When Risk Becomes Theatre

There is apparently a greater than 10 per cent chance that artificial intelligence will destroy humanity within the next few years.

At least, that is the impression created by some recent headlines.

Look underneath and the claim becomes less solid. One researcher said AI could kill everyone by the end of the decade, but gave no probability. Another offered a personal estimate of more than 10 per cent within the next ten years, which is not the same timeframe. A third said catastrophe could occur within a few years, again without attaching a number.

Three different statements become one frightening proposition: a one-in-ten chance of extinction by 2030.

That proposition was not the finding of a study. It was not derived from accident data, a validated forecasting model or an observed rate of failure. It was assembled from opinion, inference and headline compression.

This does not make the underlying concern ridiculous. Advanced AI could create serious harm through misuse, excessive delegation, cyberattack, biological research or the loss of meaningful human control. A risk need not be precisely measurable before it deserves attention. Aviation, medicine and engineering all act on credible hazards before bodies accumulate.

But recognising uncertainty is not permission to manufacture precision. There is an important difference between saying, “We cannot yet rule this out”, and saying, “There is a 10 per cent chance this will happen.” The first identifies uncertainty. The second claims knowledge about its size.

That difference matters because catastrophising can make us less capable of managing real risk.

A pile of concerns is not a causal chain

The argument for near-term AI extinction usually arrives as a collection of observations. AI can write malicious code. A model behaved deceptively in a laboratory test. Another exploited a loophole. Systems are becoming more capable. Companies are spending billions. Researchers do not fully understand what happens inside the models.

Each observation may be relevant. Together they create an atmosphere of danger. But they do not automatically establish the proposed outcome.

For AI to eliminate humanity autonomously, an entire chain would have to hold. Progress would have to continue or accelerate dramatically. Systems would need to outperform humans across research, strategy, cyber operations and relevant physical sciences. They would need sufficiently coherent objectives that conflict with human survival, the ability to conceal those objectives, access to consequential systems and resources, persistence after humans attempted to shut them down, and the capacity to defeat coordinated countermeasures. Finally, they would need an extinction mechanism capable of reaching isolated populations and preventing recovery.

Evidence for fragments of this chain is not evidence that the whole chain is close to completion. In a conjunctive risk sequence, every necessary link matters. Adding more examples of limited, simulated or loosely related behaviour does not repair a missing link.

Human judgement complicates this further. We do not reliably add the evidential value of arguments as though completing a calculation. We often form an overall impression, effectively averaging their apparent strength. A vivid example can dominate several sober qualifications. Conversely, adding numerous weak warnings can dilute the significance of the important one.

This is familiar from pharmaceutical information. A warning leaflet may place mild nausea, temporary dizziness and a vanishingly rare fatal reaction in the same dense field of text. It has disclosed more, but it may have communicated less. Research into multiple warnings has found that adding low-criticality warnings can reduce perceived overall risk. When everything carries a warning, the reader loses the hierarchy.

AI discourse can suffer from both distortions at once. Rudimentary deception, speculative self-improvement, job displacement, misinformation, cybercrime and human extinction are presented as one accumulating case. The volume of concerns creates weight, while their different probabilities, mechanisms, timescales and consequences disappear.

The result is not necessarily better vigilance. It may simply be cognitive overload.

Catastrophe can provide useful cover

Catastrophe narratives can be sincerely believed and still serve other purposes.

They attract attention. Negative words increase engagement with online headlines. They confer importance on the people issuing the warning. If a company says its technology may transform the world, that is marketing. If it says the same technology might end the world, it can sound like reluctant expert testimony.

The narrative can also change the political frame. Present AI as an immediate threat to human existence and today’s more ordinary questions can appear trivial: Who owns the data? Whose work was used? Who benefits from automation? Where is decision authority moving? Are organisations removing human expertise before systems are dependable? How much energy and infrastructure are being consumed? Who can afford to comply with the proposed regulation, and which competitors might it exclude?

These are not objections to taking extreme risks seriously. Nor does the existence of commercial or political incentives prove that a warning is false. Motive is not evidence either way.

It does mean we should ask what becomes easier, more profitable or less visible when the conversation moves towards extinction. Sometimes the future catastrophe is so captivating that it protects the present arrangement from scrutiny.

There is another subtle benefit. An unbounded catastrophe claim is difficult to falsify. If the predicted event does not occur, advocates can say the date has moved, the probability was only subjective or that their warning helped prevent it. Meanwhile, every improvement in capability is treated as confirmation, even when none of the critical causal links has been demonstrated.

This is not a sound basis for public risk management.

Rasmussen and the boundaries that matter

The safety scientist Jens Rasmussen offered a more useful way to think about risk in dynamic systems.

Organisations operate within several competing boundaries. There is a boundary beyond which performance becomes functionally unacceptable and people are harmed. There is a boundary of economic failure, beyond which the organisation cannot remain viable. There is also a boundary of unacceptable workload, beyond which people cannot continue to carry the burden.

Work does not remain neatly in the safe centre. Pressure for efficiency, lower cost, faster delivery and less effort gradually moves activity. People adapt locally. Shortcuts that work become normal. Margins narrow. The organisation drifts towards a boundary that may be difficult to see until it is crossed.

This is a better lens for AI than a single argument about whether extinction is imminent.

Where are organisations being pushed towards excessive dependence on systems they cannot adequately inspect? Where is human oversight still claimed but no longer practically possible? Who has authority to stop deployment? What commercial pressure rewards moving faster than governance can adapt? Which signals would reveal that the margin is narrowing? What recovery capacity remains when a model is wrong, compromised or unavailable?

These questions recognise that risk emerges from the interaction of technology, incentives, workload, authority and organisational design. The model is rarely the whole system. Human beings choose its access, permissions, objectives and operating environment. Leaders decide whether warnings are investigated, whether awkward evidence travels upwards and whether somebody can still intervene.

Rasmussen’s advice is not to predict the precise moment of catastrophe. It is to make the boundaries visible, monitor movement towards them and preserve enough margin to recover.

For AI, that means distinguishing capability demonstrations from operational reliability; limiting access and autonomy according to consequence; retaining meaningful human judgement; testing failure and recovery, not merely benchmark performance; monitoring weak signals; protecting challenge and escalation; and reviewing controls as both the technology and its use change.

It also means ranking risks. A speculative extinction pathway and a documented security vulnerability may both deserve attention, but not the same evidence label, owner, timeframe or response.

Neither complacency nor catastrophe

The choice is not between unrestricted acceleration and technological extinction. That false binary may be the most distracting claim of all.

Human progress has rarely come from eliminating risk. It has come from learning how to engage with risk intelligently. Commercial aviation did not become transformative because aircraft were declared perfectly safe. It progressed through better design, disciplined operation, reporting, investigation, training, redundancy and the continuous revision of what safe enough meant.

AI can extend human capability in science, medicine, education, accessibility, engineering and everyday work. It can help people see patterns, test ideas, translate knowledge and complete tasks that previously consumed scarce time. Those benefits are neither automatic nor evenly distributed, but they are real enough to pursue.

Responsible optimism begins by refusing two temptations: pretending the technology carries no serious hazards, and pretending our darkest imaginable scenario is already a quantified forecast.

We can recognise emerging danger without surrendering judgement to fear. We can demand evidence without waiting for certainty. We can impose limits without abandoning invention. We can preserve human authority while using machines to amplify human intelligence.

Good risk management does not predict the apocalypse. It makes danger visible early enough to act, preserves the capacity to recover, and makes valuable progress safe enough to continue.

That is not complacency. It is how human beings have turned dangerous technologies into instruments of progress before, and it is how we can do so again.