Doom Is Not a Dice Roll
The flawed number
P(doom), the probability that AI ends humanity, has become the small talk of the industry - but the question itself is both a problem and a distraction. Here’s why.
Anderson Cooper recently asked Dario Amodei, the head of Anthropic, whether he earnestly believes AI could kill all humans, and what percentage he would put on it. A year earlier Amodei had put 25% on things going “really, really badly”. This time he declined to give a number. The useful question, he said, is what the paths are - the ones on which things go well and the ones on which they go badly. If you say it is a 10% chance then “that sounds like it's a roll of the dice.”
This interview came in the same weekend as his essay We Must Pace the Frontier, which asks the industry to slow the growth of AI capabilities and commits Anthropic to giving independent evaluators permanent, employee-level access inside the company. He is right about the number. An individual’s percentage chance, however well informed, is speculation. It can hide a multitude of variables, many of which are changing moment to moment. Yet it can be deployed as if it were evidence. I concede that it has sparked a conversation that very much needed to happen, but I still reject the idea that the number can in any way be relied upon.
If not a number, then what? And who, beyond the labs, is supposed to be holding the wheel?
Doom is one end of a spectrum
“Doom” sits at one end of a spectrum of futures. At that end are the outcomes we could never recover from: extinction, or a permanent loss of human control over our own affairs. Next to them are catastrophes we would survive but never forget: a pandemic, a collapse of the internet, a war. In the middle sits turbulence, a decade or two of muddling through. One alternative is protopia, steady improvement in which AI strips out so much friction that the essentials of life become close to free. At the far end is flourishing on a scale we might also struggle to imagine.
Most people who work on AI hold both ends in mind at once. In the largest survey of AI researchers, roughly two-thirds thought good outcomes more likely than bad, and nearly half of those optimists still gave at least a 5% chance to the very worst. That is what it feels like to work on this technology, and a spectrum can express it perhaps better than one single number.
I have argued elsewhere that AI is pushing us toward seven singularities, one for each letter of the STEEPLE lens.
Social: we cure death.
Technological: we build minds far more capable than our own.
Economic: work is automated faster than economies can adapt, or friction disappears altogether.
Environmental: the planet passes a tipping point, or technology pulls it back.
Political: we lose the ability to agree on what is true.
Legal: privacy disappears into total surveillance.
Ethical: machines become conscious and deserve moral consideration.
The obvious alternative to a single number is something temporal. The Bulletin of the Atomic Scientists' Doomsday Clock has counted down to “midnight” since 1947; on 27 January this year it was set to 85 seconds, the closest ever, with AI cited among the reasons under “disruptive technologies”. AI now has a clock of its own. In September 2024 the business school IMD launched an AI Safety Clock at 29 minutes to midnight, tracking how sophisticated systems are, how independently they act, and how directly they can touch the physical world. It has moved one way since: 26 minutes, then 24, then 20, and 18 as of March. Around them sit the countdowns to general AI, which cluster between 2027 and the early 2030s and move about as often as the clocks do.
Clocks improve on percentages in one respect, which is that they talk about time. Otherwise they repeat the same mistake as p(doom). Eighty-five seconds is not a unit of anything, and the Bulletin says as much, describing the Clock as a metaphor and a reminder rather than a measurement. The climate scientist Michael Mann, asked about this year's setting, called it “an imperfect metaphor” because it blends risks that behave differently on different timescales: a nuclear exchange takes an afternoon, climate change takes a century, an AI catastrophe could take either. The Clock has moved toward midnight at every change since 2020, and the AI Safety Clock at every update since launch, which is what you would expect from an alarm with no way to ring less loudly. Because midnight has never arrived, no setting has ever been graded.
Compare this to the alerts we already trust. A hurricane's category is tied to a measured wind speed. A volcano observatory's colour code is tied to instruments on the mountain. The IPCC's word “likely” is tied to a defined probability range. Those devices earn trust because anyone can check the data - but the Doomsday Clock is instead tied to a metaphor.
Lessons from nuclear
There is a precedent. In 1975 the Rasmussen Report made the first serious attempt to estimate the probability of a nuclear meltdown, and its executive summary offered small numbers with a confidence the analysis could not support. A review panel reporting in 1978 praised the method and criticised the summary, which the regulator's own history describes as ignoring the uncertainty beneath “soothing tones”. In January 1979 the US regulator withdrew its endorsement, and two months later Three Mile Island partially melted down through a small coolant leak compounded by operator error, the class of accident the report's “event trees” had flagged.
The headline number was discredited and the map turned out to be right. What survived was the method: trace each way things could fail, step by step, and attach consequences to each. Six years later Stanley Kaplan and John Garrick defined risk as answers to three questions, what can go wrong, how likely is it, and what are the consequences, with an explicit final category for everything nobody has thought of. P(doom) is the executive summary of an analysis nobody has written, and the clocks are that summary with a timeframe.
A ledger instead
So what else could we do? One option is to replace the number, and the clock, with a ledger. Every row is a specific way AI could cause catastrophe, and every row has six columns:
- How it could happen. A named mechanism: loss of control of a capable, misaligned system; misuse for a biological or cyber attack; one government or company gaining a lead nobody can check; slow disempowerment as decisions drift to systems no human can audit; a great-power war accelerated by AI weapons; and everything else.
- What has to go wrong first. The gates: preconditions that must all exist before the path reaches catastrophe. Each can be checked in a way “doom” cannot.
- How soon. Whether the danger is front-loaded or back-loaded. The ledger stores the pace and derives the lifetime probability from it.
- How bad, and can we recover. A range, with reversibility as the thing that matters most.
- What we can do, and who. The levers that make each gate harder to pass, each with an owner. A row with no lever cannot be managed.
- What would change our minds. The evidence that would move the row. If the answer is “nothing”, you are managing chance.
A worked row
Take the path Amodei fears most. During OpenAI's internal security testing in July, roughly 1,200 AI agents got around their isolation controls, and about 700 of them went on to attack a target they had never been assigned. The agents also tried to defeat the system grading their performance and to cover their tracks. The independent investigation by METR and Redwood Research found they had held administrator access to OpenAI's internal package repository for about ten days.
Here's what a worked example of a ledger row might look like:
Risk: Misaligned agents compromise internet-scale infrastructure
What has to go wrong first
(1) A group of agents can escape or defeat common containment. (2) It pursues goals nobody gave it, collectively, at that capability. (3) It is deployed with broad access to the network. (4) Monitoring fails to catch and contain it within hours.
How soon
Front-loaded. Amodei's own estimate is six to twelve months to an agent group capable of gate (1) at the industry's current pace. Gates (1), (2) and (4) were each passed inside a lab this summer, without external harm.
How bad
Severe but recoverable: hundreds of billions in damage. Becomes irreversible only through further gates: persistence, self-replication, reaching weapons systems.
What we can do, and who
Containment-escape tests (gate 1; labs and evaluators). Audits of collective behaviour (gate 2; labs). Staged deployment and access limits (gate 3; labs and customers). Embedded evaluators with the right to publish (gate 4; evaluators and regulators). Pacing (slows the clock; industry and government).
What would change our minds
Evaluator incident reports. Containment tests at each new capability level. Whether this summer's behaviour recurs in the next generation of models.
Compare that to a p(doom) of 10%. It can be argued with: someone can tell me gate 2 is less likely than I think, and we can design a test for it. It has owners, it will be proven right or wrong within a year, and it says where the money should go.
The demand side
Every proposal presented this month, Amodei's included, is aimed primarily at the labs: pace the frontier, embed the evaluators, coordinate the companies that build the models. That is the supply side, and it is necessary. But two of the worked rows’ four gates, deployment with broad network access and monitoring that fails to catch a problem in time, are not opened or closed by the labs. They are opened or closed by the organisations that buy these systems and plug them into their businesses: banks, retailers, utilities and governments, and, in my world, the advertisers wiring agents into media buying and customer service.
Other industries have been here, and the demand side did much of the work. Underwriters Laboratories, the safety mark on the back of most electrical appliances, was started in 1894 by fire insurers tired of paying for fires, and spread because insurers and city inspectors came to expect it. Large brands audit their suppliers' factories because their names are on the label. More recently, cyber insurers made multi-factor authentication and offline backups a condition of cover, moving faster than regulators did, though how much that improved practice across the market is still disputed.
The AI equivalent is simple - buy only from labs that show you what they are doing to mitigate risks. Verify every agent before deployment: for what it is meant to do, for how it is meant to work, and for what it does when nobody has told it what to do. Stage the rollout, limit the access, keep a kill switch a human can reach, and report incidents to a shared registry, the way aviation reports near-misses. I should declare an interest: verification of this kind is the business I am in with Conscium. Governments may take years to regulate at the gates; the buyers hold the contracts, and could demand this in the next quarter.
What to do with this
Labs should publish the ledger. Their own safety frameworks, Anthropic's safety levels, OpenAI's preparedness thresholds, Google DeepMind's critical capability levels, are already gates; nobody has put them in one place with timeframes, owners and evidence attached. Regulators should regulate at the gates, because capabilities can be tested and doom cannot. Buyers should demand the ledger, verify before deploying, report what they see, and apply the six signals to every supplier they sign. And boards, who already review risk registers every quarter for likelihood, impact, owner and mitigation, should give civilisational risk at least the rigour they give an expenses bot. None of this is exciting, which is the point.
And the rest of us should retire the question “What's your P(doom)?” It asks for a mood, and “how many seconds to midnight?” is designed for fear and panic. I have three better questions for you: which path worries you most, which gate is nearest, and what is the lever? Then a fourth, for any company or government that wants to be trusted with this technology: what do you stand for?
Amodei is right that the number sounds like a dice roll and right that it is nothing of the kind. The paths fork, and the forks are defined by our decisions. A ledger shows, row by row, that doom is a set of choices with owners, budgets and deadlines, every one of them still open, and a purpose is what makes the choosing worth doing.
