Percentage estimates of the risk of human extinction from AI became common among rationalists accustomed to quantifying uncertainty, and a handful of foundations fund forecasts of these risks, journalist Sharon Goldman found

Percentage estimates of the risk of human extinction from AI became common among rationalists accustomed to quantifying uncertainty, and a handful of foundations fund forecasts of these risks, journalist Sharon Goldman found

On 26 October, Goldman published a report in her Ground Level AI newsletter on where these numbers come from, based on interviews with more than a dozen forecasters and researchers.

On 11 October, Eli Lifland, a co-author of the “AI 2027” scenario, wrote on X that if AI companies do not slow down, the probability of an AI takeover is about 67%, and the probability of human extinction following such a takeover is about 40%. In September, Evan Hubinger of the AI company Anthropic put the probability of all humans dying within the next ten years at more than 10%. Goldman was struck by how casually people cite these numbers, as though giving a weather forecast. A weather forecast can be checked against what happens, but how to assess a 67% probability of an AI takeover is less clear.

Futurist Anders Sandberg thinks the practice may have begun on the Extropians mailing list, launched in 1991, where futurists frequently discussed prediction markets, in which people bet on the outcomes of future events. In the 2000s, the practice spread among the rationalists who gathered around Eliezer Yudkowsky. According to Sandberg, Yudkowsky founded the community to make AI safe for humans, and learning to reason numerically about an uncertain future was an important part of that work. “When I say ‘10%,’ that is shorthand for what I believe, given what I know,” he explains.

The reliability of these estimates is another concern. Psychologist Philip Tetlock found that expert forecasts of political and economic events were no more accurate than simply extrapolating past trends, while his Good Judgment Project identified “superforecasters” with consistently high accuracy. In 2021, the Long-Term Future Fund helped fund a forecasting tournament on risks to humanity involving Tetlock. His reputation was expected to make the resulting estimates more credible to outsiders than the best estimates then available.

The tournament took place in 2022 and included 89 superforecasters and 80 experts. The median estimate of the probability of human extinction from AI by 2100 was 0,38% among superforecasters and 3% among AI risk experts. This roughly eightfold gap persisted even after months of debate. Tetlock’s Forecasting Research Institute (FRI) later compared participants’ long-term risk estimates with their accuracy on questions that could be resolved sooner and found no statistically significant relationship. According to the authors, accuracy on near-term questions so far offers little guidance on whose long-term forecasts to trust.

Since 2022, FRI and the forecasting platform Metaculus have received more than 26 million dollars from Open Philanthropy, a foundation that funds AI safety research. Philosopher David Thorstad sees a “problem of insularity”: much of the money comes from a few people with particular views on the risks, and skilled forecasters were often already concerned about AI.

In April, Marcus Abramovitch, who says he spent a year at the top of the forecasting platform Manifold’s leaderboard, called for an end to funding for forecasting. He forecasts when advanced AI will arrive for FRI and estimates that he spends half an hour a month on the task. “It is largely a matter of intuition,” Abramovitch says. “Even researchers at Anthropic base their estimates on intuition.”

Rob Bensinger of MIRI, an institute that studies the dangers of advanced AI, describes such estimates as “very, very rough even at best.” Still, he says, a number conveys what is at stake: “Either you are wrong, or the whole world needs to pay attention.”