Nick Bostrom calculates that the optimal pause in developing superintelligence is months or a few years, because each year of delay costs millions of lives
Nick Bostrom calculates that the optimal pause in developing superintelligence is months or a few years, because each year of delay costs millions of lives
On November 19, venture capital firm a16z released a conversation with philosopher Nick Bostrom about the pace of developing superintelligence, AI that surpasses human intelligence. The calculation considers only people alive today, and Bostrom regards it as just one consideration.
In the episode of the a16z Show, Bostrom responded to Yudkowsky and Soares’s book “If Anyone Builds It, Everyone Dies,” whose authors call for a global ban on advanced AI. He said there is just as much evidence for the converse claim, “if nobody builds it, everyone dies”: 95% of all people who have ever lived are already dead, the rest will follow within a few decades, and safe superintelligence could deliver breakthroughs against aging.
“If you think about what our life expectancy would be without aging, it would be around 1200, perhaps 1400 years. And that is probably quite a conservative lower bound: in that situation, we could also reduce other risks of death, such as accidents.”
The figure of 1400 comes from Bostrom’s calculation in a working paper: if people at every age had the same low mortality rate as healthy twenty-year-olds in developed countries (about 7 per 10 thousand per year), average lifespan would be about 1400 years.
In the calculation, each year of AI safety work reduces the risk of catastrophe, but millions die during each year of a pause. One host put the figure at around 150 thousand deaths a day; Bostrom agreed and added: “We must not delay a single day longer than we really need to.”
If the risk of catastrophe falls quickly during a pause, a few months will suffice. If it falls slowly, waiting is not worthwhile: more people will die of natural causes in a year than the calculation predicts the pause would save. A longer wait is justified only when risk falls at an intermediate rate. Across a broad range of assumptions, the optimal pause is “months or a few years, rather than decades.” He believes the largest safety gains may come at the beginning of a pause, when a nearly complete system can be studied and cautiously deployed; after a few years, safety improves more slowly.
Life expectancy, he said, is increasing by two to three months a year, but it needs to increase by “a year per year” for remaining life expectancy to stop shrinking. He considers that pace unlikely without superintelligence. A longer wait could be justified if aging could be slowed by other means.
Bostrom “has not been a big enthusiast of an AI halt or pause,” but sees merit in “regulating the pace at the frontier”: he would consider a brief pause or a modest slowdown if safety work falls behind. He would be concerned if a slowdown were sustained by hostility toward AI and global surveillance of data centers. Such surveillance could become entrenched as an “unquestionable orthodoxy,” leading humanity to forgo superintelligence forever, an outcome he would regard as a catastrophe.
The acceptable risk of catastrophe, Bostrom argues, may depend partly on how one assesses the best possible outcome. If that means the same people with new medicines, a moderate risk is justified; if “we haven’t seen anything yet,” there is greater willingness to “take more risk for a chance at that.”
Adam Gries, co-founder of the anti-aging movement Vitalism, agreed in response to the a16z clip, but added that the argument “may ring somewhat hollow” unless AI is deliberately applied to aging. Psychologist Geoffrey Miller objected: risking one’s family and species for the hope of achieving longevity within a couple of decades is not worth it; investing another “tens of billions” in longevity research would be preferable.