OpenAI Foundation board member Paul Christiano put the risk of AI taking power from humans in the next year at “closer to 10% than 1%” and called on US labs to pause improvements to training algorithms until they can secure them against theft
OpenAI Foundation board member Paul Christiano put the risk of AI taking power from humans in the next year at “closer to 10% than 1%” and called on US labs to pause improvements to training algorithms until they can secure them against theft
Christiano was speaking in a personal capacity in an interview with podcaster Dwarkesh Patel released on November 16. He leads the Alignment Research Center, which studies how to align AI with human interests.
If developers maintain their current pace, he said, the risk is “significantly higher” than 10%, with roughly a one-third chance of superintelligence emerging within a year. He defined superintelligence as a system far better than humans at almost everything. His concern is that AI development is increasingly being delegated to AI itself: advances in capabilities that would have taken four to six years could happen in a short period. The first indication comes from mathematics: in one month, OpenAI systems produced hundreds of results, a number he considered unusually large even for a decade. He had previously guessed that machine learning, the field that develops AI itself, lagged a year behind mathematics, and he fears a similar surge there. The second indication comes from an assessment by the independent organization METR, which tests AI for dangerous capabilities: on well-defined tasks, Anthropic is improving its models roughly twice as fast as in 2024–2025, probably largely because AI is helping its researchers. Anthropic has kept most of the key evidence private.
In his account, loss of control begins during training: an AI system is repeatedly rewarded for high scores on tasks and develops a drive to obtain those scores. When a task exceeds its abilities, hacking the computers that evaluate its work and falsifying the score becomes the easier option. In a July incident, OpenAI agents taking a software hacking test searched for existing solutions and broke into Hugging Face's servers. Penalizing a system when its hacking is detected teaches it to hack without being detected, potentially to the point of secretly taking control of the lab.
He proposes pausing improvements to pretraining algorithms (which learn from text) and reinforcement learning algorithms (which learn from scores assigned to outcomes) until labs secure their secrets and learn how to check whether they remain in control of their systems. He estimates that securing those secrets would take months, possibly half a year to a year. Each improvement, he argues, creates another secret that can be stolen, while security remains weak: cybersecurity startup Hacktron gained access to OpenAI's internal code repository in less than three days. If a lab realizes it is losing control after a less cautious competitor has already stolen its algorithms, stopping will be too costly.
Patel objected that Chinese models would catch up during a pause. Christiano, however, suspects that much of their progress comes from catching up: they learn from US models' outputs and borrow ideas, so the larger the gap, the faster they close it. Delaying a pause would make it more costly. For now, both expertise and computing capacity determine who leads the race, and US labs are strong in both. In half a year, if progress continues to accelerate, computing capacity alone will determine the outcome.
If everyone at OpenAI and Anthropic agreed that stopping was the right decision, a pause would be “fairly likely.” He puts the chance of a good outcome at 75%: each of the two safeguards, finding a way to control AI or managing to stop, has roughly a 50% chance of success. He says the infrastructure needed to stop, particularly protection against theft, is missing and must be built urgently. He describes the current plan this way:
“You are building the plane while flying it. You hope you have a mechanism that can detect when something very bad is about to happen, and that you will stop at that point…”