Anthropic’s Evan Hubinger puts his personal estimate of the risk that AI will kill everyone within the next ten years above 10%
Anthropic’s Evan Hubinger puts his personal estimate of the risk that AI will kill everyone within the next ten years above 10%
On September 9, Hubinger, who leads AI alignment at Anthropic, replied to a thread by his former colleague Jacob Coxon. He endorsed Coxon’s warning about the race toward self-improving superintelligence and wrote that Anthropic does not yet have a plan to align such a system with human interests.
Coxon announced his departure from Anthropic that same day. In his thread, he described a race toward self-improving superintelligence, urged researchers to seek different conditions for AI development, and suggested that a temporary ban on increasing model capabilities might be needed to prevent a global race.
In his reply to the thread, Hubinger offered his personal estimate and identified an unresolved problem:
“Jacob is right: we really do take seriously the possibility that AI could kill everyone. Personally, I think the probability of that happening within the next ten years is above 10%. I think Anthropic is doing everything it can, but we do not yet have a plan to align superintelligence with human interests or a clear path toward one.”
AI alignment is the task of ensuring that a system’s goals and actions serve human interests. Superintelligence refers to an AI system whose capabilities exceed those of humans. Hubinger is discussing the alignment of such a system.
Anthropic has a public safety policy, the Responsible Scaling Policy. It ties model capability thresholds to safety measures, and the company publishes risk reports. Hubinger is addressing a separate question: how to align the goals of a system that will surpass human capabilities with human interests in advance.
In Coxon’s account, Anthropic is participating in the race because it believes it will have to take responsibility for acting responsibly itself. He proposes seeking coordination; Hubinger identifies the technical problem that such coordination must allow enough time to solve.