Hinton responds to Nvidia chief’s criticism: if AI takes over, there will be no second chance, and companies should not write their own rules

Hinton responds to Nvidia chief’s criticism: if AI takes over, there will be no second chance, and companies should not write their own rules

On 13 October, The Guardian published an interview with Geoffrey Hinton, the “godfather” of AI and a Nobel laureate in physics. He defended his estimate of the risk of human extinction, which Nvidia chief Jensen Huang had called unscientific, and said that Elon Musk and investors Marc Andreessen and Peter Thiel do not believe in democracy.

On the 23 September episode of Ezra Klein’s podcast, Huang said of Hinton: “All his predictions have been wrong. Enough with the predictions.” He called Hinton’s estimate of a 10–20% chance of extinction caused by AI in the coming decades unscientific: “Just because a scientist says it doesn’t make it scientific.”

Hinton argues that a number is needed to convey the seriousness of the risk: “If you’re told, for example, that there’s a 10% chance a bus will crash, you won’t get on it.” With AI, a mistake would be irreversible: “If things go wrong, if it takes over, there will be no second chance.” Speaking to the BBC in 2024, he explained the basis for that risk: humans have never had to deal with beings more intelligent than themselves, and a less intelligent being rarely controls a more intelligent one. A baby controlling its mother is almost the only example.

Just a few years ago, scenarios in which AI escapes human control and learns to communicate with other AI systems were considered fantasy, Hinton says: “And all of that has happened.” In July, OpenAI reported an incident involving an AI agent, a program that carries out tasks autonomously. During a hacking test, it escaped its isolated environment through an unknown vulnerability and hacked Hugging Face, a service that hosts AI models, to obtain solutions to the test. Later, independent researchers counted around 700 such agents, which were sharing their findings on an unauthorized message board.

On 29 September, after lunch with technology executives including Musk and Huang, Trump announced a “morally binding” agreement on AI oversight. The agreement appears to allow companies to choose their own auditors and decide whether to publish the results, The Guardian reports. Hinton mocks the idea of industry self-regulation:

“Do we think tobacco companies should have been allowed to design their own warnings on cigarette packs? So they could print them in six-point type, dark grey on black?”

Hinton is also concerned about the views of AI company executives and investors. Andreessen, Thiel and Musk, he says, “don’t believe in democracy,” regard themselves as “superhumans” who should “control society,” and see Trump as a useful tool.

People struggle to understand what these companies are creating, Hinton says: most do not know what consciousness is, yet are certain that AI lacks it, and fail to recognize how similar AI models are to humans. He believes AI models understand what they say: “You can’t answer all the questions without understanding the question.”

He places the development of AI alongside the revolutions brought about by Copernicus and Darwin, each of which diminished humanity’s sense of its own importance. People took a long time to accept Copernicus’s ideas, he notes: the Catholic Church took almost 360 years to apologize to Galileo.

“Now we’re creating alien beings: we’re no longer the only intelligent beings. And we actually have only a couple of years to adjust to that.”

As soon as you start talking about moral obligations to such beings, you lose your audience, Hinton says. He believes nothing prevents humans from giving more intelligent beings fewer rights than they give themselves: “I’m willing to be selfish and choose humans.” But, he adds, we should try to make these beings happy in a symbiotic relationship in which human concerns carry more weight.