Kevin Esvelt: 36 of 38 DNA synthesis companies sent him fragments of the 1918 influenza virus, and a language model showed him a way to create a biological weapon he had not considered

Kevin Esvelt: 36 of 38 DNA synthesis companies sent him fragments of the 1918 influenza virus, and a language model showed him a way to create a biological weapon he had not considered

On September 13, physicist Sam Rodriques, head of the AI laboratory FutureHouse, and David Bellamy publicly described concerns about AI-enabled biological weapons as alarmism. Three days later, on September 16, Kevin Esvelt, creator of the DNA order screening system SecureDNA, responded to his friend Rodriques: experts' statements shape political willingness to invest in protection, and remaining silent now costs lives.

Rodriques, a FutureHouse co-founder who was named to the TIME100 AI list, wrote on X:

We should avoid talking about AI-enabled biological weapons as though they were literally the end of the world.

His argument is that evolution will quickly eliminate any mutation that impairs a virus's ability to spread. That same day, David Bellamy, who has trained language models and synthesized viruses in the laboratory, called these fears “complete nonsense.”

Their argument rests on a limitation of conventional biosecurity: it assesses risk based on viruses that natural selection has favored for their ability to spread. An engineered agent has not undergone that selection.

Esvelt's first piece of evidence is a paper in Nature Communications: researchers posing as a student without a laboratory ordered short gene fragments from the 1918 influenza virus and the gene encoding ricin, a potent plant toxin, from 38 companies. 36 of 38 suppliers shipped the orders, and the fragments were sufficient for several complete assemblies. The synthetic DNA for a virus that once killed more than 50 million people costs about 3000 dollars. Screening systems detect very few fragments shorter than 50 base pairs. SecureDNA detects them, but its use is optional. Only one company requested proof of identity, prompted by the ricin order; it missed the influenza fragments. The vulnerability extends beyond short fragments: an AI-rewritten ricin gene retaining the same lethal function also passed DNA suppliers' screening filters.

His second piece of evidence comes from personal experience:

A language model has already revealed a new kind of biological weapon to me that I had not realized was possible. This required little creativity from the model, only a deeper knowledge of several narrow areas of biology than I had.

The model identified a possibility that one of the world's best-prepared specialists had missed. In this instance, the claim that AI still falls short of experts could not account for what it contributed.

Esvelt accepts that evolution eliminates mutations that impair transmission. He has spent years making such mutations persist in the laboratory. But a virus only needs to remain stable long enough to cause a single outbreak, and a model that selects immune-evading mutations in advance undermines reliance on vaccines as the sole protection.

Alongside these warnings, his laboratory is developing a defense: SecureBio reads DNA and RNA from wastewater and swabs to detect an unknown pathogen before an outbreak. This provides a line of defense if DNA screening has already been bypassed.

Esvelt also has a proposal for AI models. GPT-Rosalind, OpenAI's biology model, is currently available only to organizations through a trusted access program. He proposes extending access to researchers with verifiable scientific publications and excluding molecular biology data entirely from the training of open models, which users can download and run themselves. The arrangement would require comparable access for China; otherwise, China would have to build a separate closed model. He expects even superintelligent AI to need time to master the physical world: biological design models are constrained by the shortage of experimental data linking genes to their functions, rather than by model size.

The debate over how quickly AI will advance is taking place after a demonstrated failure of existing safeguards: screening at 36 of 38 suppliers allowed fragments of the virus through.