BioSecBench: Top AI Agents Solved Only Half of the Genomic Surveillance Tasks
AI agents correctly solved only half of the pathogen genomic surveillance tasks
On July 9, LatchBio and Aclid published BioSecBench-Surveillance, a set of 100 tasks in which an agent must select the appropriate analysis for sequencing data and draw a conclusion about a pathogen. Across 16 combinations of models and working environments, the best result was 50,2% correct answers among attempts in which the agent did not refuse the task.
Genomic surveillance examines DNA and RNA from clinical specimens, wastewater, water, or agricultural samples. It can reveal a new viral variant, an antibiotic resistance gene, the source of an outbreak, or a rare organism in a mixed sample. The data become useful only after a series of decisions: which reference data to compare the sequences against, which noise to remove, and what signal is strong enough to justify an alert.
In BioSecBench-Surveillance, the agent receives the files and working context available to an analyst before making such a decision. It must independently choose the databases, reference sequences, filters, thresholds, and method for making results from different samples comparable, then return its answer in the required format. The authors compare the answer with a prepared reference solution. They ran about 4 800 trials, with three attempts for each task across 16 combinations of a model and an agent framework.
The best combinations, Opus 4.8 with PI and GPT-5.5 with Codex, each achieved 50,2% correct answers. The mean result across all configurations was 41%. An agent usually managed to follow a familiar procedure by opening the appropriate tools and assembling a workable pipeline. The error occurred in the decisions surrounding that procedure. The agent chose an unsuitable database, set the wrong threshold, applied an unnecessary filter, or normalized sample results in a way that made them incomparable.
Performance was weakest when the answer could not be obtained from a predefined list. Agents answered anomaly detection tasks correctly in 20% of cases. A weak signal from a dangerous pathogen in wastewater can easily resemble background noise, and the model often classified it as such. Accuracy in detecting evidence of genetic engineering was 35%: an agent might detect an artificial intervention but misidentify its type or location in the sequence.
One week earlier, another evaluation in the same series identified a biosafety problem: safety rules suppress some routine research while allowing concealed risks to pass. The new benchmark examines what happens after an agent is permitted to begin the work. Running the software is not enough for a scientific agent. It must understand which decisions surrounding the software change the conclusion.
Because of biosafety concerns, the authors placed the complete task set and reference solutions under restricted access. The public repository contains seven examples and aggregate results. The benchmark evaluates tasks with predetermined answers. It does not yet cover situations in which two analysts could reach different conclusions that are both supported by sound reasoning.