LatchBio Introduces BioSecBench-Function, a Benchmark for AI Agents Working with Pathogen Data

LatchBio Introduces BioSecBench-Function, a Benchmark for AI Agents Working with Pathogen Data

On 27 August, LatchBio introduced BioSecBench-Function, a set of 111 verifiable tasks concerning the properties of viruses, bacteria, and toxins. An agent receives a genetic sequence, molecular structure, or experimental results, selects an analysis, and states a conclusion that is compared with a known answer. Among 22 combinations of language models and agent software, the best configuration completed 50.3% of the tasks when refusals triggered by biosafety rules were excluded. When refusals were counted as errors, a different configuration led with 44.1%.

When a surveillance system detects a new pathogen, researchers need to determine its properties and choose an experiment that can test their hypothesis. This decision depends on which measurements provide sufficient evidence and how their results should be interpreted together.

BioSecBench-Function turns this work into verifiable tasks. In each task, an agent reads the data, selects the appropriate analysis, and states a conclusion, which is then compared with a known answer. The benchmark assesses how well the system connects the data, its chosen analysis, and its final conclusion.

The two reported scores answer different questions. The 50.3% score includes only attempts in which the system did not refuse the task under its biosafety rules. The 44.1% score counts such refusals as errors. The authors evaluate data-based reasoning separately from the behavior of safety filters, so the scoring rule determines which configuration ranks first.

The response trajectories show what lies behind these percentages. In one task, two types of experiment produced conflicting results, so the appropriate next step was a confirmatory experiment. Almost none of the agents proposed one. In another task, measurement uncertainty differed by more than a factor of one hundred across variants. Agents that applied a single numerical threshold interpreted noise as drug resistance.

In tasks like these, an agent must interpret the data together, determine whether they provide enough evidence for a conclusion, and select the next experiment when the results conflict. BioSecBench-Function evaluates this decision along with the final answer.