OmicsBench v1 preprint brings together 784 biological data analysis tasks: leading AI agents solved 49,4% of short tasks and 32,8% of long tasks
OmicsBench v1 preprint brings together 784 biological data analysis tasks: leading AI agents solved 49,4% of short tasks and 32,8% of long tasks
On 19 October, the LatchBio team released the OmicsBench v1 preprint, a common benchmark for AI agents that analyze omics data: large datasets of RNA, DNA, and protein measurements. The authors combined eight existing task sets and compared 31 pairings of a language model with a software environment that provides access to tools and data.
For each task, the agent receives an isolated workspace containing a snapshot of real data, a scientific question, a time limit, and an automated assessment of its answer. The authors reproduced published analyses and divided them into steps, from selecting and preparing data to choosing a method and interpreting the result. OmicsBench evaluates how an agent works through this sequence from data to conclusion.
The benchmark contains 741 short tasks and 43 long tasks. A short task tests a limited part of an analysis; a long task requires several successive decisions. On short tasks, Grok 4.7 paired with Grok Build achieved 49,4%; GPT-6 Astra with Pi achieved 49,3%, and Opus 5 with Pi achieved 47,7%. On long tasks, GPT-5.6 Sol with Codex achieved 32,8%.
The authors’ timeline shows that the highest score on short tasks rose from 32,8% on 17 February to 47,3% on 24 July; the next three releases brought it to 49,0%. The highest score on long tasks, 32,8%, was set on 9 July and remained unbeaten after eleven subsequent model releases.
Six days earlier, the more narrowly focused OncologyBench had shown a similar problem: in 67% of attempts, the agent performed individual analyses correctly but reached an incorrect final oncology decision. OmicsBench extends this assessment from 25 clinical cases to six types of biological data.
The authors examined 4 556 unsuccessful sequences of actions across four configurations. In 49,7% of cases, the agent chose a threshold, method, or subset of data without checking its choice; in roughly one in seven cases, it ignored a result from its own analysis that contradicted its answer.