BixBench3 Tested How AI Agents Reproduce Biological Analyses from Raw Data

BixBench3 Tested How AI Agents Reproduce Biological Analyses from Raw Data

On August 26, Edison Scientific introduced BixBench3, a benchmark comprising 20 tasks based on published studies. An agent receives the research objective, methodological instructions, and raw data, and its outputs are compared with those of the original study. Among 13 models, the highest mean score was 48%.

Reproducing a published analysis begins with raw measurements. These measurements must be processed through a sequence of intermediate outputs. For example, the analysis may first produce a gene activity table, then a list of genes that differ between groups, and finally the cellular processes associated with those genes. An error at an early stage affects every subsequent stage in the sequence.

BixBench3 evaluates this type of work. For 20 published studies, the authors collected the available raw data, prepared a research objective and methodological instructions, and divided each analysis into a total of 138 assessable outputs. In each task, the agent produces between four and 14 outputs, including tables, gene lists, and results from subsequent stages of the analysis. The benchmark compares these outputs with the reference results from the original study. An output passes if it preserves the study's main biological interpretation.

GPT 5.6 Sol achieved the best result, with an average of 48% of its outputs passing evaluation. Across tasks with more than 100 GB of source data, all models passed an average of 10% of outputs. The corresponding figures were 34% for tasks containing 50–100 GB and 37% for tasks containing less than 50 GB. When an output depended on three or more consecutive analysis steps, the mean pass rate was 24%.

In BixBench3, a scientist specifies the research objective and the general methodological framework in advance. The agent then executes the sequence from data to results, while the researcher retains responsibility for selecting the scientific question and method.

In the earlier BixBench, agents answered questions based on real bioinformatics scenarios. BixBench3 extends this approach by evaluating a connected sequence of outputs in which each subsequent step depends on the preceding one.