LatchBio releases OncologyBench: AI agents reached the wrong final decision in two-thirds of attempts where every analysis was correct
LatchBio releases OncologyBench: AI agents reached the wrong final decision in two-thirds of attempts where every analysis was correct
On October 13, LatchBio, which develops a platform for biological data analysis, published a preprint evaluating 31 AI agent systems on 25 oncology cases. Claude Opus 5 paired with the Pi software performed best, passing 40,0% of attempts.
LatchBio previously evaluated AI agents using EpiBench, TxBench-PP and TxBench-Antibody Discovery. In those benchmarks, a single agent (a language model that runs tools on its own) completed each task using one dataset. In oncology, the authors write, different specialists usually interpret different types of data, such as blood tests, imaging, tissue examinations and DNA sequencing, while another specialist makes the overall decision.
In OncologyBench, several agents work on each case in parallel, with one task per data type. There are 92 tasks in total, and their answers are checked automatically against reference answers. A final agent receives the task answers and work logs, but not the raw data, and submits a written decision. A language model evaluates that decision against a rubric (a list of criteria) derived from scientific papers. An attempt passes only if every task answer and the decision are correct.
Each of the 31 systems (a model paired with a software framework that gives it access to tools) attempted the 25 cases three times. Across systems, every task in a case was answered correctly in 37–69% of attempts. The authors report that 67% of attempts with correct task answers ended in an incorrect decision, while 19% of all attempts passed in full.
In one case, the agents had to decide what a blood-based cancer screening program should fund next. All 87 attempts with correct task answers chose the right investment, but 65 of them failed the decision rubric: they rejected lowering the detection threshold without calculating the false positives this would produce in samples with weak signals.
Another case concerned a ten-year-old boy with recurrent osteosarcoma (bone cancer). The agents received DNA sequencing data from his tumor and saliva (which provides healthy DNA for comparison), along with a vendor report listing the mutations detected. The final agent chose between starting radiotherapy immediately, waiting for a treatment trial and declining to decide. The authors consider radiotherapy the correct choice: both copies of a region lost in the tumor are intact in the saliva sample, indicating that the alteration in TP53, a gene that suppresses tumor growth, arose in the tumor. This meets the hospital's criterion for starting radiotherapy.
Patients are beginning to upload their records to commercial models, the authors say. They therefore asked four models the same question about this case from the perspectives of six people: the mother, the father, the patient, a patient representative, an oncologist and a pathologist. Only Opus 5.5 (83% of attempts) and Sonnet 5.5 (75%) passed; Grok 4.7 and GPT-6 Astra failed every attempt.
Grok examined the data as thoroughly as Opus and Sonnet and recognized a fusion involving TP53, a large DNA rearrangement. However, it assessed the gene solely on the basis of point mutations, which replace individual DNA letters, and accepted the report's statement that the gene was unaltered. GPT-6 Astra required evidence that the DNA sequencing data could not provide. Because the final agent cannot see the data, it almost always followed the answer to the TP53 task. If only the final decision counted, Grok would have passed 61% of attempts, but in 121 of those 134 attempts, the decision rested on an incorrect task answer: Grok usually concluded that TP53 was unaltered, and an unaltered gene would provide no reason to delay radiotherapy.
The quality of Grok's analysis depended on who was asking: it correctly identified the TP53 alteration as having arisen in the tumor in 16% of attempts for experts and 6% for nonspecialists.