Thirty AI agents proposed 18 ways to repeat one bacterial experiment, and a Galaxy server log helped identify unspecified details in a Paper2Agent demonstration as well, preprint authors report

Thirty AI agents proposed 18 ways to repeat one bacterial experiment, and a Galaxy server log helped identify unspecified details in a Paper2Agent demonstration as well, preprint authors report

On 28 October, the team behind Galaxy, an open platform for biological data analysis, posted a preprint on bioRxiv, a manuscript that has yet to undergo peer review. The authors compared plans from thirty agents, ten each from Anthropic (Claude), OpenAI (Codex), and Google (Gemini), then repeated seven published analyses on Galaxy.

All thirty agents received the same task: use a published paper to plan how to repeat an experiment. In that experiment, the bacterium Bacillus cereus (strain ATCC 10987) was grown without supplements and with 0,5% glucose, and RNA reads were used to compare the expression of its toxin genes.

The paper's methods omit two details that affect the final numbers: the genome used to align the RNA reads and whether the statistical model included all 19 samples or only the five being compared. Of the 30 plans, 29 differed from the procedure used in the paper, and the thirty plans comprised 18 distinct analysis workflows. All ten Claude plans used the genome of a related strain, Bacillus pacificus anQ-h4, and a joint model of 19 samples; the Codex plans used the genome of the experimental strain and five samples.

One agent from each company then ran its plan on a local computer. All three found changes in the same direction as the paper: glucose suppresses expression of the toxin gene nheB and increases expression of entFM. When reads are aligned to the ATCC 10987 genome, the cytotoxin K gene (cytK2) shows expression in every sample; the anQ-h4 genome lacks this gene, leaving its RNA reads with no matching sequence. The original paper reports that the cytotoxin K gene was not expressed under either condition, and the genes in its supplementary material carry identifiers from the anQ-h4 genome.

According to the authors, an agent's account of its work omits parameter changes and failed runs. On Galaxy, they therefore recorded every step taken by the Gemini agent in a server log that the agent cannot edit. A researcher saw in the log that the agent had chosen HISAT2 for alignment, a program for organisms with introns in their genes, whereas bacteria lack introns. The researcher directed the agent to Kallisto, which is about five times faster and produces nearly identical results.

An agent can ignore a written instruction: open models followed the rule to request approval before saving a plan in 0 of 15 runs. In response, a development version of Orbit, the application through which an agent works with Galaxy, uses a software wrapper that prevents the agent from marking a plan step as complete while the server reports that the associated job is still running.

Orbit reproduced on Galaxy a demonstration of Paper2Agent, a system that builds an agent from the authors' tutorials and checks its output against their results. In a paper in Nature, one such agent used a tutorial for the TISSUE method to predict Acta2 expression in mouse cortical cells. The paper references a data archive containing 5613 cells, while the tutorial uses 1000 of them; from the colour scale in the Nature figure, the preprint authors concluded that the demonstration used 1000 cells. Their calculations put the correlation between predicted and measured expression (where 1 indicates a perfect correlation) at three times higher for the full dataset: 0,2553 versus 0,0861.

The authors consider the main output of an agent-conducted study to be an immutable archive of input files, workflows, and results, hosted on Zenodo with a persistent identifier (DOI). They describe Orbit's guarantees as follows: “These guarantees apply to the record, rather than the science: Orbit does not make the agent's analytical decisions correct, but the logic of its software wrapper ensures that the agent's actions can be verified.”