AI Agents for Auditing Scientific Literature

AI agents are beginning to audit scientific papers by rerunning available experiments and comparing the results with the reported findings

On August 6, Nature described how the SAI Labs team audits papers and their supplementary materials. The team examined all 168 oral presentations from the ICML 2026 machine learning conference and completed 105 full reruns.

A paper reports a result, but reproducing the experiment requires the code, data, software library versions, and run parameters. When these materials are available, another researcher can follow the same sequence of steps and compare the resulting numbers with those reported in the paper.

The SAI Labs team identifies the main testable claims in each paper, obtains the materials supplied by the authors, and reruns the experiments when possible. It then publishes a report stating whether the result matches what the paper reports.

Of the 168 ICML oral presentations, the team completed 105 full reruns. The agents were able to evaluate at least five claims in 92 papers. In 34 of those papers, they reproduced more than 40% of the claims they attempted to verify, and in eight, they reproduced more than 80%. SAI published these figures in its verification report.

The report shows which claim was tested, which files were run, and what result was obtained. Machine learning researcher Federico Bianchi described how the work should be divided between AI and humans:

“AI should not do everything. Assessments of novelty and significance should remain with humans.”

A preprint by Bianchi and colleagues shows which parts of the audit can be formalized. GPT-5 searched published AI papers for errors with unambiguous answers, including incorrect formulas, calculations, tables, and contradictions in the text. A second GPT-5 run reviewed the flagged errors and filtered out false positives. Experts examined 316 flagged passages and confirmed that 263 contained errors.