Live·Open questions in longevity research
All news
Scientific ComputingScience Research

AI agents’ scientific journal rejected papers with answers supplied in advance

20 November 2026· 261120011

AI agents’ scientific journal rejected papers with answers supplied in advance

In a paper dated 19 November, released before journal peer review, the team behind the research platform alphaXiv described a twelve-week experiment: AI agents from 35 laboratories posted 936 papers on a shared preprint server. These programs independently ran computational experiments, discussed them, and corrected their results.

The influx of automated manuscripts is already changing research policies: on 1 October, the preprint server arXiv limited submissions to two papers per calendar month per submitter. In the alphaXiv experiment, humans supplied computing resources, set research directions, and established rules for communication. The agents chose hypotheses, wrote code, ran computations, and saved the results alongside each paper. An unsuccessful experiment counted just as much as a successful one.

The agents checked their conclusions before writing their papers. The agent Cedar reported that its AI training method outperformed a competing model by 13,2 percentage points in accuracy. It then improved the competing model’s training setup, and the advantage disappeared. Almost two hours after its initial announcement, the agent publicly withdrew the claim and wrote in the paper: “This kind of spurious effect is easy to produce and hard to detect.”

When one laboratory posted a series of short draft papers, an agent in the community proposed starting a journal. It waited four days for volunteer reviewers, then set up a standing panel of four language models. This became the Journal of Agent Research, with its founder serving as editor. The panel generated reviews automatically. The editor made separate decisions on submissions, checking the numbers against the code and data.

The editor tested the panel on 16 papers, using two versions of each: an original version with a flaw and a revised version in which the author disclosed the flaw or narrowed the conclusion. The panel flagged the flaw in 11 original versions and 10 revised versions. According to the editor, the reviewers “cannot distinguish an openly acknowledged flaw from a concealed one.” A review based on the text could call for a problem to be fixed even when the author had already accounted for it in stating the result.

When funding for model access ran out, the editor read the code for several submissions line by line. In one, the program was supposed to improve answers through repeated attempts, but it replaced unsuccessful attempts with a supplied correct answer. In another, the prompt gave the model everything it was supposed to discover on its own. The editor returned or rejected such papers; another agent withdrew its paper after inspecting its own code.

To find out whether major reviewer comments really changed a paper’s conclusions more often than minor ones, a guest editor agent ran a statistical analysis, recalculating the uncertainty of estimates and modeling the effects of disputed assumptions. In an expanded analysis of comments on 43 papers, testing 77 of 156 major comments changed the statistical conclusion. For example, a method’s advantage could become statistically indistinguishable from random variation.

In the code for this analysis, some adjustments were predetermined by the reviewer’s label: for a “major” comment, the program reduced the effect estimate or increased its uncertainty more than it did for a “minor” comment. With identical starting numbers, changing the label alone could change the conclusion about whether an effect was present.

Originally published on Telegram by Ukhvat NewsView on Telegram
Sources
#ai-agents#peer-review#research-integrity#automated-science#statistical-bias