A GPT-6.1 Sol agent flagged 39 of 656 mathematics papers on arXiv as containing a serious error in the main theorem

A GPT-6.1 Sol agent flagged 39 of 656 mathematics papers on arXiv as containing a serious error in the main theorem

On October 15, the MathArena team published an audit of 793 mathematics papers on arXiv. An agent powered by GPT-6.1 Sol selected up to five central theorems from each paper and searched for counterexamples: cases in which a theorem’s assumptions hold but its conclusion is false.

For the MathArena report, the team chose a task that could be checked against the theorem statement. For each theorem, the agent extracted the assumptions and definitions and searched for a counterexample. A separate run checked the proposed counterexample. The agent then checked its finding against a later version of the paper.

Of 656 papers first posted from January 2025 through July 2026, 39 were assigned the MAJOR category. This category means that retaining the theorem requires substantially restricting its assumptions or changing its conclusion. These findings account for 5,95% of papers and 58 of the 2 543 selected theorems. Among papers with a newer version, the later text resolved the issue or reduced its severity in half of the cases.

In the August to September cohort, two of 137 papers, or 1,46%, received the MAJOR category. These rates describe different samples: the authors attribute the difference both to changes in the composition of the samples and to the possibility that Sol is less effective at finding errors in texts written with its assistance.

The public data release contains the agent instructions and reports for each case; its README describes the categories as model judgments. The team manually checked 20 findings, used Opus 5.5 to assess 75 serious cases, and ran Sol again. Opus retained the MAJOR category in 78,7% of cases, and the repeat Sol run retained it in 90%. On this basis, the authors estimate that approximately 10% of findings classified as serious are false positives; each entry in the archive allows readers to compare the report with the original theorem.