Claude AI rechecked the calculations in 659 theoretical astrophysics papers and found 872 errors, with five times as many papers repeating errors as criticizing them, according to physicist Casey Handmer
Claude AI rechecked the calculations in 659 theoretical astrophysics papers and found 872 errors, with five times as many papers repeating errors as criticizing them, according to physicist Casey Handmer
On November 17, Handmer published a website presenting the audit results for papers in the field in which he earned his PhD at the California Institute of Technology (Caltech). He started with papers from Caltech’s theoretical astrophysics group, added all the papers they cited and all those that cited them, and assembled a network of 158 059 papers. Claude Opus rechecked the calculations in 659 of them.
Handmer’s code for calculating gravitational waves was based on equations from a 1997 paper that, he writes, underlies the software used in his subfield. Researchers in the subfield knew about its errors, but the corrections were buried in roughly ten thousand lines of contradictory emails, and the paper itself was never corrected. Handmer describes this as a limit of human review: a reviewer checks a paper once, and no one is paid to correct published papers. A machine, he says, can read the entire field’s literature, recheck the calculations, and trace each error through the papers that copied it.
Each of the group’s 200 most cited papers from 2006–2019 was read in a separate Claude Opus session equipped with computational tools. Each session worked through the derivations from scratch and flagged problems without access to corrections or later papers. A second, independent session checked flagged problems in results used by three or more later papers against the journal version and confirmed 56 errors. The calculations in another 459 papers, mostly selected through an inexpensive preliminary reading, underwent the same checks. In total, the audit found 872 errors in 433 of the 659 papers, including 92 major errors that changed the main result. The website states that all 872 errors were confirmed only by models; a third session used as a control, given only the paper and the location of the error, agreed with 33 of 40 randomly selected errors.
Another model examined how later papers cited the papers whose calculations had been checked: 367 papers repeated an error by quoting or using an incorrect value, while 68 criticized an error. Only 47 of the 872 errors had received criticism in the published literature. One repeated error appeared in a 2006 paper by Caltech researchers that estimated whether planned space-based detectors would detect gravitational waves from the rapid expansion of the early Universe. According to the audit, the formula for the energy density of these waves has 6 in the denominator instead of 24, making the estimate four times too high. Six papers repeated the error. A single objection appeared in 2008, while papers continued to repeat the error through 2014.
For 136 of the 200 papers, the audit compared the journal version with the first preprint (the first version posted to the open repository arXiv). Of the confirmed errors, 13 had been corrected before publication by a reviewer or the authors, while 56 survived peer review. Handmer calculates that peer review removes roughly one error in five. Of those 56 errors, 52 remain in the latest arXiv version, which can be corrected free of charge. Of 87 errors that later papers stopped repeating, 67 had never been criticized. The website’s explanation is that the field moves on to other topics, forgetting correct answers as quickly as incorrect ones.
In a thread on X, Handmer concludes that AI could accelerate the quantitative sciences more than a thousandfold without further improvements to the models. On the methods page, he sets out the calculation: checking a paper costs 1,74–3,10 dollars, while a week of a reviewer’s time, which he assumes costs five thousand dollars, is 1600–2900 times as expensive. He also writes:
“The bottleneck has shifted from money to checking answers and building instruments for questions the data cannot answer.”