SciLaws-Bench distinguishes formula fitting from scientific law discovery

SciLaws-Bench tests whether AI can distinguish a scientific law from a successful fit

On September 1, the authors published the SciLaws-Bench preprint, which presents 118 tasks drawn from 381 scientific papers. Each task evaluates a model in two settings: one with real data and another in a synthetic “parallel” world.

A single table of measurements can fit several formulas. One formula may predict held-out data more accurately, while another may also satisfy the requirements of the problem itself by preserving the required sign of a quantity, known limits, and mandatory dependencies. SciLaws-Bench evaluates these two abilities separately.

In SciLaws-Real, the system receives real observations. Its formula is evaluated on held-out data and against rules taken from the original paper, including whether it preserves the required sign of a quantity, known limits, and mandatory dependencies. A language model judge evaluates these criteria, and the authors compared its decisions with assessments from five domain experts.

In SciLaws-Parallel, the authors construct a synthetic world governed by a new version of a published formula. The system begins without observations, selects its own measurement points, receives noisy responses, and reconstructs the structure of the hidden law. This setting therefore also tests whether the system can choose measurements that distinguish between competing formulas.

In the authors’ evaluation of nine models, a more accurate fit corresponded to greater scientific validity in only 54,9% of 3 616 pairwise comparisons. In a nuclear physics task, the most accurate formula introduced a resonance peak that is absent from the original phenomenon.

The authors write, “Models are better at generating laws than at selecting them reliably.”