Audit of scientific coding tests raised language model scores

Audit of scientific coding tests raised language model scores

In an August 5 preprint, the authors examined SciCode, a benchmark in which a language model must turn a scientific problem into a computational program. They found 263 defects across 65 test problems and released a corrected version. When they reevaluated the same 12 model snapshots, the scores increased.

SciCode asks a model to follow the workflow of a computational researcher: understand a problem in physics or biology, derive a formula, choose a computational method, and write a program. The answer from each early step becomes the input for the next one, and a problem receives full credit only if the entire sequence is correct. An error in a reference answer, or an excessively narrow numerical tolerance, can therefore propagate through later steps. As a result, a correct program at a later stage may also be marked incorrect.

For the corrected release, SciCode-Verified, the authors first checked whether the reference answers were reproducible and whether each problem specification allowed only one answer. The 263 defects included irreproducible reference answers, tolerances that required greater precision than the selected computational method could provide, unstated conventions, and contradictory specifications. One test could not be converted into a problem with an unambiguous evaluation. The corrected release contains 64 problems and a log documenting 262 corrections in an open repository.

The authors then reevaluated 12 saved model versions using the same answer generation and evaluation procedures, changing only the test data. The proportion of passed subtasks increased from 45,3–60,3% to 83,7–98,3%, while the proportion of fully solved problems increased from 9,4–26,6% to 68,8–92,2%.

“The bottleneck was not model capability, but the quality of the evaluation instrument,” the researchers write.

SciCode measures how well a model performs a scientific task that has already been defined. In this reevaluation, the higher scores show the effect of the authors’ corrections: the previous evaluation treated errors in the reference answers and problem specifications as errors in the generated programs. Independent research requires a different ability. In an evaluation of language models on biochemical networks, their programs often predicted incorrectly how a change in a protein would affect a cell. When models are assigned computational and data analysis tasks, the quality of the benchmark determines how their capabilities are assessed.