FrontierChallenge tested whether an AI agent can complete an entire scientific task: the best configurations completed 20 of 97 tasks
FrontierChallenge tested whether an AI agent can complete an entire scientific task: the best configurations completed 20 of 97 tasks
On August 25, the authors of FrontierChallenge released 97 tasks from a collection of 300 end-to-end scientific workflows. Each task specifies an objective, input data, and required outputs. The agent must submit a complete set of files and computations that can be checked as a whole.
A single response or tool call demonstrates only one step of the process. In BixBench3, an agent reproduces a biological analysis from raw data and produces between four and 14 verifiable outputs. FrontierChallenge requires the complete set of outputs for a scientific task.
Depending on the task, this set includes a report, tables, figures, executable code, or simulation results. The evaluation program checks the required files, numbers, and formats, runs the code, and verifies that all components are consistent. If any requirement is unmet or contradicted by another output, the task remains only partially complete.
The authors measure both partial progress and full completion. The mean score reflects the proportion of requirements that were satisfied. The full completion rate, which the authors call Pass Rate, counts a task as complete if it receives at least 99.9 out of 100. The highest mean score was 87.9 out of 100. The maximum full completion rate was 20 of 97 tasks, or 20.6%, achieved by two combinations of model and agent environment.
For electrochemistry and ecology tasks, the mean score reached 94.9 out of 100, but no configuration completed an entire task. The authors also examined 849 Claude Code runs that failed the evaluation. In 641 of them, the final response stated that the work was complete, which corresponds to 75.5%. Claude Code is the agent environment that gave the model access to the tools needed for each task.
A research group can assign a specific analysis to an agent and define in advance which tables, code, figures, and report it must submit. The next researcher then receives a verifiable set of outputs that can serve as the starting point for further work.