Google’s internal benchmark for research automation included tasks that no solution could pass

Google’s internal benchmark for research automation included tasks that no solution could pass

On October 16, Simon Mendelsohn examined Google DeepMind’s report on Gemini 4 Argon. Its internal benchmark, GRB, tests whether an AI agent can carry out research engineering work independently. Google’s audit found bugs in some tasks that made it impossible for any solution to pass.

GRB consists of 74 machine learning research engineering tasks in which an agent works with data, runs experiments and interprets the results. Each task has a two-hour time limit, and grading programs check the agent’s actions and results against the success criteria.

Google uses GRB results to assess whether a model has reached the threshold for fully automating the work of any of its AI research teams. A model that solves fewer than 90% of valid tasks on its first attempt is classified as below that threshold. The reliability of the grading programs therefore affects the assessment of the model’s capabilities.

In the audit in Argon’s safety report, Google wrote:

“About a quarter contained bugs that prevented any attempt from passing: for example, grading programs crashed or the reference answer was incorrect.”

Several other tasks assessed details that were absent from the instructions. If a script rejects every answer, a low score reflects both the agent’s performance and a failure of the measurement itself.

On the subset of original tasks retained in the current version of the benchmark, infrastructure timeouts cut short many of Argon’s runs. Google reported results for completed attempts: the model solved roughly half, but those attempts disproportionately involved short tasks and overestimate its success rate across the full subset. Google also calculated two separate best-case scenarios for Argon: counting known defective tasks as solved, and treating every unfinished run as successful. The model remained below the threshold in both cases.

Mendelsohn considers this conclusion sufficient for Argon. In his analysis on LessWrong, he proposes that, for the next model to approach the threshold, Google disclose the size of the evaluated subset, the number of defects and timeouts, the denominator used in the 90% rule and a calculation showing the combined effect of the adjustments. That calculation would help distinguish the effects of benchmark errors from the model’s performance when a few percentage points could change the decision.