DiG-bench tests whether AI can discover hidden rules through experimentation
DiG-bench tests whether AI can discover hidden rules through experimentation
On August 12, the DiG-bench authors released a benchmark comprising 70 text-based games with hidden rules. Gemini 3.1 Pro completed 69 of the 70 games when given descriptions of the rules, but only 18 when it had to infer them independently.
In DiG-bench, each game is a small world governed by its own rules. The player sees a short string of symbols and a set of available actions, but does not know how those actions change the game state or what must be done to win. Each move produces a new state. The player must choose the next action so that its outcome distinguishes between competing hypotheses about the rules.
The public game P-21 demonstrates this cycle through a single mechanism. On the second level, the player learns that pressing the period while holding the n symbol creates a bridge one cell long. On the third level, a three-cell obstacle blocks the path. In creative mode, a separate sandbox for experimentation, moves do not count against the level limit. When the player steps onto the ~ symbol, the action is performed three times. The player then restarts the level, moves ~ to the obstacle, picks up n, builds a three-cell bridge, and continues. The experiment turns an observation into a rule, which then provides the solution to the next problem.
Two days earlier, Eric Schmidt and Suhas Mahesh described a scientific agent that connects data, forms a hypothesis, and selects the next experiment. DiG-bench isolates an early, measurable step in that process: whether an agent can design an experiment that distinguishes between possible rules while those rules remain unknown.
The authors gave Gemini 3.1 Pro a brief description of each game’s dynamics and victory condition, but provided no guidance about strategy, tactics, or the sequence of moves. The model completed 69 of the 70 games, compared with 18 without this information. At least one person completed each of the 70 games on their first encounter with it.
On levels 6 and 7, the authors evaluated software wrappers that maintain an action history for the model and provide access to files and tools. In direct comparisons, Kimi K3 and Gemini 3.1 Pro did not outperform their base versions when used with these wrappers. Prime Agent with Opus 5 likewise did not improve on the result of the base Opus 5 model. In these comparisons, the additional capabilities provided no advantage in discovering the game’s rules.
The DiG-bench website provides access to 21 games and an API that external teams can use to run their models on the public portion of the benchmark. A scientific agent faces the same problem before an answer is available: which observation could change its current explanation?