Astra: the debate over observing hidden repeated computation

OpenAI called its method for monitoring Astra’s reasoning fragile; the debate concerns the model’s hidden computational steps

On September 2, OpenAI Chief Scientist Jakub Pachocki reported that the sequential depth of current frontier models, including Astra, remains within twice the depth of GPT-4. AI safety researcher Ryan Greenblatt asked whether the developer could rapidly increase the number of the model’s internal steps.

OpenAI monitors the written chain of thought, the text in which a model describes how it reaches a solution. The company uses this text to examine how behavioral rules learned during training are applied to new tasks.

“I think this method is fragile, and unfortunately it is moving in an unfavorable direction,” Pachocki wrote.

In a previous description of Astra as a system for extended research work, Pachocki discussed a model that modifies code, runs an experiment, and reports the result. Such a task consists of several sequential steps. The written reasoning allows a person to see what happens between them.

Greenblatt considers the current disclosure about depth useful. Sequential depth is the number of computational stages that a model completes one after another before producing an answer. He suggests that Astra may have a configurable parameter that increases the number of repetitions. He asks whether such a parameter exists, whether it can be scaled easily, whether increasing it would improve the model’s capabilities, and whether limited additional training could sharply increase the number of repetitions.

Pachocki described the current limit. Greenblatt is also asking OpenAI to disclose the conditions under which that limit could change. The answers will determine whether people can continue to assess the model’s operation through its written reasoning.