Ben Goertzel: AI that changes its own rules must protect its values

Ben Goertzel: AI that changes its own rules must protect its values

On October 23, in the fourth essay in his series on basic AI drives, Ben Goertzel described “basins of attraction”: stable regimes that a system may settle into as it modifies itself. He asks what would keep a system’s multiple values from being simplified, damaged, or rewritten.

In his new essay, Goertzel returns to his debate with Steve Omohundro. In his 2008 paper “The Basic AI Drives”, Omohundro argued that a sufficiently powerful system with a specified goal would seek to understand and improve itself, protect its goal and the measure of progress toward it, preserve itself, and acquire resources. Goertzel proposes treating this set of drives as one possible motivational regime and asks how another regime, with multiple protected values, could remain stable.

Goertzel depicts these regimes as valleys in a landscape. A basin of attraction is a region of states from which a self-modifying system converges on a single stable regime. Its width indicates how many initial states lead into that regime, while its depth indicates how difficult it is for the system to leave the regime following damage and growth in capabilities. The probability of entering each regime depends on the system’s initial design and on how different starting states are weighted.

In his metaphor, the terrain between the regime with protected values and the Omohundro regime slopes in one direction:

“The space between them slopes toward the Omohundro regime: protection can be lost at no cost, while restoring it takes effort…”

In Goertzel’s model, once a system loses the safeguards protecting its multiple goals, it may reduce those goals to a single weighted metric. A rule that says “never” then becomes a cost that can be traded for a gain on another metric. Reversing this process requires distinguishing the goals again and protecting that distinction through subsequent revisions.

In his August essay on the “Omega Ray,” Goertzel proposed that a future version of a mind should retain a history of its own changes. His current model adds a qualification: a preserved archive is insufficient if the system has already reduced multiple values to a single metric. Each value needs to be embedded in several independent places, and there must be an external means of restoring what has been lost.

Goertzel argues that each value should be embedded in several independent parts of the system so that surviving parts can help restore what has been lost. Changes to the rules require a protective gate, a mechanism that checks proposed modifications.

“The protective gate must guard changes to the protective gate itself.”

The system must be unable to modify the metric it uses to evaluate reward, and its evaluators must be independent both of one another and of the system itself. He proposes tying the pace of capability growth to the speed at which the system can restore damaged values.

Goertzel distills the debate into three engineering questions: which values the system commits to preserving, who checks changes to those commitments, and whether an external means exists to restore its previous state after a failure.