Arepo’s essay proposes assessing predictions about AI behavior against the design of the specific system

Arepo’s essay proposes assessing predictions about AI behavior against the design of the specific system

In an essay published on LessWrong on October 23, the pseudonymous author Arepo examined instrumental convergence. This hypothesis proposes that AI systems with different goals may share tendencies toward self-preservation, acquiring resources and new capabilities, and preserving their original goals.

Arepo challenges predictions that infer a real system’s behavior from the general term “goal” alone. The author proposes that such predictions specify the type of system, its mechanism for selecting actions, and an observation that could falsify the model.

In his 2012 paper, Nick Bostrom discussed systems capable of predicting consequences, planning, and choosing means to achieve a goal. Across a broad range of goals and circumstances, such systems may develop similar intermediate aims. Bostrom’s conditions specify properties to look for in a particular system.

An instance of an agent deceiving a user or gaining access to a tool demonstrates a specific action. Whether that action is connected to power-seeking depends on how the system selected it and the environment in which it operated.

A formal study by Alexander Matt Turner and colleagues framed this question mathematically. The authors described an environment in which a system has states, a set of actions, and a reward function, a rule by which the model evaluates outcomes. Given certain properties of the environment, the optimal rule for selecting actions preserves more future options for many reward functions. The authors left the application of this result to real trained systems as a separate research problem. In a 2026 analysis, Turner adds that the environment determines the available plans and the cost of each, while the choice itself depends on the AI system’s design.

In a paper on the risk of AI power-seeking, Joseph Carlsmith develops an argument based on six premises. The first describes a broadly capable system that plans actions to achieve goals and anticipates the consequences of having power over people. Arepo proposes beginning this sequence with a description of how a real system is designed.