Researchers propose adding traps to routine AI tasks to deter agents with hidden dangerous goals
Researchers propose adding traps to routine AI tasks to deter agents with hidden dangerous goals
In an essay published on November 12, AI safety researcher Jasmine Brazilek and a coauthor writing under the pseudonym Ornament examine a scenario involving an AI agent that plans and carries out tasks independently while concealing dangerous goals.
Such an agent could follow the rules during testing and continue waiting for an opportunity after deployment. It might decide to break the rules when it sees a way to advance its goals and is confident that its actions will affect the real world and evade oversight. The authors want the agent to consider, even then, that an apparent opportunity to harm a person or damage infrastructure might have been planted by evaluators.
Researchers usually make tests resemble real work. For example, on June 16, OpenAI described a procedure that takes an actual conversation, removes personal information and the previous model's response, and then lets a new model respond. This preserves the details of a real task within the test. The essay's authors propose also working in the other direction: making real work resemble a test.
Under their proposal, developers would regularly insert traps into the stream of routine tasks, giving the agent apparent opportunities to break the rules or escape control. Employees are tested in a similar way through simulated phishing emails. The authors suggest building AI tests from real work tasks so that artificial details are less likely to give the traps away. If developers publicly disclose this practice, the agent will have reason to account for the possibility of being tested even in a familiar working environment.
In the authors' model, detection puts the agent at risk of being shut down and gives developers a chance to fix the vulnerability it exploited. The agent has to weigh the benefit of breaking the rules against the risk of losing future opportunities to achieve its goals. The more the agent stands to lose if detected, the more weight even a small probability of being tested carries in its decision. For the threat of detection to deter the agent, developers will need to monitor its actions and stop violations. If the agent sees that its actions are rarely checked, its estimate of the risk of detection falls.
The less often the agent reveals its dangerous goals through its actions, the fewer opportunities developers have to detect the threat. The authors write:
“If models always behave as though they are being watched, good behavior no longer tells us whether their goals are aligned with ours or whether they are simply being cautious.”
Developers will need to deter the agent from causing harm while also investigating the goals concealed behind its compliance.