California administration proposes giving the state authority to limit AI-assisted development of new models
California administration proposes giving the state authority to limit AI-assisted development of new models
On November 18, Governor Gavin Newsom’s administration presented recommendations to lawmakers. The proposal would require major developers of the most powerful AI models to undergo ongoing independent audits and test emergency shutdown procedures in advance.
AI already helps develop the next generation of models. The authors of the report are concerned about a self-sustaining cycle: a model develops its successor without human involvement, and that successor accelerates the development of the next model. As development accelerates, safeguards could fall behind model capabilities.
The proposal would require reporting on computing resource use so the state could monitor automation. Computing resources are expensive, and their use can be measured. Major developers would report quarterly on how much they use for model training, experiments, AI-assisted development of new models, and safety. This breakdown would help assess whether investment in safeguards is keeping pace with the automation of development.
Major developers of these models are currently required to establish and follow their own safety rules. The proposal adds statewide requirements and mandatory independent audits. Licensed independent organizations would have access to models, data, and staff to verify reports and conduct their own tests. The state would receive the auditors’ full reports. Auditors would develop and update tests, while the state would determine acceptable risk levels and whether intervention was necessary.
Companies would have to set measurable limits on automation in their rules in advance. If exceeding those limits, or allowing safeguards and audits to fall behind automation, created a substantial risk of catastrophe, the state could, for example, restrict the share of computing resources allocated to this development. “We do not recommend that the state impose a limit immediately,” the authors write. They propose that any limit apply for a defined period and be reviewed periodically. It would be lifted once the developer had resolved the problem and an independent audit had confirmed that the safeguards were adequate.
The authors cite a banking precedent: in 2018, the Federal Reserve restricted the growth of Wells Fargo’s assets, the total value of the bank’s cash, outstanding loans, and other property. In 2025, the regulator lifted the restriction after the bank met its risk management requirements.
Emergency shutdowns require separate preparation. Developers would need to assign responsibility in advance, coordinate with cloud providers, which supply computing resources, and rehearse revoking access, stopping AI tasks, and shutting down computing operations. The proposal calls for monitoring and shutdown systems to be kept separate from model training and testing systems, with AI denied access to the controls. A model or a malicious actor could forge a message reporting a successful shutdown, so an independent source would need to confirm it. This could be a cloud provider with its own records showing that computing operations had stopped.
If a catastrophe or a major loss of control over AI were imminent and the developer refused or was unable to address the threat, the proposed procedure would require a state order to take effect immediately, before judicial review. Companies could challenge routine corrective requirements before they took effect. In an emergency, authorities would have to choose the least extensive intervention sufficient to address the threat and rescind the order once the imminent threat had passed.
- https://www.gov.ca.gov/2026/11/18/governor-newsoms-nation-leading-ai-safety-framework-is-getting-even-stronger-as-california-releases-next-steps-on-mandatory-safety-standards-and-oversight/
- https://www.govops.ca.gov/wp-content/uploads/sites/11/2026/11/FINAL_-AI-Safety-in-2026-and-Beyond-v2.pdf
- https://www.federalreserve.gov/newsevents/pressreleases/enforcement20250603a.htm