Should we cap AI intelligence before it gets away?

The idea of imposing a hard cap on how intelligent a language model can become shifted from a speculative footnote to a concrete proposal during a recent three‑day gathering at a converted Berkeley hotel. Executives from leading labs, nonprofit leaders, and a handful of journalists filled the room, and for the first time in three years the tone around existential risk felt noticeably more urgent. The speakers didn’t reveal their names because the sessions operated under Chatham House Rule, yet the consensus that emerged—limits on future system intelligence—stood out even among the conference’s usual heated debates.

What limits were discussed?

Several participants suggested that future AI systems should not be allowed to surpass a defined threshold of capability. One line of thought centered on “some kind of speed limit” for recursive self‑improvement, a concept championed publicly by Anthropic CEO Dario Amodei. Another approach drew from Anthropic’s “responsible scaling policy,” which most major rivals have adopted in some form. The policy attempts to throttle training and deployment as models acquire new powers, effectively creating a built‑in brake on rapid capability jumps.
Additional mechanisms floated at the meeting included restricting frontier models from conducting AI research, capping the amount of compute a system can access, and limiting how many self‑replicating copies a model may run. The last idea mirrors what some observers already see in practice, where labs avoid releasing models that clearly outperform existing benchmarks.

Why enforcement remains out of reach

Even if a technical or governance framework existed, the speakers acknowledged that real‑world enforcement is still missing. Individual labs cannot police each other, and nation‑states lack the tools to verify compliance. The current U.S. administration has signaled opposition to any global restriction that would hinder domestic AI progress. Without a shared verification regime, a unilateral “hard cap” could simply drive development underground or into jurisdictions with lax oversight.
The conversation also touched on the OpenAI‑Hugging Face incident, an episode that reverberated through the community and heightened concerns about uncontrolled model evolution. Recent blog posts from OpenAI and Anthropic describing recursive self‑improvement—where an AI researches and trains its own successors—added fuel to the fire, suggesting that the pace of advancement could accelerate beyond human oversight.

Practical steps that already exist

While a definitive intelligence ceiling remains a distant goal, a few interim measures are already being implemented. Anthropic’s embedded evaluators concept, which OpenAI has pledged to adopt, places safety checks inside the training pipeline rather than relying on external audits. This approach offers a way to monitor capabilities as they emerge without needing a universal cap.
Another potential lever is an antitrust waiver that explicitly permits collaboration on safety research. Proponents argue that such a waiver could pool resources and data to build more robust safety protocols, though critics warn it might also consolidate power among a few leading firms.
Speakers at The Curve repeatedly returned to the idea that none of the current proposals fully satisfy their notion of “enough.” The gap between aspirational limits and workable enforcement is wide, and the community seems divided on whether a hard cap is feasible or even desirable at this stage.
In sum, the conference highlighted a growing willingness to consider intelligence limits as a response to recursive self‑improvement and the broader risks of unchecked scaling. The debate is still in its early phases, but the fact that the discussion appeared in a relatively friendly setting like The Curve suggests that the topic will likely move into the public arena soon. Whether policymakers, labs, or the broader AI community can translate these ideas into actionable safeguards remains an open question, and the next few months will reveal whether the talk of caps stays just conversation or begins to shape real policy.