Recursive self-improvement likely causes lossy gains instead of an exponential explosion
The belief in a singularity often ignores how limited automatable research is and how scaling law costs erase potential gains. Although Anthropic and OpenAI utilize thousands of concurrent agents to refine internal workflows, this does not signal an imminent singularity. Adding more AI agents in parallel leads to diminishing returns. AI cannot fix bottlenecks rooted in politics or resource scarcity through code alone.
The San Francisco AI community frequently exaggerates concerns. Claims that open-source AI would fail or that specific risks were imminent during 2023 and 2024 did not materialize as predicted. Since agents achieved better product-market fit at the start of 2026, the focus has moved toward extinction risks.
Richard Ngo notes that the AI safety community expects an intelligence explosion in a few years. He argues that while these members might be "directionally correct," they remain "factually wrong." Superintelligence will likely not appear within the next 8 years, though the speed of progress may make short-timeline predictions seem accurate.
Three specific frictions prevent a sudden intelligence explosion:
- Scaling Law Costs: Net acceleration is hindered because scaling costs grow exponentially.
- Parallel Diminishing Returns: Breakthroughs do not increase linearly as more agents run in parallel.
- Physical and Political Bottlenecks: Software agents cannot bypass the need for political navigation or new hardware to build next-generation models.
RSI serves as a method to maintain progress against rising costs unless a foundational, imagination-based breakthrough alters AI evolution. It is not a force that triggers an unstable or unpredictable explosion.
The current shift involves massive inference capacity rather than "superintelligence." Labs expand compute capacity and task thousands of agents with measurable problems. This results in a short-term acceleration of problem-solving that people mistake for a recursive loop. Scaling agents productively is a technical milestone rather than a sign of extinction.
To address the loop collapse in my pipelines, I’ve found that explicitly limiting each agent’s iteration depth to a fixed number (e.g., 3–5 steps) while enforcing a hard stop after reaching the limit prevents infinite recursion. This mimics how OpenAI and Anthropic scale agents by capping their parallel execution to avoid spiraling costs—even if the gains aren’t exponential. The real win comes from optimizing how these agents interact rather than how many run.