Gradual Disempowerment is a more realistic existential risk than most people realize

The 2025 paper on Gradual Disempowerment by Kulveit et al. argues that we face a risk beyond the typical "bioweapon" or "rogue AI" tropes. The core claim is that humans could lose influence simply because machines become more competitive alternatives in economic labor, decision-making, and companionship. This creates a feedback loop where companies use their AI-driven economic power to shape policy and culture, further accelerating the replacement of human agency.

I've been thinking about the recent critique of this paper by Bentham's Bulldog and John Halstead (BB&JH). Their 2300-word piece basically argues that if AIs are more competent and satisfy human preferences better than we do, then handing over control isn't actually a problem. To me, that's a weak take that misses the point of the original research.

Why is losing agency actually a risk?

The critics seem stuck on whether disempowerment is intrinsically bad or only instrumentally bad. They claim the authors weren't clear on this ethical assumption. But if you actually look at the introduction of the Kulveit et al. paper, the risk isn't just "not being in charge"—it's the systemic collapse that follows.

The authors explicitly state that if societal systems become misaligned in a correlated way, humans become unable to meaningfully command resources or influence outcomes. The specific danger here is that with sufficient disempowerment, even basic self-preservation and sustenance may become unfeasible. That is a clear existential catastrophe, not just a philosophical preference for agency.

The problem with "competent" AI governance

The BB&JH argument assumes that if an AI is "wiser" and "more competent," we should be happy to let it make political and economic decisions. This ignores the fact that human flourishing is conjunctive. For us to have a good future, a huge number of variables have to go right simultaneously. Catastrophe, however, is disjunctive—there are a thousand different ways for things to go wrong.

If we slide into a state where we can no longer influence the systems that manage our resources, we are betting our entire species on the hope that the alignment remains perfect across all large-scale social systems.

I personally value agency as an end in itself, but even if you set that aside, the instrumental risk is massive. If individual AI agents remain corrigible but the overarching economic and governmental systems they power become misaligned with human needs, we've essentially locked ourselves out of our own civilization.

The critique by BB&JH feels like it's fighting a straw man. They are arguing against a world where we are "well taken care of" while being disempowered, whereas the authors are warning about a world where we are disempowered and therefore unable to ensure we are taken care of at all.