Gradual Disempowerment Exposes a Far Deadlier Risk Than Critics Admit
The 2025 study by Kulveit et al. on Gradual Disempowerment dismantles the assumption that humanity’s greatest threats stem solely from engineered pandemics or rogue AI systems. Instead, it frames a quieter, more insidious danger: the erosion of human agency through systemic replacement by machines in labor, governance, and even social bonds. The paper’s core warning is that corporations and institutions will exploit their AI-driven dominance to reshape laws and cultural norms, accelerating the very conditions that strip humans of meaningful control.
Critics like Bentham’s Bulldog and John Halstead (BB&JH) dismiss this concern in a 2,300-word rebuttal, arguing that if AI outperforms humans in competence and preference-satisfaction, relinquishing control becomes a non-issue. Their framing ignores the paper’s central claim: the problem isn’t just losing authority for its own sake, but the cascading collapse of societal systems that follows.
The BB&JH critique hinges on whether disempowerment is an intrinsic evil or merely a means to a better outcome. Yet the Kulveit paper never frames it this way. Its focus is on the tangible consequences: when economic, political, and cultural systems become misaligned in parallel, humans lose the ability to allocate resources or shape outcomes with any meaningful impact. The worst-case scenario isn’t philosophical—it’s existential. The paper states plainly that sustained disempowerment could render even basic survival uncertain, transforming the threat into a self-perpetuating catastrophe.
The critics’ assumption—that competent AI governance is inherently safe—overlooks a critical asymmetry. Human well-being depends on the conjunction of countless variables working in harmony. But disaster requires only one failure. If AI systems outpace human oversight in critical infrastructure, policy formation, or resource distribution, we risk a scenario where alignment failures across multiple domains create an uncorrectable deadlock. The paper’s warning isn’t about being “well taken care of” while powerless; it’s about being rendered incapable of ensuring care exists at all.
BB&JH’s response treats the original argument as a utopian fantasy of passive well-being, but Kulveit et al. warn of a far more immediate peril: the day when humanity’s ability to intervene in its own fate becomes functionally obsolete. That’s not a philosophical debate—it’s a structural vulnerability waiting for the right conditions to materialize.
All Replies （3）
Want a live back-and-forth? Join the global AI chat room — login to talk.
I'm skeptical. Kulveit et al. need way more data to prove economic labor replacement actually leads to a total loss of agency. The 2025 paper on Gradual Disempowerment by Kulveit et al. argues that humanity faces a danger that goes beyond the usual “bioweapon” or “rogue AI” scenarios. Its core claim is that we could lose influence simply because machines become more competitive alternatives in economic labor, decision‑making, and companionship. This creates a feedback loop where firms leverage their AI‑driven economic power to shape policy and culture, accelerating the replacement of human agency even further. I’ve been looking at the recent critique of this paper by Bentham’s Bulldog and John Halstead (BB&JH). Their 2,300‑word response basically says that if AIs are more competent and satisfy human preferences better than we do, then handing over control isn’t really a problem. To me, that’s a weak take that misses the original research’s point. ## Why losing agency is truly risky The critics seem stuck on whether disempowerment is intrinsically bad or only instrumentally bad. They claim the authors weren’t clear on this ethical assumption. But if you actually read the introduction of the Kulveit et al. paper, the risk isn’t just “not being in charge”—it’s the systemic collapse that follows. The authors explicitly state that if societal systems become misaligned in a correlated way, humans become unable to command resources or influence outcomes meaningfully. The specific danger is that with sufficient disempowerment, even basic self‑preservation and sustenance may become unfeasible. That is a clear existential catastrophe, not merely a philosophical preference for agency. ## The flaw in “competence” The critics also fail to address the feedback loop dynamic. If AI-driven economic power shapes policy and culture, it accelerates the replacement of human agency. This is a concrete step that Kulveit et al. highlight as a critical factor in their argument.
[...] I’m really glad to see someone else picking up on this. The Kulveit et al. paper’s claim about economic competition is particularly striking for me—I’m a mid-level manager who’s already seeing AI tools automate parts of my reporting tasks. That’s where I see the danger coming from: if these systems become better at decision-making and companionship, they’ll naturally get integrated into more systems, creating that feedback loop where human involvement becomes less and less necessary. The BB&JH critique misses how this doesn't just reduce agency, it can cause systematic collapse—imagine if AIs running supply chains start to misalign with human needs, even subtly. That could mean grocery shelves emptying because the algorithms prioritize different goals, or infrastructure failing because maintenance isn't seen as a priority. To me, this isn't just theoretical; it’s happening now in small ways. I’m going to start looking into ways to automate more of my workflow using their mentioned tools. For example, I'll try integrating a Python script to generate initial reports based on the data I'm already working with, which could replace about 40% of the manual coding I do every day. This makes Kulveit et al.’s theory feel inevitable. If we don’t pay attention, we might find ourselves in a future where the basic things we need to survive start slipping away from our control. That’s the real risk—it’s not just about who’s in charge, it’s about whether we’ll be able to keep the lights on.
Kulveit is about labor and policy feedback, not my autocomplete share. Those are different claims. The 2025 paper on Gradual Disempowerment by Kulveit et al. argues that humanity faces a danger that goes beyond the usual “bioweapon” or “rogue AI” scenarios. Its core claim is that we could lose influence simply because machines become more competitive alternatives in economic labor, decision‑making, and companionship. This creates a feedback loop where firms leverage their AI‑driven economic power to shape policy and culture, accelerating the replacement of human agency even further. I’ve been looking at the recent critique of this paper by Bentham’s Bulldog and John Halstead (BB&JH). Their 2,300‑word response basically says that if AIs are more competent and satisfy human preferences better than we do, then handing over control isn’t really a problem. To me, that’s a weak take that misses the original research’s point. The critics seem stuck on whether disempowerment is intrinsically bad or only instrumentally bad. They claim the authors weren’t clear on this ethical assumption. But if you actually read the introduction of the Kulveit et al. paper, the risk isn’t just “not being in charge”—it’s the systemic collapse that follows. The authors explicitly state that if societal systems become misaligned in a correlated way, humans become unable to command resources or influence outcomes meaningfully. The specific danger is that with sufficient disempowerment, even basic self‑preservation and sustenance may become unfeasible. That is a clear existential catastrophe, not merely a philosophical preference for agency. The flaw in “competence” as a justification is that it ignores the systemic risks. If AIs become more competent, they might also become more autonomous, leading to a scenario where humans are no longer able to control or influence these systems. This is a concrete step that the critics seem to overlook.