How AI providers' throttling backfires — and a tool to prove it

What if every time a provider downshifts to a smaller model to save power, they're actually increasing load instead of reducing it? That's the paradox I kept hitting while building a scheduler simulator that models exactly this behavior.

I started this after noticing that during peak hours, responses got noticeably dumber — not just slower. My hunch was that providers swap in quantized or smaller-tier models when demand spikes, which makes sense on paper: less compute, less electricity. But here's the catch — bad answers make users retry, and retries multiply the load.

Using queueing theory and finite-horizon dynamic programming, I modeled an AI fleet serving heterogeneous users. The result? Standard threshold-based throttling (swap models once queue depth crosses X) is exactly what causes retry storms. Agents and power users are far more sensitive to quality degradation than casual askers — so lumping them together under a single policy is inefficient and counterproductive.

The tool visualizes this with a Flask + vanilla JS frontend (~100 lines, LLM-assisted). It shows how splitting traffic into quality-sensitive vs. quality-resilient cohorts leads to better server utilization and fewer retries overall.

The paper at arxiv.org/abs/2608.23986 has the proofs and some calibrated toy instances. Real-world calibration would require internal telemetry most providers don't expose — but the framework is extensible.

This isn't just academic. If you've seen latency spikes or seen models act weird during high traffic, this might explain why throttling feels like it makes things worse.

Code and demo link in the repo — feedback welcome if you're working on similar infra problems.