Speculative Decoding: Hardware Tax vs. Model Math

Speculative decoding is sold as a massive win for LLM inference speed, but in practice, it often fails to hit the theoretical speedups promised in research papers. I've been digging into why a working implementation—complete with KV caching on both the draft and verifier sides—can actually result in slower performance than just running the verifier alone.

The Performance Gap

Running a gamma sweep on open-ended prompts produced some frustrating results. With a verifier-only setup, I was seeing 14.4 tok/s. Switching to speculative decoding with a gamma of 4 (acceptance rate ~39.2%) actually dropped the speed to 12.3 tok/s. At gamma 7, it tanked further to 9.5 tok/s.

The logic was correct and the acceptance rates were stable, yet the "speedup" was 0.86x. It was objectively slower.

Testing the Hardware Hypothesis

My first instinct was to blame the Apple Silicon MPS backend. MPS has a notorious fixed dispatch cost per model call. Because the draft model is significantly smaller, it should be lightning fast, but the overhead doesn't scale down. In microbenchmarks, a 6x smaller draft model was only 2.8x faster per token.

To isolate this, I ran a FLOPs-based theoretical speedup calculation using the Leviathan et al. formula: E[speedup] = (1 - α^(γ+1)) / ((1 - α)(γ·c + 1)). By using the cost ratio c based on architecture (layers, hidden size, etc.) rather than wall-clock time, I could see what the speedup should be without hardware interference.

Verifier (Qwen2.5-3B-Instruct): 6,171,394,048 FLOPs/token
Draft (Qwen2.5-0.5B-Instruct): 987,922,432 FLOPs/token
c = 0.1601

The results were eye-opening: at a 39.2% acceptance rate, the theoretical FLOPs-only speedup was only 0.99x. The hardware overhead was making it worse, but the algorithm itself was already failing to break even.

The Draft Model Dilemma

I wondered if a larger draft model (Qwen2.5-1.5B instead of 0.5B) would fix this by increasing the acceptance rate. On paper, this looked like a disaster—the cost ratio c jumped to 0.5002. I'd need an acceptance rate of 69-84% just to hit a 1.0x speedup.

However, real-world testing showed a wild variance. In one run with a 86.4% acceptance rate, I finally saw a real win: 1.37x speedup. But in three other runs, the acceptance rate plummeted back to the 20-40% range, dragging performance back down.

This confirms that speculative decoding isn't a "plug-and-play" optimization. It's a precarious balance between the cost ratio of the two models and the highly volatile acceptance rate of the draft. If the draft model isn't predicting the verifier with extreme accuracy, you're just paying a "speculation tax" for no reason.