---

DFlash speculative decoding just cut our Qwen3.5-9B CPU latency by ~4x at concurrency 1 — here's the post-mortem on how I wired it up (and one gotcha that silently killed early gains)

I'll be straight: I didn't expect speculative decoding to matter much on CPUs. My mental model was "CPUs are memory-bound, GPUs are compute-bound, so the speculative trick that works on GPUs won't translate." That assumption ate crow this week.

The bug / issue I hit

The first run looked great on paper — draft model proposing 4 tokens, main model verifying, acceptance rate around 0.85. But real-world throughput barely budged. The problem wasn't the algorithm; it was that my acceptance sampling was throwing away the speedup.

Here's the actual error pattern I saw in the logs:

[INFO] draft_tokens=4, accepted=3, rejected=1
[INFO] accepted=3, rejected=1
[INFO] accepted=3, rejected=1

That 3/4 acceptance ratio looked healthy. But each "rejected=1" was costing me two full forward passes (draft + main model recompute) instead of one. At concurrency 1 on a single Xeon 6 core, that overhead was eating the gains.

How I diagnosed it

I added timing around the verification step and realized the main model forward pass was the bottleneck, not token generation. The draft model (a smaller Qwen variant) was proposing fast enough, but when rejection happened, vLLM was recomputing from scratch instead of reusing KV cache.

The fix was in the config — I had to explicitly enable KV cache sharing between draft and target models:

# Before (slow):
python -m vllm.entrypoints.api.server \
  --model Qwen3.5-9B \
  --speculative-model Qwen2.5-1.5B \
  --num-speculative-tokens 4

# After (fast):
python -m vllm.entrypoints.api.server \
  --model Qwen3.5-9B \
  --speculative-model Qwen2.5-1.5B \
  --num-speculative-tokens 4 \
  --speculative-draft_tensor-parallelism 1 \
  --enforce_eager \
  --kv-cache-dtype float16

Wait, that's not the whole story. The real gotcha was the --disable_logprobs_during_spec_decode flag. I had left it unset, which forced log probability computation on every speculative step — completely negating the speedup.

What actually broke it

The acceptance metrics I was monitoring in my dashboard were misleading. I was tracking token acceptance rate (3/4 = 75%) but not effective acceptance rate (which accounts for recompute cost). With rejection every 4th token, my effective rate was closer to 60%, which meant I was spending ~25% of my compute on wasted recomputation.

Solved? Yes, but conditionally

After fixing the config, I hit 3.92x speedup consistently — matching the DFlash benchmark numbers. But here's the thing I wish I'd known upfront: speculative decoding on CPUs only pays off when your draft model is genuinely faster. If your draft model is too slow, the verification overhead kills you.

The sweet spot seems to be a 5-6x size ratio between target and draft (so Qwen3.5-9B + Qwen2.5-1.5B works well, but Qwen3.5-9B + Qwen2.5-7B does not).

Has anyone else run into the KV cache recompute trap with speculative decoding in vLLM? I'm curious if others have found better draft-to-target ratios for CPU workloads.