Why is Q2_0 so painfully slow on some x86 chips?

I've been digging through the llama.cpp PRs because I'm tired of seeing "5% improvement" updates that don't actually move the needle. PR #26348 actually looks interesting because it claims a 3.0x to 3.6x speedup for Q2_0 on x86 CPUs. We're talking about an 8B model jumping from 2.39 tok/s to 8.20 tok/s. That's a massive leap, but I'm skeptical about how this translates to actual consumer gear.

The technical shift here is adding an x86 VNNI implementation for the Q2_0 × Q8_0 dot product. Essentially, it swaps a generic implementation for AVX-VNNI / AVX-512 VNNI. The benchmarks were run on an AMD EPYC 9645 (8 cores, CPU only, GGML_NATIVE=ON, BLAS off), and the numbers are wild:

• 1.7B Model: pp512 went from 14.07 to 50.47 tok/s (3.59x)
• 8B Model: tg128 went from 2.39 to 8.20 tok/s (3.43x)
• 27B Model: tg32 went from 0.72 to 2.37 tok/s (3.32x)

Here is where it gets annoying for the rest of us. There's a mention of a "silent failure" on consumer Intel chips. Apparently, on i5-13400 (12th–14th gen), these CPUs have AVX-VNNI but AVX-512 is fused off. The system doesn't tell you it's skipping the fast path; it just runs Q2_0 at a snail's pace. When the VNNI path was actually forced in a reference implementation, an 8B model's decode speed jumped from 2.17 to 6.92 tok/s.

I have a few concerns before I call this a win. First, this is strictly for Q2_0. Don't expect your Q4_K_M or Q8 models to magically triple in speed. Second, the precision isn't 100% identical due to fused multiply-add behavior. The author claims a 99.216% match on top tokens in perplexity tests, which is "fine" for most, but for those of us obsessing over prompt engineering and deterministic outputs, it's a variable to track.

For anyone wanting to do a deep dive or a real-world deployment test, you'll need to compile the PR branch manually since it isn't merged into the main branch yet.

I'm really curious if these numbers hold up on laptops. Server CPUs have massive memory bandwidth, but a Zen 4 or Raptor Lake laptop might hit a thermal or bandwidth wall that shrinks this 3x gain significantly. If you have a modern x86 laptop and can run a llama-bench comparison, I'd love to see if this actually makes Q2_0 usable or if it's just a benchmark win.