Cerebras WSE-3 smokes H100 on Llama 3 70B inference at a

Spent last week digging into the new Cerebras WSE-3 numbers they posted alongside the Llama 3 70B benchmark run. The headline: 1,800 tokens/sec on a single CS-3 system versus ~850 tokens/sec on an 8×H100 DGX box. That's not a typo — one wafer-scale engine beating eight Hopper GPUs on throughput while pulling ~23 kW total versus ~56 kW for the DGX.

The architecture difference is the whole story. Nvidia's approach is scale-out: stitch GPUs together with NVLink/NVSwitch, fight tail latency across the fabric, burn cycles on kernel launches and synchronization. Cerebras went scale-up — 900,000 cores on one piece of silicon, 44 GB on-die SRAM (no HBM, no DRAM hops), and a 2D torus mesh that moves data at 100+ TB/s internally. For dense LLM inference where the model fits in SRAM, you eliminate the memory wall entirely.

# Rough power/perf comparison from their published figures
CS-3 (1 system):     1,800 tok/s  @ 23 kW  →  78 tok/s/W
8×H100 DGX:          ~850 tok/s  @ 56 kW  →  15 tok/s/W

Five-to-one perf-per-watt is the kind of gap that makes CFOs pay attention when you're planning a 10 MW cluster.

Caveats matter though. The WSE-3 only shines when your model + KV cache fits in that 44 GB SRAM. Llama 3 70B at FP16 is ~140 GB — so they're running INT4 or INT8 quantization, and even then it's tight. For models above ~30B params at decent precision, you're back to multi-system clustering with their MemoryX boxes, and the programming model gets more complex. Nvidia's advantage stays the software stack: CUDA, TensorRT-LLM, vLLM, Triton — the entire ecosystem assumes Nvidia. Cerebras SDK is Python-based but you're writing to a completely different execution model (dataflow vs. SIMT). Porting effort is real.

Also worth noting: these are inference numbers. Training is a different beast — Nvidia still owns that mindshare and the tooling maturity. Cerebras has done impressive work on sparse training and they've shown GPT-3 scale runs, but the day-to-day ML engineer experience is years behind.

If you're building a dedicated inference fleet for a known model family (Llama, Mistral, proprietary fine-tunes) and power/rack space are constraints, the CS-3 deserves a serious eval. For general-purpose training + inference clusters where flexibility wins? H100/B200 still the default call.

Curious if anyone here has touched the Cerebras SDK for a real workload — the dataflow programming model is a mental shift.