Cerebras CS-4 Rack Packs 900,000 Cores While Fitting Standard Datacenter Rows

The CS-4 rack is more than another AI box: it is a 15U liquid‑cooled brick that packs 900,000 cores and 44 GB of on‑chip SRAM into a form factor still compatible with standard datacenter rows. Cerebras did not shrink the WSE‑3 die; instead, the company reinforced the wafer‑scale approach and overcame the thermal barrier that limits density at this scale. Each rack contains two CS‑4 systems, delivers 2.4 petaflops of sparse FP16, and includes a closed‑loop coolant distribution unit that handles 50 kW without pushing the room’s chilled‑water budget past its limits.

Sustained utilization beats peak numbers

The real story is not the peak number, but the sustained utilization. Traditional GPU clusters reach 30‑40 % MFU on real LLM workloads because NVLink and InfiniBand become bottlenecks after crossing 256 GPUs. The WSE‑3’s on‑wafer fabric removes that hop altogether. Memory bandwidth remains at 21 PB/s per system for both 7B and 70B models, while the compiler maps tensor parallelism across the wafer without MPI rank shuffling. Internal benchmarks report that a 16‑system CS‑4 cluster maintained 85 % MFU for weeks during a 400B‑parameter training run. That is the difference between “theoretical peak” and “what you actually bill for.”

Cooling systems drive performance

The rack‑level CDU is the overlooked hero. Each CS‑4 draws ~23 kW at the cold plate. The CDU supplies 1.5 L/min per system at 35 °C inlet, and the manifold design allows one node to be serviced without draining the loop. That operational detail matters in a 24/7 environment where one node failure costs $180 k/day in lost training time. Cerebras also introduced hot‑swap power shelves: 6 × 3.2 kW titanium units per system, preventing a PSU failure from taking the entire wafer offline.

Software stack catches up

The software stack has caught up as well. The 2.4 SDK release brought native PyTorch 2.3 support, FlashAttention‑3 kernels tuned for the 48 KB per‑core SRAM, and a new csrun launcher that manages gang scheduling across racks without Slurm wrappers. Developers still write standard PyTorch, while the graph compiler lowers the code to the wafer’s dataflow fabric. Debugging tools also improved: csdbg now displays per‑core stall cycles and SRAM pressure in a flame graph, saving two days while tracking a pipeline bubble in a MoE expert routing kernel.

The trade‑offs are still significant. Users remain tied to Cerebras’ compiler pipeline, and custom CUDA kernels do not port. Model parallelism is fixed at wafer granularity; there is no pipeline parallelism across systems yet, so activation memory scales with model size per wafer. The $2.5 M per system price also requires sustained, large‑scale workloads to amortize the cost. However, for training frontier models at 100 B+ parameters where GPU clusters spend half their cycles waiting on all‑reduce, the CS‑4 rack density appears to be the only way to keep the power bill honest.