HotPin: Running 120B MoE on 24GB RAM

大熊爱学习 Intermediate 1h ago Updated Jul 26, 2026 154 views 6 likes 1 min read

Running a 120B MoE model on 24GB of RAM without losing a single bit of precision is technically possible thanks to a clever memory management approach called HotPin. Instead of trying to cram the whole model into RAM or accepting the quality loss of heavy quantization, this method treats the NVMe drive as a tiered memory layer.

I've been looking into the benchmarks for this llama.cpp patch, and the results on an AMD Ryzen AI 9 HX 370 (23.6GB LPDDR5X) are pretty revealing for anyone running LLM agents on consumer hardware:

gpt-oss:120b: Disk 58.5GB → Min RAM 19.1GB (67% savings) at 3.84 tok/s
qwen3:30b-a3b: Disk 18.0GB → Min RAM 10.4GB (42% savings) at 19.7 tok/s
gemma4:26b-a4b: Disk 16.2GB → Min RAM 10.6GB (35% savings) at 11.5 tok/s
GLM-4.7-Flash: Disk 19.0GB → Min RAM 13.3GB (30% savings) at 12.4 tok/s

The most impressive part is that the output is SHA-256 bit-identical to a full-RAM run. This isn't a "compressed" version of the model; it's a lossless deployment.

The technical implementation is surprisingly lean (around 50 lines of C++). It works by profiling MoE expert routing frequencies, using mmap for the whole model, but only using mlock for the "hottest" experts. It then leverages posix_fadvise to prefetch cold experts from the NVMe just before they are needed.

For those of us obsessed with local deployment, this solves a massive pain point. If the model exceeds your RAM, you usually see a performance cliff. HotPin actually provides a 45% speedup in those scenarios (for gpt-oss, it jumped from 2.64 to 3.84 tok/s) compared to standard swapping.

If you want to dive into the implementation, the code is available here: