Four RTX 3060s can actually push 100 tok/s prompt processing on

Getting a 144 GiB model to run on a total of 48 GB VRAM sounds like a nightmare, but I managed to get DeepSeek-V4-Flash-0731 (UD-Q4_K_XL GGUF) stable on four RTX 3060 12GB cards. The real win here isn't just that it runs, but that I'm hitting nearly 100 tok/s during prompt processing while maintaining a massive context window of around 360k tokens.

The secret sauce is a very specific, non-intuitive tensor split and expert offloading strategy in llama.cpp. If you try to calculate the layout analytically, you'll probably fail because the interaction between -ncmoe and explicit tensor overrides is weird.

The Hardware Stack

• GPU: 4× NVIDIA RTX 3060 12GB (48 GB Total VRAM)
• CPU: Intel Core i9-10920X (12C/24T)
• RAM: 128 GB DDR4-3200 (Quad-channel is key here since most of the model sits in system memory)
• Engine: llama.cpp build b10181

The Optimized Deployment

To get this working, I had to push almost all non-expert tensors to GPU0 and surgically place the remaining experts on the other cards. Here is the exact command I used for the best balance of speed and stability:

llama-server \
-m DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf \
-c 368640 \
-ncmoe 34 \
-ts 100,1,1,1 \
-ot 'blk.(3[4-6]).ffn_.*_exps=CUDA1,blk.(3[7-9]).ffn_.*_exps=CUDA2,blk.(4[0-2]).ffn_.*_exps=CUDA3' \
-ctk q8_0 \
-ctv q8_0 \
-b 2048 \
-ub 2048 \
-np 1 \
-lm none \
--threads 20 \
--flash-attn on

Performance Breakdown

I tested this with a ~20.5k token prompt, and the results were surprisingly snappy for a setup that is heavily relying on system RAM:

• Prompt processing: 99.4 tok/s
• Text generation: 10.1 tok/s
• VRAM Headroom (GPU0): 671 MiB free

Key Technical Takeaways

The most critical part of this AI workflow is how the experts are handled. By setting -ncmoe 34, I keep experts from blocks 0–33 in system RAM. I then manually distribute the remaining nine expert layers across GPUs 1, 2, and 3 (three layers per card).

The -ts 100,1,1,1 split is aggressive; it forces the attention and KV allocations onto GPU0, leaving just enough room on the other three cards to hold those specific expert weights.

I also found that the physical microbatch size (-ub) is the biggest performance lever. Dropping -ub to 1024 tanked my prompt processing to about 63.4 tok/s. Boosting it to 2048 is what got me to that 100 tok/s mark, though it eats more VRAM. If you need a safer margin or a larger context (up to 524k), stick with 1024.

A few other stability notes:

• KV Cache: Using q8_0 is the sweet spot. F16 KV almost OOM'd my cards.
• Memory Mapping: I disabled it with -lm none for better stability.
• Slots: Keep -np 1 because multiple slots multiply the KV-cache requirements and will kill your VRAM instantly.