DGX Spark Inference: 90 tok/s on Large Models

Hitting 55-90 tok/s on large models without even touching speculative decoding is a massive win for our current deployment. We've been testing a new inference stack specifically for DGX Spark clusters to handle multi-model agentic workflows, and the LlamaBench numbers are actually holding up under simulated traffic.

The real-world problem we're solving at work is that enterprise agentic apps usually need a mix of specialized models of different sizes. Buying dedicated multi-GPU stacks for every single model in a workflow is way too expensive. This setup lets us run a private, lower-cost stack that handles multiple models across a 2 DGX Spark cluster.

Here is the raw performance data from the LlamaBenchy tests (no quantization, no spec decode):

• DeepSeek V4 Flash C4: 55.99 decode-system-tok/s
• Gemma 4 26B A4B C4: 63.75 decode-system-tok/s
• Nemotron 3 Nano Omni 30B NVFP4 C4: 90.83 decode-system-tok/s

We also tested a scenario with one endpoint controlling three different model activations. The scheduler batches bursts and coordinates ranks, only swapping the resident model at safe boundaries. The results were:

• DeepSeek V4 Flash: 49.30 decode-tok/s (16s activation wait)
• Gemma 4 26B A4B: 64.67 decode-tok/s (6s activation wait)
• Nemotron 3 Nano Omni 30B NVFP4: 93.31 decode-tok/s (2s activation wait)

This is a solid deep dive into how we're scaling our AI workflow internally. We're expecting another 20% bump in performance once we finish the next round of optimization.

For those wanting the full technical breakdown, the report is here:
https://woolyai.com/ai-compute-software/dgx-spark-inference-stack/