NVIDIA's new Nemotron-Labs-3 "Puzzle" is a game changer

NVIDIA just dropped some fascinating research with their Nemotron-Labs-3-Puzzle-75B, and it’s a masterclass in architectural efficiency. Instead of just making a smaller model and hoping for the best, they used an "iterative puzzle" approach—compressing, distilling, and rescoring repeatedly to maintain intelligence while slashing parameter counts.

The technical specifics are wild. They managed to shrink a 120B model down to 75B without losing the hybrid Mamba-Transformer MoE structure. The real magic is in the serving metrics. On an 8xB200 setup, they're seeing 2.03x server throughput for decode-heavy tasks. Even more impressive is the memory efficiency: by dropping the weights from 70GB to 44.5GB, they’ve increased concurrency on a single H100 from one request to eight at a 1M context window.

What strikes me is the "search-to-fit" strategy. They didn't just scale down a teacher model; they searched for an architecture that hits a specific serving operating point. It’s a shift from "how big can we make this" to "how efficient can we make this for production." While the compression gains are less pronounced in compute-bound prefill scenarios, the massive boost in concurrency and throughput for long-context decoding makes this a massive win for anyone running inference at scale.