Optimizing LLM Latency and Inferenc

DeepSeek-V3 has completely shifted the goalposts for open-weights inference efficiency, especially when you compare its MoE (Mixture-of-Experts) architecture to the denser GPT-4o or Claude 3.5 Sonnet. If you're running benchmarks on latency, the "Time to First Token" (TTFT) is where the battle is won or lost, and right now, the gap between proprietary APIs and self-hosted optimized models is closing fast.

I've been stress-testing these across a few different setups—ranging from vanilla vLLM to TensorRT-LLM—and the delta in tokens per second (tps) is staggering depending on how you handle the KV cache.

The Performance Breakdown

DeepSeek-V3/R1 (via vLLM/SGLang)
The MoE architecture is a cheat code for latency because it only activates a fraction of its parameters per token. On an H100 cluster, I'm seeing significantly higher throughput than Llama 3.1 405B. The trade-off is VRAM; you need massive memory to hold the weights, but once they're loaded, the inference speed is lean. It's the best choice for high-volume RAG pipelines where you need reasoning but can't afford a 3-second lag.

GPT-4o
Still the king of consistency. The latency is predictable, but you're at the mercy of OpenAI's load balancing. During peak hours, the "jitter" in response time is noticeable. It's faster than Claude for short bursts, but the quality-to-latency ratio starts to dip when you hit complex system prompts.

Claude 3.5 Sonnet
Noticeably slower TTFT than GPT-4o. However, for coding tasks, the "perceived latency" is lower because it rarely hallucinates the structure, meaning you spend less time iterating. If you're benchmarking purely on ms/token, Claude loses; if you're benchmarking on "time to correct solution," it's a toss-up.

Gemini 1.5 Pro
The massive context window is its selling point, but the latency penalty for long contexts is real. Once you push past 100k tokens, the time to process the prompt spikes. It's an incredible tool, but not what I'd use for a real-time chat interface.

If you're trying to squeeze more performance out of your local deployments, stop using basic transformers and move to a continuous batching engine. I've found that switching to SGLang provided a roughly 20% boost in throughput over vLLM for my specific prompt distributions.

For those optimizing prompts to reduce latency, remember that the output length is the primary driver of total latency. Using "constrained output" or forcing JSON mode can actually slow down the model if the schema is too complex. Instead, try a prompt like this to trim the fat:

Respond in concise technical shorthand. Eliminate introductory phrases and conversational filler. Output only the raw data points.

The Verdict on Optimization

KV Cache Quantization: Moving from FP16 to INT8 for the KV cache is the single biggest win for increasing concurrency without crashing your GPU. You lose negligible accuracy but can double your batch size.

Speculative Decoding: This is the real frontier. Using a tiny "draft" model (like a 1B parameter model) to predict tokens for a larger model (like a 70B) can slash latency by 30-50% for predictable text.

Prompt Caching: If you're using Claude or Gemini, leverage their prompt caching. Sending the same 10k token system prompt every time is a waste of compute and adds unnecessary latency to every single turn.