Optimizing vLLM Throughput for High-Concurrency Llama 3 Deployment on A100s

A100s are beasts, but if you're just running python -m vllm.entrypoints.openai.api_server with default settings, you're leaving a massive amount of throughput on the table. I've been stress-testing Llama 3 (70B) on a cluster of A100s, and the gap between "out-of-the-box" and "tuned" performance is shocking—we're talking about 2-3x more requests per second once you nail the KV cache and scheduling.

The biggest bottleneck in high-concurrency scenarios is almost always the KV cache memory pressure. vLLM uses PagedAttention to manage this, but the gpu_memory_utilization parameter is a blunt instrument. By default, it's 0.9, but if you're seeing "out of memory" crashes during peak spikes, don't just lower it. Instead, focus on max_num_seqs and max_model_len.

For Llama 3, if you don't actually need the full 8k or 128k context window for every single request, capping max_model_len is the fastest way to boost throughput. It frees up slots in the block manager, allowing more concurrent sequences to reside in memory.

Here is the specific launch config I'm currently using for a high-throughput production endpoint:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B \
    --tensor-parallel-size 4 \
    --max-model-len 4096 \
    --max-num-seqs 256 \
    --gpu-memory-utilization 0.95 \
    --kv-cache-dtype fp8

The --kv-cache-dtype fp8 flag is a game-changer on A100s. It effectively doubles the number of tokens you can store in the cache with negligible loss in precision. If you're running 70B, this is non-negotiable for high concurrency.

One "gotcha" I hit: if you increase max_num_seqs too high, you'll see your Time Per Output Token (TPOT) climb. There's a sweet spot where the GPU is fully saturated but not thrashing. I use a simple loop with curl to measure the latency ceiling. If the TPOT spikes beyond 50ms, I back off the max_num_seqs by 10%.

For those using Cursor to write their deployment scripts, I've found that prompting Claude 3.5 Sonnet with the actual vLLM source code for the llm_engine.py helps it suggest better batching strategies. Instead of asking "how to optimize vLLM," I feed it the specific version's scheduler logic and ask: "Based on this scheduler, how will increasing max_num_seqs impact the prefill phase latency for Llama 3?"

Key Productivity Gains from this Setup:

• FP8 KV Cache: Allowed us to move from 64 to 256 concurrent requests without hitting OOM.
• Tensor Parallelism (TP): Set to 4 for 70B on A100s to ensure the model fits while keeping communication overhead low.
• Context Capping: Reducing max_model_len from 8k to 4k increased our total throughput (tokens/sec) by roughly 40% because we stopped wasting memory on unused context headroom.

If you're still seeing bottlenecks, check your max_num_batched_tokens. For Llama 3, aligning this with your hardware's compute capability prevents the "stutter" during the prefill phase of long prompts. I usually set this to match the max_model_len or slightly higher to keep the pipeline full.