Optimizing vLLM Throughput for High-Concurrency Serving on NVIDIA A100 GPUs

Getting the most out of vLLM on A100s isn't just about installing the library and hitting the API endpoint; if you leave the defaults, you're likely wasting 30-50% of your potential throughput. I've spent the last few weeks tuning a cluster of A100 80GBs for a high-concurrency internal tool, and the delta between "out-of-the-box" and "tuned" is massive.

The biggest bottleneck in high-concurrency serving is almost always KV cache fragmentation. While PagedAttention solves the memory waste, the gpu_memory_utilization parameter is a dangerous lever. By default, it's 0.9, but on A100s, if you have other processes (or even monitoring agents) eating VRAM, you'll hit OOMs during peak bursts. I found that dropping it to 0.85 actually improved stability without a noticeable hit to throughput, as it gave the system more breathing room for the activation tensors.

The real magic happens when you tune the max_num_seqs and max_model_len. If you're serving short-to-medium length responses (like a chat bot), don't let vLLM reserve space for 32k tokens per request. Hard-capping the model length forces the engine to pack more sequences into a single batch.

Here is the launch command I'm currently using for Llama-3-70B on A100s to maximize requests per second (RPS):

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B \
    --tensor-parallel-size 4 \
    --max-model-len 8192 \
    --max-num-seqs 256 \
    --gpu-memory-utilization 0.85 \
    --kv-cache-dtype fp8

A few critical takeaways from this setup:

FP8 KV Cache
If you're on A100s (and especially H100s), use --kv-cache-dtype fp8. This effectively doubles the number of requests you can fit in the KV cache without a perceptible drop in perplexity. It's the single fastest way to increase concurrency.

Tensor Parallelism (TP) vs. Throughput
I noticed that increasing TP beyond what is strictly necessary to fit the model actually introduces communication overhead. For 70B models, TP=4 on A100s is the sweet spot. Going higher often yields diminishing returns on latency and can actually hurt total throughput due to the NCCL overhead.

Handling the "Request Spike"
When concurrency hits 500+ simultaneous users, the scheduler can struggle. I've been using a lightweight Nginx load balancer in front of multiple vLLM replicas. The trick is to use a least_conn algorithm rather than round-robin. Since different requests have different output lengths, round-robin leads to some GPUs being slammed while others are idle.

The Cursor Workflow for Tuning
To iterate on these configs, I don't manually edit shell scripts. I keep a deploy.env file and use Cursor's Composer (Cmd+I) to generate different benchmark permutations. I'll prompt it: "Create 5 variations of the vLLM launch command, incrementing max_num_seqs by 32 and decreasing gpu_memory_utilization by 0.02, then format them as a bash loop for testing." This allows me to run a grid search for the optimal throughput settings in minutes.

One gotcha: watch your CPU RAM. vLLM's memory management is aggressive on the GPU, but the Python overhead for managing thousands of concurrent requests can spike system RAM. If you see the process getting killed without a CUDA OOM, check your dmesg for OOM-killer events on the host.