Tuning vLLM Parameters is Essential for Maintaining High Throughput in Production Serving

PagedAttention solves memory fragmentation, making vLLM a standard for serving large language models. However, default settings often fail during production traffic surges. As concurrent requests move from a few testers to thousands, the primary bottleneck shifts from GPU compute to request scheduling and KV cache management.

High-concurrency environments require a balance between total tokens generated per second and individual request latency, specifically inter-token delivery and time-to-first-token. Increasing max_num_seqs without considering KV cache memory pressure is a common error. This leads to frequent preemptions where vLLM offloads requests from GPU to CPU RAM, which harms the user experience.

Maximizing performance under heavy loads requires exact adjustments to max_model_len and gpu_memory_utilization. A common setting of 0.9 for gpu_memory_utilization can trigger Out-Of-Memory (OOM) crashes during traffic spikes or in multi-tenant setups, as it ignores temporary buffers and LoRA adapter overhead.

Combining an optimized scheduler with Continuous Batching provides the best throughput gains. While short prompts and long generations allow for larger batches, RAG applications with long contexts fill the KV cache quickly. Setting enable_prefix_caching=True avoids recomputing KV caches for shared context or system prompts, which can improve TTFT by 2x-5x.

A typical production configuration for A100/H100 hardware looks like this:

python -m vllm.entrypoints.openai.api_server \
    --model facebook/opt-13b \
    --gpu-memory-utilization 0.90 \
    --max-model-len 8192 \
    --max-num-seqs 256 \
    --enable-prefix-caching \
    --tensor-parallel-size 2

Modern deployments are moving toward dynamic scaling, where max_num_seqs changes based on real-time telemetry. This optimization increases hardware utilization and reduces the cost per token.

Treating vLLM as a static configuration is a mistake, as an optimized setup may require only two GPUs compared to ten for a default deployment. If server logs show "preempted" messages, concurrency settings have exceeded available VRAM, sacrificing stability for theoretical throughput.