Optimizing vLLM Throughput for Multi-LoRA Serving on NVIDIA A100 GPUs

Running multi-LoRA adapters on A100s often feels like a gamble with VRAM and throughput until you nail the max_loras and max_lora_rank settings. Most people just throw the default vLLM config at their cluster and wonder why the tokens per second (TPS) tank as soon as they scale to 10+ different adapters.

The bottleneck isn't usually the raw compute of the A100; it's the memory overhead of managing multiple adapter weights and the resulting fragmentation. If you're serving a base model (like Llama-3) with dozens of specialized LoRAs, you need to stop treating them as separate models and start leveraging vLLM's dynamic LoRA swapping.

My current production setup for A100-80GB looks like this:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-8B \
    --enable-loras \
    --max-loras 32 \
    --max-lora-rank 64 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 8192

The "gotcha" here is the max-lora-rank. If you set this higher than your actual adapter ranks, vLLM pre-allocates a larger buffer than necessary, eating into your KV cache. I spent two days debugging a "CUDA Out of Memory" error only to realize I had max-lora-rank at 256 while all my adapters were actually rank 16. Tightening that value immediately recovered 4GB of VRAM, which allowed me to increase the batch size.

To actually maximize throughput, you have to manage the request distribution. Since vLLM uses a modified version of PagedAttention to handle LoRA, the scheduler tries to group requests using the same adapter. If your traffic is perfectly distributed (1 request per adapter), you'll see a performance dip. If you can batch requests for the same adapter, the throughput spikes.

If you're writing a wrapper or a gateway in front of vLLM, use this logic to prevent the scheduler from thrashing:

# Pseudo-code for a basic request grouper
import asyncio
from collections import deque

pending_requests = deque()

async def dispatch_to_vllm():
    while True:
        # Group by adapter_id to maximize vLLM's internal batching
        current_batch = group_by_adapter(pending_requests) 
        for req in current_batch:
            await send_to_vllm_api(req)
        await asyncio.sleep(0.01)

Another productivity gain comes from using the cpu_offload_gb flag if you have hundreds of adapters. While A100s have massive HBM, you can't fit 200 LoRAs in there without sacrificing the KV cache. Offloading the least-used adapters to CPU RAM is a lifesaver, though you'll take a slight latency hit on the first request for a swapped-in adapter.

Key Performance Checklist:

• Align max-lora-rank with your actual weights. Don't guess; check your adapter_config.json.
• Monitor the KV cache usage. If gpu_cache_usage is constantly at 95%, your max-loras setting is likely too aggressive, causing frequent swapping.
• Use FlashAttention-2. Ensure your environment is properly configured so vLLM isn't falling back to slower kernels.
• Prefer FP16 over BF16 for the adapters if you aren't seeing stability issues, as it can sometimes provide a marginal speedup on specific A100 kernels.

The biggest productivity jump for me was moving away from deploying one model per adapter. Running one vLLM instance with 32 LoRAs is roughly 4x more resource-efficient than running 32 separate small instances, simply because the base model weights are shared in memory.