Optimizing Local LLM Inference Speed Using vLLM and NVIDIA Docker

Running LLMs locally often feels like a battle against VRAM bottlenecks and sluggish tokens-per-second, but switching from basic Transformers-based loading to vLLM changes the game entirely. The secret sauce is PagedAttention, which stops the KV cache from fragmenting your memory, allowing for much larger batch sizes and significantly faster inference.

To get this running without polluting your host OS with CUDA version conflicts, NVIDIA Container Toolkit is non-negotiable. Once you have the toolkit installed, you can spin up a vLLM instance in seconds.

Here is the exact command I use to launch a Llama-3-8B instance. I've tuned the --gpu-memory-utilization to 0.9 to leave just enough breathing room for the system while maximizing the cache:

docker run --gpus all \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --gpu-memory-utilization 0.9 \
    --max-model-len 8192

Crucial config tips for performance:

The --ipc=host flag. If you omit this, you'll likely hit shared memory limits during high-concurrency requests, leading to random crashes or degraded performance.

Quantization. If you're tight on VRAM, don't just settle for a smaller model. Use AWQ or GPTQ versions. vLLM handles these natively. Just add --quantization awq to the command. I've seen a 2x throughput increase on a single RTX 3090 when moving from FP16 to AWQ with negligible loss in logic.

Max Model Length. vLLM pre-allocates the KV cache based on the --max-model-len. If you leave this at the model's default (which could be 128k for some newer models), it will eat all your VRAM immediately and throw an Out-Of-Memory (OOM) error before it even starts. Always cap this to what you actually need for your specific use case.

One major "gotcha" I encountered was the interaction between vLLM and local firewall settings when trying to access the API from a different container. Since vLLM mimics the OpenAI API format, it's incredibly easy to plug into existing tools, but remember that by default, it binds to 0.0.0.0 inside the container. Ensure your Docker port mapping is explicit.

For those integrating this into a Python app, stop using heavy libraries for the request. A simple httpx call is all you need since the output is standard JSON:

import httpx

async def get_ai_response(prompt):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "http://localhost:8000/v1/chat/completions",
            json={
                "model": "meta-llama/Meta-Llama-3-8B-Instruct",
                "messages": [{"role": "user", "content": prompt}],
                "temperature": 0.7
            },
            timeout=60.0
        )
        return response.json()['choices'][0]['message']['content']

The productivity gain here is massive. I went from waiting 3-5 seconds for a response using a naive PyTorch implementation to nearly instantaneous streaming. If you are running a local RAG pipeline or a coding assistant, vLLM is the only way to make the latency feel "natural" rather than "robotic."