Optimizing Local LLM Inference with vLLM and Ollama for Python Development

Local LLM inference is a game-changer for Python devs who can't send proprietary code to a cloud API, but the "out-of-the-box" experience is often sluggish. If you're toggling between Ollama for quick tests and vLLM for actual throughput, you've probably noticed that memory management is where most people trip up.

Ollama is fantastic for the "it just works" experience, but for a development workflow where you're running a local agent or a RAG pipeline, the latency on large prompts can be killer. I've found that switching to vLLM for the heavy lifting—specifically using PagedAttention—cuts my time-to-first-token significantly when dealing with long context windows.

The real productivity gain comes from setting up a vLLM server as an OpenAI-compatible backend and pointing your AI coding tools (like Cursor or a custom Python script) to that local endpoint.

Here is how I actually deploy the vLLM engine to maximize VRAM utilization without crashing my Xformers:

# Install vLLM
pip install vllm

# Launch a model (e.g., Mistral-7B) with optimized GPU memory utilization
# gpu_memory_utilization=0.9 prevents the system from OOMing by leaving a bit of headroom
python -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-Instruct-v0.2 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 8192

One major "gotcha" I encountered: if you're running Ollama and vLLM simultaneously, they will fight for the same VRAM. Ollama tends to keep models loaded in memory for a set timeout. To stop them from clashing, I force Ollama to unload models immediately after use by setting the OLLAMA_KEEP_ALIVE environment variable to 0s.

For the Python side, don't rewrite your client code for every different local runner. Use the openai library and just swap the base_url. This makes switching between a lightweight Ollama instance (for simple unit test generation) and a beefy vLLM instance (for codebase analysis) a one-line config change.

from openai import OpenAI

# Switch this URL based on whether you're using Ollama (11434) or vLLM (8000)
client = OpenAI(
    base_url="http://localhost:8000/v1", 
    api_key="token-not-needed-for-local"
)

response = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    messages=[{"role": "user", "content": "Refactor this Python function for time complexity: ..."}]
)
print(response.choices[0].message.content)

My current optimization stack for local dev:

• Model Quantization: Always go for GGUF via Ollama for rapid prototyping, but use AWQ or FP8 via vLLM when I need the inference speed for automated scripts.
• KV Cache Tuning: In vLLM, if you're hitting memory limits with long files, tweak the --max-model-len. Setting it too high wastes VRAM; too low and the AI "forgets" the top of your script.
• Context Shifting: Ollama's handling of prompt caching is surprisingly good for iterative chat, but for batch processing 50+ Python files, vLLM's continuous batching is objectively superior.

The biggest jump in my workflow happened when I stopped treating local LLMs as "chatbots" and started treating them as local microservices. By decoupling the inference engine (vLLM) from the interface, I can restart my IDE or crash my Python script without having to reload a 10GB model into the GPU every single time.