Optimizing Local LLM Inference Speed Using Ollama and GPU Memory Tuning

Getting a local LLM to run is one thing, but getting it to actually feel snappy is where most developers hit a wall. The gap between "it works" and "it's usable" usually comes down to how Ollama handles VRAM allocation and the specific quantization of the model you're pulling. If you're seeing token generation speeds that feel like a slow typewriter, you're likely dealing with memory swapping or inefficient layer offloading.

The core issue is that Ollama tries to be "magic" by automating the distribution of model layers between your GPU and CPU. While this is great for beginners, it's a nightmare for performance tuning. When a model is slightly too large for your VRAM, Ollama might offload just a few layers to the CPU. This creates a massive bottleneck because the system has to wait for the slower system RAM to catch up, killing your tokens-per-second (t/s) rate.

To actually optimize this, you need to stop treating the .ollama library as a black box. The most immediate win is manipulating the num_gpu parameter in the Modelfile. If you have a GPU with 12GB of VRAM and you're running a 7B model that's pushing the limits, explicitly forcing the layer count can prevent the driver from stuttering.

You can create a custom version of a model to lock in these settings:

FROM llama3
PARAMETER num_gpu 33
PARAMETER num_ctx 4096

By creating a new model from this Modelfile, you ensure that Ollama isn't guessing how many layers to shove into the GPU on every boot. More importantly, pay attention to num_ctx. The context window consumes VRAM linearly. If you leave it at a high default, you're eating into the space needed for the model weights, which triggers that dreaded CPU fallback. Dropping your context window to 2048 or 4096 if you don't need long-form memory can often jump your inference speed by 2x or 3x.

The industry trend is moving toward "small but mighty" models (like Phi-3 or Mistral), but the bottleneck remains the memory bus. We're seeing a shift where quantization isn't just about saving space; it's about fitting the entire KV cache into the fastest tier of memory. If you're using 4-bit quantization (q4_K_M), you're usually in the sweet spot. Going to q2 is too lossy, but q8 often pushes users over their VRAM limit, leading to the performance cliff mentioned earlier.

For developers building local RAG pipelines, this tuning is non-negotiable. A slow LLM makes the entire application feel broken, regardless of how fast your vector database is. The goal should always be "Zero CPU Offloading." If you can't fit the model entirely on the GPU, you're better off dropping to a smaller, more aggressively quantized model that fits entirely in VRAM than trying to run a larger model in a hybrid state.

Key optimization levers for local inference:

• Quantization Selection: Stick to q4_K_M for the best balance of intelligence and speed.
• Context Window Capping: Lower num_ctx to free up VRAM for weights.
• Layer Locking: Use a custom Modelfile to set num_gpu and avoid dynamic allocation stutters.
• VRAM Monitoring: Use nvidia-smi during the first token generation to see if you're hitting the ceiling.

Ultimately, Ollama has democratized local LLMs, but the "one-click" experience hides the reality that GPU memory management is still a manual art. The developers who take ten minutes to tune their Modelfiles are the ones who actually get production-ready speeds on consumer hardware.