Tuning Ollama GPU Offloading and VRAM Settings Improves Local LLM Inference Speed

Local LLM adoption is currently hindered by execution speed rather than compatibility. While Ollama hides CUDA complexities, poor layer distribution causes models to spill from VRAM into slower RAM, potentially dropping performance from 40 tokens-per-second to 2.

The num_gpu parameter controls VRAM layer placement. Inaccurate estimates lead to system RAM leakage in DDR4/DDR5; for instance, 7B parameter models with Q4_K_M quantization require 4.5GB to 5GB. Splitting 70B models across consumer GPUs introduces PCIe bottlenecks that slow execution. To prevent this, users can manually override distribution in a custom Modelfile:

FROM llama3
PARAMETER num_gpu 32

Performance peaks when models stay entirely in GPU memory to avoid OS swapping. Increasing context windows to 32k causes linear memory growth, necessitating layer adjustments to stop crashes. Switching from Q8 to Q4 quantization can double speed with minimal perplexity loss, provided 1-2GB of VRAM remains free for system tasks. Hardware differences are stark; an RTX 4090 may process a prompt in 1 second while a Mac M2 takes 15 seconds.

Beyond hardware, Ollama released structured outputs in v0. At the minute, this feature is limited to producing JSON formats. Investigating how structured outputs work compared to OpenAI reveals that generating reliable JSON transforms LLMs from simple chatbots into more useful tools. Since predicting next tokens is often flaky, the v0 Git diff shows the specific additions that enable this reliability. Understanding how Ollama interacts with models is essential for developers who should integrate hardware detection to dynamically adjust num_gpu or suggest lower quantizations.