Fine-tuning Ollama settings unlocks hidden speed gains on weak GPUs

Ollama lets users run large language models locally, but most configurations overlook how hardware constraints—especially in systems with modest GPUs—drag down performance. The default ollama run llama3 setup leaves significant room for improvement, particularly when managing memory allocation and quantization.

The biggest slowdown often comes from VRAM overflow. When a model exceeds GPU memory, Ollama automatically shifts layers to system RAM, turning a compute bottleneck into a PCIe bandwidth problem. For users with 8GB or 12GB GPUs, aggressive quantization and targeted layer offloading can outperform simply switching to smaller models.

Adjusting quantization levels makes a measurable difference. The default 4-bit (q4_0) is common, but switching to q3_K_M or q2_K cuts VRAM usage while keeping chat performance nearly intact. To enforce these settings, users must edit the Modelfile to explicitly specify a GGUF version rather than relying on Ollama’s defaults.

The num_gpu parameter also demands attention—it determines how many layers stay on the GPU. Without manual tuning, even a few layers may default to CPU, creating unpredictable slowdowns. For example, on a 12GB GPU running a model close to capacity, manually pinning critical layers keeps the GPU fully utilized and avoids RAM thrashing.

Context window size (num_ctx) silently impacts speed too. The KV cache scales directly with input length, and reducing it from 8k to 4k can prevent disk swapping when VRAM is tight. This single change can push tokens-per-second from as low as 5 to over 50 in constrained setups.

For API users, tracking prompt_eval and eval_duration reveals hidden issues. A sudden spike in prompt_eval time often means memory pressure during prefill, which can inflate first-token latency beyond 100ms—making real-time local inference impractical.

While newer techniques like Mixture of Experts (MoE) promise efficiency gains, practical optimization for consumer hardware still hinges on memory management. The focus shifts from merely running models to extracting every possible performance increment from limited hardware.