Optimizing Ollama Local Deployment for Faster Code Completion in VS Code

Running local LLMs for code completion often feels like a trade-off between privacy and latency, but the "lag" people complain about with Ollama in VS Code is usually a configuration mismatch rather than a hardware limitation. If you're using the Continue.dev or Llama Coder extensions, the default settings often leave too much performance on the table.

The biggest bottleneck isn't just the model size; it's the context window and the way the GPU handles KV cache. For autocomplete (tab-completion), you need a model optimized for "fill-in-the-middle" (FIM) tasks. DeepSeek-Coder-V2 or StarCoder2 are the gold standards here, but running them with default Ollama settings often leads to a 1-2 second delay that breaks your flow.

To actually make local completion feel instantaneous, you need to create a custom Modelfile to tweak the parameters. Don't just ollama run deepseek-coder; create a specific profile for completion:

FROM deepseek-coder-v2:16b
PARAMETER num_ctx 4096
PARAMETER num_gpu 99
PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER stop "<｜end of sentence｜>"
PARAMETER stop "</filler>"

The logic behind these tweaks:
num_ctx 4096: Setting this too high (like 32k) eats up VRAM and slows down the Time To First Token (TTFT). For autocomplete, the model only needs the immediate surrounding code, not your entire project history. 4k is the sweet spot for speed.
temperature 0.2: You don't want "creativity" in a semicolon or a bracket. Low temperature ensures the completion is deterministic and precise.
num_gpu 99: This forces Ollama to push as many layers as possible to the VRAM. If you see "CPU" usage spiking in your activity monitor during completion, your layers are spilling over, which kills the speed.

After saving this as Modelfile, build it:

ollama create coder-fast -f Modelfile

Once the model is optimized, the VS Code side needs a specific setup. In your config.json for Continue, separate your "Chat" model from your "Autocomplete" model. Using a massive model for both is a rookie mistake. Use a heavy-hitter like Llama 3.1 for the chat sidebar, but point the tabAutocompleteModel specifically to your coder-fast build.

One major "gotcha" is the prompt template. If the extension doesn't send the FIM markers correctly, the model will just try to "continue" the text rather than "filling" the gap, leading to weird repetitions. Ensure your provider is set to ollama and the model name matches your custom build exactly.

If you're still seeing sluggishness on a Mac M-series chip, check if other apps are hogging the Unified Memory. Ollama is aggressive with memory allocation, but if Chrome is eating 8GB, the swap file will slow your completions to a crawl. I've found that limiting the num_ctx even further to 2048 makes a noticeable difference on 16GB RAM machines without sacrificing much accuracy.

The productivity gain here is moving from "waiting for the AI" to "accepting the AI." When the latency drops below 200ms, it stops being a tool you call upon and starts being a natural extension of your typing.