Running LLMs locally is finally bec

Ollama and LM Studio have basically turned my local machine into a legitimate development sandbox, meaning I no longer have to send every single snippet of boilerplate or sensitive config to a cloud API. For a long time, "local LLMs" felt like a hobby for people with four A100s, but with the optimization of Llama 3.1 and Mistral, the 12GB-24GB VRAM range is actually productive.

The biggest productivity gain for me isn't using a local model for complex architecture—Claude 3.5 Sonnet still wins there—but using local models for "janitorial" coding tasks. I use a local Llama 3.1 8B via Ollama to handle things like writing unit tests for simple utility functions or converting JSON schemas to TypeScript interfaces. It's instant, it's free, and it doesn't eat into my monthly token quota.

If you're trying to integrate this into your workflow, don't just use the chat UI. The real power is hooking it into your IDE. I’ve been using the Continue.dev extension in VS Code, which lets me swap between a local Ollama instance and a cloud LLM on the fly.

Here is the config I use in my config.json for Continue to route simple completions to a local model while keeping the heavy lifting for the cloud:

{
  "models": [
    {
      "title": "Llama 3.1 8B (Local)",
      "provider": "ollama",
      "model": "llama3.1"
    },
    {
      "title": "Claude 3.5 Sonnet",
      "provider": "anthropic",
      "model": "claude-3-5-sonnet-20240620"
    }
  ],
  "tabAutocompleteModel": {
    "title": "StarCoder2-3b",
    "provider": "ollama",
    "model": "starcoder2:3b"
  }
}

One major gotcha: watch your context window. Local models often default to a small context (like 2k or 4k tokens). If you're asking it to analyze a file and it starts hallucinating or forgetting the top of the code, you need to explicitly bump the num_ctx in your Ollama Modelfile.

To fix this, I create a custom model file:

FROM llama3.1
PARAMETER num_ctx 16384
PARAMETER temperature 0.2

Then run: ollama create llama3-big-ctx -f Modelfile

The performance hit is noticeable once you cross the 8k threshold on consumer hardware, but it's better than the model losing its mind halfway through a function.

Another pro tip: stop trying to run 70B models on a laptop unless you have 64GB+ of unified memory on a Mac. The quantization loss on 4-bit models is acceptable, but the latency on a 70B model running on system RAM (CPU offloading) is painful. Stick to 8B or 12B models for coding assistance; they are snappy enough to maintain your flow state.

The real win here is the privacy. When I'm working on a project with a strict NDA or handling actual production API keys in a .env file, I just toggle to the local model and I can feed the entire file into the prompt without worrying about where that data is being stored or trained.