LangChain + Ollama: Run a Local LLM Without the Cloud Bill

My OpenAI bill hit $47 last month. Not because I was building something clever — a loop I forgot to kill ran for 12 hours, puking through a GPT-4 JSON extraction chain. That was the moment I decided to move at least part of my workflow onto a local model. This walkthrough is how I run LangChain with Ollama instead of a pricey API. No GPU, no Docker hoops, just a laptop and a stubborn streak.

Why bother running a local model at all?

Honest take up front: a 7-billion-parameter model running on my M1 MacBook is not going to out-code GPT-4o. It stumbles on multi-step reasoning, and its answer quality depends heavily on how I phrase the prompt. But for the stuff I actually use LLMs for — extracting fields from messy text, summarizing internal docs, writing boilerplate tests — it's shockingly good. The privacy angle matters too: I work with client code that should never leave the building. And the price. Zero. Forever.

Local is also just a better way to learn. I know exactly which model is running, how much RAM it's eating, and what happens when one component changes. No black box.

What you need

Install Ollama. On macOS or Linux it's one command:

curl -fsSL https://ollama.com/install.sh | sh

On Windows you'll want WSL2 and then the same command. As of writing, I'm on Ollama 0.1.29 and LangChain 0.2.5. These move fast, so pin your versions if you want reproducibility.

Then pull a model:

ollama pull llama3.1:8b

That's about 4.7 GB of download. If you're on a small machine, phi3:mini (3.8B) runs okay in 8 GB of RAM, but I found its output a bit dull. There's also qwen2.5:7b which is faster on my machine and better at code than llama3.1 at the same size. To be fair, I still default to llama3.1 because the ecosystem of examples around it is bigger.

The LangChain part

Now the part that trips people up: LangChain feels needlessly complicated until you stop fighting its abstractions. The modern way is to use the langchain-ollama package, which gives you a clean ChatOllama wrapper.

Create a virtual environment and install:

python -m venv .venv
source .venv/bin/activate
pip install langchain langchain-community langchain-ollama

Then the smallest useful script:

from langchain_ollama import ChatOllama

llm = ChatOllama(model="llama3.1:8b", temperature=0.1)
response = llm.invoke("Write a Python function that reads a CSV and prints the first 5 lines.")
print(response.content)

Run it. If you get a connection error, Ollama is probably not running as a server. Start it with ollama serve in another terminal, or just run any ollama command to trigger the daemon.

Making it actually useful: a small extraction workflow

A raw call is fine, but chains make things repeatable. Here's a chain that takes a messy block of text and returns JSON with a company, a product, and a price.

from langchain.prompts import ChatPromptTemplate
from langchain.schema import StrOutputParser
from langchain_ollama import ChatOllama

prompt = ChatPromptTemplate.from_template("""
Extract the company name, product name, and price from text.
Return your answer as plain JSON with keys: company, product, price.

Text: {text}
""")

llm = ChatOllama(model="llama3.1:8b", temperature=0)
chain = prompt | llm | StrOutputParser()

out = chain.invoke({"text": "Acme Corp launched WidgetX at $99 per unit last Tuesday."})
print(out)

The pipe operator (|) composes the steps. That's the whole trick. Prompt template goes in, LLM processes, string parser cleans it up.

Here's the bug I hit last Tuesday afternoon: the output came back as “Here is the extracted JSON for you: {...}. Let me know if you need anything else.” The parser doesn't magically strip that. My fix, which is ugly but works:

import json, re

match = re.search(r"\{.*\}", out, re.DOTALL)
data = json.loads(match.group(0)) if match else {}
print(data)

Yes, regex. It's not elegant, but a local model's commentary around structured output is so consistent that the regex rarely misses. If you use qwen2.5:7b, add an empty <|im_end|> handling too — that model loves its token.

If you want to build a more sophisticated pipeline around this, the Workflows section in the PromptCube community has real-world examples with step-by-step debugging notes. My chain here is intentionally minimal so you can see the bones.

Retrieval-augmented generation without the cloud

RAG is the killer feature of local LLMs. You stuff your docs into a vector store, then make the model answer only from that context. The cloud vendors charge per token for this; here it's just CPU cycles.

Pull an embedding model first:

ollama pull nomic-embed-text

Now the code. It involves loading a document, splitting it, embedding with Ollama, storing in FAISS, and wiring a retriever into the chain:

from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import OllamaEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.prompts import ChatPromptTemplate
from langchain.schema import StrOutputParser
from langchain_ollama import ChatOllama

# 1. load a text file
with open("docs/incident_report.md") as f:
    content = f.read()

# 2. split into chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
docs = splitter.create_documents([content])

# 3. embed and store locally
embeddings = OllamaEmbeddings(model="nomic-embed-text")
db = FAISS.from_documents(docs, embeddings)

# 4. build the QA chain
prompt = ChatPromptTemplate.from_template("""
Answer only from the provided context: {context}

Question: {question}
""")
llm = ChatOllama(model="llama3.1:8b")
retriever = db.as_retriever()

from langchain_core.runnables import RunnablePassthrough

chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

print(chain.invoke("When was the incident resolved?"))

The RunnablePassthrough() line is not magic. It just passes the original question downstream while the retriever contributes the context. It took me a while to get comfortable with this pattern because the original LangChain docs show a totally different memory-heavy API that's now deprecated. Ignore anything older than about eight months.

Where the wheels fall off

Let's be honest about the pain points, because you will hit them.

• Speed. My M1 MacBook runs llama3.1:8b at roughly 12.3 tokens per second. That's readable, barely. A long conversation with lots of context can take 20 seconds just to produce the first word. If your workflow needs interactive brainstorming, local is not it.
• Context loading. Smaller models choke if you stuff them with 8,000 tokens. The answer quality degrades incoherently, not gradually. I now keep a hard cap of about 2,000 tokens of context per query.
• Whitespace and JSON formatting. As mentioned, the regex fix. Also, sometimes the model refuses to include a trailing } because the tokenizer ate it. When that happens, append a closing brace yourself.

I also had a weird Windows WSL2 issue where Ollama would time out after 30 seconds of idle. Restarting the service always fixed it. Not great, but fine.

If you're chasing help with these exact failures, the shared command logs and model comparison notes in the community Resources have saved me at least a day of digging. It's genuinely one of the few places where people post the failed attempt alongside the fix.

Even after all this, do I still use the cloud?

Constantly. For really hard code generation, refactoring at scale, or untangling a gnarly error trace, I open Claude or GPT-4. But for a repeating pipeline that runs against private docs, or a test runner that needs to analyze logs all night long, I want the local chain. No API key, no cost per token, no surprise bill at the end of the month.

The 12.3 tokens per second is slow. The zero on the invoice is not.