Homebench: Local LLM Benchmark Results Across Speed, Memory

After running Homebench on a handful of popular local models over the past week, I have a few takeaways that might save you some time if you're trying to pick the right one. Spoiler: there's no universal winner, and the trade-offs are sharper than I expected.

What I tested

I stuck to models that run comfortably on a 16GB VRAM card, since that's the most common setup I see people asking about:

• LLaMA 3 8B (GGUF Q4_K_M)
• Qwen 2.5 7B (GGUF Q4_K_M)
• Mistral Nemo 12B (GGUF Q4_K_M)
• Gemma 2 9B (GGUF Q4_K_M)
• DeepSeek 7B (GGUF Q4_K_M)

All runs were on an RTX 3060 12GB, Ubuntu 22.04, using llama.cpp with CUDA offload. Homebench's three pillars it measures are throughput (tokens/sec), peak memory usage, and an LLM-judged quality score on a fixed prompt set.

The numbers

• Throughput (tokens/sec): Qwen 2.5 7B leads at ~58 t/s, followed by Gemma 2 9B at ~49 t/s. LLaMA 3 8B sits at ~42 t/s, which surprised me — it felt faster in practice but the benchmark doesn't lie. Mistral Nemo drags at ~28 t/s, likely because it's the only 12B model here and the memory bandwidth gets saturated.
• Peak memory: Gemma 2 9B uses the least at 6.1GB, Qwen 2.5 7B at 6.8GB. LLaMA 3 8B peaks at 7.4GB. Mistral Nemo 12B eats 9.2GB, and DeepSeek 7B hits 8.1GB — odd, since it's smaller on paper.
• Quality score: LLaMA 3 8B wins here, scoring 8.2/10. Qwen 2.5 7B is close behind at 7.9/10. Gemma 2 9B drops to 7.1/10, DeepSeek 7B to 6.8/10, and Mistral Nemo 12B surprisingly comes in last at 6.5/10 despite being the biggest.

What this tells me

If you're chasing raw speed and your workload is mostly short responses or code completion, Qwen 2.5 7B is the sweet spot. It's fast, light on memory, and the quality dip from LLaMA 3 is small enough that you probably won't notice in most contexts.

If you care about getting the best possible answer from a 7-9B model and can tolerate a ~30% speed hit, LLaMA 3 8B is still the benchmark to beat. The quality gap is real, especially on reasoning and multi-turn prompts.

Gemma 2 9B is interesting — it's the most memory-efficient and nearly as fast as Qwen, but the quality score makes me hesitant unless you're severely VRAM-constrained.

Mistral Nemo 12B is the disappointment. It's bigger, slower, hungrier for memory, and scored lowest on quality. I'm not sure what niche it's carving out, honestly.

A practical note on Homebench itself

The tool is straightforward — install via pip, point it at your model, and it runs through its preset suite. What I appreciate is that it doesn't try to hide the hardware context. It reports your GPU model, driver version, and llama.cpp commit, so you can actually compare apples to apples across different machines. Fair warning: the quality scoring uses a reference LLM (currently GPT-4o-mini via API), so you need an OpenAI key handy. That's fine for me, but worth flagging if you're going fully offline.

If you're tuning your local LLM setup for a specific use case, I'd recommend running Homebench with your own prompt templates rather than relying on the defaults. The stock prompts favor general-chat scenarios, which may not reflect your actual workload.

pip install homebench
homebench run --model ./models/qwen2.5-7b-q4km.gguf --prompts custom_prompts.json

The custom_prompts.json format is just a list of strings — simple enough to script up your own test set from real user queries.