Qwen 27B with 128k Context on 24GB VRAM?

Running a 27B model with a 128k context window on a single 24GB card sounds like a recipe for an OOM (Out of Memory) crash, but I'm seeing reports that it's actually working. The key seems to be a specific combo of Ollama environment variables that drastically cuts down the memory footprint of the KV cache.

If you're trying to squeeze long-context LLM agents into limited VRAM, you need to look at these two flags:

OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q4_0

The q4_0 KV cache quantization is the real heavy lifter here. Instead of storing the cache in FP16, it compresses it, which is why someone is reporting 83% VRAM usage at 128k context on a 24GB card. Without this, the KV cache for a 27B model at that context length would easily blow past 24GB on its own, regardless of the model quantization.

The performance hit is surprisingly minimal too—hitting around 30 tok/s. If it spills over into system RAM, you're looking at a drop to maybe 4 tok/s, which makes the local deployment basically unusable for real-time work.

A few observations from this setup:

• Q4 Quantization: 128k context fits comfortably.
• Q5 Quantization: Hits a wall around 96k context.

It's a decent deep dive into how much we can optimize the AI workflow by tweaking the backend rather than just hoping for a smaller model. If you've been capping your context at 32k because of VRAM limits, definitely test these flags.