Local LLM Selection for Mac Mini M4 Pro 24GB

24GB of unified memory on the M4 Pro is a weird middle ground—too much for tiny models but not quite enough to run a Llama 3.1 70B comfortably without heavy quantization that kills the intelligence. I'm currently building a few apps that need to run locally on user hardware, and I'm hitting a wall with the current model options.

The trade-off between instruction following and raw reasoning is killing me. I tried Gemma 2 (which is fast and handles reasoning well), but the tool calling is hit-or-miss. Then I shifted to Qwen 2.5 7B/14B (MLX versions), and while the tool calling is actually reliable, the generic reasoning feels a step down from Gemma.

If you're targeting a 24GB footprint, you have to account for the OS overhead and the KV cache. You're effectively looking at about 16-18GB of usable VRAM for the model.

The Model Dilemma

For a real-world AI workflow where you need both reasoning and strict tool adherence, I've found that Mistral NeMo 12B is usually the sweet spot for this specific hardware. It fits comfortably in memory and doesn't suffer from the "lobotomy" effect that happens when you squeeze a larger model into 4-bit quantization.

If you're still struggling with tool calling, I'd suggest tweaking the system prompt to be extremely explicit about the JSON schema. Instead of relying on the model's native tool-calling capability, force a structured output.

The Ollama Reranker Crash

The bigger headache right now is the reranker. I've been trying to integrate a reranking step into my RAG pipeline using Ollama, but qllama/bge-reranker-v2-m3:Q4_K_M keeps crashing my instance.

The error usually looks something like this in the logs:

# Typical Ollama crash signal during embedding/reranking
sigkill: process terminated by signal 9 (out of memory)

Since rerankers are cross-encoders, they process pairs of documents, which can spike memory usage unexpectedly. On a 24GB Mac, if you already have a 12B-14B model loaded in the GPU, loading a reranker on top of it often pushes the system over the edge, triggering the macOS OOM killer.

To fix this, I've had to implement a strict "unload" logic or use a smaller embedding model for initial retrieval and skip the reranker until I can find a more stable implementation.

Current Config Attempt

Here is the basic setup I'm testing to see if I can stabilize the memory:

# My current local deployment attempt
model: "mistral-nemo"
parameters:
  num_ctx: 8192
  num_gpu: 1
  temperature: 0.2
  top_p: 0.9
# Trying to keep ctx low to leave room for the reranker

If anyone has a stable reranking setup on M4 Pro that doesn't nukes the Ollama process, I'm all ears. I need something that can handle the bge-reranker logic without eating all 24GB of unified memory.