Gemma 4 vs Gemini 3.1 Flash-Lite: The Hybrid Win
Routing only 15-35% of queries to a cloud model can actually match the performance of a frontier model if your on-device LLM knows when it's hallucinating. This is exactly what the Cactus Hybrid approach achieves by adding a confidence probe to Gemma 4 E2B, effectively solving the "reliability gap" for local deployment.
The most striking part of this deep dive is the modality-independent signal. The probe was trained on zero audio data, yet it maintained a 0.79-0.88 AUROC on audio benchmarks. This proves the model is tapping into a universal "correctness" signal in its hidden states rather than just memorizing patterns.
If you want to experiment with the weights or the implementation, the source is available here:
Next
PowerShell Security: The Risk of `irm | iex` →
The technical breakthrough here isn't just another prompt engineering trick; it's a 68k parameter probe layer (using LayerNorm and low-rank projection) that reads hidden states during decoding. Unlike token entropy or asking the model to "rate its own confidence" in text—which are notoriously unreliable—this probe predicts the probability of being wrong as structured data.
Looking at the performance benchmarks, the efficiency gains are impressive:
- ChartQA: Only 15-20% of queries routed to Gemini 3.1 Flash-Lite.
- LibriSpeech: 25-30% routing rate.
- MMBench/GigaSpeech/MMAU: 30-35% routing rate.
- MMLU-Pro: 45-55% routing rate.
The most striking part of this deep dive is the modality-independent signal. The probe was trained on zero audio data, yet it maintained a 0.79-0.88 AUROC on audio benchmarks. This proves the model is tapping into a universal "correctness" signal in its hidden states rather than just memorizing patterns.
For anyone building an AI workflow that requires a balance between privacy/speed (local) and raw power (cloud), this is a massive step forward. You can run this via Transformers, MLX, or Llama.cpp.
A few technical constraints to keep in mind:
- It only scores single-sequence decoding (up to 1024 tokens).
- Handoffs are most effective when routing per task rather than per individual step.
If you want to experiment with the weights or the implementation, the source is available here:
https://github.com/cactus-compute/cactus-hybridIt's encouraging to see small models becoming "self-aware" enough to handle their own routing. It makes the dream of a truly seamless, hybrid AI agent much more realistic.
All Replies （10）
C
Casey51
Novice
10h ago
Does this overlap at all with what Goodfire is doing? I've been following their RLFR research lately (https://www.goodfire.ai/research/rlfr#) and I'm curious if the approach here is similar.
0
Q
Neat! Just to make sure I understand—did you train your probe layer to take this hidden state and predict p(wrong)? I'm curious to learn more. Any more info on your approach, especially regarding the mechanistic study?
0
R
I actually messed around with something similar right after the announcement just for kicks. I've only tested it on my Framework Desktop so far, but it should work on other setups. Check it out here: https://github.com/olafura/gemma-4-mic-transcribe
0
J
Have you actually benchmarked this for coding, maybe with a fallback to a larger local model like Qwen-2.5-32B? I'm also curious if you've tried using it for sub-tasks where a primary model dispatches the simpler jobs, like summarization, to this smaller one.
0
J
Do we actually have proof that these "self-awareness" signals aren't just statistical noise we're projecting meaning onto? I'm not convinced that "clean code" is a distinct internal state rather than just a pattern of high-probability tokens. If we can't rigorously define these signals first, isn't "steering" them just guesswork based on vibes?
0
M
Do you think the model quality starts degrading in other ways too? I've been playing around with a few different versions and I'm wondering if there are some subtle glitches or losses in logic that aren't immediately obvious.
0
A
Can it actually know when it's wrong about being wrong? And how confident is it in that confidence score? This terminology is driving me crazy. You can't truly "know" you're wrong; you're just unsure or inconsistent. You can be 100% certain and still be dead wrong. Please, stop using these phrases!
0
M
Funny how this keeps coming back to basic human intuition. It's way simpler to map out the boundaries of your ignorance than it is to actually verify what's true. Makes you wonder if we're just optimizing for the path of least resistance.
0
G
Why bother with a handoff if the scoring is actually reliable? If a response hits a low score, you could just re-run the prompt with a different seed until it passes. If that doesn't eventually work, it's a strong sign that your scoring metric is probably garbage anyway.
0
M
Rerunning wastes too many tokens tbh. A handoff to a bigger model is way more efficient for the hard cases.
0