Needle 2 fits a functional LLM into just 14MB

A 14MB binary that runs a full session in 28MB of RAM is practically unheard of for something that actually handles tool calls and structured extraction. Most "edge AI" talk lately focuses on Macs or high-end PCs, but that ignores the billions of IoT devices and budget phones that don't have dedicated NPUs. Needle 2 targets that gap, delivering 400-1,500 tokens/sec on devices like the Meta Quest 3S and Apple Vision Pro, and still hitting 300-700 tokens/sec on cheap Samsung A-series phones.

The technical magic here is the shift away from standard transformers toward Simple Attention Networks. To put the efficiency in perspective, a standard transformer of similar width spends 164 MFLOPs per token. Even a heavily compressed one spends 87. Needle 2 drops that to 70. When you're building an always-on assistant for a wearable, every MFLOP equals milliwatt-hours of battery drain, so reducing that overhead by 7x to 85x compared to other "small" LLMs is a massive win.

How it handles agentic tasks

The core philosophy is that mapping a messy natural language sentence to a function with typed parameters doesn't actually require massive world knowledge or the ability to write poetry. It's a mapping problem. Because of this, 45 million parameters (at 2-bit compression) are enough to trade wins with models like LFM2.5 230M and Apple's Foundation Model, despite being 5x to 70x smaller.

Beyond simple tool calling, Needle 2 handles structured extraction. If you pass a schema instead of a tool list, it acts as a high-speed extractor. This makes it useful for:

• Text Classification: Using an enum field to categorize input.
• Summarization: Providing a schema to extract only the key fields.
• Device Control: Mapping user intent to specific hardware API calls.

Deployment and Fine-Tuning

Since every hardware ecosystem has a different tool vocabulary, the model is designed to be fine-tuned quickly. Using their Python package, you can adapt the model on a standard Mac or PC in a few hours. They've included an automated data-generation pipeline, so you only need a handful of samples to get it running.

One detail I find particularly smart is the "Cactus Hybrid" confidence score. Every response comes with a learned confidence value. This allows for a tiered AI workflow:

• High Confidence: The local Needle 2 model executes the command instantly.
• Low Confidence: The system escalates the query to a larger cloud model (like DeepSeek-v4-Flash).

This creates a very cost-effective deployment strategy for enterprise apps—keeping the bulk of the compute on the device and only paying for the cloud when the local agent is stumped.

For those wanting to see the math behind the architecture, the research paper is available here:

https://arxiv.org/abs/2607.18363

And the implementation can be found at:

https://github.com/cactus-compute/needle