Maple-Preview: 120 tok/s 20B MoE on iPhone Defies Expectations

The headline looks like vaporware until you realize it's running a ternary 20B parameter Mixture-of-Experts model — most of those weights are literally baked into 2-bit values, which is why an iPhone can chew through 120 tokens per second without melting.

What's Real Here

Maple-Preview isn't some benchmark stunt. It's a trimmed-down demo of what the full Maple stack aims to ship: a 20B MoE transformer quantized down to ternary activations and weights, then compiled with a custom backend that leans hard on Apple's Neural Engine. The trick is that only a fraction of experts fire per token, and the ternary representation cuts memory bandwidth demands so drastically that the ANE doesn't become the bottleneck.

The Caveats (Because I'm Skeptical Too)

The model is, unsurprisingly, dumb as a rock compared to its dense counterparts. You're trading accuracy for sheer speed — think basic instruction following, not nuanced reasoning. The demo also caps sequence length aggressively, so it's fine for quick prompts but falls apart on anything requiring long context. This is very much a proof-of-concept: here's the raw performance, now go build something useful on top.

Setup Notes for Hackers

The repo ships with a Swift package wrapper around a quantized Core ML model. Running it locally requires:

# iOS 17+ required for full ANE support
xcodebuild -project MaplePreview.xcodeproj -scheme MaplePreview -destination 'platform=iOS Simulator,name=iPhone 15'

Memory footprint sits around 400MB once loaded, which is honestly impressive given the parameter count. The tokenizer is a stripped BPE variant, and the MoE routing is handled by a lightweight learned gate that runs entirely on-device.

Why This Matters

Most "LLM on mobile" demos either punt to the cloud or squeeze a tiny 1B model onto device. Maple-Preview flips that script — it keeps the parameter count high but slashes the computational cost through quantization and expert sparsity. If the full release holds up, we might finally get local LLMs that feel fast instead of tolerating sluggish inference.

The real test will be whether the accuracy cliff is survivable for niche use cases like coding assistants or local knowledge bases. For now, the 120 tok/s number is legit, and that alone makes this worth a look.