Zhipu's Mythos benchmark leak suggests GLM-4.

The numbers circulating on Chinese forums this week — supposedly from internal Mythos-tier evaluation runs — put Zhipu's next flagship model roughly between GPT-4o and o1-preview on reasoning-heavy benchmarks. If the leak is even directionally accurate, the gap between domestic Chinese labs and the US frontier just compressed another notch.

What's actually in the screenshots

Three benchmark suites keep appearing: MMLU-Pro (87.2%), GPQA-Diamond (78.5%), and a 128k-context needle-in-haystack retrieval test at 99.1% accuracy. The MMLU-Pro score would edge out GPT-4o's reported 86.8%, while GPQA-Diamond sits just shy of o1-preview's 80%+. Context retrieval at that length with near-perfect recall is the standout — most models still degrade noticeably past 64k.

Code generation numbers are fuzzier. HumanEval+ shows 92.3% pass@1, but the dataset contamination debate makes that hard to trust. More interesting: a leaked internal eval on SWE-bench Verified reportedly hits 38.7% resolve rate, which would place it above Claude 3.5 Sonnet's 35.4% from last month's update.

Architecture rumors worth tracking

Multiple independent sources point to a mixture-of-experts backbone with 1.8T total parameters and 256B active per token. That's denser than DeepSeek-V2's 236B active but with significantly more total capacity. The tokenizer allegedly expanded to 150k vocab with heavy Chinese/English/code balancing — a deliberate move to reduce token overhead on bilingual workloads.

Training compute estimates cluster around 3.2e25 FLOPs, roughly 1.4x GLM-4's budget. Zhipu's been quiet about H100 allocation, but their partnership with QingCloud and recent datacenter builds in Guian New District suggest they've had cluster access since Q1.

Why the Mythos codename matters

Zhipu internally uses Greek mythology codenames for model tiers: ChatGLM was "Olympus", GLM-4 was "Titan", and Mythos signals the generation after. The leap from Titan to Mythos historically correlates with architectural rewrites, not just scaling. If pattern holds, we're looking at a new attention mechanism — possibly the native sparse attention they patented last November — rather than just more layers.

Deployment implications

The 256B active parameter count means single-node H100 (80GB x 8) inference is borderline. You'd need tensor parallelism across 2 nodes for comfortable KV cache headroom at 128k context. That's a meaningful operational shift from GLM-4's single-node deployability. API pricing will likely reflect this — expect input tokens around $2.50/M and output $10/M if they follow current tier ratios.

What's missing from the leak

No multimodal evals. No long-form reasoning traces. No safety/alignment benchmarks. And critically — no timestamp on the evaluation harness. Zhipu updated their eval framework in March; if these numbers are from the old harness, they're not comparable to current frontier model reports.

Still, the consistency across three separate benchmark families makes fabrication less likely. Either Zhipu's internal build from ~6 weeks ago genuinely hits these marks, or someone carefully reverse-engineered plausible numbers from public scaling laws.

Either way, the next 60 days should clarify. Zhipu typically open-weights a distilled version 3-4 months after flagship API launch. If Mythos follows the Titan timeline, we'll see a 32B-ish open model by October — and that's the one that actually matters for local deployment.