Rethinking LLM scaling after Jie Tang's latest breakdown

Been digging through Jie Tang's recent talks on scaling laws and the Z.ai approach, and honestly it forced me to reconsider a few assumptions I've held since the Chinchilla paper dropped. The guy's been thinking about this from both the academic side at Tsinghua and the trenches at Z.ai — that dual lens shows.

Most scaling discussions treat compute as a single knob: throw more FLOPs at the problem, get predictable returns. Tang's framing separates the training compute budget from the inference reality in a way that feels obvious once you hear it but rarely gets modeled explicitly. His team's argument: if you're optimizing for a fixed inference budget (which every real deployment is), the optimal training run looks different than what pure loss curves suggest.

The practical takeaway that stuck: they're advocating for smaller, deeper architectures trained longer rather than the wider/shallower trend we've seen since GPT-3. Their internal ablation shows a 7B model trained on 3T tokens beating a 13B model on 1.5T tokens across their target benchmarks — coding, reasoning, Chinese NLP. The depth helps with the multi-step reasoning chains that benchmarks like GSM8K and HumanEval actually reward.

# Rough architecture comparison they reference
# Wide-shallow (traditional scaling)
wide_config = {
    "n_layers": 32,
    "d_model": 5120,
    "n_heads": 40,
    "params": "~13B"
}

# Narrow-deep (Z.ai preference)
deep_config = {
    "n_layers": 48,
    "d_model": 4096,
    "n_heads": 32,
    "params": "~7B"
}

The inference speed difference is non-trivial. At batch size 1, the 7B deep model runs ~2.3x faster on H100s because the smaller hidden dimension means less memory bandwidth pressure per token. That compounds when you're serving thousands of concurrent requests.

What's interesting is how this interacts with mixture-of-experts. Tang mentioned they experimented with MoE but found the routing overhead ate most gains at their scale — the expert specialization only pays off past ~30B active params. For sub-10B models, dense still wins. That contradicts a lot of the current open-source MoE hype.

Data curriculum matters more than people admit. Their pipeline does three passes: deduplication → quality filtering (they use a small classifier trained on human annotations) → domain reweighting (code/math/reasoning upweighted 3x). The reweighting alone bought them ~0.8% on MMLU compared to uniform sampling. Not huge, but it's free compute.

One detail that surprised me: they're not chasing context length aggressively. 32k is their ceiling for now. The argument: long-context attention scales quadratically, and their target workloads (coding agents, RAG) rarely need more than 8-16k effective context once you account for retrieval. They'd rather spend that compute on deeper reasoning at standard lengths.

If you're building a practical LLM system today — not a research artifact — the Z.ai playbook is worth studying. Small dense model, deep architecture, long training run, careful data curriculum, optimize for your actual inference constraints. The math works out.