AI Smuggles a Bug into Lean 4 While 'Proving' Collatz — Wait

I’ve been watching the “AI stumbles into a Lean 4 bug while pretending to prove Collatz” story float around, and it’s the funniest thing to come out of a theorem prover all year.

So someone (I think it was one of the groups playing with LLM + proof assistant pipelines) fed GPT-4 the Collatz conjecture and asked it to produce a Lean 4 proof. The model churned out something that looked plausible to a human eye, but when the machine tried to verify it, Lean 4 crashed. Not a type error — a full-on internal bug. Something in the kernel was triggered by the specific structure of the generated code. The AI accidentally uncovered a defect in the prover itself.

That's like asking a student to prove Fermat's Last Theorem and they set the chalkboard on fire instead.

People are rightfully focused on “AI finds bugs in code” but here the twist is that the bug was in the verifier, not the proof. Lean 4 is supposed to be the incorruptible judge, and it got reduced to a segfault by a half-baked attempt at a problem that’s famously impossible.

A few thoughts:

• Is this a testament to how weird LLM outputs are? These models produce code that statistically mimics correct syntax but often violates deeper invariants. Here, the generated proof somehow violated an invariant inside Lean’s kernel, causing an unexpected state. That’s wild — it means the model accidentally explored a corner case the Lean developers never tested.

• Or is it just evidence that Lean 4 is still maturing? The prover is relatively new, bugs happen. But the fact that an AI generated the triggering input by accident is a sobering reminder that our tools are more fragile than we believe.

• Practical takeaway: If you’re building any kind of AI + formal verification pipeline, treat the verifier as a black box that can itself fail. Don’t assume that because Lean accepts a proof, the proof is sound — at least until you’re sure the prover isn’t the next Collatz victim.

The whole thing feels like a meta joke: an intractable conjecture remains unproven, but the attempt to fake a proof broke the proof checker. Maybe the real lesson is that AI’s talent for finding edge cases in human-built systems is only going to grow.

Anyway, I’ll be over here watching my Lean 4 terminal for any signs of sentient bugs.