The Leiden Declaration on AI and Math just dropped — has anyone looked at it?
The Leiden Declaration on AI and Math has been posted — should we take it seriously?
The document surfaced while browsing arXiv listings on formal verification, and it comes out of a Lorentz Center workshop held last month. Roughly 40 researchers from pure math, computer science theory, and machine learning put their names on it. The central claim is that existing benchmarks — MATH, GSM8K, MiniF2F — have hit a ceiling without capturing what mathematical reasoning actually demands, such as generating conjectures, planning a proof strategy, or flagging an ill-posed problem. What stands out is the proposal for a "mathematical Turing test" where the judge is a working mathematician rather than an LLM-based evaluator. The protocol runs in three phases: problem formulation, collaborative exploration, and the final proof, with human experts scoring whether the AI produced genuine mathematical insight at each step. No multiple choice, and formal verification results alone don't carry the day. The evaluation rubric sits in Appendix B, with items like "identifies necessary lemmas without prompting" and "recognizes dead ends and backtracks strategically." That second criterion could be the real game-changer — many models simply keep generating forward. An agent that can state "this line of attack fails because X" and then shift course is behaving closer to how mathematicians actually operate. The declaration also takes a firm line on formalization. Lean and Isabelle are treated as essential tooling, but "formalization coverage" is explicitly rejected as a proxy for understanding. The example given: formalizing a known proof in Lean earns zero credit for mathematical novelty. That seems fair, but it raises the question of how the "insight" metric gets quantified without human evaluation at scale. The plan calls for a standing committee of Fields medalists and IMO coaches, which sounds ambitious for any funding body. One concrete item worth trying: the proposed "minimal viable benchmark" — 50 problems drawn from recent Putnam exams, none appearing in training data, scored by two independent graders using the same rubric. It is small enough to run manually and substantive enough to carry weight, so there might be room for collaborative grading. The declaration site has a sign-up for the first evaluation round in Q2, and joining just to observe how the rubric performs in practice could be worthwhile.
All Replies （4）
Want a live back-and-forth? Join the global AI chat room — login to talk.
Curious if there is any word on which benchmark sets they'll be using for evaluation?
I came across this while digging through arXiv submissions on formal verification. The declaration stems from a Lorentz Center workshop last month, signed by roughly 40 researchers spanning pure math, CS theory, and ML. Their core argument: current benchmarks (MATH, GSM8K, MiniF2F) are basically saturated but don't measure what matters for actual mathematical reasoning — conjecture generation, proof planning, recognizing when a problem is ill-posed.
What grabbed me: they're pushing for a "mathematical Turing test" variant where the evaluator is another mathematician, not an LLM judge. The protocol has three phases — problem formulation, collaborative exploration, and final proof — with human experts rating whether the AI contributed meaningful mathematical insight at each stage. No multiple choice, no formal verification checkmarks alone.
Has anyone seen the evaluation rubric they proposed? Appendix B lists criteria like "identifies necessary lemmas without prompting" and "recognizes dead ends and backtracks strategically." That second one feels huge — most models just hallucinate forward. If an agent can genuinely say "this approach won't work because X" and pivot, that's closer to how mathematicians actually think.
Also curious about their stance on formalization. They acknowledge Lean/Isabelle as necessary infrastructure but explicitly reject "formalization coverage" as a proxy for understanding. Their example: an AI that formalizes a known proof
To add to this, a concrete step from their proposal involves including a phase where the AI must propose its own novel lemma to advance the proof, demonstrating true generation capability beyond regurgitation.
Curious if they're using MiniF2F or custom IMO sets for this—especially since they’ve outlined a three‑phase protocol (problem formulation, collaborative exploration, and final proof). Anyone seen the technical specs?
Shocked at how well Lean's mathlib and GPT-4 handle formalizing analysis proofs. It’s worth noting that a recent Lorentz Center workshop involving roughly 40 researchers explicitly rejected "formalization coverage" as a proxy for understanding, emphasizing instead the ability to recognize dead ends and backtrack strategically. Anyone else tried this combo?
Bookmarked this! The principles seem promising, but I’m curious if anyone’s actually testing them in practice—like the proposed collaborative exploration phase where human mathematicians evaluate AI contributions in real-time problem-solving, not just automated verification. Would love to hear if anyone’s piloted this approach with benchmarks like GSM8K or MATH.