Testing LLMs against seL4: Can AI break formally proven code?

Formal verification is the "gold standard" of software reliability, and seL4 is the most famous example of a microkernel that has been mathematically proven to be bug-free relative to its specification. The logical question is whether modern LLMs, which are surprisingly adept at spotting edge-case security flaws in traditional C code, can find a hole in something that is mathematically "proven."

In a perfect world, the AI should find absolutely nothing because the proof covers the implementation. However, there is a massive gap between a mathematical proof and the actual execution of code on physical hardware. If you're planning a deep dive into this, here are the specific angles where an LLM agent might actually find something useful:

The Specification Gap

Formal verification doesn't prove a system is "perfect"; it proves the code matches the specification. If the specification itself has a logical flaw or misses a real-world hardware quirk, the proof is valid, but the system is still vulnerable. An AI trained on thousands of CVEs might recognize a pattern—like a specific race condition or a side-channel leak—that the original formal specification simply didn't account for.

Compiler and Hardware Mismatches

The seL4 proof typically targets a specific model of the CPU. But we aren't running code on a mathematical model; we're running it on silicon. There are three layers where things break:
1. The C code is verified.
2. The compiler transforms it into binary.
3. The binary runs on a CPU with branch predictors and speculative execution.

AI is particularly good at identifying "weird" hardware behaviors. If you feed an LLM the assembly output of a verified kernel and ask it to look for speculative execution vulnerabilities (like Spectre-style leaks), it might spot a sequence that the formal proof ignored because the proof assumed the CPU executes instructions linearly.

Practical Testing Workflow

If anyone wants to try this as a real-world experiment, don't just paste a file into a chat window. To get actual results, you need a structured AI workflow:

1. Context Loading: Feed the AI the formal specification documents first so it understands the "rules" the kernel is supposed to follow.
2. Differential Analysis: Provide the verified C source and the compiled binary. Ask the AI to find discrepancies where the compiler might have introduced an optimization that violates the intended security property.
3. Targeted Prompting: Instead of asking "find a bug," use specific prompt engineering to target known microkernel weaknesses:

Analyze the following memory management routine for potential TOCTOU (Time-of-Check to Time-of-Use) vulnerabilities, specifically focusing on how the hardware cache might interact with the verified state transitions.

Ultimately, using AI on proven software isn't about proving the mathematicians wrong; it's about finding the "seams" where the math ends and the physical hardware begins. That's where the real security flaws usually hide.