Wilson 95% lower bounds on rule survival across 29

So I finally sat down with this paper properly and it's one of those rare stats-meets-LegalTech things where the method is the point, not the headline. The headline is "machines disagree about statutes" — boring. The interesting part is the certificate they built around that disagreement.

The setup: two independent extractors parsed Missouri statutes, and on numeric-threshold presence they hit a 0.43 false-negative rate. Almost half the time one misses what the other catches. That number alone should make anyone running RAG over legal corpora pause.

What they actually built:

1. Take the Duquenne-Guigues implication basis of the extracted contexts (for anyone who's dealt with FCA, this is the minimal non-redundant rule set).
2. Measure per-attribute disagreement between extractors.
3. Replay that disagreement against the basis in 1,000 Monte Carlo trials.
4. An implication only "survives" if a one-sided Wilson 95% lower bound on its survival rate hits 0.95.

Each surviving rule ships with its premise spans and a minimal counterexample. That's the audit trail. That's the part I like — you can actually point at a rule and say "here's the exact statutory text, and here's the smallest thing that would break it."

The empirical results are messy in a useful way:

• 29,365 Missouri sections + 502 Indian central-Act sections
• Preregistered held-out gate: 10 statute families across 7 Titles exact, 16 across 11 with 5% tolerance
• Under one globally deployed error model, 93.2% of held-out chapters fall below the informativeness floor
• 2x2 factorial pins that collapse on calibration-rate transfer, not selection

Translation: the rules themselves are fine. The extraction error doesn't transfer cleanly across chapters, so a globally-tuned error model quietly kills your rules. If you calibrate per-chapter, or accept some error tolerance, the certificate survives. Otherwise you're shipping dead logic.

There's also a retracted claim in the released audit trail, which I'm pointing out because more authors should do this.

# The Wilson 95% one-sided lower bound (the core gate)
def wilson_lower(k, n, z=1.6448536269514722):
    p = k / n
    denom = 1 + z*z/n
    centre = p + z*z/(2*n)
    adj = z * ((p*(1-p) + z*z/(4*n))/n) ** 0.5
    return (centre - adj) / denom

Two takeaways I'm carrying forward into my own LLM-extraction work:

• Don't trust a single extractor's output without a survival check. Even at 0.43 FNR, rules with premise+counterexample chains can still certify — but only if you measure per-attribute noise, not aggregate accuracy.
• Global error models are a trap for legal/scientific text. Domain drift between chapters is real, and a single threshold turns 93% of your signal into noise.

If you're doing prompt engineering for legal RAG, structured extraction, or building LLM agents over regulatory corpora, the per-chapter calibration point is the one to internalize. The Duquenne-Guigues machinery is overkill for most use cases, but the discipline — measure disagreement, bound survival, ship counterexamples — isn't.

Has anyone else been hitting this calibration-transfer wall with extraction pipelines? I'm curious whether the 93.2% collapse rate holds up outside US statutes.