Publicly auditing my OSS release proved that your documentation

Then a stranger decided to audit my claims in public.

He didn't just offer feedback; he proposed a rigorous protocol. We froze five specific claims from the release before he inspected them. The rule was simple: verify the claims using only public artifacts. If the evidence didn't match the claim, it was recorded as a divergence—no excuses, no "explaining it away" based on what I knew happened in my local environment.

This was a brutal reality check on how we handle AI workflow documentation and LLM agent testing. Most of us release software based on a mix of CI logs, local memory, and general confidence. We know the code works, so we assume our summaries of its performance are also correct. This audit proved that assumption is a trap.

The breakdown of the audit

We froze five claims. Two were clean. Three were blatant contradictions.

• The Test Count Discrepancy: I claimed 1295 deterministic tests passed. The public CI run actually showed 1294 passed, 1 failed (a flaky SQLite concurrency issue), and 14 skipped. While the failure didn't break the engine, my documentation was mathematically incorrect compared to the public record.
• The Internal Contradiction: This was the most embarrassing one. In my field-test report, I explicitly wrote "zero true failures." However, just two tables down in Scorecard B, I had recorded True Fail = 1. I had published two incompatible truths in the same document and never noticed.
• The Metric Mismatch: I stated that plan_oscillation_detected fired for 3 goals. The actual stored artifacts showed it actually fired for 5.

Why this matters for prompt engineering and agent evaluation

The most uncomfortable takeaway isn't that the software failed—the engine actually performed beautifully. The live boundary evaluator returned perfect rates for label_flip_rate and evidence_drift_rate. The underlying logic of the PlannerCritic was robust.

The failure was in the meta-layer: the bridge between the technical execution and the human-readable report.

When we are building complex LLM agents or fine-tuning prompt engineering workflows, we often rely on high-level summaries to communicate success. If your evaluation pipeline produces a metric, but your documentation manual-entry process "rounds" that metric or misinterprets a flag, you lose technical credibility.

In a real-world deployment, if your agent's performance metrics are even slightly decoupled from the actual logs, you cannot perform a reliable post-mortem when things go wrong. This experience taught me that an AI workflow is only as reliable as its most visible documentation. If you can't reconstruct your claims from public artifacts, you haven't actually proven anything.