Debugging AI: Fixed or Just Hidden?

"The bug is fixed" is the most dangerous phrase in an AI-assisted workflow. Usually, it just means the symptom vanished, but the root cause is still there—or worse, the AI just wrapped the whole thing in a try-catch block and silenced the error.

I've stopped treating AI bug fixes as conclusions and started treating them as claims that require evidence. If you're doing a deep dive into your own code, the rule is simple: a bug is only fixed when the failure is gone, the intended behavior works, and you haven't accidentally nuked a neighboring feature.

Documenting the Failure First

Stop telling the AI "saving is broken." That's useless. Before you touch a single line of code, you need a reproducible failure path. I use this format to keep the LLM honest:

Starting state:
[what must already be true]

Actions:
1. [action]
2. [action]
3. [action]

Expected result:
[what a user should see or be able to do]

Actual result:
[what happens instead]

Evidence:
[error text, screenshot, console output, or saved record]

For instance, instead of "it doesn't save," use:
"Starting state: Signed in with one saved task. Actions: Edit title, press Save, refresh. Expected: New title stays. Actual: Says 'Saved' but old title returns. Evidence: Network request 200, but DB record is unchanged."

The Four-Point Verification Process

Once the AI gives you a patch, don't just merge it. I run these four checks to make sure it's a real fix and not a mask.

1. The Original Repro Must Pass
Run the exact steps from your documentation. No paraphrasing. If you suddenly "can't reproduce it," that's a red flag. "Cannot reproduce" is not the same as "fixed."

2. Verify the Underlying State
UI is a liar. A "Saved!" toast notification doesn't mean the data hit the disk.

• Refresh the page.
• Hard restart the app.
• Check the DB dashboard.

If the state didn't actually persist where it belongs, the AI just polished the surface.

3. Ensure "Honest" Failures
A common AI move is to make every input return a success message. Try an invalid input or a timeout. If the AI turned a specific validation error into a generic "Success" or a blank screen, it didn't fix the bug—it just hid the error message.

4. Check for Side-Effect Regressions
This is the classic "fixed the login, broke the password reset" scenario. Check the features immediately adjacent to the fix.

For anyone building an AI workflow from scratch, the goal isn't to make the AI sound confident; it's to make your proof requests specific.