Testing AI comprehension vs. pattern matching
I've been trying to build a way to actually stress-test this—to see if an agent understands the "where the bodies are buried" aspect of a repo before you let it touch a load-bearing model.
I developed a "change-impact teardown audit." The logic is simple but brutal: you pick a central hub model—the kind of file that half the system depends on—and you give the agent a specific maintainer task:
Before you change how this model is torn down, find every place in the codebase that depends on it.
This isn't a grep task. To pass, the agent has to trace polymorphic associations, workers that load objects by ID deep in a call stack, or config-string registries. It has to understand the edges, not just the center.
To keep it from being a joke, I implemented three strict rules for my test:
1. I hand-build the answer key. I manually map every dependent to an exact file:line. The agent never sees this.
2. I grade against the key, not the prose. If the agent says "it's used in the worker module" without the exact line citation, it gets zero credit. I learned this the hard way when a reference-blind judge called a 44% accurate audit "exhaustive."
3. I run every prompt twice. Inference is non-deterministic; I've seen one repo return 2 hits then 0 hits on the exact same prompt.
I ran this against thirteen Ruby codebases. When I provided a structural map for the agent to query, the accuracy on large apps jumped significantly, while small, colocated repos stayed flat. This suggests the model is actually tracking structure, not just guessing based on repo size.
But I'm skeptical of my own framework. It's heavily biased toward static call-and-dependency edges. It would likely fail miserably against a heavily event-driven system using a pub-sub bus or dynamic dispatch where the wiring only exists at runtime.
If you're building agents for complex, asynchronous architectures, how are you actually verifying they aren't just hallucinating a mental map of your dependencies?