Logic Drift in Agentic Memory Management

I was sitting there staring at my benchmark results, watching an LLM agent pull a verbatim quote from a document that explicitly stated "the policy hasn't changed," only to have the agent immediately inform the user that the policy had indeed changed. It was such a jarring moment because the model wasn't technically hallucinating the text itself—it was hallucinating the logical relationship between that text and the current state of reality. This kind of drift is exactly what broke my initial attempt at a "memory-authority gate." I had been trying to build a system that prevents an agent from blindly overwriting established knowledge just because a new, sounding "official" note comes in, but the logic was too soft.

To make sure I wasn't just tweaking parameters to fit the results I wanted, I actually pre-registered all my predictions, pass/fail thresholds, and specific failure cases in a public repository before I even touched the codebase for Version 2. I wanted to be sure I wasn't moving the goalposts once the data started coming in.

The solution I landed on for the second iteration was almost frustratingly simple. I realized that a quote shouldn't be treated as a logical relationship unless the quote itself explicitly names that relationship. I moved away from purely probabilistic checks and implemented a "relation-span clause" within a deterministic confirmer layer. It essentially forces the model to prove its work using a very specific, rigid set of rules.

The logic for this confirmer layer looks like this:

1. Operator present: The cited sentence must contain a change word from a frozen list: replaced, retired, deprecated, superseded, overridden, discontinued, revoked, "no longer," "instead," "only," "now."
2. The sentence test: At least one sentence inside the cited span must carry BOTH a change word AND a scope term of the rule on trial — in the same sentence.

The system remains incredibly strict: the quote must be verbatim, the scope must match perfectly, and the confidence score must clear a 0.60 threshold. For example, if a changelog says "v2.1 supersedes v2.0," the gate rejects it because it doesn't name the specific rule being modified. But if it says "The old retention rule is replaced: nightly backups are kept for 90 days," it passes.

However, I'm still losing sleep over the "implicit contradiction" problem. If Rule B flatly contradicts Rule A without using any of my "change" keywords, this deterministic gate is completely blind to it. I've had to relegate those cases to a low-trust tier for human review, which isn't ideal for a fully autonomous system.

How are you all approaching those subtle, implicit contradictions in your RAG pipelines? Is there a way to catch those without just adding more heavy-handed deterministic rules, or is that just the "final boss" we all have to face?

If you want to look at the raw data or see the commits where I mapped out my predicted failures, you can find everything here:

https://github.com/example-repo/memory-gate-tests