My custom memory graph actually beat Memora on the LoCoMo
.md files or basic MCP tools to track state, but flat files are too primitive. If you're just using a claude.md file and you're happy with it, you don't need a complex system. But once your project scales, you realize you need a knowledge graph that can actually handle contradictions and versioning.I started by building a tool called memoryweb to handle my own workflow. It worked well for a solo loop—recording problems and fixes as "stories"—but it fell apart the moment I thought about multi-tenancy. When multiple users are feeding an agent information, you run into a mess: conflicting decisions get written to the same file, stale information lingers, and you lose the "why" behind a specific architectural choice.
To solve this, I built Recordari. Instead of just storing text, it uses typed memories and typed edges. This allows the agent to explicitly label nodes as "contradicts" or "resolved." The server can flag semantically similar memories that might conflict, and the agent then has to adjudicate which one is correct. It also tracks ownership, so an agent doesn't just blindly overwrite a human's decision.
I decided to see how this stacked up against the industry standards using the open-source benchmark harness from Microsoft Research's Memora (ICML 2026). I kept everything strictly controlled: same model, same LoCoMo dataset, and the same category-exclusion rules.
The results were surprising:
- Recordari end-to-end score: 0.831
- Memora published score: 0.801
- Multi-session questions (Recordari): 0.869
- Raw storage baseline: 0.760
What's important here is that these aren't mocked results. The harness runs against the production API—the exact MCP search and recall API a real agent uses—with a
top_k=30 configuration and a 1-hop edge expansion. This setup works across the board with Claude Code, ChatGPT, and any other LLM agent that supports MCP. If you're looking for a real-world AI workflow that doesn't suffer from "amnesia" across sessions, moving from flat files to a structured graph is the way to go.
If you want to check out the benchmark data, the results are public here:
https://github.com/corbym/locomo-recordariAnd for those who want to test an agent against it using the LoCoMo benchmark themselves, you can use the anonymous login here:
https://admin.recordar.io/start