Building an AI Second Brain That Compounds Expert Knowledge

We built an AI agent at Meta that functions as a secondary domain expert, capturing how specialists actually reason and making that judgment available across the org. Saving SMEs substantial time on routine questions so they can focus on genuinely novel work.

Why a plain RAG agent wasn't enough

A typical domain-specific agent retrieves document chunks at inference and re-derives reasoning from raw text every run. That's slow, error-prone, and inconsistent. The real knowledge in a compliance domain isn't in the documents themselves, it's implicit: how experts weigh competing considerations, what they prioritize when signals conflict, and how they resolve ambiguity. You can't retrieve judgment with a vector search.

Four layers, each doing one job

The system we designed has four tightly coupled layers:

• Knowledge layer — a structured file hierarchy (analogous to a filesystem) that separates what the agent knows from how it reasons. The directory structure itself makes automated editing tractable because edits map cleanly to files.
• Reasoning layer — explicit procedures that mirror how domain experts actually think through a question, encoded as inspectable steps rather than buried in prompts.
• Evaluation framework — regression tests gated on every change. Every proposed edit to knowledge or reasoning must pass the suite before merging.
• Improvement loop — expert feedback is compiled into diffs, run against the eval framework, and merged only if metrics hold. No model retraining required.

Remove any layer and the others degrade. The knowledge structure makes automated editing possible. The explicit reasoning procedures make failure attribution tractable. The eval framework gates every change. The improvement loop feeds back into both.

Two pieces of novelty that generalize

The contribution isn't a smarter model — it's the integration:

• A structured, auditable architecture that separates knowledge from reasoning. Auditors can read what the agent believes and why without unpicking a prompt.
• A self-improvement loop that turns one-off feedback into regression-tested updates without retraining. Expert effort compounds rather than evaporates.

The pattern generalizes to any domain governed by retrievable text rather than model weights — finance, security, engineering, legal review.

Where this actually hurts

In our compliance domain, the same question archetypes were being answered manually across hundreds of product reviews. Expert assessments took days of manual research, and inconsistency between assessments created real organizational risk — two reviewers could reach opposite conclusions on the same facts. The agent standardizes the routine path so reviewers only spend their judgment on the genuinely novel and ambiguous cases.

The honest limitation

This works when domain knowledge is articulable in text and procedures. It does not work when expert judgment depends on embodied experience, physical intuition, or pattern recognition that practitioners can't verbalize. We had to interview SMEs repeatedly and watch them work before we could encode their reasoning — the loop is only as good as the experts' ability to externalize their own thought process. That's a real ceiling, and it's why the system is a second brain, not a replacement.

If you're sitting on a corpus of institutional knowledge and watching your senior people spend half their week answering the same five questions, the architecture is worth studying. The eval-gated improvement loop is the part most teams skip, and it's the part that determines whether the system gets better over time or silently rots as the domain shifts.