Mind Silo: Building a Cross-LLM Memory Layer

Fragmented context is the biggest productivity killer in the AI workflow. The context I build up in ChatGPT is invisible to Claude, and my Cursor sessions know nothing about either. Even worse is the "sidebar graveyard" where genuinely useful long conversations sink into an infinite list and are never found again. To fix this, I built Mind Silo—a centralized memory layer that all these tools can read from and write to.

The Technical Architecture

The system isn't just a database; it's a pipeline that converts raw chat logs into a queryable knowledge graph.

1. Passive Capture: I developed a Chrome MV3 extension that monitors conversations on Claude, ChatGPT, Gemini, Cursor, and Claude Code. This removes the friction of manual saving or copy-pasting.
2. Atomic Extraction: Raw transcripts are too noisy for a vector store. I use a background worker (Redis/Arq) to distill conversations into "atomic memories"—single, standalone facts, decisions, or preferences.
3. Vectorized Storage: I'm using Postgres with pgvector. Instead of a flat list, each memory is embedded, and semantically related nodes are connected into a graph.
4. MCP Integration: This is the key for the AI workflow. I published an MCP server (mind-silo-mcp on npm) that exposes three specific tools. Any MCP-compatible client (Claude Desktop, Claude Code, Cursor) can now query the memory graph or commit new memories back to it in real-time.
5. Crypto-Shredding: For privacy, I implemented per-user AES-256 encryption with derived keys. Deleting a user isn't just a DELETE query; it's a crypto-shred where destroying the key makes the data permanently unrecoverable.

The Stack

• Backend: FastAPI (Python 3.12, uv)
• Frontend: Next.js 14 & Three.js
• Database/Cache: Postgres + pgvector & Redis/Arq
• Auth/Billing: Clerk & Stripe
• Infrastructure: pnpm monorepo deployed on Render and Vercel

Visualization as a UX Tool

I originally built the 3D brain visualization using Three.js as a side project. Memories render as nodes in 3D space, edges represent semantic connections, and projects cluster by color.

Surprisingly, this became the most critical part of the product. Most memory tools provide a text list or "buckets," but nobody wants to scroll through 400 text entries. A 3D graph that you can fly through allows you to visually map how your thoughts and projects are interconnected, turning a database into a mental map.

Solving the "Temporal Contradiction" Bug

The biggest technical hurdle I'm facing right now isn't storage, but temporal correctness. Vector stores are naive regarding time.

The Scenario:

• March: I tell Claude, "I am using Postgres for my project." → Memory stored.
• June: I tell Claude, "I've migrated my project to SQLite." → Memory stored.

When I query my stack later, the vector store finds both. Because they are semantically similar, the LLM may retrieve the outdated Postgres memory and hallucinate that I'm still using it.

The Fix:
I am currently implementing typed edges with validity windows. Instead of a simple vector match, the system needs to identify when a new memory supersedes an old one.

{
  "memory_id": "mem_123",
  "content": "Using SQLite for Project X",
  "metadata": {
    "supersedes": "mem_045",
    "valid_from": "2024-06-01T10:00:00Z",
    "category": "tech_stack"
  }
}

By tracking these relationships, the retrieval logic can filter out "expired" truths, ensuring the LLM only sees the most current version of a fact.

The project is live at https://mind-silo.com if you want to test the integration with your own MCP clients.