RAG Cost Analysis: Where the Money Actually Goes

Embedding isn't the bottleneck. I spent way too long believing the common dev wisdom that "embeddings are expensive" and you should avoid re-embedding at all costs. I even built my semantic cache around this fear, opting for basic key-value pairs because I assumed a Redis-based semantic layer would bleed money. I was wrong.

When I actually ran the numbers on a 1,000-page document using OpenAI's pricing, the embedding cost was roughly $0.18. That's a rounding error in a production budget. The real costs are hiding in the infrastructure and the LLM generation phase.

To get a real-world view of the spend, I mapped out the entire AI workflow to see where the leakages actually occur:

 Upload PDF
 │
 ▼
 Extract + clean text ── billing quietly starts here
 │
 ▼
 Is this doc already known? ── the check that saves you real money
 │
 ┌──┴──┐
 same changed
 │ │
 skip continue
 │ │
 ▼ ▼
 Chunk only what changed
 │
 ▼
 Attach metadata
 │
 ▼
 Embed (cheap — ~18¢ per 1,000 pages)
 │
 ▼
 Store in vector DB ── millions of chunks, running 24×7
 │
 ▼
 Index + retrieve ── costs more than embedding, at scale
 │
 ▼
 LLM reads chunks, answers ── the real money and latency eater

The most overlooked optimization is the deduplication check. In my experience, about 80% of a knowledge base remains static; only 20% evolves. Treating an updated document as a brand-new file is a rookie mistake that spikes your costs. The professional move is to diff the content and only process the modified sections. This is where precise metadata (versioning, page markers) becomes a financial asset rather than just a technical detail.

The vector database is another deceptive cost center. You aren't paying for the storage of vectors; you're paying for the high-availability infrastructure required to keep those vectors searchable in milliseconds. At a small scale, it's negligible. At production scale, the recurring cost of maintaining the index and the compute for retrieval far outweighs the one-time cost of embedding.

Ultimately, if you're trying to optimize a RAG pipeline from scratch, stop obsessing over embedding tokens and start looking at your indexing overhead and LLM context window usage. That's where the real money is.