Effective Strategies for Reducing Token Costs in Long-Context RAG Pipelines

Context window expansion in models like Claude 3.5 and Gemini 1.5 has made it tempting to just dump entire documentation folders into the prompt, but the token bill catches up quickly when you're running RAG pipelines at scale. I've spent the last month optimizing a technical support bot that was burning through credits because it was retrieving too many "relevant" chunks that were actually noise.

The biggest win for me was moving away from simple top-k retrieval to a Reranking-first architecture. Instead of sending the top 20 chunks from a vector search directly to the LLM, I pull 50 candidates using a cheap embedding model, then pass them through a cross-encoder reranker (like BGE-Reranker). I only feed the top 5 highest-scoring chunks to the LLM. This cuts the input tokens by 70% without sacrificing accuracy, because the LLM isn't wading through irrelevant context.

Another massive cost sink is redundant metadata. I noticed my chunks were carrying huge JSON headers (source URL, page number, timestamp, category) that the LLM doesn't actually need to answer the question. I wrote a simple pre-processing script to strip everything but the raw text and a short ID before the final prompt construction.

If you're using Cursor for this kind of development, I highly recommend creating a .cursorrules file to enforce strict context management. I use a rule that forces the AI to suggest "summarized context" instead of "full file" when it's helping me write the retrieval logic.

Here is the basic logic I use for the "Context Pruning" step before hitting the API:

def prune_context(chunks, threshold=0.7):
    # Sort by relevance score from the reranker
    sorted_chunks = sorted(chunks, key=lambda x: x['score'], reverse=True)
    
    # Only keep chunks that meet a minimum confidence threshold
    # and cap the total token count strictly
    final_context = []
    current_tokens = 0
    max_tokens = 2000 
    
    for chunk in sorted_chunks:
        if chunk['score'] < threshold:
            break
        
        token_count = len(chunk['text'].split()) # Rough estimate
        if current_tokens + token_count > max_tokens:
            break
            
        final_context.append(chunk['text'])
        current_tokens += token_count
        
    return "\n---\n".join(final_context)

I've also started implementing Prompt Caching, especially for the "System Prompt" and the static part of the knowledge base. If your RAG pipeline has a large set of "global rules" or a fixed reference manual that stays the same across multiple queries, caching that prefix is a game changer. On Claude, this can drop the cost of repeated long-context queries significantly.

A few hard-learned gotchas:

Over-aggressive pruning kills nuance. If you set your reranker threshold too high, the LLM will start hallucinating because it lacks the supporting evidence, which actually increases costs because you end up retrying the prompt.

Vector distance isn't relevance. Don't trust the cosine similarity score from your vector DB to decide what to send to the LLM. Vector search is great for finding the neighborhood, but terrible at pinpointing the exact sentence. Always rerank.

Avoid "Chat History" bloat. In multi-turn RAG, people often send the entire conversation history back with every new retrieval. I now use a "summary buffer" where the AI summarizes the previous 5 turns into a few bullet points, drastically reducing the token overhead for long sessions.