My RAG retrieval quality is tanking as my dataset grows

I've been building a RAG pipeline using SentenceTransformer and FAISS. On a tiny test set, everything looks great, but as I scale up, I'm hitting a wall with retrieval precision. I'm currently using IndexFlatIP with L2-normalized vectors to simulate cosine similarity. It's basically a brute-force search, which is fine for a few hundred documents, but I'm worried about the latency and accuracy trade-offs as I move toward a real-world deployment.

Here is the core of my current implementation:

class VectorStore:

 def __init__(self):
 self.model = SentenceTransformer(EMBEDDING_MODEL)
 self.index = None
 self.chunks = []

 def build(self, chunks):

 self.chunks = chunks

 texts = [chunk["text"] for chunk in chunks]

 embeddings = self.model.encode(
 texts,
 convert_to_numpy=True,
 show_progress_bar=True
 ).astype("float32")

 # Normalize embeddings so inner product = cosine similarity
 faiss.normalize_L2(embeddings)

 dimension = embeddings.shape[1]

 self.index = faiss.IndexFlatIP(dimension)
 self.index.add(embeddings)

 def search(self, query, k=5):

 query_embedding = self.model.encode(
 [query],
 convert_to_numpy=True
 ).astype("float32")

 faiss.normalize_L2(query_embedding)

 scores, indices = self.index.search(
 query_embedding,
 k
 )

 results = []

 for score, index in zip(scores[0], indices[0]):
 if index == -1:
 continue

 results.append({
 "text": self.chunks[index]["text"],
 "chunk_id": self.chunks[index]["chunk_id"],
 "score": float(score)
 })

 return results

I've realized that just having "similar" vectors isn't enough for a production-grade AI workflow. I have a few specific technical hurdles I'm trying to clear:

The Indexing Bottleneck

Since I'm using IndexFlatIP, every query scans the entire index. I'm considering switching to IndexHNSWFlat or IndexIVFFlat to speed things up, but I'm unsure about the recall hit. In a practical tutorial for RAG, you often see HNSW recommended for speed, but is the memory overhead worth it compared to something like IndexIVFPQ for larger datasets?

The "Vector Search is Not Enough" Problem

I'm noticing that cosine similarity often retrieves chunks that are semantically related but logically useless for the actual answer. I'm thinking about adding a reranker stage—pulling 50 candidates via FAISS and then narrowing it down to the top 5 using a Cross-Encoder. Does this actually solve the precision issue, or is it just adding latency to the pipeline?

Embedding Model Selection

I'm using all-MiniLM-L6-v2 because it's fast, but I suspect it's the weak link. Should I be focusing on Recall@K metrics when picking a new model, or does the choice of embedding model actually impact how the FAISS index behaves?

Metadata Filtering Struggles

Right now, I store metadata in a separate Python list (self.chunks) and map it back via indices. This makes filtering (e.g., "only search within document_X") a nightmare because I have to filter after the search, which ruins my Top-K results. I need a better way to handle filtered vector search without iterating through the entire result set manually.