How to Chunk Multimodal PDFs for RAG
Here is the mess I'm currently navigating. I have a workflow where I extract images from PDFs as separate files so my text splitter doesn't choke on raw bytes, and I'm planning to use Google's multimodal embedding models to map everything into a single vector space. I'm even using semantic chunking for the text portions to keep the prose coherent.
The problem is the "spatial awareness" gap. LangChain and other common utilities are blind to where an image actually lives in the document flow. Because my text and image pipelines are decoupled, I'm hitting a wall on three specific fronts:
1. Contextual Anchoring: How do I prevent a retrieval result from returning a diagram with zero explanatory text, or a paragraph of text that refers to "Figure 4" without actually pulling "Figure 4"?
2. Chunk Granularity: Should an image be treated as its own standalone chunk, or is it more cost-effective and accurate to bundle it with the immediate surrounding text as a single multimodal unit?
3. Database Association: I need to store these in a vector database, but I'm worried about losing the relational link between the visual and the textual during the retrieval phase.
I'm looking for a production-grade pattern. I've heard people mention using placeholder markers (like [IMAGE_ID_123]) within the text stream to maintain a link, or using hierarchical chunking where a parent chunk contains the full page context.
If you've actually deployed a multimodal RAG system that doesn't fall apart when a user asks about a specific chart, how are you handling the metadata linking? Are you injecting coordinate data into the metadata to reconstruct the context, or are you just brute-forcing it with heavy semantic overlaps?