Multimodal RAG chunking is a mess if you don't have a strict
My current setup:
The problem is the structural disconnect. Standard utilities like LangChain's splitters treat the document as a flat string. They have zero awareness of where an image was physically located relative to the text. If I retrieve a chunk of text that refers to "the figure below," but the image is stored in a completely separate pipeline without a spatial pointer, the context is dead on arrival.
I’m stuck on the implementation details for three specific points:
1. Context Preservation: How do I prevent an image from becoming a "homeless" chunk? If the retrieval engine pulls a diagram, it needs the surrounding text to make sense, but if I bundle them, I risk bloating the context window.
2. Chunk Unit Logic: Should I be treating an image + its caption as a single atomic unit, or should the image exist as its own entity with metadata pointers to the text block above it?
3. Database Schema: How are people actually structuring the vector DB to maintain this association? I'm trying to avoid a situation where the similarity search returns a high-scoring image but the metadata link to the explanatory text is broken or misaligned.
Is anyone actually running production systems using placeholder markers or hierarchical chunking to solve this? I don't want to over-engineer a complex metadata mapping system if a simpler layout-aware parsing approach exists.