Multimodal RAG chunking is a mess i

My PDF parser was hallucinating spatial relationships because I treated images as separate entities from the text blocks, effectively killing the context for any figure-heavy document. I'm building a multimodal RAG pipeline for technical manuals, and the "naive" approach of just extracting images and putting them in a vector DB with a caption is a complete disaster.

The problem manifested when I queried the system about specific circuit diagrams. The LLM would find the image (via the caption) but couldn't tell me where the component was located or why it mattered because the surrounding textual explanation was in a different chunk entirely. I was getting these weird, confident lies where the AI would describe a part of the image that didn't exist, simply because it was blending the caption with text from a different page.

I spent two days digging through the retrieval logs and found the culprit. My chunking strategy was purely character-based for text and file-based for images. When the RAG pipeline retrieved the top-k chunks, it would grab the image embedding and maybe one paragraph of text, but if the actual explanation for that image started three paragraphs prior, the context window was missing the "bridge."

The specific error I kept hitting during the evaluation phase looked something like this in my logs:

{
  "query": "What is the voltage limit for the capacitor in Figure 3?",
  "retrieved_chunks": [
    {"id": "img_03", "content": "Figure 3: Power Supply Layout", "score": 0.89},
    {"id": "text_112", "content": "The system operates at 12V. Ensure all connectors are tight.", "score": 0.72}
  ],
  "llm_response": "The voltage limit is 12V.",
  "ground_truth": "The capacitor in Figure 3 has a limit of 5V; exceeding this will cause failure."
}

The LLM just guessed based on the nearest number it found in the text chunk. It didn't "see" the 5V warning because that was tucked away in text_111, which didn't have a high enough similarity score to the query to be retrieved.

I finally stopped fighting the vector search and switched to a "layout-aware" chunking approach. Instead of splitting by characters, I'm now using a vision-based layout analysis tool to identify "semantic zones." I’ve started grouping images with their surrounding text anchors into a single "multimodal document object" before embedding.

Current setup for the fix:

• Layout Analysis: Using an object detection model to find bounding boxes for figures and tables.

Context Windowing: I'm now appending the 500 words before and after* an image to the image's metadata.

• Indexing: I'm indexing the text and the image separately but linking them via a shared parent_id. When the image is hit, the system automatically pulls the parent text block regardless of the similarity score.

It's a tedious workaround, but it's the only way to stop the AI from treating images like isolated islands. If you're just throwing images into a CLIP-based index and hoping for the best, you're going to run into this exact wall. The "R" in RAG is broken if your chunking doesn't respect the visual hierarchy of the original document.