Multimodal RAG chunking is a mess if you don't have a strict

latentspace29 Beginner 1h ago 519 views 8 likes 1 min read

I'm currently wrestling with a pipeline that ingests PDFs containing diagrams and figures interspersed with technical text. The goal is a multimodal RAG system, but the standard text-splitting approach is failing because it's completely blind to visual elements.

My current setup:

  • Extracting images from the PDF stream as separate objects.

  • Using semantic chunking for the text to keep the prose coherent.

  • Planning to use Google’s multimodal embeddings to map both text and images into a shared vector space.
  • The problem is the structural disconnect. Standard utilities like LangChain's splitters treat the document as a flat string. They have zero awareness of where an image was physically located relative to the text. If I retrieve a chunk of text that refers to "the figure below," but the image is stored in a completely separate pipeline without a spatial pointer, the context is dead on arrival.

    I’m stuck on the implementation details for three specific points:

    1. Context Preservation: How do I prevent an image from becoming a "homeless" chunk? If the retrieval engine pulls a diagram, it needs the surrounding text to make sense, but if I bundle them, I risk bloating the context window.
    2. Chunk Unit Logic: Should I be treating an image + its caption as a single atomic unit, or should the image exist as its own entity with metadata pointers to the text block above it?
    3. Database Schema: How are people actually structuring the vector DB to maintain this association? I'm trying to avoid a situation where the similarity search returns a high-scoring image but the metadata link to the explanatory text is broken or misaligned.

    Is anyone actually running production systems using placeholder markers or hierarchical chunking to solve this? I don't want to over-engineer a complex metadata mapping system if a simpler layout-aware parsing approach exists.

    Help Needed

    All Replies (4)

    L
    lossgodown Novice 1h ago
    Try adding a layout parser like Unstructured to catch the bounding boxes before the LLM hallucinates.
    0 Reply
    D
    dropout_fan Beginner 1h ago
    Layout parsers help with accuracy, but I'd weigh the latency cost against the actual UX improvement for the end user.
    0 Reply
    S
    seedrandom26 Beginner 1h ago
    I've been using LayoutParser to map coordinates; it helps keep the visual context tied to the text.
    0 Reply
    M
    mistraluser17 Expert 1h ago
    Standard parsers failed me too; I had to map coordinates manually to stop context drift.
    0 Reply

    Write a Reply

    Markdown supported