Multimodal LLMs produce wildly incorrect visual reasoning despite correct object detection

A meme showing a red plane triggers a response that exposes a serious disconnection between what the model sees and how it reasons. This is not a minor slip-up but a structural problem: the visual embedding pipeline fails to hand off meaning to the higher-level language understanding that was built during training.

The breakdown follows a consistent sequence. First, detection itself works—the system correctly names the "red plane" as a concept. Then, context falls apart: irony and visual punchlines are ignored, and everything is read literally. Finally, to keep the output coherent, the model invents relationships between the plane and its surroundings that simply are not in the image.

Forcing structured, step-by-step prompts does not stop the drift. Even with a strict template that asks for object identification, spatial layout, text extraction, and evidence-based conclusions, the model still produces confident but baseless reasoning by the fourth step. The template may look like this:

Analyze the provided image using a step-by-step visual decomposition:
1. Identify all primary objects and their colors.
2. Describe the spatial relationship between these objects (e.g., foreground, background, overlapping).
3. Identify any text or symbolic elements present.
4. Based on the visual evidence alone, describe the logical interaction occurring.
5. Avoid making assumptions about humor or sarcasm unless explicitly supported by text.

The practical cost is real. Systems that rely on visual input—UI navigation, video monitoring, assistive tools—cannot afford hallucinations that parade as certainty. Text-based reasoning has matured, but multimodal integration is still in an experimental stage, and the misalignment between vision encoders and language decoders remains an open problem with no obvious fix in sight.