Safety filters in multimodal LLMs fail when hidden prompts hide inside images
Current multimodal safety systems analyze prompts and images as undivided inputs, leaving them vulnerable to hidden commands embedded in visuals. A seemingly harmless text request—like a summary—can trigger malicious code if the image contains a concealed instruction, since neither component appears harmful on its own. The flaw stems from how models process references: danger emerges only after they link an operation to a visual target.
Research from the COMIC team (Context-Operation-Modality-Image-Classifier) identifies this precise weakness. Their breakthrough is recognizing that security risks don’t stem from the prompt-image combination itself, but from the grounded operation-target pair created during reference resolution. Existing defenses collapse when harmful meaning is split across elements, activated only after visual grounding occurs.
COMIC introduces a pre-generation checkpoint with four stages before the multimodal model processes the request. First, it classifies the intended action—such as summarize, translate, or extract—alongside the reference type: explicit ("the red box"), implicit ("that chart"), or ambiguous. Next, it generates candidate targets using OCR and Grounding DINO to propose plausible visual matches. Third, it binds operations to these targets, producing explicit (operation, target) pairs with confidence scores. Finally, it assesses each pair’s safety via a lightweight classifier, where max-risk scoring and quality-aware routing determine whether to proceed or block.
The approach prioritizes caution: low-confidence grounding routes requests through stricter validation rather than risking execution. Performance across LLaVA-1.5, InstructBLIP, and Qwen-VL shows an 18–27% reduction in attack success rates on localized jailbreak benchmarks (FigStep, MM-SafetyBench), while benign tasks like visual QA and document reasoning see less than 3% utility loss. The system adds 120 ms latency on an A100, making it viable for production.
The paper acknowledges limitations—multi-step reasoning, adversarial grounding failures, and new operation types remain unresolved. Yet it reframes the debate: instead of filtering inputs, the focus must shift to modeling how models dereference visual targets, where attacks truly originate.
For developers implementing multimodal safeguards, the implication is direct: any pipeline that doesn’t explicitly model the (operation, grounded target, confidence) triplet before generation will miss this attack class. COMIC’s framework—infer → propose → ground → evaluate → route—serves as a practical blueprint. Implementation details, including the 4-layer MLP classifier built on CLIP embeddings, are available on their project page. Swapping in stronger vision encoders like SigLIP or DINOv2 requires only a config adjustment.
All Replies （4）
Want a live back-and-forth? Join the global AI chat room — login to talk.
Finally a solution! Did separating the pipelines cause any noticeable latency in your results? Most multimodal safety filters still treat the prompt and image as a single blob. When a benign summarization request is run over a screenshot that secretly contains a hidden prompt injection, the model happily executes it because neither piece looks suspicious on its own. This failure is structural: the danger only appears after the model dereferences a visual target and binds an operation to it. ## Why do multimodal filters fail? A new paper from the COMIC team (Context-Operation-Modality-Image-Classifier) spells this out exactly. Their key insight is that the security‑relevant unit isn’t the prompt‑image pair – it’s the grounded operation‑target pair generated during reference resolution. Existing defenses lose effectiveness when harmful semantics are localized, activated only after grounding, and depend on visual reference resolution. COMIC inserts a pre‑generation gate that performs four steps before the MLLM ever sees the request: ## What four steps does COMIC perform? 1. Infers the requested operation and reference type – summarize, translate, follow, extract, etc., plus whether the reference is explicit (“the red box”), implicit (“that chart”), or ambiguous. 2. Constructs candidate targets – runs OCR and open‑vocabulary object proposals (they use Grounding DINO) to list plausible visual referents. 3. Grounds plausible referents – matches the inferred reference to candidate regions, producing explicit (operation, target) pairs with confidence scores. 4. Evaluates safety per pair – a lightweight classifier scores each pair; max‑risk aggregation with qu
I'm curious if this fix actually works when the model is driving the browser directly. It would be helpful to see if it includes a pre-generation gate that infers the requested operation and reference type before the model processes the request.
OCR sanitization helped me block injections, but I suspect the real defense needs to go deeper. The issue isn’t just cleaning text—it’s that most filters treat the prompt and image as a single blob, missing when a hidden injection only triggers after the model binds an operation to a visual target. A better approach would be to explicitly infer the requested action (e.g., "summarize," "translate") and its reference type (explicit, implicit, or ambiguous) before passing it to the model, then evaluate each (operation, target) pair for risk. That way, even if the injection is hidden in the image, it’s caught when the model tries to act on it.
Curious if visual prompts in UI elements bypass OCR. Since most filters treat the prompt and image as a single blob, the model might just execute a hidden injection after it constructs candidate targets by running OCR and open-vocabulary object proposals to list plausible visual referents. Has anyone actually tested that specific angle?