MedProb shows that medical VLMs know more than they can actually

Stop assuming that a Vision-Language Model (VLM) needs massive medical fine-tuning or a complex multi-agent pipeline to handle medical visual question answering (Med-VQA). The MedProb framework (detailed in arXiv:2609.04336v1) basically proves that the "knowledge" is often already sitting there in the frozen representations of the model, but the generation head—the part that actually turns those representations into words—is the bottleneck.

The core idea here is to use a lightweight probing framework. Instead of asking a model to generate a free-text answer (which is where things usually fall apart), MedProb looks at the internal representations of the frozen VLM and predicts multiple-choice answers directly. This bypasses the generation phase entirely.

The performance gap between probing and prompting

The results across the PATH-VQA, SLAKE, and VQA-RAD datasets are pretty surprising. MedProb recovered significantly more answer-relevant signals than standard prompting. More interestingly, it narrows the gap between small and large models. When you use prompting, a small model looks incompetent compared to a giant one. But when you probe the internal states, you realize the smaller model actually possesses a lot of the correct signal; it just can't "generate" it effectively.

A few technical takeaways from the research:

• Medical Adaptation: The researchers tested 14 matched pairs of general-purpose and medical VLMs. Surprisingly, medical-specific adaptation didn't consistently improve linear decodability. This suggests that general VLMs are already capturing a lot of the necessary medical visual features.
• The Position Bias Problem: We all know LLMs have a bias toward certain answer positions in multiple-choice questions. The paper notes that free-text generation can show an answer-position bias of up to 10 percentage points. While MedProb also has a positional bias, it operates differently than the bias seen in prompting.
• Extending to Open-Ended QA: While the main focus is on multiclass/multiple-choice settings, they found a way to use the probe for open-ended generation through a rejection-sampling scoring procedure.

How this changes the AI workflow for medical VQA

For anyone building a medical AI workflow, this is a bit of a wake-up call. We spend so much time on complex prompt engineering or expensive fine-tuning, but the "signal" is often already present in the latent space of the model. If you are trying to extract specific labels from a medical image, a linear probe on a frozen model might actually be more reliable than trying to coax a "correct" sentence out of the model via a prompt.

If you're doing a deep dive into how to deploy these models, you might find that a simple linear layer on top of a frozen backbone is more efficient than a full-blown agentic system. It's a much more lightweight deployment strategy that avoids the unpredictability of free-text generation.

The fact that medical fine-tuning doesn't always improve the decodability of these representations suggests we might be over-fitting during the fine-tuning process or that the "knowledge" is already there, and we just need better ways to extract it. This makes me wonder if we should be spending less time on massive medical datasets and more time on how to probe these representations for specific clinical markers.