Why MCQ-based evaluation beats BLEU for video captions

Traditional metrics like BLEU or METEOR are useless for video captioning because they penalize any word that doesn't match a ground-truth reference, even if the description is accurate. I've found that shifting to Multiple-Choice Question Answering (MCQA) is the only way to actually measure information fidelity. Instead of checking if a caption "looks like" a reference, you use an LLM to ask if the caption contains the correct facts about the video, treating the caption as the source of truth.

How the MCQA evaluation process actually works

The goal is to see if a caption maximizes coverage of the video's content without hallucinating. I've been testing this by generating a set of questions where only one answer is supported by the video. If the model can answer the question correctly using only the generated caption, the caption is considered high-fidelity.

1. Question Generation: Use a VLLM to generate a set of questions based on the video and the ground-truth reference.
2. Distractor Creation: Generate three incorrect but plausible options (distractors) for each question.
3. Caption-based Answering: Feed the generated caption into a separate LLM. The model must choose the correct answer among the four options based solely on the text of the caption.
4. Scoring: The "Fidelity Score" is the percentage of questions correctly answered. If the model picks the wrong option, it means the caption either missed a key detail or introduced a hallucination.

Where this breaks compared to reference matching

When I ran this against standard benchmarks, the discrepancy was jarring. A caption could have a very low BLEU score because it used different adjectives or focused on a different part of the scene, yet it would score 90% on the MCQA test because the factual density was high.

The failure point here is the "distractor quality." If the distractors are too obvious, the LLM might guess the right answer even if the caption is vague. To get a real reading, the distractors need to be visually similar to the actual scene but factually wrong.

Performance trade-offs and costs

This method is computationally expensive. While a BLEU score takes milliseconds to calculate, an MCQA evaluation requires multiple LLM passes.

• Latency: For a dataset of 1,000 videos, you are looking at 4,000+ LLM calls (1 for question generation and 3+ for distractor validation per video).
• Cost: Using GPT-4o for the evaluation phase can get pricey. I've had better luck using a smaller, specialized model for the answering phase to keep costs down, though you risk "model bias" where the evaluator is too lenient on the captioner.
• Accuracy: It eliminates the "one-to-many" problem where a video of a dog running in a park could be described as "a golden retriever in the grass" or "a dog sprinting through a field." Both are correct, but reference matching would penalize one of them.

If you are benchmarking a VLLM's ability to describe complex scenes, stop relying on n-gram overlap. Use a set of checkable questions to verify if the facts in the caption actually match the pixels in the video.