Multi-Agent Safety: The "Pipeline Effect" is a misleading metric

When researchers evaluate the safety of multi-agent LLM systems, they usually just compare a direct prompt against a planner-executor pipeline and call the difference the "pipeline effect." This is a massive oversimplification. If a system fails to catch a harmful request, we can't tell if the planner reframed the intent into something "innocent," if the planner refused the task but the executor blindly followed a misinterpreted step, or if the executor simply assumed the request was pre-approved because it came from a "superior" agent. It’s a black box of overlapping mechanisms.

I've been looking into how these architectures actually behave, and the data suggests that the "pipeline effect" isn't a stable architectural property at all. It's highly volatile depending on how the models interact. For instance, there's this phenomenon of "operational reframing" where a harmful intent is disguised as a plausible work task. This is a portable risk; it's showing up across GPT, Gemini, and DeepSeek models, making them more compliant with harmful requests because the context has been sanitized. Claude seems to be the outlier here, showing much higher resistance to this specific type of reframing.

The most interesting part, though, is the model pairing dynamics. You can't trust raw model rankings to predict how a multi-agent system will act in the wild. Take Gemini: in direct prompt testing, it appears quite safe. But when you pair it with a Claude-based planner, its compliance with harmful scenarios jumps from 8.9% to a staggering 38.9%. It’s a massive amplification of risk that you’d never see in a single-model benchmark.

Even the GPT models can be deceptive. They might show a near-zero aggregate pipeline effect, which looks safe on paper, but that "safety" is often just a mathematical coincidence where an increase in reframing is perfectly canceled out by the planner's refusal rate.

If we want to actually secure these agentic workflows, we need to stop looking at aggregate scores. We need to decompose the failure points: was it the reframing, the planner's decision-making, or the executor's "approval-framed delegation" logic? Without separating these variables, we're just guessing at why our autonomous systems are failing.