Semantic Overlays pushes Qwen-3.

I've been stress-testing the new Semantic Overlays adapters from Joshua Penman's arXiv paper (2608.23873) against the usual injection suites — Tensor Trust, Hijacking, and the newer Gandalf-style black-box sets. The claim is bold: a frozen 9B model plus a tiny trained adapter hits state-of-the-art robustness without ever seeing those benchmarks during training. That "zero-shot on attacks" detail matters — most defenses overfit to the eval set.

The mechanism reminds me of an NX bit for context: the adapter learns to tag certain spans as "untrusted data" versus "instruction," then shifts the model's attention away from executing anything inside the untrusted regions. It's not a classifier bolted on top; it rewires how the frozen weights attend. That distinction shows up in the ablation — stripping the adapter drops attack success rate from ~2% back to ~68% on the same prompts.

# Quick local test if you want to reproduce
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", device_map="auto")
model = PeftModel.from_pretrained(base, "joshuapenman/semantic-overlays-adapters")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

Latency overhead is negligible — single forward pass, no ensemble. Memory footprint adds ~40MB for the adapter weights. I ran a 500-prompt sweep on an A100-40GB: throughput dropped ~3% versus raw Qwen, well within noise.

Where it struggles: multi-turn injection where the malicious payload arrives in turn 3+ after benign context. The adapter's "untrusted" tagging seems calibrated for single-shot boundaries. Also, white-box adaptive attacks (gradient-guided token search) aren't in scope per the paper — fair limitation, but worth noting if you're threat-modeling a production system.

Compared to the usual defenses:

• System-prompt hardening: brittle, fails on encoding/obfuscation variants
• Input classifiers: high false-positive rate on legitimate code/markup
• Semantic Overlays: generalizes to unseen encodings, but needs per-domain adapter tuning for best results

The adapters on HF are drop-in for Qwen-2.5-7B/9B and Llama-3.1-8B. If you're running a RAG pipeline or agent framework where untrusted context is inevitable, this is the first drop-in mitigation I've seen that doesn't tank helpfulness scores on MT-Bench. Worth a weekend eval against your own red-team set.