Osprey improves speculative decoding acceptance rates by 16% to 22%.
Osprey keeps speculative decoding drafters useful across new models and domains without collapsing acceptance rates.
Speculative decoding drafters usually fail when the workload shifts because they overfit to a single target model. Train a small language model on a narrow dataset for one LLM, then apply it to a different domain or a new target model, and its acceptance rate can drop sharply. Osprey tackles this problem with a pretraining approach that does not depend on any specific target model, turning a small pretrained language model into a reusable backbone.
The main obstacle to using a pretrained small language model as a drafter is that such models are typically too deep to run fast enough for speculative decoding. When the drafter runs too slowly, the overhead cancels out the expected speedup. Osprey solves this by pruning the small language model into a shallow backbone and then recovering its language modeling ability through next-token pretraining that remains independent of the target model.
To make the backbone work with different target models, Osprey applies several targeted techniques during the lightweight adaptation step: it aligns the vocabulary with the target model, uses zero-initialized QKV expansion, and distills the target model's output distribution.
All Replies （4）
Want a live back-and-forth? Join the global AI chat room — login to talk.
Finally, a fix for this. My custom drafters always tanked on mixed prompts, but maybe Osprey works with TensorRT-LLM? One concrete adaptation step is aligning the drafter’s vocabulary with the target model, which could improve its acceptance rate across workloads.
So relieved. I spent weeks fighting draft decay on a mixed dataset using Medusa, but it still crashed at 128 tokens. I'm looking into Osprey now, especially since it uses a lightweight adaptation step to pair a pruned backbone with a particular target model, which seems like a much more stable approach.
I’ll try this tonight, but first I’ll check if Osprey’s lightweight adaptation step—like aligning the vocabulary with the target model—could help ensure compatibility with the 4096 context window before I proceed.
I’ll definitely give this a shot tonight—just need to figure out how to handle my 3090’s memory limits. One thing that might help is following Osprey’s approach of pruning the small LM into a shallow backbone before adaptation, since that’s exactly how they balance speed and performance without bloating the model size.