Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

Speculative-decoding drafters are trained per-target on narrow distributions, so acceptance rate (and thus speedup) collapses under workload/domain shift, and EAGLE-style drafters cannot reuse broad pretraining because they consume target hidden states and are distilled per target.

Stage 1 prunes an off-the-shelf small LM (e.g., Qwen3-4B to 2 layers, ~591M params) keeping embedding, LM head, and the first N blocks. Stage 2 continues next-token pretraining of the pruned backbone on FineWeb (~100BT sample, 55k steps ≈ 462B tokens), once, target-agnostically. Stage 3 (per target) swaps the tokenizer, initializes target-vocabulary embedding/LM-head rows by copy-or-subtoken-mean, and expands each attention QKV projection with zero-initialized columns fed projected low/mid/high target hidden states, so the pretrained function is preserved at adaptation start. Stage 4 distills per target with the EAGLE-3 TTT multi-step CE loss (lambda=0.8) on target-generated data. This replaces training a drafter body from scratch with a reusable warm-started shallow LM; the target model and verification procedure are unchanged.

setupcustom runtime, distributed stack

1. evidence 01

   Osprey improves mean accepted length over EAGLE-3 on Qwen3-8B across a 5x5 train/eval domain matrix

   result

   Osprey mean AL 3.686 vs EAGLE-3 3.176, +16.1%; higher AL for every domain pair

   baseline

   EAGLE-3 one-layer drafter trained from scratch, matched adaptation budget

   conditions

   Qwen3-8B target, five domains (chat, code, commonsense, finance, math), 65k prompts/domain, 512 held-out eval prompts/domain, SGLang batch=1, steps=5, topk=1, draft_tokens=6

   Source not located automatically in the PDF
2. evidence 02

   Gains are larger out-of-domain than in-domain on Qwen3-8B

   result

   OOD mean AL 3.493 vs 2.977 (+17.3%); ID 4.465 vs 3.971 (+12.4%); OOD/ID ratio 0.782 vs 0.750

   baseline

   EAGLE-3 one-layer drafter

   conditions

   Qwen3-8B cross-domain 5x5 evaluation, single runs per domain pair

   Source not located automatically in the PDF
3. evidence 03

   Same backbone transfers to Llama-3.3-70B-Instruct with higher AL and throughput

   result

   Mean AL 2.97 vs 2.45 (+21.2%); throughput 183.7 vs 155.8 tok/s (+17.9%); +27.5% AL vs official EAGLE-3 checkpoint

   baseline

   Reproduced EAGLE-3 trained on same target-regenerated Open-PerfectBlend data; also official EAGLE-3 checkpoint

   conditions

   Llama-3.3-70B-Instruct target, 37.5k adaptation steps, five evaluation domains

   Its mean AL and throughput are respectively 21.2% and 17.9% higher than those of our reproduced EAGLE-3 baseline, and 27.5% and 22.2% higher than those of the official EAGLE-3 checkpoint.

   Open page 7 in the PDF
4. evidence 04

   On MiniMax-M2.5 (229B MoE, FP8), Osprey beats the public EAGLE-3 draft on mean AL and throughput, but roughly ties on chat/commonsense

   result

   Mean AL 3.232 vs 2.634 (+22.7%); throughput 219.8 vs 187.1 TPS (+17.5%); MT-BENCH AL 2.805 vs 2.868 (Osprey lower)

   baseline

   Publicly available one-layer EAGLE-3 checkpoint (Lu et al., 2025); cost-parity not claimed

   conditions

   Code-only adaptation (70k prompts), five benchmarks with 64 prompts each, 4xB200 FP8 serving

   Source not located automatically in the PDF
5. evidence 05

   The advantage comes primarily from the pretrained prior, not from the extra layer, and most of Stage-2 benefit is captured by 10k pretraining steps

   result

   Osprey AL 3.790 vs parameter-matched 2-layer EAGLE-3 3.385 (+0.405); 10k vs 55k pretraining steps differ by only 0.026 overall AL

   baseline

   Two-layer EAGLE-3 trained from scratch, same adaptation data/optimizer/LR; 10k-step pretrained checkpoint

   conditions

   Qwen3-8B, ablation over chat/code/finance/math (16 pairs, commonsense excluded); lr=10^-5 for the budget comparison

   Source not located automatically in the PDF

1. Skip the 462B-token pretraining: obtain or build a shallow pruned backbone (e.g., 2-layer Qwen3-4B) and use the authors' Stage-2 ablation insight (10k steps ≈ 576 H100-hours claimed, or reuse their released checkpoint from github.com/LeanModels/Osprey).
2. Pick your actual serving target model and one workload that is out-of-domain relative to the authors' five domains (e.g., your production traffic), generate ~65k target-regenerated prompts for adaptation.
3. Adapt the Osprey drafter and a from-scratch EAGLE-3 baseline on identical data and steps (authors used 2xH100 for Qwen3-8B-scale targets; an 8B target fits on 1-2 80GB GPUs).
4. Serve both drafters in SGLang under your real configuration (batch size, draft tokens, temperature) on one 80GB GPU for an 8B-class target; the paper itself shows Osprey's throughput edge shrinks from 24% at batch 1 to 13% at batch 16, so test at your operating batch size.
5. Measure mean accepted length and tokens/s on held-out prompts from your workload, including an OOD slice; the claim is disproven for your setting if Osprey does not beat the matched from-scratch EAGLE-3 on your OOD traffic or if the gain is smaller than the cost of adaptation.

• Most results are single runs, not multiple seeds (stated in Limitations); small differences such as MT-BENCH ties are not statistically established.
• Osprey is not uniformly better: on MiniMax-M2.5 it is slightly worse than EAGLE-3 on MT-BENCH (2.805 vs 2.868 AL, 201.6 vs 214.4 T/s) and nearly tied on COMMONSENSE-EVAL.
• Throughput advantage narrows at larger batch sizes (24% at batch 1 down to 13% at batch 16) and degrades with sampling temperature (35% at T=1.0 vs 41% at T=0).
• Upfront pretraining cost (576-3,040 H100-hours) is only amortized under high-volume serving or multi-target reuse; the paper itself states it may not be recovered in low-traffic deployments.
• Gains depend on adapting with target-regenerated data; MiniMax-M2.5 results use code-only adaptation data, so non-code targets/domains are untested at that scale.
• Only three targets and one serving stack (SGLang) were evaluated; behavior on other targets, hardware, or verification configurations is unknown.

• Whether the released code/checkpoints are sufficient to skip Stage-2 pretraining entirely and only run per-target adaptation (per-target cost ~48 H100-hours for 8B targets).
• Statistical significance of the gains, given single-run reporting; multi-seed replication needed before adoption.
• Performance at the reader's actual batch size, temperature, and draft-token count, since gains are sensitive to all three.
• Whether a much shorter pretraining (or an existing shallow model) suffices, since 10k steps already captures nearly all of the benefit.
• Adaptation cost and drafter size for the reader's target family, especially cross-tokenizer alignment quality for targets far from Qwen/Llama tokenizers.
• For the MiniMax-M2.5 comparison, the public EAGLE-3 baseline's training cost is unknown, so cost parity is unverified there.