Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

Possible product use: a self-hosted writing or coding assistant where output generation is a measured bottleneck.

What you would change

Osprey pretrains a shallow draft model, then adapts it to a target model. The target still verifies proposed tokens. Reusing a compatible checkpoint matters because the preparation stage requires substantial compute.

Try this only after checking target and runtime compatibility. The paper evaluates Qwen, Llama, and MiniMax targets; these results do not establish support or speedups for GLM-4.7-Flash. Table 10 on page 19 reports 576–3,040 H100-hours for pretraining, before target adaptation. Hardware-hours are not a rental quote.

Source version reviewed 2026-09-14. No experiment reproduced by Paperraft.

Begin with an artifact check. Verify a usable drafter for your exact target and serving version. If none exists, record the adaptation work and budget before booking GPUs; a new target is an experiment of its own.

Compare against

The same target with speculation disabled, your current drafter if present, and the candidate drafter. Keep the GPU, precision, sampling, context, and request trace fixed.

1. Inspect the official repository linked by the paper for weights, license, target support, and pinned runtime instructions. Confirm whether you can reuse pretraining and what adaptation remains. Do not infer compatibility from model size.
2. Save 50 held-out product prompts with realistic input and output lengths. Include an unusual-domain slice and define quality, response-time, and cost limits before running. This is a proposed pilot size.
3. Once the artifact and budget checks pass, replay the same trace for each condition at low load and your expected concurrency. Repeat paired runs after warm-up, then inspect outputs for regressions.
4. Record both accepted draft length and end-to-end performance. Keep preparation, adaptation, idle GPU time, and serving costs in the result sheet. Preserve a configuration that disables the drafter.

What to measure

First-token and completion latency, tokens per second, completed requests, output correctness, peak memory, and total cost per successful request. Report low-load and concurrent results separately.

When to keep it

Keep the drafter only if it improves the workload that matters while preserving quality and meeting latency limits. Divide setup cost by measured savings per request to estimate break-even traffic; if savings are zero or negative, there is no cost break-even. A pilot on another model does not validate GLM support.

recommendation

test

research area

speculative decoding

technical change

Target-agnostic pretrained drafters for speculative decoding

published

2026-09-08

The method is directly relevant to lowering LLM serving latency on a single-GPU self-hosted stack, but its benefit depends on hidden-state access, target model choice, and workload-specific acceptance rates, so it warrants a controlled pilot rather than immediate production adoption.

equations from arXiv HTML · fetched 2026-09-28

1. claim 01Paperraft inference

   Osprey improves mean accepted length over EAGLE-3 on Qwen3-8B across a 5x5 train/eval domain matrix

   result

   Osprey mean AL 3.686 vs EAGLE-3 3.176, +16.1%; higher AL for every domain pair

   baseline

   EAGLE-3 one-layer drafter trained from scratch, matched adaptation budget

   conditions

   Qwen3-8B target, five domains (chat, code, commonsense, finance, math), 65k prompts/domain, 512 held-out eval prompts/domain, SGLang batch=1, steps=5, topk=1, draft_tokens=6

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Gains are larger out-of-domain than in-domain on Qwen3-8B

   result

   OOD mean AL 3.493 vs 2.977 (+17.3%); ID 4.465 vs 3.971 (+12.4%); OOD/ID ratio 0.782 vs 0.750

   baseline

   EAGLE-3 one-layer drafter

   conditions

   Qwen3-8B cross-domain 5x5 evaluation, single runs per domain pair

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
3. claim 03source-linked

   Same backbone transfers to Llama-3.3-70B-Instruct with higher AL and throughput

   result

   Mean AL 2.97 vs 2.45 (+21.2%); throughput 183.7 vs 155.8 tok/s (+17.9%); +27.5% AL vs official EAGLE-3 checkpoint

   baseline

   Reproduced EAGLE-3 trained on same target-regenerated Open-PerfectBlend data; also official EAGLE-3 checkpoint

   conditions

   Llama-3.3-70B-Instruct target, 37.5k adaptation steps, five evaluation domains

   Its mean AL and throughput are respectively 21.2% and 17.9% higher than those of our reproduced EAGLE-3 baseline, and 27.5% and 22.2% higher than those of the official EAGLE-3 checkpoint.

   Open supporting passage on PDF page 7
4. claim 04Paperraft inference

   On MiniMax-M2.5 (229B MoE, FP8), Osprey beats the public EAGLE-3 draft on mean AL and throughput, but roughly ties on chat/commonsense

   result

   Mean AL 3.232 vs 2.634 (+22.7%); throughput 219.8 vs 187.1 TPS (+17.5%); MT-BENCH AL 2.805 vs 2.868 (Osprey lower)

   baseline

   Publicly available one-layer EAGLE-3 checkpoint (Lu et al., 2025); cost-parity not claimed

   conditions

   Code-only adaptation (70k prompts), five benchmarks with 64 prompts each, 4xB200 FP8 serving

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
5. claim 05Paperraft inference

   The advantage comes primarily from the pretrained prior, not from the extra layer, and most of Stage-2 benefit is captured by 10k pretraining steps

   result

   Osprey AL 3.790 vs parameter-matched 2-layer EAGLE-3 3.385 (+0.405); 10k vs 55k pretraining steps differ by only 0.026 overall AL

   baseline

   Two-layer EAGLE-3 trained from scratch, same adaptation data/optimizer/LR; 10k-step pretrained checkpoint

   conditions

   Qwen3-8B, ablation over chat/code/finance/math (16 pairs, commonsense excluded); lr=10^-5 for the budget comparison

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source

1. risk 01Paperraft inference

   Most results are single runs, not multiple seeds (stated in Limitations); small differences such as MT-BENCH ties are not statistically established.

2. risk 02Paperraft inference

   Osprey is not uniformly better: on MiniMax-M2.5 it is slightly worse than EAGLE-3 on MT-BENCH (2.805 vs 2.868 AL, 201.6 vs 214.4 T/s) and nearly tied on COMMONSENSE-EVAL.

3. risk 03Paperraft inference

   Throughput advantage narrows at larger batch sizes (24% at batch 1 down to 13% at batch 16) and degrades with sampling temperature (35% at T=1.0 vs 41% at T=0).

4. risk 04Paperraft inference

   Upfront pretraining cost (576-3,040 H100-hours) is only amortized under high-volume serving or multi-target reuse; the paper itself states it may not be recovered in low-traffic deployments.

5. risk 05Paperraft inference

   Gains depend on adapting with target-regenerated data; MiniMax-M2.5 results use code-only adaptation data, so non-code targets/domains are untested at that scale.

6. risk 06Paperraft inference

   Only three targets and one serving stack (SGLang) were evaluated; behavior on other targets, hardware, or verification configurations is unknown.

1. Skip the 462B-token pretraining: obtain or build a shallow pruned backbone (e.g., 2-layer Qwen3-4B) and use the authors' Stage-2 ablation insight (10k steps ≈ 576 H100-hours claimed, or reuse their released checkpoint from github.com/LeanModels/Osprey).
2. Pick your actual serving target model and one workload that is out-of-domain relative to the authors' five domains (e.g., your production traffic), generate ~65k target-regenerated prompts for adaptation.
3. Adapt the Osprey drafter and a from-scratch EAGLE-3 baseline on identical data and steps (authors used 2xH100 for Qwen3-8B-scale targets; an 8B target fits on 1-2 80GB GPUs).
4. Serve both drafters in SGLang under your real configuration (batch size, draft tokens, temperature) on one 80GB GPU for an 8B-class target; the paper itself shows Osprey's throughput edge shrinks from 24% at batch 1 to 13% at batch 16, so test at your operating batch size.
5. Measure mean accepted length and tokens/s on held-out prompts from your workload, including an OOD slice; the claim is disproven for your setting if Osprey does not beat the matched from-scratch EAGLE-3 on your OOD traffic or if the gain is smaller than the cost of adaptation.

• Whether the released code/checkpoints are sufficient to skip Stage-2 pretraining entirely and only run per-target adaptation (per-target cost ~48 H100-hours for 8B targets).
• Statistical significance of the gains, given single-run reporting; multi-seed replication needed before adoption.
• Performance at the reader's actual batch size, temperature, and draft-token count, since gains are sensitive to all three.
• Whether a much shorter pretraining (or an existing shallow model) suffices, since 10k steps already captures nearly all of the benefit.
• Adaptation cost and drafter size for the reader's target family, especially cross-tokenizer alignment quality for targets far from Qwen/Llama tokenizers.
• For the MiniMax-M2.5 comparison, the public EAGLE-3 baseline's training cost is unknown, so cost parity is unverified there.