AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines

Your self-hosted model spends too long generating answers token by token.

What you would change

Check whether a compatible draft model and serving integration exist before renting compute.

Check whether a compatible draft model and serving integration exist before renting compute.

Compare against

The same target with speculation off, then your existing drafter if you have one.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Accepted draft length, tokens per second, tail latency, output quality, and total GPU cost.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

speculative decoding

technical change

Retrieval-based speculative decoding with session corpora and adaptive draft length

published

2026-10-01

The speedup is directly relevant to a single-GPU batch-1 deployment and requires no model training, but it is specific to retrieval-heavy coding-agent workloads and demands infrastructure work whose payoff must be validated on the reader's own pipeline.

1. claim 01Paperraft inference
   Read the brief interpretation

   AgSpec replaces learned-draft speculative decoders (e.g., EAGLE-3) and generic retrieval drafters by retrieving token continuations from session-trajectory, workspace, and global corpora indexed in the agent's emission format, with per-agent draft lengths capped by offline profiling and adapted online from verification feedback. It costs the engineering of building and maintaining three corpora plus offline profiling of draft-length caps per agent, and adds retrieval and verification overhead per step, though it avoids training a draft model entirely. Gains depend on the agent actually repeating prior text; they shrink on workloads with little token reuse, and a stale or misformatted corpus or a miscalibrated draft-length cap can degrade throughput below the autoregressive baseline.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Raises generation throughput over autoregressive decoding up to 4.37x at batch size 1 and 4.76x at batch size 16, outperforming five retrieval-based drafters and EAGLE-3 in most evaluated settings on repository-level multi-agent coding benchmarks.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source