DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch

Long customer conversations exhaust memory or limit concurrent requests.

What you would change

Change only the attention cache policy, after checking model and runtime support.

Change only the attention cache policy, after checking model and runtime support.

Compare against

The same model with its ordinary attention cache.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak cache memory, long-context answer quality, first-token latency, and requests completed.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

KV cache

technical change

Predictive KV prefetch via dual-token decoding

published

2026-07-29

The throughput gain is specific to high-concurrency long-context serving with host-memory KV offload, so the reader should validate on their own single-GPU workload before adopting, ideally once a maintained implementation is available.

1. claim 01Paperraft inference
   Read the brief interpretation

   DualDecoder replaces existing sparse KV cache offloading systems (which keep large auxiliary retrieval-management states in GPU memory) by predicting the critical KV entries from the preceding speculated token and prefetching them from host memory via a dual-token decoding pipeline, layer-aware transfer schedule, and layer-scoped memory manager. The cost is added serving-stack complexity: a speculative decoding pipeline tightly coupled with PCIe transfer scheduling and custom memory management, with implementation effort that is non-trivial and not yet a standard library feature. It can fail if prefetch prediction accuracy degrades on workloads unlike the evaluated ones, if host-to-GPU bandwidth is insufficient to overlap transfers with compute, or if the auxiliary-state savings do not materialize at the reader's lower concurrency levels.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Improves decoding throughput by up to 2.62x over state-of-the-art sparse KV cache offloading systems while preserving decoding latency and model quality.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source