Read What Matters: Query-Adaptive Quantization for KV Caches

Long customer conversations exhaust memory or limit concurrent requests.

What you would change

Change only the attention cache policy, after checking model and runtime support.

Change only the attention cache policy, after checking model and runtime support.

Compare against

The same model with its ordinary attention cache.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak cache memory, long-context answer quality, first-token latency, and requests completed.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

KV cache

technical change

Query-adaptive progressive KV-cache quantization

published

2026-10-08

The method directly targets the dominant cost in long-context decoding on a single GPU—cache bytes moved per step—with negligible perplexity impact, but the latency evidence is narrow (one workload, batch one, single layer), so the reader should validate it on their own serving stack before committing to production.

1. claim 01Paperraft inference
   Read the brief interpretation

   ReadKV replaces uniform per-token KV-cache quantization (fixed-bit stored-and-read schemes such as 4-bit caches or TurboQuant-style codecs) with a progressive code: more bits are retained than are fetched per query, and each decoding query adaptively selects which key-channel and value-token prefixes to read. The cost is retained cache capacity roughly double the payload actually read (eight stored bits versus four read), plus added per-query allocation logic and calibration of distortion objectives, which increases implementation complexity over a static codec. It can fail if the per-query prefix-selection overhead outweighs read savings at small batch or short context, if real kernels do not expose the benchmarked latency gains outside the tested single-layer A10G setup, or if attention-guided allocation misjudges important entries on out-of-distribution long-context workloads.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   On an 8K-token, batch-one, single-layer A10G workload, an eight-bit ReadKV reader with a two-bit mean payload-read budget has 39% lower latency than the tested TurboQuant codec; reading four bits on average from an eight-bit cache raises C4 perplexity by at most 0.66% across six models.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source