PatchKV: Weight-Space Compensation of KV Cache

Long customer conversations exhaust memory or limit concurrent requests.

What you would change

Change only the attention cache policy, after checking model and runtime support.

Change only the attention cache policy, after checking model and runtime support.

Compare against

The same model with its ordinary attention cache.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak cache memory, long-context answer quality, first-token latency, and requests completed.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

KV cache

technical change

Weight-space compensation for compressed KV caches

published

2026-09-30

The method is training-free and runs on a single GPU, making it feasible for the reader's infrastructure, but its benefit depends on a stable single-context multi-query workload and on validating the patch-swap overhead and quality recovery on the reader's specific compressor and tasks.

1. claim 01Paperraft inference
   Read the brief interpretation

   PatchKV does not replace KV cache compression; it augments an existing compressed cache (eviction or approximation based) with a context-specific weight patch computed once via closed-form ridge regression that aligns activations of reference queries between the full and compressed cache. Costs are a one-time per-context patch computation at context-loading time, storage of one patch per served context (or re-merge per context switch), and the engineering complexity of merging and unmerging weight deltas in the serving stack; per-query inference cost is claimed unchanged only in the single-context multi-query setting. It can fail when workloads serve many interleaved contexts (patch swapping overhead and weight mutation in a shared server), when reference queries poorly represent downstream queries, or at budgets where the base compressor's error exceeds what a low-rank-style closed-form correction can recover.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   The abstract reports that PatchKV consistently improves cache compression methods across long-context QA (SCBench up to 170K tokens, SQuAD, NIAH) and GSM8K on three architectures, with no single quantified accuracy or compression factor stated.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source