CacheReforge: Bounded Recovery for Stale KV Caches under Evolving Adapters

Long customer conversations exhaust memory or limit concurrent requests.

What you would change

Change only the attention cache policy, after checking model and runtime support.

Change only the attention cache policy, after checking model and runtime support.

Compare against

The same model with its ordinary attention cache.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak cache memory, long-context answer quality, first-token latency, and requests completed.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

KV cache

technical change

Version-aware bounded recovery of stale KV caches

published

2026-09-25

The technique is directly implementable on a single GPU serving adapter-based models, but its benefit only materializes if the reader's production workload involves frequently updated adapters over long cached contexts, so workload-specific validation is required before adoption.

1. claim 01Paperraft inference
   Read the brief interpretation

   CacheReforge replaces the binary choice between reusing stale KV caches after a LoRA adapter update (which distorts outputs) and recomputing the full affected suffix, by treating caches as layerwise mixed-version objects and selecting per-layer reuse, bounded recomputation, or full suffix recovery via sensitivity calibration and drift tracking. It costs additional bookkeeping: per-layer adapter anchors, calibrated sensitivity scores, accumulated drift state, and executable restart boundaries must be maintained and computed, and a small fraction of layers (5.44% in the reported setting) is still recomputed. It can fail when cumulative tail influence is underestimated, causing bounded recovery to stop short of the true functional recomputation horizon and silently degrade fidelity, and its calibration may not transfer to other model families, adapters, or update schedules.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Reduces mean KL divergence by 92.4% relative to stale cache reuse while recomputing only 5.44% of layers and cutting cache-maintenance time by 93.2% relative to fresh full prefill, on Qwen2.5-1.5B/7B with continual LoRA updates and 16K-context HotpotQA/2WikiMQA workloads.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source