The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems

Long customer conversations exhaust memory or limit concurrent requests.

What you would change

Change only the attention cache policy, after checking model and runtime support.

Change only the attention cache policy, after checking model and runtime support.

Compare against

The same model with its ordinary attention cache.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak cache memory, long-context answer quality, first-token latency, and requests completed.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

watch

research area

KV cache

technical change

Online KV cache working-set estimation via Mattson stack algorithm

published

2026-09-23

Working-set capacity planning is a real concern for multi-tenant or agentic serving, but on a single 24 GB GPU the cache capacity decision is trivial and the tool only pays off at serving scales the reader does not operate.

1. claim 01Paperraft inference
   Read the brief interpretation

   KVSET replaces brute-force capacity-by-capacity simulation of KV prefix caches with a single-pass Mattson stack analysis that computes hit rates across all candidate capacities from LRU stack distances. The cost is modest: it runs as an online analyzer over request traces with low compute and memory overhead, and an open-source implementation supports both live traffic and offline replay, but it presumes an existing serving stack with paged KV caching and trace instrumentation. It can fail when production traffic drifts from observed traces, when workloads lack the prefix reuse patterns the analysis assumes, or when eviction policy in the deployed cache deviates from the LRU model the stack algorithm encodes.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   The paper claims KVSET estimates closely match measurements from real cache deployments and that it substantially reduces the computational and memory overhead of capacity-by-capacity simulation, but reports no quantified multiplicative improvement factor in the abstract.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source