Emergent Structure in the Marginal Attention Space of Language Models

Long customer conversations exhaust memory or limit concurrent requests.

What you would change

Change only the attention cache policy, after checking model and runtime support.

Change only the attention cache policy, after checking model and runtime support.

Compare against

The same model with its ordinary attention cache.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak cache memory, long-context answer quality, first-token latency, and requests completed.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

KV cache

technical change

Offline per-head KV cache eviction budgets

published

2026-10-02

It is training-free, requires only a single offline pass on one GPU, and fits constrained infrastructure, but the gain is asserted as benchmark competitiveness rather than a quantified improvement, so workload-specific validation against existing eviction baselines is warranted before production adoption.

1. claim 01Paperraft inference
   Read the brief interpretation

   Replaces per-document recomputation or per-target training of KV cache eviction budgets with a per-head budget derived from marginal attention statistics, precomputed once offline on pretraining text, paired with a training-free token-importance score. Costs one offline profiling pass over representative text plus storage of a small per-head budget vector, with no fine-tuning or auxiliary models; inference-time overhead is limited to token scoring. Can fail when deployment text differs substantially from the pretraining corpus used for profiling, since the token-wise signal is text-intrinsic but the head-wise budget is estimated from a specific distribution; competitiveness is claimed only on standard eviction benchmarks and may not transfer to very long contexts or non-standard workloads.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   A per-head eviction budget precomputed offline on pretraining text, combined with a training-free token score, is reported as competitive with methods that recompute the budget per document or train it per target; no quantitative factor is given in the abstract.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source