Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

Long customer conversations exhaust memory or limit concurrent requests.

What you would change

Change only the attention cache policy, after checking model and runtime support.

Change only the attention cache policy, after checking model and runtime support.

Compare against

The same model with its ordinary attention cache.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak cache memory, long-context answer quality, first-token latency, and requests completed.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

watch

research area

KV cache

technical change

Per-query bit-depth key scan for offloaded KV caches

published

2026-09-15

The method targets a bottleneck (host-memory scan traffic at million-token, multi-session scale) that a constrained-infrastructure practitioner is unlikely to hit, and it explicitly provides no speedup when the KV index fits in GPU memory, so adoption is premature until such workloads exist.

1. claim 01Paperraft inference
   Read the brief interpretation

   Fathom replaces fixed-width key scans (Double Sparsity, Loki, SparQ) in top-k sparse attention over host-memory-offloaded KV caches with a per-query variable bit-depth read over bit-plane-organized 4-bit keys, where a reverse water-filling rule allocates bits across channels by variance-weighted importance. It costs little in storage since it reuses the 4-bit K copy a quantized serving stack already holds, but adds channel-major bit-plane layout and per-query bit-budget computation to the decode path, and quality depends on the bit budget chosen. It can fail to deliver any benefit when the index is GPU-resident, as the authors state the method is not faster in that case, and its gains are demonstrated only at very long contexts with many concurrent sessions where host-memory scan traffic is the bottleneck.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   At one million tokens on Qwen3-8B, a decode step is 1.67x faster in GPU time than 136-bit scans (Double Sparsity, Loki, SparQ r=32); at SparQ's 68-bit read budget it reads 18% fewer bytes with lower attention error on six of seven settings, and matches the most accurate 136-bit scan's step agreement at 92 bits.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source