HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

Long customer conversations exhaust memory or limit concurrent requests.

What you would change

Change only the attention cache policy, after checking model and runtime support.

Change only the attention cache policy, after checking model and runtime support.

Compare against

The same model with its ordinary attention cache.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak cache memory, long-context answer quality, first-token latency, and requests completed.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

watch

research area

KV cache

technical change

Hybrid sparse attention with two-level KV sharing

published

2026-09-22

The method is an architectural redesign validated only on an internally trained 80B MoE model, so a team without foundation-model training capacity cannot adopt it and should wait for released checkpoints or API exposure.

1. claim 01Paperraft inference
   Read the brief interpretation

   HySparse2 replaces standard full-attention KV caching with a YOCO-style self-decoder/cross-decoder split in which all cross-decoder KV caches are derived from self-decoder hidden states, combining sliding-window attention, token-level sparse selection with a forced recent-token window, and KV reuse across layers. The cost is a bespoke architecture requiring training from scratch or substantial retraining, plus added implementation complexity in the sparse-selection and bridging paths, with potential retrieval quality risk from token-level sparsity and forced-window heuristics. It can fail for anyone without the means to pretrain an 80B-scale MoE, since the gains depend on the co-designed architecture rather than a drop-in inference-time modification.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   The authors report that on an 80B-A3B MoE model, HySparse2 outperforms HySparse and Hybrid SWA on long-context retrieval and multi-turn agentic tasks while substantially reducing prefill computation and KV-cache storage; no specific figures appear in the abstract.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source