How Divergence Becomes Decision Flips in Compressed Language Models

A self-hosted assistant is too expensive because its model uses most of the GPU memory.

What you would change

Try the paper's supported lower-precision weights on a copy of your serving setup.

Try the paper's supported lower-precision weights on a copy of your serving setup.

Compare against

The same model at its original precision, on the same GPU.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak memory, task pass rate, latency, and total serving cost at the same context and load.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

adopt

research area

quantization

technical change

Total variation as flip-rate proxy for compressed models

published

2026-09-30

For a team shipping compressed models with a fixed GPU budget, swapping KL for total variation in compression reports is a zero-cost measurement change that directly predicts decision flips and speculative-decoding acceptance, with the caveat that the ratio must be rechecked on out-of-domain data such as code.

1. claim 01Paperraft inference
   Read the brief interpretation

   Replaces KL divergence as the reported and decision-relevant distance between a compressed model and its dense reference when what matters is how many arg-max decisions flip. Costs nothing beyond computing per-token total variation (or a first-order statistic such as Hellinger distance) over a sample of outputs, which is feasible on a single GPU; no retraining or calibration is required. Can fail when transferring across distributions unlike the validation corpora: the ratio fell below one on a held-out code corpus for two of three newly tested models, so per-domain verification on the deployment data is still needed.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Flip rate tracks total variation at a ratio with median 1.05 across 802 model copies; KL misranks compressors whose flip rates differ by >=10% in 11% of cases versus 1% for total variation; in vLLM speculative decoding, teacher-forced total variation predicts greedy draft acceptance with 1.1-2.4% mean relative error without task-specific calibration.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source