On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics

A narrow, repeated product task may not need your largest model.

What you would change

Check whether a released student model fits the task before generating training data.

Check whether a released student model fits the task before generating training data.

Compare against

Your current model and an off-the-shelf small model on the same held-out tasks.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Student quality, serving cost, teacher calls, training cost, and failures on unusual inputs.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

distillation

technical change

Controlled on/off-policy distillation via token-level KL

published

2026-09-28

For a constrained team doing distillation fine-tuning, the actionable finding is that cheap off-policy forward-KL distillation is robust and learning rate tuning matters more than expensive on-policy rollouts, but the recommendation to validate on their own tasks makes this a workload-specific test rather than blind adoption.

1. claim 01Paperraft inference
   Read the brief interpretation

   This is not a new algorithm but a systematic comparison that replaces the default assumption that on-policy rollouts are inherently preferable for distillation; it shows token-level KL direction and learning rate, not rollout policy, are the dominant levers for performance, coverage, and forgetting. It costs no additional infrastructure beyond standard SFT/distillation on a single GPU, but requires deliberate selection of KL direction and learning rate rather than pipeline defaults. It can fail if reverse KL is paired with off-policy (teacher-generated) rollouts, which the paper shows is a substantially more sensitive combination, or if gains expected from on-policy data are assumed to survive later RLVR stages.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Forward KL gives stable, strong task performance across rollout policies; on-policy data improves generalisation to harder Countdown variants under both KL directions, though the advantage does not reliably persist after subsequent RLVR.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source