QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs

A self-hosted assistant is too expensive because its model uses most of the GPU memory.

What you would change

Try the paper's supported lower-precision weights on a copy of your serving setup.

Try the paper's supported lower-precision weights on a copy of your serving setup.

Compare against

The same model at its original precision, on the same GPU.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak memory, task pass rate, latency, and total serving cost at the same context and load.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

quantization

technical change

Deployment-aligned quantization-aware distillation for FP4 formats

published

2026-09-30

The framework runs on BF16 hardware without FP4 Tensor Cores and supports LoRA-based training, making small-model QAD plausibly feasible on one 24 GB GPU, but the quality gain over PTQ is workload- and format-dependent and must be validated on the reader's own model before production adoption.

1. claim 01Paperraft inference
   Read the brief interpretation

   The method replaces post-training quantization (PTQ) with quantization-aware distillation that simulates the deployment format while computing in BF16, so no FP4-native training hardware is needed and checkpoints export directly to vLLM or llama.cpp. The cost is a training run with a teacher model, a token budget that favors long sequences, and format-dependent hyperparameter choices (weights-only fake quantization for NVFP4 versus weights-plus-activations for MXFP4), plus engineering complexity beyond a one-shot PTQ pass. It can fail if the 24 GB GPU cannot hold even a LoRA-based QAD setup for the target model, if the chosen training configuration mismatches the deployment format, or if the quality gain does not justify training cost for models where PTQ is already acceptable.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   On Qwen3.5-9B, QAD reaches 68.9% average accuracy under NVFP4 and 66.0% under MXFP4, versus best PTQ results of 65.4% and 56.4%; longer 32K training sequences add 1.9 points over 4K sequences at equal token budget.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source