Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families

A self-hosted assistant is too expensive because its model uses most of the GPU memory.

What you would change

Try the paper's supported lower-precision weights on a copy of your serving setup.

Try the paper's supported lower-precision weights on a copy of your serving setup.

Compare against

The same model at its original precision, on the same GPU.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Peak memory, task pass rate, latency, and total serving cost at the same context and load.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

quantization

technical change

Weight-only post-training quantization for retrieval embedders, with task-specific distillation as the alternative at extreme widths

published

2026-09-14

An engineer on a 24 GB GPU running production retrieval can apply INT4 PTQ safely, but at extreme widths the paper's measured evidence favors distilling a small student over heuristically quantizing the teacher, provided the distillation task matches production traffic.

1. claim 01Paperraft inference
   Read the brief interpretation

   For retrieval embedders below INT4, a small task-distilled student quantized to INT3 replaces aggressive quantization of the larger teacher, rather than applying LLM-carried heuristics (embedding-table protection, module-wise bit allocation, ranking-aware objectives), which the paper shows do not transfer to embedders. The cost is a distillation pipeline and task-representative training data, plus the engineering of validating each (family, width, group-size) point instead of relying on a reusable sensitivity ordering; reconstruction-error proxies remain usable for screening uniform widths but not for selecting protected tensors. It can fail when the deployment task drifts from the distillation task, where the student's advantage over the quantized teacher does not hold, and joint damage at INT2-INT3 is not predictable from isolated module measurements.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   A distilled 109M student at INT3 holds 78.04 NDCG@10 in 68.4 MB, versus its 0.6B teacher under extreme PTQ at 64.46 NDCG@10 in 297.9 MB, dominating on both size and quality within the distillation task.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source