Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA

You already need task-specific training and want to reduce its memory or compute cost.

What you would change

Change only the training technique on a small controlled run after checking prerequisites.

Change only the training technique on a small controlled run after checking prerequisites.

Compare against

Your current training recipe with the same model, data split, and evaluation.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Held-out quality, peak memory, training time, and full compute cost to the target quality.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

efficient training

technical change

Freezing LoRA input factor A on top singular directions

published

2026-10-05

The finding is a cheap, infrastructure-compatible hyperparameter choice with reported double-digit gains at low rank, but the limited task coverage, rank-dependent attenuation, and weakened significance claims require validation on the reader's own workload before production adoption.

1. claim 01Paperraft inference
   Read the brief interpretation

   The paper studies which LoRA factor to freeze and on which singular band, replacing the default practice of fully training both A and B or adopting spectral-frozen variants such as MiCA without examining the factor choice; at low rank, freezing the input factor A on top singular directions is the better default. It costs nothing beyond standard LoRA: fewer trainable parameters, no extra memory or latency, only a subspace selection step at initialization. The advantage weakens as rank increases, the factor-versus-band ordering is descriptive rather than causal, an approximate multiplicity audit weakened several significance claims, and results cover only a handful of task-model pairs, so the benefit may not transfer to other models or tasks.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Freezing A outperforms freezing B by 8-18 percentage points at comparable trainable budgets on five task-model pairs, and PEFT's MiCA trails comparable-budget LoRA by 3.08 points on OpenBookQA/Qwen2.5-1.5B at rank 16 under a shared recipe.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source