Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

You already need task-specific training and want to reduce its memory or compute cost.

What you would change

Change only the training technique on a small controlled run after checking prerequisites.

Change only the training technique on a small controlled run after checking prerequisites.

Compare against

Your current training recipe with the same model, data split, and evaluation.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Held-out quality, peak memory, training time, and full compute cost to the target quality.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

efficient training

technical change

Encoder adapter pre-training plus LoRA joint fine-tuning for ASR

published

2026-09-16

The two-stage adapter recipe is feasible on a single 24 GB GPU for encoder-scale models and addresses a real failure mode (encoder under-adaptation under domain shift), but the quality gain is unquantified in the abstract and only relevant if the reader actually deploys Speech-LLM ASR on shifted acoustic domains.

1. claim 01Paperraft inference
   Read the brief interpretation

   It replaces direct joint fine-tuning of a Speech-LLM (or LoRA-only LLM adaptation) with a two-stage procedure: first train lightweight adapters inserted into each frozen speech-encoder layer, then jointly fine-tune the whole model with LoRA on the LLM. The cost is an extra training stage, modest additional adapter parameters in the encoder, and a more complex pipeline than a single LoRA pass; it presumes an existing Speech-LLM checkpoint and target-domain labeled audio. It can fail if the target domain is not acoustic-shifted (e.g., the gap is lexical or semantic rather than child/dialectal acoustics), if very limited target data overfits the encoder adapters, or if the base model is not an encoder-projector-LLM ASR stack, making the method inapplicable.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   EAVA consistently outperforms vanilla fine-tuning and other baselines, achieving new state-of-the-art performance on three domain-shifted ASR datasets (child and dialectal speech); no numeric margin is stated in the abstract.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source