Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

feat(nano): support LoRA finetuning of the Qwen3 LLM (#3456)

张思远 (ZhangSiYuan)0738581f4b3e
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md b/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md
index 7354b42e6..62a141981 100644
--- a/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md
+++ b/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md
@@ -93,6 +93,46 @@ bash finetune.sh
- For training data less than 5000 hours, it is recommended to fine-tune the audio_encoder and audio_adaptor.
- For training data greater than 10000 hours, it is recommended to perform full-parameter fine-tuning.
+## LoRA Finetune
+
+As an alternative to full-parameter / partial fine-tuning, you can LoRA-tune the
+Qwen3-0.6B LLM (attaching adapters to its `q_proj` / `v_proj` linear layers):
+
+```
+bash lora_finetune.sh
+```
+
+What it sets:
+
+- `llm_conf.use_lora=true` — injects `LoRALinear` adapters into the target layers
+ of the LLM (base weights shared and frozen, trainable `lora_A` / `lora_B` added).
+- `lora_only=true` — freezes every non-LoRA parameter (audio encoder, adaptor and
+ CTC decoder), so only the adapters train. Checkpoints still save the full state
+ dict (base weights unchanged + adapter params), so you can resume or decode with
+ the same `use_lora=true` config.
+- `llm_conf.freeze=true` — keeps the LLM base weights frozen (redundant with
+ `lora_only`, but explicit).
+
+LoRA hyper-parameters live under `llm_conf.lora_conf` (defaults ship with the
+model config): `r`, `lora_alpha`, `lora_dropout`, `target_modules`. Override them
+on the command line, e.g.:
+
+```
+++llm_conf.lora_conf.r=32
+```
+
+To keep the audio encoder / adaptor **trainable** while LoRA-tweaking only the LLM
+(a good middle ground for a few hundred hours of domain data), set
+`lora_only=false`, `audio_encoder_conf.freeze=false` and
+`audio_adaptor_conf.freeze=false`; the LLM base weights stay frozen via
+`llm_conf.freeze=true`.
+
+**Decoding / deployment:** the fine-tuned checkpoint works with `decode.py` below
+without any merge step — the forward pass adds the adapter output on top of the
+base weights. To fold the adapters into the base weights for a standalone
+deployment checkpoint, compute `W' = W + (lora_alpha / r) * lora_B @ lora_A` for
+each target module and drop the `lora_A` / `lora_B` keys.
+
## Model Evaluation
After model fine-tuning is completed, you can decode the model using the decode.py script:
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune_zh.md b/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune_zh.md
index bed08cbd0..4a076f20a 100644
--- a/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune_zh.md
+++ b/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune_zh.md
@@ -93,6 +93,36 @@ bash finetune.sh
- 训练数据少于 5000 小时,建议微调 audio_encoder 和 audio_adaptor
- 训练数据大于 10000 小时,建议全量参数微调
+## LoRA 微调
+
+作为全量/部分微调的替代方案,可以对 Qwen3-0.6B LLM 做 LoRA 微调(在其
+`q_proj` / `v_proj` 线性层上挂适配器):
+
+```
+bash lora_finetune.sh
+```
+
+关键参数说明:
+
+- `llm_conf.use_lora=true` — 在 LLM 目标层注入 `LoRALinear` 适配器(基座权重共享并冻结,新增可训练的 `lora_A` / `lora_B`)。
+- `lora_only=true` — 冻结所有非 LoRA 参数(音频编码器、适配器、CTC 解码器),只训练适配器。checkpoint 仍保存完整 state dict(基座权重不变 + 适配器参数),因此用相同的 `use_lora=true` 配置即可续训或解码。
+- `llm_conf.freeze=true` — 保持 LLM 基座权重冻结(与 `lora_only` 重复,但更显式)。
+
+LoRA 超参数位于 `llm_conf.lora_conf` 下(模型配置已自带默认值):
+`r`、`lora_alpha`、`lora_dropout`、`target_modules`。可在命令行覆盖,例如:
+
+```
+++llm_conf.lora_conf.r=32
+```
+
+如果希望在只对 LLM 做 LoRA 的同时**保持音频编码器/适配器可训练**(对数百小时领域数据是个不错的折中),
+可设置 `lora_only=false`、`audio_encoder_conf.freeze=false`、`audio_adaptor_conf.freeze=false`;
+LLM 基座权重仍通过 `llm_conf.freeze=true` 保持冻结。
+
+**解码 / 部署:** 微调后的 checkpoint 可直接用下文的 `decode.py` 解码,无需合并步骤——
+前向时在基座输出上叠加适配器输出。若要把适配器折叠进基座权重以得到独立部署的 checkpoint,
+对每个目标模块计算 `W' = W + (lora_alpha / r) * lora_B @ lora_A`,并删除 `lora_A` / `lora_B` 键即可。
+
## 模型评测
当模型微调结束后,可以使用 decode.py 脚本对模型进行解码:
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/lora_finetune.sh b/examples/industrial_data_pretraining/fun_asr_nano/lora_finetune.sh
new file mode 100644
index 000000000..083523d8a
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/lora_finetune.sh
@@ -0,0 +1,82 @@
+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
+# MIT License (https://opensource.org/licenses/MIT)
+
+workspace=`pwd`
+
+# which gpu to train or finetune
+export CUDA_VISIBLE_DEVICES="0"
+gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
+
+# model_name from model_hub, or model_dir in local path
+model_name_or_model_dir="FunAudioLLM/Fun-ASR-Nano-2512"
+
+# data dir, which contains: train.json, val.json
+train_data=${workspace}/data/train_example.jsonl
+val_data=${workspace}/data/val_example.jsonl
+
+# exp output dir
+output_dir="./outputs_lora"
+log_file="${output_dir}/log.txt"
+
+deepspeed_config=${workspace}/deepspeed_conf/ds_stage1.json
+
+mkdir -p ${output_dir}
+echo "log_file: ${log_file}"
+
+# LoRA hyper-parameters. The defaults below match the values shipped in the
+# model config (`llm_conf.lora_conf`); override with ++key=value if needed.
+# --llm_conf.lora_conf.r=16
+# --llm_conf.lora_conf.lora_alpha=32
+# --llm_conf.lora_conf.lora_dropout=0.05
+# --llm_conf.lora_conf.target_modules="[q_proj, v_proj]"
+#
+# Freeze semantics with LoRA:
+# * llm_conf.freeze=true keeps the Qwen3 base weights frozen; only the
+# injected lora_A / lora_B are trainable inside the LLM.
+# * lora_only=true additionally freezes every non-LoRA parameter
+# (audio_encoder / audio_adaptor / ctc_decoder), i.e. pure-LoRA training.
+# * To keep the encoder/adaptor trainable while LoRA-tweaking only the LLM,
+# set lora_only=false and flip the corresponding *_conf.freeze to false.
+
+DISTRIBUTED_ARGS="
+ --nnodes ${WORLD_SIZE:-1} \
+ --nproc_per_node $gpu_num \
+ --node_rank ${RANK:-0} \
+ --master_addr ${MASTER_ADDR:-127.0.0.1} \
+ --master_port ${MASTER_PORT:-26669}
+"
+echo $DISTRIBUTED_ARGS
+
+# funasr trainer path
+train_tool=`which funasr-train-ds`
+echo "Using funasr trainer: ${train_tool}"
+
+torchrun $DISTRIBUTED_ARGS \
+${train_tool} \
+++model="${model_name_or_model_dir}" \
+++trust_remote_code=true \
+++train_data_set_list="${train_data}" \
+++valid_data_set_list="${val_data}" \
+++dataset_conf.data_split_num=1 \
+++dataset_conf.batch_sampler="BatchSampler" \
+++dataset_conf.batch_size=6000 \
+++dataset_conf.sort_size=1024 \
+++dataset_conf.batch_type="token" \
+++dataset_conf.num_workers=4 \
+++train_conf.max_epoch=50 \
+++train_conf.log_interval=1 \
+++train_conf.resume=true \
+++train_conf.validate_interval=2000 \
+++train_conf.save_checkpoint_interval=2000 \
+++train_conf.effective_save_name_excludes="None" \
+++train_conf.keep_nbest_models=20 \
+++train_conf.avg_nbest_model=10 \
+++train_conf.use_deepspeed=false \
+++train_conf.deepspeed_config=${deepspeed_config} \
+++optim_conf.lr=0.0001 \
+++llm_conf.use_lora=true \
+++lora_only=true \
+++llm_conf.freeze=true \
+++audio_encoder_conf.freeze=true \
+++audio_adaptor_conf.freeze=true \
+++output_dir="${output_dir}" &> ${log_file}
diff --git a/funasr/models/fun_asr_nano/model.py b/funasr/models/fun_asr_nano/model.py
index 108392137..609fe1cb9 100644
--- a/funasr/models/fun_asr_nano/model.py
+++ b/funasr/models/fun_asr_nano/model.py
@@ -129,6 +129,10 @@ class FunASRNano(nn.Module):
self.llm = model.to(dtype_map[self.llm_dtype])
llm_dim = model.get_input_embeddings().weight.shape[-1]
+ # lora: inject LoRA adapters into the LLM target Linear layers
+ if self.llm is not None and llm_conf.get("use_lora", False):
+ self._apply_lora_to_llm(llm_conf)
+
# adaptor
adaptor_class = tables.adaptor_classes.get(audio_adaptor)
if audio_encoder_output_size > 0:
@@ -206,6 +210,75 @@ class FunASRNano(nn.Module):
rank = int(os.environ.get("RANK", 0))
logging.info(f"rank: {rank}, model is builded.")
+ def _apply_lora_to_llm(self, llm_conf: dict):
+ """Replace the LLM target Linear layers with LoRA adapters.
+
+ When ``llm_conf.use_lora`` is true, every ``nn.Linear`` in the LLM whose
+ module name contains one of ``lora_conf.target_modules`` is swapped for a
+ ``lora.Linear`` (base weight shared and frozen, trainable ``lora_A`` /
+ ``lora_B`` added). The base weights stay untouched in the state dict, so a
+ LoRA checkpoint can be loaded back into a model built with the same
+ ``lora_conf``, or folded for deployment (W' = W + alpha/r * B @ A).
+
+ Frozen-base behaviour: the LLM is frozen by ``llm_conf.freeze``; the
+ ``lora_A``/``lora_B`` parameters created here are trainable regardless.
+ With ``lora_only: true`` in the training config, ``mark_only_lora_as_trainable``
+ additionally freezes every non-LoRA parameter (encoder/adaptor/CTC), giving
+ pure-LoRA training. To keep the encoder/adaptor trainable while LoRA-tweaking
+ only the LLM, set ``lora_only: false`` and unfreeze them via their conf.
+ """
+ lora_conf = llm_conf.get("lora_conf", {})
+ lora_r = lora_conf.get("r", 16)
+ lora_alpha = lora_conf.get("lora_alpha", 32)
+ lora_dropout = lora_conf.get("lora_dropout", 0.05)
+ target_modules = lora_conf.get("target_modules", ["q_proj", "v_proj"])
+
+ from funasr.models.lora.layers import Linear as LoRALinear
+
+ lora_applied = 0
+ for name, module in list(self.llm.named_modules()):
+ if not isinstance(module, nn.Linear):
+ continue
+ if not any(target in name.split(".") for target in target_modules):
+ continue
+ parts = name.split(".")
+ parent = self.llm
+ for p in parts[:-1]:
+ parent = getattr(parent, p)
+ new_linear = LoRALinear(
+ in_features=module.in_features,
+ out_features=module.out_features,
+ r=lora_r,
+ lora_alpha=lora_alpha,
+ lora_dropout=lora_dropout,
+ bias=module.bias is not None,
+ # keep the adapter params in the base weight's dtype (e.g. bf16),
+ # so the LoRA path does not depend on autocast to reconcile dtypes
+ dtype=module.weight.dtype,
+ )
+ # share (and keep frozen) the pretrained base weight
+ new_linear.weight = module.weight
+ new_linear.bias = module.bias
+ setattr(parent, parts[-1], new_linear)
+ lora_applied += 1
+
+ if lora_applied > 0:
+ logging.info(
+ "LoRA applied to %d Linear layers in the LLM "
+ "(r=%d, alpha=%d, dropout=%s, targets=%s)",
+ lora_applied,
+ lora_r,
+ lora_alpha,
+ lora_dropout,
+ target_modules,
+ )
+ else:
+ logging.warning(
+ "use_lora=true but no target modules found in the LLM "
+ "(target_modules=%s)",
+ target_modules,
+ )
+
def on_pretrained_model_loaded(self, loaded_keys):
"""Fail closed when a checkpoint configures CTC without trained weights."""
loaded_keys = set(loaded_keys).union(getattr(self, "_externally_loaded_ctc_keys", ()))
diff --git a/tests/test_fun_asr_nano_lora_injection.py b/tests/test_fun_asr_nano_lora_injection.py
new file mode 100644
index 000000000..e56a160e0
--- /dev/null
+++ b/tests/test_fun_asr_nano_lora_injection.py
@@ -0,0 +1,256 @@
+"""Offline unit tests for Fun-ASR-Nano LoRA injection.
+
+The Qwen3 LLM is far too heavy to construct in a unit test, so these tests
+drive the real injection method ``FunASRNano._apply_lora_to_llm`` against a
+tiny fake LLM that mirrors the Qwen3 parameter layout
+(``layers.<i>.self_attn.{q,k,v,o}_proj`` and ``layers.<i>.mlp.{gate,up,down}_proj``).
+No model download is needed.
+
+The fake LLM is frozen (``requires_grad=False`` + ``eval()``) *before*
+injection, exactly mirroring the ``llm_conf.freeze`` (default true) step that
+``FunASRNano.__init__`` runs prior to calling ``_apply_lora_to_llm``.
+
+Coverage requested in review (modelscope/FunASR#3456):
+ 1. only configured target modules are replaced;
+ 2. the initial output matches the original linear layers;
+ 3. base weights stay frozen while ``lora_A``/``lora_B`` remain trainable
+ with the documented configuration;
+ 4. a non-zero adapter survives a state-dict save/load round trip and
+ train/eval transitions;
+ 5. the no-matching-target configuration has an explicit tested outcome.
+"""
+
+import os
+
+import torch
+import torch.nn as nn
+
+from funasr.models.fun_asr_nano.model import FunASRNano
+from funasr.models.lora.layers import Linear as LoRALinear
+
+HIDDEN = 8
+N_LAYERS = 2
+
+DEFAULT_CONF = {"r": 4, "lora_alpha": 8, "lora_dropout": 0.05,
+ "target_modules": ["q_proj", "v_proj"]}
+
+
+class _FakeQwen3(nn.Module):
+ """Minimal Qwen3-layout LLM: N layers x self_attn (q/k/v/o) + mlp (gate/up/down)."""
+
+ def __init__(self, hidden=HIDDEN, n_layers=N_LAYERS):
+ super().__init__()
+ self.embed_tokens = nn.Embedding(32, hidden)
+ layers = []
+ for _ in range(n_layers):
+ layer = nn.Module()
+ attn = nn.Module()
+ attn.q_proj = nn.Linear(hidden, hidden)
+ attn.k_proj = nn.Linear(hidden, hidden)
+ attn.v_proj = nn.Linear(hidden, hidden)
+ attn.o_proj = nn.Linear(hidden, hidden)
+ mlp = nn.Module()
+ mlp.gate_proj = nn.Linear(hidden, hidden)
+ mlp.up_proj = nn.Linear(hidden, hidden)
+ mlp.down_proj = nn.Linear(hidden, hidden)
+ layer.self_attn = attn
+ layer.mlp = mlp
+ layers.append(layer)
+ self.layers = nn.ModuleList(layers)
+ # mirror the pre-injection freeze applied by llm_conf.freeze (default true)
+ self.freeze()
+
+ def freeze(self):
+ for p in self.parameters():
+ p.requires_grad = False
+ self.eval()
+
+
+def _apply(fake_llm, lora_conf):
+ """Run the real injection method without building a full FunASRNano.
+
+ ``__new__`` + ``nn.Module.__init__`` initialize the module machinery while
+ skipping ``FunASRNano.__init__`` (which would try to download a real LLM).
+ """
+ inst = FunASRNano.__new__(FunASRNano)
+ nn.Module.__init__(inst)
+ inst.llm = fake_llm
+ inst._apply_lora_to_llm({"use_lora": True, "lora_conf": lora_conf})
+
+
+def _lora_layers(fake_llm):
+ return [m for m in fake_llm.modules() if isinstance(m, LoRALinear)]
+
+
+def test_only_configured_target_modules_replaced():
+ fake = _FakeQwen3()
+ _apply(fake, DEFAULT_CONF)
+
+ for layer in fake.layers:
+ # the two configured targets are LoRA adapters
+ assert isinstance(layer.self_attn.q_proj, LoRALinear)
+ assert isinstance(layer.self_attn.v_proj, LoRALinear)
+ # every other linear stays a plain nn.Linear
+ assert type(layer.self_attn.k_proj) is nn.Linear
+ assert type(layer.self_attn.o_proj) is nn.Linear
+ assert type(layer.mlp.gate_proj) is nn.Linear
+ assert type(layer.mlp.up_proj) is nn.Linear
+ assert type(layer.mlp.down_proj) is nn.Linear
+
+ # exactly 2 targets x N_LAYERS adapters, each carrying lora_A / lora_B
+ adapters = _lora_layers(fake)
+ assert len(adapters) == 2 * N_LAYERS
+ for m in adapters:
+ assert hasattr(m, "lora_A") and hasattr(m, "lora_B")
+
+
+def test_custom_target_modules_restrict_replacement():
+ fake = _FakeQwen3()
+ _apply(fake, {"r": 4, "lora_alpha": 8, "lora_dropout": 0.0,
+ "target_modules": ["o_proj"]})
+
+ for layer in fake.layers:
+ assert isinstance(layer.self_attn.o_proj, LoRALinear)
+ assert type(layer.self_attn.q_proj) is nn.Linear
+ assert type(layer.self_attn.v_proj) is nn.Linear
+ assert len(_lora_layers(fake)) == N_LAYERS
+
+
+def test_initial_output_matches_original_linear():
+ fake = _FakeQwen3()
+ x = torch.randn(3, HIDDEN)
+
+ with torch.no_grad():
+ ref = torch.stack(
+ [fake.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]
+ )
+
+ _apply(fake, DEFAULT_CONF)
+ # lora_B is initialized to zero, so the adapter contributes nothing and the
+ # shared base weight reproduces the original linear's output bit-for-bit.
+ with torch.no_grad():
+ after = torch.stack(
+ [fake.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]
+ )
+ assert torch.equal(after, ref)
+
+
+def test_base_frozen_adapters_trainable():
+ fake = _FakeQwen3() # frozen before injection, as llm_conf.freeze does
+ _apply(fake, DEFAULT_CONF)
+
+ for layer in fake.layers:
+ q = layer.self_attn.q_proj
+ # shared pretrained base stays frozen
+ assert q.weight.requires_grad is False
+ assert q.bias.requires_grad is False
+ # the new adapter parameters are trainable
+ assert q.lora_A.requires_grad is True
+ assert q.lora_B.requires_grad is True
+
+
+def test_adapter_dtype_matches_base_weight_dtype():
+ fake = _FakeQwen3().to(torch.bfloat16) # llm_dtype=bf16 on a real run
+ _apply(fake, DEFAULT_CONF)
+
+ for layer in fake.layers:
+ q = layer.self_attn.q_proj
+ assert q.weight.dtype is torch.bfloat16
+ assert q.lora_A.dtype is torch.bfloat16
+ assert q.lora_B.dtype is torch.bfloat16
+
+
+def test_state_dict_round_trip_and_train_eval_transitions(tmp_path):
+ lora_conf = {"r": 4, "lora_alpha": 8, "lora_dropout": 0.0,
+ "target_modules": ["q_proj", "v_proj"]}
+ fake = _FakeQwen3()
+ _apply(fake, lora_conf)
+
+ # force a non-zero adapter
+ with torch.no_grad():
+ for m in _lora_layers(fake):
+ m.lora_A.normal_()
+ m.lora_B.normal_(std=0.5)
+
+ x = torch.randn(3, HIDDEN)
+ with torch.no_grad():
+ train_out = torch.stack(
+ [fake.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]
+ )
+
+ ckpt_path = os.path.join(tmp_path, "llm_lora.pt")
+ torch.save(fake.state_dict(), ckpt_path)
+
+ # model.eval() / model.train() transitions never corrupt the adapter: the
+ # top-level eval() keeps the adapter in its unmerged state (nn.Module.eval
+ # dispatches through LoRALinear.train(False)), dropout is off in eval, and
+ # with dropout=0 the outputs are identical.
+ with torch.no_grad():
+ fake.eval()
+ eval_out = torch.stack(
+ [fake.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]
+ )
+ fake.train()
+ train_out2 = torch.stack(
+ [fake.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]
+ )
+ assert torch.allclose(eval_out, train_out2)
+ assert torch.equal(train_out, train_out2)
+
+ # The documented merge/unmerge convention on the adapter layer itself:
+ # eval() folds W' = W + alpha/r * B @ A into the shared base weight, train()
+ # unfolds it again, restoring the exact unmerged output.
+ q = fake.layers[0].self_attn.q_proj
+ with torch.no_grad():
+ q.eval()
+ assert q.merged is True
+ merged_out = q(x)
+ q.train()
+ assert q.merged is False
+ unmerged_out = q(x)
+ assert torch.allclose(merged_out, unmerged_out)
+ # the merge/unmerge round trip restores the unmerged behavior (weight is
+ # restored to ~1 ulp, not bit-exact: (w + delta) - delta rounds in IEEE)
+ assert torch.allclose(train_out[0], unmerged_out)
+
+ # a fresh model built with the same lora_conf loads the checkpoint; the
+ # non-zero adapter (and its forward behavior) survives the round trip.
+ fresh = _FakeQwen3()
+ _apply(fresh, lora_conf)
+ fresh.load_state_dict(torch.load(ckpt_path))
+ with torch.no_grad():
+ loaded_out = torch.stack(
+ [fresh.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]
+ )
+ assert torch.equal(loaded_out, train_out)
+
+
+def test_no_matching_target_is_explicit_noop():
+ import logging
+
+ fake = _FakeQwen3()
+ records = []
+
+ class _Collect(logging.Handler):
+ def emit(self, record):
+ records.append(record.getMessage())
+
+ handler = _Collect(level=logging.WARNING)
+ # model.py logs through module-level logging.warning -> the root logger
+ logging.getLogger().addHandler(handler)
+ try:
+ # must not raise; the configuration is accepted with zero adapters
+ _apply(fake, {"r": 4, "lora_alpha": 8, "lora_dropout": 0.0,
+ "target_modules": ["does_not_exist"]})
+ finally:
+ logging.getLogger().removeHandler(handler)
+
+ assert len(_lora_layers(fake)) == 0
+ for layer in fake.layers:
+ assert type(layer.self_attn.q_proj) is nn.Linear
+ assert type(layer.self_attn.v_proj) is nn.Linear
+ # the model still forwards
+ with torch.no_grad():
+ fake.layers[0].self_attn.q_proj(torch.randn(2, HIDDEN))
+ # the outcome is explicit: a warning documents that nothing was replaced
+ assert any("no target modules found" in r for r in records)