Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
feat(nano): support LoRA finetuning of the Qwen3 LLM (#3456)
张思远 (ZhangSiYuan)0738581f4b3e
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md b/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.mdindex 7354b42e6..62a141981 100644--- a/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md+++ b/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md@@ -93,6 +93,46 @@ bash finetune.sh- For training data less than 5000 hours, it is recommended to fine-tune the audio_encoder and audio_adaptor.- For training data greater than 10000 hours, it is recommended to perform full-parameter fine-tuning.+## LoRA Finetune++As an alternative to full-parameter / partial fine-tuning, you can LoRA-tune the+Qwen3-0.6B LLM (attaching adapters to its `q_proj` / `v_proj` linear layers):++```+bash lora_finetune.sh+```++What it sets:++- `llm_conf.use_lora=true` — injects `LoRALinear` adapters into the target layers+ of the LLM (base weights shared and frozen, trainable `lora_A` / `lora_B` added).+- `lora_only=true` — freezes every non-LoRA parameter (audio encoder, adaptor and+ CTC decoder), so only the adapters train. Checkpoints still save the full state+ dict (base weights unchanged + adapter params), so you can resume or decode with+ the same `use_lora=true` config.+- `llm_conf.freeze=true` — keeps the LLM base weights frozen (redundant with+ `lora_only`, but explicit).++LoRA hyper-parameters live under `llm_conf.lora_conf` (defaults ship with the+model config): `r`, `lora_alpha`, `lora_dropout`, `target_modules`. Override them+on the command line, e.g.:++```+++llm_conf.lora_conf.r=32+```++To keep the audio encoder / adaptor **trainable** while LoRA-tweaking only the LLM+(a good middle ground for a few hundred hours of domain data), set+`lora_only=false`, `audio_encoder_conf.freeze=false` and+`audio_adaptor_conf.freeze=false`; the LLM base weights stay frozen via+`llm_conf.freeze=true`.++**Decoding / deployment:** the fine-tuned checkpoint works with `decode.py` below+without any merge step — the forward pass adds the adapter output on top of the+base weights. To fold the adapters into the base weights for a standalone+deployment checkpoint, compute `W' = W + (lora_alpha / r) * lora_B @ lora_A` for+each target module and drop the `lora_A` / `lora_B` keys.+## Model EvaluationAfter model fine-tuning is completed, you can decode the model using the decode.py script:diff --git a/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune_zh.md b/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune_zh.mdindex bed08cbd0..4a076f20a 100644--- a/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune_zh.md+++ b/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune_zh.md@@ -93,6 +93,36 @@ bash finetune.sh- 训练数据少于 5000 小时,建议微调 audio_encoder 和 audio_adaptor- 训练数据大于 10000 小时,建议全量参数微调+## LoRA 微调++作为全量/部分微调的替代方案,可以对 Qwen3-0.6B LLM 做 LoRA 微调(在其+`q_proj` / `v_proj` 线性层上挂适配器):++```+bash lora_finetune.sh+```++关键参数说明:++- `llm_conf.use_lora=true` — 在 LLM 目标层注入 `LoRALinear` 适配器(基座权重共享并冻结,新增可训练的 `lora_A` / `lora_B`)。+- `lora_only=true` — 冻结所有非 LoRA 参数(音频编码器、适配器、CTC 解码器),只训练适配器。checkpoint 仍保存完整 state dict(基座权重不变 + 适配器参数),因此用相同的 `use_lora=true` 配置即可续训或解码。+- `llm_conf.freeze=true` — 保持 LLM 基座权重冻结(与 `lora_only` 重复,但更显式)。++LoRA 超参数位于 `llm_conf.lora_conf` 下(模型配置已自带默认值):+`r`、`lora_alpha`、`lora_dropout`、`target_modules`。可在命令行覆盖,例如:++```+++llm_conf.lora_conf.r=32+```++如果希望在只对 LLM 做 LoRA 的同时**保持音频编码器/适配器可训练**(对数百小时领域数据是个不错的折中),+可设置 `lora_only=false`、`audio_encoder_conf.freeze=false`、`audio_adaptor_conf.freeze=false`;+LLM 基座权重仍通过 `llm_conf.freeze=true` 保持冻结。++**解码 / 部署:** 微调后的 checkpoint 可直接用下文的 `decode.py` 解码,无需合并步骤——+前向时在基座输出上叠加适配器输出。若要把适配器折叠进基座权重以得到独立部署的 checkpoint,+对每个目标模块计算 `W' = W + (lora_alpha / r) * lora_B @ lora_A`,并删除 `lora_A` / `lora_B` 键即可。+## 模型评测当模型微调结束后,可以使用 decode.py 脚本对模型进行解码:diff --git a/examples/industrial_data_pretraining/fun_asr_nano/lora_finetune.sh b/examples/industrial_data_pretraining/fun_asr_nano/lora_finetune.shnew file mode 100644index 000000000..083523d8a--- /dev/null+++ b/examples/industrial_data_pretraining/fun_asr_nano/lora_finetune.sh@@ -0,0 +1,82 @@+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.+# MIT License (https://opensource.org/licenses/MIT)++workspace=`pwd`++# which gpu to train or finetune+export CUDA_VISIBLE_DEVICES="0"+gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')++# model_name from model_hub, or model_dir in local path+model_name_or_model_dir="FunAudioLLM/Fun-ASR-Nano-2512"++# data dir, which contains: train.json, val.json+train_data=${workspace}/data/train_example.jsonl+val_data=${workspace}/data/val_example.jsonl++# exp output dir+output_dir="./outputs_lora"+log_file="${output_dir}/log.txt"++deepspeed_config=${workspace}/deepspeed_conf/ds_stage1.json++mkdir -p ${output_dir}+echo "log_file: ${log_file}"++# LoRA hyper-parameters. The defaults below match the values shipped in the+# model config (`llm_conf.lora_conf`); override with ++key=value if needed.+# --llm_conf.lora_conf.r=16+# --llm_conf.lora_conf.lora_alpha=32+# --llm_conf.lora_conf.lora_dropout=0.05+# --llm_conf.lora_conf.target_modules="[q_proj, v_proj]"+#+# Freeze semantics with LoRA:+# * llm_conf.freeze=true keeps the Qwen3 base weights frozen; only the+# injected lora_A / lora_B are trainable inside the LLM.+# * lora_only=true additionally freezes every non-LoRA parameter+# (audio_encoder / audio_adaptor / ctc_decoder), i.e. pure-LoRA training.+# * To keep the encoder/adaptor trainable while LoRA-tweaking only the LLM,+# set lora_only=false and flip the corresponding *_conf.freeze to false.++DISTRIBUTED_ARGS="+ --nnodes ${WORLD_SIZE:-1} \+ --nproc_per_node $gpu_num \+ --node_rank ${RANK:-0} \+ --master_addr ${MASTER_ADDR:-127.0.0.1} \+ --master_port ${MASTER_PORT:-26669}+"+echo $DISTRIBUTED_ARGS++# funasr trainer path+train_tool=`which funasr-train-ds`+echo "Using funasr trainer: ${train_tool}"++torchrun $DISTRIBUTED_ARGS \+${train_tool} \+++model="${model_name_or_model_dir}" \+++trust_remote_code=true \+++train_data_set_list="${train_data}" \+++valid_data_set_list="${val_data}" \+++dataset_conf.data_split_num=1 \+++dataset_conf.batch_sampler="BatchSampler" \+++dataset_conf.batch_size=6000 \+++dataset_conf.sort_size=1024 \+++dataset_conf.batch_type="token" \+++dataset_conf.num_workers=4 \+++train_conf.max_epoch=50 \+++train_conf.log_interval=1 \+++train_conf.resume=true \+++train_conf.validate_interval=2000 \+++train_conf.save_checkpoint_interval=2000 \+++train_conf.effective_save_name_excludes="None" \+++train_conf.keep_nbest_models=20 \+++train_conf.avg_nbest_model=10 \+++train_conf.use_deepspeed=false \+++train_conf.deepspeed_config=${deepspeed_config} \+++optim_conf.lr=0.0001 \+++llm_conf.use_lora=true \+++lora_only=true \+++llm_conf.freeze=true \+++audio_encoder_conf.freeze=true \+++audio_adaptor_conf.freeze=true \+++output_dir="${output_dir}" &> ${log_file}diff --git a/funasr/models/fun_asr_nano/model.py b/funasr/models/fun_asr_nano/model.pyindex 108392137..609fe1cb9 100644--- a/funasr/models/fun_asr_nano/model.py+++ b/funasr/models/fun_asr_nano/model.py@@ -129,6 +129,10 @@ class FunASRNano(nn.Module):self.llm = model.to(dtype_map[self.llm_dtype])llm_dim = model.get_input_embeddings().weight.shape[-1]+ # lora: inject LoRA adapters into the LLM target Linear layers+ if self.llm is not None and llm_conf.get("use_lora", False):+ self._apply_lora_to_llm(llm_conf)+# adaptoradaptor_class = tables.adaptor_classes.get(audio_adaptor)if audio_encoder_output_size > 0:@@ -206,6 +210,75 @@ class FunASRNano(nn.Module):rank = int(os.environ.get("RANK", 0))logging.info(f"rank: {rank}, model is builded.")+ def _apply_lora_to_llm(self, llm_conf: dict):+ """Replace the LLM target Linear layers with LoRA adapters.++ When ``llm_conf.use_lora`` is true, every ``nn.Linear`` in the LLM whose+ module name contains one of ``lora_conf.target_modules`` is swapped for a+ ``lora.Linear`` (base weight shared and frozen, trainable ``lora_A`` /+ ``lora_B`` added). The base weights stay untouched in the state dict, so a+ LoRA checkpoint can be loaded back into a model built with the same+ ``lora_conf``, or folded for deployment (W' = W + alpha/r * B @ A).++ Frozen-base behaviour: the LLM is frozen by ``llm_conf.freeze``; the+ ``lora_A``/``lora_B`` parameters created here are trainable regardless.+ With ``lora_only: true`` in the training config, ``mark_only_lora_as_trainable``+ additionally freezes every non-LoRA parameter (encoder/adaptor/CTC), giving+ pure-LoRA training. To keep the encoder/adaptor trainable while LoRA-tweaking+ only the LLM, set ``lora_only: false`` and unfreeze them via their conf.+ """+ lora_conf = llm_conf.get("lora_conf", {})+ lora_r = lora_conf.get("r", 16)+ lora_alpha = lora_conf.get("lora_alpha", 32)+ lora_dropout = lora_conf.get("lora_dropout", 0.05)+ target_modules = lora_conf.get("target_modules", ["q_proj", "v_proj"])++ from funasr.models.lora.layers import Linear as LoRALinear++ lora_applied = 0+ for name, module in list(self.llm.named_modules()):+ if not isinstance(module, nn.Linear):+ continue+ if not any(target in name.split(".") for target in target_modules):+ continue+ parts = name.split(".")+ parent = self.llm+ for p in parts[:-1]:+ parent = getattr(parent, p)+ new_linear = LoRALinear(+ in_features=module.in_features,+ out_features=module.out_features,+ r=lora_r,+ lora_alpha=lora_alpha,+ lora_dropout=lora_dropout,+ bias=module.bias is not None,+ # keep the adapter params in the base weight's dtype (e.g. bf16),+ # so the LoRA path does not depend on autocast to reconcile dtypes+ dtype=module.weight.dtype,+ )+ # share (and keep frozen) the pretrained base weight+ new_linear.weight = module.weight+ new_linear.bias = module.bias+ setattr(parent, parts[-1], new_linear)+ lora_applied += 1++ if lora_applied > 0:+ logging.info(+ "LoRA applied to %d Linear layers in the LLM "+ "(r=%d, alpha=%d, dropout=%s, targets=%s)",+ lora_applied,+ lora_r,+ lora_alpha,+ lora_dropout,+ target_modules,+ )+ else:+ logging.warning(+ "use_lora=true but no target modules found in the LLM "+ "(target_modules=%s)",+ target_modules,+ )+def on_pretrained_model_loaded(self, loaded_keys):"""Fail closed when a checkpoint configures CTC without trained weights."""loaded_keys = set(loaded_keys).union(getattr(self, "_externally_loaded_ctc_keys", ()))diff --git a/tests/test_fun_asr_nano_lora_injection.py b/tests/test_fun_asr_nano_lora_injection.pynew file mode 100644index 000000000..e56a160e0--- /dev/null+++ b/tests/test_fun_asr_nano_lora_injection.py@@ -0,0 +1,256 @@+"""Offline unit tests for Fun-ASR-Nano LoRA injection.++The Qwen3 LLM is far too heavy to construct in a unit test, so these tests+drive the real injection method ``FunASRNano._apply_lora_to_llm`` against a+tiny fake LLM that mirrors the Qwen3 parameter layout+(``layers.<i>.self_attn.{q,k,v,o}_proj`` and ``layers.<i>.mlp.{gate,up,down}_proj``).+No model download is needed.++The fake LLM is frozen (``requires_grad=False`` + ``eval()``) *before*+injection, exactly mirroring the ``llm_conf.freeze`` (default true) step that+``FunASRNano.__init__`` runs prior to calling ``_apply_lora_to_llm``.++Coverage requested in review (modelscope/FunASR#3456):+ 1. only configured target modules are replaced;+ 2. the initial output matches the original linear layers;+ 3. base weights stay frozen while ``lora_A``/``lora_B`` remain trainable+ with the documented configuration;+ 4. a non-zero adapter survives a state-dict save/load round trip and+ train/eval transitions;+ 5. the no-matching-target configuration has an explicit tested outcome.+"""++import os++import torch+import torch.nn as nn++from funasr.models.fun_asr_nano.model import FunASRNano+from funasr.models.lora.layers import Linear as LoRALinear++HIDDEN = 8+N_LAYERS = 2++DEFAULT_CONF = {"r": 4, "lora_alpha": 8, "lora_dropout": 0.05,+ "target_modules": ["q_proj", "v_proj"]}+++class _FakeQwen3(nn.Module):+ """Minimal Qwen3-layout LLM: N layers x self_attn (q/k/v/o) + mlp (gate/up/down)."""++ def __init__(self, hidden=HIDDEN, n_layers=N_LAYERS):+ super().__init__()+ self.embed_tokens = nn.Embedding(32, hidden)+ layers = []+ for _ in range(n_layers):+ layer = nn.Module()+ attn = nn.Module()+ attn.q_proj = nn.Linear(hidden, hidden)+ attn.k_proj = nn.Linear(hidden, hidden)+ attn.v_proj = nn.Linear(hidden, hidden)+ attn.o_proj = nn.Linear(hidden, hidden)+ mlp = nn.Module()+ mlp.gate_proj = nn.Linear(hidden, hidden)+ mlp.up_proj = nn.Linear(hidden, hidden)+ mlp.down_proj = nn.Linear(hidden, hidden)+ layer.self_attn = attn+ layer.mlp = mlp+ layers.append(layer)+ self.layers = nn.ModuleList(layers)+ # mirror the pre-injection freeze applied by llm_conf.freeze (default true)+ self.freeze()++ def freeze(self):+ for p in self.parameters():+ p.requires_grad = False+ self.eval()+++def _apply(fake_llm, lora_conf):+ """Run the real injection method without building a full FunASRNano.++ ``__new__`` + ``nn.Module.__init__`` initialize the module machinery while+ skipping ``FunASRNano.__init__`` (which would try to download a real LLM).+ """+ inst = FunASRNano.__new__(FunASRNano)+ nn.Module.__init__(inst)+ inst.llm = fake_llm+ inst._apply_lora_to_llm({"use_lora": True, "lora_conf": lora_conf})+++def _lora_layers(fake_llm):+ return [m for m in fake_llm.modules() if isinstance(m, LoRALinear)]+++def test_only_configured_target_modules_replaced():+ fake = _FakeQwen3()+ _apply(fake, DEFAULT_CONF)++ for layer in fake.layers:+ # the two configured targets are LoRA adapters+ assert isinstance(layer.self_attn.q_proj, LoRALinear)+ assert isinstance(layer.self_attn.v_proj, LoRALinear)+ # every other linear stays a plain nn.Linear+ assert type(layer.self_attn.k_proj) is nn.Linear+ assert type(layer.self_attn.o_proj) is nn.Linear+ assert type(layer.mlp.gate_proj) is nn.Linear+ assert type(layer.mlp.up_proj) is nn.Linear+ assert type(layer.mlp.down_proj) is nn.Linear++ # exactly 2 targets x N_LAYERS adapters, each carrying lora_A / lora_B+ adapters = _lora_layers(fake)+ assert len(adapters) == 2 * N_LAYERS+ for m in adapters:+ assert hasattr(m, "lora_A") and hasattr(m, "lora_B")+++def test_custom_target_modules_restrict_replacement():+ fake = _FakeQwen3()+ _apply(fake, {"r": 4, "lora_alpha": 8, "lora_dropout": 0.0,+ "target_modules": ["o_proj"]})++ for layer in fake.layers:+ assert isinstance(layer.self_attn.o_proj, LoRALinear)+ assert type(layer.self_attn.q_proj) is nn.Linear+ assert type(layer.self_attn.v_proj) is nn.Linear+ assert len(_lora_layers(fake)) == N_LAYERS+++def test_initial_output_matches_original_linear():+ fake = _FakeQwen3()+ x = torch.randn(3, HIDDEN)++ with torch.no_grad():+ ref = torch.stack(+ [fake.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]+ )++ _apply(fake, DEFAULT_CONF)+ # lora_B is initialized to zero, so the adapter contributes nothing and the+ # shared base weight reproduces the original linear's output bit-for-bit.+ with torch.no_grad():+ after = torch.stack(+ [fake.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]+ )+ assert torch.equal(after, ref)+++def test_base_frozen_adapters_trainable():+ fake = _FakeQwen3() # frozen before injection, as llm_conf.freeze does+ _apply(fake, DEFAULT_CONF)++ for layer in fake.layers:+ q = layer.self_attn.q_proj+ # shared pretrained base stays frozen+ assert q.weight.requires_grad is False+ assert q.bias.requires_grad is False+ # the new adapter parameters are trainable+ assert q.lora_A.requires_grad is True+ assert q.lora_B.requires_grad is True+++def test_adapter_dtype_matches_base_weight_dtype():+ fake = _FakeQwen3().to(torch.bfloat16) # llm_dtype=bf16 on a real run+ _apply(fake, DEFAULT_CONF)++ for layer in fake.layers:+ q = layer.self_attn.q_proj+ assert q.weight.dtype is torch.bfloat16+ assert q.lora_A.dtype is torch.bfloat16+ assert q.lora_B.dtype is torch.bfloat16+++def test_state_dict_round_trip_and_train_eval_transitions(tmp_path):+ lora_conf = {"r": 4, "lora_alpha": 8, "lora_dropout": 0.0,+ "target_modules": ["q_proj", "v_proj"]}+ fake = _FakeQwen3()+ _apply(fake, lora_conf)++ # force a non-zero adapter+ with torch.no_grad():+ for m in _lora_layers(fake):+ m.lora_A.normal_()+ m.lora_B.normal_(std=0.5)++ x = torch.randn(3, HIDDEN)+ with torch.no_grad():+ train_out = torch.stack(+ [fake.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]+ )++ ckpt_path = os.path.join(tmp_path, "llm_lora.pt")+ torch.save(fake.state_dict(), ckpt_path)++ # model.eval() / model.train() transitions never corrupt the adapter: the+ # top-level eval() keeps the adapter in its unmerged state (nn.Module.eval+ # dispatches through LoRALinear.train(False)), dropout is off in eval, and+ # with dropout=0 the outputs are identical.+ with torch.no_grad():+ fake.eval()+ eval_out = torch.stack(+ [fake.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]+ )+ fake.train()+ train_out2 = torch.stack(+ [fake.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]+ )+ assert torch.allclose(eval_out, train_out2)+ assert torch.equal(train_out, train_out2)++ # The documented merge/unmerge convention on the adapter layer itself:+ # eval() folds W' = W + alpha/r * B @ A into the shared base weight, train()+ # unfolds it again, restoring the exact unmerged output.+ q = fake.layers[0].self_attn.q_proj+ with torch.no_grad():+ q.eval()+ assert q.merged is True+ merged_out = q(x)+ q.train()+ assert q.merged is False+ unmerged_out = q(x)+ assert torch.allclose(merged_out, unmerged_out)+ # the merge/unmerge round trip restores the unmerged behavior (weight is+ # restored to ~1 ulp, not bit-exact: (w + delta) - delta rounds in IEEE)+ assert torch.allclose(train_out[0], unmerged_out)++ # a fresh model built with the same lora_conf loads the checkpoint; the+ # non-zero adapter (and its forward behavior) survives the round trip.+ fresh = _FakeQwen3()+ _apply(fresh, lora_conf)+ fresh.load_state_dict(torch.load(ckpt_path))+ with torch.no_grad():+ loaded_out = torch.stack(+ [fresh.layers[i].self_attn.q_proj(x) for i in range(N_LAYERS)]+ )+ assert torch.equal(loaded_out, train_out)+++def test_no_matching_target_is_explicit_noop():+ import logging++ fake = _FakeQwen3()+ records = []++ class _Collect(logging.Handler):+ def emit(self, record):+ records.append(record.getMessage())++ handler = _Collect(level=logging.WARNING)+ # model.py logs through module-level logging.warning -> the root logger+ logging.getLogger().addHandler(handler)+ try:+ # must not raise; the configuration is accepted with zero adapters+ _apply(fake, {"r": 4, "lora_alpha": 8, "lora_dropout": 0.0,+ "target_modules": ["does_not_exist"]})+ finally:+ logging.getLogger().removeHandler(handler)++ assert len(_lora_layers(fake)) == 0+ for layer in fake.layers:+ assert type(layer.self_attn.q_proj) is nn.Linear+ assert type(layer.self_attn.v_proj) is nn.Linear+ # the model still forwards+ with torch.no_grad():+ fake.layers[0].self_attn.q_proj(torch.randn(2, HIDDEN))+ # the outcome is explicit: a warning documents that nothing was replaced+ assert any("no target modules found" in r for r in records)