Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

docs: add SenseVoice continual fine-tuning guide (#3396)

zhifu gaoeff1080856a4
diff --git a/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING.md b/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING.md
new file mode 100644
index 000000000..86679756a
--- /dev/null
+++ b/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING.md
@@ -0,0 +1,161 @@
+# Continual fine-tuning SenseVoiceSmall
+
+This guide covers adding an accent, dialect, or domain to SenseVoiceSmall while
+retaining its existing languages. Full-parameter, partial-parameter, adapter,
+and replay setups cannot guarantee zero regression. Treat retention as an explicit
+evaluation constraint.
+
+## 1. Lock the evaluation sets first
+
+Create one fixed validation set for every language or domain that must be
+preserved and one for the new domain. The speakers in validation must be
+speaker-disjoint from all training and replay data. Do not use a single mixed
+CER to select checkpoints because a large Mandarin set can hide Cantonese
+regression.
+
+Record the original model's CER or WER on every set before generating
+pseudo-labels. Define an acceptable relative regression for each retained
+language before training:
+
+```text
+relative regression = (candidate CER - baseline CER) / baseline CER
+```
+
+The threshold is a product decision, not a FunASR guarantee. Select a checkpoint
+only when it meets every retention threshold and the new-domain target.
+
+A real starting point from [issue #3388](https://github.com/modelscope/FunASR/issues/3388)
+is:
+
+| Validation set | Utterances | Baseline CER |
+| --- | ---: | ---: |
+| Mandarin FLEURS test | 300 | 8.9% |
+| Cantonese ASCEND dialogue | 314 | 9.1% |
+| Teochew held-out, about 20 speakers in the source corpus | 300 | 93.3% |
+
+Keep the Teochew held-out speakers out of the 8.45-hour training set.
+
+## 2. Build a replay manifest
+
+When manually labeled old-domain data is unavailable, decode owned or
+appropriately licensed Mandarin and Cantonese audio with the original model.
+Store the model revision and raw decoding output with each pseudo-label so the
+manifest can be audited and regenerated.
+
+Filter or review at least these cases before training:
+
+- empty, repeated, or obviously truncated pseudo-labels;
+- language mismatches;
+- audio outside the duration range supported by the training setup;
+- low decoder scores when a comparable score is available;
+- a random sample from every speaker and language, including high-score samples.
+
+A pseudo-label is not ground truth. Keep validation labels independent from
+pseudo-label generation.
+
+Sample a fixed replay budget for each training epoch instead of copying the
+entire old-domain pool. Start with this small matrix:
+
+| replay:new | New Teochew | Replay per epoch |
+| --- | ---: | ---: |
+| 1:1 | 8.45 h | 8.45 h |
+| 2:1 | 8.45 h | 16.9 h |
+| 3:1 | 8.45 h | 25.35 h |
+
+For this example, start with 2:1. A practical first replay split is about
+10.1 hours Mandarin and 6.8 hours Cantonese per epoch. The 60:40 split is an
+experiment starting point, not an official ratio: it deliberately oversamples
+Cantonese relative to a 100 h / 10 h source pool because Cantonese retention is
+a separate requirement. Sample different records across epochs and keep a
+minimum Cantonese share. Use the manifest as the source of truth when reported
+pool totals disagree.
+
+## 3. Use a supported language token
+
+SenseVoiceSmall currently has a closed language-id set: `<|zh|>`, `<|en|>`,
+`<|yue|>`, `<|ja|>`, and `<|ko|>`. For Teochew transcripts written as
+standard Chinese, use `<|zh|>` for the first experiment.
+
+Do not add `<|teochew|>` to the JSONL and assume it becomes a new language id.
+The current tokenizer does not encode it as one supported special token. A real
+custom language id requires coordinated tokenizer, vocabulary, language
+embedding, checkpoint migration, export, and inference changes.
+
+## 4. Train in two stages
+
+Copy `finetune.sh` to an experiment directory, point it at the mixed training
+manifest and fixed validation manifests, and replace the default long,
+high-learning-rate run with a short first stage.
+
+### Stage 1: head warm-up
+
+Freeze the complete acoustic encoder and train the language-query embedding and
+CTC/output parameters:
+
+```bash
+++train_conf.max_epoch=5 \
+++freeze_param="encoder" \
+++optim_conf.lr=0.00002 \
+```
+
+Three to five epochs are an initial budget, not a required value. Validate and
+save often enough to observe all three curves. Do not wait for 50 epochs before
+checking retention.
+
+### Stage 2: release the later encoder blocks
+
+If Teochew CER stalls while Mandarin and Cantonese remain inside their
+retention gates, resume the best Stage 1 checkpoint and freeze only the input
+block and the first three regular encoder blocks:
+
+```bash
+++freeze_param="encoder.encoders0,encoder.encoders.0,encoder.encoders.1,encoder.encoders.2" \
+++optim_conf.lr=0.00002 \
+```
+
+The value is a comma-separated list of parameter-name prefixes. With the
+current SenseVoiceSmall encoder, later `encoder.encoders.*` blocks remain
+trainable. Lower the learning rate further, for example to `0.00001`, or stop
+earlier if a retained language regresses.
+
+At startup, verify that the log contains the expected lines:
+
+```text
+Setting encoder...requires_grad = False
+```
+
+Also inspect the model summary. A misspelled prefix silently leaves the intended
+parameters trainable because no parameter name matches it.
+
+## 5. Select the checkpoint by constraints
+
+Evaluate every candidate checkpoint on Mandarin, Cantonese, and Teochew
+separately.
+
+1. Discard checkpoints that violate either retained-language threshold.
+2. Among the remaining checkpoints, choose the lowest Teochew CER.
+3. If none pass, increase replay from 1:1 to 2:1 or 3:1, increase the minimum
+ Cantonese share, reduce the learning rate, or shorten training.
+4. If all retained metrics pass but Teochew stalls, move from Stage 1 to the
+ partial-freeze Stage 2 configuration.
+
+Keep a table with the model revision, manifest revision, replay mix, frozen
+prefixes, learning rate, step, and all three metrics. This makes the selected
+checkpoint reproducible.
+
+## 6. LoRA status
+
+FunASR contains LoRA support for specific Paraformer/SANM paths, but
+SenseVoiceSmall does not currently expose a complete, documented LoRA recipe.
+Do not assume that setting `lora_only` creates trainable SenseVoice adapters.
+Verify the trainable parameter list before relying on any adapter experiment.
+
+## Launch checklist
+
+- Validation speakers do not occur in train or replay data.
+- Baseline CER/WER and acceptable relative regression are recorded per language.
+- Pseudo-labels are filtered and manually sampled for audit.
+- Replay is sampled to an explicit per-epoch budget with a Cantonese floor.
+- Teochew uses the supported `<|zh|>` path unless the model is deliberately forked.
+- Freeze logs and trainable parameters match the intended stage.
+- Every saved checkpoint is evaluated on all retained and new domains.
diff --git a/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING_zh.md b/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING_zh.md
new file mode 100644
index 000000000..6264d7360
--- /dev/null
+++ b/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING_zh.md
@@ -0,0 +1,138 @@
+# SenseVoiceSmall 持续微调指南
+
+本文适用于在 SenseVoiceSmall 中增加口音、方言或新业务域,同时尽量保持已有语种能力。
+无论是全量微调、部分参数微调、adapter 还是 replay,都不能保证零退化;必须把旧语种
+保持能力作为独立的评测约束。
+
+## 1. 先锁定验证集
+
+为每个需要保留的语种或业务域建立固定验证集,并为新域单独建立验证集。验证集和训练、
+replay 数据之间必须说话人不重叠。不要用一个混合 CER 选择 checkpoint,因为较大的
+普通话集合可能掩盖粤语退化。
+
+生成伪标签之前,先记录原始模型在每套验证集上的 CER 或 WER,并预先定义每个旧语种
+可接受的相对退化:
+
+```text
+相对退化 = (候选模型 CER - 基线 CER) / 基线 CER
+```
+
+阈值应由业务要求决定,不是 FunASR 的保证值。候选 checkpoint 必须同时满足所有旧语种
+阈值和新域目标。
+
+[issue #3388](https://github.com/modelscope/FunASR/issues/3388) 提供了一个真实起点:
+
+| 验证集 | 句数 | 基线 CER |
+| --- | ---: | ---: |
+| 普通话 FLEURS test | 300 | 8.9% |
+| 粤语 ASCEND 口语对话 | 314 | 9.1% |
+| 潮汕话 held-out(源语料约 20 名说话人) | 300 | 93.3% |
+
+潮汕话 held-out 中的说话人不能出现在 8.45 小时训练数据中。
+
+## 2. 构建 replay 清单
+
+没有人工标注的旧域数据时,可以使用原始模型转写自有或许可兼容的普通话、粤语音频。
+每条伪标签应保存模型版本和原始解码输出,便于审计和重新生成。
+
+训练前至少过滤或检查:
+
+- 空文本、重复文本和明显截断的伪标签;
+- 语种不匹配的样本;
+- 超出当前训练配置支持时长范围的音频;
+- 解码器能提供可比较分数时的低分样本;
+- 每个语种、每位说话人的随机样本,包括高分样本。
+
+伪标签不等同于真实标签。验证集标签不能来自本轮伪标签流程。
+
+每个 epoch 按固定预算采样 replay,不要把全部旧域池无差别复制进每轮训练。建议先跑
+下面的小矩阵:
+
+| replay:new | 新增潮汕话 | 每轮 replay |
+| --- | ---: | ---: |
+| 1:1 | 8.45 小时 | 8.45 小时 |
+| 2:1 | 8.45 小时 | 16.9 小时 |
+| 3:1 | 8.45 小时 | 25.35 小时 |
+
+这个案例可先从 2:1 开始,即每轮约采样普通话 10.1 小时、粤语 6.8 小时。60:40 只是
+实验起点,不是官方固定比例;它有意提高粤语相对占比,避免按 100 小时普通话、10 小时
+粤语的原始池比例采样后,独立的粤语保持目标被淹没。不同 epoch 应轮换样本,并为粤语
+设置最低占比。若口头统计的总小时数与分语种小时数不一致,以实际 manifest 为准。
+
+## 3. 使用现有支持的语言 token
+
+SenseVoiceSmall 当前语言 id 是闭集:`<|zh|>`、`<|en|>`、`<|yue|>`、
+`<|ja|>` 和 `<|ko|>`。潮汕话转写目标为标准中文时,第一轮实验使用 `<|zh|>`。
+
+不要直接在 JSONL 中添加 `<|teochew|>` 并假设它会成为新语言 id。当前 tokenizer
+不会把它编码成一个受支持的特殊 token。真正新增自定义语言 id,需要同步修改 tokenizer、
+词表、language embedding、旧权重迁移、导出和推理代码。
+
+## 4. 分两阶段训练
+
+把 `finetune.sh` 复制到实验目录,指向混合训练 manifest 和固定验证集,然后把默认的
+长周期、高学习率训练替换为短周期第一阶段。
+
+### 阶段一:输出层 warm-up
+
+冻结完整声学 encoder,只训练语言查询 embedding 和 CTC/output 参数:
+
+```bash
+++train_conf.max_epoch=5 \
+++freeze_param="encoder" \
+++optim_conf.lr=0.00002 \
+```
+
+3 至 5 个 epoch 只是初始预算,不是固定要求。应提高验证和保存频率,及时观察三条曲线,
+不要等默认 50 epoch 结束后才检查旧语种退化。
+
+### 阶段二:放开 encoder 后部层
+
+若潮汕话 CER 停滞,而普通话、粤语仍满足保持阈值,则从阶段一最佳 checkpoint 继续,
+只冻结输入层和前三个常规 encoder block:
+
+```bash
+++freeze_param="encoder.encoders0,encoder.encoders.0,encoder.encoders.1,encoder.encoders.2" \
+++optim_conf.lr=0.00002 \
+```
+
+该参数是逗号分隔的参数名前缀。按当前 SenseVoiceSmall 结构,后面的
+`encoder.encoders.*` block 仍可训练。如果旧语种开始退化,应进一步把学习率降到
+例如 `0.00001`,或提前停止。
+
+启动后确认日志包含预期内容:
+
+```text
+Setting encoder...requires_grad = False
+```
+
+同时检查 model summary。前缀拼写错误时不会匹配任何参数,预期参数会保持可训练状态。
+
+## 5. 按约束选择 checkpoint
+
+每个候选 checkpoint 都要分别评估普通话、粤语和潮汕话。
+
+1. 丢弃任何一个旧语种超过退化阈值的 checkpoint。
+2. 在剩余 checkpoint 中选择潮汕话 CER 最低者。
+3. 如果没有候选通过,依次增加 replay 到 2:1 或 3:1、提高粤语最低占比、降低学习率或
+ 缩短训练。
+4. 如果旧语种全部通过但潮汕话停滞,再从阶段一切换到部分冻结的阶段二。
+
+实验表应保存模型版本、manifest 版本、replay 配比、冻结前缀、学习率、step 和三套指标,
+保证最终 checkpoint 可以复现。
+
+## 6. LoRA 现状
+
+FunASR 中已有面向部分 Paraformer/SANM 路径的 LoRA 支持,但 SenseVoiceSmall 目前
+没有完整且文档化的 LoRA 配方。不要假设设置 `lora_only` 就会自动生成可训练的
+SenseVoice adapter;任何 adapter 实验都必须先核对实际可训练参数列表。
+
+## 启动前检查
+
+- 验证集与训练、replay 数据之间说话人不重叠。
+- 已记录每个语种的基线 CER/WER 和可接受相对退化。
+- 已过滤伪标签,并按语种和说话人做人工抽检。
+- replay 使用明确的每轮预算,并设置粤语最低占比。
+- 未 fork 模型时,潮汕话使用当前支持的 `<|zh|>` 路径。
+- 冻结日志和可训练参数与当前阶段一致。
+- 所有保存的 checkpoint 都分别评估旧语种和新域。
diff --git a/examples/industrial_data_pretraining/sense_voice/README.md b/examples/industrial_data_pretraining/sense_voice/README.md
index c59eccdff..c5e89ea42 100644
--- a/examples/industrial_data_pretraining/sense_voice/README.md
+++ b/examples/industrial_data_pretraining/sense_voice/README.md
@@ -352,6 +352,8 @@ Ensure to modify the train_tool in finetune.sh to the absolute path of `funasr/b
bash finetune.sh
```
+For adding a new accent, language, or domain while retaining existing capabilities, see [Continual fine-tuning](CONTINUAL_FINETUNING.md).
+
## WebUI
```shell
diff --git a/examples/industrial_data_pretraining/sense_voice/README_zh.md b/examples/industrial_data_pretraining/sense_voice/README_zh.md
index a54df6253..558414098 100644
--- a/examples/industrial_data_pretraining/sense_voice/README_zh.md
+++ b/examples/industrial_data_pretraining/sense_voice/README_zh.md
@@ -380,6 +380,8 @@ sensevoice2jsonl \
bash finetune.sh
```
+需要增加新口音、语种或业务域并保持已有能力时,请参考[持续微调指南](CONTINUAL_FINETUNING_zh.md)。
+
## WebUI
```shell
diff --git a/tests/test_sensevoice_continual_finetuning_docs.py b/tests/test_sensevoice_continual_finetuning_docs.py
new file mode 100644
index 000000000..0c80dae3c
--- /dev/null
+++ b/tests/test_sensevoice_continual_finetuning_docs.py
@@ -0,0 +1,63 @@
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+SENSEVOICE_DIR = ROOT / "examples" / "industrial_data_pretraining" / "sense_voice"
+
+
+def test_readmes_link_continual_finetuning_guides():
+ readme = (SENSEVOICE_DIR / "README.md").read_text(encoding="utf-8")
+ readme_zh = (SENSEVOICE_DIR / "README_zh.md").read_text(encoding="utf-8")
+
+ assert "[Continual fine-tuning](CONTINUAL_FINETUNING.md)" in readme
+ assert "[持续微调指南](CONTINUAL_FINETUNING_zh.md)" in readme_zh
+
+
+def test_english_guide_covers_retention_controls_and_worked_example():
+ guide = (SENSEVOICE_DIR / "CONTINUAL_FINETUNING.md").read_text(encoding="utf-8")
+
+ required = [
+ "cannot guarantee zero regression",
+ "speaker-disjoint",
+ "pseudo-label",
+ "replay:new",
+ "1:1",
+ "2:1",
+ "3:1",
+ '++freeze_param="encoder"',
+ '++freeze_param="encoder.encoders0,encoder.encoders.0,encoder.encoders.1,encoder.encoders.2"',
+ "++optim_conf.lr=0.00002",
+ "<|zh|>",
+ "<|teochew|>",
+ "8.45",
+ "16.9",
+ "Mandarin",
+ "Cantonese",
+ "Teochew",
+ ]
+ assert all(item in guide for item in required)
+
+
+def test_chinese_guide_covers_retention_controls_and_worked_example():
+ guide = (SENSEVOICE_DIR / "CONTINUAL_FINETUNING_zh.md").read_text(encoding="utf-8")
+
+ required = [
+ "不能保证零退化",
+ "说话人不重叠",
+ "伪标签",
+ "replay:new",
+ "1:1",
+ "2:1",
+ "3:1",
+ '++freeze_param="encoder"',
+ '++freeze_param="encoder.encoders0,encoder.encoders.0,encoder.encoders.1,encoder.encoders.2"',
+ "++optim_conf.lr=0.00002",
+ "<|zh|>",
+ "<|teochew|>",
+ "8.45",
+ "16.9",
+ "普通话",
+ "粤语",
+ "潮汕话",
+ ]
+ assert all(item in guide for item in required)