Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
docs: add SenseVoice continual fine-tuning guide (#3396)
zhifu gaoeff1080856a4
diff --git a/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING.md b/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING.mdnew file mode 100644index 000000000..86679756a--- /dev/null+++ b/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING.md@@ -0,0 +1,161 @@+# Continual fine-tuning SenseVoiceSmall++This guide covers adding an accent, dialect, or domain to SenseVoiceSmall while+retaining its existing languages. Full-parameter, partial-parameter, adapter,+and replay setups cannot guarantee zero regression. Treat retention as an explicit+evaluation constraint.++## 1. Lock the evaluation sets first++Create one fixed validation set for every language or domain that must be+preserved and one for the new domain. The speakers in validation must be+speaker-disjoint from all training and replay data. Do not use a single mixed+CER to select checkpoints because a large Mandarin set can hide Cantonese+regression.++Record the original model's CER or WER on every set before generating+pseudo-labels. Define an acceptable relative regression for each retained+language before training:++```text+relative regression = (candidate CER - baseline CER) / baseline CER+```++The threshold is a product decision, not a FunASR guarantee. Select a checkpoint+only when it meets every retention threshold and the new-domain target.++A real starting point from [issue #3388](https://github.com/modelscope/FunASR/issues/3388)+is:++| Validation set | Utterances | Baseline CER |+| --- | ---: | ---: |+| Mandarin FLEURS test | 300 | 8.9% |+| Cantonese ASCEND dialogue | 314 | 9.1% |+| Teochew held-out, about 20 speakers in the source corpus | 300 | 93.3% |++Keep the Teochew held-out speakers out of the 8.45-hour training set.++## 2. Build a replay manifest++When manually labeled old-domain data is unavailable, decode owned or+appropriately licensed Mandarin and Cantonese audio with the original model.+Store the model revision and raw decoding output with each pseudo-label so the+manifest can be audited and regenerated.++Filter or review at least these cases before training:++- empty, repeated, or obviously truncated pseudo-labels;+- language mismatches;+- audio outside the duration range supported by the training setup;+- low decoder scores when a comparable score is available;+- a random sample from every speaker and language, including high-score samples.++A pseudo-label is not ground truth. Keep validation labels independent from+pseudo-label generation.++Sample a fixed replay budget for each training epoch instead of copying the+entire old-domain pool. Start with this small matrix:++| replay:new | New Teochew | Replay per epoch |+| --- | ---: | ---: |+| 1:1 | 8.45 h | 8.45 h |+| 2:1 | 8.45 h | 16.9 h |+| 3:1 | 8.45 h | 25.35 h |++For this example, start with 2:1. A practical first replay split is about+10.1 hours Mandarin and 6.8 hours Cantonese per epoch. The 60:40 split is an+experiment starting point, not an official ratio: it deliberately oversamples+Cantonese relative to a 100 h / 10 h source pool because Cantonese retention is+a separate requirement. Sample different records across epochs and keep a+minimum Cantonese share. Use the manifest as the source of truth when reported+pool totals disagree.++## 3. Use a supported language token++SenseVoiceSmall currently has a closed language-id set: `<|zh|>`, `<|en|>`,+`<|yue|>`, `<|ja|>`, and `<|ko|>`. For Teochew transcripts written as+standard Chinese, use `<|zh|>` for the first experiment.++Do not add `<|teochew|>` to the JSONL and assume it becomes a new language id.+The current tokenizer does not encode it as one supported special token. A real+custom language id requires coordinated tokenizer, vocabulary, language+embedding, checkpoint migration, export, and inference changes.++## 4. Train in two stages++Copy `finetune.sh` to an experiment directory, point it at the mixed training+manifest and fixed validation manifests, and replace the default long,+high-learning-rate run with a short first stage.++### Stage 1: head warm-up++Freeze the complete acoustic encoder and train the language-query embedding and+CTC/output parameters:++```bash+++train_conf.max_epoch=5 \+++freeze_param="encoder" \+++optim_conf.lr=0.00002 \+```++Three to five epochs are an initial budget, not a required value. Validate and+save often enough to observe all three curves. Do not wait for 50 epochs before+checking retention.++### Stage 2: release the later encoder blocks++If Teochew CER stalls while Mandarin and Cantonese remain inside their+retention gates, resume the best Stage 1 checkpoint and freeze only the input+block and the first three regular encoder blocks:++```bash+++freeze_param="encoder.encoders0,encoder.encoders.0,encoder.encoders.1,encoder.encoders.2" \+++optim_conf.lr=0.00002 \+```++The value is a comma-separated list of parameter-name prefixes. With the+current SenseVoiceSmall encoder, later `encoder.encoders.*` blocks remain+trainable. Lower the learning rate further, for example to `0.00001`, or stop+earlier if a retained language regresses.++At startup, verify that the log contains the expected lines:++```text+Setting encoder...requires_grad = False+```++Also inspect the model summary. A misspelled prefix silently leaves the intended+parameters trainable because no parameter name matches it.++## 5. Select the checkpoint by constraints++Evaluate every candidate checkpoint on Mandarin, Cantonese, and Teochew+separately.++1. Discard checkpoints that violate either retained-language threshold.+2. Among the remaining checkpoints, choose the lowest Teochew CER.+3. If none pass, increase replay from 1:1 to 2:1 or 3:1, increase the minimum+ Cantonese share, reduce the learning rate, or shorten training.+4. If all retained metrics pass but Teochew stalls, move from Stage 1 to the+ partial-freeze Stage 2 configuration.++Keep a table with the model revision, manifest revision, replay mix, frozen+prefixes, learning rate, step, and all three metrics. This makes the selected+checkpoint reproducible.++## 6. LoRA status++FunASR contains LoRA support for specific Paraformer/SANM paths, but+SenseVoiceSmall does not currently expose a complete, documented LoRA recipe.+Do not assume that setting `lora_only` creates trainable SenseVoice adapters.+Verify the trainable parameter list before relying on any adapter experiment.++## Launch checklist++- Validation speakers do not occur in train or replay data.+- Baseline CER/WER and acceptable relative regression are recorded per language.+- Pseudo-labels are filtered and manually sampled for audit.+- Replay is sampled to an explicit per-epoch budget with a Cantonese floor.+- Teochew uses the supported `<|zh|>` path unless the model is deliberately forked.+- Freeze logs and trainable parameters match the intended stage.+- Every saved checkpoint is evaluated on all retained and new domains.diff --git a/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING_zh.md b/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING_zh.mdnew file mode 100644index 000000000..6264d7360--- /dev/null+++ b/examples/industrial_data_pretraining/sense_voice/CONTINUAL_FINETUNING_zh.md@@ -0,0 +1,138 @@+# SenseVoiceSmall 持续微调指南++本文适用于在 SenseVoiceSmall 中增加口音、方言或新业务域,同时尽量保持已有语种能力。+无论是全量微调、部分参数微调、adapter 还是 replay,都不能保证零退化;必须把旧语种+保持能力作为独立的评测约束。++## 1. 先锁定验证集++为每个需要保留的语种或业务域建立固定验证集,并为新域单独建立验证集。验证集和训练、+replay 数据之间必须说话人不重叠。不要用一个混合 CER 选择 checkpoint,因为较大的+普通话集合可能掩盖粤语退化。++生成伪标签之前,先记录原始模型在每套验证集上的 CER 或 WER,并预先定义每个旧语种+可接受的相对退化:++```text+相对退化 = (候选模型 CER - 基线 CER) / 基线 CER+```++阈值应由业务要求决定,不是 FunASR 的保证值。候选 checkpoint 必须同时满足所有旧语种+阈值和新域目标。++[issue #3388](https://github.com/modelscope/FunASR/issues/3388) 提供了一个真实起点:++| 验证集 | 句数 | 基线 CER |+| --- | ---: | ---: |+| 普通话 FLEURS test | 300 | 8.9% |+| 粤语 ASCEND 口语对话 | 314 | 9.1% |+| 潮汕话 held-out(源语料约 20 名说话人) | 300 | 93.3% |++潮汕话 held-out 中的说话人不能出现在 8.45 小时训练数据中。++## 2. 构建 replay 清单++没有人工标注的旧域数据时,可以使用原始模型转写自有或许可兼容的普通话、粤语音频。+每条伪标签应保存模型版本和原始解码输出,便于审计和重新生成。++训练前至少过滤或检查:++- 空文本、重复文本和明显截断的伪标签;+- 语种不匹配的样本;+- 超出当前训练配置支持时长范围的音频;+- 解码器能提供可比较分数时的低分样本;+- 每个语种、每位说话人的随机样本,包括高分样本。++伪标签不等同于真实标签。验证集标签不能来自本轮伪标签流程。++每个 epoch 按固定预算采样 replay,不要把全部旧域池无差别复制进每轮训练。建议先跑+下面的小矩阵:++| replay:new | 新增潮汕话 | 每轮 replay |+| --- | ---: | ---: |+| 1:1 | 8.45 小时 | 8.45 小时 |+| 2:1 | 8.45 小时 | 16.9 小时 |+| 3:1 | 8.45 小时 | 25.35 小时 |++这个案例可先从 2:1 开始,即每轮约采样普通话 10.1 小时、粤语 6.8 小时。60:40 只是+实验起点,不是官方固定比例;它有意提高粤语相对占比,避免按 100 小时普通话、10 小时+粤语的原始池比例采样后,独立的粤语保持目标被淹没。不同 epoch 应轮换样本,并为粤语+设置最低占比。若口头统计的总小时数与分语种小时数不一致,以实际 manifest 为准。++## 3. 使用现有支持的语言 token++SenseVoiceSmall 当前语言 id 是闭集:`<|zh|>`、`<|en|>`、`<|yue|>`、+`<|ja|>` 和 `<|ko|>`。潮汕话转写目标为标准中文时,第一轮实验使用 `<|zh|>`。++不要直接在 JSONL 中添加 `<|teochew|>` 并假设它会成为新语言 id。当前 tokenizer+不会把它编码成一个受支持的特殊 token。真正新增自定义语言 id,需要同步修改 tokenizer、+词表、language embedding、旧权重迁移、导出和推理代码。++## 4. 分两阶段训练++把 `finetune.sh` 复制到实验目录,指向混合训练 manifest 和固定验证集,然后把默认的+长周期、高学习率训练替换为短周期第一阶段。++### 阶段一:输出层 warm-up++冻结完整声学 encoder,只训练语言查询 embedding 和 CTC/output 参数:++```bash+++train_conf.max_epoch=5 \+++freeze_param="encoder" \+++optim_conf.lr=0.00002 \+```++3 至 5 个 epoch 只是初始预算,不是固定要求。应提高验证和保存频率,及时观察三条曲线,+不要等默认 50 epoch 结束后才检查旧语种退化。++### 阶段二:放开 encoder 后部层++若潮汕话 CER 停滞,而普通话、粤语仍满足保持阈值,则从阶段一最佳 checkpoint 继续,+只冻结输入层和前三个常规 encoder block:++```bash+++freeze_param="encoder.encoders0,encoder.encoders.0,encoder.encoders.1,encoder.encoders.2" \+++optim_conf.lr=0.00002 \+```++该参数是逗号分隔的参数名前缀。按当前 SenseVoiceSmall 结构,后面的+`encoder.encoders.*` block 仍可训练。如果旧语种开始退化,应进一步把学习率降到+例如 `0.00001`,或提前停止。++启动后确认日志包含预期内容:++```text+Setting encoder...requires_grad = False+```++同时检查 model summary。前缀拼写错误时不会匹配任何参数,预期参数会保持可训练状态。++## 5. 按约束选择 checkpoint++每个候选 checkpoint 都要分别评估普通话、粤语和潮汕话。++1. 丢弃任何一个旧语种超过退化阈值的 checkpoint。+2. 在剩余 checkpoint 中选择潮汕话 CER 最低者。+3. 如果没有候选通过,依次增加 replay 到 2:1 或 3:1、提高粤语最低占比、降低学习率或+ 缩短训练。+4. 如果旧语种全部通过但潮汕话停滞,再从阶段一切换到部分冻结的阶段二。++实验表应保存模型版本、manifest 版本、replay 配比、冻结前缀、学习率、step 和三套指标,+保证最终 checkpoint 可以复现。++## 6. LoRA 现状++FunASR 中已有面向部分 Paraformer/SANM 路径的 LoRA 支持,但 SenseVoiceSmall 目前+没有完整且文档化的 LoRA 配方。不要假设设置 `lora_only` 就会自动生成可训练的+SenseVoice adapter;任何 adapter 实验都必须先核对实际可训练参数列表。++## 启动前检查++- 验证集与训练、replay 数据之间说话人不重叠。+- 已记录每个语种的基线 CER/WER 和可接受相对退化。+- 已过滤伪标签,并按语种和说话人做人工抽检。+- replay 使用明确的每轮预算,并设置粤语最低占比。+- 未 fork 模型时,潮汕话使用当前支持的 `<|zh|>` 路径。+- 冻结日志和可训练参数与当前阶段一致。+- 所有保存的 checkpoint 都分别评估旧语种和新域。diff --git a/examples/industrial_data_pretraining/sense_voice/README.md b/examples/industrial_data_pretraining/sense_voice/README.mdindex c59eccdff..c5e89ea42 100644--- a/examples/industrial_data_pretraining/sense_voice/README.md+++ b/examples/industrial_data_pretraining/sense_voice/README.md@@ -352,6 +352,8 @@ Ensure to modify the train_tool in finetune.sh to the absolute path of `funasr/bbash finetune.sh```+For adding a new accent, language, or domain while retaining existing capabilities, see [Continual fine-tuning](CONTINUAL_FINETUNING.md).+## WebUI```shelldiff --git a/examples/industrial_data_pretraining/sense_voice/README_zh.md b/examples/industrial_data_pretraining/sense_voice/README_zh.mdindex a54df6253..558414098 100644--- a/examples/industrial_data_pretraining/sense_voice/README_zh.md+++ b/examples/industrial_data_pretraining/sense_voice/README_zh.md@@ -380,6 +380,8 @@ sensevoice2jsonl \bash finetune.sh```+需要增加新口音、语种或业务域并保持已有能力时,请参考[持续微调指南](CONTINUAL_FINETUNING_zh.md)。+## WebUI```shelldiff --git a/tests/test_sensevoice_continual_finetuning_docs.py b/tests/test_sensevoice_continual_finetuning_docs.pynew file mode 100644index 000000000..0c80dae3c--- /dev/null+++ b/tests/test_sensevoice_continual_finetuning_docs.py@@ -0,0 +1,63 @@+from pathlib import Path+++ROOT = Path(__file__).resolve().parents[1]+SENSEVOICE_DIR = ROOT / "examples" / "industrial_data_pretraining" / "sense_voice"+++def test_readmes_link_continual_finetuning_guides():+ readme = (SENSEVOICE_DIR / "README.md").read_text(encoding="utf-8")+ readme_zh = (SENSEVOICE_DIR / "README_zh.md").read_text(encoding="utf-8")++ assert "[Continual fine-tuning](CONTINUAL_FINETUNING.md)" in readme+ assert "[持续微调指南](CONTINUAL_FINETUNING_zh.md)" in readme_zh+++def test_english_guide_covers_retention_controls_and_worked_example():+ guide = (SENSEVOICE_DIR / "CONTINUAL_FINETUNING.md").read_text(encoding="utf-8")++ required = [+ "cannot guarantee zero regression",+ "speaker-disjoint",+ "pseudo-label",+ "replay:new",+ "1:1",+ "2:1",+ "3:1",+ '++freeze_param="encoder"',+ '++freeze_param="encoder.encoders0,encoder.encoders.0,encoder.encoders.1,encoder.encoders.2"',+ "++optim_conf.lr=0.00002",+ "<|zh|>",+ "<|teochew|>",+ "8.45",+ "16.9",+ "Mandarin",+ "Cantonese",+ "Teochew",+ ]+ assert all(item in guide for item in required)+++def test_chinese_guide_covers_retention_controls_and_worked_example():+ guide = (SENSEVOICE_DIR / "CONTINUAL_FINETUNING_zh.md").read_text(encoding="utf-8")++ required = [+ "不能保证零退化",+ "说话人不重叠",+ "伪标签",+ "replay:new",+ "1:1",+ "2:1",+ "3:1",+ '++freeze_param="encoder"',+ '++freeze_param="encoder.encoders0,encoder.encoders.0,encoder.encoders.1,encoder.encoders.2"',+ "++optim_conf.lr=0.00002",+ "<|zh|>",+ "<|teochew|>",+ "8.45",+ "16.9",+ "普通话",+ "粤语",+ "潮汕话",+ ]+ assert all(item in guide for item in required)