Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

docs: add VAD fine-tuning guide and finetune script

游雁ef0e46f1c34a
diff --git a/docs/finetune_vad.md b/docs/finetune_vad.md
new file mode 100644
index 000000000..19e46f586
--- /dev/null
+++ b/docs/finetune_vad.md
@@ -0,0 +1,109 @@
+# Fine-tuning FSMN-VAD
+
+This guide shows how to fine-tune the FSMN-VAD (Voice Activity Detection) model on your own data. Fine-tuning is recommended when the default model does not perform well on your specific audio domain (e.g., noisy environments, walkie-talkie audio, call center recordings).
+
+## When to Fine-tune
+
+- High miss rate (speech segments not detected)
+- High false alarm rate (non-speech detected as speech)
+- Domain-specific audio that differs significantly from the training data (e.g., 8kHz telephony, high-noise industrial environments)
+
+If you only need minor adjustments, try [parameter tuning](#parameter-tuning-without-fine-tuning) first.
+
+## Data Preparation
+
+### Required files
+
+1. **wav.scp** — Audio file list
+2. **vad.txt** — VAD labels (speech segment timestamps)
+
+### wav.scp format
+
+```
+utt001 /path/to/audio1.wav
+utt002 /path/to/audio2.wav
+utt003 /path/to/audio3.wav
+```
+
+### vad.txt format
+
+Each line contains the utterance ID followed by a list of speech segments `[start_ms, end_ms]`:
+
+```
+utt001 [[320, 2150], [3800, 6420], [8100, 12500]]
+utt002 [[0, 5000], [6200, 9800]]
+utt003 [[100, 3000]]
+```
+
+Prepare separate `train_wav.scp` / `train_vad.txt` and `val_wav.scp` / `val_vad.txt` for training and validation.
+
+### Recommended data size
+
+- **Minimum**: 50-100 labeled audio files
+- **Recommended**: 500+ for robust fine-tuning
+- Include both positive (speech) and negative (silence/noise) examples representative of your target domain
+
+## Fine-tuning
+
+### Step 1: Place data
+
+```
+data/list/
+├── train_wav.scp
+├── train_vad.txt
+├── val_wav.scp
+└── val_vad.txt
+```
+
+### Step 2: Run fine-tuning
+
+```bash
+cd examples/industrial_data_pretraining/fsmn_vad_streaming
+bash finetune.sh
+```
+
+Key parameters in `finetune.sh`:
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `model_name_or_model_dir` | `iic/speech_fsmn_vad_zh-cn-16k-common-pytorch` | Base model (use `*-8k-*` for 8kHz audio) |
+| `train_conf.max_epoch` | 20 | Training epochs |
+| `optim_conf.lr` | 0.00005 | Learning rate (lower = more conservative) |
+| `train_conf.validate_interval` | 1000 | Validation frequency (steps) |
+
+### Step 3: Use the fine-tuned model
+
+```python
+from funasr import AutoModel
+
+model = AutoModel(model="/path/to/outputs/model_dir")
+result = model.generate(input="test.wav")
+```
+
+## Parameter Tuning (without fine-tuning)
+
+For minor adjustments, pass VAD parameters directly:
+
+```python
+from funasr import AutoModel
+
+model = AutoModel(
+ model="paraformer-zh",
+ vad_model="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch",
+ vad_kwargs={
+ "max_end_silence_time": 500, # Max silence within a sentence (ms)
+ "speech_noise_thres": 0.5, # Speech/noise threshold (lower = more sensitive)
+ "max_single_segment_time": 60000, # Max segment duration (ms)
+ }
+)
+```
+
+### Key parameters
+
+| Parameter | Default | Effect |
+|-----------|---------|--------|
+| `max_end_silence_time` | 800 | Increase to tolerate longer pauses within sentences |
+| `speech_noise_thres` | 0.6 | Decrease to detect more speech (may increase false alarms) |
+| `max_single_segment_time` | 60000 | Max duration of a single speech segment |
+| `speech_2_noise_ratio` | 1.0 | Decrease to be more sensitive to speech |
+| `max_start_silence_time` | 3000 | Max leading silence before speech starts |
diff --git a/docs/finetune_vad_zh.md b/docs/finetune_vad_zh.md
new file mode 100644
index 000000000..ea82da4d8
--- /dev/null
+++ b/docs/finetune_vad_zh.md
@@ -0,0 +1,107 @@
+# FSMN-VAD 模型微调指南
+
+本文介绍如何在自定义数据上微调 FSMN-VAD(语音活动检测)模型。当默认模型在特定音频场景(如高噪声环境、对讲机音频、电话录音等)表现不佳时,建议进行微调。
+
+## 何时需要微调
+
+- 漏检率高(语音片段未被检测到)
+- 误检率高(非语音被误判为语音)
+- 音频域与训练数据差异大(如 8kHz 电话音频、工业噪声环境)
+
+如果只需要小幅调整,可以先尝试[参数调优](#参数调优无需微调)。
+
+## 数据准备
+
+### 所需文件
+
+1. **wav.scp** — 音频文件列表
+2. **vad.txt** — VAD 标注(语音段时间戳)
+
+### wav.scp 格式
+
+```
+utt001 /path/to/audio1.wav
+utt002 /path/to/audio2.wav
+```
+
+### vad.txt 格式
+
+每行包含语音 ID 和语音段列表 `[起始毫秒, 结束毫秒]`:
+
+```
+utt001 [[320, 2150], [3800, 6420], [8100, 12500]]
+utt002 [[0, 5000], [6200, 9800]]
+```
+
+分别准备训练集(`train_wav.scp` / `train_vad.txt`)和验证集(`val_wav.scp` / `val_vad.txt`)。
+
+### 建议数据量
+
+- **最少**: 50-100 条标注音频
+- **推荐**: 500+ 条,效果更稳定
+- 应包含目标场景的典型音频(语音和静音/噪声样本)
+
+## 微调步骤
+
+### 第一步:放置数据
+
+```
+data/list/
+├── train_wav.scp
+├── train_vad.txt
+├── val_wav.scp
+└── val_vad.txt
+```
+
+### 第二步:运行微调
+
+```bash
+cd examples/industrial_data_pretraining/fsmn_vad_streaming
+bash finetune.sh
+```
+
+`finetune.sh` 中的关键参数:
+
+| 参数 | 默认值 | 说明 |
+|------|--------|------|
+| `model_name_or_model_dir` | `iic/speech_fsmn_vad_zh-cn-16k-common-pytorch` | 基础模型(8kHz 音频用 `*-8k-*`) |
+| `train_conf.max_epoch` | 20 | 训练轮数 |
+| `optim_conf.lr` | 0.00005 | 学习率(越低越保守) |
+| `train_conf.validate_interval` | 1000 | 验证频率(步数) |
+
+### 第三步:使用微调后的模型
+
+```python
+from funasr import AutoModel
+
+model = AutoModel(model="/path/to/outputs/model_dir")
+result = model.generate(input="test.wav")
+```
+
+## 参数调优(无需微调)
+
+对于小幅调整,可以直接传入 VAD 参数:
+
+```python
+from funasr import AutoModel
+
+model = AutoModel(
+ model="paraformer-zh",
+ vad_model="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch",
+ vad_kwargs={
+ "max_end_silence_time": 500, # 句内最大静音时长(ms)
+ "speech_noise_thres": 0.5, # 语音/噪声阈值(越低越敏感)
+ "max_single_segment_time": 60000, # 单段最大时长(ms)
+ }
+)
+```
+
+### 常用参数说明
+
+| 参数 | 默认值 | 作用 |
+|------|--------|------|
+| `max_end_silence_time` | 800 | 增大可容忍句内更长的停顿 |
+| `speech_noise_thres` | 0.6 | 减小可检测更多语音(可能增加误检) |
+| `max_single_segment_time` | 60000 | 单个语音段的最大时长 |
+| `speech_2_noise_ratio` | 1.0 | 减小可提高对语音的敏感度 |
+| `max_start_silence_time` | 3000 | 起始段最大静音时长 |
diff --git a/examples/industrial_data_pretraining/fsmn_vad_streaming/finetune.sh b/examples/industrial_data_pretraining/fsmn_vad_streaming/finetune.sh
new file mode 100755
index 000000000..6be48366b
--- /dev/null
+++ b/examples/industrial_data_pretraining/fsmn_vad_streaming/finetune.sh
@@ -0,0 +1,64 @@
+#!/bin/bash
+# Copyright FunASR (https://github.com/modelscope/FunASR). All Rights Reserved.
+# MIT License (https://opensource.org/licenses/MIT)
+#
+# Fine-tune FSMN-VAD model on custom data.
+# Usage: bash finetune.sh
+
+workspace=`pwd`
+
+export CUDA_VISIBLE_DEVICES="0"
+gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
+
+# Model: choose 16k or 8k version
+model_name_or_model_dir="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch"
+# For 8k audio: model_name_or_model_dir="iic/speech_fsmn_vad_zh-cn-8k-common"
+
+# Data directory
+data_dir="../../../data/list"
+
+train_data="${data_dir}/train.jsonl"
+val_data="${data_dir}/val.jsonl"
+
+# Generate train.jsonl from wav.scp and vad.txt
+# wav.scp format: utt_id /path/to/audio.wav
+# vad.txt format: utt_id [[start_ms, end_ms], [start_ms, end_ms], ...]
+scp2jsonl \
+++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_vad.txt"]' \
+++data_type_list='["source", "target"]' \
+++jsonl_file_out="${train_data}"
+
+scp2jsonl \
+++scp_file_list='["../../../data/list/val_wav.scp", "../../../data/list/val_vad.txt"]' \
+++data_type_list='["source", "target"]' \
+++jsonl_file_out="${val_data}"
+
+output_dir="./outputs"
+log_file="${output_dir}/log.txt"
+mkdir -p ${output_dir}
+
+torchrun \
+ --nnodes 1 \
+ --nproc_per_node $gpu_num \
+../../../funasr/bin/train_ds.py \
+++model="${model_name_or_model_dir}" \
+++train_data_set_list="${train_data}" \
+++valid_data_set_list="${val_data}" \
+++dataset="AudioDataset" \
+++dataset_conf.index_ds="IndexDSJsonl" \
+++dataset_conf.data_split_num=1 \
+++dataset_conf.batch_sampler="BatchSampler" \
+++dataset_conf.batch_size=6000 \
+++dataset_conf.sort_size=1024 \
+++dataset_conf.batch_type="token" \
+++dataset_conf.num_workers=4 \
+++train_conf.max_epoch=20 \
+++train_conf.log_interval=50 \
+++train_conf.resume=true \
+++train_conf.validate_interval=1000 \
+++train_conf.save_checkpoint_interval=1000 \
+++train_conf.keep_nbest_models=5 \
+++train_conf.avg_nbest_model=3 \
+++train_conf.use_deepspeed=false \
+++optim_conf.lr=0.00005 \
+++output_dir="${output_dir}" &> ${log_file}