Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
docs: add VAD fine-tuning guide and finetune script
游雁ef0e46f1c34a
diff --git a/docs/finetune_vad.md b/docs/finetune_vad.mdnew file mode 100644index 000000000..19e46f586--- /dev/null+++ b/docs/finetune_vad.md@@ -0,0 +1,109 @@+# Fine-tuning FSMN-VAD++This guide shows how to fine-tune the FSMN-VAD (Voice Activity Detection) model on your own data. Fine-tuning is recommended when the default model does not perform well on your specific audio domain (e.g., noisy environments, walkie-talkie audio, call center recordings).++## When to Fine-tune++- High miss rate (speech segments not detected)+- High false alarm rate (non-speech detected as speech)+- Domain-specific audio that differs significantly from the training data (e.g., 8kHz telephony, high-noise industrial environments)++If you only need minor adjustments, try [parameter tuning](#parameter-tuning-without-fine-tuning) first.++## Data Preparation++### Required files++1. **wav.scp** — Audio file list+2. **vad.txt** — VAD labels (speech segment timestamps)++### wav.scp format++```+utt001 /path/to/audio1.wav+utt002 /path/to/audio2.wav+utt003 /path/to/audio3.wav+```++### vad.txt format++Each line contains the utterance ID followed by a list of speech segments `[start_ms, end_ms]`:++```+utt001 [[320, 2150], [3800, 6420], [8100, 12500]]+utt002 [[0, 5000], [6200, 9800]]+utt003 [[100, 3000]]+```++Prepare separate `train_wav.scp` / `train_vad.txt` and `val_wav.scp` / `val_vad.txt` for training and validation.++### Recommended data size++- **Minimum**: 50-100 labeled audio files+- **Recommended**: 500+ for robust fine-tuning+- Include both positive (speech) and negative (silence/noise) examples representative of your target domain++## Fine-tuning++### Step 1: Place data++```+data/list/+├── train_wav.scp+├── train_vad.txt+├── val_wav.scp+└── val_vad.txt+```++### Step 2: Run fine-tuning++```bash+cd examples/industrial_data_pretraining/fsmn_vad_streaming+bash finetune.sh+```++Key parameters in `finetune.sh`:++| Parameter | Default | Description |+|-----------|---------|-------------|+| `model_name_or_model_dir` | `iic/speech_fsmn_vad_zh-cn-16k-common-pytorch` | Base model (use `*-8k-*` for 8kHz audio) |+| `train_conf.max_epoch` | 20 | Training epochs |+| `optim_conf.lr` | 0.00005 | Learning rate (lower = more conservative) |+| `train_conf.validate_interval` | 1000 | Validation frequency (steps) |++### Step 3: Use the fine-tuned model++```python+from funasr import AutoModel++model = AutoModel(model="/path/to/outputs/model_dir")+result = model.generate(input="test.wav")+```++## Parameter Tuning (without fine-tuning)++For minor adjustments, pass VAD parameters directly:++```python+from funasr import AutoModel++model = AutoModel(+ model="paraformer-zh",+ vad_model="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch",+ vad_kwargs={+ "max_end_silence_time": 500, # Max silence within a sentence (ms)+ "speech_noise_thres": 0.5, # Speech/noise threshold (lower = more sensitive)+ "max_single_segment_time": 60000, # Max segment duration (ms)+ }+)+```++### Key parameters++| Parameter | Default | Effect |+|-----------|---------|--------|+| `max_end_silence_time` | 800 | Increase to tolerate longer pauses within sentences |+| `speech_noise_thres` | 0.6 | Decrease to detect more speech (may increase false alarms) |+| `max_single_segment_time` | 60000 | Max duration of a single speech segment |+| `speech_2_noise_ratio` | 1.0 | Decrease to be more sensitive to speech |+| `max_start_silence_time` | 3000 | Max leading silence before speech starts |diff --git a/docs/finetune_vad_zh.md b/docs/finetune_vad_zh.mdnew file mode 100644index 000000000..ea82da4d8--- /dev/null+++ b/docs/finetune_vad_zh.md@@ -0,0 +1,107 @@+# FSMN-VAD 模型微调指南++本文介绍如何在自定义数据上微调 FSMN-VAD(语音活动检测)模型。当默认模型在特定音频场景(如高噪声环境、对讲机音频、电话录音等)表现不佳时,建议进行微调。++## 何时需要微调++- 漏检率高(语音片段未被检测到)+- 误检率高(非语音被误判为语音)+- 音频域与训练数据差异大(如 8kHz 电话音频、工业噪声环境)++如果只需要小幅调整,可以先尝试[参数调优](#参数调优无需微调)。++## 数据准备++### 所需文件++1. **wav.scp** — 音频文件列表+2. **vad.txt** — VAD 标注(语音段时间戳)++### wav.scp 格式++```+utt001 /path/to/audio1.wav+utt002 /path/to/audio2.wav+```++### vad.txt 格式++每行包含语音 ID 和语音段列表 `[起始毫秒, 结束毫秒]`:++```+utt001 [[320, 2150], [3800, 6420], [8100, 12500]]+utt002 [[0, 5000], [6200, 9800]]+```++分别准备训练集(`train_wav.scp` / `train_vad.txt`)和验证集(`val_wav.scp` / `val_vad.txt`)。++### 建议数据量++- **最少**: 50-100 条标注音频+- **推荐**: 500+ 条,效果更稳定+- 应包含目标场景的典型音频(语音和静音/噪声样本)++## 微调步骤++### 第一步:放置数据++```+data/list/+├── train_wav.scp+├── train_vad.txt+├── val_wav.scp+└── val_vad.txt+```++### 第二步:运行微调++```bash+cd examples/industrial_data_pretraining/fsmn_vad_streaming+bash finetune.sh+```++`finetune.sh` 中的关键参数:++| 参数 | 默认值 | 说明 |+|------|--------|------|+| `model_name_or_model_dir` | `iic/speech_fsmn_vad_zh-cn-16k-common-pytorch` | 基础模型(8kHz 音频用 `*-8k-*`) |+| `train_conf.max_epoch` | 20 | 训练轮数 |+| `optim_conf.lr` | 0.00005 | 学习率(越低越保守) |+| `train_conf.validate_interval` | 1000 | 验证频率(步数) |++### 第三步:使用微调后的模型++```python+from funasr import AutoModel++model = AutoModel(model="/path/to/outputs/model_dir")+result = model.generate(input="test.wav")+```++## 参数调优(无需微调)++对于小幅调整,可以直接传入 VAD 参数:++```python+from funasr import AutoModel++model = AutoModel(+ model="paraformer-zh",+ vad_model="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch",+ vad_kwargs={+ "max_end_silence_time": 500, # 句内最大静音时长(ms)+ "speech_noise_thres": 0.5, # 语音/噪声阈值(越低越敏感)+ "max_single_segment_time": 60000, # 单段最大时长(ms)+ }+)+```++### 常用参数说明++| 参数 | 默认值 | 作用 |+|------|--------|------|+| `max_end_silence_time` | 800 | 增大可容忍句内更长的停顿 |+| `speech_noise_thres` | 0.6 | 减小可检测更多语音(可能增加误检) |+| `max_single_segment_time` | 60000 | 单个语音段的最大时长 |+| `speech_2_noise_ratio` | 1.0 | 减小可提高对语音的敏感度 |+| `max_start_silence_time` | 3000 | 起始段最大静音时长 |diff --git a/examples/industrial_data_pretraining/fsmn_vad_streaming/finetune.sh b/examples/industrial_data_pretraining/fsmn_vad_streaming/finetune.shnew file mode 100755index 000000000..6be48366b--- /dev/null+++ b/examples/industrial_data_pretraining/fsmn_vad_streaming/finetune.sh@@ -0,0 +1,64 @@+#!/bin/bash+# Copyright FunASR (https://github.com/modelscope/FunASR). All Rights Reserved.+# MIT License (https://opensource.org/licenses/MIT)+#+# Fine-tune FSMN-VAD model on custom data.+# Usage: bash finetune.sh++workspace=`pwd`++export CUDA_VISIBLE_DEVICES="0"+gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')++# Model: choose 16k or 8k version+model_name_or_model_dir="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch"+# For 8k audio: model_name_or_model_dir="iic/speech_fsmn_vad_zh-cn-8k-common"++# Data directory+data_dir="../../../data/list"++train_data="${data_dir}/train.jsonl"+val_data="${data_dir}/val.jsonl"++# Generate train.jsonl from wav.scp and vad.txt+# wav.scp format: utt_id /path/to/audio.wav+# vad.txt format: utt_id [[start_ms, end_ms], [start_ms, end_ms], ...]+scp2jsonl \+++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_vad.txt"]' \+++data_type_list='["source", "target"]' \+++jsonl_file_out="${train_data}"++scp2jsonl \+++scp_file_list='["../../../data/list/val_wav.scp", "../../../data/list/val_vad.txt"]' \+++data_type_list='["source", "target"]' \+++jsonl_file_out="${val_data}"++output_dir="./outputs"+log_file="${output_dir}/log.txt"+mkdir -p ${output_dir}++torchrun \+ --nnodes 1 \+ --nproc_per_node $gpu_num \+../../../funasr/bin/train_ds.py \+++model="${model_name_or_model_dir}" \+++train_data_set_list="${train_data}" \+++valid_data_set_list="${val_data}" \+++dataset="AudioDataset" \+++dataset_conf.index_ds="IndexDSJsonl" \+++dataset_conf.data_split_num=1 \+++dataset_conf.batch_sampler="BatchSampler" \+++dataset_conf.batch_size=6000 \+++dataset_conf.sort_size=1024 \+++dataset_conf.batch_type="token" \+++dataset_conf.num_workers=4 \+++train_conf.max_epoch=20 \+++train_conf.log_interval=50 \+++train_conf.resume=true \+++train_conf.validate_interval=1000 \+++train_conf.save_checkpoint_interval=1000 \+++train_conf.keep_nbest_models=5 \+++train_conf.avg_nbest_model=3 \+++train_conf.use_deepspeed=false \+++optim_conf.lr=0.00005 \+++output_dir="${output_dir}" &> ${log_file}