Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
revert: remove incorrect VAD finetune docs and script
游雁1065d2e5ab9d
diff --git a/README.md b/README.mdindex 0bd7a90cf..626386c82 100644--- a/README.md+++ b/README.md@@ -21,7 +21,7 @@</p><p align="center">- <a href="#quick-start">Quick Start</a> · <a href="./examples/colab/">Colab</a> · <a href="#benchmark">Benchmark</a> · <a href="./docs/model_selection.md">Model selection</a> · <a href="./docs/migration_from_whisper.md">Migration guide</a> · <a href="./docs/use_case_showcase.md">Use cases</a> · <a href="./docs/deployment_matrix.md">Deployment matrix</a> · <a href="./docs/finetune_vad.md">VAD fine-tuning</a> · <a href="#model-zoo">Models</a> · <a href="https://modelscope.github.io/FunASR/agent.html">Agent Integration</a> · <a href="https://modelscope.github.io/FunASR/">Docs</a> · <a href="./CONTRIBUTING.md">Contribute</a>+ <a href="#quick-start">Quick Start</a> · <a href="./examples/colab/">Colab</a> · <a href="#benchmark">Benchmark</a> · <a href="./docs/model_selection.md">Model selection</a> · <a href="./docs/migration_from_whisper.md">Migration guide</a> · <a href="./docs/use_case_showcase.md">Use cases</a> · <a href="./docs/deployment_matrix.md">Deployment matrix</a> · <a href="#model-zoo">Models</a> · <a href="https://modelscope.github.io/FunASR/agent.html">Agent Integration</a> · <a href="https://modelscope.github.io/FunASR/">Docs</a> · <a href="./CONTRIBUTING.md">Contribute</a></p>---diff --git a/README_zh.md b/README_zh.mdindex a70821cb8..919d20fce 100644--- a/README_zh.md+++ b/README_zh.md@@ -21,7 +21,7 @@</p><p align="center">- <a href="#快速开始">快速开始</a> · <a href="./examples/colab/README_zh.md">Colab</a> · <a href="#性能评测">性能评测</a> · <a href="./docs/model_selection_zh.md">模型选择</a> · <a href="./docs/migration_from_whisper_zh.md">迁移指南</a> · <a href="./docs/use_case_showcase_zh.md">场景速览</a> · <a href="./docs/deployment_matrix_zh.md">部署选型</a> · <a href="./docs/finetune_vad_zh.md">VAD微调</a> · <a href="#模型列表">模型列表</a> · <a href="https://modelscope.github.io/FunASR/agent.html">Agent 集成</a> · <a href="https://modelscope.github.io/FunASR/zh/">文档</a> · <a href="./CONTRIBUTING.md">贡献</a>+ <a href="#快速开始">快速开始</a> · <a href="./examples/colab/README_zh.md">Colab</a> · <a href="#性能评测">性能评测</a> · <a href="./docs/model_selection_zh.md">模型选择</a> · <a href="./docs/migration_from_whisper_zh.md">迁移指南</a> · <a href="./docs/use_case_showcase_zh.md">场景速览</a> · <a href="./docs/deployment_matrix_zh.md">部署选型</a> · <a href="#模型列表">模型列表</a> · <a href="https://modelscope.github.io/FunASR/agent.html">Agent 集成</a> · <a href="https://modelscope.github.io/FunASR/zh/">文档</a> · <a href="./CONTRIBUTING.md">贡献</a></p>---diff --git a/docs/finetune_vad.md b/docs/finetune_vad.mddeleted file mode 100644index 19e46f586..000000000--- a/docs/finetune_vad.md+++ /dev/null@@ -1,109 +0,0 @@-# Fine-tuning FSMN-VAD--This guide shows how to fine-tune the FSMN-VAD (Voice Activity Detection) model on your own data. Fine-tuning is recommended when the default model does not perform well on your specific audio domain (e.g., noisy environments, walkie-talkie audio, call center recordings).--## When to Fine-tune--- High miss rate (speech segments not detected)-- High false alarm rate (non-speech detected as speech)-- Domain-specific audio that differs significantly from the training data (e.g., 8kHz telephony, high-noise industrial environments)--If you only need minor adjustments, try [parameter tuning](#parameter-tuning-without-fine-tuning) first.--## Data Preparation--### Required files--1. **wav.scp** — Audio file list-2. **vad.txt** — VAD labels (speech segment timestamps)--### wav.scp format--```-utt001 /path/to/audio1.wav-utt002 /path/to/audio2.wav-utt003 /path/to/audio3.wav-```--### vad.txt format--Each line contains the utterance ID followed by a list of speech segments `[start_ms, end_ms]`:--```-utt001 [[320, 2150], [3800, 6420], [8100, 12500]]-utt002 [[0, 5000], [6200, 9800]]-utt003 [[100, 3000]]-```--Prepare separate `train_wav.scp` / `train_vad.txt` and `val_wav.scp` / `val_vad.txt` for training and validation.--### Recommended data size--- **Minimum**: 50-100 labeled audio files-- **Recommended**: 500+ for robust fine-tuning-- Include both positive (speech) and negative (silence/noise) examples representative of your target domain--## Fine-tuning--### Step 1: Place data--```-data/list/-├── train_wav.scp-├── train_vad.txt-├── val_wav.scp-└── val_vad.txt-```--### Step 2: Run fine-tuning--```bash-cd examples/industrial_data_pretraining/fsmn_vad_streaming-bash finetune.sh-```--Key parameters in `finetune.sh`:--| Parameter | Default | Description |-|-----------|---------|-------------|-| `model_name_or_model_dir` | `iic/speech_fsmn_vad_zh-cn-16k-common-pytorch` | Base model (use `*-8k-*` for 8kHz audio) |-| `train_conf.max_epoch` | 20 | Training epochs |-| `optim_conf.lr` | 0.00005 | Learning rate (lower = more conservative) |-| `train_conf.validate_interval` | 1000 | Validation frequency (steps) |--### Step 3: Use the fine-tuned model--```python-from funasr import AutoModel--model = AutoModel(model="/path/to/outputs/model_dir")-result = model.generate(input="test.wav")-```--## Parameter Tuning (without fine-tuning)--For minor adjustments, pass VAD parameters directly:--```python-from funasr import AutoModel--model = AutoModel(- model="paraformer-zh",- vad_model="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch",- vad_kwargs={- "max_end_silence_time": 500, # Max silence within a sentence (ms)- "speech_noise_thres": 0.5, # Speech/noise threshold (lower = more sensitive)- "max_single_segment_time": 60000, # Max segment duration (ms)- }-)-```--### Key parameters--| Parameter | Default | Effect |-|-----------|---------|--------|-| `max_end_silence_time` | 800 | Increase to tolerate longer pauses within sentences |-| `speech_noise_thres` | 0.6 | Decrease to detect more speech (may increase false alarms) |-| `max_single_segment_time` | 60000 | Max duration of a single speech segment |-| `speech_2_noise_ratio` | 1.0 | Decrease to be more sensitive to speech |-| `max_start_silence_time` | 3000 | Max leading silence before speech starts |diff --git a/docs/finetune_vad_zh.md b/docs/finetune_vad_zh.mddeleted file mode 100644index ea82da4d8..000000000--- a/docs/finetune_vad_zh.md+++ /dev/null@@ -1,107 +0,0 @@-# FSMN-VAD 模型微调指南--本文介绍如何在自定义数据上微调 FSMN-VAD(语音活动检测)模型。当默认模型在特定音频场景(如高噪声环境、对讲机音频、电话录音等)表现不佳时,建议进行微调。--## 何时需要微调--- 漏检率高(语音片段未被检测到)-- 误检率高(非语音被误判为语音)-- 音频域与训练数据差异大(如 8kHz 电话音频、工业噪声环境)--如果只需要小幅调整,可以先尝试[参数调优](#参数调优无需微调)。--## 数据准备--### 所需文件--1. **wav.scp** — 音频文件列表-2. **vad.txt** — VAD 标注(语音段时间戳)--### wav.scp 格式--```-utt001 /path/to/audio1.wav-utt002 /path/to/audio2.wav-```--### vad.txt 格式--每行包含语音 ID 和语音段列表 `[起始毫秒, 结束毫秒]`:--```-utt001 [[320, 2150], [3800, 6420], [8100, 12500]]-utt002 [[0, 5000], [6200, 9800]]-```--分别准备训练集(`train_wav.scp` / `train_vad.txt`)和验证集(`val_wav.scp` / `val_vad.txt`)。--### 建议数据量--- **最少**: 50-100 条标注音频-- **推荐**: 500+ 条,效果更稳定-- 应包含目标场景的典型音频(语音和静音/噪声样本)--## 微调步骤--### 第一步:放置数据--```-data/list/-├── train_wav.scp-├── train_vad.txt-├── val_wav.scp-└── val_vad.txt-```--### 第二步:运行微调--```bash-cd examples/industrial_data_pretraining/fsmn_vad_streaming-bash finetune.sh-```--`finetune.sh` 中的关键参数:--| 参数 | 默认值 | 说明 |-|------|--------|------|-| `model_name_or_model_dir` | `iic/speech_fsmn_vad_zh-cn-16k-common-pytorch` | 基础模型(8kHz 音频用 `*-8k-*`) |-| `train_conf.max_epoch` | 20 | 训练轮数 |-| `optim_conf.lr` | 0.00005 | 学习率(越低越保守) |-| `train_conf.validate_interval` | 1000 | 验证频率(步数) |--### 第三步:使用微调后的模型--```python-from funasr import AutoModel--model = AutoModel(model="/path/to/outputs/model_dir")-result = model.generate(input="test.wav")-```--## 参数调优(无需微调)--对于小幅调整,可以直接传入 VAD 参数:--```python-from funasr import AutoModel--model = AutoModel(- model="paraformer-zh",- vad_model="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch",- vad_kwargs={- "max_end_silence_time": 500, # 句内最大静音时长(ms)- "speech_noise_thres": 0.5, # 语音/噪声阈值(越低越敏感)- "max_single_segment_time": 60000, # 单段最大时长(ms)- }-)-```--### 常用参数说明--| 参数 | 默认值 | 作用 |-|------|--------|------|-| `max_end_silence_time` | 800 | 增大可容忍句内更长的停顿 |-| `speech_noise_thres` | 0.6 | 减小可检测更多语音(可能增加误检) |-| `max_single_segment_time` | 60000 | 单个语音段的最大时长 |-| `speech_2_noise_ratio` | 1.0 | 减小可提高对语音的敏感度 |-| `max_start_silence_time` | 3000 | 起始段最大静音时长 |diff --git a/examples/industrial_data_pretraining/fsmn_vad_streaming/finetune.sh b/examples/industrial_data_pretraining/fsmn_vad_streaming/finetune.shdeleted file mode 100755index 6be4926f9..000000000--- a/examples/industrial_data_pretraining/fsmn_vad_streaming/finetune.sh+++ /dev/null@@ -1,66 +0,0 @@-#!/bin/bash-# Copyright FunASR (https://github.com/modelscope/FunASR). All Rights Reserved.-# MIT License (https://opensource.org/licenses/MIT)-#-# Fine-tune FSMN-VAD model on custom data.-# Usage: bash finetune.sh--workspace=`pwd`--export CUDA_VISIBLE_DEVICES="0"-gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')--# Model: choose 16k or 8k version-model_name_or_model_dir="iic/speech_fsmn_vad_zh-cn-16k-common-pytorch"-# For 8k audio: model_name_or_model_dir="iic/speech_fsmn_vad_zh-cn-8k-common"--# Data directory-data_dir="../../../data/list"--train_data="${data_dir}/train.jsonl"-val_data="${data_dir}/val.jsonl"--# Generate train.jsonl from wav.scp and vad.txt-# wav.scp format: utt_id /path/to/audio.wav-# vad.txt format: utt_id [[start_ms, end_ms], [start_ms, end_ms], ...]-scp2jsonl \-++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_vad.txt"]' \-++data_type_list='["source", "target"]' \-++jsonl_file_out="${train_data}"--scp2jsonl \-++scp_file_list='["../../../data/list/val_wav.scp", "../../../data/list/val_vad.txt"]' \-++data_type_list='["source", "target"]' \-++jsonl_file_out="${val_data}"--output_dir="./outputs"-log_file="${output_dir}/log.txt"-mkdir -p ${output_dir}--torchrun \- --nnodes 1 \- --nproc_per_node $gpu_num \-../../../funasr/bin/train_ds.py \-++model="${model_name_or_model_dir}" \-++train_data_set_list="${train_data}" \-++valid_data_set_list="${val_data}" \-++dataset="AudioDataset" \-++dataset_conf.index_ds="IndexDSJsonl" \-++dataset_conf.data_split_num=1 \-++dataset_conf.batch_sampler="BatchSampler" \-++dataset_conf.batch_size=6000 \-++dataset_conf.sort_size=1024 \-++dataset_conf.batch_type="token" \-++dataset_conf.num_workers=4 \-++train_conf.max_epoch=20 \-++train_conf.log_interval=50 \-++train_conf.resume=true \-++train_conf.validate_interval=1000 \-++train_conf.save_checkpoint_interval=1000 \-++train_conf.keep_nbest_models=5 \-++train_conf.avg_nbest_model=3 \-++train_conf.use_deepspeed=false \-++optim_conf.lr=0.00005 \-++scheduler="warmuplr" \-++scheduler_conf.warmup_steps=1000 \-++output_dir="${output_dir}" &> ${log_file}