Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
docs: add bilingual streaming VAD task guide
zhifu gao3ac13e50494d
diff --git a/.github/workflows/product-site.yml b/.github/workflows/product-site.ymlindex 5ee0cb5fa..8256de757 100644--- a/.github/workflows/product-site.yml+++ b/.github/workflows/product-site.yml@@ -89,6 +89,7 @@ jobs:tests/test_vllm_examples_contract.pytests/test_training_docs_contract.pytests/test_python_api_docs_contract.py+ tests/test_streaming_vad_docs.pytests/test_service_docs_contract.pytests/test_generated_api_docs.pytests/test_api_signatures.pydiff --git a/docs/README.md b/docs/README.mdindex 5742ce54a..7b0d9ec93 100644--- a/docs/README.md+++ b/docs/README.md@@ -28,6 +28,7 @@| Your task | Guide || --- | --- || Compare CPU, GPU, realtime and API runtimes | [Deployment matrix](deployment_matrix.md) / [中文](deployment_matrix_zh.md) |+| Handle streaming speech boundaries and finalization | [Streaming VAD](streaming_vad.md) / [中文](streaming_vad_zh.md) || Transcribe and diarize with third-party MOSS | [MOSS](moss_transcribe_diarize.md) / [中文](moss_transcribe_diarize_zh.md) || Accelerate with the FunASR vLLM split engine | [vLLM](vllm_guide.md) / [中文](vllm_guide_zh.md) || Evaluate native vLLM serving | [Validation record](vllm_native_funasr_validation.md) |diff --git a/docs/index.rst b/docs/index.rstindex 944875224..e22567dfa 100644--- a/docs/index.rst+++ b/docs/index.rst@@ -40,6 +40,8 @@ entry to these guides. Source Markdown remains in this repository.:maxdepth: 1:caption: Models and Deployment+ streaming_vad+ streaming_vad_zhdeployment_matrixdeployment_matrix_zh./moss_transcribe_diarize.mddiff --git a/docs/streaming_vad.md b/docs/streaming_vad.mdnew file mode 100644index 000000000..ab5d84823--- /dev/null+++ b/docs/streaming_vad.md@@ -0,0 +1,162 @@+[简体中文](streaming_vad_zh.md) | English++# Streaming Voice Activity Detection++Use FSMN VAD to receive speech-start and speech-end events while feeding one+recording in order. VAD does not transcribe speech, identify speakers, or locate+linguistic sentence boundaries. For a first offline VAD call, use the+[SDK tutorial](tutorial/README.md); for streaming ASR, use the separate+[Paraformer cache example](python_api.md#streaming-cache-lifecycle).++## Prepare the Model and Audio++Complete the [installation checks](installation/installation.md). Prepare a+complete local snapshot of `iic/speech_fsmn_vad_zh-cn-16k-common-pytorch`,+including configuration, frontend files and weights. Record the resolved model+revision and review its model license. This recipe expects that 16 kHz checkpoint,+not an arbitrary VAD model or a streaming ASR checkpoint.++Use a nonempty mono 16 kHz WAV. The program below takes the local model directory+and audio path as its two positional command-line arguments. It reads the file+once and simulates ordered streaming chunks; it is not microphone capture or a+WebSocket service. The entire file remains in memory.++## Interpret Events++Each returned dictionary has a `value` list. Every pair uses **milliseconds+relative to the beginning of this stream**, not the current input chunk.+Do not add the chunk offset a second time.++| Event | Meaning |+| --- | --- |+| `[]` | No new boundary; an earlier speech span may still be open. |+| `[[start, -1]]` | A speech span started; retain the start for a later call. |+| `[[-1, end]]` | The previously started span ended. |+| `[[start, end]]` | Both boundaries are available in this call. |++One call can return several pairs. `-1` is a missing-boundary sentinel, never a+timestamp to pass to a waveform slice. Keep the pending start per stream.+The strict example below reports unexpected event order instead of inventing+a missing start or silently overwriting one.++## Run Ordered Chunks++The two small helpers are application-side example code, not new FunASR APIs.+The last **nonempty** chunk carries `is_final=True`, including when the input+length is exactly a multiple of the stride.++```python+import argparse+from pathlib import Path+import soundfile as sf+from funasr import AutoModel+++def chunk_ranges(length, stride):+ if length <= 0 or stride <= 0:+ raise ValueError("Audio length and chunk stride must be positive")+ for start in range(0, length, stride):+ end = min(start + stride, length)+ yield start, end, end == length+++def consume_events(events, pending_start):+ spans = []+ for start_ms, end_ms in events:+ if start_ms < 0 and end_ms < 0:+ raise ValueError("An event must provide a boundary")+ if start_ms >= 0:+ if pending_start is not None:+ raise ValueError("New start before the previous speech span ended")+ pending_start = start_ms+ if end_ms >= 0:+ if pending_start is None or end_ms < pending_start:+ raise ValueError("End without a matching earlier start")+ spans.append((pending_start, end_ms))+ pending_start = None+ return pending_start, spans+++parser = argparse.ArgumentParser()+parser.add_argument("model_dir")+parser.add_argument("audio")+args = parser.parse_args()+model_dir = Path(args.model_dir).expanduser().resolve(strict=True)+if not model_dir.is_dir():+ raise ValueError("Expected a complete local FSMN VAD model directory")+speech, sample_rate = sf.read(args.audio, dtype="float32")+if sample_rate != 16000 or speech.ndim != 1 or len(speech) == 0:+ raise ValueError("Expected nonempty mono 16 kHz audio")++model = AutoModel(+ model=str(model_dir), device="cpu", ncpu=4,+ disable_update=True, trust_remote_code=False,+)+chunk_ms = 200+stride = sample_rate * chunk_ms // 1000+cache = {}+pending_start = None+for start, end, final in chunk_ranges(len(speech), stride):+ results = model.generate(+ input=speech[start:end], fs=sample_rate, cache=cache,+ chunk_size=chunk_ms, is_streaming_input=True, is_final=final,+ batch_size=1, dynamic_silence=False,+ )+ for item in results:+ pending_start, spans = consume_events(item.get("value", []), pending_start)+ for start_ms, end_ms in spans:+ print(start_ms, end_ms)+if pending_start is not None:+ print("Unclosed speech start (ms):", pending_start)+# End this session; never share its cache with another recording.+cache = {}+```++At 16 kHz, `chunk_size=200` means 200 ms, or 3200 input samples. It is a+scalar duration for FSMN VAD, not Paraformer streaming's three-element+`chunk_size` list. Keep the same cache dictionary and chunk settings during+a stream, use `batch_size=1`, and create a new cache for every other recording,+user or cancelled session. This implementation reinitializes its cache after a+nonempty final call; discard it at the application boundary anyway.++Do not append an empty final call to flush this example: the current+[inference path](../funasr/models/fsmn_vad_streaming/model.py) returns early for+empty input, before its normal finalization path. For live input whose end is+not known in advance, hold the latest nonempty chunk. When the next chunk+arrives, submit the held chunk with `is_final=False` and hold the new one.+Only when end-of-stream arrives, submit the held chunk with `is_final=True`.+That strategy adds one chunk of input buffering; do not resend an already+consumed chunk. Cancelling a session instead discards its held audio and cache.++## Tune Only After Measuring++The example explicitly sets `is_streaming_input=True` to avoid the+implementation's mode default depending on `chunk_size`. It also sets+`dynamic_silence=False` to keep checkpoint/configured end-silence settings+instead of the current dynamic schedule. This is a reproducibility choice,+not a recommended threshold or a latency guarantee. Inspect+`max_end_silence_time`, `max_single_segment_time` and model configuration+before tuning; their durations are milliseconds. Input chunk duration is not+the total detection latency or a promise of phonetic boundary accuracy.++Keep raw boundary events, model/SDK revisions, configuration and the original+audio when investigating clipping. A missing or late boundary is not fixed by+globally padding every segment. If a final start remains unclosed, report it;+do not fabricate an end at the recording duration.++## SDK and Deployment Boundaries++- This recipe emits boundaries, not ASR text or subtitles. Combining VAD, ASR,+ punctuation and speakers has a different [SDK pipeline contract](python_api.md).+- Python cache events are not a C++ server wire protocol; use the+ [WebSocket protocol](../runtime/docs/websocket_protocol.md) for that service.+- Third-party OpenMOSS [MOSS-Transcribe-Diarize](moss_transcribe_diarize.md)+ provides its own joint transcription/diarization path. Do not add this external+ VAD stage as an assumed prerequisite.+- Source references: [upstream demo](../examples/industrial_data_pretraining/fsmn_vad_streaming/demo.py),+ [model implementation](../funasr/models/fsmn_vad_streaming/model.py),+ [buffer tests](../tests/test_fsmn_vad_streaming_buffers.py) and+ [dynamic-silence tests](../tests/test_dynamic_streaming_vad.py).+ [Recipe tests](../tests/test_streaming_vad_docs.py) execute chunk arithmetic and+ event assembly without downloading weights; they are not acoustic accuracy,+ live microphone, or deployment benchmarks.diff --git a/docs/streaming_vad_zh.md b/docs/streaming_vad_zh.mdnew file mode 100644index 000000000..93a4f13a3--- /dev/null+++ b/docs/streaming_vad_zh.md@@ -0,0 +1,148 @@+简体中文 | [English](streaming_vad.md)++# 流式语音活动检测++使用 FSMN VAD,按顺序输入同一条录音的音频块,接收语音起点与终点事件。+VAD 不负责转写、说话人身份识别或语言学断句。首次离线 VAD 调用见+[SDK 教程](tutorial/README_zh.md);流式 ASR 请使用独立的+[Paraformer 缓存示例](python_api_zh.md),不要混用两者的协议。++## 准备模型与音频++先完成[安装检查](installation/installation_zh.md),准备+`iic/speech_fsmn_vad_zh-cn-16k-common-pytorch` 的完整本地快照,+包括配置、前端文件与权重,记录实际模型 revision 并阅读模型自己的许可证。+本例针对该 16 kHz 检查点,不适用于任意 VAD 或流式 ASR 模型。++输入必须为非空、单声道、16 kHz WAV。下方完整程序的两个命令行位置参数+依次是本地模型目录和音频路径。它一次读取文件,再模拟按序到达的音频块;+不是麦克风采集或 WebSocket 服务,整段文件仍会占用内存。++## 理解边界事件++返回的每个字典包含 `value` 列表。每一对边界的单位都是**毫秒**,+相对于**本条流的起点**,不是当前音频块的起点,不要再叠加块偏移量。++| 事件 | 含义 |+| --- | --- |+| `[]` | 没有新边界;之前的语音段仍可能未结束。 |+| `[[start, -1]]` | 检测到起点,保存它,等待后续调用给出终点。 |+| `[[-1, end]]` | 之前开始的语音段结束。 |+| `[[start, end]]` | 本次已经获得完整起止边界。 |++一次调用可以返回多对边界。`-1` 是缺少边界的标记,不能拿它直接切音频。+每条流独立保存待配对的起点。下面的严格示例遇到异常事件顺序会报错,+不会凭空补起点或悄悄覆盖尚未结束的语音段。++## 按序输入音频块++两个辅助函数属于应用示例,不是新增的 FunASR API。将 `is_final=True`+放在最后一个**非空**音频块上;音频长度恰好整除块长度时也遵守这一规则。++```python+import argparse+from pathlib import Path+import soundfile as sf+from funasr import AutoModel+++def chunk_ranges(length, stride):+ if length <= 0 or stride <= 0:+ raise ValueError("Audio length and chunk stride must be positive")+ for start in range(0, length, stride):+ end = min(start + stride, length)+ yield start, end, end == length+++def consume_events(events, pending_start):+ spans = []+ for start_ms, end_ms in events:+ if start_ms < 0 and end_ms < 0:+ raise ValueError("An event must provide a boundary")+ if start_ms >= 0:+ if pending_start is not None:+ raise ValueError("New start before the previous speech span ended")+ pending_start = start_ms+ if end_ms >= 0:+ if pending_start is None or end_ms < pending_start:+ raise ValueError("End without a matching earlier start")+ spans.append((pending_start, end_ms))+ pending_start = None+ return pending_start, spans+++parser = argparse.ArgumentParser()+parser.add_argument("model_dir")+parser.add_argument("audio")+args = parser.parse_args()+model_dir = Path(args.model_dir).expanduser().resolve(strict=True)+if not model_dir.is_dir():+ raise ValueError("Expected a complete local FSMN VAD model directory")+speech, sample_rate = sf.read(args.audio, dtype="float32")+if sample_rate != 16000 or speech.ndim != 1 or len(speech) == 0:+ raise ValueError("Expected nonempty mono 16 kHz audio")++model = AutoModel(+ model=str(model_dir), device="cpu", ncpu=4,+ disable_update=True, trust_remote_code=False,+)+chunk_ms = 200+stride = sample_rate * chunk_ms // 1000+cache = {}+pending_start = None+for start, end, final in chunk_ranges(len(speech), stride):+ results = model.generate(+ input=speech[start:end], fs=sample_rate, cache=cache,+ chunk_size=chunk_ms, is_streaming_input=True, is_final=final,+ batch_size=1, dynamic_silence=False,+ )+ for item in results:+ pending_start, spans = consume_events(item.get("value", []), pending_start)+ for start_ms, end_ms in spans:+ print(start_ms, end_ms)+if pending_start is not None:+ print("Unclosed speech start (ms):", pending_start)+# End this session; never share its cache with another recording.+cache = {}+```++16 kHz 下,`chunk_size=200` 代表 200 毫秒,即 3200 个输入采样点。+FSMN VAD 的这个参数是单个时长数值,不是 Paraformer 流式模型的三元素列表。+同一条流保持同一个缓存字典和固定块设置,使用 `batch_size=1`。+不同录音、用户或取消后的新会话必须使用新缓存。当前实现会在非空最终调用后+重新初始化缓存;应用结束会话时仍应丢弃旧缓存。++不要另加一次空输入来刷新本例的尾部:[当前推理实现](../funasr/models/fsmn_vad_streaming/model.py)+会在空输入时提前返回,尚未走到正常的收尾路径。实时输入若事先不知道何时结束,+可保留最新一个非空块。下一个块到达时,以 `is_final=False` 发送保留块,+再保留新到达的块;只有收到结束信号时,才以 `is_final=True` 发送保留块。+这种方式增加一个输入块的缓冲延迟,不能重复发送已经消费过的音频块。+如果是取消会话,则丢弃保留的音频和缓存。++## 测量之后再调参++本例显式设置 `is_streaming_input=True`,避免流式模式默认值随+`chunk_size` 改变。同时设置 `dynamic_silence=False`,使用检查点或配置的+结束静音设置,而不是当前的动态策略。这是便于复现的选择,不是推荐阈值,+也不保证检测延迟。调参前检查 `max_end_silence_time`、+`max_single_segment_time` 和模型配置,它们的时长单位都是毫秒。+音频块时长不等于整体检测延迟,也不代表音素边界精度。++排查截断问题时保留原始边界事件、模型与 SDK revision、配置及原音频。+边界缺失或偏晚,不代表给所有段全局增加 padding 就能解决。+最终仍有未闭合起点时应报告这一状态,不要擅自把录音长度当作终点。++## SDK 与部署边界++- 本例只产生语音边界,不输出转写或字幕。组合 VAD、ASR、标点和说话人时,+ 应遵循另外的 [SDK 流水线契约](python_api_zh.md)。+- Python 缓存事件不是 C++ 服务网络协议;连接对应服务请看+ [WebSocket 协议](../runtime/docs/websocket_protocol_zh.md)。+- 第三方 OpenMOSS 的 [MOSS-Transcribe-Diarize](moss_transcribe_diarize_zh.md)+ 有自己的联合转写与说话人分离路径,不要假定它必须增加本例的外部 VAD。+- 实现依据:[原始示例](../examples/industrial_data_pretraining/fsmn_vad_streaming/demo.py)、+ [模型源码](../funasr/models/fsmn_vad_streaming/model.py)、+ [缓冲测试](../tests/test_fsmn_vad_streaming_buffers.py)、+ [动态静音测试](../tests/test_dynamic_streaming_vad.py)。+ [本文示例测试](../tests/test_streaming_vad_docs.py) 无需下载权重,只执行分块计算+ 和事件配对;不构成声学准确率、实时麦克风或部署性能测试。diff --git a/tests/test_streaming_vad_docs.py b/tests/test_streaming_vad_docs.pynew file mode 100644index 000000000..108c70a69--- /dev/null+++ b/tests/test_streaming_vad_docs.py@@ -0,0 +1,67 @@+"""Execute the documented event/chunk handling without loading model weights."""++import ast+from pathlib import Path++import pytest+from bs4 import BeautifulSoup+from markdown import markdown++ROOT = Path(__file__).resolve().parents[1]++[email protected](params=["streaming_vad.md", "streaming_vad_zh.md"])+def recipe(request):+ return lambda: load_recipe(ROOT / "docs" / request.param)+++def load_recipe(path):+ assert path.is_file(), f"Missing runnable guide: {path}"+ soup = BeautifulSoup(markdown(path.read_text(), extensions=["fenced_code"]), "html.parser")+ blocks = soup.select("code.language-python")+ assert len(blocks) == 1+ tree = ast.parse(blocks[0].get_text())+ functions = [node for node in tree.body if isinstance(node, ast.FunctionDef)]+ scope = {}+ exec(compile(ast.Module(body=functions, type_ignores=[]), str(path), "exec"), scope)+ return scope++[email protected]("length,want", [+ (1, [(0, 1, True)]),+ (3200, [(0, 3200, True)]),+ (3201, [(0, 3200, False), (3200, 3201, True)]),+ (6400, [(0, 3200, False), (3200, 6400, True)]),+])+def test_last_nonempty_chunk_is_final(recipe, length, want):+ recipe = recipe()+ assert list(recipe["chunk_ranges"](length, 3200)) == want++[email protected]("length,stride", [(0, 3200), (-1, 3200), (10, 0), (10, -1)])+def test_invalid_audio_chunk_dimensions_are_rejected(recipe, length, stride):+ recipe = recipe()+ with pytest.raises(ValueError):+ list(recipe["chunk_ranges"](length, stride))+++def test_partial_events_pair_across_calls_without_chunk_offsets(recipe):+ recipe = recipe()+ consume = recipe["consume_events"]+ pending, spans = consume([[100, -1]], None)+ assert (pending, spans) == (100, [])+ pending, spans = consume([], pending)+ assert (pending, spans) == (100, [])+ pending, spans = consume([[-1, 600], [800, 1000], [1200, -1]], pending)+ assert (pending, spans) == (1200, [(100, 600), (800, 1000)])+ assert consume([[-1, 1500]], pending) == (None, [(1200, 1500)])++[email protected]("events,pending", [+ ([[-1, 600]], None), ([[100, 50]], None),+ ([[200, -1]], 100), ([[-1, -1]], None),+])+def test_invalid_event_order_does_not_silently_invent_spans(recipe, events, pending):+ recipe = recipe()+ with pytest.raises(ValueError):+ recipe["consume_events"](events, pending)diff --git a/web-pages/product-site/data/documentation.json b/web-pages/product-site/data/documentation.jsonindex 1992b2a20..cee4f7abe 100644--- a/web-pages/product-site/data/documentation.json+++ b/web-pages/product-site/data/documentation.json@@ -17,6 +17,7 @@{"slug": "migration", "group": "start", "zh": "从 Whisper 与云 API 迁移", "en": "Migrate from Whisper & cloud APIs", "source_zh": "docs/migration_from_whisper_zh.md", "source_en": "docs/migration_from_whisper.md"},{"slug": "moss-transcribe-diarize", "group": "models", "zh": "MOSS 转写与说话人分离", "en": "MOSS transcription & diarization", "source_zh": "docs/moss_transcribe_diarize_zh.md", "source_en": "docs/moss_transcribe_diarize.md"},{"slug": "model-zoo", "group": "models", "zh": "Model Zoo", "en": "Model Zoo", "source_zh": "model_zoo/readme_zh.md", "source_en": "model_zoo/readme.md"},+ {"slug": "streaming-vad", "group": "models", "zh": "流式语音活动检测", "en": "Streaming voice activity detection", "source_zh": "docs/streaming_vad_zh.md", "source_en": "docs/streaming_vad.md"},{"slug": "training", "group": "train", "zh": "微调与评测", "en": "Fine-tuning & evaluation", "source_zh": "docs/training_zh.md", "source_en": "docs/training.md"},{"slug": "model-registration", "group": "train", "zh": "自定义模型注册", "en": "Custom model registration", "source_zh": "docs/model_registration_zh.md", "source_en": "docs/model_registration.md"},{"slug": "docker", "group": "deploy", "zh": "容器环境与镜像选择", "en": "Containers & image selection", "source_zh": "docs/installation/docker_zh.md", "source_en": "docs/installation/docker.md"},