Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

docs: rewrite English vLLM guide with high-quality manual translation

游雁8325d5eabf29
diff --git a/docs/vllm_guide.md b/docs/vllm_guide.md
index 6c65c028c..4570d3b60 100644
--- a/docs/vllm_guide.md
+++ b/docs/vllm_guide.md
@@ -1,133 +1,164 @@
# FunASR vLLM Inference Engine Guide
-FunASR integrates the vLLM high-throughput inference engine to accelerate autoregressive decoding of LLM-based ASR models. Three modes are supported: offline SDK inference, streaming SDK inference, and production-grade WebSocket real-time service.
+---
+
+## Benchmark
+
+**Test set**: 184 files, 11,541 seconds total. Models: Fun-ASR-Nano / GLM-ASR-Nano.
+
+| Model | Engine | VAD | RTFx | CER | Notes |
+|-------|--------|-----|------|-----|-------|
+| Fun-ASR-Nano | PyTorch | dynamic | 21 | 8.06% | Baseline |
+| Fun-ASR-Nano | **vLLM batch** | dynamic | **340** | **8.20%** | 16x speedup |
+| Fun-ASR-Nano | **Offline service (no SPK)** | dynamic | **102** | 8.14% | |
+| Fun-ASR-Nano | **Offline service (+SPK)** | dynamic | **46** | 8.19% | SPK off by default |
+| GLM-ASR-Nano | **vLLM batch** | fixed | **265** | 12.93% | No long-audio support |
+
+> vLLM matches PyTorch CER exactly (delta < 0.2%) while achieving 16–340x speedup.
---
## Table of Contents
-1. [Overview](#1-overview)
-2. [Installation & Environment](#2-installation--environment)
+1. [Installation & Environment](#1-installation--environment)
+2. [vLLM Engine Architecture](#2-vllm-engine-architecture)
3. [Offline SDK Inference](#3-offline-sdk-inference)
4. [Streaming SDK Inference](#4-streaming-sdk-inference)
-5. [Real-time WebSocket Service](#5-real-time-websocket-service)
-6. [Performance Comparison](#6-performance-comparison)
-7. [API Reference](#7-api-reference)
-8. [FAQ](#8-faq)
+5. [Offline Speech Recognition Service](#5-offline-speech-recognition-service)
+6. [Streaming Speech Recognition Service](#6-streaming-speech-recognition-service)
+7. [Dynamic VAD](#7-dynamic-vad)
+8. [API Reference](#8-api-reference)
+9. [FAQ](#9-faq)
---
-## 1. Overview
+## 1. Installation & Environment
-### Supported Models
+```bash
+pip install funasr>=1.3.0
+pip install vllm>=0.12.0
+pip install safetensors tiktoken websockets regex fastapi uvicorn python-multipart
+
+cd /path/to/FunASR && pip install -e .
+```
-| Model | vLLM Support | Notes |
-|-------|-------------|-------|
-| **FunASRNano** | ✓ | Qwen3-0.6B LLM, supports offline and streaming |
-| **LLMASR** | ✓ | Whisper + Qwen/Vicuna/LLaMA |
-| **GLMASR** | ✓ | GLM-ASR-Nano |
-| **QwenAudioWarp** | ✓ | Qwen-Audio |
-| Paraformer | ✗ | Non-autoregressive model (CIF predictor), no LLM decoding |
-| SenseVoice | ✗ | Whisper-like encoder-decoder, not LLM |
-| Conformer/Transformer | ✗ | CTC/attention decoding, not LLM |
-
-### Architecture
-
-```
-┌────────────────────────────────────────────────────────┐
-│ FunASR + vLLM │
-├────────────────────────────────────────────────────────┤
-│ │
-│ Audio ──→ Frontend ──→ Audio Encoder ──→ Adaptor │
-│ (fbank) (SenseVoice) (Transformer) │
-│ │ │
-│ ▼ │
-│ Audio Embeddings │
-│ │ │
-│ Text Prompt ──→ Tokenize ──→ Embed ──→ [Concat] │
-│ (hotwords/language/itn) │ │
-│ ▼ │
-│ ┌──────────────┐ │
-│ │ vLLM Engine │ │
-│ │ (Qwen3 etc) │ │
-│ │ TP parallel │ │
-│ └──────┬───────┘ │
-│ ▼ │
-│ Generated Text │
-│ │
-│ (Optional) CTC Forced Alignment ──→ Character-level │
-│ timestamps │
-└────────────────────────────────────────────────────────┘
-```
-
-### Three Usage Modes
-
-| Mode | Entry Point | Use Case |
-|------|-------------|----------|
-| [Offline SDK Inference](#3-offline-sdk-inference) | `AutoModelVLLM` / `FunASRNanoVLLM` | Large-scale transcription, batch processing |
-| [Streaming SDK Inference](#4-streaming-sdk-inference) | `FunASRNanoStreamingVLLM` | Real-time subtitles (SDK integration) |
-| [WebSocket Real-time Service](#5-real-time-websocket-service) | `serve_realtime_ws.py` | Production deployment, multi-client access |
+**Hardware**: GPU ≥ 8 GB VRAM, CUDA ≥ 11.8. 16 GB+ recommended.
---
-## 2. Installation & Environment
+## 2. vLLM Engine Architecture
-### Dependencies
+### Overall Architecture
-```bash
-pip install funasr>=1.3.0
-pip install vllm>=0.12.0
-pip install safetensors tiktoken websockets regex
+FunASR's vLLM integration splits the ASR model into two independently running components:
-# FunASR development mode install (requires auto_model_vllm)
-cd /path/to/FunASR && pip install -e .
```
+┌──────────────────────────────────────────────────────────────┐
+│ FunASR + vLLM Inference Architecture │
+├──────────────────────────────────────────────────────────────┤
+│ │
+│ ┌─────────────── PyTorch (single GPU) ───────────┐ │
+│ │ │ │
+│ │ Audio ──→ Frontend ──→ Audio Encoder ──→ Adaptor │
+│ │ (fbank) (SenseVoice/ (Transformer/ │
+│ │ Whisper) MLP) │
+│ │ │ │
+│ │ ▼ │
+│ │ Audio Embeddings │
+│ │ │ │
+│ │ Text Prompt ──→ Tokenize ──→ Embed │
+│ │ (system/user/ │ │
+│ │ hotwords/language) │ │
+│ │ ▼ │
+│ │ [Concat Embeddings] │
+│ └─────────────────────────────────┼─────────────┘ │
+│ │ │
+│ ▼ EmbedsPrompt │
+│ ┌─────────────── vLLM Engine ────────────────────┐ │
+│ │ │ │
+│ │ PagedAttention + Continuous Batching │ │
+│ │ KV Cache management + CUDA Graph │ │
+│ │ Tensor Parallel (multi-GPU) │ │
+│ │ │ │
+│ │ Qwen3-0.6B / Llama-2B (LLM decoding) │ │
+│ │ │ │
+│ └────────────────────┬───────────────────────────┘ │
+│ │ │
+│ ▼ │
+│ Generated Text │
+│ │ │
+│ ┌────────────────────┼──────────────────────────┐ │
+│ │ (Optional) CTC Decoder ──→ Forced Alignment │ │
+│ │ ──→ Character-level timestamps │ │
+│ └───────────────────────────────────────────────┘ │
+└──────────────────────────────────────────────────────────────┘
+```
+
+### Why vLLM?
+
+| Feature | PyTorch generate() | vLLM |
+|---------|-------------------|------|
+| KV Cache management | Fixed allocation, wastes memory | PagedAttention, on-demand allocation |
+| Batching | Manual padding required | Continuous Batching, automatic scheduling |
+| CUDA optimization | None | CUDA Graph + operator fusion |
+| Multi-GPU parallelism | Manual implementation | Tensor Parallel with one-line config |
+| Throughput | RTFx ~20 | **RTFx 340+** |
-### Hardware Requirements
+### Supported Models
+
+| Model | LLM component | Audio encoder | vLLM speedup |
+|-------|--------------|---------------|-------------|
+| **Fun-ASR-Nano** | Qwen3-0.6B | SenseVoice | ✓ 21.7x |
+| **GLM-ASR-Nano** | Llama-2B | Whisper-like | ✓ 7.6x |
+| LLMASR | Qwen/Vicuna | Whisper | ✓ |
+| Paraformer | No LLM | — | ✗ Non-autoregressive |
+| SenseVoice | No LLM | — | ✗ Encoder-decoder |
+
+### Key Implementation Details
-| Configuration | Minimum | Recommended |
-|--------------|---------|-------------|
-| GPU Memory | 8GB | 16GB+ |
-| CUDA | 11.8 | 12.0+ |
-| GPU Count | 1 | 2+ (tensor parallel) |
+1. **Weight separation**: LLM weights are extracted from `model.pt` and converted to HuggingFace format for vLLM loading
+2. **EmbedsPrompt**: Audio embeddings and text embeddings are concatenated and fed to vLLM as a single prompt embedding
+3. **use_low_frame_rate**: Fun-ASR-Nano's adaptor output must be truncated to the correct token count via a formula (critical for consistency)
+4. **Batch encode**: Multiple audio files pass through `extract_fbank` → `audio_encoder` → `audio_adaptor` in a single forward pass
+5. **CTC timestamps**: Encoder output is retained; after text generation, forced alignment yields character-level timing
---
## 3. Offline SDK Inference
-Best for large-scale audio transcription and offline batch processing. vLLM's batching capability provides the greatest advantage in this scenario.
+Best suited for large-scale audio transcription and offline batch processing. vLLM's batching capability provides the greatest advantage in this scenario.
### Design Principles
-The offline SDK inference splits the ASR pipeline into two stages executed independently:
+Offline SDK inference splits the ASR pipeline into two stages executed independently:
```
┌─────────────────────────────────────────────────────────────────────┐
-│ Stage 1: Audio Encoding (PyTorch, single GPU) │
+│ Stage 1: Audio Encoding (PyTorch, single GPU) │
├─────────────────────────────────────────────────────────────────────┤
│ │
-│ Audio file list ──→ Group (batch=8) ──→ Frontend (Fbank) │
-│ │ │ │
-│ │ ▼ │
-│ │ SenseVoice Encoder │
-│ │ │ │
-│ │ ▼ │
-│ │ Audio Adaptor │
-│ │ (dim transform + LFR truncation)│
-│ │ │ │
-│ └─── Shared text prompt encoding ─┐ ▼ │
-│ (system/hotwords/language) │ audio_embeds │
-│ │ │ │ │
-│ ▼ │ ▼ │
+│ Audio file list ──→ Group (batch of 8) ──→ Frontend (Fbank) │
+│ │ │ │
+│ │ ▼ │
+│ │ SenseVoice Encoder │
+│ │ │ │
+│ │ ▼ │
+│ │ Audio Adaptor │
+│ │ (dim transform + LFR trunc) │
+│ │ │ │
+│ └─── Shared text prompt encoding ────┐ ▼ │
+│ (system/hotwords/language) │ audio_embeds │
+│ │ │ │ │
+│ ▼ │ ▼ │
│ prefix_emb ──→ [concat: prefix | audio | suffix] │
-│ │ │
-│ ▼ │
-│ EmbedsPrompt (N samples) │
-└──────────────────────────────────────────────┼─────────────────────┘
- │
- ▼
+│ │ │
+│ ▼ │
+│ EmbedsPrompt (N samples) │
+└──────────────────────────────────────────────────┼─────────────────┘
+ │
+ ▼
┌─────────────────────────────────────────────────────────────────────┐
-│ Stage 2: LLM Decoding (vLLM, multi-GPU Tensor Parallel) │
+│ Stage 2: LLM Decoding (vLLM, multi-GPU Tensor Parallel) │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ EmbedsPrompt × N ──→ vLLM Continuous Batching │
@@ -137,7 +168,7 @@ The offline SDK inference splits the ASR pipeline into two stages executed indep
│ Generated token_ids × N │
│ │ │
│ ▼ │
-│ Decode + post-processing (clean special tokens) │
+│ Decode + post-processing (strip special tokens) │
│ │ │
│ ▼ │
│ (Optional) CTC Forced Alignment → char timestamps│
@@ -146,20 +177,20 @@ The offline SDK inference splits the ASR pipeline into two stages executed indep
**Key design decisions:**
-1. **Weight separation**: On first run, extracts `llm.*` prefixed weights from `model.pt` and saves them in HuggingFace safetensors format for vLLM (cached in `Qwen3-0.6B-vllm/` directory)
-2. **Embedding concatenation**: Text prompt is encoded via the LLM's `embed_tokens` layer, then concatenated with audio adaptor output along the sequence dimension: `[prefix_emb | audio_emb | suffix_emb]`, submitted to vLLM as `EmbedsPrompt`
-3. **Low Frame Rate truncation**: Adaptor output is truncated to the correct length using: `fake_token_len = ((((fbank_len - 3 + 2) // 2 - 3 + 2) // 2) - 1) // 2 + 1`, ensuring consistency with PyTorch training
-4. **Batch audio encoding**: Multiple audio files are grouped (batch_size=8) through encoder + adaptor forward pass, reducing GPU kernel launch overhead
-5. **Shared text prompt**: When hotwords/language are the same within a batch, prefix_emb and suffix_emb are computed only once
-6. **CTC timestamps**: Encoder output is preserved; after LLM generates text, forced alignment produces character-level timestamps
+1. **Weight separation**: On first run, weights with the `llm.*` prefix are extracted from `model.pt` and saved in HuggingFace safetensors format for vLLM (cached in the `Qwen3-0.6B-vllm/` directory)
+2. **Embedding concatenation**: The text prompt is encoded through the LLM's `embed_tokens` layer into embeddings, then concatenated with the audio adaptor output along the sequence dimension: `[prefix_emb | audio_emb | suffix_emb]`, and submitted to vLLM as an `EmbedsPrompt`
+3. **Low Frame Rate truncation**: Adaptor output must be truncated to the correct length using: `fake_token_len = ((((fbank_len - 3 + 2) // 2 - 3 + 2) // 2) - 1) // 2 + 1`, ensuring consistency with the PyTorch training pipeline
+4. **Batch audio encoding**: Multiple audio files are grouped in batches of 8 through the encoder + adaptor forward pass, reducing GPU kernel launch overhead
+5. **Shared text prompt**: When hotwords and language are identical within a batch, prefix_emb and suffix_emb are computed only once
+6. **CTC timestamps**: Encoder output is preserved; after LLM text generation, forced alignment produces character-level timestamps
**Why faster than PyTorch generate()?**
| Dimension | PyTorch | vLLM |
|-----------|---------|------|
| KV Cache | Fixed pre-allocation (wastes memory) | PagedAttention on-demand allocation |
-| Batching | Manual padding required | Continuous Batching auto-scheduling |
-| CUDA | Sequential per-sample | CUDA Graph + operator fusion |
+| Batching | Manual padding alignment | Continuous Batching auto-scheduling |
+| CUDA | Sequential per-sample execution | CUDA Graph + operator fusion |
| Multi-GPU | Manual implementation | Tensor Parallel one-line config |
| Result | RTFx ~20 | **RTFx 340+** (16x speedup) |
@@ -195,7 +226,7 @@ engine = FunASRNanoVLLM.from_pretrained(
)
results = engine.generate(
- inputs="wav.scp", # supports scp/jsonl/file list
+ inputs="wav.scp", # supports scp/jsonl/file lists
hotwords=["开放时间"],
language="中文",
max_new_tokens=512,
@@ -221,17 +252,17 @@ python demo_vllm.py --input audio.wav --hotwords 张三 北京 --output results.
## 4. Streaming SDK Inference
-Processes audio in 720ms chunks incrementally, outputting progressively stable recognition results. Suitable for SDK-integrated real-time subtitle scenarios.
+Processes audio in 720 ms chunks incrementally, outputting progressively stable recognition results. Suited for SDK-integrated real-time subtitle scenarios.
-### Design Principle
+### Design Principles
```
-Audio stream (720ms chunks)
- │ Cumulative re-encoding (each chunk contains all audio from start to current)
+Audio stream (720 ms chunks)
+ │ Cumulative re-encoding (each chunk covers all audio from the start)
▼
┌──────────────────────────┐
-│ Stage 1: First 10 chunks │ ← No prev_text, batch generation
-│ Find stable output │
+│ Stage 1: First 10 chunks │ ← No prev_text; batch generation
+│ Identify stable output │
└──────────┬───────────────┘
▼
┌──────────────────────────┐
@@ -261,408 +292,392 @@ for result in engine.streaming_generate("audio.wav", language="中文"):
### Output Characteristics
-| Accumulated Audio | Output Quality |
+| Accumulated audio | Output quality |
|-------------------|---------------|
-| < 1.5s | Empty or noise |
-| 1.5-3.0s | Partially correct |
-| > 3.0s | Accurate output |
+| < 1.5 s | Empty or noise |
+| 1.5–3.0 s | Partially correct |
+| > 3.0 s | Accurate output |
> Note: `repetition_penalty=1.3` is hardcoded internally to prevent short-chunk repetition degradation.
---
-## 5. Real-time WebSocket Service
-
-Production-grade real-time speech recognition service integrating VAD segmentation + vLLM inference + speaker diarization + hotwords.
-
-### Inference Logic
+## 5. Offline Speech Recognition Service
-**Core design: VAD endpoint-based segment decoding (not fixed chunks)**
+### 5.1 Service Architecture
```
-Audio stream ──→ StreamingVAD (60ms) ──→ Endpoint detected ──→ vLLM decode segment
- │ │
- │ ▼
- │ locked_sentences (locked)
- │ │
- │ ▼
- │ SPK assign (speaker assignment)
- │
- └──→ Not ended ──→ vLLM partial decode (every 0.48s)
- │
- ▼
- partial text (preview, will be overwritten)
+Client serve_vllm.py
+ │ │
+ │── HTTP / OpenAI / WebSocket ─────────→│
+ │ │
+ │ ┌────┴────────────────────────┐
+ │ │ 1. Receive complete audio │
+ │ │ 2. Dynamic VAD (≤60 s/seg) │
+ │ │ 3. vLLM batch all segments │
+ │ │ 4. CTC timestamps (per-char)│
+ │ │ 5. Speaker diarization (opt)│
+ │ └────┬────────────────────────┘
+ │ │
+ │←── JSON result ───────────────────────│
```
-**Two inference paths:**
+**Characteristics**:
+- Processes audio only after it arrives in full — ideal for file transcription
+- Dynamic VAD preserves long segments (≤60 s), reducing boundary-cut losses
+- Batch inference over all VAD segments maximizes throughput
+- Automatically outputs character-level timestamps
+- Speaker diarization is off by default; clients can enable it
-| Path | Trigger Condition | Output |
-|------|-------------------|--------|
-| Confirmed segment decode | VAD detects silence endpoint | Locked to sentences, never changes |
-| Partial preview | Every 0.48s + new audio ≥ 960ms | Temporary text, overwritten anytime |
+### 5.2 Starting the Service
-**Dynamic VAD thresholds:**
-
-| Accumulated Duration | Silence Threshold | Effect |
-|---------------------|-------------------|--------|
-| ≤ 5s | 2.0s | Preserve short sentences |
-| 5-10s | 1.5s | Normal segmentation |
-| 10-15s | 1.0s | Start tightening |
-| 15-30s | 0.8s | Faster splitting |
-| 30-45s | 0.4s | Prevent overly long segments |
-| > 45s | 0.1s | Force split |
+```bash
+CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_vllm.py \
+ --port 8899 \
+ --model FunAudioLLM/Fun-ASR-Nano-2512 \
+ --gpu-memory-utilization 0.5
+```
-**STOP final processing:**
-1. Feed remaining audio to VAD (is_final=True)
-2. Force-end currently speaking segment
-3. Global SPK re-clustering (correct speaker IDs)
-4. Return `is_final: true`
+### 5.3 Protocol 1: HTTP REST — `POST /asr`
-### Deployment
+The most feature-complete interface, supporting speaker diarization, timestamps, and hotwords.
-```bash
-cd examples/industrial_data_pretraining/fun_asr_nano
+**Request**: `multipart/form-data`
-# Single GPU
-CUDA_VISIBLE_DEVICES=0 python serve_realtime_ws.py --port 10095 --language 中文
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `file` | file | required | Audio file (wav/mp3/flac) |
+| `language` | string | None | Language ("中文" / "English" / ...), None for auto |
+| `hotwords` | string | "" | Hotwords, comma-separated |
+| `spk` | bool | false | Enable speaker diarization |
+| `timestamp` | bool | true | Output character-level timestamps |
-# Multi-GPU
-CUDA_VISIBLE_DEVICES=0,1 python serve_realtime_ws.py \
- --port 10095 --tensor-parallel-size 2 --language 中文
+**Response**:
-# Full parameters
-python serve_realtime_ws.py \
- --port 10095 \
- --model FunAudioLLM/Fun-ASR-Nano-2512 \
- --hub ms \
- --device cuda:0 \
- --decode-interval 0.48 \
- --hotword-file hotword_list \
- --language 中文 \
- --dtype bf16 \
- --tensor-parallel-size 1 \
- --gpu-memory-utilization 0.8 \
- --max-model-len 2048
+```json
+{
+ "text": "Full transcription text",
+ "segments": [
+ {
+ "text": "Segment text",
+ "start": 1.7,
+ "end": 14.8,
+ "speaker": "SPK0",
+ "words": [
+ {"word": "砸", "start": 2.02, "end": 2.08},
+ {"word": "了", "start": 2.26, "end": 2.32}
+ ]
+ }
+ ],
+ "duration": 227.4,
+ "processing_time": 3.422,
+ "rtf": 0.015
+}
```
-### Clients
+**Client examples**:
-| Client | Usage |
-|--------|-------|
-| Browser | Open `client_mic.html` (microphone/file/hotwords/speakers) |
-| Python CLI | `python client_python.py --server ws://localhost:10095 --mic` |
-| Test script | `python client_test.py --server ws://localhost:10095 --file audio.wav` |
+```bash
+# cURL
+curl -X POST http://localhost:8899/asr \
+ -F "[email protected]" -F "language=中文" -F "spk=true"
+```
-Remote access requires SSH port forwarding: `ssh -L 10095:localhost:10095 <server>`
+```python
+# Python requests
+import requests
+resp = requests.post("http://localhost:8899/asr",
+ files={"file": open("audio.wav", "rb")},
+ data={"language": "中文", "spk": "true"})
+result = resp.json()
+```
-### WebSocket Protocol
+```javascript
+// JavaScript fetch
+const form = new FormData();
+form.append("file", audioBlob, "audio.wav");
+form.append("language", "中文");
+form.append("spk", "true");
+const resp = await fetch("http://localhost:8899/asr", { method: "POST", body: form });
+const result = await resp.json();
+```
-**Client → Server:**
+### 5.4 Protocol 2: OpenAI Whisper Compatible — `POST /v1/audio/transcriptions`
-| Message | Format | Description |
-|---------|--------|-------------|
-| Start session | `"START"` | Initialize session |
-| Set hotwords | `"HOTWORDS:word1,word2"` | Optional |
-| Set language | `"LANGUAGE:中文"` | Optional |
-| Audio data | `bytes` | PCM16, 16kHz, mono |
-| End session | `"STOP"` | Trigger final decoding |
+Compatible with the OpenAI Whisper API standard; works directly with the OpenAI SDK.
-**Server → Client:**
+**Request**: `multipart/form-data`
-```json
-// Events
-{"event": "started"}
-{"event": "hotwords_set", "hotwords": ["word1", "word2"]}
-{"event": "language_set", "language": "中文"}
-{"event": "stopped"}
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `file` | file | required | Audio file |
+| `model` | string | "fun-asr-nano" | Model name (compatibility field) |
+| `language` | string | None | Language |
+| `response_format` | string | "json" | "json" / "text" / "verbose_json" |
+| `timestamp_granularities` | string | "word" | "word" / "segment" |
+| `spk` | bool | false | Speaker diarization (FunASR extension) |
-// Real-time results
-{
- "sentences": [{"text": "confirmed text", "start": 1700, "end": 5500, "spk": 0}],
- "partial": "currently speaking...",
- "partial_start_ms": 5800,
- "duration_ms": 7200,
- "is_final": false
-}
+**Response** (`verbose_json`):
-// Final result
+```json
{
- "sentences": [{"text": "...", "start": ..., "end": ..., "spk": ...}, ...],
- "partial": "",
- "partial_start_ms": 0,
- "duration_ms": 10000,
- "is_final": true
+ "task": "transcribe",
+ "language": "zh",
+ "duration": 5.17,
+ "text": "我一直没有照顾孩子,但是我想要抚养权。",
+ "segments": [
+ {
+ "id": 0, "start": 0.0, "end": 5.15,
+ "text": "我一直没有照顾孩子,但是我想要抚养权。",
+ "words": [{"word": "我", "start": 0.42, "end": 0.48}, ...]
+ }
+ ]
}
```
-**Interaction sequence:**
+**Client examples**:
+```python
+# OpenAI SDK (recommended)
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8899/v1", api_key="none")
+result = client.audio.transcriptions.create(
+ model="fun-asr-nano",
+ file=open("audio.wav", "rb"),
+ response_format="verbose_json",
+)
+print(result.text)
```
-Client Server
- │── "START" ─────────────────→│
- │←─ {"event":"started"} ──────│
- │── "HOTWORDS:张三,北京" ────→│
- │←─ {"event":"hotwords_set"} ─│
- │── "LANGUAGE:中文" ─────────→│
- │←─ {"event":"language_set"} ─│
- │── [audio bytes] ───────────→│
- │←─ {sentences,partial} ──────│ (real-time push)
- │── [audio bytes] ───────────→│
- │←─ {sentences,partial} ──────│
- │── "STOP" ──────────────────→│
- │←─ {sentences,is_final:true} │ (final result)
- │←─ {"event":"stopped"} ──────│
-```
-
-### Hotword File
-
-Default filename `hotword_list` (specified via `--hotword-file`), one word per line:
-```
-张三
-李四
-北京大学
+```bash
+# cURL
+curl -X POST http://localhost:8899/v1/audio/transcriptions \
+ -F "[email protected]" -F "model=fun-asr-nano" -F "response_format=verbose_json"
```
-Can also be set dynamically via WebSocket: `HOTWORDS:word1,word2,word3`
+### 5.5 Protocol 3: WebSocket — `ws://host:port/ws`
----
+WebSocket interface for the offline service. Send complete audio, then receive results. Speaker clustering is performed automatically on STOP, and results include the `spk` field.
-## 6. Performance Comparison
+**Client → Server**:
-### Offline Inference
+| Message | Description |
+|---------|-------------|
+| `"START"` | Begin session |
+| `"LANGUAGE:中文"` | Set language (optional) |
+| `"HOTWORDS:word1,word2"` | Set hotwords (optional) |
+| `[binary]` | PCM16 16 kHz mono audio data |
+| `"STOP"` | End session; request recognition result |
-| Configuration | 5.6s Audio Latency | Relative Speedup |
-|--------------|-------------------|-----------------|
-| PyTorch (baseline) | 0.89s | 1x |
-| vLLM 1-GPU | 0.30s | **3x** |
-| vLLM 2-GPU TP | ~0.20s | **4.5x** |
+**Server → Client**:
-### Batch Throughput
+```json
+{"event": "started"}
+{"event": "language_set", "language": "中文"}
+{"sentences": [{"text":"...","start":..,"end":..}], "is_final": true, "duration_ms": 5170}
+{"event": "stopped"}
+```
-| Batch Size | 1-GPU | 2-GPU | 4-GPU |
-|-----------|-------|-------|-------|
-| 1 | ~1.5x | ~2x | ~2.5x |
-| 16 | ~4x | ~7x | ~12x |
-| 32 | ~5x | ~9x | ~15x |
+**Client example**:
-### WebSocket Real-time Service
+```python
+import asyncio, websockets, json, numpy as np, soundfile as sf
+
+async def offline_ws(audio_path):
+ audio, sr = sf.read(audio_path)
+ pcm = (audio * 32768).astype(np.int16)
+
+ async with websockets.connect("ws://localhost:8899/ws") as ws:
+ await ws.send("START")
+ await ws.recv()
+ await ws.send("LANGUAGE:中文")
+ await ws.recv()
+
+ # Send complete audio
+ await ws.send(pcm.tobytes())
+ await ws.send("STOP")
+
+ # Receive result
+ async for msg in ws:
+ data = json.loads(msg)
+ if data.get("is_final"):
+ for s in data["sentences"]:
+ print(f"[{s['start']/1000:.1f}s] {s['text']}")
+ break
+
+asyncio.run(offline_ws("audio.wav"))
+```
-| Metric | Value |
-|--------|-------|
-| RTF | < 0.08 |
-| First-word latency | ~480ms |
-| Total time for 30s audio | ~2.3s |
-| Concurrency | Multiple WebSocket connections |
+---
-### VAD + vLLM Pipeline
+## 6. Streaming Speech Recognition Service
-| Scenario | PyTorch Sequential | vLLM Batch | Speedup |
-|----------|-------------------|------------|---------|
-| 10 segments × 5s | ~9s | ~1.5s | **6x** |
-| 20 segments × 5s | ~18s | ~2.5s | **7x** |
+### 6.1 Service Architecture
----
+```
+Client (microphone / audio stream) serve_realtime_ws.py
+ │ │
+ │── WebSocket PCM16 16 kHz ──────────→│
+ │ (~100 ms per frame, continuous) │
+ │ │
+ │ ┌────┴─────────────────────────┐
+ │ │ Real-time loop: │
+ │ │ ├─ Dynamic VAD (60 ms chunk) │
+ │ │ ├─ Endpoint → vLLM decode │
+ │ │ ├─ No endpoint → partial │
+ │ │ └─ Streaming SPK assignment │
+ │ └────┬─────────────────────────┘
+ │ │
+ │←── JSON real-time push ─────────────│
+```
-## 7. API Reference
+**Characteristics**:
+- Audio arrives frame by frame; processing starts immediately
+- Natural sentence segmentation based on VAD endpoints
+- Confirmed segment text is locked and never changes; partial text updates in real time
+- Streaming speaker assignment + global re-clustering on STOP
+- First-word latency ~480 ms
-### AutoModelVLLM
+### 6.2 Starting the Service
-```python
-from funasr.auto.auto_model_vllm import AutoModelVLLM
+```bash
+CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
+ --port 10095 --language 中文 --hotword-file hotword_list
```
-Universal entry point that automatically detects model type and selects the corresponding vLLM implementation.
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `model` | - | Model name (hub) or local path |
-| `hub` | `"ms"` | `"ms"` / `"hf"` |
-| `device` | `"cuda:0"` | Audio encoder device |
-| `dtype` | `"bf16"` | Precision |
-| `tensor_parallel_size` | `1` | vLLM GPU parallel count |
-| `gpu_memory_utilization` | `0.8` | KV Cache memory ratio |
-| `max_model_len` | `4096` | Maximum sequence length |
-
-### AutoModelVLLM.generate()
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `inputs` | - | Audio path/list/numpy/tensor |
-| `language` | `None` | Language hint |
-| `hotwords` | `None` | Hotword list |
-| `itn` | `True` | Inverse text normalization |
-| `max_new_tokens` | `512` | Maximum generated tokens |
-| `temperature` | `0.0` | Sampling temperature |
-| `repetition_penalty` | `1.0` | Repetition penalty |
-
-**Returns**: `[{"key": str, "text": str, "timestamps": [...]}]`
-
-### FunASRNanoStreamingVLLM.from_pretrained()
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `model` | `"FunAudioLLM/Fun-ASR-Nano-2512"` | Model name or path |
-| `hub` | `"ms"` | Model source |
-| `device` | `"cuda:0"` | Device |
-| `dtype` | `"bf16"` | Precision |
-| `tensor_parallel_size` | `1` | GPU parallel count |
-| `gpu_memory_utilization` | `0.8` | Memory ratio |
-| `max_model_len` | `2048` | Sequence length |
-| `chunk_ms` | `720` | Chunk duration |
-| `rollback_chars` | `8` | Rollback character count |
-
-### streaming_generate()
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `audio_input` | - | Audio path/data |
-| `chunk_ms` | `720` | Chunk size |
-| `rollback_chars` | `8` | Rollback characters |
-| `hotwords` | `None` | Hotwords |
-| `language` | `None` | Language |
-| `max_new_tokens` | `200` | Max tokens per chunk |
-| `temperature` | `0.0` | Sampling temperature |
-
-**Yields**: `{"text", "fixed_text", "is_final", "chunk_idx", "audio_duration_ms"}`
-
-> `repetition_penalty=1.3` is hardcoded internally.
-
-### serve_realtime_ws.py Parameters
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `--port` | `10095` | WebSocket port |
-| `--model` | `FunAudioLLM/Fun-ASR-Nano-2512` | Model |
-| `--hub` | `ms` | Source |
-| `--device` | `cuda:0` | Device |
-| `--decode-interval` | `0.48` | Partial decode interval (seconds) |
-| `--hotword-file` | `hotword_list` | Hotword file |
-| `--language` | `None` | Language |
-| `--dtype` | `bf16` | Precision |
-| `--tensor-parallel-size` | `1` | GPU parallel |
-| `--gpu-memory-utilization` | `0.8` | Memory ratio |
-| `--max-model-len` | `2048` | Sequence length |
+### 6.3 WebSocket Protocol
----
+**Connection**: `ws://host:10095`
-## 8. FAQ
+**Client → Server**:
-### Q: Why is the first startup slow?
-vLLM needs to initialize KV Cache and perform CUDA Graph warmup, taking approximately 60-90 seconds. Subsequent inferences respond instantly.
+| Message | Format | Description |
+|---------|--------|-------------|
+| Start | `"START"` | Initialize session |
+| Hotwords | `"HOTWORDS:word1,word2"` | Optional |
+| Language | `"LANGUAGE:中文"` | Optional |
+| Audio | `binary` | PCM16 16 kHz mono |
+| End | `"STOP"` | Final decode + SPK re-clustering |
-### Q: How to handle CUDA OOM?
-- Reduce `gpu_memory_utilization` (e.g., 0.6)
-- Increase `tensor_parallel_size` to distribute across multiple GPUs
-- Reduce `max_model_len`
+**Server → Client**:
-### Q: Can Paraformer use vLLM?
-No. Paraformer is a non-autoregressive model (CIF predictor) where all tokens are generated in parallel without KV-cache. vLLM only accelerates autoregressive LLM decoding.
+```json
+{"event": "started"}
+{"sentences": [{"text":"你好","start":300,"end":1200,"spk":0}], "partial": "世界", "is_final": false}
+{"sentences": [...], "is_final": true}
+{"event": "stopped"}
+```
-### Q: What's the difference between WebSocket service and streaming_generate?
+**Fields**: `sentences[]` = locked segments, `partial` = text being spoken (may change), `is_final` = true after STOP.
-| | WebSocket Service | streaming_generate |
-|---|---|---|
-| Segmentation | VAD natural endpoints | Fixed 720ms chunks |
-| Inference | Decode entire VAD segment | Cumulative re-encode all audio |
-| Accuracy | Higher | Lower for first 3s |
-| Use case | Production deployment | SDK integration |
+**Sequence diagram**:
+```
+Client Server
+ │── START ───────→│
+ │←─ started ──────│
+ │── [audio] ─────→│
+ │←─ {partial} ────│
+ │── [audio] ─────→│
+ │←─ {sentences+partial} ─│ (VAD cut a sentence)
+ │── STOP ────────→│
+ │←─ {is_final:true} ────│
+ │←─ stopped ─────│
+```
-### Q: Why is output empty for the first few seconds of streaming?
-Normal. The model needs ~3 seconds of accumulated audio to produce meaningful output. This is a model characteristic, not a vLLM limitation.
+### 6.4 Client Usage
-### Q: What audio formats are supported?
-wav, mp3, flac and other mainstream formats. Sample rate is automatically converted to 16kHz.
+**Python CLI**:
+```bash
+python client_python.py --server ws://localhost:10095 --mic
+python client_python.py --server ws://localhost:10095 --file audio.wav
+```
-### Q: Browser cannot access microphone?
-Chrome requires HTTPS or localhost. For remote servers, use SSH port forwarding: `ssh -L 10095:localhost:10095 <server>`
+**Browser**: Open `client_mic.html`
-### Q: Will multiple concurrent connections interfere with each other?
-No. Each WebSocket connection has an independent session (VAD/ASR state isolation). vLLM handles scheduling internally.
+**Custom Python**:
+```python
+import asyncio, websockets, numpy as np, json
+
+async def stream(audio_path):
+ import soundfile as sf
+ audio, sr = sf.read(audio_path)
+ pcm = (audio * 32768).astype(np.int16)
+
+ async with websockets.connect("ws://localhost:10095") as ws:
+ await ws.send("START")
+ await ws.recv()
+
+ for i in range(0, len(pcm), 1600):
+ await ws.send(pcm[i:i+1600].tobytes())
+ await asyncio.sleep(0.05)
+
+ await ws.send("STOP")
+ async for msg in ws:
+ data = json.loads(msg)
+ if data.get("is_final"):
+ for s in data["sentences"]:
+ print(f"[{s['start']/1000:.1f}s] {s['text']}")
+ break
+
+asyncio.run(stream("audio.wav"))
+```
---
-## Appendix: DynamicStreamingVAD
+## 7. Dynamic VAD
-`funasr.models.fsmn_vad_streaming.dynamic_vad.DynamicStreamingVAD` is a generic dynamic-threshold streaming VAD wrapper that adjusts silence splitting thresholds dynamically based on the accumulated duration of the current speech segment, built on top of fsmn-vad.
+fsmn-vad enables dynamic silence thresholds by default. Offline and streaming modes use different configurations.
-### Design Motivation
+| Accumulated duration | Offline (preserve long segs ≤60 s) | Streaming (balance latency) |
+|---------------------|-----------------------------------|-----------------------------|
+| ≤ 5 s | 2000 ms | 2000 ms |
+| 5–10 s | 2000 ms | 1500 ms |
+| 10–15 s | 1000 ms | 1000 ms |
+| 15–20 s | 1000 ms | 800 ms |
+| 20–30 s | 800 ms | 800 ms |
+| 30–45 s | 600 ms | 400 ms |
+| 45–60 s | 200–400 ms | 100 ms |
+| > 60 s | 100 ms | 100 ms |
-fsmn-vad uses a fixed silence threshold (800ms) by default. In practice:
-- Short utterances (e.g., "okay") need longer silence before splitting, otherwise sentences get fragmented
-- Long segments (e.g., 30s+ meeting speeches) need faster splitting, otherwise ASR input becomes too long and quality degrades
+Offline mode favors longer segments to reduce boundary-cut losses; streaming mode tightens faster to reduce latency.
-### Usage
+### Customization
```python
-from funasr import AutoModel
-from funasr.models.fsmn_vad_streaming.dynamic_vad import DynamicStreamingVAD
+model.generate(input="audio.wav", silence_schedule=[(5000,1500), (20000,800), (float('inf'),300)])
+```
-vad_model = AutoModel(model="fsmn-vad", device="cuda:0")
+> GLM-ASR does not support long-segment inference; pass `dynamic_silence=False` when using it.
-# Use default threshold configuration
-vad = DynamicStreamingVAD(vad_model)
+---
-# Or customize thresholds
-vad = DynamicStreamingVAD(
- vad_model,
- silence_schedule=[
- (3000, 1500), # accumulated <=3s: wait 1.5s silence
- (10000, 800), # accumulated 3-10s: wait 0.8s
- (float('inf'), 300), # accumulated >10s: wait 0.3s
- ],
- speech_noise_thres=0.5,
-)
-```
+## 8. API Reference
-#### Streaming Call
+| Parameter | AutoModelVLLM | serve_vllm.py | serve_realtime_ws.py |
+|-----------|--------------|---------------|---------------------|
+| model | ✓ | --model | --model |
+| gpu_memory_utilization | ✓ | --gpu-memory-utilization | --gpu-memory-utilization |
+| tensor_parallel_size | ✓ | — | --tensor-parallel-size |
+| max_model_len | ✓ | --max-model-len | --max-model-len |
+| language | generate() param | API param | --language / LANGUAGE: |
+| hotwords | generate() param | API param | --hotword-file / HOTWORDS: |
-```python
-import torch
+---
-for audio_chunk in audio_stream: # real-time audio stream
- segments = vad.feed(torch.from_numpy(audio_chunk).float())
- for seg in segments:
- print(f"Speech: {seg[0]}-{seg[1]}ms")
+## 9. FAQ
-# At the end
-final_segments = vad.finalize()
-```
+**Q: Offline or streaming?**
+Complete files → offline (high throughput). Microphone / live stream → streaming (low latency).
-#### Non-streaming Call
+**Q: Can GLM-ASR use dynamic VAD?**
+It does not support long-segment inference. Use `dynamic_silence=False`.
-```python
-segments = vad.process(full_audio_tensor)
-for seg in segments:
- print(f"Speech: {seg[0]}-{seg[1]}ms")
-```
-
-### Default Threshold Configuration
-
-| Accumulated Duration | Silence Threshold | Description |
-|---------------------|-------------------|-------------|
-| ≤ 5s | 2.0s | Preserve short sentences |
-| 5-10s | 1.5s | Normal segmentation |
-| 10-15s | 1.0s | Start tightening |
-| 15-30s | 0.8s | Faster splitting |
-| 30-45s | 0.4s | Prevent overly long segments |
-| > 45s | 0.1s | Force split |
-
-### Parameters
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `vad_model` | - | fsmn-vad instance loaded via FunASR AutoModel |
-| `chunk_size_ms` | 60 | VAD internal processing chunk size |
-| `speech_noise_thres` | 0.5 | Speech/noise discrimination threshold |
-| `speech_to_sil_thres_ms` | 150 | Speech-to-silence base time |
-| `silence_schedule` | See table above | Dynamic threshold config `[(upper_ms, silence_ms), ...]` |
-| `sample_rate` | 16000 | Sample rate |
-
-### Properties
-
-| Property | Description |
-|----------|-------------|
-| `vad.is_speaking` | Whether currently in speech state |
-| `vad.current_duration_ms` | Current segment accumulated duration |
-| `vad.current_threshold_ms` | Current silence threshold in use |
+**Q: Performance impact of SPK?**
+RTFx drops from 102 to 46. CER is unchanged. Disabled by default.
+
+**Q: Entry points for custom development?**
+Offline: `serve_vllm.process_audio()` / `FunASRNanoVLLM.generate()`
+Streaming: `serve_realtime_ws.RealtimeASRSession`
+
+**Q: Slow first startup?**
+vLLM initialization takes 60–90 s (KV Cache + CUDA Graph warmup). Subsequent inferences are instant.