Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
docs: sync vllm_streaming.md with code - fix params, add AutoModelVLLM comparison
xiaoyunchong.xyc6e3663c7ac42
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/docs/vllm_streaming.md b/examples/industrial_data_pretraining/fun_asr_nano/docs/vllm_streaming.mdindex cbf8e4e7a..f742be1fb 100644--- a/examples/industrial_data_pretraining/fun_asr_nano/docs/vllm_streaming.md+++ b/examples/industrial_data_pretraining/fun_asr_nano/docs/vllm_streaming.md@@ -62,10 +62,13 @@ for result in engine.streaming_generate("audio.wav", language="中文"):|-----------|---------|-------------|| `model` | `"FunAudioLLM/Fun-ASR-Nano-2512"` | Model name or local path || `hub` | `"ms"` | `"ms"` or `"hf"` |+| `device` | `"cuda:0"` | Device for audio encoder/adaptor |+| `dtype` | `"bf16"` | Compute precision || `tensor_parallel_size` | `1` | GPUs for vLLM |+| `gpu_memory_utilization` | `0.8` | GPU memory fraction for vLLM KV cache |+| `max_model_len` | `2048` | Maximum sequence length || `chunk_ms` | `720` | Chunk duration in ms || `rollback_chars` | `8` | Unfixed region size |-| `dtype` | `"bf16"` | Compute precision |### `engine.streaming_generate()`@@ -80,6 +83,8 @@ for result in engine.streaming_generate("audio.wav", language="中文"):| `max_new_tokens` | `200` | Max tokens per chunk || `temperature` | `0.0` | Sampling temperature |+> Note: `repetition_penalty=1.3` is applied internally to prevent degenerate repetitions on short chunks.+**Yields** per chunk:```python{@@ -118,21 +123,30 @@ The model needs ~3 seconds of cumulative audio before producing accurate results[FINAL] 5616ms | "期限,开放时间早上九点至下午五点。"```-## Compared with Offline Mode+## Compared with Other Modes++| Mode | Class | Latency | Accuracy | Use Case |+|------|-------|---------|----------|----------|+| Offline | `AutoModelVLLM` / `FunASRNanoVLLM` | Full audio | Best | Batch transcription |+| Streaming SDK | `FunASRNanoStreamingVLLM` | Per-chunk | Good (after ~3s) | Real-time display (SDK) |+| WebSocket Server | `serve_realtime_ws.py` | Per-VAD-segment | Best | Production service with VAD+SPK |-For non-streaming (offline) inference with maximum accuracy, use `FunASRNanoVLLM`:+For offline batch inference:```python-from funasr.models.fun_asr_nano.inference_vllm import FunASRNanoVLLM+from funasr.auto.auto_model_vllm import AutoModelVLLM++model = AutoModelVLLM(model="FunAudioLLM/Fun-ASR-Nano-2512")+results = model.generate(["audio.wav"], language="中文")+```++For real-time WebSocket service (VAD + ASR + Speaker Diarization):-engine = FunASRNanoVLLM.from_pretrained("FunAudioLLM/Fun-ASR-Nano-2512")-results = engine.generate(["audio.wav"], language="中文")+```bash+python serve_realtime_ws.py --port 10095 --language 中文```-| Mode | Latency | Accuracy | Use Case |-|------|---------|----------|----------|-| Offline (`FunASRNanoVLLM`) | Full audio | Best | Batch transcription |-| Streaming (`FunASRNanoStreamingVLLM`) | Per-chunk | Good (after ~3s) | Real-time display |+> Note: `serve_realtime_ws.py` uses `AutoModelVLLM.generate()` per-segment (not `streaming_generate`), because VAD already provides natural sentence boundaries. This gives better accuracy than chunk-level streaming.## Related