Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

feat: add streaming vLLM inference for Fun-ASR-Nano

游雁3817ca31b377
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/docs/vllm_streaming.md b/examples/industrial_data_pretraining/fun_asr_nano/docs/vllm_streaming.md
new file mode 100644
index 000000000..69264affd
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/docs/vllm_streaming.md
@@ -0,0 +1,135 @@
+# Fun-ASR-Nano Streaming vLLM Inference
+
+Chunk-by-chunk streaming ASR using vLLM for high-throughput LLM decoding.
+
+## Design
+
+```
+Audio stream (720ms chunks)
+ │
+ ▼ (cumulative re-encoding)
+┌────────────────────┐
+│ Encoder + Adaptor │ (PyTorch, single GPU)
+└────────┬───────────┘
+ ▼
+┌────────────────────┐
+│ vLLM Batch Gen │ (all chunks batched in single call)
+└────────┬───────────┘
+ ▼
+Per-chunk results with fixed/unfixed regions:
+ chunk1: ""
+ chunk2: "开"
+ chunk3: "开放时间早"
+ chunk4: "开放时间早上九点至下午五点。" ← stabilized
+```
+
+### Key Design Decisions
+
+1. **Cumulative audio re-encoding**: Each chunk re-processes ALL audio from start. This ensures the encoder has full context.
+
+2. **Batch-all-chunks**: All chunk prompts are batched into a single vLLM `generate()` call. This avoids KV-cache corruption issues with sequential EmbedsPrompt calls and maximizes throughput.
+
+3. **8-character rollback**: The last 8 characters of each chunk's output are "unfixed" (may change in subsequent chunks). Text beyond the rollback boundary is "fixed".
+
+4. **Repetition penalty**: `repetition_penalty=1.3` prevents degenerate repetitions for short audio chunks.
+
+## Quick Start
+
+```python
+from funasr.models.fun_asr_nano.inference_vllm_streaming import FunASRNanoStreamingVLLM
+
+engine = FunASRNanoStreamingVLLM.from_pretrained(
+ model="FunAudioLLM/Fun-ASR-Nano-2512",
+ tensor_parallel_size=1,
+ chunk_ms=720,
+ rollback_chars=8,
+)
+
+# Stream results
+for result in engine.streaming_generate("audio.wav", language="中文"):
+ if result["is_final"]:
+ print(f"Final: {result['text']}")
+ else:
+ print(f"[{result['audio_duration_ms']:.0f}ms] {result['text']}")
+ print(f" Fixed: {result['fixed_text']}")
+```
+
+## API Reference
+
+### `FunASRNanoStreamingVLLM.from_pretrained()`
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `model` | `"FunAudioLLM/Fun-ASR-Nano-2512"` | Model name or local path |
+| `hub` | `"ms"` | `"ms"` or `"hf"` |
+| `tensor_parallel_size` | `1` | GPUs for vLLM |
+| `chunk_ms` | `720` | Chunk duration in ms |
+| `rollback_chars` | `8` | Unfixed region size |
+| `dtype` | `"bf16"` | Compute precision |
+
+### `engine.streaming_generate()`
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `audio_input` | - | File path, numpy array, or tensor (16kHz) |
+| `chunk_ms` | `720` | Override chunk size |
+| `rollback_chars` | `8` | Override rollback |
+| `hotwords` | `None` | List of hotwords |
+| `language` | `None` | Language hint (e.g. "中文") |
+| `itn` | `True` | Inverse text normalization |
+| `max_new_tokens` | `200` | Max tokens per chunk |
+| `temperature` | `0.0` | Sampling temperature |
+
+**Yields** per chunk:
+```python
+{
+ "text": "full recognized text so far",
+ "fixed_text": "confirmed text (won't change)",
+ "is_final": False, # True for last chunk
+ "chunk_idx": 4,
+ "audio_duration_ms": 2880.0,
+}
+```
+
+## Output Characteristics
+
+| Audio Duration | Output Quality |
+|---------------|---------------|
+| < 1.5s | Empty or garbage (insufficient context) |
+| 1.5 - 3.0s | Partial/noisy (model warming up) |
+| > 3.0s | Correct transcription |
+| Full audio | Best accuracy |
+
+The model needs ~3 seconds of cumulative audio before producing accurate results. This is inherent to the model architecture, not a vLLM limitation.
+
+## Example Output
+
+```
+[c01] 720ms | ""
+[c02] 1440ms | "-"
+[c03] 2160ms | "time/day."
+[c04] 2880ms | "time早上九"
+[c05] 3600ms | "期限:早上九点至"
+ fixed: "期限:早上"
+[c06] 4320ms | "期限:开放时间早上9点~下午10:35/4."
+ fixed: "期限:开放时间早上9点~下午10:"
+[c07] 5040ms | "期限:开放时间早上九点至下午五点。"
+ fixed: "期限:开放时间早上九点至下午"
+[FINAL] 5616ms | "期限,开放时间早上九点至下午五点。"
+```
+
+## Compared with Offline Mode
+
+For non-streaming (offline) inference with maximum accuracy, use `FunASRNanoVLLM`:
+
+```python
+from funasr.models.fun_asr_nano.inference_vllm import FunASRNanoVLLM
+
+engine = FunASRNanoVLLM.from_pretrained("FunAudioLLM/Fun-ASR-Nano-2512")
+results = engine.generate(["audio.wav"], language="中文")
+```
+
+| Mode | Latency | Accuracy | Use Case |
+|------|---------|----------|----------|
+| Offline (`FunASRNanoVLLM`) | Full audio | Best | Batch transcription |
+| Streaming (`FunASRNanoStreamingVLLM`) | Per-chunk | Good (after ~3s) | Real-time display |
diff --git a/funasr/models/fun_asr_nano/inference_vllm_streaming.py b/funasr/models/fun_asr_nano/inference_vllm_streaming.py
new file mode 100644
index 000000000..f20102af0
--- /dev/null
+++ b/funasr/models/fun_asr_nano/inference_vllm_streaming.py
@@ -0,0 +1,290 @@
+#!/usr/bin/env python3
+# -*- encoding: utf-8 -*-
+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
+# MIT License (https://opensource.org/licenses/MIT)
+
+"""
+Fun-ASR-Nano Streaming vLLM Inference Engine.
+
+Design:
+ - Audio split into 720ms chunks (cumulative re-encoding)
+ - ALL chunks batched into single vLLM generate call for correctness
+ - Fixed/Unfixed: last 8 chars are unfixed (may change on next chunk)
+ - Output stabilizes as more audio accumulates (~3s+)
+
+Note: vLLM processes all chunks in one batch for throughput.
+For real-time streaming, use the torch-based inference in demo2.py.
+"""
+
+import logging
+import os
+import re
+from typing import Generator, List, Union
+
+import numpy as np
+import torch
+import torch.nn as nn
+
+logger = logging.getLogger(__name__)
+dtype_map = {"bf16": torch.bfloat16, "fp16": torch.float16, "fp32": torch.float32}
+
+_CJK_RE = re.compile(r"[一-鿿]")
+
+
+def _clean_text(text: str) -> str:
+ """Remove tags, repetitive garbage, and invalid chars."""
+ text = re.sub(r"<[^>]*>|</[^>]*>", "", text)
+ text = re.sub(r"(>.{2,8}?)\1{3,}", "", text)
+ text = text.replace("�", "").lstrip(">")
+ return text.strip()
+
+
+def _is_meaningful(text: str) -> bool:
+ """Check if text has real ASR content."""
+ return len(_CJK_RE.findall(text)) >= 2
+
+
+class FunASRNanoStreamingVLLM:
+ """Streaming ASR with vLLM backend (batch-all-chunks approach).
+
+ Processes audio in 720ms chunks. All chunks are encoded and batched
+ into a single vLLM generate() call for correct and efficient inference.
+ Results are returned per-chunk with fixed/unfixed regions.
+
+ Args:
+ model_dir: Path to Fun-ASR-Nano model directory.
+ device: Device for audio encoder/adaptor.
+ dtype: Compute dtype ("bf16", "fp16", "fp32").
+ tensor_parallel_size: GPUs for vLLM tensor parallelism.
+ gpu_memory_utilization: GPU memory fraction for KV cache.
+ max_model_len: Maximum sequence length.
+ chunk_ms: Chunk duration in ms (default 720).
+ rollback_chars: Characters to rollback per chunk (default 8).
+ """
+
+ def __init__(self, model_dir, device="cuda:0", dtype="bf16",
+ tensor_parallel_size=1, gpu_memory_utilization=0.8,
+ max_model_len=2048, enforce_eager=False,
+ chunk_ms=720, rollback_chars=8, **kwargs):
+ from vllm import LLM
+ from funasr.models.fun_asr_nano.inference_vllm import prepare_vllm_model_dir
+
+ self.device = device
+ self.dtype = dtype
+ self.torch_dtype = dtype_map.get(dtype, torch.bfloat16)
+ self.model_dir = model_dir
+ self.chunk_ms = chunk_ms
+ self.rollback_chars = rollback_chars
+
+ vllm_model_dir = prepare_vllm_model_dir(model_dir)
+ self._load_audio_components(model_dir)
+
+ vllm_kwargs = kwargs.get("vllm_kwargs", {})
+ self.vllm_engine = LLM(
+ enable_prompt_embeds=True, model=vllm_model_dir,
+ tensor_parallel_size=tensor_parallel_size,
+ gpu_memory_utilization=gpu_memory_utilization,
+ max_model_len=max_model_len, enforce_eager=enforce_eager,
+ dtype={"bf16": "bfloat16", "fp16": "float16", "fp32": "auto"}.get(dtype, dtype),
+ trust_remote_code=True, **vllm_kwargs,
+ )
+ self.tokenizer = self.vllm_engine.get_tokenizer()
+ self._load_embedding_layer(model_dir)
+ self.sample_rate = self.frontend.fs
+ self.chunk_samples = int(self.sample_rate * self.chunk_ms / 1000)
+
+ def _load_audio_components(self, model_dir):
+ from omegaconf import OmegaConf
+ from funasr.register import tables
+ config = OmegaConf.load(os.path.join(model_dir, "config.yaml"))
+ self._config = OmegaConf.to_container(config, resolve=True)
+
+ frontend_class = tables.frontend_classes.get(config["frontend"])
+ frontend_conf = OmegaConf.to_container(config.get("frontend_conf", {}), resolve=True)
+ self.frontend = frontend_class(**frontend_conf)
+ self.frontend.eval()
+
+ encoder_conf = OmegaConf.to_container(config.get("audio_encoder_conf", {}), resolve=True)
+ if encoder_conf.get("hub") == "ms":
+ from funasr import AutoModel as FAM
+ enc_m = FAM(model=config["audio_encoder"], model_revision="master", disable_update=True)
+ self.audio_encoder_output_size = getattr(enc_m.model, "encoder_output_size", -1)
+ self.audio_encoder = enc_m.model.model.encoder if hasattr(enc_m.model, "model") else enc_m.model.encoder
+ else:
+ encoder_class = tables.encoder_classes.get(config["audio_encoder"])
+ self.audio_encoder = encoder_class(input_size=self.frontend.output_size(), **encoder_conf)
+ self.audio_encoder_output_size = self.audio_encoder.output_size()
+ self.audio_encoder.eval()
+ for p in self.audio_encoder.parameters(): p.requires_grad = False
+
+ adaptor_conf = OmegaConf.to_container(config.get("audio_adaptor_conf", {}), resolve=True)
+ adaptor_class = tables.adaptor_classes.get(config["audio_adaptor"])
+ if self.audio_encoder_output_size > 0:
+ adaptor_conf["encoder_dim"] = self.audio_encoder_output_size
+ self.audio_adaptor = adaptor_class(**adaptor_conf)
+ self.audio_adaptor.eval()
+ for p in self.audio_adaptor.parameters(): p.requires_grad = False
+
+ model_pt = os.path.join(model_dir, "model.pt")
+ if os.path.exists(model_pt):
+ ckpt = torch.load(model_pt, map_location="cpu")
+ sd = ckpt.get("state_dict", ckpt)
+ enc_s = {k[len("audio_encoder."):]: v for k, v in sd.items() if k.startswith("audio_encoder.")}
+ if enc_s: self.audio_encoder.load_state_dict(enc_s, strict=False)
+ adp_s = {k[len("audio_adaptor."):]: v for k, v in sd.items() if k.startswith("audio_adaptor.")}
+ if adp_s: self.audio_adaptor.load_state_dict(adp_s, strict=False)
+
+ self.audio_encoder = self.audio_encoder.to(self.device, dtype=torch.float32)
+ self.audio_adaptor = self.audio_adaptor.to(self.device, dtype=self.torch_dtype)
+
+ def _load_embedding_layer(self, model_dir):
+ ckpt = torch.load(os.path.join(model_dir, "model.pt"), map_location="cpu")
+ sd = ckpt.get("state_dict", ckpt)
+ for key in sd:
+ if "embed_tokens.weight" in key and key.startswith("llm."):
+ self.embed_tokens = nn.Embedding.from_pretrained(sd[key], freeze=True)
+ self.embed_tokens = self.embed_tokens.to(self.device, dtype=self.torch_dtype)
+ return
+ raise RuntimeError("Could not find LLM embedding weights")
+
+ @torch.no_grad()
+ def _encode_audio(self, audio_samples):
+ from funasr.utils.load_utils import extract_fbank
+ speech, speech_lengths = extract_fbank(
+ audio_samples, data_type="sound", frontend=self.frontend, is_final=True)
+ speech = speech.to(self.device, dtype=torch.float32)
+ speech_lengths = speech_lengths.to(self.device)
+ enc_out, enc_lens = self.audio_encoder(speech, speech_lengths)
+ adp_out, adp_lens = self.audio_adaptor(enc_out.to(dtype=self.torch_dtype), enc_lens)
+ return adp_out, adp_lens
+
+ def _build_prompt_text(self, hotwords=None, language=None, itn=True):
+ hotwords = hotwords or []
+ if hotwords:
+ prompt = "请结合上下文信息,更加准确地完成语音转写任务。如果没有相关信息,我们会留空。\n\n\n**上下文信息:**\n\n\n"
+ prompt += f"热词列表:[{', '.join(hotwords)}]\n"
+ else:
+ prompt = ""
+ prompt += f"语音转写成{language}" if language else "语音转写"
+ if not itn: prompt += ",不进行文本规整"
+ return prompt + ":"
+
+ @torch.no_grad()
+ def _build_embeds(self, audio_embeds, audio_embed_lens, hotwords=None, language=None, itn=True):
+ """Build input embeddings for a single chunk (no prev_text - fresh each time)."""
+ prompt = self._build_prompt_text(hotwords, language, itn)
+ prefix_text = f"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n{prompt}<|startofspeech|>"
+ suffix_text = "<|endofspeech|><|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
+
+ prefix_ids = self.tokenizer.encode(prefix_text, add_special_tokens=False)
+ suffix_ids = self.tokenizer.encode(suffix_text, add_special_tokens=False)
+ prefix_emb = self.embed_tokens(torch.tensor(prefix_ids, dtype=torch.long, device=self.device))
+ suffix_emb = self.embed_tokens(torch.tensor(suffix_ids, dtype=torch.long, device=self.device))
+ audio_emb = audio_embeds[0, :audio_embed_lens[0].item(), :]
+ return torch.cat([prefix_emb, audio_emb, suffix_emb], dim=0)
+
+ def streaming_generate(self, audio_input, chunk_ms=None, rollback_chars=None,
+ hotwords=None, language=None, itn=True,
+ max_new_tokens=200, temperature=0.0, **kwargs):
+ """Streaming ASR: process all chunks and yield results per chunk.
+
+ All chunks are batched into a single vLLM generate() call for
+ correct results. Yields incrementally improving transcriptions.
+
+ Args:
+ audio_input: File path, numpy array, or tensor (16kHz).
+ chunk_ms: Chunk size in ms (default 720).
+ rollback_chars: Chars to rollback (default 8).
+ hotwords: Hotword list.
+ language: Language hint (e.g. "中文").
+ itn: Inverse text normalization.
+ max_new_tokens: Max tokens per chunk generation.
+ temperature: Sampling temperature (0 = greedy).
+
+ Yields:
+ {"text": full_text, "fixed_text": confirmed_text,
+ "is_final": bool, "chunk_idx": int, "audio_duration_ms": float}
+ """
+ from vllm import SamplingParams
+ from vllm.inputs.data import EmbedsPrompt
+ from funasr.utils.load_utils import load_audio_text_image_video
+
+ chunk_ms = chunk_ms or self.chunk_ms
+ rollback_chars = rollback_chars or self.rollback_chars
+
+ if isinstance(audio_input, str):
+ audio_data = load_audio_text_image_video(audio_input, fs=self.sample_rate)
+ elif isinstance(audio_input, np.ndarray):
+ audio_data = torch.from_numpy(audio_input).float()
+ elif isinstance(audio_input, torch.Tensor):
+ audio_data = audio_input.float()
+ else:
+ raise ValueError(f"Unsupported audio type: {type(audio_input)}")
+ if audio_data.dim() > 1:
+ audio_data = audio_data.squeeze()
+
+ total_samples = audio_data.shape[0]
+ chunk_samples = int(self.sample_rate * chunk_ms / 1000)
+ num_chunks = (total_samples + chunk_samples - 1) // chunk_samples
+
+ # Pre-compute all chunk embeddings
+ prompts = []
+ chunk_infos = []
+ for i in range(num_chunks):
+ end_sample = min((i + 1) * chunk_samples, total_samples)
+ cumulative_audio = audio_data[:end_sample]
+ adaptor_out, adaptor_out_lens = self._encode_audio(cumulative_audio)
+ input_embeds = self._build_embeds(adaptor_out, adaptor_out_lens,
+ hotwords=hotwords, language=language, itn=itn)
+ prompts.append(EmbedsPrompt(prompt_embeds=input_embeds.float()))
+ chunk_infos.append({
+ "chunk_idx": i + 1,
+ "is_final": end_sample >= total_samples,
+ "audio_duration_ms": end_sample * 1000 / self.sample_rate,
+ })
+
+ # Single batch vLLM generate call
+ params = SamplingParams(max_tokens=max_new_tokens, temperature=temperature,
+ repetition_penalty=1.3, skip_special_tokens=True)
+ outputs = self.vllm_engine.generate(prompts, params, use_tqdm=False)
+
+ # Yield results per chunk
+ for i, (output, info) in enumerate(zip(outputs, chunk_infos)):
+ text = output.outputs[0].text
+ if not text and output.outputs[0].token_ids:
+ text = self.tokenizer.decode(list(output.outputs[0].token_ids), skip_special_tokens=True)
+ text = _clean_text(text)
+
+ if info["is_final"]:
+ fixed_text = text
+ elif _is_meaningful(text) and len(text) > rollback_chars:
+ fixed_text = text[:-rollback_chars]
+ else:
+ fixed_text = ""
+
+ yield {"text": text, "fixed_text": fixed_text, **info}
+
+ def generate(self, audio_input, **kwargs):
+ """Run streaming and return all chunk results."""
+ return list(self.streaming_generate(audio_input, **kwargs))
+
+ @classmethod
+ def from_pretrained(cls, model="FunAudioLLM/Fun-ASR-Nano-2512", hub="ms",
+ device="cuda:0", dtype="bf16", tensor_parallel_size=1,
+ gpu_memory_utilization=0.8, max_model_len=2048,
+ chunk_ms=720, rollback_chars=8, **kwargs):
+ """Load from hub or local path."""
+ if os.path.isdir(model):
+ model_dir = model
+ else:
+ if hub in ("ms", "modelscope"):
+ from modelscope.hub.snapshot_download import snapshot_download
+ model_dir = snapshot_download(model, revision=kwargs.pop("revision", "master"))
+ else:
+ from huggingface_hub import snapshot_download
+ model_dir = snapshot_download(model)
+ return cls(model_dir=model_dir, device=device, dtype=dtype,
+ tensor_parallel_size=tensor_parallel_size,
+ gpu_memory_utilization=gpu_memory_utilization,
+ max_model_len=max_model_len, chunk_ms=chunk_ms,
+ rollback_chars=rollback_chars, **kwargs)