Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

docs: clarify ONNX hotword deployment choice (#3120)

zhifu gao7237ef890332
diff --git a/docs/deployment_matrix.md b/docs/deployment_matrix.md
index 5ebe0c0d9..f323161a5 100644
--- a/docs/deployment_matrix.md
+++ b/docs/deployment_matrix.md
@@ -13,6 +13,7 @@ Use this page to choose the shortest deployment path for a product, demo, benchm
| Docker Compose API | Reproducible local smoke test or small internal service | [OpenAI API Docker docs](../examples/openai_api/#docker-deployment) | CPU by default; adapt the image before using CUDA in containers. |
| Kubernetes API | Internal speech API for cluster services | [Kubernetes template](../examples/openai_api/kubernetes/) | Starts as private `ClusterIP`; add auth, TLS, network policy, and GPU scheduling before broader exposure. |
| Runtime WebSocket service | Live captions, meetings, call-center streams | [Runtime service docs](../runtime/readme.md) | Use when partial results, endpointing, or long-lived audio streams matter. |
+| ONNX/C++ runtime | High-concurrency CPU services or embedded realtime ASR | [ONNX runtime docs](../runtime/onnxruntime/readme.md) | Keep this path when latency/concurrency is already proven; add text post-processing for fixed business terms before moving to GPU LLMs. |
| vLLM acceleration | Higher-throughput LLM-based ASR with Fun-ASR-Nano | [vLLM guide](./vllm_guide.md) | Use for LLM decoder throughput; does not apply to non-autoregressive Paraformer. |
| MCP server | Claude/Cursor/desktop agent speech tools | [MCP example](../examples/mcp_server/) | Good when the ASR result should be exposed as a local tool. |
| Subtitle generator | SRT/VTT from long audio or video | [Subtitle example](../examples/subtitle/) | Use verbose segments and speaker labels when readability matters. |
@@ -59,6 +60,27 @@ Use the [Kubernetes template](../examples/openai_api/kubernetes/) for a private
Use the runtime WebSocket service. Validate chunk size, VAD, endpointing, punctuation, speaker diarization, reconnect behavior, and client backpressure with real audio before production rollout.
+### I already have high-concurrency ONNX and need hotwords
+
+Do not move an entire working CPU ONNX/C++ realtime stack to a GPU LLM only
+because the current path lacks hotwords. First decide what "hotword" means in
+your product:
+
+- For fixed company names, product names, or common misrecognitions, keep the
+ ONNX path and add deterministic text post-processing on final results. This
+ preserves proven CPU concurrency and is easier to audit.
+- For true decoder-time biasing, pronunciation ambiguity, or multilingual
+ accuracy gaps, test a GPU path such as Fun-ASR-Nano or Qwen3-ASR, but size it
+ against your target first-token latency, final latency, audio chunk length,
+ VAD policy, and simultaneous speaking sessions.
+- If both are needed, use the GPU model only where it wins on quality or
+ language coverage, and keep ONNX for the high-volume traffic that already
+ meets latency and accuracy requirements.
+
+When opening a deployment issue, include your current ONNX concurrency, CPU
+cores, target terms, acceptable end-to-end latency, GPU model, model command,
+and whether hotwords are post-recognition corrections or decoder-time biasing.
+
### I need more LLM-based ASR throughput
Use the vLLM path for Fun-ASR-Nano. Benchmark with your own audio distribution and watch GPU memory, tensor parallel size, first-token latency, and warmup time.
@@ -74,6 +96,9 @@ Fun-ASR-Nano's LLM-based path is currently documented and validated for CUDA/vLL
- Run a short public smoke sample and at least one realistic private sample.
- Log audio duration, model, device, latency, response format, and error type for every request.
- Add upload-size limits, authentication, TLS, and rate limits before exposing an API outside a trusted network; use the [security and gateway guide](../examples/openai_api/SECURITY.md) to plan the boundary.
+- For hotword or correction requirements, record whether the change is
+ deterministic post-processing or decoder-time biasing, then benchmark quality
+ and latency before replacing an existing runtime.
- For streaming, test silence, noise, overlapping speakers, long sessions, reconnects, and slow clients.
- For benchmark claims, include input duration, hardware, batch size, model, runtime path, and whether model download/warmup time is excluded.
diff --git a/docs/deployment_matrix_zh.md b/docs/deployment_matrix_zh.md
index d2ef396f1..e0e9b0047 100644
--- a/docs/deployment_matrix_zh.md
+++ b/docs/deployment_matrix_zh.md
@@ -13,6 +13,7 @@
| Docker Compose API | 可复现本地 smoke test 或小型内部服务 | [OpenAI API Docker 文档](../examples/openai_api/README_zh.md) | 默认 CPU;容器里使用 CUDA 前需要先适配 CUDA-capable 镜像。 |
| Kubernetes API | 集群内私有语音 API | [Kubernetes 模板](../examples/openai_api/kubernetes/README_zh.md) | 默认私有 `ClusterIP`;对外开放前补齐鉴权、TLS、网络策略和 GPU 调度。 |
| Runtime WebSocket 服务 | 实时字幕、会议、客服流式音频 | [Runtime 服务文档](../runtime/readme_cn.md) | 需要中间结果、断句或长连接音频流时选择。 |
+| ONNX/C++ Runtime | 高并发 CPU 服务或嵌入式实时 ASR | [ONNX Runtime 文档](../runtime/onnxruntime/readme.md) | 如果延迟和并发已经验证,不要轻易替换;固定业务词优先做文本后处理。 |
| vLLM 加速 | Fun-ASR-Nano 等 LLM-based ASR 高吞吐 | [vLLM 指南](./vllm_guide.md) | 适合 LLM 解码吞吐;不适用于非自回归 Paraformer。 |
| MCP 服务 | Claude/Cursor/桌面 Agent 语音工具 | [MCP 示例](../examples/mcp_server/) | 适合把 ASR 结果暴露成一个本地工具。 |
| 字幕生成 | 从长音频或视频生成 SRT/VTT | [字幕示例](../examples/subtitle/) | 需要可读性时使用 verbose segments 和说话人标签。 |
@@ -53,6 +54,16 @@ docker compose up --build
使用 Runtime WebSocket 服务。上线前请用真实音频验证 chunk size、VAD、断句、标点、说话人分离、重连行为和客户端背压。
+### 我已有高并发 ONNX,但需要热词
+
+不要只因为当前 CPU ONNX/C++ 实时链路缺少热词,就把整条已经跑稳的链路迁到 GPU 大模型。先确认产品里的“热词”到底是哪一种:
+
+- 如果是公司名、产品名、专有名词或固定误识别纠错,优先保留 ONNX 链路,在最终文本上做确定性后处理。这样能保留已经验证过的 CPU 并发能力,也更容易审计。
+- 如果必须在解码阶段影响识别,或者确实存在多语言/语义理解上的质量缺口,再测试 Fun-ASR-Nano、Qwen3-ASR 等 GPU 路径;但要按首包延迟、尾包延迟、音频切片长度、VAD 策略和同时说话路数重新压测。
+- 如果两类需求都有,只在 GPU 模型质量或语言覆盖明显更好的流量上使用 GPU,其他高吞吐稳定流量继续走 ONNX。
+
+开部署 issue 时,请附上当前 ONNX 并发、CPU 核数、目标热词、可接受端到端延迟、GPU 型号、模型启动命令,以及热词是“识别后纠错”还是“解码阶段偏置”。
+
### 我需要更高的 LLM-based ASR 吞吐
Fun-ASR-Nano 走 vLLM 路径。请用自己的音频分布做 benchmark,并关注 GPU 显存、tensor parallel size、首 token 延迟和 warmup 时间。
@@ -68,6 +79,7 @@ Fun-ASR-Nano 的 LLM-based 路径目前主要按 CUDA/vLLM、标准 PyTorch CPU/
- 跑一个公开短音频 smoke sample,再跑至少一个真实私有样本。
- 每次请求记录音频时长、模型、设备、延迟、响应格式和错误类型。
- API 暴露到可信网络外之前,增加上传大小限制、鉴权、TLS 和限流;可用 [安全与网关指南](../examples/openai_api/SECURITY_zh.md) 规划边界。
+- 热词或纠错需求要先记录它是确定性后处理,还是解码阶段偏置;替换已有 runtime 前同时 benchmark 质量和延迟。
- 流式场景需要测试静音、噪声、多人重叠、长连接、重连和慢客户端。
- 发布 benchmark 结论时,说明输入时长、硬件、batch size、模型、运行路径,以及是否排除模型下载和 warmup 时间。