Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
Document realtime WS partial rendering and concurrency (#3092)
zhifu gao4a412f8c2251
diff --git a/docs/vllm_guide.md b/docs/vllm_guide.mdindex 1b3d9b576..04be3834f 100644--- a/docs/vllm_guide.md+++ b/docs/vllm_guide.md@@ -617,7 +617,7 @@ Speaker diarization is disabled by default; add `--enable-spk` only when the `sp{"event": "stopped"}```-**Fields**: `sentences[]` = locked segments, `partial` = text being spoken (may change), `is_final` = true after STOP. When `--enable-spk` is enabled, `sentences[]` also includes `spk`.+**Fields**: `sentences[]` = locked segments, `partial` = text being spoken (may change), `partial_start_ms` = where the current provisional `partial` begins, `is_final` = true after STOP. When `--enable-spk` is enabled, `sentences[]` also includes `spk`.**Sequence diagram**:```@@ -678,6 +678,17 @@ While the user is speaking, the streaming service periodically (default `decode_> Note: `serve_vllm.py`'s `/ws` (§5) has **no partial** and only returns at sentence end; use `serve_realtime_ws.py` for live preview.+**Frontend rendering rule**+Treat `partial` as a replaceable preview, not as text to append. A good UI keeps locked text and preview text separate:++```js+const committed = data.sentences.map((s) => s.text).join("");+const preview = data.partial || "";+render(committed + preview);+```++If `partial_start_ms` moves forward because `--partial-window-sec` is active, the preview only describes the current bounded decode window. Replace the preview area on each message; append only VAD-locked `sentences` or the final `is_final=true` result.+**Principle: why each partial re-encodes the whole segment from the start**Fun-ASR-Nano's acoustic encoder (SenseVoice) is a **full-context, non-streaming** encoder — each frame's representation depends on the context of the entire segment. When the sentence continues and the audio grows, the context of the earlier frames changes, so **the previously computed encoding no longer holds**. It therefore cannot cache history and encode only the new frames the way a streaming / causal encoder would; it must run the whole "start → now" segment through the encoder again.@@ -702,6 +713,7 @@ Because each refresh re-encodes from the sentence start, the longer a sentence,- **The single-process concurrency ceiling comes from event-loop serialization, not GPU compute.** Under high concurrency GPU utilization stays low and the encoder runs at ~86× real time; mistaking this for insufficient GPU and adding cards or tensor parallelism yields little (TP only splits the LLM, not the standalone encoder).- **The right way to scale (currently) = multiple independent processes on one card + CUDA MPS + nginx round-robin**: each process has its own GIL and CUDA context, sidestepping the single-loop serialization; MPS lets the processes truly share the GPU concurrently and fill the idle compute; nginx round-robins across the WebSocket backends. Beyond a single card's headroom, scale out horizontally (one instance per card + a load balancer).+- **vLLM is not always more efficient for small real-time streams than several PyTorch processes.** vLLM helps most when requests can be batched or when LLM token decoding dominates. The current real-time WebSocket path submits many small, synchronous per-connection decode calls through one event loop, so it may reserve much more memory while still leaving GPU utilization modest. For a small number of continuous microphone streams, several lighter PyTorch processes can be easier to pack on one card. For vLLM, benchmark with the real traffic shape and start with lower `--gpu-memory-utilization` plus multiple service processes instead of assuming one vLLM process should carry every stream.- **Sustainable concurrency has no universal "supports N connections" number.** The ceiling is set not by the number of connections but by **how many are speaking at the same moment** — each speaking connection triggers a partial decode roughly once per second, all serialized on that single event loop. It mainly varies with: **① silence ratio** — in real turn-taking users spend most of the time listening, so far fewer are decoding simultaneously than are connected, whereas a continuous monologue keeps nearly every connection decoding; **② sentence length** — longer sentences make each partial encode more expensive (see 6.5's O(L²)), raising load at the same connection count. So the same "single L20 + multi-process + MPS" setup can sustain dozens of connections under turn-taking-like load but markedly fewer under long, pauseless speech. **Any "supports X connections" figure holds only for the traffic profile it was measured under** — benchmark with your own real traffic (sentence length, pauses, continuous or not) rather than treating someone else's number as your spec.---diff --git a/docs/vllm_guide_zh.md b/docs/vllm_guide_zh.mdindex d453697ca..17d23ab6a 100644--- a/docs/vllm_guide_zh.md+++ b/docs/vllm_guide_zh.md@@ -621,7 +621,7 @@ CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/{"event": "stopped"}```-**字段**:`sentences[]` = 已锁定,`partial` = 正在说(会变),`is_final` = STOP 后为 true。启用 `--enable-spk` 后,`sentences[]` 会包含 `spk`。+**字段**:`sentences[]` = 已锁定句子,`partial` = 当前正在说的临时文本(可能变化),`partial_start_ms` = 当前 `partial` 对应音频窗口的起点,`is_final` = STOP 后为 true。启用 `--enable-spk` 后,`sentences[]` 会包含 `spk`。**时序**:```@@ -682,6 +682,17 @@ asyncio.run(stream("audio.wav"))> 注:`serve_vllm.py`(§5)的 `/ws` **没有 partial**、只在句尾返回;要实时预览请用 `serve_realtime_ws.py`。+**前端渲染原则**+`partial` 只能当作“可替换预览”,不要把连续两次 `partial` 直接追加到最终文本里。推荐把已锁定文本和临时预览分开:++```js+const committed = data.sentences.map((s) => s.text).join("");+const preview = data.partial || "";+render(committed + preview);+```++如果启用了 `--partial-window-sec`,`partial_start_ms` 可能随着窗口向前滑动;这时 `partial` 只描述当前受限窗口内的临时识别结果。前端应每次替换 preview 区域,只把 VAD 已锁定的 `sentences` 或最终 `is_final=true` 结果追加到正式转写区。+**原理:为什么每次 partial 都从句首整段重编**Fun-ASR-Nano 的声学编码器(SenseVoice)是**全上下文、非流式**编码器——每一帧的表示都依赖整段音频的前后文。当这句话又往下说了一截、音频变长时,先前那些帧的上下文随之改变,**之前算出的编码不再成立**,因此无法像流式 / 因果编码器那样"缓存历史、只算新增帧",只能把"句首→当前"的整段重新过一遍编码器。@@ -706,6 +717,7 @@ Fun-ASR-Nano 的声学编码器(SenseVoice)是**全上下文、非流式**- **单进程并发墙来自事件循环串行,不是 GPU 算力**。- **目前扩展可行方案 = 单卡多个独立进程 + CUDA MPS + nginx 轮询**:每个进程有独立的 GIL 与 CUDA 上下文,绕开单循环串行;MPS 让多进程真正并发共享 GPU、填满空闲算力;nginx 在多个 WebSocket 后端间轮询。超过单卡余量后,再横向加卡(每卡一实例 + 负载均衡)。+- **小并发实时流不一定比多个 PyTorch 进程更适合 vLLM**。vLLM 的优势主要来自批处理和 LLM token decode 调度;而当前实时 WebSocket 路径会把多路小请求通过单事件循环同步送进解码,无法自然形成大 batch,所以可能显存占用更高但 GPU 利用率仍不高。对于少量连续麦克风流,多个轻量 PyTorch 进程有时更容易在一张卡上排布;使用 vLLM 时请按真实话务压测,先配合较低的 `--gpu-memory-utilization` 和多进程服务,而不是假设一个 vLLM 进程就应该承载所有连接。- **可持续并发没有通用的"支持 N 路"数字**:决定上限的不是在线连接数,而是**同一时刻有多少路正在"说话"**——每路只要在说,就每约 1 秒触发一次 partial 解码,全部串行在那个单事件循环上。它主要随两点变化:**① 停顿 / 静音占比**——真实一问一答中用户大半时间在听、不出声,同时解码的路数远少于在线连接数;连续独白则几乎每路都在持续解码,负载高得多。**② 句长**——句子越长,单次 partial 的编码越贵(见 6.5 的 O(L²)),同样路数下负载更高。因此同一套"单卡 L20 + 多进程 + MPS",在接近真实 turn-taking 的负载下可稳定支撑数十路,而在长句、连续不停顿的负载下会显著更低。**任何"支持 X 路"的数字都只在它被测出来的那种话务下成立**——请按自己的真实话务(句长、停顿、是否连续说话)压测确定,别把别处测出的某个并发数当成自己的规格。---diff --git a/docs/vllm_guide_zh_v2.md b/docs/vllm_guide_zh_v2.mdindex d453697ca..17d23ab6a 100644--- a/docs/vllm_guide_zh_v2.md+++ b/docs/vllm_guide_zh_v2.md@@ -621,7 +621,7 @@ CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/{"event": "stopped"}```-**字段**:`sentences[]` = 已锁定,`partial` = 正在说(会变),`is_final` = STOP 后为 true。启用 `--enable-spk` 后,`sentences[]` 会包含 `spk`。+**字段**:`sentences[]` = 已锁定句子,`partial` = 当前正在说的临时文本(可能变化),`partial_start_ms` = 当前 `partial` 对应音频窗口的起点,`is_final` = STOP 后为 true。启用 `--enable-spk` 后,`sentences[]` 会包含 `spk`。**时序**:```@@ -682,6 +682,17 @@ asyncio.run(stream("audio.wav"))> 注:`serve_vllm.py`(§5)的 `/ws` **没有 partial**、只在句尾返回;要实时预览请用 `serve_realtime_ws.py`。+**前端渲染原则**+`partial` 只能当作“可替换预览”,不要把连续两次 `partial` 直接追加到最终文本里。推荐把已锁定文本和临时预览分开:++```js+const committed = data.sentences.map((s) => s.text).join("");+const preview = data.partial || "";+render(committed + preview);+```++如果启用了 `--partial-window-sec`,`partial_start_ms` 可能随着窗口向前滑动;这时 `partial` 只描述当前受限窗口内的临时识别结果。前端应每次替换 preview 区域,只把 VAD 已锁定的 `sentences` 或最终 `is_final=true` 结果追加到正式转写区。+**原理:为什么每次 partial 都从句首整段重编**Fun-ASR-Nano 的声学编码器(SenseVoice)是**全上下文、非流式**编码器——每一帧的表示都依赖整段音频的前后文。当这句话又往下说了一截、音频变长时,先前那些帧的上下文随之改变,**之前算出的编码不再成立**,因此无法像流式 / 因果编码器那样"缓存历史、只算新增帧",只能把"句首→当前"的整段重新过一遍编码器。@@ -706,6 +717,7 @@ Fun-ASR-Nano 的声学编码器(SenseVoice)是**全上下文、非流式**- **单进程并发墙来自事件循环串行,不是 GPU 算力**。- **目前扩展可行方案 = 单卡多个独立进程 + CUDA MPS + nginx 轮询**:每个进程有独立的 GIL 与 CUDA 上下文,绕开单循环串行;MPS 让多进程真正并发共享 GPU、填满空闲算力;nginx 在多个 WebSocket 后端间轮询。超过单卡余量后,再横向加卡(每卡一实例 + 负载均衡)。+- **小并发实时流不一定比多个 PyTorch 进程更适合 vLLM**。vLLM 的优势主要来自批处理和 LLM token decode 调度;而当前实时 WebSocket 路径会把多路小请求通过单事件循环同步送进解码,无法自然形成大 batch,所以可能显存占用更高但 GPU 利用率仍不高。对于少量连续麦克风流,多个轻量 PyTorch 进程有时更容易在一张卡上排布;使用 vLLM 时请按真实话务压测,先配合较低的 `--gpu-memory-utilization` 和多进程服务,而不是假设一个 vLLM 进程就应该承载所有连接。- **可持续并发没有通用的"支持 N 路"数字**:决定上限的不是在线连接数,而是**同一时刻有多少路正在"说话"**——每路只要在说,就每约 1 秒触发一次 partial 解码,全部串行在那个单事件循环上。它主要随两点变化:**① 停顿 / 静音占比**——真实一问一答中用户大半时间在听、不出声,同时解码的路数远少于在线连接数;连续独白则几乎每路都在持续解码,负载高得多。**② 句长**——句子越长,单次 partial 的编码越贵(见 6.5 的 O(L²)),同样路数下负载更高。因此同一套"单卡 L20 + 多进程 + MPS",在接近真实 turn-taking 的负载下可稳定支撑数十路,而在长句、连续不停顿的负载下会显著更低。**任何"支持 X 路"的数字都只在它被测出来的那种话务下成立**——请按自己的真实话务(句长、停顿、是否连续说话)压测确定,别把别处测出的某个并发数当成自己的规格。---