Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
docs: add realtime websocket benchmark (#3119)
zhifu gaod8df61a17a3e
diff --git a/docs/benchmark/realtime_ws_benchmark.md b/docs/benchmark/realtime_ws_benchmark.mdnew file mode 100644index 000000000..11672bc3b--- /dev/null+++ b/docs/benchmark/realtime_ws_benchmark.md@@ -0,0 +1,92 @@+# Realtime WebSocket Benchmark++Use this benchmark when you need to measure the client-observable behavior of+`examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py` under+real streaming traffic. Offline `RTFx` and realtime service latency are+different metrics: this page focuses on first update latency, final latency+after `STOP`, response lag, and multi-client behavior.++The benchmark client accepts only 16 kHz mono PCM16 WAV input. Keeping the input+format strict removes resampling and file decoding from the measurement.++## Start the Service++For long continuous speech or multiple browser clients, start with a bounded+partial window and a moderate partial refresh interval:++```bash+CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \+ --port 10095 --language 中文 \+ --partial-window-sec 8 --decode-interval 0.8+```++Speaker diarization is disabled by default. Add `--enable-spk` only when the+`spk` field is required, and report that setting with the benchmark result.++## Run a Single Realtime Replay++```bash+python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \+ audio_16k_mono_pcm16.wav \+ --server ws://localhost:10095 \+ --clients 1 \+ --output-jsonl realtime_ws_1c.jsonl+```++With pacing enabled, the client sends audio at realtime speed using 100 ms+frames. This is the closest mode to a microphone or browser stream.++## Run Concurrent Replays++```bash+python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \+ audio_16k_mono_pcm16.wav \+ --server ws://localhost:10095 \+ --clients 8 \+ --loops 3 \+ --chunk-ms 100 \+ --language 中文 \+ --output-jsonl realtime_ws_8c.jsonl+```++Use a representative audio file. A long, pauseless monologue creates a very+different load shape from turn-taking meetings, because nearly every client is+speaking and triggering partial decodes at the same time.++For an unpaced stress test, add `--no-pace`. Treat that result as a throughput+stress signal, not as user-facing realtime latency.++## Metrics++| Metric | Meaning |+|--------|---------|+| `aggregate_audio_per_wall` | Total input audio seconds across all clients divided by benchmark wall time |+| `first_update_ms_p50/p95` | Time from first audio frame to first result message with `sentences`, `partial`, or `is_final` |+| `final_after_stop_ms_p50/p95` | Time from sending `STOP` to receiving the final result |+| `client_response_lag_ms_p95_max` | Largest per-client p95 of `(client receive time - audio start) - server duration_ms`; useful mainly in paced mode |+| `partial_messages` | Count of non-final result messages with a non-empty `partial` |+| `final_messages` | Count of final result messages |+| `errors` | Connection, timeout, protocol, or client-side validation errors |++The script can observe only client-side timing and fields returned by the+server. If you are debugging service internals, collect server logs separately+for queue wait, VAD time, ASR decode time, speaker diarization time, GPU memory,+and GPU utilization.++## Report Template++When publishing a realtime WebSocket benchmark or issue report, include:++| Category | What to record |+|----------|----------------|+| Data | Audio duration, sample rate, language/domain, silence ratio or speaking pattern, and whether the same file was looped |+| Load | `--clients`, `--loops`, `--chunk-ms`, paced or `--no-pace`, and total benchmark wall time |+| Service | `serve_realtime_ws.py` command, `--partial-window-sec`, `--decode-interval`, `--enable-spk`, language, and hotwords |+| Hardware | GPU/NPU model, GPU count, memory, driver, CUDA/CANN/runtime versions, CPU model, and available RAM |+| Software | `funasr`, PyTorch, torchaudio, vLLM, Python, OS, and container image if any |+| Output | Summary line, JSONL artifact, server logs, and any failed client IDs |++Do not reuse an offline `RTFx` number as a concurrency claim. For realtime+service sizing, benchmark with the actual traffic shape, especially sentence+length, pause distribution, simultaneous speakers, and whether speaker+diarization is enabled.diff --git a/docs/benchmark/rtf_reproducibility.md b/docs/benchmark/rtf_reproducibility.mdindex 1b74201e8..f6d7e7821 100644--- a/docs/benchmark/rtf_reproducibility.md+++ b/docs/benchmark/rtf_reproducibility.md@@ -27,6 +27,8 @@ for the whole set if the same scope is used:Do not compare an offline batch `RTFx` result with streaming first-token latencyor end-to-end product latency. They measure different things.+For realtime WebSocket service sizing, use the+[Realtime WebSocket Benchmark](./realtime_ws_benchmark.md) instead.## Current Public vLLM Benchmark Scopediff --git a/docs/index.rst b/docs/index.rstindex 335b2d059..3fc0b01ec 100644--- a/docs/index.rst+++ b/docs/index.rst@@ -85,6 +85,7 @@ Overview:caption: Benchmark and Leaderboard./benchmark/rtf_reproducibility.md+ ./benchmark/realtime_ws_benchmark.md./benchmark/benchmark_onnx.md./benchmark/benchmark_onnx_cpp.md./benchmark/benchmark_libtorch.mddiff --git a/docs/vllm_guide.md b/docs/vllm_guide.mdindex eaac27a5c..b898942c9 100644--- a/docs/vllm_guide.md+++ b/docs/vllm_guide.md@@ -641,6 +641,15 @@ python client_python.py --server ws://localhost:10095 --micpython client_python.py --server ws://localhost:10095 --file audio.wav```+**Realtime benchmark**:+```bash+python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \+ audio_16k_mono_pcm16.wav --server ws://localhost:10095 --clients 4 \+ --output-jsonl realtime_ws_4c.jsonl+```++For metric definitions and reporting fields, see [Realtime WebSocket Benchmark](./benchmark/realtime_ws_benchmark.md).+**Browser**: Open `client_mic.html`**Custom Python**:diff --git a/docs/vllm_guide_zh.md b/docs/vllm_guide_zh.mdindex 7b9381438..9b66e3ea2 100644--- a/docs/vllm_guide_zh.md+++ b/docs/vllm_guide_zh.md@@ -645,6 +645,15 @@ python client_python.py --server ws://localhost:10095 --micpython client_python.py --server ws://localhost:10095 --file audio.wav```+**实时压测**:+```bash+python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \+ audio_16k_mono_pcm16.wav --server ws://localhost:10095 --clients 4 \+ --output-jsonl realtime_ws_4c.jsonl+```++指标定义和报告字段见 [Realtime WebSocket Benchmark](./benchmark/realtime_ws_benchmark.md)。+**浏览器**:打开 `client_mic.html`**自定义 Python**:diff --git a/examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py b/examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.pynew file mode 100644index 000000000..fc06890f6--- /dev/null+++ b/examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py@@ -0,0 +1,311 @@+#!/usr/bin/env python3+"""Benchmark client for serve_realtime_ws.py.++The script replays a 16 kHz mono PCM16 WAV file to one or more realtime+WebSocket sessions and reports client-observable latency metrics. It does not+change the service and it does not require soundfile/librosa.+"""++import argparse+import asyncio+import json+import statistics+import sys+import time+import wave+from pathlib import Path++try:+ import websockets+except ImportError:+ print("Please install websockets: pip install websockets", file=sys.stderr)+ sys.exit(1)+++SAMPLE_RATE = 16000+SAMPLE_WIDTH_BYTES = 2+++def load_pcm16_wav(path):+ """Read a mono 16 kHz PCM16 WAV file and return raw bytes plus duration."""+ wav_path = Path(path)+ with wave.open(str(wav_path), "rb") as wf:+ channels = wf.getnchannels()+ sample_width = wf.getsampwidth()+ sample_rate = wf.getframerate()+ frames = wf.getnframes()+ if channels != 1 or sample_width != SAMPLE_WIDTH_BYTES or sample_rate != SAMPLE_RATE:+ raise ValueError(+ f"{wav_path} must be 16 kHz mono PCM16 WAV; got "+ f"{sample_rate} Hz, {channels} channel(s), {sample_width * 8}-bit samples"+ )+ audio = wf.readframes(frames)+ return audio, frames / SAMPLE_RATE+++def percentile(values, pct):+ if not values:+ return None+ ordered = sorted(values)+ if len(ordered) == 1:+ return ordered[0]+ index = (len(ordered) - 1) * pct / 100.0+ lower = int(index)+ upper = min(lower + 1, len(ordered) - 1)+ weight = index - lower+ return ordered[lower] * (1 - weight) + ordered[upper] * weight+++def round_or_none(value, digits=3):+ return None if value is None else round(value, digits)+++async def receive_message(ws, timeout):+ message = await asyncio.wait_for(ws.recv(), timeout=timeout)+ if isinstance(message, bytes):+ return {"_binary_bytes": len(message)}+ try:+ return json.loads(message)+ except json.JSONDecodeError:+ return {"_raw": message}+++async def send_command(ws, command, expected_event, timeout):+ await ws.send(command)+ response = await receive_message(ws, timeout)+ if expected_event and response.get("event") != expected_event:+ raise RuntimeError(f"{command} expected event={expected_event!r}, got {response}")+ return response+++async def recv_results(ws, metrics, audio_started_at, stop_sent_at_ref, timeout):+ while True:+ try:+ data = await receive_message(ws, timeout)+ except asyncio.TimeoutError:+ if metrics["final_messages"] == 0 and not metrics["stopped"]:+ metrics["errors"].append(f"timed out waiting for server message after {timeout}s")+ return+ except websockets.exceptions.ConnectionClosed as exc:+ if metrics["final_messages"] == 0 and not metrics["stopped"]:+ metrics["errors"].append(f"connection closed while receiving: {exc}")+ return++ now = time.perf_counter()+ metrics["messages"] += 1++ event = data.get("event")+ if event:+ metrics["events"][event] = metrics["events"].get(event, 0) + 1++ has_result = "sentences" in data or "partial" in data or data.get("is_final") is True+ if has_result:+ metrics["result_messages"] += 1+ if metrics["first_update_ms"] is None:+ metrics["first_update_ms"] = (now - audio_started_at) * 1000.0+ if data.get("partial"):+ metrics["partial_messages"] += 1+ duration_ms = data.get("duration_ms")+ if isinstance(duration_ms, (int, float)):+ metrics["response_lag_ms"].append((now - audio_started_at) * 1000.0 - duration_ms)++ if data.get("is_final") is True:+ metrics["final_messages"] += 1+ metrics["final_update_ms"] = (now - audio_started_at) * 1000.0+ if stop_sent_at_ref["value"] is not None:+ metrics["final_after_stop_ms"] = (now - stop_sent_at_ref["value"]) * 1000.0++ if event == "stopped":+ metrics["stopped"] = True+ return+++async def run_client(client_id, args, audio_bytes, audio_seconds):+ chunk_bytes = max(1, int(SAMPLE_RATE * args.chunk_ms / 1000.0)) * SAMPLE_WIDTH_BYTES+ total_audio_seconds = audio_seconds * args.loops+ metrics = {+ "client_id": client_id,+ "audio_seconds": total_audio_seconds,+ "chunk_ms": args.chunk_ms,+ "messages": 0,+ "result_messages": 0,+ "partial_messages": 0,+ "final_messages": 0,+ "events": {},+ "first_update_ms": None,+ "final_update_ms": None,+ "final_after_stop_ms": None,+ "response_lag_ms": [],+ "stopped": False,+ "errors": [],+ }++ wall_started_at = time.perf_counter()+ stop_sent_at_ref = {"value": None}++ try:+ async with websockets.connect(+ args.server,+ ping_interval=None,+ open_timeout=args.connect_timeout,+ max_size=args.max_message_size,+ ) as ws:+ await send_command(ws, "START", "started", args.recv_timeout)+ if args.hotwords:+ await send_command(ws, f"HOTWORDS:{args.hotwords}", "hotwords_set", args.recv_timeout)+ if args.language:+ await send_command(ws, f"LANGUAGE:{args.language}", "language_set", args.recv_timeout)++ audio_started_at = time.perf_counter()+ recv_task = asyncio.create_task(+ recv_results(ws, metrics, audio_started_at, stop_sent_at_ref, args.recv_timeout)+ )++ sent_audio_seconds = 0.0+ for _ in range(args.loops):+ for offset in range(0, len(audio_bytes), chunk_bytes):+ chunk = audio_bytes[offset : offset + chunk_bytes]+ await ws.send(chunk)+ sent_audio_seconds += len(chunk) / (SAMPLE_RATE * SAMPLE_WIDTH_BYTES)+ if args.pace:+ target_elapsed = sent_audio_seconds+ elapsed = time.perf_counter() - audio_started_at+ delay = target_elapsed - elapsed+ if delay > 0:+ await asyncio.sleep(delay)++ stop_sent_at_ref["value"] = time.perf_counter()+ metrics["send_seconds"] = stop_sent_at_ref["value"] - audio_started_at+ await ws.send("STOP")+ await recv_task+ except Exception as exc:+ metrics["errors"].append(str(exc))++ wall_seconds = time.perf_counter() - wall_started_at+ lags = metrics.pop("response_lag_ms")+ send_seconds = metrics.get("send_seconds")+ audio_per_wall = total_audio_seconds / wall_seconds if wall_seconds else None+ send_audio_per_wall = total_audio_seconds / send_seconds if send_seconds else None+ response_lag_ms_max = max(lags) if lags else None+ response_lag_ms_p95 = percentile(lags, 95)+ metrics.update(+ {+ "first_update_ms": round_or_none(metrics["first_update_ms"], 1),+ "final_update_ms": round_or_none(metrics["final_update_ms"], 1),+ "final_after_stop_ms": round_or_none(metrics["final_after_stop_ms"], 1),+ "wall_seconds": round(wall_seconds, 3),+ "send_seconds": round_or_none(send_seconds, 3),+ "audio_per_wall": round_or_none(audio_per_wall),+ "send_audio_per_wall": round_or_none(send_audio_per_wall),+ "response_lag_ms_max": round_or_none(response_lag_ms_max, 1),+ "response_lag_ms_p95": round_or_none(response_lag_ms_p95, 1),+ }+ )+ return metrics+++def summarize(results, elapsed_seconds):+ total_audio = sum(item["audio_seconds"] for item in results)+ first_updates = [item["first_update_ms"] for item in results if item["first_update_ms"] is not None]+ final_after_stop = [+ item["final_after_stop_ms"] for item in results if item["final_after_stop_ms"] is not None+ ]+ lag_p95 = [item["response_lag_ms_p95"] for item in results if item["response_lag_ms_p95"] is not None]+ errors = sum(len(item["errors"]) for item in results)+ return {+ "clients": len(results),+ "total_audio_seconds": round(total_audio, 3),+ "elapsed_seconds": round(elapsed_seconds, 3),+ "aggregate_audio_per_wall": round_or_none(total_audio / elapsed_seconds if elapsed_seconds else None),+ "first_update_ms_p50": round_or_none(statistics.median(first_updates), 1)+ if first_updates+ else None,+ "first_update_ms_p95": round_or_none(percentile(first_updates, 95), 1),+ "final_after_stop_ms_p50": round_or_none(statistics.median(final_after_stop), 1)+ if final_after_stop+ else None,+ "final_after_stop_ms_p95": round_or_none(percentile(final_after_stop, 95), 1),+ "client_response_lag_ms_p95_max": round_or_none(max(lag_p95), 1) if lag_p95 else None,+ "partial_messages": sum(item["partial_messages"] for item in results),+ "final_messages": sum(item["final_messages"] for item in results),+ "errors": errors,+ }+++def print_summary(summary, results):+ print("FunASR realtime WebSocket benchmark")+ print(f"clients: {summary['clients']}")+ print(f"total audio seconds: {summary['total_audio_seconds']}")+ print(f"elapsed seconds: {summary['elapsed_seconds']}")+ print(f"aggregate audio/wall: {summary['aggregate_audio_per_wall']}x")+ print(f"first update p50/p95 ms: {summary['first_update_ms_p50']} / {summary['first_update_ms_p95']}")+ print(+ "final after STOP p50/p95 ms: "+ f"{summary['final_after_stop_ms_p50']} / {summary['final_after_stop_ms_p95']}"+ )+ print(f"max client response-lag p95 ms: {summary['client_response_lag_ms_p95_max']}")+ print(f"partial/final messages: {summary['partial_messages']} / {summary['final_messages']}")+ print(f"errors: {summary['errors']}")+ if summary["errors"]:+ for item in results:+ for error in item["errors"]:+ print(f"client {item['client_id']} error: {error}")+++async def async_main(args):+ audio_bytes, audio_seconds = load_pcm16_wav(args.wav)+ started_at = time.perf_counter()+ results = await asyncio.gather(+ *(run_client(client_id, args, audio_bytes, audio_seconds) for client_id in range(args.clients))+ )+ elapsed = time.perf_counter() - started_at+ summary = summarize(results, elapsed)+ if args.output_jsonl:+ output_path = Path(args.output_jsonl)+ with output_path.open("w", encoding="utf-8") as f:+ for result in results:+ f.write(json.dumps({"type": "client", **result}, ensure_ascii=False) + "\n")+ f.write(json.dumps({"type": "summary", **summary}, ensure_ascii=False) + "\n")+ print_summary(summary, results)+ return 1 if summary["errors"] else 0+++def parse_args():+ parser = argparse.ArgumentParser(description="Benchmark serve_realtime_ws.py with one or more clients")+ parser.add_argument("wav", help="16 kHz mono PCM16 WAV file")+ parser.add_argument("--server", default="ws://localhost:10095", help="WebSocket server URL")+ parser.add_argument("--clients", type=int, default=1, help="Concurrent clients")+ parser.add_argument("--loops", type=int, default=1, help="Times each client replays the WAV before STOP")+ parser.add_argument("--chunk-ms", type=int, default=100, help="PCM frame duration per WebSocket send")+ parser.add_argument("--language", default="", help="Optional LANGUAGE command value")+ parser.add_argument("--hotwords", default="", help="Optional HOTWORDS command value, comma separated")+ parser.add_argument("--no-pace", dest="pace", action="store_false", help="Send as fast as possible")+ parser.add_argument("--output-jsonl", default="", help="Write per-client metrics and summary JSONL")+ parser.add_argument("--connect-timeout", type=float, default=10.0, help="Connection timeout seconds")+ parser.add_argument("--recv-timeout", type=float, default=30.0, help="Timeout waiting for server messages")+ parser.add_argument("--max-message-size", type=int, default=16 * 1024 * 1024, help="WebSocket max message size")+ parser.set_defaults(pace=True)+ args = parser.parse_args()+ if args.clients < 1:+ parser.error("--clients must be >= 1")+ if args.loops < 1:+ parser.error("--loops must be >= 1")+ if args.chunk_ms < 10:+ parser.error("--chunk-ms must be >= 10")+ return args+++def main():+ args = parse_args()+ try:+ raise SystemExit(asyncio.run(async_main(args)))+ except KeyboardInterrupt:+ print("Interrupted.", file=sys.stderr)+ raise SystemExit(130)+ except ValueError as exc:+ print(f"ERROR: {exc}", file=sys.stderr)+ raise SystemExit(2)+++if __name__ == "__main__":+ main()