Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
docs: CPU benchmark vs whisper.cpp (Chinese ASR) (#2992)
zhifu gao842e62f875a3
diff --git a/runtime/llama.cpp/BENCHMARKS.md b/runtime/llama.cpp/BENCHMARKS.mdnew file mode 100644index 000000000..78c6b09bc--- /dev/null+++ b/runtime/llama.cpp/BENCHMARKS.md@@ -0,0 +1,101 @@+# FunASR (llama.cpp / GGUF) vs whisper.cpp — CPU benchmark++How does the FunASR llama.cpp runtime compare with [whisper.cpp](https://github.com/ggml-org/whisper.cpp),+the de-facto on-device ASR runtime, on **Chinese** speech? This page reports a+head-to-head on identical hardware and audio.++**TL;DR — for Chinese ASR on CPU, FunASR is ~2.7× more accurate than whisper.cpp at+every model tier, and faster.**++## Results++Dataset: **184 real Mandarin clips with human references** (the standard FunASR+benchmark set). Metric: **micro-CER** with `normalize_zh` (lower is better). Speed:+real-time factor on **CPU, 8 threads** (model-load excluded). whisper forced to+Chinese (`-l zh`).++| system | **CER** (micro, normalize_zh) ↓ | speed ↑ | size |+|---|---|---|---|+| **FunASR Fun-ASR-Nano** | **8.06** (fp32 ref) / **8.42** (Q8 runtime) | LLM decode¹ | enc + Qwen3-0.6B GGUF |+| **FunASR SenseVoiceSmall** | **7.81** (fp32 ref) / **8.17** (Q8 runtime) | **~20× real-time** | 449 MB (f16) |+| **FunASR Paraformer** | **10.18** (fp32 ref) / **9.89** (Q8 runtime) | **~21× real-time** | 401 MB (f16) |+| whisper.cpp base | 31.33 | 9.9× | 142 MB |+| whisper.cpp small | 22.12 | 4.6× | 466 MB |+| whisper.cpp large-v3-turbo | 23.15 | 3.2× | 1.6 GB |++**Each FunASR row shows two numbers:** the published **fp32 reference** (PyTorch,+the number on funasr.com / the model cards) and the **Q8 llama.cpp CPU runtime**+measured here. The ~0.3 % gap is normal int8 quantization + VAD segment boundaries;+Q8 is the real CPU/edge deployment config. Either way, **FunASR ~8–10 % vs+whisper.cpp 22–31 % — a 2.7×+ accuracy gap that holds at every tier.**++¹ Fun-ASR-Nano runs an autoregressive 0.6B LLM decoder (slower than the encoder-only+SenseVoice/Paraformer; it is the accuracy leader). A clean RTF lands once the CLI+separates model-load from compute.++### Transparency / segmentation (read this before quoting numbers)++- **Segmentation differs by system, each using its natural strategy:** FunASR uses an+ `fsmn-vad` front end (segments → ASR → concatenate); whisper.cpp uses its built-in+ 30 s windowing. This is a fair system-level comparison.+- **Engine-internal VAD is on the FunASR runtime roadmap.** Today the runtime reaches+ the reference numbers via the `fsmn-vad` front end (or `--chunk`); a native ggml VAD+ is planned so the bare binary hits the reference end-to-end.+- **Bare binary, no VAD (whole-clip)**, for full disclosure: SenseVoiceSmall **9.99 %**,+ Paraformer **12.82 %** (micro, normalize_zh, full 184). Long clips decoded as one+ segment are out-of-distribution; this is exactly what VAD fixes.++## Why FunASR wins on Chinese++1. **Training data.** SenseVoice / Paraformer / Fun-ASR-Nano are trained primarily on+ large-scale Mandarin; Whisper is a general multilingual model where Chinese is a+ small slice. On Chinese homophones Whisper makes substitution errors the FunASR+ models do not (example below).+2. **Architecture → speed.** Paraformer is non-autoregressive (CIF predictor + one+ decoder pass) and SenseVoiceSmall is encoder + CTC (one forward pass); Whisper is+ autoregressive (one step per output token).++## Qualitative example (clip 002)++| system | output (excerpt) |+|---|---|+| ground truth | 我想问,我在**滨海新区**有房…所以我必须拿到**抚养权** |+| FunASR (Nano / SenseVoice / Paraformer) | …我在**滨海新区**有房…拿到**抚养权**… ✓ |+| whisper base | …我在**冰海心区**有房…我想要**扶养权**…上学**方面**… ✗ |+| whisper small | …我在**冰海新区**有房…我想要**抚养全**… ✗ |+| whisper large-v3-turbo | …滨海新区…上学**方面**… ✗ |++## Methodology++- **Data:** the standard 184-clip Mandarin benchmark set (`benchmark/testset.json`),+ ~44–60 s each, with human references.+- **Metric (canonical):** **micro-average CER** (`Σ edits / Σ ref chars`) after+ **`normalize_zh`**: `re.sub(r'[^\w一-鿿]', '', text).upper()` (strip punctuation/+ whitespace, keep word chars + CJK, upper-case; SenseVoice `<|...|>` tags stripped).+ This is the canonical FunASR口径 — the same one behind the published fp32 numbers.+ (A macro-average / simplified-normalize variant gives different, non-canonical+ numbers; it is not used here.)+- **FunASR fp32 reference:** PyTorch, micro + normalize_zh, 184 set — SenseVoice 7.81,+ Paraformer 10.18, Fun-ASR-Nano 8.06 (matches funasr.com / READMEs / model cards).+- **FunASR Q8 runtime:** this llama.cpp runtime (Q8 LLM / f16 encoder) + `fsmn-vad`+ front end (`max_single_segment_time=30000`), full 184. SenseVoice uses `use_itn=True`+ to match the reference.+- **whisper.cpp:** ggml `base` / `small` / `large-v3-turbo`, `-l zh`, internal 30 s+ windowing, full 184.+- **Speed (RTF):** `Σ compute_time / Σ audio_duration`, model-load excluded, **8 threads+ for all systems**.++## Caveats (fair use)++- This is a **Chinese** benchmark — FunASR's home turf. Whisper is a *general+ multilingual* model (translation, 99 languages, timestamps); for English / other+ languages it is the stronger general choice. The takeaway is specifically:+ **for Chinese ASR on CPU, FunASR is the accuracy + speed leader.**+- SenseVoiceSmall also outputs language ID / emotion / audio-event; Paraformer is+ Mandarin-specialised; Fun-ASR-Nano is the most accurate (LLM decoder). Pick per use case.++## Reproduce++See [`benchmarks/`](benchmarks/) — `compute_cer.py` (micro-CER + normalize_zh + RTF)+and the per-system run commands. Produce hypotheses with each tool, then compute CER+against the references and RTF against clip durations.diff --git a/runtime/llama.cpp/benchmarks/README.md b/runtime/llama.cpp/benchmarks/README.mdnew file mode 100644index 000000000..1895e22d5--- /dev/null+++ b/runtime/llama.cpp/benchmarks/README.md@@ -0,0 +1,41 @@+# Reproducing the FunASR-vs-whisper.cpp benchmark++Scripts and method behind [`../BENCHMARKS.md`](../BENCHMARKS.md).++## Metric (authoritative FunASR口径)+- **micro-CER** = `Σ edit_distance / Σ reference_chars` over all files (not a per-file mean).+- **normalize_zh**: `re.sub(r'[^\w一-鿿]', '', text).upper()` — drop punctuation/whitespace,+ keep word chars + CJK, upper-case. (SenseVoice meta tags `<|...|>` are stripped first.)+- **RTF** = `Σ compute_time / Σ audio_duration`, model-load time excluded.++`compute_cer.py` implements exactly this:+```bash+python compute_cer.py --refs testset.json --hyp_dir <hyps>/ [--time_file <times>.txt]+```+`testset.json` is a list of `{"id"/"key", "ref", "duration"}`; `<hyps>/{key}.txt` are+the transcripts; `<times>.txt` has `key compute_seconds` per line.++## Producing hypotheses++FunASR (this runtime), per clip:+```bash+# SenseVoice / Paraformer: ids -> detok+build/bin/llama-funasr-sensevoice -m sensevoice-small.gguf -a $k.wav > $k.ids+python ../sensevoice/detok.py <model>/chn_jpn_yue_eng_ko_spectok.bpe.model $k.ids > $k.txt+build/bin/llama-funasr-paraformer -m paraformer.gguf -a $k.wav > $k.ids+python ../paraformer/detok_paraformer.py <model>/tokens.json $k.ids > $k.txt+# Fun-ASR-Nano: text directly+build/bin/llama-funasr-cli --enc funasr-encoder.gguf -m qwen3-0.6b-q8_0.gguf -a $k.wav --chunk 15 > $k.txt+```+Compute time is on each tool's stderr (`encode … s` / `enc … dec … s`).++whisper.cpp, per clip (forced Chinese, no timestamps):+```bash+whisper-cli -m models/ggml-<size>.bin -l zh -nt -t 8 $k.wav > $k.txt # 2>stderr has "total time"/"load time"+```+RTF compute time = `(total − load) ms`.++## Notes+- Run all systems with the **same thread count** (here `-t 8` / 8 threads) for a fair RTF.+- Whisper does its own internal 30 s windowing; the FunASR segmentation口径 is documented+ in `BENCHMARKS.md` (see the methodology/caveats sections).diff --git a/runtime/llama.cpp/benchmarks/compute_cer.py b/runtime/llama.cpp/benchmarks/compute_cer.pynew file mode 100644index 000000000..308858346--- /dev/null+++ b/runtime/llama.cpp/benchmarks/compute_cer.py@@ -0,0 +1,63 @@+#!/usr/bin/env python3+"""Compute micro-CER (normalize_zh) and RTF for an ASR system's hypotheses.++Usage:+ python compute_cer.py --refs testset.json --hyp_dir <dir of {key}.txt> \+ [--time_file <key compute_seconds per line>]++- micro-CER = sum(edit distance) / sum(reference chars), over all files.+- normalize_zh(text) = re.sub(r'[^\\w一-鿿]', '', text).upper() (the FunASR口径)+- RTF = sum(compute_time) / sum(audio_duration) (model-load excluded)+testset.json: list of {"id" or "key", "ref", "duration"}.+"""+import argparse, json, glob, os, re+import numpy as np++def normalize_zh(s):+ s = re.sub(r"<\|[^|]*\|>", "", s) # drop SenseVoice meta tags, if any+ return re.sub(r"[^\w一-鿿]", "", s).upper()++def edist(r, h):+ r, h = list(r), list(h)+ if not r: return len(h)+ d = np.arange(len(h)+1)+ for i in range(1, len(r)+1):+ prev = d[0]; d[0] = i+ for j in range(1, len(h)+1):+ cur = d[j]+ d[j] = min(d[j]+1, d[j-1]+1, prev + (r[i-1] != h[j-1]))+ prev = cur+ return int(d[len(h)])++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--refs", required=True)+ ap.add_argument("--hyp_dir", required=True)+ ap.add_argument("--time_file", default=None)+ a = ap.parse_args()+ refs = {}+ dur = {}+ for it in json.load(open(a.refs)):+ k = f"{it.get('id', it.get('key')):03d}" if isinstance(it.get('id', it.get('key')), int) else str(it.get('key'))+ refs[k] = it["ref"]; dur[k] = float(it.get("duration", 0))+ times = {}+ if a.time_file and os.path.exists(a.time_file):+ for ln in open(a.time_file):+ p = ln.split()+ if len(p) >= 2:+ try: times[p[0]] = float(p[1])+ except ValueError: pass+ E = N = 0; rt = ad = 0.0; n = 0+ for p in glob.glob(os.path.join(a.hyp_dir, "*.txt")):+ k = os.path.splitext(os.path.basename(p))[0]+ if k not in refs: continue+ h = normalize_zh(open(p).read()); r = normalize_zh(refs[k])+ E += edist(r, h); N += len(r); n += 1+ if k in times: rt += times[k]; ad += dur[k]+ cer = E / max(N, 1) * 100+ print(f"files={n} micro-CER={cer:.2f}%", end="")+ if ad > 0: print(f" RTF={rt/ad:.4f} ({ad/rt:.1f}x real-time)")+ else: print()++if __name__ == "__main__":+ main()