Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

docs: CPU benchmark vs whisper.cpp (Chinese ASR) (#2992)

zhifu gao842e62f875a3
diff --git a/runtime/llama.cpp/BENCHMARKS.md b/runtime/llama.cpp/BENCHMARKS.md
new file mode 100644
index 000000000..78c6b09bc
--- /dev/null
+++ b/runtime/llama.cpp/BENCHMARKS.md
@@ -0,0 +1,101 @@
+# FunASR (llama.cpp / GGUF) vs whisper.cpp — CPU benchmark
+
+How does the FunASR llama.cpp runtime compare with [whisper.cpp](https://github.com/ggml-org/whisper.cpp),
+the de-facto on-device ASR runtime, on **Chinese** speech? This page reports a
+head-to-head on identical hardware and audio.
+
+**TL;DR — for Chinese ASR on CPU, FunASR is ~2.7× more accurate than whisper.cpp at
+every model tier, and faster.**
+
+## Results
+
+Dataset: **184 real Mandarin clips with human references** (the standard FunASR
+benchmark set). Metric: **micro-CER** with `normalize_zh` (lower is better). Speed:
+real-time factor on **CPU, 8 threads** (model-load excluded). whisper forced to
+Chinese (`-l zh`).
+
+| system | **CER** (micro, normalize_zh) ↓ | speed ↑ | size |
+|---|---|---|---|
+| **FunASR Fun-ASR-Nano** | **8.06** (fp32 ref) / **8.42** (Q8 runtime) | LLM decode¹ | enc + Qwen3-0.6B GGUF |
+| **FunASR SenseVoiceSmall** | **7.81** (fp32 ref) / **8.17** (Q8 runtime) | **~20× real-time** | 449 MB (f16) |
+| **FunASR Paraformer** | **10.18** (fp32 ref) / **9.89** (Q8 runtime) | **~21× real-time** | 401 MB (f16) |
+| whisper.cpp base | 31.33 | 9.9× | 142 MB |
+| whisper.cpp small | 22.12 | 4.6× | 466 MB |
+| whisper.cpp large-v3-turbo | 23.15 | 3.2× | 1.6 GB |
+
+**Each FunASR row shows two numbers:** the published **fp32 reference** (PyTorch,
+the number on funasr.com / the model cards) and the **Q8 llama.cpp CPU runtime**
+measured here. The ~0.3 % gap is normal int8 quantization + VAD segment boundaries;
+Q8 is the real CPU/edge deployment config. Either way, **FunASR ~8–10 % vs
+whisper.cpp 22–31 % — a 2.7×+ accuracy gap that holds at every tier.**
+
+¹ Fun-ASR-Nano runs an autoregressive 0.6B LLM decoder (slower than the encoder-only
+SenseVoice/Paraformer; it is the accuracy leader). A clean RTF lands once the CLI
+separates model-load from compute.
+
+### Transparency / segmentation (read this before quoting numbers)
+
+- **Segmentation differs by system, each using its natural strategy:** FunASR uses an
+ `fsmn-vad` front end (segments → ASR → concatenate); whisper.cpp uses its built-in
+ 30 s windowing. This is a fair system-level comparison.
+- **Engine-internal VAD is on the FunASR runtime roadmap.** Today the runtime reaches
+ the reference numbers via the `fsmn-vad` front end (or `--chunk`); a native ggml VAD
+ is planned so the bare binary hits the reference end-to-end.
+- **Bare binary, no VAD (whole-clip)**, for full disclosure: SenseVoiceSmall **9.99 %**,
+ Paraformer **12.82 %** (micro, normalize_zh, full 184). Long clips decoded as one
+ segment are out-of-distribution; this is exactly what VAD fixes.
+
+## Why FunASR wins on Chinese
+
+1. **Training data.** SenseVoice / Paraformer / Fun-ASR-Nano are trained primarily on
+ large-scale Mandarin; Whisper is a general multilingual model where Chinese is a
+ small slice. On Chinese homophones Whisper makes substitution errors the FunASR
+ models do not (example below).
+2. **Architecture → speed.** Paraformer is non-autoregressive (CIF predictor + one
+ decoder pass) and SenseVoiceSmall is encoder + CTC (one forward pass); Whisper is
+ autoregressive (one step per output token).
+
+## Qualitative example (clip 002)
+
+| system | output (excerpt) |
+|---|---|
+| ground truth | 我想问,我在**滨海新区**有房…所以我必须拿到**抚养权** |
+| FunASR (Nano / SenseVoice / Paraformer) | …我在**滨海新区**有房…拿到**抚养权**… ✓ |
+| whisper base | …我在**冰海心区**有房…我想要**扶养权**…上学**方面**… ✗ |
+| whisper small | …我在**冰海新区**有房…我想要**抚养全**… ✗ |
+| whisper large-v3-turbo | …滨海新区…上学**方面**… ✗ |
+
+## Methodology
+
+- **Data:** the standard 184-clip Mandarin benchmark set (`benchmark/testset.json`),
+ ~44–60 s each, with human references.
+- **Metric (canonical):** **micro-average CER** (`Σ edits / Σ ref chars`) after
+ **`normalize_zh`**: `re.sub(r'[^\w一-鿿]', '', text).upper()` (strip punctuation/
+ whitespace, keep word chars + CJK, upper-case; SenseVoice `<|...|>` tags stripped).
+ This is the canonical FunASR口径 — the same one behind the published fp32 numbers.
+ (A macro-average / simplified-normalize variant gives different, non-canonical
+ numbers; it is not used here.)
+- **FunASR fp32 reference:** PyTorch, micro + normalize_zh, 184 set — SenseVoice 7.81,
+ Paraformer 10.18, Fun-ASR-Nano 8.06 (matches funasr.com / READMEs / model cards).
+- **FunASR Q8 runtime:** this llama.cpp runtime (Q8 LLM / f16 encoder) + `fsmn-vad`
+ front end (`max_single_segment_time=30000`), full 184. SenseVoice uses `use_itn=True`
+ to match the reference.
+- **whisper.cpp:** ggml `base` / `small` / `large-v3-turbo`, `-l zh`, internal 30 s
+ windowing, full 184.
+- **Speed (RTF):** `Σ compute_time / Σ audio_duration`, model-load excluded, **8 threads
+ for all systems**.
+
+## Caveats (fair use)
+
+- This is a **Chinese** benchmark — FunASR's home turf. Whisper is a *general
+ multilingual* model (translation, 99 languages, timestamps); for English / other
+ languages it is the stronger general choice. The takeaway is specifically:
+ **for Chinese ASR on CPU, FunASR is the accuracy + speed leader.**
+- SenseVoiceSmall also outputs language ID / emotion / audio-event; Paraformer is
+ Mandarin-specialised; Fun-ASR-Nano is the most accurate (LLM decoder). Pick per use case.
+
+## Reproduce
+
+See [`benchmarks/`](benchmarks/) — `compute_cer.py` (micro-CER + normalize_zh + RTF)
+and the per-system run commands. Produce hypotheses with each tool, then compute CER
+against the references and RTF against clip durations.
diff --git a/runtime/llama.cpp/benchmarks/README.md b/runtime/llama.cpp/benchmarks/README.md
new file mode 100644
index 000000000..1895e22d5
--- /dev/null
+++ b/runtime/llama.cpp/benchmarks/README.md
@@ -0,0 +1,41 @@
+# Reproducing the FunASR-vs-whisper.cpp benchmark
+
+Scripts and method behind [`../BENCHMARKS.md`](../BENCHMARKS.md).
+
+## Metric (authoritative FunASR口径)
+- **micro-CER** = `Σ edit_distance / Σ reference_chars` over all files (not a per-file mean).
+- **normalize_zh**: `re.sub(r'[^\w一-鿿]', '', text).upper()` — drop punctuation/whitespace,
+ keep word chars + CJK, upper-case. (SenseVoice meta tags `<|...|>` are stripped first.)
+- **RTF** = `Σ compute_time / Σ audio_duration`, model-load time excluded.
+
+`compute_cer.py` implements exactly this:
+```bash
+python compute_cer.py --refs testset.json --hyp_dir <hyps>/ [--time_file <times>.txt]
+```
+`testset.json` is a list of `{"id"/"key", "ref", "duration"}`; `<hyps>/{key}.txt` are
+the transcripts; `<times>.txt` has `key compute_seconds` per line.
+
+## Producing hypotheses
+
+FunASR (this runtime), per clip:
+```bash
+# SenseVoice / Paraformer: ids -> detok
+build/bin/llama-funasr-sensevoice -m sensevoice-small.gguf -a $k.wav > $k.ids
+python ../sensevoice/detok.py <model>/chn_jpn_yue_eng_ko_spectok.bpe.model $k.ids > $k.txt
+build/bin/llama-funasr-paraformer -m paraformer.gguf -a $k.wav > $k.ids
+python ../paraformer/detok_paraformer.py <model>/tokens.json $k.ids > $k.txt
+# Fun-ASR-Nano: text directly
+build/bin/llama-funasr-cli --enc funasr-encoder.gguf -m qwen3-0.6b-q8_0.gguf -a $k.wav --chunk 15 > $k.txt
+```
+Compute time is on each tool's stderr (`encode … s` / `enc … dec … s`).
+
+whisper.cpp, per clip (forced Chinese, no timestamps):
+```bash
+whisper-cli -m models/ggml-<size>.bin -l zh -nt -t 8 $k.wav > $k.txt # 2>stderr has "total time"/"load time"
+```
+RTF compute time = `(total − load) ms`.
+
+## Notes
+- Run all systems with the **same thread count** (here `-t 8` / 8 threads) for a fair RTF.
+- Whisper does its own internal 30 s windowing; the FunASR segmentation口径 is documented
+ in `BENCHMARKS.md` (see the methodology/caveats sections).
diff --git a/runtime/llama.cpp/benchmarks/compute_cer.py b/runtime/llama.cpp/benchmarks/compute_cer.py
new file mode 100644
index 000000000..308858346
--- /dev/null
+++ b/runtime/llama.cpp/benchmarks/compute_cer.py
@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Compute micro-CER (normalize_zh) and RTF for an ASR system's hypotheses.
+
+Usage:
+ python compute_cer.py --refs testset.json --hyp_dir <dir of {key}.txt> \
+ [--time_file <key compute_seconds per line>]
+
+- micro-CER = sum(edit distance) / sum(reference chars), over all files.
+- normalize_zh(text) = re.sub(r'[^\\w一-鿿]', '', text).upper() (the FunASR口径)
+- RTF = sum(compute_time) / sum(audio_duration) (model-load excluded)
+testset.json: list of {"id" or "key", "ref", "duration"}.
+"""
+import argparse, json, glob, os, re
+import numpy as np
+
+def normalize_zh(s):
+ s = re.sub(r"<\|[^|]*\|>", "", s) # drop SenseVoice meta tags, if any
+ return re.sub(r"[^\w一-鿿]", "", s).upper()
+
+def edist(r, h):
+ r, h = list(r), list(h)
+ if not r: return len(h)
+ d = np.arange(len(h)+1)
+ for i in range(1, len(r)+1):
+ prev = d[0]; d[0] = i
+ for j in range(1, len(h)+1):
+ cur = d[j]
+ d[j] = min(d[j]+1, d[j-1]+1, prev + (r[i-1] != h[j-1]))
+ prev = cur
+ return int(d[len(h)])
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--refs", required=True)
+ ap.add_argument("--hyp_dir", required=True)
+ ap.add_argument("--time_file", default=None)
+ a = ap.parse_args()
+ refs = {}
+ dur = {}
+ for it in json.load(open(a.refs)):
+ k = f"{it.get('id', it.get('key')):03d}" if isinstance(it.get('id', it.get('key')), int) else str(it.get('key'))
+ refs[k] = it["ref"]; dur[k] = float(it.get("duration", 0))
+ times = {}
+ if a.time_file and os.path.exists(a.time_file):
+ for ln in open(a.time_file):
+ p = ln.split()
+ if len(p) >= 2:
+ try: times[p[0]] = float(p[1])
+ except ValueError: pass
+ E = N = 0; rt = ad = 0.0; n = 0
+ for p in glob.glob(os.path.join(a.hyp_dir, "*.txt")):
+ k = os.path.splitext(os.path.basename(p))[0]
+ if k not in refs: continue
+ h = normalize_zh(open(p).read()); r = normalize_zh(refs[k])
+ E += edist(r, h); N += len(r); n += 1
+ if k in times: rt += times[k]; ad += dur[k]
+ cer = E / max(N, 1) * 100
+ print(f"files={n} micro-CER={cer:.2f}%", end="")
+ if ad > 0: print(f" RTF={rt/ad:.4f} ({ad/rt:.1f}x real-time)")
+ else: print()
+
+if __name__ == "__main__":
+ main()