Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
Add llama.cpp / GGUF runtime for FunASR models (#2988)
zhifu gaob7da5cc70680
diff --git a/runtime/llama.cpp/DESIGN.md b/runtime/llama.cpp/DESIGN.mdnew file mode 100644index 000000000..c427bedd4--- /dev/null+++ b/runtime/llama.cpp/DESIGN.md@@ -0,0 +1,337 @@+# FunASR on llama.cpp / GGUF — Design Document++This document describes the design of the `runtime/llama.cpp` directory: a C++ /+ggml runtime that runs FunASR models (Fun-ASR-Nano, SenseVoiceSmall, Paraformer)+without PyTorch, on CPU and edge devices, with quantized GGUF weights. It is the+counterpart of [whisper.cpp](https://github.com/ggml-org/whisper.cpp) for FunASR.++It is written to be read without the source: it explains *why* the runtime exists,+*how* each model maps onto ggml, the GGUF weight format, the numerical-fidelity and+validation methodology, the non-obvious gotchas discovered during the port, and the+roadmap.++> This is the **shared design document** for the FunASR-on-llama.cpp effort and is+> kept identical across the FunASR family repos (modelscope/FunASR, Fun-ASR,+> SenseVoice). The three models share one ggml SAN-M encoder / FSMN / fbank+> foundation, so the design is documented once here in full; a single-model repo+> ships only the relevant model directory (§2) but the shared design still applies.++---++## 1. Motivation++FunASR's reference inference runs on PyTorch (and vLLM for the LLM-based models) on+GPU. That is the right tool for a server that batches many requests and wants to+saturate a GPU. It is the wrong tool when there is **no GPU and no Python**: a+laptop, a phone, a Raspberry Pi, an embedded C/C++ application, an offline desktop+app. There, you want a single self-contained binary, a few hundred MB of quantized+weights, and CPU SIMD.++[llama.cpp](https://github.com/ggml-org/llama.cpp) / ggml is the de-facto runtime+for that world (Ollama, LM Studio, whisper.cpp all build on it). Porting FunASR to+ggml + GGUF makes FunASR run anywhere llama.cpp runs, dramatically widening the+deployment surface.++| | PyTorch / vLLM (existing) | this runtime (llama.cpp) |+|---|---|---|+| target | GPU server, high QPS | CPU / edge / embedded |+| deps | Python + CUDA + PyTorch | none (C/C++ single binary) |+| weights | HF fp16/bf16 safetensors | GGUF, 2–8 bit quantization |+| key tech | PagedAttention, continuous batching | quantization, mmap, CPU SIMD |+| best for | online service, batch eval | offline, on-device, embedded |++These are complementary, not competing: cloud serving stays on vLLM; this runtime+covers the on-device / offline case.++---++## 2. System overview++Three models are supported. They share more than they differ — all three use the+same **SAN-M encoder**, the same **FSMN memory block**, the same **kaldi-compatible+fbank front end**, and the same ggml building blocks.++```+ ┌─────────────────────── shared C++ / ggml ───────────────────────┐+ audio.wav (16k mono) ──► kaldi 80-mel fbank + LFR(7/6) ──► SAN-M encoder (50 layers, ggml)+ └──────────────────────────────────────────────────────────────────┘+ │ encoder_out [T, 512]+ ┌───────────────────────────────────┼───────────────────────────────────┐+ Fun-ASR-Nano SenseVoiceSmall Paraformer+ adaptor → audio embeds + 4 query tokens CIF predictor (host)+ → inject into Qwen3-0.6B CTC head → greedy CTC → SAN-M decoder (cross-attn)+ (llama_decode embd path) → SentencePiece → argmax → tokens.json+ → text → text → text+```++| model | head / decoder | autoregressive? | output units |+|---|---|---|---|+| Fun-ASR-Nano | adaptor + Qwen3-0.6B LLM | yes (LLM) | Qwen3 BPE |+| SenseVoiceSmall | CTC | no | spectok BPE (25055) |+| Paraformer | CIF + SAN-M decoder | no (parallel) | char/BPE (8404) |++Directory layout:+```+runtime/llama.cpp/+ README.md overview+ DESIGN.md this document+ fun-asr-nano/ funasr-cli, funasr-encoder, funasr-embd, export_encoder_gguf.py+ sensevoice/ funasr-sensevoice, export_sensevoice_gguf.py, detok.py+ paraformer/ funasr-paraformer, export_paraformer_gguf.py, detok_paraformer.py+```+Each model dir holds the llama.cpp example sources (drop-in under `examples/`), a+GGUF export script, and a model-specific README.++---++## 3. Audio front end (kaldi fbank in C++)++All models use FunASR's `WavFrontend`: kaldi-compatible 80-bin log-mel fbank with a+hamming window (25 ms / 10 ms), pre-emphasis 0.97, DC removal, 512-pt FFT, then+**Low-Frame-Rate (LFR)** stacking of 7 frames with stride 6 → a 560-dim feature+per output frame.++The C++ implementation (`compute_fbank`) reproduces this exactly:+1. upscale the waveform by 32768 (FunASR feeds int16-range samples to kaldi),+2. per frame: remove DC offset, pre-emphasis, hamming window, zero-pad to 512,+3. radix-2 FFT, power spectrum, 80 triangular mel filters (kaldi mel: `1127·ln(1+f/700)`,+ low 20 Hz, high 8000 Hz), log floor `FLT_EPSILON`,+4. LFR: left-pad 3 copies of frame 0, stack 7 frames stride 6 → 560-dim.++**Validation:** vs torchaudio kaldi.fbank (dither=0), cosine **1.000000**,+max_abs_diff 1.75e-3.++**Gotcha — dither.** FunASR's frontend uses `dither=1.0` by default, which adds+random noise per sample, so the fbank (and everything downstream) is *non-deterministic*+in the reference. The C++ front end uses dither=0 (deterministic). The model is+robust to this; it accounts for the small (<1%) cosine gap seen when comparing+against a dithered reference.++---++## 4. The SAN-M encoder in ggml++The SenseVoice/Paraformer encoder is a 50-layer (Paraformer) or 50+20-layer+(SenseVoice, with extra `tp_encoders`) **SAN-M** stack. Each layer is pre-norm:++```+x → LN → SAN-M self-attention → +residual → LN → FFN(relu) → +residual+```++SAN-M self-attention = standard multi-head attention **plus** an FSMN memory branch+that runs in parallel on the value projection and is added to the attention output:++```+q,k,v = split(linear_q_k_v(x)) # one fused projection+fsmn = FSMN(v) # depthwise conv over time + residual+attn = softmax(qkᵀ/√d)·v → linear_out+out = attn + fsmn+```++### 4.1 FSMN as an exact f32 shift-accumulate (design decision)++FSMN is a per-channel (depthwise) 1-D convolution over time with a symmetric+kernel (size 11). ggml has `ggml_conv_1d_dw`, but it (a) requires the kernel in+F16 and (b) is flagged as "very likely wrong for some cases" upstream. Both are+unacceptable for a faithful port.++Instead FSMN is implemented as an **exact f32 shift-accumulate**: the kernel is+exported as `[K, D]`, the value tensor is zero-padded by `(K-1)/2` on each side+along time, and the output is `Σ_j kernel[:,j] ⊙ pad(v)[:, t+j]` plus the residual.+This is 11 element-wise multiply-adds — exact in f32, no F16 rounding, no dependence+on the questionable conv kernel. It dropped the full-encoder max_abs_diff vs PyTorch+from 2.93 to **0.0052**.++### 4.2 Position encoding & input scaling++Input is pre-scaled by `√(d_model)=√512` then a sinusoidal position encoding is+added, with **depth = the input feature dim (560)** and **positions starting at 1**+(not 0) — both quirks of the FunASR encoder that must be matched exactly.++### 4.3 LayerNorm++eps = 1e-5 everywhere.++**Validation:** first layer cosine 1.0 (max_abs_diff 1.8e-4); full encoder cosine+**1.000000**, max_abs_diff 5.2e-3 (f32).++---++## 5. Per-model design++### 5.1 Fun-ASR-Nano (encoder + adaptor + LLM)++Pipeline: `fbank → encoder → adaptor → audio embeds [T', 1024] → inject into+Qwen3-0.6B → text`.++- **LLM half is native.** Qwen3 is supported by llama.cpp, so the extracted+ Qwen3-0.6B converts to GGUF with the stock `convert_hf_to_gguf.py` and runs+ unchanged.+- **Embedding injection.** The audio embeddings are fed into the LLM through+ `llama_decode`'s embedding-input path — exactly how llava/mtmd inject vision+ embeddings. The integrated CLI builds the prompt as a *mixed* sequence:+ `[prefix tokens | audio embeds | suffix tokens]`, where prefix/suffix are fed as+ token ids (llama.cpp embeds them internally; `llama_tokenize(parse_special=true)`+ reproduces the exact 18-token prefix) and the audio slot is fed as embeddings.+- **Low-frame-rate truncation (critical).** The adaptor emits `T'` frames, but the+ model only uses the first `fake_token_len` of them as audio tokens, where+ `fake_token_len` derives from the fbank length by a 3-stage `÷2` formula+ (≈ T'/8). Feeding all `T'` frames is out-of-distribution and makes the LLM loop.+- **Chunking.** Decoding a long (e.g. 60 s) clip as one segment is OOD and triggers+ greedy repetition; the CLI's `--chunk 15` splits into windows with a fresh KV per+ window, dropping micro-CER from ~29% to ~9.5%.+- **Numerics.** The adaptor output has large magnitude (std ≈ 28, |max| ≈ 1187), so+ fp16 can overflow; the runtime uses f32/f16 weights with f32 activations.++### 5.2 SenseVoiceSmall (encoder + CTC)++Pipeline: `fbank → prepend 4 query tokens → encoder → CTC head → greedy CTC →+SentencePiece`.++- **Query tokens.** Four learned embeddings are prepended: `[language(auto), event,+ emotion, textnorm]` (indices `[0,1,2,15]` for auto/woitn). They are 560-dim and+ prepended *before* the encoder's `√512` scaling and position encoding.+- **CTC decode.** `argmax → collapse consecutive → drop blank(0)` → ids → SentencePiece.+- **Gotcha — no CMVN at inference.** SenseVoice's `inference()` feeds the **raw**+ log-mel fbank to the encoder; it does **not** apply `am.mvn` CMVN (that code path+ is unused at inference). Applying CMVN makes the model predict `<|nospeech|>`.++**Validation:** CTC token ids **identical** to PyTorch (108/108 on a clip); text+matches `AutoModel` exactly.++### 5.3 Paraformer (encoder + CIF + decoder, non-autoregressive)++Pipeline: `fbank → CMVN → encoder → CIF predictor → acoustic embeds [N, 512] →+SAN-M decoder (cross-attn to encoder) → argmax → tokens.json`.++- **CMVN IS applied** here (unlike SenseVoice): `(fbank + shift)·scale`, per-dim+ (560), from `am.mvn`.+- **CIF predictor (runs on host).** Continuous Integrate-and-Fire: a 1-D conv+ (k=3) + residual + relu + linear → sigmoid → per-frame weight α; then a sequential+ integrate-and-fire loop emits one acoustic embedding each time the running α-sum+ crosses 1.0. This both decides the token count and produces the decoder input. It+ is inherently sequential, so it runs in plain C++ (cheap: ~0.5 G MACs); the+ encoder and decoder run in ggml.+- **SAN-M decoder (ggml).** 16 layers, each: `FFN → FSMN self-attention →+ cross-attention to the encoder output`. The self-attention is **FSMN-only** (no+ QK attention); cross-attention has q from the decoder slots and k,v from a fused+ `linear_k_v` of the encoder output. A 17th `decoders3` layer is FFN-only. The+ decoder FFN has an internal LayerNorm and the second linear has no bias. The+ layer ordering (FFN *before* the attention inside the residual) is unusual and is+ matched exactly.++**Validation:** decoded text **identical** to `AutoModel`; CIF token count exact+(105/105). Encoder cosine 0.997 (residual is the reference's random dither).++**Gotcha — `am.mvn` has three bracketed blocks.** `[Splice idx]`, `[AddShift=shift]`,+`[Rescale=scale]`. The shift/scale are the two 560-length vectors; naively taking+the first two blocks grabs `[0]` as the shift and mis-scales everything, which makes+CIF emit ~4× too few tokens. Parse by length.++---++## 6. GGUF conversion & weight layout++Each model has an `export_*_gguf.py` that packs weights + architecture metadata into+a single GGUF.++- Tensor names are kept verbatim from the checkpoint (e.g.+ `encoder.encoders.3.norm1.weight`); the C++ looks them up by name.+- **FSMN kernels** are transposed from `(D,1,K)` to `[K,D]` at export so the C+++ shift-accumulate can take a contiguous per-tap `[D]` vector.+- **CMVN** (`am.mvn`) is parsed to `cmvn.shift` / `cmvn.scale` tensors (Paraformer+ uses them; SenseVoice ships them but the runtime ignores them).+- **Quantization.** `--wtype f16` stores the 2-D matmul weights as F16 (norms,+ biases, FSMN kernels stay f32), halving the encoder GGUF (e.g. 935 → 469 MB) with+ cosine 0.999999. The Qwen3 LLM uses the standard llama.cpp quantizer (Q8_0 / Q4_K_M).++| file | model | dtype | size |+|---|---|---|---|+| funasr-encoder.gguf | Nano | f32 / f16 | 935 / 469 MB |+| qwen3-0.6b-q8_0.gguf | Nano LLM | Q8_0 | 805 MB |+| sensevoice-small.gguf | SenseVoice | f32 | 936 MB |+| paraformer.gguf | Paraformer | f32 | 863 MB |++---++## 7. Numerical fidelity & validation methodology++The port is validated **stage by stage** against the PyTorch reference, using golden+dumps (fbank, encoder output, adaptor/CIF output, logits/ids) compared by cosine+similarity and max-abs-diff, then end-to-end by transcription text / CER.++Summary of results (benchmark clip / set):++| stage | metric |+|---|---|+| kaldi fbank vs torchaudio | cosine 1.000000 |+| SAN-M encoder (full) vs PyTorch | cosine 1.000000, max_abs_diff 5e-3 (f32) |+| SenseVoice CTC ids | identical (108/108) |+| Paraformer text / token count | identical / 105 = 105 |+| Fun-ASR-Nano end-to-end CER (same conditions) | C++ 11.68% vs PyTorch 11.70% (Δ0.02%) |++**Why not bit-exact tokens everywhere?** Greedy decoding is chaotic: a ~5e-3+difference (from ggml-CPU vs torch-GPU matmul summation order) can flip a token on+a borderline frame, and over a long sequence the paths diverge — *this also happens+between PyTorch's own GPU and CPU*. What is faithful and what we verify is (a) the+per-tensor numerics (cosine 1.0) and (b) the **aggregate CER**, which matches the+reference under identical conditions.++**fp16 caution.** The Fun-ASR-Nano adaptor output magnitude (std ≈ 28) can overflow+fp16; the audio path is kept in f32 (weights may be f16, activations f32).++---++## 8. Performance++CPU, 8 threads, a 44 s clip:+- Encoder (50 layers): ~1.2 s. Paraformer decoder: ~0.5 s.+- Fun-ASR-Nano end-to-end (with LLM): ~7 s.+- Fully-quantized footprint (f16 encoder + Q8 LLM) ≈ 1.3 GB.++These are first-correctness numbers; quantizing the encoder and threading/batching+the front end are open optimizations.++---++## 9. Design decisions & trade-offs++- **ggml for encoder/decoder, host C++ for CIF.** The neural matmul-heavy parts run+ in ggml (SIMD, future GPU backends); CIF is a sequential scalar loop with data-+ dependent control flow, so it is clearer and not slower in plain C++.+- **Exact f32 FSMN instead of `ggml_conv_1d_dw`.** Correctness and f32 precision+ over reusing a flagged, F16-only op (§4.1).+- **Prompt as tokens, not a Python embedding table.** The integrated CLI tokenizes+ the prompt with `llama_tokenize` and lets llama.cpp embed it, so no embedding+ matrix needs to be shipped or matched (Fun-ASR-Nano).+- **f32 by default, f16/Q8 opt-in.** f32 is the faithful default; quantization is a+ size/latency lever the user opts into. (Interestingly, Q8 on the LLM slightly+ *helps* greedy stability by regularizing away from repetition loops.)+- **Per-model self-contained example dirs.** Mirrors llama.cpp's `examples/` layout+ so each builds as a drop-in target; the shared code is duplicated rather than+ factored to keep each example independently buildable.++---++## 10. Limitations & roadmap++- **WAV input** assumes 16 kHz mono PCM16; arbitrary formats / resampling are TODO.+- **VAD.** Long audio needs segmentation; today Fun-ASR-Nano uses fixed `--chunk`+ windows. A real FSMN-VAD front end would close the last ~1.3% CER gap to the+ production VAD-segmented number and is the highest-value next step.+- **Single packaged GGUF** (encoder + adaptor + LLM in one file) and a one-command+ converter.+- **Encoder/decoder quantization** (Q8 via gguf-py quants), streaming, timestamps+ (Paraformer CIF peaks give alignment; SenseVoice/Nano via CTC).+- **Upstream.** The example sources are drop-in for llama.cpp; upstreaming the+ runtime to ggml-org/llama.cpp (as whisper.cpp-style tools) is a separate track.++---++## 11. Reproducing the validation++Each model dir's README has the build + convert + run quickstart. The export+scripts read a standard FunASR checkpoint (`model.pt` + `config.yaml` + `am.mvn` /+tokenizer). To reproduce a stage comparison, dump the corresponding PyTorch tensor+(`model.encode`, `model.calc_predictor`, …) and compare with cosine / max-abs-diff;+the numbers in §7 should reproduce within dither noise.diff --git a/runtime/llama.cpp/README.md b/runtime/llama.cpp/README.mdnew file mode 100644index 000000000..f3bf03d65--- /dev/null+++ b/runtime/llama.cpp/README.md@@ -0,0 +1,54 @@+# FunASR on llama.cpp / GGUF++Run FunASR models on the [llama.cpp](https://github.com/ggml-org/llama.cpp) / ggml+stack — **CPU, edge, a single binary, no Python at runtime, quantized weights**.+This is to FunASR what [whisper.cpp](https://github.com/ggml-org/whisper.cpp) is to+Whisper: it lets the models run where there is no GPU and no Python (laptops,+phones, edge boxes, embedded C/C++ apps), complementing the PyTorch / ONNX / vLLM+paths used for GPU serving.++## Models++| model | architecture | runtime | status |+|---|---|---|---|+| [Fun-ASR-Nano](fun-asr-nano/) | SenseVoice SAN-M encoder + adaptor + Qwen3-0.6B LLM | `llama-funasr-cli` | validated vs PyTorch |+| [SenseVoiceSmall](sensevoice/) | SAN-M encoder + CTC | `llama-funasr-sensevoice` | CTC ids identical to PyTorch |+| [Paraformer](paraformer/) | SAN-M encoder + CIF predictor + SAN-M decoder (non-autoregressive) | `llama-funasr-paraformer` | text identical to PyTorch |++All three share the same ggml SAN-M encoder / FSMN / attention primitives and the+same kaldi-compatible fbank front end (80-mel, LFR 7/6), so the C++ is consistent+across models.++## How it works++Each model's neural path is implemented as a ggml graph; the audio front end (kaldi+fbank) is plain C++. Weights are converted to GGUF (f32 or f16) with the per-model+`export_*_gguf.py` script. For Fun-ASR-Nano the LLM half is a standard Qwen3 GGUF+and the audio embeddings are injected into it via `llama_decode`'s embedding input+(the llava/mtmd mechanism). See each model's README for the architecture diagram,+build/convert/run quickstart, validation numbers, and gotchas.++## Build (shared)+```bash+git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp+cp -r /path/to/runtime/llama.cpp/<model>/<example-dir> examples/+echo 'add_subdirectory(<example-dir>)' >> examples/CMakeLists.txt+cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF+cmake --build build -j --target <target>+```++## Validation++Each model was validated against the FunASR PyTorch reference (encoder cosine ≈ 1.0;+SenseVoice CTC token ids identical; Paraformer text identical; Fun-ASR-Nano aggregate+CER matches PyTorch within 0.02% under identical conditions). See per-model READMEs.++## Status / notes+- WAV input currently assumes 16 kHz mono PCM16.+- For long audio, Fun-ASR-Nano supports `--chunk` windowing; a proper VAD front end+ is on the roadmap.+- This adds a new `runtime/llama.cpp/` directory only; no existing code is modified.++## Further reading++See [DESIGN.md](DESIGN.md) for the full system design — architecture, the shared SAN-M encoder, GGUF weight format, numerical-fidelity and validation methodology, design trade-offs, and gotchas.diff --git a/runtime/llama.cpp/fun-asr-nano/README.md b/runtime/llama.cpp/fun-asr-nano/README.mdnew file mode 100644index 000000000..64eefd675--- /dev/null+++ b/runtime/llama.cpp/fun-asr-nano/README.md@@ -0,0 +1,144 @@+# Fun-ASR-Nano on llama.cpp / GGUF++Run **Fun-ASR-Nano** entirely on the [llama.cpp](https://github.com/ggml-org/llama.cpp)+/ ggml stack — **CPU, edge, a single binary, no Python at runtime**. This is to+Fun-ASR what [whisper.cpp](https://github.com/ggml-org/whisper.cpp) is to Whisper.++## Why this exists++Fun-ASR-Nano normally runs on PyTorch / vLLM (GPU). That is great for a server+serving many requests, but it cannot run where there is no GPU and no Python.+This runtime ports the model to **ggml + GGUF**, so Fun-ASR-Nano can run:++- on a laptop / phone / Raspberry Pi / edge box, offline, CPU-only;+- embedded directly into a C/C++ application (one static binary);+- with quantized weights (Q8 / Q4), shrinking the model to ~1.3 GB total.++| | vLLM (existing) | this runtime (llama.cpp) |+|---|---|---|+| target | GPU server, high QPS | CPU / edge / embedded |+| deps | Python + CUDA + PyTorch | none (C/C++ single binary) |+| weights | HF fp16/bf16 | GGUF, quantized |+| best for | online service, batch | offline, on-device |++## Architecture++Fun-ASR-Nano = **SenseVoice SAN-M encoder (70 layers) + adaptor + Qwen3-0.6B LLM**.+The whole pipeline runs in C++:++```+ audio.wav (16k mono)+ │ kaldi 80-mel fbank + LFR (C++)+ ▼+ features [T, 560]+ │ SAN-M encoder + adaptor (ggml) ── funasr-encoder.gguf+ ▼+ audio embeds [T', 1024]+ │ keep first fake_token_len frames (low-frame-rate)+ ▼+ [ prefix tokens | audio embeds | suffix tokens ]+ │ Qwen3-0.6B, embeds injected via llama_decode (llava/mtmd style) ── qwen3-0.6b.gguf+ ▼+ transcription+```++The audio embeddings are fed into the LLM through `llama_decode`'s embedding-input+path — exactly how llava/mtmd inject vision embeddings.++## Quickstart++**1. Build** (drop the examples into a llama.cpp checkout):+```bash+git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp+cp -r /path/to/runtime/llama.cpp/funasr-cli examples/+echo 'add_subdirectory(funasr-cli)' >> examples/CMakeLists.txt+cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF+cmake --build build -j --target llama-funasr-cli+```++**2. Convert weights to GGUF** (one-time; needs the checkpoint, e.g.+`FunAudioLLM/Fun-ASR-Nano-2512`):+```bash+# LLM half — Qwen3-0.6B is natively supported by llama.cpp+python llama.cpp/convert_hf_to_gguf.py <model>/Qwen3-0.6B-vllm \+ --outfile qwen3-0.6b-f32.gguf --outtype f32+build/bin/llama-quantize qwen3-0.6b-f32.gguf qwen3-0.6b-q8_0.gguf Q8_0 # smaller, recommended++# audio half — SenseVoice encoder + adaptor+python runtime/llama.cpp/export_encoder_gguf.py \+ --model_pt <model>/model.pt --out funasr-encoder.gguf # f32, 935 MB+python runtime/llama.cpp/export_encoder_gguf.py \+ --model_pt <model>/model.pt --out funasr-encoder-f16.gguf --wtype f16 # 469 MB+```++**3. Transcribe:**+```bash+build/bin/llama-funasr-cli \+ --enc funasr-encoder.gguf -m qwen3-0.6b-q8_0.gguf \+ -a audio.wav --chunk 15+```+Expected output (one of the benchmark clips):+```+我想问我在滨海新区有房我一直没有照顾孩子但是我想要抚养权...你觉得这是正常的想法吗+[done] 7.40s ; chunk=15s+```++## Models & sizes++| file | dtype | size |+|---|---|---|+| funasr-encoder.gguf | f32 | 935 MB |+| funasr-encoder-f16.gguf | f16 (matmul weights) | 469 MB |+| qwen3-0.6b-f32.gguf | f32 | 3.0 GB |+| qwen3-0.6b-q8_0.gguf | Q8_0 | 805 MB |+| qwen3-0.6b-q4km.gguf | Q4_K_M | 484 MB |++Fully-quantized config (f16 encoder + Q8 LLM) ≈ **1.3 GB**, edge-friendly.++## Accuracy & validation++Validated against the PyTorch reference on the 184-file benchmark:++- **Encoder + adaptor (ggml) vs PyTorch:** cosine **1.000000**, max_abs_diff **5e-3** (f32).+- **kaldi fbank (C++) vs torchaudio:** cosine **1.000000**.+- **End-to-end CER, identical conditions (f32 LLM, 15 s chunking):**+ C++ macro 17.41% / micro 11.68% vs PyTorch macro 17.42% / micro 11.70%+ → aggregate CER matches to **0.02%**; the port is faithful.+- Best practical config (Q8 LLM + 15 s chunking): **micro-CER 9.51%** (production+ VAD-segmented reference is ~8.2%; the gap is fixed-window vs VAD, not the port).++## Tips & gotchas++- **Use `--chunk 15`** for long audio. Decoding a whole 60 s clip in one segment is+ out-of-distribution and makes greedy decoding loop; 15 s windows fix it+ (micro-CER 29% → 9.5%).+- **Low-frame-rate truncation** is required: only the first `fake_token_len`+ adaptor frames are real audio tokens. The CLI does this automatically; feeding+ all frames makes the LLM repeat.+- **Use bf16/fp32, avoid fp16 for the audio path** — the adaptor output has large+ magnitude (std ≈ 28, |max| ≈ 1187); fp16 can overflow. The GGUFs here are f32/f16+ weights with f32 activations, which is safe.+- **WAV input** currently assumes 16 kHz mono PCM16. Resample first if needed.+- Q8 quantization slightly *helps* greedy stability (quant noise regularizes away+ from repetition loops), so Q8 is a good default.++## Implementation notes++- FSMN depthwise memory is an exact f32 shift-accumulate (avoids the F16-only,+ upstream-flagged `ggml_conv_1d_dw`).+- LayerNorm eps = 1e-5; sinusoidal position encoding depth = input feature dim (560),+ positions start at 1; encoder input pre-scaled by sqrt(512).+- Prompt is fed as tokens via `llama_tokenize(parse_special=true)` (prefix = 18+ tokens, matching the HF tokenizer), so no Python embedding table is needed.++## Files+```+funasr-cli/ integrated binary: WAV → transcription+funasr-encoder/ encoder+adaptor only (ggml) — validation/debugging+funasr-embd/ LLM decode from precomputed embeds — validation/debugging+export_encoder_gguf.py export the audio encoder + adaptor to GGUF+```++## Roadmap+- True FSMN-VAD segmentation (replace fixed windows; closes the last ~1.3% CER).+- Arbitrary WAV formats / resampling; encoder Q8 quantization; single packaged GGUF.diff --git a/runtime/llama.cpp/fun-asr-nano/export_encoder_gguf.py b/runtime/llama.cpp/fun-asr-nano/export_encoder_gguf.pynew file mode 100644index 000000000..2804200f1--- /dev/null+++ b/runtime/llama.cpp/fun-asr-nano/export_encoder_gguf.py@@ -0,0 +1,72 @@+#!/usr/bin/env python3+"""Export Fun-ASR-Nano audio encoder + adaptor weights to a GGUF file.++Packs all `audio_encoder.*` and `audio_adaptor.*` tensors (bf16 -> f32) plus+architecture metadata into funasr-encoder.gguf, for the ggml C++ forward pass.+Tensor names are kept verbatim (e.g. audio_encoder.encoders.3.norm1.weight) so+the C++ side can look them up directly.+"""+import argparse, os+import numpy as np+import torch+import gguf+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--model_pt", required=True)+ ap.add_argument("--out", required=True)+ ap.add_argument("--wtype", default="f32", choices=["f32", "f16"],+ help="dtype for 2D Linear (matmul) weights; norms/bias/fsmn stay f32")+ args = ap.parse_args()++ sd = torch.load(args.model_pt, map_location="cpu")+ sd = sd.get("state_dict", sd)++ w = gguf.GGUFWriter(args.out, "funasr-sensevoice-encoder")++ # --- architecture metadata (from config.yaml) ---+ w.add_uint32("funasr.enc.input_size", 560) # lfr_m(7) * n_mels(80)+ w.add_uint32("funasr.enc.output_size", 512)+ w.add_uint32("funasr.enc.attention_heads", 4)+ w.add_uint32("funasr.enc.linear_units", 2048)+ w.add_uint32("funasr.enc.num_blocks", 50) # encoders0(1) + encoders(49)+ w.add_uint32("funasr.enc.tp_blocks", 20)+ w.add_uint32("funasr.enc.kernel_size", 11)+ w.add_uint32("funasr.enc.sanm_shfit", 0)+ w.add_uint32("funasr.adp.llm_dim", 1024)+ w.add_uint32("funasr.adp.encoder_dim", 512)+ w.add_uint32("funasr.adp.ffn_dim", 2048)+ w.add_uint32("funasr.adp.n_layer", 2)+ w.add_uint32("funasr.adp.attention_heads", 8)+ w.add_uint32("funasr.adp.downsample_rate", 1)+ w.add_uint32("funasr.frontend.n_mels", 80)+ w.add_uint32("funasr.frontend.lfr_m", 7)+ w.add_uint32("funasr.frontend.lfr_n", 6)++ n = 0+ for k, v in sd.items():+ if not (k.startswith("audio_encoder.") or k.startswith("audio_adaptor.")):+ continue+ arr = v.detach().to(torch.float32).contiguous().numpy()+ # FSMN depthwise kernel: store as (K, D) so the C++ side can slice a+ # contiguous per-tap [D] vector and do an exact f32 shift-accumulate+ # (avoids the F16-only ggml_conv_1d_dw path).+ if k.endswith("fsmn_block.weight"): # (D, 1, K) -> (K, D)+ arr = np.ascontiguousarray(arr[:, 0, :].T)+ # matmul (Linear) weights -> optional f16; norms/biases/fsmn stay f32+ elif args.wtype == "f16" and arr.ndim == 2 and "norm" not in k:+ arr = arr.astype(np.float16)+ w.add_tensor(k, arr)+ n += 1+ print(f"writing {n} tensors to {args.out}")++ w.write_header_to_file()+ w.write_kv_data_to_file()+ w.write_tensors_to_file()+ w.close()+ print(f"done: {args.out} ({os.path.getsize(args.out)/1e6:.1f} MB)")+++if __name__ == "__main__":+ main()diff --git a/runtime/llama.cpp/fun-asr-nano/funasr-cli/CMakeLists.txt b/runtime/llama.cpp/fun-asr-nano/funasr-cli/CMakeLists.txtnew file mode 100644index 000000000..6c2d09d33--- /dev/null+++ b/runtime/llama.cpp/fun-asr-nano/funasr-cli/CMakeLists.txt@@ -0,0 +1,5 @@+set(TARGET llama-funasr-cli)+add_executable(${TARGET} funasr-cli.cpp)+install(TARGETS ${TARGET} RUNTIME)+target_link_libraries(${TARGET} PRIVATE llama ggml ${CMAKE_THREAD_LIBS_INIT})+target_compile_features(${TARGET} PRIVATE cxx_std_17)diff --git a/runtime/llama.cpp/fun-asr-nano/funasr-cli/funasr-cli.cpp b/runtime/llama.cpp/fun-asr-nano/funasr-cli/funasr-cli.cppnew file mode 100644index 000000000..f63fdd339--- /dev/null+++ b/runtime/llama.cpp/fun-asr-nano/funasr-cli/funasr-cli.cpp@@ -0,0 +1,258 @@+// funasr-cli: end-to-end Fun-ASR-Nano in C++ on the llama.cpp / ggml stack.+//+// wav(16k mono) -> kaldi fbank -> SAN-M encoder + adaptor (ggml) ->+// low-frame-rate truncation -> [prefix tokens | audio embeds | suffix tokens]+// -> Qwen3 LLM (llama.cpp) -> transcription.+//+// This is the whisper.cpp-style single-binary path: no Python at runtime.+//+// funasr-cli --enc funasr-encoder.gguf -m qwen3-0.6b.gguf -a audio.wav++#include "ggml.h"+#include "ggml-cpu.h"+#include "ggml-alloc.h"+#include "ggml-backend.h"+#include "gguf.h"+#include "llama.h"++#include <cmath>+#include <cstdint>+#include <cstdio>+#include <cstring>+#include <map>+#include <string>+#include <vector>++// ======================= WAV (16 kHz mono PCM16) =======================+static bool read_wav16(const char * path, std::vector<float> & out) {+ FILE * f = fopen(path, "rb"); if (!f) return false;+ char riff[4]; fread(riff, 1, 4, f);+ if (strncmp(riff, "RIFF", 4)) { fclose(f); return false; }+ fseek(f, 4, SEEK_CUR); char wave[4]; fread(wave, 1, 4, f);+ if (strncmp(wave, "WAVE", 4)) { fclose(f); return false; }+ uint16_t ch = 1, bits = 16; uint32_t sr = 16000;+ while (!feof(f)) {+ char id[4]; if (fread(id, 1, 4, f) != 4) break;+ uint32_t sz; if (fread(&sz, 4, 1, f) != 1) break;+ if (!strncmp(id, "fmt ", 4)) {+ uint16_t fmt; fread(&fmt, 2, 1, f); fread(&ch, 2, 1, f);+ fread(&sr, 4, 1, f); fseek(f, 6, SEEK_CUR); fread(&bits, 2, 1, f);+ if (sz > 16) fseek(f, sz - 16, SEEK_CUR);+ } else if (!strncmp(id, "data", 4)) {+ if (ch < 1 || bits != 16) { fclose(f); fprintf(stderr, "only 16-bit PCM WAV supported (ch=%u bits=%u)\n", ch, bits); return false; }+ int n = sz / (bits / 8);+ std::vector<int16_t> pcm(n); fread(pcm.data(), 2, n, f);+ out.resize(n / ch);+ for (int i = 0; i < n / ch; i++) out[i] = pcm[i * ch] / 32768.0f; // ch0+ fclose(f);+ if (sr != 16000) fprintf(stderr, "warning: sample rate %u != 16000\n", sr);+ return true;+ } else fseek(f, sz, SEEK_CUR);+ }+ fclose(f); return false;+}++// ======================= kaldi fbank + LFR =======================+static const int FS=16000, WINLEN=400, SHIFT=160, NFFT=512, NMEL=80, LFR_M=7, LFR_N=6;+static const float PREEMPH=0.97f, LOWF=20.0f, HIGHF=8000.0f;+static inline float mel(float f){ return 1127.0f*logf(1.0f+f/700.0f); }+static void fft(std::vector<float>&re,std::vector<float>&im,int n){+ for(int i=1,j=0;i<n;i++){int b=n>>1;for(;j&b;b>>=1)j^=b;j^=b;if(i<j){std::swap(re[i],re[j]);std::swap(im[i],im[j]);}}+ for(int len=2;len<=n;len<<=1){double a=-2.0*M_PI/len;float wr=cosf(a),wi=sinf(a);+ for(int i=0;i<n;i+=len){float cr=1,ci=0;for(int k=0;k<len/2;k++){+ float ur=re[i+k],ui=im[i+k];float vr=re[i+k+len/2]*cr-im[i+k+len/2]*ci,vi=re[i+k+len/2]*ci+im[i+k+len/2]*cr;+ re[i+k]=ur+vr;im[i+k]=ui+vi;re[i+k+len/2]=ur-vr;im[i+k+len/2]=ui-vi;+ float n2=cr*wr-ci*wi;ci=cr*wi+ci*wr;cr=n2;}}}+}+// returns [T x 560] row-major, sets T+static std::vector<float> compute_fbank(std::vector<float> wav, int & T_out) {+ for (auto & v : wav) v *= 32768.0f;+ std::vector<float> win(WINLEN);+ for (int i=0;i<WINLEN;i++) win[i]=0.54f-0.46f*cosf(2.0f*M_PI*i/(WINLEN-1));+ const int NBIN=NFFT/2+1; float bw=(float)FS/NFFT, ml=mel(LOWF), mh=mel(HIGHF), dm=(mh-ml)/(NMEL+1);+ std::vector<std::vector<float>> fb(NMEL, std::vector<float>(NBIN,0.0f));+ for(int m=0;m<NMEL;m++){float L=ml+m*dm,C=ml+(m+1)*dm,R=ml+(m+2)*dm;+ for(int k=0;k<NBIN;k++){float mf=mel(bw*k); if(mf>L&&mf<R) fb[m][k]=mf<=C?(mf-L)/(C-L):(R-mf)/(R-C);}}+ int N=wav.size(); int T=(N-WINLEN)/SHIFT+1;+ std::vector<std::vector<float>> feat(T, std::vector<float>(NMEL));+ std::vector<float> re(NFFT),im(NFFT),fr(WINLEN);+ const float fl=1.1920929e-07f;+ for(int t=0;t<T;t++){const float*s=wav.data()+t*SHIFT;+ double mn=0;for(int i=0;i<WINLEN;i++)mn+=s[i];mn/=WINLEN;+ for(int i=0;i<WINLEN;i++)fr[i]=s[i]-(float)mn;+ for(int i=WINLEN-1;i>0;i--)fr[i]-=PREEMPH*fr[i-1];fr[0]-=PREEMPH*fr[0];+ for(int i=0;i<NFFT;i++){re[i]=i<WINLEN?fr[i]*win[i]:0.0f;im[i]=0.0f;}+ fft(re,im,NFFT);+ for(int m=0;m<NMEL;m++){float e=0;for(int k=0;k<NBIN;k++)if(fb[m][k]>0)e+=fb[m][k]*(re[k]*re[k]+im[k]*im[k]);+ feat[t][m]=logf(e>fl?e:fl);}}+ // LFR+ const int pad=(LFR_M-1)/2; int T_lfr=(T+LFR_N-1)/LFR_N;+ std::vector<std::vector<float>> pd; pd.reserve(T+pad+LFR_M);+ for(int i=0;i<pad;i++)pd.push_back(feat[0]);+ for(int t=0;t<T;t++)pd.push_back(feat[t]);+ while((int)pd.size()<(T_lfr-1)*LFR_N+LFR_M)pd.push_back(feat[T-1]);+ int D=LFR_M*NMEL; std::vector<float> out((size_t)T_lfr*D);+ for(int i=0;i<T_lfr;i++)for(int j=0;j<LFR_M;j++)+ memcpy(&out[(size_t)i*D+j*NMEL],pd[i*LFR_N+j].data(),NMEL*sizeof(float));+ T_out=T_lfr; return out;+}++// ======================= ggml SAN-M encoder + adaptor =======================+struct cfg { int d_model=512,n_head=4,num_blocks=50,tp_blocks=20,kernel=11,adp_llm=1024,adp_layers=2,adp_head=8; };+struct enc_model { cfg c; ggml_context*ctx_w=nullptr; std::map<std::string,ggml_tensor*> t;+ ggml_tensor* g(const std::string&n){auto it=t.find(n);if(it==t.end()){fprintf(stderr,"missing %s\n",n.c_str());exit(1);}return it->second;} };+static const float LN_EPS=1e-5f;+static bool load_enc(const char*p, enc_model&m){+ gguf_init_params gp={false,&m.ctx_w}; gguf_context*g=gguf_init_from_file(p,gp); if(!g)return false;+ auto rd=[&](const char*k,int d){int i=gguf_find_key(g,k);return i<0?d:(int)gguf_get_val_u32(g,i);};+ m.c.d_model=rd("funasr.enc.output_size",512); m.c.n_head=rd("funasr.enc.attention_heads",4);+ m.c.num_blocks=rd("funasr.enc.num_blocks",50); m.c.tp_blocks=rd("funasr.enc.tp_blocks",20);+ m.c.kernel=rd("funasr.enc.kernel_size",11); m.c.adp_llm=rd("funasr.adp.llm_dim",1024);+ m.c.adp_layers=rd("funasr.adp.n_layer",2); m.c.adp_head=rd("funasr.adp.attention_heads",8);+ int n=gguf_get_n_tensors(g); for(int i=0;i<n;i++){const char*nm=gguf_get_tensor_name(g,i);m.t[nm]=ggml_get_tensor(m.ctx_w,nm);}+ gguf_free(g); return true;+}+static ggml_tensor* lin(ggml_context*c,ggml_tensor*w,ggml_tensor*b,ggml_tensor*x){auto y=ggml_mul_mat(c,w,x);return b?ggml_add(c,y,b):y;}+static ggml_tensor* lnorm(ggml_context*c,ggml_tensor*x,ggml_tensor*g,ggml_tensor*b){return ggml_add(c,ggml_mul(c,ggml_norm(c,x,LN_EPS),g),b);}+static ggml_tensor* sanm_attn(ggml_context*c,enc_model&m,const std::string&p,ggml_tensor*x,int T){+ const int D=m.c.d_model,H=m.c.n_head,dk=D/H,K=m.c.kernel;+ ggml_tensor*qkv=lin(c,m.g(p+"linear_q_k_v.weight"),m.g(p+"linear_q_k_v.bias"),x); size_t nb1=qkv->nb[1];+ ggml_tensor*q=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,0));+ ggml_tensor*k=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)D*sizeof(float)));+ ggml_tensor*v=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)2*D*sizeof(float)));+ const int pad=(K-1)/2; ggml_tensor*fk=m.g(p+"fsmn_block.weight");+ ggml_tensor*vp=ggml_pad_ext(c,v,0,0,pad,pad,0,0,0,0); ggml_tensor*fsmn=v;+ for(int j=0;j<K;j++){auto sl=ggml_view_2d(c,vp,D,T,vp->nb[1],(size_t)j*vp->nb[1]);+ auto wj=ggml_view_1d(c,fk,D,(size_t)j*fk->nb[1]); fsmn=ggml_add(c,fsmn,ggml_mul(c,ggml_cont(c,sl),wj));}+ q=ggml_permute(c,ggml_reshape_3d(c,q,dk,H,T),0,2,1,3); k=ggml_permute(c,ggml_reshape_3d(c,k,dk,H,T),0,2,1,3);+ ggml_tensor*vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,v,dk,H,T),1,2,0,3));+ ggml_tensor*kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk)));+ ggml_tensor*o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,T);+ return ggml_add(c,lin(c,m.g(p+"linear_out.weight"),m.g(p+"linear_out.bias"),o),fsmn);+}+static ggml_tensor* sanm_layer(ggml_context*c,enc_model&m,const std::string&p,ggml_tensor*x,int T,bool res){+ auto r=x; auto h=lnorm(c,x,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));+ auto sa=sanm_attn(c,m,p+"self_attn.",h,T); x=res?ggml_add(c,r,sa):sa; r=x;+ h=lnorm(c,x,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));+ h=lin(c,m.g(p+"feed_forward.w_1.weight"),m.g(p+"feed_forward.w_1.bias"),h); h=ggml_relu(c,h);+ h=lin(c,m.g(p+"feed_forward.w_2.weight"),m.g(p+"feed_forward.w_2.bias"),h); return ggml_add(c,r,h);+}+static ggml_tensor* adp_layer(ggml_context*c,enc_model&m,const std::string&p,ggml_tensor*x,int T){+ const int D=m.c.adp_llm,H=m.c.adp_head,dk=D/H; auto r=x;+ auto h=lnorm(c,x,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));+ auto q=ggml_permute(c,ggml_reshape_3d(c,lin(c,m.g(p+"self_attn.linear_q.weight"),m.g(p+"self_attn.linear_q.bias"),h),dk,H,T),0,2,1,3);+ auto k=ggml_permute(c,ggml_reshape_3d(c,lin(c,m.g(p+"self_attn.linear_k.weight"),m.g(p+"self_attn.linear_k.bias"),h),dk,H,T),0,2,1,3);+ auto vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,lin(c,m.g(p+"self_attn.linear_v.weight"),m.g(p+"self_attn.linear_v.bias"),h),dk,H,T),1,2,0,3));+ auto kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk)));+ auto o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,T);+ x=ggml_add(c,r,lin(c,m.g(p+"self_attn.linear_out.weight"),m.g(p+"self_attn.linear_out.bias"),o)); r=x;+ h=lnorm(c,x,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));+ h=lin(c,m.g(p+"feed_forward.w_1.weight"),m.g(p+"feed_forward.w_1.bias"),h); h=ggml_relu(c,h);+ h=lin(c,m.g(p+"feed_forward.w_2.weight"),m.g(p+"feed_forward.w_2.bias"),h); return ggml_add(c,r,h);+}+static void add_posenc(std::vector<float>&x,int T,int depth){+ double inc=log(10000.0)/(depth/2.0-1.0);+ for(int t=0;t<T;t++){double pos=t+1;for(int i=0;i<depth/2;i++){double its=exp(i*-inc),st=pos*its;+ x[(size_t)t*depth+i]+=(float)sin(st);x[(size_t)t*depth+depth/2+i]+=(float)cos(st);}}+}+// fbank [T x F] -> adaptor out [T x adp_llm] row-major+static std::vector<float> run_encoder(enc_model&m,std::vector<float> fbank,int T,int F,int&Dout){+ float sc=sqrtf((float)m.c.d_model); for(auto&v:fbank)v*=sc; add_posenc(fbank,T,F);+ ggml_backend_t be=ggml_backend_cpu_init();+ ggml_init_params cp={(size_t)1024*1024*1024,nullptr,true}; ggml_context*c=ggml_init(cp);+ ggml_tensor*inp=ggml_new_tensor_2d(c,GGML_TYPE_F32,F,T); ggml_set_input(inp);+ ggml_tensor*x=sanm_layer(c,m,"audio_encoder.encoders0.0.",inp,T,false);+ for(int i=0;i<m.c.num_blocks-1;i++) x=sanm_layer(c,m,"audio_encoder.encoders."+std::to_string(i)+".",x,T,true);+ x=lnorm(c,x,m.g("audio_encoder.after_norm.weight"),m.g("audio_encoder.after_norm.bias"));+ for(int i=0;i<m.c.tp_blocks;i++) x=sanm_layer(c,m,"audio_encoder.tp_encoders."+std::to_string(i)+".",x,T,true);+ x=lnorm(c,x,m.g("audio_encoder.tp_norm.weight"),m.g("audio_encoder.tp_norm.bias"));+ x=lin(c,m.g("audio_adaptor.linear1.weight"),m.g("audio_adaptor.linear1.bias"),x); x=ggml_relu(c,x);+ x=lin(c,m.g("audio_adaptor.linear2.weight"),m.g("audio_adaptor.linear2.bias"),x);+ for(int i=0;i<m.c.adp_layers;i++) x=adp_layer(c,m,"audio_adaptor.blocks."+std::to_string(i)+".",x,T);+ ggml_set_output(x);+ ggml_cgraph*gf=ggml_new_graph_custom(c,32768,false); ggml_build_forward_expand(gf,x);+ ggml_gallocr_t ga=ggml_gallocr_new(ggml_backend_cpu_buffer_type()); ggml_gallocr_alloc_graph(ga,gf);+ ggml_backend_tensor_set(inp,fbank.data(),0,ggml_nbytes(inp));+ ggml_backend_cpu_set_n_threads(be,8); ggml_backend_graph_compute(be,gf);+ Dout=(int)x->ne[0]; std::vector<float> out((size_t)Dout*T); ggml_backend_tensor_get(x,out.data(),0,ggml_nbytes(x));+ ggml_gallocr_free(ga); ggml_free(c); ggml_backend_free(be); return out;+}++// ======================= LLM (llama.cpp) =======================+static int decode_batch(llama_context*ctx,int n,llama_token*tok,float*embd,int n_embd,int&n_past,bool last_logits){+ std::vector<llama_pos> pos(n); std::vector<int32_t> nsid(n,1);+ std::vector<llama_seq_id> s0(1,0); std::vector<llama_seq_id*> sid(n); std::vector<int8_t> lg(n,0);+ for(int i=0;i<n;i++){pos[i]=n_past+i;sid[i]=s0.data();}+ if(last_logits) lg[n-1]=1;+ llama_batch b={n,tok,embd,pos.data(),nsid.data(),sid.data(),lg.data()};+ int r=llama_decode(ctx,b); n_past+=n; return r;+}++int main(int argc,char**argv){+ std::string enc_path,llm_path,wav_path; int npred=512; double chunk_sec=0; float rep=1.0f;+ for(int i=1;i<argc;i++){+ if(!strcmp(argv[i],"--enc")&&i+1<argc)enc_path=argv[++i];+ else if(!strcmp(argv[i],"-m")&&i+1<argc)llm_path=argv[++i];+ else if(!strcmp(argv[i],"-a")&&i+1<argc)wav_path=argv[++i];+ else if(!strcmp(argv[i],"-n")&&i+1<argc)npred=atoi(argv[++i]);+ else if(!strcmp(argv[i],"--chunk")&&i+1<argc)chunk_sec=atof(argv[++i]);+ else if(!strcmp(argv[i],"--rep")&&i+1<argc)rep=atof(argv[++i]);+ else {fprintf(stderr,"usage: %s --enc enc.gguf -m llm.gguf -a audio.wav [-n npred] [--chunk sec]\n",argv[0]);return 1;}+ }+ if(enc_path.empty()||llm_path.empty()||wav_path.empty()){fprintf(stderr,"missing args\n");return 1;}++ std::vector<float> wav;+ if(!read_wav16(wav_path.c_str(),wav)){fprintf(stderr,"failed to read wav\n");return 1;}+ int64_t t0=ggml_time_us();++ enc_model em; if(!load_enc(enc_path.c_str(),em))return 1;+ ggml_backend_load_all();+ llama_model_params mp=llama_model_default_params(); mp.n_gpu_layers=0;+ llama_model*model=llama_model_load_from_file(llm_path.c_str(),mp); if(!model)return 1;+ const llama_vocab*vocab=llama_model_get_vocab(model);+ llama_context_params cp=llama_context_default_params();+ cp.n_ctx=2048; cp.n_batch=2048; cp.n_ubatch=2048;+ llama_context*ctx=llama_init_from_model(model,cp);+ if(!ctx){fprintf(stderr,"failed to create llama context\n");llama_model_free(model);return 1;}+ auto sp=llama_sampler_chain_default_params(); llama_sampler*smpl=llama_sampler_chain_init(sp);+ if(rep!=1.0f) llama_sampler_chain_add(smpl,llama_sampler_init_penalties(256,rep,0.0f,0.0f));+ llama_sampler_chain_add(smpl,llama_sampler_init_greedy());++ const char*prefix="<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n语音转写:";+ const char*suffix="<|im_end|>\n<|im_start|>assistant\n";+ auto tokenize=[&](const char*s){int n=-llama_tokenize(vocab,s,strlen(s),nullptr,0,false,true);+ std::vector<llama_token> v(n); llama_tokenize(vocab,s,strlen(s),v.data(),n,false,true); return v;};+ auto pre=tokenize(prefix); auto suf=tokenize(suffix);++ // split into chunks (chunk_sec<=0 -> whole file)+ int chunk_n = chunk_sec > 0 ? std::max(1, (int)(chunk_sec*16000)) : (int)wav.size();+ std::string full;+ for (size_t off = 0; off < wav.size(); off += chunk_n) {+ int len = std::min((size_t)chunk_n, wav.size()-off);+ if (len < WINLEN) break; // too short for one frame+ std::vector<float> seg(wav.begin()+off, wav.begin()+off+len);+ int T=0; auto fbank=compute_fbank(seg,T);+ int D=0; auto adp=run_encoder(em,fbank,T,560,D);+ int ol=1+(T-3+2)/2; ol=1+(ol-3+2)/2; int n_aud=(ol-1)/2+1;++ llama_memory_clear(llama_get_memory(ctx), true); // fresh context per chunk+ int n_past=0;+ decode_batch(ctx,pre.size(),pre.data(),nullptr,0,n_past,false);+ decode_batch(ctx,n_aud,nullptr,adp.data(),D,n_past,false);+ decode_batch(ctx,suf.size(),suf.data(),nullptr,0,n_past,true);+ llama_token tk=llama_sampler_sample(smpl,ctx,-1);+ for(int i=0;i<npred;i++){+ if(llama_vocab_is_eog(vocab,tk))break;+ char buf[256]; int k=llama_token_to_piece(vocab,tk,buf,sizeof(buf),0,true);+ if(k>0) full.append(buf,k);+ decode_batch(ctx,1,&tk,nullptr,0,n_past,true);+ tk=llama_sampler_sample(smpl,ctx,-1);+ }+ }+ printf("%s\n", full.c_str());+ int64_t t2=ggml_time_us();+ fprintf(stderr,"[done] %.2fs ; chunk=%.0fs\n",(t2-t0)/1e6, chunk_sec);+ llama_sampler_free(smpl); llama_free(ctx); llama_model_free(model);+ if(em.ctx_w) ggml_free(em.ctx_w);+ return 0;+}diff --git a/runtime/llama.cpp/fun-asr-nano/funasr-embd/CMakeLists.txt b/runtime/llama.cpp/fun-asr-nano/funasr-embd/CMakeLists.txtnew file mode 100644index 000000000..2d682d00d--- /dev/null+++ b/runtime/llama.cpp/fun-asr-nano/funasr-embd/CMakeLists.txt@@ -0,0 +1,5 @@+set(TARGET llama-funasr-embd)+add_executable(${TARGET} funasr-embd.cpp)+install(TARGETS ${TARGET} RUNTIME)+target_link_libraries(${TARGET} PRIVATE llama ${CMAKE_THREAD_LIBS_INIT})+target_compile_features(${TARGET} PRIVATE cxx_std_17)diff --git a/runtime/llama.cpp/fun-asr-nano/funasr-embd/funasr-embd.cpp b/runtime/llama.cpp/fun-asr-nano/funasr-embd/funasr-embd.cppnew file mode 100644index 000000000..851b94a6c--- /dev/null+++ b/runtime/llama.cpp/fun-asr-nano/funasr-embd/funasr-embd.cpp@@ -0,0 +1,141 @@+// funasr-embd: decode Fun-ASR-Nano audio embeddings through a Qwen3 GGUF.+//+// Reads an inputs_embeds matrix (produced by the FunASR audio encoder+adaptor,+// concatenated with the text prompt embeddings) and feeds it directly to the+// LLM via llama_decode's embedding input path -- the same mechanism llava/mtmd+// use to inject vision embeddings. This bridges FunASR's audio frontend to the+// llama.cpp / GGUF ecosystem.+//+// embeds.bin format: int32 n_tokens, int32 n_embd, then n_tokens*n_embd float32+// (row-major). n_embd must equal the model's input embedding dim.++#include "llama.h"++#include <cstdint>+#include <cstdio>+#include <cstring>+#include <string>+#include <vector>++static void print_usage(char ** argv) {+ printf("\nusage: %s -m model.gguf -e embeds.bin [-n n_predict] [-ngl n_gpu_layers]\n\n", argv[0]);+}++// read embeds.bin -> (n_tokens, n_embd, data)+static bool read_embeds(const std::string & path, int & n_tokens, int & n_embd, std::vector<float> & data) {+ FILE * f = fopen(path.c_str(), "rb");+ if (!f) { fprintf(stderr, "error: cannot open %s\n", path.c_str()); return false; }+ int32_t hdr[2];+ if (fread(hdr, sizeof(int32_t), 2, f) != 2) { fclose(f); return false; }+ n_tokens = hdr[0];+ n_embd = hdr[1];+ if (n_tokens <= 0 || n_embd <= 0) { fclose(f); return false; }+ data.resize((size_t) n_tokens * n_embd);+ size_t got = fread(data.data(), sizeof(float), data.size(), f);+ fclose(f);+ if (got != data.size()) {+ fprintf(stderr, "error: short read (%zu/%zu floats)\n", got, data.size());+ return false;+ }+ return true;+}++int main(int argc, char ** argv) {+ std::string model_path, embeds_path;+ int n_predict = 512;+ int ngl = 0; // CPU by default; the whole point is CPU/edge++ for (int i = 1; i < argc; i++) {+ if (!strcmp(argv[i], "-m") && i + 1 < argc) model_path = argv[++i];+ else if (!strcmp(argv[i], "-e") && i + 1 < argc) embeds_path = argv[++i];+ else if (!strcmp(argv[i], "-n") && i + 1 < argc) n_predict = std::stoi(argv[++i]);+ else if (!strcmp(argv[i], "-ngl") && i + 1 < argc) ngl = std::stoi(argv[++i]);+ else { print_usage(argv); return 1; }+ }+ if (model_path.empty() || embeds_path.empty()) { print_usage(argv); return 1; }++ int n_tokens = 0, n_embd_in = 0;+ std::vector<float> embd;+ if (!read_embeds(embeds_path, n_tokens, n_embd_in, embd)) return 1;+ fprintf(stderr, "loaded embeds: n_tokens=%d n_embd=%d\n", n_tokens, n_embd_in);++ ggml_backend_load_all();++ llama_model_params mparams = llama_model_default_params();+ mparams.n_gpu_layers = ngl;+ llama_model * model = llama_model_load_from_file(model_path.c_str(), mparams);+ if (!model) { fprintf(stderr, "error: unable to load model\n"); return 1; }++ const llama_vocab * vocab = llama_model_get_vocab(model);+ const int n_embd_model = llama_model_n_embd_inp(model);+ if (n_embd_in != n_embd_model) {+ fprintf(stderr, "error: embd dim %d != model input embd dim %d\n", n_embd_in, n_embd_model);+ llama_model_free(model);+ return 1;+ }++ llama_context_params cparams = llama_context_default_params();+ cparams.n_ctx = n_tokens + n_predict + 8;+ cparams.n_batch = n_tokens + 8; // process the whole embd prompt in one ubatch+ cparams.n_ubatch = n_tokens + 8;+ cparams.no_perf = false;+ llama_context * ctx = llama_init_from_model(model, cparams);+ if (!ctx) { fprintf(stderr, "error: failed to create context\n"); llama_model_free(model); return 1; }++ auto sparams = llama_sampler_chain_default_params();+ llama_sampler * smpl = llama_sampler_chain_init(sparams);+ llama_sampler_chain_add(smpl, llama_sampler_init_greedy());++ // --- decode the embedding prompt (causal, single sequence, positions 0..n-1) ---+ std::vector<llama_pos> pos(n_tokens);+ std::vector<int32_t> n_seq_id(n_tokens, 1);+ std::vector<llama_seq_id> seq_id_0(1, 0);+ std::vector<llama_seq_id *> seq_id(n_tokens);+ std::vector<int8_t> logits(n_tokens, 0);+ for (int i = 0; i < n_tokens; i++) { pos[i] = i; seq_id[i] = seq_id_0.data(); }+ logits[n_tokens - 1] = 1; // only need logits for the last position++ llama_batch batch = {+ /*n_tokens =*/ n_tokens,+ /*token =*/ nullptr,+ /*embd =*/ embd.data(),+ /*pos =*/ pos.data(),+ /*n_seq_id =*/ n_seq_id.data(),+ /*seq_id =*/ seq_id.data(),+ /*logits =*/ logits.data(),+ };++ const int64_t t_start = ggml_time_us();+ if (llama_decode(ctx, batch) != 0) {+ fprintf(stderr, "error: llama_decode failed on embd prompt\n");+ return 1;+ }++ // --- generation loop ---+ std::string out;+ int n_decode = 0;+ llama_token tok = llama_sampler_sample(smpl, ctx, -1);+ for (int n_pos = n_tokens; n_pos < n_tokens + n_predict; ) {+ if (llama_vocab_is_eog(vocab, tok)) break;+ char buf[256];+ int n = llama_token_to_piece(vocab, tok, buf, sizeof(buf), 0, true);+ if (n > 0) { out.append(buf, n); }+ printf("%.*s", n > 0 ? n : 0, buf);+ fflush(stdout);++ llama_batch tb = llama_batch_get_one(&tok, 1);+ if (llama_decode(ctx, tb) != 0) { fprintf(stderr, "error: decode failed\n"); return 1; }+ n_pos += 1;+ n_decode += 1;+ tok = llama_sampler_sample(smpl, ctx, -1);+ }+ printf("\n");+ const int64_t t_end = ggml_time_us();+ fprintf(stderr, "\n[funasr-embd] generated %d tokens in %.2f s (%.1f tok/s)\n",+ n_decode, (t_end - t_start) / 1e6, n_decode / ((t_end - t_start) / 1e6));++ llama_sampler_free(smpl);+ llama_free(ctx);+ llama_model_free(model);+ return 0;+}diff --git a/runtime/llama.cpp/fun-asr-nano/funasr-encoder/CMakeLists.txt b/runtime/llama.cpp/fun-asr-nano/funasr-encoder/CMakeLists.txtnew file mode 100644index 000000000..d30f2ef05--- /dev/null+++ b/runtime/llama.cpp/fun-asr-nano/funasr-encoder/CMakeLists.txt@@ -0,0 +1,5 @@+set(TARGET llama-funasr-encoder)+add_executable(${TARGET} funasr-encoder.cpp)+install(TARGETS ${TARGET} RUNTIME)+target_link_libraries(${TARGET} PRIVATE ggml ${CMAKE_THREAD_LIBS_INIT})+target_compile_features(${TARGET} PRIVATE cxx_std_17)diff --git a/runtime/llama.cpp/fun-asr-nano/funasr-encoder/funasr-encoder.cpp b/runtime/llama.cpp/fun-asr-nano/funasr-encoder/funasr-encoder.cppnew file mode 100644index 000000000..d176cbc98--- /dev/null+++ b/runtime/llama.cpp/fun-asr-nano/funasr-encoder/funasr-encoder.cpp@@ -0,0 +1,275 @@+// funasr-encoder: ggml C++ forward pass for the Fun-ASR-Nano audio encoder+// (SenseVoice SAN-M, 50+20 layers) + Transformer adaptor.+//+// Input : fbank.bin (T x 560 f32, the encoder input features)+// Output: out.bin (T' x 1024 f32, audio embeddings for the LLM)+// Weights: funasr-encoder.gguf (exported by export_encoder_gguf.py)+//+// Validated layer-by-layer against PyTorch golden dumps. fbank is currently+// produced in Python; porting the fbank frontend to C++ is the remaining piece.++#include "ggml.h"+#include "ggml-cpu.h"+#include "ggml-alloc.h"+#include "ggml-backend.h"+#include "gguf.h"++#include <cmath>+#include <cstdint>+#include <cstdio>+#include <cstring>+#include <map>+#include <string>+#include <vector>++struct cfg {+ int input_size = 560, d_model = 512, n_head = 4, ffn = 2048;+ int num_blocks = 50, tp_blocks = 20, kernel = 11;+ int adp_llm = 1024, adp_ffn = 2048, adp_layers = 2, adp_head = 8;+};++static const float LN_EPS = 1e-5f;++struct funasr_model {+ cfg c;+ struct ggml_context * ctx_w = nullptr; // weights (CPU malloc, data set)+ std::map<std::string, struct ggml_tensor *> t;+ struct ggml_tensor * get(const std::string & n) {+ auto it = t.find(n);+ if (it == t.end()) { fprintf(stderr, "missing tensor: %s\n", n.c_str()); exit(1); }+ return it->second;+ }+};++static bool load_model(const char * path, funasr_model & m) {+ struct gguf_init_params p = { /*no_alloc=*/false, /*ctx=*/&m.ctx_w };+ struct gguf_context * gguf = gguf_init_from_file(path, p);+ if (!gguf) { fprintf(stderr, "failed to load gguf %s\n", path); return false; }+ auto rd = [&](const char * k, int def) {+ int i = gguf_find_key(gguf, k); return i < 0 ? def : (int) gguf_get_val_u32(gguf, i);+ };+ m.c.input_size = rd("funasr.enc.input_size", 560);+ m.c.d_model = rd("funasr.enc.output_size", 512);+ m.c.n_head = rd("funasr.enc.attention_heads", 4);+ m.c.ffn = rd("funasr.enc.linear_units", 2048);+ m.c.num_blocks = rd("funasr.enc.num_blocks", 50);+ m.c.tp_blocks = rd("funasr.enc.tp_blocks", 20);+ m.c.kernel = rd("funasr.enc.kernel_size", 11);+ m.c.adp_llm = rd("funasr.adp.llm_dim", 1024);+ m.c.adp_ffn = rd("funasr.adp.ffn_dim", 2048);+ m.c.adp_layers = rd("funasr.adp.n_layer", 2);+ m.c.adp_head = rd("funasr.adp.attention_heads", 8);+ int n = gguf_get_n_tensors(gguf);+ for (int i = 0; i < n; i++) {+ const char * name = gguf_get_tensor_name(gguf, i);+ m.t[name] = ggml_get_tensor(m.ctx_w, name);+ }+ fprintf(stderr, "loaded %d tensors; cfg: d_model=%d heads=%d blocks=%d tp=%d kernel=%d adp_llm=%d\n",+ n, m.c.d_model, m.c.n_head, m.c.num_blocks, m.c.tp_blocks, m.c.kernel, m.c.adp_llm);+ gguf_free(gguf);+ return true;+}++// helpers ---------------------------------------------------------------+static struct ggml_tensor * linear(ggml_context * ctx, ggml_tensor * w, ggml_tensor * b, ggml_tensor * x) {+ struct ggml_tensor * y = ggml_mul_mat(ctx, w, x); // [out, T]+ if (b) y = ggml_add(ctx, y, b);+ return y;+}+static struct ggml_tensor * layernorm(ggml_context * ctx, ggml_tensor * x, ggml_tensor * g, ggml_tensor * b) {+ x = ggml_norm(ctx, x, LN_EPS);+ x = ggml_mul(ctx, x, g);+ x = ggml_add(ctx, x, b);+ return x;+}++// SAN-M self-attention + FSMN. x:[D_in,T] -> [d_model,T]+static struct ggml_tensor * sanm_attn(ggml_context * ctx, funasr_model & m, const std::string & pfx,+ ggml_tensor * x, int T) {+ const int D = m.c.d_model, H = m.c.n_head, dk = D / H, K = m.c.kernel;+ struct ggml_tensor * qkv = linear(ctx, m.get(pfx + "linear_q_k_v.weight"),+ m.get(pfx + "linear_q_k_v.bias"), x); // [3D, T]+ size_t nb1 = qkv->nb[1];+ struct ggml_tensor * q = ggml_cont(ctx, ggml_view_2d(ctx, qkv, D, T, nb1, 0));+ struct ggml_tensor * k = ggml_cont(ctx, ggml_view_2d(ctx, qkv, D, T, nb1, (size_t) D * sizeof(float)));+ struct ggml_tensor * v = ggml_cont(ctx, ggml_view_2d(ctx, qkv, D, T, nb1, (size_t) 2 * D * sizeof(float)));++ // FSMN: depthwise conv1d along time (per-channel kernel K, "same" padding),+ // plus residual v. Implemented as an exact f32 shift-accumulate to avoid the+ // F16-only ggml_conv_1d_dw path. fsmn kernel stored as [D, K] (ne0=D, ne1=K).+ const int pad = (K - 1) / 2;+ struct ggml_tensor * fk = m.get(pfx + "fsmn_block.weight"); // [D, K]+ struct ggml_tensor * vpad = ggml_pad_ext(ctx, v, 0, 0, pad, pad, 0, 0, 0, 0); // [D, T+2*pad]+ struct ggml_tensor * fsmn = v; // residual+ for (int j = 0; j < K; j++) {+ struct ggml_tensor * sl = ggml_view_2d(ctx, vpad, D, T, vpad->nb[1], (size_t) j * vpad->nb[1]);+ struct ggml_tensor * wj = ggml_view_1d(ctx, fk, D, (size_t) j * fk->nb[1]);+ fsmn = ggml_add(ctx, fsmn, ggml_mul(ctx, ggml_cont(ctx, sl), wj));+ }++ // multi-head attention+ q = ggml_reshape_3d(ctx, q, dk, H, T);+ k = ggml_reshape_3d(ctx, k, dk, H, T);+ struct ggml_tensor * vh = ggml_reshape_3d(ctx, v, dk, H, T);+ q = ggml_permute(ctx, q, 0, 2, 1, 3); // [dk, T, H]+ k = ggml_permute(ctx, k, 0, 2, 1, 3); // [dk, T, H]+ vh = ggml_cont(ctx, ggml_permute(ctx, vh, 1, 2, 0, 3)); // [T, dk, H]+ struct ggml_tensor * kq = ggml_mul_mat(ctx, k, q); // [T, T, H]+ kq = ggml_scale(ctx, kq, 1.0f / sqrtf((float) dk));+ kq = ggml_soft_max(ctx, kq);+ struct ggml_tensor * kqv = ggml_mul_mat(ctx, vh, kq); // [dk, T, H]+ kqv = ggml_permute(ctx, kqv, 0, 2, 1, 3); // [dk, H, T]+ kqv = ggml_cont_2d(ctx, kqv, D, T); // [D, T]+ struct ggml_tensor * att = linear(ctx, m.get(pfx + "linear_out.weight"),+ m.get(pfx + "linear_out.bias"), kqv);+ return ggml_add(ctx, att, fsmn);+}++// one SAN-M encoder layer. in_size may differ from d_model (first layer)+static struct ggml_tensor * sanm_layer(ggml_context * ctx, funasr_model & m, const std::string & pfx,+ ggml_tensor * x, int T, bool residual_attn) {+ struct ggml_tensor * res = x;+ struct ggml_tensor * h = layernorm(ctx, x, m.get(pfx + "norm1.weight"), m.get(pfx + "norm1.bias"));+ struct ggml_tensor * sa = sanm_attn(ctx, m, pfx + "self_attn.", h, T);+ x = residual_attn ? ggml_add(ctx, res, sa) : sa;+ res = x;+ h = layernorm(ctx, x, m.get(pfx + "norm2.weight"), m.get(pfx + "norm2.bias"));+ h = linear(ctx, m.get(pfx + "feed_forward.w_1.weight"), m.get(pfx + "feed_forward.w_1.bias"), h);+ h = ggml_relu(ctx, h);+ h = linear(ctx, m.get(pfx + "feed_forward.w_2.weight"), m.get(pfx + "feed_forward.w_2.bias"), h);+ return ggml_add(ctx, res, h);+}++// standard transformer layer (adaptor). d=adp_llm+static struct ggml_tensor * adp_layer(ggml_context * ctx, funasr_model & m, const std::string & pfx,+ ggml_tensor * x, int T) {+ const int D = m.c.adp_llm, H = m.c.adp_head, dk = D / H;+ struct ggml_tensor * res = x;+ struct ggml_tensor * h = layernorm(ctx, x, m.get(pfx + "norm1.weight"), m.get(pfx + "norm1.bias"));+ struct ggml_tensor * q = linear(ctx, m.get(pfx + "self_attn.linear_q.weight"), m.get(pfx + "self_attn.linear_q.bias"), h);+ struct ggml_tensor * k = linear(ctx, m.get(pfx + "self_attn.linear_k.weight"), m.get(pfx + "self_attn.linear_k.bias"), h);+ struct ggml_tensor * v = linear(ctx, m.get(pfx + "self_attn.linear_v.weight"), m.get(pfx + "self_attn.linear_v.bias"), h);+ q = ggml_permute(ctx, ggml_reshape_3d(ctx, q, dk, H, T), 0, 2, 1, 3);+ k = ggml_permute(ctx, ggml_reshape_3d(ctx, k, dk, H, T), 0, 2, 1, 3);+ struct ggml_tensor * vh = ggml_cont(ctx, ggml_permute(ctx, ggml_reshape_3d(ctx, v, dk, H, T), 1, 2, 0, 3));+ struct ggml_tensor * kq = ggml_soft_max(ctx, ggml_scale(ctx, ggml_mul_mat(ctx, k, q), 1.0f / sqrtf((float) dk)));+ struct ggml_tensor * kqv = ggml_cont_2d(ctx, ggml_permute(ctx, ggml_mul_mat(ctx, vh, kq), 0, 2, 1, 3), D, T);+ struct ggml_tensor * att = linear(ctx, m.get(pfx + "self_attn.linear_out.weight"), m.get(pfx + "self_attn.linear_out.bias"), kqv);+ x = ggml_add(ctx, res, att);+ res = x;+ h = layernorm(ctx, x, m.get(pfx + "norm2.weight"), m.get(pfx + "norm2.bias"));+ h = linear(ctx, m.get(pfx + "feed_forward.w_1.weight"), m.get(pfx + "feed_forward.w_1.bias"), h);+ h = ggml_relu(ctx, h);+ h = linear(ctx, m.get(pfx + "feed_forward.w_2.weight"), m.get(pfx + "feed_forward.w_2.bias"), h);+ return ggml_add(ctx, res, h);+}++// sinusoidal position encoding, depth = input feature dim, positions 1..T+static void add_posenc(std::vector<float> & x, int T, int depth) {+ double inc = log(10000.0) / (depth / 2.0 - 1.0);+ for (int t = 0; t < T; t++) {+ double pos = t + 1; // positions start at 1+ for (int i = 0; i < depth / 2; i++) {+ double its = exp(i * -inc);+ double st = pos * its;+ x[(size_t) t * depth + i] += (float) sin(st);+ x[(size_t) t * depth + depth / 2 + i] += (float) cos(st);+ }+ }+}++int main(int argc, char ** argv) {+ std::string gguf_path, fbank_path, out_path = "out.bin";+ int limit = -1; // -L: run only first N (encoders0+encoders) layers, dump running x+ bool run_adaptor = true;+ for (int i = 1; i < argc; i++) {+ if (!strcmp(argv[i], "-m") && i+1 < argc) gguf_path = argv[++i];+ else if (!strcmp(argv[i], "-f") && i+1 < argc) fbank_path = argv[++i];+ else if (!strcmp(argv[i], "-o") && i+1 < argc) out_path = argv[++i];+ else if (!strcmp(argv[i], "-L") && i+1 < argc) { limit = atoi(argv[++i]); run_adaptor = false; }+ else { fprintf(stderr, "usage: %s -m enc.gguf -f fbank.bin [-o out.bin] [-L nlayers]\n", argv[0]); return 1; }+ }++ funasr_model m;+ if (!load_model(gguf_path.c_str(), m)) return 1;++ // read fbank.bin (T x F)+ FILE * f = fopen(fbank_path.c_str(), "rb");+ if (!f) { fprintf(stderr, "cannot open %s\n", fbank_path.c_str()); return 1; }+ int32_t T, F; if (fread(&T, 4, 1, f) != 1 || fread(&F, 4, 1, f) != 1) { fclose(f); return 1; }+ std::vector<float> fbank((size_t) T * F);+ if (fread(fbank.data(), sizeof(float), fbank.size(), f) != fbank.size()) { fclose(f); return 1; }+ fclose(f);+ fprintf(stderr, "fbank: T=%d F=%d\n", T, F);++ // pre-scale (*sqrt(d_model)) and add position encoding on the host+ float scale = sqrtf((float) m.c.d_model);+ for (auto & v : fbank) v *= scale;+ add_posenc(fbank, T, F);++ // backend + compute context+ ggml_backend_t backend = ggml_backend_cpu_init();+ size_t ctx_size = (size_t) 1024*1024*1024; // graph metadata+ struct ggml_init_params cp = { ctx_size, nullptr, true };+ struct ggml_context * ctx = ggml_init(cp);++ struct ggml_tensor * inp = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, F, T);+ ggml_set_name(inp, "inp");+ ggml_set_input(inp);++ struct ggml_tensor * x = inp;+ int done = 0;+ bool stop = false;+ // encoders0 (1 layer, in_size=input_size != d_model -> no attn residual)+ x = sanm_layer(ctx, m, "audio_encoder.encoders0.0.", x, T, /*residual_attn=*/false);+ done++;+ if (limit >= 0 && done >= limit) stop = true;+ // encoders (num_blocks-1 layers)+ for (int i = 0; i < m.c.num_blocks - 1 && !stop; i++) {+ x = sanm_layer(ctx, m, "audio_encoder.encoders." + std::to_string(i) + ".", x, T, true);+ done++;+ if (limit >= 0 && done >= limit) stop = true;+ }+ if (!stop) {+ x = layernorm(ctx, x, m.get("audio_encoder.after_norm.weight"), m.get("audio_encoder.after_norm.bias"));+ for (int i = 0; i < m.c.tp_blocks; i++)+ x = sanm_layer(ctx, m, "audio_encoder.tp_encoders." + std::to_string(i) + ".", x, T, true);+ x = layernorm(ctx, x, m.get("audio_encoder.tp_norm.weight"), m.get("audio_encoder.tp_norm.bias"));+ // adaptor: downsample_rate=1 -> linear1(relu)linear2 then blocks+ if (run_adaptor) {+ x = linear(ctx, m.get("audio_adaptor.linear1.weight"), m.get("audio_adaptor.linear1.bias"), x);+ x = ggml_relu(ctx, x);+ x = linear(ctx, m.get("audio_adaptor.linear2.weight"), m.get("audio_adaptor.linear2.bias"), x);+ for (int i = 0; i < m.c.adp_layers; i++)+ x = adp_layer(ctx, m, "audio_adaptor.blocks." + std::to_string(i) + ".", x, T);+ }+ }+ ggml_set_output(x);++ struct ggml_cgraph * gf = ggml_new_graph_custom(ctx, 32768, false);+ ggml_build_forward_expand(gf, x);++ ggml_gallocr_t galloc = ggml_gallocr_new(ggml_backend_cpu_buffer_type());+ ggml_gallocr_alloc_graph(galloc, gf);+ ggml_backend_tensor_set(inp, fbank.data(), 0, ggml_nbytes(inp));++ ggml_backend_cpu_set_n_threads(backend, 8);+ int64_t t0 = ggml_time_us();+ if (ggml_backend_graph_compute(backend, gf) != GGML_STATUS_SUCCESS) {+ fprintf(stderr, "compute failed\n"); return 1;+ }+ int64_t t1 = ggml_time_us();++ int D = (int) x->ne[0];+ std::vector<float> out((size_t) D * T);+ ggml_backend_tensor_get(x, out.data(), 0, ggml_nbytes(x));+ FILE * fo = fopen(out_path.c_str(), "wb");+ if (!fo) { fprintf(stderr, "failed to open output file %s\n", out_path.c_str()); return 1; }+ fwrite(&T, 4, 1, fo); fwrite(&D, 4, 1, fo); fwrite(out.data(), sizeof(float), out.size(), fo);+ fclose(fo);+ fprintf(stderr, "done: wrote %s [%d x %d] in %.2f s (layers run=%d, adaptor=%d)\n",+ out_path.c_str(), T, D, (t1 - t0)/1e6, done, run_adaptor && !stop);+ ggml_gallocr_free(galloc); ggml_free(ctx); ggml_backend_free(backend);+ if (m.ctx_w) ggml_free(m.ctx_w);+ return 0;+}diff --git a/runtime/llama.cpp/paraformer/README.md b/runtime/llama.cpp/paraformer/README.mdnew file mode 100644index 000000000..21b40c74b--- /dev/null+++ b/runtime/llama.cpp/paraformer/README.md@@ -0,0 +1,103 @@+# Paraformer on llama.cpp / GGUF++Run **Paraformer** (the non-autoregressive ASR model) on the+[llama.cpp](https://github.com/ggml-org/llama.cpp) / ggml stack — **CPU, edge,+a single binary, no Python at runtime**. Like whisper.cpp, but for Paraformer.++## Why this exists++Paraformer normally runs on PyTorch / ONNX. This runtime ports it to **ggml ++GGUF** so it runs CPU-only, offline, embedded in a C/C++ app, with quantized+weights — laptops, phones, edge boxes, no GPU and no Python. (For high-QPS GPU+serving, the PyTorch path is still the way.)++## Architecture++Paraformer is **non-autoregressive**: it predicts all output tokens in one pass.++```+ audio.wav (16k mono)+ │ kaldi 80-mel fbank + LFR + CMVN (C++)+ ▼+ features [T, 560]+ │ SANM encoder (50 layers: LN + fused QKV + FSMN + FFN) (ggml)+ ▼+ encoder_out [T, 512]+ │ CIF predictor: conv1d → sigmoid → α; integrate-and-fire (host)+ ▼+ acoustic embeds [N_tok, 512] (N_tok = number of output tokens)+ │ SANM decoder (16 layers: FFN → FSMN self-attn → cross-attn to encoder) (ggml)+ ▼+ logits [N_tok, vocab] → argmax → token ids → text+```++CIF (Continuous Integrate-and-Fire) walks the encoder output accumulating a+predicted "weight" α per frame; each time the running sum crosses 1.0 it emits one+acoustic token. This both decides the token count and produces the acoustic+embeddings the decoder consumes. The SANM encoder/FSMN/attention primitives are+shared with the Fun-ASR-Nano and SenseVoice runtimes.++## Quickstart++**1. Build:**+```bash+git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp+cp -r /path/to/runtime/llama.cpp/funasr-paraformer examples/+echo 'add_subdirectory(funasr-paraformer)' >> examples/CMakeLists.txt+cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF+cmake --build build -j --target llama-funasr-paraformer+```++**2. Convert weights** (needs the checkpoint, e.g. `funasr/paraformer-zh`):+```bash+python runtime/llama.cpp/export_paraformer_gguf.py \+ --model_pt <model>/model.pt --mvn <model>/am.mvn \+ --out paraformer.gguf # f32, ~863 MB+python runtime/llama.cpp/export_paraformer_gguf.py --wtype f16 \+ --model_pt <model>/model.pt --mvn <model>/am.mvn \+ --out paraformer-f16.gguf # half size+```++**3. Transcribe:**+```bash+build/bin/llama-funasr-paraformer -m paraformer.gguf -a audio.wav > ids.txt+python runtime/llama.cpp/detok.py <model>/tokens.json ids.txt+```+Expected output:+```+我想问我在滨海新区有房我一直没有照顾孩子...你觉得这是正常的想法吗+[paraformer] T=742 N_tok=105 enc 1.24s dec 0.48s+```++## Accuracy & validation++- Decoded text is **character-for-character identical** to the FunASR `AutoModel`+ output on a benchmark clip; the CIF token count matches exactly (105/105).+- Stage-by-stage vs PyTorch: encoder cosine 0.997, acoustic embeds cosine 0.993+ (the small residual is the reference frontend's random `dither=1.0`; the C+++ front end is deterministic, dither=0).+- Encode ≈ 1.2 s + decode ≈ 0.5 s on CPU for a 44 s clip.++## Tips & gotchas++- **CMVN IS applied** (unlike SenseVoice): `(fbank + shift) * scale`, per-dim (560),+ from `am.mvn`. Parsing note: `am.mvn` has three `[...]` blocks — `[Splice idx]`,+ `[AddShift=shift]`, `[Rescale=scale]`; use the two 560-length vectors. Getting+ this wrong makes the CIF predictor emit ~4× too few tokens.+- **CIF/predictor runs on host** (it's a sequential integrate-and-fire loop);+ the encoder and decoder run in ggml.+- The decoder self-attention is **FSMN-only** (no QK attention); cross-attention+ attends to the encoder output. The decoder FFN has an internal LayerNorm and the+ second linear has no bias.+- WAV input assumes 16 kHz mono PCM16.++## Files+```+funasr-paraformer/ ggml runtime: WAV → token ids+export_paraformer_gguf.py export encoder + predictor + decoder + CMVN to GGUF+detok.py token-id → text (tokens.json)+```++## Roadmap+- Built-in detok; timestamps (CIF peaks give alignment); arbitrary WAV / resampling;+ encoder/decoder quantization.diff --git a/runtime/llama.cpp/paraformer/detok_paraformer.py b/runtime/llama.cpp/paraformer/detok_paraformer.pynew file mode 100644index 000000000..0e56aaab9--- /dev/null+++ b/runtime/llama.cpp/paraformer/detok_paraformer.py@@ -0,0 +1,8 @@+#!/usr/bin/env python3+"""Detokenize Paraformer token ids -> text via tokens.json.+Usage: python detok_paraformer.py <tokens.json> <ids.txt>"""+import sys, json+toks = json.load(open(sys.argv[1]))+ids = [int(x) for x in open(sys.argv[2]).read().split() if int(x) not in (1, 2)] # drop sos/eos+out = "".join(toks[i] for i in ids if 0 <= i < len(toks))+print(out.replace("@@", "").replace("▁", " ").strip())diff --git a/runtime/llama.cpp/paraformer/export_paraformer_gguf.py b/runtime/llama.cpp/paraformer/export_paraformer_gguf.pynew file mode 100644index 000000000..476f554d6--- /dev/null+++ b/runtime/llama.cpp/paraformer/export_paraformer_gguf.py@@ -0,0 +1,55 @@+#!/usr/bin/env python3+"""Export Paraformer (SANM encoder + CIF predictor + SANM decoder) to GGUF.+Encoder reuses the shared SAN-M forward. Predictor (CIF) runs on host in C++.+"""+import argparse, os, re+import numpy as np, torch, gguf+++def parse_mvn(path):+ # am.mvn (kaldi nnet) has 3 bracketed blocks: [Splice idx], [AddShift=shift],+ # [Rescale=scale]. Take the two 560-length vectors (shift then scale).+ # apply: out = (x + shift) * scale+ blocks = [np.array([float(x) for x in b.split()], np.float32)+ for b in re.findall(r"\[([^\]]*)\]", open(path).read())]+ vecs = [b for b in blocks if b.size > 1]+ return vecs[0], vecs[1]+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--model_pt", required=True); ap.add_argument("--mvn", required=True)+ ap.add_argument("--out", required=True); ap.add_argument("--wtype", default="f32", choices=["f32","f16"])+ a = ap.parse_args()+ sd = torch.load(a.model_pt, map_location="cpu"); sd = sd.get("state_dict", sd)+ w = gguf.GGUFWriter(a.out, "paraformer")+ w.add_uint32("pf.enc.output_size", 512); w.add_uint32("pf.enc.attention_heads", 4)+ w.add_uint32("pf.enc.num_blocks", 50); w.add_uint32("pf.enc.kernel_size", 11)+ w.add_uint32("pf.dec.num_blocks", 16); w.add_uint32("pf.dec.att_layer_num", 16)+ w.add_uint32("pf.dec.decoders3", 1); w.add_uint32("pf.dec.attention_heads", 4)+ w.add_uint32("pf.dec.kernel_size", 11); w.add_uint32("pf.vocab_size", 8404)+ w.add_float32("pf.predictor.tail_threshold", 0.45)+ w.add_float32("pf.predictor.threshold", 1.0)+ shift, scale = parse_mvn(a.mvn)+ w.add_tensor("cmvn.shift", shift); w.add_tensor("cmvn.scale", scale)+ n = 0+ for k, v in sd.items():+ if not (k.startswith("encoder.") or k.startswith("decoder.") or k.startswith("predictor.")):+ continue+ if k == "decoder.embed.0.weight": # token embedding, unused at NAR inference+ continue+ arr = v.detach().to(torch.float32).contiguous().numpy()+ if k.endswith("fsmn_block.weight") and arr.ndim == 3: # (D,1,K)->(K,D)+ arr = np.ascontiguousarray(arr[:, 0, :].T)+ elif args_f16(a) and arr.ndim == 2 and "norm" not in k and "cif_output" not in k:+ arr = arr.astype(np.float16)+ w.add_tensor(k, arr); n += 1+ print(f"writing {n} tensors (+cmvn) to {a.out}")+ w.write_header_to_file(); w.write_kv_data_to_file(); w.write_tensors_to_file(); w.close()+ print(f"done: {a.out} ({os.path.getsize(a.out)/1e6:.1f} MB)")+++def args_f16(a): return a.wtype == "f16"++if __name__ == "__main__":+ main()diff --git a/runtime/llama.cpp/paraformer/funasr-paraformer/CMakeLists.txt b/runtime/llama.cpp/paraformer/funasr-paraformer/CMakeLists.txtnew file mode 100644index 000000000..7fb44e1f6--- /dev/null+++ b/runtime/llama.cpp/paraformer/funasr-paraformer/CMakeLists.txt@@ -0,0 +1,5 @@+set(TARGET llama-funasr-paraformer)+add_executable(${TARGET} funasr-paraformer.cpp)+install(TARGETS ${TARGET} RUNTIME)+target_link_libraries(${TARGET} PRIVATE ggml ${CMAKE_THREAD_LIBS_INIT})+target_compile_features(${TARGET} PRIVATE cxx_std_17)diff --git a/runtime/llama.cpp/paraformer/funasr-paraformer/funasr-paraformer.cpp b/runtime/llama.cpp/paraformer/funasr-paraformer/funasr-paraformer.cppnew file mode 100644index 000000000..0b8876e49--- /dev/null+++ b/runtime/llama.cpp/paraformer/funasr-paraformer/funasr-paraformer.cpp@@ -0,0 +1,213 @@+// funasr-paraformer: Paraformer (non-autoregressive ASR) on ggml.+// WAV → kaldi fbank → CMVN → SANM encoder (ggml) → CIF predictor (host) →+// SANM decoder w/ cross-attn (ggml) → argmax → token ids (stdout).+// Encoder/FSMN/attention primitives are shared with the Fun-ASR-Nano runtime.++#include "ggml.h"+#include "ggml-cpu.h"+#include "ggml-alloc.h"+#include "ggml-backend.h"+#include "gguf.h"++#include <cmath>+#include <cstdint>+#include <cstdio>+#include <cstring>+#include <map>+#include <string>+#include <vector>++static const float LN_EPS = 1e-5f;++// ===== WAV + kaldi fbank (shared) =====+static bool read_wav16(const char*p,std::vector<float>&o){FILE*f=fopen(p,"rb");if(!f)return false;char r[4];fread(r,1,4,f);+ if(strncmp(r,"RIFF",4)){fclose(f);return false;}fseek(f,4,SEEK_CUR);char wv[4];fread(wv,1,4,f);if(strncmp(wv,"WAVE",4)){fclose(f);return false;}+ uint16_t ch=1,bits=16;uint32_t sr=16000;while(!feof(f)){char id[4];if(fread(id,1,4,f)!=4)break;uint32_t sz;if(fread(&sz,4,1,f)!=1)break;+ if(!strncmp(id,"fmt ",4)){uint16_t fm;fread(&fm,2,1,f);fread(&ch,2,1,f);fread(&sr,4,1,f);fseek(f,6,SEEK_CUR);fread(&bits,2,1,f);if(sz>16)fseek(f,sz-16,SEEK_CUR);}+ else if(!strncmp(id,"data",4)){if(ch<1||bits!=16){fclose(f);fprintf(stderr,"only 16-bit PCM WAV (ch=%u bits=%u)\n",ch,bits);return false;}int n=sz/(bits/8);std::vector<int16_t>pc(n);fread(pc.data(),2,n,f);o.resize(n/ch);for(int i=0;i<n/ch;i++)o[i]=pc[i*ch]/32768.0f;fclose(f);return true;}+ else fseek(f,sz,SEEK_CUR);}fclose(f);return false;}+static const int FS=16000,WINLEN=400,SHIFT=160,NFFT=512,NMEL=80,LFR_M=7,LFR_N=6;+static const float PREEMPH=0.97f,LOWF=20.0f,HIGHF=8000.0f;+static inline float melf(float f){return 1127.0f*logf(1.0f+f/700.0f);}+static void fftc(std::vector<float>&re,std::vector<float>&im,int n){for(int i=1,j=0;i<n;i++){int b=n>>1;for(;j&b;b>>=1)j^=b;j^=b;if(i<j){std::swap(re[i],re[j]);std::swap(im[i],im[j]);}}+ for(int len=2;len<=n;len<<=1){double a=-2.0*M_PI/len;float wr=cosf(a),wi=sinf(a);for(int i=0;i<n;i+=len){float cr=1,ci=0;for(int k=0;k<len/2;k++){float ur=re[i+k],ui=im[i+k];+ float vr=re[i+k+len/2]*cr-im[i+k+len/2]*ci,vi=re[i+k+len/2]*ci+im[i+k+len/2]*cr;re[i+k]=ur+vr;im[i+k]=ui+vi;re[i+k+len/2]=ur-vr;im[i+k+len/2]=ui-vi;float nc=cr*wr-ci*wi;ci=cr*wi+ci*wr;cr=nc;}}}}+static std::vector<float> compute_fbank(std::vector<float> wav,int&T_out){+ for(auto&v:wav)v*=32768.0f;std::vector<float>win(WINLEN);for(int i=0;i<WINLEN;i++)win[i]=0.54f-0.46f*cosf(2.0f*M_PI*i/(WINLEN-1));+ const int NB=NFFT/2+1;float bw=(float)FS/NFFT,ml=melf(LOWF),mh=melf(HIGHF),dm=(mh-ml)/(NMEL+1);+ std::vector<std::vector<float>>fb(NMEL,std::vector<float>(NB,0.0f));+ for(int m=0;m<NMEL;m++){float L=ml+m*dm,C=ml+(m+1)*dm,R=ml+(m+2)*dm;for(int k=0;k<NB;k++){float mf=melf(bw*k);if(mf>L&&mf<R)fb[m][k]=mf<=C?(mf-L)/(C-L):(R-mf)/(R-C);}}+ int N=wav.size(),T=(N-WINLEN)/SHIFT+1;std::vector<std::vector<float>>feat(T,std::vector<float>(NMEL));+ std::vector<float>re(NFFT),im(NFFT),fr(WINLEN);const float fl=1.1920929e-07f;+ for(int t=0;t<T;t++){const float*s=wav.data()+t*SHIFT;double mn=0;for(int i=0;i<WINLEN;i++)mn+=s[i];mn/=WINLEN;+ for(int i=0;i<WINLEN;i++)fr[i]=s[i]-(float)mn;for(int i=WINLEN-1;i>0;i--)fr[i]-=PREEMPH*fr[i-1];fr[0]-=PREEMPH*fr[0];+ for(int i=0;i<NFFT;i++){re[i]=i<WINLEN?fr[i]*win[i]:0.0f;im[i]=0.0f;}fftc(re,im,NFFT);+ for(int m=0;m<NMEL;m++){float e=0;for(int k=0;k<NB;k++)if(fb[m][k]>0)e+=fb[m][k]*(re[k]*re[k]+im[k]*im[k]);feat[t][m]=logf(e>fl?e:fl);}}+ const int pad=(LFR_M-1)/2;int Tl=(T+LFR_N-1)/LFR_N;std::vector<std::vector<float>>pd;pd.reserve(T+pad+LFR_M);+ for(int i=0;i<pad;i++)pd.push_back(feat[0]);for(int t=0;t<T;t++)pd.push_back(feat[t]);while((int)pd.size()<(Tl-1)*LFR_N+LFR_M)pd.push_back(feat[T-1]);+ int D=LFR_M*NMEL;std::vector<float>out((size_t)Tl*D);for(int i=0;i<Tl;i++)for(int j=0;j<LFR_M;j++)memcpy(&out[(size_t)i*D+j*NMEL],pd[i*LFR_N+j].data(),NMEL*sizeof(float));+ T_out=Tl;return out;}++// ===== model =====+struct cfg{int d_model=512,enc_head=4,enc_blocks=50,enc_kernel=11,dec_blocks=16,dec_att=16,dec3=1,dec_head=4,dec_kernel=11,vocab=8404;float tail=0.45f,thresh=1.0f;};+struct model{cfg c;ggml_context*ctx_w=nullptr;std::map<std::string,ggml_tensor*>t;+ ggml_tensor*g(const std::string&n){auto it=t.find(n);if(it==t.end()){fprintf(stderr,"missing %s\n",n.c_str());exit(1);}return it->second;}+ bool has(const std::string&n){return t.count(n);} };++static ggml_tensor* lin(ggml_context*c,ggml_tensor*w,ggml_tensor*b,ggml_tensor*x){auto y=ggml_mul_mat(c,w,x);return b?ggml_add(c,y,b):y;}+static ggml_tensor* lnorm(ggml_context*c,ggml_tensor*x,ggml_tensor*g,ggml_tensor*b){return ggml_add(c,ggml_mul(c,ggml_norm(c,x,LN_EPS),g),b);}+// FSMN depthwise (kernel stored [K,D]): out = sum_j w[:,j]*pad(v)[:, t+j] (+ residual v)+static ggml_tensor* fsmn(ggml_context*c,ggml_tensor*v,ggml_tensor*fk,int D,int T,int K){+ const int pad=(K-1)/2;ggml_tensor*vp=ggml_pad_ext(c,v,0,0,pad,pad,0,0,0,0);ggml_tensor*acc=v;+ for(int j=0;j<K;j++){auto sl=ggml_view_2d(c,vp,D,T,vp->nb[1],(size_t)j*vp->nb[1]);auto wj=ggml_view_1d(c,fk,D,(size_t)j*fk->nb[1]);+ acc=ggml_add(c,acc,ggml_mul(c,ggml_cont(c,sl),wj));}return acc;}+// SAN-M encoder self-attn (fused qkv + fsmn)+static ggml_tensor* enc_attn(ggml_context*c,model&m,const std::string&p,ggml_tensor*x,int T){+ const int D=m.c.d_model,H=m.c.enc_head,dk=D/H,K=m.c.enc_kernel;+ ggml_tensor*qkv=lin(c,m.g(p+"linear_q_k_v.weight"),m.g(p+"linear_q_k_v.bias"),x);size_t nb1=qkv->nb[1];+ ggml_tensor*q=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,0));+ ggml_tensor*k=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)D*sizeof(float)));+ ggml_tensor*v=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)2*D*sizeof(float)));+ ggml_tensor*fm=fsmn(c,v,m.g(p+"fsmn_block.weight"),D,T,K);+ q=ggml_permute(c,ggml_reshape_3d(c,q,dk,H,T),0,2,1,3);k=ggml_permute(c,ggml_reshape_3d(c,k,dk,H,T),0,2,1,3);+ ggml_tensor*vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,v,dk,H,T),1,2,0,3));+ ggml_tensor*kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk)));+ ggml_tensor*o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,T);+ return ggml_add(c,lin(c,m.g(p+"linear_out.weight"),m.g(p+"linear_out.bias"),o),fm);}+static ggml_tensor* enc_layer(ggml_context*c,model&m,const std::string&p,ggml_tensor*x,int T,bool res){+ auto r=x;auto h=lnorm(c,x,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));auto sa=enc_attn(c,m,p+"self_attn.",h,T);+ x=res?ggml_add(c,r,sa):sa;r=x;h=lnorm(c,x,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));+ h=lin(c,m.g(p+"feed_forward.w_1.weight"),m.g(p+"feed_forward.w_1.bias"),h);h=ggml_relu(c,h);+ h=lin(c,m.g(p+"feed_forward.w_2.weight"),m.g(p+"feed_forward.w_2.bias"),h);return ggml_add(c,r,h);}+// decoder FFN-SANM: w_2(LayerNorm(relu(w_1(x)))) (w_2 has no bias)+static ggml_tensor* dec_ffn(ggml_context*c,model&m,const std::string&p,ggml_tensor*x){+ auto h=lin(c,m.g(p+"w_1.weight"),m.g(p+"w_1.bias"),x);h=ggml_relu(c,h);+ h=lnorm(c,h,m.g(p+"norm.weight"),m.g(p+"norm.bias"));return ggml_mul_mat(c,m.g(p+"w_2.weight"),h);}+// cross attn: q=linear_q(tgt)[D,N], kv=linear_k_v(mem)[2D,T]+static ggml_tensor* cross_attn(ggml_context*c,model&m,const std::string&p,ggml_tensor*tgt,ggml_tensor*mem,int N,int T){+ const int D=m.c.d_model,H=m.c.dec_head,dk=D/H;+ ggml_tensor*q=lin(c,m.g(p+"linear_q.weight"),m.g(p+"linear_q.bias"),tgt); // [D,N]+ ggml_tensor*kv=lin(c,m.g(p+"linear_k_v.weight"),m.g(p+"linear_k_v.bias"),mem);// [2D,T]+ size_t nb1=kv->nb[1];+ ggml_tensor*k=ggml_cont(c,ggml_view_2d(c,kv,D,T,nb1,0));+ ggml_tensor*v=ggml_cont(c,ggml_view_2d(c,kv,D,T,nb1,(size_t)D*sizeof(float)));+ q=ggml_permute(c,ggml_reshape_3d(c,q,dk,H,N),0,2,1,3); // [dk,N,H]+ k=ggml_permute(c,ggml_reshape_3d(c,k,dk,H,T),0,2,1,3); // [dk,T,H]+ ggml_tensor*vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,v,dk,H,T),1,2,0,3)); // [T,dk,H]+ ggml_tensor*kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk))); // [T,N,H]+ ggml_tensor*o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,N);+ return lin(c,m.g(p+"linear_out.weight"),m.g(p+"linear_out.bias"),o);}+static ggml_tensor* dec_layer(ggml_context*c,model&m,const std::string&p,ggml_tensor*tgt,ggml_tensor*mem,int N,int T){+ const int D=m.c.d_model,K=m.c.dec_kernel;+ auto residual=tgt;auto h=lnorm(c,tgt,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));+ h=dec_ffn(c,m,p+"feed_forward.",h); // FFN first+ auto y=lnorm(c,h,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));+ auto sa=fsmn(c,y,m.g(p+"self_attn.fsmn_block.weight"),D,N,K); // FSMN self-attn (+residual y inside)+ auto x=ggml_add(c,residual,sa);+ residual=x;auto z=lnorm(c,x,m.g(p+"norm3.weight"),m.g(p+"norm3.bias"));+ auto ca=cross_attn(c,m,p+"src_attn.",z,mem,N,T);+ return ggml_add(c,residual,ca);}+static ggml_tensor* dec3_layer(ggml_context*c,model&m,const std::string&p,ggml_tensor*tgt){+ auto h=lnorm(c,tgt,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));return dec_ffn(c,m,p+"feed_forward.",h);}+static void add_posenc(std::vector<float>&x,int T,int depth){double inc=log(10000.0)/(depth/2.0-1.0);+ for(int t=0;t<T;t++){double pos=t+1;for(int i=0;i<depth/2;i++){double its=exp(i*-inc),st=pos*its;x[(size_t)t*depth+i]+=(float)sin(st);x[(size_t)t*depth+depth/2+i]+=(float)cos(st);}}}++// run a ggml graph on CPU, return output [ne0 x ne1] row-major (ne1 rows)+static std::vector<float> run_graph(ggml_context*c,ggml_tensor*out,ggml_tensor*in1,const float*d1,ggml_tensor*in2,const float*d2){+ ggml_backend_t be=ggml_backend_cpu_init();ggml_cgraph*gf=ggml_new_graph_custom(c,32768,false);ggml_build_forward_expand(gf,out);+ ggml_gallocr_t ga=ggml_gallocr_new(ggml_backend_cpu_buffer_type());ggml_gallocr_alloc_graph(ga,gf);+ ggml_backend_tensor_set(in1,d1,0,ggml_nbytes(in1));if(in2)ggml_backend_tensor_set(in2,d2,0,ggml_nbytes(in2));+ ggml_backend_cpu_set_n_threads(be,8);ggml_backend_graph_compute(be,gf);+ int D=out->ne[0],N=out->ne[1];std::vector<float>r((size_t)D*N);ggml_backend_tensor_get(out,r.data(),0,ggml_nbytes(out));+ ggml_gallocr_free(ga);ggml_backend_free(be);return r;}++int main(int argc,char**argv){+ std::string gguf_path,wav_path;+ for(int i=1;i<argc;i++){if(!strcmp(argv[i],"-m")&&i+1<argc)gguf_path=argv[++i];else if(!strcmp(argv[i],"-a")&&i+1<argc)wav_path=argv[++i];+ else{fprintf(stderr,"usage: %s -m paraformer.gguf -a audio.wav\n",argv[0]);return 1;}}+ if(gguf_path.empty()||wav_path.empty()){fprintf(stderr,"missing args\n");return 1;}+ model m;gguf_init_params gp={false,&m.ctx_w};gguf_context*gg=gguf_init_from_file(gguf_path.c_str(),gp);if(!gg){fprintf(stderr,"gguf load failed\n");return 1;}+ auto rdi=[&](const char*k,int d){int i=gguf_find_key(gg,k);return i<0?d:(int)gguf_get_val_u32(gg,i);};+ auto rdf=[&](const char*k,float d){int i=gguf_find_key(gg,k);return i<0?d:gguf_get_val_f32(gg,i);};+ m.c.enc_blocks=rdi("pf.enc.num_blocks",50);m.c.dec_blocks=rdi("pf.dec.num_blocks",16);m.c.dec_att=rdi("pf.dec.att_layer_num",16);+ m.c.dec3=rdi("pf.dec.decoders3",1);m.c.vocab=rdi("pf.vocab_size",8404);m.c.tail=rdf("pf.predictor.tail_threshold",0.45f);m.c.thresh=rdf("pf.predictor.threshold",1.0f);+ for(int i=0;i<gguf_get_n_tensors(gg);i++){const char*nm=gguf_get_tensor_name(gg,i);m.t[nm]=ggml_get_tensor(m.ctx_w,nm);}gguf_free(gg);+ const int D=m.c.d_model,F=560,V=m.c.vocab;++ // fbank+ std::vector<float>wav;if(!read_wav16(wav_path.c_str(),wav)){fprintf(stderr,"read wav failed\n");return 1;}+ int64_t t0=ggml_time_us();int T=0;auto fb=compute_fbank(wav,T);+ // CMVN (Paraformer applies it): (x+shift)*scale+ float*shift=(float*)m.g("cmvn.shift")->data,*scale=(float*)m.g("cmvn.scale")->data;+ for(int t=0;t<T;t++)for(int d=0;d<F;d++)fb[(size_t)t*F+d]=(fb[(size_t)t*F+d]+shift[d])*scale[d];+ // encoder pre-scale + posenc+ {float sc=sqrtf((float)D);for(auto&v:fb)v*=sc;}add_posenc(fb,T,F);++ // ===== encoder graph =====+ std::vector<float>enc;+ {ggml_init_params cp={(size_t)1024*1024*1024,nullptr,true};ggml_context*c=ggml_init(cp);+ ggml_tensor*x=ggml_new_tensor_2d(c,GGML_TYPE_F32,F,T);ggml_set_input(x);+ ggml_tensor*h=enc_layer(c,m,"encoder.encoders0.0.",x,T,false);+ for(int i=0;i<m.c.enc_blocks-1;i++)h=enc_layer(c,m,"encoder.encoders."+std::to_string(i)+".",h,T,true);+ h=lnorm(c,h,m.g("encoder.after_norm.weight"),m.g("encoder.after_norm.bias"));ggml_set_output(h);+ enc=run_graph(c,h,x,fb.data(),nullptr,nullptr);ggml_free(c);} // enc: [D, T] -> stored row-major as T rows? ggml out ne0=D,ne1=T+ // run_graph returns row-major [N=ne1 rows, D=ne0]; here rows=T, cols=D+ int64_t t1=ggml_time_us();+ // enc[t*D + d]++ // ===== CIF predictor (host) =====+ float*cw=(float*)m.g("predictor.cif_conv1d.weight")->data; // [512,512,3] = [o][i][j]+ float*cb=(float*)m.g("predictor.cif_conv1d.bias")->data; // [512]+ float*ow=(float*)m.g("predictor.cif_output.weight")->data; // [1,512]+ float ob=((float*)m.g("predictor.cif_output.bias")->data)[0];+ // conv1d (k=3,pad1) + residual + relu, then cif_output -> sigmoid -> alpha+ std::vector<float> alphas(T);+ std::vector<float> outp((size_t)T*D);+ for(int t=0;t<T;t++){+ for(int o=0;o<D;o++){+ float acc=cb[o];+ for(int j=0;j<3;j++){int tt=t+j-1; if(tt<0||tt>=T)continue; const float*ev=&enc[(size_t)tt*D];+ const float*wo=&cw[(size_t)o*D*3]; for(int i=0;i<D;i++) acc+=wo[i*3+j]*ev[i];}+ outp[(size_t)t*D+o]=acc+enc[(size_t)t*D+o]; // + context residual+ }+ float a=ob; for(int o=0;o<D;o++){float r=outp[(size_t)t*D+o]; if(r<0)r=0; a+=ow[o]*r;} // relu then linear+ float s=1.0f/(1.0f+expf(-a)); alphas[t]=s>0?s:0; // sigmoid, smooth=1 noise=0 -> relu(s)=s+ }+ // tail: append zero hidden frame + alpha=tail_threshold+ std::vector<float> hid=enc; hid.resize((size_t)(T+1)*D,0.0f);+ std::vector<float> al=alphas; al.push_back(m.c.tail);+ int L=T+1;+ // integrate-and-fire+ std::vector<float> acoustic; acoustic.reserve(64*D);+ float integrate=0; std::vector<float> frame(D,0.0f);+ for(int t=0;t<L;t++){+ float alpha=al[t]; float dc=1.0f-integrate; integrate+=alpha;+ bool fire=integrate>=m.c.thresh; float cur=fire?dc:alpha; float rem=alpha-cur;+ for(int d=0;d<D;d++) frame[d]+=cur*hid[(size_t)t*D+d];+ if(fire){ acoustic.insert(acoustic.end(),frame.begin(),frame.end()); integrate-=1.0f;+ for(int d=0;d<D;d++) frame[d]=rem*hid[(size_t)t*D+d]; }+ }+ int N=acoustic.size()/D;+ if(N<1){printf("\n");fprintf(stderr,"[paraformer] no tokens\n");return 0;}++ // ===== decoder graph =====+ std::vector<float> logits;+ {ggml_init_params cp={(size_t)2048*1024*1024,nullptr,true};ggml_context*c=ggml_init(cp);+ ggml_tensor*tgt=ggml_new_tensor_2d(c,GGML_TYPE_F32,D,N);ggml_set_input(tgt);+ ggml_tensor*mem=ggml_new_tensor_2d(c,GGML_TYPE_F32,D,T);ggml_set_input(mem);+ ggml_tensor*x=tgt;+ for(int i=0;i<m.c.dec_att;i++)x=dec_layer(c,m,"decoder.decoders."+std::to_string(i)+".",x,mem,N,T);+ for(int i=0;i<m.c.dec3;i++)x=dec3_layer(c,m,"decoder.decoders3."+std::to_string(i)+".",x);+ x=lnorm(c,x,m.g("decoder.after_norm.weight"),m.g("decoder.after_norm.bias"));+ x=lin(c,m.g("decoder.output_layer.weight"),m.g("decoder.output_layer.bias"),x); // [V,N]+ ggml_set_output(x);+ logits=run_graph(c,x,tgt,acoustic.data(),mem,enc.data());ggml_free(c);}+ int64_t t2=ggml_time_us();++ // argmax per slot -> ids+ for(int n=0;n<N;n++){const float*col=&logits[(size_t)n*V];int am=0;float best=col[0];for(int v=1;v<V;v++)if(col[v]>best){best=col[v];am=v;}printf("%d ",am);}+ printf("\n");+ fprintf(stderr,"[paraformer] T=%d N_tok=%d enc %.2fs dec %.2fs\n",T,N,(t1-t0)/1e6,(t2-t1)/1e6);+ if(m.ctx_w) ggml_free(m.ctx_w);+ return 0;+}diff --git a/runtime/llama.cpp/sensevoice/README.md b/runtime/llama.cpp/sensevoice/README.mdnew file mode 100644index 000000000..907a7323f--- /dev/null+++ b/runtime/llama.cpp/sensevoice/README.md@@ -0,0 +1,102 @@+# SenseVoiceSmall on llama.cpp / GGUF++Run **SenseVoiceSmall** on the [llama.cpp](https://github.com/ggml-org/llama.cpp)+/ ggml stack — **CPU, edge, a single binary, no Python at runtime**. Like+[whisper.cpp](https://github.com/ggml-org/whisper.cpp), but for SenseVoice.++## Why this exists++SenseVoiceSmall normally runs on PyTorch / ONNX / libtorch. This runtime ports it+to **ggml + GGUF** so it can run CPU-only, offline, embedded in a C/C++ app, with+quantized weights. Use it on laptops / phones / edge boxes where there is no GPU+and no Python. (For high-QPS GPU serving, the PyTorch/vLLM path is still the way.)++## Architecture++SenseVoiceSmall = **SAN-M encoder (70 layers) + CTC head** — no LLM, no autoregression.+The whole pipeline runs in C++:++```+ audio.wav (16k mono)+ │ kaldi 80-mel fbank + LFR (C++)+ ▼+ features [T, 560]+ │ prepend 4 query tokens [lang, event, emotion, itn]+ ▼+ [4 + T, 560]+ │ SAN-M encoder (ggml) ── sensevoice-small.gguf+ ▼+ encoder out [4+T, 512]+ │ CTC head (Linear 512→25055) → greedy CTC (argmax, dedup, drop blank)+ ▼+ token ids+ │ SentencePiece detok (detok.py)+ ▼+ <|zh|><|NEUTRAL|><|Speech|><|woitn|> transcription...+```++The SAN-M encoder is the same architecture as Fun-ASR-Nano's, so the ggml forward+is shared between the two runtimes.++## Quickstart++**1. Build:**+```bash+git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp+cp -r /path/to/runtime/llama.cpp/funasr-sensevoice examples/+echo 'add_subdirectory(funasr-sensevoice)' >> examples/CMakeLists.txt+cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF+cmake --build build -j --target llama-funasr-sensevoice+```++**2. Convert weights** (needs the checkpoint, e.g. `FunAudioLLM/SenseVoiceSmall`):+```bash+python runtime/llama.cpp/export_sensevoice_gguf.py \+ --model_pt <model>/model.pt --mvn <model>/am.mvn \+ --out sensevoice-small.gguf # f32, ~936 MB+python runtime/llama.cpp/export_sensevoice_gguf.py --wtype f16 \+ --model_pt <model>/model.pt --mvn <model>/am.mvn \+ --out sensevoice-small-f16.gguf # half size+```++**3. Transcribe:**+```bash+build/bin/llama-funasr-sensevoice -m sensevoice-small.gguf -a audio.wav > ids.txt+python runtime/llama.cpp/detok.py <model>/chn_jpn_yue_eng_ko_spectok.bpe.model ids.txt+```+Expected output:+```+<|zh|><|NEUTRAL|><|Speech|><|woitn|>我想问我在滨海新区有房我一直没有照顾孩子...你觉得这是正常的想法吗+[sensevoice] N=746 (q4+T742) encode 1.32s+```+The leading `<|...|>` tags are the predicted language / emotion / event / ITN.++## Accuracy & validation++- **CTC token ids (C++) vs PyTorch:** **identical** (108/108 on a benchmark clip).+- **Detokenized text:** matches the FunASR `AutoModel` output **exactly**.+- Encoder validated against PyTorch (shared with Fun-ASR-Nano runtime): cosine 1.0.+- Encode time ≈ **1.3 s** on CPU for a 44 s clip.++## Tips & gotchas++- **No CMVN at inference.** SenseVoice `inference()` feeds the **raw** log-mel fbank+ to the encoder; it does **not** apply `am.mvn`. Applying CMVN makes the model+ predict `<|nospeech|>`. (The export script reads `am.mvn` for completeness but the+ runtime does not use it.)+- **Query tokens (4)** are prepended from `embed.weight`, default indices+ `[language=auto(0), event=1, emotion=2, textnorm=woitn(15)]`. Change them for a+ fixed language or to enable ITN (`withitn=14`).+- **WAV input** assumes 16 kHz mono PCM16.+- LayerNorm eps = 1e-5; FSMN = exact f32 shift-accumulate; fbank matches torchaudio.++## Files+```+funasr-sensevoice/ ggml runtime: WAV → CTC token ids+export_sensevoice_gguf.py export encoder + CTC head + query embeddings to GGUF+detok.py SentencePiece id → text (bpe model ships with the checkpoint)+```++## Roadmap+- Built-in SentencePiece detok (drop the Python step); arbitrary WAV formats;+ encoder Q8 quantization; timestamps.diff --git a/runtime/llama.cpp/sensevoice/detok.py b/runtime/llama.cpp/sensevoice/detok.pynew file mode 100644index 000000000..48b11c953--- /dev/null+++ b/runtime/llama.cpp/sensevoice/detok.py@@ -0,0 +1,10 @@+#!/usr/bin/env python3+"""Detokenize SenseVoice CTC token ids -> text via the SentencePiece bpe model.+Usage: python detok.py <bpe.model> <ids.txt> (ids.txt = space-separated ints)+"""+import sys+import sentencepiece as spm++sp = spm.SentencePieceProcessor(model_file=sys.argv[1])+ids = [int(x) for x in open(sys.argv[2]).read().split()]+print(sp.decode(ids))diff --git a/runtime/llama.cpp/sensevoice/export_sensevoice_gguf.py b/runtime/llama.cpp/sensevoice/export_sensevoice_gguf.pynew file mode 100644index 000000000..997cca5f9--- /dev/null+++ b/runtime/llama.cpp/sensevoice/export_sensevoice_gguf.py@@ -0,0 +1,62 @@+#!/usr/bin/env python3+"""Export SenseVoiceSmall (encoder + CTC head + query embeddings + CMVN) to GGUF+for the ggml C++ runtime. The encoder is the same SAN-M architecture as+Fun-ASR-Nano, so the C++ forward is shared.+"""+import argparse, os, re+import numpy as np, torch, gguf+++def parse_mvn(path):+ """am.mvn (kaldi nnet): two `<LearnRateCoef> 0 [ ... ]` blocks -> shift, scale.+ apply: out = (in + shift) * scale, per-dim (560)."""+ txt = open(path).read()+ blocks = re.findall(r"\[([^\]]*)\]", txt)+ shift = np.array([float(x) for x in blocks[0].split()], dtype=np.float32)+ scale = np.array([float(x) for x in blocks[1].split()], dtype=np.float32)+ return shift, scale+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("--model_pt", required=True)+ ap.add_argument("--mvn", required=True)+ ap.add_argument("--out", required=True)+ ap.add_argument("--wtype", default="f32", choices=["f32", "f16"])+ args = ap.parse_args()++ sd = torch.load(args.model_pt, map_location="cpu"); sd = sd.get("state_dict", sd)+ w = gguf.GGUFWriter(args.out, "sensevoice-small")+ w.add_uint32("sv.input_size", 560)+ w.add_uint32("sv.output_size", 512)+ w.add_uint32("sv.attention_heads", 4)+ w.add_uint32("sv.num_blocks", 50)+ w.add_uint32("sv.tp_blocks", 20)+ w.add_uint32("sv.kernel_size", 11)+ w.add_uint32("sv.vocab_size", 25055)+ w.add_uint32("sv.blank_id", 0)+ # query token embed indices used at inference: [lid(auto=0), 1, 2, textnorm(woitn=15)]+ w.add_array("sv.query_tokens", [0, 1, 2, 15])++ shift, scale = parse_mvn(args.mvn)+ w.add_tensor("cmvn.shift", shift) # (560,)+ w.add_tensor("cmvn.scale", scale) # (560,)++ n = 0+ for k, v in sd.items():+ if not (k.startswith("encoder.") or k.startswith("ctc.") or k == "embed.weight"):+ continue+ arr = v.detach().to(torch.float32).contiguous().numpy()+ if k.endswith("fsmn_block.weight"): # (D,1,K) -> (K,D)+ arr = np.ascontiguousarray(arr[:, 0, :].T)+ elif args.wtype == "f16" and arr.ndim == 2 and "norm" not in k:+ arr = arr.astype(np.float16)+ w.add_tensor(k, arr)+ n += 1+ print(f"writing {n} tensors (+cmvn) to {args.out}")+ w.write_header_to_file(); w.write_kv_data_to_file(); w.write_tensors_to_file(); w.close()+ print(f"done: {args.out} ({os.path.getsize(args.out)/1e6:.1f} MB)")+++if __name__ == "__main__":+ main()diff --git a/runtime/llama.cpp/sensevoice/funasr-sensevoice/CMakeLists.txt b/runtime/llama.cpp/sensevoice/funasr-sensevoice/CMakeLists.txtnew file mode 100644index 000000000..41ab1e443--- /dev/null+++ b/runtime/llama.cpp/sensevoice/funasr-sensevoice/CMakeLists.txt@@ -0,0 +1,5 @@+set(TARGET llama-funasr-sensevoice)+add_executable(${TARGET} funasr-sensevoice.cpp)+install(TARGETS ${TARGET} RUNTIME)+target_link_libraries(${TARGET} PRIVATE ggml ${CMAKE_THREAD_LIBS_INIT})+target_compile_features(${TARGET} PRIVATE cxx_std_17)diff --git a/runtime/llama.cpp/sensevoice/funasr-sensevoice/funasr-sensevoice.cpp b/runtime/llama.cpp/sensevoice/funasr-sensevoice/funasr-sensevoice.cppnew file mode 100644index 000000000..2f9f20046--- /dev/null+++ b/runtime/llama.cpp/sensevoice/funasr-sensevoice/funasr-sensevoice.cpp@@ -0,0 +1,179 @@+// funasr-sensevoice: SenseVoiceSmall (SAN-M encoder + CTC) on ggml.+// fbank.bin (T x 560) -> CMVN -> prepend 4 query tokens -> SAN-M encoder ->+// CTC head -> greedy CTC decode -> token ids (stdout).+// The encoder is the same SAN-M arch as Fun-ASR-Nano (shared forward).+// Detokenize the printed ids with the SentencePiece bpe model (Python side for now).++#include "ggml.h"+#include "ggml-cpu.h"+#include "ggml-alloc.h"+#include "ggml-backend.h"+#include "gguf.h"++#include <cmath>+#include <cstdint>+#include <cstdio>+#include <cstring>+#include <map>+#include <string>+#include <vector>++static const float LN_EPS = 1e-5f;++// ---- WAV (16k mono PCM16) + kaldi fbank (shared with funasr-cli) ----+static bool read_wav16(const char*path,std::vector<float>&out){+ FILE*f=fopen(path,"rb"); if(!f)return false; char r[4]; fread(r,1,4,f);+ if(strncmp(r,"RIFF",4)){fclose(f);return false;} fseek(f,4,SEEK_CUR); char wv[4]; fread(wv,1,4,f);+ if(strncmp(wv,"WAVE",4)){fclose(f);return false;} uint16_t ch=1,bits=16; uint32_t sr=16000;+ while(!feof(f)){char id[4]; if(fread(id,1,4,f)!=4)break; uint32_t sz; if(fread(&sz,4,1,f)!=1)break;+ if(!strncmp(id,"fmt ",4)){uint16_t fmt; fread(&fmt,2,1,f); fread(&ch,2,1,f); fread(&sr,4,1,f); fseek(f,6,SEEK_CUR); fread(&bits,2,1,f); if(sz>16)fseek(f,sz-16,SEEK_CUR);}+ else if(!strncmp(id,"data",4)){if(ch<1||bits!=16){fclose(f);fprintf(stderr,"only 16-bit PCM WAV (ch=%u bits=%u)\n",ch,bits);return false;} int n=sz/(bits/8); std::vector<int16_t> pcm(n); fread(pcm.data(),2,n,f); out.resize(n/ch);+ for(int i=0;i<n/ch;i++)out[i]=pcm[i*ch]/32768.0f; fclose(f); if(sr!=16000)fprintf(stderr,"warn: sr %u\n",sr); return true;}+ else fseek(f,sz,SEEK_CUR);}+ fclose(f); return false;+}+static const int FS=16000,WINLEN=400,SHIFT=160,NFFT=512,NMEL=80,LFR_M=7,LFR_N=6;+static const float PREEMPH=0.97f,LOWF=20.0f,HIGHF=8000.0f;+static inline float melf(float f){return 1127.0f*logf(1.0f+f/700.0f);}+static void fftc(std::vector<float>&re,std::vector<float>&im,int n){+ for(int i=1,j=0;i<n;i++){int b=n>>1;for(;j&b;b>>=1)j^=b;j^=b;if(i<j){std::swap(re[i],re[j]);std::swap(im[i],im[j]);}}+ for(int len=2;len<=n;len<<=1){double a=-2.0*M_PI/len;float wr=cosf(a),wi=sinf(a);+ for(int i=0;i<n;i+=len){float cr=1,ci=0;for(int k=0;k<len/2;k++){float ur=re[i+k],ui=im[i+k];+ float vr=re[i+k+len/2]*cr-im[i+k+len/2]*ci,vi=re[i+k+len/2]*ci+im[i+k+len/2]*cr;+ re[i+k]=ur+vr;im[i+k]=ui+vi;re[i+k+len/2]=ur-vr;im[i+k+len/2]=ui-vi;float nc=cr*wr-ci*wi;ci=cr*wi+ci*wr;cr=nc;}}}+}+static std::vector<float> compute_fbank(std::vector<float> wav,int&T_out){+ for(auto&v:wav)v*=32768.0f; std::vector<float> win(WINLEN);+ for(int i=0;i<WINLEN;i++)win[i]=0.54f-0.46f*cosf(2.0f*M_PI*i/(WINLEN-1));+ const int NBIN=NFFT/2+1; float bw=(float)FS/NFFT,ml=melf(LOWF),mh=melf(HIGHF),dm=(mh-ml)/(NMEL+1);+ std::vector<std::vector<float>> fb(NMEL,std::vector<float>(NBIN,0.0f));+ for(int m=0;m<NMEL;m++){float L=ml+m*dm,C=ml+(m+1)*dm,R=ml+(m+2)*dm;+ for(int k=0;k<NBIN;k++){float mf=melf(bw*k); if(mf>L&&mf<R)fb[m][k]=mf<=C?(mf-L)/(C-L):(R-mf)/(R-C);}}+ int N=wav.size(),T=(N-WINLEN)/SHIFT+1; std::vector<std::vector<float>> feat(T,std::vector<float>(NMEL));+ std::vector<float> re(NFFT),im(NFFT),fr(WINLEN); const float fl=1.1920929e-07f;+ for(int t=0;t<T;t++){const float*s=wav.data()+t*SHIFT; double mn=0; for(int i=0;i<WINLEN;i++)mn+=s[i]; mn/=WINLEN;+ for(int i=0;i<WINLEN;i++)fr[i]=s[i]-(float)mn; for(int i=WINLEN-1;i>0;i--)fr[i]-=PREEMPH*fr[i-1]; fr[0]-=PREEMPH*fr[0];+ for(int i=0;i<NFFT;i++){re[i]=i<WINLEN?fr[i]*win[i]:0.0f;im[i]=0.0f;} fftc(re,im,NFFT);+ for(int m=0;m<NMEL;m++){float e=0;for(int k=0;k<NBIN;k++)if(fb[m][k]>0)e+=fb[m][k]*(re[k]*re[k]+im[k]*im[k]); feat[t][m]=logf(e>fl?e:fl);}}+ const int pad=(LFR_M-1)/2; int Tl=(T+LFR_N-1)/LFR_N; std::vector<std::vector<float>> pd; pd.reserve(T+pad+LFR_M);+ for(int i=0;i<pad;i++)pd.push_back(feat[0]); for(int t=0;t<T;t++)pd.push_back(feat[t]);+ while((int)pd.size()<(Tl-1)*LFR_N+LFR_M)pd.push_back(feat[T-1]);+ int D=LFR_M*NMEL; std::vector<float> out((size_t)Tl*D);+ for(int i=0;i<Tl;i++)for(int j=0;j<LFR_M;j++)memcpy(&out[(size_t)i*D+j*NMEL],pd[i*LFR_N+j].data(),NMEL*sizeof(float));+ T_out=Tl; return out;+}++struct cfg { int d_model=512,n_head=4,num_blocks=50,tp_blocks=20,kernel=11,vocab=25055,blank=0; };+struct model { cfg c; ggml_context*ctx_w=nullptr; std::map<std::string,ggml_tensor*> t;+ ggml_tensor* g(const std::string&n){auto it=t.find(n);if(it==t.end()){fprintf(stderr,"missing %s\n",n.c_str());exit(1);}return it->second;} };++static ggml_tensor* lin(ggml_context*c,ggml_tensor*w,ggml_tensor*b,ggml_tensor*x){auto y=ggml_mul_mat(c,w,x);return b?ggml_add(c,y,b):y;}+static ggml_tensor* lnorm(ggml_context*c,ggml_tensor*x,ggml_tensor*g,ggml_tensor*b){return ggml_add(c,ggml_mul(c,ggml_norm(c,x,LN_EPS),g),b);}+static ggml_tensor* sanm_attn(ggml_context*c,model&m,const std::string&p,ggml_tensor*x,int T){+ const int D=m.c.d_model,H=m.c.n_head,dk=D/H,K=m.c.kernel;+ ggml_tensor*qkv=lin(c,m.g(p+"linear_q_k_v.weight"),m.g(p+"linear_q_k_v.bias"),x); size_t nb1=qkv->nb[1];+ ggml_tensor*q=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,0));+ ggml_tensor*k=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)D*sizeof(float)));+ ggml_tensor*v=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)2*D*sizeof(float)));+ const int pad=(K-1)/2; ggml_tensor*fk=m.g(p+"fsmn_block.weight");+ ggml_tensor*vp=ggml_pad_ext(c,v,0,0,pad,pad,0,0,0,0); ggml_tensor*fsmn=v;+ for(int j=0;j<K;j++){auto sl=ggml_view_2d(c,vp,D,T,vp->nb[1],(size_t)j*vp->nb[1]);+ auto wj=ggml_view_1d(c,fk,D,(size_t)j*fk->nb[1]); fsmn=ggml_add(c,fsmn,ggml_mul(c,ggml_cont(c,sl),wj));}+ q=ggml_permute(c,ggml_reshape_3d(c,q,dk,H,T),0,2,1,3); k=ggml_permute(c,ggml_reshape_3d(c,k,dk,H,T),0,2,1,3);+ ggml_tensor*vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,v,dk,H,T),1,2,0,3));+ ggml_tensor*kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk)));+ ggml_tensor*o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,T);+ return ggml_add(c,lin(c,m.g(p+"linear_out.weight"),m.g(p+"linear_out.bias"),o),fsmn);+}+static ggml_tensor* sanm_layer(ggml_context*c,model&m,const std::string&p,ggml_tensor*x,int T,bool res){+ auto r=x; auto h=lnorm(c,x,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));+ auto sa=sanm_attn(c,m,p+"self_attn.",h,T); x=res?ggml_add(c,r,sa):sa; r=x;+ h=lnorm(c,x,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));+ h=lin(c,m.g(p+"feed_forward.w_1.weight"),m.g(p+"feed_forward.w_1.bias"),h); h=ggml_relu(c,h);+ h=lin(c,m.g(p+"feed_forward.w_2.weight"),m.g(p+"feed_forward.w_2.bias"),h); return ggml_add(c,r,h);+}+static void add_posenc(std::vector<float>&x,int T,int depth){+ double inc=log(10000.0)/(depth/2.0-1.0);+ for(int t=0;t<T;t++){double pos=t+1;for(int i=0;i<depth/2;i++){double its=exp(i*-inc),st=pos*its;+ x[(size_t)t*depth+i]+=(float)sin(st);x[(size_t)t*depth+depth/2+i]+=(float)cos(st);}}+}++int main(int argc,char**argv){+ std::string gguf_path,fbank_path,wav_path;+ for(int i=1;i<argc;i++){ if(!strcmp(argv[i],"-m")&&i+1<argc)gguf_path=argv[++i];+ else if(!strcmp(argv[i],"-f")&&i+1<argc)fbank_path=argv[++i];+ else if(!strcmp(argv[i],"-a")&&i+1<argc)wav_path=argv[++i];+ else {fprintf(stderr,"usage: %s -m sensevoice.gguf (-a audio.wav | -f fbank.bin)\n",argv[0]);return 1;} }+ if(gguf_path.empty()||(fbank_path.empty()&&wav_path.empty())){fprintf(stderr,"missing args\n");return 1;}++ // load model+ model m; gguf_init_params gp={false,&m.ctx_w}; gguf_context*gg=gguf_init_from_file(gguf_path.c_str(),gp);+ if(!gg){fprintf(stderr,"load gguf failed\n");return 1;}+ auto rd=[&](const char*k,int d){int i=gguf_find_key(gg,k);return i<0?d:(int)gguf_get_val_u32(gg,i);};+ m.c.d_model=rd("sv.output_size",512); m.c.n_head=rd("sv.attention_heads",4);+ m.c.num_blocks=rd("sv.num_blocks",50); m.c.tp_blocks=rd("sv.tp_blocks",20);+ m.c.kernel=rd("sv.kernel_size",11); m.c.vocab=rd("sv.vocab_size",25055); m.c.blank=rd("sv.blank_id",0);+ int qi=gguf_find_key(gg,"sv.query_tokens"); int nq=qi<0?0:(int)gguf_get_arr_n(gg,qi);+ std::vector<int> qtok(nq); for(int i=0;i<nq;i++) qtok[i]=((const int32_t*)gguf_get_arr_data(gg,qi))[i];+ for(int i=0;i<gguf_get_n_tensors(gg);i++){const char*nm=gguf_get_tensor_name(gg,i);m.t[nm]=ggml_get_tensor(m.ctx_w,nm);}+ gguf_free(gg);+ const int F=560, D=m.c.d_model, V=m.c.vocab;++ // fbank [T,F]: from wav (-a) or precomputed (-f)+ int32_t T=0,Fc=F; std::vector<float> fb;+ if(!wav_path.empty()){+ std::vector<float> wav; if(!read_wav16(wav_path.c_str(),wav)){fprintf(stderr,"read wav failed\n");return 1;}+ int t=0; fb=compute_fbank(wav,t); T=t;+ } else {+ FILE*f=fopen(fbank_path.c_str(),"rb"); if(!f){fprintf(stderr,"open fbank\n");return 1;}+ if(fread(&T,4,1,f)!=1||fread(&Fc,4,1,f)!=1){fclose(f);return 1;}+ fb.resize((size_t)T*Fc); if((int)fread(fb.data(),4,fb.size(),f)!=(int)fb.size()){fclose(f);return 1;} fclose(f);+ }++ // NOTE: SenseVoiceSmall inference() feeds the RAW log-mel fbank to the encoder;+ // it does NOT apply am.mvn CMVN (that path is unused at inference). Applying it+ // makes the encoder predict <|nospeech|>. So no CMVN here.++ // prepend nq query tokens (rows of embed.weight) -> input [nq+T, F]+ float*emb=(float*)m.g("embed.weight")->data; // [16, 560] row-major+ int N=nq+T; std::vector<float> inp((size_t)N*F);+ for(int i=0;i<nq;i++) memcpy(&inp[(size_t)i*F], &emb[(size_t)qtok[i]*F], F*sizeof(float));+ memcpy(&inp[(size_t)nq*F], fb.data(), (size_t)T*F*sizeof(float));++ // encoder pre-scale + posenc+ float sc=sqrtf((float)D); for(auto&v:inp)v*=sc; add_posenc(inp,N,F);++ // build graph+ ggml_backend_t be=ggml_backend_cpu_init();+ ggml_init_params cp={(size_t)1024*1024*1024,nullptr,true}; ggml_context*c=ggml_init(cp);+ ggml_tensor*x=ggml_new_tensor_2d(c,GGML_TYPE_F32,F,N); ggml_set_input(x);+ ggml_tensor*h=sanm_layer(c,m,"encoder.encoders0.0.",x,N,false);+ for(int i=0;i<m.c.num_blocks-1;i++) h=sanm_layer(c,m,"encoder.encoders."+std::to_string(i)+".",h,N,true);+ h=lnorm(c,h,m.g("encoder.after_norm.weight"),m.g("encoder.after_norm.bias"));+ for(int i=0;i<m.c.tp_blocks;i++) h=sanm_layer(c,m,"encoder.tp_encoders."+std::to_string(i)+".",h,N,true);+ h=lnorm(c,h,m.g("encoder.tp_norm.weight"),m.g("encoder.tp_norm.bias"));+ ggml_tensor*logits=lin(c,m.g("ctc.ctc_lo.weight"),m.g("ctc.ctc_lo.bias"),h); // [V, N]+ ggml_set_output(logits);+ ggml_cgraph*gf=ggml_new_graph_custom(c,32768,false); ggml_build_forward_expand(gf,logits);+ ggml_gallocr_t ga=ggml_gallocr_new(ggml_backend_cpu_buffer_type()); ggml_gallocr_alloc_graph(ga,gf);+ ggml_backend_tensor_set(x,inp.data(),0,ggml_nbytes(x));+ ggml_backend_cpu_set_n_threads(be,8);+ int64_t t0=ggml_time_us();+ if(ggml_backend_graph_compute(be,gf)!=GGML_STATUS_SUCCESS){fprintf(stderr,"compute failed\n");return 1;}+ int64_t t1=ggml_time_us();++ std::vector<float> lg((size_t)V*N); ggml_backend_tensor_get(logits,lg.data(),0,ggml_nbytes(logits));+ // greedy CTC: argmax per frame -> collapse consecutive -> drop blank+ int prev=-1;+ for(int n=0;n<N;n++){+ const float*col=&lg[(size_t)n*V]; int am=0; float best=col[0];+ for(int v=1;v<V;v++) if(col[v]>best){best=col[v];am=v;}+ if(am!=prev && am!=m.c.blank) printf("%d ", am);+ prev=am;+ }+ printf("\n");+ fprintf(stderr,"[sensevoice] N=%d (q%d+T%d) encode %.2fs\n",N,nq,T,(t1-t0)/1e6);+ ggml_gallocr_free(ga); ggml_free(c); ggml_backend_free(be);+ if(m.ctx_w) ggml_free(m.ctx_w);+ return 0;+}