Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
feat(runtime): add Vulkan llama.cpp backend packaging
zhifu gaoe3327be1bde6
diff --git a/.github/workflows/build-llamacpp-binaries.yml b/.github/workflows/build-llamacpp-binaries.ymlindex 18834dd83..342001d0e 100644--- a/.github/workflows/build-llamacpp-binaries.yml+++ b/.github/workflows/build-llamacpp-binaries.yml@@ -48,6 +48,24 @@ jobs:-DGGML_FMA=ON-DGGML_F16C=ON-DGGML_BMI2=ON+ - os: ubuntu-latest+ name: linux-x64-vulkan+ vulkan: true+ build_target: llama-funasr-sensevoice+ timeout_minutes: 90+ cmake_flags: >-+ -DGGML_NATIVE=OFF+ -DGGML_VULKAN=ON+ -DGGML_AVX=OFF+ -DGGML_AVX2=OFF+ -DGGML_AVX_VNNI=OFF+ -DGGML_AVX512=OFF+ -DGGML_AVX512_VBMI=OFF+ -DGGML_AVX512_VNNI=OFF+ -DGGML_AVX512_BF16=OFF+ -DGGML_FMA=OFF+ -DGGML_F16C=OFF+ -DGGML_BMI2=OFF- os: ubuntu-24.04-armname: linux-arm64cmake_flags: -DGGML_NATIVE=OFF@@ -124,6 +142,12 @@ jobs:- name: Show CUDA compilerif: matrix.cudarun: nvcc --version+ - name: Install Vulkan SDK packages+ if: matrix.vulkan+ run: |+ sudo apt-get update+ sudo apt-get install -y libvulkan-dev glslc spirv-headers vulkan-tools+ vulkaninfo --summary || true- name: Configure and buildrun: |# Release binaries must be portable across user machines, not tuned to@@ -131,9 +155,9 @@ jobs:# AVX-VNNI instructions and then crash with SIGILL on ordinary CPUs.# Keep the default x64 package conservative, and publish explicit# x64-avx2 assets for machines that support AVX2/FMA/F16C/BMI2.- # The CUDA release asset is SenseVoiceSmall-specific for now; building- # only that target keeps the Windows CUDA job bounded and matches the- # backend currently exposed by `--backend cuda`.+ # The GPU release assets are SenseVoiceSmall-specific for now; building+ # only that target keeps the bounded jobs aligned with the backend+ # currently exposed by `--backend cuda` / `--backend vulkan`.cmake_args=(-B build-DCMAKE_BUILD_TYPE=Releasediff --git a/runtime/llama.cpp/README.md b/runtime/llama.cpp/README.mdindex 6f7cd62e2..d751ee089 100644--- a/runtime/llama.cpp/README.md+++ b/runtime/llama.cpp/README.md@@ -94,6 +94,38 @@ with the CUDA Toolkit version configured by the release workflow. A binary builtwithout `-DGGML_CUDA=ON` exits with a clear message if `--backend cuda` isrequested.+### Optional Linux Vulkan backend for SenseVoiceSmall++Tagged releases also publish `funasr-llamacpp-linux-x64-vulkan.tar.gz` for+SenseVoiceSmall graph execution through ggml's Vulkan backend. This is useful on+Linux systems with AMD, Intel, NVIDIA, or integrated GPUs that expose a working+Vulkan driver/ICD. Download the `linux-x64-vulkan` asset, install your vendor GPU+driver, then select the backend at runtime:++```bash+# From the extracted linux-x64-vulkan package:+./llama-funasr-sensevoice \+ -m sensevoice-small-q8.gguf --vad fsmn-vad.gguf -a sample.wav --backend vulkan+```++Build from source when you need a local Vulkan SDK, distro-specific driver+stack, or to validate a device before release packaging:++```bash+sudo apt-get install libvulkan-dev glslc spirv-headers vulkan-tools+vulkaninfo --summary+cmake -B build-vulkan -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON+cmake --build build-vulkan -j --target llama-funasr-sensevoice+./build-vulkan/bin/llama-funasr-sensevoice \+ -m sensevoice-small-f16.gguf -a sample.wav --backend vulkan+```++`--backend cpu` remains the default and is what the portable cross-platform+prebuilt binaries use. A binary built without `-DGGML_VULKAN=ON` exits with a+clear message if `--backend vulkan` is requested. Vulkan performance and device+availability depend on the installed GPU driver/ICD rather than on CUDA compute+capability.+## Build (shared)```bashgit clone https://github.com/ggml-org/llama.cpp && cd llama.cpp@@ -137,12 +169,13 @@ Response:{"text": "transcribed text"}```-CUDA-enabled SenseVoice builds can be selected with `--backend cuda`. Extra-binary flags can be forwarded with repeated `--extra-arg`, for example-`--extra-arg --keep-tags`. This wrapper starts one subprocess per request, so it-is best for local tools, demos, and integration tests. For sustained production-traffic, use the Python `funasr-server` OpenAI-compatible service or build a-dedicated native server around the C++ runtime.+CUDA- and Vulkan-enabled SenseVoice builds can be selected with `--backend cuda`+or `--backend vulkan`. Extra binary flags can be forwarded with repeated+`--extra-arg`, for example `--extra-arg --keep-tags`. This wrapper starts one subprocess per request,+so it is best for local tools, demos, and integration+tests. For sustained production traffic, use the Python `funasr-server`+OpenAI-compatible service or build a dedicated native server around the C+++runtime.## Validationdiff --git a/runtime/llama.cpp/sensevoice/funasr-sensevoice/funasr-sensevoice.cpp b/runtime/llama.cpp/sensevoice/funasr-sensevoice/funasr-sensevoice.cppindex 4d915ea40..99e8577d6 100644--- a/runtime/llama.cpp/sensevoice/funasr-sensevoice/funasr-sensevoice.cpp+++ b/runtime/llama.cpp/sensevoice/funasr-sensevoice/funasr-sensevoice.cpp@@ -10,6 +10,7 @@#include "ggml-backend.h"#include "gguf.h"+#include <cctype>#include <cmath>#include <cstdint>#include <cstdio>@@ -66,6 +67,29 @@ struct graph_backend {bool is_cpu=true;};+static std::string lower_copy(const char*s){+ std::string out=s?s:"";+ for(char&c:out)c=(char)std::tolower((unsigned char)c);+ return out;+}++static ggml_backend_dev_t find_gpu_backend_device(const std::string&backend_name){+ ggml_backend_load_all();+ for(size_t i=0;i<ggml_backend_dev_count();i++){+ ggml_backend_dev_t dev=ggml_backend_dev_get(i);+ if(ggml_backend_dev_type(dev)!=GGML_BACKEND_DEVICE_TYPE_GPU) continue;+ std::string reg=lower_copy(ggml_backend_reg_name(ggml_backend_dev_backend_reg(dev)));+ std::string dev_name=lower_copy(ggml_backend_dev_name(dev));+ std::string dev_desc=lower_copy(ggml_backend_dev_description(dev));+ if(reg.find(backend_name)!=std::string::npos||+ dev_name.find(backend_name)!=std::string::npos||+ dev_desc.find(backend_name)!=std::string::npos){+ return dev;+ }+ }+ return nullptr;+}+static graph_backend make_graph_backend(const std::string&name){graph_backend out;if(name=="cpu"){@@ -73,8 +97,7 @@ static graph_backend make_graph_backend(const std::string&name){out.buffer_type=ggml_backend_get_default_buffer_type(out.backend);out.is_cpu=true;} else if(name=="cuda"){- ggml_backend_load_all();- ggml_backend_dev_t dev=ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_GPU);+ ggml_backend_dev_t dev=find_gpu_backend_device("cuda");if(!dev){fprintf(stderr,"CUDA backend requested, but no GPU backend is available; build with -DGGML_CUDA=ON\n");exit(1);@@ -82,8 +105,17 @@ static graph_backend make_graph_backend(const std::string&name){out.backend=ggml_backend_dev_init(dev,nullptr);out.buffer_type=ggml_backend_get_default_buffer_type(out.backend);out.is_cpu=false;+ } else if(name=="vulkan"){+ ggml_backend_dev_t dev=find_gpu_backend_device("vulkan");+ if(!dev){+ fprintf(stderr,"Vulkan backend requested, but no Vulkan GPU backend is available; build with -DGGML_VULKAN=ON and install a Vulkan driver/ICD\n");+ exit(1);+ }+ out.backend=ggml_backend_dev_init(dev,nullptr);+ out.buffer_type=ggml_backend_get_default_buffer_type(out.backend);+ out.is_cpu=false;} else {- fprintf(stderr,"unsupported backend '%s' (expected cpu|cuda)\n",name.c_str());+ fprintf(stderr,"unsupported backend '%s' (expected cpu|cuda|vulkan)\n",name.c_str());exit(1);}if(!out.backend||!out.buffer_type){@@ -146,7 +178,7 @@ int main(int argc,char**argv){else if(!strcmp(argv[i],"--backend")&&i+1<argc)backend_name=argv[++i];else if(!strcmp(argv[i],"--ids"))ids_mode=true;else if(!strcmp(argv[i],"--keep-tags"))keep_tags=true;- else {fprintf(stderr,"usage: %s -m sensevoice.gguf (-a audio.wav | -f fbank.bin) [--vad fsmn-vad.gguf [--vad-maxseg ms]] [--backend cpu|cuda] [--ids] [--keep-tags]\n",argv[0]);return 1;} }+ else {fprintf(stderr,"usage: %s -m sensevoice.gguf (-a audio.wav | -f fbank.bin) [--vad fsmn-vad.gguf [--vad-maxseg ms]] [--backend cpu|cuda|vulkan] [--ids] [--keep-tags]\n",argv[0]);return 1;} }if(gguf_path.empty()||(fbank_path.empty()&&wav_path.empty())){fprintf(stderr,"missing args\n");return 1;}graph_backend graph_be=make_graph_backend(backend_name);diff --git a/runtime/llama.cpp/tests/test_backend_flags.py b/runtime/llama.cpp/tests/test_backend_flags.pyindex 7310eebb4..4f1adb707 100644--- a/runtime/llama.cpp/tests/test_backend_flags.py+++ b/runtime/llama.cpp/tests/test_backend_flags.py@@ -9,7 +9,7 @@ def test_sensevoice_exposes_backend_flag():source = SENSEVOICE.read_text(encoding="utf-8")assert "--backend" in source- assert "cpu|cuda" in source+ assert "cpu|cuda|vulkan" in sourcedef test_sensevoice_does_not_hardcode_cpu_graph_backend():@@ -20,3 +20,11 @@ def test_sensevoice_does_not_hardcode_cpu_graph_backend():assert "graph_be.buffer_type" in run_seg_bodyassert "ggml_backend_cpu_init()" not in run_seg_bodyassert "ggml_backend_cpu_buffer_type()" not in run_seg_body+++def test_sensevoice_vulkan_backend_has_dedicated_error_message():+ source = SENSEVOICE.read_text(encoding="utf-8")++ assert 'name=="vulkan"' in source+ assert "GGML_VULKAN=ON" in source+ assert "unsupported backend '%s' (expected cpu|cuda|vulkan)" in sourcediff --git a/runtime/llama.cpp/tests/test_release_workflow.py b/runtime/llama.cpp/tests/test_release_workflow.pyindex bddd44587..06e87a3be 100644--- a/runtime/llama.cpp/tests/test_release_workflow.py+++ b/runtime/llama.cpp/tests/test_release_workflow.py@@ -16,6 +16,15 @@ def test_windows_cuda_release_asset_is_in_matrix():assert "timeout_minutes: 90" in workflow+def test_linux_vulkan_release_asset_is_in_matrix():+ workflow = WORKFLOW.read_text(encoding="utf-8")++ assert "name: linux-x64-vulkan" in workflow+ assert "vulkan: true" in workflow+ assert "build_target: llama-funasr-sensevoice" in workflow+ assert "timeout_minutes: 90" in workflow++def test_windows_cuda_build_uses_cuda_toolkit_and_flags():workflow = WORKFLOW.read_text(encoding="utf-8")@@ -30,6 +39,15 @@ def test_windows_cuda_build_uses_cuda_toolkit_and_flags():assert "--target \"${{ matrix.build_target }}\"" in workflow+def test_linux_vulkan_build_uses_vulkan_sdk_and_flags():+ workflow = WORKFLOW.read_text(encoding="utf-8")++ assert "libvulkan-dev" in workflow+ assert "vulkan-tools" in workflow+ assert "if: matrix.vulkan" in workflow+ assert "-DGGML_VULKAN=ON" in workflow++def test_release_notes_explain_cpu_and_cuda_windows_assets():readme = (ROOT / "runtime" / "llama.cpp" / "README.md").read_text(encoding="utf-8")@@ -43,6 +61,15 @@ def test_release_notes_explain_cpu_and_cuda_windows_assets():assert "sm_120" in readme+def test_release_notes_explain_vulkan_linux_asset():+ readme = (ROOT / "runtime" / "llama.cpp" / "README.md").read_text(encoding="utf-8")++ assert "linux-x64-vulkan" in readme+ assert "--backend vulkan" in readme+ assert "Linux Vulkan" in readme+ assert "GGML_VULKAN=ON" in readme++def test_readme_documents_lightweight_http_server():readme = (ROOT / "runtime" / "llama.cpp" / "README.md").read_text(encoding="utf-8")