Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
feat(runtime): add JSONL timestamp output (#3462)
zhifu gao4dc8e5b4f072
diff --git a/docs/superpowers/plans/2026-08-03-funasr-binary-json-timestamps.md b/docs/superpowers/plans/2026-08-03-funasr-binary-json-timestamps.mdnew file mode 100644index 000000000..8b9bed5c6--- /dev/null+++ b/docs/superpowers/plans/2026-08-03-funasr-binary-json-timestamps.md@@ -0,0 +1,166 @@+# FunASR Binary JSON Timestamp Output Implementation Plan++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.++**Goal:** Give native FunASR ONNX Runtime binaries an opt-in, machine-readable JSONL transcript and timestamp output while preserving default behavior.++**Architecture:** A small shared C++ formatter parses native timestamp strings with nlohmann/json and emits a fixed record schema. Four existing binary entry points add one validated CLI option and call the formatter; RTF callers guard complete-line writes with a mutex. Focused C++ and Python contract tests cover behavior and integration without requiring model downloads.++**Tech Stack:** C++14, nlohmann/json 3.11.2 already fetched by the runtime build, TCLAP, CMake, Python subprocess tests.++## Global Constraints++- `--output-format log` remains the default and preserves current result logging.+- `--output-format jsonl` writes one completed-input record to stdout; diagnostics remain on stderr.+- Every JSON record contains `key`, `mode`, `text`, `timestamp`, and `stamp_sents`.+- Missing, malformed, or non-array native timestamp payloads serialize as `[]`.+- RTF records are atomic but may be emitted out of input order.+- No model download is required for the formatter test.++---++### Task 1: Shared JSON result formatter++**Files:**+- Create: `runtime/onnxruntime/bin/result-json.h`+- Create: `runtime/onnxruntime/bin/result-json.cpp`+- Create: `runtime/onnxruntime/bin/tests/result-json-test.cpp`+- Modify: `runtime/onnxruntime/CMakeLists.txt`+- Modify: `runtime/onnxruntime/bin/CMakeLists.txt`++**Interfaces:**+- Produces: `funasr::OutputFormat ParseOutputFormat(const std::string&)`.+- Produces: `std::string FormatResultJson(const std::string& key, const std::string& mode, const std::string& text, const std::string& timestamp, const std::string& stamp_sents)`.++- [ ] **Step 1: Write the failing C++ test**++Cover `log`/`jsonl`, invalid values, Unicode and quote escaping, valid timestamp arrays, and malformed/empty arrays.++- [ ] **Step 2: Configure and build the test to verify RED**++Run:++```bash+cmake -S runtime/onnxruntime -B /tmp/funasr-3457-build -DFUNASR_BUILD_TESTS=ON -DONNXRUNTIME_DIR=/path/to/onnxruntime+cmake --build /tmp/funasr-3457-build --target funasr-result-json-test -j2+```++Expected: configuration or compilation fails because the formatter and target do not exist.++- [ ] **Step 3: Implement the minimal formatter and test target**++Use `nlohmann::json::parse(raw, nullptr, false)` and accept only arrays; always construct the outer object with nlohmann/json. Add `FUNASR_BUILD_TESTS` defaulting to `OFF`, and register `funasr-result-json-test` only when enabled.++- [ ] **Step 4: Run the C++ test to verify GREEN**++Run:++```bash+cmake --build /tmp/funasr-3457-build --target funasr-result-json-test -j2+/tmp/funasr-3457-build/bin/funasr-result-json-test+```++Expected: exit 0 with all formatter assertions satisfied.++### Task 2: Wire JSONL into four native binaries++**Files:**+- Modify: `runtime/onnxruntime/bin/funasr-onnx-offline.cpp`+- Modify: `runtime/onnxruntime/bin/funasr-onnx-offline-rtf.cpp`+- Modify: `runtime/onnxruntime/bin/funasr-onnx-2pass.cpp`+- Modify: `runtime/onnxruntime/bin/funasr-onnx-2pass-rtf.cpp`+- Create: `runtime/onnxruntime/bin/tests/cli-output-format-test.py`+- Modify: `runtime/onnxruntime/bin/CMakeLists.txt`++**Interfaces:**+- Consumes: `ParseOutputFormat` and `FormatResultJson` from Task 1.+- Produces: `--output-format log|jsonl` on all four binaries.++- [ ] **Step 1: Write the failing executable integration test**++Run each built binary with `--help` and assert that it exposes `--output-format`. Run each binary with its required model/audio arguments pointed at nonexistent paths plus `--output-format yaml`, and assert exit status 2 plus the validation error. This proves the option is wired and rejected before model initialization without scanning implementation text.++- [ ] **Step 2: Run the focused test to verify RED**++Run: `python runtime/onnxruntime/bin/tests/cli-output-format-test.py --bin-dir /tmp/funasr-3457-build/bin`++Expected: failures because all four executables omit the option.++- [ ] **Step 3: Add the option and result emission**++Keep the existing log branch unchanged. In JSONL mode, emit one final record per successful input. Use `offline` for offline binaries and the selected `online`/`offline`/`2pass` value for two-pass binaries. In two-pass mode, use the corrected offline transcript as `text`.++- [ ] **Step 4: Run focused Python and C++ tests**++Run:++```bash+/tmp/funasr-3457-build/bin/funasr-result-json-test+python runtime/onnxruntime/bin/tests/cli-output-format-test.py --bin-dir /tmp/funasr-3457-build/bin+```++Expected: all pass.++### Task 3: Bilingual usage documentation++**Files:**+- Create: `runtime/docs/onnxruntime_binary_output.md`+- Create: `runtime/docs/onnxruntime_binary_output_zh.md`+- Modify: `runtime/onnxruntime/readme.md`+- Modify: `runtime/readme.md`+- Modify: `runtime/readme_cn.md`++**Interfaces:**+- Produces: discoverable English and Chinese guides for the stable record schema.++- [ ] **Step 1: Add the guides and links from the ONNX Runtime build guide and both runtime indexes**++Provide commands for offline and two-pass binaries, a valid one-line JSON example, and explicit empty-array behavior.++- [ ] **Step 2: Run the repository markdown-link validator**++Run:++```bash+python -m pytest -q tests/test_markdown_relative_links.py+```++Expected: all pass.++### Task 4: Full verification, backup, and publication++**Files:**+- Verify all files changed by Tasks 1-3.++**Interfaces:**+- Produces: signed commit, exact-head backup bundle/patch, and a ready PR linked to #3457.++- [ ] **Step 1: Build all changed runtime targets**++Run:++```bash+cmake --build /tmp/funasr-3457-build --target funasr-onnx-offline funasr-onnx-offline-rtf funasr-onnx-2pass funasr-onnx-2pass-rtf -j2+```++Expected: all four binaries link successfully.++- [ ] **Step 2: Run focused and repository quality gates**++Run:++```bash+python runtime/onnxruntime/bin/tests/cli-output-format-test.py --bin-dir /tmp/funasr-3457-build/bin+python -m pytest -q tests/test_markdown_relative_links.py+git diff --check+```++Expected: all tests pass and diff check is clean.++- [ ] **Step 3: Preserve rollback evidence**++Create an exact-head bundle, patch, changed-file copies, build/test logs, and SHA-256 manifest under `/cpfs_speech/user/zhifu.gzf/.cache/funasr-ops/backups/funasr-issue3457-json-timestamps-20260803`.++- [ ] **Step 4: Commit, push, and open the PR**++Create an SSH-signed+DCO commit, push `codex/funasr-binary-json-timestamps-3457-20260803`, open a ready PR that fixes #3457, and monitor all checks/reviews before merging.diff --git a/docs/superpowers/specs/2026-08-03-funasr-binary-json-timestamps-design.md b/docs/superpowers/specs/2026-08-03-funasr-binary-json-timestamps-design.mdnew file mode 100644index 000000000..9a3c2baf4--- /dev/null+++ b/docs/superpowers/specs/2026-08-03-funasr-binary-json-timestamps-design.md@@ -0,0 +1,58 @@+# FunASR Binary JSON Timestamp Output Design++## Context++FunASR issue #3457 asks for text and timestamps from the native binaries so non-Python applications can consume recognition results. The four relevant ONNX Runtime binaries already call `FunASRGetStamp`, and the offline variants also call `FunASRGetStampSents`, but they only write human-oriented glog messages. That output is hard to parse reliably because it includes timestamps, thread ids, and unrelated runtime logs.++## Decision++Add an opt-in `--output-format jsonl` mode to:++- `funasr-onnx-offline`+- `funasr-onnx-offline-rtf`+- `funasr-onnx-2pass`+- `funasr-onnx-2pass-rtf`++The default remains `--output-format log`, preserving every existing invocation and log-oriented workflow. JSON records are written to stdout; glog continues to write diagnostics to stderr.++## Record Contract++Each completed input produces exactly one JSON object followed by one newline:++```json+{"key":"utt-1","mode":"offline","text":"recognized text","timestamp":[[0,320],[320,660]],"stamp_sents":[]}+```++The fields are always present:++- `key`: the `wav.scp` key, or `wav_default_id` for a direct audio path.+- `mode`: `offline`, `online`, or `2pass`.+- `text`: the final transcript. Two-pass mode emits the corrected offline transcript.+- `timestamp`: a JSON array parsed from `FunASRGetStamp`; it is `[]` when the model/runtime returns no valid timestamp array.+- `stamp_sents`: a JSON array parsed from `FunASRGetStampSents`; it is `[]` when unavailable. The current two-pass binary does not expose sentence timestamps, so its field remains an empty array rather than changing type or disappearing.++Malformed native timestamp payloads must not produce malformed JSON or crash result emission; they degrade to an empty array. Text and keys are escaped by nlohmann/json rather than handwritten string manipulation.++## Concurrency++RTF binaries may finish inputs out of order. They must serialize the complete JSON string before taking an output mutex, then write the line while holding that mutex. This guarantees valid, non-interleaved JSONL records without imposing ordered buffering or reducing inference concurrency.++## Components++`runtime/onnxruntime/bin/result-json.{h,cpp}` owns output-format parsing and JSON serialization. The four binary entry points only gather the existing result fields and call that module. `runtime/onnxruntime/bin/tests/result-json-test.cpp` tests real C++ behavior without loading a model. A Python contract test verifies that all four entry points expose the option and that the bilingual documentation names the stable fields.++## Error Handling++Only `log` and `jsonl` are accepted. An unsupported value logs a clear error and exits with status 2 before model initialization. Empty, non-array, or malformed timestamp payloads become `[]`; inference failures retain the existing error path and do not emit a false successful record.++## Documentation++Add a bilingual native-binary output guide with exact commands, stdout/stderr separation, schema definitions, timestamp-capable model requirements, direct-file versus `wav.scp` keys, and the unordered nature of RTF output. Link it from the ONNX Runtime build guide and the English and Chinese runtime indexes.++## Verification++1. Red-first C++ tests cover Unicode/escaping, parsed arrays, empty arrays, malformed payloads, and output-format validation.+2. Red-first Python contract tests cover all four CLI entry points; the repository markdown-link validator covers all three documentation links.+3. Build and run the isolated `funasr-result-json-test` target.+4. Compile all four changed binary translation units through the repository CMake target when the local ONNX Runtime SDK permits it; otherwise record the exact environmental gate and still run syntax checks against the repository headers.+5. Run focused tests, markdown-link validation, formatting, and `git diff --check` before push.diff --git a/runtime/docs/onnxruntime_binary_output.md b/runtime/docs/onnxruntime_binary_output.mdnew file mode 100644index 000000000..b802011af--- /dev/null+++ b/runtime/docs/onnxruntime_binary_output.md@@ -0,0 +1,58 @@+# ONNX Runtime binary JSONL output++The native ONNX Runtime binaries log results by default. Add `--output-format jsonl` when another process needs one machine-readable record for every completed input.++## Offline recognition++From `runtime/onnxruntime`, run:++```shell+./build/bin/funasr-onnx-offline \+ --model-dir /path/to/timestamp-capable-model \+ --wav-path /path/to/audio.wav \+ --output-format jsonl \+ > results.jsonl 2> runtime.log+```++The same option is available on `funasr-onnx-offline-rtf`.++## Two-pass recognition++```shell+./build/bin/funasr-onnx-2pass \+ --model-dir /path/to/offline-model \+ --online-model-dir /path/to/online-model \+ --vad-dir /path/to/vad-model \+ --punc-dir /path/to/online-punctuation-model \+ --wav-path /path/to/audio.wav \+ --mode 2pass \+ --output-format jsonl \+ > results.jsonl 2> runtime.log+```++The same option is available on `funasr-onnx-2pass-rtf`. Valid modes are `offline`, `online`, and `2pass`. The native two-pass runtime requires both a VAD model and an online punctuation model for successful inference.++## Record schema++Each stdout line is one JSON object:++```json+{"key":"utt-001","mode":"offline","stamp_sents":[{"end":920,"start":0,"text_seg":"hello world"}],"text":"hello world","timestamp":[[0,420],[460,920]]}+```++| Field | Type | Meaning |+| --- | --- | --- |+| `key` | string | The first column of a `wav.scp` row, or `wav_default_id` for a direct file input. |+| `mode` | string | `offline`, `online`, or `2pass`. |+| `text` | string | The completed transcript. In `2pass` mode this is the corrected offline transcript. |+| `timestamp` | array | The native token/word timestamp array returned by the model. |+| `stamp_sents` | array | Native sentence timestamp objects when the binary and model provide them. Two-pass binaries currently return an empty array here. |++Use a timestamp-capable model to receive timestamp values. If a model returns no timestamp, or the native payload is empty, malformed, or not a JSON array, the corresponding field is `[]`.++## Streaming and batch behavior++- JSON records are written only to stdout. Runtime diagnostics and existing progress logs remain on stderr.+- A `wav.scp` input produces one line per successfully completed key.+- RTF binaries protect each complete line from interleaving, but worker completion order is not guaranteed. Join records by `key` instead of relying on line order.+- Omitting `--output-format` is equivalent to `--output-format log` and preserves the existing logging behavior.diff --git a/runtime/docs/onnxruntime_binary_output_zh.md b/runtime/docs/onnxruntime_binary_output_zh.mdnew file mode 100644index 000000000..cd4a7ddfb--- /dev/null+++ b/runtime/docs/onnxruntime_binary_output_zh.md@@ -0,0 +1,58 @@+# ONNX Runtime 原生二进制 JSONL 输出++ONNX Runtime 原生二进制默认通过日志输出识别结果。当其他程序需要逐条读取结构化结果时,增加 `--output-format jsonl`,每个完成的输入会输出一行 JSON。++## 离线识别++在 `runtime/onnxruntime` 目录执行:++```shell+./build/bin/funasr-onnx-offline \+ --model-dir /path/to/timestamp-capable-model \+ --wav-path /path/to/audio.wav \+ --output-format jsonl \+ > results.jsonl 2> runtime.log+```++`funasr-onnx-offline-rtf` 同样支持该选项。++## 两遍识别++```shell+./build/bin/funasr-onnx-2pass \+ --model-dir /path/to/offline-model \+ --online-model-dir /path/to/online-model \+ --vad-dir /path/to/vad-model \+ --punc-dir /path/to/online-punctuation-model \+ --wav-path /path/to/audio.wav \+ --mode 2pass \+ --output-format jsonl \+ > results.jsonl 2> runtime.log+```++`funasr-onnx-2pass-rtf` 同样支持该选项。合法模式为 `offline`、`online` 和 `2pass`。原生两遍识别运行时需要同时提供 VAD 模型和在线标点模型才能完成推理。++## 输出结构++标准输出中的每一行都是一个独立 JSON 对象:++```json+{"key":"utt-001","mode":"offline","stamp_sents":[{"end":920,"start":0,"text_seg":"你好世界"}],"text":"你好世界","timestamp":[[0,420],[460,920]]}+```++| 字段 | 类型 | 含义 |+| --- | --- | --- |+| `key` | string | `wav.scp` 每行第一列;直接传入单个文件时为 `wav_default_id`。 |+| `mode` | string | `offline`、`online` 或 `2pass`。 |+| `text` | string | 完整转写文本;`2pass` 模式下为离线纠错后的最终文本。 |+| `timestamp` | array | 模型返回的原生词元或词级时间戳数组。 |+| `stamp_sents` | array | 二进制和模型支持时返回的原生句级时间戳对象;两遍识别二进制当前在此字段返回空数组。 |++需要使用支持时间戳的模型才能得到时间戳值。模型没有返回时间戳,或原生数据为空、格式错误、不是 JSON 数组时,相应字段固定输出 `[]`。++## 流式与批量行为++- JSON 记录只写入标准输出;运行诊断和现有进度日志仍写入标准错误。+- 输入 `wav.scp` 时,每个成功完成的 key 输出一行。+- RTF 二进制会保证单行 JSON 不被多个线程交错写入,但不同任务的完成顺序不固定。请通过 `key` 关联结果,不要依赖行顺序。+- 不传 `--output-format` 等同于 `--output-format log`,保留原有日志行为。diff --git a/runtime/onnxruntime/CMakeLists.txt b/runtime/onnxruntime/CMakeLists.txtindex 2e6cd09b1..c6506de97 100644--- a/runtime/onnxruntime/CMakeLists.txt+++ b/runtime/onnxruntime/CMakeLists.txt@@ -6,6 +6,11 @@ option(ENABLE_GLOG "Whether to build glog" ON)option(ENABLE_FST "Whether to build openfst" ON) # ITN need openfst compiledoption(ENABLE_FFMPEG "Whether to enable ffmpeg audio decoding" OFF)option(GPU "Whether to build with GPU" OFF)+option(FUNASR_BUILD_TESTS "Build FunASR ONNX Runtime unit tests" OFF)++if(FUNASR_BUILD_TESTS)+ enable_testing()+endif()# set(CMAKE_CXX_STANDARD 11)set(CMAKE_CXX_STANDARD 14 CACHE STRING "The C++ version to be used.")diff --git a/runtime/onnxruntime/bin/CMakeLists.txt b/runtime/onnxruntime/bin/CMakeLists.txtindex a9f79d077..7bc1c61fb 100644--- a/runtime/onnxruntime/bin/CMakeLists.txt+++ b/runtime/onnxruntime/bin/CMakeLists.txt@@ -11,9 +11,12 @@ include_directories(${PROJECT_SOURCE_DIR}/third_party)SET(RELATION_SOURCE "../src/resample.cpp" "../src/util.cpp" "../src/alignedmem.cpp" "../src/encode_converter.cpp")endif()+add_library(funasr-result-json STATIC "result-json.cpp")+target_include_directories(funasr-result-json PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})+add_executable(funasr-onnx-offline "funasr-onnx-offline.cpp" ${RELATION_SOURCE})target_link_options(funasr-onnx-offline PRIVATE "-Wl,--no-as-needed")-target_link_libraries(funasr-onnx-offline PUBLIC funasr)+target_link_libraries(funasr-onnx-offline PUBLIC funasr funasr-result-json)add_executable(funasr-onnx-offline-vad "funasr-onnx-offline-vad.cpp" ${RELATION_SOURCE})target_link_options(funasr-onnx-offline-vad PRIVATE "-Wl,--no-as-needed")@@ -37,20 +40,32 @@ target_link_libraries(funasr-onnx-online-punc PUBLIC funasr)add_executable(funasr-onnx-offline-rtf "funasr-onnx-offline-rtf.cpp" ${RELATION_SOURCE})target_link_options(funasr-onnx-offline-rtf PRIVATE "-Wl,--no-as-needed")-target_link_libraries(funasr-onnx-offline-rtf PUBLIC funasr)+target_link_libraries(funasr-onnx-offline-rtf PUBLIC funasr funasr-result-json)add_executable(funasr-onnx-2pass "funasr-onnx-2pass.cpp" ${RELATION_SOURCE})target_link_options(funasr-onnx-2pass PRIVATE "-Wl,--no-as-needed")-target_link_libraries(funasr-onnx-2pass PUBLIC funasr)+target_link_libraries(funasr-onnx-2pass PUBLIC funasr funasr-result-json)add_executable(funasr-onnx-2pass-rtf "funasr-onnx-2pass-rtf.cpp" ${RELATION_SOURCE})target_link_options(funasr-onnx-2pass-rtf PRIVATE "-Wl,--no-as-needed")-target_link_libraries(funasr-onnx-2pass-rtf PUBLIC funasr)+target_link_libraries(funasr-onnx-2pass-rtf PUBLIC funasr funasr-result-json)add_executable(funasr-onnx-online-rtf "funasr-onnx-online-rtf.cpp" ${RELATION_SOURCE})target_link_options(funasr-onnx-online-rtf PRIVATE "-Wl,--no-as-needed")target_link_libraries(funasr-onnx-online-rtf PUBLIC funasr)+if(FUNASR_BUILD_TESTS)+ find_package(Python3 REQUIRED COMPONENTS Interpreter)+ add_executable(funasr-result-json-test "tests/result-json-test.cpp")+ target_link_libraries(funasr-result-json-test PRIVATE funasr-result-json)+ add_test(NAME funasr-result-json-test COMMAND funasr-result-json-test)+ add_test(+ NAME funasr-cli-output-format-test+ COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/tests/cli-output-format-test.py+ --bin-dir ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}+ )+endif()+# include_directories(${FFMPEG_DIR}/include)# add_executable(ff "ffmpeg.cpp")# target_link_libraries(ff PUBLIC avutil avcodec avformat swresample)diff --git a/runtime/onnxruntime/bin/funasr-onnx-2pass-rtf.cpp b/runtime/onnxruntime/bin/funasr-onnx-2pass-rtf.cppindex d4abacd1f..092def85f 100644--- a/runtime/onnxruntime/bin/funasr-onnx-2pass-rtf.cpp+++ b/runtime/onnxruntime/bin/funasr-onnx-2pass-rtf.cpp@@ -22,6 +22,7 @@#include "tclap/CmdLine.h"#include "com-define.h"#include "audio.h"+#include "result-json.h"using namespace std;@@ -45,7 +46,8 @@ void GetValue(TCLAP::ValueArg<std::string>& value_arg, string key, std::map<std:void runReg(FUNASR_HANDLE tpass_handle, std::vector<int> chunk_size, vector<string> wav_list, vector<string> wav_ids, int audio_fs,float* total_length, long* total_time, int core_id, ASR_TYPE asr_mode_, string nn_hotwords_,- float glob_beam, float lat_beam, float am_scale, int inc_bias, unordered_map<string, int> hws_map) {+ float glob_beam, float lat_beam, float am_scale, int inc_bias, unordered_map<string, int> hws_map,+ funasr::OutputFormat output_format) {struct timeval start, end;long seconds = 0;@@ -136,6 +138,7 @@ void runReg(FUNASR_HANDLE tpass_handle, std::vector<int> chunk_size, vector<stristring online_res="";string tpass_res="";string time_stamp_res="";+ bool has_result = false;std::vector<std::vector<string>> punc_cache(2);for (int sample_offset = 0; sample_offset < buff_len; sample_offset += std::min(step, buff_len - sample_offset)) {if (sample_offset + step >= buff_len - 1) {@@ -154,6 +157,7 @@ void runReg(FUNASR_HANDLE tpass_handle, std::vector<int> chunk_size, vector<striif (result){+ has_result = true;string online_msg = FunASRGetResult(result, 0);online_res += online_msg;if(online_msg != ""){@@ -183,16 +187,24 @@ void runReg(FUNASR_HANDLE tpass_handle, std::vector<int> chunk_size, vector<striLOG(ERROR) << ("No return data!\n");}}- if(asr_mode_ == 2){- LOG(INFO) <<"Thread: " << this_thread::get_id() <<" " << wav_ids[i] << " Final online results "<<" : "<<online_res;- }- if(asr_mode_==1){- LOG(INFO) <<"Thread: " << this_thread::get_id() <<" " << wav_ids[i] << " Final online results "<<" : "<<tpass_res;- }- if(asr_mode_ == 0 || asr_mode_==2){- LOG(INFO) <<"Thread: " << this_thread::get_id() <<" " << wav_ids[i] << " Final offline results " <<" : "<<tpass_res;- if(time_stamp_res!=""){- LOG(INFO) <<"Thread: " << this_thread::get_id() <<" " << wav_ids[i] << " Final timestamp results " <<" : "<<time_stamp_res;+ if (output_format == funasr::OutputFormat::kJsonl && has_result) {+ const string mode = asr_mode_ == 0 ? "offline" : (asr_mode_ == 1 ? "online" : "2pass");+ const string line = funasr::FormatResultJson(+ wav_ids[i], mode, tpass_res, time_stamp_res, "");+ lock_guard<mutex> guard(mtx);+ cout << line << endl;+ } else {+ if(asr_mode_ == 2){+ LOG(INFO) <<"Thread: " << this_thread::get_id() <<" " << wav_ids[i] << " Final online results "<<" : "<<online_res;+ }+ if(asr_mode_==1){+ LOG(INFO) <<"Thread: " << this_thread::get_id() <<" " << wav_ids[i] << " Final online results "<<" : "<<tpass_res;+ }+ if(asr_mode_ == 0 || asr_mode_==2){+ LOG(INFO) <<"Thread: " << this_thread::get_id() <<" " << wav_ids[i] << " Final offline results " <<" : "<<tpass_res;+ if(time_stamp_res!=""){+ LOG(INFO) <<"Thread: " << this_thread::get_id() <<" " << wav_ids[i] << " Final timestamp results " <<" : "<<time_stamp_res;+ }}}@@ -236,6 +248,7 @@ int main(int argc, char** argv)TCLAP::ValueArg<std::string> wav_path("", WAV_PATH, "the input could be: wav_path, e.g.: asr_example.wav; pcm_path, e.g.: asr_example.pcm; wav.scp, kaldi style wav list (wav_id \t wav_path)", true, "", "string");TCLAP::ValueArg<std::int32_t> audio_fs("", AUDIO_FS, "the sample rate of audio", false, 16000, "int32_t");TCLAP::ValueArg<std::string> hotword("", HOTWORD, "the hotword file, one hotword perline, Format: Hotword Weight (could be: 阿里巴巴 20)", false, "", "string");+ TCLAP::ValueArg<std::string> output_format("", "output-format", "result output: log (Default) or jsonl", false, "log", "string");cmd.add(offline_model_dir);cmd.add(online_model_dir);@@ -256,8 +269,17 @@ int main(int argc, char** argv)cmd.add(onnx_thread);cmd.add(thread_num_);cmd.add(hotword);+ cmd.add(output_format);cmd.parse(argc, argv);+ funasr::OutputFormat output_format_;+ try {+ output_format_ = funasr::ParseOutputFormat(output_format.getValue());+ } catch (const std::invalid_argument& error) {+ LOG(ERROR) << error.what();+ return 2;+ }+std::map<std::string, std::string> model_path;GetValue(offline_model_dir, OFFLINE_MODEL_DIR, model_path);GetValue(online_model_dir, ONLINE_MODEL_DIR, model_path);@@ -350,7 +372,7 @@ int main(int argc, char** argv)for (int i = 0; i < rtf_threds; i++){threads.emplace_back(thread(runReg, tpass_hanlde, chunk_size, wav_list, wav_ids, audio_fs.getValue(), &total_length, &total_time, i, (ASR_TYPE)asr_mode_, nn_hotwords_,- glob_beam, lat_beam, am_sc, fst_inc_wts.getValue(), hws_map));+ glob_beam, lat_beam, am_sc, fst_inc_wts.getValue(), hws_map, output_format_));}for (auto& thread : threads)@@ -366,4 +388,3 @@ int main(int argc, char** argv)FunTpassUninit(tpass_hanlde);return 0;}-diff --git a/runtime/onnxruntime/bin/funasr-onnx-2pass.cpp b/runtime/onnxruntime/bin/funasr-onnx-2pass.cppindex 694115104..f61818609 100644--- a/runtime/onnxruntime/bin/funasr-onnx-2pass.cpp+++ b/runtime/onnxruntime/bin/funasr-onnx-2pass.cpp@@ -18,6 +18,7 @@#include "funasrruntime.h"#include "tclap/CmdLine.h"#include "com-define.h"+#include "result-json.h"#include "audio.h"using namespace std;@@ -62,6 +63,7 @@ int main(int argc, char** argv)TCLAP::ValueArg<std::string> wav_path("", WAV_PATH, "the input could be: wav_path, e.g.: asr_example.wav; pcm_path, e.g.: asr_example.pcm; wav.scp, kaldi style wav list (wav_id \t wav_path)", true, "", "string");TCLAP::ValueArg<std::int32_t> audio_fs("", AUDIO_FS, "the sample rate of audio", false, 16000, "int32_t");TCLAP::ValueArg<std::string> hotword("", HOTWORD, "the hotword file, one hotword perline, Format: Hotword Weight (could be: 阿里巴巴 20)", false, "", "string");+ TCLAP::ValueArg<std::string> output_format("", "output-format", "result output: log (Default) or jsonl", false, "log", "string");cmd.add(offline_model_dir);cmd.add(online_model_dir);@@ -81,8 +83,17 @@ int main(int argc, char** argv)cmd.add(asr_mode);cmd.add(onnx_thread);cmd.add(hotword);+ cmd.add(output_format);cmd.parse(argc, argv);+ funasr::OutputFormat output_format_;+ try {+ output_format_ = funasr::ParseOutputFormat(output_format.getValue());+ } catch (const std::invalid_argument& error) {+ LOG(ERROR) << error.what();+ return 2;+ }+std::map<std::string, std::string> model_path;GetValue(offline_model_dir, OFFLINE_MODEL_DIR, model_path);GetValue(online_model_dir, ONLINE_MODEL_DIR, model_path);@@ -206,6 +217,7 @@ int main(int argc, char** argv)string online_res="";string tpass_res="";string time_stamp_res="";+ bool has_result = false;std::vector<std::vector<string>> punc_cache(2);for (int sample_offset = 0; sample_offset < buff_len; sample_offset += std::min(step, buff_len - sample_offset)) {if (sample_offset + step >= buff_len - 1) {@@ -224,6 +236,7 @@ int main(int argc, char** argv)if (result){+ has_result = true;string online_msg = FunASRGetResult(result, 0);online_res += online_msg;if(online_msg != ""){@@ -248,16 +261,21 @@ int main(int argc, char** argv)FunASRFreeResult(result);}}- if(asr_mode_==2){- LOG(INFO) << wav_id << " Final online results "<<" : "<<online_res;- }- if(asr_mode_==1){- LOG(INFO) << wav_id << " Final online results "<<" : "<<tpass_res;- }- if(asr_mode_==0 || asr_mode_==2){- LOG(INFO) << wav_id << " Final offline results " <<" : "<<tpass_res;- if(time_stamp_res!=""){- LOG(INFO) << wav_id << " Final timestamp results " <<" : "<<time_stamp_res;+ if (output_format_ == funasr::OutputFormat::kJsonl && has_result) {+ cout << funasr::FormatResultJson(+ wav_id, asr_mode.getValue(), tpass_res, time_stamp_res, "") << endl;+ } else {+ if(asr_mode_==2){+ LOG(INFO) << wav_id << " Final online results "<<" : "<<online_res;+ }+ if(asr_mode_==1){+ LOG(INFO) << wav_id << " Final online results "<<" : "<<tpass_res;+ }+ if(asr_mode_==0 || asr_mode_==2){+ LOG(INFO) << wav_id << " Final offline results " <<" : "<<tpass_res;+ if(time_stamp_res!=""){+ LOG(INFO) << wav_id << " Final timestamp results " <<" : "<<time_stamp_res;+ }}}}@@ -271,4 +289,3 @@ int main(int argc, char** argv)FunTpassUninit(tpass_handle);return 0;}-diff --git a/runtime/onnxruntime/bin/funasr-onnx-offline-rtf.cpp b/runtime/onnxruntime/bin/funasr-onnx-offline-rtf.cppindex d8d94734c..2e41c2150 100644--- a/runtime/onnxruntime/bin/funasr-onnx-offline-rtf.cpp+++ b/runtime/onnxruntime/bin/funasr-onnx-offline-rtf.cpp@@ -13,6 +13,7 @@#include "funasrruntime.h"#include "tclap/CmdLine.h"#include "com-define.h"+#include "result-json.h"#include <iostream>#include <fstream>@@ -31,7 +32,8 @@ std::mutex mtx;void runReg(FUNASR_HANDLE asr_handle, vector<string> wav_list, vector<string> wav_ids, int audio_fs,float* total_length, long* total_time, int core_id, float glob_beam = 3.0f, float lat_beam = 3.0f, float am_sc = 10.0f,- int fst_inc_wts = 20, string hotword_path = "") {+ int fst_inc_wts = 20, string hotword_path = "",+ funasr::OutputFormat output_format = funasr::OutputFormat::kLog) {struct timeval start, end;long seconds = 0;@@ -77,14 +79,21 @@ void runReg(FUNASR_HANDLE asr_handle, vector<string> wav_list, vector<string> waif(result){string msg = FunASRGetResult(result, 0);- LOG(INFO) << "Thread: " << this_thread::get_id() << "," << wav_ids[i] << " : " << msg;string stamp = FunASRGetStamp(result);- if(stamp !=""){- LOG(INFO) << "Thread: " << this_thread::get_id() << "," << wav_ids[i] << " : " << stamp;- }string stamp_sents = FunASRGetStampSents(result);- if(stamp_sents !=""){- LOG(INFO)<< wav_ids[i] <<" : "<<stamp_sents;+ if (output_format == funasr::OutputFormat::kJsonl) {+ const string line = funasr::FormatResultJson(+ wav_ids[i], "offline", msg, stamp, stamp_sents);+ lock_guard<mutex> guard(mtx);+ cout << line << endl;+ } else {+ LOG(INFO) << "Thread: " << this_thread::get_id() << "," << wav_ids[i] << " : " << msg;+ if(stamp !=""){+ LOG(INFO) << "Thread: " << this_thread::get_id() << "," << wav_ids[i] << " : " << stamp;+ }+ if(stamp_sents !=""){+ LOG(INFO)<< wav_ids[i] <<" : "<<stamp_sents;+ }}float snippet_time = FunASRGetRetSnippetTime(result);n_total_length += snippet_time;@@ -143,6 +152,7 @@ int main(int argc, char *argv[])TCLAP::ValueArg<std::int32_t> audio_fs("", AUDIO_FS, "the sample rate of audio", false, 16000, "int32_t");TCLAP::ValueArg<std::int32_t> thread_num("", THREAD_NUM, "multi-thread num for rtf", false, 1, "int32_t");TCLAP::ValueArg<std::string> hotword("", HOTWORD, "the hotword file, one hotword perline, Format: Hotword Weight (could be: 阿里巴巴 20)", false, "", "string");+ TCLAP::ValueArg<std::string> output_format("", "output-format", "result output: log (Default) or jsonl", false, "log", "string");TCLAP::SwitchArg use_gpu("", INFER_GPU, "Whether to use GPU for inference, default is false", false);TCLAP::ValueArg<std::int32_t> batch_size("", BATCHSIZE, "batch_size for ASR model when using GPU", false, 4, "int32_t");@@ -159,6 +169,7 @@ int main(int argc, char *argv[])cmd.add(lattice_beam);cmd.add(am_scale);cmd.add(hotword);+ cmd.add(output_format);cmd.add(fst_inc_wts);cmd.add(wav_path);cmd.add(audio_fs);@@ -167,6 +178,14 @@ int main(int argc, char *argv[])cmd.add(batch_size);cmd.parse(argc, argv);+ funasr::OutputFormat output_format_;+ try {+ output_format_ = funasr::ParseOutputFormat(output_format.getValue());+ } catch (const std::invalid_argument& error) {+ LOG(ERROR) << error.what();+ return 2;+ }+std::map<std::string, std::string> model_path;GetValue(model_dir, MODEL_DIR, model_path);GetValue(quantize, QUANTIZE, model_path);@@ -246,7 +265,7 @@ int main(int argc, char *argv[])}for (int i = 0; i < rtf_threds; i++){- threads.emplace_back(thread(runReg, asr_handle, wav_list, wav_ids, audio_fs.getValue(), &total_length, &total_time, i, glob_beam, lat_beam, am_sc, value_bias, hotword_path));+ threads.emplace_back(thread(runReg, asr_handle, wav_list, wav_ids, audio_fs.getValue(), &total_length, &total_time, i, glob_beam, lat_beam, am_sc, value_bias, hotword_path, output_format_));}for (auto& thread : threads)diff --git a/runtime/onnxruntime/bin/funasr-onnx-offline.cpp b/runtime/onnxruntime/bin/funasr-onnx-offline.cppindex 87928f59f..1f3741eb7 100644--- a/runtime/onnxruntime/bin/funasr-onnx-offline.cpp+++ b/runtime/onnxruntime/bin/funasr-onnx-offline.cpp@@ -17,6 +17,7 @@#include "funasrruntime.h"#include "tclap/CmdLine.h"#include "com-define.h"+#include "result-json.h"#include <unordered_map>#include "util.h"#include "audio.h"@@ -59,6 +60,7 @@ int main(int argc, char** argv)TCLAP::ValueArg<std::string> wav_path("", WAV_PATH, "the input could be: wav_path, e.g.: asr_example.wav; pcm_path, e.g.: asr_example.pcm; wav.scp, kaldi style wav list (wav_id \t wav_path)", true, "", "string");TCLAP::ValueArg<std::int32_t> audio_fs("", AUDIO_FS, "the sample rate of audio", false, 16000, "int32_t");TCLAP::ValueArg<std::string> hotword("", HOTWORD, "the hotword file, one hotword perline, Format: Hotword Weight (could be: 阿里巴巴 20)", false, "", "string");+ TCLAP::ValueArg<std::string> output_format("", "output-format", "result output: log (Default) or jsonl", false, "log", "string");TCLAP::SwitchArg use_gpu("", INFER_GPU, "Whether to use GPU for inference, default is false", false);TCLAP::ValueArg<std::int32_t> batch_size("", BATCHSIZE, "batch_size for ASR model when using GPU", false, 4, "int32_t");@@ -78,10 +80,19 @@ int main(int argc, char** argv)cmd.add(wav_path);cmd.add(audio_fs);cmd.add(hotword);+ cmd.add(output_format);cmd.add(use_gpu);cmd.add(batch_size);cmd.parse(argc, argv);+ funasr::OutputFormat output_format_;+ try {+ output_format_ = funasr::ParseOutputFormat(output_format.getValue());+ } catch (const std::invalid_argument& error) {+ LOG(ERROR) << error.what();+ return 2;+ }+std::map<std::string, std::string> model_path;GetValue(model_dir, MODEL_DIR, model_path);GetValue(quantize, QUANTIZE, model_path);@@ -201,14 +212,18 @@ int main(int argc, char** argv)if (result){string msg = FunASRGetResult(result, 0);- LOG(INFO)<< wav_id <<" : "<<msg;string stamp = FunASRGetStamp(result);- if(stamp !=""){- LOG(INFO)<< wav_id <<" : "<<stamp;- }string stamp_sents = FunASRGetStampSents(result);- if(stamp_sents !=""){- LOG(INFO)<< wav_id <<" : "<<stamp_sents;+ if (output_format_ == funasr::OutputFormat::kJsonl) {+ cout << funasr::FormatResultJson(wav_id, "offline", msg, stamp, stamp_sents) << endl;+ } else {+ LOG(INFO)<< wav_id <<" : "<<msg;+ if(stamp !=""){+ LOG(INFO)<< wav_id <<" : "<<stamp;+ }+ if(stamp_sents !=""){+ LOG(INFO)<< wav_id <<" : "<<stamp_sents;+ }}snippet_time += FunASRGetRetSnippetTime(result);FunASRFreeResult(result);@@ -227,4 +242,3 @@ int main(int argc, char** argv)FunOfflineUninit(asr_hanlde);return 0;}-diff --git a/runtime/onnxruntime/bin/result-json.cpp b/runtime/onnxruntime/bin/result-json.cppnew file mode 100644index 000000000..7c4e53fe2--- /dev/null+++ b/runtime/onnxruntime/bin/result-json.cpp@@ -0,0 +1,48 @@+#include "result-json.h"++#include <stdexcept>++#include "nlohmann/json.hpp"++namespace funasr {+namespace {++nlohmann::json ParseArrayOrEmpty(const std::string& value) {+ if (value.empty()) {+ return nlohmann::json::array();+ }++ auto parsed = nlohmann::json::parse(value, nullptr, false);+ if (parsed.is_discarded() || !parsed.is_array()) {+ return nlohmann::json::array();+ }+ return parsed;+}++} // namespace++OutputFormat ParseOutputFormat(const std::string& value) {+ if (value == "log") {+ return OutputFormat::kLog;+ }+ if (value == "jsonl") {+ return OutputFormat::kJsonl;+ }+ throw std::invalid_argument("unsupported output format '" + value ++ "'; expected 'log' or 'jsonl'");+}++std::string FormatResultJson(const std::string& key, const std::string& mode,+ const std::string& text, const std::string& timestamp,+ const std::string& stamp_sents) {+ nlohmann::json record = {+ {"key", key},+ {"mode", mode},+ {"text", text},+ {"timestamp", ParseArrayOrEmpty(timestamp)},+ {"stamp_sents", ParseArrayOrEmpty(stamp_sents)},+ };+ return record.dump();+}++} // namespace funasrdiff --git a/runtime/onnxruntime/bin/result-json.h b/runtime/onnxruntime/bin/result-json.hnew file mode 100644index 000000000..25806647e--- /dev/null+++ b/runtime/onnxruntime/bin/result-json.h@@ -0,0 +1,19 @@+#pragma once++#include <stdexcept>+#include <string>++namespace funasr {++enum class OutputFormat {+ kLog,+ kJsonl,+};++OutputFormat ParseOutputFormat(const std::string& value);++std::string FormatResultJson(const std::string& key, const std::string& mode,+ const std::string& text, const std::string& timestamp,+ const std::string& stamp_sents);++} // namespace funasrdiff --git a/runtime/onnxruntime/bin/tests/cli-output-format-test.py b/runtime/onnxruntime/bin/tests/cli-output-format-test.pynew file mode 100644index 000000000..5587833a5--- /dev/null+++ b/runtime/onnxruntime/bin/tests/cli-output-format-test.py@@ -0,0 +1,71 @@+#!/usr/bin/env python3++import argparse+import subprocess+from pathlib import Path+++BINARIES = {+ "funasr-onnx-offline": ["--model-dir", "/missing/offline", "--wav-path", "/missing/audio.wav"],+ "funasr-onnx-offline-rtf": ["--model-dir", "/missing/offline", "--wav-path", "/missing/audio.wav"],+ "funasr-onnx-2pass": [+ "--model-dir",+ "/missing/offline",+ "--online-model-dir",+ "/missing/online",+ "--wav-path",+ "/missing/audio.wav",+ ],+ "funasr-onnx-2pass-rtf": [+ "--model-dir",+ "/missing/offline",+ "--online-model-dir",+ "/missing/online",+ "--wav-path",+ "/missing/audio.wav",+ ],+}+++def run(command):+ return subprocess.run(command, capture_output=True, text=True, check=False)+++def main():+ parser = argparse.ArgumentParser()+ parser.add_argument("--bin-dir", type=Path, required=True)+ args = parser.parse_args()++ failures = []+ for name, required_args in BINARIES.items():+ binary = args.bin_dir / name+ if not binary.is_file():+ failures.append(f"{name}: executable not found at {binary}")+ continue++ help_result = run([str(binary), "--help"])+ help_output = help_result.stdout + help_result.stderr+ if help_result.returncode != 0:+ failures.append(f"{name}: --help exited {help_result.returncode}")+ if "--output-format" not in help_output:+ failures.append(f"{name}: --help does not expose --output-format")++ invalid_result = run(+ [str(binary), *required_args, "--output-format", "yaml"]+ )+ invalid_output = invalid_result.stdout + invalid_result.stderr+ if invalid_result.returncode != 2:+ failures.append(+ f"{name}: invalid format exited {invalid_result.returncode}, expected 2"+ )+ if "unsupported output format 'yaml'" not in invalid_output:+ failures.append(f"{name}: invalid format error is not actionable")+ if "FunASR init failed" in invalid_output:+ failures.append(f"{name}: invalid format reached model initialization")++ if failures:+ raise AssertionError("\n".join(failures))+++if __name__ == "__main__":+ main()diff --git a/runtime/onnxruntime/bin/tests/result-json-test.cpp b/runtime/onnxruntime/bin/tests/result-json-test.cppnew file mode 100644index 000000000..0f8a698d1--- /dev/null+++ b/runtime/onnxruntime/bin/tests/result-json-test.cpp@@ -0,0 +1,45 @@+#include "result-json.h"++#include <cassert>+#include <stdexcept>+#include <string>++#include "nlohmann/json.hpp"++int main() {+ using funasr::FormatResultJson;+ using funasr::OutputFormat;+ using funasr::ParseOutputFormat;++ assert(ParseOutputFormat("log") == OutputFormat::kLog);+ assert(ParseOutputFormat("jsonl") == OutputFormat::kJsonl);++ bool rejected_invalid_format = false;+ try {+ ParseOutputFormat("yaml");+ } catch (const std::invalid_argument&) {+ rejected_invalid_format = true;+ }+ assert(rejected_invalid_format);++ const auto record = nlohmann::json::parse(FormatResultJson(+ "utt-\"1", "offline", "你好 \"FunASR\"", "[[0,320],[320,660]]",+ "[{\"text_seg\":\"你好\",\"start\":0,\"end\":660}]"));+ assert(record.at("key") == "utt-\"1");+ assert(record.at("mode") == "offline");+ assert(record.at("text") == "你好 \"FunASR\"");+ assert(record.at("timestamp") == nlohmann::json::parse("[[0,320],[320,660]]"));+ assert(record.at("stamp_sents").at(0).at("start") == 0);++ const auto empty_record = nlohmann::json::parse(+ FormatResultJson("utt-2", "2pass", "done", "", "not-json"));+ assert(empty_record.at("timestamp") == nlohmann::json::array());+ assert(empty_record.at("stamp_sents") == nlohmann::json::array());++ const auto wrong_type_record = nlohmann::json::parse(+ FormatResultJson("utt-3", "online", "partial", "{\"start\":0}", "null"));+ assert(wrong_type_record.at("timestamp") == nlohmann::json::array());+ assert(wrong_type_record.at("stamp_sents") == nlohmann::json::array());++ return 0;+}diff --git a/runtime/onnxruntime/readme.md b/runtime/onnxruntime/readme.mdindex 39c3c4d06..d3ee72d6b 100644--- a/runtime/onnxruntime/readme.md+++ b/runtime/onnxruntime/readme.md@@ -1,5 +1,7 @@# Please ref to [websocket service](https://github.com/modelscope/FunASR/tree/main/runtime/websocket)+Machine-readable transcript and timestamp output: [JSONL output guide](../docs/onnxruntime_binary_output.md) | [中文](../docs/onnxruntime_binary_output_zh.md)+# If you want to compile the file yourself, you can follow the steps below.## Building for Linux/Unix### Download onnxruntimediff --git a/runtime/readme.md b/runtime/readme.mdindex 6a4b22eae..723c8c970 100644--- a/runtime/readme.md+++ b/runtime/readme.md@@ -1,6 +1,8 @@# FunASR Runtime Roadmap中文文档([点击此处](./readme_cn.md))+Native ONNX Runtime JSONL transcript and timestamp output: [usage guide](./docs/onnxruntime_binary_output.md)+FunASR is a speech recognition framework developed by the Speech Lab of DAMO Academy, which integrates industrial-level models in the fields of speech endpoint detection, speech recognition, punctuation segmentation, and more.It has attracted many developers to participate in experiencing and developing. To solve the last mile of industrial landing and integrate models into business, we have developed the FunASR runtime-SDK. The SDK supports several service deployments, including:diff --git a/runtime/readme_cn.md b/runtime/readme_cn.mdindex 122f9a5ef..e6c4745b3 100644--- a/runtime/readme_cn.md+++ b/runtime/readme_cn.md@@ -2,6 +2,8 @@English Version([docs](./readme.md))+原生 ONNX Runtime JSONL 转写与时间戳输出:[使用文档](./docs/onnxruntime_binary_output_zh.md)+FunASR是由阿里巴巴通义实验室语音团队开源的一款语音识别基础框架,集成了语音端点检测、语音识别、标点断句等领域的工业级别模型,吸引了众多开发者参与体验和开发。为了解决工业落地的最后一公里,将模型集成到业务中去,我们开发了社区软件包。支持以下几种服务部署: