Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
Codex/create training and inference scripts (#2791)
ZZUZSL1024a397d0b27eaf
diff --git a/examples/industrial_data_pretraining/paraformer/README_LoRA_zh.md b/examples/industrial_data_pretraining/paraformer/README_LoRA_zh.mdnew file mode 100644index 000000000..dac4e76b1--- /dev/null+++ b/examples/industrial_data_pretraining/paraformer/README_LoRA_zh.md@@ -0,0 +1,112 @@+# Paraformer LoRA 微调说明++本文档说明如何在 FunASR 中使用 LoRA 微调 Paraformer,并提供训练、推理与 CER 评测的完整示例。++## 1. 前置准备++1. 已准备好符合 FunASR 要求的 `train.jsonl` 与 `val.jsonl`。+2. 进入仓库根目录(示例路径):++```bash+cd /workspace/FunASR+```++## 2. 训练配置++LoRA 配置文件:++```+examples/industrial_data_pretraining/paraformer/conf/paraformer_lora.yaml+```++关键字段说明:+- `model`: 基座模型名称或本地模型路径。+- `lora_only`: 是否只训练 LoRA 参数。+- `lora_bias`: LoRA 偏置训练策略(`none`/`all`/`lora_only`)。+- `encoder_conf.lora_*` / `decoder_conf.lora_*`: LoRA 参数(rank/alpha/dropout)。+- `train_data_set_list`/`valid_data_set_list`: 训练/验证集 jsonl。++如需覆盖配置,请通过命令行 `++key=value` 传参。++## 3. 训练脚本++脚本:++```+examples/industrial_data_pretraining/paraformer/lora_finetune.sh+```++你只需要确认脚本中的数据路径:++```bash+data_dir="${workspace}/data/list"+train_data="${data_dir}/train.jsonl"+val_data="${data_dir}/val.jsonl"+```++运行:++```bash+bash examples/industrial_data_pretraining/paraformer/lora_finetune.sh+```++训练日志与模型输出将保存在:++```+examples/industrial_data_pretraining/paraformer/outputs_lora+```++## 4. 推理脚本++推理脚本会读取 jsonl 输入并生成 `text.hyp` / `text.ref`:++- Python 脚本:`examples/industrial_data_pretraining/paraformer/lora_infer.py`+- Shell 封装:`examples/industrial_data_pretraining/paraformer/lora_infer.sh`++修改 `lora_infer.sh` 中路径后运行:++```bash+bash examples/industrial_data_pretraining/paraformer/lora_infer.sh+```++输出目录默认:++```+examples/industrial_data_pretraining/paraformer/outputs_lora/infer+```++## 5. CER 评测++评测脚本:++```+examples/industrial_data_pretraining/paraformer/lora_cer.sh+```++运行:++```bash+bash examples/industrial_data_pretraining/paraformer/lora_cer.sh+```++结果会输出 CER 统计到:++```+examples/industrial_data_pretraining/paraformer/outputs_lora/infer/text.cer+```++## 6. 常见问题++1. **训练不收敛或效果差**+ - 尝试调整 `lora_rank`、`lora_alpha`、`lora_dropout`。+ - 调整 `optim_conf.lr` 与 `train_conf.max_epoch`。++2. **推理报错找不到配置**+ - 确保训练输出目录中存在 `config.yaml`,并在推理脚本中设置正确的 `config_path` 和 `config_name`。++3. **多卡训练**+ - 设置 `CUDA_VISIBLE_DEVICES`,脚本会自动计算 `gpu_num`。++---++如需进一步定制,可直接在 `paraformer_lora.yaml` 中修改配置或在命令行传参覆盖。diff --git a/examples/industrial_data_pretraining/paraformer/conf/paraformer_lora.yaml b/examples/industrial_data_pretraining/paraformer/conf/paraformer_lora.yamlnew file mode 100644index 000000000..d0776b328--- /dev/null+++ b/examples/industrial_data_pretraining/paraformer/conf/paraformer_lora.yaml@@ -0,0 +1,51 @@+# LoRA finetune config for Paraformer+# You can override data paths and hyper-parameters by command-line ++key=value.++# model hub name or local model dir+model: iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch+model_revision: master++# LoRA settings+lora_only: true+lora_bias: none++encoder_conf:+ lora_list: ["q", "k", "v", "o"]+ lora_rank: 8+ lora_alpha: 16+ lora_dropout: 0.05++decoder_conf:+ lora_list: ["q", "k", "v", "o"]+ lora_rank: 8+ lora_alpha: 16+ lora_dropout: 0.05++# dataset+train_data_set_list: data/list/train.jsonl+valid_data_set_list: data/list/val.jsonl++dataset: AudioDataset+dataset_conf:+ index_ds: IndexDSJsonl+ data_split_num: 1+ batch_sampler: BatchSampler+ batch_size: 6000+ sort_size: 1024+ batch_type: token+ num_workers: 4++# training+train_conf:+ max_epoch: 30+ log_interval: 10+ resume: true+ validate_interval: 2000+ save_checkpoint_interval: 2000+ keep_nbest_models: 10+ avg_nbest_model: 5+ use_deepspeed: false++optim: adam+optim_conf:+ lr: 0.0001diff --git a/examples/industrial_data_pretraining/paraformer/lora_cer.sh b/examples/industrial_data_pretraining/paraformer/lora_cer.shnew file mode 100755index 000000000..6ede74892--- /dev/null+++ b/examples/industrial_data_pretraining/paraformer/lora_cer.sh@@ -0,0 +1,23 @@+#!/usr/bin/env bash+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.+# MIT License (https://opensource.org/licenses/MIT)++set -euo pipefail++workspace=$(pwd)++infer_dir="${workspace}/examples/industrial_data_pretraining/paraformer/outputs_lora/infer"+ref_file="${infer_dir}/text.ref"+hyp_file="${infer_dir}/text.hyp"+cer_file="${infer_dir}/text.cer"++python -m funasr.metrics.wer \+ ++ref_file="${ref_file}" \+ ++hyp_file="${hyp_file}" \+ ++cer_file="${cer_file}" \+ ++cn_postprocess=false++# Show final CER summary+if [ -f "${cer_file}" ]; then+ tail -n 3 "${cer_file}"+fidiff --git a/examples/industrial_data_pretraining/paraformer/lora_finetune.sh b/examples/industrial_data_pretraining/paraformer/lora_finetune.shnew file mode 100755index 000000000..3411bfd77--- /dev/null+++ b/examples/industrial_data_pretraining/paraformer/lora_finetune.sh@@ -0,0 +1,46 @@+#!/usr/bin/env bash+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.+# MIT License (https://opensource.org/licenses/MIT)++set -euo pipefail++workspace=$(pwd)++# which gpu to train or finetune+export CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-"0"}+gpu_num=$(echo ${CUDA_VISIBLE_DEVICES} | awk -F "," '{print NF}')++# data dir, which contains train.jsonl/val.jsonl+# NOTE: update these paths to your dataset jsonl files.+data_dir="${workspace}/data/list"+train_data="${data_dir}/train.jsonl"+val_data="${data_dir}/val.jsonl"++# config+config_path="${workspace}/examples/industrial_data_pretraining/paraformer/conf"+config_name="paraformer_lora.yaml"++# exp output dir+output_dir="${workspace}/examples/industrial_data_pretraining/paraformer/outputs_lora"+log_file="${output_dir}/log.txt"++mkdir -p "${output_dir}"++DISTRIBUTED_ARGS="+ --nnodes ${WORLD_SIZE:-1} \+ --nproc_per_node ${gpu_num} \+ --node_rank ${RANK:-0} \+ --master_addr ${MASTER_ADDR:-127.0.0.1} \+ --master_port ${MASTER_PORT:-26669}+"++echo "log_file: ${log_file}"++torchrun ${DISTRIBUTED_ARGS} \+ funasr/bin/train_ds.py \+ --config-path "${config_path}" \+ --config-name "${config_name}" \+ ++train_data_set_list="${train_data}" \+ ++valid_data_set_list="${val_data}" \+ ++output_dir="${output_dir}" \+ &> "${log_file}"diff --git a/examples/industrial_data_pretraining/paraformer/lora_infer.py b/examples/industrial_data_pretraining/paraformer/lora_infer.pynew file mode 100755index 000000000..1bfb7907d--- /dev/null+++ b/examples/industrial_data_pretraining/paraformer/lora_infer.py@@ -0,0 +1,83 @@+#!/usr/bin/env python3+# -*- encoding: utf-8 -*-++import argparse+import json+import os+from typing import List, Tuple++from omegaconf import OmegaConf++from funasr import AutoModel+++def load_jsonl(jsonl_path: str) -> Tuple[List[str], List[str]]:+ keys = []+ targets = []+ with open(jsonl_path, "r", encoding="utf-8") as f:+ for line in f:+ if not line.strip():+ continue+ record = json.loads(line)+ key = record.get("key")+ if key is None and isinstance(record.get("source"), dict):+ key = record["source"].get("key")+ keys.append(key or "")+ targets.append(record.get("target", ""))+ return keys, targets+++def build_model(args: argparse.Namespace):+ kwargs = {}+ if args.config_path and args.config_name:+ cfg_path = os.path.join(args.config_path, args.config_name)+ cfg = OmegaConf.load(cfg_path)+ kwargs.update(OmegaConf.to_container(cfg, resolve=True))+ if args.model:+ kwargs["model"] = args.model+ if args.init_param:+ kwargs["init_param"] = args.init_param+ kwargs["device"] = args.device+ if args.batch_size:+ kwargs["batch_size"] = args.batch_size+ return AutoModel(**kwargs)+++def main() -> None:+ parser = argparse.ArgumentParser()+ parser.add_argument("--model", type=str, default=None, help="model name or model dir")+ parser.add_argument("--config-path", type=str, default=None, help="config directory")+ parser.add_argument("--config-name", type=str, default=None, help="config filename")+ parser.add_argument("--init-param", type=str, default=None, help="model checkpoint path")+ parser.add_argument("--input-jsonl", type=str, required=True, help="input jsonl with source/target")+ parser.add_argument("--output-dir", type=str, required=True, help="output directory")+ parser.add_argument("--device", type=str, default="cuda:0", help="cuda:0 or cpu")+ parser.add_argument("--batch-size", type=int, default=1, help="batch size for inference")+ args = parser.parse_args()++ os.makedirs(args.output_dir, exist_ok=True)++ keys, targets = load_jsonl(args.input_jsonl)++ model = build_model(args)+ results = model.generate(input=args.input_jsonl, batch_size=args.batch_size)++ hyp_path = os.path.join(args.output_dir, "text.hyp")+ ref_path = os.path.join(args.output_dir, "text.ref")++ with open(hyp_path, "w", encoding="utf-8") as hyp_f, open(+ ref_path, "w", encoding="utf-8"+ ) as ref_f:+ for idx, result in enumerate(results):+ key = keys[idx] if idx < len(keys) else result.get("key", f"utt_{idx}")+ hyp = result.get("text", "")+ ref = targets[idx] if idx < len(targets) else ""+ hyp_f.write(f"{key} {hyp}\n")+ ref_f.write(f"{key} {ref}\n")++ print(f"hyp saved to: {hyp_path}")+ print(f"ref saved to: {ref_path}")+++if __name__ == "__main__":+ main()diff --git a/examples/industrial_data_pretraining/paraformer/lora_infer.sh b/examples/industrial_data_pretraining/paraformer/lora_infer.shnew file mode 100755index 000000000..c6ef892e5--- /dev/null+++ b/examples/industrial_data_pretraining/paraformer/lora_infer.sh@@ -0,0 +1,32 @@+#!/usr/bin/env bash+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.+# MIT License (https://opensource.org/licenses/MIT)++set -euo pipefail++workspace=$(pwd)++# model path and config (from training output)+model_dir="${workspace}/examples/industrial_data_pretraining/paraformer/outputs_lora"+init_param="${model_dir}/model.pt"+config_path="${model_dir}"+config_name="config.yaml"++# input jsonl (must contain source/target)+input_jsonl="${workspace}/data/list/val.jsonl"++# output directory+output_dir="${model_dir}/infer"++# device+device="cuda:0"++python ${workspace}/examples/industrial_data_pretraining/paraformer/lora_infer.py \+ --model "${model_dir}" \+ --config-path "${config_path}" \+ --config-name "${config_name}" \+ --init-param "${init_param}" \+ --input-jsonl "${input_jsonl}" \+ --output-dir "${output_dir}" \+ --device "${device}" \+ --batch-size 1diff --git a/funasr/bin/train.py b/funasr/bin/train.pyindex c56d04752..5878bce9b 100644--- a/funasr/bin/train.py+++ b/funasr/bin/train.py@@ -108,6 +108,11 @@ def main(**kwargs):if k.startswith(t + ".") or k == t:logging.info(f"Setting {k}.requires_grad = False")p.requires_grad = False+ lora_only = kwargs.get("lora_only", False)+ if lora_only:+ lora_bias = kwargs.get("lora_bias", "none")+ logging.info("Enable LoRA-only training with bias=%s", lora_bias)+ mark_only_lora_as_trainable(model, bias=lora_bias)if local_rank == 0:logging.info(f"{model_summary(model)}")diff --git a/funasr/bin/train_ds.py b/funasr/bin/train_ds.pyindex 2241b0c52..d2bef0663 100644--- a/funasr/bin/train_ds.py+++ b/funasr/bin/train_ds.py@@ -119,6 +119,11 @@ def main(**kwargs):if k.startswith(t + ".") or k == t:logging.info(f"Setting {k}.requires_grad = False")p.requires_grad = False+ lora_only = kwargs.get("lora_only", False)+ if lora_only:+ lora_bias = kwargs.get("lora_bias", "none")+ logging.info("Enable LoRA-only training with bias=%s", lora_bias)+ mark_only_lora_as_trainable(model, bias=lora_bias)if local_rank == 0:logging.info(f"{model_summary(model)}")diff --git a/runtime/python/websocket/funASR-Service.py b/runtime/python/websocket/funASR-Service.pynew file mode 100644index 000000000..a8d8803b8--- /dev/null+++ b/runtime/python/websocket/funASR-Service.py@@ -0,0 +1,278 @@+'''+功能概述:音频分段摘要+步骤:+1、传入mp3,wav格式音频文件+2、调用funasr生成分段文本内容+3、调用大模型生成摘要+4、输出结果++'''+from flask import Flask, request, jsonify+import re+from modelscope.pipelines import pipeline+from modelscope.utils.constant import Tasks+from werkzeug.utils import secure_filename+import torch+import os+from openai import OpenAI+from funasr import AutoModel++app = Flask(__name__)++# 配置上传文件夹+UPLOAD_FOLDER = 'uploads'+os.makedirs(UPLOAD_FOLDER, exist_ok=True)+app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER++auto_model = AutoModel(model="damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch",+ vad_model="damo/speech_fsmn_vad_zh-cn-16k-common-pytorch",+ punc_model="damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch",+ spk_model="damo/speech_campplus_sv_zh-cn_16k-common",+ device="cuda:0" if torch.cuda.is_available() else "cpu"+ )+# 声纹对比+sv_pipeline = pipeline(+ task='speaker-verification',+ model='damo/speech_campplus_sv_zh-cn_16k-common',+ model_revision='v1.0.0',+ device="cuda:0" if torch.cuda.is_available() else "cpu"+ )++++# 注册声纹:传音频wav文件,实现注册音频返回音频的embedding数据[email protected]('/Register_Speaker', methods=['POST'])+def Register_Speaker():+ # 检查文件上传+ if 'file' not in request.files:+ return jsonify({"error": "No audio file provided"}), 400++ file = request.files['file']+ if file.filename == '':+ return jsonify({"error": "Empty filename"}), 400++ # 保存上传文件+ filename = secure_filename(file.filename)+ filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename)+ file.save(filepath)+ try:+ # 执行音频文件Embeding+ result = sv_pipeline([filepath], output_emb=True)+ embedding = result['embs'][0]+ # 删除临时文件+ os.remove(filepath)+ if len(embedding) == 0:+ return jsonify({+ "status": "error",+ "result": "音频解析结果为空"+ })+ else:+ if isinstance(embedding, np.ndarray):+ embedding_list = embedding.tolist()+ else:+ embedding_list = list(embedding)+ return jsonify({+ "status": "success",+ "result": embedding_list+ })++ except Exception as e:+ # 清理文件+ print(f"错误类型: {type(e).__name__}")+ print(f"错误信息: {str(e)}")+ if os.path.exists(filepath):+ os.remove(filepath)+ return jsonify({"error": str(e)}), 500++# 会议撰写[email protected]('/AsrCamWithIdentify', methods=['POST'])+def speech_recognition_Timestamp_cam_identify_speakers():+ # 检查文件上传+ if 'file' not in request.files:+ return jsonify({"error": "No audio file provided"}), 400++ file = request.files['file']+ if file.filename == '':+ return jsonify({"error": "Empty filename"}), 400++ # 保存上传文件+ filename = secure_filename(file.filename)+ filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename)+ file.save(filepath)+ # 2. 从FormData读取参数并解析JSON+ # 读取identify_speakers(默认False)+ identify_speakers_str = request.form.get('identify_speakers', 'false')+ identify_speakers = json.loads(identify_speakers_str.lower()) # 转为bool++ # 读取speaker_db(默认空字典)+ speaker_db_str = request.form.get('speaker_db', '{}')+ speaker_db = json.loads(speaker_db_str)++ # 3. 验证声纹库(如果需要对比)+ if identify_speakers:+ if not isinstance(speaker_db, dict) or len(speaker_db) == 0:+ return jsonify({+ "status": "error",+ "result": "声纹库为空或格式错误,无法进行对比"+ }), 400+ try:+ # 执行语音识别+ result = auto_model.generate(input=filepath,+ batch_size_s=300,+ hotword='',+ )++ # 处理结果+ processed_result = process_cam_result_with_identify_speakers(result,speaker_db,filepath,identify_speakers)++ os.remove(filepath)+ if len(processed_result) == 0:+ return jsonify({+ "status": "error",+ "result": "音频解析结果为空"+ })+ else:+ return jsonify({+ "status": "success",+ "result": processed_result+ })++ except Exception as e:+ # 清理文件+ print(f"错误类型: {type(e).__name__}")+ print(f"错误信息: {str(e)}")+ if os.path.exists(filepath):+ os.remove(filepath)+ return jsonify({"error": str(e)}), 500++import torchaudio+def _extract_audio_segment(audio_path, start_sec, end_sec):+ """根据时间戳提取音频片段"""+ waveform, sample_rate = torchaudio.load(audio_path)++ # 计算起止采样点+ start_sample = int(start_sec * sample_rate)+ end_sample = int(end_sec * sample_rate)++ # 提取片段+ segment = waveform[:, start_sample:end_sample]++ # 保存为临时文件(pipeline需要文件路径)+ temp_path = f"/tmp/temp_segment_{start_sec}_{end_sec}.wav"+ torchaudio.save(temp_path, segment, sample_rate)++ return temp_path+# 是否使用声纹转化的结果处理+import json+def process_cam_result_with_identify_speakers(result,speaker_db,filepath,identify_speakers=False,threshold=0.45):+ """处理ASR结果,返回包含时间和内容的JSON对象列表"""+ if not isinstance(result, list) or len(result) == 0:+ return []++ data = result[0]+ best_match = "unknown"+ best_score = 0.0+ # 创建JSON格式的输出+ output = []+ sentence_infos = data.get('sentence_info',[])++ current_sentence = {+ "spk": sentence_infos[0]["spk"],+ "spk_name": best_match,+ "confidence":best_score,+ "start": sentence_infos[0]["start"],+ "end": sentence_infos[0]["end"],+ "text": sentence_infos[0]["text"]+ }+++ for i in range(1, len(sentence_infos)):+ sentence_info = sentence_infos[i]+ # 检查合并条件:相同说话人且时间间隔≤1000ms+ if (current_sentence["spk"] == sentence_info["spk"] and+ sentence_info["start"] - current_sentence["end"] <= 1000):++ # 合并文本内容(中文无需加空格)+ current_sentence["text"] += sentence_info["text"]++ # 更新整句结束时间+ current_sentence["end"] = sentence_info["end"]++ else:+ # 保存合并完成的句子+ if identify_speakers: # 提取说话人+ segment_audio = _extract_audio_segment( #提取对应的音频+ filepath, current_sentence['start']/1000, current_sentence['end']/1000+ )+ result_b = sv_pipeline([segment_audio], output_emb=True)['embs'][0] #获取音频向量+ os.remove(segment_audio)+ # 遍历声纹库++ for name, db_emb in speaker_db.items():+ # 计算余弦相似度+ data_list = json.loads(db_emb)+ arr = np.array(data_list, dtype=np.float32)+ similarity = 1 - cosine(result_b, arr)+ similarity = float(similarity)+ if similarity > best_score and similarity > threshold:+ best_score = similarity+ best_match = name+ output.append({+ "spk": current_sentence["spk"],+ "spk_name": best_match,+ "confidence":best_score,+ "text": current_sentence["text"],+ "start": current_sentence["start"],+ "end": current_sentence["end"]+ })+ # 重新开始新句子+ current_sentence = sentence_info.copy()+ best_match = "unknown"+ best_score = 0.0++ # 保存合并完成的句子+ if identify_speakers: # 提取说话人+ segment_audio = _extract_audio_segment( # 提取对应的音频+ filepath, current_sentence['start']/1000, current_sentence['end']/1000+ )+ result_b = sv_pipeline([segment_audio], output_emb=True)['embs'][0] # 获取音频向量+ os.remove(segment_audio)+ # 遍历声纹库++ for name, db_emb in speaker_db.items():+ # 计算余弦相似度+ data_list = json.loads(db_emb)+ arr = np.array(data_list, dtype=np.float32)+ similarity = 1 - cosine(result_b, arr)+ similarity = float(similarity)+ if similarity > best_score and similarity > threshold:+ best_score = similarity+ best_match = name+ output.append({+ "spk": current_sentence["spk"],+ "spk_name": best_match,+ "confidence":best_score,+ "text": current_sentence["text"],+ "start": current_sentence["start"],+ "end": current_sentence["end"]+ })++ return output # 返回JSON对象列表+++from scipy.spatial.distance import cosine+import numpy as np[email protected]('/calculate_similarity', methods=['POST'])+def calculate_similarity():+ """计算两个特征向量的余弦相似度"""+ data = request.get_json(force=True)+ emb1 = np.array(data['emb1'], dtype=np.float32)+ emb2 = np.array(data['emb2'], dtype=np.float32)+ print(1-cosine(emb1,emb2))+ return jsonify({+ "status": "success",+ "result": float(1 - cosine(emb1, emb2))+ })++if __name__ == '__main__':+ app.run(host='0.0.0.0', port=10099, debug=True)diff --git a/runtime/python/websocket/funasr_wss_client.py b/runtime/python/websocket/funasr_wss_client.pyindex b56f64fae..884579d2b 100644--- a/runtime/python/websocket/funasr_wss_client.py+++ b/runtime/python/websocket/funasr_wss_client.py@@ -4,14 +4,11 @@ import timeimport websockets, sslimport asyncio-# import threadingimport argparseimport jsonimport tracebackfrom multiprocessing import Process-# from funasr.fileio.datadir_writer import DatadirWriter-import logginglogging.basicConfig(level=logging.ERROR)@@ -31,14 +28,22 @@ parser.add_argument(default="",help="hotword file path, one hotword perline (e.g.:阿里巴巴 20)",)-parser.add_argument("--audio_in", type=str, default=None, help="audio_in")+parser.add_argument(+ "--audio_in",+ type=str,+ default=None,+ help="音频输入路径;不传则使用麦克风(需安装 PyAudio)",+)parser.add_argument("--audio_fs", type=int, default=16000, help="audio_fs")++# ✅ 修复语义:默认 False;传入参数则不 sleep(用于压测)parser.add_argument("--send_without_sleep",action="store_true",- default=True,- help="if audio_in is set, send_without_sleep",+ default=False,+ help="若设置:发送音频不按实时节奏 sleep(用于压测)",)+parser.add_argument("--thread_num", type=int, default=1, help="thread_num")parser.add_argument("--words_max_print", type=int, default=10000, help="chunk")parser.add_argument("--output_dir", type=str, default=None, help="output_dir")@@ -46,28 +51,83 @@ parser.add_argument("--ssl", type=int, default=1, help="1 for ssl connect, 0 forparser.add_argument("--use_itn", type=int, default=1, help="1 for using itn, 0 for not itn")parser.add_argument("--mode", type=str, default="2pass", help="offline, online, 2pass")+# ✅ 验收日志输出目录(每个 meeting 单独写,避免多进程抢文件)+parser.add_argument("--log_dir", type=str, default="./asr_logs", help="验收日志输出目录")+parser.add_argument("--log_flush_every", type=int, default=1, help="events.jsonl 每写N行flush一次(默认1更安全)")+args = parser.parse_args()args.chunk_size = [int(x) for x in args.chunk_size.split(",")]print(args)-# voices = asyncio.Queue()+from queue import Queue+from datetime import datetimevoices = Queue()offline_msg_done = False-if args.output_dir is not None:- # if os.path.exists(args.output_dir):- # os.remove(args.output_dir)+# === 延迟统计相关:对每个 wav_name 记录首包/末包发送时间 & 是否已经打印过延迟 ===+latency_first_audio_time = {} # {wav_name: t_first_chunk_send}+latency_last_audio_time = {} # {wav_name: t_last_chunk_send}+latency_first_text_printed = {} # {wav_name: bool}+++def _iso(ts: float) -> str:+ return datetime.fromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]+++class MeetingWriter:+ """+ 每个进程/meeting 单独写:+ - events.jsonl:收到的每条服务端消息(在线/离线/2pass)+ - meta.json:本次运行参数(方便复现)+ """+ def __init__(self, log_dir: str, meeting_id: str, flush_every: int = 1):+ self.meeting_id = str(meeting_id)+ self.base = os.path.join(log_dir, f"meeting_{self.meeting_id}")+ os.makedirs(self.base, exist_ok=True)++ self.fp_events = open(os.path.join(self.base, "events.jsonl"), "a", encoding="utf-8")+ self.flush_every = max(1, int(flush_every))+ self._cnt = 0++ meta_path = os.path.join(self.base, "meta.json")+ if not os.path.exists(meta_path):+ with open(meta_path, "w", encoding="utf-8") as f:+ meta = {+ "created_at": _iso(time.time()),+ "meeting_id": self.meeting_id,+ "args": vars(args),+ }+ f.write(json.dumps(meta, ensure_ascii=False, indent=2))++ def write_event(self, obj: dict):+ self.fp_events.write(json.dumps(obj, ensure_ascii=False) + "\n")+ self._cnt += 1+ if self._cnt % self.flush_every == 0:+ self.fp_events.flush()++ def close(self):+ try:+ self.fp_events.flush()+ self.fp_events.close()+ except Exception:+ pass++if args.output_dir is not None:if not os.path.exists(args.output_dir):os.makedirs(args.output_dir)async def record_microphone():- is_finished = False- import pyaudio+ """从麦克风实时录音发送到服务端(一般单路测试使用)"""+ try:+ import pyaudio+ except ImportError as e:+ raise ImportError(+ "缺少 PyAudio,麦克风推流前请先运行 `pip install pyaudio`"+ ) from e- # print("2")global voicesFORMAT = pyaudio.paInt16CHANNELS = 1@@ -96,7 +156,7 @@ async def record_microphone():fst_dict[" ".join(words[:-1])] = int(words[-1])except ValueError:print("Please checkout format of hotwords")- hotword_msg = json.dumps(fst_dict)+ hotword_msg = json.dumps(fst_dict, ensure_ascii=False)else:hotword_msg = args.hotword@@ -115,21 +175,19 @@ async def record_microphone():"is_speaking": True,"hotwords": hotword_msg,"itn": use_itn,- }+ },+ ensure_ascii=False,)- # voices.put(message)await websocket.send(message)while True:data = stream.read(CHUNK)- message = data- # voices.put(message)- await websocket.send(message)- await asyncio.sleep(0.005)+ await websocket.send(data)+ await asyncio.sleep(0.01)async def record_from_scp(chunk_begin, chunk_size):- global voices- is_finished = False+ """从 wav/scp 文件读取音频分片发送,用于压测和延迟测试"""+ global voices, latency_first_audio_time, latency_last_audio_timeif args.audio_in.endswith(".scp"):f_scp = open(args.audio_in)wavs = f_scp.readlines()@@ -137,25 +195,25 @@ async def record_from_scp(chunk_begin, chunk_size):wavs = [args.audio_in]# hotwords- fst_dict = {}hotword_msg = ""if args.hotword.strip() != "":if os.path.exists(args.hotword):- f_scp = open(args.hotword)- hot_lines = f_scp.readlines()+ with open(args.hotword, encoding="utf-8") as f_scp:+ hot_lines = f_scp.readlines()++ hot_list = []for line in hot_lines:- words = line.strip().split(" ")- if len(words) < 2:- print("Please checkout format of hotwords")+ words = line.strip().split()+ if not words:continue- try:- fst_dict[" ".join(words[:-1])] = int(words[-1])- except ValueError:- print("Please checkout format of hotwords")- hotword_msg = json.dumps(fst_dict)+ # Python AutoModel: 用逗号分隔多个热词+ hot_list.append(words[0])++ hotword_msg = ",".join(hot_list)else:hotword_msg = args.hotword- print(hotword_msg)++ print("hotword", hotword_msg)sample_rate = args.audio_fswav_format = "pcm"@@ -164,7 +222,8 @@ async def record_from_scp(chunk_begin, chunk_size):use_itn = Falseif chunk_size > 0:- wavs = wavs[chunk_begin : chunk_begin + chunk_size]+ wavs = wavs[chunk_begin: chunk_begin + chunk_size]+for wav in wavs:wav_splits = wav.strip().split()@@ -172,14 +231,13 @@ async def record_from_scp(chunk_begin, chunk_size):wav_path = wav_splits[1] if len(wav_splits) > 1 else wav_splits[0]if not len(wav_path.strip()) > 0:continue+if wav_path.endswith(".pcm"):with open(wav_path, "rb") as f:audio_bytes = f.read()elif wav_path.endswith(".wav"):import wave-with wave.open(wav_path, "rb") as wav_file:- params = wav_file.getparams()sample_rate = wav_file.getframerate()frames = wav_file.readframes(wav_file.getnframes())audio_bytes = bytes(frames)@@ -190,9 +248,8 @@ async def record_from_scp(chunk_begin, chunk_size):stride = int(60 * args.chunk_size[1] / args.chunk_interval / 1000 * sample_rate * 2)chunk_num = (len(audio_bytes) - 1) // stride + 1- # print(stride)- # send first time+ # send first control messagemessage = json.dumps({"mode": args.mode,@@ -206,36 +263,46 @@ async def record_from_scp(chunk_begin, chunk_size):"is_speaking": True,"hotwords": hotword_msg,"itn": use_itn,- }+ },+ ensure_ascii=False,)- # voices.put(message)await websocket.send(message)is_speaking = True- for i in range(chunk_num):+ # 初始化该 wav 的统计状态+ latency_first_audio_time[wav_name] = None+ latency_last_audio_time[wav_name] = None+ latency_first_text_printed[wav_name] = False++ for i in range(chunk_num):beg = i * stride- data = audio_bytes[beg : beg + stride]- message = data- # voices.put(message)- await websocket.send(message)+ data = audio_bytes[beg: beg + stride]++ now_ts = time.time()+ if latency_first_audio_time[wav_name] is None:+ latency_first_audio_time[wav_name] = now_ts+ latency_last_audio_time[wav_name] = now_ts++ await websocket.send(data)+if i == chunk_num - 1:is_speaking = False- message = json.dumps({"is_speaking": is_speaking})- # voices.put(message)- await websocket.send(message)-- sleep_duration = (- 0.001- if args.mode == "offline"- else 60 * args.chunk_size[1] / args.chunk_interval / 1000- )+ await websocket.send(json.dumps({"is_speaking": is_speaking}, ensure_ascii=False))+ # ✅ sleep策略:默认按实时节奏;若开启 send_without_sleep 则几乎不 sleep(压测)+ if args.send_without_sleep:+ sleep_duration = 0.001+ else:+ sleep_duration = (+ 0.001+ if args.mode == "offline"+ else 60 * args.chunk_size[1] / args.chunk_interval / 1000+ )await asyncio.sleep(sleep_duration)if not args.mode == "offline":await asyncio.sleep(2)- # offline model need to wait for message recvedif args.mode == "offline":global offline_msg_done@@ -247,30 +314,87 @@ async def record_from_scp(chunk_begin, chunk_size):await websocket.close()-async def message(id):+async def message(id, writer: MeetingWriter):+ """接收服务端识别结果 + 打印实时文本 + 打印延迟 + 写验收日志(events.jsonl)"""+ import websocketsglobal websocket, voices, offline_msg_done+ global latency_first_audio_time, latency_last_audio_time, latency_first_text_printed++ multi_mode = args.thread_num > 1 # 多路并发时,打印风格更简洁text_print = ""text_print_2pass_online = ""text_print_2pass_offline = ""+if args.output_dir is not None:ibest_writer = open(os.path.join(args.output_dir, "text.{}".format(id)), "a", encoding="utf-8")else:ibest_writer = None+try:while True:-meg = await websocket.recv()meg = json.loads(meg)+wav_name = meg.get("wav_name", "demo")- text = meg["text"]- timestamp = ""+ text = meg.get("text", "")+ mode = meg.get("mode", "")+ spk_name = meg.get("spk_name", "")+ spk_score = meg.get("spk_score", None)+ now_ts = time.time()++ # === 延迟统计:仅在首条 online/2pass-online 文本时计算并打印一次 ===+ latency_last_ms = None+ latency_first_ms = None+ if text and mode in ("online", "2pass-online"):+ if not latency_first_text_printed.get(wav_name, False):+ t_last = latency_last_audio_time.get(wav_name, None)+ t_first = latency_first_audio_time.get(wav_name, None)+ latency_last_ms = (now_ts - t_last) * 1000.0 if t_last is not None else None+ latency_first_ms = (now_ts - t_first) * 1000.0 if t_first is not None else None++ latency_first_text_printed[wav_name] = True++ if multi_mode:+ parts = [f"[MEETING {id}][LATENCY] wav={wav_name}, mode={mode}"]+ if latency_last_ms is not None:+ parts.append(f"from_last_chunk={latency_last_ms:.1f} ms")+ if latency_first_ms is not None:+ parts.append(f"from_first_chunk={latency_first_ms:.1f} ms")+ print(" ".join(parts))+ else:+ print(+ f"[LATENCY] wav={wav_name}, mode={mode}, "+ f"from_last_chunk={(latency_last_ms or 0):.1f} ms, "+ f"from_first_chunk={(latency_first_ms or 0):.1f} ms"+ )++ timestamp = meg.get("timestamp", "")offline_msg_done = meg.get("is_final", False)- if "timestamp" in meg:- timestamp = meg["timestamp"]- if ibest_writer is not None:+ # ✅ 验收友好:每条消息落 events.jsonl(便于后处理)+ event = {+ "ts": _iso(now_ts),+ "recv_ts": now_ts,+ "meeting_id": str(id),+ "wav_name": wav_name,+ "mode": mode,+ "is_final": bool(meg.get("is_final", False)),+ "text": text,+ "spk_name": spk_name,+ "spk_score": spk_score,+ "latency_first_ms": latency_first_ms,+ "latency_last_ms": latency_last_ms,+ "server_timestamp": meg.get("timestamp", None),+ "sentence_info": meg.get("sentence_info", None),+ "punc_array": meg.get("punc_array", None),+ }+ if writer is not None:+ writer.write_event(event)++ # 保存到 output_dir(保留你原来的逻辑)+ if ibest_writer is not None and text:if timestamp != "":text_write_line = "{}\t{}\t{}\n".format(wav_name, text, timestamp)else:@@ -279,22 +403,44 @@ async def message(id):if "mode" not in meg:continue++ # ===== 多路并发输出风格:只打印精简行 =====+ if multi_mode:+ if mode in ("offline", "2pass-offline") and text:+ spk_name2 = meg.get("spk_name", "unknown")+ spk_score2 = meg.get("spk_score", 0.0)+ print(+ f"[MEETING {id}][FINAL][{wav_name}] "+ f"spk={spk_name2}({float(spk_score2):.3f}) text=\"{text}\""+ )+ if timestamp:+ print(f"[MEETING {id}][TIMESTAMP][{wav_name}] {timestamp}")+ continue++ # ===== 单路模式输出:保留原滚动体验 =====if meg["mode"] == "online":text_print += "{}".format(text)- text_print = text_print[-args.words_max_print :]- os.system("clear")- print("\rpid" + str(id) + ": " + text_print)+ text_print = text_print[-args.words_max_print:]+ print("pid" + str(id) + ": " + text_print)+elif meg["mode"] == "offline":if timestamp != "":text_print += "{} timestamp: {}".format(text, timestamp)else:text_print += "{}".format(text)- # text_print = text_print[-args.words_max_print:]- # os.system('clear')- print("\rpid" + str(id) + ": " + wav_name + ": " + text_print)+ spk_info = ""+ if spk_name:+ if spk_score is not None:+ spk_info = f" [spk={spk_name} score={float(spk_score):.3f}]"+ else:+ spk_info = f" [spk={spk_name}]"++ print("pid" + str(id) + ": " + wav_name + ": " + text_print + spk_info)offline_msg_done = True+else:+ # 2pass 模式if meg["mode"] == "2pass-online":text_print_2pass_online += "{}".format(text)text_print = text_print_2pass_offline + text_print_2pass_online@@ -302,15 +448,27 @@ async def message(id):text_print_2pass_online = ""text_print = text_print_2pass_offline + "{}".format(text)text_print_2pass_offline += "{}".format(text)- text_print = text_print[-args.words_max_print :]- os.system("clear")- print("\rpid" + str(id) + ": " + text_print)- # offline_msg_done=True+ if spk_name:+ if spk_score is not None:+ text_print += f" [spk={spk_name} score={float(spk_score):.3f}]"+ else:+ text_print += f" [spk={spk_name}]"++ text_print = text_print[-args.words_max_print:]+ print("pid" + str(id) + ": " + text_print)++ except websockets.exceptions.ConnectionClosedOK:+ print(f"[MEETING {id}] connection closed normally")except Exception as e:- print("Exception:", e)- # traceback.print_exc()- # await websocket.close()+ print(f"[MEETING {id}] Exception:", e)+ finally:+ try:+ if ibest_writer is not None:+ ibest_writer.flush()+ ibest_writer.close()+ except Exception:+ passasync def ws_client(id, chunk_begin, chunk_size):@@ -322,30 +480,39 @@ async def ws_client(id, chunk_begin, chunk_size):for i in range(chunk_begin, chunk_begin + chunk_size):offline_msg_done = Falsevoices = Queue()+if args.ssl == 1:- ssl_context = ssl.SSLContext()+ ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)ssl_context.check_hostname = Falsessl_context.verify_mode = ssl.CERT_NONEuri = "wss://{}:{}".format(args.host, args.port)else:uri = "ws://{}:{}".format(args.host, args.port)ssl_context = None+print("connect to", uri)+async with websockets.connect(uri, subprotocols=["binary"], ping_interval=None, ssl=ssl_context) as websocket:- if args.audio_in is not None:- task = asyncio.create_task(record_from_scp(i, 1))- else:- task = asyncio.create_task(record_microphone())- task3 = asyncio.create_task(message(str(id) + "_" + str(i))) # processid+fileid- await asyncio.gather(task, task3)- exit(0)+ meeting_tag = f"{id}_{i}"+ writer = MeetingWriter(args.log_dir, meeting_id=meeting_tag, flush_every=args.log_flush_every)+ try:+ if args.audio_in is not None:+ task = asyncio.create_task(record_from_scp(i, 1))+ else:+ task = asyncio.create_task(record_microphone())+ task3 = asyncio.create_task(message(str(id) + "_" + str(i), writer)) # processid+fileid+ await asyncio.gather(task, task3)+ finally:+ writer.close()++ returndef one_thread(id, chunk_begin, chunk_size):- asyncio.get_event_loop().run_until_complete(ws_client(id, chunk_begin, chunk_size))- asyncio.get_event_loop().run_forever()+ # ✅ 子进程里用 asyncio.run 更稳+ asyncio.run(ws_client(id, chunk_begin, chunk_size))if __name__ == "__main__":@@ -356,17 +523,12 @@ if __name__ == "__main__":p.join()print("end")else:- # calculate the number of wavs for each preocess+ # calculate the number of wavs for each processif args.audio_in.endswith(".scp"):f_scp = open(args.audio_in)wavs = f_scp.readlines()else:wavs = [args.audio_in]- for wav in wavs:- wav_splits = wav.strip().split()- wav_name = wav_splits[0] if len(wav_splits) > 1 else "demo"- wav_path = wav_splits[1] if len(wav_splits) > 1 else wav_splits[0]- audio_type = os.path.splitext(wav_path)[-1].lower()total_len = len(wavs)if total_len >= args.thread_num:@@ -383,7 +545,7 @@ if __name__ == "__main__":if remain_wavs > 0:now_chunk_size = chunk_size + 1remain_wavs = remain_wavs - 1- # process i handle wavs at chunk_begin and size of now_chunk_size+p = Process(target=one_thread, args=(i, chunk_begin, now_chunk_size))chunk_begin = chunk_begin + now_chunk_sizep.start()diff --git a/runtime/python/websocket/funasr_wss_server.py b/runtime/python/websocket/funasr_wss_server.pyindex 1f957a92e..ae5cad2fb 100644--- a/runtime/python/websocket/funasr_wss_server.py+++ b/runtime/python/websocket/funasr_wss_server.py@@ -2,25 +2,54 @@ import asyncioimport jsonimport websocketsimport time-import logging-import tracemallocimport numpy as npimport argparseimport ssl+import os+import wave+import functools+from concurrent.futures import ThreadPoolExecutor+from scipy.spatial.distance import cosine++import torch # 保留不影响+++def to_python(obj):+ """递归地把 numpy / torch 等类型转成纯 Python,可 JSON 序列化。"""+ try:+ import numpy as np # noqa+ import torch # noqa+ except Exception:+ np = None+ torch = None++ if np is not None and isinstance(obj, np.generic):+ return obj.item()+ if np is not None and isinstance(obj, np.ndarray):+ return obj.tolist()+ if torch is not None and isinstance(obj, torch.Tensor):+ return obj.cpu().tolist()++ if isinstance(obj, dict):+ return {k: to_python(v) for k, v in obj.items()}+ if isinstance(obj, (list, tuple)):+ return [to_python(v) for v in obj]++ return objparser = argparse.ArgumentParser()-parser.add_argument(- "--host", type=str, default="0.0.0.0", required=False, help="host ip, localhost, 0.0.0.0"-)+parser.add_argument("--host", type=str, default="0.0.0.0", required=False, help="host ip")parser.add_argument("--port", type=int, default=10095, required=False, help="grpc server port")+parser.add_argument("--asr_model",type=str,- default="iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch",+ default="iic/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404",help="model from modelscope",)parser.add_argument("--asr_model_revision", type=str, default="v2.0.4", help="")+parser.add_argument("--asr_model_online",type=str,@@ -28,6 +57,7 @@ parser.add_argument(help="model from modelscope",)parser.add_argument("--asr_model_online_revision", type=str, default="v2.0.4", help="")+parser.add_argument("--vad_model",type=str,@@ -35,6 +65,7 @@ parser.add_argument(help="model from modelscope",)parser.add_argument("--vad_model_revision", type=str, default="v2.0.4", help="")+parser.add_argument("--punc_model",type=str,@@ -42,9 +73,11 @@ parser.add_argument(help="model from modelscope",)parser.add_argument("--punc_model_revision", type=str, default="v2.0.4", help="")+parser.add_argument("--ngpu", type=int, default=1, help="0 for cpu, 1 for gpu")parser.add_argument("--device", type=str, default="cuda", help="cuda, cpu")parser.add_argument("--ncpu", type=int, default=4, help="cpu cores")+parser.add_argument("--certfile",type=str,@@ -52,7 +85,6 @@ parser.add_argument(required=False,help="certfile for ssl",)-parser.add_argument("--keyfile",type=str,@@ -60,25 +92,159 @@ parser.add_argument(required=False,help="keyfile for ssl",)-args = parser.parse_args()+# ====== 保存 2pass 离线阶段送入 ASR 的音频片段(排查 VAD 切分)======+parser.add_argument(+ "--save_offline_segments",+ action="store_true",+ help="Save each offline (2pass) audio segment sent to offline ASR as wav for debugging VAD split.",+)+parser.add_argument(+ "--save_offline_segments_dir",+ type=str,+ default="./offline_segments",+ help="Directory to save offline wav segments when --save_offline_segments is enabled.",+)++# ====== 并发控制:核心新增 ======+parser.add_argument(+ "--worker_threads",+ type=int,+ default=max(4, (os.cpu_count() or 4)),+ help="ThreadPoolExecutor max_workers. Used to offload blocking inference so event loop won't be blocked.",+)+parser.add_argument("--concurrent_vad", type=int, default=4, help="Max concurrent VAD generate() calls.")+parser.add_argument("--concurrent_asr_online", type=int, default=4, help="Max concurrent streaming ASR generate() calls.")+parser.add_argument("--concurrent_asr_offline", type=int, default=2, help="Max concurrent offline ASR generate() calls.")+parser.add_argument("--concurrent_punc", type=int, default=1, help="Max concurrent punctuation generate() calls.")+parser.add_argument("--concurrent_sv", type=int, default=1, help="Max concurrent speaker verification generate() calls.")+parser.add_argument(+ "--speaker_db_reload_sec",+ type=int,+ default=5,+ help="Reload speaker_db.json at most once every N seconds (avoid frequent disk IO).",+)++args = parser.parse_args()websocket_users = set()+SPEAKER_DB_PATH = os.path.join(os.path.dirname(__file__), "speaker_db.json")+++def _ensure_dir(p: str):+ try:+ os.makedirs(p, exist_ok=True)+ except Exception:+ pass+++def _pcm_duration_ms(pcm_bytes: bytes, fs: int, ch: int = 1, sampwidth: int = 2) -> int:+ """根据 fs/ch/sampwidth 计算 PCM 时长,避免写死 16k -> 32 bytes/ms。"""+ if not pcm_bytes:+ return 0+ bytes_per_ms = (fs * ch * sampwidth) / 1000.0+ if bytes_per_ms <= 0:+ return 0+ return int(len(pcm_bytes) / bytes_per_ms)+++def _safe_int(v, default):+ try:+ return int(v)+ except Exception:+ return default+++# ========= speaker db:加缓存,避免每段都读盘 =========+_SPEAKER_DB_CACHE = {}+_SPEAKER_DB_CACHE_TS = 0.0+++def _load_speaker_db_sync():+ if not os.path.exists(SPEAKER_DB_PATH):+ return {}+ try:+ with open(SPEAKER_DB_PATH, "r", encoding="utf-8") as f:+ data = json.load(f)+ return data if isinstance(data, dict) else {}+ except Exception:+ return {}+++def get_speaker_db_cached(now_ts: float, reload_sec: int):+ global _SPEAKER_DB_CACHE, _SPEAKER_DB_CACHE_TS+ if (now_ts - _SPEAKER_DB_CACHE_TS) >= max(1, int(reload_sec)):+ _SPEAKER_DB_CACHE = _load_speaker_db_sync()+ _SPEAKER_DB_CACHE_TS = now_ts+ return _SPEAKER_DB_CACHE or {}+++def _save_wav_sync(out_path: str, audio_bytes: bytes, fs: int, ch: int, sampwidth: int):+ with wave.open(out_path, "wb") as wf:+ wf.setnchannels(ch)+ wf.setsampwidth(sampwidth)+ wf.setframerate(fs)+ wf.writeframes(audio_bytes)+++def save_offline_wav_segment_sync(websocket, audio_bytes: bytes, reason: str = "offline"):+ """+ 保存离线阶段送入 ASR 的音频片段,方便人工试听排查 VAD 切分是否正确。+ 约定:audio_bytes 为 单声道 PCM16 little-endian(默认 16k)。+ (注意:这是同步函数,外层会放线程池执行)+ """+ if not getattr(websocket, "save_offline_segments", False):+ return+ if "2pass" not in (getattr(websocket, "mode", "") or ""):+ return+ if not audio_bytes:+ return++ fs = int(getattr(websocket, "audio_fs", 16000) or 16000)+ ch = 1+ sampwidth = 2 # int16++ # int16 对齐+ if len(audio_bytes) % 2 == 1:+ audio_bytes = audio_bytes[:-1]+ if not audio_bytes:+ return++ seg_idx = int(getattr(websocket, "offline_seg_idx", 0))+ websocket.offline_seg_idx = seg_idx + 1++ duration_ms = _pcm_duration_ms(audio_bytes, fs=fs, ch=ch, sampwidth=sampwidth)++ base_dir = getattr(websocket, "offline_save_dir", args.save_offline_segments_dir)+ _ensure_dir(base_dir)++ wav_name = (getattr(websocket, "wav_name", "microphone") or "microphone").replace("/", "_")+ ts = int(time.time() * 1000)+ fname = f"{wav_name}_{ts}_seg{seg_idx:04d}_{reason}_{duration_ms}ms.wav"+ out_path = os.path.join(base_dir, fname)++ try:+ _save_wav_sync(out_path, audio_bytes, fs=fs, ch=ch, sampwidth=sampwidth)+ print(f"[SAVE_OFFLINE_SEG] {out_path} ({duration_ms} ms, {len(audio_bytes)} bytes)")+ except Exception as e:+ print(f"[SAVE_OFFLINE_SEG] failed: {e}")+print("model loading")-from funasr import AutoModel+from funasr import AutoModel # noqa-# asr+# ====== 离线 ASR ======model_asr = AutoModel(- model=args.asr_model,- model_revision=args.asr_model_revision,+ model="paraformer-zh",+ model_revision="v2.0.4",ngpu=args.ngpu,ncpu=args.ncpu,device=args.device,disable_pbar=True,disable_log=True,)-# asr++# streaming asrmodel_asr_streaming = AutoModel(model=args.asr_model_online,model_revision=args.asr_model_online_revision,@@ -88,6 +254,7 @@ model_asr_streaming = AutoModel(disable_pbar=True,disable_log=True,)+# vadmodel_vad = AutoModel(model=args.vad_model,@@ -97,9 +264,9 @@ model_vad = AutoModel(device=args.device,disable_pbar=True,disable_log=True,- # chunk_size=60,)+# puncif args.punc_model != "":model_punc = AutoModel(model=args.punc_model,@@ -113,8 +280,45 @@ if args.punc_model != "":else:model_punc = None+# sv+model_sv = AutoModel(+ model="iic/speech_campplus_sv_zh-cn_16k-common",+ ngpu=args.ngpu,+ device=args.device,+ disable_pbar=True,+ disable_log=True,+)++print("model loaded! (now supports multi-client with non-blocking inference)")+-print("model loaded! only support one client at the same time now!!!!")+# ====== 线程池 + 并发阈值(核心)======+EXECUTOR = ThreadPoolExecutor(max_workers=int(args.worker_threads))++SEM_VAD = asyncio.Semaphore(max(1, int(args.concurrent_vad)))+SEM_ASR_ONLINE = asyncio.Semaphore(max(1, int(args.concurrent_asr_online)))+SEM_ASR_OFFLINE = asyncio.Semaphore(max(1, int(args.concurrent_asr_offline)))+SEM_PUNC = asyncio.Semaphore(max(1, int(args.concurrent_punc)))+SEM_SV = asyncio.Semaphore(max(1, int(args.concurrent_sv)))+SEM_WAV = asyncio.Semaphore(max(1, 4)) # 保存 wav 一般不需要太大+++async def run_blocking(fn, *a, sem: asyncio.Semaphore | None = None, **kw):+ """+ 把阻塞函数丢线程池执行,避免卡 event loop。+ sem 用于限流(避免 GPU / 模型被打爆)。+ """+ loop = asyncio.get_running_loop()+ call = functools.partial(fn, *a, **kw)+ if sem is None:+ return await loop.run_in_executor(EXECUTOR, call)+ async with sem:+ return await loop.run_in_executor(EXECUTOR, call)+++def _generate_sync(model, audio_or_text, status_dict):+ # 注意:status_dict 里包含 cache,会被 generate 更新+ return model.generate(input=audio_or_text, **status_dict)async def ws_reset(websocket):@@ -130,130 +334,212 @@ async def ws_reset(websocket):async def clear_websocket():- for websocket in websocket_users:+ for websocket in list(websocket_users):await ws_reset(websocket)websocket_users.clear()-async def ws_serve(websocket, path):+async def ws_serve(websocket, path=None):+ # websockets 新版本不会传 path,这里做兼容+ if path is None:+ path = getattr(websocket, "path", None)frames = []frames_asr = []frames_asr_online = []global websocket_users- # await clear_websocket()websocket_users.add(websocket)- websocket.status_dict_asr = {}++ websocket.status_dict_asr = {} # hotword 等websocket.status_dict_asr_online = {"cache": {}, "is_final": False}websocket.status_dict_vad = {"cache": {}, "is_final": False}websocket.status_dict_punc = {"cache": {}}+websocket.chunk_interval = 10websocket.vad_pre_idx = 0speech_start = Falsespeech_end_i = -1+websocket.wav_name = "microphone"websocket.mode = "2pass"+ websocket.is_speaking = True # ✅ 默认初始化,避免 AttributeError++ # 保存离线片段+ websocket.audio_fs = 16000+ websocket.offline_seg_idx = 0+ websocket.save_offline_segments = bool(args.save_offline_segments)+ websocket.offline_save_dir = args.save_offline_segments_dir+ if websocket.save_offline_segments:+ _ensure_dir(websocket.offline_save_dir)+ print(f"[SAVE_OFFLINE_SEG] enabled, dir={websocket.offline_save_dir}")+print("new user connected", flush=True)try:async for message in websocket:+ # ========== 1) 先处理“文本配置消息” ==========if isinstance(message, str):- messagejson = json.loads(message)+ try:+ messagejson = json.loads(message)+ except Exception as e:+ print("bad json message:", e, message[:200])+ continue++ print("=============messagejson============", messagejson)if "is_speaking" in messagejson:- websocket.is_speaking = messagejson["is_speaking"]- websocket.status_dict_asr_online["is_final"] = not websocket.is_speaking+ websocket.is_speaking = bool(messagejson["is_speaking"])+ websocket.status_dict_asr_online["is_final"] = (not websocket.is_speaking)+if "chunk_interval" in messagejson:- websocket.chunk_interval = messagejson["chunk_interval"]+ websocket.chunk_interval = _safe_int(+ messagejson["chunk_interval"], websocket.chunk_interval+ )+if "wav_name" in messagejson:- websocket.wav_name = messagejson.get("wav_name")+ websocket.wav_name = messagejson.get("wav_name") or websocket.wav_name+if "chunk_size" in messagejson:chunk_size = messagejson["chunk_size"]if isinstance(chunk_size, str):- chunk_size = chunk_size.split(",")+ chunk_size = [x.strip() for x in chunk_size.split(",") if x.strip()]websocket.status_dict_asr_online["chunk_size"] = [int(x) for x in chunk_size]+if "encoder_chunk_look_back" in messagejson:websocket.status_dict_asr_online["encoder_chunk_look_back"] = messagejson["encoder_chunk_look_back"]+if "decoder_chunk_look_back" in messagejson:websocket.status_dict_asr_online["decoder_chunk_look_back"] = messagejson["decoder_chunk_look_back"]+if "hotwords" in messagejson:- websocket.status_dict_asr["hotword"] = messagejson["hotwords"]+ hotword_data = messagejson["hotwords"]+ websocket.status_dict_asr["hotword"] = hotword_data+ websocket.status_dict_asr_online["hotword"] = hotword_data+ print(f"热词已更新: {hotword_data}")+if "mode" in messagejson:- websocket.mode = messagejson["mode"]+ websocket.mode = messagejson["mode"] or websocket.mode- websocket.status_dict_vad["chunk_size"] = int(- websocket.status_dict_asr_online["chunk_size"][1] * 60 / websocket.chunk_interval- )- if len(frames_asr_online) > 0 or len(frames_asr) >= 0 or not isinstance(message, str):- if not isinstance(message, str):- frames.append(message)- duration_ms = len(message) // 32- websocket.vad_pre_idx += duration_ms-- # asr online- frames_asr_online.append(message)- websocket.status_dict_asr_online["is_final"] = speech_end_i != -1- if (- len(frames_asr_online) % websocket.chunk_interval == 0- or websocket.status_dict_asr_online["is_final"]- ):- if websocket.mode == "2pass" or websocket.mode == "online":- audio_in = b"".join(frames_asr_online)- try:- await async_asr_online(websocket, audio_in)- except:- print(f"error in asr streaming, {websocket.status_dict_asr_online}")- frames_asr_online = []- if speech_start:- frames_asr.append(message)- # vad online+ if "audio_fs" in messagejson:+ websocket.audio_fs = _safe_int(messagejson["audio_fs"], 16000)++ continue++ # ========== 2) 处理“二进制音频消息” ==========+ if "chunk_size" not in websocket.status_dict_asr_online:+ print("[WARN] chunk_size not set yet, skip audio frame (send config first).")+ continue++ try:+ websocket.status_dict_vad["chunk_size"] = int(+ websocket.status_dict_asr_online["chunk_size"][1] * 60 / websocket.chunk_interval+ )+ except Exception as e:+ print("[WARN] set vad chunk_size failed:", e)+ continue++ pcm = message+ frames.append(pcm)++ duration_ms = _pcm_duration_ms(pcm, fs=websocket.audio_fs, ch=1, sampwidth=2)+ websocket.vad_pre_idx += duration_ms++ # online asr+ frames_asr_online.append(pcm)+ websocket.status_dict_asr_online["is_final"] = (speech_end_i != -1)++ if (len(frames_asr_online) % websocket.chunk_interval == 0) or websocket.status_dict_asr_online["is_final"]:+ if websocket.mode in ("2pass", "online"):+ audio_in = b"".join(frames_asr_online)try:- speech_start_i, speech_end_i = await async_vad(websocket, message)- except:- print("error in vad")- if speech_start_i != -1:- speech_start = True- beg_bias = (websocket.vad_pre_idx - speech_start_i) // duration_ms- frames_pre = frames[-beg_bias:]- frames_asr = []- frames_asr.extend(frames_pre)- # asr punc offline- if speech_end_i != -1 or not websocket.is_speaking:- # print("vad end point")- if websocket.mode == "2pass" or websocket.mode == "offline":- audio_in = b"".join(frames_asr)+ await async_asr_online(websocket, audio_in)+ except Exception:+ print(f"error in asr streaming, {websocket.status_dict_asr_online}")+ frames_asr_online = []++ if speech_start:+ frames_asr.append(pcm)++ # vad online+ try:+ speech_start_i, speech_end_i = await async_vad(websocket, pcm)+ except Exception as e:+ print("error in vad:", e)+ speech_start_i, speech_end_i = -1, -1++ if speech_start_i != -1:+ speech_start = True+ if duration_ms > 0:+ beg_bias = (websocket.vad_pre_idx - speech_start_i) // duration_ms+ else:+ beg_bias = 0+ frames_pre = frames[-beg_bias:] if beg_bias > 0 else []+ frames_asr = []+ frames_asr.extend(frames_pre)++ # ========== 3) 2pass:离线阶段触发点 ==========+ if (speech_end_i != -1) or (not websocket.is_speaking):+ if websocket.mode in ("2pass", "offline"):+ audio_in = b"".join(frames_asr)+ reason = "vad_end" if speech_end_i != -1 else "not_speaking"++ # 保存 wav:放线程池,避免磁盘 IO 卡 loop+ if websocket.save_offline_segments and audio_in:try:- await async_asr(websocket, audio_in)- except:- print("error in asr offline")- frames_asr = []- speech_start = False- frames_asr_online = []- websocket.status_dict_asr_online["cache"] = {}- if not websocket.is_speaking:- websocket.vad_pre_idx = 0- frames = []- websocket.status_dict_vad["cache"] = {}- else:- frames = frames[-20:]+ await run_blocking(+ save_offline_wav_segment_sync,+ websocket,+ audio_in,+ reason,+ sem=SEM_WAV,+ )+ except Exception as e:+ print("[SAVE_OFFLINE_SEG] async failed:", e)++ try:+ await async_asr(websocket, audio_in)+ except Exception as e:+ print("error in asr offline:", e)++ frames_asr = []+ speech_start = False+ frames_asr_online = []+ websocket.status_dict_asr_online["cache"] = {}++ if not websocket.is_speaking:+ websocket.vad_pre_idx = 0+ frames = []+ websocket.status_dict_vad["cache"] = {}+ speech_end_i = -1+ else:+ frames = frames[-20:]except websockets.ConnectionClosed:print("ConnectionClosed...", websocket_users, flush=True)await ws_reset(websocket)- websocket_users.remove(websocket)+ if websocket in websocket_users:+ websocket_users.remove(websocket)except websockets.InvalidState:print("InvalidState...")except Exception as e:print("Exception:", e)+ try:+ await ws_reset(websocket)+ except Exception:+ pass+ if websocket in websocket_users:+ websocket_users.remove(websocket)-async def async_vad(websocket, audio_in):+# ===================== 推理:全部改为“线程池 + 限流” =====================- segments_result = model_vad.generate(input=audio_in, **websocket.status_dict_vad)[0]["value"]- # print(segments_result)+async def async_vad(websocket, audio_in: bytes):+ # model_vad.generate 是阻塞的,必须 offload+ out = await run_blocking(_generate_sync, model_vad, audio_in, websocket.status_dict_vad, sem=SEM_VAD)+ segments_result = out[0].get("value", [])speech_start = -1speech_end = -1@@ -267,79 +553,197 @@ async def async_vad(websocket, audio_in):return speech_start, speech_end-async def async_asr(websocket, audio_in):- if len(audio_in) > 0:- # print(len(audio_in))- rec_result = model_asr.generate(input=audio_in, **websocket.status_dict_asr)[0]- # print("offline_asr, ", rec_result)- if model_punc is not None and len(rec_result["text"]) > 0:- # print("offline, before punc", rec_result, "cache", websocket.status_dict_punc)- rec_result = model_punc.generate(- input=rec_result["text"], **websocket.status_dict_punc- )[0]- # print("offline, after punc", rec_result)- if len(rec_result["text"]) > 0:- # print("offline", rec_result)- mode = "2pass-offline" if "2pass" in websocket.mode else websocket.mode- message = json.dumps(- {- "mode": mode,- "text": rec_result["text"],- "wav_name": websocket.wav_name,- "is_final": websocket.is_speaking,- }- )- await websocket.send(message)+def _sv_and_match_sync(audio_in: bytes, reload_sec: int):+ """+ 同步执行:SV embedding + speaker_db 匹配+ 返回 (spk_name, best_score)+ """+ spk_name = "unknown"+ best_score = 0.0++ sv_out = model_sv.generate(input=audio_in, embedding=True)[0]+ embedding = sv_out["spk_embedding"][0].cpu().numpy()++ now_ts = time.time()+ local_speaker_db = get_speaker_db_cached(now_ts, reload_sec=reload_sec)+ if local_speaker_db:+ for name, ref_embedding in local_speaker_db.items():+ if ref_embedding is None:+ continue+ arr = np.array(ref_embedding, dtype=np.float32)+ similarity = 1.0 - cosine(embedding, arr)+ print("sv similarity with {}: {}".format(name, similarity))+ if similarity > best_score and similarity > 0.2:+ best_score = similarity+ spk_name = name++ return spk_name, float(best_score)+++async def async_asr(websocket, audio_in: bytes):+ mode = "2pass-offline" if "2pass" in (websocket.mode or "") else websocket.mode++ if len(audio_in) <= 0:+ message = {+ "mode": mode,+ "text": "",+ "wav_name": websocket.wav_name,+ "is_final": True,+ }+ await websocket.send(json.dumps(message, ensure_ascii=False))+ return++ # 1) ASR(阻塞,线程池执行)+ rec_result_list = await run_blocking(+ _generate_sync,+ model_asr,+ audio_in,+ websocket.status_dict_asr,+ sem=SEM_ASR_OFFLINE,+ )+ rec_result = rec_result_list[0]- else:- mode = "2pass-offline" if "2pass" in websocket.mode else websocket.mode- message = json.dumps(- {- "mode": mode,- "text": "",- "wav_name": websocket.wav_name,- "is_final": websocket.is_speaking,- }+ print("offline_asr, raw:", rec_result)+ print("offline_asr, keys:", rec_result.keys())++ text = rec_result.get("text", "")+ timestamp = rec_result.get("timestamp", None)+ sentence_info = rec_result.get("sentence_info", None)++ # 2) 声纹识别(阻塞,线程池执行)+ spk_name = "unknown"+ best_score = 0.0+ try:+ spk_name, best_score = await run_blocking(+ _sv_and_match_sync,+ audio_in,+ int(args.speaker_db_reload_sec),+ sem=SEM_SV,)- await websocket.send(message)--async def async_asr_online(websocket, audio_in):- if len(audio_in) > 0:- # print(websocket.status_dict_asr_online.get("is_final", False))- rec_result = model_asr_streaming.generate(- input=audio_in, **websocket.status_dict_asr_online- )[0]- # print("online, ", rec_result)- if websocket.mode == "2pass" and websocket.status_dict_asr_online.get("is_final", False):- return- # websocket.status_dict_asr_online["cache"] = dict()- if len(rec_result["text"]):- mode = "2pass-online" if "2pass" in websocket.mode else websocket.mode- message = json.dumps(- {- "mode": mode,- "text": rec_result["text"],- "wav_name": websocket.wav_name,- "is_final": websocket.is_speaking,- }+ except Exception as e:+ print(f"声纹识别失败: {e}")++ # 3) 标点(阻塞,线程池执行)+ punc_array = None+ if model_punc is not None and len(text) > 0:+ try:+ # punc 只对文本处理+ punc_out = await run_blocking(+ _generate_sync,+ model_punc,+ text,+ websocket.status_dict_punc,+ sem=SEM_PUNC,)- await websocket.send(message)-+ punc_result = punc_out[0]+ print("offline, after punc", punc_result)++ if "text" in punc_result and punc_result["text"]:+ text = punc_result["text"]+ if "punc_array" in punc_result:+ punc_array = punc_result["punc_array"]+ except Exception as e:+ print("punc failed:", e)++ # 4) 构造最终 message+ if len(text) > 0:+ print("======offline final text:", text)+ message = {+ "mode": mode,+ "spk_name": spk_name,+ "spk_score": float(best_score),+ "text": text,+ "wav_name": websocket.wav_name,+ "is_final": True,+ }+ if timestamp is not None:+ message["timestamp"] = to_python(timestamp)+ if sentence_info is not None:+ message["sentence_info"] = to_python(sentence_info)+ if punc_array is not None:+ message["punc_array"] = to_python(punc_array)++ try:+ await websocket.send(json.dumps(message, ensure_ascii=False))+ except Exception as e:+ print("send json failed:", e)+ print("message types:", {k: type(v) for k, v in message.items()})+ else:+ message = {+ "mode": mode,+ "spk_name": spk_name,+ "spk_score": float(best_score),+ "text": "",+ "wav_name": websocket.wav_name,+ "is_final": True,+ }+ await websocket.send(json.dumps(message, ensure_ascii=False))+++async def async_asr_online(websocket, audio_in: bytes):+ if len(audio_in) <= 0:+ return++ # streaming generate 也是阻塞:线程池执行+ rec_out = await run_blocking(+ _generate_sync,+ model_asr_streaming,+ audio_in,+ websocket.status_dict_asr_online,+ sem=SEM_ASR_ONLINE,+ )+ rec_result = rec_out[0]+ print("online, ", rec_result)++ # 2pass:online 只要 partial,不发 final(final 交给 offline)+ if websocket.mode == "2pass" and websocket.status_dict_asr_online.get("is_final", False):+ return++ if rec_result.get("text"):+ mode = "2pass-online" if "2pass" in (websocket.mode or "") else websocket.mode+ message = {+ "mode": mode,+ "text": rec_result["text"],+ "wav_name": websocket.wav_name,+ "is_final": bool(+ websocket.status_dict_asr_online.get("is_final", False) or (not websocket.is_speaking)+ ),+ }+ await websocket.send(json.dumps(message, ensure_ascii=False))+++# ===================== 启动服务 =====================++async def main():+ if len(args.certfile) > 0:+ ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)+ ssl_context.load_cert_chain(args.certfile, keyfile=args.keyfile)+ server = await websockets.serve(+ ws_serve,+ args.host,+ args.port,+ subprotocols=["binary"],+ ping_interval=None,+ ssl=ssl_context,+ )+ else:+ server = await websockets.serve(+ ws_serve,+ args.host,+ args.port,+ subprotocols=["binary"],+ ping_interval=None,+ )-if len(args.certfile) > 0:- ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)+ print(f"WS server started at ws(s)://{args.host}:{args.port}")+ await server.wait_closed()- # Generate with Lets Encrypt, copied to this location, chown to current user and 400 permissions- ssl_cert = args.certfile- ssl_key = args.keyfile- ssl_context.load_cert_chain(ssl_cert, keyfile=ssl_key)- start_server = websockets.serve(- ws_serve, args.host, args.port, subprotocols=["binary"], ping_interval=None, ssl=ssl_context- )-else:- start_server = websockets.serve(- ws_serve, args.host, args.port, subprotocols=["binary"], ping_interval=None- )-asyncio.get_event_loop().run_until_complete(start_server)-asyncio.get_event_loop().run_forever()+if __name__ == "__main__":+ try:+ asyncio.run(main())+ finally:+ try:+ EXECUTOR.shutdown(wait=False, cancel_futures=True)+ except Exception:+ pass