Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
Python asraudiochineseemotion-recognitionfunasrmcp-servermultilingual-asropenai-compatible-apiparaformerpunctuationpytorchreal-time-asrspeaker-diarizationspeech-recognitionspeech-to-textstreaming-asrtranscriptionvllmvoice-activity-detectionwhisper-alternative
trainer
游雁24aea85b5bc3
diff --git a/README.md b/README.mdindex a4511b972..e7ff1b10e 100644--- a/README.md+++ b/README.md@@ -97,7 +97,7 @@ FunASR has open-sourced a large number of pre-trained models on industrial data.<a name="quick-start"></a>## Quick Start-Below is a quick start tutorial. Test audio files ([Mandarin](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav), [English]()).+Below is a quick start tutorial. Test audio files ([Mandarin](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav), [English](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav)).### Command-line usagediff --git a/README_zh.md b/README_zh.mdindex 3e58aa77a..46af926ee 100644--- a/README_zh.md+++ b/README_zh.md@@ -91,7 +91,7 @@ FunASR开源了大量在工业数据上预训练模型,您可以在[模型许<a name="快速开始"></a>## 快速开始-下面为快速上手教程,测试音频([中文](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav),[英文]())+下面为快速上手教程,测试音频([中文](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav),[英文](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav))### 可执行命令行diff --git a/data/list/train.jsonl b/data/list/train.jsonlnew file mode 100644index 000000000..618af3880--- /dev/null+++ b/data/list/train.jsonl@@ -0,0 +1,4 @@+{"key": "BAC009S0764W0121", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav", "source_len": 90, "target": "甚至出现交易几乎停滞的情况", "target_len": 13}+{"key": "BAC009S0916W0489", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav", "source_len": 90, "target": "湖北一公司以员工名义贷款数十员工负债千万", "target_len": 20}+{"key": "asr_example_cn_en", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_cn_en.wav", "source_len": 91, "target": "所有只要处理 data 不管你是做 machine learning 做 deep learning 做 data analytics 做 data science 也好 scientist 也好通通都要都做的基本功啊那 again 先先对有一些也许对", "target_len": 19}+{"key": "ID0012W0014", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav", "source_len": 88, "target": "he tried to think how it could be", "target_len": 8}diff --git a/data/list/train_text.txt b/data/list/train_text.txtindex f4d4fe457..2fc2aa54f 100644--- a/data/list/train_text.txt+++ b/data/list/train_text.txt@@ -1,2 +1,4 @@-ID0012W0013 当客户风险承受能力评估依据发生变化时-ID0012W0014 杨涛不得不将工厂关掉\ No newline at end of file+BAC009S0764W0121 甚至出现交易几乎停滞的情况+BAC009S0916W0489 湖北一公司以员工名义贷款数十员工负债千万+asr_example_cn_en 所有只要处理 data 不管你是做 machine learning 做 deep learning 做 data analytics 做 data science 也好 scientist 也好通通都要都做的基本功啊那 again 先先对有一些也许对+ID0012W0014 he tried to think how it could be\ No newline at end of filediff --git a/data/list/train_wav.scp b/data/list/train_wav.scpindex 325a340c6..210f3cb48 100644--- a/data/list/train_wav.scp+++ b/data/list/train_wav.scp@@ -1,2 +1,4 @@-ID0012W0013 /Users/zhifu/funasr_github/test_local/aishell2_dev_ios/wav/D0012/ID0012W0013.wav-ID0012W0014 /Users/zhifu/funasr_github/test_local/aishell2_dev_ios/wav/D0012/ID0012W0014.wav\ No newline at end of file+BAC009S0764W0121 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav+BAC009S0916W0489 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav+asr_example_cn_en https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_cn_en.wav+ID0012W0014 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav\ No newline at end of filediff --git a/data/list/val.jsonl b/data/list/val.jsonlnew file mode 100644index 000000000..8cd75ad2b--- /dev/null+++ b/data/list/val.jsonl@@ -0,0 +1,2 @@+{"key": "ID0012W0013", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav", "source_len": 88, "target": "欢迎大家来体验达摩院推出的语音识别模型", "target_len": 19}+{"key": "ID0012W0014", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav", "source_len": 88, "target": "he tried to think how it could be", "target_len": 8}diff --git a/data/list/val_text.txt b/data/list/val_text.txtindex f4d4fe457..c1aa42d57 100644--- a/data/list/val_text.txt+++ b/data/list/val_text.txt@@ -1,2 +1,2 @@-ID0012W0013 当客户风险承受能力评估依据发生变化时-ID0012W0014 杨涛不得不将工厂关掉\ No newline at end of file+ID0012W0013 欢迎大家来体验达摩院推出的语音识别模型+ID0012W0014 he tried to think how it could be\ No newline at end of filediff --git a/data/list/val_wav.scp b/data/list/val_wav.scpindex 325a340c6..4054ad33b 100644--- a/data/list/val_wav.scp+++ b/data/list/val_wav.scp@@ -1,2 +1,2 @@-ID0012W0013 /Users/zhifu/funasr_github/test_local/aishell2_dev_ios/wav/D0012/ID0012W0013.wav-ID0012W0014 /Users/zhifu/funasr_github/test_local/aishell2_dev_ios/wav/D0012/ID0012W0014.wav\ No newline at end of file+ID0012W0013 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav+ID0012W0014 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav\ No newline at end of filediff --git a/examples/industrial_data_pretraining/contextual_paraformer/finetune_from_local.sh b/examples/industrial_data_pretraining/bicif_paraformer/finetune.shsimilarity index 53%rename from examples/industrial_data_pretraining/contextual_paraformer/finetune_from_local.shrename to examples/industrial_data_pretraining/bicif_paraformer/finetune.shindex 4dbe8552f..885e5c0e2 100644--- a/examples/industrial_data_pretraining/contextual_paraformer/finetune_from_local.sh+++ b/examples/industrial_data_pretraining/bicif_paraformer/finetune.sh@@ -1,22 +1,24 @@# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.# MIT License (https://opensource.org/licenses/MIT)-# method2, finetune from local model+# method1, finetune from model hub-workspace=`pwd`+# which gpu to train or finetune+export CUDA_VISIBLE_DEVICES="0,1"+gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')-echo "current path: ${workspace}" # /xxxx/funasr/examples/industrial_data_pretraining/paraformer+# model_name from model_hub, or model_dir in local path-# download model-local_path_root=${workspace}/modelscope_models-mkdir -p ${local_path_root}-local_path=${local_path_root}/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch-git clone https://www.modelscope.cn/iic/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404.git ${local_path}+## option 1, download model automatically+model_name_or_model_dir="iic/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch"+model_revision="v2.0.4"+## option 2, download model by git+#local_path_root=${workspace}/modelscope_models+#mkdir -p ${local_path_root}/${model_name_or_model_dir}+#git clone https://www.modelscope.cn/${model_name_or_model_dir}.git ${local_path_root}/${model_name_or_model_dir}+#model_name_or_model_dir=${local_path_root}/${model_name_or_model_dir}-# which gpu to train or finetune-export CUDA_VISIBLE_DEVICES="0,1"-gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')# data dir, which contains: train.json, val.jsondata_dir="../../../data/list"@@ -24,29 +26,22 @@ data_dir="../../../data/list"train_data="${data_dir}/train.jsonl"val_data="${data_dir}/val.jsonl"-# generate train.jsonl and val.jsonl from wav.scp and text.txt-python -m funasr.datasets.audio_datasets.scp2jsonl \+scp2jsonl \++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt"]' \++data_type_list='["source", "target"]' \++jsonl_file_out="${train_data}"-python -m funasr.datasets.audio_datasets.scp2jsonl \+scp2jsonl \++scp_file_list='["../../../data/list/val_wav.scp", "../../../data/list/val_text.txt"]' \++data_type_list='["source", "target"]' \++jsonl_file_out="${val_data}"-tokens="${local_path}/tokens.json"-cmvn_file="${local_path}/am.mvn"--# output dir+# exp output diroutput_dir="./outputs"log_file="${output_dir}/log.txt"-config_name="config.yaml"--init_param="${local_path}/model.pt"mkdir -p ${output_dir}echo "log_file: ${log_file}"@@ -55,17 +50,18 @@ torchrun \--nnodes 1 \--nproc_per_node ${gpu_num} \../../../funasr/bin/train.py \---config-path "${local_path}" \---config-name "${config_name}" \+++model="${model_name_or_model_dir}" \+++model_revision="${model_revision}" \++train_data_set_list="${train_data}" \++valid_data_set_list="${val_data}" \-++tokenizer_conf.token_list="${tokens}" \-++frontend_conf.cmvn_file="${cmvn_file}" \-++dataset_conf.batch_size=32 \-++dataset_conf.batch_type="example" \+++dataset_conf.batch_size=20000 \+++dataset_conf.batch_type="token" \++dataset_conf.num_workers=4 \-++train_conf.max_epoch=20 \-++optim_conf.lr=0.0002 \+++train_conf.max_epoch=50 \++train_conf.log_interval=1 \-++init_param="${init_param}" \-++output_dir="${output_dir}" &> ${log_file}+++train_conf.resume=false \+++train_conf.validate_interval=2000 \+++train_conf.save_checkpoint_interval=2000 \+++train_conf.keep_nbest_models=20 \+++optim_conf.lr=0.0002 \+++output_dir="${output_dir}" &> ${log_file}\ No newline at end of filediff --git a/examples/industrial_data_pretraining/contextual_paraformer/finetune.sh b/examples/industrial_data_pretraining/contextual_paraformer/finetune.shindex 98cc73ceb..de8896853 100644--- a/examples/industrial_data_pretraining/contextual_paraformer/finetune.sh+++ b/examples/industrial_data_pretraining/contextual_paraformer/finetune.sh@@ -7,21 +7,39 @@export CUDA_VISIBLE_DEVICES="0,1"gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')-# data dir, which contains: train.json, val.json-data_dir="/Users/zhifu/funasr1.0/data/list"+# model_name from model_hub, or model_dir in local path++## option 1, download model automatically+model_name_or_model_dir="iic/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404"+model_revision="v2.0.4"-## generate jsonl from wav.scp and text.txt-#python -m funasr.datasets.audio_datasets.scp2jsonl \-#++scp_file_list='["/Users/zhifu/funasr1.0/test_local/wav.scp", "/Users/zhifu/funasr1.0/test_local/text.txt"]' \-#++data_type_list='["source", "target"]' \-#++jsonl_file_out=/Users/zhifu/funasr1.0/test_local/audio_datasets.jsonl+## option 2, download model by git+#local_path_root=${workspace}/modelscope_models+#mkdir -p ${local_path_root}/${model_name_or_model_dir}+#git clone https://www.modelscope.cn/${model_name_or_model_dir}.git ${local_path_root}/${model_name_or_model_dir}+#model_name_or_model_dir=${local_path_root}/${model_name_or_model_dir}+++# data dir, which contains: train.json, val.json+data_dir="../../../data/list"train_data="${data_dir}/train.jsonl"val_data="${data_dir}/val.jsonl"+# generate train.jsonl and val.jsonl from wav.scp and text.txt+scp2jsonl \+++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt"]' \+++data_type_list='["source", "target"]' \+++jsonl_file_out="${train_data}"++scp2jsonl \+++scp_file_list='["../../../data/list/val_wav.scp", "../../../data/list/val_text.txt"]' \+++data_type_list='["source", "target"]' \+++jsonl_file_out="${val_data}"+# exp output dir-output_dir="/Users/zhifu/exp"+output_dir="./outputs"log_file="${output_dir}/log.txt"@@ -31,14 +49,19 @@ echo "log_file: ${log_file}"torchrun \--nnodes 1 \--nproc_per_node ${gpu_num} \-funasr/bin/train.py \-++model="iic/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404" \-++model_revision="v2.0.5" \+../../../funasr/bin/train.py \+++model="${model_name_or_model_dir}" \+++model_revision="${model_revision}" \++train_data_set_list="${train_data}" \++valid_data_set_list="${val_data}" \-++dataset_conf.batch_size=32 \-++dataset_conf.batch_type="example" \+++dataset_conf.batch_size=20000 \+++dataset_conf.batch_type="token" \++dataset_conf.num_workers=4 \-++train_conf.max_epoch=20 \+++train_conf.max_epoch=50 \+++train_conf.log_interval=1 \+++train_conf.resume=false \+++train_conf.validate_interval=2000 \+++train_conf.save_checkpoint_interval=2000 \+++train_conf.keep_nbest_models=20 \++optim_conf.lr=0.0002 \++output_dir="${output_dir}" &> ${log_file}\ No newline at end of filediff --git a/examples/industrial_data_pretraining/paraformer/finetune.sh b/examples/industrial_data_pretraining/paraformer/finetune.shindex 9fc8bf06d..720925223 100644--- a/examples/industrial_data_pretraining/paraformer/finetune.sh+++ b/examples/industrial_data_pretraining/paraformer/finetune.sh@@ -1,27 +1,44 @@# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.# MIT License (https://opensource.org/licenses/MIT)-# method1, finetune from model hub# which gpu to train or finetuneexport CUDA_VISIBLE_DEVICES="0,1"gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')-# data dir, which contains: train.json, val.json-data_dir="/Users/zhifu/funasr1.0/data/list"+# model_name from model_hub, or model_dir in local path++## option 1, download model automatically+model_name_or_model_dir="iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch"+model_revision="v2.0.4"-## generate jsonl from wav.scp and text.txt-#python -m funasr.datasets.audio_datasets.scp2jsonl \-#++scp_file_list='["/Users/zhifu/funasr1.0/test_local/wav.scp", "/Users/zhifu/funasr1.0/test_local/text.txt"]' \-#++data_type_list='["source", "target"]' \-#++jsonl_file_out=/Users/zhifu/funasr1.0/test_local/audio_datasets.jsonl+## option 2, download model by git+#local_path_root=${workspace}/modelscope_models+#mkdir -p ${local_path_root}/${model_name_or_model_dir}+#git clone https://www.modelscope.cn/${model_name_or_model_dir}.git ${local_path_root}/${model_name_or_model_dir}+#model_name_or_model_dir=${local_path_root}/${model_name_or_model_dir}+++# data dir, which contains: train.json, val.json+data_dir="../../../data/list"train_data="${data_dir}/train.jsonl"val_data="${data_dir}/val.jsonl"+# generate train.jsonl and val.jsonl from wav.scp and text.txt+scp2jsonl \+++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt"]' \+++data_type_list='["source", "target"]' \+++jsonl_file_out="${train_data}"++scp2jsonl \+++scp_file_list='["../../../data/list/val_wav.scp", "../../../data/list/val_text.txt"]' \+++data_type_list='["source", "target"]' \+++jsonl_file_out="${val_data}"+# exp output dir-output_dir="/Users/zhifu/exp"+output_dir="./outputs"log_file="${output_dir}/log.txt"@@ -31,19 +48,19 @@ echo "log_file: ${log_file}"torchrun \--nnodes 1 \--nproc_per_node ${gpu_num} \-funasr/bin/train.py \-++model="iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch" \-++model_revision="v2.0.4" \+../../../funasr/bin/train.py \+++model="${model_name_or_model_dir}" \+++model_revision="${model_revision}" \++train_data_set_list="${train_data}" \++valid_data_set_list="${val_data}" \++dataset_conf.batch_size=20000 \++dataset_conf.batch_type="token" \++dataset_conf.num_workers=4 \++train_conf.max_epoch=50 \-++train_conf.log_interval=10 \+++train_conf.log_interval=1 \++train_conf.resume=false \-++train_conf.validate_interval=15 \-++train_conf.save_checkpoint_interval=15 \-++train_conf.keep_nbest_models=50 \+++train_conf.validate_interval=2000 \+++train_conf.save_checkpoint_interval=2000 \+++train_conf.keep_nbest_models=20 \++optim_conf.lr=0.0002 \++output_dir="${output_dir}" &> ${log_file}\ No newline at end of filediff --git a/examples/industrial_data_pretraining/paraformer/finetune_from_local.sh b/examples/industrial_data_pretraining/paraformer/train_from_local.shsimilarity index 100%rename from examples/industrial_data_pretraining/paraformer/finetune_from_local.shrename to examples/industrial_data_pretraining/paraformer/train_from_local.shdiff --git a/examples/industrial_data_pretraining/paraformer_streaming/demo.py b/examples/industrial_data_pretraining/paraformer_streaming/demo.pyindex 601a531ef..9fcee3a8c 100644--- a/examples/industrial_data_pretraining/paraformer_streaming/demo.py+++ b/examples/industrial_data_pretraining/paraformer_streaming/demo.py@@ -3,13 +3,16 @@# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.# MIT License (https://opensource.org/licenses/MIT)+import os+from funasr import AutoModel-chunk_size = [5, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms-encoder_chunk_look_back = 0 #number of chunks to lookback for encoder self-attention-decoder_chunk_look_back = 0 #number of encoder chunks to lookback for decoder cross-attention-wav_file="/Users/zhifu/Downloads/NCYzUhAtZNI_0015.wav"+chunk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms+encoder_chunk_look_back = 4 #number of chunks to lookback for encoder self-attention+decoder_chunk_look_back = 1 #number of encoder chunks to lookback for decoder cross-attentionmodel = AutoModel(model="iic/speech_paraformer_asr_nat-zh-cn-16k-common-vocab8404-online", model_revision="v2.0.4")++wav_file = os.path.join(model.model_path, "example/asr_example.wav")res = model.generate(input=wav_file,chunk_size=chunk_size,encoder_chunk_look_back=encoder_chunk_look_back,@@ -17,12 +20,11 @@ res = model.generate(input=wav_file,)print(res)-# exit()import soundfile-import os-# wav_file = os.path.join(model.model_path, "example/asr_example.wav")++wav_file = os.path.join(model.model_path, "example/asr_example.wav")speech, sample_rate = soundfile.read(wav_file)chunk_stride = chunk_size[1] * 960 # 600ms、480msdiff --git a/examples/industrial_data_pretraining/seaco_paraformer/finetune_from_local.sh b/examples/industrial_data_pretraining/paraformer_streaming/finetune.shsimilarity index 53%rename from examples/industrial_data_pretraining/seaco_paraformer/finetune_from_local.shrename to examples/industrial_data_pretraining/paraformer_streaming/finetune.shindex 9593671d9..2bfb2bf39 100644--- a/examples/industrial_data_pretraining/seaco_paraformer/finetune_from_local.sh+++ b/examples/industrial_data_pretraining/paraformer_streaming/finetune.sh@@ -1,22 +1,23 @@# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.# MIT License (https://opensource.org/licenses/MIT)-# method2, finetune from local model-workspace=`pwd`+# which gpu to train or finetune+export CUDA_VISIBLE_DEVICES="0,1"+gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')-echo "current path: ${workspace}" # /xxxx/funasr/examples/industrial_data_pretraining/paraformer+# model_name from model_hub, or model_dir in local path-# download model-local_path_root=${workspace}/modelscope_models-mkdir -p ${local_path_root}-local_path=${local_path_root}/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch-git clone https://www.modelscope.cn/iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch.git ${local_path}+## option 1, download model automatically+model_name_or_model_dir="iic/speech_paraformer_asr_nat-zh-cn-16k-common-vocab8404-online"+model_revision="v2.0.4"+## option 2, download model by git+#local_path_root=${workspace}/modelscope_models+#mkdir -p ${local_path_root}/${model_name_or_model_dir}+#git clone https://www.modelscope.cn/${model_name_or_model_dir}.git ${local_path_root}/${model_name_or_model_dir}+#model_name_or_model_dir=${local_path_root}/${model_name_or_model_dir}-# which gpu to train or finetune-export CUDA_VISIBLE_DEVICES="0,1"-gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')# data dir, which contains: train.json, val.jsondata_dir="../../../data/list"@@ -24,29 +25,22 @@ data_dir="../../../data/list"train_data="${data_dir}/train.jsonl"val_data="${data_dir}/val.jsonl"-# generate train.jsonl and val.jsonl from wav.scp and text.txt-python -m funasr.datasets.audio_datasets.scp2jsonl \+scp2jsonl \++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt"]' \++data_type_list='["source", "target"]' \++jsonl_file_out="${train_data}"-python -m funasr.datasets.audio_datasets.scp2jsonl \+scp2jsonl \++scp_file_list='["../../../data/list/val_wav.scp", "../../../data/list/val_text.txt"]' \++data_type_list='["source", "target"]' \++jsonl_file_out="${val_data}"-tokens="${local_path}/tokens.json"-cmvn_file="${local_path}/am.mvn"--# output dir+# exp output diroutput_dir="./outputs"log_file="${output_dir}/log.txt"-config_name="config.yaml"--init_param="${local_path}/model.pt"mkdir -p ${output_dir}echo "log_file: ${log_file}"@@ -55,17 +49,18 @@ torchrun \--nnodes 1 \--nproc_per_node ${gpu_num} \../../../funasr/bin/train.py \---config-path "${local_path}" \---config-name "${config_name}" \+++model="${model_name_or_model_dir}" \+++model_revision="${model_revision}" \++train_data_set_list="${train_data}" \++valid_data_set_list="${val_data}" \-++tokenizer_conf.token_list="${tokens}" \-++frontend_conf.cmvn_file="${cmvn_file}" \-++dataset_conf.batch_size=32 \-++dataset_conf.batch_type="example" \+++dataset_conf.batch_size=20000 \+++dataset_conf.batch_type="token" \++dataset_conf.num_workers=4 \-++train_conf.max_epoch=20 \-++optim_conf.lr=0.0002 \+++train_conf.max_epoch=50 \++train_conf.log_interval=1 \-++init_param="${init_param}" \-++output_dir="${output_dir}" &> ${log_file}+++train_conf.resume=false \+++train_conf.validate_interval=2000 \+++train_conf.save_checkpoint_interval=2000 \+++train_conf.keep_nbest_models=20 \+++optim_conf.lr=0.0002 \+++output_dir="${output_dir}" &> ${log_file}\ No newline at end of filediff --git a/examples/industrial_data_pretraining/seaco_paraformer/finetune.sh b/examples/industrial_data_pretraining/seaco_paraformer/finetune.shindex 88f0e2314..cfdec7779 100644--- a/examples/industrial_data_pretraining/seaco_paraformer/finetune.sh+++ b/examples/industrial_data_pretraining/seaco_paraformer/finetune.sh@@ -1,27 +1,44 @@# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.# MIT License (https://opensource.org/licenses/MIT)-# method1, finetune from model hub# which gpu to train or finetuneexport CUDA_VISIBLE_DEVICES="0,1"gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')-# data dir, which contains: train.json, val.json-data_dir="/Users/zhifu/funasr1.0/data/list"+# model_name from model_hub, or model_dir in local path++## option 1, download model automatically+model_name_or_model_dir="iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch"+model_revision="v2.0.4"-## generate jsonl from wav.scp and text.txt-#python -m funasr.datasets.audio_datasets.scp2jsonl \-#++scp_file_list='["/Users/zhifu/funasr1.0/test_local/wav.scp", "/Users/zhifu/funasr1.0/test_local/text.txt"]' \-#++data_type_list='["source", "target"]' \-#++jsonl_file_out=/Users/zhifu/funasr1.0/test_local/audio_datasets.jsonl+## option 2, download model by git+#local_path_root=${workspace}/modelscope_models+#mkdir -p ${local_path_root}/${model_name_or_model_dir}+#git clone https://www.modelscope.cn/${model_name_or_model_dir}.git ${local_path_root}/${model_name_or_model_dir}+#model_name_or_model_dir=${local_path_root}/${model_name_or_model_dir}+++# data dir, which contains: train.json, val.json+data_dir="../../../data/list"train_data="${data_dir}/train.jsonl"val_data="${data_dir}/val.jsonl"+# generate train.jsonl and val.jsonl from wav.scp and text.txt+scp2jsonl \+++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt"]' \+++data_type_list='["source", "target"]' \+++jsonl_file_out="${train_data}"++scp2jsonl \+++scp_file_list='["../../../data/list/val_wav.scp", "../../../data/list/val_text.txt"]' \+++data_type_list='["source", "target"]' \+++jsonl_file_out="${val_data}"+# exp output dir-output_dir="/Users/zhifu/exp"+output_dir="./outputs"log_file="${output_dir}/log.txt"@@ -31,14 +48,19 @@ echo "log_file: ${log_file}"torchrun \--nnodes 1 \--nproc_per_node ${gpu_num} \-funasr/bin/train.py \-++model="iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch" \-++model_revision="v2.0.6" \+../../../funasr/bin/train.py \+++model="${model_name_or_model_dir}" \+++model_revision="${model_revision}" \++train_data_set_list="${train_data}" \++valid_data_set_list="${val_data}" \-++dataset_conf.batch_size=32 \-++dataset_conf.batch_type="example" \+++dataset_conf.batch_size=20000 \+++dataset_conf.batch_type="token" \++dataset_conf.num_workers=4 \-++train_conf.max_epoch=20 \+++train_conf.max_epoch=50 \+++train_conf.log_interval=1 \+++train_conf.resume=false \+++train_conf.validate_interval=2000 \+++train_conf.save_checkpoint_interval=2000 \+++train_conf.keep_nbest_models=20 \++optim_conf.lr=0.0002 \++output_dir="${output_dir}" &> ${log_file}\ No newline at end of filediff --git a/funasr/datasets/audio_datasets/scp2jsonl.py b/funasr/datasets/audio_datasets/scp2jsonl.pyindex e09a84a61..00754ddb6 100644--- a/funasr/datasets/audio_datasets/scp2jsonl.py+++ b/funasr/datasets/audio_datasets/scp2jsonl.py@@ -29,15 +29,19 @@ def gen_jsonl_from_wav_text_list(path, data_type_list=("source", "target"), jsondata_file_lists = f.readlines()lines_for_each_th = (len(data_file_lists)-1)//cpu_cores + 1task_num = cpu_cores if len(data_file_lists) > cpu_cores else 1- with concurrent.futures.ThreadPoolExecutor(max_workers=cpu_cores) as executor:-- futures = [executor.submit(parse_context_length, data_file_lists[i*lines_for_each_th:(i+1)*lines_for_each_th], data_type) for i in range(task_num)]+ # import pdb;pdb.set_trace()+ if task_num > 1:+ with concurrent.futures.ThreadPoolExecutor(max_workers=cpu_cores) as executor:- for future in concurrent.futures.as_completed(futures):-- json_dict[data_type].update(future.result())- # print(json_dict)+ futures = [executor.submit(parse_context_length, data_file_lists[i*lines_for_each_th:(i+1)*lines_for_each_th], data_type) for i in range(task_num)]+ for future in concurrent.futures.as_completed(futures):++ json_dict[data_type].update(future.result())+ else:+ res = parse_context_length(data_file_lists, data_type)+ json_dict[data_type].update(res)+with open(jsonl_file_out, "w") as f:for key in json_dict[data_type_list[0]].keys():jsonl_line = {"key": key}@@ -46,6 +50,7 @@ def gen_jsonl_from_wav_text_list(path, data_type_list=("source", "target"), jsonjsonl_line = json.dumps(jsonl_line, ensure_ascii=False)f.write(jsonl_line+"\n")f.flush()+ print(f"processed {len(json_dict[data_type_list[0]])} samples")else:passdiff --git a/setup.py b/setup.pyindex 5cb56658f..9a69fcd9d 100644--- a/setup.py+++ b/setup.py@@ -140,6 +140,9 @@ setup(],entry_points={"console_scripts": ["funasr = funasr.bin.inference:main_hydra",+ "funasr-train = funasr.bin.train:main_hydra","funasr-export = funasr.bin.export:main_hydra",+ "scp2jsonl = funasr.datasets.audio_datasets.scp2jsonl:main_hydra",+ "jsonl2scp = funasr.datasets.audio_datasets.jsonl2scp:main_hydra",]},)