Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

fun-asr-nano

游雁c42f2540cc46
diff --git a/README.md b/README.md
index d9b11b427..925691fd1 100644
--- a/README.md
+++ b/README.md
@@ -107,23 +107,24 @@ FunASR has open-sourced a large number of pre-trained models on industrial data.
(Note: ⭐ represents the ModelScope model zoo, 🤗 represents the Huggingface model zoo, 🍀 represents the OpenAI model zoo)
-| Model Name | Task Details | Training Data | Parameters |
-| :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------: | :------------------------------: | :--------: |
-| SenseVoiceSmall <br> ([⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) ) | multiple speech understanding capabilities, including ASR, ITN, LID, SER, and AED, support languages such as zh, yue, en, ja, ko | 300000 hours | 234M |
-| paraformer-zh <br> ([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) ) | speech recognition, with timestamps, non-streaming | 60000 hours, Mandarin | 220M |
-| <nobr>paraformer-zh-streaming <br> ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗](https://huggingface.co/funasr/paraformer-zh-streaming) )</nobr> | speech recognition, streaming | 60000 hours, Mandarin | 220M |
-| paraformer-en <br> ( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗](https://huggingface.co/funasr/paraformer-en) ) | speech recognition, without timestamps, non-streaming | 50000 hours, English | 220M |
-| conformer-en <br> ( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗](https://huggingface.co/funasr/conformer-en) ) | speech recognition, non-streaming | 50000 hours, English | 220M |
-| ct-punc <br> ( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗](https://huggingface.co/funasr/ct-punc) ) | punctuation restoration | 100M, Mandarin and English | 290M |
-| fsmn-vad <br> ( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) ) | voice activity detection | 5000 hours, Mandarin and English | 0.4M |
-| fsmn-kws <br> ( [⭐](https://modelscope.cn/models/iic/speech_charctc_kws_phone-xiaoyun/summary) ) | keyword spotting,streaming | 5000 hours, Mandarin | 0.7M |
-| fa-zh <br> ( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗](https://huggingface.co/funasr/fa-zh) ) | timestamp prediction | 5000 hours, Mandarin | 38M |
-| cam++ <br> ( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) ) | speaker verification/diarization | 5000 hours | 7.2M |
-| Whisper-large-v3 <br> ([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3/summary) [🍀](https://github.com/openai/whisper) ) | speech recognition, with timestamps, non-streaming | multilingual | 1550 M |
-| Whisper-large-v3-turbo <br> ([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3-turbo/summary) [🍀](https://github.com/openai/whisper) ) | speech recognition, with timestamps, non-streaming | multilingual | 809 M |
-| Qwen-Audio <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio) ) | audio-text multimodal models (pretraining) | multilingual | 8B |
-| Qwen-Audio-Chat <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo_chat.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio-Chat) ) | audio-text multimodal models (chat) | multilingual | 8B |
-| emotion2vec+large <br> ([⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) ) | speech emotion recongintion | 40000 hours | 300M |
+| Model Name | Task Details | Training Data | Parameters |
+|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:--------------------------------:| :--------: |
+| Fun-ASR-Nano <br> ([⭐](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) [🤗](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) ) |Speech recognition supports Chinese, English, and Japanese. Chinese includes support for 7 dialects and 26 regional accents. English and Japanese cover multiple regional accents. Additional features include lyric recognition and rap speech recognition. | Tens of millions of hours | 800M |
+| SenseVoiceSmall <br> ([⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) ) | multiple speech understanding capabilities, including ASR, ITN, LID, SER, and AED, support languages such as zh, yue, en, ja, ko | 300000 hours | 234M |
+| paraformer-zh <br> ([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) ) | speech recognition, with timestamps, non-streaming | 60000 hours, Mandarin | 220M |
+| <nobr>paraformer-zh-streaming <br> ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗](https://huggingface.co/funasr/paraformer-zh-streaming) )</nobr> | speech recognition, streaming | 60000 hours, Mandarin | 220M |
+| paraformer-en <br> ( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗](https://huggingface.co/funasr/paraformer-en) ) | speech recognition, without timestamps, non-streaming | 50000 hours, English | 220M |
+| conformer-en <br> ( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗](https://huggingface.co/funasr/conformer-en) ) | speech recognition, non-streaming | 50000 hours, English | 220M |
+| ct-punc <br> ( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗](https://huggingface.co/funasr/ct-punc) ) | punctuation restoration | 100M, Mandarin and English | 290M |
+| fsmn-vad <br> ( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) ) | voice activity detection | 5000 hours, Mandarin and English | 0.4M |
+| fsmn-kws <br> ( [⭐](https://modelscope.cn/models/iic/speech_charctc_kws_phone-xiaoyun/summary) ) | keyword spotting,streaming | 5000 hours, Mandarin | 0.7M |
+| fa-zh <br> ( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗](https://huggingface.co/funasr/fa-zh) ) | timestamp prediction | 5000 hours, Mandarin | 38M |
+| cam++ <br> ( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) ) | speaker verification/diarization | 5000 hours | 7.2M |
+| Whisper-large-v3 <br> ([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3/summary) [🍀](https://github.com/openai/whisper) ) | speech recognition, with timestamps, non-streaming | multilingual | 1550 M |
+| Whisper-large-v3-turbo <br> ([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3-turbo/summary) [🍀](https://github.com/openai/whisper) ) | speech recognition, with timestamps, non-streaming | multilingual | 809 M |
+| Qwen-Audio <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio) ) | audio-text multimodal models (pretraining) | multilingual | 8B |
+| Qwen-Audio-Chat <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo_chat.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio-Chat) ) | audio-text multimodal models (chat) | multilingual | 8B |
+| emotion2vec+large <br> ([⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) ) | speech emotion recongintion | 40000 hours | 300M |
[//]: #
[//]: # "FunASR supports pre-trained or further fine-tuned models for deployment as a service. The CPU version of the Chinese offline file conversion service has been released, details can be found in [docs](funasr/runtime/docs/SDK_tutorial.md). More detailed information about service deployment can be found in the [deployment roadmap](funasr/runtime/readme_cn.md)."
@@ -144,6 +145,32 @@ Notes: Support recognition of single audio file, as well as file list in Kaldi-s
### Speech Recognition (Non-streaming)
+#### Fun-ASR-Nano
+
+```python
+from funasr import AutoModel
+
+model_dir = "FunAudioLLM/Fun-ASR-Nano-2512"
+
+model = AutoModel(
+ model=model_dir,
+ vad_model="fsmn-vad",
+ vad_kwargs={"max_single_segment_time": 30000},
+ device="cuda:0",
+)
+res = model.generate(input=[wav_path], cache={}, batch_size_s=0)
+text = res[0]["text"]
+print(text)
+```
+
+Parameter Description:
+
+- `model_dir`: The name of the model, or the path to the model on the local disk.
+- `vad_model`: This indicates the activation of VAD (Voice Activity Detection). The purpose of VAD is to split long audio into shorter clips. In this case, the inference time includes both VAD and SenseVoice total consumption, and represents the end-to-end latency. If you wish to test the SenseVoice model's inference time separately, the VAD model can be disabled.
+- `vad_kwargs`: Specifies the configurations for the VAD model. `max_single_segment_time`: denotes the maximum duration for audio segmentation by the `vad_model`, with the unit being milliseconds (ms).
+- `use_itn`: Whether the output result includes punctuation and inverse text normalization.
+- `batch_size_s`: Indicates the use of dynamic batching, where the total duration of audio in the batch is measured in seconds (s).
+
#### SenseVoice
```python
diff --git a/README_zh.md b/README_zh.md
index ecdba3ea0..e1e8fe37b 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -107,23 +107,24 @@ FunASR 开源了大量在工业数据上预训练模型,您可以在[模型许
(注:⭐ 表示 ModelScope 模型仓库,🤗 表示 Huggingface 模型仓库,🍀 表示 OpenAI 模型仓库)
-| 模型名字 | 任务详情 | 训练数据 | 参数量 |
-| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------: | :----------------------: | :----: |
-| SenseVoiceSmall <br> ([⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) ) | 多种语音理解能力,涵盖了自动语音识别(ASR)、语言识别(LID)、情感识别(SER)以及音频事件检测(AED) | 400000 小时,中文 | 330M |
-| paraformer-zh <br> ([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) ) | 语音识别,带时间戳输出,非实时 | 60000 小时,中文 | 220M |
-| paraformer-zh-streaming <br> ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗](https://huggingface.co/funasr/paraformer-zh-streaming) ) | 语音识别,实时 | 60000 小时,中文 | 220M |
-| paraformer-en <br> ( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗](https://huggingface.co/funasr/paraformer-en) ) | 语音识别,非实时 | 50000 小时,英文 | 220M |
-| conformer-en <br> ( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗](https://huggingface.co/funasr/conformer-en) ) | 语音识别,非实时 | 50000 小时,英文 | 220M |
-| ct-punc <br> ( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗](https://huggingface.co/funasr/ct-punc) ) | 标点恢复 | 100M,中文与英文 | 290M |
-| fsmn-vad <br> ( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) ) | 语音端点检测,实时 | 5000 小时,中文与英文 | 0.4M |
-| fsmn-kws <br> ( [⭐](https://modelscope.cn/models/iic/speech_charctc_kws_phone-xiaoyun/summary) ) | 语音唤醒,实时 | 5000 小时,中文 | 0.7M |
-| fa-zh <br> ( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗](https://huggingface.co/funasr/fa-zh) ) | 字级别时间戳预测 | 50000 小时,中文 | 38M |
-| cam++ <br> ( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) ) | 说话人确认/分割 | 5000 小时 | 7.2M |
-| Whisper-large-v3 <br> ([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3/summary) [🍀](https://github.com/openai/whisper) ) | 语音识别,带时间戳输出,非实时 | 多语言 | 1550 M |
-| Whisper-large-v3-turbo <br> ([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3-turbo/summary) [🍀](https://github.com/openai/whisper) ) | 语音识别,带时间戳输出,非实时 | 多语言 | 809 M |
-| Qwen-Audio <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio) ) | 音频文本多模态大模型(预训练) | 多语言 | 8B |
-| Qwen-Audio-Chat <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo_chat.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio-Chat) ) | 音频文本多模态大模型(chat 版本) | 多语言 | 8B |
-| emotion2vec+large <br> ([⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) ) | 情感识别模型 | 40000 小时,4 种情感类别 | 300M |
+| 模型名字 | 任务详情 | 训练数据 | 参数量 |
+|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------:|:----------------:|:------:|
+| Fun-ASR-Nano <br> ([⭐](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) [🤗](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) ) | 语音识别,支持中文、英文与日语,其中中文支持7个方言,26个地方口音,英文与日语覆盖多地区口音,歌词识别,说唱等 | 数千万小时 | 800M |
+| SenseVoiceSmall <br> ([⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) ) | 多种语音理解能力,涵盖了自动语音识别(ASR)、语言识别(LID)、情感识别(SER)以及音频事件检测(AED) | 400000 小时,中文 | 330M |
+| paraformer-zh <br> ([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) ) | 语音识别,带时间戳输出,非实时 | 60000 小时,中文 | 220M |
+| paraformer-zh-streaming <br> ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗](https://huggingface.co/funasr/paraformer-zh-streaming) ) | 语音识别,实时 | 60000 小时,中文 | 220M |
+| paraformer-en <br> ( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗](https://huggingface.co/funasr/paraformer-en) ) | 语音识别,非实时 | 50000 小时,英文 | 220M |
+| conformer-en <br> ( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗](https://huggingface.co/funasr/conformer-en) ) | 语音识别,非实时 | 50000 小时,英文 | 220M |
+| ct-punc <br> ( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗](https://huggingface.co/funasr/ct-punc) ) | 标点恢复 | 100M,中文与英文 | 290M |
+| fsmn-vad <br> ( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) ) | 语音端点检测,实时 | 5000 小时,中文与英文 | 0.4M |
+| fsmn-kws <br> ( [⭐](https://modelscope.cn/models/iic/speech_charctc_kws_phone-xiaoyun/summary) ) | 语音唤醒,实时 | 5000 小时,中文 | 0.7M |
+| fa-zh <br> ( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗](https://huggingface.co/funasr/fa-zh) ) | 字级别时间戳预测 | 50000 小时,中文 | 38M |
+| cam++ <br> ( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) ) | 说话人确认/分割 | 5000 小时 | 7.2M |
+| Whisper-large-v3 <br> ([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3/summary) [🍀](https://github.com/openai/whisper) ) | 语音识别,带时间戳输出,非实时 | 多语言 | 1550 M |
+| Whisper-large-v3-turbo <br> ([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3-turbo/summary) [🍀](https://github.com/openai/whisper) ) | 语音识别,带时间戳输出,非实时 | 多语言 | 809 M |
+| Qwen-Audio <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio) ) | 音频文本多模态大模型(预训练) | 多语言 | 8B |
+| Qwen-Audio-Chat <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo_chat.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio-Chat) ) | 音频文本多模态大模型(chat 版本) | 多语言 | 8B |
+| emotion2vec+large <br> ([⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) ) | 情感识别模型 | 40000 小时,4 种情感类别 | 300M |
<a name="快速开始"></a>
@@ -141,6 +142,31 @@ funasr ++model=paraformer-zh ++vad_model="fsmn-vad" ++punc_model="ct-punc" ++inp
### 非实时语音识别
+#### Fun-ASR-Nano
+
+```python
+from funasr import AutoModel
+
+model_dir = "FunAudioLLM/Fun-ASR-Nano-2512"
+
+model = AutoModel(
+ model=model_dir,
+ vad_model="fsmn-vad",
+ vad_kwargs={"max_single_segment_time": 30000},
+ device="cuda:0",
+)
+res = model.generate(input=[wav_path], cache={}, batch_size_s=0)
+text = res[0]["text"]
+print(text)
+```
+参数说明:
+- `model_dir`:模型名称,或本地磁盘中的模型路径。
+- `vad_model`:表示开启 VAD,VAD 的作用是将长音频切割成短音频,此时推理耗时包括了 VAD 与 SenseVoice 总耗时,为链路耗时,如果需要单独测试 SenseVoice 模型耗时,可以关闭 VAD 模型。
+- `vad_kwargs`:表示 VAD 模型配置,`max_single_segment_time`: 表示`vad_model`最大切割音频时长, 单位是毫秒 ms。
+- `batch_size_s` 表示采用动态 batch,batch 中总音频时长,单位为秒 s。
+
+
+
#### SenseVoice
```python
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/LICENSE b/examples/industrial_data_pretraining/fun_asr_nano/LICENSE
new file mode 100644
index 000000000..261eeb9e9
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/README.md b/examples/industrial_data_pretraining/fun_asr_nano/README.md
new file mode 100644
index 000000000..fa8725371
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/README.md
@@ -0,0 +1,137 @@
+# Fun-ASR
+
+「[简体中文](README_zh.md)」|「English」
+
+Fun-ASR is an end-to-end speech recognition large model launched by Tongyi Lab. It is trained on tens of millions of hours of real speech data, possessing powerful contextual understanding capabilities and industry adaptability. It supports low-latency real-time transcription and covers 31 languages. It excels in vertical domains such as education and finance, accurately recognizing professional terminology and industry expressions, effectively addressing challenges like "hallucination" generation and language confusion, achieving "clear hearing, understanding meaning, and accurate writing."
+
+<div align="center">
+<img src="images/funasr-v2.png">
+</div>
+
+<div align="center">
+<h4>
+<a href="https://funaudiollm.github.io/funasr"> Homepage </a>
+|<a href="#core-features"> Core Features </a>
+|<a href="#performance-evaluation"> Performance Evaluation </a>
+|<a href="#environment-setup"> Environment Setup </a>
+|<a href="#usage-tutorial"> Usage Tutorial </a>
+
+</h4>
+
+Model Repository: [modelscope](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512), [huggingface](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512)
+
+Online Experience:
+[ModelScope Community Space](https://modelscope.cn/studios/FunAudioLLM/Fun-ASR-Nano), [huggingface space](https://huggingface.co/spaces/FunAudioLLM/Fun-ASR-Nano)
+
+</div>
+
+<a name="What's News"></a>
+
+# What's New 🔥
+
+- 2025/12: [Fun-ASR-Nano-2512](https://modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) is an end-to-end speech recognition large model trained on tens of millions of hours real speech data. It supports low-latency real-time transcription and covers 31 languages.
+- 2024/7: [FunASR](https://github.com/modelscope/FunASR) is a fundamental speech recognition toolkit that offers a variety of features, including speech recognition (ASR), Voice Activity Detection (VAD), Punctuation Restoration, Language Models, Speaker Verification, Speaker Diarization and multi-talker ASR.
+
+# Core Features 🎯
+
+**Fun-ASR** focuses on high-precision speech recognition, multi-language support, and industry customization capabilities
+
+- **Far-field High-noise Recognition:** Deeply optimized for far-distance sound pickup and high-noise scenarios (such as conference rooms, in-vehicle environments, industrial sites, etc.), improving recognition accuracy to **93%**.
+- **Chinese Dialects and Regional Accents:**
+ - Supports **7 major dialects**: Wu, Cantonese, Min, Hakka, Gan, Xiang, Jin
+ - Covers **26 regional accents**: including Henan, Shaanxi, Hubei, Sichuan, Chongqing, Yunnan, Guizhou, Guangdong, Guangxi and more than 20 other regions
+- **Multi-language Free Speech:** Supports recognition of **31 languages**, with focused optimization on East and Southeast Asian languages, supporting free language switching and mixed recognition.
+- **Music Background Lyric Recognition:** Enhanced speech recognition performance under music background interference, supporting accurate recognition of lyric content in songs.
+
+# Environment Setup 🐍
+
+```shell
+pip install -r requirements.txt
+```
+
+<a name="usage-tutorial"></a>
+
+# TODO
+
+- [ ] Support returning timestamps
+- [ ] Support speaker diarization
+- [ ] Support model training
+
+# Usage 🛠️
+
+## Inference
+
+### Using funasr for inference
+
+```python
+from funasr import AutoModel
+
+
+def main():
+ model_dir = "FunAudioLLM/Fun-ASR-Nano-2512"
+ model = AutoModel(
+ model=model_dir,
+ trust_remote_code=True,
+ remote_code="./model.py",
+ device="cuda:0",
+ )
+
+ wav_path = f"{model.model_path}/example/zh.mp3"
+ res = model.generate(input=[wav_path], cache={}, batch_size=1)
+ text = res[0]["text"]
+ print(text)
+
+ model = AutoModel(
+ model=model_dir,
+ trust_remote_code=True,
+ vad_model="fsmn-vad",
+ vad_kwargs={"max_single_segment_time": 30000},
+ remote_code="./model.py",
+ device="cuda:0",
+ )
+ res = model.generate(input=[wav_path], cache={}, batch_size=1)
+ text = res[0]["text"]
+ print(text)
+
+
+if __name__ == "__main__":
+ main()
+```
+
+### Direct Inference
+
+```python
+from model import FunASRNano
+
+
+def main():
+ model_dir = "FunAudioLLM/Fun-ASR-Nano-2512"
+ m, kwargs = FunASRNano.from_pretrained(model=model_dir, device="cuda:0")
+ m.eval()
+
+ wav_path = f"{kwargs['model_path']}/example/zh.mp3"
+ res = m.inference(data_in=[wav_path], **kwargs)
+ text = res[0][0]["text"]
+ print(text)
+
+
+if __name__ == "__main__":
+ main()
+```
+
+<details><summary> Parameter Description (click to expand) </summary>
+
+- `model_dir`: Model name or local disk model path.
+- `trust_remote_code`: Whether to trust remote code for loading custom model implementations.
+- `remote_code`: Specify the location of specific model code (e.g., `model.py` in the current directory), supporting both absolute and relative paths.
+- `device`: Specify the device to use, such as "cuda:0" or "cpu".
+
+</details>
+
+# Performance Evaluation 📝
+
+We compared the multi-language speech recognition performance of Fun-ASR with other models on open-source benchmark datasets (including AISHELL-1, AISHELL-2, Wenetspeech, Librispeech, and Common Voice).
+
+<div align="center">
+<img src="images/compare_en.png" width="800" />
+</div>
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/README_zh.md b/examples/industrial_data_pretraining/fun_asr_nano/README_zh.md
new file mode 100644
index 000000000..fbdc34239
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/README_zh.md
@@ -0,0 +1,137 @@
+# Fun-ASR
+
+「简体中文」|「[English](README.md)」
+
+Fun-ASR 是通义实验室推出的端到端语音识别大模型,是基于数千万小时真实语音数据训练而成,具备强大的上下文理解能力与行业适应性,支持低延迟实时听写,并且覆盖 31 个语种。在教育、金融等垂直领域表现出色,能准确识别专业术语与行业表达,有效应对"幻觉"生成和语种混淆等挑战,实现"听得清、懂其意、写得准"。
+
+<div align="center">
+<img src="images/funasr-v2.png">
+</div>
+
+<div align="center">
+<h4>
+<a href="https://funaudiollm.github.io/funasr"> Homepage </a>
+|<a href="#核心特性"> 核心特性 </a>
+|<a href="#性能评测"> 性能评测 </a>
+|<a href="#环境安装"> 环境安装 </a>
+|<a href="#用法教程"> 用法教程 </a>
+
+</h4>
+
+模型仓库:[modelscope](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512),[huggingface](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512)
+
+在线体验:
+[魔搭社区创空间](https://modelscope.cn/studios/FunAudioLLM/Fun-ASR),[huggingface space](https://huggingface.co/spaces/FunAudioLLM/Fun-ASR)
+
+</div>
+
+<a name="最新动态"></a>
+
+# 最新动态 🔥
+
+- 2025/12: [Fun-ASR-Nano-2512](https://modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) 是一款基于数千万小时真实语音数据训练的端到端语音识别大模型。它支持低延迟实时转写,并涵盖 31 种语言识别功能。
+- 2024/7: [FunASR](https://github.com/modelscope/FunASR) 是一款功能全面的语音识别基础工具包,集成了多项核心功能,包括自动语音识别(ASR)、语音活动检测(VAD)、标点恢复、语言模型、说话人验证、说话人日志记录以及多说话人语音识别。
+
+# 核心特性 🎯
+
+**Fun-ASR** 专注于高精度语音识别、多语言支持和行业定制化能力
+
+- **远场高噪声识别:** 针对远距离拾音及高噪声场景(如会议室、车载环境、工业现场等)进行深度优化,识别准确率提升至 **93%**。
+- **中文方言与地方口音:**
+ - 支持 **7 大方言**:吴语、粤语、闽语、客家话、赣语、湘语、晋语
+ - 覆盖 **26 个地区口音**:包括河南、陕西、湖北、四川、重庆、云南、贵州、广东、广西等 20 多个地区
+- **多语言自由说:** 支持 **31 种语言**识别,重点优化东亚与东南亚语种,支持语种自由切换和混合识别。
+- **音乐背景歌词识别:** 强化在音乐背景干扰下的语音识别性能,支持对歌曲中歌词内容的精准识别。
+
+# 环境安装 🐍
+
+```shell
+pip install -r requirements.txt
+```
+
+<a name="用法教程"></a>
+
+# TODO
+
+- [ ] 支持返回时间戳
+- [ ] 支持区分说话人识别
+- [ ] 支持模型训练
+
+# 用法 🛠️
+
+## 推理
+
+### 使用 funasr 推理
+
+```python
+from funasr import AutoModel
+
+
+def main():
+ model_dir = "FunAudioLLM/Fun-ASR-Nano-2512"
+ model = AutoModel(
+ model=model_dir,
+ trust_remote_code=True,
+ remote_code="./model.py",
+ device="cuda:0",
+ )
+
+ wav_path = f"{model.model_path}/example/zh.mp3"
+ res = model.generate(input=[wav_path], cache={}, batch_size=1)
+ text = res[0]["text"]
+ print(text)
+
+ model = AutoModel(
+ model=model_dir,
+ trust_remote_code=True,
+ vad_model="fsmn-vad",
+ vad_kwargs={"max_single_segment_time": 30000},
+ remote_code="./model.py",
+ device="cuda:0",
+ )
+ res = model.generate(input=[wav_path], cache={}, batch_size=1)
+ text = res[0]["text"]
+ print(text)
+
+
+if __name__ == "__main__":
+ main()
+```
+
+### 直接推理
+
+```python
+from model import FunASRNano
+
+
+def main():
+ model_dir = "FunAudioLLM/Fun-ASR-Nano-2512"
+ m, kwargs = FunASRNano.from_pretrained(model=model_dir, device="cuda:0")
+ m.eval()
+
+ wav_path = f"{kwargs['model_path']}/example/zh.mp3"
+ res = m.inference(data_in=[wav_path], **kwargs)
+ text = res[0][0]["text"]
+ print(text)
+
+
+if __name__ == "__main__":
+ main()
+```
+
+<details><summary> 参数说明(点击展开)</summary>
+
+- `model_dir`:模型名称,或本地磁盘中的模型路径。
+- `trust_remote_code`:是否信任远程代码,用于加载自定义模型实现。
+- `remote_code`:指定模型具体代码的位置(例如,当前目录下的 `model.py`),支持绝对路径与相对路径。
+- `device`:指定使用的设备,如 "cuda:0" 或 "cpu"。
+
+</details>
+
+# 性能评测 📝
+
+我们在开源基准数据集、中文方言测试集和工业测试集上,比较了 Fun-ASR 与其他模型的多语言语音识别性能。Fun-ASR 模型均具有明显的效果优势。
+
+<div align="center">
+<img src="images/compare_zh.png" width="800" />
+</div>
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/demo1.py b/examples/industrial_data_pretraining/fun_asr_nano/demo1.py
new file mode 100644
index 000000000..7992b1098
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/demo1.py
@@ -0,0 +1,32 @@
+from funasr import AutoModel
+
+
+def main():
+ model_dir = "FunAudioLLM/Fun-ASR-Nano-2512"
+ model = AutoModel(
+ model=model_dir,
+ trust_remote_code=True,
+ remote_code="./model.py",
+ device="cuda:0",
+ )
+
+ wav_path = f"{model.model_path}/example/zh.mp3"
+ res = model.generate(input=[wav_path], cache={}, batch_size=1)
+ text = res[0]["text"]
+ print(text)
+
+ model = AutoModel(
+ model=model_dir,
+ trust_remote_code=True,
+ vad_model="fsmn-vad",
+ vad_kwargs={"max_single_segment_time": 30000},
+ remote_code="./model.py",
+ device="cuda:0",
+ )
+ res = model.generate(input=[wav_path], cache={}, batch_size=1)
+ text = res[0]["text"]
+ print(text)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/demo2.py b/examples/industrial_data_pretraining/fun_asr_nano/demo2.py
new file mode 100644
index 000000000..f53642939
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/demo2.py
@@ -0,0 +1,16 @@
+from model import FunASRNano
+
+
+def main():
+ model_dir = "FunAudioLLM/Fun-ASR-Nano-2512"
+ m, kwargs = FunASRNano.from_pretrained(model=model_dir, device="cuda:0")
+ m.eval()
+
+ wav_path = f"{kwargs['model_path']}/example/zh.mp3"
+ res = m.inference(data_in=[wav_path], **kwargs)
+ text = res[0][0]["text"]
+ print(text)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/model.py b/examples/industrial_data_pretraining/fun_asr_nano/model.py
new file mode 100644
index 000000000..f0d22a800
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/model.py
@@ -0,0 +1,660 @@
+import json
+import logging
+import os
+import random
+import re
+import string
+import time
+import traceback
+
+import torch
+import torch.nn as nn
+from funasr import AutoModel
+from funasr.metrics.compute_acc import compute_accuracy
+from funasr.register import tables
+from funasr.train_utils.device_funcs import force_gatherable, to_device
+from funasr.utils.datadir_writer import DatadirWriter
+from funasr.utils.load_utils import extract_fbank, load_audio_text_image_video
+from transformers import AutoConfig, AutoModelForCausalLM
+
+dtype_map = {"bf16": torch.bfloat16, "fp16": torch.float16, "fp32": torch.float32}
+
+
[email protected]("model_classes", "FunASRNano")
+class FunASRNano(nn.Module):
+ def __init__(
+ self,
+ audio_encoder: str = None,
+ audio_encoder_conf: dict = None,
+ audio_adaptor: str = None,
+ audio_adaptor_conf: dict = None,
+ llm: str = None,
+ llm_conf: dict = None,
+ input_size: int = 80,
+ length_normalized_loss: bool = False,
+ **kwargs,
+ ):
+ super().__init__()
+
+ # audio encoder
+ hub = audio_encoder_conf.get("hub", None)
+ self.audio_encoder_activation_checkpoint = audio_encoder_conf.get(
+ "activation_checkpoint", False
+ )
+ if hub == "ms":
+ model = AutoModel(model=audio_encoder, model_revision="master")
+ audio_encoder_output_size = (
+ model.model.encoder_output_size
+ if hasattr(model.model, "encoder_output_size")
+ else -1
+ )
+ audio_encoder = (
+ model.model.model.encoder if hasattr(model.model, "model") else model.model.encoder
+ )
+ else:
+ encoder_class = tables.encoder_classes.get(audio_encoder)
+ audio_encoder = encoder_class(input_size=input_size, **audio_encoder_conf)
+ audio_encoder_output_size = audio_encoder.output_size()
+ freeze = audio_encoder_conf.get("freeze", True)
+ freeze_layer_num = int(audio_encoder_conf.get("freeze_layer_num", -1))
+
+ if freeze:
+ for name, param in audio_encoder.named_parameters():
+ param.requires_grad = False
+ audio_encoder.eval()
+ self.audio_encoder = audio_encoder
+ # llm
+ self.llm = None
+ init_param_path = llm_conf.get("init_param_path", None)
+ llm_dim = None
+
+ llm_load_kwargs = llm_conf.get("load_kwargs", {})
+ config = AutoConfig.from_pretrained(init_param_path)
+ model = AutoModelForCausalLM.from_config(config, **llm_load_kwargs)
+
+ freeze = llm_conf.get("freeze", True)
+ if freeze:
+ for name, param in model.named_parameters():
+ param.requires_grad = False
+ model.eval()
+ logging.info(f"use_lora: {llm_conf.get('use_lora', False)}")
+ if llm_conf.get("use_lora", False):
+ from omegaconf import DictConfig, OmegaConf
+
+ lora_conf = llm_conf.get("lora_conf", {})
+ if isinstance(lora_conf, (OmegaConf, DictConfig)):
+ lora_conf = OmegaConf.to_container(lora_conf, resolve=True)
+ from peft import LoraConfig, PeftModel, get_peft_model
+
+ lora_init_param_path = lora_conf.get("init_param_path", None)
+ if lora_init_param_path is not None:
+ logging.info(f"lora_init_param_path: {lora_init_param_path}")
+ model = PeftModel.from_pretrained(model, lora_init_param_path)
+ for name, param in model.named_parameters():
+ if not lora_conf.get("freeze_lora", False):
+ if "lora_" in name:
+ param.requires_grad = True
+ else:
+ peft_config = LoraConfig(**lora_conf)
+ model = get_peft_model(model, peft_config)
+ model.print_trainable_parameters()
+
+ if llm_conf.get("activation_checkpoint", False):
+ model.gradient_checkpointing_enable()
+
+ self.llm_dtype = llm_conf.get("llm_dtype", "fp32")
+ self.llm = model.to(dtype_map[self.llm_dtype])
+ llm_dim = model.get_input_embeddings().weight.shape[-1]
+
+ # adaptor
+ adaptor_class = tables.adaptor_classes.get(audio_adaptor)
+ if audio_encoder_output_size > 0:
+ audio_adaptor_conf["encoder_dim"] = audio_encoder_output_size
+ audio_adaptor_conf["llm_dim"] = (
+ llm_dim if llm_dim is not None else audio_adaptor_conf["llm_dim"]
+ )
+ audio_adaptor = adaptor_class(**audio_adaptor_conf)
+ init_param_path = audio_adaptor_conf.get("init_param_path", None)
+ if init_param_path is not None:
+ src_state = torch.load(init_param_path, map_location="cpu")
+ flag = audio_adaptor.load_state_dict(src_state, strict=False)
+ logging.info(f"Loading audio_adaptor ckpt: {init_param_path}, status: {flag}")
+ freeze = audio_adaptor_conf.get("freeze", False)
+ if freeze:
+ for name, param in audio_adaptor.named_parameters():
+ param.requires_grad = False
+ audio_adaptor.eval()
+ self.audio_adaptor = audio_adaptor
+
+ self.length_normalized_loss = length_normalized_loss
+ self.feat_permute = audio_encoder_conf.get("feat_permute", True)
+ rank = int(os.environ.get("RANK", 0))
+ logging.info(f"rank: {rank}, model is builded.")
+
+ def forward(
+ self,
+ speech: torch.Tensor = None,
+ speech_lengths: torch.Tensor = None,
+ input_ids: torch.Tensor = None,
+ attention_mask: torch.Tensor = None,
+ labels_ids: torch.Tensor = None,
+ fbank_beg: torch.Tensor = None,
+ fbank_mask: torch.Tensor = None,
+ **kwargs,
+ ):
+ batch_size, token_num = input_ids.shape
+ stats = {}
+ input_ids[input_ids < 0] = 0
+ inputs_embeds = self.llm.model.get_input_embeddings()(input_ids)
+ if speech is not None:
+ if len(speech_lengths.size()) > 1:
+ speech_lengths = speech_lengths[:, 0]
+ batch_size_speech, frames, _ = speech.shape
+
+ # audio encoder
+ if self.audio_encoder_activation_checkpoint:
+ from torch.utils.checkpoint import checkpoint
+
+ encoder_out, encoder_out_lens = checkpoint(
+ self.encode, speech, speech_lengths, use_reentrant=False
+ )
+ else:
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ # audio_adaptor
+ encoder_out, encoder_out_lens = self.audio_adaptor(encoder_out, encoder_out_lens)
+
+ batch_size, token_num, dims = inputs_embeds.shape
+ fake_token_len = kwargs.get("fake_token_len")
+ fake_token_len[fake_token_len < 0] = 0
+ fbank_beg[fbank_beg < 0] = 0
+
+ speech_idx = 0
+ for batch_idx in range(batch_size):
+ for turn_id in range(fbank_beg.shape[1]):
+ fbank_beg_idx = fbank_beg[batch_idx, turn_id].item()
+ if fbank_beg_idx > 0:
+ speech_token_len = fake_token_len[batch_idx, turn_id]
+ speech_token = encoder_out[speech_idx, :speech_token_len, :]
+
+ try:
+ inputs_embeds[
+ batch_idx,
+ fbank_beg_idx : fbank_beg_idx + speech_token_len,
+ :,
+ ] = speech_token
+ except Exception as e:
+ logging.error(f"{str(e)}, {traceback.format_exc()}")
+ logging.info(
+ f"batch_idx: {batch_idx}, inputs_embeds: {inputs_embeds.shape}, fbank_beg_idx: {fbank_beg_idx}, speech_token_len: {speech_token_len}, encoder_out: {encoder_out.shape}, encoder_out_lens: {encoder_out_lens}, fake_token_len: {fake_token_len}, speech_lengths: {speech_lengths}"
+ )
+ speech_token_len = encoder_out_lens[speech_idx].item()
+ speech_token = encoder_out[speech_idx, :speech_token_len, :]
+ inputs_embeds[
+ batch_idx,
+ fbank_beg_idx : fbank_beg_idx + speech_token_len,
+ :,
+ ] = speech_token
+
+ speech_idx += 1
+
+ stats["batch_size_speech"] = batch_size_speech
+ stats["batch_size_x_frames"] = frames * batch_size_speech
+ stats["batch_size_real_frames"] = speech_lengths.sum().item()
+ stats["padding_frames"] = stats["batch_size_x_frames"] - stats["batch_size_real_frames"]
+
+ with torch.cuda.amp.autocast(
+ enabled=True if self.llm_dtype != "fp32" else False,
+ dtype=dtype_map[self.llm_dtype],
+ ):
+ labels_ids[labels_ids == -1] = -100
+ attention_mask[attention_mask < 0] = 0
+ model_outputs = self.llm(
+ inputs_embeds=inputs_embeds.to(dtype_map[self.llm_dtype]),
+ attention_mask=attention_mask,
+ labels=labels_ids,
+ )
+ loss = model_outputs.loss
+
+ with torch.no_grad():
+ preds = torch.argmax(model_outputs.logits, -1)
+ acc_att = compute_accuracy(preds[:, :-1], labels_ids[:, 1:], ignore_label=-100)
+ stats["acc"] = acc_att
+
+ stats["loss"] = torch.clone(loss.detach())
+ stats["batch_size"] = batch_size
+
+ stats["batch_size_x_tokens"] = token_num * batch_size
+ stats["batch_size_real_tokens"] = attention_mask.sum().item()
+ stats["padding_tokens"] = stats["batch_size_x_tokens"] - stats["batch_size_real_tokens"]
+
+ dialog_turns = (fbank_beg > 0).sum(-1)
+ dialog_turns_max = torch.max(dialog_turns).int().item()
+ dialog_turns_avg = dialog_turns.sum().item() / batch_size
+ stats["dialog_turns_max"] = dialog_turns_max
+ stats["dialog_turns_avg"] = dialog_turns_avg
+
+ # force_gatherable: to-device and to-tensor if scalar for DataParallel
+ if self.length_normalized_loss:
+ batch_size = int((labels_ids > 0 + 1).sum())
+ loss, stats, weight = force_gatherable((loss, stats, batch_size), loss.device)
+ return loss, stats, weight
+
+ def forward_export(self, speech, speech_lengths, **kwargs):
+ x, olens = self.audio_encoder(speech, speech_lengths)
+ encoder_out, encoder_out_lens = self.audio_adaptor(x, olens)
+ return encoder_out, encoder_out_lens
+
+ def encode(self, speech, speech_lengths):
+ # audio encoder
+ if self.feat_permute:
+ encoder_out, encoder_out_lens = self.audio_encoder(
+ speech.permute(0, 2, 1), speech_lengths
+ )
+ else:
+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)
+
+ return encoder_out, encoder_out_lens
+
+ def data_template(self, data):
+ system, user, assistant = [], [], []
+ for i, item in enumerate(data):
+ role = item["role"]
+ content = item["content"]
+ if role == "system":
+ system.append(content)
+ elif role == "user":
+ if "audio" in item:
+ audio = item["audio"]
+ content = [content, audio]
+ user.append(content)
+ elif role == "assistant":
+ assistant.append(content)
+
+ system = system * len(user)
+
+ contents = {
+ "system": system,
+ "user": user,
+ "assistant": assistant,
+ }
+
+ return contents
+
+ def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):
+ system = contents["system"]
+ user = contents["user"]
+ assistant = contents["assistant"]
+ pattern = re.compile(r"(<\|startofspeech\|>.*?<\|endofspeech\|>)")
+ do_think = True
+ sys_prompt = True
+ if "dataset_conf" in kwargs:
+ do_think = kwargs["dataset_conf"].get("do_think", True)
+ sys_prompt = kwargs["dataset_conf"].get("sys_prompt", True)
+
+ input_ids, labels, fbank, fbank_lens, fbank_mask, fbank_beg, fake_token_len = (
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ )
+ input_source_ids = []
+ for i, (system_prompt, user_prompt, target_out) in enumerate(zip(system, user, assistant)):
+ if i >= kwargs.get("multiturn_num_max", 5):
+ break
+ if len(input_ids) > kwargs.get("max_token_length", 1500):
+ break
+ if isinstance(user_prompt, (list, tuple)):
+ user_prompt, audio = user_prompt
+ if i == 0:
+ if kwargs.get("infer_with_assistant_input", False):
+ source_input = f"<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_prompt}"
+ if not sys_prompt:
+ source_input = f"<|im_start|>user\n{user_prompt}"
+ else:
+ source_input = f"<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"
+ if not sys_prompt:
+ source_input = (
+ f"<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"
+ )
+ else:
+ if kwargs.get("infer_with_assistant_input", False):
+ source_input = f"<|im_start|>user\n{user_prompt}"
+ else:
+ source_input = (
+ f"<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"
+ )
+ if not do_think:
+ source_input += "<think>\n\n</think>\n\n"
+
+ splits = pattern.split(source_input)
+ source_ids = []
+ fbank_mask_i = []
+ fake_token_len_i = 0
+ fbank_beg_i = -1
+ speech, speech_lengths = [], []
+ for k, sub_str in enumerate(splits):
+ if not sub_str.startswith("<|startofspeech|>"):
+ sub_token = tokenizer.encode(sub_str)
+ source_ids += sub_token
+ fbank_mask_i += [0] * len(sub_token)
+ else:
+ sub_str = sub_str.replace("<|startofspeech|>", "").replace(
+ "<|endofspeech|>", ""
+ )
+ if sub_str.startswith("!"):
+ sub_str = sub_str[1:]
+ if sub_str.startswith("!"): # !!: audio sample point
+ sub_str = audio
+ try:
+ time1 = time.perf_counter()
+ data_src = load_audio_text_image_video(
+ sub_str, fs=frontend.fs, **kwargs
+ )
+ time2 = time.perf_counter()
+ meta_data["load_data"] = f"{time2 - time1:0.3f}"
+ except Exception as e:
+ logging.error(f"Loading wav failed! {str(e)}, {traceback.format_exc()}")
+
+ speech, speech_lengths = extract_fbank(
+ data_src,
+ data_type=kwargs.get("data_type", "sound"),
+ frontend=frontend,
+ is_final=True,
+ ) # speech: [b, T, d]
+
+ time3 = time.perf_counter()
+ meta_data["extract_feat"] = f"{time3 - time2:0.3f}"
+ meta_data["batch_data_time"] = (
+ speech_lengths.sum().item()
+ * frontend.frame_shift
+ * frontend.lfr_n
+ / 1000
+ )
+
+ if self.feat_permute:
+ speech = speech.permute(0, 2, 1)
+
+ olens = 1 + (speech_lengths[0].item() - 3 + 2 * 1) // 2
+ olens = 1 + (olens - 3 + 2 * 1) // 2
+ fake_token_len_i = (olens - 1) // 2 + 1
+ fake_token = [0] * fake_token_len_i
+ fbank_beg_i = len(source_ids)
+ source_ids += fake_token
+ fbank_mask_i += [1] * len(fake_token)
+
+ fbank_beg += [fbank_beg_i + len(input_ids)]
+ fake_token_len += [fake_token_len_i]
+ source_mask = [-100] * len(source_ids)
+ target_out = f"{target_out}<|im_end|>"
+ target_ids = tokenizer.encode(target_out)
+ input_source_ids = input_ids + source_ids
+ input_ids += source_ids + target_ids
+ labels += source_mask + target_ids
+ fbank_mask += fbank_mask_i
+ if len(speech) > 0:
+ fbank.append(speech[0, :, :])
+ fbank_lens.append(speech_lengths)
+
+ input_ids = torch.tensor(input_ids, dtype=torch.int64) # [: self.max_token_length]
+ attention_mask = torch.tensor([1] * len(input_ids), dtype=torch.int32)
+ labels = torch.tensor(labels, dtype=torch.int64) # [: self.max_token_length]
+
+ fbank_mask = torch.tensor(fbank_mask, dtype=torch.float32)
+ fbank_beg = torch.tensor(fbank_beg, dtype=torch.int32)
+ fake_token_len = torch.tensor(fake_token_len, dtype=torch.int32)
+ source_ids = torch.tensor(input_source_ids, dtype=torch.int64)
+ target_ids = torch.tensor(target_ids, dtype=torch.int64)
+
+ if len(fbank) > 0:
+ speech = torch.nn.utils.rnn.pad_sequence(fbank, batch_first=True, padding_value=0.0)
+ speech_lengths = torch.nn.utils.rnn.pad_sequence(
+ fbank_lens, batch_first=True, padding_value=-1
+ )
+ else:
+ speech = []
+ speech_lengths = []
+ output = {
+ "speech": speech,
+ "speech_lengths": speech_lengths,
+ "fbank_mask": fbank_mask[None, :],
+ "fbank_beg": fbank_beg[None,],
+ "fake_token_len": fake_token_len[None, :],
+ "input_ids": input_ids[None,],
+ "attention_mask": attention_mask[None,],
+ "labels_ids": labels,
+ "source_ids": source_ids[None, :],
+ "target_ids": target_ids[None, :],
+ }
+
+ return output
+
+ def inference_prepare(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ meta_data = {}
+
+ if kwargs.get("batch_size", 1) > 1:
+ raise NotImplementedError("batch decoding is not implemented")
+
+ contents = self.data_template(data_in[0])
+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)
+ batch = to_device(output, kwargs["device"])
+
+ # audio encoder
+ speech = batch["speech"]
+
+ if len(speech) > 0:
+ if "audio_embedding" in kwargs and "audio_embedding_lens" in kwargs:
+ encoder_out = kwargs["audio_embedding"]
+ encoder_out_lens = kwargs["audio_embedding_lens"]
+ else:
+ speech_lengths = batch["speech_lengths"][:, 0]
+ # fp16
+ if kwargs.get("fp16", False):
+ speech = speech.to(torch.float16)
+ elif kwargs.get("bf16", False):
+ speech = speech.to(torch.bfloat16)
+ # audio encoder
+ encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
+
+ # audio_adaptor
+ encoder_out, encoder_out_lens = self.audio_adaptor(encoder_out, encoder_out_lens)
+ meta_data["audio_adaptor_out"] = encoder_out
+ meta_data["audio_adaptor_out_lens"] = encoder_out_lens
+
+ input_ids = batch["input_ids"]
+ source_ids = batch["source_ids"]
+ fbank_beg = batch["fbank_beg"]
+ fake_token_len = batch["fake_token_len"]
+
+ if not kwargs.get("tearchforing", False):
+ input_ids = source_ids
+
+ input_ids[input_ids < 0] = 0
+ inputs_embeds = self.llm.model.get_input_embeddings()(input_ids)
+
+ batch_size, token_num, dims = inputs_embeds.shape
+
+ fake_token_len[fake_token_len < 0] = 0
+ fbank_beg[fbank_beg < 0] = 0
+
+ speech_idx = 0
+ for batch_idx in range(batch_size):
+ for turn_id in range(fbank_beg.shape[1]):
+ fbank_beg_idx = fbank_beg[batch_idx, turn_id].item()
+ if fbank_beg_idx > 0:
+ speech_token_len = fake_token_len[batch_idx, turn_id]
+ speech_token = encoder_out[speech_idx, :speech_token_len, :]
+
+ try:
+ inputs_embeds[
+ batch_idx,
+ fbank_beg_idx : fbank_beg_idx + speech_token_len,
+ :,
+ ] = speech_token
+ except Exception as e:
+ #
+ logging.error(f"{str(e)}, {traceback.format_exc()}")
+ logging.info(
+ f"batch_idx: {batch_idx}, inputs_embeds: {inputs_embeds.shape}, fbank_beg_idx: {fbank_beg_idx}, speech_token_len: {speech_token_len}, encoder_out: {encoder_out.shape}, encoder_out_lens: {encoder_out_lens}, fake_token_len: {fake_token_len}, speech_lengths: {speech_lengths}"
+ )
+ speech_token_len = encoder_out_lens[speech_idx].item()
+ speech_token = encoder_out[speech_idx, :speech_token_len, :]
+ inputs_embeds[
+ batch_idx,
+ fbank_beg_idx : fbank_beg_idx + speech_token_len,
+ :,
+ ] = speech_token
+
+ speech_idx += 1
+ return inputs_embeds, contents, batch, source_ids, meta_data
+
+ def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ new_data_in = []
+ for data in data_in:
+ if isinstance(data, str):
+ new_data_in.append(
+ [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {
+ "role": "user",
+ "content": f"语音转写:<|startofspeech|>!{data}<|endofspeech|>",
+ },
+ {"role": "assistant", "content": "null"},
+ ]
+ )
+ elif isinstance(data, torch.Tensor):
+ new_data_in.append(
+ [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {
+ "role": "user",
+ "content": f"语音转写:<|startofspeech|>!!<|endofspeech|>",
+ "audio": data,
+ },
+ {"role": "assistant", "content": "null"},
+ ]
+ )
+ data_in = new_data_in
+
+ if key is None:
+ key = []
+ for _ in data_in:
+ chars = string.ascii_letters + string.digits
+ key.append("rand_key_" + "".join(random.choice(chars) for _ in range(13)))
+
+ return self.inference_llm(
+ data_in,
+ data_lengths=data_lengths,
+ key=key,
+ tokenizer=tokenizer,
+ frontend=frontend,
+ **kwargs,
+ )
+
+ def inference_llm(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(
+ data_in, data_lengths, key, tokenizer, frontend, **kwargs
+ )
+ llm_dtype = kwargs.get("llm_dtype", "fp32")
+ if llm_dtype == "fp32":
+ llm_dtype = "fp16" if kwargs.get("fp16", False) else llm_dtype
+ llm_dtype = "bf16" if kwargs.get("bf16", False) else llm_dtype
+
+ with torch.cuda.amp.autocast(
+ enabled=True if llm_dtype != "fp32" else False, dtype=dtype_map[llm_dtype]
+ ):
+ label = contents["assistant"][-1]
+ self.llm = self.llm.to(dtype_map[llm_dtype])
+ inputs_embeds = inputs_embeds.to(dtype_map[llm_dtype])
+ llm_kwargs = kwargs.get("llm_kwargs", {})
+ if not kwargs.get("teachforing", False):
+ generated_ids = self.llm.generate(
+ inputs_embeds=inputs_embeds,
+ max_new_tokens=kwargs.get("max_length", 512),
+ **llm_kwargs,
+ )
+
+ response = tokenizer.batch_decode(
+ generated_ids,
+ skip_special_tokens=kwargs.get("skip_special_tokens", True),
+ )[0]
+
+ loss = None
+ else:
+ labels_ids = batch["labels_ids"]
+ labels_ids[labels_ids == -1] = -100
+ attention_mask = batch.get("attention_mask", None)
+ model_outputs = self.llm(
+ inputs_embeds=inputs_embeds,
+ attention_mask=attention_mask,
+ labels=labels_ids,
+ **llm_kwargs,
+ )
+
+ preds = torch.argmax(model_outputs.logits, -1)[:, source_ids.shape[1] :]
+ response = tokenizer.batch_decode(
+ preds,
+ add_special_tokens=False,
+ skip_special_tokens=kwargs.get("skip_special_tokens", True),
+ )[0]
+ loss = model_outputs.loss.item()
+
+ ibest_writer = None
+ if kwargs.get("output_dir") is not None:
+ if not hasattr(self, "writer"):
+ self.writer = DatadirWriter(kwargs.get("output_dir"))
+ ibest_writer = self.writer[f"{0 + 1}best_recog"]
+
+ results = []
+ response_clean = re.sub(r"[^\w\s\u3000\u4e00-\u9fff]+", "", response)
+ result_i = {
+ "key": key[0],
+ "text": re.sub(r"\s+", " ", response.replace("/sil", " ")),
+ "text_tn": response_clean,
+ "label": label,
+ }
+ if loss is not None:
+ result_i["loss"] = loss
+ results.append(result_i)
+
+ if ibest_writer is not None:
+ ibest_writer["text"][key[0]] = response.replace("\n", " ")
+ ibest_writer["label"][key[0]] = label.replace("\n", " ")
+ ibest_writer["text_tn"][key[0]] = response_clean
+
+ return results, meta_data
+
+ @staticmethod
+ def from_pretrained(model: str = None, **kwargs):
+ from funasr import AutoModel
+
+ model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)
+
+ return model, kwargs
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/requirements.txt b/examples/industrial_data_pretraining/fun_asr_nano/requirements.txt
new file mode 100644
index 000000000..596c1121d
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/requirements.txt
@@ -0,0 +1,4 @@
+torch
+torchaudio
+transformers
+funasr>=1.2.7
diff --git a/funasr/version.txt b/funasr/version.txt
index a77d7d927..5975b143a 100644
--- a/funasr/version.txt
+++ b/funasr/version.txt
@@ -1 +1 @@
-1.2.7
\ No newline at end of file
+1.2.8
\ No newline at end of file