Skip to content

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

update

游雁335eb1ea6156
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/README.md b/examples/industrial_data_pretraining/fun_asr_nano/README.md
index fa8725371..08c7a61ab 100644
--- a/examples/industrial_data_pretraining/fun_asr_nano/README.md
+++ b/examples/industrial_data_pretraining/fun_asr_nano/README.md
@@ -25,6 +25,11 @@ Online Experience:
</div>
+| Model Name | Task Details | Training Data | Parameters |
+| :-------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------: | :--------: |
+| Fun-ASR-Nano <br> ([⭐](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) [🤗](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512)) | Speech recognition supports Chinese, English, and Japanese. Chinese includes support for 7 dialects (Wu, Cantonese, Min, Hakka, Gan, Xiang, Jin) and 26 regional accents (Henan, Shanxi, Hubei, Sichuan, Chongqing, Yunnan, Guizhou, Guangdong, Guangxi and more than 20 other regions). English and Japanese cover multiple regional accents. Additional features include lyric recognition and rap speech recognition. | Tens of millions of hours | 800M |
+| Fun-ASR-MLT-Nano <br> ([⭐](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-MLT-Nano-2512) [🤗](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512)) | Speech recognition supports Chinese, English, Cantonese, Japanese, Korean, Vietnamese, Indonesian, Thai, Malay, Filipino, Arabic, Hindi, Bulgarian, Croatian, Czech, Danish, Dutch, Estonian, Finnish, Greek, Hungarian, Irish, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, Swedish, and 31 languages in total. | Hundreds of thousands of hours | 800M |
+
<a name="What's News"></a>
# What's New 🔥
@@ -46,6 +51,8 @@ Online Experience:
# Environment Setup 🐍
```shell
+git clone https://github.com/FunAudioLLM/Fun-ASR.git
+cd Fun-ASR
pip install -r requirements.txt
```
@@ -53,9 +60,9 @@ pip install -r requirements.txt
# TODO
-- [ ] Support returning timestamps
+- [x] Support returning timestamps
- [ ] Support speaker diarization
-- [ ] Support model training
+- [x] Support model training
# Usage 🛠️
@@ -74,10 +81,24 @@ def main():
trust_remote_code=True,
remote_code="./model.py",
device="cuda:0",
+ # hub:download models from ms (for ModelScope) or hf (for Hugging Face).
+ hub="hf"
)
wav_path = f"{model.model_path}/example/zh.mp3"
- res = model.generate(input=[wav_path], cache={}, batch_size=1)
+ res = model.generate(
+ input=[wav_path],
+ cache={},
+ batch_size=1,
+ hotwords=["开放时间"],
+ # 中文、英文、日文 for Fun-ASR-Nano-2512
+ # 中文、英文、粤语、日文、韩文、越南语、印尼语、泰语、马来语、菲律宾语、阿拉伯语、
+ # 印地语、保加利亚语、克罗地亚语、捷克语、丹麦语、荷兰语、爱沙尼亚语、芬兰语、希腊语、
+ # 匈牙利语、爱尔兰语、拉脱维亚语、立陶宛语、马耳他语、波兰语、葡萄牙语、罗马尼亚语、
+ # 斯洛伐克语、斯洛文尼亚语、瑞典语 for Fun-ASR-MLT-Nano-2512
+ language="中文",
+ itn=True, # or False
+ )
text = res[0]["text"]
print(text)
@@ -128,10 +149,66 @@ if __name__ == "__main__":
</details>
-# Performance Evaluation 📝
+# Finetune
+
+Please refer to [docs/finetune.md](docs/finetune.md)
+
+# Performance 📝
+
+We evaluated Fun-ASR against other state-of-the-art models on open-source benchmarks, Chinese dialect datasets, and industry-specific test sets. The results demonstrate that Fun-ASR achieves superior performance across various scenarios.
+
+### 1. Open-Source Dataset Performance (WER %)
+
+| Test set | GLM-ASR-nano | GLM-ASR-nano\* | Whisper-large-v3 | Seed-ASR | Seed-ASR\* | Kimi-Audio | Step-Audio2 | FireRed-ASR | Fun-ASR-nano | Fun-ASR |
+| :------------------ | :----------: | :------------: | :--------------: | :------: | :--------: | :--------: | :---------: | :---------: | :----------: | :-----: |
+| **Model Size** | 1.5B | 1.5B | 1.6B | - | - | - | - | 1.1B | 0.8B | 7.7B |
+| **OpenSource** | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ |
+| AIShell1 | 1.81 | 2.17 | 4.72 | 0.68 | 1.63 | 0.71 | 0.63 | 0.54 | 1.80 | 1.22 |
+| AIShell2 | - | 3.47 | 4.68 | 2.27 | 2.76 | 2.86 | 2.10 | 2.58 | 2.75 | 2.39 |
+| Fleurs-zh | - | 3.65 | 5.18 | 3.43 | 3.23 | 3.11 | 2.68 | 4.81 | 2.56 | 2.53 |
+| Fleurs-en | 5.78 | 6.95 | 6.23 | 9.39 | 9.39 | 6.99 | 3.03 | 10.79 | 5.96 | 4.74 |
+| Librispeech-clean | 2.00 | 2.17 | 1.86 | 1.58 | 2.8 | 1.32 | 1.17 | 1.84 | 1.76 | 1.51 |
+| Librispeech-other | 4.19 | 4.43 | 3.43 | 2.84 | 5.69 | 2.63 | 2.42 | 4.52 | 4.33 | 3.03 |
+| WenetSpeech Meeting | 6.73 | 8.21 | 18.39 | 5.69 | 7.07 | 6.24 | 4.75 | 4.95 | 6.60 | 6.17 |
+| WenetSpeech Net | - | 6.33 | 11.89 | 4.66 | 4.84 | 6.45 | 4.67 | 4.94 | 6.01 | 5.46 |
-We compared the multi-language speech recognition performance of Fun-ASR with other models on open-source benchmark datasets (including AISHELL-1, AISHELL-2, Wenetspeech, Librispeech, and Common Voice).
+> _Note: Seed-ASR\* results are evaluated using the official API on volcengine; GLM-ASR-nano\* results are evaluated using the open-source checkpoint._
+
+### 2. Industry Dataset Performance (WER %)
+
+| Test set | GLM-ASR-Nano | Whisper-large-v3 | Seed-ASR | FireRed-ASR | Kimi-Audio | Paraformer v2 | Fun-ASR-nano | Fun-ASR |
+| :----------------- | :----------: | :--------------: | :-------: | :---------: | :--------: | :-----------: | :----------: | :-------: |
+| **Model Size** | 1.5B | 1.6B | - | 1.1B | 8B | 0.2B | 0.8B | 7.7B |
+| **OpenSource** | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ |
+| Nearfield | 16.95 | 16.58 | 7.20 | 10.10 | 9.02 | 8.11 | 7.79 | 6.31 |
+| Farfield | 9.44 | 22.21 | 4.59 | 7.49 | 10.95 | 9.55 | 5.79 | 4.34 |
+| Complex Background | 23.79 | 32.57 | 12.90 | 15.56 | 15.56 | 15.19 | 14.59 | 11.45 |
+| English General | 16.47 | 18.56 | 15.65 | 21.62 | 18.12 | 19.48 | 15.28 | 13.73 |
+| Opensource | 4.67 | 7.05 | 3.83 | 5.31 | 3.79 | 6.23 | 4.22 | 3.38 |
+| Dialect | 54.21 | 66.14 | 29.45 | 52.82 | 71.94 | 41.16 | 28.18 | 15.21 |
+| Accent | 19.78 | 36.03 | 10.23 | 14.05 | 27.20 | 17.80 | 12.90 | 10.31 |
+| Lyrics | 46.56 | 54.82 | 30.26 | 42.87 | 65.18 | 50.14 | 30.85 | 21.00 |
+| Hiphop | 43.32 | 46.56 | 29.46 | 33.88 | 57.25 | 43.79 | 30.87 | 28.58 |
+| **Average** | **26.13** | **33.39** | **15.95** | **22.63** | **31.00** | **23.49** | **16.72** | **12.70** |
<div align="center">
<img src="images/compare_en.png" width="800" />
</div>
+
+## Remarkable Third-Party Work
+
+- vLLM (GPU) Deployment Best Practices: An accelerated implementation of Fun-ASR using vLLM. [Repository](https://github.com/yuekaizhang/Fun-ASR-vllm)
+
+## Citations
+
+```bibtex
+@misc{an2025funasrtechnicalreport,
+ title={Fun-ASR Technical Report},
+ author={Keyu An and Yanni Chen and Zhigao Chen and Chong Deng and Zhihao Du and Changfeng Gao and Zhifu Gao and Bo Gong and Xiangang Li and Yabin Li and Ying Liu and Xiang Lv and Yunjie Ji and Yiheng Jiang and Bin Ma and Haoneng Luo and Chongjia Ni and Zexu Pan and Yiping Peng and Zhendong Peng and Peiyao Wang and Hao Wang and Haoxu Wang and Wen Wang and Wupeng Wang and Yuzhong Wu and Biao Tian and Zhentao Tan and Nan Yang and Bin Yuan and Jieping Ye and Jixing Yu and Qinglin Zhang and Kun Zou and Han Zhao and Shengkui Zhao and Jingren Zhou and Yanqiao Zhu},
+ year={2025},
+ eprint={2509.12508},
+ archivePrefix={arXiv},
+ primaryClass={cs.CL},
+ url={https://arxiv.org/abs/2509.12508},
+}
+```
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/README_zh.md b/examples/industrial_data_pretraining/fun_asr_nano/README_zh.md
index fbdc34239..0757f305d 100644
--- a/examples/industrial_data_pretraining/fun_asr_nano/README_zh.md
+++ b/examples/industrial_data_pretraining/fun_asr_nano/README_zh.md
@@ -21,10 +21,15 @@ Fun-ASR 是通义实验室推出的端到端语音识别大模型,是基于数
模型仓库:[modelscope](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512),[huggingface](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512)
在线体验:
-[魔搭社区创空间](https://modelscope.cn/studios/FunAudioLLM/Fun-ASR),[huggingface space](https://huggingface.co/spaces/FunAudioLLM/Fun-ASR)
+[魔搭社区创空间](https://modelscope.cn/studios/FunAudioLLM/Fun-ASR-Nano),[huggingface space](https://huggingface.co/spaces/FunAudioLLM/Fun-ASR-Nano)
</div>
+| 模型 | 介绍 | 训练数据 | 参数 |
+| :-------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------: | :--: |
+| Fun-ASR-Nano <br> ([⭐](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) [🤗](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512)) | 支持中文、英文、日文。中文包含 7 种方言(吴语、粤语、闽语、客家话、赣语、湘语、晋语)及 26 种地域口音支持(河南、陕西、湖北、四川、重庆、云南、贵州、广东、广西、河北、天津、山东、安徽、南京、江苏、杭州、甘肃、宁夏)。英文、日文涵盖多种地域口音。额外功能包括歌词识别与说唱语音识别。 | 数千万小时 | 8 亿 |
+| Fun-ASR-MLT-Nano <br> ([⭐](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-MLT-Nano-2512) [🤗](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512)) | 支持中文、英文、粤语、日文、韩文、越南语、印尼语、泰语、马来语、菲律宾语、阿拉伯语、印地语、保加利亚语、克罗地亚语、捷克语、丹麦语、荷兰语、爱沙尼亚语、芬兰语、希腊语、匈牙利语、爱尔兰语、拉脱维亚语、立陶宛语、马耳他语、波兰语、葡萄牙语、罗马尼亚语、斯洛伐克语、斯洛文尼亚语、瑞典语,共 31 种语言。 | 数十万小时 | 8 亿 |
+
<a name="最新动态"></a>
# 最新动态 🔥
@@ -46,6 +51,8 @@ Fun-ASR 是通义实验室推出的端到端语音识别大模型,是基于数
# 环境安装 🐍
```shell
+git clone https://github.com/FunAudioLLM/Fun-ASR.git
+cd Fun-ASR
pip install -r requirements.txt
```
@@ -53,9 +60,9 @@ pip install -r requirements.txt
# TODO
-- [ ] 支持返回时间戳
+- [x] 支持返回时间戳
- [ ] 支持区分说话人识别
-- [ ] 支持模型训练
+- [x] 支持模型训练
# 用法 🛠️
@@ -74,10 +81,23 @@ def main():
trust_remote_code=True,
remote_code="./model.py",
device="cuda:0",
+ hub="ms"
)
wav_path = f"{model.model_path}/example/zh.mp3"
- res = model.generate(input=[wav_path], cache={}, batch_size=1)
+ res = model.generate(
+ input=[wav_path],
+ cache={},
+ batch_size=1,
+ hotwords=["开放时间"],
+ # 中文、英文、日文 for Fun-ASR-Nano-2512
+ # 中文、英文、粤语、日文、韩文、越南语、印尼语、泰语、马来语、菲律宾语、阿拉伯语、
+ # 印地语、保加利亚语、克罗地亚语、捷克语、丹麦语、荷兰语、爱沙尼亚语、芬兰语、希腊语、
+ # 匈牙利语、爱尔兰语、拉脱维亚语、立陶宛语、马耳他语、波兰语、葡萄牙语、罗马尼亚语、
+ # 斯洛伐克语、斯洛文尼亚语、瑞典语 for Fun-ASR-MLT-Nano-2512
+ language="中文",
+ itn=True, # or False
+ )
text = res[0]["text"]
print(text)
@@ -128,10 +148,66 @@ if __name__ == "__main__":
</details>
+# 微调
+
+详情请参考 [docs/finetune_zh.md](docs/finetune.md)
+
# 性能评测 📝
我们在开源基准数据集、中文方言测试集和工业测试集上,比较了 Fun-ASR 与其他模型的多语言语音识别性能。Fun-ASR 模型均具有明显的效果优势。
+### 1. 开源数据集性能 (WER %)
+
+| Test set | GLM-ASR-nano | GLM-ASR-nano\* | Whisper-large-v3 | Seed-ASR | Seed-ASR\* | Kimi-Audio | Step-Audio2 | FireRed-ASR | Fun-ASR-nano | Fun-ASR |
+| :------------------ | :----------: | :------------: | :--------------: | :------: | :--------: | :--------: | :---------: | :---------: | :----------: | :-----: |
+| **Model Size** | 1.5B | 1.5B | 1.6B | - | - | - | - | 1.1B | 0.8B | 7.7B |
+| **OpenSource** | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ |
+| AIShell1 | 1.81 | 2.17 | 4.72 | 0.68 | 1.63 | 0.71 | 0.63 | 0.54 | 1.80 | 1.22 |
+| AIShell2 | - | 3.47 | 4.68 | 2.27 | 2.76 | 2.86 | 2.10 | 2.58 | 2.75 | 2.39 |
+| Fleurs-zh | - | 3.65 | 5.18 | 3.43 | 3.23 | 3.11 | 2.68 | 4.81 | 2.56 | 2.53 |
+| Fleurs-en | 5.78 | 6.95 | 6.23 | 9.39 | 9.39 | 6.99 | 3.03 | 10.79 | 5.96 | 4.74 |
+| Librispeech-clean | 2.00 | 2.17 | 1.86 | 1.58 | 2.8 | 1.32 | 1.17 | 1.84 | 1.76 | 1.51 |
+| Librispeech-other | 4.19 | 4.43 | 3.43 | 2.84 | 5.69 | 2.63 | 2.42 | 4.52 | 4.33 | 3.03 |
+| WenetSpeech Meeting | 6.73 | 8.21 | 18.39 | 5.69 | 7.07 | 6.24 | 4.75 | 4.95 | 6.60 | 6.17 |
+| WenetSpeech Net | - | 6.33 | 11.89 | 4.66 | 4.84 | 6.45 | 4.67 | 4.94 | 6.01 | 5.46 |
+
+> _注:Seed-ASR\* 结果使用 volcengine 上的官方 API 评估;GLM-ASR-nano\* 结果使用开源 checkpoint 评估。_
+
+### 2. 工业数据集性能 (WER %)
+
+| Test set | GLM-ASR-Nano | Whisper-large-v3 | Seed-ASR | FireRed-ASR | Kimi-Audio | Paraformer v2 | Fun-ASR-nano | Fun-ASR |
+| :----------------- | :----------: | :--------------: | :-------: | :---------: | :--------: | :-----------: | :----------: | :-------: |
+| **Model Size** | 1.5B | 1.6B | - | 1.1B | 8B | 0.2B | 0.8B | 7.7B |
+| **OpenSource** | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ |
+| Nearfield | 16.95 | 16.58 | 7.20 | 10.10 | 9.02 | 8.11 | 7.79 | 6.31 |
+| Farfield | 9.44 | 22.21 | 4.59 | 7.49 | 10.95 | 9.55 | 5.79 | 4.34 |
+| Complex Background | 23.79 | 32.57 | 12.90 | 15.56 | 15.56 | 15.19 | 14.59 | 11.45 |
+| English General | 16.47 | 18.56 | 15.65 | 21.62 | 18.12 | 19.48 | 15.28 | 13.73 |
+| Opensource | 4.67 | 7.05 | 3.83 | 5.31 | 3.79 | 6.23 | 4.22 | 3.38 |
+| Dialect | 54.21 | 66.14 | 29.45 | 52.82 | 71.94 | 41.16 | 28.18 | 15.21 |
+| Accent | 19.78 | 36.03 | 10.23 | 14.05 | 27.20 | 17.80 | 12.90 | 10.31 |
+| Lyrics | 46.56 | 54.82 | 30.26 | 42.87 | 65.18 | 50.14 | 30.85 | 21.00 |
+| Hiphop | 43.32 | 46.56 | 29.46 | 33.88 | 57.25 | 43.79 | 30.87 | 28.58 |
+| **Average** | **26.13** | **33.39** | **15.95** | **22.63** | **31.00** | **23.49** | **16.72** | **12.70** |
+
<div align="center">
<img src="images/compare_zh.png" width="800" />
</div>
+
+## 优秀三方工作
+
+- vLLM (GPU) 最佳部署实践: 使用 vLLM 实现对 Fun-ASR 的加速. [Repository](https://github.com/yuekaizhang/Fun-ASR-vllm)
+
+## Citations
+
+```bibtex
+@misc{an2025funasrtechnicalreport,
+ title={Fun-ASR Technical Report},
+ author={Keyu An and Yanni Chen and Zhigao Chen and Chong Deng and Zhihao Du and Changfeng Gao and Zhifu Gao and Bo Gong and Xiangang Li and Yabin Li and Ying Liu and Xiang Lv and Yunjie Ji and Yiheng Jiang and Bin Ma and Haoneng Luo and Chongjia Ni and Zexu Pan and Yiping Peng and Zhendong Peng and Peiyao Wang and Hao Wang and Haoxu Wang and Wen Wang and Wupeng Wang and Yuzhong Wu and Biao Tian and Zhentao Tan and Nan Yang and Bin Yuan and Jieping Ye and Jixing Yu and Qinglin Zhang and Kun Zou and Han Zhao and Shengkui Zhao and Jingren Zhou and Yanqiao Zhu},
+ year={2025},
+ eprint={2509.12508},
+ archivePrefix={arXiv},
+ primaryClass={cs.CL},
+ url={https://arxiv.org/abs/2509.12508},
+}
+```
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/__init__.py b/examples/industrial_data_pretraining/fun_asr_nano/__init__.py
new file mode 100644
index 000000000..8b1378917
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/__init__.py
@@ -0,0 +1 @@
+
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/ctc.py b/examples/industrial_data_pretraining/fun_asr_nano/ctc.py
new file mode 100644
index 000000000..f2d592759
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/ctc.py
@@ -0,0 +1,60 @@
+import torch
+import torch.nn.functional as F
+
+
+class CTC(torch.nn.Module):
+ """CTC module.
+
+ Args:
+ odim: dimension of outputs
+ encoder_output_size: number of encoder projection units
+ dropout_rate: dropout rate (0.0 ~ 1.0)
+ reduce: reduce the CTC loss into a scalar
+ """
+
+ def __init__(
+ self,
+ odim: int,
+ encoder_output_size: int,
+ dropout_rate: float = 0.0,
+ reduce: bool = True,
+ blank_id: int = 0,
+ **kwargs,
+ ):
+ super().__init__()
+ eprojs = encoder_output_size
+ self.dropout_rate = dropout_rate
+ self.ctc_lo = torch.nn.Linear(eprojs, odim)
+ self.blank_id = blank_id
+ self.ctc_loss = torch.nn.CTCLoss(reduction="none", blank=blank_id)
+ self.reduce = reduce
+
+ def softmax(self, hs_pad):
+ """softmax of frame activations
+
+ Args:
+ Tensor hs_pad: 3d tensor (B, Tmax, eprojs)
+ Returns:
+ torch.Tensor: softmax applied 3d tensor (B, Tmax, odim)
+ """
+ return F.softmax(self.ctc_lo(hs_pad), dim=2)
+
+ def log_softmax(self, hs_pad):
+ """log_softmax of frame activations
+
+ Args:
+ Tensor hs_pad: 3d tensor (B, Tmax, eprojs)
+ Returns:
+ torch.Tensor: log softmax applied 3d tensor (B, Tmax, odim)
+ """
+ return F.log_softmax(self.ctc_lo(hs_pad), dim=2)
+
+ def argmax(self, hs_pad):
+ """argmax of frame activations
+
+ Args:
+ torch.Tensor hs_pad: 3d tensor (B, Tmax, eprojs)
+ Returns:
+ torch.Tensor: argmax applied 2d tensor (B, Tmax)
+ """
+ return torch.argmax(self.ctc_lo(hs_pad), dim=2)
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/data/train_example.jsonl b/examples/industrial_data_pretraining/fun_asr_nano/data/train_example.jsonl
new file mode 100644
index 000000000..b2ea6c968
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/data/train_example.jsonl
@@ -0,0 +1,20 @@
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0002.wav<|endofspeech|>"}, {"role": "assistant", "content": "几点了?"}], "speech_length": 145, "text_length": 3}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1688-142285-0001.wav<|endofspeech|>"}, {"role": "assistant", "content": "Margaret said Mister Hale, as he returned from showing his guest downstairs, I could not help watching your face with some anxiety when Mister Thornton made his confession of having been a shop boy."}], "speech_length": 1261, "text_length": 38}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写成英文:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0004.wav<|endofspeech|>"}, {"role": "assistant", "content": "Number ten. Fresh Nelly is waiting on you. Good night, husband."}], "speech_length": 520, "text_length": 16}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0002.wav<|endofspeech|>"}, {"role": "assistant", "content": "After early nightfall, the yellow lamps would light up here and there the squalid quarter of the brothels."}], "speech_length": 661, "text_length": 24}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0126.wav<|endofspeech|>"}, {"role": "assistant", "content": "因此,土地储备至关重要。"}], "speech_length": 348, "text_length": 6}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription without text normalization:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1688-142285-0002.wav<|endofspeech|>"}, {"role": "assistant", "content": "You don't mean that you thought me so silly"}], "speech_length": 282, "text_length": 11}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写成英文:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0001.wav<|endofspeech|>"}, {"role": "assistant", "content": "Stuff it into you, his belly counselled him."}], "speech_length": 326, "text_length": 11}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Transcribe speech into Chinese:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0127.wav<|endofspeech|>"}, {"role": "assistant", "content": "中原地产首席分析师张大伟说。"}], "speech_length": 443, "text_length": 9}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写,不进行文本规整:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0001.wav<|endofspeech|>"}, {"role": "assistant", "content": "换一首歌"}], "speech_length": 197, "text_length": 4}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Transcribe speech into Chinese:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0124.wav<|endofspeech|>"}, {"role": "assistant", "content": "为了规避三四线城市明显过剩的市场风险,"}], "speech_length": 522, "text_length": 11}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写成英文:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0003.wav<|endofspeech|>"}, {"role": "assistant", "content": "Hello, Bertie, any good in your mind?"}], "speech_length": 266, "text_length": 11}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Transcribe speech into Chinese:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0125.wav<|endofspeech|>"}, {"role": "assistant", "content": "标杆房企必然调整市场战略。"}], "speech_length": 429, "text_length": 7}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1688-142285-0000.wav<|endofspeech|>"}, {"role": "assistant", "content": "There's iron, they say, in all our blood, and a grain or two perhaps is good; but his, he makes me harshly feel, has got a little too much of steel. Anon."}], "speech_length": 1498, "text_length": 43}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0006.wav<|endofspeech|>"}, {"role": "assistant", "content": "The dull light fell more faintly upon the page whereon another equation began to unfold itself slowly and to spread abroad its widening tail."}], "speech_length": 1054, "text_length": 27}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写成英文:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0008.wav<|endofspeech|>"}, {"role": "assistant", "content": "The chaos in which his ardour extinguished itself was a cold indifferent knowledge of himself."}], "speech_length": 671, "text_length": 18}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0003.wav<|endofspeech|>"}, {"role": "assistant", "content": "早上好。"}], "speech_length": 147, "text_length": 3}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0005.wav<|endofspeech|>"}, {"role": "assistant", "content": "The music came nearer, and he recalled the words, the words of Shelley's fragment upon the moon, wandering companionless, pale for weariness."}], "speech_length": 962, "text_length": 30}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1688-142285-0004.wav<|endofspeech|>"}, {"role": "assistant", "content": "His statement of having been a shop boy was the thing I liked best of all."}], "speech_length": 446, "text_length": 17}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写成英文:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0007.wav<|endofspeech|>"}, {"role": "assistant", "content": "A cold lucid indifference reigned in his soul."}], "speech_length": 426, "text_length": 11}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0129.wav<|endofspeech|>"}, {"role": "assistant", "content": "也助推了土地市场的火爆。"}], "speech_length": 357, "text_length": 7}
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/data/train_text.txt b/examples/industrial_data_pretraining/fun_asr_nano/data/train_text.txt
new file mode 100644
index 000000000..65cc0d5ce
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/data/train_text.txt
@@ -0,0 +1,20 @@
+1089-134686-0004 Number ten. Fresh Nelly is waiting on you. Good night, husband.
+BAC009S0764W0127 中原地产首席分析师张大伟说。
+1089-134686-0002 After early nightfall, the yellow lamps would light up here and there the squalid quarter of the brothels.
+1688-142285-0001 Margaret said Mister Hale, as he returned from showing his guest downstairs, I could not help watching your face with some anxiety when Mister Thornton made his confession of having been a shop boy.
+1089-134686-0001 Stuff it into you, his belly counselled him.
+1688-142285-0002 You don't mean that you thought me so silly
+IT0011W0002 几点了?
+BAC009S0764W0126 因此,土地储备至关重要。
+BAC009S0764W0125 标杆房企必然调整市场战略。
+IT0011W0001 换一首歌。
+1688-142285-0000 There's iron, they say, in all our blood, and a grain or two perhaps is good; but his, he makes me harshly feel, has got a little too much of steel. Anon.
+BAC009S0764W0124 为了规避三四线城市明显过剩的市场风险,
+1089-134686-0003 Hello, Bertie, any good in your mind?
+1089-134686-0006 The dull light fell more faintly upon the page whereon another equation began to unfold itself slowly and to spread abroad its widening tail.
+IT0011W0003 早上好。
+1089-134686-0008 The chaos in which his ardour extinguished itself was a cold indifferent knowledge of himself.
+BAC009S0764W0129 也助推了土地市场的火爆。
+1089-134686-0005 The music came nearer, and he recalled the words, the words of Shelley's fragment upon the moon, wandering companionless, pale for weariness.
+1089-134686-0007 A cold lucid indifference reigned in his soul.
+1688-142285-0004 His statement of having been a shop boy was the thing I liked best of all.
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/data/train_wav.scp b/examples/industrial_data_pretraining/fun_asr_nano/data/train_wav.scp
new file mode 100644
index 000000000..7f0ce2b88
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/data/train_wav.scp
@@ -0,0 +1,20 @@
+1089-134686-0004 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0004.wav
+BAC009S0764W0127 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0127.wav
+1089-134686-0002 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0002.wav
+1688-142285-0001 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1688-142285-0001.wav
+1089-134686-0001 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0001.wav
+1688-142285-0002 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1688-142285-0002.wav
+IT0011W0002 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0002.wav
+BAC009S0764W0126 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0126.wav
+BAC009S0764W0125 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0125.wav
+IT0011W0001 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0001.wav
+1688-142285-0000 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1688-142285-0000.wav
+BAC009S0764W0124 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0124.wav
+1089-134686-0003 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0003.wav
+1089-134686-0006 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0006.wav
+IT0011W0003 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0003.wav
+1089-134686-0008 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0008.wav
+BAC009S0764W0129 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0129.wav
+1089-134686-0005 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0005.wav
+1089-134686-0007 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0007.wav
+1688-142285-0004 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1688-142285-0004.wav
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/data/val_example.jsonl b/examples/industrial_data_pretraining/fun_asr_nano/data/val_example.jsonl
new file mode 100644
index 000000000..cd72302c5
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/data/val_example.jsonl
@@ -0,0 +1,10 @@
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写成英文:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0000.wav<|endofspeech|>"}, {"role": "assistant", "content": "He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick peppered flour-fattened sauce."}], "speech_length": 1042, "text_length": 37}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0004.wav<|endofspeech|>"}, {"role": "assistant", "content": "放歌。"}], "speech_length": 131, "text_length": 3}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0130.wav<|endofspeech|>"}, {"role": "assistant", "content": "北京仅新增住宅土地供应10宗。"}], "speech_length": 523, "text_length": 9}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0005.wav<|endofspeech|>"}, {"role": "assistant", "content": "放首歌。"}], "speech_length": 155, "text_length": 4}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0121.wav<|endofspeech|>"}, {"role": "assistant", "content": "甚至出现交易几乎停滞的情况。"}], "speech_length": 418, "text_length": 7}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0009.wav<|endofspeech|>"}, {"role": "assistant", "content": "At most, by an alms given to a beggar whose blessing he fled from, he might hope wearily to win for himself some measure of actual grace."}], "speech_length": 1056, "text_length": 33}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写,不进行文本规整:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0123.wav<|endofspeech|>"}, {"role": "assistant", "content": "但因为聚集了过多公共资源"}], "speech_length": 398, "text_length": 7}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1688-142285-0003.wav<|endofspeech|>"}, {"role": "assistant", "content": "I really liked that account of himself better than anything else he said."}], "speech_length": 504, "text_length": 14}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "语音转写:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0128.wav<|endofspeech|>"}, {"role": "assistant", "content": "一线城市土地供应量减少,"}], "speech_length": 355, "text_length": 6}
+{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Speech transcription without text normalization:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0122.wav<|endofspeech|>"}, {"role": "assistant", "content": "一二线城市虽然也处于调整中"}], "speech_length": 410, "text_length": 9}
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/data/val_text.txt b/examples/industrial_data_pretraining/fun_asr_nano/data/val_text.txt
new file mode 100644
index 000000000..b487460f3
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/data/val_text.txt
@@ -0,0 +1,10 @@
+1688-142285-0003 I really liked that account of himself better than anything else he said.
+BAC009S0764W0130 北京仅新增住宅土地供应10宗。
+BAC009S0764W0123 但因为聚集了过多公共资源,
+IT0011W0004 放歌。
+IT0011W0005 放首歌。
+1089-134686-0009 At most, by an alms given to a beggar whose blessing he fled from, he might hope wearily to win for himself some measure of actual grace.
+1089-134686-0000 He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick peppered flour-fattened sauce.
+BAC009S0764W0121 甚至出现交易几乎停滞的情况。
+BAC009S0764W0128 一线城市土地供应量减少
+BAC009S0764W0122 一二线城市虽然也处于调整中,
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/data/val_wav.scp b/examples/industrial_data_pretraining/fun_asr_nano/data/val_wav.scp
new file mode 100644
index 000000000..eba943d9c
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/data/val_wav.scp
@@ -0,0 +1,10 @@
+1688-142285-0003 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1688-142285-0003.wav
+BAC009S0764W0130 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0130.wav
+BAC009S0764W0123 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0123.wav
+IT0011W0004 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0004.wav
+IT0011W0005 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0005.wav
+1089-134686-0009 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0009.wav
+1089-134686-0000 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/1089-134686-0000.wav
+BAC009S0764W0121 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0121.wav
+BAC009S0764W0128 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0128.wav
+BAC009S0764W0122 https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/BAC009S0764W0122.wav
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/decode.py b/examples/industrial_data_pretraining/fun_asr_nano/decode.py
new file mode 100644
index 000000000..6eb107a35
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/decode.py
@@ -0,0 +1,58 @@
+import os
+
+import hydra
+import torch
+from omegaconf import DictConfig, ListConfig, OmegaConf
+
+
[email protected](config_name=None, version_base=None)
+def main_hydra(cfg: DictConfig):
+ def to_plain_list(cfg_item):
+ if isinstance(cfg_item, ListConfig):
+ return OmegaConf.to_container(cfg_item, resolve=True)
+ elif isinstance(cfg_item, DictConfig):
+ return {k: to_plain_list(v) for k, v in cfg_item.items()}
+ else:
+ return cfg_item
+
+ kwargs = to_plain_list(cfg)
+
+ model_dir = kwargs.get("model_dir", "FunAudioLLM/Fun-ASR-Nano-2512")
+ scp_file = kwargs["scp_file"]
+ output_file = kwargs["output_file"]
+
+ device = (
+ "cuda:0"
+ if torch.cuda.is_available()
+ else "mps" if torch.backends.mps.is_available() else "cpu"
+ )
+
+ from funasr import AutoModel
+
+ model = AutoModel(
+ model=model_dir,
+ trust_remote_code=True,
+ vad_model="fsmn-vad",
+ vad_kwargs={"max_single_segment_time": 30000},
+ remote_code="./model.py",
+ device=device,
+ )
+
+ output_dir = os.path.dirname(output_file)
+ if output_dir and not os.path.exists(output_dir):
+ os.makedirs(output_dir, exist_ok=True)
+
+ with open(scp_file, "r", encoding="utf-8") as f1:
+ with open(output_file, "w", encoding="utf-8") as f2:
+ for line in f1:
+ line = line.strip()
+ if not line:
+ continue
+ parts = line.split(maxsplit=1)
+ if len(parts) == 2:
+ text = model.generate(input=[parts[1]], cache={}, batch_size=1)[0]["text"]
+ f2.write(f"{parts[0]}\t{text}\n")
+
+
+if __name__ == "__main__":
+ main_hydra()
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/deepspeed_conf/ds_stage1.json b/examples/industrial_data_pretraining/fun_asr_nano/deepspeed_conf/ds_stage1.json
new file mode 100644
index 000000000..51804c1f1
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/deepspeed_conf/ds_stage1.json
@@ -0,0 +1,33 @@
+{
+ "train_micro_batch_size_per_gpu": 1,
+ "gradient_accumulation_steps": 1,
+ "steps_per_print": 100,
+ "gradient_clipping": 5,
+ "fp16": {
+ "enabled": false,
+ "auto_cast": false,
+ "loss_scale": 0,
+ "initial_scale_power": 16,
+ "loss_scale_window": 1000,
+ "hysteresis": 2,
+ "consecutive_hysteresis": false,
+ "min_loss_scale": 1
+ },
+ "bf16": {
+ "enabled": true
+ },
+ "zero_force_ds_cpu_optimizer": false,
+ "zero_optimization": {
+ "stage": 1,
+ "offload_optimizer": {
+ "device": "none",
+ "pin_memory": true
+ },
+ "allgather_partitions": true,
+ "allgather_bucket_size": 5e8,
+ "overlap_comm": true,
+ "reduce_scatter": true,
+ "reduce_bucket_size": 5e8,
+ "contiguous_gradients" : true
+ }
+}
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/demo1.py b/examples/industrial_data_pretraining/fun_asr_nano/demo1.py
index 7992b1098..2d1c441df 100644
--- a/examples/industrial_data_pretraining/fun_asr_nano/demo1.py
+++ b/examples/industrial_data_pretraining/fun_asr_nano/demo1.py
@@ -1,17 +1,39 @@
-from funasr import AutoModel
+import torch
def main():
model_dir = "FunAudioLLM/Fun-ASR-Nano-2512"
+ device = (
+ "cuda:0"
+ if torch.cuda.is_available()
+ else "mps" if torch.backends.mps.is_available() else "cpu"
+ )
+
+ from funasr import AutoModel
+
model = AutoModel(
model=model_dir,
trust_remote_code=True,
remote_code="./model.py",
- device="cuda:0",
+ device=device,
+ # hub:download models from ms (for ModelScope) or hf (for Hugging Face).
+ hub="ms",
)
wav_path = f"{model.model_path}/example/zh.mp3"
- res = model.generate(input=[wav_path], cache={}, batch_size=1)
+ res = model.generate(
+ input=[wav_path],
+ cache={},
+ batch_size=1,
+ hotwords=["开放时间"],
+ # 中文、英文、日文 for Fun-ASR-Nano-2512
+ # 中文、英文、粤语、日文、韩文、越南语、印尼语、泰语、马来语、菲律宾语、阿拉伯语、
+ # 印地语、保加利亚语、克罗地亚语、捷克语、丹麦语、荷兰语、爱沙尼亚语、芬兰语、希腊语、
+ # 匈牙利语、爱尔兰语、拉脱维亚语、立陶宛语、马耳他语、波兰语、葡萄牙语、罗马尼亚语、
+ # 斯洛伐克语、斯洛文尼亚语、瑞典语 for Fun-ASR-MLT-Nano-2512
+ language="中文",
+ itn=True, # or False
+ )
text = res[0]["text"]
print(text)
@@ -21,7 +43,7 @@ def main():
vad_model="fsmn-vad",
vad_kwargs={"max_single_segment_time": 30000},
remote_code="./model.py",
- device="cuda:0",
+ device=device,
)
res = model.generate(input=[wav_path], cache={}, batch_size=1)
text = res[0]["text"]
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/demo2.py b/examples/industrial_data_pretraining/fun_asr_nano/demo2.py
index f53642939..7efc81e43 100644
--- a/examples/industrial_data_pretraining/fun_asr_nano/demo2.py
+++ b/examples/industrial_data_pretraining/fun_asr_nano/demo2.py
@@ -1,16 +1,39 @@
+import numpy as np
+import soundfile as sf
+import torch
+
from model import FunASRNano
+from tools.utils import load_audio
def main():
model_dir = "FunAudioLLM/Fun-ASR-Nano-2512"
- m, kwargs = FunASRNano.from_pretrained(model=model_dir, device="cuda:0")
+ device = (
+ "cuda:0"
+ if torch.cuda.is_available()
+ else "mps" if torch.backends.mps.is_available() else "cpu"
+ )
+ m, kwargs = FunASRNano.from_pretrained(model=model_dir, device=device)
+ tokenizer = kwargs.get("tokenizer", None)
m.eval()
wav_path = f"{kwargs['model_path']}/example/zh.mp3"
res = m.inference(data_in=[wav_path], **kwargs)
- text = res[0][0]["text"]
+ text = res[0][0]
print(text)
+ chunk_size = 0.72
+ duration = sf.info(wav_path).duration
+ cum_durations = np.arange(chunk_size, duration + chunk_size, chunk_size)
+ prev_text = ""
+ for idx, cum_duration in enumerate(cum_durations):
+ audio, rate = load_audio(wav_path, 16000, duration=round(cum_duration, 3))
+ prev_text = m.inference([torch.tensor(audio)], prev_text=prev_text, **kwargs)[0][0]["text"]
+ if idx != len(cum_durations) - 1:
+ prev_text = tokenizer.decode(tokenizer.encode(prev_text)[:-5]).replace("�", "")
+ if prev_text:
+ print(prev_text)
+
if __name__ == "__main__":
main()
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md b/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md
new file mode 100644
index 000000000..5f864de06
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md
@@ -0,0 +1,114 @@
+# Finetune
+
+「[简体中文](fintune_zh.md)」|「English」
+
+## Requirements
+
+```
+pip install funasr>=1.3.0
+```
+
+## Data Prepare
+
+Data examples
+
+```
+head -n1 data/train_example.jsonl | jq
+
+{
+ "messages": [
+ {
+ "role": "system",
+ "content": "You are a helpful assistant."
+ },
+ {
+ "role": "user",
+ "content": "语音转写:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0002.wav<|endofspeech|>"
+ },
+ {
+ "role": "assistant",
+ "content": "几点了?"
+ }
+ ],
+ "speech_length": 145,
+ "text_length": 3
+}
+```
+
+Full ref to `data/train_example.jsonl`
+
+Description:
+
+- The content of systemis fixed as `You are a helpful assistant.`
+- The content of userincludes the prompt and the path to the audio file (enclosed between `<|startofspeech|>!`and `<|endofspeech|>`).
+ - The default prompts are `语音转写:`and `Speech transcription: `.
+ - For corresponding languages, prompts can be combined, such as `语音转写成英文:`and `Transcribe speech into Chinese: `.
+ - When the text annotation corresponding to the audio file contains no Arabic numerals or punctuation marks, you can use `语音转写,不进行文本规整:`and `Speech transcription without text normalization: `.
+- The content of assistant corresponds to the text annotation of the audio file.
+- speech_length: The number of fbank frames of the audio file (10ms per frame).
+- text_length: The number of tokens in the annotation text of the audio file (encoded using `Qwen/Qwen3-0.6B`).
+
+- `messages[2]["content"]`: transcription
+- `speech_length`: number of fbank frames of the audio file
+- `text_length`: number of tokens of the transcription (tokenized by `Qwen3-0.6B`)
+
+We provide a data format conversion tool `scp2jsonl.py`, which can convert common speech recognition training data formats such as wav scp and transcription into the ChatML format.
+
+`train_wav.scp`
+
+```
+BAC009S0764W0121 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav
+BAC009S0916W0489 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav
+```
+
+`train_text.txt`
+
+```
+BAC009S0764W0121 甚至出现交易几乎停滞的情况
+BAC009S0916W0489 湖北一公司以员工名义贷款数十员工负债千万
+```
+
+```
+python tools/scp2jsonl.py \
+ ++scp_file=data/train_wav.scp \
+ ++transcript_file=data/train_text.txt \
+ ++jsonl_file=data/train_example.jsonl
+```
+
+## Finetune
+
+Modify the `audio_encoder_conf.freeze`, `audio_adaptor_conf.freeze`, and `llm_conf.freeze` in `finetune.sh`.
+
+Set the `freeze` parameter of the modules to be fine-tuned to false(by default, only the LLM is fine-tuned).
+
+For more detailed parameters, refer to: [SenseVoice Model Training and Testing](https://github.com/modelscope/FunASR/blob/main/docs/tutorial/README.md#Model%20Training%20and%20Testing)
+
+```
+bash finetune.sh
+```
+
+### Recommended Configuration
+
+- For training data less than 1000 hours, it is recommended to fine-tune the audio_adaptor.
+- For training data less than 5000 hours, it is recommended to fine-tune the audio_encoder and audio_adaptor.
+- For training data greater than 10000 hours, it is recommended to perform full-parameter fine-tuning.
+
+## Model Evaluation
+
+After model fine-tuning is completed, you can decode the model using the decode.py script:
+
+```
+python decode.py \
+ ++model_dir=/path/to/finetuned \
+ ++scp_file=data/val_wav.scp \
+ ++output_file=output.txt
+```
+
+After decoding is completed, text inverse normalization needs to be applied to the annotations and recognition results, and then the WER should be calculated:
+
+```
+python tools/whisper_mix_normalize.py data/val_text.txt data/val_norm.txt
+python tools/whisper_mix_normalize.py output.txt output_norm.txt
+compute-wer data/val_norm.txt output_norm.txt cer.txt
+tail -n8 cer.txt
+```
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/docs/fintune_zh.md b/examples/industrial_data_pretraining/fun_asr_nano/docs/fintune_zh.md
new file mode 100644
index 000000000..25e839487
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/docs/fintune_zh.md
@@ -0,0 +1,114 @@
+# 微调
+
+「简体中文」|「[English](finetune.md)」
+
+## 安装训练环境
+
+```
+pip install funasr>=1.3.0
+```
+
+## 数据准备
+
+数据格式需要包括如下几个字段:
+
+```
+head -n1 data/train_example.jsonl | jq
+
+{
+ "messages": [
+ {
+ "role": "system",
+ "content": "You are a helpful assistant."
+ },
+ {
+ "role": "user",
+ "content": "语音转写:<|startofspeech|>!https://modelscope.cn/datasets/FunAudioLLM/funasr-demo/resolve/master/audios/IT0011W0002.wav<|endofspeech|>"
+ },
+ {
+ "role": "assistant",
+ "content": "几点了?"
+ }
+ ],
+ "speech_length": 145,
+ "text_length": 3
+}
+```
+
+详细可以参考:`data/train_example.jsonl`
+
+数据准备细节介绍:
+
+- system 的 content 固定为 `You are a helpful assistant.`
+- user 的 content 包含了 prompt 和音频文件的路径(位于 `<|startofspeech|>!` 和 `<|endofspeech|>`之间)
+ - prompt 默认为`语音转写:`和`Speech transcription: `
+ - 可以结合对应的语种为`语音转写成英文:`和`Transcribe speech into Chinese: `
+ - 当音频文件对应的文本标注不含阿拉伯数字或者标点符号时,可以使用`语音转写,不进行文本规整:`和 `Speech transcription without text normalization: `
+- assistant 的 content 对应音频文件对应的文本标注
+- speech_length:音频文件的 fbank 帧数(一帧 10ms)
+- text_length:音频文件标注文本的 token 数 (用 `Qwen/Qwen3-0.6B` 编码)
+
+我们提供了数据格式转换工具 `scp2jsonl.py`,可以将常见的语音识别训练数据格式 wav scp 和 transcription 转成 ChatML 格式。
+
+`train_wav.scp`
+
+左边为数据唯一 ID,需与 `train_text.txt` 中的 ID 一一对应 右边为音频文件的路径,格式如下
+
+```
+BAC009S0764W0121 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav
+BAC009S0916W0489 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav
+```
+
+`train_text.txt`
+
+左边为数据唯一 ID,需与 `train_wav.scp` 中的 ID 一一对应 右边为音频文件标注文本,格式如下:
+
+```
+BAC009S0764W0121 甚至出现交易几乎停滞的情况
+BAC009S0916W0489 湖北一公司以员工名义贷款数十员工负债千万
+```
+
+```
+python tools/scp2jsonl.py \
+ ++scp_file=data/train_wav.scp \
+ ++transcript_file=data/train_text.txt \
+ ++jsonl_file=data/train_example.jsonl
+```
+
+## 启动训练
+
+修改 `finetune.sh` 中的 `audio_encoder_conf.freeze`, `audio_adaptor_conf.freeze` 和 `llm_conf.freeze`。
+
+将需要微调的模块 `freeze` 设置成 `false`(默认只微调 llm)。
+
+更多参数细节参考:[SenseVoice 模型训练与测试](https://github.com/modelscope/FunASR/blob/main/docs/tutorial/README_zh.md#%E6%A8%A1%E5%9E%8B%E8%AE%AD%E7%BB%83%E4%B8%8E%E6%B5%8B%E8%AF%95)
+
+```
+bash finetune.sh
+```
+
+### 推荐配置
+
+- 训练数据少于 1000 小时,建议微调 audio_adaptor
+- 训练数据少于 5000 小时,建议微调 audio_encoder 和 audio_adaptor
+- 训练数据大于 10000 小时,建议全量参数微调
+
+## 模型评测
+
+当模型微调结束后,可以使用 decode.py 脚本对模型进行解码:
+
+```
+python decode.py \
+ ++model_dir=/path/to/finetuned \
+ ++scp_file=data/val_wav.scp \
+ ++output_file=output.txt
+```
+
+解码结束后,需要对标注和识别结果做文本逆归一化,然后计算 WER:
+
+```
+python tools/whisper_mix_normalize.py data/val_text.txt data/val_norm.txt
+python tools/whisper_mix_normalize.py output.txt output_norm.txt
+compute-wer data/val_norm.txt output_norm.txt cer.txt
+tail -n8 cer.txt
+```
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/finetune.sh b/examples/industrial_data_pretraining/fun_asr_nano/finetune.sh
new file mode 100644
index 000000000..0e14453ad
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/finetune.sh
@@ -0,0 +1,65 @@
+# Copyright FunASR (https://github.com/alibaba-damo-academy/FunASR). All Rights Reserved.
+# MIT License (https://opensource.org/licenses/MIT)
+
+workspace=`pwd`
+
+# which gpu to train or finetune
+export CUDA_VISIBLE_DEVICES="0"
+gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
+
+# model_name from model_hub, or model_dir in local path
+model_name_or_model_dir="FunAudioLLM/Fun-ASR-Nano-2512"
+
+# data dir, which contains: train.json, val.json
+train_data=${workspace}/data/train_example.jsonl
+val_data=${workspace}/data/val_example.jsonl
+
+# exp output dir
+output_dir="./outputs"
+log_file="${output_dir}/log.txt"
+
+deepspeed_config=${workspace}/deepspeed_conf/ds_stage1.json
+
+mkdir -p ${output_dir}
+echo "log_file: ${log_file}"
+
+DISTRIBUTED_ARGS="
+ --nnodes ${WORLD_SIZE:-1} \
+ --nproc_per_node $gpu_num \
+ --node_rank ${RANK:-0} \
+ --master_addr ${MASTER_ADDR:-127.0.0.1} \
+ --master_port ${MASTER_PORT:-26669}
+"
+echo $DISTRIBUTED_ARGS
+
+# funasr trainer path
+train_tool=`which funasr-train-ds`
+echo "Using funasr trainer: ${train_tool}"
+
+torchrun $DISTRIBUTED_ARGS \
+${train_tool} \
+++model="${model_name_or_model_dir}" \
+++trust_remote_code=true \
+++train_data_set_list="${train_data}" \
+++valid_data_set_list="${val_data}" \
+++dataset_conf.data_split_num=1 \
+++dataset_conf.batch_sampler="BatchSampler" \
+++dataset_conf.batch_size=6000 \
+++dataset_conf.sort_size=1024 \
+++dataset_conf.batch_type="token" \
+++dataset_conf.num_workers=4 \
+++train_conf.max_epoch=50 \
+++train_conf.log_interval=1 \
+++train_conf.resume=true \
+++train_conf.validate_interval=2000 \
+++train_conf.save_checkpoint_interval=2000 \
+++train_conf.effective_save_name_excludes="None" \
+++train_conf.keep_nbest_models=20 \
+++train_conf.avg_nbest_model=10 \
+++train_conf.use_deepspeed=false \
+++train_conf.deepspeed_config=${deepspeed_config} \
+++optim_conf.lr=0.0002 \
+++audio_encoder_conf.freeze=true \
+++audio_adaptor_conf.freeze=true \
+++llm_conf.freeze=false \
+++output_dir="${output_dir}" &> ${log_file}
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/images/compare_en.png b/examples/industrial_data_pretraining/fun_asr_nano/images/compare_en.png
new file mode 100644
index 000000000..9637cbac1
Binary files /dev/null and b/examples/industrial_data_pretraining/fun_asr_nano/images/compare_en.png differ
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/images/compare_zh.png b/examples/industrial_data_pretraining/fun_asr_nano/images/compare_zh.png
new file mode 100644
index 000000000..861277566
Binary files /dev/null and b/examples/industrial_data_pretraining/fun_asr_nano/images/compare_zh.png differ
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/images/funasr-v2.png b/examples/industrial_data_pretraining/fun_asr_nano/images/funasr-v2.png
new file mode 100644
index 000000000..1de77e810
Binary files /dev/null and b/examples/industrial_data_pretraining/fun_asr_nano/images/funasr-v2.png differ
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/model.py b/examples/industrial_data_pretraining/fun_asr_nano/model.py
index f0d22a800..308740083 100644
--- a/examples/industrial_data_pretraining/fun_asr_nano/model.py
+++ b/examples/industrial_data_pretraining/fun_asr_nano/model.py
@@ -1,4 +1,3 @@
-import json
import logging
import os
import random
@@ -6,10 +5,11 @@ import re
import string
import time
import traceback
+from typing import Union
import torch
import torch.nn as nn
-from funasr import AutoModel
+
from funasr.metrics.compute_acc import compute_accuracy
from funasr.register import tables
from funasr.train_utils.device_funcs import force_gatherable, to_device
@@ -17,6 +17,9 @@ from funasr.utils.datadir_writer import DatadirWriter
from funasr.utils.load_utils import extract_fbank, load_audio_text_image_video
from transformers import AutoConfig, AutoModelForCausalLM
+from ctc import CTC
+from tools.utils import forced_align
+
dtype_map = {"bf16": torch.bfloat16, "fp16": torch.float16, "fp32": torch.float32}
@@ -42,6 +45,8 @@ class FunASRNano(nn.Module):
"activation_checkpoint", False
)
if hub == "ms":
+ from funasr import AutoModel
+
model = AutoModel(model=audio_encoder, model_revision="master")
audio_encoder_output_size = (
model.model.encoder_output_size
@@ -56,13 +61,13 @@ class FunASRNano(nn.Module):
audio_encoder = encoder_class(input_size=input_size, **audio_encoder_conf)
audio_encoder_output_size = audio_encoder.output_size()
freeze = audio_encoder_conf.get("freeze", True)
- freeze_layer_num = int(audio_encoder_conf.get("freeze_layer_num", -1))
if freeze:
- for name, param in audio_encoder.named_parameters():
+ for _, param in audio_encoder.named_parameters():
param.requires_grad = False
audio_encoder.eval()
self.audio_encoder = audio_encoder
+
# llm
self.llm = None
init_param_path = llm_conf.get("init_param_path", None)
@@ -74,31 +79,9 @@ class FunASRNano(nn.Module):
freeze = llm_conf.get("freeze", True)
if freeze:
- for name, param in model.named_parameters():
+ for _, param in model.named_parameters():
param.requires_grad = False
model.eval()
- logging.info(f"use_lora: {llm_conf.get('use_lora', False)}")
- if llm_conf.get("use_lora", False):
- from omegaconf import DictConfig, OmegaConf
-
- lora_conf = llm_conf.get("lora_conf", {})
- if isinstance(lora_conf, (OmegaConf, DictConfig)):
- lora_conf = OmegaConf.to_container(lora_conf, resolve=True)
- from peft import LoraConfig, PeftModel, get_peft_model
-
- lora_init_param_path = lora_conf.get("init_param_path", None)
- if lora_init_param_path is not None:
- logging.info(f"lora_init_param_path: {lora_init_param_path}")
- model = PeftModel.from_pretrained(model, lora_init_param_path)
- for name, param in model.named_parameters():
- if not lora_conf.get("freeze_lora", False):
- if "lora_" in name:
- param.requires_grad = True
- else:
- peft_config = LoraConfig(**lora_conf)
- model = get_peft_model(model, peft_config)
- model.print_trainable_parameters()
-
if llm_conf.get("activation_checkpoint", False):
model.gradient_checkpointing_enable()
@@ -114,20 +97,64 @@ class FunASRNano(nn.Module):
llm_dim if llm_dim is not None else audio_adaptor_conf["llm_dim"]
)
audio_adaptor = adaptor_class(**audio_adaptor_conf)
- init_param_path = audio_adaptor_conf.get("init_param_path", None)
- if init_param_path is not None:
- src_state = torch.load(init_param_path, map_location="cpu")
- flag = audio_adaptor.load_state_dict(src_state, strict=False)
- logging.info(f"Loading audio_adaptor ckpt: {init_param_path}, status: {flag}")
freeze = audio_adaptor_conf.get("freeze", False)
if freeze:
- for name, param in audio_adaptor.named_parameters():
+ for _, param in audio_adaptor.named_parameters():
param.requires_grad = False
audio_adaptor.eval()
self.audio_adaptor = audio_adaptor
+ self.use_low_frame_rate = audio_adaptor_conf.get("use_low_frame_rate", False)
+
+ # ctc decoder
+ self.ctc_decoder = None
+ # TODO: fix table name
+ ctc_decoder_class = tables.adaptor_classes.get(kwargs.get("ctc_decoder", None))
+ if ctc_decoder_class is not None:
+ ctc_tokenizer = (
+ kwargs.get("ctc_tokenizer", None)
+ if "ctc_tokenizer" in kwargs
+ else kwargs["dataset_conf"]["ctc_tokenizer"]
+ )
+ ctc_tokenizer_conf = (
+ kwargs.get("ctc_tokenizer_conf", None)
+ if "ctc_tokenizer_conf" in kwargs
+ else kwargs["dataset_conf"]["ctc_tokenizer_conf"]
+ )
+ if ctc_tokenizer is not None and ctc_tokenizer_conf is not None:
+ ctc_tokenizer_class = tables.tokenizer_classes.get(ctc_tokenizer)
+ ctc_tokenizer = ctc_tokenizer_class(**ctc_tokenizer_conf)
+ self.ctc_tokenizer = ctc_tokenizer
+ assert ctc_tokenizer is not None, f"ctc_tokenizer must be set"
+
+ ctc_vocab_size = kwargs.get("ctc_vocab_size", 60515)
+ ctc_decoder_conf = kwargs.get("ctc_decoder_conf", {})
+ if audio_encoder_output_size > 0:
+ ctc_decoder_conf["encoder_dim"] = audio_encoder_output_size
+ self.ctc_decoder = ctc_decoder_class(**ctc_decoder_conf)
+ init_param_path = ctc_decoder_conf.get("init_param_path", None)
+ if init_param_path is not None:
+ src_state = torch.load(init_param_path, map_location="cpu")
+ flag = self.ctc_decoder.load_state_dict(src_state, strict=False)
+ logging.info(f"Loading ctc_decoder ckpt: {init_param_path}, status: {flag}")
+ freeze = ctc_decoder_conf.get("freeze", False)
+ if freeze:
+ for _, param in self.ctc_decoder.named_parameters():
+ param.requires_grad = False
+ self.ctc_decoder.eval()
+
+ ctc_conf = kwargs.get("ctc_conf", {})
+ self.blank_id = ctc_conf.get("blank_id", ctc_vocab_size - 1)
+ self.ctc_weight = kwargs.get("ctc_weight", 0.3)
+ self.ctc = CTC(
+ odim=ctc_vocab_size,
+ encoder_output_size=audio_encoder_output_size,
+ blank_id=self.blank_id,
+ **ctc_conf,
+ )
+ self.detach_ctc_decoder = kwargs.get("detach_ctc_decoder", True)
+ self.error_calculator = None
self.length_normalized_loss = length_normalized_loss
- self.feat_permute = audio_encoder_conf.get("feat_permute", True)
rank = int(os.environ.get("RANK", 0))
logging.info(f"rank: {rank}, model is builded.")
@@ -203,7 +230,9 @@ class FunASRNano(nn.Module):
stats["batch_size_real_frames"] = speech_lengths.sum().item()
stats["padding_frames"] = stats["batch_size_x_frames"] - stats["batch_size_real_frames"]
- with torch.cuda.amp.autocast(
+ device_type = next(self.parameters()).device.type
+ with torch.autocast(
+ device_type=device_type if device_type in ["cuda", "xpu", "mps"] else "cpu",
enabled=True if self.llm_dtype != "fp32" else False,
dtype=dtype_map[self.llm_dtype],
):
@@ -247,12 +276,7 @@ class FunASRNano(nn.Module):
def encode(self, speech, speech_lengths):
# audio encoder
- if self.feat_permute:
- encoder_out, encoder_out_lens = self.audio_encoder(
- speech.permute(0, 2, 1), speech_lengths
- )
- else:
- encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)
+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)
return encoder_out, encoder_out_lens
@@ -329,6 +353,8 @@ class FunASRNano(nn.Module):
)
if not do_think:
source_input += "<think>\n\n</think>\n\n"
+ if kwargs.get("prev_text", None) is not None:
+ source_input += kwargs["prev_text"]
splits = pattern.split(source_input)
source_ids = []
@@ -375,12 +401,12 @@ class FunASRNano(nn.Module):
/ 1000
)
- if self.feat_permute:
- speech = speech.permute(0, 2, 1)
-
- olens = 1 + (speech_lengths[0].item() - 3 + 2 * 1) // 2
- olens = 1 + (olens - 3 + 2 * 1) // 2
- fake_token_len_i = (olens - 1) // 2 + 1
+ if self.use_low_frame_rate:
+ olens = 1 + (speech_lengths[0].item() - 3 + 2 * 1) // 2
+ olens = 1 + (olens - 3 + 2 * 1) // 2
+ fake_token_len_i = (olens - 1) // 2 + 1
+ else:
+ fake_token_len_i = speech_lengths[0].item()
fake_token = [0] * fake_token_len_i
fbank_beg_i = len(source_ids)
source_ids += fake_token
@@ -468,16 +494,18 @@ class FunASRNano(nn.Module):
encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
# audio_adaptor
- encoder_out, encoder_out_lens = self.audio_adaptor(encoder_out, encoder_out_lens)
- meta_data["audio_adaptor_out"] = encoder_out
- meta_data["audio_adaptor_out_lens"] = encoder_out_lens
+ adaptor_out, adaptor_out_lens = self.audio_adaptor(encoder_out, encoder_out_lens)
+ meta_data["encoder_out"] = encoder_out
+ meta_data["encoder_out_lens"] = encoder_out_lens
+ meta_data["audio_adaptor_out"] = adaptor_out
+ meta_data["audio_adaptor_out_lens"] = adaptor_out_lens
input_ids = batch["input_ids"]
source_ids = batch["source_ids"]
fbank_beg = batch["fbank_beg"]
fake_token_len = batch["fake_token_len"]
- if not kwargs.get("tearchforing", False):
+ if not kwargs.get("teacherforcing", False):
input_ids = source_ids
input_ids[input_ids < 0] = 0
@@ -494,7 +522,7 @@ class FunASRNano(nn.Module):
fbank_beg_idx = fbank_beg[batch_idx, turn_id].item()
if fbank_beg_idx > 0:
speech_token_len = fake_token_len[batch_idx, turn_id]
- speech_token = encoder_out[speech_idx, :speech_token_len, :]
+ speech_token = adaptor_out[speech_idx, :speech_token_len, :]
try:
inputs_embeds[
@@ -506,10 +534,10 @@ class FunASRNano(nn.Module):
#
logging.error(f"{str(e)}, {traceback.format_exc()}")
logging.info(
- f"batch_idx: {batch_idx}, inputs_embeds: {inputs_embeds.shape}, fbank_beg_idx: {fbank_beg_idx}, speech_token_len: {speech_token_len}, encoder_out: {encoder_out.shape}, encoder_out_lens: {encoder_out_lens}, fake_token_len: {fake_token_len}, speech_lengths: {speech_lengths}"
+ f"batch_idx: {batch_idx}, inputs_embeds: {inputs_embeds.shape}, fbank_beg_idx: {fbank_beg_idx}, speech_token_len: {speech_token_len}, adaptor_out: {adaptor_out.shape}, adaptor_out_lens: {adaptor_out_lens}, fake_token_len: {fake_token_len}, speech_lengths: {speech_lengths}"
)
- speech_token_len = encoder_out_lens[speech_idx].item()
- speech_token = encoder_out[speech_idx, :speech_token_len, :]
+ speech_token_len = adaptor_out_lens[speech_idx].item()
+ speech_token = adaptor_out[speech_idx, :speech_token_len, :]
inputs_embeds[
batch_idx,
fbank_beg_idx : fbank_beg_idx + speech_token_len,
@@ -519,6 +547,39 @@ class FunASRNano(nn.Module):
speech_idx += 1
return inputs_embeds, contents, batch, source_ids, meta_data
+ def get_prompt(self, hotwords: list[str], language: str = None, itn: bool = True):
+ if len(hotwords) > 0:
+ hotwords = ", ".join(hotwords)
+ prompt = f"请结合上下文信息,更加准确地完成语音转写任务。如果没有相关信息,我们会留空。\n\n\n**上下文信息:**\n\n\n"
+ prompt += f"热词列表:[{hotwords}]\n"
+ else:
+ prompt = ""
+ if language is None:
+ prompt += "语音转写"
+ else:
+ prompt += f"语音转写成{language}"
+ if not itn:
+ prompt += ",不进行文本规整"
+ return prompt + ":"
+
+ def generate_chatml(self, prompt: str, data: Union[str, torch.Tensor]):
+ if isinstance(data, str):
+ return [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {"role": "user", "content": f"{prompt}<|startofspeech|>!{data}<|endofspeech|>"},
+ {"role": "assistant", "content": "null"},
+ ]
+ elif isinstance(data, torch.Tensor):
+ return [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {
+ "role": "user",
+ "content": f"{prompt}<|startofspeech|>!!<|endofspeech|>",
+ "audio": data,
+ },
+ {"role": "assistant", "content": "null"},
+ ]
+
def inference(
self,
data_in,
@@ -528,32 +589,10 @@ class FunASRNano(nn.Module):
frontend=None,
**kwargs,
):
- new_data_in = []
- for data in data_in:
- if isinstance(data, str):
- new_data_in.append(
- [
- {"role": "system", "content": "You are a helpful assistant."},
- {
- "role": "user",
- "content": f"语音转写:<|startofspeech|>!{data}<|endofspeech|>",
- },
- {"role": "assistant", "content": "null"},
- ]
- )
- elif isinstance(data, torch.Tensor):
- new_data_in.append(
- [
- {"role": "system", "content": "You are a helpful assistant."},
- {
- "role": "user",
- "content": f"语音转写:<|startofspeech|>!!<|endofspeech|>",
- "audio": data,
- },
- {"role": "assistant", "content": "null"},
- ]
- )
- data_in = new_data_in
+ prompt = self.get_prompt(
+ kwargs.get("hotwords", []), kwargs.get("language", None), kwargs.get("itn", True)
+ )
+ data_in = [self.generate_chatml(prompt, data) for data in data_in]
if key is None:
key = []
@@ -582,22 +621,51 @@ class FunASRNano(nn.Module):
inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(
data_in, data_lengths, key, tokenizer, frontend, **kwargs
)
+
+ ctc_results = []
+ if self.ctc_decoder is not None:
+ encoder_out = meta_data["encoder_out"]
+ encoder_out_lens = meta_data["encoder_out_lens"]
+ decoder_out, decoder_out_lens = self.ctc_decoder(encoder_out, encoder_out_lens)
+ ctc_logits = self.ctc.log_softmax(decoder_out)
+
+ b, n, d = encoder_out.size()
+ if isinstance(key[0], (list, tuple)):
+ key = key[0]
+ if len(key) < b:
+ key = key * b
+ for i in range(b):
+ x = ctc_logits[i, : encoder_out_lens[i].item(), :]
+ yseq = x.argmax(dim=-1)
+ yseq = torch.unique_consecutive(yseq, dim=-1)
+ mask = yseq != self.blank_id
+ token_int = yseq[mask].tolist()
+ # Change integer-ids to tokens
+ text = self.ctc_tokenizer.decode(token_int)
+ ctc_results.append({"key": key[i], "text": text, "ctc_logits": x})
+
llm_dtype = kwargs.get("llm_dtype", "fp32")
if llm_dtype == "fp32":
llm_dtype = "fp16" if kwargs.get("fp16", False) else llm_dtype
llm_dtype = "bf16" if kwargs.get("bf16", False) else llm_dtype
- with torch.cuda.amp.autocast(
- enabled=True if llm_dtype != "fp32" else False, dtype=dtype_map[llm_dtype]
+ device_type = torch.device(kwargs.get("device", "cuda")).type
+ with torch.autocast(
+ device_type=device_type if device_type in ["cuda", "xpu", "mps"] else "cpu",
+ enabled=True if llm_dtype != "fp32" else False,
+ dtype=dtype_map[llm_dtype],
):
label = contents["assistant"][-1]
self.llm = self.llm.to(dtype_map[llm_dtype])
inputs_embeds = inputs_embeds.to(dtype_map[llm_dtype])
llm_kwargs = kwargs.get("llm_kwargs", {})
- if not kwargs.get("teachforing", False):
+ if not kwargs.get("teacherforcing", False):
+ attention_mask = batch.get("attention_mask", None)
generated_ids = self.llm.generate(
inputs_embeds=inputs_embeds,
+ attention_mask=attention_mask,
max_new_tokens=kwargs.get("max_length", 512),
+ pad_token_id=self.llm.config.pad_token_id or self.llm.config.eos_token_id,
**llm_kwargs,
)
@@ -615,6 +683,7 @@ class FunASRNano(nn.Module):
inputs_embeds=inputs_embeds,
attention_mask=attention_mask,
labels=labels_ids,
+ pad_token_id=self.llm.config.pad_token_id or self.llm.config.eos_token_id,
**llm_kwargs,
)
@@ -625,6 +694,7 @@ class FunASRNano(nn.Module):
skip_special_tokens=kwargs.get("skip_special_tokens", True),
)[0]
loss = model_outputs.loss.item()
+ response = kwargs.get("prev_text", "") + response
ibest_writer = None
if kwargs.get("output_dir") is not None:
@@ -644,6 +714,22 @@ class FunASRNano(nn.Module):
result_i["loss"] = loss
results.append(result_i)
+ for ctc_result, result in zip(ctc_results, results):
+ result["ctc_text"] = ctc_result["text"].replace("<|nospeech|>", "")
+ target_ids = torch.tensor(
+ self.ctc_tokenizer.encode(result["ctc_text"]), dtype=torch.int64
+ )
+ result["ctc_timestamps"] = forced_align(
+ ctc_result["ctc_logits"], target_ids, self.blank_id
+ )
+ target_ids = torch.tensor(self.ctc_tokenizer.encode(result["text"]), dtype=torch.int64)
+ result["timestamps"] = forced_align(ctc_result["ctc_logits"], target_ids, self.blank_id)
+ for timestamps in [result["timestamps"], result["ctc_timestamps"]]:
+ for timestamp in timestamps:
+ timestamp["token"] = self.ctc_tokenizer.decode([timestamp["token"]])
+ timestamp["start_time"] = timestamp["start_time"] * 6 * 10 / 1000
+ timestamp["end_time"] = timestamp["end_time"] * 6 * 10 / 1000
+
if ibest_writer is not None:
ibest_writer["text"][key[0]] = response.replace("\n", " ")
ibest_writer["label"][key[0]] = label.replace("\n", " ")
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/requirements.txt b/examples/industrial_data_pretraining/fun_asr_nano/requirements.txt
index 596c1121d..eac7d4a42 100644
--- a/examples/industrial_data_pretraining/fun_asr_nano/requirements.txt
+++ b/examples/industrial_data_pretraining/fun_asr_nano/requirements.txt
@@ -1,4 +1,9 @@
-torch
-torchaudio
-transformers
-funasr>=1.2.7
+torch>=2.9.0
+torchaudio>=2.9.0
+transformers>=4.51.3
+funasr>=1.3.0
+zhconv
+whisper_normalizer
+pyopenjtalk-plus
+compute-wer
+openai-whisper
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/tools/__init__.py b/examples/industrial_data_pretraining/fun_asr_nano/tools/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/tools/format5res.py b/examples/industrial_data_pretraining/fun_asr_nano/tools/format5res.py
new file mode 100644
index 000000000..3417337bb
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/tools/format5res.py
@@ -0,0 +1,301 @@
+# -*- coding: utf-8 -*-
+#!/usr/bin/python
+# Author: Mengze Chen
+
+import re
+import sys
+
+
+def scoreformat(name, line, flag=1):
+ newline = ""
+ for i in range(0, len(line)):
+ curr = line[i]
+ currEn = False
+ if curr == "":
+ continue
+ if (
+ (curr >= "\u0041" and curr <= "\u005a") # eng
+ or (curr >= "\u0061" and curr <= "\u007a") # eng
+ or (curr >= "\u0000" and curr <= "\u007f") # de fr es it
+ or (curr >= "\u0400" and curr <= "\u04ff") # ru
+ or (curr >= "\u0100" and curr <= "\u017f") # latin1
+ or (curr >= "\u0080" and curr <= "\u00ff") # latin2
+ or curr == "'"
+ ) and (curr < "\u0030" or curr > "\u0039"):
+ currEn = True
+ if i == 0:
+ newline = newline + curr
+ else:
+ if lastEn == True and currEn == True:
+ newline = newline + curr
+ else:
+ newline = newline + " " + curr
+ if flag == -1:
+ lastEn = False
+ else:
+ lastEn = currEn
+ ret = re.sub("[ ]{1,}", " ", newline)
+ ret = ret
+ if name == "":
+ ret = ret
+ else:
+ if flag <= 0:
+ ret = ret + " " + "(" + name + ")"
+ else:
+ ret = name + "\t" + ret
+ return ret
+
+
+def recoformat(line):
+ newline = ""
+ en_flag = 0 # 0: no-english 1 : english 2: former
+ for i in range(0, len(line)):
+ word = line[i]
+ if ord(word) == 32:
+ if en_flag == 0:
+ continue
+ else:
+ en_flag = 0
+ newline += " "
+ if (word >= "\u4e00" and word <= "\u9fa5") or (word >= "\u0030" and word <= "\u0039"):
+ if en_flag == 1:
+ newline += " " + word
+ else:
+ newline += word
+ en_flag = 0
+ elif (
+ (word >= "\u0041" and word <= "\u005a") # eng
+ or (word >= "\u0061" and word <= "\u007a") # eng
+ or (word >= "\u0000" and word <= "\u007f") # de fr es it
+ or (word >= "\u0400" and word <= "\u04ff") # ru
+ or (word >= "\u0100" and word <= "\u017f") # latin1
+ or (word >= "\u0080" and word <= "\u00ff") # latin2
+ or word == "'"
+ ):
+ if en_flag == 0:
+ newline += " " + ("" if (word == "'") else word)
+ else:
+ newline += word
+ en_flag = 1
+ else:
+ newline += " " + word
+ newline = newline
+ newline = re.sub("[ ]{1,}", " ", newline)
+ newline = newline
+ return newline
+
+
+def numbersingle(line):
+ chnu = ["零", "一", "二", "两", "三", "四", "五", "六", "七", "八", "九", "点"]
+ newline = ""
+ for id in range(len(line)):
+ if re.findall(r"\.", line[id]):
+ if re.findall(r"\.\s*$", line[id]):
+ newline += "."
+ else:
+ newline += chnu[10]
+ elif re.search(r"0", line[id]):
+ if id > 0 and id < len(line) - 1:
+ if (
+ re.search(r"\d", line[id - 1])
+ and (not re.search(r"\d", line[id + 1]))
+ and (not re.search(r"0", line[id - 1]))
+ ):
+ if id > 2 and len(line) > 2 and (not re.search(r"\d", line[id - 1])):
+ newline = newline[:-1]
+ newline += chnu[int(line[id - 1])] + "十"
+ else:
+ newline += chnu[int(line[id])]
+ else:
+ newline += chnu[int(line[id])]
+ else:
+ newline += chnu[int(line[id])]
+ elif re.search(r"\d", line[id]):
+ newline += chnu[int(line[id])]
+ else:
+ newline += line[id]
+ return newline
+
+
+def ch_number2digit(line):
+ number_flag = 0
+ zero_flag = 0
+ bits = {
+ "零": "1",
+ "十": "2",
+ "百": "3",
+ "千": "4",
+ "万": "5",
+ "十万": "6",
+ "百万": "7",
+ "千万": "8",
+ }
+ chsh = {
+ "一": "1",
+ "二": "2",
+ "三": "3",
+ "四": "4",
+ "五": "5",
+ "六": "6",
+ "七": "7",
+ "八": "8",
+ "九": "9",
+ "两": "2",
+ "幺": "1",
+ }
+ unit = {"里": "1", "克": "1", "米": "1"}
+ newline = ""
+ digit = []
+ bit = []
+ onebit = ""
+ for i in range(len(line)):
+ if ord(line[i]) == 32:
+ newline += " "
+ continue
+ if line[i] in chsh:
+ number_flag = 1
+ if line[i] == "两":
+ if (i == len(line) - 1) or (
+ (line[i + 1] not in chsh.keys()) and (line[i + 1] not in bits.keys())
+ ):
+ number_flag = -1
+ if number_flag == 1:
+ digit.append(chsh[line[i]])
+
+ elif "十" == line[i] and number_flag == 0:
+ number_flag = 2
+ digit.append("1")
+ bit.append(line[i])
+ elif "十" == line[i] and number_flag == 3:
+ digit.append("1")
+ bit.append(line[i])
+ elif ("零" == line[i]) and (number_flag == 0 or number_flag == 1):
+ digit.append("0")
+ elif ("零" == line[i]) and number_flag == 3:
+ zero_flag = 1
+ elif number_flag == 1 and line[i] in bits:
+ number_flag = 3
+ if line[i] == "千":
+ if i < len(line) - 1:
+ if line[i + 1] in unit:
+ number_flag = -1
+ if number_flag == 3:
+ onebit = line[i]
+ bit.append(onebit)
+ elif number_flag == 3 and line[i] in bits:
+ onebit = bit[-1] + line[i]
+ if onebit in bits:
+ bit[-1] = onebit
+ else:
+ number_flag = -2
+ else:
+ number_flag = -1
+ if len(digit) > 0 and number_flag == -1:
+ number_flag = -2
+ if i == (len(line) - 1) and number_flag >= 0:
+ number_flag = -1
+ if number_flag < 0:
+ newdigit = ""
+ if len(digit) > 0: # and (len(digit) == len(bit))):
+ if len(bit) == 1 and zero_flag == 0 and bit[0] == "百" and len(bit) != len(digit):
+ bit.append("十")
+ if len(digit) == (len(bit) + 1):
+ bit.append("零")
+ if len(digit) == len(bit):
+ for m in range(len(digit))[-1::-1]:
+ if int(bits[bit[m]]) == int(len(newdigit) + 1):
+ newdigit += digit[m]
+ else:
+ nu = int(bits[bit[m]]) - len(newdigit) - 1
+ for n in range(nu):
+ newdigit += "0"
+ newdigit += digit[m]
+ for z in range(len(newdigit))[-1::-1]:
+ newline += newdigit[z]
+ else:
+ newline += "".join(digit)
+ bit = []
+ digit = []
+ zero_flag = 0
+ else:
+ newline += line[i]
+ if number_flag == -2:
+ newline += line[i]
+ number_flag = 0
+ return newline
+
+
+def special(line):
+ newline = ""
+ for e in range(len(line)):
+ if ord(line[e]) == 247:
+ newline += "除以"
+ elif ord(line[e]) == 215:
+ newline += "乘以"
+ elif ord(line[e]) == 61:
+ newline += "等于"
+ elif ord(line[e]) == 43:
+ newline += "加"
+ elif ord(line[e]) == 45:
+ newline += "负"
+ elif ord(line[e]) == 8451:
+ newline += "摄氏度"
+ elif ord(line[e]) == 13217:
+ newline += "平方米"
+ elif ord(line[e]) == 8240 or ord(line[e]) == 65130:
+ newline += "%"
+ elif ord(line[e]) == 46:
+ newline += "点"
+ elif ord(line[e]) == 176:
+ newline += "度"
+ angel = 1
+ elif ord(line[e]) == 8242 and angel == 1:
+ newline += "分"
+ else:
+ newline += line[e]
+ return newline
+
+
+def all_convert(content):
+ content = recoformat(content)
+ content = numbersingle(content)
+ content = ch_number2digit(content)
+ content = special(content)
+ content = scoreformat("", content)
+ return content
+
+
+if __name__ == "__main__":
+ if len(sys.argv[1:]) < 1:
+ sys.stderr.write("Usage:\n .py reco.result\n")
+ sys.stderr.write(" reco.result: id<tab>recoresult\n")
+ sys.exit(1)
+ f = open(sys.argv[1])
+ flag = 0
+ if len(sys.argv[1:]) > 1:
+ flag = int(sys.argv[2])
+ for line in f.readlines():
+ if not line:
+ continue
+ line = line.rstrip()
+ tmp = line.split("\t")
+ if len(tmp) < 2:
+ tmp = line.split(",")
+ if len(tmp) < 2:
+ tmp = line.split(" ", 1)
+ if len(tmp) < 2:
+ name = tmp[0]
+ content = ""
+ print(content)
+ continue
+ name = tmp[0]
+ content = tmp[1]
+ name = re.sub("\.pcm", "", name)
+ name = re.sub("\.wav", "", name)
+ content = recoformat(content)
+ content = numbersingle(content)
+ content = ch_number2digit(content)
+ content = special(content)
+ content = scoreformat(name, content, flag)
+ print(content)
+ f.close()
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/tools/scp2jsonl.py b/examples/industrial_data_pretraining/fun_asr_nano/tools/scp2jsonl.py
new file mode 100644
index 000000000..488173ff9
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/tools/scp2jsonl.py
@@ -0,0 +1,142 @@
+import hydra
+import json
+import os
+import threading
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from io import BytesIO
+from typing import Dict, Optional, Tuple
+from urllib.request import urlopen
+
+import soundfile as sf
+from modelscope import AutoTokenizer
+from tqdm import tqdm
+from omegaconf import DictConfig, OmegaConf, ListConfig
+
+
+class LineProcessor:
+ def __init__(self, tokenizer):
+ self.tokenizer = tokenizer
+ self.lock = threading.Lock()
+
+ def process_line(self, line_pair: Tuple[str, str]) -> Optional[Dict]:
+ line1, line2 = line_pair
+
+ line1, line2 = line1.strip(), line2.strip()
+ if not line1 or not line2:
+ return None
+
+ parts1, parts2 = line1.split(maxsplit=1), line2.split(maxsplit=1)
+ if len(parts1) != 2 or len(parts2) != 2:
+ return None
+
+ utt1, utt2 = parts1[0], parts2[0]
+ wav_path, text = parts1[1], parts2[1]
+
+ if utt1 != utt2:
+ return {"error": f"UTT mismatch: {utt1} vs {utt2}"}
+
+ try:
+ if wav_path.startswith("http"):
+ response = urlopen(wav_path)
+ if response.status != 200:
+ return {"error": f"WAV not found: {wav_path}"}
+ audio_file = BytesIO(response.read())
+ duration = sf.info(audio_file).duration
+ else:
+ if not os.path.exists(wav_path):
+ return {"error": f"WAV not found: {wav_path}"}
+ duration = sf.info(wav_path).duration
+
+ data = {
+ "messages": [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {
+ "role": "user",
+ "content": f"语音转写:<|startofspeech|>!{wav_path}<|endofspeech|>",
+ },
+ {"role": "assistant", "content": text},
+ ],
+ "speech_length": int((duration * 1000 - 25) // 10 + 1),
+ "text_length": len(self.tokenizer.tokenize(text)),
+ }
+ return {"success": data, "utt": utt1}
+
+ except Exception as e:
+ return {"error": f"Error processing {wav_path}: {str(e)}"}
+
+
[email protected](config_name=None, version_base=None)
+def main_hydra(cfg: DictConfig):
+ def to_plain_list(cfg_item):
+ if isinstance(cfg_item, ListConfig):
+ return OmegaConf.to_container(cfg_item, resolve=True)
+ elif isinstance(cfg_item, DictConfig):
+ return {k: to_plain_list(v) for k, v in cfg_item.items()}
+ else:
+ return cfg_item
+
+ kwargs = to_plain_list(cfg)
+ scp_file = kwargs["scp_file"]
+ transcript_file = kwargs["transcript_file"]
+ max_workers = kwargs.get("max_workers", os.cpu_count())
+ jsonl_file = kwargs["jsonl_file"]
+
+ with open(scp_file, "r") as f1, open(transcript_file, "r") as f2:
+ scp_lines = f1.readlines()
+ transcript_lines = f2.readlines()
+
+ if len(scp_lines) != len(transcript_lines):
+ print(
+ f"Warning: Line count mismatch - scp: {len(scp_lines)}, transcript: {len(transcript_lines)}"
+ )
+
+ tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
+ processor = LineProcessor(tokenizer)
+
+ data_pairs = list(zip(scp_lines, transcript_lines))
+
+ processed_count = 0
+ failed_count = 0
+ error_messages = []
+
+ with tqdm(total=len(data_pairs), desc="Processing") as pbar:
+ with ThreadPoolExecutor(max_workers=max_workers) as executor:
+ with open(jsonl_file, "w") as f_out:
+ futures = {
+ executor.submit(processor.process_line, pair): i
+ for i, pair in enumerate(data_pairs)
+ }
+
+ for future in as_completed(futures):
+ result = future.result()
+
+ if result and "success" in result:
+ with processor.lock:
+ json.dump(result["success"], f_out, ensure_ascii=False)
+ f_out.write("\n")
+ processed_count += 1
+ elif result and "error" in result:
+ failed_count += 1
+ error_messages.append(result["error"])
+
+ pbar.update(1)
+ pbar.set_postfix({"processed": processed_count, "failed": failed_count})
+
+ print(f"\nProcessing completed:")
+ print(f" Total lines: {len(data_pairs)}")
+ print(f" Successfully processed: {processed_count}")
+ print(f" Failed: {failed_count}")
+
+ if error_messages and len(error_messages) <= 10:
+ print(f"\nSample errors:")
+ for error in error_messages[:10]:
+ print(f" - {error}")
+ elif error_messages:
+ print(f"\nFirst 10 errors:")
+ for error in error_messages[:10]:
+ print(f" - {error}")
+ print(f" ... and {len(error_messages) - 10} more errors")
+
+
+if __name__ == "__main__":
+ main_hydra()
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/tools/utils.py b/examples/industrial_data_pretraining/fun_asr_nano/tools/utils.py
new file mode 100644
index 000000000..7125bca5c
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/tools/utils.py
@@ -0,0 +1,57 @@
+from itertools import groupby
+
+import soundfile as sf
+import torch
+import torchaudio
+import torchaudio.functional as F
+
+
+def load_audio(wav_path, rate: int = None, offset: float = 0, duration: float = None):
+ with sf.SoundFile(wav_path) as f:
+ start_frame = int(offset * f.samplerate)
+ if duration is None:
+ frames_to_read = f.frames - start_frame
+ else:
+ frames_to_read = int(duration * f.samplerate)
+ f.seek(start_frame)
+ audio_data = f.read(frames_to_read, dtype="float32")
+ audio_tensor = torch.from_numpy(audio_data)
+ if rate is not None and f.samplerate != rate:
+ if audio_tensor.ndim == 1:
+ audio_tensor = audio_tensor.unsqueeze(0)
+ else:
+ audio_tensor = audio_tensor.T
+ resampler = torchaudio.transforms.Resample(orig_freq=f.samplerate, new_freq=rate)
+ audio_tensor = resampler(audio_tensor)
+ if audio_tensor.shape[0] == 1:
+ audio_tensor = audio_tensor.squeeze(0)
+ return audio_tensor, rate if rate is not None else f.samplerate
+
+
+def forced_align(log_probs: torch.Tensor, targets: torch.Tensor, blank: int = 0):
+ items = []
+ try:
+ # The current version only supports batch_size==1.
+ log_probs, targets = log_probs.unsqueeze(0).cpu(), targets.unsqueeze(0).cpu()
+ assert log_probs.shape[1] >= targets.shape[1]
+ alignments, scores = F.forced_align(log_probs, targets, blank=blank)
+ alignments, scores = alignments[0], torch.exp(scores[0]).tolist()
+ # use enumerate to keep track of the original indices, then group by token value
+ for token, group in groupby(enumerate(alignments), key=lambda item: item[1]):
+ if token == blank:
+ continue
+ group = list(group)
+ start = group[0][0]
+ end = start + len(group)
+ score = max(scores[start:end])
+ items.append(
+ {
+ "token": token.item(),
+ "start_time": start,
+ "end_time": end,
+ "score": round(score, 3),
+ }
+ )
+ except:
+ pass
+ return items
diff --git a/examples/industrial_data_pretraining/fun_asr_nano/tools/whisper_mix_normalize.py b/examples/industrial_data_pretraining/fun_asr_nano/tools/whisper_mix_normalize.py
new file mode 100644
index 000000000..632b52906
--- /dev/null
+++ b/examples/industrial_data_pretraining/fun_asr_nano/tools/whisper_mix_normalize.py
@@ -0,0 +1,135 @@
+# -*- coding: utf-8 -*-
+#!/usr/bin/python
+# Author: Mengze Chen
+
+import re
+import sys
+
+import cn_tn as cn_tn
+import format5res as cn_itn
+import pyopenjtalk
+import zhconv
+from whisper_normalizer.basic import BasicTextNormalizer
+from whisper_normalizer.english import EnglishTextNormalizer
+
+basic_normalizer = BasicTextNormalizer()
+english_normalizer = EnglishTextNormalizer()
+
+
+def is_only_chinese_and_english(s):
+ # 定义正则表达式模式,匹配中文字符范围和英文字母(包括大小写)
+ pattern = r"^[\u4e00-\u9fa5A-Za-z0-9,\.!\?:;,。!?:;、%\'\s\-\~]+$"
+ # 使用正则表达式进行匹配
+ return re.match(pattern, s) is not None
+
+
+def is_only_english(s):
+ # 定义正则表达式模式,匹配中文字符范围和英文字母(包括大小写)
+ pattern = r"^[A-Za-z0-9,\.!\?:;,。!?:;、%\'\s\-\~]+$"
+ # 使用正则表达式进行匹配
+ return re.match(pattern, s) is not None
+
+
+def is_number(s):
+ # 定义正则表达式模式,匹配中文字符范围和英文字母(包括大小写)
+ pattern = r"^[0-9,\.!\?:;,。!?:;、%\'\s]+$"
+ # 使用正则表达式进行匹配
+ return re.match(pattern, s) is not None
+
+
+def safe_ja_g2p(text, kana=True, max_length=100):
+ if len(text) > max_length:
+ # 如果文本过长,分段处理
+ parts = []
+ for i in range(0, len(text), max_length):
+ part = text[i : i + max_length]
+ try:
+ converted = pyopenjtalk.g2p(part, kana=kana)
+ parts.append(converted)
+ except:
+ parts.append(part) # 如果转换失败,使用原文本
+ return " ".join(parts)
+ else:
+ try:
+ return pyopenjtalk.g2p(text, kana=kana)
+ except:
+ return text # 如果转换失败,返回原文本
+
+
+def normalize_text(srcfn, dstfn, kana=False):
+ with open(srcfn, "r") as f_read, open(dstfn, "w") as f_write:
+ all_lines = f_read.readlines()
+ for line in all_lines:
+ line = line.strip()
+ line_arr = line.split(maxsplit=1)
+ if len(line_arr) < 1:
+ continue
+ if len(line_arr) == 1:
+ line_arr.append("")
+ key = line_arr[0]
+ line_arr[1] = re.sub(r"=", " ", line_arr[1])
+ line_arr[1] = re.sub(r"\(", " ", line_arr[1])
+ line_arr[1] = re.sub(r"\)", " ", line_arr[1])
+ # From Chongjia Ni
+ if kana:
+ line_arr[1] = safe_ja_g2p(line_arr[1], kana=True, max_length=100)
+
+ line_arr = f"{key}\t{line_arr[1]}".split()
+ conts = []
+ language_bak = ""
+ part = []
+ for i in range(1, len(line_arr)):
+ out_part = ""
+ chn_eng_bool = is_only_chinese_and_english(line_arr[i])
+ eng_bool = is_only_english(line_arr[i])
+ num_bool = is_number(line_arr[i])
+ if eng_bool and not num_bool:
+ language = "en"
+ elif chn_eng_bool:
+ language = "chn_en"
+ else:
+ language = "not_chn_en"
+ if language == language_bak or language_bak == "":
+ part.append(line_arr[i])
+ language_bak = language
+ else:
+ if language_bak == "en":
+ out_part1 = english_normalizer(" ".join(part))
+ out_part = cn_itn.scoreformat("", out_part1)
+ elif language_bak == "chn_en":
+ out_part1 = english_normalizer(" ".join(part))
+ out_part2 = cn_tn.normalize_nsw(out_part1)
+ out_part3 = cn_itn.all_convert(out_part2)
+ out_part = zhconv.convert(out_part3, "zh-cn")
+ else:
+ out_part1 = basic_normalizer(" ".join(part))
+ out_part2 = cn_tn.normalize_nsw(out_part1)
+ out_part3 = cn_itn.all_convert(out_part2)
+ out_part = zhconv.convert(out_part3, "zh-cn")
+ conts.append(out_part)
+ language_bak = language
+ part = []
+ part.append(line_arr[i])
+ if i == len(line_arr) - 1:
+ if language == "en":
+ out_part1 = english_normalizer(" ".join(part))
+ out_part = cn_itn.scoreformat("", out_part1)
+ elif language == "chn_en":
+ out_part1 = english_normalizer(" ".join(part))
+ out_part2 = cn_tn.normalize_nsw(out_part1)
+ out_part3 = cn_itn.all_convert(out_part2)
+ out_part = zhconv.convert(out_part3, "zh-cn")
+ else:
+ out_part1 = basic_normalizer(" ".join(part))
+ out_part2 = cn_tn.normalize_nsw(out_part1)
+ out_part3 = cn_itn.all_convert(out_part2)
+ out_part = zhconv.convert(out_part3, "zh-cn")
+ conts.append(out_part)
+
+ f_write.write("{0}\t{1}\n".format(key, " ".join(conts).strip()))
+
+
+if __name__ == "__main__":
+ srcfn = sys.argv[1]
+ dstfn = sys.argv[2]
+ normalize_text(srcfn, dstfn, True if len(sys.argv) > 3 else False)
diff --git a/funasr/models/fun_asr_nano/ctc.py b/funasr/models/fun_asr_nano/ctc.py
new file mode 100644
index 000000000..f2d592759
--- /dev/null
+++ b/funasr/models/fun_asr_nano/ctc.py
@@ -0,0 +1,60 @@
+import torch
+import torch.nn.functional as F
+
+
+class CTC(torch.nn.Module):
+ """CTC module.
+
+ Args:
+ odim: dimension of outputs
+ encoder_output_size: number of encoder projection units
+ dropout_rate: dropout rate (0.0 ~ 1.0)
+ reduce: reduce the CTC loss into a scalar
+ """
+
+ def __init__(
+ self,
+ odim: int,
+ encoder_output_size: int,
+ dropout_rate: float = 0.0,
+ reduce: bool = True,
+ blank_id: int = 0,
+ **kwargs,
+ ):
+ super().__init__()
+ eprojs = encoder_output_size
+ self.dropout_rate = dropout_rate
+ self.ctc_lo = torch.nn.Linear(eprojs, odim)
+ self.blank_id = blank_id
+ self.ctc_loss = torch.nn.CTCLoss(reduction="none", blank=blank_id)
+ self.reduce = reduce
+
+ def softmax(self, hs_pad):
+ """softmax of frame activations
+
+ Args:
+ Tensor hs_pad: 3d tensor (B, Tmax, eprojs)
+ Returns:
+ torch.Tensor: softmax applied 3d tensor (B, Tmax, odim)
+ """
+ return F.softmax(self.ctc_lo(hs_pad), dim=2)
+
+ def log_softmax(self, hs_pad):
+ """log_softmax of frame activations
+
+ Args:
+ Tensor hs_pad: 3d tensor (B, Tmax, eprojs)
+ Returns:
+ torch.Tensor: log softmax applied 3d tensor (B, Tmax, odim)
+ """
+ return F.log_softmax(self.ctc_lo(hs_pad), dim=2)
+
+ def argmax(self, hs_pad):
+ """argmax of frame activations
+
+ Args:
+ torch.Tensor hs_pad: 3d tensor (B, Tmax, eprojs)
+ Returns:
+ torch.Tensor: argmax applied 2d tensor (B, Tmax)
+ """
+ return torch.argmax(self.ctc_lo(hs_pad), dim=2)
diff --git a/funasr/models/fun_asr_nano/model.py b/funasr/models/fun_asr_nano/model.py
index 49294d405..308740083 100644
--- a/funasr/models/fun_asr_nano/model.py
+++ b/funasr/models/fun_asr_nano/model.py
@@ -1,4 +1,3 @@
-import json
import logging
import os
import random
@@ -6,9 +5,11 @@ import re
import string
import time
import traceback
+from typing import Union
import torch
import torch.nn as nn
+
from funasr.metrics.compute_acc import compute_accuracy
from funasr.register import tables
from funasr.train_utils.device_funcs import force_gatherable, to_device
@@ -16,6 +17,9 @@ from funasr.utils.datadir_writer import DatadirWriter
from funasr.utils.load_utils import extract_fbank, load_audio_text_image_video
from transformers import AutoConfig, AutoModelForCausalLM
+from ctc import CTC
+from tools.utils import forced_align
+
dtype_map = {"bf16": torch.bfloat16, "fp16": torch.float16, "fp32": torch.float32}
@@ -42,6 +46,7 @@ class FunASRNano(nn.Module):
)
if hub == "ms":
from funasr import AutoModel
+
model = AutoModel(model=audio_encoder, model_revision="master")
audio_encoder_output_size = (
model.model.encoder_output_size
@@ -49,22 +54,20 @@ class FunASRNano(nn.Module):
else -1
)
audio_encoder = (
- model.model.model.encoder
- if hasattr(model.model, "model")
- else model.model.encoder
+ model.model.model.encoder if hasattr(model.model, "model") else model.model.encoder
)
else:
encoder_class = tables.encoder_classes.get(audio_encoder)
audio_encoder = encoder_class(input_size=input_size, **audio_encoder_conf)
audio_encoder_output_size = audio_encoder.output_size()
freeze = audio_encoder_conf.get("freeze", True)
- freeze_layer_num = int(audio_encoder_conf.get("freeze_layer_num", -1))
if freeze:
- for name, param in audio_encoder.named_parameters():
+ for _, param in audio_encoder.named_parameters():
param.requires_grad = False
audio_encoder.eval()
self.audio_encoder = audio_encoder
+
# llm
self.llm = None
init_param_path = llm_conf.get("init_param_path", None)
@@ -76,31 +79,9 @@ class FunASRNano(nn.Module):
freeze = llm_conf.get("freeze", True)
if freeze:
- for name, param in model.named_parameters():
+ for _, param in model.named_parameters():
param.requires_grad = False
model.eval()
- logging.info(f"use_lora: {llm_conf.get('use_lora', False)}")
- if llm_conf.get("use_lora", False):
- from omegaconf import DictConfig, OmegaConf
-
- lora_conf = llm_conf.get("lora_conf", {})
- if isinstance(lora_conf, (OmegaConf, DictConfig)):
- lora_conf = OmegaConf.to_container(lora_conf, resolve=True)
- from peft import LoraConfig, PeftModel, get_peft_model
-
- lora_init_param_path = lora_conf.get("init_param_path", None)
- if lora_init_param_path is not None:
- logging.info(f"lora_init_param_path: {lora_init_param_path}")
- model = PeftModel.from_pretrained(model, lora_init_param_path)
- for name, param in model.named_parameters():
- if not lora_conf.get("freeze_lora", False):
- if "lora_" in name:
- param.requires_grad = True
- else:
- peft_config = LoraConfig(**lora_conf)
- model = get_peft_model(model, peft_config)
- model.print_trainable_parameters()
-
if llm_conf.get("activation_checkpoint", False):
model.gradient_checkpointing_enable()
@@ -118,13 +99,62 @@ class FunASRNano(nn.Module):
audio_adaptor = adaptor_class(**audio_adaptor_conf)
freeze = audio_adaptor_conf.get("freeze", False)
if freeze:
- for name, param in audio_adaptor.named_parameters():
+ for _, param in audio_adaptor.named_parameters():
param.requires_grad = False
audio_adaptor.eval()
self.audio_adaptor = audio_adaptor
+ self.use_low_frame_rate = audio_adaptor_conf.get("use_low_frame_rate", False)
+
+ # ctc decoder
+ self.ctc_decoder = None
+ # TODO: fix table name
+ ctc_decoder_class = tables.adaptor_classes.get(kwargs.get("ctc_decoder", None))
+ if ctc_decoder_class is not None:
+ ctc_tokenizer = (
+ kwargs.get("ctc_tokenizer", None)
+ if "ctc_tokenizer" in kwargs
+ else kwargs["dataset_conf"]["ctc_tokenizer"]
+ )
+ ctc_tokenizer_conf = (
+ kwargs.get("ctc_tokenizer_conf", None)
+ if "ctc_tokenizer_conf" in kwargs
+ else kwargs["dataset_conf"]["ctc_tokenizer_conf"]
+ )
+ if ctc_tokenizer is not None and ctc_tokenizer_conf is not None:
+ ctc_tokenizer_class = tables.tokenizer_classes.get(ctc_tokenizer)
+ ctc_tokenizer = ctc_tokenizer_class(**ctc_tokenizer_conf)
+ self.ctc_tokenizer = ctc_tokenizer
+ assert ctc_tokenizer is not None, f"ctc_tokenizer must be set"
+
+ ctc_vocab_size = kwargs.get("ctc_vocab_size", 60515)
+ ctc_decoder_conf = kwargs.get("ctc_decoder_conf", {})
+ if audio_encoder_output_size > 0:
+ ctc_decoder_conf["encoder_dim"] = audio_encoder_output_size
+ self.ctc_decoder = ctc_decoder_class(**ctc_decoder_conf)
+ init_param_path = ctc_decoder_conf.get("init_param_path", None)
+ if init_param_path is not None:
+ src_state = torch.load(init_param_path, map_location="cpu")
+ flag = self.ctc_decoder.load_state_dict(src_state, strict=False)
+ logging.info(f"Loading ctc_decoder ckpt: {init_param_path}, status: {flag}")
+ freeze = ctc_decoder_conf.get("freeze", False)
+ if freeze:
+ for _, param in self.ctc_decoder.named_parameters():
+ param.requires_grad = False
+ self.ctc_decoder.eval()
+
+ ctc_conf = kwargs.get("ctc_conf", {})
+ self.blank_id = ctc_conf.get("blank_id", ctc_vocab_size - 1)
+ self.ctc_weight = kwargs.get("ctc_weight", 0.3)
+ self.ctc = CTC(
+ odim=ctc_vocab_size,
+ encoder_output_size=audio_encoder_output_size,
+ blank_id=self.blank_id,
+ **ctc_conf,
+ )
+ self.detach_ctc_decoder = kwargs.get("detach_ctc_decoder", True)
+ self.error_calculator = None
self.length_normalized_loss = length_normalized_loss
- self.feat_permute = audio_encoder_conf.get("feat_permute", True)
rank = int(os.environ.get("RANK", 0))
logging.info(f"rank: {rank}, model is builded.")
@@ -159,9 +189,7 @@ class FunASRNano(nn.Module):
encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
# audio_adaptor
- encoder_out, encoder_out_lens = self.audio_adaptor(
- encoder_out, encoder_out_lens
- )
+ encoder_out, encoder_out_lens = self.audio_adaptor(encoder_out, encoder_out_lens)
batch_size, token_num, dims = inputs_embeds.shape
fake_token_len = kwargs.get("fake_token_len")
@@ -200,11 +228,11 @@ class FunASRNano(nn.Module):
stats["batch_size_speech"] = batch_size_speech
stats["batch_size_x_frames"] = frames * batch_size_speech
stats["batch_size_real_frames"] = speech_lengths.sum().item()
- stats["padding_frames"] = (
- stats["batch_size_x_frames"] - stats["batch_size_real_frames"]
- )
+ stats["padding_frames"] = stats["batch_size_x_frames"] - stats["batch_size_real_frames"]
- with torch.cuda.amp.autocast(
+ device_type = next(self.parameters()).device.type
+ with torch.autocast(
+ device_type=device_type if device_type in ["cuda", "xpu", "mps"] else "cpu",
enabled=True if self.llm_dtype != "fp32" else False,
dtype=dtype_map[self.llm_dtype],
):
@@ -219,9 +247,7 @@ class FunASRNano(nn.Module):
with torch.no_grad():
preds = torch.argmax(model_outputs.logits, -1)
- acc_att = compute_accuracy(
- preds[:, :-1], labels_ids[:, 1:], ignore_label=-100
- )
+ acc_att = compute_accuracy(preds[:, :-1], labels_ids[:, 1:], ignore_label=-100)
stats["acc"] = acc_att
stats["loss"] = torch.clone(loss.detach())
@@ -229,9 +255,7 @@ class FunASRNano(nn.Module):
stats["batch_size_x_tokens"] = token_num * batch_size
stats["batch_size_real_tokens"] = attention_mask.sum().item()
- stats["padding_tokens"] = (
- stats["batch_size_x_tokens"] - stats["batch_size_real_tokens"]
- )
+ stats["padding_tokens"] = stats["batch_size_x_tokens"] - stats["batch_size_real_tokens"]
dialog_turns = (fbank_beg > 0).sum(-1)
dialog_turns_max = torch.max(dialog_turns).int().item()
@@ -252,12 +276,7 @@ class FunASRNano(nn.Module):
def encode(self, speech, speech_lengths):
# audio encoder
- if self.feat_permute:
- encoder_out, encoder_out_lens = self.audio_encoder(
- speech.permute(0, 2, 1), speech_lengths
- )
- else:
- encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)
+ encoder_out, encoder_out_lens = self.audio_encoder(speech, speech_lengths)
return encoder_out, encoder_out_lens
@@ -286,9 +305,7 @@ class FunASRNano(nn.Module):
return contents
- def data_load_speech(
- self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs
- ):
+ def data_load_speech(self, contents: dict, tokenizer, frontend, meta_data={}, **kwargs):
system = contents["system"]
user = contents["user"]
assistant = contents["assistant"]
@@ -309,9 +326,7 @@ class FunASRNano(nn.Module):
[],
)
input_source_ids = []
- for i, (system_prompt, user_prompt, target_out) in enumerate(
- zip(system, user, assistant)
- ):
+ for i, (system_prompt, user_prompt, target_out) in enumerate(zip(system, user, assistant)):
if i >= kwargs.get("multiturn_num_max", 5):
break
if len(input_ids) > kwargs.get("max_token_length", 1500):
@@ -326,14 +341,20 @@ class FunASRNano(nn.Module):
else:
source_input = f"<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"
if not sys_prompt:
- source_input = f"<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"
+ source_input = (
+ f"<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"
+ )
else:
if kwargs.get("infer_with_assistant_input", False):
source_input = f"<|im_start|>user\n{user_prompt}"
else:
- source_input = f"<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"
+ source_input = (
+ f"<|im_start|>user\n{user_prompt}<|im_end|>\n<|im_start|>assistant\n"
+ )
if not do_think:
source_input += "<think>\n\n</think>\n\n"
+ if kwargs.get("prev_text", None) is not None:
+ source_input += kwargs["prev_text"]
splits = pattern.split(source_input)
source_ids = []
@@ -362,9 +383,7 @@ class FunASRNano(nn.Module):
time2 = time.perf_counter()
meta_data["load_data"] = f"{time2 - time1:0.3f}"
except Exception as e:
- logging.error(
- f"Loading wav failed! {str(e)}, {traceback.format_exc()}"
- )
+ logging.error(f"Loading wav failed! {str(e)}, {traceback.format_exc()}")
speech, speech_lengths = extract_fbank(
data_src,
@@ -382,12 +401,12 @@ class FunASRNano(nn.Module):
/ 1000
)
- if self.feat_permute:
- speech = speech.permute(0, 2, 1)
-
- olens = 1 + (speech_lengths[0].item() - 3 + 2 * 1) // 2
- olens = 1 + (olens - 3 + 2 * 1) // 2
- fake_token_len_i = (olens - 1) // 2 + 1
+ if self.use_low_frame_rate:
+ olens = 1 + (speech_lengths[0].item() - 3 + 2 * 1) // 2
+ olens = 1 + (olens - 3 + 2 * 1) // 2
+ fake_token_len_i = (olens - 1) // 2 + 1
+ else:
+ fake_token_len_i = speech_lengths[0].item()
fake_token = [0] * fake_token_len_i
fbank_beg_i = len(source_ids)
source_ids += fake_token
@@ -406,9 +425,7 @@ class FunASRNano(nn.Module):
fbank.append(speech[0, :, :])
fbank_lens.append(speech_lengths)
- input_ids = torch.tensor(
- input_ids, dtype=torch.int64
- ) # [: self.max_token_length]
+ input_ids = torch.tensor(input_ids, dtype=torch.int64) # [: self.max_token_length]
attention_mask = torch.tensor([1] * len(input_ids), dtype=torch.int32)
labels = torch.tensor(labels, dtype=torch.int64) # [: self.max_token_length]
@@ -419,9 +436,7 @@ class FunASRNano(nn.Module):
target_ids = torch.tensor(target_ids, dtype=torch.int64)
if len(fbank) > 0:
- speech = torch.nn.utils.rnn.pad_sequence(
- fbank, batch_first=True, padding_value=0.0
- )
+ speech = torch.nn.utils.rnn.pad_sequence(fbank, batch_first=True, padding_value=0.0)
speech_lengths = torch.nn.utils.rnn.pad_sequence(
fbank_lens, batch_first=True, padding_value=-1
)
@@ -458,9 +473,7 @@ class FunASRNano(nn.Module):
raise NotImplementedError("batch decoding is not implemented")
contents = self.data_template(data_in[0])
- output = self.data_load_speech(
- contents, tokenizer, frontend, meta_data=meta_data, **kwargs
- )
+ output = self.data_load_speech(contents, tokenizer, frontend, meta_data=meta_data, **kwargs)
batch = to_device(output, kwargs["device"])
# audio encoder
@@ -481,18 +494,18 @@ class FunASRNano(nn.Module):
encoder_out, encoder_out_lens = self.encode(speech, speech_lengths)
# audio_adaptor
- encoder_out, encoder_out_lens = self.audio_adaptor(
- encoder_out, encoder_out_lens
- )
- meta_data["audio_adaptor_out"] = encoder_out
- meta_data["audio_adaptor_out_lens"] = encoder_out_lens
+ adaptor_out, adaptor_out_lens = self.audio_adaptor(encoder_out, encoder_out_lens)
+ meta_data["encoder_out"] = encoder_out
+ meta_data["encoder_out_lens"] = encoder_out_lens
+ meta_data["audio_adaptor_out"] = adaptor_out
+ meta_data["audio_adaptor_out_lens"] = adaptor_out_lens
input_ids = batch["input_ids"]
source_ids = batch["source_ids"]
fbank_beg = batch["fbank_beg"]
fake_token_len = batch["fake_token_len"]
- if not kwargs.get("tearchforing", False):
+ if not kwargs.get("teacherforcing", False):
input_ids = source_ids
input_ids[input_ids < 0] = 0
@@ -509,7 +522,7 @@ class FunASRNano(nn.Module):
fbank_beg_idx = fbank_beg[batch_idx, turn_id].item()
if fbank_beg_idx > 0:
speech_token_len = fake_token_len[batch_idx, turn_id]
- speech_token = encoder_out[speech_idx, :speech_token_len, :]
+ speech_token = adaptor_out[speech_idx, :speech_token_len, :]
try:
inputs_embeds[
@@ -521,10 +534,10 @@ class FunASRNano(nn.Module):
#
logging.error(f"{str(e)}, {traceback.format_exc()}")
logging.info(
- f"batch_idx: {batch_idx}, inputs_embeds: {inputs_embeds.shape}, fbank_beg_idx: {fbank_beg_idx}, speech_token_len: {speech_token_len}, encoder_out: {encoder_out.shape}, encoder_out_lens: {encoder_out_lens}, fake_token_len: {fake_token_len}, speech_lengths: {speech_lengths}"
+ f"batch_idx: {batch_idx}, inputs_embeds: {inputs_embeds.shape}, fbank_beg_idx: {fbank_beg_idx}, speech_token_len: {speech_token_len}, adaptor_out: {adaptor_out.shape}, adaptor_out_lens: {adaptor_out_lens}, fake_token_len: {fake_token_len}, speech_lengths: {speech_lengths}"
)
- speech_token_len = encoder_out_lens[speech_idx].item()
- speech_token = encoder_out[speech_idx, :speech_token_len, :]
+ speech_token_len = adaptor_out_lens[speech_idx].item()
+ speech_token = adaptor_out[speech_idx, :speech_token_len, :]
inputs_embeds[
batch_idx,
fbank_beg_idx : fbank_beg_idx + speech_token_len,
@@ -534,69 +547,58 @@ class FunASRNano(nn.Module):
speech_idx += 1
return inputs_embeds, contents, batch, source_ids, meta_data
- def inference(
- self,
- data_in,
- data_lengths=None,
- key: list = None,
- tokenizer=None,
- frontend=None,
- **kwargs,
- ):
- hotwords = kwargs.get("hotwords", [])
+ def get_prompt(self, hotwords: list[str], language: str = None, itn: bool = True):
if len(hotwords) > 0:
hotwords = ", ".join(hotwords)
prompt = f"请结合上下文信息,更加准确地完成语音转写任务。如果没有相关信息,我们会留空。\n\n\n**上下文信息:**\n\n\n"
prompt += f"热词列表:[{hotwords}]\n"
else:
prompt = ""
- language = kwargs.get("language", "auto")
- if language not in ("auto", "zh", "en", "ja"):
- language = "auto"
- if language == "auto":
+ if language is None:
prompt += "语音转写"
else:
- LANGUAGE_MAP = {"zh": "中文", "en": "英文", "ja": "日文"}
- prompt += f"语音转写成{LANGUAGE_MAP[language]}"
- itn = kwargs.get("itn", True)
+ prompt += f"语音转写成{language}"
if not itn:
prompt += ",不进行文本规整"
- prompt += ":"
-
- new_data_in = []
- for data in data_in:
- if isinstance(data, str):
- new_data_in.append(
- [
- {"role": "system", "content": "You are a helpful assistant."},
- {
- "role": "user",
- "content": f"{prompt}<|startofspeech|>!{data}<|endofspeech|>",
- },
- {"role": "assistant", "content": "null"},
- ]
- )
- elif isinstance(data, torch.Tensor):
- new_data_in.append(
- [
- {"role": "system", "content": "You are a helpful assistant."},
- {
- "role": "user",
- "content": f"{prompt}<|startofspeech|>!!<|endofspeech|>",
- "audio": data,
- },
- {"role": "assistant", "content": "null"},
- ]
- )
- data_in = new_data_in
+ return prompt + ":"
+
+ def generate_chatml(self, prompt: str, data: Union[str, torch.Tensor]):
+ if isinstance(data, str):
+ return [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {"role": "user", "content": f"{prompt}<|startofspeech|>!{data}<|endofspeech|>"},
+ {"role": "assistant", "content": "null"},
+ ]
+ elif isinstance(data, torch.Tensor):
+ return [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {
+ "role": "user",
+ "content": f"{prompt}<|startofspeech|>!!<|endofspeech|>",
+ "audio": data,
+ },
+ {"role": "assistant", "content": "null"},
+ ]
+
+ def inference(
+ self,
+ data_in,
+ data_lengths=None,
+ key: list = None,
+ tokenizer=None,
+ frontend=None,
+ **kwargs,
+ ):
+ prompt = self.get_prompt(
+ kwargs.get("hotwords", []), kwargs.get("language", None), kwargs.get("itn", True)
+ )
+ data_in = [self.generate_chatml(prompt, data) for data in data_in]
if key is None:
key = []
for _ in data_in:
chars = string.ascii_letters + string.digits
- key.append(
- "rand_key_" + "".join(random.choice(chars) for _ in range(13))
- )
+ key.append("rand_key_" + "".join(random.choice(chars) for _ in range(13)))
return self.inference_llm(
data_in,
@@ -619,22 +621,51 @@ class FunASRNano(nn.Module):
inputs_embeds, contents, batch, source_ids, meta_data = self.inference_prepare(
data_in, data_lengths, key, tokenizer, frontend, **kwargs
)
+
+ ctc_results = []
+ if self.ctc_decoder is not None:
+ encoder_out = meta_data["encoder_out"]
+ encoder_out_lens = meta_data["encoder_out_lens"]
+ decoder_out, decoder_out_lens = self.ctc_decoder(encoder_out, encoder_out_lens)
+ ctc_logits = self.ctc.log_softmax(decoder_out)
+
+ b, n, d = encoder_out.size()
+ if isinstance(key[0], (list, tuple)):
+ key = key[0]
+ if len(key) < b:
+ key = key * b
+ for i in range(b):
+ x = ctc_logits[i, : encoder_out_lens[i].item(), :]
+ yseq = x.argmax(dim=-1)
+ yseq = torch.unique_consecutive(yseq, dim=-1)
+ mask = yseq != self.blank_id
+ token_int = yseq[mask].tolist()
+ # Change integer-ids to tokens
+ text = self.ctc_tokenizer.decode(token_int)
+ ctc_results.append({"key": key[i], "text": text, "ctc_logits": x})
+
llm_dtype = kwargs.get("llm_dtype", "fp32")
if llm_dtype == "fp32":
llm_dtype = "fp16" if kwargs.get("fp16", False) else llm_dtype
llm_dtype = "bf16" if kwargs.get("bf16", False) else llm_dtype
- with torch.cuda.amp.autocast(
- enabled=True if llm_dtype != "fp32" else False, dtype=dtype_map[llm_dtype]
+ device_type = torch.device(kwargs.get("device", "cuda")).type
+ with torch.autocast(
+ device_type=device_type if device_type in ["cuda", "xpu", "mps"] else "cpu",
+ enabled=True if llm_dtype != "fp32" else False,
+ dtype=dtype_map[llm_dtype],
):
label = contents["assistant"][-1]
self.llm = self.llm.to(dtype_map[llm_dtype])
inputs_embeds = inputs_embeds.to(dtype_map[llm_dtype])
llm_kwargs = kwargs.get("llm_kwargs", {})
- if not kwargs.get("teachforing", False):
+ if not kwargs.get("teacherforcing", False):
+ attention_mask = batch.get("attention_mask", None)
generated_ids = self.llm.generate(
inputs_embeds=inputs_embeds,
+ attention_mask=attention_mask,
max_new_tokens=kwargs.get("max_length", 512),
+ pad_token_id=self.llm.config.pad_token_id or self.llm.config.eos_token_id,
**llm_kwargs,
)
@@ -652,6 +683,7 @@ class FunASRNano(nn.Module):
inputs_embeds=inputs_embeds,
attention_mask=attention_mask,
labels=labels_ids,
+ pad_token_id=self.llm.config.pad_token_id or self.llm.config.eos_token_id,
**llm_kwargs,
)
@@ -662,6 +694,7 @@ class FunASRNano(nn.Module):
skip_special_tokens=kwargs.get("skip_special_tokens", True),
)[0]
loss = model_outputs.loss.item()
+ response = kwargs.get("prev_text", "") + response
ibest_writer = None
if kwargs.get("output_dir") is not None:
@@ -673,7 +706,7 @@ class FunASRNano(nn.Module):
response_clean = re.sub(r"[^\w\s\u3000\u4e00-\u9fff]+", "", response)
result_i = {
"key": key[0],
- "text": re.sub(r'\s+', ' ', response.replace("/sil", " ")),
+ "text": re.sub(r"\s+", " ", response.replace("/sil", " ")),
"text_tn": response_clean,
"label": label,
}
@@ -681,9 +714,33 @@ class FunASRNano(nn.Module):
result_i["loss"] = loss
results.append(result_i)
+ for ctc_result, result in zip(ctc_results, results):
+ result["ctc_text"] = ctc_result["text"].replace("<|nospeech|>", "")
+ target_ids = torch.tensor(
+ self.ctc_tokenizer.encode(result["ctc_text"]), dtype=torch.int64
+ )
+ result["ctc_timestamps"] = forced_align(
+ ctc_result["ctc_logits"], target_ids, self.blank_id
+ )
+ target_ids = torch.tensor(self.ctc_tokenizer.encode(result["text"]), dtype=torch.int64)
+ result["timestamps"] = forced_align(ctc_result["ctc_logits"], target_ids, self.blank_id)
+ for timestamps in [result["timestamps"], result["ctc_timestamps"]]:
+ for timestamp in timestamps:
+ timestamp["token"] = self.ctc_tokenizer.decode([timestamp["token"]])
+ timestamp["start_time"] = timestamp["start_time"] * 6 * 10 / 1000
+ timestamp["end_time"] = timestamp["end_time"] * 6 * 10 / 1000
+
if ibest_writer is not None:
ibest_writer["text"][key[0]] = response.replace("\n", " ")
ibest_writer["label"][key[0]] = label.replace("\n", " ")
ibest_writer["text_tn"][key[0]] = response_clean
return results, meta_data
+
+ @staticmethod
+ def from_pretrained(model: str = None, **kwargs):
+ from funasr import AutoModel
+
+ model, kwargs = AutoModel.build_model(model=model, trust_remote_code=True, **kwargs)
+
+ return model, kwargs
diff --git a/funasr/models/fun_asr_nano/tools/__init__.py b/funasr/models/fun_asr_nano/tools/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/funasr/models/fun_asr_nano/tools/format5res.py b/funasr/models/fun_asr_nano/tools/format5res.py
new file mode 100644
index 000000000..3417337bb
--- /dev/null
+++ b/funasr/models/fun_asr_nano/tools/format5res.py
@@ -0,0 +1,301 @@
+# -*- coding: utf-8 -*-
+#!/usr/bin/python
+# Author: Mengze Chen
+
+import re
+import sys
+
+
+def scoreformat(name, line, flag=1):
+ newline = ""
+ for i in range(0, len(line)):
+ curr = line[i]
+ currEn = False
+ if curr == "":
+ continue
+ if (
+ (curr >= "\u0041" and curr <= "\u005a") # eng
+ or (curr >= "\u0061" and curr <= "\u007a") # eng
+ or (curr >= "\u0000" and curr <= "\u007f") # de fr es it
+ or (curr >= "\u0400" and curr <= "\u04ff") # ru
+ or (curr >= "\u0100" and curr <= "\u017f") # latin1
+ or (curr >= "\u0080" and curr <= "\u00ff") # latin2
+ or curr == "'"
+ ) and (curr < "\u0030" or curr > "\u0039"):
+ currEn = True
+ if i == 0:
+ newline = newline + curr
+ else:
+ if lastEn == True and currEn == True:
+ newline = newline + curr
+ else:
+ newline = newline + " " + curr
+ if flag == -1:
+ lastEn = False
+ else:
+ lastEn = currEn
+ ret = re.sub("[ ]{1,}", " ", newline)
+ ret = ret
+ if name == "":
+ ret = ret
+ else:
+ if flag <= 0:
+ ret = ret + " " + "(" + name + ")"
+ else:
+ ret = name + "\t" + ret
+ return ret
+
+
+def recoformat(line):
+ newline = ""
+ en_flag = 0 # 0: no-english 1 : english 2: former
+ for i in range(0, len(line)):
+ word = line[i]
+ if ord(word) == 32:
+ if en_flag == 0:
+ continue
+ else:
+ en_flag = 0
+ newline += " "
+ if (word >= "\u4e00" and word <= "\u9fa5") or (word >= "\u0030" and word <= "\u0039"):
+ if en_flag == 1:
+ newline += " " + word
+ else:
+ newline += word
+ en_flag = 0
+ elif (
+ (word >= "\u0041" and word <= "\u005a") # eng
+ or (word >= "\u0061" and word <= "\u007a") # eng
+ or (word >= "\u0000" and word <= "\u007f") # de fr es it
+ or (word >= "\u0400" and word <= "\u04ff") # ru
+ or (word >= "\u0100" and word <= "\u017f") # latin1
+ or (word >= "\u0080" and word <= "\u00ff") # latin2
+ or word == "'"
+ ):
+ if en_flag == 0:
+ newline += " " + ("" if (word == "'") else word)
+ else:
+ newline += word
+ en_flag = 1
+ else:
+ newline += " " + word
+ newline = newline
+ newline = re.sub("[ ]{1,}", " ", newline)
+ newline = newline
+ return newline
+
+
+def numbersingle(line):
+ chnu = ["零", "一", "二", "两", "三", "四", "五", "六", "七", "八", "九", "点"]
+ newline = ""
+ for id in range(len(line)):
+ if re.findall(r"\.", line[id]):
+ if re.findall(r"\.\s*$", line[id]):
+ newline += "."
+ else:
+ newline += chnu[10]
+ elif re.search(r"0", line[id]):
+ if id > 0 and id < len(line) - 1:
+ if (
+ re.search(r"\d", line[id - 1])
+ and (not re.search(r"\d", line[id + 1]))
+ and (not re.search(r"0", line[id - 1]))
+ ):
+ if id > 2 and len(line) > 2 and (not re.search(r"\d", line[id - 1])):
+ newline = newline[:-1]
+ newline += chnu[int(line[id - 1])] + "十"
+ else:
+ newline += chnu[int(line[id])]
+ else:
+ newline += chnu[int(line[id])]
+ else:
+ newline += chnu[int(line[id])]
+ elif re.search(r"\d", line[id]):
+ newline += chnu[int(line[id])]
+ else:
+ newline += line[id]
+ return newline
+
+
+def ch_number2digit(line):
+ number_flag = 0
+ zero_flag = 0
+ bits = {
+ "零": "1",
+ "十": "2",
+ "百": "3",
+ "千": "4",
+ "万": "5",
+ "十万": "6",
+ "百万": "7",
+ "千万": "8",
+ }
+ chsh = {
+ "一": "1",
+ "二": "2",
+ "三": "3",
+ "四": "4",
+ "五": "5",
+ "六": "6",
+ "七": "7",
+ "八": "8",
+ "九": "9",
+ "两": "2",
+ "幺": "1",
+ }
+ unit = {"里": "1", "克": "1", "米": "1"}
+ newline = ""
+ digit = []
+ bit = []
+ onebit = ""
+ for i in range(len(line)):
+ if ord(line[i]) == 32:
+ newline += " "
+ continue
+ if line[i] in chsh:
+ number_flag = 1
+ if line[i] == "两":
+ if (i == len(line) - 1) or (
+ (line[i + 1] not in chsh.keys()) and (line[i + 1] not in bits.keys())
+ ):
+ number_flag = -1
+ if number_flag == 1:
+ digit.append(chsh[line[i]])
+
+ elif "十" == line[i] and number_flag == 0:
+ number_flag = 2
+ digit.append("1")
+ bit.append(line[i])
+ elif "十" == line[i] and number_flag == 3:
+ digit.append("1")
+ bit.append(line[i])
+ elif ("零" == line[i]) and (number_flag == 0 or number_flag == 1):
+ digit.append("0")
+ elif ("零" == line[i]) and number_flag == 3:
+ zero_flag = 1
+ elif number_flag == 1 and line[i] in bits:
+ number_flag = 3
+ if line[i] == "千":
+ if i < len(line) - 1:
+ if line[i + 1] in unit:
+ number_flag = -1
+ if number_flag == 3:
+ onebit = line[i]
+ bit.append(onebit)
+ elif number_flag == 3 and line[i] in bits:
+ onebit = bit[-1] + line[i]
+ if onebit in bits:
+ bit[-1] = onebit
+ else:
+ number_flag = -2
+ else:
+ number_flag = -1
+ if len(digit) > 0 and number_flag == -1:
+ number_flag = -2
+ if i == (len(line) - 1) and number_flag >= 0:
+ number_flag = -1
+ if number_flag < 0:
+ newdigit = ""
+ if len(digit) > 0: # and (len(digit) == len(bit))):
+ if len(bit) == 1 and zero_flag == 0 and bit[0] == "百" and len(bit) != len(digit):
+ bit.append("十")
+ if len(digit) == (len(bit) + 1):
+ bit.append("零")
+ if len(digit) == len(bit):
+ for m in range(len(digit))[-1::-1]:
+ if int(bits[bit[m]]) == int(len(newdigit) + 1):
+ newdigit += digit[m]
+ else:
+ nu = int(bits[bit[m]]) - len(newdigit) - 1
+ for n in range(nu):
+ newdigit += "0"
+ newdigit += digit[m]
+ for z in range(len(newdigit))[-1::-1]:
+ newline += newdigit[z]
+ else:
+ newline += "".join(digit)
+ bit = []
+ digit = []
+ zero_flag = 0
+ else:
+ newline += line[i]
+ if number_flag == -2:
+ newline += line[i]
+ number_flag = 0
+ return newline
+
+
+def special(line):
+ newline = ""
+ for e in range(len(line)):
+ if ord(line[e]) == 247:
+ newline += "除以"
+ elif ord(line[e]) == 215:
+ newline += "乘以"
+ elif ord(line[e]) == 61:
+ newline += "等于"
+ elif ord(line[e]) == 43:
+ newline += "加"
+ elif ord(line[e]) == 45:
+ newline += "负"
+ elif ord(line[e]) == 8451:
+ newline += "摄氏度"
+ elif ord(line[e]) == 13217:
+ newline += "平方米"
+ elif ord(line[e]) == 8240 or ord(line[e]) == 65130:
+ newline += "%"
+ elif ord(line[e]) == 46:
+ newline += "点"
+ elif ord(line[e]) == 176:
+ newline += "度"
+ angel = 1
+ elif ord(line[e]) == 8242 and angel == 1:
+ newline += "分"
+ else:
+ newline += line[e]
+ return newline
+
+
+def all_convert(content):
+ content = recoformat(content)
+ content = numbersingle(content)
+ content = ch_number2digit(content)
+ content = special(content)
+ content = scoreformat("", content)
+ return content
+
+
+if __name__ == "__main__":
+ if len(sys.argv[1:]) < 1:
+ sys.stderr.write("Usage:\n .py reco.result\n")
+ sys.stderr.write(" reco.result: id<tab>recoresult\n")
+ sys.exit(1)
+ f = open(sys.argv[1])
+ flag = 0
+ if len(sys.argv[1:]) > 1:
+ flag = int(sys.argv[2])
+ for line in f.readlines():
+ if not line:
+ continue
+ line = line.rstrip()
+ tmp = line.split("\t")
+ if len(tmp) < 2:
+ tmp = line.split(",")
+ if len(tmp) < 2:
+ tmp = line.split(" ", 1)
+ if len(tmp) < 2:
+ name = tmp[0]
+ content = ""
+ print(content)
+ continue
+ name = tmp[0]
+ content = tmp[1]
+ name = re.sub("\.pcm", "", name)
+ name = re.sub("\.wav", "", name)
+ content = recoformat(content)
+ content = numbersingle(content)
+ content = ch_number2digit(content)
+ content = special(content)
+ content = scoreformat(name, content, flag)
+ print(content)
+ f.close()
diff --git a/funasr/models/fun_asr_nano/tools/scp2jsonl.py b/funasr/models/fun_asr_nano/tools/scp2jsonl.py
new file mode 100644
index 000000000..488173ff9
--- /dev/null
+++ b/funasr/models/fun_asr_nano/tools/scp2jsonl.py
@@ -0,0 +1,142 @@
+import hydra
+import json
+import os
+import threading
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from io import BytesIO
+from typing import Dict, Optional, Tuple
+from urllib.request import urlopen
+
+import soundfile as sf
+from modelscope import AutoTokenizer
+from tqdm import tqdm
+from omegaconf import DictConfig, OmegaConf, ListConfig
+
+
+class LineProcessor:
+ def __init__(self, tokenizer):
+ self.tokenizer = tokenizer
+ self.lock = threading.Lock()
+
+ def process_line(self, line_pair: Tuple[str, str]) -> Optional[Dict]:
+ line1, line2 = line_pair
+
+ line1, line2 = line1.strip(), line2.strip()
+ if not line1 or not line2:
+ return None
+
+ parts1, parts2 = line1.split(maxsplit=1), line2.split(maxsplit=1)
+ if len(parts1) != 2 or len(parts2) != 2:
+ return None
+
+ utt1, utt2 = parts1[0], parts2[0]
+ wav_path, text = parts1[1], parts2[1]
+
+ if utt1 != utt2:
+ return {"error": f"UTT mismatch: {utt1} vs {utt2}"}
+
+ try:
+ if wav_path.startswith("http"):
+ response = urlopen(wav_path)
+ if response.status != 200:
+ return {"error": f"WAV not found: {wav_path}"}
+ audio_file = BytesIO(response.read())
+ duration = sf.info(audio_file).duration
+ else:
+ if not os.path.exists(wav_path):
+ return {"error": f"WAV not found: {wav_path}"}
+ duration = sf.info(wav_path).duration
+
+ data = {
+ "messages": [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {
+ "role": "user",
+ "content": f"语音转写:<|startofspeech|>!{wav_path}<|endofspeech|>",
+ },
+ {"role": "assistant", "content": text},
+ ],
+ "speech_length": int((duration * 1000 - 25) // 10 + 1),
+ "text_length": len(self.tokenizer.tokenize(text)),
+ }
+ return {"success": data, "utt": utt1}
+
+ except Exception as e:
+ return {"error": f"Error processing {wav_path}: {str(e)}"}
+
+
[email protected](config_name=None, version_base=None)
+def main_hydra(cfg: DictConfig):
+ def to_plain_list(cfg_item):
+ if isinstance(cfg_item, ListConfig):
+ return OmegaConf.to_container(cfg_item, resolve=True)
+ elif isinstance(cfg_item, DictConfig):
+ return {k: to_plain_list(v) for k, v in cfg_item.items()}
+ else:
+ return cfg_item
+
+ kwargs = to_plain_list(cfg)
+ scp_file = kwargs["scp_file"]
+ transcript_file = kwargs["transcript_file"]
+ max_workers = kwargs.get("max_workers", os.cpu_count())
+ jsonl_file = kwargs["jsonl_file"]
+
+ with open(scp_file, "r") as f1, open(transcript_file, "r") as f2:
+ scp_lines = f1.readlines()
+ transcript_lines = f2.readlines()
+
+ if len(scp_lines) != len(transcript_lines):
+ print(
+ f"Warning: Line count mismatch - scp: {len(scp_lines)}, transcript: {len(transcript_lines)}"
+ )
+
+ tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
+ processor = LineProcessor(tokenizer)
+
+ data_pairs = list(zip(scp_lines, transcript_lines))
+
+ processed_count = 0
+ failed_count = 0
+ error_messages = []
+
+ with tqdm(total=len(data_pairs), desc="Processing") as pbar:
+ with ThreadPoolExecutor(max_workers=max_workers) as executor:
+ with open(jsonl_file, "w") as f_out:
+ futures = {
+ executor.submit(processor.process_line, pair): i
+ for i, pair in enumerate(data_pairs)
+ }
+
+ for future in as_completed(futures):
+ result = future.result()
+
+ if result and "success" in result:
+ with processor.lock:
+ json.dump(result["success"], f_out, ensure_ascii=False)
+ f_out.write("\n")
+ processed_count += 1
+ elif result and "error" in result:
+ failed_count += 1
+ error_messages.append(result["error"])
+
+ pbar.update(1)
+ pbar.set_postfix({"processed": processed_count, "failed": failed_count})
+
+ print(f"\nProcessing completed:")
+ print(f" Total lines: {len(data_pairs)}")
+ print(f" Successfully processed: {processed_count}")
+ print(f" Failed: {failed_count}")
+
+ if error_messages and len(error_messages) <= 10:
+ print(f"\nSample errors:")
+ for error in error_messages[:10]:
+ print(f" - {error}")
+ elif error_messages:
+ print(f"\nFirst 10 errors:")
+ for error in error_messages[:10]:
+ print(f" - {error}")
+ print(f" ... and {len(error_messages) - 10} more errors")
+
+
+if __name__ == "__main__":
+ main_hydra()
diff --git a/funasr/models/fun_asr_nano/tools/utils.py b/funasr/models/fun_asr_nano/tools/utils.py
new file mode 100644
index 000000000..7125bca5c
--- /dev/null
+++ b/funasr/models/fun_asr_nano/tools/utils.py
@@ -0,0 +1,57 @@
+from itertools import groupby
+
+import soundfile as sf
+import torch
+import torchaudio
+import torchaudio.functional as F
+
+
+def load_audio(wav_path, rate: int = None, offset: float = 0, duration: float = None):
+ with sf.SoundFile(wav_path) as f:
+ start_frame = int(offset * f.samplerate)
+ if duration is None:
+ frames_to_read = f.frames - start_frame
+ else:
+ frames_to_read = int(duration * f.samplerate)
+ f.seek(start_frame)
+ audio_data = f.read(frames_to_read, dtype="float32")
+ audio_tensor = torch.from_numpy(audio_data)
+ if rate is not None and f.samplerate != rate:
+ if audio_tensor.ndim == 1:
+ audio_tensor = audio_tensor.unsqueeze(0)
+ else:
+ audio_tensor = audio_tensor.T
+ resampler = torchaudio.transforms.Resample(orig_freq=f.samplerate, new_freq=rate)
+ audio_tensor = resampler(audio_tensor)
+ if audio_tensor.shape[0] == 1:
+ audio_tensor = audio_tensor.squeeze(0)
+ return audio_tensor, rate if rate is not None else f.samplerate
+
+
+def forced_align(log_probs: torch.Tensor, targets: torch.Tensor, blank: int = 0):
+ items = []
+ try:
+ # The current version only supports batch_size==1.
+ log_probs, targets = log_probs.unsqueeze(0).cpu(), targets.unsqueeze(0).cpu()
+ assert log_probs.shape[1] >= targets.shape[1]
+ alignments, scores = F.forced_align(log_probs, targets, blank=blank)
+ alignments, scores = alignments[0], torch.exp(scores[0]).tolist()
+ # use enumerate to keep track of the original indices, then group by token value
+ for token, group in groupby(enumerate(alignments), key=lambda item: item[1]):
+ if token == blank:
+ continue
+ group = list(group)
+ start = group[0][0]
+ end = start + len(group)
+ score = max(scores[start:end])
+ items.append(
+ {
+ "token": token.item(),
+ "start_time": start,
+ "end_time": end,
+ "score": round(score, 3),
+ }
+ )
+ except:
+ pass
+ return items
diff --git a/funasr/models/fun_asr_nano/tools/whisper_mix_normalize.py b/funasr/models/fun_asr_nano/tools/whisper_mix_normalize.py
new file mode 100644
index 000000000..632b52906
--- /dev/null
+++ b/funasr/models/fun_asr_nano/tools/whisper_mix_normalize.py
@@ -0,0 +1,135 @@
+# -*- coding: utf-8 -*-
+#!/usr/bin/python
+# Author: Mengze Chen
+
+import re
+import sys
+
+import cn_tn as cn_tn
+import format5res as cn_itn
+import pyopenjtalk
+import zhconv
+from whisper_normalizer.basic import BasicTextNormalizer
+from whisper_normalizer.english import EnglishTextNormalizer
+
+basic_normalizer = BasicTextNormalizer()
+english_normalizer = EnglishTextNormalizer()
+
+
+def is_only_chinese_and_english(s):
+ # 定义正则表达式模式,匹配中文字符范围和英文字母(包括大小写)
+ pattern = r"^[\u4e00-\u9fa5A-Za-z0-9,\.!\?:;,。!?:;、%\'\s\-\~]+$"
+ # 使用正则表达式进行匹配
+ return re.match(pattern, s) is not None
+
+
+def is_only_english(s):
+ # 定义正则表达式模式,匹配中文字符范围和英文字母(包括大小写)
+ pattern = r"^[A-Za-z0-9,\.!\?:;,。!?:;、%\'\s\-\~]+$"
+ # 使用正则表达式进行匹配
+ return re.match(pattern, s) is not None
+
+
+def is_number(s):
+ # 定义正则表达式模式,匹配中文字符范围和英文字母(包括大小写)
+ pattern = r"^[0-9,\.!\?:;,。!?:;、%\'\s]+$"
+ # 使用正则表达式进行匹配
+ return re.match(pattern, s) is not None
+
+
+def safe_ja_g2p(text, kana=True, max_length=100):
+ if len(text) > max_length:
+ # 如果文本过长,分段处理
+ parts = []
+ for i in range(0, len(text), max_length):
+ part = text[i : i + max_length]
+ try:
+ converted = pyopenjtalk.g2p(part, kana=kana)
+ parts.append(converted)
+ except:
+ parts.append(part) # 如果转换失败,使用原文本
+ return " ".join(parts)
+ else:
+ try:
+ return pyopenjtalk.g2p(text, kana=kana)
+ except:
+ return text # 如果转换失败,返回原文本
+
+
+def normalize_text(srcfn, dstfn, kana=False):
+ with open(srcfn, "r") as f_read, open(dstfn, "w") as f_write:
+ all_lines = f_read.readlines()
+ for line in all_lines:
+ line = line.strip()
+ line_arr = line.split(maxsplit=1)
+ if len(line_arr) < 1:
+ continue
+ if len(line_arr) == 1:
+ line_arr.append("")
+ key = line_arr[0]
+ line_arr[1] = re.sub(r"=", " ", line_arr[1])
+ line_arr[1] = re.sub(r"\(", " ", line_arr[1])
+ line_arr[1] = re.sub(r"\)", " ", line_arr[1])
+ # From Chongjia Ni
+ if kana:
+ line_arr[1] = safe_ja_g2p(line_arr[1], kana=True, max_length=100)
+
+ line_arr = f"{key}\t{line_arr[1]}".split()
+ conts = []
+ language_bak = ""
+ part = []
+ for i in range(1, len(line_arr)):
+ out_part = ""
+ chn_eng_bool = is_only_chinese_and_english(line_arr[i])
+ eng_bool = is_only_english(line_arr[i])
+ num_bool = is_number(line_arr[i])
+ if eng_bool and not num_bool:
+ language = "en"
+ elif chn_eng_bool:
+ language = "chn_en"
+ else:
+ language = "not_chn_en"
+ if language == language_bak or language_bak == "":
+ part.append(line_arr[i])
+ language_bak = language
+ else:
+ if language_bak == "en":
+ out_part1 = english_normalizer(" ".join(part))
+ out_part = cn_itn.scoreformat("", out_part1)
+ elif language_bak == "chn_en":
+ out_part1 = english_normalizer(" ".join(part))
+ out_part2 = cn_tn.normalize_nsw(out_part1)
+ out_part3 = cn_itn.all_convert(out_part2)
+ out_part = zhconv.convert(out_part3, "zh-cn")
+ else:
+ out_part1 = basic_normalizer(" ".join(part))
+ out_part2 = cn_tn.normalize_nsw(out_part1)
+ out_part3 = cn_itn.all_convert(out_part2)
+ out_part = zhconv.convert(out_part3, "zh-cn")
+ conts.append(out_part)
+ language_bak = language
+ part = []
+ part.append(line_arr[i])
+ if i == len(line_arr) - 1:
+ if language == "en":
+ out_part1 = english_normalizer(" ".join(part))
+ out_part = cn_itn.scoreformat("", out_part1)
+ elif language == "chn_en":
+ out_part1 = english_normalizer(" ".join(part))
+ out_part2 = cn_tn.normalize_nsw(out_part1)
+ out_part3 = cn_itn.all_convert(out_part2)
+ out_part = zhconv.convert(out_part3, "zh-cn")
+ else:
+ out_part1 = basic_normalizer(" ".join(part))
+ out_part2 = cn_tn.normalize_nsw(out_part1)
+ out_part3 = cn_itn.all_convert(out_part2)
+ out_part = zhconv.convert(out_part3, "zh-cn")
+ conts.append(out_part)
+
+ f_write.write("{0}\t{1}\n".format(key, " ".join(conts).strip()))
+
+
+if __name__ == "__main__":
+ srcfn = sys.argv[1]
+ dstfn = sys.argv[2]
+ normalize_text(srcfn, dstfn, True if len(sys.argv) > 3 else False)