FunASR-powered video transcription, subtitle generation, and LLM-assisted clipping tool with a local Gradio UI.
Python ai-toolsai-video-editingasrauto-subtitleschinesecontent-creationfunasrfunclipgradiollmparaformerspeech-recognitionspeech-to-textsubtitles-generatortranscriptionvideo-editingvideo-processingvideo-subtitlesvideo-transcriptionwhisper-alternative
Fix ASR model routing across language modes (#176)
zhifu gao80ea6ea18b89
diff --git a/README.md b/README.mdindex 302beca..a6340bf 100644--- a/README.md+++ b/README.md@@ -34,7 +34,7 @@<a name="What's New"></a>## What's New🚀-- 2026/05/20 FunClip now supports Fun-ASR-Nano and SenseVoice models. Fun-ASR-Nano provides higher accuracy for 31 languages; SenseVoice adds emotion recognition and audio event detection. Run `python funclip/launch.py -m fun-asr-nano` or `-m sensevoice` to try.+- 2026/05/20 FunClip now supports Fun-ASR-Nano and SenseVoice models. The `fun-asr-nano` option loads the flagship Fun-ASR-Nano-2512 checkpoint for Mandarin, English, Japanese, 7 Chinese dialect groups, and 26 regional accents; it does not load the separate 31-language Fun-ASR-MLT-Nano-2512 checkpoint. SenseVoice adds emotion recognition and audio event detection. Run `python funclip/launch.py -m fun-asr-nano` or `-m sensevoice` to try. For precise text-based clipping, use Paraformer because the released Nano checkpoint does not provide reliable character-level timestamps.- 2024/06/12 FunClip supports recognize and clip English audio files now. Run `python funclip/launch.py -l en` to try.- 🔥2024/05/13 FunClip v2.0.0 now supports smart clipping with large language models, integrating models from the qwen series, GPT series, etc., providing default prompts. You can also explore and share tips for setting prompts, the usage is as follows:1. After the recognition, select the name of the large model and configure your own apikey;@@ -108,7 +108,8 @@ wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/STHeitiMedYou can establish your own FunClip service which is same as [Modelscope Space](https://modelscope.cn/studios/iic/funasr_app_clipvideo/summary) as follow:```shellpython funclip/launch.py-# '-m fun-asr-nano' for Fun-ASR-Nano model (higher accuracy, 31 languages)+# '-m fun-asr-nano' for the flagship Fun-ASR-Nano model (Mandarin, English,+# Japanese, 7 Chinese dialect groups, and 26 regional accents)# '-m sensevoice' for SenseVoice model (multilingual ASR + emotion + audio event detection)# '-l en' for English audio recognize# '-p xxx' for setting port number@@ -120,7 +121,7 @@ python funclip/launch.py| Scenario | Command || --- | --- || Default Chinese video clipping with Paraformer | `python funclip/launch.py` |-| 31-language ASR with Fun-ASR-Nano | `python funclip/launch.py -m fun-asr-nano` |+| High-accuracy transcription with the flagship Fun-ASR-Nano checkpoint (use Paraformer for precise text-based clipping) | `python funclip/launch.py -m fun-asr-nano` || Multilingual ASR with emotion and audio event tags | `python funclip/launch.py -m sensevoice` || English video clipping with the Paraformer English model | `python funclip/launch.py -l en` |@@ -188,7 +189,7 @@ FunClip is part of the **FunAudioLLM** family:| Project | Description | Stars ||---------|-------------|-------|| [FunASR](https://github.com/modelscope/FunASR) | Industrial speech recognition toolkit — VAD, ASR, punctuation, diarization | [](https://github.com/modelscope/FunASR) |-| [Fun-ASR-Nano](https://github.com/FunAudioLLM/Fun-ASR) | End-to-end LLM-based ASR — 31 languages, streaming, hotwords | [](https://github.com/FunAudioLLM/Fun-ASR) |+| [Fun-ASR-Nano](https://github.com/FunAudioLLM/Fun-ASR) | End-to-end LLM-based ASR — flagship and separate 31-language MLT checkpoints, streaming, hotwords | [](https://github.com/FunAudioLLM/Fun-ASR) || [SenseVoice](https://github.com/FunAudioLLM/SenseVoice) | Multilingual speech understanding — ASR + emotion + audio events | [](https://github.com/FunAudioLLM/SenseVoice) || [CosyVoice](https://github.com/FunAudioLLM/CosyVoice) | Natural speech generation — multi-language, zero-shot cloning | [](https://github.com/FunAudioLLM/CosyVoice) |diff --git a/README_zh.md b/README_zh.mdindex ec05593..24b6146 100644--- a/README_zh.md+++ b/README_zh.md@@ -36,7 +36,7 @@<a name="近期更新"></a>## 近期更新🚀-- 2026/05/20 FunClip 现在支持 Fun-ASR-Nano 与 SenseVoice 模型。Fun-ASR-Nano 支持 31 种语言的高精度识别;SenseVoice 支持多语种识别,并额外输出情绪识别与音频事件检测标签。可通过 `python funclip/launch.py -m fun-asr-nano` 或 `python funclip/launch.py -m sensevoice` 启动体验。+- 2026/05/20 FunClip 现在支持 Fun-ASR-Nano 与 SenseVoice 模型。`fun-asr-nano` 选项加载旗舰版 Fun-ASR-Nano-2512,支持普通话、英语、日语、7 类中文方言和 26 种地域口音;该选项不会加载独立的 31 语种 Fun-ASR-MLT-Nano-2512。SenseVoice 支持多语种识别,并额外输出情绪识别与音频事件检测标签。可通过 `python funclip/launch.py -m fun-asr-nano` 或 `python funclip/launch.py -m sensevoice` 启动体验。需要精确按文本裁剪时请使用 Paraformer,因为当前发布的 Nano checkpoint 不提供可靠的字符级时间戳。- 2024/06/12 FunClip现在支持识别与裁剪英文视频,通过`python funclip/launch.py -l en`来启动英文版本服务。- 🔥2024/05/13 FunClip v2.0.0加入大语言模型智能裁剪功能,集成qwen系列,gpt系列等模型,提供默认prompt,您也可以探索并分享prompt的设置技巧,使用方法如下:1. 在进行识别之后,选择大模型名称,配置你自己的apikey;@@ -108,7 +108,8 @@ wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/STHeitiMed```shellpython funclip/launch.py-# '-m fun-asr-nano' 使用 Fun-ASR-Nano 模型(更高精度,支持 31 种语言)+# '-m fun-asr-nano' 使用旗舰版 Fun-ASR-Nano(普通话、英语、日语、+# 7 类中文方言和 26 种地域口音)# '-m sensevoice' 使用 SenseVoice 模型(多语种 ASR + 情绪识别 + 音频事件检测)# '-l en' for English audio recognize# '-p xxx' for setting port number@@ -119,7 +120,7 @@ python funclip/launch.py| 场景 | 启动命令 || --- | --- || 默认中文视频裁剪,使用 Paraformer | `python funclip/launch.py` |-| 使用 Fun-ASR-Nano 进行 31 种语言识别 | `python funclip/launch.py -m fun-asr-nano` |+| 使用旗舰版 Fun-ASR-Nano 进行高精度转写(精确按文本裁剪请使用 Paraformer) | `python funclip/launch.py -m fun-asr-nano` || 使用 SenseVoice 进行多语种识别、情绪识别和音频事件检测 | `python funclip/launch.py -m sensevoice` || 使用 Paraformer 英文模型裁剪英文视频 | `python funclip/launch.py -l en` |diff --git a/funclip/launch.py b/funclip/launch.pyindex 76e77d7..e712318 100644--- a/funclip/launch.py+++ b/funclip/launch.py@@ -20,43 +20,48 @@ from utils.trans_utils import extract_timestampsfrom introduction import top_md_1, top_md_3, top_md_4+def create_asr_model(model_name, lang, auto_model_cls=AutoModel):+ if model_name == "fun-asr-nano":+ return auto_model_cls(+ model="FunAudioLLM/Fun-ASR-Nano-2512",+ trust_remote_code=True,+ remote_code="./model.py",+ vad_model="fsmn-vad",+ vad_kwargs={"max_single_segment_time": 30000},+ spk_model="cam++",+ hub="hf",+ )+ if model_name == "sensevoice":+ return auto_model_cls(+ model="iic/SenseVoiceSmall",+ vad_model="fsmn-vad",+ vad_kwargs={"max_single_segment_time": 30000},+ spk_model="cam++",+ )++ paraformer_model = (+ "iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch"+ if lang == "zh"+ else "iic/speech_paraformer_asr-en-16k-vocab4199-pytorch"+ )+ return auto_model_cls(+ model=paraformer_model,+ vad_model="damo/speech_fsmn_vad_zh-cn-16k-common-pytorch",+ punc_model="damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch",+ spk_model="damo/speech_campplus_sv_zh-cn_16k-common",+ )++if __name__ == "__main__":parser = argparse.ArgumentParser(description='argparse testing')- parser.add_argument('--lang', '-l', type=str, default = "zh", help="language")- parser.add_argument('--model', '-m', type=str, default="paraformer", choices=["paraformer", "fun-asr-nano", "sensevoice"], help="ASR model: paraformer, fun-asr-nano, or sensevoice")+ parser.add_argument('--lang', '-l', type=str, default = "zh", help="language mode; selects the Paraformer checkpoint but does not override --model")+ parser.add_argument('--model', '-m', type=str, default="paraformer", choices=["paraformer", "fun-asr-nano", "sensevoice"], help="ASR model: paraformer, fun-asr-nano, or sensevoice (takes precedence over --lang)")parser.add_argument('--share', '-s', action='store_true', help="if to establish gradio share link")parser.add_argument('--port', '-p', type=int, default=7860, help='port number')parser.add_argument('--listen', action='store_true', help="if to listen to all hosts")args = parser.parse_args()- if args.lang == 'zh':- if hasattr(args, 'model') and args.model == 'fun-asr-nano':- funasr_model = AutoModel(model="FunAudioLLM/Fun-ASR-Nano-2512",- trust_remote_code=True,- remote_code="./model.py",- vad_model="fsmn-vad",- vad_kwargs={"max_single_segment_time": 30000},- spk_model="cam++",- hub="hf",- )- elif hasattr(args, 'model') and args.model == 'sensevoice':- funasr_model = AutoModel(model="iic/SenseVoiceSmall",- vad_model="fsmn-vad",- vad_kwargs={"max_single_segment_time": 30000},- spk_model="cam++",- )- else:- funasr_model = AutoModel(model="iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch",- vad_model="damo/speech_fsmn_vad_zh-cn-16k-common-pytorch",- punc_model="damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch",- spk_model="damo/speech_campplus_sv_zh-cn_16k-common",- )- else:- funasr_model = AutoModel(model="iic/speech_paraformer_asr-en-16k-vocab4199-pytorch",- vad_model="damo/speech_fsmn_vad_zh-cn-16k-common-pytorch",- punc_model="damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch",- spk_model="damo/speech_campplus_sv_zh-cn_16k-common",- )+ funasr_model = create_asr_model(args.model, args.lang)audio_clipper = VideoClipper(funasr_model)audio_clipper.lang = args.langdiff --git a/requirements.txt b/requirements.txtindex 2482ae4..8fc2625 100644--- a/requirements.txt+++ b/requirements.txt@@ -2,6 +2,8 @@ librosasoundfilescikit-learn>=1.3.2funasr>=1.1.2+transformers>=4.32.0,<5.0+huggingface_hub>=0.19.3,<1.0moviepy==1.0.3numpy==1.26.4gradio>=4.31.3,<5.0diff --git a/tests/test_model_selection.py b/tests/test_model_selection.pynew file mode 100644index 0000000..86a494b--- /dev/null+++ b/tests/test_model_selection.py@@ -0,0 +1,78 @@+import sys+import unittest+from pathlib import Path+++ROOT = Path(__file__).resolve().parents[1]+sys.path.insert(0, str(ROOT / "funclip"))++from launch import create_asr_model+++class RecordingAutoModel:+ def __init__(self, **kwargs):+ self.kwargs = kwargs+++class TestModelSelection(unittest.TestCase):+ def test_fun_asr_nano_is_selected_independently_of_language(self):+ model = create_asr_model(+ "fun-asr-nano", "en", auto_model_cls=RecordingAutoModel+ )++ self.assertEqual(+ model.kwargs,+ {+ "model": "FunAudioLLM/Fun-ASR-Nano-2512",+ "trust_remote_code": True,+ "remote_code": "./model.py",+ "vad_model": "fsmn-vad",+ "vad_kwargs": {"max_single_segment_time": 30000},+ "spk_model": "cam++",+ "hub": "hf",+ },+ )++ def test_sensevoice_is_selected_independently_of_language(self):+ model = create_asr_model(+ "sensevoice", "en", auto_model_cls=RecordingAutoModel+ )++ self.assertEqual(+ model.kwargs,+ {+ "model": "iic/SenseVoiceSmall",+ "vad_model": "fsmn-vad",+ "vad_kwargs": {"max_single_segment_time": 30000},+ "spk_model": "cam++",+ },+ )++ def test_paraformer_keeps_language_specific_model_mapping(self):+ cases = {+ "zh": "iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch",+ "en": "iic/speech_paraformer_asr-en-16k-vocab4199-pytorch",+ }++ for language, expected_model in cases.items():+ with self.subTest(language=language):+ model = create_asr_model(+ "paraformer", language, auto_model_cls=RecordingAutoModel+ )+ self.assertEqual(model.kwargs["model"], expected_model)+ self.assertEqual(+ model.kwargs["vad_model"],+ "damo/speech_fsmn_vad_zh-cn-16k-common-pytorch",+ )+ self.assertEqual(+ model.kwargs["punc_model"],+ "damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch",+ )+ self.assertEqual(+ model.kwargs["spk_model"],+ "damo/speech_campplus_sv_zh-cn_16k-common",+ )+++if __name__ == "__main__":+ unittest.main()