FunASR-powered video transcription, subtitle generation, and LLM-assisted clipping tool with a local Gradio UI.
Python ai-toolsai-video-editingasrauto-subtitleschinesecontent-creationfunasrfunclipgradiollmparaformerspeech-recognitionspeech-to-textsubtitles-generatortranscriptionvideo-editingvideo-processingvideo-subtitlesvideo-transcriptionwhisper-alternative
update readme
维石7acec36804a8
diff --git a/README.md b/README.mdindex 446f751..363db9c 100644--- a/README.md+++ b/README.md@@ -1,80 +1,97 @@-# FunClip🎥+# FunClip🎥-FunClip是一款完全开源、本地部署的自动化视频剪辑工具,通过调用阿里巴巴通义实验室开源的[FunASR](https://github.com/alibaba-damo-academy/FunASR) Paraformer系列模型进行视频的语音识别,随后用户可以自由选择识别结果中的文本片段或说话人,点击裁剪按钮即可获取对应片段的视频([快速体验](https://modelscope.cn/studios/damo/funasr_app_funclip/summary))。+([简体中文](./README_zh.md)|English)-在上述基本功能的基础上,FunClip有以下特色:-- FunClip集成了阿里巴巴开源的工业级模型[Paraformer-Large](https://modelscope.cn/models/iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary),是当前识别效果最优的开源中文ASR模型之一,Modelscope下载量1300w+次,并且能够一体化的准确预测时间戳。-- FunClip集成了[SeACo-Paraformer](https://modelscope.cn/models/iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary)的热词定制化功能,在ASR过程中可以指定一些实体词、人名等作为热词,提升识别效果。-- FunClip集成了[CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary)说话人识别模型,用户可以将自动识别出的说话人ID作为裁剪目标,将某一说话人的段落裁剪出来。-- 通过Gradio交互实现上述功能,安装简单使用方便,并且可以在服务端搭建服务通过浏览器使用。-- FunClip支持多段自由剪辑,并且会自动返回全视频SRT字幕、目标段落SRT字幕,使用简单方便。+FunClip is a fully open-source, locally deployed automated video editing tool. It leverages Alibaba DAMO Academy's open-source [FunASR](https://github.com/alibaba-damo-academy/FunASR) Paraformer series models to perform speech recognition on videos. Then, users can freely choose text segments or speakers from the recognition results and click the trim button to obtain the video corresponding to the selected segments ([Quick Experience](https://modelscope.cn/studios/iic/funasr_app_clipvideo/summary)).-欢迎体验使用,欢迎提出关于字幕生成或语音识别的需求与宝贵建议~+On top of the basic features mentioned above, FunClip has the following highlights:-## 近期更新🚀+- FunClip integrates Alibaba's open-source industrial-grade model [Paraformer-Large](https://modelscope.cn/models/iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), which is one of the best-performing open-source Chinese ASR models available, with over 13 million downloads on Modelscope. It can also accurately predict timestamps in an integrated manner.+- FunClip incorporates the hotword customization feature of [SeACo-Paraformer](https://modelscope.cn/models/iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), allowing users to specify certain entity words, names, etc., as hotwords during the ASR process to enhance recognition results.+- FunClip integrates the [CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) speaker recognition model, enabling users to use the auto-recognized speaker ID as the target for trimming, to clip segments from a specific speaker.+- The functionalities are realized through Gradio interaction, offering simple installation and ease of use. It can also be deployed on a server and accessed via a browser.+- FunClip supports multi-segment free clipping and automatically returns full video SRT subtitles and target segment SRT subtitles, offering a simple and convenient user experience.-- 2024/03/06 命令行调用方式更新与问题修复,相关功能可以正常使用。-- 2024/02/28 FunClip升级到FunASR1.0模型调用方式,通过FunASR开源的SeACo-Paraformer模型在视频剪辑中进一步支持热词定制化功能。-- 2024/02/28 原FunASR-APP/ClipVideo更名为FunClip。+You're welcome to try it out, and we look forward to any requests and valuable suggestions you may have about subtitle generation or speech recognition~-## 使用FunClip进行视频剪辑✂️+## What's New🚀++- 2024/03/06 Fix bugs in using FunClip with command line.+- 2024/02/28 Update call function of funasr into funasr1.0, use SeACo_Paraformer thus ASR now supports hotword. Also support destination transcription like 'abcd[-100,150]#efgh[200,200]' to adjust offset time for every sub-sentence.+- 2023/10/17 Fix bugs in multiple periods chosen, used to return video with wrong length.+- 2023/10/10 FunClipper now supports recognizing with speaker diarization ability, choose 'yes' button in 'Recognize Speakers' and you will get recognition results with speaker id for each sentence. And then you can clip out the periods of one or some speakers (e.g. 'spk0' or 'spk0#spk3') using FunClipper.++## Use FunClipper to Clip Your Video✂️++FunClipper</strong> enables users to clip ```.mp4``` video files or ```.wav``` audio files with chosen text segments out of the recognition results generated by [Paraformer-long model](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary).++Under the help of FunClipper you can get the video clips easily with the following steps (in Gradio service):+- Step1: Upload your video file (or try the example videos below)+- Step2: Copy the text segments you need to 'Text to Clip'+- Step3: Adjust subtitle settings (if needed)+- Step4: Click 'Clip' or 'Clip and Generate Subtitles'++### Install++#### Python env install-### 安装```shell-# 安装FunASR(必须)+# install funasr andpip install -U funasr-# 可以通过源码安装funasr+# or install funasr throuth source codegit clone https://github.com/alibaba-damo-academy/FunASR.gitcd FunASR; pip install -e ./-# 安装FunClip的Python依赖(必须)-pip install -r ./requirments.txt-# 根据你的环境(CUDA,Python版本等)安装torch和torchaudio-pip install torch torchaudio+# install python dependencies+pip install -r funclip/requirments.txt+# install torch and torchaudio based on your CUDA and python version+pip install torch==xxx torchaudio==xxx```-### 安装imagemagick(可选)-如果你希望使用自动生成字幕的视频裁剪功能,需要安装imagemagick-- Ubuntu+#### imagemagick install (Optional)++If you want to clip video file with embedded subtitles++1. ffmpeg and imagemagick is required++- On Ubuntu```shellapt-get -y update && apt-get -y install ffmpeg imagemagicksed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xml```-- MacOS+- On MacOS```shellbrew install imagemagicksed -i 's/none/read,write/g' /usr/local/Cellar/imagemagick/7.1.1-8_1/etc/ImageMagick-7/policy.xml```+2. Download font file to funclip/font-并且下载你需要的字体文件,这里我们提供一个默认的黑体字体文件```shell-wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/funclip/STHeitiMedium.ttc -O font/STHeitiMedium.ttc+wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/STHeitiMedium.ttc -O font/STHeitiMedium.ttc```-### 使用FunClip--#### A.在本地启动Gradio服务+### Use FunClip+#### A. Use FunClip as local Gradio Service+You can establish your own FunClip service which is same as [Modelscope Space](https://modelscope.cn/studios/iic/funasr_app_clipvideo/summary) as follow:```shell-python funclip/launch.py+python funclip/gradio_service.py```-随后在浏览器中访问```localhost:7860```即可看到如下图所示的界面,按如下步骤即可进行视频剪辑-1. 上传你的视频(或使用下方的视频用例)-2. (可选)设置热词,勾选是否使用说话人识别功能-3. 点击识别按钮获取识别结果-4. 将识别结果中的选段复制到对应位置,或者将说话人ID输入到对应为止-5. (可选)配置剪辑参数,偏移量与字幕设置等-6. 点击“裁剪”或“裁剪并添加字幕”按钮+then visit ```localhost:7860``` you will get a Gradio service like below and you can use FunClip following the steps:+<img src="funclip/docs/images/show2.0.png"/>-<img src="docs/images/demo.png"/>+#### B. Experience FunClip in Modelscope+You can try FunClip in modelscope space: [link](https://modelscope.cn/studios/iic/funasr_app_clipvideo/summary).-#### B.通过命令行调用使用FunClip的相关功能(更新中)+#### C. Use FunClip in command line++FunClip supports you to recognize and clip with commands:```shell-# 步骤一:识别+# step1: Recognizepython funclip/videoclipper.py --stage 1 \--file examples/2022云栖大会_片段.mp4 \--output_dir ./output-# ./output中生成了识别结果与srt字幕等-# 步骤二:裁剪+# now you can find recognition results and entire SRT file in ./output/+# step2: Clippython funclip/videoclipper.py --stage 2 \--file examples/2022云栖大会_片段.mp4 \--output_dir ./output \@@ -84,24 +101,18 @@ python funclip/videoclipper.py --stage 2 \--output_file './output/res.mp4'```-#### C.通过Modelscope创空间体验FunClip-[funclip创空间](https://modelscope.cn/studios/damo/funasr_app_funclip/summary)--### 社区交流+## Community Communication+If you encounter problems in use, you can directly raise Issues on the github page.-FunClip开源项目由FunASR社区维护,欢迎加入社区,交流与讨论,以及合作开发等。+You can also scan the following DingTalk group or WeChat group QR code to join the community group for communication and discussion.-| 钉钉群 | 微信群 |+| DingTalk group | WeChat group ||:-------------------------------------------------------------------:|:-----------------------------------------------------:|| <div align="left"><img src="docs/images/dingding.png" width="250"/> | <img src="docs/images/wechat.png" width="215"/></div> |+### Get Speech Models in FunASR-### 通过FunASR了解语音识别相关技术--[FunASR](https://github.com/alibaba-damo-academy/FunASR)是阿里巴巴通义实验室开源的端到端语音识别工具包,目前已经成为主流ASR工具包之一。其主要包括Python pipeline,SDK部署与海量开源工业ASR模型等。--📚FunASR论文: <a href="https://arxiv.org/abs/2305.11013"><img src="https://img.shields.io/badge/Arxiv-2305.11013-orange"></a>--📚SeACo-Paraformer论文:<a href="https://arxiv.org/abs/2308.03266"><img src="https://img.shields.io/badge/Arxiv-2308.03266-orange"></a>+[FunASR](https://github.com/alibaba-damo-academy/FunASR) hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model released on ModelScope, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun!-⭐支持FunASR: <a href='https://github.com/alibaba-damo-academy/FunASR.stargazers'><img src='https://img.shields.io/github/stars/alibaba-damo-academy/FunASR.svg?style=social'></a>+📚FunASR Paper: <a href="https://arxiv.org/abs/2305.11013"><img src="https://img.shields.io/badge/Arxiv-2305.11013-orange"></a>+🌟Support FunASR: <a href='https://github.com/alibaba-damo-academy/FunASR/stargazers'><img src='https://img.shields.io/github/stars/alibaba-damo-academy/FunASR.svg?style=social'></a>diff --git a/README_en.md b/README_en.mddeleted file mode 100644index 9b74392..0000000--- a/README_en.md+++ /dev/null@@ -1,90 +0,0 @@-# FunClipper--## What's New--- 2024/02/28 Update call function of funasr into funasr1.0, use SeACo_Paraformer thus ASR now supports hotword. Also support destination transcription like 'abcd[-100,150]#efgh[200,200]' to adjust offset time for every sub-sentence (TODO: use FunClipper with command line is not updated currently).-- 2023/10/17 Bug fix for multiple periods chosen, used to return video with wrong length.-- 2023/10/10 FunClipper now supports recognizing with speaker diarization ability, choose 'yes' button in 'Recognize Speakers' and you will get recognition results with speaker id for each sentence. And then you can clip out the periods of one or some speakers (e.g. 'spk0' or 'spk0#spk3') using FunClipper.--## Use FunClipper to Clip Your Video--FunClipper</strong> enables users to clip ```.mp4``` video files or ```.wav``` audio files with chosen text segments out of the recognition results generated by [Paraformer-long model](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary).--Under the help of FunClipper you can get the video clips easily with the following steps (in Gradio service):-- Step1: Upload your video file (or try the example videos below)-- Step2: Copy the text segments you need to 'Text to Clip'-- Step3: Adjust subtitle settings (if needed)-- Step4: Click 'Clip' or 'Clip and Generate Subtitles'--### Usage-```shell-# install funasr (necessary)-pip install -U funasr-# install python dependency-pip install -r FunClipper/requirments.txt-```-(Optional) If you want to clip video file with embedded subtitles--1. ffmpeg and imagemagick is required--- On Ubuntu-```shell-apt-get -y update && apt-get -y install ffmpeg imagemagick-sed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xml-```-- On MacOS-```shell-brew install imagemagick-sed -i 's/none/read,write/g' /usr/local/Cellar/imagemagick/7.1.1-8_1/etc/ImageMagick-7/policy.xml-```-2. Download font file to FunClipper/font--```shell-wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/FunClipper/STHeitiMedium.ttc -O FunClipper/font/STHeitiMedium.ttc-```--#### Experience FunClipper in Modelscope-You can try FunClipper in modelscope space: [link](https://modelscope.cn/studios/damo/funasr_app_FunClipper/summary).--#### Use FunClipper as Gradio Service-You can establish your own FunClipper service which is same as [Modelscope Space](https://modelscope.cn/studios/damo/funasr_app_FunClipper/summary) as follow:-```shell-python FunClipper/gradio_service.py-```-then visit ```localhost:7860``` you will get a Gradio service like below and you can use FunClipper following the steps:-<img src="FunClipper/docs/images/show2.0.png"/>--#### Use FunClipper in command line-FunClipper supports you to recognize and clip with commands:-```shell-# working in FunClipper/-# step1: Recognize-python FunClipper/videoclipper.py --stage 1 \- --file examples/2022云栖大会_片段.mp4 \- --output_dir ./output-# now you can find recognition results and entire SRT file in ./output/-# step2: Clip-python FunClipper/videoclipper.py --stage 2 \- --file examples/2022云栖大会_片段.mp4 \- --output_dir ./output \- --dest_text '我们把它跟乡村振兴去结合起来,利用我们的设计的能力' \- --start_ost 0 \- --end_ost 100 \- --output_file './output/res.mp4'-```--## Community Communication-If you encounter problems in use, you can directly raise Issues on the github page.--You can also scan the following DingTalk group or WeChat group QR code to join the community group for communication and discussion.--| DingTalk group | WeChat group |-|:-------------------------------------------------------------------:|:-----------------------------------------------------:|-| <div align="left"><img src="docs/images/dingding.png" width="250"/> | <img src="docs/images/wechat.png" width="215"/></div> |--### Get Speech Models in FunASR--[FunASR](https://github.com/alibaba-damo-academy/FunASR) hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model released on ModelScope, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun!--📚FunASR Paper: <a href="https://arxiv.org/abs/2305.11013"><img src="https://img.shields.io/badge/Arxiv-2305.11013-orange"></a>-🌟Support FunASR: <a href='https://github.com/alibaba-damo-academy/FunASR/stargazers'><img src='https://img.shields.io/github/stars/alibaba-damo-academy/FunASR.svg?style=social'></a>diff --git a/README_zh.md b/README_zh.mdnew file mode 100644index 0000000..e4d64a9--- /dev/null+++ b/README_zh.md@@ -0,0 +1,113 @@+# FunClip🎥++(简体中文|[English](./README.md))++FunClip是一款完全开源、本地部署的自动化视频剪辑工具,通过调用阿里巴巴通义实验室开源的[FunASR](https://github.com/alibaba-damo-academy/FunASR) Paraformer系列模型进行视频的语音识别,随后用户可以自由选择识别结果中的文本片段或说话人,点击裁剪按钮即可获取对应片段的视频([快速体验](https://modelscope.cn/studios/iic/funasr_app_clipvideo/summary))。++在上述基本功能的基础上,FunClip有以下特色:+- FunClip集成了阿里巴巴开源的工业级模型[Paraformer-Large](https://modelscope.cn/models/iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary),是当前识别效果最优的开源中文ASR模型之一,Modelscope下载量1300w+次,并且能够一体化的准确预测时间戳。+- FunClip集成了[SeACo-Paraformer](https://modelscope.cn/models/iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary)的热词定制化功能,在ASR过程中可以指定一些实体词、人名等作为热词,提升识别效果。+- FunClip集成了[CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary)说话人识别模型,用户可以将自动识别出的说话人ID作为裁剪目标,将某一说话人的段落裁剪出来。+- 通过Gradio交互实现上述功能,安装简单使用方便,并且可以在服务端搭建服务通过浏览器使用。+- FunClip支持多段自由剪辑,并且会自动返回全视频SRT字幕、目标段落SRT字幕,使用简单方便。++欢迎体验使用,欢迎提出关于字幕生成或语音识别的需求与宝贵建议~++## 近期更新🚀++- 2024/03/06 命令行调用方式更新与问题修复,相关功能可以正常使用。+- 2024/02/28 FunClip升级到FunASR1.0模型调用方式,通过FunASR开源的SeACo-Paraformer模型在视频剪辑中进一步支持热词定制化功能。+- 2024/02/28 原FunASR-APP/ClipVideo更名为FunClip。++### 安装++#### Python环境安装++```shell+# 安装FunASR+pip install -U funasr+# 可以通过源码安装funasr+git clone https://github.com/alibaba-damo-academy/FunASR.git+cd FunASR; pip install -e ./+# 安装FunClip的Python依赖+pip install -r ./requirments.txt+# 根据你的环境(CUDA,Python版本等)安装torch和torchaudio+pip install torch==xxx torchaudio==xxx+```++#### 安装imagemagick(可选)++1. 如果你希望使用自动生成字幕的视频裁剪功能,需要安装imagemagick++- Ubuntu+```shell+apt-get -y update && apt-get -y install ffmpeg imagemagick+sed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xml+```+- MacOS+```shell+brew install imagemagick+sed -i 's/none/read,write/g' /usr/local/Cellar/imagemagick/7.1.1-8_1/etc/ImageMagick-7/policy.xml+```++2. 下载你需要的字体文件,这里我们提供一个默认的黑体字体文件++```shell+wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/STHeitiMedium.ttc -O font/STHeitiMedium.ttc+```++### 使用FunClip++#### A.在本地启动Gradio服务++```shell+python funclip/launch.py+```+随后在浏览器中访问```localhost:7860```即可看到如下图所示的界面,按如下步骤即可进行视频剪辑+1. 上传你的视频(或使用下方的视频用例)+2. (可选)设置热词,勾选是否使用说话人识别功能+3. 点击识别按钮获取识别结果+4. 将识别结果中的选段复制到对应位置,或者将说话人ID输入到对应为止+5. (可选)配置剪辑参数,偏移量与字幕设置等+6. 点击“裁剪”或“裁剪并添加字幕”按钮++<img src="docs/images/demo.png"/>++#### B.通过命令行调用使用FunClip的相关功能(更新中)+```shell+# 步骤一:识别+python funclip/videoclipper.py --stage 1 \+ --file examples/2022云栖大会_片段.mp4 \+ --output_dir ./output+# ./output中生成了识别结果与srt字幕等+# 步骤二:裁剪+python funclip/videoclipper.py --stage 2 \+ --file examples/2022云栖大会_片段.mp4 \+ --output_dir ./output \+ --dest_text '我们把它跟乡村振兴去结合起来,利用我们的设计的能力' \+ --start_ost 0 \+ --end_ost 100 \+ --output_file './output/res.mp4'+```++#### C.通过Modelscope创空间体验FunClip+[funclip创空间](https://modelscope.cn/studios/iic/funasr_app_clipvideo/summary)++### 社区交流++FunClip开源项目由FunASR社区维护,欢迎加入社区,交流与讨论,以及合作开发等。++| 钉钉群 | 微信群 |+|:-------------------------------------------------------------------:|:-----------------------------------------------------:|+| <div align="left"><img src="docs/images/dingding.png" width="250"/> | <img src="docs/images/wechat.png" width="215"/></div> |+++### 通过FunASR了解语音识别相关技术++[FunASR](https://github.com/alibaba-damo-academy/FunASR)是阿里巴巴通义实验室开源的端到端语音识别工具包,目前已经成为主流ASR工具包之一。其主要包括Python pipeline,SDK部署与海量开源工业ASR模型等。++📚FunASR论文: <a href="https://arxiv.org/abs/2305.11013"><img src="https://img.shields.io/badge/Arxiv-2305.11013-orange"></a>++📚SeACo-Paraformer论文:<a href="https://arxiv.org/abs/2308.03266"><img src="https://img.shields.io/badge/Arxiv-2308.03266-orange"></a>++⭐支持FunASR: <a href='https://github.com/alibaba-damo-academy/FunASR.stargazers'><img src='https://img.shields.io/github/stars/alibaba-damo-academy/FunASR.svg?style=social'></a>