FunASR-powered video transcription, subtitle generation, and LLM-assisted clipping tool with a local Gradio UI.
Python ai-toolsai-video-editingasrauto-subtitleschinesecontent-creationfunasrfunclipgradiollmparaformerspeech-recognitionspeech-to-textsubtitles-generatortranscriptionvideo-editingvideo-processingvideo-subtitlesvideo-transcriptionwhisper-alternative
update to paraclipper
shixian.shiafb6714ee811
diff --git a/ClipVideo/README.md b/ClipVideo/README.mddeleted file mode 100644index 4f9ec5d..0000000--- a/ClipVideo/README.md+++ /dev/null@@ -1,68 +0,0 @@-## ClipVideo--As the first application toolkit based of FunASR-APP, <strong>ClipVideo</strong> enables users to clip ```.mp4``` video files or ```.wav``` audio files with chosen text segments out of the recognition results generated by [Paraformer-long model](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary).--Under the help of ClipVideo you can get the video clips easily with the following steps (in Gradio service):-- Step1: Chose your video file (or try the example videos below)-- Step2: Copy the text segments you need to 'Text to Clip'-- Step3: Adjust subtitle settings (if needed)-- Step4: Click 'Clip' or 'Clip and Generate Subtitles'--### Usage-```shell-git clone https://github.com/alibaba-damo-academy/FunASR-APP.git-cd FunASR-APP-# install modelscope-pip install "modelscope[audio_asr]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html-# python environments-pip install -r ClipVideo/requirments.txt-```-(Optional) If you want to clip video file with embedded subtitles--1. ffmpeg and imagemagick is required--- On Ubuntu-```shell-apt-get -y update && apt-get -y install ffmpeg imagemagick-sed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xml-```-- On MacOS-```shell-brew install imagemagick-sed -i 's/none/read,write/g' /usr/local/Cellar/imagemagick/7.1.1-8_1/etc/ImageMagick-7/policy.xml-```-2. Download font file to ClipVideo/font--```shell-wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/STHeitiMedium.ttc -O ClipVideo/font/STHeitiMedium.ttc-```--#### Experience ClipVideo in Modelscope-You can try ClipVideo in modelscope space: [link](https://modelscope.cn/studios/damo/funasr_app_clipvideo/summary).--#### Use ClipVideo by Gradio Service-You can establish your own ClipVideo service which is same as [Modelscope Space](https://modelscope.cn/studios/damo/funasr_app_clipvideo/summary) as follow:-```shell-python clipvideo/gradio_service.py-```-then visit ```localhost:7860``` you will get a Gradio service like below and you can use ClipVideo following the steps:-<img src="./docs/images/show.png"/>--#### Use ClipVideo in command line-ClipVideo supports you to recognize and clip with commands:-```shell-# working in ClipVideo/-# step1: Recognize-python clipvideo/videoclipper.py --stage 1 \- --file examples/2022云栖大会_片段.mp4 \- --output_dir ./output-# now you can find recognition results and entire SRT file in ./output/-# step2: Clip-python clipvideo/videoclipper.py --stage 2 \- --file examples/2022云栖大会_片段.mp4 \- --output_dir ./output \- --dest_text '我们把它跟乡村振兴去结合起来,利用我们的设计的能力' \- --start_ost 0 \- --end_ost 100 \- --output_file './output/res.mp4'-```diff --git a/ClipVideo/clipvideo/video.py b/ClipVideo/clipvideo/video.pydeleted file mode 100644index ee70592..0000000--- a/ClipVideo/clipvideo/video.py+++ /dev/null@@ -1,72 +0,0 @@-import gradio-from pathlib import Path-from ffmpy import FFmpeg-from gradio import utils-from gradio_client.documentation import document, set_documentation_group-set_documentation_group("component")--@document()-class Video(gradio.Video):- """- This is adapted from gradio.Video. Only `def preprocess()` is modified for auto recoginzing input x is local data or url- """- def preprocess(self, x):- """- Parameters:- x: A tuple of (video file data, subtitle file data) or just video file data.- Returns:- A string file path or URL to the preprocessed video. Subtitle file data is ignored.- """- if x is None:- return None- elif isinstance(x, dict):- video = x- else:- video = x[0]-- file_name, file_data, is_file = (- video.get("name"),- video["data"],- video.get("is_file", False),- )-- if is_file:- assert file_name is not None, "Received file data without a file name."- if utils.validate_url(file_name): # URL path- file_name = Path(self.download_temp_copy_if_needed(file_name))- else: # Local path- file_name = Path(self.make_temp_copy_if_needed(file_name))- else:- assert file_data is not None, "Received empty file data."- file_name = Path(self.base64_to_temp_file_if_needed(file_data, file_name))-- uploaded_format = file_name.suffix.replace(".", "")- needs_formatting = self.format is not None and uploaded_format != self.format- flip = self.source == "webcam" and self.mirror_webcam-- if needs_formatting or flip:- format = f".{self.format if needs_formatting else uploaded_format}"- output_options = ["-vf", "hflip", "-c:a", "copy"] if flip else []- output_options += ["-an"] if not self.include_audio else []- flip_suffix = "_flip" if flip else ""- output_file_name = str(- file_name.with_name(f"{file_name.stem}{flip_suffix}{format}")- )- if Path(output_file_name).exists():- return output_file_name- ff = FFmpeg(- inputs={str(file_name): None},- outputs={output_file_name: output_options},- )- ff.run()- return output_file_name- elif not self.include_audio:- output_file_name = str(file_name.with_name(f"muted_{file_name.name}"))- ff = FFmpeg(- inputs={str(file_name): None},- outputs={output_file_name: ["-an"]},- )- ff.run()- return output_file_name- else:- return str(file_name)diff --git a/ClipVideo/clipvideo/video.pyi b/ClipVideo/clipvideo/video.pyideleted file mode 100644index ab346c5..0000000--- a/ClipVideo/clipvideo/video.pyi+++ /dev/null@@ -1,73 +0,0 @@-import gradio-from pathlib import Path-from ffmpy import FFmpeg-from gradio import utils-from gradio_client.documentation import document, set_documentation_group-set_documentation_group("component")-from gradio.events import Dependency--@document()-class Video(gradio.Video):- """- This is adapted from gradio.Video. Only `def preprocess()` is modified for auto recoginzing input x is local data or url- """- def preprocess(self, x):- """- Parameters:- x: A tuple of (video file data, subtitle file data) or just video file data.- Returns:- A string file path or URL to the preprocessed video. Subtitle file data is ignored.- """- if x is None:- return None- elif isinstance(x, dict):- video = x- else:- video = x[0]-- file_name, file_data, is_file = (- video.get("name"),- video["data"],- video.get("is_file", False),- )-- if is_file:- assert file_name is not None, "Received file data without a file name."- if utils.validate_url(file_name): # URL path- file_name = Path(self.download_temp_copy_if_needed(file_name))- else: # Local path- file_name = Path(self.make_temp_copy_if_needed(file_name))- else:- assert file_data is not None, "Received empty file data."- file_name = Path(self.base64_to_temp_file_if_needed(file_data, file_name))-- uploaded_format = file_name.suffix.replace(".", "")- needs_formatting = self.format is not None and uploaded_format != self.format- flip = self.source == "webcam" and self.mirror_webcam-- if needs_formatting or flip:- format = f".{self.format if needs_formatting else uploaded_format}"- output_options = ["-vf", "hflip", "-c:a", "copy"] if flip else []- output_options += ["-an"] if not self.include_audio else []- flip_suffix = "_flip" if flip else ""- output_file_name = str(- file_name.with_name(f"{file_name.stem}{flip_suffix}{format}")- )- if Path(output_file_name).exists():- return output_file_name- ff = FFmpeg(- inputs={str(file_name): None},- outputs={output_file_name: output_options},- )- ff.run()- return output_file_name- elif not self.include_audio:- output_file_name = str(file_name.with_name(f"muted_{file_name.name}"))- ff = FFmpeg(- inputs={str(file_name): None},- outputs={output_file_name: ["-an"]},- )- ff.run()- return output_file_name- else:- return str(file_name)\ No newline at end of filediff --git a/ParaClipper/_TransAudio/README.md b/ParaClipper/_TransAudio/README.mdnew file mode 100644index 0000000..6c36bfd--- /dev/null+++ b/ParaClipper/_TransAudio/README.md@@ -0,0 +1 @@+TransAudio is no longer under support, please use SDK in funasr(https://github.com/alibaba-damo-academy/FunASR) to establish your ASR service.\ No newline at end of filediff --git a/TransAudio/funasr-runtime-deploy.sh b/ParaClipper/_TransAudio/funasr-runtime-deploy.shsimilarity index 100%rename from TransAudio/funasr-runtime-deploy.shrename to ParaClipper/_TransAudio/funasr-runtime-deploy.shdiff --git a/ClipVideo/docs/images/show.png b/ParaClipper/docs/images/show.pngsimilarity index 100%rename from ClipVideo/docs/images/show.pngrename to ParaClipper/docs/images/show.pngdiff --git a/ClipVideo/docs/images/show2.0.png b/ParaClipper/docs/images/show2.0.pngsimilarity index 100%rename from ClipVideo/docs/images/show2.0.pngrename to ParaClipper/docs/images/show2.0.pngdiff --git a/ClipVideo/font/STHeitiMedium.ttc b/ParaClipper/font/STHeitiMedium.ttcsimilarity index 100%rename from ClipVideo/font/STHeitiMedium.ttcrename to ParaClipper/font/STHeitiMedium.ttcdiff --git a/ClipVideo/requirments.txt b/ParaClipper/requirments.txtsimilarity index 100%rename from ClipVideo/requirments.txtrename to ParaClipper/requirments.txtdiff --git a/ClipVideo/clipvideo/__init__.py b/ParaClipper/src/__init__.pysimilarity index 100%rename from ClipVideo/clipvideo/__init__.pyrename to ParaClipper/src/__init__.pydiff --git a/ClipVideo/clipvideo/argparse_tools.py b/ParaClipper/src/argparse_tools.pysimilarity index 100%rename from ClipVideo/clipvideo/argparse_tools.pyrename to ParaClipper/src/argparse_tools.pydiff --git a/ClipVideo/clipvideo/imagemagick_test.py b/ParaClipper/src/imagemagick_test.pysimilarity index 100%rename from ClipVideo/clipvideo/imagemagick_test.pyrename to ParaClipper/src/imagemagick_test.pydiff --git a/ClipVideo/clipvideo/gradio_service.py b/ParaClipper/src/launch.pysimilarity index 97%rename from ClipVideo/clipvideo/gradio_service.pyrename to ParaClipper/src/launch.pyindex a4302c7..db77b33 100644--- a/ClipVideo/clipvideo/gradio_service.py+++ b/ParaClipper/src/launch.py@@ -1,7 +1,6 @@import gradio as grfrom funasr import AutoModelfrom videoclipper import VideoClipper-# from video import Videoif __name__ == "__main__":@@ -33,7 +32,7 @@ if __name__ == "__main__":return audio_clipper.video_clip(dest_text, start_ost, end_ost, state, font_size, font_color, add_sub=True, dest_spk=video_spk_input)top_md_1 = ("""- 基于达摩院自研Paraformer-长音频版的语音识别、端点检测、标点预测、时间戳功能、角色区分功能+ 基于阿里巴巴通义实验室自研SeACo-Paraformer-长音频版的语音识别、端点检测、标点预测、时间戳功能、角色区分、热词定制化功能准确识别,自由复制所需段落,或者设置说话人标识,一键裁剪、添加字幕@@ -53,7 +52,7 @@ if __name__ == "__main__":</div>""")- top_md_3 = ("""访问FunASR项目与论文能够帮助您深入了解ClipVideo中所使用的语音处理相关模型:+ top_md_3 = ("""访问FunASR项目与论文能够帮助您深入了解ParaClipper中所使用的语音处理相关模型:<div align="center"><div style="display:flex; gap: 0.25rem;" align="center">FunASR: <a href='https://github.com/alibaba-damo-academy/FunASR'><img src='https://img.shields.io/badge/Github-Code-blue'></a>diff --git a/ClipVideo/clipvideo/subtitle_utils.py b/ParaClipper/src/subtitle_utils.pysimilarity index 100%rename from ClipVideo/clipvideo/subtitle_utils.pyrename to ParaClipper/src/subtitle_utils.pydiff --git a/ClipVideo/clipvideo/test.sh b/ParaClipper/src/test.shsimilarity index 100%rename from ClipVideo/clipvideo/test.shrename to ParaClipper/src/test.shdiff --git a/ClipVideo/clipvideo/trans_utils.py b/ParaClipper/src/trans_utils.pysimilarity index 100%rename from ClipVideo/clipvideo/trans_utils.pyrename to ParaClipper/src/trans_utils.pydiff --git a/ClipVideo/clipvideo/videoclipper.py b/ParaClipper/src/videoclipper.pysimilarity index 100%rename from ClipVideo/clipvideo/videoclipper.pyrename to ParaClipper/src/videoclipper.pydiff --git a/README.md b/README_en.mdsimilarity index 51%rename from README.mdrename to README_en.mdindex 5f86273..804408f 100644--- a/README.md+++ b/README_en.md@@ -1,19 +1,16 @@-# FunASR-APP--<strong>FunASR-APP</strong> is a comprehensive speech application toolkit designed to facilitate the application and integration of [FunASR](https://github.com/alibaba-damo-academy/FunASR)'s open-source speech models. Its primary goal is to package the models into convenient application packages, enabling easy application and seamless integration.+# ParaClipper## What's New-- 2024/02/28 Update call function of funasr into funasr1.0, use SeACo_Paraformer thus ASR now supports hotword. Also support destination transcription like 'abcd[-100,150]#efgh[200,200]' to adjust offset time for every sub-sentence (TODO: use ClipVideo with command line is not updated currently).+- 2024/02/28 Update call function of funasr into funasr1.0, use SeACo_Paraformer thus ASR now supports hotword. Also support destination transcription like 'abcd[-100,150]#efgh[200,200]' to adjust offset time for every sub-sentence (TODO: use ParaClipper with command line is not updated currently).- 2023/10/17 Bug fix for multiple periods chosen, used to return video with wrong length.-- 2023/10/10 ClipVideo now supports recognizing with speaker diarization ability, choose 'yes' button in 'Recognize Speakers' and you will get recognition results with speaker id for each sentence. And then you can clip out the periods of one or some speakers (e.g. 'spk0' or 'spk0#spk3') using ClipVideo.-+- 2023/10/10 ParaClipper now supports recognizing with speaker diarization ability, choose 'yes' button in 'Recognize Speakers' and you will get recognition results with speaker id for each sentence. And then you can clip out the periods of one or some speakers (e.g. 'spk0' or 'spk0#spk3') using ParaClipper.-## ClipVideo+## Use ParaClipper to Clip Your Video-As the first application toolkit of FunASR-APP, <strong>ClipVideo</strong> enables users to clip ```.mp4``` video files or ```.wav``` audio files with chosen text segments out of the recognition results generated by [Paraformer-long model](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary).+ParaClipper</strong> enables users to clip ```.mp4``` video files or ```.wav``` audio files with chosen text segments out of the recognition results generated by [Paraformer-long model](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary).-Under the help of ClipVideo you can get the video clips easily with the following steps (in Gradio service):+Under the help of ParaClipper you can get the video clips easily with the following steps (in Gradio service):- Step1: Upload your video file (or try the example videos below)- Step2: Copy the text segments you need to 'Text to Clip'- Step3: Adjust subtitle settings (if needed)@@ -22,13 +19,9 @@ Under the help of ClipVideo you can get the video clips easily with the followin### Usage```shell# install funasr (necessary)-git clone https://github.com/alibaba-damo-academy/FunASR.git-cd FunASR-pip install -e ./-# python environments-git clone https://github.com/alibaba-damo-academy/FunASR-APP.git-cd FunASR-APP-pip install -r ClipVideo/requirments.txt+pip install -U funasr+# install python dependency+pip install -r ParaClipper/requirments.txt```(Optional) If you want to clip video file with embedded subtitles@@ -44,34 +37,34 @@ sed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xmlbrew install imagemagicksed -i 's/none/read,write/g' /usr/local/Cellar/imagemagick/7.1.1-8_1/etc/ImageMagick-7/policy.xml```-2. Download font file to ClipVideo/font+2. Download font file to ParaClipper/font```shell-wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/STHeitiMedium.ttc -O ClipVideo/font/STHeitiMedium.ttc+wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ParaClipper/STHeitiMedium.ttc -O ParaClipper/font/STHeitiMedium.ttc```-#### Experience ClipVideo in Modelscope-You can try ClipVideo in modelscope space: [link](https://modelscope.cn/studios/damo/funasr_app_clipvideo/summary).+#### Experience ParaClipper in Modelscope+You can try ParaClipper in modelscope space: [link](https://modelscope.cn/studios/damo/funasr_app_ParaClipper/summary).-#### Use ClipVideo as Gradio Service-You can establish your own ClipVideo service which is same as [Modelscope Space](https://modelscope.cn/studios/damo/funasr_app_clipvideo/summary) as follow:+#### Use ParaClipper as Gradio Service+You can establish your own ParaClipper service which is same as [Modelscope Space](https://modelscope.cn/studios/damo/funasr_app_ParaClipper/summary) as follow:```shell-python clipvideo/gradio_service.py+python ParaClipper/gradio_service.py```-then visit ```localhost:7860``` you will get a Gradio service like below and you can use ClipVideo following the steps:-<img src="ClipVideo/docs/images/show2.0.png"/>+then visit ```localhost:7860``` you will get a Gradio service like below and you can use ParaClipper following the steps:+<img src="ParaClipper/docs/images/show2.0.png"/>-#### Use ClipVideo in command line-ClipVideo supports you to recognize and clip with commands:+#### Use ParaClipper in command line+ParaClipper supports you to recognize and clip with commands:```shell-# working in ClipVideo/+# working in ParaClipper/# step1: Recognize-python clipvideo/videoclipper.py --stage 1 \+python ParaClipper/videoclipper.py --stage 1 \--file examples/2022云栖大会_片段.mp4 \--output_dir ./output# now you can find recognition results and entire SRT file in ./output/# step2: Clip-python clipvideo/videoclipper.py --stage 2 \+python ParaClipper/videoclipper.py --stage 2 \--file examples/2022云栖大会_片段.mp4 \--output_dir ./output \--dest_text '我们把它跟乡村振兴去结合起来,利用我们的设计的能力' \diff --git a/TransAudio/README.md b/TransAudio/README.mddeleted file mode 100644index e69de29..0000000