FunASR-powered video transcription, subtitle generation, and LLM-assisted clipping tool with a local Gradio UI.
Python ai-toolsai-video-editingasrauto-subtitleschinesecontent-creationfunasrfunclipgradiollmparaformerspeech-recognitionspeech-to-textsubtitles-generatortranscriptionvideo-editingvideo-processingvideo-subtitlesvideo-transcriptionwhisper-alternative
update files
shixian.shid3dc616d22f8
diff --git a/.gitignore b/.gitignorenew file mode 100644index 0000000..b7b6fd3--- /dev/null+++ b/.gitignore@@ -0,0 +1,5 @@+*.pyc+.DS_Store+*.DS_Store+ClipVideo/clipvideo/output+*__pycache__\ No newline at end of filediff --git a/ClipVideo/README.md b/ClipVideo/README.mdnew file mode 100644index 0000000..4f9ec5d--- /dev/null+++ b/ClipVideo/README.md@@ -0,0 +1,68 @@+## ClipVideo++As the first application toolkit based of FunASR-APP, <strong>ClipVideo</strong> enables users to clip ```.mp4``` video files or ```.wav``` audio files with chosen text segments out of the recognition results generated by [Paraformer-long model](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary).++Under the help of ClipVideo you can get the video clips easily with the following steps (in Gradio service):+- Step1: Chose your video file (or try the example videos below)+- Step2: Copy the text segments you need to 'Text to Clip'+- Step3: Adjust subtitle settings (if needed)+- Step4: Click 'Clip' or 'Clip and Generate Subtitles'++### Usage+```shell+git clone https://github.com/alibaba-damo-academy/FunASR-APP.git+cd FunASR-APP+# install modelscope+pip install "modelscope[audio_asr]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html+# python environments+pip install -r ClipVideo/requirments.txt+```+(Optional) If you want to clip video file with embedded subtitles++1. ffmpeg and imagemagick is required++- On Ubuntu+```shell+apt-get -y update && apt-get -y install ffmpeg imagemagick+sed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xml+```+- On MacOS+```shell+brew install imagemagick+sed -i 's/none/read,write/g' /usr/local/Cellar/imagemagick/7.1.1-8_1/etc/ImageMagick-7/policy.xml+```+2. Download font file to ClipVideo/font++```shell+wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/STHeitiMedium.ttc -O ClipVideo/font/STHeitiMedium.ttc+```++#### Experience ClipVideo in Modelscope+You can try ClipVideo in modelscope space: [link](https://modelscope.cn/studios/damo/funasr_app_clipvideo/summary).++#### Use ClipVideo by Gradio Service+You can establish your own ClipVideo service which is same as [Modelscope Space](https://modelscope.cn/studios/damo/funasr_app_clipvideo/summary) as follow:+```shell+python clipvideo/gradio_service.py+```+then visit ```localhost:7860``` you will get a Gradio service like below and you can use ClipVideo following the steps:+<img src="./docs/images/show.png"/>++#### Use ClipVideo in command line+ClipVideo supports you to recognize and clip with commands:+```shell+# working in ClipVideo/+# step1: Recognize+python clipvideo/videoclipper.py --stage 1 \+ --file examples/2022云栖大会_片段.mp4 \+ --output_dir ./output+# now you can find recognition results and entire SRT file in ./output/+# step2: Clip+python clipvideo/videoclipper.py --stage 2 \+ --file examples/2022云栖大会_片段.mp4 \+ --output_dir ./output \+ --dest_text '我们把它跟乡村振兴去结合起来,利用我们的设计的能力' \+ --start_ost 0 \+ --end_ost 100 \+ --output_file './output/res.mp4'+```diff --git a/ClipVideo/clipvideo/__init__.py b/ClipVideo/clipvideo/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/ClipVideo/clipvideo/argparse_tools.py b/ClipVideo/clipvideo/argparse_tools.pynew file mode 100644index 0000000..fc621ed--- /dev/null+++ b/ClipVideo/clipvideo/argparse_tools.py@@ -0,0 +1,83 @@+import argparse+from pathlib import Path++import yaml+import sys+++class ArgumentParser(argparse.ArgumentParser):+ """Simple implementation of ArgumentParser supporting config file++ This class is originated from https://github.com/bw2/ConfigArgParse,+ but this class is lack of some features that it has.++ - Not supporting multiple config files+ - Automatically adding "--config" as an option.+ - Not supporting any formats other than yaml+ - Not checking argument type++ """++ def __init__(self, *args, **kwargs):+ super().__init__(*args, **kwargs)+ self.add_argument("--config", help="Give config file in yaml format")++ def parse_known_args(self, args=None, namespace=None):+ # Once parsing for setting from "--config"+ _args, _ = super().parse_known_args(args, namespace)+ if _args.config is not None:+ if not Path(_args.config).exists():+ self.error(f"No such file: {_args.config}")++ with open(_args.config, "r", encoding="utf-8") as f:+ d = yaml.safe_load(f)+ if not isinstance(d, dict):+ self.error("Config file has non dict value: {_args.config}")++ for key in d:+ for action in self._actions:+ if key == action.dest:+ break+ else:+ self.error(f"unrecognized arguments: {key} (from {_args.config})")++ # NOTE(kamo): Ignore "--config" from a config file+ # NOTE(kamo): Unlike "configargparse", this module doesn't check type.+ # i.e. We can set any type value regardless of argument type.+ self.set_defaults(**d)+ return super().parse_known_args(args, namespace)+++def get_commandline_args():+ extra_chars = [+ " ",+ ";",+ "&",+ "(",+ ")",+ "|",+ "^",+ "<",+ ">",+ "?",+ "*",+ "[",+ "]",+ "$",+ "`",+ '"',+ "\\",+ "!",+ "{",+ "}",+ ]++ # Escape the extra characters for shell+ argv = [+ arg.replace("'", "'\\''")+ if all(char not in arg for char in extra_chars)+ else "'" + arg.replace("'", "'\\''") + "'"+ for arg in sys.argv+ ]++ return sys.executable + " " + " ".join(argv)\ No newline at end of filediff --git a/ClipVideo/clipvideo/gradio_service.py b/ClipVideo/clipvideo/gradio_service.pynew file mode 100644index 0000000..514479d--- /dev/null+++ b/ClipVideo/clipvideo/gradio_service.py@@ -0,0 +1,137 @@+import gradio as gr+from modelscope.pipelines import pipeline+from modelscope.utils.constant import Tasks+from videoclipper import VideoClipper+++if __name__ == "__main__":+ inference_pipeline = pipeline(+ task=Tasks.auto_speech_recognition,+ model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',+ vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',+ punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',+ )+ audio_clipper = VideoClipper(inference_pipeline)++ def audio_recog(audio_input):+ return audio_clipper.recog(audio_input)++ def audio_clip(dest_text, start_ost, end_ost, state):+ return audio_clipper.clip(dest_text, start_ost, end_ost, state)++ def video_recog(video_input):+ return audio_clipper.video_recog(video_input)++ def video_clip(dest_text, start_ost, end_ost, state):+ return audio_clipper.video_clip(dest_text, start_ost, end_ost, state)++ def video_clip_addsub(dest_text, start_ost, end_ost, state, font_size, font_color):+ return audio_clipper.video_clip(dest_text, start_ost, end_ost, state, font_size, font_color, add_sub=True)++ '''+ top_md_1 = ("""+ 基于达摩院自研Paraformer-长音频版的语音识别、端点检测、标点预测、时间戳功能++ 准确识别,自由复制所需段落并一键裁剪、添加字幕++ * Step1: 上传视频文件(或使用下方的用例体验),点击 **<font color="#f7802b">识别</font>** 按钮+ * Step2: 复制识别结果中所需的文字至右上方,设置偏移与字幕配置(可选)+ * Step3: 点击 **<font color="#f7802b">裁剪</font>** 按钮或 **<font color="#f7802b">裁剪并添加字幕</font>** 按钮获得结果+ """)+ '''++ top_md_2 = ("""+ 受到网络传输与服务资源的限制,用于体验的视频最好大小在40mb以下+ 过大的视频可以尝试分离音轨使用音频剪辑,或 **<font color="#1785c4">通过源代码将您的ClipVideo服务部署在本地(推荐)</font>** :+ <div align="center">+ <div style="display:flex; gap: 0.25rem;" align="center">+ FunASR_APP: <a href='https://github.com/alibaba/funasr-app'><img src='https://img.shields.io/badge/Github-Code-blue'></a>+ 🌟支持我们: <a href='https://github.com/alibaba/funasr-app/stargazers'><img src='https://img.shields.io/github/stars/alibaba/funasr-app.svg?style=social'></a>+ </div>+ </div>+ """)++ top_md_3 = ("""访问FunASR项目与论文能够帮助您深入了解ClipVideo中所使用的语音处理相关模型:+ <div align="center">+ <div style="display:flex; gap: 0.25rem;" align="center">+ FunASR: <a href='https://github.com/alibaba-damo-academy/FunASR'><img src='https://img.shields.io/badge/Github-Code-blue'></a>+ FunASR Paper: <a href="https://arxiv.org/abs/2305.11013"><img src="https://img.shields.io/badge/Arxiv-2305.11013-orange"></a>+ 🌟Star FunASR: <a href='https://github.com/alibaba-damo-academy/FunASR/stargazers'><img src='https://img.shields.io/github/stars/alibaba-damo-academy/FunASR.svg?style=social'></a>+ </div>+ </div>+ """)++ # gradio interface+ with gr.Blocks() as demo:+ #gr.Image("./examples/guide.png", show_label=False)+ # gr.Markdown(top_md_1)+ #gr.Markdown(top_md_2)+ #gr.Markdown(top_md_3)+ video_state = gr.State()+ audio_state = gr.State()+ with gr.Tab("🎥✂️视频裁剪 Video Clipping"):+ with gr.Row():+ with gr.Column():+ video_input = gr.Video(label="🎥视频输入 Video Input")+ gr.Examples(['https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/%E4%B8%BA%E4%BB%80%E4%B9%88%E8%A6%81%E5%A4%9A%E8%AF%BB%E4%B9%A6%EF%BC%9F%E8%BF%99%E6%98%AF%E6%88%91%E5%90%AC%E8%BF%87%E6%9C%80%E5%A5%BD%E7%9A%84%E7%AD%94%E6%A1%88-%E7%89%87%E6%AE%B5.mp4',+ 'https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/2022%E4%BA%91%E6%A0%96%E5%A4%A7%E4%BC%9A_%E7%89%87%E6%AE%B5.mp4',+ 'https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/2022%E4%BA%91%E6%A0%96%E5%A4%A7%E4%BC%9A_%E7%89%87%E6%AE%B52.mp4',+ 'https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/%E4%BD%BF%E7%94%A8chatgpt_%E7%89%87%E6%AE%B5.mp4'],+ [video_input])+ recog_button2 = gr.Button("👂识别 Recognize")+ video_text_output = gr.Textbox(label="✏️识别结果 Recognition Result")+ video_srt_output = gr.Textbox(label="📖SRT字幕内容 RST Subtitles")+ with gr.Column():+ video_text_input = gr.Textbox(label="✏️待裁剪文本 Text to Clip (多段文本使用'#'连接)")+ with gr.Row():+ video_start_ost = gr.Slider(minimum=-500, maximum=1000, value=0, step=50, label="⏪开始位置偏移 Start Offset (ms)")+ video_end_ost = gr.Slider(minimum=-500, maximum=1000, value=100, step=50, label="⏩结束位置偏移 End Offset (ms)")+ with gr.Row():+ font_size = gr.Slider(minimum=10, maximum=100, value=32, step=2, label="🔠字幕字体大小 Subtitle Font Size")+ font_color = gr.Radio(["black", "white", "green", "red"], label="🌈字幕颜色 Subtitle Color", value='white')+ # font = gr.Radio(["黑体", "Alibaba Sans"], label="字体 Font")+ with gr.Row():+ clip_button2 = gr.Button("✂️裁剪\nClip")+ clip_button3 = gr.Button("✂️裁剪并添加字幕\nClip and Generate Subtitles")+ video_output = gr.Video(label="🎥裁剪结果 Audio Clipped")+ video_mess_output = gr.Textbox(label="ℹ️裁剪信息 Clipping Log")+ video_srt_clip_output = gr.Textbox(label="📖裁剪部分SRT字幕内容 Clipped RST Subtitles")++ with gr.Tab("🔊✂️音频裁剪 Audio Clipping"):+ with gr.Row():+ with gr.Column():+ audio_input = gr.Audio(label="🔊音频输入 Audio Input")+ gr.Examples(['https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/%E9%B2%81%E8%82%83%E9%87%87%E8%AE%BF%E7%89%87%E6%AE%B51.wav'], [audio_input])+ recog_button1 = gr.Button("👂识别 Recognize")+ audio_text_output = gr.Textbox(label="✏️识别结果 Recognition Result")+ audio_srt_output = gr.Textbox(label="📖SRT字幕内容 RST Subtitles")+ with gr.Column():+ audio_text_input = gr.Textbox(label="✏️待裁剪文本 Text to Clip (多段文本使用'#'连接)")+ with gr.Row():+ audio_start_ost = gr.Slider(minimum=-500, maximum=1000, value=0, step=50, label="⏪开始位置偏移 Start Offset (ms)")+ audio_end_ost = gr.Slider(minimum=-500, maximum=1000, value=100, step=50, label="⏩结束位置偏移 End Offset (ms)")+ with gr.Row():+ clip_button1 = gr.Button("✂️裁剪 Clip")+ audio_output = gr.Audio(label="🔊裁剪结果 Audio Clipped")+ audio_mess_output = gr.Textbox(label="ℹ️裁剪信息 Clipping Log")+ audio_srt_clip_output = gr.Textbox(label="📖裁剪部分SRT字幕内容 Clipped RST Subtitles")++ recog_button1.click(audio_recog,+ inputs=audio_input,+ outputs=[audio_text_output, audio_srt_output, audio_state])+ clip_button1.click(audio_clip,+ inputs=[audio_text_input, audio_start_ost, audio_end_ost, audio_state],+ outputs=[audio_output, audio_mess_output, audio_srt_clip_output])++ recog_button2.click(video_recog,+ inputs=video_input,+ outputs=[video_text_output, video_srt_output, video_state])+ clip_button2.click(video_clip,+ inputs=[video_text_input, video_start_ost, video_end_ost, video_state],+ outputs=[video_output, video_mess_output, video_srt_clip_output])+ clip_button3.click(video_clip_addsub,+ inputs=[video_text_input, video_start_ost, video_end_ost, video_state, font_size, font_color],+ outputs=[video_output, video_mess_output, video_srt_clip_output])++ # start gradio service in local+ demo.queue(concurrency_count=3).launch()diff --git a/ClipVideo/clipvideo/imagemagick_test.py b/ClipVideo/clipvideo/imagemagick_test.pynew file mode 100644index 0000000..662c5f6--- /dev/null+++ b/ClipVideo/clipvideo/imagemagick_test.py@@ -0,0 +1,16 @@+from moviepy.editor import *+from moviepy.video.tools.subtitles import SubtitlesClip++generator = lambda txt: TextClip(txt, font='./font/STHeitiMedium.ttc', fontsize=48, color='white')+subs = [((0, 2), 'sub1中文字幕'),+ ((2, 4), 'subs2'),+ ((4, 6), 'subs3'),+ ((6, 8), 'subs4')]++subtitles = SubtitlesClip(subs, generator)++video = VideoFileClip("examples/2022云栖大会_片段.mp4.mp4")+video = video.subclip(0, 8)+video = CompositeVideoClip([video, subtitles.set_pos(('center','bottom'))])++video.write_videofile("test_output.mp4")\ No newline at end of filediff --git a/ClipVideo/clipvideo/subtitle_utils.py b/ClipVideo/clipvideo/subtitle_utils.pynew file mode 100644index 0000000..a3d9a47--- /dev/null+++ b/ClipVideo/clipvideo/subtitle_utils.py@@ -0,0 +1,105 @@+def time_convert(ms):+ ms = int(ms)+ tail = ms % 1000+ s = ms // 1000+ mi = s // 60+ s = s % 60+ h = mi // 60+ mi = mi % 60+ h = "00" if h == 0 else str(h)+ mi = "00" if mi == 0 else str(mi)+ s = "00" if s == 0 else str(s)+ tail = str(tail)+ if len(h) == 1: h = '0' + h+ if len(mi) == 1: mi = '0' + mi+ if len(s) == 1: s = '0' + s+ return "{}:{}:{},{}".format(h, mi, s, tail)+++class Text2SRT():+ def __init__(self, text_seg, ts_list, offset=0):+ self.token_list = [i for i in text_seg.split() if len(i)]+ self.ts_list = ts_list+ start, end = ts_list[0][0] - offset, ts_list[-1][1] - offset+ self.start_sec, self.end_sec = start, end+ self.start_time = time_convert(start)+ self.end_time = time_convert(end)+ def text(self):+ res = ""+ for word in self.token_list:+ if '\u4e00' <= word <= '\u9fff':+ res += word+ else:+ res += " " + word+ return res+ def len(self):+ return len(self.token_list)+ def srt(self):+ return "{} --> {}\n{}\n".format(self.start_time, self.end_time, self.text())+ def time(self):+ return (self.start_sec/1000, self.end_sec/1000)+++def generate_srt(sentence_list):+ srt_total = ''+ for i, d in enumerate(sentence_list):+ t2s = Text2SRT(d['text_seg'], d['ts_list'])+ srt_total += "{}\n{}".format(i, t2s.srt())+ return srt_total++def generate_srt_clip(sentence_list, start, end, begin_index=0):+ start, end = int(start * 1000), int(end * 1000)+ srt_total = ''+ cc = 1 + begin_index+ subs = []+ for i, d in enumerate(sentence_list):+ if d['ts_list'][-1][1] <= start:+ continue+ if d['ts_list'][0][0] >= end:+ break+ # parts in between+ if (d['ts_list'][-1][1] < end and d['ts_list'][0][0] > start) or (d['ts_list'][-1][1] == end and d['ts_list'][0][0] == start):+ t2s = Text2SRT(d['text_seg'], d['ts_list'], offset=start)+ srt_total += "{}\n{}".format(cc, t2s.srt())+ subs.append((t2s.time(), t2s.text()))+ cc += 1+ continue+ if d['ts_list'][0][0] <= start:+ if not d['ts_list'][-1][1] > end:+ for j, ts in enumerate(d['ts_list']):+ if ts[1] > start:+ break+ _text = " ".join(d['text_seg'].split()[j:])+ _ts = d['ts_list'][j:]+ else:+ for j, ts in enumerate(d['ts_list']):+ if ts[1] > start:+ _start = j+ break+ for j, ts in enumerate(d['ts_list']):+ if ts[1] > end:+ _end = j+ break+ _text = " ".join(d['text_seg'].split()[_start:_end])+ _ts = d['ts_list'][_start:_end]+ if len(ts):+ t2s = Text2SRT(_text, _ts, offset=start)+ srt_total += "{}\n{}".format(cc, t2s.srt())+ subs.append((t2s.time(), t2s.text()))+ cc += 1+ continue+ if d['ts_list'][-1][1] > end:+ for j, ts in enumerate(d['ts_list']):+ if ts[1] > end:+ break+ _text = " ".join(d['text_seg'].split()[:j])+ _ts = d['ts_list'][:j]+ if len(_ts):+ t2s = Text2SRT(_text, _ts, offset=start)+ srt_total += "{}\n{}".format(cc, t2s.srt())+ subs.append(+ (t2s.time(), t2s.text())+ )+ cc += 1+ continue+ return srt_total, subs, ccdiff --git a/ClipVideo/clipvideo/test.sh b/ClipVideo/clipvideo/test.shnew file mode 100755index 0000000..fa2549b--- /dev/null+++ b/ClipVideo/clipvideo/test.sh@@ -0,0 +1,13 @@+# step1: Recognize+python videoclipper.py --stage 1 \+ --file ../examples/2022云栖大会_片段.mp4 \+ --output_dir ./output+# now you can find recognition results and entire SRT file in ./output/+# step2: Clip+python videoclipper.py --stage 2 \+ --file ../examples/2022云栖大会_片段.mp4 \+ --output_dir ./output \+ --dest_text '所以这个是我们办这个奖的初心啊,我们也会一届一届的办下去' \+ --start_ost 0 \+ --end_ost 100 \+ --output_file './output/res.mp4'\ No newline at end of filediff --git a/ClipVideo/clipvideo/trans_utils.py b/ClipVideo/clipvideo/trans_utils.pynew file mode 100644index 0000000..10bd7c7--- /dev/null+++ b/ClipVideo/clipvideo/trans_utils.py@@ -0,0 +1,57 @@+PUNC_LIST = [',', '。', '!', '?', '、']+++def pre_proc(text):+ res = ''+ for i in range(len(text)):+ if text[i] in PUNC_LIST:+ continue+ if '\u4e00' <= text[i] <= '\u9fff':+ if len(res) and res[-1] != " ":+ res += ' ' + text[i]+' '+ else:+ res += text[i]+' '+ else:+ res += text[i]+ if res[-1] == ' ':+ res = res[:-1]+ return res++def proc(raw_text, timestamp, dest_text):+ # simple matching+ ld = len(dest_text.split())+ mi, ts = [], []+ offset = 0+ while True:+ fi = raw_text.find(dest_text, offset, len(raw_text))+ # import pdb; pdb.set_trace()+ ti = raw_text[:fi].count(' ')+ if fi == -1:+ break+ offset = fi + ld+ mi.append(fi)+ ts.append([timestamp[ti][0]*16, timestamp[ti+ld-1][1]*16])+ # import pdb; pdb.set_trace()+ return ts+++def write_state(output_dir, state):+ for key in ['/recog_res_raw', '/timestamp', '/sentences']:+ with open(output_dir+key, 'w') as fout:+ fout.write(str(state[key[1:]]))+++def load_state(output_dir):+ state = {}+ with open(output_dir+'/recog_res_raw') as fin:+ line = fin.read()+ state['recog_res_raw'] = line+ with open(output_dir+'/timestamp') as fin:+ line = fin.read()+ state['timestamp'] = eval(line)+ with open(output_dir+'/sentences') as fin:+ line = fin.read()+ state['sentences'] = eval(line)+ return state++\ No newline at end of filediff --git a/ClipVideo/clipvideo/videoclipper.py b/ClipVideo/clipvideo/videoclipper.pynew file mode 100644index 0000000..cb30fcb--- /dev/null+++ b/ClipVideo/clipvideo/videoclipper.py@@ -0,0 +1,278 @@+import sys+import copy+import librosa+import logging+import argparse+import numpy as np+import soundfile as sf+import moviepy.editor as mpy+# from modelscope.pipelines import pipeline+# from modelscope.utils.constant import Tasks+from subtitle_utils import generate_srt, generate_srt_clip+from trans_utils import pre_proc, proc, write_state, load_state+from argparse_tools import ArgumentParser, get_commandline_args++from moviepy.editor import *+from moviepy.video.tools.subtitles import SubtitlesClip+++class VideoClipper():+ def __init__(self, asr_pipeline):+ logging.warning("Initializing VideoClipper.")+ self.asr_pipeline = asr_pipeline++ def recog(self, audio_input, state=None):+ if state is None:+ state = {}+ state['audio_input'] = audio_input+ _, data = audio_input+ data = data.astype(np.float64)+ rec_result = self.asr_pipeline(audio_in=data)+ state['recog_res_raw'] = rec_result['text_postprocessed']+ state['timestamp'] = rec_result['time_stamp']+ state['sentences'] = rec_result['sentences']+ res_text = rec_result['text']+ res_srt = generate_srt(rec_result['sentences'])+ return res_text, res_srt, state++ def clip(self, dest_text, start_ost, end_ost, state):+ # get from state+ audio_input = state['audio_input']+ recog_res_raw = state['recog_res_raw']+ timestamp = state['timestamp']+ sentences = state['sentences']+ sr, data = audio_input+ data = data.astype(np.float64)++ all_ts = []+ for _dest_text in dest_text.split('#'):+ _dest_text = pre_proc(_dest_text)+ ts = proc(recog_res_raw, timestamp, _dest_text)+ for _ts in ts: all_ts.append(_ts)+ ts = all_ts+ srt_index = 0+ clip_srt = ""+ if len(ts):+ start, end = ts[0]+ start = min(max(0, start+start_ost*16), len(data))+ end = min(max(0, end+end_ost*16), len(data))+ res_audio = data[start:end]+ start_end_info = "from {} to {}".format(start/16000, end/16000)+ srt_clip, _, srt_index = generate_srt_clip(sentences, start/16000.0, end/16000.0, begin_index=srt_index)+ clip_srt += srt_clip+ for _ts in ts[1:]: # multiple sentence input or multiple output matched+ start, end = _ts+ start = min(max(0, start+start_ost*16), len(data))+ end = min(max(0, end+end_ost*16), len(data))+ start_end_info += ", from {} to {}".format(start, end)+ res_audio = np.concatenate([res_audio, data[start+start_ost*16:end+end_ost*16]], -1)+ srt_clip, _, srt_index = generate_srt_clip(sentences, start/16000.0, end/16000.0, begin_index=srt_index-1)+ clip_srt += srt_clip+ if len(ts):+ message = "{} periods found in the speech: ".format(len(ts)) + start_end_info+ else:+ message = "No period found in the speech, return raw speech. You may check the recognition result and try other destination text."+ return (sr, res_audio), message, clip_srt++ def video_recog(self, vedio_filename):+ vedio_filename = vedio_filename+ clip_video_file = vedio_filename[:-4] + '_clip.mp4'+ video = mpy.VideoFileClip(vedio_filename)+ audio_file = vedio_filename[:-3] + 'wav'+ video.audio.write_audiofile(audio_file)+ wav = librosa.load(audio_file, sr=16000)[0]+ state = {+ 'vedio_filename': vedio_filename,+ 'clip_video_file': clip_video_file,+ 'video': video,+ }+ # res_text, res_srt = self.recog((16000, wav), state)+ return self.recog((16000, wav), state)++ def video_clip(self, dest_text, start_ost, end_ost, state, font_size=32, font_color='white', add_sub=False):+ # get from state+ recog_res_raw = state['recog_res_raw']+ timestamp = state['timestamp']+ sentences = state['sentences']+ video = state['video']+ clip_video_file = state['clip_video_file']+ vedio_filename = state['vedio_filename']++ all_ts = []+ srt_index = 0+ for _dest_text in dest_text.split('#'):+ _dest_text = pre_proc(_dest_text)+ ts = proc(recog_res_raw, timestamp, _dest_text)+ for _ts in ts: all_ts.append(_ts)+ ts = all_ts+ clip_srt = ""+ if len(ts):+ start, end = ts[0][0] / 16000, ts[0][1] / 16000+ start, end = start+start_ost/1000.0, end+end_ost/1000.0+ video_clip = video.subclip(start, end)+ start_end_info = "from {} to {}".format(start, end)+ srt_clip, subs, srt_index = generate_srt_clip(sentences, start, end, begin_index=srt_index)+ clip_srt += srt_clip+ if add_sub:+ generator = lambda txt: TextClip(txt, font='./font/STHeitiMedium.ttc', fontsize=font_size, color=font_color)+ subtitles = SubtitlesClip(subs, generator)+ video_clip = CompositeVideoClip([video_clip, subtitles.set_pos(('center','bottom'))])+ concate_clip = [video_clip]+ for _ts in ts[1:]:+ start, end = _ts[0] / 16000, _ts[1] / 16000+ start, end = start+start_ost/1000.0, end+end_ost/1000.0+ _video_clip = video.subclip(start, end)+ start_end_info += ", from {} to {}".format(start, end)+ srt_clip, subs, srt_index = generate_srt_clip(sentences, start, end, begin_index=srt_index-1)+ clip_srt += srt_clip+ if add_sub:+ generator = lambda txt: TextClip(txt, font='./font/STHeitiMedium.ttc', fontsize=font_size, color=font_color)+ subtitles = SubtitlesClip(subs, generator)+ _video_clip = CompositeVideoClip([_video_clip, subtitles.set_pos(('center','bottom'))])+ concate_clip.append(copy.copy(_video_clip))+ message = "{} periods found in the audio: ".format(len(ts)) + start_end_info+ logging.warning("Concating...")+ if len(concate_clip) > 1:+ video_clip = concatenate_videoclips(concate_clip)+ video_clip.write_videofile(clip_video_file, audio_codec="aac")+ else:+ clip_video_file = vedio_filename+ message = "No period found in the audio, return raw speech. You may check the recognition result and try other destination text."+ srt_clip = ''+ return clip_video_file, message, clip_srt+++def get_parser():+ parser = ArgumentParser(+ description="ClipVideo Argument",+ formatter_class=argparse.ArgumentDefaultsHelpFormatter,+ )+ parser.add_argument(+ "--stage",+ type=int,+ choices=(1, 2),+ help="Stage, 0 for recognizing and 1 for clipping",+ required=True+ )+ parser.add_argument(+ "--file",+ type=str,+ default=None,+ help="Input file path",+ required=True+ )+ parser.add_argument(+ "--output_dir",+ type=str,+ default='./output',+ help="Output files path",+ )+ parser.add_argument(+ "--dest_text",+ type=str,+ default=None,+ help="Destination text string for clipping",+ )+ parser.add_argument(+ "--start_ost",+ type=int,+ default=0,+ help="Offset time in ms at beginning for clipping"+ )+ parser.add_argument(+ "--end_ost",+ type=int,+ default=0,+ help="Offset time in ms at ending for clipping"+ )+ parser.add_argument(+ "--output_file",+ type=str,+ default=None,+ help="Output file path"+ )+ return parser+++def runner(stage, file, output_dir, dest_text, start_ost, end_ost, output_file, config=None):+ audio_suffixs = ['wav']+ video_suffixs = ['mp4']+ if file[-3:] in audio_suffixs:+ mode = 'audio'+ elif file[-3:] in video_suffixs:+ mode = 'video'+ else:+ logging.error("Unsupported file format: {}".format(file))+ while output_dir.endswith('/'):+ output_dir = output_dir[:-1]+ if stage == 1:+ from modelscope.pipelines import pipeline+ from modelscope.utils.constant import Tasks+ # initialize modelscope asr pipeline+ logging.warning("Initializing modelscope asr pipeline.")+ inference_pipeline = pipeline(+ task=Tasks.auto_speech_recognition,+ model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',+ vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',+ punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',+ output_dir=output_dir,+ )+ audio_clipper = VideoClipper(inference_pipeline)+ if mode == 'audio':+ logging.warning("Recognizing audio file: {}".format(file))+ wav, sr = librosa.load(file, sr=16000)+ res_text, res_srt, state = audio_clipper.recog((sr, wav))+ if mode == 'video':+ logging.warning("Recognizing video file: {}".format(file))+ res_text, res_srt, state = audio_clipper.video_recog(file)+ total_srt_file = output_dir + '/total.srt'+ with open(total_srt_file, 'w') as fout:+ fout.write(res_srt)+ logging.warning("Write total subtitile to {}".format(total_srt_file))+ write_state(output_dir, state)+ logging.warning("Recognition successed. You can copy the text segment from below and use stage 2.")+ print(res_text)+ if stage == 2:+ audio_clipper = VideoClipper(None)+ if mode == 'audio':+ state = load_state(output_dir)+ wav, sr = librosa.load(file, sr=16000)+ state['audio_input'] = (sr, wav)+ (sr, audio), message, srt_clip = audio_clipper.clip(dest_text, start_ost, end_ost, state)+ if output_file is None:+ output_file = output_dir + '/result.wav'+ clip_srt_file = output_file[:-3] + 'srt'+ logging.warning(message)+ sf.write(output_file, audio, 16000)+ assert output_file.endswith('.wav'), "output_file must ends with '.wav'"+ logging.warning("Save clipped wav file to {}".format(output_file))+ with open(clip_srt_file, 'w') as fout:+ fout.write(srt_clip)+ logging.warning("Write clipped subtitile to {}".format(clip_srt_file))+ if mode == 'video':+ state = load_state(output_dir)+ state['vedio_filename'] = file+ if output_file is None:+ state['clip_video_file'] = file[:-4] + '_clip.mp4'+ else:+ state['clip_video_file'] = output_file+ clip_srt_file = state['clip_video_file'][:-3] + 'srt'+ state['video'] = mpy.VideoFileClip(file)+ clip_video_file, message, srt_clip = audio_clipper.video_clip(dest_text, start_ost, end_ost, state)+ logging.warning("Clipping Log: {}".format(message))+ logging.warning("Save clipped mp4 file to {}".format(clip_video_file))+ with open(clip_srt_file, 'w') as fout:+ fout.write(srt_clip)+ logging.warning("Write clipped subtitile to {}".format(clip_srt_file))+++def main(cmd=None):+ print(get_commandline_args(), file=sys.stderr)+ parser = get_parser()+ args = parser.parse_args(cmd)+ kwargs = vars(args)+ runner(**kwargs)+++if __name__ == '__main__':+ main()\ No newline at end of filediff --git a/ClipVideo/docs/images/show.png b/ClipVideo/docs/images/show.pngnew file mode 100644index 0000000..5f561adBinary files /dev/null and b/ClipVideo/docs/images/show.png differdiff --git a/ClipVideo/requirments.txt b/ClipVideo/requirments.txtnew file mode 100644index 0000000..106dedc--- /dev/null+++ b/ClipVideo/requirments.txt@@ -0,0 +1,6 @@+librosa+soundfile+funasr>=0.5.5+moviepy+numpy+gradio\ No newline at end of filediff --git a/README.md b/README.mdnew file mode 100644index 0000000..afeb058--- /dev/null+++ b/README.md@@ -0,0 +1,79 @@+# FunASR-APP++<strong>FunASR-APP</strong> is a comprehensive speech application toolkit designed to facilitate the application and integration of [FunASR](https://github.com/alibaba-damo-academy/FunASR)'s open-source speech models. Its primary goal is to package the models into convenient application packages, enabling easy application and seamless integration.++## ClipVideo++As the first application toolkit of FunASR-APP, <strong>ClipVideo</strong> enables users to clip ```.mp4``` video files or ```.wav``` audio files with chosen text segments out of the recognition results generated by [Paraformer-long model](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary).++Under the help of ClipVideo you can get the video clips easily with the following steps (in Gradio service):+- Step1: Upload your video file (or try the example videos below)+- Step2: Copy the text segments you need to 'Text to Clip'+- Step3: Adjust subtitle settings (if needed)+- Step4: Click 'Clip' or 'Clip and Generate Subtitles'++### Usage+```shell+git clone https://github.com/alibaba-damo-academy/FunASR-APP.git+cd FunASR-APP+# install modelscope+pip install "modelscope[audio_asr]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html+# python environments+pip install -r ClipVideo/requirments.txt+```+(Optional) If you want to clip video file with embedded subtitles++1. ffmpeg and imagemagick is required++- On Ubuntu+```shell+apt-get -y update && apt-get -y install ffmpeg imagemagick+sed -i 's/none/read,write/g' /etc/ImageMagick-6/policy.xml+```+- On MacOS+```shell+brew install imagemagick+sed -i 's/none/read,write/g' /usr/local/Cellar/imagemagick/7.1.1-8_1/etc/ImageMagick-7/policy.xml+```+2. Download font file to ClipVideo/font++```shell+wget https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ClipVideo/STHeitiMedium.ttc -O ClipVideo/font/STHeitiMedium.ttc+```++#### Experience ClipVideo in Modelscope+You can try ClipVideo in modelscope space: [link](https://modelscope.cn/studios/damo/funasr_app_clipvideo/summary).++#### Use ClipVideo as Gradio Service+You can establish your own ClipVideo service which is same as [Modelscope Space](https://modelscope.cn/studios/damo/funasr_app_clipvideo/summary) as follow:+```shell+python clipvideo/gradio_service.py+```+then visit ```localhost:7860``` you will get a Gradio service like below and you can use ClipVideo following the steps:+<img src="ClipVideo/docs/images/show.png"/>++#### Use ClipVideo in command line+ClipVideo supports you to recognize and clip with commands:+```shell+# working in ClipVideo/+# step1: Recognize+python clipvideo/videoclipper.py --stage 1 \+ --file examples/2022云栖大会_片段.mp4 \+ --output_dir ./output+# now you can find recognition results and entire SRT file in ./output/+# step2: Clip+python clipvideo/videoclipper.py --stage 2 \+ --file examples/2022云栖大会_片段.mp4 \+ --output_dir ./output \+ --dest_text '我们把它跟乡村振兴去结合起来,利用我们的设计的能力' \+ --start_ost 0 \+ --end_ost 100 \+ --output_file './output/res.mp4'+```++### Study Speech Related Models in FunASR++[FunASR](https://github.com/alibaba-damo-academy/FunASR) hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model released on ModelScope, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun!++📚FunASR Paper: <a href="https://arxiv.org/abs/2305.11013"><img src="https://img.shields.io/badge/Arxiv-2305.11013-orange"></a>+🌟Support FunASR: <a href='https://github.com/alibaba-damo-academy/FunASR/stargazers'><img src='https://img.shields.io/github/stars/alibaba-damo-academy/FunASR.svg?style=social'></a>\ No newline at end of filediff --git a/TransAudio/README.md b/TransAudio/README.mdnew file mode 100644index 0000000..e69de29