Skip to content

FunASR-powered video transcription, subtitle generation, and LLM-assisted clipping tool with a local Gradio UI.

llm update

维石6c12e2b7c3a0
diff --git a/README.md b/README.md
index bdd3cdf..250addb 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,7 @@
# <p align="center"> FunClip🎥</p>
**<p align="center"> ⚡ 开源、精准、方便的视频切片工具 ⚡ </p>**
+**<p align="center"> 🧠 通过FunClip探索基于大模型的视频剪辑 🧠 </p>**
<p align="center"> <img src="docs/images/interface.jpg" width=444/></p>
@@ -18,6 +19,7 @@
**FunClip**是一款完全开源、本地部署的自动化视频剪辑工具,通过调用阿里巴巴通义实验室开源的[FunASR](https://github.com/alibaba-damo-academy/FunASR) Paraformer系列模型进行视频的语音识别,随后用户可以自由选择识别结果中的文本片段或说话人,点击裁剪按钮即可获取对应片段的视频([快速体验](https://modelscope.cn/studios/iic/funasr_app_clipvideo/summary))。
在上述基本功能的基础上,FunClip有以下特色:
+- 🔥FunClip集成了多种大语言模型调用方式并提供了prompt配置接口,尝试通过大语言模型进行视频裁剪~
- FunClip集成了阿里巴巴开源的工业级模型[Paraformer-Large](https://modelscope.cn/models/iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary),是当前识别效果最优的开源中文ASR模型之一,Modelscope下载量1300w+次,并且能够一体化的准确预测时间戳。
- FunClip集成了[SeACo-Paraformer](https://modelscope.cn/models/iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary)的热词定制化功能,在ASR过程中可以指定一些实体词、人名等作为热词,提升识别效果。
- FunClip集成了[CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary)说话人识别模型,用户可以将自动识别出的说话人ID作为裁剪目标,将某一说话人的段落裁剪出来。
@@ -29,6 +31,12 @@
<a name="近期更新"></a>
## 近期更新🚀
+- 🔥2024/05/13 FunClip更新基于大语言模型的剪辑功能,使用方法如下:
+ 1. 在进行识别之后,选择大模型名称,配置你自己的apikey;
+ 2. 点击'LLM智能段落选择'按钮,FunClip将自动组合两个prompt与视频的srt字幕;
+ 3. 点击'LLM智能裁剪'按钮,基于前一步的大语言模型输出结果,FunClip将提取其中的时间戳进行裁剪;
+ 4. 您可以尝试改变prompt来借助大语言模型的能力来获取您想要的结果;
+- 🔥FunClip加入大语言模型智能裁剪功能,集成qwen系列,gpt系列等模型,提供默认prompt,您也可以探索并分享prompt的设置技巧。
- 2024/05/09 FunClip更新至v1.1.0,包含如下更新与修复:
- 支持配置输出文件目录,保存ASR中间结果与视频裁剪中间文件;
- UI升级(见下方演示图例),视频与音频裁剪功能在同一页,按钮位置调整;
@@ -55,7 +63,7 @@
git clone https://github.com/alibaba-damo-academy/FunClip.git
cd FunClip
# 安装相关Python依赖
-pip install -r ./requirments.txt
+pip install -r ./requirements.txt
```
### 安装imagemagick(可选)
diff --git a/README_en.md b/README_en.md
index 096b6b4..9ca41c2 100644
--- a/README_en.md
+++ b/README_en.md
@@ -20,6 +20,7 @@
On top of the basic features mentioned above, FunClip has following highlights:
+- Try AI clipping using LLM in FunClip now.
- FunClip integrates Alibaba's open-source industrial-grade model [Paraformer-Large](https://modelscope.cn/models/iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), which is one of the best-performing open-source Chinese ASR models available, with over 13 million downloads on Modelscope. It can also accurately predict timestamps in an integrated manner.
- FunClip incorporates the hotword customization feature of [SeACo-Paraformer](https://modelscope.cn/models/iic/speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), allowing users to specify certain entity words, names, etc., as hotwords during the ASR process to enhance recognition results.
- FunClip integrates the [CAM++](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) speaker recognition model, enabling users to use the auto-recognized speaker ID as the target for trimming, to clip segments from a specific speaker.
@@ -30,7 +31,12 @@ You're welcome to try it out, and we look forward to any requests and valuable s
<a name="What's New"></a>
## What's New🚀
-
+- 🔥2024/05/13 FunClip updates its clipping feature based on large language models, and the usage is as follows:
+ 1. After the recognition, select the name of the large model and configure your own apikey;
+ 2. Click on the 'LLM Smart Paragraph Selection' button, and FunClip will automatically combine two prompts with the video's srt subtitles;
+ 3. Click on the 'LLM Smart Clipping' button, and based on the output results of the large language model from the previous step, FunClip will extract the timestamps for clipping;
+ 4. You can try changing the prompt to leverage the capabilities of the large language models to get the results you want;
+- 🔥FunClip adds smart clipping functionality with large language models, integrating models from the qwen series, gpt series, etc., providing default prompts. You can also explore and share tips for setting prompts.
- 2024/05/09 FunClip updated to v1.1.0, including the following updates and fixes:
- Support configuration of output file directory, saving ASR intermediate results and video clipping intermediate files;
- UI upgrade (see guide picture below), video and audio cropping function on the same page, button position adjustment;
@@ -58,7 +64,7 @@ You're welcome to try it out, and we look forward to any requests and valuable s
git clone https://github.com/alibaba-damo-academy/FunClip.git
cd FunClip
# install Python requirments
-pip install -r ./requirments.txt
+pip install -r ./requirements.txt
```
### imagemagick install (Optional)
diff --git a/funclip/introduction.py b/funclip/introduction.py
index a069e64..ac731be 100644
--- a/funclip/introduction.py
+++ b/funclip/introduction.py
@@ -29,11 +29,12 @@ top_md_3 = ("""访问FunASR项目与论文能够帮助您深入了解ParaClipper
</div>
""")
-top_md_4 = ("""我们在「LLM智能裁剪」模块中提供两种LLM调用方式,
- 1. 选择openai开头的模型即为调用openai官方api,此时需要您自备sk与网络环境;
- 2. [gpt4free](https://github.com/xtekky/gpt4free?tab=readme-ov-file)项目也被集成进FunClip,可以通过它免费调用gpt模型;
+top_md_4 = ("""我们在「LLM智能裁剪」模块中提供三种LLM调用方式,
+ 1. 选择阿里云百炼平台通过api调用qwen系列模型,此时需要您准备百炼平台的apikey,请访问[阿里云百炼](https://bailian.console.aliyun.com/#/home);
+ 2. 选择GPT开头的模型即为调用openai官方api,此时需要您自备sk与网络环境;
+ 3. [gpt4free](https://github.com/xtekky/gpt4free?tab=readme-ov-file)项目也被集成进FunClip,可以通过它免费调用gpt模型;
- 其中方式1中可以通过在界面中传入sk,或通过`export OPENAI_API_KEY=YOUR_SK`的方式在linux或mac中配置sk
-
- 方式2而可能非常不稳定,返回时间可能很长或者结果获取失败,可以多多尝试或者自己准备sk使用方式1
+ 其中方式1与方式2需要在界面中传入相应的apikey
+
+ 方式3而可能非常不稳定,返回时间可能很长或者结果获取失败,可以多多尝试或者自己准备sk使用方式1
""")
diff --git a/funclip/launch.py b/funclip/launch.py
index 0a2512d..4eff9fe 100644
--- a/funclip/launch.py
+++ b/funclip/launch.py
@@ -11,6 +11,7 @@ from videoclipper import VideoClipper
from introduction import top_md_1, top_md_2, top_md_3, top_md_4
from llm.openai_api import openai_call
from llm.g4f_openai_api import g4f_openai_call
+from llm.qwen_api import call_qwen_model
from utils.trans_utils import extract_timestamps
@@ -88,14 +89,15 @@ if __name__ == "__main__":
add_sub=True, dest_spk=video_spk_input, output_dir=output_dir
)
- def llm_inference(prompt_head, srt_text, model):
- SUPPORT_LLM_PREFIX = ['openai', 'g4f']
- prefix = model.split('-')[0]
- model_wo_prefix = "-".join(model.split('-')[1:])
- if prefix == 'openai':
- return openai_call(prompt_head+'\n'+srt_text, model_wo_prefix)
- elif prefix.startswith('g4f'):
- return g4f_openai_call(prompt_head+'\n'+srt_text, model_wo_prefix)
+ def llm_inference(system_content, user_content, srt_text, model, apikey):
+ SUPPORT_LLM_PREFIX = ['qwen', 'gpt', 'g4f', 'moonshot']
+ if model.startswith('qwen'):
+ return call_qwen_model(apikey, model, system_content, user_content+'\n'+srt_text)
+ if model.startswith('gpt') or model.startswith('moonshot'):
+ return openai_call(apikey, model, system_content, user_content+'\n'+srt_text)
+ elif model.startswith('g4f'):
+ model = "-".join(model.split('-')[1:])
+ return g4f_openai_call(model, system_content, user_content+'\n'+srt_text)
else:
logging.error("LLM name error, only {} are supported as LLM name prefix."
.format(SUPPORT_LLM_PREFIX))
@@ -146,8 +148,8 @@ if __name__ == "__main__":
with gr.Column():
# with gr.Row():
# video_sd_switch = gr.Radio(["No", "Yes"], label="👥区分说话人 Get Speakers", value='No')
- hotwords_input = gr.Textbox(label="🚒 热词 | Hotwords(多个热词使用空格分隔,仅支持中文热词)")
- output_dir = gr.Textbox(label="📁 文件输出路径 | File Output Dir", value=" ")
+ hotwords_input = gr.Textbox(label="🚒 热词 | Hotwords(可以为空,多个热词使用空格分隔,仅支持中文热词)")
+ output_dir = gr.Textbox(label="📁 文件输出路径 | File Output Dir (可以为空,Linux, mac系统可以稳定使用)", value=" ")
with gr.Row():
recog_button = gr.Button("👂 识别 | ASR", variant="primary")
recog_button2 = gr.Button("👂👫 识别+区分说话人 | ASR+SD")
@@ -156,20 +158,23 @@ if __name__ == "__main__":
with gr.Column():
with gr.Tab("🧠 LLM智能裁剪 | LLM Clipping"):
with gr.Column():
- prompt_head = gr.Textbox(label="Prompt Head", value=("你是一个视频srt字幕剪辑工具,"
- "输入视频的srt字幕之后根据如下要求剪辑出最精彩的并且尽可能连续的片段,输出每个段落的开始与结束时间,"
- "输出需严格按照如下格式:1. [开始时间-结束时间] 文本,\n"
- "原始srt字幕如下:"))
- with gr.Row():
- llm_model = gr.Dropdown(
- choices=["openai-gpt-3.5-turbo",
- "openai-gpt-3.5-turbo-0125",
- "openai-gpt-4-turbo",
- "g4fopenai-gpt-3.5-turbo"],
- value="openai-gpt-3.5-turbo",
- label="LLM Model Name with Prefix",
- allow_custom_value=True)
- llm_button = gr.Button("LLM Clip", variant="primary")
+ prompt_head = gr.Textbox(label="Prompt System", value=("你是一个视频srt字幕分析剪辑器,输入视频的srt字幕,"
+ "分析其中的精彩且尽可能连续的片段并裁剪出来,将片段中在时间上连续的多个句子及它们的时间戳合并为一条,"
+ "注意确保文字与时间戳的正确匹配。输出需严格按照如下格式:1. [开始时间-结束时间] 文本"))
+ prompt_head2 = gr.Textbox(label="Prompt User", value=("这是待裁剪的视频srt字幕:"))
+ with gr.Column():
+ with gr.Row():
+ llm_model = gr.Dropdown(
+ choices=["qwen-plus",
+ "gpt-3.5-turbo",
+ "gpt-3.5-turbo-0125",
+ "gpt-4-turbo",
+ "g4f-gpt-3.5-turbo"],
+ value="qwen-plus",
+ label="LLM Model Name",
+ allow_custom_value=True)
+ apikey_input = gr.Textbox(label="APIKEY")
+ llm_button = gr.Button("LLM智能段落选择(首先进行识别,非g4f需配置对应apikey)", variant="primary")
llm_result = gr.Textbox(label="LLM Clipper Result")
with gr.Row():
llm_clip_button = gr.Button("🧠 LLM智能裁剪 | AI Clip", variant="primary")
@@ -228,7 +233,7 @@ if __name__ == "__main__":
],
outputs=[video_output, clip_message, srt_clipped])
llm_button.click(llm_inference,
- inputs=[prompt_head, video_srt_output, llm_model],
+ inputs=[prompt_head, prompt_head2, video_srt_output, llm_model, apikey_input],
outputs=[llm_result])
llm_clip_button.click(AI_clip,
inputs=[llm_result,
diff --git a/funclip/llm/g4f_openai_api.py b/funclip/llm/g4f_openai_api.py
index 478af87..ca7a099 100644
--- a/funclip/llm/g4f_openai_api.py
+++ b/funclip/llm/g4f_openai_api.py
@@ -10,10 +10,21 @@ if __name__ == '__main__':
print(response.choices[0].message.content)
-def g4f_openai_call(prompt, model="gpt-3.5-turbo"):
+def g4f_openai_call(model="gpt-3.5-turbo",
+ user_content="如何做西红柿炖牛腩?",
+ system_content=None):
client = Client()
+ if system_content is not None and len(system_content.strip()):
+ messages = [
+ {'role': 'system', 'content': system_content},
+ {'role': 'user', 'content': user_content}
+ ]
+ else:
+ messages = [
+ {'role': 'user', 'content': user_content}
+ ]
response = client.chat.completions.create(
model=model,
- messages=[{"role": "user", "content": prompt}],
+ messages=messages,
)
return(response.choices[0].message.content)
\ No newline at end of file
diff --git a/funclip/llm/openai_api.py b/funclip/llm/openai_api.py
index 5be31cb..3a9fd39 100644
--- a/funclip/llm/openai_api.py
+++ b/funclip/llm/openai_api.py
@@ -22,19 +22,26 @@ if __name__ == '__main__':
print(chat_completion.choices[0].message.content)
-def openai_call(prompt, model="gpt-3.5-turbo"):
+def openai_call(apikey,
+ model="gpt-3.5-turbo",
+ user_content="如何做西红柿炖牛腩?",
+ system_content=None):
client = OpenAI(
# This is the default and can be omitted
- api_key=os.environ.get("OPENAI_API_KEY"),
+ api_key=apikey,
)
-
+ if system_content is not None and len(system_content.strip()):
+ messages = [
+ {'role': 'system', 'content': system_content},
+ {'role': 'user', 'content': user_content}
+ ]
+ else:
+ messages = [
+ {'role': 'user', 'content': user_content}
+ ]
+
chat_completion = client.chat.completions.create(
- messages=[
- {
- "role": "user",
- "content": prompt,
- }
- ],
+ messages=messages,
model=model,
)
logging.info("Openai model inference done.")
diff --git a/funclip/llm/qwen_api.py b/funclip/llm/qwen_api.py
new file mode 100644
index 0000000..a1c806a
--- /dev/null
+++ b/funclip/llm/qwen_api.py
@@ -0,0 +1,30 @@
+import dashscope
+from dashscope import Generation
+
+
+def call_qwen_model(key=None,
+ model="qwen_plus",
+ user_content="如何做西红柿炖牛腩?",
+ system_content=None):
+ dashscope.api_key = key
+ if system_content is not None and len(system_content.strip()):
+ messages = [
+ {'role': 'system', 'content': system_content},
+ {'role': 'user', 'content': user_content}
+ ]
+ else:
+ messages = [
+ {'role': 'user', 'content': user_content}
+ ]
+ responses = Generation.call(model,
+ messages=messages,
+ result_format='message', # 设置输出为'message'格式
+ stream=False, # 设置输出方式为流式输出
+ incremental_output=False # 增量式流式输出
+ )
+ print(responses)
+ return responses['output']['choices'][0]['message']['content']
+
+
+if __name__ == '__main__':
+ call_qwen_model('YOUR_BAILIAN_APIKEY')
\ No newline at end of file
diff --git a/requirments.txt b/requirements.txt
similarity index 100%
rename from requirments.txt
rename to requirements.txt