利用 AI 大模型和自动化工作流,根据主题或关键词一键生成高清短视频。Generate HD short videos from a topic or keyword with an automated AI workflow.
Python ai-video-generatorcontent-creationffmpeginstagram-reelsllmpythonshort-videosubtitlestext-to-speechtiktokvideo-automationvideo-workflowworkflow-automationyoutube-shorts
feat(material): add optional TwelveLabs video-AI integration
mohit28ecfab021ce
diff --git a/README-en.md b/README-en.mdindex b2bf1172..398a99f8 100644--- a/README-en.md+++ b/README-en.md@@ -104,6 +104,7 @@ materials, video subtitles, and video background music before synthesizing a hig- [x] Supports **background music**, either random or specified music files, with adjustable `background music volume`- [x] Video material sources are **high-definition** and **royalty-free**, and you can also use your own **local materials**- [x] Supports multiple stock video providers: **Pexels**, **Pixabay**, and **Coverr**+- [x] Optional **TwelveLabs** video AI integration: use **Marengo** embeddings to semantically rank B-roll search terms against your subject, and **Pegasus** to QA/describe clips ([free API key](https://twelvelabs.io))- [x] Supports integration with various models such as **OpenAI**, **AIHubMix**, **AIML API**, **EvoLink**, **Moonshot**, **Azure**, **gpt4free**, **one-api**, **Qwen**, **Google Gemini**, **Ollama**, **DeepSeek**, **MiniMax**, **ERNIE**, **Pollinations**, **ModelScope** and more## Video Demos 📺diff --git a/README.md b/README.mdindex 5dad5677..3afaa98d 100644--- a/README.md+++ b/README.md@@ -101,6 +101,7 @@- [x] 支持 **背景音乐**,随机或者指定音乐文件,可设置`背景音乐音量`- [x] 视频素材来源 **高清**,而且 **无版权**,也可以使用自己的 **本地素材**- [x] 支持多种素材源:**Pexels**、**Pixabay**、**Coverr**+- [x] 可选接入 **TwelveLabs** 视频 AI:用 **Marengo** 多模态向量按主题语义重排素材关键词,用 **Pegasus** 对素材做内容 QA/描述([免费 API key](https://twelvelabs.io))- [x] 支持 **OpenAI**、**AIHubMix**、**AIML API**、**EvoLink**、**Moonshot**、**Azure**、**gpt4free**、**one-api**、**通义千问**、**Google Gemini**、**Ollama**、**DeepSeek**、**MiniMax**、 **文心一言**, **Pollinations**、**ModelScope** 等多种模型接入- [x] 支持一键 **跨平台发布**,生成完成后自动上传至 **TikTok**、**Instagram** 和 **YouTube Shorts**(需 [Upload-Post](https://upload-post.com) 账号;YouTube 发布时自动标注 AI 生成内容);在 `config.toml` 中配置 `upload_post_platforms`、`upload_post_youtube_privacy_status` 等参数diff --git a/app/services/task.py b/app/services/task.pyindex 7f192e9c..9e583467 100644--- a/app/services/task.py+++ b/app/services/task.py@@ -8,7 +8,7 @@ from loguru import loggerfrom app.config import configfrom app.models import constfrom app.models.schema import VideoConcatMode, VideoParams-from app.services import llm, material, subtitle, video, voice, upload_post+from app.services import llm, material, subtitle, twelvelabs, video, voice, upload_postfrom app.services import state as smfrom app.utils import file_security, utils@@ -63,6 +63,14 @@ def generate_terms(task_id, params, video_script):logger.error("failed to generate video terms.")return None+ # 可选的 TwelveLabs Marengo 语义重排:未启用时返回原顺序,无任何副作用。+ # 顺序匹配模式下关键词顺序本身就是脚本叙事顺序,必须保持原样,故跳过。+ if not params.match_materials_to_script:+ video_terms = twelvelabs.rerank_terms_by_subject(+ video_subject=params.video_subject,+ search_terms=video_terms,+ )+return video_termsdiff --git a/app/services/twelvelabs.py b/app/services/twelvelabs.pynew file mode 100644index 00000000..4554ac88--- /dev/null+++ b/app/services/twelvelabs.py@@ -0,0 +1,166 @@+"""+TwelveLabs (https://twelvelabs.io) integration — optional, opt-in helpers.++This module wraps two TwelveLabs models so MoneyPrinterTurbo can make better+use of the stock/B-roll footage it downloads:++ * Marengo (multimodal embeddings, 512-dim) — used to *semantically reorder*+ the LLM-generated search terms against the video subject, so that when the+ timeline budget runs out the most on-topic footage is the footage that made+ it in (instead of whatever the LLM happened to list first).++ * Pegasus (video understanding) — used to QA / describe a generated clip from+ a public URL, e.g. to sanity-check that a downloaded clip actually matches+ the script before it ships.++The integration is fully opt-in and non-breaking:+ * If `twelvelabs_api_keys` is not configured, every public function here is a+ no-op that returns its input unchanged (or None), so default behavior is+ identical to a build without TwelveLabs.+ * The `twelvelabs` SDK is imported lazily, so the dependency is only required+ when the feature is actually used.++Config (config.toml, [app] section):+ twelvelabs_api_keys = ["tlk_xxx"] # required to enable+ twelvelabs_rerank_terms = true # opt-in: reorder search terms by relevance+ twelvelabs_marengo_model = "marengo3.0" # optional override+ twelvelabs_pegasus_model = "pegasus1.5" # optional override++Get a free API key at https://twelvelabs.io (generous free tier).+"""++import math+from functools import lru_cache+from typing import List, Optional++from loguru import logger++from app.config import config+from app.services import material++DEFAULT_MARENGO_MODEL = "marengo3.0"+DEFAULT_PEGASUS_MODEL = "pegasus1.5"+# Pegasus requires max_tokens in [512, 98304]; 512 is plenty for a one-line QA.+_PEGASUS_MIN_MAX_TOKENS = 512+++def is_enabled() -> bool:+ """True only when at least one TwelveLabs API key is configured."""+ keys = config.app.get("twelvelabs_api_keys")+ return bool(keys)+++def _client():+ # Lazy import + rotated key reuse mirrors the other providers in+ # material.py (get_api_key rotates across configured keys).+ from twelvelabs import TwelveLabs++ api_key = material.get_api_key("twelvelabs_api_keys")+ return TwelveLabs(api_key=api_key)+++def _cosine(a: List[float], b: List[float]) -> float:+ dot = sum(x * y for x, y in zip(a, b))+ na = math.sqrt(sum(x * x for x in a))+ nb = math.sqrt(sum(x * x for x in b))+ if na == 0 or nb == 0:+ return 0.0+ return dot / (na * nb)+++def embed_text(text: str, model: Optional[str] = None) -> Optional[List[float]]:+ """+ Return a 512-dim Marengo text embedding, or None on failure / when disabled.++ Cached so repeated terms across a session don't re-hit the API.+ """+ if not is_enabled() or not text or not text.strip():+ return None+ model = model or config.app.get("twelvelabs_marengo_model", DEFAULT_MARENGO_MODEL)+ try:+ # lru_cache only memoizes successful returns; a raised exception is not+ # cached, so a transient API error never poisons the cache.+ return _embed_text_cached(text.strip(), model)+ except Exception as e: # noqa: BLE001 - never break the pipeline on TL errors+ logger.warning(f"TwelveLabs embed_text failed, skipping rerank: {e}")+ return None+++@lru_cache(maxsize=512)+def _embed_text_cached(text: str, model: str) -> List[float]:+ client = _client()+ resp = client.embed.create(model_name=model, text=text)+ # SDK aliases the raw JSON 'float' vector key to `float_`.+ return list(resp.text_embedding.segments[0].float_)+++def rerank_terms_by_subject(+ video_subject: str,+ search_terms: List[str],+ model: Optional[str] = None,+) -> List[str]:+ """+ Reorder `search_terms` so the terms most semantically relevant to+ `video_subject` come first (Marengo cosine similarity).++ Opt-in: only runs when TwelveLabs is enabled AND+ `twelvelabs_rerank_terms` is truthy. Falls back to the original order on+ any failure, so it can never make the pipeline worse.+ """+ if not is_enabled() or not config.app.get("twelvelabs_rerank_terms"):+ return search_terms+ if not video_subject or len(search_terms) < 2:+ return search_terms++ subject_vec = embed_text(video_subject, model)+ if subject_vec is None:+ return search_terms++ scored = []+ for term in search_terms:+ vec = embed_text(term, model)+ if vec is None:+ # If any term can't be embedded, don't risk a partial reorder.+ return search_terms+ scored.append((term, _cosine(subject_vec, vec)))++ ranked = [term for term, _ in sorted(scored, key=lambda x: x[1], reverse=True)]+ logger.info(+ f"TwelveLabs Marengo reranked {len(ranked)} search terms by relevance "+ f"to subject '{video_subject}': {ranked}"+ )+ return ranked+++def analyze_clip(+ video_url: str,+ prompt: str = "Describe what happens in this video in one sentence.",+ model: Optional[str] = None,+ max_tokens: int = _PEGASUS_MIN_MAX_TOKENS,+) -> Optional[str]:+ """+ QA / describe a clip from a public URL with Pegasus, returning the model's+ text answer (or None when disabled / on failure).++ Notes (TwelveLabs API constraints):+ * Pegasus needs a publicly reachable URL (or an uploaded asset), not a+ bare local path; the analyzed window must be >= 4s.+ * max_tokens must be >= 512 for this model.+ """+ if not is_enabled() or not video_url:+ return None+ model = model or config.app.get("twelvelabs_pegasus_model", DEFAULT_PEGASUS_MODEL)+ try:+ from twelvelabs.types import VideoContext_Url++ client = _client()+ resp = client.analyze(+ model_name=model,+ video=VideoContext_Url(url=video_url),+ prompt=prompt,+ max_tokens=max(max_tokens, _PEGASUS_MIN_MAX_TOKENS),+ )+ return resp.data+ except Exception as e: # noqa: BLE001+ logger.warning(f"TwelveLabs analyze_clip failed: {e}")+ return Nonediff --git a/config.example.toml b/config.example.tomlindex 6a268cb7..703eaa5d 100644--- a/config.example.toml+++ b/config.example.toml@@ -48,6 +48,31 @@ pixabay_api_keys = []# 特别注意格式,Key 用英文双引号括起来,多个Key用逗号隔开coverr_api_keys = []+# TwelveLabs API Key (optional) — https://twelvelabs.io+# TwelveLabs (可选) 视频理解 / 多模态嵌入集成。+# Enables two opt-in features:+# * Marengo (multimodal embeddings) semantically reorders the auto-generated+# search terms by relevance to your video subject, so the most on-topic+# B-roll is downloaded first when the timeline budget is tight.+# * Pegasus (video understanding) can QA / describe a clip from its URL.+# Leave empty to keep MoneyPrinterTurbo's behavior completely unchanged.+# Get a free key (generous free tier) at https://twelvelabs.io and install the+# optional dependency with `uv sync --extra twelvelabs`.+# Register at https://playground.twelvelabs.io to get your API key.+# You can use multiple keys to avoid rate limits.+# For example: twelvelabs_api_keys = ["tlk_123adsf4567","tlk_abd1321cd13ef"]+# 请使用英文半角双引号包裹每个 Key;从富文本复制来的弯引号会导致 TOML 解析失败。+twelvelabs_api_keys = []++# Reorder search terms by Marengo relevance before downloading footage.+# Requires twelvelabs_api_keys above. Ignored in "match materials to script"+# mode, where term order is intentionally the script's narrative order.+twelvelabs_rerank_terms = false++# Optional model overrides (defaults shown).+# twelvelabs_marengo_model = "marengo3.0"+# twelvelabs_pegasus_model = "pegasus1.5"+# 支持的提供商 (Supported providers):# openai# aihubmix (OpenAI-compatible AI model gateway)diff --git a/pyproject.toml b/pyproject.tomlindex d875ce3c..b2e96107 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -37,6 +37,12 @@ dependencies = [g4f = ["g4f==0.5.2.2",]+# Optional TwelveLabs (https://twelvelabs.io) video-understanding/embedding+# integration. Only needed when `twelvelabs_api_keys` is configured; install+# with `uv sync --extra twelvelabs`.+twelvelabs = [+ "twelvelabs>=1.2.8",+][tool.hatch.build.targets.wheel]packages = ["app"]diff --git a/test/services/test_twelvelabs.py b/test/services/test_twelvelabs.pynew file mode 100644index 00000000..ce090e19--- /dev/null+++ b/test/services/test_twelvelabs.py@@ -0,0 +1,169 @@+import os+import sys+import unittest+from pathlib import Path+from unittest.mock import MagicMock, patch++sys.path.insert(0, str(Path(__file__).parent.parent.parent))++from app.config import config+from app.services import twelvelabs++RUN_INTEGRATION_TESTS = os.environ.get("MPT_RUN_INTEGRATION_TESTS", "").lower() in {+ "1",+ "true",+ "yes",+}+++class TestTwelveLabsService(unittest.TestCase):+ """+ TwelveLabs 集成是完全 opt-in 的:未配置 twelvelabs_api_keys 时所有函数+ 都必须是无副作用的 no-op,行为与不接入 TwelveLabs 完全一致。+ 这些用例全部用 mock 替换 SDK 客户端,CI 不依赖真实网络或真实 API key。+ """++ def setUp(self):+ self.original_app_config = dict(config.app)+ twelvelabs._embed_text_cached.cache_clear()++ def tearDown(self):+ config.app.clear()+ config.app.update(self.original_app_config)+ twelvelabs._embed_text_cached.cache_clear()++ # ---------------- disabled / no-op behavior ----------------++ def test_disabled_when_no_api_key(self):+ config.app.pop("twelvelabs_api_keys", None)+ self.assertFalse(twelvelabs.is_enabled())+ # rerank must return the input list unchanged+ terms = ["b", "a", "c"]+ self.assertEqual(+ twelvelabs.rerank_terms_by_subject("subject", terms), terms+ )+ # analyze must be a no-op returning None+ self.assertIsNone(twelvelabs.analyze_clip("https://x/y.mp4"))++ def test_rerank_skipped_when_flag_off(self):+ config.app["twelvelabs_api_keys"] = ["tlk_test"]+ config.app["twelvelabs_rerank_terms"] = False+ terms = ["b", "a"]+ # Even enabled, with the flag off we must not touch order or call the API.+ with patch.object(twelvelabs, "_client") as client:+ result = twelvelabs.rerank_terms_by_subject("subject", terms)+ self.assertEqual(result, terms)+ client.assert_not_called()++ # ---------------- enabled rerank behavior ----------------++ def _client_returning(self, vectors_by_text):+ """Build a fake TwelveLabs client whose embed.create returns canned vectors."""++ def fake_create(*, model_name, text):+ seg = MagicMock()+ seg.float_ = vectors_by_text[text]+ resp = MagicMock()+ resp.text_embedding.segments = [seg]+ return resp++ client = MagicMock()+ client.embed.create.side_effect = fake_create+ return client++ def test_rerank_orders_by_cosine_to_subject(self):+ config.app["twelvelabs_api_keys"] = ["tlk_test"]+ config.app["twelvelabs_rerank_terms"] = True++ # subject aligned with "city"; "kitten" is orthogonal.+ vectors = {+ "city skyline": [1.0, 0.0, 0.0],+ "downtown buildings": [0.9, 0.1, 0.0], # close to subject+ "cute kitten": [0.0, 1.0, 0.0], # far from subject+ }+ client = self._client_returning(vectors)++ with patch.object(twelvelabs, "_client", return_value=client):+ result = twelvelabs.rerank_terms_by_subject(+ "city skyline", ["cute kitten", "downtown buildings"]+ )++ # most relevant term must come first+ self.assertEqual(result, ["downtown buildings", "cute kitten"])++ def test_rerank_falls_back_on_embed_failure(self):+ config.app["twelvelabs_api_keys"] = ["tlk_test"]+ config.app["twelvelabs_rerank_terms"] = True++ client = MagicMock()+ client.embed.create.side_effect = RuntimeError("api down")++ terms = ["alpha", "beta"]+ with patch.object(twelvelabs, "_client", return_value=client):+ result = twelvelabs.rerank_terms_by_subject("subject", terms)++ # any failure must preserve the original order (never make things worse)+ self.assertEqual(result, terms)++ def test_rerank_noop_for_single_term(self):+ config.app["twelvelabs_api_keys"] = ["tlk_test"]+ config.app["twelvelabs_rerank_terms"] = True+ with patch.object(twelvelabs, "_client") as client:+ result = twelvelabs.rerank_terms_by_subject("subject", ["only"])+ self.assertEqual(result, ["only"])+ client.assert_not_called()++ # ---------------- analyze_clip ----------------++ def test_analyze_clip_returns_model_text(self):+ config.app["twelvelabs_api_keys"] = ["tlk_test"]++ resp = MagicMock()+ resp.data = "A city skyline at dusk."+ client = MagicMock()+ client.analyze.return_value = resp++ with patch.object(twelvelabs, "_client", return_value=client):+ out = twelvelabs.analyze_clip(+ "https://example.com/clip.mp4", prompt="describe"+ )++ self.assertEqual(out, "A city skyline at dusk.")+ # max_tokens must be clamped to the Pegasus minimum (>=512)+ self.assertGreaterEqual(client.analyze.call_args.kwargs["max_tokens"], 512)+++ RUN_INTEGRATION_TESTS and os.getenv("TWELVELABS_API_KEY"),+ "live test: set MPT_RUN_INTEGRATION_TESTS=1 and TWELVELABS_API_KEY to run "+ "against the real TwelveLabs API",+)+class TestTwelveLabsLive(unittest.TestCase):+ """Live contract check — only runs with MPT_RUN_INTEGRATION_TESTS=1 + a key."""++ def setUp(self):+ self.original_app_config = dict(config.app)+ config.app["twelvelabs_api_keys"] = [os.environ["TWELVELABS_API_KEY"]]+ config.app["twelvelabs_rerank_terms"] = True+ twelvelabs._embed_text_cached.cache_clear()++ def tearDown(self):+ config.app.clear()+ config.app.update(self.original_app_config)+ twelvelabs._embed_text_cached.cache_clear()++ def test_marengo_embedding_is_512_dim(self):+ vec = twelvelabs.embed_text("a city skyline at night")+ self.assertIsNotNone(vec)+ self.assertEqual(len(vec), 512)++ def test_rerank_puts_relevant_term_first(self):+ result = twelvelabs.rerank_terms_by_subject(+ "city skyline at night",+ ["cute kitten playing with yarn", "downtown buildings and traffic at dusk"],+ )+ self.assertEqual(result[0], "downtown buildings and traffic at dusk")+++if __name__ == "__main__":+ unittest.main()diff --git a/uv.lock b/uv.lockindex a5d50c7c..7a9f1745 100644--- a/uv.lock+++ b/uv.lock@@ -1127,6 +1127,9 @@ dependencies = [g4f = [{ name = "g4f" },]+twelvelabs = [+ { name = "twelvelabs" },+][package.metadata]requires-dist = [@@ -1148,9 +1151,10 @@ requires-dist = [{ name = "requests", specifier = "==2.33.1" },{ name = "socksio", specifier = "==1.0.0" },{ name = "streamlit", specifier = "==1.58.0" },+ { name = "twelvelabs", marker = "extra == 'twelvelabs'", specifier = ">=1.2.8" },{ name = "uvicorn", specifier = "==0.32.1" },]-provides-extras = ["g4f"]+provides-extras = ["g4f", "twelvelabs"][[package]]name = "moviepy"@@ -2069,6 +2073,21 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf", size = 78374 },]+[[package]]+name = "twelvelabs"+version = "1.2.8"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "httpx" },+ { name = "pydantic" },+ { name = "pydantic-core" },+ { name = "typing-extensions" },+]+sdist = { url = "https://files.pythonhosted.org/packages/5f/24/2604563f5a528bf5f12f9bbb7e3102e91c0e4f9bfd7aeb9963dced3a0b0f/twelvelabs-1.2.8.tar.gz", hash = "sha256:bb0846cc839845c800de8061bb7b99212a472e24bf20770d569f6f620b845e3c", size = 178449 }+wheels = [+ { url = "https://files.pythonhosted.org/packages/4e/a6/6380129c65222bb2497cbef27334a496a1a31f4f38973fac46b62b9224fc/twelvelabs-1.2.8-py3-none-any.whl", hash = "sha256:c0487dd892b03728ac2afe3e4ebd4ea5e30ff404b61896667eb5a39264db282b", size = 372073 },+]+[[package]]name = "typer"version = "0.26.5"