Skip to content

利用 AI 大模型和自动化工作流,根据主题或关键词一键生成高清短视频。Generate HD short videos from a topic or keyword with an automated AI workflow.

feat(material): add optional TwelveLabs video-AI integration

mohit28ecfab021ce
diff --git a/README-en.md b/README-en.md
index b2bf1172..398a99f8 100644
--- a/README-en.md
+++ b/README-en.md
@@ -104,6 +104,7 @@ materials, video subtitles, and video background music before synthesizing a hig
- [x] Supports **background music**, either random or specified music files, with adjustable `background music volume`
- [x] Video material sources are **high-definition** and **royalty-free**, and you can also use your own **local materials**
- [x] Supports multiple stock video providers: **Pexels**, **Pixabay**, and **Coverr**
+- [x] Optional **TwelveLabs** video AI integration: use **Marengo** embeddings to semantically rank B-roll search terms against your subject, and **Pegasus** to QA/describe clips ([free API key](https://twelvelabs.io))
- [x] Supports integration with various models such as **OpenAI**, **AIHubMix**, **AIML API**, **EvoLink**, **Moonshot**, **Azure**, **gpt4free**, **one-api**, **Qwen**, **Google Gemini**, **Ollama**, **DeepSeek**, **MiniMax**, **ERNIE**, **Pollinations**, **ModelScope** and more
## Video Demos 📺
diff --git a/README.md b/README.md
index 5dad5677..3afaa98d 100644
--- a/README.md
+++ b/README.md
@@ -101,6 +101,7 @@
- [x] 支持 **背景音乐**,随机或者指定音乐文件,可设置`背景音乐音量`
- [x] 视频素材来源 **高清**,而且 **无版权**,也可以使用自己的 **本地素材**
- [x] 支持多种素材源:**Pexels**、**Pixabay**、**Coverr**
+- [x] 可选接入 **TwelveLabs** 视频 AI:用 **Marengo** 多模态向量按主题语义重排素材关键词,用 **Pegasus** 对素材做内容 QA/描述([免费 API key](https://twelvelabs.io))
- [x] 支持 **OpenAI**、**AIHubMix**、**AIML API**、**EvoLink**、**Moonshot**、**Azure**、**gpt4free**、**one-api**、**通义千问**、**Google Gemini**、**Ollama**、**DeepSeek**、**MiniMax**、 **文心一言**, **Pollinations**、**ModelScope** 等多种模型接入
- [x] 支持一键 **跨平台发布**,生成完成后自动上传至 **TikTok**、**Instagram** 和 **YouTube Shorts**(需 [Upload-Post](https://upload-post.com) 账号;YouTube 发布时自动标注 AI 生成内容);在 `config.toml` 中配置 `upload_post_platforms`、`upload_post_youtube_privacy_status` 等参数
diff --git a/app/services/task.py b/app/services/task.py
index 7f192e9c..9e583467 100644
--- a/app/services/task.py
+++ b/app/services/task.py
@@ -8,7 +8,7 @@ from loguru import logger
from app.config import config
from app.models import const
from app.models.schema import VideoConcatMode, VideoParams
-from app.services import llm, material, subtitle, video, voice, upload_post
+from app.services import llm, material, subtitle, twelvelabs, video, voice, upload_post
from app.services import state as sm
from app.utils import file_security, utils
@@ -63,6 +63,14 @@ def generate_terms(task_id, params, video_script):
logger.error("failed to generate video terms.")
return None
+ # 可选的 TwelveLabs Marengo 语义重排:未启用时返回原顺序,无任何副作用。
+ # 顺序匹配模式下关键词顺序本身就是脚本叙事顺序,必须保持原样,故跳过。
+ if not params.match_materials_to_script:
+ video_terms = twelvelabs.rerank_terms_by_subject(
+ video_subject=params.video_subject,
+ search_terms=video_terms,
+ )
+
return video_terms
diff --git a/app/services/twelvelabs.py b/app/services/twelvelabs.py
new file mode 100644
index 00000000..4554ac88
--- /dev/null
+++ b/app/services/twelvelabs.py
@@ -0,0 +1,166 @@
+"""
+TwelveLabs (https://twelvelabs.io) integration — optional, opt-in helpers.
+
+This module wraps two TwelveLabs models so MoneyPrinterTurbo can make better
+use of the stock/B-roll footage it downloads:
+
+ * Marengo (multimodal embeddings, 512-dim) — used to *semantically reorder*
+ the LLM-generated search terms against the video subject, so that when the
+ timeline budget runs out the most on-topic footage is the footage that made
+ it in (instead of whatever the LLM happened to list first).
+
+ * Pegasus (video understanding) — used to QA / describe a generated clip from
+ a public URL, e.g. to sanity-check that a downloaded clip actually matches
+ the script before it ships.
+
+The integration is fully opt-in and non-breaking:
+ * If `twelvelabs_api_keys` is not configured, every public function here is a
+ no-op that returns its input unchanged (or None), so default behavior is
+ identical to a build without TwelveLabs.
+ * The `twelvelabs` SDK is imported lazily, so the dependency is only required
+ when the feature is actually used.
+
+Config (config.toml, [app] section):
+ twelvelabs_api_keys = ["tlk_xxx"] # required to enable
+ twelvelabs_rerank_terms = true # opt-in: reorder search terms by relevance
+ twelvelabs_marengo_model = "marengo3.0" # optional override
+ twelvelabs_pegasus_model = "pegasus1.5" # optional override
+
+Get a free API key at https://twelvelabs.io (generous free tier).
+"""
+
+import math
+from functools import lru_cache
+from typing import List, Optional
+
+from loguru import logger
+
+from app.config import config
+from app.services import material
+
+DEFAULT_MARENGO_MODEL = "marengo3.0"
+DEFAULT_PEGASUS_MODEL = "pegasus1.5"
+# Pegasus requires max_tokens in [512, 98304]; 512 is plenty for a one-line QA.
+_PEGASUS_MIN_MAX_TOKENS = 512
+
+
+def is_enabled() -> bool:
+ """True only when at least one TwelveLabs API key is configured."""
+ keys = config.app.get("twelvelabs_api_keys")
+ return bool(keys)
+
+
+def _client():
+ # Lazy import + rotated key reuse mirrors the other providers in
+ # material.py (get_api_key rotates across configured keys).
+ from twelvelabs import TwelveLabs
+
+ api_key = material.get_api_key("twelvelabs_api_keys")
+ return TwelveLabs(api_key=api_key)
+
+
+def _cosine(a: List[float], b: List[float]) -> float:
+ dot = sum(x * y for x, y in zip(a, b))
+ na = math.sqrt(sum(x * x for x in a))
+ nb = math.sqrt(sum(x * x for x in b))
+ if na == 0 or nb == 0:
+ return 0.0
+ return dot / (na * nb)
+
+
+def embed_text(text: str, model: Optional[str] = None) -> Optional[List[float]]:
+ """
+ Return a 512-dim Marengo text embedding, or None on failure / when disabled.
+
+ Cached so repeated terms across a session don't re-hit the API.
+ """
+ if not is_enabled() or not text or not text.strip():
+ return None
+ model = model or config.app.get("twelvelabs_marengo_model", DEFAULT_MARENGO_MODEL)
+ try:
+ # lru_cache only memoizes successful returns; a raised exception is not
+ # cached, so a transient API error never poisons the cache.
+ return _embed_text_cached(text.strip(), model)
+ except Exception as e: # noqa: BLE001 - never break the pipeline on TL errors
+ logger.warning(f"TwelveLabs embed_text failed, skipping rerank: {e}")
+ return None
+
+
+@lru_cache(maxsize=512)
+def _embed_text_cached(text: str, model: str) -> List[float]:
+ client = _client()
+ resp = client.embed.create(model_name=model, text=text)
+ # SDK aliases the raw JSON 'float' vector key to `float_`.
+ return list(resp.text_embedding.segments[0].float_)
+
+
+def rerank_terms_by_subject(
+ video_subject: str,
+ search_terms: List[str],
+ model: Optional[str] = None,
+) -> List[str]:
+ """
+ Reorder `search_terms` so the terms most semantically relevant to
+ `video_subject` come first (Marengo cosine similarity).
+
+ Opt-in: only runs when TwelveLabs is enabled AND
+ `twelvelabs_rerank_terms` is truthy. Falls back to the original order on
+ any failure, so it can never make the pipeline worse.
+ """
+ if not is_enabled() or not config.app.get("twelvelabs_rerank_terms"):
+ return search_terms
+ if not video_subject or len(search_terms) < 2:
+ return search_terms
+
+ subject_vec = embed_text(video_subject, model)
+ if subject_vec is None:
+ return search_terms
+
+ scored = []
+ for term in search_terms:
+ vec = embed_text(term, model)
+ if vec is None:
+ # If any term can't be embedded, don't risk a partial reorder.
+ return search_terms
+ scored.append((term, _cosine(subject_vec, vec)))
+
+ ranked = [term for term, _ in sorted(scored, key=lambda x: x[1], reverse=True)]
+ logger.info(
+ f"TwelveLabs Marengo reranked {len(ranked)} search terms by relevance "
+ f"to subject '{video_subject}': {ranked}"
+ )
+ return ranked
+
+
+def analyze_clip(
+ video_url: str,
+ prompt: str = "Describe what happens in this video in one sentence.",
+ model: Optional[str] = None,
+ max_tokens: int = _PEGASUS_MIN_MAX_TOKENS,
+) -> Optional[str]:
+ """
+ QA / describe a clip from a public URL with Pegasus, returning the model's
+ text answer (or None when disabled / on failure).
+
+ Notes (TwelveLabs API constraints):
+ * Pegasus needs a publicly reachable URL (or an uploaded asset), not a
+ bare local path; the analyzed window must be >= 4s.
+ * max_tokens must be >= 512 for this model.
+ """
+ if not is_enabled() or not video_url:
+ return None
+ model = model or config.app.get("twelvelabs_pegasus_model", DEFAULT_PEGASUS_MODEL)
+ try:
+ from twelvelabs.types import VideoContext_Url
+
+ client = _client()
+ resp = client.analyze(
+ model_name=model,
+ video=VideoContext_Url(url=video_url),
+ prompt=prompt,
+ max_tokens=max(max_tokens, _PEGASUS_MIN_MAX_TOKENS),
+ )
+ return resp.data
+ except Exception as e: # noqa: BLE001
+ logger.warning(f"TwelveLabs analyze_clip failed: {e}")
+ return None
diff --git a/config.example.toml b/config.example.toml
index 6a268cb7..703eaa5d 100644
--- a/config.example.toml
+++ b/config.example.toml
@@ -48,6 +48,31 @@ pixabay_api_keys = []
# 特别注意格式,Key 用英文双引号括起来,多个Key用逗号隔开
coverr_api_keys = []
+# TwelveLabs API Key (optional) — https://twelvelabs.io
+# TwelveLabs (可选) 视频理解 / 多模态嵌入集成。
+# Enables two opt-in features:
+# * Marengo (multimodal embeddings) semantically reorders the auto-generated
+# search terms by relevance to your video subject, so the most on-topic
+# B-roll is downloaded first when the timeline budget is tight.
+# * Pegasus (video understanding) can QA / describe a clip from its URL.
+# Leave empty to keep MoneyPrinterTurbo's behavior completely unchanged.
+# Get a free key (generous free tier) at https://twelvelabs.io and install the
+# optional dependency with `uv sync --extra twelvelabs`.
+# Register at https://playground.twelvelabs.io to get your API key.
+# You can use multiple keys to avoid rate limits.
+# For example: twelvelabs_api_keys = ["tlk_123adsf4567","tlk_abd1321cd13ef"]
+# 请使用英文半角双引号包裹每个 Key;从富文本复制来的弯引号会导致 TOML 解析失败。
+twelvelabs_api_keys = []
+
+# Reorder search terms by Marengo relevance before downloading footage.
+# Requires twelvelabs_api_keys above. Ignored in "match materials to script"
+# mode, where term order is intentionally the script's narrative order.
+twelvelabs_rerank_terms = false
+
+# Optional model overrides (defaults shown).
+# twelvelabs_marengo_model = "marengo3.0"
+# twelvelabs_pegasus_model = "pegasus1.5"
+
# 支持的提供商 (Supported providers):
# openai
# aihubmix (OpenAI-compatible AI model gateway)
diff --git a/pyproject.toml b/pyproject.toml
index d875ce3c..b2e96107 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -37,6 +37,12 @@ dependencies = [
g4f = [
"g4f==0.5.2.2",
]
+# Optional TwelveLabs (https://twelvelabs.io) video-understanding/embedding
+# integration. Only needed when `twelvelabs_api_keys` is configured; install
+# with `uv sync --extra twelvelabs`.
+twelvelabs = [
+ "twelvelabs>=1.2.8",
+]
[tool.hatch.build.targets.wheel]
packages = ["app"]
diff --git a/test/services/test_twelvelabs.py b/test/services/test_twelvelabs.py
new file mode 100644
index 00000000..ce090e19
--- /dev/null
+++ b/test/services/test_twelvelabs.py
@@ -0,0 +1,169 @@
+import os
+import sys
+import unittest
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+sys.path.insert(0, str(Path(__file__).parent.parent.parent))
+
+from app.config import config
+from app.services import twelvelabs
+
+RUN_INTEGRATION_TESTS = os.environ.get("MPT_RUN_INTEGRATION_TESTS", "").lower() in {
+ "1",
+ "true",
+ "yes",
+}
+
+
+class TestTwelveLabsService(unittest.TestCase):
+ """
+ TwelveLabs 集成是完全 opt-in 的:未配置 twelvelabs_api_keys 时所有函数
+ 都必须是无副作用的 no-op,行为与不接入 TwelveLabs 完全一致。
+ 这些用例全部用 mock 替换 SDK 客户端,CI 不依赖真实网络或真实 API key。
+ """
+
+ def setUp(self):
+ self.original_app_config = dict(config.app)
+ twelvelabs._embed_text_cached.cache_clear()
+
+ def tearDown(self):
+ config.app.clear()
+ config.app.update(self.original_app_config)
+ twelvelabs._embed_text_cached.cache_clear()
+
+ # ---------------- disabled / no-op behavior ----------------
+
+ def test_disabled_when_no_api_key(self):
+ config.app.pop("twelvelabs_api_keys", None)
+ self.assertFalse(twelvelabs.is_enabled())
+ # rerank must return the input list unchanged
+ terms = ["b", "a", "c"]
+ self.assertEqual(
+ twelvelabs.rerank_terms_by_subject("subject", terms), terms
+ )
+ # analyze must be a no-op returning None
+ self.assertIsNone(twelvelabs.analyze_clip("https://x/y.mp4"))
+
+ def test_rerank_skipped_when_flag_off(self):
+ config.app["twelvelabs_api_keys"] = ["tlk_test"]
+ config.app["twelvelabs_rerank_terms"] = False
+ terms = ["b", "a"]
+ # Even enabled, with the flag off we must not touch order or call the API.
+ with patch.object(twelvelabs, "_client") as client:
+ result = twelvelabs.rerank_terms_by_subject("subject", terms)
+ self.assertEqual(result, terms)
+ client.assert_not_called()
+
+ # ---------------- enabled rerank behavior ----------------
+
+ def _client_returning(self, vectors_by_text):
+ """Build a fake TwelveLabs client whose embed.create returns canned vectors."""
+
+ def fake_create(*, model_name, text):
+ seg = MagicMock()
+ seg.float_ = vectors_by_text[text]
+ resp = MagicMock()
+ resp.text_embedding.segments = [seg]
+ return resp
+
+ client = MagicMock()
+ client.embed.create.side_effect = fake_create
+ return client
+
+ def test_rerank_orders_by_cosine_to_subject(self):
+ config.app["twelvelabs_api_keys"] = ["tlk_test"]
+ config.app["twelvelabs_rerank_terms"] = True
+
+ # subject aligned with "city"; "kitten" is orthogonal.
+ vectors = {
+ "city skyline": [1.0, 0.0, 0.0],
+ "downtown buildings": [0.9, 0.1, 0.0], # close to subject
+ "cute kitten": [0.0, 1.0, 0.0], # far from subject
+ }
+ client = self._client_returning(vectors)
+
+ with patch.object(twelvelabs, "_client", return_value=client):
+ result = twelvelabs.rerank_terms_by_subject(
+ "city skyline", ["cute kitten", "downtown buildings"]
+ )
+
+ # most relevant term must come first
+ self.assertEqual(result, ["downtown buildings", "cute kitten"])
+
+ def test_rerank_falls_back_on_embed_failure(self):
+ config.app["twelvelabs_api_keys"] = ["tlk_test"]
+ config.app["twelvelabs_rerank_terms"] = True
+
+ client = MagicMock()
+ client.embed.create.side_effect = RuntimeError("api down")
+
+ terms = ["alpha", "beta"]
+ with patch.object(twelvelabs, "_client", return_value=client):
+ result = twelvelabs.rerank_terms_by_subject("subject", terms)
+
+ # any failure must preserve the original order (never make things worse)
+ self.assertEqual(result, terms)
+
+ def test_rerank_noop_for_single_term(self):
+ config.app["twelvelabs_api_keys"] = ["tlk_test"]
+ config.app["twelvelabs_rerank_terms"] = True
+ with patch.object(twelvelabs, "_client") as client:
+ result = twelvelabs.rerank_terms_by_subject("subject", ["only"])
+ self.assertEqual(result, ["only"])
+ client.assert_not_called()
+
+ # ---------------- analyze_clip ----------------
+
+ def test_analyze_clip_returns_model_text(self):
+ config.app["twelvelabs_api_keys"] = ["tlk_test"]
+
+ resp = MagicMock()
+ resp.data = "A city skyline at dusk."
+ client = MagicMock()
+ client.analyze.return_value = resp
+
+ with patch.object(twelvelabs, "_client", return_value=client):
+ out = twelvelabs.analyze_clip(
+ "https://example.com/clip.mp4", prompt="describe"
+ )
+
+ self.assertEqual(out, "A city skyline at dusk.")
+ # max_tokens must be clamped to the Pegasus minimum (>=512)
+ self.assertGreaterEqual(client.analyze.call_args.kwargs["max_tokens"], 512)
+
+
+ RUN_INTEGRATION_TESTS and os.getenv("TWELVELABS_API_KEY"),
+ "live test: set MPT_RUN_INTEGRATION_TESTS=1 and TWELVELABS_API_KEY to run "
+ "against the real TwelveLabs API",
+)
+class TestTwelveLabsLive(unittest.TestCase):
+ """Live contract check — only runs with MPT_RUN_INTEGRATION_TESTS=1 + a key."""
+
+ def setUp(self):
+ self.original_app_config = dict(config.app)
+ config.app["twelvelabs_api_keys"] = [os.environ["TWELVELABS_API_KEY"]]
+ config.app["twelvelabs_rerank_terms"] = True
+ twelvelabs._embed_text_cached.cache_clear()
+
+ def tearDown(self):
+ config.app.clear()
+ config.app.update(self.original_app_config)
+ twelvelabs._embed_text_cached.cache_clear()
+
+ def test_marengo_embedding_is_512_dim(self):
+ vec = twelvelabs.embed_text("a city skyline at night")
+ self.assertIsNotNone(vec)
+ self.assertEqual(len(vec), 512)
+
+ def test_rerank_puts_relevant_term_first(self):
+ result = twelvelabs.rerank_terms_by_subject(
+ "city skyline at night",
+ ["cute kitten playing with yarn", "downtown buildings and traffic at dusk"],
+ )
+ self.assertEqual(result[0], "downtown buildings and traffic at dusk")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/uv.lock b/uv.lock
index a5d50c7c..7a9f1745 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1127,6 +1127,9 @@ dependencies = [
g4f = [
{ name = "g4f" },
]
+twelvelabs = [
+ { name = "twelvelabs" },
+]
[package.metadata]
requires-dist = [
@@ -1148,9 +1151,10 @@ requires-dist = [
{ name = "requests", specifier = "==2.33.1" },
{ name = "socksio", specifier = "==1.0.0" },
{ name = "streamlit", specifier = "==1.58.0" },
+ { name = "twelvelabs", marker = "extra == 'twelvelabs'", specifier = ">=1.2.8" },
{ name = "uvicorn", specifier = "==0.32.1" },
]
-provides-extras = ["g4f"]
+provides-extras = ["g4f", "twelvelabs"]
[[package]]
name = "moviepy"
@@ -2069,6 +2073,21 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf", size = 78374 },
]
+[[package]]
+name = "twelvelabs"
+version = "1.2.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "httpx" },
+ { name = "pydantic" },
+ { name = "pydantic-core" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/5f/24/2604563f5a528bf5f12f9bbb7e3102e91c0e4f9bfd7aeb9963dced3a0b0f/twelvelabs-1.2.8.tar.gz", hash = "sha256:bb0846cc839845c800de8061bb7b99212a472e24bf20770d569f6f620b845e3c", size = 178449 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4e/a6/6380129c65222bb2497cbef27334a496a1a31f4f38973fac46b62b9224fc/twelvelabs-1.2.8-py3-none-any.whl", hash = "sha256:c0487dd892b03728ac2afe3e4ebd4ea5e30ff404b61896667eb5a39264db282b", size = 372073 },
+]
+
[[package]]
name = "typer"
version = "0.26.5"