利用 AI 大模型和自动化工作流,根据主题或关键词一键生成高清短视频。Generate HD short videos from a topic or keyword with an automated AI workflow.
Python ai-video-generatorcontent-creationffmpeginstagram-reelsllmpythonshort-videosubtitlestext-to-speechtiktokvideo-automationvideo-workflowworkflow-automationyoutube-shorts
feat(material): record source provenance
harry07037ddb11d5bfe1
diff --git a/README-en.md b/README-en.mdindex 63ade920..3a82aa58 100644--- a/README-en.md+++ b/README-en.md@@ -31,12 +31,12 @@ English | [简体中文](README.md) | [Releases](https://github.com/harry0703/Mo## Special Thanks ❤️<div align="center">- <a href="https://platform.kimi.ai?aff=MoneyPrinterTurbo" target="_blank"><img src="https://gcdn.moonshot.cn/growth-cdn/sponsor/kimi-en.png" alt="Kimi sponsors MoneyPrinterTurbo" width="100%"></a>+ <a href="https://platform.kimi.ai/?aff=MoneyPrinterTurbo" target="_blank"><img src="https://gcdn.moonshot.cn/growth-cdn/sponsor/kimi-en.png" alt="Kimi sponsors MoneyPrinterTurbo" width="100%"></a></div>-Thanks to [Kimi](https://platform.kimi.ai?aff=MoneyPrinterTurbo) for sponsoring this project! [Kimi K2.7 Code](https://platform.kimi.ai/docs/guide/kimi-k2-7-code-quickstart) is an open-source, coding-focused agentic model developed by Moonshot AI, with substantial gains on real-world long-horizon coding tasks and higher end-to-end success across complex software engineering workflows. It also cuts thinking-token usage by approximately 30% compared with K2.6. Within MoneyPrinterTurbo, Kimi's LLM powers video creation: it writes the video script and extracts the search keywords that decide the final footage, so the sharper its understanding, the more on-topic the results.+Thanks to [Kimi](https://platform.kimi.ai/?aff=MoneyPrinterTurbo) for sponsoring this project! [Kimi K3](https://www.kimi.com/blog/kimi-k3?aff=MoneyPrinterTurbo) is Moonshot AI's most capable model and the world's first open 3T-class model. With native vision and a 1-million-token context window, K3 delivers frontier performance across knowledge work, reasoning, and long-horizon tasks. Within MoneyPrinterTurbo, K3 powers video creation by writing scripts and extracting the search keywords that determine the final footage—the better it understands the content, the more relevant the results.-**MoneyPrinterTurbo already supports Kimi. Visit the [Kimi Open Platform](https://platform.kimi.ai?aff=MoneyPrinterTurbo) ([中文站](https://platform.kimi.com?aff=MoneyPrinterTurbo) | [Global](https://platform.kimi.ai?aff=MoneyPrinterTurbo)) to try the API, or explore the cost-effective [Coding Plan](https://www.kimi.com/code?aff=MoneyPrinterTurbo).**+**MoneyPrinterTurbo already supports Kimi. Visit the Kimi Open Platform ([中文站](https://platform.kimi.com/?aff=MoneyPrinterTurbo) | [Global](https://platform.kimi.ai/?aff=MoneyPrinterTurbo)) to try the API, or explore the [Kimi Code subscription](https://www.kimi.com/code?aff=MoneyPrinterTurbo).**<br><table align="center">diff --git a/README.md b/README.mdindex 60807138..64a2ca9b 100644--- a/README.md+++ b/README.md@@ -31,12 +31,12 @@## 特别感谢 ❤️<div align="center">- <a href="https://platform.kimi.com?aff=MoneyPrinterTurbo" target="_blank"><img src="https://gcdn.moonshot.cn/growth-cdn/sponsor/kimi-zh.png" alt="Kimi 赞助 MoneyPrinterTurbo" width="100%"></a>+ <a href="https://platform.kimi.com/?aff=MoneyPrinterTurbo" target="_blank"><img src="https://gcdn.moonshot.cn/growth-cdn/sponsor/kimi-zh.png" alt="Kimi 赞助 MoneyPrinterTurbo" width="100%"></a></div>-感谢 [Kimi](https://platform.kimi.com?aff=MoneyPrinterTurbo) 赞助本项目 是 Moonshot AI 推出的编程专用开源智能体模型,在真实长程编程与复杂软件工程工作流中显著提升端到端任务成功率,同时优化推理效率,相比 K2.6 平均减少约 30% 的推理 token 消耗。而在本项目中,Kimi 大模型能直接驱动视频创作,不仅撰写视频文案,还会提炼素材搜索关键词、决定成片画面,文案理解越精准,匹配到的素材就越贴题。+感谢 [Kimi](https://platform.kimi.com/?aff=MoneyPrinterTurbo) 赞助本项目 是 Moonshot AI 迄今能力最强的模型,也是全球首个开源 3T 级模型,拥有原生视觉能力与 100 万 Token 上下文,在知识工作、推理和长周期任务中展现前沿性能。在 MoneyPrinterTurbo 中,K3 能直接驱动视频创作,不仅撰写视频文案,还会提炼素材搜索关键词、决定成片画面;对内容理解越准确,匹配到的素材就越贴题。-**本项目已接入 Kimi 大模型。前往 [Kimi 开放平台](https://platform.kimi.com?aff=MoneyPrinterTurbo)([中文站](https://platform.kimi.com?aff=MoneyPrinterTurbo)|[Global](https://platform.kimi.ai?aff=MoneyPrinterTurbo))体验 API,或了解高性价比 [Coding Plan 套餐](https://www.kimi.com/code?aff=MoneyPrinterTurbo)。**+**MoneyPrinterTurbo 已接入 Kimi。前往 Kimi 开放平台([中文站](https://platform.kimi.com/?aff=MoneyPrinterTurbo)|[Global](https://platform.kimi.ai/?aff=MoneyPrinterTurbo))体验 API,或了解 [Kimi Code 订阅](https://www.kimi.com/code?aff=MoneyPrinterTurbo)。**<br><table align="center">diff --git a/app/models/schema.py b/app/models/schema.pyindex 5c3b4ce1..ff7b0807 100644--- a/app/models/schema.py+++ b/app/models/schema.py@@ -55,6 +55,10 @@ class MaterialInfo:provider: str = "pexels"url: str = ""duration: int = 0+ # 在线素材搜索会附带经过筛选的公开来源信息,供搜索缓存和任务记录复用。+ # 本地上传素材不需要填写;写入任务文件前仍会按字段白名单重新构造,+ # 避免外部请求传入的签名 URL、凭据或无关字段进入持久化数据。+ source_info: Optional[dict[str, Any]] = Noneclass VideoParams(BaseModel):diff --git a/app/services/material.py b/app/services/material.pyindex d73ff43f..cf50b812 100644--- a/app/services/material.py+++ b/app/services/material.py@@ -1,8 +1,9 @@import osimport randomimport threading-from typing import Callable, List-from urllib.parse import quote_plus, urlencode+from pathlib import Path+from typing import Any, Callable, List+from urllib.parse import quote_plus, urlencode, urlsplit, urlunsplitimport requestsfrom loguru import logger@@ -10,7 +11,7 @@ from moviepy.video.io.VideoFileClip import VideoFileClipfrom app.config import configfrom app.models.schema import MaterialInfo, VideoAspect, VideoConcatMode-from app.services import material_cache+from app.services import material_cache, task_artifactsfrom app.utils import utils# Thread-safe counter for API key rotation@@ -18,6 +19,124 @@ _api_key_counter = 0_api_key_lock = threading.Lock()+def _safe_public_url(value: Any) -> str | None:+ """+ 只保留可公开展示的 HTTP(S) 页面地址,并移除查询参数和凭据。++ 素材下载地址可能携带 API Key、签名 JWT 或临时 token。任务清单只需要+ 帮助用户回到供应商的公开素材页,不应保存鉴权参数;用户信息形式的 URL+ 同样拒绝,避免 ``https://user:[email protected]`` 一类内容落盘。+ """+ if not isinstance(value, str) or not value.strip():+ return None++ try:+ parsed = urlsplit(value.strip())+ except ValueError:+ return None+ if (+ parsed.scheme not in {"http", "https"}+ or not parsed.hostname+ or parsed.username is not None+ or parsed.password is not None+ ):+ return None+ return urlunsplit((parsed.scheme, parsed.netloc, parsed.path, "", ""))+++def _creator_info(value: Any) -> dict[str, str] | None:+ """从不同供应商的作者结构中提取统一的公开字段。"""+ if isinstance(value, str) and value.strip():+ return {"name": value.strip()}+ if not isinstance(value, dict):+ return None++ creator: dict[str, str] = {}+ creator_id = value.get("id")+ creator_name = value.get("name") or value.get("username")+ creator_page = _safe_public_url(+ value.get("url") or value.get("profile_url") or value.get("profile_page")+ )+ if creator_id is not None:+ creator["id"] = str(creator_id)+ if creator_name:+ creator["name"] = str(creator_name)+ if creator_page:+ creator["profile_page"] = creator_page+ return creator or None+++def _material_source_record(item: MaterialInfo, local_path: str) -> dict[str, Any]:+ """+ 为成功下载的素材生成轻量来源记录。++ ``source_info`` 可能来自缓存,甚至来自外部构造的 ``MaterialInfo``,因此+ 不能原样写入。这里按白名单重新构造,只保留公开页面、业务标识和尺寸,+ 并只记录本地文件名,避免用户目录或 Docker 挂载路径进入任务文件。+ """+ source = item.source_info if isinstance(item.source_info, dict) else {}+ record: dict[str, Any] = {+ "provider": str(item.provider or source.get("provider") or ""),+ "local_file": Path(local_path).name,+ "duration": int(item.duration),+ }++ search_term = source.get("search_term")+ asset_id = source.get("asset_id")+ source_page = _safe_public_url(source.get("source_page"))+ if isinstance(search_term, str) and search_term.strip():+ record["search_term"] = search_term.strip()+ if asset_id not in (None, ""):+ record["asset_id"] = str(asset_id)+ if source_page:+ record["source_page"] = source_page++ creator = _creator_info(source.get("creator"))+ if creator:+ record["creator"] = creator++ raw_rendition = source.get("rendition")+ if isinstance(raw_rendition, dict):+ rendition = {}+ for field in ("id", "width", "height"):+ value = raw_rendition.get(field)+ if value not in (None, ""):+ rendition[field] = str(value) if field == "id" else value+ if rendition:+ record["rendition"] = rendition+ return record+++def _persist_material_sources(+ task_id: str,+ material_sources: list[dict[str, Any]],+) -> None:+ """+ 将当前实际下载成功的素材来源补充到任务清单。++ 任务记录是辅助能力,不能改变视频下载函数的返回值,也不能因为写盘失败+ 中断成片主流程。``patch_script_data`` 会负责原子替换和异常日志;这里仅在+ 成功后记录数量,便于确认任务追溯信息是否已经落盘。+ """+ try:+ saved = task_artifacts.patch_script_data(+ task_id,+ material_sources=material_sources,+ )+ if saved:+ logger.info(+ f"saved material source records: "+ f"task_id={task_id}, count={len(material_sources)}"+ )+ except Exception as exc:+ # task_artifacts 自身已经按失败降级设计,这里仍保留最后一道隔离,+ # 防止未来实现调整或目录解析异常意外影响素材下载返回值。+ logger.warning(+ "failed to persist material source records: "+ f"task_id={task_id}, error={type(exc).__name__}, detail={exc}"+ )++def _get_tls_verify() -> bool:# 默认开启 TLS 证书校验,防止素材搜索和下载过程被中间人篡改。# 仅在企业代理、自签证书等明确需要的场景下,允许用户通过@@ -39,8 +158,8 @@ def get_api_key(cfg_key: str):api_keys = config.app.get(cfg_key)if not api_keys:raise ValueError(- f"\n\n##### {cfg_key} is not set #####\n\nPlease set it in the config.toml file: {config.config_file}\n\n"- f"{utils.to_json(config.app)}"+ f"\n\n##### {cfg_key} is not set #####\n\n"+ f"Please set it in the config.toml file: {config.config_file}\n")# if only one key is provided, return it@@ -72,6 +191,21 @@ def _redact_secret(message: str, secret: str) -> str:return safe_message+def _redact_request_error(error: Exception, *secrets: str) -> str:+ """+ 保留网络异常的可排查信息,同时移除 API Key 和代理凭据。++ 直接只记录异常类型会丢失 DNS、证书、超时等关键上下文;直接记录原始异常+ 又可能回显完整请求 URL。统一入口可以让三个素材供应商使用相同脱敏规则。+ """+ safe_message = str(error)+ for secret in secrets:+ safe_message = _redact_secret(safe_message, str(secret or ""))+ for proxy_url in config.proxy.values():+ safe_message = _redact_secret(safe_message, str(proxy_url))+ return safe_message++def _is_cloudflare_challenge(response: requests.Response) -> bool:"""识别 Cloudflare 返回的 HTML Challenge,而不是把它当成 Pixabay JSON。@@ -108,7 +242,7 @@ def search_videos_pexels(# Build URLparams = {"query": search_term, "per_page": 20, "orientation": video_orientation}query_url = f"https://api.pexels.com/videos/search?{urlencode(params)}"- logger.info(f"searching videos: {query_url}, with proxies: {config.proxy}")+ logger.info(f"searching videos on pexels: term={search_term!r}")try:r = requests.get(@@ -121,7 +255,7 @@ def search_videos_pexels(response = r.json()video_items = []if "videos" not in response:- logger.error(f"search videos failed: {response}")+ logger.error("pexels video search returned an unsupported response")return video_itemsvideos = response["videos"]# loop through each video in the result@@ -140,11 +274,32 @@ def search_videos_pexels(item.provider = "pexels"item.url = video["link"]item.duration = duration+ item.source_info = {+ "provider": "pexels",+ "search_term": search_term,+ "asset_id": (+ str(v.get("id")) if v.get("id") is not None else None+ ),+ "source_page": _safe_public_url(v.get("url")),+ "creator": _creator_info(v.get("user")),+ "rendition": {+ "id": (+ str(video.get("id"))+ if video.get("id") is not None+ else None+ ),+ "width": w,+ "height": h,+ },+ }video_items.append(item)breakreturn video_itemsexcept Exception as e:- logger.error(f"search videos failed: {str(e)}")+ logger.error(+ "pexels video search failed: "+ f"error={type(e).__name__}, detail={_redact_request_error(e, api_key)}"+ )return []@@ -215,7 +370,7 @@ def search_videos_pixabay(video_items = []if "hits" not in response:- logger.error(f"search videos failed: {response}")+ logger.error("pixabay video search returned an unsupported response")return video_itemsvideos = response["hits"]# loop through each video in the result@@ -235,16 +390,30 @@ def search_videos_pixabay(item.provider = "pixabay"item.url = video["url"]item.duration = duration+ item.source_info = {+ "provider": "pixabay",+ "search_term": search_term,+ "asset_id": (+ str(v.get("id")) if v.get("id") is not None else None+ ),+ "source_page": _safe_public_url(v.get("pageURL")),+ "creator": _creator_info(+ {+ "id": v.get("user_id"),+ "name": v.get("user"),+ }+ ),+ "rendition": {+ "id": video_type,+ "width": w,+ "height": video.get("height"),+ },+ }video_items.append(item)breakreturn video_itemsexcept Exception as e:- error_message = _redact_secret(str(e), api_key)- # ProxyError 等异常可能回显完整代理 URL,其中可能包含用户名和密码。- # 仅替换用户实际配置的代理值,不对普通错误文本做宽泛正则处理,- # 避免误删 DNS、超时、证书等真正有助于排查的信息。- for proxy_url in config.proxy.values():- error_message = _redact_secret(error_message, str(proxy_url))+ error_message = _redact_request_error(e, api_key)logger.error("pixabay search request failed: "f"error={type(e).__name__}, detail={error_message}"@@ -286,7 +455,7 @@ def search_videos_coverr("sort": "popular",}query_url = f"https://api.coverr.co/videos?{urlencode(params)}"- logger.info(f"searching videos: {query_url}, with proxies: {config.proxy}")+ logger.info(f"searching videos on coverr: term={search_term!r}")try:r = requests.get(@@ -300,7 +469,7 @@ def search_videos_coverr(video_items: List[MaterialInfo] = []if not isinstance(response, dict) or "hits" not in response:- logger.error(f"search videos failed: {response}")+ logger.error("coverr video search returned an unsupported response")return video_itemsfor v in response["hits"]:@@ -321,10 +490,25 @@ def search_videos_coverr(item.provider = "coverr"item.url = mp4_download_urlitem.duration = duration+ item.source_info = {+ "provider": "coverr",+ "search_term": search_term,+ "asset_id": str(video_id),+ "source_page": _safe_public_url(v.get("canonical_url") or v.get("url")),+ "creator": _creator_info(v.get("creator") or v.get("author")),+ "rendition": {+ "id": "mp4_download",+ "width": v.get("max_width"),+ "height": v.get("max_height"),+ },+ }video_items.append(item)return video_itemsexcept Exception as e:- logger.error(f"search videos failed: {str(e)}")+ logger.error(+ "coverr video search failed: "+ f"error={type(e).__name__}, detail={_redact_request_error(e, api_key)}"+ )return []@@ -439,6 +623,13 @@ def _search_videos_with_cache(minimum_duration=minimum_duration,video_aspect=video_aspect,)+ # Provider 正常会写入当前关键词,但测试替身、第三方扩展或旧实现可能+ # 遗漏或携带错误值。缓存读取会根据缓存键恢复该字段,因此远端结果也在+ # 同一入口校正,保证首次搜索与缓存命中的任务来源记录保持一致。+ for item in items:+ if isinstance(item.source_info, dict):+ item.source_info = dict(item.source_info)+ item.source_info["search_term"] = search_termif items:try:material_cache.save_material_search_cache(@@ -523,6 +714,7 @@ def download_videos(f"found total videos: {len(valid_video_items)}, required duration: {audio_duration} seconds, found duration: {found_duration} seconds")video_paths = []+ material_sources: list[dict[str, Any]] = []concat_mode_value = getattr(video_concat_mode, "value", video_concat_mode)if concat_mode_value == VideoConcatMode.random.value:@@ -531,13 +723,29 @@ def download_videos(total_duration = 0.0for item in valid_video_items:try:- logger.info(f"downloading video: {item.url}")+ source_info = item.source_info if isinstance(item.source_info, dict) else {}+ logger.info(+ f"downloading {item.provider} video: "+ f"asset_id={source_info.get('asset_id') or 'unknown'}"+ )saved_video_path = save_video(video_url=item.url, save_dir=material_directory)if saved_video_path:logger.info(f"video saved: {saved_video_path}")video_paths.append(saved_video_path)+ try:+ material_sources.append(+ _material_source_record(item, saved_video_path)+ )+ except Exception as source_error:+ # 来源记录异常不能把已经成功下载的素材视为下载失败,更不能+ # 阻断视频生成;保留供应商和异常类型用于后续定位。+ logger.warning(+ "failed to prepare material source record: "+ f"provider={item.provider}, "+ f"error={type(source_error).__name__}, detail={source_error}"+ )seconds = min(max_clip_duration, item.duration)total_duration += secondsif total_duration > audio_duration:@@ -546,8 +754,13 @@ def download_videos()breakexcept Exception as e:- logger.error(f"failed to download video: {utils.to_json(item)} => {str(e)}")+ logger.error(+ "failed to download material video: "+ f"provider={item.provider}, error={type(e).__name__}, "+ f"detail={_redact_request_error(e, item.url)}"+ )logger.success(f"downloaded {len(video_paths)} videos")+ _persist_material_sources(task_id, material_sources)return video_paths@@ -599,6 +812,7 @@ def _download_videos_by_script_order()video_paths = []+ material_sources: list[dict[str, Any]] = []total_duration = 0.0candidate_index = 0while candidate_groups and total_duration <= audio_duration:@@ -610,8 +824,12 @@ def _download_videos_by_script_order(has_candidate = Trueitem = term_items[candidate_index]try:+ source_info = (+ item.source_info if isinstance(item.source_info, dict) else {}+ )logger.info(- f"downloading ordered video for '{search_term}': {item.url}"+ f"downloading ordered {item.provider} video for {search_term!r}: "+ f"asset_id={source_info.get('asset_id') or 'unknown'}")saved_video_path = save_video(video_url=item.url, save_dir=material_directory@@ -619,6 +837,17 @@ def _download_videos_by_script_order(if saved_video_path:logger.info(f"video saved: {saved_video_path}")video_paths.append(saved_video_path)+ try:+ material_sources.append(+ _material_source_record(item, saved_video_path)+ )+ except Exception as source_error:+ logger.warning(+ "failed to prepare ordered material source record: "+ f"provider={item.provider}, "+ f"error={type(source_error).__name__}, "+ f"detail={source_error}"+ )total_duration += min(max_clip_duration, item.duration)if total_duration > audio_duration:logger.info(@@ -627,7 +856,9 @@ def _download_videos_by_script_order(breakexcept Exception as e:logger.error(- f"failed to download ordered video: {utils.to_json(item)} => {str(e)}"+ "failed to download ordered material video: "+ f"provider={item.provider}, error={type(e).__name__}, "+ f"detail={_redact_request_error(e, item.url)}")if not has_candidate:@@ -635,6 +866,7 @@ def _download_videos_by_script_order(candidate_index += 1logger.success(f"downloaded {len(video_paths)} ordered videos")+ _persist_material_sources(task_id, material_sources)return video_pathsdiff --git a/app/services/material_cache.py b/app/services/material_cache.pyindex edea3c4a..77cc0c58 100644--- a/app/services/material_cache.py+++ b/app/services/material_cache.py@@ -11,6 +11,7 @@ import threadingimport timefrom pathlib import Pathfrom typing import Iterable+from urllib.parse import urlsplit, urlunsplitfrom loguru import logger@@ -19,7 +20,7 @@ from app.utils import utilsMATERIAL_SEARCH_CACHE_TTL_SECONDS = 24 * 60 * 60-_CACHE_FORMAT_VERSION = 1+_CACHE_FORMAT_VERSION = 2_CACHE_CLEANUP_INTERVAL_SECONDS = 60 * 60_CACHE_FILE_PATTERN = re.compile(r"^[0-9a-f]{64}\.json$")@@ -31,6 +32,73 @@ _cleanup_state_lock = threading.Lock()_last_cleanup_monotonic: float | None = None+def _safe_public_url(value) -> str | None:+ """移除公开页面 URL 的查询参数和用户凭据,避免缓存意外保存 token。"""+ if not isinstance(value, str) or not value.strip():+ return None+ try:+ parsed = urlsplit(value.strip())+ except ValueError:+ return None+ if (+ parsed.scheme not in {"http", "https"}+ or not parsed.hostname+ or parsed.username is not None+ or parsed.password is not None+ ):+ return None+ return urlunsplit((parsed.scheme, parsed.netloc, parsed.path, "", ""))+++def _cached_source_info(item: MaterialInfo) -> dict | None:+ """+ 按白名单构造可落盘的来源信息。++ 搜索关键词已经包含在缓存键中,不再明文写入缓存内容;读取时由调用参数+ 恢复。下载 URL 由 ``MaterialInfo.url`` 单独保存,这里只允许公开素材页、+ 作者公开页和稳定业务标识,避免任意扩展字段进入磁盘缓存。+ """+ source = item.source_info+ if not isinstance(source, dict) or not source:+ return None++ cached: dict = {+ "provider": str(source.get("provider") or item.provider),+ }+ asset_id = source.get("asset_id")+ source_page = _safe_public_url(source.get("source_page"))+ if asset_id not in (None, ""):+ cached["asset_id"] = str(asset_id)+ if source_page:+ cached["source_page"] = source_page++ raw_creator = source.get("creator")+ if isinstance(raw_creator, dict):+ creator = {}+ creator_id = raw_creator.get("id")+ creator_name = raw_creator.get("name")+ creator_page = _safe_public_url(raw_creator.get("profile_page"))+ if creator_id not in (None, ""):+ creator["id"] = str(creator_id)+ if creator_name not in (None, ""):+ creator["name"] = str(creator_name)+ if creator_page:+ creator["profile_page"] = creator_page+ if creator:+ cached["creator"] = creator++ raw_rendition = source.get("rendition")+ if isinstance(raw_rendition, dict):+ rendition = {}+ for field in ("id", "width", "height"):+ value = raw_rendition.get(field)+ if value not in (None, ""):+ rendition[field] = str(value) if field == "id" else value+ if rendition:+ cached["rendition"] = rendition+ return cached++def _cache_dir() -> Path:"""返回所有运行入口共用的素材搜索缓存目录。@@ -124,6 +192,25 @@ def load_material_search_cache(``None`` 表示缓存未命中,需要请求远端 API;空列表不作为有效缓存返回,避免网络错误或上游异常被误缓存后持续阻断后续任务。"""+ if str(provider).strip().lower() == "coverr":+ # Coverr 的下载地址包含绑定 API Key 的签名 JWT。它只用于当前请求,+ # 不能进入磁盘缓存;查询相同条件时顺带删除旧版本可能留下的缓存。+ try:+ _remove_invalid_cache(+ _cache_path(+ provider=provider,+ search_term=search_term,+ minimum_duration=minimum_duration,+ video_aspect=video_aspect,+ )+ )+ except Exception as exc:+ logger.warning(+ "failed to remove disabled Coverr material search cache: "+ f"error={type(exc).__name__}, detail={exc}"+ )+ return None+try:cache_path = _cache_path(provider=provider,@@ -177,6 +264,7 @@ def load_material_search_cache(item_provider = raw_item.get("provider")item_url = raw_item.get("url")item_duration = raw_item.get("duration")+ source_info = raw_item.get("source_info")if (not isinstance(item_provider, str)or not item_provider@@ -185,19 +273,23 @@ def load_material_search_cache(or isinstance(item_duration, bool)or not isinstance(item_duration, (int, float))or item_duration <= 0+ or not isinstance(source_info, dict)+ or not source_info):raise ValueError("invalid material fields")+ source_info = dict(source_info)+ source_info["search_term"] = search_termitems.append(MaterialInfo(provider=item_provider,url=item_url,duration=int(item_duration),+ source_info=source_info,))except (OSError, ValueError, TypeError) as exc:logger.warning(- f"failed to load material search cache: "- f"file={cache_path.name}, error={exc}"+ f"failed to load material search cache: file={cache_path.name}, error={exc}")_remove_invalid_cache(cache_path)return None@@ -223,17 +315,24 @@ def save_material_search_cache(``os.replace`` 发布,可以保证读进程只会看到完整旧文件或完整新文件;即使两个写进程同时完成,最终内容也都是同一缓存键对应的合法结果。"""+ if str(provider).strip().lower() == "coverr":+ return False+temp_path = Nonetry:- serialized_items = [- {- "provider": item.provider,- "url": item.url,- "duration": int(item.duration),- }- for item in items- if item.url and item.duration > 0- ]+ serialized_items = []+ for item in items:+ source_info = _cached_source_info(item)+ if not item.url or item.duration <= 0 or not source_info:+ continue+ serialized_items.append(+ {+ "provider": item.provider,+ "url": item.url,+ "duration": int(item.duration),+ "source_info": source_info,+ }+ )if not serialized_items:return Falsediff --git a/app/services/task.py b/app/services/task.pyindex cd08d3bc..5ea42ac5 100644--- a/app/services/task.py+++ b/app/services/task.py@@ -21,6 +21,7 @@ from app.services import (material,sonilo,subtitle,+ task_artifacts,twelvelabs,video,voice,@@ -328,15 +329,12 @@ def generate_terms(task_id, params, video_script):def save_script_data(task_id, video_script, video_terms, params):- script_file = path.join(utils.task_dir(task_id), "script.json")script_data = {"script": video_script,"search_terms": video_terms,"params": params,}-- with open(script_file, "w", encoding="utf-8") as f:- f.write(utils.to_json(script_data))+ task_artifacts.write_script_data(task_id, script_data)def resolve_custom_audio_file(task_id: str, custom_audio_file: str | None) -> str:diff --git a/app/services/task_artifacts.py b/app/services/task_artifacts.pynew file mode 100644index 00000000..95b45ef4--- /dev/null+++ b/app/services/task_artifacts.py@@ -0,0 +1,96 @@+"""任务目录中持久化文件的安全读写。"""++from __future__ import annotations++import json+import os+import tempfile+from pathlib import Path+from typing import Any, Mapping++from loguru import logger++from app.utils import utils+++def _script_file(task_id: str) -> Path:+ """返回任务脚本清单路径,并复用统一的任务目录创建逻辑。"""+ return Path(utils.task_dir(task_id)) / "script.json"+++def _write_json_atomic(target: Path, payload: Mapping[str, Any]) -> None:+ """+ 在目标目录内原子写入 JSON,避免进程中断留下半个文件。++ 临时文件和目标文件必须位于同一目录,才能保证 ``os.replace`` 在常见+ 本地文件系统和 Docker 挂载目录中保持原子替换语义。写入成功前不会修改+ 现有文件;异常时只清理本次创建的临时文件,并把错误交给调用方决定是否+ 影响主流程。+ """+ temp_path: Path | None = None+ try:+ with tempfile.NamedTemporaryFile(+ mode="w",+ encoding="utf-8",+ dir=target.parent,+ prefix=f".{target.name}.",+ suffix=".tmp",+ delete=False,+ ) as temp_file:+ temp_path = Path(temp_file.name)+ json.dump(+ payload,+ temp_file,+ ensure_ascii=False,+ indent=4,+ default=lambda value: value.__dict__,+ )+ temp_file.write("\n")+ temp_file.flush()+ os.fsync(temp_file.fileno())++ os.replace(temp_path, target)+ temp_path = None+ finally:+ if temp_path is not None:+ temp_path.unlink(missing_ok=True)+++def write_script_data(task_id: str, payload: Mapping[str, Any]) -> None:+ """创建或完整替换任务的 ``script.json`` 清单。"""+ _write_json_atomic(_script_file(task_id), payload)+++def patch_script_data(task_id: str, **updates: Any) -> bool:+ """+ 在保留原有字段的前提下补充任务清单,失败时返回 ``False``。++ 素材来源属于辅助诊断信息,不能因为文件权限、磁盘瞬时异常或历史文件损坏+ 阻断视频生成。因此该入口会记录完整异常并降级;首次创建任务清单仍使用+ ``write_script_data``,由主流程决定基础任务数据写入失败时如何处理。+ """+ try:+ target = _script_file(task_id)+ with target.open("r", encoding="utf-8") as script_file:+ payload = json.load(script_file)+ if not isinstance(payload, dict):+ raise ValueError("task script data must be a JSON object")++ payload.update(updates)+ _write_json_atomic(target, payload)+ return True+ except FileNotFoundError:+ # ``download_videos`` 也可能被测试、脚本或第三方代码独立调用,此时没有+ # 任务清单属于正常场景,不应制造警告或为了辅助记录创建残缺文件。+ logger.debug(+ f"skip task script update because script.json does not exist: "+ f"task_id={task_id}"+ )+ return False+ except Exception as exc:+ logger.warning(+ "failed to update task script data: "+ f"task_id={task_id}, fields={sorted(updates)}, "+ f"error={type(exc).__name__}, detail={exc}"+ )+ return Falsediff --git a/test/services/test_material.py b/test/services/test_material.pyindex 09f3a4f6..a9140c2e 100644--- a/test/services/test_material.py+++ b/test/services/test_material.py@@ -38,9 +38,17 @@ class TestMaterialTlsVerification(unittest.TestCase):json=lambda: {"videos": [{+ "id": 321,+ "url": "https://www.pexels.com/video/example-321/?token=drop","duration": 8,+ "user": {+ "id": 654,+ "name": "Pexels Creator",+ "url": "https://www.pexels.com/@creator/?key=drop",+ },"video_files": [{+ "id": 987,"width": 1080,"height": 1920,"link": "https://example.com/video.mp4",@@ -56,6 +64,16 @@ class TestMaterialTlsVerification(unittest.TestCase):self.assertEqual(len(results), 1)self.assertTrue(get.call_args.kwargs["verify"])+ self.assertEqual(results[0].source_info["asset_id"], "321")+ self.assertEqual(+ results[0].source_info["source_page"],+ "https://www.pexels.com/video/example-321/",+ )+ self.assertEqual(+ results[0].source_info["creator"]["profile_page"],+ "https://www.pexels.com/@creator/",+ )+ self.assertEqual(results[0].source_info["rendition"]["id"], "987")def test_search_pixabay_allows_explicit_tls_disable_for_proxy(self):"""@@ -284,6 +302,59 @@ class TestMaterialTlsVerification(unittest.TestCase):self.assertEqual(result, [])+ def test_material_source_record_uses_public_whitelist(self):+ """+ 任务清单只应包含可追溯的公开字段,不能写入签名参数、下载地址、+ 调用方传入的额外字段或本机绝对路径。+ """+ item = material.MaterialInfo(+ provider="pixabay",+ url="https://cdn.example.com/video.mp4?token=secret",+ duration=12,+ source_info={+ "provider": "pixabay",+ "search_term": "city",+ "asset_id": 123,+ "source_page": "https://pixabay.com/videos/city-123/?key=secret",+ "creator": {+ "id": 456,+ "name": "Creator",+ "profile_page": "https://pixabay.com/users/creator/?token=secret",+ "email": "[email protected]",+ },+ "rendition": {+ "id": "large",+ "width": 1920,+ "height": 1080,+ "download_url": "https://cdn.example.com/private",+ },+ "api_key": "must-not-persist",+ },+ )++ record = material._material_source_record(+ item,+ "/Users/example/private/task/vid-123.mp4",+ )+ serialized = str(record)++ self.assertEqual(record["local_file"], "vid-123.mp4")+ self.assertEqual(+ record["source_page"],+ "https://pixabay.com/videos/city-123/",+ )+ self.assertEqual(+ record["creator"]["profile_page"],+ "https://pixabay.com/users/creator/",+ )+ self.assertEqual(+ record["rendition"],+ {"id": "large", "width": 1920, "height": 1080},+ )+ self.assertNotIn("secret", serialized)+ self.assertNotIn("/Users/example", serialized)+ self.assertNotIn("[email protected]", serialized)+def test_download_videos_can_round_robin_terms_in_script_order(self):"""开启按文案顺序匹配素材后,不能让第一个关键词的多个候选先把@@ -292,12 +363,48 @@ class TestMaterialTlsVerification(unittest.TestCase):"""search_results = {"opening city": [- material.MaterialInfo(provider="pexels", url="https://v.example/a1.mp4", duration=3),- material.MaterialInfo(provider="pexels", url="https://v.example/a2.mp4", duration=3),+ material.MaterialInfo(+ provider="pexels",+ url="https://v.example/a1.mp4",+ duration=3,+ source_info={+ "provider": "pexels",+ "search_term": "opening city",+ "asset_id": "a1",+ },+ ),+ material.MaterialInfo(+ provider="pexels",+ url="https://v.example/a2.mp4",+ duration=3,+ source_info={+ "provider": "pexels",+ "search_term": "opening city",+ "asset_id": "a2",+ },+ ),],"middle office": [- material.MaterialInfo(provider="pexels", url="https://v.example/b1.mp4", duration=3),- material.MaterialInfo(provider="pexels", url="https://v.example/b2.mp4", duration=3),+ material.MaterialInfo(+ provider="pexels",+ url="https://v.example/b1.mp4",+ duration=3,+ source_info={+ "provider": "pexels",+ "search_term": "middle office",+ "asset_id": "b1",+ },+ ),+ material.MaterialInfo(+ provider="pexels",+ url="https://v.example/b2.mp4",+ duration=3,+ source_info={+ "provider": "pexels",+ "search_term": "middle office",+ "asset_id": "b2",+ },+ ),],}downloaded_urls = []@@ -319,6 +426,11 @@ class TestMaterialTlsVerification(unittest.TestCase):return_value=None,),patch.object(material.material_cache, "save_material_search_cache"),+ patch.object(+ material.task_artifacts,+ "patch_script_data",+ return_value=True,+ ) as patch_script,):result = material.download_videos(task_id="ordered-materials",@@ -338,6 +450,52 @@ class TestMaterialTlsVerification(unittest.TestCase):],)self.assertEqual(result, ["/tmp/a1.mp4", "/tmp/b1.mp4", "/tmp/a2.mp4"])+ recorded_sources = patch_script.call_args.kwargs["material_sources"]+ self.assertEqual(+ [source["asset_id"] for source in recorded_sources],+ ["a1", "b1", "a2"],+ )+ self.assertEqual(+ [source["local_file"] for source in recorded_sources],+ ["a1.mp4", "b1.mp4", "a2.mp4"],+ )++ def test_material_source_persistence_failure_does_not_break_download(self):+ """辅助任务记录失败时,已经下载成功的素材仍应正常返回给成片主流程。"""+ item = material.MaterialInfo(+ provider="pexels",+ url="https://v.example/a1.mp4",+ duration=5,+ source_info={"provider": "pexels", "asset_id": "a1"},+ )++ with (+ patch.dict(config.app, {"material_directory": ""}),+ patch.object(material, "search_videos_pexels", return_value=[item]),+ patch.object(material, "save_video", return_value="/tmp/a1.mp4"),+ patch.object(+ material.material_cache,+ "load_material_search_cache",+ return_value=None,+ ),+ patch.object(material.material_cache, "save_material_search_cache"),+ patch.object(+ material.task_artifacts,+ "patch_script_data",+ side_effect=OSError("disk unavailable"),+ ),+ patch.object(material.logger, "warning") as warning,+ ):+ result = material.download_videos(+ task_id="persist-failure",+ search_terms=["city"],+ source="pexels",+ audio_duration=1,+ max_clip_duration=5,+ )++ self.assertEqual(result, ["/tmp/a1.mp4"])+ self.assertTrue(warning.called)class TestCoverrProvider(unittest.TestCase):@@ -381,6 +539,14 @@ class TestCoverrProvider(unittest.TestCase):"id": "S1YbPl1NfI","duration": 11.625,"aspect_ratio": "16:9",+ "canonical_url": "https://coverr.co/videos/example?token=drop",+ "creator": {+ "id": "creator-1",+ "name": "Coverr Creator",+ "profile_url": "https://coverr.co/creators/example?key=drop",+ },+ "max_width": 3840,+ "max_height": 2160,"urls": {"mp4": "https://storage.coverr.co/videos/abc?token=xyz","mp4_preview": "https://storage.coverr.co/videos/abc/preview?token=xyz",@@ -404,6 +570,15 @@ class TestCoverrProvider(unittest.TestCase):self.assertEqual(item.url, "https://storage.coverr.co/videos/abc/download?token=xyz")+ self.assertEqual(item.source_info["asset_id"], "S1YbPl1NfI")+ self.assertEqual(+ item.source_info["source_page"],+ "https://coverr.co/videos/example",+ )+ self.assertEqual(+ item.source_info["creator"]["profile_page"],+ "https://coverr.co/creators/example",+ )# Bearer auth + TLS verify on by defaultself.assertEqual(get.call_args.kwargs["headers"]["Authorization"], "Bearer coverr-key"diff --git a/test/services/test_material_cache.py b/test/services/test_material_cache.pyindex 442b986b..13033a2f 100644--- a/test/services/test_material_cache.py+++ b/test/services/test_material_cache.py@@ -26,7 +26,27 @@ class TestMaterialSearchCache(unittest.TestCase):@staticmethoddef _item(url: str = "https://example.com/video.mp4") -> MaterialInfo:- return MaterialInfo(provider="pixabay", url=url, duration=12)+ return MaterialInfo(+ provider="pixabay",+ url=url,+ duration=12,+ source_info={+ "provider": "pixabay",+ "search_term": "nature",+ "asset_id": "123",+ "source_page": "https://pixabay.com/videos/example-123/",+ "creator": {+ "id": "456",+ "name": "Creator",+ "profile_page": "https://pixabay.com/users/creator-456/",+ },+ "rendition": {+ "id": "large",+ "width": 1920,+ "height": 1080,+ },+ },+ )def _cache_path(self) -> Path:return material_cache._cache_path(@@ -60,6 +80,16 @@ class TestMaterialSearchCache(unittest.TestCase):self.assertEqual(loaded[0].provider, "pixabay")self.assertEqual(loaded[0].url, "https://example.com/video.mp4")self.assertEqual(loaded[0].duration, 12)+ self.assertEqual(loaded[0].source_info["search_term"], "nature")+ self.assertEqual(loaded[0].source_info["asset_id"], "123")+ self.assertEqual(+ loaded[0].source_info["source_page"],+ "https://pixabay.com/videos/example-123/",+ )+ self.assertEqual(+ loaded[0].source_info["creator"]["profile_page"],+ "https://pixabay.com/users/creator-456/",+ )def test_expired_cache_is_removed_and_treated_as_miss(self):"""@@ -155,19 +185,107 @@ class TestMaterialSearchCache(unittest.TestCase):缓存文件名使用摘要,内容只保存素材字段。即使用户共享 storage 目录,文件中也不应出现关键词、API Key 或其它请求配置。"""+ item = self._item()+ item.source_info["source_page"] += "?token=drop"+ item.source_info["creator"]["profile_page"] += "?key=drop"material_cache.save_material_search_cache(provider="pixabay",search_term="private search term",minimum_duration=5,video_aspect=VideoAspect.portrait,- items=[self._item()],+ items=[item],)cache_files = list(Path(self.temp_dir.name).glob("*.json"))self.assertEqual(len(cache_files), 1)self.assertNotIn("private search term", cache_files[0].name)- payload = json.loads(cache_files[0].read_text(encoding="utf-8"))+ raw_payload = cache_files[0].read_text(encoding="utf-8")+ payload = json.loads(raw_payload)self.assertEqual(set(payload), {"version", "items"})+ self.assertNotIn("private search term", raw_payload)+ self.assertNotIn("token=drop", raw_payload)++ def test_coverr_signed_urls_are_never_cached(self):+ """Coverr 下载地址包含签名 JWT,不能进入可长期保留的磁盘缓存。"""+ item = self._item(+ "https://storage.coverr.co/video/download?token=signed-jwt"+ )+ item.provider = "coverr"+ item.source_info["provider"] = "coverr"++ saved = material_cache.save_material_search_cache(+ provider="coverr",+ search_term="nature",+ minimum_duration=5,+ video_aspect=VideoAspect.portrait,+ items=[item],+ )++ self.assertFalse(saved)+ self.assertEqual(list(Path(self.temp_dir.name).glob("*.json")), [])++ def test_coverr_cache_load_removes_legacy_signed_url(self):+ """访问 Coverr 时应清理旧版本可能留下的签名下载地址缓存。"""+ cache_path = material_cache._cache_path(+ provider="coverr",+ search_term="nature",+ minimum_duration=5,+ video_aspect=VideoAspect.portrait,+ )+ cache_path.write_text(+ json.dumps(+ {+ "version": 1,+ "items": [+ {+ "provider": "coverr",+ "url": "https://storage.coverr.co/video?token=signed-jwt",+ "duration": 12,+ }+ ],+ }+ ),+ encoding="utf-8",+ )++ loaded = material_cache.load_material_search_cache(+ provider="coverr",+ search_term="nature",+ minimum_duration=5,+ video_aspect=VideoAspect.portrait,+ )++ self.assertIsNone(loaded)+ self.assertFalse(cache_path.exists())++ def test_version_one_cache_is_invalidated(self):+ """旧缓存缺少来源信息,升级后必须重新查询而不能生成残缺任务记录。"""+ cache_path = self._cache_path()+ cache_path.write_text(+ json.dumps(+ {+ "version": 1,+ "items": [+ {+ "provider": "pixabay",+ "url": "https://example.com/old.mp4",+ "duration": 12,+ }+ ],+ }+ ),+ encoding="utf-8",+ )++ loaded = material_cache.load_material_search_cache(+ provider="pixabay",+ search_term="nature",+ minimum_duration=5,+ video_aspect=VideoAspect.portrait,+ )++ self.assertIsNone(loaded)+ self.assertFalse(cache_path.exists())def test_cache_key_separates_provider_duration_and_aspect(self):"""diff --git a/test/services/test_task_artifacts.py b/test/services/test_task_artifacts.pynew file mode 100644index 00000000..b93c4b88--- /dev/null+++ b/test/services/test_task_artifacts.py@@ -0,0 +1,101 @@+import json+import tempfile+import unittest+from pathlib import Path+from unittest.mock import patch++from app.models.schema import VideoParams+from app.services import task_artifacts+++class TestTaskArtifacts(unittest.TestCase):+ def setUp(self):+ self.temp_dir = tempfile.TemporaryDirectory()+ self.task_dir = Path(self.temp_dir.name)+ self.task_dir_patch = patch(+ "app.services.task_artifacts.utils.task_dir",+ return_value=str(self.task_dir),+ )+ self.task_dir_patch.start()++ def tearDown(self):+ self.task_dir_patch.stop()+ self.temp_dir.cleanup()++ def test_patch_preserves_existing_script_fields(self):+ """补充素材来源时不能覆盖历史任务恢复依赖的文案、关键词和参数。"""+ original = {+ "script": "existing script",+ "search_terms": ["nature"],+ "params": {"video_source": "pixabay"},+ }+ task_artifacts.write_script_data("task-1", original)++ updated = task_artifacts.patch_script_data(+ "task-1",+ material_sources=[+ {+ "provider": "pixabay",+ "asset_id": "123",+ "local_file": "vid-123.mp4",+ }+ ],+ )+ payload = json.loads((self.task_dir / "script.json").read_text())++ self.assertTrue(updated)+ self.assertEqual(payload["script"], original["script"])+ self.assertEqual(payload["search_terms"], original["search_terms"])+ self.assertEqual(payload["params"], original["params"])+ self.assertEqual(payload["material_sources"][0]["asset_id"], "123")+ self.assertEqual(list(self.task_dir.glob(".script.json.*.tmp")), [])++ def test_write_script_data_serializes_video_params(self):+ """原子写入替换旧实现后,仍需完整兼容任务主流程传入的 Pydantic 参数。"""+ params = VideoParams(+ video_subject="test subject",+ video_terms=["city", "night"],+ )++ task_artifacts.write_script_data(+ "task-params",+ {+ "script": "test script",+ "search_terms": ["city"],+ "params": params,+ },+ )+ payload = json.loads((self.task_dir / "script.json").read_text())++ self.assertEqual(payload["params"]["video_subject"], "test subject")+ self.assertEqual(payload["params"]["video_terms"], ["city", "night"])+ self.assertEqual(payload["params"]["video_source"], "pexels")++ def test_patch_missing_script_is_non_blocking(self):+ """独立调用素材下载时没有任务清单,应静默跳过而不是创建残缺 JSON。"""+ updated = task_artifacts.patch_script_data(+ "standalone",+ material_sources=[],+ )++ self.assertFalse(updated)+ self.assertFalse((self.task_dir / "script.json").exists())++ def test_patch_invalid_script_returns_false_without_overwrite(self):+ """历史 JSON 损坏时必须保留原文件、记录错误,并允许视频主流程继续。"""+ target = self.task_dir / "script.json"+ target.write_text("{invalid-json", encoding="utf-8")++ with patch.object(task_artifacts.logger, "warning") as warning:+ updated = task_artifacts.patch_script_data(+ "task-1",+ material_sources=[],+ )++ self.assertFalse(updated)+ self.assertEqual(target.read_text(encoding="utf-8"), "{invalid-json")+ self.assertTrue(warning.called)+++if __name__ == "__main__":+ unittest.main()