ReShot — copy the shot, not the actors. Video → depth map, OpenPose skeleton or canny lines for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Apache-2.0. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system.
Python ai-short-dramaai-videocannycomfyuicontrolnetdepth-anythingdepth-estimationdwposemaosikaminimaxopenposepose-controlnetpose-estimationseedanceshort-dramavideo-generationvideo-to-videowan
docs: README as a pitch — problem first, tagline 'Copy the shot, not the actors', positioning, use cases; social preview card
wangfeng3fef94d8c685
diff --git a/README.md b/README.mdindex a8cade3..9ac9a46 100644--- a/README.md+++ b/README.md@@ -1,8 +1,8 @@-<p align="center"><img src="docs/demo.gif" width="560" alt="left: source video, right: depth blockout produced by depthcat"></p>-<h1 align="center">depthcat</h1>+<p align="center"><b>Copy the shot, not the actors.</b></p>+<p align="center">Turn a reference video into a depth blockout your video model can follow — same staging, same camera, your cast, your style.</p>-<p align="center"><b>Turn any video into a depth blockout — the control signal that lets a video model copy a shot's staging and camera, without copying its faces, clothes or style.</b></p>+<p align="center"><img src="docs/demo.gif" width="560" alt="left: reference video, right: the depth blockout depthcat makes from it"></p><p align="center"><a href="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml"><img src="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml/badge.svg" alt="ci"></a>@@ -10,115 +10,113 @@<img src="https://img.shields.io/badge/python-3.10%20%7C%203.12-blue" alt="python"><img src="https://img.shields.io/badge/runs%20on-CUDA%20%C2%B7%20Apple%20Silicon%20%C2%B7%20CPU-555" alt="platforms"></p>-<p align="center"><a href="README.zh-CN.md">中文</a> · <a href="docs/USAGE.md">User guide</a> · <a href="https://huggingface.co/spaces/maosika/depthcat">Hugging Face</a> · <a href="CHANGELOG.md">Changelog</a></p>----```bash-pip install git+https://github.com/maosika-ai/depthcat-depthcat reference.mp4 -o blockout.mp4 --target h3-```--That is the whole workflow. Feed `blockout.mp4` to a depth-conditioned video model-(MiniMax H3 Fun ControlNet, Wan VACE, …) together with your own prompt, and the-generated clip follows the reference's **blocking and camera** while its **look** comes-entirely from your prompt.--## Why a depth blockout+## The problem-A reference video carries two kinds of information: *how the shot is staged* (who is-where, how big, how they move, how the camera moves) and *what it looks like* (faces,-wardrobe, lighting, style). Depth keeps the first and discards the second. That makes it+You found a clip with exactly the blocking you want — she walks in from the door, he+struggles in the background, the camera pushes in slowly. Now try to get that out of a+video model:-- **the cleanest control signal** for "shoot my script like this reference",-- **a compliance layer** — no likeness, no wardrobe, no brand leaves the reference,-- **model-agnostic** — every depth ControlNet was trained on exactly this kind of image.+- **Prompts don't do staging.** "Woman in foreground, man behind her, slow push-in" comes+ out different every run.+- **Feeding the clip itself copies too much.** Faces, wardrobe, lighting and style come+ along — and real-people footage gets rejected by most models' content checks.-depthcat produces that signal correctly and reproducibly, with the details that ad-hoc-pipelines get wrong already handled.+## What depthcat does-## What it gets right--| | depthcat | typical "run a depth model per frame" script |-|---|---|---|-| Temporal consistency | native video model (32-frame windows, overlap, cross-window alignment) | flickers |-| Normalisation | once over the whole clip | per frame → static walls "breathe" |-| Resolution | inference at model resolution, output upscaled frame by frame | full-res float depth for every frame → 8–20 GB RAM |-| Frame rate | picks frames by timestamp (30 → 24 really is 24) | integer stride → still 30 |-| Output size | crops to the generator's multiple (32 for H3) | pads → black border reads as a wall |-| Encoding | x264 CRF 12, key frame every 2 s | default CRF → banding → jitter in the generated clip |-| Licence | Apache-2.0 model by default, non-commercial variants opt-in with a warning | whatever was on disk |-| Safety | RAM estimate before running; refuses with a concrete `--max-frames` | swaps the machine to death |--## Install+It strips the reference down to the one thing you actually wanted from it: **where things+are, how big they are, how they and the camera move** — as a near-white / far-black depth+video, the format depth ControlNets are trained on.```bash-pip install git+https://github.com/maosika-ai/depthcat # needs ffmpeg on PATH-pip install "depthcat[ffmpeg] @ git+https://github.com/maosika-ai/depthcat" # bundles an ffmpeg binary+pip install git+https://github.com/maosika-ai/depthcat+depthcat reference.mp4 -o blockout.mp4 --target h3```-Weights (111 MB, Apache-2.0) download from Hugging Face on first run.-Behind the Great Firewall: `export HF_ENDPOINT=https://hf-mirror.com`.+```+reference.mp4 ──▶ depthcat ──▶ blockout.mp4 ──┐+ ├──▶ video model (depth ControlNet) ──▶ your clip+your prompt: cast, wardrobe, lighting, style ──┘+```-## Use+Feed `blockout.mp4` to MiniMax H3 Fun ControlNet, Wan VACE or any depth-conditioned+generator, describe the *look* in the prompt, and the result keeps the reference's+staging and camera with none of its identity.-```bash-depthcat in.mp4 -o out.mp4 # keep fps and size, near = white-depthcat in.mp4 -o out.mp4 --target h3 # MiniMax H3 ControlNet: 24 fps, ×32, ≤ 15 s-depthcat in.mp4 -o out.mp4 --npz d.npz --metrics run.json-```+## Made for-```python-from pathlib import Path-from depthcat import RunConfig, run+- **Short-drama and film teams** — reuse a shot's blocking with your own characters.+- **Ads and product video** — put the camera move of a reference spot on your product.+- **Anyone iterating on a scene** — lock staging once, change cast and style freely.-result = run(RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3"))-print(result.ms_per_frame, result.peak_rss_bytes)-```+## Why not …++| alternative | what you give up |+|---|---|+| Depth capture inside a hosted video app | Works only in that app; nothing to script, batch, or plug into your own pipeline. |+| A single-image depth node per frame in ComfyUI | Flickers — each frame is normalised on its own. You also build fps, size and encoding yourself. |+| A hosted depth-video API | Pay per second and upload your footage. Fine for a test, not for a pipeline. |+| **depthcat** | A CLI and a Python function. Video-native model, whole-clip normalisation, generator presets, Apache-2.0 end to end. Runs on an 8 GB GPU or a Mac. |-Every option, preset, exit code and recipe: **[docs/USAGE.md](docs/USAGE.md)**.+## Details that matter-## Targets+- **Temporal consistency from the model, not a blur.** Video Depth Anything runs 32-frame+ windows with overlap and aligns them; single-image models cannot do this.+- **One normalisation for the whole clip.** Per-frame min/max makes a static wall change+ brightness when someone walks toward the camera; the generator reads that as motion.+- **Inference at model resolution, output at yours.** A 12 s 720p clip needs ~4 GB of RAM,+ not 8–20 GB. The RAM need is estimated before running and refused if it won't fit.+- **Frames by timestamp, size by crop, quality by CRF.** 30 → 24 fps really gives 24; frame+ size is cropped to the generator's multiple, never padded; x264 CRF 12 so the control+ signal has no banding.+- **Licence you can ship with.** Apache-2.0 model by default; the non-commercial variants+ are opt-in and say so out loud.++## Presets| `--target` | fps | frame size | max | for ||---|---|---|---|---|-| `none` | source | even | – | anything || `h3` | 24 | multiple of 32 | 15 s | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) |-| `wan` | 16 | multiple of 16 | – | Wan 2.1 VACE (values from the docs; not yet verified end-to-end) |+| `wan` | 16 | multiple of 16 | – | Wan 2.1 VACE (values from the docs, not yet verified end-to-end) |+| `none` | source | even | – | anything |-## Performance (measured, Small model, 294 frames @ 736×1280)+## Measured-| device | precision | ms / frame | host RAM peak |+| device | precision | ms / frame | host RAM ||---|---|---|---|| RTX 4090, torch 2.8 cu128 | fp16 | **70** (31 with `--input-size 364`) | 4.5 GB || Apple M2 Max, torch 2.9.1 | fp32 | ~500 | ~4 GB |-| Apple M2 Max, torch 2.6.0 | fp32 | ~1700 | — |-| CPU (M2 Max) | fp32 | ~1850 | — |-fp16 is enabled on CUDA only: on MPS it is pathologically slow. Upgrade torch to ≥ 2.9 on Apple Silicon.+294 frames at 736×1280, Small model. Details and more platforms in the [user guide](docs/USAGE.md).++## Python-## Models and licences+```python+from pathlib import Path+from depthcat import RunConfig, run++result = run(RunConfig(input=Path("reference.mp4"), output=Path("blockout.mp4"), target="h3"))+print(result.frames, result.ms_per_frame)+```-| weights | licence | commercial |-|---|---|---|-| Video-Depth-Anything-**Small** (default) | Apache-2.0 | ✅ |-| Video-Depth-Anything-Base / -Large | CC-BY-NC-4.0 | ❌ opt-in with `--model`, prints a warning |+## Licence-depthcat's own code is Apache-2.0. The model code from-[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything)-(CVPR 2025) is vendored under `depthcat/third_party/` — see [NOTICE](NOTICE).+Code: Apache-2.0. Default weights: Video-Depth-Anything-**Small**, Apache-2.0+([DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything),+CVPR 2025; model code vendored under `depthcat/third_party/`, see [NOTICE](NOTICE)).+Base/Large weights are CC-BY-NC-4.0 and only load when asked for explicitly.## Roadmap-- `--people-only`: keep person silhouettes, flatten the background (SAM 2)-- `--also pose,canny,normal`: the other MiniMax H3 Fun ControlNet inputs from one pass-- Docker image, PyPI package, ComfyUI example workflow-- Runnable Hugging Face Space (the current one is a landing page; a compute Space needs a paid plan)+`--people-only` (SAM 2 silhouettes) · pose / canny / normal outputs from the same pass ·+PyPI · Docker · ComfyUI example workflow · a runnable Hugging Face Space## Contributing-Issues and PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Run `ruff check` and-`pytest` before pushing; the CI also runs the real model on CPU.+Issues and PRs welcome — [CONTRIBUTING.md](CONTRIBUTING.md). `ruff check` and `pytest`+run in CI on Ubuntu and Windows, plus the real model on CPU.-Built at [Maoska (猫斯卡)](https://www.maosika.com), an AI short-drama studio.+<p align="center"><sub>Built at <a href="https://www.maosika.com">Maoska 猫斯卡</a>, a studio that ships AI short drama every day.</sub></p>diff --git a/README.zh-CN.md b/README.zh-CN.mdindex 5ce63b8..11f745a 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -1,8 +1,8 @@-<p align="center"><img src="docs/demo.gif" width="560" alt="左:原片;右:depthcat 抽出的深度白模"></p>-<h1 align="center">depthcat</h1>+<p align="center"><b>复制走位,不复制演员。</b></p>+<p align="center">把参考视频抽成视频模型能跟着走的深度白模——走位一样、镜头一样,人是你的,画风是你的。</p>-<p align="center"><b>把任意视频抽成深度白模——让视频模型复制一段参考片的走位和运镜,却不带走它的人脸、服装和画风。</b></p>+<p align="center"><img src="docs/demo.gif" width="560" alt="左:参考片;右:depthcat 抽出的深度白模"></p><p align="center"><a href="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml"><img src="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml/badge.svg" alt="ci"></a>@@ -10,110 +10,106 @@<img src="https://img.shields.io/badge/python-3.10%20%7C%203.12-blue" alt="python"><img src="https://img.shields.io/badge/%E8%BF%90%E8%A1%8C%E4%BA%8E-CUDA%20%C2%B7%20Apple%20Silicon%20%C2%B7%20CPU-555" alt="platforms"></p>-<p align="center"><a href="README.md">English</a> · <a href="docs/USAGE.zh-CN.md">使用手册</a> · <a href="https://huggingface.co/spaces/maosika/depthcat">Hugging Face</a> · <a href="CHANGELOG.md">更新日志</a></p>----```bash-pip install git+https://github.com/maosika-ai/depthcat-depthcat 参考片.mp4 -o 白模.mp4 --target h3-```--整个流程就这两行。把 `白模.mp4` 连同你自己的提示词一起交给带深度控制的视频模型-(MiniMax H3 Fun ControlNet、Wan VACE……),生成出来的片子**走位和镜头跟参考片**,-**长相和画风全听你的提示词**。就是 LibTV「深度动作捕捉」那种输出,但你自己掌握、可商用。--## 为什么是深度白模+## 你遇到的问题-一段参考视频里有两种信息:**怎么拍的**(谁站哪、多大、怎么动、镜头怎么动)和-**长什么样**(人脸、衣服、光线、风格)。深度图只保留前者、丢掉后者。所以它是+你找到一段走位、运镜都对的片子——女主从门口走进来,男人在后面挣扎,镜头缓缓推近。+现在想让视频模型照这个拍:-- **最干净的控制信号**——"按这段参考片的拍法拍我的剧本"-- **一层合规隔离**——人脸、服装、品牌一样都不会从参考片带出来-- **模型无关**——所有深度 ControlNet 训练时吃的就是这种图+- **提示词管不了走位。**"女主在前景、男人在她身后、镜头慢推",写一百遍,每次出来都不一样。+- **直接喂参考片,带走的太多。** 人脸、服装、光线、画风全跟过来;真人素材还大概率过不了模型的审核。-depthcat 负责把这个信号做对、做稳,把那些"随手跑一下深度模型"必然踩的坑提前处理掉。+这就是 LibTV「深度动作捕捉」在做的事——把参考片抽成深度白模再去生成。depthcat+把这一步做成了你自己电脑上的一条命令:开源、可商用、能进你自己的流水线。-## 做对了什么+## depthcat 做什么-| | depthcat | 常见的"逐帧跑深度模型"脚本 |-|---|---|---|-| 时序一致性 | 原生视频模型(32 帧窗口、重叠、跨窗口对齐) | 闪烁 |-| 归一化 | 整段视频只做一次 | 逐帧 → 静止的墙在"呼吸" |-| 分辨率 | 按模型分辨率推理,8 位结果逐帧放大 | 每帧存全分辨率浮点深度 → 8–20 GB 内存 |-| 帧率 | 按时间戳选帧(30→24 真的是 24) | 整数步长 → 还是 30 |-| 输出尺寸 | 裁到生成模型要的倍数(H3 是 32) | 补黑边 → 模型当成一堵墙 |-| 编码 | x264 CRF 12,2 秒一个关键帧 | 默认 CRF → 色带 → 出片抖动 |-| 许可 | 默认 Apache-2.0 权重,非商用权重需显式指定并警告 | 硬盘上有啥用啥 |-| 安全 | 跑前估内存,超预算拒跑并给出 `--max-frames` 建议 | 把机器交换到死 |--## 安装+把参考片剥到只剩你真正想要的那一层:**谁在哪、多大、怎么动、镜头怎么动**——+输出近白远黑的深度视频,正是深度 ControlNet 训练时吃的格式。```bash-pip install git+https://github.com/maosika-ai/depthcat # 需要 PATH 里有 ffmpeg-pip install "depthcat[ffmpeg] @ git+https://github.com/maosika-ai/depthcat" # 自带 ffmpeg 二进制+pip install git+https://github.com/maosika-ai/depthcat+depthcat 参考片.mp4 -o 白模.mp4 --target h3```-首次运行自动下载权重(111 MB,Apache-2.0)。国内:`export HF_ENDPOINT=https://hf-mirror.com`。+```+参考片.mp4 ──▶ depthcat ──▶ 白模.mp4 ──┐+ ├──▶ 视频模型(深度 ControlNet)──▶ 你的片子+你的提示词:人物、服装、光线、画风 ─────┘+```-## 用法+把 `白模.mp4` 交给 MiniMax H3 Fun ControlNet、Wan VACE 或任何带深度控制的生成模型,+提示词只写**长相**,出来的片子保留参考片的走位和镜头,不带一点它的身份。-```bash-depthcat in.mp4 -o out.mp4 # 保持原 fps / 尺寸,近白远黑-depthcat in.mp4 -o out.mp4 --target h3 # MiniMax H3 ControlNet:24 fps、边长 32 的倍数、≤15 秒-depthcat in.mp4 -o out.mp4 --npz d.npz --metrics run.json-```+## 给谁用-```python-from pathlib import Path-from depthcat import RunConfig, run+- **短剧、影视团队** —— 把一段好走位套到自己的角色身上。+- **广告、产品视频** —— 把参考广告的运镜放到自己的产品上。+- **反复打磨一个场景的人** —— 走位锁一次,人物和画风随便换。-result = run(RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3"))-print(result.ms_per_frame, result.peak_rss_bytes)-```+## 为什么不用别的-全部参数、预设、退出码和实战配方:**[docs/USAGE.zh-CN.md](docs/USAGE.zh-CN.md)**。+| 替代方案 | 你要放弃什么 |+|---|---|+| 某个视频 App 里的深度捕捉 | 只能在那个 App 里用;没法写脚本、批处理、接进自己的流水线。 |+| ComfyUI 里逐帧跑单图深度节点 | 会闪——每帧各自归一化。fps、尺寸、编码还得自己搭。 |+| 托管的深度视频 API | 按秒收费,素材要传到别人服务器。测一下行,当流水线不行。 |+| **depthcat** | 一条命令、一个 Python 函数。视频原生模型、整段归一化、生成模型预设、端到端 Apache-2.0。8 GB 显卡或 Mac 就能跑。 |-## 目标预设+## 决定成败的细节++- **时序一致性来自模型,不是靠模糊。** Video Depth Anything 用 32 帧窗口重叠推理并对齐;单图模型做不到。+- **整段视频只归一化一次。** 逐帧 min/max 会让静止的墙在有人走近镜头时忽明忽暗,生成模型把它读成运动。+- **按模型分辨率推理,按你要的尺寸输出。** 12 秒 720p 只要约 4 GB 内存,不是 8–20 GB。跑之前先估算,放不下就拒绝。+- **帧按时间戳选、尺寸靠裁、质量靠 CRF。** 30→24 fps 真的是 24;尺寸裁到生成模型要的倍数、绝不补边;x264 CRF 12,控制信号没有色带。+- **能随产品一起发的许可。** 默认 Apache-2.0 权重;非商用的变体要显式指定,运行时会明说。++## 预设| `--target` | fps | 尺寸 | 上限 | 对应 ||---|---|---|---|---|-| `none` | 原片 | 偶数 | – | 通用 || `h3` | 24 | 32 的倍数 | 15 秒 | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) || `wan` | 16 | 16 的倍数 | – | Wan 2.1 VACE(按文档取值,尚未端到端实测) |+| `none` | 原片 | 偶数 | – | 通用 |-## 实测性能(Small 模型,294 帧 @ 736×1280)+## 实测-| 设备 | 精度 | ms / 帧 | 宿主内存峰值 |+| 设备 | 精度 | ms / 帧 | 宿主内存 ||---|---|---|---|| RTX 4090,torch 2.8 cu128 | fp16 | **70**(`--input-size 364` 时 31) | 4.5 GB || Apple M2 Max,torch 2.9.1 | fp32 | 约 500 | 约 4 GB |-| Apple M2 Max,torch 2.6.0 | fp32 | 约 1700 | — |-| CPU(M2 Max) | fp32 | 约 1850 | — |-fp16 只在 CUDA 上开:MPS 上慢到不可用。Apple 芯片请把 torch 升到 ≥ 2.9。+294 帧 @ 736×1280,Small 模型。更多平台与细节见[使用手册](docs/USAGE.zh-CN.md)。-## 模型与许可+安装时国内请先 `export HF_ENDPOINT=https://hf-mirror.com`,权重 111 MB。++## Python++```python+from pathlib import Path+from depthcat import RunConfig, run++result = run(RunConfig(input=Path("参考片.mp4"), output=Path("白模.mp4"), target="h3"))+print(result.frames, result.ms_per_frame)+```-| 权重 | 许可 | 商用 |-|---|---|---|-| Video-Depth-Anything-**Small**(默认) | Apache-2.0 | ✅ |-| Video-Depth-Anything-Base / -Large | CC-BY-NC-4.0 | ❌ 需 `--model` 显式指定,运行时警告 |+## 许可-depthcat 自身代码 Apache-2.0。模型代码来自-[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything)-(CVPR 2025),vendored 在 `depthcat/third_party/`,见 [NOTICE](NOTICE)。+代码 Apache-2.0。默认权重 Video-Depth-Anything-**Small**,Apache-2.0+([DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything),+CVPR 2025;模型代码 vendored 在 `depthcat/third_party/`,见 [NOTICE](NOTICE))。+Base/Large 权重是 CC-BY-NC-4.0,只在显式指定时加载。## 路线图-- `--people-only`:只留人物剪影、背景抹平(SAM 2)-- `--also pose,canny,normal`:一遍出 MiniMax H3 Fun ControlNet 的其余输入-- Docker 镜像、PyPI 包、ComfyUI 示例工作流-- 可在线运行的 Hugging Face Space(现在是静态介绍页;能跑计算的 Space 需要付费档)+`--people-only`(SAM 2 人物剪影)· 同一遍推理顺带出骨架 / 边缘 / 法线 ·+PyPI · Docker · ComfyUI 示例工作流 · 可在线运行的 Hugging Face Space## 参与-欢迎 issue 和 PR,见 [CONTRIBUTING.md](CONTRIBUTING.md)。推之前跑 `ruff check` 和 `pytest`;CI 还会在 CPU 上真跑一遍模型。+欢迎 issue 和 PR —— [CONTRIBUTING.md](CONTRIBUTING.md)。CI 在 Ubuntu 和 Windows 上跑 `ruff check` 和 `pytest`,外加 CPU 真模型。-出品:[猫斯卡](https://www.maosika.com),AI 短剧工作室。+<p align="center"><sub>出品:<a href="https://www.maosika.com">猫斯卡</a>,每天都在出 AI 短剧的工作室。</sub></p>diff --git a/docs/social-preview.png b/docs/social-preview.pngnew file mode 100644index 0000000..102944eBinary files /dev/null and b/docs/social-preview.png differ