Skip to content

ReShot — copy the shot, not the actors. Video → depth map, OpenPose skeleton or canny lines for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Apache-2.0. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system.

docs: README as a pitch — problem first, tagline 'Copy the shot, not the actors', positioning, use cases; social preview card

wangfeng3fef94d8c685
diff --git a/README.md b/README.md
index a8cade3..9ac9a46 100644
--- a/README.md
+++ b/README.md
@@ -1,8 +1,8 @@
-<p align="center"><img src="docs/demo.gif" width="560" alt="left: source video, right: depth blockout produced by depthcat"></p>
-
<h1 align="center">depthcat</h1>
+<p align="center"><b>Copy the shot, not the actors.</b></p>
+<p align="center">Turn a reference video into a depth blockout your video model can follow — same staging, same camera, your cast, your style.</p>
-<p align="center"><b>Turn any video into a depth blockout — the control signal that lets a video model copy a shot's staging and camera, without copying its faces, clothes or style.</b></p>
+<p align="center"><img src="docs/demo.gif" width="560" alt="left: reference video, right: the depth blockout depthcat makes from it"></p>
<p align="center">
<a href="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml"><img src="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml/badge.svg" alt="ci"></a>
@@ -10,115 +10,113 @@
<img src="https://img.shields.io/badge/python-3.10%20%7C%203.12-blue" alt="python">
<img src="https://img.shields.io/badge/runs%20on-CUDA%20%C2%B7%20Apple%20Silicon%20%C2%B7%20CPU-555" alt="platforms">
</p>
-
<p align="center"><a href="README.zh-CN.md">中文</a> · <a href="docs/USAGE.md">User guide</a> · <a href="https://huggingface.co/spaces/maosika/depthcat">Hugging Face</a> · <a href="CHANGELOG.md">Changelog</a></p>
---
-```bash
-pip install git+https://github.com/maosika-ai/depthcat
-depthcat reference.mp4 -o blockout.mp4 --target h3
-```
-
-That is the whole workflow. Feed `blockout.mp4` to a depth-conditioned video model
-(MiniMax H3 Fun ControlNet, Wan VACE, …) together with your own prompt, and the
-generated clip follows the reference's **blocking and camera** while its **look** comes
-entirely from your prompt.
-
-## Why a depth blockout
+## The problem
-A reference video carries two kinds of information: *how the shot is staged* (who is
-where, how big, how they move, how the camera moves) and *what it looks like* (faces,
-wardrobe, lighting, style). Depth keeps the first and discards the second. That makes it
+You found a clip with exactly the blocking you want — she walks in from the door, he
+struggles in the background, the camera pushes in slowly. Now try to get that out of a
+video model:
-- **the cleanest control signal** for "shoot my script like this reference",
-- **a compliance layer** — no likeness, no wardrobe, no brand leaves the reference,
-- **model-agnostic** — every depth ControlNet was trained on exactly this kind of image.
+- **Prompts don't do staging.** "Woman in foreground, man behind her, slow push-in" comes
+ out different every run.
+- **Feeding the clip itself copies too much.** Faces, wardrobe, lighting and style come
+ along — and real-people footage gets rejected by most models' content checks.
-depthcat produces that signal correctly and reproducibly, with the details that ad-hoc
-pipelines get wrong already handled.
+## What depthcat does
-## What it gets right
-
-| | depthcat | typical "run a depth model per frame" script |
-|---|---|---|
-| Temporal consistency | native video model (32-frame windows, overlap, cross-window alignment) | flickers |
-| Normalisation | once over the whole clip | per frame → static walls "breathe" |
-| Resolution | inference at model resolution, output upscaled frame by frame | full-res float depth for every frame → 8–20 GB RAM |
-| Frame rate | picks frames by timestamp (30 → 24 really is 24) | integer stride → still 30 |
-| Output size | crops to the generator's multiple (32 for H3) | pads → black border reads as a wall |
-| Encoding | x264 CRF 12, key frame every 2 s | default CRF → banding → jitter in the generated clip |
-| Licence | Apache-2.0 model by default, non-commercial variants opt-in with a warning | whatever was on disk |
-| Safety | RAM estimate before running; refuses with a concrete `--max-frames` | swaps the machine to death |
-
-## Install
+It strips the reference down to the one thing you actually wanted from it: **where things
+are, how big they are, how they and the camera move** — as a near-white / far-black depth
+video, the format depth ControlNets are trained on.
```bash
-pip install git+https://github.com/maosika-ai/depthcat # needs ffmpeg on PATH
-pip install "depthcat[ffmpeg] @ git+https://github.com/maosika-ai/depthcat" # bundles an ffmpeg binary
+pip install git+https://github.com/maosika-ai/depthcat
+depthcat reference.mp4 -o blockout.mp4 --target h3
```
-Weights (111 MB, Apache-2.0) download from Hugging Face on first run.
-Behind the Great Firewall: `export HF_ENDPOINT=https://hf-mirror.com`.
+```
+reference.mp4 ──▶ depthcat ──▶ blockout.mp4 ──┐
+ ├──▶ video model (depth ControlNet) ──▶ your clip
+your prompt: cast, wardrobe, lighting, style ──┘
+```
-## Use
+Feed `blockout.mp4` to MiniMax H3 Fun ControlNet, Wan VACE or any depth-conditioned
+generator, describe the *look* in the prompt, and the result keeps the reference's
+staging and camera with none of its identity.
-```bash
-depthcat in.mp4 -o out.mp4 # keep fps and size, near = white
-depthcat in.mp4 -o out.mp4 --target h3 # MiniMax H3 ControlNet: 24 fps, ×32, ≤ 15 s
-depthcat in.mp4 -o out.mp4 --npz d.npz --metrics run.json
-```
+## Made for
-```python
-from pathlib import Path
-from depthcat import RunConfig, run
+- **Short-drama and film teams** — reuse a shot's blocking with your own characters.
+- **Ads and product video** — put the camera move of a reference spot on your product.
+- **Anyone iterating on a scene** — lock staging once, change cast and style freely.
-result = run(RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3"))
-print(result.ms_per_frame, result.peak_rss_bytes)
-```
+## Why not …
+
+| alternative | what you give up |
+|---|---|
+| Depth capture inside a hosted video app | Works only in that app; nothing to script, batch, or plug into your own pipeline. |
+| A single-image depth node per frame in ComfyUI | Flickers — each frame is normalised on its own. You also build fps, size and encoding yourself. |
+| A hosted depth-video API | Pay per second and upload your footage. Fine for a test, not for a pipeline. |
+| **depthcat** | A CLI and a Python function. Video-native model, whole-clip normalisation, generator presets, Apache-2.0 end to end. Runs on an 8 GB GPU or a Mac. |
-Every option, preset, exit code and recipe: **[docs/USAGE.md](docs/USAGE.md)**.
+## Details that matter
-## Targets
+- **Temporal consistency from the model, not a blur.** Video Depth Anything runs 32-frame
+ windows with overlap and aligns them; single-image models cannot do this.
+- **One normalisation for the whole clip.** Per-frame min/max makes a static wall change
+ brightness when someone walks toward the camera; the generator reads that as motion.
+- **Inference at model resolution, output at yours.** A 12 s 720p clip needs ~4 GB of RAM,
+ not 8–20 GB. The RAM need is estimated before running and refused if it won't fit.
+- **Frames by timestamp, size by crop, quality by CRF.** 30 → 24 fps really gives 24; frame
+ size is cropped to the generator's multiple, never padded; x264 CRF 12 so the control
+ signal has no banding.
+- **Licence you can ship with.** Apache-2.0 model by default; the non-commercial variants
+ are opt-in and say so out loud.
+
+## Presets
| `--target` | fps | frame size | max | for |
|---|---|---|---|---|
-| `none` | source | even | – | anything |
| `h3` | 24 | multiple of 32 | 15 s | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) |
-| `wan` | 16 | multiple of 16 | – | Wan 2.1 VACE (values from the docs; not yet verified end-to-end) |
+| `wan` | 16 | multiple of 16 | – | Wan 2.1 VACE (values from the docs, not yet verified end-to-end) |
+| `none` | source | even | – | anything |
-## Performance (measured, Small model, 294 frames @ 736×1280)
+## Measured
-| device | precision | ms / frame | host RAM peak |
+| device | precision | ms / frame | host RAM |
|---|---|---|---|
| RTX 4090, torch 2.8 cu128 | fp16 | **70** (31 with `--input-size 364`) | 4.5 GB |
| Apple M2 Max, torch 2.9.1 | fp32 | ~500 | ~4 GB |
-| Apple M2 Max, torch 2.6.0 | fp32 | ~1700 | — |
-| CPU (M2 Max) | fp32 | ~1850 | — |
-fp16 is enabled on CUDA only: on MPS it is pathologically slow. Upgrade torch to ≥ 2.9 on Apple Silicon.
+294 frames at 736×1280, Small model. Details and more platforms in the [user guide](docs/USAGE.md).
+
+## Python
-## Models and licences
+```python
+from pathlib import Path
+from depthcat import RunConfig, run
+
+result = run(RunConfig(input=Path("reference.mp4"), output=Path("blockout.mp4"), target="h3"))
+print(result.frames, result.ms_per_frame)
+```
-| weights | licence | commercial |
-|---|---|---|
-| Video-Depth-Anything-**Small** (default) | Apache-2.0 | ✅ |
-| Video-Depth-Anything-Base / -Large | CC-BY-NC-4.0 | ❌ opt-in with `--model`, prints a warning |
+## Licence
-depthcat's own code is Apache-2.0. The model code from
-[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything)
-(CVPR 2025) is vendored under `depthcat/third_party/` — see [NOTICE](NOTICE).
+Code: Apache-2.0. Default weights: Video-Depth-Anything-**Small**, Apache-2.0
+([DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything),
+CVPR 2025; model code vendored under `depthcat/third_party/`, see [NOTICE](NOTICE)).
+Base/Large weights are CC-BY-NC-4.0 and only load when asked for explicitly.
## Roadmap
-- `--people-only`: keep person silhouettes, flatten the background (SAM 2)
-- `--also pose,canny,normal`: the other MiniMax H3 Fun ControlNet inputs from one pass
-- Docker image, PyPI package, ComfyUI example workflow
-- Runnable Hugging Face Space (the current one is a landing page; a compute Space needs a paid plan)
+`--people-only` (SAM 2 silhouettes) · pose / canny / normal outputs from the same pass ·
+PyPI · Docker · ComfyUI example workflow · a runnable Hugging Face Space
## Contributing
-Issues and PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Run `ruff check` and
-`pytest` before pushing; the CI also runs the real model on CPU.
+Issues and PRs welcome — [CONTRIBUTING.md](CONTRIBUTING.md). `ruff check` and `pytest`
+run in CI on Ubuntu and Windows, plus the real model on CPU.
-Built at [Maoska (猫斯卡)](https://www.maosika.com), an AI short-drama studio.
+<p align="center"><sub>Built at <a href="https://www.maosika.com">Maoska 猫斯卡</a>, a studio that ships AI short drama every day.</sub></p>
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 5ce63b8..11f745a 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -1,8 +1,8 @@
-<p align="center"><img src="docs/demo.gif" width="560" alt="左:原片;右:depthcat 抽出的深度白模"></p>
-
<h1 align="center">depthcat</h1>
+<p align="center"><b>复制走位,不复制演员。</b></p>
+<p align="center">把参考视频抽成视频模型能跟着走的深度白模——走位一样、镜头一样,人是你的,画风是你的。</p>
-<p align="center"><b>把任意视频抽成深度白模——让视频模型复制一段参考片的走位和运镜,却不带走它的人脸、服装和画风。</b></p>
+<p align="center"><img src="docs/demo.gif" width="560" alt="左:参考片;右:depthcat 抽出的深度白模"></p>
<p align="center">
<a href="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml"><img src="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml/badge.svg" alt="ci"></a>
@@ -10,110 +10,106 @@
<img src="https://img.shields.io/badge/python-3.10%20%7C%203.12-blue" alt="python">
<img src="https://img.shields.io/badge/%E8%BF%90%E8%A1%8C%E4%BA%8E-CUDA%20%C2%B7%20Apple%20Silicon%20%C2%B7%20CPU-555" alt="platforms">
</p>
-
<p align="center"><a href="README.md">English</a> · <a href="docs/USAGE.zh-CN.md">使用手册</a> · <a href="https://huggingface.co/spaces/maosika/depthcat">Hugging Face</a> · <a href="CHANGELOG.md">更新日志</a></p>
---
-```bash
-pip install git+https://github.com/maosika-ai/depthcat
-depthcat 参考片.mp4 -o 白模.mp4 --target h3
-```
-
-整个流程就这两行。把 `白模.mp4` 连同你自己的提示词一起交给带深度控制的视频模型
-(MiniMax H3 Fun ControlNet、Wan VACE……),生成出来的片子**走位和镜头跟参考片**,
-**长相和画风全听你的提示词**。就是 LibTV「深度动作捕捉」那种输出,但你自己掌握、可商用。
-
-## 为什么是深度白模
+## 你遇到的问题
-一段参考视频里有两种信息:**怎么拍的**(谁站哪、多大、怎么动、镜头怎么动)和
-**长什么样**(人脸、衣服、光线、风格)。深度图只保留前者、丢掉后者。所以它是
+你找到一段走位、运镜都对的片子——女主从门口走进来,男人在后面挣扎,镜头缓缓推近。
+现在想让视频模型照这个拍:
-- **最干净的控制信号**——"按这段参考片的拍法拍我的剧本"
-- **一层合规隔离**——人脸、服装、品牌一样都不会从参考片带出来
-- **模型无关**——所有深度 ControlNet 训练时吃的就是这种图
+- **提示词管不了走位。**"女主在前景、男人在她身后、镜头慢推",写一百遍,每次出来都不一样。
+- **直接喂参考片,带走的太多。** 人脸、服装、光线、画风全跟过来;真人素材还大概率过不了模型的审核。
-depthcat 负责把这个信号做对、做稳,把那些"随手跑一下深度模型"必然踩的坑提前处理掉。
+这就是 LibTV「深度动作捕捉」在做的事——把参考片抽成深度白模再去生成。depthcat
+把这一步做成了你自己电脑上的一条命令:开源、可商用、能进你自己的流水线。
-## 做对了什么
+## depthcat 做什么
-| | depthcat | 常见的"逐帧跑深度模型"脚本 |
-|---|---|---|
-| 时序一致性 | 原生视频模型(32 帧窗口、重叠、跨窗口对齐) | 闪烁 |
-| 归一化 | 整段视频只做一次 | 逐帧 → 静止的墙在"呼吸" |
-| 分辨率 | 按模型分辨率推理,8 位结果逐帧放大 | 每帧存全分辨率浮点深度 → 8–20 GB 内存 |
-| 帧率 | 按时间戳选帧(30→24 真的是 24) | 整数步长 → 还是 30 |
-| 输出尺寸 | 裁到生成模型要的倍数(H3 是 32) | 补黑边 → 模型当成一堵墙 |
-| 编码 | x264 CRF 12,2 秒一个关键帧 | 默认 CRF → 色带 → 出片抖动 |
-| 许可 | 默认 Apache-2.0 权重,非商用权重需显式指定并警告 | 硬盘上有啥用啥 |
-| 安全 | 跑前估内存,超预算拒跑并给出 `--max-frames` 建议 | 把机器交换到死 |
-
-## 安装
+把参考片剥到只剩你真正想要的那一层:**谁在哪、多大、怎么动、镜头怎么动**——
+输出近白远黑的深度视频,正是深度 ControlNet 训练时吃的格式。
```bash
-pip install git+https://github.com/maosika-ai/depthcat # 需要 PATH 里有 ffmpeg
-pip install "depthcat[ffmpeg] @ git+https://github.com/maosika-ai/depthcat" # 自带 ffmpeg 二进制
+pip install git+https://github.com/maosika-ai/depthcat
+depthcat 参考片.mp4 -o 白模.mp4 --target h3
```
-首次运行自动下载权重(111 MB,Apache-2.0)。国内:`export HF_ENDPOINT=https://hf-mirror.com`。
+```
+参考片.mp4 ──▶ depthcat ──▶ 白模.mp4 ──┐
+ ├──▶ 视频模型(深度 ControlNet)──▶ 你的片子
+你的提示词:人物、服装、光线、画风 ─────┘
+```
-## 用法
+把 `白模.mp4` 交给 MiniMax H3 Fun ControlNet、Wan VACE 或任何带深度控制的生成模型,
+提示词只写**长相**,出来的片子保留参考片的走位和镜头,不带一点它的身份。
-```bash
-depthcat in.mp4 -o out.mp4 # 保持原 fps / 尺寸,近白远黑
-depthcat in.mp4 -o out.mp4 --target h3 # MiniMax H3 ControlNet:24 fps、边长 32 的倍数、≤15 秒
-depthcat in.mp4 -o out.mp4 --npz d.npz --metrics run.json
-```
+## 给谁用
-```python
-from pathlib import Path
-from depthcat import RunConfig, run
+- **短剧、影视团队** —— 把一段好走位套到自己的角色身上。
+- **广告、产品视频** —— 把参考广告的运镜放到自己的产品上。
+- **反复打磨一个场景的人** —— 走位锁一次,人物和画风随便换。
-result = run(RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3"))
-print(result.ms_per_frame, result.peak_rss_bytes)
-```
+## 为什么不用别的
-全部参数、预设、退出码和实战配方:**[docs/USAGE.zh-CN.md](docs/USAGE.zh-CN.md)**。
+| 替代方案 | 你要放弃什么 |
+|---|---|
+| 某个视频 App 里的深度捕捉 | 只能在那个 App 里用;没法写脚本、批处理、接进自己的流水线。 |
+| ComfyUI 里逐帧跑单图深度节点 | 会闪——每帧各自归一化。fps、尺寸、编码还得自己搭。 |
+| 托管的深度视频 API | 按秒收费,素材要传到别人服务器。测一下行,当流水线不行。 |
+| **depthcat** | 一条命令、一个 Python 函数。视频原生模型、整段归一化、生成模型预设、端到端 Apache-2.0。8 GB 显卡或 Mac 就能跑。 |
-## 目标预设
+## 决定成败的细节
+
+- **时序一致性来自模型,不是靠模糊。** Video Depth Anything 用 32 帧窗口重叠推理并对齐;单图模型做不到。
+- **整段视频只归一化一次。** 逐帧 min/max 会让静止的墙在有人走近镜头时忽明忽暗,生成模型把它读成运动。
+- **按模型分辨率推理,按你要的尺寸输出。** 12 秒 720p 只要约 4 GB 内存,不是 8–20 GB。跑之前先估算,放不下就拒绝。
+- **帧按时间戳选、尺寸靠裁、质量靠 CRF。** 30→24 fps 真的是 24;尺寸裁到生成模型要的倍数、绝不补边;x264 CRF 12,控制信号没有色带。
+- **能随产品一起发的许可。** 默认 Apache-2.0 权重;非商用的变体要显式指定,运行时会明说。
+
+## 预设
| `--target` | fps | 尺寸 | 上限 | 对应 |
|---|---|---|---|---|
-| `none` | 原片 | 偶数 | – | 通用 |
| `h3` | 24 | 32 的倍数 | 15 秒 | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) |
| `wan` | 16 | 16 的倍数 | – | Wan 2.1 VACE(按文档取值,尚未端到端实测) |
+| `none` | 原片 | 偶数 | – | 通用 |
-## 实测性能(Small 模型,294 帧 @ 736×1280)
+## 实测
-| 设备 | 精度 | ms / 帧 | 宿主内存峰值 |
+| 设备 | 精度 | ms / 帧 | 宿主内存 |
|---|---|---|---|
| RTX 4090,torch 2.8 cu128 | fp16 | **70**(`--input-size 364` 时 31) | 4.5 GB |
| Apple M2 Max,torch 2.9.1 | fp32 | 约 500 | 约 4 GB |
-| Apple M2 Max,torch 2.6.0 | fp32 | 约 1700 | — |
-| CPU(M2 Max) | fp32 | 约 1850 | — |
-fp16 只在 CUDA 上开:MPS 上慢到不可用。Apple 芯片请把 torch 升到 ≥ 2.9。
+294 帧 @ 736×1280,Small 模型。更多平台与细节见[使用手册](docs/USAGE.zh-CN.md)。
-## 模型与许可
+安装时国内请先 `export HF_ENDPOINT=https://hf-mirror.com`,权重 111 MB。
+
+## Python
+
+```python
+from pathlib import Path
+from depthcat import RunConfig, run
+
+result = run(RunConfig(input=Path("参考片.mp4"), output=Path("白模.mp4"), target="h3"))
+print(result.frames, result.ms_per_frame)
+```
-| 权重 | 许可 | 商用 |
-|---|---|---|
-| Video-Depth-Anything-**Small**(默认) | Apache-2.0 | ✅ |
-| Video-Depth-Anything-Base / -Large | CC-BY-NC-4.0 | ❌ 需 `--model` 显式指定,运行时警告 |
+## 许可
-depthcat 自身代码 Apache-2.0。模型代码来自
-[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything)
-(CVPR 2025),vendored 在 `depthcat/third_party/`,见 [NOTICE](NOTICE)。
+代码 Apache-2.0。默认权重 Video-Depth-Anything-**Small**,Apache-2.0
+([DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything),
+CVPR 2025;模型代码 vendored 在 `depthcat/third_party/`,见 [NOTICE](NOTICE))。
+Base/Large 权重是 CC-BY-NC-4.0,只在显式指定时加载。
## 路线图
-- `--people-only`:只留人物剪影、背景抹平(SAM 2)
-- `--also pose,canny,normal`:一遍出 MiniMax H3 Fun ControlNet 的其余输入
-- Docker 镜像、PyPI 包、ComfyUI 示例工作流
-- 可在线运行的 Hugging Face Space(现在是静态介绍页;能跑计算的 Space 需要付费档)
+`--people-only`(SAM 2 人物剪影)· 同一遍推理顺带出骨架 / 边缘 / 法线 ·
+PyPI · Docker · ComfyUI 示例工作流 · 可在线运行的 Hugging Face Space
## 参与
-欢迎 issue 和 PR,见 [CONTRIBUTING.md](CONTRIBUTING.md)。推之前跑 `ruff check` 和 `pytest`;CI 还会在 CPU 上真跑一遍模型。
+欢迎 issue 和 PR —— [CONTRIBUTING.md](CONTRIBUTING.md)。CI 在 Ubuntu 和 Windows 上跑 `ruff check` 和 `pytest`,外加 CPU 真模型。
-出品:[猫斯卡](https://www.maosika.com),AI 短剧工作室。
+<p align="center"><sub>出品:<a href="https://www.maosika.com">猫斯卡</a>,每天都在出 AI 短剧的工作室。</sub></p>
diff --git a/docs/social-preview.png b/docs/social-preview.png
new file mode 100644
index 0000000..102944e
Binary files /dev/null and b/docs/social-preview.png differ