ReShot — copy the shot, not the actors. Video → depth map, OpenPose skeleton or canny lines for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Apache-2.0. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system.
Python ai-short-dramaai-videocannycomfyuicontrolnetdepth-anythingdepth-estimationdwposemaosikaminimaxopenposepose-controlnetpose-estimationseedanceshort-dramavideo-generationvideo-to-videowan
docs: rewrite README (en/zh) as a project introduction; add user guides, CONTRIBUTING, issue templates
wangfeng52bbba1ef482
diff --git a/.github/ISSUE_TEMPLATE/bug.md b/.github/ISSUE_TEMPLATE/bug.mdnew file mode 100644index 0000000..8fedac8--- /dev/null+++ b/.github/ISSUE_TEMPLATE/bug.md@@ -0,0 +1,19 @@+---+name: Bug+about: Something is wrong+---++**Command**++```+depthcat … -v+```++**stderr output**++```+```++**Environment**: OS · GPU/CPU · torch version · depthcat version (`depthcat --version`)++**Clip** (optional): link or description (resolution, fps, length)diff --git a/.github/ISSUE_TEMPLATE/feature.md b/.github/ISSUE_TEMPLATE/feature.mdnew file mode 100644index 0000000..9d59c56--- /dev/null+++ b/.github/ISSUE_TEMPLATE/feature.md@@ -0,0 +1,8 @@+---+name: Feature+about: Something depthcat should do+---++**What** (one sentence)++**Why** — which generator / workflow needs it, and what happens today without itdiff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdnew file mode 100644index 0000000..ea055e4--- /dev/null+++ b/CONTRIBUTING.md@@ -0,0 +1,35 @@+# Contributing++Thanks for looking. Small, focused PRs are easiest to review.++## Setup++```bash+git clone [email protected]:maosika-ai/depthcat.git && cd depthcat+python -m venv .venv && . .venv/bin/activate+pip install torch torchvision # pick the build for your machine+pip install -e ".[ffmpeg,dev]"+```++## Before you push++```bash+ruff check depthcat tests && ruff format depthcat tests+pytest -q # ~30 tests, no model, ~2 s+RUN_MODEL_TESTS=1 pytest -q tests/test_model_smoke.py # optional: real model on CPU, downloads 111 MB+```++CI runs the same on Ubuntu and Windows, plus the model smoke on CPU.++## Ground rules++- `depthcat/third_party/` is vendored upstream code — do not edit it; fix things in our layer.+- User-facing failures raise a `DepthcatError` subclass with a concrete fix in the message.+- Anything that changes output (normalisation, encoding, presets) needs a test and a+ CHANGELOG line, and the README/USAGE tables updated in **both** languages.+- Performance claims in docs must come from a measurement; say where it was measured.++## Reporting a bug++Include: OS, GPU/CPU, `python -c "import torch;print(torch.__version__)"`, the exact command,+and the stderr output with `-v`. If a clip reproduces it and you can share it, link it.diff --git a/README.md b/README.mdindex 77e36c2..0bab3c4 100644--- a/README.md+++ b/README.md@@ -1,110 +1,124 @@-# depthcat+<p align="center"><img src="docs/demo.gif" width="560" alt="left: source video, right: depth blockout produced by depthcat"></p>-[](https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml) [](LICENSE)+<h1 align="center">depthcat</h1>-Turn any video into a **depth "blockout" video** — the grey near-white / far-black-clip that video-generation ControlNets (MiniMax H3 Fun ControlNet, Wan VACE, …) use to-copy the *staging and camera* of a reference shot without copying its faces, clothes-or style.+<p align="center"><b>Turn any video into a depth blockout — the control signal that lets a video model copy a shot's staging and camera, without copying its faces, clothes or style.</b></p>-<p align="center"><img src="docs/demo.gif" width="560" alt="left: source, right: depth blockout"></p>+<p align="center">+<a href="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml"><img src="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml/badge.svg" alt="ci"></a>+<a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-blue.svg" alt="Apache-2.0"></a>+<img src="https://img.shields.io/badge/python-3.10%20%7C%203.12-blue" alt="python">+<img src="https://img.shields.io/badge/runs%20on-CUDA%20%C2%B7%20Apple%20Silicon%20%C2%B7%20CPU-555" alt="platforms">+</p>-One command, one model download (Apache-2.0), runs on an 8 GB GPU or an Apple M-series.+<p align="center"><a href="README.zh-CN.md">中文</a> · <a href="docs/USAGE.md">User guide</a> · <a href="CHANGELOG.md">Changelog</a></p>-[中文说明](README.zh-CN.md)+---++```bash+pip install git+https://github.com/maosika-ai/depthcat+depthcat reference.mp4 -o blockout.mp4 --target h3+```++That is the whole workflow. Feed `blockout.mp4` to a depth-conditioned video model+(MiniMax H3 Fun ControlNet, Wan VACE, …) together with your own prompt, and the+generated clip follows the reference's **blocking and camera** while its **look** comes+entirely from your prompt.++## Why a depth blockout++A reference video carries two kinds of information: *how the shot is staged* (who is+where, how big, how they move, how the camera moves) and *what it looks like* (faces,+wardrobe, lighting, style). Depth keeps the first and discards the second. That makes it++- **the cleanest control signal** for "shoot my script like this reference",+- **a compliance layer** — no likeness, no wardrobe, no brand leaves the reference,+- **model-agnostic** — every depth ControlNet was trained on exactly this kind of image.++depthcat produces that signal correctly and reproducibly, with the details that ad-hoc+pipelines get wrong already handled.++## What it gets right++| | depthcat | typical "run a depth model per frame" script |+|---|---|---|+| Temporal consistency | native video model (32-frame windows, overlap, cross-window alignment) | flickers |+| Normalisation | once over the whole clip | per frame → static walls "breathe" |+| Resolution | inference at model resolution, output upscaled frame by frame | full-res float depth for every frame → 8–20 GB RAM |+| Frame rate | picks frames by timestamp (30 → 24 really is 24) | integer stride → still 30 |+| Output size | crops to the generator's multiple (32 for H3) | pads → black border reads as a wall |+| Encoding | x264 CRF 12, key frame every 2 s | default CRF → banding → jitter in the generated clip |+| Licence | Apache-2.0 model by default, non-commercial variants opt-in with a warning | whatever was on disk |+| Safety | RAM estimate before running; refuses with a concrete `--max-frames` | swaps the machine to death |## Install```bash-pip install git+https://github.com/maosika-ai/depthcat-# needs ffmpeg on PATH (or: pip install "depthcat[ffmpeg]")+pip install git+https://github.com/maosika-ai/depthcat # needs ffmpeg on PATH+pip install "depthcat[ffmpeg] @ git+https://github.com/maosika-ai/depthcat" # bundles an ffmpeg binary```-Weights (111 MB) download from Hugging Face on first run. In China set-`HF_ENDPOINT=https://hf-mirror.com`.+Weights (111 MB, Apache-2.0) download from Hugging Face on first run.+Behind the Great Firewall: `export HF_ENDPOINT=https://hf-mirror.com`.## Use```bash-depthcat in.mp4 -o blockout.mp4 # keep fps/size, near = white-depthcat in.mp4 -o blockout.mp4 --target h3 # MiniMax H3: 24 fps, ×32 dims, ≤15 s-depthcat in.mp4 -o blockout.mp4 --npz depth.npz # also keep raw float depth+depthcat in.mp4 -o out.mp4 # keep fps and size, near = white+depthcat in.mp4 -o out.mp4 --target h3 # MiniMax H3 ControlNet: 24 fps, ×32, ≤ 15 s+depthcat in.mp4 -o out.mp4 --npz d.npz --metrics run.json``````python-from depthcat import extract, to_gray, write_gray_video-depths, fps = extract("in.mp4") # float32 [T, H, W], larger = closer-write_gray_video(to_gray(depths), "out.mp4", fps)-```+from pathlib import Path+from depthcat import RunConfig, run-| Option | Default | What it does |-|---|---|---|-| `--model small\|base\|large` | `small` | Only `small` is Apache-2.0. `base`/`large` are CC-BY-NC and print a warning. |-| `--target none\|h3\|wan` | `none` | fps + frame-size preset for a generator (see below). |-| `--fps N` | source | Override output fps. Frames are picked by timestamp, so 30 → 24 really gives 24. |-| `--max-res N` | 1280 | Cap the longer side before inference; output is written at that size. |-| `--invert` | off | far = white instead of near = white. |-| `--clip PCT` | 0 | Percentile clip on both tails so one hot pixel can't crush contrast. |-| `--gamma G` | 1.0 | > 1 darkens mid-tones (more separation near camera). |-| `--crf N` | 12 | x264 quality. Deliberately generous — banding in a control video becomes jitter in the output. |-| `--npz PATH` | – | Save raw float depth for your own post-processing. |--### Targets--| `--target` | fps | frame size | max length | for |-|---|---|---|---|---|-| `none` | source | even | – | anything |-| `h3` | 24 | multiple of 32 (centre-crop) | 15 s | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) depth input |-| `wan` | 16 | multiple of 16 | – | Wan 2.1 VACE control video |+result = run(RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3"))+print(result.ms_per_frame, result.peak_rss_bytes)+```-We crop rather than pad: a padded black border reads to the generator as a far wall.+Every option, preset, exit code and recipe: **[docs/USAGE.md](docs/USAGE.md)**.-## What it gets right (that ad-hoc pipelines get wrong)+## Targets-1. **One normalisation for the whole clip.** Per-frame min/max makes a static wall- change brightness whenever someone walks toward the camera; the generator reads that- as the scene breathing. Depth is scaled once over all frames.-2. **Temporal consistency comes from the model, not a blur.** Video Depth Anything runs- 32-frame windows with 10-frame overlap and aligns scale/shift across windows. We call- that path as-is instead of re-chunking outside it.-3. **Near is white**, the convention depth ControlNets were trained on. `--invert` if- your tool wants the opposite.-4. **fps by timestamp, size by crop, quality by CRF.** See table above.+| `--target` | fps | frame size | max | for |+|---|---|---|---|---|+| `none` | source | even | – | anything |+| `h3` | 24 | multiple of 32 | 15 s | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) |+| `wan` | 16 | multiple of 16 | – | Wan 2.1 VACE (values from the docs; not yet verified end-to-end) |-## Performance (measured end-to-end, Small model, 294 frames @ 736×1280)+## Performance (measured, Small model, 294 frames @ 736×1280)-| device | precision | speed | note |+| device | precision | ms / frame | host RAM peak ||---|---|---|---|-| Apple M2 Max, torch 2.9.1 | fp32 | ~500 ms/frame (12 s clip in 2.7 min) | fp16 on MPS is pathologically slow — we force fp32 there |-| Apple M2 Max, torch 2.6.0 | fp32 | ~1700 ms/frame | 3.5× slower — **upgrade torch ≥ 2.9 on Apple Silicon** |-| CPU (M2 Max) | fp32 | ~1850 ms/frame | works, slow |-| NVIDIA RTX 4090 (AutoDL), torch 2.8 cu128 | fp16 | **70 ms/frame**, 31 ms with `--input-size 364` | host RAM peak 4.5 GB (was 8.2 before the model-resolution change) |-| NVIDIA A100 (upstream figure) | fp16 | ~8 ms/frame | 6.8 GB VRAM at 32-frame batch |+| RTX 4090, torch 2.8 cu128 | fp16 | **70** (31 with `--input-size 364`) | 4.5 GB |+| Apple M2 Max, torch 2.9.1 | fp32 | ~500 | ~4 GB |+| Apple M2 Max, torch 2.6.0 | fp32 | ~1700 | — |+| CPU (M2 Max) | fp32 | ~1850 | — |++fp16 is enabled on CUDA only: on MPS it is pathologically slow. Upgrade torch to ≥ 2.9 on Apple Silicon.## Models and licences| weights | licence | commercial ||---|---|---|-| `Video-Depth-Anything-Small` (default) | Apache-2.0 | ✅ |-| `Video-Depth-Anything-Base` / `-Large` | CC-BY-NC-4.0 | ❌ opt-in only, with warning |+| Video-Depth-Anything-**Small** (default) | Apache-2.0 | ✅ |+| Video-Depth-Anything-Base / -Large | CC-BY-NC-4.0 | ❌ opt-in with `--model`, prints a warning |-Code is Apache-2.0. The model code from+depthcat's own code is Apache-2.0. The model code from[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything)-is vendored under `depthcat/third_party/` (see `NOTICE`).--## ComfyUI--There is already a good node — [ComfyUI-Video-Depth-Anything](https://github.com/yuvraj108c/ComfyUI-Video-Depth-Anything).-This project is for people who want a CLI / Python API, a licence-safe default, and the-target presets.+(CVPR 2025) is vendored under `depthcat/third_party/` — see [NOTICE](NOTICE).## Roadmap- `--people-only`: keep person silhouettes, flatten the background (SAM 2)- `--also pose,canny,normal`: the other MiniMax H3 Fun ControlNet inputs from one pass-- Depth Anything 3 (per-frame + smoothing) and ViGeo backends-- Gradio demo / Hugging Face Space+- Docker image, PyPI package, ComfyUI example workflow+- Hugging Face Space demo++## Contributing-## Credits+Issues and PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Run `ruff check` and+`pytest` before pushing; the CI also runs the real model on CPU.-Video Depth Anything — Chen et al., CVPR 2025. Built at [Maoska](https://www.maosika.com).+Built at [Maoska (猫斯卡)](https://www.maosika.com), an AI short-drama studio.diff --git a/README.zh-CN.md b/README.zh-CN.mdindex bfbfced..6ffd873 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -1,100 +1,119 @@-# depthcat+<p align="center"><img src="docs/demo.gif" width="560" alt="左:原片;右:depthcat 抽出的深度白模"></p>-[](https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml) [](LICENSE)+<h1 align="center">depthcat</h1>-把任意视频抽成**深度白模视频**——近白远黑的灰度片,喂给视频生成模型的 ControlNet-(MiniMax H3 Fun ControlNet、Wan VACE……),只复制参考片的**走位与镜头**,不带走人脸、-服装和画风。就是 LibTV「深度动作捕捉」那种输出。+<p align="center"><b>把任意视频抽成深度白模——让视频模型复制一段参考片的走位和运镜,却不带走它的人脸、服装和画风。</b></p>-<p align="center"><img src="docs/demo.gif" width="560" alt="左:原片;右:深度白模"></p>+<p align="center">+<a href="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml"><img src="https://github.com/maosika-ai/depthcat/actions/workflows/ci.yml/badge.svg" alt="ci"></a>+<a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-blue.svg" alt="Apache-2.0"></a>+<img src="https://img.shields.io/badge/python-3.10%20%7C%203.12-blue" alt="python">+<img src="https://img.shields.io/badge/%E8%BF%90%E8%A1%8C%E4%BA%8E-CUDA%20%C2%B7%20Apple%20Silicon%20%C2%B7%20CPU-555" alt="platforms">+</p>-一条命令,一个 Apache-2.0 的模型(111 MB),8G 显卡或 Apple M 系列芯片都能跑。+<p align="center"><a href="README.md">English</a> · <a href="docs/USAGE.zh-CN.md">使用手册</a> · <a href="CHANGELOG.md">更新日志</a></p>-## 安装+---```bashpip install git+https://github.com/maosika-ai/depthcat-# 需要 PATH 里有 ffmpeg(或 pip install "depthcat[ffmpeg]")+depthcat 参考片.mp4 -o 白模.mp4 --target h3```-首次运行自动从 Hugging Face 下载权重。国内请设置 `HF_ENDPOINT=https://hf-mirror.com`。+整个流程就这两行。把 `白模.mp4` 连同你自己的提示词一起交给带深度控制的视频模型+(MiniMax H3 Fun ControlNet、Wan VACE……),生成出来的片子**走位和镜头跟参考片**,+**长相和画风全听你的提示词**。就是 LibTV「深度动作捕捉」那种输出,但你自己掌握、可商用。++## 为什么是深度白模++一段参考视频里有两种信息:**怎么拍的**(谁站哪、多大、怎么动、镜头怎么动)和+**长什么样**(人脸、衣服、光线、风格)。深度图只保留前者、丢掉后者。所以它是++- **最干净的控制信号**——"按这段参考片的拍法拍我的剧本"+- **一层合规隔离**——人脸、服装、品牌一样都不会从参考片带出来+- **模型无关**——所有深度 ControlNet 训练时吃的就是这种图++depthcat 负责把这个信号做对、做稳,把那些"随手跑一下深度模型"必然踩的坑提前处理掉。++## 做对了什么++| | depthcat | 常见的"逐帧跑深度模型"脚本 |+|---|---|---|+| 时序一致性 | 原生视频模型(32 帧窗口、重叠、跨窗口对齐) | 闪烁 |+| 归一化 | 整段视频只做一次 | 逐帧 → 静止的墙在"呼吸" |+| 分辨率 | 按模型分辨率推理,8 位结果逐帧放大 | 每帧存全分辨率浮点深度 → 8–20 GB 内存 |+| 帧率 | 按时间戳选帧(30→24 真的是 24) | 整数步长 → 还是 30 |+| 输出尺寸 | 裁到生成模型要的倍数(H3 是 32) | 补黑边 → 模型当成一堵墙 |+| 编码 | x264 CRF 12,2 秒一个关键帧 | 默认 CRF → 色带 → 出片抖动 |+| 许可 | 默认 Apache-2.0 权重,非商用权重需显式指定并警告 | 硬盘上有啥用啥 |+| 安全 | 跑前估内存,超预算拒跑并给出 `--max-frames` 建议 | 把机器交换到死 |++## 安装++```bash+pip install git+https://github.com/maosika-ai/depthcat # 需要 PATH 里有 ffmpeg+pip install "depthcat[ffmpeg] @ git+https://github.com/maosika-ai/depthcat" # 自带 ffmpeg 二进制+```++首次运行自动下载权重(111 MB,Apache-2.0)。国内:`export HF_ENDPOINT=https://hf-mirror.com`。## 用法```bash-depthcat in.mp4 -o blockout.mp4 # 保持原 fps / 尺寸,近白远黑-depthcat in.mp4 -o blockout.mp4 --target h3 # MiniMax H3:24fps、边长 32 的倍数、≤15 秒-depthcat in.mp4 -o blockout.mp4 --npz depth.npz # 同时保存原始浮点深度+depthcat in.mp4 -o out.mp4 # 保持原 fps / 尺寸,近白远黑+depthcat in.mp4 -o out.mp4 --target h3 # MiniMax H3 ControlNet:24 fps、边长 32 的倍数、≤15 秒+depthcat in.mp4 -o out.mp4 --npz d.npz --metrics run.json``````python-from depthcat import extract, to_gray, write_gray_video-depths, fps = extract("in.mp4") # float32 [T, H, W],越大越近-write_gray_video(to_gray(depths), "out.mp4", fps)-```+from pathlib import Path+from depthcat import RunConfig, run-| 参数 | 默认 | 作用 |-|---|---|---|-| `--model small\|base\|large` | `small` | 只有 `small` 是 Apache-2.0;`base`/`large` 是 CC-BY-NC,加载时会警告 |-| `--target none\|h3\|wan` | `none` | 按目标模型设 fps 与尺寸(见下表) |-| `--fps N` | 原片 | 按**时间戳**选帧,30→24 真的是 24,不是整数步长那种假 24 |-| `--max-res N` | 1280 | 推理前把长边压到 N;输出也是这个尺寸 |-| `--invert` | 关 | 改成远白近黑 |-| `--clip PCT` | 0 | 两端各裁掉 PCT% 再归一,防一个热点像素把整段压暗 |-| `--gamma G` | 1.0 | >1 压暗中间调(近处层次更开) |-| `--crf N` | 12 | x264 质量。故意给高:控制视频里的色带会变成出片里的抖动 |-| `--npz PATH` | – | 保存原始深度,留给自己后处理 |--### 目标预设--| `--target` | fps | 尺寸 | 时长上限 | 对应 |-|---|---|---|---|---|-| `none` | 原片 | 偶数 | – | 通用 |-| `h3` | 24 | 32 的倍数(居中裁) | 15 秒 | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) 深度输入 |-| `wan` | 16 | 16 的倍数 | – | Wan 2.1 VACE 控制视频 |+result = run(RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3"))+print(result.ms_per_frame, result.peak_rss_bytes)+```-尺寸用裁不用补:补上去的黑边在生成模型眼里是一堵远墙。+全部参数、预设、退出码和实战配方:**[docs/USAGE.zh-CN.md](docs/USAGE.zh-CN.md)**。-## 做对的四个细节+## 目标预设-1. **整段视频只归一化一次,不逐帧。** 逐帧 min/max 会让静止的墙在有人走近镜头时忽明忽暗,生成模型读到的就是"场景在呼吸"。-2. **时序一致性来自模型,不是靠模糊。** Video Depth Anything 用 32 帧窗口、10 帧重叠、跨窗口对齐尺度;我们原样调用它的推理路径,不在外面重新切段。-3. **近处白。** 深度 ControlNet 训练时用的就是这个约定;要反过来用 `--invert`。-4. **fps 按时间戳、尺寸靠裁剪、质量靠 CRF。** 见上表。+| `--target` | fps | 尺寸 | 上限 | 对应 |+|---|---|---|---|---|+| `none` | 原片 | 偶数 | – | 通用 |+| `h3` | 24 | 32 的倍数 | 15 秒 | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) |+| `wan` | 16 | 16 的倍数 | – | Wan 2.1 VACE(按文档取值,尚未端到端实测) |-## 实测性能(全片端到端,Small 模型,294 帧 @ 736×1280)+## 实测性能(Small 模型,294 帧 @ 736×1280)-| 设备 | 精度 | 速度 | 说明 |+| 设备 | 精度 | ms / 帧 | 宿主内存峰值 ||---|---|---|---|-| Apple M2 Max,torch 2.9.1 | fp32 | 约 500 ms/帧(12 秒片 2.7 分钟) | MPS 上 fp16 慢到不可用,已强制 fp32 |-| Apple M2 Max,torch 2.6.0 | fp32 | 约 1700 ms/帧 | 慢 3.5 倍——**Apple 芯片请升级 torch ≥ 2.9** |-| CPU(M2 Max) | fp32 | 约 1850 ms/帧 | 能用,慢 |-| NVIDIA RTX 4090(AutoDL),torch 2.8 cu128 | fp16 | **70 ms/帧**,`--input-size 364` 时 31 ms | 宿主内存峰值 4.5 GB(改成模型分辨率推理前是 8.2) |-| NVIDIA A100(官方数据) | fp16 | 约 8 ms/帧 | 32 帧批次占 6.8 GB 显存 |+| RTX 4090,torch 2.8 cu128 | fp16 | **70**(`--input-size 364` 时 31) | 4.5 GB |+| Apple M2 Max,torch 2.9.1 | fp32 | 约 500 | 约 4 GB |+| Apple M2 Max,torch 2.6.0 | fp32 | 约 1700 | — |+| CPU(M2 Max) | fp32 | 约 1850 | — |++fp16 只在 CUDA 上开:MPS 上慢到不可用。Apple 芯片请把 torch 升到 ≥ 2.9。## 模型与许可| 权重 | 许可 | 商用 ||---|---|---|-| `Video-Depth-Anything-Small`(默认) | Apache-2.0 | ✅ |-| `Video-Depth-Anything-Base` / `-Large` | CC-BY-NC-4.0 | ❌ 需显式指定,加载时警告 |+| Video-Depth-Anything-**Small**(默认) | Apache-2.0 | ✅ |+| Video-Depth-Anything-Base / -Large | CC-BY-NC-4.0 | ❌ 需 `--model` 显式指定,运行时警告 |-代码 Apache-2.0。模型代码来自-[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything),-vendored 在 `depthcat/third_party/`(见 `NOTICE`)。--## ComfyUI--已有现成节点 [ComfyUI-Video-Depth-Anything](https://github.com/yuvraj108c/ComfyUI-Video-Depth-Anything)。-本项目面向要 CLI / Python API、要许可证安全默认值、要目标模型预设的人。+depthcat 自身代码 Apache-2.0。模型代码来自+[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything)+(CVPR 2025),vendored 在 `depthcat/third_party/`,见 [NOTICE](NOTICE)。## 路线图- `--people-only`:只留人物剪影、背景抹平(SAM 2)-- `--also pose,canny,normal`:一遍出 MiniMax H3 Fun ControlNet 的其余几种输入-- Depth Anything 3(逐帧 + 平滑)与 ViGeo 后端-- Gradio 演示 / Hugging Face Space+- `--also pose,canny,normal`:一遍出 MiniMax H3 Fun ControlNet 的其余输入+- Docker 镜像、PyPI 包、ComfyUI 示例工作流+- Hugging Face Space 在线试用++## 参与-## 致谢+欢迎 issue 和 PR,见 [CONTRIBUTING.md](CONTRIBUTING.md)。推之前跑 `ruff check` 和 `pytest`;CI 还会在 CPU 上真跑一遍模型。-Video Depth Anything — Chen et al., CVPR 2025。由 [猫斯卡](https://www.maosika.com) 出品。+出品:[猫斯卡](https://www.maosika.com),AI 短剧工作室。diff --git a/docs/USAGE.md b/docs/USAGE.mdnew file mode 100644index 0000000..39d6333--- /dev/null+++ b/docs/USAGE.md@@ -0,0 +1,215 @@+# depthcat — user guide++[中文版](USAGE.zh-CN.md)++1. [Install](#install)+2. [Command line](#command-line)+3. [Python API](#python-api)+4. [Recipes](#recipes)+5. [How it works](#how-it-works)+6. [Troubleshooting](#troubleshooting)+7. [FAQ](#faq)++## Install++Requirements: Python ≥ 3.10, PyTorch ≥ 2.1 (CPU, CUDA or Apple MPS), ffmpeg.++```bash+# with your own ffmpeg on PATH+pip install git+https://github.com/maosika-ai/depthcat++# no ffmpeg? this extra bundles a static binary+pip install "depthcat[ffmpeg] @ git+https://github.com/maosika-ai/depthcat"+```++PyTorch is not pinned to a CUDA version; install the build for your machine first if+`pip` picks the wrong one (see https://pytorch.org/get-started/locally/).++Model weights (111 MB) download from Hugging Face on first run into `~/.cache/huggingface`.+Offline machines: copy that cache, or pass `--checkpoint /path/video_depth_anything_vits.pth`.+In China: `export HF_ENDPOINT=https://hf-mirror.com`.++## Command line++```+depthcat INPUT -o OUTPUT [options]+```++### Model++| option | default | notes |+|---|---|---|+| `--model small\|base\|large` | `small` | Only `small` is Apache-2.0. `base`/`large` are CC-BY-NC-4.0: allowed for research, a warning is printed. |+| `--device auto\|cuda\|mps\|cpu` | `auto` | fp16 on CUDA, fp32 elsewhere (fp16 on MPS is unusably slow). |+| `--input-size N` | `518` | Model short side, multiple of 14. `364` is ~2× faster with visibly softer edges. |+| `--checkpoint PATH` | – | Local `.pth`, skips the download. |++### Output++| option | default | notes |+|---|---|---|+| `--target none\|h3\|wan` | `none` | fps + frame-size preset, see [Targets](#targets). |+| `--fps N` | preset or source | Frames are chosen by timestamp, so 30 → 24 yields 24, not 30. Never upsamples. |+| `--max-res N` | `1280` | Cap on the output's longer side. Inference always runs at model resolution regardless. |+| `--max-frames N` | all | Stop after N source frames. |+| `--invert` | off | far = white. Default is near = white (what depth ControlNets expect). |+| `--clip PCT` | `0` | Trim PCT % from each tail before scaling to 0–255, so one hot pixel can't crush contrast. `0` = exact min/max. |+| `--gamma G` | `1.0` | > 1 darkens mid-tones: more separation close to the camera. |+| `--crf N` | `12` | x264 quality. Generous on purpose — banding in a control video becomes jitter in the generated clip. |+| `--npz PATH` | – | Raw float32 depth at processing resolution (`depths[T,H,W]`, `fps`). |+| `--metrics PATH` | – | JSON with config, plan, timings, peak RSS, output size. |++### Safety++| option | notes |+|---|---|+| `--force` | Run even when the host-RAM estimate exceeds half of physical RAM. |++### Exit codes++| code | meaning | typical fix |+|---|---|---|+| 0 | done | |+| 1 | unexpected error (a bug — please report with `-v` output) | |+| 2 | bad input or arguments | check the path / the option value in the message |+| 3 | RAM estimate over budget | use the suggested `--max-frames`, split the clip, or `--force` |+| 4 | ffmpeg missing | `pip install imageio-ffmpeg` or install ffmpeg |+| 5 | weights could not be obtained | set `HF_ENDPOINT` to a mirror, or `--checkpoint` |++### Targets++| `--target` | fps | frame size | max length | notes |+|---|---|---|---|---|+| `none` | source | even | – | |+| `h3` | 24 | multiple of 32 (centre crop) | 15 s (warns) | MiniMax-H3-Fun-Controlnet-Union depth input |+| `wan` | 16 | multiple of 16 | – | Wan 2.1 VACE control video; values from the docs, not yet verified end-to-end |++Frame size is **cropped**, never padded: a padded black border reads to the generator as a far wall.++## Python API++```python+from pathlib import Path+from depthcat import RunConfig, run, plan++cfg = RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3", metrics=Path("m.json"))+print(plan(cfg)) # sizes, frame counts, RAM estimate — no decoding yet+result = run(cfg) # same code path as the CLI+print(result.frames, result.fps, result.ms_per_frame, result.peak_rss_bytes)+```++Lower-level pieces, for your own post-processing:++```python+from depthcat import extract, to_gray, upsample_frames, write_gray_video++depths, fps = extract("in.mp4") # float32 [T, H, W], larger = closer+gray = to_gray(depths, clip_percent=0.5, gamma=1.2) # uint8 [T, H, W], whole-clip normalised+write_gray_video(upsample_frames(gray, 1280, 736), "out.mp4", fps, size=(1280, 736))+```++Errors that are the *user's* to fix are subclasses of `depthcat.DepthcatError` and carry an+`exit_code`; anything else is a bug.++`RunConfig(..., backend="fake")` runs the entire pipeline with a synthetic depth field and no+model — handy for testing your own integration in milliseconds.++## Recipes++### MiniMax H3 Fun ControlNet (ComfyUI)++1. `depthcat ref.mp4 -o ref_depth.mp4 --target h3` (24 fps, ×32, ≤ 15 s).+2. In ComfyUI with the [Fun ControlNet Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union)+ weights, load `ref_depth.mp4` as the control video, choose the **depth** condition, write your+ prompt for the *look* (cast, wardrobe, lighting, style). Staging and camera come from the control video.+3. Keep the generator's resolution equal to the control video's, or let the node resize — but do not+ pad.++### Wan 2.1 VACE++`depthcat ref.mp4 -o ref_depth.mp4 --target wan`, then use it as the depth control video in your+VACE workflow. Report back if the fps/size values need adjusting — this preset has not been+verified end-to-end yet.++### Batch++```bash+for f in clips/*.mp4; do depthcat "$f" -o "out/$(basename "$f" .mp4)_depth.mp4" --target h3 --metrics "out/$(basename "$f" .mp4).json"; done+```++The model loads once per process; for hundreds of clips write a short Python loop with+`extract()` and reuse the backend object:++```python+from depthcat.backends import get_backend+from depthcat import read_video, to_gray, write_gray_video+b = get_backend("vda", model="small")+for src in sources:+ frames, fps = read_video(src, max_res=900)+ write_gray_video(to_gray(b.infer(frames, fps)), src.with_suffix(".depth.mp4"), fps)+```++### Long videos++Everything is held in RAM for the whole clip (frames + depth). depthcat estimates the need and+refuses beyond half of physical RAM with a `--max-frames` suggestion. For a 3-minute clip, split+it into ≤ 15 s pieces first (`ffmpeg -ss … -t 15 …`); the generators want short control videos anyway.++### Apple Silicon++Works out of the box (`--device auto` picks MPS). Two things measured on an M2 Max:+torch 2.9.1 is ~3.5× faster than 2.6.0, and fp16 is disabled on MPS because it does not finish.+Expect ~0.5 s/frame at 736×1280; a 12 s clip takes ~2.5 minutes.++### Reproducible depth for research++`--npz` writes the raw float depth at processing resolution. Combined with `--metrics` (which+records the exact config and plan) a run is fully reproducible.++## How it works++```+input video ─▶ probe ─▶ plan (sizes, frames, RAM) ─▶ decode at model resolution+ ─▶ Video Depth Anything (32-frame windows, 10-frame overlap, cross-window alignment)+ ─▶ whole-clip normalisation → uint8, near = white+ ─▶ per-frame upscale to output size ─▶ centre-crop to target multiple ─▶ x264+```++- **Model**: [Video Depth Anything](https://github.com/DepthAnything/Video-Depth-Anything)+ (CVPR 2025). Its temporal module is what keeps the depth stable across frames; single-image+ models flicker. The model code is vendored unmodified (two import lines) under `third_party/`.+- **Resolution**: the model works on a 518-px short side. Feeding larger frames only inflates the+ float depth kept per frame; depthcat decodes at that size and upscales the 8-bit result.+- **Normalisation**: `(d − min) / (max − min)` over *all* frames; optional percentile clip.+- **Encoding**: `libx264 -crf 12 -pix_fmt yuv420p -g 2·fps -movflags +faststart`.++## Troubleshooting++| symptom | cause | fix |+|---|---|---|+| `ModuleNotFoundError: torchvision` | torch installed without torchvision | `pip install torchvision` (matching your torch build) |+| exit 4 / "ffmpeg not found" | no ffmpeg | `pip install imageio-ffmpeg` or install ffmpeg |+| exit 5 / weights download fails | no route to huggingface.co | `export HF_ENDPOINT=https://hf-mirror.com`, or `--checkpoint` |+| exit 3 | clip too long for RAM | use the suggested `--max-frames`, split the clip |+| very slow on Mac | torch < 2.9, or fp16 forced | `pip install -U torch`; never force fp16 on MPS |+| output is 30 fps although `--fps 24` | source fps < 24 | depthcat never upsamples |+| output smaller than source | `--target` crop or `--max-res` | expected; see Targets |+| banding in the blockout | `--crf` raised | keep ≤ 14 |+| people look flat / merged with background | scene has little depth range | try `--clip 0.5 --gamma 1.3` |++## FAQ++**Is Small good enough?** For a control video, yes: the generator reads silhouette, size and+motion, not fine geometry, and Small's silhouettes are clean (see the demo GIF). We have not+benchmarked Base/Large against Small ourselves; they are CC-BY-NC, so Small is also the only+choice for commercial work.++**Why near = white?** Depth ControlNets (H3 Fun, Wan VACE, SD depth) were trained on inverse-depth+maps where closer is brighter. `--invert` if a tool wants the opposite.++**Can it output pose / canny / normals?** Not yet — on the roadmap.++**Does it remove the background / keep only people?** Not yet (`--people-only` is on the roadmap).+The full-frame depth already carries no identity or style.++**GPU memory?** Small in fp16 needs ~2 GB at the default input size; a 4090 is idle most of the time.diff --git a/docs/USAGE.zh-CN.md b/docs/USAGE.zh-CN.mdnew file mode 100644index 0000000..5b39bfd--- /dev/null+++ b/docs/USAGE.zh-CN.md@@ -0,0 +1,201 @@+# depthcat 使用手册++[English](USAGE.md)++1. [安装](#安装)+2. [命令行](#命令行)+3. [Python 接口](#python-接口)+4. [实战配方](#实战配方)+5. [原理](#原理)+6. [排错](#排错)+7. [常见问题](#常见问题)++## 安装++要求:Python ≥ 3.10、PyTorch ≥ 2.1(CPU / CUDA / Apple MPS 皆可)、ffmpeg。++```bash+# 自己有 ffmpeg+pip install git+https://github.com/maosika-ai/depthcat++# 没有 ffmpeg?这个 extra 自带一个静态二进制+pip install "depthcat[ffmpeg] @ git+https://github.com/maosika-ai/depthcat"+```++PyTorch 不锁 CUDA 版本;如果 `pip` 装错了,先按 https://pytorch.org/get-started/locally/ 装好再装 depthcat。++模型权重(111 MB)首次运行时从 Hugging Face 下到 `~/.cache/huggingface`。+离线机器:把这个缓存目录拷过去,或用 `--checkpoint /路径/video_depth_anything_vits.pth`。+国内:`export HF_ENDPOINT=https://hf-mirror.com`。++## 命令行++```+depthcat 输入 -o 输出 [选项]+```++### 模型++| 选项 | 默认 | 说明 |+|---|---|---|+| `--model small\|base\|large` | `small` | 只有 `small` 是 Apache-2.0。`base`/`large` 是 CC-BY-NC-4.0:研究可用,运行时会警告。 |+| `--device auto\|cuda\|mps\|cpu` | `auto` | CUDA 用 fp16,其余 fp32(MPS 上 fp16 慢到不可用)。 |+| `--input-size N` | `518` | 模型短边,14 的倍数。`364` 快约一倍,边缘略软。 |+| `--checkpoint 路径` | – | 本地 `.pth`,跳过下载。 |++### 输出++| 选项 | 默认 | 说明 |+|---|---|---|+| `--target none\|h3\|wan` | `none` | fps + 尺寸预设,见[目标预设](#目标预设)。 |+| `--fps N` | 预设或原片 | 按**时间戳**选帧,30→24 真的是 24。绝不向上插帧。 |+| `--max-res N` | `1280` | 输出长边上限。推理永远在模型分辨率进行,与此无关。 |+| `--max-frames N` | 全部 | 只处理前 N 帧。 |+| `--invert` | 关 | 改成远白近黑。默认近白远黑(深度 ControlNet 要的就是这个)。 |+| `--clip PCT` | `0` | 两端各裁 PCT% 再拉到 0–255,防一个热点像素压暗全片。`0` 为精确 min/max。 |+| `--gamma G` | `1.0` | >1 压暗中间调,近处层次更开。 |+| `--crf N` | `12` | x264 质量。故意给高:控制视频里的色带会变成出片里的抖动。 |+| `--npz 路径` | – | 处理分辨率下的原始 float32 深度(`depths[T,H,W]`、`fps`)。 |+| `--metrics 路径` | – | JSON:配置、计划、耗时、峰值内存、输出尺寸。 |++### 安全++| 选项 | 说明 |+|---|---|+| `--force` | 内存估算超过物理内存一半时仍然运行。 |++### 退出码++| 码 | 含义 | 怎么办 |+|---|---|---|+| 0 | 完成 | |+| 1 | 意外错误(是 bug——请带 `-v` 输出提 issue) | |+| 2 | 输入或参数有问题 | 按报错里指出的路径 / 参数改 |+| 3 | 内存估算超预算 | 用建议的 `--max-frames`、把片子切短,或 `--force` |+| 4 | 缺 ffmpeg | `pip install imageio-ffmpeg` 或装 ffmpeg |+| 5 | 拿不到权重 | `HF_ENDPOINT` 指向镜像,或 `--checkpoint` |++### 目标预设++| `--target` | fps | 尺寸 | 时长上限 | 说明 |+|---|---|---|---|---|+| `none` | 原片 | 偶数 | – | |+| `h3` | 24 | 32 的倍数(居中裁) | 15 秒(超了警告) | MiniMax-H3-Fun-Controlnet-Union 深度输入 |+| `wan` | 16 | 16 的倍数 | – | Wan 2.1 VACE 控制视频;按文档取值,尚未端到端实测 |++尺寸用**裁**不用补:补上去的黑边在生成模型眼里是一堵远墙。++## Python 接口++```python+from pathlib import Path+from depthcat import RunConfig, run, plan++cfg = RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3", metrics=Path("m.json"))+print(plan(cfg)) # 尺寸、帧数、内存估算——此时还没解码+result = run(cfg) # 与命令行完全同一条代码路径+print(result.frames, result.fps, result.ms_per_frame, result.peak_rss_bytes)+```++更底层的零件,给自己做后处理用:++```python+from depthcat import extract, to_gray, upsample_frames, write_gray_video++depths, fps = extract("in.mp4") # float32 [T, H, W],越大越近+gray = to_gray(depths, clip_percent=0.5, gamma=1.2) # uint8 [T, H, W],整段归一+write_gray_video(upsample_frames(gray, 1280, 736), "out.mp4", fps, size=(1280, 736))+```++属于"用户该修"的错误都是 `depthcat.DepthcatError` 的子类、带 `exit_code`;其余都是 bug。++`RunConfig(..., backend="fake")` 用合成深度场跑完整条流水线、不载模型——几毫秒验证你自己的集成。++## 实战配方++### MiniMax H3 Fun ControlNet(ComfyUI)++1. `depthcat 参考.mp4 -o 参考_depth.mp4 --target h3`(24 fps、32 倍数、≤15 秒)。+2. ComfyUI 里装 [Fun ControlNet Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) 权重,+ 把 `参考_depth.mp4` 作为控制视频、条件选 **depth**,提示词只写**长相**(人物、服装、光线、画风)。+ 走位和镜头由控制视频决定。+3. 生成分辨率和控制视频保持一致,或让节点缩放——但别补边。++### Wan 2.1 VACE++`depthcat 参考.mp4 -o 参考_depth.mp4 --target wan`,在 VACE 工作流里当深度控制视频用。+这个预设还没端到端验过,fps/尺寸如需调整欢迎反馈。++### 批量++```bash+for f in clips/*.mp4; do depthcat "$f" -o "out/$(basename "$f" .mp4)_depth.mp4" --target h3 --metrics "out/$(basename "$f" .mp4).json"; done+```++每个进程载一次模型;几百条片子的话写个 Python 循环复用后端对象:++```python+from depthcat.backends import get_backend+from depthcat import read_video, to_gray, write_gray_video+b = get_backend("vda", model="small")+for src in sources:+ frames, fps = read_video(src, max_res=900)+ write_gray_video(to_gray(b.infer(frames, fps)), src.with_suffix(".depth.mp4"), fps)+```++### 长视频++整段视频的帧和深度都在内存里。depthcat 会先估算,超过物理内存一半就拒跑并给 `--max-frames` 建议。+三分钟的片子先用 `ffmpeg -ss … -t 15 …` 切成 ≤15 秒的段——生成模型本来也只吃短的控制视频。++### Apple 芯片++开箱即用(`--device auto` 会选 MPS)。M2 Max 上两条实测:torch 2.9.1 比 2.6.0 快约 3.5 倍;+MPS 上 fp16 跑不完,所以强制 fp32。736×1280 约 0.5 秒/帧,12 秒的片约 2.5 分钟。++### 可复现的深度数据(研究用)++`--npz` 输出处理分辨率下的原始浮点深度;配合 `--metrics`(记录完整配置与计划),一次运行可完全复现。++## 原理++```+输入视频 ─▶ 探测 ─▶ 计划(尺寸、帧数、内存)─▶ 按模型分辨率解码+ ─▶ Video Depth Anything(32 帧窗口、10 帧重叠、跨窗口对齐)+ ─▶ 整段归一化 → uint8,近白远黑+ ─▶ 逐帧放大到输出尺寸 ─▶ 居中裁到目标倍数 ─▶ x264+```++- **模型**:[Video Depth Anything](https://github.com/DepthAnything/Video-Depth-Anything)(CVPR 2025)。+ 它的时序模块是深度不闪的原因;单图模型逐帧跑必闪。模型代码原样 vendored 在 `third_party/`(只改两行 import)。+- **分辨率**:模型工作在短边 518 px。喂更大的帧只会让每帧存的浮点深度变大;depthcat 按这个尺寸解码,放大的是 8 位结果。+- **归一化**:`(d − min) / (max − min)` 对**全部帧**做一次;可选百分位裁剪。+- **编码**:`libx264 -crf 12 -pix_fmt yuv420p -g 2·fps -movflags +faststart`。++## 排错++| 现象 | 原因 | 处理 |+|---|---|---|+| `ModuleNotFoundError: torchvision` | torch 装了但没装 torchvision | `pip install torchvision`(与 torch 同一构建) |+| 退出码 4 / "ffmpeg not found" | 没 ffmpeg | `pip install imageio-ffmpeg` 或装 ffmpeg |+| 退出码 5 / 权重下载失败 | 连不上 huggingface.co | `export HF_ENDPOINT=https://hf-mirror.com`,或 `--checkpoint` |+| 退出码 3 | 片子太长内存不够 | 用建议的 `--max-frames`,或切片 |+| Mac 上极慢 | torch < 2.9,或强开了 fp16 | `pip install -U torch`;MPS 上别开 fp16 |+| 给了 `--fps 24` 输出还是 30 | 原片 fps 低于 24 | depthcat 不向上插帧 |+| 输出比原片小 | `--target` 裁剪或 `--max-res` | 正常,见目标预设 |+| 白模有色带 | 调高了 `--crf` | 保持 ≤14 |+| 人和背景糊在一起 | 场景纵深本身很小 | 试 `--clip 0.5 --gamma 1.3` |++## 常见问题++**Small 够用吗?** 做控制视频够:生成模型读的是轮廓、体量和运动,不是精细几何,Small 的剪影已经很干净(见首页动图)。+Base/Large 和 Small 的差距我们没有自己测过;它们是 CC-BY-NC,商用本来也只能选 Small。++**为什么近白远黑?** 深度 ControlNet(H3 Fun、Wan VACE、SD depth)训练用的都是"越近越亮"的逆深度图。+要反过来用 `--invert`。++**能出骨架 / 边缘 / 法线吗?** 还不能,在路线图上。++**能去背景、只留人吗?** 还不能(`--people-only` 在路线图上)。全画面深度本身已经不含任何身份和画风信息。++**显存?** Small fp16 默认输入尺寸约 2 GB;4090 大部分时间在等 CPU。