Skip to content

ReShot — copy the shot, not the actors. Video → depth map, OpenPose skeleton or canny lines for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Apache-2.0. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system.

video2blockout 0.1.0: video → depth blockout video for video-generation ControlNets

wangfengcd490fa795a5
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
new file mode 100644
index 0000000..2d36c29
--- /dev/null
+++ b/.github/workflows/ci.yml
@@ -0,0 +1,40 @@
+name: ci
+on:
+ push: { branches: [main] }
+ pull_request:
+jobs:
+ unit:
+ name: unit (${{ matrix.os }}, py${{ matrix.python }})
+ runs-on: ${{ matrix.os }}
+ strategy:
+ fail-fast: false
+ matrix:
+ os: [ubuntu-latest, windows-latest]
+ python: ["3.10", "3.12"]
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v5
+ with: { python-version: "${{ matrix.python }}" }
+ - name: Install (CPU torch)
+ run: |
+ python -m pip install -U pip
+ python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
+ python -m pip install -e ".[ffmpeg,dev]"
+ - run: python -m pytest -q tests
+
+ model-smoke:
+ name: real model on CPU (8 frames)
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-python@v5
+ with: { python-version: "3.12" }
+ - uses: actions/cache@v4
+ with:
+ path: ~/.cache/huggingface
+ key: hf-vda-small-v1
+ - run: |
+ python -m pip install -U pip
+ python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
+ python -m pip install -e ".[ffmpeg,dev]"
+ - run: RUN_MODEL_TESTS=1 python -m pytest -q tests/test_model_smoke.py -s
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..b91c3cd
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,11 @@
+__pycache__/
+*.pyc
+*.egg-info/
+build/
+dist/
+.pytest_cache/
+.ruff_cache/
+*.npz
+outputs/
+examples/*_depth.mp4
+runs/
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000..12a6646
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,8 @@
+# Changelog
+
+## 0.1.0 — 2026-09-11
+- First release: `video2blockout in.mp4 -o out.mp4` on Video Depth Anything Small (Apache-2.0).
+- Depth estimated at model resolution, whole-clip normalisation, near = white, x264 CRF 12.
+- `--target h3` / `wan` presets (fps, size multiple, max length), `--npz`, `--metrics`.
+- Host-RAM estimate with refusal over 50 % of physical RAM; MPS allocator cap on Apple Silicon.
+- Measured: RTX 4090 fp16 70 ms/frame; Apple M2 Max fp32 ~500 ms/frame.
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000..d645695
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,202 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/NOTICE b/NOTICE
new file mode 100644
index 0000000..9bd0966
--- /dev/null
+++ b/NOTICE
@@ -0,0 +1,12 @@
+video2blockout
+Copyright 2026 Maoska
+
+This product includes software developed by Bytedance Ltd. and/or its affiliates:
+Video Depth Anything (https://github.com/DepthAnything/Video-Depth-Anything),
+licensed under the Apache License 2.0. It is vendored, unmodified except for
+two import lines, under video2blockout/third_party/video_depth_anything/.
+
+Model weights are downloaded at runtime from Hugging Face and carry their own licences:
+ depth-anything/Video-Depth-Anything-Small Apache-2.0 (default; commercial use OK)
+ depth-anything/Video-Depth-Anything-Base CC-BY-NC-4.0 (opt-in; non-commercial)
+ depth-anything/Video-Depth-Anything-Large CC-BY-NC-4.0 (opt-in; non-commercial)
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..b9ab44c
--- /dev/null
+++ b/README.md
@@ -0,0 +1,110 @@
+# video2blockout
+
+[![ci](https://github.com/maosika-ai/video2blockout/actions/workflows/ci.yml/badge.svg)](https://github.com/maosika-ai/video2blockout/actions/workflows/ci.yml) [![license](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
+
+Turn any video into a **depth "blockout" video** — the grey near-white / far-black
+clip that video-generation ControlNets (MiniMax H3 Fun ControlNet, Wan VACE, …) use to
+copy the *staging and camera* of a reference shot without copying its faces, clothes
+or style.
+
+<p align="center"><img src="docs/demo.gif" width="560" alt="left: source, right: depth blockout"></p>
+
+One command, one model download (Apache-2.0), runs on an 8 GB GPU or an Apple M-series.
+
+[中文说明](README.zh-CN.md)
+
+## Install
+
+```bash
+pip install git+https://github.com/maosika-ai/video2blockout
+# needs ffmpeg on PATH (or: pip install "video2blockout[ffmpeg]")
+```
+
+Weights (111 MB) download from Hugging Face on first run. In China set
+`HF_ENDPOINT=https://hf-mirror.com`.
+
+## Use
+
+```bash
+video2blockout in.mp4 -o blockout.mp4 # keep fps/size, near = white
+video2blockout in.mp4 -o blockout.mp4 --target h3 # MiniMax H3: 24 fps, ×32 dims, ≤15 s
+video2blockout in.mp4 -o blockout.mp4 --npz depth.npz # also keep raw float depth
+```
+
+```python
+from video2blockout import extract, to_gray, write_gray_video
+depths, fps = extract("in.mp4") # float32 [T, H, W], larger = closer
+write_gray_video(to_gray(depths), "out.mp4", fps)
+```
+
+| Option | Default | What it does |
+|---|---|---|
+| `--model small\|base\|large` | `small` | Only `small` is Apache-2.0. `base`/`large` are CC-BY-NC and print a warning. |
+| `--target none\|h3\|wan` | `none` | fps + frame-size preset for a generator (see below). |
+| `--fps N` | source | Override output fps. Frames are picked by timestamp, so 30 → 24 really gives 24. |
+| `--max-res N` | 1280 | Cap the longer side before inference; output is written at that size. |
+| `--invert` | off | far = white instead of near = white. |
+| `--clip PCT` | 0 | Percentile clip on both tails so one hot pixel can't crush contrast. |
+| `--gamma G` | 1.0 | > 1 darkens mid-tones (more separation near camera). |
+| `--crf N` | 12 | x264 quality. Deliberately generous — banding in a control video becomes jitter in the output. |
+| `--npz PATH` | – | Save raw float depth for your own post-processing. |
+
+### Targets
+
+| `--target` | fps | frame size | max length | for |
+|---|---|---|---|---|
+| `none` | source | even | – | anything |
+| `h3` | 24 | multiple of 32 (centre-crop) | 15 s | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) depth input |
+| `wan` | 16 | multiple of 16 | – | Wan 2.1 VACE control video |
+
+We crop rather than pad: a padded black border reads to the generator as a far wall.
+
+## What it gets right (that ad-hoc pipelines get wrong)
+
+1. **One normalisation for the whole clip.** Per-frame min/max makes a static wall
+ change brightness whenever someone walks toward the camera; the generator reads that
+ as the scene breathing. Depth is scaled once over all frames.
+2. **Temporal consistency comes from the model, not a blur.** Video Depth Anything runs
+ 32-frame windows with 10-frame overlap and aligns scale/shift across windows. We call
+ that path as-is instead of re-chunking outside it.
+3. **Near is white**, the convention depth ControlNets were trained on. `--invert` if
+ your tool wants the opposite.
+4. **fps by timestamp, size by crop, quality by CRF.** See table above.
+
+## Performance (measured end-to-end, Small model, 294 frames @ 736×1280)
+
+| device | precision | speed | note |
+|---|---|---|---|
+| Apple M2 Max, torch 2.9.1 | fp32 | ~500 ms/frame (12 s clip in 2.7 min) | fp16 on MPS is pathologically slow — we force fp32 there |
+| Apple M2 Max, torch 2.6.0 | fp32 | ~1700 ms/frame | 3.5× slower — **upgrade torch ≥ 2.9 on Apple Silicon** |
+| CPU (M2 Max) | fp32 | ~1850 ms/frame | works, slow |
+| NVIDIA RTX 4090 (AutoDL), torch 2.8 cu128 | fp16 | **70 ms/frame**, 31 ms with `--input-size 364` | host RAM peak 4.5 GB (was 8.2 before the model-resolution change) |
+| NVIDIA A100 (upstream figure) | fp16 | ~8 ms/frame | 6.8 GB VRAM at 32-frame batch |
+
+## Models and licences
+
+| weights | licence | commercial |
+|---|---|---|
+| `Video-Depth-Anything-Small` (default) | Apache-2.0 | ✅ |
+| `Video-Depth-Anything-Base` / `-Large` | CC-BY-NC-4.0 | ❌ opt-in only, with warning |
+
+Code is Apache-2.0. The model code from
+[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything)
+is vendored under `video2blockout/third_party/` (see `NOTICE`).
+
+## ComfyUI
+
+There is already a good node — [ComfyUI-Video-Depth-Anything](https://github.com/yuvraj108c/ComfyUI-Video-Depth-Anything).
+This project is for people who want a CLI / Python API, a licence-safe default, and the
+target presets.
+
+## Roadmap
+
+- `--people-only`: keep person silhouettes, flatten the background (SAM 2)
+- `--also pose,canny,normal`: the other MiniMax H3 Fun ControlNet inputs from one pass
+- Depth Anything 3 (per-frame + smoothing) and ViGeo backends
+- Gradio demo / Hugging Face Space
+
+## Credits
+
+Video Depth Anything — Chen et al., CVPR 2025. Built at [Maoska](https://www.maosika.com).
diff --git a/README.zh-CN.md b/README.zh-CN.md
new file mode 100644
index 0000000..8a2ccb5
--- /dev/null
+++ b/README.zh-CN.md
@@ -0,0 +1,100 @@
+# video2blockout
+
+[![ci](https://github.com/maosika-ai/video2blockout/actions/workflows/ci.yml/badge.svg)](https://github.com/maosika-ai/video2blockout/actions/workflows/ci.yml) [![license](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
+
+把任意视频抽成**深度白模视频**——近白远黑的灰度片,喂给视频生成模型的 ControlNet
+(MiniMax H3 Fun ControlNet、Wan VACE……),只复制参考片的**走位与镜头**,不带走人脸、
+服装和画风。就是 LibTV「深度动作捕捉」那种输出。
+
+<p align="center"><img src="docs/demo.gif" width="560" alt="左:原片;右:深度白模"></p>
+
+一条命令,一个 Apache-2.0 的模型(111 MB),8G 显卡或 Apple M 系列芯片都能跑。
+
+## 安装
+
+```bash
+pip install git+https://github.com/maosika-ai/video2blockout
+# 需要 PATH 里有 ffmpeg(或 pip install "video2blockout[ffmpeg]")
+```
+
+首次运行自动从 Hugging Face 下载权重。国内请设置 `HF_ENDPOINT=https://hf-mirror.com`。
+
+## 用法
+
+```bash
+video2blockout in.mp4 -o blockout.mp4 # 保持原 fps / 尺寸,近白远黑
+video2blockout in.mp4 -o blockout.mp4 --target h3 # MiniMax H3:24fps、边长 32 的倍数、≤15 秒
+video2blockout in.mp4 -o blockout.mp4 --npz depth.npz # 同时保存原始浮点深度
+```
+
+```python
+from video2blockout import extract, to_gray, write_gray_video
+depths, fps = extract("in.mp4") # float32 [T, H, W],越大越近
+write_gray_video(to_gray(depths), "out.mp4", fps)
+```
+
+| 参数 | 默认 | 作用 |
+|---|---|---|
+| `--model small\|base\|large` | `small` | 只有 `small` 是 Apache-2.0;`base`/`large` 是 CC-BY-NC,加载时会警告 |
+| `--target none\|h3\|wan` | `none` | 按目标模型设 fps 与尺寸(见下表) |
+| `--fps N` | 原片 | 按**时间戳**选帧,30→24 真的是 24,不是整数步长那种假 24 |
+| `--max-res N` | 1280 | 推理前把长边压到 N;输出也是这个尺寸 |
+| `--invert` | 关 | 改成远白近黑 |
+| `--clip PCT` | 0 | 两端各裁掉 PCT% 再归一,防一个热点像素把整段压暗 |
+| `--gamma G` | 1.0 | >1 压暗中间调(近处层次更开) |
+| `--crf N` | 12 | x264 质量。故意给高:控制视频里的色带会变成出片里的抖动 |
+| `--npz PATH` | – | 保存原始深度,留给自己后处理 |
+
+### 目标预设
+
+| `--target` | fps | 尺寸 | 时长上限 | 对应 |
+|---|---|---|---|---|
+| `none` | 原片 | 偶数 | – | 通用 |
+| `h3` | 24 | 32 的倍数(居中裁) | 15 秒 | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) 深度输入 |
+| `wan` | 16 | 16 的倍数 | – | Wan 2.1 VACE 控制视频 |
+
+尺寸用裁不用补:补上去的黑边在生成模型眼里是一堵远墙。
+
+## 做对的四个细节
+
+1. **整段视频只归一化一次,不逐帧。** 逐帧 min/max 会让静止的墙在有人走近镜头时忽明忽暗,生成模型读到的就是"场景在呼吸"。
+2. **时序一致性来自模型,不是靠模糊。** Video Depth Anything 用 32 帧窗口、10 帧重叠、跨窗口对齐尺度;我们原样调用它的推理路径,不在外面重新切段。
+3. **近处白。** 深度 ControlNet 训练时用的就是这个约定;要反过来用 `--invert`。
+4. **fps 按时间戳、尺寸靠裁剪、质量靠 CRF。** 见上表。
+
+## 实测性能(全片端到端,Small 模型,294 帧 @ 736×1280)
+
+| 设备 | 精度 | 速度 | 说明 |
+|---|---|---|---|
+| Apple M2 Max,torch 2.9.1 | fp32 | 约 500 ms/帧(12 秒片 2.7 分钟) | MPS 上 fp16 慢到不可用,已强制 fp32 |
+| Apple M2 Max,torch 2.6.0 | fp32 | 约 1700 ms/帧 | 慢 3.5 倍——**Apple 芯片请升级 torch ≥ 2.9** |
+| CPU(M2 Max) | fp32 | 约 1850 ms/帧 | 能用,慢 |
+| NVIDIA RTX 4090(AutoDL),torch 2.8 cu128 | fp16 | **70 ms/帧**,`--input-size 364` 时 31 ms | 宿主内存峰值 4.5 GB(改成模型分辨率推理前是 8.2) |
+| NVIDIA A100(官方数据) | fp16 | 约 8 ms/帧 | 32 帧批次占 6.8 GB 显存 |
+
+## 模型与许可
+
+| 权重 | 许可 | 商用 |
+|---|---|---|
+| `Video-Depth-Anything-Small`(默认) | Apache-2.0 | ✅ |
+| `Video-Depth-Anything-Base` / `-Large` | CC-BY-NC-4.0 | ❌ 需显式指定,加载时警告 |
+
+代码 Apache-2.0。模型代码来自
+[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything),
+vendored 在 `video2blockout/third_party/`(见 `NOTICE`)。
+
+## ComfyUI
+
+已有现成节点 [ComfyUI-Video-Depth-Anything](https://github.com/yuvraj108c/ComfyUI-Video-Depth-Anything)。
+本项目面向要 CLI / Python API、要许可证安全默认值、要目标模型预设的人。
+
+## 路线图
+
+- `--people-only`:只留人物剪影、背景抹平(SAM 2)
+- `--also pose,canny,normal`:一遍出 MiniMax H3 Fun ControlNet 的其余几种输入
+- Depth Anything 3(逐帧 + 平滑)与 ViGeo 后端
+- Gradio 演示 / Hugging Face Space
+
+## 致谢
+
+Video Depth Anything — Chen et al., CVPR 2025。由 [猫斯卡](https://www.maosika.com) 出品。
diff --git a/docs/demo.gif b/docs/demo.gif
new file mode 100644
index 0000000..efb563a
Binary files /dev/null and b/docs/demo.gif differ
diff --git a/docs/example_city_night.jpg b/docs/example_city_night.jpg
new file mode 100644
index 0000000..7f2250f
Binary files /dev/null and b/docs/example_city_night.jpg differ
diff --git a/docs/example_city_night_3s.jpg b/docs/example_city_night_3s.jpg
new file mode 100644
index 0000000..13274d0
Binary files /dev/null and b/docs/example_city_night_3s.jpg differ
diff --git a/examples/city_night.mp4 b/examples/city_night.mp4
new file mode 100644
index 0000000..b8a518c
Binary files /dev/null and b/examples/city_night.mp4 differ
diff --git a/examples/land_16x9.mp4 b/examples/land_16x9.mp4
new file mode 100644
index 0000000..31b2a55
Binary files /dev/null and b/examples/land_16x9.mp4 differ
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..2eef4df
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,47 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "video2blockout"
+version = "0.1.0"
+description = "Turn any video into a depth blockout video for video-generation ControlNets (MiniMax H3, Wan VACE, ...)."
+readme = "README.md"
+license = { text = "Apache-2.0" }
+requires-python = ">=3.10"
+authors = [{ name = "Maoska" }]
+keywords = ["video", "depth", "controlnet", "video-generation", "blockout", "depth-anything"]
+classifiers = [
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3",
+ "Topic :: Multimedia :: Video",
+]
+dependencies = [
+ "torch>=2.1",
+ "numpy>=1.24",
+ "opencv-python-headless>=4.8",
+ "einops>=0.4",
+ "easydict",
+ "tqdm",
+ "huggingface_hub>=0.20",
+]
+
+[project.optional-dependencies]
+ffmpeg = ["imageio-ffmpeg>=0.4.7"] # bundled ffmpeg if none on PATH
+dev = ["pytest>=8", "ruff"]
+
+[project.scripts]
+video2blockout = "video2blockout.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/maosika-ai/video2blockout"
+
+[tool.setuptools.packages.find]
+include = ["video2blockout*"]
+
+[tool.setuptools.package-data]
+"video2blockout.third_party.video_depth_anything" = ["LICENSE"]
+
+[tool.ruff]
+line-length = 110
+extend-exclude = ["video2blockout/third_party"]
diff --git a/tests/test_io.py b/tests/test_io.py
new file mode 100644
index 0000000..a2cab78
--- /dev/null
+++ b/tests/test_io.py
@@ -0,0 +1,80 @@
+"""ffmpeg round-trip on synthetic frames — no model."""
+
+import cv2
+import numpy as np
+import pytest
+
+from video2blockout.io import read_video, write_gray_video
+
+def _has_ffmpeg():
+ try:
+ from video2blockout.io import ffmpeg_binary
+
+ return bool(ffmpeg_binary())
+ except Exception:
+ return False
+
+
+pytestmark = pytest.mark.skipif(not _has_ffmpeg(), reason="no ffmpeg (PATH or imageio-ffmpeg)")
+
+
+def test_gray_video_roundtrip(tmp_path):
+ t, h, w = 12, 64, 96
+ ramp = np.linspace(0, 255, w, dtype=np.float32)
+ gray = np.broadcast_to(ramp, (t, h, w)).astype(np.uint8).copy()
+ out = write_gray_video(gray, tmp_path / "g.mp4", fps=24)
+
+ cap = cv2.VideoCapture(str(out))
+ assert int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) == t
+ ok, frame = cap.read()
+ cap.release()
+ assert ok and frame.shape[:2] == (h, w)
+ # yuv420p + x264 at CRF 12 should keep a smooth ramp within a couple of levels
+ err = np.abs(frame[:, :, 0].astype(int) - gray[0].astype(int))
+ assert err.mean() < 2.0
+
+
+def test_write_rejects_odd_dimensions(tmp_path):
+ with pytest.raises(ValueError):
+ write_gray_video(np.zeros((1, 63, 64), np.uint8), tmp_path / "x.mp4", 24)
+
+
+def test_read_video_resamples_by_timestamp(tmp_path):
+ # 30 fps, 30 frames → 1 s; ask for 24 fps and expect 24 frames, not 30 (stride=1 trap)
+ p = tmp_path / "src.mp4"
+ vw = cv2.VideoWriter(str(p), cv2.VideoWriter_fourcc(*"mp4v"), 30, (64, 48))
+ for i in range(30):
+ vw.write(np.full((48, 64, 3), i * 8, np.uint8))
+ vw.release()
+ frames, fps = read_video(p, target_fps=24)
+ assert fps == 24 and len(frames) == 24
+
+
+def test_read_video_caps_longer_side(tmp_path):
+ p = tmp_path / "big.mp4"
+ vw = cv2.VideoWriter(str(p), cv2.VideoWriter_fourcc(*"mp4v"), 24, (400, 200))
+ for _ in range(3):
+ vw.write(np.zeros((200, 400, 3), np.uint8))
+ vw.release()
+ frames, _ = read_video(p, max_res=100)
+ assert frames.shape[1:3] == (50, 100)
+
+
+def test_write_accepts_frame_iterator_with_size(tmp_path):
+ frames = (np.full((32, 48), i * 7, np.uint8) for i in range(6))
+ out = write_gray_video(frames, tmp_path / "it.mp4", 24, size=(32, 48))
+ cap = cv2.VideoCapture(str(out))
+ assert int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) == 6
+ cap.release()
+
+
+def test_probe_video(tmp_path):
+ p = tmp_path / "p.mp4"
+ vw = cv2.VideoWriter(str(p), cv2.VideoWriter_fourcc(*"mp4v"), 25, (64, 48))
+ for _ in range(5):
+ vw.write(np.zeros((48, 64, 3), np.uint8))
+ vw.release()
+ from video2blockout.io import probe_video
+
+ w, h, fps, n = probe_video(p)
+ assert (w, h, n) == (64, 48, 5) and abs(fps - 25) < 0.01
diff --git a/tests/test_model_smoke.py b/tests/test_model_smoke.py
new file mode 100644
index 0000000..879ef2f
--- /dev/null
+++ b/tests/test_model_smoke.py
@@ -0,0 +1,49 @@
+"""End-to-end with the real Small model on CPU. Opt-in (downloads 111 MB):
+
+ RUN_MODEL_TESTS=1 pytest tests/test_model_smoke.py
+"""
+
+import os
+
+import cv2
+import numpy as np
+import pytest
+
+pytestmark = pytest.mark.skipif(not os.environ.get("RUN_MODEL_TESTS"), reason="set RUN_MODEL_TESTS=1")
+
+
+def _synthetic_clip(path, frames=8, w=224, h=160):
+ vw = cv2.VideoWriter(str(path), cv2.VideoWriter_fourcc(*"mp4v"), 24, (w, h))
+ for i in range(frames):
+ img = np.full((h, w, 3), 40, np.uint8)
+ # a bright square that drifts right, on a gradient background
+ img[:, :, 1] = np.linspace(20, 200, w, dtype=np.uint8)
+ x = 20 + i * 8
+ cv2.rectangle(img, (x, 50), (x + 60, 120), (230, 230, 230), -1)
+ vw.write(img)
+ vw.release()
+
+
+def test_extract_and_write(tmp_path):
+ from video2blockout import extract, to_gray, write_gray_video
+
+ src = tmp_path / "src.mp4"
+ _synthetic_clip(src)
+ depths, fps = extract(str(src), device="cpu")
+ assert depths.shape[0] == 8 and depths.ndim == 3
+ assert np.isfinite(depths).all() and depths.max() > depths.min()
+ gray = to_gray(depths)
+ assert gray.dtype == np.uint8 and gray.max() == 255 and gray.min() == 0
+ out = write_gray_video(gray, tmp_path / "out.mp4", fps)
+ assert out.stat().st_size > 1000
+
+
+def test_cli_end_to_end(tmp_path):
+ from video2blockout.cli import main
+
+ src = tmp_path / "src.mp4"
+ _synthetic_clip(src)
+ out = tmp_path / "out.mp4"
+ metrics = tmp_path / "m.json"
+ assert main([str(src), "-o", str(out), "--device", "cpu", "--target", "h3", "--metrics", str(metrics)]) == 0
+ assert out.exists() and metrics.exists()
diff --git a/tests/test_planning.py b/tests/test_planning.py
new file mode 100644
index 0000000..a31a678
--- /dev/null
+++ b/tests/test_planning.py
@@ -0,0 +1,36 @@
+"""Planning is pure arithmetic — test it without a model."""
+
+from video2blockout.planning import (
+ HOST_RAM_BUDGET_RATIO,
+ estimate_host_bytes,
+ memory_verdict,
+ processing_max_res,
+)
+
+
+def test_processing_res_puts_short_side_at_input_size():
+ # 736×1280 portrait: short side 736 → 518 means long side 1280·518/736 = 900.87 → 900 (even)
+ assert processing_max_res(736, 1280) == 900
+ assert processing_max_res(1280, 736) == 900
+
+
+def test_processing_res_never_upscales():
+ assert processing_max_res(400, 300) == 400
+
+
+def test_estimate_scales_with_pixels_and_frames():
+ assert estimate_host_bytes(2, 100, 100) == 2 * estimate_host_bytes(1, 100, 100)
+ assert estimate_host_bytes(1, 200, 100) == 2 * estimate_host_bytes(1, 100, 100)
+
+
+def test_verdict_refuses_over_budget_and_suggests_frames():
+ ram = 8 * 2**30
+ v = memory_verdict(10_000, 900, 518, ram_bytes=ram)
+ assert not v["ok"]
+ assert v["budget"] == int(ram * HOST_RAM_BUDGET_RATIO)
+ # The suggested frame count itself must fit the budget
+ assert estimate_host_bytes(v["max_frames_ok"], 900, 518) <= v["budget"]
+
+
+def test_verdict_passes_small_job():
+ assert memory_verdict(8, 320, 180, ram_bytes=8 * 2**30)["ok"]
diff --git a/tests/test_postprocess.py b/tests/test_postprocess.py
new file mode 100644
index 0000000..911a823
--- /dev/null
+++ b/tests/test_postprocess.py
@@ -0,0 +1,60 @@
+"""Unit tests that need no model and no GPU."""
+
+import numpy as np
+import pytest
+
+from video2blockout.postprocess import to_gray
+from video2blockout.targets import TARGETS, center_crop, fit_dimensions
+
+
+def test_global_normalisation_not_per_frame():
+ # Frame 0 spans 0..1, frame 1 spans 0..10. Per-frame min/max would map both to
+ # full range; global must keep frame 0 dark.
+ d = np.zeros((2, 4, 4), np.float32)
+ d[0] = np.linspace(0, 1, 16).reshape(4, 4)
+ d[1] = np.linspace(0, 10, 16).reshape(4, 4)
+ g = to_gray(d)
+ assert g[0].max() <= 26 # 1/10 of 255
+ assert g[1].max() == 255
+
+
+def test_near_is_white_by_default_and_invert_flips():
+ d = np.array([[[0.0, 5.0]]], np.float32) # larger = closer (VDA convention)
+ assert to_gray(d)[0, 0].tolist() == [0, 255]
+ assert to_gray(d, invert=True)[0, 0].tolist() == [255, 0]
+
+
+def test_clip_percent_ignores_hot_pixel():
+ # A 0..1 ramp plus one hot pixel at 1000. Exact min/max lets the hot pixel crush the
+ # ramp into the bottom few grey levels; a 2 % tail clip restores its full contrast.
+ d = np.tile(np.linspace(0, 1, 100, dtype=np.float32), (1, 10, 1)).reshape(1, 10, 100)
+ d[0, 0, 0] = 1000.0
+ assert to_gray(d)[0, 5, 99] < 2
+ assert to_gray(d, clip_percent=2)[0, 5, 99] >= 250
+
+
+def test_flat_input_does_not_divide_by_zero():
+ assert to_gray(np.ones((1, 2, 2), np.float32)).max() == 0
+
+
+def test_to_gray_rejects_wrong_rank():
+ with pytest.raises(ValueError):
+ to_gray(np.zeros((2, 2), np.float32))
+
+
[email protected]("h,w,m,exp", [(1280, 736, 32, (1280, 736)), (1080, 1920, 32, (1056, 1920)),
+ (7, 9, 2, (6, 8)), (10, 10, 32, (32, 32))])
+def test_fit_dimensions_rounds_down_to_multiple(h, w, m, exp):
+ assert fit_dimensions(h, w, m) == exp
+
+
+def test_center_crop_is_centred():
+ f = np.arange(6 * 6).reshape(1, 6, 6)
+ c = center_crop(f, 4, 4)
+ assert c.shape == (1, 4, 4)
+ assert c[0, 0, 0] == 7 and c[0, -1, -1] == 28
+
+
+def test_h3_preset_matches_published_constraints():
+ t = TARGETS["h3"]
+ assert (t.fps, t.multiple, t.max_seconds) == (24.0, 32, 15.0)
diff --git a/video2blockout/__init__.py b/video2blockout/__init__.py
new file mode 100644
index 0000000..b57148b
--- /dev/null
+++ b/video2blockout/__init__.py
@@ -0,0 +1,65 @@
+"""video2blockout — turn any video into a depth "blockout" video for video-generation control.
+
+ from video2blockout import extract, to_gray, write_gray_video
+ depths, fps = extract("in.mp4") # float32 [T, H, W], larger = closer
+ write_gray_video(to_gray(depths), "out.mp4", fps)
+"""
+
+from __future__ import annotations
+
+import os
+
+# Apple Silicon: cap the MPS allocator at 60 % of the recommended working set. Unified
+# memory means GPU allocations count against the same RAM as everything else; without
+# a cap a long clip grows until the OS starts killing processes (seen 2026-09-11).
+# With the cap we get a clean MPS out-of-memory error instead. Must be set before
+# torch is imported, which is why it lives here and not in the backend.
+os.environ.setdefault("PYTORCH_MPS_HIGH_WATERMARK_RATIO", "0.6")
+
+import numpy as np
+
+from .backends import VDABackend, pick_device
+from .io import probe_video, read_video, write_gray_video
+from .planning import memory_verdict, processing_max_res
+from .postprocess import to_gray, upsample_frames
+from .targets import TARGETS, center_crop, fit_dimensions
+
+__version__ = "0.1.0"
+
+
+def extract(
+ video: str,
+ *,
+ model: str = "small",
+ device: str = "auto",
+ max_res: int = 1280,
+ target_fps: float = -1,
+ max_frames: int = -1,
+ input_size: int = 518,
+ checkpoint: str | None = None,
+) -> tuple[np.ndarray, float]:
+ """One call: decode → depth. Returns `(depths[T,H,W] float32, fps)`."""
+ # Infer at model resolution regardless of `max_res` — see planning.processing_max_res.
+ w, h, _, _ = probe_video(video)
+ proc = min(processing_max_res(w, h, input_size), max_res if max_res > 0 else 10**9)
+ frames, fps = read_video(video, max_frames=max_frames, target_fps=target_fps, max_res=proc)
+ backend = VDABackend(model, device=device, checkpoint=checkpoint)
+ return backend.infer(frames, fps, input_size=input_size), fps
+
+
+__all__ = [
+ "__version__",
+ "extract",
+ "probe_video",
+ "read_video",
+ "write_gray_video",
+ "to_gray",
+ "upsample_frames",
+ "processing_max_res",
+ "memory_verdict",
+ "TARGETS",
+ "fit_dimensions",
+ "center_crop",
+ "VDABackend",
+ "pick_device",
+]
diff --git a/video2blockout/backends/__init__.py b/video2blockout/backends/__init__.py
new file mode 100644
index 0000000..f8e2ef8
--- /dev/null
+++ b/video2blockout/backends/__init__.py
@@ -0,0 +1,4 @@
+from .base import BackendInfo, DepthBackend
+from .vda import VDABackend, pick_device
+
+__all__ = ["BackendInfo", "DepthBackend", "VDABackend", "pick_device"]
diff --git a/video2blockout/backends/base.py b/video2blockout/backends/base.py
new file mode 100644
index 0000000..f4ba3fb
--- /dev/null
+++ b/video2blockout/backends/base.py
@@ -0,0 +1,30 @@
+"""Backend protocol: a depth model that turns a frame stack into per-frame depth.
+
+Every backend returns *relative inverse depth* (larger = closer to camera) as float32
+`[T, H, W]` at the input frame resolution, already temporally aligned across the whole
+clip. Normalisation to 8-bit grey happens later in :mod:`video2blockout.postprocess`,
+so backends never decide what "white" means.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Protocol
+
+import numpy as np
+
+
+@dataclass(frozen=True)
+class BackendInfo:
+ name: str
+ variant: str
+ license: str
+ commercial_ok: bool
+
+
+class DepthBackend(Protocol):
+ info: BackendInfo
+
+ def infer(self, frames: np.ndarray, fps: float, *, input_size: int = 518) -> np.ndarray:
+ """`frames`: uint8 RGB `[T, H, W, 3]` → float32 inverse depth `[T, H, W]`."""
+ ...
diff --git a/video2blockout/backends/vda.py b/video2blockout/backends/vda.py
new file mode 100644
index 0000000..8b30bda
--- /dev/null
+++ b/video2blockout/backends/vda.py
@@ -0,0 +1,111 @@
+"""Video Depth Anything backend (default).
+
+Why this model: it is the only open video-depth model that ships a temporal module
+*and* an Apache-2.0 checkpoint (Small). Per-frame models (Depth Anything V2/3) flicker
+on video unless you bolt on your own smoothing; diffusion models (DepthCrafter) are
+~100× slower. VDA-Small at 28M params runs on an 8 GB consumer GPU or an Apple M-series.
+
+Precision policy (measured 2026-09-11 on M2 Max, 32 frames @ 736×1280):
+ mps fp32 → 7.0 s mps fp16 → did not finish in 3 min (autocast pathological)
+ cpu fp32 → 59 s
+ Full 294-frame clip end-to-end on mps fp32: 147 s (~500 ms/frame incl. alignment).
+So fp16 is enabled on CUDA only. Do not "optimise" this by turning fp16 on for MPS.
+
+Licence policy: `small` is Apache-2.0. `base`/`large` are CC-BY-NC-4.0 — loading them
+prints a warning and sets ``info.commercial_ok = False``; the CLI surfaces this.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import warnings
+
+import numpy as np
+import torch
+
+from .base import BackendInfo
+
+log = logging.getLogger(__name__)
+
+_VARIANTS = {
+ "small": {"encoder": "vits", "features": 64, "out_channels": [48, 96, 192, 384]},
+ "base": {"encoder": "vitb", "features": 128, "out_channels": [96, 192, 384, 768]},
+ "large": {"encoder": "vitl", "features": 256, "out_channels": [256, 512, 1024, 1024]},
+}
+_LICENSES = {"small": "Apache-2.0", "base": "CC-BY-NC-4.0", "large": "CC-BY-NC-4.0"}
+_HF_REPOS = {
+ "small": "depth-anything/Video-Depth-Anything-Small",
+ "base": "depth-anything/Video-Depth-Anything-Base",
+ "large": "depth-anything/Video-Depth-Anything-Large",
+}
+
+
+def pick_device(requested: str = "auto") -> str:
+ if requested != "auto":
+ return requested
+ if torch.cuda.is_available():
+ return "cuda"
+ if getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
+ return "mps"
+ return "cpu"
+
+
+def _weights_path(variant: str, checkpoint: str | None) -> str:
+ if checkpoint:
+ return checkpoint
+ # huggingface_hub honours HF_ENDPOINT, so users in China can point it at a mirror
+ # (e.g. https://hf-mirror.com) without us special-casing anything.
+ from huggingface_hub import hf_hub_download
+
+ filename = f"video_depth_anything_{_VARIANTS[variant]['encoder']}.pth"
+ return hf_hub_download(_HF_REPOS[variant], filename)
+
+
+class VDABackend:
+ def __init__(
+ self,
+ variant: str = "small",
+ device: str = "auto",
+ checkpoint: str | None = None,
+ ) -> None:
+ if variant not in _VARIANTS:
+ raise ValueError(f"unknown variant {variant!r}; choose from {sorted(_VARIANTS)}")
+ from ..third_party.video_depth_anything.video_depth import VideoDepthAnything
+
+ self.variant = variant
+ self.device = pick_device(device)
+ # fp16 only where it is known to be fast; see module docstring for the measurement.
+ self.fp32 = self.device != "cuda"
+ self.info = BackendInfo(
+ name="video-depth-anything",
+ variant=variant,
+ license=_LICENSES[variant],
+ commercial_ok=_LICENSES[variant].startswith("Apache"),
+ )
+ if not self.info.commercial_ok:
+ warnings.warn(
+ f"Video Depth Anything '{variant}' weights are {_LICENSES[variant]}: "
+ "non-commercial use only. Use 'small' for commercial projects.",
+ stacklevel=2,
+ )
+
+ path = _weights_path(variant, checkpoint)
+ log.info("loading %s from %s on %s (fp32=%s)", variant, path, self.device, self.fp32)
+ model = VideoDepthAnything(**_VARIANTS[variant])
+ model.load_state_dict(torch.load(path, map_location="cpu"), strict=True)
+ self.model = model.to(self.device).eval()
+
+ def infer(self, frames: np.ndarray, fps: float, *, input_size: int = 518) -> np.ndarray:
+ if frames.ndim != 4 or frames.shape[-1] != 3:
+ raise ValueError(f"expected [T, H, W, 3] uint8 RGB, got {frames.shape}")
+ # Upstream's infer_video_depth already does 32-frame windows with 10-frame overlap,
+ # keyframe-based scale/shift alignment across windows and interpolation over the
+ # seam. That is the temporal-consistency machinery — do not re-chunk outside it.
+ depths, _ = self.model.infer_video_depth(
+ frames, fps, input_size=input_size, device=self.device, fp32=self.fp32
+ )
+ return np.asarray(depths, dtype=np.float32)
+
+
+__all__ = ["VDABackend", "pick_device"]
diff --git a/video2blockout/cli.py b/video2blockout/cli.py
new file mode 100644
index 0000000..d0a8e0c
--- /dev/null
+++ b/video2blockout/cli.py
@@ -0,0 +1,174 @@
+"""`video2blockout` command line.
+
+ video2blockout in.mp4 -o out.mp4 # Apache-2.0 Small model, auto device
+ video2blockout in.mp4 -o out.mp4 --target h3 # 24 fps, ×32 dims, ≤15 s for MiniMax H3
+ video2blockout in.mp4 -o out.mp4 --npz depth.npz --metrics run.json
+
+Memory model: depth is estimated at the model's working resolution (short side 518 px)
+and only the 8-bit result is upscaled to the output size, frame by frame while
+encoding. A host-RAM estimate is printed and, if it exceeds half of physical RAM, the
+run is refused with a concrete `--max-frames` suggestion (override with `--force`).
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import os
+import platform
+import resource
+import sys
+import time
+
+import numpy as np
+
+from . import __version__
+from .backends import VDABackend, pick_device
+from .io import probe_video, read_video, write_gray_video
+from .planning import gib, memory_verdict, processing_max_res
+from .postprocess import to_gray, upsample_frames
+from .targets import TARGETS, fit_dimensions
+
+
+def build_parser() -> argparse.ArgumentParser:
+ p = argparse.ArgumentParser(
+ prog="video2blockout",
+ description="Video → depth blockout video for video-generation ControlNets.",
+ )
+ p.add_argument("input", help="input video (anything ffmpeg/OpenCV can read)")
+ p.add_argument("-o", "--output", required=True, help="output .mp4")
+ p.add_argument("--model", default="small", choices=["small", "base", "large"],
+ help="VDA variant; only 'small' is Apache-2.0 (default)")
+ p.add_argument("--device", default="auto", help="auto | cuda | mps | cpu")
+ p.add_argument("--target", default="none", choices=sorted(TARGETS),
+ help="fps / size preset for a specific generator")
+ p.add_argument("--fps", type=float, default=-1, help="override output fps (-1 = preset or source)")
+ p.add_argument("--max-res", type=int, default=1280,
+ help="cap the longer side of the OUTPUT (inference always runs at model resolution)")
+ p.add_argument("--max-frames", type=int, default=-1, help="stop after N frames (-1 = all)")
+ p.add_argument("--input-size", type=int, default=518, help="model short side (multiple of 14)")
+ p.add_argument("--invert", action="store_true", help="far = white instead of near = white")
+ p.add_argument("--clip", type=float, default=0.0, metavar="PCT",
+ help="percentile clip on both tails before scaling (0 = exact min/max)")
+ p.add_argument("--gamma", type=float, default=1.0)
+ p.add_argument("--crf", type=int, default=12, help="x264 CRF; lower = larger, cleaner")
+ p.add_argument("--npz", help="also save raw float depth (at processing resolution) to this .npz")
+ p.add_argument("--metrics", help="write timing / memory / size metrics as JSON to this path")
+ p.add_argument("--checkpoint", help="local .pth instead of the Hugging Face download")
+ p.add_argument("--force", action="store_true", help="run even if the RAM estimate exceeds the budget")
+ p.add_argument("-v", "--verbose", action="store_true")
+ p.add_argument("--version", action="version", version=__version__)
+ return p
+
+
+def _peak_rss_bytes() -> int:
+ ru = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+ # macOS reports bytes, Linux kilobytes.
+ return int(ru if platform.system() == "Darwin" else ru * 1024)
+
+
+def _say(msg: str) -> None:
+ print(msg, file=sys.stderr, flush=True)
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = build_parser().parse_args(argv)
+ logging.basicConfig(level=logging.INFO if args.verbose else logging.WARNING,
+ format="%(levelname)s %(name)s: %(message)s")
+ target = TARGETS[args.target]
+ fps_req = args.fps if args.fps > 0 else (target.fps or -1)
+ metrics: dict = {"version": __version__, "input": args.input, "model": args.model, "target": args.target}
+ t0 = time.time()
+
+ # ── plan: how big will this run be? ────────────────────────────────────────────
+ src_w, src_h, src_fps, src_n = probe_video(args.input)
+ out_fps = src_fps if fps_req <= 0 or fps_req >= src_fps else fps_req
+ n_frames = src_n if args.max_frames <= 0 else min(src_n, args.max_frames)
+ n_frames = int(round(n_frames * out_fps / src_fps)) if out_fps < src_fps else n_frames
+ proc_res = min(processing_max_res(src_w, src_h, args.input_size), args.max_res if args.max_res > 0 else 10**9)
+ scale = min(1.0, proc_res / max(src_w, src_h))
+ proc_w, proc_h = int(round(src_w * scale / 2) * 2), int(round(src_h * scale / 2) * 2)
+ verdict = memory_verdict(n_frames, proc_w, proc_h)
+ _say(f"plan {src_w}x{src_h} @ {src_fps:.3f} fps × {src_n} frames → process {proc_w}x{proc_h} × ~{n_frames} frames, "
+ f"host RAM ≈ {gib(verdict['estimate'])} (budget {gib(verdict['budget'])})")
+ metrics.update(source=[src_w, src_h, src_fps, src_n], process=[proc_w, proc_h, n_frames],
+ ram_estimate=verdict["estimate"], ram_budget=verdict["budget"])
+ if not verdict["ok"] and not args.force:
+ _say(f"abort estimated RAM exceeds budget. Try --max-frames {verdict['max_frames_ok']} "
+ f"(or split the clip), a smaller --input-size, or --force if you know the machine can take it.")
+ return 3
+
+ # ── read ────────────────────────────────────────────────────────────────────────
+ frames, fps = read_video(args.input, max_frames=args.max_frames, target_fps=fps_req, max_res=proc_res)
+ t, h, w = frames.shape[:3]
+ _say(f"read {t} frames @ {fps:.3f} fps, {w}x{h} ({time.time() - t0:.1f}s)")
+ if target.max_seconds and t / fps > target.max_seconds + 1e-6:
+ _say(f"warn clip is {t / fps:.1f}s; {target.key} accepts ≤ {target.max_seconds:.0f}s — trim it")
+
+ # ── depth ───────────────────────────────────────────────────────────────────────
+ device = pick_device(args.device)
+ if os.environ.get("VIDEO2BLOCKOUT_FAKE_BACKEND"):
+ # Dry-run mode for exercising the whole pipeline (I/O, encoding, metrics, remote
+ # runbook) without loading a model. Produces a plausible radial gradient.
+ _say("model FAKE backend (VIDEO2BLOCKOUT_FAKE_BACKEND set) — no inference")
+ yy, xx = np.mgrid[0:h, 0:w].astype(np.float32)
+ base = 1.0 / (1.0 + ((xx - w / 2) ** 2 + (yy - h / 2) ** 2) / (0.15 * w * h))
+ depths = np.stack([base * (0.8 + 0.2 * i / max(1, t - 1)) for i in range(t)]).astype(np.float32)
+ precision, license_ = "none", "n/a"
+ dt = 0.0
+ else:
+ backend = VDABackend(args.model, device=device, checkpoint=args.checkpoint)
+ precision, license_ = ("fp32" if backend.fp32 else "fp16"), backend.info.license
+ _say(f"model {backend.info.name}/{backend.info.variant} on {device} ({precision}), licence {license_}")
+ t1 = time.time()
+ depths = backend.infer(frames, fps, input_size=args.input_size)
+ dt = time.time() - t1
+ del backend
+ _release_accelerator(device)
+ _say(f"depth {dt:.1f}s = {dt / t * 1000:.0f} ms/frame")
+ del frames
+ metrics.update(device=device, precision=precision, license=license_, frames=t, fps=fps,
+ depth_seconds=round(dt, 2), ms_per_frame=round(dt / t * 1000, 1))
+
+ if args.npz:
+ np.savez_compressed(args.npz, depths=depths, fps=fps)
+ _say(f"npz {args.npz}")
+
+ # ── grey → output size → encode (frame by frame) ────────────────────────────────
+ gray = to_gray(depths, invert=args.invert, clip_percent=args.clip, gamma=args.gamma)
+ del depths
+ out_scale = min(1.0, (args.max_res if args.max_res > 0 else 10**9) / max(src_w, src_h))
+ out_h, out_w = fit_dimensions(int(src_h * out_scale), int(src_w * out_scale), target.multiple)
+ if (out_h, out_w) != (h, w):
+ _say(f"size {w}x{h} → {out_w}x{out_h} (multiple of {target.multiple})")
+ out = write_gray_video(upsample_frames(gray, out_h, out_w), args.output, fps, crf=args.crf, size=(out_h, out_w))
+ total = time.time() - t0
+ peak = _peak_rss_bytes()
+ _say(f"wrote {out} ({total:.1f}s total, peak RSS {gib(peak)})")
+ metrics.update(output=str(out), output_size=[out_w, out_h], total_seconds=round(total, 1),
+ peak_rss_bytes=peak, output_bytes=os.path.getsize(out))
+ if args.metrics:
+ with open(args.metrics, "w", encoding="utf-8") as fh:
+ json.dump(metrics, fh, ensure_ascii=False, indent=2)
+ _say(f"metrics {args.metrics}")
+ if license_ not in ("Apache-2.0", "n/a"):
+ _say("note weights are non-commercial; use --model small for commercial work")
+ return 0
+
+
+def _release_accelerator(device: str) -> None:
+ """Give cached GPU/MPS memory back before encoding; harmless on CPU."""
+ try:
+ import torch
+
+ if device == "cuda":
+ torch.cuda.empty_cache()
+ elif device == "mps":
+ torch.mps.empty_cache()
+ except Exception: # pragma: no cover
+ pass
+
+
+if __name__ == "__main__": # pragma: no cover
+ sys.exit(main())
diff --git a/video2blockout/io.py b/video2blockout/io.py
new file mode 100644
index 0000000..31724d8
--- /dev/null
+++ b/video2blockout/io.py
@@ -0,0 +1,156 @@
+"""Video in / video out.
+
+Reading uses OpenCV (no decord dependency — decord has no arm64 macOS wheels).
+Writing pipes raw 8-bit grey frames into ffmpeg. We deliberately do not go through
+imageio's writer: we need explicit control over pixel format, GOP length and CRF,
+because the output is a *control signal*, not a video for humans.
+"""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+from pathlib import Path
+from typing import Iterable
+
+import cv2
+import numpy as np
+
+
+def probe_video(path: str | Path) -> tuple[int, int, float, int]:
+ """Cheap header read → `(width, height, fps, frame_count)`; nothing is decoded."""
+ cap = cv2.VideoCapture(str(path))
+ if not cap.isOpened():
+ raise FileNotFoundError(path)
+ try:
+ return (
+ int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)),
+ int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)),
+ float(cap.get(cv2.CAP_PROP_FPS) or 30.0),
+ int(cap.get(cv2.CAP_PROP_FRAME_COUNT)),
+ )
+ finally:
+ cap.release()
+
+
+def read_video(
+ path: str | Path,
+ *,
+ max_frames: int = -1,
+ target_fps: float = -1,
+ max_res: int = -1,
+) -> tuple[np.ndarray, float]:
+ """Decode to uint8 RGB `[T, H, W, 3]` and return `(frames, fps)`.
+
+ `target_fps` picks frames by *timestamp*, not by integer stride, so 30 → 24 fps
+ actually yields 24 fps instead of silently staying at 30 (which is what a
+ round(30/24)=1 stride does).
+ `max_res` downsizes the longer side; depth is estimated at that size and the
+ output video is written at that size too.
+ """
+ cap = cv2.VideoCapture(str(path))
+ if not cap.isOpened():
+ raise FileNotFoundError(path)
+ src_fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
+ out_fps = src_fps if target_fps <= 0 or target_fps >= src_fps else float(target_fps)
+
+ frames: list[np.ndarray] = []
+ src_index = 0
+ next_pick_time = 0.0
+ while True:
+ ok, bgr = cap.read()
+ if not ok:
+ break
+ t = src_index / src_fps
+ src_index += 1
+ if out_fps < src_fps and t + 1e-9 < next_pick_time:
+ continue
+ next_pick_time += 1.0 / out_fps
+ if max_res > 0 and max(bgr.shape[:2]) > max_res:
+ scale = max_res / max(bgr.shape[:2])
+ # Even dims keep yuv420p encoders happy later; rounding here avoids a crop.
+ new_w = int(round(bgr.shape[1] * scale / 2) * 2)
+ new_h = int(round(bgr.shape[0] * scale / 2) * 2)
+ bgr = cv2.resize(bgr, (new_w, new_h), interpolation=cv2.INTER_AREA)
+ frames.append(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))
+ if 0 < max_frames <= len(frames):
+ break
+ cap.release()
+ if not frames:
+ raise ValueError(f"no frames decoded from {path}")
+ return np.stack(frames), out_fps
+
+
+def ffmpeg_binary() -> str:
+ exe = shutil.which("ffmpeg")
+ if exe:
+ return exe
+ try:
+ import imageio_ffmpeg
+
+ return imageio_ffmpeg.get_ffmpeg_exe()
+ except Exception as exc: # pragma: no cover - environment dependent
+ raise RuntimeError("ffmpeg not found; install ffmpeg or `pip install imageio-ffmpeg`") from exc
+
+
+def write_gray_video(
+ gray: "np.ndarray | Iterable[np.ndarray]",
+ path: str | Path,
+ fps: float,
+ *,
+ crf: int = 12,
+ keyint_seconds: float = 2.0,
+ size: tuple[int, int] | None = None,
+) -> Path:
+ """Encode uint8 `[T, H, W]` as H.264 (yuv420p, neutral chroma).
+
+ CRF 12 is generous on purpose: depth maps are large smooth gradients, exactly the
+ content where quantisation shows up as banding, and banding in a control video
+ becomes jitter in the generated clip. Measured cost is small (~0.5–1 Mbps at 720p).
+ A key frame every `keyint_seconds` lets downstream tools cut the clip anywhere.
+ """
+ # Accept either a whole [T, H, W] array or a lazy frame iterator (with `size`), so
+ # the caller can upsample frame by frame instead of materialising the full-res clip.
+ if isinstance(gray, np.ndarray):
+ if gray.ndim != 3 or gray.dtype != np.uint8:
+ raise ValueError(f"expected uint8 [T, H, W], got {gray.shape} {gray.dtype}")
+ h, w = gray.shape[1], gray.shape[2]
+ frames: Iterable[np.ndarray] = gray
+ else:
+ if size is None:
+ raise ValueError("size=(h, w) is required when passing a frame iterator")
+ h, w = size
+ frames = gray
+ if h % 2 or w % 2:
+ raise ValueError("frame size must be even for yuv420p; use targets.fit_dimensions first")
+ path = Path(path)
+ path.parent.mkdir(parents=True, exist_ok=True)
+ cmd = [
+ ffmpeg_binary(), "-y", "-loglevel", "error",
+ "-f", "rawvideo", "-pix_fmt", "gray", "-s", f"{w}x{h}", "-r", f"{fps:.6f}", "-i", "-",
+ "-an",
+ "-c:v", "libx264", "-preset", "slow", "-crf", str(crf),
+ "-pix_fmt", "yuv420p", "-profile:v", "high",
+ "-g", str(max(1, int(round(fps * keyint_seconds)))),
+ "-movflags", "+faststart",
+ str(path),
+ ]
+ proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
+ assert proc.stdin is not None and proc.stderr is not None
+ try:
+ # Stream frame by frame so a 300-frame 720p clip doesn't need a 300 MB
+ # intermediate bytes object. BrokenPipe means ffmpeg died early; its stderr
+ # (read below) carries the real reason, so swallow the pipe error here.
+ for frame in frames:
+ if frame.shape != (h, w) or frame.dtype != np.uint8:
+ raise ValueError(f"frame {frame.shape} {frame.dtype} does not match {(h, w)} uint8")
+ proc.stdin.write(np.ascontiguousarray(frame).tobytes())
+ except BrokenPipeError:
+ pass
+ finally:
+ proc.stdin.close()
+ err = proc.stderr.read()
+ proc.wait()
+ if proc.returncode != 0:
+ raise RuntimeError(f"ffmpeg failed: {err.decode(errors='replace')}")
+ return path
diff --git a/video2blockout/planning.py b/video2blockout/planning.py
new file mode 100644
index 0000000..85cdbef
--- /dev/null
+++ b/video2blockout/planning.py
@@ -0,0 +1,76 @@
+"""Decide *how big* to run before touching the GPU: processing resolution and a host-RAM
+estimate. Pure functions, unit-tested without a model.
+
+Why this module exists: on 2026-09-11 a 12 s 736×1280 clip took the reference pipeline to
+20.8 GB of RSS on a 32 GB Mac and the OS killed the desktop. The model itself only ever
+sees a 518 px short side; everything above that was full-resolution float32 depth kept
+for every frame. So: infer at model resolution, estimate memory up front, refuse loudly
+instead of swapping the machine to death.
+"""
+
+from __future__ import annotations
+
+import os
+
+#: Host bytes per (frame × pixel at processing resolution). Measured on the reference
+#: pipeline (RSS 8.2 GB / (294 frames × 1280 × 736 px) ≈ 30 B). Kept deliberately
+#: conservative until re-measured with the model-resolution path; recalibrate from
+#: `metrics.jsonl` of a real run, not by guessing.
+HOST_BYTES_PER_PIXEL_FRAME = 32.0
+
+#: Refuse when the estimate exceeds this share of physical RAM. Half, not 90 %: the
+#: OS, the browser and the GPU driver (unified memory on Apple Silicon) all need room.
+HOST_RAM_BUDGET_RATIO = 0.5
+
+
+def processing_max_res(width: int, height: int, input_size: int = 518) -> int:
+ """Longest side (even) at which the *short* side equals `input_size`.
+
+ Feeding frames larger than this is pure waste: the model resizes the short side to
+ `input_size` anyway, and the extra pixels only inflate the float depth we keep.
+ """
+ short, long = min(width, height), max(width, height)
+ if short <= input_size:
+ return long
+ scaled = long * input_size / short
+ return int(round(scaled / 2) * 2)
+
+
+def estimate_host_bytes(frames: int, proc_w: int, proc_h: int) -> int:
+ return int(frames * proc_w * proc_h * HOST_BYTES_PER_PIXEL_FRAME)
+
+
+def physical_ram_bytes() -> int:
+ """Total RAM; falls back to 16 GB when the platform won't tell us."""
+ try:
+ import psutil # optional
+
+ return int(psutil.virtual_memory().total)
+ except Exception:
+ pass
+ try:
+ pages = os.sysconf("SC_PHYS_PAGES")
+ page = os.sysconf("SC_PAGE_SIZE")
+ if pages > 0 and page > 0:
+ return int(pages * page)
+ except (ValueError, OSError, AttributeError):
+ pass
+ return 16 * 2**30
+
+
+def memory_verdict(frames: int, proc_w: int, proc_h: int, ram_bytes: int | None = None) -> dict:
+ """`{"estimate", "budget", "ok", "max_frames_ok"}` — the CLI turns this into text."""
+ ram = ram_bytes if ram_bytes is not None else physical_ram_bytes()
+ budget = int(ram * HOST_RAM_BUDGET_RATIO)
+ est = estimate_host_bytes(frames, proc_w, proc_h)
+ per_frame = max(1, estimate_host_bytes(1, proc_w, proc_h))
+ return {
+ "estimate": est,
+ "budget": budget,
+ "ok": est <= budget,
+ "max_frames_ok": max(1, budget // per_frame),
+ }
+
+
+def gib(n: int | float) -> str:
+ return f"{n / 2**30:.1f} GiB"
diff --git a/video2blockout/postprocess.py b/video2blockout/postprocess.py
new file mode 100644
index 0000000..347bbdd
--- /dev/null
+++ b/video2blockout/postprocess.py
@@ -0,0 +1,65 @@
+"""Float depth → 8-bit grey the way video ControlNets expect it.
+
+Two rules, both learned the hard way:
+
+1. **Normalise over the whole clip, never per frame.** Per-frame min/max makes the
+ brightness of a static wall change whenever someone walks closer to the camera,
+ and the generator reads that as the scene "breathing".
+2. **Near is white.** Video Depth Anything outputs inverse depth (larger = closer), and
+ depth ControlNets (MiniMax H3 Fun, Wan VACE, SD ControlNet-depth) were trained on
+ that convention. `invert=True` exists for tools that want the opposite.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+
+def to_gray(
+ depths: np.ndarray,
+ *,
+ invert: bool = False,
+ clip_percent: float = 0.0,
+ gamma: float = 1.0,
+) -> np.ndarray:
+ """`[T, H, W]` float → `[T, H, W]` uint8.
+
+ `clip_percent` (0–10) trims that many percent from each tail before scaling, so a
+ single hot pixel can't crush the contrast of the whole clip. 0 reproduces the
+ upstream reference behaviour exactly.
+ `gamma` > 1 pushes mid-tones darker (more separation near the camera); 1 is linear.
+ """
+ d = np.asarray(depths, dtype=np.float32)
+ if d.ndim != 3:
+ raise ValueError(f"expected [T, H, W], got {d.shape}")
+ if not 0 <= clip_percent < 50:
+ raise ValueError("clip_percent must be in [0, 50)")
+ if clip_percent > 0:
+ lo, hi = np.percentile(d, (clip_percent, 100 - clip_percent))
+ else:
+ lo, hi = float(d.min()), float(d.max())
+ if hi - lo < 1e-6:
+ return np.zeros(d.shape, dtype=np.uint8)
+ n = np.clip((d - lo) / (hi - lo), 0.0, 1.0)
+ if gamma != 1.0:
+ n = n ** float(gamma)
+ if invert:
+ n = 1.0 - n
+ return (n * 255.0 + 0.5).astype(np.uint8)
+
+
+def upsample_frames(gray: np.ndarray, height: int, width: int):
+ """Yield each `[H, W]` uint8 frame resized to `(height, width)`.
+
+ Depth is estimated at the model's working resolution (short side 518 px) and only
+ the 8-bit result is scaled up — the upstream reference upsamples the *float* depth
+ of every frame to full size before alignment, which is what made a 12 s 720p clip
+ need 8–20 GB of RAM. Bilinear on 8-bit grey is visually identical for a control video.
+ """
+ import cv2
+
+ for frame in gray:
+ if frame.shape == (height, width):
+ yield frame
+ else:
+ yield cv2.resize(frame, (width, height), interpolation=cv2.INTER_LINEAR)
diff --git a/video2blockout/targets.py b/video2blockout/targets.py
new file mode 100644
index 0000000..9b34f4c
--- /dev/null
+++ b/video2blockout/targets.py
@@ -0,0 +1,45 @@
+"""Output presets for specific video-generation models.
+
+A control video only helps if its frame rate and size match what the generator was
+trained to consume; otherwise the generator resamples it and you get drift. Values
+here are the published constraints of each model's control/ControlNet path.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import numpy as np
+
+
+@dataclass(frozen=True)
+class Target:
+ key: str
+ fps: float | None # None = keep source fps
+ multiple: int # frame size rounded down to a multiple of this
+ max_seconds: float | None # warn beyond this
+ note: str
+
+
+TARGETS: dict[str, Target] = {
+ "none": Target("none", None, 2, None, "keep source fps; only make dimensions even"),
+ # MiniMax-H3-Fun-Controlnet-Union: 24 fps fixed, H/W multiples of 32, ≤15 s.
+ "h3": Target("h3", 24.0, 32, 15.0, "MiniMax H3 Fun ControlNet (depth/pose/canny/hed/mlsd)"),
+ # Wan 2.1 VACE control videos: 16 fps, H/W multiples of 16.
+ "wan": Target("wan", 16.0, 16, None, "Wan 2.1 VACE control video"),
+}
+
+
+def fit_dimensions(h: int, w: int, multiple: int) -> tuple[int, int]:
+ """Largest (h, w) ≤ input that are multiples of `multiple` (min one block)."""
+ return max(multiple, h // multiple * multiple), max(multiple, w // multiple * multiple)
+
+
+def center_crop(frames: np.ndarray, h: int, w: int) -> np.ndarray:
+ """Crop `[T, H, W, ...]` to `(h, w)` around the centre.
+
+ Crop, not pad: a padded black border reads to the generator as a far wall.
+ """
+ H, W = frames.shape[1], frames.shape[2]
+ top, left = (H - h) // 2, (W - w) // 2
+ return frames[:, top : top + h, left : left + w]
diff --git a/video2blockout/third_party/__init__.py b/video2blockout/third_party/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/video2blockout/third_party/video_depth_anything/LICENSE b/video2blockout/third_party/video_depth_anything/LICENSE
new file mode 100644
index 0000000..f49a4e1
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
\ No newline at end of file
diff --git a/video2blockout/third_party/video_depth_anything/__init__.py b/video2blockout/third_party/video_depth_anything/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/video2blockout/third_party/video_depth_anything/dinov2.py b/video2blockout/third_party/video_depth_anything/dinov2.py
new file mode 100644
index 0000000..83d2508
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dinov2.py
@@ -0,0 +1,415 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the Apache License, Version 2.0
+# found in the LICENSE file in the root directory of this source tree.
+
+# References:
+# https://github.com/facebookresearch/dino/blob/main/vision_transformer.py
+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/models/vision_transformer.py
+
+from functools import partial
+import math
+import logging
+from typing import Sequence, Tuple, Union, Callable
+
+import torch
+import torch.nn as nn
+import torch.utils.checkpoint
+from torch.nn.init import trunc_normal_
+
+from .dinov2_layers import Mlp, PatchEmbed, SwiGLUFFNFused, MemEffAttention, NestedTensorBlock as Block
+
+
+logger = logging.getLogger("dinov2")
+
+
+def named_apply(fn: Callable, module: nn.Module, name="", depth_first=True, include_root=False) -> nn.Module:
+ if not depth_first and include_root:
+ fn(module=module, name=name)
+ for child_name, child_module in module.named_children():
+ child_name = ".".join((name, child_name)) if name else child_name
+ named_apply(fn=fn, module=child_module, name=child_name, depth_first=depth_first, include_root=True)
+ if depth_first and include_root:
+ fn(module=module, name=name)
+ return module
+
+
+class BlockChunk(nn.ModuleList):
+ def forward(self, x):
+ for b in self:
+ x = b(x)
+ return x
+
+
+class DinoVisionTransformer(nn.Module):
+ def __init__(
+ self,
+ img_size=224,
+ patch_size=16,
+ in_chans=3,
+ embed_dim=768,
+ depth=12,
+ num_heads=12,
+ mlp_ratio=4.0,
+ qkv_bias=True,
+ ffn_bias=True,
+ proj_bias=True,
+ drop_path_rate=0.0,
+ drop_path_uniform=False,
+ init_values=None, # for layerscale: None or 0 => no layerscale
+ embed_layer=PatchEmbed,
+ act_layer=nn.GELU,
+ block_fn=Block,
+ ffn_layer="mlp",
+ block_chunks=1,
+ num_register_tokens=0,
+ interpolate_antialias=False,
+ interpolate_offset=0.1,
+ ):
+ """
+ Args:
+ img_size (int, tuple): input image size
+ patch_size (int, tuple): patch size
+ in_chans (int): number of input channels
+ embed_dim (int): embedding dimension
+ depth (int): depth of transformer
+ num_heads (int): number of attention heads
+ mlp_ratio (int): ratio of mlp hidden dim to embedding dim
+ qkv_bias (bool): enable bias for qkv if True
+ proj_bias (bool): enable bias for proj in attn if True
+ ffn_bias (bool): enable bias for ffn if True
+ drop_path_rate (float): stochastic depth rate
+ drop_path_uniform (bool): apply uniform drop rate across blocks
+ weight_init (str): weight init scheme
+ init_values (float): layer-scale init values
+ embed_layer (nn.Module): patch embedding layer
+ act_layer (nn.Module): MLP activation layer
+ block_fn (nn.Module): transformer block class
+ ffn_layer (str): "mlp", "swiglu", "swiglufused" or "identity"
+ block_chunks: (int) split block sequence into block_chunks units for FSDP wrap
+ num_register_tokens: (int) number of extra cls tokens (so-called "registers")
+ interpolate_antialias: (str) flag to apply anti-aliasing when interpolating positional embeddings
+ interpolate_offset: (float) work-around offset to apply when interpolating positional embeddings
+ """
+ super().__init__()
+ norm_layer = partial(nn.LayerNorm, eps=1e-6)
+
+ self.num_features = self.embed_dim = embed_dim # num_features for consistency with other models
+ self.num_tokens = 1
+ self.n_blocks = depth
+ self.num_heads = num_heads
+ self.patch_size = patch_size
+ self.num_register_tokens = num_register_tokens
+ self.interpolate_antialias = interpolate_antialias
+ self.interpolate_offset = interpolate_offset
+
+ self.patch_embed = embed_layer(img_size=img_size, patch_size=patch_size, in_chans=in_chans, embed_dim=embed_dim)
+ num_patches = self.patch_embed.num_patches
+
+ self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
+ self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + self.num_tokens, embed_dim))
+ assert num_register_tokens >= 0
+ self.register_tokens = (
+ nn.Parameter(torch.zeros(1, num_register_tokens, embed_dim)) if num_register_tokens else None
+ )
+
+ if drop_path_uniform is True:
+ dpr = [drop_path_rate] * depth
+ else:
+ dpr = [x.item() for x in torch.linspace(0, drop_path_rate, depth)] # stochastic depth decay rule
+
+ if ffn_layer == "mlp":
+ logger.info("using MLP layer as FFN")
+ ffn_layer = Mlp
+ elif ffn_layer == "swiglufused" or ffn_layer == "swiglu":
+ logger.info("using SwiGLU layer as FFN")
+ ffn_layer = SwiGLUFFNFused
+ elif ffn_layer == "identity":
+ logger.info("using Identity layer as FFN")
+
+ def f(*args, **kwargs):
+ return nn.Identity()
+
+ ffn_layer = f
+ else:
+ raise NotImplementedError
+
+ blocks_list = [
+ block_fn(
+ dim=embed_dim,
+ num_heads=num_heads,
+ mlp_ratio=mlp_ratio,
+ qkv_bias=qkv_bias,
+ proj_bias=proj_bias,
+ ffn_bias=ffn_bias,
+ drop_path=dpr[i],
+ norm_layer=norm_layer,
+ act_layer=act_layer,
+ ffn_layer=ffn_layer,
+ init_values=init_values,
+ )
+ for i in range(depth)
+ ]
+ if block_chunks > 0:
+ self.chunked_blocks = True
+ chunked_blocks = []
+ chunksize = depth // block_chunks
+ for i in range(0, depth, chunksize):
+ # this is to keep the block index consistent if we chunk the block list
+ chunked_blocks.append([nn.Identity()] * i + blocks_list[i : i + chunksize])
+ self.blocks = nn.ModuleList([BlockChunk(p) for p in chunked_blocks])
+ else:
+ self.chunked_blocks = False
+ self.blocks = nn.ModuleList(blocks_list)
+
+ self.norm = norm_layer(embed_dim)
+ self.head = nn.Identity()
+
+ self.mask_token = nn.Parameter(torch.zeros(1, embed_dim))
+
+ self.init_weights()
+
+ def init_weights(self):
+ trunc_normal_(self.pos_embed, std=0.02)
+ nn.init.normal_(self.cls_token, std=1e-6)
+ if self.register_tokens is not None:
+ nn.init.normal_(self.register_tokens, std=1e-6)
+ named_apply(init_weights_vit_timm, self)
+
+ def interpolate_pos_encoding(self, x, w, h):
+ previous_dtype = x.dtype
+ npatch = x.shape[1] - 1
+ N = self.pos_embed.shape[1] - 1
+ if npatch == N and w == h:
+ return self.pos_embed
+ pos_embed = self.pos_embed.float()
+ class_pos_embed = pos_embed[:, 0]
+ patch_pos_embed = pos_embed[:, 1:]
+ dim = x.shape[-1]
+ w0 = w // self.patch_size
+ h0 = h // self.patch_size
+ # we add a small number to avoid floating point error in the interpolation
+ # see discussion at https://github.com/facebookresearch/dino/issues/8
+ # DINOv2 with register modify the interpolate_offset from 0.1 to 0.0
+ w0, h0 = w0 + self.interpolate_offset, h0 + self.interpolate_offset
+ # w0, h0 = w0 + 0.1, h0 + 0.1
+
+ sqrt_N = math.sqrt(N)
+ sx, sy = float(w0) / sqrt_N, float(h0) / sqrt_N
+ patch_pos_embed = nn.functional.interpolate(
+ patch_pos_embed.reshape(1, int(sqrt_N), int(sqrt_N), dim).permute(0, 3, 1, 2),
+ scale_factor=(sx, sy),
+ # (int(w0), int(h0)), # to solve the upsampling shape issue
+ mode="bicubic",
+ antialias=self.interpolate_antialias
+ )
+
+ assert int(w0) == patch_pos_embed.shape[-2]
+ assert int(h0) == patch_pos_embed.shape[-1]
+ patch_pos_embed = patch_pos_embed.permute(0, 2, 3, 1).view(1, -1, dim)
+ return torch.cat((class_pos_embed.unsqueeze(0), patch_pos_embed), dim=1).to(previous_dtype)
+
+ def prepare_tokens_with_masks(self, x, masks=None):
+ B, nc, w, h = x.shape
+ x = self.patch_embed(x)
+ if masks is not None:
+ x = torch.where(masks.unsqueeze(-1), self.mask_token.to(x.dtype).unsqueeze(0), x)
+
+ x = torch.cat((self.cls_token.expand(x.shape[0], -1, -1), x), dim=1)
+ x = x + self.interpolate_pos_encoding(x, w, h)
+
+ if self.register_tokens is not None:
+ x = torch.cat(
+ (
+ x[:, :1],
+ self.register_tokens.expand(x.shape[0], -1, -1),
+ x[:, 1:],
+ ),
+ dim=1,
+ )
+
+ return x
+
+ def forward_features_list(self, x_list, masks_list):
+ x = [self.prepare_tokens_with_masks(x, masks) for x, masks in zip(x_list, masks_list)]
+ for blk in self.blocks:
+ x = blk(x)
+
+ all_x = x
+ output = []
+ for x, masks in zip(all_x, masks_list):
+ x_norm = self.norm(x)
+ output.append(
+ {
+ "x_norm_clstoken": x_norm[:, 0],
+ "x_norm_regtokens": x_norm[:, 1 : self.num_register_tokens + 1],
+ "x_norm_patchtokens": x_norm[:, self.num_register_tokens + 1 :],
+ "x_prenorm": x,
+ "masks": masks,
+ }
+ )
+ return output
+
+ def forward_features(self, x, masks=None):
+ if isinstance(x, list):
+ return self.forward_features_list(x, masks)
+
+ x = self.prepare_tokens_with_masks(x, masks)
+
+ for blk in self.blocks:
+ x = blk(x)
+
+ x_norm = self.norm(x)
+ return {
+ "x_norm_clstoken": x_norm[:, 0],
+ "x_norm_regtokens": x_norm[:, 1 : self.num_register_tokens + 1],
+ "x_norm_patchtokens": x_norm[:, self.num_register_tokens + 1 :],
+ "x_prenorm": x,
+ "masks": masks,
+ }
+
+ def _get_intermediate_layers_not_chunked(self, x, n=1):
+ x = self.prepare_tokens_with_masks(x)
+ # If n is an int, take the n last blocks. If it's a list, take them
+ output, total_block_len = [], len(self.blocks)
+ blocks_to_take = range(total_block_len - n, total_block_len) if isinstance(n, int) else n
+ for i, blk in enumerate(self.blocks):
+ x = blk(x)
+ if i in blocks_to_take:
+ output.append(x)
+ assert len(output) == len(blocks_to_take), f"only {len(output)} / {len(blocks_to_take)} blocks found"
+ return output
+
+ def _get_intermediate_layers_chunked(self, x, n=1):
+ x = self.prepare_tokens_with_masks(x)
+ output, i, total_block_len = [], 0, len(self.blocks[-1])
+ # If n is an int, take the n last blocks. If it's a list, take them
+ blocks_to_take = range(total_block_len - n, total_block_len) if isinstance(n, int) else n
+ for block_chunk in self.blocks:
+ for blk in block_chunk[i:]: # Passing the nn.Identity()
+ x = blk(x)
+ if i in blocks_to_take:
+ output.append(x)
+ i += 1
+ assert len(output) == len(blocks_to_take), f"only {len(output)} / {len(blocks_to_take)} blocks found"
+ return output
+
+ def get_intermediate_layers(
+ self,
+ x: torch.Tensor,
+ n: Union[int, Sequence] = 1, # Layers or n last layers to take
+ reshape: bool = False,
+ return_class_token: bool = False,
+ norm=True
+ ) -> Tuple[Union[torch.Tensor, Tuple[torch.Tensor]]]:
+ if self.chunked_blocks:
+ outputs = self._get_intermediate_layers_chunked(x, n)
+ else:
+ outputs = self._get_intermediate_layers_not_chunked(x, n)
+ if norm:
+ outputs = [self.norm(out) for out in outputs]
+ class_tokens = [out[:, 0] for out in outputs]
+ outputs = [out[:, 1 + self.num_register_tokens:] for out in outputs]
+ if reshape:
+ B, _, w, h = x.shape
+ outputs = [
+ out.reshape(B, w // self.patch_size, h // self.patch_size, -1).permute(0, 3, 1, 2).contiguous()
+ for out in outputs
+ ]
+ if return_class_token:
+ return tuple(zip(outputs, class_tokens))
+ return tuple(outputs)
+
+ def forward(self, *args, is_training=False, **kwargs):
+ ret = self.forward_features(*args, **kwargs)
+ if is_training:
+ return ret
+ else:
+ return self.head(ret["x_norm_clstoken"])
+
+
+def init_weights_vit_timm(module: nn.Module, name: str = ""):
+ """ViT weight initialization, original timm impl (for reproducibility)"""
+ if isinstance(module, nn.Linear):
+ trunc_normal_(module.weight, std=0.02)
+ if module.bias is not None:
+ nn.init.zeros_(module.bias)
+
+
+def vit_small(patch_size=16, num_register_tokens=0, **kwargs):
+ model = DinoVisionTransformer(
+ patch_size=patch_size,
+ embed_dim=384,
+ depth=12,
+ num_heads=6,
+ mlp_ratio=4,
+ block_fn=partial(Block, attn_class=MemEffAttention),
+ num_register_tokens=num_register_tokens,
+ **kwargs,
+ )
+ return model
+
+
+def vit_base(patch_size=16, num_register_tokens=0, **kwargs):
+ model = DinoVisionTransformer(
+ patch_size=patch_size,
+ embed_dim=768,
+ depth=12,
+ num_heads=12,
+ mlp_ratio=4,
+ block_fn=partial(Block, attn_class=MemEffAttention),
+ num_register_tokens=num_register_tokens,
+ **kwargs,
+ )
+ return model
+
+
+def vit_large(patch_size=16, num_register_tokens=0, **kwargs):
+ model = DinoVisionTransformer(
+ patch_size=patch_size,
+ embed_dim=1024,
+ depth=24,
+ num_heads=16,
+ mlp_ratio=4,
+ block_fn=partial(Block, attn_class=MemEffAttention),
+ num_register_tokens=num_register_tokens,
+ **kwargs,
+ )
+ return model
+
+
+def vit_giant2(patch_size=16, num_register_tokens=0, **kwargs):
+ """
+ Close to ViT-giant, with embed-dim 1536 and 24 heads => embed-dim per head 64
+ """
+ model = DinoVisionTransformer(
+ patch_size=patch_size,
+ embed_dim=1536,
+ depth=40,
+ num_heads=24,
+ mlp_ratio=4,
+ block_fn=partial(Block, attn_class=MemEffAttention),
+ num_register_tokens=num_register_tokens,
+ **kwargs,
+ )
+ return model
+
+
+def DINOv2(model_name):
+ model_zoo = {
+ "vits": vit_small,
+ "vitb": vit_base,
+ "vitl": vit_large,
+ "vitg": vit_giant2
+ }
+
+ return model_zoo[model_name](
+ img_size=518,
+ patch_size=14,
+ init_values=1.0,
+ ffn_layer="mlp" if model_name != "vitg" else "swiglufused",
+ block_chunks=0,
+ num_register_tokens=0,
+ interpolate_antialias=False,
+ interpolate_offset=0.1
+ )
diff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/__init__.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/__init__.py
new file mode 100644
index 0000000..8120f4b
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/__init__.py
@@ -0,0 +1,11 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+from .mlp import Mlp
+from .patch_embed import PatchEmbed
+from .swiglu_ffn import SwiGLUFFN, SwiGLUFFNFused
+from .block import NestedTensorBlock
+from .attention import MemEffAttention
diff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/attention.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/attention.py
new file mode 100644
index 0000000..815a2bf
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/attention.py
@@ -0,0 +1,83 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# References:
+# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py
+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/models/vision_transformer.py
+
+import logging
+
+from torch import Tensor
+from torch import nn
+
+
+logger = logging.getLogger("dinov2")
+
+
+try:
+ from xformers.ops import memory_efficient_attention, unbind, fmha
+
+ XFORMERS_AVAILABLE = True
+except ImportError:
+ logger.warning("xFormers not available")
+ XFORMERS_AVAILABLE = False
+
+
+class Attention(nn.Module):
+ def __init__(
+ self,
+ dim: int,
+ num_heads: int = 8,
+ qkv_bias: bool = False,
+ proj_bias: bool = True,
+ attn_drop: float = 0.0,
+ proj_drop: float = 0.0,
+ ) -> None:
+ super().__init__()
+ self.num_heads = num_heads
+ head_dim = dim // num_heads
+ self.scale = head_dim**-0.5
+
+ self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
+ self.attn_drop = nn.Dropout(attn_drop)
+ self.proj = nn.Linear(dim, dim, bias=proj_bias)
+ self.proj_drop = nn.Dropout(proj_drop)
+
+ def forward(self, x: Tensor) -> Tensor:
+ B, N, C = x.shape
+ qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
+
+ q, k, v = qkv[0] * self.scale, qkv[1], qkv[2]
+ attn = q @ k.transpose(-2, -1)
+
+ attn = attn.softmax(dim=-1)
+ attn = self.attn_drop(attn)
+
+ x = (attn @ v).transpose(1, 2).reshape(B, N, C)
+ x = self.proj(x)
+ x = self.proj_drop(x)
+ return x
+
+
+class MemEffAttention(Attention):
+ def forward(self, x: Tensor, attn_bias=None) -> Tensor:
+ if not XFORMERS_AVAILABLE:
+ assert attn_bias is None, "xFormers is required for nested tensors usage"
+ return super().forward(x)
+
+ B, N, C = x.shape
+ qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads)
+
+ q, k, v = unbind(qkv, 2)
+
+ x = memory_efficient_attention(q, k, v, attn_bias=attn_bias)
+ x = x.reshape([B, N, C])
+
+ x = self.proj(x)
+ x = self.proj_drop(x)
+ return x
+
+
\ No newline at end of file
diff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/block.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/block.py
new file mode 100644
index 0000000..25488f5
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/block.py
@@ -0,0 +1,252 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# References:
+# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py
+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/patch_embed.py
+
+import logging
+from typing import Callable, List, Any, Tuple, Dict
+
+import torch
+from torch import nn, Tensor
+
+from .attention import Attention, MemEffAttention
+from .drop_path import DropPath
+from .layer_scale import LayerScale
+from .mlp import Mlp
+
+
+logger = logging.getLogger("dinov2")
+
+
+try:
+ from xformers.ops import fmha
+ from xformers.ops import scaled_index_add, index_select_cat
+
+ XFORMERS_AVAILABLE = True
+except ImportError:
+ logger.warning("xFormers not available")
+ XFORMERS_AVAILABLE = False
+
+
+class Block(nn.Module):
+ def __init__(
+ self,
+ dim: int,
+ num_heads: int,
+ mlp_ratio: float = 4.0,
+ qkv_bias: bool = False,
+ proj_bias: bool = True,
+ ffn_bias: bool = True,
+ drop: float = 0.0,
+ attn_drop: float = 0.0,
+ init_values=None,
+ drop_path: float = 0.0,
+ act_layer: Callable[..., nn.Module] = nn.GELU,
+ norm_layer: Callable[..., nn.Module] = nn.LayerNorm,
+ attn_class: Callable[..., nn.Module] = Attention,
+ ffn_layer: Callable[..., nn.Module] = Mlp,
+ ) -> None:
+ super().__init__()
+ # print(f"biases: qkv: {qkv_bias}, proj: {proj_bias}, ffn: {ffn_bias}")
+ self.norm1 = norm_layer(dim)
+ self.attn = attn_class(
+ dim,
+ num_heads=num_heads,
+ qkv_bias=qkv_bias,
+ proj_bias=proj_bias,
+ attn_drop=attn_drop,
+ proj_drop=drop,
+ )
+ self.ls1 = LayerScale(dim, init_values=init_values) if init_values else nn.Identity()
+ self.drop_path1 = DropPath(drop_path) if drop_path > 0.0 else nn.Identity()
+
+ self.norm2 = norm_layer(dim)
+ mlp_hidden_dim = int(dim * mlp_ratio)
+ self.mlp = ffn_layer(
+ in_features=dim,
+ hidden_features=mlp_hidden_dim,
+ act_layer=act_layer,
+ drop=drop,
+ bias=ffn_bias,
+ )
+ self.ls2 = LayerScale(dim, init_values=init_values) if init_values else nn.Identity()
+ self.drop_path2 = DropPath(drop_path) if drop_path > 0.0 else nn.Identity()
+
+ self.sample_drop_ratio = drop_path
+
+ def forward(self, x: Tensor) -> Tensor:
+ def attn_residual_func(x: Tensor) -> Tensor:
+ return self.ls1(self.attn(self.norm1(x)))
+
+ def ffn_residual_func(x: Tensor) -> Tensor:
+ return self.ls2(self.mlp(self.norm2(x)))
+
+ if self.training and self.sample_drop_ratio > 0.1:
+ # the overhead is compensated only for a drop path rate larger than 0.1
+ x = drop_add_residual_stochastic_depth(
+ x,
+ residual_func=attn_residual_func,
+ sample_drop_ratio=self.sample_drop_ratio,
+ )
+ x = drop_add_residual_stochastic_depth(
+ x,
+ residual_func=ffn_residual_func,
+ sample_drop_ratio=self.sample_drop_ratio,
+ )
+ elif self.training and self.sample_drop_ratio > 0.0:
+ x = x + self.drop_path1(attn_residual_func(x))
+ x = x + self.drop_path1(ffn_residual_func(x)) # FIXME: drop_path2
+ else:
+ x = x + attn_residual_func(x)
+ x = x + ffn_residual_func(x)
+ return x
+
+
+def drop_add_residual_stochastic_depth(
+ x: Tensor,
+ residual_func: Callable[[Tensor], Tensor],
+ sample_drop_ratio: float = 0.0,
+) -> Tensor:
+ # 1) extract subset using permutation
+ b, n, d = x.shape
+ sample_subset_size = max(int(b * (1 - sample_drop_ratio)), 1)
+ brange = (torch.randperm(b, device=x.device))[:sample_subset_size]
+ x_subset = x[brange]
+
+ # 2) apply residual_func to get residual
+ residual = residual_func(x_subset)
+
+ x_flat = x.flatten(1)
+ residual = residual.flatten(1)
+
+ residual_scale_factor = b / sample_subset_size
+
+ # 3) add the residual
+ x_plus_residual = torch.index_add(x_flat, 0, brange, residual.to(dtype=x.dtype), alpha=residual_scale_factor)
+ return x_plus_residual.view_as(x)
+
+
+def get_branges_scales(x, sample_drop_ratio=0.0):
+ b, n, d = x.shape
+ sample_subset_size = max(int(b * (1 - sample_drop_ratio)), 1)
+ brange = (torch.randperm(b, device=x.device))[:sample_subset_size]
+ residual_scale_factor = b / sample_subset_size
+ return brange, residual_scale_factor
+
+
+def add_residual(x, brange, residual, residual_scale_factor, scaling_vector=None):
+ if scaling_vector is None:
+ x_flat = x.flatten(1)
+ residual = residual.flatten(1)
+ x_plus_residual = torch.index_add(x_flat, 0, brange, residual.to(dtype=x.dtype), alpha=residual_scale_factor)
+ else:
+ x_plus_residual = scaled_index_add(
+ x, brange, residual.to(dtype=x.dtype), scaling=scaling_vector, alpha=residual_scale_factor
+ )
+ return x_plus_residual
+
+
+attn_bias_cache: Dict[Tuple, Any] = {}
+
+
+def get_attn_bias_and_cat(x_list, branges=None):
+ """
+ this will perform the index select, cat the tensors, and provide the attn_bias from cache
+ """
+ batch_sizes = [b.shape[0] for b in branges] if branges is not None else [x.shape[0] for x in x_list]
+ all_shapes = tuple((b, x.shape[1]) for b, x in zip(batch_sizes, x_list))
+ if all_shapes not in attn_bias_cache.keys():
+ seqlens = []
+ for b, x in zip(batch_sizes, x_list):
+ for _ in range(b):
+ seqlens.append(x.shape[1])
+ attn_bias = fmha.BlockDiagonalMask.from_seqlens(seqlens)
+ attn_bias._batch_sizes = batch_sizes
+ attn_bias_cache[all_shapes] = attn_bias
+
+ if branges is not None:
+ cat_tensors = index_select_cat([x.flatten(1) for x in x_list], branges).view(1, -1, x_list[0].shape[-1])
+ else:
+ tensors_bs1 = tuple(x.reshape([1, -1, *x.shape[2:]]) for x in x_list)
+ cat_tensors = torch.cat(tensors_bs1, dim=1)
+
+ return attn_bias_cache[all_shapes], cat_tensors
+
+
+def drop_add_residual_stochastic_depth_list(
+ x_list: List[Tensor],
+ residual_func: Callable[[Tensor, Any], Tensor],
+ sample_drop_ratio: float = 0.0,
+ scaling_vector=None,
+) -> Tensor:
+ # 1) generate random set of indices for dropping samples in the batch
+ branges_scales = [get_branges_scales(x, sample_drop_ratio=sample_drop_ratio) for x in x_list]
+ branges = [s[0] for s in branges_scales]
+ residual_scale_factors = [s[1] for s in branges_scales]
+
+ # 2) get attention bias and index+concat the tensors
+ attn_bias, x_cat = get_attn_bias_and_cat(x_list, branges)
+
+ # 3) apply residual_func to get residual, and split the result
+ residual_list = attn_bias.split(residual_func(x_cat, attn_bias=attn_bias)) # type: ignore
+
+ outputs = []
+ for x, brange, residual, residual_scale_factor in zip(x_list, branges, residual_list, residual_scale_factors):
+ outputs.append(add_residual(x, brange, residual, residual_scale_factor, scaling_vector).view_as(x))
+ return outputs
+
+
+class NestedTensorBlock(Block):
+ def forward_nested(self, x_list: List[Tensor]) -> List[Tensor]:
+ """
+ x_list contains a list of tensors to nest together and run
+ """
+ assert isinstance(self.attn, MemEffAttention)
+
+ if self.training and self.sample_drop_ratio > 0.0:
+
+ def attn_residual_func(x: Tensor, attn_bias=None) -> Tensor:
+ return self.attn(self.norm1(x), attn_bias=attn_bias)
+
+ def ffn_residual_func(x: Tensor, attn_bias=None) -> Tensor:
+ return self.mlp(self.norm2(x))
+
+ x_list = drop_add_residual_stochastic_depth_list(
+ x_list,
+ residual_func=attn_residual_func,
+ sample_drop_ratio=self.sample_drop_ratio,
+ scaling_vector=self.ls1.gamma if isinstance(self.ls1, LayerScale) else None,
+ )
+ x_list = drop_add_residual_stochastic_depth_list(
+ x_list,
+ residual_func=ffn_residual_func,
+ sample_drop_ratio=self.sample_drop_ratio,
+ scaling_vector=self.ls2.gamma if isinstance(self.ls1, LayerScale) else None,
+ )
+ return x_list
+ else:
+
+ def attn_residual_func(x: Tensor, attn_bias=None) -> Tensor:
+ return self.ls1(self.attn(self.norm1(x), attn_bias=attn_bias))
+
+ def ffn_residual_func(x: Tensor, attn_bias=None) -> Tensor:
+ return self.ls2(self.mlp(self.norm2(x)))
+
+ attn_bias, x = get_attn_bias_and_cat(x_list)
+ x = x + attn_residual_func(x, attn_bias=attn_bias)
+ x = x + ffn_residual_func(x)
+ return attn_bias.split(x)
+
+ def forward(self, x_or_x_list):
+ if isinstance(x_or_x_list, Tensor):
+ return super().forward(x_or_x_list)
+ elif isinstance(x_or_x_list, list):
+ assert XFORMERS_AVAILABLE, "Please install xFormers for nested tensors usage"
+ return self.forward_nested(x_or_x_list)
+ else:
+ raise AssertionError
diff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/drop_path.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/drop_path.py
new file mode 100644
index 0000000..af05625
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/drop_path.py
@@ -0,0 +1,35 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# References:
+# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py
+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/drop.py
+
+
+from torch import nn
+
+
+def drop_path(x, drop_prob: float = 0.0, training: bool = False):
+ if drop_prob == 0.0 or not training:
+ return x
+ keep_prob = 1 - drop_prob
+ shape = (x.shape[0],) + (1,) * (x.ndim - 1) # work with diff dim tensors, not just 2D ConvNets
+ random_tensor = x.new_empty(shape).bernoulli_(keep_prob)
+ if keep_prob > 0.0:
+ random_tensor.div_(keep_prob)
+ output = x * random_tensor
+ return output
+
+
+class DropPath(nn.Module):
+ """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks)."""
+
+ def __init__(self, drop_prob=None):
+ super(DropPath, self).__init__()
+ self.drop_prob = drop_prob
+
+ def forward(self, x):
+ return drop_path(x, self.drop_prob, self.training)
diff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/layer_scale.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/layer_scale.py
new file mode 100644
index 0000000..ca5daa5
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/layer_scale.py
@@ -0,0 +1,28 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Modified from: https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/vision_transformer.py#L103-L110
+
+from typing import Union
+
+import torch
+from torch import Tensor
+from torch import nn
+
+
+class LayerScale(nn.Module):
+ def __init__(
+ self,
+ dim: int,
+ init_values: Union[float, Tensor] = 1e-5,
+ inplace: bool = False,
+ ) -> None:
+ super().__init__()
+ self.inplace = inplace
+ self.gamma = nn.Parameter(init_values * torch.ones(dim))
+
+ def forward(self, x: Tensor) -> Tensor:
+ return x.mul_(self.gamma) if self.inplace else x * self.gamma
diff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/mlp.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/mlp.py
new file mode 100644
index 0000000..5e4b315
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/mlp.py
@@ -0,0 +1,41 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# References:
+# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py
+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/mlp.py
+
+
+from typing import Callable, Optional
+
+from torch import Tensor, nn
+
+
+class Mlp(nn.Module):
+ def __init__(
+ self,
+ in_features: int,
+ hidden_features: Optional[int] = None,
+ out_features: Optional[int] = None,
+ act_layer: Callable[..., nn.Module] = nn.GELU,
+ drop: float = 0.0,
+ bias: bool = True,
+ ) -> None:
+ super().__init__()
+ out_features = out_features or in_features
+ hidden_features = hidden_features or in_features
+ self.fc1 = nn.Linear(in_features, hidden_features, bias=bias)
+ self.act = act_layer()
+ self.fc2 = nn.Linear(hidden_features, out_features, bias=bias)
+ self.drop = nn.Dropout(drop)
+
+ def forward(self, x: Tensor) -> Tensor:
+ x = self.fc1(x)
+ x = self.act(x)
+ x = self.drop(x)
+ x = self.fc2(x)
+ x = self.drop(x)
+ return x
diff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/patch_embed.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/patch_embed.py
new file mode 100644
index 0000000..574abe4
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/patch_embed.py
@@ -0,0 +1,89 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+# References:
+# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py
+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/patch_embed.py
+
+from typing import Callable, Optional, Tuple, Union
+
+from torch import Tensor
+import torch.nn as nn
+
+
+def make_2tuple(x):
+ if isinstance(x, tuple):
+ assert len(x) == 2
+ return x
+
+ assert isinstance(x, int)
+ return (x, x)
+
+
+class PatchEmbed(nn.Module):
+ """
+ 2D image to patch embedding: (B,C,H,W) -> (B,N,D)
+
+ Args:
+ img_size: Image size.
+ patch_size: Patch token size.
+ in_chans: Number of input image channels.
+ embed_dim: Number of linear projection output channels.
+ norm_layer: Normalization layer.
+ """
+
+ def __init__(
+ self,
+ img_size: Union[int, Tuple[int, int]] = 224,
+ patch_size: Union[int, Tuple[int, int]] = 16,
+ in_chans: int = 3,
+ embed_dim: int = 768,
+ norm_layer: Optional[Callable] = None,
+ flatten_embedding: bool = True,
+ ) -> None:
+ super().__init__()
+
+ image_HW = make_2tuple(img_size)
+ patch_HW = make_2tuple(patch_size)
+ patch_grid_size = (
+ image_HW[0] // patch_HW[0],
+ image_HW[1] // patch_HW[1],
+ )
+
+ self.img_size = image_HW
+ self.patch_size = patch_HW
+ self.patches_resolution = patch_grid_size
+ self.num_patches = patch_grid_size[0] * patch_grid_size[1]
+
+ self.in_chans = in_chans
+ self.embed_dim = embed_dim
+
+ self.flatten_embedding = flatten_embedding
+
+ self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_HW, stride=patch_HW)
+ self.norm = norm_layer(embed_dim) if norm_layer else nn.Identity()
+
+ def forward(self, x: Tensor) -> Tensor:
+ _, _, H, W = x.shape
+ patch_H, patch_W = self.patch_size
+
+ assert H % patch_H == 0, f"Input image height {H} is not a multiple of patch height {patch_H}"
+ assert W % patch_W == 0, f"Input image width {W} is not a multiple of patch width: {patch_W}"
+
+ x = self.proj(x) # B C H W
+ H, W = x.size(2), x.size(3)
+ x = x.flatten(2).transpose(1, 2) # B HW C
+ x = self.norm(x)
+ if not self.flatten_embedding:
+ x = x.reshape(-1, H, W, self.embed_dim) # B H W C
+ return x
+
+ def flops(self) -> float:
+ Ho, Wo = self.patches_resolution
+ flops = Ho * Wo * self.embed_dim * self.in_chans * (self.patch_size[0] * self.patch_size[1])
+ if self.norm is not None:
+ flops += Ho * Wo * self.embed_dim
+ return flops
diff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/swiglu_ffn.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/swiglu_ffn.py
new file mode 100644
index 0000000..b3324b2
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/swiglu_ffn.py
@@ -0,0 +1,63 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+from typing import Callable, Optional
+
+from torch import Tensor, nn
+import torch.nn.functional as F
+
+
+class SwiGLUFFN(nn.Module):
+ def __init__(
+ self,
+ in_features: int,
+ hidden_features: Optional[int] = None,
+ out_features: Optional[int] = None,
+ act_layer: Callable[..., nn.Module] = None,
+ drop: float = 0.0,
+ bias: bool = True,
+ ) -> None:
+ super().__init__()
+ out_features = out_features or in_features
+ hidden_features = hidden_features or in_features
+ self.w12 = nn.Linear(in_features, 2 * hidden_features, bias=bias)
+ self.w3 = nn.Linear(hidden_features, out_features, bias=bias)
+
+ def forward(self, x: Tensor) -> Tensor:
+ x12 = self.w12(x)
+ x1, x2 = x12.chunk(2, dim=-1)
+ hidden = F.silu(x1) * x2
+ return self.w3(hidden)
+
+
+try:
+ from xformers.ops import SwiGLU
+
+ XFORMERS_AVAILABLE = True
+except ImportError:
+ SwiGLU = SwiGLUFFN
+ XFORMERS_AVAILABLE = False
+
+
+class SwiGLUFFNFused(SwiGLU):
+ def __init__(
+ self,
+ in_features: int,
+ hidden_features: Optional[int] = None,
+ out_features: Optional[int] = None,
+ act_layer: Callable[..., nn.Module] = None,
+ drop: float = 0.0,
+ bias: bool = True,
+ ) -> None:
+ out_features = out_features or in_features
+ hidden_features = hidden_features or in_features
+ hidden_features = (int(hidden_features * 2 / 3) + 7) // 8 * 8
+ super().__init__(
+ in_features=in_features,
+ hidden_features=hidden_features,
+ out_features=out_features,
+ bias=bias,
+ )
diff --git a/video2blockout/third_party/video_depth_anything/dpt.py b/video2blockout/third_party/video_depth_anything/dpt.py
new file mode 100644
index 0000000..8c43a14
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dpt.py
@@ -0,0 +1,160 @@
+# Copyright (2025) Bytedance Ltd. and/or its affiliates
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from .util.blocks import FeatureFusionBlock, _make_scratch
+
+
+def _make_fusion_block(features, use_bn, size=None):
+ return FeatureFusionBlock(
+ features,
+ nn.ReLU(False),
+ deconv=False,
+ bn=use_bn,
+ expand=False,
+ align_corners=True,
+ size=size,
+ )
+
+
+class ConvBlock(nn.Module):
+ def __init__(self, in_feature, out_feature):
+ super().__init__()
+
+ self.conv_block = nn.Sequential(
+ nn.Conv2d(in_feature, out_feature, kernel_size=3, stride=1, padding=1),
+ nn.BatchNorm2d(out_feature),
+ nn.ReLU(True)
+ )
+
+ def forward(self, x):
+ return self.conv_block(x)
+
+
+class DPTHead(nn.Module):
+ def __init__(
+ self,
+ in_channels,
+ features=256,
+ use_bn=False,
+ out_channels=[256, 512, 1024, 1024],
+ use_clstoken=False
+ ):
+ super(DPTHead, self).__init__()
+
+ self.use_clstoken = use_clstoken
+
+ self.projects = nn.ModuleList([
+ nn.Conv2d(
+ in_channels=in_channels,
+ out_channels=out_channel,
+ kernel_size=1,
+ stride=1,
+ padding=0,
+ ) for out_channel in out_channels
+ ])
+
+ self.resize_layers = nn.ModuleList([
+ nn.ConvTranspose2d(
+ in_channels=out_channels[0],
+ out_channels=out_channels[0],
+ kernel_size=4,
+ stride=4,
+ padding=0),
+ nn.ConvTranspose2d(
+ in_channels=out_channels[1],
+ out_channels=out_channels[1],
+ kernel_size=2,
+ stride=2,
+ padding=0),
+ nn.Identity(),
+ nn.Conv2d(
+ in_channels=out_channels[3],
+ out_channels=out_channels[3],
+ kernel_size=3,
+ stride=2,
+ padding=1)
+ ])
+
+ if use_clstoken:
+ self.readout_projects = nn.ModuleList()
+ for _ in range(len(self.projects)):
+ self.readout_projects.append(
+ nn.Sequential(
+ nn.Linear(2 * in_channels, in_channels),
+ nn.GELU()))
+
+ self.scratch = _make_scratch(
+ out_channels,
+ features,
+ groups=1,
+ expand=False,
+ )
+
+ self.scratch.stem_transpose = None
+
+ self.scratch.refinenet1 = _make_fusion_block(features, use_bn)
+ self.scratch.refinenet2 = _make_fusion_block(features, use_bn)
+ self.scratch.refinenet3 = _make_fusion_block(features, use_bn)
+ self.scratch.refinenet4 = _make_fusion_block(features, use_bn)
+
+ head_features_1 = features
+ head_features_2 = 32
+
+ self.scratch.output_conv1 = nn.Conv2d(head_features_1, head_features_1 // 2, kernel_size=3, stride=1, padding=1)
+ self.scratch.output_conv2 = nn.Sequential(
+ nn.Conv2d(head_features_1 // 2, head_features_2, kernel_size=3, stride=1, padding=1),
+ nn.ReLU(True),
+ nn.Conv2d(head_features_2, 1, kernel_size=1, stride=1, padding=0),
+ nn.ReLU(True),
+ nn.Identity(),
+ )
+
+ def forward(self, out_features, patch_h, patch_w):
+ out = []
+ for i, x in enumerate(out_features):
+ if self.use_clstoken:
+ x, cls_token = x[0], x[1]
+ readout = cls_token.unsqueeze(1).expand_as(x)
+ x = self.readout_projects[i](torch.cat((x, readout), -1))
+ else:
+ x = x[0]
+
+ x = x.permute(0, 2, 1).reshape((x.shape[0], x.shape[-1], patch_h, patch_w))
+
+ x = self.projects[i](x)
+ x = self.resize_layers[i](x)
+
+ out.append(x)
+
+ layer_1, layer_2, layer_3, layer_4 = out
+
+ layer_1_rn = self.scratch.layer1_rn(layer_1)
+ layer_2_rn = self.scratch.layer2_rn(layer_2)
+ layer_3_rn = self.scratch.layer3_rn(layer_3)
+ layer_4_rn = self.scratch.layer4_rn(layer_4)
+
+ path_4 = self.scratch.refinenet4(layer_4_rn, size=layer_3_rn.shape[2:])
+ path_3 = self.scratch.refinenet3(path_4, layer_3_rn, size=layer_2_rn.shape[2:])
+ path_2 = self.scratch.refinenet2(path_3, layer_2_rn, size=layer_1_rn.shape[2:])
+ path_1 = self.scratch.refinenet1(path_2, layer_1_rn)
+
+ out = self.scratch.output_conv1(path_1)
+ out = F.interpolate(out, (int(patch_h * 14), int(patch_w * 14)), mode="bilinear", align_corners=True)
+ out = self.scratch.output_conv2(out)
+
+ return out
+
\ No newline at end of file
diff --git a/video2blockout/third_party/video_depth_anything/dpt_temporal.py b/video2blockout/third_party/video_depth_anything/dpt_temporal.py
new file mode 100644
index 0000000..d629a11
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/dpt_temporal.py
@@ -0,0 +1,125 @@
+# Copyright (2025) Bytedance Ltd. and/or its affiliates
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+import torch.nn.functional as F
+import torch.nn as nn
+from .dpt import DPTHead
+from .motion_module.motion_module import TemporalModule
+from easydict import EasyDict
+
+
+class DPTHeadTemporal(DPTHead):
+ def __init__(self,
+ in_channels,
+ features=256,
+ use_bn=False,
+ out_channels=[256, 512, 1024, 1024],
+ use_clstoken=False,
+ num_frames=32,
+ pe='ape'
+ ):
+ super().__init__(in_channels, features, use_bn, out_channels, use_clstoken)
+
+ assert num_frames > 0
+ motion_module_kwargs = EasyDict(num_attention_heads = 8,
+ num_transformer_block = 1,
+ num_attention_blocks = 2,
+ temporal_max_len = num_frames,
+ zero_initialize = True,
+ pos_embedding_type = pe)
+
+ self.motion_modules = nn.ModuleList([
+ TemporalModule(in_channels=out_channels[2],
+ **motion_module_kwargs),
+ TemporalModule(in_channels=out_channels[3],
+ **motion_module_kwargs),
+ TemporalModule(in_channels=features,
+ **motion_module_kwargs),
+ TemporalModule(in_channels=features,
+ **motion_module_kwargs)
+ ])
+
+ def forward(self, out_features, patch_h, patch_w, frame_length, micro_batch_size=4, cached_hidden_state_list=None):
+ out = []
+ for i, x in enumerate(out_features):
+ if self.use_clstoken:
+ x, cls_token = x[0], x[1]
+ readout = cls_token.unsqueeze(1).expand_as(x)
+ x = self.readout_projects[i](torch.cat((x, readout), -1))
+ else:
+ x = x[0]
+
+ x = x.permute(0, 2, 1).reshape((x.shape[0], x.shape[-1], patch_h, patch_w)).contiguous()
+
+ B, T = x.shape[0] // frame_length, frame_length
+ x = self.projects[i](x)
+ x = self.resize_layers[i](x)
+
+ out.append(x)
+
+ layer_1, layer_2, layer_3, layer_4 = out
+
+ B, T = layer_1.shape[0] // frame_length, frame_length
+ if cached_hidden_state_list is not None:
+ N = len(cached_hidden_state_list) // len(self.motion_modules)
+ else:
+ N = 0
+
+ layer_3, h0 = self.motion_modules[0](layer_3.unflatten(0, (B, T)).permute(0, 2, 1, 3, 4), None, None, cached_hidden_state_list[0:N] if N else None)
+ layer_3 = layer_3.permute(0, 2, 1, 3, 4).flatten(0, 1)
+ layer_4, h1 = self.motion_modules[1](layer_4.unflatten(0, (B, T)).permute(0, 2, 1, 3, 4), None, None, cached_hidden_state_list[N:2*N] if N else None)
+ layer_4 = layer_4.permute(0, 2, 1, 3, 4).flatten(0, 1)
+
+ layer_1_rn = self.scratch.layer1_rn(layer_1)
+ layer_2_rn = self.scratch.layer2_rn(layer_2)
+ layer_3_rn = self.scratch.layer3_rn(layer_3)
+ layer_4_rn = self.scratch.layer4_rn(layer_4)
+
+ path_4 = self.scratch.refinenet4(layer_4_rn, size=layer_3_rn.shape[2:])
+ path_4, h2 = self.motion_modules[2](path_4.unflatten(0, (B, T)).permute(0, 2, 1, 3, 4), None, None, cached_hidden_state_list[2*N:3*N] if N else None)
+ path_4 = path_4.permute(0, 2, 1, 3, 4).flatten(0, 1)
+ path_3 = self.scratch.refinenet3(path_4, layer_3_rn, size=layer_2_rn.shape[2:])
+ path_3, h3 = self.motion_modules[3](path_3.unflatten(0, (B, T)).permute(0, 2, 1, 3, 4), None, None, cached_hidden_state_list[3*N:] if N else None)
+ path_3 = path_3.permute(0, 2, 1, 3, 4).flatten(0, 1)
+
+ batch_size = layer_1_rn.shape[0]
+ if batch_size <= micro_batch_size or batch_size % micro_batch_size != 0:
+ path_2 = self.scratch.refinenet2(path_3, layer_2_rn, size=layer_1_rn.shape[2:])
+ path_1 = self.scratch.refinenet1(path_2, layer_1_rn)
+
+ out = self.scratch.output_conv1(path_1)
+ out = F.interpolate(
+ out, (int(patch_h * 14), int(patch_w * 14)), mode="bilinear", align_corners=True
+ )
+ ori_type = out.dtype
+ with torch.autocast(device_type="cuda", enabled=False):
+ out = self.scratch.output_conv2(out.float())
+
+ output = out.to(ori_type)
+ else:
+ ret = []
+ for i in range(0, batch_size, micro_batch_size):
+ path_2 = self.scratch.refinenet2(path_3[i:i + micro_batch_size], layer_2_rn[i:i + micro_batch_size], size=layer_1_rn[i:i + micro_batch_size].shape[2:])
+ path_1 = self.scratch.refinenet1(path_2, layer_1_rn[i:i + micro_batch_size])
+ out = self.scratch.output_conv1(path_1)
+ out = F.interpolate(
+ out, (int(patch_h * 14), int(patch_w * 14)), mode="bilinear", align_corners=True
+ )
+ ori_type = out.dtype
+ with torch.autocast(device_type="cuda", enabled=False):
+ out = self.scratch.output_conv2(out.float())
+ ret.append(out.to(ori_type))
+ output = torch.cat(ret, dim=0)
+
+ return output, h0 + h1 + h2 + h3
diff --git a/video2blockout/third_party/video_depth_anything/motion_module/__init__.py b/video2blockout/third_party/video_depth_anything/motion_module/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/video2blockout/third_party/video_depth_anything/motion_module/attention.py b/video2blockout/third_party/video_depth_anything/motion_module/attention.py
new file mode 100644
index 0000000..41f551b
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/motion_module/attention.py
@@ -0,0 +1,429 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import Optional, Tuple
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+try:
+ import xformers
+ import xformers.ops
+
+ XFORMERS_AVAILABLE = True
+except ImportError:
+ print("xFormers not available")
+ XFORMERS_AVAILABLE = False
+
+
+class CrossAttention(nn.Module):
+ r"""
+ A cross attention layer.
+
+ Parameters:
+ query_dim (`int`): The number of channels in the query.
+ cross_attention_dim (`int`, *optional*):
+ The number of channels in the encoder_hidden_states. If not given, defaults to `query_dim`.
+ heads (`int`, *optional*, defaults to 8): The number of heads to use for multi-head attention.
+ dim_head (`int`, *optional*, defaults to 64): The number of channels in each head.
+ dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.
+ bias (`bool`, *optional*, defaults to False):
+ Set to `True` for the query, key, and value linear layers to contain a bias parameter.
+ """
+
+ def __init__(
+ self,
+ query_dim: int,
+ cross_attention_dim: Optional[int] = None,
+ heads: int = 8,
+ dim_head: int = 64,
+ dropout: float = 0.0,
+ bias=False,
+ upcast_attention: bool = False,
+ upcast_softmax: bool = False,
+ added_kv_proj_dim: Optional[int] = None,
+ norm_num_groups: Optional[int] = None,
+ ):
+ super().__init__()
+ inner_dim = dim_head * heads
+ cross_attention_dim = cross_attention_dim if cross_attention_dim is not None else query_dim
+ self.upcast_attention = upcast_attention
+ self.upcast_softmax = upcast_softmax
+ self.upcast_efficient_attention = False
+
+ self.scale = dim_head**-0.5
+
+ self.heads = heads
+ # for slice_size > 0 the attention score computation
+ # is split across the batch axis to save memory
+ # You can set slice_size with `set_attention_slice`
+ self.sliceable_head_dim = heads
+ self._slice_size = None
+ self._use_memory_efficient_attention_xformers = False
+ self.added_kv_proj_dim = added_kv_proj_dim
+
+ if norm_num_groups is not None:
+ self.group_norm = nn.GroupNorm(num_channels=inner_dim, num_groups=norm_num_groups, eps=1e-5, affine=True)
+ else:
+ self.group_norm = None
+
+ self.to_q = nn.Linear(query_dim, inner_dim, bias=bias)
+ self.to_k = nn.Linear(cross_attention_dim, inner_dim, bias=bias)
+ self.to_v = nn.Linear(cross_attention_dim, inner_dim, bias=bias)
+
+ if self.added_kv_proj_dim is not None:
+ self.add_k_proj = nn.Linear(added_kv_proj_dim, cross_attention_dim)
+ self.add_v_proj = nn.Linear(added_kv_proj_dim, cross_attention_dim)
+
+ self.to_out = nn.ModuleList([])
+ self.to_out.append(nn.Linear(inner_dim, query_dim))
+ self.to_out.append(nn.Dropout(dropout))
+
+ def reshape_heads_to_batch_dim(self, tensor):
+ batch_size, seq_len, dim = tensor.shape
+ head_size = self.heads
+ tensor = tensor.reshape(batch_size, seq_len, head_size, dim // head_size).contiguous()
+ tensor = tensor.permute(0, 2, 1, 3).reshape(batch_size * head_size, seq_len, dim // head_size).contiguous()
+ return tensor
+
+ def reshape_heads_to_4d(self, tensor):
+ batch_size, seq_len, dim = tensor.shape
+ head_size = self.heads
+ tensor = tensor.reshape(batch_size, seq_len, head_size, dim // head_size).contiguous()
+ return tensor
+
+ def reshape_batch_dim_to_heads(self, tensor):
+ batch_size, seq_len, dim = tensor.shape
+ head_size = self.heads
+ tensor = tensor.reshape(batch_size // head_size, head_size, seq_len, dim).contiguous()
+ tensor = tensor.permute(0, 2, 1, 3).reshape(batch_size // head_size, seq_len, dim * head_size).contiguous()
+ return tensor
+
+ def reshape_4d_to_heads(self, tensor):
+ batch_size, seq_len, head_size, dim = tensor.shape
+ head_size = self.heads
+ tensor = tensor.reshape(batch_size, seq_len, dim * head_size).contiguous()
+ return tensor
+
+ def set_attention_slice(self, slice_size):
+ if slice_size is not None and slice_size > self.sliceable_head_dim:
+ raise ValueError(f"slice_size {slice_size} has to be smaller or equal to {self.sliceable_head_dim}.")
+
+ self._slice_size = slice_size
+
+ def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None):
+ batch_size, sequence_length, _ = hidden_states.shape
+
+ encoder_hidden_states = encoder_hidden_states
+
+ if self.group_norm is not None:
+ hidden_states = self.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)
+
+ query = self.to_q(hidden_states)
+ dim = query.shape[-1]
+ query = self.reshape_heads_to_batch_dim(query)
+
+ if self.added_kv_proj_dim is not None:
+ key = self.to_k(hidden_states)
+ value = self.to_v(hidden_states)
+ encoder_hidden_states_key_proj = self.add_k_proj(encoder_hidden_states)
+ encoder_hidden_states_value_proj = self.add_v_proj(encoder_hidden_states)
+
+ key = self.reshape_heads_to_batch_dim(key)
+ value = self.reshape_heads_to_batch_dim(value)
+ encoder_hidden_states_key_proj = self.reshape_heads_to_batch_dim(encoder_hidden_states_key_proj)
+ encoder_hidden_states_value_proj = self.reshape_heads_to_batch_dim(encoder_hidden_states_value_proj)
+
+ key = torch.concat([encoder_hidden_states_key_proj, key], dim=1)
+ value = torch.concat([encoder_hidden_states_value_proj, value], dim=1)
+ else:
+ encoder_hidden_states = encoder_hidden_states if encoder_hidden_states is not None else hidden_states
+ key = self.to_k(encoder_hidden_states)
+ value = self.to_v(encoder_hidden_states)
+
+ key = self.reshape_heads_to_batch_dim(key)
+ value = self.reshape_heads_to_batch_dim(value)
+
+ if attention_mask is not None:
+ if attention_mask.shape[-1] != query.shape[1]:
+ target_length = query.shape[1]
+ attention_mask = F.pad(attention_mask, (0, target_length), value=0.0)
+ attention_mask = attention_mask.repeat_interleave(self.heads, dim=0)
+
+ # attention, what we cannot get enough of
+ if XFORMERS_AVAILABLE and self._use_memory_efficient_attention_xformers:
+ hidden_states = self._memory_efficient_attention_xformers(query, key, value, attention_mask)
+ # Some versions of xformers return output in fp32, cast it back to the dtype of the input
+ hidden_states = hidden_states.to(query.dtype)
+ else:
+ if self._slice_size is None or query.shape[0] // self._slice_size == 1:
+ hidden_states = self._attention(query, key, value, attention_mask)
+ else:
+ hidden_states = self._sliced_attention(query, key, value, sequence_length, dim, attention_mask)
+
+ # linear proj
+ hidden_states = self.to_out[0](hidden_states)
+
+ # dropout
+ hidden_states = self.to_out[1](hidden_states)
+ return hidden_states
+
+ def _attention(self, query, key, value, attention_mask=None):
+ if self.upcast_attention:
+ query = query.float()
+ key = key.float()
+
+ attention_scores = torch.baddbmm(
+ torch.empty(query.shape[0], query.shape[1], key.shape[1], dtype=query.dtype, device=query.device),
+ query,
+ key.transpose(-1, -2),
+ beta=0,
+ alpha=self.scale,
+ )
+
+ if attention_mask is not None:
+ attention_scores = attention_scores + attention_mask
+
+ if self.upcast_softmax:
+ attention_scores = attention_scores.float()
+
+ attention_probs = attention_scores.softmax(dim=-1)
+
+ # cast back to the original dtype
+ attention_probs = attention_probs.to(value.dtype)
+
+ # compute attention output
+ hidden_states = torch.bmm(attention_probs, value)
+
+ # reshape hidden_states
+ hidden_states = self.reshape_batch_dim_to_heads(hidden_states)
+ return hidden_states
+
+ def _sliced_attention(self, query, key, value, sequence_length, dim, attention_mask):
+ batch_size_attention = query.shape[0]
+ hidden_states = torch.zeros(
+ (batch_size_attention, sequence_length, dim // self.heads), device=query.device, dtype=query.dtype
+ )
+ slice_size = self._slice_size if self._slice_size is not None else hidden_states.shape[0]
+ for i in range(hidden_states.shape[0] // slice_size):
+ start_idx = i * slice_size
+ end_idx = (i + 1) * slice_size
+
+ query_slice = query[start_idx:end_idx]
+ key_slice = key[start_idx:end_idx]
+
+ if self.upcast_attention:
+ query_slice = query_slice.float()
+ key_slice = key_slice.float()
+
+ attn_slice = torch.baddbmm(
+ torch.empty(slice_size, query.shape[1], key.shape[1], dtype=query_slice.dtype, device=query.device),
+ query_slice,
+ key_slice.transpose(-1, -2),
+ beta=0,
+ alpha=self.scale,
+ )
+
+ if attention_mask is not None:
+ attn_slice = attn_slice + attention_mask[start_idx:end_idx]
+
+ if self.upcast_softmax:
+ attn_slice = attn_slice.float()
+
+ attn_slice = attn_slice.softmax(dim=-1)
+
+ # cast back to the original dtype
+ attn_slice = attn_slice.to(value.dtype)
+ attn_slice = torch.bmm(attn_slice, value[start_idx:end_idx])
+
+ hidden_states[start_idx:end_idx] = attn_slice
+
+ # reshape hidden_states
+ hidden_states = self.reshape_batch_dim_to_heads(hidden_states)
+ return hidden_states
+
+ def _memory_efficient_attention_xformers(self, query, key, value, attention_mask):
+ if self.upcast_efficient_attention:
+ org_dtype = query.dtype
+ query = query.float()
+ key = key.float()
+ value = value.float()
+ if attention_mask is not None:
+ attention_mask = attention_mask.float()
+ hidden_states = self._memory_efficient_attention_split(query, key, value, attention_mask)
+
+ if self.upcast_efficient_attention:
+ hidden_states = hidden_states.to(org_dtype)
+
+ hidden_states = self.reshape_4d_to_heads(hidden_states)
+ return hidden_states
+
+ # print("Errror: no xformers")
+ # raise NotImplementedError
+
+ def _memory_efficient_attention_split(self, query, key, value, attention_mask):
+ batch_size = query.shape[0]
+ max_batch_size = 65535
+ num_batches = (batch_size + max_batch_size - 1) // max_batch_size
+ results = []
+ for i in range(num_batches):
+ start_idx = i * max_batch_size
+ end_idx = min((i + 1) * max_batch_size, batch_size)
+ query_batch = query[start_idx:end_idx]
+ key_batch = key[start_idx:end_idx]
+ value_batch = value[start_idx:end_idx]
+ if attention_mask is not None:
+ attention_mask_batch = attention_mask[start_idx:end_idx]
+ else:
+ attention_mask_batch = None
+ result = xformers.ops.memory_efficient_attention(query_batch, key_batch, value_batch, attn_bias=attention_mask_batch)
+ results.append(result)
+ full_result = torch.cat(results, dim=0)
+ return full_result
+
+
+class FeedForward(nn.Module):
+ r"""
+ A feed-forward layer.
+
+ Parameters:
+ dim (`int`): The number of channels in the input.
+ dim_out (`int`, *optional*): The number of channels in the output. If not given, defaults to `dim`.
+ mult (`int`, *optional*, defaults to 4): The multiplier to use for the hidden dimension.
+ dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.
+ activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.
+ """
+
+ def __init__(
+ self,
+ dim: int,
+ dim_out: Optional[int] = None,
+ mult: int = 4,
+ dropout: float = 0.0,
+ activation_fn: str = "geglu",
+ ):
+ super().__init__()
+ inner_dim = int(dim * mult)
+ dim_out = dim_out if dim_out is not None else dim
+
+ if activation_fn == "gelu":
+ act_fn = GELU(dim, inner_dim)
+ elif activation_fn == "geglu":
+ act_fn = GEGLU(dim, inner_dim)
+ elif activation_fn == "geglu-approximate":
+ act_fn = ApproximateGELU(dim, inner_dim)
+
+ self.net = nn.ModuleList([])
+ # project in
+ self.net.append(act_fn)
+ # project dropout
+ self.net.append(nn.Dropout(dropout))
+ # project out
+ self.net.append(nn.Linear(inner_dim, dim_out))
+
+ def forward(self, hidden_states):
+ for module in self.net:
+ hidden_states = module(hidden_states)
+ return hidden_states
+
+
+class GELU(nn.Module):
+ r"""
+ GELU activation function
+ """
+
+ def __init__(self, dim_in: int, dim_out: int):
+ super().__init__()
+ self.proj = nn.Linear(dim_in, dim_out)
+
+ def gelu(self, gate):
+ if gate.device.type != "mps":
+ return F.gelu(gate)
+ # mps: gelu is not implemented for float16
+ return F.gelu(gate.to(dtype=torch.float32)).to(dtype=gate.dtype)
+
+ def forward(self, hidden_states):
+ hidden_states = self.proj(hidden_states)
+ hidden_states = self.gelu(hidden_states)
+ return hidden_states
+
+
+# feedforward
+class GEGLU(nn.Module):
+ r"""
+ A variant of the gated linear unit activation function from https://arxiv.org/abs/2002.05202.
+
+ Parameters:
+ dim_in (`int`): The number of channels in the input.
+ dim_out (`int`): The number of channels in the output.
+ """
+
+ def __init__(self, dim_in: int, dim_out: int):
+ super().__init__()
+ self.proj = nn.Linear(dim_in, dim_out * 2)
+
+ def gelu(self, gate):
+ if gate.device.type != "mps":
+ return F.gelu(gate)
+ # mps: gelu is not implemented for float16
+ return F.gelu(gate.to(dtype=torch.float32)).to(dtype=gate.dtype)
+
+ def forward(self, hidden_states):
+ hidden_states, gate = self.proj(hidden_states).chunk(2, dim=-1)
+ return hidden_states * self.gelu(gate)
+
+
+class ApproximateGELU(nn.Module):
+ """
+ The approximate form of Gaussian Error Linear Unit (GELU)
+
+ For more details, see section 2: https://arxiv.org/abs/1606.08415
+ """
+
+ def __init__(self, dim_in: int, dim_out: int):
+ super().__init__()
+ self.proj = nn.Linear(dim_in, dim_out)
+
+ def forward(self, x):
+ x = self.proj(x)
+ return x * torch.sigmoid(1.702 * x)
+
+
+def precompute_freqs_cis(dim: int, end: int, theta: float = 10000.0):
+ freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))
+ t = torch.arange(end, device=freqs.device, dtype=torch.float32)
+ freqs = torch.outer(t, freqs)
+ freqs_cis = torch.polar(torch.ones_like(freqs), freqs) # complex64
+ return freqs_cis
+
+
+def reshape_for_broadcast(freqs_cis: torch.Tensor, x: torch.Tensor):
+ ndim = x.ndim
+ assert 0 <= 1 < ndim
+ assert freqs_cis.shape == (x.shape[1], x.shape[-1])
+ shape = [d if i == 1 or i == ndim - 1 else 1 for i, d in enumerate(x.shape)]
+ return freqs_cis.view(*shape)
+
+
+def apply_rotary_emb(
+ xq: torch.Tensor,
+ xk: torch.Tensor,
+ freqs_cis: torch.Tensor,
+) -> Tuple[torch.Tensor, torch.Tensor]:
+ xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2).contiguous())
+ xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2).contiguous())
+ freqs_cis = reshape_for_broadcast(freqs_cis, xq_)
+ xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(2)
+ xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(2)
+ return xq_out.type_as(xq), xk_out.type_as(xk)
diff --git a/video2blockout/third_party/video_depth_anything/motion_module/motion_module.py b/video2blockout/third_party/video_depth_anything/motion_module/motion_module.py
new file mode 100644
index 0000000..d4d81a3
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/motion_module/motion_module.py
@@ -0,0 +1,321 @@
+# This file is originally from AnimateDiff/animatediff/models/motion_module.py at main · guoyww/AnimateDiff
+# SPDX-License-Identifier: Apache-2.0 license
+#
+# This file may have been modified by ByteDance Ltd. and/or its affiliates on [date of modification]
+# Original file was released under [ Apache-2.0 license], with the full license text available at [https://github.com/guoyww/AnimateDiff?tab=Apache-2.0-1-ov-file#readme].
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from .attention import CrossAttention, FeedForward, apply_rotary_emb, precompute_freqs_cis
+
+from einops import rearrange, repeat
+import math
+
+try:
+ import xformers
+ import xformers.ops
+
+ XFORMERS_AVAILABLE = True
+except ImportError:
+ print("xFormers not available")
+ XFORMERS_AVAILABLE = False
+
+
+def zero_module(module):
+ # Zero out the parameters of a module and return it.
+ for p in module.parameters():
+ p.detach().zero_()
+ return module
+
+
+class TemporalModule(nn.Module):
+ def __init__(
+ self,
+ in_channels,
+ num_attention_heads = 8,
+ num_transformer_block = 2,
+ num_attention_blocks = 2,
+ norm_num_groups = 32,
+ temporal_max_len = 32,
+ zero_initialize = True,
+ pos_embedding_type = "ape",
+ ):
+ super().__init__()
+
+ self.temporal_transformer = TemporalTransformer3DModel(
+ in_channels=in_channels,
+ num_attention_heads=num_attention_heads,
+ attention_head_dim=in_channels // num_attention_heads,
+ num_layers=num_transformer_block,
+ num_attention_blocks=num_attention_blocks,
+ norm_num_groups=norm_num_groups,
+ temporal_max_len=temporal_max_len,
+ pos_embedding_type=pos_embedding_type,
+ )
+
+ if zero_initialize:
+ self.temporal_transformer.proj_out = zero_module(self.temporal_transformer.proj_out)
+
+ def forward(self, input_tensor, encoder_hidden_states, attention_mask=None, cached_hidden_state_list=None):
+ hidden_states = input_tensor
+ hidden_states, output_hidden_state_list = self.temporal_transformer(hidden_states, encoder_hidden_states, attention_mask, cached_hidden_state_list)
+
+ output = hidden_states
+ return output, output_hidden_state_list # list of hidden states
+
+
+class TemporalTransformer3DModel(nn.Module):
+ def __init__(
+ self,
+ in_channels,
+ num_attention_heads,
+ attention_head_dim,
+ num_layers,
+ num_attention_blocks = 2,
+ norm_num_groups = 32,
+ temporal_max_len = 32,
+ pos_embedding_type = "ape",
+ ):
+ super().__init__()
+
+ inner_dim = num_attention_heads * attention_head_dim
+
+ self.norm = torch.nn.GroupNorm(num_groups=norm_num_groups, num_channels=in_channels, eps=1e-6, affine=True)
+ self.proj_in = nn.Linear(in_channels, inner_dim)
+
+ self.transformer_blocks = nn.ModuleList(
+ [
+ TemporalTransformerBlock(
+ dim=inner_dim,
+ num_attention_heads=num_attention_heads,
+ attention_head_dim=attention_head_dim,
+ num_attention_blocks=num_attention_blocks,
+ temporal_max_len=temporal_max_len,
+ pos_embedding_type=pos_embedding_type,
+ )
+ for d in range(num_layers)
+ ]
+ )
+ self.proj_out = nn.Linear(inner_dim, in_channels)
+
+ def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, cached_hidden_state_list=None):
+ assert hidden_states.dim() == 5, f"Expected hidden_states to have ndim=5, but got ndim={hidden_states.dim()}."
+ output_hidden_state_list = []
+
+ video_length = hidden_states.shape[2]
+ hidden_states = rearrange(hidden_states, "b c f h w -> (b f) c h w")
+
+ batch, channel, height, width = hidden_states.shape
+ residual = hidden_states
+
+ hidden_states = self.norm(hidden_states)
+ inner_dim = hidden_states.shape[1]
+ hidden_states = hidden_states.permute(0, 2, 3, 1).reshape(batch, height * width, inner_dim).contiguous()
+ hidden_states = self.proj_in(hidden_states)
+
+ # Transformer Blocks
+ if cached_hidden_state_list is not None:
+ n = len(cached_hidden_state_list) // len(self.transformer_blocks)
+ else:
+ n = 0
+ for i, block in enumerate(self.transformer_blocks):
+ hidden_states, hidden_state_list = block(hidden_states, encoder_hidden_states=encoder_hidden_states, video_length=video_length, attention_mask=attention_mask,
+ cached_hidden_state_list=cached_hidden_state_list[i*n:(i+1)*n] if n else None)
+ output_hidden_state_list.extend(hidden_state_list)
+
+ # output
+ hidden_states = self.proj_out(hidden_states)
+ hidden_states = hidden_states.reshape(batch, height, width, inner_dim).permute(0, 3, 1, 2).contiguous()
+
+ output = hidden_states + residual
+ output = rearrange(output, "(b f) c h w -> b c f h w", f=video_length)
+
+ return output, output_hidden_state_list
+
+
+class TemporalTransformerBlock(nn.Module):
+ def __init__(
+ self,
+ dim,
+ num_attention_heads,
+ attention_head_dim,
+ num_attention_blocks = 2,
+ temporal_max_len = 32,
+ pos_embedding_type = "ape",
+ ):
+ super().__init__()
+
+ self.attention_blocks = nn.ModuleList(
+ [
+ TemporalAttention(
+ query_dim=dim,
+ heads=num_attention_heads,
+ dim_head=attention_head_dim,
+ temporal_max_len=temporal_max_len,
+ pos_embedding_type=pos_embedding_type,
+ )
+ for i in range(num_attention_blocks)
+ ]
+ )
+ self.norms = nn.ModuleList(
+ [
+ nn.LayerNorm(dim)
+ for i in range(num_attention_blocks)
+ ]
+ )
+
+ self.ff = FeedForward(dim, dropout=0.0, activation_fn="geglu")
+ self.ff_norm = nn.LayerNorm(dim)
+
+
+ def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, video_length=None, cached_hidden_state_list=None):
+ output_hidden_state_list = []
+ for i, (attention_block, norm) in enumerate(zip(self.attention_blocks, self.norms)):
+ norm_hidden_states = norm(hidden_states)
+ residual_hidden_states, output_hidden_states = attention_block(
+ norm_hidden_states,
+ encoder_hidden_states=encoder_hidden_states,
+ video_length=video_length,
+ attention_mask=attention_mask,
+ cached_hidden_states=cached_hidden_state_list[i] if cached_hidden_state_list is not None else None,
+ )
+ hidden_states = residual_hidden_states + hidden_states
+ output_hidden_state_list.append(output_hidden_states)
+
+ hidden_states = self.ff(self.ff_norm(hidden_states)) + hidden_states
+
+ output = hidden_states
+ return output, output_hidden_state_list
+
+
+class PositionalEncoding(nn.Module):
+ def __init__(
+ self,
+ d_model,
+ dropout = 0.,
+ max_len = 32
+ ):
+ super().__init__()
+ self.dropout = nn.Dropout(p=dropout)
+ position = torch.arange(max_len).unsqueeze(1)
+ div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
+ pe = torch.zeros(1, max_len, d_model)
+ pe[0, :, 0::2] = torch.sin(position * div_term)
+ pe[0, :, 1::2] = torch.cos(position * div_term)
+ self.register_buffer('pe', pe)
+
+ def forward(self, x):
+ x = x + self.pe[:, :x.size(1)].to(x.dtype)
+ return self.dropout(x)
+
+class TemporalAttention(CrossAttention):
+ def __init__(
+ self,
+ temporal_max_len = 32,
+ pos_embedding_type = "ape",
+ *args, **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+
+ self.pos_embedding_type = pos_embedding_type
+ self._use_memory_efficient_attention_xformers = True
+
+ self.pos_encoder = None
+ self.freqs_cis = None
+ if self.pos_embedding_type == "ape":
+ self.pos_encoder = PositionalEncoding(
+ kwargs["query_dim"],
+ dropout=0.,
+ max_len=temporal_max_len
+ )
+
+ elif self.pos_embedding_type == "rope":
+ self.freqs_cis = precompute_freqs_cis(
+ kwargs["query_dim"],
+ temporal_max_len
+ )
+
+ else:
+ raise NotImplementedError
+
+ def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, video_length=None, cached_hidden_states=None):
+ # TODO: support cache for these
+ assert encoder_hidden_states is None
+ assert attention_mask is None
+
+ d = hidden_states.shape[1]
+ d_in = 0
+ if cached_hidden_states is None:
+ hidden_states = rearrange(hidden_states, "(b f) d c -> (b d) f c", f=video_length)
+ input_hidden_states = hidden_states # (bxd) f c
+ else:
+ hidden_states = rearrange(hidden_states, "(b f) d c -> (b d) f c", f=1)
+ input_hidden_states = hidden_states
+ d_in = cached_hidden_states.shape[1]
+ hidden_states = torch.cat([cached_hidden_states, hidden_states], dim=1)
+
+ if self.pos_encoder is not None:
+ hidden_states = self.pos_encoder(hidden_states)
+
+ encoder_hidden_states = repeat(encoder_hidden_states, "b n c -> (b d) n c", d=d) if encoder_hidden_states is not None else encoder_hidden_states
+
+ if self.group_norm is not None:
+ hidden_states = self.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)
+
+ query = self.to_q(hidden_states[:, d_in:, ...])
+ dim = query.shape[-1]
+
+ if self.added_kv_proj_dim is not None:
+ raise NotImplementedError
+
+ encoder_hidden_states = encoder_hidden_states if encoder_hidden_states is not None else hidden_states
+ key = self.to_k(encoder_hidden_states)
+ value = self.to_v(encoder_hidden_states)
+
+ if self.freqs_cis is not None:
+ seq_len = query.shape[1]
+ freqs_cis = self.freqs_cis[:seq_len].to(query.device)
+ query, key = apply_rotary_emb(query, key, freqs_cis)
+
+ if attention_mask is not None:
+ if attention_mask.shape[-1] != query.shape[1]:
+ target_length = query.shape[1]
+ attention_mask = F.pad(attention_mask, (0, target_length), value=0.0)
+ attention_mask = attention_mask.repeat_interleave(self.heads, dim=0)
+
+
+ use_memory_efficient = XFORMERS_AVAILABLE and self._use_memory_efficient_attention_xformers
+ if use_memory_efficient and (dim // self.heads) % 8 != 0:
+ # print('Warning: the dim {} cannot be divided by 8. Fall into normal attention'.format(dim // self.heads))
+ use_memory_efficient = False
+
+ # attention, what we cannot get enough of
+ if use_memory_efficient:
+ query = self.reshape_heads_to_4d(query)
+ key = self.reshape_heads_to_4d(key)
+ value = self.reshape_heads_to_4d(value)
+
+ hidden_states = self._memory_efficient_attention_xformers(query, key, value, attention_mask)
+ # Some versions of xformers return output in fp32, cast it back to the dtype of the input
+ hidden_states = hidden_states.to(query.dtype)
+ else:
+ query = self.reshape_heads_to_batch_dim(query)
+ key = self.reshape_heads_to_batch_dim(key)
+ value = self.reshape_heads_to_batch_dim(value)
+
+ if self._slice_size is None or query.shape[0] // self._slice_size == 1:
+ hidden_states = self._attention(query, key, value, attention_mask)
+ else:
+ raise NotImplementedError
+ # hidden_states = self._sliced_attention(query, key, value, sequence_length, dim, attention_mask)
+
+ # linear proj
+ hidden_states = self.to_out[0](hidden_states)
+
+ # dropout
+ hidden_states = self.to_out[1](hidden_states)
+
+ hidden_states = rearrange(hidden_states, "(b d) f c -> (b f) d c", d=d)
+
+ return hidden_states, input_hidden_states
diff --git a/video2blockout/third_party/video_depth_anything/util/__init__.py b/video2blockout/third_party/video_depth_anything/util/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/video2blockout/third_party/video_depth_anything/util/align.py b/video2blockout/third_party/video_depth_anything/util/align.py
new file mode 100644
index 0000000..75ff80a
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/util/align.py
@@ -0,0 +1,74 @@
+# Copyright (2025) Bytedance Ltd. and/or its affiliates
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import numpy as np
+
+def compute_scale_and_shift(prediction, target, mask, scale_only=False):
+ if scale_only:
+ return compute_scale(prediction, target, mask), 0
+ else:
+ return compute_scale_and_shift_full(prediction, target, mask)
+
+
+def compute_scale(prediction, target, mask):
+ # system matrix: A = [[a_00, a_01], [a_10, a_11]]
+ prediction = prediction.astype(np.float32)
+ target = target.astype(np.float32)
+ mask = mask.astype(np.float32)
+
+ a_00 = np.sum(mask * prediction * prediction)
+ a_01 = np.sum(mask * prediction)
+ a_11 = np.sum(mask)
+
+ # right hand side: b = [b_0, b_1]
+ b_0 = np.sum(mask * prediction * target)
+
+ x_0 = b_0 / (a_00 + 1e-6)
+
+ return x_0
+
+def compute_scale_and_shift_full(prediction, target, mask):
+ # system matrix: A = [[a_00, a_01], [a_10, a_11]]
+ prediction = prediction.astype(np.float32)
+ target = target.astype(np.float32)
+ mask = mask.astype(np.float32)
+
+ a_00 = np.sum(mask * prediction * prediction)
+ a_01 = np.sum(mask * prediction)
+ a_11 = np.sum(mask)
+
+ b_0 = np.sum(mask * prediction * target)
+ b_1 = np.sum(mask * target)
+
+ x_0 = 1
+ x_1 = 0
+
+ det = a_00 * a_11 - a_01 * a_01
+
+ if det != 0:
+ x_0 = (a_11 * b_0 - a_01 * b_1) / det
+ x_1 = (-a_01 * b_0 + a_00 * b_1) / det
+
+ return x_0, x_1
+
+
+def get_interpolate_frames(frame_list_pre, frame_list_post):
+ assert len(frame_list_pre) == len(frame_list_post)
+ min_w = 0.0
+ max_w = 1.0
+ step = (max_w - min_w) / (len(frame_list_pre)-1)
+ post_w_list = [min_w] + [i * step for i in range(1,len(frame_list_pre)-1)] + [max_w]
+ interpolated_frames = []
+ for i in range(len(frame_list_pre)):
+ interpolated_frames.append(frame_list_pre[i] * (1-post_w_list[i]) + frame_list_post[i] * post_w_list[i])
+ return interpolated_frames
\ No newline at end of file
diff --git a/video2blockout/third_party/video_depth_anything/util/blocks.py b/video2blockout/third_party/video_depth_anything/util/blocks.py
new file mode 100644
index 0000000..0be16c0
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/util/blocks.py
@@ -0,0 +1,162 @@
+import torch.nn as nn
+
+
+def _make_scratch(in_shape, out_shape, groups=1, expand=False):
+ scratch = nn.Module()
+
+ out_shape1 = out_shape
+ out_shape2 = out_shape
+ out_shape3 = out_shape
+ if len(in_shape) >= 4:
+ out_shape4 = out_shape
+
+ if expand:
+ out_shape1 = out_shape
+ out_shape2 = out_shape * 2
+ out_shape3 = out_shape * 4
+ if len(in_shape) >= 4:
+ out_shape4 = out_shape * 8
+
+ scratch.layer1_rn = nn.Conv2d(
+ in_shape[0], out_shape1, kernel_size=3, stride=1, padding=1, bias=False, groups=groups
+ )
+ scratch.layer2_rn = nn.Conv2d(
+ in_shape[1], out_shape2, kernel_size=3, stride=1, padding=1, bias=False, groups=groups
+ )
+ scratch.layer3_rn = nn.Conv2d(
+ in_shape[2], out_shape3, kernel_size=3, stride=1, padding=1, bias=False, groups=groups
+ )
+ if len(in_shape) >= 4:
+ scratch.layer4_rn = nn.Conv2d(
+ in_shape[3], out_shape4, kernel_size=3, stride=1, padding=1, bias=False, groups=groups
+ )
+
+ return scratch
+
+
+class ResidualConvUnit(nn.Module):
+ """Residual convolution module."""
+
+ def __init__(self, features, activation, bn):
+ """Init.
+
+ Args:
+ features (int): number of features
+ """
+ super().__init__()
+
+ self.bn = bn
+
+ self.groups = 1
+
+ self.conv1 = nn.Conv2d(
+ features, features, kernel_size=3, stride=1, padding=1, bias=True, groups=self.groups
+ )
+
+ self.conv2 = nn.Conv2d(
+ features, features, kernel_size=3, stride=1, padding=1, bias=True, groups=self.groups
+ )
+
+ if self.bn is True:
+ self.bn1 = nn.BatchNorm2d(features)
+ self.bn2 = nn.BatchNorm2d(features)
+
+ self.activation = activation
+
+ self.skip_add = nn.quantized.FloatFunctional()
+
+ def forward(self, x):
+ """Forward pass.
+
+ Args:
+ x (tensor): input
+
+ Returns:
+ tensor: output
+ """
+
+ out = self.activation(x)
+ out = self.conv1(out)
+ if self.bn is True:
+ out = self.bn1(out)
+
+ out = self.activation(out)
+ out = self.conv2(out)
+ if self.bn is True:
+ out = self.bn2(out)
+
+ if self.groups > 1:
+ out = self.conv_merge(out)
+
+ return self.skip_add.add(out, x)
+
+
+class FeatureFusionBlock(nn.Module):
+ """Feature fusion block."""
+
+ def __init__(
+ self,
+ features,
+ activation,
+ deconv=False,
+ bn=False,
+ expand=False,
+ align_corners=True,
+ size=None,
+ ):
+ """Init.
+
+ Args:
+ features (int): number of features
+ """
+ super().__init__()
+
+ self.deconv = deconv
+ self.align_corners = align_corners
+
+ self.groups = 1
+
+ self.expand = expand
+ out_features = features
+ if self.expand is True:
+ out_features = features // 2
+
+ self.out_conv = nn.Conv2d(
+ features, out_features, kernel_size=1, stride=1, padding=0, bias=True, groups=1
+ )
+
+ self.resConfUnit1 = ResidualConvUnit(features, activation, bn)
+ self.resConfUnit2 = ResidualConvUnit(features, activation, bn)
+
+ self.skip_add = nn.quantized.FloatFunctional()
+
+ self.size = size
+
+ def forward(self, *xs, size=None):
+ """Forward pass.
+
+ Returns:
+ tensor: output
+ """
+ output = xs[0]
+
+ if len(xs) == 2:
+ res = self.resConfUnit1(xs[1])
+ output = self.skip_add.add(output, res)
+
+ output = self.resConfUnit2(output)
+
+ if (size is None) and (self.size is None):
+ modifier = {"scale_factor": 2}
+ elif size is None:
+ modifier = {"size": self.size}
+ else:
+ modifier = {"size": size}
+
+ output = nn.functional.interpolate(
+ output.contiguous(), **modifier, mode="bilinear", align_corners=self.align_corners
+ )
+
+ output = self.out_conv(output)
+
+ return output
diff --git a/video2blockout/third_party/video_depth_anything/util/transform.py b/video2blockout/third_party/video_depth_anything/util/transform.py
new file mode 100644
index 0000000..b14aacd
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/util/transform.py
@@ -0,0 +1,158 @@
+import numpy as np
+import cv2
+
+
+class Resize(object):
+ """Resize sample to given size (width, height).
+ """
+
+ def __init__(
+ self,
+ width,
+ height,
+ resize_target=True,
+ keep_aspect_ratio=False,
+ ensure_multiple_of=1,
+ resize_method="lower_bound",
+ image_interpolation_method=cv2.INTER_AREA,
+ ):
+ """Init.
+
+ Args:
+ width (int): desired output width
+ height (int): desired output height
+ resize_target (bool, optional):
+ True: Resize the full sample (image, mask, target).
+ False: Resize image only.
+ Defaults to True.
+ keep_aspect_ratio (bool, optional):
+ True: Keep the aspect ratio of the input sample.
+ Output sample might not have the given width and height, and
+ resize behaviour depends on the parameter 'resize_method'.
+ Defaults to False.
+ ensure_multiple_of (int, optional):
+ Output width and height is constrained to be multiple of this parameter.
+ Defaults to 1.
+ resize_method (str, optional):
+ "lower_bound": Output will be at least as large as the given size.
+ "upper_bound": Output will be at max as large as the given size. (Output size might be smaller than given size.)
+ "minimal": Scale as least as possible. (Output size might be smaller than given size.)
+ Defaults to "lower_bound".
+ """
+ self.__width = width
+ self.__height = height
+
+ self.__resize_target = resize_target
+ self.__keep_aspect_ratio = keep_aspect_ratio
+ self.__multiple_of = ensure_multiple_of
+ self.__resize_method = resize_method
+ self.__image_interpolation_method = image_interpolation_method
+
+ def constrain_to_multiple_of(self, x, min_val=0, max_val=None):
+ y = (np.round(x / self.__multiple_of) * self.__multiple_of).astype(int)
+
+ if max_val is not None and y > max_val:
+ y = (np.floor(x / self.__multiple_of) * self.__multiple_of).astype(int)
+
+ if y < min_val:
+ y = (np.ceil(x / self.__multiple_of) * self.__multiple_of).astype(int)
+
+ return y
+
+ def get_size(self, width, height):
+ # determine new height and width
+ scale_height = self.__height / height
+ scale_width = self.__width / width
+
+ if self.__keep_aspect_ratio:
+ if self.__resize_method == "lower_bound":
+ # scale such that output size is lower bound
+ if scale_width > scale_height:
+ # fit width
+ scale_height = scale_width
+ else:
+ # fit height
+ scale_width = scale_height
+ elif self.__resize_method == "upper_bound":
+ # scale such that output size is upper bound
+ if scale_width < scale_height:
+ # fit width
+ scale_height = scale_width
+ else:
+ # fit height
+ scale_width = scale_height
+ elif self.__resize_method == "minimal":
+ # scale as least as possbile
+ if abs(1 - scale_width) < abs(1 - scale_height):
+ # fit width
+ scale_height = scale_width
+ else:
+ # fit height
+ scale_width = scale_height
+ else:
+ raise ValueError(f"resize_method {self.__resize_method} not implemented")
+
+ if self.__resize_method == "lower_bound":
+ new_height = self.constrain_to_multiple_of(scale_height * height, min_val=self.__height)
+ new_width = self.constrain_to_multiple_of(scale_width * width, min_val=self.__width)
+ elif self.__resize_method == "upper_bound":
+ new_height = self.constrain_to_multiple_of(scale_height * height, max_val=self.__height)
+ new_width = self.constrain_to_multiple_of(scale_width * width, max_val=self.__width)
+ elif self.__resize_method == "minimal":
+ new_height = self.constrain_to_multiple_of(scale_height * height)
+ new_width = self.constrain_to_multiple_of(scale_width * width)
+ else:
+ raise ValueError(f"resize_method {self.__resize_method} not implemented")
+
+ return (new_width, new_height)
+
+ def __call__(self, sample):
+ width, height = self.get_size(sample["image"].shape[1], sample["image"].shape[0])
+
+ # resize sample
+ sample["image"] = cv2.resize(sample["image"], (width, height), interpolation=self.__image_interpolation_method)
+
+ if self.__resize_target:
+ if "depth" in sample:
+ sample["depth"] = cv2.resize(sample["depth"], (width, height), interpolation=cv2.INTER_NEAREST)
+
+ if "mask" in sample:
+ sample["mask"] = cv2.resize(sample["mask"].astype(np.float32), (width, height), interpolation=cv2.INTER_NEAREST)
+
+ return sample
+
+
+class NormalizeImage(object):
+ """Normlize image by given mean and std.
+ """
+
+ def __init__(self, mean, std):
+ self.__mean = mean
+ self.__std = std
+
+ def __call__(self, sample):
+ sample["image"] = (sample["image"] - self.__mean) / self.__std
+
+ return sample
+
+
+class PrepareForNet(object):
+ """Prepare sample for usage as network input.
+ """
+
+ def __init__(self):
+ pass
+
+ def __call__(self, sample):
+ image = np.transpose(sample["image"], (2, 0, 1))
+ sample["image"] = np.ascontiguousarray(image).astype(np.float32)
+
+ if "depth" in sample:
+ depth = sample["depth"].astype(np.float32)
+ sample["depth"] = np.ascontiguousarray(depth)
+
+ if "mask" in sample:
+ sample["mask"] = sample["mask"].astype(np.float32)
+ sample["mask"] = np.ascontiguousarray(sample["mask"])
+
+ return sample
\ No newline at end of file
diff --git a/video2blockout/third_party/video_depth_anything/video_depth.py b/video2blockout/third_party/video_depth_anything/video_depth.py
new file mode 100644
index 0000000..cf3bce6
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/video_depth.py
@@ -0,0 +1,163 @@
+# Copyright (2025) Bytedance Ltd. and/or its affiliates
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+import torch.nn.functional as F
+import torch.nn as nn
+from torchvision.transforms import Compose
+import cv2
+from tqdm import tqdm
+import numpy as np
+import gc
+
+from .dinov2 import DINOv2
+from .dpt_temporal import DPTHeadTemporal
+from .util.transform import Resize, NormalizeImage, PrepareForNet
+
+from .util.align import compute_scale_and_shift, get_interpolate_frames # vendored: was `utils.util`
+
+# infer settings, do not change
+INFER_LEN = 32
+OVERLAP = 10
+KEYFRAMES = [0,12,24,25,26,27,28,29,30,31]
+INTERP_LEN = 8
+
+class VideoDepthAnything(nn.Module):
+ def __init__(
+ self,
+ encoder='vitl',
+ features=256,
+ out_channels=[256, 512, 1024, 1024],
+ use_bn=False,
+ use_clstoken=False,
+ num_frames=32,
+ pe='ape',
+ metric=False,
+ ):
+ super(VideoDepthAnything, self).__init__()
+
+ self.intermediate_layer_idx = {
+ 'vits': [2, 5, 8, 11],
+ "vitb": [2, 5, 8, 11],
+ 'vitl': [4, 11, 17, 23]
+ }
+
+ self.encoder = encoder
+ self.pretrained = DINOv2(model_name=encoder)
+
+ self.head = DPTHeadTemporal(self.pretrained.embed_dim, features, use_bn, out_channels=out_channels, use_clstoken=use_clstoken, num_frames=num_frames, pe=pe)
+ self.metric = metric
+
+ def forward(self, x):
+ B, T, C, H, W = x.shape
+ patch_h, patch_w = H // 14, W // 14
+ features = self.pretrained.get_intermediate_layers(x.flatten(0,1), self.intermediate_layer_idx[self.encoder], return_class_token=True)
+ depth = self.head(features, patch_h, patch_w, T)[0]
+ depth = F.interpolate(depth, size=(H, W), mode="bilinear", align_corners=True)
+ depth = F.relu(depth)
+ return depth.squeeze(1).unflatten(0, (B, T)) # return shape [B, T, H, W]
+
+ def infer_video_depth(self, frames, target_fps, input_size=518, device='cuda', fp32=False):
+ frame_height, frame_width = frames[0].shape[:2]
+ ratio = max(frame_height, frame_width) / min(frame_height, frame_width)
+ if ratio > 1.78: # we recommend to process video with ratio smaller than 16:9 due to memory limitation
+ input_size = int(input_size * 1.777 / ratio)
+ input_size = round(input_size / 14) * 14
+
+ transform = Compose([
+ Resize(
+ width=input_size,
+ height=input_size,
+ resize_target=False,
+ keep_aspect_ratio=True,
+ ensure_multiple_of=14,
+ resize_method='lower_bound',
+ image_interpolation_method=cv2.INTER_CUBIC,
+ ),
+ NormalizeImage(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+ PrepareForNet(),
+ ])
+
+ frame_list = [frames[i] for i in range(frames.shape[0])]
+ frame_step = INFER_LEN - OVERLAP
+ org_video_len = len(frame_list)
+ append_frame_len = (frame_step - (org_video_len % frame_step)) % frame_step + (INFER_LEN - frame_step)
+ frame_list = frame_list + [frame_list[-1].copy()] * append_frame_len
+
+ depth_list = []
+ pre_input = None
+ for frame_id in tqdm(range(0, org_video_len, frame_step)):
+ cur_list = []
+ for i in range(INFER_LEN):
+ cur_list.append(torch.from_numpy(transform({'image': frame_list[frame_id+i].astype(np.float32) / 255.0})['image']).unsqueeze(0).unsqueeze(0))
+ cur_input = torch.cat(cur_list, dim=1).to(device)
+ if pre_input is not None:
+ cur_input[:, :OVERLAP, ...] = pre_input[:, KEYFRAMES, ...]
+
+ with torch.no_grad():
+ with torch.autocast(device_type=device, enabled=(not fp32)):
+ depth = self.forward(cur_input) # depth shape: [1, T, H, W]
+
+ depth = depth.to(cur_input.dtype)
+ depth = F.interpolate(depth.flatten(0,1).unsqueeze(1), size=(frame_height, frame_width), mode='bilinear', align_corners=True)
+ depth_list += [depth[i][0].cpu().numpy() for i in range(depth.shape[0])]
+
+ pre_input = cur_input
+
+ del frame_list
+ gc.collect()
+
+ depth_list_aligned = []
+ ref_align = []
+ align_len = OVERLAP - INTERP_LEN
+ kf_align_list = KEYFRAMES[:align_len]
+
+ for frame_id in range(0, len(depth_list), INFER_LEN):
+ if len(depth_list_aligned) == 0:
+ depth_list_aligned += depth_list[:INFER_LEN]
+ for kf_id in kf_align_list:
+ ref_align.append(depth_list[frame_id+kf_id])
+ else:
+ curr_align = []
+ for i in range(len(kf_align_list)):
+ curr_align.append(depth_list[frame_id+i])
+
+ if self.metric:
+ scale, shift = 1.0, 0.0
+ else:
+ scale, shift = compute_scale_and_shift(np.concatenate(curr_align),
+ np.concatenate(ref_align),
+ np.concatenate(np.ones_like(ref_align)==1))
+
+ pre_depth_list = depth_list_aligned[-INTERP_LEN:]
+ post_depth_list = depth_list[frame_id+align_len:frame_id+OVERLAP]
+ for i in range(len(post_depth_list)):
+ post_depth_list[i] = post_depth_list[i] * scale + shift
+ post_depth_list[i][post_depth_list[i]<0] = 0
+ depth_list_aligned[-INTERP_LEN:] = get_interpolate_frames(pre_depth_list, post_depth_list)
+
+ for i in range(OVERLAP, INFER_LEN):
+ new_depth = depth_list[frame_id+i] * scale + shift
+ new_depth[new_depth<0] = 0
+ depth_list_aligned.append(new_depth)
+
+ ref_align = ref_align[:1]
+ for kf_id in kf_align_list[1:]:
+ new_depth = depth_list[frame_id+kf_id] * scale + shift
+ new_depth[new_depth<0] = 0
+ ref_align.append(new_depth)
+
+ depth_list = depth_list_aligned
+
+ return np.stack(depth_list[:org_video_len], axis=0), target_fps
+
diff --git a/video2blockout/third_party/video_depth_anything/video_depth_stream.py b/video2blockout/third_party/video_depth_anything/video_depth_stream.py
new file mode 100644
index 0000000..dbee3fc
--- /dev/null
+++ b/video2blockout/third_party/video_depth_anything/video_depth_stream.py
@@ -0,0 +1,161 @@
+# Copyright (2025) Bytedance Ltd. and/or its affiliates
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import torch
+import torch.nn.functional as F
+import torch.nn as nn
+from torchvision.transforms import Compose
+import cv2
+import numpy as np
+
+from .dinov2 import DINOv2
+from .dpt_temporal import DPTHeadTemporal
+from .util.transform import Resize, NormalizeImage, PrepareForNet
+
+from .util.align import compute_scale_and_shift, get_interpolate_frames # vendored: was `utils.util`
+
+# infer settings, do not change
+INFER_LEN = 32
+OVERLAP = 10
+INTERP_LEN = 8
+
+class VideoDepthAnything(nn.Module):
+ def __init__(
+ self,
+ encoder='vitl',
+ features=256,
+ out_channels=[256, 512, 1024, 1024],
+ use_bn=False,
+ use_clstoken=False,
+ num_frames=32,
+ pe='ape'
+ ):
+ super(VideoDepthAnything, self).__init__()
+
+ self.intermediate_layer_idx = {
+ 'vits': [2, 5, 8, 11],
+ "vitb": [2, 5, 8, 11],
+ 'vitl': [4, 11, 17, 23]
+ }
+
+ self.encoder = encoder
+ self.pretrained = DINOv2(model_name=encoder)
+
+ self.head = DPTHeadTemporal(self.pretrained.embed_dim, features, use_bn, out_channels=out_channels, use_clstoken=use_clstoken, num_frames=num_frames, pe=pe)
+ self.transform = None
+ self.frame_id_list = []
+ self.frame_cache_list = []
+ self.gap = (INFER_LEN - OVERLAP) * 2 - 1 - (OVERLAP - INTERP_LEN)
+ assert self.gap == 41
+ self.id = -1
+
+ def forward(self, x):
+ return self.forward_depth(self.forward_features(x), x.shape)[0]
+
+ def forward_features(self, x):
+ features = self.pretrained.get_intermediate_layers(x.flatten(0,1), self.intermediate_layer_idx[self.encoder], return_class_token=True)
+ return features
+
+ def forward_depth(self, features, x_shape, cached_hidden_state_list=None):
+ B, T, C, H, W = x_shape
+ patch_h, patch_w = H // 14, W // 14
+ depth, cur_cached_hidden_state_list = self.head(features, patch_h, patch_w, T, cached_hidden_state_list=cached_hidden_state_list)
+ depth = F.interpolate(depth, size=(H, W), mode="bilinear", align_corners=True)
+ depth = F.relu(depth)
+ return depth.squeeze(1).unflatten(0, (B, T)), cur_cached_hidden_state_list # return shape [B, T, H, W]
+
+ def infer_video_depth_one(self, frame, input_size=518, device='cuda', fp32=False):
+ self.id += 1
+
+ if self.transform is None: # first frame
+ # Initialize the transform
+ frame_height, frame_width = frame.shape[:2]
+ self.frame_height = frame_height
+ self.frame_width = frame_width
+ ratio = max(frame_height, frame_width) / min(frame_height, frame_width)
+ if ratio > 1.78: # we recommend to process video with ratio smaller than 16:9 due to memory limitation
+ input_size = int(input_size * 1.777 / ratio)
+ input_size = round(input_size / 14) * 14
+
+ self.transform = Compose([
+ Resize(
+ width=input_size,
+ height=input_size,
+ resize_target=False,
+ keep_aspect_ratio=True,
+ ensure_multiple_of=14,
+ resize_method='lower_bound',
+ image_interpolation_method=cv2.INTER_CUBIC,
+ ),
+ NormalizeImage(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
+ PrepareForNet(),
+ ])
+
+ # Inference the first frame
+ cur_list = [torch.from_numpy(self.transform({'image': frame.astype(np.float32) / 255.0})['image']).unsqueeze(0).unsqueeze(0)]
+ cur_input = torch.cat(cur_list, dim=1).to(device)
+
+ with torch.no_grad():
+ with torch.autocast(device_type=device, enabled=(not fp32)):
+ cur_feature = self.forward_features(cur_input)
+ x_shape = cur_input.shape
+ depth, cached_hidden_state_list = self.forward_depth(cur_feature, x_shape)
+
+ depth = depth.to(cur_input.dtype)
+ depth = F.interpolate(depth.flatten(0,1).unsqueeze(1), size=(frame_height, frame_width), mode='bilinear', align_corners=True)
+
+ # Copy multiple cache to simulate the windows
+ self.frame_cache_list = [cached_hidden_state_list] * INFER_LEN
+ self.frame_id_list.extend([0] * (INFER_LEN - 1))
+
+ new_depth = depth[0][0].cpu().numpy()
+ else:
+ frame_height, frame_width = frame.shape[:2]
+ assert frame_height == self.frame_height
+ assert frame_width == self.frame_width
+
+ # infer feature
+ cur_input = torch.from_numpy(self.transform({'image': frame.astype(np.float32) / 255.0})['image']).unsqueeze(0).unsqueeze(0).to(device)
+ with torch.no_grad():
+ with torch.autocast(device_type=device, enabled=(not fp32)):
+ cur_feature = self.forward_features(cur_input)
+ x_shape = cur_input.shape
+
+ cur_list = self.frame_cache_list[0:2] + self.frame_cache_list[-INFER_LEN+3:]
+ '''
+ cur_id = self.frame_id_list[0:2] + self.frame_id_list[-INFER_LEN+3:]
+ print(f"cur_id: {cur_id}")
+ '''
+ assert len(cur_list) == INFER_LEN - 1
+ cur_cache = [torch.cat([h[i] for h in cur_list], dim=1) for i in range(len(cur_list[0]))]
+
+ # infer depth
+ with torch.no_grad():
+ with torch.autocast(device_type=device, enabled=(not fp32)):
+ depth, new_cache = self.forward_depth(cur_feature, x_shape, cached_hidden_state_list=cur_cache)
+
+ depth = depth.to(cur_input.dtype)
+ depth = F.interpolate(depth.flatten(0,1).unsqueeze(1), size=(frame_height, frame_width), mode='bilinear', align_corners=True)
+ depth_list = [depth[i][0].cpu().numpy() for i in range(depth.shape[0])]
+
+ new_depth = depth_list[-1]
+
+ self.frame_cache_list.append(new_cache)
+
+ # adjust the sliding window
+ self.frame_id_list.append(self.id)
+ if self.id + INFER_LEN > self.gap + 1:
+ del self.frame_id_list[1]
+ del self.frame_cache_list[1]
+
+ return new_depth