ReShot — copy the shot, not the actors. Video → depth map, OpenPose skeleton or canny lines for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Apache-2.0. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system.
Python ai-short-dramaai-videocannycomfyuicontrolnetdepth-anythingdepth-estimationdwposemaosikaminimaxopenposepose-controlnetpose-estimationseedanceshort-dramavideo-generationvideo-to-videowan
video2blockout 0.1.0: video → depth blockout video for video-generation ControlNets
wangfengcd490fa795a5
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlnew file mode 100644index 0000000..2d36c29--- /dev/null+++ b/.github/workflows/ci.yml@@ -0,0 +1,40 @@+name: ci+on:+ push: { branches: [main] }+ pull_request:+jobs:+ unit:+ name: unit (${{ matrix.os }}, py${{ matrix.python }})+ runs-on: ${{ matrix.os }}+ strategy:+ fail-fast: false+ matrix:+ os: [ubuntu-latest, windows-latest]+ python: ["3.10", "3.12"]+ steps:+ - uses: actions/checkout@v4+ - uses: actions/setup-python@v5+ with: { python-version: "${{ matrix.python }}" }+ - name: Install (CPU torch)+ run: |+ python -m pip install -U pip+ python -m pip install torch --index-url https://download.pytorch.org/whl/cpu+ python -m pip install -e ".[ffmpeg,dev]"+ - run: python -m pytest -q tests++ model-smoke:+ name: real model on CPU (8 frames)+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@v4+ - uses: actions/setup-python@v5+ with: { python-version: "3.12" }+ - uses: actions/cache@v4+ with:+ path: ~/.cache/huggingface+ key: hf-vda-small-v1+ - run: |+ python -m pip install -U pip+ python -m pip install torch --index-url https://download.pytorch.org/whl/cpu+ python -m pip install -e ".[ffmpeg,dev]"+ - run: RUN_MODEL_TESTS=1 python -m pytest -q tests/test_model_smoke.py -sdiff --git a/.gitignore b/.gitignorenew file mode 100644index 0000000..b91c3cd--- /dev/null+++ b/.gitignore@@ -0,0 +1,11 @@+__pycache__/+*.pyc+*.egg-info/+build/+dist/+.pytest_cache/+.ruff_cache/+*.npz+outputs/+examples/*_depth.mp4+runs/diff --git a/CHANGELOG.md b/CHANGELOG.mdnew file mode 100644index 0000000..12a6646--- /dev/null+++ b/CHANGELOG.md@@ -0,0 +1,8 @@+# Changelog++## 0.1.0 — 2026-09-11+- First release: `video2blockout in.mp4 -o out.mp4` on Video Depth Anything Small (Apache-2.0).+- Depth estimated at model resolution, whole-clip normalisation, near = white, x264 CRF 12.+- `--target h3` / `wan` presets (fps, size multiple, max length), `--npz`, `--metrics`.+- Host-RAM estimate with refusal over 50 % of physical RAM; MPS allocator cap on Apple Silicon.+- Measured: RTX 4090 fp16 70 ms/frame; Apple M2 Max fp32 ~500 ms/frame.diff --git a/LICENSE b/LICENSEnew file mode 100644index 0000000..d645695--- /dev/null+++ b/LICENSE@@ -0,0 +1,202 @@++ Apache License+ Version 2.0, January 2004+ http://www.apache.org/licenses/++ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION++ 1. Definitions.++ "License" shall mean the terms and conditions for use, reproduction,+ and distribution as defined by Sections 1 through 9 of this document.++ "Licensor" shall mean the copyright owner or entity authorized by+ the copyright owner that is granting the License.++ "Legal Entity" shall mean the union of the acting entity and all+ other entities that control, are controlled by, or are under common+ control with that entity. For the purposes of this definition,+ "control" means (i) the power, direct or indirect, to cause the+ direction or management of such entity, whether by contract or+ otherwise, or (ii) ownership of fifty percent (50%) or more of the+ outstanding shares, or (iii) beneficial ownership of such entity.++ "You" (or "Your") shall mean an individual or Legal Entity+ exercising permissions granted by this License.++ "Source" form shall mean the preferred form for making modifications,+ including but not limited to software source code, documentation+ source, and configuration files.++ "Object" form shall mean any form resulting from mechanical+ transformation or translation of a Source form, including but+ not limited to compiled object code, generated documentation,+ and conversions to other media types.++ "Work" shall mean the work of authorship, whether in Source or+ Object form, made available under the License, as indicated by a+ copyright notice that is included in or attached to the work+ (an example is provided in the Appendix below).++ "Derivative Works" shall mean any work, whether in Source or Object+ form, that is based on (or derived from) the Work and for which the+ editorial revisions, annotations, elaborations, or other modifications+ represent, as a whole, an original work of authorship. For the purposes+ of this License, Derivative Works shall not include works that remain+ separable from, or merely link (or bind by name) to the interfaces of,+ the Work and Derivative Works thereof.++ "Contribution" shall mean any work of authorship, including+ the original version of the Work and any modifications or additions+ to that Work or Derivative Works thereof, that is intentionally+ submitted to Licensor for inclusion in the Work by the copyright owner+ or by an individual or Legal Entity authorized to submit on behalf of+ the copyright owner. For the purposes of this definition, "submitted"+ means any form of electronic, verbal, or written communication sent+ to the Licensor or its representatives, including but not limited to+ communication on electronic mailing lists, source code control systems,+ and issue tracking systems that are managed by, or on behalf of, the+ Licensor for the purpose of discussing and improving the Work, but+ excluding communication that is conspicuously marked or otherwise+ designated in writing by the copyright owner as "Not a Contribution."++ "Contributor" shall mean Licensor and any individual or Legal Entity+ on behalf of whom a Contribution has been received by Licensor and+ subsequently incorporated within the Work.++ 2. Grant of Copyright License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ copyright license to reproduce, prepare Derivative Works of,+ publicly display, publicly perform, sublicense, and distribute the+ Work and such Derivative Works in Source or Object form.++ 3. Grant of Patent License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ (except as stated in this section) patent license to make, have made,+ use, offer to sell, sell, import, and otherwise transfer the Work,+ where such license applies only to those patent claims licensable+ by such Contributor that are necessarily infringed by their+ Contribution(s) alone or by combination of their Contribution(s)+ with the Work to which such Contribution(s) was submitted. If You+ institute patent litigation against any entity (including a+ cross-claim or counterclaim in a lawsuit) alleging that the Work+ or a Contribution incorporated within the Work constitutes direct+ or contributory patent infringement, then any patent licenses+ granted to You under this License for that Work shall terminate+ as of the date such litigation is filed.++ 4. Redistribution. You may reproduce and distribute copies of the+ Work or Derivative Works thereof in any medium, with or without+ modifications, and in Source or Object form, provided that You+ meet the following conditions:++ (a) You must give any other recipients of the Work or+ Derivative Works a copy of this License; and++ (b) You must cause any modified files to carry prominent notices+ stating that You changed the files; and++ (c) You must retain, in the Source form of any Derivative Works+ that You distribute, all copyright, patent, trademark, and+ attribution notices from the Source form of the Work,+ excluding those notices that do not pertain to any part of+ the Derivative Works; and++ (d) If the Work includes a "NOTICE" text file as part of its+ distribution, then any Derivative Works that You distribute must+ include a readable copy of the attribution notices contained+ within such NOTICE file, excluding those notices that do not+ pertain to any part of the Derivative Works, in at least one+ of the following places: within a NOTICE text file distributed+ as part of the Derivative Works; within the Source form or+ documentation, if provided along with the Derivative Works; or,+ within a display generated by the Derivative Works, if and+ wherever such third-party notices normally appear. The contents+ of the NOTICE file are for informational purposes only and+ do not modify the License. You may add Your own attribution+ notices within Derivative Works that You distribute, alongside+ or as an addendum to the NOTICE text from the Work, provided+ that such additional attribution notices cannot be construed+ as modifying the License.++ You may add Your own copyright statement to Your modifications and+ may provide additional or different license terms and conditions+ for use, reproduction, or distribution of Your modifications, or+ for any such Derivative Works as a whole, provided Your use,+ reproduction, and distribution of the Work otherwise complies with+ the conditions stated in this License.++ 5. Submission of Contributions. Unless You explicitly state otherwise,+ any Contribution intentionally submitted for inclusion in the Work+ by You to the Licensor shall be under the terms and conditions of+ this License, without any additional terms or conditions.+ Notwithstanding the above, nothing herein shall supersede or modify+ the terms of any separate license agreement you may have executed+ with Licensor regarding such Contributions.++ 6. Trademarks. This License does not grant permission to use the trade+ names, trademarks, service marks, or product names of the Licensor,+ except as required for reasonable and customary use in describing the+ origin of the Work and reproducing the content of the NOTICE file.++ 7. Disclaimer of Warranty. Unless required by applicable law or+ agreed to in writing, Licensor provides the Work (and each+ Contributor provides its Contributions) on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or+ implied, including, without limitation, any warranties or conditions+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A+ PARTICULAR PURPOSE. You are solely responsible for determining the+ appropriateness of using or redistributing the Work and assume any+ risks associated with Your exercise of permissions under this License.++ 8. Limitation of Liability. In no event and under no legal theory,+ whether in tort (including negligence), contract, or otherwise,+ unless required by applicable law (such as deliberate and grossly+ negligent acts) or agreed to in writing, shall any Contributor be+ liable to You for damages, including any direct, indirect, special,+ incidental, or consequential damages of any character arising as a+ result of this License or out of the use or inability to use the+ Work (including but not limited to damages for loss of goodwill,+ work stoppage, computer failure or malfunction, or any and all+ other commercial damages or losses), even if such Contributor+ has been advised of the possibility of such damages.++ 9. Accepting Warranty or Additional Liability. While redistributing+ the Work or Derivative Works thereof, You may choose to offer,+ and charge a fee for, acceptance of support, warranty, indemnity,+ or other liability obligations and/or rights consistent with this+ License. However, in accepting such obligations, You may act only+ on Your own behalf and on Your sole responsibility, not on behalf+ of any other Contributor, and only if You agree to indemnify,+ defend, and hold each Contributor harmless for any liability+ incurred by, or claims asserted against, such Contributor by reason+ of your accepting any such warranty or additional liability.++ END OF TERMS AND CONDITIONS++ APPENDIX: How to apply the Apache License to your work.++ To apply the Apache License to your work, attach the following+ boilerplate notice, with the fields enclosed by brackets "[]"+ replaced with your own identifying information. (Don't include+ the brackets!) The text should be enclosed in the appropriate+ comment syntax for the file format. We also recommend that a+ file or class name and description of purpose be included on the+ same "printed page" as the copyright notice for easier+ identification within third-party archives.++ Copyright [yyyy] [name of copyright owner]++ Licensed under the Apache License, Version 2.0 (the "License");+ you may not use this file except in compliance with the License.+ You may obtain a copy of the License at++ http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing, software+ distributed under the License is distributed on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ See the License for the specific language governing permissions and+ limitations under the License.diff --git a/NOTICE b/NOTICEnew file mode 100644index 0000000..9bd0966--- /dev/null+++ b/NOTICE@@ -0,0 +1,12 @@+video2blockout+Copyright 2026 Maoska++This product includes software developed by Bytedance Ltd. and/or its affiliates:+Video Depth Anything (https://github.com/DepthAnything/Video-Depth-Anything),+licensed under the Apache License 2.0. It is vendored, unmodified except for+two import lines, under video2blockout/third_party/video_depth_anything/.++Model weights are downloaded at runtime from Hugging Face and carry their own licences:+ depth-anything/Video-Depth-Anything-Small Apache-2.0 (default; commercial use OK)+ depth-anything/Video-Depth-Anything-Base CC-BY-NC-4.0 (opt-in; non-commercial)+ depth-anything/Video-Depth-Anything-Large CC-BY-NC-4.0 (opt-in; non-commercial)diff --git a/README.md b/README.mdnew file mode 100644index 0000000..b9ab44c--- /dev/null+++ b/README.md@@ -0,0 +1,110 @@+# video2blockout++[](https://github.com/maosika-ai/video2blockout/actions/workflows/ci.yml) [](LICENSE)++Turn any video into a **depth "blockout" video** — the grey near-white / far-black+clip that video-generation ControlNets (MiniMax H3 Fun ControlNet, Wan VACE, …) use to+copy the *staging and camera* of a reference shot without copying its faces, clothes+or style.++<p align="center"><img src="docs/demo.gif" width="560" alt="left: source, right: depth blockout"></p>++One command, one model download (Apache-2.0), runs on an 8 GB GPU or an Apple M-series.++[中文说明](README.zh-CN.md)++## Install++```bash+pip install git+https://github.com/maosika-ai/video2blockout+# needs ffmpeg on PATH (or: pip install "video2blockout[ffmpeg]")+```++Weights (111 MB) download from Hugging Face on first run. In China set+`HF_ENDPOINT=https://hf-mirror.com`.++## Use++```bash+video2blockout in.mp4 -o blockout.mp4 # keep fps/size, near = white+video2blockout in.mp4 -o blockout.mp4 --target h3 # MiniMax H3: 24 fps, ×32 dims, ≤15 s+video2blockout in.mp4 -o blockout.mp4 --npz depth.npz # also keep raw float depth+```++```python+from video2blockout import extract, to_gray, write_gray_video+depths, fps = extract("in.mp4") # float32 [T, H, W], larger = closer+write_gray_video(to_gray(depths), "out.mp4", fps)+```++| Option | Default | What it does |+|---|---|---|+| `--model small\|base\|large` | `small` | Only `small` is Apache-2.0. `base`/`large` are CC-BY-NC and print a warning. |+| `--target none\|h3\|wan` | `none` | fps + frame-size preset for a generator (see below). |+| `--fps N` | source | Override output fps. Frames are picked by timestamp, so 30 → 24 really gives 24. |+| `--max-res N` | 1280 | Cap the longer side before inference; output is written at that size. |+| `--invert` | off | far = white instead of near = white. |+| `--clip PCT` | 0 | Percentile clip on both tails so one hot pixel can't crush contrast. |+| `--gamma G` | 1.0 | > 1 darkens mid-tones (more separation near camera). |+| `--crf N` | 12 | x264 quality. Deliberately generous — banding in a control video becomes jitter in the output. |+| `--npz PATH` | – | Save raw float depth for your own post-processing. |++### Targets++| `--target` | fps | frame size | max length | for |+|---|---|---|---|---|+| `none` | source | even | – | anything |+| `h3` | 24 | multiple of 32 (centre-crop) | 15 s | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) depth input |+| `wan` | 16 | multiple of 16 | – | Wan 2.1 VACE control video |++We crop rather than pad: a padded black border reads to the generator as a far wall.++## What it gets right (that ad-hoc pipelines get wrong)++1. **One normalisation for the whole clip.** Per-frame min/max makes a static wall+ change brightness whenever someone walks toward the camera; the generator reads that+ as the scene breathing. Depth is scaled once over all frames.+2. **Temporal consistency comes from the model, not a blur.** Video Depth Anything runs+ 32-frame windows with 10-frame overlap and aligns scale/shift across windows. We call+ that path as-is instead of re-chunking outside it.+3. **Near is white**, the convention depth ControlNets were trained on. `--invert` if+ your tool wants the opposite.+4. **fps by timestamp, size by crop, quality by CRF.** See table above.++## Performance (measured end-to-end, Small model, 294 frames @ 736×1280)++| device | precision | speed | note |+|---|---|---|---|+| Apple M2 Max, torch 2.9.1 | fp32 | ~500 ms/frame (12 s clip in 2.7 min) | fp16 on MPS is pathologically slow — we force fp32 there |+| Apple M2 Max, torch 2.6.0 | fp32 | ~1700 ms/frame | 3.5× slower — **upgrade torch ≥ 2.9 on Apple Silicon** |+| CPU (M2 Max) | fp32 | ~1850 ms/frame | works, slow |+| NVIDIA RTX 4090 (AutoDL), torch 2.8 cu128 | fp16 | **70 ms/frame**, 31 ms with `--input-size 364` | host RAM peak 4.5 GB (was 8.2 before the model-resolution change) |+| NVIDIA A100 (upstream figure) | fp16 | ~8 ms/frame | 6.8 GB VRAM at 32-frame batch |++## Models and licences++| weights | licence | commercial |+|---|---|---|+| `Video-Depth-Anything-Small` (default) | Apache-2.0 | ✅ |+| `Video-Depth-Anything-Base` / `-Large` | CC-BY-NC-4.0 | ❌ opt-in only, with warning |++Code is Apache-2.0. The model code from+[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything)+is vendored under `video2blockout/third_party/` (see `NOTICE`).++## ComfyUI++There is already a good node — [ComfyUI-Video-Depth-Anything](https://github.com/yuvraj108c/ComfyUI-Video-Depth-Anything).+This project is for people who want a CLI / Python API, a licence-safe default, and the+target presets.++## Roadmap++- `--people-only`: keep person silhouettes, flatten the background (SAM 2)+- `--also pose,canny,normal`: the other MiniMax H3 Fun ControlNet inputs from one pass+- Depth Anything 3 (per-frame + smoothing) and ViGeo backends+- Gradio demo / Hugging Face Space++## Credits++Video Depth Anything — Chen et al., CVPR 2025. Built at [Maoska](https://www.maosika.com).diff --git a/README.zh-CN.md b/README.zh-CN.mdnew file mode 100644index 0000000..8a2ccb5--- /dev/null+++ b/README.zh-CN.md@@ -0,0 +1,100 @@+# video2blockout++[](https://github.com/maosika-ai/video2blockout/actions/workflows/ci.yml) [](LICENSE)++把任意视频抽成**深度白模视频**——近白远黑的灰度片,喂给视频生成模型的 ControlNet+(MiniMax H3 Fun ControlNet、Wan VACE……),只复制参考片的**走位与镜头**,不带走人脸、+服装和画风。就是 LibTV「深度动作捕捉」那种输出。++<p align="center"><img src="docs/demo.gif" width="560" alt="左:原片;右:深度白模"></p>++一条命令,一个 Apache-2.0 的模型(111 MB),8G 显卡或 Apple M 系列芯片都能跑。++## 安装++```bash+pip install git+https://github.com/maosika-ai/video2blockout+# 需要 PATH 里有 ffmpeg(或 pip install "video2blockout[ffmpeg]")+```++首次运行自动从 Hugging Face 下载权重。国内请设置 `HF_ENDPOINT=https://hf-mirror.com`。++## 用法++```bash+video2blockout in.mp4 -o blockout.mp4 # 保持原 fps / 尺寸,近白远黑+video2blockout in.mp4 -o blockout.mp4 --target h3 # MiniMax H3:24fps、边长 32 的倍数、≤15 秒+video2blockout in.mp4 -o blockout.mp4 --npz depth.npz # 同时保存原始浮点深度+```++```python+from video2blockout import extract, to_gray, write_gray_video+depths, fps = extract("in.mp4") # float32 [T, H, W],越大越近+write_gray_video(to_gray(depths), "out.mp4", fps)+```++| 参数 | 默认 | 作用 |+|---|---|---|+| `--model small\|base\|large` | `small` | 只有 `small` 是 Apache-2.0;`base`/`large` 是 CC-BY-NC,加载时会警告 |+| `--target none\|h3\|wan` | `none` | 按目标模型设 fps 与尺寸(见下表) |+| `--fps N` | 原片 | 按**时间戳**选帧,30→24 真的是 24,不是整数步长那种假 24 |+| `--max-res N` | 1280 | 推理前把长边压到 N;输出也是这个尺寸 |+| `--invert` | 关 | 改成远白近黑 |+| `--clip PCT` | 0 | 两端各裁掉 PCT% 再归一,防一个热点像素把整段压暗 |+| `--gamma G` | 1.0 | >1 压暗中间调(近处层次更开) |+| `--crf N` | 12 | x264 质量。故意给高:控制视频里的色带会变成出片里的抖动 |+| `--npz PATH` | – | 保存原始深度,留给自己后处理 |++### 目标预设++| `--target` | fps | 尺寸 | 时长上限 | 对应 |+|---|---|---|---|---|+| `none` | 原片 | 偶数 | – | 通用 |+| `h3` | 24 | 32 的倍数(居中裁) | 15 秒 | [MiniMax-H3-Fun-Controlnet-Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) 深度输入 |+| `wan` | 16 | 16 的倍数 | – | Wan 2.1 VACE 控制视频 |++尺寸用裁不用补:补上去的黑边在生成模型眼里是一堵远墙。++## 做对的四个细节++1. **整段视频只归一化一次,不逐帧。** 逐帧 min/max 会让静止的墙在有人走近镜头时忽明忽暗,生成模型读到的就是"场景在呼吸"。+2. **时序一致性来自模型,不是靠模糊。** Video Depth Anything 用 32 帧窗口、10 帧重叠、跨窗口对齐尺度;我们原样调用它的推理路径,不在外面重新切段。+3. **近处白。** 深度 ControlNet 训练时用的就是这个约定;要反过来用 `--invert`。+4. **fps 按时间戳、尺寸靠裁剪、质量靠 CRF。** 见上表。++## 实测性能(全片端到端,Small 模型,294 帧 @ 736×1280)++| 设备 | 精度 | 速度 | 说明 |+|---|---|---|---|+| Apple M2 Max,torch 2.9.1 | fp32 | 约 500 ms/帧(12 秒片 2.7 分钟) | MPS 上 fp16 慢到不可用,已强制 fp32 |+| Apple M2 Max,torch 2.6.0 | fp32 | 约 1700 ms/帧 | 慢 3.5 倍——**Apple 芯片请升级 torch ≥ 2.9** |+| CPU(M2 Max) | fp32 | 约 1850 ms/帧 | 能用,慢 |+| NVIDIA RTX 4090(AutoDL),torch 2.8 cu128 | fp16 | **70 ms/帧**,`--input-size 364` 时 31 ms | 宿主内存峰值 4.5 GB(改成模型分辨率推理前是 8.2) |+| NVIDIA A100(官方数据) | fp16 | 约 8 ms/帧 | 32 帧批次占 6.8 GB 显存 |++## 模型与许可++| 权重 | 许可 | 商用 |+|---|---|---|+| `Video-Depth-Anything-Small`(默认) | Apache-2.0 | ✅ |+| `Video-Depth-Anything-Base` / `-Large` | CC-BY-NC-4.0 | ❌ 需显式指定,加载时警告 |++代码 Apache-2.0。模型代码来自+[DepthAnything/Video-Depth-Anything](https://github.com/DepthAnything/Video-Depth-Anything),+vendored 在 `video2blockout/third_party/`(见 `NOTICE`)。++## ComfyUI++已有现成节点 [ComfyUI-Video-Depth-Anything](https://github.com/yuvraj108c/ComfyUI-Video-Depth-Anything)。+本项目面向要 CLI / Python API、要许可证安全默认值、要目标模型预设的人。++## 路线图++- `--people-only`:只留人物剪影、背景抹平(SAM 2)+- `--also pose,canny,normal`:一遍出 MiniMax H3 Fun ControlNet 的其余几种输入+- Depth Anything 3(逐帧 + 平滑)与 ViGeo 后端+- Gradio 演示 / Hugging Face Space++## 致谢++Video Depth Anything — Chen et al., CVPR 2025。由 [猫斯卡](https://www.maosika.com) 出品。diff --git a/docs/demo.gif b/docs/demo.gifnew file mode 100644index 0000000..efb563aBinary files /dev/null and b/docs/demo.gif differdiff --git a/docs/example_city_night.jpg b/docs/example_city_night.jpgnew file mode 100644index 0000000..7f2250fBinary files /dev/null and b/docs/example_city_night.jpg differdiff --git a/docs/example_city_night_3s.jpg b/docs/example_city_night_3s.jpgnew file mode 100644index 0000000..13274d0Binary files /dev/null and b/docs/example_city_night_3s.jpg differdiff --git a/examples/city_night.mp4 b/examples/city_night.mp4new file mode 100644index 0000000..b8a518cBinary files /dev/null and b/examples/city_night.mp4 differdiff --git a/examples/land_16x9.mp4 b/examples/land_16x9.mp4new file mode 100644index 0000000..31b2a55Binary files /dev/null and b/examples/land_16x9.mp4 differdiff --git a/pyproject.toml b/pyproject.tomlnew file mode 100644index 0000000..2eef4df--- /dev/null+++ b/pyproject.toml@@ -0,0 +1,47 @@+[build-system]+requires = ["setuptools>=68", "wheel"]+build-backend = "setuptools.build_meta"++[project]+name = "video2blockout"+version = "0.1.0"+description = "Turn any video into a depth blockout video for video-generation ControlNets (MiniMax H3, Wan VACE, ...)."+readme = "README.md"+license = { text = "Apache-2.0" }+requires-python = ">=3.10"+authors = [{ name = "Maoska" }]+keywords = ["video", "depth", "controlnet", "video-generation", "blockout", "depth-anything"]+classifiers = [+ "License :: OSI Approved :: Apache Software License",+ "Programming Language :: Python :: 3",+ "Topic :: Multimedia :: Video",+]+dependencies = [+ "torch>=2.1",+ "numpy>=1.24",+ "opencv-python-headless>=4.8",+ "einops>=0.4",+ "easydict",+ "tqdm",+ "huggingface_hub>=0.20",+]++[project.optional-dependencies]+ffmpeg = ["imageio-ffmpeg>=0.4.7"] # bundled ffmpeg if none on PATH+dev = ["pytest>=8", "ruff"]++[project.scripts]+video2blockout = "video2blockout.cli:main"++[project.urls]+Homepage = "https://github.com/maosika-ai/video2blockout"++[tool.setuptools.packages.find]+include = ["video2blockout*"]++[tool.setuptools.package-data]+"video2blockout.third_party.video_depth_anything" = ["LICENSE"]++[tool.ruff]+line-length = 110+extend-exclude = ["video2blockout/third_party"]diff --git a/tests/test_io.py b/tests/test_io.pynew file mode 100644index 0000000..a2cab78--- /dev/null+++ b/tests/test_io.py@@ -0,0 +1,80 @@+"""ffmpeg round-trip on synthetic frames — no model."""++import cv2+import numpy as np+import pytest++from video2blockout.io import read_video, write_gray_video++def _has_ffmpeg():+ try:+ from video2blockout.io import ffmpeg_binary++ return bool(ffmpeg_binary())+ except Exception:+ return False+++pytestmark = pytest.mark.skipif(not _has_ffmpeg(), reason="no ffmpeg (PATH or imageio-ffmpeg)")+++def test_gray_video_roundtrip(tmp_path):+ t, h, w = 12, 64, 96+ ramp = np.linspace(0, 255, w, dtype=np.float32)+ gray = np.broadcast_to(ramp, (t, h, w)).astype(np.uint8).copy()+ out = write_gray_video(gray, tmp_path / "g.mp4", fps=24)++ cap = cv2.VideoCapture(str(out))+ assert int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) == t+ ok, frame = cap.read()+ cap.release()+ assert ok and frame.shape[:2] == (h, w)+ # yuv420p + x264 at CRF 12 should keep a smooth ramp within a couple of levels+ err = np.abs(frame[:, :, 0].astype(int) - gray[0].astype(int))+ assert err.mean() < 2.0+++def test_write_rejects_odd_dimensions(tmp_path):+ with pytest.raises(ValueError):+ write_gray_video(np.zeros((1, 63, 64), np.uint8), tmp_path / "x.mp4", 24)+++def test_read_video_resamples_by_timestamp(tmp_path):+ # 30 fps, 30 frames → 1 s; ask for 24 fps and expect 24 frames, not 30 (stride=1 trap)+ p = tmp_path / "src.mp4"+ vw = cv2.VideoWriter(str(p), cv2.VideoWriter_fourcc(*"mp4v"), 30, (64, 48))+ for i in range(30):+ vw.write(np.full((48, 64, 3), i * 8, np.uint8))+ vw.release()+ frames, fps = read_video(p, target_fps=24)+ assert fps == 24 and len(frames) == 24+++def test_read_video_caps_longer_side(tmp_path):+ p = tmp_path / "big.mp4"+ vw = cv2.VideoWriter(str(p), cv2.VideoWriter_fourcc(*"mp4v"), 24, (400, 200))+ for _ in range(3):+ vw.write(np.zeros((200, 400, 3), np.uint8))+ vw.release()+ frames, _ = read_video(p, max_res=100)+ assert frames.shape[1:3] == (50, 100)+++def test_write_accepts_frame_iterator_with_size(tmp_path):+ frames = (np.full((32, 48), i * 7, np.uint8) for i in range(6))+ out = write_gray_video(frames, tmp_path / "it.mp4", 24, size=(32, 48))+ cap = cv2.VideoCapture(str(out))+ assert int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) == 6+ cap.release()+++def test_probe_video(tmp_path):+ p = tmp_path / "p.mp4"+ vw = cv2.VideoWriter(str(p), cv2.VideoWriter_fourcc(*"mp4v"), 25, (64, 48))+ for _ in range(5):+ vw.write(np.zeros((48, 64, 3), np.uint8))+ vw.release()+ from video2blockout.io import probe_video++ w, h, fps, n = probe_video(p)+ assert (w, h, n) == (64, 48, 5) and abs(fps - 25) < 0.01diff --git a/tests/test_model_smoke.py b/tests/test_model_smoke.pynew file mode 100644index 0000000..879ef2f--- /dev/null+++ b/tests/test_model_smoke.py@@ -0,0 +1,49 @@+"""End-to-end with the real Small model on CPU. Opt-in (downloads 111 MB):++ RUN_MODEL_TESTS=1 pytest tests/test_model_smoke.py+"""++import os++import cv2+import numpy as np+import pytest++pytestmark = pytest.mark.skipif(not os.environ.get("RUN_MODEL_TESTS"), reason="set RUN_MODEL_TESTS=1")+++def _synthetic_clip(path, frames=8, w=224, h=160):+ vw = cv2.VideoWriter(str(path), cv2.VideoWriter_fourcc(*"mp4v"), 24, (w, h))+ for i in range(frames):+ img = np.full((h, w, 3), 40, np.uint8)+ # a bright square that drifts right, on a gradient background+ img[:, :, 1] = np.linspace(20, 200, w, dtype=np.uint8)+ x = 20 + i * 8+ cv2.rectangle(img, (x, 50), (x + 60, 120), (230, 230, 230), -1)+ vw.write(img)+ vw.release()+++def test_extract_and_write(tmp_path):+ from video2blockout import extract, to_gray, write_gray_video++ src = tmp_path / "src.mp4"+ _synthetic_clip(src)+ depths, fps = extract(str(src), device="cpu")+ assert depths.shape[0] == 8 and depths.ndim == 3+ assert np.isfinite(depths).all() and depths.max() > depths.min()+ gray = to_gray(depths)+ assert gray.dtype == np.uint8 and gray.max() == 255 and gray.min() == 0+ out = write_gray_video(gray, tmp_path / "out.mp4", fps)+ assert out.stat().st_size > 1000+++def test_cli_end_to_end(tmp_path):+ from video2blockout.cli import main++ src = tmp_path / "src.mp4"+ _synthetic_clip(src)+ out = tmp_path / "out.mp4"+ metrics = tmp_path / "m.json"+ assert main([str(src), "-o", str(out), "--device", "cpu", "--target", "h3", "--metrics", str(metrics)]) == 0+ assert out.exists() and metrics.exists()diff --git a/tests/test_planning.py b/tests/test_planning.pynew file mode 100644index 0000000..a31a678--- /dev/null+++ b/tests/test_planning.py@@ -0,0 +1,36 @@+"""Planning is pure arithmetic — test it without a model."""++from video2blockout.planning import (+ HOST_RAM_BUDGET_RATIO,+ estimate_host_bytes,+ memory_verdict,+ processing_max_res,+)+++def test_processing_res_puts_short_side_at_input_size():+ # 736×1280 portrait: short side 736 → 518 means long side 1280·518/736 = 900.87 → 900 (even)+ assert processing_max_res(736, 1280) == 900+ assert processing_max_res(1280, 736) == 900+++def test_processing_res_never_upscales():+ assert processing_max_res(400, 300) == 400+++def test_estimate_scales_with_pixels_and_frames():+ assert estimate_host_bytes(2, 100, 100) == 2 * estimate_host_bytes(1, 100, 100)+ assert estimate_host_bytes(1, 200, 100) == 2 * estimate_host_bytes(1, 100, 100)+++def test_verdict_refuses_over_budget_and_suggests_frames():+ ram = 8 * 2**30+ v = memory_verdict(10_000, 900, 518, ram_bytes=ram)+ assert not v["ok"]+ assert v["budget"] == int(ram * HOST_RAM_BUDGET_RATIO)+ # The suggested frame count itself must fit the budget+ assert estimate_host_bytes(v["max_frames_ok"], 900, 518) <= v["budget"]+++def test_verdict_passes_small_job():+ assert memory_verdict(8, 320, 180, ram_bytes=8 * 2**30)["ok"]diff --git a/tests/test_postprocess.py b/tests/test_postprocess.pynew file mode 100644index 0000000..911a823--- /dev/null+++ b/tests/test_postprocess.py@@ -0,0 +1,60 @@+"""Unit tests that need no model and no GPU."""++import numpy as np+import pytest++from video2blockout.postprocess import to_gray+from video2blockout.targets import TARGETS, center_crop, fit_dimensions+++def test_global_normalisation_not_per_frame():+ # Frame 0 spans 0..1, frame 1 spans 0..10. Per-frame min/max would map both to+ # full range; global must keep frame 0 dark.+ d = np.zeros((2, 4, 4), np.float32)+ d[0] = np.linspace(0, 1, 16).reshape(4, 4)+ d[1] = np.linspace(0, 10, 16).reshape(4, 4)+ g = to_gray(d)+ assert g[0].max() <= 26 # 1/10 of 255+ assert g[1].max() == 255+++def test_near_is_white_by_default_and_invert_flips():+ d = np.array([[[0.0, 5.0]]], np.float32) # larger = closer (VDA convention)+ assert to_gray(d)[0, 0].tolist() == [0, 255]+ assert to_gray(d, invert=True)[0, 0].tolist() == [255, 0]+++def test_clip_percent_ignores_hot_pixel():+ # A 0..1 ramp plus one hot pixel at 1000. Exact min/max lets the hot pixel crush the+ # ramp into the bottom few grey levels; a 2 % tail clip restores its full contrast.+ d = np.tile(np.linspace(0, 1, 100, dtype=np.float32), (1, 10, 1)).reshape(1, 10, 100)+ d[0, 0, 0] = 1000.0+ assert to_gray(d)[0, 5, 99] < 2+ assert to_gray(d, clip_percent=2)[0, 5, 99] >= 250+++def test_flat_input_does_not_divide_by_zero():+ assert to_gray(np.ones((1, 2, 2), np.float32)).max() == 0+++def test_to_gray_rejects_wrong_rank():+ with pytest.raises(ValueError):+ to_gray(np.zeros((2, 2), np.float32))++[email protected]("h,w,m,exp", [(1280, 736, 32, (1280, 736)), (1080, 1920, 32, (1056, 1920)),+ (7, 9, 2, (6, 8)), (10, 10, 32, (32, 32))])+def test_fit_dimensions_rounds_down_to_multiple(h, w, m, exp):+ assert fit_dimensions(h, w, m) == exp+++def test_center_crop_is_centred():+ f = np.arange(6 * 6).reshape(1, 6, 6)+ c = center_crop(f, 4, 4)+ assert c.shape == (1, 4, 4)+ assert c[0, 0, 0] == 7 and c[0, -1, -1] == 28+++def test_h3_preset_matches_published_constraints():+ t = TARGETS["h3"]+ assert (t.fps, t.multiple, t.max_seconds) == (24.0, 32, 15.0)diff --git a/video2blockout/__init__.py b/video2blockout/__init__.pynew file mode 100644index 0000000..b57148b--- /dev/null+++ b/video2blockout/__init__.py@@ -0,0 +1,65 @@+"""video2blockout — turn any video into a depth "blockout" video for video-generation control.++ from video2blockout import extract, to_gray, write_gray_video+ depths, fps = extract("in.mp4") # float32 [T, H, W], larger = closer+ write_gray_video(to_gray(depths), "out.mp4", fps)+"""++from __future__ import annotations++import os++# Apple Silicon: cap the MPS allocator at 60 % of the recommended working set. Unified+# memory means GPU allocations count against the same RAM as everything else; without+# a cap a long clip grows until the OS starts killing processes (seen 2026-09-11).+# With the cap we get a clean MPS out-of-memory error instead. Must be set before+# torch is imported, which is why it lives here and not in the backend.+os.environ.setdefault("PYTORCH_MPS_HIGH_WATERMARK_RATIO", "0.6")++import numpy as np++from .backends import VDABackend, pick_device+from .io import probe_video, read_video, write_gray_video+from .planning import memory_verdict, processing_max_res+from .postprocess import to_gray, upsample_frames+from .targets import TARGETS, center_crop, fit_dimensions++__version__ = "0.1.0"+++def extract(+ video: str,+ *,+ model: str = "small",+ device: str = "auto",+ max_res: int = 1280,+ target_fps: float = -1,+ max_frames: int = -1,+ input_size: int = 518,+ checkpoint: str | None = None,+) -> tuple[np.ndarray, float]:+ """One call: decode → depth. Returns `(depths[T,H,W] float32, fps)`."""+ # Infer at model resolution regardless of `max_res` — see planning.processing_max_res.+ w, h, _, _ = probe_video(video)+ proc = min(processing_max_res(w, h, input_size), max_res if max_res > 0 else 10**9)+ frames, fps = read_video(video, max_frames=max_frames, target_fps=target_fps, max_res=proc)+ backend = VDABackend(model, device=device, checkpoint=checkpoint)+ return backend.infer(frames, fps, input_size=input_size), fps+++__all__ = [+ "__version__",+ "extract",+ "probe_video",+ "read_video",+ "write_gray_video",+ "to_gray",+ "upsample_frames",+ "processing_max_res",+ "memory_verdict",+ "TARGETS",+ "fit_dimensions",+ "center_crop",+ "VDABackend",+ "pick_device",+]diff --git a/video2blockout/backends/__init__.py b/video2blockout/backends/__init__.pynew file mode 100644index 0000000..f8e2ef8--- /dev/null+++ b/video2blockout/backends/__init__.py@@ -0,0 +1,4 @@+from .base import BackendInfo, DepthBackend+from .vda import VDABackend, pick_device++__all__ = ["BackendInfo", "DepthBackend", "VDABackend", "pick_device"]diff --git a/video2blockout/backends/base.py b/video2blockout/backends/base.pynew file mode 100644index 0000000..f4ba3fb--- /dev/null+++ b/video2blockout/backends/base.py@@ -0,0 +1,30 @@+"""Backend protocol: a depth model that turns a frame stack into per-frame depth.++Every backend returns *relative inverse depth* (larger = closer to camera) as float32+`[T, H, W]` at the input frame resolution, already temporally aligned across the whole+clip. Normalisation to 8-bit grey happens later in :mod:`video2blockout.postprocess`,+so backends never decide what "white" means.+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import Protocol++import numpy as np+++@dataclass(frozen=True)+class BackendInfo:+ name: str+ variant: str+ license: str+ commercial_ok: bool+++class DepthBackend(Protocol):+ info: BackendInfo++ def infer(self, frames: np.ndarray, fps: float, *, input_size: int = 518) -> np.ndarray:+ """`frames`: uint8 RGB `[T, H, W, 3]` → float32 inverse depth `[T, H, W]`."""+ ...diff --git a/video2blockout/backends/vda.py b/video2blockout/backends/vda.pynew file mode 100644index 0000000..8b30bda--- /dev/null+++ b/video2blockout/backends/vda.py@@ -0,0 +1,111 @@+"""Video Depth Anything backend (default).++Why this model: it is the only open video-depth model that ships a temporal module+*and* an Apache-2.0 checkpoint (Small). Per-frame models (Depth Anything V2/3) flicker+on video unless you bolt on your own smoothing; diffusion models (DepthCrafter) are+~100× slower. VDA-Small at 28M params runs on an 8 GB consumer GPU or an Apple M-series.++Precision policy (measured 2026-09-11 on M2 Max, 32 frames @ 736×1280):+ mps fp32 → 7.0 s mps fp16 → did not finish in 3 min (autocast pathological)+ cpu fp32 → 59 s+ Full 294-frame clip end-to-end on mps fp32: 147 s (~500 ms/frame incl. alignment).+So fp16 is enabled on CUDA only. Do not "optimise" this by turning fp16 on for MPS.++Licence policy: `small` is Apache-2.0. `base`/`large` are CC-BY-NC-4.0 — loading them+prints a warning and sets ``info.commercial_ok = False``; the CLI surfaces this.+"""++from __future__ import annotations++import logging+import os+import warnings++import numpy as np+import torch++from .base import BackendInfo++log = logging.getLogger(__name__)++_VARIANTS = {+ "small": {"encoder": "vits", "features": 64, "out_channels": [48, 96, 192, 384]},+ "base": {"encoder": "vitb", "features": 128, "out_channels": [96, 192, 384, 768]},+ "large": {"encoder": "vitl", "features": 256, "out_channels": [256, 512, 1024, 1024]},+}+_LICENSES = {"small": "Apache-2.0", "base": "CC-BY-NC-4.0", "large": "CC-BY-NC-4.0"}+_HF_REPOS = {+ "small": "depth-anything/Video-Depth-Anything-Small",+ "base": "depth-anything/Video-Depth-Anything-Base",+ "large": "depth-anything/Video-Depth-Anything-Large",+}+++def pick_device(requested: str = "auto") -> str:+ if requested != "auto":+ return requested+ if torch.cuda.is_available():+ return "cuda"+ if getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():+ return "mps"+ return "cpu"+++def _weights_path(variant: str, checkpoint: str | None) -> str:+ if checkpoint:+ return checkpoint+ # huggingface_hub honours HF_ENDPOINT, so users in China can point it at a mirror+ # (e.g. https://hf-mirror.com) without us special-casing anything.+ from huggingface_hub import hf_hub_download++ filename = f"video_depth_anything_{_VARIANTS[variant]['encoder']}.pth"+ return hf_hub_download(_HF_REPOS[variant], filename)+++class VDABackend:+ def __init__(+ self,+ variant: str = "small",+ device: str = "auto",+ checkpoint: str | None = None,+ ) -> None:+ if variant not in _VARIANTS:+ raise ValueError(f"unknown variant {variant!r}; choose from {sorted(_VARIANTS)}")+ from ..third_party.video_depth_anything.video_depth import VideoDepthAnything++ self.variant = variant+ self.device = pick_device(device)+ # fp16 only where it is known to be fast; see module docstring for the measurement.+ self.fp32 = self.device != "cuda"+ self.info = BackendInfo(+ name="video-depth-anything",+ variant=variant,+ license=_LICENSES[variant],+ commercial_ok=_LICENSES[variant].startswith("Apache"),+ )+ if not self.info.commercial_ok:+ warnings.warn(+ f"Video Depth Anything '{variant}' weights are {_LICENSES[variant]}: "+ "non-commercial use only. Use 'small' for commercial projects.",+ stacklevel=2,+ )++ path = _weights_path(variant, checkpoint)+ log.info("loading %s from %s on %s (fp32=%s)", variant, path, self.device, self.fp32)+ model = VideoDepthAnything(**_VARIANTS[variant])+ model.load_state_dict(torch.load(path, map_location="cpu"), strict=True)+ self.model = model.to(self.device).eval()++ def infer(self, frames: np.ndarray, fps: float, *, input_size: int = 518) -> np.ndarray:+ if frames.ndim != 4 or frames.shape[-1] != 3:+ raise ValueError(f"expected [T, H, W, 3] uint8 RGB, got {frames.shape}")+ # Upstream's infer_video_depth already does 32-frame windows with 10-frame overlap,+ # keyframe-based scale/shift alignment across windows and interpolation over the+ # seam. That is the temporal-consistency machinery — do not re-chunk outside it.+ depths, _ = self.model.infer_video_depth(+ frames, fps, input_size=input_size, device=self.device, fp32=self.fp32+ )+ return np.asarray(depths, dtype=np.float32)+++__all__ = ["VDABackend", "pick_device"]diff --git a/video2blockout/cli.py b/video2blockout/cli.pynew file mode 100644index 0000000..d0a8e0c--- /dev/null+++ b/video2blockout/cli.py@@ -0,0 +1,174 @@+"""`video2blockout` command line.++ video2blockout in.mp4 -o out.mp4 # Apache-2.0 Small model, auto device+ video2blockout in.mp4 -o out.mp4 --target h3 # 24 fps, ×32 dims, ≤15 s for MiniMax H3+ video2blockout in.mp4 -o out.mp4 --npz depth.npz --metrics run.json++Memory model: depth is estimated at the model's working resolution (short side 518 px)+and only the 8-bit result is upscaled to the output size, frame by frame while+encoding. A host-RAM estimate is printed and, if it exceeds half of physical RAM, the+run is refused with a concrete `--max-frames` suggestion (override with `--force`).+"""++from __future__ import annotations++import argparse+import json+import logging+import os+import platform+import resource+import sys+import time++import numpy as np++from . import __version__+from .backends import VDABackend, pick_device+from .io import probe_video, read_video, write_gray_video+from .planning import gib, memory_verdict, processing_max_res+from .postprocess import to_gray, upsample_frames+from .targets import TARGETS, fit_dimensions+++def build_parser() -> argparse.ArgumentParser:+ p = argparse.ArgumentParser(+ prog="video2blockout",+ description="Video → depth blockout video for video-generation ControlNets.",+ )+ p.add_argument("input", help="input video (anything ffmpeg/OpenCV can read)")+ p.add_argument("-o", "--output", required=True, help="output .mp4")+ p.add_argument("--model", default="small", choices=["small", "base", "large"],+ help="VDA variant; only 'small' is Apache-2.0 (default)")+ p.add_argument("--device", default="auto", help="auto | cuda | mps | cpu")+ p.add_argument("--target", default="none", choices=sorted(TARGETS),+ help="fps / size preset for a specific generator")+ p.add_argument("--fps", type=float, default=-1, help="override output fps (-1 = preset or source)")+ p.add_argument("--max-res", type=int, default=1280,+ help="cap the longer side of the OUTPUT (inference always runs at model resolution)")+ p.add_argument("--max-frames", type=int, default=-1, help="stop after N frames (-1 = all)")+ p.add_argument("--input-size", type=int, default=518, help="model short side (multiple of 14)")+ p.add_argument("--invert", action="store_true", help="far = white instead of near = white")+ p.add_argument("--clip", type=float, default=0.0, metavar="PCT",+ help="percentile clip on both tails before scaling (0 = exact min/max)")+ p.add_argument("--gamma", type=float, default=1.0)+ p.add_argument("--crf", type=int, default=12, help="x264 CRF; lower = larger, cleaner")+ p.add_argument("--npz", help="also save raw float depth (at processing resolution) to this .npz")+ p.add_argument("--metrics", help="write timing / memory / size metrics as JSON to this path")+ p.add_argument("--checkpoint", help="local .pth instead of the Hugging Face download")+ p.add_argument("--force", action="store_true", help="run even if the RAM estimate exceeds the budget")+ p.add_argument("-v", "--verbose", action="store_true")+ p.add_argument("--version", action="version", version=__version__)+ return p+++def _peak_rss_bytes() -> int:+ ru = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss+ # macOS reports bytes, Linux kilobytes.+ return int(ru if platform.system() == "Darwin" else ru * 1024)+++def _say(msg: str) -> None:+ print(msg, file=sys.stderr, flush=True)+++def main(argv: list[str] | None = None) -> int:+ args = build_parser().parse_args(argv)+ logging.basicConfig(level=logging.INFO if args.verbose else logging.WARNING,+ format="%(levelname)s %(name)s: %(message)s")+ target = TARGETS[args.target]+ fps_req = args.fps if args.fps > 0 else (target.fps or -1)+ metrics: dict = {"version": __version__, "input": args.input, "model": args.model, "target": args.target}+ t0 = time.time()++ # ── plan: how big will this run be? ────────────────────────────────────────────+ src_w, src_h, src_fps, src_n = probe_video(args.input)+ out_fps = src_fps if fps_req <= 0 or fps_req >= src_fps else fps_req+ n_frames = src_n if args.max_frames <= 0 else min(src_n, args.max_frames)+ n_frames = int(round(n_frames * out_fps / src_fps)) if out_fps < src_fps else n_frames+ proc_res = min(processing_max_res(src_w, src_h, args.input_size), args.max_res if args.max_res > 0 else 10**9)+ scale = min(1.0, proc_res / max(src_w, src_h))+ proc_w, proc_h = int(round(src_w * scale / 2) * 2), int(round(src_h * scale / 2) * 2)+ verdict = memory_verdict(n_frames, proc_w, proc_h)+ _say(f"plan {src_w}x{src_h} @ {src_fps:.3f} fps × {src_n} frames → process {proc_w}x{proc_h} × ~{n_frames} frames, "+ f"host RAM ≈ {gib(verdict['estimate'])} (budget {gib(verdict['budget'])})")+ metrics.update(source=[src_w, src_h, src_fps, src_n], process=[proc_w, proc_h, n_frames],+ ram_estimate=verdict["estimate"], ram_budget=verdict["budget"])+ if not verdict["ok"] and not args.force:+ _say(f"abort estimated RAM exceeds budget. Try --max-frames {verdict['max_frames_ok']} "+ f"(or split the clip), a smaller --input-size, or --force if you know the machine can take it.")+ return 3++ # ── read ────────────────────────────────────────────────────────────────────────+ frames, fps = read_video(args.input, max_frames=args.max_frames, target_fps=fps_req, max_res=proc_res)+ t, h, w = frames.shape[:3]+ _say(f"read {t} frames @ {fps:.3f} fps, {w}x{h} ({time.time() - t0:.1f}s)")+ if target.max_seconds and t / fps > target.max_seconds + 1e-6:+ _say(f"warn clip is {t / fps:.1f}s; {target.key} accepts ≤ {target.max_seconds:.0f}s — trim it")++ # ── depth ───────────────────────────────────────────────────────────────────────+ device = pick_device(args.device)+ if os.environ.get("VIDEO2BLOCKOUT_FAKE_BACKEND"):+ # Dry-run mode for exercising the whole pipeline (I/O, encoding, metrics, remote+ # runbook) without loading a model. Produces a plausible radial gradient.+ _say("model FAKE backend (VIDEO2BLOCKOUT_FAKE_BACKEND set) — no inference")+ yy, xx = np.mgrid[0:h, 0:w].astype(np.float32)+ base = 1.0 / (1.0 + ((xx - w / 2) ** 2 + (yy - h / 2) ** 2) / (0.15 * w * h))+ depths = np.stack([base * (0.8 + 0.2 * i / max(1, t - 1)) for i in range(t)]).astype(np.float32)+ precision, license_ = "none", "n/a"+ dt = 0.0+ else:+ backend = VDABackend(args.model, device=device, checkpoint=args.checkpoint)+ precision, license_ = ("fp32" if backend.fp32 else "fp16"), backend.info.license+ _say(f"model {backend.info.name}/{backend.info.variant} on {device} ({precision}), licence {license_}")+ t1 = time.time()+ depths = backend.infer(frames, fps, input_size=args.input_size)+ dt = time.time() - t1+ del backend+ _release_accelerator(device)+ _say(f"depth {dt:.1f}s = {dt / t * 1000:.0f} ms/frame")+ del frames+ metrics.update(device=device, precision=precision, license=license_, frames=t, fps=fps,+ depth_seconds=round(dt, 2), ms_per_frame=round(dt / t * 1000, 1))++ if args.npz:+ np.savez_compressed(args.npz, depths=depths, fps=fps)+ _say(f"npz {args.npz}")++ # ── grey → output size → encode (frame by frame) ────────────────────────────────+ gray = to_gray(depths, invert=args.invert, clip_percent=args.clip, gamma=args.gamma)+ del depths+ out_scale = min(1.0, (args.max_res if args.max_res > 0 else 10**9) / max(src_w, src_h))+ out_h, out_w = fit_dimensions(int(src_h * out_scale), int(src_w * out_scale), target.multiple)+ if (out_h, out_w) != (h, w):+ _say(f"size {w}x{h} → {out_w}x{out_h} (multiple of {target.multiple})")+ out = write_gray_video(upsample_frames(gray, out_h, out_w), args.output, fps, crf=args.crf, size=(out_h, out_w))+ total = time.time() - t0+ peak = _peak_rss_bytes()+ _say(f"wrote {out} ({total:.1f}s total, peak RSS {gib(peak)})")+ metrics.update(output=str(out), output_size=[out_w, out_h], total_seconds=round(total, 1),+ peak_rss_bytes=peak, output_bytes=os.path.getsize(out))+ if args.metrics:+ with open(args.metrics, "w", encoding="utf-8") as fh:+ json.dump(metrics, fh, ensure_ascii=False, indent=2)+ _say(f"metrics {args.metrics}")+ if license_ not in ("Apache-2.0", "n/a"):+ _say("note weights are non-commercial; use --model small for commercial work")+ return 0+++def _release_accelerator(device: str) -> None:+ """Give cached GPU/MPS memory back before encoding; harmless on CPU."""+ try:+ import torch++ if device == "cuda":+ torch.cuda.empty_cache()+ elif device == "mps":+ torch.mps.empty_cache()+ except Exception: # pragma: no cover+ pass+++if __name__ == "__main__": # pragma: no cover+ sys.exit(main())diff --git a/video2blockout/io.py b/video2blockout/io.pynew file mode 100644index 0000000..31724d8--- /dev/null+++ b/video2blockout/io.py@@ -0,0 +1,156 @@+"""Video in / video out.++Reading uses OpenCV (no decord dependency — decord has no arm64 macOS wheels).+Writing pipes raw 8-bit grey frames into ffmpeg. We deliberately do not go through+imageio's writer: we need explicit control over pixel format, GOP length and CRF,+because the output is a *control signal*, not a video for humans.+"""++from __future__ import annotations++import shutil+import subprocess+from pathlib import Path+from typing import Iterable++import cv2+import numpy as np+++def probe_video(path: str | Path) -> tuple[int, int, float, int]:+ """Cheap header read → `(width, height, fps, frame_count)`; nothing is decoded."""+ cap = cv2.VideoCapture(str(path))+ if not cap.isOpened():+ raise FileNotFoundError(path)+ try:+ return (+ int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)),+ int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)),+ float(cap.get(cv2.CAP_PROP_FPS) or 30.0),+ int(cap.get(cv2.CAP_PROP_FRAME_COUNT)),+ )+ finally:+ cap.release()+++def read_video(+ path: str | Path,+ *,+ max_frames: int = -1,+ target_fps: float = -1,+ max_res: int = -1,+) -> tuple[np.ndarray, float]:+ """Decode to uint8 RGB `[T, H, W, 3]` and return `(frames, fps)`.++ `target_fps` picks frames by *timestamp*, not by integer stride, so 30 → 24 fps+ actually yields 24 fps instead of silently staying at 30 (which is what a+ round(30/24)=1 stride does).+ `max_res` downsizes the longer side; depth is estimated at that size and the+ output video is written at that size too.+ """+ cap = cv2.VideoCapture(str(path))+ if not cap.isOpened():+ raise FileNotFoundError(path)+ src_fps = cap.get(cv2.CAP_PROP_FPS) or 30.0+ out_fps = src_fps if target_fps <= 0 or target_fps >= src_fps else float(target_fps)++ frames: list[np.ndarray] = []+ src_index = 0+ next_pick_time = 0.0+ while True:+ ok, bgr = cap.read()+ if not ok:+ break+ t = src_index / src_fps+ src_index += 1+ if out_fps < src_fps and t + 1e-9 < next_pick_time:+ continue+ next_pick_time += 1.0 / out_fps+ if max_res > 0 and max(bgr.shape[:2]) > max_res:+ scale = max_res / max(bgr.shape[:2])+ # Even dims keep yuv420p encoders happy later; rounding here avoids a crop.+ new_w = int(round(bgr.shape[1] * scale / 2) * 2)+ new_h = int(round(bgr.shape[0] * scale / 2) * 2)+ bgr = cv2.resize(bgr, (new_w, new_h), interpolation=cv2.INTER_AREA)+ frames.append(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))+ if 0 < max_frames <= len(frames):+ break+ cap.release()+ if not frames:+ raise ValueError(f"no frames decoded from {path}")+ return np.stack(frames), out_fps+++def ffmpeg_binary() -> str:+ exe = shutil.which("ffmpeg")+ if exe:+ return exe+ try:+ import imageio_ffmpeg++ return imageio_ffmpeg.get_ffmpeg_exe()+ except Exception as exc: # pragma: no cover - environment dependent+ raise RuntimeError("ffmpeg not found; install ffmpeg or `pip install imageio-ffmpeg`") from exc+++def write_gray_video(+ gray: "np.ndarray | Iterable[np.ndarray]",+ path: str | Path,+ fps: float,+ *,+ crf: int = 12,+ keyint_seconds: float = 2.0,+ size: tuple[int, int] | None = None,+) -> Path:+ """Encode uint8 `[T, H, W]` as H.264 (yuv420p, neutral chroma).++ CRF 12 is generous on purpose: depth maps are large smooth gradients, exactly the+ content where quantisation shows up as banding, and banding in a control video+ becomes jitter in the generated clip. Measured cost is small (~0.5–1 Mbps at 720p).+ A key frame every `keyint_seconds` lets downstream tools cut the clip anywhere.+ """+ # Accept either a whole [T, H, W] array or a lazy frame iterator (with `size`), so+ # the caller can upsample frame by frame instead of materialising the full-res clip.+ if isinstance(gray, np.ndarray):+ if gray.ndim != 3 or gray.dtype != np.uint8:+ raise ValueError(f"expected uint8 [T, H, W], got {gray.shape} {gray.dtype}")+ h, w = gray.shape[1], gray.shape[2]+ frames: Iterable[np.ndarray] = gray+ else:+ if size is None:+ raise ValueError("size=(h, w) is required when passing a frame iterator")+ h, w = size+ frames = gray+ if h % 2 or w % 2:+ raise ValueError("frame size must be even for yuv420p; use targets.fit_dimensions first")+ path = Path(path)+ path.parent.mkdir(parents=True, exist_ok=True)+ cmd = [+ ffmpeg_binary(), "-y", "-loglevel", "error",+ "-f", "rawvideo", "-pix_fmt", "gray", "-s", f"{w}x{h}", "-r", f"{fps:.6f}", "-i", "-",+ "-an",+ "-c:v", "libx264", "-preset", "slow", "-crf", str(crf),+ "-pix_fmt", "yuv420p", "-profile:v", "high",+ "-g", str(max(1, int(round(fps * keyint_seconds)))),+ "-movflags", "+faststart",+ str(path),+ ]+ proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE)+ assert proc.stdin is not None and proc.stderr is not None+ try:+ # Stream frame by frame so a 300-frame 720p clip doesn't need a 300 MB+ # intermediate bytes object. BrokenPipe means ffmpeg died early; its stderr+ # (read below) carries the real reason, so swallow the pipe error here.+ for frame in frames:+ if frame.shape != (h, w) or frame.dtype != np.uint8:+ raise ValueError(f"frame {frame.shape} {frame.dtype} does not match {(h, w)} uint8")+ proc.stdin.write(np.ascontiguousarray(frame).tobytes())+ except BrokenPipeError:+ pass+ finally:+ proc.stdin.close()+ err = proc.stderr.read()+ proc.wait()+ if proc.returncode != 0:+ raise RuntimeError(f"ffmpeg failed: {err.decode(errors='replace')}")+ return pathdiff --git a/video2blockout/planning.py b/video2blockout/planning.pynew file mode 100644index 0000000..85cdbef--- /dev/null+++ b/video2blockout/planning.py@@ -0,0 +1,76 @@+"""Decide *how big* to run before touching the GPU: processing resolution and a host-RAM+estimate. Pure functions, unit-tested without a model.++Why this module exists: on 2026-09-11 a 12 s 736×1280 clip took the reference pipeline to+20.8 GB of RSS on a 32 GB Mac and the OS killed the desktop. The model itself only ever+sees a 518 px short side; everything above that was full-resolution float32 depth kept+for every frame. So: infer at model resolution, estimate memory up front, refuse loudly+instead of swapping the machine to death.+"""++from __future__ import annotations++import os++#: Host bytes per (frame × pixel at processing resolution). Measured on the reference+#: pipeline (RSS 8.2 GB / (294 frames × 1280 × 736 px) ≈ 30 B). Kept deliberately+#: conservative until re-measured with the model-resolution path; recalibrate from+#: `metrics.jsonl` of a real run, not by guessing.+HOST_BYTES_PER_PIXEL_FRAME = 32.0++#: Refuse when the estimate exceeds this share of physical RAM. Half, not 90 %: the+#: OS, the browser and the GPU driver (unified memory on Apple Silicon) all need room.+HOST_RAM_BUDGET_RATIO = 0.5+++def processing_max_res(width: int, height: int, input_size: int = 518) -> int:+ """Longest side (even) at which the *short* side equals `input_size`.++ Feeding frames larger than this is pure waste: the model resizes the short side to+ `input_size` anyway, and the extra pixels only inflate the float depth we keep.+ """+ short, long = min(width, height), max(width, height)+ if short <= input_size:+ return long+ scaled = long * input_size / short+ return int(round(scaled / 2) * 2)+++def estimate_host_bytes(frames: int, proc_w: int, proc_h: int) -> int:+ return int(frames * proc_w * proc_h * HOST_BYTES_PER_PIXEL_FRAME)+++def physical_ram_bytes() -> int:+ """Total RAM; falls back to 16 GB when the platform won't tell us."""+ try:+ import psutil # optional++ return int(psutil.virtual_memory().total)+ except Exception:+ pass+ try:+ pages = os.sysconf("SC_PHYS_PAGES")+ page = os.sysconf("SC_PAGE_SIZE")+ if pages > 0 and page > 0:+ return int(pages * page)+ except (ValueError, OSError, AttributeError):+ pass+ return 16 * 2**30+++def memory_verdict(frames: int, proc_w: int, proc_h: int, ram_bytes: int | None = None) -> dict:+ """`{"estimate", "budget", "ok", "max_frames_ok"}` — the CLI turns this into text."""+ ram = ram_bytes if ram_bytes is not None else physical_ram_bytes()+ budget = int(ram * HOST_RAM_BUDGET_RATIO)+ est = estimate_host_bytes(frames, proc_w, proc_h)+ per_frame = max(1, estimate_host_bytes(1, proc_w, proc_h))+ return {+ "estimate": est,+ "budget": budget,+ "ok": est <= budget,+ "max_frames_ok": max(1, budget // per_frame),+ }+++def gib(n: int | float) -> str:+ return f"{n / 2**30:.1f} GiB"diff --git a/video2blockout/postprocess.py b/video2blockout/postprocess.pynew file mode 100644index 0000000..347bbdd--- /dev/null+++ b/video2blockout/postprocess.py@@ -0,0 +1,65 @@+"""Float depth → 8-bit grey the way video ControlNets expect it.++Two rules, both learned the hard way:++1. **Normalise over the whole clip, never per frame.** Per-frame min/max makes the+ brightness of a static wall change whenever someone walks closer to the camera,+ and the generator reads that as the scene "breathing".+2. **Near is white.** Video Depth Anything outputs inverse depth (larger = closer), and+ depth ControlNets (MiniMax H3 Fun, Wan VACE, SD ControlNet-depth) were trained on+ that convention. `invert=True` exists for tools that want the opposite.+"""++from __future__ import annotations++import numpy as np+++def to_gray(+ depths: np.ndarray,+ *,+ invert: bool = False,+ clip_percent: float = 0.0,+ gamma: float = 1.0,+) -> np.ndarray:+ """`[T, H, W]` float → `[T, H, W]` uint8.++ `clip_percent` (0–10) trims that many percent from each tail before scaling, so a+ single hot pixel can't crush the contrast of the whole clip. 0 reproduces the+ upstream reference behaviour exactly.+ `gamma` > 1 pushes mid-tones darker (more separation near the camera); 1 is linear.+ """+ d = np.asarray(depths, dtype=np.float32)+ if d.ndim != 3:+ raise ValueError(f"expected [T, H, W], got {d.shape}")+ if not 0 <= clip_percent < 50:+ raise ValueError("clip_percent must be in [0, 50)")+ if clip_percent > 0:+ lo, hi = np.percentile(d, (clip_percent, 100 - clip_percent))+ else:+ lo, hi = float(d.min()), float(d.max())+ if hi - lo < 1e-6:+ return np.zeros(d.shape, dtype=np.uint8)+ n = np.clip((d - lo) / (hi - lo), 0.0, 1.0)+ if gamma != 1.0:+ n = n ** float(gamma)+ if invert:+ n = 1.0 - n+ return (n * 255.0 + 0.5).astype(np.uint8)+++def upsample_frames(gray: np.ndarray, height: int, width: int):+ """Yield each `[H, W]` uint8 frame resized to `(height, width)`.++ Depth is estimated at the model's working resolution (short side 518 px) and only+ the 8-bit result is scaled up — the upstream reference upsamples the *float* depth+ of every frame to full size before alignment, which is what made a 12 s 720p clip+ need 8–20 GB of RAM. Bilinear on 8-bit grey is visually identical for a control video.+ """+ import cv2++ for frame in gray:+ if frame.shape == (height, width):+ yield frame+ else:+ yield cv2.resize(frame, (width, height), interpolation=cv2.INTER_LINEAR)diff --git a/video2blockout/targets.py b/video2blockout/targets.pynew file mode 100644index 0000000..9b34f4c--- /dev/null+++ b/video2blockout/targets.py@@ -0,0 +1,45 @@+"""Output presets for specific video-generation models.++A control video only helps if its frame rate and size match what the generator was+trained to consume; otherwise the generator resamples it and you get drift. Values+here are the published constraints of each model's control/ControlNet path.+"""++from __future__ import annotations++from dataclasses import dataclass++import numpy as np+++@dataclass(frozen=True)+class Target:+ key: str+ fps: float | None # None = keep source fps+ multiple: int # frame size rounded down to a multiple of this+ max_seconds: float | None # warn beyond this+ note: str+++TARGETS: dict[str, Target] = {+ "none": Target("none", None, 2, None, "keep source fps; only make dimensions even"),+ # MiniMax-H3-Fun-Controlnet-Union: 24 fps fixed, H/W multiples of 32, ≤15 s.+ "h3": Target("h3", 24.0, 32, 15.0, "MiniMax H3 Fun ControlNet (depth/pose/canny/hed/mlsd)"),+ # Wan 2.1 VACE control videos: 16 fps, H/W multiples of 16.+ "wan": Target("wan", 16.0, 16, None, "Wan 2.1 VACE control video"),+}+++def fit_dimensions(h: int, w: int, multiple: int) -> tuple[int, int]:+ """Largest (h, w) ≤ input that are multiples of `multiple` (min one block)."""+ return max(multiple, h // multiple * multiple), max(multiple, w // multiple * multiple)+++def center_crop(frames: np.ndarray, h: int, w: int) -> np.ndarray:+ """Crop `[T, H, W, ...]` to `(h, w)` around the centre.++ Crop, not pad: a padded black border reads to the generator as a far wall.+ """+ H, W = frames.shape[1], frames.shape[2]+ top, left = (H - h) // 2, (W - w) // 2+ return frames[:, top : top + h, left : left + w]diff --git a/video2blockout/third_party/__init__.py b/video2blockout/third_party/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/video2blockout/third_party/video_depth_anything/LICENSE b/video2blockout/third_party/video_depth_anything/LICENSEnew file mode 100644index 0000000..f49a4e1--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/LICENSE@@ -0,0 +1,201 @@+ Apache License+ Version 2.0, January 2004+ http://www.apache.org/licenses/++ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION++ 1. Definitions.++ "License" shall mean the terms and conditions for use, reproduction,+ and distribution as defined by Sections 1 through 9 of this document.++ "Licensor" shall mean the copyright owner or entity authorized by+ the copyright owner that is granting the License.++ "Legal Entity" shall mean the union of the acting entity and all+ other entities that control, are controlled by, or are under common+ control with that entity. For the purposes of this definition,+ "control" means (i) the power, direct or indirect, to cause the+ direction or management of such entity, whether by contract or+ otherwise, or (ii) ownership of fifty percent (50%) or more of the+ outstanding shares, or (iii) beneficial ownership of such entity.++ "You" (or "Your") shall mean an individual or Legal Entity+ exercising permissions granted by this License.++ "Source" form shall mean the preferred form for making modifications,+ including but not limited to software source code, documentation+ source, and configuration files.++ "Object" form shall mean any form resulting from mechanical+ transformation or translation of a Source form, including but+ not limited to compiled object code, generated documentation,+ and conversions to other media types.++ "Work" shall mean the work of authorship, whether in Source or+ Object form, made available under the License, as indicated by a+ copyright notice that is included in or attached to the work+ (an example is provided in the Appendix below).++ "Derivative Works" shall mean any work, whether in Source or Object+ form, that is based on (or derived from) the Work and for which the+ editorial revisions, annotations, elaborations, or other modifications+ represent, as a whole, an original work of authorship. For the purposes+ of this License, Derivative Works shall not include works that remain+ separable from, or merely link (or bind by name) to the interfaces of,+ the Work and Derivative Works thereof.++ "Contribution" shall mean any work of authorship, including+ the original version of the Work and any modifications or additions+ to that Work or Derivative Works thereof, that is intentionally+ submitted to Licensor for inclusion in the Work by the copyright owner+ or by an individual or Legal Entity authorized to submit on behalf of+ the copyright owner. For the purposes of this definition, "submitted"+ means any form of electronic, verbal, or written communication sent+ to the Licensor or its representatives, including but not limited to+ communication on electronic mailing lists, source code control systems,+ and issue tracking systems that are managed by, or on behalf of, the+ Licensor for the purpose of discussing and improving the Work, but+ excluding communication that is conspicuously marked or otherwise+ designated in writing by the copyright owner as "Not a Contribution."++ "Contributor" shall mean Licensor and any individual or Legal Entity+ on behalf of whom a Contribution has been received by Licensor and+ subsequently incorporated within the Work.++ 2. Grant of Copyright License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ copyright license to reproduce, prepare Derivative Works of,+ publicly display, publicly perform, sublicense, and distribute the+ Work and such Derivative Works in Source or Object form.++ 3. Grant of Patent License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ (except as stated in this section) patent license to make, have made,+ use, offer to sell, sell, import, and otherwise transfer the Work,+ where such license applies only to those patent claims licensable+ by such Contributor that are necessarily infringed by their+ Contribution(s) alone or by combination of their Contribution(s)+ with the Work to which such Contribution(s) was submitted. If You+ institute patent litigation against any entity (including a+ cross-claim or counterclaim in a lawsuit) alleging that the Work+ or a Contribution incorporated within the Work constitutes direct+ or contributory patent infringement, then any patent licenses+ granted to You under this License for that Work shall terminate+ as of the date such litigation is filed.++ 4. Redistribution. You may reproduce and distribute copies of the+ Work or Derivative Works thereof in any medium, with or without+ modifications, and in Source or Object form, provided that You+ meet the following conditions:++ (a) You must give any other recipients of the Work or+ Derivative Works a copy of this License; and++ (b) You must cause any modified files to carry prominent notices+ stating that You changed the files; and++ (c) You must retain, in the Source form of any Derivative Works+ that You distribute, all copyright, patent, trademark, and+ attribution notices from the Source form of the Work,+ excluding those notices that do not pertain to any part of+ the Derivative Works; and++ (d) If the Work includes a "NOTICE" text file as part of its+ distribution, then any Derivative Works that You distribute must+ include a readable copy of the attribution notices contained+ within such NOTICE file, excluding those notices that do not+ pertain to any part of the Derivative Works, in at least one+ of the following places: within a NOTICE text file distributed+ as part of the Derivative Works; within the Source form or+ documentation, if provided along with the Derivative Works; or,+ within a display generated by the Derivative Works, if and+ wherever such third-party notices normally appear. The contents+ of the NOTICE file are for informational purposes only and+ do not modify the License. You may add Your own attribution+ notices within Derivative Works that You distribute, alongside+ or as an addendum to the NOTICE text from the Work, provided+ that such additional attribution notices cannot be construed+ as modifying the License.++ You may add Your own copyright statement to Your modifications and+ may provide additional or different license terms and conditions+ for use, reproduction, or distribution of Your modifications, or+ for any such Derivative Works as a whole, provided Your use,+ reproduction, and distribution of the Work otherwise complies with+ the conditions stated in this License.++ 5. Submission of Contributions. Unless You explicitly state otherwise,+ any Contribution intentionally submitted for inclusion in the Work+ by You to the Licensor shall be under the terms and conditions of+ this License, without any additional terms or conditions.+ Notwithstanding the above, nothing herein shall supersede or modify+ the terms of any separate license agreement you may have executed+ with Licensor regarding such Contributions.++ 6. Trademarks. This License does not grant permission to use the trade+ names, trademarks, service marks, or product names of the Licensor,+ except as required for reasonable and customary use in describing the+ origin of the Work and reproducing the content of the NOTICE file.++ 7. Disclaimer of Warranty. Unless required by applicable law or+ agreed to in writing, Licensor provides the Work (and each+ Contributor provides its Contributions) on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or+ implied, including, without limitation, any warranties or conditions+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A+ PARTICULAR PURPOSE. You are solely responsible for determining the+ appropriateness of using or redistributing the Work and assume any+ risks associated with Your exercise of permissions under this License.++ 8. Limitation of Liability. In no event and under no legal theory,+ whether in tort (including negligence), contract, or otherwise,+ unless required by applicable law (such as deliberate and grossly+ negligent acts) or agreed to in writing, shall any Contributor be+ liable to You for damages, including any direct, indirect, special,+ incidental, or consequential damages of any character arising as a+ result of this License or out of the use or inability to use the+ Work (including but not limited to damages for loss of goodwill,+ work stoppage, computer failure or malfunction, or any and all+ other commercial damages or losses), even if such Contributor+ has been advised of the possibility of such damages.++ 9. Accepting Warranty or Additional Liability. While redistributing+ the Work or Derivative Works thereof, You may choose to offer,+ and charge a fee for, acceptance of support, warranty, indemnity,+ or other liability obligations and/or rights consistent with this+ License. However, in accepting such obligations, You may act only+ on Your own behalf and on Your sole responsibility, not on behalf+ of any other Contributor, and only if You agree to indemnify,+ defend, and hold each Contributor harmless for any liability+ incurred by, or claims asserted against, such Contributor by reason+ of your accepting any such warranty or additional liability.++ END OF TERMS AND CONDITIONS++ APPENDIX: How to apply the Apache License to your work.++ To apply the Apache License to your work, attach the following+ boilerplate notice, with the fields enclosed by brackets "[]"+ replaced with your own identifying information. (Don't include+ the brackets!) The text should be enclosed in the appropriate+ comment syntax for the file format. We also recommend that a+ file or class name and description of purpose be included on the+ same "printed page" as the copyright notice for easier+ identification within third-party archives.++ Copyright [yyyy] [name of copyright owner]++ Licensed under the Apache License, Version 2.0 (the "License");+ you may not use this file except in compliance with the License.+ You may obtain a copy of the License at++ http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing, software+ distributed under the License is distributed on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ See the License for the specific language governing permissions and+ limitations under the License.\ No newline at end of filediff --git a/video2blockout/third_party/video_depth_anything/__init__.py b/video2blockout/third_party/video_depth_anything/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/video2blockout/third_party/video_depth_anything/dinov2.py b/video2blockout/third_party/video_depth_anything/dinov2.pynew file mode 100644index 0000000..83d2508--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dinov2.py@@ -0,0 +1,415 @@+# Copyright (c) Meta Platforms, Inc. and affiliates.+#+# This source code is licensed under the Apache License, Version 2.0+# found in the LICENSE file in the root directory of this source tree.++# References:+# https://github.com/facebookresearch/dino/blob/main/vision_transformer.py+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/models/vision_transformer.py++from functools import partial+import math+import logging+from typing import Sequence, Tuple, Union, Callable++import torch+import torch.nn as nn+import torch.utils.checkpoint+from torch.nn.init import trunc_normal_++from .dinov2_layers import Mlp, PatchEmbed, SwiGLUFFNFused, MemEffAttention, NestedTensorBlock as Block+++logger = logging.getLogger("dinov2")+++def named_apply(fn: Callable, module: nn.Module, name="", depth_first=True, include_root=False) -> nn.Module:+ if not depth_first and include_root:+ fn(module=module, name=name)+ for child_name, child_module in module.named_children():+ child_name = ".".join((name, child_name)) if name else child_name+ named_apply(fn=fn, module=child_module, name=child_name, depth_first=depth_first, include_root=True)+ if depth_first and include_root:+ fn(module=module, name=name)+ return module+++class BlockChunk(nn.ModuleList):+ def forward(self, x):+ for b in self:+ x = b(x)+ return x+++class DinoVisionTransformer(nn.Module):+ def __init__(+ self,+ img_size=224,+ patch_size=16,+ in_chans=3,+ embed_dim=768,+ depth=12,+ num_heads=12,+ mlp_ratio=4.0,+ qkv_bias=True,+ ffn_bias=True,+ proj_bias=True,+ drop_path_rate=0.0,+ drop_path_uniform=False,+ init_values=None, # for layerscale: None or 0 => no layerscale+ embed_layer=PatchEmbed,+ act_layer=nn.GELU,+ block_fn=Block,+ ffn_layer="mlp",+ block_chunks=1,+ num_register_tokens=0,+ interpolate_antialias=False,+ interpolate_offset=0.1,+ ):+ """+ Args:+ img_size (int, tuple): input image size+ patch_size (int, tuple): patch size+ in_chans (int): number of input channels+ embed_dim (int): embedding dimension+ depth (int): depth of transformer+ num_heads (int): number of attention heads+ mlp_ratio (int): ratio of mlp hidden dim to embedding dim+ qkv_bias (bool): enable bias for qkv if True+ proj_bias (bool): enable bias for proj in attn if True+ ffn_bias (bool): enable bias for ffn if True+ drop_path_rate (float): stochastic depth rate+ drop_path_uniform (bool): apply uniform drop rate across blocks+ weight_init (str): weight init scheme+ init_values (float): layer-scale init values+ embed_layer (nn.Module): patch embedding layer+ act_layer (nn.Module): MLP activation layer+ block_fn (nn.Module): transformer block class+ ffn_layer (str): "mlp", "swiglu", "swiglufused" or "identity"+ block_chunks: (int) split block sequence into block_chunks units for FSDP wrap+ num_register_tokens: (int) number of extra cls tokens (so-called "registers")+ interpolate_antialias: (str) flag to apply anti-aliasing when interpolating positional embeddings+ interpolate_offset: (float) work-around offset to apply when interpolating positional embeddings+ """+ super().__init__()+ norm_layer = partial(nn.LayerNorm, eps=1e-6)++ self.num_features = self.embed_dim = embed_dim # num_features for consistency with other models+ self.num_tokens = 1+ self.n_blocks = depth+ self.num_heads = num_heads+ self.patch_size = patch_size+ self.num_register_tokens = num_register_tokens+ self.interpolate_antialias = interpolate_antialias+ self.interpolate_offset = interpolate_offset++ self.patch_embed = embed_layer(img_size=img_size, patch_size=patch_size, in_chans=in_chans, embed_dim=embed_dim)+ num_patches = self.patch_embed.num_patches++ self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))+ self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + self.num_tokens, embed_dim))+ assert num_register_tokens >= 0+ self.register_tokens = (+ nn.Parameter(torch.zeros(1, num_register_tokens, embed_dim)) if num_register_tokens else None+ )++ if drop_path_uniform is True:+ dpr = [drop_path_rate] * depth+ else:+ dpr = [x.item() for x in torch.linspace(0, drop_path_rate, depth)] # stochastic depth decay rule++ if ffn_layer == "mlp":+ logger.info("using MLP layer as FFN")+ ffn_layer = Mlp+ elif ffn_layer == "swiglufused" or ffn_layer == "swiglu":+ logger.info("using SwiGLU layer as FFN")+ ffn_layer = SwiGLUFFNFused+ elif ffn_layer == "identity":+ logger.info("using Identity layer as FFN")++ def f(*args, **kwargs):+ return nn.Identity()++ ffn_layer = f+ else:+ raise NotImplementedError++ blocks_list = [+ block_fn(+ dim=embed_dim,+ num_heads=num_heads,+ mlp_ratio=mlp_ratio,+ qkv_bias=qkv_bias,+ proj_bias=proj_bias,+ ffn_bias=ffn_bias,+ drop_path=dpr[i],+ norm_layer=norm_layer,+ act_layer=act_layer,+ ffn_layer=ffn_layer,+ init_values=init_values,+ )+ for i in range(depth)+ ]+ if block_chunks > 0:+ self.chunked_blocks = True+ chunked_blocks = []+ chunksize = depth // block_chunks+ for i in range(0, depth, chunksize):+ # this is to keep the block index consistent if we chunk the block list+ chunked_blocks.append([nn.Identity()] * i + blocks_list[i : i + chunksize])+ self.blocks = nn.ModuleList([BlockChunk(p) for p in chunked_blocks])+ else:+ self.chunked_blocks = False+ self.blocks = nn.ModuleList(blocks_list)++ self.norm = norm_layer(embed_dim)+ self.head = nn.Identity()++ self.mask_token = nn.Parameter(torch.zeros(1, embed_dim))++ self.init_weights()++ def init_weights(self):+ trunc_normal_(self.pos_embed, std=0.02)+ nn.init.normal_(self.cls_token, std=1e-6)+ if self.register_tokens is not None:+ nn.init.normal_(self.register_tokens, std=1e-6)+ named_apply(init_weights_vit_timm, self)++ def interpolate_pos_encoding(self, x, w, h):+ previous_dtype = x.dtype+ npatch = x.shape[1] - 1+ N = self.pos_embed.shape[1] - 1+ if npatch == N and w == h:+ return self.pos_embed+ pos_embed = self.pos_embed.float()+ class_pos_embed = pos_embed[:, 0]+ patch_pos_embed = pos_embed[:, 1:]+ dim = x.shape[-1]+ w0 = w // self.patch_size+ h0 = h // self.patch_size+ # we add a small number to avoid floating point error in the interpolation+ # see discussion at https://github.com/facebookresearch/dino/issues/8+ # DINOv2 with register modify the interpolate_offset from 0.1 to 0.0+ w0, h0 = w0 + self.interpolate_offset, h0 + self.interpolate_offset+ # w0, h0 = w0 + 0.1, h0 + 0.1++ sqrt_N = math.sqrt(N)+ sx, sy = float(w0) / sqrt_N, float(h0) / sqrt_N+ patch_pos_embed = nn.functional.interpolate(+ patch_pos_embed.reshape(1, int(sqrt_N), int(sqrt_N), dim).permute(0, 3, 1, 2),+ scale_factor=(sx, sy),+ # (int(w0), int(h0)), # to solve the upsampling shape issue+ mode="bicubic",+ antialias=self.interpolate_antialias+ )++ assert int(w0) == patch_pos_embed.shape[-2]+ assert int(h0) == patch_pos_embed.shape[-1]+ patch_pos_embed = patch_pos_embed.permute(0, 2, 3, 1).view(1, -1, dim)+ return torch.cat((class_pos_embed.unsqueeze(0), patch_pos_embed), dim=1).to(previous_dtype)++ def prepare_tokens_with_masks(self, x, masks=None):+ B, nc, w, h = x.shape+ x = self.patch_embed(x)+ if masks is not None:+ x = torch.where(masks.unsqueeze(-1), self.mask_token.to(x.dtype).unsqueeze(0), x)++ x = torch.cat((self.cls_token.expand(x.shape[0], -1, -1), x), dim=1)+ x = x + self.interpolate_pos_encoding(x, w, h)++ if self.register_tokens is not None:+ x = torch.cat(+ (+ x[:, :1],+ self.register_tokens.expand(x.shape[0], -1, -1),+ x[:, 1:],+ ),+ dim=1,+ )++ return x++ def forward_features_list(self, x_list, masks_list):+ x = [self.prepare_tokens_with_masks(x, masks) for x, masks in zip(x_list, masks_list)]+ for blk in self.blocks:+ x = blk(x)++ all_x = x+ output = []+ for x, masks in zip(all_x, masks_list):+ x_norm = self.norm(x)+ output.append(+ {+ "x_norm_clstoken": x_norm[:, 0],+ "x_norm_regtokens": x_norm[:, 1 : self.num_register_tokens + 1],+ "x_norm_patchtokens": x_norm[:, self.num_register_tokens + 1 :],+ "x_prenorm": x,+ "masks": masks,+ }+ )+ return output++ def forward_features(self, x, masks=None):+ if isinstance(x, list):+ return self.forward_features_list(x, masks)++ x = self.prepare_tokens_with_masks(x, masks)++ for blk in self.blocks:+ x = blk(x)++ x_norm = self.norm(x)+ return {+ "x_norm_clstoken": x_norm[:, 0],+ "x_norm_regtokens": x_norm[:, 1 : self.num_register_tokens + 1],+ "x_norm_patchtokens": x_norm[:, self.num_register_tokens + 1 :],+ "x_prenorm": x,+ "masks": masks,+ }++ def _get_intermediate_layers_not_chunked(self, x, n=1):+ x = self.prepare_tokens_with_masks(x)+ # If n is an int, take the n last blocks. If it's a list, take them+ output, total_block_len = [], len(self.blocks)+ blocks_to_take = range(total_block_len - n, total_block_len) if isinstance(n, int) else n+ for i, blk in enumerate(self.blocks):+ x = blk(x)+ if i in blocks_to_take:+ output.append(x)+ assert len(output) == len(blocks_to_take), f"only {len(output)} / {len(blocks_to_take)} blocks found"+ return output++ def _get_intermediate_layers_chunked(self, x, n=1):+ x = self.prepare_tokens_with_masks(x)+ output, i, total_block_len = [], 0, len(self.blocks[-1])+ # If n is an int, take the n last blocks. If it's a list, take them+ blocks_to_take = range(total_block_len - n, total_block_len) if isinstance(n, int) else n+ for block_chunk in self.blocks:+ for blk in block_chunk[i:]: # Passing the nn.Identity()+ x = blk(x)+ if i in blocks_to_take:+ output.append(x)+ i += 1+ assert len(output) == len(blocks_to_take), f"only {len(output)} / {len(blocks_to_take)} blocks found"+ return output++ def get_intermediate_layers(+ self,+ x: torch.Tensor,+ n: Union[int, Sequence] = 1, # Layers or n last layers to take+ reshape: bool = False,+ return_class_token: bool = False,+ norm=True+ ) -> Tuple[Union[torch.Tensor, Tuple[torch.Tensor]]]:+ if self.chunked_blocks:+ outputs = self._get_intermediate_layers_chunked(x, n)+ else:+ outputs = self._get_intermediate_layers_not_chunked(x, n)+ if norm:+ outputs = [self.norm(out) for out in outputs]+ class_tokens = [out[:, 0] for out in outputs]+ outputs = [out[:, 1 + self.num_register_tokens:] for out in outputs]+ if reshape:+ B, _, w, h = x.shape+ outputs = [+ out.reshape(B, w // self.patch_size, h // self.patch_size, -1).permute(0, 3, 1, 2).contiguous()+ for out in outputs+ ]+ if return_class_token:+ return tuple(zip(outputs, class_tokens))+ return tuple(outputs)++ def forward(self, *args, is_training=False, **kwargs):+ ret = self.forward_features(*args, **kwargs)+ if is_training:+ return ret+ else:+ return self.head(ret["x_norm_clstoken"])+++def init_weights_vit_timm(module: nn.Module, name: str = ""):+ """ViT weight initialization, original timm impl (for reproducibility)"""+ if isinstance(module, nn.Linear):+ trunc_normal_(module.weight, std=0.02)+ if module.bias is not None:+ nn.init.zeros_(module.bias)+++def vit_small(patch_size=16, num_register_tokens=0, **kwargs):+ model = DinoVisionTransformer(+ patch_size=patch_size,+ embed_dim=384,+ depth=12,+ num_heads=6,+ mlp_ratio=4,+ block_fn=partial(Block, attn_class=MemEffAttention),+ num_register_tokens=num_register_tokens,+ **kwargs,+ )+ return model+++def vit_base(patch_size=16, num_register_tokens=0, **kwargs):+ model = DinoVisionTransformer(+ patch_size=patch_size,+ embed_dim=768,+ depth=12,+ num_heads=12,+ mlp_ratio=4,+ block_fn=partial(Block, attn_class=MemEffAttention),+ num_register_tokens=num_register_tokens,+ **kwargs,+ )+ return model+++def vit_large(patch_size=16, num_register_tokens=0, **kwargs):+ model = DinoVisionTransformer(+ patch_size=patch_size,+ embed_dim=1024,+ depth=24,+ num_heads=16,+ mlp_ratio=4,+ block_fn=partial(Block, attn_class=MemEffAttention),+ num_register_tokens=num_register_tokens,+ **kwargs,+ )+ return model+++def vit_giant2(patch_size=16, num_register_tokens=0, **kwargs):+ """+ Close to ViT-giant, with embed-dim 1536 and 24 heads => embed-dim per head 64+ """+ model = DinoVisionTransformer(+ patch_size=patch_size,+ embed_dim=1536,+ depth=40,+ num_heads=24,+ mlp_ratio=4,+ block_fn=partial(Block, attn_class=MemEffAttention),+ num_register_tokens=num_register_tokens,+ **kwargs,+ )+ return model+++def DINOv2(model_name):+ model_zoo = {+ "vits": vit_small,+ "vitb": vit_base,+ "vitl": vit_large,+ "vitg": vit_giant2+ }++ return model_zoo[model_name](+ img_size=518,+ patch_size=14,+ init_values=1.0,+ ffn_layer="mlp" if model_name != "vitg" else "swiglufused",+ block_chunks=0,+ num_register_tokens=0,+ interpolate_antialias=False,+ interpolate_offset=0.1+ )diff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/__init__.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/__init__.pynew file mode 100644index 0000000..8120f4b--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/__init__.py@@ -0,0 +1,11 @@+# Copyright (c) Meta Platforms, Inc. and affiliates.+# All rights reserved.+#+# This source code is licensed under the license found in the+# LICENSE file in the root directory of this source tree.++from .mlp import Mlp+from .patch_embed import PatchEmbed+from .swiglu_ffn import SwiGLUFFN, SwiGLUFFNFused+from .block import NestedTensorBlock+from .attention import MemEffAttentiondiff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/attention.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/attention.pynew file mode 100644index 0000000..815a2bf--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/attention.py@@ -0,0 +1,83 @@+# Copyright (c) Meta Platforms, Inc. and affiliates.+# All rights reserved.+#+# This source code is licensed under the license found in the+# LICENSE file in the root directory of this source tree.++# References:+# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/models/vision_transformer.py++import logging++from torch import Tensor+from torch import nn+++logger = logging.getLogger("dinov2")+++try:+ from xformers.ops import memory_efficient_attention, unbind, fmha++ XFORMERS_AVAILABLE = True+except ImportError:+ logger.warning("xFormers not available")+ XFORMERS_AVAILABLE = False+++class Attention(nn.Module):+ def __init__(+ self,+ dim: int,+ num_heads: int = 8,+ qkv_bias: bool = False,+ proj_bias: bool = True,+ attn_drop: float = 0.0,+ proj_drop: float = 0.0,+ ) -> None:+ super().__init__()+ self.num_heads = num_heads+ head_dim = dim // num_heads+ self.scale = head_dim**-0.5++ self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)+ self.attn_drop = nn.Dropout(attn_drop)+ self.proj = nn.Linear(dim, dim, bias=proj_bias)+ self.proj_drop = nn.Dropout(proj_drop)++ def forward(self, x: Tensor) -> Tensor:+ B, N, C = x.shape+ qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)++ q, k, v = qkv[0] * self.scale, qkv[1], qkv[2]+ attn = q @ k.transpose(-2, -1)++ attn = attn.softmax(dim=-1)+ attn = self.attn_drop(attn)++ x = (attn @ v).transpose(1, 2).reshape(B, N, C)+ x = self.proj(x)+ x = self.proj_drop(x)+ return x+++class MemEffAttention(Attention):+ def forward(self, x: Tensor, attn_bias=None) -> Tensor:+ if not XFORMERS_AVAILABLE:+ assert attn_bias is None, "xFormers is required for nested tensors usage"+ return super().forward(x)++ B, N, C = x.shape+ qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads)++ q, k, v = unbind(qkv, 2)++ x = memory_efficient_attention(q, k, v, attn_bias=attn_bias)+ x = x.reshape([B, N, C])++ x = self.proj(x)+ x = self.proj_drop(x)+ return x++\ No newline at end of filediff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/block.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/block.pynew file mode 100644index 0000000..25488f5--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/block.py@@ -0,0 +1,252 @@+# Copyright (c) Meta Platforms, Inc. and affiliates.+# All rights reserved.+#+# This source code is licensed under the license found in the+# LICENSE file in the root directory of this source tree.++# References:+# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/patch_embed.py++import logging+from typing import Callable, List, Any, Tuple, Dict++import torch+from torch import nn, Tensor++from .attention import Attention, MemEffAttention+from .drop_path import DropPath+from .layer_scale import LayerScale+from .mlp import Mlp+++logger = logging.getLogger("dinov2")+++try:+ from xformers.ops import fmha+ from xformers.ops import scaled_index_add, index_select_cat++ XFORMERS_AVAILABLE = True+except ImportError:+ logger.warning("xFormers not available")+ XFORMERS_AVAILABLE = False+++class Block(nn.Module):+ def __init__(+ self,+ dim: int,+ num_heads: int,+ mlp_ratio: float = 4.0,+ qkv_bias: bool = False,+ proj_bias: bool = True,+ ffn_bias: bool = True,+ drop: float = 0.0,+ attn_drop: float = 0.0,+ init_values=None,+ drop_path: float = 0.0,+ act_layer: Callable[..., nn.Module] = nn.GELU,+ norm_layer: Callable[..., nn.Module] = nn.LayerNorm,+ attn_class: Callable[..., nn.Module] = Attention,+ ffn_layer: Callable[..., nn.Module] = Mlp,+ ) -> None:+ super().__init__()+ # print(f"biases: qkv: {qkv_bias}, proj: {proj_bias}, ffn: {ffn_bias}")+ self.norm1 = norm_layer(dim)+ self.attn = attn_class(+ dim,+ num_heads=num_heads,+ qkv_bias=qkv_bias,+ proj_bias=proj_bias,+ attn_drop=attn_drop,+ proj_drop=drop,+ )+ self.ls1 = LayerScale(dim, init_values=init_values) if init_values else nn.Identity()+ self.drop_path1 = DropPath(drop_path) if drop_path > 0.0 else nn.Identity()++ self.norm2 = norm_layer(dim)+ mlp_hidden_dim = int(dim * mlp_ratio)+ self.mlp = ffn_layer(+ in_features=dim,+ hidden_features=mlp_hidden_dim,+ act_layer=act_layer,+ drop=drop,+ bias=ffn_bias,+ )+ self.ls2 = LayerScale(dim, init_values=init_values) if init_values else nn.Identity()+ self.drop_path2 = DropPath(drop_path) if drop_path > 0.0 else nn.Identity()++ self.sample_drop_ratio = drop_path++ def forward(self, x: Tensor) -> Tensor:+ def attn_residual_func(x: Tensor) -> Tensor:+ return self.ls1(self.attn(self.norm1(x)))++ def ffn_residual_func(x: Tensor) -> Tensor:+ return self.ls2(self.mlp(self.norm2(x)))++ if self.training and self.sample_drop_ratio > 0.1:+ # the overhead is compensated only for a drop path rate larger than 0.1+ x = drop_add_residual_stochastic_depth(+ x,+ residual_func=attn_residual_func,+ sample_drop_ratio=self.sample_drop_ratio,+ )+ x = drop_add_residual_stochastic_depth(+ x,+ residual_func=ffn_residual_func,+ sample_drop_ratio=self.sample_drop_ratio,+ )+ elif self.training and self.sample_drop_ratio > 0.0:+ x = x + self.drop_path1(attn_residual_func(x))+ x = x + self.drop_path1(ffn_residual_func(x)) # FIXME: drop_path2+ else:+ x = x + attn_residual_func(x)+ x = x + ffn_residual_func(x)+ return x+++def drop_add_residual_stochastic_depth(+ x: Tensor,+ residual_func: Callable[[Tensor], Tensor],+ sample_drop_ratio: float = 0.0,+) -> Tensor:+ # 1) extract subset using permutation+ b, n, d = x.shape+ sample_subset_size = max(int(b * (1 - sample_drop_ratio)), 1)+ brange = (torch.randperm(b, device=x.device))[:sample_subset_size]+ x_subset = x[brange]++ # 2) apply residual_func to get residual+ residual = residual_func(x_subset)++ x_flat = x.flatten(1)+ residual = residual.flatten(1)++ residual_scale_factor = b / sample_subset_size++ # 3) add the residual+ x_plus_residual = torch.index_add(x_flat, 0, brange, residual.to(dtype=x.dtype), alpha=residual_scale_factor)+ return x_plus_residual.view_as(x)+++def get_branges_scales(x, sample_drop_ratio=0.0):+ b, n, d = x.shape+ sample_subset_size = max(int(b * (1 - sample_drop_ratio)), 1)+ brange = (torch.randperm(b, device=x.device))[:sample_subset_size]+ residual_scale_factor = b / sample_subset_size+ return brange, residual_scale_factor+++def add_residual(x, brange, residual, residual_scale_factor, scaling_vector=None):+ if scaling_vector is None:+ x_flat = x.flatten(1)+ residual = residual.flatten(1)+ x_plus_residual = torch.index_add(x_flat, 0, brange, residual.to(dtype=x.dtype), alpha=residual_scale_factor)+ else:+ x_plus_residual = scaled_index_add(+ x, brange, residual.to(dtype=x.dtype), scaling=scaling_vector, alpha=residual_scale_factor+ )+ return x_plus_residual+++attn_bias_cache: Dict[Tuple, Any] = {}+++def get_attn_bias_and_cat(x_list, branges=None):+ """+ this will perform the index select, cat the tensors, and provide the attn_bias from cache+ """+ batch_sizes = [b.shape[0] for b in branges] if branges is not None else [x.shape[0] for x in x_list]+ all_shapes = tuple((b, x.shape[1]) for b, x in zip(batch_sizes, x_list))+ if all_shapes not in attn_bias_cache.keys():+ seqlens = []+ for b, x in zip(batch_sizes, x_list):+ for _ in range(b):+ seqlens.append(x.shape[1])+ attn_bias = fmha.BlockDiagonalMask.from_seqlens(seqlens)+ attn_bias._batch_sizes = batch_sizes+ attn_bias_cache[all_shapes] = attn_bias++ if branges is not None:+ cat_tensors = index_select_cat([x.flatten(1) for x in x_list], branges).view(1, -1, x_list[0].shape[-1])+ else:+ tensors_bs1 = tuple(x.reshape([1, -1, *x.shape[2:]]) for x in x_list)+ cat_tensors = torch.cat(tensors_bs1, dim=1)++ return attn_bias_cache[all_shapes], cat_tensors+++def drop_add_residual_stochastic_depth_list(+ x_list: List[Tensor],+ residual_func: Callable[[Tensor, Any], Tensor],+ sample_drop_ratio: float = 0.0,+ scaling_vector=None,+) -> Tensor:+ # 1) generate random set of indices for dropping samples in the batch+ branges_scales = [get_branges_scales(x, sample_drop_ratio=sample_drop_ratio) for x in x_list]+ branges = [s[0] for s in branges_scales]+ residual_scale_factors = [s[1] for s in branges_scales]++ # 2) get attention bias and index+concat the tensors+ attn_bias, x_cat = get_attn_bias_and_cat(x_list, branges)++ # 3) apply residual_func to get residual, and split the result+ residual_list = attn_bias.split(residual_func(x_cat, attn_bias=attn_bias)) # type: ignore++ outputs = []+ for x, brange, residual, residual_scale_factor in zip(x_list, branges, residual_list, residual_scale_factors):+ outputs.append(add_residual(x, brange, residual, residual_scale_factor, scaling_vector).view_as(x))+ return outputs+++class NestedTensorBlock(Block):+ def forward_nested(self, x_list: List[Tensor]) -> List[Tensor]:+ """+ x_list contains a list of tensors to nest together and run+ """+ assert isinstance(self.attn, MemEffAttention)++ if self.training and self.sample_drop_ratio > 0.0:++ def attn_residual_func(x: Tensor, attn_bias=None) -> Tensor:+ return self.attn(self.norm1(x), attn_bias=attn_bias)++ def ffn_residual_func(x: Tensor, attn_bias=None) -> Tensor:+ return self.mlp(self.norm2(x))++ x_list = drop_add_residual_stochastic_depth_list(+ x_list,+ residual_func=attn_residual_func,+ sample_drop_ratio=self.sample_drop_ratio,+ scaling_vector=self.ls1.gamma if isinstance(self.ls1, LayerScale) else None,+ )+ x_list = drop_add_residual_stochastic_depth_list(+ x_list,+ residual_func=ffn_residual_func,+ sample_drop_ratio=self.sample_drop_ratio,+ scaling_vector=self.ls2.gamma if isinstance(self.ls1, LayerScale) else None,+ )+ return x_list+ else:++ def attn_residual_func(x: Tensor, attn_bias=None) -> Tensor:+ return self.ls1(self.attn(self.norm1(x), attn_bias=attn_bias))++ def ffn_residual_func(x: Tensor, attn_bias=None) -> Tensor:+ return self.ls2(self.mlp(self.norm2(x)))++ attn_bias, x = get_attn_bias_and_cat(x_list)+ x = x + attn_residual_func(x, attn_bias=attn_bias)+ x = x + ffn_residual_func(x)+ return attn_bias.split(x)++ def forward(self, x_or_x_list):+ if isinstance(x_or_x_list, Tensor):+ return super().forward(x_or_x_list)+ elif isinstance(x_or_x_list, list):+ assert XFORMERS_AVAILABLE, "Please install xFormers for nested tensors usage"+ return self.forward_nested(x_or_x_list)+ else:+ raise AssertionErrordiff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/drop_path.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/drop_path.pynew file mode 100644index 0000000..af05625--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/drop_path.py@@ -0,0 +1,35 @@+# Copyright (c) Meta Platforms, Inc. and affiliates.+# All rights reserved.+#+# This source code is licensed under the license found in the+# LICENSE file in the root directory of this source tree.++# References:+# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/drop.py+++from torch import nn+++def drop_path(x, drop_prob: float = 0.0, training: bool = False):+ if drop_prob == 0.0 or not training:+ return x+ keep_prob = 1 - drop_prob+ shape = (x.shape[0],) + (1,) * (x.ndim - 1) # work with diff dim tensors, not just 2D ConvNets+ random_tensor = x.new_empty(shape).bernoulli_(keep_prob)+ if keep_prob > 0.0:+ random_tensor.div_(keep_prob)+ output = x * random_tensor+ return output+++class DropPath(nn.Module):+ """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks)."""++ def __init__(self, drop_prob=None):+ super(DropPath, self).__init__()+ self.drop_prob = drop_prob++ def forward(self, x):+ return drop_path(x, self.drop_prob, self.training)diff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/layer_scale.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/layer_scale.pynew file mode 100644index 0000000..ca5daa5--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/layer_scale.py@@ -0,0 +1,28 @@+# Copyright (c) Meta Platforms, Inc. and affiliates.+# All rights reserved.+#+# This source code is licensed under the license found in the+# LICENSE file in the root directory of this source tree.++# Modified from: https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/vision_transformer.py#L103-L110++from typing import Union++import torch+from torch import Tensor+from torch import nn+++class LayerScale(nn.Module):+ def __init__(+ self,+ dim: int,+ init_values: Union[float, Tensor] = 1e-5,+ inplace: bool = False,+ ) -> None:+ super().__init__()+ self.inplace = inplace+ self.gamma = nn.Parameter(init_values * torch.ones(dim))++ def forward(self, x: Tensor) -> Tensor:+ return x.mul_(self.gamma) if self.inplace else x * self.gammadiff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/mlp.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/mlp.pynew file mode 100644index 0000000..5e4b315--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/mlp.py@@ -0,0 +1,41 @@+# Copyright (c) Meta Platforms, Inc. and affiliates.+# All rights reserved.+#+# This source code is licensed under the license found in the+# LICENSE file in the root directory of this source tree.++# References:+# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/mlp.py+++from typing import Callable, Optional++from torch import Tensor, nn+++class Mlp(nn.Module):+ def __init__(+ self,+ in_features: int,+ hidden_features: Optional[int] = None,+ out_features: Optional[int] = None,+ act_layer: Callable[..., nn.Module] = nn.GELU,+ drop: float = 0.0,+ bias: bool = True,+ ) -> None:+ super().__init__()+ out_features = out_features or in_features+ hidden_features = hidden_features or in_features+ self.fc1 = nn.Linear(in_features, hidden_features, bias=bias)+ self.act = act_layer()+ self.fc2 = nn.Linear(hidden_features, out_features, bias=bias)+ self.drop = nn.Dropout(drop)++ def forward(self, x: Tensor) -> Tensor:+ x = self.fc1(x)+ x = self.act(x)+ x = self.drop(x)+ x = self.fc2(x)+ x = self.drop(x)+ return xdiff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/patch_embed.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/patch_embed.pynew file mode 100644index 0000000..574abe4--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/patch_embed.py@@ -0,0 +1,89 @@+# Copyright (c) Meta Platforms, Inc. and affiliates.+# All rights reserved.+#+# This source code is licensed under the license found in the+# LICENSE file in the root directory of this source tree.++# References:+# https://github.com/facebookresearch/dino/blob/master/vision_transformer.py+# https://github.com/rwightman/pytorch-image-models/tree/master/timm/layers/patch_embed.py++from typing import Callable, Optional, Tuple, Union++from torch import Tensor+import torch.nn as nn+++def make_2tuple(x):+ if isinstance(x, tuple):+ assert len(x) == 2+ return x++ assert isinstance(x, int)+ return (x, x)+++class PatchEmbed(nn.Module):+ """+ 2D image to patch embedding: (B,C,H,W) -> (B,N,D)++ Args:+ img_size: Image size.+ patch_size: Patch token size.+ in_chans: Number of input image channels.+ embed_dim: Number of linear projection output channels.+ norm_layer: Normalization layer.+ """++ def __init__(+ self,+ img_size: Union[int, Tuple[int, int]] = 224,+ patch_size: Union[int, Tuple[int, int]] = 16,+ in_chans: int = 3,+ embed_dim: int = 768,+ norm_layer: Optional[Callable] = None,+ flatten_embedding: bool = True,+ ) -> None:+ super().__init__()++ image_HW = make_2tuple(img_size)+ patch_HW = make_2tuple(patch_size)+ patch_grid_size = (+ image_HW[0] // patch_HW[0],+ image_HW[1] // patch_HW[1],+ )++ self.img_size = image_HW+ self.patch_size = patch_HW+ self.patches_resolution = patch_grid_size+ self.num_patches = patch_grid_size[0] * patch_grid_size[1]++ self.in_chans = in_chans+ self.embed_dim = embed_dim++ self.flatten_embedding = flatten_embedding++ self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_HW, stride=patch_HW)+ self.norm = norm_layer(embed_dim) if norm_layer else nn.Identity()++ def forward(self, x: Tensor) -> Tensor:+ _, _, H, W = x.shape+ patch_H, patch_W = self.patch_size++ assert H % patch_H == 0, f"Input image height {H} is not a multiple of patch height {patch_H}"+ assert W % patch_W == 0, f"Input image width {W} is not a multiple of patch width: {patch_W}"++ x = self.proj(x) # B C H W+ H, W = x.size(2), x.size(3)+ x = x.flatten(2).transpose(1, 2) # B HW C+ x = self.norm(x)+ if not self.flatten_embedding:+ x = x.reshape(-1, H, W, self.embed_dim) # B H W C+ return x++ def flops(self) -> float:+ Ho, Wo = self.patches_resolution+ flops = Ho * Wo * self.embed_dim * self.in_chans * (self.patch_size[0] * self.patch_size[1])+ if self.norm is not None:+ flops += Ho * Wo * self.embed_dim+ return flopsdiff --git a/video2blockout/third_party/video_depth_anything/dinov2_layers/swiglu_ffn.py b/video2blockout/third_party/video_depth_anything/dinov2_layers/swiglu_ffn.pynew file mode 100644index 0000000..b3324b2--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dinov2_layers/swiglu_ffn.py@@ -0,0 +1,63 @@+# Copyright (c) Meta Platforms, Inc. and affiliates.+# All rights reserved.+#+# This source code is licensed under the license found in the+# LICENSE file in the root directory of this source tree.++from typing import Callable, Optional++from torch import Tensor, nn+import torch.nn.functional as F+++class SwiGLUFFN(nn.Module):+ def __init__(+ self,+ in_features: int,+ hidden_features: Optional[int] = None,+ out_features: Optional[int] = None,+ act_layer: Callable[..., nn.Module] = None,+ drop: float = 0.0,+ bias: bool = True,+ ) -> None:+ super().__init__()+ out_features = out_features or in_features+ hidden_features = hidden_features or in_features+ self.w12 = nn.Linear(in_features, 2 * hidden_features, bias=bias)+ self.w3 = nn.Linear(hidden_features, out_features, bias=bias)++ def forward(self, x: Tensor) -> Tensor:+ x12 = self.w12(x)+ x1, x2 = x12.chunk(2, dim=-1)+ hidden = F.silu(x1) * x2+ return self.w3(hidden)+++try:+ from xformers.ops import SwiGLU++ XFORMERS_AVAILABLE = True+except ImportError:+ SwiGLU = SwiGLUFFN+ XFORMERS_AVAILABLE = False+++class SwiGLUFFNFused(SwiGLU):+ def __init__(+ self,+ in_features: int,+ hidden_features: Optional[int] = None,+ out_features: Optional[int] = None,+ act_layer: Callable[..., nn.Module] = None,+ drop: float = 0.0,+ bias: bool = True,+ ) -> None:+ out_features = out_features or in_features+ hidden_features = hidden_features or in_features+ hidden_features = (int(hidden_features * 2 / 3) + 7) // 8 * 8+ super().__init__(+ in_features=in_features,+ hidden_features=hidden_features,+ out_features=out_features,+ bias=bias,+ )diff --git a/video2blockout/third_party/video_depth_anything/dpt.py b/video2blockout/third_party/video_depth_anything/dpt.pynew file mode 100644index 0000000..8c43a14--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dpt.py@@ -0,0 +1,160 @@+# Copyright (2025) Bytedance Ltd. and/or its affiliates++# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at++# http://www.apache.org/licenses/LICENSE-2.0++# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+import torch+import torch.nn as nn+import torch.nn.functional as F++from .util.blocks import FeatureFusionBlock, _make_scratch+++def _make_fusion_block(features, use_bn, size=None):+ return FeatureFusionBlock(+ features,+ nn.ReLU(False),+ deconv=False,+ bn=use_bn,+ expand=False,+ align_corners=True,+ size=size,+ )+++class ConvBlock(nn.Module):+ def __init__(self, in_feature, out_feature):+ super().__init__()++ self.conv_block = nn.Sequential(+ nn.Conv2d(in_feature, out_feature, kernel_size=3, stride=1, padding=1),+ nn.BatchNorm2d(out_feature),+ nn.ReLU(True)+ )++ def forward(self, x):+ return self.conv_block(x)+++class DPTHead(nn.Module):+ def __init__(+ self,+ in_channels,+ features=256,+ use_bn=False,+ out_channels=[256, 512, 1024, 1024],+ use_clstoken=False+ ):+ super(DPTHead, self).__init__()++ self.use_clstoken = use_clstoken++ self.projects = nn.ModuleList([+ nn.Conv2d(+ in_channels=in_channels,+ out_channels=out_channel,+ kernel_size=1,+ stride=1,+ padding=0,+ ) for out_channel in out_channels+ ])++ self.resize_layers = nn.ModuleList([+ nn.ConvTranspose2d(+ in_channels=out_channels[0],+ out_channels=out_channels[0],+ kernel_size=4,+ stride=4,+ padding=0),+ nn.ConvTranspose2d(+ in_channels=out_channels[1],+ out_channels=out_channels[1],+ kernel_size=2,+ stride=2,+ padding=0),+ nn.Identity(),+ nn.Conv2d(+ in_channels=out_channels[3],+ out_channels=out_channels[3],+ kernel_size=3,+ stride=2,+ padding=1)+ ])++ if use_clstoken:+ self.readout_projects = nn.ModuleList()+ for _ in range(len(self.projects)):+ self.readout_projects.append(+ nn.Sequential(+ nn.Linear(2 * in_channels, in_channels),+ nn.GELU()))++ self.scratch = _make_scratch(+ out_channels,+ features,+ groups=1,+ expand=False,+ )++ self.scratch.stem_transpose = None++ self.scratch.refinenet1 = _make_fusion_block(features, use_bn)+ self.scratch.refinenet2 = _make_fusion_block(features, use_bn)+ self.scratch.refinenet3 = _make_fusion_block(features, use_bn)+ self.scratch.refinenet4 = _make_fusion_block(features, use_bn)++ head_features_1 = features+ head_features_2 = 32++ self.scratch.output_conv1 = nn.Conv2d(head_features_1, head_features_1 // 2, kernel_size=3, stride=1, padding=1)+ self.scratch.output_conv2 = nn.Sequential(+ nn.Conv2d(head_features_1 // 2, head_features_2, kernel_size=3, stride=1, padding=1),+ nn.ReLU(True),+ nn.Conv2d(head_features_2, 1, kernel_size=1, stride=1, padding=0),+ nn.ReLU(True),+ nn.Identity(),+ )++ def forward(self, out_features, patch_h, patch_w):+ out = []+ for i, x in enumerate(out_features):+ if self.use_clstoken:+ x, cls_token = x[0], x[1]+ readout = cls_token.unsqueeze(1).expand_as(x)+ x = self.readout_projects[i](torch.cat((x, readout), -1))+ else:+ x = x[0]++ x = x.permute(0, 2, 1).reshape((x.shape[0], x.shape[-1], patch_h, patch_w))++ x = self.projects[i](x)+ x = self.resize_layers[i](x)++ out.append(x)++ layer_1, layer_2, layer_3, layer_4 = out++ layer_1_rn = self.scratch.layer1_rn(layer_1)+ layer_2_rn = self.scratch.layer2_rn(layer_2)+ layer_3_rn = self.scratch.layer3_rn(layer_3)+ layer_4_rn = self.scratch.layer4_rn(layer_4)++ path_4 = self.scratch.refinenet4(layer_4_rn, size=layer_3_rn.shape[2:])+ path_3 = self.scratch.refinenet3(path_4, layer_3_rn, size=layer_2_rn.shape[2:])+ path_2 = self.scratch.refinenet2(path_3, layer_2_rn, size=layer_1_rn.shape[2:])+ path_1 = self.scratch.refinenet1(path_2, layer_1_rn)++ out = self.scratch.output_conv1(path_1)+ out = F.interpolate(out, (int(patch_h * 14), int(patch_w * 14)), mode="bilinear", align_corners=True)+ out = self.scratch.output_conv2(out)++ return out+\ No newline at end of filediff --git a/video2blockout/third_party/video_depth_anything/dpt_temporal.py b/video2blockout/third_party/video_depth_anything/dpt_temporal.pynew file mode 100644index 0000000..d629a11--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/dpt_temporal.py@@ -0,0 +1,125 @@+# Copyright (2025) Bytedance Ltd. and/or its affiliates++# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at++# http://www.apache.org/licenses/LICENSE-2.0++# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+import torch+import torch.nn.functional as F+import torch.nn as nn+from .dpt import DPTHead+from .motion_module.motion_module import TemporalModule+from easydict import EasyDict+++class DPTHeadTemporal(DPTHead):+ def __init__(self,+ in_channels,+ features=256,+ use_bn=False,+ out_channels=[256, 512, 1024, 1024],+ use_clstoken=False,+ num_frames=32,+ pe='ape'+ ):+ super().__init__(in_channels, features, use_bn, out_channels, use_clstoken)++ assert num_frames > 0+ motion_module_kwargs = EasyDict(num_attention_heads = 8,+ num_transformer_block = 1,+ num_attention_blocks = 2,+ temporal_max_len = num_frames,+ zero_initialize = True,+ pos_embedding_type = pe)++ self.motion_modules = nn.ModuleList([+ TemporalModule(in_channels=out_channels[2],+ **motion_module_kwargs),+ TemporalModule(in_channels=out_channels[3],+ **motion_module_kwargs),+ TemporalModule(in_channels=features,+ **motion_module_kwargs),+ TemporalModule(in_channels=features,+ **motion_module_kwargs)+ ])++ def forward(self, out_features, patch_h, patch_w, frame_length, micro_batch_size=4, cached_hidden_state_list=None):+ out = []+ for i, x in enumerate(out_features):+ if self.use_clstoken:+ x, cls_token = x[0], x[1]+ readout = cls_token.unsqueeze(1).expand_as(x)+ x = self.readout_projects[i](torch.cat((x, readout), -1))+ else:+ x = x[0]++ x = x.permute(0, 2, 1).reshape((x.shape[0], x.shape[-1], patch_h, patch_w)).contiguous()++ B, T = x.shape[0] // frame_length, frame_length+ x = self.projects[i](x)+ x = self.resize_layers[i](x)++ out.append(x)++ layer_1, layer_2, layer_3, layer_4 = out++ B, T = layer_1.shape[0] // frame_length, frame_length+ if cached_hidden_state_list is not None:+ N = len(cached_hidden_state_list) // len(self.motion_modules)+ else:+ N = 0++ layer_3, h0 = self.motion_modules[0](layer_3.unflatten(0, (B, T)).permute(0, 2, 1, 3, 4), None, None, cached_hidden_state_list[0:N] if N else None)+ layer_3 = layer_3.permute(0, 2, 1, 3, 4).flatten(0, 1)+ layer_4, h1 = self.motion_modules[1](layer_4.unflatten(0, (B, T)).permute(0, 2, 1, 3, 4), None, None, cached_hidden_state_list[N:2*N] if N else None)+ layer_4 = layer_4.permute(0, 2, 1, 3, 4).flatten(0, 1)++ layer_1_rn = self.scratch.layer1_rn(layer_1)+ layer_2_rn = self.scratch.layer2_rn(layer_2)+ layer_3_rn = self.scratch.layer3_rn(layer_3)+ layer_4_rn = self.scratch.layer4_rn(layer_4)++ path_4 = self.scratch.refinenet4(layer_4_rn, size=layer_3_rn.shape[2:])+ path_4, h2 = self.motion_modules[2](path_4.unflatten(0, (B, T)).permute(0, 2, 1, 3, 4), None, None, cached_hidden_state_list[2*N:3*N] if N else None)+ path_4 = path_4.permute(0, 2, 1, 3, 4).flatten(0, 1)+ path_3 = self.scratch.refinenet3(path_4, layer_3_rn, size=layer_2_rn.shape[2:])+ path_3, h3 = self.motion_modules[3](path_3.unflatten(0, (B, T)).permute(0, 2, 1, 3, 4), None, None, cached_hidden_state_list[3*N:] if N else None)+ path_3 = path_3.permute(0, 2, 1, 3, 4).flatten(0, 1)++ batch_size = layer_1_rn.shape[0]+ if batch_size <= micro_batch_size or batch_size % micro_batch_size != 0:+ path_2 = self.scratch.refinenet2(path_3, layer_2_rn, size=layer_1_rn.shape[2:])+ path_1 = self.scratch.refinenet1(path_2, layer_1_rn)++ out = self.scratch.output_conv1(path_1)+ out = F.interpolate(+ out, (int(patch_h * 14), int(patch_w * 14)), mode="bilinear", align_corners=True+ )+ ori_type = out.dtype+ with torch.autocast(device_type="cuda", enabled=False):+ out = self.scratch.output_conv2(out.float())++ output = out.to(ori_type)+ else:+ ret = []+ for i in range(0, batch_size, micro_batch_size):+ path_2 = self.scratch.refinenet2(path_3[i:i + micro_batch_size], layer_2_rn[i:i + micro_batch_size], size=layer_1_rn[i:i + micro_batch_size].shape[2:])+ path_1 = self.scratch.refinenet1(path_2, layer_1_rn[i:i + micro_batch_size])+ out = self.scratch.output_conv1(path_1)+ out = F.interpolate(+ out, (int(patch_h * 14), int(patch_w * 14)), mode="bilinear", align_corners=True+ )+ ori_type = out.dtype+ with torch.autocast(device_type="cuda", enabled=False):+ out = self.scratch.output_conv2(out.float())+ ret.append(out.to(ori_type))+ output = torch.cat(ret, dim=0)++ return output, h0 + h1 + h2 + h3diff --git a/video2blockout/third_party/video_depth_anything/motion_module/__init__.py b/video2blockout/third_party/video_depth_anything/motion_module/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/video2blockout/third_party/video_depth_anything/motion_module/attention.py b/video2blockout/third_party/video_depth_anything/motion_module/attention.pynew file mode 100644index 0000000..41f551b--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/motion_module/attention.py@@ -0,0 +1,429 @@+# Copyright 2022 The HuggingFace Team. All rights reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+from typing import Optional, Tuple++import torch+import torch.nn.functional as F+from torch import nn++try:+ import xformers+ import xformers.ops++ XFORMERS_AVAILABLE = True+except ImportError:+ print("xFormers not available")+ XFORMERS_AVAILABLE = False+++class CrossAttention(nn.Module):+ r"""+ A cross attention layer.++ Parameters:+ query_dim (`int`): The number of channels in the query.+ cross_attention_dim (`int`, *optional*):+ The number of channels in the encoder_hidden_states. If not given, defaults to `query_dim`.+ heads (`int`, *optional*, defaults to 8): The number of heads to use for multi-head attention.+ dim_head (`int`, *optional*, defaults to 64): The number of channels in each head.+ dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.+ bias (`bool`, *optional*, defaults to False):+ Set to `True` for the query, key, and value linear layers to contain a bias parameter.+ """++ def __init__(+ self,+ query_dim: int,+ cross_attention_dim: Optional[int] = None,+ heads: int = 8,+ dim_head: int = 64,+ dropout: float = 0.0,+ bias=False,+ upcast_attention: bool = False,+ upcast_softmax: bool = False,+ added_kv_proj_dim: Optional[int] = None,+ norm_num_groups: Optional[int] = None,+ ):+ super().__init__()+ inner_dim = dim_head * heads+ cross_attention_dim = cross_attention_dim if cross_attention_dim is not None else query_dim+ self.upcast_attention = upcast_attention+ self.upcast_softmax = upcast_softmax+ self.upcast_efficient_attention = False++ self.scale = dim_head**-0.5++ self.heads = heads+ # for slice_size > 0 the attention score computation+ # is split across the batch axis to save memory+ # You can set slice_size with `set_attention_slice`+ self.sliceable_head_dim = heads+ self._slice_size = None+ self._use_memory_efficient_attention_xformers = False+ self.added_kv_proj_dim = added_kv_proj_dim++ if norm_num_groups is not None:+ self.group_norm = nn.GroupNorm(num_channels=inner_dim, num_groups=norm_num_groups, eps=1e-5, affine=True)+ else:+ self.group_norm = None++ self.to_q = nn.Linear(query_dim, inner_dim, bias=bias)+ self.to_k = nn.Linear(cross_attention_dim, inner_dim, bias=bias)+ self.to_v = nn.Linear(cross_attention_dim, inner_dim, bias=bias)++ if self.added_kv_proj_dim is not None:+ self.add_k_proj = nn.Linear(added_kv_proj_dim, cross_attention_dim)+ self.add_v_proj = nn.Linear(added_kv_proj_dim, cross_attention_dim)++ self.to_out = nn.ModuleList([])+ self.to_out.append(nn.Linear(inner_dim, query_dim))+ self.to_out.append(nn.Dropout(dropout))++ def reshape_heads_to_batch_dim(self, tensor):+ batch_size, seq_len, dim = tensor.shape+ head_size = self.heads+ tensor = tensor.reshape(batch_size, seq_len, head_size, dim // head_size).contiguous()+ tensor = tensor.permute(0, 2, 1, 3).reshape(batch_size * head_size, seq_len, dim // head_size).contiguous()+ return tensor++ def reshape_heads_to_4d(self, tensor):+ batch_size, seq_len, dim = tensor.shape+ head_size = self.heads+ tensor = tensor.reshape(batch_size, seq_len, head_size, dim // head_size).contiguous()+ return tensor++ def reshape_batch_dim_to_heads(self, tensor):+ batch_size, seq_len, dim = tensor.shape+ head_size = self.heads+ tensor = tensor.reshape(batch_size // head_size, head_size, seq_len, dim).contiguous()+ tensor = tensor.permute(0, 2, 1, 3).reshape(batch_size // head_size, seq_len, dim * head_size).contiguous()+ return tensor++ def reshape_4d_to_heads(self, tensor):+ batch_size, seq_len, head_size, dim = tensor.shape+ head_size = self.heads+ tensor = tensor.reshape(batch_size, seq_len, dim * head_size).contiguous()+ return tensor++ def set_attention_slice(self, slice_size):+ if slice_size is not None and slice_size > self.sliceable_head_dim:+ raise ValueError(f"slice_size {slice_size} has to be smaller or equal to {self.sliceable_head_dim}.")++ self._slice_size = slice_size++ def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None):+ batch_size, sequence_length, _ = hidden_states.shape++ encoder_hidden_states = encoder_hidden_states++ if self.group_norm is not None:+ hidden_states = self.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)++ query = self.to_q(hidden_states)+ dim = query.shape[-1]+ query = self.reshape_heads_to_batch_dim(query)++ if self.added_kv_proj_dim is not None:+ key = self.to_k(hidden_states)+ value = self.to_v(hidden_states)+ encoder_hidden_states_key_proj = self.add_k_proj(encoder_hidden_states)+ encoder_hidden_states_value_proj = self.add_v_proj(encoder_hidden_states)++ key = self.reshape_heads_to_batch_dim(key)+ value = self.reshape_heads_to_batch_dim(value)+ encoder_hidden_states_key_proj = self.reshape_heads_to_batch_dim(encoder_hidden_states_key_proj)+ encoder_hidden_states_value_proj = self.reshape_heads_to_batch_dim(encoder_hidden_states_value_proj)++ key = torch.concat([encoder_hidden_states_key_proj, key], dim=1)+ value = torch.concat([encoder_hidden_states_value_proj, value], dim=1)+ else:+ encoder_hidden_states = encoder_hidden_states if encoder_hidden_states is not None else hidden_states+ key = self.to_k(encoder_hidden_states)+ value = self.to_v(encoder_hidden_states)++ key = self.reshape_heads_to_batch_dim(key)+ value = self.reshape_heads_to_batch_dim(value)++ if attention_mask is not None:+ if attention_mask.shape[-1] != query.shape[1]:+ target_length = query.shape[1]+ attention_mask = F.pad(attention_mask, (0, target_length), value=0.0)+ attention_mask = attention_mask.repeat_interleave(self.heads, dim=0)++ # attention, what we cannot get enough of+ if XFORMERS_AVAILABLE and self._use_memory_efficient_attention_xformers:+ hidden_states = self._memory_efficient_attention_xformers(query, key, value, attention_mask)+ # Some versions of xformers return output in fp32, cast it back to the dtype of the input+ hidden_states = hidden_states.to(query.dtype)+ else:+ if self._slice_size is None or query.shape[0] // self._slice_size == 1:+ hidden_states = self._attention(query, key, value, attention_mask)+ else:+ hidden_states = self._sliced_attention(query, key, value, sequence_length, dim, attention_mask)++ # linear proj+ hidden_states = self.to_out[0](hidden_states)++ # dropout+ hidden_states = self.to_out[1](hidden_states)+ return hidden_states++ def _attention(self, query, key, value, attention_mask=None):+ if self.upcast_attention:+ query = query.float()+ key = key.float()++ attention_scores = torch.baddbmm(+ torch.empty(query.shape[0], query.shape[1], key.shape[1], dtype=query.dtype, device=query.device),+ query,+ key.transpose(-1, -2),+ beta=0,+ alpha=self.scale,+ )++ if attention_mask is not None:+ attention_scores = attention_scores + attention_mask++ if self.upcast_softmax:+ attention_scores = attention_scores.float()++ attention_probs = attention_scores.softmax(dim=-1)++ # cast back to the original dtype+ attention_probs = attention_probs.to(value.dtype)++ # compute attention output+ hidden_states = torch.bmm(attention_probs, value)++ # reshape hidden_states+ hidden_states = self.reshape_batch_dim_to_heads(hidden_states)+ return hidden_states++ def _sliced_attention(self, query, key, value, sequence_length, dim, attention_mask):+ batch_size_attention = query.shape[0]+ hidden_states = torch.zeros(+ (batch_size_attention, sequence_length, dim // self.heads), device=query.device, dtype=query.dtype+ )+ slice_size = self._slice_size if self._slice_size is not None else hidden_states.shape[0]+ for i in range(hidden_states.shape[0] // slice_size):+ start_idx = i * slice_size+ end_idx = (i + 1) * slice_size++ query_slice = query[start_idx:end_idx]+ key_slice = key[start_idx:end_idx]++ if self.upcast_attention:+ query_slice = query_slice.float()+ key_slice = key_slice.float()++ attn_slice = torch.baddbmm(+ torch.empty(slice_size, query.shape[1], key.shape[1], dtype=query_slice.dtype, device=query.device),+ query_slice,+ key_slice.transpose(-1, -2),+ beta=0,+ alpha=self.scale,+ )++ if attention_mask is not None:+ attn_slice = attn_slice + attention_mask[start_idx:end_idx]++ if self.upcast_softmax:+ attn_slice = attn_slice.float()++ attn_slice = attn_slice.softmax(dim=-1)++ # cast back to the original dtype+ attn_slice = attn_slice.to(value.dtype)+ attn_slice = torch.bmm(attn_slice, value[start_idx:end_idx])++ hidden_states[start_idx:end_idx] = attn_slice++ # reshape hidden_states+ hidden_states = self.reshape_batch_dim_to_heads(hidden_states)+ return hidden_states++ def _memory_efficient_attention_xformers(self, query, key, value, attention_mask):+ if self.upcast_efficient_attention:+ org_dtype = query.dtype+ query = query.float()+ key = key.float()+ value = value.float()+ if attention_mask is not None:+ attention_mask = attention_mask.float()+ hidden_states = self._memory_efficient_attention_split(query, key, value, attention_mask)++ if self.upcast_efficient_attention:+ hidden_states = hidden_states.to(org_dtype)++ hidden_states = self.reshape_4d_to_heads(hidden_states)+ return hidden_states++ # print("Errror: no xformers")+ # raise NotImplementedError++ def _memory_efficient_attention_split(self, query, key, value, attention_mask):+ batch_size = query.shape[0]+ max_batch_size = 65535+ num_batches = (batch_size + max_batch_size - 1) // max_batch_size+ results = []+ for i in range(num_batches):+ start_idx = i * max_batch_size+ end_idx = min((i + 1) * max_batch_size, batch_size)+ query_batch = query[start_idx:end_idx]+ key_batch = key[start_idx:end_idx]+ value_batch = value[start_idx:end_idx]+ if attention_mask is not None:+ attention_mask_batch = attention_mask[start_idx:end_idx]+ else:+ attention_mask_batch = None+ result = xformers.ops.memory_efficient_attention(query_batch, key_batch, value_batch, attn_bias=attention_mask_batch)+ results.append(result)+ full_result = torch.cat(results, dim=0)+ return full_result+++class FeedForward(nn.Module):+ r"""+ A feed-forward layer.++ Parameters:+ dim (`int`): The number of channels in the input.+ dim_out (`int`, *optional*): The number of channels in the output. If not given, defaults to `dim`.+ mult (`int`, *optional*, defaults to 4): The multiplier to use for the hidden dimension.+ dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.+ activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.+ """++ def __init__(+ self,+ dim: int,+ dim_out: Optional[int] = None,+ mult: int = 4,+ dropout: float = 0.0,+ activation_fn: str = "geglu",+ ):+ super().__init__()+ inner_dim = int(dim * mult)+ dim_out = dim_out if dim_out is not None else dim++ if activation_fn == "gelu":+ act_fn = GELU(dim, inner_dim)+ elif activation_fn == "geglu":+ act_fn = GEGLU(dim, inner_dim)+ elif activation_fn == "geglu-approximate":+ act_fn = ApproximateGELU(dim, inner_dim)++ self.net = nn.ModuleList([])+ # project in+ self.net.append(act_fn)+ # project dropout+ self.net.append(nn.Dropout(dropout))+ # project out+ self.net.append(nn.Linear(inner_dim, dim_out))++ def forward(self, hidden_states):+ for module in self.net:+ hidden_states = module(hidden_states)+ return hidden_states+++class GELU(nn.Module):+ r"""+ GELU activation function+ """++ def __init__(self, dim_in: int, dim_out: int):+ super().__init__()+ self.proj = nn.Linear(dim_in, dim_out)++ def gelu(self, gate):+ if gate.device.type != "mps":+ return F.gelu(gate)+ # mps: gelu is not implemented for float16+ return F.gelu(gate.to(dtype=torch.float32)).to(dtype=gate.dtype)++ def forward(self, hidden_states):+ hidden_states = self.proj(hidden_states)+ hidden_states = self.gelu(hidden_states)+ return hidden_states+++# feedforward+class GEGLU(nn.Module):+ r"""+ A variant of the gated linear unit activation function from https://arxiv.org/abs/2002.05202.++ Parameters:+ dim_in (`int`): The number of channels in the input.+ dim_out (`int`): The number of channels in the output.+ """++ def __init__(self, dim_in: int, dim_out: int):+ super().__init__()+ self.proj = nn.Linear(dim_in, dim_out * 2)++ def gelu(self, gate):+ if gate.device.type != "mps":+ return F.gelu(gate)+ # mps: gelu is not implemented for float16+ return F.gelu(gate.to(dtype=torch.float32)).to(dtype=gate.dtype)++ def forward(self, hidden_states):+ hidden_states, gate = self.proj(hidden_states).chunk(2, dim=-1)+ return hidden_states * self.gelu(gate)+++class ApproximateGELU(nn.Module):+ """+ The approximate form of Gaussian Error Linear Unit (GELU)++ For more details, see section 2: https://arxiv.org/abs/1606.08415+ """++ def __init__(self, dim_in: int, dim_out: int):+ super().__init__()+ self.proj = nn.Linear(dim_in, dim_out)++ def forward(self, x):+ x = self.proj(x)+ return x * torch.sigmoid(1.702 * x)+++def precompute_freqs_cis(dim: int, end: int, theta: float = 10000.0):+ freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))+ t = torch.arange(end, device=freqs.device, dtype=torch.float32)+ freqs = torch.outer(t, freqs)+ freqs_cis = torch.polar(torch.ones_like(freqs), freqs) # complex64+ return freqs_cis+++def reshape_for_broadcast(freqs_cis: torch.Tensor, x: torch.Tensor):+ ndim = x.ndim+ assert 0 <= 1 < ndim+ assert freqs_cis.shape == (x.shape[1], x.shape[-1])+ shape = [d if i == 1 or i == ndim - 1 else 1 for i, d in enumerate(x.shape)]+ return freqs_cis.view(*shape)+++def apply_rotary_emb(+ xq: torch.Tensor,+ xk: torch.Tensor,+ freqs_cis: torch.Tensor,+) -> Tuple[torch.Tensor, torch.Tensor]:+ xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2).contiguous())+ xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2).contiguous())+ freqs_cis = reshape_for_broadcast(freqs_cis, xq_)+ xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(2)+ xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(2)+ return xq_out.type_as(xq), xk_out.type_as(xk)diff --git a/video2blockout/third_party/video_depth_anything/motion_module/motion_module.py b/video2blockout/third_party/video_depth_anything/motion_module/motion_module.pynew file mode 100644index 0000000..d4d81a3--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/motion_module/motion_module.py@@ -0,0 +1,321 @@+# This file is originally from AnimateDiff/animatediff/models/motion_module.py at main · guoyww/AnimateDiff+# SPDX-License-Identifier: Apache-2.0 license+#+# This file may have been modified by ByteDance Ltd. and/or its affiliates on [date of modification]+# Original file was released under [ Apache-2.0 license], with the full license text available at [https://github.com/guoyww/AnimateDiff?tab=Apache-2.0-1-ov-file#readme].+import torch+import torch.nn.functional as F+from torch import nn++from .attention import CrossAttention, FeedForward, apply_rotary_emb, precompute_freqs_cis++from einops import rearrange, repeat+import math++try:+ import xformers+ import xformers.ops++ XFORMERS_AVAILABLE = True+except ImportError:+ print("xFormers not available")+ XFORMERS_AVAILABLE = False+++def zero_module(module):+ # Zero out the parameters of a module and return it.+ for p in module.parameters():+ p.detach().zero_()+ return module+++class TemporalModule(nn.Module):+ def __init__(+ self,+ in_channels,+ num_attention_heads = 8,+ num_transformer_block = 2,+ num_attention_blocks = 2,+ norm_num_groups = 32,+ temporal_max_len = 32,+ zero_initialize = True,+ pos_embedding_type = "ape",+ ):+ super().__init__()++ self.temporal_transformer = TemporalTransformer3DModel(+ in_channels=in_channels,+ num_attention_heads=num_attention_heads,+ attention_head_dim=in_channels // num_attention_heads,+ num_layers=num_transformer_block,+ num_attention_blocks=num_attention_blocks,+ norm_num_groups=norm_num_groups,+ temporal_max_len=temporal_max_len,+ pos_embedding_type=pos_embedding_type,+ )++ if zero_initialize:+ self.temporal_transformer.proj_out = zero_module(self.temporal_transformer.proj_out)++ def forward(self, input_tensor, encoder_hidden_states, attention_mask=None, cached_hidden_state_list=None):+ hidden_states = input_tensor+ hidden_states, output_hidden_state_list = self.temporal_transformer(hidden_states, encoder_hidden_states, attention_mask, cached_hidden_state_list)++ output = hidden_states+ return output, output_hidden_state_list # list of hidden states+++class TemporalTransformer3DModel(nn.Module):+ def __init__(+ self,+ in_channels,+ num_attention_heads,+ attention_head_dim,+ num_layers,+ num_attention_blocks = 2,+ norm_num_groups = 32,+ temporal_max_len = 32,+ pos_embedding_type = "ape",+ ):+ super().__init__()++ inner_dim = num_attention_heads * attention_head_dim++ self.norm = torch.nn.GroupNorm(num_groups=norm_num_groups, num_channels=in_channels, eps=1e-6, affine=True)+ self.proj_in = nn.Linear(in_channels, inner_dim)++ self.transformer_blocks = nn.ModuleList(+ [+ TemporalTransformerBlock(+ dim=inner_dim,+ num_attention_heads=num_attention_heads,+ attention_head_dim=attention_head_dim,+ num_attention_blocks=num_attention_blocks,+ temporal_max_len=temporal_max_len,+ pos_embedding_type=pos_embedding_type,+ )+ for d in range(num_layers)+ ]+ )+ self.proj_out = nn.Linear(inner_dim, in_channels)++ def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, cached_hidden_state_list=None):+ assert hidden_states.dim() == 5, f"Expected hidden_states to have ndim=5, but got ndim={hidden_states.dim()}."+ output_hidden_state_list = []++ video_length = hidden_states.shape[2]+ hidden_states = rearrange(hidden_states, "b c f h w -> (b f) c h w")++ batch, channel, height, width = hidden_states.shape+ residual = hidden_states++ hidden_states = self.norm(hidden_states)+ inner_dim = hidden_states.shape[1]+ hidden_states = hidden_states.permute(0, 2, 3, 1).reshape(batch, height * width, inner_dim).contiguous()+ hidden_states = self.proj_in(hidden_states)++ # Transformer Blocks+ if cached_hidden_state_list is not None:+ n = len(cached_hidden_state_list) // len(self.transformer_blocks)+ else:+ n = 0+ for i, block in enumerate(self.transformer_blocks):+ hidden_states, hidden_state_list = block(hidden_states, encoder_hidden_states=encoder_hidden_states, video_length=video_length, attention_mask=attention_mask,+ cached_hidden_state_list=cached_hidden_state_list[i*n:(i+1)*n] if n else None)+ output_hidden_state_list.extend(hidden_state_list)++ # output+ hidden_states = self.proj_out(hidden_states)+ hidden_states = hidden_states.reshape(batch, height, width, inner_dim).permute(0, 3, 1, 2).contiguous()++ output = hidden_states + residual+ output = rearrange(output, "(b f) c h w -> b c f h w", f=video_length)++ return output, output_hidden_state_list+++class TemporalTransformerBlock(nn.Module):+ def __init__(+ self,+ dim,+ num_attention_heads,+ attention_head_dim,+ num_attention_blocks = 2,+ temporal_max_len = 32,+ pos_embedding_type = "ape",+ ):+ super().__init__()++ self.attention_blocks = nn.ModuleList(+ [+ TemporalAttention(+ query_dim=dim,+ heads=num_attention_heads,+ dim_head=attention_head_dim,+ temporal_max_len=temporal_max_len,+ pos_embedding_type=pos_embedding_type,+ )+ for i in range(num_attention_blocks)+ ]+ )+ self.norms = nn.ModuleList(+ [+ nn.LayerNorm(dim)+ for i in range(num_attention_blocks)+ ]+ )++ self.ff = FeedForward(dim, dropout=0.0, activation_fn="geglu")+ self.ff_norm = nn.LayerNorm(dim)+++ def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, video_length=None, cached_hidden_state_list=None):+ output_hidden_state_list = []+ for i, (attention_block, norm) in enumerate(zip(self.attention_blocks, self.norms)):+ norm_hidden_states = norm(hidden_states)+ residual_hidden_states, output_hidden_states = attention_block(+ norm_hidden_states,+ encoder_hidden_states=encoder_hidden_states,+ video_length=video_length,+ attention_mask=attention_mask,+ cached_hidden_states=cached_hidden_state_list[i] if cached_hidden_state_list is not None else None,+ )+ hidden_states = residual_hidden_states + hidden_states+ output_hidden_state_list.append(output_hidden_states)++ hidden_states = self.ff(self.ff_norm(hidden_states)) + hidden_states++ output = hidden_states+ return output, output_hidden_state_list+++class PositionalEncoding(nn.Module):+ def __init__(+ self,+ d_model,+ dropout = 0.,+ max_len = 32+ ):+ super().__init__()+ self.dropout = nn.Dropout(p=dropout)+ position = torch.arange(max_len).unsqueeze(1)+ div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))+ pe = torch.zeros(1, max_len, d_model)+ pe[0, :, 0::2] = torch.sin(position * div_term)+ pe[0, :, 1::2] = torch.cos(position * div_term)+ self.register_buffer('pe', pe)++ def forward(self, x):+ x = x + self.pe[:, :x.size(1)].to(x.dtype)+ return self.dropout(x)++class TemporalAttention(CrossAttention):+ def __init__(+ self,+ temporal_max_len = 32,+ pos_embedding_type = "ape",+ *args, **kwargs+ ):+ super().__init__(*args, **kwargs)++ self.pos_embedding_type = pos_embedding_type+ self._use_memory_efficient_attention_xformers = True++ self.pos_encoder = None+ self.freqs_cis = None+ if self.pos_embedding_type == "ape":+ self.pos_encoder = PositionalEncoding(+ kwargs["query_dim"],+ dropout=0.,+ max_len=temporal_max_len+ )++ elif self.pos_embedding_type == "rope":+ self.freqs_cis = precompute_freqs_cis(+ kwargs["query_dim"],+ temporal_max_len+ )++ else:+ raise NotImplementedError++ def forward(self, hidden_states, encoder_hidden_states=None, attention_mask=None, video_length=None, cached_hidden_states=None):+ # TODO: support cache for these+ assert encoder_hidden_states is None+ assert attention_mask is None++ d = hidden_states.shape[1]+ d_in = 0+ if cached_hidden_states is None:+ hidden_states = rearrange(hidden_states, "(b f) d c -> (b d) f c", f=video_length)+ input_hidden_states = hidden_states # (bxd) f c+ else:+ hidden_states = rearrange(hidden_states, "(b f) d c -> (b d) f c", f=1)+ input_hidden_states = hidden_states+ d_in = cached_hidden_states.shape[1]+ hidden_states = torch.cat([cached_hidden_states, hidden_states], dim=1)++ if self.pos_encoder is not None:+ hidden_states = self.pos_encoder(hidden_states)++ encoder_hidden_states = repeat(encoder_hidden_states, "b n c -> (b d) n c", d=d) if encoder_hidden_states is not None else encoder_hidden_states++ if self.group_norm is not None:+ hidden_states = self.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)++ query = self.to_q(hidden_states[:, d_in:, ...])+ dim = query.shape[-1]++ if self.added_kv_proj_dim is not None:+ raise NotImplementedError++ encoder_hidden_states = encoder_hidden_states if encoder_hidden_states is not None else hidden_states+ key = self.to_k(encoder_hidden_states)+ value = self.to_v(encoder_hidden_states)++ if self.freqs_cis is not None:+ seq_len = query.shape[1]+ freqs_cis = self.freqs_cis[:seq_len].to(query.device)+ query, key = apply_rotary_emb(query, key, freqs_cis)++ if attention_mask is not None:+ if attention_mask.shape[-1] != query.shape[1]:+ target_length = query.shape[1]+ attention_mask = F.pad(attention_mask, (0, target_length), value=0.0)+ attention_mask = attention_mask.repeat_interleave(self.heads, dim=0)+++ use_memory_efficient = XFORMERS_AVAILABLE and self._use_memory_efficient_attention_xformers+ if use_memory_efficient and (dim // self.heads) % 8 != 0:+ # print('Warning: the dim {} cannot be divided by 8. Fall into normal attention'.format(dim // self.heads))+ use_memory_efficient = False++ # attention, what we cannot get enough of+ if use_memory_efficient:+ query = self.reshape_heads_to_4d(query)+ key = self.reshape_heads_to_4d(key)+ value = self.reshape_heads_to_4d(value)++ hidden_states = self._memory_efficient_attention_xformers(query, key, value, attention_mask)+ # Some versions of xformers return output in fp32, cast it back to the dtype of the input+ hidden_states = hidden_states.to(query.dtype)+ else:+ query = self.reshape_heads_to_batch_dim(query)+ key = self.reshape_heads_to_batch_dim(key)+ value = self.reshape_heads_to_batch_dim(value)++ if self._slice_size is None or query.shape[0] // self._slice_size == 1:+ hidden_states = self._attention(query, key, value, attention_mask)+ else:+ raise NotImplementedError+ # hidden_states = self._sliced_attention(query, key, value, sequence_length, dim, attention_mask)++ # linear proj+ hidden_states = self.to_out[0](hidden_states)++ # dropout+ hidden_states = self.to_out[1](hidden_states)++ hidden_states = rearrange(hidden_states, "(b d) f c -> (b f) d c", d=d)++ return hidden_states, input_hidden_statesdiff --git a/video2blockout/third_party/video_depth_anything/util/__init__.py b/video2blockout/third_party/video_depth_anything/util/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/video2blockout/third_party/video_depth_anything/util/align.py b/video2blockout/third_party/video_depth_anything/util/align.pynew file mode 100644index 0000000..75ff80a--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/util/align.py@@ -0,0 +1,74 @@+# Copyright (2025) Bytedance Ltd. and/or its affiliates++# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at++# http://www.apache.org/licenses/LICENSE-2.0++# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+import numpy as np++def compute_scale_and_shift(prediction, target, mask, scale_only=False):+ if scale_only:+ return compute_scale(prediction, target, mask), 0+ else:+ return compute_scale_and_shift_full(prediction, target, mask)+++def compute_scale(prediction, target, mask):+ # system matrix: A = [[a_00, a_01], [a_10, a_11]]+ prediction = prediction.astype(np.float32)+ target = target.astype(np.float32)+ mask = mask.astype(np.float32)++ a_00 = np.sum(mask * prediction * prediction)+ a_01 = np.sum(mask * prediction)+ a_11 = np.sum(mask)++ # right hand side: b = [b_0, b_1]+ b_0 = np.sum(mask * prediction * target)++ x_0 = b_0 / (a_00 + 1e-6)++ return x_0++def compute_scale_and_shift_full(prediction, target, mask):+ # system matrix: A = [[a_00, a_01], [a_10, a_11]]+ prediction = prediction.astype(np.float32)+ target = target.astype(np.float32)+ mask = mask.astype(np.float32)++ a_00 = np.sum(mask * prediction * prediction)+ a_01 = np.sum(mask * prediction)+ a_11 = np.sum(mask)++ b_0 = np.sum(mask * prediction * target)+ b_1 = np.sum(mask * target)++ x_0 = 1+ x_1 = 0++ det = a_00 * a_11 - a_01 * a_01++ if det != 0:+ x_0 = (a_11 * b_0 - a_01 * b_1) / det+ x_1 = (-a_01 * b_0 + a_00 * b_1) / det++ return x_0, x_1+++def get_interpolate_frames(frame_list_pre, frame_list_post):+ assert len(frame_list_pre) == len(frame_list_post)+ min_w = 0.0+ max_w = 1.0+ step = (max_w - min_w) / (len(frame_list_pre)-1)+ post_w_list = [min_w] + [i * step for i in range(1,len(frame_list_pre)-1)] + [max_w]+ interpolated_frames = []+ for i in range(len(frame_list_pre)):+ interpolated_frames.append(frame_list_pre[i] * (1-post_w_list[i]) + frame_list_post[i] * post_w_list[i])+ return interpolated_frames\ No newline at end of filediff --git a/video2blockout/third_party/video_depth_anything/util/blocks.py b/video2blockout/third_party/video_depth_anything/util/blocks.pynew file mode 100644index 0000000..0be16c0--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/util/blocks.py@@ -0,0 +1,162 @@+import torch.nn as nn+++def _make_scratch(in_shape, out_shape, groups=1, expand=False):+ scratch = nn.Module()++ out_shape1 = out_shape+ out_shape2 = out_shape+ out_shape3 = out_shape+ if len(in_shape) >= 4:+ out_shape4 = out_shape++ if expand:+ out_shape1 = out_shape+ out_shape2 = out_shape * 2+ out_shape3 = out_shape * 4+ if len(in_shape) >= 4:+ out_shape4 = out_shape * 8++ scratch.layer1_rn = nn.Conv2d(+ in_shape[0], out_shape1, kernel_size=3, stride=1, padding=1, bias=False, groups=groups+ )+ scratch.layer2_rn = nn.Conv2d(+ in_shape[1], out_shape2, kernel_size=3, stride=1, padding=1, bias=False, groups=groups+ )+ scratch.layer3_rn = nn.Conv2d(+ in_shape[2], out_shape3, kernel_size=3, stride=1, padding=1, bias=False, groups=groups+ )+ if len(in_shape) >= 4:+ scratch.layer4_rn = nn.Conv2d(+ in_shape[3], out_shape4, kernel_size=3, stride=1, padding=1, bias=False, groups=groups+ )++ return scratch+++class ResidualConvUnit(nn.Module):+ """Residual convolution module."""++ def __init__(self, features, activation, bn):+ """Init.++ Args:+ features (int): number of features+ """+ super().__init__()++ self.bn = bn++ self.groups = 1++ self.conv1 = nn.Conv2d(+ features, features, kernel_size=3, stride=1, padding=1, bias=True, groups=self.groups+ )++ self.conv2 = nn.Conv2d(+ features, features, kernel_size=3, stride=1, padding=1, bias=True, groups=self.groups+ )++ if self.bn is True:+ self.bn1 = nn.BatchNorm2d(features)+ self.bn2 = nn.BatchNorm2d(features)++ self.activation = activation++ self.skip_add = nn.quantized.FloatFunctional()++ def forward(self, x):+ """Forward pass.++ Args:+ x (tensor): input++ Returns:+ tensor: output+ """++ out = self.activation(x)+ out = self.conv1(out)+ if self.bn is True:+ out = self.bn1(out)++ out = self.activation(out)+ out = self.conv2(out)+ if self.bn is True:+ out = self.bn2(out)++ if self.groups > 1:+ out = self.conv_merge(out)++ return self.skip_add.add(out, x)+++class FeatureFusionBlock(nn.Module):+ """Feature fusion block."""++ def __init__(+ self,+ features,+ activation,+ deconv=False,+ bn=False,+ expand=False,+ align_corners=True,+ size=None,+ ):+ """Init.++ Args:+ features (int): number of features+ """+ super().__init__()++ self.deconv = deconv+ self.align_corners = align_corners++ self.groups = 1++ self.expand = expand+ out_features = features+ if self.expand is True:+ out_features = features // 2++ self.out_conv = nn.Conv2d(+ features, out_features, kernel_size=1, stride=1, padding=0, bias=True, groups=1+ )++ self.resConfUnit1 = ResidualConvUnit(features, activation, bn)+ self.resConfUnit2 = ResidualConvUnit(features, activation, bn)++ self.skip_add = nn.quantized.FloatFunctional()++ self.size = size++ def forward(self, *xs, size=None):+ """Forward pass.++ Returns:+ tensor: output+ """+ output = xs[0]++ if len(xs) == 2:+ res = self.resConfUnit1(xs[1])+ output = self.skip_add.add(output, res)++ output = self.resConfUnit2(output)++ if (size is None) and (self.size is None):+ modifier = {"scale_factor": 2}+ elif size is None:+ modifier = {"size": self.size}+ else:+ modifier = {"size": size}++ output = nn.functional.interpolate(+ output.contiguous(), **modifier, mode="bilinear", align_corners=self.align_corners+ )++ output = self.out_conv(output)++ return outputdiff --git a/video2blockout/third_party/video_depth_anything/util/transform.py b/video2blockout/third_party/video_depth_anything/util/transform.pynew file mode 100644index 0000000..b14aacd--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/util/transform.py@@ -0,0 +1,158 @@+import numpy as np+import cv2+++class Resize(object):+ """Resize sample to given size (width, height).+ """++ def __init__(+ self,+ width,+ height,+ resize_target=True,+ keep_aspect_ratio=False,+ ensure_multiple_of=1,+ resize_method="lower_bound",+ image_interpolation_method=cv2.INTER_AREA,+ ):+ """Init.++ Args:+ width (int): desired output width+ height (int): desired output height+ resize_target (bool, optional):+ True: Resize the full sample (image, mask, target).+ False: Resize image only.+ Defaults to True.+ keep_aspect_ratio (bool, optional):+ True: Keep the aspect ratio of the input sample.+ Output sample might not have the given width and height, and+ resize behaviour depends on the parameter 'resize_method'.+ Defaults to False.+ ensure_multiple_of (int, optional):+ Output width and height is constrained to be multiple of this parameter.+ Defaults to 1.+ resize_method (str, optional):+ "lower_bound": Output will be at least as large as the given size.+ "upper_bound": Output will be at max as large as the given size. (Output size might be smaller than given size.)+ "minimal": Scale as least as possible. (Output size might be smaller than given size.)+ Defaults to "lower_bound".+ """+ self.__width = width+ self.__height = height++ self.__resize_target = resize_target+ self.__keep_aspect_ratio = keep_aspect_ratio+ self.__multiple_of = ensure_multiple_of+ self.__resize_method = resize_method+ self.__image_interpolation_method = image_interpolation_method++ def constrain_to_multiple_of(self, x, min_val=0, max_val=None):+ y = (np.round(x / self.__multiple_of) * self.__multiple_of).astype(int)++ if max_val is not None and y > max_val:+ y = (np.floor(x / self.__multiple_of) * self.__multiple_of).astype(int)++ if y < min_val:+ y = (np.ceil(x / self.__multiple_of) * self.__multiple_of).astype(int)++ return y++ def get_size(self, width, height):+ # determine new height and width+ scale_height = self.__height / height+ scale_width = self.__width / width++ if self.__keep_aspect_ratio:+ if self.__resize_method == "lower_bound":+ # scale such that output size is lower bound+ if scale_width > scale_height:+ # fit width+ scale_height = scale_width+ else:+ # fit height+ scale_width = scale_height+ elif self.__resize_method == "upper_bound":+ # scale such that output size is upper bound+ if scale_width < scale_height:+ # fit width+ scale_height = scale_width+ else:+ # fit height+ scale_width = scale_height+ elif self.__resize_method == "minimal":+ # scale as least as possbile+ if abs(1 - scale_width) < abs(1 - scale_height):+ # fit width+ scale_height = scale_width+ else:+ # fit height+ scale_width = scale_height+ else:+ raise ValueError(f"resize_method {self.__resize_method} not implemented")++ if self.__resize_method == "lower_bound":+ new_height = self.constrain_to_multiple_of(scale_height * height, min_val=self.__height)+ new_width = self.constrain_to_multiple_of(scale_width * width, min_val=self.__width)+ elif self.__resize_method == "upper_bound":+ new_height = self.constrain_to_multiple_of(scale_height * height, max_val=self.__height)+ new_width = self.constrain_to_multiple_of(scale_width * width, max_val=self.__width)+ elif self.__resize_method == "minimal":+ new_height = self.constrain_to_multiple_of(scale_height * height)+ new_width = self.constrain_to_multiple_of(scale_width * width)+ else:+ raise ValueError(f"resize_method {self.__resize_method} not implemented")++ return (new_width, new_height)++ def __call__(self, sample):+ width, height = self.get_size(sample["image"].shape[1], sample["image"].shape[0])++ # resize sample+ sample["image"] = cv2.resize(sample["image"], (width, height), interpolation=self.__image_interpolation_method)++ if self.__resize_target:+ if "depth" in sample:+ sample["depth"] = cv2.resize(sample["depth"], (width, height), interpolation=cv2.INTER_NEAREST)++ if "mask" in sample:+ sample["mask"] = cv2.resize(sample["mask"].astype(np.float32), (width, height), interpolation=cv2.INTER_NEAREST)++ return sample+++class NormalizeImage(object):+ """Normlize image by given mean and std.+ """++ def __init__(self, mean, std):+ self.__mean = mean+ self.__std = std++ def __call__(self, sample):+ sample["image"] = (sample["image"] - self.__mean) / self.__std++ return sample+++class PrepareForNet(object):+ """Prepare sample for usage as network input.+ """++ def __init__(self):+ pass++ def __call__(self, sample):+ image = np.transpose(sample["image"], (2, 0, 1))+ sample["image"] = np.ascontiguousarray(image).astype(np.float32)++ if "depth" in sample:+ depth = sample["depth"].astype(np.float32)+ sample["depth"] = np.ascontiguousarray(depth)++ if "mask" in sample:+ sample["mask"] = sample["mask"].astype(np.float32)+ sample["mask"] = np.ascontiguousarray(sample["mask"])++ return sample\ No newline at end of filediff --git a/video2blockout/third_party/video_depth_anything/video_depth.py b/video2blockout/third_party/video_depth_anything/video_depth.pynew file mode 100644index 0000000..cf3bce6--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/video_depth.py@@ -0,0 +1,163 @@+# Copyright (2025) Bytedance Ltd. and/or its affiliates++# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at++# http://www.apache.org/licenses/LICENSE-2.0++# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+import torch+import torch.nn.functional as F+import torch.nn as nn+from torchvision.transforms import Compose+import cv2+from tqdm import tqdm+import numpy as np+import gc++from .dinov2 import DINOv2+from .dpt_temporal import DPTHeadTemporal+from .util.transform import Resize, NormalizeImage, PrepareForNet++from .util.align import compute_scale_and_shift, get_interpolate_frames # vendored: was `utils.util`++# infer settings, do not change+INFER_LEN = 32+OVERLAP = 10+KEYFRAMES = [0,12,24,25,26,27,28,29,30,31]+INTERP_LEN = 8++class VideoDepthAnything(nn.Module):+ def __init__(+ self,+ encoder='vitl',+ features=256,+ out_channels=[256, 512, 1024, 1024],+ use_bn=False,+ use_clstoken=False,+ num_frames=32,+ pe='ape',+ metric=False,+ ):+ super(VideoDepthAnything, self).__init__()++ self.intermediate_layer_idx = {+ 'vits': [2, 5, 8, 11],+ "vitb": [2, 5, 8, 11],+ 'vitl': [4, 11, 17, 23]+ }++ self.encoder = encoder+ self.pretrained = DINOv2(model_name=encoder)++ self.head = DPTHeadTemporal(self.pretrained.embed_dim, features, use_bn, out_channels=out_channels, use_clstoken=use_clstoken, num_frames=num_frames, pe=pe)+ self.metric = metric++ def forward(self, x):+ B, T, C, H, W = x.shape+ patch_h, patch_w = H // 14, W // 14+ features = self.pretrained.get_intermediate_layers(x.flatten(0,1), self.intermediate_layer_idx[self.encoder], return_class_token=True)+ depth = self.head(features, patch_h, patch_w, T)[0]+ depth = F.interpolate(depth, size=(H, W), mode="bilinear", align_corners=True)+ depth = F.relu(depth)+ return depth.squeeze(1).unflatten(0, (B, T)) # return shape [B, T, H, W]++ def infer_video_depth(self, frames, target_fps, input_size=518, device='cuda', fp32=False):+ frame_height, frame_width = frames[0].shape[:2]+ ratio = max(frame_height, frame_width) / min(frame_height, frame_width)+ if ratio > 1.78: # we recommend to process video with ratio smaller than 16:9 due to memory limitation+ input_size = int(input_size * 1.777 / ratio)+ input_size = round(input_size / 14) * 14++ transform = Compose([+ Resize(+ width=input_size,+ height=input_size,+ resize_target=False,+ keep_aspect_ratio=True,+ ensure_multiple_of=14,+ resize_method='lower_bound',+ image_interpolation_method=cv2.INTER_CUBIC,+ ),+ NormalizeImage(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),+ PrepareForNet(),+ ])++ frame_list = [frames[i] for i in range(frames.shape[0])]+ frame_step = INFER_LEN - OVERLAP+ org_video_len = len(frame_list)+ append_frame_len = (frame_step - (org_video_len % frame_step)) % frame_step + (INFER_LEN - frame_step)+ frame_list = frame_list + [frame_list[-1].copy()] * append_frame_len++ depth_list = []+ pre_input = None+ for frame_id in tqdm(range(0, org_video_len, frame_step)):+ cur_list = []+ for i in range(INFER_LEN):+ cur_list.append(torch.from_numpy(transform({'image': frame_list[frame_id+i].astype(np.float32) / 255.0})['image']).unsqueeze(0).unsqueeze(0))+ cur_input = torch.cat(cur_list, dim=1).to(device)+ if pre_input is not None:+ cur_input[:, :OVERLAP, ...] = pre_input[:, KEYFRAMES, ...]++ with torch.no_grad():+ with torch.autocast(device_type=device, enabled=(not fp32)):+ depth = self.forward(cur_input) # depth shape: [1, T, H, W]++ depth = depth.to(cur_input.dtype)+ depth = F.interpolate(depth.flatten(0,1).unsqueeze(1), size=(frame_height, frame_width), mode='bilinear', align_corners=True)+ depth_list += [depth[i][0].cpu().numpy() for i in range(depth.shape[0])]++ pre_input = cur_input++ del frame_list+ gc.collect()++ depth_list_aligned = []+ ref_align = []+ align_len = OVERLAP - INTERP_LEN+ kf_align_list = KEYFRAMES[:align_len]++ for frame_id in range(0, len(depth_list), INFER_LEN):+ if len(depth_list_aligned) == 0:+ depth_list_aligned += depth_list[:INFER_LEN]+ for kf_id in kf_align_list:+ ref_align.append(depth_list[frame_id+kf_id])+ else:+ curr_align = []+ for i in range(len(kf_align_list)):+ curr_align.append(depth_list[frame_id+i])++ if self.metric:+ scale, shift = 1.0, 0.0+ else:+ scale, shift = compute_scale_and_shift(np.concatenate(curr_align),+ np.concatenate(ref_align),+ np.concatenate(np.ones_like(ref_align)==1))++ pre_depth_list = depth_list_aligned[-INTERP_LEN:]+ post_depth_list = depth_list[frame_id+align_len:frame_id+OVERLAP]+ for i in range(len(post_depth_list)):+ post_depth_list[i] = post_depth_list[i] * scale + shift+ post_depth_list[i][post_depth_list[i]<0] = 0+ depth_list_aligned[-INTERP_LEN:] = get_interpolate_frames(pre_depth_list, post_depth_list)++ for i in range(OVERLAP, INFER_LEN):+ new_depth = depth_list[frame_id+i] * scale + shift+ new_depth[new_depth<0] = 0+ depth_list_aligned.append(new_depth)++ ref_align = ref_align[:1]+ for kf_id in kf_align_list[1:]:+ new_depth = depth_list[frame_id+kf_id] * scale + shift+ new_depth[new_depth<0] = 0+ ref_align.append(new_depth)++ depth_list = depth_list_aligned++ return np.stack(depth_list[:org_video_len], axis=0), target_fps+diff --git a/video2blockout/third_party/video_depth_anything/video_depth_stream.py b/video2blockout/third_party/video_depth_anything/video_depth_stream.pynew file mode 100644index 0000000..dbee3fc--- /dev/null+++ b/video2blockout/third_party/video_depth_anything/video_depth_stream.py@@ -0,0 +1,161 @@+# Copyright (2025) Bytedance Ltd. and/or its affiliates++# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at++# http://www.apache.org/licenses/LICENSE-2.0++# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+import torch+import torch.nn.functional as F+import torch.nn as nn+from torchvision.transforms import Compose+import cv2+import numpy as np++from .dinov2 import DINOv2+from .dpt_temporal import DPTHeadTemporal+from .util.transform import Resize, NormalizeImage, PrepareForNet++from .util.align import compute_scale_and_shift, get_interpolate_frames # vendored: was `utils.util`++# infer settings, do not change+INFER_LEN = 32+OVERLAP = 10+INTERP_LEN = 8++class VideoDepthAnything(nn.Module):+ def __init__(+ self,+ encoder='vitl',+ features=256,+ out_channels=[256, 512, 1024, 1024],+ use_bn=False,+ use_clstoken=False,+ num_frames=32,+ pe='ape'+ ):+ super(VideoDepthAnything, self).__init__()++ self.intermediate_layer_idx = {+ 'vits': [2, 5, 8, 11],+ "vitb": [2, 5, 8, 11],+ 'vitl': [4, 11, 17, 23]+ }++ self.encoder = encoder+ self.pretrained = DINOv2(model_name=encoder)++ self.head = DPTHeadTemporal(self.pretrained.embed_dim, features, use_bn, out_channels=out_channels, use_clstoken=use_clstoken, num_frames=num_frames, pe=pe)+ self.transform = None+ self.frame_id_list = []+ self.frame_cache_list = []+ self.gap = (INFER_LEN - OVERLAP) * 2 - 1 - (OVERLAP - INTERP_LEN)+ assert self.gap == 41+ self.id = -1++ def forward(self, x):+ return self.forward_depth(self.forward_features(x), x.shape)[0]++ def forward_features(self, x):+ features = self.pretrained.get_intermediate_layers(x.flatten(0,1), self.intermediate_layer_idx[self.encoder], return_class_token=True)+ return features++ def forward_depth(self, features, x_shape, cached_hidden_state_list=None):+ B, T, C, H, W = x_shape+ patch_h, patch_w = H // 14, W // 14+ depth, cur_cached_hidden_state_list = self.head(features, patch_h, patch_w, T, cached_hidden_state_list=cached_hidden_state_list)+ depth = F.interpolate(depth, size=(H, W), mode="bilinear", align_corners=True)+ depth = F.relu(depth)+ return depth.squeeze(1).unflatten(0, (B, T)), cur_cached_hidden_state_list # return shape [B, T, H, W]++ def infer_video_depth_one(self, frame, input_size=518, device='cuda', fp32=False):+ self.id += 1++ if self.transform is None: # first frame+ # Initialize the transform+ frame_height, frame_width = frame.shape[:2]+ self.frame_height = frame_height+ self.frame_width = frame_width+ ratio = max(frame_height, frame_width) / min(frame_height, frame_width)+ if ratio > 1.78: # we recommend to process video with ratio smaller than 16:9 due to memory limitation+ input_size = int(input_size * 1.777 / ratio)+ input_size = round(input_size / 14) * 14++ self.transform = Compose([+ Resize(+ width=input_size,+ height=input_size,+ resize_target=False,+ keep_aspect_ratio=True,+ ensure_multiple_of=14,+ resize_method='lower_bound',+ image_interpolation_method=cv2.INTER_CUBIC,+ ),+ NormalizeImage(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),+ PrepareForNet(),+ ])++ # Inference the first frame+ cur_list = [torch.from_numpy(self.transform({'image': frame.astype(np.float32) / 255.0})['image']).unsqueeze(0).unsqueeze(0)]+ cur_input = torch.cat(cur_list, dim=1).to(device)++ with torch.no_grad():+ with torch.autocast(device_type=device, enabled=(not fp32)):+ cur_feature = self.forward_features(cur_input)+ x_shape = cur_input.shape+ depth, cached_hidden_state_list = self.forward_depth(cur_feature, x_shape)++ depth = depth.to(cur_input.dtype)+ depth = F.interpolate(depth.flatten(0,1).unsqueeze(1), size=(frame_height, frame_width), mode='bilinear', align_corners=True)++ # Copy multiple cache to simulate the windows+ self.frame_cache_list = [cached_hidden_state_list] * INFER_LEN+ self.frame_id_list.extend([0] * (INFER_LEN - 1))++ new_depth = depth[0][0].cpu().numpy()+ else:+ frame_height, frame_width = frame.shape[:2]+ assert frame_height == self.frame_height+ assert frame_width == self.frame_width++ # infer feature+ cur_input = torch.from_numpy(self.transform({'image': frame.astype(np.float32) / 255.0})['image']).unsqueeze(0).unsqueeze(0).to(device)+ with torch.no_grad():+ with torch.autocast(device_type=device, enabled=(not fp32)):+ cur_feature = self.forward_features(cur_input)+ x_shape = cur_input.shape++ cur_list = self.frame_cache_list[0:2] + self.frame_cache_list[-INFER_LEN+3:]+ '''+ cur_id = self.frame_id_list[0:2] + self.frame_id_list[-INFER_LEN+3:]+ print(f"cur_id: {cur_id}")+ '''+ assert len(cur_list) == INFER_LEN - 1+ cur_cache = [torch.cat([h[i] for h in cur_list], dim=1) for i in range(len(cur_list[0]))]++ # infer depth+ with torch.no_grad():+ with torch.autocast(device_type=device, enabled=(not fp32)):+ depth, new_cache = self.forward_depth(cur_feature, x_shape, cached_hidden_state_list=cur_cache)++ depth = depth.to(cur_input.dtype)+ depth = F.interpolate(depth.flatten(0,1).unsqueeze(1), size=(frame_height, frame_width), mode='bilinear', align_corners=True)+ depth_list = [depth[i][0].cpu().numpy() for i in range(depth.shape[0])]++ new_depth = depth_list[-1]++ self.frame_cache_list.append(new_cache)++ # adjust the sliding window+ self.frame_id_list.append(self.id)+ if self.id + INFER_LEN > self.gap + 1:+ del self.frame_id_list[1]+ del self.frame_cache_list[1]++ return new_depth