Skip to content

[SIGGRAPH Asia 2026] 4DAnyone: Create Anyone in 4D from a Casual Monocular Video

Initial release

krahetsf8afd50764c0
diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml
new file mode 100644
index 0000000..18e0c43
--- /dev/null
+++ b/.github/workflows/pages.yml
@@ -0,0 +1,122 @@
+name: Deploy project page
+run-name: Deploy page ${{ inputs.release_sha }}
+
+on:
+ workflow_dispatch:
+ inputs:
+ source_sha:
+ description: Exact commit on the temporary gh-pages branch
+ required: true
+ type: string
+ release_sha:
+ description: Source snapshot in krahets/4danyone-page
+ required: true
+ type: string
+
+permissions:
+ contents: write
+ pages: write
+ id-token: write
+
+concurrency:
+ group: github-pages
+ cancel-in-progress: false
+
+jobs:
+ package:
+ runs-on: ubuntu-latest
+ timeout-minutes: 30
+ steps:
+ - name: Validate repository and dispatch inputs
+ env:
+ RELEASE_SHA: ${{ inputs.release_sha }}
+ SOURCE_SHA: ${{ inputs.source_sha }}
+ shell: bash
+ run: |
+ set -euo pipefail
+ case "$GITHUB_REPOSITORY" in
+ 4danyone/4danyone.github.io|ant-research/4DAnyone) ;;
+ *) echo "Unsupported Pages target: $GITHUB_REPOSITORY" >&2; exit 1 ;;
+ esac
+ [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]]
+ [[ "$RELEASE_SHA" =~ ^[0-9a-f]{40}$ ]]
+
+ - name: Check out the temporary page snapshot
+ uses: actions/checkout@v6
+ with:
+ ref: gh-pages
+ path: site
+ fetch-depth: 1
+ persist-credentials: false
+
+ - name: Verify the static snapshot
+ env:
+ SOURCE_SHA: ${{ inputs.source_sha }}
+ shell: bash
+ run: |
+ set -euo pipefail
+ actual_sha=$(git -C site rev-parse HEAD)
+ if [[ "$actual_sha" != "$SOURCE_SHA" ]]; then
+ echo "Checked out $actual_sha, expected $SOURCE_SHA" >&2
+ exit 1
+ fi
+ test -f site/index.html
+ test -f site/.nojekyll
+ test -d site/assets
+ for path in .dev .github tools CLAUDE.md README.md .gitignore; do
+ test ! -e "site/$path"
+ done
+ find site/.git -depth -delete
+ if find site -type l -print -quit | grep -q .; then
+ echo "The Pages artifact must not contain symbolic links" >&2
+ exit 1
+ fi
+ du -sh site
+
+ - name: Configure GitHub Pages
+ uses: actions/configure-pages@v5
+
+ - name: Upload the Pages artifact
+ uses: actions/upload-pages-artifact@v4
+ with:
+ path: site
+
+ - name: Remove the temporary gh-pages branch
+ if: always()
+ continue-on-error: true
+ env:
+ GH_TOKEN: ${{ github.token }}
+ SOURCE_SHA: ${{ inputs.source_sha }}
+ shell: bash
+ run: |
+ set -euo pipefail
+ ref_json=$(curl --fail-with-body --silent --show-error \
+ --header "Accept: application/vnd.github+json" \
+ --header "Authorization: Bearer $GH_TOKEN" \
+ --header "X-GitHub-Api-Version: 2022-11-28" \
+ "https://api.github.com/repos/${GITHUB_REPOSITORY}/git/ref/heads/gh-pages" || true)
+ remote_sha=$(printf '%s\n' "$ref_json" | jq -r '.object.sha // empty')
+ if [[ -z "$remote_sha" ]]; then
+ echo "gh-pages is already absent"
+ elif [[ "$remote_sha" != "$SOURCE_SHA" ]]; then
+ echo "gh-pages moved to $remote_sha; refusing to delete it" >&2
+ else
+ curl --fail-with-body --silent --show-error \
+ --request DELETE \
+ --header "Accept: application/vnd.github+json" \
+ --header "Authorization: Bearer $GH_TOKEN" \
+ --header "X-GitHub-Api-Version: 2022-11-28" \
+ "https://api.github.com/repos/${GITHUB_REPOSITORY}/git/refs/heads/gh-pages"
+ fi
+
+ deploy:
+ needs: package
+ runs-on: ubuntu-latest
+ timeout-minutes: 15
+ environment:
+ name: github-pages
+ url: ${{ steps.deployment.outputs.page_url }}
+ steps:
+ - name: Deploy to GitHub Pages
+ id: deployment
+ uses: actions/deploy-pages@v4
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..c5e6274
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,35 @@
+.DS_Store
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+.venv/
+dist/
+build/
+*.egg-info/
+
+# Runtime artifacts and large assets
+outputs/
+/data/fdanyone/
+/data/gvhmr/
+/data/nerfstudio/
+/data/.download/
+/data/source/pexels/*.mp4
+/models/
+/checkpoints/
+*.safetensors
+*.ckpt
+*.pth
+*.pt
+
+# Local-only configuration
+.env
+.env.*
+!.env.example
+
+# Local agent state may contain private provenance.
+.claude/
+work/
diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 0000000..0f5d511
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "third_party/GVHMR"]
+ path = third_party/GVHMR
+ url = https://github.com/zju3dv/GVHMR.git
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000..a664f84
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright 2026 4DAnyone authors
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..f179c7e
--- /dev/null
+++ b/README.md
@@ -0,0 +1,132 @@
+<p align="center"><a href="https://4danyone.github.io/"><img src="docs/assets/logo_title.png" width="300" alt="4DAnyone"></a></p>
+
+<h2 align="center">4DAnyone: Create Anyone in 4D from a Casual Monocular Video</h2>
+
+<p align="center"><a href="https://4danyone.github.io/"><strong>Project Page</strong></a> &nbsp;|&nbsp; <a href="https://arxiv.org/abs/2608.20335"><strong>Paper</strong></a></p>
+
+<p align="center"><img src="docs/assets/teaser.gif" width="100%" alt="4DAnyone teaser"></p>
+
+<p align="center">4DAnyone turns a casual monocular video into multi-view videos, enabling downstream 4DGS reconstruction.</p>
+
+## Installation
+
+```bash
+git clone https://github.com/ant-research/4DAnyone.git
+cd 4DAnyone
+git submodule update --init third_party/GVHMR
+
+conda create -n 4danyone python=3.11 -y
+conda activate 4danyone
+pip install -r requirements.txt
+```
+
+For faster inference, optionally install [FlashAttention-3](https://github.com/Dao-AILab/flash-attention/tree/main/hopper) or [SageAttention](https://github.com/thu-ml/SageAttention).
+
+Missing models and examples are downloaded automatically on first use. You can also download them manually:
+
+```bash
+python scripts/download_smplx.py
+python scripts/download_model.py
+python scripts/download_example.py
+```
+
+## Inference
+
+### Generate target views
+
+```bash
+# Generate 6 evenly spaced views around the subject
+python inference.py \
+ --video_path "data/source/pexels/10331522-uhd_2160_4096_25fps.mp4" \
+ --views_per_layer 6
+
+# Generate 24 evenly spaced views around the subject
+python inference.py \
+ --video_path "data/source/pexels/10331522-uhd_2160_4096_25fps.mp4" \
+ --views_per_layer 24
+
+# Generate 48 views with 3 pitch layers and 16 views per layer
+python inference.py \
+ --video_path "data/source/pexels/10331522-uhd_2160_4096_25fps.mp4" \
+ --views_per_layer 16 --layer_pitches '[-10,15,35]'
+
+# Generate 8 views over the frontal 180-degree arc
+python inference.py \
+ --video_path "data/source/pexels/10331522-uhd_2160_4096_25fps.mp4" \
+ --views_per_layer 8 --start_yaw -90 --yaw_span 180
+
+# Generate one view at pitch 15° and yaw 60°
+python inference.py \
+ --video_path "data/source/pexels/10331522-uhd_2160_4096_25fps.mp4" \
+ --views_per_layer 1 --layer_pitches '[15]' --start_yaw 60
+```
+
+Camera layout options:
+
+- `layer_pitches`: pitch angle of each camera layer, in degrees; positive values place the cameras above the person.
+- `views_per_layer`: number of evenly spaced views in each camera layer. Total views are `views_per_layer × len(layer_pitches)`.
+- `start_yaw`: horizontal angle of the first view, in degrees; yaw `0` is the front view.
+- `yaw_span`: horizontal range covered by each camera layer, in degrees.
+
+Run `python inference.py --help` to see all inference options.
+
+### Output
+
+With the default `--data_dir data`, results follow this layout. See the [output documentation](docs/output.md) for the complete format.
+
+```text
+data/
+├── gvhmr/results/<clip>/ # reusable motion-recovery result
+└── fdanyone/<clip>/
+ ├── metadata.json # run settings, timings, resources
+ ├── cameras.json # the final N-camera rig
+ ├── skeletons/00.mp4 ... <N-1>.mp4
+ └── videos/
+ ├── sparse/{00,04,09,12,14,19}.mp4 # default 24-view RCP proposals
+ └── dense/00.mp4 ... <N-1>.mp4 # generated target views
+```
+
+### Custom data
+
+Use a handheld-like video that:
+
+- has a portrait 9:16 aspect ratio;
+- shows one person, either full body or upper body;
+- contains at least 121 frames;
+- has only mild camera movement.
+
+## 3DGS Reconstruction
+
+Install [Nerfstudio](https://docs.nerf.studio/quickstart/installation.html) before running `ns-train`.
+
+```bash
+# Export frame 0 across all generated views
+python scripts/export_nerfstudio.py \
+ --result_dir data/fdanyone/<clip> \
+ --frame_index 0
+
+# Train foreground-only 3DGS
+ns-train splatfacto \
+ --data data/nerfstudio/<clip>/frame_000 \
+ --pipeline.model.background-color random
+```
+
+## Todos
+
+- [ ] Low-memory inference (<32 GB)
+- [ ] Faster inference with TensorRT and sparse attention
+- [ ] Support 4DGS reconstruction with an open-source method
+
+## Citation
+
+If you find 4DAnyone useful or interesting, please cite our work and consider giving the repository a star ⭐:
+
+```bibtex
+@article{jin2026fdanyone,
+ title={4DAnyone: Create Anyone in 4D from a Casual Monocular Video},
+ author={Jin, Yudong and Xie, Tao and Zhang, Qihang and Shen, Zehong and Xu, Zhen and Shen, Yujun and Bao, Hujun and Zhou, Xiaowei and Xu, Yinghao},
+ journal={arXiv preprint arXiv:2608.20335},
+ year={2026},
+ url={https://arxiv.org/abs/2608.20335}
+}
+```
diff --git a/docs/THIRD_PARTY_NOTICES.md b/docs/THIRD_PARTY_NOTICES.md
new file mode 100644
index 0000000..2cb5751
--- /dev/null
+++ b/docs/THIRD_PARTY_NOTICES.md
@@ -0,0 +1,80 @@
+# Third-party notices
+
+The root Apache-2.0 license applies only to first-party 4DAnyone code. The following components and assets retain their own terms.
+
+## GVHMR
+
+- Upstream source: <https://github.com/zju3dv/GVHMR>
+- Pinned upstream revision: `6ec3ca39336c50492c0fae65fba2fb831fc7d866`
+- Location: `third_party/GVHMR`
+- License: `third_party/GVHMR/LICENSE`
+
+4DAnyone uses the official upstream source without a GVHMR source patch. The pinned revision is immutable and can be fetched anonymously from the upstream repository.
+
+GVHMR, HMR2, ViTPose, YOLO, SMPL, and SMPL-X model files are not bundled in 4DAnyone. The complete public inference closure is anchored by the immutable Hugging Face revision frozen in `fdanyone/assets.py`; users obtain those files through the applicable upstream or licensed setup flows.
+
+GVHMR permits use, copying, modification, and distribution for educational, research, and non-profit purposes only. It requires modifications based on the work to be open source and prohibits commercial use. Its checkpoints and transitive model assets may have additional terms.
+
+## BiRefNet
+
+- Upstream source and model: <https://huggingface.co/ZhengPeng7/BiRefNet>
+- Pinned model revision: `e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4`
+- Upstream code: <https://github.com/ZhengPeng7/BiRefNet>
+- License: MIT, copied at `third_party/licenses/BIREFNET_LICENSE`
+
+The adaptive preprocessing path downloads the pinned BiRefNet configuration, inference source, and weights on demand. It uses the resulting foreground masks for source cropping and source-framing analysis; those downloaded files retain their upstream terms.
+
+## PyTorch3D compatibility subset
+
+- Upstream source: <https://github.com/facebookresearch/pytorch3d>
+- Upstream revision: `f34104cf6ebefacd7b7e07955ee7aaa823e616ac`
+- Location: `fdanyone/vendor/pytorch3d_compat`
+- License: BSD 3-Clause, copied at `fdanyone/vendor/pytorch3d_compat/LICENSE`
+
+Classic GVHMR imports PyTorch3D rotation conversions through a broader training-time module surface. 4DAnyone retains the two upstream source files needed by inference, plus small first-party module aliases and a native PyTorch fallback for an imported optional KNN helper. The renderer and compiled PyTorch3D operators are not included or required. `UPSTREAM.md` and `VENDORED_FILES.txt` record the exact retained closure and source hashes.
+
+## Pexels example media
+
+- Source platform: <https://www.pexels.com/>
+- Location: `data/source/pexels`
+- Terms: <https://www.pexels.com/license/>
+
+The companion Hugging Face repository distributes eight modified 121-frame excerpts for running the public demo; they are installed under the location above on demand. The excerpts derive from Pexels videos [10331522](https://www.pexels.com/video/10331522/), [2785536](https://www.pexels.com/video/2785536/), [5435720](https://www.pexels.com/video/5435720/), [5885633](https://www.pexels.com/video/5885633/), [5999210](https://www.pexels.com/video/5999210/), [6980035](https://www.pexels.com/video/6980035/), [7080903](https://www.pexels.com/video/7080903/), and [7480858](https://www.pexels.com/video/7480858/). Pexels permits its media to be used and modified for free and does not require attribution; the source links are retained for provenance and creator credit.
+
+## DiffSynth-Studio inference runtime
+
+- Public source: <https://github.com/modelscope/DiffSynth-Studio>
+- Public base revision: `04e39f7de53df7276a7b40ca1791c2a393e05ff3`
+- Original experiment fork revision: `c00782d90c872c97bda4745a9e6a41a0a4a7c4db`
+- Location: `fdanyone/vendor/diffsynth`
+- License: Apache-2.0, copied at `fdanyone/vendor/diffsynth/LICENSE`
+
+`UPSTREAM.md`, `UPSTREAM.patch`, and `VENDORED_FILES.txt` in that directory record the exact provenance, research patch, and retained file set.
+
+Two retained DiffSynth files carry code-level upstream attribution:
+
+- `models/wan_video_pose_encoder.py` derives its pose network from [Tencent/MimicMotion](https://github.com/Tencent/MimicMotion/tree/c053153a1d124abae8c08568925ae88debc63001), Copyright Tencent, under Apache-2.0.
+- `models/wan_video_camera_controller.py` contains camera-coordinate helpers from [hehao13/CameraCtrl](https://github.com/hehao13/CameraCtrl), under Apache-2.0. The exact retained bytes remain identified through the DiffSynth public-base revision above.
+
+## Sapiens2-derived Goliath/MHR schema material
+
+- Source project: <https://github.com/facebookresearch/sapiens2>
+- Source revision: `0e51c12d7c7257d88431b2d50e523a7b03004854`
+- Source file: `sapiens/pose/configs/_base_/keypoints308.py`
+- Retained material: the exact MHR70 name/order and Goliath40 link closure needed to reproduce model conditioning
+- First-party Apache-2.0 material: RGB palette, color assignment policy, and renderer implementation in `fdanyone/skeleton`
+- License copy: `third_party/licenses/SAPIENS2_LICENSE.md`
+
+No Sapiens2 model weights are distributed by this repository.
+
+The retained schema material is conservatively treated as Sapiens2-derived and the complete agreement is included rather than assuming that the root Apache-2.0 license applies. The agreement contains explicit prohibited-use terms, including surveillance, biometric processing, identification or re-identification, deepfakes or deceptive content, and the other activities listed in section 1(b)(vi).
+
+## Model and body assets
+
+The 4DAnyone checkpoint and sanitized MHR70 sparse regression tensors are first-party release artifacts licensed under Apache-2.0, published at the Hugging Face revision frozen in `fdanyone/assets.py`. The regressor file also embeds the exact keypoint-name order required to interpret those tensors; that schema material retains the Sapiens2 notice above and is not relicensed by the tensor license. Wan VAE/T5/tokenizer, GVHMR checkpoints, HMR2, ViTPose, detector weights, and SMPL-X body models remain third-party assets. The redistributable files are mirrored by the companion Hugging Face repository; users obtain SMPL-X from its official site and remain responsible for every applicable license.
+
+The frozen base-model assets come from the official [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B) repository at revision `37685f96025fc1425edccdd4b2bca3836ae917ff` and the official [Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) repository at revision `3f40b6dc4ca5c02dd23c9db74d9d2ccb82903b86`. Their official model cards license those model assets under Apache-2.0; the redistributed copies are pinned by the frozen Hugging Face revision.
+
+The MHR70 asset is a sanitized inference artifact containing only the sparse support/weight tensors, keypoint names, and a path-free structural schema. The first-party regression tensors are redistributed under Apache-2.0; the embedded name/order metadata remains covered by the schema notice above.
+
+That regressor was trained by the 4DAnyone project on DNA-Rendering frames, using outputs from [SAM 3D Body](https://github.com/facebookresearch/sam-3d-body) and the [Momentum Human Rig](https://github.com/facebookresearch/MHR) conversion tools. SAM 3D Body is governed by the SAM License, MHR source is Apache-2.0, and DNA-Rendering access requires a separate signed dataset agreement. The regressor is not bundled in the source repository; its Hugging Face model card must retain these source-model and dataset citations alongside the split tensor/schema license declaration in this notice.
diff --git a/docs/assets/logo_title.png b/docs/assets/logo_title.png
new file mode 100644
index 0000000..14bde2d
Binary files /dev/null and b/docs/assets/logo_title.png differ
diff --git a/docs/assets/teaser.gif b/docs/assets/teaser.gif
new file mode 100644
index 0000000..d778d1a
Binary files /dev/null and b/docs/assets/teaser.gif differ
diff --git a/docs/output.md b/docs/output.md
new file mode 100644
index 0000000..5e5043a
--- /dev/null
+++ b/docs/output.md
@@ -0,0 +1,70 @@
+# Output
+
+With the default `--data_dir data`, one inference run writes:
+
+```bash
+data/
+├── gvhmr/
+│ └── results/<clip>/
+│ ├── motion.json
+│ └── motion.safetensors
+└── fdanyone/
+ └── <clip>/
+ ├── metadata.json
+ ├── cameras.json
+ ├── skeletons/00.mp4 ... <N-1>.mp4
+ └── videos/
+ ├── sparse/{00,04,09,12,14,19}.mp4 # default 24-view RCP proposals
+ └── dense/00.mp4 ... <N-1>.mp4
+```
+
+`<clip>` is the input filename without its extension. Results are written atomically; a run never overwrites an existing result, and temporary intermediates are removed after a successful run.
+
+## GVHMR motion
+
+`motion.safetensors` contains the global and input-camera SMPL parameters, per-frame camera intrinsics, and observed ViTPose keypoints recovered from the input. `motion.json` records the GVHMR revision, timeline, frame identity, raster, and coordinate convention. When generation is retried with the same input clip, the validated motion result is reused instead of recomputed.
+
+## 4DAnyone videos
+
+`videos/dense/` contains `views_per_layer × len(layer_pitches)` final views. Camera IDs are layer-major: every pitch layer contains the requested evenly spaced yaw views before the next layer begins. Yaw `0` faces the person's front, positive pitch places the camera above the subject, and the default camera `00` is therefore frontal. Runs with at most six targets generate them directly; larger runs use RCP by default, with one, four, or six proposal videos following `views_per_group` and stored in `videos/sparse/`. At most the first four proposals are used as target-stage references. `skeletons/` contains one synchronized Goliath40 conditioning video per final camera, with IDs matching `videos/dense/`. All videos contain 121 frames at 704×1280 and use the selected input clip FPS. In `auto` mode, the source FPS is preserved unless it can be evenly reduced to 24, 25, or 30 FPS. Audio is not copied.
+
+Before generation, the canonical source frames are segmented by the pinned BiRefNet revision at its standard 1024×1024 inference resolution, using FP16 batches of four. Their union mask is cropped with a 4% margin and a center-contain policy. Projected input-camera anatomy, foreground support, and ViTPose agreement classify the source as full-body, half-body, or close-up. The generated views then share one sequence-level camera solve:
+
+- the base radius protects the 95th-percentile body height and 80th-percentile body width around the full camera ring;
+- confident half-body and close-up inputs additionally adapt target height and focal length;
+- target and RCP conditioning use a center-aspect crop, while skeleton line width follows projected 3D body scale.
+
+`metadata.json` records this preprocessing solve alongside the resolved layer pitches, yaw range, grouping, input timeline, motion and checkpoint identity, timings, and resource measurements. `cameras.json` stores each camera's layer, pitch, yaw, intrinsics, and pose in OpenCV camera coordinates and the canonical 4DAnyone Y-up human world.
+
+## Nerfstudio export
+
+Nerfstudio consumes images rather than MP4 files. Export one synchronized timestamp after inference:
+
+```bash
+python scripts/export_nerfstudio.py \
+ --result_dir data/fdanyone/<clip> \
+ --frame_index 60
+```
+
+This creates:
+
+```bash
+data/nerfstudio/<clip>/frame_060/
+├── transforms.json
+├── images/00.jpg ... <N-1>.jpg
+└── masks/00.png ... <N-1>.png
+```
+
+The exporter runs the same pinned standard BiRefNet model used by inference, at 1024×1024 in FP16 batches of four. Missing BiRefNet files are downloaded automatically into `--model_dir` on first use. Its soft predictions are thresholded at 0.5 and written as single-channel binary PNGs: white pixels are foreground and black pixels are excluded from training.
+
+`transforms.json` follows the [Nerfstudio `OPENCV` dataset schema](https://docs.nerf.studio/quickstart/data_conventions.html). Every frame contains both `file_path` and `mask_path`; Nerfstudio loads those masks automatically. Intrinsics are stored per image so each exported frame remains self-contained. Camera poses are converted from 4DAnyone's OpenCV/Y-up convention to Nerfstudio's OpenGL/Z-up camera-to-world convention. The directory can be passed directly to Splatfacto:
+
+```bash
+ns-train splatfacto \
+ --data data/nerfstudio/<clip>/frame_060 \
+ --pipeline.model.background-color random
+```
+
+The masks remove generated background pixels from the image loss. The explicit random rendering background also discourages transparent Gaussians from relying on one fixed composite color.
+
+Each export is one static multi-view timestamp. Exporting all 121 frames into one vanilla Nerfstudio dataset would mix a moving person across time; dynamic methods should instead use a method-specific temporal data parser.
diff --git a/fdanyone/__init__.py b/fdanyone/__init__.py
new file mode 100644
index 0000000..523aa32
--- /dev/null
+++ b/fdanyone/__init__.py
@@ -0,0 +1 @@
+"""4DAnyone inference."""
diff --git a/fdanyone/assets.py b/fdanyone/assets.py
new file mode 100644
index 0000000..4f63abd
--- /dev/null
+++ b/fdanyone/assets.py
@@ -0,0 +1,128 @@
+"""Locate the model files used by 4DAnyone.
+
+Every published file is anchored by one immutable Hugging Face revision and
+downloaded on demand. ``fdanyone.download`` fetches missing files;
+the resolvers here only locate them.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+from fdanyone.errors import AssetError
+
+HF_REPO_ID = "AntResearch/4DAnyone"
+HF_REVISION = "eab457d2761b15c7fe7b7b05484ae75a94119677"
+
+BIREFNET_REPO_ID = "ZhengPeng7/BiRefNet"
+BIREFNET_REVISION = "e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4"
+BIREFNET_DIR = "birefnet"
+BIREFNET_FILES = (
+ "BiRefNet_config.py",
+ "birefnet.py",
+ "config.json",
+ "model.safetensors",
+)
+
+CHECKPOINT = "4danyone/model.safetensors"
+MHR70_REGRESSOR = "4danyone/smplx_to_goliath70.pt"
+WAN_VAE = "4danyone/Wan2.2_VAE.pth"
+TEXT_ENCODER = "4danyone/models_t5_umt5-xxl-enc-bf16.pth"
+TOKENIZER_DIR = "4danyone/umt5-xxl"
+TOKENIZER_FILES = tuple(
+ f"{TOKENIZER_DIR}/{name}"
+ for name in ("special_tokens_map.json", "spiece.model", "tokenizer.json", "tokenizer_config.json")
+)
+
+GVHMR_CHECKPOINT = "gvhmr/gvhmr_siga24_release.ckpt"
+HMR2_CHECKPOINT = "gvhmr/epoch=10-step=25000.ckpt"
+VITPOSE_CHECKPOINT = "gvhmr/vitpose-h-multi-coco.pth"
+YOLO_CHECKPOINT = "gvhmr/yolov8x.pt"
+
+SMPLX_MODEL = "body_models/smplx/SMPLX_NEUTRAL.npz"
+
+MODEL_FILES = (
+ CHECKPOINT,
+ MHR70_REGRESSOR,
+ WAN_VAE,
+ TEXT_ENCODER,
+ *TOKENIZER_FILES,
+ GVHMR_CHECKPOINT,
+ HMR2_CHECKPOINT,
+ VITPOSE_CHECKPOINT,
+ YOLO_CHECKPOINT,
+)
+
+EXAMPLE_FILES = (
+ "data/source/pexels/10331522-uhd_2160_4096_25fps.mp4",
+ "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4",
+ "data/source/pexels/5435720-uhd_2160_4096_25fps.mp4",
+ "data/source/pexels/5885633-hd_1080_1920_25fps.mp4",
+ "data/source/pexels/5999210-uhd_2160_4096_25fps.mp4",
+ "data/source/pexels/6980035-uhd_2160_4096_30fps.mp4",
+ "data/source/pexels/7080903-hd_1080_1920_30fps.mp4",
+ "data/source/pexels/7480858-uhd_2160_3840_25fps.mp4",
+)
+
+# Upstream GVHMR resolves its model files relative to its own checkout, so the
+# install commands link each downloaded file to the location GVHMR expects.
+GVHMR_LINKS = (
+ (GVHMR_CHECKPOINT, "inputs/checkpoints/gvhmr/gvhmr_siga24_release.ckpt"),
+ (HMR2_CHECKPOINT, "inputs/checkpoints/hmr2/epoch=10-step=25000.ckpt"),
+ (VITPOSE_CHECKPOINT, "inputs/checkpoints/vitpose/vitpose-h-multi-coco.pth"),
+ (YOLO_CHECKPOINT, "inputs/checkpoints/yolo/yolov8x.pt"),
+ (SMPLX_MODEL, "inputs/checkpoints/body_models/smplx/SMPLX_NEUTRAL.npz"),
+)
+
+
+@dataclass(frozen=True)
+class BaseAssets:
+ vae: Path
+ text_encoder: Path
+ tokenizer: Path
+
+
+def _require_file(path: Path, label: str, command: str) -> Path:
+ resolved = path.expanduser().resolve()
+ if not resolved.is_file():
+ raise AssetError(f"{label} does not exist: {resolved}. Run `python {command}` to install it.")
+ return resolved
+
+
+def resolve_checkpoint(path: str | Path | None = None, model_dir: str | Path = "models") -> Path:
+ if path is not None:
+ resolved = Path(path).expanduser().resolve()
+ if not resolved.is_file():
+ raise AssetError(f"Checkpoint override does not exist: {resolved}")
+ return resolved
+ return _require_file(Path(model_dir) / CHECKPOINT, "Checkpoint", "scripts/download_model.py")
+
+
+def resolve_regressor(path: str | Path | None = None, model_dir: str | Path = "models") -> Path:
+ if path is not None:
+ resolved = Path(path).expanduser().resolve()
+ if not resolved.is_file():
+ raise AssetError(f"MHR70 regressor override does not exist: {resolved}")
+ return resolved
+ return _require_file(Path(model_dir) / MHR70_REGRESSOR, "MHR70 regressor", "scripts/download_model.py")
+
+
+def resolve_foreground_model(model_dir: str | Path = "models") -> Path:
+ root = Path(model_dir).expanduser() / BIREFNET_DIR
+ for relative in BIREFNET_FILES:
+ _require_file(root / relative, "BiRefNet file", "scripts/download_model.py")
+ return root.resolve()
+
+
+def resolve_base_assets(model_dir: str | Path = "models") -> BaseAssets:
+ """Resolve the local VAE, T5 encoder, and tokenizer."""
+
+ root = Path(model_dir).expanduser()
+ for relative in TOKENIZER_FILES:
+ _require_file(root / relative, "Tokenizer file", "scripts/download_model.py")
+ return BaseAssets(
+ vae=_require_file(root / WAN_VAE, "VAE", "scripts/download_model.py"),
+ text_encoder=_require_file(root / TEXT_ENCODER, "Text encoder", "scripts/download_model.py"),
+ tokenizer=(root / TOKENIZER_DIR).expanduser().resolve(),
+ )
diff --git a/fdanyone/config.py b/fdanyone/config.py
new file mode 100644
index 0000000..60a3414
--- /dev/null
+++ b/fdanyone/config.py
@@ -0,0 +1,113 @@
+"""Fixed model and preprocessing settings used by the released method.
+
+Only reader-useful choices live in the CLI. These values describe the trained
+model and therefore stay together here instead of being exposed as knobs.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class InferenceConfig:
+ num_frames: int = 121
+ height: int = 1280
+ width: int = 704
+ prompt: str = "视频中的人在做动作"
+ auto_downsample_fps: tuple[tuple[int, int], ...] = (
+ (24, 1),
+ (24000, 1001),
+ (25, 1),
+ (30, 1),
+ (30000, 1001),
+ )
+ temporal_sampling_policy: str = "nearest_source_pts_on_zero_based_cfr_clock"
+ rcp_jpeg_quality: int = 85
+ skeleton_h264_crf: int = 17
+ target_h264_crf: int = 18
+ h264_preset: str = "medium"
+ skeleton_max_dimension: int = 2048
+ num_inference_steps: int = 24
+ denoising_strength: float = 1.0
+ scheduler_shift: float = 5.0
+ use_mvs_attention: bool = True
+ mvs_attention_range: tuple[int | None, int | None, int | None] = (0, None, 1)
+ disable_video_attention: bool = False
+ use_4d_attention: bool = False
+ use_source_self_attention: bool = False
+ use_source_cross_attention: bool = False
+ freqs_src_shift: int = 121
+ use_pose_encoder: bool = True
+ pose_encoder_type: str = "rgb"
+ use_camera_encoder: bool = False
+ use_lbm: bool = False
+ tiled_vae: bool = False
+ vae_tile_size: tuple[int, int] = (52, 30)
+ vae_tile_stride: tuple[int, int] = (26, 15)
+
+
+@dataclass(frozen=True)
+class CameraConfig:
+ count: int = 24
+ pitch_degrees: float = 15.0
+
+ def __post_init__(self) -> None:
+ if self.count <= 0:
+ raise ValueError("Camera count must be positive.")
+
+
+@dataclass(frozen=True)
+class ForegroundConfig:
+ """Pinned standard BiRefNet inference contract."""
+
+ image_size: tuple[int, int] = (1024, 1024)
+ batch_size: int = 4
+
+
+@dataclass(frozen=True)
+class FramingConfig:
+ """Sequence-level camera solve matching the current GVHMR demo."""
+
+ reference_radius: float = 3.0
+ reference_target_height: float = 1.0
+ reference_focal_normalized: float = 1664.0 / 1280.0
+ height_target_ratio: float = 0.80
+ height_percentile: float = 95.0
+ width_target_ratio: float = 0.90
+ width_percentile: float = 80.0
+ min_radius: float = 1.5
+ max_radius: float = 8.0
+ input_min_confidence: float = 0.55
+ max_focal_normalized: float = 4.0
+ cutoff_target_ratio: float = 0.99
+ cutoff_percentile: float = 80.0
+
+
+@dataclass(frozen=True)
+class CropConfig:
+ """Source-mask crop; generated cameras use a plain center aspect crop."""
+
+ margin_top: float = 0.04
+ margin_right: float = 0.04
+ margin_bottom: float = 0.04
+ margin_left: float = 0.04
+ allow_upscale: bool = True
+ mask_threshold: float = 0.05
+
+ @property
+ def margins(self) -> tuple[float, float, float, float]:
+ return (self.margin_top, self.margin_right, self.margin_bottom, self.margin_left)
+
+
+@dataclass(frozen=True)
+class SkeletonConfig:
+ draw_body_reference_px: float = 640.0
+
+
+INFERENCE = InferenceConfig()
+CAMERA = CameraConfig()
+FOREGROUND = ForegroundConfig()
+FRAMING = FramingConfig()
+CROP = CropConfig()
+SKELETON = SkeletonConfig()
diff --git a/fdanyone/device.py b/fdanyone/device.py
new file mode 100644
index 0000000..73bd828
--- /dev/null
+++ b/fdanyone/device.py
@@ -0,0 +1,25 @@
+"""CUDA device selection shared by pipeline and isolated workers."""
+
+from __future__ import annotations
+
+from fdanyone.errors import ConfigurationError
+
+
+def select_cuda_device(device: str) -> tuple[str, int]:
+ """Validate, select, and normalize one CUDA device."""
+
+ import torch
+
+ try:
+ requested = torch.device(device)
+ except (RuntimeError, TypeError, ValueError) as exc:
+ raise ConfigurationError(f"Invalid CUDA device {device!r}.") from exc
+ if requested.type != "cuda" or not torch.cuda.is_available():
+ raise ConfigurationError(f"4DAnyone requires an available CUDA device, got {device!r}.")
+ index = torch.cuda.current_device() if requested.index is None else requested.index
+ if index < 0 or index >= torch.cuda.device_count():
+ raise ConfigurationError(
+ f"CUDA device index {index} is unavailable; visible device count is {torch.cuda.device_count()}."
+ )
+ torch.cuda.set_device(index)
+ return f"cuda:{index}", index
diff --git a/fdanyone/download.py b/fdanyone/download.py
new file mode 100644
index 0000000..e51be2a
--- /dev/null
+++ b/fdanyone/download.py
@@ -0,0 +1,362 @@
+"""Download the published 4DAnyone assets from Hugging Face.
+
+Missing model checkpoints and bundled example clips are fetched automatically
+when inference needs them; the scripts under ``scripts/`` pre-fetch the same
+files. SMPL-X is licensed separately, so first-run inference starts its
+interactive installer only when a terminal is available.
+"""
+
+from __future__ import annotations
+
+import getpass
+import logging
+import os
+import shlex
+import shutil
+import sys
+import tempfile
+import urllib.error
+import urllib.parse
+import urllib.request
+import zipfile
+from pathlib import Path, PurePosixPath
+
+from fdanyone.assets import (
+ BIREFNET_DIR,
+ BIREFNET_FILES,
+ BIREFNET_REPO_ID,
+ BIREFNET_REVISION,
+ EXAMPLE_FILES,
+ GVHMR_LINKS,
+ HF_REPO_ID,
+ HF_REVISION,
+ MODEL_FILES,
+ SMPLX_MODEL,
+)
+from fdanyone.errors import AssetError
+
+LOGGER = logging.getLogger("fdanyone")
+
+SMPLX_HOME = "https://smpl-x.is.tue.mpg.de/"
+SMPLX_DOWNLOAD_URL = "https://download.is.tue.mpg.de/download.php?domain=smplx&sfile=models_smplx_v1_1.zip"
+SMPLX_ARCHIVE_MEMBER = ("models", "smplx", "SMPLX_NEUTRAL.npz")
+
+
+def _snapshot(
+ allow_patterns: list[str],
+ local_dir: Path,
+ *,
+ repo_id: str = HF_REPO_ID,
+ revision: str = HF_REVISION,
+) -> None:
+ try:
+ from huggingface_hub import snapshot_download
+ except ImportError as exc:
+ raise AssetError("Install requirements.txt before downloading assets.") from exc
+
+ try:
+ snapshot_download(
+ repo_id=repo_id,
+ revision=revision,
+ allow_patterns=allow_patterns,
+ local_dir=local_dir,
+ )
+ except Exception as exc:
+ raise AssetError(
+ f"Could not download {repo_id}@{revision}. Check the network connection and Hugging Face access."
+ ) from exc
+
+
+def ensure_foreground_model(model_dir: str | Path = "models") -> Path:
+ root = Path(model_dir).expanduser().resolve() / BIREFNET_DIR
+ missing = [relative for relative in BIREFNET_FILES if not (root / relative).is_file()]
+ if missing:
+ LOGGER.info("Downloading BiRefNet foreground model (first run only)")
+ _snapshot(
+ missing,
+ root,
+ repo_id=BIREFNET_REPO_ID,
+ revision=BIREFNET_REVISION,
+ )
+ return root
+
+
+def require_gvhmr_checkout(gvhmr_root: str | Path) -> Path:
+ root = Path(gvhmr_root).expanduser().resolve()
+ if not (root / "hmr4d/__init__.py").is_file():
+ raise AssetError(
+ f"GVHMR is not initialized at {root}. Run `git submodule update --init third_party/GVHMR` first."
+ )
+ return root
+
+
+def _ensure_link(source: Path, destination: Path) -> None:
+ source = source.expanduser().resolve()
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ if destination.is_symlink():
+ try:
+ if destination.resolve(strict=True).samefile(source):
+ return
+ except FileNotFoundError:
+ pass
+ destination.unlink()
+ elif destination.exists():
+ if destination.samefile(source):
+ return
+ raise AssetError(
+ f"GVHMR asset location is occupied by an unrelated file: {destination}. "
+ f"Move it away so the downloaded {source.name} can be linked."
+ )
+ relative = os.path.relpath(source, start=destination.parent)
+ destination.symlink_to(relative)
+
+
+def create_classic_gvhmr_links(
+ model_dir: str | Path = "models",
+ gvhmr_root: str | Path = "third_party/GVHMR",
+ *,
+ require_models: bool = True,
+ require_smplx: bool = True,
+) -> Path:
+ """Create the ignored compatibility links expected by upstream GVHMR."""
+
+ root = require_gvhmr_checkout(gvhmr_root)
+ models = Path(model_dir).expanduser().resolve()
+ for relative, target in GVHMR_LINKS:
+ source = models / relative
+ if not source.is_file():
+ required = require_smplx if relative == SMPLX_MODEL else require_models
+ if required:
+ command = "scripts/download_smplx.py" if relative == SMPLX_MODEL else "scripts/download_model.py"
+ raise AssetError(f"Model file is missing: {source}. Run `python {command}` first.")
+ continue
+ _ensure_link(source, root / target)
+ return root
+
+
+def ensure_models(
+ model_dir: str | Path = "models",
+ gvhmr_root: str | Path = "third_party/GVHMR",
+) -> Path:
+ """Download any missing published model file and refresh the GVHMR links."""
+
+ require_gvhmr_checkout(gvhmr_root)
+ models = Path(model_dir).expanduser().resolve()
+ missing = [relative for relative in MODEL_FILES if not (models / relative).is_file()]
+ if missing:
+ LOGGER.info("Downloading %d model files from %s (first run only)", len(missing), HF_REPO_ID)
+ # Repository paths match the local layout, so download straight into
+ # place; huggingface_hub stages and resumes partial files itself.
+ _snapshot(missing, models)
+ ensure_foreground_model(models)
+ create_classic_gvhmr_links(models, gvhmr_root, require_smplx=False)
+ return models
+
+
+def download_model(
+ model_dir: str = "models",
+ gvhmr_root: str = "third_party/GVHMR",
+) -> dict[str, str]:
+ """Download the published model checkpoints."""
+
+ models = ensure_models(model_dir, gvhmr_root)
+ return {
+ "models": str(models),
+ "revision": HF_REVISION,
+ "foreground_revision": BIREFNET_REVISION,
+ }
+
+
+def download_example(data_dir: str = "data") -> dict[str, str]:
+ """Download the bundled example clips."""
+
+ data = Path(data_dir).expanduser().resolve()
+ destinations = {relative: data / Path(relative).relative_to("data") for relative in EXAMPLE_FILES}
+ missing = [relative for relative, destination in destinations.items() if not destination.is_file()]
+ if missing:
+ # Repository paths carry a leading ``data/`` prefix while --data_dir is
+ # the local root itself, so stage the snapshot and move each file.
+ staging = data / ".download"
+ _snapshot(missing, staging)
+ for relative in missing:
+ destination = destinations[relative]
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ (staging / relative).replace(destination)
+ shutil.rmtree(staging)
+ return {"examples": str(data / "source/pexels"), "revision": HF_REVISION}
+
+
+def ensure_example_video(video_path: str | Path) -> Path:
+ """Fetch a bundled example clip when its expected file is missing."""
+
+ path = Path(video_path).expanduser()
+ if path.is_file():
+ return path
+ matches = [relative for relative in EXAMPLE_FILES if PurePosixPath(relative).name == path.name]
+ if not matches:
+ raise AssetError(f"Input video does not exist: {path.resolve()}")
+ LOGGER.info("Downloading the bundled example clip %s", path.name)
+ path.parent.mkdir(parents=True, exist_ok=True)
+ # Stage beside the destination so the final rename stays on one filesystem.
+ staging = path.parent / ".download"
+ _snapshot(matches[:1], staging)
+ (staging / matches[0]).replace(path)
+ shutil.rmtree(staging)
+ return path
+
+
+def _parse_interactive_path(value: str) -> Path:
+ try:
+ parts = shlex.split(value.strip())
+ except ValueError as exc:
+ raise AssetError(f"Could not parse the archive path: {exc}") from None
+ if len(parts) != 1:
+ raise AssetError("Enter one ZIP or SMPLX_NEUTRAL.npz path.")
+ return Path(parts[0]).expanduser()
+
+
+def _copy_model_from_source(source: Path, destination: Path) -> None:
+ if source.name == "SMPLX_NEUTRAL.npz":
+ shutil.copyfile(source, destination)
+ return
+ if zipfile.is_zipfile(source):
+ try:
+ with zipfile.ZipFile(source) as archive:
+ candidates = [
+ info
+ for info in archive.infolist()
+ if not info.is_dir() and PurePosixPath(info.filename).parts[-3:] == SMPLX_ARCHIVE_MEMBER
+ ]
+ if len(candidates) != 1:
+ raise AssetError(
+ "The archive must contain exactly one models/smplx/SMPLX_NEUTRAL.npz file. "
+ "Download models_smplx_v1_1.zip from the official SMPL-X website."
+ )
+ with archive.open(candidates[0]) as model, destination.open("wb") as output:
+ shutil.copyfileobj(model, output, length=8 * 1024 * 1024)
+ except zipfile.BadZipFile:
+ raise AssetError(f"SMPL-X archive is invalid: {source}") from None
+ return
+ raise AssetError("Select models_smplx_v1_1.zip or SMPLX_NEUTRAL.npz.")
+
+
+def install_smplx(
+ source_path: str | Path,
+ model_dir: str | Path = "models",
+ gvhmr_root: str | Path = "third_party/GVHMR",
+) -> Path:
+ """Install a user-provided official ZIP or neutral NPZ."""
+
+ source = Path(source_path).expanduser().resolve()
+ if not source.is_file():
+ raise AssetError(f"SMPL-X source does not exist: {source}")
+
+ target = Path(model_dir).expanduser().resolve() / SMPLX_MODEL
+ target.parent.mkdir(parents=True, exist_ok=True)
+ temporary = target.parent / f".{target.name}.download-{os.getpid()}"
+ try:
+ _copy_model_from_source(source, temporary)
+ temporary.replace(target)
+ finally:
+ temporary.unlink(missing_ok=True)
+ create_classic_gvhmr_links(model_dir, gvhmr_root, require_models=False, require_smplx=True)
+ return target
+
+
+def _download_official(username: str, password: str, destination: Path) -> None:
+ payload = urllib.parse.urlencode({"username": username, "password": password}).encode()
+ request = urllib.request.Request(
+ SMPLX_DOWNLOAD_URL,
+ data=payload,
+ headers={"User-Agent": "4DAnyone SMPL-X installer"},
+ method="POST",
+ )
+ try:
+ with urllib.request.urlopen(request, timeout=120) as response, destination.open("wb") as output:
+ shutil.copyfileobj(response, output, length=8 * 1024 * 1024)
+ except (OSError, urllib.error.URLError) as exc:
+ raise AssetError(f"Official SMPL-X download failed: {exc}") from None
+ if not zipfile.is_zipfile(destination):
+ raise AssetError(
+ "The SMPL-X website did not return a ZIP archive. Check the account, license acceptance, or website."
+ )
+
+
+def _prompt_for_archive(model_dir: str, gvhmr_root: str) -> dict[str, str] | None:
+ print(f"Download models_smplx_v1_1.zip from:\n {SMPLX_DOWNLOAD_URL}")
+ while True:
+ try:
+ value = input("Archive path (drag the downloaded ZIP here): ").strip()
+ except EOFError:
+ value = ""
+ if not value:
+ print("SMPL-X setup cancelled; the downloaded ZIP was not modified.")
+ return None
+ try:
+ installed = install_smplx(_parse_interactive_path(value), model_dir, gvhmr_root)
+ except AssetError as exc:
+ print(f"error: {exc}")
+ continue
+ return {"installed": str(installed)}
+
+
+def download_smplx(
+ archive_path: str | None = None,
+ model_dir: str = "models",
+ gvhmr_root: str = "third_party/GVHMR",
+) -> dict[str, str] | None:
+ """Install the separately licensed SMPL-X neutral body model."""
+
+ target = Path(model_dir).expanduser().resolve() / SMPLX_MODEL
+ if target.is_file():
+ create_classic_gvhmr_links(model_dir, gvhmr_root, require_models=False, require_smplx=True)
+ return {"installed": str(target)}
+
+ if archive_path is not None:
+ return {"installed": str(install_smplx(archive_path, model_dir, gvhmr_root))}
+
+ print(f"SMPL-X requires a free account and license acceptance at {SMPLX_HOME}")
+ try:
+ accepted = input("Have you registered and accepted the SMPL-X license? [y/N]: ").strip().lower()
+ except EOFError:
+ accepted = ""
+ if accepted in {"y", "yes"}:
+ username = input("SMPL-X username or email: ").strip()
+ password = getpass.getpass("SMPL-X password: ")
+ if username and password:
+ with tempfile.TemporaryDirectory(prefix="fdanyone-smplx-") as temporary_dir:
+ archive = Path(temporary_dir) / "models_smplx_v1_1.zip"
+ try:
+ _download_official(username, password, archive)
+ installed = install_smplx(archive, model_dir, gvhmr_root)
+ except AssetError as exc:
+ print(f"Automatic download was unavailable: {exc}")
+ else:
+ return {"installed": str(installed)}
+ return _prompt_for_archive(model_dir, gvhmr_root)
+
+
+def ensure_smplx(
+ model_dir: str | Path = "models",
+ gvhmr_root: str | Path = "third_party/GVHMR",
+) -> Path:
+ """Install SMPL-X interactively on first use, without blocking jobs."""
+
+ require_gvhmr_checkout(gvhmr_root)
+ target = Path(model_dir).expanduser().resolve() / SMPLX_MODEL
+ if target.is_file():
+ create_classic_gvhmr_links(model_dir, gvhmr_root, require_models=False, require_smplx=True)
+ return target
+
+ if not getattr(sys.stdin, "isatty", lambda: False)():
+ raise AssetError(
+ f"SMPL-X is not installed at {target}, and inference has no interactive terminal. "
+ "Run `python scripts/download_smplx.py` before starting this job."
+ )
+
+ print("SMPL-X is required and has not been installed; starting its licensed setup.")
+ result = download_smplx(model_dir=str(model_dir), gvhmr_root=str(gvhmr_root))
+ if result is None or not target.is_file():
+ raise AssetError("SMPL-X setup was cancelled; inference cannot continue.")
+ LOGGER.info("SMPL-X installed; continuing inference")
+ return target
diff --git a/fdanyone/errors.py b/fdanyone/errors.py
new file mode 100644
index 0000000..3093eff
--- /dev/null
+++ b/fdanyone/errors.py
@@ -0,0 +1,17 @@
+"""Project-specific errors with actionable user-facing messages."""
+
+
+class FourDAnyoneError(RuntimeError):
+ """Base class for expected pipeline failures."""
+
+
+class ConfigurationError(FourDAnyoneError):
+ """Raised when a frozen inference contract is violated."""
+
+
+class AssetError(FourDAnyoneError):
+ """Raised when a model or gated asset is missing or invalid."""
+
+
+class VideoContractError(FourDAnyoneError):
+ """Raised when an input cannot provide the canonical 121-frame clip."""
diff --git a/fdanyone/foreground.py b/fdanyone/foreground.py
new file mode 100644
index 0000000..5ea44de
--- /dev/null
+++ b/fdanyone/foreground.py
@@ -0,0 +1,65 @@
+"""Pinned BiRefNet inference over the canonical source clip."""
+
+from __future__ import annotations
+
+import gc
+from pathlib import Path
+
+import numpy as np
+from PIL import Image
+
+from fdanyone.config import FOREGROUND
+
+
+def predict_foreground_masks(
+ frames: tuple[np.ndarray, ...],
+ model_path: str | Path,
+ device: str,
+ *,
+ batch_size: int = FOREGROUND.batch_size,
+) -> np.ndarray:
+ """Return full-raster 8-bit foreground masks for the canonical clip."""
+
+ import torch
+ from torchvision import transforms
+ from torchvision.transforms.functional import to_pil_image
+ from transformers import AutoModelForImageSegmentation
+
+ if not frames:
+ raise ValueError("Foreground inference requires at least one frame.")
+ if batch_size <= 0:
+ raise ValueError("batch_size must be positive.")
+ shape = frames[0].shape
+ if any(frame.dtype != np.uint8 or frame.shape != shape for frame in frames):
+ raise ValueError("Foreground frames must share one RGB uint8 raster.")
+
+ model = AutoModelForImageSegmentation.from_pretrained(
+ str(Path(model_path).expanduser().resolve()),
+ local_files_only=True,
+ trust_remote_code=True,
+ )
+ model = model.eval().half().to(device)
+ transform = transforms.Compose(
+ [
+ transforms.Resize(FOREGROUND.image_size),
+ transforms.ToTensor(),
+ transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
+ ]
+ )
+ output: list[np.ndarray] = []
+ try:
+ for start in range(0, len(frames), batch_size):
+ images = [Image.fromarray(frame, mode="RGB") for frame in frames[start : start + batch_size]]
+ inputs = torch.stack([transform(image) for image in images]).to(device=device, dtype=torch.float16)
+ with torch.inference_mode():
+ predictions = model(inputs)[-1].sigmoid().cpu()
+ for image, prediction in zip(images, predictions, strict=True):
+ mask = to_pil_image(prediction).resize(image.size).convert("L")
+ output.append(np.asarray(mask, dtype=np.uint8).copy())
+ del inputs, predictions
+ finally:
+ del model
+ gc.collect()
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+ return np.stack(output)
diff --git a/fdanyone/geometry/__init__.py b/fdanyone/geometry/__init__.py
new file mode 100644
index 0000000..7c99e8a
--- /dev/null
+++ b/fdanyone/geometry/__init__.py
@@ -0,0 +1 @@
+"""Camera and crop geometry."""
diff --git a/fdanyone/geometry/cameras.py b/fdanyone/geometry/cameras.py
new file mode 100644
index 0000000..8f91e06
--- /dev/null
+++ b/fdanyone/geometry/cameras.py
@@ -0,0 +1,228 @@
+"""Canonical uniform camera rings and camera serialization."""
+
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass
+
+import numpy as np
+
+from fdanyone.config import CAMERA, FRAMING, CameraConfig
+
+WORLD_FRAME = {
+ "name": "canonical_human_world",
+ "handedness": "right",
+ "axes": {
+ "x": "right in the source-facing ring camera (the subject's anatomical left)",
+ "y": "up",
+ "z": "front; the subject initially faces +z and the source-facing camera lies on the +z side",
+ },
+ "origin": "initial root projected to the ground plane",
+ "units": "meters",
+}
+
+CAMERA_FRAME = {
+ "name": "opencv_camera",
+ "handedness": "right",
+ "axes": {"x": "image right", "y": "image down", "z": "forward from camera into the scene"},
+ "matrix_convention": "column vectors: x_camera = world_to_camera @ x_world_homogeneous",
+ "intrinsics_convention": "pixels with origin at the top-left",
+}
+
+
+def _normalize(vector: np.ndarray) -> np.ndarray:
+ norm = float(np.linalg.norm(vector))
+ if norm <= 1e-12:
+ raise ValueError("Cannot normalize a zero-length direction vector.")
+ return vector / norm
+
+
+def look_at_pytorch3d(position: np.ndarray, target: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+ """Return the column-vector w2c used by PyTorch3D's look_at_rotation."""
+
+ position = np.asarray(position, dtype=np.float64)
+ target = np.asarray(target, dtype=np.float64)
+ up = np.array([0.0, 1.0, 0.0], dtype=np.float64)
+ z_axis = _normalize(target - position)
+ x_axis = _normalize(np.cross(up, z_axis))
+ y_axis = _normalize(np.cross(z_axis, x_axis))
+ rotation = np.stack([x_axis, y_axis, z_axis], axis=0)
+ translation = -(rotation @ position)
+ return rotation, translation
+
+
+def pytorch3d_to_opencv(rotation: np.ndarray, translation: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+ rotation_cv = np.asarray(rotation, dtype=np.float64).copy()
+ translation_cv = np.asarray(translation, dtype=np.float64).copy()
+ rotation_cv[:2] *= -1.0
+ translation_cv[:2] *= -1.0
+ return rotation_cv, translation_cv
+
+
+def homogeneous_w2c(rotation: np.ndarray, translation: np.ndarray) -> np.ndarray:
+ matrix = np.eye(4, dtype=np.float64)
+ matrix[:3, :3] = rotation
+ matrix[:3, 3] = translation
+ return matrix
+
+
+@dataclass(frozen=True)
+class Camera:
+ camera_id: int
+ layer_index: int
+ yaw_degrees: float
+ azimuth_degrees: float
+ pitch_degrees: float
+ position: tuple[float, float, float]
+ K: tuple[tuple[float, float, float], ...]
+ world_to_camera: tuple[tuple[float, float, float, float], ...]
+ camera_to_world: tuple[tuple[float, float, float, float], ...]
+ image_width: int
+ image_height: int
+
+ def to_dict(self) -> dict:
+ return asdict(self)
+
+
+def reference_intrinsics(
+ image_height: int,
+ image_width: int,
+ max_render_height: int = 1280,
+ *,
+ focal_normalized: float = FRAMING.reference_focal_normalized,
+) -> np.ndarray:
+ """Reproduce the renderer's downscale-then-rescale intrinsic construction."""
+
+ divisor = 2
+ while image_height / divisor > max_render_height:
+ divisor += 1
+ render_height = image_height // divisor
+ render_width = image_width // divisor
+ scale = image_height / render_height
+ if focal_normalized <= 0:
+ raise ValueError("focal_normalized must be positive.")
+ focal = focal_normalized * render_height
+ intrinsic = np.array(
+ [[focal, 0.0, render_width / 2.0], [0.0, focal, render_height / 2.0], [0.0, 0.0, 1.0]],
+ dtype=np.float64,
+ )
+ intrinsic *= scale
+ intrinsic[2, 2] = 1.0
+ return intrinsic
+
+
+def camera_ring(
+ *,
+ center: np.ndarray,
+ front_direction: np.ndarray,
+ K: np.ndarray,
+ image_height: int,
+ image_width: int,
+ radius: float = FRAMING.reference_radius,
+ target_height: float = FRAMING.reference_target_height,
+ spec: CameraConfig = CAMERA,
+ start_yaw_degrees: float = 0.0,
+ yaw_span_degrees: float = 360.0,
+ layer_index: int = 0,
+ camera_id_offset: int = 0,
+) -> tuple[Camera, ...]:
+ center = np.asarray(center, dtype=np.float64).copy()
+ front_direction = np.asarray(front_direction, dtype=np.float64)
+ front_xz = front_direction[[0, 2]]
+ front_azimuth = np.arctan2(front_xz[1], front_xz[0])
+ # Relative yaw zero faces the person's front; start_yaw chooses where the
+ # first camera lies and IDs then advance uniformly through the span.
+ azimuth_start = front_azimuth + np.pi + np.deg2rad(start_yaw_degrees)
+ target = center.copy()
+ if radius <= 0:
+ raise ValueError("radius must be positive.")
+ target[1] = target_height
+ camera_height = target_height + radius * np.tan(np.deg2rad(spec.pitch_degrees))
+
+ cameras: list[Camera] = []
+ yaw_span_radians = np.deg2rad(yaw_span_degrees)
+ for view_index in range(spec.count):
+ camera_id = camera_id_offset + view_index
+ yaw_degrees = start_yaw_degrees + view_index / spec.count * yaw_span_degrees
+ azimuth = azimuth_start + view_index / spec.count * yaw_span_radians
+ position = np.array(
+ [
+ center[0] + radius * np.cos(azimuth),
+ camera_height,
+ center[2] + radius * np.sin(azimuth),
+ ],
+ dtype=np.float64,
+ )
+ rotation_p3d, translation_p3d = look_at_pytorch3d(position, target)
+ rotation_cv, translation_cv = pytorch3d_to_opencv(rotation_p3d, translation_p3d)
+ w2c = homogeneous_w2c(rotation_cv, translation_cv)
+ c2w = np.linalg.inv(w2c)
+ cameras.append(
+ Camera(
+ camera_id=camera_id,
+ layer_index=layer_index,
+ yaw_degrees=float(yaw_degrees),
+ azimuth_degrees=float(np.rad2deg(azimuth) % 360.0),
+ pitch_degrees=spec.pitch_degrees,
+ position=tuple(float(value) for value in position),
+ K=tuple(tuple(float(value) for value in row) for row in K),
+ world_to_camera=tuple(tuple(float(value) for value in row) for row in w2c),
+ camera_to_world=tuple(tuple(float(value) for value in row) for row in c2w),
+ image_width=image_width,
+ image_height=image_height,
+ )
+ )
+ return tuple(cameras)
+
+
+def camera_grid(
+ *,
+ center: np.ndarray,
+ front_direction: np.ndarray,
+ K: np.ndarray,
+ image_height: int,
+ image_width: int,
+ views_per_layer: int,
+ layer_pitches: tuple[int, ...],
+ start_yaw: int,
+ yaw_span: int,
+ radius: float = FRAMING.reference_radius,
+ target_height: float = FRAMING.reference_target_height,
+) -> tuple[Camera, ...]:
+ """Create a layer-major target camera grid."""
+
+ return tuple(
+ camera
+ for layer_index, pitch in enumerate(layer_pitches)
+ for camera in camera_ring(
+ center=center,
+ front_direction=front_direction,
+ K=K,
+ image_height=image_height,
+ image_width=image_width,
+ radius=radius,
+ target_height=target_height,
+ spec=CameraConfig(count=views_per_layer, pitch_degrees=float(pitch)),
+ start_yaw_degrees=float(start_yaw),
+ yaw_span_degrees=float(yaw_span),
+ layer_index=layer_index,
+ camera_id_offset=layer_index * views_per_layer,
+ )
+ )
+
+
+def project_points(points_world: np.ndarray, camera: Camera) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ points = np.asarray(points_world, dtype=np.float64)
+ w2c = np.asarray(camera.world_to_camera)
+ K = np.asarray(camera.K)
+ points_h = np.concatenate([points, np.ones((points.shape[0], 1), dtype=points.dtype)], axis=1)
+ points_camera = points_h @ w2c[:3].T
+ homogeneous = points_camera @ K.T
+ xy = homogeneous[:, :2] / np.maximum(homogeneous[:, 2:3], 1e-8)
+ valid = (
+ (points_camera[:, 2] > 0.0)
+ & (xy[:, 0] >= 0.0)
+ & (xy[:, 0] < camera.image_width)
+ & (xy[:, 1] >= 0.0)
+ & (xy[:, 1] < camera.image_height)
+ )
+ return xy.astype(np.float32), points_camera[:, 2].astype(np.float32), valid
diff --git a/fdanyone/geometry/crop.py b/fdanyone/geometry/crop.py
new file mode 100644
index 0000000..08f501c
--- /dev/null
+++ b/fdanyone/geometry/crop.py
@@ -0,0 +1,144 @@
+"""Deterministic source-mask and center-aspect crops."""
+
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass
+
+import numpy as np
+
+
+@dataclass(frozen=True)
+class Crop:
+ top: int
+ left: int
+ height: int
+ width: int
+ original_height: int
+ original_width: int
+ output_height: int
+ output_width: int
+
+ @property
+ def scale_y(self) -> float:
+ return self.output_height / self.height
+
+ @property
+ def scale_x(self) -> float:
+ return self.output_width / self.width
+
+
+def center_crop(image_height: int, image_width: int, output_height: int, output_width: int) -> Crop:
+ aspect_ratio = output_height / output_width
+ if image_height / image_width >= aspect_ratio:
+ crop_width = image_width
+ crop_height = min(image_height, max(1, int(round(crop_width * aspect_ratio))))
+ else:
+ crop_height = image_height
+ crop_width = min(image_width, max(1, int(round(crop_height / aspect_ratio))))
+ return Crop(
+ top=(image_height - crop_height) // 2,
+ left=(image_width - crop_width) // 2,
+ height=crop_height,
+ width=crop_width,
+ original_height=image_height,
+ original_width=image_width,
+ output_height=output_height,
+ output_width=output_width,
+ )
+
+
+def mask_bounds(masks: np.ndarray, threshold: float) -> tuple[int, int, int, int] | None:
+ """Return union bounds as left/top inclusive and right/bottom exclusive."""
+
+ values = np.asarray(masks)
+ if values.ndim == 4 and values.shape[1] == 1:
+ values = values[:, 0]
+ if values.ndim not in (2, 3):
+ raise ValueError(f"Expected masks [H,W] or [F,H,W], got {values.shape}.")
+ if not 0 <= threshold <= 1:
+ raise ValueError("threshold must be in [0, 1].")
+ cutoff = threshold * 255.0 if np.issubdtype(values.dtype, np.integer) else threshold
+ union = values.max(axis=0) if values.ndim == 3 else values
+ foreground = np.asarray(union > cutoff)
+ rows = np.flatnonzero(foreground.any(axis=1))
+ columns = np.flatnonzero(foreground.any(axis=0))
+ if rows.size == 0 or columns.size == 0:
+ return None
+ return int(columns[0]), int(rows[0]), int(columns[-1] + 1), int(rows[-1] + 1)
+
+
+def expand_bounds(bounds, margins, image_height: int, image_width: int):
+ if bounds is None:
+ return None
+ xmin, ymin, xmax, ymax = (float(value) for value in bounds)
+ top, right, bottom, left = (float(value) for value in margins)
+ box_width, box_height = xmax - xmin, ymax - ymin
+ return (
+ max(0, int(math.floor(xmin - left * box_width))),
+ max(0, int(math.floor(ymin - top * box_height))),
+ min(image_width, int(math.ceil(xmax + right * box_width))),
+ min(image_height, int(math.ceil(ymax + bottom * box_height))),
+ )
+
+
+def crop_from_bounds(
+ *,
+ bounds: tuple[int, int, int, int] | None,
+ image_height: int,
+ image_width: int,
+ output_height: int,
+ output_width: int,
+ margins: tuple[float, float, float, float],
+ allow_upscale: bool = True,
+) -> Crop:
+ bounds = expand_bounds(bounds, margins, image_height, image_width)
+ if bounds is None:
+ return center_crop(image_height, image_width, output_height, output_width)
+
+ xmin, ymin, xmax, ymax = bounds
+ required_height = float(ymax - ymin)
+ required_width = float(xmax - xmin)
+ if not allow_upscale:
+ required_height = max(required_height, min(output_height, image_height))
+ required_width = max(required_width, min(output_width, image_width))
+
+ aspect_ratio = output_height / output_width
+ crop_height = max(required_height, required_width * aspect_ratio)
+ crop_width = crop_height / aspect_ratio
+ max_height = min(float(image_height), float(image_width) * aspect_ratio)
+ crop_height = max(1, min(image_height, int(math.ceil(min(crop_height, max_height)))))
+ crop_width = max(1, min(image_width, int(math.ceil(crop_height / aspect_ratio))))
+
+ left_low = max(0.0, xmax - crop_width)
+ left_high = min(float(xmin), image_width - crop_width)
+ top_low = max(0.0, ymax - crop_height)
+ top_high = min(float(ymin), image_height - crop_height)
+ if left_low <= left_high:
+ left = int(math.floor((left_low + left_high) / 2.0))
+ else:
+ left = max(0, min(int(math.floor((xmin + xmax - crop_width) / 2.0)), image_width - crop_width))
+ if top_low <= top_high:
+ top = int(math.floor((top_low + top_high) / 2.0))
+ else:
+ top = max(0, min(int(math.floor((ymin + ymax - crop_height) / 2.0)), image_height - crop_height))
+ return Crop(
+ top=top,
+ left=left,
+ height=crop_height,
+ width=crop_width,
+ original_height=image_height,
+ original_width=image_width,
+ output_height=output_height,
+ output_width=output_width,
+ )
+
+
+def transform_intrinsics(K: np.ndarray, crop: Crop) -> np.ndarray:
+ transformed = np.asarray(K, dtype=np.float64).copy()
+ transformed[0, 2] -= crop.left
+ transformed[1, 2] -= crop.top
+ transformed[0] *= crop.scale_x
+ transformed[1] *= crop.scale_y
+ transformed[2, 2] = 1.0
+ return transformed
diff --git a/fdanyone/geometry/framing.py b/fdanyone/geometry/framing.py
new file mode 100644
index 0000000..fad54a1
--- /dev/null
+++ b/fdanyone/geometry/framing.py
@@ -0,0 +1,579 @@
+"""Source-aware static-camera framing for the canonical camera ring."""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Sequence
+from dataclasses import asdict, dataclass
+
+import cv2
+import numpy as np
+
+from fdanyone.config import FRAMING, FramingConfig
+from fdanyone.geometry.cameras import Camera
+
+FULL_BODY_BOTTOM = 0.82
+CLOSE_UP_BOTTOM = 0.35
+HALF_BODY_BOTTOM = 0.42
+_ANATOMY_COORDS = (0.0, 0.18, 0.45, 0.72, 0.94, 1.0)
+_FINGER_TOKENS = ("thumb", "index", "middle", "ring", "pinky")
+
+
+@dataclass(frozen=True)
+class InputFraming:
+ label: str
+ visible_body_bottom: float
+ visible_body_bottom_p50: float
+ visible_body_bottom_p80: float
+ visible_height_ratio: float
+ wrist_out_ratio_x: float
+ wrist_out_ratio_y: float
+ valid_frame_ratio: float
+ torso_valid_ratio: float
+ projection_alignment_error_ratio: float | None
+ confidence: float
+
+ def to_dict(self) -> dict[str, object]:
+ return {**asdict(self), "fmask_used": True}
+
+
+@dataclass(frozen=True)
+class AdaptiveThresholds:
+ closeup_strength: float
+ height_target_ratio: float
+ height_percentile: float
+ width_target_ratio: float
+ width_percentile: float
+
+
+@dataclass(frozen=True)
+class RadiusSolve:
+ radius: float
+ height_ratio: float
+ width_ratio: float
+ bound: str | None
+ limiting_constraint: str
+
+
+@dataclass(frozen=True)
+class FocalSolve:
+ focal_normalized: float
+ height_ratio: float
+ width_ratio: float
+ bound: str | None
+ limiting_constraint: str
+
+
+@dataclass(frozen=True)
+class SequenceFraming:
+ radius: float
+ target_height: float
+ focal_normalized: float
+ input: InputFraming
+ input_applied: bool
+ radius_solve: RadiusSolve
+ adaptive_thresholds: AdaptiveThresholds | None = None
+ focal_solve: FocalSolve | None = None
+ cutoff_ratio: float | None = None
+ target_bound: str | None = None
+
+ def to_dict(self) -> dict[str, object]:
+ return {
+ "method": "sequence_input_profile_static_radius_target_focal",
+ "radius": self.radius,
+ "target_height": self.target_height,
+ "focal_normalized": self.focal_normalized,
+ "input_framing_applied": self.input_applied,
+ "input_framing": self.input.to_dict(),
+ "adaptive_thresholds": (None if self.adaptive_thresholds is None else asdict(self.adaptive_thresholds)),
+ "radius_solver": asdict(self.radius_solve),
+ "focal_solver": None if self.focal_solve is None else asdict(self.focal_solve),
+ "cutoff_ratio": self.cutoff_ratio,
+ "target_bound": self.target_bound,
+ }
+
+
+def _name_index(names: Sequence[str]) -> dict[str, int]:
+ normalized = [str(name).strip().lower().replace("_", "-") for name in names]
+ mapping = dict(zip(normalized, range(len(normalized)), strict=True))
+ if len(mapping) != len(normalized):
+ raise ValueError("Keypoint names must be unique after normalization.")
+ return mapping
+
+
+def _required_indices(names: Sequence[str]) -> dict[str, int]:
+ required = ["nose", "left-eye", "right-eye", "left-ear", "right-ear", "neck"]
+ for side in ("left", "right"):
+ required.extend(
+ f"{side}-{part}"
+ for part in (
+ "shoulder",
+ "hip",
+ "knee",
+ "ankle",
+ "big-toe-tip",
+ "small-toe-tip",
+ "heel",
+ )
+ )
+ mapping = _name_index(names)
+ missing = [name for name in required if name not in mapping]
+ if missing:
+ raise ValueError(f"Missing framing keypoints: {missing}.")
+ return {name: mapping[name] for name in required}
+
+
+def _sample_path(anchors: Sequence[np.ndarray], samples_per_segment: int) -> tuple[np.ndarray, np.ndarray]:
+ point_chunks: list[np.ndarray] = []
+ coordinate_chunks: list[np.ndarray] = []
+ for index, (start, end) in enumerate(zip(anchors, anchors[1:], strict=False)):
+ endpoint = index == len(anchors) - 2
+ weights = np.linspace(
+ 0.0,
+ 1.0,
+ samples_per_segment + int(endpoint),
+ endpoint=endpoint,
+ dtype=np.float64,
+ )
+ point_chunks.append(start[:, None] * (1.0 - weights[None, :, None]) + end[:, None] * weights[None, :, None])
+ coordinate_chunks.append(_ANATOMY_COORDS[index] * (1.0 - weights) + _ANATOMY_COORDS[index + 1] * weights)
+ return np.concatenate(point_chunks, axis=1), np.concatenate(coordinate_chunks)
+
+
+def anatomy_samples(
+ keypoints: np.ndarray,
+ names: Sequence[str],
+ samples_per_segment: int = 8,
+) -> tuple[np.ndarray, np.ndarray]:
+ points = np.asarray(keypoints, dtype=np.float64)
+ if points.ndim != 3 or points.shape[1:] != (len(names), 3) or not np.isfinite(points).all():
+ raise ValueError(f"Expected finite keypoints [frames,{len(names)},3], got {points.shape}.")
+ if samples_per_segment < 2:
+ raise ValueError("samples_per_segment must be at least two.")
+ ids = _required_indices(names)
+ face = np.mean(
+ points[:, [ids[name] for name in ("nose", "left-eye", "right-eye", "left-ear", "right-ear")]], axis=1
+ )
+ head_top = face + 0.65 * (face - points[:, ids["neck"]])
+ paths: list[np.ndarray] = []
+ coordinates: list[np.ndarray] = []
+ for side in ("left", "right"):
+ foot = np.mean(
+ points[:, [ids[f"{side}-{part}"] for part in ("big-toe-tip", "small-toe-tip", "heel")]],
+ axis=1,
+ )
+ path, path_coordinates = _sample_path(
+ [
+ head_top,
+ points[:, ids[f"{side}-shoulder"]],
+ points[:, ids[f"{side}-hip"]],
+ points[:, ids[f"{side}-knee"]],
+ points[:, ids[f"{side}-ankle"]],
+ foot,
+ ],
+ samples_per_segment,
+ )
+ paths.append(path)
+ coordinates.append(path_coordinates)
+ return np.concatenate(paths, axis=1), np.concatenate(coordinates)
+
+
+def project_incam(points: np.ndarray, intrinsics: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+ points = np.asarray(points, dtype=np.float64)
+ cameras = np.asarray(intrinsics, dtype=np.float64)
+ if points.ndim != 3 or points.shape[-1] != 3:
+ raise ValueError(f"Expected points [frames,keypoints,3], got {points.shape}.")
+ if cameras.shape == (3, 3):
+ cameras = np.broadcast_to(cameras, (points.shape[0], 3, 3))
+ if cameras.shape == (1, 3, 3):
+ cameras = np.broadcast_to(cameras, (points.shape[0], 3, 3))
+ if cameras.shape != (points.shape[0], 3, 3):
+ raise ValueError(f"Expected intrinsics [{points.shape[0]},3,3], got {cameras.shape}.")
+ if not np.isfinite(points).all() or not np.isfinite(cameras).all():
+ raise ValueError("Projection inputs must be finite.")
+ homogeneous = np.einsum("fij,fkj->fki", cameras, points)
+ depths = points[..., 2]
+ xy = homogeneous[..., :2] / np.maximum(homogeneous[..., 2:3], 1e-8)
+ return xy, depths
+
+
+def _inside(xy: np.ndarray, depths: np.ndarray, width: int, height: int) -> np.ndarray:
+ return (depths > 1e-6) & (xy[..., 0] >= 0) & (xy[..., 0] < width) & (xy[..., 1] >= 0) & (xy[..., 1] < height)
+
+
+def _mask_support(xy: np.ndarray, inside: np.ndarray, masks: np.ndarray) -> np.ndarray:
+ masks = np.asarray(masks)
+ if masks.ndim != 3 or masks.shape[0] != xy.shape[0]:
+ raise ValueError(f"Expected masks [frames,height,width], got {masks.shape}.")
+ height, width = masks.shape[1:]
+ patch_radius = max(3, int(round(min(width, height) * 0.005)))
+ kernel = np.ones((2 * patch_radius + 1,) * 2, dtype=np.uint8)
+ support = np.zeros_like(inside)
+ for frame_index, mask in enumerate(masks):
+ dilated = cv2.dilate((mask >= 64).astype(np.uint8), kernel)
+ candidates = np.flatnonzero(inside[frame_index])
+ if candidates.size:
+ pixels = np.rint(xy[frame_index, candidates]).astype(np.int64)
+ pixels[:, 0] = np.clip(pixels[:, 0], 0, width - 1)
+ pixels[:, 1] = np.clip(pixels[:, 1], 0, height - 1)
+ support[frame_index, candidates] = dilated[pixels[:, 1], pixels[:, 0]] > 0
+ return support
+
+
+def _torso_valid(vitpose: np.ndarray, width: int, height: int) -> np.ndarray:
+ detector = np.asarray(vitpose, dtype=np.float64)
+ if detector.ndim != 3 or detector.shape[1:] != (17, 3):
+ raise ValueError(f"Expected VitPose [frames,17,3], got {detector.shape}.")
+ valid = (
+ (detector[..., 2] >= 0.3)
+ & (detector[..., 0] >= 0)
+ & (detector[..., 0] < width)
+ & (detector[..., 1] >= 0)
+ & (detector[..., 1] < height)
+ )
+ shoulders = valid[:, 5] & valid[:, 6]
+ torso = shoulders & (valid[:, 11] | valid[:, 12])
+ return shoulders if float(torso.mean()) < 0.5 else torso
+
+
+def _alignment_error(
+ projected: np.ndarray,
+ names: Sequence[str],
+ vitpose: np.ndarray,
+ width: int,
+ height: int,
+) -> float | None:
+ coco_names = (
+ "nose",
+ "left-eye",
+ "right-eye",
+ "left-ear",
+ "right-ear",
+ "left-shoulder",
+ "right-shoulder",
+ "left-elbow",
+ "right-elbow",
+ "left-wrist",
+ "right-wrist",
+ "left-hip",
+ "right-hip",
+ "left-knee",
+ "right-knee",
+ "left-ankle",
+ "right-ankle",
+ )
+ mapping = _name_index(names)
+ if any(name not in mapping for name in coco_names):
+ return None
+ detector = np.asarray(vitpose, dtype=np.float64)
+ valid = (
+ (detector[..., 2] >= 0.3)
+ & (detector[..., 0] >= 0)
+ & (detector[..., 0] < width)
+ & (detector[..., 1] >= 0)
+ & (detector[..., 1] < height)
+ )
+ if not valid.any():
+ return None
+ errors = np.linalg.norm(projected[:, [mapping[name] for name in coco_names]] - detector[..., :2], axis=-1)
+ return float(np.median(errors[valid]) / height)
+
+
+def analyze_input_framing(
+ incam_keypoints: np.ndarray,
+ names: Sequence[str],
+ intrinsics: np.ndarray,
+ vitpose: np.ndarray,
+ masks: np.ndarray,
+) -> InputFraming:
+ masks = np.asarray(masks)
+ if masks.ndim != 3:
+ raise ValueError(f"Expected masks [frames,height,width], got {masks.shape}.")
+ frame_count, height, width = masks.shape
+ if np.asarray(incam_keypoints).shape[0] != frame_count or np.asarray(vitpose).shape[0] != frame_count:
+ raise ValueError("Input-framing arrays do not share one frame count.")
+
+ anatomy, coordinates = anatomy_samples(incam_keypoints, names)
+ anatomy_xy, anatomy_depth = project_incam(anatomy, intrinsics)
+ support = _mask_support(anatomy_xy, _inside(anatomy_xy, anatomy_depth, width, height), masks)
+ torso_valid = _torso_valid(vitpose, width, height)
+ bottoms = np.full(frame_count, np.nan)
+ visible_heights = np.full(frame_count, np.nan)
+ for frame_index in range(frame_count):
+ valid = np.flatnonzero(support[frame_index])
+ if valid.size and torso_valid[frame_index]:
+ bottoms[frame_index] = float(coordinates[valid].max())
+ y = anatomy_xy[frame_index, valid, 1]
+ visible_heights[frame_index] = float(np.clip((y.max() - y.min()) / height, 0.0, 1.0))
+ valid_frames = np.isfinite(bottoms)
+ if not valid_frames.any():
+ raise ValueError("No valid frames for input framing analysis.")
+
+ projected, depths = project_incam(incam_keypoints, intrinsics)
+ mapping = _name_index(names)
+ wrists = [mapping[name] for name in ("left-wrist", "right-wrist")]
+ wrist_xy = projected[:, wrists]
+ wrist_depth = depths[:, wrists]
+ wrist_out_x = (wrist_depth <= 1e-6) | (wrist_xy[..., 0] < 0) | (wrist_xy[..., 0] >= width)
+ wrist_out_y = (wrist_depth <= 1e-6) | (wrist_xy[..., 1] < 0) | (wrist_xy[..., 1] >= height)
+ alignment = _alignment_error(projected, names, vitpose, width, height)
+ valid_ratio = float(valid_frames.mean())
+ torso_ratio = float(torso_valid.mean())
+ alignment_score = 0.6 if alignment is None else float(np.clip(1.0 - alignment / 0.15, 0.0, 1.0))
+ confidence = float(np.clip(0.45 * valid_ratio + 0.35 * torso_ratio + 0.20 * alignment_score, 0.0, 1.0))
+ bottom = float(np.percentile(bottoms[valid_frames], 20.0))
+ if bottom >= FULL_BODY_BOTTOM:
+ label = "full_body"
+ elif bottom >= HALF_BODY_BOTTOM:
+ label = "half_body"
+ else:
+ label = "close_up"
+ finite_heights = visible_heights[np.isfinite(visible_heights)]
+ return InputFraming(
+ label=label,
+ visible_body_bottom=round(bottom, 6),
+ visible_body_bottom_p50=round(float(np.percentile(bottoms[valid_frames], 50.0)), 6),
+ visible_body_bottom_p80=round(float(np.percentile(bottoms[valid_frames], 80.0)), 6),
+ visible_height_ratio=round(float(np.percentile(finite_heights, 50.0)) if finite_heights.size else 0.0, 6),
+ wrist_out_ratio_x=round(float(np.any(wrist_out_x, axis=1).mean()), 6),
+ wrist_out_ratio_y=round(float(np.any(wrist_out_y, axis=1).mean()), 6),
+ valid_frame_ratio=round(valid_ratio, 6),
+ torso_valid_ratio=round(torso_ratio, 6),
+ projection_alignment_error_ratio=None if alignment is None else round(alignment, 6),
+ confidence=round(confidence, 6),
+ )
+
+
+def _camera_coordinates(points: np.ndarray, cameras: Sequence[Camera]) -> np.ndarray:
+ world = np.asarray(points, dtype=np.float64)
+ points_h = np.concatenate([world, np.ones((*world.shape[:-1], 1), dtype=np.float64)], axis=-1)
+ w2c = np.asarray([camera.world_to_camera for camera in cameras], dtype=np.float64)
+ return np.einsum("cij,fkj->cfki", w2c[:, :3], points_h)
+
+
+def projected_axis_ratios(
+ points: np.ndarray,
+ cameras: Sequence[Camera],
+ focal_normalized: float,
+ *,
+ axis: int,
+ axis_scale: float = 1.0,
+) -> np.ndarray:
+ camera_points = _camera_coordinates(points, cameras)
+ depths = camera_points[..., 2]
+ coordinates = focal_normalized * camera_points[..., axis] / np.maximum(depths, 1e-6) / axis_scale
+ ratios = coordinates.max(axis=2) - coordinates.min(axis=2)
+ ratios[np.any(depths <= 1e-6, axis=2)] = np.inf
+ return ratios.max(axis=0)
+
+
+def _selected(points: np.ndarray, names: Sequence[str], *, exclude_hands: bool, exclude_fingers: bool) -> np.ndarray:
+ excluded = _FINGER_TOKENS
+ if exclude_hands:
+ excluded = ("wrist", *_FINGER_TOKENS)
+ elif not exclude_fingers:
+ excluded = ()
+ ids = [index for index, name in enumerate(names) if not any(token in name.lower() for token in excluded)]
+ return np.asarray(points)[:, ids]
+
+
+def solve_radius(
+ points: np.ndarray,
+ names: Sequence[str],
+ camera_factory: Callable[[float, float], Sequence[Camera]],
+ aspect_ratio: float,
+ spec: FramingConfig = FRAMING,
+) -> RadiusSolve:
+ height_points = _selected(points, names, exclude_hands=True, exclude_fingers=True)
+ width_points = _selected(points, names, exclude_hands=False, exclude_fingers=True)
+
+ def evaluate(radius: float) -> tuple[float, float, float]:
+ cameras = camera_factory(radius, spec.reference_target_height)
+ heights = projected_axis_ratios(height_points, cameras, spec.reference_focal_normalized, axis=1)
+ widths = projected_axis_ratios(
+ width_points, cameras, spec.reference_focal_normalized, axis=0, axis_scale=aspect_ratio
+ )
+ height = float(np.percentile(heights, spec.height_percentile))
+ width = float(np.percentile(widths, spec.width_percentile))
+ return max(height / spec.height_target_ratio, width / spec.width_target_ratio), height, width
+
+ def result(radius: float, values: tuple[float, float, float], bound: str | None) -> RadiusSolve:
+ _, height, width = values
+ limiting = "width" if width / spec.width_target_ratio > height / spec.height_target_ratio else "height"
+ return RadiusSolve(radius, height, width, bound, limiting)
+
+ lower_values = evaluate(spec.min_radius)
+ if lower_values[0] <= 1.0:
+ return result(spec.min_radius, lower_values, "min")
+ upper_values = evaluate(spec.max_radius)
+ if upper_values[0] > 1.0:
+ return result(spec.max_radius, upper_values, "max")
+ lower, upper, best = spec.min_radius, spec.max_radius, upper_values
+ for _ in range(24):
+ midpoint = (lower + upper) / 2
+ values = evaluate(midpoint)
+ if values[0] > 1.0:
+ lower = midpoint
+ else:
+ upper, best = midpoint, values
+ if values[0] <= 1.0 and abs(values[0] - 1.0) <= 1e-4:
+ break
+ return result(upper, best, None)
+
+
+def adaptive_thresholds(profile: InputFraming) -> AdaptiveThresholds:
+ strength = float(
+ np.clip((FULL_BODY_BOTTOM - profile.visible_body_bottom) / (FULL_BODY_BOTTOM - CLOSE_UP_BOTTOM), 0.0, 1.0)
+ )
+ return AdaptiveThresholds(strength, 0.80 + 0.12 * strength, 95.0, 0.90 + 0.20 * strength, 80.0 - 30.0 * strength)
+
+
+def _cutoff_points(points: np.ndarray, coordinates: np.ndarray, cutoff: float) -> np.ndarray:
+ path_length = points.shape[1] // 2
+ output = []
+ for offset in (0, path_length):
+ local_coordinates = coordinates[offset : offset + path_length]
+ local_points = points[:, offset : offset + path_length]
+ after = int(np.searchsorted(local_coordinates, cutoff, side="right"))
+ if after == 0:
+ output.append(local_points[:, 0])
+ elif after >= len(local_coordinates):
+ output.append(local_points[:, -1])
+ else:
+ before = after - 1
+ weight = (cutoff - local_coordinates[before]) / (local_coordinates[after] - local_coordinates[before])
+ output.append(local_points[:, before] * (1.0 - weight) + local_points[:, after] * weight)
+ return np.stack(output, axis=1)
+
+
+def _visible_anatomy(
+ points: np.ndarray,
+ names: Sequence[str],
+ bottom: float,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ samples, coordinates = anatomy_samples(points, names)
+ core = samples[:, coordinates <= bottom + 1e-8]
+ cutoff = _cutoff_points(samples, coordinates, bottom)
+ if not np.any(np.isclose(coordinates, bottom, atol=1e-8)):
+ core = np.concatenate([core, cutoff], axis=1)
+ mapping = _name_index(names)
+ arm_tokens = ("shoulder", "acromion", "elbow", "olecranon", "cubital-fossa", "wrist")
+ arms = np.asarray(points)[
+ :, [index for name, index in mapping.items() if any(token in name for token in arm_tokens)]
+ ]
+ return (
+ np.asarray(core, dtype=np.float32),
+ np.asarray(np.concatenate([core, arms], axis=1), dtype=np.float32),
+ np.asarray(cutoff, dtype=np.float32),
+ )
+
+
+def _solve_focal(
+ core: np.ndarray,
+ width_points: np.ndarray,
+ cameras: Sequence[Camera],
+ thresholds: AdaptiveThresholds,
+ aspect_ratio: float,
+ spec: FramingConfig,
+) -> tuple[FocalSolve, np.ndarray]:
+ unit_heights = projected_axis_ratios(core, cameras, 1.0, axis=1)
+ unit_widths = projected_axis_ratios(width_points, cameras, 1.0, axis=0, axis_scale=aspect_ratio)
+ unit_height = float(np.percentile(unit_heights, thresholds.height_percentile))
+ unit_width = float(np.percentile(unit_widths, thresholds.width_percentile))
+ height_focal = thresholds.height_target_ratio / unit_height
+ width_focal = thresholds.width_target_ratio / unit_width if unit_width > 0 else np.inf
+ unconstrained = min(height_focal, width_focal)
+ focal = float(np.clip(unconstrained, spec.reference_focal_normalized, spec.max_focal_normalized))
+ bound = (
+ "min"
+ if unconstrained < spec.reference_focal_normalized
+ else "max"
+ if unconstrained > spec.max_focal_normalized
+ else None
+ )
+ return (
+ FocalSolve(
+ focal,
+ float(np.percentile(unit_heights * focal, thresholds.height_percentile)),
+ float(np.percentile(unit_widths * focal, thresholds.width_percentile)),
+ bound,
+ "width" if width_focal < height_focal else "height",
+ ),
+ unit_heights,
+ )
+
+
+def _cutoff_ratio(points: np.ndarray, cameras: Sequence[Camera], focal: float, percentile: float) -> float:
+ camera_points = _camera_coordinates(points, cameras)
+ positions = 0.5 + focal * camera_points[..., 1] / np.maximum(camera_points[..., 2], 1e-6)
+ positions[camera_points[..., 2] <= 1e-6] = np.nan
+ return float(np.percentile(positions[np.isfinite(positions)], percentile))
+
+
+def solve_sequence_framing(
+ keypoints: np.ndarray,
+ names: Sequence[str],
+ profile: InputFraming,
+ camera_factory: Callable[[float, float], Sequence[Camera]],
+ aspect_ratio: float,
+ spec: FramingConfig = FRAMING,
+) -> SequenceFraming:
+ radius_result = solve_radius(keypoints, names, camera_factory, aspect_ratio, spec)
+ applied = profile.confidence >= spec.input_min_confidence
+ thresholds = adaptive_thresholds(profile) if applied else None
+ if not applied or thresholds is None or thresholds.closeup_strength <= 0:
+ return SequenceFraming(
+ radius_result.radius,
+ spec.reference_target_height,
+ spec.reference_focal_normalized,
+ profile,
+ applied,
+ radius_result,
+ thresholds,
+ )
+
+ core, width_points, cutoff = _visible_anatomy(keypoints, names, profile.visible_body_bottom)
+ centers = (core[..., 1].min(axis=1) + core[..., 1].max(axis=1)) / 2
+ anatomical_target = float(np.median(centers))
+ alignment = min(1.0, 2.0 * thresholds.closeup_strength)
+ initial_target = spec.reference_target_height * (1.0 - alignment) + anatomical_target * alignment
+
+ def evaluate(target_height: float) -> tuple[FocalSolve, float]:
+ cameras = camera_factory(radius_result.radius, target_height)
+ focal, _ = _solve_focal(core, width_points, cameras, thresholds, aspect_ratio, spec)
+ return focal, _cutoff_ratio(cutoff, cameras, focal.focal_normalized, spec.cutoff_percentile)
+
+ lower, upper = initial_target - 0.5, initial_target + 0.5
+ lower_value, upper_value = evaluate(lower), evaluate(upper)
+ increasing = upper_value[1] > lower_value[1]
+ if not min(lower_value[1], upper_value[1]) <= spec.cutoff_target_ratio <= max(lower_value[1], upper_value[1]):
+ if abs(lower_value[1] - spec.cutoff_target_ratio) <= abs(upper_value[1] - spec.cutoff_target_ratio):
+ target, value, target_bound = lower, lower_value, "min"
+ else:
+ target, value, target_bound = upper, upper_value, "max"
+ else:
+ target, value, target_bound = lower, lower_value, None
+ for _ in range(20):
+ midpoint = (lower + upper) / 2
+ candidate = evaluate(midpoint)
+ error = candidate[1] - spec.cutoff_target_ratio
+ if abs(error) < abs(value[1] - spec.cutoff_target_ratio):
+ target, value = midpoint, candidate
+ if abs(error) <= 1e-4:
+ break
+ if (error < 0) == increasing:
+ lower = midpoint
+ else:
+ upper = midpoint
+
+ return SequenceFraming(
+ radius_result.radius,
+ target,
+ value[0].focal_normalized,
+ profile,
+ True,
+ radius_result,
+ thresholds,
+ value[0],
+ value[1],
+ target_bound,
+ )
diff --git a/fdanyone/io.py b/fdanyone/io.py
new file mode 100644
index 0000000..1503c53
--- /dev/null
+++ b/fdanyone/io.py
@@ -0,0 +1,97 @@
+"""Filesystem helpers for crash-safe result publication."""
+
+from __future__ import annotations
+
+import errno
+import json
+import os
+import shutil
+import time
+import uuid
+from contextlib import AbstractContextManager
+from pathlib import Path
+
+from fdanyone.errors import FourDAnyoneError
+
+_RETRYABLE_TREE_ERRORS = {errno.EBUSY, errno.ENOTEMPTY, errno.ESTALE}
+
+
+def write_json(path: str | Path, value: object, *, sort_keys: bool = True) -> None:
+ target = Path(path)
+ target.parent.mkdir(parents=True, exist_ok=True)
+ temporary = target.with_name(f".{target.name}.{uuid.uuid4().hex}.tmp")
+ temporary.write_text(json.dumps(value, indent=2, sort_keys=sort_keys) + "\n")
+ os.replace(temporary, target)
+
+
+def remove_tree(
+ path: str | Path,
+ *,
+ attempts: int = 8,
+ initial_delay_seconds: float = 0.1,
+ ignore_errors: bool = False,
+) -> None:
+ """Remove a tree, tolerating short directory-entry lag on network filesystems."""
+
+ target = Path(path)
+ if attempts <= 0:
+ raise ValueError(f"attempts must be positive, got {attempts}.")
+ for attempt in range(attempts):
+ try:
+ shutil.rmtree(target)
+ return
+ except FileNotFoundError:
+ return
+ except OSError as exc:
+ retryable = exc.errno in _RETRYABLE_TREE_ERRORS and attempt + 1 < attempts
+ if not retryable:
+ if ignore_errors:
+ return
+ raise
+ time.sleep(initial_delay_seconds * (2**attempt))
+
+
+class AtomicResultDirectory(AbstractContextManager[Path]):
+ """Build beside the destination and rename only after all validation passes."""
+
+ def __init__(self, destination: str | Path):
+ expanded = Path(destination).expanduser()
+ # Resolve the parent for a stable absolute location, but preserve the
+ # leaf itself so a dangling output symlink cannot be followed and
+ # mistaken for a nonexistent destination.
+ self.destination = expanded.parent.resolve() / expanded.name
+ self.working = self.destination.with_name(f".{self.destination.name}.work-{uuid.uuid4().hex[:10]}")
+ self._committed = False
+
+ def _destination_exists(self) -> bool:
+ return os.path.lexists(self.destination)
+
+ def __enter__(self) -> Path:
+ if self._destination_exists():
+ raise FourDAnyoneError(
+ f"Output directory already exists: {self.destination}. Choose a new directory to avoid mixed runs."
+ )
+ self.working.mkdir(parents=True)
+ return self.working
+
+ def commit(self) -> Path:
+ if self._committed:
+ return self.destination
+ if self._destination_exists():
+ raise FourDAnyoneError(
+ f"Output directory appeared during inference: {self.destination}. Refusing to overwrite it."
+ )
+ os.replace(self.working, self.destination)
+ self._committed = True
+ return self.destination
+
+ def __exit__(self, exc_type, exc_value, traceback) -> bool:
+ if exc_type is None:
+ try:
+ self.commit()
+ except BaseException:
+ remove_tree(self.working, ignore_errors=True)
+ raise
+ else:
+ remove_tree(self.working, ignore_errors=True)
+ return False
diff --git a/fdanyone/model/__init__.py b/fdanyone/model/__init__.py
new file mode 100644
index 0000000..1a13461
--- /dev/null
+++ b/fdanyone/model/__init__.py
@@ -0,0 +1 @@
+"""4DAnyone model loading and stage inference."""
diff --git a/fdanyone/model/inference.py b/fdanyone/model/inference.py
new file mode 100644
index 0000000..1a3e362
--- /dev/null
+++ b/fdanyone/model/inference.py
@@ -0,0 +1,521 @@
+"""Hydra/Lightning-free multi-view generation.
+
+RCP and final target generation share one source encoding and prompt embedding.
+Target groups execute sequentially on one GPU; TCR optionally shifts their
+membership between denoising steps.
+"""
+
+from __future__ import annotations
+
+import gc
+import logging
+import time
+from collections.abc import Iterable
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+from PIL import Image
+
+from fdanyone.assets import BaseAssets
+from fdanyone.config import INFERENCE
+from fdanyone.errors import FourDAnyoneError
+from fdanyone.model.loader import load_pipeline
+from fdanyone.model.routing import routing_steps
+from fdanyone.skeleton.pipeline import Conditioning, SkeletonVideo
+from fdanyone.video import CanonicalClip, write_video
+from fdanyone.views import ViewPlan
+
+LOGGER = logging.getLogger("fdanyone")
+
+
+@dataclass(frozen=True)
+class GeneratedViews:
+ """Paths and measurements produced by one resolved view plan."""
+
+ rcp_videos: tuple[Path, ...]
+ target_videos: tuple[Path, ...]
+ view_plan: ViewPlan
+ seed: int
+ device: str
+ elapsed_seconds: dict[str, float]
+ peak_vram_allocated_bytes: int
+ peak_vram_reserved_bytes: int
+
+
+def _empty_cuda_cache() -> None:
+ import torch
+
+ gc.collect()
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+
+
+def _move_model(model, device: str) -> None:
+ model.to(device)
+ _empty_cuda_cache()
+
+
+def _offload_model(model) -> None:
+ model.to("cpu")
+ _empty_cuda_cache()
+
+
+def _tiler_kwargs() -> dict:
+ return {
+ "tiled": INFERENCE.tiled_vae,
+ "tile_size": INFERENCE.vae_tile_size,
+ "tile_stride": INFERENCE.vae_tile_stride,
+ }
+
+
+def _bf16_autocast():
+ """Match Lightning's ``bf16-mixed`` inference context without Lightning."""
+
+ import torch
+
+ return torch.autocast(device_type="cuda", dtype=torch.bfloat16)
+
+
+def _channels_last_source_layout(video):
+ """Preserve the frozen source tensor's VFHWC-backed VCFHW layout."""
+
+ import torch
+
+ if video.ndim != 5:
+ raise FourDAnyoneError(f"Expected a 5D video tensor, got shape {tuple(video.shape)}.")
+ return video.contiguous(memory_format=torch.channels_last_3d)
+
+
+def _encode_prompt(pipe, device: str) -> dict[str, object]:
+ import torch
+
+ _move_model(pipe.text_encoder, device)
+ with torch.inference_mode(), _bf16_autocast():
+ context = pipe.prompter.encode_prompt(INFERENCE.prompt, positive=True, device=device)
+ prompt = {"context": context.detach().to("cpu")}
+ _offload_model(pipe.text_encoder)
+ return prompt
+
+
+def _encode_videos(pipe, videos, device: str):
+ import torch
+
+ videos = videos.to(dtype=pipe.torch_dtype, device=device)
+ with torch.inference_mode(), _bf16_autocast():
+ latents = pipe.encode_video(videos, **_tiler_kwargs())
+ return latents.detach().to("cpu")
+
+
+def _noise(pipe, num_views: int, num_frames: int, seed: int, device: str):
+ import torch
+
+ shape = (
+ num_views,
+ pipe.vae.model.z_dim,
+ (num_frames - 1) // 4 + 1,
+ INFERENCE.height // pipe.vae.upsampling_factor,
+ INFERENCE.width // pipe.vae.upsampling_factor,
+ )
+ generator = torch.Generator("cpu").manual_seed(seed)
+ return torch.randn(shape, generator=generator, device="cpu", dtype=torch.float32).to(
+ dtype=pipe.torch_dtype, device=device
+ )
+
+
+def _load_skeleton_cache(
+ conditioning: Conditioning,
+ skeletons: Iterable[SkeletonVideo],
+ cache: dict[SkeletonVideo, object],
+) -> None:
+ import torch
+
+ for skeleton in skeletons:
+ if skeleton in cache:
+ continue
+ LOGGER.info("Loading skeleton conditioning from %s", skeleton.path.name)
+ tensor = conditioning.load_skeleton_tensor([skeleton]).to(dtype=torch.bfloat16, device="cpu")
+ cache[skeleton] = tensor.contiguous()
+
+
+def _skeleton_group(
+ cache: dict[SkeletonVideo, object],
+ skeletons: Iterable[SkeletonVideo],
+ device: str,
+ *,
+ channels_last: bool = False,
+):
+ import torch
+
+ items = tuple(skeletons)
+ first = cache[items[0]]
+ memory_format = torch.channels_last_3d if channels_last else torch.contiguous_format
+ group = torch.empty(
+ (len(items), *first.shape[1:]),
+ dtype=first.dtype,
+ device=device,
+ memory_format=memory_format,
+ )
+ for output_index, skeleton in enumerate(items):
+ group[output_index].copy_(cache[skeleton][0])
+ return group
+
+
+def _denoise_rcp(
+ pipe,
+ src_latents,
+ prompt,
+ camera_ids: tuple[int, ...],
+ skeletons: tuple[SkeletonVideo, ...],
+ skeleton_cache: dict[SkeletonVideo, object],
+ device: str,
+ seed: int,
+):
+ import torch
+ from tqdm.auto import tqdm
+
+ if len(skeletons) != len(camera_ids):
+ raise FourDAnyoneError(f"RCP requires {len(camera_ids)} skeleton views, got {len(skeletons)}.")
+ pipe.scheduler.set_timesteps(
+ INFERENCE.num_inference_steps,
+ denoising_strength=INFERENCE.denoising_strength,
+ shift=INFERENCE.scheduler_shift,
+ )
+ latents = _noise(pipe, len(skeletons), INFERENCE.num_frames, seed, device)
+ source = src_latents.to(dtype=pipe.torch_dtype, device=device)
+ context = {name: value.to(dtype=pipe.torch_dtype, device=device) for name, value in prompt.items()}
+ # Preserve the reviewed RCP tensor layout: it changes CUDA kernel selection.
+ # Target generation below materializes each sequential group contiguously.
+ skeleton_tensor = _skeleton_group(skeleton_cache, skeletons, device, channels_last=True)
+
+ with torch.inference_mode(), _bf16_autocast():
+ for timestep in tqdm(pipe.scheduler.timesteps, desc=f"RCP 1-to-{len(camera_ids)}"):
+ batched_timestep = timestep.unsqueeze(0).to(dtype=pipe.torch_dtype, device=device)
+ batched_timestep = torch.cat([batched_timestep] * len(skeletons), dim=0)
+ prediction = pipe.dit(
+ x=latents,
+ x_src=source,
+ timestep=batched_timestep,
+ skeletons=skeleton_tensor,
+ **context,
+ **pipe.prepare_extra_input(latents),
+ )
+ latents = pipe.scheduler.step(prediction, timestep, latents)
+ return latents.detach().to("cpu")
+
+
+def _tensor_frames(video) -> Iterable[np.ndarray]:
+ """Match DiffSynth's float-to-uint8 truncation exactly."""
+
+ frames = video.detach().float().add_(1.0).mul_(127.5).clamp_(0.0, 255.0).to("cpu")
+ for frame_index in range(frames.shape[1]):
+ yield frames[:, frame_index].permute(1, 2, 0).numpy().astype(np.uint8)
+
+
+def _save_rcp_jpegs(video, camera_id: int, root: Path) -> Path:
+ import torchvision.transforms.functional as transform
+
+ frame_dir = root / f"{camera_id:06d}"
+ frame_dir.mkdir(parents=True, exist_ok=False)
+ normalized = video.detach().float().mul(0.5).add_(0.5).clamp_(0.0, 1.0).to("cpu")
+ for frame_index in range(normalized.shape[1]):
+ image = transform.to_pil_image(normalized[:, frame_index])
+ image.save(frame_dir / f"{frame_index:06d}.jpg", quality=INFERENCE.rcp_jpeg_quality)
+ return frame_dir
+
+
+def _rcp_reference_video_layout(frame_first_video):
+ """Match ``prepare_batch`` for JPEG-backed ``[V,F,C,H,W]`` data."""
+
+ if frame_first_video.ndim != 5:
+ raise FourDAnyoneError(f"Expected a 5D frame-first video tensor, got shape {tuple(frame_first_video.shape)}.")
+ return frame_first_video.permute(0, 2, 1, 3, 4)
+
+
+def _load_rcp_reference_videos(frame_dirs: Iterable[Path], num_frames: int):
+ """Decode RCP references exactly like the frozen JPEG-backed input path."""
+
+ import torch
+ import torchvision.transforms.functional as transform
+
+ videos = []
+ for frame_dir in frame_dirs:
+ frames = []
+ for frame_index in range(num_frames):
+ path = frame_dir / f"{frame_index:06d}.jpg"
+ with Image.open(path) as image:
+ frames.append(transform.to_tensor(image.convert("RGB")))
+ videos.append(torch.stack(frames, dim=0))
+ frame_first_video = torch.stack(videos, dim=0).mul_(2.0).sub_(1.0)
+ return _rcp_reference_video_layout(frame_first_video)
+
+
+def _fill_rcp_reference_slots(latents):
+ """Match the checkpoint's fixed four-reference ViewPack input."""
+
+ count = latents.shape[0]
+ if count == 4:
+ return latents
+ if count == 1:
+ return latents.expand(4, *latents.shape[1:])
+ raise FourDAnyoneError(f"RCP must provide one or four reference latents, got {count}.")
+
+
+def _decode_rcp(
+ pipe,
+ latents,
+ camera_ids: tuple[int, ...],
+ output_dir: Path,
+ clip: CanonicalClip,
+ device: str,
+) -> tuple[tuple[Path, ...], tuple[Path, ...]]:
+ import torch
+
+ if latents.shape[0] != len(camera_ids):
+ raise FourDAnyoneError(f"RCP decode expected {len(camera_ids)} latent views, got {latents.shape[0]}.")
+ frame_root = output_dir / "frames"
+ video_root = output_dir / "videos"
+ frame_root.mkdir(parents=True, exist_ok=False)
+ video_root.mkdir(parents=True, exist_ok=False)
+ frame_outputs: list[Path] = []
+ video_outputs: list[Path] = []
+ with torch.inference_mode(), _bf16_autocast():
+ for latent_index, camera_id in enumerate(camera_ids):
+ LOGGER.info("Decoding RCP camera %02d", camera_id)
+ video = pipe.decode_video(
+ latents[latent_index : latent_index + 1].to(dtype=pipe.torch_dtype, device=device),
+ **_tiler_kwargs(),
+ )[0]
+ frame_outputs.append(_save_rcp_jpegs(video, camera_id, frame_root))
+ video_outputs.append(
+ write_video(
+ _tensor_frames(video),
+ video_root / f"{camera_id:02d}.mp4",
+ clip.fps,
+ crf=INFERENCE.target_h264_crf,
+ preset=INFERENCE.h264_preset,
+ )
+ )
+ del video
+ torch.cuda.empty_cache()
+ return tuple(frame_outputs), tuple(video_outputs)
+
+
+def _denoise_targets(
+ pipe,
+ src_latents,
+ prompt,
+ skeletons: tuple[SkeletonVideo, ...],
+ skeleton_cache: dict[SkeletonVideo, object],
+ view_plan: ViewPlan,
+ device: str,
+ seed: int,
+):
+ import torch
+ from tqdm.auto import tqdm
+
+ num_views = len(skeletons)
+ if num_views != view_plan.num_target_views:
+ raise FourDAnyoneError(
+ f"Target generation requires {view_plan.num_target_views} skeleton views, got {num_views}."
+ )
+ pipe.scheduler.set_timesteps(
+ INFERENCE.num_inference_steps,
+ denoising_strength=INFERENCE.denoising_strength,
+ shift=INFERENCE.scheduler_shift,
+ )
+ latents = _noise(pipe, num_views, INFERENCE.num_frames, seed, device)
+ source = src_latents.to(dtype=pipe.torch_dtype, device=device)
+ context = {name: value.to(dtype=pipe.torch_dtype, device=device) for name, value in prompt.items()}
+ routes = routing_steps(
+ views_per_layer=view_plan.views_per_layer,
+ num_layers=view_plan.num_layers,
+ group_size=view_plan.views_per_group,
+ num_steps=INFERENCE.num_inference_steps,
+ enable_tcr=view_plan.tcr_active,
+ circular=view_plan.closed_yaw,
+ )
+
+ with torch.inference_mode(), _bf16_autocast():
+ for step_index, groups in enumerate(tqdm(routes, desc=f"Generate {num_views} target views")):
+ for view_indices in groups:
+ index = torch.tensor(view_indices, dtype=torch.long, device=device)
+ local_latents = torch.index_select(latents, 0, index)
+ local_skeletons = _skeleton_group(
+ skeleton_cache,
+ (skeletons[view_index] for view_index in view_indices),
+ device,
+ )
+ timestep = pipe.scheduler.timesteps[step_index]
+ batched_timestep = timestep.unsqueeze(0).to(dtype=pipe.torch_dtype, device=device)
+ batched_timestep = torch.cat([batched_timestep] * len(view_indices), dim=0)
+ prediction = pipe.dit(
+ x=local_latents,
+ x_src=source,
+ timestep=batched_timestep,
+ skeletons=local_skeletons,
+ **context,
+ **pipe.prepare_extra_input(local_latents),
+ )
+ local_latents = pipe.scheduler.step(prediction, timestep, local_latents)
+ latents.index_copy_(0, index, local_latents)
+ del local_latents, local_skeletons, prediction
+ return latents.detach().to("cpu")
+
+
+def _decode_targets(pipe, latents, output_dir: Path, clip: CanonicalClip, device: str) -> tuple[Path, ...]:
+ import torch
+
+ video_root = output_dir / "videos"
+ video_root.mkdir(parents=True, exist_ok=False)
+ outputs: list[Path] = []
+ with torch.inference_mode(), _bf16_autocast():
+ for camera_id in range(latents.shape[0]):
+ LOGGER.info("Decoding target camera %02d", camera_id)
+ video = pipe.decode_video(
+ latents[camera_id : camera_id + 1].to(dtype=pipe.torch_dtype, device=device),
+ **_tiler_kwargs(),
+ )[0]
+ path = write_video(
+ _tensor_frames(video),
+ video_root / f"{camera_id:02d}.mp4",
+ clip.fps,
+ crf=INFERENCE.target_h264_crf,
+ preset=INFERENCE.h264_preset,
+ )
+ outputs.append(path)
+ del video
+ torch.cuda.empty_cache()
+ return tuple(outputs)
+
+
+def generate_views(
+ *,
+ clip: CanonicalClip,
+ conditioning: Conditioning,
+ checkpoint_path: str | Path,
+ assets: BaseAssets,
+ output_dir: str | Path,
+ device: str,
+ seed: int,
+) -> GeneratedViews:
+ """Generate the proposal (when enabled) and the requested target views."""
+
+ import torch
+
+ if conditioning.num_frames != INFERENCE.num_frames or len(clip.frames) != INFERENCE.num_frames:
+ raise FourDAnyoneError("Generation requires the frozen 121-frame contract.")
+ if seed < 0:
+ raise FourDAnyoneError(f"seed must be non-negative, got {seed}.")
+ device_index = int(device.removeprefix("cuda:"))
+ LOGGER.info("Using %s (%s)", device, torch.cuda.get_device_name(device_index))
+
+ root = Path(output_dir).expanduser().resolve()
+ root.mkdir(parents=True, exist_ok=False)
+ view_plan = conditioning.view_plan
+ if len(conditioning.target_skeletons) != view_plan.num_target_views:
+ raise FourDAnyoneError("Target skeleton count does not match the resolved view plan.")
+ if len(conditioning.rcp_skeletons) != len(view_plan.rcp_camera_ids):
+ raise FourDAnyoneError("RCP skeleton count does not match the resolved view plan.")
+ loaded = load_pipeline(checkpoint_path=checkpoint_path, assets=assets, device=device)
+ pipe = loaded.pipe
+ timings: dict[str, float] = {}
+ skeleton_cache: dict[SkeletonVideo, object] = {}
+ torch.cuda.reset_peak_memory_stats(device_index)
+
+ started = time.monotonic()
+ prompt = _encode_prompt(pipe, device)
+ loaded.release_text_encoder()
+ timings["prompt"] = time.monotonic() - started
+
+ started = time.monotonic()
+ _move_model(pipe.vae, device)
+ source_video = _channels_last_source_layout(conditioning.load_source_tensor())
+ source_latents = _encode_videos(pipe, source_video, device)
+ del source_video
+ _offload_model(pipe.vae)
+ timings["source_encode"] = time.monotonic() - started
+
+ rcp_videos: tuple[Path, ...] = ()
+ target_sources = source_latents
+ if view_plan.enable_rcp:
+ started = time.monotonic()
+ _load_skeleton_cache(conditioning, conditioning.rcp_skeletons, skeleton_cache)
+ _move_model(pipe.dit, device)
+ rcp_latents = _denoise_rcp(
+ pipe,
+ source_latents,
+ prompt,
+ view_plan.rcp_camera_ids,
+ conditioning.rcp_skeletons,
+ skeleton_cache,
+ device,
+ seed,
+ )
+ _offload_model(pipe.dit)
+ timings["rcp_denoise"] = time.monotonic() - started
+
+ started = time.monotonic()
+ rcp_root = root / "rcp"
+ rcp_root.mkdir()
+ _move_model(pipe.vae, device)
+ frame_dirs, rcp_videos = _decode_rcp(
+ pipe,
+ rcp_latents,
+ view_plan.rcp_camera_ids,
+ rcp_root,
+ clip,
+ device,
+ )
+ # Match the reference JPEG-backed RCP boundary before the second VAE
+ # encode; backing layout changes can otherwise select different BF16
+ # CUDA kernels even when tensor values are identical.
+ reference_count = min(4, len(view_plan.rcp_camera_ids))
+ rcp_reference_videos = _load_rcp_reference_videos(frame_dirs[:reference_count], INFERENCE.num_frames)
+ rcp_reference_latents = _encode_videos(pipe, rcp_reference_videos, device)
+ rcp_reference_latents = _fill_rcp_reference_slots(rcp_reference_latents)
+ # VAE38 encodes batch elements independently, so the existing source
+ # encoding is identical to re-encoding source plus the RCP references.
+ target_sources = torch.cat([source_latents, rcp_reference_latents], dim=0)
+ del rcp_latents, rcp_reference_videos, rcp_reference_latents
+ _offload_model(pipe.vae)
+ timings["rcp_decode_and_reference_encode"] = time.monotonic() - started
+
+ started = time.monotonic()
+ _load_skeleton_cache(conditioning, conditioning.target_skeletons, skeleton_cache)
+ _move_model(pipe.dit, device)
+ target_latents = _denoise_targets(
+ pipe,
+ target_sources,
+ prompt,
+ conditioning.target_skeletons,
+ skeleton_cache,
+ view_plan,
+ device,
+ seed,
+ )
+ _offload_model(pipe.dit)
+ timings["target_denoise"] = time.monotonic() - started
+
+ started = time.monotonic()
+ target_root = root / "target"
+ target_root.mkdir()
+ _move_model(pipe.vae, device)
+ target_videos = _decode_targets(pipe, target_latents, target_root, clip, device)
+ _offload_model(pipe.vae)
+ timings["target_decode"] = time.monotonic() - started
+ peak_vram_allocated = int(torch.cuda.max_memory_allocated(device_index))
+ peak_vram_reserved = int(torch.cuda.max_memory_reserved(device_index))
+
+ del target_latents, target_sources, source_latents, prompt, skeleton_cache, loaded, pipe
+ _empty_cuda_cache()
+ return GeneratedViews(
+ rcp_videos=rcp_videos,
+ target_videos=target_videos,
+ view_plan=view_plan,
+ seed=seed,
+ device=device,
+ elapsed_seconds=timings,
+ peak_vram_allocated_bytes=peak_vram_allocated,
+ peak_vram_reserved_bytes=peak_vram_reserved,
+ )
diff --git a/fdanyone/model/loader.py b/fdanyone/model/loader.py
new file mode 100644
index 0000000..d2c021e
--- /dev/null
+++ b/fdanyone/model/loader.py
@@ -0,0 +1,259 @@
+"""Direct, registry-free loading of the frozen Wan/SpaTem inference stack."""
+
+from __future__ import annotations
+
+import gc
+from dataclasses import dataclass
+from pathlib import Path
+
+from fdanyone.assets import BaseAssets
+from fdanyone.config import INFERENCE
+from fdanyone.errors import AssetError, ConfigurationError
+
+WAN22_TI2V_5B_CONFIG = {
+ "has_image_input": False,
+ "patch_size": (1, 2, 2),
+ "in_dim": 48,
+ "dim": 3072,
+ "ffn_dim": 14336,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 48,
+ "num_heads": 24,
+ "num_layers": 30,
+ "eps": 1e-6,
+ "seperated_timestep": True,
+ "require_clip_embedding": False,
+ "require_vae_embedding": False,
+ "fuse_vae_embedding_in_latents": True,
+}
+
+
+@dataclass
+class LoadedPipeline:
+ pipe: object
+
+ def release_text_encoder(self) -> None:
+ """Release T5 after the single fixed prompt has been encoded."""
+
+ import torch
+
+ self.pipe.text_encoder = None
+ self.pipe.prompter.text_encoder = None
+ gc.collect()
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+
+
+def _load_checkpoint(path: Path):
+ try:
+ from safetensors.torch import load_file
+ except ImportError as exc:
+ raise AssetError("safetensors is required to load the 4DAnyone checkpoint.") from exc
+ return load_file(str(path), device="cpu")
+
+
+def _strict_assign(module, state_dict: dict, label: str) -> None:
+ """Load into a meta-initialized module without a second parameter copy."""
+
+ try:
+ incompatible = module.load_state_dict(state_dict, strict=True, assign=True)
+ except TypeError as exc:
+ raise ConfigurationError("4DAnyone requires PyTorch >=2.8 for assign-based model loading.") from exc
+ except RuntimeError as exc:
+ raise AssetError(f"{label} is incompatible with the released architecture: {exc}") from exc
+ if incompatible.missing_keys or incompatible.unexpected_keys:
+ raise AssetError(
+ f"{label} strict load failed; missing={incompatible.missing_keys}, "
+ f"unexpected={incompatible.unexpected_keys}"
+ )
+
+
+def _load_dit(checkpoint_path: Path, dtype):
+ import torch
+
+ from fdanyone.vendor.diffsynth.models.wan_video_dit import WanModel, precompute_freqs_cis_3d
+ from fdanyone.vendor.diffsynth.pipelines.wan_video_spatem import WanVideoSpaTemPipeline
+
+ with torch.device("meta"):
+ dit = WanModel(**WAN22_TI2V_5B_CONFIG)
+ shell = WanVideoSpaTemPipeline(device="cpu", torch_dtype=dtype)
+ shell.dit = dit
+ shell.init_spatem_modules(
+ disable_video_attn=INFERENCE.disable_video_attention,
+ use_4d_attn=INFERENCE.use_4d_attention,
+ use_mvs_attn=INFERENCE.use_mvs_attention,
+ use_src_self_attn=INFERENCE.use_source_self_attention,
+ use_src_cross_attn=INFERENCE.use_source_cross_attention,
+ freqs_src_shift=INFERENCE.freqs_src_shift,
+ range_mvs_attn=INFERENCE.mvs_attention_range,
+ # ``ViewPack`` is the upstream module name for RCP references.
+ use_viewpack=True,
+ use_pose_encoder=INFERENCE.use_pose_encoder,
+ pose_encoder_type=INFERENCE.pose_encoder_type,
+ use_cam_encoder=INFERENCE.use_camera_encoder,
+ use_lbm=INFERENCE.use_lbm,
+ )
+ state_dict = _load_checkpoint(checkpoint_path)
+ _strict_assign(dit, state_dict, "4DAnyone DiT checkpoint")
+ del state_dict
+ # ``freqs`` is a derived, non-persistent tensor and therefore is not in the
+ # state dict populated above.
+ dit.freqs = precompute_freqs_cis_3d(WAN22_TI2V_5B_CONFIG["dim"] // WAN22_TI2V_5B_CONFIG["num_heads"])
+ return dit.eval().requires_grad_(False)
+
+
+def _load_vae(path: Path, dtype):
+ import torch
+
+ from fdanyone.vendor.diffsynth.models.wan_video_vae import WanVideoVAE38
+
+ state_dict = torch.load(path, map_location="cpu", weights_only=True)
+ state_dict = WanVideoVAE38.state_dict_converter().from_civitai(state_dict)
+ with torch.device("meta"):
+ vae = WanVideoVAE38()
+ _strict_assign(vae, state_dict, "Wan2.2 VAE")
+ del state_dict
+ # Wan's latent normalization tensors are plain attributes rather than
+ # registered buffers, so materialize them after meta initialization.
+ mean = (
+ -0.2289,
+ -0.0052,
+ -0.1323,
+ -0.2339,
+ -0.2799,
+ 0.0174,
+ 0.1838,
+ 0.1557,
+ -0.1382,
+ 0.0542,
+ 0.2813,
+ 0.0891,
+ 0.1570,
+ -0.0098,
+ 0.0375,
+ -0.1825,
+ -0.2246,
+ -0.1207,
+ -0.0698,
+ 0.5109,
+ 0.2665,
+ -0.2108,
+ -0.2158,
+ 0.2502,
+ -0.2055,
+ -0.0322,
+ 0.1109,
+ 0.1567,
+ -0.0729,
+ 0.0899,
+ -0.2799,
+ -0.1230,
+ -0.0313,
+ -0.1649,
+ 0.0117,
+ 0.0723,
+ -0.2839,
+ -0.2083,
+ -0.0520,
+ 0.3748,
+ 0.0152,
+ 0.1957,
+ 0.1433,
+ -0.2944,
+ 0.3573,
+ -0.0548,
+ -0.1681,
+ -0.0667,
+ )
+ std = (
+ 0.4765,
+ 1.0364,
+ 0.4514,
+ 1.1677,
+ 0.5313,
+ 0.4990,
+ 0.4818,
+ 0.5013,
+ 0.8158,
+ 1.0344,
+ 0.5894,
+ 1.0901,
+ 0.6885,
+ 0.6165,
+ 0.8454,
+ 0.4978,
+ 0.5759,
+ 0.3523,
+ 0.7135,
+ 0.6804,
+ 0.5833,
+ 1.4146,
+ 0.8986,
+ 0.5659,
+ 0.7069,
+ 0.5338,
+ 0.4889,
+ 0.4917,
+ 0.4069,
+ 0.4999,
+ 0.6866,
+ 0.4093,
+ 0.5709,
+ 0.6065,
+ 0.6415,
+ 0.4944,
+ 0.5726,
+ 1.2042,
+ 0.5458,
+ 1.6887,
+ 0.3971,
+ 1.0600,
+ 0.3943,
+ 0.5537,
+ 0.5444,
+ 0.4089,
+ 0.7468,
+ 0.7744,
+ )
+ vae.mean = torch.tensor(mean)
+ vae.std = torch.tensor(std)
+ vae.scale = [vae.mean, 1.0 / vae.std]
+ return vae.to(dtype=dtype).eval().requires_grad_(False)
+
+
+def _load_text_encoder(path: Path, dtype):
+ import torch
+
+ from fdanyone.vendor.diffsynth.models.wan_video_text_encoder import WanTextEncoder
+
+ state_dict = torch.load(path, map_location="cpu", weights_only=True)
+ state_dict = WanTextEncoder.state_dict_converter().from_civitai(state_dict)
+ with torch.device("meta"):
+ text_encoder = WanTextEncoder()
+ _strict_assign(text_encoder, state_dict, "Wan T5 text encoder")
+ del state_dict
+ return text_encoder.to(dtype=dtype).eval().requires_grad_(False)
+
+
+def load_pipeline(
+ *,
+ checkpoint_path: str | Path,
+ assets: BaseAssets,
+ device: str,
+) -> LoadedPipeline:
+ """Load exactly the runtime models required by the released checkpoint."""
+
+ import torch
+
+ from fdanyone.vendor.diffsynth.pipelines.wan_video_spatem import WanVideoSpaTemPipeline
+
+ dtype = torch.bfloat16
+ pipe = WanVideoSpaTemPipeline(device=device, torch_dtype=dtype, tokenizer_path=str(assets.tokenizer))
+ pipe.dit = _load_dit(Path(checkpoint_path), dtype)
+ pipe.vae = _load_vae(assets.vae, dtype)
+ pipe.text_encoder = _load_text_encoder(assets.text_encoder, dtype)
+ pipe.prompter.fetch_models(pipe.text_encoder)
+ pipe.height_division_factor = pipe.vae.upsampling_factor * 2
+ pipe.width_division_factor = pipe.vae.upsampling_factor * 2
+ return LoadedPipeline(pipe=pipe)
diff --git a/fdanyone/model/routing.py b/fdanyone/model/routing.py
new file mode 100644
index 0000000..c40b8f2
--- /dev/null
+++ b/fdanyone/model/routing.py
@@ -0,0 +1,69 @@
+"""Target-context routing across view groups."""
+
+from __future__ import annotations
+
+from itertools import pairwise
+
+
+def _validate_grouping(num_views: int, group_size: int) -> int:
+ if num_views <= 0 or group_size <= 0 or num_views % group_size:
+ raise ValueError(f"num_views={num_views} must be divisible by positive group_size={group_size}.")
+ return num_views // group_size
+
+
+def view_groups(
+ num_views: int,
+ group_size: int,
+ offset: int = 0,
+ *,
+ circular: bool = True,
+) -> tuple[tuple[int, ...], ...]:
+ """Partition one camera layer, optionally without joining its endpoints."""
+
+ num_groups = _validate_grouping(num_views, group_size)
+ if circular:
+ return tuple(
+ tuple((group_index * group_size + offset + local_index) % num_views for local_index in range(group_size))
+ for group_index in range(num_groups)
+ )
+
+ # Shifting an open sequence creates smaller boundary groups instead of a
+ # false neighborhood between the two ends of a partial yaw span.
+ offset %= group_size
+ boundaries = [0]
+ if offset:
+ boundaries.append(offset)
+ boundaries.extend(range(offset + group_size, num_views, group_size))
+ boundaries.append(num_views)
+ return tuple(tuple(range(start, end)) for start, end in pairwise(boundaries))
+
+
+def routing_steps(
+ *,
+ views_per_layer: int,
+ num_layers: int,
+ group_size: int,
+ num_steps: int,
+ enable_tcr: bool,
+ circular: bool,
+) -> tuple[tuple[tuple[int, ...], ...], ...]:
+ """Return layer-local target groups for every denoising step."""
+
+ if num_steps <= 0:
+ raise ValueError(f"num_steps must be positive, got {num_steps}.")
+ if num_layers <= 0:
+ raise ValueError(f"num_layers must be positive, got {num_layers}.")
+ _validate_grouping(views_per_layer, group_size)
+ return tuple(
+ tuple(
+ tuple(layer_index * views_per_layer + view_index for view_index in group)
+ for layer_index in range(num_layers)
+ for group in view_groups(
+ views_per_layer,
+ group_size,
+ step_index if enable_tcr else 0,
+ circular=circular,
+ )
+ )
+ for step_index in range(num_steps)
+ )
diff --git a/fdanyone/motion/__init__.py b/fdanyone/motion/__init__.py
new file mode 100644
index 0000000..5fdaf00
--- /dev/null
+++ b/fdanyone/motion/__init__.py
@@ -0,0 +1,5 @@
+"""GVHMR motion recovery."""
+
+from fdanyone.motion.result import MotionResult
+
+__all__ = ["MotionResult"]
diff --git a/fdanyone/motion/gvhmr.py b/fdanyone/motion/gvhmr.py
new file mode 100644
index 0000000..81a73eb
--- /dev/null
+++ b/fdanyone/motion/gvhmr.py
@@ -0,0 +1,315 @@
+"""Classic GVHMR inference used by 4DAnyone.
+
+The official demo imports training, evaluation, visualization, and
+moving-camera modules eagerly. This file keeps the released static-camera path
+in one place without exposing Hydra or backend abstractions to 4DAnyone users.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import importlib
+import importlib.util
+import itertools
+import json
+import os
+import subprocess
+import sys
+import types
+import warnings
+from collections.abc import Iterator
+from contextlib import contextmanager
+from pathlib import Path
+
+from fdanyone.errors import AssetError, VideoContractError
+from fdanyone.motion.result import SMPL_PARAMETER_NAMES, MotionResult
+from fdanyone.vendor.pytorch3d_compat import install_if_needed as install_pytorch3d_compat
+from fdanyone.video import CanonicalClip
+
+GVHMR_ASSETS = (
+ "inputs/checkpoints/gvhmr/gvhmr_siga24_release.ckpt",
+ "inputs/checkpoints/hmr2/epoch=10-step=25000.ckpt",
+ "inputs/checkpoints/vitpose/vitpose-h-multi-coco.pth",
+ "inputs/checkpoints/yolo/yolov8x.pt",
+ "inputs/checkpoints/body_models/smplx/SMPLX_NEUTRAL.npz",
+)
+
+
+def validate_gvhmr(root: str | Path) -> tuple[Path, str]:
+ """Locate the GVHMR checkout and files consumed by inference."""
+
+ path = Path(root).expanduser().resolve()
+ required = ("hmr4d/__init__.py", "tools/demo/demo.py", *GVHMR_ASSETS)
+ missing = [relative for relative in required if not (path / relative).is_file()]
+ if missing:
+ formatted = "\n - ".join(missing)
+ raise AssetError(
+ f"GVHMR is incomplete under {path}. Run `git submodule update --init third_party/GVHMR`, "
+ f"`python scripts/download_model.py`, and `python scripts/download_smplx.py`; missing:\n"
+ f" - {formatted}"
+ )
+ try:
+ revision = subprocess.check_output(
+ ["git", "-C", str(path), "rev-parse", "HEAD"],
+ text=True,
+ stderr=subprocess.DEVNULL,
+ ).strip()
+ except (OSError, subprocess.CalledProcessError) as exc:
+ raise AssetError(f"GVHMR must be a git checkout: {path}") from exc
+ if len(revision) != 40:
+ raise AssetError(f"Cannot identify the GVHMR revision at {path}.")
+ return path, revision
+
+
+def hydra_override(name: str, value: str | Path) -> str:
+ """Quote a path for the internal GVHMR Hydra config."""
+
+ if not name.isidentifier():
+ raise ValueError(f"Invalid Hydra field name: {name!r}.")
+ return f"{name}={json.dumps(str(value), ensure_ascii=False)}"
+
+
+@contextmanager
+def gvhmr_imports(root: Path) -> Iterator[None]:
+ """Temporarily import GVHMR as if its checkout were the working tree."""
+
+ old_cwd = Path.cwd()
+ root_text = str(root)
+ already_present = root_text in sys.path
+ if not already_present:
+ sys.path.insert(0, root_text)
+ os.chdir(root)
+ try:
+ yield
+ finally:
+ os.chdir(old_cwd)
+ if not already_present:
+ with contextlib.suppress(ValueError):
+ sys.path.remove(root_text)
+
+
+@contextmanager
+def _legacy_checkpoint_loading():
+ """Restore pre-2.6 ``torch.load`` behavior for trusted GVHMR assets."""
+
+ name = "TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD"
+ previous = os.environ.get(name)
+ os.environ[name] = "1"
+ try:
+ with warnings.catch_warnings():
+ warnings.filterwarnings(
+ "ignore",
+ message=r"Environment variable TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD detected.*",
+ category=UserWarning,
+ )
+ yield
+ finally:
+ if previous is None:
+ os.environ.pop(name, None)
+ else:
+ os.environ[name] = previous
+
+
+def _install_optional_import_stubs() -> None:
+ """Avoid visualization and moving-camera dependencies we never call."""
+
+ def unavailable(*_args, **_kwargs):
+ raise RuntimeError("Wis3D visualization is not part of 4DAnyone inference.")
+
+ if importlib.util.find_spec("wis3d") is None:
+ module = types.ModuleType("hmr4d.utils.wis3d_utils")
+ module.make_wis3d = unavailable
+ module.add_motion_as_lines = unavailable
+ sys.modules[module.__name__] = module
+
+ def moving_camera_unavailable(*_args, **_kwargs):
+ raise RuntimeError("SimpleVO is unavailable in the static-camera 4DAnyone runtime.")
+
+ module = types.ModuleType("hmr4d.utils.preproc.relpose.simple_vo")
+ module.SimpleVO = moving_camera_unavailable
+ sys.modules[module.__name__] = module
+
+
+def _register_inference_store() -> None:
+ """Register only the Hydra groups referenced by GVHMR's demo config."""
+
+ _install_optional_import_stubs()
+ for module in (
+ "hmr4d.model.gvhmr.gvhmr_pl_demo",
+ "hmr4d.model.gvhmr.utils.endecoder",
+ "hmr4d.network.gvhmr.relative_transformer",
+ ):
+ importlib.import_module(module)
+
+ # GVHMR installs a colored handler on the root logger. Remove only that
+ # duplicate because the public pipeline already owns a handler.
+ logger_module = sys.modules.get("hmr4d.utils.pylogger")
+ logger = getattr(logger_module, "Log", None)
+ handler = getattr(logger_module, "ch", None)
+ if (
+ logger is not None
+ and handler in logger.handlers
+ and any(candidate is not handler for candidate in logger.handlers)
+ ):
+ logger.removeHandler(handler)
+
+
+def _run_preprocess(cfg) -> None:
+ """Run tracker, ViTPose, and image-feature extraction."""
+
+ import torch
+ from hmr4d.utils.geo.hmr_cam import get_bbx_xys_from_xyxy
+ from hmr4d.utils.preproc.tracker import Tracker
+ from hmr4d.utils.preproc.vitfeat_extractor import Extractor
+ from hmr4d.utils.preproc.vitpose import VitPoseExtractor
+ from hmr4d.utils.pylogger import Log
+
+ if not bool(cfg.static_cam):
+ raise ValueError("4DAnyone requires GVHMR static_cam=true.")
+
+ Log.info("[Preprocess] Start!")
+ started = Log.time()
+ video_path = cfg.video_path
+ paths = cfg.paths
+
+ if not Path(paths.bbx).exists():
+ tracker = Tracker()
+ bbx_xyxy = tracker.get_one_track(video_path).float()
+ bbx_xys = get_bbx_xys_from_xyxy(bbx_xyxy, base_enlarge=1.2).float()
+ torch.save({"bbx_xyxy": bbx_xyxy, "bbx_xys": bbx_xys}, paths.bbx)
+ del tracker
+ else:
+ bbx_xys = torch.load(paths.bbx, weights_only=True)["bbx_xys"]
+ Log.info("[Preprocess] bbx (xyxy, xys) from %s", paths.bbx)
+
+ if not Path(paths.vitpose).exists():
+ extractor = VitPoseExtractor()
+ torch.save(extractor.extract(video_path, bbx_xys), paths.vitpose)
+ del extractor
+ else:
+ Log.info("[Preprocess] vitpose from %s", paths.vitpose)
+
+ if not Path(paths.vit_features).exists():
+ extractor = Extractor()
+ torch.save(extractor.extract_video_features(video_path, bbx_xys), paths.vit_features)
+ del extractor
+ else:
+ Log.info("[Preprocess] vit_features from %s", paths.vit_features)
+
+ Log.info("[Preprocess] End. Time elapsed: %.2fs", Log.time() - started)
+
+
+def _load_data(cfg):
+ """Build the static-camera tensors consumed by GVHMR."""
+
+ import torch
+ from hmr4d.utils.geo.hmr_cam import estimate_K
+ from hmr4d.utils.geo_transform import compute_cam_angvel
+ from hmr4d.utils.video_io_utils import get_video_lwh
+
+ if not bool(cfg.static_cam):
+ raise ValueError("4DAnyone requires GVHMR static_cam=true.")
+ paths = cfg.paths
+ length, width, height = get_video_lwh(cfg.video_path)
+ rotation_world_to_camera = torch.eye(3).repeat(length, 1, 1)
+ intrinsics = estimate_K(width, height).repeat(length, 1, 1)
+ return {
+ "length": torch.tensor(length),
+ "bbx_xys": torch.load(paths.bbx, weights_only=True)["bbx_xys"],
+ "kp2d": torch.load(paths.vitpose, weights_only=True),
+ "K_fullimg": intrinsics,
+ "cam_angvel": compute_cam_angvel(rotation_world_to_camera),
+ "f_imgseq": torch.load(paths.vit_features, weights_only=True),
+ }
+
+
+def _verify_gvhmr_decode(clip: CanonicalClip, working_video: Path, reader_factory) -> None:
+ """Ensure GVHMR's own video reader sees the canonical RGB frames."""
+
+ import numpy as np
+
+ reader = reader_factory(str(working_video))
+ sentinel = object()
+ try:
+ for index, (actual, expected) in enumerate(itertools.zip_longest(reader, clip.rgb_frames, fillvalue=sentinel)):
+ if actual is sentinel or expected is sentinel or not np.array_equal(actual, expected):
+ raise VideoContractError(f"GVHMR decoded a different canonical frame at index {index}.")
+ finally:
+ close = getattr(reader, "close", None)
+ if close is not None:
+ close()
+
+
+def run_gvhmr(
+ *,
+ clip: CanonicalClip,
+ working_video: str | Path,
+ output_dir: str | Path,
+ gvhmr_root: str | Path,
+ device: str,
+) -> MotionResult:
+ """Recover static-camera human motion from the canonical source clip."""
+
+ root, revision = validate_gvhmr(gvhmr_root)
+ working_video = Path(working_video).expanduser().resolve()
+ output_root = Path(output_dir).expanduser().resolve()
+ output_root.mkdir(parents=True, exist_ok=True)
+
+ with gvhmr_imports(root), _legacy_checkpoint_loading():
+ install_pytorch3d_compat()
+ import hydra
+ import torch
+ from hmr4d.model.gvhmr.gvhmr_pl_demo import DemoPL
+ from hmr4d.utils.net_utils import detach_to_cpu
+ from hmr4d.utils.video_io_utils import get_video_reader
+ from hydra import compose, initialize_config_module
+ from omegaconf import open_dict
+
+ _register_inference_store()
+ with initialize_config_module(version_base="1.3", config_module="hmr4d.configs"):
+ cfg = compose(
+ config_name="demo",
+ overrides=[
+ hydra_override("video_name", working_video.stem),
+ "static_cam=true",
+ "verbose=false",
+ "use_dpvo=false",
+ hydra_override("output_root", output_root),
+ ],
+ )
+ Path(cfg.output_dir).mkdir(parents=True, exist_ok=True)
+ Path(cfg.preprocess_dir).mkdir(parents=True, exist_ok=True)
+ with open_dict(cfg):
+ cfg.video_path = str(working_video)
+
+ _verify_gvhmr_decode(clip, working_video, get_video_reader)
+ _run_preprocess(cfg)
+ data = _load_data(cfg)
+ if int(data["length"]) != len(clip.frames):
+ raise RuntimeError(f"GVHMR decoded {int(data['length'])} frames, expected {len(clip.frames)}.")
+ observed_keypoints_2d = data["kp2d"].detach().cpu()
+ model: DemoPL = hydra.utils.instantiate(cfg.model, _recursive_=False)
+ model.load_pretrained_model(cfg.ckpt_path)
+ model = model.eval().to(device)
+ with torch.inference_mode():
+ prediction = detach_to_cpu(model.predict(data, static_cam=True))
+ del model, data
+ torch.cuda.empty_cache()
+
+ result = MotionResult(
+ gvhmr_revision=revision,
+ fps=clip.fps,
+ frame_timestamps_sec=tuple(float(frame.canonical_timestamp) for frame in clip.frames),
+ source_frame_indices=tuple(frame.source_index for frame in clip.frames),
+ source_pts=tuple(frame.source_pts for frame in clip.frames),
+ source_size_bytes=clip.source_size_bytes,
+ source_mtime_ns=clip.source_mtime_ns,
+ image_height=clip.height,
+ image_width=clip.width,
+ smpl_params_global={name: prediction["smpl_params_global"][name] for name in SMPL_PARAMETER_NAMES},
+ smpl_params_incam={name: prediction["smpl_params_incam"][name] for name in SMPL_PARAMETER_NAMES},
+ K_fullimg=prediction["K_fullimg"],
+ observed_keypoints_2d=observed_keypoints_2d,
+ )
+ result.validate(expected_frames=len(clip.frames))
+ return result
diff --git a/fdanyone/motion/result.py b/fdanyone/motion/result.py
new file mode 100644
index 0000000..85b2995
--- /dev/null
+++ b/fdanyone/motion/result.py
@@ -0,0 +1,188 @@
+"""Reusable GVHMR result stored as JSON plus safetensors."""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+from fractions import Fraction
+from pathlib import Path
+from typing import Any
+
+from fdanyone.errors import FourDAnyoneError
+
+SMPL_PARAMETER_NAMES = ("body_pose", "betas", "global_orient", "transl")
+SMPL_PARAMETER_WIDTHS = {"body_pose": 63, "betas": 10, "global_orient": 3, "transl": 3}
+
+
+@dataclass(frozen=True)
+class MotionResult:
+ gvhmr_revision: str
+ fps: Fraction
+ frame_timestamps_sec: tuple[float, ...]
+ source_frame_indices: tuple[int, ...]
+ source_pts: tuple[int | None, ...]
+ source_size_bytes: int
+ source_mtime_ns: int
+ image_height: int
+ image_width: int
+ smpl_params_global: dict[str, Any]
+ smpl_params_incam: dict[str, Any]
+ K_fullimg: Any
+ observed_keypoints_2d: Any
+ motion_world: str = "gvhmr_gravity_aligned_y_up"
+
+ @property
+ def num_frames(self) -> int:
+ return len(self.frame_timestamps_sec)
+
+ def validate(self, expected_frames: int = 121) -> None:
+ try:
+ import torch
+ except ImportError as exc:
+ raise FourDAnyoneError("PyTorch is required to validate motion tensors.") from exc
+ if self.motion_world != "gvhmr_gravity_aligned_y_up":
+ raise FourDAnyoneError(f"Unknown motion world convention: {self.motion_world!r}.")
+ if (
+ not isinstance(self.gvhmr_revision, str)
+ or len(self.gvhmr_revision) != 40
+ or any(character not in "0123456789abcdef" for character in self.gvhmr_revision.lower())
+ ):
+ raise FourDAnyoneError("MotionResult requires a 40-character GVHMR git revision.")
+ if self.fps <= 0:
+ raise FourDAnyoneError(f"MotionResult FPS must be positive, got {self.fps}.")
+ if self.num_frames != expected_frames:
+ raise FourDAnyoneError(f"MotionResult has {self.num_frames} frames, expected {expected_frames}.")
+ for name, values in (
+ ("source_frame_indices", self.source_frame_indices),
+ ("source_pts", self.source_pts),
+ ):
+ if len(values) != self.num_frames:
+ raise FourDAnyoneError(f"MotionResult {name} has {len(values)} values, expected {self.num_frames}.")
+ expected_timestamps = tuple(float(Fraction(index, 1) / self.fps) for index in range(self.num_frames))
+ if self.frame_timestamps_sec != expected_timestamps:
+ raise FourDAnyoneError("MotionResult timestamps are not the exact zero-based CFR timeline.")
+ if any(index < 0 for index in self.source_frame_indices) or any(
+ right < left for left, right in zip(self.source_frame_indices, self.source_frame_indices[1:], strict=False)
+ ):
+ raise FourDAnyoneError("MotionResult source-frame indices must be non-negative and monotonic.")
+ if self.source_size_bytes <= 0 or self.source_mtime_ns <= 0:
+ raise FourDAnyoneError("MotionResult has an invalid source-file identity.")
+ if self.image_height <= 0 or self.image_width <= 0:
+ raise FourDAnyoneError("MotionResult image dimensions must be positive.")
+ for group_name, parameters in (
+ ("smpl_params_global", self.smpl_params_global),
+ ("smpl_params_incam", self.smpl_params_incam),
+ ):
+ if set(parameters) != set(SMPL_PARAMETER_NAMES):
+ raise FourDAnyoneError(
+ f"MotionResult {group_name} must contain {SMPL_PARAMETER_NAMES}, got {tuple(parameters)}."
+ )
+ for name, tensor in parameters.items():
+ expected_shape = (self.num_frames, SMPL_PARAMETER_WIDTHS[name])
+ if not isinstance(tensor, torch.Tensor) or tuple(tensor.shape) != expected_shape:
+ raise FourDAnyoneError(f"{group_name}.{name} must have shape {expected_shape}.")
+ if not bool(torch.isfinite(tensor).all()):
+ raise FourDAnyoneError(f"{group_name}.{name} contains non-finite values.")
+ if not isinstance(self.K_fullimg, torch.Tensor) or tuple(self.K_fullimg.shape) != (self.num_frames, 3, 3):
+ raise FourDAnyoneError(f"K_fullimg must have shape ({self.num_frames}, 3, 3).")
+ if not bool(torch.isfinite(self.K_fullimg).all()):
+ raise FourDAnyoneError("K_fullimg contains non-finite values.")
+ expected_keypoint_shape = (self.num_frames, 17, 3)
+ if (
+ not isinstance(self.observed_keypoints_2d, torch.Tensor)
+ or tuple(self.observed_keypoints_2d.shape) != expected_keypoint_shape
+ ):
+ raise FourDAnyoneError(f"observed_keypoints_2d must have shape {expected_keypoint_shape}.")
+ if not bool(torch.isfinite(self.observed_keypoints_2d).all()):
+ raise FourDAnyoneError("observed_keypoints_2d contains non-finite values.")
+
+ def validate_against_clip(self, clip) -> None:
+ """Reject a cached result produced from a different video timeline."""
+
+ self.validate(expected_frames=len(clip.frames))
+ expected_timestamps = tuple(float(frame.canonical_timestamp) for frame in clip.frames)
+ expected_indices = tuple(frame.source_index for frame in clip.frames)
+ expected_pts = tuple(frame.source_pts for frame in clip.frames)
+ if self.fps != clip.fps:
+ raise FourDAnyoneError(f"Motion FPS {self.fps} does not match canonical FPS {clip.fps}.")
+ if self.frame_timestamps_sec != expected_timestamps:
+ raise FourDAnyoneError("Motion timestamps do not match the canonical clip.")
+ if self.source_frame_indices != expected_indices or self.source_pts != expected_pts:
+ raise FourDAnyoneError("Motion source-frame identity does not match the canonical clip.")
+ if (self.source_size_bytes, self.source_mtime_ns) != (
+ clip.source_size_bytes,
+ clip.source_mtime_ns,
+ ):
+ raise FourDAnyoneError("Cached GVHMR motion belongs to a different source file.")
+
+ def save(self, directory: str | Path) -> Path:
+ from safetensors.torch import save_file
+
+ self.validate()
+ root = Path(directory).expanduser().resolve()
+ root.mkdir(parents=True, exist_ok=True)
+ tensor_path = root / "motion.safetensors"
+ tensors = {
+ **{
+ f"smpl_params_global.{name}": self.smpl_params_global[name].detach().cpu().contiguous()
+ for name in SMPL_PARAMETER_NAMES
+ },
+ **{
+ f"smpl_params_incam.{name}": self.smpl_params_incam[name].detach().cpu().contiguous()
+ for name in SMPL_PARAMETER_NAMES
+ },
+ "K_fullimg": self.K_fullimg.detach().cpu().contiguous(),
+ "observed_keypoints_2d": self.observed_keypoints_2d.detach().cpu().contiguous(),
+ }
+ save_file(tensors, str(tensor_path))
+ metadata = {
+ "gvhmr_revision": self.gvhmr_revision,
+ "num_frames": self.num_frames,
+ "fps_num": self.fps.numerator,
+ "fps_den": self.fps.denominator,
+ "frame_timestamps_sec": self.frame_timestamps_sec,
+ "source_frame_indices": self.source_frame_indices,
+ "source_pts": self.source_pts,
+ "source_size_bytes": self.source_size_bytes,
+ "source_mtime_ns": self.source_mtime_ns,
+ "image_height": self.image_height,
+ "image_width": self.image_width,
+ "motion_world": self.motion_world,
+ "tensor_file": tensor_path.name,
+ }
+ (root / "motion.json").write_text(json.dumps(metadata, indent=2, sort_keys=True) + "\n")
+ return root
+
+ @classmethod
+ def load(cls, directory: str | Path) -> MotionResult:
+ from safetensors.torch import load_file
+
+ root = Path(directory).expanduser().resolve()
+ metadata = json.loads((root / "motion.json").read_text())
+ tensors = load_file(str(root / metadata["tensor_file"]), device="cpu")
+ # ``safetensors`` may keep the source file memory-mapped for as long as
+ # returned tensors own that storage. Motion results live until final
+ # export, which in turn can leave an in-use ``.efc_*`` tombstone on
+ # CPFS/NFS when scratch cleanup unlinks the backing file. The payload
+ # is tiny compared with the generated videos, so take ownership here
+ # and close the file mapping at this explicit process boundary.
+ owned = {name: tensor.clone() for name, tensor in tensors.items()}
+ del tensors
+ result = cls(
+ gvhmr_revision=str(metadata["gvhmr_revision"]),
+ fps=Fraction(int(metadata["fps_num"]), int(metadata["fps_den"])),
+ frame_timestamps_sec=tuple(float(value) for value in metadata["frame_timestamps_sec"]),
+ source_frame_indices=tuple(int(value) for value in metadata["source_frame_indices"]),
+ source_pts=tuple(None if value is None else int(value) for value in metadata["source_pts"]),
+ source_size_bytes=int(metadata["source_size_bytes"]),
+ source_mtime_ns=int(metadata["source_mtime_ns"]),
+ image_height=int(metadata["image_height"]),
+ image_width=int(metadata["image_width"]),
+ smpl_params_global={name: owned[f"smpl_params_global.{name}"] for name in SMPL_PARAMETER_NAMES},
+ smpl_params_incam={name: owned[f"smpl_params_incam.{name}"] for name in SMPL_PARAMETER_NAMES},
+ K_fullimg=owned["K_fullimg"],
+ observed_keypoints_2d=owned["observed_keypoints_2d"],
+ motion_world=str(metadata["motion_world"]),
+ )
+ result.validate()
+ return result
diff --git a/fdanyone/motion/worker.py b/fdanyone/motion/worker.py
new file mode 100644
index 0000000..bfb9751
--- /dev/null
+++ b/fdanyone/motion/worker.py
@@ -0,0 +1,31 @@
+"""Private subprocess entry point for motion inference."""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+from fdanyone.device import select_cuda_device
+from fdanyone.motion.gvhmr import run_gvhmr
+from fdanyone.video import load_canonical_working_clip
+
+
+def main(request_path: str) -> None:
+ request = json.loads(Path(request_path).read_text())
+ device, _ = select_cuda_device(request["device"])
+ clip = load_canonical_working_clip(request["working_video"], request["clip_metadata"])
+ common = {
+ "clip": clip,
+ "working_video": request["working_video"],
+ "output_dir": request["output_dir"],
+ "device": device,
+ }
+ result = run_gvhmr(gvhmr_root=request["gvhmr_root"], **common)
+ result.save(request["result_dir"])
+
+
+if __name__ == "__main__":
+ if len(sys.argv) != 2:
+ raise SystemExit("Usage: python -m fdanyone.motion.worker REQUEST.json")
+ main(sys.argv[1])
diff --git a/fdanyone/nerfstudio.py b/fdanyone/nerfstudio.py
new file mode 100644
index 0000000..3970244
--- /dev/null
+++ b/fdanyone/nerfstudio.py
@@ -0,0 +1,222 @@
+"""Export one synchronized generated frame and its foreground masks for Nerfstudio."""
+
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+
+import av
+import numpy as np
+from PIL import Image
+
+from fdanyone.assets import resolve_foreground_model
+from fdanyone.config import INFERENCE
+from fdanyone.device import select_cuda_device
+from fdanyone.download import ensure_foreground_model
+from fdanyone.errors import FourDAnyoneError
+from fdanyone.foreground import predict_foreground_masks
+from fdanyone.io import AtomicResultDirectory, write_json
+
+NERFSTUDIO_JPEG_QUALITY = 85
+# Nerfstudio casts mask pixels directly to bool, so soft BiRefNet predictions
+# must cross a real decision boundary before serialization.
+NERFSTUDIO_MASK_THRESHOLD = 128
+
+# 4DAnyone uses a right-handed Y-up world. Nerfstudio uses a right-handed
+# Z-up world, so rotate +Y onto +Z while keeping +X fixed.
+_Y_UP_TO_Z_UP = np.array(
+ [
+ [1.0, 0.0, 0.0, 0.0],
+ [0.0, 0.0, -1.0, 0.0],
+ [0.0, 1.0, 0.0, 0.0],
+ [0.0, 0.0, 0.0, 1.0],
+ ],
+ dtype=np.float64,
+)
+
+# OpenCV camera axes are +X right, +Y down, +Z forward. Nerfstudio follows
+# OpenGL/Blender: +X right, +Y up, +Z back.
+_OPENCV_TO_OPENGL = np.diag([1.0, -1.0, -1.0, 1.0])
+
+
+def camera_to_nerfstudio(camera_to_world: object) -> list[list[float]]:
+ """Convert an OpenCV/Y-up camera-to-world matrix to OpenGL/Z-up."""
+
+ matrix = np.asarray(camera_to_world, dtype=np.float64)
+ if matrix.shape != (4, 4) or not np.isfinite(matrix).all():
+ raise FourDAnyoneError("Camera-to-world must be a finite 4x4 matrix.")
+ if not np.allclose(matrix[3], [0.0, 0.0, 0.0, 1.0]):
+ raise FourDAnyoneError("Camera-to-world must be a homogeneous transform.")
+ converted = _Y_UP_TO_Z_UP @ matrix @ _OPENCV_TO_OPENGL
+ return converted.tolist()
+
+
+def _read_cameras(result: Path) -> dict:
+ path = result / "cameras.json"
+ try:
+ cameras = json.loads(path.read_text())
+ except (FileNotFoundError, json.JSONDecodeError) as exc:
+ raise FourDAnyoneError(f"Cannot read {path}: {exc}") from exc
+ if not isinstance(cameras, dict):
+ raise FourDAnyoneError("cameras.json must contain a JSON object.")
+ return cameras
+
+
+def _camera_records(rig: dict) -> list[dict]:
+ if not isinstance(rig, dict) or rig.get("camera_model") != "OPENCV":
+ raise FourDAnyoneError("cameras.json has no supported OPENCV camera rig.")
+ cameras = rig.get("cameras")
+ if not isinstance(cameras, list) or not cameras:
+ raise FourDAnyoneError("Camera rig must contain at least one camera.")
+ if [camera.get("camera_id") for camera in cameras if isinstance(camera, dict)] != list(range(len(cameras))):
+ raise FourDAnyoneError("Camera rig must be ordered by camera ID.")
+ return cameras
+
+
+def _extract_frame(video_path: Path, frame_index: int) -> np.ndarray:
+ try:
+ with av.open(str(video_path), mode="r") as container:
+ streams = container.streams.video
+ if len(streams) != 1:
+ raise FourDAnyoneError(f"Expected one video stream in {video_path}.")
+ image = next(
+ (
+ frame.to_ndarray(format="rgb24")
+ for index, frame in enumerate(container.decode(streams[0]))
+ if index == frame_index
+ ),
+ None,
+ )
+ except OSError as exc:
+ raise FourDAnyoneError(f"Cannot decode generated video {video_path}: {exc}") from exc
+ if image is None:
+ raise FourDAnyoneError(f"Video {video_path} has no frame {frame_index}.")
+ return image
+
+
+def _dense_video_paths(result: Path, cameras: list[dict]) -> tuple[Path, ...]:
+ paths = []
+ for camera in cameras:
+ camera_id = int(camera["camera_id"])
+ relative = f"videos/dense/{camera_id:02d}.mp4"
+ if camera.get("video") != relative:
+ raise FourDAnyoneError(f"Camera {camera_id:02d} points to the wrong target video.")
+ paths.append(result / relative)
+ return tuple(paths)
+
+
+def _validate_raster(image: np.ndarray, camera: dict) -> None:
+ camera_id = int(camera["camera_id"])
+ expected = (int(camera["image_height"]), int(camera["image_width"]), 3)
+ if image.dtype != np.uint8 or image.shape != expected:
+ raise FourDAnyoneError(
+ f"Camera {camera_id:02d} frame has raster {image.shape} and dtype {image.dtype}; "
+ f"expected RGB uint8 {expected}."
+ )
+
+
+def _write_images(images: tuple[np.ndarray, ...], root: Path) -> None:
+ root.mkdir()
+ for camera_id, image in enumerate(images):
+ Image.fromarray(image).save(root / f"{camera_id:02d}.jpg", format="JPEG", quality=NERFSTUDIO_JPEG_QUALITY)
+
+
+def _write_masks(masks: np.ndarray, images: tuple[np.ndarray, ...], root: Path) -> None:
+ expected_shape = (len(images), *(images[0].shape[:2]))
+ if masks.dtype != np.uint8 or masks.shape != expected_shape:
+ raise FourDAnyoneError(
+ f"BiRefNet returned masks with shape {masks.shape} and dtype {masks.dtype}; "
+ f"expected uint8 {expected_shape}."
+ )
+
+ root.mkdir()
+ for camera_id, mask in enumerate(masks):
+ binary = np.where(mask >= NERFSTUDIO_MASK_THRESHOLD, 255, 0).astype(np.uint8)
+ if not np.any(binary):
+ raise FourDAnyoneError(f"BiRefNet found no foreground in camera {camera_id:02d}.")
+ Image.fromarray(binary).save(root / f"{camera_id:02d}.png", format="PNG")
+
+
+def _transforms(cameras: list[dict]) -> dict:
+ frames = []
+ for camera in cameras:
+ camera_id = int(camera["camera_id"])
+ intrinsic = np.asarray(camera.get("K"), dtype=np.float64)
+ if intrinsic.shape != (3, 3) or not np.isfinite(intrinsic).all():
+ raise FourDAnyoneError(f"Camera {camera_id:02d} has an invalid intrinsic matrix.")
+ try:
+ width = int(camera["image_width"])
+ height = int(camera["image_height"])
+ except (KeyError, TypeError, ValueError) as exc:
+ raise FourDAnyoneError(f"Camera {camera_id:02d} has invalid image dimensions.") from exc
+ frames.append(
+ {
+ "file_path": f"images/{camera_id:02d}.jpg",
+ "mask_path": f"masks/{camera_id:02d}.png",
+ "fl_x": float(intrinsic[0, 0]),
+ "fl_y": float(intrinsic[1, 1]),
+ "cx": float(intrinsic[0, 2]),
+ "cy": float(intrinsic[1, 2]),
+ "h": height,
+ "w": width,
+ "transform_matrix": camera_to_nerfstudio(camera.get("camera_to_world")),
+ }
+ )
+ return {"camera_model": "OPENCV", "frames": frames}
+
+
+def export_nerfstudio(
+ result_dir: str,
+ frame_index: int = 0,
+ output_dir: str | None = None,
+ model_dir: str = "models",
+ device: str = "cuda:0",
+) -> dict:
+ """Export one masked multi-view timestamp for direct Nerfstudio training.
+
+ Args:
+ result_dir: A completed data/fdanyone/<clip> result.
+ frame_index: Synchronized frame index from 0 through 120.
+ output_dir: Destination; defaults to data/nerfstudio/<clip>/frame_NNN.
+ model_dir: Model root containing (or receiving) the pinned BiRefNet files.
+ device: CUDA device used for foreground segmentation.
+ """
+
+ result = Path(result_dir).expanduser().resolve()
+ if not result.is_dir():
+ raise FourDAnyoneError(f"4DAnyone result does not exist: {result}")
+ if not 0 <= frame_index < INFERENCE.num_frames:
+ raise FourDAnyoneError(f"frame_index must be in [0, {INFERENCE.num_frames - 1}], got {frame_index}.")
+ cameras = _camera_records(_read_cameras(result))
+ transforms = _transforms(cameras)
+ videos = _dense_video_paths(result, cameras)
+
+ if output_dir is None:
+ data_root = result.parent.parent if result.parent.name == "fdanyone" else result.parent
+ destination = data_root / "nerfstudio" / result.name / f"frame_{frame_index:03d}"
+ else:
+ destination = Path(output_dir).expanduser().resolve()
+ atomic = AtomicResultDirectory(destination)
+ if os.path.lexists(atomic.destination):
+ raise FourDAnyoneError(f"Nerfstudio dataset already exists: {atomic.destination}")
+
+ with atomic as work:
+ images = tuple(_extract_frame(video, frame_index) for video in videos)
+ for image, camera in zip(images, cameras, strict=True):
+ _validate_raster(image, camera)
+ _write_images(images, work / "images")
+
+ device, _ = select_cuda_device(device)
+ ensure_foreground_model(model_dir)
+ foreground_model = resolve_foreground_model(model_dir)
+ masks = predict_foreground_masks(images, foreground_model, device)
+ _write_masks(masks, images, work / "masks")
+ write_json(work / "transforms.json", transforms, sort_keys=False)
+
+ return {
+ "output_dir": str(destination),
+ "frame_index": frame_index,
+ "num_images": len(cameras),
+ "num_masks": len(cameras),
+ }
diff --git a/fdanyone/output.py b/fdanyone/output.py
new file mode 100644
index 0000000..a0fa8a4
--- /dev/null
+++ b/fdanyone/output.py
@@ -0,0 +1,214 @@
+"""Publish generated videos and their camera metadata."""
+
+from __future__ import annotations
+
+import json
+import platform
+import shutil
+import sys
+import time
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from fdanyone.config import INFERENCE
+from fdanyone.errors import FourDAnyoneError
+from fdanyone.io import write_json
+
+if TYPE_CHECKING:
+ from fdanyone.model.inference import GeneratedViews
+ from fdanyone.motion.result import MotionResult
+ from fdanyone.skeleton.pipeline import Conditioning
+ from fdanyone.video import CanonicalClip
+
+
+def _copy_file(source: Path, destination: Path) -> Path:
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copy2(source, destination)
+ return destination
+
+
+def _runtime_metadata(device: str) -> dict:
+ import torch
+
+ cuda = {
+ "available": torch.cuda.is_available(),
+ "torch_cuda": torch.version.cuda,
+ "cudnn": torch.backends.cudnn.version(),
+ }
+ if torch.cuda.is_available():
+ torch_device = torch.device(device)
+ properties = torch.cuda.get_device_properties(torch_device)
+ cuda.update(
+ {
+ "device": device,
+ "device_name": torch.cuda.get_device_name(torch_device),
+ "device_capability": list(torch.cuda.get_device_capability(torch_device)),
+ "device_total_memory_bytes": properties.total_memory,
+ }
+ )
+ return {
+ "python": sys.version.split()[0],
+ "platform": platform.platform(),
+ "torch": torch.__version__,
+ "cuda": cuda,
+ }
+
+
+def _camera_rig_payload(payload: dict, cameras: list[dict]) -> dict:
+ """Keep the final OpenCV camera rig needed by downstream tools."""
+
+ records = []
+ for camera in cameras:
+ camera_id = int(camera["camera_id"])
+ records.append(
+ {
+ "camera_id": camera_id,
+ "layer_index": int(camera["layer_index"]),
+ "pitch": int(camera["pitch_degrees"]),
+ "yaw": float(camera["yaw_degrees"]),
+ "K": camera["K"],
+ "camera_to_world": camera["camera_to_world"],
+ "image_width": int(camera["image_width"]),
+ "image_height": int(camera["image_height"]),
+ "video": f"videos/dense/{camera_id:02d}.mp4",
+ "skeleton_video": f"skeletons/{camera_id:02d}.mp4",
+ }
+ )
+ return {
+ "camera_model": "OPENCV",
+ "world_frame": payload["world_frame"],
+ "camera_frame": payload["camera_frame"],
+ "front_camera_ids": payload["front_camera_ids"],
+ "framing": payload["framing"],
+ "cameras": records,
+ }
+
+
+def _target_cameras(payload: object, expected_count: int) -> list[dict]:
+ """Read the camera records produced by the conditioning stage."""
+
+ if not isinstance(payload, dict) or payload.get("camera_model") != "OPENCV":
+ raise FourDAnyoneError("Conditioning did not produce an OpenCV camera rig.")
+ cameras = payload.get("cameras")
+ if not isinstance(cameras, list) or len(cameras) != expected_count:
+ raise FourDAnyoneError(f"Conditioning must contain {expected_count} target cameras.")
+ if [camera.get("camera_id") for camera in cameras if isinstance(camera, dict)] != list(range(expected_count)):
+ raise FourDAnyoneError("Target cameras are not in canonical order.")
+ return cameras
+
+
+def export_result(
+ *,
+ clip: CanonicalClip,
+ conditioning: Conditioning,
+ generated: GeneratedViews,
+ destination: str | Path,
+ motion: MotionResult,
+ model_identity: dict,
+ pipeline_started: float,
+) -> dict:
+ """Publish proposal, target, skeleton, camera, and metadata artifacts."""
+
+ from fdanyone.vendor.diffsynth.models.wan_video_dit import get_attention_backend
+
+ root = Path(destination).expanduser().resolve()
+ attention_backend = get_attention_backend()
+ view_plan = generated.view_plan
+ if conditioning.view_plan != view_plan:
+ raise FourDAnyoneError("Conditioning and generation resolved different view plans.")
+ if len(generated.rcp_videos) != len(view_plan.rcp_camera_ids):
+ raise FourDAnyoneError(
+ f"Generation returned {len(generated.rcp_videos)} RCP videos, expected {len(view_plan.rcp_camera_ids)}."
+ )
+ if len(generated.target_videos) != view_plan.num_target_views:
+ raise FourDAnyoneError(
+ f"Generation returned {len(generated.target_videos)} target videos, expected {view_plan.num_target_views}."
+ )
+ if len(conditioning.target_skeletons) != view_plan.num_target_views:
+ raise FourDAnyoneError(
+ f"Conditioning returned {len(conditioning.target_skeletons)} target skeletons, "
+ f"expected {view_plan.num_target_views}."
+ )
+
+ sparse_root = root / "videos" / "sparse"
+ dense_root = root / "videos" / "dense"
+ skeletons_root = root / "skeletons"
+ dense_root.mkdir(parents=True, exist_ok=False)
+ skeletons_root.mkdir(exist_ok=False)
+ if generated.rcp_videos:
+ sparse_root.mkdir(exist_ok=False)
+
+ output_sparse = tuple(
+ _copy_file(source, sparse_root / f"{camera_id:02d}.mp4")
+ for camera_id, source in zip(view_plan.rcp_camera_ids, generated.rcp_videos, strict=True)
+ )
+ output_dense = tuple(
+ _copy_file(source, dense_root / f"{camera_id:02d}.mp4")
+ for camera_id, source in enumerate(generated.target_videos)
+ )
+ for camera_id, skeleton in enumerate(conditioning.target_skeletons):
+ _copy_file(skeleton.path, skeletons_root / f"{camera_id:02d}.mp4")
+
+ camera_payload = json.loads((conditioning.root / "cameras.json").read_text())
+ conditioning_metadata = json.loads((conditioning.root / "metadata.json").read_text())
+ camera_records = _target_cameras(camera_payload, view_plan.num_target_views)
+ total_elapsed = time.monotonic() - pipeline_started
+
+ metadata = {
+ "input": {
+ "filename": clip.source_path.name,
+ "fps": f"{clip.fps_num}/{clip.fps_den}",
+ "start_time_seconds": float(clip.start_time),
+ "num_frames": len(clip.frames),
+ "width": clip.width,
+ "height": clip.height,
+ },
+ "motion": {
+ "method": "GVHMR",
+ "revision": motion.gvhmr_revision,
+ },
+ "preprocessing": {
+ "source_crop_policy": conditioning_metadata["source_crop_policy"],
+ "foreground_model": conditioning_metadata["foreground_model"],
+ "framing": conditioning_metadata["framing"],
+ "skeleton_draw_scale": conditioning_metadata["skeleton_draw_scale"],
+ },
+ "model": dict(model_identity),
+ "generation": {
+ "seed": generated.seed,
+ "view_plan": {
+ **view_plan.to_dict(),
+ "num_layers": view_plan.num_layers,
+ "num_target_views": view_plan.num_target_views,
+ "groups_per_layer": view_plan.groups_per_layer,
+ "tcr_active": view_plan.tcr_active,
+ "routing_topology": "circular" if view_plan.closed_yaw else "open",
+ },
+ "attention_backend": attention_backend,
+ "inference_steps": INFERENCE.num_inference_steps,
+ "elapsed_seconds": generated.elapsed_seconds,
+ "total_elapsed_seconds": total_elapsed,
+ "peak_vram_allocated_bytes": generated.peak_vram_allocated_bytes,
+ "peak_vram_reserved_bytes": generated.peak_vram_reserved_bytes,
+ },
+ "output": {
+ "rcp_views": len(output_sparse),
+ "target_views": len(output_dense),
+ "frames_per_video": INFERENCE.num_frames,
+ "width": INFERENCE.width,
+ "height": INFERENCE.height,
+ "fps": f"{clip.fps_num}/{clip.fps_den}",
+ },
+ "runtime": _runtime_metadata(generated.device),
+ }
+ write_json(root / "cameras.json", _camera_rig_payload(camera_payload, camera_records))
+ write_json(root / "metadata.json", metadata)
+ return {
+ "attention_backend": attention_backend,
+ "num_rcp_videos": len(output_sparse),
+ "num_target_videos": len(output_dense),
+ "fps": f"{clip.fps_num}/{clip.fps_den}",
+ "peak_vram_allocated_bytes": generated.peak_vram_allocated_bytes,
+ "peak_vram_reserved_bytes": generated.peak_vram_reserved_bytes,
+ "total_pipeline_elapsed_seconds": total_elapsed,
+ }
diff --git a/fdanyone/pipeline.py b/fdanyone/pipeline.py
new file mode 100644
index 0000000..7fa0abe
--- /dev/null
+++ b/fdanyone/pipeline.py
@@ -0,0 +1,323 @@
+"""Top-level inference orchestration."""
+
+from __future__ import annotations
+
+import logging
+import os
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+
+from fdanyone.assets import (
+ CHECKPOINT,
+ HF_REPO_ID,
+ HF_REVISION,
+ resolve_base_assets,
+ resolve_checkpoint,
+ resolve_foreground_model,
+ resolve_regressor,
+)
+from fdanyone.config import INFERENCE
+from fdanyone.device import select_cuda_device
+from fdanyone.download import ensure_example_video, ensure_models, ensure_smplx
+from fdanyone.errors import ConfigurationError
+from fdanyone.io import AtomicResultDirectory, remove_tree, write_json
+from fdanyone.motion.gvhmr import validate_gvhmr
+from fdanyone.motion.result import MotionResult
+from fdanyone.video import (
+ decode_canonical_clip,
+ validate_required_video_codecs,
+ verify_lossless_video,
+ write_gvhmr_video,
+)
+from fdanyone.views import ViewPlan, resolve_view_plan
+
+LOGGER = logging.getLogger("fdanyone")
+
+
+def _data_paths(data_dir: str, video_path: str) -> tuple[Path, Path, Path]:
+ data_root = Path(data_dir).expanduser().resolve()
+ run_name = Path(video_path).stem
+ return (
+ data_root,
+ data_root / "gvhmr" / "results" / run_name,
+ data_root / "fdanyone" / run_name,
+ )
+
+
+def _discard_scratch(path: Path) -> None:
+ """Best-effort cleanup that can never invalidate a published result.
+
+ Some network filesystems keep an open, hidden tombstone after a file is
+ unlinked. Such a tombstone may remain ``EBUSY`` until this process exits,
+ so cleanup must not be part of the atomic publication transaction.
+ """
+
+ try:
+ remove_tree(path)
+ except OSError as exc:
+ LOGGER.warning(
+ "Could not remove temporary files at %s (%s). "
+ "The result is unaffected; the hidden scratch directory can be removed after this process exits.",
+ path,
+ exc,
+ )
+
+
+def _worker_environment() -> dict[str, str]:
+ """Give the short-lived GVHMR workers this checkout and stable CUDA flags."""
+
+ environment = os.environ.copy()
+ environment.update(
+ {
+ "TORCH_CUDNN_V8_API_DISABLED": "1",
+ "CUDNN_FRONTEND_DISABLE": "1",
+ "CUDNN_LOGINFO_DBG": "0",
+ "CUDNN_LOGDEST_DBG": "stderr",
+ "CUDA_DEVICE_MAX_CONNECTIONS": "1",
+ "NVIDIA_TF32_OVERRIDE": "0",
+ }
+ )
+ environment.pop("PYTHONHOME", None)
+ environment["PYTHONPATH"] = str(Path(__file__).resolve().parent.parent)
+ return environment
+
+
+def _run_motion(
+ *,
+ working_video: Path,
+ output_dir: Path,
+ gvhmr_root: Path,
+ device: str,
+ worker_python: str,
+ clip_metadata: Path,
+):
+ output_dir.mkdir(parents=True, exist_ok=True)
+ request_path = output_dir / ".motion-worker-request.json"
+ result_dir = output_dir / "result"
+ write_json(
+ request_path,
+ {
+ "gvhmr_root": str(gvhmr_root),
+ "working_video": str(working_video),
+ "clip_metadata": str(clip_metadata),
+ "output_dir": str(output_dir / "runtime"),
+ "result_dir": str(result_dir),
+ "device": device,
+ },
+ )
+ try:
+ subprocess.run(
+ [worker_python, "-m", "fdanyone.motion.worker", str(request_path)],
+ check=True,
+ env=_worker_environment(),
+ )
+ finally:
+ request_path.unlink(missing_ok=True)
+ return MotionResult.load(result_dir)
+
+
+def _build_conditioning(
+ *,
+ regressor: Path,
+ foreground_model: Path,
+ gvhmr_root: Path,
+ output_dir: Path,
+ device: str,
+ worker_python: str,
+ working_video: Path,
+ clip_metadata: Path,
+ motion_result_dir: Path,
+ view_plan: ViewPlan,
+):
+ from fdanyone.skeleton.pipeline import Conditioning
+
+ request_path = output_dir.parent / ".skeleton-worker-request.json"
+ write_json(
+ request_path,
+ {
+ "working_video": str(working_video),
+ "clip_metadata": str(clip_metadata),
+ "motion_result_dir": str(motion_result_dir),
+ "regressor_path": str(regressor),
+ "foreground_model_path": str(foreground_model),
+ "gvhmr_root": str(gvhmr_root),
+ "output_dir": str(output_dir),
+ "device": device,
+ "view_plan": view_plan.to_dict(),
+ },
+ )
+ try:
+ subprocess.run(
+ [
+ worker_python,
+ "-m",
+ "fdanyone.skeleton.worker",
+ str(request_path),
+ ],
+ check=True,
+ env=_worker_environment(),
+ )
+ finally:
+ request_path.unlink(missing_ok=True)
+ return Conditioning.load(output_dir)
+
+
+def run_pipeline(
+ *,
+ video_path: str,
+ data_dir: str,
+ model_dir: str,
+ checkpoint_path: str | None,
+ mhr70_regressor_path: str | None,
+ gvhmr_root: str,
+ device: str,
+ start_time: float,
+ target_fps: str | int | float,
+ seed: int,
+ views_per_layer: int,
+ layer_pitches: list[int],
+ start_yaw: int,
+ yaw_span: int,
+ views_per_group: int | str,
+ enable_rcp: bool,
+ enable_tcr: bool,
+) -> dict:
+ """Execute inference and publish reusable GVHMR plus 4DAnyone results."""
+
+ pipeline_started = time.monotonic()
+ logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
+ if seed < 0:
+ raise ConfigurationError(f"seed must be non-negative, got {seed}.")
+ view_plan = resolve_view_plan(
+ views_per_layer=views_per_layer,
+ layer_pitches=layer_pitches,
+ start_yaw=start_yaw,
+ yaw_span=yaw_span,
+ views_per_group=views_per_group,
+ enable_rcp=enable_rcp,
+ enable_tcr=enable_tcr,
+ )
+ data_root, motion_dir, result_dir = _data_paths(data_dir, video_path)
+ run_name = Path(video_path).stem
+ atomic = AtomicResultDirectory(result_dir)
+ # Fail before asset resolution or video decode; the context manager
+ # checks again later in case another process creates the path.
+ if os.path.lexists(atomic.destination):
+ raise ConfigurationError(
+ f"4DAnyone result already exists: {atomic.destination}. Choose a new --data_dir or input filename."
+ )
+ validate_required_video_codecs()
+ device, _ = select_cuda_device(device)
+
+ ensure_example_video(video_path)
+ # Resolve the licensed body model before starting the much larger public
+ # model download. Interactive use continues automatically after setup;
+ # background jobs receive an actionable error instead of hanging.
+ ensure_smplx(model_dir, gvhmr_root)
+ ensure_models(model_dir, gvhmr_root)
+ gvhmr_root, gvhmr_revision = validate_gvhmr(gvhmr_root)
+ worker_python = os.path.abspath(sys.executable)
+
+ regressor = resolve_regressor(mhr70_regressor_path, model_dir=model_dir)
+ foreground_model = resolve_foreground_model(model_dir)
+ canonical_fps = None if str(target_fps).lower() == "auto" else target_fps
+ clip = decode_canonical_clip(
+ video_path,
+ num_frames=INFERENCE.num_frames,
+ start_time=start_time,
+ fps=canonical_fps,
+ )
+ data_root.mkdir(parents=True, exist_ok=True)
+ scratch = Path(tempfile.mkdtemp(prefix=f".{run_name}.scratch-", dir=data_root))
+ try:
+ clip_metadata = scratch / "canonical_clip.json"
+ clip.write_metadata(clip_metadata)
+ working_video = write_gvhmr_video(clip, scratch / "canonical_clip.mp4")
+
+ if os.path.lexists(motion_dir):
+ if motion_dir.is_symlink() or not motion_dir.is_dir():
+ raise ConfigurationError(f"GVHMR result path is not a regular directory: {motion_dir}")
+ motion = MotionResult.load(motion_dir)
+ if motion.gvhmr_revision != gvhmr_revision:
+ raise ConfigurationError(
+ f"Existing GVHMR result at {motion_dir} was produced by "
+ f"GVHMR@{motion.gvhmr_revision}, not GVHMR@{gvhmr_revision}."
+ )
+ motion.validate_against_clip(clip)
+ LOGGER.info("Reusing validated GVHMR result at %s", motion_dir)
+ else:
+ with AtomicResultDirectory(motion_dir) as motion_work:
+ motion = _run_motion(
+ working_video=working_video,
+ output_dir=scratch / "gvhmr",
+ gvhmr_root=gvhmr_root,
+ device=device,
+ worker_python=worker_python,
+ clip_metadata=clip_metadata,
+ )
+ motion.validate_against_clip(clip)
+ motion.save(motion_work)
+
+ checkpoint = resolve_checkpoint(checkpoint_path, model_dir=model_dir)
+ base_assets = resolve_base_assets(model_dir)
+ # Record the published identity only for the published checkpoint; an
+ # explicit override must not claim the frozen Hugging Face coordinates.
+ if checkpoint_path is None:
+ model_identity = {"checkpoint": CHECKPOINT, "repo_id": HF_REPO_ID, "revision": HF_REVISION}
+ else:
+ model_identity = {"checkpoint": checkpoint.name, "source": "local_override"}
+
+ with atomic as work:
+ # Heavy rendering and generation are imported only after the motion
+ # contract has been materialized, keeping CLI/help and CPU tests light.
+ from fdanyone.model.inference import generate_views
+ from fdanyone.output import export_result
+
+ conditioning = _build_conditioning(
+ regressor=regressor,
+ foreground_model=foreground_model,
+ gvhmr_root=gvhmr_root,
+ output_dir=scratch / "conditioning",
+ device=device,
+ worker_python=worker_python,
+ working_video=working_video,
+ clip_metadata=clip_metadata,
+ motion_result_dir=motion_dir,
+ view_plan=view_plan,
+ )
+ if conditioning.num_frames != len(clip.frames) or (
+ conditioning.fps_num,
+ conditioning.fps_den,
+ ) != (
+ clip.fps_num,
+ clip.fps_den,
+ ):
+ raise ConfigurationError("Skeleton conditioning does not match the canonical clip timeline.")
+ # Re-decode the worker-produced source before it becomes a model tensor.
+ verify_lossless_video(clip, conditioning.source_video)
+ generated = generate_views(
+ clip=clip,
+ conditioning=conditioning,
+ checkpoint_path=checkpoint,
+ assets=base_assets,
+ output_dir=scratch / "generation",
+ device=device,
+ seed=seed,
+ )
+ summary = export_result(
+ clip=clip,
+ conditioning=conditioning,
+ generated=generated,
+ destination=work,
+ motion=motion,
+ model_identity=model_identity,
+ pipeline_started=pipeline_started,
+ )
+ finally:
+ _discard_scratch(scratch)
+ summary["result_dir"] = str(result_dir)
+ summary["motion_dir"] = str(motion_dir)
+ return summary
diff --git a/fdanyone/skeleton/__init__.py b/fdanyone/skeleton/__init__.py
new file mode 100644
index 0000000..c0f0eff
--- /dev/null
+++ b/fdanyone/skeleton/__init__.py
@@ -0,0 +1 @@
+"""MHR70 projection and Goliath40 conditioning renderer."""
diff --git a/fdanyone/skeleton/keypoints.py b/fdanyone/skeleton/keypoints.py
new file mode 100644
index 0000000..5bae6d9
--- /dev/null
+++ b/fdanyone/skeleton/keypoints.py
@@ -0,0 +1,170 @@
+"""Minimal MHR70/Goliath40 conditioning schema.
+
+The exact keypoint names/order and no-finger link closure were extracted from
+``facebookresearch/sapiens2`` revision
+``0e51c12d7c7257d88431b2d50e523a7b03004854`` and remain subject to the
+Sapiens2 License. A copy is kept at
+``third_party/licenses/SAPIENS2_LICENSE.md``. The RGB palette, per-keypoint
+color policy, and renderer implementation are original 4DAnyone material
+licensed under Apache-2.0.
+"""
+
+from __future__ import annotations
+
+# The exact names/order below are derived from Sapiens2, Copyright (c) Meta
+# Platforms, Inc. and affiliates, under the Sapiens2 License noted above.
+
+KEYPOINT_NAMES = (
+ "nose",
+ "left-eye",
+ "right-eye",
+ "left-ear",
+ "right-ear",
+ "left-shoulder",
+ "right-shoulder",
+ "left-elbow",
+ "right-elbow",
+ "left-hip",
+ "right-hip",
+ "left-knee",
+ "right-knee",
+ "left-ankle",
+ "right-ankle",
+ "left-big-toe-tip",
+ "left-small-toe-tip",
+ "left-heel",
+ "right-big-toe-tip",
+ "right-small-toe-tip",
+ "right-heel",
+ "right-thumb-tip",
+ "right-thumb-first-joint",
+ "right-thumb-second-joint",
+ "right-thumb-third-joint",
+ "right-index-tip",
+ "right-index-first-joint",
+ "right-index-second-joint",
+ "right-index-third-joint",
+ "right-middle-tip",
+ "right-middle-first-joint",
+ "right-middle-second-joint",
+ "right-middle-third-joint",
+ "right-ring-tip",
+ "right-ring-first-joint",
+ "right-ring-second-joint",
+ "right-ring-third-joint",
+ "right-pinky-tip",
+ "right-pinky-first-joint",
+ "right-pinky-second-joint",
+ "right-pinky-third-joint",
+ "right-wrist",
+ "left-thumb-tip",
+ "left-thumb-first-joint",
+ "left-thumb-second-joint",
+ "left-thumb-third-joint",
+ "left-index-tip",
+ "left-index-first-joint",
+ "left-index-second-joint",
+ "left-index-third-joint",
+ "left-middle-tip",
+ "left-middle-first-joint",
+ "left-middle-second-joint",
+ "left-middle-third-joint",
+ "left-ring-tip",
+ "left-ring-first-joint",
+ "left-ring-second-joint",
+ "left-ring-third-joint",
+ "left-pinky-tip",
+ "left-pinky-first-joint",
+ "left-pinky-second-joint",
+ "left-pinky-third-joint",
+ "left-wrist",
+ "left-olecranon",
+ "right-olecranon",
+ "left-cubital-fossa",
+ "right-cubital-fossa",
+ "left-acromion",
+ "right-acromion",
+ "neck",
+)
+
+# First-party 4DAnyone color palette, licensed under Apache-2.0.
+BLUE = (116, 192, 252)
+GREEN = (130, 186, 129)
+ORANGE = (248, 129, 81)
+TEAL = (99, 230, 190)
+YELLOW = (255, 212, 59)
+PINK = (229, 153, 247)
+PURPLE = (177, 151, 252)
+RED = (255, 135, 135)
+
+# The schema ids and link endpoints are Sapiens2-derived. The RGB assignments
+# are first-party 4DAnyone material.
+# (schema id, first keypoint, second keypoint, RGB, major)
+LINKS = (
+ (0, 13, 11, TEAL, True),
+ (1, 11, 9, TEAL, True),
+ (2, 14, 12, YELLOW, True),
+ (3, 12, 10, YELLOW, True),
+ (4, 9, 10, BLUE, True),
+ (5, 5, 9, GREEN, True),
+ (6, 6, 10, ORANGE, True),
+ (7, 5, 6, BLUE, True),
+ (8, 5, 7, TEAL, True),
+ (9, 6, 8, YELLOW, True),
+ (10, 7, 62, TEAL, True),
+ (11, 8, 41, YELLOW, True),
+ (12, 1, 2, BLUE, True),
+ (13, 0, 1, GREEN, True),
+ (14, 0, 2, ORANGE, True),
+ (15, 1, 3, GREEN, True),
+ (16, 2, 4, ORANGE, True),
+ (17, 3, 5, GREEN, True),
+ (18, 4, 6, ORANGE, True),
+ (19, 13, 15, TEAL, True),
+ (20, 13, 16, TEAL, True),
+ (21, 13, 17, TEAL, True),
+ (22, 14, 18, YELLOW, True),
+ (23, 14, 19, YELLOW, True),
+ (24, 14, 20, YELLOW, True),
+ (25, 62, 45, YELLOW, False),
+ (29, 62, 49, PINK, False),
+ (33, 62, 53, PURPLE, False),
+ (37, 62, 57, RED, False),
+ (41, 62, 61, TEAL, False),
+ (45, 41, 24, YELLOW, False),
+ (49, 41, 28, PINK, False),
+ (53, 41, 32, PURPLE, False),
+ (57, 41, 36, RED, False),
+ (61, 41, 40, TEAL, False),
+ (65, 5, 10, PURPLE, True),
+ (66, 6, 9, PURPLE, True),
+ (67, 3, 4, PURPLE, True),
+)
+
+EXTRA_KEYPOINT_IDS = frozenset(range(63, 70))
+
+
+def keypoint_color(keypoint_id: int) -> tuple[int, int, int]:
+ name = KEYPOINT_NAMES[keypoint_id]
+ if name == "nose" or name == "neck":
+ return BLUE
+ if name in {"left-eye", "left-ear"}:
+ return GREEN
+ if name in {"right-eye", "right-ear"}:
+ return ORANGE
+ if "thumb" in name:
+ return YELLOW
+ if "forefinger" in name or "index" in name:
+ return PINK
+ if "middle" in name:
+ return PURPLE
+ if "ring" in name:
+ return RED
+ if "pinky" in name:
+ return TEAL
+ return TEAL if name.startswith("left-") else YELLOW if name.startswith("right-") else BLUE
+
+
+VISIBLE_KEYPOINT_IDS = (
+ frozenset({index for _, first, second, _, _ in LINKS for index in (first, second)}) | EXTRA_KEYPOINT_IDS
+)
diff --git a/fdanyone/skeleton/pipeline.py b/fdanyone/skeleton/pipeline.py
new file mode 100644
index 0000000..9984180
--- /dev/null
+++ b/fdanyone/skeleton/pipeline.py
@@ -0,0 +1,581 @@
+"""Build source-aware Goliath40 conditioning for multi-view generation."""
+
+from __future__ import annotations
+
+import json
+import logging
+from collections.abc import Iterable
+from contextlib import contextmanager
+from dataclasses import asdict, dataclass, replace
+from pathlib import Path
+
+import numpy as np
+
+from fdanyone.assets import BIREFNET_REPO_ID, BIREFNET_REVISION
+from fdanyone.config import CAMERA, CROP, FOREGROUND, INFERENCE, SKELETON, CameraConfig
+from fdanyone.errors import AssetError, FourDAnyoneError
+from fdanyone.foreground import predict_foreground_masks
+from fdanyone.geometry.cameras import (
+ CAMERA_FRAME,
+ WORLD_FRAME,
+ Camera,
+ camera_grid,
+ camera_ring,
+ project_points,
+ reference_intrinsics,
+)
+from fdanyone.geometry.crop import Crop, center_crop, crop_from_bounds, mask_bounds, transform_intrinsics
+from fdanyone.geometry.framing import analyze_input_framing, solve_sequence_framing
+from fdanyone.io import write_json
+from fdanyone.motion.gvhmr import gvhmr_imports, validate_gvhmr
+from fdanyone.motion.result import MotionResult
+from fdanyone.skeleton.keypoints import KEYPOINT_NAMES
+from fdanyone.skeleton.renderer import estimate_body_height, projected_body_scales, render_goliath40
+from fdanyone.vendor.pytorch3d_compat import install_if_needed as install_pytorch3d_compat
+from fdanyone.video import CanonicalClip, iter_rgb_video, write_lossless_video, write_video
+from fdanyone.views import ViewPlan
+
+LOGGER = logging.getLogger("fdanyone")
+
+
+@dataclass(frozen=True)
+class SkeletonVideo:
+ path: Path
+ crop: Crop
+
+
+@dataclass(frozen=True)
+class Conditioning:
+ root: Path
+ source_video: Path
+ source_crop: Crop
+ target_skeletons: tuple[SkeletonVideo, ...]
+ rcp_skeletons: tuple[SkeletonVideo, ...]
+ view_plan: ViewPlan
+ fps_num: int
+ fps_den: int
+ num_frames: int
+
+ def load_source_tensor(self):
+ return _video_tensor(self.source_video, self.num_frames, crop=self.source_crop)
+
+ def load_skeleton_tensor(self, skeletons: Iterable[SkeletonVideo]):
+ import torch
+
+ videos = [_video_tensor(item.path, self.num_frames, crop=item.crop) for item in skeletons]
+ return torch.cat(videos, dim=0)
+
+ @classmethod
+ def load(cls, directory: str | Path) -> Conditioning:
+ root = Path(directory).expanduser().resolve()
+ camera_payload = json.loads((root / "cameras.json").read_text())
+ metadata = json.loads((root / "metadata.json").read_text())
+ try:
+ view_plan = ViewPlan.from_dict(metadata["view_plan"])
+ except (KeyError, TypeError) as exc:
+ raise FourDAnyoneError("Conditioning artifacts have no valid view plan.") from exc
+ records = camera_payload["cameras"]
+ if [int(record["camera_id"]) for record in records] != list(range(view_plan.num_target_views)):
+ raise FourDAnyoneError("Target conditioning cameras are not in canonical order.")
+ for record, view in zip(records, view_plan.target_views, strict=True):
+ if (
+ int(record.get("layer_index", -1)) != view.layer_index
+ or int(record.get("pitch_degrees", 1000)) != view.pitch
+ or abs(float(record.get("yaw_degrees", 1000.0)) - view.yaw) > 1e-8
+ ):
+ raise FourDAnyoneError("Target conditioning cameras do not match the resolved view layout.")
+ if camera_payload.get("front_camera_ids") != list(view_plan.front_camera_ids):
+ raise FourDAnyoneError("Target conditioning has the wrong frontal-camera IDs.")
+ rcp_records = camera_payload.get("rcp_cameras", [])
+ if [int(record["camera_id"]) for record in rcp_records] != list(view_plan.rcp_camera_ids):
+ raise FourDAnyoneError("RCP conditioning cameras do not match the resolved view plan.")
+
+ def skeletons(camera_records: list[dict]) -> tuple[SkeletonVideo, ...]:
+ return tuple(
+ SkeletonVideo(root / record["skeleton_video"], Crop(**record["crop"])) for record in camera_records
+ )
+
+ target_skeletons = skeletons(records)
+ rcp_skeletons = skeletons(rcp_records)
+ required = (
+ root / metadata["source_video"],
+ *(item.path for item in (*target_skeletons, *rcp_skeletons)),
+ )
+ missing = [str(path) for path in required if not path.is_file()]
+ if missing:
+ raise FourDAnyoneError(f"Conditioning artifacts are incomplete: {missing}.")
+ return cls(
+ root=root,
+ source_video=root / metadata["source_video"],
+ source_crop=Crop(**metadata["source_crop"]),
+ target_skeletons=target_skeletons,
+ rcp_skeletons=rcp_skeletons,
+ view_plan=view_plan,
+ fps_num=int(metadata["fps_num"]),
+ fps_den=int(metadata["fps_den"]),
+ num_frames=int(metadata["num_frames"]),
+ )
+
+
+@dataclass(frozen=True)
+class _BodyGeometry:
+ vertices_world: np.ndarray
+ joints_world: np.ndarray
+ keypoints_world: np.ndarray
+ keypoints_incam: np.ndarray
+ motion_world_to_canonical_world: np.ndarray
+ regressor_metadata: dict[str, int | str]
+
+
+def _video_tensor(path: Path, num_frames: int, *, crop: Crop | None = None):
+ import torch
+ import torchvision.transforms.functional as transform
+ from torchvision.transforms import InterpolationMode
+
+ output_frames = []
+ for frame in iter_rgb_video(path):
+ tensor = torch.from_numpy(frame).permute(2, 0, 1).float().div_(255.0)
+ if crop is not None:
+ height, width = tensor.shape[-2:]
+ scaled = _scale_crop(
+ crop,
+ scale_y=height / crop.original_height,
+ scale_x=width / crop.original_width,
+ )
+ tensor = transform.crop(tensor, scaled.top, scaled.left, scaled.height, scaled.width)
+ if tensor.shape[-2:] != (crop.output_height, crop.output_width):
+ tensor = transform.resize(
+ tensor,
+ (crop.output_height, crop.output_width),
+ interpolation=InterpolationMode.BICUBIC,
+ antialias=True,
+ )
+ tensor = tensor.clamp_(0.0, 1.0)
+ output_frames.append(tensor.mul_(2.0).sub_(1.0))
+ if len(output_frames) != num_frames:
+ raise FourDAnyoneError(f"Video {path} has {len(output_frames)} decoded frames, expected {num_frames}.")
+ return torch.stack(output_frames, dim=1).unsqueeze(0).contiguous()
+
+
+def _safe_regressor_metadata(raw_metadata: object, support_shape: tuple[int, ...]) -> dict[str, int | str]:
+ del raw_metadata
+ return {
+ "format": "sparse_vertex_regressor",
+ "num_keypoints": int(support_shape[0]),
+ "support_vertices_per_keypoint": int(support_shape[1]),
+ }
+
+
+def _load_regressor(path: Path, device):
+ import torch
+
+ data = torch.load(path, map_location="cpu", weights_only=True)
+ support = data["support_vertex_ids"].detach().long().to(device)
+ weights = data["weights"].detach().float().to(device)
+ names = tuple(str(value) for value in data["keypoint_names"])
+ if support.shape != weights.shape or support.shape[0] != 70:
+ raise AssetError(f"Unexpected MHR70 regressor shapes: support={support.shape}, weights={weights.shape}.")
+ if names != KEYPOINT_NAMES:
+ raise AssetError("MHR70 regressor keypoint order does not match the frozen Goliath70 schema.")
+ return support, weights, _safe_regressor_metadata(data.get("metadata"), tuple(support.shape))
+
+
+@contextmanager
+def _gvhmr_geometry_context(gvhmr_root: Path):
+ install_pytorch3d_compat()
+ with gvhmr_imports(gvhmr_root):
+ yield
+
+
+def _body_geometry(
+ motion: MotionResult,
+ regressor_path: Path,
+ gvhmr_root: Path,
+ device: str,
+) -> _BodyGeometry:
+ import torch
+
+ body_model = gvhmr_root / "inputs/checkpoints/body_models/smplx/SMPLX_NEUTRAL.npz"
+ if not body_model.is_file():
+ raise AssetError(
+ "The licensed SMPL-X body model is missing. Run `python scripts/download_smplx.py`; "
+ f"expected the GVHMR compatibility link at {body_model}."
+ )
+ utility_root = gvhmr_root / "hmr4d/utils/body_model"
+ smplx_to_smpl_path = utility_root / "smplx2smpl_sparse.pt"
+ joint_regressor_path = utility_root / "smpl_neutral_J_regressor.pt"
+ for path in (smplx_to_smpl_path, joint_regressor_path):
+ if not path.is_file():
+ raise AssetError(f"GVHMR body-model utility is missing: {path}")
+
+ torch_device = torch.device(device)
+ support, weights, regressor_metadata = _load_regressor(regressor_path, torch_device)
+ with _gvhmr_geometry_context(gvhmr_root):
+ from hmr4d.utils.geo_transform import apply_T_on_points, compute_T_ayfz2ay
+ from hmr4d.utils.smplx_utils import make_smplx
+
+ smplx = make_smplx("supermotion").to(torch_device).eval()
+ global_parameters = {name: value.to(torch_device) for name, value in motion.smpl_params_global.items()}
+ incam_parameters = {name: value.to(torch_device) for name, value in motion.smpl_params_incam.items()}
+ with torch.inference_mode():
+ vertices_global = smplx(**global_parameters).vertices.detach()
+ vertices_incam = smplx(**incam_parameters).vertices.detach()
+ if tuple(vertices_global.shape[1:]) != (10475, 3) or vertices_incam.shape != vertices_global.shape:
+ raise FourDAnyoneError(
+ "Expected matching global/incam SMPL-X vertices [frames,10475,3], got "
+ f"{tuple(vertices_global.shape)} and {tuple(vertices_incam.shape)}."
+ )
+ keypoints_global = (vertices_global[:, support] * weights[None, :, :, None]).sum(dim=2)
+ keypoints_incam = (vertices_incam[:, support] * weights[None, :, :, None]).sum(dim=2)
+ smplx_to_smpl = torch.load(smplx_to_smpl_path, map_location=torch_device, weights_only=True)
+ joint_regressor = torch.load(joint_regressor_path, map_location=torch_device, weights_only=True)
+ vertices_smpl = torch.stack([torch.matmul(smplx_to_smpl, frame) for frame in vertices_global])
+ offset = torch.einsum("jv,vi->ji", joint_regressor, vertices_smpl[0])[0]
+ offset = offset.clone()
+ offset[1] = vertices_smpl[..., 1].min()
+ vertices_offset = vertices_smpl - offset
+ first_joints = torch.einsum("jv,lvi->lji", joint_regressor, vertices_offset[[0]])
+ transform = compute_T_ayfz2ay(first_joints, inverse=True)
+ vertices_world = apply_T_on_points(vertices_offset, transform)
+ keypoints_world = apply_T_on_points(keypoints_global - offset, transform)
+ joints_world = torch.einsum("jv,lvi->lji", joint_regressor, vertices_world)
+
+ world_transform = transform[0].detach().clone()
+ world_transform[:3, 3] -= world_transform[:3, :3] @ offset
+ result = _BodyGeometry(
+ vertices_world.detach().cpu().numpy().astype(np.float32),
+ joints_world.detach().cpu().numpy().astype(np.float32),
+ keypoints_world.detach().cpu().numpy().astype(np.float32),
+ keypoints_incam.detach().cpu().numpy().astype(np.float32),
+ world_transform.detach().cpu().numpy().astype(np.float64),
+ regressor_metadata,
+ )
+ del smplx, vertices_global, vertices_incam, vertices_smpl, vertices_world, keypoints_global, keypoints_world
+ torch.cuda.empty_cache()
+ return result
+
+
+def _front_direction(joints: np.ndarray) -> np.ndarray:
+ first = joints[0]
+ left = first[1, [0, 2]] - first[2, [0, 2]] + first[16, [0, 2]] - first[17, [0, 2]]
+ norm = float(np.linalg.norm(left))
+ if norm <= 1e-8:
+ return np.array([0.0, 0.0, -1.0], dtype=np.float64)
+ left /= norm
+ return np.array([left[1], 0.0, -left[0]], dtype=np.float64)
+
+
+def _projection_shape(height: int, width: int, max_render_height: int = 1280) -> tuple[int, int]:
+ divisor = 2
+ while height / divisor > max_render_height:
+ divisor += 1
+ return height // divisor, width // divisor
+
+
+def _output_skeleton_shape(height: int, width: int) -> tuple[int, int]:
+ while max(height, width) > INFERENCE.skeleton_max_dimension:
+ height //= 2
+ width //= 2
+ return max(2, height - height % 2), max(2, width - width % 2)
+
+
+def _scale_crop(crop: Crop, *, scale_y: float, scale_x: float) -> Crop:
+ original_height = max(1, int(round(crop.original_height * scale_y)))
+ original_width = max(1, int(round(crop.original_width * scale_x)))
+ top = min(max(0, int(round(crop.top * scale_y))), original_height - 1)
+ left = min(max(0, int(round(crop.left * scale_x))), original_width - 1)
+ height = min(max(1, int(round(crop.height * scale_y))), original_height - top)
+ width = min(max(1, int(round(crop.width * scale_x))), original_width - left)
+ return Crop(
+ top,
+ left,
+ height,
+ width,
+ original_height,
+ original_width,
+ crop.output_height,
+ crop.output_width,
+ )
+
+
+def _cropped_camera(camera: Camera, crop: Crop) -> Camera:
+ intrinsic = transform_intrinsics(np.asarray(camera.K), crop)
+ return replace(
+ camera,
+ K=tuple(tuple(float(value) for value in row) for row in intrinsic),
+ image_width=crop.output_width,
+ image_height=crop.output_height,
+ )
+
+
+def _resized_camera(camera: Camera, image_height: int, image_width: int) -> Camera:
+ intrinsic = np.asarray(camera.K, dtype=np.float64).copy()
+ intrinsic[0] *= image_width / camera.image_width
+ intrinsic[1] *= image_height / camera.image_height
+ intrinsic[2, 2] = 1.0
+ return replace(
+ camera,
+ K=tuple(tuple(float(value) for value in row) for row in intrinsic),
+ image_width=image_width,
+ image_height=image_height,
+ )
+
+
+def build_skeleton_conditioning(
+ *,
+ motion: MotionResult,
+ clip: CanonicalClip,
+ regressor_path: str | Path,
+ foreground_model_path: str | Path,
+ gvhmr_root: str | Path,
+ output_dir: str | Path,
+ device: str,
+ view_plan: ViewPlan,
+) -> Conditioning:
+ """Build source, RCP, and target conditioning on one camera grid."""
+
+ gvhmr_root, _ = validate_gvhmr(gvhmr_root)
+ root = Path(output_dir).expanduser().resolve()
+ root.mkdir(parents=True, exist_ok=True)
+
+ LOGGER.info("Estimating source foreground masks with BiRefNet")
+ masks = predict_foreground_masks(clip.rgb_frames, foreground_model_path, device)
+ geometry = _body_geometry(motion, Path(regressor_path), gvhmr_root, device)
+ input_framing = analyze_input_framing(
+ geometry.keypoints_incam,
+ KEYPOINT_NAMES,
+ motion.K_fullimg.detach().cpu().numpy(),
+ motion.observed_keypoints_2d.detach().cpu().numpy(),
+ masks,
+ )
+
+ targets = geometry.vertices_world.mean(axis=1)
+ targets[:, 1] = 0.0
+ center = targets.mean(axis=0)
+ front_direction = _front_direction(geometry.joints_world)
+ reference_K = reference_intrinsics(clip.height, clip.width)
+ framing_pitches = tuple(dict.fromkeys((*view_plan.layer_pitches, int(CAMERA.pitch_degrees))))
+
+ def camera_factory(radius: float, target_height: float) -> tuple[Camera, ...]:
+ # A full safety ring at every requested pitch keeps framing independent
+ # of view density while covering all target and canonical RCP cameras.
+ candidate_center = center.copy()
+ candidate_center[1] = target_height
+ return tuple(
+ camera
+ for layer_index, pitch in enumerate(framing_pitches)
+ for camera in camera_ring(
+ center=candidate_center,
+ front_direction=front_direction,
+ K=reference_K,
+ image_height=clip.height,
+ image_width=clip.width,
+ radius=radius,
+ target_height=target_height,
+ spec=CameraConfig(count=CAMERA.count, pitch_degrees=float(pitch)),
+ layer_index=layer_index,
+ camera_id_offset=layer_index * CAMERA.count,
+ )
+ )
+
+ projection_height, projection_width = _projection_shape(clip.height, clip.width)
+ framing = solve_sequence_framing(
+ geometry.keypoints_world,
+ KEYPOINT_NAMES,
+ input_framing,
+ camera_factory,
+ projection_width / projection_height,
+ )
+ LOGGER.info(
+ "Adaptive framing: input=%s confidence=%.3f radius=%.3f target_height=%.3f f/H=%.3f",
+ input_framing.label,
+ input_framing.confidence,
+ framing.radius,
+ framing.target_height,
+ framing.focal_normalized,
+ )
+ center[1] = framing.target_height
+ raw_intrinsic = reference_intrinsics(
+ clip.height,
+ clip.width,
+ focal_normalized=framing.focal_normalized,
+ )
+ raw_target_cameras = camera_grid(
+ center=center,
+ front_direction=front_direction,
+ K=raw_intrinsic,
+ image_height=clip.height,
+ image_width=clip.width,
+ radius=framing.radius,
+ target_height=framing.target_height,
+ views_per_layer=view_plan.views_per_layer,
+ layer_pitches=view_plan.layer_pitches,
+ start_yaw=view_plan.start_yaw,
+ yaw_span=view_plan.yaw_span,
+ )
+
+ source_crop = crop_from_bounds(
+ bounds=mask_bounds(masks, CROP.mask_threshold),
+ image_height=clip.height,
+ image_width=clip.width,
+ output_height=INFERENCE.height,
+ output_width=INFERENCE.width,
+ margins=CROP.margins,
+ allow_upscale=CROP.allow_upscale,
+ )
+ target_crop = center_crop(clip.height, clip.width, INFERENCE.height, INFERENCE.width)
+ source_video = write_lossless_video(clip, root / "source.mkv")
+ skeleton_root = root / "goliath40"
+ skeleton_root.mkdir()
+ skeleton_height, skeleton_width = _output_skeleton_shape(clip.height, clip.width)
+ body_height = estimate_body_height(geometry.keypoints_world, KEYPOINT_NAMES)
+ np.save(root / "keypoints_3d.npy", geometry.keypoints_world)
+
+ def render_skeleton(camera: Camera, path: Path) -> Path:
+ projected, depths = [], []
+ for frame_points in geometry.keypoints_world:
+ xy, depth, _ = project_points(frame_points, camera)
+ projected.append(xy)
+ depths.append(depth)
+ keypoints_2d = np.stack(projected)
+ keypoint_depths = np.stack(depths)
+ body_scales = projected_body_scales(
+ keypoint_depths,
+ KEYPOINT_NAMES,
+ body_height,
+ float(np.sqrt(raw_intrinsic[0, 0] * raw_intrinsic[1, 1])),
+ )
+
+ def rendered_frames(
+ points=keypoints_2d,
+ point_depths=keypoint_depths,
+ scales=body_scales,
+ ):
+ scores = np.ones(len(KEYPOINT_NAMES), dtype=np.float32)
+ for frame_index in range(motion.num_frames):
+ yield render_goliath40(
+ points[frame_index],
+ point_depths[frame_index],
+ scores,
+ canvas_height=clip.height,
+ canvas_width=clip.width,
+ output_height=skeleton_height,
+ output_width=skeleton_width,
+ body_scale_px=float(scales[frame_index]),
+ )
+
+ return write_video(
+ rendered_frames(),
+ path,
+ clip.fps,
+ crf=INFERENCE.skeleton_h264_crf,
+ preset=INFERENCE.h264_preset,
+ )
+
+ target_paths = tuple(
+ render_skeleton(camera, skeleton_root / f"{camera.camera_id:02d}.mp4") for camera in raw_target_cameras
+ )
+ cropped_target_cameras = tuple(_cropped_camera(camera, target_crop) for camera in raw_target_cameras)
+
+ if not view_plan.enable_rcp:
+ raw_rcp_cameras: tuple[Camera, ...] = ()
+ rcp_paths: tuple[Path, ...] = ()
+ cropped_rcp_cameras: tuple[Camera, ...] = ()
+ elif view_plan.is_canonical_target_ring:
+ raw_rcp_cameras = tuple(raw_target_cameras[camera_id] for camera_id in view_plan.rcp_camera_ids)
+ rcp_paths = tuple(target_paths[camera_id] for camera_id in view_plan.rcp_camera_ids)
+ cropped_rcp_cameras = tuple(cropped_target_cameras[camera_id] for camera_id in view_plan.rcp_camera_ids)
+ else:
+ canonical_cameras = camera_ring(
+ center=center,
+ front_direction=front_direction,
+ K=raw_intrinsic,
+ image_height=clip.height,
+ image_width=clip.width,
+ radius=framing.radius,
+ target_height=framing.target_height,
+ layer_index=-1,
+ )
+ raw_rcp_cameras = tuple(canonical_cameras[camera_id] for camera_id in view_plan.rcp_camera_ids)
+ rcp_root = root / "rcp_goliath40"
+ rcp_root.mkdir()
+ rcp_paths = tuple(
+ render_skeleton(camera, rcp_root / f"{camera.camera_id:02d}.mp4") for camera in raw_rcp_cameras
+ )
+ cropped_rcp_cameras = tuple(_cropped_camera(camera, target_crop) for camera in raw_rcp_cameras)
+
+ def camera_records(
+ raw_cameras: tuple[Camera, ...],
+ cropped_cameras: tuple[Camera, ...],
+ paths: tuple[Path, ...],
+ ) -> list[dict]:
+ return [
+ {
+ **camera.to_dict(),
+ "crop": asdict(target_crop),
+ "raw_camera": raw.to_dict(),
+ "skeleton_camera": _resized_camera(raw, skeleton_height, skeleton_width).to_dict(),
+ "skeleton_video": path.relative_to(root).as_posix(),
+ }
+ for raw, camera, path in zip(raw_cameras, cropped_cameras, paths, strict=True)
+ ]
+
+ framing_payload = framing.to_dict()
+ camera_payload = {
+ "camera_model": "OPENCV",
+ "world_frame": WORLD_FRAME,
+ "camera_frame": CAMERA_FRAME,
+ "front_camera_ids": list(view_plan.front_camera_ids),
+ "motion_world": motion.motion_world,
+ "motion_world_to_canonical_world": geometry.motion_world_to_canonical_world.tolist(),
+ "ring_center": center.tolist(),
+ "framing": framing_payload,
+ "cameras": camera_records(raw_target_cameras, cropped_target_cameras, target_paths),
+ "rcp_cameras": camera_records(raw_rcp_cameras, cropped_rcp_cameras, rcp_paths),
+ }
+ write_json(root / "cameras.json", camera_payload)
+ write_json(
+ root / "metadata.json",
+ {
+ "view_plan": view_plan.to_dict(),
+ "num_frames": motion.num_frames,
+ "fps_num": clip.fps_num,
+ "fps_den": clip.fps_den,
+ "source_video": source_video.name,
+ "source_crop": asdict(source_crop),
+ "source_crop_policy": {
+ "subject": "fmask",
+ "threshold": CROP.mask_threshold,
+ "margins": CROP.margins,
+ "allow_upscale": CROP.allow_upscale,
+ },
+ "foreground_model": {
+ "repo_id": BIREFNET_REPO_ID,
+ "revision": BIREFNET_REVISION,
+ "image_size": FOREGROUND.image_size,
+ "batch_size": FOREGROUND.batch_size,
+ },
+ "framing": framing_payload,
+ "regressor_metadata": geometry.regressor_metadata,
+ "keypoint_names": KEYPOINT_NAMES,
+ "visible_keypoint_set": "goliath40",
+ "skeleton_codec_boundary": f"libx264_crf{INFERENCE.skeleton_h264_crf}",
+ "skeleton_canvas": {"height": skeleton_height, "width": skeleton_width},
+ "skeleton_draw_scale": {
+ "mode": "kp3d",
+ "body_height_3d": body_height,
+ "body_reference_px": SKELETON.draw_body_reference_px,
+ },
+ },
+ )
+ return Conditioning(
+ root=root,
+ source_video=source_video,
+ source_crop=source_crop,
+ target_skeletons=tuple(SkeletonVideo(path, target_crop) for path in target_paths),
+ rcp_skeletons=tuple(SkeletonVideo(path, target_crop) for path in rcp_paths),
+ view_plan=view_plan,
+ fps_num=clip.fps_num,
+ fps_den=clip.fps_den,
+ num_frames=motion.num_frames,
+ )
diff --git a/fdanyone/skeleton/renderer.py b/fdanyone/skeleton/renderer.py
new file mode 100644
index 0000000..09e3d7f
--- /dev/null
+++ b/fdanyone/skeleton/renderer.py
@@ -0,0 +1,230 @@
+"""Dependency-light, depth-aware Goliath40 rasterization."""
+
+from __future__ import annotations
+
+import math
+
+import cv2
+import numpy as np
+
+from fdanyone.config import INFERENCE, SKELETON
+from fdanyone.skeleton.keypoints import EXTRA_KEYPOINT_IDS, LINKS, VISIBLE_KEYPOINT_IDS, keypoint_color
+
+_BODY_SCALE_SEGMENTS = (
+ (("left-shoulder",), ("right-shoulder",), 0.26),
+ (("left-hip",), ("right-hip",), 0.18),
+ (("left-shoulder", "right-shoulder"), ("left-hip", "right-hip"), 0.32),
+ (("left-shoulder",), ("left-hip",), 0.32),
+ (("right-shoulder",), ("right-hip",), 0.32),
+ (("left-shoulder",), ("left-elbow",), 0.19),
+ (("right-shoulder",), ("right-elbow",), 0.19),
+ (("left-elbow",), ("left-wrist",), 0.16),
+ (("right-elbow",), ("right-wrist",), 0.16),
+ (("left-hip",), ("left-knee",), 0.245),
+ (("right-hip",), ("right-knee",), 0.245),
+ (("left-knee",), ("left-ankle",), 0.245),
+ (("right-knee",), ("right-ankle",), 0.245),
+)
+_BODY_CENTER_NAMES = ("left-shoulder", "right-shoulder", "left-hip", "right-hip")
+
+
+def _name_index(names) -> dict[str, int]:
+ normalized = [str(name).strip().lower().replace("_", "-") for name in names]
+ mapping = dict(zip(normalized, range(len(normalized)), strict=True))
+ if len(mapping) != len(normalized):
+ raise ValueError("Keypoint names must be unique after normalization.")
+ return mapping
+
+
+def estimate_body_height(keypoints_3d: np.ndarray, names) -> float:
+ """Robustly infer physical body height from stable anatomical segments."""
+
+ points = np.asarray(keypoints_3d, dtype=np.float64)
+ if points.ndim != 3 or points.shape[1:] != (len(names), 3):
+ raise ValueError(f"Expected keypoints [frames,{len(names)},3], got {points.shape}.")
+ mapping = _name_index(names)
+ required = {name for first, second, _ in _BODY_SCALE_SEGMENTS for group in (first, second) for name in group}
+ missing = sorted(required - mapping.keys())
+ if missing:
+ raise ValueError(f"Missing body-scale keypoints: {missing}.")
+
+ frame_heights = []
+ for frame in points:
+ candidates = []
+ for first, second, height_ratio in _BODY_SCALE_SEGMENTS:
+ point_a = frame[[mapping[name] for name in first]].mean(axis=0)
+ point_b = frame[[mapping[name] for name in second]].mean(axis=0)
+ length = float(np.linalg.norm(point_a - point_b))
+ if np.isfinite(length) and length > 0:
+ candidates.append(length / height_ratio)
+ if candidates:
+ frame_heights.append(float(np.median(candidates)))
+ heights = np.asarray(frame_heights, dtype=np.float64)
+ heights = heights[np.isfinite(heights) & (heights > 0)]
+ if not heights.size:
+ raise ValueError("Cannot estimate body height from the supplied keypoints.")
+ if heights.size >= 5:
+ lower, upper = np.percentile(heights, [10.0, 90.0])
+ trimmed = heights[(heights >= lower) & (heights <= upper)]
+ if trimmed.size:
+ heights = trimmed
+ return float(np.median(heights))
+
+
+def projected_body_scales(
+ keypoint_depths: np.ndarray,
+ names,
+ body_height_3d: float,
+ focal_px: float,
+) -> np.ndarray:
+ """Convert physical body height and camera depth to per-frame pixel scale."""
+
+ depths = np.asarray(keypoint_depths, dtype=np.float64)
+ if depths.ndim != 2 or depths.shape[1] != len(names):
+ raise ValueError(f"Expected keypoint depths [frames,{len(names)}], got {depths.shape}.")
+ if not np.isfinite(body_height_3d) or body_height_3d <= 0:
+ raise ValueError("body_height_3d must be positive and finite.")
+ if not np.isfinite(focal_px) or focal_px <= 0:
+ raise ValueError("focal_px must be positive and finite.")
+ mapping = _name_index(names)
+ missing = [name for name in _BODY_CENTER_NAMES if name not in mapping]
+ if missing:
+ raise ValueError(f"Missing body-center keypoints: {missing}.")
+ center_depths = depths[:, [mapping[name] for name in _BODY_CENTER_NAMES]]
+ valid = np.isfinite(center_depths) & (center_depths > 1e-6)
+ counts = valid.sum(axis=1)
+ mean_depths = np.full(depths.shape[0], np.nan, dtype=np.float64)
+ enough = counts >= 2
+ mean_depths[enough] = np.where(valid, center_depths, 0.0).sum(axis=1)[enough] / counts[enough]
+ scales = np.full(depths.shape[0], np.nan, dtype=np.float32)
+ scales[enough] = (body_height_3d * focal_px / mean_depths[enough]).astype(np.float32)
+ return scales
+
+
+def _draw_point(z_buffer, canvas, point, depth, radius, color) -> None:
+ if not np.isfinite(depth) or depth <= 0:
+ return
+ height, width = z_buffer.shape
+ x, y = point
+ radius = max(1, int(radius))
+ x0, x1 = max(0, x - radius), min(width, x + radius + 1)
+ y0, y1 = max(0, y - radius), min(height, y + radius + 1)
+ if x0 >= x1 or y0 >= y1:
+ return
+ yy, xx = np.ogrid[y0:y1, x0:x1]
+ mask = (xx - x) ** 2 + (yy - y) ** 2 <= radius**2
+ view = z_buffer[y0:y1, x0:x1]
+ update = mask & (depth <= view)
+ view[update] = depth
+ canvas[y0:y1, x0:x1][update] = color
+
+
+def _draw_line(z_buffer, canvas, p1, p2, d1, d2, thickness, color) -> None:
+ if not np.isfinite(d1) or not np.isfinite(d2) or max(d1, d2) <= 0:
+ return
+ height, width = z_buffer.shape
+ x1, y1 = p1
+ x2, y2 = p2
+ dx, dy = float(x2 - x1), float(y2 - y1)
+ length_sq = dx * dx + dy * dy
+ if length_sq < 1e-6:
+ _draw_point(z_buffer, canvas, p1, (d1 + d2) / 2.0, max(1, thickness // 2), color)
+ return
+ radius = max(0.5, float(thickness) / 2.0)
+ pad = int(math.ceil(radius)) + 1
+ x0, x3 = max(0, min(x1, x2) - pad), min(width, max(x1, x2) + pad + 1)
+ y0, y3 = max(0, min(y1, y2) - pad), min(height, max(y1, y2) + pad + 1)
+ if x0 >= x3 or y0 >= y3:
+ return
+ yy, xx = np.ogrid[y0:y3, x0:x3]
+ t = np.clip(((xx - x1) * dx + (yy - y1) * dy) / length_sq, 0.0, 1.0)
+ mask = (xx - (x1 + t * dx)) ** 2 + (yy - (y1 + t * dy)) ** 2 <= radius**2
+ depth = d1 + t * (d2 - d1)
+ view = z_buffer[y0:y3, x0:x3]
+ update = mask & (depth > 0) & (depth <= view)
+ view[update] = depth[update]
+ canvas[y0:y3, x0:x3][update] = color
+
+
+def render_goliath40(
+ keypoints: np.ndarray,
+ depths: np.ndarray,
+ scores: np.ndarray,
+ *,
+ canvas_height: int,
+ canvas_width: int,
+ output_height: int,
+ output_width: int,
+ score_threshold: float = 0.3,
+ body_scale_px: float | None = None,
+) -> np.ndarray:
+ """Render one RGB frame using the reference Sapiens2 sizing rules."""
+
+ canvas_scale = max(1.0, INFERENCE.skeleton_max_dimension / max(output_height, output_width))
+ render_height = int(round(output_height * canvas_scale))
+ render_width = int(round(output_width * canvas_scale))
+ points = np.asarray(keypoints, dtype=np.float32).copy()
+ points[:, 0] *= render_width / canvas_width
+ points[:, 1] *= render_height / canvas_height
+ depths = np.asarray(depths, dtype=np.float32)
+ scores = np.asarray(scores, dtype=np.float32)
+ # Keep the reference renderer's BGR draw -> resize -> RGB conversion order.
+ # Resizing a channel-permuted uint8 image can differ by one LSB in OpenCV's
+ # optimized interpolation kernels, so drawing directly in RGB is not quite
+ # byte-exact even though the colors are semantically identical.
+ canvas = np.zeros((render_height, render_width, 3), dtype=np.uint8)
+ z_buffer = np.full((render_height, render_width), np.inf, dtype=np.float32)
+ line_scale = render_height / 1024.0
+ if body_scale_px is not None and np.isfinite(body_scale_px) and body_scale_px > 0:
+ keypoint_scale = render_height / canvas_height
+ line_scale = max(
+ 0.25,
+ float(body_scale_px) * keypoint_scale / SKELETON.draw_body_reference_px,
+ )
+ base_radius = max(1, int(round(2 * line_scale)))
+ base_thickness = max(1, int(round(2 * line_scale)))
+ point_items: dict[int, tuple[tuple[int, int], float, int, tuple[int, int, int]]] = {}
+
+ def remember(index: int, radius: int) -> None:
+ if scores[index] < score_threshold or not np.isfinite(points[index]).all():
+ return
+ item = (
+ (int(round(points[index, 0])), int(round(points[index, 1]))),
+ float(depths[index]),
+ radius,
+ keypoint_color(index)[::-1],
+ )
+ if index not in point_items or radius > point_items[index][2]:
+ point_items[index] = item
+
+ for _, first, second, color, major in LINKS:
+ if min(float(scores[first]), float(scores[second])) < score_threshold:
+ continue
+ if not np.isfinite(points[[first, second]]).all():
+ continue
+ p1 = (int(round(points[first, 0])), int(round(points[first, 1])))
+ p2 = (int(round(points[second, 0])), int(round(points[second, 1])))
+ thickness = base_thickness * (2 if major else 1)
+ _draw_line(
+ z_buffer,
+ canvas,
+ p1,
+ p2,
+ float(depths[first]),
+ float(depths[second]),
+ thickness,
+ color[::-1],
+ )
+ radius = max(1, int(round(base_radius * (1.75 if major else 1.0))))
+ remember(first, radius)
+ remember(second, radius)
+
+ for index in EXTRA_KEYPOINT_IDS:
+ remember(index, max(2, int(round(base_radius * 1.5))))
+ for index, (point, depth, radius, color) in point_items.items():
+ if index in VISIBLE_KEYPOINT_IDS:
+ _draw_point(z_buffer, canvas, point, depth, radius, color)
+ if (render_height, render_width) != (output_height, output_width):
+ canvas = cv2.resize(canvas, (output_width, output_height), interpolation=cv2.INTER_AREA)
+ canvas = cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB)
+ return np.ascontiguousarray(canvas)
diff --git a/fdanyone/skeleton/worker.py b/fdanyone/skeleton/worker.py
new file mode 100644
index 0000000..6fee93c
--- /dev/null
+++ b/fdanyone/skeleton/worker.py
@@ -0,0 +1,36 @@
+"""Private subprocess entry point for licensed body-model conditioning."""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+from fdanyone.device import select_cuda_device
+from fdanyone.motion.result import MotionResult
+from fdanyone.skeleton.pipeline import build_skeleton_conditioning
+from fdanyone.video import load_canonical_working_clip
+from fdanyone.views import ViewPlan
+
+
+def main(request_path: str) -> None:
+ request = json.loads(Path(request_path).read_text())
+ device, _ = select_cuda_device(request["device"])
+ clip = load_canonical_working_clip(request["working_video"], request["clip_metadata"])
+ motion = MotionResult.load(request["motion_result_dir"])
+ build_skeleton_conditioning(
+ motion=motion,
+ clip=clip,
+ regressor_path=request["regressor_path"],
+ foreground_model_path=request["foreground_model_path"],
+ gvhmr_root=request["gvhmr_root"],
+ output_dir=request["output_dir"],
+ device=device,
+ view_plan=ViewPlan.from_dict(request["view_plan"]),
+ )
+
+
+if __name__ == "__main__":
+ if len(sys.argv) != 2:
+ raise SystemExit("Usage: python -m fdanyone.skeleton.worker REQUEST.json")
+ main(sys.argv[1])
diff --git a/fdanyone/vendor/__init__.py b/fdanyone/vendor/__init__.py
new file mode 100644
index 0000000..62562d9
--- /dev/null
+++ b/fdanyone/vendor/__init__.py
@@ -0,0 +1 @@
+"""Third-party inference code redistributed with its upstream notices."""
diff --git a/fdanyone/vendor/diffsynth/LICENSE b/fdanyone/vendor/diffsynth/LICENSE
new file mode 100644
index 0000000..0e5a49f
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [2023] [Zhongjie Duan]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/fdanyone/vendor/diffsynth/UPSTREAM.md b/fdanyone/vendor/diffsynth/UPSTREAM.md
new file mode 100644
index 0000000..2fa8f9b
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/UPSTREAM.md
@@ -0,0 +1,23 @@
+# DiffSynth-Studio provenance
+
+This directory is a deliberately small inference-only extract from [modelscope/DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), licensed under Apache-2.0.
+
+- Public base revision: `04e39f7de53df7276a7b40ca1791c2a393e05ff3`
+- Research fork revision used by the original experiment: `c00782d90c872c97bda4745a9e6a41a0a4a7c4db`
+- `UPSTREAM.patch` SHA-256: `178e6035e451f94a2122fa4d2c876a488546964768b90275549cbf609f97daba`
+- Extracted: 2026-07-16
+
+The research fork revision was not anonymously reachable when the release contract was audited. `UPSTREAM.patch` therefore records the exact binary-safe diff from the public base to the research revision for the retained Wan/SpaTem/scheduler source files. Unrelated research-fork changes are deliberately excluded. `VENDORED_FILES.txt` is the reviewable extraction manifest.
+
+4DAnyone subsequently removed registry, downloader, training, image-encoder, and unrelated pipeline surfaces; changed imports to the local vendored package; exposes direct, typed model construction in first-party code; and exposes the implementation selected by the retained automatic attention fallback chain for run metadata. These pruning and observability changes do not alter the retained Wan/SpaTem architecture or scheduler math.
+
+To reproduce the retained research sources before pruning:
+
+```bash
+git clone https://github.com/modelscope/DiffSynth-Studio.git
+git -C DiffSynth-Studio checkout 04e39f7de53df7276a7b40ca1791c2a393e05ff3
+git -C DiffSynth-Studio apply --check /path/to/UPSTREAM.patch
+git -C DiffSynth-Studio apply /path/to/UPSTREAM.patch
+```
+
+The patch contains research-code additions and is itself source code. It must remain covered by this directory's Apache-2.0 `LICENSE` and attribution.
diff --git a/fdanyone/vendor/diffsynth/UPSTREAM.patch b/fdanyone/vendor/diffsynth/UPSTREAM.patch
new file mode 100644
index 0000000..c1906b3
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/UPSTREAM.patch
@@ -0,0 +1,1334 @@
+diff --git a/diffsynth/models/wan_video_dit.py b/diffsynth/models/wan_video_dit.py
+index 1a54728..b663722 100644
+--- a/diffsynth/models/wan_video_dit.py
++++ b/diffsynth/models/wan_video_dit.py
+@@ -3,7 +3,7 @@ import torch.nn as nn
+ import torch.nn.functional as F
+ import math
+ from typing import Tuple, Optional
+-from einops import rearrange
++from einops import rearrange, repeat
+ from .utils import hash_state_dict_keys
+ from .wan_video_camera_controller import SimpleAdapter
+ try:
+@@ -23,7 +23,12 @@ try:
+ SAGE_ATTN_AVAILABLE = True
+ except ModuleNotFoundError:
+ SAGE_ATTN_AVAILABLE = False
+-
++
++try:
++ from spas_sage_attn import spas_sage2_attn_meansim_topk_cuda
++ SPARGE_ATTN_AVAILABLE = True
++except ModuleNotFoundError:
++ SPARGE_ATTN_AVAILABLE = False
+
+ def flash_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, num_heads: int, compatibility_mode=False):
+ if compatibility_mode:
+@@ -46,6 +51,13 @@ def flash_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, num_heads
+ v = rearrange(v, "b s (n d) -> b s n d", n=num_heads)
+ x = flash_attn.flash_attn_func(q, k, v)
+ x = rearrange(x, "b s n d -> b s (n d)", n=num_heads)
++ # TODO: test spas_sage2_attn_meansim_topk_cuda
++ # elif SPARGE_ATTN_AVAILABLE:
++ # q = rearrange(q, "b s (n d) -> b n s d", n=num_heads)
++ # k = rearrange(k, "b s (n d) -> b n s d", n=num_heads)
++ # v = rearrange(v, "b s (n d) -> b n s d", n=num_heads)
++ # x = spas_sage2_attn_meansim_topk_cuda(q, k, v, simthreshd1=-0.1, topk=0.5, pvthreshd=15, is_causal=False)
++ # x = rearrange(x, "b n s d -> b s (n d)", n=num_heads)
+ elif SAGE_ATTN_AVAILABLE:
+ q = rearrange(q, "b s (n d) -> b n s d", n=num_heads)
+ k = rearrange(k, "b s (n d) -> b n s d", n=num_heads)
+@@ -186,6 +198,32 @@ class CrossAttention(nn.Module):
+ return self.o(x)
+
+
++class CrossAttentionSrcCam(nn.Module):
++ def __init__(self, dim: int, num_heads: int, eps: float = 1e-6):
++ super().__init__()
++ self.dim = dim
++ self.num_heads = num_heads
++ self.head_dim = dim // num_heads
++
++ self.q = nn.Linear(dim, dim)
++ self.k = nn.Linear(dim, dim)
++ self.v = nn.Linear(dim, dim)
++ self.o = nn.Linear(dim, dim)
++ self.norm_q = RMSNorm(dim, eps=eps)
++ self.norm_k = RMSNorm(dim, eps=eps)
++
++ self.attn = AttentionModule(self.num_heads)
++
++ def forward(self, x, x_src, freqs, freqs_src):
++ q = self.norm_q(self.q(x))
++ k = self.norm_k(self.k(x_src))
++ v = self.v(x_src)
++ q = rope_apply(q, freqs, self.num_heads)
++ k = rope_apply(k, freqs_src, self.num_heads)
++ x = self.attn(q, k, v)
++ return self.o(x)
++
++
+ class GateModule(nn.Module):
+ def __init__(self,):
+ super().__init__()
+@@ -200,6 +238,13 @@ class DiTBlock(nn.Module):
+ self.num_heads = num_heads
+ self.ffn_dim = ffn_dim
+
++ self.disable_video_attn = False
++ self.use_4d_attn = False
++ self.use_mvs_attn = False
++ self.use_src_self_attn = False
++ self.use_src_cross_attn = False
++ self.use_cam_encoder = False
++
+ self.self_attn = SelfAttention(dim, num_heads, eps)
+ self.cross_attn = CrossAttention(
+ dim, num_heads, eps, has_image_input=has_image_input)
+@@ -211,7 +256,10 @@ class DiTBlock(nn.Module):
+ self.modulation = nn.Parameter(torch.randn(1, 6, dim) / dim**0.5)
+ self.gate = GateModule()
+
+- def forward(self, x, context, t_mod, freqs):
++ def forward(self, x, x_src, context, t_mod, freqs, freqs_mvs, freqs_src, cam_emb, shape):
++ # breakpoint()
++ v, f, h, w = shape
++
+ has_seq = len(t_mod.shape) == 4
+ chunk_dim = 2 if has_seq else 1
+ # msa: multi-head self-attention mlp: multi-layer perceptron
+@@ -222,12 +270,80 @@ class DiTBlock(nn.Module):
+ shift_msa.squeeze(2), scale_msa.squeeze(2), gate_msa.squeeze(2),
+ shift_mlp.squeeze(2), scale_mlp.squeeze(2), gate_mlp.squeeze(2),
+ )
+- input_x = modulate(self.norm1(x), shift_msa, scale_msa)
+- x = self.gate(x, gate_msa, self.self_attn(input_x, freqs))
++
++ # video self-attention
++ if not self.disable_video_attn:
++ input_x = modulate(self.norm1(x), shift_msa, scale_msa)
++ if self.use_4d_attn:
++ input_x = rearrange(input_x, "v fhw c -> (v fhw) c").unsqueeze(0)
++ freqs = repeat(freqs, "fhw 1 c -> (v fhw) 1 c", v=v)
++ input_x = self.self_attn(input_x, freqs)
++ if self.use_4d_attn:
++ input_x = rearrange(input_x.squeeze(0), "(v fhw) c -> v fhw c", v=v)
++ x = self.gate(x, gate_msa, input_x)
++
++ # source-view self-attention
++ if self.use_src_self_attn:
++ x_cat = torch.cat([x, x_src], dim=1)
++ shift_src, scale_src, gate_src = (
++ self.modulation_src.to(dtype=t_mod.dtype, device=t_mod.device) + t_mod[:, :3, :]).chunk(3, dim=1)
++ input_x_cat = modulate(self.norm1_src(x_cat), shift_src, scale_src)
++
++ freqs_cat = torch.cat([freqs, freqs_src], dim=0)
++ input_x_cat = self.self_attn_src(input_x_cat, freqs_cat)
++ x_cat = self.gate(x_cat, gate_src, input_x_cat)
++ len_src = x_src.shape[1]
++ x, x_src = x_cat[:, :-len_src, ...], x_cat[:, -len_src:, ...]
++
++ # source-view cross-attention
++ if self.use_src_cross_attn:
++ shift_src, scale_src, gate_src = (
++ self.modulation_src.to(dtype=t_mod.dtype, device=t_mod.device) + t_mod[:, :3, :]).chunk(3, dim=1)
++ input_x = modulate(self.norm1_src(x), shift_src, scale_src)
++ input_x_src = self.norm1_src(x_src)
++
++ input_x = self.cross_attn_src(input_x, input_x_src, freqs, freqs_src)
++ x = self.gate(x, gate_src, input_x)
++
++ # multiview self-attention
++ if self.use_mvs_attn:
++ shift_mvs, scale_mvs, gate_mvs = (
++ self.modulation_mvs.to(dtype=t_mod.dtype, device=t_mod.device) + t_mod[:, :3, :]).chunk(3, dim=1)
++ input_x = modulate(self.norm1_mvs(x), shift_mvs, scale_mvs)
++
++ # add camera embedding before multiview attention
++ if self.use_cam_encoder and cam_emb is not None:
++ cam_proj = self.cam_encoder(cam_emb) # (v, 1, dim)
++ cam_proj = cam_proj.unsqueeze(2).unsqueeze(3).expand(-1, f, h, w, -1) # (v, f, h, w, dim)
++ cam_proj = rearrange(cam_proj, "v f h w d -> v (f h w) d")
++ input_x = input_x + cam_proj
++
++ input_x = rearrange(input_x, "v (f h w) c -> f (v h w) c", v=v, f=f, h=h, w=w)
++ input_x = self.self_attn_mvs(input_x, freqs_mvs)
++ input_x = rearrange(input_x, "f (v h w) c -> v (f h w) c", v=v, f=f, h=h, w=w)
++
++ # projector wraps multiview attention output
++ if self.use_cam_encoder:
++ input_x = self.projector(input_x)
++
++ x = self.gate(x, gate_mvs, input_x)
++
++ # prompt cross-attention
++ context = repeat(context, "1 l c -> v l c", v=x.shape[0])
+ x = x + self.cross_attn(self.norm3(x), context)
++
++ # feed-forward network
+ input_x = modulate(self.norm2(x), shift_mlp, scale_mlp)
+ x = self.gate(x, gate_mlp, self.ffn(input_x))
+- return x
++
++ if self.use_src_self_attn:
++ # for src self-attention, the x_src is updated as well
++ x_src = x_src + self.cross_attn(self.norm3(x_src), context)
++
++ input_x_src = modulate(self.norm2(x_src), shift_mlp, scale_mlp)
++ x_src = self.gate(x_src, gate_mlp, self.ffn(input_x_src))
++
++ return x, x_src
+
+
+ class MLP(torch.nn.Module):
+@@ -264,11 +380,57 @@ class Head(nn.Module):
+ shift, scale = (self.modulation.unsqueeze(0).to(dtype=t_mod.dtype, device=t_mod.device) + t_mod.unsqueeze(2)).chunk(2, dim=2)
+ x = (self.head(self.norm(x) * (1 + scale.squeeze(2)) + shift.squeeze(2)))
+ else:
++ if t_mod.shape[0] != 1:
++ t_mod = t_mod[:, None, :] # [b, d] -> [b, 1, d]
+ shift, scale = (self.modulation.to(dtype=t_mod.dtype, device=t_mod.device) + t_mod).chunk(2, dim=1)
+ x = (self.head(self.norm(x) * (1 + scale) + shift))
+ return x
+
+
++def pad_for_3d_conv(x, kernel_size):
++ """Pad to be divisible by kernel_size. From FramePack."""
++ _, _, t, h, w = x.shape
++ pt, ph, pw = kernel_size
++ pad_t = (pt - (t % pt)) % pt
++ pad_h = (ph - (h % ph)) % ph
++ pad_w = (pw - (w % pw)) % pw
++ if pad_t == 0 and pad_h == 0 and pad_w == 0:
++ return x
++ return F.pad(x, (0, pad_w, 0, pad_h, 0, pad_t), mode='replicate')
++
++
++class ViewPackEmbedding(nn.Module):
++ """Multi-resolution spatial patch embedding for clean source views.
++ Spatial-only downsampling: temporal dim preserved.
++ Ref: HunyuanVideoPatchEmbedForCleanLatents in FramePack.
++
++ Supported configurations (all exactly fill 4 quadrants):
++ - 4×2x views (each 2x view → 1 quadrant)
++ - 3×2x + 4×4x (3 quadrants from 2x + 1 quadrant from 4×4x tile)
++ """
++
++ def __init__(self, in_dim, dim, patch_size):
++ super().__init__()
++ pt, ph, pw = patch_size # (1, 2, 2) for Wan
++ # 2x: spatial 2x downsample relative to 1x -> kernel (1, 4, 4)
++ self.proj_2x = nn.Conv3d(in_dim, dim, kernel_size=(pt, ph*2, pw*2), stride=(pt, ph*2, pw*2))
++ # 4x: spatial 4x downsample relative to 1x -> kernel (1, 8, 8)
++ self.proj_4x = nn.Conv3d(in_dim, dim, kernel_size=(pt, ph*4, pw*4), stride=(pt, ph*4, pw*4))
++
++ @torch.no_grad()
++ def initialize_from_patch_embedding(self, patch_embedding: nn.Conv3d):
++ """FramePack-style init: tile spatial dims and scale by 1/area_ratio."""
++ weight = patch_embedding.weight.detach().clone() # (dim, in_dim, 1, 2, 2)
++ bias = patch_embedding.bias.detach().clone()
++ sd = {
++ 'proj_2x.weight': repeat(weight, 'b c t h w -> b c t (h 2) (w 2)') / 4.0,
++ 'proj_2x.bias': bias.clone(),
++ 'proj_4x.weight': repeat(weight, 'b c t h w -> b c t (h 4) (w 4)') / 16.0,
++ 'proj_4x.bias': bias.clone(),
++ }
++ self.load_state_dict(sd)
++
++
+ class WanModel(torch.nn.Module):
+ def __init__(
+ self,
+@@ -355,32 +517,121 @@ class WanModel(torch.nn.Module):
+
+ def forward(self,
+ x: torch.Tensor,
++ x_src: torch.Tensor,
+ timestep: torch.Tensor,
+ context: torch.Tensor,
++ skeletons: Optional[torch.Tensor] = None,
++ cam_emb: Optional[torch.Tensor] = None,
++ drop_viewpack_tokens: bool = False,
+ clip_feature: Optional[torch.Tensor] = None,
+ y: Optional[torch.Tensor] = None,
+ use_gradient_checkpointing: bool = False,
+ use_gradient_checkpointing_offload: bool = False,
+ **kwargs,
+ ):
+- t = self.time_embedding(
+- sinusoidal_embedding_1d(self.freq_dim, timestep))
+- t_mod = self.time_projection(t).unflatten(1, (6, self.dim))
++ # breakpoint()
+ context = self.text_embedding(context)
+-
++
+ if self.has_image_input:
+ x = torch.cat([x, y], dim=1) # (b, c_x + c_y, f, h, w)
+ clip_embdding = self.img_emb(clip_feature)
+ context = torch.cat([clip_embdding, context], dim=1)
+-
++
+ x, (f, h, w) = self.patchify(x)
+-
++
+ freqs = torch.cat([
+ self.freqs[0][:f].view(f, 1, 1, -1).expand(f, h, w, -1),
+ self.freqs[1][:h].view(1, h, 1, -1).expand(f, h, w, -1),
+ self.freqs[2][:w].view(1, 1, w, -1).expand(f, h, w, -1)
+ ], dim=-1).reshape(f * h * w, 1, -1).to(x.device)
+-
++
++ # build packed views from 1x/2x/4x source views
++ v_src = x_src.shape[0]
++ if v_src == 1:
++ # 1×1x: no 2x/4x sources
++ x_src_2x, x_src_4x = None, None
++ elif v_src == 5:
++ # 1×1x + 4×2x
++ x_src, x_src_2x, x_src_4x = x_src[:1], x_src[1:], None
++ elif v_src == 8:
++ # 1×1x + 3×2x + 4×4x
++ x_src, x_src_2x, x_src_4x = x_src[:1], x_src[1:4], x_src[4:]
++ else:
++ raise ValueError(f"Unsupported number of source views: {v_src}")
++
++ # 1x source: always packed as one extra view
++ x_src, (f_src, h_src, w_src) = self.patchify(x_src)
++ if not self.use_src_attn:
++ # use viewpack for 1x source views
++ x = torch.cat([x, x_src], dim=0)
++ x_src = None
++ freqs_src = None
++ v_pack = 1
++ else:
++ # use src self/cross-attention for 1x source views
++ x_src = repeat(x_src, "1 fhw_src c -> v fhw_src c", v=x.shape[0])
++ freqs_src = torch.cat([
++ self.freqs[0][self.freqs_src_shift:f_src + self.freqs_src_shift].view(f_src, 1, 1, -1).expand(f_src, h_src, w_src, -1),
++ self.freqs[1][:h_src].view(1, h_src, 1, -1).expand(f_src, h_src, w_src, -1),
++ self.freqs[2][:w_src].view(1, 1, w_src, -1).expand(f_src, h_src, w_src, -1)
++ ], dim=-1).reshape(f_src * h_src * w_src, 1, -1).to(x.device)
++ v_pack = 0
++
++ # 2x/4x source views: packed after 1x source views
++ if self.use_viewpack:
++ if x_src_2x is not None:
++ x_src_2x = pad_for_3d_conv(x_src_2x, self.viewpack_embedding.proj_2x.kernel_size)
++ x_src_2x = self.viewpack_embedding.proj_2x(x_src_2x) # (v_2x, dim, f, h//2, w//2)
++
++ if x_src_4x is not None:
++ # tile 4x source views into 2x source views
++ x_src_4x = pad_for_3d_conv(x_src_4x, self.viewpack_embedding.proj_4x.kernel_size)
++ x_src_4x = self.viewpack_embedding.proj_4x(x_src_4x) # (4, dim, f, h//4, w//4)
++ x_src_4x = rearrange(x_src_4x, '(g1 g2) c f h w -> 1 c f (g1 h) (g2 w)', g1=2, g2=2) # (1, dim, f, h//2, w//2)
++ f_2x, h_2x, w_2x = x_src_2x.shape[2:] # crop padding surplus to match 2x source views (f, h, w)
++ x_src_4x = x_src_4x[:, :, :f_2x, :h_2x, :w_2x]
++ x_src_2x = torch.cat([x_src_2x, x_src_4x], dim=0) # (4, dim, f, h//2, w//2)
++
++ # tile 2x source views into 1x source views
++ x_pack = rearrange(x_src_2x, '(g1 g2) c f h w -> 1 c f (g1 h) (g2 w)', g1=2, g2=2)
++ x_pack = x_pack[:, :, :f, :h, :w] # crop padding surplus to match 1x source views (f, h, w)
++ x_pack = rearrange(x_pack, '1 c f h w -> 1 (f h w) c')
++ x_pack = x_pack.to(dtype=x.dtype)
++ if drop_viewpack_tokens:
++ # Keep viewpack parameters in the autograd graph across distributed ranks.
++ zero_dependency = x_pack.float().mean().to(dtype=x.dtype) * 0.0
++ x = x + zero_dependency
++ else:
++ x = torch.cat([x, x_pack], dim=0)
++ v_pack += 1
++
++ timestep = torch.cat([timestep, torch.zeros(v_pack, device=timestep.device, dtype=timestep.dtype)])
++ if skeletons is not None:
++ skeletons = torch.cat([skeletons, -torch.ones_like(skeletons[:1]).expand(v_pack, -1, -1, -1, -1)], dim=0)
++
++ # Expand cam_emb for viewpack views (zero vectors for packed source views)
++ if cam_emb is not None:
++ cam_emb = torch.cat([cam_emb, torch.zeros(v_pack, cam_emb.shape[-1],
++ device=cam_emb.device, dtype=cam_emb.dtype)], dim=0)
++ cam_emb = cam_emb.unsqueeze(1) # (v, 1, 12)
++
++ # Compute time embeddings (after concat since timestep may have been extended)
++ t = self.time_embedding(
++ sinusoidal_embedding_1d(self.freq_dim, timestep).to(x.dtype))
++ t_mod = self.time_projection(t).unflatten(1, (6, self.dim))
++
++ if self.use_pose_encoder:
++ skeleton_latents = self.pose_encoder(skeletons)
++ skeleton_tokens = rearrange(skeleton_latents, "v c f h w -> v (f h w) c")
++ x = x + skeleton_tokens
++
++ v = x.shape[0]
++ freqs_mvs = torch.cat([
++ self.freqs[0][:v].view(v, 1, 1, -1).expand(v, h, w, -1),
++ self.freqs[1][:h].view(1, h, 1, -1).expand(v, h, w, -1),
++ self.freqs[2][:w].view(1, 1, w, -1).expand(v, h, w, -1)
++ ], dim=-1).reshape(v * h * w, 1, -1).to(x.device)
++
+ def create_custom_forward(module):
+ def custom_forward(*inputs):
+ return module(*inputs)
+@@ -390,19 +641,24 @@ class WanModel(torch.nn.Module):
+ if self.training and use_gradient_checkpointing:
+ if use_gradient_checkpointing_offload:
+ with torch.autograd.graph.save_on_cpu():
+- x = torch.utils.checkpoint.checkpoint(
++ x, x_src = torch.utils.checkpoint.checkpoint(
+ create_custom_forward(block),
+- x, context, t_mod, freqs,
++ x, x_src, context, t_mod, freqs, freqs_mvs, freqs_src, cam_emb, (v, f, h, w),
+ use_reentrant=False,
+ )
+ else:
+- x = torch.utils.checkpoint.checkpoint(
++ x, x_src = torch.utils.checkpoint.checkpoint(
+ create_custom_forward(block),
+- x, context, t_mod, freqs,
++ x, x_src, context, t_mod, freqs, freqs_mvs, freqs_src, cam_emb, (v, f, h, w),
+ use_reentrant=False,
+ )
+ else:
+- x = block(x, context, t_mod, freqs)
++ x, x_src = block(x, x_src, context, t_mod, freqs, freqs_mvs, freqs_src, cam_emb, (v, f, h, w))
++
++ # Strip added views (viewpack)
++ if v_pack > 0:
++ x = x[:-v_pack, ...]
++ t = t[:-v_pack, ...]
+
+ x = self.head(x, t)
+ x = self.unpatchify(x, (f, h, w))
+@@ -411,7 +667,7 @@ class WanModel(torch.nn.Module):
+ @staticmethod
+ def state_dict_converter():
+ return WanModelStateDictConverter()
+-
++
+
+ class WanModelStateDictConverter:
+ def __init__(self):
+diff --git a/diffsynth/models/wan_video_pose_encoder.py b/diffsynth/models/wan_video_pose_encoder.py
+new file mode 100644
+index 0000000..5180625
+--- /dev/null
++++ b/diffsynth/models/wan_video_pose_encoder.py
+@@ -0,0 +1,81 @@
++import torch
++import torch.nn as nn
++import numpy as np
++from torch.nn import init
++
++# PoseEncoder is 3D version of PoseNet in MimicMotion:
++# https://github.com/Tencent/MimicMotion/blob/c053153a1d124abae8c08568925ae88debc63001/mimicmotion/modules/pose_net.py
++
++
++class PoseEncoder(nn.Module):
++ def __init__(self, out_dim=5120, in_channels=3):
++ super().__init__()
++
++ if out_dim in (5120, 1536):
++ # Wan2.1-T2V-14B / 1.3B
++ t_strides = (1, 1, 1, 2, 2) # downsampled by 4
++ s_strides = (2, 2, 1, 2, 2) # downsampled by 16
++ kernel_size = (3, 3, 3)
++ elif out_dim == 3072:
++ # Wan2.2-TI2V-5B
++ t_strides = (1, 1, 1, 2, 2) # downsampled by 4
++ s_strides = (2, 2, 2, 2, 2) # downsampled by 32
++ kernel_size = (3, 4, 4)
++ else:
++ raise ValueError(f"Invalid out_dim: {out_dim}")
++
++ strides = [(t, s, s) for t, s in zip(t_strides, s_strides)]
++
++ self.conv_layers = nn.Sequential(
++ nn.Conv3d(in_channels, in_channels, kernel_size=3, stride=1, padding=1),
++ nn.SiLU(),
++ nn.Conv3d(in_channels, 16, kernel_size=kernel_size, stride=strides[0], padding=(1, 1, 1)),
++ nn.SiLU(),
++ nn.Conv3d(16, 16, kernel_size=3, stride=1, padding=1),
++ nn.SiLU(),
++ nn.Conv3d(16, 32, kernel_size=kernel_size, stride=strides[1], padding=(1, 1, 1)),
++ nn.SiLU(),
++ nn.Conv3d(32, 32, kernel_size=3, stride=1, padding=1),
++ nn.SiLU(),
++ nn.Conv3d(32, 64, kernel_size=kernel_size, stride=strides[2], padding=(1, 1, 1)),
++ nn.SiLU(),
++ nn.Conv3d(64, 64, kernel_size=3, stride=1, padding=1),
++ nn.SiLU(),
++ nn.Conv3d(64, 128, kernel_size=kernel_size, stride=strides[3], padding=(1, 1, 1)),
++ nn.SiLU(),
++ nn.Conv3d(128, 128, kernel_size=3, stride=1, padding=1),
++ nn.SiLU(),
++ nn.Conv3d(128, 256, kernel_size=kernel_size, stride=strides[4], padding=(1, 1, 1)),
++ nn.SiLU(),
++ )
++
++ self.final_proj = nn.Conv3d(256, out_dim, kernel_size=1)
++
++ self.scale = nn.Parameter(torch.ones(1) * 2.0)
++
++ self._initialize_weights()
++
++ def _initialize_weights(self):
++ for m in self.modules():
++ if isinstance(m, nn.Conv3d):
++ # He (Kaiming) initialization in fan‑in mode
++ receptive = np.prod(m.kernel_size) * m.in_channels
++ init.normal_(m.weight, mean=0.0, std=np.sqrt(2.0 / receptive))
++ if m.bias is not None:
++ init.zeros_(m.bias)
++ # start with zero output so model behaves like unconditional
++ init.zeros_(self.final_proj.weight)
++ if self.final_proj.bias is not None:
++ init.zeros_(self.final_proj.bias)
++
++ def forward(self, x: torch.Tensor) -> torch.Tensor:
++ """
++ x: (B, C, F, H, W) -> latent grid matching the DiT patch tokens.
++ Wan2.1 uses F/4, H/16, W/16; Wan2.2-TI2V-5B uses F/4, H/32, W/32.
++ """
++ # Wan pattern: 1 -> 4 -> 4 -> ...
++ x = torch.cat([x[:, :, :1].repeat(1, 1, 3, 1, 1), x], dim=2)
++
++ x = self.conv_layers(x)
++ x = self.final_proj(x)
++ return x * self.scale
+diff --git a/diffsynth/models/wan_video_vae.py b/diffsynth/models/wan_video_vae.py
+index 397a2e7..43057ff 100644
+--- a/diffsynth/models/wan_video_vae.py
++++ b/diffsynth/models/wan_video_vae.py
+@@ -1121,7 +1121,7 @@ class WanVideoVAE(nn.Module):
+ weight = torch.zeros((1, 1, out_T, H * self.upsampling_factor, W * self.upsampling_factor), dtype=hidden_states.dtype, device=data_device)
+ values = torch.zeros((1, 3, out_T, H * self.upsampling_factor, W * self.upsampling_factor), dtype=hidden_states.dtype, device=data_device)
+
+- for h, h_, w, w_ in tqdm(tasks, desc="VAE decoding"):
++ for h, h_, w, w_ in tasks:
+ hidden_states_batch = hidden_states[:, :, :, h:h_, w:w_].to(computation_device)
+ hidden_states_batch = self.model.decode(hidden_states_batch, self.scale).to(data_device)
+
+@@ -1173,7 +1173,7 @@ class WanVideoVAE(nn.Module):
+ weight = torch.zeros((1, 1, out_T, H // self.upsampling_factor, W // self.upsampling_factor), dtype=video.dtype, device=data_device)
+ values = torch.zeros((1, self.z_dim, out_T, H // self.upsampling_factor, W // self.upsampling_factor), dtype=video.dtype, device=data_device)
+
+- for h, h_, w, w_ in tqdm(tasks, desc="VAE encoding"):
++ for h, h_, w, w_ in tasks:
+ hidden_states_batch = video[:, :, :, h:h_, w:w_].to(computation_device)
+ hidden_states_batch = self.model.encode(hidden_states_batch, self.scale).to(data_device)
+
+@@ -1216,10 +1216,9 @@ class WanVideoVAE(nn.Module):
+
+
+ def encode(self, videos, device, tiled=False, tile_size=(34, 34), tile_stride=(18, 16)):
+-
+ videos = [video.to("cpu") for video in videos]
+ hidden_states = []
+- for video in videos:
++ for video in tqdm(videos, desc="VAE encoding", disable=not tiled):
+ video = video.unsqueeze(0)
+ if tiled:
+ tile_size = (tile_size[0] * self.upsampling_factor, tile_size[1] * self.upsampling_factor)
+@@ -1234,11 +1233,18 @@ class WanVideoVAE(nn.Module):
+
+
+ def decode(self, hidden_states, device, tiled=False, tile_size=(34, 34), tile_stride=(18, 16)):
+- if tiled:
+- video = self.tiled_decode(hidden_states, device, tile_size, tile_stride)
+- else:
+- video = self.single_decode(hidden_states, device)
+- return video
++ hidden_states = [hidden_state.to("cpu") for hidden_state in hidden_states]
++ videos = []
++ for hidden_state in tqdm(hidden_states, desc="VAE decoding", disable=not tiled):
++ hidden_state = hidden_state.unsqueeze(0)
++ if tiled:
++ video = self.tiled_decode(hidden_state, device, tile_size, tile_stride)
++ else:
++ video = self.single_decode(hidden_state, device)
++ video = video.squeeze(0)
++ videos.append(video)
++ videos = torch.stack(videos)
++ return videos
+
+
+ @staticmethod
+diff --git a/diffsynth/pipelines/__init__.py b/diffsynth/pipelines/__init__.py
+index e2ad551..f878ad8 100644
+--- a/diffsynth/pipelines/__init__.py
++++ b/diffsynth/pipelines/__init__.py
+@@ -12,4 +12,5 @@ from .pipeline_runner import SDVideoPipelineRunner
+ from .hunyuan_video import HunyuanVideoPipeline
+ from .step_video import StepVideoPipeline
+ from .wan_video import WanVideoPipeline
++from .wan_video_spatem import WanVideoSpaTemPipeline
+ KolorsImagePipeline = SDXLImagePipeline
+diff --git a/diffsynth/pipelines/wan_video_spatem.py b/diffsynth/pipelines/wan_video_spatem.py
+new file mode 100644
+index 0000000..85b4f65
+--- /dev/null
++++ b/diffsynth/pipelines/wan_video_spatem.py
+@@ -0,0 +1,659 @@
++from ..models import ModelManager
++from ..models.wan_video_dit import WanModel
++from ..models.wan_video_pose_encoder import PoseEncoder
++from ..models.wan_video_text_encoder import WanTextEncoder
++from ..models.wan_video_vae import WanVideoVAE
++from ..models.wan_video_image_encoder import WanImageEncoder
++from ..schedulers.flow_match import FlowMatchScheduler
++from ..schedulers.bride_match import BridgeMatchScheduler
++from ..pipelines.base import BasePipeline
++from ..prompters import WanPrompter
++import torch, os
++import torch.nn as nn
++import numpy as np
++import torch.nn.functional as F
++from PIL import Image
++from tqdm import tqdm
++from typing import Optional, Union
++from functools import partial
++from einops import rearrange
++
++from ..vram_management import enable_vram_management, AutoWrappedModule, AutoWrappedLinear
++from ..models.wan_video_text_encoder import T5RelativeEmbedding, T5LayerNorm
++from ..models.wan_video_dit import RMSNorm, SelfAttention, CrossAttentionSrcCam, ViewPackEmbedding
++from ..models.wan_video_vae import RMS_norm, CausalConv3d, Upsample
++from ..utils import ModelConfig
++
++
++class WanVideoSpaTemPipeline(BasePipeline):
++
++ def __init__(self, device="cuda", torch_dtype=torch.float16, tokenizer_path=None):
++ super().__init__(device=device, torch_dtype=torch_dtype)
++ self.scheduler = FlowMatchScheduler(shift=5, sigma_min=0.0, extra_one_step=True)
++ self.prompter = WanPrompter(tokenizer_path=tokenizer_path)
++ self.text_encoder: WanTextEncoder = None
++ self.image_encoder: WanImageEncoder = None
++ self.dit: WanModel = None
++ self.vae: WanVideoVAE = None
++ self.model_names = ["text_encoder", "dit", "vae"]
++ self.height_division_factor = 16
++ self.width_division_factor = 16
++
++ def enable_vram_management(self, num_persistent_param_in_dit=None):
++ dtype = next(iter(self.text_encoder.parameters())).dtype
++ enable_vram_management(
++ self.text_encoder,
++ module_map={
++ torch.nn.Linear: AutoWrappedLinear,
++ torch.nn.Embedding: AutoWrappedModule,
++ T5RelativeEmbedding: AutoWrappedModule,
++ T5LayerNorm: AutoWrappedModule,
++ },
++ module_config=dict(
++ offload_dtype=dtype,
++ offload_device="cpu",
++ onload_dtype=dtype,
++ onload_device="cpu",
++ computation_dtype=self.torch_dtype,
++ computation_device=self.device,
++ ),
++ )
++ dtype = next(iter(self.dit.parameters())).dtype
++ enable_vram_management(
++ self.dit,
++ module_map={
++ torch.nn.Linear: AutoWrappedLinear,
++ torch.nn.Conv3d: AutoWrappedModule,
++ torch.nn.LayerNorm: AutoWrappedModule,
++ RMSNorm: AutoWrappedModule,
++ },
++ module_config=dict(
++ offload_dtype=dtype,
++ offload_device="cpu",
++ onload_dtype=dtype,
++ onload_device=self.device,
++ computation_dtype=self.torch_dtype,
++ computation_device=self.device,
++ ),
++ max_num_param=num_persistent_param_in_dit,
++ overflow_module_config=dict(
++ offload_dtype=dtype,
++ offload_device="cpu",
++ onload_dtype=dtype,
++ onload_device="cpu",
++ computation_dtype=self.torch_dtype,
++ computation_device=self.device,
++ ),
++ )
++ dtype = next(iter(self.vae.parameters())).dtype
++ enable_vram_management(
++ self.vae,
++ module_map={
++ torch.nn.Linear: AutoWrappedLinear,
++ torch.nn.Conv2d: AutoWrappedModule,
++ RMS_norm: AutoWrappedModule,
++ CausalConv3d: AutoWrappedModule,
++ Upsample: AutoWrappedModule,
++ torch.nn.SiLU: AutoWrappedModule,
++ torch.nn.Dropout: AutoWrappedModule,
++ },
++ module_config=dict(
++ offload_dtype=dtype,
++ offload_device="cpu",
++ onload_dtype=dtype,
++ onload_device=self.device,
++ computation_dtype=self.torch_dtype,
++ computation_device=self.device,
++ ),
++ )
++ if self.image_encoder is not None:
++ dtype = next(iter(self.image_encoder.parameters())).dtype
++ enable_vram_management(
++ self.image_encoder,
++ module_map={
++ torch.nn.Linear: AutoWrappedLinear,
++ torch.nn.Conv2d: AutoWrappedModule,
++ torch.nn.LayerNorm: AutoWrappedModule,
++ },
++ module_config=dict(
++ offload_dtype=dtype,
++ offload_device="cpu",
++ onload_dtype=dtype,
++ onload_device="cpu",
++ computation_dtype=dtype,
++ computation_device=self.device,
++ ),
++ )
++ self.enable_cpu_offload()
++
++ def fetch_models(self, model_manager: ModelManager):
++ text_encoder_model_and_path = model_manager.fetch_model("wan_video_text_encoder", require_model_path=True)
++ if text_encoder_model_and_path is not None:
++ self.text_encoder, tokenizer_path = text_encoder_model_and_path
++ self.prompter.fetch_models(self.text_encoder)
++ self.prompter.fetch_tokenizer(os.path.join(os.path.dirname(tokenizer_path), "google/umt5-xxl"))
++ self.dit = model_manager.fetch_model("wan_video_dit")
++ self.vae = model_manager.fetch_model("wan_video_vae")
++ self.image_encoder = model_manager.fetch_model("wan_video_image_encoder")
++
++ @staticmethod
++ def from_model_manager(model_manager: ModelManager, torch_dtype=None, device=None):
++ if device is None:
++ device = model_manager.device
++ if torch_dtype is None:
++ torch_dtype = model_manager.torch_dtype
++ pipe = WanVideoSpaTemPipeline(device=device, torch_dtype=torch_dtype)
++ pipe.fetch_models(model_manager)
++ return pipe
++
++ @staticmethod
++ def from_pretrained(
++ torch_dtype: torch.dtype = torch.bfloat16,
++ device: Union[str, torch.device] = "cuda",
++ model_configs: list[ModelConfig] = [],
++ tokenizer_config: ModelConfig = ModelConfig(model_id="Wan-AI/Wan2.1-T2V-1.3B", origin_file_pattern="google/*"),
++ redirect_common_files: bool = True,
++ use_usp=False,
++ ):
++ # Redirect model path
++ if redirect_common_files:
++ redirect_dict = {
++ "models_t5_umt5-xxl-enc-bf16.pth": "Wan-AI/Wan2.1-T2V-1.3B",
++ "Wan2.1_VAE.pth": "Wan-AI/Wan2.1-T2V-1.3B",
++ "models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth": "Wan-AI/Wan2.1-I2V-14B-480P",
++ }
++ for model_config in model_configs:
++ if model_config.origin_file_pattern is None or model_config.model_id is None:
++ continue
++ if (
++ model_config.origin_file_pattern in redirect_dict
++ and model_config.model_id != redirect_dict[model_config.origin_file_pattern]
++ ):
++ print(
++ f"To avoid repeatedly downloading model files, ({model_config.model_id}, {model_config.origin_file_pattern}) is redirected to ({redirect_dict[model_config.origin_file_pattern]}, {model_config.origin_file_pattern}). You can use `redirect_common_files=False` to disable file redirection."
++ )
++ model_config.model_id = redirect_dict[model_config.origin_file_pattern]
++
++ # Initialize pipeline
++ pipe = WanVideoSpaTemPipeline(device=device, torch_dtype=torch_dtype)
++ if use_usp:
++ pipe.initialize_usp()
++
++ # Download and load models
++ model_manager = ModelManager()
++ for model_config in model_configs:
++ model_config.download_if_necessary(use_usp=use_usp)
++ model_manager.load_model(
++ model_config.path,
++ device=model_config.offload_device or device,
++ torch_dtype=model_config.offload_dtype or torch_dtype,
++ )
++
++ # Load models
++ pipe.text_encoder = model_manager.fetch_model("wan_video_text_encoder")
++ dit = model_manager.fetch_model("wan_video_dit", index=2)
++ if isinstance(dit, list):
++ pipe.dit, pipe.dit2 = dit
++ else:
++ pipe.dit = dit
++ pipe.vae = model_manager.fetch_model("wan_video_vae")
++ pipe.image_encoder = model_manager.fetch_model("wan_video_image_encoder")
++ pipe.motion_controller = model_manager.fetch_model("wan_video_motion_controller")
++ pipe.vace = model_manager.fetch_model("wan_video_vace")
++
++ # Size division factor
++ if pipe.vae is not None:
++ pipe.height_division_factor = pipe.vae.upsampling_factor * 2
++ pipe.width_division_factor = pipe.vae.upsampling_factor * 2
++
++ # Initialize tokenizer
++ tokenizer_config.local_model_path = model_configs[0].local_model_path
++ tokenizer_config.skip_download = model_configs[0].skip_download
++ tokenizer_config.download_if_necessary(use_usp=use_usp)
++ pipe.prompter.fetch_models(pipe.text_encoder)
++ pipe.prompter.fetch_tokenizer(tokenizer_config.path)
++
++ # Unified Sequence Parallel
++ if use_usp:
++ pipe.enable_usp()
++
++ return pipe
++
++ def init_spatem_modules(
++ self,
++ disable_video_attn: bool = False,
++ use_4d_attn: bool = False,
++ use_mvs_attn: bool = False,
++ use_src_self_attn: bool = False,
++ use_src_cross_attn: bool = False,
++ freqs_src_shift: int = 121,
++ use_viewpack: bool = True,
++ viewpack_dropout_prob: float = 0.0,
++ use_pose_encoder: bool = True,
++ pose_encoder_type: str = "rgb",
++ use_cam_encoder: bool = False,
++ range_4d_attn: tuple[int, int, int] = (0, None, 2),
++ range_mvs_attn: tuple[int, int, int] = (1, None, 2),
++ range_src_self_attn: tuple[int, int, int] = (0, None, 2),
++ range_src_cross_attn: tuple[int, int, int] = (0, None, 2),
++ use_lbm: bool = False,
++ fill_wpmask_with_noise: bool = False,
++ ):
++ # breakpoint()
++ device, dtype = self.dit.patch_embedding.weight.device, self.dit.patch_embedding.weight.dtype
++
++ if disable_video_attn:
++ # todo: delete self_attn layers from the model
++ if use_4d_attn:
++ raise ValueError("Cannot use 4D attention when video attention is disabled")
++ for block in self.dit.blocks:
++ block.disable_video_attn = True
++
++ if use_4d_attn:
++ b, e, s = range_4d_attn
++ for block in self.dit.blocks[b:e:s]:
++ block.use_4d_attn = True
++
++ if use_mvs_attn:
++ b, e, s = range_mvs_attn
++ for block in self.dit.blocks[b:e:s]:
++ block.use_mvs_attn = True
++
++ dim = block.self_attn.q.weight.shape[0]
++ block.modulation_mvs = nn.Parameter(block.modulation[:, :3, :].detach().clone())
++ block.norm1_mvs = nn.LayerNorm(dim, eps=block.norm1.eps, elementwise_affine=False).to(
++ device=device, dtype=dtype
++ )
++ block.self_attn_mvs = SelfAttention(dim, block.self_attn.num_heads, block.self_attn.norm_q.eps).to(
++ device=device, dtype=dtype
++ )
++ block.self_attn_mvs.load_state_dict(block.self_attn.state_dict(), strict=True)
++
++ if not 0.0 <= viewpack_dropout_prob <= 1.0:
++ raise ValueError("viewpack_dropout_prob should be between 0 and 1")
++ if viewpack_dropout_prob > 0.0 and not use_viewpack:
++ raise ValueError("viewpack_dropout_prob requires use_viewpack=True")
++
++ if use_viewpack:
++ viewpack_emb = ViewPackEmbedding(
++ in_dim=self.dit.patch_embedding.weight.shape[1],
++ dim=self.dit.patch_embedding.weight.shape[0],
++ patch_size=list(self.dit.patch_embedding.kernel_size),
++ )
++ viewpack_emb.initialize_from_patch_embedding(self.dit.patch_embedding)
++ self.dit.viewpack_embedding = viewpack_emb.to(device=device, dtype=dtype)
++ elif use_src_self_attn:
++ if use_src_cross_attn:
++ raise ValueError("Cannot use both src self-attention and src cross-attention")
++
++ b, e, s = range_src_self_attn
++ for block in self.dit.blocks[b:e:s]:
++ block.use_src_self_attn = True
++
++ dim = block.self_attn.q.weight.shape[0]
++ block.modulation_src = nn.Parameter(block.modulation[:, :3, :].detach().clone())
++ block.norm1_src = nn.LayerNorm(dim, eps=block.norm1.eps, elementwise_affine=False).to(
++ device=device, dtype=dtype
++ )
++ block.self_attn_src = SelfAttention(dim, block.self_attn.num_heads, block.self_attn.norm_q.eps).to(
++ device=device, dtype=dtype
++ )
++ block.self_attn_src.load_state_dict(block.self_attn.state_dict(), strict=True)
++ elif use_src_cross_attn:
++ b, e, s = range_src_cross_attn
++ for block in self.dit.blocks[b:e:s]:
++ block.use_src_cross_attn = True
++
++ dim = block.self_attn.q.weight.shape[0]
++ block.modulation_src = nn.Parameter(block.modulation[:, :3, :].detach().clone())
++ block.norm1_src = nn.LayerNorm(dim, eps=block.norm1.eps, elementwise_affine=False).to(
++ device=device, dtype=dtype
++ )
++ block.cross_attn_src = CrossAttentionSrcCam(
++ dim, block.self_attn.num_heads, block.self_attn.norm_q.eps
++ ).to(device=device, dtype=dtype)
++ block.cross_attn_src.load_state_dict(block.self_attn.state_dict(), strict=True)
++
++ if use_pose_encoder:
++ if pose_encoder_type == "rgb":
++ in_channels = 3
++ elif pose_encoder_type == "rgbd":
++ in_channels = 4
++ else:
++ raise ValueError(f"Invalid pose_encoder_type: {pose_encoder_type}")
++ pose_encoder = PoseEncoder(out_dim=self.dit.patch_embedding.out_channels, in_channels=in_channels)
++ self.dit.pose_encoder = pose_encoder.to(device=device, dtype=dtype)
++
++ if use_cam_encoder:
++ dim = self.dit.blocks[0].self_attn.q.weight.shape[0]
++ for block in self.dit.blocks:
++ block.use_cam_encoder = True
++ block.cam_encoder = nn.Linear(12, dim).to(device=device, dtype=dtype)
++ block.projector = nn.Linear(dim, dim).to(device=device, dtype=dtype)
++ block.cam_encoder.weight.data.zero_()
++ block.cam_encoder.bias.data.zero_()
++ block.projector.weight = nn.Parameter(torch.eye(dim, device=device, dtype=dtype))
++ block.projector.bias = nn.Parameter(torch.zeros(dim, device=device, dtype=dtype))
++
++ if use_lbm:
++ # TODO: hard-code for now
++ self.scheduler = BridgeMatchScheduler()
++ self.dit.fill_wpmask_with_noise = fill_wpmask_with_noise
++
++ self.dit.use_pose_encoder = use_pose_encoder
++ self.dit.use_cam_encoder = use_cam_encoder
++ self.dit.use_viewpack = use_viewpack
++ self.dit.viewpack_dropout_prob = viewpack_dropout_prob
++ self.dit.use_src_attn = use_src_self_attn or use_src_cross_attn
++ self.dit.use_lbm = use_lbm
++ self.dit.freqs_src_shift = freqs_src_shift
++
++ def denoising_model(self):
++ return self.dit
++
++ def encode_prompt(self, prompt, positive=True):
++ prompt_emb = self.prompter.encode_prompt(prompt, positive=positive)
++ return {"context": prompt_emb}
++
++ def encode_image(self, image, num_frames, height, width):
++ image = self.preprocess_image(image.resize((width, height))).to(self.device)
++ clip_context = self.image_encoder.encode_image([image])
++ msk = torch.ones(1, num_frames, height // 8, width // 8, device=self.device)
++ msk[:, 1:] = 0
++ msk = torch.concat([torch.repeat_interleave(msk[:, 0:1], repeats=4, dim=1), msk[:, 1:]], dim=1)
++ msk = msk.view(1, msk.shape[1] // 4, 4, height // 8, width // 8)
++ msk = msk.transpose(1, 2)[0]
++
++ vae_input = torch.concat(
++ [image.transpose(0, 1), torch.zeros(3, num_frames - 1, height, width).to(image.device)], dim=1
++ )
++ y = self.vae.encode([vae_input.to(dtype=self.torch_dtype, device=self.device)], device=self.device)[0]
++ y = torch.concat([msk, y])
++ y = y.unsqueeze(0)
++ clip_context = clip_context.to(dtype=self.torch_dtype, device=self.device)
++ y = y.to(dtype=self.torch_dtype, device=self.device)
++ return {"clip_feature": clip_context, "y": y}
++
++ def tensor2video(self, frames):
++ frames = rearrange(frames, "c f h w -> f h w c")
++ frames = ((frames.float() + 1) * 127.5).clip(0, 255).cpu().numpy().astype(np.uint8)
++ frames = [Image.fromarray(frame) for frame in frames]
++ return frames
++
++ def prepare_extra_input(self, latents=None):
++ return {}
++
++ def encode_video(self, input_video, tiled=True, tile_size=(34, 34), tile_stride=(18, 16)):
++ latents = self.vae.encode(
++ input_video, device=self.device, tiled=tiled, tile_size=tile_size, tile_stride=tile_stride
++ )
++ return latents
++
++ def decode_video(self, latents, tiled=True, tile_size=(34, 34), tile_stride=(18, 16)):
++ frames = self.vae.decode(latents, device=self.device, tiled=tiled, tile_size=tile_size, tile_stride=tile_stride)
++ return frames
++
++ def encode_fmask(self, mask, size):
++ f_, h_, w_ = size
++ g_ = (mask.shape[2] - 1) // (f_ - 1)
++
++ # union of the frames in each latent (the first frame is encoded independently)
++ mask = torch.cat([mask[:, :, :1].repeat(1, 1, g_ - 1, 1, 1), mask], dim=2)
++ mask = rearrange(mask, "v c (f g) h w -> v c f g h w", f=f_, g=g_)
++ mask = mask.max(dim=3).values
++
++ # interpolate along the spatial dimensions
++ mask = rearrange(mask, "v c f h w -> (v f) c h w")
++ mask = F.interpolate(mask, size=(h_, w_), mode="area")
++ mask = rearrange(mask, "(v f) c h w -> v c f h w", f=f_)
++ return mask
++
++ def encode_wpmask(self, mask, size):
++ f_, h_, w_ = size
++ g_ = (mask.shape[2] - 1) // (f_ - 1)
++
++ # intersection of the frames in each latent (the first frame is encoded independently)
++ mask = torch.cat([mask[:, :, :1].repeat(1, 1, g_ - 1, 1, 1), mask], dim=2)
++ mask = rearrange(mask, "v c (f g) h w -> v c f g h w", f=f_, g=g_)
++ mask = mask.min(dim=3).values
++
++ # interpolate along the spatial dimensions
++ mask = rearrange(mask, "v c f h w -> (v f) c h w")
++ mask = F.interpolate(mask, size=(h_, w_), mode="area")
++ mask = rearrange(mask, "(v f) c h w -> v c f h w", f=f_)
++ return mask
++
++ @torch.no_grad()
++ def __call__(
++ self,
++ prompt,
++ negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走",
++ src_videos: torch.Tensor = None,
++ skeletons: torch.Tensor = None,
++ wpvideos: torch.Tensor = None,
++ wpmasks: torch.Tensor = None,
++ cam_emb: torch.Tensor = None,
++ input_image: Image.Image = None,
++ input_video: torch.Tensor = None,
++ denoising_strength: float = 1.0,
++ seed: int = None,
++ rand_device: str = "cpu",
++ height: int = 832,
++ width: int = 480,
++ num_frames: int = None,
++ cfg_scale: float = 5.0,
++ num_inference_steps: int = 50,
++ sigma_shift: float = 5.0,
++ tiled: bool = True,
++ tile_size: tuple[int, int] = (52, 30),
++ tile_stride: tuple[int, int] = (26, 15),
++ tea_cache_l1_thresh: float = None,
++ tea_cache_model_id: str = "",
++ progress_bar_cmd=partial(tqdm, desc="Denoising"),
++ progress_bar_st=None,
++ return_tensor=False,
++ ):
++ # breakpoint()
++ assert num_frames is None, "num_frames is not supported for WanVideoSpaTemPipeline"
++ assert input_image is None, "input_image is not supported for WanVideoSpaTemPipeline"
++ assert input_video is None, "input_video is not supported for WanVideoSpaTemPipeline"
++ assert tea_cache_l1_thresh is None, "tea_cache_l1_thresh is not supported for WanVideoSpaTemPipeline"
++
++ # Parameter check
++ height, width = self.check_resize_height_width(height, width)
++
++ # Tiler parameters
++ tiler_kwargs = {"tiled": tiled, "tile_size": tile_size, "tile_stride": tile_stride}
++
++ # Scheduler
++ if self.dit.use_lbm:
++ # bridge matching scheduler
++ self.scheduler.set_timesteps(num_inference_steps)
++ else:
++ # flow matching scheduler
++ self.scheduler.set_timesteps(num_inference_steps, denoising_strength=denoising_strength, shift=sigma_shift)
++
++ src_videos = src_videos.to(dtype=self.torch_dtype, device=self.device)
++ if skeletons is not None:
++ skeletons = skeletons.to(dtype=self.torch_dtype, device=self.device)
++ if wpvideos is not None:
++ wpvideos = wpvideos.to(dtype=self.torch_dtype, device=self.device)
++ if wpmasks is not None:
++ wpmasks = wpmasks.to(dtype=self.torch_dtype, device=self.device)
++ if cam_emb is not None:
++ cam_emb = cam_emb.to(dtype=self.torch_dtype, device=self.device)
++
++ if skeletons is not None:
++ num_cameras = skeletons.shape[0]
++ num_frames = skeletons.shape[2]
++ elif wpvideos is not None:
++ num_cameras = wpvideos.shape[0]
++ num_frames = wpvideos.shape[2]
++ else:
++ raise ValueError("Either skeletons or wpvideos must be provided")
++
++ # Initialize noise
++ noise_shape = (
++ num_cameras,
++ self.vae.model.z_dim,
++ (num_frames - 1) // 4 + 1,
++ height // self.vae.upsampling_factor,
++ width // self.vae.upsampling_factor,
++ )
++ noise = self.generate_noise(noise_shape, seed=seed, device=rand_device, dtype=torch.float32)
++ noise = noise.to(dtype=self.torch_dtype, device=self.device)
++
++ if input_video is not None:
++ self.load_models_to_device(["vae"])
++ input_video = self.preprocess_images(input_video)
++ input_video = torch.stack(input_video, dim=2).to(dtype=self.torch_dtype, device=self.device)
++ latents = self.encode_video(input_video, **tiler_kwargs).to(dtype=self.torch_dtype, device=self.device)
++ latents = self.scheduler.add_noise(latents, noise, timestep=self.scheduler.timesteps[0])
++ else:
++ latents = noise
++
++ # Encode source video
++ self.load_models_to_device(["vae"])
++ src_latents = self.encode_video(src_videos, **tiler_kwargs)
++ src_latents = src_latents.to(dtype=self.torch_dtype, device=self.device)
++ src_latents_nega = torch.zeros_like(src_latents)
++
++ # Latent bridge matching
++ if self.dit.use_lbm:
++ if skeletons is not None:
++ # skeleton-based: use primary src_latents as bridge source
++ lbm_src_latents = src_latents[:1].expand_as(latents)
++ elif wpvideos is not None:
++ lbm_src_latents = self.encode_video(wpvideos, **tiler_kwargs).to(
++ dtype=self.torch_dtype, device=self.device
++ )
++ if self.dit.fill_wpmask_with_noise:
++ wpmask_latents = self.encode_wpmask(wpmasks, size=lbm_src_latents.shape[-3:])
++ lbm_src_latents = lbm_src_latents * wpmask_latents + noise * (1 - wpmask_latents)
++
++ latents = lbm_src_latents
++
++ # Encode prompts
++ self.load_models_to_device(["text_encoder"])
++ prompt_emb_posi = self.encode_prompt(prompt, positive=True)
++ if cfg_scale != 1.0:
++ prompt_emb_nega = self.encode_prompt(negative_prompt, positive=False)
++
++ # Encode image
++ if input_image is not None and self.image_encoder is not None:
++ self.load_models_to_device(["image_encoder", "vae"])
++ image_emb = self.encode_image(input_image, num_frames, height, width)
++ else:
++ image_emb = {}
++
++ # Extra input
++ extra_input = self.prepare_extra_input(latents)
++
++ # Denoise
++ self.load_models_to_device(["dit"])
++ for progress_id, timestep in enumerate(progress_bar_cmd(self.scheduler.timesteps)):
++ timestep = timestep.unsqueeze(0).to(dtype=self.torch_dtype, device=self.device)
++ timestep = torch.cat([timestep] * num_cameras, dim=0)
++
++ # Inference
++ noise_pred_posi = self.denoising_model()(
++ x=latents,
++ x_src=src_latents,
++ timestep=timestep,
++ skeletons=skeletons,
++ cam_emb=cam_emb,
++ **prompt_emb_posi,
++ **image_emb,
++ **extra_input,
++ )
++ if cfg_scale != 1.0:
++ noise_pred_nega = self.denoising_model()(
++ x=latents,
++ x_src=src_latents_nega,
++ timestep=timestep,
++ skeletons=skeletons,
++ cam_emb=cam_emb,
++ **prompt_emb_nega,
++ **image_emb,
++ **extra_input,
++ )
++ noise_pred = noise_pred_nega + cfg_scale * (noise_pred_posi - noise_pred_nega)
++ else:
++ noise_pred = noise_pred_posi
++
++ # Scheduler
++ latents = self.scheduler.step(noise_pred, self.scheduler.timesteps[progress_id], latents)
++
++ # Decode
++ self.load_models_to_device(["vae"])
++ pred_videos = self.decode_video(latents, **tiler_kwargs)
++
++ if return_tensor:
++ return pred_videos
++
++ self.load_models_to_device([])
++ pred_video_list = []
++ for pred_video in pred_videos:
++ pred_video_list.append(self.tensor2video(pred_video))
++ return pred_video_list
++
++
++class TeaCache:
++ def __init__(self, num_inference_steps, rel_l1_thresh, model_id):
++ self.num_inference_steps = num_inference_steps
++ self.step = 0
++ self.accumulated_rel_l1_distance = 0
++ self.previous_modulated_input = None
++ self.rel_l1_thresh = rel_l1_thresh
++ self.previous_residual = None
++ self.previous_hidden_states = None
++
++ self.coefficients_dict = {
++ "Wan2.1-T2V-1.3B": [-5.21862437e04, 9.23041404e03, -5.28275948e02, 1.36987616e01, -4.99875664e-02],
++ "Wan2.1-T2V-14B": [-3.03318725e05, 4.90537029e04, -2.65530556e03, 5.87365115e01, -3.15583525e-01],
++ "Wan2.1-I2V-14B-480P": [2.57151496e05, -3.54229917e04, 1.40286849e03, -1.35890334e01, 1.32517977e-01],
++ "Wan2.1-I2V-14B-720P": [8.10705460e03, 2.13393892e03, -3.72934672e02, 1.66203073e01, -4.17769401e-02],
++ }
++ if model_id not in self.coefficients_dict:
++ supported_model_ids = ", ".join([i for i in self.coefficients_dict])
++ raise ValueError(
++ f"{model_id} is not a supported TeaCache model id. Please choose a valid model id in ({supported_model_ids})."
++ )
++ self.coefficients = self.coefficients_dict[model_id]
++
++ def check(self, dit: WanModel, x, t_mod):
++ modulated_inp = t_mod.clone()
++ if self.step == 0 or self.step == self.num_inference_steps - 1:
++ should_calc = True
++ self.accumulated_rel_l1_distance = 0
++ else:
++ coefficients = self.coefficients
++ rescale_func = np.poly1d(coefficients)
++ self.accumulated_rel_l1_distance += rescale_func(
++ (
++ (modulated_inp - self.previous_modulated_input).abs().mean()
++ / self.previous_modulated_input.abs().mean()
++ )
++ .cpu()
++ .item()
++ )
++ if self.accumulated_rel_l1_distance < self.rel_l1_thresh:
++ should_calc = False
++ else:
++ should_calc = True
++ self.accumulated_rel_l1_distance = 0
++ self.previous_modulated_input = modulated_inp
++ self.step += 1
++ if self.step == self.num_inference_steps:
++ self.step = 0
++ if should_calc:
++ self.previous_hidden_states = x.clone()
++ return not should_calc
++
++ def store(self, hidden_states):
++ self.previous_residual = hidden_states - self.previous_hidden_states
++ self.previous_hidden_states = None
++
++ def update(self, hidden_states):
++ hidden_states = hidden_states + self.previous_residual
++ return hidden_states
+diff --git a/diffsynth/schedulers/__init__.py b/diffsynth/schedulers/__init__.py
+index 0ec4325..03cae00 100644
+--- a/diffsynth/schedulers/__init__.py
++++ b/diffsynth/schedulers/__init__.py
+@@ -1,3 +1,4 @@
+ from .ddim import EnhancedDDIMScheduler
+ from .continuous_ode import ContinuousODEScheduler
+ from .flow_match import FlowMatchScheduler
++from .bride_match import BridgeMatchScheduler
+diff --git a/diffsynth/schedulers/bride_match.py b/diffsynth/schedulers/bride_match.py
+new file mode 100644
+index 0000000..c157702
+--- /dev/null
++++ b/diffsynth/schedulers/bride_match.py
+@@ -0,0 +1,71 @@
++import torch, math
++
++
++class BridgeMatchScheduler:
++
++ def __init__(
++ self,
++ num_train_timesteps=1000,
++ num_inference_steps=8,
++ sigma_max=1.0,
++ bridge_noise_sigma=0.005,
++ ):
++ self.sigma_max = sigma_max
++ self.sigma_min = sigma_max / num_train_timesteps
++ self.num_train_timesteps = num_train_timesteps # train timesteps for base model
++ self.num_inference_steps = num_inference_steps # inference steps for bridge matching
++ self.bridge_noise_sigma = bridge_noise_sigma
++
++ self.set_timesteps(self.num_inference_steps)
++
++ def set_timesteps(self, num_inference_steps=8, training=False):
++ sigma_start = self.sigma_max
++ sigma_end = self.sigma_max / num_inference_steps
++ self.sigmas = torch.linspace(sigma_start, sigma_end, num_inference_steps)
++ self.timesteps = self.sigmas * self.num_train_timesteps
++
++ self.training = training
++
++ def retrieve_sigma(self, timestep):
++ if isinstance(timestep, torch.Tensor):
++ timestep = timestep.cpu()
++ timestep_id = torch.argmin((self.timesteps - timestep).abs())
++ sigma = self.sigmas[timestep_id]
++ return sigma
++
++ def get_noise_term(self, sigma, sample):
++ # bridge noise term == 0 when sigma == 1 or 0
++ return self.bridge_noise_sigma * (sigma * (1.0 - sigma)) ** 0.5 * torch.randn_like(sample)
++
++ def step(self, model_output, timestep, sample, to_final=False, **kwargs):
++ if isinstance(timestep, torch.Tensor):
++ timestep = timestep.cpu()
++ timestep_id = torch.argmin((self.timesteps - timestep).abs())
++ sigma = self.sigmas[timestep_id]
++ if to_final or timestep_id + 1 >= len(self.timesteps):
++ sigma_ = 0
++ else:
++ sigma_ = self.sigmas[timestep_id + 1]
++
++ prev_sample = sample + model_output * (sigma_ - sigma) + self.get_noise_term(sigma_, sample)
++ return prev_sample
++
++ def add_noise(self, tgt_sample, src_sample, timestep):
++ sigma = self.retrieve_sigma(timestep)
++ noisy_sample = sigma * src_sample + (1 - sigma) * tgt_sample + self.get_noise_term(sigma, tgt_sample)
++ return noisy_sample
++
++ def training_target(self, tgt_sample, noisy_sample, timestep):
++ sigma = self.retrieve_sigma(timestep)
++
++ target = (noisy_sample - tgt_sample) / sigma
++ return target
++
++ def denoised_sample(self, prediction, noisy_sample, timestep):
++ sigma = self.retrieve_sigma(timestep)
++
++ sample = noisy_sample - prediction * sigma
++ return sample
++
++ def training_weight(self, timestep):
++ return 1.0
+diff --git a/diffsynth/schedulers/flow_match.py b/diffsynth/schedulers/flow_match.py
+index 6a8e235..0cd3e08 100644
+--- a/diffsynth/schedulers/flow_match.py
++++ b/diffsynth/schedulers/flow_match.py
+@@ -98,7 +98,12 @@ class FlowMatchScheduler():
+ def training_target(self, sample, noise, timestep):
+ target = noise - sample
+ return target
+-
++
++
++ def denoised_sample(self, prediction, noise, timestep):
++ sample = noise - prediction
++ return sample
++
+
+ def training_weight(self, timestep):
+ timestep_id = torch.argmin((self.timesteps - timestep.to(self.timesteps.device)).abs())
diff --git a/fdanyone/vendor/diffsynth/VENDORED_FILES.txt b/fdanyone/vendor/diffsynth/VENDORED_FILES.txt
new file mode 100644
index 0000000..136e462
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/VENDORED_FILES.txt
@@ -0,0 +1,25 @@
+# Paths are relative to fdanyone/vendor/diffsynth.
+LICENSE
+UPSTREAM.md
+UPSTREAM.patch
+VENDORED_FILES.txt
+__init__.py
+models/__init__.py
+models/utils.py
+models/wan_video_camera_controller.py
+models/wan_video_dit.py
+models/wan_video_pose_encoder.py
+models/wan_video_text_encoder.py
+models/wan_video_vae.py
+pipelines/__init__.py
+pipelines/base.py
+pipelines/wan_video_spatem.py
+prompters/__init__.py
+prompters/base_prompter.py
+prompters/wan_prompter.py
+schedulers/__init__.py
+schedulers/bride_match.py
+schedulers/flow_match.py
+vram_management/__init__.py
+vram_management/gradient_checkpointing.py
+vram_management/layers.py
diff --git a/fdanyone/vendor/diffsynth/__init__.py b/fdanyone/vendor/diffsynth/__init__.py
new file mode 100644
index 0000000..3d29a80
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/__init__.py
@@ -0,0 +1 @@
+"""Minimal DiffSynth-Studio inference closure used by 4DAnyone."""
diff --git a/fdanyone/vendor/diffsynth/models/__init__.py b/fdanyone/vendor/diffsynth/models/__init__.py
new file mode 100644
index 0000000..125a76b
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/models/__init__.py
@@ -0,0 +1,7 @@
+"""Vendored Wan model definitions."""
+
+from .wan_video_dit import WanModel
+from .wan_video_text_encoder import WanTextEncoder
+from .wan_video_vae import WanVideoVAE
+
+__all__ = ["WanModel", "WanTextEncoder", "WanVideoVAE"]
diff --git a/fdanyone/vendor/diffsynth/models/utils.py b/fdanyone/vendor/diffsynth/models/utils.py
new file mode 100644
index 0000000..86104d0
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/models/utils.py
@@ -0,0 +1,182 @@
+import torch, os
+from safetensors import safe_open
+from contextlib import contextmanager
+import hashlib
+
+@contextmanager
+def init_weights_on_device(device = torch.device("meta"), include_buffers :bool = False):
+
+ old_register_parameter = torch.nn.Module.register_parameter
+ if include_buffers:
+ old_register_buffer = torch.nn.Module.register_buffer
+
+ def register_empty_parameter(module, name, param):
+ old_register_parameter(module, name, param)
+ if param is not None:
+ param_cls = type(module._parameters[name])
+ kwargs = module._parameters[name].__dict__
+ kwargs["requires_grad"] = param.requires_grad
+ module._parameters[name] = param_cls(module._parameters[name].to(device), **kwargs)
+
+ def register_empty_buffer(module, name, buffer, persistent=True):
+ old_register_buffer(module, name, buffer, persistent=persistent)
+ if buffer is not None:
+ module._buffers[name] = module._buffers[name].to(device)
+
+ def patch_tensor_constructor(fn):
+ def wrapper(*args, **kwargs):
+ kwargs["device"] = device
+ return fn(*args, **kwargs)
+
+ return wrapper
+
+ if include_buffers:
+ tensor_constructors_to_patch = {
+ torch_function_name: getattr(torch, torch_function_name)
+ for torch_function_name in ["empty", "zeros", "ones", "full"]
+ }
+ else:
+ tensor_constructors_to_patch = {}
+
+ try:
+ torch.nn.Module.register_parameter = register_empty_parameter
+ if include_buffers:
+ torch.nn.Module.register_buffer = register_empty_buffer
+ for torch_function_name in tensor_constructors_to_patch.keys():
+ setattr(torch, torch_function_name, patch_tensor_constructor(getattr(torch, torch_function_name)))
+ yield
+ finally:
+ torch.nn.Module.register_parameter = old_register_parameter
+ if include_buffers:
+ torch.nn.Module.register_buffer = old_register_buffer
+ for torch_function_name, old_torch_function in tensor_constructors_to_patch.items():
+ setattr(torch, torch_function_name, old_torch_function)
+
+def load_state_dict_from_folder(file_path, torch_dtype=None):
+ state_dict = {}
+ for file_name in os.listdir(file_path):
+ if "." in file_name and file_name.split(".")[-1] in [
+ "safetensors", "bin", "ckpt", "pth", "pt"
+ ]:
+ state_dict.update(load_state_dict(os.path.join(file_path, file_name), torch_dtype=torch_dtype))
+ return state_dict
+
+
+def load_state_dict(file_path, torch_dtype=None, device="cpu"):
+ if file_path.endswith(".safetensors"):
+ return load_state_dict_from_safetensors(file_path, torch_dtype=torch_dtype, device=device)
+ else:
+ return load_state_dict_from_bin(file_path, torch_dtype=torch_dtype, device=device)
+
+
+def load_state_dict_from_safetensors(file_path, torch_dtype=None, device="cpu"):
+ state_dict = {}
+ with safe_open(file_path, framework="pt", device=str(device)) as f:
+ for k in f.keys():
+ state_dict[k] = f.get_tensor(k)
+ if torch_dtype is not None:
+ state_dict[k] = state_dict[k].to(torch_dtype)
+ return state_dict
+
+
+def load_state_dict_from_bin(file_path, torch_dtype=None, device="cpu"):
+ state_dict = torch.load(file_path, map_location=device, weights_only=True)
+ if torch_dtype is not None:
+ for i in state_dict:
+ if isinstance(state_dict[i], torch.Tensor):
+ state_dict[i] = state_dict[i].to(torch_dtype)
+ return state_dict
+
+
+def search_for_embeddings(state_dict):
+ embeddings = []
+ for k in state_dict:
+ if isinstance(state_dict[k], torch.Tensor):
+ embeddings.append(state_dict[k])
+ elif isinstance(state_dict[k], dict):
+ embeddings += search_for_embeddings(state_dict[k])
+ return embeddings
+
+
+def search_parameter(param, state_dict):
+ for name, param_ in state_dict.items():
+ if param.numel() == param_.numel():
+ if param.shape == param_.shape:
+ if torch.dist(param, param_) < 1e-3:
+ return name
+ else:
+ if torch.dist(param.flatten(), param_.flatten()) < 1e-3:
+ return name
+ return None
+
+
+def build_rename_dict(source_state_dict, target_state_dict, split_qkv=False):
+ matched_keys = set()
+ with torch.no_grad():
+ for name in source_state_dict:
+ rename = search_parameter(source_state_dict[name], target_state_dict)
+ if rename is not None:
+ print(f'"{name}": "{rename}",')
+ matched_keys.add(rename)
+ elif split_qkv and len(source_state_dict[name].shape)>=1 and source_state_dict[name].shape[0]%3==0:
+ length = source_state_dict[name].shape[0] // 3
+ rename = []
+ for i in range(3):
+ rename.append(search_parameter(source_state_dict[name][i*length: i*length+length], target_state_dict))
+ if None not in rename:
+ print(f'"{name}": {rename},')
+ for rename_ in rename:
+ matched_keys.add(rename_)
+ for name in target_state_dict:
+ if name not in matched_keys:
+ print("Cannot find", name, target_state_dict[name].shape)
+
+
+def search_for_files(folder, extensions):
+ files = []
+ if os.path.isdir(folder):
+ for file in sorted(os.listdir(folder)):
+ files += search_for_files(os.path.join(folder, file), extensions)
+ elif os.path.isfile(folder):
+ for extension in extensions:
+ if folder.endswith(extension):
+ files.append(folder)
+ break
+ return files
+
+
+def convert_state_dict_keys_to_single_str(state_dict, with_shape=True):
+ keys = []
+ for key, value in state_dict.items():
+ if isinstance(key, str):
+ if isinstance(value, torch.Tensor):
+ if with_shape:
+ shape = "_".join(map(str, list(value.shape)))
+ keys.append(key + ":" + shape)
+ keys.append(key)
+ elif isinstance(value, dict):
+ keys.append(key + "|" + convert_state_dict_keys_to_single_str(value, with_shape=with_shape))
+ keys.sort()
+ keys_str = ",".join(keys)
+ return keys_str
+
+
+def split_state_dict_with_prefix(state_dict):
+ keys = sorted([key for key in state_dict if isinstance(key, str)])
+ prefix_dict = {}
+ for key in keys:
+ prefix = key if "." not in key else key.split(".")[0]
+ if prefix not in prefix_dict:
+ prefix_dict[prefix] = []
+ prefix_dict[prefix].append(key)
+ state_dicts = []
+ for prefix, keys in prefix_dict.items():
+ sub_state_dict = {key: state_dict[key] for key in keys}
+ state_dicts.append(sub_state_dict)
+ return state_dicts
+
+
+def hash_state_dict_keys(state_dict, with_shape=True):
+ keys_str = convert_state_dict_keys_to_single_str(state_dict, with_shape=with_shape)
+ keys_str = keys_str.encode(encoding="UTF-8")
+ return hashlib.md5(keys_str).hexdigest()
\ No newline at end of file
diff --git a/fdanyone/vendor/diffsynth/models/wan_video_camera_controller.py b/fdanyone/vendor/diffsynth/models/wan_video_camera_controller.py
new file mode 100644
index 0000000..8655804
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/models/wan_video_camera_controller.py
@@ -0,0 +1,209 @@
+import torch
+import torch.nn as nn
+import numpy as np
+from einops import rearrange
+import os
+from typing_extensions import Literal
+
+# Camera-coordinate helpers retained below are attributed to CameraCtrl
+# (https://github.com/hehao13/CameraCtrl), licensed under Apache-2.0.
+
+class SimpleAdapter(nn.Module):
+ def __init__(self, in_dim, out_dim, kernel_size, stride, num_residual_blocks=1):
+ super(SimpleAdapter, self).__init__()
+
+ # Pixel Unshuffle: reduce spatial dimensions by a factor of 8
+ self.pixel_unshuffle = nn.PixelUnshuffle(downscale_factor=8)
+
+ # Convolution: reduce spatial dimensions by a factor
+ # of 2 (without overlap)
+ self.conv = nn.Conv2d(in_dim * 64, out_dim, kernel_size=kernel_size, stride=stride, padding=0)
+
+ # Residual blocks for feature extraction
+ self.residual_blocks = nn.Sequential(
+ *[ResidualBlock(out_dim) for _ in range(num_residual_blocks)]
+ )
+
+ def forward(self, x):
+ # Reshape to merge the frame dimension into batch
+ bs, c, f, h, w = x.size()
+ x = x.permute(0, 2, 1, 3, 4).contiguous().view(bs * f, c, h, w)
+
+ # Pixel Unshuffle operation
+ x_unshuffled = self.pixel_unshuffle(x)
+
+ # Convolution operation
+ x_conv = self.conv(x_unshuffled)
+
+ # Feature extraction with residual blocks
+ out = self.residual_blocks(x_conv)
+
+ # Reshape to restore original bf dimension
+ out = out.view(bs, f, out.size(1), out.size(2), out.size(3))
+
+ # Permute dimensions to reorder (if needed), e.g., swap channels and feature frames
+ out = out.permute(0, 2, 1, 3, 4)
+
+ return out
+
+ def process_camera_coordinates(
+ self,
+ direction: Literal["Left", "Right", "Up", "Down", "LeftUp", "LeftDown", "RightUp", "RightDown"],
+ length: int,
+ height: int,
+ width: int,
+ speed: float = 1/54,
+ origin=(0, 0.532139961, 0.946026558, 0.5, 0.5, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0)
+ ):
+ if origin is None:
+ origin = (0, 0.532139961, 0.946026558, 0.5, 0.5, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0)
+ coordinates = generate_camera_coordinates(direction, length, speed, origin)
+ plucker_embedding = process_pose_file(coordinates, width, height)
+ return plucker_embedding
+
+
+
+class ResidualBlock(nn.Module):
+ def __init__(self, dim):
+ super(ResidualBlock, self).__init__()
+ self.conv1 = nn.Conv2d(dim, dim, kernel_size=3, padding=1)
+ self.relu = nn.ReLU(inplace=True)
+ self.conv2 = nn.Conv2d(dim, dim, kernel_size=3, padding=1)
+
+ def forward(self, x):
+ residual = x
+ out = self.relu(self.conv1(x))
+ out = self.conv2(out)
+ out += residual
+ return out
+
+class Camera(object):
+ """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+ """
+ def __init__(self, entry):
+ fx, fy, cx, cy = entry[1:5]
+ self.fx = fx
+ self.fy = fy
+ self.cx = cx
+ self.cy = cy
+ w2c_mat = np.array(entry[7:]).reshape(3, 4)
+ w2c_mat_4x4 = np.eye(4)
+ w2c_mat_4x4[:3, :] = w2c_mat
+ self.w2c_mat = w2c_mat_4x4
+ self.c2w_mat = np.linalg.inv(w2c_mat_4x4)
+
+def get_relative_pose(cam_params):
+ """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+ """
+ abs_w2cs = [cam_param.w2c_mat for cam_param in cam_params]
+ abs_c2ws = [cam_param.c2w_mat for cam_param in cam_params]
+ cam_to_origin = 0
+ target_cam_c2w = np.array([
+ [1, 0, 0, 0],
+ [0, 1, 0, -cam_to_origin],
+ [0, 0, 1, 0],
+ [0, 0, 0, 1]
+ ])
+ abs2rel = target_cam_c2w @ abs_w2cs[0]
+ ret_poses = [target_cam_c2w, ] + [abs2rel @ abs_c2w for abs_c2w in abs_c2ws[1:]]
+ ret_poses = np.array(ret_poses, dtype=np.float32)
+ return ret_poses
+
+def custom_meshgrid(*args):
+ # torch>=2.0.0 only
+ return torch.meshgrid(*args, indexing='ij')
+
+
+def ray_condition(K, c2w, H, W, device):
+ """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+ """
+ # c2w: B, V, 4, 4
+ # K: B, V, 4
+
+ B = K.shape[0]
+
+ j, i = custom_meshgrid(
+ torch.linspace(0, H - 1, H, device=device, dtype=c2w.dtype),
+ torch.linspace(0, W - 1, W, device=device, dtype=c2w.dtype),
+ )
+ i = i.reshape([1, 1, H * W]).expand([B, 1, H * W]) + 0.5 # [B, HxW]
+ j = j.reshape([1, 1, H * W]).expand([B, 1, H * W]) + 0.5 # [B, HxW]
+
+ fx, fy, cx, cy = K.chunk(4, dim=-1) # B,V, 1
+
+ zs = torch.ones_like(i) # [B, HxW]
+ xs = (i - cx) / fx * zs
+ ys = (j - cy) / fy * zs
+ zs = zs.expand_as(ys)
+
+ directions = torch.stack((xs, ys, zs), dim=-1) # B, V, HW, 3
+ directions = directions / directions.norm(dim=-1, keepdim=True) # B, V, HW, 3
+
+ rays_d = directions @ c2w[..., :3, :3].transpose(-1, -2) # B, V, 3, HW
+ rays_o = c2w[..., :3, 3] # B, V, 3
+ rays_o = rays_o[:, :, None].expand_as(rays_d) # B, V, 3, HW
+ # c2w @ dirctions
+ rays_dxo = torch.linalg.cross(rays_o, rays_d)
+ plucker = torch.cat([rays_dxo, rays_d], dim=-1)
+ plucker = plucker.reshape(B, c2w.shape[1], H, W, 6) # B, V, H, W, 6
+ # plucker = plucker.permute(0, 1, 4, 2, 3)
+ return plucker
+
+
+def process_pose_file(cam_params, width=672, height=384, original_pose_width=1280, original_pose_height=720, device='cpu', return_poses=False):
+ if return_poses:
+ return cam_params
+ else:
+ cam_params = [Camera(cam_param) for cam_param in cam_params]
+
+ sample_wh_ratio = width / height
+ pose_wh_ratio = original_pose_width / original_pose_height # Assuming placeholder ratios, change as needed
+
+ if pose_wh_ratio > sample_wh_ratio:
+ resized_ori_w = height * pose_wh_ratio
+ for cam_param in cam_params:
+ cam_param.fx = resized_ori_w * cam_param.fx / width
+ else:
+ resized_ori_h = width / pose_wh_ratio
+ for cam_param in cam_params:
+ cam_param.fy = resized_ori_h * cam_param.fy / height
+
+ intrinsic = np.asarray([[cam_param.fx * width,
+ cam_param.fy * height,
+ cam_param.cx * width,
+ cam_param.cy * height]
+ for cam_param in cam_params], dtype=np.float32)
+
+ K = torch.as_tensor(intrinsic)[None] # [1, 1, 4]
+ c2ws = get_relative_pose(cam_params) # Assuming this function is defined elsewhere
+ c2ws = torch.as_tensor(c2ws)[None] # [1, n_frame, 4, 4]
+ plucker_embedding = ray_condition(K, c2ws, height, width, device=device)[0].permute(0, 3, 1, 2).contiguous() # V, 6, H, W
+ plucker_embedding = plucker_embedding[None]
+ plucker_embedding = rearrange(plucker_embedding, "b f c h w -> b f h w c")[0]
+ return plucker_embedding
+
+
+
+def generate_camera_coordinates(
+ direction: Literal["Left", "Right", "Up", "Down", "LeftUp", "LeftDown", "RightUp", "RightDown", "In", "Out"],
+ length: int,
+ speed: float = 1/54,
+ origin=(0, 0.532139961, 0.946026558, 0.5, 0.5, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0)
+):
+ coordinates = [list(origin)]
+ while len(coordinates) < length:
+ coor = coordinates[-1].copy()
+ if "Left" in direction:
+ coor[9] += speed
+ if "Right" in direction:
+ coor[9] -= speed
+ if "Up" in direction:
+ coor[13] += speed
+ if "Down" in direction:
+ coor[13] -= speed
+ if "In" in direction:
+ coor[18] -= speed
+ if "Out" in direction:
+ coor[18] += speed
+ coordinates.append(coor)
+ return coordinates
diff --git a/fdanyone/vendor/diffsynth/models/wan_video_dit.py b/fdanyone/vendor/diffsynth/models/wan_video_dit.py
new file mode 100644
index 0000000..f2cc2e7
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/models/wan_video_dit.py
@@ -0,0 +1,1022 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import math
+from typing import Tuple, Optional
+from einops import rearrange, repeat
+from .utils import hash_state_dict_keys
+from .wan_video_camera_controller import SimpleAdapter
+try:
+ import flash_attn_interface
+ FLASH_ATTN_3_AVAILABLE = True
+except (ImportError, OSError, RuntimeError):
+ FLASH_ATTN_3_AVAILABLE = False
+
+try:
+ import flash_attn
+ FLASH_ATTN_2_AVAILABLE = True
+except (ImportError, OSError, RuntimeError):
+ FLASH_ATTN_2_AVAILABLE = False
+
+try:
+ from sageattention import sageattn
+ SAGE_ATTN_AVAILABLE = True
+except (ImportError, OSError, RuntimeError):
+ SAGE_ATTN_AVAILABLE = False
+
+try:
+ from spas_sage_attn import spas_sage2_attn_meansim_topk_cuda
+ SPARGE_ATTN_AVAILABLE = True
+except (ImportError, OSError, RuntimeError):
+ SPARGE_ATTN_AVAILABLE = False
+
+
+def get_attention_backend() -> str:
+ """Return the implementation selected by the vendored auto policy."""
+
+ if FLASH_ATTN_3_AVAILABLE:
+ return "flash_attn_3"
+ if FLASH_ATTN_2_AVAILABLE:
+ return "flash_attn_2"
+ if SAGE_ATTN_AVAILABLE:
+ return "sageattention"
+ return "sdpa"
+
+
+def flash_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, num_heads: int, compatibility_mode=False):
+ if compatibility_mode:
+ q = rearrange(q, "b s (n d) -> b n s d", n=num_heads)
+ k = rearrange(k, "b s (n d) -> b n s d", n=num_heads)
+ v = rearrange(v, "b s (n d) -> b n s d", n=num_heads)
+ x = F.scaled_dot_product_attention(q, k, v)
+ x = rearrange(x, "b n s d -> b s (n d)", n=num_heads)
+ elif FLASH_ATTN_3_AVAILABLE:
+ q = rearrange(q, "b s (n d) -> b s n d", n=num_heads)
+ k = rearrange(k, "b s (n d) -> b s n d", n=num_heads)
+ v = rearrange(v, "b s (n d) -> b s n d", n=num_heads)
+ x = flash_attn_interface.flash_attn_func(q, k, v)
+ if isinstance(x,tuple):
+ x = x[0]
+ x = rearrange(x, "b s n d -> b s (n d)", n=num_heads)
+ elif FLASH_ATTN_2_AVAILABLE:
+ q = rearrange(q, "b s (n d) -> b s n d", n=num_heads)
+ k = rearrange(k, "b s (n d) -> b s n d", n=num_heads)
+ v = rearrange(v, "b s (n d) -> b s n d", n=num_heads)
+ x = flash_attn.flash_attn_func(q, k, v)
+ x = rearrange(x, "b s n d -> b s (n d)", n=num_heads)
+ # TODO: test spas_sage2_attn_meansim_topk_cuda
+ # elif SPARGE_ATTN_AVAILABLE:
+ # q = rearrange(q, "b s (n d) -> b n s d", n=num_heads)
+ # k = rearrange(k, "b s (n d) -> b n s d", n=num_heads)
+ # v = rearrange(v, "b s (n d) -> b n s d", n=num_heads)
+ # x = spas_sage2_attn_meansim_topk_cuda(q, k, v, simthreshd1=-0.1, topk=0.5, pvthreshd=15, is_causal=False)
+ # x = rearrange(x, "b n s d -> b s (n d)", n=num_heads)
+ elif SAGE_ATTN_AVAILABLE:
+ q = rearrange(q, "b s (n d) -> b n s d", n=num_heads)
+ k = rearrange(k, "b s (n d) -> b n s d", n=num_heads)
+ v = rearrange(v, "b s (n d) -> b n s d", n=num_heads)
+ x = sageattn(q, k, v)
+ x = rearrange(x, "b n s d -> b s (n d)", n=num_heads)
+ else:
+ q = rearrange(q, "b s (n d) -> b n s d", n=num_heads)
+ k = rearrange(k, "b s (n d) -> b n s d", n=num_heads)
+ v = rearrange(v, "b s (n d) -> b n s d", n=num_heads)
+ x = F.scaled_dot_product_attention(q, k, v)
+ x = rearrange(x, "b n s d -> b s (n d)", n=num_heads)
+ return x
+
+
+def modulate(x: torch.Tensor, shift: torch.Tensor, scale: torch.Tensor):
+ return (x * (1 + scale) + shift)
+
+
+def sinusoidal_embedding_1d(dim, position):
+ sinusoid = torch.outer(position.type(torch.float64), torch.pow(
+ 10000, -torch.arange(dim//2, dtype=torch.float64, device=position.device).div(dim//2)))
+ x = torch.cat([torch.cos(sinusoid), torch.sin(sinusoid)], dim=1)
+ return x.to(position.dtype)
+
+
+def precompute_freqs_cis_3d(dim: int, end: int = 1024, theta: float = 10000.0):
+ # 3d rope precompute
+ f_freqs_cis = precompute_freqs_cis(dim - 2 * (dim // 3), end, theta)
+ h_freqs_cis = precompute_freqs_cis(dim // 3, end, theta)
+ w_freqs_cis = precompute_freqs_cis(dim // 3, end, theta)
+ return f_freqs_cis, h_freqs_cis, w_freqs_cis
+
+
+def precompute_freqs_cis(dim: int, end: int = 1024, theta: float = 10000.0):
+ # 1d rope precompute
+ freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)
+ [: (dim // 2)].double() / dim))
+ freqs = torch.outer(torch.arange(end, device=freqs.device), freqs)
+ freqs_cis = torch.polar(torch.ones_like(freqs), freqs) # complex64
+ return freqs_cis
+
+
+def rope_apply(x, freqs, num_heads):
+ x = rearrange(x, "b s (n d) -> b s n d", n=num_heads)
+ x_out = torch.view_as_complex(x.to(torch.float64).reshape(
+ x.shape[0], x.shape[1], x.shape[2], -1, 2))
+ x_out = torch.view_as_real(x_out * freqs).flatten(2)
+ return x_out.to(x.dtype)
+
+
+class RMSNorm(nn.Module):
+ def __init__(self, dim, eps=1e-5):
+ super().__init__()
+ self.eps = eps
+ self.weight = nn.Parameter(torch.ones(dim))
+
+ def norm(self, x):
+ return x * torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.eps)
+
+ def forward(self, x):
+ dtype = x.dtype
+ return self.norm(x.float()).to(dtype) * self.weight
+
+
+class AttentionModule(nn.Module):
+ def __init__(self, num_heads):
+ super().__init__()
+ self.num_heads = num_heads
+
+ def forward(self, q, k, v):
+ x = flash_attention(q=q, k=k, v=v, num_heads=self.num_heads)
+ return x
+
+
+class SelfAttention(nn.Module):
+ def __init__(self, dim: int, num_heads: int, eps: float = 1e-6):
+ super().__init__()
+ self.dim = dim
+ self.num_heads = num_heads
+ self.head_dim = dim // num_heads
+
+ self.q = nn.Linear(dim, dim)
+ self.k = nn.Linear(dim, dim)
+ self.v = nn.Linear(dim, dim)
+ self.o = nn.Linear(dim, dim)
+ self.norm_q = RMSNorm(dim, eps=eps)
+ self.norm_k = RMSNorm(dim, eps=eps)
+
+ self.attn = AttentionModule(self.num_heads)
+
+ def forward(self, x, freqs):
+ q = self.norm_q(self.q(x))
+ k = self.norm_k(self.k(x))
+ v = self.v(x)
+ q = rope_apply(q, freqs, self.num_heads)
+ k = rope_apply(k, freqs, self.num_heads)
+ x = self.attn(q, k, v)
+ return self.o(x)
+
+
+class CrossAttention(nn.Module):
+ def __init__(self, dim: int, num_heads: int, eps: float = 1e-6, has_image_input: bool = False):
+ super().__init__()
+ self.dim = dim
+ self.num_heads = num_heads
+ self.head_dim = dim // num_heads
+
+ self.q = nn.Linear(dim, dim)
+ self.k = nn.Linear(dim, dim)
+ self.v = nn.Linear(dim, dim)
+ self.o = nn.Linear(dim, dim)
+ self.norm_q = RMSNorm(dim, eps=eps)
+ self.norm_k = RMSNorm(dim, eps=eps)
+ self.has_image_input = has_image_input
+ if has_image_input:
+ self.k_img = nn.Linear(dim, dim)
+ self.v_img = nn.Linear(dim, dim)
+ self.norm_k_img = RMSNorm(dim, eps=eps)
+
+ self.attn = AttentionModule(self.num_heads)
+
+ def forward(self, x: torch.Tensor, y: torch.Tensor):
+ if self.has_image_input:
+ img = y[:, :257]
+ ctx = y[:, 257:]
+ else:
+ ctx = y
+ q = self.norm_q(self.q(x))
+ k = self.norm_k(self.k(ctx))
+ v = self.v(ctx)
+ x = self.attn(q, k, v)
+ if self.has_image_input:
+ k_img = self.norm_k_img(self.k_img(img))
+ v_img = self.v_img(img)
+ y = flash_attention(q, k_img, v_img, num_heads=self.num_heads)
+ x = x + y
+ return self.o(x)
+
+
+class CrossAttentionSrcCam(nn.Module):
+ def __init__(self, dim: int, num_heads: int, eps: float = 1e-6):
+ super().__init__()
+ self.dim = dim
+ self.num_heads = num_heads
+ self.head_dim = dim // num_heads
+
+ self.q = nn.Linear(dim, dim)
+ self.k = nn.Linear(dim, dim)
+ self.v = nn.Linear(dim, dim)
+ self.o = nn.Linear(dim, dim)
+ self.norm_q = RMSNorm(dim, eps=eps)
+ self.norm_k = RMSNorm(dim, eps=eps)
+
+ self.attn = AttentionModule(self.num_heads)
+
+ def forward(self, x, x_src, freqs, freqs_src):
+ q = self.norm_q(self.q(x))
+ k = self.norm_k(self.k(x_src))
+ v = self.v(x_src)
+ q = rope_apply(q, freqs, self.num_heads)
+ k = rope_apply(k, freqs_src, self.num_heads)
+ x = self.attn(q, k, v)
+ return self.o(x)
+
+
+class GateModule(nn.Module):
+ def __init__(self,):
+ super().__init__()
+
+ def forward(self, x, gate, residual):
+ return x + gate * residual
+
+class DiTBlock(nn.Module):
+ def __init__(self, has_image_input: bool, dim: int, num_heads: int, ffn_dim: int, eps: float = 1e-6):
+ super().__init__()
+ self.dim = dim
+ self.num_heads = num_heads
+ self.ffn_dim = ffn_dim
+
+ self.disable_video_attn = False
+ self.use_4d_attn = False
+ self.use_mvs_attn = False
+ self.use_src_self_attn = False
+ self.use_src_cross_attn = False
+ self.use_cam_encoder = False
+
+ self.self_attn = SelfAttention(dim, num_heads, eps)
+ self.cross_attn = CrossAttention(
+ dim, num_heads, eps, has_image_input=has_image_input)
+ self.norm1 = nn.LayerNorm(dim, eps=eps, elementwise_affine=False)
+ self.norm2 = nn.LayerNorm(dim, eps=eps, elementwise_affine=False)
+ self.norm3 = nn.LayerNorm(dim, eps=eps)
+ self.ffn = nn.Sequential(nn.Linear(dim, ffn_dim), nn.GELU(
+ approximate='tanh'), nn.Linear(ffn_dim, dim))
+ self.modulation = nn.Parameter(torch.randn(1, 6, dim) / dim**0.5)
+ self.gate = GateModule()
+
+ def forward(self, x, x_src, context, t_mod, freqs, freqs_mvs, freqs_src, cam_emb, shape):
+ v, f, h, w = shape
+
+ has_seq = len(t_mod.shape) == 4
+ chunk_dim = 2 if has_seq else 1
+ # msa: multi-head self-attention mlp: multi-layer perceptron
+ shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = (
+ self.modulation.to(dtype=t_mod.dtype, device=t_mod.device) + t_mod).chunk(6, dim=chunk_dim)
+ if has_seq:
+ shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = (
+ shift_msa.squeeze(2), scale_msa.squeeze(2), gate_msa.squeeze(2),
+ shift_mlp.squeeze(2), scale_mlp.squeeze(2), gate_mlp.squeeze(2),
+ )
+
+ # video self-attention
+ if not self.disable_video_attn:
+ input_x = modulate(self.norm1(x), shift_msa, scale_msa)
+ if self.use_4d_attn:
+ input_x = rearrange(input_x, "v fhw c -> (v fhw) c").unsqueeze(0)
+ freqs = repeat(freqs, "fhw 1 c -> (v fhw) 1 c", v=v)
+ input_x = self.self_attn(input_x, freqs)
+ if self.use_4d_attn:
+ input_x = rearrange(input_x.squeeze(0), "(v fhw) c -> v fhw c", v=v)
+ x = self.gate(x, gate_msa, input_x)
+
+ # source-view self-attention
+ if self.use_src_self_attn:
+ x_cat = torch.cat([x, x_src], dim=1)
+ shift_src, scale_src, gate_src = (
+ self.modulation_src.to(dtype=t_mod.dtype, device=t_mod.device) + t_mod[:, :3, :]).chunk(3, dim=1)
+ input_x_cat = modulate(self.norm1_src(x_cat), shift_src, scale_src)
+
+ freqs_cat = torch.cat([freqs, freqs_src], dim=0)
+ input_x_cat = self.self_attn_src(input_x_cat, freqs_cat)
+ x_cat = self.gate(x_cat, gate_src, input_x_cat)
+ len_src = x_src.shape[1]
+ x, x_src = x_cat[:, :-len_src, ...], x_cat[:, -len_src:, ...]
+
+ # source-view cross-attention
+ if self.use_src_cross_attn:
+ shift_src, scale_src, gate_src = (
+ self.modulation_src.to(dtype=t_mod.dtype, device=t_mod.device) + t_mod[:, :3, :]).chunk(3, dim=1)
+ input_x = modulate(self.norm1_src(x), shift_src, scale_src)
+ input_x_src = self.norm1_src(x_src)
+
+ input_x = self.cross_attn_src(input_x, input_x_src, freqs, freqs_src)
+ x = self.gate(x, gate_src, input_x)
+
+ # multiview self-attention
+ if self.use_mvs_attn:
+ shift_mvs, scale_mvs, gate_mvs = (
+ self.modulation_mvs.to(dtype=t_mod.dtype, device=t_mod.device) + t_mod[:, :3, :]).chunk(3, dim=1)
+ input_x = modulate(self.norm1_mvs(x), shift_mvs, scale_mvs)
+
+ # add camera embedding before multiview attention
+ if self.use_cam_encoder and cam_emb is not None:
+ cam_proj = self.cam_encoder(cam_emb) # (v, 1, dim)
+ cam_proj = cam_proj.unsqueeze(2).unsqueeze(3).expand(-1, f, h, w, -1) # (v, f, h, w, dim)
+ cam_proj = rearrange(cam_proj, "v f h w d -> v (f h w) d")
+ input_x = input_x + cam_proj
+
+ input_x = rearrange(input_x, "v (f h w) c -> f (v h w) c", v=v, f=f, h=h, w=w)
+ input_x = self.self_attn_mvs(input_x, freqs_mvs)
+ input_x = rearrange(input_x, "f (v h w) c -> v (f h w) c", v=v, f=f, h=h, w=w)
+
+ # projector wraps multiview attention output
+ if self.use_cam_encoder:
+ input_x = self.projector(input_x)
+
+ x = self.gate(x, gate_mvs, input_x)
+
+ # prompt cross-attention
+ context = repeat(context, "1 l c -> v l c", v=x.shape[0])
+ x = x + self.cross_attn(self.norm3(x), context)
+
+ # feed-forward network
+ input_x = modulate(self.norm2(x), shift_mlp, scale_mlp)
+ x = self.gate(x, gate_mlp, self.ffn(input_x))
+
+ if self.use_src_self_attn:
+ # for src self-attention, the x_src is updated as well
+ x_src = x_src + self.cross_attn(self.norm3(x_src), context)
+
+ input_x_src = modulate(self.norm2(x_src), shift_mlp, scale_mlp)
+ x_src = self.gate(x_src, gate_mlp, self.ffn(input_x_src))
+
+ return x, x_src
+
+
+class MLP(torch.nn.Module):
+ def __init__(self, in_dim, out_dim, has_pos_emb=False):
+ super().__init__()
+ self.proj = torch.nn.Sequential(
+ nn.LayerNorm(in_dim),
+ nn.Linear(in_dim, in_dim),
+ nn.GELU(),
+ nn.Linear(in_dim, out_dim),
+ nn.LayerNorm(out_dim)
+ )
+ self.has_pos_emb = has_pos_emb
+ if has_pos_emb:
+ self.emb_pos = torch.nn.Parameter(torch.zeros((1, 514, 1280)))
+
+ def forward(self, x):
+ if self.has_pos_emb:
+ x = x + self.emb_pos.to(dtype=x.dtype, device=x.device)
+ return self.proj(x)
+
+
+class Head(nn.Module):
+ def __init__(self, dim: int, out_dim: int, patch_size: Tuple[int, int, int], eps: float):
+ super().__init__()
+ self.dim = dim
+ self.patch_size = patch_size
+ self.norm = nn.LayerNorm(dim, eps=eps, elementwise_affine=False)
+ self.head = nn.Linear(dim, out_dim * math.prod(patch_size))
+ self.modulation = nn.Parameter(torch.randn(1, 2, dim) / dim**0.5)
+
+ def forward(self, x, t_mod):
+ if len(t_mod.shape) == 3:
+ shift, scale = (self.modulation.unsqueeze(0).to(dtype=t_mod.dtype, device=t_mod.device) + t_mod.unsqueeze(2)).chunk(2, dim=2)
+ x = (self.head(self.norm(x) * (1 + scale.squeeze(2)) + shift.squeeze(2)))
+ else:
+ if t_mod.shape[0] != 1:
+ t_mod = t_mod[:, None, :] # [b, d] -> [b, 1, d]
+ shift, scale = (self.modulation.to(dtype=t_mod.dtype, device=t_mod.device) + t_mod).chunk(2, dim=1)
+ x = (self.head(self.norm(x) * (1 + scale) + shift))
+ return x
+
+
+def pad_for_3d_conv(x, kernel_size):
+ """Pad to be divisible by kernel_size. From FramePack."""
+ _, _, t, h, w = x.shape
+ pt, ph, pw = kernel_size
+ pad_t = (pt - (t % pt)) % pt
+ pad_h = (ph - (h % ph)) % ph
+ pad_w = (pw - (w % pw)) % pw
+ if pad_t == 0 and pad_h == 0 and pad_w == 0:
+ return x
+ return F.pad(x, (0, pad_w, 0, pad_h, 0, pad_t), mode='replicate')
+
+
+class ViewPackEmbedding(nn.Module):
+ """Multi-resolution spatial patch embedding for clean source views.
+ Spatial-only downsampling: temporal dim preserved.
+ Ref: HunyuanVideoPatchEmbedForCleanLatents in FramePack.
+
+ Supported configurations (all exactly fill 4 quadrants):
+ - 4×2x views (each 2x view → 1 quadrant)
+ - 3×2x + 4×4x (3 quadrants from 2x + 1 quadrant from 4×4x tile)
+ """
+
+ def __init__(self, in_dim, dim, patch_size):
+ super().__init__()
+ pt, ph, pw = patch_size # (1, 2, 2) for Wan
+ # 2x: spatial 2x downsample relative to 1x -> kernel (1, 4, 4)
+ self.proj_2x = nn.Conv3d(in_dim, dim, kernel_size=(pt, ph*2, pw*2), stride=(pt, ph*2, pw*2))
+ # 4x: spatial 4x downsample relative to 1x -> kernel (1, 8, 8)
+ self.proj_4x = nn.Conv3d(in_dim, dim, kernel_size=(pt, ph*4, pw*4), stride=(pt, ph*4, pw*4))
+
+ @torch.no_grad()
+ def initialize_from_patch_embedding(self, patch_embedding: nn.Conv3d):
+ """FramePack-style init: tile spatial dims and scale by 1/area_ratio."""
+ weight = patch_embedding.weight.detach().clone() # (dim, in_dim, 1, 2, 2)
+ bias = patch_embedding.bias.detach().clone()
+ sd = {
+ 'proj_2x.weight': repeat(weight, 'b c t h w -> b c t (h 2) (w 2)') / 4.0,
+ 'proj_2x.bias': bias.clone(),
+ 'proj_4x.weight': repeat(weight, 'b c t h w -> b c t (h 4) (w 4)') / 16.0,
+ 'proj_4x.bias': bias.clone(),
+ }
+ self.load_state_dict(sd)
+
+
+class WanModel(torch.nn.Module):
+ def __init__(
+ self,
+ dim: int,
+ in_dim: int,
+ ffn_dim: int,
+ out_dim: int,
+ text_dim: int,
+ freq_dim: int,
+ eps: float,
+ patch_size: Tuple[int, int, int],
+ num_heads: int,
+ num_layers: int,
+ has_image_input: bool,
+ has_image_pos_emb: bool = False,
+ has_ref_conv: bool = False,
+ add_control_adapter: bool = False,
+ in_dim_control_adapter: int = 24,
+ seperated_timestep: bool = False,
+ require_vae_embedding: bool = True,
+ require_clip_embedding: bool = True,
+ fuse_vae_embedding_in_latents: bool = False,
+ ):
+ super().__init__()
+ self.dim = dim
+ self.in_dim = in_dim
+ self.freq_dim = freq_dim
+ self.has_image_input = has_image_input
+ self.patch_size = patch_size
+ self.seperated_timestep = seperated_timestep
+ self.require_vae_embedding = require_vae_embedding
+ self.require_clip_embedding = require_clip_embedding
+ self.fuse_vae_embedding_in_latents = fuse_vae_embedding_in_latents
+
+ self.patch_embedding = nn.Conv3d(
+ in_dim, dim, kernel_size=patch_size, stride=patch_size)
+ self.text_embedding = nn.Sequential(
+ nn.Linear(text_dim, dim),
+ nn.GELU(approximate='tanh'),
+ nn.Linear(dim, dim)
+ )
+ self.time_embedding = nn.Sequential(
+ nn.Linear(freq_dim, dim),
+ nn.SiLU(),
+ nn.Linear(dim, dim)
+ )
+ self.time_projection = nn.Sequential(
+ nn.SiLU(), nn.Linear(dim, dim * 6))
+ self.blocks = nn.ModuleList([
+ DiTBlock(has_image_input, dim, num_heads, ffn_dim, eps)
+ for _ in range(num_layers)
+ ])
+ self.head = Head(dim, out_dim, patch_size, eps)
+ head_dim = dim // num_heads
+ self.freqs = precompute_freqs_cis_3d(head_dim)
+
+ if has_image_input:
+ self.img_emb = MLP(1280, dim, has_pos_emb=has_image_pos_emb) # clip_feature_dim = 1280
+ if has_ref_conv:
+ self.ref_conv = nn.Conv2d(16, dim, kernel_size=(2, 2), stride=(2, 2))
+ self.has_image_pos_emb = has_image_pos_emb
+ self.has_ref_conv = has_ref_conv
+ if add_control_adapter:
+ self.control_adapter = SimpleAdapter(in_dim_control_adapter, dim, kernel_size=patch_size[1:], stride=patch_size[1:])
+ else:
+ self.control_adapter = None
+
+ def patchify(self, x: torch.Tensor, control_camera_latents_input: Optional[torch.Tensor] = None):
+ x = self.patch_embedding(x)
+ if self.control_adapter is not None and control_camera_latents_input is not None:
+ y_camera = self.control_adapter(control_camera_latents_input)
+ x = [u + v for u, v in zip(x, y_camera)]
+ x = x[0].unsqueeze(0)
+ grid_size = x.shape[2:]
+ x = rearrange(x, 'b c f h w -> b (f h w) c').contiguous()
+ return x, grid_size # x, grid_size: (f, h, w)
+
+ def unpatchify(self, x: torch.Tensor, grid_size: torch.Tensor):
+ return rearrange(
+ x, 'b (f h w) (x y z c) -> b c (f x) (h y) (w z)',
+ f=grid_size[0], h=grid_size[1], w=grid_size[2],
+ x=self.patch_size[0], y=self.patch_size[1], z=self.patch_size[2]
+ )
+
+ def forward(self,
+ x: torch.Tensor,
+ x_src: torch.Tensor,
+ timestep: torch.Tensor,
+ context: torch.Tensor,
+ skeletons: Optional[torch.Tensor] = None,
+ cam_emb: Optional[torch.Tensor] = None,
+ drop_viewpack_tokens: bool = False,
+ clip_feature: Optional[torch.Tensor] = None,
+ y: Optional[torch.Tensor] = None,
+ use_gradient_checkpointing: bool = False,
+ use_gradient_checkpointing_offload: bool = False,
+ **kwargs,
+ ):
+ context = self.text_embedding(context)
+
+ if self.has_image_input:
+ x = torch.cat([x, y], dim=1) # (b, c_x + c_y, f, h, w)
+ clip_embdding = self.img_emb(clip_feature)
+ context = torch.cat([clip_embdding, context], dim=1)
+
+ x, (f, h, w) = self.patchify(x)
+
+ freqs = torch.cat([
+ self.freqs[0][:f].view(f, 1, 1, -1).expand(f, h, w, -1),
+ self.freqs[1][:h].view(1, h, 1, -1).expand(f, h, w, -1),
+ self.freqs[2][:w].view(1, 1, w, -1).expand(f, h, w, -1)
+ ], dim=-1).reshape(f * h * w, 1, -1).to(x.device)
+
+ # build packed views from 1x/2x/4x source views
+ v_src = x_src.shape[0]
+ if v_src == 1:
+ # 1×1x: no 2x/4x sources
+ x_src_2x, x_src_4x = None, None
+ elif v_src == 5:
+ # 1×1x + 4×2x
+ x_src, x_src_2x, x_src_4x = x_src[:1], x_src[1:], None
+ elif v_src == 8:
+ # 1×1x + 3×2x + 4×4x
+ x_src, x_src_2x, x_src_4x = x_src[:1], x_src[1:4], x_src[4:]
+ else:
+ raise ValueError(f"Unsupported number of source views: {v_src}")
+
+ # 1x source: always packed as one extra view
+ x_src, (f_src, h_src, w_src) = self.patchify(x_src)
+ if not self.use_src_attn:
+ # use viewpack for 1x source views
+ x = torch.cat([x, x_src], dim=0)
+ x_src = None
+ freqs_src = None
+ v_pack = 1
+ else:
+ # use src self/cross-attention for 1x source views
+ x_src = repeat(x_src, "1 fhw_src c -> v fhw_src c", v=x.shape[0])
+ freqs_src = torch.cat([
+ self.freqs[0][self.freqs_src_shift:f_src + self.freqs_src_shift].view(f_src, 1, 1, -1).expand(f_src, h_src, w_src, -1),
+ self.freqs[1][:h_src].view(1, h_src, 1, -1).expand(f_src, h_src, w_src, -1),
+ self.freqs[2][:w_src].view(1, 1, w_src, -1).expand(f_src, h_src, w_src, -1)
+ ], dim=-1).reshape(f_src * h_src * w_src, 1, -1).to(x.device)
+ v_pack = 0
+
+ # 2x/4x source views: packed after 1x source views
+ if self.use_viewpack:
+ if x_src_2x is not None:
+ x_src_2x = pad_for_3d_conv(x_src_2x, self.viewpack_embedding.proj_2x.kernel_size)
+ x_src_2x = self.viewpack_embedding.proj_2x(x_src_2x) # (v_2x, dim, f, h//2, w//2)
+
+ if x_src_4x is not None:
+ # tile 4x source views into 2x source views
+ x_src_4x = pad_for_3d_conv(x_src_4x, self.viewpack_embedding.proj_4x.kernel_size)
+ x_src_4x = self.viewpack_embedding.proj_4x(x_src_4x) # (4, dim, f, h//4, w//4)
+ x_src_4x = rearrange(x_src_4x, '(g1 g2) c f h w -> 1 c f (g1 h) (g2 w)', g1=2, g2=2) # (1, dim, f, h//2, w//2)
+ f_2x, h_2x, w_2x = x_src_2x.shape[2:] # crop padding surplus to match 2x source views (f, h, w)
+ x_src_4x = x_src_4x[:, :, :f_2x, :h_2x, :w_2x]
+ x_src_2x = torch.cat([x_src_2x, x_src_4x], dim=0) # (4, dim, f, h//2, w//2)
+
+ # tile 2x source views into 1x source views
+ x_pack = rearrange(x_src_2x, '(g1 g2) c f h w -> 1 c f (g1 h) (g2 w)', g1=2, g2=2)
+ x_pack = x_pack[:, :, :f, :h, :w] # crop padding surplus to match 1x source views (f, h, w)
+ x_pack = rearrange(x_pack, '1 c f h w -> 1 (f h w) c')
+ x_pack = x_pack.to(dtype=x.dtype)
+ if drop_viewpack_tokens:
+ # Keep viewpack parameters in the autograd graph across distributed ranks.
+ zero_dependency = x_pack.float().mean().to(dtype=x.dtype) * 0.0
+ x = x + zero_dependency
+ else:
+ x = torch.cat([x, x_pack], dim=0)
+ v_pack += 1
+
+ timestep = torch.cat([timestep, torch.zeros(v_pack, device=timestep.device, dtype=timestep.dtype)])
+ if skeletons is not None:
+ skeletons = torch.cat([skeletons, -torch.ones_like(skeletons[:1]).expand(v_pack, -1, -1, -1, -1)], dim=0)
+
+ # Expand cam_emb for viewpack views (zero vectors for packed source views)
+ if cam_emb is not None:
+ cam_emb = torch.cat([cam_emb, torch.zeros(v_pack, cam_emb.shape[-1],
+ device=cam_emb.device, dtype=cam_emb.dtype)], dim=0)
+ cam_emb = cam_emb.unsqueeze(1) # (v, 1, 12)
+
+ # Compute time embeddings (after concat since timestep may have been extended)
+ t = self.time_embedding(
+ sinusoidal_embedding_1d(self.freq_dim, timestep).to(x.dtype))
+ t_mod = self.time_projection(t).unflatten(1, (6, self.dim))
+
+ if self.use_pose_encoder:
+ skeleton_latents = self.pose_encoder(skeletons)
+ skeleton_tokens = rearrange(skeleton_latents, "v c f h w -> v (f h w) c")
+ x = x + skeleton_tokens
+
+ v = x.shape[0]
+ freqs_mvs = torch.cat([
+ self.freqs[0][:v].view(v, 1, 1, -1).expand(v, h, w, -1),
+ self.freqs[1][:h].view(1, h, 1, -1).expand(v, h, w, -1),
+ self.freqs[2][:w].view(1, 1, w, -1).expand(v, h, w, -1)
+ ], dim=-1).reshape(v * h * w, 1, -1).to(x.device)
+
+ def create_custom_forward(module):
+ def custom_forward(*inputs):
+ return module(*inputs)
+ return custom_forward
+
+ for block in self.blocks:
+ if self.training and use_gradient_checkpointing:
+ if use_gradient_checkpointing_offload:
+ with torch.autograd.graph.save_on_cpu():
+ x, x_src = torch.utils.checkpoint.checkpoint(
+ create_custom_forward(block),
+ x, x_src, context, t_mod, freqs, freqs_mvs, freqs_src, cam_emb, (v, f, h, w),
+ use_reentrant=False,
+ )
+ else:
+ x, x_src = torch.utils.checkpoint.checkpoint(
+ create_custom_forward(block),
+ x, x_src, context, t_mod, freqs, freqs_mvs, freqs_src, cam_emb, (v, f, h, w),
+ use_reentrant=False,
+ )
+ else:
+ x, x_src = block(x, x_src, context, t_mod, freqs, freqs_mvs, freqs_src, cam_emb, (v, f, h, w))
+
+ # Strip added views (viewpack)
+ if v_pack > 0:
+ x = x[:-v_pack, ...]
+ t = t[:-v_pack, ...]
+
+ x = self.head(x, t)
+ x = self.unpatchify(x, (f, h, w))
+ return x
+
+ @staticmethod
+ def state_dict_converter():
+ return WanModelStateDictConverter()
+
+
+class WanModelStateDictConverter:
+ def __init__(self):
+ pass
+
+ def from_diffusers(self, state_dict):
+ rename_dict = {
+ "blocks.0.attn1.norm_k.weight": "blocks.0.self_attn.norm_k.weight",
+ "blocks.0.attn1.norm_q.weight": "blocks.0.self_attn.norm_q.weight",
+ "blocks.0.attn1.to_k.bias": "blocks.0.self_attn.k.bias",
+ "blocks.0.attn1.to_k.weight": "blocks.0.self_attn.k.weight",
+ "blocks.0.attn1.to_out.0.bias": "blocks.0.self_attn.o.bias",
+ "blocks.0.attn1.to_out.0.weight": "blocks.0.self_attn.o.weight",
+ "blocks.0.attn1.to_q.bias": "blocks.0.self_attn.q.bias",
+ "blocks.0.attn1.to_q.weight": "blocks.0.self_attn.q.weight",
+ "blocks.0.attn1.to_v.bias": "blocks.0.self_attn.v.bias",
+ "blocks.0.attn1.to_v.weight": "blocks.0.self_attn.v.weight",
+ "blocks.0.attn2.norm_k.weight": "blocks.0.cross_attn.norm_k.weight",
+ "blocks.0.attn2.norm_q.weight": "blocks.0.cross_attn.norm_q.weight",
+ "blocks.0.attn2.to_k.bias": "blocks.0.cross_attn.k.bias",
+ "blocks.0.attn2.to_k.weight": "blocks.0.cross_attn.k.weight",
+ "blocks.0.attn2.to_out.0.bias": "blocks.0.cross_attn.o.bias",
+ "blocks.0.attn2.to_out.0.weight": "blocks.0.cross_attn.o.weight",
+ "blocks.0.attn2.to_q.bias": "blocks.0.cross_attn.q.bias",
+ "blocks.0.attn2.to_q.weight": "blocks.0.cross_attn.q.weight",
+ "blocks.0.attn2.to_v.bias": "blocks.0.cross_attn.v.bias",
+ "blocks.0.attn2.to_v.weight": "blocks.0.cross_attn.v.weight",
+ "blocks.0.ffn.net.0.proj.bias": "blocks.0.ffn.0.bias",
+ "blocks.0.ffn.net.0.proj.weight": "blocks.0.ffn.0.weight",
+ "blocks.0.ffn.net.2.bias": "blocks.0.ffn.2.bias",
+ "blocks.0.ffn.net.2.weight": "blocks.0.ffn.2.weight",
+ "blocks.0.norm2.bias": "blocks.0.norm3.bias",
+ "blocks.0.norm2.weight": "blocks.0.norm3.weight",
+ "blocks.0.scale_shift_table": "blocks.0.modulation",
+ "condition_embedder.text_embedder.linear_1.bias": "text_embedding.0.bias",
+ "condition_embedder.text_embedder.linear_1.weight": "text_embedding.0.weight",
+ "condition_embedder.text_embedder.linear_2.bias": "text_embedding.2.bias",
+ "condition_embedder.text_embedder.linear_2.weight": "text_embedding.2.weight",
+ "condition_embedder.time_embedder.linear_1.bias": "time_embedding.0.bias",
+ "condition_embedder.time_embedder.linear_1.weight": "time_embedding.0.weight",
+ "condition_embedder.time_embedder.linear_2.bias": "time_embedding.2.bias",
+ "condition_embedder.time_embedder.linear_2.weight": "time_embedding.2.weight",
+ "condition_embedder.time_proj.bias": "time_projection.1.bias",
+ "condition_embedder.time_proj.weight": "time_projection.1.weight",
+ "patch_embedding.bias": "patch_embedding.bias",
+ "patch_embedding.weight": "patch_embedding.weight",
+ "scale_shift_table": "head.modulation",
+ "proj_out.bias": "head.head.bias",
+ "proj_out.weight": "head.head.weight",
+ }
+ state_dict_ = {}
+ for name, param in state_dict.items():
+ if name in rename_dict:
+ state_dict_[rename_dict[name]] = param
+ else:
+ name_ = ".".join(name.split(".")[:1] + ["0"] + name.split(".")[2:])
+ if name_ in rename_dict:
+ name_ = rename_dict[name_]
+ name_ = ".".join(name_.split(".")[:1] + [name.split(".")[1]] + name_.split(".")[2:])
+ state_dict_[name_] = param
+ if hash_state_dict_keys(state_dict) == "cb104773c6c2cb6df4f9529ad5c60d0b":
+ config = {
+ "model_type": "t2v",
+ "patch_size": (1, 2, 2),
+ "text_len": 512,
+ "in_dim": 16,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "window_size": (-1, -1),
+ "qk_norm": True,
+ "cross_attn_norm": True,
+ "eps": 1e-6,
+ }
+ else:
+ config = {}
+ return state_dict_, config
+
+ def from_civitai(self, state_dict):
+ state_dict = {name: param for name, param in state_dict.items() if not name.startswith("vace")}
+ if hash_state_dict_keys(state_dict) == "9269f8db9040a9d860eaca435be61814":
+ config = {
+ "has_image_input": False,
+ "patch_size": [1, 2, 2],
+ "in_dim": 16,
+ "dim": 1536,
+ "ffn_dim": 8960,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 12,
+ "num_layers": 30,
+ "eps": 1e-6
+ }
+ elif hash_state_dict_keys(state_dict) == "aafcfd9672c3a2456dc46e1cb6e52c70":
+ config = {
+ "has_image_input": False,
+ "patch_size": [1, 2, 2],
+ "in_dim": 16,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "eps": 1e-6
+ }
+ elif hash_state_dict_keys(state_dict) == "6bfcfb3b342cb286ce886889d519a77e":
+ config = {
+ "has_image_input": True,
+ "patch_size": [1, 2, 2],
+ "in_dim": 36,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "eps": 1e-6
+ }
+ elif hash_state_dict_keys(state_dict) == "6d6ccde6845b95ad9114ab993d917893":
+ config = {
+ "has_image_input": True,
+ "patch_size": [1, 2, 2],
+ "in_dim": 36,
+ "dim": 1536,
+ "ffn_dim": 8960,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 12,
+ "num_layers": 30,
+ "eps": 1e-6
+ }
+ elif hash_state_dict_keys(state_dict) == "6bfcfb3b342cb286ce886889d519a77e":
+ config = {
+ "has_image_input": True,
+ "patch_size": [1, 2, 2],
+ "in_dim": 36,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "eps": 1e-6
+ }
+ elif hash_state_dict_keys(state_dict) == "349723183fc063b2bfc10bb2835cf677":
+ # 1.3B PAI control
+ config = {
+ "has_image_input": True,
+ "patch_size": [1, 2, 2],
+ "in_dim": 48,
+ "dim": 1536,
+ "ffn_dim": 8960,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 12,
+ "num_layers": 30,
+ "eps": 1e-6
+ }
+ elif hash_state_dict_keys(state_dict) == "efa44cddf936c70abd0ea28b6cbe946c":
+ # 14B PAI control
+ config = {
+ "has_image_input": True,
+ "patch_size": [1, 2, 2],
+ "in_dim": 48,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "eps": 1e-6
+ }
+ elif hash_state_dict_keys(state_dict) == "3ef3b1f8e1dab83d5b71fd7b617f859f":
+ config = {
+ "has_image_input": True,
+ "patch_size": [1, 2, 2],
+ "in_dim": 36,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "eps": 1e-6,
+ "has_image_pos_emb": True
+ }
+ elif hash_state_dict_keys(state_dict) == "70ddad9d3a133785da5ea371aae09504":
+ # 1.3B PAI control v1.1
+ config = {
+ "has_image_input": True,
+ "patch_size": [1, 2, 2],
+ "in_dim": 48,
+ "dim": 1536,
+ "ffn_dim": 8960,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 12,
+ "num_layers": 30,
+ "eps": 1e-6,
+ "has_ref_conv": True
+ }
+ elif hash_state_dict_keys(state_dict) == "26bde73488a92e64cc20b0a7485b9e5b":
+ # 14B PAI control v1.1
+ config = {
+ "has_image_input": True,
+ "patch_size": [1, 2, 2],
+ "in_dim": 48,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "eps": 1e-6,
+ "has_ref_conv": True
+ }
+ elif hash_state_dict_keys(state_dict) == "ac6a5aa74f4a0aab6f64eb9a72f19901":
+ # 1.3B PAI control-camera v1.1
+ config = {
+ "has_image_input": True,
+ "patch_size": [1, 2, 2],
+ "in_dim": 32,
+ "dim": 1536,
+ "ffn_dim": 8960,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 12,
+ "num_layers": 30,
+ "eps": 1e-6,
+ "has_ref_conv": False,
+ "add_control_adapter": True,
+ "in_dim_control_adapter": 24,
+ }
+ elif hash_state_dict_keys(state_dict) == "b61c605c2adbd23124d152ed28e049ae":
+ # 14B PAI control-camera v1.1
+ config = {
+ "has_image_input": True,
+ "patch_size": [1, 2, 2],
+ "in_dim": 32,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "eps": 1e-6,
+ "has_ref_conv": False,
+ "add_control_adapter": True,
+ "in_dim_control_adapter": 24,
+ }
+ elif hash_state_dict_keys(state_dict) == "1f5ab7703c6fc803fdded85ff040c316":
+ # Wan-AI/Wan2.2-TI2V-5B
+ config = {
+ "has_image_input": False,
+ "patch_size": [1, 2, 2],
+ "in_dim": 48,
+ "dim": 3072,
+ "ffn_dim": 14336,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 48,
+ "num_heads": 24,
+ "num_layers": 30,
+ "eps": 1e-6,
+ "seperated_timestep": True,
+ "require_clip_embedding": False,
+ "require_vae_embedding": False,
+ "fuse_vae_embedding_in_latents": True,
+ }
+ elif hash_state_dict_keys(state_dict) == "5b013604280dd715f8457c6ed6d6a626":
+ # Wan-AI/Wan2.2-I2V-A14B
+ config = {
+ "has_image_input": False,
+ "patch_size": [1, 2, 2],
+ "in_dim": 36,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "eps": 1e-6,
+ "require_clip_embedding": False,
+ }
+ elif hash_state_dict_keys(state_dict) == "2267d489f0ceb9f21836532952852ee5":
+ # Wan2.2-Fun-A14B-Control
+ config = {
+ "has_image_input": False,
+ "patch_size": [1, 2, 2],
+ "in_dim": 52,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "eps": 1e-6,
+ "has_ref_conv": True,
+ "require_clip_embedding": False,
+ }
+ elif hash_state_dict_keys(state_dict) == "47dbeab5e560db3180adf51dc0232fb1":
+ # Wan2.2-Fun-A14B-Control-Camera
+ config = {
+ "has_image_input": False,
+ "patch_size": [1, 2, 2],
+ "in_dim": 36,
+ "dim": 5120,
+ "ffn_dim": 13824,
+ "freq_dim": 256,
+ "text_dim": 4096,
+ "out_dim": 16,
+ "num_heads": 40,
+ "num_layers": 40,
+ "eps": 1e-6,
+ "has_ref_conv": False,
+ "add_control_adapter": True,
+ "in_dim_control_adapter": 24,
+ "require_clip_embedding": False,
+ }
+ else:
+ config = {}
+ return state_dict, config
diff --git a/fdanyone/vendor/diffsynth/models/wan_video_pose_encoder.py b/fdanyone/vendor/diffsynth/models/wan_video_pose_encoder.py
new file mode 100644
index 0000000..5fcfcb0
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/models/wan_video_pose_encoder.py
@@ -0,0 +1,82 @@
+import torch
+import torch.nn as nn
+import numpy as np
+from torch.nn import init
+
+# PoseEncoder is a 3D version of PoseNet in MimicMotion. Copyright Tencent;
+# the source is licensed under Apache-2.0:
+# https://github.com/Tencent/MimicMotion/blob/c053153a1d124abae8c08568925ae88debc63001/mimicmotion/modules/pose_net.py
+
+
+class PoseEncoder(nn.Module):
+ def __init__(self, out_dim=5120, in_channels=3):
+ super().__init__()
+
+ if out_dim in (5120, 1536):
+ # Wan2.1-T2V-14B / 1.3B
+ t_strides = (1, 1, 1, 2, 2) # downsampled by 4
+ s_strides = (2, 2, 1, 2, 2) # downsampled by 16
+ kernel_size = (3, 3, 3)
+ elif out_dim == 3072:
+ # Wan2.2-TI2V-5B
+ t_strides = (1, 1, 1, 2, 2) # downsampled by 4
+ s_strides = (2, 2, 2, 2, 2) # downsampled by 32
+ kernel_size = (3, 4, 4)
+ else:
+ raise ValueError(f"Invalid out_dim: {out_dim}")
+
+ strides = [(t, s, s) for t, s in zip(t_strides, s_strides)]
+
+ self.conv_layers = nn.Sequential(
+ nn.Conv3d(in_channels, in_channels, kernel_size=3, stride=1, padding=1),
+ nn.SiLU(),
+ nn.Conv3d(in_channels, 16, kernel_size=kernel_size, stride=strides[0], padding=(1, 1, 1)),
+ nn.SiLU(),
+ nn.Conv3d(16, 16, kernel_size=3, stride=1, padding=1),
+ nn.SiLU(),
+ nn.Conv3d(16, 32, kernel_size=kernel_size, stride=strides[1], padding=(1, 1, 1)),
+ nn.SiLU(),
+ nn.Conv3d(32, 32, kernel_size=3, stride=1, padding=1),
+ nn.SiLU(),
+ nn.Conv3d(32, 64, kernel_size=kernel_size, stride=strides[2], padding=(1, 1, 1)),
+ nn.SiLU(),
+ nn.Conv3d(64, 64, kernel_size=3, stride=1, padding=1),
+ nn.SiLU(),
+ nn.Conv3d(64, 128, kernel_size=kernel_size, stride=strides[3], padding=(1, 1, 1)),
+ nn.SiLU(),
+ nn.Conv3d(128, 128, kernel_size=3, stride=1, padding=1),
+ nn.SiLU(),
+ nn.Conv3d(128, 256, kernel_size=kernel_size, stride=strides[4], padding=(1, 1, 1)),
+ nn.SiLU(),
+ )
+
+ self.final_proj = nn.Conv3d(256, out_dim, kernel_size=1)
+
+ self.scale = nn.Parameter(torch.ones(1) * 2.0)
+
+ self._initialize_weights()
+
+ def _initialize_weights(self):
+ for m in self.modules():
+ if isinstance(m, nn.Conv3d):
+ # He (Kaiming) initialization in fan‑in mode
+ receptive = np.prod(m.kernel_size) * m.in_channels
+ init.normal_(m.weight, mean=0.0, std=np.sqrt(2.0 / receptive))
+ if m.bias is not None:
+ init.zeros_(m.bias)
+ # start with zero output so model behaves like unconditional
+ init.zeros_(self.final_proj.weight)
+ if self.final_proj.bias is not None:
+ init.zeros_(self.final_proj.bias)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ """
+ x: (B, C, F, H, W) -> latent grid matching the DiT patch tokens.
+ Wan2.1 uses F/4, H/16, W/16; Wan2.2-TI2V-5B uses F/4, H/32, W/32.
+ """
+ # Wan pattern: 1 -> 4 -> 4 -> ...
+ x = torch.cat([x[:, :, :1].repeat(1, 1, 3, 1, 1), x], dim=2)
+
+ x = self.conv_layers(x)
+ x = self.final_proj(x)
+ return x * self.scale
diff --git a/fdanyone/vendor/diffsynth/models/wan_video_text_encoder.py b/fdanyone/vendor/diffsynth/models/wan_video_text_encoder.py
new file mode 100644
index 0000000..c288737
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/models/wan_video_text_encoder.py
@@ -0,0 +1,269 @@
+import math
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+def fp16_clamp(x):
+ if x.dtype == torch.float16 and torch.isinf(x).any():
+ clamp = torch.finfo(x.dtype).max - 1000
+ x = torch.clamp(x, min=-clamp, max=clamp)
+ return x
+
+
+class GELU(nn.Module):
+
+ def forward(self, x):
+ return 0.5 * x * (1.0 + torch.tanh(
+ math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))
+
+
+class T5LayerNorm(nn.Module):
+
+ def __init__(self, dim, eps=1e-6):
+ super(T5LayerNorm, self).__init__()
+ self.dim = dim
+ self.eps = eps
+ self.weight = nn.Parameter(torch.ones(dim))
+
+ def forward(self, x):
+ x = x * torch.rsqrt(x.float().pow(2).mean(dim=-1, keepdim=True) +
+ self.eps)
+ if self.weight.dtype in [torch.float16, torch.bfloat16]:
+ x = x.type_as(self.weight)
+ return self.weight * x
+
+
+class T5Attention(nn.Module):
+
+ def __init__(self, dim, dim_attn, num_heads, dropout=0.1):
+ assert dim_attn % num_heads == 0
+ super(T5Attention, self).__init__()
+ self.dim = dim
+ self.dim_attn = dim_attn
+ self.num_heads = num_heads
+ self.head_dim = dim_attn // num_heads
+
+ # layers
+ self.q = nn.Linear(dim, dim_attn, bias=False)
+ self.k = nn.Linear(dim, dim_attn, bias=False)
+ self.v = nn.Linear(dim, dim_attn, bias=False)
+ self.o = nn.Linear(dim_attn, dim, bias=False)
+ self.dropout = nn.Dropout(dropout)
+
+ def forward(self, x, context=None, mask=None, pos_bias=None):
+ """
+ x: [B, L1, C].
+ context: [B, L2, C] or None.
+ mask: [B, L2] or [B, L1, L2] or None.
+ """
+ # check inputs
+ context = x if context is None else context
+ b, n, c = x.size(0), self.num_heads, self.head_dim
+
+ # compute query, key, value
+ q = self.q(x).view(b, -1, n, c)
+ k = self.k(context).view(b, -1, n, c)
+ v = self.v(context).view(b, -1, n, c)
+
+ # attention bias
+ attn_bias = x.new_zeros(b, n, q.size(1), k.size(1))
+ if pos_bias is not None:
+ attn_bias += pos_bias
+ if mask is not None:
+ assert mask.ndim in [2, 3]
+ mask = mask.view(b, 1, 1,
+ -1) if mask.ndim == 2 else mask.unsqueeze(1)
+ attn_bias.masked_fill_(mask == 0, torch.finfo(x.dtype).min)
+
+ # compute attention (T5 does not use scaling)
+ attn = torch.einsum('binc,bjnc->bnij', q, k) + attn_bias
+ attn = F.softmax(attn.float(), dim=-1).type_as(attn)
+ x = torch.einsum('bnij,bjnc->binc', attn, v)
+
+ # output
+ x = x.reshape(b, -1, n * c)
+ x = self.o(x)
+ x = self.dropout(x)
+ return x
+
+
+class T5FeedForward(nn.Module):
+
+ def __init__(self, dim, dim_ffn, dropout=0.1):
+ super(T5FeedForward, self).__init__()
+ self.dim = dim
+ self.dim_ffn = dim_ffn
+
+ # layers
+ self.gate = nn.Sequential(nn.Linear(dim, dim_ffn, bias=False), GELU())
+ self.fc1 = nn.Linear(dim, dim_ffn, bias=False)
+ self.fc2 = nn.Linear(dim_ffn, dim, bias=False)
+ self.dropout = nn.Dropout(dropout)
+
+ def forward(self, x):
+ x = self.fc1(x) * self.gate(x)
+ x = self.dropout(x)
+ x = self.fc2(x)
+ x = self.dropout(x)
+ return x
+
+
+class T5SelfAttention(nn.Module):
+
+ def __init__(self,
+ dim,
+ dim_attn,
+ dim_ffn,
+ num_heads,
+ num_buckets,
+ shared_pos=True,
+ dropout=0.1):
+ super(T5SelfAttention, self).__init__()
+ self.dim = dim
+ self.dim_attn = dim_attn
+ self.dim_ffn = dim_ffn
+ self.num_heads = num_heads
+ self.num_buckets = num_buckets
+ self.shared_pos = shared_pos
+
+ # layers
+ self.norm1 = T5LayerNorm(dim)
+ self.attn = T5Attention(dim, dim_attn, num_heads, dropout)
+ self.norm2 = T5LayerNorm(dim)
+ self.ffn = T5FeedForward(dim, dim_ffn, dropout)
+ self.pos_embedding = None if shared_pos else T5RelativeEmbedding(
+ num_buckets, num_heads, bidirectional=True)
+
+ def forward(self, x, mask=None, pos_bias=None):
+ e = pos_bias if self.shared_pos else self.pos_embedding(
+ x.size(1), x.size(1))
+ x = fp16_clamp(x + self.attn(self.norm1(x), mask=mask, pos_bias=e))
+ x = fp16_clamp(x + self.ffn(self.norm2(x)))
+ return x
+
+
+class T5RelativeEmbedding(nn.Module):
+
+ def __init__(self, num_buckets, num_heads, bidirectional, max_dist=128):
+ super(T5RelativeEmbedding, self).__init__()
+ self.num_buckets = num_buckets
+ self.num_heads = num_heads
+ self.bidirectional = bidirectional
+ self.max_dist = max_dist
+
+ # layers
+ self.embedding = nn.Embedding(num_buckets, num_heads)
+
+ def forward(self, lq, lk):
+ device = self.embedding.weight.device
+ # rel_pos = torch.arange(lk).unsqueeze(0).to(device) - \
+ # torch.arange(lq).unsqueeze(1).to(device)
+ rel_pos = torch.arange(lk, device=device).unsqueeze(0) - \
+ torch.arange(lq, device=device).unsqueeze(1)
+ rel_pos = self._relative_position_bucket(rel_pos)
+ rel_pos_embeds = self.embedding(rel_pos)
+ rel_pos_embeds = rel_pos_embeds.permute(2, 0, 1).unsqueeze(
+ 0) # [1, N, Lq, Lk]
+ return rel_pos_embeds.contiguous()
+
+ def _relative_position_bucket(self, rel_pos):
+ # preprocess
+ if self.bidirectional:
+ num_buckets = self.num_buckets // 2
+ rel_buckets = (rel_pos > 0).long() * num_buckets
+ rel_pos = torch.abs(rel_pos)
+ else:
+ num_buckets = self.num_buckets
+ rel_buckets = 0
+ rel_pos = -torch.min(rel_pos, torch.zeros_like(rel_pos))
+
+ # embeddings for small and large positions
+ max_exact = num_buckets // 2
+ rel_pos_large = max_exact + (torch.log(rel_pos.float() / max_exact) /
+ math.log(self.max_dist / max_exact) *
+ (num_buckets - max_exact)).long()
+ rel_pos_large = torch.min(
+ rel_pos_large, torch.full_like(rel_pos_large, num_buckets - 1))
+ rel_buckets += torch.where(rel_pos < max_exact, rel_pos, rel_pos_large)
+ return rel_buckets
+
+def init_weights(m):
+ if isinstance(m, T5LayerNorm):
+ nn.init.ones_(m.weight)
+ elif isinstance(m, T5FeedForward):
+ nn.init.normal_(m.gate[0].weight, std=m.dim**-0.5)
+ nn.init.normal_(m.fc1.weight, std=m.dim**-0.5)
+ nn.init.normal_(m.fc2.weight, std=m.dim_ffn**-0.5)
+ elif isinstance(m, T5Attention):
+ nn.init.normal_(m.q.weight, std=(m.dim * m.dim_attn)**-0.5)
+ nn.init.normal_(m.k.weight, std=m.dim**-0.5)
+ nn.init.normal_(m.v.weight, std=m.dim**-0.5)
+ nn.init.normal_(m.o.weight, std=(m.num_heads * m.dim_attn)**-0.5)
+ elif isinstance(m, T5RelativeEmbedding):
+ nn.init.normal_(
+ m.embedding.weight, std=(2 * m.num_buckets * m.num_heads)**-0.5)
+
+
+class WanTextEncoder(torch.nn.Module):
+
+ def __init__(self,
+ vocab=256384,
+ dim=4096,
+ dim_attn=4096,
+ dim_ffn=10240,
+ num_heads=64,
+ num_layers=24,
+ num_buckets=32,
+ shared_pos=False,
+ dropout=0.1):
+ super(WanTextEncoder, self).__init__()
+ self.dim = dim
+ self.dim_attn = dim_attn
+ self.dim_ffn = dim_ffn
+ self.num_heads = num_heads
+ self.num_layers = num_layers
+ self.num_buckets = num_buckets
+ self.shared_pos = shared_pos
+
+ # layers
+ self.token_embedding = vocab if isinstance(vocab, nn.Embedding) \
+ else nn.Embedding(vocab, dim)
+ self.pos_embedding = T5RelativeEmbedding(
+ num_buckets, num_heads, bidirectional=True) if shared_pos else None
+ self.dropout = nn.Dropout(dropout)
+ self.blocks = nn.ModuleList([
+ T5SelfAttention(dim, dim_attn, dim_ffn, num_heads, num_buckets,
+ shared_pos, dropout) for _ in range(num_layers)
+ ])
+ self.norm = T5LayerNorm(dim)
+
+ # initialize weights
+ self.apply(init_weights)
+
+ def forward(self, ids, mask=None):
+ x = self.token_embedding(ids)
+ x = self.dropout(x)
+ e = self.pos_embedding(x.size(1),
+ x.size(1)) if self.shared_pos else None
+ for block in self.blocks:
+ x = block(x, mask, pos_bias=e)
+ x = self.norm(x)
+ x = self.dropout(x)
+ return x
+
+ @staticmethod
+ def state_dict_converter():
+ return WanTextEncoderStateDictConverter()
+
+
+class WanTextEncoderStateDictConverter:
+ def __init__(self):
+ pass
+
+ def from_diffusers(self, state_dict):
+ return state_dict
+
+ def from_civitai(self, state_dict):
+ return state_dict
diff --git a/fdanyone/vendor/diffsynth/models/wan_video_vae.py b/fdanyone/vendor/diffsynth/models/wan_video_vae.py
new file mode 100644
index 0000000..43057ff
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/models/wan_video_vae.py
@@ -0,0 +1,1382 @@
+from einops import rearrange, repeat
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from tqdm import tqdm
+
+CACHE_T = 2
+
+
+def check_is_instance(model, module_class):
+ if isinstance(model, module_class):
+ return True
+ if hasattr(model, "module") and isinstance(model.module, module_class):
+ return True
+ return False
+
+
+def block_causal_mask(x, block_size):
+ # params
+ b, n, s, _, device = *x.size(), x.device
+ assert s % block_size == 0
+ num_blocks = s // block_size
+
+ # build mask
+ mask = torch.zeros(b, n, s, s, dtype=torch.bool, device=device)
+ for i in range(num_blocks):
+ mask[:, :,
+ i * block_size:(i + 1) * block_size, :(i + 1) * block_size] = 1
+ return mask
+
+
+class CausalConv3d(nn.Conv3d):
+ """
+ Causal 3d convolusion.
+ """
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self._padding = (self.padding[2], self.padding[2], self.padding[1],
+ self.padding[1], 2 * self.padding[0], 0)
+ self.padding = (0, 0, 0)
+
+ def forward(self, x, cache_x=None):
+ padding = list(self._padding)
+ if cache_x is not None and self._padding[4] > 0:
+ cache_x = cache_x.to(x.device)
+ x = torch.cat([cache_x, x], dim=2)
+ padding[4] -= cache_x.shape[2]
+ x = F.pad(x, padding)
+
+ return super().forward(x)
+
+
+class RMS_norm(nn.Module):
+
+ def __init__(self, dim, channel_first=True, images=True, bias=False):
+ super().__init__()
+ broadcastable_dims = (1, 1, 1) if not images else (1, 1)
+ shape = (dim, *broadcastable_dims) if channel_first else (dim,)
+
+ self.channel_first = channel_first
+ self.scale = dim**0.5
+ self.gamma = nn.Parameter(torch.ones(shape))
+ self.bias = nn.Parameter(torch.zeros(shape)) if bias else 0.
+
+ def forward(self, x):
+ return F.normalize(
+ x, dim=(1 if self.channel_first else
+ -1)) * self.scale * self.gamma + self.bias
+
+
+class Upsample(nn.Upsample):
+
+ def forward(self, x):
+ """
+ Fix bfloat16 support for nearest neighbor interpolation.
+ """
+ return super().forward(x.float()).type_as(x)
+
+
+class Resample(nn.Module):
+
+ def __init__(self, dim, mode):
+ assert mode in ('none', 'upsample2d', 'upsample3d', 'downsample2d',
+ 'downsample3d')
+ super().__init__()
+ self.dim = dim
+ self.mode = mode
+
+ # layers
+ if mode == 'upsample2d':
+ self.resample = nn.Sequential(
+ Upsample(scale_factor=(2., 2.), mode='nearest-exact'),
+ nn.Conv2d(dim, dim // 2, 3, padding=1))
+ elif mode == 'upsample3d':
+ self.resample = nn.Sequential(
+ Upsample(scale_factor=(2., 2.), mode='nearest-exact'),
+ nn.Conv2d(dim, dim // 2, 3, padding=1))
+ self.time_conv = CausalConv3d(dim,
+ dim * 2, (3, 1, 1),
+ padding=(1, 0, 0))
+
+ elif mode == 'downsample2d':
+ self.resample = nn.Sequential(
+ nn.ZeroPad2d((0, 1, 0, 1)),
+ nn.Conv2d(dim, dim, 3, stride=(2, 2)))
+ elif mode == 'downsample3d':
+ self.resample = nn.Sequential(
+ nn.ZeroPad2d((0, 1, 0, 1)),
+ nn.Conv2d(dim, dim, 3, stride=(2, 2)))
+ self.time_conv = CausalConv3d(dim,
+ dim, (3, 1, 1),
+ stride=(2, 1, 1),
+ padding=(0, 0, 0))
+
+ else:
+ self.resample = nn.Identity()
+
+ def forward(self, x, feat_cache=None, feat_idx=[0]):
+ b, c, t, h, w = x.size()
+ if self.mode == 'upsample3d':
+ if feat_cache is not None:
+ idx = feat_idx[0]
+ if feat_cache[idx] is None:
+ feat_cache[idx] = 'Rep'
+ feat_idx[0] += 1
+ else:
+
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[
+ idx] is not None and feat_cache[idx] != 'Rep':
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ if cache_x.shape[2] < 2 and feat_cache[
+ idx] is not None and feat_cache[idx] == 'Rep':
+ cache_x = torch.cat([
+ torch.zeros_like(cache_x).to(cache_x.device),
+ cache_x
+ ],
+ dim=2)
+ if feat_cache[idx] == 'Rep':
+ x = self.time_conv(x)
+ else:
+ x = self.time_conv(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+
+ x = x.reshape(b, 2, c, t, h, w)
+ x = torch.stack((x[:, 0, :, :, :, :], x[:, 1, :, :, :, :]),
+ 3)
+ x = x.reshape(b, c, t * 2, h, w)
+ t = x.shape[2]
+ x = rearrange(x, 'b c t h w -> (b t) c h w')
+ x = self.resample(x)
+ x = rearrange(x, '(b t) c h w -> b c t h w', t=t)
+
+ if self.mode == 'downsample3d':
+ if feat_cache is not None:
+ idx = feat_idx[0]
+ if feat_cache[idx] is None:
+ feat_cache[idx] = x.clone()
+ feat_idx[0] += 1
+ else:
+ cache_x = x[:, :, -1:, :, :].clone()
+ x = self.time_conv(
+ torch.cat([feat_cache[idx][:, :, -1:, :, :], x], 2))
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ return x
+
+ def init_weight(self, conv):
+ conv_weight = conv.weight
+ nn.init.zeros_(conv_weight)
+ c1, c2, t, h, w = conv_weight.size()
+ one_matrix = torch.eye(c1, c2)
+ init_matrix = one_matrix
+ nn.init.zeros_(conv_weight)
+ conv_weight.data[:, :, 1, 0, 0] = init_matrix
+ conv.weight.data.copy_(conv_weight)
+ nn.init.zeros_(conv.bias.data)
+
+ def init_weight2(self, conv):
+ conv_weight = conv.weight.data
+ nn.init.zeros_(conv_weight)
+ c1, c2, t, h, w = conv_weight.size()
+ init_matrix = torch.eye(c1 // 2, c2)
+ conv_weight[:c1 // 2, :, -1, 0, 0] = init_matrix
+ conv_weight[c1 // 2:, :, -1, 0, 0] = init_matrix
+ conv.weight.data.copy_(conv_weight)
+ nn.init.zeros_(conv.bias.data)
+
+
+
+def patchify(x, patch_size):
+ if patch_size == 1:
+ return x
+ if x.dim() == 4:
+ x = rearrange(x, "b c (h q) (w r) -> b (c r q) h w", q=patch_size, r=patch_size)
+ elif x.dim() == 5:
+ x = rearrange(x,
+ "b c f (h q) (w r) -> b (c r q) f h w",
+ q=patch_size,
+ r=patch_size)
+ else:
+ raise ValueError(f"Invalid input shape: {x.shape}")
+ return x
+
+
+def unpatchify(x, patch_size):
+ if patch_size == 1:
+ return x
+ if x.dim() == 4:
+ x = rearrange(x, "b (c r q) h w -> b c (h q) (w r)", q=patch_size, r=patch_size)
+ elif x.dim() == 5:
+ x = rearrange(x,
+ "b (c r q) f h w -> b c f (h q) (w r)",
+ q=patch_size,
+ r=patch_size)
+ return x
+
+
+class Resample38(Resample):
+
+ def __init__(self, dim, mode):
+ assert mode in (
+ "none",
+ "upsample2d",
+ "upsample3d",
+ "downsample2d",
+ "downsample3d",
+ )
+ super(Resample, self).__init__()
+ self.dim = dim
+ self.mode = mode
+
+ # layers
+ if mode == "upsample2d":
+ self.resample = nn.Sequential(
+ Upsample(scale_factor=(2.0, 2.0), mode="nearest-exact"),
+ nn.Conv2d(dim, dim, 3, padding=1),
+ )
+ elif mode == "upsample3d":
+ self.resample = nn.Sequential(
+ Upsample(scale_factor=(2.0, 2.0), mode="nearest-exact"),
+ nn.Conv2d(dim, dim, 3, padding=1),
+ )
+ self.time_conv = CausalConv3d(dim, dim * 2, (3, 1, 1), padding=(1, 0, 0))
+ elif mode == "downsample2d":
+ self.resample = nn.Sequential(
+ nn.ZeroPad2d((0, 1, 0, 1)), nn.Conv2d(dim, dim, 3, stride=(2, 2))
+ )
+ elif mode == "downsample3d":
+ self.resample = nn.Sequential(
+ nn.ZeroPad2d((0, 1, 0, 1)), nn.Conv2d(dim, dim, 3, stride=(2, 2))
+ )
+ self.time_conv = CausalConv3d(
+ dim, dim, (3, 1, 1), stride=(2, 1, 1), padding=(0, 0, 0)
+ )
+ else:
+ self.resample = nn.Identity()
+
+class ResidualBlock(nn.Module):
+
+ def __init__(self, in_dim, out_dim, dropout=0.0):
+ super().__init__()
+ self.in_dim = in_dim
+ self.out_dim = out_dim
+
+ # layers
+ self.residual = nn.Sequential(
+ RMS_norm(in_dim, images=False), nn.SiLU(),
+ CausalConv3d(in_dim, out_dim, 3, padding=1),
+ RMS_norm(out_dim, images=False), nn.SiLU(), nn.Dropout(dropout),
+ CausalConv3d(out_dim, out_dim, 3, padding=1))
+ self.shortcut = CausalConv3d(in_dim, out_dim, 1) \
+ if in_dim != out_dim else nn.Identity()
+
+ def forward(self, x, feat_cache=None, feat_idx=[0]):
+ h = self.shortcut(x)
+ for layer in self.residual:
+ if check_is_instance(layer, CausalConv3d) and feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ x = layer(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = layer(x)
+ return x + h
+
+
+class AttentionBlock(nn.Module):
+ """
+ Causal self-attention with a single head.
+ """
+
+ def __init__(self, dim):
+ super().__init__()
+ self.dim = dim
+
+ # layers
+ self.norm = RMS_norm(dim)
+ self.to_qkv = nn.Conv2d(dim, dim * 3, 1)
+ self.proj = nn.Conv2d(dim, dim, 1)
+
+ # zero out the last layer params
+ nn.init.zeros_(self.proj.weight)
+
+ def forward(self, x):
+ identity = x
+ b, c, t, h, w = x.size()
+ x = rearrange(x, 'b c t h w -> (b t) c h w')
+ x = self.norm(x)
+ # compute query, key, value
+ q, k, v = self.to_qkv(x).reshape(b * t, 1, c * 3, -1).permute(
+ 0, 1, 3, 2).contiguous().chunk(3, dim=-1)
+
+ # apply attention
+ x = F.scaled_dot_product_attention(
+ q,
+ k,
+ v,
+ #attn_mask=block_causal_mask(q, block_size=h * w)
+ )
+ x = x.squeeze(1).permute(0, 2, 1).reshape(b * t, c, h, w)
+
+ # output
+ x = self.proj(x)
+ x = rearrange(x, '(b t) c h w-> b c t h w', t=t)
+ return x + identity
+
+
+class AvgDown3D(nn.Module):
+ def __init__(
+ self,
+ in_channels,
+ out_channels,
+ factor_t,
+ factor_s=1,
+ ):
+ super().__init__()
+ self.in_channels = in_channels
+ self.out_channels = out_channels
+ self.factor_t = factor_t
+ self.factor_s = factor_s
+ self.factor = self.factor_t * self.factor_s * self.factor_s
+
+ assert in_channels * self.factor % out_channels == 0
+ self.group_size = in_channels * self.factor // out_channels
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ pad_t = (self.factor_t - x.shape[2] % self.factor_t) % self.factor_t
+ pad = (0, 0, 0, 0, pad_t, 0)
+ x = F.pad(x, pad)
+ B, C, T, H, W = x.shape
+ x = x.view(
+ B,
+ C,
+ T // self.factor_t,
+ self.factor_t,
+ H // self.factor_s,
+ self.factor_s,
+ W // self.factor_s,
+ self.factor_s,
+ )
+ x = x.permute(0, 1, 3, 5, 7, 2, 4, 6).contiguous()
+ x = x.view(
+ B,
+ C * self.factor,
+ T // self.factor_t,
+ H // self.factor_s,
+ W // self.factor_s,
+ )
+ x = x.view(
+ B,
+ self.out_channels,
+ self.group_size,
+ T // self.factor_t,
+ H // self.factor_s,
+ W // self.factor_s,
+ )
+ x = x.mean(dim=2)
+ return x
+
+
+class DupUp3D(nn.Module):
+ def __init__(
+ self,
+ in_channels: int,
+ out_channels: int,
+ factor_t,
+ factor_s=1,
+ ):
+ super().__init__()
+ self.in_channels = in_channels
+ self.out_channels = out_channels
+
+ self.factor_t = factor_t
+ self.factor_s = factor_s
+ self.factor = self.factor_t * self.factor_s * self.factor_s
+
+ assert out_channels * self.factor % in_channels == 0
+ self.repeats = out_channels * self.factor // in_channels
+
+ def forward(self, x: torch.Tensor, first_chunk=False) -> torch.Tensor:
+ x = x.repeat_interleave(self.repeats, dim=1)
+ x = x.view(
+ x.size(0),
+ self.out_channels,
+ self.factor_t,
+ self.factor_s,
+ self.factor_s,
+ x.size(2),
+ x.size(3),
+ x.size(4),
+ )
+ x = x.permute(0, 1, 5, 2, 6, 3, 7, 4).contiguous()
+ x = x.view(
+ x.size(0),
+ self.out_channels,
+ x.size(2) * self.factor_t,
+ x.size(4) * self.factor_s,
+ x.size(6) * self.factor_s,
+ )
+ if first_chunk:
+ x = x[:, :, self.factor_t - 1 :, :, :]
+ return x
+
+
+class Down_ResidualBlock(nn.Module):
+ def __init__(
+ self, in_dim, out_dim, dropout, mult, temperal_downsample=False, down_flag=False
+ ):
+ super().__init__()
+
+ # Shortcut path with downsample
+ self.avg_shortcut = AvgDown3D(
+ in_dim,
+ out_dim,
+ factor_t=2 if temperal_downsample else 1,
+ factor_s=2 if down_flag else 1,
+ )
+
+ # Main path with residual blocks and downsample
+ downsamples = []
+ for _ in range(mult):
+ downsamples.append(ResidualBlock(in_dim, out_dim, dropout))
+ in_dim = out_dim
+
+ # Add the final downsample block
+ if down_flag:
+ mode = "downsample3d" if temperal_downsample else "downsample2d"
+ downsamples.append(Resample38(out_dim, mode=mode))
+
+ self.downsamples = nn.Sequential(*downsamples)
+
+ def forward(self, x, feat_cache=None, feat_idx=[0]):
+ x_copy = x.clone()
+ for module in self.downsamples:
+ x = module(x, feat_cache, feat_idx)
+
+ return x + self.avg_shortcut(x_copy)
+
+
+class Up_ResidualBlock(nn.Module):
+ def __init__(
+ self, in_dim, out_dim, dropout, mult, temperal_upsample=False, up_flag=False
+ ):
+ super().__init__()
+ # Shortcut path with upsample
+ if up_flag:
+ self.avg_shortcut = DupUp3D(
+ in_dim,
+ out_dim,
+ factor_t=2 if temperal_upsample else 1,
+ factor_s=2 if up_flag else 1,
+ )
+ else:
+ self.avg_shortcut = None
+
+ # Main path with residual blocks and upsample
+ upsamples = []
+ for _ in range(mult):
+ upsamples.append(ResidualBlock(in_dim, out_dim, dropout))
+ in_dim = out_dim
+
+ # Add the final upsample block
+ if up_flag:
+ mode = "upsample3d" if temperal_upsample else "upsample2d"
+ upsamples.append(Resample38(out_dim, mode=mode))
+
+ self.upsamples = nn.Sequential(*upsamples)
+
+ def forward(self, x, feat_cache=None, feat_idx=[0], first_chunk=False):
+ x_main = x.clone()
+ for module in self.upsamples:
+ x_main = module(x_main, feat_cache, feat_idx)
+ if self.avg_shortcut is not None:
+ x_shortcut = self.avg_shortcut(x, first_chunk)
+ return x_main + x_shortcut
+ else:
+ return x_main
+
+
+class Encoder3d(nn.Module):
+
+ def __init__(self,
+ dim=128,
+ z_dim=4,
+ dim_mult=[1, 2, 4, 4],
+ num_res_blocks=2,
+ attn_scales=[],
+ temperal_downsample=[True, True, False],
+ dropout=0.0):
+ super().__init__()
+ self.dim = dim
+ self.z_dim = z_dim
+ self.dim_mult = dim_mult
+ self.num_res_blocks = num_res_blocks
+ self.attn_scales = attn_scales
+ self.temperal_downsample = temperal_downsample
+
+ # dimensions
+ dims = [dim * u for u in [1] + dim_mult]
+ scale = 1.0
+
+ # init block
+ self.conv1 = CausalConv3d(3, dims[0], 3, padding=1)
+
+ # downsample blocks
+ downsamples = []
+ for i, (in_dim, out_dim) in enumerate(zip(dims[:-1], dims[1:])):
+ # residual (+attention) blocks
+ for _ in range(num_res_blocks):
+ downsamples.append(ResidualBlock(in_dim, out_dim, dropout))
+ if scale in attn_scales:
+ downsamples.append(AttentionBlock(out_dim))
+ in_dim = out_dim
+
+ # downsample block
+ if i != len(dim_mult) - 1:
+ mode = 'downsample3d' if temperal_downsample[
+ i] else 'downsample2d'
+ downsamples.append(Resample(out_dim, mode=mode))
+ scale /= 2.0
+ self.downsamples = nn.Sequential(*downsamples)
+
+ # middle blocks
+ self.middle = nn.Sequential(ResidualBlock(out_dim, out_dim, dropout),
+ AttentionBlock(out_dim),
+ ResidualBlock(out_dim, out_dim, dropout))
+
+ # output blocks
+ self.head = nn.Sequential(RMS_norm(out_dim, images=False), nn.SiLU(),
+ CausalConv3d(out_dim, z_dim, 3, padding=1))
+
+ def forward(self, x, feat_cache=None, feat_idx=[0]):
+ if feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ x = self.conv1(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = self.conv1(x)
+
+ ## downsamples
+ for layer in self.downsamples:
+ if feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## middle
+ for layer in self.middle:
+ if check_is_instance(layer, ResidualBlock) and feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## head
+ for layer in self.head:
+ if check_is_instance(layer, CausalConv3d) and feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ x = layer(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = layer(x)
+ return x
+
+
+class Encoder3d_38(nn.Module):
+
+ def __init__(self,
+ dim=128,
+ z_dim=4,
+ dim_mult=[1, 2, 4, 4],
+ num_res_blocks=2,
+ attn_scales=[],
+ temperal_downsample=[False, True, True],
+ dropout=0.0):
+ super().__init__()
+ self.dim = dim
+ self.z_dim = z_dim
+ self.dim_mult = dim_mult
+ self.num_res_blocks = num_res_blocks
+ self.attn_scales = attn_scales
+ self.temperal_downsample = temperal_downsample
+
+ # dimensions
+ dims = [dim * u for u in [1] + dim_mult]
+ scale = 1.0
+
+ # init block
+ self.conv1 = CausalConv3d(12, dims[0], 3, padding=1)
+
+ # downsample blocks
+ downsamples = []
+ for i, (in_dim, out_dim) in enumerate(zip(dims[:-1], dims[1:])):
+ t_down_flag = (
+ temperal_downsample[i] if i < len(temperal_downsample) else False
+ )
+ downsamples.append(
+ Down_ResidualBlock(
+ in_dim=in_dim,
+ out_dim=out_dim,
+ dropout=dropout,
+ mult=num_res_blocks,
+ temperal_downsample=t_down_flag,
+ down_flag=i != len(dim_mult) - 1,
+ )
+ )
+ scale /= 2.0
+ self.downsamples = nn.Sequential(*downsamples)
+
+ # middle blocks
+ self.middle = nn.Sequential(
+ ResidualBlock(out_dim, out_dim, dropout),
+ AttentionBlock(out_dim),
+ ResidualBlock(out_dim, out_dim, dropout),
+ )
+
+ # # output blocks
+ self.head = nn.Sequential(
+ RMS_norm(out_dim, images=False),
+ nn.SiLU(),
+ CausalConv3d(out_dim, z_dim, 3, padding=1),
+ )
+
+
+ def forward(self, x, feat_cache=None, feat_idx=[0]):
+
+ if feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ cache_x = torch.cat(
+ [
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(cache_x.device),
+ cache_x,
+ ],
+ dim=2,
+ )
+ x = self.conv1(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = self.conv1(x)
+
+ ## downsamples
+ for layer in self.downsamples:
+ if feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## middle
+ for layer in self.middle:
+ if isinstance(layer, ResidualBlock) and feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## head
+ for layer in self.head:
+ if isinstance(layer, CausalConv3d) and feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ cache_x = torch.cat(
+ [
+ feat_cache[idx][:, :, -1, :, :]
+ .unsqueeze(2)
+ .to(cache_x.device),
+ cache_x,
+ ],
+ dim=2,
+ )
+ x = layer(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = layer(x)
+
+ return x
+
+
+class Decoder3d(nn.Module):
+
+ def __init__(self,
+ dim=128,
+ z_dim=4,
+ dim_mult=[1, 2, 4, 4],
+ num_res_blocks=2,
+ attn_scales=[],
+ temperal_upsample=[False, True, True],
+ dropout=0.0):
+ super().__init__()
+ self.dim = dim
+ self.z_dim = z_dim
+ self.dim_mult = dim_mult
+ self.num_res_blocks = num_res_blocks
+ self.attn_scales = attn_scales
+ self.temperal_upsample = temperal_upsample
+
+ # dimensions
+ dims = [dim * u for u in [dim_mult[-1]] + dim_mult[::-1]]
+ scale = 1.0 / 2**(len(dim_mult) - 2)
+
+ # init block
+ self.conv1 = CausalConv3d(z_dim, dims[0], 3, padding=1)
+
+ # middle blocks
+ self.middle = nn.Sequential(ResidualBlock(dims[0], dims[0], dropout),
+ AttentionBlock(dims[0]),
+ ResidualBlock(dims[0], dims[0], dropout))
+
+ # upsample blocks
+ upsamples = []
+ for i, (in_dim, out_dim) in enumerate(zip(dims[:-1], dims[1:])):
+ # residual (+attention) blocks
+ if i == 1 or i == 2 or i == 3:
+ in_dim = in_dim // 2
+ for _ in range(num_res_blocks + 1):
+ upsamples.append(ResidualBlock(in_dim, out_dim, dropout))
+ if scale in attn_scales:
+ upsamples.append(AttentionBlock(out_dim))
+ in_dim = out_dim
+
+ # upsample block
+ if i != len(dim_mult) - 1:
+ mode = 'upsample3d' if temperal_upsample[i] else 'upsample2d'
+ upsamples.append(Resample(out_dim, mode=mode))
+ scale *= 2.0
+ self.upsamples = nn.Sequential(*upsamples)
+
+ # output blocks
+ self.head = nn.Sequential(RMS_norm(out_dim, images=False), nn.SiLU(),
+ CausalConv3d(out_dim, 3, 3, padding=1))
+
+ def forward(self, x, feat_cache=None, feat_idx=[0]):
+ ## conv1
+ if feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ x = self.conv1(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = self.conv1(x)
+
+ ## middle
+ for layer in self.middle:
+ if check_is_instance(layer, ResidualBlock) and feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## upsamples
+ for layer in self.upsamples:
+ if feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## head
+ for layer in self.head:
+ if check_is_instance(layer, CausalConv3d) and feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ x = layer(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = layer(x)
+ return x
+
+
+
+class Decoder3d_38(nn.Module):
+
+ def __init__(self,
+ dim=128,
+ z_dim=4,
+ dim_mult=[1, 2, 4, 4],
+ num_res_blocks=2,
+ attn_scales=[],
+ temperal_upsample=[False, True, True],
+ dropout=0.0):
+ super().__init__()
+ self.dim = dim
+ self.z_dim = z_dim
+ self.dim_mult = dim_mult
+ self.num_res_blocks = num_res_blocks
+ self.attn_scales = attn_scales
+ self.temperal_upsample = temperal_upsample
+
+ # dimensions
+ dims = [dim * u for u in [dim_mult[-1]] + dim_mult[::-1]]
+ scale = 1.0 / 2 ** (len(dim_mult) - 2)
+ # init block
+ self.conv1 = CausalConv3d(z_dim, dims[0], 3, padding=1)
+
+ # middle blocks
+ self.middle = nn.Sequential(ResidualBlock(dims[0], dims[0], dropout),
+ AttentionBlock(dims[0]),
+ ResidualBlock(dims[0], dims[0], dropout))
+
+ # upsample blocks
+ upsamples = []
+ for i, (in_dim, out_dim) in enumerate(zip(dims[:-1], dims[1:])):
+ t_up_flag = temperal_upsample[i] if i < len(temperal_upsample) else False
+ upsamples.append(
+ Up_ResidualBlock(in_dim=in_dim,
+ out_dim=out_dim,
+ dropout=dropout,
+ mult=num_res_blocks + 1,
+ temperal_upsample=t_up_flag,
+ up_flag=i != len(dim_mult) - 1))
+ self.upsamples = nn.Sequential(*upsamples)
+
+ # output blocks
+ self.head = nn.Sequential(RMS_norm(out_dim, images=False), nn.SiLU(),
+ CausalConv3d(out_dim, 12, 3, padding=1))
+
+
+ def forward(self, x, feat_cache=None, feat_idx=[0], first_chunk=False):
+ if feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ cache_x = torch.cat(
+ [
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(cache_x.device),
+ cache_x,
+ ],
+ dim=2,
+ )
+ x = self.conv1(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = self.conv1(x)
+
+ for layer in self.middle:
+ if check_is_instance(layer, ResidualBlock) and feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## upsamples
+ for layer in self.upsamples:
+ if feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx, first_chunk)
+ else:
+ x = layer(x)
+
+ ## head
+ for layer in self.head:
+ if check_is_instance(layer, CausalConv3d) and feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ cache_x = torch.cat(
+ [
+ feat_cache[idx][:, :, -1, :, :]
+ .unsqueeze(2)
+ .to(cache_x.device),
+ cache_x,
+ ],
+ dim=2,
+ )
+ x = layer(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = layer(x)
+ return x
+
+
+def count_conv3d(model):
+ count = 0
+ for m in model.modules():
+ if isinstance(m, CausalConv3d):
+ count += 1
+ return count
+
+
+class VideoVAE_(nn.Module):
+
+ def __init__(self,
+ dim=96,
+ z_dim=16,
+ dim_mult=[1, 2, 4, 4],
+ num_res_blocks=2,
+ attn_scales=[],
+ temperal_downsample=[False, True, True],
+ dropout=0.0):
+ super().__init__()
+ self.dim = dim
+ self.z_dim = z_dim
+ self.dim_mult = dim_mult
+ self.num_res_blocks = num_res_blocks
+ self.attn_scales = attn_scales
+ self.temperal_downsample = temperal_downsample
+ self.temperal_upsample = temperal_downsample[::-1]
+
+ # modules
+ self.encoder = Encoder3d(dim, z_dim * 2, dim_mult, num_res_blocks,
+ attn_scales, self.temperal_downsample, dropout)
+ self.conv1 = CausalConv3d(z_dim * 2, z_dim * 2, 1)
+ self.conv2 = CausalConv3d(z_dim, z_dim, 1)
+ self.decoder = Decoder3d(dim, z_dim, dim_mult, num_res_blocks,
+ attn_scales, self.temperal_upsample, dropout)
+
+ def forward(self, x):
+ mu, log_var = self.encode(x)
+ z = self.reparameterize(mu, log_var)
+ x_recon = self.decode(z)
+ return x_recon, mu, log_var
+
+ def encode(self, x, scale):
+ self.clear_cache()
+ ## cache
+ t = x.shape[2]
+ iter_ = 1 + (t - 1) // 4
+
+ for i in range(iter_):
+ self._enc_conv_idx = [0]
+ if i == 0:
+ out = self.encoder(x[:, :, :1, :, :],
+ feat_cache=self._enc_feat_map,
+ feat_idx=self._enc_conv_idx)
+ else:
+ out_ = self.encoder(x[:, :, 1 + 4 * (i - 1):1 + 4 * i, :, :],
+ feat_cache=self._enc_feat_map,
+ feat_idx=self._enc_conv_idx)
+ out = torch.cat([out, out_], 2)
+ mu, log_var = self.conv1(out).chunk(2, dim=1)
+ if isinstance(scale[0], torch.Tensor):
+ scale = [s.to(dtype=mu.dtype, device=mu.device) for s in scale]
+ mu = (mu - scale[0].view(1, self.z_dim, 1, 1, 1)) * scale[1].view(
+ 1, self.z_dim, 1, 1, 1)
+ else:
+ scale = scale.to(dtype=mu.dtype, device=mu.device)
+ mu = (mu - scale[0]) * scale[1]
+ return mu
+
+ def decode(self, z, scale):
+ self.clear_cache()
+ # z: [b,c,t,h,w]
+ if isinstance(scale[0], torch.Tensor):
+ scale = [s.to(dtype=z.dtype, device=z.device) for s in scale]
+ z = z / scale[1].view(1, self.z_dim, 1, 1, 1) + scale[0].view(
+ 1, self.z_dim, 1, 1, 1)
+ else:
+ scale = scale.to(dtype=z.dtype, device=z.device)
+ z = z / scale[1] + scale[0]
+ iter_ = z.shape[2]
+ x = self.conv2(z)
+ for i in range(iter_):
+ self._conv_idx = [0]
+ if i == 0:
+ out = self.decoder(x[:, :, i:i + 1, :, :],
+ feat_cache=self._feat_map,
+ feat_idx=self._conv_idx)
+ else:
+ out_ = self.decoder(x[:, :, i:i + 1, :, :],
+ feat_cache=self._feat_map,
+ feat_idx=self._conv_idx)
+ out = torch.cat([out, out_], 2) # may add tensor offload
+ return out
+
+ def reparameterize(self, mu, log_var):
+ std = torch.exp(0.5 * log_var)
+ eps = torch.randn_like(std)
+ return eps * std + mu
+
+ def sample(self, imgs, deterministic=False):
+ mu, log_var = self.encode(imgs)
+ if deterministic:
+ return mu
+ std = torch.exp(0.5 * log_var.clamp(-30.0, 20.0))
+ return mu + std * torch.randn_like(std)
+
+ def clear_cache(self):
+ self._conv_num = count_conv3d(self.decoder)
+ self._conv_idx = [0]
+ self._feat_map = [None] * self._conv_num
+ # cache encode
+ self._enc_conv_num = count_conv3d(self.encoder)
+ self._enc_conv_idx = [0]
+ self._enc_feat_map = [None] * self._enc_conv_num
+
+
+class WanVideoVAE(nn.Module):
+
+ def __init__(self, z_dim=16):
+ super().__init__()
+
+ mean = [
+ -0.7571, -0.7089, -0.9113, 0.1075, -0.1745, 0.9653, -0.1517, 1.5508,
+ 0.4134, -0.0715, 0.5517, -0.3632, -0.1922, -0.9497, 0.2503, -0.2921
+ ]
+ std = [
+ 2.8184, 1.4541, 2.3275, 2.6558, 1.2196, 1.7708, 2.6052, 2.0743,
+ 3.2687, 2.1526, 2.8652, 1.5579, 1.6382, 1.1253, 2.8251, 1.9160
+ ]
+ self.mean = torch.tensor(mean)
+ self.std = torch.tensor(std)
+ self.scale = [self.mean, 1.0 / self.std]
+
+ # init model
+ self.model = VideoVAE_(z_dim=z_dim).eval().requires_grad_(False)
+ self.upsampling_factor = 8
+ self.z_dim = z_dim
+
+
+ def build_1d_mask(self, length, left_bound, right_bound, border_width):
+ x = torch.ones((length,))
+ if not left_bound:
+ x[:border_width] = (torch.arange(border_width) + 1) / border_width
+ if not right_bound:
+ x[-border_width:] = torch.flip((torch.arange(border_width) + 1) / border_width, dims=(0,))
+ return x
+
+
+ def build_mask(self, data, is_bound, border_width):
+ _, _, _, H, W = data.shape
+ h = self.build_1d_mask(H, is_bound[0], is_bound[1], border_width[0])
+ w = self.build_1d_mask(W, is_bound[2], is_bound[3], border_width[1])
+
+ h = repeat(h, "H -> H W", H=H, W=W)
+ w = repeat(w, "W -> H W", H=H, W=W)
+
+ mask = torch.stack([h, w]).min(dim=0).values
+ mask = rearrange(mask, "H W -> 1 1 1 H W")
+ return mask
+
+
+ def tiled_decode(self, hidden_states, device, tile_size, tile_stride):
+ _, _, T, H, W = hidden_states.shape
+ size_h, size_w = tile_size
+ stride_h, stride_w = tile_stride
+
+ # Split tasks
+ tasks = []
+ for h in range(0, H, stride_h):
+ if (h-stride_h >= 0 and h-stride_h+size_h >= H): continue
+ for w in range(0, W, stride_w):
+ if (w-stride_w >= 0 and w-stride_w+size_w >= W): continue
+ h_, w_ = h + size_h, w + size_w
+ tasks.append((h, h_, w, w_))
+
+ data_device = "cpu"
+ computation_device = device
+
+ out_T = T * 4 - 3
+ weight = torch.zeros((1, 1, out_T, H * self.upsampling_factor, W * self.upsampling_factor), dtype=hidden_states.dtype, device=data_device)
+ values = torch.zeros((1, 3, out_T, H * self.upsampling_factor, W * self.upsampling_factor), dtype=hidden_states.dtype, device=data_device)
+
+ for h, h_, w, w_ in tasks:
+ hidden_states_batch = hidden_states[:, :, :, h:h_, w:w_].to(computation_device)
+ hidden_states_batch = self.model.decode(hidden_states_batch, self.scale).to(data_device)
+
+ mask = self.build_mask(
+ hidden_states_batch,
+ is_bound=(h==0, h_>=H, w==0, w_>=W),
+ border_width=((size_h - stride_h) * self.upsampling_factor, (size_w - stride_w) * self.upsampling_factor)
+ ).to(dtype=hidden_states.dtype, device=data_device)
+
+ target_h = h * self.upsampling_factor
+ target_w = w * self.upsampling_factor
+ values[
+ :,
+ :,
+ :,
+ target_h:target_h + hidden_states_batch.shape[3],
+ target_w:target_w + hidden_states_batch.shape[4],
+ ] += hidden_states_batch * mask
+ weight[
+ :,
+ :,
+ :,
+ target_h: target_h + hidden_states_batch.shape[3],
+ target_w: target_w + hidden_states_batch.shape[4],
+ ] += mask
+ values = values / weight
+ values = values.clamp_(-1, 1)
+ return values
+
+
+ def tiled_encode(self, video, device, tile_size, tile_stride):
+ _, _, T, H, W = video.shape
+ size_h, size_w = tile_size
+ stride_h, stride_w = tile_stride
+
+ # Split tasks
+ tasks = []
+ for h in range(0, H, stride_h):
+ if (h-stride_h >= 0 and h-stride_h+size_h >= H): continue
+ for w in range(0, W, stride_w):
+ if (w-stride_w >= 0 and w-stride_w+size_w >= W): continue
+ h_, w_ = h + size_h, w + size_w
+ tasks.append((h, h_, w, w_))
+
+ data_device = "cpu"
+ computation_device = device
+
+ out_T = (T + 3) // 4
+ weight = torch.zeros((1, 1, out_T, H // self.upsampling_factor, W // self.upsampling_factor), dtype=video.dtype, device=data_device)
+ values = torch.zeros((1, self.z_dim, out_T, H // self.upsampling_factor, W // self.upsampling_factor), dtype=video.dtype, device=data_device)
+
+ for h, h_, w, w_ in tasks:
+ hidden_states_batch = video[:, :, :, h:h_, w:w_].to(computation_device)
+ hidden_states_batch = self.model.encode(hidden_states_batch, self.scale).to(data_device)
+
+ mask = self.build_mask(
+ hidden_states_batch,
+ is_bound=(h==0, h_>=H, w==0, w_>=W),
+ border_width=((size_h - stride_h) // self.upsampling_factor, (size_w - stride_w) // self.upsampling_factor)
+ ).to(dtype=video.dtype, device=data_device)
+
+ target_h = h // self.upsampling_factor
+ target_w = w // self.upsampling_factor
+ values[
+ :,
+ :,
+ :,
+ target_h:target_h + hidden_states_batch.shape[3],
+ target_w:target_w + hidden_states_batch.shape[4],
+ ] += hidden_states_batch * mask
+ weight[
+ :,
+ :,
+ :,
+ target_h: target_h + hidden_states_batch.shape[3],
+ target_w: target_w + hidden_states_batch.shape[4],
+ ] += mask
+ values = values / weight
+ return values
+
+
+ def single_encode(self, video, device):
+ video = video.to(device)
+ x = self.model.encode(video, self.scale)
+ return x
+
+
+ def single_decode(self, hidden_state, device):
+ hidden_state = hidden_state.to(device)
+ video = self.model.decode(hidden_state, self.scale)
+ return video.clamp_(-1, 1)
+
+
+ def encode(self, videos, device, tiled=False, tile_size=(34, 34), tile_stride=(18, 16)):
+ videos = [video.to("cpu") for video in videos]
+ hidden_states = []
+ for video in tqdm(videos, desc="VAE encoding", disable=not tiled):
+ video = video.unsqueeze(0)
+ if tiled:
+ tile_size = (tile_size[0] * self.upsampling_factor, tile_size[1] * self.upsampling_factor)
+ tile_stride = (tile_stride[0] * self.upsampling_factor, tile_stride[1] * self.upsampling_factor)
+ hidden_state = self.tiled_encode(video, device, tile_size, tile_stride)
+ else:
+ hidden_state = self.single_encode(video, device)
+ hidden_state = hidden_state.squeeze(0)
+ hidden_states.append(hidden_state)
+ hidden_states = torch.stack(hidden_states)
+ return hidden_states
+
+
+ def decode(self, hidden_states, device, tiled=False, tile_size=(34, 34), tile_stride=(18, 16)):
+ hidden_states = [hidden_state.to("cpu") for hidden_state in hidden_states]
+ videos = []
+ for hidden_state in tqdm(hidden_states, desc="VAE decoding", disable=not tiled):
+ hidden_state = hidden_state.unsqueeze(0)
+ if tiled:
+ video = self.tiled_decode(hidden_state, device, tile_size, tile_stride)
+ else:
+ video = self.single_decode(hidden_state, device)
+ video = video.squeeze(0)
+ videos.append(video)
+ videos = torch.stack(videos)
+ return videos
+
+
+ @staticmethod
+ def state_dict_converter():
+ return WanVideoVAEStateDictConverter()
+
+
+class WanVideoVAEStateDictConverter:
+
+ def __init__(self):
+ pass
+
+ def from_civitai(self, state_dict):
+ state_dict_ = {}
+ if 'model_state' in state_dict:
+ state_dict = state_dict['model_state']
+ for name in state_dict:
+ state_dict_['model.' + name] = state_dict[name]
+ return state_dict_
+
+
+class VideoVAE38_(VideoVAE_):
+
+ def __init__(self,
+ dim=160,
+ z_dim=48,
+ dec_dim=256,
+ dim_mult=[1, 2, 4, 4],
+ num_res_blocks=2,
+ attn_scales=[],
+ temperal_downsample=[False, True, True],
+ dropout=0.0):
+ super(VideoVAE_, self).__init__()
+ self.dim = dim
+ self.z_dim = z_dim
+ self.dim_mult = dim_mult
+ self.num_res_blocks = num_res_blocks
+ self.attn_scales = attn_scales
+ self.temperal_downsample = temperal_downsample
+ self.temperal_upsample = temperal_downsample[::-1]
+
+ # modules
+ self.encoder = Encoder3d_38(dim, z_dim * 2, dim_mult, num_res_blocks,
+ attn_scales, self.temperal_downsample, dropout)
+ self.conv1 = CausalConv3d(z_dim * 2, z_dim * 2, 1)
+ self.conv2 = CausalConv3d(z_dim, z_dim, 1)
+ self.decoder = Decoder3d_38(dec_dim, z_dim, dim_mult, num_res_blocks,
+ attn_scales, self.temperal_upsample, dropout)
+
+
+ def encode(self, x, scale):
+ self.clear_cache()
+ x = patchify(x, patch_size=2)
+ t = x.shape[2]
+ iter_ = 1 + (t - 1) // 4
+ for i in range(iter_):
+ self._enc_conv_idx = [0]
+ if i == 0:
+ out = self.encoder(x[:, :, :1, :, :],
+ feat_cache=self._enc_feat_map,
+ feat_idx=self._enc_conv_idx)
+ else:
+ out_ = self.encoder(x[:, :, 1 + 4 * (i - 1):1 + 4 * i, :, :],
+ feat_cache=self._enc_feat_map,
+ feat_idx=self._enc_conv_idx)
+ out = torch.cat([out, out_], 2)
+ mu, log_var = self.conv1(out).chunk(2, dim=1)
+ if isinstance(scale[0], torch.Tensor):
+ scale = [s.to(dtype=mu.dtype, device=mu.device) for s in scale]
+ mu = (mu - scale[0].view(1, self.z_dim, 1, 1, 1)) * scale[1].view(
+ 1, self.z_dim, 1, 1, 1)
+ else:
+ scale = scale.to(dtype=mu.dtype, device=mu.device)
+ mu = (mu - scale[0]) * scale[1]
+ self.clear_cache()
+ return mu
+
+
+ def decode(self, z, scale):
+ self.clear_cache()
+ if isinstance(scale[0], torch.Tensor):
+ scale = [s.to(dtype=z.dtype, device=z.device) for s in scale]
+ z = z / scale[1].view(1, self.z_dim, 1, 1, 1) + scale[0].view(
+ 1, self.z_dim, 1, 1, 1)
+ else:
+ scale = scale.to(dtype=z.dtype, device=z.device)
+ z = z / scale[1] + scale[0]
+ iter_ = z.shape[2]
+ x = self.conv2(z)
+ for i in range(iter_):
+ self._conv_idx = [0]
+ if i == 0:
+ out = self.decoder(x[:, :, i:i + 1, :, :],
+ feat_cache=self._feat_map,
+ feat_idx=self._conv_idx,
+ first_chunk=True)
+ else:
+ out_ = self.decoder(x[:, :, i:i + 1, :, :],
+ feat_cache=self._feat_map,
+ feat_idx=self._conv_idx)
+ out = torch.cat([out, out_], 2)
+ out = unpatchify(out, patch_size=2)
+ self.clear_cache()
+ return out
+
+
+class WanVideoVAE38(WanVideoVAE):
+
+ def __init__(self, z_dim=48, dim=160):
+ super(WanVideoVAE, self).__init__()
+
+ mean = [
+ -0.2289, -0.0052, -0.1323, -0.2339, -0.2799, 0.0174, 0.1838, 0.1557,
+ -0.1382, 0.0542, 0.2813, 0.0891, 0.1570, -0.0098, 0.0375, -0.1825,
+ -0.2246, -0.1207, -0.0698, 0.5109, 0.2665, -0.2108, -0.2158, 0.2502,
+ -0.2055, -0.0322, 0.1109, 0.1567, -0.0729, 0.0899, -0.2799, -0.1230,
+ -0.0313, -0.1649, 0.0117, 0.0723, -0.2839, -0.2083, -0.0520, 0.3748,
+ 0.0152, 0.1957, 0.1433, -0.2944, 0.3573, -0.0548, -0.1681, -0.0667
+ ]
+ std = [
+ 0.4765, 1.0364, 0.4514, 1.1677, 0.5313, 0.4990, 0.4818, 0.5013,
+ 0.8158, 1.0344, 0.5894, 1.0901, 0.6885, 0.6165, 0.8454, 0.4978,
+ 0.5759, 0.3523, 0.7135, 0.6804, 0.5833, 1.4146, 0.8986, 0.5659,
+ 0.7069, 0.5338, 0.4889, 0.4917, 0.4069, 0.4999, 0.6866, 0.4093,
+ 0.5709, 0.6065, 0.6415, 0.4944, 0.5726, 1.2042, 0.5458, 1.6887,
+ 0.3971, 1.0600, 0.3943, 0.5537, 0.5444, 0.4089, 0.7468, 0.7744
+ ]
+ self.mean = torch.tensor(mean)
+ self.std = torch.tensor(std)
+ self.scale = [self.mean, 1.0 / self.std]
+
+ # init model
+ self.model = VideoVAE38_(z_dim=z_dim, dim=dim).eval().requires_grad_(False)
+ self.upsampling_factor = 16
+ self.z_dim = z_dim
diff --git a/fdanyone/vendor/diffsynth/pipelines/__init__.py b/fdanyone/vendor/diffsynth/pipelines/__init__.py
new file mode 100644
index 0000000..3117b0d
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/pipelines/__init__.py
@@ -0,0 +1,5 @@
+"""Vendored inference pipelines."""
+
+from .wan_video_spatem import WanVideoSpaTemPipeline
+
+__all__ = ["WanVideoSpaTemPipeline"]
diff --git a/fdanyone/vendor/diffsynth/pipelines/base.py b/fdanyone/vendor/diffsynth/pipelines/base.py
new file mode 100644
index 0000000..2a4f01c
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/pipelines/base.py
@@ -0,0 +1,127 @@
+import torch
+import numpy as np
+from PIL import Image
+from torchvision.transforms import GaussianBlur
+
+
+
+class BasePipeline(torch.nn.Module):
+
+ def __init__(self, device="cuda", torch_dtype=torch.float16, height_division_factor=64, width_division_factor=64):
+ super().__init__()
+ self.device = device
+ self.torch_dtype = torch_dtype
+ self.height_division_factor = height_division_factor
+ self.width_division_factor = width_division_factor
+ self.cpu_offload = False
+ self.model_names = []
+
+
+ def check_resize_height_width(self, height, width):
+ if height % self.height_division_factor != 0:
+ height = (height + self.height_division_factor - 1) // self.height_division_factor * self.height_division_factor
+ print(f"The height cannot be evenly divided by {self.height_division_factor}. We round it up to {height}.")
+ if width % self.width_division_factor != 0:
+ width = (width + self.width_division_factor - 1) // self.width_division_factor * self.width_division_factor
+ print(f"The width cannot be evenly divided by {self.width_division_factor}. We round it up to {width}.")
+ return height, width
+
+
+ def preprocess_image(self, image):
+ image = torch.Tensor(np.array(image, dtype=np.float32) * (2 / 255) - 1).permute(2, 0, 1).unsqueeze(0)
+ return image
+
+
+ def preprocess_images(self, images):
+ return [self.preprocess_image(image) for image in images]
+
+
+ def vae_output_to_image(self, vae_output):
+ image = vae_output[0].cpu().float().permute(1, 2, 0).numpy()
+ image = Image.fromarray(((image / 2 + 0.5).clip(0, 1) * 255).astype("uint8"))
+ return image
+
+
+ def vae_output_to_video(self, vae_output):
+ video = vae_output.cpu().permute(1, 2, 0).numpy()
+ video = [Image.fromarray(((image / 2 + 0.5).clip(0, 1) * 255).astype("uint8")) for image in video]
+ return video
+
+
+ def merge_latents(self, value, latents, masks, scales, blur_kernel_size=33, blur_sigma=10.0):
+ if len(latents) > 0:
+ blur = GaussianBlur(kernel_size=blur_kernel_size, sigma=blur_sigma)
+ height, width = value.shape[-2:]
+ weight = torch.ones_like(value)
+ for latent, mask, scale in zip(latents, masks, scales):
+ mask = self.preprocess_image(mask.resize((width, height))).mean(dim=1, keepdim=True) > 0
+ mask = mask.repeat(1, latent.shape[1], 1, 1).to(dtype=latent.dtype, device=latent.device)
+ mask = blur(mask)
+ value += latent * mask * scale
+ weight += mask * scale
+ value /= weight
+ return value
+
+
+ def control_noise_via_local_prompts(self, prompt_emb_global, prompt_emb_locals, masks, mask_scales, inference_callback, special_kwargs=None, special_local_kwargs_list=None):
+ if special_kwargs is None:
+ noise_pred_global = inference_callback(prompt_emb_global)
+ else:
+ noise_pred_global = inference_callback(prompt_emb_global, special_kwargs)
+ if special_local_kwargs_list is None:
+ noise_pred_locals = [inference_callback(prompt_emb_local) for prompt_emb_local in prompt_emb_locals]
+ else:
+ noise_pred_locals = [inference_callback(prompt_emb_local, special_kwargs) for prompt_emb_local, special_kwargs in zip(prompt_emb_locals, special_local_kwargs_list)]
+ noise_pred = self.merge_latents(noise_pred_global, noise_pred_locals, masks, mask_scales)
+ return noise_pred
+
+
+ def extend_prompt(self, prompt, local_prompts, masks, mask_scales):
+ local_prompts = local_prompts or []
+ masks = masks or []
+ mask_scales = mask_scales or []
+ extended_prompt_dict = self.prompter.extend_prompt(prompt)
+ prompt = extended_prompt_dict.get("prompt", prompt)
+ local_prompts += extended_prompt_dict.get("prompts", [])
+ masks += extended_prompt_dict.get("masks", [])
+ mask_scales += [100.0] * len(extended_prompt_dict.get("masks", []))
+ return prompt, local_prompts, masks, mask_scales
+
+
+ def enable_cpu_offload(self):
+ self.cpu_offload = True
+
+
+ def load_models_to_device(self, loadmodel_names=[]):
+ # only load models to device if cpu_offload is enabled
+ if not self.cpu_offload:
+ return
+ # offload the unneeded models to cpu
+ for model_name in self.model_names:
+ if model_name not in loadmodel_names:
+ model = getattr(self, model_name)
+ if model is not None:
+ if hasattr(model, "vram_management_enabled") and model.vram_management_enabled:
+ for module in model.modules():
+ if hasattr(module, "offload"):
+ module.offload()
+ else:
+ model.cpu()
+ # load the needed models to device
+ for model_name in loadmodel_names:
+ model = getattr(self, model_name)
+ if model is not None:
+ if hasattr(model, "vram_management_enabled") and model.vram_management_enabled:
+ for module in model.modules():
+ if hasattr(module, "onload"):
+ module.onload()
+ else:
+ model.to(self.device)
+ # fresh the cuda cache
+ torch.cuda.empty_cache()
+
+
+ def generate_noise(self, shape, seed=None, device="cpu", dtype=torch.float16):
+ generator = None if seed is None else torch.Generator(device).manual_seed(seed)
+ noise = torch.randn(shape, generator=generator, device=device, dtype=dtype)
+ return noise
diff --git a/fdanyone/vendor/diffsynth/pipelines/wan_video_spatem.py b/fdanyone/vendor/diffsynth/pipelines/wan_video_spatem.py
new file mode 100644
index 0000000..53ad33c
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/pipelines/wan_video_spatem.py
@@ -0,0 +1,518 @@
+from ..models.wan_video_dit import WanModel
+from ..models.wan_video_pose_encoder import PoseEncoder
+from ..models.wan_video_text_encoder import WanTextEncoder
+from ..models.wan_video_vae import WanVideoVAE
+from ..schedulers.flow_match import FlowMatchScheduler
+from ..schedulers.bride_match import BridgeMatchScheduler
+from ..pipelines.base import BasePipeline
+from ..prompters.wan_prompter import WanPrompter
+import torch
+import torch.nn as nn
+import numpy as np
+import torch.nn.functional as F
+from PIL import Image
+from tqdm import tqdm
+from typing import Optional
+from functools import partial
+from einops import rearrange
+
+from ..vram_management import enable_vram_management, AutoWrappedModule, AutoWrappedLinear
+from ..models.wan_video_text_encoder import T5RelativeEmbedding, T5LayerNorm
+from ..models.wan_video_dit import RMSNorm, SelfAttention, CrossAttentionSrcCam, ViewPackEmbedding
+from ..models.wan_video_vae import RMS_norm, CausalConv3d, Upsample
+
+
+class WanVideoSpaTemPipeline(BasePipeline):
+
+ def __init__(self, device="cuda", torch_dtype=torch.float16, tokenizer_path=None):
+ super().__init__(device=device, torch_dtype=torch_dtype)
+ self.scheduler = FlowMatchScheduler(shift=5, sigma_min=0.0, extra_one_step=True)
+ self.prompter = WanPrompter(tokenizer_path=tokenizer_path)
+ self.text_encoder: WanTextEncoder = None
+ self.dit: WanModel = None
+ self.vae: WanVideoVAE = None
+ self.model_names = ["text_encoder", "dit", "vae"]
+ self.height_division_factor = 16
+ self.width_division_factor = 16
+
+ def enable_vram_management(self, num_persistent_param_in_dit=None):
+ dtype = next(iter(self.text_encoder.parameters())).dtype
+ enable_vram_management(
+ self.text_encoder,
+ module_map={
+ torch.nn.Linear: AutoWrappedLinear,
+ torch.nn.Embedding: AutoWrappedModule,
+ T5RelativeEmbedding: AutoWrappedModule,
+ T5LayerNorm: AutoWrappedModule,
+ },
+ module_config=dict(
+ offload_dtype=dtype,
+ offload_device="cpu",
+ onload_dtype=dtype,
+ onload_device="cpu",
+ computation_dtype=self.torch_dtype,
+ computation_device=self.device,
+ ),
+ )
+ dtype = next(iter(self.dit.parameters())).dtype
+ enable_vram_management(
+ self.dit,
+ module_map={
+ torch.nn.Linear: AutoWrappedLinear,
+ torch.nn.Conv3d: AutoWrappedModule,
+ torch.nn.LayerNorm: AutoWrappedModule,
+ RMSNorm: AutoWrappedModule,
+ },
+ module_config=dict(
+ offload_dtype=dtype,
+ offload_device="cpu",
+ onload_dtype=dtype,
+ onload_device=self.device,
+ computation_dtype=self.torch_dtype,
+ computation_device=self.device,
+ ),
+ max_num_param=num_persistent_param_in_dit,
+ overflow_module_config=dict(
+ offload_dtype=dtype,
+ offload_device="cpu",
+ onload_dtype=dtype,
+ onload_device="cpu",
+ computation_dtype=self.torch_dtype,
+ computation_device=self.device,
+ ),
+ )
+ dtype = next(iter(self.vae.parameters())).dtype
+ enable_vram_management(
+ self.vae,
+ module_map={
+ torch.nn.Linear: AutoWrappedLinear,
+ torch.nn.Conv2d: AutoWrappedModule,
+ RMS_norm: AutoWrappedModule,
+ CausalConv3d: AutoWrappedModule,
+ Upsample: AutoWrappedModule,
+ torch.nn.SiLU: AutoWrappedModule,
+ torch.nn.Dropout: AutoWrappedModule,
+ },
+ module_config=dict(
+ offload_dtype=dtype,
+ offload_device="cpu",
+ onload_dtype=dtype,
+ onload_device=self.device,
+ computation_dtype=self.torch_dtype,
+ computation_device=self.device,
+ ),
+ )
+ self.enable_cpu_offload()
+
+ def init_spatem_modules(
+ self,
+ disable_video_attn: bool = False,
+ use_4d_attn: bool = False,
+ use_mvs_attn: bool = False,
+ use_src_self_attn: bool = False,
+ use_src_cross_attn: bool = False,
+ freqs_src_shift: int = 121,
+ use_viewpack: bool = True,
+ viewpack_dropout_prob: float = 0.0,
+ use_pose_encoder: bool = True,
+ pose_encoder_type: str = "rgb",
+ use_cam_encoder: bool = False,
+ range_4d_attn: tuple[int, int, int] = (0, None, 2),
+ range_mvs_attn: tuple[int, int, int] = (1, None, 2),
+ range_src_self_attn: tuple[int, int, int] = (0, None, 2),
+ range_src_cross_attn: tuple[int, int, int] = (0, None, 2),
+ use_lbm: bool = False,
+ fill_wpmask_with_noise: bool = False,
+ ):
+ device, dtype = self.dit.patch_embedding.weight.device, self.dit.patch_embedding.weight.dtype
+
+ if disable_video_attn:
+ # todo: delete self_attn layers from the model
+ if use_4d_attn:
+ raise ValueError("Cannot use 4D attention when video attention is disabled")
+ for block in self.dit.blocks:
+ block.disable_video_attn = True
+
+ if use_4d_attn:
+ b, e, s = range_4d_attn
+ for block in self.dit.blocks[b:e:s]:
+ block.use_4d_attn = True
+
+ if use_mvs_attn:
+ b, e, s = range_mvs_attn
+ for block in self.dit.blocks[b:e:s]:
+ block.use_mvs_attn = True
+
+ dim = block.self_attn.q.weight.shape[0]
+ block.modulation_mvs = nn.Parameter(block.modulation[:, :3, :].detach().clone())
+ block.norm1_mvs = nn.LayerNorm(dim, eps=block.norm1.eps, elementwise_affine=False).to(
+ device=device, dtype=dtype
+ )
+ block.self_attn_mvs = SelfAttention(dim, block.self_attn.num_heads, block.self_attn.norm_q.eps).to(
+ device=device, dtype=dtype
+ )
+ block.self_attn_mvs.load_state_dict(block.self_attn.state_dict(), strict=True)
+
+ if not 0.0 <= viewpack_dropout_prob <= 1.0:
+ raise ValueError("viewpack_dropout_prob should be between 0 and 1")
+ if viewpack_dropout_prob > 0.0 and not use_viewpack:
+ raise ValueError("viewpack_dropout_prob requires use_viewpack=True")
+
+ if use_viewpack:
+ viewpack_emb = ViewPackEmbedding(
+ in_dim=self.dit.patch_embedding.weight.shape[1],
+ dim=self.dit.patch_embedding.weight.shape[0],
+ patch_size=list(self.dit.patch_embedding.kernel_size),
+ )
+ viewpack_emb.initialize_from_patch_embedding(self.dit.patch_embedding)
+ self.dit.viewpack_embedding = viewpack_emb.to(device=device, dtype=dtype)
+ elif use_src_self_attn:
+ if use_src_cross_attn:
+ raise ValueError("Cannot use both src self-attention and src cross-attention")
+
+ b, e, s = range_src_self_attn
+ for block in self.dit.blocks[b:e:s]:
+ block.use_src_self_attn = True
+
+ dim = block.self_attn.q.weight.shape[0]
+ block.modulation_src = nn.Parameter(block.modulation[:, :3, :].detach().clone())
+ block.norm1_src = nn.LayerNorm(dim, eps=block.norm1.eps, elementwise_affine=False).to(
+ device=device, dtype=dtype
+ )
+ block.self_attn_src = SelfAttention(dim, block.self_attn.num_heads, block.self_attn.norm_q.eps).to(
+ device=device, dtype=dtype
+ )
+ block.self_attn_src.load_state_dict(block.self_attn.state_dict(), strict=True)
+ elif use_src_cross_attn:
+ b, e, s = range_src_cross_attn
+ for block in self.dit.blocks[b:e:s]:
+ block.use_src_cross_attn = True
+
+ dim = block.self_attn.q.weight.shape[0]
+ block.modulation_src = nn.Parameter(block.modulation[:, :3, :].detach().clone())
+ block.norm1_src = nn.LayerNorm(dim, eps=block.norm1.eps, elementwise_affine=False).to(
+ device=device, dtype=dtype
+ )
+ block.cross_attn_src = CrossAttentionSrcCam(
+ dim, block.self_attn.num_heads, block.self_attn.norm_q.eps
+ ).to(device=device, dtype=dtype)
+ block.cross_attn_src.load_state_dict(block.self_attn.state_dict(), strict=True)
+
+ if use_pose_encoder:
+ if pose_encoder_type == "rgb":
+ in_channels = 3
+ elif pose_encoder_type == "rgbd":
+ in_channels = 4
+ else:
+ raise ValueError(f"Invalid pose_encoder_type: {pose_encoder_type}")
+ pose_encoder = PoseEncoder(out_dim=self.dit.patch_embedding.out_channels, in_channels=in_channels)
+ self.dit.pose_encoder = pose_encoder.to(device=device, dtype=dtype)
+
+ if use_cam_encoder:
+ dim = self.dit.blocks[0].self_attn.q.weight.shape[0]
+ for block in self.dit.blocks:
+ block.use_cam_encoder = True
+ block.cam_encoder = nn.Linear(12, dim).to(device=device, dtype=dtype)
+ block.projector = nn.Linear(dim, dim).to(device=device, dtype=dtype)
+ block.cam_encoder.weight.data.zero_()
+ block.cam_encoder.bias.data.zero_()
+ block.projector.weight = nn.Parameter(torch.eye(dim, device=device, dtype=dtype))
+ block.projector.bias = nn.Parameter(torch.zeros(dim, device=device, dtype=dtype))
+
+ if use_lbm:
+ # TODO: hard-code for now
+ self.scheduler = BridgeMatchScheduler()
+ self.dit.fill_wpmask_with_noise = fill_wpmask_with_noise
+
+ self.dit.use_pose_encoder = use_pose_encoder
+ self.dit.use_cam_encoder = use_cam_encoder
+ self.dit.use_viewpack = use_viewpack
+ self.dit.viewpack_dropout_prob = viewpack_dropout_prob
+ self.dit.use_src_attn = use_src_self_attn or use_src_cross_attn
+ self.dit.use_lbm = use_lbm
+ self.dit.freqs_src_shift = freqs_src_shift
+
+ def denoising_model(self):
+ return self.dit
+
+ def encode_prompt(self, prompt, positive=True):
+ prompt_emb = self.prompter.encode_prompt(prompt, positive=positive)
+ return {"context": prompt_emb}
+
+ def tensor2video(self, frames):
+ frames = rearrange(frames, "c f h w -> f h w c")
+ frames = ((frames.float() + 1) * 127.5).clip(0, 255).cpu().numpy().astype(np.uint8)
+ frames = [Image.fromarray(frame) for frame in frames]
+ return frames
+
+ def prepare_extra_input(self, latents=None):
+ return {}
+
+ def encode_video(self, input_video, tiled=True, tile_size=(34, 34), tile_stride=(18, 16)):
+ latents = self.vae.encode(
+ input_video, device=self.device, tiled=tiled, tile_size=tile_size, tile_stride=tile_stride
+ )
+ return latents
+
+ def decode_video(self, latents, tiled=True, tile_size=(34, 34), tile_stride=(18, 16)):
+ frames = self.vae.decode(latents, device=self.device, tiled=tiled, tile_size=tile_size, tile_stride=tile_stride)
+ return frames
+
+ def encode_fmask(self, mask, size):
+ f_, h_, w_ = size
+ g_ = (mask.shape[2] - 1) // (f_ - 1)
+
+ # union of the frames in each latent (the first frame is encoded independently)
+ mask = torch.cat([mask[:, :, :1].repeat(1, 1, g_ - 1, 1, 1), mask], dim=2)
+ mask = rearrange(mask, "v c (f g) h w -> v c f g h w", f=f_, g=g_)
+ mask = mask.max(dim=3).values
+
+ # interpolate along the spatial dimensions
+ mask = rearrange(mask, "v c f h w -> (v f) c h w")
+ mask = F.interpolate(mask, size=(h_, w_), mode="area")
+ mask = rearrange(mask, "(v f) c h w -> v c f h w", f=f_)
+ return mask
+
+ def encode_wpmask(self, mask, size):
+ f_, h_, w_ = size
+ g_ = (mask.shape[2] - 1) // (f_ - 1)
+
+ # intersection of the frames in each latent (the first frame is encoded independently)
+ mask = torch.cat([mask[:, :, :1].repeat(1, 1, g_ - 1, 1, 1), mask], dim=2)
+ mask = rearrange(mask, "v c (f g) h w -> v c f g h w", f=f_, g=g_)
+ mask = mask.min(dim=3).values
+
+ # interpolate along the spatial dimensions
+ mask = rearrange(mask, "v c f h w -> (v f) c h w")
+ mask = F.interpolate(mask, size=(h_, w_), mode="area")
+ mask = rearrange(mask, "(v f) c h w -> v c f h w", f=f_)
+ return mask
+
+ @torch.no_grad()
+ def __call__(
+ self,
+ prompt,
+ negative_prompt="色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,三条腿,背景人很多,倒着走",
+ src_videos: torch.Tensor = None,
+ skeletons: torch.Tensor = None,
+ wpvideos: torch.Tensor = None,
+ wpmasks: torch.Tensor = None,
+ cam_emb: torch.Tensor = None,
+ input_image: Image.Image = None,
+ input_video: torch.Tensor = None,
+ denoising_strength: float = 1.0,
+ seed: int = None,
+ rand_device: str = "cpu",
+ height: int = 832,
+ width: int = 480,
+ num_frames: int = None,
+ cfg_scale: float = 5.0,
+ num_inference_steps: int = 50,
+ sigma_shift: float = 5.0,
+ tiled: bool = True,
+ tile_size: tuple[int, int] = (52, 30),
+ tile_stride: tuple[int, int] = (26, 15),
+ tea_cache_l1_thresh: float = None,
+ tea_cache_model_id: str = "",
+ progress_bar_cmd=partial(tqdm, desc="Denoising"),
+ progress_bar_st=None,
+ return_tensor=False,
+ ):
+ assert num_frames is None, "num_frames is not supported for WanVideoSpaTemPipeline"
+ assert input_image is None, "input_image is not supported for WanVideoSpaTemPipeline"
+ assert input_video is None, "input_video is not supported for WanVideoSpaTemPipeline"
+ assert tea_cache_l1_thresh is None, "tea_cache_l1_thresh is not supported for WanVideoSpaTemPipeline"
+
+ # Parameter check
+ height, width = self.check_resize_height_width(height, width)
+
+ # Tiler parameters
+ tiler_kwargs = {"tiled": tiled, "tile_size": tile_size, "tile_stride": tile_stride}
+
+ # Scheduler
+ if self.dit.use_lbm:
+ # bridge matching scheduler
+ self.scheduler.set_timesteps(num_inference_steps)
+ else:
+ # flow matching scheduler
+ self.scheduler.set_timesteps(num_inference_steps, denoising_strength=denoising_strength, shift=sigma_shift)
+
+ src_videos = src_videos.to(dtype=self.torch_dtype, device=self.device)
+ if skeletons is not None:
+ skeletons = skeletons.to(dtype=self.torch_dtype, device=self.device)
+ if wpvideos is not None:
+ wpvideos = wpvideos.to(dtype=self.torch_dtype, device=self.device)
+ if wpmasks is not None:
+ wpmasks = wpmasks.to(dtype=self.torch_dtype, device=self.device)
+ if cam_emb is not None:
+ cam_emb = cam_emb.to(dtype=self.torch_dtype, device=self.device)
+
+ if skeletons is not None:
+ num_cameras = skeletons.shape[0]
+ num_frames = skeletons.shape[2]
+ elif wpvideos is not None:
+ num_cameras = wpvideos.shape[0]
+ num_frames = wpvideos.shape[2]
+ else:
+ raise ValueError("Either skeletons or wpvideos must be provided")
+
+ # Initialize noise
+ noise_shape = (
+ num_cameras,
+ self.vae.model.z_dim,
+ (num_frames - 1) // 4 + 1,
+ height // self.vae.upsampling_factor,
+ width // self.vae.upsampling_factor,
+ )
+ noise = self.generate_noise(noise_shape, seed=seed, device=rand_device, dtype=torch.float32)
+ noise = noise.to(dtype=self.torch_dtype, device=self.device)
+
+ if input_video is not None:
+ self.load_models_to_device(["vae"])
+ input_video = self.preprocess_images(input_video)
+ input_video = torch.stack(input_video, dim=2).to(dtype=self.torch_dtype, device=self.device)
+ latents = self.encode_video(input_video, **tiler_kwargs).to(dtype=self.torch_dtype, device=self.device)
+ latents = self.scheduler.add_noise(latents, noise, timestep=self.scheduler.timesteps[0])
+ else:
+ latents = noise
+
+ # Encode source video
+ self.load_models_to_device(["vae"])
+ src_latents = self.encode_video(src_videos, **tiler_kwargs)
+ src_latents = src_latents.to(dtype=self.torch_dtype, device=self.device)
+ src_latents_nega = torch.zeros_like(src_latents)
+
+ # Latent bridge matching
+ if self.dit.use_lbm:
+ if skeletons is not None:
+ # skeleton-based: use primary src_latents as bridge source
+ lbm_src_latents = src_latents[:1].expand_as(latents)
+ elif wpvideos is not None:
+ lbm_src_latents = self.encode_video(wpvideos, **tiler_kwargs).to(
+ dtype=self.torch_dtype, device=self.device
+ )
+ if self.dit.fill_wpmask_with_noise:
+ wpmask_latents = self.encode_wpmask(wpmasks, size=lbm_src_latents.shape[-3:])
+ lbm_src_latents = lbm_src_latents * wpmask_latents + noise * (1 - wpmask_latents)
+
+ latents = lbm_src_latents
+
+ # Encode prompts
+ self.load_models_to_device(["text_encoder"])
+ prompt_emb_posi = self.encode_prompt(prompt, positive=True)
+ if cfg_scale != 1.0:
+ prompt_emb_nega = self.encode_prompt(negative_prompt, positive=False)
+
+ image_emb = {}
+
+ # Extra input
+ extra_input = self.prepare_extra_input(latents)
+
+ # Denoise
+ self.load_models_to_device(["dit"])
+ for progress_id, timestep in enumerate(progress_bar_cmd(self.scheduler.timesteps)):
+ timestep = timestep.unsqueeze(0).to(dtype=self.torch_dtype, device=self.device)
+ timestep = torch.cat([timestep] * num_cameras, dim=0)
+
+ # Inference
+ noise_pred_posi = self.denoising_model()(
+ x=latents,
+ x_src=src_latents,
+ timestep=timestep,
+ skeletons=skeletons,
+ cam_emb=cam_emb,
+ **prompt_emb_posi,
+ **image_emb,
+ **extra_input,
+ )
+ if cfg_scale != 1.0:
+ noise_pred_nega = self.denoising_model()(
+ x=latents,
+ x_src=src_latents_nega,
+ timestep=timestep,
+ skeletons=skeletons,
+ cam_emb=cam_emb,
+ **prompt_emb_nega,
+ **image_emb,
+ **extra_input,
+ )
+ noise_pred = noise_pred_nega + cfg_scale * (noise_pred_posi - noise_pred_nega)
+ else:
+ noise_pred = noise_pred_posi
+
+ # Scheduler
+ latents = self.scheduler.step(noise_pred, self.scheduler.timesteps[progress_id], latents)
+
+ # Decode
+ self.load_models_to_device(["vae"])
+ pred_videos = self.decode_video(latents, **tiler_kwargs)
+
+ if return_tensor:
+ return pred_videos
+
+ self.load_models_to_device([])
+ pred_video_list = []
+ for pred_video in pred_videos:
+ pred_video_list.append(self.tensor2video(pred_video))
+ return pred_video_list
+
+
+class TeaCache:
+ def __init__(self, num_inference_steps, rel_l1_thresh, model_id):
+ self.num_inference_steps = num_inference_steps
+ self.step = 0
+ self.accumulated_rel_l1_distance = 0
+ self.previous_modulated_input = None
+ self.rel_l1_thresh = rel_l1_thresh
+ self.previous_residual = None
+ self.previous_hidden_states = None
+
+ self.coefficients_dict = {
+ "Wan2.1-T2V-1.3B": [-5.21862437e04, 9.23041404e03, -5.28275948e02, 1.36987616e01, -4.99875664e-02],
+ "Wan2.1-T2V-14B": [-3.03318725e05, 4.90537029e04, -2.65530556e03, 5.87365115e01, -3.15583525e-01],
+ "Wan2.1-I2V-14B-480P": [2.57151496e05, -3.54229917e04, 1.40286849e03, -1.35890334e01, 1.32517977e-01],
+ "Wan2.1-I2V-14B-720P": [8.10705460e03, 2.13393892e03, -3.72934672e02, 1.66203073e01, -4.17769401e-02],
+ }
+ if model_id not in self.coefficients_dict:
+ supported_model_ids = ", ".join([i for i in self.coefficients_dict])
+ raise ValueError(
+ f"{model_id} is not a supported TeaCache model id. Please choose a valid model id in ({supported_model_ids})."
+ )
+ self.coefficients = self.coefficients_dict[model_id]
+
+ def check(self, dit: WanModel, x, t_mod):
+ modulated_inp = t_mod.clone()
+ if self.step == 0 or self.step == self.num_inference_steps - 1:
+ should_calc = True
+ self.accumulated_rel_l1_distance = 0
+ else:
+ coefficients = self.coefficients
+ rescale_func = np.poly1d(coefficients)
+ self.accumulated_rel_l1_distance += rescale_func(
+ (
+ (modulated_inp - self.previous_modulated_input).abs().mean()
+ / self.previous_modulated_input.abs().mean()
+ )
+ .cpu()
+ .item()
+ )
+ if self.accumulated_rel_l1_distance < self.rel_l1_thresh:
+ should_calc = False
+ else:
+ should_calc = True
+ self.accumulated_rel_l1_distance = 0
+ self.previous_modulated_input = modulated_inp
+ self.step += 1
+ if self.step == self.num_inference_steps:
+ self.step = 0
+ if should_calc:
+ self.previous_hidden_states = x.clone()
+ return not should_calc
+
+ def store(self, hidden_states):
+ self.previous_residual = hidden_states - self.previous_hidden_states
+ self.previous_hidden_states = None
+
+ def update(self, hidden_states):
+ hidden_states = hidden_states + self.previous_residual
+ return hidden_states
diff --git a/fdanyone/vendor/diffsynth/prompters/__init__.py b/fdanyone/vendor/diffsynth/prompters/__init__.py
new file mode 100644
index 0000000..d986931
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/prompters/__init__.py
@@ -0,0 +1,5 @@
+"""Vendored Wan prompt helpers."""
+
+from .wan_prompter import WanPrompter
+
+__all__ = ["WanPrompter"]
diff --git a/fdanyone/vendor/diffsynth/prompters/base_prompter.py b/fdanyone/vendor/diffsynth/prompters/base_prompter.py
new file mode 100644
index 0000000..749fc3f
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/prompters/base_prompter.py
@@ -0,0 +1,69 @@
+import torch
+
+
+
+def tokenize_long_prompt(tokenizer, prompt, max_length=None):
+ # Get model_max_length from self.tokenizer
+ length = tokenizer.model_max_length if max_length is None else max_length
+
+ # To avoid the warning. set self.tokenizer.model_max_length to +oo.
+ tokenizer.model_max_length = 99999999
+
+ # Tokenize it!
+ input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+
+ # Determine the real length.
+ max_length = (input_ids.shape[1] + length - 1) // length * length
+
+ # Restore tokenizer.model_max_length
+ tokenizer.model_max_length = length
+
+ # Tokenize it again with fixed length.
+ input_ids = tokenizer(
+ prompt,
+ return_tensors="pt",
+ padding="max_length",
+ max_length=max_length,
+ truncation=True
+ ).input_ids
+
+ # Reshape input_ids to fit the text encoder.
+ num_sentence = input_ids.shape[1] // length
+ input_ids = input_ids.reshape((num_sentence, length))
+
+ return input_ids
+
+
+
+class BasePrompter:
+ def __init__(self):
+ self.refiners = []
+ self.extenders = []
+
+
+ def load_prompt_refiners(self, model_manager, refiner_classes=[]):
+ for refiner_class in refiner_classes:
+ refiner = refiner_class.from_model_manager(model_manager)
+ self.refiners.append(refiner)
+
+ def load_prompt_extenders(self, model_manager, extender_classes=[]):
+ for extender_class in extender_classes:
+ extender = extender_class.from_model_manager(model_manager)
+ self.extenders.append(extender)
+
+
+ @torch.no_grad()
+ def process_prompt(self, prompt, positive=True):
+ if isinstance(prompt, list):
+ prompt = [self.process_prompt(prompt_, positive=positive) for prompt_ in prompt]
+ else:
+ for refiner in self.refiners:
+ prompt = refiner(prompt, positive=positive)
+ return prompt
+
+ @torch.no_grad()
+ def extend_prompt(self, prompt:str, positive=True):
+ extended_prompt = dict(prompt=prompt)
+ for extender in self.extenders:
+ extended_prompt = extender(extended_prompt)
+ return extended_prompt
diff --git a/fdanyone/vendor/diffsynth/prompters/wan_prompter.py b/fdanyone/vendor/diffsynth/prompters/wan_prompter.py
new file mode 100644
index 0000000..01a765d
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/prompters/wan_prompter.py
@@ -0,0 +1,109 @@
+from .base_prompter import BasePrompter
+from ..models.wan_video_text_encoder import WanTextEncoder
+from transformers import AutoTokenizer
+import os, torch
+import ftfy
+import html
+import string
+import regex as re
+
+
+def basic_clean(text):
+ text = ftfy.fix_text(text)
+ text = html.unescape(html.unescape(text))
+ return text.strip()
+
+
+def whitespace_clean(text):
+ text = re.sub(r'\s+', ' ', text)
+ text = text.strip()
+ return text
+
+
+def canonicalize(text, keep_punctuation_exact_string=None):
+ text = text.replace('_', ' ')
+ if keep_punctuation_exact_string:
+ text = keep_punctuation_exact_string.join(
+ part.translate(str.maketrans('', '', string.punctuation))
+ for part in text.split(keep_punctuation_exact_string))
+ else:
+ text = text.translate(str.maketrans('', '', string.punctuation))
+ text = text.lower()
+ text = re.sub(r'\s+', ' ', text)
+ return text.strip()
+
+
+class HuggingfaceTokenizer:
+
+ def __init__(self, name, seq_len=None, clean=None, **kwargs):
+ assert clean in (None, 'whitespace', 'lower', 'canonicalize')
+ self.name = name
+ self.seq_len = seq_len
+ self.clean = clean
+
+ # init tokenizer
+ self.tokenizer = AutoTokenizer.from_pretrained(name, **kwargs)
+ self.vocab_size = self.tokenizer.vocab_size
+
+ def __call__(self, sequence, **kwargs):
+ return_mask = kwargs.pop('return_mask', False)
+
+ # arguments
+ _kwargs = {'return_tensors': 'pt'}
+ if self.seq_len is not None:
+ _kwargs.update({
+ 'padding': 'max_length',
+ 'truncation': True,
+ 'max_length': self.seq_len
+ })
+ _kwargs.update(**kwargs)
+
+ # tokenization
+ if isinstance(sequence, str):
+ sequence = [sequence]
+ if self.clean:
+ sequence = [self._clean(u) for u in sequence]
+ ids = self.tokenizer(sequence, **_kwargs)
+
+ # output
+ if return_mask:
+ return ids.input_ids, ids.attention_mask
+ else:
+ return ids.input_ids
+
+ def _clean(self, text):
+ if self.clean == 'whitespace':
+ text = whitespace_clean(basic_clean(text))
+ elif self.clean == 'lower':
+ text = whitespace_clean(basic_clean(text)).lower()
+ elif self.clean == 'canonicalize':
+ text = canonicalize(basic_clean(text))
+ return text
+
+
+class WanPrompter(BasePrompter):
+
+ def __init__(self, tokenizer_path=None, text_len=512):
+ super().__init__()
+ self.text_len = text_len
+ self.text_encoder = None
+ self.fetch_tokenizer(tokenizer_path)
+
+ def fetch_tokenizer(self, tokenizer_path=None):
+ if tokenizer_path is not None:
+ self.tokenizer = HuggingfaceTokenizer(name=tokenizer_path, seq_len=self.text_len, clean='whitespace')
+
+ def fetch_models(self, text_encoder: WanTextEncoder = None):
+ self.text_encoder = text_encoder
+
+ def encode_prompt(self, prompt, positive=True, device="cuda"):
+ prompt = self.process_prompt(prompt, positive=positive)
+
+ ids, mask = self.tokenizer(prompt, return_mask=True, add_special_tokens=True)
+ ids = ids.to(device)
+ mask = mask.to(device)
+ seq_lens = mask.gt(0).sum(dim=1).long()
+ prompt_emb = self.text_encoder(ids, mask)
+ for i, v in enumerate(seq_lens):
+ prompt_emb[:, v:] = 0
+ return prompt_emb
diff --git a/fdanyone/vendor/diffsynth/schedulers/__init__.py b/fdanyone/vendor/diffsynth/schedulers/__init__.py
new file mode 100644
index 0000000..79d3552
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/schedulers/__init__.py
@@ -0,0 +1,5 @@
+"""Vendored diffusion schedulers."""
+
+from .flow_match import FlowMatchScheduler
+
+__all__ = ["FlowMatchScheduler"]
diff --git a/fdanyone/vendor/diffsynth/schedulers/bride_match.py b/fdanyone/vendor/diffsynth/schedulers/bride_match.py
new file mode 100644
index 0000000..c157702
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/schedulers/bride_match.py
@@ -0,0 +1,71 @@
+import torch, math
+
+
+class BridgeMatchScheduler:
+
+ def __init__(
+ self,
+ num_train_timesteps=1000,
+ num_inference_steps=8,
+ sigma_max=1.0,
+ bridge_noise_sigma=0.005,
+ ):
+ self.sigma_max = sigma_max
+ self.sigma_min = sigma_max / num_train_timesteps
+ self.num_train_timesteps = num_train_timesteps # train timesteps for base model
+ self.num_inference_steps = num_inference_steps # inference steps for bridge matching
+ self.bridge_noise_sigma = bridge_noise_sigma
+
+ self.set_timesteps(self.num_inference_steps)
+
+ def set_timesteps(self, num_inference_steps=8, training=False):
+ sigma_start = self.sigma_max
+ sigma_end = self.sigma_max / num_inference_steps
+ self.sigmas = torch.linspace(sigma_start, sigma_end, num_inference_steps)
+ self.timesteps = self.sigmas * self.num_train_timesteps
+
+ self.training = training
+
+ def retrieve_sigma(self, timestep):
+ if isinstance(timestep, torch.Tensor):
+ timestep = timestep.cpu()
+ timestep_id = torch.argmin((self.timesteps - timestep).abs())
+ sigma = self.sigmas[timestep_id]
+ return sigma
+
+ def get_noise_term(self, sigma, sample):
+ # bridge noise term == 0 when sigma == 1 or 0
+ return self.bridge_noise_sigma * (sigma * (1.0 - sigma)) ** 0.5 * torch.randn_like(sample)
+
+ def step(self, model_output, timestep, sample, to_final=False, **kwargs):
+ if isinstance(timestep, torch.Tensor):
+ timestep = timestep.cpu()
+ timestep_id = torch.argmin((self.timesteps - timestep).abs())
+ sigma = self.sigmas[timestep_id]
+ if to_final or timestep_id + 1 >= len(self.timesteps):
+ sigma_ = 0
+ else:
+ sigma_ = self.sigmas[timestep_id + 1]
+
+ prev_sample = sample + model_output * (sigma_ - sigma) + self.get_noise_term(sigma_, sample)
+ return prev_sample
+
+ def add_noise(self, tgt_sample, src_sample, timestep):
+ sigma = self.retrieve_sigma(timestep)
+ noisy_sample = sigma * src_sample + (1 - sigma) * tgt_sample + self.get_noise_term(sigma, tgt_sample)
+ return noisy_sample
+
+ def training_target(self, tgt_sample, noisy_sample, timestep):
+ sigma = self.retrieve_sigma(timestep)
+
+ target = (noisy_sample - tgt_sample) / sigma
+ return target
+
+ def denoised_sample(self, prediction, noisy_sample, timestep):
+ sigma = self.retrieve_sigma(timestep)
+
+ sample = noisy_sample - prediction * sigma
+ return sample
+
+ def training_weight(self, timestep):
+ return 1.0
diff --git a/fdanyone/vendor/diffsynth/schedulers/flow_match.py b/fdanyone/vendor/diffsynth/schedulers/flow_match.py
new file mode 100644
index 0000000..0cd3e08
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/schedulers/flow_match.py
@@ -0,0 +1,125 @@
+import torch, math
+
+
+
+class FlowMatchScheduler():
+
+ def __init__(
+ self,
+ num_inference_steps=100,
+ num_train_timesteps=1000,
+ shift=3.0,
+ sigma_max=1.0,
+ sigma_min=0.003/1.002,
+ inverse_timesteps=False,
+ extra_one_step=False,
+ reverse_sigmas=False,
+ exponential_shift=False,
+ exponential_shift_mu=None,
+ shift_terminal=None,
+ ):
+ self.num_train_timesteps = num_train_timesteps
+ self.shift = shift
+ self.sigma_max = sigma_max
+ self.sigma_min = sigma_min
+ self.inverse_timesteps = inverse_timesteps
+ self.extra_one_step = extra_one_step
+ self.reverse_sigmas = reverse_sigmas
+ self.exponential_shift = exponential_shift
+ self.exponential_shift_mu = exponential_shift_mu
+ self.shift_terminal = shift_terminal
+ self.set_timesteps(num_inference_steps)
+
+
+ def set_timesteps(self, num_inference_steps=100, denoising_strength=1.0, training=False, shift=None, dynamic_shift_len=None):
+ if shift is not None:
+ self.shift = shift
+ sigma_start = self.sigma_min + (self.sigma_max - self.sigma_min) * denoising_strength
+ if self.extra_one_step:
+ self.sigmas = torch.linspace(sigma_start, self.sigma_min, num_inference_steps + 1)[:-1]
+ else:
+ self.sigmas = torch.linspace(sigma_start, self.sigma_min, num_inference_steps)
+ if self.inverse_timesteps:
+ self.sigmas = torch.flip(self.sigmas, dims=[0])
+ if self.exponential_shift:
+ mu = self.calculate_shift(dynamic_shift_len) if dynamic_shift_len is not None else self.exponential_shift_mu
+ self.sigmas = math.exp(mu) / (math.exp(mu) + (1 / self.sigmas - 1))
+ else:
+ self.sigmas = self.shift * self.sigmas / (1 + (self.shift - 1) * self.sigmas)
+ if self.shift_terminal is not None:
+ one_minus_z = 1 - self.sigmas
+ scale_factor = one_minus_z[-1] / (1 - self.shift_terminal)
+ self.sigmas = 1 - (one_minus_z / scale_factor)
+ if self.reverse_sigmas:
+ self.sigmas = 1 - self.sigmas
+ self.timesteps = self.sigmas * self.num_train_timesteps
+ if training:
+ x = self.timesteps
+ y = torch.exp(-2 * ((x - num_inference_steps / 2) / num_inference_steps) ** 2)
+ y_shifted = y - y.min()
+ bsmntw_weighing = y_shifted * (num_inference_steps / y_shifted.sum())
+ self.linear_timesteps_weights = bsmntw_weighing
+ self.training = True
+ else:
+ self.training = False
+
+
+ def step(self, model_output, timestep, sample, to_final=False, **kwargs):
+ if isinstance(timestep, torch.Tensor):
+ timestep = timestep.cpu()
+ timestep_id = torch.argmin((self.timesteps - timestep).abs())
+ sigma = self.sigmas[timestep_id]
+ if to_final or timestep_id + 1 >= len(self.timesteps):
+ sigma_ = 1 if (self.inverse_timesteps or self.reverse_sigmas) else 0
+ else:
+ sigma_ = self.sigmas[timestep_id + 1]
+ prev_sample = sample + model_output * (sigma_ - sigma)
+ return prev_sample
+
+
+ def return_to_timestep(self, timestep, sample, sample_stablized):
+ if isinstance(timestep, torch.Tensor):
+ timestep = timestep.cpu()
+ timestep_id = torch.argmin((self.timesteps - timestep).abs())
+ sigma = self.sigmas[timestep_id]
+ model_output = (sample - sample_stablized) / sigma
+ return model_output
+
+
+ def add_noise(self, original_samples, noise, timestep):
+ if isinstance(timestep, torch.Tensor):
+ timestep = timestep.cpu()
+ timestep_id = torch.argmin((self.timesteps - timestep).abs())
+ sigma = self.sigmas[timestep_id]
+ sample = (1 - sigma) * original_samples + sigma * noise
+ return sample
+
+
+ def training_target(self, sample, noise, timestep):
+ target = noise - sample
+ return target
+
+
+ def denoised_sample(self, prediction, noise, timestep):
+ sample = noise - prediction
+ return sample
+
+
+ def training_weight(self, timestep):
+ timestep_id = torch.argmin((self.timesteps - timestep.to(self.timesteps.device)).abs())
+ weights = self.linear_timesteps_weights[timestep_id]
+ return weights
+
+
+ def calculate_shift(
+ self,
+ image_seq_len,
+ base_seq_len: int = 256,
+ max_seq_len: int = 8192,
+ base_shift: float = 0.5,
+ max_shift: float = 0.9,
+ ):
+ m = (max_shift - base_shift) / (max_seq_len - base_seq_len)
+ b = base_shift - m * base_seq_len
+ mu = image_seq_len * m + b
+ return mu
diff --git a/fdanyone/vendor/diffsynth/vram_management/__init__.py b/fdanyone/vendor/diffsynth/vram_management/__init__.py
new file mode 100644
index 0000000..5b07580
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/vram_management/__init__.py
@@ -0,0 +1,2 @@
+from .layers import *
+from .gradient_checkpointing import *
diff --git a/fdanyone/vendor/diffsynth/vram_management/gradient_checkpointing.py b/fdanyone/vendor/diffsynth/vram_management/gradient_checkpointing.py
new file mode 100644
index 0000000..b356415
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/vram_management/gradient_checkpointing.py
@@ -0,0 +1,34 @@
+import torch
+
+
+def create_custom_forward(module):
+ def custom_forward(*inputs, **kwargs):
+ return module(*inputs, **kwargs)
+ return custom_forward
+
+
+def gradient_checkpoint_forward(
+ model,
+ use_gradient_checkpointing,
+ use_gradient_checkpointing_offload,
+ *args,
+ **kwargs,
+):
+ if use_gradient_checkpointing_offload:
+ with torch.autograd.graph.save_on_cpu():
+ model_output = torch.utils.checkpoint.checkpoint(
+ create_custom_forward(model),
+ *args,
+ **kwargs,
+ use_reentrant=False,
+ )
+ elif use_gradient_checkpointing:
+ model_output = torch.utils.checkpoint.checkpoint(
+ create_custom_forward(model),
+ *args,
+ **kwargs,
+ use_reentrant=False,
+ )
+ else:
+ model_output = model(*args, **kwargs)
+ return model_output
diff --git a/fdanyone/vendor/diffsynth/vram_management/layers.py b/fdanyone/vendor/diffsynth/vram_management/layers.py
new file mode 100644
index 0000000..c6f1ec6
--- /dev/null
+++ b/fdanyone/vendor/diffsynth/vram_management/layers.py
@@ -0,0 +1,213 @@
+import torch, copy
+from ..models.utils import init_weights_on_device
+
+
+def cast_to(weight, dtype, device):
+ r = torch.empty_like(weight, dtype=dtype, device=device)
+ r.copy_(weight)
+ return r
+
+
+class AutoTorchModule(torch.nn.Module):
+ def __init__(self):
+ super().__init__()
+
+ def check_free_vram(self):
+ gpu_mem_state = torch.cuda.mem_get_info(self.computation_device)
+ used_memory = (gpu_mem_state[1] - gpu_mem_state[0]) / (1024 ** 3)
+ return used_memory < self.vram_limit
+
+ def offload(self):
+ if self.state != 0:
+ self.to(dtype=self.offload_dtype, device=self.offload_device)
+ self.state = 0
+
+ def onload(self):
+ if self.state != 1:
+ self.to(dtype=self.onload_dtype, device=self.onload_device)
+ self.state = 1
+
+ def keep(self):
+ if self.state != 2:
+ self.to(dtype=self.computation_dtype, device=self.computation_device)
+ self.state = 2
+
+
+class AutoWrappedModule(AutoTorchModule):
+ def __init__(self, module: torch.nn.Module, offload_dtype, offload_device, onload_dtype, onload_device, computation_dtype, computation_device, vram_limit, **kwargs):
+ super().__init__()
+ self.module = module.to(dtype=offload_dtype, device=offload_device)
+ self.offload_dtype = offload_dtype
+ self.offload_device = offload_device
+ self.onload_dtype = onload_dtype
+ self.onload_device = onload_device
+ self.computation_dtype = computation_dtype
+ self.computation_device = computation_device
+ self.vram_limit = vram_limit
+ self.state = 0
+
+ def forward(self, *args, **kwargs):
+ if self.state == 2:
+ module = self.module
+ else:
+ if self.onload_dtype == self.computation_dtype and self.onload_device == self.computation_device:
+ module = self.module
+ elif self.vram_limit is not None and self.check_free_vram():
+ self.keep()
+ module = self.module
+ else:
+ module = copy.deepcopy(self.module).to(dtype=self.computation_dtype, device=self.computation_device)
+ return module(*args, **kwargs)
+
+
+class WanAutoCastLayerNorm(torch.nn.LayerNorm, AutoTorchModule):
+ def __init__(self, module: torch.nn.LayerNorm, offload_dtype, offload_device, onload_dtype, onload_device, computation_dtype, computation_device, vram_limit, **kwargs):
+ with init_weights_on_device(device=torch.device("meta")):
+ super().__init__(module.normalized_shape, eps=module.eps, elementwise_affine=module.elementwise_affine, bias=module.bias is not None, dtype=offload_dtype, device=offload_device)
+ self.weight = module.weight
+ self.bias = module.bias
+ self.offload_dtype = offload_dtype
+ self.offload_device = offload_device
+ self.onload_dtype = onload_dtype
+ self.onload_device = onload_device
+ self.computation_dtype = computation_dtype
+ self.computation_device = computation_device
+ self.vram_limit = vram_limit
+ self.state = 0
+
+ def forward(self, x, *args, **kwargs):
+ if self.state == 2:
+ weight, bias = self.weight, self.bias
+ else:
+ if self.onload_dtype == self.computation_dtype and self.onload_device == self.computation_device:
+ weight, bias = self.weight, self.bias
+ elif self.vram_limit is not None and self.check_free_vram():
+ self.keep()
+ weight, bias = self.weight, self.bias
+ else:
+ weight = None if self.weight is None else cast_to(self.weight, self.computation_dtype, self.computation_device)
+ bias = None if self.bias is None else cast_to(self.bias, self.computation_dtype, self.computation_device)
+ with torch.amp.autocast(device_type=x.device.type):
+ x = torch.nn.functional.layer_norm(x.float(), self.normalized_shape, weight, bias, self.eps).type_as(x)
+ return x
+
+
+class AutoWrappedLinear(torch.nn.Linear, AutoTorchModule):
+ def __init__(self, module: torch.nn.Linear, offload_dtype, offload_device, onload_dtype, onload_device, computation_dtype, computation_device, vram_limit, name="", **kwargs):
+ with init_weights_on_device(device=torch.device("meta")):
+ super().__init__(in_features=module.in_features, out_features=module.out_features, bias=module.bias is not None, dtype=offload_dtype, device=offload_device)
+ self.weight = module.weight
+ self.bias = module.bias
+ self.offload_dtype = offload_dtype
+ self.offload_device = offload_device
+ self.onload_dtype = onload_dtype
+ self.onload_device = onload_device
+ self.computation_dtype = computation_dtype
+ self.computation_device = computation_device
+ self.vram_limit = vram_limit
+ self.state = 0
+ self.name = name
+ self.lora_A_weights = []
+ self.lora_B_weights = []
+ self.lora_merger = None
+ self.enable_fp8 = computation_dtype in [torch.float8_e4m3fn, torch.float8_e4m3fnuz]
+
+ def fp8_linear(
+ self,
+ input: torch.Tensor,
+ weight: torch.Tensor,
+ bias: torch.Tensor | None = None,
+ ) -> torch.Tensor:
+ device = input.device
+ origin_dtype = input.dtype
+ origin_shape = input.shape
+ input = input.reshape(-1, origin_shape[-1])
+
+ x_max = torch.max(torch.abs(input), dim=-1, keepdim=True).values
+ fp8_max = 448.0
+ # For float8_e4m3fnuz, the maximum representable value is half of that of e4m3fn.
+ # To avoid overflow and ensure numerical compatibility during FP8 computation,
+ # we scale down the input by 2.0 in advance.
+ # This scaling will be compensated later during the final result scaling.
+ if self.computation_dtype == torch.float8_e4m3fnuz:
+ fp8_max = fp8_max / 2.0
+ scale_a = torch.clamp(x_max / fp8_max, min=1.0).float().to(device=device)
+ scale_b = torch.ones((weight.shape[0], 1)).to(device=device)
+ input = input / (scale_a + 1e-8)
+ input = input.to(self.computation_dtype)
+ weight = weight.to(self.computation_dtype)
+ bias = bias.to(torch.bfloat16)
+
+ result = torch._scaled_mm(
+ input,
+ weight.T,
+ scale_a=scale_a,
+ scale_b=scale_b.T,
+ bias=bias,
+ out_dtype=origin_dtype,
+ )
+ new_shape = origin_shape[:-1] + result.shape[-1:]
+ result = result.reshape(new_shape)
+ return result
+
+ def forward(self, x, *args, **kwargs):
+ # VRAM management
+ if self.state == 2:
+ weight, bias = self.weight, self.bias
+ else:
+ if self.onload_dtype == self.computation_dtype and self.onload_device == self.computation_device:
+ weight, bias = self.weight, self.bias
+ elif self.vram_limit is not None and self.check_free_vram():
+ self.keep()
+ weight, bias = self.weight, self.bias
+ else:
+ weight = cast_to(self.weight, self.computation_dtype, self.computation_device)
+ bias = None if self.bias is None else cast_to(self.bias, self.computation_dtype, self.computation_device)
+
+ # Linear forward
+ if self.enable_fp8:
+ out = self.fp8_linear(x, weight, bias)
+ else:
+ out = torch.nn.functional.linear(x, weight, bias)
+
+ # LoRA
+ if len(self.lora_A_weights) == 0:
+ # No LoRA
+ return out
+ elif self.lora_merger is None:
+ # Native LoRA inference
+ for lora_A, lora_B in zip(self.lora_A_weights, self.lora_B_weights):
+ out = out + x @ lora_A.T @ lora_B.T
+ else:
+ # LoRA fusion
+ lora_output = []
+ for lora_A, lora_B in zip(self.lora_A_weights, self.lora_B_weights):
+ lora_output.append(x @ lora_A.T @ lora_B.T)
+ lora_output = torch.stack(lora_output)
+ out = self.lora_merger(out, lora_output)
+ return out
+
+
+def enable_vram_management_recursively(model: torch.nn.Module, module_map: dict, module_config: dict, max_num_param=None, overflow_module_config: dict = None, total_num_param=0, vram_limit=None, name_prefix=""):
+ for name, module in model.named_children():
+ layer_name = name if name_prefix == "" else name_prefix + "." + name
+ for source_module, target_module in module_map.items():
+ if isinstance(module, source_module):
+ num_param = sum(p.numel() for p in module.parameters())
+ if max_num_param is not None and total_num_param + num_param > max_num_param:
+ module_config_ = overflow_module_config
+ else:
+ module_config_ = module_config
+ module_ = target_module(module, **module_config_, vram_limit=vram_limit, name=layer_name)
+ setattr(model, name, module_)
+ total_num_param += num_param
+ break
+ else:
+ total_num_param = enable_vram_management_recursively(module, module_map, module_config, max_num_param, overflow_module_config, total_num_param, vram_limit=vram_limit, name_prefix=layer_name)
+ return total_num_param
+
+
+def enable_vram_management(model: torch.nn.Module, module_map: dict, module_config: dict, max_num_param=None, overflow_module_config: dict = None, vram_limit=None):
+ enable_vram_management_recursively(model, module_map, module_config, max_num_param, overflow_module_config, total_num_param=0, vram_limit=vram_limit)
+ model.vram_management_enabled = True
+
diff --git a/fdanyone/vendor/pytorch3d_compat/LICENSE b/fdanyone/vendor/pytorch3d_compat/LICENSE
new file mode 100644
index 0000000..c55382f
--- /dev/null
+++ b/fdanyone/vendor/pytorch3d_compat/LICENSE
@@ -0,0 +1,30 @@
+BSD License
+
+For PyTorch3D software
+
+Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+ * Neither the name Meta nor the names of its contributors may be used to
+ endorse or promote products derived from this software without specific
+ prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/fdanyone/vendor/pytorch3d_compat/UPSTREAM.md b/fdanyone/vendor/pytorch3d_compat/UPSTREAM.md
new file mode 100644
index 0000000..0945964
--- /dev/null
+++ b/fdanyone/vendor/pytorch3d_compat/UPSTREAM.md
@@ -0,0 +1,8 @@
+# PyTorch3D compatibility subset
+
+The two retained source files are copied byte-for-byte from [`facebookresearch/pytorch3d@f34104c`](https://github.com/facebookresearch/pytorch3d/tree/f34104cf6ebefacd7b7e07955ee7aaa823e616ac) (release `v0.7.6`):
+
+- `common/datatypes.py`
+- `transforms/rotation_conversions.py`
+
+PyTorch3D is BSD-licensed; its license is preserved in `LICENSE`. The smaller `__init__.py` files and native-PyTorch KNN fallback are 4DAnyone adapter code. They expose only the rotation/geometry surface imported by classic GVHMR inference and intentionally omit PyTorch3D's renderer and compiled operators.
diff --git a/fdanyone/vendor/pytorch3d_compat/VENDORED_FILES.txt b/fdanyone/vendor/pytorch3d_compat/VENDORED_FILES.txt
new file mode 100644
index 0000000..dc0a7ab
--- /dev/null
+++ b/fdanyone/vendor/pytorch3d_compat/VENDORED_FILES.txt
@@ -0,0 +1,11 @@
+# Complete file list for this compatibility subset.
+LICENSE
+UPSTREAM.md
+VENDORED_FILES.txt
+__init__.py
+common/__init__.py
+common/datatypes.py
+ops/__init__.py
+ops/knn.py
+transforms/__init__.py
+transforms/rotation_conversions.py
diff --git a/fdanyone/vendor/pytorch3d_compat/__init__.py b/fdanyone/vendor/pytorch3d_compat/__init__.py
new file mode 100644
index 0000000..ab7a327
--- /dev/null
+++ b/fdanyone/vendor/pytorch3d_compat/__init__.py
@@ -0,0 +1,38 @@
+"""Minimal PyTorch3D 0.7.6 compatibility surface for classic GVHMR.
+
+The official GVHMR inference path needs rotation conversions but imports the
+compiled PyTorch3D package through broader training/demo modules. 4DAnyone
+registers this BSD-licensed, inference-only subset when full PyTorch3D is not
+installed, allowing GVHMR and generation to share the same modern PyTorch
+environment.
+"""
+
+from __future__ import annotations
+
+import importlib
+import importlib.util
+import sys
+
+
+def install_if_needed() -> bool:
+ """Expose the compatibility modules as ``pytorch3d`` when it is absent."""
+
+ if importlib.util.find_spec("pytorch3d") is not None:
+ return False
+
+ root = sys.modules[__name__]
+ transforms = importlib.import_module(f"{__name__}.transforms")
+ ops = importlib.import_module(f"{__name__}.ops")
+ knn = importlib.import_module(f"{__name__}.ops.knn")
+ sys.modules.update(
+ {
+ "pytorch3d": root,
+ "pytorch3d.transforms": transforms,
+ "pytorch3d.ops": ops,
+ "pytorch3d.ops.knn": knn,
+ }
+ )
+ return True
+
+
+__all__ = ["install_if_needed"]
diff --git a/fdanyone/vendor/pytorch3d_compat/common/__init__.py b/fdanyone/vendor/pytorch3d_compat/common/__init__.py
new file mode 100644
index 0000000..818c53c
--- /dev/null
+++ b/fdanyone/vendor/pytorch3d_compat/common/__init__.py
@@ -0,0 +1,5 @@
+"""Small type helpers retained from PyTorch3D 0.7.6."""
+
+from .datatypes import Device, get_device, make_device
+
+__all__ = ["Device", "get_device", "make_device"]
diff --git a/fdanyone/vendor/pytorch3d_compat/common/datatypes.py b/fdanyone/vendor/pytorch3d_compat/common/datatypes.py
new file mode 100644
index 0000000..03fe3ef
--- /dev/null
+++ b/fdanyone/vendor/pytorch3d_compat/common/datatypes.py
@@ -0,0 +1,58 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+from typing import Optional, Union
+
+import torch
+
+
+Device = Union[str, torch.device]
+
+
+def make_device(device: Device) -> torch.device:
+ """
+ Makes an actual torch.device object from the device specified as
+ either a string or torch.device object. If the device is `cuda` without
+ a specific index, the index of the current device is assigned.
+
+ Args:
+ device: Device (as str or torch.device)
+
+ Returns:
+ A matching torch.device object
+ """
+ device = torch.device(device) if isinstance(device, str) else device
+ if device.type == "cuda" and device.index is None:
+ # If cuda but with no index, then the current cuda device is indicated.
+ # In that case, we fix to that device
+ device = torch.device(f"cuda:{torch.cuda.current_device()}")
+ return device
+
+
+def get_device(x, device: Optional[Device] = None) -> torch.device:
+ """
+ Gets the device of the specified variable x if it is a tensor, or
+ falls back to a default CPU device otherwise. Allows overriding by
+ providing an explicit device.
+
+ Args:
+ x: a torch.Tensor to get the device from or another type
+ device: Device (as str or torch.device) to fall back to
+
+ Returns:
+ A matching torch.device object
+ """
+
+ # User overrides device
+ if device is not None:
+ return make_device(device)
+
+ # Set device based on input tensor
+ if torch.is_tensor(x):
+ return x.device
+
+ # Default device is cpu
+ return torch.device("cpu")
diff --git a/fdanyone/vendor/pytorch3d_compat/ops/__init__.py b/fdanyone/vendor/pytorch3d_compat/ops/__init__.py
new file mode 100644
index 0000000..a6d8c6f
--- /dev/null
+++ b/fdanyone/vendor/pytorch3d_compat/ops/__init__.py
@@ -0,0 +1,5 @@
+"""Native-PyTorch operations needed to import classic GVHMR."""
+
+from . import knn
+
+__all__ = ["knn"]
diff --git a/fdanyone/vendor/pytorch3d_compat/ops/knn.py b/fdanyone/vendor/pytorch3d_compat/ops/knn.py
new file mode 100644
index 0000000..8663614
--- /dev/null
+++ b/fdanyone/vendor/pytorch3d_compat/ops/knn.py
@@ -0,0 +1,22 @@
+"""Inference-only fallback for the one optional GVHMR KNN helper."""
+
+from __future__ import annotations
+
+
+def knn_points(p1, p2, *, K: int = 1, return_nn: bool = False):
+ """Return the PyTorch3D-compatible tuple using native PyTorch operations."""
+
+ import torch
+
+ if p1.ndim != 3 or p2.ndim != 3 or p1.shape[0] != p2.shape[0]:
+ raise ValueError("p1 and p2 must have shapes (N, P, D) and (N, Q, D).")
+ if K < 1 or p2.shape[1] < K:
+ raise ValueError(f"K must be in [1, {p2.shape[1]}], got {K}.")
+ distances = torch.cdist(p1, p2).square()
+ squared_distances, indices = distances.topk(K, dim=-1, largest=False, sorted=True)
+ neighbors = None
+ if return_nn:
+ expanded = p2[:, None].expand(-1, p1.shape[1], -1, -1)
+ gather_index = indices[..., None].expand(-1, -1, -1, p2.shape[-1])
+ neighbors = torch.gather(expanded, 2, gather_index)
+ return squared_distances, indices, neighbors
diff --git a/fdanyone/vendor/pytorch3d_compat/transforms/__init__.py b/fdanyone/vendor/pytorch3d_compat/transforms/__init__.py
new file mode 100644
index 0000000..32dd82a
--- /dev/null
+++ b/fdanyone/vendor/pytorch3d_compat/transforms/__init__.py
@@ -0,0 +1,44 @@
+"""PyTorch3D 0.7.6 rotation conversions used by classic GVHMR inference."""
+
+from .rotation_conversions import (
+ axis_angle_to_matrix,
+ euler_angles_to_matrix,
+ matrix_to_axis_angle,
+ matrix_to_quaternion,
+ matrix_to_rotation_6d,
+ quaternion_to_axis_angle,
+ quaternion_to_matrix,
+ rotation_6d_to_matrix,
+)
+
+
+def so3_exp_map(log_rot, eps: float = 0.0001):
+ """Match the PyTorch3D 0.7.6 SO(3) exponential map used by GVHMR."""
+
+ del eps
+ if log_rot.ndim != 2 or log_rot.shape[1] != 3:
+ raise ValueError("Input tensor shape has to be Nx3.")
+ return axis_angle_to_matrix(log_rot)
+
+
+def so3_log_map(rotation, eps: float = 0.0001, cos_bound: float = 1e-4):
+ """Match the PyTorch3D 0.7.6 SO(3) logarithm used by GVHMR."""
+
+ del eps, cos_bound
+ if rotation.ndim != 3 or rotation.shape[1:] != (3, 3):
+ raise ValueError("Input has to be a batch of 3x3 Tensors.")
+ return matrix_to_axis_angle(rotation)
+
+
+__all__ = [
+ "axis_angle_to_matrix",
+ "euler_angles_to_matrix",
+ "matrix_to_axis_angle",
+ "matrix_to_quaternion",
+ "matrix_to_rotation_6d",
+ "quaternion_to_axis_angle",
+ "quaternion_to_matrix",
+ "rotation_6d_to_matrix",
+ "so3_exp_map",
+ "so3_log_map",
+]
diff --git a/fdanyone/vendor/pytorch3d_compat/transforms/rotation_conversions.py b/fdanyone/vendor/pytorch3d_compat/transforms/rotation_conversions.py
new file mode 100644
index 0000000..459441c
--- /dev/null
+++ b/fdanyone/vendor/pytorch3d_compat/transforms/rotation_conversions.py
@@ -0,0 +1,596 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+from typing import Optional
+
+import torch
+import torch.nn.functional as F
+
+from ..common.datatypes import Device
+
+
+"""
+The transformation matrices returned from the functions in this file assume
+the points on which the transformation will be applied are column vectors.
+i.e. the R matrix is structured as
+
+ R = [
+ [Rxx, Rxy, Rxz],
+ [Ryx, Ryy, Ryz],
+ [Rzx, Rzy, Rzz],
+ ] # (3, 3)
+
+This matrix can be applied to column vectors by post multiplication
+by the points e.g.
+
+ points = [[0], [1], [2]] # (3 x 1) xyz coordinates of a point
+ transformed_points = R * points
+
+To apply the same matrix to points which are row vectors, the R matrix
+can be transposed and pre multiplied by the points:
+
+e.g.
+ points = [[0, 1, 2]] # (1 x 3) xyz coordinates of a point
+ transformed_points = points * R.transpose(1, 0)
+"""
+
+
+def quaternion_to_matrix(quaternions: torch.Tensor) -> torch.Tensor:
+ """
+ Convert rotations given as quaternions to rotation matrices.
+
+ Args:
+ quaternions: quaternions with real part first,
+ as tensor of shape (..., 4).
+
+ Returns:
+ Rotation matrices as tensor of shape (..., 3, 3).
+ """
+ r, i, j, k = torch.unbind(quaternions, -1)
+ # pyre-fixme[58]: `/` is not supported for operand types `float` and `Tensor`.
+ two_s = 2.0 / (quaternions * quaternions).sum(-1)
+
+ o = torch.stack(
+ (
+ 1 - two_s * (j * j + k * k),
+ two_s * (i * j - k * r),
+ two_s * (i * k + j * r),
+ two_s * (i * j + k * r),
+ 1 - two_s * (i * i + k * k),
+ two_s * (j * k - i * r),
+ two_s * (i * k - j * r),
+ two_s * (j * k + i * r),
+ 1 - two_s * (i * i + j * j),
+ ),
+ -1,
+ )
+ return o.reshape(quaternions.shape[:-1] + (3, 3))
+
+
+def _copysign(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
+ """
+ Return a tensor where each element has the absolute value taken from the,
+ corresponding element of a, with sign taken from the corresponding
+ element of b. This is like the standard copysign floating-point operation,
+ but is not careful about negative 0 and NaN.
+
+ Args:
+ a: source tensor.
+ b: tensor whose signs will be used, of the same shape as a.
+
+ Returns:
+ Tensor of the same shape as a with the signs of b.
+ """
+ signs_differ = (a < 0) != (b < 0)
+ return torch.where(signs_differ, -a, a)
+
+
+def _sqrt_positive_part(x: torch.Tensor) -> torch.Tensor:
+ """
+ Returns torch.sqrt(torch.max(0, x))
+ but with a zero subgradient where x is 0.
+ """
+ ret = torch.zeros_like(x)
+ positive_mask = x > 0
+ ret[positive_mask] = torch.sqrt(x[positive_mask])
+ return ret
+
+
+def matrix_to_quaternion(matrix: torch.Tensor) -> torch.Tensor:
+ """
+ Convert rotations given as rotation matrices to quaternions.
+
+ Args:
+ matrix: Rotation matrices as tensor of shape (..., 3, 3).
+
+ Returns:
+ quaternions with real part first, as tensor of shape (..., 4).
+ """
+ if matrix.size(-1) != 3 or matrix.size(-2) != 3:
+ raise ValueError(f"Invalid rotation matrix shape {matrix.shape}.")
+
+ batch_dim = matrix.shape[:-2]
+ m00, m01, m02, m10, m11, m12, m20, m21, m22 = torch.unbind(
+ matrix.reshape(batch_dim + (9,)), dim=-1
+ )
+
+ q_abs = _sqrt_positive_part(
+ torch.stack(
+ [
+ 1.0 + m00 + m11 + m22,
+ 1.0 + m00 - m11 - m22,
+ 1.0 - m00 + m11 - m22,
+ 1.0 - m00 - m11 + m22,
+ ],
+ dim=-1,
+ )
+ )
+
+ # we produce the desired quaternion multiplied by each of r, i, j, k
+ quat_by_rijk = torch.stack(
+ [
+ # pyre-fixme[58]: `**` is not supported for operand types `Tensor` and
+ # `int`.
+ torch.stack([q_abs[..., 0] ** 2, m21 - m12, m02 - m20, m10 - m01], dim=-1),
+ # pyre-fixme[58]: `**` is not supported for operand types `Tensor` and
+ # `int`.
+ torch.stack([m21 - m12, q_abs[..., 1] ** 2, m10 + m01, m02 + m20], dim=-1),
+ # pyre-fixme[58]: `**` is not supported for operand types `Tensor` and
+ # `int`.
+ torch.stack([m02 - m20, m10 + m01, q_abs[..., 2] ** 2, m12 + m21], dim=-1),
+ # pyre-fixme[58]: `**` is not supported for operand types `Tensor` and
+ # `int`.
+ torch.stack([m10 - m01, m20 + m02, m21 + m12, q_abs[..., 3] ** 2], dim=-1),
+ ],
+ dim=-2,
+ )
+
+ # We floor here at 0.1 but the exact level is not important; if q_abs is small,
+ # the candidate won't be picked.
+ flr = torch.tensor(0.1).to(dtype=q_abs.dtype, device=q_abs.device)
+ quat_candidates = quat_by_rijk / (2.0 * q_abs[..., None].max(flr))
+
+ # if not for numerical problems, quat_candidates[i] should be same (up to a sign),
+ # forall i; we pick the best-conditioned one (with the largest denominator)
+ out = quat_candidates[
+ F.one_hot(q_abs.argmax(dim=-1), num_classes=4) > 0.5, :
+ ].reshape(batch_dim + (4,))
+ return standardize_quaternion(out)
+
+
+def _axis_angle_rotation(axis: str, angle: torch.Tensor) -> torch.Tensor:
+ """
+ Return the rotation matrices for one of the rotations about an axis
+ of which Euler angles describe, for each value of the angle given.
+
+ Args:
+ axis: Axis label "X" or "Y or "Z".
+ angle: any shape tensor of Euler angles in radians
+
+ Returns:
+ Rotation matrices as tensor of shape (..., 3, 3).
+ """
+
+ cos = torch.cos(angle)
+ sin = torch.sin(angle)
+ one = torch.ones_like(angle)
+ zero = torch.zeros_like(angle)
+
+ if axis == "X":
+ R_flat = (one, zero, zero, zero, cos, -sin, zero, sin, cos)
+ elif axis == "Y":
+ R_flat = (cos, zero, sin, zero, one, zero, -sin, zero, cos)
+ elif axis == "Z":
+ R_flat = (cos, -sin, zero, sin, cos, zero, zero, zero, one)
+ else:
+ raise ValueError("letter must be either X, Y or Z.")
+
+ return torch.stack(R_flat, -1).reshape(angle.shape + (3, 3))
+
+
+def euler_angles_to_matrix(euler_angles: torch.Tensor, convention: str) -> torch.Tensor:
+ """
+ Convert rotations given as Euler angles in radians to rotation matrices.
+
+ Args:
+ euler_angles: Euler angles in radians as tensor of shape (..., 3).
+ convention: Convention string of three uppercase letters from
+ {"X", "Y", and "Z"}.
+
+ Returns:
+ Rotation matrices as tensor of shape (..., 3, 3).
+ """
+ if euler_angles.dim() == 0 or euler_angles.shape[-1] != 3:
+ raise ValueError("Invalid input euler angles.")
+ if len(convention) != 3:
+ raise ValueError("Convention must have 3 letters.")
+ if convention[1] in (convention[0], convention[2]):
+ raise ValueError(f"Invalid convention {convention}.")
+ for letter in convention:
+ if letter not in ("X", "Y", "Z"):
+ raise ValueError(f"Invalid letter {letter} in convention string.")
+ matrices = [
+ _axis_angle_rotation(c, e)
+ for c, e in zip(convention, torch.unbind(euler_angles, -1))
+ ]
+ # return functools.reduce(torch.matmul, matrices)
+ return torch.matmul(torch.matmul(matrices[0], matrices[1]), matrices[2])
+
+
+def _angle_from_tan(
+ axis: str, other_axis: str, data, horizontal: bool, tait_bryan: bool
+) -> torch.Tensor:
+ """
+ Extract the first or third Euler angle from the two members of
+ the matrix which are positive constant times its sine and cosine.
+
+ Args:
+ axis: Axis label "X" or "Y or "Z" for the angle we are finding.
+ other_axis: Axis label "X" or "Y or "Z" for the middle axis in the
+ convention.
+ data: Rotation matrices as tensor of shape (..., 3, 3).
+ horizontal: Whether we are looking for the angle for the third axis,
+ which means the relevant entries are in the same row of the
+ rotation matrix. If not, they are in the same column.
+ tait_bryan: Whether the first and third axes in the convention differ.
+
+ Returns:
+ Euler Angles in radians for each matrix in data as a tensor
+ of shape (...).
+ """
+
+ i1, i2 = {"X": (2, 1), "Y": (0, 2), "Z": (1, 0)}[axis]
+ if horizontal:
+ i2, i1 = i1, i2
+ even = (axis + other_axis) in ["XY", "YZ", "ZX"]
+ if horizontal == even:
+ return torch.atan2(data[..., i1], data[..., i2])
+ if tait_bryan:
+ return torch.atan2(-data[..., i2], data[..., i1])
+ return torch.atan2(data[..., i2], -data[..., i1])
+
+
+def _index_from_letter(letter: str) -> int:
+ if letter == "X":
+ return 0
+ if letter == "Y":
+ return 1
+ if letter == "Z":
+ return 2
+ raise ValueError("letter must be either X, Y or Z.")
+
+
+def matrix_to_euler_angles(matrix: torch.Tensor, convention: str) -> torch.Tensor:
+ """
+ Convert rotations given as rotation matrices to Euler angles in radians.
+
+ Args:
+ matrix: Rotation matrices as tensor of shape (..., 3, 3).
+ convention: Convention string of three uppercase letters.
+
+ Returns:
+ Euler angles in radians as tensor of shape (..., 3).
+ """
+ if len(convention) != 3:
+ raise ValueError("Convention must have 3 letters.")
+ if convention[1] in (convention[0], convention[2]):
+ raise ValueError(f"Invalid convention {convention}.")
+ for letter in convention:
+ if letter not in ("X", "Y", "Z"):
+ raise ValueError(f"Invalid letter {letter} in convention string.")
+ if matrix.size(-1) != 3 or matrix.size(-2) != 3:
+ raise ValueError(f"Invalid rotation matrix shape {matrix.shape}.")
+ i0 = _index_from_letter(convention[0])
+ i2 = _index_from_letter(convention[2])
+ tait_bryan = i0 != i2
+ if tait_bryan:
+ central_angle = torch.asin(
+ matrix[..., i0, i2] * (-1.0 if i0 - i2 in [-1, 2] else 1.0)
+ )
+ else:
+ central_angle = torch.acos(matrix[..., i0, i0])
+
+ o = (
+ _angle_from_tan(
+ convention[0], convention[1], matrix[..., i2], False, tait_bryan
+ ),
+ central_angle,
+ _angle_from_tan(
+ convention[2], convention[1], matrix[..., i0, :], True, tait_bryan
+ ),
+ )
+ return torch.stack(o, -1)
+
+
+def random_quaternions(
+ n: int, dtype: Optional[torch.dtype] = None, device: Optional[Device] = None
+) -> torch.Tensor:
+ """
+ Generate random quaternions representing rotations,
+ i.e. versors with nonnegative real part.
+
+ Args:
+ n: Number of quaternions in a batch to return.
+ dtype: Type to return.
+ device: Desired device of returned tensor. Default:
+ uses the current device for the default tensor type.
+
+ Returns:
+ Quaternions as tensor of shape (N, 4).
+ """
+ if isinstance(device, str):
+ device = torch.device(device)
+ o = torch.randn((n, 4), dtype=dtype, device=device)
+ s = (o * o).sum(1)
+ o = o / _copysign(torch.sqrt(s), o[:, 0])[:, None]
+ return o
+
+
+def random_rotations(
+ n: int, dtype: Optional[torch.dtype] = None, device: Optional[Device] = None
+) -> torch.Tensor:
+ """
+ Generate random rotations as 3x3 rotation matrices.
+
+ Args:
+ n: Number of rotation matrices in a batch to return.
+ dtype: Type to return.
+ device: Device of returned tensor. Default: if None,
+ uses the current device for the default tensor type.
+
+ Returns:
+ Rotation matrices as tensor of shape (n, 3, 3).
+ """
+ quaternions = random_quaternions(n, dtype=dtype, device=device)
+ return quaternion_to_matrix(quaternions)
+
+
+def random_rotation(
+ dtype: Optional[torch.dtype] = None, device: Optional[Device] = None
+) -> torch.Tensor:
+ """
+ Generate a single random 3x3 rotation matrix.
+
+ Args:
+ dtype: Type to return
+ device: Device of returned tensor. Default: if None,
+ uses the current device for the default tensor type
+
+ Returns:
+ Rotation matrix as tensor of shape (3, 3).
+ """
+ return random_rotations(1, dtype, device)[0]
+
+
+def standardize_quaternion(quaternions: torch.Tensor) -> torch.Tensor:
+ """
+ Convert a unit quaternion to a standard form: one in which the real
+ part is non negative.
+
+ Args:
+ quaternions: Quaternions with real part first,
+ as tensor of shape (..., 4).
+
+ Returns:
+ Standardized quaternions as tensor of shape (..., 4).
+ """
+ return torch.where(quaternions[..., 0:1] < 0, -quaternions, quaternions)
+
+
+def quaternion_raw_multiply(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
+ """
+ Multiply two quaternions.
+ Usual torch rules for broadcasting apply.
+
+ Args:
+ a: Quaternions as tensor of shape (..., 4), real part first.
+ b: Quaternions as tensor of shape (..., 4), real part first.
+
+ Returns:
+ The product of a and b, a tensor of quaternions shape (..., 4).
+ """
+ aw, ax, ay, az = torch.unbind(a, -1)
+ bw, bx, by, bz = torch.unbind(b, -1)
+ ow = aw * bw - ax * bx - ay * by - az * bz
+ ox = aw * bx + ax * bw + ay * bz - az * by
+ oy = aw * by - ax * bz + ay * bw + az * bx
+ oz = aw * bz + ax * by - ay * bx + az * bw
+ return torch.stack((ow, ox, oy, oz), -1)
+
+
+def quaternion_multiply(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
+ """
+ Multiply two quaternions representing rotations, returning the quaternion
+ representing their composition, i.e. the versor with nonnegative real part.
+ Usual torch rules for broadcasting apply.
+
+ Args:
+ a: Quaternions as tensor of shape (..., 4), real part first.
+ b: Quaternions as tensor of shape (..., 4), real part first.
+
+ Returns:
+ The product of a and b, a tensor of quaternions of shape (..., 4).
+ """
+ ab = quaternion_raw_multiply(a, b)
+ return standardize_quaternion(ab)
+
+
+def quaternion_invert(quaternion: torch.Tensor) -> torch.Tensor:
+ """
+ Given a quaternion representing rotation, get the quaternion representing
+ its inverse.
+
+ Args:
+ quaternion: Quaternions as tensor of shape (..., 4), with real part
+ first, which must be versors (unit quaternions).
+
+ Returns:
+ The inverse, a tensor of quaternions of shape (..., 4).
+ """
+
+ scaling = torch.tensor([1, -1, -1, -1], device=quaternion.device)
+ return quaternion * scaling
+
+
+def quaternion_apply(quaternion: torch.Tensor, point: torch.Tensor) -> torch.Tensor:
+ """
+ Apply the rotation given by a quaternion to a 3D point.
+ Usual torch rules for broadcasting apply.
+
+ Args:
+ quaternion: Tensor of quaternions, real part first, of shape (..., 4).
+ point: Tensor of 3D points of shape (..., 3).
+
+ Returns:
+ Tensor of rotated points of shape (..., 3).
+ """
+ if point.size(-1) != 3:
+ raise ValueError(f"Points are not in 3D, {point.shape}.")
+ real_parts = point.new_zeros(point.shape[:-1] + (1,))
+ point_as_quaternion = torch.cat((real_parts, point), -1)
+ out = quaternion_raw_multiply(
+ quaternion_raw_multiply(quaternion, point_as_quaternion),
+ quaternion_invert(quaternion),
+ )
+ return out[..., 1:]
+
+
+def axis_angle_to_matrix(axis_angle: torch.Tensor) -> torch.Tensor:
+ """
+ Convert rotations given as axis/angle to rotation matrices.
+
+ Args:
+ axis_angle: Rotations given as a vector in axis angle form,
+ as a tensor of shape (..., 3), where the magnitude is
+ the angle turned anticlockwise in radians around the
+ vector's direction.
+
+ Returns:
+ Rotation matrices as tensor of shape (..., 3, 3).
+ """
+ return quaternion_to_matrix(axis_angle_to_quaternion(axis_angle))
+
+
+def matrix_to_axis_angle(matrix: torch.Tensor) -> torch.Tensor:
+ """
+ Convert rotations given as rotation matrices to axis/angle.
+
+ Args:
+ matrix: Rotation matrices as tensor of shape (..., 3, 3).
+
+ Returns:
+ Rotations given as a vector in axis angle form, as a tensor
+ of shape (..., 3), where the magnitude is the angle
+ turned anticlockwise in radians around the vector's
+ direction.
+ """
+ return quaternion_to_axis_angle(matrix_to_quaternion(matrix))
+
+
+def axis_angle_to_quaternion(axis_angle: torch.Tensor) -> torch.Tensor:
+ """
+ Convert rotations given as axis/angle to quaternions.
+
+ Args:
+ axis_angle: Rotations given as a vector in axis angle form,
+ as a tensor of shape (..., 3), where the magnitude is
+ the angle turned anticlockwise in radians around the
+ vector's direction.
+
+ Returns:
+ quaternions with real part first, as tensor of shape (..., 4).
+ """
+ angles = torch.norm(axis_angle, p=2, dim=-1, keepdim=True)
+ half_angles = angles * 0.5
+ eps = 1e-6
+ small_angles = angles.abs() < eps
+ sin_half_angles_over_angles = torch.empty_like(angles)
+ sin_half_angles_over_angles[~small_angles] = (
+ torch.sin(half_angles[~small_angles]) / angles[~small_angles]
+ )
+ # for x small, sin(x/2) is about x/2 - (x/2)^3/6
+ # so sin(x/2)/x is about 1/2 - (x*x)/48
+ sin_half_angles_over_angles[small_angles] = (
+ 0.5 - (angles[small_angles] * angles[small_angles]) / 48
+ )
+ quaternions = torch.cat(
+ [torch.cos(half_angles), axis_angle * sin_half_angles_over_angles], dim=-1
+ )
+ return quaternions
+
+
+def quaternion_to_axis_angle(quaternions: torch.Tensor) -> torch.Tensor:
+ """
+ Convert rotations given as quaternions to axis/angle.
+
+ Args:
+ quaternions: quaternions with real part first,
+ as tensor of shape (..., 4).
+
+ Returns:
+ Rotations given as a vector in axis angle form, as a tensor
+ of shape (..., 3), where the magnitude is the angle
+ turned anticlockwise in radians around the vector's
+ direction.
+ """
+ norms = torch.norm(quaternions[..., 1:], p=2, dim=-1, keepdim=True)
+ half_angles = torch.atan2(norms, quaternions[..., :1])
+ angles = 2 * half_angles
+ eps = 1e-6
+ small_angles = angles.abs() < eps
+ sin_half_angles_over_angles = torch.empty_like(angles)
+ sin_half_angles_over_angles[~small_angles] = (
+ torch.sin(half_angles[~small_angles]) / angles[~small_angles]
+ )
+ # for x small, sin(x/2) is about x/2 - (x/2)^3/6
+ # so sin(x/2)/x is about 1/2 - (x*x)/48
+ sin_half_angles_over_angles[small_angles] = (
+ 0.5 - (angles[small_angles] * angles[small_angles]) / 48
+ )
+ return quaternions[..., 1:] / sin_half_angles_over_angles
+
+
+def rotation_6d_to_matrix(d6: torch.Tensor) -> torch.Tensor:
+ """
+ Converts 6D rotation representation by Zhou et al. [1] to rotation matrix
+ using Gram--Schmidt orthogonalization per Section B of [1].
+ Args:
+ d6: 6D rotation representation, of size (*, 6)
+
+ Returns:
+ batch of rotation matrices of size (*, 3, 3)
+
+ [1] Zhou, Y., Barnes, C., Lu, J., Yang, J., & Li, H.
+ On the Continuity of Rotation Representations in Neural Networks.
+ IEEE Conference on Computer Vision and Pattern Recognition, 2019.
+ Retrieved from http://arxiv.org/abs/1812.07035
+ """
+
+ a1, a2 = d6[..., :3], d6[..., 3:]
+ b1 = F.normalize(a1, dim=-1)
+ b2 = a2 - (b1 * a2).sum(-1, keepdim=True) * b1
+ b2 = F.normalize(b2, dim=-1)
+ b3 = torch.cross(b1, b2, dim=-1)
+ return torch.stack((b1, b2, b3), dim=-2)
+
+
+def matrix_to_rotation_6d(matrix: torch.Tensor) -> torch.Tensor:
+ """
+ Converts rotation matrices to 6D rotation representation by Zhou et al. [1]
+ by dropping the last row. Note that 6D representation is not unique.
+ Args:
+ matrix: batch of rotation matrices of size (*, 3, 3)
+
+ Returns:
+ 6D rotation representation, of size (*, 6)
+
+ [1] Zhou, Y., Barnes, C., Lu, J., Yang, J., & Li, H.
+ On the Continuity of Rotation Representations in Neural Networks.
+ IEEE Conference on Computer Vision and Pattern Recognition, 2019.
+ Retrieved from http://arxiv.org/abs/1812.07035
+ """
+ batch_dim = matrix.size()[:-2]
+ return matrix[..., :2, :].clone().reshape(batch_dim + (6,))
diff --git a/fdanyone/video.py b/fdanyone/video.py
new file mode 100644
index 0000000..6af9bcc
--- /dev/null
+++ b/fdanyone/video.py
@@ -0,0 +1,517 @@
+"""PTS-aware canonical video decoding and encoding."""
+
+from __future__ import annotations
+
+import itertools
+import json
+import math
+from collections.abc import Iterator
+from dataclasses import dataclass
+from fractions import Fraction
+from pathlib import Path
+
+import av
+import numpy as np
+
+from fdanyone.config import INFERENCE
+from fdanyone.errors import VideoContractError
+
+AUTO_DOWNSAMPLE_FPS = tuple(
+ Fraction(numerator, denominator) for numerator, denominator in INFERENCE.auto_downsample_fps
+)
+SAMPLING_POLICY = INFERENCE.temporal_sampling_policy
+INTEGER_RATIO_TOLERANCE = 1e-3
+
+
+def validate_required_video_codecs() -> None:
+ missing = []
+ for codec in ("libx264", "libx264rgb"):
+ try:
+ av.codec.Codec(codec, "w")
+ except (ValueError, av.FFmpegError):
+ missing.append(codec)
+ if missing:
+ raise VideoContractError(
+ f"The installed FFmpeg/PyAV build lacks required video encoders: {missing}. "
+ "Install a PyAV build with libx264 and libx264rgb support."
+ )
+
+
+def choose_canonical_fps(input_rate: Fraction) -> Fraction:
+ """Keep the source rate unless it supports clean integer downsampling.
+
+ High-frame-rate sources may be reduced to an exact 24/25/30-family
+ divisor (48 -> 24, 50 -> 25, 60 -> 30, 120 -> 30). Rates without such a
+ divisor, including 40 FPS, are preserved exactly.
+ """
+
+ if input_rate <= 0:
+ raise VideoContractError(f"Input frame rate must be positive, got {input_rate}.")
+
+ divisible: list[tuple[Fraction, int]] = []
+ for candidate in AUTO_DOWNSAMPLE_FPS:
+ ratio = float(input_rate / candidate)
+ multiple = round(ratio)
+ if multiple >= 2 and abs(ratio - multiple) <= INTEGER_RATIO_TOLERANCE:
+ divisible.append((candidate, multiple))
+ if divisible:
+ _, multiple = max(divisible, key=lambda match: float(match[0]))
+ return input_rate / multiple
+
+ return input_rate
+
+
+def _stream_rate(stream: av.video.stream.VideoStream) -> Fraction:
+ for value in (stream.average_rate, stream.guessed_rate, stream.base_rate):
+ if value is not None and value > 0:
+ return Fraction(value.numerator, value.denominator)
+ raise VideoContractError("The input video does not declare a usable frame rate.")
+
+
+def _normalize_rotation_degrees(raw: object) -> int:
+ """Normalize a display-matrix rotation to a supported CCW quarter turn."""
+
+ try:
+ rotation = int(round(float(raw))) % 360
+ except (TypeError, ValueError):
+ rotation = 0
+ if rotation not in (0, 90, 180, 270):
+ raise VideoContractError(f"Unsupported video rotation metadata: {raw!r} degrees.")
+ return rotation
+
+
+def _rotation_degrees(stream: av.video.stream.VideoStream) -> int:
+ return _normalize_rotation_degrees(stream.metadata.get("rotate", "0"))
+
+
+def _frame_rotation_degrees(frame: av.VideoFrame, metadata_rotation: int) -> int:
+ """Prefer FFmpeg display-matrix side data over the legacy rotate tag."""
+
+ frame_rotation = getattr(frame, "rotation", 0)
+ if frame_rotation is None or float(frame_rotation) == 0.0:
+ return metadata_rotation
+ return _normalize_rotation_degrees(frame_rotation)
+
+
+def _frame_timestamp(frame: av.VideoFrame, stream: av.video.stream.VideoStream, index: int) -> Fraction:
+ if frame.pts is not None and frame.time_base is not None:
+ return Fraction(frame.pts * frame.time_base)
+ rate = _stream_rate(stream)
+ return Fraction(index, 1) / rate
+
+
+def _frame_to_rgb(frame: av.VideoFrame, rotation_degrees: int) -> np.ndarray:
+ rgb = frame.to_ndarray(format="rgb24")
+ if rotation_degrees == 90:
+ rgb = np.rot90(rgb, k=1)
+ elif rotation_degrees == 180:
+ rgb = np.rot90(rgb, k=2)
+ elif rotation_degrees == 270:
+ rgb = np.rot90(rgb, k=3)
+ return np.ascontiguousarray(rgb)
+
+
+@dataclass(frozen=True)
+class CanonicalFrame:
+ rgb: np.ndarray
+ source_index: int
+ source_pts: int | None
+ source_timestamp: Fraction
+ canonical_timestamp: Fraction
+
+
+@dataclass(frozen=True)
+class CanonicalClip:
+ source_path: Path
+ source_size_bytes: int
+ source_mtime_ns: int
+ fps: Fraction
+ input_rate: Fraction
+ source_time_base: Fraction
+ start_time: Fraction
+ frames: tuple[CanonicalFrame, ...]
+ rotation_degrees: int
+
+ @property
+ def height(self) -> int:
+ return int(self.frames[0].rgb.shape[0])
+
+ @property
+ def width(self) -> int:
+ return int(self.frames[0].rgb.shape[1])
+
+ @property
+ def fps_num(self) -> int:
+ return self.fps.numerator
+
+ @property
+ def fps_den(self) -> int:
+ return self.fps.denominator
+
+ @property
+ def rgb_frames(self) -> tuple[np.ndarray, ...]:
+ return tuple(frame.rgb for frame in self.frames)
+
+ def metadata(self) -> dict:
+ return {
+ # The subprocess boundary only needs source identity, not a
+ # machine-specific absolute path. Keep the file-protocol payload
+ # safe to share by storing the basename.
+ "source_path": self.source_path.name,
+ "source_size_bytes": self.source_size_bytes,
+ "source_mtime_ns": self.source_mtime_ns,
+ "fps_num": self.fps_num,
+ "fps_den": self.fps_den,
+ "input_rate_num": self.input_rate.numerator,
+ "input_rate_den": self.input_rate.denominator,
+ "source_time_base_num": self.source_time_base.numerator,
+ "source_time_base_den": self.source_time_base.denominator,
+ "start_time_num": self.start_time.numerator,
+ "start_time_den": self.start_time.denominator,
+ "sampling_policy": SAMPLING_POLICY,
+ "num_frames": len(self.frames),
+ "height": self.height,
+ "width": self.width,
+ "rotation_degrees_applied": self.rotation_degrees,
+ "frames": [
+ {
+ "canonical_index": index,
+ "canonical_timestamp_sec": float(frame.canonical_timestamp),
+ "source_index": frame.source_index,
+ "source_pts": frame.source_pts,
+ "source_timestamp_sec": float(frame.source_timestamp),
+ }
+ for index, frame in enumerate(self.frames)
+ ],
+ }
+
+ def write_metadata(self, path: str | Path) -> None:
+ Path(path).write_text(json.dumps(self.metadata(), indent=2, sort_keys=True) + "\n")
+
+
+@dataclass(frozen=True)
+class _DecodedFrame:
+ rgb: np.ndarray
+ index: int
+ pts: int | None
+ timestamp: Fraction
+ time_base: Fraction
+ rotation_degrees: int
+
+
+def _decode_frames(
+ container: av.container.InputContainer,
+ stream: av.video.stream.VideoStream,
+ rotation: int,
+) -> Iterator[_DecodedFrame]:
+ observed_rotation = None
+ for index, frame in enumerate(container.decode(stream)):
+ frame_rotation = _frame_rotation_degrees(frame, rotation)
+ if observed_rotation is None:
+ observed_rotation = frame_rotation
+ elif frame_rotation != observed_rotation:
+ raise VideoContractError(
+ f"Video display rotation changes at frame {index}: {observed_rotation} -> {frame_rotation}."
+ )
+ yield _DecodedFrame(
+ rgb=_frame_to_rgb(frame, frame_rotation),
+ index=index,
+ pts=frame.pts,
+ timestamp=_frame_timestamp(frame, stream, index),
+ time_base=(
+ Fraction(frame.time_base)
+ if frame.time_base is not None
+ else Fraction(stream.time_base)
+ if stream.time_base is not None
+ else Fraction(1, 1) / _stream_rate(stream)
+ ),
+ rotation_degrees=frame_rotation,
+ )
+
+
+def decode_canonical_clip(
+ video_path: str | Path,
+ *,
+ num_frames: int = 121,
+ start_time: float = 0.0,
+ fps: str | int | float | Fraction | None = None,
+) -> CanonicalClip:
+ """Decode one canonical clip, selecting frames by source presentation time."""
+
+ path = Path(video_path).expanduser().resolve()
+ if not path.is_file():
+ raise VideoContractError(f"Input video does not exist: {path}")
+ if num_frames <= 0:
+ raise VideoContractError(f"num_frames must be positive, got {num_frames}.")
+ if not math.isfinite(start_time) or start_time < 0:
+ raise VideoContractError(f"start_time must be non-negative, got {start_time}.")
+
+ source_stat = path.stat()
+
+ with av.open(str(path), mode="r") as container:
+ if not container.streams.video:
+ raise VideoContractError(f"Input has no video stream: {path}")
+ stream = container.streams.video[0]
+ input_rate = _stream_rate(stream)
+ try:
+ output_rate = choose_canonical_fps(input_rate) if fps is None else Fraction(str(fps))
+ except (ValueError, ZeroDivisionError) as exc:
+ raise VideoContractError(f"Cannot parse target fps {fps!r} as a rational frame rate.") from exc
+ if output_rate <= 0:
+ raise VideoContractError(f"Target fps must be positive, got {output_rate}.")
+ metadata_rotation = _rotation_degrees(stream)
+ decoded = _decode_frames(container, stream, metadata_rotation)
+ try:
+ previous = next(decoded)
+ except StopIteration as exc:
+ raise VideoContractError(f"Input video contains no decodable frames: {path}") from exc
+
+ origin = previous.timestamp
+ start_offset = Fraction(str(start_time))
+ start = origin + start_offset
+ targets = [start + Fraction(index, 1) / output_rate for index in range(num_frames)]
+ selected: list[_DecodedFrame] = []
+ target_index = 0
+ current = previous
+
+ for current in decoded:
+ if current.timestamp < previous.timestamp:
+ raise VideoContractError(
+ f"Input presentation timestamps are not monotonic at source frame {current.index}: "
+ f"{float(current.timestamp):.6f}s < {float(previous.timestamp):.6f}s."
+ )
+ while target_index < num_frames and targets[target_index] <= current.timestamp:
+ target = targets[target_index]
+ candidate = previous if abs(previous.timestamp - target) <= abs(current.timestamp - target) else current
+ if selected and candidate.index < selected[-1].index:
+ raise VideoContractError("Temporal sampling produced non-monotonic source-frame order.")
+ selected.append(candidate)
+ target_index += 1
+ if target_index >= num_frames:
+ break
+ previous = current
+
+ max_error = Fraction(3, 4) / output_rate
+ if target_index < num_frames:
+ # A faster canonical clock can legitimately select the final source
+ # frame more than once, just as it may reuse frames in the middle of
+ # the clip. Stop once the nearest-frame error exceeds the same gap
+ # bound enforced below; that remains a short-input failure rather
+ # than temporal padding.
+ while target_index < num_frames and abs(current.timestamp - targets[target_index]) <= max_error:
+ if selected and current.index < selected[-1].index:
+ raise VideoContractError("Temporal sampling produced non-monotonic source-frame order.")
+ selected.append(current)
+ target_index += 1
+
+ if len(selected) != num_frames:
+ duration = float(current.timestamp - origin)
+ required = float(Fraction(num_frames - 1, 1) / output_rate + start_offset)
+ raise VideoContractError(
+ f"Input is too short for {num_frames} frames at {float(output_rate):.6f} FPS from "
+ f"start_time={start_time}: decoded duration={duration:.3f}s, required={required:.3f}s."
+ )
+
+ errors = [abs(frame.timestamp - target) for frame, target in zip(selected, targets, strict=True)]
+ if max(errors) > max_error:
+ raise VideoContractError(
+ "Input timestamps contain a gap too large for stable sampling: "
+ f"max error={float(max(errors)):.6f}s, limit={float(max_error):.6f}s."
+ )
+ expected_shape = selected[0].rgb.shape
+ if any(frame.rgb.shape != expected_shape for frame in selected[1:]):
+ raise VideoContractError("Input frame dimensions change inside the selected canonical clip.")
+ source_time_base = selected[0].time_base
+ if any(frame.time_base != source_time_base for frame in selected[1:]):
+ raise VideoContractError("Input frame time base changes inside the selected canonical clip.")
+
+ canonical_frames = tuple(
+ CanonicalFrame(
+ rgb=frame.rgb,
+ source_index=frame.index,
+ source_pts=frame.pts,
+ source_timestamp=frame.timestamp,
+ canonical_timestamp=Fraction(index, 1) / output_rate,
+ )
+ for index, frame in enumerate(selected)
+ )
+
+ final_stat = path.stat()
+ if (final_stat.st_size, final_stat.st_mtime_ns) != (source_stat.st_size, source_stat.st_mtime_ns):
+ raise VideoContractError(f"Input video changed while it was being decoded: {path}")
+ return CanonicalClip(
+ source_path=path,
+ source_size_bytes=source_stat.st_size,
+ source_mtime_ns=source_stat.st_mtime_ns,
+ fps=output_rate,
+ input_rate=input_rate,
+ source_time_base=source_time_base,
+ start_time=start_offset,
+ frames=canonical_frames,
+ rotation_degrees=selected[0].rotation_degrees,
+ )
+
+
+def write_lossless_video(clip: CanonicalClip, path: str | Path) -> Path:
+ """Write an FFV1 working video. A subsequent decode must be RGB-identical."""
+
+ output_path = Path(path).expanduser().resolve()
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ with av.open(str(output_path), mode="w", format="matroska") as container:
+ stream = container.add_stream("ffv1", rate=clip.fps)
+ stream.width = clip.width
+ stream.height = clip.height
+ # FFV1 does not expose 8-bit GBR planar. BGR0 is an exact 8-bit RGB
+ # representation (the fourth byte is padding) and round-trips to rgb24.
+ stream.pix_fmt = "bgr0"
+ for index, canonical_frame in enumerate(clip.frames):
+ frame = av.VideoFrame.from_ndarray(canonical_frame.rgb, format="rgb24")
+ frame.pts = index
+ frame.time_base = Fraction(1, 1) / clip.fps
+ for packet in stream.encode(frame):
+ container.mux(packet)
+ for packet in stream.encode():
+ container.mux(packet)
+ verify_lossless_video(clip, output_path)
+ return output_path
+
+
+def write_gvhmr_video(clip: CanonicalClip, path: str | Path) -> Path:
+ """Write the frame-counted, RGB-lossless MP4 consumed by GVHMR.
+
+ GVHMR's imageio metadata probe cannot determine the frame count of an
+ FFV1 Matroska stream. ``libx264rgb`` in lossless mode preserves every RGB
+ byte while placing an exact frame count in the MP4 index.
+ """
+
+ output_path = Path(path).expanduser().resolve()
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ with av.open(str(output_path), mode="w") as container:
+ stream = container.add_stream("libx264rgb", rate=clip.fps)
+ stream.width = clip.width
+ stream.height = clip.height
+ stream.pix_fmt = "rgb24"
+ stream.options = {"crf": "0", "preset": "medium"}
+ for index, canonical_frame in enumerate(clip.frames):
+ frame = av.VideoFrame.from_ndarray(canonical_frame.rgb, format="rgb24")
+ frame.pts = index
+ frame.time_base = Fraction(1, 1) / clip.fps
+ for packet in stream.encode(frame):
+ container.mux(packet)
+ for packet in stream.encode():
+ container.mux(packet)
+ verify_lossless_video(clip, output_path)
+ with av.open(str(output_path), mode="r") as container:
+ declared_frames = int(container.streams.video[0].frames)
+ if declared_frames != len(clip.frames):
+ raise VideoContractError(
+ f"Backend MP4 declares {declared_frames} frames, expected {len(clip.frames)}: {output_path}."
+ )
+ return output_path
+
+
+def verify_lossless_video(clip: CanonicalClip, path: str | Path) -> None:
+ decoded = iter_rgb_video(path)
+ sentinel = object()
+ for index, (actual, expected) in enumerate(itertools.zip_longest(decoded, clip.rgb_frames, fillvalue=sentinel)):
+ if actual is sentinel or expected is sentinel or not np.array_equal(actual, expected):
+ raise VideoContractError(f"Lossless working-video verification failed at frame {index}.")
+
+
+def write_video(
+ frames: Iterator[np.ndarray] | tuple[np.ndarray, ...],
+ path: str | Path,
+ fps: Fraction,
+ *,
+ crf: int = 18,
+ preset: str = "medium",
+) -> Path:
+ """Encode RGB frames as a broadly playable H.264 MP4."""
+
+ output_path = Path(path).expanduser().resolve()
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ iterator = iter(frames)
+ try:
+ first = next(iterator)
+ except StopIteration as exc:
+ raise VideoContractError("Cannot encode an empty video.") from exc
+ height, width = first.shape[:2]
+ with av.open(str(output_path), mode="w") as container:
+ stream = container.add_stream("libx264", rate=fps)
+ stream.width = width
+ stream.height = height
+ stream.pix_fmt = "yuv420p"
+ stream.options = {"crf": str(crf), "preset": preset}
+ for index, rgb in enumerate(itertools.chain((first,), iterator)):
+ if rgb.shape != first.shape:
+ raise VideoContractError(f"Frame {index} has shape {rgb.shape}, expected {first.shape}.")
+ frame = av.VideoFrame.from_ndarray(np.ascontiguousarray(rgb), format="rgb24")
+ frame.pts = index
+ frame.time_base = Fraction(1, 1) / fps
+ for packet in stream.encode(frame):
+ container.mux(packet)
+ for packet in stream.encode():
+ container.mux(packet)
+ return output_path
+
+
+def iter_rgb_video(path: str | Path) -> Iterator[np.ndarray]:
+ """Stream RGB frames without interpreting or resampling timestamps."""
+
+ video_path = Path(path).expanduser().resolve()
+ with av.open(str(video_path), mode="r") as container:
+ if not container.streams.video:
+ raise VideoContractError(f"Video has no stream: {video_path}")
+ for frame in container.decode(container.streams.video[0]):
+ yield np.ascontiguousarray(frame.to_ndarray(format="rgb24"))
+
+
+def read_rgb_video(path: str | Path, *, expected_frames: int | None = None) -> tuple[np.ndarray, ...]:
+ """Decode an RGB video without interpreting or resampling its timestamps."""
+
+ frames = tuple(iter_rgb_video(path))
+ if expected_frames is not None and len(frames) != expected_frames:
+ raise VideoContractError(
+ f"{Path(path).expanduser().resolve()} has {len(frames)} frames, expected {expected_frames}."
+ )
+ return frames
+
+
+def load_canonical_working_clip(video_path: str | Path, metadata_path: str | Path) -> CanonicalClip:
+ """Rehydrate the canonical clip inside a short-lived worker."""
+
+ video_path = Path(video_path).expanduser().resolve()
+ metadata = json.loads(Path(metadata_path).read_text())
+ if metadata.get("sampling_policy") != SAMPLING_POLICY:
+ raise VideoContractError(f"Unsupported canonical sampling policy: {metadata.get('sampling_policy')!r}.")
+ records = metadata["frames"]
+ decoded = read_rgb_video(video_path, expected_frames=int(metadata["num_frames"]))
+ if len(records) != len(decoded):
+ raise VideoContractError(
+ f"Canonical metadata has {len(records)} frame records, but {video_path} has {len(decoded)} frames."
+ )
+ frames = []
+ for canonical_index, (rgb, record) in enumerate(zip(decoded, records, strict=True)):
+ if int(record["canonical_index"]) != canonical_index:
+ raise VideoContractError("Canonical subprocess metadata is not in frame-index order.")
+ frames.append(
+ CanonicalFrame(
+ rgb=rgb,
+ source_index=int(record["source_index"]),
+ source_pts=None if record["source_pts"] is None else int(record["source_pts"]),
+ source_timestamp=Fraction(str(record["source_timestamp_sec"])),
+ canonical_timestamp=Fraction(canonical_index, 1)
+ / Fraction(int(metadata["fps_num"]), int(metadata["fps_den"])),
+ )
+ )
+ return CanonicalClip(
+ source_path=Path(metadata["source_path"]),
+ source_size_bytes=int(metadata["source_size_bytes"]),
+ source_mtime_ns=int(metadata["source_mtime_ns"]),
+ fps=Fraction(int(metadata["fps_num"]), int(metadata["fps_den"])),
+ input_rate=Fraction(int(metadata["input_rate_num"]), int(metadata["input_rate_den"])),
+ source_time_base=Fraction(int(metadata["source_time_base_num"]), int(metadata["source_time_base_den"])),
+ start_time=Fraction(int(metadata["start_time_num"]), int(metadata["start_time_den"])),
+ frames=tuple(frames),
+ rotation_degrees=int(metadata["rotation_degrees_applied"]),
+ )
diff --git a/fdanyone/views.py b/fdanyone/views.py
new file mode 100644
index 0000000..7e9e2f0
--- /dev/null
+++ b/fdanyone/views.py
@@ -0,0 +1,203 @@
+"""Resolve the reader-facing target-view layout and inference grouping."""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+
+from fdanyone.config import CAMERA
+from fdanyone.errors import ConfigurationError
+
+VALID_VIEWS_PER_GROUP = (1, 4, 6)
+MIN_PITCH = -15
+MAX_PITCH = 45
+
+# RCP uses the canonical proposal cameras seen during training. The resolved
+# group size selects a prefix; at most the first four become target references.
+RCP_CAMERA_ORDER = (4, 9, 14, 19, 0, 12)
+
+
+@dataclass(frozen=True)
+class TargetView:
+ """One requested camera in the layer-major public view order."""
+
+ camera_id: int
+ layer_index: int
+ pitch: int
+ yaw: float
+
+
+@dataclass(frozen=True)
+class ViewPlan:
+ """Validated target cameras and method components for one run."""
+
+ views_per_layer: int
+ layer_pitches: tuple[int, ...]
+ start_yaw: int
+ yaw_span: int
+ views_per_group: int
+ enable_rcp: bool
+ enable_tcr: bool
+
+ @property
+ def num_layers(self) -> int:
+ return len(self.layer_pitches)
+
+ @property
+ def num_target_views(self) -> int:
+ return self.views_per_layer * self.num_layers
+
+ @property
+ def groups_per_layer(self) -> int:
+ return self.views_per_layer // self.views_per_group
+
+ @property
+ def num_groups(self) -> int:
+ return self.groups_per_layer * self.num_layers
+
+ @property
+ def closed_yaw(self) -> bool:
+ return self.yaw_span == 360
+
+ @property
+ def tcr_active(self) -> bool:
+ # A one-view group has no cross-view context to rearrange. All other
+ # layouts honor enable_tcr, including partial yaw spans and one group.
+ return self.enable_tcr and self.views_per_group > 1
+
+ @property
+ def target_views(self) -> tuple[TargetView, ...]:
+ step = self.yaw_span / self.views_per_layer
+ return tuple(
+ TargetView(
+ camera_id=layer_index * self.views_per_layer + view_index,
+ layer_index=layer_index,
+ pitch=pitch,
+ yaw=self.start_yaw + view_index * step,
+ )
+ for layer_index, pitch in enumerate(self.layer_pitches)
+ for view_index in range(self.views_per_layer)
+ )
+
+ @property
+ def front_camera_ids(self) -> tuple[int, ...]:
+ return tuple(view.camera_id for view in self.target_views if abs(view.yaw % 360.0) < 1e-8)
+
+ @property
+ def rcp_camera_ids(self) -> tuple[int, ...]:
+ return RCP_CAMERA_ORDER[: self.views_per_group] if self.enable_rcp else ()
+
+ @property
+ def is_canonical_target_ring(self) -> bool:
+ return (
+ self.views_per_layer == CAMERA.count
+ and self.layer_pitches == (int(CAMERA.pitch_degrees),)
+ and self.start_yaw == 0
+ and self.yaw_span == 360
+ )
+
+ def to_dict(self) -> dict[str, object]:
+ return {
+ "views_per_layer": self.views_per_layer,
+ "layer_pitches": list(self.layer_pitches),
+ "start_yaw": self.start_yaw,
+ "yaw_span": self.yaw_span,
+ "views_per_group": self.views_per_group,
+ "enable_rcp": self.enable_rcp,
+ "enable_tcr": self.enable_tcr,
+ }
+
+ @classmethod
+ def from_dict(cls, value: object) -> ViewPlan:
+ if not isinstance(value, dict):
+ raise ConfigurationError("View plan must be a JSON object.")
+ try:
+ return resolve_view_plan(
+ views_per_layer=value["views_per_layer"],
+ layer_pitches=value["layer_pitches"],
+ start_yaw=value["start_yaw"],
+ yaw_span=value["yaw_span"],
+ views_per_group=value["views_per_group"],
+ enable_rcp=value["enable_rcp"],
+ enable_tcr=value["enable_tcr"],
+ )
+ except KeyError as exc:
+ raise ConfigurationError(f"View plan is missing {exc.args[0]!r}.") from None
+
+
+def _integer(name: str, value: object) -> int:
+ if isinstance(value, bool) or not isinstance(value, int):
+ raise ConfigurationError(f"{name} must be an integer, got {value!r}.")
+ return value
+
+
+def _layer_pitches(value: object) -> tuple[int, ...]:
+ if isinstance(value, (str, bytes)) or not isinstance(value, Sequence) or not value:
+ raise ConfigurationError("layer_pitches must be a non-empty list of integer degrees.")
+ pitches = tuple(_integer("Each layer pitch", pitch) for pitch in value)
+ if len(set(pitches)) != len(pitches):
+ raise ConfigurationError(f"layer_pitches must not contain duplicates, got {list(pitches)}.")
+ invalid = [pitch for pitch in pitches if not MIN_PITCH <= pitch <= MAX_PITCH]
+ if invalid:
+ raise ConfigurationError(
+ f"Each layer pitch must be between {MIN_PITCH} and {MAX_PITCH} degrees, got {invalid}."
+ )
+ return pitches
+
+
+def _group_size(value: int | str, views_per_layer: int) -> int:
+ if isinstance(value, str):
+ if value.lower() == "auto":
+ return max(size for size in VALID_VIEWS_PER_GROUP if views_per_layer % size == 0)
+ try:
+ value = int(value)
+ except ValueError:
+ raise ConfigurationError(
+ f"views_per_group must be 'auto' or one of {VALID_VIEWS_PER_GROUP}, got {value!r}."
+ ) from None
+ value = _integer("views_per_group", value)
+ if value not in VALID_VIEWS_PER_GROUP:
+ raise ConfigurationError(f"views_per_group must be one of {VALID_VIEWS_PER_GROUP}, got {value!r}.")
+ if views_per_layer % value:
+ raise ConfigurationError(f"views_per_layer ({views_per_layer}) must be divisible by views_per_group ({value}).")
+ return value
+
+
+def resolve_view_plan(
+ *,
+ views_per_layer: int = 24,
+ layer_pitches: Sequence[int] = (15,),
+ start_yaw: int = 0,
+ yaw_span: int = 360,
+ views_per_group: int | str = "auto",
+ enable_rcp: bool = True,
+ enable_tcr: bool = True,
+) -> ViewPlan:
+ """Validate the compact CLI settings before expensive work starts."""
+
+ views_per_layer = _integer("views_per_layer", views_per_layer)
+ if views_per_layer <= 0:
+ raise ConfigurationError(f"views_per_layer must be positive, got {views_per_layer}.")
+ pitches = _layer_pitches(layer_pitches)
+ start_yaw = _integer("start_yaw", start_yaw)
+ start_yaw = (start_yaw + 180) % 360 - 180
+ yaw_span = _integer("yaw_span", yaw_span)
+ if not 0 < yaw_span <= 360:
+ raise ConfigurationError(f"yaw_span must be between 1 and 360 degrees, got {yaw_span}.")
+ resolved_group_size = _group_size(views_per_group, views_per_layer)
+ if not isinstance(enable_rcp, bool):
+ raise ConfigurationError(f"enable_rcp must be True or False, got {enable_rcp!r}.")
+ if not isinstance(enable_tcr, bool):
+ raise ConfigurationError(f"enable_tcr must be True or False, got {enable_tcr!r}.")
+
+ # Up to six requested targets are cheaper and clearer to generate directly.
+ rcp_active = enable_rcp and views_per_layer * len(pitches) > 6
+ return ViewPlan(
+ views_per_layer=views_per_layer,
+ layer_pitches=pitches,
+ start_yaw=start_yaw,
+ yaw_span=yaw_span,
+ views_per_group=resolved_group_size,
+ enable_rcp=rcp_active,
+ enable_tcr=enable_tcr,
+ )
diff --git a/inference.py b/inference.py
new file mode 100644
index 0000000..1c4e702
--- /dev/null
+++ b/inference.py
@@ -0,0 +1,91 @@
+"""Run 4DAnyone inference from a monocular video."""
+
+from __future__ import annotations
+
+import sys
+
+from fire import Fire
+
+from fdanyone.errors import FourDAnyoneError
+
+
+def inference(
+ video_path: str,
+ views_per_layer: int = 24,
+ layer_pitches: list[int] = [15], # noqa: B006 - normalized without mutation
+ start_yaw: int = 0,
+ yaw_span: int = 360,
+ views_per_group: int | str = "auto",
+ enable_rcp: bool = True,
+ enable_tcr: bool = True,
+ data_dir: str = "data",
+ model_dir: str = "models",
+ checkpoint_path: str | None = None,
+ mhr70_regressor_path: str | None = None,
+ gvhmr_root: str = "third_party/GVHMR",
+ device: str = "cuda:0",
+ target_fps: str | int | float = "auto",
+ start_time: float = 0.0,
+ seed: int = 42,
+) -> dict:
+ """Generate synchronized target-view videos from one monocular video.
+
+ Args:
+ video_path: Input video; it must contain at least 121 usable frames.
+ views_per_layer: Number of evenly spaced yaw views at each pitch.
+ layer_pitches: Camera pitch for each layer in degrees, for example
+ [-10,15,35]. Positive values place the camera above the subject;
+ each value must be between -15 and 45.
+ start_yaw: First yaw in every layer, in degrees; 0 faces the person.
+ yaw_span: Angular range sampled by each layer, from 1 to 360 degrees.
+ The end angle is excluded so a full ring never duplicates a view.
+ views_per_group: Maximum target views generated together. auto chooses
+ the largest divisor among 6, 4, and 1; a manual value must be 1,
+ 4, or 6 and divide views_per_layer.
+ enable_rcp: Use proposal views before generating more than six targets.
+ The proposal count follows views_per_group.
+ enable_tcr: Shift view groups between denoising steps. Partial yaw
+ ranges remain open sequences rather than wrapping their endpoints.
+ data_dir: Root for reusable GVHMR motion and final 4DAnyone outputs.
+ model_dir: Model root; missing public checkpoints download here.
+ checkpoint_path: Local 4DAnyone checkpoint override.
+ mhr70_regressor_path: Local SMPL-X-to-MHR70 regressor override.
+ gvhmr_root: Path to the GVHMR source checkout.
+ device: CUDA device, for example cuda:0.
+ target_fps: auto preserves the input clock unless it divides evenly
+ to 24, 25, or 30 FPS; a positive number requests an explicit FPS.
+ start_time: Clip start time on the input timeline, in seconds.
+ seed: Random seed shared by proposal and target generation.
+ """
+
+ # Keep model imports out of module scope so ``--help`` stays lightweight.
+ from fdanyone.pipeline import run_pipeline
+
+ return run_pipeline(
+ video_path=video_path,
+ views_per_layer=views_per_layer,
+ layer_pitches=layer_pitches,
+ start_yaw=start_yaw,
+ yaw_span=yaw_span,
+ views_per_group=views_per_group,
+ enable_rcp=enable_rcp,
+ enable_tcr=enable_tcr,
+ data_dir=data_dir,
+ model_dir=model_dir,
+ checkpoint_path=checkpoint_path,
+ mhr70_regressor_path=mhr70_regressor_path,
+ gvhmr_root=gvhmr_root,
+ device=device,
+ target_fps=target_fps,
+ start_time=start_time,
+ seed=seed,
+ )
+
+
+if __name__ == "__main__":
+ try:
+ Fire(inference)
+ except FourDAnyoneError as exc:
+ message = " ".join(line.strip() for line in str(exc).splitlines())
+ print(f"error: {message}", file=sys.stderr)
+ raise SystemExit(1) from None
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..32a6c5e
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,29 @@
+av>=16,<17
+colorlog>=6,<7
+einops>=0.8,<1
+ffmpeg-python>=0.2,<1
+fire>=0.7,<1
+ftfy>=6,<7
+huggingface-hub>=0.36,<1
+hydra-core>=1.3,<2
+hydra-zen>=0.16,<1
+imageio>=2.34,<3
+kornia>=0.7,<1
+lapx>=0.5,<1
+numpy>=1.26,<2
+opencv-python>=4.10,<5
+pillow>=10,<13
+pytorch-lightning>=2.3,<3
+regex>=2024.11
+safetensors>=0.5,<1
+scipy>=1.10,<2
+sentencepiece>=0.2,<1
+smplx==0.1.28
+timm==0.9.12
+torch>=2.8,<2.9
+torchvision>=0.23,<0.24
+tqdm>=4.66,<5
+transformers>=4.57,<5
+typing-extensions>=4.10,<5
+ultralytics==8.2.42
+yacs>=0.1,<1
diff --git a/scripts/download_example.py b/scripts/download_example.py
new file mode 100644
index 0000000..c9ff15e
--- /dev/null
+++ b/scripts/download_example.py
@@ -0,0 +1,25 @@
+"""Download the bundled 4DAnyone example clips."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+
+def main() -> None:
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+ from fire import Fire
+
+ from fdanyone.download import download_example
+ from fdanyone.errors import FourDAnyoneError
+
+ try:
+ Fire(download_example)
+ except FourDAnyoneError as exc:
+ message = " ".join(line.strip() for line in str(exc).splitlines())
+ print(f"error: {message}", file=sys.stderr)
+ raise SystemExit(1) from None
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/download_model.py b/scripts/download_model.py
new file mode 100644
index 0000000..c486c15
--- /dev/null
+++ b/scripts/download_model.py
@@ -0,0 +1,25 @@
+"""Download the published 4DAnyone model checkpoints."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+
+def main() -> None:
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+ from fire import Fire
+
+ from fdanyone.download import download_model
+ from fdanyone.errors import FourDAnyoneError
+
+ try:
+ Fire(download_model)
+ except FourDAnyoneError as exc:
+ message = " ".join(line.strip() for line in str(exc).splitlines())
+ print(f"error: {message}", file=sys.stderr)
+ raise SystemExit(1) from None
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/download_smplx.py b/scripts/download_smplx.py
new file mode 100644
index 0000000..7d605f1
--- /dev/null
+++ b/scripts/download_smplx.py
@@ -0,0 +1,25 @@
+"""Install the separately licensed SMPL-X body model."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+
+def main() -> None:
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+ from fire import Fire
+
+ from fdanyone.download import download_smplx
+ from fdanyone.errors import FourDAnyoneError
+
+ try:
+ Fire(download_smplx)
+ except FourDAnyoneError as exc:
+ message = " ".join(line.strip() for line in str(exc).splitlines())
+ print(f"error: {message}", file=sys.stderr)
+ raise SystemExit(1) from None
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/export_nerfstudio.py b/scripts/export_nerfstudio.py
new file mode 100644
index 0000000..4d59c25
--- /dev/null
+++ b/scripts/export_nerfstudio.py
@@ -0,0 +1,25 @@
+"""Export one synchronized 4DAnyone frame for Nerfstudio."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+
+def main() -> None:
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+ from fire import Fire
+
+ from fdanyone.errors import FourDAnyoneError
+ from fdanyone.nerfstudio import export_nerfstudio
+
+ try:
+ Fire(export_nerfstudio)
+ except FourDAnyoneError as exc:
+ message = " ".join(line.strip() for line in str(exc).splitlines())
+ print(f"error: {message}", file=sys.stderr)
+ raise SystemExit(1) from None
+
+
+if __name__ == "__main__":
+ main()
diff --git a/third_party/GVHMR b/third_party/GVHMR
new file mode 160000
index 0000000..6ec3ca3
--- /dev/null
+++ b/third_party/GVHMR
@@ -0,0 +1 @@
+Subproject commit 6ec3ca39336c50492c0fae65fba2fb831fc7d866
diff --git a/third_party/licenses/BIREFNET_LICENSE b/third_party/licenses/BIREFNET_LICENSE
new file mode 100644
index 0000000..485921e
--- /dev/null
+++ b/third_party/licenses/BIREFNET_LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 ZhengPeng
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/third_party/licenses/SAPIENS2_LICENSE.md b/third_party/licenses/SAPIENS2_LICENSE.md
new file mode 100644
index 0000000..f51b769
--- /dev/null
+++ b/third_party/licenses/SAPIENS2_LICENSE.md
@@ -0,0 +1,79 @@
+# Sapiens2 License
+
+## Definitions
+
+**"Agreement"** means the terms and conditions for use, reproduction, distribution and modification of the Sapiens Materials set forth herein.
+
+**"Sapiens Materials"** means, collectively, Documentation and the models, software and algorithms, including machine-learning model code, trained model weights, inference-enabling code, training-enabling code, fine-tuning enabling code, and other elements of the foregoing distributed by Meta and made available under this Agreement.
+
+**"Documentation"** means the specifications, manuals and documentation accompanying Sapiens Materials distributed by Meta.
+
+**"Licensee"** or **"you"** means you, or your employer or any other person or entity (if you are entering into this Agreement on such person or entity's behalf), of the age required under applicable laws, rules or regulations to provide legal consent and that has legal authority to bind your employer or such other person or entity if you are entering in this Agreement on their behalf.
+
+**"Meta"** or **"we"** means Meta Platforms Ireland Limited (if you are located in or, if you are an entity, your principal place of business is in the EEA or Switzerland) or Meta Platforms, Inc. (if you are located outside of the EEA or Switzerland).
+
+**"Sanctions"** means any economic or trade sanctions or restrictions administered or enforced by the United States (including the Office of Foreign Assets Control of the U.S. Department of the Treasury ("OFAC"), the U.S. Department of State and the U.S. Department of Commerce), the United Nations, the European Union, or the United Kingdom.
+
+**"Trade Controls"** means any of the following: Sanctions and applicable export and import controls.
+
+By clicking **"I Accept"** below or by using or distributing any portion or element of the Sapiens Materials, you agree to be bound by this Agreement.
+
+## 1. License Rights and Redistribution
+
+**a. Grant of Rights.** You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Meta's intellectual property or other rights owned by Meta embodied in the Sapiens Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Sapiens Materials.
+
+**b. Redistribution and Use.**
+
+ i. Distribution of Sapiens Materials, and any derivative works thereof, are subject to the terms of this Agreement. If you distribute or make the Sapiens Materials, or any derivative works thereof, available to a third party, you may only do so under the terms of this Agreement and you shall provide a copy of this Agreement with any such Sapiens Materials.
+
+ ii. If you submit for publication the results of research you perform on, using, or otherwise in connection with Sapiens Materials, you must acknowledge the use of Sapiens Materials in your publication.
+
+ iii. Your use of the Sapiens Materials must comply with applicable laws and regulations, including Trade Control Laws and applicable privacy, data protection and security laws, rules, regulations, directives, or governmental requirements including the General Data Privacy Regulation (Regulation (EU) 2016/679), the California Consumer Privacy Act, U.S. data security regulations set forth at 28 C.F.R. Part 202, the Lei Geral de Proteção de Dados, and any and all laws governing the processing of biometric information, as well as all amendments and successor laws to any of the foregoing.
+
+ iv. Your use of the Sapiens Materials will not involve or encourage others to reverse engineer, decompile or discover the underlying components of the Sapiens Materials.
+
+ v. You are not the target of Trade Controls and your use of Sapiens Materials must comply with Trade Controls.
+
+ vi. Your use of the Sapiens Materials will not involve or encourage others to use for:
+
+ - (i) for purposes of surveillance, including any research or development relating to surveillance;
+ - (ii) for biometric processing;
+ - (iii) in any manner that infringes, misappropriates, or otherwise violates any third-party rights;
+ - (iv) for attempting to identify or re-identify any individual included in the Sapiens Materials;
+ - (v) to create deepfakes, surveillance, or other misleading, deceptive, or inappropriate content including but not limited to impersonating any individual or facilitating false online engagement;
+ - (vi) to collect, process, disclose, generate, or infer health, demographic, or other sensitive personal or private information about individuals without rights and consents required by applicable laws;
+ - (vii) in a pornographic or defamatory manner;
+ - (viii) to engage in the unauthorized or unlicensed practice of any profession including but not limited to financial, legal, medical/health, or related professional practices; or
+ - (ix) to engage in, promote, incite, facilitate, or assist in the planning or development of any activities subject to the International Traffic in Arms Regulations (ITAR) or end uses prohibited by Trade Controls or any other activities that present a risk of death or bodily harm to individuals, including those related to military or warfare purposes, nuclear industries or applications, espionage, the development or use of guns or illegal weapons or illegal drugs, and the operation of critical infrastructure or machinery.
+
+**c.** If Meta reasonably believes, as determined by Meta in its sole discretion, that you are or are likely to be in violation of the terms of this Agreement, Meta or Meta's designee, at Meta's sole expense, may audit your use, storage, and distribution of the Sapiens Materials including, without limitation, any and all records and files associated with the Licensed Content and this Agreement. You hereby agree to cooperate with such an audit. Meta may also require You to delete all copies of the Sapiens Materials (in whole or in part) in Your possession and control. You will promptly comply with any and all such requests. Upon Meta's request, You shall provide Meta with written confirmation of Your compliance with such requirement.
+
+## 2. User Support
+
+Your use of the Sapiens Materials is done at your own discretion; Meta does not process any information nor provide any service in relation to such use. Meta is under no obligation to provide any support services for the Sapiens Materials. Any support provided is "as is", "with all faults", and without warranty of any kind.
+
+## 3. Disclaimer of Warranty
+
+UNLESS REQUIRED BY APPLICABLE LAW, THE SAPIENS MATERIALS AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, AND META DISCLAIMS ALL WARRANTIES OF ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE SAPIENS MATERIALS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE SAPIENS MATERIALS AND ANY OUTPUT AND RESULTS.
+
+## 4. Limitation of Liability
+
+IN NO EVENT WILL META OR ITS AFFILIATES BE LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST PROFITS OR ANY DIRECT OR INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES, EVEN IF META OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF ANY OF THE FOREGOING.
+
+## 5. Intellectual Property
+
+**a.** Subject to Meta's ownership of Sapiens Materials and derivatives made by or for Meta, with respect to any derivative works and modifications of the Sapiens Materials that are made by you, as between you and Meta, you are and will be the owner of such derivative works and modifications.
+
+**b.** If you institute litigation or other proceedings against Meta or any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Sapiens Materials, outputs or results, or any portion of any of the foregoing, constitutes infringement of intellectual property or other rights owned or licensable by you, then any licenses granted to you under this Agreement shall terminate as of the date such litigation or claim is filed or instituted. You will indemnify and hold harmless Meta from and against any claim by any third party arising out of or related to your use or distribution of the Sapiens Materials.
+
+## 6. Term and Termination
+
+The term of this Agreement will commence upon your acceptance of this Agreement or access to the Sapiens Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein. Meta may terminate this Agreement if you are in breach of any term or condition of this Agreement. Upon termination of this Agreement, you shall delete and cease use of the Sapiens Materials. Sections 3, 4 and 7 shall survive the termination of this Agreement.
+
+## 7. Governing Law and Jurisdiction
+
+This Agreement will be governed and construed under the laws of the State of California without regard to choice of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement. The courts of California shall have exclusive jurisdiction of any dispute arising out of this Agreement.
+
+## 8. Modifications and Amendments
+
+Meta may modify this Agreement from time to time; provided that they are similar in spirit to the current version of the Agreement, but may differ in detail to address new problems or concerns. All such changes will be effective immediately. Your continued use of the Sapiens Materials after any modification to this Agreement constitutes your agreement to such modification. Except as provided in this Agreement, no modification or addition to any provision of this Agreement will be binding unless it is in writing and signed by an authorized representative of both you and Meta.