Skip to content

[SIGGRAPH Asia 2026] 4DAnyone: Create Anyone in 4D from a Casual Monocular Video

Release: enable Wan2.2 5B Turbo inference (#21)

Yudong Jin488c7890ebb2
diff --git a/README.md b/README.md
index 7483908..84a2f61 100644
--- a/README.md
+++ b/README.md
@@ -10,8 +10,9 @@
## News
+- **2026-09-02**: Achieved a **5.58×** denoising speedup over the base model using the Wan2.2 5B Turbo LoRA. See [Inference performance](docs/inference_performance.md) for details.
- **2026-08-28**: Achieved a **1.42×** end-to-end speedup for the complete 24-view generation pipeline.
-- **2026-08-28**: Reduced peak GPU memory to **25.4 GiB** while slightly improving speed. See [Inference performance](docs/inference_performance.md) for details.
+- **2026-08-28**: Reduced peak GPU memory to **25.4 GiB** while slightly improving speed.
## Installation
@@ -94,6 +95,7 @@ Run `python inference.py --help` for the full list.
- `start_yaw`: horizontal angle of the first view, in degrees. Yaw `0` is the front view.
- `yaw_span`: horizontal range covered by each camera layer, in degrees.
- `gpu_ids`: GPU IDs used for parallel pose/VAE view stages and target denoising. Defaults to all visible GPUs.
+- `enable_turbo`: enable the Wan2.2 5B Turbo LoRA. Defaults to `True`; set it to `False` to use the base model.
### Output
@@ -140,8 +142,7 @@ See the [nerfstudio guide](docs/nerfstudio.md) for details.
### Inference Acceleration
- [x] Achieve up to 1.42× end-to-end speedup by parallelizing pose encoding and VAE processing across GPUs.
-- [ ] Integrate [Sol-Engine](https://nvlabs.github.io/Sana/Sol-Engine/) (expected 2× speedup).
-- [ ] Distill the model for few-step inference (expected 5× speedup).
+- [x] Accelerate inference with the Wan2.2 5B Turbo LoRA.
### Reconstruction
diff --git a/docs/THIRD_PARTY_NOTICES.md b/docs/THIRD_PARTY_NOTICES.md
index 983615c..e018b14 100644
--- a/docs/THIRD_PARTY_NOTICES.md
+++ b/docs/THIRD_PARTY_NOTICES.md
@@ -39,7 +39,7 @@ Classic GVHMR imports PyTorch3D rotation conversions through a broader training-
- Location: `data/source/pexels`
- Terms: <https://www.pexels.com/license/>
-The companion Hugging Face repository distributes eight modified 121-frame excerpts for running the public demo. They are installed under the location above on demand. The excerpts derive from Pexels videos [10331522](https://www.pexels.com/video/10331522/), [2785536](https://www.pexels.com/video/2785536/), [5435720](https://www.pexels.com/video/5435720/), [5885633](https://www.pexels.com/video/5885633/), [5999210](https://www.pexels.com/video/5999210/), [6980035](https://www.pexels.com/video/6980035/), [7080903](https://www.pexels.com/video/7080903/), and [7480858](https://www.pexels.com/video/7480858/). Pexels permits its media to be used and modified for free and does not require attribution. The source links are retained for provenance and creator credit.
+The companion Hugging Face repository distributes 20 modified 121-frame excerpts for running the public demo. They are installed under the location above on demand. Their filenames, source videos, extraction ranges, and checksums are recorded in the pinned [Pexels asset manifest](https://huggingface.co/AntResearch/4DAnyone/blob/4c80e87b805a5f8461cf339cdbe2fb4249e585aa/data/source/pexels/README.md). Pexels permits its media to be used and modified for free and does not require attribution. The source links are retained for provenance and creator credit.
## DiffSynth-Studio inference runtime
@@ -74,6 +74,10 @@ The 4DAnyone checkpoint and sanitized MHR70 sparse regression tensors are first-
The frozen base-model assets come from the official [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B) repository at revision `37685f96025fc1425edccdd4b2bca3836ae917ff` and the official [Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) repository at revision `3f40b6dc4ca5c02dd23c9db74d9d2ccb82903b86`. The latter is used offline to produce the fixed prompt-conditioning tensor; its T5 weights and tokenizer are not part of reader downloads. Their official model cards license those model assets under Apache-2.0. The redistributed copies are pinned by the frozen Hugging Face revision.
+### Wan2.2 TI2V 5B Turbo LoRA
+
+`4danyone/Wan22_TI2V_5B_Turbo_lora_rank_64_fp16.safetensors` is a byte-for-byte mirror of the Rank64 difference-LoRA published by [`Kijai/WanVideo_comfy`](https://huggingface.co/Kijai/WanVideo_comfy) at revision `86c2b0442e01eeee630b48fd7efc0cd37af03252`. Kijai identifies its source model as [`quanhaol/Wan2.2-TI2V-5B-Turbo`](https://huggingface.co/quanhaol/Wan2.2-TI2V-5B-Turbo), whose weights are published under [CC BY-NC-SA 4.0](https://github.com/quanhaol/Wan2.2-TI2V-5B-Turbo/blob/main/LICENSE.md). The adapter is redistributed under those upstream terms. Kijai declares no separate adapter license, and the safetensors metadata is empty. Its exact SHA-256 remains pinned in `fdanyone/assets.py`; the model card retains the full upstream file identity and provenance.
+
The MHR70 asset is a sanitized inference artifact containing only the sparse support/weight tensors, keypoint names, and a path-free structural schema. The first-party regression tensors are redistributed under Apache-2.0. The embedded name/order metadata remains covered by the schema notice above.
That regressor was trained by the 4DAnyone project on DNA-Rendering frames, using outputs from [SAM 3D Body](https://github.com/facebookresearch/sam-3d-body) and the [Momentum Human Rig](https://github.com/facebookresearch/MHR) conversion tools. SAM 3D Body is governed by the SAM License, MHR source is Apache-2.0, and DNA-Rendering access requires a separate signed dataset agreement. The regressor is not bundled in the source repository. Its Hugging Face model card must retain these source-model and dataset citations alongside the split tensor/schema license declaration in this notice.
diff --git a/docs/inference_performance.md b/docs/inference_performance.md
index 806b546..d378867 100644
--- a/docs/inference_performance.md
+++ b/docs/inference_performance.md
@@ -1,6 +1,6 @@
# Inference performance
-This page reports inference efficiency for the [6-view full-orbit](../README.md#6-view-full-orbit) example using:
+This page reports matched inference efficiency for the default Wan2.2 5B Turbo LoRA and the unfused base model on the [6-view full-orbit](../README.md#6-view-full-orbit) example:
```bash
python inference.py \
@@ -8,30 +8,52 @@ python inference.py \
--views_per_layer 6
```
-Each configuration reports the median of three runs.
+The command above uses Turbo by default. Add `--enable_turbo=False` to run the base model. Each denoising configuration below is the median of three complete runs from the same source revision.
## Denoising
-This table compares denoising time across attention backends.
+These tables compare denoising time across attention backends for the Turbo and base models.
- `SDPA`, `FlashAttention-3`, and `SageAttention`: denoising time using each attention backend.
-- `Peak CUDA allocated`: maximum live tensor allocation during generation. It was identical across the tested backends on each GPU.
+- `Peak CUDA allocated`: maximum live tensor allocation observed across the three runs and available attention backends for each GPU and model profile.
- `GPU memory`: total device memory capacity.
- `—`: backend unavailable on that GPU.
+### Turbo (default)
+
+| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GiB) | GPU memory (GiB) |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| H200 | 0.98 | 0.77 | 0.78 | 26.63 | 139.81 |
+| H20-3E | 3.22 | 2.55 | 2.10 | 26.63 | 139.81 |
+| RTX 5880 Ada | 3.51 | — | 3.04 | 25.47 | 47.37 |
+| RTX A6000 | 3.80 | — | 3.71 | 24.91 | 47.53 |
+
+### Base
+
| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GiB) | GPU memory (GiB) |
| --- | ---: | ---: | ---: | ---: | ---: |
-| H200 | 8.87 | 7.55 | 7.59 | 25.37 | 139.81 |
-| H20-3E | 19.55 | 15.63 | 12.89 | 25.37 | 139.81 |
-| RTX 5880 Ada | 20.28 | — | 17.51 | 24.21 | 47.37 |
-| RTX A6000 | 22.41 | — | 21.78 | 24.06 | 47.53 |
+| H200 | 5.35 | 4.03 | 4.09 | 26.63 | 139.81 |
+| H20-3E | 18.71 | 14.73 | 11.98 | 26.63 | 139.81 |
+| RTX 5880 Ada | 20.40 | — | 17.58 | 25.47 | 47.37 |
+| RTX A6000 | 22.02 | — | 21.42 | 24.91 | 47.53 |
+
+### Base / Turbo speedup
+
+Each value divides the Base median by the Turbo median for the same GPU and attention backend.
+
+| GPU | SDPA | FlashAttention-3 | SageAttention |
+| --- | ---: | ---: | ---: |
+| H200 | 5.45× | 5.26× | 5.23× |
+| H20-3E | 5.81× | 5.78× | 5.69× |
+| RTX 5880 Ada | 5.81× | — | 5.79× |
+| RTX A6000 | 5.79× | — | 5.77× |
> [!NOTE]
-> SageAttention results use the official `sageattn` entry point, which automatically dispatches an architecture-specific kernel for each GPU.
+> FlashAttention-3 is available on the tested SM90 GPUs and unavailable on RTX A6000 (SM86) and RTX 5880 Ada (SM89). SageAttention results use the official `sageattn` entry point, which dispatches an architecture-specific kernel.
## Preprocessing
-This table reports preprocessing time before generation.
+Preprocessing is shared by Turbo and Base, so it is reported once. These profile-independent values are retained from the preceding release benchmark; this refresh remeasured the matched denoising path.
- `GVHMR`: tracking, ViTPose, image-feature extraction, and motion prediction.
- `Skeleton extraction`: target-view skeleton-condition rendering.
diff --git a/docs/output.md b/docs/output.md
index 3f01789..4d6d0fc 100644
--- a/docs/output.md
+++ b/docs/output.md
@@ -24,7 +24,7 @@ data/
- `videos/sparse/` contains intermediate proposal views for larger camera layouts.
- `skeletons/` contains pose-conditioning videos aligned with the target views.
- `cameras.json` contains camera intrinsics and poses for every target view.
-- `metadata.json` records the input, generation settings, model versions, timings, and worker metrics when target denoising uses multiple GPUs.
+- `metadata.json` records the input, denoising profile, model identities, timings, and worker metrics when target denoising uses multiple GPUs.
- `gvhmr/results/` caches the recovered input motion for reuse.
All output videos contain 121 frames at 704×1280, use the selected output FPS, and contain no audio.
diff --git a/fdanyone/assets.py b/fdanyone/assets.py
index 284c5f7..629f300 100644
--- a/fdanyone/assets.py
+++ b/fdanyone/assets.py
@@ -11,9 +11,10 @@ from dataclasses import dataclass
from pathlib import Path
from fdanyone.errors import AssetError
+from fdanyone.io import sha256_file
HF_REPO_ID = "AntResearch/4DAnyone"
-HF_REVISION = "442816913e7cc75be2ede1a5c93a86d936d032f1"
+HF_REVISION = "4c80e87b805a5f8461cf339cdbe2fb4249e585aa"
BIREFNET_REPO_ID = "ZhengPeng7/BiRefNet"
BIREFNET_REVISION = "e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4"
@@ -30,6 +31,11 @@ MHR70_REGRESSOR = "4danyone/smplx_to_goliath70.pt"
WAN_VAE = "4danyone/Wan2.2_VAE.pth"
PROMPT_CONTEXT = "4danyone/prompt_context.safetensors"
+TURBO_LORA = "4danyone/Wan22_TI2V_5B_Turbo_lora_rank_64_fp16.safetensors"
+TURBO_LORA_NAME = "wan22_ti2v_5b_turbo_lora"
+TURBO_LORA_SIZE_BYTES = 332_348_584
+TURBO_LORA_SHA256 = "0ace5244e3d1256f884662c261b017249796cf5b95f05d5ed93cc02a478967b8"
+
GVHMR_CHECKPOINT = "gvhmr/gvhmr_siga24_release.ckpt"
HMR2_CHECKPOINT = "gvhmr/epoch=10-step=25000.ckpt"
VITPOSE_CHECKPOINT = "gvhmr/vitpose-h-multi-coco.pth"
@@ -48,6 +54,7 @@ MODEL_FILES = (
VITPOSE_CHECKPOINT,
YOLO_CHECKPOINT,
PERCEPTUAL_VGG19,
+ TURBO_LORA,
)
EXAMPLE_FILES = (
@@ -106,6 +113,19 @@ def resolve_checkpoint(path: str | Path | None = None, model_dir: str | Path = "
return _require_file(Path(model_dir) / CHECKPOINT, "Checkpoint", "scripts/download_model.py")
+def resolve_turbo_lora(model_dir: str | Path = "models") -> Path:
+ """Resolve and authenticate the exact Wan2.2 5B Turbo LoRA."""
+
+ resolved = _require_file(Path(model_dir) / TURBO_LORA, "Turbo LoRA", "scripts/download_model.py")
+ size = resolved.stat().st_size
+ if size != TURBO_LORA_SIZE_BYTES:
+ raise AssetError(f"Turbo LoRA size mismatch: {size} != {TURBO_LORA_SIZE_BYTES} bytes ({resolved})")
+ digest = sha256_file(resolved)
+ if digest != TURBO_LORA_SHA256:
+ raise AssetError(f"Turbo LoRA SHA-256 mismatch: {digest} != {TURBO_LORA_SHA256} ({resolved})")
+ return resolved
+
+
def resolve_regressor(path: str | Path | None = None, model_dir: str | Path = "models") -> Path:
if path is not None:
resolved = Path(path).expanduser().resolve()
diff --git a/fdanyone/config.py b/fdanyone/config.py
index 355c543..d203898 100644
--- a/fdanyone/config.py
+++ b/fdanyone/config.py
@@ -28,14 +28,43 @@ class InferenceConfig:
target_h264_crf: int = 18
h264_preset: str = "medium"
skeleton_max_dimension: int = 2048
- num_inference_steps: int = 24
- denoising_strength: float = 1.0
- scheduler_shift: float = 5.0
tiled_vae: bool = False
vae_tile_size: tuple[int, int] = (52, 30)
vae_tile_stride: tuple[int, int] = (26, 15)
+@dataclass(frozen=True)
+class DenoisingProfile:
+ """Scheduler and target-context routing used by one inference mode."""
+
+ name: str
+ num_inference_steps: int
+ denoising_strength: float
+ scheduler_shift: float
+ tcr_stride: int
+ freeze_tcr_after_one_cycle: bool
+
+ def __post_init__(self) -> None:
+ if self.num_inference_steps <= 0:
+ raise ValueError("Denoising steps must be positive.")
+ if not 0.0 < self.denoising_strength <= 1.0:
+ raise ValueError("Denoising strength must be in (0, 1].")
+ if self.scheduler_shift <= 0.0:
+ raise ValueError("Scheduler shift must be positive.")
+ if self.tcr_stride <= 0:
+ raise ValueError("TCR stride must be positive.")
+
+ def to_dict(self) -> dict[str, object]:
+ return {
+ "name": self.name,
+ "num_inference_steps": self.num_inference_steps,
+ "denoising_strength": self.denoising_strength,
+ "scheduler_shift": self.scheduler_shift,
+ "tcr_stride": self.tcr_stride,
+ "freeze_tcr_after_one_cycle": self.freeze_tcr_after_one_cycle,
+ }
+
+
@dataclass(frozen=True)
class CameraConfig:
count: int = 24
@@ -95,6 +124,22 @@ class SkeletonConfig:
INFERENCE = InferenceConfig()
+BASE24 = DenoisingProfile(
+ name="base24",
+ num_inference_steps=24,
+ denoising_strength=1.0,
+ scheduler_shift=5.0,
+ tcr_stride=1,
+ freeze_tcr_after_one_cycle=False,
+)
+RANK64_DELTA4 = DenoisingProfile(
+ name="rank64_delta4",
+ num_inference_steps=4,
+ denoising_strength=1.0,
+ scheduler_shift=5.0,
+ tcr_stride=2,
+ freeze_tcr_after_one_cycle=True,
+)
CAMERA = CameraConfig()
FOREGROUND = ForegroundConfig()
FRAMING = FramingConfig()
diff --git a/fdanyone/io.py b/fdanyone/io.py
index 1503c53..556c591 100644
--- a/fdanyone/io.py
+++ b/fdanyone/io.py
@@ -3,6 +3,7 @@
from __future__ import annotations
import errno
+import hashlib
import json
import os
import shutil
@@ -16,6 +17,16 @@ from fdanyone.errors import FourDAnyoneError
_RETRYABLE_TREE_ERRORS = {errno.EBUSY, errno.ENOTEMPTY, errno.ESTALE}
+def sha256_file(path: str | Path, *, chunk_size: int = 8 * 1024 * 1024) -> str:
+ """Hash a file without materializing it in memory."""
+
+ digest = hashlib.sha256()
+ with Path(path).open("rb") as handle:
+ for chunk in iter(lambda: handle.read(chunk_size), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
def write_json(path: str | Path, value: object, *, sort_keys: bool = True) -> None:
target = Path(path)
target.parent.mkdir(parents=True, exist_ok=True)
diff --git a/fdanyone/model/distributed.py b/fdanyone/model/distributed.py
index 2b753bf..a393e17 100644
--- a/fdanyone/model/distributed.py
+++ b/fdanyone/model/distributed.py
@@ -25,6 +25,7 @@ from fdanyone.errors import FourDAnyoneError
if TYPE_CHECKING:
from torch import Tensor
+ from fdanyone.config import DenoisingProfile
from fdanyone.model.conditioning import PoseFeatureBank
from fdanyone.model.loader import Denoiser
@@ -51,11 +52,11 @@ class WorkerReport(TypedDict):
@dataclass(frozen=True)
class DistributedDenoiseRequest:
checkpoint_path: str
- work_dir: str
- devices: tuple[str, ...]
+ turbo_lora_path: str | None
+ denoising_profile: DenoisingProfile
routes: Routes
- denoising_strength: float
- scheduler_shift: float
+ devices: tuple[str, ...]
+ work_dir: str
pose_feature_shape: tuple[int, ...]
init_method: str
@@ -260,11 +261,6 @@ def _load_worker_state(rank: int, request: DistributedDenoiseRequest, denoiser:
size=math.prod(request.pose_feature_shape),
dtype=torch.bfloat16,
).view(request.pose_feature_shape)
- denoiser.scheduler.set_timesteps(
- len(request.routes),
- denoising_strength=request.denoising_strength,
- shift=request.scheduler_shift,
- )
return _WorkerState(
rank=rank,
request=request,
@@ -321,8 +317,12 @@ def _worker(rank: int, request: DistributedDenoiseRequest) -> None:
resolved_backend = get_attention_backend()
model_started = time.monotonic()
- denoiser = load_denoiser(checkpoint_path=request.checkpoint_path)
- denoiser.model.to(device)
+ denoiser = load_denoiser(
+ checkpoint_path=request.checkpoint_path,
+ turbo_lora_path=request.turbo_lora_path,
+ profile=request.denoising_profile,
+ )
+ denoiser.prepare_on_device(device)
torch.cuda.synchronize(device_index)
model_load_seconds = time.monotonic() - model_started
@@ -360,16 +360,16 @@ def _worker(rank: int, request: DistributedDenoiseRequest) -> None:
def denoise_targets_distributed(
*,
+ checkpoint_path: str | Path,
+ turbo_lora_path: str | Path | None,
+ denoising_profile: DenoisingProfile,
+ routes: Routes,
src_latents: Tensor,
context: Tensor,
initial_latents: Tensor,
pose_features: PoseFeatureBank,
- routes: Routes,
- checkpoint_path: str | Path,
work_dir: str | Path,
devices: Sequence[str],
- denoising_strength: float,
- scheduler_shift: float,
) -> tuple[Tensor, list[WorkerReport]]:
"""Prepare shared inputs, launch NCCL workers, and return canonical latents."""
@@ -382,6 +382,11 @@ def denoise_targets_distributed(
if not torch.distributed.is_available() or not torch.distributed.is_nccl_available():
raise FourDAnyoneError("Multi-GPU inference requires a PyTorch build with NCCL support.")
validate_routes(routes, int(initial_latents.shape[0]))
+ if len(routes) != denoising_profile.num_inference_steps:
+ raise ValueError(
+ f"Denoising profile {denoising_profile.name!r} requires "
+ f"{denoising_profile.num_inference_steps} routing steps, got {len(routes)}."
+ )
num_groups = len(routes[0])
root = _resolve_empty_workspace(work_dir)
@@ -400,11 +405,11 @@ def denoise_targets_distributed(
)
request = DistributedDenoiseRequest(
checkpoint_path=str(Path(checkpoint_path).expanduser().resolve()),
- work_dir=str(root),
- devices=devices,
+ turbo_lora_path=(None if turbo_lora_path is None else str(Path(turbo_lora_path).expanduser().resolve())),
+ denoising_profile=denoising_profile,
routes=routes,
- denoising_strength=denoising_strength,
- scheduler_shift=scheduler_shift,
+ devices=devices,
+ work_dir=str(root),
pose_feature_shape=pose_feature_shape,
init_method=f"tcp://127.0.0.1:{_free_local_port()}",
)
diff --git a/fdanyone/model/inference.py b/fdanyone/model/inference.py
index 9c74cf8..2b8e6c5 100644
--- a/fdanyone/model/inference.py
+++ b/fdanyone/model/inference.py
@@ -18,11 +18,21 @@ from tempfile import TemporaryDirectory
from typing import TYPE_CHECKING, TypedDict
from fdanyone.assets import BaseAssets
-from fdanyone.config import INFERENCE
+from fdanyone.config import INFERENCE, DenoisingProfile
from fdanyone.errors import FourDAnyoneError
-from fdanyone.model.conditioning import PoseFeatureBank, PoseFeatureCache, build_pose_feature_cache, load_prompt_context
+from fdanyone.model.conditioning import (
+ PoseFeatureBank,
+ PoseFeatureCache,
+ build_pose_feature_cache,
+ load_prompt_context,
+)
from fdanyone.model.denoise import denoise_group
-from fdanyone.model.distributed import Routes, WorkerReport, denoise_targets_distributed, select_worker_devices
+from fdanyone.model.distributed import (
+ Routes,
+ WorkerReport,
+ denoise_targets_distributed,
+ select_worker_devices,
+)
from fdanyone.model.loader import Denoiser, load_denoiser
from fdanyone.model.metrics import GenerationMetrics
from fdanyone.model.routing import routing_steps
@@ -32,7 +42,7 @@ from fdanyone.video import CanonicalClip
from fdanyone.views import ViewPlan
if TYPE_CHECKING:
- from torch import Tensor, nn
+ from torch import Tensor
LOGGER = logging.getLogger("fdanyone")
@@ -55,6 +65,7 @@ class GeneratedViews:
rcp_videos: tuple[Path, ...]
target_videos: tuple[Path, ...]
view_plan: ViewPlan
+ denoising_profile: DenoisingProfile
seed: int
device: str
elapsed_seconds: dict[str, float]
@@ -69,12 +80,11 @@ class _GenerationPlan:
view_plan: ViewPlan
candidate_devices: tuple[str, ...]
primary_device: str
- primary_device_index: int
dit_devices: tuple[str, ...]
@property
- def view_devices(self) -> tuple[str, ...]:
- return self.candidate_devices
+ def primary_device_index(self) -> int:
+ return int(self.primary_device.removeprefix("cuda:"))
@property
def distributed(self) -> bool:
@@ -94,15 +104,15 @@ def _empty_cuda_cache() -> None:
@contextmanager
-def _model_on_device(model: nn.Module, device: str) -> Iterator[nn.Module]:
- """Keep one model resident for exactly one exception-safe stage."""
+def _denoiser_on_device(denoiser: Denoiser, device: str) -> Iterator[None]:
+ """Keep one denoiser resident and Turbo-fused for one safe stage."""
- model.to(device)
- _empty_cuda_cache()
try:
- yield model
+ denoiser.prepare_on_device(device)
+ _empty_cuda_cache()
+ yield
finally:
- model.to("cpu")
+ denoiser.model.to("cpu")
_empty_cuda_cache()
@@ -124,7 +134,14 @@ def _channels_last_source_layout(video):
return video.contiguous(memory_format=torch.channels_last_3d)
-def _noise(vae: VaeExecutor, num_views: int, num_frames: int, seed: int, device: str) -> Tensor:
+def _noise(
+ *,
+ vae: VaeExecutor,
+ num_views: int,
+ num_frames: int,
+ seed: int,
+ device: str,
+) -> Tensor:
import torch
shape = (
@@ -141,26 +158,28 @@ def _noise(vae: VaeExecutor, num_views: int, num_frames: int, seed: int, device:
def _denoise_rcp(
+ *,
denoiser: Denoiser,
vae: VaeExecutor,
src_latents: Tensor,
context: Tensor,
camera_ids: tuple[int, ...],
pose_features: PoseFeatureBank,
- device: str,
seed: int,
+ device: str,
) -> Tensor:
import torch
from tqdm.auto import tqdm
if pose_features.num_features != len(camera_ids):
raise FourDAnyoneError(f"RCP requires {len(camera_ids)} pose features, got {pose_features.num_features}.")
- denoiser.scheduler.set_timesteps(
- INFERENCE.num_inference_steps,
- denoising_strength=INFERENCE.denoising_strength,
- shift=INFERENCE.scheduler_shift,
+ latents = _noise(
+ vae=vae,
+ num_views=len(camera_ids),
+ num_frames=INFERENCE.num_frames,
+ seed=seed,
+ device=device,
)
- latents = _noise(vae, len(camera_ids), INFERENCE.num_frames, seed, device)
source = src_latents.to(dtype=denoiser.dtype, device=device)
context = context.to(dtype=denoiser.dtype, device=device)
pose_feature_batch = pose_features.allocate_group(len(camera_ids), device)
@@ -182,6 +201,7 @@ def _denoise_rcp(
def _denoise_targets_single(
+ *,
denoiser: Denoiser,
src_latents: Tensor,
context: Tensor,
@@ -198,11 +218,12 @@ def _denoise_targets_single(
raise FourDAnyoneError(
f"Target generation requires {num_views} pose features, got {pose_features.num_features}."
)
- denoiser.scheduler.set_timesteps(
- INFERENCE.num_inference_steps,
- denoising_strength=INFERENCE.denoising_strength,
- shift=INFERENCE.scheduler_shift,
- )
+ num_timesteps = len(denoiser.scheduler.timesteps)
+ if len(routes) != num_timesteps:
+ raise FourDAnyoneError(
+ f"Denoising requires one route per scheduler timestep; got "
+ f"{len(routes)} routes for {num_timesteps} timesteps."
+ )
latents = initial_latents
source = src_latents.to(dtype=denoiser.dtype, device=device)
context = context.to(dtype=denoiser.dtype, device=device)
@@ -231,17 +252,15 @@ def _denoise_targets_single(
def _resolve_generation_plan(
+ *,
clip: CanonicalClip,
conditioning: Conditioning,
devices: tuple[str, ...],
- seed: int,
) -> _GenerationPlan:
import torch
if conditioning.num_frames != INFERENCE.num_frames or len(clip.frames) != INFERENCE.num_frames:
raise FourDAnyoneError("Generation requires the frozen 121-frame contract.")
- if seed < 0:
- raise FourDAnyoneError(f"seed must be non-negative, got {seed}.")
if not devices:
raise FourDAnyoneError("Generation requires at least one CUDA device.")
@@ -254,7 +273,11 @@ def _resolve_generation_plan(
primary_device = devices[0]
primary_device_index = int(primary_device.removeprefix("cuda:"))
dit_devices = select_worker_devices(devices, view_plan.num_groups)
- LOGGER.info("Using %s (%s)", primary_device, torch.cuda.get_device_name(primary_device_index))
+ LOGGER.info(
+ "Using %s (%s)",
+ primary_device,
+ torch.cuda.get_device_name(primary_device_index),
+ )
if len(dit_devices) < len(devices):
LOGGER.info(
"Using %d of %d candidate GPUs for DiT and all %d for independent view stages",
@@ -266,7 +289,6 @@ def _resolve_generation_plan(
view_plan=view_plan,
candidate_devices=devices,
primary_device=primary_device,
- primary_device_index=primary_device_index,
dit_devices=dit_devices,
)
@@ -275,13 +297,13 @@ def _generate_rcp_and_references(
*,
denoiser: Denoiser,
vae: VaeExecutor,
+ clip: CanonicalClip,
+ plan: _GenerationPlan,
+ seed: int,
source_latents: Tensor,
context: Tensor,
pose_cache: PoseFeatureCache,
- plan: _GenerationPlan,
root: Path,
- clip: CanonicalClip,
- seed: int,
metrics: GenerationMetrics,
) -> tuple[Tensor, tuple[Path, ...]]:
import torch
@@ -289,16 +311,19 @@ def _generate_rcp_and_references(
rcp_pose_features = pose_cache.rcp
if rcp_pose_features is None:
raise FourDAnyoneError("RCP was enabled without precomputed proposal pose features.")
- with metrics.stage("rcp_denoise"), _model_on_device(denoiser.model, plan.primary_device):
+ with (
+ metrics.stage("rcp_denoise"),
+ _denoiser_on_device(denoiser, plan.primary_device),
+ ):
rcp_latents = _denoise_rcp(
- denoiser,
- vae,
- source_latents,
- context,
- plan.view_plan.rcp_camera_ids,
- rcp_pose_features,
- plan.primary_device,
- seed,
+ denoiser=denoiser,
+ vae=vae,
+ src_latents=source_latents,
+ context=context,
+ camera_ids=plan.view_plan.rcp_camera_ids,
+ pose_features=rcp_pose_features,
+ seed=seed,
+ device=plan.primary_device,
)
with metrics.stage("rcp_decode_and_publish"):
@@ -334,48 +359,50 @@ def _merge_view_stage_peak(metrics: GenerationMetrics, stage: str, vae: VaeExecu
)
-def _target_routes(view_plan: ViewPlan) -> Routes:
+def _target_routes(*, view_plan: ViewPlan, profile: DenoisingProfile) -> Routes:
return routing_steps(
views_per_layer=view_plan.views_per_layer,
num_layers=view_plan.num_layers,
group_size=view_plan.views_per_group,
- num_steps=INFERENCE.num_inference_steps,
+ num_steps=profile.num_inference_steps,
enable_tcr=view_plan.enable_tcr,
+ tcr_stride=profile.tcr_stride,
+ freeze_after_one_cycle=profile.freeze_tcr_after_one_cycle,
)
def _denoise_targets_multi_gpu(
*,
+ checkpoint_path: str | Path,
+ turbo_lora_path: str | Path | None,
+ denoising_profile: DenoisingProfile,
+ plan: _GenerationPlan,
target_sources: Tensor,
context: Tensor,
initial_latents: Tensor,
pose_features: PoseFeatureBank,
routes: Routes,
- checkpoint_path: str | Path,
root: Path,
- devices: tuple[str, ...],
- candidate_devices: tuple[str, ...],
- num_groups: int,
) -> tuple[Tensor, ParallelismReport]:
with TemporaryDirectory(prefix=".distributed-", dir=root) as work_dir:
target_latents, workers = denoise_targets_distributed(
+ checkpoint_path=checkpoint_path,
+ turbo_lora_path=turbo_lora_path,
+ denoising_profile=denoising_profile,
src_latents=target_sources,
context=context,
initial_latents=initial_latents,
pose_features=pose_features,
routes=routes,
- checkpoint_path=checkpoint_path,
work_dir=work_dir,
- devices=devices,
- denoising_strength=INFERENCE.denoising_strength,
- scheduler_shift=INFERENCE.scheduler_shift,
+ devices=plan.dit_devices,
)
return target_latents, {
"backend": "nccl",
- "candidate_devices": list(candidate_devices),
- "used_devices": list(devices),
- "groups_per_step": num_groups,
- "waves_per_step": math.ceil(num_groups / len(devices)),
+ "candidate_devices": list(plan.candidate_devices),
+ "used_devices": list(plan.dit_devices),
+ "groups_per_step": plan.view_plan.num_groups,
+ "waves_per_step": math.ceil(plan.view_plan.num_groups / len(plan.dit_devices)),
"workers": workers,
}
@@ -385,6 +412,8 @@ def generate_views(
clip: CanonicalClip,
conditioning: Conditioning,
checkpoint_path: str | Path,
+ turbo_lora_path: str | Path | None,
+ denoising_profile: DenoisingProfile,
assets: BaseAssets,
output_dir: str | Path,
devices: tuple[str, ...],
@@ -392,7 +421,13 @@ def generate_views(
) -> GeneratedViews:
"""Generate the proposal (when enabled) and the requested target views."""
- plan = _resolve_generation_plan(clip, conditioning, devices, seed)
+ if seed < 0:
+ raise FourDAnyoneError(f"seed must be non-negative, got {seed}.")
+ plan = _resolve_generation_plan(
+ clip=clip,
+ conditioning=conditioning,
+ devices=devices,
+ )
root = Path(output_dir).expanduser().resolve()
root.mkdir(parents=True, exist_ok=False)
metrics = GenerationMetrics(plan.primary_device_index)
@@ -404,12 +439,20 @@ def generate_views(
pose_cache = build_pose_feature_cache(
conditioning=conditioning,
checkpoint_path=checkpoint_path,
- devices=plan.view_devices,
+ devices=plan.candidate_devices,
)
with metrics.stage("model_load"):
- denoiser = load_denoiser(checkpoint_path=checkpoint_path) if plan.needs_primary_denoiser else None
- vae = VaeExecutor.load(assets.vae, plan.view_devices)
+ denoiser = (
+ load_denoiser(
+ checkpoint_path=checkpoint_path,
+ turbo_lora_path=turbo_lora_path,
+ profile=denoising_profile,
+ )
+ if plan.needs_primary_denoiser
+ else None
+ )
+ vae = VaeExecutor.load(assets.vae, plan.candidate_devices)
try:
with metrics.stage("source_encode"):
@@ -426,13 +469,13 @@ def generate_views(
target_sources, rcp_videos = _generate_rcp_and_references(
denoiser=denoiser,
vae=vae,
+ clip=clip,
+ plan=plan,
+ seed=seed,
source_latents=source_latents,
context=context,
pose_cache=pose_cache,
- plan=plan,
root=root,
- clip=clip,
- seed=seed,
metrics=metrics,
)
del source_latents
@@ -444,42 +487,42 @@ def generate_views(
parallelism = None
with metrics.stage("target_denoise"):
- routes = _target_routes(plan.view_plan)
+ routes = _target_routes(view_plan=plan.view_plan, profile=denoising_profile)
initial_latents = _noise(
- vae,
- plan.view_plan.num_target_views,
- INFERENCE.num_frames,
- seed,
- plan.primary_device,
+ vae=vae,
+ num_views=plan.view_plan.num_target_views,
+ num_frames=INFERENCE.num_frames,
+ seed=seed,
+ device=plan.primary_device,
)
if plan.distributed:
denoiser = None
initial_latents = initial_latents.to("cpu")
_empty_cuda_cache()
target_latents, parallelism = _denoise_targets_multi_gpu(
+ checkpoint_path=checkpoint_path,
+ turbo_lora_path=turbo_lora_path,
+ denoising_profile=denoising_profile,
+ plan=plan,
target_sources=target_sources,
context=context,
initial_latents=initial_latents,
pose_features=target_pose_features,
routes=routes,
- checkpoint_path=checkpoint_path,
root=root,
- devices=plan.dit_devices,
- candidate_devices=plan.candidate_devices,
- num_groups=plan.view_plan.num_groups,
)
else:
if denoiser is None:
raise RuntimeError("Single-GPU target generation requires a primary-process denoiser.")
- with _model_on_device(denoiser.model, plan.primary_device):
+ with _denoiser_on_device(denoiser, plan.primary_device):
target_latents = _denoise_targets_single(
- denoiser,
- target_sources,
- context,
- target_pose_features,
- initial_latents,
- routes,
- plan.primary_device,
+ denoiser=denoiser,
+ src_latents=target_sources,
+ context=context,
+ pose_features=target_pose_features,
+ initial_latents=initial_latents,
+ routes=routes,
+ device=plan.primary_device,
)
denoiser = None
del initial_latents
@@ -503,6 +546,7 @@ def generate_views(
rcp_videos=rcp_videos,
target_videos=target_videos,
view_plan=plan.view_plan,
+ denoising_profile=denoising_profile,
seed=seed,
device=plan.primary_device,
elapsed_seconds=metrics.elapsed_seconds,
diff --git a/fdanyone/model/loader.py b/fdanyone/model/loader.py
index 3964297..d6a68fb 100644
--- a/fdanyone/model/loader.py
+++ b/fdanyone/model/loader.py
@@ -2,10 +2,11 @@
from __future__ import annotations
-from dataclasses import dataclass
+from dataclasses import dataclass, field
from pathlib import Path
from typing import TYPE_CHECKING
+from fdanyone.config import DenoisingProfile
from fdanyone.errors import AssetError, ConfigurationError
if TYPE_CHECKING:
@@ -17,13 +18,25 @@ if TYPE_CHECKING:
POSE_ENCODER_PREFIX = "pose_encoder."
-@dataclass(frozen=True)
+@dataclass
class Denoiser:
- """The exact DiT, scheduler, and dtype used by one denoising process."""
+ """A loaded DiT, its configured scheduler, and optional Turbo fusion."""
model: nn.Module
scheduler: FlowMatchScheduler
dtype: torch.dtype
+ turbo_lora_path: Path | None = None
+ turbo_lora_applied: bool = field(default=False, init=False)
+
+ def prepare_on_device(self, device: str | torch.device) -> None:
+ """Move the DiT to a device and apply its configured Turbo delta once."""
+
+ self.model.to(device=device, dtype=self.dtype)
+ if self.turbo_lora_path is not None and not self.turbo_lora_applied:
+ from fdanyone.model.turbo_lora import fuse_turbo_lora
+
+ fuse_turbo_lora(self.model, self.turbo_lora_path)
+ self.turbo_lora_applied = True
def _load_checkpoint(
@@ -44,7 +57,10 @@ def _load_checkpoint(
if (include_prefix is None or key.startswith(include_prefix))
and not any(key.startswith(prefix) for prefix in exclude_prefixes)
)
- return {key: checkpoint.get_tensor(key) for key in keys}
+ return (
+ {key: checkpoint.get_tensor(key) for key in keys},
+ dict(checkpoint.metadata() or {}),
+ )
def _strict_assign(module, state_dict: dict, label: str) -> None:
@@ -75,13 +91,13 @@ def _load_dit(checkpoint_path: Path):
with torch.device("meta"):
dit = FourDAnyoneDiT()
- state_dict = _load_checkpoint(checkpoint_path, exclude_prefixes=(POSE_ENCODER_PREFIX,))
+ state_dict, metadata = _load_checkpoint(checkpoint_path, exclude_prefixes=(POSE_ENCODER_PREFIX,))
_strict_assign(dit, state_dict, "4DAnyone DiT checkpoint")
del state_dict
# ``freqs`` is a derived, non-persistent tensor and therefore is not in the
# state dict populated above.
dit.freqs = precompute_freqs_cis_3d(MODEL_DIM // NUM_HEADS)
- return dit.eval().requires_grad_(False)
+ return dit.eval().requires_grad_(False), metadata
def load_pose_encoder(checkpoint_path: str | Path, device: str):
@@ -92,10 +108,9 @@ def load_pose_encoder(checkpoint_path: str | Path, device: str):
from fdanyone.vendor.diffsynth.models.wan_video_dit import MODEL_DIM
from fdanyone.vendor.diffsynth.models.wan_video_pose_encoder import PoseEncoder
- state_dict = {
- key.removeprefix(POSE_ENCODER_PREFIX): value
- for key, value in _load_checkpoint(Path(checkpoint_path), include_prefix=POSE_ENCODER_PREFIX).items()
- }
+ checkpoint, _ = _load_checkpoint(Path(checkpoint_path), include_prefix=POSE_ENCODER_PREFIX)
+ state_dict = {key.removeprefix(POSE_ENCODER_PREFIX): value for key, value in checkpoint.items()}
+ del checkpoint
with torch.device("meta"):
pose_encoder = PoseEncoder(out_dim=MODEL_DIM, in_channels=3)
_strict_assign(pose_encoder, state_dict, "4DAnyone pose encoder checkpoint")
@@ -128,16 +143,40 @@ def load_vae(path: str | Path):
return _load_vae(Path(path), torch.bfloat16)
-def load_denoiser(*, checkpoint_path: str | Path) -> Denoiser:
- """Load the DiT and scheduler required by one distributed worker."""
+def load_denoiser(
+ *,
+ checkpoint_path: str | Path,
+ turbo_lora_path: str | Path | None,
+ profile: DenoisingProfile,
+) -> Denoiser:
+ """Load one DiT and configure its denoising trajectory."""
import torch
from fdanyone.vendor.diffsynth.schedulers.flow_match import FlowMatchScheduler
dtype = torch.bfloat16
+ checkpoint = Path(checkpoint_path).expanduser().resolve()
+ if not checkpoint.is_file():
+ raise AssetError(f"4DAnyone checkpoint does not exist: {checkpoint}")
+ turbo_path = None if turbo_lora_path is None else Path(turbo_lora_path).expanduser().resolve()
+ if turbo_path is not None and not turbo_path.is_file():
+ raise AssetError(f"Turbo LoRA does not exist: {turbo_path}")
+ model, metadata = _load_dit(checkpoint)
+ if turbo_path is not None:
+ from fdanyone.model.turbo_lora import validate_turbo_base_metadata
+
+ validate_turbo_base_metadata(metadata)
+ # ``step`` only reads the trajectory, so RCP and target generation can
+ # safely share one scheduler configured when this denoiser is loaded.
+ scheduler = FlowMatchScheduler(shift=profile.scheduler_shift, sigma_min=0.0, extra_one_step=True)
+ scheduler.set_timesteps(
+ profile.num_inference_steps,
+ denoising_strength=profile.denoising_strength,
+ )
return Denoiser(
- model=_load_dit(Path(checkpoint_path)),
- scheduler=FlowMatchScheduler(shift=5, sigma_min=0.0, extra_one_step=True),
+ model=model,
+ scheduler=scheduler,
dtype=dtype,
+ turbo_lora_path=turbo_path,
)
diff --git a/fdanyone/model/routing.py b/fdanyone/model/routing.py
index 2799979..3621411 100644
--- a/fdanyone/model/routing.py
+++ b/fdanyone/model/routing.py
@@ -30,6 +30,8 @@ def routing_steps(
group_size: int,
num_steps: int,
enable_tcr: bool,
+ tcr_stride: int = 1,
+ freeze_after_one_cycle: bool = False,
) -> tuple[tuple[tuple[int, ...], ...], ...]:
"""Return layer-local target groups for every denoising step."""
@@ -37,7 +39,18 @@ def routing_steps(
raise ValueError(f"num_steps must be positive, got {num_steps}.")
if num_layers <= 0:
raise ValueError(f"num_layers must be positive, got {num_layers}.")
+ if tcr_stride <= 0:
+ raise ValueError(f"tcr_stride must be positive, got {tcr_stride}.")
_validate_grouping(views_per_layer, group_size)
+
+ def step_offset(step_index: int) -> int:
+ if not enable_tcr:
+ return 0
+ offset = step_index * tcr_stride
+ if freeze_after_one_cycle and offset >= group_size:
+ return 0
+ return offset
+
return tuple(
tuple(
tuple(layer_index * views_per_layer + view_index for view_index in group)
@@ -45,7 +58,7 @@ def routing_steps(
for group in view_groups(
views_per_layer,
group_size,
- step_index if enable_tcr else 0,
+ step_offset(step_index),
)
)
for step_index in range(num_steps)
diff --git a/fdanyone/model/turbo_lora.py b/fdanyone/model/turbo_lora.py
new file mode 100644
index 0000000..8e0c1ca
--- /dev/null
+++ b/fdanyone/model/turbo_lora.py
@@ -0,0 +1,148 @@
+"""Apply the pinned Wan2.2 Turbo difference-LoRA to a resident DiT."""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from fdanyone.errors import AssetError
+
+if TYPE_CHECKING:
+ from torch import Tensor, nn
+
+_PREFIX = "diffusion_model."
+_DOWN = ".lora_down.weight"
+_UP = ".lora_up.weight"
+_BIAS_DIFF = ".diff_b"
+_TENSOR_DIFF = ".diff"
+_FUSED_METADATA_KEYS = (
+ "4danyone.lora_sha256",
+ "4danyone.turbo_lora_sha256",
+ "4danyone.turbo_lora",
+)
+
+
+def _stem(key: str, suffix: str) -> str:
+ stem = key[: -len(suffix)]
+ if not stem.startswith(_PREFIX):
+ raise ValueError(f"Turbo tensor lacks the {_PREFIX!r} prefix: {key}")
+ return stem.removeprefix(_PREFIX)
+
+
+def _direct_target(state_dict: Mapping[str, Tensor], stem: str) -> str:
+ candidates = tuple(key for key in (stem, f"{stem}.weight") if key in state_dict)
+ if len(candidates) != 1:
+ raise ValueError(f"Turbo tensor {stem!r} must resolve to one base tensor, got {candidates}.")
+ return candidates[0]
+
+
+def validate_turbo_base_metadata(base_metadata: Mapping[str, str] | None) -> None:
+ """Reject checkpoints that explicitly identify an earlier Turbo fusion."""
+
+ fused_markers = sorted(set(base_metadata or {}).intersection(_FUSED_METADATA_KEYS))
+ if fused_markers:
+ raise AssetError(f"Refusing to apply Turbo to an already-fused checkpoint: {fused_markers}.")
+
+
+def _fuse_low_rank_delta(state_dict: Mapping[str, Tensor], adapter: Mapping[str, Tensor], down_key: str) -> None:
+ import torch
+
+ up_key = f"{down_key[: -len(_DOWN)]}{_UP}"
+ if up_key not in adapter:
+ raise ValueError(f"Turbo LoRA-down tensor has no paired LoRA-up tensor: {down_key}")
+
+ target = f"{_stem(down_key, _DOWN)}.weight"
+ if target not in state_dict:
+ raise ValueError(f"Turbo tensor {down_key} targets missing base tensor {target}.")
+
+ base = state_dict[target]
+ down = adapter[down_key]
+ up = adapter[up_key]
+ if (
+ base.ndim != 2
+ or down.ndim != 2
+ or up.ndim != 2
+ or up.shape[1] != down.shape[0]
+ or tuple(base.shape) != (up.shape[0], down.shape[1])
+ ):
+ raise ValueError(
+ f"Turbo matrix shapes do not reconstruct {target}: "
+ f"base={tuple(base.shape)}, up={tuple(up.shape)}, down={tuple(down.shape)}."
+ )
+
+ merged = base.detach().to(dtype=torch.float32, copy=True)
+ merged.addmm_(up.float(), down.float())
+ base.copy_(merged)
+
+
+def _fuse_direct_delta(state_dict: Mapping[str, Tensor], adapter: Mapping[str, Tensor], key: str) -> None:
+ import torch
+
+ if key.endswith(_BIAS_DIFF):
+ target = f"{_stem(key, _BIAS_DIFF)}.bias"
+ elif key.endswith(_TENSOR_DIFF):
+ target = _direct_target(state_dict, _stem(key, _TENSOR_DIFF))
+ else:
+ raise ValueError(f"Unrecognized Turbo tensor suffix: {key}")
+
+ if target not in state_dict:
+ raise ValueError(f"Turbo tensor {key} targets missing base tensor {target}.")
+ base = state_dict[target]
+ delta = adapter[key]
+ if tuple(base.shape) != tuple(delta.shape):
+ raise ValueError(
+ f"Turbo tensor shape mismatch for {target}: base={tuple(base.shape)}, delta={tuple(delta.shape)}."
+ )
+
+ merged = base.detach().to(dtype=torch.float32, copy=True)
+ merged.add_(delta.float())
+ base.copy_(merged)
+
+
+def fuse_turbo_lora(model: nn.Module, adapter_path: str | Path) -> None:
+ """Fuse the fixed-strength Wan2.2 5B Turbo LoRA into one resident DiT.
+
+ The complete adapter remains in its serialized FP16 dtype on the target
+ device. Each target uses one temporary FP32 base tensor, converts only its
+ current adapter components to FP32, and copies the merged result back into
+ the original base storage and dtype.
+ """
+
+ state_dict = model.state_dict(keep_vars=True)
+
+ path = Path(adapter_path).expanduser().resolve()
+ if not path.is_file():
+ raise AssetError(f"Turbo LoRA does not exist: {path}")
+ if not state_dict:
+ raise AssetError("Cannot apply Turbo LoRA to an empty state dict.")
+
+ try:
+ import torch
+ from safetensors.torch import load_file
+
+ devices = {tensor.device for tensor in state_dict.values()}
+ if len(devices) != 1:
+ raise ValueError(f"Turbo base tensors must share one device, got {sorted(map(str, devices))}.")
+ device = devices.pop()
+ adapter = load_file(str(path), device=str(device))
+ adapter_dtypes = {tensor.dtype for tensor in adapter.values()}
+ if adapter_dtypes != {torch.float16}:
+ raise ValueError(
+ f"Turbo LoRA tensors must all be serialized as FP16, got {sorted(map(str, adapter_dtypes))}."
+ )
+ with torch.no_grad():
+ for key in sorted(adapter):
+ if key.endswith(_UP):
+ down_key = f"{key[: -len(_UP)]}{_DOWN}"
+ if down_key not in adapter:
+ raise ValueError(f"Turbo LoRA-up tensor has no paired LoRA-down tensor: {key}")
+ continue
+ if key.endswith(_DOWN):
+ _fuse_low_rank_delta(state_dict, adapter, key)
+ else:
+ _fuse_direct_delta(state_dict, adapter, key)
+ except ImportError as exc:
+ raise AssetError("PyTorch and safetensors are required to load the Turbo LoRA.") from exc
+ except (OSError, RuntimeError, ValueError) as exc:
+ raise AssetError(f"Turbo LoRA is incompatible with this 4DAnyone checkpoint: {exc}") from exc
diff --git a/fdanyone/output.py b/fdanyone/output.py
index 591d2e2..ea77fde 100644
--- a/fdanyone/output.py
+++ b/fdanyone/output.py
@@ -165,7 +165,8 @@ def export_result(
"groups_per_layer": view_plan.groups_per_layer,
},
"attention_backend": attention_backend,
- "inference_steps": INFERENCE.num_inference_steps,
+ "inference_steps": generated.denoising_profile.num_inference_steps,
+ "denoising_profile": generated.denoising_profile.to_dict(),
"elapsed_seconds": generated.elapsed_seconds,
"stage_peak_vram_bytes": generated.stage_peak_vram_bytes,
"total_elapsed_seconds": total_elapsed,
diff --git a/fdanyone/pipeline.py b/fdanyone/pipeline.py
index 9399a1a..f03c32e 100644
--- a/fdanyone/pipeline.py
+++ b/fdanyone/pipeline.py
@@ -14,12 +14,16 @@ from fdanyone.assets import (
CHECKPOINT,
HF_REPO_ID,
HF_REVISION,
+ TURBO_LORA,
+ TURBO_LORA_NAME,
+ TURBO_LORA_SHA256,
resolve_base_assets,
resolve_checkpoint,
resolve_foreground_model,
resolve_regressor,
+ resolve_turbo_lora,
)
-from fdanyone.config import INFERENCE
+from fdanyone.config import BASE24, INFERENCE, RANK64_DELTA4
from fdanyone.device import CUDA_ALLOCATOR_CONF, select_cuda_devices
from fdanyone.download import ensure_example_video, ensure_models, ensure_smplx
from fdanyone.errors import ConfigurationError
@@ -174,6 +178,7 @@ def run_pipeline(
data_dir: str,
model_dir: str,
checkpoint_path: str | None,
+ enable_turbo: bool,
mhr70_regressor_path: str | None,
gvhmr_root: str,
gpu_ids: list[int] | None,
@@ -194,6 +199,9 @@ def run_pipeline(
logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
if seed < 0:
raise ConfigurationError(f"seed must be non-negative, got {seed}.")
+ if not isinstance(enable_turbo, bool):
+ raise ConfigurationError(f"enable_turbo must be True or False, got {enable_turbo!r}.")
+ denoising_profile = RANK64_DELTA4 if enable_turbo else BASE24
view_plan = resolve_view_plan(
views_per_layer=views_per_layer,
layer_pitches=layer_pitches,
@@ -222,6 +230,7 @@ def run_pipeline(
# background jobs receive an actionable error instead of hanging.
ensure_smplx(model_dir, gvhmr_root)
ensure_models(model_dir, gvhmr_root)
+ turbo_lora = resolve_turbo_lora(model_dir) if enable_turbo else None
gvhmr_root, gvhmr_revision = validate_gvhmr(gvhmr_root)
worker_python = os.path.abspath(sys.executable)
@@ -273,6 +282,12 @@ def run_pipeline(
model_identity = {"checkpoint": CHECKPOINT, "repo_id": HF_REPO_ID, "revision": HF_REVISION}
else:
model_identity = {"checkpoint": checkpoint.name, "source": "local_override"}
+ if turbo_lora is not None:
+ model_identity["turbo_lora"] = {
+ "name": TURBO_LORA_NAME,
+ "file": TURBO_LORA,
+ "sha256": TURBO_LORA_SHA256,
+ }
with atomic as work:
# Heavy rendering and generation are imported only after the motion
@@ -306,6 +321,8 @@ def run_pipeline(
clip=clip,
conditioning=conditioning,
checkpoint_path=checkpoint,
+ turbo_lora_path=turbo_lora,
+ denoising_profile=denoising_profile,
assets=base_assets,
output_dir=scratch / "generation",
devices=devices,
diff --git a/inference.py b/inference.py
index 16a34f4..86de4a4 100644
--- a/inference.py
+++ b/inference.py
@@ -17,6 +17,7 @@ def inference(
views_per_group: int | str = "auto",
enable_rcp: bool = True,
enable_tcr: bool = True,
+ enable_turbo: bool = True,
data_dir: str = "data",
model_dir: str = "models",
checkpoint_path: str | None = None,
@@ -45,6 +46,8 @@ def inference(
enable_rcp: Use proposal views before generating more than six targets.
The proposal count follows views_per_group.
enable_tcr: Shift view groups cyclically between denoising steps.
+ enable_turbo: Use the Wan2.2 5B Turbo LoRA for accelerated denoising.
+ Disable it to use the unfused base model.
data_dir: Root for reusable GVHMR motion and final 4DAnyone outputs.
model_dir: Model root; missing public checkpoints download here.
checkpoint_path: Local 4DAnyone checkpoint override.
@@ -73,6 +76,7 @@ def inference(
views_per_group=views_per_group,
enable_rcp=enable_rcp,
enable_tcr=enable_tcr,
+ enable_turbo=enable_turbo,
data_dir=data_dir,
model_dir=model_dir,
checkpoint_path=checkpoint_path,