Skip to content

WorldPlay2: Extending Real-Time Interactive World Models in Control and Horizon

Python

release

aejionzhang6bb655c701e7
diff --git a/LICENSE.txt b/LICENSE.txt
new file mode 100644
index 0000000..4c3c3a9
--- /dev/null
+++ b/LICENSE.txt
@@ -0,0 +1 @@
+license: cc-by-nc-4.0
\ No newline at end of file
diff --git a/README.md b/README.md
index ae9b5d4..6ae35a1 100644
--- a/README.md
+++ b/README.md
@@ -4,13 +4,190 @@
<div align="center">
<a href=https://worldplay2.github.io/ target="_blank"><img src=https://img.shields.io/badge/Project%20Page-333399.svg?logo=homepage height=22px></a>
- <a href=https://arxiv.org/abs/xxxx.xxxxx target="_blank"><img src=https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv height=22px></a>
+ <a href=https://arxiv.org/abs/2609.35560 target="_blank"><img src=https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv height=22px></a>
<a href=https://www.youtube.com/watch?v=_pFzHHslhgc target="_blank"><img src=https://img.shields.io/badge/YouTube%20Video-FF0000.svg?logo=youtube height=22px></a>
</div>
-We are still organizing the code and weights. Thank you all for your patience!
-
## 🎥 Video
https://github.com/user-attachments/assets/3264efa7-ef6a-4c7b-a043-d819264a93bd
+
+## 🛠️ Installation
+
+Requires Python 3.10+ and a CUDA-capable NVIDIA GPU.
+
+Create and activate a Conda environment, then install the dependencies:
+
+```bash
+conda create -n worldplay2 python=3.10 -y
+conda activate worldplay2
+python -m pip install -r requirements.txt
+```
+
+Install `flash_attn`:
+
+```bash
+pip install flash-attn --no-build-isolation
+```
+
+#### Optional SageAttention
+
+Install SageAttention in your CUDA environment:
+
+```bash
+pip install sageattention==2.2.0 --no-build-isolation
+```
+
+All three launchers support `ATTENTION_BACKEND` (default: `flash`). For direct
+CLI usage, pass `--attention_backend sage`.
+
+#### Optional FP8
+
+Install the additional dependencies on supported hardware:
+
+```bash
+pip install transformer_engine[pytorch]
+```
+
+After setting the model and input paths in `run_few_step.sh` or exporting them
+as environment variables, enable FP8:
+
+```bash
+ENABLE_FP8_FFN=true ENABLE_FP8_QKV=true USE_DIT_FSDP=false bash run_few_step.sh
+```
+
+## 🧱 Model Download
+
+### Base Model
+
+Download [Wan2.2 I2V-A14B](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B)
+and set `CKPT_DIR` to its local directory.
+
+### WorldPlay2 Models
+
+Download the WorldPlay2 checkpoints for your inference mode. Set
+`LOW_NOISE_CKPT` and `HIGH_NOISE_CKPT` to the corresponding checkpoint files.
+
+<!-- TODO: Replace each # below with the corresponding Hugging Face model URL. -->
+
+| Model | Model Type | Default steps | Download Links |
+|---------------------|-------------------------------------|---------------| --- |
+| **WorldPlay2-Fast** | Autoregressive, 4-step (`few_step`) | 4 | [🤗 Hugging Face](https://huggingface.co/aejion/WorldPlay2-Fast) |
+| **WorldPlay2-BI** | Bidirectional (`bi`) | 40 | [🤗 Hugging Face](https://huggingface.co/aejion/WorldPlay2-BI) |
+| **WorldPlay2-AR** | Autoregressive (`ar`) | 40 | [🤗 Hugging Face](https://huggingface.co/aejion/WorldPlay2-AR) |
+| **WorldPlay2-TAE** | Causal VAE | --- | (coming soon) |
+
+Download the base model and WorldPlay2 few-step model using the Hugging Face CLI.
+
+```bash
+pip install "huggingface_hub[cli]>=0.34,<1.0"
+
+# Base Wan2.2 model
+hf download Wan-AI/Wan2.2-I2V-A14B --local-dir ./checkpoints/Wan2.2-I2V-A14B
+
+# WorldPlay2 models
+hf download aejion/WorldPlay2-Fast --local-dir ./checkpoints/WorldPlay2-Fast
+hf download aejion/WorldPlay2-BI --local-dir ./checkpoints/WorldPlay2-BI
+hf download aejion/WorldPlay2-AR --local-dir ./checkpoints/WorldPlay2-AR
+```
+
+Set `CKPT_DIR` to `./checkpoints/Wan2.2-I2V-A14B`, and set `LOW_NOISE_CKPT`
+and `HIGH_NOISE_CKPT` to the checkpoint files in `./checkpoints/WorldPlay2-Fast`.
+
+## 🎮 Quick Start
+
+Set the paths to the WorldPlay2 few-step checkpoints and run from the repository root:
+
+```bash
+# CKPT_DIR: path to the original Wan2.2 I2V-A14B base model directory.
+# LOW_NOISE_CKPT: path to the WorldPlay2 few-step low-noise checkpoint.
+# HIGH_NOISE_CKPT: path to the WorldPlay2 few-step high-noise checkpoint.
+CKPT_DIR=/path/to/model \
+LOW_NOISE_CKPT=/path/to/model/low_noise_model.pt \
+HIGH_NOISE_CKPT=/path/to/model/high_noise_model.pt \
+INPUT_JSON=/path/to/input.json \
+OUTPUT_PATH=./outputs/few_step \
+bash run_few_step.sh
+```
+
+Use `run_ar.sh` or `run_bi.sh` for the other modes, with their corresponding
+checkpoints. Launchers default to 8 GPUs with sequence parallelism and FSDP;
+set `NPROC_PER_NODE` to match your GPU count. For one GPU, also set
+`USE_DIT_FSDP=false USE_T5_FSDP=false`.
+
+All input items are processed, and videos are saved as `<output_name>.mp4` in
+`OUTPUT_PATH`.
+
+## 📖 Input Format
+
+Provide a JSON array. Relative image paths are resolved from the JSON file's
+directory.
+
+```json
+[
+ {
+ "image_path": "input.jpg",
+ "perspective": "tps",
+ "action": "w-32,s-32",
+ "prompt_event": "prompt1-32,prompt2-32",
+ "prompt1": "The scene in the video is: a quiet forest trail. The character is: a hiker wearing a blue jacket. The event is: dark clouds gather and rain begins to fall.",
+ "prompt2": "The scene in the video is: a quiet forest trail. The character is: a hiker wearing a blue jacket. The event is: the rain stops and sunlight breaks through the clouds.",
+ "output_name": "sample_000"
+ }
+]
+```
+
+- **Perspective:** `fps` or `tps`.
+- **Actions:** comma-separated `action-duration` commands. Movement actions are
+ `w`, `s`, `a`, `d`, `wa`, `wd`, `sa`, `sd`; rotation actions are `left`, `right`,
+ `up`, `down`; special action is `space`. Combine tokens with `+`, for example `right+space-32`.
+- **Durations:** measured in latent frames; the first latent is stationary.
+ Action and prompt durations must have the same total. The total and prompt
+ boundaries must align with `chunk_length` (4 in the few-step and AR launchers,
+ 32 in BI).
+
+#### Prompt Format
+
+Construct each prompt field (`prompt1`, `prompt2`, etc.) using the template for
+your perspective, replacing the placeholders with your descriptions.
+
+**FPS (`fps`):** omit the `The character is: ...` sentence.
+
+```text
+The scene in the video is: {scene description}. The event is: {event description}.
+```
+
+**TPS (`tps`):** include the character description, as in the JSON example above.
+
+```text
+The scene in the video is: {scene description}. The character is: {character description}. The event is: {event description}.
+```
+
+The scene description is required for both perspectives; the character
+description is only required for TPS. The event is optional for both: omit
+the entire `The event is: ...` sentence when it is not needed. For example,
+an FPS prompt without an event is:
+
+```text
+The scene in the video is: a quiet forest trail.
+```
+
+The `prompt_event` scheduling field is still required, even when prompts omit
+the event description.
+
+## 🔑 Configuration
+
+Paths and launcher settings can be overridden through environment variables:
+
+| Variable | Purpose |
+| --- | --- |
+| `NPROC_PER_NODE` | Number of GPUs |
+| `ATTENTION_BACKEND` | Visual self-attention: `flash` (default) or `sage` |
+| `SAMPLE_STEPS` | Sampling steps; few-step inference requires 4 |
+| `GUIDE_SCALE` | Guidance scale for BI/AR |
+| `YAW_ROTATION_SPEED_DEG` | Left/right rotation per latent |
+| `PITCH_ROTATION_SPEED_DEG` | Up/down rotation per latent |
+
+## 🙏 Acknowledgements
+We would like to thank [HY-World 1.5](https://github.com/Tencent-Hunyuan/HY-WorldPlay), [HY-World 2.0](https://github.com/Tencent-Hunyuan/HY-World-2.0), and [HunyuanWorld-Mirror](https://github.com/Tencent-Hunyuan/HunyuanWorld-Mirror) for their great work.
diff --git a/configs/__init__.py b/configs/__init__.py
new file mode 100644
index 0000000..527fffd
--- /dev/null
+++ b/configs/__init__.py
@@ -0,0 +1,3 @@
+from .i2v_a14b import I2V_A14B_CONFIG
+
+__all__ = ["I2V_A14B_CONFIG"]
diff --git a/configs/i2v_a14b.py b/configs/i2v_a14b.py
new file mode 100644
index 0000000..ce5ce65
--- /dev/null
+++ b/configs/i2v_a14b.py
@@ -0,0 +1,32 @@
+from dataclasses import dataclass
+
+import torch
+
+
+@dataclass(frozen=True)
+class I2VA14BConfig:
+ text_len: int = 512
+ t5_dtype: torch.dtype = torch.bfloat16
+ param_dtype: torch.dtype = torch.bfloat16
+ num_train_timesteps: int = 1000
+ boundary: float = 0.9
+ sample_shift: float = 5.0
+ sample_guide_scale: tuple[float, float] = (3.5, 3.5)
+ sample_neg_prompt: str = (
+ "色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,"
+ "画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,"
+ "残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,"
+ "毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,"
+ "三条腿,背景人很多,倒着走"
+ )
+ t5_checkpoint: str = "models_t5_umt5-xxl-enc-bf16.pth"
+ t5_tokenizer: str = "google/umt5-xxl"
+ vae_checkpoint: str = "Wan2.1_VAE.pth"
+ low_noise_checkpoint: str = "low_noise_model"
+ high_noise_checkpoint: str = "high_noise_model"
+ vae_stride: tuple[int, int, int] = (4, 8, 8)
+ patch_size: tuple[int, int, int] = (1, 2, 2)
+ sample_fps: int = 16
+
+
+I2V_A14B_CONFIG = I2VA14BConfig()
diff --git a/generate.py b/generate.py
new file mode 100644
index 0000000..50c40de
--- /dev/null
+++ b/generate.py
@@ -0,0 +1,233 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import gc
+import logging
+import sys
+
+import torch
+from PIL import Image
+
+from configs import I2V_A14B_CONFIG
+from worldplay2.inputs import (
+ initialize_runtime,
+ load_inference_inputs,
+)
+from worldplay2.pipeline import InferenceMode, WorldPlay2Pipeline
+from worldplay2.video import save_video
+
+
+def str2bool(value):
+ if isinstance(value, bool):
+ return value
+ value = value.lower()
+ if value in {"1", "true", "yes", "on"}:
+ return True
+ if value in {"0", "false", "no", "off"}:
+ return False
+ raise argparse.ArgumentTypeError(f"invalid boolean: {value}")
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="WorldPlay2 action-conditioned video inference"
+ )
+ source = parser.add_argument_group("input")
+ source.add_argument("--input_json", required=True)
+ source.add_argument(
+ "--rotation_speed_deg",
+ type=float,
+ default=3.0,
+ help="Fallback rotation degrees per latent for axes without an override.",
+ )
+ source.add_argument(
+ "--yaw_rotation_speed_deg",
+ type=float,
+ help="Left/right rotation degrees per latent; defaults to rotation_speed_deg.",
+ )
+ source.add_argument(
+ "--pitch_rotation_speed_deg",
+ type=float,
+ help="Up/down rotation degrees per latent; defaults to rotation_speed_deg.",
+ )
+
+ model = parser.add_argument_group("model")
+ model.add_argument("--ckpt_dir", required=True)
+ model.add_argument("--low_noise_mem_ckpt", required=True)
+ model.add_argument("--high_noise_mem_ckpt", required=True)
+ model.add_argument(
+ "--inference_mode",
+ choices=[mode.value for mode in InferenceMode],
+ default=InferenceMode.BI.value,
+ )
+ model.add_argument("--vae_type", choices=["wan2_1", "causal_tae"], default="wan2_1")
+ model.add_argument("--tae_ckpt")
+
+ generation = parser.add_argument_group("generation")
+ generation.add_argument("--height", type=int, default=448)
+ generation.add_argument("--width", type=int, default=832)
+ generation.add_argument("--chunk_length", type=int, default=4)
+ generation.add_argument("--sink_size", type=int, default=1)
+ generation.add_argument("--temporal_size", type=int, default=1)
+ generation.add_argument("--sample_steps", type=int)
+ generation.add_argument("--sample_shift", type=float)
+ generation.add_argument("--sample_guide_scale", type=float)
+ generation.add_argument("--sample_solver", choices=["unipc", "dpm++"], default="unipc")
+ generation.add_argument("--negative_prompt", default="")
+ generation.add_argument("--seed", type=int, default=-1)
+ generation.add_argument("--output_dir", default="outputs")
+ generation.add_argument(
+ "--compile_cache_dir",
+ default="~/.cache",
+ help="Persistent torch.compile cache directory.",
+ )
+
+ runtime = parser.add_argument_group("runtime")
+ runtime.add_argument("--offload_model", type=str2bool)
+ runtime.add_argument(
+ "--attention_backend",
+ choices=["flash", "sage"],
+ default="flash",
+ help="Visual self-attention backend: FlashAttention or SageAttention. "
+ "Text cross-attention and masked prefill keep their existing backends.",
+ )
+ runtime.add_argument("--ulysses_size", type=int, default=1)
+ runtime.add_argument(
+ "--disable_compiled_qk_rope",
+ action="store_true",
+ help="Use the eager QK RMSNorm + RoPE kernel.",
+ )
+ runtime.add_argument(
+ "--disable_conditioning_cache",
+ action="store_true",
+ help="Disable text projection and cross-attention K/V caches.",
+ )
+ runtime.add_argument(
+ "--disable_self_qkv_fusion",
+ action="store_true",
+ help="Keep separate self-attention Q/K/V projection GEMMs.",
+ )
+ runtime.add_argument(
+ "--disable_compiled_block_glue",
+ action="store_true",
+ help="Use eager modulation and gated residual expressions.",
+ )
+ runtime.add_argument(
+ "--disable_compiled_ffn",
+ action="store_true",
+ help="Use the eager Linear-GELU-Linear FFN module.",
+ )
+ runtime.add_argument(
+ "--fp8_ffn",
+ action="store_true",
+ help="Use Transformer Engine FP8 E4M3 for selected FFN blocks. "
+ "Requires DiT FSDP and model offload to be disabled.",
+ )
+ runtime.add_argument("--fp8_start_block", type=int, default=4)
+ runtime.add_argument("--fp8_end_block", type=int, default=35)
+ runtime.add_argument(
+ "--fp8_qkv",
+ action="store_true",
+ help="Use Transformer Engine FP8 E4M3 for selected fused "
+ "self-attention QKV projections.",
+ )
+ runtime.add_argument("--fp8_qkv_start_block", type=int, default=4)
+ runtime.add_argument("--fp8_qkv_end_block", type=int, default=35)
+ runtime.add_argument("--t5_fsdp", action="store_true")
+ runtime.add_argument("--dit_fsdp", action="store_true")
+ return parser.parse_args()
+
+
+def main():
+ args = parse_args()
+ inference = load_inference_inputs(
+ args,
+ default_guide_scale=I2V_A14B_CONFIG.sample_guide_scale,
+ )
+ runtime = initialize_runtime(args)
+ mode = InferenceMode(inference.mode)
+
+ logging.basicConfig(
+ level=logging.INFO,
+ format="[%(asctime)s] %(levelname)s: %(message)s",
+ handlers=[
+ logging.StreamHandler(stream=sys.stdout)
+ if runtime.rank == 0
+ else logging.NullHandler()
+ ],
+ force=True,
+ )
+
+ pipeline = WorldPlay2Pipeline(
+ config=I2V_A14B_CONFIG,
+ checkpoint_dir=args.ckpt_dir,
+ low_noise_mem_ckpt=args.low_noise_mem_ckpt,
+ high_noise_mem_ckpt=args.high_noise_mem_ckpt,
+ device_id=runtime.local_rank,
+ rank=runtime.rank,
+ t5_fsdp=args.t5_fsdp,
+ dit_fsdp=args.dit_fsdp,
+ use_sp=args.ulysses_size > 1,
+ attention_backend=args.attention_backend,
+ compile_qk_rope=not args.disable_compiled_qk_rope,
+ conditioning_cache=not args.disable_conditioning_cache,
+ fuse_self_qkv=not args.disable_self_qkv_fusion,
+ compile_block_glue=not args.disable_compiled_block_glue,
+ compile_ffn=not args.disable_compiled_ffn,
+ fp8_ffn=args.fp8_ffn,
+ fp8_start_block=args.fp8_start_block,
+ fp8_end_block=args.fp8_end_block,
+ fp8_qkv=args.fp8_qkv,
+ fp8_qkv_start_block=args.fp8_qkv_start_block,
+ fp8_qkv_end_block=args.fp8_qkv_end_block,
+ inference_mode=mode,
+ vae_type=args.vae_type,
+ tae_ckpt=args.tae_ckpt,
+ )
+ # FSDP construction can temporarily materialize full parameters. Clear
+ # allocator cache so nvidia-smi reflects persistent usage rather than
+ # stale reserved blocks from initialization.
+ gc.collect()
+ torch.cuda.empty_cache()
+ logging.info(
+ "FSDP requested: t5=%s dit=%s; wrapper counts: t5=%d low=%d high=%d",
+ args.t5_fsdp,
+ args.dit_fsdp,
+ pipeline.dtype_report["t5_fsdp_units"],
+ pipeline.dtype_report["low_noise_fsdp_units"],
+ pipeline.dtype_report["high_noise_fsdp_units"],
+ )
+
+ for item in inference.items:
+ image = Image.open(item.image).convert("RGB")
+ video = pipeline.generate_chunked(
+ prompts=item.prompts,
+ chunk_prompts=item.chunk_prompts,
+ img=image,
+ hr_action=item.actions.tensor,
+ total_latent_frames=item.actions.latent_frames,
+ chunk_length=args.chunk_length,
+ shift=inference.sample_shift,
+ sample_solver=args.sample_solver,
+ sampling_steps=inference.sample_steps,
+ guide_scale=inference.guide_scale,
+ n_prompt=args.negative_prompt,
+ seed=runtime.seed,
+ offload_model=runtime.offload_model,
+ sink_size=args.sink_size,
+ temporal_size=args.temporal_size,
+ height=args.height,
+ width=args.width,
+ )
+ if runtime.rank == 0 and video is not None:
+ save_video(
+ video,
+ runtime.output_dir / f"{item.output_name}.mp4",
+ I2V_A14B_CONFIG.sample_fps,
+ )
+ runtime.finish()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..1e1afcf
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,15 @@
+torch>=2.4.0
+torchvision>=0.19.0
+diffusers>=0.31.0
+transformers>=4.49.0,<=4.51.3
+tokenizers>=0.20.3
+accelerate>=1.1.1
+safetensors
+einops
+tqdm
+imageio[ffmpeg]
+ftfy
+regex
+numpy>=1.23.5,<2
+Pillow
+pytest
diff --git a/run_ar.sh b/run_ar.sh
new file mode 100755
index 0000000..2031ea7
--- /dev/null
+++ b/run_ar.sh
@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+NPROC_PER_NODE="${NPROC_PER_NODE:-8}"
+# Attention backend: flash (default) or sage.
+ATTENTION_BACKEND="${ATTENTION_BACKEND:-flash}"
+YAW_ROTATION_SPEED_DEG="${YAW_ROTATION_SPEED_DEG:-3.0}"
+PITCH_ROTATION_SPEED_DEG="${PITCH_ROTATION_SPEED_DEG:-3.0}"
+SAMPLE_STEPS="${SAMPLE_STEPS:-40}"
+GUIDE_SCALE="${GUIDE_SCALE:-3.5}"
+
+DISABLE_COMPILED_QK_ROPE="${DISABLE_COMPILED_QK_ROPE:-false}"
+DISABLE_CONDITIONING_CACHE="${DISABLE_CONDITIONING_CACHE:-false}"
+DISABLE_SELF_QKV_FUSION="${DISABLE_SELF_QKV_FUSION:-false}"
+DISABLE_COMPILED_BLOCK_GLUE="${DISABLE_COMPILED_BLOCK_GLUE:-false}"
+DISABLE_COMPILED_FFN="${DISABLE_COMPILED_FFN:-false}"
+USE_DIT_FSDP="${USE_DIT_FSDP:-false}"
+USE_T5_FSDP="${USE_T5_FSDP:-false}"
+ENABLE_FP8_FFN="${ENABLE_FP8_FFN:-false}"
+FP8_START_BLOCK="${FP8_START_BLOCK:-0}"
+FP8_END_BLOCK="${FP8_END_BLOCK:-39}"
+ENABLE_FP8_QKV="${ENABLE_FP8_QKV:-false}"
+FP8_QKV_START_BLOCK="${FP8_QKV_START_BLOCK:-0}"
+FP8_QKV_END_BLOCK="${FP8_QKV_END_BLOCK:-39}"
+
+CKPT_DIR="${CKPT_DIR:-/path/to/model}"
+LOW_NOISE_CKPT="${LOW_NOISE_CKPT:-/path/to/model/low_noise_model.safetensors}"
+HIGH_NOISE_CKPT="${HIGH_NOISE_CKPT:-/path/to/model/high_noise_model.safetensors}"
+
+INPUT_JSON="${INPUT_JSON:-/path/to/input.json}"
+
+OUTPUT_PATH="${OUTPUT_PATH:-./outputs/ar}"
+
+run_case() {
+ local json_path="$1"
+ local output_dir="${OUTPUT_PATH}"
+ local qk_rope_args=()
+ if [[ "${DISABLE_COMPILED_QK_ROPE}" == "true" ]]; then
+ qk_rope_args+=(--disable_compiled_qk_rope)
+ fi
+ local conditioning_args=()
+ if [[ "${DISABLE_CONDITIONING_CACHE}" == "true" ]]; then
+ conditioning_args+=(--disable_conditioning_cache)
+ fi
+ local qkv_fusion_args=()
+ if [[ "${DISABLE_SELF_QKV_FUSION}" == "true" ]]; then
+ qkv_fusion_args+=(--disable_self_qkv_fusion)
+ fi
+ local compiled_region_args=()
+ if [[ "${DISABLE_COMPILED_BLOCK_GLUE}" == "true" ]]; then
+ compiled_region_args+=(--disable_compiled_block_glue)
+ fi
+ if [[ "${DISABLE_COMPILED_FFN}" == "true" ]]; then
+ compiled_region_args+=(--disable_compiled_ffn)
+ fi
+ local fsdp_args=()
+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then
+ fsdp_args+=(--dit_fsdp)
+ fi
+ if [[ "${USE_T5_FSDP}" == "true" ]]; then
+ fsdp_args+=(--t5_fsdp)
+ fi
+ local fp8_args=()
+ if [[ "${ENABLE_FP8_FFN}" == "true" || "${ENABLE_FP8_QKV}" == "true" ]]; then
+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then
+ echo "DiT FP8 requires USE_DIT_FSDP=false" >&2
+ return 2
+ fi
+ fp8_args+=(--offload_model false)
+ fi
+ if [[ "${ENABLE_FP8_FFN}" == "true" ]]; then
+ fp8_args+=(
+ --fp8_ffn
+ --fp8_start_block "${FP8_START_BLOCK}"
+ --fp8_end_block "${FP8_END_BLOCK}"
+ )
+ fi
+ if [[ "${ENABLE_FP8_QKV}" == "true" ]]; then
+ fp8_args+=(
+ --fp8_qkv
+ --fp8_qkv_start_block "${FP8_QKV_START_BLOCK}"
+ --fp8_qkv_end_block "${FP8_QKV_END_BLOCK}"
+ )
+ fi
+
+ torchrun \
+ --standalone \
+ --nproc_per_node="${NPROC_PER_NODE}" \
+ generate.py \
+ --inference_mode ar \
+ --attention_backend "${ATTENTION_BACKEND}" \
+ --ckpt_dir "${CKPT_DIR}" \
+ --low_noise_mem_ckpt "${LOW_NOISE_CKPT}" \
+ --high_noise_mem_ckpt "${HIGH_NOISE_CKPT}" \
+ --input_json "${json_path}" \
+ --yaw_rotation_speed_deg "${YAW_ROTATION_SPEED_DEG}" \
+ --pitch_rotation_speed_deg "${PITCH_ROTATION_SPEED_DEG}" \
+ --height 448 \
+ --width 832 \
+ --chunk_length 4 \
+ --sink_size 1 \
+ --temporal_size 1 \
+ --sample_steps "${SAMPLE_STEPS}" \
+ --sample_shift 5.0 \
+ --sample_guide_scale "${GUIDE_SCALE}" \
+ --seed 42 \
+ "${qk_rope_args[@]}" \
+ "${conditioning_args[@]}" \
+ "${qkv_fusion_args[@]}" \
+ "${compiled_region_args[@]}" \
+ "${fsdp_args[@]}" \
+ "${fp8_args[@]}" \
+ --ulysses_size "${NPROC_PER_NODE}" \
+ --output_dir "${output_dir}"
+}
+
+run_case "${INPUT_JSON}"
diff --git a/run_bi.sh b/run_bi.sh
new file mode 100755
index 0000000..f7804f6
--- /dev/null
+++ b/run_bi.sh
@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+NPROC_PER_NODE="${NPROC_PER_NODE:-8}"
+# Visual self-attention backend: flash (default) or sage.
+ATTENTION_BACKEND="${ATTENTION_BACKEND:-sage}"
+YAW_ROTATION_SPEED_DEG="${YAW_ROTATION_SPEED_DEG:-3.0}"
+PITCH_ROTATION_SPEED_DEG="${PITCH_ROTATION_SPEED_DEG:-3.0}"
+SAMPLE_STEPS="${SAMPLE_STEPS:-40}"
+GUIDE_SCALE="${GUIDE_SCALE:-3.5}"
+
+DISABLE_COMPILED_QK_ROPE="${DISABLE_COMPILED_QK_ROPE:-false}"
+DISABLE_CONDITIONING_CACHE="${DISABLE_CONDITIONING_CACHE:-false}"
+DISABLE_SELF_QKV_FUSION="${DISABLE_SELF_QKV_FUSION:-false}"
+DISABLE_COMPILED_BLOCK_GLUE="${DISABLE_COMPILED_BLOCK_GLUE:-false}"
+DISABLE_COMPILED_FFN="${DISABLE_COMPILED_FFN:-false}"
+USE_DIT_FSDP="${USE_DIT_FSDP:-false}"
+USE_T5_FSDP="${USE_T5_FSDP:-false}"
+ENABLE_FP8_FFN="${ENABLE_FP8_FFN:-false}"
+FP8_START_BLOCK="${FP8_START_BLOCK:-0}"
+FP8_END_BLOCK="${FP8_END_BLOCK:-39}"
+ENABLE_FP8_QKV="${ENABLE_FP8_QKV:-false}"
+FP8_QKV_START_BLOCK="${FP8_QKV_START_BLOCK:-0}"
+FP8_QKV_END_BLOCK="${FP8_QKV_END_BLOCK:-39}"
+
+CKPT_DIR="${CKPT_DIR:-/path/to/model}"
+LOW_NOISE_CKPT="${LOW_NOISE_CKPT:-/path/to/model/low_noise_model.safetensors}"
+HIGH_NOISE_CKPT="${HIGH_NOISE_CKPT:-/path/to/model/high_noise_model.safetensors}"
+
+INPUT_JSON="${INPUT_JSON:-/path/to/input.json}"
+
+OUTPUT_PATH="${OUTPUT_PATH:-./outputs/bi}"
+
+run_case() {
+ local json_path="$1"
+ local output_dir="${OUTPUT_PATH}"
+ local qk_rope_args=()
+ if [[ "${DISABLE_COMPILED_QK_ROPE}" == "true" ]]; then
+ qk_rope_args+=(--disable_compiled_qk_rope)
+ fi
+ local conditioning_args=()
+ if [[ "${DISABLE_CONDITIONING_CACHE}" == "true" ]]; then
+ conditioning_args+=(--disable_conditioning_cache)
+ fi
+ local qkv_fusion_args=()
+ if [[ "${DISABLE_SELF_QKV_FUSION}" == "true" ]]; then
+ qkv_fusion_args+=(--disable_self_qkv_fusion)
+ fi
+ local compiled_region_args=()
+ if [[ "${DISABLE_COMPILED_BLOCK_GLUE}" == "true" ]]; then
+ compiled_region_args+=(--disable_compiled_block_glue)
+ fi
+ if [[ "${DISABLE_COMPILED_FFN}" == "true" ]]; then
+ compiled_region_args+=(--disable_compiled_ffn)
+ fi
+ local fsdp_args=()
+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then
+ fsdp_args+=(--dit_fsdp)
+ fi
+ if [[ "${USE_T5_FSDP}" == "true" ]]; then
+ fsdp_args+=(--t5_fsdp)
+ fi
+ local fp8_args=()
+ if [[ "${ENABLE_FP8_FFN}" == "true" || "${ENABLE_FP8_QKV}" == "true" ]]; then
+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then
+ echo "DiT FP8 requires USE_DIT_FSDP=false" >&2
+ return 2
+ fi
+ fp8_args+=(--offload_model false)
+ fi
+ if [[ "${ENABLE_FP8_FFN}" == "true" ]]; then
+ fp8_args+=(
+ --fp8_ffn
+ --fp8_start_block "${FP8_START_BLOCK}"
+ --fp8_end_block "${FP8_END_BLOCK}"
+ )
+ fi
+ if [[ "${ENABLE_FP8_QKV}" == "true" ]]; then
+ fp8_args+=(
+ --fp8_qkv
+ --fp8_qkv_start_block "${FP8_QKV_START_BLOCK}"
+ --fp8_qkv_end_block "${FP8_QKV_END_BLOCK}"
+ )
+ fi
+
+ torchrun \
+ --standalone \
+ --nproc_per_node="${NPROC_PER_NODE}" \
+ generate.py \
+ --inference_mode bi \
+ --attention_backend "${ATTENTION_BACKEND}" \
+ --ckpt_dir "${CKPT_DIR}" \
+ --low_noise_mem_ckpt "${LOW_NOISE_CKPT}" \
+ --high_noise_mem_ckpt "${HIGH_NOISE_CKPT}" \
+ --input_json "${json_path}" \
+ --yaw_rotation_speed_deg "${YAW_ROTATION_SPEED_DEG}" \
+ --pitch_rotation_speed_deg "${PITCH_ROTATION_SPEED_DEG}" \
+ --height 448 \
+ --width 832 \
+ --chunk_length 32 \
+ --sink_size 1 \
+ --temporal_size 1 \
+ --sample_steps "${SAMPLE_STEPS}" \
+ --sample_shift 5.0 \
+ --sample_guide_scale "${GUIDE_SCALE}" \
+ --seed 42 \
+ "${qk_rope_args[@]}" \
+ "${conditioning_args[@]}" \
+ "${qkv_fusion_args[@]}" \
+ "${compiled_region_args[@]}" \
+ "${fsdp_args[@]}" \
+ "${fp8_args[@]}" \
+ --ulysses_size "${NPROC_PER_NODE}" \
+ --output_dir "${output_dir}"
+}
+
+run_case "${INPUT_JSON}"
diff --git a/run_few_step.sh b/run_few_step.sh
new file mode 100644
index 0000000..30460ed
--- /dev/null
+++ b/run_few_step.sh
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+NPROC_PER_NODE="${NPROC_PER_NODE:-8}"
+# Visual self-attention backend: flash (default) or sage.
+ATTENTION_BACKEND="${ATTENTION_BACKEND:-sage}"
+YAW_ROTATION_SPEED_DEG="${YAW_ROTATION_SPEED_DEG:-3.0}"
+PITCH_ROTATION_SPEED_DEG="${PITCH_ROTATION_SPEED_DEG:-1.0}"
+SAMPLE_STEPS="${SAMPLE_STEPS:-4}"
+
+DISABLE_COMPILED_QK_ROPE="${DISABLE_COMPILED_QK_ROPE:-false}"
+DISABLE_CONDITIONING_CACHE="${DISABLE_CONDITIONING_CACHE:-false}"
+DISABLE_SELF_QKV_FUSION="${DISABLE_SELF_QKV_FUSION:-false}"
+DISABLE_COMPILED_BLOCK_GLUE="${DISABLE_COMPILED_BLOCK_GLUE:-false}"
+DISABLE_COMPILED_FFN="${DISABLE_COMPILED_FFN:-false}"
+USE_DIT_FSDP="${USE_DIT_FSDP:-false}"
+USE_T5_FSDP="${USE_T5_FSDP:-false}"
+ENABLE_FP8_FFN="${ENABLE_FP8_FFN:-false}"
+FP8_START_BLOCK="${FP8_START_BLOCK:-0}"
+FP8_END_BLOCK="${FP8_END_BLOCK:-39}"
+ENABLE_FP8_QKV="${ENABLE_FP8_QKV:-false}"
+FP8_QKV_START_BLOCK="${FP8_QKV_START_BLOCK:-0}"
+FP8_QKV_END_BLOCK="${FP8_QKV_END_BLOCK:-39}"
+
+CKPT_DIR="${CKPT_DIR:-/path/to/model}"
+LOW_NOISE_CKPT="${LOW_NOISE_CKPT:-/path/to/model/low_noise_model.pt}"
+HIGH_NOISE_CKPT="${HIGH_NOISE_CKPT:-/path/to/model/high_noise_model.pt}"
+
+INPUT_JSON="${INPUT_JSON:-/path/to/input.json}"
+
+OUTPUT_PATH="${OUTPUT_PATH:-./outputs/few_step}"
+
+run_case() {
+ local json_path="$1"
+ local yaw_rotation_speed_deg="$2"
+ local pitch_rotation_speed_deg="$3"
+ local output_dir="${OUTPUT_PATH}"
+ local qk_rope_args=()
+ if [[ "${DISABLE_COMPILED_QK_ROPE}" == "true" ]]; then
+ qk_rope_args+=(--disable_compiled_qk_rope)
+ fi
+ local conditioning_args=()
+ if [[ "${DISABLE_CONDITIONING_CACHE}" == "true" ]]; then
+ conditioning_args+=(--disable_conditioning_cache)
+ fi
+ local qkv_fusion_args=()
+ if [[ "${DISABLE_SELF_QKV_FUSION}" == "true" ]]; then
+ qkv_fusion_args+=(--disable_self_qkv_fusion)
+ fi
+ local compiled_region_args=()
+ if [[ "${DISABLE_COMPILED_BLOCK_GLUE}" == "true" ]]; then
+ compiled_region_args+=(--disable_compiled_block_glue)
+ fi
+ if [[ "${DISABLE_COMPILED_FFN}" == "true" ]]; then
+ compiled_region_args+=(--disable_compiled_ffn)
+ fi
+ local fsdp_args=()
+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then
+ fsdp_args+=(--dit_fsdp)
+ fi
+ if [[ "${USE_T5_FSDP}" == "true" ]]; then
+ fsdp_args+=(--t5_fsdp)
+ fi
+ local fp8_args=()
+ if [[ "${ENABLE_FP8_FFN}" == "true" || "${ENABLE_FP8_QKV}" == "true" ]]; then
+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then
+ echo "DiT FP8 requires USE_DIT_FSDP=false" >&2
+ return 2
+ fi
+ fp8_args+=(--offload_model false)
+ fi
+ if [[ "${ENABLE_FP8_FFN}" == "true" ]]; then
+ fp8_args+=(
+ --fp8_ffn
+ --fp8_start_block "${FP8_START_BLOCK}"
+ --fp8_end_block "${FP8_END_BLOCK}"
+ )
+ fi
+ if [[ "${ENABLE_FP8_QKV}" == "true" ]]; then
+ fp8_args+=(
+ --fp8_qkv
+ --fp8_qkv_start_block "${FP8_QKV_START_BLOCK}"
+ --fp8_qkv_end_block "${FP8_QKV_END_BLOCK}"
+ )
+ fi
+
+ echo "Running yaw_rotation_speed_deg=${yaw_rotation_speed_deg}, pitch_rotation_speed_deg=${pitch_rotation_speed_deg}, output_dir=${output_dir}"
+ torchrun \
+ --standalone \
+ --nproc_per_node="${NPROC_PER_NODE}" \
+ generate.py \
+ --inference_mode pdd \
+ --attention_backend "${ATTENTION_BACKEND}" \
+ --ckpt_dir "${CKPT_DIR}" \
+ --low_noise_mem_ckpt "${LOW_NOISE_CKPT}" \
+ --high_noise_mem_ckpt "${HIGH_NOISE_CKPT}" \
+ --input_json "${json_path}" \
+ --yaw_rotation_speed_deg "${yaw_rotation_speed_deg}" \
+ --pitch_rotation_speed_deg "${pitch_rotation_speed_deg}" \
+ --height 448 \
+ --width 832 \
+ --chunk_length 4 \
+ --sink_size 1 \
+ --temporal_size 1 \
+ --sample_steps "${SAMPLE_STEPS}" \
+ --sample_shift 7.0 \
+ --sample_guide_scale 1.0 \
+ --seed 42 \
+ "${qk_rope_args[@]}" \
+ "${conditioning_args[@]}" \
+ "${qkv_fusion_args[@]}" \
+ "${compiled_region_args[@]}" \
+ "${fsdp_args[@]}" \
+ "${fp8_args[@]}" \
+ --ulysses_size "${NPROC_PER_NODE}" \
+ --output_dir "${output_dir}"
+}
+
+run_case "${INPUT_JSON}" "${YAW_ROTATION_SPEED_DEG}" "${PITCH_ROTATION_SPEED_DEG}"
diff --git a/worldplay2/__init__.py b/worldplay2/__init__.py
new file mode 100644
index 0000000..67fecfd
--- /dev/null
+++ b/worldplay2/__init__.py
@@ -0,0 +1,10 @@
+"""WorldPlay2 inference package.
+
+Heavy CUDA/model modules are intentionally not imported here, so lightweight
+utilities such as input and action parsing can be used without torchvision,
+diffusers, or flash-attn.
+"""
+
+from .inputs import ActionSequence, parse_action_string
+
+__all__ = ["ActionSequence", "parse_action_string"]
diff --git a/worldplay2/checkpoint.py b/worldplay2/checkpoint.py
new file mode 100644
index 0000000..9a3f72c
--- /dev/null
+++ b/worldplay2/checkpoint.py
@@ -0,0 +1,67 @@
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+
+import torch
+from safetensors.torch import load_file
+
+
+def load_state_dict(path: str, *, prefer_ema: bool = False) -> dict[str, torch.Tensor]:
+ checkpoint_path = Path(path).expanduser()
+ if not checkpoint_path.is_file():
+ raise FileNotFoundError(checkpoint_path)
+ state = (
+ load_file(str(checkpoint_path))
+ if checkpoint_path.suffix == ".safetensors"
+ else torch.load(checkpoint_path, map_location="cpu", weights_only=False)
+ )
+
+ if isinstance(state, dict) and ("generator" in state or "generator_ema" in state):
+ key = "generator_ema" if prefer_ema and "generator_ema" in state else "generator"
+ state = state[key]
+ for wrapper in ("state_dict", "model_state_dict", "model"):
+ if isinstance(state, dict) and wrapper in state and isinstance(state[wrapper], dict):
+ state = state[wrapper]
+
+ if not isinstance(state, dict) or not state:
+ raise ValueError(f"checkpoint {checkpoint_path} does not contain a state dict")
+
+ prefixes = ("module.", "model.")
+ for prefix in prefixes:
+ if all(key.startswith(prefix) for key in state):
+ state = {key[len(prefix):]: value for key, value in state.items()}
+ return state
+
+
+def load_expert_checkpoint(
+ model,
+ path: str,
+ *,
+ tag: str,
+ prefer_ema: bool = False,
+ strict: bool = True,
+) -> None:
+ state = load_state_dict(path, prefer_ema=prefer_ema)
+ provided = set(state)
+ required_prefixes = (
+ "memory_compress.",
+ "action_in_cont.",
+ "action_in_disc.",
+ )
+ absent = [prefix for prefix in required_prefixes if not any(
+ key.startswith(prefix) for key in provided
+ )]
+ if absent:
+ raise ValueError(
+ f"{tag} checkpoint is missing required parameter groups: {absent}"
+ )
+ missing, unexpected = model.load_state_dict(state, strict=strict)
+ logging.info(
+ "%s checkpoint: loaded=%d missing=%d unexpected=%d strict=%s",
+ tag,
+ len(state),
+ len(missing),
+ len(unexpected),
+ strict,
+ )
diff --git a/worldplay2/distributed/__init__.py b/worldplay2/distributed/__init__.py
new file mode 100644
index 0000000..388f037
--- /dev/null
+++ b/worldplay2/distributed/__init__.py
@@ -0,0 +1,3 @@
+from .ops import get_rank, get_world_size, init_distributed_group
+
+__all__ = ["get_rank", "get_world_size", "init_distributed_group"]
diff --git a/worldplay2/distributed/fsdp.py b/worldplay2/distributed/fsdp.py
new file mode 100644
index 0000000..247b5eb
--- /dev/null
+++ b/worldplay2/distributed/fsdp.py
@@ -0,0 +1,45 @@
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+import gc
+from functools import partial
+
+import torch
+from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
+from torch.distributed.fsdp import MixedPrecision, ShardingStrategy
+from torch.distributed.fsdp.wrap import lambda_auto_wrap_policy
+from torch.distributed.utils import _free_storage
+
+
+def shard_model(
+ model,
+ device_id,
+ param_dtype=torch.bfloat16,
+ reduce_dtype=torch.float32,
+ buffer_dtype=torch.float32,
+ process_group=None,
+ sharding_strategy=ShardingStrategy.FULL_SHARD,
+ sync_module_states=True,
+ use_lora=False
+):
+ model = FSDP(
+ module=model,
+ process_group=process_group,
+ sharding_strategy=sharding_strategy,
+ auto_wrap_policy=partial(
+ lambda_auto_wrap_policy, lambda_fn=lambda m: m in model.blocks),
+ mixed_precision=MixedPrecision(
+ param_dtype=param_dtype,
+ reduce_dtype=reduce_dtype,
+ buffer_dtype=buffer_dtype),
+ device_id=device_id,
+ sync_module_states=sync_module_states,
+ use_orig_params=True if use_lora else False)
+ return model
+
+
+def free_model(model):
+ for m in model.modules():
+ if isinstance(m, FSDP):
+ _free_storage(m._handle.flat_param.data)
+ del model
+ gc.collect()
+ torch.cuda.empty_cache()
diff --git a/worldplay2/distributed/ops.py b/worldplay2/distributed/ops.py
new file mode 100644
index 0000000..cc64632
--- /dev/null
+++ b/worldplay2/distributed/ops.py
@@ -0,0 +1,17 @@
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+import torch.distributed as dist
+
+
+def init_distributed_group():
+ """r initialize sequence parallel group.
+ """
+ if not dist.is_initialized():
+ dist.init_process_group(backend='nccl')
+
+
+def get_rank():
+ return dist.get_rank()
+
+
+def get_world_size():
+ return dist.get_world_size()
diff --git a/worldplay2/distributed/ulysses.py b/worldplay2/distributed/ulysses.py
new file mode 100644
index 0000000..da714c8
--- /dev/null
+++ b/worldplay2/distributed/ulysses.py
@@ -0,0 +1,358 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from functools import lru_cache
+
+import torch
+import torch.distributed as dist
+
+
+@dataclass(frozen=True)
+class UlyssesSplitPlan:
+ global_sequence: int
+ num_heads: int
+ world_size: int
+ rank: int
+ sequence_splits: tuple[int, ...]
+ sequence_start: int
+ local_sequence: int
+ local_heads: int
+
+
+def _balanced_splits(total: int, parts: int) -> tuple[int, ...]:
+ if parts <= 0:
+ raise ValueError("parts must be positive")
+ if total < parts:
+ raise ValueError(
+ f"sequence length {total} must be >= world size {parts}"
+ )
+ base, remainder = divmod(total, parts)
+ return tuple(
+ base + (1 if rank < remainder else 0)
+ for rank in range(parts)
+ )
+
+
+@lru_cache(maxsize=256)
+def get_ulysses_split_plan(
+ global_sequence: int,
+ num_heads: int,
+ world_size: int,
+ rank: int,
+) -> UlyssesSplitPlan:
+ if num_heads % world_size:
+ raise ValueError(
+ f"num_heads={num_heads} must be divisible by world_size={world_size}"
+ )
+ splits = _balanced_splits(global_sequence, world_size)
+ return UlyssesSplitPlan(
+ global_sequence=global_sequence,
+ num_heads=num_heads,
+ world_size=world_size,
+ rank=rank,
+ sequence_splits=splits,
+ sequence_start=sum(splits[:rank]),
+ local_sequence=splits[rank],
+ local_heads=num_heads // world_size,
+ )
+
+
+_RECEIVE_BUFFERS: dict[tuple, torch.Tensor] = {}
+_STATS = {
+ "packed_qkv_calls": 0,
+ "output_calls": 0,
+ "final_gather_calls": 0,
+ "qkv_collectives_eliminated": 0,
+ "buffer_hits": 0,
+ "buffer_misses": 0,
+ "bytes_exchanged": 0,
+}
+
+
+def _receive_buffer(name: str, reference: torch.Tensor, elements: int):
+ key = (
+ name,
+ reference.device.type,
+ reference.device.index,
+ reference.dtype,
+ )
+ buffer = _RECEIVE_BUFFERS.get(key)
+ if buffer is None or buffer.numel() < elements:
+ buffer = torch.empty(
+ elements,
+ dtype=reference.dtype,
+ device=reference.device,
+ )
+ _RECEIVE_BUFFERS[key] = buffer
+ _STATS["buffer_misses"] += 1
+ else:
+ _STATS["buffer_hits"] += 1
+ return buffer[:elements]
+
+
+def shard_sequence(
+ tensor: torch.Tensor | None,
+ *,
+ global_sequence: int,
+ rank: int,
+ world_size: int,
+ dim: int,
+) -> torch.Tensor | None:
+ if tensor is None or world_size <= 1:
+ return tensor
+ splits = _balanced_splits(global_sequence, world_size)
+ start = sum(splits[:rank])
+ return tensor.narrow(dim, start, splits[rank]).contiguous()
+
+
+def _pack_qkv_for_destinations(q, k, v, world_size, local_heads):
+ batch, local_sequence, _, head_dim = q.shape
+
+ def destination_major(tensor):
+ return tensor.contiguous().view(
+ batch,
+ local_sequence,
+ world_size,
+ local_heads,
+ head_dim,
+ ).permute(2, 0, 1, 3, 4)
+
+ return torch.stack(
+ [destination_major(q), destination_major(k), destination_major(v)],
+ dim=1,
+ ).contiguous()
+
+
+def _unpack_received_qkv(
+ output_flat,
+ *,
+ sequence_splits,
+ batch,
+ local_heads,
+ head_dim,
+):
+ source_chunks = []
+ offset = 0
+ for source_sequence in sequence_splits:
+ elements = (
+ 3 * batch * source_sequence * local_heads * head_dim
+ )
+ source_chunks.append(
+ output_flat[offset:offset + elements].view(
+ 3,
+ batch,
+ source_sequence,
+ local_heads,
+ head_dim,
+ )
+ )
+ offset += elements
+ return torch.cat(source_chunks, dim=2).contiguous()
+
+
+def _pack_output_for_destinations(output, sequence_splits):
+ return torch.cat([
+ chunk.contiguous().view(-1)
+ for chunk in torch.split(output, sequence_splits, dim=1)
+ ])
+
+
+def _unpack_received_output(
+ output_flat,
+ *,
+ world_size,
+ batch,
+ local_sequence,
+ local_heads,
+ num_heads,
+ head_dim,
+):
+ return output_flat.view(
+ world_size,
+ batch,
+ local_sequence,
+ local_heads,
+ head_dim,
+ ).permute(1, 2, 0, 3, 4).reshape(
+ batch,
+ local_sequence,
+ num_heads,
+ head_dim,
+ ).contiguous()
+
+
+def packed_qkv_all_to_all(
+ q: torch.Tensor,
+ k: torch.Tensor,
+ v: torch.Tensor,
+ *,
+ global_sequence: int,
+ group=None,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Gather sequence and shard heads with one packed all-to-all-single."""
+ if not dist.is_initialized():
+ raise RuntimeError("packed QKV all-to-all requires distributed init")
+ world_size = dist.get_world_size(group)
+ if world_size == 1:
+ return q, k, v
+ rank = dist.get_rank(group)
+ if q.shape != k.shape or q.shape != v.shape:
+ raise ValueError("Q, K, and V must have identical local shapes")
+
+ batch, local_sequence, num_heads, head_dim = q.shape
+ plan = get_ulysses_split_plan(
+ global_sequence, num_heads, world_size, rank)
+ if local_sequence != plan.local_sequence:
+ raise ValueError(
+ f"rank {rank} expected {plan.local_sequence} local tokens, "
+ f"got {local_sequence}"
+ )
+
+ # [destination, qkv, batch, local_sequence, local_heads, head_dim]
+ packed = _pack_qkv_for_destinations(
+ q, k, v, world_size, plan.local_heads)
+ input_flat = packed.view(-1)
+ per_destination = (
+ 3 * batch * local_sequence * plan.local_heads * head_dim
+ )
+ input_splits = [per_destination] * world_size
+ output_splits = [
+ 3 * batch * seq * plan.local_heads * head_dim
+ for seq in plan.sequence_splits
+ ]
+ output_flat = _receive_buffer(
+ "packed_qkv", input_flat, sum(output_splits))
+ dist.all_to_all_single(
+ output_flat,
+ input_flat,
+ output_split_sizes=output_splits,
+ input_split_sizes=input_splits,
+ group=group,
+ )
+
+ packed_global = _unpack_received_qkv(
+ output_flat,
+ sequence_splits=plan.sequence_splits,
+ batch=batch,
+ local_heads=plan.local_heads,
+ head_dim=head_dim,
+ )
+
+ _STATS["packed_qkv_calls"] += 1
+ _STATS["qkv_collectives_eliminated"] += 2
+ _STATS["bytes_exchanged"] += (
+ input_flat.numel() * input_flat.element_size())
+ return tuple(packed_global.unbind(0))
+
+
+def output_all_to_all(
+ output: torch.Tensor,
+ *,
+ global_sequence: int,
+ num_heads: int,
+ group=None,
+) -> torch.Tensor:
+ """Shard sequence and gather heads after attention."""
+ if not dist.is_initialized():
+ raise RuntimeError("output all-to-all requires distributed init")
+ world_size = dist.get_world_size(group)
+ if world_size == 1:
+ return output
+ rank = dist.get_rank(group)
+
+ batch, sequence, local_heads, head_dim = output.shape
+ plan = get_ulysses_split_plan(
+ global_sequence, num_heads, world_size, rank)
+ if sequence != global_sequence or local_heads != plan.local_heads:
+ raise ValueError(
+ "attention output shape does not match the Ulysses split plan: "
+ f"shape={tuple(output.shape)}, plan={plan}"
+ )
+
+ input_flat = _pack_output_for_destinations(
+ output, plan.sequence_splits)
+ input_splits = [
+ batch * seq * local_heads * head_dim
+ for seq in plan.sequence_splits
+ ]
+ per_source = (
+ batch * plan.local_sequence * local_heads * head_dim
+ )
+ output_splits = [per_source] * world_size
+ output_flat = _receive_buffer(
+ "attention_output", input_flat, sum(output_splits))
+ dist.all_to_all_single(
+ output_flat,
+ input_flat,
+ output_split_sizes=output_splits,
+ input_split_sizes=input_splits,
+ group=group,
+ )
+ local = _unpack_received_output(
+ output_flat,
+ world_size=world_size,
+ batch=batch,
+ local_sequence=plan.local_sequence,
+ local_heads=local_heads,
+ num_heads=num_heads,
+ head_dim=head_dim,
+ )
+
+ _STATS["output_calls"] += 1
+ _STATS["bytes_exchanged"] += (
+ input_flat.numel() * input_flat.element_size())
+ return local
+
+
+def gather_sequence(
+ tensor: torch.Tensor,
+ *,
+ global_sequence: int,
+ group=None,
+) -> torch.Tensor:
+ """Gather a known balanced sequence split without object collectives."""
+ if not dist.is_initialized():
+ return tensor
+ world_size = dist.get_world_size(group)
+ if world_size == 1:
+ return tensor
+ rank = dist.get_rank(group)
+ splits = _balanced_splits(global_sequence, world_size)
+ if tensor.shape[1] != splits[rank]:
+ raise ValueError(
+ f"rank {rank} expected {splits[rank]} tokens, got {tensor.shape[1]}"
+ )
+
+ max_sequence = max(splits)
+ padded_shape = list(tensor.shape)
+ padded_shape[1] = max_sequence
+ padded = tensor.new_zeros(padded_shape)
+ padded[:, :tensor.shape[1]].copy_(tensor)
+ padded_flat = padded.contiguous().view(-1)
+ gathered_flat = _receive_buffer(
+ "final_sequence_gather",
+ padded_flat,
+ world_size * padded_flat.numel(),
+ )
+ dist.all_gather_into_tensor(
+ gathered_flat,
+ padded_flat,
+ group=group,
+ )
+ gathered = gathered_flat.view(world_size, *padded.shape)
+ result = torch.cat(
+ [chunk[:, :length] for chunk, length in zip(gathered, splits)],
+ dim=1,
+ ).contiguous()
+ _STATS["final_gather_calls"] += 1
+ _STATS["bytes_exchanged"] += (
+ padded.numel() * padded.element_size())
+ return result
+
+
+def get_ulysses_stats() -> dict:
+ return dict(_STATS)
+
+
+def clear_ulysses_buffers() -> None:
+ _RECEIVE_BUFFERS.clear()
diff --git a/worldplay2/inputs.py b/worldplay2/inputs.py
new file mode 100644
index 0000000..3246082
--- /dev/null
+++ b/worldplay2/inputs.py
@@ -0,0 +1,555 @@
+from __future__ import annotations
+
+import json
+import math
+import os
+import random
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import torch
+import torch.distributed as dist
+
+from worldplay2.distributed.ops import init_distributed_group
+
+
+_MOVEMENT = {
+ "w": (1, 0),
+ "s": (-1, 0),
+ "a": (0, -1),
+ "d": (0, 1),
+ "wa": (1, -1),
+ "wd": (1, 1),
+ "sa": (-1, -1),
+ "sd": (-1, 1),
+}
+_ROTATION = {
+ "up": (1, 0),
+ "down": (-1, 0),
+ "left": (0, -1),
+ "right": (0, 1),
+}
+
+
+@dataclass(frozen=True)
+class ActionSequence:
+ tensor: torch.Tensor
+ source: str
+ perspective: str
+
+ @property
+ def latent_frames(self) -> int:
+ return int(self.tensor.shape[0])
+
+
+@dataclass(frozen=True)
+class GenerationInput:
+ image: Path
+ actions: ActionSequence
+ prompts: dict[str, str]
+ chunk_prompts: tuple[str, ...]
+ output_name: str
+
+
+@dataclass(frozen=True)
+class InferenceInputs:
+ items: list[GenerationInput]
+ mode: str
+ sample_shift: float
+ sample_steps: int
+ guide_scale: float | tuple[float, ...]
+
+
+@dataclass(frozen=True)
+class RuntimeContext:
+ rank: int
+ world_size: int
+ local_rank: int
+ output_dir: Path
+ offload_model: bool
+ seed: int
+
+ def finish(self) -> None:
+ if dist.is_initialized():
+ dist.barrier(device_ids=[self.local_rank])
+ dist.destroy_process_group()
+
+
+def parse_action_string(
+ value: str,
+ *,
+ perspective: str = "tps",
+ rotation_speed_deg: float = 3.0,
+ yaw_rotation_speed_deg: float | None = None,
+ pitch_rotation_speed_deg: float | None = None,
+) -> ActionSequence:
+ """Convert ``w+right-8,s-4`` directly to model actions.
+
+ Command durations sum to the total latent length. The first latent is
+ always stationary, so the first command applies for ``duration - 1``
+ subsequent latents.
+
+ Yaw (left/right) and pitch (up/down) use independent degrees per latent.
+ Each unspecified axis falls back to ``rotation_speed_deg``.
+ """
+ if perspective not in {"tps", "fps"}:
+ raise ValueError("perspective must be 'tps' or 'fps'")
+ if yaw_rotation_speed_deg is None:
+ yaw_rotation_speed_deg = rotation_speed_deg
+ if pitch_rotation_speed_deg is None:
+ pitch_rotation_speed_deg = rotation_speed_deg
+ for name, speed in (
+ ("yaw_rotation_speed_deg", yaw_rotation_speed_deg),
+ ("pitch_rotation_speed_deg", pitch_rotation_speed_deg),
+ ):
+ if not math.isfinite(speed) or speed <= 0:
+ raise ValueError(f"{name} must be a positive finite number")
+ if not value or not value.strip():
+ raise ValueError("action string cannot be empty")
+
+ perspective_value = 0.0 if perspective == "tps" else 1.0
+ rows: list[list[float]] = [[0.0, 0.0, 0.0, 0.0, perspective_value, 0.0]]
+ first_command = True
+
+ for raw_command in value.split(","):
+ command = raw_command.strip()
+ if not command:
+ continue
+ action_text, separator, duration_text = command.rpartition("-")
+ if not separator or not action_text.strip():
+ raise ValueError(
+ f"invalid action command {command!r}; expected action-duration"
+ )
+ try:
+ duration = int(duration_text.strip())
+ except ValueError as exc:
+ raise ValueError(
+ f"duration must be a positive integer in {command!r}"
+ ) from exc
+ if duration <= 0 or str(duration) != duration_text.strip():
+ raise ValueError(
+ f"duration must be a positive integer in {command!r}"
+ )
+
+ pitch = yaw = forward = lateral = space = 0.0
+ tokens = [token.strip().lower() for token in action_text.split("+")]
+ if not all(tokens):
+ raise ValueError(f"empty action token in {command!r}")
+ seen_categories: set[str] = set()
+ for token in tokens:
+ if token in _MOVEMENT:
+ if "movement" in seen_categories:
+ raise ValueError(f"multiple movement tokens in {command!r}")
+ seen_categories.add("movement")
+ forward, lateral = map(float, _MOVEMENT[token])
+ elif token in _ROTATION:
+ if token in {"up", "down"}:
+ if "pitch" in seen_categories:
+ raise ValueError(f"multiple pitch tokens in {command!r}")
+ seen_categories.add("pitch")
+ else:
+ if "yaw" in seen_categories:
+ raise ValueError(f"multiple yaw tokens in {command!r}")
+ seen_categories.add("yaw")
+ pitch_sign, yaw_sign = _ROTATION[token]
+ pitch += pitch_sign * pitch_rotation_speed_deg
+ yaw += yaw_sign * yaw_rotation_speed_deg
+ elif token == "space":
+ if "space" in seen_categories:
+ raise ValueError(f"duplicate space token in {command!r}")
+ seen_categories.add("space")
+ space = 1.0
+ elif token == "none":
+ if len(tokens) != 1:
+ raise ValueError(
+ f"'none' cannot be combined with other actions "
+ f"in {command!r}"
+ )
+ else:
+ supported = sorted(
+ set(_MOVEMENT) | set(_ROTATION) | {"none", "space"}
+ )
+ raise ValueError(
+ f"unknown action token {token!r}; supported: {supported}"
+ )
+
+ row = [pitch, yaw, forward, lateral, perspective_value, space]
+ repeat_count = duration - 1 if first_command else duration
+ rows.extend([row.copy() for _ in range(repeat_count)])
+ first_command = False
+
+ if first_command:
+ raise ValueError("action string did not contain any commands")
+ return ActionSequence(
+ tensor=torch.tensor(rows, dtype=torch.float32),
+ source=value,
+ perspective=perspective,
+ )
+
+
+def _parse_prompt_event(
+ *,
+ value: str,
+ prompts: dict[str, str],
+ latent_frames: int,
+ chunk_length: int,
+) -> tuple[str, ...]:
+ """Parse ``prompt1-8,prompt2-8`` into one prompt name per chunk."""
+ if not value or not value.strip():
+ raise ValueError("prompt_event cannot be empty")
+
+ latent_prompts: list[str] = []
+ cumulative = 0
+ for raw_event in value.split(","):
+ event = raw_event.strip()
+ prompt_name, separator, duration_text = event.rpartition("-")
+ prompt_name = prompt_name.strip()
+ if not separator or not prompt_name:
+ raise ValueError(
+ f"invalid prompt event {event!r}; expected prompt_name-duration"
+ )
+ if prompt_name not in prompts:
+ raise ValueError(
+ f"prompt_event references missing prompt {prompt_name!r}"
+ )
+ try:
+ duration = int(duration_text.strip())
+ except ValueError as exc:
+ raise ValueError(
+ f"prompt duration must be a positive integer in {event!r}"
+ ) from exc
+ if duration <= 0 or str(duration) != duration_text.strip():
+ raise ValueError(
+ f"prompt duration must be a positive integer in {event!r}"
+ )
+
+ cumulative += duration
+ if cumulative < latent_frames and cumulative % chunk_length:
+ raise ValueError(
+ "prompt_event boundaries must align with chunk_length; "
+ f"boundary {cumulative} is not divisible by {chunk_length}"
+ )
+ latent_prompts.extend([prompt_name] * duration)
+
+ if len(latent_prompts) != latent_frames:
+ raise ValueError(
+ f"prompt_event covers {len(latent_prompts)} latent frames, "
+ f"but action produces {latent_frames}"
+ )
+ return tuple(
+ latent_prompts[index]
+ for index in range(0, latent_frames, chunk_length)
+ )
+
+
+def load_generation_inputs(
+ *,
+ input_json: str,
+ rotation_speed_deg: float = 3.0,
+ yaw_rotation_speed_deg: float | None = None,
+ pitch_rotation_speed_deg: float | None = None,
+ chunk_length: int,
+ sink_size: int,
+ temporal_size: int,
+) -> list[GenerationInput]:
+ json_path = Path(input_json).expanduser()
+ with json_path.open("r", encoding="utf-8") as handle:
+ payload = json.load(handle)
+ if not isinstance(payload, list) or not payload:
+ raise ValueError("input JSON must contain a non-empty list")
+
+ items = []
+ for index, raw_item in enumerate(payload):
+ if not isinstance(raw_item, dict):
+ raise ValueError(f"input JSON item {index} must be an object")
+ image_path = raw_item.get("image_path")
+ action = raw_item.get("action")
+ prompt_event = raw_item.get("prompt_event")
+ perspective = raw_item.get("perspective")
+ if not isinstance(image_path, str) or not image_path.strip():
+ raise ValueError(f"input JSON item {index} requires image_path")
+ if not isinstance(action, str):
+ raise ValueError(f"input JSON item {index} requires action")
+ if not isinstance(prompt_event, str):
+ raise ValueError(f"input JSON item {index} requires prompt_event")
+ if perspective not in {"tps", "fps"}:
+ raise ValueError(
+ f"input JSON item {index} perspective must be 'tps' or 'fps'"
+ )
+
+ prompts = {
+ key: value
+ for key, value in raw_item.items()
+ if key.startswith("prompt")
+ and key != "prompt_event"
+ and isinstance(value, str)
+ and value.strip()
+ }
+ if not prompts:
+ raise ValueError(
+ f"input JSON item {index} requires at least one prompt field"
+ )
+ actions = load_action_sequence(
+ action=action,
+ perspective=perspective,
+ rotation_speed_deg=rotation_speed_deg,
+ yaw_rotation_speed_deg=yaw_rotation_speed_deg,
+ pitch_rotation_speed_deg=pitch_rotation_speed_deg,
+ chunk_length=chunk_length,
+ sink_size=sink_size,
+ temporal_size=temporal_size,
+ )
+ chunk_prompts = _parse_prompt_event(
+ value=prompt_event,
+ prompts=prompts,
+ latent_frames=actions.latent_frames,
+ chunk_length=chunk_length,
+ )
+ image = Path(image_path).expanduser()
+ if not image.is_absolute():
+ image = json_path.parent / image
+ output_name = str(
+ raw_item.get("output_name") or f"{index:04d}_{image.stem}"
+ ).strip()
+ items.append(
+ GenerationInput(
+ image=image,
+ actions=actions,
+ prompts=prompts,
+ chunk_prompts=chunk_prompts,
+ output_name=output_name,
+ )
+ )
+ return items
+
+
+def validate_geometry(height: int, width: int) -> None:
+ if height <= 0 or width <= 0:
+ raise ValueError("height and width must be positive")
+ if height % 64 or width % 64:
+ raise ValueError(
+ "height and width must be divisible by 64 for HR/LR VAE and patch alignment"
+ )
+
+
+def load_action_sequence(
+ *,
+ action: str,
+ perspective: str,
+ rotation_speed_deg: float = 3.0,
+ yaw_rotation_speed_deg: float | None = None,
+ pitch_rotation_speed_deg: float | None = None,
+ chunk_length: int,
+ sink_size: int,
+ temporal_size: int,
+) -> ActionSequence:
+ """Parse an action string and validate its generation layout.
+
+ This validates both the action syntax and its relationships with the
+ chunked stage-three inference inputs.
+ """
+ if chunk_length <= 0:
+ raise ValueError("chunk_length must be positive")
+ if chunk_length % 2:
+ raise ValueError(
+ "chunk_length must be even for HR-to-LR temporal downsampling"
+ )
+ if sink_size < 0:
+ raise ValueError("sink_size must be non-negative")
+ if sink_size > chunk_length:
+ raise ValueError("sink_size must not exceed chunk_length")
+ if not 1 <= temporal_size <= chunk_length:
+ raise ValueError("temporal_size must be in [1, chunk_length]")
+
+ actions = parse_action_string(
+ action,
+ perspective=perspective,
+ rotation_speed_deg=rotation_speed_deg,
+ yaw_rotation_speed_deg=yaw_rotation_speed_deg,
+ pitch_rotation_speed_deg=pitch_rotation_speed_deg,
+ )
+ if actions.latent_frames % chunk_length:
+ raise ValueError(
+ f"action produces {actions.latent_frames} latent frames, which is "
+ f"not divisible by chunk_length={chunk_length}"
+ )
+ if sink_size >= actions.latent_frames:
+ raise ValueError("sink_size must be smaller than the action length")
+ return actions
+
+
+def load_inference_inputs(
+ args: Any,
+ *,
+ default_guide_scale: float | tuple[float, ...],
+) -> InferenceInputs:
+ """Resolve and validate all data and sampling inputs from CLI arguments."""
+ validate_geometry(args.height, args.width)
+ items = load_generation_inputs(
+ input_json=args.input_json,
+ rotation_speed_deg=args.rotation_speed_deg,
+ yaw_rotation_speed_deg=getattr(args, "yaw_rotation_speed_deg", None),
+ pitch_rotation_speed_deg=getattr(args, "pitch_rotation_speed_deg", None),
+ chunk_length=args.chunk_length,
+ sink_size=args.sink_size,
+ temporal_size=args.temporal_size,
+ )
+
+ mode = str(args.inference_mode)
+ if mode not in {"bi", "ar", "pdd"}:
+ raise ValueError(f"unsupported inference mode: {mode!r}")
+ sample_shift = (
+ float(args.sample_shift)
+ if args.sample_shift is not None
+ else (7.0 if mode == "pdd" else 5.0)
+ )
+ if not math.isfinite(sample_shift) or sample_shift <= 0:
+ raise ValueError("sample_shift must be a positive finite number")
+ sample_steps = (
+ int(args.sample_steps)
+ if args.sample_steps is not None
+ else (4 if mode == "pdd" else 40)
+ )
+ if sample_steps <= 0:
+ raise ValueError("sample_steps must be positive")
+ guide_scale = (
+ args.sample_guide_scale
+ if args.sample_guide_scale is not None
+ else (1.0 if mode == "pdd" else default_guide_scale)
+ )
+ guide_scales = (
+ (float(guide_scale),)
+ if isinstance(guide_scale, (int, float))
+ else tuple(float(scale) for scale in guide_scale)
+ )
+ if not guide_scales or any(
+ not math.isfinite(scale) or scale < 0 for scale in guide_scales
+ ):
+ raise ValueError(
+ "sample_guide_scale must contain finite non-negative values"
+ )
+
+ if mode == "pdd":
+ if sample_steps != 4:
+ raise ValueError("--inference_mode pdd requires --sample_steps 4")
+ if not math.isclose(sample_shift, 7.0, rel_tol=0.0, abs_tol=1e-8):
+ raise ValueError("--inference_mode pdd requires --sample_shift 7")
+ if any(
+ not math.isclose(scale, 1.0, rel_tol=0.0, abs_tol=1e-8)
+ for scale in guide_scales
+ ):
+ raise ValueError(
+ "--inference_mode pdd requires --sample_guide_scale 1"
+ )
+
+ return InferenceInputs(
+ items=items,
+ mode=mode,
+ sample_shift=sample_shift,
+ sample_steps=sample_steps,
+ guide_scale=guide_scale,
+ )
+
+
+def initialize_runtime(args: Any) -> RuntimeContext:
+ """Validate runtime options and initialize distributed execution."""
+ rank = int(os.getenv("RANK", "0"))
+ world_size = int(os.getenv("WORLD_SIZE", "1"))
+ local_rank = int(os.getenv("LOCAL_RANK", "0"))
+ if world_size <= 0:
+ raise ValueError("WORLD_SIZE must be positive")
+ if not 0 <= rank < world_size:
+ raise ValueError("RANK must be in [0, WORLD_SIZE)")
+ if not 0 <= local_rank < world_size:
+ raise ValueError("LOCAL_RANK must be in [0, WORLD_SIZE)")
+
+ if world_size == 1 and (
+ args.t5_fsdp or args.dit_fsdp or args.ulysses_size != 1
+ ):
+ raise ValueError(
+ "FSDP/SP options require torchrun with multiple processes"
+ )
+
+ if args.ulysses_size <= 0:
+ raise ValueError("ulysses_size must be positive")
+ if args.ulysses_size > 1 and args.ulysses_size != world_size:
+ raise ValueError("ulysses_size must equal world size")
+
+ fp8_enabled = bool(args.fp8_ffn or args.fp8_qkv)
+ if fp8_enabled and args.dit_fsdp:
+ raise ValueError("DiT FP8 cannot be combined with --dit_fsdp")
+ if args.fp8_qkv and args.disable_self_qkv_fusion:
+ raise ValueError(
+ "--fp8_qkv requires self-QKV fusion; remove "
+ "--disable_self_qkv_fusion"
+ )
+ if args.fp8_ffn and not (
+ 0 <= args.fp8_start_block <= args.fp8_end_block < 40
+ ):
+ raise ValueError(
+ "FP8 block range must satisfy 0 <= start <= end < 40"
+ )
+ if args.fp8_qkv and not (
+ 0 <= args.fp8_qkv_start_block <= args.fp8_qkv_end_block < 40
+ ):
+ raise ValueError(
+ "FP8 QKV block range must satisfy 0 <= start <= end < 40"
+ )
+
+ offload_model = args.offload_model
+ if offload_model is None:
+ offload_model = world_size == 1
+ if fp8_enabled and offload_model:
+ raise ValueError("DiT FP8 requires --offload_model false")
+
+ output_dir = Path(args.output_dir).expanduser()
+ if rank == 0:
+ output_dir.mkdir(parents=True, exist_ok=True)
+
+ if not (
+ args.disable_compiled_qk_rope
+ and args.disable_compiled_block_glue
+ and args.disable_compiled_ffn
+ ):
+ compile_root = (
+ Path(args.compile_cache_dir).expanduser()
+ / f"rank_{rank:02d}"
+ )
+ inductor_cache = compile_root / "torchinductor"
+ triton_cache = compile_root / "triton"
+ for cache_dir in (inductor_cache, triton_cache):
+ cache_dir.mkdir(parents=True, exist_ok=True)
+ os.environ["TORCHINDUCTOR_CACHE_DIR"] = str(inductor_cache)
+ os.environ["TRITON_CACHE_DIR"] = str(triton_cache)
+
+ if world_size > 1:
+ torch.cuda.set_device(local_rank)
+ dist.init_process_group(
+ "nccl",
+ init_method="env://",
+ device_id=torch.device("cuda", local_rank),
+ )
+ if args.ulysses_size > 1:
+ init_distributed_group()
+ dist.barrier(device_ids=[local_rank])
+
+ seed = (
+ int(args.seed)
+ if args.seed >= 0
+ else random.randint(0, sys.maxsize)
+ )
+ if dist.is_initialized():
+ seed_box = [seed if rank == 0 else None]
+ dist.broadcast_object_list(seed_box, src=0)
+ seed = int(seed_box[0])
+
+ return RuntimeContext(
+ rank=rank,
+ world_size=world_size,
+ local_rank=local_rank,
+ output_dir=output_dir,
+ offload_model=bool(offload_model),
+ seed=seed,
+ )
diff --git a/worldplay2/models/__init__.py b/worldplay2/models/__init__.py
new file mode 100644
index 0000000..482f8f5
--- /dev/null
+++ b/worldplay2/models/__init__.py
@@ -0,0 +1,3 @@
+from .model import WorldPlay2Model
+
+__all__ = ["WorldPlay2Model"]
diff --git a/worldplay2/models/attention.py b/worldplay2/models/attention.py
new file mode 100644
index 0000000..843b194
--- /dev/null
+++ b/worldplay2/models/attention.py
@@ -0,0 +1,282 @@
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+from functools import lru_cache
+
+import torch
+
+try:
+ import flash_attn_interface
+ FLASH_ATTN_3_AVAILABLE = True
+except ModuleNotFoundError:
+ flash_attn_interface = None
+ FLASH_ATTN_3_AVAILABLE = False
+
+try:
+ import flash_attn
+ FLASH_ATTN_2_AVAILABLE = True
+except ModuleNotFoundError:
+ flash_attn = None
+ FLASH_ATTN_2_AVAILABLE = False
+
+import warnings
+
+__all__ = [
+ 'flash_attention',
+ 'flash_attention_dense',
+ 'sage_attention',
+ 'resolve_attention_backend',
+ 'attention',
+]
+
+
+@lru_cache(maxsize=1)
+def _load_sage_attention():
+ """Load the optional CUDA backend only when it is selected."""
+ try:
+ from sageattention import sageattn
+ except (ImportError, OSError) as exc:
+ raise RuntimeError(
+ "attention_backend='sage' requires a working SageAttention "
+ "installation. Install it with `pip install sageattention "
+ "--no-build-isolation` in your CUDA environment, or use "
+ "attention_backend='flash'."
+ ) from exc
+ return sageattn
+
+
+def resolve_attention_backend(backend: str) -> str:
+ """Normalize the backend name and check optional dependencies early."""
+ backend = backend.lower()
+ if backend not in ('flash', 'sage'):
+ raise ValueError(
+ "attention_backend must be 'flash' or 'sage', "
+ f"got {backend!r}"
+ )
+ if backend == 'sage':
+ _load_sage_attention()
+ return backend
+
+
+def sage_attention(
+ q,
+ k,
+ v,
+ *,
+ k_lens=None,
+ softmax_scale=None,
+ causal=False,
+ window_size=(-1, -1),
+):
+ """Dense SageAttention in the model's ``[B, S, H, D]`` layout.
+
+ Only full key sequences are supported here. Masked history prefill and
+ text cross-attention keep their existing SDPA/FlashAttention paths.
+ """
+ if tuple(window_size) != (-1, -1):
+ raise NotImplementedError(
+ "SageAttention does not support sliding-window attention in "
+ "WorldPlay2; use attention_backend='flash'."
+ )
+ if k_lens is not None and not torch.all(k_lens == k.shape[1]):
+ raise NotImplementedError(
+ "SageAttention requires dense, equal-length keys; "
+ "use attention_backend='flash' for padded/variable lengths."
+ )
+ return _load_sage_attention()(
+ q.contiguous(),
+ k.contiguous(),
+ v.contiguous(),
+ tensor_layout='NHD',
+ is_causal=causal,
+ sm_scale=softmax_scale,
+ )
+
+
+def flash_attention_dense(
+ q,
+ k,
+ v,
+ *,
+ softmax_scale=None,
+ causal=False,
+):
+ """Dense attention using FA3 when available, otherwise PyTorch SDPA.
+
+ Inputs and output use the model-native layout ``[B, S, H, D]``. This is
+ preferable to the varlen wrapper when every batch item has the same
+ sequence length because it avoids flattening and constructing cu_seqlens.
+ """
+ if FLASH_ATTN_3_AVAILABLE:
+ return flash_attn_interface.flash_attn_func(
+ q=q.contiguous(),
+ k=k.contiguous(),
+ v=v.contiguous(),
+ softmax_scale=softmax_scale,
+ causal=causal,
+ )
+
+ out = torch.nn.functional.scaled_dot_product_attention(
+ q.transpose(1, 2),
+ k.transpose(1, 2),
+ v.transpose(1, 2),
+ is_causal=causal,
+ scale=softmax_scale,
+ )
+ return out.transpose(1, 2).contiguous()
+
+
+def flash_attention(
+ q,
+ k,
+ v,
+ q_lens=None,
+ k_lens=None,
+ dropout_p=0.,
+ softmax_scale=None,
+ q_scale=None,
+ causal=False,
+ window_size=(-1, -1),
+ deterministic=False,
+ dtype=torch.bfloat16,
+ version=None,
+):
+ """
+ q: [B, Lq, Nq, C1].
+ k: [B, Lk, Nk, C1].
+ v: [B, Lk, Nk, C2]. Nq must be divisible by Nk.
+ q_lens: [B].
+ k_lens: [B].
+ dropout_p: float. Dropout probability.
+ softmax_scale: float. The scaling of QK^T before applying softmax.
+ causal: bool. Whether to apply causal attention mask.
+ window_size: (left right). If not (-1, -1), apply sliding window local attention.
+ deterministic: bool. If True, slightly slower and uses more memory.
+ dtype: torch.dtype. Apply when dtype of q/k/v is not float16/bfloat16.
+ """
+ half_dtypes = (torch.float16, torch.bfloat16)
+ assert dtype in half_dtypes
+ assert q.device.type == 'cuda' and q.size(-1) <= 256
+
+ # params
+ b, lq, lk, out_dtype = q.size(0), q.size(1), k.size(1), q.dtype
+
+ def half(x):
+ return x if x.dtype in half_dtypes else x.to(dtype)
+
+ # preprocess query
+ if q_lens is None:
+ q = half(q.flatten(0, 1))
+ q_lens = torch.tensor(
+ [lq] * b, dtype=torch.int32).to(
+ device=q.device, non_blocking=True)
+ else:
+ q = half(torch.cat([u[:v] for u, v in zip(q, q_lens)]))
+
+ # preprocess key, value
+ if k_lens is None:
+ k = half(k.flatten(0, 1))
+ v = half(v.flatten(0, 1))
+ k_lens = torch.tensor(
+ [lk] * b, dtype=torch.int32).to(
+ device=k.device, non_blocking=True)
+ else:
+ k = half(torch.cat([u[:v] for u, v in zip(k, k_lens)]))
+ v = half(torch.cat([u[:v] for u, v in zip(v, k_lens)]))
+
+ q = q.to(v.dtype)
+ k = k.to(v.dtype)
+
+ if q_scale is not None:
+ q = q * q_scale
+
+ if version is not None and version == 3 and not FLASH_ATTN_3_AVAILABLE:
+ warnings.warn(
+ 'Flash attention 3 is not available, use flash attention 2 instead.'
+ )
+
+ # apply attention
+ if (version is None or version == 3) and FLASH_ATTN_3_AVAILABLE:
+ # Note: dropout_p, window_size are not supported in FA3 now.
+ x = flash_attn_interface.flash_attn_varlen_func(
+ q=q,
+ k=k,
+ v=v,
+ cu_seqlens_q=torch.cat([q_lens.new_zeros([1]), q_lens]).cumsum(
+ 0, dtype=torch.int32).to(q.device, non_blocking=True),
+ cu_seqlens_k=torch.cat([k_lens.new_zeros([1]), k_lens]).cumsum(
+ 0, dtype=torch.int32).to(q.device, non_blocking=True),
+ seqused_q=None,
+ seqused_k=None,
+ max_seqlen_q=lq,
+ max_seqlen_k=lk,
+ softmax_scale=softmax_scale,
+ causal=causal,
+ deterministic=deterministic).unflatten(0, (b, lq))
+ else:
+ assert FLASH_ATTN_2_AVAILABLE
+ x = flash_attn.flash_attn_varlen_func(
+ q=q,
+ k=k,
+ v=v,
+ cu_seqlens_q=torch.cat([q_lens.new_zeros([1]), q_lens]).cumsum(
+ 0, dtype=torch.int32).to(q.device, non_blocking=True),
+ cu_seqlens_k=torch.cat([k_lens.new_zeros([1]), k_lens]).cumsum(
+ 0, dtype=torch.int32).to(q.device, non_blocking=True),
+ max_seqlen_q=lq,
+ max_seqlen_k=lk,
+ dropout_p=dropout_p,
+ softmax_scale=softmax_scale,
+ causal=causal,
+ window_size=window_size,
+ deterministic=deterministic).unflatten(0, (b, lq))
+
+ # output
+ return x.type(out_dtype)
+
+
+def attention(
+ q,
+ k,
+ v,
+ q_lens=None,
+ k_lens=None,
+ dropout_p=0.,
+ softmax_scale=None,
+ q_scale=None,
+ causal=False,
+ window_size=(-1, -1),
+ deterministic=False,
+ dtype=torch.bfloat16,
+ fa_version=None,
+):
+ if FLASH_ATTN_2_AVAILABLE or FLASH_ATTN_3_AVAILABLE:
+ return flash_attention(
+ q=q,
+ k=k,
+ v=v,
+ q_lens=q_lens,
+ k_lens=k_lens,
+ dropout_p=dropout_p,
+ softmax_scale=softmax_scale,
+ q_scale=q_scale,
+ causal=causal,
+ window_size=window_size,
+ deterministic=deterministic,
+ dtype=dtype,
+ version=fa_version,
+ )
+ else:
+ if q_lens is not None or k_lens is not None:
+ warnings.warn(
+ 'Padding mask is disabled when using scaled_dot_product_attention. It can have a significant impact on performance.'
+ )
+ attn_mask = None
+
+ q = q.transpose(1, 2).to(dtype)
+ k = k.transpose(1, 2).to(dtype)
+ v = v.transpose(1, 2).to(dtype)
+
+ out = torch.nn.functional.scaled_dot_product_attention(
+ q, k, v, attn_mask=attn_mask, is_causal=causal, dropout_p=dropout_p)
+
+ out = out.transpose(1, 2).contiguous()
+ return out
diff --git a/worldplay2/models/causal_compressor.py b/worldplay2/models/causal_compressor.py
new file mode 100644
index 0000000..e4ba8dd
--- /dev/null
+++ b/worldplay2/models/causal_compressor.py
@@ -0,0 +1,167 @@
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+"""Causal memory compressor for WorldPlay2.
+
+This is the Wan2.2 inference-side port of the FastVideo training module
+``fastvideo/models/hyvideo/models/transformers/modules/causal_memory_compress.py``
+(``CausalMemCompressModel`` / ``CausalCompressConv3D``).
+
+The ONLY difference from the non-causal ``MemCompressModel`` is that every
+convolution spanning the TEMPORAL axis is made causal: an output frame at time
+``t`` may depend on input frames ``<= t`` (current + past) only, never on
+future frames. This is achieved by padding the temporal axis on the LEFT
+(past) side only -- exactly like ``CausalConv3d`` in the Wan VAE
+(``worldplay2/vae/wan21.py``) and ``WanCausalConv3d`` in the FastVideo wanvae.
+
+Concretely (matching the FastVideo training module verbatim):
+ - ``self.conv`` (kernel 3, temporal padding 1) becomes a ``CausalConv3d``:
+ instead of symmetric temporal padding (1 left + 1 right) it pads
+ ``2*1 = 2`` frames on the left and 0 on the right, so the output length is
+ unchanged but each frame only sees the past.
+ - ``self.time_conv`` (kernel 3, temporal stride 2 -- the temporal
+ downsample) becomes a ``CausalConv3d`` with temporal padding 1 -> 2 frames
+ of left padding, 0 right. The old non-causal implementation zero-padded 1
+ frame on the RIGHT (future) side, which leaked one future frame into every
+ downsampled output.
+
+ OUTPUT-LENGTH INVARIANT: the temporal downsample must still map
+ ``T_hr -> T_hr/2`` so ``hr_tokens`` line up token-by-token with
+ ``lr_tokens`` (which live at ``T_lr = T_hr/2``). For the wan latent
+ cadence ``T_hr`` is always EVEN; with an even input:
+ old (right-pad 1): out = (T_hr + 1 - 3)//2 + 1 = T_hr/2
+ new (left-pad 2): out = (T_hr + 2 - 3)//2 + 1 = T_hr/2
+ so the output length is IDENTICAL -- only the temporal receptive field
+ shifts from centered to left/causal.
+ - ``self.resample`` only touches H/W (spatial), so it is left unchanged.
+ - ``_AvgDown3D`` (the temporal shortcut) already pads temporal on the left
+ only, so it is already causal.
+
+The compressor here is run over the WHOLE history in one shot (not streamed),
+so ``CausalConv3d`` is always called with ``cache_x=None`` (standard causal
+left-pad). The ``cache_x`` streaming interface is inherited but unused.
+"""
+
+import torch
+import torch.nn as nn
+from einops import rearrange
+
+# Reuse the causal 3d conv from the Wan VAE (identical padding semantics to
+# FastVideo's WanCausalConv3d).
+from ..vae.wan21 import CausalConv3d
+# Reuse the (already-causal / time-independent) helper blocks from the
+# non-causal compressor so weights and shapes match the training module.
+from .compressor_blocks import _AvgDown3D, _WanAttentionBlock
+
+
+class CausalCompressConv3D(nn.Module):
+ """Causal-in-time version of ``memory_compress.CompressConv3D``.
+
+ Same structure and parameter layout as ``CompressConv3D`` except the two
+ temporal-spanning convolutions (``conv`` and ``time_conv``) are causal.
+ """
+
+ def __init__(self, in_dim, out_dim, spatial_down, temporal_down,
+ non_linearity='silu'):
+ super().__init__()
+ # Causal in time: kernel 3 with padding 1 -> left-only temporal pad.
+ self.conv = CausalConv3d(in_channels=in_dim, out_channels=out_dim,
+ kernel_size=3, stride=1, padding=1)
+ if non_linearity == 'silu':
+ self.nonlinearity = nn.SiLU()
+ else:
+ raise ValueError(
+ f"Unsupported non_linearity={non_linearity!r}. "
+ "Only 'silu' is used by Wan2.2 5B; extend if needed.")
+ self.spatial_down = spatial_down
+ self.temporal_down = temporal_down
+
+ if self.spatial_down:
+ # Spatial-only downsample -- no temporal axis touched, unchanged.
+ self.resample = nn.Sequential(
+ nn.ZeroPad2d((0, 1, 0, 1)),
+ nn.Conv2d(out_dim, out_dim, 3, stride=(2, 2)),
+ )
+
+ if self.temporal_down:
+ # Causal temporal downsample: kernel 3, stride 2, temporal
+ # padding 1 -> CausalConv3d pads 2 frames on the LEFT (past) and 0
+ # on the right, so no future frame leaks into the output.
+ self.time_conv = CausalConv3d(
+ out_dim, out_dim, kernel_size=(3, 1, 1),
+ stride=(2, 1, 1), padding=(1, 0, 0))
+
+ if self.spatial_down or self.temporal_down:
+ self.avg_shortcut = _AvgDown3D(
+ in_dim, out_dim,
+ factor_t=2 if temporal_down else 1,
+ factor_s=2 if spatial_down else 1,
+ )
+ else:
+ self.avg_shortcut = nn.Conv3d(in_dim, out_dim, kernel_size=1,
+ stride=1, padding=0)
+
+ def forward(self, x):
+ x_copy = x.clone()
+ x = self.conv(x)
+ x = self.nonlinearity(x)
+ b, c, t, h, w = x.shape
+ if self.spatial_down:
+ x = x.permute(0, 2, 1, 3, 4).reshape(b * t, c, h, w)
+ x = self.resample(x)
+ x = x.view(b, t, x.size(1), x.size(2),
+ x.size(3)).permute(0, 2, 1, 3, 4)
+ if self.temporal_down:
+ x = self.time_conv(x)
+ return x + self.avg_shortcut(x_copy)
+
+
+class CausalMemCompressModel(nn.Module):
+ """Causal-in-time variant of ``memory_compress.MemCompressModel``.
+
+ Identical to ``MemCompressModel`` except each ``CompressConv3D`` block is
+ replaced by ``CausalCompressConv3D`` so the temporal receptive field is
+ strictly causal (an output token never depends on future latent frames).
+ The ``attn_blocks`` are per-frame spatial attention (no temporal mixing),
+ so they are already causal and reused verbatim.
+ """
+
+ def __init__(
+ self,
+ output_dim=2048,
+ input_dim=65,
+ dims=(64, 64, 128, 256, 256, 512, 512),
+ spatial_down=(1, 1, 0, 0, 0, 0),
+ temporal_down=(1, 0, 0, 0, 0, 0),
+ attn_num=2,
+ non_linearity: str = "silu",
+ ):
+ super().__init__()
+ dims = list(dims)
+ spatial_down = list(spatial_down)
+ temporal_down = list(temporal_down)
+
+ self.input_layer = nn.Linear(input_dim, dims[0])
+
+ self.blocks = nn.ModuleList()
+ for i, (in_dim, out_dim) in enumerate(zip(dims[:-1], dims[1:])):
+ self.blocks.append(CausalCompressConv3D(
+ in_dim, out_dim, spatial_down[i], temporal_down[i],
+ non_linearity=non_linearity))
+
+ self.attn_blocks = nn.ModuleList(
+ [_WanAttentionBlock(dims[-1], head_dim=8) for _ in range(attn_num)])
+
+ self.output_layer = nn.Linear(dims[-1], output_dim)
+ nn.init.zeros_(self.output_layer.weight)
+ nn.init.zeros_(self.output_layer.bias)
+
+ def forward(self, x):
+ B, C, T, H, W = x.shape
+ x = rearrange(x, 'B C T H W -> B (T H W) C')
+ x = self.input_layer(x)
+ x = rearrange(x, 'B (T H W) C -> B C T H W', T=T, H=H, W=W)
+ for block in self.blocks:
+ x = block(x)
+ for attn_block in self.attn_blocks:
+ x = attn_block(x)
+ x = rearrange(x, 'B C T H W -> B (T H W) C')
+ return self.output_layer(x)
diff --git a/worldplay2/models/compressor_blocks.py b/worldplay2/models/compressor_blocks.py
new file mode 100644
index 0000000..6c9cff8
--- /dev/null
+++ b/worldplay2/models/compressor_blocks.py
@@ -0,0 +1,129 @@
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+"""Hierarchical HR -> compressed-memory feature extractor used by the
+stage-2 / stage-3 chunked inference recipe.
+
+This is a self-contained port of ``MemCompressModel`` from the FastVideo
+training pipeline. It only depends on ``torch.nn`` plus ``einops`` so
+that it can be imported from ``worldplay2/models/model.py`` without dragging in
+any FastVideo-specific layers / autocast helpers.
+
+Wan2.2 5B contract (matches ``WorldPlay2Model.add_memory_compress_model``):
+ input_dim = 36 (16 hr-latent + 20 hr-y)
+ spatial_down = [1, 1, 1, 0, 0, 0] (3 spatial down blocks * 2 each = /8)
+ temporal_down = [1, 0, 0, 0, 0, 0] (1 temporal down block * 2 = /2)
+"""
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from einops import rearrange
+
+
+# ---------------------------------------------------------------------------
+# Local copies of the 3 helpers MemCompressModel uses from FastVideo.
+# Kept verbatim so checkpoints trained against FastVideo load directly.
+# ---------------------------------------------------------------------------
+
+class _AvgDown3D(nn.Module):
+ """Channel-folding average-pool used as the residual shortcut inside
+ ``CompressConv3D``. Identical to ``fastvideo.models.vaes.wanvae.AvgDown3D``.
+ """
+
+ def __init__(self, in_channels, out_channels, factor_t, factor_s=1):
+ super().__init__()
+ self.in_channels = in_channels
+ self.out_channels = out_channels
+ self.factor_t = factor_t
+ self.factor_s = factor_s
+ self.factor = factor_t * factor_s * factor_s
+
+ assert in_channels * self.factor % out_channels == 0
+ self.group_size = in_channels * self.factor // out_channels
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ pad_t = (self.factor_t - x.shape[2] % self.factor_t) % self.factor_t
+ pad = (0, 0, 0, 0, pad_t, 0)
+ x = F.pad(x, pad)
+ B, C, T, H, W = x.shape
+ x = x.view(
+ B, C,
+ T // self.factor_t, self.factor_t,
+ H // self.factor_s, self.factor_s,
+ W // self.factor_s, self.factor_s,
+ )
+ x = x.permute(0, 1, 3, 5, 7, 2, 4, 6).contiguous()
+ x = x.view(
+ B, C * self.factor,
+ T // self.factor_t,
+ H // self.factor_s,
+ W // self.factor_s,
+ )
+ x = x.view(
+ B, self.out_channels, self.group_size,
+ T // self.factor_t,
+ H // self.factor_s,
+ W // self.factor_s,
+ )
+ return x.mean(dim=2)
+
+
+class _WanRMS_norm(nn.Module):
+ """Same as ``fastvideo.models.vaes.wanvae.WanRMS_norm``. Operates
+ per-frame on (B*T, C, H, W) tensors.
+ """
+
+ def __init__(self, dim, channel_first=True, images=True, bias=False):
+ super().__init__()
+ broadcastable_dims = (1, 1, 1) if not images else (1, 1)
+ shape = (dim, *broadcastable_dims) if channel_first else (dim,)
+ self.channel_first = channel_first
+ self.scale = dim ** 0.5
+ self.gamma = nn.Parameter(torch.ones(shape))
+ self.bias = nn.Parameter(torch.zeros(shape)) if bias else 0.0
+
+ def forward(self, x):
+ return F.normalize(
+ x, dim=(1 if self.channel_first else -1)
+ ) * self.scale * self.gamma + self.bias
+
+
+class _WanAttentionBlock(nn.Module):
+ """Multi-head per-frame self-attention. Identical to
+ ``fastvideo.models.vaes.wanvae.WanAttentionBlock``; this is the
+ ``head_dim=8`` variant used by ``MemCompressModel``.
+ """
+
+ def __init__(self, dim, head_dim=1):
+ super().__init__()
+ self.dim = dim
+ self.head_dim = head_dim
+ self.norm = _WanRMS_norm(dim)
+ self.to_qkv = nn.Conv2d(dim, dim * 3, 1)
+ self.proj = nn.Conv2d(dim, dim, 1)
+
+ def forward(self, x):
+ identity = x
+ batch_size, channels, time, height, width = x.size()
+
+ x = x.permute(0, 2, 1, 3, 4).reshape(
+ batch_size * time, channels, height, width)
+ x = self.norm(x)
+
+ qkv = self.to_qkv(x)
+ qkv = qkv.reshape(batch_size * time, 1, channels * 3, -1)
+ qkv = qkv.permute(0, 1, 3, 2).contiguous()
+ q, k, v = qkv.chunk(3, dim=-1)
+ q = rearrange(q, "b h l (H d) -> b (h H) l d", H=self.head_dim)
+ k = rearrange(k, "b h l (H d) -> b (h H) l d", H=self.head_dim)
+ v = rearrange(v, "b h l (H d) -> b (h H) l d", H=self.head_dim)
+
+ x = F.scaled_dot_product_attention(q, k, v)
+ x = rearrange(x, "b h l d -> b (h d) l")
+
+ x = x.reshape(batch_size * time, channels, height, width)
+ x = self.proj(x)
+
+ x = x.view(batch_size, time, channels, height, width)
+ x = x.permute(0, 2, 1, 3, 4)
+ return x + identity
+
diff --git a/worldplay2/models/fp8.py b/worldplay2/models/fp8.py
new file mode 100644
index 0000000..80de6f5
--- /dev/null
+++ b/worldplay2/models/fp8.py
@@ -0,0 +1,400 @@
+from __future__ import annotations
+
+import gc
+from contextlib import nullcontext
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+def _load_transformer_engine():
+ try:
+ import transformer_engine.pytorch as te
+ from transformer_engine.common.recipe import DelayedScaling, Format
+ except ImportError as exc:
+ raise RuntimeError(
+ "FFN FP8 requires NVIDIA Transformer Engine. Install the "
+ "PyTorch extension with `pip install 'transformer_engine[pytorch]'`."
+ ) from exc
+ return te, DelayedScaling, Format
+
+
+def make_fp8_recipe():
+ """Hopper-friendly tensor-wise E4M3 delayed scaling for inference."""
+ te, DelayedScaling, Format = _load_transformer_engine()
+ available = te.is_fp8_available(return_reason=True)
+ if isinstance(available, tuple):
+ is_available, reason = available
+ else:
+ is_available, reason = available, "unsupported GPU/runtime"
+ if not is_available:
+ raise RuntimeError(f"Transformer Engine FP8 is unavailable: {reason}")
+ return DelayedScaling(
+ fp8_format=Format.E4M3,
+ amax_history_len=16,
+ amax_compute_algo="max",
+ # Sequence-parallel ranks process different tokens. Local activation
+ # scales avoid an extra amax all-reduce and are dequantized before the
+ # next distributed attention.
+ reduce_amax=False,
+ )
+
+
+def fp8_autocast(recipe, enabled: bool):
+ if not enabled:
+ return nullcontext()
+ te, _, _ = _load_transformer_engine()
+ autocast = getattr(te, "autocast", None)
+ if autocast is None:
+ autocast = te.fp8_autocast
+ return autocast(enabled=True, recipe=recipe)
+
+
+def _primary_fp8_init_context(te, recipe):
+ """Create TE modules whose primary weights are stored only in FP8."""
+ quantized_model_init = getattr(te, "quantized_model_init", None)
+ if quantized_model_init is None:
+ raise RuntimeError(
+ "This FP8 inference path requires "
+ "transformer_engine.pytorch.quantized_model_init. "
+ "Transformer Engine 2.18.0 provides this API."
+ )
+ return quantized_model_init(
+ enabled=True,
+ recipe=recipe,
+ preserve_high_precision_init_val=False,
+ )
+
+
+def _copy_weight_to_primary_fp8(destination, source, recipe) -> None:
+ """Calibrate a static E4M3 weight, then quantize it into TE storage."""
+ quantizer_getter = getattr(destination, "_get_quantizer", None)
+ if quantizer_getter is not None:
+ quantizer = quantizer_getter()
+ scale = getattr(quantizer, "scale", None)
+ scale_inv = getattr(destination, "_scale_inv", None)
+ if scale is not None and scale_inv is not None:
+ # DelayedScaling would otherwise reuse the random initialization
+ # scale for this first copy. Static inference weights can be
+ # calibrated exactly once from their checkpoint amax.
+ amax = source.detach().abs().amax().float()
+ fp8_max = 448.0 # E4M3 finite maximum
+ margin = int(getattr(recipe, "margin", 0))
+ calibrated_scale = (fp8_max / amax.clamp_min(
+ torch.finfo(torch.float32).tiny)) / (2 ** margin)
+ scale.copy_(calibrated_scale.to(scale))
+ scale_inv.copy_(calibrated_scale.reciprocal().to(scale_inv))
+
+ destination.copy_(source)
+
+
+class TransformerEngineFP8FFN(nn.Module):
+ """FFN with FP8-only primary weights and FP8 E4M3 GEMM execution."""
+
+ uses_fp8 = True
+
+ def __init__(self, source: nn.Sequential, recipe, name: str):
+ super().__init__()
+ if (
+ len(source) != 3
+ or not isinstance(source[0], nn.Linear)
+ or not isinstance(source[1], nn.GELU)
+ or not isinstance(source[2], nn.Linear)
+ ):
+ raise TypeError("FP8 FFN expects Linear -> GELU -> Linear")
+
+ te, _, _ = _load_transformer_engine()
+ first, activation, second = source
+ if first.in_features % 16 or first.out_features % 16:
+ raise ValueError(
+ f"{name}.ffn[0] shape is not FP8 GEMM eligible: "
+ f"{first.in_features}x{first.out_features}"
+ )
+ if second.in_features % 16 or second.out_features % 16:
+ raise ValueError(
+ f"{name}.ffn[2] shape is not FP8 GEMM eligible: "
+ f"{second.in_features}x{second.out_features}"
+ )
+
+ device = first.weight.device
+ dtype = first.weight.dtype
+ # TE only honors quantized_model_init during no-grad construction.
+ # params_dtype remains BF16 as the nominal compute/output dtype, while
+ # the actual primary weight storage is a TE Float8Tensor.
+ with torch.no_grad(), _primary_fp8_init_context(te, recipe):
+ self.fc1 = te.Linear(
+ first.in_features,
+ first.out_features,
+ bias=first.bias is not None,
+ params_dtype=dtype,
+ device=device,
+ name=f"{name}.fc1",
+ )
+ self.fc2 = te.Linear(
+ second.in_features,
+ second.out_features,
+ bias=second.bias is not None,
+ params_dtype=dtype,
+ device=device,
+ name=f"{name}.fc2",
+ )
+
+ if not (
+ getattr(self.fc1, "primary_weights_in_fp8", False)
+ and getattr(self.fc2, "primary_weights_in_fp8", False)
+ ):
+ raise RuntimeError(
+ f"{name}: Transformer Engine did not create FP8 primary "
+ "weights. Check that TE 2.18.0 is installed and that "
+ "quantized_model_init is enabled."
+ )
+
+ # QuantizedTensor.copy_ quantizes the BF16 checkpoint tensor directly
+ # into the FP8 destination and updates its scale metadata.
+ with torch.no_grad():
+ _copy_weight_to_primary_fp8(
+ self.fc1.weight, first.weight, recipe)
+ _copy_weight_to_primary_fp8(
+ self.fc2.weight, second.weight, recipe)
+ if first.bias is not None:
+ self.fc1.bias.copy_(first.bias)
+ if second.bias is not None:
+ self.fc2.bias.copy_(second.bias)
+
+ self.activation = nn.GELU(approximate=activation.approximate)
+ self.recipe = recipe
+ self.primary_weights_in_fp8 = True
+ self.forward_calls = 0
+
+ def forward(self, hidden_states):
+ # Transformer Engine FP8 GEMM requires:
+ # M = product(hidden_states.shape[:-1]) to be divisible by 8
+ # K = hidden_states.shape[-1] to be divisible by 16
+ #
+ # Sequence parallelism can produce local token counts such as 387 or
+ # 386. FFN is token-wise, so pad only inside the FFN and remove the
+ # padding before returning. Padded tokens never enter attention.
+ original_shape = hidden_states.shape
+ hidden_size = original_shape[-1]
+ hidden_states = hidden_states.reshape(-1, hidden_size)
+ num_tokens = hidden_states.shape[0]
+ padded_tokens = (-num_tokens) % 8
+ if padded_tokens:
+ hidden_states = F.pad(
+ hidden_states, (0, 0, 0, padded_tokens), value=0.0)
+
+ # The weights are already primary Float8Tensors. Do not request the
+ # BF16->FP8 weight workspace/cache used by regular TE Linear modules.
+ hidden_states = self.fc1(hidden_states)
+ hidden_states = self.activation(hidden_states)
+ hidden_states = self.fc2(hidden_states)
+
+ if padded_tokens:
+ hidden_states = hidden_states[:num_tokens]
+ hidden_states = hidden_states.reshape(*original_shape[:-1], -1)
+
+ self.forward_calls += 1
+ return hidden_states
+
+
+class TransformerEngineFP8Linear(nn.Module):
+ """Drop-in inference Linear with FP8-only primary weight storage."""
+
+ uses_fp8 = True
+ primary_weights_in_fp8 = True
+
+ def __init__(self, source: nn.Linear, recipe, name: str):
+ super().__init__()
+ if not isinstance(source, nn.Linear):
+ raise TypeError("FP8 Linear conversion expects nn.Linear")
+ if source.in_features % 16 or source.out_features % 16:
+ raise ValueError(
+ f"{name} shape is not FP8 GEMM eligible: "
+ f"{source.in_features}x{source.out_features}"
+ )
+
+ te, _, _ = _load_transformer_engine()
+ with torch.no_grad(), _primary_fp8_init_context(te, recipe):
+ self.linear = te.Linear(
+ source.in_features,
+ source.out_features,
+ bias=source.bias is not None,
+ params_dtype=source.weight.dtype,
+ device=source.weight.device,
+ name=name,
+ )
+ if not getattr(self.linear, "primary_weights_in_fp8", False):
+ raise RuntimeError(
+ f"{name}: Transformer Engine did not create an FP8 primary "
+ "weight. Check the TE 2.18.0 installation."
+ )
+
+ with torch.no_grad():
+ _copy_weight_to_primary_fp8(
+ self.linear.weight, source.weight, recipe)
+ if source.bias is not None:
+ self.linear.bias.copy_(source.bias)
+
+ self.in_features = source.in_features
+ self.out_features = source.out_features
+ self.forward_calls = 0
+
+ @property
+ def weight(self):
+ return self.linear.weight
+
+ @property
+ def bias(self):
+ return self.linear.bias
+
+ def forward(self, hidden_states):
+ original_shape = hidden_states.shape
+ hidden_states = hidden_states.reshape(-1, original_shape[-1])
+ num_tokens = hidden_states.shape[0]
+ padded_tokens = (-num_tokens) % 8
+ if padded_tokens:
+ hidden_states = F.pad(
+ hidden_states, (0, 0, 0, padded_tokens), value=0.0)
+
+ hidden_states = self.linear(hidden_states)
+
+ if padded_tokens:
+ hidden_states = hidden_states[:num_tokens]
+ hidden_states = hidden_states.reshape(
+ *original_shape[:-1], self.out_features)
+ self.forward_calls += 1
+ return hidden_states
+
+
+def convert_model_ffn_to_fp8(
+ model,
+ *,
+ start_block: int,
+ end_block: int,
+) -> dict:
+ """Replace inclusive middle-block FFNs with Transformer Engine modules."""
+ if start_block < 0 or end_block < start_block:
+ raise ValueError("invalid FP8 block range")
+ if end_block >= len(model.blocks):
+ raise ValueError(
+ f"FP8 end block {end_block} exceeds model depth "
+ f"{len(model.blocks)}"
+ )
+
+ recipe = model.fp8_recipe or make_fp8_recipe()
+ converted = []
+ for index in range(start_block, end_block + 1):
+ block = model.blocks[index]
+ if getattr(block.ffn, "uses_fp8", False):
+ converted.append(index)
+ continue
+ block.ffn = TransformerEngineFP8FFN(
+ block.ffn, recipe, name=f"blocks.{index}.ffn")
+ block.ffn.eval().requires_grad_(False)
+ converted.append(index)
+
+ model.fp8_ffn_enabled = True
+ model.fp8_enabled = True
+ model.fp8_recipe = recipe
+ model.fp8_block_range = (start_block, end_block)
+ gc.collect()
+ torch.cuda.empty_cache()
+ return {
+ "enabled": True,
+ "recipe": "DelayedScaling",
+ "format": "E4M3",
+ "weight_storage": "primary_fp8",
+ "bf16_master_weights": False,
+ "reduce_amax": False,
+ "blocks": converted,
+ }
+
+
+def convert_model_qkv_to_fp8(
+ model,
+ *,
+ start_block: int,
+ end_block: int,
+) -> dict:
+ """Convert selected fused self-attention QKV projections to FP8."""
+ if start_block < 0 or end_block < start_block:
+ raise ValueError("invalid FP8 QKV block range")
+ if end_block >= len(model.blocks):
+ raise ValueError(
+ f"FP8 QKV end block {end_block} exceeds model depth "
+ f"{len(model.blocks)}"
+ )
+ if not getattr(model, "self_qkv_fused", False):
+ raise ValueError(
+ "QKV FP8 requires fused self-attention QKV projections")
+
+ recipe = model.fp8_recipe or make_fp8_recipe()
+ converted = []
+ for index in range(start_block, end_block + 1):
+ attention = model.blocks[index].self_attn
+ if attention.qkv is None:
+ raise RuntimeError(
+ f"blocks.{index}.self_attn has no fused QKV projection")
+ if getattr(attention.qkv, "uses_fp8", False):
+ converted.append(index)
+ continue
+ attention.qkv = TransformerEngineFP8Linear(
+ attention.qkv,
+ recipe,
+ name=f"blocks.{index}.self_attn.qkv",
+ )
+ attention.qkv.eval().requires_grad_(False)
+ converted.append(index)
+
+ model.fp8_qkv_enabled = True
+ model.fp8_enabled = True
+ model.fp8_recipe = recipe
+ model.fp8_qkv_block_range = (start_block, end_block)
+ gc.collect()
+ torch.cuda.empty_cache()
+ return {
+ "enabled": True,
+ "recipe": "DelayedScaling",
+ "format": "E4M3",
+ "weight_storage": "primary_fp8",
+ "bf16_master_weights": False,
+ "reduce_amax": False,
+ "blocks": converted,
+ }
+
+
+def fp8_model_stats(model) -> dict:
+ ffn_blocks = []
+ ffn_calls = 0
+ qkv_blocks = []
+ qkv_calls = 0
+ for index, block in enumerate(model.blocks):
+ if getattr(block.ffn, "uses_fp8", False):
+ ffn_blocks.append(index)
+ ffn_calls += int(block.ffn.forward_calls)
+ if getattr(block.self_attn.qkv, "uses_fp8", False):
+ qkv_blocks.append(index)
+ qkv_calls += int(block.self_attn.qkv.forward_calls)
+ enabled = bool(ffn_blocks or qkv_blocks)
+ return {
+ "enabled": enabled,
+ "ffn": {
+ "enabled": bool(ffn_blocks),
+ "blocks": ffn_blocks,
+ "forward_calls": ffn_calls,
+ },
+ "qkv": {
+ "enabled": bool(qkv_blocks),
+ "blocks": qkv_blocks,
+ "forward_calls": qkv_calls,
+ },
+ "primary_weights_in_fp8": all(
+ model.blocks[index].ffn.primary_weights_in_fp8
+ for index in ffn_blocks
+ ) and all(
+ model.blocks[index].self_attn.qkv.primary_weights_in_fp8
+ for index in qkv_blocks
+ ) if enabled else False,
+ "bf16_master_weights": False if enabled else None,
+ }
diff --git a/worldplay2/models/kernels.py b/worldplay2/models/kernels.py
new file mode 100644
index 0000000..62b64b6
--- /dev/null
+++ b/worldplay2/models/kernels.py
@@ -0,0 +1,496 @@
+from __future__ import annotations
+
+import logging
+from functools import lru_cache
+
+import torch
+import torch.nn.functional as F
+
+try:
+ import triton
+ import triton.language as tl
+except ImportError:
+ triton = None
+ tl = None
+
+
+_QK_ROPE_STATS = {
+ "compiled_calls": 0,
+ "eager_calls": 0,
+ "compile_failures": 0,
+}
+_REGION_STATS = {
+ "layer_norm_modulation": {
+ "triton_calls": 0,
+ "eager_calls": 0,
+ "failures": 0,
+ },
+ "modulation": {"compiled_calls": 0, "eager_calls": 0, "failures": 0},
+ "gated_residual": {
+ "compiled_calls": 0,
+ "eager_calls": 0,
+ "failures": 0,
+ },
+ "action_mlp": {
+ "compiled_calls": 0,
+ "eager_calls": 0,
+ "failures": 0,
+ },
+ "ffn": {"compiled_calls": 0, "eager_calls": 0, "failures": 0},
+}
+_DISABLED_REGIONS: set[str] = set()
+
+
+if triton is not None:
+
+ @triton.jit
+ def _layer_norm_modulation_kernel(
+ x_ptr,
+ scale_ptr,
+ shift_ptr,
+ output_ptr,
+ sequence_length: tl.constexpr,
+ hidden_size: tl.constexpr,
+ x_stride_b: tl.constexpr,
+ x_stride_s: tl.constexpr,
+ scale_stride_b: tl.constexpr,
+ scale_stride_s: tl.constexpr,
+ shift_stride_b: tl.constexpr,
+ shift_stride_s: tl.constexpr,
+ eps: tl.constexpr,
+ INPUT_DTYPE: tl.constexpr,
+ BLOCK_SIZE: tl.constexpr,
+ ):
+ """One Triton program normalizes and modulates one token row."""
+ row = tl.program_id(0)
+ batch_index = row // sequence_length
+ sequence_index = row - batch_index * sequence_length
+ columns = tl.arange(0, BLOCK_SIZE)
+ mask = columns < hidden_size
+
+ x_offsets = (
+ batch_index * x_stride_b
+ + sequence_index * x_stride_s
+ + columns
+ )
+ x = tl.load(x_ptr + x_offsets, mask=mask, other=0.0).to(tl.float32)
+ mean = tl.sum(x, axis=0) / hidden_size
+ centered = tl.where(mask, x - mean, 0.0)
+ variance = tl.sum(centered * centered, axis=0) / hidden_size
+ normalized = centered * tl.rsqrt(variance + eps)
+ # Match WorldPlay2LayerNorm exactly: FP32 normalization is rounded back to
+ # the activation dtype before adaptive modulation evaluates in FP32.
+ if INPUT_DTYPE == 1:
+ normalized = normalized.to(tl.bfloat16).to(tl.float32)
+ elif INPUT_DTYPE == 0:
+ normalized = normalized.to(tl.float16).to(tl.float32)
+
+ scale_offsets = (
+ batch_index * scale_stride_b
+ + sequence_index * scale_stride_s
+ + columns
+ )
+ shift_offsets = (
+ batch_index * shift_stride_b
+ + sequence_index * shift_stride_s
+ + columns
+ )
+ scale = tl.load(
+ scale_ptr + scale_offsets, mask=mask, other=0.0
+ ).to(tl.float32)
+ shift = tl.load(
+ shift_ptr + shift_offsets, mask=mask, other=0.0
+ ).to(tl.float32)
+ output = normalized * (1.0 + scale) + shift
+
+ output_offsets = row * hidden_size + columns
+ tl.store(output_ptr + output_offsets, output, mask=mask)
+
+
+def qk_norm_rope_eager(
+ q: torch.Tensor,
+ k: torch.Tensor,
+ norm_q_weight: torch.Tensor,
+ norm_k_weight: torch.Tensor,
+ freqs_real: torch.Tensor,
+ freqs_imag: torch.Tensor,
+ *,
+ eps: float,
+ num_heads: int,
+ head_dim: int,
+) -> tuple[torch.Tensor, torch.Tensor]:
+ """RMS-normalize Q/K and apply real-valued RoPE in FP32."""
+
+ def normalize_and_rotate(x, weight):
+ normalized = x.float()
+ normalized = normalized * torch.rsqrt(
+ normalized.square().mean(dim=-1, keepdim=True) + eps
+ )
+ # Preserve the original WorldPlay2RMSNorm order: normalize in FP32, cast
+ # back to projection dtype, then apply the learned norm weight.
+ normalized = normalized.to(x.dtype) * weight
+ batch, sequence, _ = normalized.shape
+ pairs = normalized.float().view(
+ batch, sequence, num_heads, head_dim // 2, 2)
+ even = pairs[..., 0]
+ odd = pairs[..., 1]
+ cos = freqs_real.float().view(
+ 1, sequence, 1, head_dim // 2)
+ sin = freqs_imag.float().view(
+ 1, sequence, 1, head_dim // 2)
+ return torch.stack(
+ [even * cos - odd * sin, even * sin + odd * cos],
+ dim=-1,
+ ).flatten(-2)
+
+ return (
+ normalize_and_rotate(q, norm_q_weight),
+ normalize_and_rotate(k, norm_k_weight),
+ )
+
+
+@lru_cache(maxsize=16)
+def _compiled_kernel(num_heads: int, head_dim: int, eps: float):
+ def kernel(q, k, norm_q_weight, norm_k_weight, freqs_real, freqs_imag):
+ return qk_norm_rope_eager(
+ q,
+ k,
+ norm_q_weight,
+ norm_k_weight,
+ freqs_real,
+ freqs_imag,
+ eps=eps,
+ num_heads=num_heads,
+ head_dim=head_dim,
+ )
+
+ return torch.compile(
+ kernel,
+ fullgraph=True,
+ dynamic=True,
+ mode="max-autotune-no-cudagraphs",
+ )
+
+
+def qk_norm_rope(
+ q: torch.Tensor,
+ k: torch.Tensor,
+ norm_q_weight: torch.Tensor,
+ norm_k_weight: torch.Tensor,
+ freqs_flat: torch.Tensor,
+ *,
+ eps: float,
+ num_heads: int,
+ head_dim: int,
+ compile_enabled: bool,
+) -> tuple[torch.Tensor, torch.Tensor, bool]:
+ """Run compiled QK norm + RoPE, falling back to the same eager kernel."""
+ args = (
+ q,
+ k,
+ norm_q_weight,
+ norm_k_weight,
+ freqs_flat.real,
+ freqs_flat.imag,
+ )
+ if compile_enabled:
+ try:
+ output = _compiled_kernel(num_heads, head_dim, eps)(*args)
+ _QK_ROPE_STATS["compiled_calls"] += 1
+ return output[0], output[1], True
+ except Exception:
+ _QK_ROPE_STATS["compile_failures"] += 1
+ logging.exception(
+ "Compiled QK norm + RoPE failed; using eager fallback")
+
+ output = qk_norm_rope_eager(
+ *args,
+ eps=eps,
+ num_heads=num_heads,
+ head_dim=head_dim,
+ )
+ _QK_ROPE_STATS["eager_calls"] += 1
+ return output[0], output[1], False
+
+
+def get_qk_rope_compile_stats() -> dict:
+ return dict(_QK_ROPE_STATS)
+
+
+# ---------------------------------------------------------------------------
+# Repeated Wan block regions
+# ---------------------------------------------------------------------------
+
+def layer_norm_modulation_eager(
+ hidden_states,
+ scale,
+ shift,
+ *,
+ eps,
+):
+ """Reference Wan LayerNorm followed by adaptive scale and shift."""
+ normalized = F.layer_norm(
+ hidden_states.float(),
+ (hidden_states.shape[-1],),
+ weight=None,
+ bias=None,
+ eps=eps,
+ ).to(hidden_states.dtype)
+ return normalized.float() * (1.0 + scale.float()) + shift.float()
+
+
+def layer_norm_modulation(
+ hidden_states,
+ scale,
+ shift,
+ *,
+ eps,
+ enabled,
+):
+ """Fuse non-affine LayerNorm and Wan modulation in one Triton kernel."""
+ name = "layer_norm_modulation"
+ can_use_triton = (
+ enabled
+ and name not in _DISABLED_REGIONS
+ and triton is not None
+ and hidden_states.is_cuda
+ and hidden_states.ndim == 3
+ and scale.ndim == 3
+ and shift.ndim == 3
+ and hidden_states.shape == scale.shape == shift.shape
+ and hidden_states.dtype in (
+ torch.bfloat16,
+ torch.float16,
+ torch.float32,
+ )
+ and hidden_states.stride(-1) == 1
+ and scale.stride(-1) == 1
+ and shift.stride(-1) == 1
+ )
+ if can_use_triton:
+ try:
+ batch, sequence, hidden_size = hidden_states.shape
+ output = torch.empty(
+ hidden_states.shape,
+ device=hidden_states.device,
+ dtype=torch.float32,
+ )
+ block_size = triton.next_power_of_2(hidden_size)
+ if block_size > 65536:
+ raise ValueError(
+ f"LayerNorm hidden size {hidden_size} is too large")
+ num_warps = 8 if block_size >= 4096 else 4
+ _layer_norm_modulation_kernel[(batch * sequence,)](
+ hidden_states,
+ scale,
+ shift,
+ output,
+ sequence_length=sequence,
+ hidden_size=hidden_size,
+ x_stride_b=hidden_states.stride(0),
+ x_stride_s=hidden_states.stride(1),
+ scale_stride_b=scale.stride(0),
+ scale_stride_s=scale.stride(1),
+ shift_stride_b=shift.stride(0),
+ shift_stride_s=shift.stride(1),
+ eps=eps,
+ INPUT_DTYPE=(
+ 1 if hidden_states.dtype == torch.bfloat16
+ else 0 if hidden_states.dtype == torch.float16
+ else 2
+ ),
+ BLOCK_SIZE=block_size,
+ num_warps=num_warps,
+ )
+ _REGION_STATS[name]["triton_calls"] += 1
+ return output, True
+ except Exception:
+ _REGION_STATS[name]["failures"] += 1
+ _DISABLED_REGIONS.add(name)
+ logging.exception(
+ "Triton fused LayerNorm + modulation failed; "
+ "disabling it and using eager fallback"
+ )
+
+ _REGION_STATS[name]["eager_calls"] += 1
+ return layer_norm_modulation_eager(
+ hidden_states,
+ scale,
+ shift,
+ eps=eps,
+ ), False
+
+
+def modulation_eager(normalized, scale, shift):
+ """Exact block modulation expression, evaluated in FP32."""
+ return normalized.float() * (1.0 + scale.float()) + shift.float()
+
+
+def gated_residual_eager(hidden_states, branch, gate):
+ """Exact gated residual expression used by WorldPlay2AttentionBlock."""
+ return hidden_states + branch * gate
+
+
+def action_mlp_eager(
+ hidden_states,
+ action,
+ input_weight,
+ input_bias,
+ output_weight,
+ output_bias,
+):
+ """Action injection: concat, MLP, and residual add."""
+ branch = torch.cat([hidden_states, action], dim=-1)
+ branch = F.linear(branch, input_weight, input_bias)
+ branch = F.silu(branch)
+ branch = F.linear(branch, output_weight, output_bias)
+ return hidden_states + branch
+
+
+def ffn_eager(
+ hidden_states,
+ input_weight,
+ input_bias,
+ output_weight,
+ output_bias,
+):
+ hidden_states = F.linear(
+ hidden_states, input_weight, input_bias)
+ hidden_states = F.gelu(hidden_states, approximate="tanh")
+ return F.linear(hidden_states, output_weight, output_bias)
+
+
+@lru_cache(maxsize=1)
+def _compiled_modulation():
+ return torch.compile(
+ modulation_eager,
+ fullgraph=True,
+ dynamic=True,
+ mode="max-autotune-no-cudagraphs",
+ )
+
+
+@lru_cache(maxsize=1)
+def _compiled_gated_residual():
+ return torch.compile(
+ gated_residual_eager,
+ fullgraph=True,
+ dynamic=True,
+ mode="max-autotune-no-cudagraphs",
+ )
+
+
+@lru_cache(maxsize=1)
+def _compiled_action_mlp():
+ return torch.compile(
+ action_mlp_eager,
+ fullgraph=True,
+ dynamic=True,
+ mode="max-autotune-no-cudagraphs",
+ )
+
+
+@lru_cache(maxsize=1)
+def _compiled_ffn():
+ return torch.compile(
+ ffn_eager,
+ fullgraph=True,
+ dynamic=True,
+ mode="max-autotune-no-cudagraphs",
+ )
+
+
+def _run_region(name, eager, compiled_factory, args, enabled):
+ if enabled and name not in _DISABLED_REGIONS:
+ try:
+ output = compiled_factory()(*args)
+ _REGION_STATS[name]["compiled_calls"] += 1
+ return output, True
+ except Exception:
+ _REGION_STATS[name]["failures"] += 1
+ _DISABLED_REGIONS.add(name)
+ logging.exception(
+ "Compiled region %s failed; disabling it and using eager",
+ name,
+ )
+ _REGION_STATS[name]["eager_calls"] += 1
+ return eager(*args), False
+
+
+def modulation(normalized, scale, shift, *, enabled):
+ return _run_region(
+ "modulation",
+ modulation_eager,
+ _compiled_modulation,
+ (normalized, scale, shift),
+ enabled,
+ )
+
+
+def gated_residual(hidden_states, branch, gate, *, enabled):
+ return _run_region(
+ "gated_residual",
+ gated_residual_eager,
+ _compiled_gated_residual,
+ (hidden_states, branch, gate),
+ enabled,
+ )
+
+
+def action_mlp(
+ hidden_states,
+ action,
+ input_weight,
+ input_bias,
+ output_weight,
+ output_bias,
+ *,
+ enabled,
+):
+ return _run_region(
+ "action_mlp",
+ action_mlp_eager,
+ _compiled_action_mlp,
+ (
+ hidden_states,
+ action,
+ input_weight,
+ input_bias,
+ output_weight,
+ output_bias,
+ ),
+ enabled,
+ )
+
+
+def ffn(
+ hidden_states,
+ input_weight,
+ input_bias,
+ output_weight,
+ output_bias,
+ *,
+ enabled,
+):
+ return _run_region(
+ "ffn",
+ ffn_eager,
+ _compiled_ffn,
+ (
+ hidden_states,
+ input_weight,
+ input_bias,
+ output_weight,
+ output_bias,
+ ),
+ enabled,
+ )
+
+
+def get_compiled_region_stats() -> dict:
+ return {
+ name: dict(values)
+ for name, values in _REGION_STATS.items()
+ } | {
+ "disabled_after_failure": sorted(_DISABLED_REGIONS),
+ }
diff --git a/worldplay2/models/model.py b/worldplay2/models/model.py
new file mode 100644
index 0000000..76bfe79
--- /dev/null
+++ b/worldplay2/models/model.py
@@ -0,0 +1,1907 @@
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+import math
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from einops import rearrange, repeat
+
+from diffusers.configuration_utils import ConfigMixin, register_to_config
+from diffusers.models.modeling_utils import ModelMixin
+
+from .attention import (
+ flash_attention,
+ flash_attention_dense,
+ resolve_attention_backend,
+ sage_attention,
+)
+from .kernels import (
+ action_mlp as run_compiled_action_mlp,
+ ffn as run_compiled_ffn,
+ gated_residual as run_compiled_gated_residual,
+ layer_norm_modulation as run_fused_layer_norm_modulation,
+ qk_norm_rope as run_qk_norm_rope,
+)
+from .fp8 import fp8_autocast
+from ..distributed.ops import get_rank, get_world_size
+from ..distributed.ulysses import (
+ gather_sequence,
+ output_all_to_all,
+ packed_qkv_all_to_all,
+ shard_sequence,
+)
+
+__all__ = ['WorldPlay2Model']
+
+
+def sinusoidal_embedding_1d(dim, position):
+ # preprocess
+ assert dim % 2 == 0
+ half = dim // 2
+ position = position.type(torch.float64)
+
+ # calculation
+ sinusoid = torch.outer(
+ position, torch.pow(10000, -torch.arange(half).to(position).div(half)))
+ x = torch.cat([torch.cos(sinusoid), torch.sin(sinusoid)], dim=1)
+ return x
+
+
[email protected]('cuda', enabled=False)
+def rope_params(max_seq_len, dim, theta=10000):
+ assert dim % 2 == 0
+ freqs = torch.outer(
+ torch.arange(max_seq_len),
+ 1.0 / torch.pow(theta,
+ torch.arange(0, dim, 2).to(torch.float64).div(dim)))
+ freqs = torch.polar(torch.ones_like(freqs), freqs)
+ return freqs
+
+
[email protected]('cuda', enabled=False)
+def rope_apply(x, grid_sizes, freqs):
+ n, c = x.size(2), x.size(3) // 2
+
+ # split freqs
+ freqs = freqs.split([c - 2 * (c // 3), c // 3, c // 3], dim=1)
+
+ # loop over samples
+ output = []
+ for i, (f, h, w) in enumerate(grid_sizes.tolist()):
+ seq_len = f * h * w
+
+ # precompute multipliers
+ x_i = torch.view_as_complex(x[i, :seq_len].to(torch.float64).reshape(
+ seq_len, n, -1, 2))
+ freqs_i = torch.cat([
+ freqs[0][:f].view(f, 1, 1, -1).expand(f, h, w, -1),
+ freqs[1][:h].view(1, h, 1, -1).expand(f, h, w, -1),
+ freqs[2][:w].view(1, 1, w, -1).expand(f, h, w, -1)
+ ],
+ dim=-1).reshape(seq_len, 1, -1)
+
+ # apply rotary embedding
+ x_i = torch.view_as_real(x_i * freqs_i).flatten(2)
+ x_i = torch.cat([x_i, x[i, seq_len:]])
+
+ # append to collection
+ output.append(x_i)
+ return torch.stack(output).float()
+
+
[email protected]('cuda', enabled=False)
+def rope_apply_flat(x, freqs_flat):
+ """Apply RoPE given a per-token freqs tensor.
+
+ Used by the memory-compress path where the sequence is a heterogeneous
+ concat of context/generate tokens, so a single (F, H, W) grid_sizes
+ description is no longer sufficient.
+
+ Args:
+ x (Tensor): Shape [B, L, Nh, C_head] (real).
+ freqs_flat (Tensor): complex64, shape [L, C_head / 2].
+
+ Returns:
+ Tensor: Same shape as x, with RoPE applied (float).
+ """
+ b, l, n = x.shape[:3]
+ x_c = torch.view_as_complex(x.to(torch.float64).reshape(b, l, n, -1, 2))
+ freqs_i = freqs_flat.view(1, l, 1, -1)
+ x_out = torch.view_as_real(x_c * freqs_i).flatten(2)
+ return x_out.float().view(b, l, n, -1)
+
+
+def build_freqs_flat(freqs, grid_size):
+ """Build per-token RoPE freqs for a single grid (F, H, W).
+
+ Returns a complex tensor of shape [F*H*W, C_head/2].
+ """
+ f, h, w = int(grid_size[0]), int(grid_size[1]), int(grid_size[2])
+ c = freqs.size(1)
+ freqs_split = freqs.split([c - 2 * (c // 3), c // 3, c // 3], dim=1)
+ freqs_i = torch.cat([
+ freqs_split[0][:f].view(f, 1, 1, -1).expand(f, h, w, -1),
+ freqs_split[1][:h].view(1, h, 1, -1).expand(f, h, w, -1),
+ freqs_split[2][:w].view(1, 1, w, -1).expand(f, h, w, -1)
+ ], dim=-1).reshape(f * h * w, -1)
+ return freqs_i
+
+
+class WorldPlay2RMSNorm(nn.Module):
+
+ def __init__(self, dim, eps=1e-5):
+ super().__init__()
+ self.dim = dim
+ self.eps = eps
+ self.weight = nn.Parameter(torch.ones(dim))
+
+ def forward(self, x):
+ r"""
+ Args:
+ x(Tensor): Shape [B, L, C]
+ """
+ return self._norm(x.float()).type_as(x) * self.weight
+
+ def _norm(self, x):
+ return x * torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.eps)
+
+
+class WorldPlay2LayerNorm(nn.LayerNorm):
+
+ def __init__(self, dim, eps=1e-6, elementwise_affine=False):
+ super().__init__(dim, elementwise_affine=elementwise_affine, eps=eps)
+
+ def forward(self, x):
+ r"""
+ Args:
+ x(Tensor): Shape [B, L, C]
+ """
+ return super().forward(x.float()).type_as(x)
+
+
+class WorldPlay2SelfAttention(nn.Module):
+
+ def __init__(self,
+ dim,
+ num_heads,
+ window_size=(-1, -1),
+ qk_norm=True,
+ eps=1e-6):
+ assert dim % num_heads == 0
+ super().__init__()
+ self.dim = dim
+ self.num_heads = num_heads
+ self.head_dim = dim // num_heads
+ self.window_size = window_size
+ self.qk_norm = qk_norm
+ self.eps = eps
+ self.attention_backend = 'flash'
+
+ # layers
+ self.q = nn.Linear(dim, dim)
+ self.k = nn.Linear(dim, dim)
+ self.v = nn.Linear(dim, dim)
+ self.o = nn.Linear(dim, dim)
+ self.norm_q = WorldPlay2RMSNorm(dim, eps=eps) if qk_norm else nn.Identity()
+ self.norm_k = WorldPlay2RMSNorm(dim, eps=eps) if qk_norm else nn.Identity()
+ self.compile_qk_rope = True
+ self.qk_rope_compile_failed = False
+ self.qkv = None
+
+ def fuse_qkv_projections(self) -> None:
+ """Replace the three self-attention projections with one exact GEMM."""
+ if self.qkv is not None:
+ return
+ if not all(hasattr(self, name) for name in ("q", "k", "v")):
+ raise RuntimeError("Q/K/V projections are unavailable for fusion")
+
+ projections = (self.q, self.k, self.v)
+ reference = projections[0]
+ if any(
+ projection.in_features != reference.in_features
+ or projection.out_features != reference.out_features
+ or (projection.bias is None) != (reference.bias is None)
+ for projection in projections[1:]
+ ):
+ raise RuntimeError("Q/K/V projection shapes are incompatible")
+
+ qkv = nn.Linear(
+ reference.in_features,
+ 3 * reference.out_features,
+ bias=reference.bias is not None,
+ device=reference.weight.device,
+ dtype=reference.weight.dtype,
+ )
+ with torch.no_grad():
+ qkv.weight.copy_(torch.cat(
+ [projection.weight for projection in projections], dim=0))
+ if qkv.bias is not None:
+ qkv.bias.copy_(torch.cat(
+ [projection.bias for projection in projections], dim=0))
+
+ self.qkv = qkv
+ # Do not retain duplicate parameters: fusion must not increase FSDP
+ # storage or checkpoint memory after installation.
+ del self.q
+ del self.k
+ del self.v
+
+ def forward(self, x, seq_lens, grid_sizes, freqs, freqs_flat=None,
+ ar=False, cmp_token_count=0, cmp_chunk_token_count=0,
+ tmp_token_count=0,
+ kv_cache_entry=None, cache_write=False,
+ sink_token_count=0, cache_update=None):
+ r"""
+ Args:
+ x(Tensor): Shape [B, L, num_heads, C / num_heads]
+ seq_lens(Tensor): Shape [B]
+ grid_sizes(Tensor): Shape [B, 3], the second dimension contains (F, H, W).
+ Ignored when ``freqs_flat`` is provided.
+ freqs(Tensor): Rope freqs, shape [1024, C / num_heads / 2]
+ freqs_flat(Tensor, optional): Per-token complex RoPE tensor of shape
+ ``[L, C / num_heads / 2]``. When provided, RoPE is applied via
+ ``rope_apply_flat`` (used by the memory-compress path where
+ the sequence is a heterogeneous concat of context/generate
+ tokens). When ``None``, falls back to the original grid-based
+ ``rope_apply``.
+ ar(bool): when True, run the chunk-AR (KV-cache) self-attention.
+ Cache behavior is selected by ``cache_write``:
+ * prefill (cache_write=True): ``x`` = ctx tokens only
+ (``[cmp | tmp]``). A chunk-causal mask is applied (cmp
+ chunk-causal to cmp; tmp -> all cmp+tmp; no pred rows).
+ The post-RoPE ctx K/V are OVERWRITE-stored into
+ ``kv_cache_entry['k_vision'/'v_vision']``. Returns the
+ ctx output (threaded layer-to-layer by the caller).
+ * cache-hit (cache_write=False): ``x`` = pred tokens only.
+ K/V = ``[cached_ctx | pred]`` (no mask -- pred sees all),
+ via the selected backend. ``cached_*`` may be None on the
+ first chunk (no compressor ctx) -> pred-only attention.
+ kv_cache_entry(dict, optional): per-layer ``{'k_vision','v_vision'}``
+ slot (see ``WorldPlay2Model.init_kv_cache``).
+ cache_write(bool): write/update context cache vs read it for denoise.
+ """
+ b, s, n, d = *x.shape[:2], self.num_heads, self.head_dim
+
+ if self.qkv is not None:
+ q_projection, k_projection, v_projection = self.qkv(x).chunk(
+ 3, dim=-1)
+ else:
+ q_projection = self.q(x)
+ k_projection = self.k(x)
+ v_projection = self.v(x)
+ v = v_projection.view(b, s, n, d)
+
+ if freqs_flat is not None:
+ if self.qk_norm:
+ q, k, compiled = run_qk_norm_rope(
+ q_projection,
+ k_projection,
+ self.norm_q.weight,
+ self.norm_k.weight,
+ freqs_flat,
+ eps=self.eps,
+ num_heads=self.num_heads,
+ head_dim=self.head_dim,
+ compile_enabled=(
+ self.compile_qk_rope
+ and not self.qk_rope_compile_failed
+ ),
+ )
+ if self.compile_qk_rope and not compiled:
+ self.qk_rope_compile_failed = True
+ else:
+ q = q_projection.view(b, s, n, d)
+ k = k_projection.view(b, s, n, d)
+ q = rope_apply_flat(q, freqs_flat)
+ k = rope_apply_flat(k, freqs_flat)
+ else:
+ q = self.norm_q(q_projection).view(b, s, n, d)
+ k = self.norm_k(k_projection).view(b, s, n, d)
+ q = rope_apply(q, grid_sizes, freqs)
+ k = rope_apply(k, grid_sizes, freqs)
+
+ # RoPE is evaluated in FP32 for stability, but attention and packed
+ # QKV communication use the value/projection dtype (BF16 in inference).
+ # This also prevents torch.stack in packed QKV A2A from promoting V
+ # and tripling its communication precision.
+ q = q.to(v.dtype)
+ k = k.to(v.dtype)
+
+ use_sp = getattr(self, "use_sp", False)
+ if use_sp:
+ if not torch.all(seq_lens == seq_lens[0]):
+ raise ValueError(
+ "sequence parallel requires equal sequence lengths "
+ "within the batch")
+ global_sequence = int(seq_lens[0].item())
+ q, k, v = packed_qkv_all_to_all(
+ q, k, v, global_sequence=global_sequence)
+ else:
+ global_sequence = q.shape[1]
+
+ def finish_attention(out):
+ if use_sp:
+ out = output_all_to_all(
+ out,
+ global_sequence=global_sequence,
+ num_heads=self.num_heads,
+ )
+ return self.o(out.flatten(2))
+
+ if not ar:
+ attention_fn = (
+ sage_attention
+ if self.attention_backend == 'sage' else flash_attention
+ )
+ x = attention_fn(
+ q=q, k=k, v=v, k_lens=seq_lens,
+ window_size=self.window_size)
+ return finish_attention(x)
+
+ assert kv_cache_entry is not None, (
+ "self_attn(ar=True) requires kv_cache_entry "
+ "(use WorldPlay2Model.init_kv_cache()).")
+
+ cache_initialized = kv_cache_entry.get('k_vision') is not None
+ if not cache_write:
+ # Case 1: regular AR denoise. Read the persistent context cache;
+ # an empty cache on the very first generated chunk is valid.
+ out = self._attend_with_ar_cache(q, k, v, kv_cache_entry)
+ elif not cache_initialized:
+ # Case 2: first history prefill. Bootstrap the cache as
+ # [sink | cmp | tmp], including the sink-specific attention mask.
+ out = self._initialize_ar_cache(
+ q=q,
+ k=k,
+ v=v,
+ kv_cache_entry=kv_cache_entry,
+ sink_token_count=sink_token_count,
+ cmp_token_count=cmp_token_count,
+ cmp_chunk_token_count=cmp_chunk_token_count,
+ tmp_token_count=tmp_token_count,
+ )
+ else:
+ # Case 3: later history prefill. Keep [sink | old_cmp], discard
+ # the previous tmp, then append [new_cmp | new_tmp].
+ if cache_update is None:
+ raise ValueError("AR cache update metadata is required")
+ out = self._update_ar_cache(
+ q, k, v, kv_cache_entry, cache_update,
+ cmp_chunk_token_count)
+ return finish_attention(out)
+
+ def _initialize_ar_cache(
+ self,
+ q,
+ k,
+ v,
+ kv_cache_entry,
+ sink_token_count,
+ cmp_token_count,
+ cmp_chunk_token_count,
+ tmp_token_count,
+ ):
+ """Bootstrap ``[sink | cmp | tmp]`` on the first history prefill."""
+ sequence_length = q.shape[1]
+ expected_length = (
+ sink_token_count + cmp_token_count + tmp_token_count
+ )
+ if sequence_length != expected_length:
+ raise ValueError(
+ f"initial AR cache expects {expected_length} tokens, "
+ f"got {sequence_length}"
+ )
+
+ q_attn = q.transpose(1, 2)
+ k_attn = k.transpose(1, 2)
+ v_attn = v.transpose(1, 2)
+ if sequence_length == 0:
+ out = torch.zeros_like(q_attn)
+ elif cmp_chunk_token_count <= 0:
+ out = F.scaled_dot_product_attention(
+ q_attn, k_attn, v_attn, is_causal=False)
+ else:
+ mask = torch.zeros(
+ sequence_length,
+ sequence_length,
+ dtype=torch.bool,
+ device=q.device,
+ )
+ if sink_token_count:
+ mask[:sink_token_count, :sink_token_count] = True
+ for chunk_start in range(
+ 0, cmp_token_count, cmp_chunk_token_count
+ ):
+ query_start = sink_token_count + chunk_start
+ query_end = min(
+ query_start + cmp_chunk_token_count,
+ sink_token_count + cmp_token_count,
+ )
+ mask[query_start:query_end, :query_end] = True
+ tmp_start = sink_token_count + cmp_token_count
+ if tmp_token_count:
+ mask[tmp_start:expected_length, :expected_length] = True
+ out = F.scaled_dot_product_attention(
+ q_attn,
+ k_attn,
+ v_attn,
+ attn_mask=mask.unsqueeze(0).unsqueeze(0),
+ is_causal=False,
+ )
+
+ kv_cache_entry['k_vision'] = k.detach()
+ kv_cache_entry['v_vision'] = v.detach()
+ return out.transpose(1, 2)
+
+ def _update_ar_cache(self, q, k, v, kv_cache_entry, cache_update,
+ cmp_chunk_token_count):
+ r"""Replace old tmp and append ``[new_cmp | new_tmp]``.
+
+ ``q/k/v`` contain only ``[new_cmp | new_tmp]``. The persistent prefix
+ ``[sink | old_cmp]`` is sliced directly from ``k_vision/v_vision``;
+ the previous tmp suffix is discarded. After attention, the cache is
+ replaced by ``[cached_prefix | new_cmp | new_tmp]``.
+ """
+ n_new_cmp = int(cache_update['n_new_cmp'])
+ n_tmp = int(cache_update['n_tmp'])
+ cached_prefix_tokens = int(cache_update['cached_prefix_tokens'])
+ Sq = q.shape[1]
+ assert Sq == n_new_cmp + n_tmp, (
+ f"ctx prefill expects q = [new_cmp | tmp] "
+ f"({n_new_cmp}+{n_tmp}), got {Sq}")
+
+ cached_k = kv_cache_entry.get('k_vision', None)
+ cached_v = kv_cache_entry.get('v_vision', None)
+ assert cached_k is not None and cached_v is not None, (
+ "Incremental prefill requires an existing k_vision/v_vision.")
+ assert cached_prefix_tokens <= cached_k.shape[1], (
+ f"cached_prefix_tokens ({cached_prefix_tokens}) exceeds cached "
+ f"length ({cached_k.shape[1]}).")
+ pre_k = cached_k[:, :cached_prefix_tokens].to(k.device).to(k.dtype)
+ pre_v = cached_v[:, :cached_prefix_tokens].to(v.device).to(v.dtype)
+ L_pre = cached_prefix_tokens
+ k_full = torch.cat([pre_k, k], dim=1)
+ v_full = torch.cat([pre_v, v], dim=1)
+ L_tot = k_full.shape[1] # = L_pre + n_new_cmp + n_tmp
+
+ # Column index landmarks within k_full = [sink | old_cmp | new_cmp | tmp]
+ cmp_new_c0 = L_pre # first new_cmp column
+ cmp_new_c1 = L_pre + n_new_cmp # end of new_cmp / start of tmp
+ tmp_c1 = L_tot # end of tmp
+
+ # Build mask over QUERY rows (Sq) x all columns (L_tot).
+ attn_mask = torch.zeros(Sq, L_tot, dtype=torch.bool, device=q.device)
+ if n_new_cmp > 0:
+ # new_cmp is chunk-causal *within itself* and sees all prefix
+ # (sink + old_cmp), but NOT tmp. Split new_cmp into cmp-chunks.
+ ccc = cmp_chunk_token_count if cmp_chunk_token_count > 0 else n_new_cmp
+ for rs in range(0, n_new_cmp, ccc):
+ re = min(rs + ccc, n_new_cmp)
+ # columns: everything up to and including this cmp sub-chunk
+ ke = cmp_new_c0 + re
+ ke = min(ke, cmp_new_c1)
+ attn_mask[rs:re, :ke] = True
+ if n_tmp > 0:
+ # tmp rows (after new_cmp in q) see EVERYTHING (sink+cmp+tmp).
+ attn_mask[n_new_cmp:Sq, :tmp_c1] = True
+ attn_mask = attn_mask.unsqueeze(0).unsqueeze(0)
+
+ out = F.scaled_dot_product_attention(
+ q.transpose(1, 2), k_full.transpose(1, 2), v_full.transpose(1, 2),
+ attn_mask=attn_mask, is_causal=False)
+ out = out.transpose(1, 2) # [B, Sq, H, D]
+
+ kv_cache_entry['k_vision'] = torch.cat(
+ [pre_k, k], dim=1).detach()
+ kv_cache_entry['v_vision'] = torch.cat(
+ [pre_v, v], dim=1).detach()
+ return out
+
+ def _attend_with_ar_cache(self, q, k, v, kv_cache_entry):
+ """Attend over ``[cached ctx | pred]`` with the selected backend."""
+ cached_k = kv_cache_entry.get('k_vision')
+ cached_v = kv_cache_entry.get('v_vision')
+ if cached_k is not None:
+ k = torch.cat([cached_k.to(k.device, k.dtype), k], dim=1)
+ v = torch.cat([cached_v.to(v.device, v.dtype), v], dim=1)
+ if self.attention_backend == 'sage':
+ return sage_attention(q, k, v, causal=False)
+ return flash_attention_dense(q, k, v, causal=False)
+
+
+
+class WorldPlay2CrossAttention(WorldPlay2SelfAttention):
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.conditioning_cache_enabled = True
+ # Keyed by the context tensor itself: torch.Tensor hashes and compares
+ # by identity, so the key doubles as the strong reference that keeps a
+ # stale entry from ever matching a different tensor.
+ self._context_kv_cache: dict[torch.Tensor, tuple] = {}
+
+ def clear_conditioning_cache(self) -> None:
+ self._context_kv_cache.clear()
+
+ def _get_context_kv(self, context):
+ if not self.conditioning_cache_enabled:
+ return self.norm_k(self.k(context)), self.v(context)
+
+ entry = self._context_kv_cache.get(context)
+ if entry is not None:
+ return entry
+
+ key_tensor = self.norm_k(self.k(context)).detach()
+ value_tensor = self.v(context).detach()
+ self._context_kv_cache[context] = (key_tensor, value_tensor)
+ return key_tensor, value_tensor
+
+ def forward(self, x, context, context_lens):
+ r"""
+ Args:
+ x(Tensor): Shape [B, L1, C]
+ context(Tensor): Shape [B, L2, C]
+ context_lens(Tensor): Shape [B]
+ """
+ b, n, d = x.size(0), self.num_heads, self.head_dim
+
+ # compute query, key, value
+ q = self.norm_q(self.q(x)).view(b, -1, n, d)
+ k, v = self._get_context_kv(context)
+ k = k.view(b, -1, n, d)
+ v = v.view(b, -1, n, d)
+
+ # compute attention
+ x = flash_attention(q, k, v, k_lens=context_lens)
+
+ # output
+ x = x.flatten(2)
+ x = self.o(x)
+ return x
+
+
+class WorldPlay2AttentionBlock(nn.Module):
+
+ def __init__(self,
+ dim,
+ ffn_dim,
+ num_heads,
+ window_size=(-1, -1),
+ qk_norm=True,
+ cross_attn_norm=False,
+ eps=1e-6):
+ super().__init__()
+ self.dim = dim
+ self.ffn_dim = ffn_dim
+ self.num_heads = num_heads
+ self.window_size = window_size
+ self.qk_norm = qk_norm
+ self.cross_attn_norm = cross_attn_norm
+ self.eps = eps
+
+ # layers
+ self.norm1 = WorldPlay2LayerNorm(dim, eps)
+ self.self_attn = WorldPlay2SelfAttention(dim, num_heads, window_size, qk_norm,
+ eps)
+ self.norm3 = WorldPlay2LayerNorm(
+ dim, eps,
+ elementwise_affine=True) if cross_attn_norm else nn.Identity()
+ self.cross_attn = WorldPlay2CrossAttention(dim, num_heads, (-1, -1), qk_norm,
+ eps)
+ self.norm2 = WorldPlay2LayerNorm(dim, eps)
+ self.ffn = nn.Sequential(
+ nn.Linear(dim, ffn_dim), nn.GELU(approximate='tanh'),
+ nn.Linear(ffn_dim, dim))
+
+ # modulation
+ self.modulation = nn.Parameter(torch.randn(1, 6, dim) / dim**0.5)
+ self.compile_block_glue = True
+ self.compile_ffn = True
+
+ def forward(
+ self,
+ x,
+ e,
+ seq_lens,
+ grid_sizes,
+ freqs,
+ context,
+ context_lens,
+ vec_action,
+ freqs_flat=None,
+ ar=False,
+ cmp_token_count=0,
+ cmp_chunk_token_count=0,
+ tmp_token_count=0,
+ kv_cache_entry=None,
+ cache_write=False,
+ sink_token_count=0,
+ cache_update=None,
+ ):
+ r"""
+ Args:
+ x(Tensor): Shape [B, L, C]
+ e(Tensor): Shape [B, L1, 6, C]
+ seq_lens(Tensor): Shape [B], length of each sequence in batch
+ grid_sizes(Tensor): Shape [B, 3], the second dimension contains (F, H, W)
+ freqs(Tensor): Rope freqs, shape [1024, C / num_heads / 2]
+ freqs_flat(Tensor, optional): Per-token RoPE freqs for the
+ memory-compress path; forwarded to ``self_attn``.
+ ar(bool): when True, run the chunk-AR (KV-cache) self-attention.
+ kv_cache_entry/cache_write: forwarded to
+ ``self_attn`` for the chunk-AR prefill / cache-hit paths.
+ cache_update(dict, optional): metadata for replacing the old tmp
+ suffix and appending the new cmp/tmp tokens.
+ """
+ assert e.dtype == torch.float32
+ with torch.amp.autocast('cuda', dtype=torch.float32):
+ e = (self.modulation.unsqueeze(0) + e).chunk(6, dim=2)
+ assert e[0].dtype == torch.float32
+
+ # self-attention (BI when ar=False; chunk-AR KV-cache when ar=True)
+ attn_in, _ = run_fused_layer_norm_modulation(
+ x,
+ e[1].squeeze(2),
+ e[0].squeeze(2),
+ eps=self.eps,
+ enabled=self.compile_block_glue,
+ )
+ y = self.self_attn(
+ attn_in, seq_lens, grid_sizes, freqs, freqs_flat=freqs_flat,
+ ar=ar,
+ cmp_token_count=cmp_token_count,
+ cmp_chunk_token_count=cmp_chunk_token_count,
+ tmp_token_count=tmp_token_count,
+ kv_cache_entry=kv_cache_entry,
+ cache_write=cache_write,
+ sink_token_count=sink_token_count,
+ cache_update=cache_update)
+ x, _ = run_compiled_gated_residual(
+ x,
+ y,
+ e[2].squeeze(2),
+ enabled=self.compile_block_glue,
+ )
+
+ # cross-attention & ffn function
+ def cross_attn_ffn(x, context, context_lens, e):
+ cross_input = self.norm3(x)
+ cross_output = self.cross_attn(
+ cross_input, context, context_lens)
+ x = x + cross_output
+
+ # mlp inject action instead of attention
+ if vec_action is not None:
+ x, _ = run_compiled_action_mlp(
+ x,
+ vec_action,
+ self.img_action_mlp[0].weight,
+ self.img_action_mlp[0].bias,
+ self.img_action_mlp[2].weight,
+ self.img_action_mlp[2].bias,
+ enabled=self.compile_block_glue,
+ )
+
+ ffn_in, _ = run_fused_layer_norm_modulation(
+ x,
+ e[4].squeeze(2),
+ e[3].squeeze(2),
+ eps=self.eps,
+ enabled=self.compile_block_glue,
+ )
+ if getattr(self.ffn, "uses_fp8", False):
+ y = self.ffn(ffn_in.to(x.dtype))
+ else:
+ y, _ = run_compiled_ffn(
+ ffn_in,
+ self.ffn[0].weight,
+ self.ffn[0].bias,
+ self.ffn[2].weight,
+ self.ffn[2].bias,
+ enabled=self.compile_ffn,
+ )
+ x, _ = run_compiled_gated_residual(
+ x,
+ y,
+ e[5].squeeze(2),
+ enabled=self.compile_block_glue,
+ )
+ return x
+
+ x = cross_attn_ffn(x, context, context_lens, e)
+ return x
+
+
+class Head(nn.Module):
+
+ def __init__(self, dim, out_dim, patch_size, eps=1e-6):
+ super().__init__()
+ self.dim = dim
+ self.out_dim = out_dim
+ self.patch_size = patch_size
+ self.eps = eps
+
+ # layers
+ out_dim = math.prod(patch_size) * out_dim
+ self.norm = WorldPlay2LayerNorm(dim, eps)
+ self.head = nn.Linear(dim, out_dim)
+
+ # modulation
+ self.modulation = nn.Parameter(torch.randn(1, 2, dim) / dim**0.5)
+ self._active_block_index = None
+ self._last_endpoint_output = None
+
+ def make_fixed_pdd(self, num_blocks: int = 2) -> None:
+ num_blocks = int(num_blocks)
+ if num_blocks <= 0:
+ raise ValueError("fixed PDD block count must be positive")
+ if hasattr(self, "block_weight"):
+ if self._num_fixed_pdd_blocks != num_blocks:
+ raise ValueError("fixed PDD block count mismatch")
+ return
+ self._num_fixed_pdd_blocks = num_blocks
+ output_dim = self.head.out_features
+ self.block_weight = nn.Parameter(torch.empty(
+ num_blocks,
+ 2 * output_dim,
+ self.dim,
+ device=self.head.weight.device,
+ dtype=self.head.weight.dtype,
+ ))
+ self.block_bias = nn.Parameter(torch.empty(
+ num_blocks,
+ 2 * output_dim,
+ device=self.head.bias.device,
+ dtype=self.head.bias.dtype,
+ ))
+ del self.head
+
+ def set_active_block(self, block_index: int) -> None:
+ if not hasattr(self, "block_weight"):
+ raise RuntimeError("set_active_block requires a fixed PDD head")
+ block_index = int(block_index)
+ if not 0 <= block_index < self._num_fixed_pdd_blocks:
+ raise IndexError(
+ f"fixed PDD block index {block_index} is out of range "
+ f"[0, {self._num_fixed_pdd_blocks})"
+ )
+ self._active_block_index = block_index
+
+ def pop_endpoint_output(self):
+ output = self._last_endpoint_output
+ self._last_endpoint_output = None
+ return output
+
+ def forward(self, x, e):
+ r"""
+ Args:
+ x(Tensor): Shape [B, L1, C]
+ e(Tensor): Shape [B, L1, C]
+ """
+ assert e.dtype == torch.float32
+ with torch.amp.autocast('cuda', dtype=torch.float32):
+ shift, scale = (
+ self.modulation.unsqueeze(0) + e.unsqueeze(2)
+ ).chunk(2, dim=2)
+ hidden = (
+ self.norm(x) * (1 + scale.squeeze(2)) + shift.squeeze(2)
+ )
+ if hasattr(self, "block_weight"):
+ if self._active_block_index is None:
+ raise RuntimeError(
+ "fixed PDD inference block was not selected"
+ )
+ projected = F.linear(
+ hidden,
+ self.block_weight[self._active_block_index],
+ self.block_bias[self._active_block_index],
+ )
+ displacement, endpoint = projected.chunk(2, dim=-1)
+ self._last_endpoint_output = endpoint
+ return displacement
+ self._last_endpoint_output = None
+ return self.head(hidden)
+
+
+class WorldPlay2Model(ModelMixin, ConfigMixin):
+ r"""
+ WorldPlay2 diffusion backbone based on Wan2.2.
+ """
+
+ ignore_for_config = [
+ 'patch_size', 'cross_attn_norm', 'qk_norm', 'text_dim', 'window_size'
+ ]
+ _no_split_modules = ['WorldPlay2AttentionBlock']
+
+ @register_to_config
+ def __init__(self,
+ model_type='t2v',
+ patch_size=(1, 2, 2),
+ text_len=512,
+ in_dim=16,
+ dim=2048,
+ ffn_dim=8192,
+ freq_dim=256,
+ text_dim=4096,
+ out_dim=16,
+ num_heads=16,
+ num_layers=32,
+ window_size=(-1, -1),
+ qk_norm=True,
+ cross_attn_norm=True,
+ eps=1e-6):
+ r"""
+ Initialize the diffusion model backbone.
+
+ Args:
+ model_type (`str`, *optional*, defaults to 't2v'):
+ Model variant - 't2v' (text-to-video) or 'i2v' (image-to-video)
+ patch_size (`tuple`, *optional*, defaults to (1, 2, 2)):
+ 3D patch dimensions for video embedding (t_patch, h_patch, w_patch)
+ text_len (`int`, *optional*, defaults to 512):
+ Fixed length for text embeddings
+ in_dim (`int`, *optional*, defaults to 16):
+ Input video channels (C_in)
+ dim (`int`, *optional*, defaults to 2048):
+ Hidden dimension of the transformer
+ ffn_dim (`int`, *optional*, defaults to 8192):
+ Intermediate dimension in feed-forward network
+ freq_dim (`int`, *optional*, defaults to 256):
+ Dimension for sinusoidal time embeddings
+ text_dim (`int`, *optional*, defaults to 4096):
+ Input dimension for text embeddings
+ out_dim (`int`, *optional*, defaults to 16):
+ Output video channels (C_out)
+ num_heads (`int`, *optional*, defaults to 16):
+ Number of attention heads
+ num_layers (`int`, *optional*, defaults to 32):
+ Number of transformer blocks
+ window_size (`tuple`, *optional*, defaults to (-1, -1)):
+ Window size for local attention (-1 indicates global attention)
+ qk_norm (`bool`, *optional*, defaults to True):
+ Enable query/key normalization
+ cross_attn_norm (`bool`, *optional*, defaults to False):
+ Enable cross-attention normalization
+ eps (`float`, *optional*, defaults to 1e-6):
+ Epsilon value for normalization layers
+ """
+
+ super().__init__()
+
+ assert model_type in ['t2v', 'i2v', 'ti2v', 's2v']
+ self.model_type = model_type
+
+ self.patch_size = patch_size
+ self.text_len = text_len
+ self.in_dim = in_dim
+ self.dim = dim
+ self.ffn_dim = ffn_dim
+ self.freq_dim = freq_dim
+ self.text_dim = text_dim
+ self.out_dim = out_dim
+ self.num_heads = num_heads
+ self.num_layers = num_layers
+ self.window_size = window_size
+ self.qk_norm = qk_norm
+ self.cross_attn_norm = cross_attn_norm
+ self.eps = eps
+
+ # embeddings
+ self.patch_embedding = nn.Conv3d(
+ in_dim, dim, kernel_size=patch_size, stride=patch_size)
+ self.text_embedding = nn.Sequential(
+ nn.Linear(text_dim, dim), nn.GELU(approximate='tanh'),
+ nn.Linear(dim, dim))
+
+ self.time_embedding = nn.Sequential(
+ nn.Linear(freq_dim, dim), nn.SiLU(), nn.Linear(dim, dim))
+ self.time_projection = nn.Sequential(nn.SiLU(), nn.Linear(dim, dim * 6))
+
+ # blocks
+ self.blocks = nn.ModuleList([
+ WorldPlay2AttentionBlock(dim, ffn_dim, num_heads, window_size, qk_norm,
+ cross_attn_norm, eps) for _ in range(num_layers)
+ ])
+
+ # head
+ self.head = Head(dim, out_dim, patch_size, eps)
+
+ # buffers (don't use register_buffer otherwise dtype will be changed in to())
+ assert (dim % num_heads) == 0 and (dim // num_heads) % 2 == 0
+ d = dim // num_heads
+ self.freqs = torch.cat([
+ rope_params(1024, d - 4 * (d // 6)),
+ rope_params(1024, 2 * (d // 6)),
+ rope_params(1024, 2 * (d // 6))
+ ],
+ dim=1)
+
+ # initialize weights
+ self.init_weights()
+
+ self.use_sp = False
+ self.attention_backend = 'flash'
+ self.conditioning_cache_enabled = True
+ self._text_projection_cache: dict[tuple, torch.Tensor] = {}
+ self.self_qkv_fused = False
+ self.fp8_enabled = False
+ self.fp8_ffn_enabled = False
+ self.fp8_qkv_enabled = False
+ self.fp8_recipe = None
+ self.fp8_block_range = None
+ self.fp8_qkv_block_range = None
+
+ def add_embedding_action_parameters(self):
+ """Continuous (first 2 dims) + per-dim discrete embeddings (last 4).
+
+ Action layout (last dim = 6):
+ [:, 0:2] continuous → ``action_in_cont`` MLP → (N, dim)
+ [:, 2] discrete in {-1, 0, 1} → Embedding(3, dim) (idx +1)
+ [:, 3] discrete in {-1, 0, 1} → Embedding(3, dim) (idx +1)
+ [:, 4] discrete in {0, 1} → Embedding(2, dim)
+ [:, 5] discrete in {0, 1} → Embedding(2, dim)
+ Final ``vec_action = cont + Σ discrete`` so the output shape and
+ downstream usage stay identical to the MLP path.
+ """
+ # continuous branch: (N, 2) -> (N, dim)
+ self.action_in_cont = nn.Sequential(
+ nn.Linear(2, 256),
+ nn.SiLU(),
+ nn.Linear(256, 1024),
+ nn.SiLU(),
+ nn.Linear(1024, self.dim),
+ )
+ # one independent embedding per discrete dim
+ self.action_in_disc = nn.ModuleList([
+ nn.Embedding(3, self.dim), # dim2: {-1,0,1} (+1 offset → {0,1,2})
+ nn.Embedding(3, self.dim), # dim3: {-1,0,1} (+1 offset → {0,1,2})
+ nn.Embedding(2, self.dim), # dim4: {0,1}
+ nn.Embedding(2, self.dim), # dim5: {0,1}
+ ])
+ # Match the per-dim contribution of the continuous branch so all 6
+ # action dims start training with comparable signal strength.
+ #
+ # The continuous branch's last layer (Linear(1024, dim)) under Xavier
+ # has weight std ≈ sqrt(2/(fan_in+fan_out)); its output std ≈ that
+ # value, spread over 2 input dims, so each continuous dim contributes
+ # std ≈ cont_std / sqrt(2). Initialising the discrete embeddings to
+ # the same per-dim std keeps the 6 dims roughly equal-strength.
+ #
+ # Doing this with explicit Xavier on the cont MLP (rather than relying
+ # on Linear's default Kaiming) also keeps it consistent with how
+ # init_weights() initialises every other Linear in this model.
+ for m in self.action_in_cont.modules():
+ if isinstance(m, nn.Linear):
+ nn.init.xavier_uniform_(m.weight)
+ if m.bias is not None:
+ nn.init.zeros_(m.bias)
+ cont_last = self.action_in_cont[-1] # Linear(1024, dim)
+ cont_std = math.sqrt(
+ 2.0 / (cont_last.in_features + cont_last.out_features))
+ per_dim_std = cont_std / math.sqrt(2)
+ for emb in self.action_in_disc:
+ nn.init.normal_(emb.weight, std=per_dim_std)
+
+ # img_action_mlp identical to the MLP path (zero-init last layer)
+ for block in self.blocks:
+ block.img_action_mlp = nn.Sequential(
+ nn.Linear(block.dim + block.dim, block.dim),
+ nn.SiLU(),
+ nn.Linear(block.dim, block.dim),
+ )
+ nn.init.zeros_(block.img_action_mlp[-1].weight)
+ if block.img_action_mlp[-1].bias is not None:
+ nn.init.zeros_(block.img_action_mlp[-1].bias)
+
+ def _encode_action(self, action: torch.Tensor) -> torch.Tensor:
+ """Encode ``[pitch,yaw,forward,lateral,perspective,space]``."""
+ a = action.reshape(-1, 6)
+ cont = self.action_in_cont(a[:, :2])
+ # discrete indices; +1 offset for the {-1,0,1} dims, clamp as a
+ # cheap guard against stray out-of-range values.
+ idx2 = (a[:, 2].long() + 1).clamp_(0, 2)
+ idx3 = (a[:, 3].long() + 1).clamp_(0, 2)
+ idx4 = a[:, 4].long().clamp_(0, 1)
+ idx5 = a[:, 5].long().clamp_(0, 1)
+ disc = (self.action_in_disc[0](idx2)
+ + self.action_in_disc[1](idx3)
+ + self.action_in_disc[2](idx4)
+ + self.action_in_disc[3](idx5))
+ return cont + disc
+
+ def add_memory_compress_model(self):
+ """Attach the hierarchical HR -> compressed-memory feature extractor
+ used by the stage-2 / stage-3 chunked inference recipe.
+
+ Wan2.2 5B contract (lr is 1/2 in T and 1/4 in H/W):
+ - hr_latent : [B, 16, T_hr, H_hr, W_hr]
+ - lr_latent : [B, 16, T_hr/2, H_hr/4, W_hr/4]
+ - y : 20ch (4 mask + 16 first-frame image latent)
+ - compressor input : 36 = 16 + 20 channels at hr resolution.
+ - compressor output : spatial_down=[1,1,1,0,0,0] + temporal_down=[1,0,...]
+ reduces hr ``[T_hr, H_hr, W_hr]`` -> ``[T_hr/2, H_hr/8, W_hr/8]``.
+ ``patch_embedding(lr)`` with patch_size=(1,2,2) also produces
+ ``[T_hr/2, H_hr/8, W_hr/8]``, so ``lr_tokens + hr_tokens`` lines up.
+
+ Only the causal-in-time compressor is supported.
+ """
+ from .causal_compressor import CausalMemCompressModel
+ self.memory_compress = CausalMemCompressModel(
+ output_dim=self.dim,
+ input_dim=self.in_dim,
+ spatial_down=[1, 1, 1, 0, 0, 0],
+ temporal_down=[1, 0, 0, 0, 0, 0],
+ )
+
+ def set_sequence_parallel(self, enabled: bool) -> None:
+ """Select SP/non-SP paths without runtime monkey-patching."""
+ self.use_sp = bool(enabled)
+ for block in self.blocks:
+ block.self_attn.use_sp = self.use_sp
+
+ def set_compiled_qk_rope(self, enabled: bool) -> None:
+ for block in self.blocks:
+ block.self_attn.compile_qk_rope = bool(enabled)
+ block.self_attn.qk_rope_compile_failed = False
+
+ def set_self_attention_backend(self, backend: str) -> None:
+ """Select visual denoising attention; preserve masked prefill and text."""
+ backend = resolve_attention_backend(backend)
+ if backend == 'sage' and any(
+ tuple(block.self_attn.window_size) != (-1, -1)
+ for block in self.blocks
+ ):
+ raise NotImplementedError(
+ "SageAttention requires window_size=(-1, -1); "
+ "use attention_backend='flash' for sliding-window attention."
+ )
+ self.attention_backend = backend
+ for block in self.blocks:
+ block.self_attn.attention_backend = backend
+
+ def set_compiled_regions(self, *, block_glue: bool, ffn: bool) -> None:
+ for block in self.blocks:
+ block.compile_block_glue = bool(block_glue)
+ block.compile_ffn = bool(ffn)
+
+ def fuse_self_attention_qkv(self) -> None:
+ """Fuse Q/K/V for every visual self-attention block."""
+ if self.self_qkv_fused:
+ return
+ for block in self.blocks:
+ block.self_attn.fuse_qkv_projections()
+ self.self_qkv_fused = True
+
+ def set_conditioning_cache(self, enabled: bool) -> None:
+ self.conditioning_cache_enabled = bool(enabled)
+ for block in self.blocks:
+ block.cross_attn.conditioning_cache_enabled = bool(enabled)
+ if not enabled:
+ self.clear_conditioning_cache()
+
+ def clear_conditioning_cache(self) -> None:
+ self._text_projection_cache.clear()
+ for block in self.blocks:
+ block.cross_attn.clear_conditioning_cache()
+
+ def _project_text_context(self, context):
+ # Keyed by the source tensors themselves: torch.Tensor hashes and
+ # compares by identity, so the key both distinguishes cond/uncond and
+ # keeps a strong reference to the tensors it was derived from. Model
+ # movement/offload explicitly clears this cache.
+ key = tuple(context)
+ if self.conditioning_cache_enabled:
+ cached = self._text_projection_cache.get(key)
+ if cached is not None:
+ return cached
+
+ padded = torch.stack([
+ torch.cat([
+ tensor,
+ tensor.new_zeros(
+ self.text_len - tensor.size(0), tensor.size(1)),
+ ])
+ for tensor in key
+ ])
+ projected = self.text_embedding(padded).detach()
+ if self.conditioning_cache_enabled:
+ self._text_projection_cache[key] = projected
+ return projected
+
+ def _forward_compress(
+ self,
+ hr_latents_input,
+ lr_latents_input,
+ lr_action,
+ memory_t,
+ hr_temporal_latents_input=None,
+ hr_temporal_action=None,
+ sink_action=None,
+ ):
+ """Build ``[optional sink | compressed history | temporal anchor]``."""
+ if hr_temporal_latents_input is None or hr_temporal_action is None:
+ raise ValueError("Stage-Three compressor requires a temporal anchor")
+
+ bs = hr_latents_input.shape[0]
+ hr_pf = hr_latents_input.shape[2] // self.patch_size[0]
+ device = hr_latents_input.device
+ if self.freqs.device != device:
+ self.freqs = self.freqs.to(device)
+
+ memory_e = self.time_embedding(
+ sinusoidal_embedding_1d(
+ self.freq_dim,
+ memory_t.reshape(-1).float(),
+ ).float()
+ )
+
+ # Compressed LR+HR history at the LR temporal cadence.
+ _, _, lr_f, lr_h, lr_w = lr_latents_input.shape
+ cmp_pf = lr_f // self.patch_size[0]
+ cmp_ph = lr_h // self.patch_size[1]
+ cmp_pw = lr_w // self.patch_size[2]
+ cmp_spatial = cmp_ph * cmp_pw
+ cmp_tokens = (
+ self.patch_embedding(lr_latents_input).flatten(2).transpose(1, 2)
+ + self.memory_compress(hr_latents_input)
+ )
+ cmp_freqs = build_freqs_flat(self.freqs,(hr_pf, cmp_ph, cmp_pw)).view(hr_pf, cmp_spatial, -1)
+ cmp_freqs = cmp_freqs[::2][:cmp_pf].reshape(cmp_pf * cmp_spatial, -1)
+ cmp_e = repeat(
+ memory_e,
+ "1 D -> B (T HW) D",
+ B=bs,
+ T=cmp_pf,
+ HW=cmp_spatial,
+ )
+ cmp_e0 = self.time_projection(cmp_e).unflatten(2, (6, self.dim))
+ cmp_action = repeat(
+ self._encode_action(lr_action),
+ "(B T) D -> B (T HW) D",
+ B=bs,
+ HW=cmp_spatial,
+ )
+
+ # Temporal anchor occupies the final HR temporal positions.
+ _, _, tmp_f, tmp_h, tmp_w = hr_temporal_latents_input.shape
+ tmp_pf = tmp_f // self.patch_size[0]
+ tmp_ph = tmp_h // self.patch_size[1]
+ tmp_pw = tmp_w // self.patch_size[2]
+ tmp_spatial = tmp_ph * tmp_pw
+
+ tmp_tokens = self.patch_embedding(hr_temporal_latents_input)
+ tmp_tokens = tmp_tokens.flatten(2).transpose(1, 2)
+ tmp_freqs = build_freqs_flat(self.freqs,(hr_pf, tmp_ph, tmp_pw)).view(hr_pf, tmp_spatial, -1)
+ tmp_freqs = tmp_freqs[-tmp_pf:].reshape(tmp_pf * tmp_spatial, -1)
+ tmp_e = repeat(
+ memory_e,
+ "1 D -> B (T HW) D",
+ B=bs,
+ T=tmp_pf,
+ HW=tmp_spatial,
+ )
+ tmp_e0 = self.time_projection(tmp_e).unflatten(2, (6, self.dim))
+ tmp_action = repeat(
+ self._encode_action(hr_temporal_action),
+ "(B T) D -> B (T HW) D",
+ B=bs,
+ HW=tmp_spatial,
+ )
+
+ sink_token_count = 0
+ if sink_action is not None and sink_action.shape[0] > 0:
+ sink_size = sink_action.shape[0]
+ sink_latents = hr_latents_input[:, :, :sink_size]
+ _, _, sink_f, sink_h, sink_w = sink_latents.shape
+ sink_pf = sink_f // self.patch_size[0]
+ sink_ph = sink_h // self.patch_size[1]
+ sink_pw = sink_w // self.patch_size[2]
+ sink_spatial = sink_ph * sink_pw
+
+ sink_tokens = self.patch_embedding(sink_latents)
+ sink_tokens = sink_tokens.flatten(2).transpose(1, 2)
+ sink_freqs = build_freqs_flat(self.freqs,(hr_pf, sink_ph, sink_pw)).view(hr_pf, sink_spatial, -1)
+ sink_freqs = sink_freqs[:sink_pf].reshape(sink_pf * sink_spatial, -1)
+ sink_e = repeat(
+ memory_e,
+ "1 D -> B (T HW) D",
+ B=bs,
+ T=sink_pf,
+ HW=sink_spatial,
+ )
+ sink_e0 = self.time_projection(sink_e).unflatten(2, (6, self.dim))
+ sink_action_embedding = repeat(
+ self._encode_action(sink_action),
+ "(B T) D -> B (T HW) D",
+ B=bs,
+ HW=sink_spatial,
+ )
+ sink_token_count = sink_tokens.shape[1]
+
+ tokens = torch.cat([sink_tokens, cmp_tokens, tmp_tokens], dim=1)
+ freqs_flat = torch.cat([sink_freqs, cmp_freqs, tmp_freqs], dim=0)
+ e0 = torch.cat([sink_e0, cmp_e0, tmp_e0], dim=1)
+ e = torch.cat([sink_e, cmp_e, tmp_e], dim=1)
+ vec_action = torch.cat([sink_action_embedding, cmp_action, tmp_action], dim=1)
+ else:
+ tokens = torch.cat([cmp_tokens, tmp_tokens], dim=1)
+ freqs_flat = torch.cat([cmp_freqs, tmp_freqs], dim=0)
+ e0 = torch.cat([cmp_e0, tmp_e0], dim=1)
+ e = torch.cat([cmp_e, tmp_e], dim=1)
+ vec_action = torch.cat([cmp_action, tmp_action], dim=1)
+
+ return {
+ "tokens": tokens,
+ "freqs_flat": freqs_flat,
+ "e0": e0,
+ "e": e,
+ "vec_action": vec_action,
+ "rope_shift_idx": hr_pf,
+ "sink_token_count": sink_token_count,
+ "cmp_token_count": cmp_tokens.shape[1],
+ "cmp_chunk_token_count": 2 * cmp_spatial,
+ "tmp_token_count": tmp_tokens.shape[1],
+ }
+
+ def forward(
+ self,
+ x=None,
+ t=None,
+ context=None,
+ seq_len=None,
+ y=None,
+ hr_action=None,
+ memory_dict=None,
+ forward_mode="bi_denoise",
+ kv_cache=None,
+ ar_chunk_start=0,
+ hr_latents_input=None,
+ lr_latents_input=None,
+ lr_action=None,
+ memory_t=None,
+ hr_temporal_latents_input=None,
+ hr_temporal_action=None,
+ sink_action=None,
+ pdd_block_index=None,
+ ):
+ """Route all model operations through one explicit entry point."""
+ with fp8_autocast(
+ self.fp8_recipe,
+ enabled=self.fp8_enabled,
+ ):
+ if forward_mode == "compress":
+ return self._forward_compress(
+ hr_latents_input=hr_latents_input,
+ lr_latents_input=lr_latents_input,
+ lr_action=lr_action,
+ memory_t=memory_t,
+ hr_temporal_latents_input=hr_temporal_latents_input,
+ hr_temporal_action=hr_temporal_action,
+ sink_action=sink_action,
+ )
+ if forward_mode == "bi_denoise":
+ return self._forward_bi_denoise(
+ x=x, t=t, context=context, seq_len=seq_len, y=y,
+ hr_action=hr_action, memory_dict=memory_dict)
+ if forward_mode == "ar_prefill":
+ return self._forward_ar_prefill(
+ context=context, memory_dict=memory_dict,
+ kv_cache=kv_cache)
+ if forward_mode == "ar_denoise":
+ return self._forward_ar_denoise(
+ x=x, t=t, context=context, seq_len=seq_len, y=y,
+ hr_action=hr_action, memory_dict=memory_dict,
+ kv_cache=kv_cache, ar_chunk_start=ar_chunk_start)
+ if forward_mode == "pdd_denoise":
+ self.head.set_active_block(pdd_block_index)
+ return self._forward_ar_denoise(
+ x=x, t=t, context=context, seq_len=seq_len, y=y,
+ hr_action=hr_action, memory_dict=memory_dict,
+ kv_cache=kv_cache, ar_chunk_start=ar_chunk_start)
+ raise ValueError(
+ f"unsupported forward_mode={forward_mode!r}; expected "
+ "'compress', 'bi_denoise', 'ar_prefill', 'ar_denoise', "
+ "'pdd_denoise'"
+ )
+
+ def _forward_bi_denoise(
+ self, x, t, context, seq_len, y, hr_action, memory_dict
+ ):
+ """Bidirectional denoise with optional compressed history context."""
+ # BI uses the same body for SP and non-SP. SP only shards the token
+ # tensors before the blocks and gathers the head output afterwards.
+ ar = False
+
+ if self.model_type == 'i2v':
+ assert y is not None
+ # params
+ device = self.patch_embedding.weight.device
+ if self.freqs.device != device:
+ self.freqs = self.freqs.to(device)
+
+ h, w = x[0].shape[2:]
+ tw = w // self.patch_size[2]
+ th = h // self.patch_size[1]
+
+ if y is not None:
+ x = [torch.cat([u, v], dim=0) for u, v in zip(x, y)]
+
+ # embeddings
+ x = [self.patch_embedding(u.unsqueeze(0)) for u in x]
+ grid_sizes = torch.stack(
+ [torch.tensor(u.shape[2:], dtype=torch.long) for u in x])
+ x = [u.flatten(2).transpose(1, 2) for u in x]
+ seq_lens = torch.tensor([u.size(1) for u in x], dtype=torch.long)
+ assert seq_lens.max() <= seq_len
+ x = torch.cat([
+ torch.cat(
+ [u, u.new_zeros(1, seq_len - u.size(1), u.size(2))],
+ dim=1,
+ )
+ for u in x
+ ])
+
+ # time embeddings
+ if t.dim() == 1:
+ t = t.expand(t.size(0), seq_len)
+ with torch.amp.autocast('cuda', dtype=torch.float32):
+ bt = t.size(0)
+ t = t.flatten()
+ e = self.time_embedding(
+ sinusoidal_embedding_1d(
+ self.freq_dim, t
+ ).unflatten(0, (bt, seq_len)).float())
+ e0 = self.time_projection(e).unflatten(2, (6, self.dim))
+ assert e.dtype == torch.float32 and e0.dtype == torch.float32
+
+ # context
+ context_lens = None
+ context = self._project_text_context(context)
+
+ vec_action = None
+ if hr_action is not None:
+ vec_action = self._encode_action(hr_action)
+ vec_action = repeat(
+ vec_action, "(B T) D -> B (T H W) D",
+ B=x.shape[0], H=th, W=tw)
+
+ # Build per-token RoPE for the generate segment. The generate
+ # segment ALWAYS uses per-token ``freqs_flat`` (via ``rope_apply_flat``)
+ # -- both BI and AR, with or without a compressor context -- so the
+ # RoPE handling is uniform and never depends on the grid-based
+ # ``rope_apply`` fallback. When fused with a compressor context the
+ # generate segment's temporal rope indices must START AFTER the
+ # context's time range (= ``rope_shift_idx``); otherwise the generate
+ # and context tokens would share the same temporal phases and the
+ # model could not distinguish "past vs future".
+ #
+ # NB: for ``rope_shift_idx == 0`` (no context) the per-token freqs are
+ # numerically identical to what grid-based ``rope_apply`` would build,
+ # so this does not change the BI first-chunk result.
+ gen_pf = int(grid_sizes[0, 0].item())
+ gen_ph = int(grid_sizes[0, 1].item())
+ gen_pw = int(grid_sizes[0, 2].item())
+ rope_shift_idx = (int(memory_dict.get("rope_shift_idx", 0))
+ if memory_dict is not None else 0)
+ full_freqs = build_freqs_flat(
+ self.freqs,
+ (rope_shift_idx + gen_pf, gen_ph, gen_pw)).to(x.device)
+ full_freqs = full_freqs.view(
+ rope_shift_idx + gen_pf, gen_ph * gen_pw, -1)
+ gen_freqs_flat = full_freqs[-gen_pf:].reshape(
+ gen_pf * gen_ph * gen_pw, -1)
+ # Pad to seq_len (pad tokens are masked out in flash_attention
+ # via ``seq_lens``).
+ if gen_freqs_flat.shape[0] < seq_len:
+ pad_rows = seq_len - gen_freqs_flat.shape[0]
+ pad = gen_freqs_flat.new_zeros(
+ pad_rows, gen_freqs_flat.shape[1])
+ gen_freqs_flat = torch.cat(
+ [gen_freqs_flat, pad], dim=0)
+
+ # -- fuse context (memory_compress output) before generate segment --
+ generate_seq_length = x.shape[1]
+ cmp_token_count = 0
+ cmp_chunk_token_count = 0
+ tmp_token_count = 0
+ if memory_dict is not None:
+ ctx_tokens = memory_dict["tokens"]
+ ctx_freqs_flat = memory_dict["freqs_flat"].to(x.device)
+ ctx_e0 = memory_dict["e0"].float()
+ ctx_e = memory_dict["e"].float()
+ ctx_vec_action = memory_dict["vec_action"]
+
+ x = torch.cat([ctx_tokens, x], dim=1)
+ freqs_flat = torch.cat(
+ [ctx_freqs_flat, gen_freqs_flat], dim=0)
+ e0 = torch.cat([ctx_e0, e0], dim=1)
+ e = torch.cat([ctx_e, e], dim=1)
+ vec_action = torch.cat(
+ [ctx_vec_action, vec_action], dim=1)
+ # seq_lens now counts the full (ctx + generate) length per batch
+ seq_lens = torch.tensor([x.shape[1]] * x.shape[0],
+ dtype=torch.long)
+ cmp_token_count = int(memory_dict.get("cmp_token_count", 0))
+ cmp_chunk_token_count = int(
+ memory_dict.get("cmp_chunk_token_count", 0))
+ tmp_token_count = int(memory_dict.get("tmp_token_count", 0))
+ else:
+ # No compressor ctx (first chunk): generate-only, per-token RoPE.
+ freqs_flat = gen_freqs_flat
+
+ if self.use_sp:
+ sp_size = get_world_size()
+ sp_rank = get_rank()
+ global_sequence = x.shape[1]
+ x = shard_sequence(x, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)
+ e = shard_sequence(e, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)
+ e0 = shard_sequence(e0, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)
+ vec_action = shard_sequence(vec_action, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)
+ freqs_flat = shard_sequence(freqs_flat, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=0)
+ else:
+ global_sequence = x.shape[1]
+
+ # arguments
+ kwargs = dict(
+ e=e0,
+ seq_lens=seq_lens,
+ grid_sizes=grid_sizes,
+ freqs=self.freqs,
+ context=context,
+ context_lens=context_lens,
+ vec_action=vec_action,
+ freqs_flat=freqs_flat,
+ ar=ar,
+ cmp_token_count=cmp_token_count,
+ cmp_chunk_token_count=cmp_chunk_token_count,
+ tmp_token_count=tmp_token_count,
+ )
+
+ for block in self.blocks:
+ x = block(x, **kwargs)
+
+ # head
+ x = self.head(x, e)
+ if self.use_sp:
+ x = gather_sequence(x, global_sequence=global_sequence)
+
+ # drop the prepended context tokens before unpatchify
+ if memory_dict is not None:
+ x = x[:, -generate_seq_length:]
+
+ # unpatchify
+ x = self.unpatchify(x, grid_sizes)
+ return [u.float() for u in x]
+
+ def init_kv_cache(self):
+ r"""Allocate contiguous per-layer ``[sink | cmp | tmp]`` K/V."""
+ return [{'k_vision': None, 'v_vision': None}
+ for _ in range(self.num_layers)]
+
+ def _forward_ar_prefill(self, context, memory_dict, kv_cache):
+ r"""Chunk-AR KV-cache incremental prefill for SP and non-SP.
+
+ Each layer stores only contiguous ``k_vision/v_vision`` with layout
+ ``[sink | accumulated cmp | latest tmp]``. The first context is
+ processed in one masked pass. Later calls infer the old cmp length
+ from the cache shape, discard the previous tmp suffix, process only
+ ``[new cmp | new tmp]``, and replace the cache with the new layout.
+
+ Args:
+ context (List[Tensor]): text embeddings (cond OR uncond).
+ memory_dict (dict | None): compressor output; ``None`` -> no-op.
+ kv_cache (list[dict]): from ``init_kv_cache()``; mutated in place.
+
+ Returns:
+ list[dict]: the same ``kv_cache`` (mutated in place).
+ """
+ assert kv_cache is not None and len(kv_cache) == self.num_layers, (
+ "ar_prefill requires a kv_cache of length num_layers "
+ "(call init_kv_cache()).")
+
+ # First chunk: no compressor ctx -> nothing to cache.
+ if memory_dict is None:
+ return kv_cache
+
+ device = self.patch_embedding.weight.device
+ if self.freqs.device != device:
+ self.freqs = self.freqs.to(device)
+
+ # text context (embed cond/uncond text the same way as the denoise path)
+ context_lens = None
+ context = self._project_text_context(context)
+
+ ctx_tokens = memory_dict["tokens"]
+ freqs_flat = memory_dict["freqs_flat"].to(ctx_tokens.device)
+ e0c = memory_dict["e0"].float() # ensure fp32 (FSDP bf16)
+ vec_action_c = memory_dict["vec_action"]
+ sink_token_count = int(memory_dict.get("sink_token_count", 0))
+ cmp_token_count = int(memory_dict.get("cmp_token_count", 0))
+ cmp_chunk_token_count = int(
+ memory_dict.get("cmp_chunk_token_count", 0))
+ tmp_token_count = int(memory_dict.get("tmp_token_count", 0))
+
+ cache_is_empty = kv_cache[0].get('k_vision') is None
+ if any(
+ (entry.get('k_vision') is None) != cache_is_empty
+ for entry in kv_cache
+ ):
+ raise ValueError("all AR cache layers must have the same state")
+
+ if cache_is_empty:
+ self._prefill_initial_ar_context(
+ tokens=ctx_tokens,
+ modulation=e0c,
+ actions=vec_action_c,
+ freqs_flat=freqs_flat,
+ text_context=context,
+ context_lens=context_lens,
+ kv_cache=kv_cache,
+ sink_token_count=sink_token_count,
+ cmp_token_count=cmp_token_count,
+ cmp_chunk_token_count=cmp_chunk_token_count,
+ tmp_token_count=tmp_token_count,
+ )
+ else:
+ self._prefill_ar_context_update(
+ tokens=ctx_tokens,
+ modulation=e0c,
+ actions=vec_action_c,
+ freqs_flat=freqs_flat,
+ text_context=context,
+ context_lens=context_lens,
+ kv_cache=kv_cache,
+ sink_token_count=sink_token_count,
+ cmp_token_count=cmp_token_count,
+ cmp_chunk_token_count=cmp_chunk_token_count,
+ tmp_token_count=tmp_token_count,
+ )
+ return kv_cache
+
+ def _prefill_initial_ar_context(
+ self,
+ *,
+ tokens,
+ modulation,
+ actions,
+ freqs_flat,
+ text_context,
+ context_lens,
+ kv_cache,
+ sink_token_count,
+ cmp_token_count,
+ cmp_chunk_token_count,
+ tmp_token_count,
+ ):
+ """First prefill: write the complete ``[sink | cmp | tmp]`` cache."""
+ self._run_ar_prefill_blocks(
+ tokens=tokens,
+ modulation=modulation,
+ actions=actions,
+ freqs_flat=freqs_flat,
+ text_context=text_context,
+ context_lens=context_lens,
+ kv_cache=kv_cache,
+ sink_token_count=sink_token_count,
+ cmp_token_count=cmp_token_count,
+ cmp_chunk_token_count=cmp_chunk_token_count,
+ tmp_token_count=tmp_token_count,
+ cache_update=None,
+ )
+
+ def _prefill_ar_context_update(
+ self,
+ *,
+ tokens,
+ modulation,
+ actions,
+ freqs_flat,
+ text_context,
+ context_lens,
+ kv_cache,
+ sink_token_count,
+ cmp_token_count,
+ cmp_chunk_token_count,
+ tmp_token_count,
+ ):
+ """Later prefill: append new cmp and replace the previous tmp."""
+ cached_total = int(kv_cache[0]['k_vision'].shape[1])
+ cached_cmp = cached_total - sink_token_count - tmp_token_count
+ if not 0 <= cached_cmp <= cmp_token_count:
+ raise ValueError(
+ "invalid AR cache layout: "
+ f"cached_total={cached_total}, sink={sink_token_count}, "
+ f"tmp={tmp_token_count}, current_cmp={cmp_token_count}"
+ )
+
+ new_cmp_count = cmp_token_count - cached_cmp
+ cmp_start = sink_token_count + cached_cmp
+ cmp_end = sink_token_count + cmp_token_count
+ tmp_start = cmp_end
+ tmp_end = tmp_start + tmp_token_count
+
+ token_parts = []
+ modulation_parts = []
+ action_parts = []
+ freq_parts = []
+ if new_cmp_count:
+ token_parts.append(tokens[:, cmp_start:cmp_end])
+ modulation_parts.append(modulation[:, cmp_start:cmp_end])
+ freq_parts.append(freqs_flat[cmp_start:cmp_end])
+ if actions is not None:
+ action_parts.append(actions[:, cmp_start:cmp_end])
+ if tmp_token_count:
+ token_parts.append(tokens[:, tmp_start:tmp_end])
+ modulation_parts.append(modulation[:, tmp_start:tmp_end])
+ freq_parts.append(freqs_flat[tmp_start:tmp_end])
+ if actions is not None:
+ action_parts.append(actions[:, tmp_start:tmp_end])
+ if not token_parts:
+ return
+
+ self._run_ar_prefill_blocks(
+ tokens=torch.cat(token_parts, dim=1),
+ modulation=torch.cat(modulation_parts, dim=1),
+ actions=(
+ torch.cat(action_parts, dim=1)
+ if action_parts else None
+ ),
+ freqs_flat=torch.cat(freq_parts, dim=0),
+ text_context=text_context,
+ context_lens=context_lens,
+ kv_cache=kv_cache,
+ sink_token_count=sink_token_count,
+ cmp_token_count=cmp_token_count,
+ cmp_chunk_token_count=cmp_chunk_token_count,
+ tmp_token_count=tmp_token_count,
+ cache_update={
+ 'n_new_cmp': new_cmp_count,
+ 'n_tmp': tmp_token_count,
+ 'cached_prefix_tokens': sink_token_count + cached_cmp,
+ },
+ )
+
+ def _run_ar_prefill_blocks(
+ self,
+ *,
+ tokens,
+ modulation,
+ actions,
+ freqs_flat,
+ text_context,
+ context_lens,
+ kv_cache,
+ sink_token_count,
+ cmp_token_count,
+ cmp_chunk_token_count,
+ tmp_token_count,
+ cache_update,
+ ):
+ """Run either initial or incremental prefill through all blocks."""
+ global_length = tokens.shape[1]
+ if self.use_sp:
+ sp_size = get_world_size()
+ sp_rank = get_rank()
+ tokens = shard_sequence(tokens, global_sequence=global_length, rank=sp_rank, world_size=sp_size, dim=1)
+ modulation = shard_sequence(modulation, global_sequence=global_length, rank=sp_rank, world_size=sp_size, dim=1)
+ freqs_flat = shard_sequence(freqs_flat, global_sequence=global_length, rank=sp_rank, world_size=sp_size, dim=0)
+ actions = shard_sequence(actions, global_sequence=global_length, rank=sp_rank, world_size=sp_size, dim=1)
+
+ seq_lens = torch.tensor([global_length] * tokens.shape[0], dtype=torch.long)
+ for index, block in enumerate(self.blocks):
+ tokens = block(
+ tokens,
+ e=modulation,
+ seq_lens=seq_lens,
+ grid_sizes=None,
+ freqs=self.freqs,
+ context=text_context,
+ context_lens=context_lens,
+ vec_action=actions,
+ freqs_flat=freqs_flat,
+ ar=True,
+ cmp_token_count=cmp_token_count,
+ cmp_chunk_token_count=cmp_chunk_token_count,
+ tmp_token_count=tmp_token_count,
+ kv_cache_entry=kv_cache[index],
+ cache_write=True,
+ sink_token_count=sink_token_count,
+ cache_update=cache_update,
+ )
+
+ def _forward_ar_denoise(
+ self,
+ x=None,
+ t=None,
+ context=None,
+ seq_len=None,
+ y=None,
+ hr_action=None,
+ memory_dict=None,
+ kv_cache=None,
+ ar_chunk_start=0,
+ ):
+ r"""Chunk-AR KV-cache denoise for both SP and non-SP.
+
+ Runs the gen (pred) tokens through every block with
+ ``cache_write=False``; each block reads its cached ctx K/V (written
+ by :meth:`_forward_ar_prefill`) and prepends it to the pred K/V. No
+ ctx tokens are concatenated to ``x`` here. Returns ``list[Tensor]``.
+
+ ``ar_chunk_start`` phase-shifts the generate segment's RoPE so K/V
+ across chunks share a consistent temporal coordinate system. When
+ ``memory_dict`` is ``None`` / the cache is empty (first chunk) this
+ degenerates to plain pred-only self-attention.
+ """
+ assert kv_cache is not None, (
+ "AR denoise requires kv_cache (call init_kv_cache()).")
+ assert len(kv_cache) == self.num_layers, (
+ f"kv_cache length ({len(kv_cache)}) must equal num_layers "
+ f"({self.num_layers}).")
+
+ if self.model_type == 'i2v':
+ assert y is not None
+ device = self.patch_embedding.weight.device
+ if self.freqs.device != device:
+ self.freqs = self.freqs.to(device)
+
+ h, w = x[0].shape[2:]
+ tw = w // self.patch_size[2]
+ th = h // self.patch_size[1]
+
+ if y is not None:
+ x = [torch.cat([u, v], dim=0) for u, v in zip(x, y)]
+
+ # embeddings (gen/pred tokens)
+ x = [self.patch_embedding(u.unsqueeze(0)) for u in x]
+ grid_sizes = torch.stack([torch.tensor(u.shape[2:], dtype=torch.long) for u in x])
+ x = [u.flatten(2).transpose(1, 2) for u in x]
+ seq_lens = torch.tensor([u.size(1) for u in x], dtype=torch.long)
+ assert seq_lens.max() <= seq_len
+ x = torch.cat([
+ torch.cat(
+ [u, u.new_zeros(1, seq_len - u.size(1), u.size(2))],
+ dim=1,
+ )
+ for u in x
+ ])
+
+ # time embeddings
+ if t.dim() == 1:
+ t = t.expand(t.size(0), seq_len)
+ with torch.amp.autocast('cuda', dtype=torch.float32):
+ bt = t.size(0)
+ t = t.flatten()
+ e = self.time_embedding(sinusoidal_embedding_1d(self.freq_dim, t).unflatten(0, (bt, seq_len)).float())
+ e0 = self.time_projection(e).unflatten(2, (6, self.dim))
+ assert e.dtype == torch.float32 and e0.dtype == torch.float32
+
+ # text context
+ context_lens = None
+ context = self._project_text_context(context)
+
+ vec_action = None
+ if hr_action is not None:
+ vec_action = self._encode_action(hr_action)
+ vec_action = repeat(vec_action, "(B T) D -> B (T H W) D", B=x.shape[0], H=th, W=tw)
+
+ # ---- per-token RoPE for the generate segment --------------------
+ # The current chunk lives at hr-latent indices
+ # [ar_chunk_start, ar_chunk_start + gen_pf). When a compressor ctx is
+ # present, the generate segment must start AFTER the ctx time range
+ # (rope_shift_idx); resolve gen_start = max(ar_chunk_start, ctx_shift).
+ gen_pf = int(grid_sizes[0, 0].item())
+ gen_ph = int(grid_sizes[0, 1].item())
+ gen_pw = int(grid_sizes[0, 2].item())
+ ctx_shift = int(memory_dict.get("rope_shift_idx", 0)) if memory_dict is not None else 0
+ gen_start = max(int(ar_chunk_start), ctx_shift)
+ full_freqs = build_freqs_flat(self.freqs, (gen_start + gen_pf, gen_ph, gen_pw)).to(x.device)
+ full_freqs = full_freqs.view(gen_start + gen_pf, gen_ph * gen_pw, -1)
+ gen_freqs_flat = full_freqs[-gen_pf:].reshape(gen_pf * gen_ph * gen_pw, -1)
+
+ # x = pred only; each block reads cached ctx K/V and prepends to pred.
+ freqs_flat = gen_freqs_flat
+ cmp_token_count = int(memory_dict.get("cmp_token_count", 0)) if memory_dict else 0
+ cmp_chunk_token_count = int(memory_dict.get("cmp_chunk_token_count", 0)) if memory_dict else 0
+ tmp_token_count = int(memory_dict.get("tmp_token_count", 0)) if memory_dict else 0
+ seq_lens = torch.tensor([x.shape[1]] * x.shape[0], dtype=torch.long)
+
+ if self.use_sp:
+ sp_size = get_world_size()
+ sp_rank = get_rank()
+ global_sequence = x.shape[1]
+ x = shard_sequence(x, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)
+ e = shard_sequence(e, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)
+ e0 = shard_sequence(e0, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)
+ vec_action = shard_sequence(vec_action, global_sequence=global_sequence, rank=sp_rank,
+ world_size=sp_size, dim=1)
+ freqs_flat = shard_sequence(freqs_flat, global_sequence=global_sequence, rank=sp_rank,
+ world_size=sp_size, dim=0)
+ else:
+ global_sequence = x.shape[1]
+
+ for idx, block in enumerate(self.blocks):
+ x = block(
+ x,
+ e=e0,
+ seq_lens=seq_lens,
+ grid_sizes=grid_sizes,
+ freqs=self.freqs,
+ context=context,
+ context_lens=context_lens,
+ vec_action=vec_action,
+ freqs_flat=freqs_flat,
+ ar=True,
+ cmp_token_count=cmp_token_count,
+ cmp_chunk_token_count=cmp_chunk_token_count,
+ tmp_token_count=tmp_token_count,
+ kv_cache_entry=kv_cache[idx],
+ cache_write=False,
+ )
+
+ # head
+ x = self.head(x, e)
+ endpoint_x = self.head.pop_endpoint_output()
+ if self.use_sp:
+ x = gather_sequence(x, global_sequence=global_sequence)
+ if endpoint_x is not None:
+ endpoint_x = gather_sequence(endpoint_x, global_sequence=global_sequence)
+
+ # unpatchify (no ctx prefix to strip -- x is pred-only throughout)
+ x = self.unpatchify(x, grid_sizes)
+ if endpoint_x is None:
+ return [u.float() for u in x]
+ endpoint_x = self.unpatchify(endpoint_x, grid_sizes)
+ return [
+ (displacement.float(), endpoint.float())
+ for displacement, endpoint in zip(x, endpoint_x)
+ ]
+
+ def unpatchify(self, x, grid_sizes):
+ r"""
+ Reconstruct video tensors from patch embeddings.
+
+ Args:
+ x (List[Tensor]):
+ List of patchified features, each with shape [L, C_out * prod(patch_size)]
+ grid_sizes (Tensor):
+ Original spatial-temporal grid dimensions before patching,
+ shape [B, 3] (3 dimensions correspond to F_patches, H_patches, W_patches)
+
+ Returns:
+ List[Tensor]:
+ Reconstructed video tensors with shape [C_out, F, H / 8, W / 8]
+ """
+
+ c = self.out_dim
+ out = []
+ for u, v in zip(x, grid_sizes.tolist()):
+ u = u[:math.prod(v)].view(*v, *self.patch_size, c)
+ u = torch.einsum('fhwpqrc->cfphqwr', u)
+ u = u.reshape(c, *[i * j for i, j in zip(v, self.patch_size)])
+ out.append(u)
+ return out
+
+ def init_weights(self):
+ r"""
+ Initialize model parameters using Xavier initialization.
+ """
+
+ # basic init
+ for m in self.modules():
+ if isinstance(m, nn.Linear):
+ nn.init.xavier_uniform_(m.weight)
+ if m.bias is not None:
+ nn.init.zeros_(m.bias)
+
+ # init embeddings
+ nn.init.xavier_uniform_(self.patch_embedding.weight.flatten(1))
+ for m in self.text_embedding.modules():
+ if isinstance(m, nn.Linear):
+ nn.init.normal_(m.weight, std=.02)
+ for m in self.time_embedding.modules():
+ if isinstance(m, nn.Linear):
+ nn.init.normal_(m.weight, std=.02)
+
+ # init output layer
+ nn.init.zeros_(self.head.head.weight)
diff --git a/worldplay2/pdd.py b/worldplay2/pdd.py
new file mode 100644
index 0000000..f0d3362
--- /dev/null
+++ b/worldplay2/pdd.py
@@ -0,0 +1,169 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import torch
+
+from .checkpoint import load_state_dict
+
+
+@dataclass(frozen=True)
+class FixedPDDBlock:
+ expert: str
+ local_index: int
+ start: int
+ end: int
+
+
+class FixedPDD4Scheduler:
+ """Fixed four-NFE consistency-PDD schedule used by Self-Forcing."""
+
+ num_intervals = 128
+ shift = 7.0
+ boundary = 0.9
+
+ def __init__(self):
+ raw_boundary = self.boundary / (
+ self.shift - (self.shift - 1.0) * self.boundary
+ )
+ high_count = int(round(self.num_intervals * (1.0 - raw_boundary)))
+ high_count = min(max(high_count, 1), self.num_intervals - 1)
+ raw = torch.cat([
+ torch.linspace(1.0, raw_boundary, high_count + 1),
+ torch.linspace(
+ raw_boundary,
+ 0.0,
+ self.num_intervals - high_count + 1,
+ )[1:],
+ ])
+ self.sigmas = (
+ self.shift * raw / (1.0 + (self.shift - 1.0) * raw)
+ ).float()
+ self.sigmas[0] = 1.0
+ self.sigmas[high_count] = self.boundary
+ self.sigmas[-1] = 0.0
+ self.timesteps = self.sigmas * 1000.0
+ self.boundary_index = high_count
+
+ high_edges = self._partition(0, high_count)
+ low_edges = self._partition(high_count, self.num_intervals)
+ self._blocks = {
+ "high": tuple(
+ FixedPDDBlock("high", i, high_edges[i], high_edges[i + 1])
+ for i in range(2)
+ ),
+ "low": tuple(
+ FixedPDDBlock("low", i, low_edges[i], low_edges[i + 1])
+ for i in range(2)
+ ),
+ }
+ self.ordered_blocks = self._blocks["high"] + self._blocks["low"]
+
+ @staticmethod
+ def _partition(start: int, end: int) -> list[int]:
+ return [start, start + (end - start) // 2, end]
+
+ def blocks(self, expert: str) -> tuple[FixedPDDBlock, ...]:
+ try:
+ return self._blocks[expert]
+ except KeyError as exc:
+ raise ValueError(f"expert must be high/low, got {expert!r}") from exc
+
+ def expert_range(self, expert: str) -> tuple[int, int]:
+ if expert == "high":
+ return 0, self.boundary_index
+ if expert == "low":
+ return self.boundary_index, self.num_intervals
+ raise ValueError(f"expert must be high/low, got {expert!r}")
+
+ def compact_head(
+ self,
+ expert: str,
+ interval_weight: torch.Tensor,
+ interval_bias: torch.Tensor,
+ ) -> tuple[torch.Tensor, torch.Tensor]:
+ expert_start, expert_end = self.expert_range(expert)
+ expected = expert_end - expert_start
+ if interval_weight.ndim != 3 or interval_weight.shape[0] != expected:
+ raise ValueError(
+ f"{expert} PDD checkpoint has interval weight shape "
+ f"{tuple(interval_weight.shape)}, expected [{expected}, O, I]"
+ )
+ if interval_bias.ndim != 2 or interval_bias.shape[0] != expected:
+ raise ValueError(
+ f"{expert} PDD checkpoint has interval bias shape "
+ f"{tuple(interval_bias.shape)}, expected [{expected}, O]"
+ )
+
+ weights = []
+ biases = []
+ for block in self.blocks(expert):
+ indices = [
+ index - expert_start
+ for index in range(block.start, block.end - 1)
+ ]
+ coefficients = (
+ self.sigmas[block.start + 1:block.end]
+ - self.sigmas[block.start:block.end - 1]
+ ).to(interval_weight)
+ displacement_weight = torch.einsum(
+ "k,koi->oi", coefficients, interval_weight[indices]
+ )
+ displacement_bias = torch.einsum(
+ "k,ko->o", coefficients, interval_bias[indices]
+ )
+ endpoint_index = block.end - 1 - expert_start
+ weights.append(torch.cat(
+ (displacement_weight, interval_weight[endpoint_index]), dim=0
+ ))
+ biases.append(torch.cat(
+ (displacement_bias, interval_bias[endpoint_index]), dim=0
+ ))
+ return torch.stack(weights), torch.stack(biases)
+
+ def prepare_checkpoint(
+ self,
+ path: str,
+ *,
+ expert: str,
+ dtype: torch.dtype,
+ ) -> dict[str, torch.Tensor]:
+ """Load either interval-head or already compact fixed-PDD weights."""
+ state = load_state_dict(path)
+ compact_weight = "head.block_weight"
+ compact_bias = "head.block_bias"
+ if compact_weight not in state or compact_bias not in state:
+ weight_key = next((
+ key for key in (
+ "head.weight",
+ "head.pdd_weight",
+ "head.head.weight",
+ )
+ if key in state and state[key].ndim == 3
+ ), None)
+ bias_key = next((
+ key for key in (
+ "head.bias",
+ "head.pdd_bias",
+ "head.head.bias",
+ )
+ if key in state and state[key].ndim == 2
+ ), None)
+ if weight_key is None or bias_key is None:
+ raise KeyError(
+ f"{path} has neither compact fixed-PDD heads nor an "
+ "interval PDD head bank"
+ )
+ block_weight, block_bias = self.compact_head(
+ expert, state.pop(weight_key), state.pop(bias_key)
+ )
+ state[compact_weight] = block_weight
+ state[compact_bias] = block_bias
+
+ # A compact resume can retain a now-unused standard projection.
+ state.pop("head.head.weight", None)
+ state.pop("head.head.bias", None)
+ for key, tensor in state.items():
+ if tensor.is_floating_point() and tensor.dtype != dtype:
+ state[key] = tensor.to(dtype=dtype)
+ return state
diff --git a/worldplay2/pipeline.py b/worldplay2/pipeline.py
new file mode 100644
index 0000000..24cc31c
--- /dev/null
+++ b/worldplay2/pipeline.py
@@ -0,0 +1,1291 @@
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+"""WorldPlay2 chunked inference with compressed memory and temporal anchors,
+built on Wan2.2 I2V-A14B.
+
+Recipe (per HY WorldPlay reference, adapted to Wan2.2):
+ * Generate ``total_latent_frames`` HR latent frames in groups of
+ ``chunk_length`` frames (default 4).
+ * For chunk 0, run a vanilla i2v denoising loop without context.
+ * For chunks > 0:
+ 1. Streaming-VAE-decode the just-finished HR chunk.
+ 2. Take ``frames[:, ::2]`` (every other frame) and bilinear
+ downsample to (H/4, W/4) in pixel space.
+ 3. Streaming-VAE-encode the downsampled clip back to LR latents.
+ 4. Accumulate (lr_latents_cache, hr_latents_cache, decoded_rgb_cache).
+ 5. Build ``hr_temporal`` = last 2 hr latents before the generate
+ window with zero y, build hr_y / lr_y context (4-mask + 16-image
+ latent, image-only at t=0), then call
+ ``model(forward_mode="compress", ...)`` to
+ produce ``memory_dict``.
+ 6. Run the chunk's denoising loop with ``memory_dict`` plumbed in.
+ * After the final chunk, decode the last chunk (continuing the same
+ streaming cache) and concatenate with the cached RGB.
+
+The pipeline builds the T5 encoder, VAE, and two ``WorldPlay2Model`` experts.
+Each expert loads its complete DiT, action encoder, and memory compressor
+weights from a separate checkpoint.
+
+Sequence-parallel behavior is owned by ``WorldPlay2Model`` and
+``WorldPlay2SelfAttention`` themselves. The pipeline only enables it through
+``model.set_sequence_parallel``; no runtime monkey-patching is used.
+"""
+
+import gc
+import logging
+import math
+import os
+import random
+import sys
+from contextlib import contextmanager
+from enum import Enum
+from functools import partial
+
+import torch
+import torch.distributed as dist
+import torch.nn.functional as F
+import torchvision.transforms.functional as TF
+from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
+from tqdm import tqdm
+
+from .checkpoint import load_expert_checkpoint
+from .distributed.fsdp import shard_model
+from .distributed.ops import get_world_size
+from .models.attention import resolve_attention_backend
+from .models.model import WorldPlay2Model
+from .models.fp8 import (
+ convert_model_ffn_to_fp8,
+ convert_model_qkv_to_fp8,
+ fp8_model_stats,
+)
+from .pdd import FixedPDD4Scheduler
+from .schedulers import (FlowDPMSolverMultistepScheduler,
+ FlowUniPCMultistepScheduler, get_sampling_sigmas,
+ retrieve_timesteps)
+from .text.encoder import T5EncoderModel
+from .vae.wan21 import Wan2_1_VAE
+
+
+# ---------------------------------------------------------------------------
+# small utilities (mirror the dataset/training conventions verbatim)
+# ---------------------------------------------------------------------------
+
+def _build_i2v_mask_4ch(t_lat: int, h_lat: int, w_lat: int,
+ device, dtype) -> torch.Tensor:
+ """Build the 4-channel "1,0,0,..." mask used as the first 4 channels
+ of ``y``. Mirrors ``bi_camera_wan22_w_mem_dataset._build_i2v_mask``.
+ """
+ m = torch.ones(1, t_lat * 4 - 3, h_lat, w_lat, device=device, dtype=dtype)
+ m[:, 1:] = 0
+ m = torch.cat([m[:, 0:1].repeat_interleave(4, dim=1), m[:, 1:]], dim=1)
+ m = m.view(1, m.shape[1] // 4, 4, h_lat, w_lat).transpose(1, 2)[0]
+ return m # [4, t_lat, h_lat, w_lat]
+
+
+def _encode_i2v_clip(vae, img_pixel: torch.Tensor, t_lat: int,
+ h_pix: int, w_pix: int, device) -> torch.Tensor:
+ """VAE-encode the I2V pixel clip ``[image, zeros..., zeros]`` into a
+ 16-channel latent of temporal length ``t_lat``.
+
+ Mirrors the upstream :meth:`WanI2V.generate` recipe verbatim::
+
+ y = vae.encode([
+ cat([
+ F.interpolate(img[None], (h, w), mode='bicubic').transpose(0,1),
+ zeros(3, F_pix - 1, h, w),
+ ], dim=1)
+ ])[0]
+
+ where ``F_pix = t_lat * 4 - 3`` so the temporal-causal VAE produces
+ exactly ``t_lat`` latent frames. ``img_pixel`` is on ``device``,
+ shape ``[3, H_in, W_in]``, value range [-1, 1].
+ """
+ F_pix = t_lat * 4 - 3
+ clip = torch.cat([
+ torch.nn.functional.interpolate(
+ img_pixel[None].cpu(), size=(h_pix, w_pix),
+ mode='bilinear').transpose(0, 1), # [3, 1, h, w]
+ torch.zeros(3, F_pix - 1, h_pix, w_pix),
+ ], dim=1).to(device) # [3, F_pix, h, w]
+ return vae.encode([clip])[0] # [16, t_lat, h, w]
+
+
+def _build_y(vae, img_pixel: torch.Tensor, t_lat: int,
+ h_pix: int, w_pix: int, device, dtype) -> torch.Tensor:
+ """Construct the 20-channel y volume = [4-mask | 16-image-clip-latent]
+ using the same recipe as upstream :meth:`WanI2V.generate`.
+ """
+ y_img = _encode_i2v_clip(vae, img_pixel, t_lat, h_pix, w_pix, device)
+ h_lat, w_lat = y_img.shape[2], y_img.shape[3]
+ msk = _build_i2v_mask_4ch(t_lat, h_lat, w_lat, device, y_img.dtype)
+ return torch.cat([msk, y_img], dim=0).to(dtype=dtype) # [20, t_lat, h_lat, w_lat]
+
+
+@contextmanager
+def _noop_no_sync():
+ yield
+
+
+def _parameter_dtype_summary(module) -> dict[str, int]:
+ counts: dict[str, int] = {}
+ for parameter in module.parameters():
+ name = str(parameter.dtype).replace("torch.", "")
+ counts[name] = counts.get(name, 0) + parameter.numel()
+ return counts
+
+
+def _fsdp_unit_count(module) -> int:
+ return sum(
+ 1 for child in module.modules() if isinstance(child, FSDP)
+ )
+
+
+class InferenceMode(str, Enum):
+ BI = "bi"
+ AR = "ar"
+ PDD = "pdd"
+
+
+# ---------------------------------------------------------------------------
+# main pipeline
+# ---------------------------------------------------------------------------
+
+class WorldPlay2Pipeline:
+ """WorldPlay2 pipeline for chunked action-conditioned video generation.
+
+ Owns: T5 encoder, Wan2.1 VAE, two ``WorldPlay2Model`` experts (low / high
+ noise) with the action MLP + memory_compress submodule installed.
+ """
+
+ def __init__(
+ self,
+ config,
+ checkpoint_dir: str,
+ low_noise_mem_ckpt: str,
+ high_noise_mem_ckpt: str,
+ device_id: int = 0,
+ rank: int = 0,
+ t5_fsdp: bool = False,
+ dit_fsdp: bool = False,
+ use_sp: bool = False,
+ compile_qk_rope: bool = True,
+ conditioning_cache: bool = True,
+ fuse_self_qkv: bool = True,
+ compile_block_glue: bool = True,
+ compile_ffn: bool = True,
+ fp8_ffn: bool = False,
+ fp8_start_block: int = 4,
+ fp8_end_block: int = 35,
+ fp8_qkv: bool = False,
+ fp8_qkv_start_block: int = 4,
+ fp8_qkv_end_block: int = 35,
+ init_on_cpu: bool = True,
+ inference_mode: InferenceMode | str = InferenceMode.BI,
+ vae_type: str = "wan2_1",
+ tae_ckpt: str = None,
+ attention_backend: str = "flash",
+ ):
+ """
+ Args:
+ config: model settings from ``configs.I2V_A14B_CONFIG``.
+ checkpoint_dir: directory holding the base T5 / VAE / DiT
+ checkpoints (same layout :class:`WanI2V` expects).
+ low_noise_mem_ckpt: path to a ``.safetensors``/``.pt`` file
+ holding the LOW-noise expert's trained ``action_in.*``,
+ ``img_action_mlp.*`` and ``memory_compress.*`` weights.
+ high_noise_mem_ckpt: same, but for the HIGH-noise expert.
+ The two experts are trained separately (see
+ ``run_bi_wan22_action_mem_stage_three.sh`` with /
+ without ``--low_noise``), so each carries its own
+ memory_compress + action weights and at inference we
+ must run TWO compressor passes per chunk and dispatch
+ by timestep just like the denoise step does.
+ device_id, rank, t5_fsdp, dit_fsdp, use_sp, init_on_cpu:
+ runtime placement options.
+ attention_backend: visual denoising attention backend, 'flash'
+ or 'sage'.
+ """
+ assert low_noise_mem_ckpt is not None and high_noise_mem_ckpt is not None, (
+ "WorldPlay2Pipeline requires BOTH --low_noise_mem_ckpt and "
+ "--high_noise_mem_ckpt; the FastVideo training pipeline "
+ "produces a separate checkpoint for each expert (each with "
+ "its own memory_compress + action weights) and the inference "
+ "loop dispatches between them by timestep.")
+
+ self.inference_mode = InferenceMode(inference_mode)
+ self.attention_backend = resolve_attention_backend(attention_backend)
+ logging.info("Visual self-attention backend: %s", self.attention_backend)
+ self.device = torch.device(f"cuda:{device_id}")
+ self.config = config
+ self.rank = rank
+ self.init_on_cpu = init_on_cpu
+ self.denoise_forward_mode = (
+ "bi_denoise"
+ if self.inference_mode is InferenceMode.BI
+ else (
+ "pdd_denoise"
+ if self.inference_mode is InferenceMode.PDD
+ else "ar_denoise"
+ )
+ )
+ self.pdd_scheduler = (
+ FixedPDD4Scheduler()
+ if self.inference_mode is InferenceMode.PDD
+ else None
+ )
+ self.use_sp = use_sp
+ self.compile_qk_rope = compile_qk_rope
+ self.conditioning_cache_enabled = conditioning_cache
+ self.fuse_self_qkv = fuse_self_qkv
+ self.compile_block_glue = compile_block_glue
+ self.compile_ffn = compile_ffn
+ self.fp8_ffn_enabled = fp8_ffn
+ self.fp8_start_block = fp8_start_block
+ self.fp8_end_block = fp8_end_block
+ self.fp8_qkv_enabled = fp8_qkv
+ self.fp8_qkv_start_block = fp8_qkv_start_block
+ self.fp8_qkv_end_block = fp8_qkv_end_block
+ self.fp8_enabled = fp8_ffn or fp8_qkv
+ if self.fp8_enabled and dit_fsdp:
+ raise ValueError(
+ "DiT FP8 currently requires --dit_fsdp to be disabled")
+ if fp8_qkv and not fuse_self_qkv:
+ raise ValueError(
+ "QKV FP8 requires self-attention QKV fusion")
+
+ self.num_train_timesteps = config.num_train_timesteps
+ self.boundary = config.boundary
+ self.param_dtype = config.param_dtype
+
+ if t5_fsdp or dit_fsdp or use_sp:
+ self.init_on_cpu = False
+
+ shard_fn = partial(shard_model, device_id=device_id)
+ self.text_encoder = T5EncoderModel(
+ text_len=config.text_len,
+ dtype=config.t5_dtype,
+ device=self.device,
+ checkpoint_path=os.path.join(checkpoint_dir, config.t5_checkpoint),
+ tokenizer_path=os.path.join(checkpoint_dir, config.t5_tokenizer),
+ shard_fn=shard_fn if t5_fsdp else None,
+ )
+
+ self.vae_stride = config.vae_stride
+ self.patch_size = config.patch_size
+ # VAE is pluggable: the default Wan2.1 VAE, or the causal TAE
+ # (lightvae) which exposes the SAME encode/decode/encode_chunk/
+ # decode_chunk list interface via CausalTAEVAE, so nothing downstream
+ # changes. Both are temporally causal (T_pix = 4*T_lat - 3).
+ assert vae_type in ("wan2_1", "causal_tae"), (
+ f"vae_type must be 'wan2_1' or 'causal_tae', got {vae_type!r}")
+ if vae_type == "causal_tae":
+ from .vae.causal_tae import CausalTAEVAE
+ assert tae_ckpt is not None, (
+ "vae_type='causal_tae' requires --tae_ckpt "
+ "(the CausalTAE encoder+decoder weights).")
+ self.vae = CausalTAEVAE(
+ vae_pth=tae_ckpt,
+ z_dim=16,
+ device=self.device)
+ else:
+ self.vae = Wan2_1_VAE(
+ vae_pth=os.path.join(checkpoint_dir, config.vae_checkpoint),
+ device=self.device)
+
+ # Build the architecture from the base model config only. The supplied
+ # expert checkpoints contain the complete DiT + action + compressor
+ # state, so loading base-model parameters through from_pretrained would
+ # be redundant and would temporarily consume additional host memory.
+ self.low_noise_model = self._build_expert_from_config(
+ checkpoint_dir=checkpoint_dir,
+ config_subfolder=config.low_noise_checkpoint,
+ expert_ckpt=low_noise_mem_ckpt,
+ tag="low_noise_model",
+ dtype=self.param_dtype,
+ fuse_self_qkv=fuse_self_qkv,
+ pdd_scheduler=self.pdd_scheduler,
+ pdd_expert="low",
+ )
+ self.high_noise_model = self._build_expert_from_config(
+ checkpoint_dir=checkpoint_dir,
+ config_subfolder=config.high_noise_checkpoint,
+ expert_ckpt=high_noise_mem_ckpt,
+ tag="high_noise_model",
+ dtype=self.param_dtype,
+ fuse_self_qkv=fuse_self_qkv,
+ pdd_scheduler=self.pdd_scheduler,
+ pdd_expert="high",
+ )
+
+ # ---- configure (FSDP / SP / dtype / device) ----------------------
+ self.low_noise_model = self._configure_model(
+ model=self.low_noise_model,
+ use_sp=use_sp,
+ attention_backend=self.attention_backend,
+ compile_qk_rope=compile_qk_rope,
+ conditioning_cache=conditioning_cache,
+ compile_block_glue=compile_block_glue,
+ compile_ffn=compile_ffn,
+ fp8_ffn=fp8_ffn,
+ fp8_start_block=fp8_start_block,
+ fp8_end_block=fp8_end_block,
+ fp8_qkv=fp8_qkv,
+ fp8_qkv_start_block=fp8_qkv_start_block,
+ fp8_qkv_end_block=fp8_qkv_end_block,
+ dit_fsdp=dit_fsdp,
+ shard_fn=shard_fn)
+ self.high_noise_model = self._configure_model(
+ model=self.high_noise_model,
+ use_sp=use_sp,
+ attention_backend=self.attention_backend,
+ compile_qk_rope=compile_qk_rope,
+ conditioning_cache=conditioning_cache,
+ compile_block_glue=compile_block_glue,
+ compile_ffn=compile_ffn,
+ fp8_ffn=fp8_ffn,
+ fp8_start_block=fp8_start_block,
+ fp8_end_block=fp8_end_block,
+ fp8_qkv=fp8_qkv,
+ fp8_qkv_start_block=fp8_qkv_start_block,
+ fp8_qkv_end_block=fp8_qkv_end_block,
+ dit_fsdp=dit_fsdp,
+ shard_fn=shard_fn)
+
+ self.sp_size = get_world_size() if use_sp else 1
+ self.sample_neg_prompt = config.sample_neg_prompt
+ self.dtype_report = {
+ "attention_backend": self.attention_backend,
+ "configured_dit_compute_dtype": str(
+ self.param_dtype).replace("torch.", ""),
+ "t5_fsdp_requested": bool(t5_fsdp),
+ "dit_fsdp": bool(dit_fsdp),
+ "t5_fsdp_units": _fsdp_unit_count(self.text_encoder.model),
+ "low_noise_fsdp_units": _fsdp_unit_count(
+ self.low_noise_model),
+ "high_noise_fsdp_units": _fsdp_unit_count(
+ self.high_noise_model),
+ "compile_qk_rope": bool(compile_qk_rope),
+ "conditioning_cache": bool(conditioning_cache),
+ "self_qkv_fused": bool(fuse_self_qkv),
+ "compile_block_glue": bool(compile_block_glue),
+ "compile_ffn": bool(compile_ffn),
+ "fp8_ffn": bool(fp8_ffn),
+ "fp8_block_range": (
+ [fp8_start_block, fp8_end_block] if fp8_ffn else None
+ ),
+ "fp8_qkv": bool(fp8_qkv),
+ "fp8_qkv_block_range": (
+ [fp8_qkv_start_block, fp8_qkv_end_block]
+ if fp8_qkv else None
+ ),
+ "low_noise_parameter_storage": _parameter_dtype_summary(
+ self.low_noise_model),
+ "high_noise_parameter_storage": _parameter_dtype_summary(
+ self.high_noise_model),
+ "t5_configured_dtype": str(
+ config.t5_dtype).replace("torch.", ""),
+ "t5_parameter_storage": _parameter_dtype_summary(
+ self.text_encoder.model),
+ "vae_configured_dtype": str(
+ self.vae.dtype).replace("torch.", ""),
+ "vae_parameter_storage": _parameter_dtype_summary(
+ self.vae.model),
+ }
+ logging.info("Dtype report: %s", self.dtype_report)
+
+ # ------------------------------------------------------------------
+ # ckpt loading + model configuration
+ # ------------------------------------------------------------------
+
+ @staticmethod
+ def _build_expert_from_config(
+ checkpoint_dir: str,
+ config_subfolder: str,
+ expert_ckpt: str,
+ tag: str,
+ dtype: torch.dtype,
+ fuse_self_qkv: bool,
+ pdd_scheduler: FixedPDD4Scheduler | None = None,
+ pdd_expert: str | None = None,
+ ) -> WorldPlay2Model:
+ """Initialize architecture from config and strictly load a full expert."""
+ config_dir = os.path.join(checkpoint_dir, config_subfolder)
+ logging.info("Creating %s from config: %s", tag, config_dir)
+ model_config = WorldPlay2Model.load_config(config_dir)
+ model = WorldPlay2Model.from_config(model_config)
+ model.add_embedding_action_parameters()
+ model.add_memory_compress_model()
+ if pdd_scheduler is not None:
+ model.head.make_fixed_pdd(num_blocks=2)
+ # Keep the actual DiT/action/compressor parameter storage in BF16,
+ # rather than relying on autocast or FSDP's transient compute cast.
+ model.to(dtype=dtype)
+ if pdd_scheduler is None:
+ load_expert_checkpoint(model, expert_ckpt, tag=tag, strict=True)
+ else:
+ state = pdd_scheduler.prepare_checkpoint(
+ expert_ckpt, expert=pdd_expert, dtype=dtype
+ )
+ provided = set(state)
+ required_prefixes = (
+ "memory_compress.",
+ "action_in_cont.",
+ "action_in_disc.",
+ )
+ absent = [
+ prefix for prefix in required_prefixes
+ if not any(key.startswith(prefix) for key in provided)
+ ]
+ if absent:
+ raise ValueError(
+ f"{tag} PDD checkpoint is missing required parameter "
+ f"groups: {absent}"
+ )
+ model.load_state_dict(state, strict=True)
+ logging.info(
+ "%s fixed-PDD checkpoint: loaded=%d expert=%s",
+ tag,
+ len(state),
+ pdd_expert,
+ )
+ # Install after strict checkpoint loading (which expects q/k/v keys)
+ # and before FSDP wrapping (which should only see the fused weights).
+ if fuse_self_qkv:
+ model.fuse_self_attention_qkv()
+ return model
+
+ def _configure_model(
+ self,
+ model,
+ use_sp,
+ attention_backend,
+ compile_qk_rope,
+ conditioning_cache,
+ compile_block_glue,
+ compile_ffn,
+ fp8_ffn,
+ fp8_start_block,
+ fp8_end_block,
+ fp8_qkv,
+ fp8_qkv_start_block,
+ fp8_qkv_end_block,
+ dit_fsdp,
+ shard_fn,
+ ):
+ """Configure dtype/device/FSDP and enable the model-owned SP path."""
+ model.eval().requires_grad_(False)
+ model.to(dtype=self.param_dtype)
+
+ model.set_sequence_parallel(use_sp)
+ model.set_self_attention_backend(attention_backend)
+ model.set_compiled_qk_rope(compile_qk_rope)
+ model.set_conditioning_cache(conditioning_cache)
+ model.set_compiled_regions(
+ block_glue=compile_block_glue,
+ ffn=compile_ffn,
+ )
+
+ if dist.is_initialized():
+ dist.barrier(device_ids=[self.device.index])
+
+ if fp8_ffn or fp8_qkv:
+ # TE primary FP8 weights must remain resident. This path
+ # intentionally does not support DiT FSDP or model offload.
+ model.to(self.device)
+ if fp8_qkv:
+ report = convert_model_qkv_to_fp8(
+ model,
+ start_block=fp8_qkv_start_block,
+ end_block=fp8_qkv_end_block,
+ )
+ logging.info("Installed self-attention QKV FP8: %s", report)
+ if fp8_ffn:
+ report = convert_model_ffn_to_fp8(
+ model,
+ start_block=fp8_start_block,
+ end_block=fp8_end_block,
+ )
+ logging.info("Installed FFN FP8: %s", report)
+
+ if dit_fsdp:
+ model = shard_fn(model)
+ else:
+ if not self.init_on_cpu:
+ model.to(self.device)
+
+ return model
+
+ @staticmethod
+ def _find_wan_model(model):
+ for module in model.modules():
+ if isinstance(module, WorldPlay2Model):
+ return module
+ raise RuntimeError("FSDP/model wrapper does not contain WorldPlay2Model")
+
+ def clear_conditioning_cache(self) -> None:
+ self._find_wan_model(
+ self.low_noise_model).clear_conditioning_cache()
+ self._find_wan_model(
+ self.high_noise_model).clear_conditioning_cache()
+
+ def fp8_stats(self) -> dict:
+ return {
+ "low_noise": fp8_model_stats(
+ self._find_wan_model(self.low_noise_model)),
+ "high_noise": fp8_model_stats(
+ self._find_wan_model(self.high_noise_model)),
+ }
+
+ def _prepare_model_for_timestep(self, t, boundary, offload_model):
+ """Mirrors :meth:`WanI2V._prepare_model_for_timestep`."""
+ if t.item() >= boundary:
+ required_model_name = 'high_noise_model'
+ offload_model_name = 'low_noise_model'
+ else:
+ required_model_name = 'low_noise_model'
+ offload_model_name = 'high_noise_model'
+ if offload_model or self.init_on_cpu:
+ other = getattr(self, offload_model_name)
+ active = getattr(self, required_model_name)
+ if next(other.parameters()).device.type == 'cuda':
+ # Cached tensors are plain inference state and are not moved by
+ # Module.to(). Drop them before offloading the expert.
+ self._find_wan_model(other).clear_conditioning_cache()
+ other.to('cpu')
+ if next(active.parameters()).device.type == 'cpu':
+ active.to(self.device)
+ return getattr(self, required_model_name)
+
+ def _prepare_named_expert(self, expert: str, offload_model: bool):
+ required = f"{expert}_noise_model"
+ other = "low_noise_model" if expert == "high" else "high_noise_model"
+ if offload_model or self.init_on_cpu:
+ other_model = getattr(self, other)
+ active_model = getattr(self, required)
+ if next(other_model.parameters()).device.type == "cuda":
+ self._find_wan_model(other_model).clear_conditioning_cache()
+ other_model.to("cpu")
+ if next(active_model.parameters()).device.type == "cpu":
+ active_model.to(self.device)
+ return getattr(self, required)
+
+ def _run_pdd_chunk(
+ self,
+ *,
+ latent,
+ y_chunk,
+ hr_action_chunk,
+ context,
+ memory_dict_high,
+ memory_dict_low,
+ kv_high,
+ kv_low,
+ chunk_start,
+ seed_g,
+ ):
+ for block in self.pdd_scheduler.ordered_blocks:
+ model = self._prepare_named_expert(
+ block.expert,
+ self.offload_model,
+ )
+
+ kv_cache = kv_high if block.expert == "high" else kv_low
+ memory_dict = (
+ memory_dict_high
+ if block.expert == "high"
+ else memory_dict_low
+ )
+ timestep = self.pdd_scheduler.timesteps[block.start:block.start + 1].to(self.device)
+ args = dict(
+ context=[context[0]],
+ seq_len=self.seq_len_chunk,
+ y=[y_chunk],
+ hr_action=hr_action_chunk,
+ forward_mode="pdd_denoise",
+ pdd_block_index=block.local_index,
+ kv_cache=kv_cache,
+ ar_chunk_start=chunk_start,
+ memory_dict=memory_dict,
+ )
+ output = model(
+ [latent.to(self.device)],
+ t=timestep,
+ **args,
+ )[0]
+ if not isinstance(output, tuple) or len(output) != 2:
+ raise RuntimeError(
+ "fixed PDD model must return "
+ "(displacement, endpoint_velocity)"
+ )
+ displacement, endpoint_velocity = output
+ sigma_endpoint = self.pdd_scheduler.sigmas[block.end - 1].to(device=latent.device, dtype=torch.float32)
+ state_endpoint = latent.float() + displacement.float()
+ x0 = (state_endpoint - sigma_endpoint * endpoint_velocity.float()).to(latent.dtype)
+
+ sigma_end = self.pdd_scheduler.sigmas[block.end].to(device=latent.device, dtype=torch.float32)
+ if block.end == self.pdd_scheduler.num_intervals:
+ latent = x0
+ else:
+ noise = torch.randn(x0.shape, generator=seed_g, device=x0.device, dtype=x0.dtype)
+ latent = ((1.0 - sigma_end) * x0.float() + sigma_end * noise.float()).to(x0.dtype)
+ return latent
+
+ def _build_chunk_scheduler(
+ self,
+ sample_solver: str,
+ sampling_steps: int,
+ shift: float,
+ ):
+ if sample_solver == "unipc":
+ scheduler = FlowUniPCMultistepScheduler(
+ num_train_timesteps=self.num_train_timesteps,
+ shift=1,
+ use_dynamic_shifting=False,
+ )
+ scheduler.set_timesteps(
+ sampling_steps,
+ device=self.device,
+ shift=shift,
+ )
+ return scheduler, scheduler.timesteps
+ if sample_solver == "dpm++":
+ scheduler = FlowDPMSolverMultistepScheduler(
+ num_train_timesteps=self.num_train_timesteps,
+ shift=1,
+ use_dynamic_shifting=False,
+ )
+ sampling_sigmas = get_sampling_sigmas(sampling_steps, shift)
+ timesteps, _ = retrieve_timesteps(
+ scheduler,
+ device=self.device,
+ sigmas=sampling_sigmas,
+ )
+ return scheduler, timesteps
+ raise NotImplementedError(f"Unsupported solver {sample_solver!r}")
+
+ def _predict_chunk_noise(
+ self,
+ *,
+ latent,
+ timestep_value,
+ boundary,
+ context,
+ context_null,
+ memory_dict_high,
+ memory_dict_low,
+ kv_high,
+ kv_high_neg,
+ kv_low,
+ kv_low_neg,
+ y_chunk,
+ hr_action_chunk,
+ ar_cache,
+ chunk_start,
+ ):
+ latent_model_input = [latent.to(self.device)]
+ timestep = torch.stack([timestep_value]).to(self.device)
+ model = self._prepare_model_for_timestep(
+ timestep_value,
+ boundary,
+ self.offload_model,
+ )
+
+ if timestep_value.item() >= boundary:
+ memory_dict = memory_dict_high
+ kv_cache, kv_cache_neg = kv_high, kv_high_neg
+ sample_guide_scale = self.guide_scale[1]
+ use_cfg = self.use_cfg_high
+ else:
+ memory_dict = memory_dict_low
+ kv_cache, kv_cache_neg = kv_low, kv_low_neg
+ sample_guide_scale = self.guide_scale[0]
+ use_cfg = self.use_cfg_low
+
+ model_args = {
+ "context": [context[0]],
+ "seq_len": self.seq_len_chunk,
+ "y": [y_chunk],
+ "hr_action": hr_action_chunk,
+ "forward_mode": self.denoise_forward_mode,
+ }
+ if ar_cache:
+ model_args.update(
+ kv_cache=kv_cache,
+ ar_chunk_start=chunk_start,
+ )
+ else:
+ model_args["memory_dict"] = memory_dict
+
+ noise_pred_cond = model(
+ latent_model_input,
+ t=timestep,
+ **model_args,
+ )[0]
+ if use_cfg:
+ null_args = {
+ "context": context_null,
+ "seq_len": self.seq_len_chunk,
+ "y": [y_chunk],
+ "hr_action": hr_action_chunk,
+ "forward_mode": self.denoise_forward_mode,
+ }
+ if ar_cache:
+ null_args.update(
+ kv_cache=kv_cache_neg,
+ ar_chunk_start=chunk_start,
+ )
+ else:
+ null_args["memory_dict"] = memory_dict
+ noise_pred_uncond = model(
+ latent_model_input,
+ t=timestep,
+ **null_args,
+ )[0]
+ if self.offload_model:
+ torch.cuda.empty_cache()
+ noise_pred_cond = noise_pred_uncond + sample_guide_scale * (
+ noise_pred_cond - noise_pred_uncond
+ )
+ return noise_pred_cond
+
+ def _run_solver_chunk(
+ self,
+ *,
+ latent,
+ sample_solver,
+ sampling_steps,
+ shift,
+ seed_g,
+ boundary,
+ context,
+ context_null,
+ memory_dict_high,
+ memory_dict_low,
+ kv_high,
+ kv_high_neg,
+ kv_low,
+ kv_low_neg,
+ y_chunk,
+ hr_action_chunk,
+ ar_cache,
+ chunk_start,
+ chunk_index,
+ ):
+ scheduler, timesteps = self._build_chunk_scheduler(
+ sample_solver,
+ sampling_steps,
+ shift,
+ )
+ if self.offload_model:
+ torch.cuda.empty_cache()
+
+ for timestep_value in tqdm(timesteps, desc=f"chunk {chunk_index}"):
+ noise_pred = self._predict_chunk_noise(
+ latent=latent,
+ timestep_value=timestep_value,
+ boundary=boundary,
+ context=context,
+ context_null=context_null,
+ memory_dict_high=memory_dict_high,
+ memory_dict_low=memory_dict_low,
+ kv_high=kv_high,
+ kv_high_neg=kv_high_neg,
+ kv_low=kv_low,
+ kv_low_neg=kv_low_neg,
+ y_chunk=y_chunk,
+ hr_action_chunk=hr_action_chunk,
+ ar_cache=ar_cache,
+ chunk_start=chunk_start,
+ )
+ latent = scheduler.step(
+ noise_pred.unsqueeze(0),
+ timestep_value,
+ latent.unsqueeze(0),
+ return_dict=False,
+ generator=seed_g,
+ )[0].squeeze(0)
+ return latent
+
+ def _prefill_kv_cache(
+ self,
+ model,
+ memory_dict,
+ kv_cache,
+ kv_cache_neg,
+ context,
+ context_null,
+ use_cfg,
+ ):
+ """Prefill conditional cache and the optional CFG cache.
+
+ The ctx self-attn K/V pass through cross-attention (text), so the
+ cond and uncond ctx K/V differ -> two separate caches. When
+ ``memory_dict`` is None (first chunk) prefill is a no-op and the
+ caches stay empty (denoise then runs pred-only attention).
+
+ Returns the (possibly mutated) ``(kv_cache, kv_cache_neg)`` so the
+ caller rebinds explicitly rather than relying on in-place mutation.
+ """
+ if memory_dict is None:
+ return kv_cache, kv_cache_neg
+ kv_cache = model(
+ forward_mode="ar_prefill",
+ context=[context[0]],
+ memory_dict=memory_dict,
+ kv_cache=kv_cache,
+ )
+ if use_cfg:
+ if context_null is None or kv_cache_neg is None:
+ raise ValueError("CFG prefill requires null context and cache")
+ kv_cache_neg = model(
+ forward_mode="ar_prefill",
+ context=context_null,
+ memory_dict=memory_dict,
+ kv_cache=kv_cache_neg,
+ )
+ return kv_cache, kv_cache_neg
+
+ # ------------------------------------------------------------------
+ # streaming VAE helpers
+ # ------------------------------------------------------------------
+
+ def _decode_chunk_and_lr_encode(
+ self,
+ hr_latent_chunk: torch.Tensor,
+ is_first_vae_chunk: bool,
+ ):
+ """Decode one HR latent chunk via streaming VAE, downsample
+ (T::2, HW//4 in pixel space), then re-encode via streaming VAE.
+
+ Returns (lr_latent, rgb_frames) where:
+ lr_latent : [16, T_lr_lat, h//4, w//4]
+ rgb_frames : [3, T_pix, H_pix, W_pix] in [-1, 1]
+ """
+ rgb = self.vae.decode_chunk([hr_latent_chunk],
+ is_first_chunk=is_first_vae_chunk)[0]
+
+ lr_pix = rgb[:, ::2, :, :] # T::2
+ h_pix, w_pix = lr_pix.shape[2:]
+ target_size = (h_pix // 4, w_pix // 4)
+ lr_pix = lr_pix.permute(1, 0, 2, 3).contiguous() # [T, C, H, W]
+ lr_pix = F.interpolate(lr_pix, size=target_size, mode='bilinear',
+ align_corners=False)
+ lr_pix = lr_pix.permute(1, 0, 2, 3).contiguous() # [C, T, H, W]
+
+ lr_lat = self.vae.encode_chunk([lr_pix],
+ is_first_chunk=is_first_vae_chunk)[0]
+ return lr_lat, rgb
+
+ def _build_lr_image_pixel(self, img_pixel_hr: torch.Tensor,
+ h_hr_pix: int, w_hr_pix: int) -> torch.Tensor:
+ """Bicubic-downsample the HR pixel image to the LR resolution
+ ``(h_hr_pix//4, w_hr_pix//4)`` -- matches the (H//4, W//4) target
+ used by :meth:`_decode_chunk_and_lr_encode`. Returns ``[3, H_lr, W_lr]``
+ on ``self.device``.
+ """
+ return F.interpolate(
+ img_pixel_hr[None].cpu(),
+ size=(h_hr_pix // 4, w_hr_pix // 4),
+ mode='bilinear')[0].to(self.device)
+
+ def _prepare_chunk_memory(
+ self,
+ *,
+ chunk_index,
+ chunk_end,
+ latents,
+ hr_y_full,
+ lr_y_full,
+ hr_action,
+ lr_action,
+ context,
+ context_null,
+ kv_high,
+ kv_high_neg,
+ kv_low,
+ kv_low_neg,
+ ):
+ """Decode a generated chunk and prepare memory for the next chunk."""
+ chunk_start = chunk_end - self.chunk_length
+ generated_latents = latents[:, chunk_start:chunk_end].contiguous()
+ new_lr_latents, decoded_rgb = self._decode_chunk_and_lr_encode(
+ generated_latents,
+ is_first_vae_chunk=chunk_index == 0,
+ )
+ if self.lr_latents_cache is None:
+ self.lr_latents_cache = new_lr_latents
+ else:
+ self.lr_latents_cache = torch.cat(
+ [self.lr_latents_cache, new_lr_latents],
+ dim=1,
+ )
+
+ hr_context_latents = latents[:, :chunk_end].contiguous()
+ hr_context_y = hr_y_full[:, :chunk_end]
+ lr_context_y = lr_y_full[:, :chunk_end // 2]
+ hr_input = torch.cat([hr_context_latents, hr_context_y], dim=0).to(dtype=self.param_dtype).unsqueeze(0)
+ lr_input = torch.cat(
+ [self.lr_latents_cache, lr_context_y],
+ dim=0,
+ ).to(dtype=self.param_dtype).unsqueeze(0)
+
+ temporal_latents = hr_context_latents[:, -self.temporal_size:].to(dtype=self.param_dtype)
+ temporal_y = hr_y_full[:, chunk_end - self.temporal_size:chunk_end].to(dtype=self.param_dtype)
+ temporal_input = torch.cat([temporal_latents, temporal_y], dim=0).unsqueeze(0)
+
+ compressor_args = {
+ "forward_mode": "compress",
+ "hr_latents_input": hr_input,
+ "lr_latents_input": lr_input,
+ "lr_action": lr_action[:chunk_end // 2],
+ "memory_t": torch.zeros(1, device=self.device, dtype=self.param_dtype),
+ "hr_temporal_latents_input": temporal_input,
+ "hr_temporal_action": hr_action[chunk_end - self.temporal_size:chunk_end],
+ "sink_action": (
+ hr_action[:self.sink_size]
+ if self.sink_size > 0
+ else None
+ ),
+ }
+
+ if self.offload_model:
+ if next(self.high_noise_model.parameters()).device.type != "cuda":
+ self.high_noise_model.to(self.device)
+ if next(self.low_noise_model.parameters()).device.type == "cuda":
+ self.low_noise_model.to("cpu")
+ memory_dict_high = self.high_noise_model(**compressor_args)
+ if self.ar_cache:
+ kv_high, kv_high_neg = self._prefill_kv_cache(
+ self.high_noise_model,
+ memory_dict_high,
+ kv_high,
+ kv_high_neg,
+ context=context,
+ context_null=context_null,
+ use_cfg=self.use_cfg_high,
+ )
+
+ if self.offload_model:
+ if next(self.low_noise_model.parameters()).device.type != "cuda":
+ self.low_noise_model.to(self.device)
+ if next(self.high_noise_model.parameters()).device.type == "cuda":
+ self.high_noise_model.to("cpu")
+ memory_dict_low = self.low_noise_model(**compressor_args)
+ if self.ar_cache:
+ kv_low, kv_low_neg = self._prefill_kv_cache(
+ self.low_noise_model,
+ memory_dict_low,
+ kv_low,
+ kv_low_neg,
+ context=context,
+ context_null=context_null,
+ use_cfg=self.use_cfg_low,
+ )
+
+ return (
+ decoded_rgb.cpu(),
+ memory_dict_high,
+ memory_dict_low,
+ kv_high,
+ kv_high_neg,
+ kv_low,
+ kv_low_neg,
+ )
+
+ # ------------------------------------------------------------------
+ # public entry: chunked autoregressive generation
+ # ------------------------------------------------------------------
+
+ def generate_chunked(
+ self,
+ prompts: dict[str, str],
+ chunk_prompts: tuple[str, ...],
+ img,
+ hr_action: torch.Tensor,
+ total_latent_frames: int = 16,
+ chunk_length: int = 4,
+ shift: float = 5.0,
+ sample_solver: str = 'unipc',
+ sampling_steps: int = 40,
+ guide_scale=5.0,
+ n_prompt: str = "",
+ seed: int = -1,
+ offload_model: bool = True,
+ sink_size: int = 0,
+ temporal_size: int = 2,
+ height: int = 448,
+ width: int = 832,
+ ):
+ """Run stage-3 chunked inference. See module docstring for
+ the recipe. Returns ``[3, T_pix, H_pix, W_pix]`` on rank 0,
+ ``None`` on other ranks.
+
+ ``sink_size`` (default 0): number of leading hr latent frames used as
+ the context "sink" (``hr_latents_input[:, :, :sink_size]``, prepended
+ to the front of the context). 0 disables the sink.
+
+ ``temporal_size`` (default 2): number of trailing hr latent frames
+ used as the temporal anchor (``hr_ctx_lat[:, -temporal_size:]``),
+ appended after the compressed lr+hr context.
+ """
+ assert total_latent_frames % chunk_length == 0, (
+ f"total_latent_frames ({total_latent_frames}) must be a multiple "
+ f"of chunk_length ({chunk_length})")
+ num_chunks = total_latent_frames // chunk_length
+ if len(chunk_prompts) != num_chunks:
+ raise ValueError(
+ f"chunk_prompts must contain {num_chunks} entries, "
+ f"got {len(chunk_prompts)}"
+ )
+ # Conditioning is sample-local. Never carry projected text or
+ # cross-attention K/V into the next JSON item.
+ self.clear_conditioning_cache()
+
+ if isinstance(guide_scale, (int, float)):
+ guide_scale = (float(guide_scale), float(guide_scale))
+ else:
+ guide_scale = tuple(float(scale) for scale in guide_scale)
+ if len(guide_scale) != 2:
+ raise ValueError(
+ "guide_scale must be a scalar or (low_noise, high_noise)"
+ )
+ use_cfg_low = not math.isclose(
+ guide_scale[0], 1.0, rel_tol=0.0, abs_tol=1e-8)
+ use_cfg_high = not math.isclose(
+ guide_scale[1], 1.0, rel_tol=0.0, abs_tol=1e-8)
+ use_any_cfg = use_cfg_low or use_cfg_high
+ if self.inference_mode is InferenceMode.PDD and use_any_cfg:
+ raise ValueError("PDD consistency inference requires guide_scale=1")
+ self.chunk_length = chunk_length
+ self.temporal_size = temporal_size
+ self.sink_size = sink_size
+ self.offload_model = offload_model
+ self.ar_cache = (
+ self.denoise_forward_mode in {"ar_denoise", "pdd_denoise"}
+ )
+ self.guide_scale = guide_scale
+ self.use_cfg_high = use_cfg_high
+ self.use_cfg_low = use_cfg_low
+ self.lr_latents_cache = None
+ logging.info(
+ "CFG enabled: low_noise=%s high_noise=%s",
+ use_cfg_low,
+ use_cfg_high,
+ )
+
+ # ---- preprocess image ----
+ img = TF.to_tensor(img).sub_(0.5).div_(0.5).to(self.device)
+ h, w = height, width
+ lat_h = h // self.vae_stride[1] # 56
+ lat_w = w // self.vae_stride[2] # 104
+ img_pixel = F.interpolate(img[None].cpu(), size=(h, w),
+ mode='bicubic')[0].to(self.device)
+
+ # ---- text encode ----
+ if n_prompt == "":
+ n_prompt = self.sample_neg_prompt
+ prompt_names = tuple(dict.fromkeys(chunk_prompts))
+ prompt_texts = [prompts[name] for name in prompt_names]
+ encoded_prompts = self.text_encoder(prompt_texts, self.device)
+ context_null = (
+ self.text_encoder([n_prompt], self.device)
+ if use_any_cfg else None
+ )
+ prompt_contexts = {
+ name: [encoded]
+ for name, encoded in zip(prompt_names, encoded_prompts)
+ }
+
+ # ---- pre-build hr_y_full / lr_y_full (upstream WanI2V recipe:
+ # encode the full pixel clip [image, zeros..., zeros] in one shot
+ # so the temporally-causal VAE produces the correct latent volume).
+ with torch.no_grad():
+ hr_y_full = _build_y(
+ self.vae, img_pixel,
+ t_lat=total_latent_frames,
+ h_pix=h, w_pix=w,
+ device=self.device, dtype=self.param_dtype)
+
+ img_pixel_lr = self._build_lr_image_pixel(img_pixel, h, w)
+ lr_y_full = _build_y(
+ self.vae, img_pixel_lr,
+ t_lat=total_latent_frames // 2,
+ h_pix=h // 4, w_pix=w // 4,
+ device=self.device, dtype=self.param_dtype)
+
+ # ---- noise allocation for the whole HR latent volume ----
+ seed = seed if seed >= 0 else random.randint(0, sys.maxsize)
+ seed_g = torch.Generator(device=self.device)
+ seed_g.manual_seed(seed)
+ noise_full = torch.randn(
+ 16, total_latent_frames, lat_h, lat_w,
+ dtype=torch.float32, generator=seed_g, device=self.device)
+ latents = noise_full.clone()
+
+ # ---- hr_action sanity ----
+ hr_action = hr_action.to(self.device)
+ assert hr_action.shape[0] >= total_latent_frames, (
+ f"hr_action must have >= total_latent_frames ({total_latent_frames}) "
+ f"rows; got {hr_action.shape[0]}.")
+ lr_action = hr_action[::2][:total_latent_frames // 2]
+
+ decoded_rgb_cache: list = []
+
+ # Chunk-AR always keeps conditional caches. Negative caches are only
+ # allocated for experts whose guide scale actually enables CFG.
+ ar_cache = self.ar_cache
+ # PERSISTENT KV caches, created ONCE and reused across all chunks.
+ # The unified prefill path is incremental: each chunk only pushes the
+ # new cmp block + recomputed tmp anchor. Each layer stores one
+ # contiguous [sink | accumulated cmp | latest tmp] K/V pair instead
+ # of re-prefilling the whole growing ctx. When AR but no KV cache,
+ # ar_cache=False and the denoise falls back to the fused [ctx|pred] AR
+ # path (memory_dict). BI leaves them None.
+ if ar_cache:
+ kv_high = self.high_noise_model.init_kv_cache()
+ kv_low = self.low_noise_model.init_kv_cache()
+ kv_high_neg = (
+ self.high_noise_model.init_kv_cache()
+ if use_cfg_high else None
+ )
+ kv_low_neg = (
+ self.low_noise_model.init_kv_cache()
+ if use_cfg_low else None
+ )
+ else:
+ kv_high = kv_high_neg = kv_low = kv_low_neg = None
+ memory_dict_high = None
+ memory_dict_low = None
+ boundary = self.boundary * self.num_train_timesteps
+ no_sync_low = getattr(self.low_noise_model, 'no_sync', _noop_no_sync)
+ no_sync_high = getattr(self.high_noise_model, 'no_sync', _noop_no_sync)
+
+ with (torch.amp.autocast('cuda', dtype=self.param_dtype),
+ torch.no_grad(), no_sync_low(), no_sync_high()):
+
+ # Exact gen token count -- no rounding to a multiple of sp_size.
+ # The unified SP forwards (BI + AR) chunk the sequence UNEVENLY
+ # (uneven all_to_all) and attend with SDPA over the true
+ # sequence, so any extra zero-padded rows would leak into the
+ # attention (BI: as real tokens; AR: into the chunk-causal mask).
+ seq_len_chunk = chunk_length * lat_h * lat_w // (
+ self.patch_size[1] * self.patch_size[2])
+ self.seq_len_chunk = seq_len_chunk
+
+ for chunk_i in range(num_chunks):
+ s = chunk_i * chunk_length
+ e = (chunk_i + 1) * chunk_length
+ prompt_name = chunk_prompts[chunk_i]
+ context = prompt_contexts[prompt_name]
+
+ logging.info(f"[stage3] chunk {chunk_i}/{num_chunks - 1} "
+ f"latent frames [{s}, {e}) prompt={prompt_name}")
+
+ # Per-chunk denoise inputs (also needed by AR prefill below).
+ latent = latents[:, s:e].contiguous()
+ hr_action_chunk = hr_action[s:e]
+ y_chunk = hr_y_full[:, s:e]
+
+ chunk_denoise_args = dict(
+ latent=latent,
+ seed_g=seed_g,
+ boundary=boundary,
+ context=context,
+ context_null=context_null,
+ memory_dict_high=memory_dict_high,
+ memory_dict_low=memory_dict_low,
+ kv_high=kv_high,
+ kv_high_neg=kv_high_neg,
+ kv_low=kv_low,
+ kv_low_neg=kv_low_neg,
+ y_chunk=y_chunk,
+ hr_action_chunk=hr_action_chunk,
+ chunk_start=s,
+ )
+
+ if self.inference_mode is InferenceMode.PDD:
+ latent = self._run_pdd_chunk(
+ latent=latent,
+ y_chunk=y_chunk,
+ hr_action_chunk=hr_action_chunk,
+ context=context,
+ memory_dict_high=memory_dict_high,
+ memory_dict_low=memory_dict_low,
+ kv_high=kv_high,
+ kv_low=kv_low,
+ chunk_start=s,
+ seed_g=seed_g,
+ )
+ elif self.inference_mode is InferenceMode.BI:
+ latent = self._run_solver_chunk(
+ sample_solver=sample_solver,
+ sampling_steps=sampling_steps,
+ shift=shift,
+ ar_cache=False,
+ chunk_index=chunk_i,
+ **chunk_denoise_args,
+ )
+ elif self.inference_mode is InferenceMode.AR:
+ latent = self._run_solver_chunk(
+ sample_solver=sample_solver,
+ sampling_steps=sampling_steps,
+ shift=shift,
+ ar_cache=True,
+ chunk_index=chunk_i,
+ **chunk_denoise_args,
+ )
+ else:
+ raise RuntimeError(
+ f"Unsupported inference mode: {self.inference_mode}"
+ )
+ latents[:, s:e] = latent.to(latents.dtype)
+
+ if chunk_i < num_chunks - 1:
+ (
+ decoded_rgb,
+ memory_dict_high,
+ memory_dict_low,
+ kv_high,
+ kv_high_neg,
+ kv_low,
+ kv_low_neg,
+ ) = self._prepare_chunk_memory(
+ chunk_index=chunk_i,
+ chunk_end=e,
+ latents=latents,
+ hr_y_full=hr_y_full,
+ lr_y_full=lr_y_full,
+ hr_action=hr_action,
+ lr_action=lr_action,
+ context=context,
+ context_null=context_null,
+ kv_high=kv_high,
+ kv_high_neg=kv_high_neg,
+ kv_low=kv_low,
+ kv_low_neg=kv_low_neg,
+ )
+ decoded_rgb_cache.append(decoded_rgb)
+ elif self.rank == 0:
+ final_rgb = self.vae.decode_chunk(
+ [latent],
+ is_first_chunk=chunk_i == 0,
+ )[0].cpu()
+ decoded_rgb_cache.append(final_rgb)
+
+ if self.rank == 0:
+ video = torch.cat(decoded_rgb_cache, dim=1)
+ else:
+ video = None
+
+ if offload_model:
+ self.clear_conditioning_cache()
+ self.low_noise_model.cpu()
+ self.high_noise_model.cpu()
+ torch.cuda.empty_cache()
+
+ if offload_model:
+ gc.collect()
+ torch.cuda.synchronize()
+ if dist.is_initialized():
+ dist.barrier(device_ids=[self.device.index])
+
+ return video
diff --git a/worldplay2/schedulers/__init__.py b/worldplay2/schedulers/__init__.py
new file mode 100644
index 0000000..4d446c5
--- /dev/null
+++ b/worldplay2/schedulers/__init__.py
@@ -0,0 +1,13 @@
+from .dpm_solver import (
+ FlowDPMSolverMultistepScheduler,
+ get_sampling_sigmas,
+ retrieve_timesteps,
+)
+from .unipc import FlowUniPCMultistepScheduler
+
+__all__ = [
+ "FlowDPMSolverMultistepScheduler",
+ "FlowUniPCMultistepScheduler",
+ "get_sampling_sigmas",
+ "retrieve_timesteps",
+]
diff --git a/worldplay2/schedulers/dpm_solver.py b/worldplay2/schedulers/dpm_solver.py
new file mode 100644
index 0000000..17bef85
--- /dev/null
+++ b/worldplay2/schedulers/dpm_solver.py
@@ -0,0 +1,859 @@
+# Copied from https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
+# Convert dpm solver for flow matching
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+
+import inspect
+import math
+from typing import List, Optional, Tuple, Union
+
+import numpy as np
+import torch
+from diffusers.configuration_utils import ConfigMixin, register_to_config
+from diffusers.schedulers.scheduling_utils import (
+ KarrasDiffusionSchedulers,
+ SchedulerMixin,
+ SchedulerOutput,
+)
+from diffusers.utils import deprecate, is_scipy_available
+from diffusers.utils.torch_utils import randn_tensor
+
+if is_scipy_available():
+ pass
+
+
+def get_sampling_sigmas(sampling_steps, shift):
+ sigma = np.linspace(1, 0, sampling_steps + 1)[:sampling_steps]
+ sigma = (shift * sigma / (1 + (shift - 1) * sigma))
+
+ return sigma
+
+
+def retrieve_timesteps(
+ scheduler,
+ num_inference_steps=None,
+ device=None,
+ timesteps=None,
+ sigmas=None,
+ **kwargs,
+):
+ if timesteps is not None and sigmas is not None:
+ raise ValueError(
+ "Only one of `timesteps` or `sigmas` can be passed. Please choose one to set custom values"
+ )
+ if timesteps is not None:
+ accepts_timesteps = "timesteps" in set(
+ inspect.signature(scheduler.set_timesteps).parameters.keys())
+ if not accepts_timesteps:
+ raise ValueError(
+ f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom"
+ f" timestep schedules. Please check whether you are using the correct scheduler."
+ )
+ scheduler.set_timesteps(timesteps=timesteps, device=device, **kwargs)
+ timesteps = scheduler.timesteps
+ num_inference_steps = len(timesteps)
+ elif sigmas is not None:
+ accept_sigmas = "sigmas" in set(
+ inspect.signature(scheduler.set_timesteps).parameters.keys())
+ if not accept_sigmas:
+ raise ValueError(
+ f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom"
+ f" sigmas schedules. Please check whether you are using the correct scheduler."
+ )
+ scheduler.set_timesteps(sigmas=sigmas, device=device, **kwargs)
+ timesteps = scheduler.timesteps
+ num_inference_steps = len(timesteps)
+ else:
+ scheduler.set_timesteps(num_inference_steps, device=device, **kwargs)
+ timesteps = scheduler.timesteps
+ return timesteps, num_inference_steps
+
+
+class FlowDPMSolverMultistepScheduler(SchedulerMixin, ConfigMixin):
+ """
+ `FlowDPMSolverMultistepScheduler` is a fast dedicated high-order solver for diffusion ODEs.
+ This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic
+ methods the library implements for all schedulers such as loading and saving.
+ Args:
+ num_train_timesteps (`int`, defaults to 1000):
+ The number of diffusion steps to train the model. This determines the resolution of the diffusion process.
+ solver_order (`int`, defaults to 2):
+ The DPMSolver order which can be `1`, `2`, or `3`. It is recommended to use `solver_order=2` for guided
+ sampling, and `solver_order=3` for unconditional sampling. This affects the number of model outputs stored
+ and used in multistep updates.
+ prediction_type (`str`, defaults to "flow_prediction"):
+ Prediction type of the scheduler function; must be `flow_prediction` for this scheduler, which predicts
+ the flow of the diffusion process.
+ shift (`float`, *optional*, defaults to 1.0):
+ A factor used to adjust the sigmas in the noise schedule. It modifies the step sizes during the sampling
+ process.
+ use_dynamic_shifting (`bool`, defaults to `False`):
+ Whether to apply dynamic shifting to the timesteps based on image resolution. If `True`, the shifting is
+ applied on the fly.
+ thresholding (`bool`, defaults to `False`):
+ Whether to use the "dynamic thresholding" method. This method adjusts the predicted sample to prevent
+ saturation and improve photorealism.
+ dynamic_thresholding_ratio (`float`, defaults to 0.995):
+ The ratio for the dynamic thresholding method. Valid only when `thresholding=True`.
+ sample_max_value (`float`, defaults to 1.0):
+ The threshold value for dynamic thresholding. Valid only when `thresholding=True` and
+ `algorithm_type="dpmsolver++"`.
+ algorithm_type (`str`, defaults to `dpmsolver++`):
+ Algorithm type for the solver; can be `dpmsolver`, `dpmsolver++`, `sde-dpmsolver` or `sde-dpmsolver++`. The
+ `dpmsolver` type implements the algorithms in the [DPMSolver](https://huggingface.co/papers/2206.00927)
+ paper, and the `dpmsolver++` type implements the algorithms in the
+ [DPMSolver++](https://huggingface.co/papers/2211.01095) paper. It is recommended to use `dpmsolver++` or
+ `sde-dpmsolver++` with `solver_order=2` for guided sampling like in Stable Diffusion.
+ solver_type (`str`, defaults to `midpoint`):
+ Solver type for the second-order solver; can be `midpoint` or `heun`. The solver type slightly affects the
+ sample quality, especially for a small number of steps. It is recommended to use `midpoint` solvers.
+ lower_order_final (`bool`, defaults to `True`):
+ Whether to use lower-order solvers in the final steps. Only valid for < 15 inference steps. This can
+ stabilize the sampling of DPMSolver for steps < 15, especially for steps <= 10.
+ euler_at_final (`bool`, defaults to `False`):
+ Whether to use Euler's method in the final step. It is a trade-off between numerical stability and detail
+ richness. This can stabilize the sampling of the SDE variant of DPMSolver for small number of inference
+ steps, but sometimes may result in blurring.
+ final_sigmas_type (`str`, *optional*, defaults to "zero"):
+ The final `sigma` value for the noise schedule during the sampling process. If `"sigma_min"`, the final
+ sigma is the same as the last sigma in the training schedule. If `zero`, the final sigma is set to 0.
+ lambda_min_clipped (`float`, defaults to `-inf`):
+ Clipping threshold for the minimum value of `lambda(t)` for numerical stability. This is critical for the
+ cosine (`squaredcos_cap_v2`) noise schedule.
+ variance_type (`str`, *optional*):
+ Set to "learned" or "learned_range" for diffusion models that predict variance. If set, the model's output
+ contains the predicted Gaussian variance.
+ """
+
+ _compatibles = [e.name for e in KarrasDiffusionSchedulers]
+ order = 1
+
+ @register_to_config
+ def __init__(
+ self,
+ num_train_timesteps: int = 1000,
+ solver_order: int = 2,
+ prediction_type: str = "flow_prediction",
+ shift: Optional[float] = 1.0,
+ use_dynamic_shifting=False,
+ thresholding: bool = False,
+ dynamic_thresholding_ratio: float = 0.995,
+ sample_max_value: float = 1.0,
+ algorithm_type: str = "dpmsolver++",
+ solver_type: str = "midpoint",
+ lower_order_final: bool = True,
+ euler_at_final: bool = False,
+ final_sigmas_type: Optional[str] = "zero", # "zero", "sigma_min"
+ lambda_min_clipped: float = -float("inf"),
+ variance_type: Optional[str] = None,
+ invert_sigmas: bool = False,
+ ):
+ if algorithm_type in ["dpmsolver", "sde-dpmsolver"]:
+ deprecation_message = f"algorithm_type {algorithm_type} is deprecated and will be removed in a future version. Choose from `dpmsolver++` or `sde-dpmsolver++` instead"
+ deprecate("algorithm_types dpmsolver and sde-dpmsolver", "1.0.0",
+ deprecation_message)
+
+ # settings for DPM-Solver
+ if algorithm_type not in [
+ "dpmsolver", "dpmsolver++", "sde-dpmsolver", "sde-dpmsolver++"
+ ]:
+ if algorithm_type == "deis":
+ self.register_to_config(algorithm_type="dpmsolver++")
+ else:
+ raise NotImplementedError(
+ f"{algorithm_type} is not implemented for {self.__class__}")
+
+ if solver_type not in ["midpoint", "heun"]:
+ if solver_type in ["logrho", "bh1", "bh2"]:
+ self.register_to_config(solver_type="midpoint")
+ else:
+ raise NotImplementedError(
+ f"{solver_type} is not implemented for {self.__class__}")
+
+ if algorithm_type not in ["dpmsolver++", "sde-dpmsolver++"
+ ] and final_sigmas_type == "zero":
+ raise ValueError(
+ f"`final_sigmas_type` {final_sigmas_type} is not supported for `algorithm_type` {algorithm_type}. Please choose `sigma_min` instead."
+ )
+
+ # setable values
+ self.num_inference_steps = None
+ alphas = np.linspace(1, 1 / num_train_timesteps,
+ num_train_timesteps)[::-1].copy()
+ sigmas = 1.0 - alphas
+ sigmas = torch.from_numpy(sigmas).to(dtype=torch.float32)
+
+ if not use_dynamic_shifting:
+ # when use_dynamic_shifting is True, we apply the timestep shifting on the fly based on the image resolution
+ sigmas = shift * sigmas / (1 +
+ (shift - 1) * sigmas) # pyright: ignore
+
+ self.sigmas = sigmas
+ self.timesteps = sigmas * num_train_timesteps
+
+ self.model_outputs = [None] * solver_order
+ self.lower_order_nums = 0
+ self._step_index = None
+ self._begin_index = None
+
+ # self.sigmas = self.sigmas.to(
+ # "cpu") # to avoid too much CPU/GPU communication
+ self.sigma_min = self.sigmas[-1].item()
+ self.sigma_max = self.sigmas[0].item()
+
+ @property
+ def step_index(self):
+ """
+ The index counter for current timestep. It will increase 1 after each scheduler step.
+ """
+ return self._step_index
+
+ @property
+ def begin_index(self):
+ """
+ The index for the first timestep. It should be set from pipeline with `set_begin_index` method.
+ """
+ return self._begin_index
+
+ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.set_begin_index
+ def set_begin_index(self, begin_index: int = 0):
+ """
+ Sets the begin index for the scheduler. This function should be run from pipeline before the inference.
+ Args:
+ begin_index (`int`):
+ The begin index for the scheduler.
+ """
+ self._begin_index = begin_index
+
+ # Modified from diffusers.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler.set_timesteps
+ def set_timesteps(
+ self,
+ num_inference_steps: Union[int, None] = None,
+ device: Union[str, torch.device] = None,
+ sigmas: Optional[List[float]] = None,
+ mu: Optional[Union[float, None]] = None,
+ shift: Optional[Union[float, None]] = None,
+ ):
+ """
+ Sets the discrete timesteps used for the diffusion chain (to be run before inference).
+ Args:
+ num_inference_steps (`int`):
+ Total number of the spacing of the time steps.
+ device (`str` or `torch.device`, *optional*):
+ The device to which the timesteps should be moved to. If `None`, the timesteps are not moved.
+ """
+
+ if self.config.use_dynamic_shifting and mu is None:
+ raise ValueError(
+ " you have to pass a value for `mu` when `use_dynamic_shifting` is set to be `True`"
+ )
+
+ if sigmas is None:
+ sigmas = np.linspace(self.sigma_max, self.sigma_min,
+ num_inference_steps +
+ 1).copy()[:-1] # pyright: ignore
+
+ if self.config.use_dynamic_shifting:
+ sigmas = self.time_shift(mu, 1.0, sigmas) # pyright: ignore
+ else:
+ if shift is None:
+ shift = self.config.shift
+ sigmas = shift * sigmas / (1 +
+ (shift - 1) * sigmas) # pyright: ignore
+
+ if self.config.final_sigmas_type == "sigma_min":
+ sigma_last = ((1 - self.alphas_cumprod[0]) /
+ self.alphas_cumprod[0])**0.5
+ elif self.config.final_sigmas_type == "zero":
+ sigma_last = 0
+ else:
+ raise ValueError(
+ f"`final_sigmas_type` must be one of 'zero', or 'sigma_min', but got {self.config.final_sigmas_type}"
+ )
+
+ timesteps = sigmas * self.config.num_train_timesteps
+ sigmas = np.concatenate([sigmas, [sigma_last]
+ ]).astype(np.float32) # pyright: ignore
+
+ self.sigmas = torch.from_numpy(sigmas)
+ self.timesteps = torch.from_numpy(timesteps).to(
+ device=device, dtype=torch.int64)
+
+ self.num_inference_steps = len(timesteps)
+
+ self.model_outputs = [
+ None,
+ ] * self.config.solver_order
+ self.lower_order_nums = 0
+
+ self._step_index = None
+ self._begin_index = None
+ # self.sigmas = self.sigmas.to(
+ # "cpu") # to avoid too much CPU/GPU communication
+
+ # Copied from diffusers.schedulers.scheduling_ddpm.DDPMScheduler._threshold_sample
+ def _threshold_sample(self, sample: torch.Tensor) -> torch.Tensor:
+ """
+ "Dynamic thresholding: At each sampling step we set s to a certain percentile absolute pixel value in xt0 (the
+ prediction of x_0 at timestep t), and if s > 1, then we threshold xt0 to the range [-s, s] and then divide by
+ s. Dynamic thresholding pushes saturated pixels (those near -1 and 1) inwards, thereby actively preventing
+ pixels from saturation at each step. We find that dynamic thresholding results in significantly better
+ photorealism as well as better image-text alignment, especially when using very large guidance weights."
+ https://arxiv.org/abs/2205.11487
+ """
+ dtype = sample.dtype
+ batch_size, channels, *remaining_dims = sample.shape
+
+ if dtype not in (torch.float32, torch.float64):
+ sample = sample.float(
+ ) # upcast for quantile calculation, and clamp not implemented for cpu half
+
+ # Flatten sample for doing quantile calculation along each image
+ sample = sample.reshape(batch_size, channels * np.prod(remaining_dims))
+
+ abs_sample = sample.abs() # "a certain percentile absolute pixel value"
+
+ s = torch.quantile(
+ abs_sample, self.config.dynamic_thresholding_ratio, dim=1)
+ s = torch.clamp(
+ s, min=1, max=self.config.sample_max_value
+ ) # When clamped to min=1, equivalent to standard clipping to [-1, 1]
+ s = s.unsqueeze(
+ 1) # (batch_size, 1) because clamp will broadcast along dim=0
+ sample = torch.clamp(
+ sample, -s, s
+ ) / s # "we threshold xt0 to the range [-s, s] and then divide by s"
+
+ sample = sample.reshape(batch_size, channels, *remaining_dims)
+ sample = sample.to(dtype)
+
+ return sample
+
+ # Copied from diffusers.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler._sigma_to_t
+ def _sigma_to_t(self, sigma):
+ return sigma * self.config.num_train_timesteps
+
+ def _sigma_to_alpha_sigma_t(self, sigma):
+ return 1 - sigma, sigma
+
+ # Copied from diffusers.schedulers.scheduling_flow_match_euler_discrete.set_timesteps
+ def time_shift(self, mu: float, sigma: float, t: torch.Tensor):
+ return math.exp(mu) / (math.exp(mu) + (1 / t - 1)**sigma)
+
+ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.convert_model_output
+ def convert_model_output(
+ self,
+ model_output: torch.Tensor,
+ *args,
+ sample: torch.Tensor = None,
+ **kwargs,
+ ) -> torch.Tensor:
+ """
+ Convert the model output to the corresponding type the DPMSolver/DPMSolver++ algorithm needs. DPM-Solver is
+ designed to discretize an integral of the noise prediction model, and DPM-Solver++ is designed to discretize an
+ integral of the data prediction model.
+ <Tip>
+ The algorithm and model type are decoupled. You can use either DPMSolver or DPMSolver++ for both noise
+ prediction and data prediction models.
+ </Tip>
+ Args:
+ model_output (`torch.Tensor`):
+ The direct output from the learned diffusion model.
+ sample (`torch.Tensor`):
+ A current instance of a sample created by the diffusion process.
+ Returns:
+ `torch.Tensor`:
+ The converted model output.
+ """
+ timestep = args[0] if len(args) > 0 else kwargs.pop("timestep", None)
+ if sample is None:
+ if len(args) > 1:
+ sample = args[1]
+ else:
+ raise ValueError(
+ "missing `sample` as a required keyward argument")
+ if timestep is not None:
+ deprecate(
+ "timesteps",
+ "1.0.0",
+ "Passing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",
+ )
+
+ # DPM-Solver++ needs to solve an integral of the data prediction model.
+ if self.config.algorithm_type in ["dpmsolver++", "sde-dpmsolver++"]:
+ if self.config.prediction_type == "flow_prediction":
+ sigma_t = self.sigmas[self.step_index]
+ x0_pred = sample - sigma_t * model_output
+ else:
+ raise ValueError(
+ f"prediction_type given as {self.config.prediction_type} must be one of `epsilon`, `sample`,"
+ " `v_prediction`, or `flow_prediction` for the FlowDPMSolverMultistepScheduler."
+ )
+
+ if self.config.thresholding:
+ x0_pred = self._threshold_sample(x0_pred)
+
+ return x0_pred
+
+ # DPM-Solver needs to solve an integral of the noise prediction model.
+ elif self.config.algorithm_type in ["dpmsolver", "sde-dpmsolver"]:
+ if self.config.prediction_type == "flow_prediction":
+ sigma_t = self.sigmas[self.step_index]
+ epsilon = sample - (1 - sigma_t) * model_output
+ else:
+ raise ValueError(
+ f"prediction_type given as {self.config.prediction_type} must be one of `epsilon`, `sample`,"
+ " `v_prediction` or `flow_prediction` for the FlowDPMSolverMultistepScheduler."
+ )
+
+ if self.config.thresholding:
+ sigma_t = self.sigmas[self.step_index]
+ x0_pred = sample - sigma_t * model_output
+ x0_pred = self._threshold_sample(x0_pred)
+ epsilon = model_output + x0_pred
+
+ return epsilon
+
+ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.dpm_solver_first_order_update
+ def dpm_solver_first_order_update(
+ self,
+ model_output: torch.Tensor,
+ *args,
+ sample: torch.Tensor = None,
+ noise: Optional[torch.Tensor] = None,
+ **kwargs,
+ ) -> torch.Tensor:
+ """
+ One step for the first-order DPMSolver (equivalent to DDIM).
+ Args:
+ model_output (`torch.Tensor`):
+ The direct output from the learned diffusion model.
+ sample (`torch.Tensor`):
+ A current instance of a sample created by the diffusion process.
+ Returns:
+ `torch.Tensor`:
+ The sample tensor at the previous timestep.
+ """
+ timestep = args[0] if len(args) > 0 else kwargs.pop("timestep", None)
+ prev_timestep = args[1] if len(args) > 1 else kwargs.pop(
+ "prev_timestep", None)
+ if sample is None:
+ if len(args) > 2:
+ sample = args[2]
+ else:
+ raise ValueError(
+ " missing `sample` as a required keyward argument")
+ if timestep is not None:
+ deprecate(
+ "timesteps",
+ "1.0.0",
+ "Passing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",
+ )
+
+ if prev_timestep is not None:
+ deprecate(
+ "prev_timestep",
+ "1.0.0",
+ "Passing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",
+ )
+
+ sigma_t, sigma_s = self.sigmas[self.step_index + 1], self.sigmas[
+ self.step_index] # pyright: ignore
+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma_t)
+ alpha_s, sigma_s = self._sigma_to_alpha_sigma_t(sigma_s)
+ lambda_t = torch.log(alpha_t) - torch.log(sigma_t)
+ lambda_s = torch.log(alpha_s) - torch.log(sigma_s)
+
+ h = lambda_t - lambda_s
+ if self.config.algorithm_type == "dpmsolver++":
+ x_t = (sigma_t /
+ sigma_s) * sample - (alpha_t *
+ (torch.exp(-h) - 1.0)) * model_output
+ elif self.config.algorithm_type == "dpmsolver":
+ x_t = (alpha_t /
+ alpha_s) * sample - (sigma_t *
+ (torch.exp(h) - 1.0)) * model_output
+ elif self.config.algorithm_type == "sde-dpmsolver++":
+ assert noise is not None
+ x_t = ((sigma_t / sigma_s * torch.exp(-h)) * sample +
+ (alpha_t * (1 - torch.exp(-2.0 * h))) * model_output +
+ sigma_t * torch.sqrt(1.0 - torch.exp(-2 * h)) * noise)
+ elif self.config.algorithm_type == "sde-dpmsolver":
+ assert noise is not None
+ x_t = ((alpha_t / alpha_s) * sample - 2.0 *
+ (sigma_t * (torch.exp(h) - 1.0)) * model_output +
+ sigma_t * torch.sqrt(torch.exp(2 * h) - 1.0) * noise)
+ return x_t # pyright: ignore
+
+ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.multistep_dpm_solver_second_order_update
+ def multistep_dpm_solver_second_order_update(
+ self,
+ model_output_list: List[torch.Tensor],
+ *args,
+ sample: torch.Tensor = None,
+ noise: Optional[torch.Tensor] = None,
+ **kwargs,
+ ) -> torch.Tensor:
+ """
+ One step for the second-order multistep DPMSolver.
+ Args:
+ model_output_list (`List[torch.Tensor]`):
+ The direct outputs from learned diffusion model at current and latter timesteps.
+ sample (`torch.Tensor`):
+ A current instance of a sample created by the diffusion process.
+ Returns:
+ `torch.Tensor`:
+ The sample tensor at the previous timestep.
+ """
+ timestep_list = args[0] if len(args) > 0 else kwargs.pop(
+ "timestep_list", None)
+ prev_timestep = args[1] if len(args) > 1 else kwargs.pop(
+ "prev_timestep", None)
+ if sample is None:
+ if len(args) > 2:
+ sample = args[2]
+ else:
+ raise ValueError(
+ " missing `sample` as a required keyward argument")
+ if timestep_list is not None:
+ deprecate(
+ "timestep_list",
+ "1.0.0",
+ "Passing `timestep_list` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",
+ )
+
+ if prev_timestep is not None:
+ deprecate(
+ "prev_timestep",
+ "1.0.0",
+ "Passing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",
+ )
+
+ sigma_t, sigma_s0, sigma_s1 = (
+ self.sigmas[self.step_index + 1], # pyright: ignore
+ self.sigmas[self.step_index],
+ self.sigmas[self.step_index - 1], # pyright: ignore
+ )
+
+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma_t)
+ alpha_s0, sigma_s0 = self._sigma_to_alpha_sigma_t(sigma_s0)
+ alpha_s1, sigma_s1 = self._sigma_to_alpha_sigma_t(sigma_s1)
+
+ lambda_t = torch.log(alpha_t) - torch.log(sigma_t)
+ lambda_s0 = torch.log(alpha_s0) - torch.log(sigma_s0)
+ lambda_s1 = torch.log(alpha_s1) - torch.log(sigma_s1)
+
+ m0, m1 = model_output_list[-1], model_output_list[-2]
+
+ h, h_0 = lambda_t - lambda_s0, lambda_s0 - lambda_s1
+ r0 = h_0 / h
+ D0, D1 = m0, (1.0 / r0) * (m0 - m1)
+ if self.config.algorithm_type == "dpmsolver++":
+ # See https://arxiv.org/abs/2211.01095 for detailed derivations
+ if self.config.solver_type == "midpoint":
+ x_t = ((sigma_t / sigma_s0) * sample -
+ (alpha_t * (torch.exp(-h) - 1.0)) * D0 - 0.5 *
+ (alpha_t * (torch.exp(-h) - 1.0)) * D1)
+ elif self.config.solver_type == "heun":
+ x_t = ((sigma_t / sigma_s0) * sample -
+ (alpha_t * (torch.exp(-h) - 1.0)) * D0 +
+ (alpha_t * ((torch.exp(-h) - 1.0) / h + 1.0)) * D1)
+ elif self.config.algorithm_type == "dpmsolver":
+ # See https://arxiv.org/abs/2206.00927 for detailed derivations
+ if self.config.solver_type == "midpoint":
+ x_t = ((alpha_t / alpha_s0) * sample -
+ (sigma_t * (torch.exp(h) - 1.0)) * D0 - 0.5 *
+ (sigma_t * (torch.exp(h) - 1.0)) * D1)
+ elif self.config.solver_type == "heun":
+ x_t = ((alpha_t / alpha_s0) * sample -
+ (sigma_t * (torch.exp(h) - 1.0)) * D0 -
+ (sigma_t * ((torch.exp(h) - 1.0) / h - 1.0)) * D1)
+ elif self.config.algorithm_type == "sde-dpmsolver++":
+ assert noise is not None
+ if self.config.solver_type == "midpoint":
+ x_t = ((sigma_t / sigma_s0 * torch.exp(-h)) * sample +
+ (alpha_t * (1 - torch.exp(-2.0 * h))) * D0 + 0.5 *
+ (alpha_t * (1 - torch.exp(-2.0 * h))) * D1 +
+ sigma_t * torch.sqrt(1.0 - torch.exp(-2 * h)) * noise)
+ elif self.config.solver_type == "heun":
+ x_t = ((sigma_t / sigma_s0 * torch.exp(-h)) * sample +
+ (alpha_t * (1 - torch.exp(-2.0 * h))) * D0 +
+ (alpha_t * ((1.0 - torch.exp(-2.0 * h)) /
+ (-2.0 * h) + 1.0)) * D1 +
+ sigma_t * torch.sqrt(1.0 - torch.exp(-2 * h)) * noise)
+ elif self.config.algorithm_type == "sde-dpmsolver":
+ assert noise is not None
+ if self.config.solver_type == "midpoint":
+ x_t = ((alpha_t / alpha_s0) * sample - 2.0 *
+ (sigma_t * (torch.exp(h) - 1.0)) * D0 -
+ (sigma_t * (torch.exp(h) - 1.0)) * D1 +
+ sigma_t * torch.sqrt(torch.exp(2 * h) - 1.0) * noise)
+ elif self.config.solver_type == "heun":
+ x_t = ((alpha_t / alpha_s0) * sample - 2.0 *
+ (sigma_t * (torch.exp(h) - 1.0)) * D0 - 2.0 *
+ (sigma_t * ((torch.exp(h) - 1.0) / h - 1.0)) * D1 +
+ sigma_t * torch.sqrt(torch.exp(2 * h) - 1.0) * noise)
+ return x_t # pyright: ignore
+
+ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.multistep_dpm_solver_third_order_update
+ def multistep_dpm_solver_third_order_update(
+ self,
+ model_output_list: List[torch.Tensor],
+ *args,
+ sample: torch.Tensor = None,
+ **kwargs,
+ ) -> torch.Tensor:
+ """
+ One step for the third-order multistep DPMSolver.
+ Args:
+ model_output_list (`List[torch.Tensor]`):
+ The direct outputs from learned diffusion model at current and latter timesteps.
+ sample (`torch.Tensor`):
+ A current instance of a sample created by diffusion process.
+ Returns:
+ `torch.Tensor`:
+ The sample tensor at the previous timestep.
+ """
+
+ timestep_list = args[0] if len(args) > 0 else kwargs.pop(
+ "timestep_list", None)
+ prev_timestep = args[1] if len(args) > 1 else kwargs.pop(
+ "prev_timestep", None)
+ if sample is None:
+ if len(args) > 2:
+ sample = args[2]
+ else:
+ raise ValueError(
+ " missing`sample` as a required keyward argument")
+ if timestep_list is not None:
+ deprecate(
+ "timestep_list",
+ "1.0.0",
+ "Passing `timestep_list` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",
+ )
+
+ if prev_timestep is not None:
+ deprecate(
+ "prev_timestep",
+ "1.0.0",
+ "Passing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",
+ )
+
+ sigma_t, sigma_s0, sigma_s1, sigma_s2 = (
+ self.sigmas[self.step_index + 1], # pyright: ignore
+ self.sigmas[self.step_index],
+ self.sigmas[self.step_index - 1], # pyright: ignore
+ self.sigmas[self.step_index - 2], # pyright: ignore
+ )
+
+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma_t)
+ alpha_s0, sigma_s0 = self._sigma_to_alpha_sigma_t(sigma_s0)
+ alpha_s1, sigma_s1 = self._sigma_to_alpha_sigma_t(sigma_s1)
+ alpha_s2, sigma_s2 = self._sigma_to_alpha_sigma_t(sigma_s2)
+
+ lambda_t = torch.log(alpha_t) - torch.log(sigma_t)
+ lambda_s0 = torch.log(alpha_s0) - torch.log(sigma_s0)
+ lambda_s1 = torch.log(alpha_s1) - torch.log(sigma_s1)
+ lambda_s2 = torch.log(alpha_s2) - torch.log(sigma_s2)
+
+ m0, m1, m2 = model_output_list[-1], model_output_list[
+ -2], model_output_list[-3]
+
+ h, h_0, h_1 = lambda_t - lambda_s0, lambda_s0 - lambda_s1, lambda_s1 - lambda_s2
+ r0, r1 = h_0 / h, h_1 / h
+ D0 = m0
+ D1_0, D1_1 = (1.0 / r0) * (m0 - m1), (1.0 / r1) * (m1 - m2)
+ D1 = D1_0 + (r0 / (r0 + r1)) * (D1_0 - D1_1)
+ D2 = (1.0 / (r0 + r1)) * (D1_0 - D1_1)
+ if self.config.algorithm_type == "dpmsolver++":
+ # See https://arxiv.org/abs/2206.00927 for detailed derivations
+ x_t = ((sigma_t / sigma_s0) * sample -
+ (alpha_t * (torch.exp(-h) - 1.0)) * D0 +
+ (alpha_t * ((torch.exp(-h) - 1.0) / h + 1.0)) * D1 -
+ (alpha_t * ((torch.exp(-h) - 1.0 + h) / h**2 - 0.5)) * D2)
+ elif self.config.algorithm_type == "dpmsolver":
+ # See https://arxiv.org/abs/2206.00927 for detailed derivations
+ x_t = ((alpha_t / alpha_s0) * sample - (sigma_t *
+ (torch.exp(h) - 1.0)) * D0 -
+ (sigma_t * ((torch.exp(h) - 1.0) / h - 1.0)) * D1 -
+ (sigma_t * ((torch.exp(h) - 1.0 - h) / h**2 - 0.5)) * D2)
+ return x_t # pyright: ignore
+
+ def index_for_timestep(self, timestep, schedule_timesteps=None):
+ if schedule_timesteps is None:
+ schedule_timesteps = self.timesteps
+
+ indices = (schedule_timesteps == timestep).nonzero()
+
+ # The sigma index that is taken for the **very** first `step`
+ # is always the second index (or the last index if there is only 1)
+ # This way we can ensure we don't accidentally skip a sigma in
+ # case we start in the middle of the denoising schedule (e.g. for image-to-image)
+ pos = 1 if len(indices) > 1 else 0
+
+ return indices[pos].item()
+
+ def _init_step_index(self, timestep):
+ """
+ Initialize the step_index counter for the scheduler.
+ """
+
+ if self.begin_index is None:
+ if isinstance(timestep, torch.Tensor):
+ timestep = timestep.to(self.timesteps.device)
+ self._step_index = self.index_for_timestep(timestep)
+ else:
+ self._step_index = self._begin_index
+
+ # Modified from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.step
+ def step(
+ self,
+ model_output: torch.Tensor,
+ timestep: Union[int, torch.Tensor],
+ sample: torch.Tensor,
+ generator=None,
+ variance_noise: Optional[torch.Tensor] = None,
+ return_dict: bool = True,
+ ) -> Union[SchedulerOutput, Tuple]:
+ """
+ Predict the sample from the previous timestep by reversing the SDE. This function propagates the sample with
+ the multistep DPMSolver.
+ Args:
+ model_output (`torch.Tensor`):
+ The direct output from learned diffusion model.
+ timestep (`int`):
+ The current discrete timestep in the diffusion chain.
+ sample (`torch.Tensor`):
+ A current instance of a sample created by the diffusion process.
+ generator (`torch.Generator`, *optional*):
+ A random number generator.
+ variance_noise (`torch.Tensor`):
+ Alternative to generating noise with `generator` by directly providing the noise for the variance
+ itself. Useful for methods such as [`LEdits++`].
+ return_dict (`bool`):
+ Whether or not to return a [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`.
+ Returns:
+ [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`:
+ If return_dict is `True`, [`~schedulers.scheduling_utils.SchedulerOutput`] is returned, otherwise a
+ tuple is returned where the first element is the sample tensor.
+ """
+ if self.num_inference_steps is None:
+ raise ValueError(
+ "Number of inference steps is 'None', you need to run 'set_timesteps' after creating the scheduler"
+ )
+
+ if self.step_index is None:
+ self._init_step_index(timestep)
+
+ # Improve numerical stability for small number of steps
+ lower_order_final = (self.step_index == len(self.timesteps) - 1) and (
+ self.config.euler_at_final or
+ (self.config.lower_order_final and len(self.timesteps) < 15) or
+ self.config.final_sigmas_type == "zero")
+ lower_order_second = ((self.step_index == len(self.timesteps) - 2) and
+ self.config.lower_order_final and
+ len(self.timesteps) < 15)
+
+ model_output = self.convert_model_output(model_output, sample=sample)
+ for i in range(self.config.solver_order - 1):
+ self.model_outputs[i] = self.model_outputs[i + 1]
+ self.model_outputs[-1] = model_output
+
+ # Upcast to avoid precision issues when computing prev_sample
+ sample = sample.to(torch.float32)
+ if self.config.algorithm_type in ["sde-dpmsolver", "sde-dpmsolver++"
+ ] and variance_noise is None:
+ noise = randn_tensor(
+ model_output.shape,
+ generator=generator,
+ device=model_output.device,
+ dtype=torch.float32)
+ elif self.config.algorithm_type in ["sde-dpmsolver", "sde-dpmsolver++"]:
+ noise = variance_noise.to(
+ device=model_output.device,
+ dtype=torch.float32) # pyright: ignore
+ else:
+ noise = None
+
+ if self.config.solver_order == 1 or self.lower_order_nums < 1 or lower_order_final:
+ prev_sample = self.dpm_solver_first_order_update(
+ model_output, sample=sample, noise=noise)
+ elif self.config.solver_order == 2 or self.lower_order_nums < 2 or lower_order_second:
+ prev_sample = self.multistep_dpm_solver_second_order_update(
+ self.model_outputs, sample=sample, noise=noise)
+ else:
+ prev_sample = self.multistep_dpm_solver_third_order_update(
+ self.model_outputs, sample=sample)
+
+ if self.lower_order_nums < self.config.solver_order:
+ self.lower_order_nums += 1
+
+ # Cast sample back to expected dtype
+ prev_sample = prev_sample.to(model_output.dtype)
+
+ # upon completion increase step index by one
+ self._step_index += 1 # pyright: ignore
+
+ if not return_dict:
+ return (prev_sample,)
+
+ return SchedulerOutput(prev_sample=prev_sample)
+
+ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.scale_model_input
+ def scale_model_input(self, sample: torch.Tensor, *args,
+ **kwargs) -> torch.Tensor:
+ """
+ Ensures interchangeability with schedulers that need to scale the denoising model input depending on the
+ current timestep.
+ Args:
+ sample (`torch.Tensor`):
+ The input sample.
+ Returns:
+ `torch.Tensor`:
+ A scaled input sample.
+ """
+ return sample
+
+ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.scale_model_input
+ def add_noise(
+ self,
+ original_samples: torch.Tensor,
+ noise: torch.Tensor,
+ timesteps: torch.IntTensor,
+ ) -> torch.Tensor:
+ # Make sure sigmas and timesteps have the same device and dtype as original_samples
+ sigmas = self.sigmas.to(
+ device=original_samples.device, dtype=original_samples.dtype)
+ if original_samples.device.type == "mps" and torch.is_floating_point(
+ timesteps):
+ # mps does not support float64
+ schedule_timesteps = self.timesteps.to(
+ original_samples.device, dtype=torch.float32)
+ timesteps = timesteps.to(
+ original_samples.device, dtype=torch.float32)
+ else:
+ schedule_timesteps = self.timesteps.to(original_samples.device)
+ timesteps = timesteps.to(original_samples.device)
+
+ # begin_index is None when the scheduler is used for training or pipeline does not implement set_begin_index
+ if self.begin_index is None:
+ step_indices = [
+ self.index_for_timestep(t, schedule_timesteps)
+ for t in timesteps
+ ]
+ elif self.step_index is not None:
+ # add_noise is called after first denoising step (for inpainting)
+ step_indices = [self.step_index] * timesteps.shape[0]
+ else:
+ # add noise is called before first denoising step to create initial latent(img2img)
+ step_indices = [self.begin_index] * timesteps.shape[0]
+
+ sigma = sigmas[step_indices].flatten()
+ while len(sigma.shape) < len(original_samples.shape):
+ sigma = sigma.unsqueeze(-1)
+
+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma)
+ noisy_samples = alpha_t * original_samples + sigma_t * noise
+ return noisy_samples
+
+ def __len__(self):
+ return self.config.num_train_timesteps
diff --git a/worldplay2/schedulers/unipc.py b/worldplay2/schedulers/unipc.py
new file mode 100644
index 0000000..fb502f2
--- /dev/null
+++ b/worldplay2/schedulers/unipc.py
@@ -0,0 +1,802 @@
+# Copied from https://github.com/huggingface/diffusers/blob/v0.31.0/src/diffusers/schedulers/scheduling_unipc_multistep.py
+# Convert unipc for flow matching
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+
+import math
+from typing import List, Optional, Tuple, Union
+
+import numpy as np
+import torch
+from diffusers.configuration_utils import ConfigMixin, register_to_config
+from diffusers.schedulers.scheduling_utils import (
+ KarrasDiffusionSchedulers,
+ SchedulerMixin,
+ SchedulerOutput,
+)
+from diffusers.utils import deprecate, is_scipy_available
+
+if is_scipy_available():
+ import scipy.stats
+
+
+class FlowUniPCMultistepScheduler(SchedulerMixin, ConfigMixin):
+ """
+ `UniPCMultistepScheduler` is a training-free framework designed for the fast sampling of diffusion models.
+
+ This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic
+ methods the library implements for all schedulers such as loading and saving.
+
+ Args:
+ num_train_timesteps (`int`, defaults to 1000):
+ The number of diffusion steps to train the model.
+ solver_order (`int`, default `2`):
+ The UniPC order which can be any positive integer. The effective order of accuracy is `solver_order + 1`
+ due to the UniC. It is recommended to use `solver_order=2` for guided sampling, and `solver_order=3` for
+ unconditional sampling.
+ prediction_type (`str`, defaults to "flow_prediction"):
+ Prediction type of the scheduler function; must be `flow_prediction` for this scheduler, which predicts
+ the flow of the diffusion process.
+ thresholding (`bool`, defaults to `False`):
+ Whether to use the "dynamic thresholding" method. This is unsuitable for latent-space diffusion models such
+ as Stable Diffusion.
+ dynamic_thresholding_ratio (`float`, defaults to 0.995):
+ The ratio for the dynamic thresholding method. Valid only when `thresholding=True`.
+ sample_max_value (`float`, defaults to 1.0):
+ The threshold value for dynamic thresholding. Valid only when `thresholding=True` and `predict_x0=True`.
+ predict_x0 (`bool`, defaults to `True`):
+ Whether to use the updating algorithm on the predicted x0.
+ solver_type (`str`, default `bh2`):
+ Solver type for UniPC. It is recommended to use `bh1` for unconditional sampling when steps < 10, and `bh2`
+ otherwise.
+ lower_order_final (`bool`, default `True`):
+ Whether to use lower-order solvers in the final steps. Only valid for < 15 inference steps. This can
+ stabilize the sampling of DPMSolver for steps < 15, especially for steps <= 10.
+ disable_corrector (`list`, default `[]`):
+ Decides which step to disable the corrector to mitigate the misalignment between `epsilon_theta(x_t, c)`
+ and `epsilon_theta(x_t^c, c)` which can influence convergence for a large guidance scale. Corrector is
+ usually disabled during the first few steps.
+ solver_p (`SchedulerMixin`, default `None`):
+ Any other scheduler that if specified, the algorithm becomes `solver_p + UniC`.
+ use_karras_sigmas (`bool`, *optional*, defaults to `False`):
+ Whether to use Karras sigmas for step sizes in the noise schedule during the sampling process. If `True`,
+ the sigmas are determined according to a sequence of noise levels {σi}.
+ use_exponential_sigmas (`bool`, *optional*, defaults to `False`):
+ Whether to use exponential sigmas for step sizes in the noise schedule during the sampling process.
+ timestep_spacing (`str`, defaults to `"linspace"`):
+ The way the timesteps should be scaled. Refer to Table 2 of the [Common Diffusion Noise Schedules and
+ Sample Steps are Flawed](https://huggingface.co/papers/2305.08891) for more information.
+ steps_offset (`int`, defaults to 0):
+ An offset added to the inference steps, as required by some model families.
+ final_sigmas_type (`str`, defaults to `"zero"`):
+ The final `sigma` value for the noise schedule during the sampling process. If `"sigma_min"`, the final
+ sigma is the same as the last sigma in the training schedule. If `zero`, the final sigma is set to 0.
+ """
+
+ _compatibles = [e.name for e in KarrasDiffusionSchedulers]
+ order = 1
+
+ @register_to_config
+ def __init__(
+ self,
+ num_train_timesteps: int = 1000,
+ solver_order: int = 2,
+ prediction_type: str = "flow_prediction",
+ shift: Optional[float] = 1.0,
+ use_dynamic_shifting=False,
+ thresholding: bool = False,
+ dynamic_thresholding_ratio: float = 0.995,
+ sample_max_value: float = 1.0,
+ predict_x0: bool = True,
+ solver_type: str = "bh2",
+ lower_order_final: bool = True,
+ disable_corrector: List[int] = [],
+ solver_p: SchedulerMixin = None,
+ timestep_spacing: str = "linspace",
+ steps_offset: int = 0,
+ final_sigmas_type: Optional[str] = "zero", # "zero", "sigma_min"
+ ):
+
+ if solver_type not in ["bh1", "bh2"]:
+ if solver_type in ["midpoint", "heun", "logrho"]:
+ self.register_to_config(solver_type="bh2")
+ else:
+ raise NotImplementedError(
+ f"{solver_type} is not implemented for {self.__class__}")
+
+ self.predict_x0 = predict_x0
+ # setable values
+ self.num_inference_steps = None
+ alphas = np.linspace(1, 1 / num_train_timesteps,
+ num_train_timesteps)[::-1].copy()
+ sigmas = 1.0 - alphas
+ sigmas = torch.from_numpy(sigmas).to(dtype=torch.float32)
+
+ if not use_dynamic_shifting:
+ # when use_dynamic_shifting is True, we apply the timestep shifting on the fly based on the image resolution
+ sigmas = shift * sigmas / (1 +
+ (shift - 1) * sigmas) # pyright: ignore
+
+ self.sigmas = sigmas
+ self.timesteps = sigmas * num_train_timesteps
+
+ self.model_outputs = [None] * solver_order
+ self.timestep_list = [None] * solver_order
+ self.lower_order_nums = 0
+ self.disable_corrector = disable_corrector
+ self.solver_p = solver_p
+ self.last_sample = None
+ self._step_index = None
+ self._begin_index = None
+
+ self.sigmas = self.sigmas.to(
+ "cpu") # to avoid too much CPU/GPU communication
+ self.sigma_min = self.sigmas[-1].item()
+ self.sigma_max = self.sigmas[0].item()
+
+ @property
+ def step_index(self):
+ """
+ The index counter for current timestep. It will increase 1 after each scheduler step.
+ """
+ return self._step_index
+
+ @property
+ def begin_index(self):
+ """
+ The index for the first timestep. It should be set from pipeline with `set_begin_index` method.
+ """
+ return self._begin_index
+
+ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.set_begin_index
+ def set_begin_index(self, begin_index: int = 0):
+ """
+ Sets the begin index for the scheduler. This function should be run from pipeline before the inference.
+
+ Args:
+ begin_index (`int`):
+ The begin index for the scheduler.
+ """
+ self._begin_index = begin_index
+
+ # Modified from diffusers.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler.set_timesteps
+ def set_timesteps(
+ self,
+ num_inference_steps: Union[int, None] = None,
+ device: Union[str, torch.device] = None,
+ sigmas: Optional[List[float]] = None,
+ mu: Optional[Union[float, None]] = None,
+ shift: Optional[Union[float, None]] = None,
+ ):
+ """
+ Sets the discrete timesteps used for the diffusion chain (to be run before inference).
+ Args:
+ num_inference_steps (`int`):
+ Total number of the spacing of the time steps.
+ device (`str` or `torch.device`, *optional*):
+ The device to which the timesteps should be moved to. If `None`, the timesteps are not moved.
+ """
+
+ if self.config.use_dynamic_shifting and mu is None:
+ raise ValueError(
+ " you have to pass a value for `mu` when `use_dynamic_shifting` is set to be `True`"
+ )
+
+ if sigmas is None:
+ sigmas = np.linspace(self.sigma_max, self.sigma_min,
+ num_inference_steps +
+ 1).copy()[:-1] # pyright: ignore
+
+ if self.config.use_dynamic_shifting:
+ sigmas = self.time_shift(mu, 1.0, sigmas) # pyright: ignore
+ else:
+ if shift is None:
+ shift = self.config.shift
+ sigmas = shift * sigmas / (1 +
+ (shift - 1) * sigmas) # pyright: ignore
+
+ if self.config.final_sigmas_type == "sigma_min":
+ sigma_last = ((1 - self.alphas_cumprod[0]) /
+ self.alphas_cumprod[0])**0.5
+ elif self.config.final_sigmas_type == "zero":
+ sigma_last = 0
+ else:
+ raise ValueError(
+ f"`final_sigmas_type` must be one of 'zero', or 'sigma_min', but got {self.config.final_sigmas_type}"
+ )
+
+ timesteps = sigmas * self.config.num_train_timesteps
+ sigmas = np.concatenate([sigmas, [sigma_last]
+ ]).astype(np.float32) # pyright: ignore
+
+ self.sigmas = torch.from_numpy(sigmas)
+ self.timesteps = torch.from_numpy(timesteps).to(
+ device=device, dtype=torch.int64)
+
+ self.num_inference_steps = len(timesteps)
+
+ self.model_outputs = [
+ None,
+ ] * self.config.solver_order
+ self.lower_order_nums = 0
+ self.last_sample = None
+ if self.solver_p:
+ self.solver_p.set_timesteps(self.num_inference_steps, device=device)
+
+ # add an index counter for schedulers that allow duplicated timesteps
+ self._step_index = None
+ self._begin_index = None
+ self.sigmas = self.sigmas.to(
+ "cpu") # to avoid too much CPU/GPU communication
+
+ # Copied from diffusers.schedulers.scheduling_ddpm.DDPMScheduler._threshold_sample
+ def _threshold_sample(self, sample: torch.Tensor) -> torch.Tensor:
+ """
+ "Dynamic thresholding: At each sampling step we set s to a certain percentile absolute pixel value in xt0 (the
+ prediction of x_0 at timestep t), and if s > 1, then we threshold xt0 to the range [-s, s] and then divide by
+ s. Dynamic thresholding pushes saturated pixels (those near -1 and 1) inwards, thereby actively preventing
+ pixels from saturation at each step. We find that dynamic thresholding results in significantly better
+ photorealism as well as better image-text alignment, especially when using very large guidance weights."
+
+ https://arxiv.org/abs/2205.11487
+ """
+ dtype = sample.dtype
+ batch_size, channels, *remaining_dims = sample.shape
+
+ if dtype not in (torch.float32, torch.float64):
+ sample = sample.float(
+ ) # upcast for quantile calculation, and clamp not implemented for cpu half
+
+ # Flatten sample for doing quantile calculation along each image
+ sample = sample.reshape(batch_size, channels * np.prod(remaining_dims))
+
+ abs_sample = sample.abs() # "a certain percentile absolute pixel value"
+
+ s = torch.quantile(
+ abs_sample, self.config.dynamic_thresholding_ratio, dim=1)
+ s = torch.clamp(
+ s, min=1, max=self.config.sample_max_value
+ ) # When clamped to min=1, equivalent to standard clipping to [-1, 1]
+ s = s.unsqueeze(
+ 1) # (batch_size, 1) because clamp will broadcast along dim=0
+ sample = torch.clamp(
+ sample, -s, s
+ ) / s # "we threshold xt0 to the range [-s, s] and then divide by s"
+
+ sample = sample.reshape(batch_size, channels, *remaining_dims)
+ sample = sample.to(dtype)
+
+ return sample
+
+ # Copied from diffusers.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler._sigma_to_t
+ def _sigma_to_t(self, sigma):
+ return sigma * self.config.num_train_timesteps
+
+ def _sigma_to_alpha_sigma_t(self, sigma):
+ return 1 - sigma, sigma
+
+ # Copied from diffusers.schedulers.scheduling_flow_match_euler_discrete.set_timesteps
+ def time_shift(self, mu: float, sigma: float, t: torch.Tensor):
+ return math.exp(mu) / (math.exp(mu) + (1 / t - 1)**sigma)
+
+ def convert_model_output(
+ self,
+ model_output: torch.Tensor,
+ *args,
+ sample: torch.Tensor = None,
+ **kwargs,
+ ) -> torch.Tensor:
+ r"""
+ Convert the model output to the corresponding type the UniPC algorithm needs.
+
+ Args:
+ model_output (`torch.Tensor`):
+ The direct output from the learned diffusion model.
+ timestep (`int`):
+ The current discrete timestep in the diffusion chain.
+ sample (`torch.Tensor`):
+ A current instance of a sample created by the diffusion process.
+
+ Returns:
+ `torch.Tensor`:
+ The converted model output.
+ """
+ timestep = args[0] if len(args) > 0 else kwargs.pop("timestep", None)
+ if sample is None:
+ if len(args) > 1:
+ sample = args[1]
+ else:
+ raise ValueError(
+ "missing `sample` as a required keyward argument")
+ if timestep is not None:
+ deprecate(
+ "timesteps",
+ "1.0.0",
+ "Passing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",
+ )
+
+ sigma = self.sigmas[self.step_index]
+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma)
+
+ if self.predict_x0:
+ if self.config.prediction_type == "flow_prediction":
+ sigma_t = self.sigmas[self.step_index]
+ x0_pred = sample - sigma_t * model_output
+ else:
+ raise ValueError(
+ f"prediction_type given as {self.config.prediction_type} must be one of `epsilon`, `sample`,"
+ " `v_prediction` or `flow_prediction` for the UniPCMultistepScheduler."
+ )
+
+ if self.config.thresholding:
+ x0_pred = self._threshold_sample(x0_pred)
+
+ return x0_pred
+ else:
+ if self.config.prediction_type == "flow_prediction":
+ sigma_t = self.sigmas[self.step_index]
+ epsilon = sample - (1 - sigma_t) * model_output
+ else:
+ raise ValueError(
+ f"prediction_type given as {self.config.prediction_type} must be one of `epsilon`, `sample`,"
+ " `v_prediction` or `flow_prediction` for the UniPCMultistepScheduler."
+ )
+
+ if self.config.thresholding:
+ sigma_t = self.sigmas[self.step_index]
+ x0_pred = sample - sigma_t * model_output
+ x0_pred = self._threshold_sample(x0_pred)
+ epsilon = model_output + x0_pred
+
+ return epsilon
+
+ def multistep_uni_p_bh_update(
+ self,
+ model_output: torch.Tensor,
+ *args,
+ sample: torch.Tensor = None,
+ order: int = None, # pyright: ignore
+ **kwargs,
+ ) -> torch.Tensor:
+ """
+ One step for the UniP (B(h) version). Alternatively, `self.solver_p` is used if is specified.
+
+ Args:
+ model_output (`torch.Tensor`):
+ The direct output from the learned diffusion model at the current timestep.
+ prev_timestep (`int`):
+ The previous discrete timestep in the diffusion chain.
+ sample (`torch.Tensor`):
+ A current instance of a sample created by the diffusion process.
+ order (`int`):
+ The order of UniP at this timestep (corresponds to the *p* in UniPC-p).
+
+ Returns:
+ `torch.Tensor`:
+ The sample tensor at the previous timestep.
+ """
+ prev_timestep = args[0] if len(args) > 0 else kwargs.pop(
+ "prev_timestep", None)
+ if sample is None:
+ if len(args) > 1:
+ sample = args[1]
+ else:
+ raise ValueError(
+ " missing `sample` as a required keyward argument")
+ if order is None:
+ if len(args) > 2:
+ order = args[2]
+ else:
+ raise ValueError(
+ " missing `order` as a required keyward argument")
+ if prev_timestep is not None:
+ deprecate(
+ "prev_timestep",
+ "1.0.0",
+ "Passing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",
+ )
+ model_output_list = self.model_outputs
+
+ s0 = self.timestep_list[-1]
+ m0 = model_output_list[-1]
+ x = sample
+
+ if self.solver_p:
+ x_t = self.solver_p.step(model_output, s0, x).prev_sample
+ return x_t
+
+ sigma_t, sigma_s0 = self.sigmas[self.step_index + 1], self.sigmas[
+ self.step_index] # pyright: ignore
+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma_t)
+ alpha_s0, sigma_s0 = self._sigma_to_alpha_sigma_t(sigma_s0)
+
+ lambda_t = torch.log(alpha_t) - torch.log(sigma_t)
+ lambda_s0 = torch.log(alpha_s0) - torch.log(sigma_s0)
+
+ h = lambda_t - lambda_s0
+ device = sample.device
+
+ rks = []
+ D1s = []
+ for i in range(1, order):
+ si = self.step_index - i # pyright: ignore
+ mi = model_output_list[-(i + 1)]
+ alpha_si, sigma_si = self._sigma_to_alpha_sigma_t(self.sigmas[si])
+ lambda_si = torch.log(alpha_si) - torch.log(sigma_si)
+ rk = (lambda_si - lambda_s0) / h
+ rks.append(rk)
+ D1s.append((mi - m0) / rk) # pyright: ignore
+
+ rks.append(1.0)
+ rks = torch.tensor(rks, device=device)
+
+ R = []
+ b = []
+
+ hh = -h if self.predict_x0 else h
+ h_phi_1 = torch.expm1(hh) # h\phi_1(h) = e^h - 1
+ h_phi_k = h_phi_1 / hh - 1
+
+ factorial_i = 1
+
+ if self.config.solver_type == "bh1":
+ B_h = hh
+ elif self.config.solver_type == "bh2":
+ B_h = torch.expm1(hh)
+ else:
+ raise NotImplementedError()
+
+ for i in range(1, order + 1):
+ R.append(torch.pow(rks, i - 1))
+ b.append(h_phi_k * factorial_i / B_h)
+ factorial_i *= i + 1
+ h_phi_k = h_phi_k / hh - 1 / factorial_i
+
+ R = torch.stack(R)
+ b = torch.tensor(b, device=device)
+
+ if len(D1s) > 0:
+ D1s = torch.stack(D1s, dim=1) # (B, K)
+ # for order 2, we use a simplified version
+ if order == 2:
+ rhos_p = torch.tensor([0.5], dtype=x.dtype, device=device)
+ else:
+ rhos_p = torch.linalg.solve(R[:-1, :-1],
+ b[:-1]).to(device).to(x.dtype)
+ else:
+ D1s = None
+
+ if self.predict_x0:
+ x_t_ = sigma_t / sigma_s0 * x - alpha_t * h_phi_1 * m0
+ if D1s is not None:
+ pred_res = torch.einsum("k,bkc...->bc...", rhos_p,
+ D1s) # pyright: ignore
+ else:
+ pred_res = 0
+ x_t = x_t_ - alpha_t * B_h * pred_res
+ else:
+ x_t_ = alpha_t / alpha_s0 * x - sigma_t * h_phi_1 * m0
+ if D1s is not None:
+ pred_res = torch.einsum("k,bkc...->bc...", rhos_p,
+ D1s) # pyright: ignore
+ else:
+ pred_res = 0
+ x_t = x_t_ - sigma_t * B_h * pred_res
+
+ x_t = x_t.to(x.dtype)
+ return x_t
+
+ def multistep_uni_c_bh_update(
+ self,
+ this_model_output: torch.Tensor,
+ *args,
+ last_sample: torch.Tensor = None,
+ this_sample: torch.Tensor = None,
+ order: int = None, # pyright: ignore
+ **kwargs,
+ ) -> torch.Tensor:
+ """
+ One step for the UniC (B(h) version).
+
+ Args:
+ this_model_output (`torch.Tensor`):
+ The model outputs at `x_t`.
+ this_timestep (`int`):
+ The current timestep `t`.
+ last_sample (`torch.Tensor`):
+ The generated sample before the last predictor `x_{t-1}`.
+ this_sample (`torch.Tensor`):
+ The generated sample after the last predictor `x_{t}`.
+ order (`int`):
+ The `p` of UniC-p at this step. The effective order of accuracy should be `order + 1`.
+
+ Returns:
+ `torch.Tensor`:
+ The corrected sample tensor at the current timestep.
+ """
+ this_timestep = args[0] if len(args) > 0 else kwargs.pop(
+ "this_timestep", None)
+ if last_sample is None:
+ if len(args) > 1:
+ last_sample = args[1]
+ else:
+ raise ValueError(
+ " missing`last_sample` as a required keyward argument")
+ if this_sample is None:
+ if len(args) > 2:
+ this_sample = args[2]
+ else:
+ raise ValueError(
+ " missing`this_sample` as a required keyward argument")
+ if order is None:
+ if len(args) > 3:
+ order = args[3]
+ else:
+ raise ValueError(
+ " missing`order` as a required keyward argument")
+ if this_timestep is not None:
+ deprecate(
+ "this_timestep",
+ "1.0.0",
+ "Passing `this_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",
+ )
+
+ model_output_list = self.model_outputs
+
+ m0 = model_output_list[-1]
+ x = last_sample
+ x_t = this_sample
+ model_t = this_model_output
+
+ sigma_t, sigma_s0 = self.sigmas[self.step_index], self.sigmas[
+ self.step_index - 1] # pyright: ignore
+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma_t)
+ alpha_s0, sigma_s0 = self._sigma_to_alpha_sigma_t(sigma_s0)
+
+ lambda_t = torch.log(alpha_t) - torch.log(sigma_t)
+ lambda_s0 = torch.log(alpha_s0) - torch.log(sigma_s0)
+
+ h = lambda_t - lambda_s0
+ device = this_sample.device
+
+ rks = []
+ D1s = []
+ for i in range(1, order):
+ si = self.step_index - (i + 1) # pyright: ignore
+ mi = model_output_list[-(i + 1)]
+ alpha_si, sigma_si = self._sigma_to_alpha_sigma_t(self.sigmas[si])
+ lambda_si = torch.log(alpha_si) - torch.log(sigma_si)
+ rk = (lambda_si - lambda_s0) / h
+ rks.append(rk)
+ D1s.append((mi - m0) / rk) # pyright: ignore
+
+ rks.append(1.0)
+ rks = torch.tensor(rks, device=device)
+
+ R = []
+ b = []
+
+ hh = -h if self.predict_x0 else h
+ h_phi_1 = torch.expm1(hh) # h\phi_1(h) = e^h - 1
+ h_phi_k = h_phi_1 / hh - 1
+
+ factorial_i = 1
+
+ if self.config.solver_type == "bh1":
+ B_h = hh
+ elif self.config.solver_type == "bh2":
+ B_h = torch.expm1(hh)
+ else:
+ raise NotImplementedError()
+
+ for i in range(1, order + 1):
+ R.append(torch.pow(rks, i - 1))
+ b.append(h_phi_k * factorial_i / B_h)
+ factorial_i *= i + 1
+ h_phi_k = h_phi_k / hh - 1 / factorial_i
+
+ R = torch.stack(R)
+ b = torch.tensor(b, device=device)
+
+ if len(D1s) > 0:
+ D1s = torch.stack(D1s, dim=1)
+ else:
+ D1s = None
+
+ # for order 1, we use a simplified version
+ if order == 1:
+ rhos_c = torch.tensor([0.5], dtype=x.dtype, device=device)
+ else:
+ rhos_c = torch.linalg.solve(R, b).to(device).to(x.dtype)
+
+ if self.predict_x0:
+ x_t_ = sigma_t / sigma_s0 * x - alpha_t * h_phi_1 * m0
+ if D1s is not None:
+ corr_res = torch.einsum("k,bkc...->bc...", rhos_c[:-1], D1s)
+ else:
+ corr_res = 0
+ D1_t = model_t - m0
+ x_t = x_t_ - alpha_t * B_h * (corr_res + rhos_c[-1] * D1_t)
+ else:
+ x_t_ = alpha_t / alpha_s0 * x - sigma_t * h_phi_1 * m0
+ if D1s is not None:
+ corr_res = torch.einsum("k,bkc...->bc...", rhos_c[:-1], D1s)
+ else:
+ corr_res = 0
+ D1_t = model_t - m0
+ x_t = x_t_ - sigma_t * B_h * (corr_res + rhos_c[-1] * D1_t)
+ x_t = x_t.to(x.dtype)
+ return x_t
+
+ def index_for_timestep(self, timestep, schedule_timesteps=None):
+ if schedule_timesteps is None:
+ schedule_timesteps = self.timesteps
+
+ indices = (schedule_timesteps == timestep).nonzero()
+
+ # The sigma index that is taken for the **very** first `step`
+ # is always the second index (or the last index if there is only 1)
+ # This way we can ensure we don't accidentally skip a sigma in
+ # case we start in the middle of the denoising schedule (e.g. for image-to-image)
+ pos = 1 if len(indices) > 1 else 0
+
+ return indices[pos].item()
+
+ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler._init_step_index
+ def _init_step_index(self, timestep):
+ """
+ Initialize the step_index counter for the scheduler.
+ """
+
+ if self.begin_index is None:
+ if isinstance(timestep, torch.Tensor):
+ timestep = timestep.to(self.timesteps.device)
+ self._step_index = self.index_for_timestep(timestep)
+ else:
+ self._step_index = self._begin_index
+
+ def step(self,
+ model_output: torch.Tensor,
+ timestep: Union[int, torch.Tensor],
+ sample: torch.Tensor,
+ return_dict: bool = True,
+ generator=None) -> Union[SchedulerOutput, Tuple]:
+ """
+ Predict the sample from the previous timestep by reversing the SDE. This function propagates the sample with
+ the multistep UniPC.
+
+ Args:
+ model_output (`torch.Tensor`):
+ The direct output from learned diffusion model.
+ timestep (`int`):
+ The current discrete timestep in the diffusion chain.
+ sample (`torch.Tensor`):
+ A current instance of a sample created by the diffusion process.
+ return_dict (`bool`):
+ Whether or not to return a [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`.
+
+ Returns:
+ [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`:
+ If return_dict is `True`, [`~schedulers.scheduling_utils.SchedulerOutput`] is returned, otherwise a
+ tuple is returned where the first element is the sample tensor.
+
+ """
+ if self.num_inference_steps is None:
+ raise ValueError(
+ "Number of inference steps is 'None', you need to run 'set_timesteps' after creating the scheduler"
+ )
+
+ if self.step_index is None:
+ self._init_step_index(timestep)
+
+ use_corrector = (
+ self.step_index > 0 and
+ self.step_index - 1 not in self.disable_corrector and
+ self.last_sample is not None # pyright: ignore
+ )
+
+ model_output_convert = self.convert_model_output(
+ model_output, sample=sample)
+ if use_corrector:
+ sample = self.multistep_uni_c_bh_update(
+ this_model_output=model_output_convert,
+ last_sample=self.last_sample,
+ this_sample=sample,
+ order=self.this_order,
+ )
+
+ for i in range(self.config.solver_order - 1):
+ self.model_outputs[i] = self.model_outputs[i + 1]
+ self.timestep_list[i] = self.timestep_list[i + 1]
+
+ self.model_outputs[-1] = model_output_convert
+ self.timestep_list[-1] = timestep # pyright: ignore
+
+ if self.config.lower_order_final:
+ this_order = min(self.config.solver_order,
+ len(self.timesteps) -
+ self.step_index) # pyright: ignore
+ else:
+ this_order = self.config.solver_order
+
+ self.this_order = min(this_order,
+ self.lower_order_nums + 1) # warmup for multistep
+ assert self.this_order > 0
+
+ self.last_sample = sample
+ prev_sample = self.multistep_uni_p_bh_update(
+ model_output=model_output, # pass the original non-converted model output, in case solver-p is used
+ sample=sample,
+ order=self.this_order,
+ )
+
+ if self.lower_order_nums < self.config.solver_order:
+ self.lower_order_nums += 1
+
+ # upon completion increase step index by one
+ self._step_index += 1 # pyright: ignore
+
+ if not return_dict:
+ return (prev_sample,)
+
+ return SchedulerOutput(prev_sample=prev_sample)
+
+ def scale_model_input(self, sample: torch.Tensor, *args,
+ **kwargs) -> torch.Tensor:
+ """
+ Ensures interchangeability with schedulers that need to scale the denoising model input depending on the
+ current timestep.
+
+ Args:
+ sample (`torch.Tensor`):
+ The input sample.
+
+ Returns:
+ `torch.Tensor`:
+ A scaled input sample.
+ """
+ return sample
+
+ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.add_noise
+ def add_noise(
+ self,
+ original_samples: torch.Tensor,
+ noise: torch.Tensor,
+ timesteps: torch.IntTensor,
+ ) -> torch.Tensor:
+ # Make sure sigmas and timesteps have the same device and dtype as original_samples
+ sigmas = self.sigmas.to(
+ device=original_samples.device, dtype=original_samples.dtype)
+ if original_samples.device.type == "mps" and torch.is_floating_point(
+ timesteps):
+ # mps does not support float64
+ schedule_timesteps = self.timesteps.to(
+ original_samples.device, dtype=torch.float32)
+ timesteps = timesteps.to(
+ original_samples.device, dtype=torch.float32)
+ else:
+ schedule_timesteps = self.timesteps.to(original_samples.device)
+ timesteps = timesteps.to(original_samples.device)
+
+ # begin_index is None when the scheduler is used for training or pipeline does not implement set_begin_index
+ if self.begin_index is None:
+ step_indices = [
+ self.index_for_timestep(t, schedule_timesteps)
+ for t in timesteps
+ ]
+ elif self.step_index is not None:
+ # add_noise is called after first denoising step (for inpainting)
+ step_indices = [self.step_index] * timesteps.shape[0]
+ else:
+ # add noise is called before first denoising step to create initial latent(img2img)
+ step_indices = [self.begin_index] * timesteps.shape[0]
+
+ sigma = sigmas[step_indices].flatten()
+ while len(sigma.shape) < len(original_samples.shape):
+ sigma = sigma.unsqueeze(-1)
+
+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma)
+ noisy_samples = alpha_t * original_samples + sigma_t * noise
+ return noisy_samples
+
+ def __len__(self):
+ return self.config.num_train_timesteps
diff --git a/worldplay2/text/__init__.py b/worldplay2/text/__init__.py
new file mode 100644
index 0000000..d6e636a
--- /dev/null
+++ b/worldplay2/text/__init__.py
@@ -0,0 +1,3 @@
+from .encoder import T5EncoderModel
+
+__all__ = ["T5EncoderModel"]
diff --git a/worldplay2/text/encoder.py b/worldplay2/text/encoder.py
new file mode 100644
index 0000000..6bc85eb
--- /dev/null
+++ b/worldplay2/text/encoder.py
@@ -0,0 +1,513 @@
+# Modified from transformers.models.t5.modeling_t5
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+import logging
+import math
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from .tokenizer import HuggingfaceTokenizer
+
+__all__ = [
+ 'T5Model',
+ 'T5Encoder',
+ 'T5Decoder',
+ 'T5EncoderModel',
+]
+
+
+def fp16_clamp(x):
+ if x.dtype == torch.float16 and torch.isinf(x).any():
+ clamp = torch.finfo(x.dtype).max - 1000
+ x = torch.clamp(x, min=-clamp, max=clamp)
+ return x
+
+
+def init_weights(m):
+ if isinstance(m, T5LayerNorm):
+ nn.init.ones_(m.weight)
+ elif isinstance(m, T5Model):
+ nn.init.normal_(m.token_embedding.weight, std=1.0)
+ elif isinstance(m, T5FeedForward):
+ nn.init.normal_(m.gate[0].weight, std=m.dim**-0.5)
+ nn.init.normal_(m.fc1.weight, std=m.dim**-0.5)
+ nn.init.normal_(m.fc2.weight, std=m.dim_ffn**-0.5)
+ elif isinstance(m, T5Attention):
+ nn.init.normal_(m.q.weight, std=(m.dim * m.dim_attn)**-0.5)
+ nn.init.normal_(m.k.weight, std=m.dim**-0.5)
+ nn.init.normal_(m.v.weight, std=m.dim**-0.5)
+ nn.init.normal_(m.o.weight, std=(m.num_heads * m.dim_attn)**-0.5)
+ elif isinstance(m, T5RelativeEmbedding):
+ nn.init.normal_(
+ m.embedding.weight, std=(2 * m.num_buckets * m.num_heads)**-0.5)
+
+
+class GELU(nn.Module):
+
+ def forward(self, x):
+ return 0.5 * x * (1.0 + torch.tanh(
+ math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))
+
+
+class T5LayerNorm(nn.Module):
+
+ def __init__(self, dim, eps=1e-6):
+ super(T5LayerNorm, self).__init__()
+ self.dim = dim
+ self.eps = eps
+ self.weight = nn.Parameter(torch.ones(dim))
+
+ def forward(self, x):
+ x = x * torch.rsqrt(x.float().pow(2).mean(dim=-1, keepdim=True) +
+ self.eps)
+ if self.weight.dtype in [torch.float16, torch.bfloat16]:
+ x = x.type_as(self.weight)
+ return self.weight * x
+
+
+class T5Attention(nn.Module):
+
+ def __init__(self, dim, dim_attn, num_heads, dropout=0.1):
+ assert dim_attn % num_heads == 0
+ super(T5Attention, self).__init__()
+ self.dim = dim
+ self.dim_attn = dim_attn
+ self.num_heads = num_heads
+ self.head_dim = dim_attn // num_heads
+
+ # layers
+ self.q = nn.Linear(dim, dim_attn, bias=False)
+ self.k = nn.Linear(dim, dim_attn, bias=False)
+ self.v = nn.Linear(dim, dim_attn, bias=False)
+ self.o = nn.Linear(dim_attn, dim, bias=False)
+ self.dropout = nn.Dropout(dropout)
+
+ def forward(self, x, context=None, mask=None, pos_bias=None):
+ """
+ x: [B, L1, C].
+ context: [B, L2, C] or None.
+ mask: [B, L2] or [B, L1, L2] or None.
+ """
+ # check inputs
+ context = x if context is None else context
+ b, n, c = x.size(0), self.num_heads, self.head_dim
+
+ # compute query, key, value
+ q = self.q(x).view(b, -1, n, c)
+ k = self.k(context).view(b, -1, n, c)
+ v = self.v(context).view(b, -1, n, c)
+
+ # attention bias
+ attn_bias = x.new_zeros(b, n, q.size(1), k.size(1))
+ if pos_bias is not None:
+ attn_bias += pos_bias
+ if mask is not None:
+ assert mask.ndim in [2, 3]
+ mask = mask.view(b, 1, 1,
+ -1) if mask.ndim == 2 else mask.unsqueeze(1)
+ attn_bias.masked_fill_(mask == 0, torch.finfo(x.dtype).min)
+
+ # compute attention (T5 does not use scaling)
+ attn = torch.einsum('binc,bjnc->bnij', q, k) + attn_bias
+ attn = F.softmax(attn.float(), dim=-1).type_as(attn)
+ x = torch.einsum('bnij,bjnc->binc', attn, v)
+
+ # output
+ x = x.reshape(b, -1, n * c)
+ x = self.o(x)
+ x = self.dropout(x)
+ return x
+
+
+class T5FeedForward(nn.Module):
+
+ def __init__(self, dim, dim_ffn, dropout=0.1):
+ super(T5FeedForward, self).__init__()
+ self.dim = dim
+ self.dim_ffn = dim_ffn
+
+ # layers
+ self.gate = nn.Sequential(nn.Linear(dim, dim_ffn, bias=False), GELU())
+ self.fc1 = nn.Linear(dim, dim_ffn, bias=False)
+ self.fc2 = nn.Linear(dim_ffn, dim, bias=False)
+ self.dropout = nn.Dropout(dropout)
+
+ def forward(self, x):
+ x = self.fc1(x) * self.gate(x)
+ x = self.dropout(x)
+ x = self.fc2(x)
+ x = self.dropout(x)
+ return x
+
+
+class T5SelfAttention(nn.Module):
+
+ def __init__(self,
+ dim,
+ dim_attn,
+ dim_ffn,
+ num_heads,
+ num_buckets,
+ shared_pos=True,
+ dropout=0.1):
+ super(T5SelfAttention, self).__init__()
+ self.dim = dim
+ self.dim_attn = dim_attn
+ self.dim_ffn = dim_ffn
+ self.num_heads = num_heads
+ self.num_buckets = num_buckets
+ self.shared_pos = shared_pos
+
+ # layers
+ self.norm1 = T5LayerNorm(dim)
+ self.attn = T5Attention(dim, dim_attn, num_heads, dropout)
+ self.norm2 = T5LayerNorm(dim)
+ self.ffn = T5FeedForward(dim, dim_ffn, dropout)
+ self.pos_embedding = None if shared_pos else T5RelativeEmbedding(
+ num_buckets, num_heads, bidirectional=True)
+
+ def forward(self, x, mask=None, pos_bias=None):
+ e = pos_bias if self.shared_pos else self.pos_embedding(
+ x.size(1), x.size(1))
+ x = fp16_clamp(x + self.attn(self.norm1(x), mask=mask, pos_bias=e))
+ x = fp16_clamp(x + self.ffn(self.norm2(x)))
+ return x
+
+
+class T5CrossAttention(nn.Module):
+
+ def __init__(self,
+ dim,
+ dim_attn,
+ dim_ffn,
+ num_heads,
+ num_buckets,
+ shared_pos=True,
+ dropout=0.1):
+ super(T5CrossAttention, self).__init__()
+ self.dim = dim
+ self.dim_attn = dim_attn
+ self.dim_ffn = dim_ffn
+ self.num_heads = num_heads
+ self.num_buckets = num_buckets
+ self.shared_pos = shared_pos
+
+ # layers
+ self.norm1 = T5LayerNorm(dim)
+ self.self_attn = T5Attention(dim, dim_attn, num_heads, dropout)
+ self.norm2 = T5LayerNorm(dim)
+ self.cross_attn = T5Attention(dim, dim_attn, num_heads, dropout)
+ self.norm3 = T5LayerNorm(dim)
+ self.ffn = T5FeedForward(dim, dim_ffn, dropout)
+ self.pos_embedding = None if shared_pos else T5RelativeEmbedding(
+ num_buckets, num_heads, bidirectional=False)
+
+ def forward(self,
+ x,
+ mask=None,
+ encoder_states=None,
+ encoder_mask=None,
+ pos_bias=None):
+ e = pos_bias if self.shared_pos else self.pos_embedding(
+ x.size(1), x.size(1))
+ x = fp16_clamp(x + self.self_attn(self.norm1(x), mask=mask, pos_bias=e))
+ x = fp16_clamp(x + self.cross_attn(
+ self.norm2(x), context=encoder_states, mask=encoder_mask))
+ x = fp16_clamp(x + self.ffn(self.norm3(x)))
+ return x
+
+
+class T5RelativeEmbedding(nn.Module):
+
+ def __init__(self, num_buckets, num_heads, bidirectional, max_dist=128):
+ super(T5RelativeEmbedding, self).__init__()
+ self.num_buckets = num_buckets
+ self.num_heads = num_heads
+ self.bidirectional = bidirectional
+ self.max_dist = max_dist
+
+ # layers
+ self.embedding = nn.Embedding(num_buckets, num_heads)
+
+ def forward(self, lq, lk):
+ device = self.embedding.weight.device
+ # rel_pos = torch.arange(lk).unsqueeze(0).to(device) - \
+ # torch.arange(lq).unsqueeze(1).to(device)
+ rel_pos = torch.arange(lk, device=device).unsqueeze(0) - \
+ torch.arange(lq, device=device).unsqueeze(1)
+ rel_pos = self._relative_position_bucket(rel_pos)
+ rel_pos_embeds = self.embedding(rel_pos)
+ rel_pos_embeds = rel_pos_embeds.permute(2, 0, 1).unsqueeze(
+ 0) # [1, N, Lq, Lk]
+ return rel_pos_embeds.contiguous()
+
+ def _relative_position_bucket(self, rel_pos):
+ # preprocess
+ if self.bidirectional:
+ num_buckets = self.num_buckets // 2
+ rel_buckets = (rel_pos > 0).long() * num_buckets
+ rel_pos = torch.abs(rel_pos)
+ else:
+ num_buckets = self.num_buckets
+ rel_buckets = 0
+ rel_pos = -torch.min(rel_pos, torch.zeros_like(rel_pos))
+
+ # embeddings for small and large positions
+ max_exact = num_buckets // 2
+ rel_pos_large = max_exact + (torch.log(rel_pos.float() / max_exact) /
+ math.log(self.max_dist / max_exact) *
+ (num_buckets - max_exact)).long()
+ rel_pos_large = torch.min(
+ rel_pos_large, torch.full_like(rel_pos_large, num_buckets - 1))
+ rel_buckets += torch.where(rel_pos < max_exact, rel_pos, rel_pos_large)
+ return rel_buckets
+
+
+class T5Encoder(nn.Module):
+
+ def __init__(self,
+ vocab,
+ dim,
+ dim_attn,
+ dim_ffn,
+ num_heads,
+ num_layers,
+ num_buckets,
+ shared_pos=True,
+ dropout=0.1):
+ super(T5Encoder, self).__init__()
+ self.dim = dim
+ self.dim_attn = dim_attn
+ self.dim_ffn = dim_ffn
+ self.num_heads = num_heads
+ self.num_layers = num_layers
+ self.num_buckets = num_buckets
+ self.shared_pos = shared_pos
+
+ # layers
+ self.token_embedding = vocab if isinstance(vocab, nn.Embedding) \
+ else nn.Embedding(vocab, dim)
+ self.pos_embedding = T5RelativeEmbedding(
+ num_buckets, num_heads, bidirectional=True) if shared_pos else None
+ self.dropout = nn.Dropout(dropout)
+ self.blocks = nn.ModuleList([
+ T5SelfAttention(dim, dim_attn, dim_ffn, num_heads, num_buckets,
+ shared_pos, dropout) for _ in range(num_layers)
+ ])
+ self.norm = T5LayerNorm(dim)
+
+ # initialize weights
+ self.apply(init_weights)
+
+ def forward(self, ids, mask=None):
+ x = self.token_embedding(ids)
+ x = self.dropout(x)
+ e = self.pos_embedding(x.size(1),
+ x.size(1)) if self.shared_pos else None
+ for block in self.blocks:
+ x = block(x, mask, pos_bias=e)
+ x = self.norm(x)
+ x = self.dropout(x)
+ return x
+
+
+class T5Decoder(nn.Module):
+
+ def __init__(self,
+ vocab,
+ dim,
+ dim_attn,
+ dim_ffn,
+ num_heads,
+ num_layers,
+ num_buckets,
+ shared_pos=True,
+ dropout=0.1):
+ super(T5Decoder, self).__init__()
+ self.dim = dim
+ self.dim_attn = dim_attn
+ self.dim_ffn = dim_ffn
+ self.num_heads = num_heads
+ self.num_layers = num_layers
+ self.num_buckets = num_buckets
+ self.shared_pos = shared_pos
+
+ # layers
+ self.token_embedding = vocab if isinstance(vocab, nn.Embedding) \
+ else nn.Embedding(vocab, dim)
+ self.pos_embedding = T5RelativeEmbedding(
+ num_buckets, num_heads, bidirectional=False) if shared_pos else None
+ self.dropout = nn.Dropout(dropout)
+ self.blocks = nn.ModuleList([
+ T5CrossAttention(dim, dim_attn, dim_ffn, num_heads, num_buckets,
+ shared_pos, dropout) for _ in range(num_layers)
+ ])
+ self.norm = T5LayerNorm(dim)
+
+ # initialize weights
+ self.apply(init_weights)
+
+ def forward(self, ids, mask=None, encoder_states=None, encoder_mask=None):
+ b, s = ids.size()
+
+ # causal mask
+ if mask is None:
+ mask = torch.tril(torch.ones(1, s, s).to(ids.device))
+ elif mask.ndim == 2:
+ mask = torch.tril(mask.unsqueeze(1).expand(-1, s, -1))
+
+ # layers
+ x = self.token_embedding(ids)
+ x = self.dropout(x)
+ e = self.pos_embedding(x.size(1),
+ x.size(1)) if self.shared_pos else None
+ for block in self.blocks:
+ x = block(x, mask, encoder_states, encoder_mask, pos_bias=e)
+ x = self.norm(x)
+ x = self.dropout(x)
+ return x
+
+
+class T5Model(nn.Module):
+
+ def __init__(self,
+ vocab_size,
+ dim,
+ dim_attn,
+ dim_ffn,
+ num_heads,
+ encoder_layers,
+ decoder_layers,
+ num_buckets,
+ shared_pos=True,
+ dropout=0.1):
+ super(T5Model, self).__init__()
+ self.vocab_size = vocab_size
+ self.dim = dim
+ self.dim_attn = dim_attn
+ self.dim_ffn = dim_ffn
+ self.num_heads = num_heads
+ self.encoder_layers = encoder_layers
+ self.decoder_layers = decoder_layers
+ self.num_buckets = num_buckets
+
+ # layers
+ self.token_embedding = nn.Embedding(vocab_size, dim)
+ self.encoder = T5Encoder(self.token_embedding, dim, dim_attn, dim_ffn,
+ num_heads, encoder_layers, num_buckets,
+ shared_pos, dropout)
+ self.decoder = T5Decoder(self.token_embedding, dim, dim_attn, dim_ffn,
+ num_heads, decoder_layers, num_buckets,
+ shared_pos, dropout)
+ self.head = nn.Linear(dim, vocab_size, bias=False)
+
+ # initialize weights
+ self.apply(init_weights)
+
+ def forward(self, encoder_ids, encoder_mask, decoder_ids, decoder_mask):
+ x = self.encoder(encoder_ids, encoder_mask)
+ x = self.decoder(decoder_ids, decoder_mask, x, encoder_mask)
+ x = self.head(x)
+ return x
+
+
+def _t5(name,
+ encoder_only=False,
+ decoder_only=False,
+ return_tokenizer=False,
+ tokenizer_kwargs={},
+ dtype=torch.float32,
+ device='cpu',
+ **kwargs):
+ # sanity check
+ assert not (encoder_only and decoder_only)
+
+ # params
+ if encoder_only:
+ model_cls = T5Encoder
+ kwargs['vocab'] = kwargs.pop('vocab_size')
+ kwargs['num_layers'] = kwargs.pop('encoder_layers')
+ _ = kwargs.pop('decoder_layers')
+ elif decoder_only:
+ model_cls = T5Decoder
+ kwargs['vocab'] = kwargs.pop('vocab_size')
+ kwargs['num_layers'] = kwargs.pop('decoder_layers')
+ _ = kwargs.pop('encoder_layers')
+ else:
+ model_cls = T5Model
+
+ # init model
+ with torch.device(device):
+ model = model_cls(**kwargs)
+
+ # set device
+ model = model.to(dtype=dtype, device=device)
+
+ # init tokenizer
+ if return_tokenizer:
+ from .tokenizer import HuggingfaceTokenizer
+ tokenizer = HuggingfaceTokenizer(f'google/{name}', **tokenizer_kwargs)
+ return model, tokenizer
+ else:
+ return model
+
+
+def umt5_xxl(**kwargs):
+ cfg = dict(
+ vocab_size=256384,
+ dim=4096,
+ dim_attn=4096,
+ dim_ffn=10240,
+ num_heads=64,
+ encoder_layers=24,
+ decoder_layers=24,
+ num_buckets=32,
+ shared_pos=False,
+ dropout=0.1)
+ cfg.update(**kwargs)
+ return _t5('umt5-xxl', **cfg)
+
+
+class T5EncoderModel:
+
+ def __init__(
+ self,
+ text_len,
+ dtype=torch.bfloat16,
+ device=torch.cuda.current_device(),
+ checkpoint_path=None,
+ tokenizer_path=None,
+ shard_fn=None,
+ ):
+ self.text_len = text_len
+ self.dtype = dtype
+ self.device = device
+ self.checkpoint_path = checkpoint_path
+ self.tokenizer_path = tokenizer_path
+
+ # init model
+ model = umt5_xxl(
+ encoder_only=True,
+ return_tokenizer=False,
+ dtype=dtype,
+ device=device).eval().requires_grad_(False)
+ logging.info(f'loading {checkpoint_path}')
+ model.load_state_dict(torch.load(checkpoint_path, map_location='cpu'))
+ self.model = model
+ if shard_fn is not None:
+ self.model = shard_fn(self.model, sync_module_states=False)
+ else:
+ self.model.to(self.device)
+ # init tokenizer
+ self.tokenizer = HuggingfaceTokenizer(
+ name=tokenizer_path, seq_len=text_len, clean='whitespace')
+
+ def __call__(self, texts, device):
+ ids, mask = self.tokenizer(
+ texts, return_mask=True, add_special_tokens=True)
+ ids = ids.to(device)
+ mask = mask.to(device)
+ seq_lens = mask.gt(0).sum(dim=1).long()
+ context = self.model(ids, mask)
+ return [u[:v] for u, v in zip(context, seq_lens)]
diff --git a/worldplay2/text/tokenizer.py b/worldplay2/text/tokenizer.py
new file mode 100644
index 0000000..121e591
--- /dev/null
+++ b/worldplay2/text/tokenizer.py
@@ -0,0 +1,82 @@
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+import html
+import string
+
+import ftfy
+import regex as re
+from transformers import AutoTokenizer
+
+__all__ = ['HuggingfaceTokenizer']
+
+
+def basic_clean(text):
+ text = ftfy.fix_text(text)
+ text = html.unescape(html.unescape(text))
+ return text.strip()
+
+
+def whitespace_clean(text):
+ text = re.sub(r'\s+', ' ', text)
+ text = text.strip()
+ return text
+
+
+def canonicalize(text, keep_punctuation_exact_string=None):
+ text = text.replace('_', ' ')
+ if keep_punctuation_exact_string:
+ text = keep_punctuation_exact_string.join(
+ part.translate(str.maketrans('', '', string.punctuation))
+ for part in text.split(keep_punctuation_exact_string))
+ else:
+ text = text.translate(str.maketrans('', '', string.punctuation))
+ text = text.lower()
+ text = re.sub(r'\s+', ' ', text)
+ return text.strip()
+
+
+class HuggingfaceTokenizer:
+
+ def __init__(self, name, seq_len=None, clean=None, **kwargs):
+ assert clean in (None, 'whitespace', 'lower', 'canonicalize')
+ self.name = name
+ self.seq_len = seq_len
+ self.clean = clean
+
+ # init tokenizer
+ self.tokenizer = AutoTokenizer.from_pretrained(name, **kwargs)
+ self.vocab_size = self.tokenizer.vocab_size
+
+ def __call__(self, sequence, **kwargs):
+ return_mask = kwargs.pop('return_mask', False)
+
+ # arguments
+ _kwargs = {'return_tensors': 'pt'}
+ if self.seq_len is not None:
+ _kwargs.update({
+ 'padding': 'max_length',
+ 'truncation': True,
+ 'max_length': self.seq_len
+ })
+ _kwargs.update(**kwargs)
+
+ # tokenization
+ if isinstance(sequence, str):
+ sequence = [sequence]
+ if self.clean:
+ sequence = [self._clean(u) for u in sequence]
+ ids = self.tokenizer(sequence, **_kwargs)
+
+ # output
+ if return_mask:
+ return ids.input_ids, ids.attention_mask
+ else:
+ return ids.input_ids
+
+ def _clean(self, text):
+ if self.clean == 'whitespace':
+ text = whitespace_clean(basic_clean(text))
+ elif self.clean == 'lower':
+ text = whitespace_clean(basic_clean(text)).lower()
+ elif self.clean == 'canonicalize':
+ text = canonicalize(basic_clean(text))
+ return text
diff --git a/worldplay2/vae/__init__.py b/worldplay2/vae/__init__.py
new file mode 100644
index 0000000..bdef9f7
--- /dev/null
+++ b/worldplay2/vae/__init__.py
@@ -0,0 +1,4 @@
+from .causal_tae import CausalTAEVAE
+from .wan21 import Wan2_1_VAE
+
+__all__ = ["CausalTAEVAE", "Wan2_1_VAE"]
diff --git a/worldplay2/vae/causal_tae.py b/worldplay2/vae/causal_tae.py
new file mode 100644
index 0000000..c1394ea
--- /dev/null
+++ b/worldplay2/vae/causal_tae.py
@@ -0,0 +1,864 @@
+#!/usr/bin/env python3
+"""
+Causal (left-padded) TAE-style ENCODER for Wan 2.1, adapted for 4n+1 frame
+video inputs.
+
+This is a variant of the TAEW2_1 encoder (see ``taew2_1.py``) with ONE
+architectural change in the temporal-pooling behaviour:
+
+ * The original ``TPool`` reshapes every ``stride`` consecutive frames
+ into the channel axis directly. It relied on ``encode_video`` padding
+ the time axis at the *end* to a multiple of ``t_downscale``.
+
+ * ``CausalTPool`` (here) instead pads ONE zero frame on the *left*
+ (the temporal-causal side) whenever the incoming time length is not a
+ multiple of ``stride`` (i.e. for stride>1 on an odd-length input),
+ then pools. For a 4n+1 input this cascades cleanly:
+ T=4n+1 --pad1--> 4n+2 --pool2--> 2n+1 --pad1--> 2n+2 --pool2--> n+1.
+
+``MemBlock`` keeps the SAME behaviour as the original parallel path: it
+takes the "previous frame" as memory by left-padding one zero frame along
+the time axis and concatenating it on the channel dim inside the block.
+That part is unchanged — this file only re-implements the pooling.
+
+Scope (per request):
+ * ENCODER ONLY. No decoder / TGrow.
+ * ONE-SHOT (parallel) encode only. No streaming work-queue path.
+
+I/O convention (same as TAEW2_1):
+ encode_video: input [N, T, C=3, H, W] in [0, 1], T = 4n+1
+ output [N, T_lat, latent_channels, H/8, W/8]
+"""
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from tqdm.auto import tqdm
+
+
+def conv(n_in, n_out, **kwargs):
+ return nn.Conv2d(n_in, n_out, 3, padding=1, **kwargs)
+
+
+class MemBlock(nn.Module):
+ """Residual block that also consumes the previous frame's activation
+ (its "memory") concatenated on the channel axis. Identical to the
+ TAEW2_1 MemBlock."""
+
+ def __init__(self, n_in, n_out):
+ super().__init__()
+ self.conv = nn.Sequential(conv(n_in * 2, n_out), nn.ReLU(inplace=True), conv(n_out, n_out), nn.ReLU(inplace=True), conv(n_out, n_out))
+ self.skip = nn.Conv2d(n_in, n_out, 1, bias=False) if n_in != n_out else nn.Identity()
+ self.act = nn.ReLU(inplace=True)
+
+ def forward(self, x, past):
+ return self.act(self.conv(torch.cat([x, past], 1)) + self.skip(x))
+
+ def forward_chunk(self, x, N, cache):
+ """Streaming forward over one chunk in [N*T, C, H, W] layout.
+
+ The per-frame memory is the *previous* frame's input. Within the
+ chunk this is the left-shifted version of ``x``; the very first
+ frame of the chunk uses ``cache`` (the last input frame of the
+ PREVIOUS chunk) instead of a zero frame -- except when ``cache`` is
+ None (the first chunk overall), where it falls back to zeros. This
+ makes the chunked result identical to the one-shot path.
+
+ Args:
+ x: [N*T, C, H, W] chunk input.
+ N: batch size.
+ cache: previous chunk's last-frame input, [N, 1, C, H, W], or None.
+ Returns:
+ (out, new_cache) where out is [N*T, C, H, W] and new_cache is
+ this chunk's last-frame input [N, 1, C, H, W].
+ """
+ NT, C, H, W = x.shape
+ T = NT // N
+ _x = x.view(N, T, C, H, W)
+ if cache is None:
+ head = _x.new_zeros(N, 1, C, H, W) # first chunk: zero history
+ else:
+ head = cache # continuation: prev last frame
+ # memory[t] = input[t-1]; memory[0] = head
+ block_memory = torch.cat([head, _x[:, :T - 1]], dim=1).reshape(x.shape)
+ new_cache = _x[:, -1:].clone()
+ out = self.forward(x, block_memory)
+ return out, new_cache
+
+
+class CausalTPool(nn.Module):
+ """Temporal pooling that LEFT-pads one zero frame when needed so a
+ 4n+1 (odd) time length becomes even before pooling every ``stride``
+ frames into the channel axis.
+
+ Operates on the flattened [N*T, C, H, W] layout used by the parallel
+ encoder path, and therefore needs ``N`` (batch size) to recover the
+ time dimension.
+ """
+
+ def __init__(self, n_f, stride):
+ super().__init__()
+ self.stride = stride
+ self.conv = nn.Conv2d(n_f * stride, n_f, 1, bias=False)
+
+ def forward(self, x, N):
+ # x: [N*T, C, H, W]
+ NT, C, H, W = x.shape
+ T = NT // N
+ if self.stride == 1:
+ # no temporal pooling; the 1x1 conv still maps n_f -> n_f
+ return self.conv(x)
+
+ x = x.view(N, T, C, H, W)
+ # left-pad zeros along the time axis until T is a multiple of stride
+ # (for 4n+1 inputs and stride=2 this pads exactly one frame).
+ rem = T % self.stride
+ if rem != 0:
+ n_pad = self.stride - rem
+ pad = x.new_zeros(N, n_pad, C, H, W)
+ x = torch.cat([pad, x], dim=1) # LEFT pad (causal side)
+ T = x.shape[1]
+
+ # group every ``stride`` consecutive frames onto the channel axis,
+ # matching the original TPool's reshape ordering (frame t occupies
+ # channels [0:C], frame t+1 -> [C:2C], ...).
+ x = x.view(N, T // self.stride, self.stride * C, H, W)
+ x = x.reshape(N * (T // self.stride), self.stride * C, H, W)
+ return self.conv(x)
+
+ def forward_chunk(self, x, N, is_first_chunk):
+ """Streaming pool for the 1,4,4,4 chunk schedule.
+
+ With that schedule and stride=2, only the FIRST chunk is odd
+ (length 1 -> left-pad 1 -> 2 -> pool -> 1); every subsequent chunk
+ already has a length that is a multiple of ``stride`` (4 -> pool ->
+ 2, and at the 2nd pooling stage 2 -> pool -> 1), so no cross-chunk
+ residual buffering is needed and the concatenation of per-chunk
+ outputs is bit-for-bit identical to the one-shot ``forward``.
+
+ Args:
+ x: [N*T, C, H, W] chunk input.
+ N: batch size.
+ is_first_chunk: True only for the very first chunk.
+ Returns:
+ [N*(T_out), C, H, W] pooled output for this chunk.
+ """
+ NT, C, H, W = x.shape
+ T = NT // N
+ if self.stride == 1:
+ return self.conv(x)
+
+ _x = x.view(N, T, C, H, W)
+ rem = T % self.stride
+ if rem != 0:
+ assert is_first_chunk, (
+ f"CausalTPool.forward_chunk: non-first chunk has time length "
+ f"{T} which is not a multiple of stride {self.stride}; the "
+ f"1,4,4,4 schedule should keep continuation chunks aligned.")
+ n_pad = self.stride - rem
+ pad = _x.new_zeros(N, n_pad, C, H, W)
+ _x = torch.cat([pad, _x], dim=1) # LEFT pad only on the first chunk
+ T = _x.shape[1]
+
+ _x = _x.view(N, T // self.stride, self.stride * C, H, W)
+ _x = _x.reshape(N * (T // self.stride), self.stride * C, H, W)
+ return self.conv(_x)
+
+
+class Clamp(nn.Module):
+ """Soft clamp to ~[-3, 3] on the latent, used at the head of the decoder
+ (identical to the TAEW2_1 Clamp)."""
+
+ def forward(self, x):
+ return torch.tanh(x / 3) * 3
+
+
+class CausalTGrow(nn.Module):
+ """Temporal upsampling, inverse of ``CausalTPool``.
+
+ ``conv`` expands the channel axis to ``stride * n_f`` and the frames are
+ then split back out along the time axis (1 latent frame -> ``stride``
+ pixel-time frames), matching the original TGrow reshape ordering.
+
+ For CAUSAL first-frame handling, the VERY FIRST latent frame of a clip
+ maps to a single output frame with ``output == input`` (it skips both the
+ conv and the temporal expansion entirely), while every subsequent latent
+ frame is expanded to ``stride`` frames. This mirrors the encoder, where
+ the first pixel frame maps to one latent; here the first latent maps back
+ to one frame, so:
+
+ T_lat = n + 1 -> T_pix = 1 + n * (product of TGrow strides) = 4n+1
+ """
+
+ def __init__(self, n_f, stride):
+ super().__init__()
+ self.stride = stride
+ self.conv = nn.Conv2d(n_f, n_f * stride, 1, bias=False)
+
+ def _expand(self, x):
+ """conv + reshape: [N*T, C, H, W] -> [N*T*stride, C, H, W]."""
+ NT, C, H, W = x.shape
+ x = self.conv(x) # [N*T, stride*C, H, W]
+ return x.reshape(NT * self.stride, C, H, W)
+
+ def forward(self, x, N, is_first_clip=True):
+ """One-shot temporal grow over the whole clip.
+
+ Args:
+ x: [N*T_lat, C, H, W].
+ N: batch size.
+ is_first_clip: if True, the first latent frame of the clip
+ expands to a single output frame (causal startup), the rest
+ expand to ``stride`` frames each. If False (continuation
+ chunk in streaming decode), ALL frames expand to ``stride``.
+ Returns:
+ [N*T_out, C, H, W].
+ """
+ NT, C, H, W = x.shape
+ T = NT // N
+ if self.stride == 1:
+ # no temporal upsampling; conv maps n_f -> n_f*1 = n_f, reshape no-op
+ return self._expand(x)
+
+ _x = x.view(N, T, C, H, W)
+ if is_first_clip:
+ # first latent frame: output == input (no temporal expansion,
+ # no conv), so it maps to exactly ONE output frame.
+ first = _x[:, :1] # [N, 1, C, H, W]
+ rest = _x[:, 1:].reshape(N * (T - 1), C, H, W) if T > 1 else None
+ outs = [first]
+ if rest is not None:
+ rest_grown = self._expand(rest) # [N*(T-1)*stride, C, H, W]
+ rest_grown = rest_grown.view(N, (T - 1) * self.stride, C, H, W)
+ outs.append(rest_grown)
+ out = torch.cat(outs, dim=1) # [N, T_out, C, H, W]
+ return out.reshape(N * out.shape[1], C, H, W)
+ else:
+ # continuation chunk: every latent frame expands to stride frames
+ return self._expand(x)
+
+
+class CausalTAEEncoder(nn.Module):
+ """One-shot causal TAE-style encoder for Wan 2.1 (4n+1 frame inputs)."""
+
+ def __init__(self,
+ encoder_time_downscale=(True, True, False),
+ latent_channels=16,
+ patch_size=1):
+ super().__init__()
+ self.patch_size = patch_size
+ self.latent_channels = latent_channels
+ self.image_channels = 3
+
+ self.encoder = nn.Sequential(
+ conv(self.image_channels * self.patch_size ** 2, 64), nn.ReLU(inplace=True),
+ CausalTPool(64, 2 if encoder_time_downscale[0] else 1), conv(64, 64, stride=2, bias=False), MemBlock(64, 64), MemBlock(64, 64), MemBlock(64, 64),
+ CausalTPool(64, 2 if encoder_time_downscale[1] else 1), conv(64, 64, stride=2, bias=False), MemBlock(64, 64), MemBlock(64, 64), MemBlock(64, 64),
+ CausalTPool(64, 2 if encoder_time_downscale[2] else 1), conv(64, 64, stride=2, bias=False), MemBlock(64, 64), MemBlock(64, 64), MemBlock(64, 64),
+ conv(64, self.latent_channels),
+ )
+
+ # temporal downscale factor (product of the pooling strides)
+ self.t_downscale = 1
+ for m in self.encoder:
+ if isinstance(m, CausalTPool):
+ self.t_downscale *= m.stride
+
+ def preprocess_input_frames(self, x):
+ """Preprocess RGB input frames prior to the main encoder sequence."""
+ if self.patch_size > 1:
+ x = F.pixel_unshuffle(x, self.patch_size)
+ return x
+
+ def _apply_encoder(self, x, show_progress_bar):
+ """Parallel (one-shot) application of the encoder sequence over the
+ time axis.
+
+ Args:
+ x: [N, T, C, H, W] tensor.
+ Returns:
+ [N, T_lat, latent_channels, H/8, W/8] tensor.
+ """
+ assert x.ndim == 5, f"expected NTCHW tensor, got {x.ndim}-dim"
+ N, T, C, H, W = x.shape
+ x = x.reshape(N * T, C, H, W)
+
+ for b in tqdm(self.encoder, disable=not show_progress_bar):
+ if isinstance(b, MemBlock):
+ # "previous frame" memory = left-pad one zero frame along
+ # time, then drop the last frame so lengths line up. This is
+ # the same causal memory the original parallel path uses.
+ NT, C, H, W = x.shape
+ T = NT // N
+ _x = x.reshape(N, T, C, H, W)
+ block_memory = F.pad(_x, (0, 0, 0, 0, 0, 0, 1, 0), value=0)[:, :T].reshape(x.shape)
+ x = b(x, block_memory)
+ elif isinstance(b, CausalTPool):
+ # CausalTPool needs N to recover the time dim and does the
+ # left-pad-then-pool internally.
+ x = b(x, N)
+ else:
+ x = b(x)
+
+ NT, C, H, W = x.shape
+ T = NT // N
+ return x.view(N, T, C, H, W)
+
+ def encode_video(self, x, show_progress_bar=True):
+ """Encode a sequence of frames (one-shot / parallel).
+
+ Args:
+ x: input [N, T, C=3, H, W] RGB tensor in [0, 1], with T = 4n+1.
+ show_progress_bar: enable tqdm over encoder blocks.
+ Returns:
+ [N, T_lat, latent_channels, H/8, W/8] latent tensor.
+
+ Unlike TAEW2_1.encode_video, there is NO end-of-sequence padding:
+ the left-padding inside each CausalTPool handles odd (4n+1) lengths
+ causally.
+ """
+ x = self.preprocess_input_frames(x)
+ return self._apply_encoder(x, show_progress_bar)
+
+ # ------------------------------------------------------------------
+ # streaming / chunked encode (1, 4, 4, 4, ... schedule)
+ # ------------------------------------------------------------------
+
+ def init_cache(self):
+ """Create a fresh per-block streaming cache.
+
+ Returns a list with one slot per encoder block; only ``MemBlock``
+ slots are ever populated (each holds that block's last-frame input
+ from the previous chunk). Pass the same object to every
+ :meth:`encode_chunk` call of one stream.
+ """
+ return [None] * len(self.encoder)
+
+ def _apply_encoder_chunk(self, x, cache, is_first_chunk):
+ """Streaming application of the encoder over one chunk.
+
+ Args:
+ x: [N, T_chunk, C, H, W] chunk (T_chunk == 1 for the first chunk,
+ then 4 for each continuation chunk).
+ cache: list from :meth:`init_cache`, mutated in place.
+ is_first_chunk: True only for the first chunk of the stream.
+ Returns:
+ [N, T_lat_chunk, latent_channels, H/8, W/8] latent for this chunk.
+ """
+ assert x.ndim == 5, f"expected NTCHW tensor, got {x.ndim}-dim"
+ N = x.shape[0]
+ x = x.reshape(N * x.shape[1], *x.shape[2:])
+
+ for i, b in enumerate(self.encoder):
+ if isinstance(b, MemBlock):
+ x, cache[i] = b.forward_chunk(x, N, cache[i])
+ elif isinstance(b, CausalTPool):
+ x = b.forward_chunk(x, N, is_first_chunk)
+ else:
+ x = b(x)
+
+ NT, C, H, W = x.shape
+ return x.view(N, NT // N, C, H, W)
+
+ def encode_chunk(self, x, cache, is_first_chunk=False):
+ """Encode ONE chunk of frames, continuing the streaming state.
+
+ Feed the video in the ``1, 4, 4, 4, ...`` schedule: the first call
+ with ``is_first_chunk=True`` passes a single frame ([N, 1, C, H, W]),
+ and each subsequent call passes 4 frames ([N, 4, C, H, W]) with
+ ``is_first_chunk=False``. Concatenating the returned latents along
+ the time axis reproduces :meth:`encode_video` on the full clip
+ exactly.
+
+ Args:
+ x: [N, T_chunk, C=3, H, W] RGB chunk in [0, 1].
+ cache: list from :meth:`init_cache`, reused across the stream.
+ is_first_chunk: True only for the very first chunk.
+ Returns:
+ [N, T_lat_chunk, latent_channels, H/8, W/8] latent for this chunk.
+ """
+ x = self.preprocess_input_frames(x)
+ return self._apply_encoder_chunk(x, cache, is_first_chunk)
+
+ def encode_video_chunked(self, x, show_progress_bar=False):
+ """Convenience wrapper: split a full 4n+1 clip into the 1,4,4,4,...
+ schedule, run :meth:`encode_chunk` over each piece, and concatenate.
+
+ Args:
+ x: [N, T=4n+1, C=3, H, W] RGB clip in [0, 1].
+ Returns:
+ [N, T_lat, latent_channels, H/8, W/8] latent (== encode_video).
+ """
+ T = x.shape[1]
+ assert T % 4 == 1, f"chunked encode expects 4n+1 frames, got T={T}"
+ cache = self.init_cache()
+ outs = []
+ # first chunk: 1 frame
+ outs.append(self.encode_chunk(x[:, :1], cache, is_first_chunk=True))
+ # continuation chunks: 4 frames each
+ starts = range(1, T, 4)
+ for s in tqdm(starts, disable=not show_progress_bar):
+ outs.append(self.encode_chunk(x[:, s:s + 4], cache, is_first_chunk=False))
+ return torch.cat(outs, dim=1)
+
+ def forward(self, x, show_progress_bar=False):
+ return self.encode_video(x, show_progress_bar=show_progress_bar)
+
+
+class CausalTAEDecoder(nn.Module):
+ """Causal TAE-style decoder for Wan 2.1, inverse of CausalTAEEncoder.
+
+ The architecture mirrors the TAEW2_1 base decoder. The only causal
+ change is in temporal upsampling: the FIRST latent frame of a clip
+ decodes to a SINGLE output frame (via CausalTGrow's first-frame path),
+ so the whole decoder maps
+
+ T_lat = n + 1 latent frames -> T_pix = 4n + 1 output frames
+
+ which is the exact inverse of the encoder. No end-of-sequence trimming
+ is required (unlike the original TAEW2_1.decode_video which trims
+ ``frames_to_trim`` startup frames).
+ """
+
+ def __init__(self,
+ decoder_time_upscale=(False, True, True),
+ decoder_space_upscale=(True, True, True),
+ latent_channels=16,
+ patch_size=1):
+ super().__init__()
+ self.patch_size = patch_size
+ self.latent_channels = latent_channels
+ self.image_channels = 3
+
+ n_f = [256, 128, 64, 64]
+ self.decoder = nn.Sequential(
+ Clamp(), conv(self.latent_channels, n_f[0]), nn.ReLU(inplace=True),
+ MemBlock(n_f[0], n_f[0]), MemBlock(n_f[0], n_f[0]), MemBlock(n_f[0], n_f[0]), nn.Upsample(scale_factor=2 if decoder_space_upscale[0] else 1), CausalTGrow(n_f[0], 2 if decoder_time_upscale[0] else 1), conv(n_f[0], n_f[1], bias=False),
+ MemBlock(n_f[1], n_f[1]), MemBlock(n_f[1], n_f[1]), MemBlock(n_f[1], n_f[1]), nn.Upsample(scale_factor=2 if decoder_space_upscale[1] else 1), CausalTGrow(n_f[1], 2 if decoder_time_upscale[1] else 1), conv(n_f[1], n_f[2], bias=False),
+ MemBlock(n_f[2], n_f[2]), MemBlock(n_f[2], n_f[2]), MemBlock(n_f[2], n_f[2]), nn.Upsample(scale_factor=2 if decoder_space_upscale[2] else 1), CausalTGrow(n_f[2], 2 if decoder_time_upscale[2] else 1), conv(n_f[2], n_f[3], bias=False),
+ nn.ReLU(inplace=True), conv(n_f[3], self.image_channels * self.patch_size ** 2),
+ )
+
+ # temporal upscale factor (product of the grow strides)
+ self.t_upscale = 1
+ for m in self.decoder:
+ if isinstance(m, CausalTGrow):
+ self.t_upscale *= m.stride
+
+ def postprocess_output_frames(self, x):
+ """Postprocess RGB frames after the main decoder sequence."""
+ if self.patch_size > 1:
+ x = F.pixel_shuffle(x, self.patch_size)
+ return x.clamp_(0, 1)
+
+ def _apply_decoder(self, x, is_first_clip, show_progress_bar):
+ """Apply the decoder sequence over the time axis (one-shot).
+
+ Args:
+ x: [N, T_lat, C, H, W] latent tensor.
+ is_first_clip: True if this batch of latents starts a clip (the
+ first latent frame decodes to a single output frame).
+ Returns:
+ [N, T_out, image_channels*patch_size**2, H*8, W*8] tensor.
+ """
+ assert x.ndim == 5, f"expected NTCHW tensor, got {x.ndim}-dim"
+ N, T, C, H, W = x.shape
+ x = x.reshape(N * T, C, H, W)
+
+ for b in tqdm(self.decoder, disable=not show_progress_bar):
+ if isinstance(b, MemBlock):
+ NT, C, H, W = x.shape
+ T = NT // N
+ _x = x.reshape(N, T, C, H, W)
+ block_memory = F.pad(_x, (0, 0, 0, 0, 0, 0, 1, 0), value=0)[:, :T].reshape(x.shape)
+ x = b(x, block_memory)
+ elif isinstance(b, CausalTGrow):
+ x = b(x, N, is_first_clip=is_first_clip)
+ else:
+ x = b(x)
+
+ NT, C, H, W = x.shape
+ T = NT // N
+ return x.view(N, T, C, H, W)
+
+ def decode_video(self, x, show_progress_bar=True):
+ """Decode a full sequence of latents (one-shot / parallel).
+
+ Args:
+ x: input [N, T_lat=n+1, C=latent_channels, H, W] latent tensor.
+ show_progress_bar: enable tqdm over decoder blocks.
+ Returns:
+ [N, T_pix=4n+1, 3, H*8, W*8] RGB tensor in [0, 1].
+ """
+ x = self._apply_decoder(x, is_first_clip=True, show_progress_bar=show_progress_bar)
+ return self.postprocess_output_frames(x)
+
+ def decode_latent(self, x, is_first_latent, show_progress_bar=False):
+ """Decode ONE (or a few) latent frame(s) with explicit first-frame
+ control, WITHOUT threading a cross-latent cache -- the low-level
+ stateless building block.
+
+ NOTE: without a MemBlock cache, calling this repeatedly is NOT
+ numerically identical to ``decode_video`` at chunk boundaries (each
+ call restarts the MemBlock memory from zeros). For an equivalent
+ one-latent-at-a-time decode, use :meth:`init_cache` +
+ :meth:`decode_chunk` instead.
+
+ Args:
+ x: [N, T_sub, C=latent_channels, H, W] latent(s).
+ is_first_latent: True if these latents include the clip's first
+ latent frame (-> single output frame for that frame).
+ Returns:
+ [N, T_out, 3, H*8, W*8] RGB tensor in [0, 1].
+ """
+ x = self._apply_decoder(x, is_first_clip=is_first_latent,
+ show_progress_bar=show_progress_bar)
+ return self.postprocess_output_frames(x)
+
+ # ------------------------------------------------------------------
+ # streaming / chunked decode (cross-latent cache)
+ # ------------------------------------------------------------------
+
+ def init_cache(self):
+ """Create a fresh per-block streaming cache.
+
+ Returns a list with one slot per decoder block; only ``MemBlock``
+ slots are ever populated (each holds that block's last-frame input
+ from the previous chunk). Pass the same object to every
+ :meth:`decode_chunk` call of one stream.
+ """
+ return [None] * len(self.decoder)
+
+ def _apply_decoder_chunk(self, x, cache, is_first_chunk):
+ """Streaming application of the decoder over one chunk of latents.
+
+ MemBlock threads its "previous frame" memory through ``cache`` (same
+ contract as the encoder), and CausalTGrow uses ``is_first_chunk`` so
+ that only the very first latent of the stream yields a single frame
+ (output == input); continuation chunks expand every latent frame by
+ ``stride``.
+
+ Args:
+ x: [N, T_sub, C, H, W] latent chunk.
+ cache: list from :meth:`init_cache`, mutated in place.
+ is_first_chunk: True only for the first chunk of the stream.
+ Returns:
+ [N, T_out, image_channels*patch_size**2, H*8, W*8] tensor.
+ """
+ assert x.ndim == 5, f"expected NTCHW tensor, got {x.ndim}-dim"
+ N = x.shape[0]
+ x = x.reshape(N * x.shape[1], *x.shape[2:])
+
+ for i, b in enumerate(self.decoder):
+ if isinstance(b, MemBlock):
+ x, cache[i] = b.forward_chunk(x, N, cache[i])
+ elif isinstance(b, CausalTGrow):
+ x = b(x, N, is_first_clip=is_first_chunk)
+ else:
+ x = b(x)
+
+ NT, C, H, W = x.shape
+ return x.view(N, NT // N, C, H, W)
+
+ def decode_chunk(self, x, cache, is_first_chunk=False):
+ """Decode ONE chunk of latents, continuing the streaming state.
+
+ Feed latents one (or a few) at a time. The first call with
+ ``is_first_chunk=True`` decodes the clip's first latent to a single
+ output frame; every subsequent call (``is_first_chunk=False``)
+ expands each latent frame to ``t_upscale`` output frames.
+ Concatenating the returned frames along the time axis reproduces
+ :meth:`decode_video` on the full latent sequence exactly.
+
+ Args:
+ x: [N, T_sub, C=latent_channels, H, W] latent chunk.
+ cache: list from :meth:`init_cache`, reused across the stream.
+ is_first_chunk: True only for the very first chunk.
+ Returns:
+ [N, T_out, 3, H*8, W*8] RGB frames in [0, 1].
+ """
+ x = self._apply_decoder_chunk(x, cache, is_first_chunk)
+ return self.postprocess_output_frames(x)
+
+ def decode_video_streamed(self, x, show_progress_bar=False):
+ """Convenience wrapper: decode a full latent sequence one latent at a
+ time through :meth:`decode_chunk` and concatenate.
+
+ Numerically identical to :meth:`decode_video` (up to float rounding).
+
+ Args:
+ x: [N, T_lat=n+1, C=latent_channels, H, W] latent tensor.
+ Returns:
+ [N, T_pix=4n+1, 3, H*8, W*8] RGB tensor in [0, 1].
+ """
+ T = x.shape[1]
+ cache = self.init_cache()
+ outs = []
+ for i in tqdm(range(T), disable=not show_progress_bar):
+ outs.append(self.decode_chunk(x[:, i:i + 1], cache,
+ is_first_chunk=(i == 0)))
+ return torch.cat(outs, dim=1)
+
+ def forward(self, x, show_progress_bar=False):
+ return self.decode_video(x, show_progress_bar=show_progress_bar)
+
+
+class CausalTAE(nn.Module):
+ """Unified causal TAE for Wan 2.1: encoder + decoder in one module.
+
+ Wraps :class:`CausalTAEEncoder` and :class:`CausalTAEDecoder` so a single
+ object exposes the full video<->latent roundtrip. Both one-shot and
+ streaming (chunked) paths are provided and are numerically equivalent.
+
+ I/O convention:
+ encode: [N, T=4n+1, C=3, H, W] RGB in [0, 1]
+ -> [N, T_lat=n+1, latent_channels, H/8, W/8]
+ decode: [N, T_lat=n+1, latent_channels, H, W]
+ -> [N, T_pix=4n+1, 3, H*8, W*8] RGB in [0, 1]
+
+ Streaming schedules (equivalent to the one-shot paths):
+ encode: feed frames as 1, 4, 4, 4, ... (encode_chunk)
+ decode: feed latents one at a time (decode_chunk)
+ """
+
+ def __init__(self,
+ encoder_time_downscale=(True, True, False),
+ decoder_time_upscale=(False, True, True),
+ decoder_space_upscale=(True, True, True),
+ latent_channels=16,
+ patch_size=1):
+ super().__init__()
+ self.latent_channels = latent_channels
+ self.patch_size = patch_size
+ self.image_channels = 3
+
+ self.enc = CausalTAEEncoder(
+ encoder_time_downscale=encoder_time_downscale,
+ latent_channels=latent_channels,
+ patch_size=patch_size,
+ )
+ self.dec = CausalTAEDecoder(
+ decoder_time_upscale=decoder_time_upscale,
+ decoder_space_upscale=decoder_space_upscale,
+ latent_channels=latent_channels,
+ patch_size=patch_size,
+ )
+ self.t_downscale = self.enc.t_downscale
+ self.t_upscale = self.dec.t_upscale
+
+ # ---- one-shot ----------------------------------------------------
+
+ def encode_video(self, x, show_progress_bar=True):
+ """One-shot encode. [N, 4n+1, 3, H, W] -> [N, n+1, C_z, H/8, W/8]."""
+ return self.enc.encode_video(x, show_progress_bar=show_progress_bar)
+
+ def decode_video(self, z, show_progress_bar=True):
+ """One-shot decode. [N, n+1, C_z, H, W] -> [N, 4n+1, 3, H*8, W*8]."""
+ return self.dec.decode_video(z, show_progress_bar=show_progress_bar)
+
+ def forward(self, x, teacher_latent=None, show_progress_bar=False):
+ """Two modes:
+
+ * ``teacher_latent is None`` (default): full roundtrip encode->decode,
+ returns reconstructed RGB.
+ * ``teacher_latent`` given: TRAINING mode. Runs both the encoder
+ (supervised by ``teacher_latent``) and the decoder (fed the teacher
+ latent, reconstructing ``x``) through this single ``forward`` so that
+ DistributedDataParallel sees the whole graph and syncs gradients
+ correctly. Returns ``(encoded, decoded)``:
+ encoded: [N, T_lat, C_z, H/8, W/8] (compare to teacher_latent)
+ decoded: [N, T_pix, 3, H*8, W*8] (compare to x)
+ """
+ if teacher_latent is None:
+ z = self.encode_video(x, show_progress_bar=show_progress_bar)
+ return self.decode_video(z, show_progress_bar=show_progress_bar)
+ encoded = self.encode_video(x, show_progress_bar=show_progress_bar)
+ decoded = self.decode_video(teacher_latent, show_progress_bar=show_progress_bar)
+ return encoded, decoded
+
+ # ---- streaming: encode (1, 4, 4, 4, ...) -------------------------
+
+ def init_encode_cache(self):
+ return self.enc.init_cache()
+
+ def encode_chunk(self, x, cache, is_first_chunk=False):
+ """Streaming encode of one chunk (1 frame first, then 4 each)."""
+ return self.enc.encode_chunk(x, cache, is_first_chunk=is_first_chunk)
+
+ def encode_video_chunked(self, x, show_progress_bar=False):
+ """Streaming encode over a full 4n+1 clip (== encode_video)."""
+ return self.enc.encode_video_chunked(x, show_progress_bar=show_progress_bar)
+
+ # ---- streaming: decode (one latent at a time) --------------------
+
+ def init_decode_cache(self):
+ return self.dec.init_cache()
+
+ def decode_chunk(self, z, cache, is_first_chunk=False):
+ """Streaming decode of one latent chunk (first -> 1 frame, then
+ t_upscale frames each)."""
+ return self.dec.decode_chunk(z, cache, is_first_chunk=is_first_chunk)
+
+ def decode_video_streamed(self, z, show_progress_bar=False):
+ """Streaming decode over a full latent sequence (== decode_video)."""
+ return self.dec.decode_video_streamed(z, show_progress_bar=show_progress_bar)
+
+
+# ===========================================================================
+# Drop-in adapter exposing the SAME interface as ``worldplay2.vae.wan21.Wan2_1_VAE``
+# so ``image2video_stage_two`` can switch VAEs without touching call sites.
+# ===========================================================================
+
+class CausalTAEVAE:
+ r"""Adapter that wraps :class:`CausalTAE` behind the exact public surface
+ of :class:`worldplay2.vae.wan21.Wan2_1_VAE`, so the stage-2/3 pipeline can
+ use it as a drop-in replacement with NO changes to its call sites.
+
+ Interface parity (all list-in / list-out, elements are ``[C, T, H, W]``
+ tensors WITHOUT a batch dim, pixels in ``[-1, 1]``):
+ * ``encode([clip])`` -> ``[z]``
+ * ``decode([z])`` -> ``[clip]``
+ * ``encode_chunk([clip], is_first_chunk)`` -> ``[z]``
+ * ``decode_chunk([z], is_first_chunk)`` -> ``[clip]``
+ plus the ``.scale`` / ``.dtype`` / ``.device`` attributes.
+
+ Differences from ``CausalTAE`` that this adapter bridges:
+ * LAYOUT: ``Wan2_1_VAE`` uses list of ``[C, T, H, W]``; ``CausalTAE``
+ uses batched ``[N, T, C, H, W]``. We add/remove the batch dim and
+ move the channel axis.
+ * VALUE RANGE: ``Wan2_1_VAE`` pixels are in ``[-1, 1]``; ``CausalTAE``
+ pixels are in ``[0, 1]``. We map ``(x+1)/2`` in and ``x*2-1`` out.
+ * STREAMING CACHE: ``CausalTAE`` streaming keeps the per-block cache in
+ an EXTERNAL list; ``Wan2_1_VAE`` keeps it INTERNAL and re-arms it when
+ ``is_first_chunk=True``. We hold the enc/dec caches on the adapter
+ and rebuild them on ``is_first_chunk=True`` to reproduce the internal
+ semantics exactly.
+
+ Temporal convention matches ``Wan2_1_VAE`` (causal, first-frame aligned):
+ ``T_pix = 4 * T_lat - 3`` (== ``CausalTAE``'s ``T_lat = n+1 <-> 4n+1``).
+ """
+
+ def __init__(self,
+ vae_pth: str,
+ z_dim: int = 16,
+ dtype=torch.float,
+ device="cuda",
+ **causal_tae_kwargs):
+ self.dtype = dtype
+ self.device = device
+ # No mean/std normalisation: the CausalTAE latent space is
+ # self-consistent (its own decoder inverts its own encoder), so we
+ # expose a no-op ``scale`` only for attribute parity with Wan2_1_VAE.
+ self.scale = [0.0, 1.0]
+
+ self.model = CausalTAE(
+ latent_channels=z_dim, **causal_tae_kwargs
+ ).eval().requires_grad_(False)
+ if vae_pth is not None:
+ sd = torch.load(vae_pth, map_location="cpu", weights_only=True)
+ # tolerate a wrapping dict (e.g. {"model": ...} / {"state_dict": ...})
+ if isinstance(sd, dict):
+ for key in ("state_dict", "model", "module"):
+ if key in sd and isinstance(sd[key], dict):
+ sd = sd[key]
+ break
+ missing, unexpected = self.model.load_state_dict(sd, strict=False)
+ if missing or unexpected:
+ import logging
+ logging.info(
+ f"CausalTAEVAE: loaded {vae_pth} "
+ f"(missing={len(missing)}, unexpected={len(unexpected)})")
+ self.model = self.model.to(device)
+
+ # streaming caches (internal, mirroring Wan2_1_VAE's internal state).
+ self._enc_cache = None
+ self._dec_cache = None
+
+ # ---- layout / range helpers -------------------------------------------
+
+ @staticmethod
+ def _to_ntchw_01(clip):
+ """``[C, T, H, W]`` in [-1,1] -> ``[1, T, C, H, W]`` in [0,1]."""
+ x = clip.permute(1, 0, 2, 3).contiguous() # [T, C, H, W]
+ x = x.unsqueeze(0) # [1, T, C, H, W]
+ return x.mul(0.5).add(0.5) # [-1,1] -> [0,1]
+
+ @staticmethod
+ def _from_ntchw_01(x):
+ """``[1, T, C, H, W]`` in [0,1] -> ``[C, T, H, W]`` in [-1,1]."""
+ x = x.squeeze(0) # [T, C, H, W]
+ x = x.permute(1, 0, 2, 3).contiguous() # [C, T, H, W]
+ return x.mul(2.0).sub(1.0).clamp_(-1, 1) # [0,1] -> [-1,1]
+
+ @staticmethod
+ def _lat_to_ntchw(z):
+ """latent ``[C, T, H, W]`` -> ``[1, T, C, H, W]`` (no range change)."""
+ return z.permute(1, 0, 2, 3).contiguous().unsqueeze(0)
+
+ @staticmethod
+ def _lat_from_ntchw(z):
+ """latent ``[1, T, C, H, W]`` -> ``[C, T, H, W]``."""
+ return z.squeeze(0).permute(1, 0, 2, 3).contiguous()
+
+ # ---- one-shot ---------------------------------------------------------
+
+ def encode(self, videos):
+ """``videos``: list of ``[3, T_pix, H, W]`` in [-1,1] ->
+ list of latent ``[16, T_lat, H/8, W/8]``.
+
+ Uses the STREAMING encode (``encode_video_chunked``: the 1,4,4,4,...
+ chunk schedule with a fresh cache per clip) rather than a one-shot
+ ``encode_video`` -- numerically identical, but bounds peak memory."""
+ with torch.amp.autocast('cuda', dtype=self.dtype), torch.no_grad():
+ out = []
+ for u in videos:
+ x = self._to_ntchw_01(u).to(self.device)
+ z = self.model.encode_video_chunked(x, show_progress_bar=False)
+ out.append(self._lat_from_ntchw(z).float())
+ return out
+
+ def decode(self, zs):
+ """``zs``: list of latent ``[16, T_lat, H, W]`` ->
+ list of pixel ``[3, T_pix, H*8, W*8]`` in [-1,1].
+
+ Uses the STREAMING decode (``decode_video_streamed``: one latent
+ chunk at a time with a fresh cache per clip) rather than a one-shot
+ ``decode_video`` -- numerically identical, but bounds peak memory."""
+ with torch.amp.autocast('cuda', dtype=self.dtype), torch.no_grad():
+ out = []
+ for u in zs:
+ z = self._lat_to_ntchw(u).to(self.device)
+ x = self.model.decode_video_streamed(z, show_progress_bar=False)
+ out.append(self._from_ntchw_01(x).float())
+ return out
+
+ # ---- streaming --------------------------------------------------------
+
+ def encode_chunk(self, videos, is_first_chunk: bool = False):
+ """Streaming encode. Same list I/O as :meth:`encode`. The adapter
+ holds the encoder cache internally and re-arms it when
+ ``is_first_chunk=True`` (mirroring ``Wan2_1_VAE.encode_chunk``)."""
+ if is_first_chunk:
+ self._enc_cache = self.model.init_encode_cache()
+ with torch.amp.autocast('cuda', dtype=self.dtype), torch.no_grad():
+ out = []
+ for u in videos:
+ x = self._to_ntchw_01(u).to(self.device)
+ z = self.model.encode_chunk(
+ x, self._enc_cache, is_first_chunk=is_first_chunk)
+ out.append(self._lat_from_ntchw(z).float())
+ return out
+
+ def decode_chunk(self, zs, is_first_chunk: bool = False):
+ """Streaming decode. Same list I/O as :meth:`decode`. The adapter
+ holds the decoder cache internally and re-arms it when
+ ``is_first_chunk=True`` (mirroring ``Wan2_1_VAE.decode_chunk``)."""
+ if is_first_chunk:
+ self._dec_cache = self.model.init_decode_cache()
+ with torch.amp.autocast('cuda', dtype=self.dtype), torch.no_grad():
+ out = []
+ for u in zs:
+ z = self._lat_to_ntchw(u).to(self.device)
+ x = self.model.decode_chunk(
+ z, self._dec_cache, is_first_chunk=is_first_chunk)
+ out.append(self._from_ntchw_01(x).float())
+ return out
diff --git a/worldplay2/vae/interface.py b/worldplay2/vae/interface.py
new file mode 100644
index 0000000..1305b0f
--- /dev/null
+++ b/worldplay2/vae/interface.py
@@ -0,0 +1,14 @@
+from typing import Protocol
+
+import torch
+
+
+class StreamingVAE(Protocol):
+ def encode(self, videos: list[torch.Tensor]) -> list[torch.Tensor]: ...
+ def decode(self, latents: list[torch.Tensor]) -> list[torch.Tensor]: ...
+ def encode_chunk(
+ self, videos: list[torch.Tensor], is_first_chunk: bool = False
+ ) -> list[torch.Tensor]: ...
+ def decode_chunk(
+ self, latents: list[torch.Tensor], is_first_chunk: bool = False
+ ) -> list[torch.Tensor]: ...
diff --git a/worldplay2/vae/wan21.py b/worldplay2/vae/wan21.py
new file mode 100644
index 0000000..ac0925d
--- /dev/null
+++ b/worldplay2/vae/wan21.py
@@ -0,0 +1,812 @@
+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
+import logging
+
+import torch
+import torch.cuda.amp as amp
+import torch.nn as nn
+import torch.nn.functional as F
+from einops import rearrange
+
+__all__ = [
+ 'Wan2_1_VAE',
+]
+
+CACHE_T = 2
+
+
+class CausalConv3d(nn.Conv3d):
+ """
+ Causal 3d convolusion.
+ """
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self._padding = (self.padding[2], self.padding[2], self.padding[1],
+ self.padding[1], 2 * self.padding[0], 0)
+ self.padding = (0, 0, 0)
+
+ def forward(self, x, cache_x=None):
+ padding = list(self._padding)
+ if cache_x is not None and self._padding[4] > 0:
+ cache_x = cache_x.to(x.device)
+ x = torch.cat([cache_x, x], dim=2)
+ padding[4] -= cache_x.shape[2]
+ x = F.pad(x, padding)
+
+ return super().forward(x)
+
+
+class RMS_norm(nn.Module):
+
+ def __init__(self, dim, channel_first=True, images=True, bias=False):
+ super().__init__()
+ broadcastable_dims = (1, 1, 1) if not images else (1, 1)
+ shape = (dim, *broadcastable_dims) if channel_first else (dim,)
+
+ self.channel_first = channel_first
+ self.scale = dim**0.5
+ self.gamma = nn.Parameter(torch.ones(shape))
+ self.bias = nn.Parameter(torch.zeros(shape)) if bias else 0.
+
+ def forward(self, x):
+ return F.normalize(
+ x, dim=(1 if self.channel_first else
+ -1)) * self.scale * self.gamma + self.bias
+
+
+class Upsample(nn.Upsample):
+
+ def forward(self, x):
+ """
+ Fix bfloat16 support for nearest neighbor interpolation.
+ """
+ return super().forward(x.float()).type_as(x)
+
+
+class Resample(nn.Module):
+
+ def __init__(self, dim, mode):
+ assert mode in ('none', 'upsample2d', 'upsample3d', 'downsample2d',
+ 'downsample3d')
+ super().__init__()
+ self.dim = dim
+ self.mode = mode
+
+ # layers
+ if mode == 'upsample2d':
+ self.resample = nn.Sequential(
+ Upsample(scale_factor=(2., 2.), mode='nearest-exact'),
+ nn.Conv2d(dim, dim // 2, 3, padding=1))
+ elif mode == 'upsample3d':
+ self.resample = nn.Sequential(
+ Upsample(scale_factor=(2., 2.), mode='nearest-exact'),
+ nn.Conv2d(dim, dim // 2, 3, padding=1))
+ self.time_conv = CausalConv3d(
+ dim, dim * 2, (3, 1, 1), padding=(1, 0, 0))
+
+ elif mode == 'downsample2d':
+ self.resample = nn.Sequential(
+ nn.ZeroPad2d((0, 1, 0, 1)),
+ nn.Conv2d(dim, dim, 3, stride=(2, 2)))
+ elif mode == 'downsample3d':
+ self.resample = nn.Sequential(
+ nn.ZeroPad2d((0, 1, 0, 1)),
+ nn.Conv2d(dim, dim, 3, stride=(2, 2)))
+ self.time_conv = CausalConv3d(
+ dim, dim, (3, 1, 1), stride=(2, 1, 1), padding=(0, 0, 0))
+
+ else:
+ self.resample = nn.Identity()
+
+ def forward(self, x, feat_cache=None, feat_idx=[0]):
+ b, c, t, h, w = x.size()
+ if self.mode == 'upsample3d':
+ if feat_cache is not None:
+ idx = feat_idx[0]
+ if feat_cache[idx] is None:
+ feat_cache[idx] = 'Rep'
+ feat_idx[0] += 1
+ else:
+
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[
+ idx] is not None and feat_cache[idx] != 'Rep':
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ if cache_x.shape[2] < 2 and feat_cache[
+ idx] is not None and feat_cache[idx] == 'Rep':
+ cache_x = torch.cat([
+ torch.zeros_like(cache_x).to(cache_x.device),
+ cache_x
+ ],
+ dim=2)
+ if feat_cache[idx] == 'Rep':
+ x = self.time_conv(x)
+ else:
+ x = self.time_conv(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+
+ x = x.reshape(b, 2, c, t, h, w)
+ x = torch.stack((x[:, 0, :, :, :, :], x[:, 1, :, :, :, :]),
+ 3)
+ x = x.reshape(b, c, t * 2, h, w)
+ t = x.shape[2]
+ x = rearrange(x, 'b c t h w -> (b t) c h w')
+ x = self.resample(x)
+ x = rearrange(x, '(b t) c h w -> b c t h w', t=t)
+
+ if self.mode == 'downsample3d':
+ if feat_cache is not None:
+ idx = feat_idx[0]
+ if feat_cache[idx] is None:
+ feat_cache[idx] = x.clone()
+ feat_idx[0] += 1
+ else:
+
+ cache_x = x[:, :, -1:, :, :].clone()
+ # if cache_x.shape[2] < 2 and feat_cache[idx] is not None and feat_cache[idx]!='Rep':
+ # # cache last frame of last two chunk
+ # cache_x = torch.cat([feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(cache_x.device), cache_x], dim=2)
+
+ x = self.time_conv(
+ torch.cat([feat_cache[idx][:, :, -1:, :, :], x], 2))
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ return x
+
+ def init_weight(self, conv):
+ conv_weight = conv.weight
+ nn.init.zeros_(conv_weight)
+ c1, c2, t, h, w = conv_weight.size()
+ one_matrix = torch.eye(c1, c2)
+ init_matrix = one_matrix
+ nn.init.zeros_(conv_weight)
+ #conv_weight.data[:,:,-1,1,1] = init_matrix * 0.5
+ conv_weight.data[:, :, 1, 0, 0] = init_matrix #* 0.5
+ conv.weight.data.copy_(conv_weight)
+ nn.init.zeros_(conv.bias.data)
+
+ def init_weight2(self, conv):
+ conv_weight = conv.weight.data
+ nn.init.zeros_(conv_weight)
+ c1, c2, t, h, w = conv_weight.size()
+ init_matrix = torch.eye(c1 // 2, c2)
+ #init_matrix = repeat(init_matrix, 'o ... -> (o 2) ...').permute(1,0,2).contiguous().reshape(c1,c2)
+ conv_weight[:c1 // 2, :, -1, 0, 0] = init_matrix
+ conv_weight[c1 // 2:, :, -1, 0, 0] = init_matrix
+ conv.weight.data.copy_(conv_weight)
+ nn.init.zeros_(conv.bias.data)
+
+
+class ResidualBlock(nn.Module):
+
+ def __init__(self, in_dim, out_dim, dropout=0.0):
+ super().__init__()
+ self.in_dim = in_dim
+ self.out_dim = out_dim
+
+ # layers
+ self.residual = nn.Sequential(
+ RMS_norm(in_dim, images=False), nn.SiLU(),
+ CausalConv3d(in_dim, out_dim, 3, padding=1),
+ RMS_norm(out_dim, images=False), nn.SiLU(), nn.Dropout(dropout),
+ CausalConv3d(out_dim, out_dim, 3, padding=1))
+ self.shortcut = CausalConv3d(in_dim, out_dim, 1) \
+ if in_dim != out_dim else nn.Identity()
+
+ def forward(self, x, feat_cache=None, feat_idx=[0]):
+ h = self.shortcut(x)
+ for layer in self.residual:
+ if isinstance(layer, CausalConv3d) and feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ x = layer(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = layer(x)
+ return x + h
+
+
+class AttentionBlock(nn.Module):
+ """
+ Causal self-attention with a single head.
+ """
+
+ def __init__(self, dim):
+ super().__init__()
+ self.dim = dim
+
+ # layers
+ self.norm = RMS_norm(dim)
+ self.to_qkv = nn.Conv2d(dim, dim * 3, 1)
+ self.proj = nn.Conv2d(dim, dim, 1)
+
+ # zero out the last layer params
+ nn.init.zeros_(self.proj.weight)
+
+ def forward(self, x):
+ identity = x
+ b, c, t, h, w = x.size()
+ x = rearrange(x, 'b c t h w -> (b t) c h w')
+ x = self.norm(x)
+ # compute query, key, value
+ q, k, v = self.to_qkv(x).reshape(b * t, 1, c * 3,
+ -1).permute(0, 1, 3,
+ 2).contiguous().chunk(
+ 3, dim=-1)
+
+ # apply attention
+ x = F.scaled_dot_product_attention(
+ q,
+ k,
+ v,
+ )
+ x = x.squeeze(1).permute(0, 2, 1).reshape(b * t, c, h, w)
+
+ # output
+ x = self.proj(x)
+ x = rearrange(x, '(b t) c h w-> b c t h w', t=t)
+ return x + identity
+
+
+class Encoder3d(nn.Module):
+
+ def __init__(self,
+ dim=128,
+ z_dim=4,
+ dim_mult=[1, 2, 4, 4],
+ num_res_blocks=2,
+ attn_scales=[],
+ temperal_downsample=[True, True, False],
+ dropout=0.0):
+ super().__init__()
+ self.dim = dim
+ self.z_dim = z_dim
+ self.dim_mult = dim_mult
+ self.num_res_blocks = num_res_blocks
+ self.attn_scales = attn_scales
+ self.temperal_downsample = temperal_downsample
+
+ # dimensions
+ dims = [dim * u for u in [1] + dim_mult]
+ scale = 1.0
+
+ # init block
+ self.conv1 = CausalConv3d(3, dims[0], 3, padding=1)
+
+ # downsample blocks
+ downsamples = []
+ for i, (in_dim, out_dim) in enumerate(zip(dims[:-1], dims[1:])):
+ # residual (+attention) blocks
+ for _ in range(num_res_blocks):
+ downsamples.append(ResidualBlock(in_dim, out_dim, dropout))
+ if scale in attn_scales:
+ downsamples.append(AttentionBlock(out_dim))
+ in_dim = out_dim
+
+ # downsample block
+ if i != len(dim_mult) - 1:
+ mode = 'downsample3d' if temperal_downsample[
+ i] else 'downsample2d'
+ downsamples.append(Resample(out_dim, mode=mode))
+ scale /= 2.0
+ self.downsamples = nn.Sequential(*downsamples)
+
+ # middle blocks
+ self.middle = nn.Sequential(
+ ResidualBlock(out_dim, out_dim, dropout), AttentionBlock(out_dim),
+ ResidualBlock(out_dim, out_dim, dropout))
+
+ # output blocks
+ self.head = nn.Sequential(
+ RMS_norm(out_dim, images=False), nn.SiLU(),
+ CausalConv3d(out_dim, z_dim, 3, padding=1))
+
+ def forward(self, x, feat_cache=None, feat_idx=[0]):
+ if feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ x = self.conv1(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = self.conv1(x)
+
+ ## downsamples
+ for layer in self.downsamples:
+ if feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## middle
+ for layer in self.middle:
+ if isinstance(layer, ResidualBlock) and feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## head
+ for layer in self.head:
+ if isinstance(layer, CausalConv3d) and feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ x = layer(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = layer(x)
+ return x
+
+
+class Decoder3d(nn.Module):
+
+ def __init__(self,
+ dim=128,
+ z_dim=4,
+ dim_mult=[1, 2, 4, 4],
+ num_res_blocks=2,
+ attn_scales=[],
+ temperal_upsample=[False, True, True],
+ dropout=0.0):
+ super().__init__()
+ self.dim = dim
+ self.z_dim = z_dim
+ self.dim_mult = dim_mult
+ self.num_res_blocks = num_res_blocks
+ self.attn_scales = attn_scales
+ self.temperal_upsample = temperal_upsample
+
+ # dimensions
+ dims = [dim * u for u in [dim_mult[-1]] + dim_mult[::-1]]
+ scale = 1.0 / 2**(len(dim_mult) - 2)
+
+ # init block
+ self.conv1 = CausalConv3d(z_dim, dims[0], 3, padding=1)
+
+ # middle blocks
+ self.middle = nn.Sequential(
+ ResidualBlock(dims[0], dims[0], dropout), AttentionBlock(dims[0]),
+ ResidualBlock(dims[0], dims[0], dropout))
+
+ # upsample blocks
+ upsamples = []
+ for i, (in_dim, out_dim) in enumerate(zip(dims[:-1], dims[1:])):
+ # residual (+attention) blocks
+ if i == 1 or i == 2 or i == 3:
+ in_dim = in_dim // 2
+ for _ in range(num_res_blocks + 1):
+ upsamples.append(ResidualBlock(in_dim, out_dim, dropout))
+ if scale in attn_scales:
+ upsamples.append(AttentionBlock(out_dim))
+ in_dim = out_dim
+
+ # upsample block
+ if i != len(dim_mult) - 1:
+ mode = 'upsample3d' if temperal_upsample[i] else 'upsample2d'
+ upsamples.append(Resample(out_dim, mode=mode))
+ scale *= 2.0
+ self.upsamples = nn.Sequential(*upsamples)
+
+ # output blocks
+ self.head = nn.Sequential(
+ RMS_norm(out_dim, images=False), nn.SiLU(),
+ CausalConv3d(out_dim, 3, 3, padding=1))
+
+ def forward(self, x, feat_cache=None, feat_idx=[0]):
+ ## conv1
+ if feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ x = self.conv1(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = self.conv1(x)
+
+ ## middle
+ for layer in self.middle:
+ if isinstance(layer, ResidualBlock) and feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## upsamples
+ for layer in self.upsamples:
+ if feat_cache is not None:
+ x = layer(x, feat_cache, feat_idx)
+ else:
+ x = layer(x)
+
+ ## head
+ for layer in self.head:
+ if isinstance(layer, CausalConv3d) and feat_cache is not None:
+ idx = feat_idx[0]
+ cache_x = x[:, :, -CACHE_T:, :, :].clone()
+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:
+ # cache last frame of last two chunk
+ cache_x = torch.cat([
+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(
+ cache_x.device), cache_x
+ ],
+ dim=2)
+ x = layer(x, feat_cache[idx])
+ feat_cache[idx] = cache_x
+ feat_idx[0] += 1
+ else:
+ x = layer(x)
+ return x
+
+
+def count_conv3d(model):
+ count = 0
+ for m in model.modules():
+ if isinstance(m, CausalConv3d):
+ count += 1
+ return count
+
+
+class WanVAE_(nn.Module):
+
+ def __init__(self,
+ dim=128,
+ z_dim=4,
+ dim_mult=[1, 2, 4, 4],
+ num_res_blocks=2,
+ attn_scales=[],
+ temperal_downsample=[True, True, False],
+ dropout=0.0):
+ super().__init__()
+ self.dim = dim
+ self.z_dim = z_dim
+ self.dim_mult = dim_mult
+ self.num_res_blocks = num_res_blocks
+ self.attn_scales = attn_scales
+ self.temperal_downsample = temperal_downsample
+ self.temperal_upsample = temperal_downsample[::-1]
+
+ # modules
+ self.encoder = Encoder3d(dim, z_dim * 2, dim_mult, num_res_blocks,
+ attn_scales, self.temperal_downsample, dropout)
+ self.conv1 = CausalConv3d(z_dim * 2, z_dim * 2, 1)
+ self.conv2 = CausalConv3d(z_dim, z_dim, 1)
+ self.decoder = Decoder3d(dim, z_dim, dim_mult, num_res_blocks,
+ attn_scales, self.temperal_upsample, dropout)
+
+ def forward(self, x):
+ mu, log_var = self.encode(x)
+ z = self.reparameterize(mu, log_var)
+ x_recon = self.decode(z)
+ return x_recon, mu, log_var
+
+ def encode(self, x, scale):
+ self.clear_encode_cache()
+ ## cache
+ t = x.shape[2]
+ iter_ = 1 + (t - 1) // 4
+ ## 对encode输入的x,按时间拆分为1、4、4、4....
+ for i in range(iter_):
+ self._enc_conv_idx = [0]
+ if i == 0:
+ out = self.encoder(
+ x[:, :, :1, :, :],
+ feat_cache=self._enc_feat_map,
+ feat_idx=self._enc_conv_idx)
+ else:
+ out_ = self.encoder(
+ x[:, :, 1 + 4 * (i - 1):1 + 4 * i, :, :],
+ feat_cache=self._enc_feat_map,
+ feat_idx=self._enc_conv_idx)
+ out = torch.cat([out, out_], 2)
+ mu, log_var = self.conv1(out).chunk(2, dim=1)
+ if isinstance(scale[0], torch.Tensor):
+ mu = (mu - scale[0].view(1, self.z_dim, 1, 1, 1)) * scale[1].view(
+ 1, self.z_dim, 1, 1, 1)
+ else:
+ mu = (mu - scale[0]) * scale[1]
+ self.clear_encode_cache()
+ return mu
+
+ def decode(self, z, scale):
+ self.clear_decode_cache()
+ # z: [b,c,t,h,w]
+ if isinstance(scale[0], torch.Tensor):
+ z = z / scale[1].view(1, self.z_dim, 1, 1, 1) + scale[0].view(
+ 1, self.z_dim, 1, 1, 1)
+ else:
+ z = z / scale[1] + scale[0]
+ iter_ = z.shape[2]
+ x = self.conv2(z)
+ for i in range(iter_):
+ self._conv_idx = [0]
+ if i == 0:
+ out = self.decoder(
+ x[:, :, i:i + 1, :, :],
+ feat_cache=self._feat_map,
+ feat_idx=self._conv_idx)
+ else:
+ out_ = self.decoder(
+ x[:, :, i:i + 1, :, :],
+ feat_cache=self._feat_map,
+ feat_idx=self._conv_idx)
+ out = torch.cat([out, out_], 2)
+ self.clear_decode_cache()
+ return out
+
+ def encode_chunk(self, x, scale, is_first_chunk: bool = False):
+ """Streaming encode for chunked inference.
+
+ Unlike :meth:`encode`, this does **not** ``clear_cache()`` between
+ calls; the per-layer ``_enc_feat_map`` accumulated by previous
+ chunks is reused so the causal convolutions see the prior temporal
+ state.
+
+ Frame-to-latent layout:
+ - is_first_chunk=True : first pixel frame -> 1 latent, then
+ groups of 4 pixel frames -> 1 latent each. ``iter_ = 1 + (T-1)//4``.
+ - is_first_chunk=False : all input frames are continuation
+ frames coming in groups of 4. ``iter_ = T // 4``.
+
+ Args:
+ x: pixel tensor of shape [B, C, T, H, W].
+ scale: (mean, 1/std) tuple matching :meth:`encode`.
+ is_first_chunk: True for the very first chunk; resets the
+ encoder cache before processing.
+ """
+ if is_first_chunk:
+ self.clear_encode_cache()
+
+ t = x.shape[2]
+ if is_first_chunk:
+ iter_ = 1 + (t - 1) // 4
+ else:
+ assert t % 4 == 0, (
+ f"encode_chunk(is_first_chunk=False) requires the number of "
+ f"pixel frames to be a multiple of 4; got T={t}.")
+ iter_ = t // 4
+
+ out = None
+ for i in range(iter_):
+ self._enc_conv_idx = [0]
+ if is_first_chunk:
+ if i == 0:
+ frame_slice = x[:, :, :1, :, :]
+ else:
+ frame_slice = x[:, :, 1 + 4 * (i - 1):1 + 4 * i, :, :]
+ else:
+ frame_slice = x[:, :, 4 * i:4 * (i + 1), :, :]
+
+ chunk_out = self.encoder(
+ frame_slice,
+ feat_cache=self._enc_feat_map,
+ feat_idx=self._enc_conv_idx)
+ out = chunk_out if out is None else torch.cat([out, chunk_out], 2)
+
+ mu, log_var = self.conv1(out).chunk(2, dim=1)
+ if isinstance(scale[0], torch.Tensor):
+ mu = (mu - scale[0].view(1, self.z_dim, 1, 1, 1)) * scale[1].view(
+ 1, self.z_dim, 1, 1, 1)
+ else:
+ mu = (mu - scale[0]) * scale[1]
+ # NOTE: cache is intentionally NOT cleared at the end so the next
+ # chunk continues the streaming state.
+ return mu
+
+ def decode_chunk(self, z, scale, is_first_chunk: bool = False):
+ """Streaming decode for chunked inference.
+
+ Unlike :meth:`decode`, this does **not** ``clear_cache()`` between
+ calls. ``is_first_chunk=True`` resets the decoder cache once at
+ the very first chunk; subsequent calls reuse the cached state so
+ the causal temporal convolutions stay consistent across chunk
+ boundaries.
+
+ Args:
+ z: latent tensor of shape [B, C, T_lat, H, W] for a single chunk.
+ scale: (mean, 1/std) tuple matching :meth:`decode`.
+ is_first_chunk: True for the very first chunk.
+
+ Returns:
+ Tensor of shape [B, 3, T_pix, H_pix, W_pix]. Not clamped here
+ (the public wrapper :meth:`Wan2_1_VAE.decode_chunk` clamps to
+ [-1, 1] to mirror :meth:`Wan2_1_VAE.decode`).
+ """
+ if is_first_chunk:
+ self.clear_decode_cache()
+
+ if isinstance(scale[0], torch.Tensor):
+ z = z / scale[1].view(1, self.z_dim, 1, 1, 1) + scale[0].view(
+ 1, self.z_dim, 1, 1, 1)
+ else:
+ z = z / scale[1] + scale[0]
+
+ iter_ = z.shape[2]
+ x = self.conv2(z)
+ out = None
+ for i in range(iter_):
+ self._conv_idx = [0]
+ cur = self.decoder(
+ x[:, :, i:i + 1, :, :],
+ feat_cache=self._feat_map,
+ feat_idx=self._conv_idx)
+ out = cur if out is None else torch.cat([out, cur], 2)
+ # NOTE: cache is intentionally NOT cleared at the end.
+ return out
+
+ def reparameterize(self, mu, log_var):
+ std = torch.exp(0.5 * log_var)
+ eps = torch.randn_like(std)
+ return eps * std + mu
+
+ def sample(self, imgs, deterministic=False):
+ mu, log_var = self.encode(imgs)
+ if deterministic:
+ return mu
+ std = torch.exp(0.5 * log_var.clamp(-30.0, 20.0))
+ return mu + std * torch.randn_like(std)
+
+ def clear_cache(self):
+ """Reset BOTH encode and decode streaming caches.
+
+ Kept for backwards-compatibility (the non-streaming
+ :meth:`encode` / :meth:`decode` paths use it as a "scratch" reset
+ because they do not interleave with streaming calls). For
+ chunked / streaming inference, prefer :meth:`clear_encode_cache`
+ and :meth:`clear_decode_cache` so that resetting the encode side
+ does not nuke an in-flight decode cache (and vice versa).
+ """
+ self.clear_decode_cache()
+ self.clear_encode_cache()
+
+ def clear_decode_cache(self):
+ self._conv_num = count_conv3d(self.decoder)
+ self._conv_idx = [0]
+ self._feat_map = [None] * self._conv_num
+
+ def clear_encode_cache(self):
+ self._enc_conv_num = count_conv3d(self.encoder)
+ self._enc_conv_idx = [0]
+ self._enc_feat_map = [None] * self._enc_conv_num
+
+
+def _video_vae(pretrained_path=None, z_dim=None, device='cpu', **kwargs):
+ """
+ Autoencoder3d adapted from Stable Diffusion 1.x, 2.x and XL.
+ """
+ # params
+ cfg = dict(
+ dim=96,
+ z_dim=z_dim,
+ dim_mult=[1, 2, 4, 4],
+ num_res_blocks=2,
+ attn_scales=[],
+ temperal_downsample=[False, True, True],
+ dropout=0.0)
+ cfg.update(**kwargs)
+
+ # init model
+ with torch.device('meta'):
+ model = WanVAE_(**cfg)
+
+ # load checkpoint
+ logging.info(f'loading {pretrained_path}')
+ model.load_state_dict(
+ torch.load(pretrained_path, map_location=device), assign=True)
+
+ return model
+
+
+class Wan2_1_VAE:
+
+ def __init__(self,
+ z_dim=16,
+ vae_pth='cache/vae_step_411000.pth',
+ dtype=torch.float,
+ device="cuda"):
+ self.dtype = dtype
+ self.device = device
+
+ mean = [
+ -0.7571, -0.7089, -0.9113, 0.1075, -0.1745, 0.9653, -0.1517, 1.5508,
+ 0.4134, -0.0715, 0.5517, -0.3632, -0.1922, -0.9497, 0.2503, -0.2921
+ ]
+ std = [
+ 2.8184, 1.4541, 2.3275, 2.6558, 1.2196, 1.7708, 2.6052, 2.0743,
+ 3.2687, 2.1526, 2.8652, 1.5579, 1.6382, 1.1253, 2.8251, 1.9160
+ ]
+ self.mean = torch.tensor(mean, dtype=dtype, device=device)
+ self.std = torch.tensor(std, dtype=dtype, device=device)
+ self.scale = [self.mean, 1.0 / self.std]
+
+ # init model
+ self.model = _video_vae(
+ pretrained_path=vae_pth,
+ z_dim=z_dim,
+ ).eval().requires_grad_(False).to(device)
+
+ def encode(self, videos):
+ """
+ videos: A list of videos each with shape [C, T, H, W].
+ """
+ with amp.autocast(dtype=self.dtype):
+ return [
+ self.model.encode(u.unsqueeze(0), self.scale).float().squeeze(0)
+ for u in videos
+ ]
+
+ def decode(self, zs):
+ with amp.autocast(dtype=self.dtype):
+ return [
+ self.model.decode(u.unsqueeze(0),
+ self.scale).float().clamp_(-1, 1).squeeze(0)
+ for u in zs
+ ]
+
+ # ------------------------------------------------------------------
+ # Streaming variants for chunked inference (stage-2 / stage-3).
+ # See ``WanVAE_.encode_chunk`` / ``decode_chunk`` for the cache contract.
+ # ------------------------------------------------------------------
+
+ def encode_chunk(self, videos, is_first_chunk: bool = False):
+ """Streaming encode. ``videos`` is a list of [C, T, H, W] tensors.
+ Returns a list of latent tensors [C_z, T_lat, H/8, W/8].
+
+ Caller is responsible for setting ``is_first_chunk=True`` exactly
+ once at the start of an autoregressive generation and ``False``
+ for every subsequent chunk.
+ """
+ with amp.autocast(dtype=self.dtype):
+ return [
+ self.model.encode_chunk(
+ u.unsqueeze(0), self.scale,
+ is_first_chunk=is_first_chunk).float().squeeze(0)
+ for u in videos
+ ]
+
+ def decode_chunk(self, zs, is_first_chunk: bool = False):
+ """Streaming decode. ``zs`` is a list of [C_z, T_lat, H, W] latent
+ tensors. Returns a list of pixel tensors [3, T_pix, H_pix, W_pix]
+ clamped to [-1, 1] (matching :meth:`decode`).
+ """
+ with amp.autocast(dtype=self.dtype):
+ return [
+ self.model.decode_chunk(
+ u.unsqueeze(0), self.scale,
+ is_first_chunk=is_first_chunk
+ ).float().clamp_(-1, 1).squeeze(0)
+ for u in zs
+ ]
diff --git a/worldplay2/video.py b/worldplay2/video.py
new file mode 100644
index 0000000..320fbf9
--- /dev/null
+++ b/worldplay2/video.py
@@ -0,0 +1,22 @@
+from pathlib import Path
+
+import imageio
+import torch
+
+
+def save_video(tensor: torch.Tensor, path: str | Path, fps: int) -> None:
+ """Save ``[3,T,H,W]`` video in ``[-1,1]``."""
+ output = Path(path)
+ output.parent.mkdir(parents=True, exist_ok=True)
+ frames = (
+ tensor.detach()
+ .float()
+ .clamp(-1, 1)
+ .add(1)
+ .mul(127.5)
+ .byte()
+ .permute(1, 2, 3, 0)
+ .cpu()
+ .numpy()
+ )
+ imageio.mimsave(output, frames, fps=fps, codec="libx264", quality=8)