WorldPlay2: Extending Real-Time Interactive World Models in Control and Horizon
Python
release
aejionzhang6bb655c701e7
diff --git a/LICENSE.txt b/LICENSE.txtnew file mode 100644index 0000000..4c3c3a9--- /dev/null+++ b/LICENSE.txt@@ -0,0 +1 @@+license: cc-by-nc-4.0\ No newline at end of filediff --git a/README.md b/README.mdindex ae9b5d4..6ae35a1 100644--- a/README.md+++ b/README.md@@ -4,13 +4,190 @@<div align="center"><a href=https://worldplay2.github.io/ target="_blank"><img src=https://img.shields.io/badge/Project%20Page-333399.svg?logo=homepage height=22px></a>- <a href=https://arxiv.org/abs/xxxx.xxxxx target="_blank"><img src=https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv height=22px></a>+ <a href=https://arxiv.org/abs/2609.35560 target="_blank"><img src=https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv height=22px></a><a href=https://www.youtube.com/watch?v=_pFzHHslhgc target="_blank"><img src=https://img.shields.io/badge/YouTube%20Video-FF0000.svg?logo=youtube height=22px></a></div>-We are still organizing the code and weights. Thank you all for your patience!-## 🎥 Videohttps://github.com/user-attachments/assets/3264efa7-ef6a-4c7b-a043-d819264a93bd++## 🛠️ Installation++Requires Python 3.10+ and a CUDA-capable NVIDIA GPU.++Create and activate a Conda environment, then install the dependencies:++```bash+conda create -n worldplay2 python=3.10 -y+conda activate worldplay2+python -m pip install -r requirements.txt+```++Install `flash_attn`:++```bash+pip install flash-attn --no-build-isolation+```++#### Optional SageAttention++Install SageAttention in your CUDA environment:++```bash+pip install sageattention==2.2.0 --no-build-isolation+```++All three launchers support `ATTENTION_BACKEND` (default: `flash`). For direct+CLI usage, pass `--attention_backend sage`.++#### Optional FP8++Install the additional dependencies on supported hardware:++```bash+pip install transformer_engine[pytorch]+```++After setting the model and input paths in `run_few_step.sh` or exporting them+as environment variables, enable FP8:++```bash+ENABLE_FP8_FFN=true ENABLE_FP8_QKV=true USE_DIT_FSDP=false bash run_few_step.sh+```++## 🧱 Model Download++### Base Model++Download [Wan2.2 I2V-A14B](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B)+and set `CKPT_DIR` to its local directory.++### WorldPlay2 Models++Download the WorldPlay2 checkpoints for your inference mode. Set+`LOW_NOISE_CKPT` and `HIGH_NOISE_CKPT` to the corresponding checkpoint files.++<!-- TODO: Replace each # below with the corresponding Hugging Face model URL. -->++| Model | Model Type | Default steps | Download Links |+|---------------------|-------------------------------------|---------------| --- |+| **WorldPlay2-Fast** | Autoregressive, 4-step (`few_step`) | 4 | [🤗 Hugging Face](https://huggingface.co/aejion/WorldPlay2-Fast) |+| **WorldPlay2-BI** | Bidirectional (`bi`) | 40 | [🤗 Hugging Face](https://huggingface.co/aejion/WorldPlay2-BI) |+| **WorldPlay2-AR** | Autoregressive (`ar`) | 40 | [🤗 Hugging Face](https://huggingface.co/aejion/WorldPlay2-AR) |+| **WorldPlay2-TAE** | Causal VAE | --- | (coming soon) |++Download the base model and WorldPlay2 few-step model using the Hugging Face CLI.++```bash+pip install "huggingface_hub[cli]>=0.34,<1.0"++# Base Wan2.2 model+hf download Wan-AI/Wan2.2-I2V-A14B --local-dir ./checkpoints/Wan2.2-I2V-A14B++# WorldPlay2 models+hf download aejion/WorldPlay2-Fast --local-dir ./checkpoints/WorldPlay2-Fast+hf download aejion/WorldPlay2-BI --local-dir ./checkpoints/WorldPlay2-BI+hf download aejion/WorldPlay2-AR --local-dir ./checkpoints/WorldPlay2-AR+```++Set `CKPT_DIR` to `./checkpoints/Wan2.2-I2V-A14B`, and set `LOW_NOISE_CKPT`+and `HIGH_NOISE_CKPT` to the checkpoint files in `./checkpoints/WorldPlay2-Fast`.++## 🎮 Quick Start++Set the paths to the WorldPlay2 few-step checkpoints and run from the repository root:++```bash+# CKPT_DIR: path to the original Wan2.2 I2V-A14B base model directory.+# LOW_NOISE_CKPT: path to the WorldPlay2 few-step low-noise checkpoint.+# HIGH_NOISE_CKPT: path to the WorldPlay2 few-step high-noise checkpoint.+CKPT_DIR=/path/to/model \+LOW_NOISE_CKPT=/path/to/model/low_noise_model.pt \+HIGH_NOISE_CKPT=/path/to/model/high_noise_model.pt \+INPUT_JSON=/path/to/input.json \+OUTPUT_PATH=./outputs/few_step \+bash run_few_step.sh+```++Use `run_ar.sh` or `run_bi.sh` for the other modes, with their corresponding+checkpoints. Launchers default to 8 GPUs with sequence parallelism and FSDP;+set `NPROC_PER_NODE` to match your GPU count. For one GPU, also set+`USE_DIT_FSDP=false USE_T5_FSDP=false`.++All input items are processed, and videos are saved as `<output_name>.mp4` in+`OUTPUT_PATH`.++## 📖 Input Format++Provide a JSON array. Relative image paths are resolved from the JSON file's+directory.++```json+[+ {+ "image_path": "input.jpg",+ "perspective": "tps",+ "action": "w-32,s-32",+ "prompt_event": "prompt1-32,prompt2-32",+ "prompt1": "The scene in the video is: a quiet forest trail. The character is: a hiker wearing a blue jacket. The event is: dark clouds gather and rain begins to fall.",+ "prompt2": "The scene in the video is: a quiet forest trail. The character is: a hiker wearing a blue jacket. The event is: the rain stops and sunlight breaks through the clouds.",+ "output_name": "sample_000"+ }+]+```++- **Perspective:** `fps` or `tps`.+- **Actions:** comma-separated `action-duration` commands. Movement actions are+ `w`, `s`, `a`, `d`, `wa`, `wd`, `sa`, `sd`; rotation actions are `left`, `right`,+ `up`, `down`; special action is `space`. Combine tokens with `+`, for example `right+space-32`.+- **Durations:** measured in latent frames; the first latent is stationary.+ Action and prompt durations must have the same total. The total and prompt+ boundaries must align with `chunk_length` (4 in the few-step and AR launchers,+ 32 in BI).++#### Prompt Format++Construct each prompt field (`prompt1`, `prompt2`, etc.) using the template for+your perspective, replacing the placeholders with your descriptions.++**FPS (`fps`):** omit the `The character is: ...` sentence.++```text+The scene in the video is: {scene description}. The event is: {event description}.+```++**TPS (`tps`):** include the character description, as in the JSON example above.++```text+The scene in the video is: {scene description}. The character is: {character description}. The event is: {event description}.+```++The scene description is required for both perspectives; the character+description is only required for TPS. The event is optional for both: omit+the entire `The event is: ...` sentence when it is not needed. For example,+an FPS prompt without an event is:++```text+The scene in the video is: a quiet forest trail.+```++The `prompt_event` scheduling field is still required, even when prompts omit+the event description.++## 🔑 Configuration++Paths and launcher settings can be overridden through environment variables:++| Variable | Purpose |+| --- | --- |+| `NPROC_PER_NODE` | Number of GPUs |+| `ATTENTION_BACKEND` | Visual self-attention: `flash` (default) or `sage` |+| `SAMPLE_STEPS` | Sampling steps; few-step inference requires 4 |+| `GUIDE_SCALE` | Guidance scale for BI/AR |+| `YAW_ROTATION_SPEED_DEG` | Left/right rotation per latent |+| `PITCH_ROTATION_SPEED_DEG` | Up/down rotation per latent |++## 🙏 Acknowledgements+We would like to thank [HY-World 1.5](https://github.com/Tencent-Hunyuan/HY-WorldPlay), [HY-World 2.0](https://github.com/Tencent-Hunyuan/HY-World-2.0), and [HunyuanWorld-Mirror](https://github.com/Tencent-Hunyuan/HunyuanWorld-Mirror) for their great work.diff --git a/configs/__init__.py b/configs/__init__.pynew file mode 100644index 0000000..527fffd--- /dev/null+++ b/configs/__init__.py@@ -0,0 +1,3 @@+from .i2v_a14b import I2V_A14B_CONFIG++__all__ = ["I2V_A14B_CONFIG"]diff --git a/configs/i2v_a14b.py b/configs/i2v_a14b.pynew file mode 100644index 0000000..ce5ce65--- /dev/null+++ b/configs/i2v_a14b.py@@ -0,0 +1,32 @@+from dataclasses import dataclass++import torch+++@dataclass(frozen=True)+class I2VA14BConfig:+ text_len: int = 512+ t5_dtype: torch.dtype = torch.bfloat16+ param_dtype: torch.dtype = torch.bfloat16+ num_train_timesteps: int = 1000+ boundary: float = 0.9+ sample_shift: float = 5.0+ sample_guide_scale: tuple[float, float] = (3.5, 3.5)+ sample_neg_prompt: str = (+ "色调艳丽,过曝,静态,细节模糊不清,字幕,风格,作品,画作,"+ "画面,静止,整体发灰,最差质量,低质量,JPEG压缩残留,丑陋的,"+ "残缺的,多余的手指,画得不好的手部,画得不好的脸部,畸形的,"+ "毁容的,形态畸形的肢体,手指融合,静止不动的画面,杂乱的背景,"+ "三条腿,背景人很多,倒着走"+ )+ t5_checkpoint: str = "models_t5_umt5-xxl-enc-bf16.pth"+ t5_tokenizer: str = "google/umt5-xxl"+ vae_checkpoint: str = "Wan2.1_VAE.pth"+ low_noise_checkpoint: str = "low_noise_model"+ high_noise_checkpoint: str = "high_noise_model"+ vae_stride: tuple[int, int, int] = (4, 8, 8)+ patch_size: tuple[int, int, int] = (1, 2, 2)+ sample_fps: int = 16+++I2V_A14B_CONFIG = I2VA14BConfig()diff --git a/generate.py b/generate.pynew file mode 100644index 0000000..50c40de--- /dev/null+++ b/generate.py@@ -0,0 +1,233 @@+#!/usr/bin/env python3+from __future__ import annotations++import argparse+import gc+import logging+import sys++import torch+from PIL import Image++from configs import I2V_A14B_CONFIG+from worldplay2.inputs import (+ initialize_runtime,+ load_inference_inputs,+)+from worldplay2.pipeline import InferenceMode, WorldPlay2Pipeline+from worldplay2.video import save_video+++def str2bool(value):+ if isinstance(value, bool):+ return value+ value = value.lower()+ if value in {"1", "true", "yes", "on"}:+ return True+ if value in {"0", "false", "no", "off"}:+ return False+ raise argparse.ArgumentTypeError(f"invalid boolean: {value}")+++def parse_args():+ parser = argparse.ArgumentParser(+ description="WorldPlay2 action-conditioned video inference"+ )+ source = parser.add_argument_group("input")+ source.add_argument("--input_json", required=True)+ source.add_argument(+ "--rotation_speed_deg",+ type=float,+ default=3.0,+ help="Fallback rotation degrees per latent for axes without an override.",+ )+ source.add_argument(+ "--yaw_rotation_speed_deg",+ type=float,+ help="Left/right rotation degrees per latent; defaults to rotation_speed_deg.",+ )+ source.add_argument(+ "--pitch_rotation_speed_deg",+ type=float,+ help="Up/down rotation degrees per latent; defaults to rotation_speed_deg.",+ )++ model = parser.add_argument_group("model")+ model.add_argument("--ckpt_dir", required=True)+ model.add_argument("--low_noise_mem_ckpt", required=True)+ model.add_argument("--high_noise_mem_ckpt", required=True)+ model.add_argument(+ "--inference_mode",+ choices=[mode.value for mode in InferenceMode],+ default=InferenceMode.BI.value,+ )+ model.add_argument("--vae_type", choices=["wan2_1", "causal_tae"], default="wan2_1")+ model.add_argument("--tae_ckpt")++ generation = parser.add_argument_group("generation")+ generation.add_argument("--height", type=int, default=448)+ generation.add_argument("--width", type=int, default=832)+ generation.add_argument("--chunk_length", type=int, default=4)+ generation.add_argument("--sink_size", type=int, default=1)+ generation.add_argument("--temporal_size", type=int, default=1)+ generation.add_argument("--sample_steps", type=int)+ generation.add_argument("--sample_shift", type=float)+ generation.add_argument("--sample_guide_scale", type=float)+ generation.add_argument("--sample_solver", choices=["unipc", "dpm++"], default="unipc")+ generation.add_argument("--negative_prompt", default="")+ generation.add_argument("--seed", type=int, default=-1)+ generation.add_argument("--output_dir", default="outputs")+ generation.add_argument(+ "--compile_cache_dir",+ default="~/.cache",+ help="Persistent torch.compile cache directory.",+ )++ runtime = parser.add_argument_group("runtime")+ runtime.add_argument("--offload_model", type=str2bool)+ runtime.add_argument(+ "--attention_backend",+ choices=["flash", "sage"],+ default="flash",+ help="Visual self-attention backend: FlashAttention or SageAttention. "+ "Text cross-attention and masked prefill keep their existing backends.",+ )+ runtime.add_argument("--ulysses_size", type=int, default=1)+ runtime.add_argument(+ "--disable_compiled_qk_rope",+ action="store_true",+ help="Use the eager QK RMSNorm + RoPE kernel.",+ )+ runtime.add_argument(+ "--disable_conditioning_cache",+ action="store_true",+ help="Disable text projection and cross-attention K/V caches.",+ )+ runtime.add_argument(+ "--disable_self_qkv_fusion",+ action="store_true",+ help="Keep separate self-attention Q/K/V projection GEMMs.",+ )+ runtime.add_argument(+ "--disable_compiled_block_glue",+ action="store_true",+ help="Use eager modulation and gated residual expressions.",+ )+ runtime.add_argument(+ "--disable_compiled_ffn",+ action="store_true",+ help="Use the eager Linear-GELU-Linear FFN module.",+ )+ runtime.add_argument(+ "--fp8_ffn",+ action="store_true",+ help="Use Transformer Engine FP8 E4M3 for selected FFN blocks. "+ "Requires DiT FSDP and model offload to be disabled.",+ )+ runtime.add_argument("--fp8_start_block", type=int, default=4)+ runtime.add_argument("--fp8_end_block", type=int, default=35)+ runtime.add_argument(+ "--fp8_qkv",+ action="store_true",+ help="Use Transformer Engine FP8 E4M3 for selected fused "+ "self-attention QKV projections.",+ )+ runtime.add_argument("--fp8_qkv_start_block", type=int, default=4)+ runtime.add_argument("--fp8_qkv_end_block", type=int, default=35)+ runtime.add_argument("--t5_fsdp", action="store_true")+ runtime.add_argument("--dit_fsdp", action="store_true")+ return parser.parse_args()+++def main():+ args = parse_args()+ inference = load_inference_inputs(+ args,+ default_guide_scale=I2V_A14B_CONFIG.sample_guide_scale,+ )+ runtime = initialize_runtime(args)+ mode = InferenceMode(inference.mode)++ logging.basicConfig(+ level=logging.INFO,+ format="[%(asctime)s] %(levelname)s: %(message)s",+ handlers=[+ logging.StreamHandler(stream=sys.stdout)+ if runtime.rank == 0+ else logging.NullHandler()+ ],+ force=True,+ )++ pipeline = WorldPlay2Pipeline(+ config=I2V_A14B_CONFIG,+ checkpoint_dir=args.ckpt_dir,+ low_noise_mem_ckpt=args.low_noise_mem_ckpt,+ high_noise_mem_ckpt=args.high_noise_mem_ckpt,+ device_id=runtime.local_rank,+ rank=runtime.rank,+ t5_fsdp=args.t5_fsdp,+ dit_fsdp=args.dit_fsdp,+ use_sp=args.ulysses_size > 1,+ attention_backend=args.attention_backend,+ compile_qk_rope=not args.disable_compiled_qk_rope,+ conditioning_cache=not args.disable_conditioning_cache,+ fuse_self_qkv=not args.disable_self_qkv_fusion,+ compile_block_glue=not args.disable_compiled_block_glue,+ compile_ffn=not args.disable_compiled_ffn,+ fp8_ffn=args.fp8_ffn,+ fp8_start_block=args.fp8_start_block,+ fp8_end_block=args.fp8_end_block,+ fp8_qkv=args.fp8_qkv,+ fp8_qkv_start_block=args.fp8_qkv_start_block,+ fp8_qkv_end_block=args.fp8_qkv_end_block,+ inference_mode=mode,+ vae_type=args.vae_type,+ tae_ckpt=args.tae_ckpt,+ )+ # FSDP construction can temporarily materialize full parameters. Clear+ # allocator cache so nvidia-smi reflects persistent usage rather than+ # stale reserved blocks from initialization.+ gc.collect()+ torch.cuda.empty_cache()+ logging.info(+ "FSDP requested: t5=%s dit=%s; wrapper counts: t5=%d low=%d high=%d",+ args.t5_fsdp,+ args.dit_fsdp,+ pipeline.dtype_report["t5_fsdp_units"],+ pipeline.dtype_report["low_noise_fsdp_units"],+ pipeline.dtype_report["high_noise_fsdp_units"],+ )++ for item in inference.items:+ image = Image.open(item.image).convert("RGB")+ video = pipeline.generate_chunked(+ prompts=item.prompts,+ chunk_prompts=item.chunk_prompts,+ img=image,+ hr_action=item.actions.tensor,+ total_latent_frames=item.actions.latent_frames,+ chunk_length=args.chunk_length,+ shift=inference.sample_shift,+ sample_solver=args.sample_solver,+ sampling_steps=inference.sample_steps,+ guide_scale=inference.guide_scale,+ n_prompt=args.negative_prompt,+ seed=runtime.seed,+ offload_model=runtime.offload_model,+ sink_size=args.sink_size,+ temporal_size=args.temporal_size,+ height=args.height,+ width=args.width,+ )+ if runtime.rank == 0 and video is not None:+ save_video(+ video,+ runtime.output_dir / f"{item.output_name}.mp4",+ I2V_A14B_CONFIG.sample_fps,+ )+ runtime.finish()+++if __name__ == "__main__":+ main()diff --git a/requirements.txt b/requirements.txtnew file mode 100644index 0000000..1e1afcf--- /dev/null+++ b/requirements.txt@@ -0,0 +1,15 @@+torch>=2.4.0+torchvision>=0.19.0+diffusers>=0.31.0+transformers>=4.49.0,<=4.51.3+tokenizers>=0.20.3+accelerate>=1.1.1+safetensors+einops+tqdm+imageio[ffmpeg]+ftfy+regex+numpy>=1.23.5,<2+Pillow+pytestdiff --git a/run_ar.sh b/run_ar.shnew file mode 100755index 0000000..2031ea7--- /dev/null+++ b/run_ar.sh@@ -0,0 +1,117 @@+#!/usr/bin/env bash+set -euo pipefail++NPROC_PER_NODE="${NPROC_PER_NODE:-8}"+# Attention backend: flash (default) or sage.+ATTENTION_BACKEND="${ATTENTION_BACKEND:-flash}"+YAW_ROTATION_SPEED_DEG="${YAW_ROTATION_SPEED_DEG:-3.0}"+PITCH_ROTATION_SPEED_DEG="${PITCH_ROTATION_SPEED_DEG:-3.0}"+SAMPLE_STEPS="${SAMPLE_STEPS:-40}"+GUIDE_SCALE="${GUIDE_SCALE:-3.5}"++DISABLE_COMPILED_QK_ROPE="${DISABLE_COMPILED_QK_ROPE:-false}"+DISABLE_CONDITIONING_CACHE="${DISABLE_CONDITIONING_CACHE:-false}"+DISABLE_SELF_QKV_FUSION="${DISABLE_SELF_QKV_FUSION:-false}"+DISABLE_COMPILED_BLOCK_GLUE="${DISABLE_COMPILED_BLOCK_GLUE:-false}"+DISABLE_COMPILED_FFN="${DISABLE_COMPILED_FFN:-false}"+USE_DIT_FSDP="${USE_DIT_FSDP:-false}"+USE_T5_FSDP="${USE_T5_FSDP:-false}"+ENABLE_FP8_FFN="${ENABLE_FP8_FFN:-false}"+FP8_START_BLOCK="${FP8_START_BLOCK:-0}"+FP8_END_BLOCK="${FP8_END_BLOCK:-39}"+ENABLE_FP8_QKV="${ENABLE_FP8_QKV:-false}"+FP8_QKV_START_BLOCK="${FP8_QKV_START_BLOCK:-0}"+FP8_QKV_END_BLOCK="${FP8_QKV_END_BLOCK:-39}"++CKPT_DIR="${CKPT_DIR:-/path/to/model}"+LOW_NOISE_CKPT="${LOW_NOISE_CKPT:-/path/to/model/low_noise_model.safetensors}"+HIGH_NOISE_CKPT="${HIGH_NOISE_CKPT:-/path/to/model/high_noise_model.safetensors}"++INPUT_JSON="${INPUT_JSON:-/path/to/input.json}"++OUTPUT_PATH="${OUTPUT_PATH:-./outputs/ar}"++run_case() {+ local json_path="$1"+ local output_dir="${OUTPUT_PATH}"+ local qk_rope_args=()+ if [[ "${DISABLE_COMPILED_QK_ROPE}" == "true" ]]; then+ qk_rope_args+=(--disable_compiled_qk_rope)+ fi+ local conditioning_args=()+ if [[ "${DISABLE_CONDITIONING_CACHE}" == "true" ]]; then+ conditioning_args+=(--disable_conditioning_cache)+ fi+ local qkv_fusion_args=()+ if [[ "${DISABLE_SELF_QKV_FUSION}" == "true" ]]; then+ qkv_fusion_args+=(--disable_self_qkv_fusion)+ fi+ local compiled_region_args=()+ if [[ "${DISABLE_COMPILED_BLOCK_GLUE}" == "true" ]]; then+ compiled_region_args+=(--disable_compiled_block_glue)+ fi+ if [[ "${DISABLE_COMPILED_FFN}" == "true" ]]; then+ compiled_region_args+=(--disable_compiled_ffn)+ fi+ local fsdp_args=()+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then+ fsdp_args+=(--dit_fsdp)+ fi+ if [[ "${USE_T5_FSDP}" == "true" ]]; then+ fsdp_args+=(--t5_fsdp)+ fi+ local fp8_args=()+ if [[ "${ENABLE_FP8_FFN}" == "true" || "${ENABLE_FP8_QKV}" == "true" ]]; then+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then+ echo "DiT FP8 requires USE_DIT_FSDP=false" >&2+ return 2+ fi+ fp8_args+=(--offload_model false)+ fi+ if [[ "${ENABLE_FP8_FFN}" == "true" ]]; then+ fp8_args+=(+ --fp8_ffn+ --fp8_start_block "${FP8_START_BLOCK}"+ --fp8_end_block "${FP8_END_BLOCK}"+ )+ fi+ if [[ "${ENABLE_FP8_QKV}" == "true" ]]; then+ fp8_args+=(+ --fp8_qkv+ --fp8_qkv_start_block "${FP8_QKV_START_BLOCK}"+ --fp8_qkv_end_block "${FP8_QKV_END_BLOCK}"+ )+ fi++ torchrun \+ --standalone \+ --nproc_per_node="${NPROC_PER_NODE}" \+ generate.py \+ --inference_mode ar \+ --attention_backend "${ATTENTION_BACKEND}" \+ --ckpt_dir "${CKPT_DIR}" \+ --low_noise_mem_ckpt "${LOW_NOISE_CKPT}" \+ --high_noise_mem_ckpt "${HIGH_NOISE_CKPT}" \+ --input_json "${json_path}" \+ --yaw_rotation_speed_deg "${YAW_ROTATION_SPEED_DEG}" \+ --pitch_rotation_speed_deg "${PITCH_ROTATION_SPEED_DEG}" \+ --height 448 \+ --width 832 \+ --chunk_length 4 \+ --sink_size 1 \+ --temporal_size 1 \+ --sample_steps "${SAMPLE_STEPS}" \+ --sample_shift 5.0 \+ --sample_guide_scale "${GUIDE_SCALE}" \+ --seed 42 \+ "${qk_rope_args[@]}" \+ "${conditioning_args[@]}" \+ "${qkv_fusion_args[@]}" \+ "${compiled_region_args[@]}" \+ "${fsdp_args[@]}" \+ "${fp8_args[@]}" \+ --ulysses_size "${NPROC_PER_NODE}" \+ --output_dir "${output_dir}"+}++run_case "${INPUT_JSON}"diff --git a/run_bi.sh b/run_bi.shnew file mode 100755index 0000000..f7804f6--- /dev/null+++ b/run_bi.sh@@ -0,0 +1,117 @@+#!/usr/bin/env bash+set -euo pipefail++NPROC_PER_NODE="${NPROC_PER_NODE:-8}"+# Visual self-attention backend: flash (default) or sage.+ATTENTION_BACKEND="${ATTENTION_BACKEND:-sage}"+YAW_ROTATION_SPEED_DEG="${YAW_ROTATION_SPEED_DEG:-3.0}"+PITCH_ROTATION_SPEED_DEG="${PITCH_ROTATION_SPEED_DEG:-3.0}"+SAMPLE_STEPS="${SAMPLE_STEPS:-40}"+GUIDE_SCALE="${GUIDE_SCALE:-3.5}"++DISABLE_COMPILED_QK_ROPE="${DISABLE_COMPILED_QK_ROPE:-false}"+DISABLE_CONDITIONING_CACHE="${DISABLE_CONDITIONING_CACHE:-false}"+DISABLE_SELF_QKV_FUSION="${DISABLE_SELF_QKV_FUSION:-false}"+DISABLE_COMPILED_BLOCK_GLUE="${DISABLE_COMPILED_BLOCK_GLUE:-false}"+DISABLE_COMPILED_FFN="${DISABLE_COMPILED_FFN:-false}"+USE_DIT_FSDP="${USE_DIT_FSDP:-false}"+USE_T5_FSDP="${USE_T5_FSDP:-false}"+ENABLE_FP8_FFN="${ENABLE_FP8_FFN:-false}"+FP8_START_BLOCK="${FP8_START_BLOCK:-0}"+FP8_END_BLOCK="${FP8_END_BLOCK:-39}"+ENABLE_FP8_QKV="${ENABLE_FP8_QKV:-false}"+FP8_QKV_START_BLOCK="${FP8_QKV_START_BLOCK:-0}"+FP8_QKV_END_BLOCK="${FP8_QKV_END_BLOCK:-39}"++CKPT_DIR="${CKPT_DIR:-/path/to/model}"+LOW_NOISE_CKPT="${LOW_NOISE_CKPT:-/path/to/model/low_noise_model.safetensors}"+HIGH_NOISE_CKPT="${HIGH_NOISE_CKPT:-/path/to/model/high_noise_model.safetensors}"++INPUT_JSON="${INPUT_JSON:-/path/to/input.json}"++OUTPUT_PATH="${OUTPUT_PATH:-./outputs/bi}"++run_case() {+ local json_path="$1"+ local output_dir="${OUTPUT_PATH}"+ local qk_rope_args=()+ if [[ "${DISABLE_COMPILED_QK_ROPE}" == "true" ]]; then+ qk_rope_args+=(--disable_compiled_qk_rope)+ fi+ local conditioning_args=()+ if [[ "${DISABLE_CONDITIONING_CACHE}" == "true" ]]; then+ conditioning_args+=(--disable_conditioning_cache)+ fi+ local qkv_fusion_args=()+ if [[ "${DISABLE_SELF_QKV_FUSION}" == "true" ]]; then+ qkv_fusion_args+=(--disable_self_qkv_fusion)+ fi+ local compiled_region_args=()+ if [[ "${DISABLE_COMPILED_BLOCK_GLUE}" == "true" ]]; then+ compiled_region_args+=(--disable_compiled_block_glue)+ fi+ if [[ "${DISABLE_COMPILED_FFN}" == "true" ]]; then+ compiled_region_args+=(--disable_compiled_ffn)+ fi+ local fsdp_args=()+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then+ fsdp_args+=(--dit_fsdp)+ fi+ if [[ "${USE_T5_FSDP}" == "true" ]]; then+ fsdp_args+=(--t5_fsdp)+ fi+ local fp8_args=()+ if [[ "${ENABLE_FP8_FFN}" == "true" || "${ENABLE_FP8_QKV}" == "true" ]]; then+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then+ echo "DiT FP8 requires USE_DIT_FSDP=false" >&2+ return 2+ fi+ fp8_args+=(--offload_model false)+ fi+ if [[ "${ENABLE_FP8_FFN}" == "true" ]]; then+ fp8_args+=(+ --fp8_ffn+ --fp8_start_block "${FP8_START_BLOCK}"+ --fp8_end_block "${FP8_END_BLOCK}"+ )+ fi+ if [[ "${ENABLE_FP8_QKV}" == "true" ]]; then+ fp8_args+=(+ --fp8_qkv+ --fp8_qkv_start_block "${FP8_QKV_START_BLOCK}"+ --fp8_qkv_end_block "${FP8_QKV_END_BLOCK}"+ )+ fi++ torchrun \+ --standalone \+ --nproc_per_node="${NPROC_PER_NODE}" \+ generate.py \+ --inference_mode bi \+ --attention_backend "${ATTENTION_BACKEND}" \+ --ckpt_dir "${CKPT_DIR}" \+ --low_noise_mem_ckpt "${LOW_NOISE_CKPT}" \+ --high_noise_mem_ckpt "${HIGH_NOISE_CKPT}" \+ --input_json "${json_path}" \+ --yaw_rotation_speed_deg "${YAW_ROTATION_SPEED_DEG}" \+ --pitch_rotation_speed_deg "${PITCH_ROTATION_SPEED_DEG}" \+ --height 448 \+ --width 832 \+ --chunk_length 32 \+ --sink_size 1 \+ --temporal_size 1 \+ --sample_steps "${SAMPLE_STEPS}" \+ --sample_shift 5.0 \+ --sample_guide_scale "${GUIDE_SCALE}" \+ --seed 42 \+ "${qk_rope_args[@]}" \+ "${conditioning_args[@]}" \+ "${qkv_fusion_args[@]}" \+ "${compiled_region_args[@]}" \+ "${fsdp_args[@]}" \+ "${fp8_args[@]}" \+ --ulysses_size "${NPROC_PER_NODE}" \+ --output_dir "${output_dir}"+}++run_case "${INPUT_JSON}"diff --git a/run_few_step.sh b/run_few_step.shnew file mode 100644index 0000000..30460ed--- /dev/null+++ b/run_few_step.sh@@ -0,0 +1,119 @@+#!/usr/bin/env bash+set -euo pipefail++NPROC_PER_NODE="${NPROC_PER_NODE:-8}"+# Visual self-attention backend: flash (default) or sage.+ATTENTION_BACKEND="${ATTENTION_BACKEND:-sage}"+YAW_ROTATION_SPEED_DEG="${YAW_ROTATION_SPEED_DEG:-3.0}"+PITCH_ROTATION_SPEED_DEG="${PITCH_ROTATION_SPEED_DEG:-1.0}"+SAMPLE_STEPS="${SAMPLE_STEPS:-4}"++DISABLE_COMPILED_QK_ROPE="${DISABLE_COMPILED_QK_ROPE:-false}"+DISABLE_CONDITIONING_CACHE="${DISABLE_CONDITIONING_CACHE:-false}"+DISABLE_SELF_QKV_FUSION="${DISABLE_SELF_QKV_FUSION:-false}"+DISABLE_COMPILED_BLOCK_GLUE="${DISABLE_COMPILED_BLOCK_GLUE:-false}"+DISABLE_COMPILED_FFN="${DISABLE_COMPILED_FFN:-false}"+USE_DIT_FSDP="${USE_DIT_FSDP:-false}"+USE_T5_FSDP="${USE_T5_FSDP:-false}"+ENABLE_FP8_FFN="${ENABLE_FP8_FFN:-false}"+FP8_START_BLOCK="${FP8_START_BLOCK:-0}"+FP8_END_BLOCK="${FP8_END_BLOCK:-39}"+ENABLE_FP8_QKV="${ENABLE_FP8_QKV:-false}"+FP8_QKV_START_BLOCK="${FP8_QKV_START_BLOCK:-0}"+FP8_QKV_END_BLOCK="${FP8_QKV_END_BLOCK:-39}"++CKPT_DIR="${CKPT_DIR:-/path/to/model}"+LOW_NOISE_CKPT="${LOW_NOISE_CKPT:-/path/to/model/low_noise_model.pt}"+HIGH_NOISE_CKPT="${HIGH_NOISE_CKPT:-/path/to/model/high_noise_model.pt}"++INPUT_JSON="${INPUT_JSON:-/path/to/input.json}"++OUTPUT_PATH="${OUTPUT_PATH:-./outputs/few_step}"++run_case() {+ local json_path="$1"+ local yaw_rotation_speed_deg="$2"+ local pitch_rotation_speed_deg="$3"+ local output_dir="${OUTPUT_PATH}"+ local qk_rope_args=()+ if [[ "${DISABLE_COMPILED_QK_ROPE}" == "true" ]]; then+ qk_rope_args+=(--disable_compiled_qk_rope)+ fi+ local conditioning_args=()+ if [[ "${DISABLE_CONDITIONING_CACHE}" == "true" ]]; then+ conditioning_args+=(--disable_conditioning_cache)+ fi+ local qkv_fusion_args=()+ if [[ "${DISABLE_SELF_QKV_FUSION}" == "true" ]]; then+ qkv_fusion_args+=(--disable_self_qkv_fusion)+ fi+ local compiled_region_args=()+ if [[ "${DISABLE_COMPILED_BLOCK_GLUE}" == "true" ]]; then+ compiled_region_args+=(--disable_compiled_block_glue)+ fi+ if [[ "${DISABLE_COMPILED_FFN}" == "true" ]]; then+ compiled_region_args+=(--disable_compiled_ffn)+ fi+ local fsdp_args=()+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then+ fsdp_args+=(--dit_fsdp)+ fi+ if [[ "${USE_T5_FSDP}" == "true" ]]; then+ fsdp_args+=(--t5_fsdp)+ fi+ local fp8_args=()+ if [[ "${ENABLE_FP8_FFN}" == "true" || "${ENABLE_FP8_QKV}" == "true" ]]; then+ if [[ "${USE_DIT_FSDP}" == "true" ]]; then+ echo "DiT FP8 requires USE_DIT_FSDP=false" >&2+ return 2+ fi+ fp8_args+=(--offload_model false)+ fi+ if [[ "${ENABLE_FP8_FFN}" == "true" ]]; then+ fp8_args+=(+ --fp8_ffn+ --fp8_start_block "${FP8_START_BLOCK}"+ --fp8_end_block "${FP8_END_BLOCK}"+ )+ fi+ if [[ "${ENABLE_FP8_QKV}" == "true" ]]; then+ fp8_args+=(+ --fp8_qkv+ --fp8_qkv_start_block "${FP8_QKV_START_BLOCK}"+ --fp8_qkv_end_block "${FP8_QKV_END_BLOCK}"+ )+ fi++ echo "Running yaw_rotation_speed_deg=${yaw_rotation_speed_deg}, pitch_rotation_speed_deg=${pitch_rotation_speed_deg}, output_dir=${output_dir}"+ torchrun \+ --standalone \+ --nproc_per_node="${NPROC_PER_NODE}" \+ generate.py \+ --inference_mode pdd \+ --attention_backend "${ATTENTION_BACKEND}" \+ --ckpt_dir "${CKPT_DIR}" \+ --low_noise_mem_ckpt "${LOW_NOISE_CKPT}" \+ --high_noise_mem_ckpt "${HIGH_NOISE_CKPT}" \+ --input_json "${json_path}" \+ --yaw_rotation_speed_deg "${yaw_rotation_speed_deg}" \+ --pitch_rotation_speed_deg "${pitch_rotation_speed_deg}" \+ --height 448 \+ --width 832 \+ --chunk_length 4 \+ --sink_size 1 \+ --temporal_size 1 \+ --sample_steps "${SAMPLE_STEPS}" \+ --sample_shift 7.0 \+ --sample_guide_scale 1.0 \+ --seed 42 \+ "${qk_rope_args[@]}" \+ "${conditioning_args[@]}" \+ "${qkv_fusion_args[@]}" \+ "${compiled_region_args[@]}" \+ "${fsdp_args[@]}" \+ "${fp8_args[@]}" \+ --ulysses_size "${NPROC_PER_NODE}" \+ --output_dir "${output_dir}"+}++run_case "${INPUT_JSON}" "${YAW_ROTATION_SPEED_DEG}" "${PITCH_ROTATION_SPEED_DEG}"diff --git a/worldplay2/__init__.py b/worldplay2/__init__.pynew file mode 100644index 0000000..67fecfd--- /dev/null+++ b/worldplay2/__init__.py@@ -0,0 +1,10 @@+"""WorldPlay2 inference package.++Heavy CUDA/model modules are intentionally not imported here, so lightweight+utilities such as input and action parsing can be used without torchvision,+diffusers, or flash-attn.+"""++from .inputs import ActionSequence, parse_action_string++__all__ = ["ActionSequence", "parse_action_string"]diff --git a/worldplay2/checkpoint.py b/worldplay2/checkpoint.pynew file mode 100644index 0000000..9a3f72c--- /dev/null+++ b/worldplay2/checkpoint.py@@ -0,0 +1,67 @@+from __future__ import annotations++import logging+from pathlib import Path++import torch+from safetensors.torch import load_file+++def load_state_dict(path: str, *, prefer_ema: bool = False) -> dict[str, torch.Tensor]:+ checkpoint_path = Path(path).expanduser()+ if not checkpoint_path.is_file():+ raise FileNotFoundError(checkpoint_path)+ state = (+ load_file(str(checkpoint_path))+ if checkpoint_path.suffix == ".safetensors"+ else torch.load(checkpoint_path, map_location="cpu", weights_only=False)+ )++ if isinstance(state, dict) and ("generator" in state or "generator_ema" in state):+ key = "generator_ema" if prefer_ema and "generator_ema" in state else "generator"+ state = state[key]+ for wrapper in ("state_dict", "model_state_dict", "model"):+ if isinstance(state, dict) and wrapper in state and isinstance(state[wrapper], dict):+ state = state[wrapper]++ if not isinstance(state, dict) or not state:+ raise ValueError(f"checkpoint {checkpoint_path} does not contain a state dict")++ prefixes = ("module.", "model.")+ for prefix in prefixes:+ if all(key.startswith(prefix) for key in state):+ state = {key[len(prefix):]: value for key, value in state.items()}+ return state+++def load_expert_checkpoint(+ model,+ path: str,+ *,+ tag: str,+ prefer_ema: bool = False,+ strict: bool = True,+) -> None:+ state = load_state_dict(path, prefer_ema=prefer_ema)+ provided = set(state)+ required_prefixes = (+ "memory_compress.",+ "action_in_cont.",+ "action_in_disc.",+ )+ absent = [prefix for prefix in required_prefixes if not any(+ key.startswith(prefix) for key in provided+ )]+ if absent:+ raise ValueError(+ f"{tag} checkpoint is missing required parameter groups: {absent}"+ )+ missing, unexpected = model.load_state_dict(state, strict=strict)+ logging.info(+ "%s checkpoint: loaded=%d missing=%d unexpected=%d strict=%s",+ tag,+ len(state),+ len(missing),+ len(unexpected),+ strict,+ )diff --git a/worldplay2/distributed/__init__.py b/worldplay2/distributed/__init__.pynew file mode 100644index 0000000..388f037--- /dev/null+++ b/worldplay2/distributed/__init__.py@@ -0,0 +1,3 @@+from .ops import get_rank, get_world_size, init_distributed_group++__all__ = ["get_rank", "get_world_size", "init_distributed_group"]diff --git a/worldplay2/distributed/fsdp.py b/worldplay2/distributed/fsdp.pynew file mode 100644index 0000000..247b5eb--- /dev/null+++ b/worldplay2/distributed/fsdp.py@@ -0,0 +1,45 @@+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.+import gc+from functools import partial++import torch+from torch.distributed.fsdp import FullyShardedDataParallel as FSDP+from torch.distributed.fsdp import MixedPrecision, ShardingStrategy+from torch.distributed.fsdp.wrap import lambda_auto_wrap_policy+from torch.distributed.utils import _free_storage+++def shard_model(+ model,+ device_id,+ param_dtype=torch.bfloat16,+ reduce_dtype=torch.float32,+ buffer_dtype=torch.float32,+ process_group=None,+ sharding_strategy=ShardingStrategy.FULL_SHARD,+ sync_module_states=True,+ use_lora=False+):+ model = FSDP(+ module=model,+ process_group=process_group,+ sharding_strategy=sharding_strategy,+ auto_wrap_policy=partial(+ lambda_auto_wrap_policy, lambda_fn=lambda m: m in model.blocks),+ mixed_precision=MixedPrecision(+ param_dtype=param_dtype,+ reduce_dtype=reduce_dtype,+ buffer_dtype=buffer_dtype),+ device_id=device_id,+ sync_module_states=sync_module_states,+ use_orig_params=True if use_lora else False)+ return model+++def free_model(model):+ for m in model.modules():+ if isinstance(m, FSDP):+ _free_storage(m._handle.flat_param.data)+ del model+ gc.collect()+ torch.cuda.empty_cache()diff --git a/worldplay2/distributed/ops.py b/worldplay2/distributed/ops.pynew file mode 100644index 0000000..cc64632--- /dev/null+++ b/worldplay2/distributed/ops.py@@ -0,0 +1,17 @@+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.+import torch.distributed as dist+++def init_distributed_group():+ """r initialize sequence parallel group.+ """+ if not dist.is_initialized():+ dist.init_process_group(backend='nccl')+++def get_rank():+ return dist.get_rank()+++def get_world_size():+ return dist.get_world_size()diff --git a/worldplay2/distributed/ulysses.py b/worldplay2/distributed/ulysses.pynew file mode 100644index 0000000..da714c8--- /dev/null+++ b/worldplay2/distributed/ulysses.py@@ -0,0 +1,358 @@+from __future__ import annotations++from dataclasses import dataclass+from functools import lru_cache++import torch+import torch.distributed as dist+++@dataclass(frozen=True)+class UlyssesSplitPlan:+ global_sequence: int+ num_heads: int+ world_size: int+ rank: int+ sequence_splits: tuple[int, ...]+ sequence_start: int+ local_sequence: int+ local_heads: int+++def _balanced_splits(total: int, parts: int) -> tuple[int, ...]:+ if parts <= 0:+ raise ValueError("parts must be positive")+ if total < parts:+ raise ValueError(+ f"sequence length {total} must be >= world size {parts}"+ )+ base, remainder = divmod(total, parts)+ return tuple(+ base + (1 if rank < remainder else 0)+ for rank in range(parts)+ )+++@lru_cache(maxsize=256)+def get_ulysses_split_plan(+ global_sequence: int,+ num_heads: int,+ world_size: int,+ rank: int,+) -> UlyssesSplitPlan:+ if num_heads % world_size:+ raise ValueError(+ f"num_heads={num_heads} must be divisible by world_size={world_size}"+ )+ splits = _balanced_splits(global_sequence, world_size)+ return UlyssesSplitPlan(+ global_sequence=global_sequence,+ num_heads=num_heads,+ world_size=world_size,+ rank=rank,+ sequence_splits=splits,+ sequence_start=sum(splits[:rank]),+ local_sequence=splits[rank],+ local_heads=num_heads // world_size,+ )+++_RECEIVE_BUFFERS: dict[tuple, torch.Tensor] = {}+_STATS = {+ "packed_qkv_calls": 0,+ "output_calls": 0,+ "final_gather_calls": 0,+ "qkv_collectives_eliminated": 0,+ "buffer_hits": 0,+ "buffer_misses": 0,+ "bytes_exchanged": 0,+}+++def _receive_buffer(name: str, reference: torch.Tensor, elements: int):+ key = (+ name,+ reference.device.type,+ reference.device.index,+ reference.dtype,+ )+ buffer = _RECEIVE_BUFFERS.get(key)+ if buffer is None or buffer.numel() < elements:+ buffer = torch.empty(+ elements,+ dtype=reference.dtype,+ device=reference.device,+ )+ _RECEIVE_BUFFERS[key] = buffer+ _STATS["buffer_misses"] += 1+ else:+ _STATS["buffer_hits"] += 1+ return buffer[:elements]+++def shard_sequence(+ tensor: torch.Tensor | None,+ *,+ global_sequence: int,+ rank: int,+ world_size: int,+ dim: int,+) -> torch.Tensor | None:+ if tensor is None or world_size <= 1:+ return tensor+ splits = _balanced_splits(global_sequence, world_size)+ start = sum(splits[:rank])+ return tensor.narrow(dim, start, splits[rank]).contiguous()+++def _pack_qkv_for_destinations(q, k, v, world_size, local_heads):+ batch, local_sequence, _, head_dim = q.shape++ def destination_major(tensor):+ return tensor.contiguous().view(+ batch,+ local_sequence,+ world_size,+ local_heads,+ head_dim,+ ).permute(2, 0, 1, 3, 4)++ return torch.stack(+ [destination_major(q), destination_major(k), destination_major(v)],+ dim=1,+ ).contiguous()+++def _unpack_received_qkv(+ output_flat,+ *,+ sequence_splits,+ batch,+ local_heads,+ head_dim,+):+ source_chunks = []+ offset = 0+ for source_sequence in sequence_splits:+ elements = (+ 3 * batch * source_sequence * local_heads * head_dim+ )+ source_chunks.append(+ output_flat[offset:offset + elements].view(+ 3,+ batch,+ source_sequence,+ local_heads,+ head_dim,+ )+ )+ offset += elements+ return torch.cat(source_chunks, dim=2).contiguous()+++def _pack_output_for_destinations(output, sequence_splits):+ return torch.cat([+ chunk.contiguous().view(-1)+ for chunk in torch.split(output, sequence_splits, dim=1)+ ])+++def _unpack_received_output(+ output_flat,+ *,+ world_size,+ batch,+ local_sequence,+ local_heads,+ num_heads,+ head_dim,+):+ return output_flat.view(+ world_size,+ batch,+ local_sequence,+ local_heads,+ head_dim,+ ).permute(1, 2, 0, 3, 4).reshape(+ batch,+ local_sequence,+ num_heads,+ head_dim,+ ).contiguous()+++def packed_qkv_all_to_all(+ q: torch.Tensor,+ k: torch.Tensor,+ v: torch.Tensor,+ *,+ global_sequence: int,+ group=None,+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:+ """Gather sequence and shard heads with one packed all-to-all-single."""+ if not dist.is_initialized():+ raise RuntimeError("packed QKV all-to-all requires distributed init")+ world_size = dist.get_world_size(group)+ if world_size == 1:+ return q, k, v+ rank = dist.get_rank(group)+ if q.shape != k.shape or q.shape != v.shape:+ raise ValueError("Q, K, and V must have identical local shapes")++ batch, local_sequence, num_heads, head_dim = q.shape+ plan = get_ulysses_split_plan(+ global_sequence, num_heads, world_size, rank)+ if local_sequence != plan.local_sequence:+ raise ValueError(+ f"rank {rank} expected {plan.local_sequence} local tokens, "+ f"got {local_sequence}"+ )++ # [destination, qkv, batch, local_sequence, local_heads, head_dim]+ packed = _pack_qkv_for_destinations(+ q, k, v, world_size, plan.local_heads)+ input_flat = packed.view(-1)+ per_destination = (+ 3 * batch * local_sequence * plan.local_heads * head_dim+ )+ input_splits = [per_destination] * world_size+ output_splits = [+ 3 * batch * seq * plan.local_heads * head_dim+ for seq in plan.sequence_splits+ ]+ output_flat = _receive_buffer(+ "packed_qkv", input_flat, sum(output_splits))+ dist.all_to_all_single(+ output_flat,+ input_flat,+ output_split_sizes=output_splits,+ input_split_sizes=input_splits,+ group=group,+ )++ packed_global = _unpack_received_qkv(+ output_flat,+ sequence_splits=plan.sequence_splits,+ batch=batch,+ local_heads=plan.local_heads,+ head_dim=head_dim,+ )++ _STATS["packed_qkv_calls"] += 1+ _STATS["qkv_collectives_eliminated"] += 2+ _STATS["bytes_exchanged"] += (+ input_flat.numel() * input_flat.element_size())+ return tuple(packed_global.unbind(0))+++def output_all_to_all(+ output: torch.Tensor,+ *,+ global_sequence: int,+ num_heads: int,+ group=None,+) -> torch.Tensor:+ """Shard sequence and gather heads after attention."""+ if not dist.is_initialized():+ raise RuntimeError("output all-to-all requires distributed init")+ world_size = dist.get_world_size(group)+ if world_size == 1:+ return output+ rank = dist.get_rank(group)++ batch, sequence, local_heads, head_dim = output.shape+ plan = get_ulysses_split_plan(+ global_sequence, num_heads, world_size, rank)+ if sequence != global_sequence or local_heads != plan.local_heads:+ raise ValueError(+ "attention output shape does not match the Ulysses split plan: "+ f"shape={tuple(output.shape)}, plan={plan}"+ )++ input_flat = _pack_output_for_destinations(+ output, plan.sequence_splits)+ input_splits = [+ batch * seq * local_heads * head_dim+ for seq in plan.sequence_splits+ ]+ per_source = (+ batch * plan.local_sequence * local_heads * head_dim+ )+ output_splits = [per_source] * world_size+ output_flat = _receive_buffer(+ "attention_output", input_flat, sum(output_splits))+ dist.all_to_all_single(+ output_flat,+ input_flat,+ output_split_sizes=output_splits,+ input_split_sizes=input_splits,+ group=group,+ )+ local = _unpack_received_output(+ output_flat,+ world_size=world_size,+ batch=batch,+ local_sequence=plan.local_sequence,+ local_heads=local_heads,+ num_heads=num_heads,+ head_dim=head_dim,+ )++ _STATS["output_calls"] += 1+ _STATS["bytes_exchanged"] += (+ input_flat.numel() * input_flat.element_size())+ return local+++def gather_sequence(+ tensor: torch.Tensor,+ *,+ global_sequence: int,+ group=None,+) -> torch.Tensor:+ """Gather a known balanced sequence split without object collectives."""+ if not dist.is_initialized():+ return tensor+ world_size = dist.get_world_size(group)+ if world_size == 1:+ return tensor+ rank = dist.get_rank(group)+ splits = _balanced_splits(global_sequence, world_size)+ if tensor.shape[1] != splits[rank]:+ raise ValueError(+ f"rank {rank} expected {splits[rank]} tokens, got {tensor.shape[1]}"+ )++ max_sequence = max(splits)+ padded_shape = list(tensor.shape)+ padded_shape[1] = max_sequence+ padded = tensor.new_zeros(padded_shape)+ padded[:, :tensor.shape[1]].copy_(tensor)+ padded_flat = padded.contiguous().view(-1)+ gathered_flat = _receive_buffer(+ "final_sequence_gather",+ padded_flat,+ world_size * padded_flat.numel(),+ )+ dist.all_gather_into_tensor(+ gathered_flat,+ padded_flat,+ group=group,+ )+ gathered = gathered_flat.view(world_size, *padded.shape)+ result = torch.cat(+ [chunk[:, :length] for chunk, length in zip(gathered, splits)],+ dim=1,+ ).contiguous()+ _STATS["final_gather_calls"] += 1+ _STATS["bytes_exchanged"] += (+ padded.numel() * padded.element_size())+ return result+++def get_ulysses_stats() -> dict:+ return dict(_STATS)+++def clear_ulysses_buffers() -> None:+ _RECEIVE_BUFFERS.clear()diff --git a/worldplay2/inputs.py b/worldplay2/inputs.pynew file mode 100644index 0000000..3246082--- /dev/null+++ b/worldplay2/inputs.py@@ -0,0 +1,555 @@+from __future__ import annotations++import json+import math+import os+import random+import sys+from dataclasses import dataclass+from pathlib import Path+from typing import Any++import torch+import torch.distributed as dist++from worldplay2.distributed.ops import init_distributed_group+++_MOVEMENT = {+ "w": (1, 0),+ "s": (-1, 0),+ "a": (0, -1),+ "d": (0, 1),+ "wa": (1, -1),+ "wd": (1, 1),+ "sa": (-1, -1),+ "sd": (-1, 1),+}+_ROTATION = {+ "up": (1, 0),+ "down": (-1, 0),+ "left": (0, -1),+ "right": (0, 1),+}+++@dataclass(frozen=True)+class ActionSequence:+ tensor: torch.Tensor+ source: str+ perspective: str++ @property+ def latent_frames(self) -> int:+ return int(self.tensor.shape[0])+++@dataclass(frozen=True)+class GenerationInput:+ image: Path+ actions: ActionSequence+ prompts: dict[str, str]+ chunk_prompts: tuple[str, ...]+ output_name: str+++@dataclass(frozen=True)+class InferenceInputs:+ items: list[GenerationInput]+ mode: str+ sample_shift: float+ sample_steps: int+ guide_scale: float | tuple[float, ...]+++@dataclass(frozen=True)+class RuntimeContext:+ rank: int+ world_size: int+ local_rank: int+ output_dir: Path+ offload_model: bool+ seed: int++ def finish(self) -> None:+ if dist.is_initialized():+ dist.barrier(device_ids=[self.local_rank])+ dist.destroy_process_group()+++def parse_action_string(+ value: str,+ *,+ perspective: str = "tps",+ rotation_speed_deg: float = 3.0,+ yaw_rotation_speed_deg: float | None = None,+ pitch_rotation_speed_deg: float | None = None,+) -> ActionSequence:+ """Convert ``w+right-8,s-4`` directly to model actions.++ Command durations sum to the total latent length. The first latent is+ always stationary, so the first command applies for ``duration - 1``+ subsequent latents.++ Yaw (left/right) and pitch (up/down) use independent degrees per latent.+ Each unspecified axis falls back to ``rotation_speed_deg``.+ """+ if perspective not in {"tps", "fps"}:+ raise ValueError("perspective must be 'tps' or 'fps'")+ if yaw_rotation_speed_deg is None:+ yaw_rotation_speed_deg = rotation_speed_deg+ if pitch_rotation_speed_deg is None:+ pitch_rotation_speed_deg = rotation_speed_deg+ for name, speed in (+ ("yaw_rotation_speed_deg", yaw_rotation_speed_deg),+ ("pitch_rotation_speed_deg", pitch_rotation_speed_deg),+ ):+ if not math.isfinite(speed) or speed <= 0:+ raise ValueError(f"{name} must be a positive finite number")+ if not value or not value.strip():+ raise ValueError("action string cannot be empty")++ perspective_value = 0.0 if perspective == "tps" else 1.0+ rows: list[list[float]] = [[0.0, 0.0, 0.0, 0.0, perspective_value, 0.0]]+ first_command = True++ for raw_command in value.split(","):+ command = raw_command.strip()+ if not command:+ continue+ action_text, separator, duration_text = command.rpartition("-")+ if not separator or not action_text.strip():+ raise ValueError(+ f"invalid action command {command!r}; expected action-duration"+ )+ try:+ duration = int(duration_text.strip())+ except ValueError as exc:+ raise ValueError(+ f"duration must be a positive integer in {command!r}"+ ) from exc+ if duration <= 0 or str(duration) != duration_text.strip():+ raise ValueError(+ f"duration must be a positive integer in {command!r}"+ )++ pitch = yaw = forward = lateral = space = 0.0+ tokens = [token.strip().lower() for token in action_text.split("+")]+ if not all(tokens):+ raise ValueError(f"empty action token in {command!r}")+ seen_categories: set[str] = set()+ for token in tokens:+ if token in _MOVEMENT:+ if "movement" in seen_categories:+ raise ValueError(f"multiple movement tokens in {command!r}")+ seen_categories.add("movement")+ forward, lateral = map(float, _MOVEMENT[token])+ elif token in _ROTATION:+ if token in {"up", "down"}:+ if "pitch" in seen_categories:+ raise ValueError(f"multiple pitch tokens in {command!r}")+ seen_categories.add("pitch")+ else:+ if "yaw" in seen_categories:+ raise ValueError(f"multiple yaw tokens in {command!r}")+ seen_categories.add("yaw")+ pitch_sign, yaw_sign = _ROTATION[token]+ pitch += pitch_sign * pitch_rotation_speed_deg+ yaw += yaw_sign * yaw_rotation_speed_deg+ elif token == "space":+ if "space" in seen_categories:+ raise ValueError(f"duplicate space token in {command!r}")+ seen_categories.add("space")+ space = 1.0+ elif token == "none":+ if len(tokens) != 1:+ raise ValueError(+ f"'none' cannot be combined with other actions "+ f"in {command!r}"+ )+ else:+ supported = sorted(+ set(_MOVEMENT) | set(_ROTATION) | {"none", "space"}+ )+ raise ValueError(+ f"unknown action token {token!r}; supported: {supported}"+ )++ row = [pitch, yaw, forward, lateral, perspective_value, space]+ repeat_count = duration - 1 if first_command else duration+ rows.extend([row.copy() for _ in range(repeat_count)])+ first_command = False++ if first_command:+ raise ValueError("action string did not contain any commands")+ return ActionSequence(+ tensor=torch.tensor(rows, dtype=torch.float32),+ source=value,+ perspective=perspective,+ )+++def _parse_prompt_event(+ *,+ value: str,+ prompts: dict[str, str],+ latent_frames: int,+ chunk_length: int,+) -> tuple[str, ...]:+ """Parse ``prompt1-8,prompt2-8`` into one prompt name per chunk."""+ if not value or not value.strip():+ raise ValueError("prompt_event cannot be empty")++ latent_prompts: list[str] = []+ cumulative = 0+ for raw_event in value.split(","):+ event = raw_event.strip()+ prompt_name, separator, duration_text = event.rpartition("-")+ prompt_name = prompt_name.strip()+ if not separator or not prompt_name:+ raise ValueError(+ f"invalid prompt event {event!r}; expected prompt_name-duration"+ )+ if prompt_name not in prompts:+ raise ValueError(+ f"prompt_event references missing prompt {prompt_name!r}"+ )+ try:+ duration = int(duration_text.strip())+ except ValueError as exc:+ raise ValueError(+ f"prompt duration must be a positive integer in {event!r}"+ ) from exc+ if duration <= 0 or str(duration) != duration_text.strip():+ raise ValueError(+ f"prompt duration must be a positive integer in {event!r}"+ )++ cumulative += duration+ if cumulative < latent_frames and cumulative % chunk_length:+ raise ValueError(+ "prompt_event boundaries must align with chunk_length; "+ f"boundary {cumulative} is not divisible by {chunk_length}"+ )+ latent_prompts.extend([prompt_name] * duration)++ if len(latent_prompts) != latent_frames:+ raise ValueError(+ f"prompt_event covers {len(latent_prompts)} latent frames, "+ f"but action produces {latent_frames}"+ )+ return tuple(+ latent_prompts[index]+ for index in range(0, latent_frames, chunk_length)+ )+++def load_generation_inputs(+ *,+ input_json: str,+ rotation_speed_deg: float = 3.0,+ yaw_rotation_speed_deg: float | None = None,+ pitch_rotation_speed_deg: float | None = None,+ chunk_length: int,+ sink_size: int,+ temporal_size: int,+) -> list[GenerationInput]:+ json_path = Path(input_json).expanduser()+ with json_path.open("r", encoding="utf-8") as handle:+ payload = json.load(handle)+ if not isinstance(payload, list) or not payload:+ raise ValueError("input JSON must contain a non-empty list")++ items = []+ for index, raw_item in enumerate(payload):+ if not isinstance(raw_item, dict):+ raise ValueError(f"input JSON item {index} must be an object")+ image_path = raw_item.get("image_path")+ action = raw_item.get("action")+ prompt_event = raw_item.get("prompt_event")+ perspective = raw_item.get("perspective")+ if not isinstance(image_path, str) or not image_path.strip():+ raise ValueError(f"input JSON item {index} requires image_path")+ if not isinstance(action, str):+ raise ValueError(f"input JSON item {index} requires action")+ if not isinstance(prompt_event, str):+ raise ValueError(f"input JSON item {index} requires prompt_event")+ if perspective not in {"tps", "fps"}:+ raise ValueError(+ f"input JSON item {index} perspective must be 'tps' or 'fps'"+ )++ prompts = {+ key: value+ for key, value in raw_item.items()+ if key.startswith("prompt")+ and key != "prompt_event"+ and isinstance(value, str)+ and value.strip()+ }+ if not prompts:+ raise ValueError(+ f"input JSON item {index} requires at least one prompt field"+ )+ actions = load_action_sequence(+ action=action,+ perspective=perspective,+ rotation_speed_deg=rotation_speed_deg,+ yaw_rotation_speed_deg=yaw_rotation_speed_deg,+ pitch_rotation_speed_deg=pitch_rotation_speed_deg,+ chunk_length=chunk_length,+ sink_size=sink_size,+ temporal_size=temporal_size,+ )+ chunk_prompts = _parse_prompt_event(+ value=prompt_event,+ prompts=prompts,+ latent_frames=actions.latent_frames,+ chunk_length=chunk_length,+ )+ image = Path(image_path).expanduser()+ if not image.is_absolute():+ image = json_path.parent / image+ output_name = str(+ raw_item.get("output_name") or f"{index:04d}_{image.stem}"+ ).strip()+ items.append(+ GenerationInput(+ image=image,+ actions=actions,+ prompts=prompts,+ chunk_prompts=chunk_prompts,+ output_name=output_name,+ )+ )+ return items+++def validate_geometry(height: int, width: int) -> None:+ if height <= 0 or width <= 0:+ raise ValueError("height and width must be positive")+ if height % 64 or width % 64:+ raise ValueError(+ "height and width must be divisible by 64 for HR/LR VAE and patch alignment"+ )+++def load_action_sequence(+ *,+ action: str,+ perspective: str,+ rotation_speed_deg: float = 3.0,+ yaw_rotation_speed_deg: float | None = None,+ pitch_rotation_speed_deg: float | None = None,+ chunk_length: int,+ sink_size: int,+ temporal_size: int,+) -> ActionSequence:+ """Parse an action string and validate its generation layout.++ This validates both the action syntax and its relationships with the+ chunked stage-three inference inputs.+ """+ if chunk_length <= 0:+ raise ValueError("chunk_length must be positive")+ if chunk_length % 2:+ raise ValueError(+ "chunk_length must be even for HR-to-LR temporal downsampling"+ )+ if sink_size < 0:+ raise ValueError("sink_size must be non-negative")+ if sink_size > chunk_length:+ raise ValueError("sink_size must not exceed chunk_length")+ if not 1 <= temporal_size <= chunk_length:+ raise ValueError("temporal_size must be in [1, chunk_length]")++ actions = parse_action_string(+ action,+ perspective=perspective,+ rotation_speed_deg=rotation_speed_deg,+ yaw_rotation_speed_deg=yaw_rotation_speed_deg,+ pitch_rotation_speed_deg=pitch_rotation_speed_deg,+ )+ if actions.latent_frames % chunk_length:+ raise ValueError(+ f"action produces {actions.latent_frames} latent frames, which is "+ f"not divisible by chunk_length={chunk_length}"+ )+ if sink_size >= actions.latent_frames:+ raise ValueError("sink_size must be smaller than the action length")+ return actions+++def load_inference_inputs(+ args: Any,+ *,+ default_guide_scale: float | tuple[float, ...],+) -> InferenceInputs:+ """Resolve and validate all data and sampling inputs from CLI arguments."""+ validate_geometry(args.height, args.width)+ items = load_generation_inputs(+ input_json=args.input_json,+ rotation_speed_deg=args.rotation_speed_deg,+ yaw_rotation_speed_deg=getattr(args, "yaw_rotation_speed_deg", None),+ pitch_rotation_speed_deg=getattr(args, "pitch_rotation_speed_deg", None),+ chunk_length=args.chunk_length,+ sink_size=args.sink_size,+ temporal_size=args.temporal_size,+ )++ mode = str(args.inference_mode)+ if mode not in {"bi", "ar", "pdd"}:+ raise ValueError(f"unsupported inference mode: {mode!r}")+ sample_shift = (+ float(args.sample_shift)+ if args.sample_shift is not None+ else (7.0 if mode == "pdd" else 5.0)+ )+ if not math.isfinite(sample_shift) or sample_shift <= 0:+ raise ValueError("sample_shift must be a positive finite number")+ sample_steps = (+ int(args.sample_steps)+ if args.sample_steps is not None+ else (4 if mode == "pdd" else 40)+ )+ if sample_steps <= 0:+ raise ValueError("sample_steps must be positive")+ guide_scale = (+ args.sample_guide_scale+ if args.sample_guide_scale is not None+ else (1.0 if mode == "pdd" else default_guide_scale)+ )+ guide_scales = (+ (float(guide_scale),)+ if isinstance(guide_scale, (int, float))+ else tuple(float(scale) for scale in guide_scale)+ )+ if not guide_scales or any(+ not math.isfinite(scale) or scale < 0 for scale in guide_scales+ ):+ raise ValueError(+ "sample_guide_scale must contain finite non-negative values"+ )++ if mode == "pdd":+ if sample_steps != 4:+ raise ValueError("--inference_mode pdd requires --sample_steps 4")+ if not math.isclose(sample_shift, 7.0, rel_tol=0.0, abs_tol=1e-8):+ raise ValueError("--inference_mode pdd requires --sample_shift 7")+ if any(+ not math.isclose(scale, 1.0, rel_tol=0.0, abs_tol=1e-8)+ for scale in guide_scales+ ):+ raise ValueError(+ "--inference_mode pdd requires --sample_guide_scale 1"+ )++ return InferenceInputs(+ items=items,+ mode=mode,+ sample_shift=sample_shift,+ sample_steps=sample_steps,+ guide_scale=guide_scale,+ )+++def initialize_runtime(args: Any) -> RuntimeContext:+ """Validate runtime options and initialize distributed execution."""+ rank = int(os.getenv("RANK", "0"))+ world_size = int(os.getenv("WORLD_SIZE", "1"))+ local_rank = int(os.getenv("LOCAL_RANK", "0"))+ if world_size <= 0:+ raise ValueError("WORLD_SIZE must be positive")+ if not 0 <= rank < world_size:+ raise ValueError("RANK must be in [0, WORLD_SIZE)")+ if not 0 <= local_rank < world_size:+ raise ValueError("LOCAL_RANK must be in [0, WORLD_SIZE)")++ if world_size == 1 and (+ args.t5_fsdp or args.dit_fsdp or args.ulysses_size != 1+ ):+ raise ValueError(+ "FSDP/SP options require torchrun with multiple processes"+ )++ if args.ulysses_size <= 0:+ raise ValueError("ulysses_size must be positive")+ if args.ulysses_size > 1 and args.ulysses_size != world_size:+ raise ValueError("ulysses_size must equal world size")++ fp8_enabled = bool(args.fp8_ffn or args.fp8_qkv)+ if fp8_enabled and args.dit_fsdp:+ raise ValueError("DiT FP8 cannot be combined with --dit_fsdp")+ if args.fp8_qkv and args.disable_self_qkv_fusion:+ raise ValueError(+ "--fp8_qkv requires self-QKV fusion; remove "+ "--disable_self_qkv_fusion"+ )+ if args.fp8_ffn and not (+ 0 <= args.fp8_start_block <= args.fp8_end_block < 40+ ):+ raise ValueError(+ "FP8 block range must satisfy 0 <= start <= end < 40"+ )+ if args.fp8_qkv and not (+ 0 <= args.fp8_qkv_start_block <= args.fp8_qkv_end_block < 40+ ):+ raise ValueError(+ "FP8 QKV block range must satisfy 0 <= start <= end < 40"+ )++ offload_model = args.offload_model+ if offload_model is None:+ offload_model = world_size == 1+ if fp8_enabled and offload_model:+ raise ValueError("DiT FP8 requires --offload_model false")++ output_dir = Path(args.output_dir).expanduser()+ if rank == 0:+ output_dir.mkdir(parents=True, exist_ok=True)++ if not (+ args.disable_compiled_qk_rope+ and args.disable_compiled_block_glue+ and args.disable_compiled_ffn+ ):+ compile_root = (+ Path(args.compile_cache_dir).expanduser()+ / f"rank_{rank:02d}"+ )+ inductor_cache = compile_root / "torchinductor"+ triton_cache = compile_root / "triton"+ for cache_dir in (inductor_cache, triton_cache):+ cache_dir.mkdir(parents=True, exist_ok=True)+ os.environ["TORCHINDUCTOR_CACHE_DIR"] = str(inductor_cache)+ os.environ["TRITON_CACHE_DIR"] = str(triton_cache)++ if world_size > 1:+ torch.cuda.set_device(local_rank)+ dist.init_process_group(+ "nccl",+ init_method="env://",+ device_id=torch.device("cuda", local_rank),+ )+ if args.ulysses_size > 1:+ init_distributed_group()+ dist.barrier(device_ids=[local_rank])++ seed = (+ int(args.seed)+ if args.seed >= 0+ else random.randint(0, sys.maxsize)+ )+ if dist.is_initialized():+ seed_box = [seed if rank == 0 else None]+ dist.broadcast_object_list(seed_box, src=0)+ seed = int(seed_box[0])++ return RuntimeContext(+ rank=rank,+ world_size=world_size,+ local_rank=local_rank,+ output_dir=output_dir,+ offload_model=bool(offload_model),+ seed=seed,+ )diff --git a/worldplay2/models/__init__.py b/worldplay2/models/__init__.pynew file mode 100644index 0000000..482f8f5--- /dev/null+++ b/worldplay2/models/__init__.py@@ -0,0 +1,3 @@+from .model import WorldPlay2Model++__all__ = ["WorldPlay2Model"]diff --git a/worldplay2/models/attention.py b/worldplay2/models/attention.pynew file mode 100644index 0000000..843b194--- /dev/null+++ b/worldplay2/models/attention.py@@ -0,0 +1,282 @@+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.+from functools import lru_cache++import torch++try:+ import flash_attn_interface+ FLASH_ATTN_3_AVAILABLE = True+except ModuleNotFoundError:+ flash_attn_interface = None+ FLASH_ATTN_3_AVAILABLE = False++try:+ import flash_attn+ FLASH_ATTN_2_AVAILABLE = True+except ModuleNotFoundError:+ flash_attn = None+ FLASH_ATTN_2_AVAILABLE = False++import warnings++__all__ = [+ 'flash_attention',+ 'flash_attention_dense',+ 'sage_attention',+ 'resolve_attention_backend',+ 'attention',+]+++@lru_cache(maxsize=1)+def _load_sage_attention():+ """Load the optional CUDA backend only when it is selected."""+ try:+ from sageattention import sageattn+ except (ImportError, OSError) as exc:+ raise RuntimeError(+ "attention_backend='sage' requires a working SageAttention "+ "installation. Install it with `pip install sageattention "+ "--no-build-isolation` in your CUDA environment, or use "+ "attention_backend='flash'."+ ) from exc+ return sageattn+++def resolve_attention_backend(backend: str) -> str:+ """Normalize the backend name and check optional dependencies early."""+ backend = backend.lower()+ if backend not in ('flash', 'sage'):+ raise ValueError(+ "attention_backend must be 'flash' or 'sage', "+ f"got {backend!r}"+ )+ if backend == 'sage':+ _load_sage_attention()+ return backend+++def sage_attention(+ q,+ k,+ v,+ *,+ k_lens=None,+ softmax_scale=None,+ causal=False,+ window_size=(-1, -1),+):+ """Dense SageAttention in the model's ``[B, S, H, D]`` layout.++ Only full key sequences are supported here. Masked history prefill and+ text cross-attention keep their existing SDPA/FlashAttention paths.+ """+ if tuple(window_size) != (-1, -1):+ raise NotImplementedError(+ "SageAttention does not support sliding-window attention in "+ "WorldPlay2; use attention_backend='flash'."+ )+ if k_lens is not None and not torch.all(k_lens == k.shape[1]):+ raise NotImplementedError(+ "SageAttention requires dense, equal-length keys; "+ "use attention_backend='flash' for padded/variable lengths."+ )+ return _load_sage_attention()(+ q.contiguous(),+ k.contiguous(),+ v.contiguous(),+ tensor_layout='NHD',+ is_causal=causal,+ sm_scale=softmax_scale,+ )+++def flash_attention_dense(+ q,+ k,+ v,+ *,+ softmax_scale=None,+ causal=False,+):+ """Dense attention using FA3 when available, otherwise PyTorch SDPA.++ Inputs and output use the model-native layout ``[B, S, H, D]``. This is+ preferable to the varlen wrapper when every batch item has the same+ sequence length because it avoids flattening and constructing cu_seqlens.+ """+ if FLASH_ATTN_3_AVAILABLE:+ return flash_attn_interface.flash_attn_func(+ q=q.contiguous(),+ k=k.contiguous(),+ v=v.contiguous(),+ softmax_scale=softmax_scale,+ causal=causal,+ )++ out = torch.nn.functional.scaled_dot_product_attention(+ q.transpose(1, 2),+ k.transpose(1, 2),+ v.transpose(1, 2),+ is_causal=causal,+ scale=softmax_scale,+ )+ return out.transpose(1, 2).contiguous()+++def flash_attention(+ q,+ k,+ v,+ q_lens=None,+ k_lens=None,+ dropout_p=0.,+ softmax_scale=None,+ q_scale=None,+ causal=False,+ window_size=(-1, -1),+ deterministic=False,+ dtype=torch.bfloat16,+ version=None,+):+ """+ q: [B, Lq, Nq, C1].+ k: [B, Lk, Nk, C1].+ v: [B, Lk, Nk, C2]. Nq must be divisible by Nk.+ q_lens: [B].+ k_lens: [B].+ dropout_p: float. Dropout probability.+ softmax_scale: float. The scaling of QK^T before applying softmax.+ causal: bool. Whether to apply causal attention mask.+ window_size: (left right). If not (-1, -1), apply sliding window local attention.+ deterministic: bool. If True, slightly slower and uses more memory.+ dtype: torch.dtype. Apply when dtype of q/k/v is not float16/bfloat16.+ """+ half_dtypes = (torch.float16, torch.bfloat16)+ assert dtype in half_dtypes+ assert q.device.type == 'cuda' and q.size(-1) <= 256++ # params+ b, lq, lk, out_dtype = q.size(0), q.size(1), k.size(1), q.dtype++ def half(x):+ return x if x.dtype in half_dtypes else x.to(dtype)++ # preprocess query+ if q_lens is None:+ q = half(q.flatten(0, 1))+ q_lens = torch.tensor(+ [lq] * b, dtype=torch.int32).to(+ device=q.device, non_blocking=True)+ else:+ q = half(torch.cat([u[:v] for u, v in zip(q, q_lens)]))++ # preprocess key, value+ if k_lens is None:+ k = half(k.flatten(0, 1))+ v = half(v.flatten(0, 1))+ k_lens = torch.tensor(+ [lk] * b, dtype=torch.int32).to(+ device=k.device, non_blocking=True)+ else:+ k = half(torch.cat([u[:v] for u, v in zip(k, k_lens)]))+ v = half(torch.cat([u[:v] for u, v in zip(v, k_lens)]))++ q = q.to(v.dtype)+ k = k.to(v.dtype)++ if q_scale is not None:+ q = q * q_scale++ if version is not None and version == 3 and not FLASH_ATTN_3_AVAILABLE:+ warnings.warn(+ 'Flash attention 3 is not available, use flash attention 2 instead.'+ )++ # apply attention+ if (version is None or version == 3) and FLASH_ATTN_3_AVAILABLE:+ # Note: dropout_p, window_size are not supported in FA3 now.+ x = flash_attn_interface.flash_attn_varlen_func(+ q=q,+ k=k,+ v=v,+ cu_seqlens_q=torch.cat([q_lens.new_zeros([1]), q_lens]).cumsum(+ 0, dtype=torch.int32).to(q.device, non_blocking=True),+ cu_seqlens_k=torch.cat([k_lens.new_zeros([1]), k_lens]).cumsum(+ 0, dtype=torch.int32).to(q.device, non_blocking=True),+ seqused_q=None,+ seqused_k=None,+ max_seqlen_q=lq,+ max_seqlen_k=lk,+ softmax_scale=softmax_scale,+ causal=causal,+ deterministic=deterministic).unflatten(0, (b, lq))+ else:+ assert FLASH_ATTN_2_AVAILABLE+ x = flash_attn.flash_attn_varlen_func(+ q=q,+ k=k,+ v=v,+ cu_seqlens_q=torch.cat([q_lens.new_zeros([1]), q_lens]).cumsum(+ 0, dtype=torch.int32).to(q.device, non_blocking=True),+ cu_seqlens_k=torch.cat([k_lens.new_zeros([1]), k_lens]).cumsum(+ 0, dtype=torch.int32).to(q.device, non_blocking=True),+ max_seqlen_q=lq,+ max_seqlen_k=lk,+ dropout_p=dropout_p,+ softmax_scale=softmax_scale,+ causal=causal,+ window_size=window_size,+ deterministic=deterministic).unflatten(0, (b, lq))++ # output+ return x.type(out_dtype)+++def attention(+ q,+ k,+ v,+ q_lens=None,+ k_lens=None,+ dropout_p=0.,+ softmax_scale=None,+ q_scale=None,+ causal=False,+ window_size=(-1, -1),+ deterministic=False,+ dtype=torch.bfloat16,+ fa_version=None,+):+ if FLASH_ATTN_2_AVAILABLE or FLASH_ATTN_3_AVAILABLE:+ return flash_attention(+ q=q,+ k=k,+ v=v,+ q_lens=q_lens,+ k_lens=k_lens,+ dropout_p=dropout_p,+ softmax_scale=softmax_scale,+ q_scale=q_scale,+ causal=causal,+ window_size=window_size,+ deterministic=deterministic,+ dtype=dtype,+ version=fa_version,+ )+ else:+ if q_lens is not None or k_lens is not None:+ warnings.warn(+ 'Padding mask is disabled when using scaled_dot_product_attention. It can have a significant impact on performance.'+ )+ attn_mask = None++ q = q.transpose(1, 2).to(dtype)+ k = k.transpose(1, 2).to(dtype)+ v = v.transpose(1, 2).to(dtype)++ out = torch.nn.functional.scaled_dot_product_attention(+ q, k, v, attn_mask=attn_mask, is_causal=causal, dropout_p=dropout_p)++ out = out.transpose(1, 2).contiguous()+ return outdiff --git a/worldplay2/models/causal_compressor.py b/worldplay2/models/causal_compressor.pynew file mode 100644index 0000000..e4ba8dd--- /dev/null+++ b/worldplay2/models/causal_compressor.py@@ -0,0 +1,167 @@+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.+"""Causal memory compressor for WorldPlay2.++This is the Wan2.2 inference-side port of the FastVideo training module+``fastvideo/models/hyvideo/models/transformers/modules/causal_memory_compress.py``+(``CausalMemCompressModel`` / ``CausalCompressConv3D``).++The ONLY difference from the non-causal ``MemCompressModel`` is that every+convolution spanning the TEMPORAL axis is made causal: an output frame at time+``t`` may depend on input frames ``<= t`` (current + past) only, never on+future frames. This is achieved by padding the temporal axis on the LEFT+(past) side only -- exactly like ``CausalConv3d`` in the Wan VAE+(``worldplay2/vae/wan21.py``) and ``WanCausalConv3d`` in the FastVideo wanvae.++Concretely (matching the FastVideo training module verbatim):+ - ``self.conv`` (kernel 3, temporal padding 1) becomes a ``CausalConv3d``:+ instead of symmetric temporal padding (1 left + 1 right) it pads+ ``2*1 = 2`` frames on the left and 0 on the right, so the output length is+ unchanged but each frame only sees the past.+ - ``self.time_conv`` (kernel 3, temporal stride 2 -- the temporal+ downsample) becomes a ``CausalConv3d`` with temporal padding 1 -> 2 frames+ of left padding, 0 right. The old non-causal implementation zero-padded 1+ frame on the RIGHT (future) side, which leaked one future frame into every+ downsampled output.++ OUTPUT-LENGTH INVARIANT: the temporal downsample must still map+ ``T_hr -> T_hr/2`` so ``hr_tokens`` line up token-by-token with+ ``lr_tokens`` (which live at ``T_lr = T_hr/2``). For the wan latent+ cadence ``T_hr`` is always EVEN; with an even input:+ old (right-pad 1): out = (T_hr + 1 - 3)//2 + 1 = T_hr/2+ new (left-pad 2): out = (T_hr + 2 - 3)//2 + 1 = T_hr/2+ so the output length is IDENTICAL -- only the temporal receptive field+ shifts from centered to left/causal.+ - ``self.resample`` only touches H/W (spatial), so it is left unchanged.+ - ``_AvgDown3D`` (the temporal shortcut) already pads temporal on the left+ only, so it is already causal.++The compressor here is run over the WHOLE history in one shot (not streamed),+so ``CausalConv3d`` is always called with ``cache_x=None`` (standard causal+left-pad). The ``cache_x`` streaming interface is inherited but unused.+"""++import torch+import torch.nn as nn+from einops import rearrange++# Reuse the causal 3d conv from the Wan VAE (identical padding semantics to+# FastVideo's WanCausalConv3d).+from ..vae.wan21 import CausalConv3d+# Reuse the (already-causal / time-independent) helper blocks from the+# non-causal compressor so weights and shapes match the training module.+from .compressor_blocks import _AvgDown3D, _WanAttentionBlock+++class CausalCompressConv3D(nn.Module):+ """Causal-in-time version of ``memory_compress.CompressConv3D``.++ Same structure and parameter layout as ``CompressConv3D`` except the two+ temporal-spanning convolutions (``conv`` and ``time_conv``) are causal.+ """++ def __init__(self, in_dim, out_dim, spatial_down, temporal_down,+ non_linearity='silu'):+ super().__init__()+ # Causal in time: kernel 3 with padding 1 -> left-only temporal pad.+ self.conv = CausalConv3d(in_channels=in_dim, out_channels=out_dim,+ kernel_size=3, stride=1, padding=1)+ if non_linearity == 'silu':+ self.nonlinearity = nn.SiLU()+ else:+ raise ValueError(+ f"Unsupported non_linearity={non_linearity!r}. "+ "Only 'silu' is used by Wan2.2 5B; extend if needed.")+ self.spatial_down = spatial_down+ self.temporal_down = temporal_down++ if self.spatial_down:+ # Spatial-only downsample -- no temporal axis touched, unchanged.+ self.resample = nn.Sequential(+ nn.ZeroPad2d((0, 1, 0, 1)),+ nn.Conv2d(out_dim, out_dim, 3, stride=(2, 2)),+ )++ if self.temporal_down:+ # Causal temporal downsample: kernel 3, stride 2, temporal+ # padding 1 -> CausalConv3d pads 2 frames on the LEFT (past) and 0+ # on the right, so no future frame leaks into the output.+ self.time_conv = CausalConv3d(+ out_dim, out_dim, kernel_size=(3, 1, 1),+ stride=(2, 1, 1), padding=(1, 0, 0))++ if self.spatial_down or self.temporal_down:+ self.avg_shortcut = _AvgDown3D(+ in_dim, out_dim,+ factor_t=2 if temporal_down else 1,+ factor_s=2 if spatial_down else 1,+ )+ else:+ self.avg_shortcut = nn.Conv3d(in_dim, out_dim, kernel_size=1,+ stride=1, padding=0)++ def forward(self, x):+ x_copy = x.clone()+ x = self.conv(x)+ x = self.nonlinearity(x)+ b, c, t, h, w = x.shape+ if self.spatial_down:+ x = x.permute(0, 2, 1, 3, 4).reshape(b * t, c, h, w)+ x = self.resample(x)+ x = x.view(b, t, x.size(1), x.size(2),+ x.size(3)).permute(0, 2, 1, 3, 4)+ if self.temporal_down:+ x = self.time_conv(x)+ return x + self.avg_shortcut(x_copy)+++class CausalMemCompressModel(nn.Module):+ """Causal-in-time variant of ``memory_compress.MemCompressModel``.++ Identical to ``MemCompressModel`` except each ``CompressConv3D`` block is+ replaced by ``CausalCompressConv3D`` so the temporal receptive field is+ strictly causal (an output token never depends on future latent frames).+ The ``attn_blocks`` are per-frame spatial attention (no temporal mixing),+ so they are already causal and reused verbatim.+ """++ def __init__(+ self,+ output_dim=2048,+ input_dim=65,+ dims=(64, 64, 128, 256, 256, 512, 512),+ spatial_down=(1, 1, 0, 0, 0, 0),+ temporal_down=(1, 0, 0, 0, 0, 0),+ attn_num=2,+ non_linearity: str = "silu",+ ):+ super().__init__()+ dims = list(dims)+ spatial_down = list(spatial_down)+ temporal_down = list(temporal_down)++ self.input_layer = nn.Linear(input_dim, dims[0])++ self.blocks = nn.ModuleList()+ for i, (in_dim, out_dim) in enumerate(zip(dims[:-1], dims[1:])):+ self.blocks.append(CausalCompressConv3D(+ in_dim, out_dim, spatial_down[i], temporal_down[i],+ non_linearity=non_linearity))++ self.attn_blocks = nn.ModuleList(+ [_WanAttentionBlock(dims[-1], head_dim=8) for _ in range(attn_num)])++ self.output_layer = nn.Linear(dims[-1], output_dim)+ nn.init.zeros_(self.output_layer.weight)+ nn.init.zeros_(self.output_layer.bias)++ def forward(self, x):+ B, C, T, H, W = x.shape+ x = rearrange(x, 'B C T H W -> B (T H W) C')+ x = self.input_layer(x)+ x = rearrange(x, 'B (T H W) C -> B C T H W', T=T, H=H, W=W)+ for block in self.blocks:+ x = block(x)+ for attn_block in self.attn_blocks:+ x = attn_block(x)+ x = rearrange(x, 'B C T H W -> B (T H W) C')+ return self.output_layer(x)diff --git a/worldplay2/models/compressor_blocks.py b/worldplay2/models/compressor_blocks.pynew file mode 100644index 0000000..6c9cff8--- /dev/null+++ b/worldplay2/models/compressor_blocks.py@@ -0,0 +1,129 @@+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.+"""Hierarchical HR -> compressed-memory feature extractor used by the+stage-2 / stage-3 chunked inference recipe.++This is a self-contained port of ``MemCompressModel`` from the FastVideo+training pipeline. It only depends on ``torch.nn`` plus ``einops`` so+that it can be imported from ``worldplay2/models/model.py`` without dragging in+any FastVideo-specific layers / autocast helpers.++Wan2.2 5B contract (matches ``WorldPlay2Model.add_memory_compress_model``):+ input_dim = 36 (16 hr-latent + 20 hr-y)+ spatial_down = [1, 1, 1, 0, 0, 0] (3 spatial down blocks * 2 each = /8)+ temporal_down = [1, 0, 0, 0, 0, 0] (1 temporal down block * 2 = /2)+"""++import torch+import torch.nn as nn+import torch.nn.functional as F+from einops import rearrange+++# ---------------------------------------------------------------------------+# Local copies of the 3 helpers MemCompressModel uses from FastVideo.+# Kept verbatim so checkpoints trained against FastVideo load directly.+# ---------------------------------------------------------------------------++class _AvgDown3D(nn.Module):+ """Channel-folding average-pool used as the residual shortcut inside+ ``CompressConv3D``. Identical to ``fastvideo.models.vaes.wanvae.AvgDown3D``.+ """++ def __init__(self, in_channels, out_channels, factor_t, factor_s=1):+ super().__init__()+ self.in_channels = in_channels+ self.out_channels = out_channels+ self.factor_t = factor_t+ self.factor_s = factor_s+ self.factor = factor_t * factor_s * factor_s++ assert in_channels * self.factor % out_channels == 0+ self.group_size = in_channels * self.factor // out_channels++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ pad_t = (self.factor_t - x.shape[2] % self.factor_t) % self.factor_t+ pad = (0, 0, 0, 0, pad_t, 0)+ x = F.pad(x, pad)+ B, C, T, H, W = x.shape+ x = x.view(+ B, C,+ T // self.factor_t, self.factor_t,+ H // self.factor_s, self.factor_s,+ W // self.factor_s, self.factor_s,+ )+ x = x.permute(0, 1, 3, 5, 7, 2, 4, 6).contiguous()+ x = x.view(+ B, C * self.factor,+ T // self.factor_t,+ H // self.factor_s,+ W // self.factor_s,+ )+ x = x.view(+ B, self.out_channels, self.group_size,+ T // self.factor_t,+ H // self.factor_s,+ W // self.factor_s,+ )+ return x.mean(dim=2)+++class _WanRMS_norm(nn.Module):+ """Same as ``fastvideo.models.vaes.wanvae.WanRMS_norm``. Operates+ per-frame on (B*T, C, H, W) tensors.+ """++ def __init__(self, dim, channel_first=True, images=True, bias=False):+ super().__init__()+ broadcastable_dims = (1, 1, 1) if not images else (1, 1)+ shape = (dim, *broadcastable_dims) if channel_first else (dim,)+ self.channel_first = channel_first+ self.scale = dim ** 0.5+ self.gamma = nn.Parameter(torch.ones(shape))+ self.bias = nn.Parameter(torch.zeros(shape)) if bias else 0.0++ def forward(self, x):+ return F.normalize(+ x, dim=(1 if self.channel_first else -1)+ ) * self.scale * self.gamma + self.bias+++class _WanAttentionBlock(nn.Module):+ """Multi-head per-frame self-attention. Identical to+ ``fastvideo.models.vaes.wanvae.WanAttentionBlock``; this is the+ ``head_dim=8`` variant used by ``MemCompressModel``.+ """++ def __init__(self, dim, head_dim=1):+ super().__init__()+ self.dim = dim+ self.head_dim = head_dim+ self.norm = _WanRMS_norm(dim)+ self.to_qkv = nn.Conv2d(dim, dim * 3, 1)+ self.proj = nn.Conv2d(dim, dim, 1)++ def forward(self, x):+ identity = x+ batch_size, channels, time, height, width = x.size()++ x = x.permute(0, 2, 1, 3, 4).reshape(+ batch_size * time, channels, height, width)+ x = self.norm(x)++ qkv = self.to_qkv(x)+ qkv = qkv.reshape(batch_size * time, 1, channels * 3, -1)+ qkv = qkv.permute(0, 1, 3, 2).contiguous()+ q, k, v = qkv.chunk(3, dim=-1)+ q = rearrange(q, "b h l (H d) -> b (h H) l d", H=self.head_dim)+ k = rearrange(k, "b h l (H d) -> b (h H) l d", H=self.head_dim)+ v = rearrange(v, "b h l (H d) -> b (h H) l d", H=self.head_dim)++ x = F.scaled_dot_product_attention(q, k, v)+ x = rearrange(x, "b h l d -> b (h d) l")++ x = x.reshape(batch_size * time, channels, height, width)+ x = self.proj(x)++ x = x.view(batch_size, time, channels, height, width)+ x = x.permute(0, 2, 1, 3, 4)+ return x + identity+diff --git a/worldplay2/models/fp8.py b/worldplay2/models/fp8.pynew file mode 100644index 0000000..80de6f5--- /dev/null+++ b/worldplay2/models/fp8.py@@ -0,0 +1,400 @@+from __future__ import annotations++import gc+from contextlib import nullcontext++import torch+import torch.nn as nn+import torch.nn.functional as F+++def _load_transformer_engine():+ try:+ import transformer_engine.pytorch as te+ from transformer_engine.common.recipe import DelayedScaling, Format+ except ImportError as exc:+ raise RuntimeError(+ "FFN FP8 requires NVIDIA Transformer Engine. Install the "+ "PyTorch extension with `pip install 'transformer_engine[pytorch]'`."+ ) from exc+ return te, DelayedScaling, Format+++def make_fp8_recipe():+ """Hopper-friendly tensor-wise E4M3 delayed scaling for inference."""+ te, DelayedScaling, Format = _load_transformer_engine()+ available = te.is_fp8_available(return_reason=True)+ if isinstance(available, tuple):+ is_available, reason = available+ else:+ is_available, reason = available, "unsupported GPU/runtime"+ if not is_available:+ raise RuntimeError(f"Transformer Engine FP8 is unavailable: {reason}")+ return DelayedScaling(+ fp8_format=Format.E4M3,+ amax_history_len=16,+ amax_compute_algo="max",+ # Sequence-parallel ranks process different tokens. Local activation+ # scales avoid an extra amax all-reduce and are dequantized before the+ # next distributed attention.+ reduce_amax=False,+ )+++def fp8_autocast(recipe, enabled: bool):+ if not enabled:+ return nullcontext()+ te, _, _ = _load_transformer_engine()+ autocast = getattr(te, "autocast", None)+ if autocast is None:+ autocast = te.fp8_autocast+ return autocast(enabled=True, recipe=recipe)+++def _primary_fp8_init_context(te, recipe):+ """Create TE modules whose primary weights are stored only in FP8."""+ quantized_model_init = getattr(te, "quantized_model_init", None)+ if quantized_model_init is None:+ raise RuntimeError(+ "This FP8 inference path requires "+ "transformer_engine.pytorch.quantized_model_init. "+ "Transformer Engine 2.18.0 provides this API."+ )+ return quantized_model_init(+ enabled=True,+ recipe=recipe,+ preserve_high_precision_init_val=False,+ )+++def _copy_weight_to_primary_fp8(destination, source, recipe) -> None:+ """Calibrate a static E4M3 weight, then quantize it into TE storage."""+ quantizer_getter = getattr(destination, "_get_quantizer", None)+ if quantizer_getter is not None:+ quantizer = quantizer_getter()+ scale = getattr(quantizer, "scale", None)+ scale_inv = getattr(destination, "_scale_inv", None)+ if scale is not None and scale_inv is not None:+ # DelayedScaling would otherwise reuse the random initialization+ # scale for this first copy. Static inference weights can be+ # calibrated exactly once from their checkpoint amax.+ amax = source.detach().abs().amax().float()+ fp8_max = 448.0 # E4M3 finite maximum+ margin = int(getattr(recipe, "margin", 0))+ calibrated_scale = (fp8_max / amax.clamp_min(+ torch.finfo(torch.float32).tiny)) / (2 ** margin)+ scale.copy_(calibrated_scale.to(scale))+ scale_inv.copy_(calibrated_scale.reciprocal().to(scale_inv))++ destination.copy_(source)+++class TransformerEngineFP8FFN(nn.Module):+ """FFN with FP8-only primary weights and FP8 E4M3 GEMM execution."""++ uses_fp8 = True++ def __init__(self, source: nn.Sequential, recipe, name: str):+ super().__init__()+ if (+ len(source) != 3+ or not isinstance(source[0], nn.Linear)+ or not isinstance(source[1], nn.GELU)+ or not isinstance(source[2], nn.Linear)+ ):+ raise TypeError("FP8 FFN expects Linear -> GELU -> Linear")++ te, _, _ = _load_transformer_engine()+ first, activation, second = source+ if first.in_features % 16 or first.out_features % 16:+ raise ValueError(+ f"{name}.ffn[0] shape is not FP8 GEMM eligible: "+ f"{first.in_features}x{first.out_features}"+ )+ if second.in_features % 16 or second.out_features % 16:+ raise ValueError(+ f"{name}.ffn[2] shape is not FP8 GEMM eligible: "+ f"{second.in_features}x{second.out_features}"+ )++ device = first.weight.device+ dtype = first.weight.dtype+ # TE only honors quantized_model_init during no-grad construction.+ # params_dtype remains BF16 as the nominal compute/output dtype, while+ # the actual primary weight storage is a TE Float8Tensor.+ with torch.no_grad(), _primary_fp8_init_context(te, recipe):+ self.fc1 = te.Linear(+ first.in_features,+ first.out_features,+ bias=first.bias is not None,+ params_dtype=dtype,+ device=device,+ name=f"{name}.fc1",+ )+ self.fc2 = te.Linear(+ second.in_features,+ second.out_features,+ bias=second.bias is not None,+ params_dtype=dtype,+ device=device,+ name=f"{name}.fc2",+ )++ if not (+ getattr(self.fc1, "primary_weights_in_fp8", False)+ and getattr(self.fc2, "primary_weights_in_fp8", False)+ ):+ raise RuntimeError(+ f"{name}: Transformer Engine did not create FP8 primary "+ "weights. Check that TE 2.18.0 is installed and that "+ "quantized_model_init is enabled."+ )++ # QuantizedTensor.copy_ quantizes the BF16 checkpoint tensor directly+ # into the FP8 destination and updates its scale metadata.+ with torch.no_grad():+ _copy_weight_to_primary_fp8(+ self.fc1.weight, first.weight, recipe)+ _copy_weight_to_primary_fp8(+ self.fc2.weight, second.weight, recipe)+ if first.bias is not None:+ self.fc1.bias.copy_(first.bias)+ if second.bias is not None:+ self.fc2.bias.copy_(second.bias)++ self.activation = nn.GELU(approximate=activation.approximate)+ self.recipe = recipe+ self.primary_weights_in_fp8 = True+ self.forward_calls = 0++ def forward(self, hidden_states):+ # Transformer Engine FP8 GEMM requires:+ # M = product(hidden_states.shape[:-1]) to be divisible by 8+ # K = hidden_states.shape[-1] to be divisible by 16+ #+ # Sequence parallelism can produce local token counts such as 387 or+ # 386. FFN is token-wise, so pad only inside the FFN and remove the+ # padding before returning. Padded tokens never enter attention.+ original_shape = hidden_states.shape+ hidden_size = original_shape[-1]+ hidden_states = hidden_states.reshape(-1, hidden_size)+ num_tokens = hidden_states.shape[0]+ padded_tokens = (-num_tokens) % 8+ if padded_tokens:+ hidden_states = F.pad(+ hidden_states, (0, 0, 0, padded_tokens), value=0.0)++ # The weights are already primary Float8Tensors. Do not request the+ # BF16->FP8 weight workspace/cache used by regular TE Linear modules.+ hidden_states = self.fc1(hidden_states)+ hidden_states = self.activation(hidden_states)+ hidden_states = self.fc2(hidden_states)++ if padded_tokens:+ hidden_states = hidden_states[:num_tokens]+ hidden_states = hidden_states.reshape(*original_shape[:-1], -1)++ self.forward_calls += 1+ return hidden_states+++class TransformerEngineFP8Linear(nn.Module):+ """Drop-in inference Linear with FP8-only primary weight storage."""++ uses_fp8 = True+ primary_weights_in_fp8 = True++ def __init__(self, source: nn.Linear, recipe, name: str):+ super().__init__()+ if not isinstance(source, nn.Linear):+ raise TypeError("FP8 Linear conversion expects nn.Linear")+ if source.in_features % 16 or source.out_features % 16:+ raise ValueError(+ f"{name} shape is not FP8 GEMM eligible: "+ f"{source.in_features}x{source.out_features}"+ )++ te, _, _ = _load_transformer_engine()+ with torch.no_grad(), _primary_fp8_init_context(te, recipe):+ self.linear = te.Linear(+ source.in_features,+ source.out_features,+ bias=source.bias is not None,+ params_dtype=source.weight.dtype,+ device=source.weight.device,+ name=name,+ )+ if not getattr(self.linear, "primary_weights_in_fp8", False):+ raise RuntimeError(+ f"{name}: Transformer Engine did not create an FP8 primary "+ "weight. Check the TE 2.18.0 installation."+ )++ with torch.no_grad():+ _copy_weight_to_primary_fp8(+ self.linear.weight, source.weight, recipe)+ if source.bias is not None:+ self.linear.bias.copy_(source.bias)++ self.in_features = source.in_features+ self.out_features = source.out_features+ self.forward_calls = 0++ @property+ def weight(self):+ return self.linear.weight++ @property+ def bias(self):+ return self.linear.bias++ def forward(self, hidden_states):+ original_shape = hidden_states.shape+ hidden_states = hidden_states.reshape(-1, original_shape[-1])+ num_tokens = hidden_states.shape[0]+ padded_tokens = (-num_tokens) % 8+ if padded_tokens:+ hidden_states = F.pad(+ hidden_states, (0, 0, 0, padded_tokens), value=0.0)++ hidden_states = self.linear(hidden_states)++ if padded_tokens:+ hidden_states = hidden_states[:num_tokens]+ hidden_states = hidden_states.reshape(+ *original_shape[:-1], self.out_features)+ self.forward_calls += 1+ return hidden_states+++def convert_model_ffn_to_fp8(+ model,+ *,+ start_block: int,+ end_block: int,+) -> dict:+ """Replace inclusive middle-block FFNs with Transformer Engine modules."""+ if start_block < 0 or end_block < start_block:+ raise ValueError("invalid FP8 block range")+ if end_block >= len(model.blocks):+ raise ValueError(+ f"FP8 end block {end_block} exceeds model depth "+ f"{len(model.blocks)}"+ )++ recipe = model.fp8_recipe or make_fp8_recipe()+ converted = []+ for index in range(start_block, end_block + 1):+ block = model.blocks[index]+ if getattr(block.ffn, "uses_fp8", False):+ converted.append(index)+ continue+ block.ffn = TransformerEngineFP8FFN(+ block.ffn, recipe, name=f"blocks.{index}.ffn")+ block.ffn.eval().requires_grad_(False)+ converted.append(index)++ model.fp8_ffn_enabled = True+ model.fp8_enabled = True+ model.fp8_recipe = recipe+ model.fp8_block_range = (start_block, end_block)+ gc.collect()+ torch.cuda.empty_cache()+ return {+ "enabled": True,+ "recipe": "DelayedScaling",+ "format": "E4M3",+ "weight_storage": "primary_fp8",+ "bf16_master_weights": False,+ "reduce_amax": False,+ "blocks": converted,+ }+++def convert_model_qkv_to_fp8(+ model,+ *,+ start_block: int,+ end_block: int,+) -> dict:+ """Convert selected fused self-attention QKV projections to FP8."""+ if start_block < 0 or end_block < start_block:+ raise ValueError("invalid FP8 QKV block range")+ if end_block >= len(model.blocks):+ raise ValueError(+ f"FP8 QKV end block {end_block} exceeds model depth "+ f"{len(model.blocks)}"+ )+ if not getattr(model, "self_qkv_fused", False):+ raise ValueError(+ "QKV FP8 requires fused self-attention QKV projections")++ recipe = model.fp8_recipe or make_fp8_recipe()+ converted = []+ for index in range(start_block, end_block + 1):+ attention = model.blocks[index].self_attn+ if attention.qkv is None:+ raise RuntimeError(+ f"blocks.{index}.self_attn has no fused QKV projection")+ if getattr(attention.qkv, "uses_fp8", False):+ converted.append(index)+ continue+ attention.qkv = TransformerEngineFP8Linear(+ attention.qkv,+ recipe,+ name=f"blocks.{index}.self_attn.qkv",+ )+ attention.qkv.eval().requires_grad_(False)+ converted.append(index)++ model.fp8_qkv_enabled = True+ model.fp8_enabled = True+ model.fp8_recipe = recipe+ model.fp8_qkv_block_range = (start_block, end_block)+ gc.collect()+ torch.cuda.empty_cache()+ return {+ "enabled": True,+ "recipe": "DelayedScaling",+ "format": "E4M3",+ "weight_storage": "primary_fp8",+ "bf16_master_weights": False,+ "reduce_amax": False,+ "blocks": converted,+ }+++def fp8_model_stats(model) -> dict:+ ffn_blocks = []+ ffn_calls = 0+ qkv_blocks = []+ qkv_calls = 0+ for index, block in enumerate(model.blocks):+ if getattr(block.ffn, "uses_fp8", False):+ ffn_blocks.append(index)+ ffn_calls += int(block.ffn.forward_calls)+ if getattr(block.self_attn.qkv, "uses_fp8", False):+ qkv_blocks.append(index)+ qkv_calls += int(block.self_attn.qkv.forward_calls)+ enabled = bool(ffn_blocks or qkv_blocks)+ return {+ "enabled": enabled,+ "ffn": {+ "enabled": bool(ffn_blocks),+ "blocks": ffn_blocks,+ "forward_calls": ffn_calls,+ },+ "qkv": {+ "enabled": bool(qkv_blocks),+ "blocks": qkv_blocks,+ "forward_calls": qkv_calls,+ },+ "primary_weights_in_fp8": all(+ model.blocks[index].ffn.primary_weights_in_fp8+ for index in ffn_blocks+ ) and all(+ model.blocks[index].self_attn.qkv.primary_weights_in_fp8+ for index in qkv_blocks+ ) if enabled else False,+ "bf16_master_weights": False if enabled else None,+ }diff --git a/worldplay2/models/kernels.py b/worldplay2/models/kernels.pynew file mode 100644index 0000000..62b64b6--- /dev/null+++ b/worldplay2/models/kernels.py@@ -0,0 +1,496 @@+from __future__ import annotations++import logging+from functools import lru_cache++import torch+import torch.nn.functional as F++try:+ import triton+ import triton.language as tl+except ImportError:+ triton = None+ tl = None+++_QK_ROPE_STATS = {+ "compiled_calls": 0,+ "eager_calls": 0,+ "compile_failures": 0,+}+_REGION_STATS = {+ "layer_norm_modulation": {+ "triton_calls": 0,+ "eager_calls": 0,+ "failures": 0,+ },+ "modulation": {"compiled_calls": 0, "eager_calls": 0, "failures": 0},+ "gated_residual": {+ "compiled_calls": 0,+ "eager_calls": 0,+ "failures": 0,+ },+ "action_mlp": {+ "compiled_calls": 0,+ "eager_calls": 0,+ "failures": 0,+ },+ "ffn": {"compiled_calls": 0, "eager_calls": 0, "failures": 0},+}+_DISABLED_REGIONS: set[str] = set()+++if triton is not None:++ @triton.jit+ def _layer_norm_modulation_kernel(+ x_ptr,+ scale_ptr,+ shift_ptr,+ output_ptr,+ sequence_length: tl.constexpr,+ hidden_size: tl.constexpr,+ x_stride_b: tl.constexpr,+ x_stride_s: tl.constexpr,+ scale_stride_b: tl.constexpr,+ scale_stride_s: tl.constexpr,+ shift_stride_b: tl.constexpr,+ shift_stride_s: tl.constexpr,+ eps: tl.constexpr,+ INPUT_DTYPE: tl.constexpr,+ BLOCK_SIZE: tl.constexpr,+ ):+ """One Triton program normalizes and modulates one token row."""+ row = tl.program_id(0)+ batch_index = row // sequence_length+ sequence_index = row - batch_index * sequence_length+ columns = tl.arange(0, BLOCK_SIZE)+ mask = columns < hidden_size++ x_offsets = (+ batch_index * x_stride_b+ + sequence_index * x_stride_s+ + columns+ )+ x = tl.load(x_ptr + x_offsets, mask=mask, other=0.0).to(tl.float32)+ mean = tl.sum(x, axis=0) / hidden_size+ centered = tl.where(mask, x - mean, 0.0)+ variance = tl.sum(centered * centered, axis=0) / hidden_size+ normalized = centered * tl.rsqrt(variance + eps)+ # Match WorldPlay2LayerNorm exactly: FP32 normalization is rounded back to+ # the activation dtype before adaptive modulation evaluates in FP32.+ if INPUT_DTYPE == 1:+ normalized = normalized.to(tl.bfloat16).to(tl.float32)+ elif INPUT_DTYPE == 0:+ normalized = normalized.to(tl.float16).to(tl.float32)++ scale_offsets = (+ batch_index * scale_stride_b+ + sequence_index * scale_stride_s+ + columns+ )+ shift_offsets = (+ batch_index * shift_stride_b+ + sequence_index * shift_stride_s+ + columns+ )+ scale = tl.load(+ scale_ptr + scale_offsets, mask=mask, other=0.0+ ).to(tl.float32)+ shift = tl.load(+ shift_ptr + shift_offsets, mask=mask, other=0.0+ ).to(tl.float32)+ output = normalized * (1.0 + scale) + shift++ output_offsets = row * hidden_size + columns+ tl.store(output_ptr + output_offsets, output, mask=mask)+++def qk_norm_rope_eager(+ q: torch.Tensor,+ k: torch.Tensor,+ norm_q_weight: torch.Tensor,+ norm_k_weight: torch.Tensor,+ freqs_real: torch.Tensor,+ freqs_imag: torch.Tensor,+ *,+ eps: float,+ num_heads: int,+ head_dim: int,+) -> tuple[torch.Tensor, torch.Tensor]:+ """RMS-normalize Q/K and apply real-valued RoPE in FP32."""++ def normalize_and_rotate(x, weight):+ normalized = x.float()+ normalized = normalized * torch.rsqrt(+ normalized.square().mean(dim=-1, keepdim=True) + eps+ )+ # Preserve the original WorldPlay2RMSNorm order: normalize in FP32, cast+ # back to projection dtype, then apply the learned norm weight.+ normalized = normalized.to(x.dtype) * weight+ batch, sequence, _ = normalized.shape+ pairs = normalized.float().view(+ batch, sequence, num_heads, head_dim // 2, 2)+ even = pairs[..., 0]+ odd = pairs[..., 1]+ cos = freqs_real.float().view(+ 1, sequence, 1, head_dim // 2)+ sin = freqs_imag.float().view(+ 1, sequence, 1, head_dim // 2)+ return torch.stack(+ [even * cos - odd * sin, even * sin + odd * cos],+ dim=-1,+ ).flatten(-2)++ return (+ normalize_and_rotate(q, norm_q_weight),+ normalize_and_rotate(k, norm_k_weight),+ )+++@lru_cache(maxsize=16)+def _compiled_kernel(num_heads: int, head_dim: int, eps: float):+ def kernel(q, k, norm_q_weight, norm_k_weight, freqs_real, freqs_imag):+ return qk_norm_rope_eager(+ q,+ k,+ norm_q_weight,+ norm_k_weight,+ freqs_real,+ freqs_imag,+ eps=eps,+ num_heads=num_heads,+ head_dim=head_dim,+ )++ return torch.compile(+ kernel,+ fullgraph=True,+ dynamic=True,+ mode="max-autotune-no-cudagraphs",+ )+++def qk_norm_rope(+ q: torch.Tensor,+ k: torch.Tensor,+ norm_q_weight: torch.Tensor,+ norm_k_weight: torch.Tensor,+ freqs_flat: torch.Tensor,+ *,+ eps: float,+ num_heads: int,+ head_dim: int,+ compile_enabled: bool,+) -> tuple[torch.Tensor, torch.Tensor, bool]:+ """Run compiled QK norm + RoPE, falling back to the same eager kernel."""+ args = (+ q,+ k,+ norm_q_weight,+ norm_k_weight,+ freqs_flat.real,+ freqs_flat.imag,+ )+ if compile_enabled:+ try:+ output = _compiled_kernel(num_heads, head_dim, eps)(*args)+ _QK_ROPE_STATS["compiled_calls"] += 1+ return output[0], output[1], True+ except Exception:+ _QK_ROPE_STATS["compile_failures"] += 1+ logging.exception(+ "Compiled QK norm + RoPE failed; using eager fallback")++ output = qk_norm_rope_eager(+ *args,+ eps=eps,+ num_heads=num_heads,+ head_dim=head_dim,+ )+ _QK_ROPE_STATS["eager_calls"] += 1+ return output[0], output[1], False+++def get_qk_rope_compile_stats() -> dict:+ return dict(_QK_ROPE_STATS)+++# ---------------------------------------------------------------------------+# Repeated Wan block regions+# ---------------------------------------------------------------------------++def layer_norm_modulation_eager(+ hidden_states,+ scale,+ shift,+ *,+ eps,+):+ """Reference Wan LayerNorm followed by adaptive scale and shift."""+ normalized = F.layer_norm(+ hidden_states.float(),+ (hidden_states.shape[-1],),+ weight=None,+ bias=None,+ eps=eps,+ ).to(hidden_states.dtype)+ return normalized.float() * (1.0 + scale.float()) + shift.float()+++def layer_norm_modulation(+ hidden_states,+ scale,+ shift,+ *,+ eps,+ enabled,+):+ """Fuse non-affine LayerNorm and Wan modulation in one Triton kernel."""+ name = "layer_norm_modulation"+ can_use_triton = (+ enabled+ and name not in _DISABLED_REGIONS+ and triton is not None+ and hidden_states.is_cuda+ and hidden_states.ndim == 3+ and scale.ndim == 3+ and shift.ndim == 3+ and hidden_states.shape == scale.shape == shift.shape+ and hidden_states.dtype in (+ torch.bfloat16,+ torch.float16,+ torch.float32,+ )+ and hidden_states.stride(-1) == 1+ and scale.stride(-1) == 1+ and shift.stride(-1) == 1+ )+ if can_use_triton:+ try:+ batch, sequence, hidden_size = hidden_states.shape+ output = torch.empty(+ hidden_states.shape,+ device=hidden_states.device,+ dtype=torch.float32,+ )+ block_size = triton.next_power_of_2(hidden_size)+ if block_size > 65536:+ raise ValueError(+ f"LayerNorm hidden size {hidden_size} is too large")+ num_warps = 8 if block_size >= 4096 else 4+ _layer_norm_modulation_kernel[(batch * sequence,)](+ hidden_states,+ scale,+ shift,+ output,+ sequence_length=sequence,+ hidden_size=hidden_size,+ x_stride_b=hidden_states.stride(0),+ x_stride_s=hidden_states.stride(1),+ scale_stride_b=scale.stride(0),+ scale_stride_s=scale.stride(1),+ shift_stride_b=shift.stride(0),+ shift_stride_s=shift.stride(1),+ eps=eps,+ INPUT_DTYPE=(+ 1 if hidden_states.dtype == torch.bfloat16+ else 0 if hidden_states.dtype == torch.float16+ else 2+ ),+ BLOCK_SIZE=block_size,+ num_warps=num_warps,+ )+ _REGION_STATS[name]["triton_calls"] += 1+ return output, True+ except Exception:+ _REGION_STATS[name]["failures"] += 1+ _DISABLED_REGIONS.add(name)+ logging.exception(+ "Triton fused LayerNorm + modulation failed; "+ "disabling it and using eager fallback"+ )++ _REGION_STATS[name]["eager_calls"] += 1+ return layer_norm_modulation_eager(+ hidden_states,+ scale,+ shift,+ eps=eps,+ ), False+++def modulation_eager(normalized, scale, shift):+ """Exact block modulation expression, evaluated in FP32."""+ return normalized.float() * (1.0 + scale.float()) + shift.float()+++def gated_residual_eager(hidden_states, branch, gate):+ """Exact gated residual expression used by WorldPlay2AttentionBlock."""+ return hidden_states + branch * gate+++def action_mlp_eager(+ hidden_states,+ action,+ input_weight,+ input_bias,+ output_weight,+ output_bias,+):+ """Action injection: concat, MLP, and residual add."""+ branch = torch.cat([hidden_states, action], dim=-1)+ branch = F.linear(branch, input_weight, input_bias)+ branch = F.silu(branch)+ branch = F.linear(branch, output_weight, output_bias)+ return hidden_states + branch+++def ffn_eager(+ hidden_states,+ input_weight,+ input_bias,+ output_weight,+ output_bias,+):+ hidden_states = F.linear(+ hidden_states, input_weight, input_bias)+ hidden_states = F.gelu(hidden_states, approximate="tanh")+ return F.linear(hidden_states, output_weight, output_bias)+++@lru_cache(maxsize=1)+def _compiled_modulation():+ return torch.compile(+ modulation_eager,+ fullgraph=True,+ dynamic=True,+ mode="max-autotune-no-cudagraphs",+ )+++@lru_cache(maxsize=1)+def _compiled_gated_residual():+ return torch.compile(+ gated_residual_eager,+ fullgraph=True,+ dynamic=True,+ mode="max-autotune-no-cudagraphs",+ )+++@lru_cache(maxsize=1)+def _compiled_action_mlp():+ return torch.compile(+ action_mlp_eager,+ fullgraph=True,+ dynamic=True,+ mode="max-autotune-no-cudagraphs",+ )+++@lru_cache(maxsize=1)+def _compiled_ffn():+ return torch.compile(+ ffn_eager,+ fullgraph=True,+ dynamic=True,+ mode="max-autotune-no-cudagraphs",+ )+++def _run_region(name, eager, compiled_factory, args, enabled):+ if enabled and name not in _DISABLED_REGIONS:+ try:+ output = compiled_factory()(*args)+ _REGION_STATS[name]["compiled_calls"] += 1+ return output, True+ except Exception:+ _REGION_STATS[name]["failures"] += 1+ _DISABLED_REGIONS.add(name)+ logging.exception(+ "Compiled region %s failed; disabling it and using eager",+ name,+ )+ _REGION_STATS[name]["eager_calls"] += 1+ return eager(*args), False+++def modulation(normalized, scale, shift, *, enabled):+ return _run_region(+ "modulation",+ modulation_eager,+ _compiled_modulation,+ (normalized, scale, shift),+ enabled,+ )+++def gated_residual(hidden_states, branch, gate, *, enabled):+ return _run_region(+ "gated_residual",+ gated_residual_eager,+ _compiled_gated_residual,+ (hidden_states, branch, gate),+ enabled,+ )+++def action_mlp(+ hidden_states,+ action,+ input_weight,+ input_bias,+ output_weight,+ output_bias,+ *,+ enabled,+):+ return _run_region(+ "action_mlp",+ action_mlp_eager,+ _compiled_action_mlp,+ (+ hidden_states,+ action,+ input_weight,+ input_bias,+ output_weight,+ output_bias,+ ),+ enabled,+ )+++def ffn(+ hidden_states,+ input_weight,+ input_bias,+ output_weight,+ output_bias,+ *,+ enabled,+):+ return _run_region(+ "ffn",+ ffn_eager,+ _compiled_ffn,+ (+ hidden_states,+ input_weight,+ input_bias,+ output_weight,+ output_bias,+ ),+ enabled,+ )+++def get_compiled_region_stats() -> dict:+ return {+ name: dict(values)+ for name, values in _REGION_STATS.items()+ } | {+ "disabled_after_failure": sorted(_DISABLED_REGIONS),+ }diff --git a/worldplay2/models/model.py b/worldplay2/models/model.pynew file mode 100644index 0000000..76bfe79--- /dev/null+++ b/worldplay2/models/model.py@@ -0,0 +1,1907 @@+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.+import math++import torch+import torch.nn as nn+import torch.nn.functional as F+from einops import rearrange, repeat++from diffusers.configuration_utils import ConfigMixin, register_to_config+from diffusers.models.modeling_utils import ModelMixin++from .attention import (+ flash_attention,+ flash_attention_dense,+ resolve_attention_backend,+ sage_attention,+)+from .kernels import (+ action_mlp as run_compiled_action_mlp,+ ffn as run_compiled_ffn,+ gated_residual as run_compiled_gated_residual,+ layer_norm_modulation as run_fused_layer_norm_modulation,+ qk_norm_rope as run_qk_norm_rope,+)+from .fp8 import fp8_autocast+from ..distributed.ops import get_rank, get_world_size+from ..distributed.ulysses import (+ gather_sequence,+ output_all_to_all,+ packed_qkv_all_to_all,+ shard_sequence,+)++__all__ = ['WorldPlay2Model']+++def sinusoidal_embedding_1d(dim, position):+ # preprocess+ assert dim % 2 == 0+ half = dim // 2+ position = position.type(torch.float64)++ # calculation+ sinusoid = torch.outer(+ position, torch.pow(10000, -torch.arange(half).to(position).div(half)))+ x = torch.cat([torch.cos(sinusoid), torch.sin(sinusoid)], dim=1)+ return x++[email protected]('cuda', enabled=False)+def rope_params(max_seq_len, dim, theta=10000):+ assert dim % 2 == 0+ freqs = torch.outer(+ torch.arange(max_seq_len),+ 1.0 / torch.pow(theta,+ torch.arange(0, dim, 2).to(torch.float64).div(dim)))+ freqs = torch.polar(torch.ones_like(freqs), freqs)+ return freqs++[email protected]('cuda', enabled=False)+def rope_apply(x, grid_sizes, freqs):+ n, c = x.size(2), x.size(3) // 2++ # split freqs+ freqs = freqs.split([c - 2 * (c // 3), c // 3, c // 3], dim=1)++ # loop over samples+ output = []+ for i, (f, h, w) in enumerate(grid_sizes.tolist()):+ seq_len = f * h * w++ # precompute multipliers+ x_i = torch.view_as_complex(x[i, :seq_len].to(torch.float64).reshape(+ seq_len, n, -1, 2))+ freqs_i = torch.cat([+ freqs[0][:f].view(f, 1, 1, -1).expand(f, h, w, -1),+ freqs[1][:h].view(1, h, 1, -1).expand(f, h, w, -1),+ freqs[2][:w].view(1, 1, w, -1).expand(f, h, w, -1)+ ],+ dim=-1).reshape(seq_len, 1, -1)++ # apply rotary embedding+ x_i = torch.view_as_real(x_i * freqs_i).flatten(2)+ x_i = torch.cat([x_i, x[i, seq_len:]])++ # append to collection+ output.append(x_i)+ return torch.stack(output).float()++[email protected]('cuda', enabled=False)+def rope_apply_flat(x, freqs_flat):+ """Apply RoPE given a per-token freqs tensor.++ Used by the memory-compress path where the sequence is a heterogeneous+ concat of context/generate tokens, so a single (F, H, W) grid_sizes+ description is no longer sufficient.++ Args:+ x (Tensor): Shape [B, L, Nh, C_head] (real).+ freqs_flat (Tensor): complex64, shape [L, C_head / 2].++ Returns:+ Tensor: Same shape as x, with RoPE applied (float).+ """+ b, l, n = x.shape[:3]+ x_c = torch.view_as_complex(x.to(torch.float64).reshape(b, l, n, -1, 2))+ freqs_i = freqs_flat.view(1, l, 1, -1)+ x_out = torch.view_as_real(x_c * freqs_i).flatten(2)+ return x_out.float().view(b, l, n, -1)+++def build_freqs_flat(freqs, grid_size):+ """Build per-token RoPE freqs for a single grid (F, H, W).++ Returns a complex tensor of shape [F*H*W, C_head/2].+ """+ f, h, w = int(grid_size[0]), int(grid_size[1]), int(grid_size[2])+ c = freqs.size(1)+ freqs_split = freqs.split([c - 2 * (c // 3), c // 3, c // 3], dim=1)+ freqs_i = torch.cat([+ freqs_split[0][:f].view(f, 1, 1, -1).expand(f, h, w, -1),+ freqs_split[1][:h].view(1, h, 1, -1).expand(f, h, w, -1),+ freqs_split[2][:w].view(1, 1, w, -1).expand(f, h, w, -1)+ ], dim=-1).reshape(f * h * w, -1)+ return freqs_i+++class WorldPlay2RMSNorm(nn.Module):++ def __init__(self, dim, eps=1e-5):+ super().__init__()+ self.dim = dim+ self.eps = eps+ self.weight = nn.Parameter(torch.ones(dim))++ def forward(self, x):+ r"""+ Args:+ x(Tensor): Shape [B, L, C]+ """+ return self._norm(x.float()).type_as(x) * self.weight++ def _norm(self, x):+ return x * torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.eps)+++class WorldPlay2LayerNorm(nn.LayerNorm):++ def __init__(self, dim, eps=1e-6, elementwise_affine=False):+ super().__init__(dim, elementwise_affine=elementwise_affine, eps=eps)++ def forward(self, x):+ r"""+ Args:+ x(Tensor): Shape [B, L, C]+ """+ return super().forward(x.float()).type_as(x)+++class WorldPlay2SelfAttention(nn.Module):++ def __init__(self,+ dim,+ num_heads,+ window_size=(-1, -1),+ qk_norm=True,+ eps=1e-6):+ assert dim % num_heads == 0+ super().__init__()+ self.dim = dim+ self.num_heads = num_heads+ self.head_dim = dim // num_heads+ self.window_size = window_size+ self.qk_norm = qk_norm+ self.eps = eps+ self.attention_backend = 'flash'++ # layers+ self.q = nn.Linear(dim, dim)+ self.k = nn.Linear(dim, dim)+ self.v = nn.Linear(dim, dim)+ self.o = nn.Linear(dim, dim)+ self.norm_q = WorldPlay2RMSNorm(dim, eps=eps) if qk_norm else nn.Identity()+ self.norm_k = WorldPlay2RMSNorm(dim, eps=eps) if qk_norm else nn.Identity()+ self.compile_qk_rope = True+ self.qk_rope_compile_failed = False+ self.qkv = None++ def fuse_qkv_projections(self) -> None:+ """Replace the three self-attention projections with one exact GEMM."""+ if self.qkv is not None:+ return+ if not all(hasattr(self, name) for name in ("q", "k", "v")):+ raise RuntimeError("Q/K/V projections are unavailable for fusion")++ projections = (self.q, self.k, self.v)+ reference = projections[0]+ if any(+ projection.in_features != reference.in_features+ or projection.out_features != reference.out_features+ or (projection.bias is None) != (reference.bias is None)+ for projection in projections[1:]+ ):+ raise RuntimeError("Q/K/V projection shapes are incompatible")++ qkv = nn.Linear(+ reference.in_features,+ 3 * reference.out_features,+ bias=reference.bias is not None,+ device=reference.weight.device,+ dtype=reference.weight.dtype,+ )+ with torch.no_grad():+ qkv.weight.copy_(torch.cat(+ [projection.weight for projection in projections], dim=0))+ if qkv.bias is not None:+ qkv.bias.copy_(torch.cat(+ [projection.bias for projection in projections], dim=0))++ self.qkv = qkv+ # Do not retain duplicate parameters: fusion must not increase FSDP+ # storage or checkpoint memory after installation.+ del self.q+ del self.k+ del self.v++ def forward(self, x, seq_lens, grid_sizes, freqs, freqs_flat=None,+ ar=False, cmp_token_count=0, cmp_chunk_token_count=0,+ tmp_token_count=0,+ kv_cache_entry=None, cache_write=False,+ sink_token_count=0, cache_update=None):+ r"""+ Args:+ x(Tensor): Shape [B, L, num_heads, C / num_heads]+ seq_lens(Tensor): Shape [B]+ grid_sizes(Tensor): Shape [B, 3], the second dimension contains (F, H, W).+ Ignored when ``freqs_flat`` is provided.+ freqs(Tensor): Rope freqs, shape [1024, C / num_heads / 2]+ freqs_flat(Tensor, optional): Per-token complex RoPE tensor of shape+ ``[L, C / num_heads / 2]``. When provided, RoPE is applied via+ ``rope_apply_flat`` (used by the memory-compress path where+ the sequence is a heterogeneous concat of context/generate+ tokens). When ``None``, falls back to the original grid-based+ ``rope_apply``.+ ar(bool): when True, run the chunk-AR (KV-cache) self-attention.+ Cache behavior is selected by ``cache_write``:+ * prefill (cache_write=True): ``x`` = ctx tokens only+ (``[cmp | tmp]``). A chunk-causal mask is applied (cmp+ chunk-causal to cmp; tmp -> all cmp+tmp; no pred rows).+ The post-RoPE ctx K/V are OVERWRITE-stored into+ ``kv_cache_entry['k_vision'/'v_vision']``. Returns the+ ctx output (threaded layer-to-layer by the caller).+ * cache-hit (cache_write=False): ``x`` = pred tokens only.+ K/V = ``[cached_ctx | pred]`` (no mask -- pred sees all),+ via the selected backend. ``cached_*`` may be None on the+ first chunk (no compressor ctx) -> pred-only attention.+ kv_cache_entry(dict, optional): per-layer ``{'k_vision','v_vision'}``+ slot (see ``WorldPlay2Model.init_kv_cache``).+ cache_write(bool): write/update context cache vs read it for denoise.+ """+ b, s, n, d = *x.shape[:2], self.num_heads, self.head_dim++ if self.qkv is not None:+ q_projection, k_projection, v_projection = self.qkv(x).chunk(+ 3, dim=-1)+ else:+ q_projection = self.q(x)+ k_projection = self.k(x)+ v_projection = self.v(x)+ v = v_projection.view(b, s, n, d)++ if freqs_flat is not None:+ if self.qk_norm:+ q, k, compiled = run_qk_norm_rope(+ q_projection,+ k_projection,+ self.norm_q.weight,+ self.norm_k.weight,+ freqs_flat,+ eps=self.eps,+ num_heads=self.num_heads,+ head_dim=self.head_dim,+ compile_enabled=(+ self.compile_qk_rope+ and not self.qk_rope_compile_failed+ ),+ )+ if self.compile_qk_rope and not compiled:+ self.qk_rope_compile_failed = True+ else:+ q = q_projection.view(b, s, n, d)+ k = k_projection.view(b, s, n, d)+ q = rope_apply_flat(q, freqs_flat)+ k = rope_apply_flat(k, freqs_flat)+ else:+ q = self.norm_q(q_projection).view(b, s, n, d)+ k = self.norm_k(k_projection).view(b, s, n, d)+ q = rope_apply(q, grid_sizes, freqs)+ k = rope_apply(k, grid_sizes, freqs)++ # RoPE is evaluated in FP32 for stability, but attention and packed+ # QKV communication use the value/projection dtype (BF16 in inference).+ # This also prevents torch.stack in packed QKV A2A from promoting V+ # and tripling its communication precision.+ q = q.to(v.dtype)+ k = k.to(v.dtype)++ use_sp = getattr(self, "use_sp", False)+ if use_sp:+ if not torch.all(seq_lens == seq_lens[0]):+ raise ValueError(+ "sequence parallel requires equal sequence lengths "+ "within the batch")+ global_sequence = int(seq_lens[0].item())+ q, k, v = packed_qkv_all_to_all(+ q, k, v, global_sequence=global_sequence)+ else:+ global_sequence = q.shape[1]++ def finish_attention(out):+ if use_sp:+ out = output_all_to_all(+ out,+ global_sequence=global_sequence,+ num_heads=self.num_heads,+ )+ return self.o(out.flatten(2))++ if not ar:+ attention_fn = (+ sage_attention+ if self.attention_backend == 'sage' else flash_attention+ )+ x = attention_fn(+ q=q, k=k, v=v, k_lens=seq_lens,+ window_size=self.window_size)+ return finish_attention(x)++ assert kv_cache_entry is not None, (+ "self_attn(ar=True) requires kv_cache_entry "+ "(use WorldPlay2Model.init_kv_cache()).")++ cache_initialized = kv_cache_entry.get('k_vision') is not None+ if not cache_write:+ # Case 1: regular AR denoise. Read the persistent context cache;+ # an empty cache on the very first generated chunk is valid.+ out = self._attend_with_ar_cache(q, k, v, kv_cache_entry)+ elif not cache_initialized:+ # Case 2: first history prefill. Bootstrap the cache as+ # [sink | cmp | tmp], including the sink-specific attention mask.+ out = self._initialize_ar_cache(+ q=q,+ k=k,+ v=v,+ kv_cache_entry=kv_cache_entry,+ sink_token_count=sink_token_count,+ cmp_token_count=cmp_token_count,+ cmp_chunk_token_count=cmp_chunk_token_count,+ tmp_token_count=tmp_token_count,+ )+ else:+ # Case 3: later history prefill. Keep [sink | old_cmp], discard+ # the previous tmp, then append [new_cmp | new_tmp].+ if cache_update is None:+ raise ValueError("AR cache update metadata is required")+ out = self._update_ar_cache(+ q, k, v, kv_cache_entry, cache_update,+ cmp_chunk_token_count)+ return finish_attention(out)++ def _initialize_ar_cache(+ self,+ q,+ k,+ v,+ kv_cache_entry,+ sink_token_count,+ cmp_token_count,+ cmp_chunk_token_count,+ tmp_token_count,+ ):+ """Bootstrap ``[sink | cmp | tmp]`` on the first history prefill."""+ sequence_length = q.shape[1]+ expected_length = (+ sink_token_count + cmp_token_count + tmp_token_count+ )+ if sequence_length != expected_length:+ raise ValueError(+ f"initial AR cache expects {expected_length} tokens, "+ f"got {sequence_length}"+ )++ q_attn = q.transpose(1, 2)+ k_attn = k.transpose(1, 2)+ v_attn = v.transpose(1, 2)+ if sequence_length == 0:+ out = torch.zeros_like(q_attn)+ elif cmp_chunk_token_count <= 0:+ out = F.scaled_dot_product_attention(+ q_attn, k_attn, v_attn, is_causal=False)+ else:+ mask = torch.zeros(+ sequence_length,+ sequence_length,+ dtype=torch.bool,+ device=q.device,+ )+ if sink_token_count:+ mask[:sink_token_count, :sink_token_count] = True+ for chunk_start in range(+ 0, cmp_token_count, cmp_chunk_token_count+ ):+ query_start = sink_token_count + chunk_start+ query_end = min(+ query_start + cmp_chunk_token_count,+ sink_token_count + cmp_token_count,+ )+ mask[query_start:query_end, :query_end] = True+ tmp_start = sink_token_count + cmp_token_count+ if tmp_token_count:+ mask[tmp_start:expected_length, :expected_length] = True+ out = F.scaled_dot_product_attention(+ q_attn,+ k_attn,+ v_attn,+ attn_mask=mask.unsqueeze(0).unsqueeze(0),+ is_causal=False,+ )++ kv_cache_entry['k_vision'] = k.detach()+ kv_cache_entry['v_vision'] = v.detach()+ return out.transpose(1, 2)++ def _update_ar_cache(self, q, k, v, kv_cache_entry, cache_update,+ cmp_chunk_token_count):+ r"""Replace old tmp and append ``[new_cmp | new_tmp]``.++ ``q/k/v`` contain only ``[new_cmp | new_tmp]``. The persistent prefix+ ``[sink | old_cmp]`` is sliced directly from ``k_vision/v_vision``;+ the previous tmp suffix is discarded. After attention, the cache is+ replaced by ``[cached_prefix | new_cmp | new_tmp]``.+ """+ n_new_cmp = int(cache_update['n_new_cmp'])+ n_tmp = int(cache_update['n_tmp'])+ cached_prefix_tokens = int(cache_update['cached_prefix_tokens'])+ Sq = q.shape[1]+ assert Sq == n_new_cmp + n_tmp, (+ f"ctx prefill expects q = [new_cmp | tmp] "+ f"({n_new_cmp}+{n_tmp}), got {Sq}")++ cached_k = kv_cache_entry.get('k_vision', None)+ cached_v = kv_cache_entry.get('v_vision', None)+ assert cached_k is not None and cached_v is not None, (+ "Incremental prefill requires an existing k_vision/v_vision.")+ assert cached_prefix_tokens <= cached_k.shape[1], (+ f"cached_prefix_tokens ({cached_prefix_tokens}) exceeds cached "+ f"length ({cached_k.shape[1]}).")+ pre_k = cached_k[:, :cached_prefix_tokens].to(k.device).to(k.dtype)+ pre_v = cached_v[:, :cached_prefix_tokens].to(v.device).to(v.dtype)+ L_pre = cached_prefix_tokens+ k_full = torch.cat([pre_k, k], dim=1)+ v_full = torch.cat([pre_v, v], dim=1)+ L_tot = k_full.shape[1] # = L_pre + n_new_cmp + n_tmp++ # Column index landmarks within k_full = [sink | old_cmp | new_cmp | tmp]+ cmp_new_c0 = L_pre # first new_cmp column+ cmp_new_c1 = L_pre + n_new_cmp # end of new_cmp / start of tmp+ tmp_c1 = L_tot # end of tmp++ # Build mask over QUERY rows (Sq) x all columns (L_tot).+ attn_mask = torch.zeros(Sq, L_tot, dtype=torch.bool, device=q.device)+ if n_new_cmp > 0:+ # new_cmp is chunk-causal *within itself* and sees all prefix+ # (sink + old_cmp), but NOT tmp. Split new_cmp into cmp-chunks.+ ccc = cmp_chunk_token_count if cmp_chunk_token_count > 0 else n_new_cmp+ for rs in range(0, n_new_cmp, ccc):+ re = min(rs + ccc, n_new_cmp)+ # columns: everything up to and including this cmp sub-chunk+ ke = cmp_new_c0 + re+ ke = min(ke, cmp_new_c1)+ attn_mask[rs:re, :ke] = True+ if n_tmp > 0:+ # tmp rows (after new_cmp in q) see EVERYTHING (sink+cmp+tmp).+ attn_mask[n_new_cmp:Sq, :tmp_c1] = True+ attn_mask = attn_mask.unsqueeze(0).unsqueeze(0)++ out = F.scaled_dot_product_attention(+ q.transpose(1, 2), k_full.transpose(1, 2), v_full.transpose(1, 2),+ attn_mask=attn_mask, is_causal=False)+ out = out.transpose(1, 2) # [B, Sq, H, D]++ kv_cache_entry['k_vision'] = torch.cat(+ [pre_k, k], dim=1).detach()+ kv_cache_entry['v_vision'] = torch.cat(+ [pre_v, v], dim=1).detach()+ return out++ def _attend_with_ar_cache(self, q, k, v, kv_cache_entry):+ """Attend over ``[cached ctx | pred]`` with the selected backend."""+ cached_k = kv_cache_entry.get('k_vision')+ cached_v = kv_cache_entry.get('v_vision')+ if cached_k is not None:+ k = torch.cat([cached_k.to(k.device, k.dtype), k], dim=1)+ v = torch.cat([cached_v.to(v.device, v.dtype), v], dim=1)+ if self.attention_backend == 'sage':+ return sage_attention(q, k, v, causal=False)+ return flash_attention_dense(q, k, v, causal=False)++++class WorldPlay2CrossAttention(WorldPlay2SelfAttention):++ def __init__(self, *args, **kwargs):+ super().__init__(*args, **kwargs)+ self.conditioning_cache_enabled = True+ # Keyed by the context tensor itself: torch.Tensor hashes and compares+ # by identity, so the key doubles as the strong reference that keeps a+ # stale entry from ever matching a different tensor.+ self._context_kv_cache: dict[torch.Tensor, tuple] = {}++ def clear_conditioning_cache(self) -> None:+ self._context_kv_cache.clear()++ def _get_context_kv(self, context):+ if not self.conditioning_cache_enabled:+ return self.norm_k(self.k(context)), self.v(context)++ entry = self._context_kv_cache.get(context)+ if entry is not None:+ return entry++ key_tensor = self.norm_k(self.k(context)).detach()+ value_tensor = self.v(context).detach()+ self._context_kv_cache[context] = (key_tensor, value_tensor)+ return key_tensor, value_tensor++ def forward(self, x, context, context_lens):+ r"""+ Args:+ x(Tensor): Shape [B, L1, C]+ context(Tensor): Shape [B, L2, C]+ context_lens(Tensor): Shape [B]+ """+ b, n, d = x.size(0), self.num_heads, self.head_dim++ # compute query, key, value+ q = self.norm_q(self.q(x)).view(b, -1, n, d)+ k, v = self._get_context_kv(context)+ k = k.view(b, -1, n, d)+ v = v.view(b, -1, n, d)++ # compute attention+ x = flash_attention(q, k, v, k_lens=context_lens)++ # output+ x = x.flatten(2)+ x = self.o(x)+ return x+++class WorldPlay2AttentionBlock(nn.Module):++ def __init__(self,+ dim,+ ffn_dim,+ num_heads,+ window_size=(-1, -1),+ qk_norm=True,+ cross_attn_norm=False,+ eps=1e-6):+ super().__init__()+ self.dim = dim+ self.ffn_dim = ffn_dim+ self.num_heads = num_heads+ self.window_size = window_size+ self.qk_norm = qk_norm+ self.cross_attn_norm = cross_attn_norm+ self.eps = eps++ # layers+ self.norm1 = WorldPlay2LayerNorm(dim, eps)+ self.self_attn = WorldPlay2SelfAttention(dim, num_heads, window_size, qk_norm,+ eps)+ self.norm3 = WorldPlay2LayerNorm(+ dim, eps,+ elementwise_affine=True) if cross_attn_norm else nn.Identity()+ self.cross_attn = WorldPlay2CrossAttention(dim, num_heads, (-1, -1), qk_norm,+ eps)+ self.norm2 = WorldPlay2LayerNorm(dim, eps)+ self.ffn = nn.Sequential(+ nn.Linear(dim, ffn_dim), nn.GELU(approximate='tanh'),+ nn.Linear(ffn_dim, dim))++ # modulation+ self.modulation = nn.Parameter(torch.randn(1, 6, dim) / dim**0.5)+ self.compile_block_glue = True+ self.compile_ffn = True++ def forward(+ self,+ x,+ e,+ seq_lens,+ grid_sizes,+ freqs,+ context,+ context_lens,+ vec_action,+ freqs_flat=None,+ ar=False,+ cmp_token_count=0,+ cmp_chunk_token_count=0,+ tmp_token_count=0,+ kv_cache_entry=None,+ cache_write=False,+ sink_token_count=0,+ cache_update=None,+ ):+ r"""+ Args:+ x(Tensor): Shape [B, L, C]+ e(Tensor): Shape [B, L1, 6, C]+ seq_lens(Tensor): Shape [B], length of each sequence in batch+ grid_sizes(Tensor): Shape [B, 3], the second dimension contains (F, H, W)+ freqs(Tensor): Rope freqs, shape [1024, C / num_heads / 2]+ freqs_flat(Tensor, optional): Per-token RoPE freqs for the+ memory-compress path; forwarded to ``self_attn``.+ ar(bool): when True, run the chunk-AR (KV-cache) self-attention.+ kv_cache_entry/cache_write: forwarded to+ ``self_attn`` for the chunk-AR prefill / cache-hit paths.+ cache_update(dict, optional): metadata for replacing the old tmp+ suffix and appending the new cmp/tmp tokens.+ """+ assert e.dtype == torch.float32+ with torch.amp.autocast('cuda', dtype=torch.float32):+ e = (self.modulation.unsqueeze(0) + e).chunk(6, dim=2)+ assert e[0].dtype == torch.float32++ # self-attention (BI when ar=False; chunk-AR KV-cache when ar=True)+ attn_in, _ = run_fused_layer_norm_modulation(+ x,+ e[1].squeeze(2),+ e[0].squeeze(2),+ eps=self.eps,+ enabled=self.compile_block_glue,+ )+ y = self.self_attn(+ attn_in, seq_lens, grid_sizes, freqs, freqs_flat=freqs_flat,+ ar=ar,+ cmp_token_count=cmp_token_count,+ cmp_chunk_token_count=cmp_chunk_token_count,+ tmp_token_count=tmp_token_count,+ kv_cache_entry=kv_cache_entry,+ cache_write=cache_write,+ sink_token_count=sink_token_count,+ cache_update=cache_update)+ x, _ = run_compiled_gated_residual(+ x,+ y,+ e[2].squeeze(2),+ enabled=self.compile_block_glue,+ )++ # cross-attention & ffn function+ def cross_attn_ffn(x, context, context_lens, e):+ cross_input = self.norm3(x)+ cross_output = self.cross_attn(+ cross_input, context, context_lens)+ x = x + cross_output++ # mlp inject action instead of attention+ if vec_action is not None:+ x, _ = run_compiled_action_mlp(+ x,+ vec_action,+ self.img_action_mlp[0].weight,+ self.img_action_mlp[0].bias,+ self.img_action_mlp[2].weight,+ self.img_action_mlp[2].bias,+ enabled=self.compile_block_glue,+ )++ ffn_in, _ = run_fused_layer_norm_modulation(+ x,+ e[4].squeeze(2),+ e[3].squeeze(2),+ eps=self.eps,+ enabled=self.compile_block_glue,+ )+ if getattr(self.ffn, "uses_fp8", False):+ y = self.ffn(ffn_in.to(x.dtype))+ else:+ y, _ = run_compiled_ffn(+ ffn_in,+ self.ffn[0].weight,+ self.ffn[0].bias,+ self.ffn[2].weight,+ self.ffn[2].bias,+ enabled=self.compile_ffn,+ )+ x, _ = run_compiled_gated_residual(+ x,+ y,+ e[5].squeeze(2),+ enabled=self.compile_block_glue,+ )+ return x++ x = cross_attn_ffn(x, context, context_lens, e)+ return x+++class Head(nn.Module):++ def __init__(self, dim, out_dim, patch_size, eps=1e-6):+ super().__init__()+ self.dim = dim+ self.out_dim = out_dim+ self.patch_size = patch_size+ self.eps = eps++ # layers+ out_dim = math.prod(patch_size) * out_dim+ self.norm = WorldPlay2LayerNorm(dim, eps)+ self.head = nn.Linear(dim, out_dim)++ # modulation+ self.modulation = nn.Parameter(torch.randn(1, 2, dim) / dim**0.5)+ self._active_block_index = None+ self._last_endpoint_output = None++ def make_fixed_pdd(self, num_blocks: int = 2) -> None:+ num_blocks = int(num_blocks)+ if num_blocks <= 0:+ raise ValueError("fixed PDD block count must be positive")+ if hasattr(self, "block_weight"):+ if self._num_fixed_pdd_blocks != num_blocks:+ raise ValueError("fixed PDD block count mismatch")+ return+ self._num_fixed_pdd_blocks = num_blocks+ output_dim = self.head.out_features+ self.block_weight = nn.Parameter(torch.empty(+ num_blocks,+ 2 * output_dim,+ self.dim,+ device=self.head.weight.device,+ dtype=self.head.weight.dtype,+ ))+ self.block_bias = nn.Parameter(torch.empty(+ num_blocks,+ 2 * output_dim,+ device=self.head.bias.device,+ dtype=self.head.bias.dtype,+ ))+ del self.head++ def set_active_block(self, block_index: int) -> None:+ if not hasattr(self, "block_weight"):+ raise RuntimeError("set_active_block requires a fixed PDD head")+ block_index = int(block_index)+ if not 0 <= block_index < self._num_fixed_pdd_blocks:+ raise IndexError(+ f"fixed PDD block index {block_index} is out of range "+ f"[0, {self._num_fixed_pdd_blocks})"+ )+ self._active_block_index = block_index++ def pop_endpoint_output(self):+ output = self._last_endpoint_output+ self._last_endpoint_output = None+ return output++ def forward(self, x, e):+ r"""+ Args:+ x(Tensor): Shape [B, L1, C]+ e(Tensor): Shape [B, L1, C]+ """+ assert e.dtype == torch.float32+ with torch.amp.autocast('cuda', dtype=torch.float32):+ shift, scale = (+ self.modulation.unsqueeze(0) + e.unsqueeze(2)+ ).chunk(2, dim=2)+ hidden = (+ self.norm(x) * (1 + scale.squeeze(2)) + shift.squeeze(2)+ )+ if hasattr(self, "block_weight"):+ if self._active_block_index is None:+ raise RuntimeError(+ "fixed PDD inference block was not selected"+ )+ projected = F.linear(+ hidden,+ self.block_weight[self._active_block_index],+ self.block_bias[self._active_block_index],+ )+ displacement, endpoint = projected.chunk(2, dim=-1)+ self._last_endpoint_output = endpoint+ return displacement+ self._last_endpoint_output = None+ return self.head(hidden)+++class WorldPlay2Model(ModelMixin, ConfigMixin):+ r"""+ WorldPlay2 diffusion backbone based on Wan2.2.+ """++ ignore_for_config = [+ 'patch_size', 'cross_attn_norm', 'qk_norm', 'text_dim', 'window_size'+ ]+ _no_split_modules = ['WorldPlay2AttentionBlock']++ @register_to_config+ def __init__(self,+ model_type='t2v',+ patch_size=(1, 2, 2),+ text_len=512,+ in_dim=16,+ dim=2048,+ ffn_dim=8192,+ freq_dim=256,+ text_dim=4096,+ out_dim=16,+ num_heads=16,+ num_layers=32,+ window_size=(-1, -1),+ qk_norm=True,+ cross_attn_norm=True,+ eps=1e-6):+ r"""+ Initialize the diffusion model backbone.++ Args:+ model_type (`str`, *optional*, defaults to 't2v'):+ Model variant - 't2v' (text-to-video) or 'i2v' (image-to-video)+ patch_size (`tuple`, *optional*, defaults to (1, 2, 2)):+ 3D patch dimensions for video embedding (t_patch, h_patch, w_patch)+ text_len (`int`, *optional*, defaults to 512):+ Fixed length for text embeddings+ in_dim (`int`, *optional*, defaults to 16):+ Input video channels (C_in)+ dim (`int`, *optional*, defaults to 2048):+ Hidden dimension of the transformer+ ffn_dim (`int`, *optional*, defaults to 8192):+ Intermediate dimension in feed-forward network+ freq_dim (`int`, *optional*, defaults to 256):+ Dimension for sinusoidal time embeddings+ text_dim (`int`, *optional*, defaults to 4096):+ Input dimension for text embeddings+ out_dim (`int`, *optional*, defaults to 16):+ Output video channels (C_out)+ num_heads (`int`, *optional*, defaults to 16):+ Number of attention heads+ num_layers (`int`, *optional*, defaults to 32):+ Number of transformer blocks+ window_size (`tuple`, *optional*, defaults to (-1, -1)):+ Window size for local attention (-1 indicates global attention)+ qk_norm (`bool`, *optional*, defaults to True):+ Enable query/key normalization+ cross_attn_norm (`bool`, *optional*, defaults to False):+ Enable cross-attention normalization+ eps (`float`, *optional*, defaults to 1e-6):+ Epsilon value for normalization layers+ """++ super().__init__()++ assert model_type in ['t2v', 'i2v', 'ti2v', 's2v']+ self.model_type = model_type++ self.patch_size = patch_size+ self.text_len = text_len+ self.in_dim = in_dim+ self.dim = dim+ self.ffn_dim = ffn_dim+ self.freq_dim = freq_dim+ self.text_dim = text_dim+ self.out_dim = out_dim+ self.num_heads = num_heads+ self.num_layers = num_layers+ self.window_size = window_size+ self.qk_norm = qk_norm+ self.cross_attn_norm = cross_attn_norm+ self.eps = eps++ # embeddings+ self.patch_embedding = nn.Conv3d(+ in_dim, dim, kernel_size=patch_size, stride=patch_size)+ self.text_embedding = nn.Sequential(+ nn.Linear(text_dim, dim), nn.GELU(approximate='tanh'),+ nn.Linear(dim, dim))++ self.time_embedding = nn.Sequential(+ nn.Linear(freq_dim, dim), nn.SiLU(), nn.Linear(dim, dim))+ self.time_projection = nn.Sequential(nn.SiLU(), nn.Linear(dim, dim * 6))++ # blocks+ self.blocks = nn.ModuleList([+ WorldPlay2AttentionBlock(dim, ffn_dim, num_heads, window_size, qk_norm,+ cross_attn_norm, eps) for _ in range(num_layers)+ ])++ # head+ self.head = Head(dim, out_dim, patch_size, eps)++ # buffers (don't use register_buffer otherwise dtype will be changed in to())+ assert (dim % num_heads) == 0 and (dim // num_heads) % 2 == 0+ d = dim // num_heads+ self.freqs = torch.cat([+ rope_params(1024, d - 4 * (d // 6)),+ rope_params(1024, 2 * (d // 6)),+ rope_params(1024, 2 * (d // 6))+ ],+ dim=1)++ # initialize weights+ self.init_weights()++ self.use_sp = False+ self.attention_backend = 'flash'+ self.conditioning_cache_enabled = True+ self._text_projection_cache: dict[tuple, torch.Tensor] = {}+ self.self_qkv_fused = False+ self.fp8_enabled = False+ self.fp8_ffn_enabled = False+ self.fp8_qkv_enabled = False+ self.fp8_recipe = None+ self.fp8_block_range = None+ self.fp8_qkv_block_range = None++ def add_embedding_action_parameters(self):+ """Continuous (first 2 dims) + per-dim discrete embeddings (last 4).++ Action layout (last dim = 6):+ [:, 0:2] continuous → ``action_in_cont`` MLP → (N, dim)+ [:, 2] discrete in {-1, 0, 1} → Embedding(3, dim) (idx +1)+ [:, 3] discrete in {-1, 0, 1} → Embedding(3, dim) (idx +1)+ [:, 4] discrete in {0, 1} → Embedding(2, dim)+ [:, 5] discrete in {0, 1} → Embedding(2, dim)+ Final ``vec_action = cont + Σ discrete`` so the output shape and+ downstream usage stay identical to the MLP path.+ """+ # continuous branch: (N, 2) -> (N, dim)+ self.action_in_cont = nn.Sequential(+ nn.Linear(2, 256),+ nn.SiLU(),+ nn.Linear(256, 1024),+ nn.SiLU(),+ nn.Linear(1024, self.dim),+ )+ # one independent embedding per discrete dim+ self.action_in_disc = nn.ModuleList([+ nn.Embedding(3, self.dim), # dim2: {-1,0,1} (+1 offset → {0,1,2})+ nn.Embedding(3, self.dim), # dim3: {-1,0,1} (+1 offset → {0,1,2})+ nn.Embedding(2, self.dim), # dim4: {0,1}+ nn.Embedding(2, self.dim), # dim5: {0,1}+ ])+ # Match the per-dim contribution of the continuous branch so all 6+ # action dims start training with comparable signal strength.+ #+ # The continuous branch's last layer (Linear(1024, dim)) under Xavier+ # has weight std ≈ sqrt(2/(fan_in+fan_out)); its output std ≈ that+ # value, spread over 2 input dims, so each continuous dim contributes+ # std ≈ cont_std / sqrt(2). Initialising the discrete embeddings to+ # the same per-dim std keeps the 6 dims roughly equal-strength.+ #+ # Doing this with explicit Xavier on the cont MLP (rather than relying+ # on Linear's default Kaiming) also keeps it consistent with how+ # init_weights() initialises every other Linear in this model.+ for m in self.action_in_cont.modules():+ if isinstance(m, nn.Linear):+ nn.init.xavier_uniform_(m.weight)+ if m.bias is not None:+ nn.init.zeros_(m.bias)+ cont_last = self.action_in_cont[-1] # Linear(1024, dim)+ cont_std = math.sqrt(+ 2.0 / (cont_last.in_features + cont_last.out_features))+ per_dim_std = cont_std / math.sqrt(2)+ for emb in self.action_in_disc:+ nn.init.normal_(emb.weight, std=per_dim_std)++ # img_action_mlp identical to the MLP path (zero-init last layer)+ for block in self.blocks:+ block.img_action_mlp = nn.Sequential(+ nn.Linear(block.dim + block.dim, block.dim),+ nn.SiLU(),+ nn.Linear(block.dim, block.dim),+ )+ nn.init.zeros_(block.img_action_mlp[-1].weight)+ if block.img_action_mlp[-1].bias is not None:+ nn.init.zeros_(block.img_action_mlp[-1].bias)++ def _encode_action(self, action: torch.Tensor) -> torch.Tensor:+ """Encode ``[pitch,yaw,forward,lateral,perspective,space]``."""+ a = action.reshape(-1, 6)+ cont = self.action_in_cont(a[:, :2])+ # discrete indices; +1 offset for the {-1,0,1} dims, clamp as a+ # cheap guard against stray out-of-range values.+ idx2 = (a[:, 2].long() + 1).clamp_(0, 2)+ idx3 = (a[:, 3].long() + 1).clamp_(0, 2)+ idx4 = a[:, 4].long().clamp_(0, 1)+ idx5 = a[:, 5].long().clamp_(0, 1)+ disc = (self.action_in_disc[0](idx2)+ + self.action_in_disc[1](idx3)+ + self.action_in_disc[2](idx4)+ + self.action_in_disc[3](idx5))+ return cont + disc++ def add_memory_compress_model(self):+ """Attach the hierarchical HR -> compressed-memory feature extractor+ used by the stage-2 / stage-3 chunked inference recipe.++ Wan2.2 5B contract (lr is 1/2 in T and 1/4 in H/W):+ - hr_latent : [B, 16, T_hr, H_hr, W_hr]+ - lr_latent : [B, 16, T_hr/2, H_hr/4, W_hr/4]+ - y : 20ch (4 mask + 16 first-frame image latent)+ - compressor input : 36 = 16 + 20 channels at hr resolution.+ - compressor output : spatial_down=[1,1,1,0,0,0] + temporal_down=[1,0,...]+ reduces hr ``[T_hr, H_hr, W_hr]`` -> ``[T_hr/2, H_hr/8, W_hr/8]``.+ ``patch_embedding(lr)`` with patch_size=(1,2,2) also produces+ ``[T_hr/2, H_hr/8, W_hr/8]``, so ``lr_tokens + hr_tokens`` lines up.++ Only the causal-in-time compressor is supported.+ """+ from .causal_compressor import CausalMemCompressModel+ self.memory_compress = CausalMemCompressModel(+ output_dim=self.dim,+ input_dim=self.in_dim,+ spatial_down=[1, 1, 1, 0, 0, 0],+ temporal_down=[1, 0, 0, 0, 0, 0],+ )++ def set_sequence_parallel(self, enabled: bool) -> None:+ """Select SP/non-SP paths without runtime monkey-patching."""+ self.use_sp = bool(enabled)+ for block in self.blocks:+ block.self_attn.use_sp = self.use_sp++ def set_compiled_qk_rope(self, enabled: bool) -> None:+ for block in self.blocks:+ block.self_attn.compile_qk_rope = bool(enabled)+ block.self_attn.qk_rope_compile_failed = False++ def set_self_attention_backend(self, backend: str) -> None:+ """Select visual denoising attention; preserve masked prefill and text."""+ backend = resolve_attention_backend(backend)+ if backend == 'sage' and any(+ tuple(block.self_attn.window_size) != (-1, -1)+ for block in self.blocks+ ):+ raise NotImplementedError(+ "SageAttention requires window_size=(-1, -1); "+ "use attention_backend='flash' for sliding-window attention."+ )+ self.attention_backend = backend+ for block in self.blocks:+ block.self_attn.attention_backend = backend++ def set_compiled_regions(self, *, block_glue: bool, ffn: bool) -> None:+ for block in self.blocks:+ block.compile_block_glue = bool(block_glue)+ block.compile_ffn = bool(ffn)++ def fuse_self_attention_qkv(self) -> None:+ """Fuse Q/K/V for every visual self-attention block."""+ if self.self_qkv_fused:+ return+ for block in self.blocks:+ block.self_attn.fuse_qkv_projections()+ self.self_qkv_fused = True++ def set_conditioning_cache(self, enabled: bool) -> None:+ self.conditioning_cache_enabled = bool(enabled)+ for block in self.blocks:+ block.cross_attn.conditioning_cache_enabled = bool(enabled)+ if not enabled:+ self.clear_conditioning_cache()++ def clear_conditioning_cache(self) -> None:+ self._text_projection_cache.clear()+ for block in self.blocks:+ block.cross_attn.clear_conditioning_cache()++ def _project_text_context(self, context):+ # Keyed by the source tensors themselves: torch.Tensor hashes and+ # compares by identity, so the key both distinguishes cond/uncond and+ # keeps a strong reference to the tensors it was derived from. Model+ # movement/offload explicitly clears this cache.+ key = tuple(context)+ if self.conditioning_cache_enabled:+ cached = self._text_projection_cache.get(key)+ if cached is not None:+ return cached++ padded = torch.stack([+ torch.cat([+ tensor,+ tensor.new_zeros(+ self.text_len - tensor.size(0), tensor.size(1)),+ ])+ for tensor in key+ ])+ projected = self.text_embedding(padded).detach()+ if self.conditioning_cache_enabled:+ self._text_projection_cache[key] = projected+ return projected++ def _forward_compress(+ self,+ hr_latents_input,+ lr_latents_input,+ lr_action,+ memory_t,+ hr_temporal_latents_input=None,+ hr_temporal_action=None,+ sink_action=None,+ ):+ """Build ``[optional sink | compressed history | temporal anchor]``."""+ if hr_temporal_latents_input is None or hr_temporal_action is None:+ raise ValueError("Stage-Three compressor requires a temporal anchor")++ bs = hr_latents_input.shape[0]+ hr_pf = hr_latents_input.shape[2] // self.patch_size[0]+ device = hr_latents_input.device+ if self.freqs.device != device:+ self.freqs = self.freqs.to(device)++ memory_e = self.time_embedding(+ sinusoidal_embedding_1d(+ self.freq_dim,+ memory_t.reshape(-1).float(),+ ).float()+ )++ # Compressed LR+HR history at the LR temporal cadence.+ _, _, lr_f, lr_h, lr_w = lr_latents_input.shape+ cmp_pf = lr_f // self.patch_size[0]+ cmp_ph = lr_h // self.patch_size[1]+ cmp_pw = lr_w // self.patch_size[2]+ cmp_spatial = cmp_ph * cmp_pw+ cmp_tokens = (+ self.patch_embedding(lr_latents_input).flatten(2).transpose(1, 2)+ + self.memory_compress(hr_latents_input)+ )+ cmp_freqs = build_freqs_flat(self.freqs,(hr_pf, cmp_ph, cmp_pw)).view(hr_pf, cmp_spatial, -1)+ cmp_freqs = cmp_freqs[::2][:cmp_pf].reshape(cmp_pf * cmp_spatial, -1)+ cmp_e = repeat(+ memory_e,+ "1 D -> B (T HW) D",+ B=bs,+ T=cmp_pf,+ HW=cmp_spatial,+ )+ cmp_e0 = self.time_projection(cmp_e).unflatten(2, (6, self.dim))+ cmp_action = repeat(+ self._encode_action(lr_action),+ "(B T) D -> B (T HW) D",+ B=bs,+ HW=cmp_spatial,+ )++ # Temporal anchor occupies the final HR temporal positions.+ _, _, tmp_f, tmp_h, tmp_w = hr_temporal_latents_input.shape+ tmp_pf = tmp_f // self.patch_size[0]+ tmp_ph = tmp_h // self.patch_size[1]+ tmp_pw = tmp_w // self.patch_size[2]+ tmp_spatial = tmp_ph * tmp_pw++ tmp_tokens = self.patch_embedding(hr_temporal_latents_input)+ tmp_tokens = tmp_tokens.flatten(2).transpose(1, 2)+ tmp_freqs = build_freqs_flat(self.freqs,(hr_pf, tmp_ph, tmp_pw)).view(hr_pf, tmp_spatial, -1)+ tmp_freqs = tmp_freqs[-tmp_pf:].reshape(tmp_pf * tmp_spatial, -1)+ tmp_e = repeat(+ memory_e,+ "1 D -> B (T HW) D",+ B=bs,+ T=tmp_pf,+ HW=tmp_spatial,+ )+ tmp_e0 = self.time_projection(tmp_e).unflatten(2, (6, self.dim))+ tmp_action = repeat(+ self._encode_action(hr_temporal_action),+ "(B T) D -> B (T HW) D",+ B=bs,+ HW=tmp_spatial,+ )++ sink_token_count = 0+ if sink_action is not None and sink_action.shape[0] > 0:+ sink_size = sink_action.shape[0]+ sink_latents = hr_latents_input[:, :, :sink_size]+ _, _, sink_f, sink_h, sink_w = sink_latents.shape+ sink_pf = sink_f // self.patch_size[0]+ sink_ph = sink_h // self.patch_size[1]+ sink_pw = sink_w // self.patch_size[2]+ sink_spatial = sink_ph * sink_pw++ sink_tokens = self.patch_embedding(sink_latents)+ sink_tokens = sink_tokens.flatten(2).transpose(1, 2)+ sink_freqs = build_freqs_flat(self.freqs,(hr_pf, sink_ph, sink_pw)).view(hr_pf, sink_spatial, -1)+ sink_freqs = sink_freqs[:sink_pf].reshape(sink_pf * sink_spatial, -1)+ sink_e = repeat(+ memory_e,+ "1 D -> B (T HW) D",+ B=bs,+ T=sink_pf,+ HW=sink_spatial,+ )+ sink_e0 = self.time_projection(sink_e).unflatten(2, (6, self.dim))+ sink_action_embedding = repeat(+ self._encode_action(sink_action),+ "(B T) D -> B (T HW) D",+ B=bs,+ HW=sink_spatial,+ )+ sink_token_count = sink_tokens.shape[1]++ tokens = torch.cat([sink_tokens, cmp_tokens, tmp_tokens], dim=1)+ freqs_flat = torch.cat([sink_freqs, cmp_freqs, tmp_freqs], dim=0)+ e0 = torch.cat([sink_e0, cmp_e0, tmp_e0], dim=1)+ e = torch.cat([sink_e, cmp_e, tmp_e], dim=1)+ vec_action = torch.cat([sink_action_embedding, cmp_action, tmp_action], dim=1)+ else:+ tokens = torch.cat([cmp_tokens, tmp_tokens], dim=1)+ freqs_flat = torch.cat([cmp_freqs, tmp_freqs], dim=0)+ e0 = torch.cat([cmp_e0, tmp_e0], dim=1)+ e = torch.cat([cmp_e, tmp_e], dim=1)+ vec_action = torch.cat([cmp_action, tmp_action], dim=1)++ return {+ "tokens": tokens,+ "freqs_flat": freqs_flat,+ "e0": e0,+ "e": e,+ "vec_action": vec_action,+ "rope_shift_idx": hr_pf,+ "sink_token_count": sink_token_count,+ "cmp_token_count": cmp_tokens.shape[1],+ "cmp_chunk_token_count": 2 * cmp_spatial,+ "tmp_token_count": tmp_tokens.shape[1],+ }++ def forward(+ self,+ x=None,+ t=None,+ context=None,+ seq_len=None,+ y=None,+ hr_action=None,+ memory_dict=None,+ forward_mode="bi_denoise",+ kv_cache=None,+ ar_chunk_start=0,+ hr_latents_input=None,+ lr_latents_input=None,+ lr_action=None,+ memory_t=None,+ hr_temporal_latents_input=None,+ hr_temporal_action=None,+ sink_action=None,+ pdd_block_index=None,+ ):+ """Route all model operations through one explicit entry point."""+ with fp8_autocast(+ self.fp8_recipe,+ enabled=self.fp8_enabled,+ ):+ if forward_mode == "compress":+ return self._forward_compress(+ hr_latents_input=hr_latents_input,+ lr_latents_input=lr_latents_input,+ lr_action=lr_action,+ memory_t=memory_t,+ hr_temporal_latents_input=hr_temporal_latents_input,+ hr_temporal_action=hr_temporal_action,+ sink_action=sink_action,+ )+ if forward_mode == "bi_denoise":+ return self._forward_bi_denoise(+ x=x, t=t, context=context, seq_len=seq_len, y=y,+ hr_action=hr_action, memory_dict=memory_dict)+ if forward_mode == "ar_prefill":+ return self._forward_ar_prefill(+ context=context, memory_dict=memory_dict,+ kv_cache=kv_cache)+ if forward_mode == "ar_denoise":+ return self._forward_ar_denoise(+ x=x, t=t, context=context, seq_len=seq_len, y=y,+ hr_action=hr_action, memory_dict=memory_dict,+ kv_cache=kv_cache, ar_chunk_start=ar_chunk_start)+ if forward_mode == "pdd_denoise":+ self.head.set_active_block(pdd_block_index)+ return self._forward_ar_denoise(+ x=x, t=t, context=context, seq_len=seq_len, y=y,+ hr_action=hr_action, memory_dict=memory_dict,+ kv_cache=kv_cache, ar_chunk_start=ar_chunk_start)+ raise ValueError(+ f"unsupported forward_mode={forward_mode!r}; expected "+ "'compress', 'bi_denoise', 'ar_prefill', 'ar_denoise', "+ "'pdd_denoise'"+ )++ def _forward_bi_denoise(+ self, x, t, context, seq_len, y, hr_action, memory_dict+ ):+ """Bidirectional denoise with optional compressed history context."""+ # BI uses the same body for SP and non-SP. SP only shards the token+ # tensors before the blocks and gathers the head output afterwards.+ ar = False++ if self.model_type == 'i2v':+ assert y is not None+ # params+ device = self.patch_embedding.weight.device+ if self.freqs.device != device:+ self.freqs = self.freqs.to(device)++ h, w = x[0].shape[2:]+ tw = w // self.patch_size[2]+ th = h // self.patch_size[1]++ if y is not None:+ x = [torch.cat([u, v], dim=0) for u, v in zip(x, y)]++ # embeddings+ x = [self.patch_embedding(u.unsqueeze(0)) for u in x]+ grid_sizes = torch.stack(+ [torch.tensor(u.shape[2:], dtype=torch.long) for u in x])+ x = [u.flatten(2).transpose(1, 2) for u in x]+ seq_lens = torch.tensor([u.size(1) for u in x], dtype=torch.long)+ assert seq_lens.max() <= seq_len+ x = torch.cat([+ torch.cat(+ [u, u.new_zeros(1, seq_len - u.size(1), u.size(2))],+ dim=1,+ )+ for u in x+ ])++ # time embeddings+ if t.dim() == 1:+ t = t.expand(t.size(0), seq_len)+ with torch.amp.autocast('cuda', dtype=torch.float32):+ bt = t.size(0)+ t = t.flatten()+ e = self.time_embedding(+ sinusoidal_embedding_1d(+ self.freq_dim, t+ ).unflatten(0, (bt, seq_len)).float())+ e0 = self.time_projection(e).unflatten(2, (6, self.dim))+ assert e.dtype == torch.float32 and e0.dtype == torch.float32++ # context+ context_lens = None+ context = self._project_text_context(context)++ vec_action = None+ if hr_action is not None:+ vec_action = self._encode_action(hr_action)+ vec_action = repeat(+ vec_action, "(B T) D -> B (T H W) D",+ B=x.shape[0], H=th, W=tw)++ # Build per-token RoPE for the generate segment. The generate+ # segment ALWAYS uses per-token ``freqs_flat`` (via ``rope_apply_flat``)+ # -- both BI and AR, with or without a compressor context -- so the+ # RoPE handling is uniform and never depends on the grid-based+ # ``rope_apply`` fallback. When fused with a compressor context the+ # generate segment's temporal rope indices must START AFTER the+ # context's time range (= ``rope_shift_idx``); otherwise the generate+ # and context tokens would share the same temporal phases and the+ # model could not distinguish "past vs future".+ #+ # NB: for ``rope_shift_idx == 0`` (no context) the per-token freqs are+ # numerically identical to what grid-based ``rope_apply`` would build,+ # so this does not change the BI first-chunk result.+ gen_pf = int(grid_sizes[0, 0].item())+ gen_ph = int(grid_sizes[0, 1].item())+ gen_pw = int(grid_sizes[0, 2].item())+ rope_shift_idx = (int(memory_dict.get("rope_shift_idx", 0))+ if memory_dict is not None else 0)+ full_freqs = build_freqs_flat(+ self.freqs,+ (rope_shift_idx + gen_pf, gen_ph, gen_pw)).to(x.device)+ full_freqs = full_freqs.view(+ rope_shift_idx + gen_pf, gen_ph * gen_pw, -1)+ gen_freqs_flat = full_freqs[-gen_pf:].reshape(+ gen_pf * gen_ph * gen_pw, -1)+ # Pad to seq_len (pad tokens are masked out in flash_attention+ # via ``seq_lens``).+ if gen_freqs_flat.shape[0] < seq_len:+ pad_rows = seq_len - gen_freqs_flat.shape[0]+ pad = gen_freqs_flat.new_zeros(+ pad_rows, gen_freqs_flat.shape[1])+ gen_freqs_flat = torch.cat(+ [gen_freqs_flat, pad], dim=0)++ # -- fuse context (memory_compress output) before generate segment --+ generate_seq_length = x.shape[1]+ cmp_token_count = 0+ cmp_chunk_token_count = 0+ tmp_token_count = 0+ if memory_dict is not None:+ ctx_tokens = memory_dict["tokens"]+ ctx_freqs_flat = memory_dict["freqs_flat"].to(x.device)+ ctx_e0 = memory_dict["e0"].float()+ ctx_e = memory_dict["e"].float()+ ctx_vec_action = memory_dict["vec_action"]++ x = torch.cat([ctx_tokens, x], dim=1)+ freqs_flat = torch.cat(+ [ctx_freqs_flat, gen_freqs_flat], dim=0)+ e0 = torch.cat([ctx_e0, e0], dim=1)+ e = torch.cat([ctx_e, e], dim=1)+ vec_action = torch.cat(+ [ctx_vec_action, vec_action], dim=1)+ # seq_lens now counts the full (ctx + generate) length per batch+ seq_lens = torch.tensor([x.shape[1]] * x.shape[0],+ dtype=torch.long)+ cmp_token_count = int(memory_dict.get("cmp_token_count", 0))+ cmp_chunk_token_count = int(+ memory_dict.get("cmp_chunk_token_count", 0))+ tmp_token_count = int(memory_dict.get("tmp_token_count", 0))+ else:+ # No compressor ctx (first chunk): generate-only, per-token RoPE.+ freqs_flat = gen_freqs_flat++ if self.use_sp:+ sp_size = get_world_size()+ sp_rank = get_rank()+ global_sequence = x.shape[1]+ x = shard_sequence(x, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)+ e = shard_sequence(e, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)+ e0 = shard_sequence(e0, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)+ vec_action = shard_sequence(vec_action, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)+ freqs_flat = shard_sequence(freqs_flat, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=0)+ else:+ global_sequence = x.shape[1]++ # arguments+ kwargs = dict(+ e=e0,+ seq_lens=seq_lens,+ grid_sizes=grid_sizes,+ freqs=self.freqs,+ context=context,+ context_lens=context_lens,+ vec_action=vec_action,+ freqs_flat=freqs_flat,+ ar=ar,+ cmp_token_count=cmp_token_count,+ cmp_chunk_token_count=cmp_chunk_token_count,+ tmp_token_count=tmp_token_count,+ )++ for block in self.blocks:+ x = block(x, **kwargs)++ # head+ x = self.head(x, e)+ if self.use_sp:+ x = gather_sequence(x, global_sequence=global_sequence)++ # drop the prepended context tokens before unpatchify+ if memory_dict is not None:+ x = x[:, -generate_seq_length:]++ # unpatchify+ x = self.unpatchify(x, grid_sizes)+ return [u.float() for u in x]++ def init_kv_cache(self):+ r"""Allocate contiguous per-layer ``[sink | cmp | tmp]`` K/V."""+ return [{'k_vision': None, 'v_vision': None}+ for _ in range(self.num_layers)]++ def _forward_ar_prefill(self, context, memory_dict, kv_cache):+ r"""Chunk-AR KV-cache incremental prefill for SP and non-SP.++ Each layer stores only contiguous ``k_vision/v_vision`` with layout+ ``[sink | accumulated cmp | latest tmp]``. The first context is+ processed in one masked pass. Later calls infer the old cmp length+ from the cache shape, discard the previous tmp suffix, process only+ ``[new cmp | new tmp]``, and replace the cache with the new layout.++ Args:+ context (List[Tensor]): text embeddings (cond OR uncond).+ memory_dict (dict | None): compressor output; ``None`` -> no-op.+ kv_cache (list[dict]): from ``init_kv_cache()``; mutated in place.++ Returns:+ list[dict]: the same ``kv_cache`` (mutated in place).+ """+ assert kv_cache is not None and len(kv_cache) == self.num_layers, (+ "ar_prefill requires a kv_cache of length num_layers "+ "(call init_kv_cache()).")++ # First chunk: no compressor ctx -> nothing to cache.+ if memory_dict is None:+ return kv_cache++ device = self.patch_embedding.weight.device+ if self.freqs.device != device:+ self.freqs = self.freqs.to(device)++ # text context (embed cond/uncond text the same way as the denoise path)+ context_lens = None+ context = self._project_text_context(context)++ ctx_tokens = memory_dict["tokens"]+ freqs_flat = memory_dict["freqs_flat"].to(ctx_tokens.device)+ e0c = memory_dict["e0"].float() # ensure fp32 (FSDP bf16)+ vec_action_c = memory_dict["vec_action"]+ sink_token_count = int(memory_dict.get("sink_token_count", 0))+ cmp_token_count = int(memory_dict.get("cmp_token_count", 0))+ cmp_chunk_token_count = int(+ memory_dict.get("cmp_chunk_token_count", 0))+ tmp_token_count = int(memory_dict.get("tmp_token_count", 0))++ cache_is_empty = kv_cache[0].get('k_vision') is None+ if any(+ (entry.get('k_vision') is None) != cache_is_empty+ for entry in kv_cache+ ):+ raise ValueError("all AR cache layers must have the same state")++ if cache_is_empty:+ self._prefill_initial_ar_context(+ tokens=ctx_tokens,+ modulation=e0c,+ actions=vec_action_c,+ freqs_flat=freqs_flat,+ text_context=context,+ context_lens=context_lens,+ kv_cache=kv_cache,+ sink_token_count=sink_token_count,+ cmp_token_count=cmp_token_count,+ cmp_chunk_token_count=cmp_chunk_token_count,+ tmp_token_count=tmp_token_count,+ )+ else:+ self._prefill_ar_context_update(+ tokens=ctx_tokens,+ modulation=e0c,+ actions=vec_action_c,+ freqs_flat=freqs_flat,+ text_context=context,+ context_lens=context_lens,+ kv_cache=kv_cache,+ sink_token_count=sink_token_count,+ cmp_token_count=cmp_token_count,+ cmp_chunk_token_count=cmp_chunk_token_count,+ tmp_token_count=tmp_token_count,+ )+ return kv_cache++ def _prefill_initial_ar_context(+ self,+ *,+ tokens,+ modulation,+ actions,+ freqs_flat,+ text_context,+ context_lens,+ kv_cache,+ sink_token_count,+ cmp_token_count,+ cmp_chunk_token_count,+ tmp_token_count,+ ):+ """First prefill: write the complete ``[sink | cmp | tmp]`` cache."""+ self._run_ar_prefill_blocks(+ tokens=tokens,+ modulation=modulation,+ actions=actions,+ freqs_flat=freqs_flat,+ text_context=text_context,+ context_lens=context_lens,+ kv_cache=kv_cache,+ sink_token_count=sink_token_count,+ cmp_token_count=cmp_token_count,+ cmp_chunk_token_count=cmp_chunk_token_count,+ tmp_token_count=tmp_token_count,+ cache_update=None,+ )++ def _prefill_ar_context_update(+ self,+ *,+ tokens,+ modulation,+ actions,+ freqs_flat,+ text_context,+ context_lens,+ kv_cache,+ sink_token_count,+ cmp_token_count,+ cmp_chunk_token_count,+ tmp_token_count,+ ):+ """Later prefill: append new cmp and replace the previous tmp."""+ cached_total = int(kv_cache[0]['k_vision'].shape[1])+ cached_cmp = cached_total - sink_token_count - tmp_token_count+ if not 0 <= cached_cmp <= cmp_token_count:+ raise ValueError(+ "invalid AR cache layout: "+ f"cached_total={cached_total}, sink={sink_token_count}, "+ f"tmp={tmp_token_count}, current_cmp={cmp_token_count}"+ )++ new_cmp_count = cmp_token_count - cached_cmp+ cmp_start = sink_token_count + cached_cmp+ cmp_end = sink_token_count + cmp_token_count+ tmp_start = cmp_end+ tmp_end = tmp_start + tmp_token_count++ token_parts = []+ modulation_parts = []+ action_parts = []+ freq_parts = []+ if new_cmp_count:+ token_parts.append(tokens[:, cmp_start:cmp_end])+ modulation_parts.append(modulation[:, cmp_start:cmp_end])+ freq_parts.append(freqs_flat[cmp_start:cmp_end])+ if actions is not None:+ action_parts.append(actions[:, cmp_start:cmp_end])+ if tmp_token_count:+ token_parts.append(tokens[:, tmp_start:tmp_end])+ modulation_parts.append(modulation[:, tmp_start:tmp_end])+ freq_parts.append(freqs_flat[tmp_start:tmp_end])+ if actions is not None:+ action_parts.append(actions[:, tmp_start:tmp_end])+ if not token_parts:+ return++ self._run_ar_prefill_blocks(+ tokens=torch.cat(token_parts, dim=1),+ modulation=torch.cat(modulation_parts, dim=1),+ actions=(+ torch.cat(action_parts, dim=1)+ if action_parts else None+ ),+ freqs_flat=torch.cat(freq_parts, dim=0),+ text_context=text_context,+ context_lens=context_lens,+ kv_cache=kv_cache,+ sink_token_count=sink_token_count,+ cmp_token_count=cmp_token_count,+ cmp_chunk_token_count=cmp_chunk_token_count,+ tmp_token_count=tmp_token_count,+ cache_update={+ 'n_new_cmp': new_cmp_count,+ 'n_tmp': tmp_token_count,+ 'cached_prefix_tokens': sink_token_count + cached_cmp,+ },+ )++ def _run_ar_prefill_blocks(+ self,+ *,+ tokens,+ modulation,+ actions,+ freqs_flat,+ text_context,+ context_lens,+ kv_cache,+ sink_token_count,+ cmp_token_count,+ cmp_chunk_token_count,+ tmp_token_count,+ cache_update,+ ):+ """Run either initial or incremental prefill through all blocks."""+ global_length = tokens.shape[1]+ if self.use_sp:+ sp_size = get_world_size()+ sp_rank = get_rank()+ tokens = shard_sequence(tokens, global_sequence=global_length, rank=sp_rank, world_size=sp_size, dim=1)+ modulation = shard_sequence(modulation, global_sequence=global_length, rank=sp_rank, world_size=sp_size, dim=1)+ freqs_flat = shard_sequence(freqs_flat, global_sequence=global_length, rank=sp_rank, world_size=sp_size, dim=0)+ actions = shard_sequence(actions, global_sequence=global_length, rank=sp_rank, world_size=sp_size, dim=1)++ seq_lens = torch.tensor([global_length] * tokens.shape[0], dtype=torch.long)+ for index, block in enumerate(self.blocks):+ tokens = block(+ tokens,+ e=modulation,+ seq_lens=seq_lens,+ grid_sizes=None,+ freqs=self.freqs,+ context=text_context,+ context_lens=context_lens,+ vec_action=actions,+ freqs_flat=freqs_flat,+ ar=True,+ cmp_token_count=cmp_token_count,+ cmp_chunk_token_count=cmp_chunk_token_count,+ tmp_token_count=tmp_token_count,+ kv_cache_entry=kv_cache[index],+ cache_write=True,+ sink_token_count=sink_token_count,+ cache_update=cache_update,+ )++ def _forward_ar_denoise(+ self,+ x=None,+ t=None,+ context=None,+ seq_len=None,+ y=None,+ hr_action=None,+ memory_dict=None,+ kv_cache=None,+ ar_chunk_start=0,+ ):+ r"""Chunk-AR KV-cache denoise for both SP and non-SP.++ Runs the gen (pred) tokens through every block with+ ``cache_write=False``; each block reads its cached ctx K/V (written+ by :meth:`_forward_ar_prefill`) and prepends it to the pred K/V. No+ ctx tokens are concatenated to ``x`` here. Returns ``list[Tensor]``.++ ``ar_chunk_start`` phase-shifts the generate segment's RoPE so K/V+ across chunks share a consistent temporal coordinate system. When+ ``memory_dict`` is ``None`` / the cache is empty (first chunk) this+ degenerates to plain pred-only self-attention.+ """+ assert kv_cache is not None, (+ "AR denoise requires kv_cache (call init_kv_cache()).")+ assert len(kv_cache) == self.num_layers, (+ f"kv_cache length ({len(kv_cache)}) must equal num_layers "+ f"({self.num_layers}).")++ if self.model_type == 'i2v':+ assert y is not None+ device = self.patch_embedding.weight.device+ if self.freqs.device != device:+ self.freqs = self.freqs.to(device)++ h, w = x[0].shape[2:]+ tw = w // self.patch_size[2]+ th = h // self.patch_size[1]++ if y is not None:+ x = [torch.cat([u, v], dim=0) for u, v in zip(x, y)]++ # embeddings (gen/pred tokens)+ x = [self.patch_embedding(u.unsqueeze(0)) for u in x]+ grid_sizes = torch.stack([torch.tensor(u.shape[2:], dtype=torch.long) for u in x])+ x = [u.flatten(2).transpose(1, 2) for u in x]+ seq_lens = torch.tensor([u.size(1) for u in x], dtype=torch.long)+ assert seq_lens.max() <= seq_len+ x = torch.cat([+ torch.cat(+ [u, u.new_zeros(1, seq_len - u.size(1), u.size(2))],+ dim=1,+ )+ for u in x+ ])++ # time embeddings+ if t.dim() == 1:+ t = t.expand(t.size(0), seq_len)+ with torch.amp.autocast('cuda', dtype=torch.float32):+ bt = t.size(0)+ t = t.flatten()+ e = self.time_embedding(sinusoidal_embedding_1d(self.freq_dim, t).unflatten(0, (bt, seq_len)).float())+ e0 = self.time_projection(e).unflatten(2, (6, self.dim))+ assert e.dtype == torch.float32 and e0.dtype == torch.float32++ # text context+ context_lens = None+ context = self._project_text_context(context)++ vec_action = None+ if hr_action is not None:+ vec_action = self._encode_action(hr_action)+ vec_action = repeat(vec_action, "(B T) D -> B (T H W) D", B=x.shape[0], H=th, W=tw)++ # ---- per-token RoPE for the generate segment --------------------+ # The current chunk lives at hr-latent indices+ # [ar_chunk_start, ar_chunk_start + gen_pf). When a compressor ctx is+ # present, the generate segment must start AFTER the ctx time range+ # (rope_shift_idx); resolve gen_start = max(ar_chunk_start, ctx_shift).+ gen_pf = int(grid_sizes[0, 0].item())+ gen_ph = int(grid_sizes[0, 1].item())+ gen_pw = int(grid_sizes[0, 2].item())+ ctx_shift = int(memory_dict.get("rope_shift_idx", 0)) if memory_dict is not None else 0+ gen_start = max(int(ar_chunk_start), ctx_shift)+ full_freqs = build_freqs_flat(self.freqs, (gen_start + gen_pf, gen_ph, gen_pw)).to(x.device)+ full_freqs = full_freqs.view(gen_start + gen_pf, gen_ph * gen_pw, -1)+ gen_freqs_flat = full_freqs[-gen_pf:].reshape(gen_pf * gen_ph * gen_pw, -1)++ # x = pred only; each block reads cached ctx K/V and prepends to pred.+ freqs_flat = gen_freqs_flat+ cmp_token_count = int(memory_dict.get("cmp_token_count", 0)) if memory_dict else 0+ cmp_chunk_token_count = int(memory_dict.get("cmp_chunk_token_count", 0)) if memory_dict else 0+ tmp_token_count = int(memory_dict.get("tmp_token_count", 0)) if memory_dict else 0+ seq_lens = torch.tensor([x.shape[1]] * x.shape[0], dtype=torch.long)++ if self.use_sp:+ sp_size = get_world_size()+ sp_rank = get_rank()+ global_sequence = x.shape[1]+ x = shard_sequence(x, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)+ e = shard_sequence(e, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)+ e0 = shard_sequence(e0, global_sequence=global_sequence, rank=sp_rank, world_size=sp_size, dim=1)+ vec_action = shard_sequence(vec_action, global_sequence=global_sequence, rank=sp_rank,+ world_size=sp_size, dim=1)+ freqs_flat = shard_sequence(freqs_flat, global_sequence=global_sequence, rank=sp_rank,+ world_size=sp_size, dim=0)+ else:+ global_sequence = x.shape[1]++ for idx, block in enumerate(self.blocks):+ x = block(+ x,+ e=e0,+ seq_lens=seq_lens,+ grid_sizes=grid_sizes,+ freqs=self.freqs,+ context=context,+ context_lens=context_lens,+ vec_action=vec_action,+ freqs_flat=freqs_flat,+ ar=True,+ cmp_token_count=cmp_token_count,+ cmp_chunk_token_count=cmp_chunk_token_count,+ tmp_token_count=tmp_token_count,+ kv_cache_entry=kv_cache[idx],+ cache_write=False,+ )++ # head+ x = self.head(x, e)+ endpoint_x = self.head.pop_endpoint_output()+ if self.use_sp:+ x = gather_sequence(x, global_sequence=global_sequence)+ if endpoint_x is not None:+ endpoint_x = gather_sequence(endpoint_x, global_sequence=global_sequence)++ # unpatchify (no ctx prefix to strip -- x is pred-only throughout)+ x = self.unpatchify(x, grid_sizes)+ if endpoint_x is None:+ return [u.float() for u in x]+ endpoint_x = self.unpatchify(endpoint_x, grid_sizes)+ return [+ (displacement.float(), endpoint.float())+ for displacement, endpoint in zip(x, endpoint_x)+ ]++ def unpatchify(self, x, grid_sizes):+ r"""+ Reconstruct video tensors from patch embeddings.++ Args:+ x (List[Tensor]):+ List of patchified features, each with shape [L, C_out * prod(patch_size)]+ grid_sizes (Tensor):+ Original spatial-temporal grid dimensions before patching,+ shape [B, 3] (3 dimensions correspond to F_patches, H_patches, W_patches)++ Returns:+ List[Tensor]:+ Reconstructed video tensors with shape [C_out, F, H / 8, W / 8]+ """++ c = self.out_dim+ out = []+ for u, v in zip(x, grid_sizes.tolist()):+ u = u[:math.prod(v)].view(*v, *self.patch_size, c)+ u = torch.einsum('fhwpqrc->cfphqwr', u)+ u = u.reshape(c, *[i * j for i, j in zip(v, self.patch_size)])+ out.append(u)+ return out++ def init_weights(self):+ r"""+ Initialize model parameters using Xavier initialization.+ """++ # basic init+ for m in self.modules():+ if isinstance(m, nn.Linear):+ nn.init.xavier_uniform_(m.weight)+ if m.bias is not None:+ nn.init.zeros_(m.bias)++ # init embeddings+ nn.init.xavier_uniform_(self.patch_embedding.weight.flatten(1))+ for m in self.text_embedding.modules():+ if isinstance(m, nn.Linear):+ nn.init.normal_(m.weight, std=.02)+ for m in self.time_embedding.modules():+ if isinstance(m, nn.Linear):+ nn.init.normal_(m.weight, std=.02)++ # init output layer+ nn.init.zeros_(self.head.head.weight)diff --git a/worldplay2/pdd.py b/worldplay2/pdd.pynew file mode 100644index 0000000..f0d3362--- /dev/null+++ b/worldplay2/pdd.py@@ -0,0 +1,169 @@+from __future__ import annotations++from dataclasses import dataclass++import torch++from .checkpoint import load_state_dict+++@dataclass(frozen=True)+class FixedPDDBlock:+ expert: str+ local_index: int+ start: int+ end: int+++class FixedPDD4Scheduler:+ """Fixed four-NFE consistency-PDD schedule used by Self-Forcing."""++ num_intervals = 128+ shift = 7.0+ boundary = 0.9++ def __init__(self):+ raw_boundary = self.boundary / (+ self.shift - (self.shift - 1.0) * self.boundary+ )+ high_count = int(round(self.num_intervals * (1.0 - raw_boundary)))+ high_count = min(max(high_count, 1), self.num_intervals - 1)+ raw = torch.cat([+ torch.linspace(1.0, raw_boundary, high_count + 1),+ torch.linspace(+ raw_boundary,+ 0.0,+ self.num_intervals - high_count + 1,+ )[1:],+ ])+ self.sigmas = (+ self.shift * raw / (1.0 + (self.shift - 1.0) * raw)+ ).float()+ self.sigmas[0] = 1.0+ self.sigmas[high_count] = self.boundary+ self.sigmas[-1] = 0.0+ self.timesteps = self.sigmas * 1000.0+ self.boundary_index = high_count++ high_edges = self._partition(0, high_count)+ low_edges = self._partition(high_count, self.num_intervals)+ self._blocks = {+ "high": tuple(+ FixedPDDBlock("high", i, high_edges[i], high_edges[i + 1])+ for i in range(2)+ ),+ "low": tuple(+ FixedPDDBlock("low", i, low_edges[i], low_edges[i + 1])+ for i in range(2)+ ),+ }+ self.ordered_blocks = self._blocks["high"] + self._blocks["low"]++ @staticmethod+ def _partition(start: int, end: int) -> list[int]:+ return [start, start + (end - start) // 2, end]++ def blocks(self, expert: str) -> tuple[FixedPDDBlock, ...]:+ try:+ return self._blocks[expert]+ except KeyError as exc:+ raise ValueError(f"expert must be high/low, got {expert!r}") from exc++ def expert_range(self, expert: str) -> tuple[int, int]:+ if expert == "high":+ return 0, self.boundary_index+ if expert == "low":+ return self.boundary_index, self.num_intervals+ raise ValueError(f"expert must be high/low, got {expert!r}")++ def compact_head(+ self,+ expert: str,+ interval_weight: torch.Tensor,+ interval_bias: torch.Tensor,+ ) -> tuple[torch.Tensor, torch.Tensor]:+ expert_start, expert_end = self.expert_range(expert)+ expected = expert_end - expert_start+ if interval_weight.ndim != 3 or interval_weight.shape[0] != expected:+ raise ValueError(+ f"{expert} PDD checkpoint has interval weight shape "+ f"{tuple(interval_weight.shape)}, expected [{expected}, O, I]"+ )+ if interval_bias.ndim != 2 or interval_bias.shape[0] != expected:+ raise ValueError(+ f"{expert} PDD checkpoint has interval bias shape "+ f"{tuple(interval_bias.shape)}, expected [{expected}, O]"+ )++ weights = []+ biases = []+ for block in self.blocks(expert):+ indices = [+ index - expert_start+ for index in range(block.start, block.end - 1)+ ]+ coefficients = (+ self.sigmas[block.start + 1:block.end]+ - self.sigmas[block.start:block.end - 1]+ ).to(interval_weight)+ displacement_weight = torch.einsum(+ "k,koi->oi", coefficients, interval_weight[indices]+ )+ displacement_bias = torch.einsum(+ "k,ko->o", coefficients, interval_bias[indices]+ )+ endpoint_index = block.end - 1 - expert_start+ weights.append(torch.cat(+ (displacement_weight, interval_weight[endpoint_index]), dim=0+ ))+ biases.append(torch.cat(+ (displacement_bias, interval_bias[endpoint_index]), dim=0+ ))+ return torch.stack(weights), torch.stack(biases)++ def prepare_checkpoint(+ self,+ path: str,+ *,+ expert: str,+ dtype: torch.dtype,+ ) -> dict[str, torch.Tensor]:+ """Load either interval-head or already compact fixed-PDD weights."""+ state = load_state_dict(path)+ compact_weight = "head.block_weight"+ compact_bias = "head.block_bias"+ if compact_weight not in state or compact_bias not in state:+ weight_key = next((+ key for key in (+ "head.weight",+ "head.pdd_weight",+ "head.head.weight",+ )+ if key in state and state[key].ndim == 3+ ), None)+ bias_key = next((+ key for key in (+ "head.bias",+ "head.pdd_bias",+ "head.head.bias",+ )+ if key in state and state[key].ndim == 2+ ), None)+ if weight_key is None or bias_key is None:+ raise KeyError(+ f"{path} has neither compact fixed-PDD heads nor an "+ "interval PDD head bank"+ )+ block_weight, block_bias = self.compact_head(+ expert, state.pop(weight_key), state.pop(bias_key)+ )+ state[compact_weight] = block_weight+ state[compact_bias] = block_bias++ # A compact resume can retain a now-unused standard projection.+ state.pop("head.head.weight", None)+ state.pop("head.head.bias", None)+ for key, tensor in state.items():+ if tensor.is_floating_point() and tensor.dtype != dtype:+ state[key] = tensor.to(dtype=dtype)+ return statediff --git a/worldplay2/pipeline.py b/worldplay2/pipeline.pynew file mode 100644index 0000000..24cc31c--- /dev/null+++ b/worldplay2/pipeline.py@@ -0,0 +1,1291 @@+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.+"""WorldPlay2 chunked inference with compressed memory and temporal anchors,+built on Wan2.2 I2V-A14B.++Recipe (per HY WorldPlay reference, adapted to Wan2.2):+ * Generate ``total_latent_frames`` HR latent frames in groups of+ ``chunk_length`` frames (default 4).+ * For chunk 0, run a vanilla i2v denoising loop without context.+ * For chunks > 0:+ 1. Streaming-VAE-decode the just-finished HR chunk.+ 2. Take ``frames[:, ::2]`` (every other frame) and bilinear+ downsample to (H/4, W/4) in pixel space.+ 3. Streaming-VAE-encode the downsampled clip back to LR latents.+ 4. Accumulate (lr_latents_cache, hr_latents_cache, decoded_rgb_cache).+ 5. Build ``hr_temporal`` = last 2 hr latents before the generate+ window with zero y, build hr_y / lr_y context (4-mask + 16-image+ latent, image-only at t=0), then call+ ``model(forward_mode="compress", ...)`` to+ produce ``memory_dict``.+ 6. Run the chunk's denoising loop with ``memory_dict`` plumbed in.+ * After the final chunk, decode the last chunk (continuing the same+ streaming cache) and concatenate with the cached RGB.++The pipeline builds the T5 encoder, VAE, and two ``WorldPlay2Model`` experts.+Each expert loads its complete DiT, action encoder, and memory compressor+weights from a separate checkpoint.++Sequence-parallel behavior is owned by ``WorldPlay2Model`` and+``WorldPlay2SelfAttention`` themselves. The pipeline only enables it through+``model.set_sequence_parallel``; no runtime monkey-patching is used.+"""++import gc+import logging+import math+import os+import random+import sys+from contextlib import contextmanager+from enum import Enum+from functools import partial++import torch+import torch.distributed as dist+import torch.nn.functional as F+import torchvision.transforms.functional as TF+from torch.distributed.fsdp import FullyShardedDataParallel as FSDP+from tqdm import tqdm++from .checkpoint import load_expert_checkpoint+from .distributed.fsdp import shard_model+from .distributed.ops import get_world_size+from .models.attention import resolve_attention_backend+from .models.model import WorldPlay2Model+from .models.fp8 import (+ convert_model_ffn_to_fp8,+ convert_model_qkv_to_fp8,+ fp8_model_stats,+)+from .pdd import FixedPDD4Scheduler+from .schedulers import (FlowDPMSolverMultistepScheduler,+ FlowUniPCMultistepScheduler, get_sampling_sigmas,+ retrieve_timesteps)+from .text.encoder import T5EncoderModel+from .vae.wan21 import Wan2_1_VAE+++# ---------------------------------------------------------------------------+# small utilities (mirror the dataset/training conventions verbatim)+# ---------------------------------------------------------------------------++def _build_i2v_mask_4ch(t_lat: int, h_lat: int, w_lat: int,+ device, dtype) -> torch.Tensor:+ """Build the 4-channel "1,0,0,..." mask used as the first 4 channels+ of ``y``. Mirrors ``bi_camera_wan22_w_mem_dataset._build_i2v_mask``.+ """+ m = torch.ones(1, t_lat * 4 - 3, h_lat, w_lat, device=device, dtype=dtype)+ m[:, 1:] = 0+ m = torch.cat([m[:, 0:1].repeat_interleave(4, dim=1), m[:, 1:]], dim=1)+ m = m.view(1, m.shape[1] // 4, 4, h_lat, w_lat).transpose(1, 2)[0]+ return m # [4, t_lat, h_lat, w_lat]+++def _encode_i2v_clip(vae, img_pixel: torch.Tensor, t_lat: int,+ h_pix: int, w_pix: int, device) -> torch.Tensor:+ """VAE-encode the I2V pixel clip ``[image, zeros..., zeros]`` into a+ 16-channel latent of temporal length ``t_lat``.++ Mirrors the upstream :meth:`WanI2V.generate` recipe verbatim::++ y = vae.encode([+ cat([+ F.interpolate(img[None], (h, w), mode='bicubic').transpose(0,1),+ zeros(3, F_pix - 1, h, w),+ ], dim=1)+ ])[0]++ where ``F_pix = t_lat * 4 - 3`` so the temporal-causal VAE produces+ exactly ``t_lat`` latent frames. ``img_pixel`` is on ``device``,+ shape ``[3, H_in, W_in]``, value range [-1, 1].+ """+ F_pix = t_lat * 4 - 3+ clip = torch.cat([+ torch.nn.functional.interpolate(+ img_pixel[None].cpu(), size=(h_pix, w_pix),+ mode='bilinear').transpose(0, 1), # [3, 1, h, w]+ torch.zeros(3, F_pix - 1, h_pix, w_pix),+ ], dim=1).to(device) # [3, F_pix, h, w]+ return vae.encode([clip])[0] # [16, t_lat, h, w]+++def _build_y(vae, img_pixel: torch.Tensor, t_lat: int,+ h_pix: int, w_pix: int, device, dtype) -> torch.Tensor:+ """Construct the 20-channel y volume = [4-mask | 16-image-clip-latent]+ using the same recipe as upstream :meth:`WanI2V.generate`.+ """+ y_img = _encode_i2v_clip(vae, img_pixel, t_lat, h_pix, w_pix, device)+ h_lat, w_lat = y_img.shape[2], y_img.shape[3]+ msk = _build_i2v_mask_4ch(t_lat, h_lat, w_lat, device, y_img.dtype)+ return torch.cat([msk, y_img], dim=0).to(dtype=dtype) # [20, t_lat, h_lat, w_lat]+++@contextmanager+def _noop_no_sync():+ yield+++def _parameter_dtype_summary(module) -> dict[str, int]:+ counts: dict[str, int] = {}+ for parameter in module.parameters():+ name = str(parameter.dtype).replace("torch.", "")+ counts[name] = counts.get(name, 0) + parameter.numel()+ return counts+++def _fsdp_unit_count(module) -> int:+ return sum(+ 1 for child in module.modules() if isinstance(child, FSDP)+ )+++class InferenceMode(str, Enum):+ BI = "bi"+ AR = "ar"+ PDD = "pdd"+++# ---------------------------------------------------------------------------+# main pipeline+# ---------------------------------------------------------------------------++class WorldPlay2Pipeline:+ """WorldPlay2 pipeline for chunked action-conditioned video generation.++ Owns: T5 encoder, Wan2.1 VAE, two ``WorldPlay2Model`` experts (low / high+ noise) with the action MLP + memory_compress submodule installed.+ """++ def __init__(+ self,+ config,+ checkpoint_dir: str,+ low_noise_mem_ckpt: str,+ high_noise_mem_ckpt: str,+ device_id: int = 0,+ rank: int = 0,+ t5_fsdp: bool = False,+ dit_fsdp: bool = False,+ use_sp: bool = False,+ compile_qk_rope: bool = True,+ conditioning_cache: bool = True,+ fuse_self_qkv: bool = True,+ compile_block_glue: bool = True,+ compile_ffn: bool = True,+ fp8_ffn: bool = False,+ fp8_start_block: int = 4,+ fp8_end_block: int = 35,+ fp8_qkv: bool = False,+ fp8_qkv_start_block: int = 4,+ fp8_qkv_end_block: int = 35,+ init_on_cpu: bool = True,+ inference_mode: InferenceMode | str = InferenceMode.BI,+ vae_type: str = "wan2_1",+ tae_ckpt: str = None,+ attention_backend: str = "flash",+ ):+ """+ Args:+ config: model settings from ``configs.I2V_A14B_CONFIG``.+ checkpoint_dir: directory holding the base T5 / VAE / DiT+ checkpoints (same layout :class:`WanI2V` expects).+ low_noise_mem_ckpt: path to a ``.safetensors``/``.pt`` file+ holding the LOW-noise expert's trained ``action_in.*``,+ ``img_action_mlp.*`` and ``memory_compress.*`` weights.+ high_noise_mem_ckpt: same, but for the HIGH-noise expert.+ The two experts are trained separately (see+ ``run_bi_wan22_action_mem_stage_three.sh`` with /+ without ``--low_noise``), so each carries its own+ memory_compress + action weights and at inference we+ must run TWO compressor passes per chunk and dispatch+ by timestep just like the denoise step does.+ device_id, rank, t5_fsdp, dit_fsdp, use_sp, init_on_cpu:+ runtime placement options.+ attention_backend: visual denoising attention backend, 'flash'+ or 'sage'.+ """+ assert low_noise_mem_ckpt is not None and high_noise_mem_ckpt is not None, (+ "WorldPlay2Pipeline requires BOTH --low_noise_mem_ckpt and "+ "--high_noise_mem_ckpt; the FastVideo training pipeline "+ "produces a separate checkpoint for each expert (each with "+ "its own memory_compress + action weights) and the inference "+ "loop dispatches between them by timestep.")++ self.inference_mode = InferenceMode(inference_mode)+ self.attention_backend = resolve_attention_backend(attention_backend)+ logging.info("Visual self-attention backend: %s", self.attention_backend)+ self.device = torch.device(f"cuda:{device_id}")+ self.config = config+ self.rank = rank+ self.init_on_cpu = init_on_cpu+ self.denoise_forward_mode = (+ "bi_denoise"+ if self.inference_mode is InferenceMode.BI+ else (+ "pdd_denoise"+ if self.inference_mode is InferenceMode.PDD+ else "ar_denoise"+ )+ )+ self.pdd_scheduler = (+ FixedPDD4Scheduler()+ if self.inference_mode is InferenceMode.PDD+ else None+ )+ self.use_sp = use_sp+ self.compile_qk_rope = compile_qk_rope+ self.conditioning_cache_enabled = conditioning_cache+ self.fuse_self_qkv = fuse_self_qkv+ self.compile_block_glue = compile_block_glue+ self.compile_ffn = compile_ffn+ self.fp8_ffn_enabled = fp8_ffn+ self.fp8_start_block = fp8_start_block+ self.fp8_end_block = fp8_end_block+ self.fp8_qkv_enabled = fp8_qkv+ self.fp8_qkv_start_block = fp8_qkv_start_block+ self.fp8_qkv_end_block = fp8_qkv_end_block+ self.fp8_enabled = fp8_ffn or fp8_qkv+ if self.fp8_enabled and dit_fsdp:+ raise ValueError(+ "DiT FP8 currently requires --dit_fsdp to be disabled")+ if fp8_qkv and not fuse_self_qkv:+ raise ValueError(+ "QKV FP8 requires self-attention QKV fusion")++ self.num_train_timesteps = config.num_train_timesteps+ self.boundary = config.boundary+ self.param_dtype = config.param_dtype++ if t5_fsdp or dit_fsdp or use_sp:+ self.init_on_cpu = False++ shard_fn = partial(shard_model, device_id=device_id)+ self.text_encoder = T5EncoderModel(+ text_len=config.text_len,+ dtype=config.t5_dtype,+ device=self.device,+ checkpoint_path=os.path.join(checkpoint_dir, config.t5_checkpoint),+ tokenizer_path=os.path.join(checkpoint_dir, config.t5_tokenizer),+ shard_fn=shard_fn if t5_fsdp else None,+ )++ self.vae_stride = config.vae_stride+ self.patch_size = config.patch_size+ # VAE is pluggable: the default Wan2.1 VAE, or the causal TAE+ # (lightvae) which exposes the SAME encode/decode/encode_chunk/+ # decode_chunk list interface via CausalTAEVAE, so nothing downstream+ # changes. Both are temporally causal (T_pix = 4*T_lat - 3).+ assert vae_type in ("wan2_1", "causal_tae"), (+ f"vae_type must be 'wan2_1' or 'causal_tae', got {vae_type!r}")+ if vae_type == "causal_tae":+ from .vae.causal_tae import CausalTAEVAE+ assert tae_ckpt is not None, (+ "vae_type='causal_tae' requires --tae_ckpt "+ "(the CausalTAE encoder+decoder weights).")+ self.vae = CausalTAEVAE(+ vae_pth=tae_ckpt,+ z_dim=16,+ device=self.device)+ else:+ self.vae = Wan2_1_VAE(+ vae_pth=os.path.join(checkpoint_dir, config.vae_checkpoint),+ device=self.device)++ # Build the architecture from the base model config only. The supplied+ # expert checkpoints contain the complete DiT + action + compressor+ # state, so loading base-model parameters through from_pretrained would+ # be redundant and would temporarily consume additional host memory.+ self.low_noise_model = self._build_expert_from_config(+ checkpoint_dir=checkpoint_dir,+ config_subfolder=config.low_noise_checkpoint,+ expert_ckpt=low_noise_mem_ckpt,+ tag="low_noise_model",+ dtype=self.param_dtype,+ fuse_self_qkv=fuse_self_qkv,+ pdd_scheduler=self.pdd_scheduler,+ pdd_expert="low",+ )+ self.high_noise_model = self._build_expert_from_config(+ checkpoint_dir=checkpoint_dir,+ config_subfolder=config.high_noise_checkpoint,+ expert_ckpt=high_noise_mem_ckpt,+ tag="high_noise_model",+ dtype=self.param_dtype,+ fuse_self_qkv=fuse_self_qkv,+ pdd_scheduler=self.pdd_scheduler,+ pdd_expert="high",+ )++ # ---- configure (FSDP / SP / dtype / device) ----------------------+ self.low_noise_model = self._configure_model(+ model=self.low_noise_model,+ use_sp=use_sp,+ attention_backend=self.attention_backend,+ compile_qk_rope=compile_qk_rope,+ conditioning_cache=conditioning_cache,+ compile_block_glue=compile_block_glue,+ compile_ffn=compile_ffn,+ fp8_ffn=fp8_ffn,+ fp8_start_block=fp8_start_block,+ fp8_end_block=fp8_end_block,+ fp8_qkv=fp8_qkv,+ fp8_qkv_start_block=fp8_qkv_start_block,+ fp8_qkv_end_block=fp8_qkv_end_block,+ dit_fsdp=dit_fsdp,+ shard_fn=shard_fn)+ self.high_noise_model = self._configure_model(+ model=self.high_noise_model,+ use_sp=use_sp,+ attention_backend=self.attention_backend,+ compile_qk_rope=compile_qk_rope,+ conditioning_cache=conditioning_cache,+ compile_block_glue=compile_block_glue,+ compile_ffn=compile_ffn,+ fp8_ffn=fp8_ffn,+ fp8_start_block=fp8_start_block,+ fp8_end_block=fp8_end_block,+ fp8_qkv=fp8_qkv,+ fp8_qkv_start_block=fp8_qkv_start_block,+ fp8_qkv_end_block=fp8_qkv_end_block,+ dit_fsdp=dit_fsdp,+ shard_fn=shard_fn)++ self.sp_size = get_world_size() if use_sp else 1+ self.sample_neg_prompt = config.sample_neg_prompt+ self.dtype_report = {+ "attention_backend": self.attention_backend,+ "configured_dit_compute_dtype": str(+ self.param_dtype).replace("torch.", ""),+ "t5_fsdp_requested": bool(t5_fsdp),+ "dit_fsdp": bool(dit_fsdp),+ "t5_fsdp_units": _fsdp_unit_count(self.text_encoder.model),+ "low_noise_fsdp_units": _fsdp_unit_count(+ self.low_noise_model),+ "high_noise_fsdp_units": _fsdp_unit_count(+ self.high_noise_model),+ "compile_qk_rope": bool(compile_qk_rope),+ "conditioning_cache": bool(conditioning_cache),+ "self_qkv_fused": bool(fuse_self_qkv),+ "compile_block_glue": bool(compile_block_glue),+ "compile_ffn": bool(compile_ffn),+ "fp8_ffn": bool(fp8_ffn),+ "fp8_block_range": (+ [fp8_start_block, fp8_end_block] if fp8_ffn else None+ ),+ "fp8_qkv": bool(fp8_qkv),+ "fp8_qkv_block_range": (+ [fp8_qkv_start_block, fp8_qkv_end_block]+ if fp8_qkv else None+ ),+ "low_noise_parameter_storage": _parameter_dtype_summary(+ self.low_noise_model),+ "high_noise_parameter_storage": _parameter_dtype_summary(+ self.high_noise_model),+ "t5_configured_dtype": str(+ config.t5_dtype).replace("torch.", ""),+ "t5_parameter_storage": _parameter_dtype_summary(+ self.text_encoder.model),+ "vae_configured_dtype": str(+ self.vae.dtype).replace("torch.", ""),+ "vae_parameter_storage": _parameter_dtype_summary(+ self.vae.model),+ }+ logging.info("Dtype report: %s", self.dtype_report)++ # ------------------------------------------------------------------+ # ckpt loading + model configuration+ # ------------------------------------------------------------------++ @staticmethod+ def _build_expert_from_config(+ checkpoint_dir: str,+ config_subfolder: str,+ expert_ckpt: str,+ tag: str,+ dtype: torch.dtype,+ fuse_self_qkv: bool,+ pdd_scheduler: FixedPDD4Scheduler | None = None,+ pdd_expert: str | None = None,+ ) -> WorldPlay2Model:+ """Initialize architecture from config and strictly load a full expert."""+ config_dir = os.path.join(checkpoint_dir, config_subfolder)+ logging.info("Creating %s from config: %s", tag, config_dir)+ model_config = WorldPlay2Model.load_config(config_dir)+ model = WorldPlay2Model.from_config(model_config)+ model.add_embedding_action_parameters()+ model.add_memory_compress_model()+ if pdd_scheduler is not None:+ model.head.make_fixed_pdd(num_blocks=2)+ # Keep the actual DiT/action/compressor parameter storage in BF16,+ # rather than relying on autocast or FSDP's transient compute cast.+ model.to(dtype=dtype)+ if pdd_scheduler is None:+ load_expert_checkpoint(model, expert_ckpt, tag=tag, strict=True)+ else:+ state = pdd_scheduler.prepare_checkpoint(+ expert_ckpt, expert=pdd_expert, dtype=dtype+ )+ provided = set(state)+ required_prefixes = (+ "memory_compress.",+ "action_in_cont.",+ "action_in_disc.",+ )+ absent = [+ prefix for prefix in required_prefixes+ if not any(key.startswith(prefix) for key in provided)+ ]+ if absent:+ raise ValueError(+ f"{tag} PDD checkpoint is missing required parameter "+ f"groups: {absent}"+ )+ model.load_state_dict(state, strict=True)+ logging.info(+ "%s fixed-PDD checkpoint: loaded=%d expert=%s",+ tag,+ len(state),+ pdd_expert,+ )+ # Install after strict checkpoint loading (which expects q/k/v keys)+ # and before FSDP wrapping (which should only see the fused weights).+ if fuse_self_qkv:+ model.fuse_self_attention_qkv()+ return model++ def _configure_model(+ self,+ model,+ use_sp,+ attention_backend,+ compile_qk_rope,+ conditioning_cache,+ compile_block_glue,+ compile_ffn,+ fp8_ffn,+ fp8_start_block,+ fp8_end_block,+ fp8_qkv,+ fp8_qkv_start_block,+ fp8_qkv_end_block,+ dit_fsdp,+ shard_fn,+ ):+ """Configure dtype/device/FSDP and enable the model-owned SP path."""+ model.eval().requires_grad_(False)+ model.to(dtype=self.param_dtype)++ model.set_sequence_parallel(use_sp)+ model.set_self_attention_backend(attention_backend)+ model.set_compiled_qk_rope(compile_qk_rope)+ model.set_conditioning_cache(conditioning_cache)+ model.set_compiled_regions(+ block_glue=compile_block_glue,+ ffn=compile_ffn,+ )++ if dist.is_initialized():+ dist.barrier(device_ids=[self.device.index])++ if fp8_ffn or fp8_qkv:+ # TE primary FP8 weights must remain resident. This path+ # intentionally does not support DiT FSDP or model offload.+ model.to(self.device)+ if fp8_qkv:+ report = convert_model_qkv_to_fp8(+ model,+ start_block=fp8_qkv_start_block,+ end_block=fp8_qkv_end_block,+ )+ logging.info("Installed self-attention QKV FP8: %s", report)+ if fp8_ffn:+ report = convert_model_ffn_to_fp8(+ model,+ start_block=fp8_start_block,+ end_block=fp8_end_block,+ )+ logging.info("Installed FFN FP8: %s", report)++ if dit_fsdp:+ model = shard_fn(model)+ else:+ if not self.init_on_cpu:+ model.to(self.device)++ return model++ @staticmethod+ def _find_wan_model(model):+ for module in model.modules():+ if isinstance(module, WorldPlay2Model):+ return module+ raise RuntimeError("FSDP/model wrapper does not contain WorldPlay2Model")++ def clear_conditioning_cache(self) -> None:+ self._find_wan_model(+ self.low_noise_model).clear_conditioning_cache()+ self._find_wan_model(+ self.high_noise_model).clear_conditioning_cache()++ def fp8_stats(self) -> dict:+ return {+ "low_noise": fp8_model_stats(+ self._find_wan_model(self.low_noise_model)),+ "high_noise": fp8_model_stats(+ self._find_wan_model(self.high_noise_model)),+ }++ def _prepare_model_for_timestep(self, t, boundary, offload_model):+ """Mirrors :meth:`WanI2V._prepare_model_for_timestep`."""+ if t.item() >= boundary:+ required_model_name = 'high_noise_model'+ offload_model_name = 'low_noise_model'+ else:+ required_model_name = 'low_noise_model'+ offload_model_name = 'high_noise_model'+ if offload_model or self.init_on_cpu:+ other = getattr(self, offload_model_name)+ active = getattr(self, required_model_name)+ if next(other.parameters()).device.type == 'cuda':+ # Cached tensors are plain inference state and are not moved by+ # Module.to(). Drop them before offloading the expert.+ self._find_wan_model(other).clear_conditioning_cache()+ other.to('cpu')+ if next(active.parameters()).device.type == 'cpu':+ active.to(self.device)+ return getattr(self, required_model_name)++ def _prepare_named_expert(self, expert: str, offload_model: bool):+ required = f"{expert}_noise_model"+ other = "low_noise_model" if expert == "high" else "high_noise_model"+ if offload_model or self.init_on_cpu:+ other_model = getattr(self, other)+ active_model = getattr(self, required)+ if next(other_model.parameters()).device.type == "cuda":+ self._find_wan_model(other_model).clear_conditioning_cache()+ other_model.to("cpu")+ if next(active_model.parameters()).device.type == "cpu":+ active_model.to(self.device)+ return getattr(self, required)++ def _run_pdd_chunk(+ self,+ *,+ latent,+ y_chunk,+ hr_action_chunk,+ context,+ memory_dict_high,+ memory_dict_low,+ kv_high,+ kv_low,+ chunk_start,+ seed_g,+ ):+ for block in self.pdd_scheduler.ordered_blocks:+ model = self._prepare_named_expert(+ block.expert,+ self.offload_model,+ )++ kv_cache = kv_high if block.expert == "high" else kv_low+ memory_dict = (+ memory_dict_high+ if block.expert == "high"+ else memory_dict_low+ )+ timestep = self.pdd_scheduler.timesteps[block.start:block.start + 1].to(self.device)+ args = dict(+ context=[context[0]],+ seq_len=self.seq_len_chunk,+ y=[y_chunk],+ hr_action=hr_action_chunk,+ forward_mode="pdd_denoise",+ pdd_block_index=block.local_index,+ kv_cache=kv_cache,+ ar_chunk_start=chunk_start,+ memory_dict=memory_dict,+ )+ output = model(+ [latent.to(self.device)],+ t=timestep,+ **args,+ )[0]+ if not isinstance(output, tuple) or len(output) != 2:+ raise RuntimeError(+ "fixed PDD model must return "+ "(displacement, endpoint_velocity)"+ )+ displacement, endpoint_velocity = output+ sigma_endpoint = self.pdd_scheduler.sigmas[block.end - 1].to(device=latent.device, dtype=torch.float32)+ state_endpoint = latent.float() + displacement.float()+ x0 = (state_endpoint - sigma_endpoint * endpoint_velocity.float()).to(latent.dtype)++ sigma_end = self.pdd_scheduler.sigmas[block.end].to(device=latent.device, dtype=torch.float32)+ if block.end == self.pdd_scheduler.num_intervals:+ latent = x0+ else:+ noise = torch.randn(x0.shape, generator=seed_g, device=x0.device, dtype=x0.dtype)+ latent = ((1.0 - sigma_end) * x0.float() + sigma_end * noise.float()).to(x0.dtype)+ return latent++ def _build_chunk_scheduler(+ self,+ sample_solver: str,+ sampling_steps: int,+ shift: float,+ ):+ if sample_solver == "unipc":+ scheduler = FlowUniPCMultistepScheduler(+ num_train_timesteps=self.num_train_timesteps,+ shift=1,+ use_dynamic_shifting=False,+ )+ scheduler.set_timesteps(+ sampling_steps,+ device=self.device,+ shift=shift,+ )+ return scheduler, scheduler.timesteps+ if sample_solver == "dpm++":+ scheduler = FlowDPMSolverMultistepScheduler(+ num_train_timesteps=self.num_train_timesteps,+ shift=1,+ use_dynamic_shifting=False,+ )+ sampling_sigmas = get_sampling_sigmas(sampling_steps, shift)+ timesteps, _ = retrieve_timesteps(+ scheduler,+ device=self.device,+ sigmas=sampling_sigmas,+ )+ return scheduler, timesteps+ raise NotImplementedError(f"Unsupported solver {sample_solver!r}")++ def _predict_chunk_noise(+ self,+ *,+ latent,+ timestep_value,+ boundary,+ context,+ context_null,+ memory_dict_high,+ memory_dict_low,+ kv_high,+ kv_high_neg,+ kv_low,+ kv_low_neg,+ y_chunk,+ hr_action_chunk,+ ar_cache,+ chunk_start,+ ):+ latent_model_input = [latent.to(self.device)]+ timestep = torch.stack([timestep_value]).to(self.device)+ model = self._prepare_model_for_timestep(+ timestep_value,+ boundary,+ self.offload_model,+ )++ if timestep_value.item() >= boundary:+ memory_dict = memory_dict_high+ kv_cache, kv_cache_neg = kv_high, kv_high_neg+ sample_guide_scale = self.guide_scale[1]+ use_cfg = self.use_cfg_high+ else:+ memory_dict = memory_dict_low+ kv_cache, kv_cache_neg = kv_low, kv_low_neg+ sample_guide_scale = self.guide_scale[0]+ use_cfg = self.use_cfg_low++ model_args = {+ "context": [context[0]],+ "seq_len": self.seq_len_chunk,+ "y": [y_chunk],+ "hr_action": hr_action_chunk,+ "forward_mode": self.denoise_forward_mode,+ }+ if ar_cache:+ model_args.update(+ kv_cache=kv_cache,+ ar_chunk_start=chunk_start,+ )+ else:+ model_args["memory_dict"] = memory_dict++ noise_pred_cond = model(+ latent_model_input,+ t=timestep,+ **model_args,+ )[0]+ if use_cfg:+ null_args = {+ "context": context_null,+ "seq_len": self.seq_len_chunk,+ "y": [y_chunk],+ "hr_action": hr_action_chunk,+ "forward_mode": self.denoise_forward_mode,+ }+ if ar_cache:+ null_args.update(+ kv_cache=kv_cache_neg,+ ar_chunk_start=chunk_start,+ )+ else:+ null_args["memory_dict"] = memory_dict+ noise_pred_uncond = model(+ latent_model_input,+ t=timestep,+ **null_args,+ )[0]+ if self.offload_model:+ torch.cuda.empty_cache()+ noise_pred_cond = noise_pred_uncond + sample_guide_scale * (+ noise_pred_cond - noise_pred_uncond+ )+ return noise_pred_cond++ def _run_solver_chunk(+ self,+ *,+ latent,+ sample_solver,+ sampling_steps,+ shift,+ seed_g,+ boundary,+ context,+ context_null,+ memory_dict_high,+ memory_dict_low,+ kv_high,+ kv_high_neg,+ kv_low,+ kv_low_neg,+ y_chunk,+ hr_action_chunk,+ ar_cache,+ chunk_start,+ chunk_index,+ ):+ scheduler, timesteps = self._build_chunk_scheduler(+ sample_solver,+ sampling_steps,+ shift,+ )+ if self.offload_model:+ torch.cuda.empty_cache()++ for timestep_value in tqdm(timesteps, desc=f"chunk {chunk_index}"):+ noise_pred = self._predict_chunk_noise(+ latent=latent,+ timestep_value=timestep_value,+ boundary=boundary,+ context=context,+ context_null=context_null,+ memory_dict_high=memory_dict_high,+ memory_dict_low=memory_dict_low,+ kv_high=kv_high,+ kv_high_neg=kv_high_neg,+ kv_low=kv_low,+ kv_low_neg=kv_low_neg,+ y_chunk=y_chunk,+ hr_action_chunk=hr_action_chunk,+ ar_cache=ar_cache,+ chunk_start=chunk_start,+ )+ latent = scheduler.step(+ noise_pred.unsqueeze(0),+ timestep_value,+ latent.unsqueeze(0),+ return_dict=False,+ generator=seed_g,+ )[0].squeeze(0)+ return latent++ def _prefill_kv_cache(+ self,+ model,+ memory_dict,+ kv_cache,+ kv_cache_neg,+ context,+ context_null,+ use_cfg,+ ):+ """Prefill conditional cache and the optional CFG cache.++ The ctx self-attn K/V pass through cross-attention (text), so the+ cond and uncond ctx K/V differ -> two separate caches. When+ ``memory_dict`` is None (first chunk) prefill is a no-op and the+ caches stay empty (denoise then runs pred-only attention).++ Returns the (possibly mutated) ``(kv_cache, kv_cache_neg)`` so the+ caller rebinds explicitly rather than relying on in-place mutation.+ """+ if memory_dict is None:+ return kv_cache, kv_cache_neg+ kv_cache = model(+ forward_mode="ar_prefill",+ context=[context[0]],+ memory_dict=memory_dict,+ kv_cache=kv_cache,+ )+ if use_cfg:+ if context_null is None or kv_cache_neg is None:+ raise ValueError("CFG prefill requires null context and cache")+ kv_cache_neg = model(+ forward_mode="ar_prefill",+ context=context_null,+ memory_dict=memory_dict,+ kv_cache=kv_cache_neg,+ )+ return kv_cache, kv_cache_neg++ # ------------------------------------------------------------------+ # streaming VAE helpers+ # ------------------------------------------------------------------++ def _decode_chunk_and_lr_encode(+ self,+ hr_latent_chunk: torch.Tensor,+ is_first_vae_chunk: bool,+ ):+ """Decode one HR latent chunk via streaming VAE, downsample+ (T::2, HW//4 in pixel space), then re-encode via streaming VAE.++ Returns (lr_latent, rgb_frames) where:+ lr_latent : [16, T_lr_lat, h//4, w//4]+ rgb_frames : [3, T_pix, H_pix, W_pix] in [-1, 1]+ """+ rgb = self.vae.decode_chunk([hr_latent_chunk],+ is_first_chunk=is_first_vae_chunk)[0]++ lr_pix = rgb[:, ::2, :, :] # T::2+ h_pix, w_pix = lr_pix.shape[2:]+ target_size = (h_pix // 4, w_pix // 4)+ lr_pix = lr_pix.permute(1, 0, 2, 3).contiguous() # [T, C, H, W]+ lr_pix = F.interpolate(lr_pix, size=target_size, mode='bilinear',+ align_corners=False)+ lr_pix = lr_pix.permute(1, 0, 2, 3).contiguous() # [C, T, H, W]++ lr_lat = self.vae.encode_chunk([lr_pix],+ is_first_chunk=is_first_vae_chunk)[0]+ return lr_lat, rgb++ def _build_lr_image_pixel(self, img_pixel_hr: torch.Tensor,+ h_hr_pix: int, w_hr_pix: int) -> torch.Tensor:+ """Bicubic-downsample the HR pixel image to the LR resolution+ ``(h_hr_pix//4, w_hr_pix//4)`` -- matches the (H//4, W//4) target+ used by :meth:`_decode_chunk_and_lr_encode`. Returns ``[3, H_lr, W_lr]``+ on ``self.device``.+ """+ return F.interpolate(+ img_pixel_hr[None].cpu(),+ size=(h_hr_pix // 4, w_hr_pix // 4),+ mode='bilinear')[0].to(self.device)++ def _prepare_chunk_memory(+ self,+ *,+ chunk_index,+ chunk_end,+ latents,+ hr_y_full,+ lr_y_full,+ hr_action,+ lr_action,+ context,+ context_null,+ kv_high,+ kv_high_neg,+ kv_low,+ kv_low_neg,+ ):+ """Decode a generated chunk and prepare memory for the next chunk."""+ chunk_start = chunk_end - self.chunk_length+ generated_latents = latents[:, chunk_start:chunk_end].contiguous()+ new_lr_latents, decoded_rgb = self._decode_chunk_and_lr_encode(+ generated_latents,+ is_first_vae_chunk=chunk_index == 0,+ )+ if self.lr_latents_cache is None:+ self.lr_latents_cache = new_lr_latents+ else:+ self.lr_latents_cache = torch.cat(+ [self.lr_latents_cache, new_lr_latents],+ dim=1,+ )++ hr_context_latents = latents[:, :chunk_end].contiguous()+ hr_context_y = hr_y_full[:, :chunk_end]+ lr_context_y = lr_y_full[:, :chunk_end // 2]+ hr_input = torch.cat([hr_context_latents, hr_context_y], dim=0).to(dtype=self.param_dtype).unsqueeze(0)+ lr_input = torch.cat(+ [self.lr_latents_cache, lr_context_y],+ dim=0,+ ).to(dtype=self.param_dtype).unsqueeze(0)++ temporal_latents = hr_context_latents[:, -self.temporal_size:].to(dtype=self.param_dtype)+ temporal_y = hr_y_full[:, chunk_end - self.temporal_size:chunk_end].to(dtype=self.param_dtype)+ temporal_input = torch.cat([temporal_latents, temporal_y], dim=0).unsqueeze(0)++ compressor_args = {+ "forward_mode": "compress",+ "hr_latents_input": hr_input,+ "lr_latents_input": lr_input,+ "lr_action": lr_action[:chunk_end // 2],+ "memory_t": torch.zeros(1, device=self.device, dtype=self.param_dtype),+ "hr_temporal_latents_input": temporal_input,+ "hr_temporal_action": hr_action[chunk_end - self.temporal_size:chunk_end],+ "sink_action": (+ hr_action[:self.sink_size]+ if self.sink_size > 0+ else None+ ),+ }++ if self.offload_model:+ if next(self.high_noise_model.parameters()).device.type != "cuda":+ self.high_noise_model.to(self.device)+ if next(self.low_noise_model.parameters()).device.type == "cuda":+ self.low_noise_model.to("cpu")+ memory_dict_high = self.high_noise_model(**compressor_args)+ if self.ar_cache:+ kv_high, kv_high_neg = self._prefill_kv_cache(+ self.high_noise_model,+ memory_dict_high,+ kv_high,+ kv_high_neg,+ context=context,+ context_null=context_null,+ use_cfg=self.use_cfg_high,+ )++ if self.offload_model:+ if next(self.low_noise_model.parameters()).device.type != "cuda":+ self.low_noise_model.to(self.device)+ if next(self.high_noise_model.parameters()).device.type == "cuda":+ self.high_noise_model.to("cpu")+ memory_dict_low = self.low_noise_model(**compressor_args)+ if self.ar_cache:+ kv_low, kv_low_neg = self._prefill_kv_cache(+ self.low_noise_model,+ memory_dict_low,+ kv_low,+ kv_low_neg,+ context=context,+ context_null=context_null,+ use_cfg=self.use_cfg_low,+ )++ return (+ decoded_rgb.cpu(),+ memory_dict_high,+ memory_dict_low,+ kv_high,+ kv_high_neg,+ kv_low,+ kv_low_neg,+ )++ # ------------------------------------------------------------------+ # public entry: chunked autoregressive generation+ # ------------------------------------------------------------------++ def generate_chunked(+ self,+ prompts: dict[str, str],+ chunk_prompts: tuple[str, ...],+ img,+ hr_action: torch.Tensor,+ total_latent_frames: int = 16,+ chunk_length: int = 4,+ shift: float = 5.0,+ sample_solver: str = 'unipc',+ sampling_steps: int = 40,+ guide_scale=5.0,+ n_prompt: str = "",+ seed: int = -1,+ offload_model: bool = True,+ sink_size: int = 0,+ temporal_size: int = 2,+ height: int = 448,+ width: int = 832,+ ):+ """Run stage-3 chunked inference. See module docstring for+ the recipe. Returns ``[3, T_pix, H_pix, W_pix]`` on rank 0,+ ``None`` on other ranks.++ ``sink_size`` (default 0): number of leading hr latent frames used as+ the context "sink" (``hr_latents_input[:, :, :sink_size]``, prepended+ to the front of the context). 0 disables the sink.++ ``temporal_size`` (default 2): number of trailing hr latent frames+ used as the temporal anchor (``hr_ctx_lat[:, -temporal_size:]``),+ appended after the compressed lr+hr context.+ """+ assert total_latent_frames % chunk_length == 0, (+ f"total_latent_frames ({total_latent_frames}) must be a multiple "+ f"of chunk_length ({chunk_length})")+ num_chunks = total_latent_frames // chunk_length+ if len(chunk_prompts) != num_chunks:+ raise ValueError(+ f"chunk_prompts must contain {num_chunks} entries, "+ f"got {len(chunk_prompts)}"+ )+ # Conditioning is sample-local. Never carry projected text or+ # cross-attention K/V into the next JSON item.+ self.clear_conditioning_cache()++ if isinstance(guide_scale, (int, float)):+ guide_scale = (float(guide_scale), float(guide_scale))+ else:+ guide_scale = tuple(float(scale) for scale in guide_scale)+ if len(guide_scale) != 2:+ raise ValueError(+ "guide_scale must be a scalar or (low_noise, high_noise)"+ )+ use_cfg_low = not math.isclose(+ guide_scale[0], 1.0, rel_tol=0.0, abs_tol=1e-8)+ use_cfg_high = not math.isclose(+ guide_scale[1], 1.0, rel_tol=0.0, abs_tol=1e-8)+ use_any_cfg = use_cfg_low or use_cfg_high+ if self.inference_mode is InferenceMode.PDD and use_any_cfg:+ raise ValueError("PDD consistency inference requires guide_scale=1")+ self.chunk_length = chunk_length+ self.temporal_size = temporal_size+ self.sink_size = sink_size+ self.offload_model = offload_model+ self.ar_cache = (+ self.denoise_forward_mode in {"ar_denoise", "pdd_denoise"}+ )+ self.guide_scale = guide_scale+ self.use_cfg_high = use_cfg_high+ self.use_cfg_low = use_cfg_low+ self.lr_latents_cache = None+ logging.info(+ "CFG enabled: low_noise=%s high_noise=%s",+ use_cfg_low,+ use_cfg_high,+ )++ # ---- preprocess image ----+ img = TF.to_tensor(img).sub_(0.5).div_(0.5).to(self.device)+ h, w = height, width+ lat_h = h // self.vae_stride[1] # 56+ lat_w = w // self.vae_stride[2] # 104+ img_pixel = F.interpolate(img[None].cpu(), size=(h, w),+ mode='bicubic')[0].to(self.device)++ # ---- text encode ----+ if n_prompt == "":+ n_prompt = self.sample_neg_prompt+ prompt_names = tuple(dict.fromkeys(chunk_prompts))+ prompt_texts = [prompts[name] for name in prompt_names]+ encoded_prompts = self.text_encoder(prompt_texts, self.device)+ context_null = (+ self.text_encoder([n_prompt], self.device)+ if use_any_cfg else None+ )+ prompt_contexts = {+ name: [encoded]+ for name, encoded in zip(prompt_names, encoded_prompts)+ }++ # ---- pre-build hr_y_full / lr_y_full (upstream WanI2V recipe:+ # encode the full pixel clip [image, zeros..., zeros] in one shot+ # so the temporally-causal VAE produces the correct latent volume).+ with torch.no_grad():+ hr_y_full = _build_y(+ self.vae, img_pixel,+ t_lat=total_latent_frames,+ h_pix=h, w_pix=w,+ device=self.device, dtype=self.param_dtype)++ img_pixel_lr = self._build_lr_image_pixel(img_pixel, h, w)+ lr_y_full = _build_y(+ self.vae, img_pixel_lr,+ t_lat=total_latent_frames // 2,+ h_pix=h // 4, w_pix=w // 4,+ device=self.device, dtype=self.param_dtype)++ # ---- noise allocation for the whole HR latent volume ----+ seed = seed if seed >= 0 else random.randint(0, sys.maxsize)+ seed_g = torch.Generator(device=self.device)+ seed_g.manual_seed(seed)+ noise_full = torch.randn(+ 16, total_latent_frames, lat_h, lat_w,+ dtype=torch.float32, generator=seed_g, device=self.device)+ latents = noise_full.clone()++ # ---- hr_action sanity ----+ hr_action = hr_action.to(self.device)+ assert hr_action.shape[0] >= total_latent_frames, (+ f"hr_action must have >= total_latent_frames ({total_latent_frames}) "+ f"rows; got {hr_action.shape[0]}.")+ lr_action = hr_action[::2][:total_latent_frames // 2]++ decoded_rgb_cache: list = []++ # Chunk-AR always keeps conditional caches. Negative caches are only+ # allocated for experts whose guide scale actually enables CFG.+ ar_cache = self.ar_cache+ # PERSISTENT KV caches, created ONCE and reused across all chunks.+ # The unified prefill path is incremental: each chunk only pushes the+ # new cmp block + recomputed tmp anchor. Each layer stores one+ # contiguous [sink | accumulated cmp | latest tmp] K/V pair instead+ # of re-prefilling the whole growing ctx. When AR but no KV cache,+ # ar_cache=False and the denoise falls back to the fused [ctx|pred] AR+ # path (memory_dict). BI leaves them None.+ if ar_cache:+ kv_high = self.high_noise_model.init_kv_cache()+ kv_low = self.low_noise_model.init_kv_cache()+ kv_high_neg = (+ self.high_noise_model.init_kv_cache()+ if use_cfg_high else None+ )+ kv_low_neg = (+ self.low_noise_model.init_kv_cache()+ if use_cfg_low else None+ )+ else:+ kv_high = kv_high_neg = kv_low = kv_low_neg = None+ memory_dict_high = None+ memory_dict_low = None+ boundary = self.boundary * self.num_train_timesteps+ no_sync_low = getattr(self.low_noise_model, 'no_sync', _noop_no_sync)+ no_sync_high = getattr(self.high_noise_model, 'no_sync', _noop_no_sync)++ with (torch.amp.autocast('cuda', dtype=self.param_dtype),+ torch.no_grad(), no_sync_low(), no_sync_high()):++ # Exact gen token count -- no rounding to a multiple of sp_size.+ # The unified SP forwards (BI + AR) chunk the sequence UNEVENLY+ # (uneven all_to_all) and attend with SDPA over the true+ # sequence, so any extra zero-padded rows would leak into the+ # attention (BI: as real tokens; AR: into the chunk-causal mask).+ seq_len_chunk = chunk_length * lat_h * lat_w // (+ self.patch_size[1] * self.patch_size[2])+ self.seq_len_chunk = seq_len_chunk++ for chunk_i in range(num_chunks):+ s = chunk_i * chunk_length+ e = (chunk_i + 1) * chunk_length+ prompt_name = chunk_prompts[chunk_i]+ context = prompt_contexts[prompt_name]++ logging.info(f"[stage3] chunk {chunk_i}/{num_chunks - 1} "+ f"latent frames [{s}, {e}) prompt={prompt_name}")++ # Per-chunk denoise inputs (also needed by AR prefill below).+ latent = latents[:, s:e].contiguous()+ hr_action_chunk = hr_action[s:e]+ y_chunk = hr_y_full[:, s:e]++ chunk_denoise_args = dict(+ latent=latent,+ seed_g=seed_g,+ boundary=boundary,+ context=context,+ context_null=context_null,+ memory_dict_high=memory_dict_high,+ memory_dict_low=memory_dict_low,+ kv_high=kv_high,+ kv_high_neg=kv_high_neg,+ kv_low=kv_low,+ kv_low_neg=kv_low_neg,+ y_chunk=y_chunk,+ hr_action_chunk=hr_action_chunk,+ chunk_start=s,+ )++ if self.inference_mode is InferenceMode.PDD:+ latent = self._run_pdd_chunk(+ latent=latent,+ y_chunk=y_chunk,+ hr_action_chunk=hr_action_chunk,+ context=context,+ memory_dict_high=memory_dict_high,+ memory_dict_low=memory_dict_low,+ kv_high=kv_high,+ kv_low=kv_low,+ chunk_start=s,+ seed_g=seed_g,+ )+ elif self.inference_mode is InferenceMode.BI:+ latent = self._run_solver_chunk(+ sample_solver=sample_solver,+ sampling_steps=sampling_steps,+ shift=shift,+ ar_cache=False,+ chunk_index=chunk_i,+ **chunk_denoise_args,+ )+ elif self.inference_mode is InferenceMode.AR:+ latent = self._run_solver_chunk(+ sample_solver=sample_solver,+ sampling_steps=sampling_steps,+ shift=shift,+ ar_cache=True,+ chunk_index=chunk_i,+ **chunk_denoise_args,+ )+ else:+ raise RuntimeError(+ f"Unsupported inference mode: {self.inference_mode}"+ )+ latents[:, s:e] = latent.to(latents.dtype)++ if chunk_i < num_chunks - 1:+ (+ decoded_rgb,+ memory_dict_high,+ memory_dict_low,+ kv_high,+ kv_high_neg,+ kv_low,+ kv_low_neg,+ ) = self._prepare_chunk_memory(+ chunk_index=chunk_i,+ chunk_end=e,+ latents=latents,+ hr_y_full=hr_y_full,+ lr_y_full=lr_y_full,+ hr_action=hr_action,+ lr_action=lr_action,+ context=context,+ context_null=context_null,+ kv_high=kv_high,+ kv_high_neg=kv_high_neg,+ kv_low=kv_low,+ kv_low_neg=kv_low_neg,+ )+ decoded_rgb_cache.append(decoded_rgb)+ elif self.rank == 0:+ final_rgb = self.vae.decode_chunk(+ [latent],+ is_first_chunk=chunk_i == 0,+ )[0].cpu()+ decoded_rgb_cache.append(final_rgb)++ if self.rank == 0:+ video = torch.cat(decoded_rgb_cache, dim=1)+ else:+ video = None++ if offload_model:+ self.clear_conditioning_cache()+ self.low_noise_model.cpu()+ self.high_noise_model.cpu()+ torch.cuda.empty_cache()++ if offload_model:+ gc.collect()+ torch.cuda.synchronize()+ if dist.is_initialized():+ dist.barrier(device_ids=[self.device.index])++ return videodiff --git a/worldplay2/schedulers/__init__.py b/worldplay2/schedulers/__init__.pynew file mode 100644index 0000000..4d446c5--- /dev/null+++ b/worldplay2/schedulers/__init__.py@@ -0,0 +1,13 @@+from .dpm_solver import (+ FlowDPMSolverMultistepScheduler,+ get_sampling_sigmas,+ retrieve_timesteps,+)+from .unipc import FlowUniPCMultistepScheduler++__all__ = [+ "FlowDPMSolverMultistepScheduler",+ "FlowUniPCMultistepScheduler",+ "get_sampling_sigmas",+ "retrieve_timesteps",+]diff --git a/worldplay2/schedulers/dpm_solver.py b/worldplay2/schedulers/dpm_solver.pynew file mode 100644index 0000000..17bef85--- /dev/null+++ b/worldplay2/schedulers/dpm_solver.py@@ -0,0 +1,859 @@+# Copied from https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py+# Convert dpm solver for flow matching+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.++import inspect+import math+from typing import List, Optional, Tuple, Union++import numpy as np+import torch+from diffusers.configuration_utils import ConfigMixin, register_to_config+from diffusers.schedulers.scheduling_utils import (+ KarrasDiffusionSchedulers,+ SchedulerMixin,+ SchedulerOutput,+)+from diffusers.utils import deprecate, is_scipy_available+from diffusers.utils.torch_utils import randn_tensor++if is_scipy_available():+ pass+++def get_sampling_sigmas(sampling_steps, shift):+ sigma = np.linspace(1, 0, sampling_steps + 1)[:sampling_steps]+ sigma = (shift * sigma / (1 + (shift - 1) * sigma))++ return sigma+++def retrieve_timesteps(+ scheduler,+ num_inference_steps=None,+ device=None,+ timesteps=None,+ sigmas=None,+ **kwargs,+):+ if timesteps is not None and sigmas is not None:+ raise ValueError(+ "Only one of `timesteps` or `sigmas` can be passed. Please choose one to set custom values"+ )+ if timesteps is not None:+ accepts_timesteps = "timesteps" in set(+ inspect.signature(scheduler.set_timesteps).parameters.keys())+ if not accepts_timesteps:+ raise ValueError(+ f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom"+ f" timestep schedules. Please check whether you are using the correct scheduler."+ )+ scheduler.set_timesteps(timesteps=timesteps, device=device, **kwargs)+ timesteps = scheduler.timesteps+ num_inference_steps = len(timesteps)+ elif sigmas is not None:+ accept_sigmas = "sigmas" in set(+ inspect.signature(scheduler.set_timesteps).parameters.keys())+ if not accept_sigmas:+ raise ValueError(+ f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom"+ f" sigmas schedules. Please check whether you are using the correct scheduler."+ )+ scheduler.set_timesteps(sigmas=sigmas, device=device, **kwargs)+ timesteps = scheduler.timesteps+ num_inference_steps = len(timesteps)+ else:+ scheduler.set_timesteps(num_inference_steps, device=device, **kwargs)+ timesteps = scheduler.timesteps+ return timesteps, num_inference_steps+++class FlowDPMSolverMultistepScheduler(SchedulerMixin, ConfigMixin):+ """+ `FlowDPMSolverMultistepScheduler` is a fast dedicated high-order solver for diffusion ODEs.+ This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic+ methods the library implements for all schedulers such as loading and saving.+ Args:+ num_train_timesteps (`int`, defaults to 1000):+ The number of diffusion steps to train the model. This determines the resolution of the diffusion process.+ solver_order (`int`, defaults to 2):+ The DPMSolver order which can be `1`, `2`, or `3`. It is recommended to use `solver_order=2` for guided+ sampling, and `solver_order=3` for unconditional sampling. This affects the number of model outputs stored+ and used in multistep updates.+ prediction_type (`str`, defaults to "flow_prediction"):+ Prediction type of the scheduler function; must be `flow_prediction` for this scheduler, which predicts+ the flow of the diffusion process.+ shift (`float`, *optional*, defaults to 1.0):+ A factor used to adjust the sigmas in the noise schedule. It modifies the step sizes during the sampling+ process.+ use_dynamic_shifting (`bool`, defaults to `False`):+ Whether to apply dynamic shifting to the timesteps based on image resolution. If `True`, the shifting is+ applied on the fly.+ thresholding (`bool`, defaults to `False`):+ Whether to use the "dynamic thresholding" method. This method adjusts the predicted sample to prevent+ saturation and improve photorealism.+ dynamic_thresholding_ratio (`float`, defaults to 0.995):+ The ratio for the dynamic thresholding method. Valid only when `thresholding=True`.+ sample_max_value (`float`, defaults to 1.0):+ The threshold value for dynamic thresholding. Valid only when `thresholding=True` and+ `algorithm_type="dpmsolver++"`.+ algorithm_type (`str`, defaults to `dpmsolver++`):+ Algorithm type for the solver; can be `dpmsolver`, `dpmsolver++`, `sde-dpmsolver` or `sde-dpmsolver++`. The+ `dpmsolver` type implements the algorithms in the [DPMSolver](https://huggingface.co/papers/2206.00927)+ paper, and the `dpmsolver++` type implements the algorithms in the+ [DPMSolver++](https://huggingface.co/papers/2211.01095) paper. It is recommended to use `dpmsolver++` or+ `sde-dpmsolver++` with `solver_order=2` for guided sampling like in Stable Diffusion.+ solver_type (`str`, defaults to `midpoint`):+ Solver type for the second-order solver; can be `midpoint` or `heun`. The solver type slightly affects the+ sample quality, especially for a small number of steps. It is recommended to use `midpoint` solvers.+ lower_order_final (`bool`, defaults to `True`):+ Whether to use lower-order solvers in the final steps. Only valid for < 15 inference steps. This can+ stabilize the sampling of DPMSolver for steps < 15, especially for steps <= 10.+ euler_at_final (`bool`, defaults to `False`):+ Whether to use Euler's method in the final step. It is a trade-off between numerical stability and detail+ richness. This can stabilize the sampling of the SDE variant of DPMSolver for small number of inference+ steps, but sometimes may result in blurring.+ final_sigmas_type (`str`, *optional*, defaults to "zero"):+ The final `sigma` value for the noise schedule during the sampling process. If `"sigma_min"`, the final+ sigma is the same as the last sigma in the training schedule. If `zero`, the final sigma is set to 0.+ lambda_min_clipped (`float`, defaults to `-inf`):+ Clipping threshold for the minimum value of `lambda(t)` for numerical stability. This is critical for the+ cosine (`squaredcos_cap_v2`) noise schedule.+ variance_type (`str`, *optional*):+ Set to "learned" or "learned_range" for diffusion models that predict variance. If set, the model's output+ contains the predicted Gaussian variance.+ """++ _compatibles = [e.name for e in KarrasDiffusionSchedulers]+ order = 1++ @register_to_config+ def __init__(+ self,+ num_train_timesteps: int = 1000,+ solver_order: int = 2,+ prediction_type: str = "flow_prediction",+ shift: Optional[float] = 1.0,+ use_dynamic_shifting=False,+ thresholding: bool = False,+ dynamic_thresholding_ratio: float = 0.995,+ sample_max_value: float = 1.0,+ algorithm_type: str = "dpmsolver++",+ solver_type: str = "midpoint",+ lower_order_final: bool = True,+ euler_at_final: bool = False,+ final_sigmas_type: Optional[str] = "zero", # "zero", "sigma_min"+ lambda_min_clipped: float = -float("inf"),+ variance_type: Optional[str] = None,+ invert_sigmas: bool = False,+ ):+ if algorithm_type in ["dpmsolver", "sde-dpmsolver"]:+ deprecation_message = f"algorithm_type {algorithm_type} is deprecated and will be removed in a future version. Choose from `dpmsolver++` or `sde-dpmsolver++` instead"+ deprecate("algorithm_types dpmsolver and sde-dpmsolver", "1.0.0",+ deprecation_message)++ # settings for DPM-Solver+ if algorithm_type not in [+ "dpmsolver", "dpmsolver++", "sde-dpmsolver", "sde-dpmsolver++"+ ]:+ if algorithm_type == "deis":+ self.register_to_config(algorithm_type="dpmsolver++")+ else:+ raise NotImplementedError(+ f"{algorithm_type} is not implemented for {self.__class__}")++ if solver_type not in ["midpoint", "heun"]:+ if solver_type in ["logrho", "bh1", "bh2"]:+ self.register_to_config(solver_type="midpoint")+ else:+ raise NotImplementedError(+ f"{solver_type} is not implemented for {self.__class__}")++ if algorithm_type not in ["dpmsolver++", "sde-dpmsolver++"+ ] and final_sigmas_type == "zero":+ raise ValueError(+ f"`final_sigmas_type` {final_sigmas_type} is not supported for `algorithm_type` {algorithm_type}. Please choose `sigma_min` instead."+ )++ # setable values+ self.num_inference_steps = None+ alphas = np.linspace(1, 1 / num_train_timesteps,+ num_train_timesteps)[::-1].copy()+ sigmas = 1.0 - alphas+ sigmas = torch.from_numpy(sigmas).to(dtype=torch.float32)++ if not use_dynamic_shifting:+ # when use_dynamic_shifting is True, we apply the timestep shifting on the fly based on the image resolution+ sigmas = shift * sigmas / (1 ++ (shift - 1) * sigmas) # pyright: ignore++ self.sigmas = sigmas+ self.timesteps = sigmas * num_train_timesteps++ self.model_outputs = [None] * solver_order+ self.lower_order_nums = 0+ self._step_index = None+ self._begin_index = None++ # self.sigmas = self.sigmas.to(+ # "cpu") # to avoid too much CPU/GPU communication+ self.sigma_min = self.sigmas[-1].item()+ self.sigma_max = self.sigmas[0].item()++ @property+ def step_index(self):+ """+ The index counter for current timestep. It will increase 1 after each scheduler step.+ """+ return self._step_index++ @property+ def begin_index(self):+ """+ The index for the first timestep. It should be set from pipeline with `set_begin_index` method.+ """+ return self._begin_index++ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.set_begin_index+ def set_begin_index(self, begin_index: int = 0):+ """+ Sets the begin index for the scheduler. This function should be run from pipeline before the inference.+ Args:+ begin_index (`int`):+ The begin index for the scheduler.+ """+ self._begin_index = begin_index++ # Modified from diffusers.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler.set_timesteps+ def set_timesteps(+ self,+ num_inference_steps: Union[int, None] = None,+ device: Union[str, torch.device] = None,+ sigmas: Optional[List[float]] = None,+ mu: Optional[Union[float, None]] = None,+ shift: Optional[Union[float, None]] = None,+ ):+ """+ Sets the discrete timesteps used for the diffusion chain (to be run before inference).+ Args:+ num_inference_steps (`int`):+ Total number of the spacing of the time steps.+ device (`str` or `torch.device`, *optional*):+ The device to which the timesteps should be moved to. If `None`, the timesteps are not moved.+ """++ if self.config.use_dynamic_shifting and mu is None:+ raise ValueError(+ " you have to pass a value for `mu` when `use_dynamic_shifting` is set to be `True`"+ )++ if sigmas is None:+ sigmas = np.linspace(self.sigma_max, self.sigma_min,+ num_inference_steps ++ 1).copy()[:-1] # pyright: ignore++ if self.config.use_dynamic_shifting:+ sigmas = self.time_shift(mu, 1.0, sigmas) # pyright: ignore+ else:+ if shift is None:+ shift = self.config.shift+ sigmas = shift * sigmas / (1 ++ (shift - 1) * sigmas) # pyright: ignore++ if self.config.final_sigmas_type == "sigma_min":+ sigma_last = ((1 - self.alphas_cumprod[0]) /+ self.alphas_cumprod[0])**0.5+ elif self.config.final_sigmas_type == "zero":+ sigma_last = 0+ else:+ raise ValueError(+ f"`final_sigmas_type` must be one of 'zero', or 'sigma_min', but got {self.config.final_sigmas_type}"+ )++ timesteps = sigmas * self.config.num_train_timesteps+ sigmas = np.concatenate([sigmas, [sigma_last]+ ]).astype(np.float32) # pyright: ignore++ self.sigmas = torch.from_numpy(sigmas)+ self.timesteps = torch.from_numpy(timesteps).to(+ device=device, dtype=torch.int64)++ self.num_inference_steps = len(timesteps)++ self.model_outputs = [+ None,+ ] * self.config.solver_order+ self.lower_order_nums = 0++ self._step_index = None+ self._begin_index = None+ # self.sigmas = self.sigmas.to(+ # "cpu") # to avoid too much CPU/GPU communication++ # Copied from diffusers.schedulers.scheduling_ddpm.DDPMScheduler._threshold_sample+ def _threshold_sample(self, sample: torch.Tensor) -> torch.Tensor:+ """+ "Dynamic thresholding: At each sampling step we set s to a certain percentile absolute pixel value in xt0 (the+ prediction of x_0 at timestep t), and if s > 1, then we threshold xt0 to the range [-s, s] and then divide by+ s. Dynamic thresholding pushes saturated pixels (those near -1 and 1) inwards, thereby actively preventing+ pixels from saturation at each step. We find that dynamic thresholding results in significantly better+ photorealism as well as better image-text alignment, especially when using very large guidance weights."+ https://arxiv.org/abs/2205.11487+ """+ dtype = sample.dtype+ batch_size, channels, *remaining_dims = sample.shape++ if dtype not in (torch.float32, torch.float64):+ sample = sample.float(+ ) # upcast for quantile calculation, and clamp not implemented for cpu half++ # Flatten sample for doing quantile calculation along each image+ sample = sample.reshape(batch_size, channels * np.prod(remaining_dims))++ abs_sample = sample.abs() # "a certain percentile absolute pixel value"++ s = torch.quantile(+ abs_sample, self.config.dynamic_thresholding_ratio, dim=1)+ s = torch.clamp(+ s, min=1, max=self.config.sample_max_value+ ) # When clamped to min=1, equivalent to standard clipping to [-1, 1]+ s = s.unsqueeze(+ 1) # (batch_size, 1) because clamp will broadcast along dim=0+ sample = torch.clamp(+ sample, -s, s+ ) / s # "we threshold xt0 to the range [-s, s] and then divide by s"++ sample = sample.reshape(batch_size, channels, *remaining_dims)+ sample = sample.to(dtype)++ return sample++ # Copied from diffusers.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler._sigma_to_t+ def _sigma_to_t(self, sigma):+ return sigma * self.config.num_train_timesteps++ def _sigma_to_alpha_sigma_t(self, sigma):+ return 1 - sigma, sigma++ # Copied from diffusers.schedulers.scheduling_flow_match_euler_discrete.set_timesteps+ def time_shift(self, mu: float, sigma: float, t: torch.Tensor):+ return math.exp(mu) / (math.exp(mu) + (1 / t - 1)**sigma)++ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.convert_model_output+ def convert_model_output(+ self,+ model_output: torch.Tensor,+ *args,+ sample: torch.Tensor = None,+ **kwargs,+ ) -> torch.Tensor:+ """+ Convert the model output to the corresponding type the DPMSolver/DPMSolver++ algorithm needs. DPM-Solver is+ designed to discretize an integral of the noise prediction model, and DPM-Solver++ is designed to discretize an+ integral of the data prediction model.+ <Tip>+ The algorithm and model type are decoupled. You can use either DPMSolver or DPMSolver++ for both noise+ prediction and data prediction models.+ </Tip>+ Args:+ model_output (`torch.Tensor`):+ The direct output from the learned diffusion model.+ sample (`torch.Tensor`):+ A current instance of a sample created by the diffusion process.+ Returns:+ `torch.Tensor`:+ The converted model output.+ """+ timestep = args[0] if len(args) > 0 else kwargs.pop("timestep", None)+ if sample is None:+ if len(args) > 1:+ sample = args[1]+ else:+ raise ValueError(+ "missing `sample` as a required keyward argument")+ if timestep is not None:+ deprecate(+ "timesteps",+ "1.0.0",+ "Passing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",+ )++ # DPM-Solver++ needs to solve an integral of the data prediction model.+ if self.config.algorithm_type in ["dpmsolver++", "sde-dpmsolver++"]:+ if self.config.prediction_type == "flow_prediction":+ sigma_t = self.sigmas[self.step_index]+ x0_pred = sample - sigma_t * model_output+ else:+ raise ValueError(+ f"prediction_type given as {self.config.prediction_type} must be one of `epsilon`, `sample`,"+ " `v_prediction`, or `flow_prediction` for the FlowDPMSolverMultistepScheduler."+ )++ if self.config.thresholding:+ x0_pred = self._threshold_sample(x0_pred)++ return x0_pred++ # DPM-Solver needs to solve an integral of the noise prediction model.+ elif self.config.algorithm_type in ["dpmsolver", "sde-dpmsolver"]:+ if self.config.prediction_type == "flow_prediction":+ sigma_t = self.sigmas[self.step_index]+ epsilon = sample - (1 - sigma_t) * model_output+ else:+ raise ValueError(+ f"prediction_type given as {self.config.prediction_type} must be one of `epsilon`, `sample`,"+ " `v_prediction` or `flow_prediction` for the FlowDPMSolverMultistepScheduler."+ )++ if self.config.thresholding:+ sigma_t = self.sigmas[self.step_index]+ x0_pred = sample - sigma_t * model_output+ x0_pred = self._threshold_sample(x0_pred)+ epsilon = model_output + x0_pred++ return epsilon++ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.dpm_solver_first_order_update+ def dpm_solver_first_order_update(+ self,+ model_output: torch.Tensor,+ *args,+ sample: torch.Tensor = None,+ noise: Optional[torch.Tensor] = None,+ **kwargs,+ ) -> torch.Tensor:+ """+ One step for the first-order DPMSolver (equivalent to DDIM).+ Args:+ model_output (`torch.Tensor`):+ The direct output from the learned diffusion model.+ sample (`torch.Tensor`):+ A current instance of a sample created by the diffusion process.+ Returns:+ `torch.Tensor`:+ The sample tensor at the previous timestep.+ """+ timestep = args[0] if len(args) > 0 else kwargs.pop("timestep", None)+ prev_timestep = args[1] if len(args) > 1 else kwargs.pop(+ "prev_timestep", None)+ if sample is None:+ if len(args) > 2:+ sample = args[2]+ else:+ raise ValueError(+ " missing `sample` as a required keyward argument")+ if timestep is not None:+ deprecate(+ "timesteps",+ "1.0.0",+ "Passing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",+ )++ if prev_timestep is not None:+ deprecate(+ "prev_timestep",+ "1.0.0",+ "Passing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",+ )++ sigma_t, sigma_s = self.sigmas[self.step_index + 1], self.sigmas[+ self.step_index] # pyright: ignore+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma_t)+ alpha_s, sigma_s = self._sigma_to_alpha_sigma_t(sigma_s)+ lambda_t = torch.log(alpha_t) - torch.log(sigma_t)+ lambda_s = torch.log(alpha_s) - torch.log(sigma_s)++ h = lambda_t - lambda_s+ if self.config.algorithm_type == "dpmsolver++":+ x_t = (sigma_t /+ sigma_s) * sample - (alpha_t *+ (torch.exp(-h) - 1.0)) * model_output+ elif self.config.algorithm_type == "dpmsolver":+ x_t = (alpha_t /+ alpha_s) * sample - (sigma_t *+ (torch.exp(h) - 1.0)) * model_output+ elif self.config.algorithm_type == "sde-dpmsolver++":+ assert noise is not None+ x_t = ((sigma_t / sigma_s * torch.exp(-h)) * sample ++ (alpha_t * (1 - torch.exp(-2.0 * h))) * model_output ++ sigma_t * torch.sqrt(1.0 - torch.exp(-2 * h)) * noise)+ elif self.config.algorithm_type == "sde-dpmsolver":+ assert noise is not None+ x_t = ((alpha_t / alpha_s) * sample - 2.0 *+ (sigma_t * (torch.exp(h) - 1.0)) * model_output ++ sigma_t * torch.sqrt(torch.exp(2 * h) - 1.0) * noise)+ return x_t # pyright: ignore++ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.multistep_dpm_solver_second_order_update+ def multistep_dpm_solver_second_order_update(+ self,+ model_output_list: List[torch.Tensor],+ *args,+ sample: torch.Tensor = None,+ noise: Optional[torch.Tensor] = None,+ **kwargs,+ ) -> torch.Tensor:+ """+ One step for the second-order multistep DPMSolver.+ Args:+ model_output_list (`List[torch.Tensor]`):+ The direct outputs from learned diffusion model at current and latter timesteps.+ sample (`torch.Tensor`):+ A current instance of a sample created by the diffusion process.+ Returns:+ `torch.Tensor`:+ The sample tensor at the previous timestep.+ """+ timestep_list = args[0] if len(args) > 0 else kwargs.pop(+ "timestep_list", None)+ prev_timestep = args[1] if len(args) > 1 else kwargs.pop(+ "prev_timestep", None)+ if sample is None:+ if len(args) > 2:+ sample = args[2]+ else:+ raise ValueError(+ " missing `sample` as a required keyward argument")+ if timestep_list is not None:+ deprecate(+ "timestep_list",+ "1.0.0",+ "Passing `timestep_list` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",+ )++ if prev_timestep is not None:+ deprecate(+ "prev_timestep",+ "1.0.0",+ "Passing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",+ )++ sigma_t, sigma_s0, sigma_s1 = (+ self.sigmas[self.step_index + 1], # pyright: ignore+ self.sigmas[self.step_index],+ self.sigmas[self.step_index - 1], # pyright: ignore+ )++ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma_t)+ alpha_s0, sigma_s0 = self._sigma_to_alpha_sigma_t(sigma_s0)+ alpha_s1, sigma_s1 = self._sigma_to_alpha_sigma_t(sigma_s1)++ lambda_t = torch.log(alpha_t) - torch.log(sigma_t)+ lambda_s0 = torch.log(alpha_s0) - torch.log(sigma_s0)+ lambda_s1 = torch.log(alpha_s1) - torch.log(sigma_s1)++ m0, m1 = model_output_list[-1], model_output_list[-2]++ h, h_0 = lambda_t - lambda_s0, lambda_s0 - lambda_s1+ r0 = h_0 / h+ D0, D1 = m0, (1.0 / r0) * (m0 - m1)+ if self.config.algorithm_type == "dpmsolver++":+ # See https://arxiv.org/abs/2211.01095 for detailed derivations+ if self.config.solver_type == "midpoint":+ x_t = ((sigma_t / sigma_s0) * sample -+ (alpha_t * (torch.exp(-h) - 1.0)) * D0 - 0.5 *+ (alpha_t * (torch.exp(-h) - 1.0)) * D1)+ elif self.config.solver_type == "heun":+ x_t = ((sigma_t / sigma_s0) * sample -+ (alpha_t * (torch.exp(-h) - 1.0)) * D0 ++ (alpha_t * ((torch.exp(-h) - 1.0) / h + 1.0)) * D1)+ elif self.config.algorithm_type == "dpmsolver":+ # See https://arxiv.org/abs/2206.00927 for detailed derivations+ if self.config.solver_type == "midpoint":+ x_t = ((alpha_t / alpha_s0) * sample -+ (sigma_t * (torch.exp(h) - 1.0)) * D0 - 0.5 *+ (sigma_t * (torch.exp(h) - 1.0)) * D1)+ elif self.config.solver_type == "heun":+ x_t = ((alpha_t / alpha_s0) * sample -+ (sigma_t * (torch.exp(h) - 1.0)) * D0 -+ (sigma_t * ((torch.exp(h) - 1.0) / h - 1.0)) * D1)+ elif self.config.algorithm_type == "sde-dpmsolver++":+ assert noise is not None+ if self.config.solver_type == "midpoint":+ x_t = ((sigma_t / sigma_s0 * torch.exp(-h)) * sample ++ (alpha_t * (1 - torch.exp(-2.0 * h))) * D0 + 0.5 *+ (alpha_t * (1 - torch.exp(-2.0 * h))) * D1 ++ sigma_t * torch.sqrt(1.0 - torch.exp(-2 * h)) * noise)+ elif self.config.solver_type == "heun":+ x_t = ((sigma_t / sigma_s0 * torch.exp(-h)) * sample ++ (alpha_t * (1 - torch.exp(-2.0 * h))) * D0 ++ (alpha_t * ((1.0 - torch.exp(-2.0 * h)) /+ (-2.0 * h) + 1.0)) * D1 ++ sigma_t * torch.sqrt(1.0 - torch.exp(-2 * h)) * noise)+ elif self.config.algorithm_type == "sde-dpmsolver":+ assert noise is not None+ if self.config.solver_type == "midpoint":+ x_t = ((alpha_t / alpha_s0) * sample - 2.0 *+ (sigma_t * (torch.exp(h) - 1.0)) * D0 -+ (sigma_t * (torch.exp(h) - 1.0)) * D1 ++ sigma_t * torch.sqrt(torch.exp(2 * h) - 1.0) * noise)+ elif self.config.solver_type == "heun":+ x_t = ((alpha_t / alpha_s0) * sample - 2.0 *+ (sigma_t * (torch.exp(h) - 1.0)) * D0 - 2.0 *+ (sigma_t * ((torch.exp(h) - 1.0) / h - 1.0)) * D1 ++ sigma_t * torch.sqrt(torch.exp(2 * h) - 1.0) * noise)+ return x_t # pyright: ignore++ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.multistep_dpm_solver_third_order_update+ def multistep_dpm_solver_third_order_update(+ self,+ model_output_list: List[torch.Tensor],+ *args,+ sample: torch.Tensor = None,+ **kwargs,+ ) -> torch.Tensor:+ """+ One step for the third-order multistep DPMSolver.+ Args:+ model_output_list (`List[torch.Tensor]`):+ The direct outputs from learned diffusion model at current and latter timesteps.+ sample (`torch.Tensor`):+ A current instance of a sample created by diffusion process.+ Returns:+ `torch.Tensor`:+ The sample tensor at the previous timestep.+ """++ timestep_list = args[0] if len(args) > 0 else kwargs.pop(+ "timestep_list", None)+ prev_timestep = args[1] if len(args) > 1 else kwargs.pop(+ "prev_timestep", None)+ if sample is None:+ if len(args) > 2:+ sample = args[2]+ else:+ raise ValueError(+ " missing`sample` as a required keyward argument")+ if timestep_list is not None:+ deprecate(+ "timestep_list",+ "1.0.0",+ "Passing `timestep_list` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",+ )++ if prev_timestep is not None:+ deprecate(+ "prev_timestep",+ "1.0.0",+ "Passing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",+ )++ sigma_t, sigma_s0, sigma_s1, sigma_s2 = (+ self.sigmas[self.step_index + 1], # pyright: ignore+ self.sigmas[self.step_index],+ self.sigmas[self.step_index - 1], # pyright: ignore+ self.sigmas[self.step_index - 2], # pyright: ignore+ )++ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma_t)+ alpha_s0, sigma_s0 = self._sigma_to_alpha_sigma_t(sigma_s0)+ alpha_s1, sigma_s1 = self._sigma_to_alpha_sigma_t(sigma_s1)+ alpha_s2, sigma_s2 = self._sigma_to_alpha_sigma_t(sigma_s2)++ lambda_t = torch.log(alpha_t) - torch.log(sigma_t)+ lambda_s0 = torch.log(alpha_s0) - torch.log(sigma_s0)+ lambda_s1 = torch.log(alpha_s1) - torch.log(sigma_s1)+ lambda_s2 = torch.log(alpha_s2) - torch.log(sigma_s2)++ m0, m1, m2 = model_output_list[-1], model_output_list[+ -2], model_output_list[-3]++ h, h_0, h_1 = lambda_t - lambda_s0, lambda_s0 - lambda_s1, lambda_s1 - lambda_s2+ r0, r1 = h_0 / h, h_1 / h+ D0 = m0+ D1_0, D1_1 = (1.0 / r0) * (m0 - m1), (1.0 / r1) * (m1 - m2)+ D1 = D1_0 + (r0 / (r0 + r1)) * (D1_0 - D1_1)+ D2 = (1.0 / (r0 + r1)) * (D1_0 - D1_1)+ if self.config.algorithm_type == "dpmsolver++":+ # See https://arxiv.org/abs/2206.00927 for detailed derivations+ x_t = ((sigma_t / sigma_s0) * sample -+ (alpha_t * (torch.exp(-h) - 1.0)) * D0 ++ (alpha_t * ((torch.exp(-h) - 1.0) / h + 1.0)) * D1 -+ (alpha_t * ((torch.exp(-h) - 1.0 + h) / h**2 - 0.5)) * D2)+ elif self.config.algorithm_type == "dpmsolver":+ # See https://arxiv.org/abs/2206.00927 for detailed derivations+ x_t = ((alpha_t / alpha_s0) * sample - (sigma_t *+ (torch.exp(h) - 1.0)) * D0 -+ (sigma_t * ((torch.exp(h) - 1.0) / h - 1.0)) * D1 -+ (sigma_t * ((torch.exp(h) - 1.0 - h) / h**2 - 0.5)) * D2)+ return x_t # pyright: ignore++ def index_for_timestep(self, timestep, schedule_timesteps=None):+ if schedule_timesteps is None:+ schedule_timesteps = self.timesteps++ indices = (schedule_timesteps == timestep).nonzero()++ # The sigma index that is taken for the **very** first `step`+ # is always the second index (or the last index if there is only 1)+ # This way we can ensure we don't accidentally skip a sigma in+ # case we start in the middle of the denoising schedule (e.g. for image-to-image)+ pos = 1 if len(indices) > 1 else 0++ return indices[pos].item()++ def _init_step_index(self, timestep):+ """+ Initialize the step_index counter for the scheduler.+ """++ if self.begin_index is None:+ if isinstance(timestep, torch.Tensor):+ timestep = timestep.to(self.timesteps.device)+ self._step_index = self.index_for_timestep(timestep)+ else:+ self._step_index = self._begin_index++ # Modified from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.step+ def step(+ self,+ model_output: torch.Tensor,+ timestep: Union[int, torch.Tensor],+ sample: torch.Tensor,+ generator=None,+ variance_noise: Optional[torch.Tensor] = None,+ return_dict: bool = True,+ ) -> Union[SchedulerOutput, Tuple]:+ """+ Predict the sample from the previous timestep by reversing the SDE. This function propagates the sample with+ the multistep DPMSolver.+ Args:+ model_output (`torch.Tensor`):+ The direct output from learned diffusion model.+ timestep (`int`):+ The current discrete timestep in the diffusion chain.+ sample (`torch.Tensor`):+ A current instance of a sample created by the diffusion process.+ generator (`torch.Generator`, *optional*):+ A random number generator.+ variance_noise (`torch.Tensor`):+ Alternative to generating noise with `generator` by directly providing the noise for the variance+ itself. Useful for methods such as [`LEdits++`].+ return_dict (`bool`):+ Whether or not to return a [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`.+ Returns:+ [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`:+ If return_dict is `True`, [`~schedulers.scheduling_utils.SchedulerOutput`] is returned, otherwise a+ tuple is returned where the first element is the sample tensor.+ """+ if self.num_inference_steps is None:+ raise ValueError(+ "Number of inference steps is 'None', you need to run 'set_timesteps' after creating the scheduler"+ )++ if self.step_index is None:+ self._init_step_index(timestep)++ # Improve numerical stability for small number of steps+ lower_order_final = (self.step_index == len(self.timesteps) - 1) and (+ self.config.euler_at_final or+ (self.config.lower_order_final and len(self.timesteps) < 15) or+ self.config.final_sigmas_type == "zero")+ lower_order_second = ((self.step_index == len(self.timesteps) - 2) and+ self.config.lower_order_final and+ len(self.timesteps) < 15)++ model_output = self.convert_model_output(model_output, sample=sample)+ for i in range(self.config.solver_order - 1):+ self.model_outputs[i] = self.model_outputs[i + 1]+ self.model_outputs[-1] = model_output++ # Upcast to avoid precision issues when computing prev_sample+ sample = sample.to(torch.float32)+ if self.config.algorithm_type in ["sde-dpmsolver", "sde-dpmsolver++"+ ] and variance_noise is None:+ noise = randn_tensor(+ model_output.shape,+ generator=generator,+ device=model_output.device,+ dtype=torch.float32)+ elif self.config.algorithm_type in ["sde-dpmsolver", "sde-dpmsolver++"]:+ noise = variance_noise.to(+ device=model_output.device,+ dtype=torch.float32) # pyright: ignore+ else:+ noise = None++ if self.config.solver_order == 1 or self.lower_order_nums < 1 or lower_order_final:+ prev_sample = self.dpm_solver_first_order_update(+ model_output, sample=sample, noise=noise)+ elif self.config.solver_order == 2 or self.lower_order_nums < 2 or lower_order_second:+ prev_sample = self.multistep_dpm_solver_second_order_update(+ self.model_outputs, sample=sample, noise=noise)+ else:+ prev_sample = self.multistep_dpm_solver_third_order_update(+ self.model_outputs, sample=sample)++ if self.lower_order_nums < self.config.solver_order:+ self.lower_order_nums += 1++ # Cast sample back to expected dtype+ prev_sample = prev_sample.to(model_output.dtype)++ # upon completion increase step index by one+ self._step_index += 1 # pyright: ignore++ if not return_dict:+ return (prev_sample,)++ return SchedulerOutput(prev_sample=prev_sample)++ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.scale_model_input+ def scale_model_input(self, sample: torch.Tensor, *args,+ **kwargs) -> torch.Tensor:+ """+ Ensures interchangeability with schedulers that need to scale the denoising model input depending on the+ current timestep.+ Args:+ sample (`torch.Tensor`):+ The input sample.+ Returns:+ `torch.Tensor`:+ A scaled input sample.+ """+ return sample++ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.scale_model_input+ def add_noise(+ self,+ original_samples: torch.Tensor,+ noise: torch.Tensor,+ timesteps: torch.IntTensor,+ ) -> torch.Tensor:+ # Make sure sigmas and timesteps have the same device and dtype as original_samples+ sigmas = self.sigmas.to(+ device=original_samples.device, dtype=original_samples.dtype)+ if original_samples.device.type == "mps" and torch.is_floating_point(+ timesteps):+ # mps does not support float64+ schedule_timesteps = self.timesteps.to(+ original_samples.device, dtype=torch.float32)+ timesteps = timesteps.to(+ original_samples.device, dtype=torch.float32)+ else:+ schedule_timesteps = self.timesteps.to(original_samples.device)+ timesteps = timesteps.to(original_samples.device)++ # begin_index is None when the scheduler is used for training or pipeline does not implement set_begin_index+ if self.begin_index is None:+ step_indices = [+ self.index_for_timestep(t, schedule_timesteps)+ for t in timesteps+ ]+ elif self.step_index is not None:+ # add_noise is called after first denoising step (for inpainting)+ step_indices = [self.step_index] * timesteps.shape[0]+ else:+ # add noise is called before first denoising step to create initial latent(img2img)+ step_indices = [self.begin_index] * timesteps.shape[0]++ sigma = sigmas[step_indices].flatten()+ while len(sigma.shape) < len(original_samples.shape):+ sigma = sigma.unsqueeze(-1)++ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma)+ noisy_samples = alpha_t * original_samples + sigma_t * noise+ return noisy_samples++ def __len__(self):+ return self.config.num_train_timestepsdiff --git a/worldplay2/schedulers/unipc.py b/worldplay2/schedulers/unipc.pynew file mode 100644index 0000000..fb502f2--- /dev/null+++ b/worldplay2/schedulers/unipc.py@@ -0,0 +1,802 @@+# Copied from https://github.com/huggingface/diffusers/blob/v0.31.0/src/diffusers/schedulers/scheduling_unipc_multistep.py+# Convert unipc for flow matching+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.++import math+from typing import List, Optional, Tuple, Union++import numpy as np+import torch+from diffusers.configuration_utils import ConfigMixin, register_to_config+from diffusers.schedulers.scheduling_utils import (+ KarrasDiffusionSchedulers,+ SchedulerMixin,+ SchedulerOutput,+)+from diffusers.utils import deprecate, is_scipy_available++if is_scipy_available():+ import scipy.stats+++class FlowUniPCMultistepScheduler(SchedulerMixin, ConfigMixin):+ """+ `UniPCMultistepScheduler` is a training-free framework designed for the fast sampling of diffusion models.++ This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic+ methods the library implements for all schedulers such as loading and saving.++ Args:+ num_train_timesteps (`int`, defaults to 1000):+ The number of diffusion steps to train the model.+ solver_order (`int`, default `2`):+ The UniPC order which can be any positive integer. The effective order of accuracy is `solver_order + 1`+ due to the UniC. It is recommended to use `solver_order=2` for guided sampling, and `solver_order=3` for+ unconditional sampling.+ prediction_type (`str`, defaults to "flow_prediction"):+ Prediction type of the scheduler function; must be `flow_prediction` for this scheduler, which predicts+ the flow of the diffusion process.+ thresholding (`bool`, defaults to `False`):+ Whether to use the "dynamic thresholding" method. This is unsuitable for latent-space diffusion models such+ as Stable Diffusion.+ dynamic_thresholding_ratio (`float`, defaults to 0.995):+ The ratio for the dynamic thresholding method. Valid only when `thresholding=True`.+ sample_max_value (`float`, defaults to 1.0):+ The threshold value for dynamic thresholding. Valid only when `thresholding=True` and `predict_x0=True`.+ predict_x0 (`bool`, defaults to `True`):+ Whether to use the updating algorithm on the predicted x0.+ solver_type (`str`, default `bh2`):+ Solver type for UniPC. It is recommended to use `bh1` for unconditional sampling when steps < 10, and `bh2`+ otherwise.+ lower_order_final (`bool`, default `True`):+ Whether to use lower-order solvers in the final steps. Only valid for < 15 inference steps. This can+ stabilize the sampling of DPMSolver for steps < 15, especially for steps <= 10.+ disable_corrector (`list`, default `[]`):+ Decides which step to disable the corrector to mitigate the misalignment between `epsilon_theta(x_t, c)`+ and `epsilon_theta(x_t^c, c)` which can influence convergence for a large guidance scale. Corrector is+ usually disabled during the first few steps.+ solver_p (`SchedulerMixin`, default `None`):+ Any other scheduler that if specified, the algorithm becomes `solver_p + UniC`.+ use_karras_sigmas (`bool`, *optional*, defaults to `False`):+ Whether to use Karras sigmas for step sizes in the noise schedule during the sampling process. If `True`,+ the sigmas are determined according to a sequence of noise levels {σi}.+ use_exponential_sigmas (`bool`, *optional*, defaults to `False`):+ Whether to use exponential sigmas for step sizes in the noise schedule during the sampling process.+ timestep_spacing (`str`, defaults to `"linspace"`):+ The way the timesteps should be scaled. Refer to Table 2 of the [Common Diffusion Noise Schedules and+ Sample Steps are Flawed](https://huggingface.co/papers/2305.08891) for more information.+ steps_offset (`int`, defaults to 0):+ An offset added to the inference steps, as required by some model families.+ final_sigmas_type (`str`, defaults to `"zero"`):+ The final `sigma` value for the noise schedule during the sampling process. If `"sigma_min"`, the final+ sigma is the same as the last sigma in the training schedule. If `zero`, the final sigma is set to 0.+ """++ _compatibles = [e.name for e in KarrasDiffusionSchedulers]+ order = 1++ @register_to_config+ def __init__(+ self,+ num_train_timesteps: int = 1000,+ solver_order: int = 2,+ prediction_type: str = "flow_prediction",+ shift: Optional[float] = 1.0,+ use_dynamic_shifting=False,+ thresholding: bool = False,+ dynamic_thresholding_ratio: float = 0.995,+ sample_max_value: float = 1.0,+ predict_x0: bool = True,+ solver_type: str = "bh2",+ lower_order_final: bool = True,+ disable_corrector: List[int] = [],+ solver_p: SchedulerMixin = None,+ timestep_spacing: str = "linspace",+ steps_offset: int = 0,+ final_sigmas_type: Optional[str] = "zero", # "zero", "sigma_min"+ ):++ if solver_type not in ["bh1", "bh2"]:+ if solver_type in ["midpoint", "heun", "logrho"]:+ self.register_to_config(solver_type="bh2")+ else:+ raise NotImplementedError(+ f"{solver_type} is not implemented for {self.__class__}")++ self.predict_x0 = predict_x0+ # setable values+ self.num_inference_steps = None+ alphas = np.linspace(1, 1 / num_train_timesteps,+ num_train_timesteps)[::-1].copy()+ sigmas = 1.0 - alphas+ sigmas = torch.from_numpy(sigmas).to(dtype=torch.float32)++ if not use_dynamic_shifting:+ # when use_dynamic_shifting is True, we apply the timestep shifting on the fly based on the image resolution+ sigmas = shift * sigmas / (1 ++ (shift - 1) * sigmas) # pyright: ignore++ self.sigmas = sigmas+ self.timesteps = sigmas * num_train_timesteps++ self.model_outputs = [None] * solver_order+ self.timestep_list = [None] * solver_order+ self.lower_order_nums = 0+ self.disable_corrector = disable_corrector+ self.solver_p = solver_p+ self.last_sample = None+ self._step_index = None+ self._begin_index = None++ self.sigmas = self.sigmas.to(+ "cpu") # to avoid too much CPU/GPU communication+ self.sigma_min = self.sigmas[-1].item()+ self.sigma_max = self.sigmas[0].item()++ @property+ def step_index(self):+ """+ The index counter for current timestep. It will increase 1 after each scheduler step.+ """+ return self._step_index++ @property+ def begin_index(self):+ """+ The index for the first timestep. It should be set from pipeline with `set_begin_index` method.+ """+ return self._begin_index++ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.set_begin_index+ def set_begin_index(self, begin_index: int = 0):+ """+ Sets the begin index for the scheduler. This function should be run from pipeline before the inference.++ Args:+ begin_index (`int`):+ The begin index for the scheduler.+ """+ self._begin_index = begin_index++ # Modified from diffusers.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler.set_timesteps+ def set_timesteps(+ self,+ num_inference_steps: Union[int, None] = None,+ device: Union[str, torch.device] = None,+ sigmas: Optional[List[float]] = None,+ mu: Optional[Union[float, None]] = None,+ shift: Optional[Union[float, None]] = None,+ ):+ """+ Sets the discrete timesteps used for the diffusion chain (to be run before inference).+ Args:+ num_inference_steps (`int`):+ Total number of the spacing of the time steps.+ device (`str` or `torch.device`, *optional*):+ The device to which the timesteps should be moved to. If `None`, the timesteps are not moved.+ """++ if self.config.use_dynamic_shifting and mu is None:+ raise ValueError(+ " you have to pass a value for `mu` when `use_dynamic_shifting` is set to be `True`"+ )++ if sigmas is None:+ sigmas = np.linspace(self.sigma_max, self.sigma_min,+ num_inference_steps ++ 1).copy()[:-1] # pyright: ignore++ if self.config.use_dynamic_shifting:+ sigmas = self.time_shift(mu, 1.0, sigmas) # pyright: ignore+ else:+ if shift is None:+ shift = self.config.shift+ sigmas = shift * sigmas / (1 ++ (shift - 1) * sigmas) # pyright: ignore++ if self.config.final_sigmas_type == "sigma_min":+ sigma_last = ((1 - self.alphas_cumprod[0]) /+ self.alphas_cumprod[0])**0.5+ elif self.config.final_sigmas_type == "zero":+ sigma_last = 0+ else:+ raise ValueError(+ f"`final_sigmas_type` must be one of 'zero', or 'sigma_min', but got {self.config.final_sigmas_type}"+ )++ timesteps = sigmas * self.config.num_train_timesteps+ sigmas = np.concatenate([sigmas, [sigma_last]+ ]).astype(np.float32) # pyright: ignore++ self.sigmas = torch.from_numpy(sigmas)+ self.timesteps = torch.from_numpy(timesteps).to(+ device=device, dtype=torch.int64)++ self.num_inference_steps = len(timesteps)++ self.model_outputs = [+ None,+ ] * self.config.solver_order+ self.lower_order_nums = 0+ self.last_sample = None+ if self.solver_p:+ self.solver_p.set_timesteps(self.num_inference_steps, device=device)++ # add an index counter for schedulers that allow duplicated timesteps+ self._step_index = None+ self._begin_index = None+ self.sigmas = self.sigmas.to(+ "cpu") # to avoid too much CPU/GPU communication++ # Copied from diffusers.schedulers.scheduling_ddpm.DDPMScheduler._threshold_sample+ def _threshold_sample(self, sample: torch.Tensor) -> torch.Tensor:+ """+ "Dynamic thresholding: At each sampling step we set s to a certain percentile absolute pixel value in xt0 (the+ prediction of x_0 at timestep t), and if s > 1, then we threshold xt0 to the range [-s, s] and then divide by+ s. Dynamic thresholding pushes saturated pixels (those near -1 and 1) inwards, thereby actively preventing+ pixels from saturation at each step. We find that dynamic thresholding results in significantly better+ photorealism as well as better image-text alignment, especially when using very large guidance weights."++ https://arxiv.org/abs/2205.11487+ """+ dtype = sample.dtype+ batch_size, channels, *remaining_dims = sample.shape++ if dtype not in (torch.float32, torch.float64):+ sample = sample.float(+ ) # upcast for quantile calculation, and clamp not implemented for cpu half++ # Flatten sample for doing quantile calculation along each image+ sample = sample.reshape(batch_size, channels * np.prod(remaining_dims))++ abs_sample = sample.abs() # "a certain percentile absolute pixel value"++ s = torch.quantile(+ abs_sample, self.config.dynamic_thresholding_ratio, dim=1)+ s = torch.clamp(+ s, min=1, max=self.config.sample_max_value+ ) # When clamped to min=1, equivalent to standard clipping to [-1, 1]+ s = s.unsqueeze(+ 1) # (batch_size, 1) because clamp will broadcast along dim=0+ sample = torch.clamp(+ sample, -s, s+ ) / s # "we threshold xt0 to the range [-s, s] and then divide by s"++ sample = sample.reshape(batch_size, channels, *remaining_dims)+ sample = sample.to(dtype)++ return sample++ # Copied from diffusers.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler._sigma_to_t+ def _sigma_to_t(self, sigma):+ return sigma * self.config.num_train_timesteps++ def _sigma_to_alpha_sigma_t(self, sigma):+ return 1 - sigma, sigma++ # Copied from diffusers.schedulers.scheduling_flow_match_euler_discrete.set_timesteps+ def time_shift(self, mu: float, sigma: float, t: torch.Tensor):+ return math.exp(mu) / (math.exp(mu) + (1 / t - 1)**sigma)++ def convert_model_output(+ self,+ model_output: torch.Tensor,+ *args,+ sample: torch.Tensor = None,+ **kwargs,+ ) -> torch.Tensor:+ r"""+ Convert the model output to the corresponding type the UniPC algorithm needs.++ Args:+ model_output (`torch.Tensor`):+ The direct output from the learned diffusion model.+ timestep (`int`):+ The current discrete timestep in the diffusion chain.+ sample (`torch.Tensor`):+ A current instance of a sample created by the diffusion process.++ Returns:+ `torch.Tensor`:+ The converted model output.+ """+ timestep = args[0] if len(args) > 0 else kwargs.pop("timestep", None)+ if sample is None:+ if len(args) > 1:+ sample = args[1]+ else:+ raise ValueError(+ "missing `sample` as a required keyward argument")+ if timestep is not None:+ deprecate(+ "timesteps",+ "1.0.0",+ "Passing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",+ )++ sigma = self.sigmas[self.step_index]+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma)++ if self.predict_x0:+ if self.config.prediction_type == "flow_prediction":+ sigma_t = self.sigmas[self.step_index]+ x0_pred = sample - sigma_t * model_output+ else:+ raise ValueError(+ f"prediction_type given as {self.config.prediction_type} must be one of `epsilon`, `sample`,"+ " `v_prediction` or `flow_prediction` for the UniPCMultistepScheduler."+ )++ if self.config.thresholding:+ x0_pred = self._threshold_sample(x0_pred)++ return x0_pred+ else:+ if self.config.prediction_type == "flow_prediction":+ sigma_t = self.sigmas[self.step_index]+ epsilon = sample - (1 - sigma_t) * model_output+ else:+ raise ValueError(+ f"prediction_type given as {self.config.prediction_type} must be one of `epsilon`, `sample`,"+ " `v_prediction` or `flow_prediction` for the UniPCMultistepScheduler."+ )++ if self.config.thresholding:+ sigma_t = self.sigmas[self.step_index]+ x0_pred = sample - sigma_t * model_output+ x0_pred = self._threshold_sample(x0_pred)+ epsilon = model_output + x0_pred++ return epsilon++ def multistep_uni_p_bh_update(+ self,+ model_output: torch.Tensor,+ *args,+ sample: torch.Tensor = None,+ order: int = None, # pyright: ignore+ **kwargs,+ ) -> torch.Tensor:+ """+ One step for the UniP (B(h) version). Alternatively, `self.solver_p` is used if is specified.++ Args:+ model_output (`torch.Tensor`):+ The direct output from the learned diffusion model at the current timestep.+ prev_timestep (`int`):+ The previous discrete timestep in the diffusion chain.+ sample (`torch.Tensor`):+ A current instance of a sample created by the diffusion process.+ order (`int`):+ The order of UniP at this timestep (corresponds to the *p* in UniPC-p).++ Returns:+ `torch.Tensor`:+ The sample tensor at the previous timestep.+ """+ prev_timestep = args[0] if len(args) > 0 else kwargs.pop(+ "prev_timestep", None)+ if sample is None:+ if len(args) > 1:+ sample = args[1]+ else:+ raise ValueError(+ " missing `sample` as a required keyward argument")+ if order is None:+ if len(args) > 2:+ order = args[2]+ else:+ raise ValueError(+ " missing `order` as a required keyward argument")+ if prev_timestep is not None:+ deprecate(+ "prev_timestep",+ "1.0.0",+ "Passing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",+ )+ model_output_list = self.model_outputs++ s0 = self.timestep_list[-1]+ m0 = model_output_list[-1]+ x = sample++ if self.solver_p:+ x_t = self.solver_p.step(model_output, s0, x).prev_sample+ return x_t++ sigma_t, sigma_s0 = self.sigmas[self.step_index + 1], self.sigmas[+ self.step_index] # pyright: ignore+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma_t)+ alpha_s0, sigma_s0 = self._sigma_to_alpha_sigma_t(sigma_s0)++ lambda_t = torch.log(alpha_t) - torch.log(sigma_t)+ lambda_s0 = torch.log(alpha_s0) - torch.log(sigma_s0)++ h = lambda_t - lambda_s0+ device = sample.device++ rks = []+ D1s = []+ for i in range(1, order):+ si = self.step_index - i # pyright: ignore+ mi = model_output_list[-(i + 1)]+ alpha_si, sigma_si = self._sigma_to_alpha_sigma_t(self.sigmas[si])+ lambda_si = torch.log(alpha_si) - torch.log(sigma_si)+ rk = (lambda_si - lambda_s0) / h+ rks.append(rk)+ D1s.append((mi - m0) / rk) # pyright: ignore++ rks.append(1.0)+ rks = torch.tensor(rks, device=device)++ R = []+ b = []++ hh = -h if self.predict_x0 else h+ h_phi_1 = torch.expm1(hh) # h\phi_1(h) = e^h - 1+ h_phi_k = h_phi_1 / hh - 1++ factorial_i = 1++ if self.config.solver_type == "bh1":+ B_h = hh+ elif self.config.solver_type == "bh2":+ B_h = torch.expm1(hh)+ else:+ raise NotImplementedError()++ for i in range(1, order + 1):+ R.append(torch.pow(rks, i - 1))+ b.append(h_phi_k * factorial_i / B_h)+ factorial_i *= i + 1+ h_phi_k = h_phi_k / hh - 1 / factorial_i++ R = torch.stack(R)+ b = torch.tensor(b, device=device)++ if len(D1s) > 0:+ D1s = torch.stack(D1s, dim=1) # (B, K)+ # for order 2, we use a simplified version+ if order == 2:+ rhos_p = torch.tensor([0.5], dtype=x.dtype, device=device)+ else:+ rhos_p = torch.linalg.solve(R[:-1, :-1],+ b[:-1]).to(device).to(x.dtype)+ else:+ D1s = None++ if self.predict_x0:+ x_t_ = sigma_t / sigma_s0 * x - alpha_t * h_phi_1 * m0+ if D1s is not None:+ pred_res = torch.einsum("k,bkc...->bc...", rhos_p,+ D1s) # pyright: ignore+ else:+ pred_res = 0+ x_t = x_t_ - alpha_t * B_h * pred_res+ else:+ x_t_ = alpha_t / alpha_s0 * x - sigma_t * h_phi_1 * m0+ if D1s is not None:+ pred_res = torch.einsum("k,bkc...->bc...", rhos_p,+ D1s) # pyright: ignore+ else:+ pred_res = 0+ x_t = x_t_ - sigma_t * B_h * pred_res++ x_t = x_t.to(x.dtype)+ return x_t++ def multistep_uni_c_bh_update(+ self,+ this_model_output: torch.Tensor,+ *args,+ last_sample: torch.Tensor = None,+ this_sample: torch.Tensor = None,+ order: int = None, # pyright: ignore+ **kwargs,+ ) -> torch.Tensor:+ """+ One step for the UniC (B(h) version).++ Args:+ this_model_output (`torch.Tensor`):+ The model outputs at `x_t`.+ this_timestep (`int`):+ The current timestep `t`.+ last_sample (`torch.Tensor`):+ The generated sample before the last predictor `x_{t-1}`.+ this_sample (`torch.Tensor`):+ The generated sample after the last predictor `x_{t}`.+ order (`int`):+ The `p` of UniC-p at this step. The effective order of accuracy should be `order + 1`.++ Returns:+ `torch.Tensor`:+ The corrected sample tensor at the current timestep.+ """+ this_timestep = args[0] if len(args) > 0 else kwargs.pop(+ "this_timestep", None)+ if last_sample is None:+ if len(args) > 1:+ last_sample = args[1]+ else:+ raise ValueError(+ " missing`last_sample` as a required keyward argument")+ if this_sample is None:+ if len(args) > 2:+ this_sample = args[2]+ else:+ raise ValueError(+ " missing`this_sample` as a required keyward argument")+ if order is None:+ if len(args) > 3:+ order = args[3]+ else:+ raise ValueError(+ " missing`order` as a required keyward argument")+ if this_timestep is not None:+ deprecate(+ "this_timestep",+ "1.0.0",+ "Passing `this_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`",+ )++ model_output_list = self.model_outputs++ m0 = model_output_list[-1]+ x = last_sample+ x_t = this_sample+ model_t = this_model_output++ sigma_t, sigma_s0 = self.sigmas[self.step_index], self.sigmas[+ self.step_index - 1] # pyright: ignore+ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma_t)+ alpha_s0, sigma_s0 = self._sigma_to_alpha_sigma_t(sigma_s0)++ lambda_t = torch.log(alpha_t) - torch.log(sigma_t)+ lambda_s0 = torch.log(alpha_s0) - torch.log(sigma_s0)++ h = lambda_t - lambda_s0+ device = this_sample.device++ rks = []+ D1s = []+ for i in range(1, order):+ si = self.step_index - (i + 1) # pyright: ignore+ mi = model_output_list[-(i + 1)]+ alpha_si, sigma_si = self._sigma_to_alpha_sigma_t(self.sigmas[si])+ lambda_si = torch.log(alpha_si) - torch.log(sigma_si)+ rk = (lambda_si - lambda_s0) / h+ rks.append(rk)+ D1s.append((mi - m0) / rk) # pyright: ignore++ rks.append(1.0)+ rks = torch.tensor(rks, device=device)++ R = []+ b = []++ hh = -h if self.predict_x0 else h+ h_phi_1 = torch.expm1(hh) # h\phi_1(h) = e^h - 1+ h_phi_k = h_phi_1 / hh - 1++ factorial_i = 1++ if self.config.solver_type == "bh1":+ B_h = hh+ elif self.config.solver_type == "bh2":+ B_h = torch.expm1(hh)+ else:+ raise NotImplementedError()++ for i in range(1, order + 1):+ R.append(torch.pow(rks, i - 1))+ b.append(h_phi_k * factorial_i / B_h)+ factorial_i *= i + 1+ h_phi_k = h_phi_k / hh - 1 / factorial_i++ R = torch.stack(R)+ b = torch.tensor(b, device=device)++ if len(D1s) > 0:+ D1s = torch.stack(D1s, dim=1)+ else:+ D1s = None++ # for order 1, we use a simplified version+ if order == 1:+ rhos_c = torch.tensor([0.5], dtype=x.dtype, device=device)+ else:+ rhos_c = torch.linalg.solve(R, b).to(device).to(x.dtype)++ if self.predict_x0:+ x_t_ = sigma_t / sigma_s0 * x - alpha_t * h_phi_1 * m0+ if D1s is not None:+ corr_res = torch.einsum("k,bkc...->bc...", rhos_c[:-1], D1s)+ else:+ corr_res = 0+ D1_t = model_t - m0+ x_t = x_t_ - alpha_t * B_h * (corr_res + rhos_c[-1] * D1_t)+ else:+ x_t_ = alpha_t / alpha_s0 * x - sigma_t * h_phi_1 * m0+ if D1s is not None:+ corr_res = torch.einsum("k,bkc...->bc...", rhos_c[:-1], D1s)+ else:+ corr_res = 0+ D1_t = model_t - m0+ x_t = x_t_ - sigma_t * B_h * (corr_res + rhos_c[-1] * D1_t)+ x_t = x_t.to(x.dtype)+ return x_t++ def index_for_timestep(self, timestep, schedule_timesteps=None):+ if schedule_timesteps is None:+ schedule_timesteps = self.timesteps++ indices = (schedule_timesteps == timestep).nonzero()++ # The sigma index that is taken for the **very** first `step`+ # is always the second index (or the last index if there is only 1)+ # This way we can ensure we don't accidentally skip a sigma in+ # case we start in the middle of the denoising schedule (e.g. for image-to-image)+ pos = 1 if len(indices) > 1 else 0++ return indices[pos].item()++ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler._init_step_index+ def _init_step_index(self, timestep):+ """+ Initialize the step_index counter for the scheduler.+ """++ if self.begin_index is None:+ if isinstance(timestep, torch.Tensor):+ timestep = timestep.to(self.timesteps.device)+ self._step_index = self.index_for_timestep(timestep)+ else:+ self._step_index = self._begin_index++ def step(self,+ model_output: torch.Tensor,+ timestep: Union[int, torch.Tensor],+ sample: torch.Tensor,+ return_dict: bool = True,+ generator=None) -> Union[SchedulerOutput, Tuple]:+ """+ Predict the sample from the previous timestep by reversing the SDE. This function propagates the sample with+ the multistep UniPC.++ Args:+ model_output (`torch.Tensor`):+ The direct output from learned diffusion model.+ timestep (`int`):+ The current discrete timestep in the diffusion chain.+ sample (`torch.Tensor`):+ A current instance of a sample created by the diffusion process.+ return_dict (`bool`):+ Whether or not to return a [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`.++ Returns:+ [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`:+ If return_dict is `True`, [`~schedulers.scheduling_utils.SchedulerOutput`] is returned, otherwise a+ tuple is returned where the first element is the sample tensor.++ """+ if self.num_inference_steps is None:+ raise ValueError(+ "Number of inference steps is 'None', you need to run 'set_timesteps' after creating the scheduler"+ )++ if self.step_index is None:+ self._init_step_index(timestep)++ use_corrector = (+ self.step_index > 0 and+ self.step_index - 1 not in self.disable_corrector and+ self.last_sample is not None # pyright: ignore+ )++ model_output_convert = self.convert_model_output(+ model_output, sample=sample)+ if use_corrector:+ sample = self.multistep_uni_c_bh_update(+ this_model_output=model_output_convert,+ last_sample=self.last_sample,+ this_sample=sample,+ order=self.this_order,+ )++ for i in range(self.config.solver_order - 1):+ self.model_outputs[i] = self.model_outputs[i + 1]+ self.timestep_list[i] = self.timestep_list[i + 1]++ self.model_outputs[-1] = model_output_convert+ self.timestep_list[-1] = timestep # pyright: ignore++ if self.config.lower_order_final:+ this_order = min(self.config.solver_order,+ len(self.timesteps) -+ self.step_index) # pyright: ignore+ else:+ this_order = self.config.solver_order++ self.this_order = min(this_order,+ self.lower_order_nums + 1) # warmup for multistep+ assert self.this_order > 0++ self.last_sample = sample+ prev_sample = self.multistep_uni_p_bh_update(+ model_output=model_output, # pass the original non-converted model output, in case solver-p is used+ sample=sample,+ order=self.this_order,+ )++ if self.lower_order_nums < self.config.solver_order:+ self.lower_order_nums += 1++ # upon completion increase step index by one+ self._step_index += 1 # pyright: ignore++ if not return_dict:+ return (prev_sample,)++ return SchedulerOutput(prev_sample=prev_sample)++ def scale_model_input(self, sample: torch.Tensor, *args,+ **kwargs) -> torch.Tensor:+ """+ Ensures interchangeability with schedulers that need to scale the denoising model input depending on the+ current timestep.++ Args:+ sample (`torch.Tensor`):+ The input sample.++ Returns:+ `torch.Tensor`:+ A scaled input sample.+ """+ return sample++ # Copied from diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler.add_noise+ def add_noise(+ self,+ original_samples: torch.Tensor,+ noise: torch.Tensor,+ timesteps: torch.IntTensor,+ ) -> torch.Tensor:+ # Make sure sigmas and timesteps have the same device and dtype as original_samples+ sigmas = self.sigmas.to(+ device=original_samples.device, dtype=original_samples.dtype)+ if original_samples.device.type == "mps" and torch.is_floating_point(+ timesteps):+ # mps does not support float64+ schedule_timesteps = self.timesteps.to(+ original_samples.device, dtype=torch.float32)+ timesteps = timesteps.to(+ original_samples.device, dtype=torch.float32)+ else:+ schedule_timesteps = self.timesteps.to(original_samples.device)+ timesteps = timesteps.to(original_samples.device)++ # begin_index is None when the scheduler is used for training or pipeline does not implement set_begin_index+ if self.begin_index is None:+ step_indices = [+ self.index_for_timestep(t, schedule_timesteps)+ for t in timesteps+ ]+ elif self.step_index is not None:+ # add_noise is called after first denoising step (for inpainting)+ step_indices = [self.step_index] * timesteps.shape[0]+ else:+ # add noise is called before first denoising step to create initial latent(img2img)+ step_indices = [self.begin_index] * timesteps.shape[0]++ sigma = sigmas[step_indices].flatten()+ while len(sigma.shape) < len(original_samples.shape):+ sigma = sigma.unsqueeze(-1)++ alpha_t, sigma_t = self._sigma_to_alpha_sigma_t(sigma)+ noisy_samples = alpha_t * original_samples + sigma_t * noise+ return noisy_samples++ def __len__(self):+ return self.config.num_train_timestepsdiff --git a/worldplay2/text/__init__.py b/worldplay2/text/__init__.pynew file mode 100644index 0000000..d6e636a--- /dev/null+++ b/worldplay2/text/__init__.py@@ -0,0 +1,3 @@+from .encoder import T5EncoderModel++__all__ = ["T5EncoderModel"]diff --git a/worldplay2/text/encoder.py b/worldplay2/text/encoder.pynew file mode 100644index 0000000..6bc85eb--- /dev/null+++ b/worldplay2/text/encoder.py@@ -0,0 +1,513 @@+# Modified from transformers.models.t5.modeling_t5+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.+import logging+import math++import torch+import torch.nn as nn+import torch.nn.functional as F++from .tokenizer import HuggingfaceTokenizer++__all__ = [+ 'T5Model',+ 'T5Encoder',+ 'T5Decoder',+ 'T5EncoderModel',+]+++def fp16_clamp(x):+ if x.dtype == torch.float16 and torch.isinf(x).any():+ clamp = torch.finfo(x.dtype).max - 1000+ x = torch.clamp(x, min=-clamp, max=clamp)+ return x+++def init_weights(m):+ if isinstance(m, T5LayerNorm):+ nn.init.ones_(m.weight)+ elif isinstance(m, T5Model):+ nn.init.normal_(m.token_embedding.weight, std=1.0)+ elif isinstance(m, T5FeedForward):+ nn.init.normal_(m.gate[0].weight, std=m.dim**-0.5)+ nn.init.normal_(m.fc1.weight, std=m.dim**-0.5)+ nn.init.normal_(m.fc2.weight, std=m.dim_ffn**-0.5)+ elif isinstance(m, T5Attention):+ nn.init.normal_(m.q.weight, std=(m.dim * m.dim_attn)**-0.5)+ nn.init.normal_(m.k.weight, std=m.dim**-0.5)+ nn.init.normal_(m.v.weight, std=m.dim**-0.5)+ nn.init.normal_(m.o.weight, std=(m.num_heads * m.dim_attn)**-0.5)+ elif isinstance(m, T5RelativeEmbedding):+ nn.init.normal_(+ m.embedding.weight, std=(2 * m.num_buckets * m.num_heads)**-0.5)+++class GELU(nn.Module):++ def forward(self, x):+ return 0.5 * x * (1.0 + torch.tanh(+ math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))+++class T5LayerNorm(nn.Module):++ def __init__(self, dim, eps=1e-6):+ super(T5LayerNorm, self).__init__()+ self.dim = dim+ self.eps = eps+ self.weight = nn.Parameter(torch.ones(dim))++ def forward(self, x):+ x = x * torch.rsqrt(x.float().pow(2).mean(dim=-1, keepdim=True) ++ self.eps)+ if self.weight.dtype in [torch.float16, torch.bfloat16]:+ x = x.type_as(self.weight)+ return self.weight * x+++class T5Attention(nn.Module):++ def __init__(self, dim, dim_attn, num_heads, dropout=0.1):+ assert dim_attn % num_heads == 0+ super(T5Attention, self).__init__()+ self.dim = dim+ self.dim_attn = dim_attn+ self.num_heads = num_heads+ self.head_dim = dim_attn // num_heads++ # layers+ self.q = nn.Linear(dim, dim_attn, bias=False)+ self.k = nn.Linear(dim, dim_attn, bias=False)+ self.v = nn.Linear(dim, dim_attn, bias=False)+ self.o = nn.Linear(dim_attn, dim, bias=False)+ self.dropout = nn.Dropout(dropout)++ def forward(self, x, context=None, mask=None, pos_bias=None):+ """+ x: [B, L1, C].+ context: [B, L2, C] or None.+ mask: [B, L2] or [B, L1, L2] or None.+ """+ # check inputs+ context = x if context is None else context+ b, n, c = x.size(0), self.num_heads, self.head_dim++ # compute query, key, value+ q = self.q(x).view(b, -1, n, c)+ k = self.k(context).view(b, -1, n, c)+ v = self.v(context).view(b, -1, n, c)++ # attention bias+ attn_bias = x.new_zeros(b, n, q.size(1), k.size(1))+ if pos_bias is not None:+ attn_bias += pos_bias+ if mask is not None:+ assert mask.ndim in [2, 3]+ mask = mask.view(b, 1, 1,+ -1) if mask.ndim == 2 else mask.unsqueeze(1)+ attn_bias.masked_fill_(mask == 0, torch.finfo(x.dtype).min)++ # compute attention (T5 does not use scaling)+ attn = torch.einsum('binc,bjnc->bnij', q, k) + attn_bias+ attn = F.softmax(attn.float(), dim=-1).type_as(attn)+ x = torch.einsum('bnij,bjnc->binc', attn, v)++ # output+ x = x.reshape(b, -1, n * c)+ x = self.o(x)+ x = self.dropout(x)+ return x+++class T5FeedForward(nn.Module):++ def __init__(self, dim, dim_ffn, dropout=0.1):+ super(T5FeedForward, self).__init__()+ self.dim = dim+ self.dim_ffn = dim_ffn++ # layers+ self.gate = nn.Sequential(nn.Linear(dim, dim_ffn, bias=False), GELU())+ self.fc1 = nn.Linear(dim, dim_ffn, bias=False)+ self.fc2 = nn.Linear(dim_ffn, dim, bias=False)+ self.dropout = nn.Dropout(dropout)++ def forward(self, x):+ x = self.fc1(x) * self.gate(x)+ x = self.dropout(x)+ x = self.fc2(x)+ x = self.dropout(x)+ return x+++class T5SelfAttention(nn.Module):++ def __init__(self,+ dim,+ dim_attn,+ dim_ffn,+ num_heads,+ num_buckets,+ shared_pos=True,+ dropout=0.1):+ super(T5SelfAttention, self).__init__()+ self.dim = dim+ self.dim_attn = dim_attn+ self.dim_ffn = dim_ffn+ self.num_heads = num_heads+ self.num_buckets = num_buckets+ self.shared_pos = shared_pos++ # layers+ self.norm1 = T5LayerNorm(dim)+ self.attn = T5Attention(dim, dim_attn, num_heads, dropout)+ self.norm2 = T5LayerNorm(dim)+ self.ffn = T5FeedForward(dim, dim_ffn, dropout)+ self.pos_embedding = None if shared_pos else T5RelativeEmbedding(+ num_buckets, num_heads, bidirectional=True)++ def forward(self, x, mask=None, pos_bias=None):+ e = pos_bias if self.shared_pos else self.pos_embedding(+ x.size(1), x.size(1))+ x = fp16_clamp(x + self.attn(self.norm1(x), mask=mask, pos_bias=e))+ x = fp16_clamp(x + self.ffn(self.norm2(x)))+ return x+++class T5CrossAttention(nn.Module):++ def __init__(self,+ dim,+ dim_attn,+ dim_ffn,+ num_heads,+ num_buckets,+ shared_pos=True,+ dropout=0.1):+ super(T5CrossAttention, self).__init__()+ self.dim = dim+ self.dim_attn = dim_attn+ self.dim_ffn = dim_ffn+ self.num_heads = num_heads+ self.num_buckets = num_buckets+ self.shared_pos = shared_pos++ # layers+ self.norm1 = T5LayerNorm(dim)+ self.self_attn = T5Attention(dim, dim_attn, num_heads, dropout)+ self.norm2 = T5LayerNorm(dim)+ self.cross_attn = T5Attention(dim, dim_attn, num_heads, dropout)+ self.norm3 = T5LayerNorm(dim)+ self.ffn = T5FeedForward(dim, dim_ffn, dropout)+ self.pos_embedding = None if shared_pos else T5RelativeEmbedding(+ num_buckets, num_heads, bidirectional=False)++ def forward(self,+ x,+ mask=None,+ encoder_states=None,+ encoder_mask=None,+ pos_bias=None):+ e = pos_bias if self.shared_pos else self.pos_embedding(+ x.size(1), x.size(1))+ x = fp16_clamp(x + self.self_attn(self.norm1(x), mask=mask, pos_bias=e))+ x = fp16_clamp(x + self.cross_attn(+ self.norm2(x), context=encoder_states, mask=encoder_mask))+ x = fp16_clamp(x + self.ffn(self.norm3(x)))+ return x+++class T5RelativeEmbedding(nn.Module):++ def __init__(self, num_buckets, num_heads, bidirectional, max_dist=128):+ super(T5RelativeEmbedding, self).__init__()+ self.num_buckets = num_buckets+ self.num_heads = num_heads+ self.bidirectional = bidirectional+ self.max_dist = max_dist++ # layers+ self.embedding = nn.Embedding(num_buckets, num_heads)++ def forward(self, lq, lk):+ device = self.embedding.weight.device+ # rel_pos = torch.arange(lk).unsqueeze(0).to(device) - \+ # torch.arange(lq).unsqueeze(1).to(device)+ rel_pos = torch.arange(lk, device=device).unsqueeze(0) - \+ torch.arange(lq, device=device).unsqueeze(1)+ rel_pos = self._relative_position_bucket(rel_pos)+ rel_pos_embeds = self.embedding(rel_pos)+ rel_pos_embeds = rel_pos_embeds.permute(2, 0, 1).unsqueeze(+ 0) # [1, N, Lq, Lk]+ return rel_pos_embeds.contiguous()++ def _relative_position_bucket(self, rel_pos):+ # preprocess+ if self.bidirectional:+ num_buckets = self.num_buckets // 2+ rel_buckets = (rel_pos > 0).long() * num_buckets+ rel_pos = torch.abs(rel_pos)+ else:+ num_buckets = self.num_buckets+ rel_buckets = 0+ rel_pos = -torch.min(rel_pos, torch.zeros_like(rel_pos))++ # embeddings for small and large positions+ max_exact = num_buckets // 2+ rel_pos_large = max_exact + (torch.log(rel_pos.float() / max_exact) /+ math.log(self.max_dist / max_exact) *+ (num_buckets - max_exact)).long()+ rel_pos_large = torch.min(+ rel_pos_large, torch.full_like(rel_pos_large, num_buckets - 1))+ rel_buckets += torch.where(rel_pos < max_exact, rel_pos, rel_pos_large)+ return rel_buckets+++class T5Encoder(nn.Module):++ def __init__(self,+ vocab,+ dim,+ dim_attn,+ dim_ffn,+ num_heads,+ num_layers,+ num_buckets,+ shared_pos=True,+ dropout=0.1):+ super(T5Encoder, self).__init__()+ self.dim = dim+ self.dim_attn = dim_attn+ self.dim_ffn = dim_ffn+ self.num_heads = num_heads+ self.num_layers = num_layers+ self.num_buckets = num_buckets+ self.shared_pos = shared_pos++ # layers+ self.token_embedding = vocab if isinstance(vocab, nn.Embedding) \+ else nn.Embedding(vocab, dim)+ self.pos_embedding = T5RelativeEmbedding(+ num_buckets, num_heads, bidirectional=True) if shared_pos else None+ self.dropout = nn.Dropout(dropout)+ self.blocks = nn.ModuleList([+ T5SelfAttention(dim, dim_attn, dim_ffn, num_heads, num_buckets,+ shared_pos, dropout) for _ in range(num_layers)+ ])+ self.norm = T5LayerNorm(dim)++ # initialize weights+ self.apply(init_weights)++ def forward(self, ids, mask=None):+ x = self.token_embedding(ids)+ x = self.dropout(x)+ e = self.pos_embedding(x.size(1),+ x.size(1)) if self.shared_pos else None+ for block in self.blocks:+ x = block(x, mask, pos_bias=e)+ x = self.norm(x)+ x = self.dropout(x)+ return x+++class T5Decoder(nn.Module):++ def __init__(self,+ vocab,+ dim,+ dim_attn,+ dim_ffn,+ num_heads,+ num_layers,+ num_buckets,+ shared_pos=True,+ dropout=0.1):+ super(T5Decoder, self).__init__()+ self.dim = dim+ self.dim_attn = dim_attn+ self.dim_ffn = dim_ffn+ self.num_heads = num_heads+ self.num_layers = num_layers+ self.num_buckets = num_buckets+ self.shared_pos = shared_pos++ # layers+ self.token_embedding = vocab if isinstance(vocab, nn.Embedding) \+ else nn.Embedding(vocab, dim)+ self.pos_embedding = T5RelativeEmbedding(+ num_buckets, num_heads, bidirectional=False) if shared_pos else None+ self.dropout = nn.Dropout(dropout)+ self.blocks = nn.ModuleList([+ T5CrossAttention(dim, dim_attn, dim_ffn, num_heads, num_buckets,+ shared_pos, dropout) for _ in range(num_layers)+ ])+ self.norm = T5LayerNorm(dim)++ # initialize weights+ self.apply(init_weights)++ def forward(self, ids, mask=None, encoder_states=None, encoder_mask=None):+ b, s = ids.size()++ # causal mask+ if mask is None:+ mask = torch.tril(torch.ones(1, s, s).to(ids.device))+ elif mask.ndim == 2:+ mask = torch.tril(mask.unsqueeze(1).expand(-1, s, -1))++ # layers+ x = self.token_embedding(ids)+ x = self.dropout(x)+ e = self.pos_embedding(x.size(1),+ x.size(1)) if self.shared_pos else None+ for block in self.blocks:+ x = block(x, mask, encoder_states, encoder_mask, pos_bias=e)+ x = self.norm(x)+ x = self.dropout(x)+ return x+++class T5Model(nn.Module):++ def __init__(self,+ vocab_size,+ dim,+ dim_attn,+ dim_ffn,+ num_heads,+ encoder_layers,+ decoder_layers,+ num_buckets,+ shared_pos=True,+ dropout=0.1):+ super(T5Model, self).__init__()+ self.vocab_size = vocab_size+ self.dim = dim+ self.dim_attn = dim_attn+ self.dim_ffn = dim_ffn+ self.num_heads = num_heads+ self.encoder_layers = encoder_layers+ self.decoder_layers = decoder_layers+ self.num_buckets = num_buckets++ # layers+ self.token_embedding = nn.Embedding(vocab_size, dim)+ self.encoder = T5Encoder(self.token_embedding, dim, dim_attn, dim_ffn,+ num_heads, encoder_layers, num_buckets,+ shared_pos, dropout)+ self.decoder = T5Decoder(self.token_embedding, dim, dim_attn, dim_ffn,+ num_heads, decoder_layers, num_buckets,+ shared_pos, dropout)+ self.head = nn.Linear(dim, vocab_size, bias=False)++ # initialize weights+ self.apply(init_weights)++ def forward(self, encoder_ids, encoder_mask, decoder_ids, decoder_mask):+ x = self.encoder(encoder_ids, encoder_mask)+ x = self.decoder(decoder_ids, decoder_mask, x, encoder_mask)+ x = self.head(x)+ return x+++def _t5(name,+ encoder_only=False,+ decoder_only=False,+ return_tokenizer=False,+ tokenizer_kwargs={},+ dtype=torch.float32,+ device='cpu',+ **kwargs):+ # sanity check+ assert not (encoder_only and decoder_only)++ # params+ if encoder_only:+ model_cls = T5Encoder+ kwargs['vocab'] = kwargs.pop('vocab_size')+ kwargs['num_layers'] = kwargs.pop('encoder_layers')+ _ = kwargs.pop('decoder_layers')+ elif decoder_only:+ model_cls = T5Decoder+ kwargs['vocab'] = kwargs.pop('vocab_size')+ kwargs['num_layers'] = kwargs.pop('decoder_layers')+ _ = kwargs.pop('encoder_layers')+ else:+ model_cls = T5Model++ # init model+ with torch.device(device):+ model = model_cls(**kwargs)++ # set device+ model = model.to(dtype=dtype, device=device)++ # init tokenizer+ if return_tokenizer:+ from .tokenizer import HuggingfaceTokenizer+ tokenizer = HuggingfaceTokenizer(f'google/{name}', **tokenizer_kwargs)+ return model, tokenizer+ else:+ return model+++def umt5_xxl(**kwargs):+ cfg = dict(+ vocab_size=256384,+ dim=4096,+ dim_attn=4096,+ dim_ffn=10240,+ num_heads=64,+ encoder_layers=24,+ decoder_layers=24,+ num_buckets=32,+ shared_pos=False,+ dropout=0.1)+ cfg.update(**kwargs)+ return _t5('umt5-xxl', **cfg)+++class T5EncoderModel:++ def __init__(+ self,+ text_len,+ dtype=torch.bfloat16,+ device=torch.cuda.current_device(),+ checkpoint_path=None,+ tokenizer_path=None,+ shard_fn=None,+ ):+ self.text_len = text_len+ self.dtype = dtype+ self.device = device+ self.checkpoint_path = checkpoint_path+ self.tokenizer_path = tokenizer_path++ # init model+ model = umt5_xxl(+ encoder_only=True,+ return_tokenizer=False,+ dtype=dtype,+ device=device).eval().requires_grad_(False)+ logging.info(f'loading {checkpoint_path}')+ model.load_state_dict(torch.load(checkpoint_path, map_location='cpu'))+ self.model = model+ if shard_fn is not None:+ self.model = shard_fn(self.model, sync_module_states=False)+ else:+ self.model.to(self.device)+ # init tokenizer+ self.tokenizer = HuggingfaceTokenizer(+ name=tokenizer_path, seq_len=text_len, clean='whitespace')++ def __call__(self, texts, device):+ ids, mask = self.tokenizer(+ texts, return_mask=True, add_special_tokens=True)+ ids = ids.to(device)+ mask = mask.to(device)+ seq_lens = mask.gt(0).sum(dim=1).long()+ context = self.model(ids, mask)+ return [u[:v] for u, v in zip(context, seq_lens)]diff --git a/worldplay2/text/tokenizer.py b/worldplay2/text/tokenizer.pynew file mode 100644index 0000000..121e591--- /dev/null+++ b/worldplay2/text/tokenizer.py@@ -0,0 +1,82 @@+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.+import html+import string++import ftfy+import regex as re+from transformers import AutoTokenizer++__all__ = ['HuggingfaceTokenizer']+++def basic_clean(text):+ text = ftfy.fix_text(text)+ text = html.unescape(html.unescape(text))+ return text.strip()+++def whitespace_clean(text):+ text = re.sub(r'\s+', ' ', text)+ text = text.strip()+ return text+++def canonicalize(text, keep_punctuation_exact_string=None):+ text = text.replace('_', ' ')+ if keep_punctuation_exact_string:+ text = keep_punctuation_exact_string.join(+ part.translate(str.maketrans('', '', string.punctuation))+ for part in text.split(keep_punctuation_exact_string))+ else:+ text = text.translate(str.maketrans('', '', string.punctuation))+ text = text.lower()+ text = re.sub(r'\s+', ' ', text)+ return text.strip()+++class HuggingfaceTokenizer:++ def __init__(self, name, seq_len=None, clean=None, **kwargs):+ assert clean in (None, 'whitespace', 'lower', 'canonicalize')+ self.name = name+ self.seq_len = seq_len+ self.clean = clean++ # init tokenizer+ self.tokenizer = AutoTokenizer.from_pretrained(name, **kwargs)+ self.vocab_size = self.tokenizer.vocab_size++ def __call__(self, sequence, **kwargs):+ return_mask = kwargs.pop('return_mask', False)++ # arguments+ _kwargs = {'return_tensors': 'pt'}+ if self.seq_len is not None:+ _kwargs.update({+ 'padding': 'max_length',+ 'truncation': True,+ 'max_length': self.seq_len+ })+ _kwargs.update(**kwargs)++ # tokenization+ if isinstance(sequence, str):+ sequence = [sequence]+ if self.clean:+ sequence = [self._clean(u) for u in sequence]+ ids = self.tokenizer(sequence, **_kwargs)++ # output+ if return_mask:+ return ids.input_ids, ids.attention_mask+ else:+ return ids.input_ids++ def _clean(self, text):+ if self.clean == 'whitespace':+ text = whitespace_clean(basic_clean(text))+ elif self.clean == 'lower':+ text = whitespace_clean(basic_clean(text)).lower()+ elif self.clean == 'canonicalize':+ text = canonicalize(basic_clean(text))+ return textdiff --git a/worldplay2/vae/__init__.py b/worldplay2/vae/__init__.pynew file mode 100644index 0000000..bdef9f7--- /dev/null+++ b/worldplay2/vae/__init__.py@@ -0,0 +1,4 @@+from .causal_tae import CausalTAEVAE+from .wan21 import Wan2_1_VAE++__all__ = ["CausalTAEVAE", "Wan2_1_VAE"]diff --git a/worldplay2/vae/causal_tae.py b/worldplay2/vae/causal_tae.pynew file mode 100644index 0000000..c1394ea--- /dev/null+++ b/worldplay2/vae/causal_tae.py@@ -0,0 +1,864 @@+#!/usr/bin/env python3+"""+Causal (left-padded) TAE-style ENCODER for Wan 2.1, adapted for 4n+1 frame+video inputs.++This is a variant of the TAEW2_1 encoder (see ``taew2_1.py``) with ONE+architectural change in the temporal-pooling behaviour:++ * The original ``TPool`` reshapes every ``stride`` consecutive frames+ into the channel axis directly. It relied on ``encode_video`` padding+ the time axis at the *end* to a multiple of ``t_downscale``.++ * ``CausalTPool`` (here) instead pads ONE zero frame on the *left*+ (the temporal-causal side) whenever the incoming time length is not a+ multiple of ``stride`` (i.e. for stride>1 on an odd-length input),+ then pools. For a 4n+1 input this cascades cleanly:+ T=4n+1 --pad1--> 4n+2 --pool2--> 2n+1 --pad1--> 2n+2 --pool2--> n+1.++``MemBlock`` keeps the SAME behaviour as the original parallel path: it+takes the "previous frame" as memory by left-padding one zero frame along+the time axis and concatenating it on the channel dim inside the block.+That part is unchanged — this file only re-implements the pooling.++Scope (per request):+ * ENCODER ONLY. No decoder / TGrow.+ * ONE-SHOT (parallel) encode only. No streaming work-queue path.++I/O convention (same as TAEW2_1):+ encode_video: input [N, T, C=3, H, W] in [0, 1], T = 4n+1+ output [N, T_lat, latent_channels, H/8, W/8]+"""+import torch+import torch.nn as nn+import torch.nn.functional as F+from tqdm.auto import tqdm+++def conv(n_in, n_out, **kwargs):+ return nn.Conv2d(n_in, n_out, 3, padding=1, **kwargs)+++class MemBlock(nn.Module):+ """Residual block that also consumes the previous frame's activation+ (its "memory") concatenated on the channel axis. Identical to the+ TAEW2_1 MemBlock."""++ def __init__(self, n_in, n_out):+ super().__init__()+ self.conv = nn.Sequential(conv(n_in * 2, n_out), nn.ReLU(inplace=True), conv(n_out, n_out), nn.ReLU(inplace=True), conv(n_out, n_out))+ self.skip = nn.Conv2d(n_in, n_out, 1, bias=False) if n_in != n_out else nn.Identity()+ self.act = nn.ReLU(inplace=True)++ def forward(self, x, past):+ return self.act(self.conv(torch.cat([x, past], 1)) + self.skip(x))++ def forward_chunk(self, x, N, cache):+ """Streaming forward over one chunk in [N*T, C, H, W] layout.++ The per-frame memory is the *previous* frame's input. Within the+ chunk this is the left-shifted version of ``x``; the very first+ frame of the chunk uses ``cache`` (the last input frame of the+ PREVIOUS chunk) instead of a zero frame -- except when ``cache`` is+ None (the first chunk overall), where it falls back to zeros. This+ makes the chunked result identical to the one-shot path.++ Args:+ x: [N*T, C, H, W] chunk input.+ N: batch size.+ cache: previous chunk's last-frame input, [N, 1, C, H, W], or None.+ Returns:+ (out, new_cache) where out is [N*T, C, H, W] and new_cache is+ this chunk's last-frame input [N, 1, C, H, W].+ """+ NT, C, H, W = x.shape+ T = NT // N+ _x = x.view(N, T, C, H, W)+ if cache is None:+ head = _x.new_zeros(N, 1, C, H, W) # first chunk: zero history+ else:+ head = cache # continuation: prev last frame+ # memory[t] = input[t-1]; memory[0] = head+ block_memory = torch.cat([head, _x[:, :T - 1]], dim=1).reshape(x.shape)+ new_cache = _x[:, -1:].clone()+ out = self.forward(x, block_memory)+ return out, new_cache+++class CausalTPool(nn.Module):+ """Temporal pooling that LEFT-pads one zero frame when needed so a+ 4n+1 (odd) time length becomes even before pooling every ``stride``+ frames into the channel axis.++ Operates on the flattened [N*T, C, H, W] layout used by the parallel+ encoder path, and therefore needs ``N`` (batch size) to recover the+ time dimension.+ """++ def __init__(self, n_f, stride):+ super().__init__()+ self.stride = stride+ self.conv = nn.Conv2d(n_f * stride, n_f, 1, bias=False)++ def forward(self, x, N):+ # x: [N*T, C, H, W]+ NT, C, H, W = x.shape+ T = NT // N+ if self.stride == 1:+ # no temporal pooling; the 1x1 conv still maps n_f -> n_f+ return self.conv(x)++ x = x.view(N, T, C, H, W)+ # left-pad zeros along the time axis until T is a multiple of stride+ # (for 4n+1 inputs and stride=2 this pads exactly one frame).+ rem = T % self.stride+ if rem != 0:+ n_pad = self.stride - rem+ pad = x.new_zeros(N, n_pad, C, H, W)+ x = torch.cat([pad, x], dim=1) # LEFT pad (causal side)+ T = x.shape[1]++ # group every ``stride`` consecutive frames onto the channel axis,+ # matching the original TPool's reshape ordering (frame t occupies+ # channels [0:C], frame t+1 -> [C:2C], ...).+ x = x.view(N, T // self.stride, self.stride * C, H, W)+ x = x.reshape(N * (T // self.stride), self.stride * C, H, W)+ return self.conv(x)++ def forward_chunk(self, x, N, is_first_chunk):+ """Streaming pool for the 1,4,4,4 chunk schedule.++ With that schedule and stride=2, only the FIRST chunk is odd+ (length 1 -> left-pad 1 -> 2 -> pool -> 1); every subsequent chunk+ already has a length that is a multiple of ``stride`` (4 -> pool ->+ 2, and at the 2nd pooling stage 2 -> pool -> 1), so no cross-chunk+ residual buffering is needed and the concatenation of per-chunk+ outputs is bit-for-bit identical to the one-shot ``forward``.++ Args:+ x: [N*T, C, H, W] chunk input.+ N: batch size.+ is_first_chunk: True only for the very first chunk.+ Returns:+ [N*(T_out), C, H, W] pooled output for this chunk.+ """+ NT, C, H, W = x.shape+ T = NT // N+ if self.stride == 1:+ return self.conv(x)++ _x = x.view(N, T, C, H, W)+ rem = T % self.stride+ if rem != 0:+ assert is_first_chunk, (+ f"CausalTPool.forward_chunk: non-first chunk has time length "+ f"{T} which is not a multiple of stride {self.stride}; the "+ f"1,4,4,4 schedule should keep continuation chunks aligned.")+ n_pad = self.stride - rem+ pad = _x.new_zeros(N, n_pad, C, H, W)+ _x = torch.cat([pad, _x], dim=1) # LEFT pad only on the first chunk+ T = _x.shape[1]++ _x = _x.view(N, T // self.stride, self.stride * C, H, W)+ _x = _x.reshape(N * (T // self.stride), self.stride * C, H, W)+ return self.conv(_x)+++class Clamp(nn.Module):+ """Soft clamp to ~[-3, 3] on the latent, used at the head of the decoder+ (identical to the TAEW2_1 Clamp)."""++ def forward(self, x):+ return torch.tanh(x / 3) * 3+++class CausalTGrow(nn.Module):+ """Temporal upsampling, inverse of ``CausalTPool``.++ ``conv`` expands the channel axis to ``stride * n_f`` and the frames are+ then split back out along the time axis (1 latent frame -> ``stride``+ pixel-time frames), matching the original TGrow reshape ordering.++ For CAUSAL first-frame handling, the VERY FIRST latent frame of a clip+ maps to a single output frame with ``output == input`` (it skips both the+ conv and the temporal expansion entirely), while every subsequent latent+ frame is expanded to ``stride`` frames. This mirrors the encoder, where+ the first pixel frame maps to one latent; here the first latent maps back+ to one frame, so:++ T_lat = n + 1 -> T_pix = 1 + n * (product of TGrow strides) = 4n+1+ """++ def __init__(self, n_f, stride):+ super().__init__()+ self.stride = stride+ self.conv = nn.Conv2d(n_f, n_f * stride, 1, bias=False)++ def _expand(self, x):+ """conv + reshape: [N*T, C, H, W] -> [N*T*stride, C, H, W]."""+ NT, C, H, W = x.shape+ x = self.conv(x) # [N*T, stride*C, H, W]+ return x.reshape(NT * self.stride, C, H, W)++ def forward(self, x, N, is_first_clip=True):+ """One-shot temporal grow over the whole clip.++ Args:+ x: [N*T_lat, C, H, W].+ N: batch size.+ is_first_clip: if True, the first latent frame of the clip+ expands to a single output frame (causal startup), the rest+ expand to ``stride`` frames each. If False (continuation+ chunk in streaming decode), ALL frames expand to ``stride``.+ Returns:+ [N*T_out, C, H, W].+ """+ NT, C, H, W = x.shape+ T = NT // N+ if self.stride == 1:+ # no temporal upsampling; conv maps n_f -> n_f*1 = n_f, reshape no-op+ return self._expand(x)++ _x = x.view(N, T, C, H, W)+ if is_first_clip:+ # first latent frame: output == input (no temporal expansion,+ # no conv), so it maps to exactly ONE output frame.+ first = _x[:, :1] # [N, 1, C, H, W]+ rest = _x[:, 1:].reshape(N * (T - 1), C, H, W) if T > 1 else None+ outs = [first]+ if rest is not None:+ rest_grown = self._expand(rest) # [N*(T-1)*stride, C, H, W]+ rest_grown = rest_grown.view(N, (T - 1) * self.stride, C, H, W)+ outs.append(rest_grown)+ out = torch.cat(outs, dim=1) # [N, T_out, C, H, W]+ return out.reshape(N * out.shape[1], C, H, W)+ else:+ # continuation chunk: every latent frame expands to stride frames+ return self._expand(x)+++class CausalTAEEncoder(nn.Module):+ """One-shot causal TAE-style encoder for Wan 2.1 (4n+1 frame inputs)."""++ def __init__(self,+ encoder_time_downscale=(True, True, False),+ latent_channels=16,+ patch_size=1):+ super().__init__()+ self.patch_size = patch_size+ self.latent_channels = latent_channels+ self.image_channels = 3++ self.encoder = nn.Sequential(+ conv(self.image_channels * self.patch_size ** 2, 64), nn.ReLU(inplace=True),+ CausalTPool(64, 2 if encoder_time_downscale[0] else 1), conv(64, 64, stride=2, bias=False), MemBlock(64, 64), MemBlock(64, 64), MemBlock(64, 64),+ CausalTPool(64, 2 if encoder_time_downscale[1] else 1), conv(64, 64, stride=2, bias=False), MemBlock(64, 64), MemBlock(64, 64), MemBlock(64, 64),+ CausalTPool(64, 2 if encoder_time_downscale[2] else 1), conv(64, 64, stride=2, bias=False), MemBlock(64, 64), MemBlock(64, 64), MemBlock(64, 64),+ conv(64, self.latent_channels),+ )++ # temporal downscale factor (product of the pooling strides)+ self.t_downscale = 1+ for m in self.encoder:+ if isinstance(m, CausalTPool):+ self.t_downscale *= m.stride++ def preprocess_input_frames(self, x):+ """Preprocess RGB input frames prior to the main encoder sequence."""+ if self.patch_size > 1:+ x = F.pixel_unshuffle(x, self.patch_size)+ return x++ def _apply_encoder(self, x, show_progress_bar):+ """Parallel (one-shot) application of the encoder sequence over the+ time axis.++ Args:+ x: [N, T, C, H, W] tensor.+ Returns:+ [N, T_lat, latent_channels, H/8, W/8] tensor.+ """+ assert x.ndim == 5, f"expected NTCHW tensor, got {x.ndim}-dim"+ N, T, C, H, W = x.shape+ x = x.reshape(N * T, C, H, W)++ for b in tqdm(self.encoder, disable=not show_progress_bar):+ if isinstance(b, MemBlock):+ # "previous frame" memory = left-pad one zero frame along+ # time, then drop the last frame so lengths line up. This is+ # the same causal memory the original parallel path uses.+ NT, C, H, W = x.shape+ T = NT // N+ _x = x.reshape(N, T, C, H, W)+ block_memory = F.pad(_x, (0, 0, 0, 0, 0, 0, 1, 0), value=0)[:, :T].reshape(x.shape)+ x = b(x, block_memory)+ elif isinstance(b, CausalTPool):+ # CausalTPool needs N to recover the time dim and does the+ # left-pad-then-pool internally.+ x = b(x, N)+ else:+ x = b(x)++ NT, C, H, W = x.shape+ T = NT // N+ return x.view(N, T, C, H, W)++ def encode_video(self, x, show_progress_bar=True):+ """Encode a sequence of frames (one-shot / parallel).++ Args:+ x: input [N, T, C=3, H, W] RGB tensor in [0, 1], with T = 4n+1.+ show_progress_bar: enable tqdm over encoder blocks.+ Returns:+ [N, T_lat, latent_channels, H/8, W/8] latent tensor.++ Unlike TAEW2_1.encode_video, there is NO end-of-sequence padding:+ the left-padding inside each CausalTPool handles odd (4n+1) lengths+ causally.+ """+ x = self.preprocess_input_frames(x)+ return self._apply_encoder(x, show_progress_bar)++ # ------------------------------------------------------------------+ # streaming / chunked encode (1, 4, 4, 4, ... schedule)+ # ------------------------------------------------------------------++ def init_cache(self):+ """Create a fresh per-block streaming cache.++ Returns a list with one slot per encoder block; only ``MemBlock``+ slots are ever populated (each holds that block's last-frame input+ from the previous chunk). Pass the same object to every+ :meth:`encode_chunk` call of one stream.+ """+ return [None] * len(self.encoder)++ def _apply_encoder_chunk(self, x, cache, is_first_chunk):+ """Streaming application of the encoder over one chunk.++ Args:+ x: [N, T_chunk, C, H, W] chunk (T_chunk == 1 for the first chunk,+ then 4 for each continuation chunk).+ cache: list from :meth:`init_cache`, mutated in place.+ is_first_chunk: True only for the first chunk of the stream.+ Returns:+ [N, T_lat_chunk, latent_channels, H/8, W/8] latent for this chunk.+ """+ assert x.ndim == 5, f"expected NTCHW tensor, got {x.ndim}-dim"+ N = x.shape[0]+ x = x.reshape(N * x.shape[1], *x.shape[2:])++ for i, b in enumerate(self.encoder):+ if isinstance(b, MemBlock):+ x, cache[i] = b.forward_chunk(x, N, cache[i])+ elif isinstance(b, CausalTPool):+ x = b.forward_chunk(x, N, is_first_chunk)+ else:+ x = b(x)++ NT, C, H, W = x.shape+ return x.view(N, NT // N, C, H, W)++ def encode_chunk(self, x, cache, is_first_chunk=False):+ """Encode ONE chunk of frames, continuing the streaming state.++ Feed the video in the ``1, 4, 4, 4, ...`` schedule: the first call+ with ``is_first_chunk=True`` passes a single frame ([N, 1, C, H, W]),+ and each subsequent call passes 4 frames ([N, 4, C, H, W]) with+ ``is_first_chunk=False``. Concatenating the returned latents along+ the time axis reproduces :meth:`encode_video` on the full clip+ exactly.++ Args:+ x: [N, T_chunk, C=3, H, W] RGB chunk in [0, 1].+ cache: list from :meth:`init_cache`, reused across the stream.+ is_first_chunk: True only for the very first chunk.+ Returns:+ [N, T_lat_chunk, latent_channels, H/8, W/8] latent for this chunk.+ """+ x = self.preprocess_input_frames(x)+ return self._apply_encoder_chunk(x, cache, is_first_chunk)++ def encode_video_chunked(self, x, show_progress_bar=False):+ """Convenience wrapper: split a full 4n+1 clip into the 1,4,4,4,...+ schedule, run :meth:`encode_chunk` over each piece, and concatenate.++ Args:+ x: [N, T=4n+1, C=3, H, W] RGB clip in [0, 1].+ Returns:+ [N, T_lat, latent_channels, H/8, W/8] latent (== encode_video).+ """+ T = x.shape[1]+ assert T % 4 == 1, f"chunked encode expects 4n+1 frames, got T={T}"+ cache = self.init_cache()+ outs = []+ # first chunk: 1 frame+ outs.append(self.encode_chunk(x[:, :1], cache, is_first_chunk=True))+ # continuation chunks: 4 frames each+ starts = range(1, T, 4)+ for s in tqdm(starts, disable=not show_progress_bar):+ outs.append(self.encode_chunk(x[:, s:s + 4], cache, is_first_chunk=False))+ return torch.cat(outs, dim=1)++ def forward(self, x, show_progress_bar=False):+ return self.encode_video(x, show_progress_bar=show_progress_bar)+++class CausalTAEDecoder(nn.Module):+ """Causal TAE-style decoder for Wan 2.1, inverse of CausalTAEEncoder.++ The architecture mirrors the TAEW2_1 base decoder. The only causal+ change is in temporal upsampling: the FIRST latent frame of a clip+ decodes to a SINGLE output frame (via CausalTGrow's first-frame path),+ so the whole decoder maps++ T_lat = n + 1 latent frames -> T_pix = 4n + 1 output frames++ which is the exact inverse of the encoder. No end-of-sequence trimming+ is required (unlike the original TAEW2_1.decode_video which trims+ ``frames_to_trim`` startup frames).+ """++ def __init__(self,+ decoder_time_upscale=(False, True, True),+ decoder_space_upscale=(True, True, True),+ latent_channels=16,+ patch_size=1):+ super().__init__()+ self.patch_size = patch_size+ self.latent_channels = latent_channels+ self.image_channels = 3++ n_f = [256, 128, 64, 64]+ self.decoder = nn.Sequential(+ Clamp(), conv(self.latent_channels, n_f[0]), nn.ReLU(inplace=True),+ MemBlock(n_f[0], n_f[0]), MemBlock(n_f[0], n_f[0]), MemBlock(n_f[0], n_f[0]), nn.Upsample(scale_factor=2 if decoder_space_upscale[0] else 1), CausalTGrow(n_f[0], 2 if decoder_time_upscale[0] else 1), conv(n_f[0], n_f[1], bias=False),+ MemBlock(n_f[1], n_f[1]), MemBlock(n_f[1], n_f[1]), MemBlock(n_f[1], n_f[1]), nn.Upsample(scale_factor=2 if decoder_space_upscale[1] else 1), CausalTGrow(n_f[1], 2 if decoder_time_upscale[1] else 1), conv(n_f[1], n_f[2], bias=False),+ MemBlock(n_f[2], n_f[2]), MemBlock(n_f[2], n_f[2]), MemBlock(n_f[2], n_f[2]), nn.Upsample(scale_factor=2 if decoder_space_upscale[2] else 1), CausalTGrow(n_f[2], 2 if decoder_time_upscale[2] else 1), conv(n_f[2], n_f[3], bias=False),+ nn.ReLU(inplace=True), conv(n_f[3], self.image_channels * self.patch_size ** 2),+ )++ # temporal upscale factor (product of the grow strides)+ self.t_upscale = 1+ for m in self.decoder:+ if isinstance(m, CausalTGrow):+ self.t_upscale *= m.stride++ def postprocess_output_frames(self, x):+ """Postprocess RGB frames after the main decoder sequence."""+ if self.patch_size > 1:+ x = F.pixel_shuffle(x, self.patch_size)+ return x.clamp_(0, 1)++ def _apply_decoder(self, x, is_first_clip, show_progress_bar):+ """Apply the decoder sequence over the time axis (one-shot).++ Args:+ x: [N, T_lat, C, H, W] latent tensor.+ is_first_clip: True if this batch of latents starts a clip (the+ first latent frame decodes to a single output frame).+ Returns:+ [N, T_out, image_channels*patch_size**2, H*8, W*8] tensor.+ """+ assert x.ndim == 5, f"expected NTCHW tensor, got {x.ndim}-dim"+ N, T, C, H, W = x.shape+ x = x.reshape(N * T, C, H, W)++ for b in tqdm(self.decoder, disable=not show_progress_bar):+ if isinstance(b, MemBlock):+ NT, C, H, W = x.shape+ T = NT // N+ _x = x.reshape(N, T, C, H, W)+ block_memory = F.pad(_x, (0, 0, 0, 0, 0, 0, 1, 0), value=0)[:, :T].reshape(x.shape)+ x = b(x, block_memory)+ elif isinstance(b, CausalTGrow):+ x = b(x, N, is_first_clip=is_first_clip)+ else:+ x = b(x)++ NT, C, H, W = x.shape+ T = NT // N+ return x.view(N, T, C, H, W)++ def decode_video(self, x, show_progress_bar=True):+ """Decode a full sequence of latents (one-shot / parallel).++ Args:+ x: input [N, T_lat=n+1, C=latent_channels, H, W] latent tensor.+ show_progress_bar: enable tqdm over decoder blocks.+ Returns:+ [N, T_pix=4n+1, 3, H*8, W*8] RGB tensor in [0, 1].+ """+ x = self._apply_decoder(x, is_first_clip=True, show_progress_bar=show_progress_bar)+ return self.postprocess_output_frames(x)++ def decode_latent(self, x, is_first_latent, show_progress_bar=False):+ """Decode ONE (or a few) latent frame(s) with explicit first-frame+ control, WITHOUT threading a cross-latent cache -- the low-level+ stateless building block.++ NOTE: without a MemBlock cache, calling this repeatedly is NOT+ numerically identical to ``decode_video`` at chunk boundaries (each+ call restarts the MemBlock memory from zeros). For an equivalent+ one-latent-at-a-time decode, use :meth:`init_cache` ++ :meth:`decode_chunk` instead.++ Args:+ x: [N, T_sub, C=latent_channels, H, W] latent(s).+ is_first_latent: True if these latents include the clip's first+ latent frame (-> single output frame for that frame).+ Returns:+ [N, T_out, 3, H*8, W*8] RGB tensor in [0, 1].+ """+ x = self._apply_decoder(x, is_first_clip=is_first_latent,+ show_progress_bar=show_progress_bar)+ return self.postprocess_output_frames(x)++ # ------------------------------------------------------------------+ # streaming / chunked decode (cross-latent cache)+ # ------------------------------------------------------------------++ def init_cache(self):+ """Create a fresh per-block streaming cache.++ Returns a list with one slot per decoder block; only ``MemBlock``+ slots are ever populated (each holds that block's last-frame input+ from the previous chunk). Pass the same object to every+ :meth:`decode_chunk` call of one stream.+ """+ return [None] * len(self.decoder)++ def _apply_decoder_chunk(self, x, cache, is_first_chunk):+ """Streaming application of the decoder over one chunk of latents.++ MemBlock threads its "previous frame" memory through ``cache`` (same+ contract as the encoder), and CausalTGrow uses ``is_first_chunk`` so+ that only the very first latent of the stream yields a single frame+ (output == input); continuation chunks expand every latent frame by+ ``stride``.++ Args:+ x: [N, T_sub, C, H, W] latent chunk.+ cache: list from :meth:`init_cache`, mutated in place.+ is_first_chunk: True only for the first chunk of the stream.+ Returns:+ [N, T_out, image_channels*patch_size**2, H*8, W*8] tensor.+ """+ assert x.ndim == 5, f"expected NTCHW tensor, got {x.ndim}-dim"+ N = x.shape[0]+ x = x.reshape(N * x.shape[1], *x.shape[2:])++ for i, b in enumerate(self.decoder):+ if isinstance(b, MemBlock):+ x, cache[i] = b.forward_chunk(x, N, cache[i])+ elif isinstance(b, CausalTGrow):+ x = b(x, N, is_first_clip=is_first_chunk)+ else:+ x = b(x)++ NT, C, H, W = x.shape+ return x.view(N, NT // N, C, H, W)++ def decode_chunk(self, x, cache, is_first_chunk=False):+ """Decode ONE chunk of latents, continuing the streaming state.++ Feed latents one (or a few) at a time. The first call with+ ``is_first_chunk=True`` decodes the clip's first latent to a single+ output frame; every subsequent call (``is_first_chunk=False``)+ expands each latent frame to ``t_upscale`` output frames.+ Concatenating the returned frames along the time axis reproduces+ :meth:`decode_video` on the full latent sequence exactly.++ Args:+ x: [N, T_sub, C=latent_channels, H, W] latent chunk.+ cache: list from :meth:`init_cache`, reused across the stream.+ is_first_chunk: True only for the very first chunk.+ Returns:+ [N, T_out, 3, H*8, W*8] RGB frames in [0, 1].+ """+ x = self._apply_decoder_chunk(x, cache, is_first_chunk)+ return self.postprocess_output_frames(x)++ def decode_video_streamed(self, x, show_progress_bar=False):+ """Convenience wrapper: decode a full latent sequence one latent at a+ time through :meth:`decode_chunk` and concatenate.++ Numerically identical to :meth:`decode_video` (up to float rounding).++ Args:+ x: [N, T_lat=n+1, C=latent_channels, H, W] latent tensor.+ Returns:+ [N, T_pix=4n+1, 3, H*8, W*8] RGB tensor in [0, 1].+ """+ T = x.shape[1]+ cache = self.init_cache()+ outs = []+ for i in tqdm(range(T), disable=not show_progress_bar):+ outs.append(self.decode_chunk(x[:, i:i + 1], cache,+ is_first_chunk=(i == 0)))+ return torch.cat(outs, dim=1)++ def forward(self, x, show_progress_bar=False):+ return self.decode_video(x, show_progress_bar=show_progress_bar)+++class CausalTAE(nn.Module):+ """Unified causal TAE for Wan 2.1: encoder + decoder in one module.++ Wraps :class:`CausalTAEEncoder` and :class:`CausalTAEDecoder` so a single+ object exposes the full video<->latent roundtrip. Both one-shot and+ streaming (chunked) paths are provided and are numerically equivalent.++ I/O convention:+ encode: [N, T=4n+1, C=3, H, W] RGB in [0, 1]+ -> [N, T_lat=n+1, latent_channels, H/8, W/8]+ decode: [N, T_lat=n+1, latent_channels, H, W]+ -> [N, T_pix=4n+1, 3, H*8, W*8] RGB in [0, 1]++ Streaming schedules (equivalent to the one-shot paths):+ encode: feed frames as 1, 4, 4, 4, ... (encode_chunk)+ decode: feed latents one at a time (decode_chunk)+ """++ def __init__(self,+ encoder_time_downscale=(True, True, False),+ decoder_time_upscale=(False, True, True),+ decoder_space_upscale=(True, True, True),+ latent_channels=16,+ patch_size=1):+ super().__init__()+ self.latent_channels = latent_channels+ self.patch_size = patch_size+ self.image_channels = 3++ self.enc = CausalTAEEncoder(+ encoder_time_downscale=encoder_time_downscale,+ latent_channels=latent_channels,+ patch_size=patch_size,+ )+ self.dec = CausalTAEDecoder(+ decoder_time_upscale=decoder_time_upscale,+ decoder_space_upscale=decoder_space_upscale,+ latent_channels=latent_channels,+ patch_size=patch_size,+ )+ self.t_downscale = self.enc.t_downscale+ self.t_upscale = self.dec.t_upscale++ # ---- one-shot ----------------------------------------------------++ def encode_video(self, x, show_progress_bar=True):+ """One-shot encode. [N, 4n+1, 3, H, W] -> [N, n+1, C_z, H/8, W/8]."""+ return self.enc.encode_video(x, show_progress_bar=show_progress_bar)++ def decode_video(self, z, show_progress_bar=True):+ """One-shot decode. [N, n+1, C_z, H, W] -> [N, 4n+1, 3, H*8, W*8]."""+ return self.dec.decode_video(z, show_progress_bar=show_progress_bar)++ def forward(self, x, teacher_latent=None, show_progress_bar=False):+ """Two modes:++ * ``teacher_latent is None`` (default): full roundtrip encode->decode,+ returns reconstructed RGB.+ * ``teacher_latent`` given: TRAINING mode. Runs both the encoder+ (supervised by ``teacher_latent``) and the decoder (fed the teacher+ latent, reconstructing ``x``) through this single ``forward`` so that+ DistributedDataParallel sees the whole graph and syncs gradients+ correctly. Returns ``(encoded, decoded)``:+ encoded: [N, T_lat, C_z, H/8, W/8] (compare to teacher_latent)+ decoded: [N, T_pix, 3, H*8, W*8] (compare to x)+ """+ if teacher_latent is None:+ z = self.encode_video(x, show_progress_bar=show_progress_bar)+ return self.decode_video(z, show_progress_bar=show_progress_bar)+ encoded = self.encode_video(x, show_progress_bar=show_progress_bar)+ decoded = self.decode_video(teacher_latent, show_progress_bar=show_progress_bar)+ return encoded, decoded++ # ---- streaming: encode (1, 4, 4, 4, ...) -------------------------++ def init_encode_cache(self):+ return self.enc.init_cache()++ def encode_chunk(self, x, cache, is_first_chunk=False):+ """Streaming encode of one chunk (1 frame first, then 4 each)."""+ return self.enc.encode_chunk(x, cache, is_first_chunk=is_first_chunk)++ def encode_video_chunked(self, x, show_progress_bar=False):+ """Streaming encode over a full 4n+1 clip (== encode_video)."""+ return self.enc.encode_video_chunked(x, show_progress_bar=show_progress_bar)++ # ---- streaming: decode (one latent at a time) --------------------++ def init_decode_cache(self):+ return self.dec.init_cache()++ def decode_chunk(self, z, cache, is_first_chunk=False):+ """Streaming decode of one latent chunk (first -> 1 frame, then+ t_upscale frames each)."""+ return self.dec.decode_chunk(z, cache, is_first_chunk=is_first_chunk)++ def decode_video_streamed(self, z, show_progress_bar=False):+ """Streaming decode over a full latent sequence (== decode_video)."""+ return self.dec.decode_video_streamed(z, show_progress_bar=show_progress_bar)+++# ===========================================================================+# Drop-in adapter exposing the SAME interface as ``worldplay2.vae.wan21.Wan2_1_VAE``+# so ``image2video_stage_two`` can switch VAEs without touching call sites.+# ===========================================================================++class CausalTAEVAE:+ r"""Adapter that wraps :class:`CausalTAE` behind the exact public surface+ of :class:`worldplay2.vae.wan21.Wan2_1_VAE`, so the stage-2/3 pipeline can+ use it as a drop-in replacement with NO changes to its call sites.++ Interface parity (all list-in / list-out, elements are ``[C, T, H, W]``+ tensors WITHOUT a batch dim, pixels in ``[-1, 1]``):+ * ``encode([clip])`` -> ``[z]``+ * ``decode([z])`` -> ``[clip]``+ * ``encode_chunk([clip], is_first_chunk)`` -> ``[z]``+ * ``decode_chunk([z], is_first_chunk)`` -> ``[clip]``+ plus the ``.scale`` / ``.dtype`` / ``.device`` attributes.++ Differences from ``CausalTAE`` that this adapter bridges:+ * LAYOUT: ``Wan2_1_VAE`` uses list of ``[C, T, H, W]``; ``CausalTAE``+ uses batched ``[N, T, C, H, W]``. We add/remove the batch dim and+ move the channel axis.+ * VALUE RANGE: ``Wan2_1_VAE`` pixels are in ``[-1, 1]``; ``CausalTAE``+ pixels are in ``[0, 1]``. We map ``(x+1)/2`` in and ``x*2-1`` out.+ * STREAMING CACHE: ``CausalTAE`` streaming keeps the per-block cache in+ an EXTERNAL list; ``Wan2_1_VAE`` keeps it INTERNAL and re-arms it when+ ``is_first_chunk=True``. We hold the enc/dec caches on the adapter+ and rebuild them on ``is_first_chunk=True`` to reproduce the internal+ semantics exactly.++ Temporal convention matches ``Wan2_1_VAE`` (causal, first-frame aligned):+ ``T_pix = 4 * T_lat - 3`` (== ``CausalTAE``'s ``T_lat = n+1 <-> 4n+1``).+ """++ def __init__(self,+ vae_pth: str,+ z_dim: int = 16,+ dtype=torch.float,+ device="cuda",+ **causal_tae_kwargs):+ self.dtype = dtype+ self.device = device+ # No mean/std normalisation: the CausalTAE latent space is+ # self-consistent (its own decoder inverts its own encoder), so we+ # expose a no-op ``scale`` only for attribute parity with Wan2_1_VAE.+ self.scale = [0.0, 1.0]++ self.model = CausalTAE(+ latent_channels=z_dim, **causal_tae_kwargs+ ).eval().requires_grad_(False)+ if vae_pth is not None:+ sd = torch.load(vae_pth, map_location="cpu", weights_only=True)+ # tolerate a wrapping dict (e.g. {"model": ...} / {"state_dict": ...})+ if isinstance(sd, dict):+ for key in ("state_dict", "model", "module"):+ if key in sd and isinstance(sd[key], dict):+ sd = sd[key]+ break+ missing, unexpected = self.model.load_state_dict(sd, strict=False)+ if missing or unexpected:+ import logging+ logging.info(+ f"CausalTAEVAE: loaded {vae_pth} "+ f"(missing={len(missing)}, unexpected={len(unexpected)})")+ self.model = self.model.to(device)++ # streaming caches (internal, mirroring Wan2_1_VAE's internal state).+ self._enc_cache = None+ self._dec_cache = None++ # ---- layout / range helpers -------------------------------------------++ @staticmethod+ def _to_ntchw_01(clip):+ """``[C, T, H, W]`` in [-1,1] -> ``[1, T, C, H, W]`` in [0,1]."""+ x = clip.permute(1, 0, 2, 3).contiguous() # [T, C, H, W]+ x = x.unsqueeze(0) # [1, T, C, H, W]+ return x.mul(0.5).add(0.5) # [-1,1] -> [0,1]++ @staticmethod+ def _from_ntchw_01(x):+ """``[1, T, C, H, W]`` in [0,1] -> ``[C, T, H, W]`` in [-1,1]."""+ x = x.squeeze(0) # [T, C, H, W]+ x = x.permute(1, 0, 2, 3).contiguous() # [C, T, H, W]+ return x.mul(2.0).sub(1.0).clamp_(-1, 1) # [0,1] -> [-1,1]++ @staticmethod+ def _lat_to_ntchw(z):+ """latent ``[C, T, H, W]`` -> ``[1, T, C, H, W]`` (no range change)."""+ return z.permute(1, 0, 2, 3).contiguous().unsqueeze(0)++ @staticmethod+ def _lat_from_ntchw(z):+ """latent ``[1, T, C, H, W]`` -> ``[C, T, H, W]``."""+ return z.squeeze(0).permute(1, 0, 2, 3).contiguous()++ # ---- one-shot ---------------------------------------------------------++ def encode(self, videos):+ """``videos``: list of ``[3, T_pix, H, W]`` in [-1,1] ->+ list of latent ``[16, T_lat, H/8, W/8]``.++ Uses the STREAMING encode (``encode_video_chunked``: the 1,4,4,4,...+ chunk schedule with a fresh cache per clip) rather than a one-shot+ ``encode_video`` -- numerically identical, but bounds peak memory."""+ with torch.amp.autocast('cuda', dtype=self.dtype), torch.no_grad():+ out = []+ for u in videos:+ x = self._to_ntchw_01(u).to(self.device)+ z = self.model.encode_video_chunked(x, show_progress_bar=False)+ out.append(self._lat_from_ntchw(z).float())+ return out++ def decode(self, zs):+ """``zs``: list of latent ``[16, T_lat, H, W]`` ->+ list of pixel ``[3, T_pix, H*8, W*8]`` in [-1,1].++ Uses the STREAMING decode (``decode_video_streamed``: one latent+ chunk at a time with a fresh cache per clip) rather than a one-shot+ ``decode_video`` -- numerically identical, but bounds peak memory."""+ with torch.amp.autocast('cuda', dtype=self.dtype), torch.no_grad():+ out = []+ for u in zs:+ z = self._lat_to_ntchw(u).to(self.device)+ x = self.model.decode_video_streamed(z, show_progress_bar=False)+ out.append(self._from_ntchw_01(x).float())+ return out++ # ---- streaming --------------------------------------------------------++ def encode_chunk(self, videos, is_first_chunk: bool = False):+ """Streaming encode. Same list I/O as :meth:`encode`. The adapter+ holds the encoder cache internally and re-arms it when+ ``is_first_chunk=True`` (mirroring ``Wan2_1_VAE.encode_chunk``)."""+ if is_first_chunk:+ self._enc_cache = self.model.init_encode_cache()+ with torch.amp.autocast('cuda', dtype=self.dtype), torch.no_grad():+ out = []+ for u in videos:+ x = self._to_ntchw_01(u).to(self.device)+ z = self.model.encode_chunk(+ x, self._enc_cache, is_first_chunk=is_first_chunk)+ out.append(self._lat_from_ntchw(z).float())+ return out++ def decode_chunk(self, zs, is_first_chunk: bool = False):+ """Streaming decode. Same list I/O as :meth:`decode`. The adapter+ holds the decoder cache internally and re-arms it when+ ``is_first_chunk=True`` (mirroring ``Wan2_1_VAE.decode_chunk``)."""+ if is_first_chunk:+ self._dec_cache = self.model.init_decode_cache()+ with torch.amp.autocast('cuda', dtype=self.dtype), torch.no_grad():+ out = []+ for u in zs:+ z = self._lat_to_ntchw(u).to(self.device)+ x = self.model.decode_chunk(+ z, self._dec_cache, is_first_chunk=is_first_chunk)+ out.append(self._from_ntchw_01(x).float())+ return outdiff --git a/worldplay2/vae/interface.py b/worldplay2/vae/interface.pynew file mode 100644index 0000000..1305b0f--- /dev/null+++ b/worldplay2/vae/interface.py@@ -0,0 +1,14 @@+from typing import Protocol++import torch+++class StreamingVAE(Protocol):+ def encode(self, videos: list[torch.Tensor]) -> list[torch.Tensor]: ...+ def decode(self, latents: list[torch.Tensor]) -> list[torch.Tensor]: ...+ def encode_chunk(+ self, videos: list[torch.Tensor], is_first_chunk: bool = False+ ) -> list[torch.Tensor]: ...+ def decode_chunk(+ self, latents: list[torch.Tensor], is_first_chunk: bool = False+ ) -> list[torch.Tensor]: ...diff --git a/worldplay2/vae/wan21.py b/worldplay2/vae/wan21.pynew file mode 100644index 0000000..ac0925d--- /dev/null+++ b/worldplay2/vae/wan21.py@@ -0,0 +1,812 @@+# Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.+import logging++import torch+import torch.cuda.amp as amp+import torch.nn as nn+import torch.nn.functional as F+from einops import rearrange++__all__ = [+ 'Wan2_1_VAE',+]++CACHE_T = 2+++class CausalConv3d(nn.Conv3d):+ """+ Causal 3d convolusion.+ """++ def __init__(self, *args, **kwargs):+ super().__init__(*args, **kwargs)+ self._padding = (self.padding[2], self.padding[2], self.padding[1],+ self.padding[1], 2 * self.padding[0], 0)+ self.padding = (0, 0, 0)++ def forward(self, x, cache_x=None):+ padding = list(self._padding)+ if cache_x is not None and self._padding[4] > 0:+ cache_x = cache_x.to(x.device)+ x = torch.cat([cache_x, x], dim=2)+ padding[4] -= cache_x.shape[2]+ x = F.pad(x, padding)++ return super().forward(x)+++class RMS_norm(nn.Module):++ def __init__(self, dim, channel_first=True, images=True, bias=False):+ super().__init__()+ broadcastable_dims = (1, 1, 1) if not images else (1, 1)+ shape = (dim, *broadcastable_dims) if channel_first else (dim,)++ self.channel_first = channel_first+ self.scale = dim**0.5+ self.gamma = nn.Parameter(torch.ones(shape))+ self.bias = nn.Parameter(torch.zeros(shape)) if bias else 0.++ def forward(self, x):+ return F.normalize(+ x, dim=(1 if self.channel_first else+ -1)) * self.scale * self.gamma + self.bias+++class Upsample(nn.Upsample):++ def forward(self, x):+ """+ Fix bfloat16 support for nearest neighbor interpolation.+ """+ return super().forward(x.float()).type_as(x)+++class Resample(nn.Module):++ def __init__(self, dim, mode):+ assert mode in ('none', 'upsample2d', 'upsample3d', 'downsample2d',+ 'downsample3d')+ super().__init__()+ self.dim = dim+ self.mode = mode++ # layers+ if mode == 'upsample2d':+ self.resample = nn.Sequential(+ Upsample(scale_factor=(2., 2.), mode='nearest-exact'),+ nn.Conv2d(dim, dim // 2, 3, padding=1))+ elif mode == 'upsample3d':+ self.resample = nn.Sequential(+ Upsample(scale_factor=(2., 2.), mode='nearest-exact'),+ nn.Conv2d(dim, dim // 2, 3, padding=1))+ self.time_conv = CausalConv3d(+ dim, dim * 2, (3, 1, 1), padding=(1, 0, 0))++ elif mode == 'downsample2d':+ self.resample = nn.Sequential(+ nn.ZeroPad2d((0, 1, 0, 1)),+ nn.Conv2d(dim, dim, 3, stride=(2, 2)))+ elif mode == 'downsample3d':+ self.resample = nn.Sequential(+ nn.ZeroPad2d((0, 1, 0, 1)),+ nn.Conv2d(dim, dim, 3, stride=(2, 2)))+ self.time_conv = CausalConv3d(+ dim, dim, (3, 1, 1), stride=(2, 1, 1), padding=(0, 0, 0))++ else:+ self.resample = nn.Identity()++ def forward(self, x, feat_cache=None, feat_idx=[0]):+ b, c, t, h, w = x.size()+ if self.mode == 'upsample3d':+ if feat_cache is not None:+ idx = feat_idx[0]+ if feat_cache[idx] is None:+ feat_cache[idx] = 'Rep'+ feat_idx[0] += 1+ else:++ cache_x = x[:, :, -CACHE_T:, :, :].clone()+ if cache_x.shape[2] < 2 and feat_cache[+ idx] is not None and feat_cache[idx] != 'Rep':+ # cache last frame of last two chunk+ cache_x = torch.cat([+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(+ cache_x.device), cache_x+ ],+ dim=2)+ if cache_x.shape[2] < 2 and feat_cache[+ idx] is not None and feat_cache[idx] == 'Rep':+ cache_x = torch.cat([+ torch.zeros_like(cache_x).to(cache_x.device),+ cache_x+ ],+ dim=2)+ if feat_cache[idx] == 'Rep':+ x = self.time_conv(x)+ else:+ x = self.time_conv(x, feat_cache[idx])+ feat_cache[idx] = cache_x+ feat_idx[0] += 1++ x = x.reshape(b, 2, c, t, h, w)+ x = torch.stack((x[:, 0, :, :, :, :], x[:, 1, :, :, :, :]),+ 3)+ x = x.reshape(b, c, t * 2, h, w)+ t = x.shape[2]+ x = rearrange(x, 'b c t h w -> (b t) c h w')+ x = self.resample(x)+ x = rearrange(x, '(b t) c h w -> b c t h w', t=t)++ if self.mode == 'downsample3d':+ if feat_cache is not None:+ idx = feat_idx[0]+ if feat_cache[idx] is None:+ feat_cache[idx] = x.clone()+ feat_idx[0] += 1+ else:++ cache_x = x[:, :, -1:, :, :].clone()+ # if cache_x.shape[2] < 2 and feat_cache[idx] is not None and feat_cache[idx]!='Rep':+ # # cache last frame of last two chunk+ # cache_x = torch.cat([feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(cache_x.device), cache_x], dim=2)++ x = self.time_conv(+ torch.cat([feat_cache[idx][:, :, -1:, :, :], x], 2))+ feat_cache[idx] = cache_x+ feat_idx[0] += 1+ return x++ def init_weight(self, conv):+ conv_weight = conv.weight+ nn.init.zeros_(conv_weight)+ c1, c2, t, h, w = conv_weight.size()+ one_matrix = torch.eye(c1, c2)+ init_matrix = one_matrix+ nn.init.zeros_(conv_weight)+ #conv_weight.data[:,:,-1,1,1] = init_matrix * 0.5+ conv_weight.data[:, :, 1, 0, 0] = init_matrix #* 0.5+ conv.weight.data.copy_(conv_weight)+ nn.init.zeros_(conv.bias.data)++ def init_weight2(self, conv):+ conv_weight = conv.weight.data+ nn.init.zeros_(conv_weight)+ c1, c2, t, h, w = conv_weight.size()+ init_matrix = torch.eye(c1 // 2, c2)+ #init_matrix = repeat(init_matrix, 'o ... -> (o 2) ...').permute(1,0,2).contiguous().reshape(c1,c2)+ conv_weight[:c1 // 2, :, -1, 0, 0] = init_matrix+ conv_weight[c1 // 2:, :, -1, 0, 0] = init_matrix+ conv.weight.data.copy_(conv_weight)+ nn.init.zeros_(conv.bias.data)+++class ResidualBlock(nn.Module):++ def __init__(self, in_dim, out_dim, dropout=0.0):+ super().__init__()+ self.in_dim = in_dim+ self.out_dim = out_dim++ # layers+ self.residual = nn.Sequential(+ RMS_norm(in_dim, images=False), nn.SiLU(),+ CausalConv3d(in_dim, out_dim, 3, padding=1),+ RMS_norm(out_dim, images=False), nn.SiLU(), nn.Dropout(dropout),+ CausalConv3d(out_dim, out_dim, 3, padding=1))+ self.shortcut = CausalConv3d(in_dim, out_dim, 1) \+ if in_dim != out_dim else nn.Identity()++ def forward(self, x, feat_cache=None, feat_idx=[0]):+ h = self.shortcut(x)+ for layer in self.residual:+ if isinstance(layer, CausalConv3d) and feat_cache is not None:+ idx = feat_idx[0]+ cache_x = x[:, :, -CACHE_T:, :, :].clone()+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:+ # cache last frame of last two chunk+ cache_x = torch.cat([+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(+ cache_x.device), cache_x+ ],+ dim=2)+ x = layer(x, feat_cache[idx])+ feat_cache[idx] = cache_x+ feat_idx[0] += 1+ else:+ x = layer(x)+ return x + h+++class AttentionBlock(nn.Module):+ """+ Causal self-attention with a single head.+ """++ def __init__(self, dim):+ super().__init__()+ self.dim = dim++ # layers+ self.norm = RMS_norm(dim)+ self.to_qkv = nn.Conv2d(dim, dim * 3, 1)+ self.proj = nn.Conv2d(dim, dim, 1)++ # zero out the last layer params+ nn.init.zeros_(self.proj.weight)++ def forward(self, x):+ identity = x+ b, c, t, h, w = x.size()+ x = rearrange(x, 'b c t h w -> (b t) c h w')+ x = self.norm(x)+ # compute query, key, value+ q, k, v = self.to_qkv(x).reshape(b * t, 1, c * 3,+ -1).permute(0, 1, 3,+ 2).contiguous().chunk(+ 3, dim=-1)++ # apply attention+ x = F.scaled_dot_product_attention(+ q,+ k,+ v,+ )+ x = x.squeeze(1).permute(0, 2, 1).reshape(b * t, c, h, w)++ # output+ x = self.proj(x)+ x = rearrange(x, '(b t) c h w-> b c t h w', t=t)+ return x + identity+++class Encoder3d(nn.Module):++ def __init__(self,+ dim=128,+ z_dim=4,+ dim_mult=[1, 2, 4, 4],+ num_res_blocks=2,+ attn_scales=[],+ temperal_downsample=[True, True, False],+ dropout=0.0):+ super().__init__()+ self.dim = dim+ self.z_dim = z_dim+ self.dim_mult = dim_mult+ self.num_res_blocks = num_res_blocks+ self.attn_scales = attn_scales+ self.temperal_downsample = temperal_downsample++ # dimensions+ dims = [dim * u for u in [1] + dim_mult]+ scale = 1.0++ # init block+ self.conv1 = CausalConv3d(3, dims[0], 3, padding=1)++ # downsample blocks+ downsamples = []+ for i, (in_dim, out_dim) in enumerate(zip(dims[:-1], dims[1:])):+ # residual (+attention) blocks+ for _ in range(num_res_blocks):+ downsamples.append(ResidualBlock(in_dim, out_dim, dropout))+ if scale in attn_scales:+ downsamples.append(AttentionBlock(out_dim))+ in_dim = out_dim++ # downsample block+ if i != len(dim_mult) - 1:+ mode = 'downsample3d' if temperal_downsample[+ i] else 'downsample2d'+ downsamples.append(Resample(out_dim, mode=mode))+ scale /= 2.0+ self.downsamples = nn.Sequential(*downsamples)++ # middle blocks+ self.middle = nn.Sequential(+ ResidualBlock(out_dim, out_dim, dropout), AttentionBlock(out_dim),+ ResidualBlock(out_dim, out_dim, dropout))++ # output blocks+ self.head = nn.Sequential(+ RMS_norm(out_dim, images=False), nn.SiLU(),+ CausalConv3d(out_dim, z_dim, 3, padding=1))++ def forward(self, x, feat_cache=None, feat_idx=[0]):+ if feat_cache is not None:+ idx = feat_idx[0]+ cache_x = x[:, :, -CACHE_T:, :, :].clone()+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:+ # cache last frame of last two chunk+ cache_x = torch.cat([+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(+ cache_x.device), cache_x+ ],+ dim=2)+ x = self.conv1(x, feat_cache[idx])+ feat_cache[idx] = cache_x+ feat_idx[0] += 1+ else:+ x = self.conv1(x)++ ## downsamples+ for layer in self.downsamples:+ if feat_cache is not None:+ x = layer(x, feat_cache, feat_idx)+ else:+ x = layer(x)++ ## middle+ for layer in self.middle:+ if isinstance(layer, ResidualBlock) and feat_cache is not None:+ x = layer(x, feat_cache, feat_idx)+ else:+ x = layer(x)++ ## head+ for layer in self.head:+ if isinstance(layer, CausalConv3d) and feat_cache is not None:+ idx = feat_idx[0]+ cache_x = x[:, :, -CACHE_T:, :, :].clone()+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:+ # cache last frame of last two chunk+ cache_x = torch.cat([+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(+ cache_x.device), cache_x+ ],+ dim=2)+ x = layer(x, feat_cache[idx])+ feat_cache[idx] = cache_x+ feat_idx[0] += 1+ else:+ x = layer(x)+ return x+++class Decoder3d(nn.Module):++ def __init__(self,+ dim=128,+ z_dim=4,+ dim_mult=[1, 2, 4, 4],+ num_res_blocks=2,+ attn_scales=[],+ temperal_upsample=[False, True, True],+ dropout=0.0):+ super().__init__()+ self.dim = dim+ self.z_dim = z_dim+ self.dim_mult = dim_mult+ self.num_res_blocks = num_res_blocks+ self.attn_scales = attn_scales+ self.temperal_upsample = temperal_upsample++ # dimensions+ dims = [dim * u for u in [dim_mult[-1]] + dim_mult[::-1]]+ scale = 1.0 / 2**(len(dim_mult) - 2)++ # init block+ self.conv1 = CausalConv3d(z_dim, dims[0], 3, padding=1)++ # middle blocks+ self.middle = nn.Sequential(+ ResidualBlock(dims[0], dims[0], dropout), AttentionBlock(dims[0]),+ ResidualBlock(dims[0], dims[0], dropout))++ # upsample blocks+ upsamples = []+ for i, (in_dim, out_dim) in enumerate(zip(dims[:-1], dims[1:])):+ # residual (+attention) blocks+ if i == 1 or i == 2 or i == 3:+ in_dim = in_dim // 2+ for _ in range(num_res_blocks + 1):+ upsamples.append(ResidualBlock(in_dim, out_dim, dropout))+ if scale in attn_scales:+ upsamples.append(AttentionBlock(out_dim))+ in_dim = out_dim++ # upsample block+ if i != len(dim_mult) - 1:+ mode = 'upsample3d' if temperal_upsample[i] else 'upsample2d'+ upsamples.append(Resample(out_dim, mode=mode))+ scale *= 2.0+ self.upsamples = nn.Sequential(*upsamples)++ # output blocks+ self.head = nn.Sequential(+ RMS_norm(out_dim, images=False), nn.SiLU(),+ CausalConv3d(out_dim, 3, 3, padding=1))++ def forward(self, x, feat_cache=None, feat_idx=[0]):+ ## conv1+ if feat_cache is not None:+ idx = feat_idx[0]+ cache_x = x[:, :, -CACHE_T:, :, :].clone()+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:+ # cache last frame of last two chunk+ cache_x = torch.cat([+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(+ cache_x.device), cache_x+ ],+ dim=2)+ x = self.conv1(x, feat_cache[idx])+ feat_cache[idx] = cache_x+ feat_idx[0] += 1+ else:+ x = self.conv1(x)++ ## middle+ for layer in self.middle:+ if isinstance(layer, ResidualBlock) and feat_cache is not None:+ x = layer(x, feat_cache, feat_idx)+ else:+ x = layer(x)++ ## upsamples+ for layer in self.upsamples:+ if feat_cache is not None:+ x = layer(x, feat_cache, feat_idx)+ else:+ x = layer(x)++ ## head+ for layer in self.head:+ if isinstance(layer, CausalConv3d) and feat_cache is not None:+ idx = feat_idx[0]+ cache_x = x[:, :, -CACHE_T:, :, :].clone()+ if cache_x.shape[2] < 2 and feat_cache[idx] is not None:+ # cache last frame of last two chunk+ cache_x = torch.cat([+ feat_cache[idx][:, :, -1, :, :].unsqueeze(2).to(+ cache_x.device), cache_x+ ],+ dim=2)+ x = layer(x, feat_cache[idx])+ feat_cache[idx] = cache_x+ feat_idx[0] += 1+ else:+ x = layer(x)+ return x+++def count_conv3d(model):+ count = 0+ for m in model.modules():+ if isinstance(m, CausalConv3d):+ count += 1+ return count+++class WanVAE_(nn.Module):++ def __init__(self,+ dim=128,+ z_dim=4,+ dim_mult=[1, 2, 4, 4],+ num_res_blocks=2,+ attn_scales=[],+ temperal_downsample=[True, True, False],+ dropout=0.0):+ super().__init__()+ self.dim = dim+ self.z_dim = z_dim+ self.dim_mult = dim_mult+ self.num_res_blocks = num_res_blocks+ self.attn_scales = attn_scales+ self.temperal_downsample = temperal_downsample+ self.temperal_upsample = temperal_downsample[::-1]++ # modules+ self.encoder = Encoder3d(dim, z_dim * 2, dim_mult, num_res_blocks,+ attn_scales, self.temperal_downsample, dropout)+ self.conv1 = CausalConv3d(z_dim * 2, z_dim * 2, 1)+ self.conv2 = CausalConv3d(z_dim, z_dim, 1)+ self.decoder = Decoder3d(dim, z_dim, dim_mult, num_res_blocks,+ attn_scales, self.temperal_upsample, dropout)++ def forward(self, x):+ mu, log_var = self.encode(x)+ z = self.reparameterize(mu, log_var)+ x_recon = self.decode(z)+ return x_recon, mu, log_var++ def encode(self, x, scale):+ self.clear_encode_cache()+ ## cache+ t = x.shape[2]+ iter_ = 1 + (t - 1) // 4+ ## 对encode输入的x,按时间拆分为1、4、4、4....+ for i in range(iter_):+ self._enc_conv_idx = [0]+ if i == 0:+ out = self.encoder(+ x[:, :, :1, :, :],+ feat_cache=self._enc_feat_map,+ feat_idx=self._enc_conv_idx)+ else:+ out_ = self.encoder(+ x[:, :, 1 + 4 * (i - 1):1 + 4 * i, :, :],+ feat_cache=self._enc_feat_map,+ feat_idx=self._enc_conv_idx)+ out = torch.cat([out, out_], 2)+ mu, log_var = self.conv1(out).chunk(2, dim=1)+ if isinstance(scale[0], torch.Tensor):+ mu = (mu - scale[0].view(1, self.z_dim, 1, 1, 1)) * scale[1].view(+ 1, self.z_dim, 1, 1, 1)+ else:+ mu = (mu - scale[0]) * scale[1]+ self.clear_encode_cache()+ return mu++ def decode(self, z, scale):+ self.clear_decode_cache()+ # z: [b,c,t,h,w]+ if isinstance(scale[0], torch.Tensor):+ z = z / scale[1].view(1, self.z_dim, 1, 1, 1) + scale[0].view(+ 1, self.z_dim, 1, 1, 1)+ else:+ z = z / scale[1] + scale[0]+ iter_ = z.shape[2]+ x = self.conv2(z)+ for i in range(iter_):+ self._conv_idx = [0]+ if i == 0:+ out = self.decoder(+ x[:, :, i:i + 1, :, :],+ feat_cache=self._feat_map,+ feat_idx=self._conv_idx)+ else:+ out_ = self.decoder(+ x[:, :, i:i + 1, :, :],+ feat_cache=self._feat_map,+ feat_idx=self._conv_idx)+ out = torch.cat([out, out_], 2)+ self.clear_decode_cache()+ return out++ def encode_chunk(self, x, scale, is_first_chunk: bool = False):+ """Streaming encode for chunked inference.++ Unlike :meth:`encode`, this does **not** ``clear_cache()`` between+ calls; the per-layer ``_enc_feat_map`` accumulated by previous+ chunks is reused so the causal convolutions see the prior temporal+ state.++ Frame-to-latent layout:+ - is_first_chunk=True : first pixel frame -> 1 latent, then+ groups of 4 pixel frames -> 1 latent each. ``iter_ = 1 + (T-1)//4``.+ - is_first_chunk=False : all input frames are continuation+ frames coming in groups of 4. ``iter_ = T // 4``.++ Args:+ x: pixel tensor of shape [B, C, T, H, W].+ scale: (mean, 1/std) tuple matching :meth:`encode`.+ is_first_chunk: True for the very first chunk; resets the+ encoder cache before processing.+ """+ if is_first_chunk:+ self.clear_encode_cache()++ t = x.shape[2]+ if is_first_chunk:+ iter_ = 1 + (t - 1) // 4+ else:+ assert t % 4 == 0, (+ f"encode_chunk(is_first_chunk=False) requires the number of "+ f"pixel frames to be a multiple of 4; got T={t}.")+ iter_ = t // 4++ out = None+ for i in range(iter_):+ self._enc_conv_idx = [0]+ if is_first_chunk:+ if i == 0:+ frame_slice = x[:, :, :1, :, :]+ else:+ frame_slice = x[:, :, 1 + 4 * (i - 1):1 + 4 * i, :, :]+ else:+ frame_slice = x[:, :, 4 * i:4 * (i + 1), :, :]++ chunk_out = self.encoder(+ frame_slice,+ feat_cache=self._enc_feat_map,+ feat_idx=self._enc_conv_idx)+ out = chunk_out if out is None else torch.cat([out, chunk_out], 2)++ mu, log_var = self.conv1(out).chunk(2, dim=1)+ if isinstance(scale[0], torch.Tensor):+ mu = (mu - scale[0].view(1, self.z_dim, 1, 1, 1)) * scale[1].view(+ 1, self.z_dim, 1, 1, 1)+ else:+ mu = (mu - scale[0]) * scale[1]+ # NOTE: cache is intentionally NOT cleared at the end so the next+ # chunk continues the streaming state.+ return mu++ def decode_chunk(self, z, scale, is_first_chunk: bool = False):+ """Streaming decode for chunked inference.++ Unlike :meth:`decode`, this does **not** ``clear_cache()`` between+ calls. ``is_first_chunk=True`` resets the decoder cache once at+ the very first chunk; subsequent calls reuse the cached state so+ the causal temporal convolutions stay consistent across chunk+ boundaries.++ Args:+ z: latent tensor of shape [B, C, T_lat, H, W] for a single chunk.+ scale: (mean, 1/std) tuple matching :meth:`decode`.+ is_first_chunk: True for the very first chunk.++ Returns:+ Tensor of shape [B, 3, T_pix, H_pix, W_pix]. Not clamped here+ (the public wrapper :meth:`Wan2_1_VAE.decode_chunk` clamps to+ [-1, 1] to mirror :meth:`Wan2_1_VAE.decode`).+ """+ if is_first_chunk:+ self.clear_decode_cache()++ if isinstance(scale[0], torch.Tensor):+ z = z / scale[1].view(1, self.z_dim, 1, 1, 1) + scale[0].view(+ 1, self.z_dim, 1, 1, 1)+ else:+ z = z / scale[1] + scale[0]++ iter_ = z.shape[2]+ x = self.conv2(z)+ out = None+ for i in range(iter_):+ self._conv_idx = [0]+ cur = self.decoder(+ x[:, :, i:i + 1, :, :],+ feat_cache=self._feat_map,+ feat_idx=self._conv_idx)+ out = cur if out is None else torch.cat([out, cur], 2)+ # NOTE: cache is intentionally NOT cleared at the end.+ return out++ def reparameterize(self, mu, log_var):+ std = torch.exp(0.5 * log_var)+ eps = torch.randn_like(std)+ return eps * std + mu++ def sample(self, imgs, deterministic=False):+ mu, log_var = self.encode(imgs)+ if deterministic:+ return mu+ std = torch.exp(0.5 * log_var.clamp(-30.0, 20.0))+ return mu + std * torch.randn_like(std)++ def clear_cache(self):+ """Reset BOTH encode and decode streaming caches.++ Kept for backwards-compatibility (the non-streaming+ :meth:`encode` / :meth:`decode` paths use it as a "scratch" reset+ because they do not interleave with streaming calls). For+ chunked / streaming inference, prefer :meth:`clear_encode_cache`+ and :meth:`clear_decode_cache` so that resetting the encode side+ does not nuke an in-flight decode cache (and vice versa).+ """+ self.clear_decode_cache()+ self.clear_encode_cache()++ def clear_decode_cache(self):+ self._conv_num = count_conv3d(self.decoder)+ self._conv_idx = [0]+ self._feat_map = [None] * self._conv_num++ def clear_encode_cache(self):+ self._enc_conv_num = count_conv3d(self.encoder)+ self._enc_conv_idx = [0]+ self._enc_feat_map = [None] * self._enc_conv_num+++def _video_vae(pretrained_path=None, z_dim=None, device='cpu', **kwargs):+ """+ Autoencoder3d adapted from Stable Diffusion 1.x, 2.x and XL.+ """+ # params+ cfg = dict(+ dim=96,+ z_dim=z_dim,+ dim_mult=[1, 2, 4, 4],+ num_res_blocks=2,+ attn_scales=[],+ temperal_downsample=[False, True, True],+ dropout=0.0)+ cfg.update(**kwargs)++ # init model+ with torch.device('meta'):+ model = WanVAE_(**cfg)++ # load checkpoint+ logging.info(f'loading {pretrained_path}')+ model.load_state_dict(+ torch.load(pretrained_path, map_location=device), assign=True)++ return model+++class Wan2_1_VAE:++ def __init__(self,+ z_dim=16,+ vae_pth='cache/vae_step_411000.pth',+ dtype=torch.float,+ device="cuda"):+ self.dtype = dtype+ self.device = device++ mean = [+ -0.7571, -0.7089, -0.9113, 0.1075, -0.1745, 0.9653, -0.1517, 1.5508,+ 0.4134, -0.0715, 0.5517, -0.3632, -0.1922, -0.9497, 0.2503, -0.2921+ ]+ std = [+ 2.8184, 1.4541, 2.3275, 2.6558, 1.2196, 1.7708, 2.6052, 2.0743,+ 3.2687, 2.1526, 2.8652, 1.5579, 1.6382, 1.1253, 2.8251, 1.9160+ ]+ self.mean = torch.tensor(mean, dtype=dtype, device=device)+ self.std = torch.tensor(std, dtype=dtype, device=device)+ self.scale = [self.mean, 1.0 / self.std]++ # init model+ self.model = _video_vae(+ pretrained_path=vae_pth,+ z_dim=z_dim,+ ).eval().requires_grad_(False).to(device)++ def encode(self, videos):+ """+ videos: A list of videos each with shape [C, T, H, W].+ """+ with amp.autocast(dtype=self.dtype):+ return [+ self.model.encode(u.unsqueeze(0), self.scale).float().squeeze(0)+ for u in videos+ ]++ def decode(self, zs):+ with amp.autocast(dtype=self.dtype):+ return [+ self.model.decode(u.unsqueeze(0),+ self.scale).float().clamp_(-1, 1).squeeze(0)+ for u in zs+ ]++ # ------------------------------------------------------------------+ # Streaming variants for chunked inference (stage-2 / stage-3).+ # See ``WanVAE_.encode_chunk`` / ``decode_chunk`` for the cache contract.+ # ------------------------------------------------------------------++ def encode_chunk(self, videos, is_first_chunk: bool = False):+ """Streaming encode. ``videos`` is a list of [C, T, H, W] tensors.+ Returns a list of latent tensors [C_z, T_lat, H/8, W/8].++ Caller is responsible for setting ``is_first_chunk=True`` exactly+ once at the start of an autoregressive generation and ``False``+ for every subsequent chunk.+ """+ with amp.autocast(dtype=self.dtype):+ return [+ self.model.encode_chunk(+ u.unsqueeze(0), self.scale,+ is_first_chunk=is_first_chunk).float().squeeze(0)+ for u in videos+ ]++ def decode_chunk(self, zs, is_first_chunk: bool = False):+ """Streaming decode. ``zs`` is a list of [C_z, T_lat, H, W] latent+ tensors. Returns a list of pixel tensors [3, T_pix, H_pix, W_pix]+ clamped to [-1, 1] (matching :meth:`decode`).+ """+ with amp.autocast(dtype=self.dtype):+ return [+ self.model.decode_chunk(+ u.unsqueeze(0), self.scale,+ is_first_chunk=is_first_chunk+ ).float().clamp_(-1, 1).squeeze(0)+ for u in zs+ ]diff --git a/worldplay2/video.py b/worldplay2/video.pynew file mode 100644index 0000000..320fbf9--- /dev/null+++ b/worldplay2/video.py@@ -0,0 +1,22 @@+from pathlib import Path++import imageio+import torch+++def save_video(tensor: torch.Tensor, path: str | Path, fps: int) -> None:+ """Save ``[3,T,H,W]`` video in ``[-1,1]``."""+ output = Path(path)+ output.parent.mkdir(parents=True, exist_ok=True)+ frames = (+ tensor.detach()+ .float()+ .clamp(-1, 1)+ .add(1)+ .mul(127.5)+ .byte()+ .permute(1, 2, 3, 0)+ .cpu()+ .numpy()+ )+ imageio.mimsave(output, frames, fps=fps, codec="libx264", quality=8)