[SIGGRAPH Asia 2026] 4DAnyone: Create Anyone in 4D from a Casual Monocular Video
Release: refresh inference performance documentation (#29)
Yudong Jin28c5c3828951
diff --git a/README.md b/README.mdindex 225333a..5d21b8e 100644--- a/README.md+++ b/README.md@@ -8,11 +8,19 @@<p align="center">4DAnyone turns a casual monocular video into multi-view videos, enabling downstream 4DGS reconstruction.</p>+> [!note]+> 4DAnyone is a multi-view video model that:+>+> - generates dozens of synchronized, view-consistent videos from a single monocular video.+> - requires **22 GB** of peak CUDA memory, enabling inference on consumer GPUs.+> - generates a 121-frame video in **27 seconds** on a single RTX 4090.+## News-- **2026-09-02**: Released **4DAnyone-Turbo**, achieving a **5.58×** denoising speedup over the base 4DAnyone model. See [Inference performance](docs/inference_performance.md) for details.+- **2026-09-05**: Reduced peak GPU memory below **24 GB**, enabling inference on consumer GPUs (RTX 4090).+- **2026-09-02**: Released **4DAnyone-Turbo**, achieving a **5.58×** denoising speedup over 4DAnyone-Base.- **2026-08-28**: Achieved a **1.42×** end-to-end speedup for the complete 24-view generation pipeline.-- **2026-08-28**: Reduced peak GPU memory to **25.4 GiB** while slightly improving speed.+- **2026-08-28**: Reduced peak GPU memory below **32 GB** while slightly improving speed.## Installation@@ -26,6 +34,8 @@ conda activate 4danyonepip install -r requirements.txt```+For faster inference, optionally install [FlashAttention-3](https://github.com/Dao-AILab/flash-attention/tree/main/hopper) or [SageAttention](https://github.com/thu-ml/SageAttention). The installed backend is enabled automatically.+Missing models and examples are downloaded automatically on first use. You can also download them manually:```bash@@ -36,7 +46,7 @@ python scripts/download_example.py## Inference-We provide two models: **4DAnyone-Turbo** for faster four-step denoising and **4DAnyone-Base** with the standard denoising schedule. Both models achieve comparable generation quality. 4DAnyone-Turbo is enabled by default. Set `--enable_turbo=False` to use 4DAnyone-Base.+This repository provides two models: **4DAnyone-Base** with the standard denoising schedule and the distilled **4DAnyone-Turbo** for faster four-step denoising. 4DAnyone-Turbo is enabled by default for faster inference while maintaining generation quality comparable to 4DAnyone-Base. See [Inference performance](docs/inference_performance.md) for GPU memory, inference speed, and generation quality benchmarks.4DAnyone supports flexible target-view counts, pitch layers, and yaw coverage. Here are several common camera configurations:@@ -115,21 +125,13 @@ data/└── dense/00.mp4 ... <N-1>.mp4 # generated target views```-### Inference Efficiency--For faster inference on supported GPUs, optionally install [FlashAttention-3](https://github.com/Dao-AILab/flash-attention/tree/main/hopper) or [SageAttention](https://github.com/thu-ml/SageAttention). Runtime selection follows one fixed order: FlashAttention-3, SageAttention, then PyTorch SDPA.--See [Inference performance](docs/inference_performance.md) for measured 6-view runtimes and peak GPU memory usage on H20-3E, H200, RTX 5880 Ada, and RTX A6000 GPUs.-### Custom Data-Use an input video that:+Use an input video with:-- is 720p or higher, with 1080p recommended.-- uses a 9:16 portrait aspect ratio.-- shows one person in a full-body or upper-body shot.-- has at least 121 frames.-- contains only mild camera motion.+- a single person in a full-body or upper-body shot.+- no large camera movements, clear footage.+- 1080p or higher, 9:16 portrait aspect ratio, at least 121 frames.## 3DGS Reconstruction@@ -139,12 +141,13 @@ See the [nerfstudio guide](docs/nerfstudio.md) for details.### Peak Memory Optimization-- [x] Reduce peak GPU memory below 32 GB through pose precomputation and memory-efficient operators.+- [x] Reduce peak GPU memory below 32 GB.+- [x] Further reduce peak GPU memory below 24 GB.### Inference Acceleration-- [x] Achieve up to 1.42× end-to-end speedup by parallelizing pose encoding and VAE processing across GPUs.-- [x] Achieve a **5.58×** denoising speedup with 4DAnyone-Turbo.+- [x] Optimize inference speed through multi-GPU parallelism.+- [x] Accelerate inference via few-step model distillation.### Reconstructiondiff --git a/docs/inference_performance.md b/docs/inference_performance.mdindex 9ad08ef..1a74eda 100644--- a/docs/inference_performance.md+++ b/docs/inference_performance.md@@ -1,21 +1,45 @@-# Inference performance+# Inference Performance-We compare **4DAnyone-Turbo**, the default accelerated model, with **4DAnyone-Base**, the original model described in the paper.+This report provides practical runtime and GPU-memory references for running 4DAnyone on common GPUs.-## Quality comparison+## Turbo vs. Base-We evaluate 32 validation scenes: 16 from DNA-Rendering, 8 from MVGameHuman, and 8 from SynCamVideo. Both models use identical target cameras, 25-frame clips, seed 42, and foreground-masked metrics; 4DAnyone-Turbo uses 4 denoising steps with scheduler shift 5, while 4DAnyone-Base uses 50 steps, serving as an upper-bound quality reference for our method.+### Generation Quality++The 4-step Turbo model achieves quality comparable to a 50-step Base reference:| Model | Denoising steps | PSNR ↑ | SSIM ↑ | LPIPS ↓ || --- | ---: | ---: | ---: | ---: |-| 4DAnyone-Turbo | 4 | 25.48 | 0.895 | **0.072** |-| 4DAnyone-Base | 50 | **25.69** | **0.899** | 0.073 |+| 4DAnyone-Turbo | 4 | 25.48 | 0.895 | 0.072 |+| 4DAnyone-Base | 50 | 25.69 | 0.899 | 0.073 |++### Denoising Speedup-The two models achieve comparable quality across PSNR, SSIM, and LPIPS.+For the 6-view example, Turbo uses 4 denoising steps and Base uses 24. The values show Turbo's speedup over Base on the same GPU and attention backend.-## Denoising performance+| GPU | SDPA | FlashAttention-3 | SageAttention |+| --- | ---: | ---: | ---: |+| H200 | 5.49× | 5.35× | 5.12× |+| H20-3E | 5.83× | 5.78× | 5.73× |+| RTX 5880 Ada | 5.79× | — | 5.80× |+| RTX A6000 | 5.77× | — | 5.75× |+| RTX 4090 | 5.85× | — | 5.78× |-We measure 4DAnyone-Turbo with 4 denoising steps and 4DAnyone-Base with 24 steps on the [6-view full-orbit](../README.md#6-view-full-orbit) example:+## 4DAnyone-Turbo Performance++The performance tables below use the following terms:++- `SDPA`, `FlashAttention-3`, and `SageAttention`: denoising time for each attention backend.+- `Peak CUDA allocated`: maximum live tensor memory during generation.+- `GPU memory`: total memory capacity per GPU.+- `—`: backend or workload unavailable on that GPU.++> [!note]+> SageAttention results use the official `sageattn` entry point, which automatically dispatches an architecture-specific kernel for each GPU.++### 6-View Full Orbit++The [6-view full-orbit](../README.md#6-view-full-orbit) benchmark uses the following command:```bashpython inference.py \@@ -23,57 +47,64 @@ python inference.py \--views_per_layer 6```-Each reported denoising time is the median of three complete runs from the same source revision.--These tables compare denoising time across attention backends for both models.+| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GB) | GPU memory (GB) |+| --- | ---: | ---: | ---: | ---: | ---: |+| H200 | 0.97 | 0.76 | 0.80 | 24.06 | 139.81 |+| H20-3E | 3.21 | 2.54 | 2.09 | 24.06 | 139.81 |+| RTX 5880 Ada | 3.47 | — | 2.99 | 24.06 | 47.37 |+| RTX A6000 | 3.78 | — | 3.69 | 24.06 | 47.53 |+| RTX 4090 | 2.71 | — | 2.13 | 22.18 | 23.52 |-- `SDPA`, `FlashAttention-3`, and `SageAttention`: denoising time using each attention backend.-- `Peak CUDA allocated`: maximum live tensor allocation across all runs and available attention backends.-- `GPU memory`: total device memory capacity.-- `—`: backend unavailable on that GPU.+### 24-View Full Orbit-### 4DAnyone-Turbo+The [24-view full-orbit](../README.md#24-view-full-orbit) benchmark runs 4DAnyone-Turbo on a single GPU:-| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GiB) | GPU memory (GiB) |-| --- | ---: | ---: | ---: | ---: | ---: |-| H200 | 0.98 | 0.77 | 0.78 | 26.63 | 139.81 |-| H20-3E | 3.22 | 2.55 | 2.10 | 26.63 | 139.81 |-| RTX 5880 Ada | 3.51 | — | 3.04 | 25.47 | 47.37 |-| RTX A6000 | 3.80 | — | 3.71 | 24.91 | 47.53 |+```bash+python inference.py \+ --video_path "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4" \+ --views_per_layer 24+```-### 4DAnyone-Base+For 24-view generation, the pipeline first denoises six RCP views to use as references and then denoises the 24 target views. The table reports the total time for both stages.-| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GiB) | GPU memory (GiB) |+| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GB) | GPU memory (GB) || --- | ---: | ---: | ---: | ---: | ---: |-| H200 | 5.35 | 4.03 | 4.09 | 26.63 | 139.81 |-| H20-3E | 18.71 | 14.73 | 11.98 | 26.63 | 139.81 |-| RTX 5880 Ada | 20.40 | — | 17.58 | 25.47 | 47.37 |-| RTX A6000 | 22.02 | — | 21.42 | 24.91 | 47.53 |+| H200 | 5.11 | 3.90 | 3.90 | 24.06 | 139.81 |+| H20-3E | 17.66 | 13.86 | 11.24 | 24.06 | 139.81 |+| RTX 5880 Ada | 18.84 | — | 16.14 | 24.06 | 47.37 |+| RTX A6000 | 20.55 | — | 19.98 | 24.06 | 47.53 |+| RTX 4090 | 16.20 | — | — | 22.18 | 23.52 |-### Speedup+## 4DAnyone-Base Performance-Speedup is calculated by dividing the 4DAnyone-Base median by the 4DAnyone-Turbo median for the same GPU and attention backend.+The 6-view 4DAnyone-Base benchmark uses 24 denoising steps:-| GPU | SDPA | FlashAttention-3 | SageAttention |-| --- | ---: | ---: | ---: |-| H200 | 5.45× | 5.26× | 5.23× |-| H20-3E | 5.81× | 5.78× | 5.69× |-| RTX 5880 Ada | 5.81× | — | 5.79× |-| RTX A6000 | 5.79× | — | 5.77× |+```bash+python inference.py \+ --video_path "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4" \+ --views_per_layer 6 \+ --enable_turbo=False+```-> [!NOTE]-> SageAttention results use the official `sageattn` entry point, which automatically dispatches an architecture-specific kernel for each GPU.+| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GB) | GPU memory (GB) |+| --- | ---: | ---: | ---: | ---: | ---: |+| H200 | 5.33 | 4.05 | 4.09 | 24.06 | 139.81 |+| H20-3E | 18.68 | 14.70 | 11.98 | 24.06 | 139.81 |+| RTX 5880 Ada | 20.10 | — | 17.36 | 24.06 | 47.37 |+| RTX A6000 | 21.82 | — | 21.20 | 24.06 | 47.53 |+| RTX 4090 | 15.86 | — | 12.29 | 22.18 | 23.52 |-## Preprocessing performance+## Preprocessing Performance-Preprocessing is identical for both models and therefore reported once.+GVHMR and skeleton extraction run before denoising and are shared by both models. These 6-view timings can vary with CPU and storage performance.- `GVHMR`: tracking, ViTPose, image-feature extraction, and motion prediction.- `Skeleton extraction`: target-view skeleton-condition rendering.| GPU | GVHMR (min) | Skeleton extraction (min) || --- | ---: | ---: |-| H200 | 0.94 | 0.57 |-| H20-3E | 0.97 | 0.55 |-| RTX 5880 Ada | 0.94 | 1.00 |-| RTX A6000 | 1.70 | 1.06 |+| H200 | 1.20 | 1.25 |+| H20-3E | 1.08 | 1.27 |+| RTX 5880 Ada | 0.98 | 1.00 |+| RTX A6000 | 1.57 | 2.12 |+| RTX 4090 | 0.72 | 0.73 |diff --git a/docs/nerfstudio.md b/docs/nerfstudio.mdindex 6b52c35..14037c7 100644--- a/docs/nerfstudio.md+++ b/docs/nerfstudio.md@@ -73,7 +73,7 @@ Example 3DGS reconstruction in the Nerfstudio viewer:-> [!NOTE]+> [!note]> This guide reconstructs a static 3DGS from a single synchronized timestamp and cannot reproduce the 4DGS results shown in our work.>> The FreeTimeGS implementation used in the paper is not publicly available. We are evaluating open-source alternatives for a reproducible 4DGS pipeline.