Skip to content

[SIGGRAPH Asia 2026] 4DAnyone: Create Anyone in 4D from a Casual Monocular Video

Release: refresh inference performance documentation (#29)

Yudong Jin28c5c3828951
diff --git a/README.md b/README.md
index 225333a..5d21b8e 100644
--- a/README.md
+++ b/README.md
@@ -8,11 +8,19 @@
<p align="center">4DAnyone turns a casual monocular video into multi-view videos, enabling downstream 4DGS reconstruction.</p>
+> [!note]
+> 4DAnyone is a multi-view video model that:
+>
+> - generates dozens of synchronized, view-consistent videos from a single monocular video.
+> - requires **22 GB** of peak CUDA memory, enabling inference on consumer GPUs.
+> - generates a 121-frame video in **27 seconds** on a single RTX 4090.
+
## News
-- **2026-09-02**: Released **4DAnyone-Turbo**, achieving a **5.58×** denoising speedup over the base 4DAnyone model. See [Inference performance](docs/inference_performance.md) for details.
+- **2026-09-05**: Reduced peak GPU memory below **24 GB**, enabling inference on consumer GPUs (RTX 4090).
+- **2026-09-02**: Released **4DAnyone-Turbo**, achieving a **5.58×** denoising speedup over 4DAnyone-Base.
- **2026-08-28**: Achieved a **1.42×** end-to-end speedup for the complete 24-view generation pipeline.
-- **2026-08-28**: Reduced peak GPU memory to **25.4 GiB** while slightly improving speed.
+- **2026-08-28**: Reduced peak GPU memory below **32 GB** while slightly improving speed.
## Installation
@@ -26,6 +34,8 @@ conda activate 4danyone
pip install -r requirements.txt
```
+For faster inference, optionally install [FlashAttention-3](https://github.com/Dao-AILab/flash-attention/tree/main/hopper) or [SageAttention](https://github.com/thu-ml/SageAttention). The installed backend is enabled automatically.
+
Missing models and examples are downloaded automatically on first use. You can also download them manually:
```bash
@@ -36,7 +46,7 @@ python scripts/download_example.py
## Inference
-We provide two models: **4DAnyone-Turbo** for faster four-step denoising and **4DAnyone-Base** with the standard denoising schedule. Both models achieve comparable generation quality. 4DAnyone-Turbo is enabled by default. Set `--enable_turbo=False` to use 4DAnyone-Base.
+This repository provides two models: **4DAnyone-Base** with the standard denoising schedule and the distilled **4DAnyone-Turbo** for faster four-step denoising. 4DAnyone-Turbo is enabled by default for faster inference while maintaining generation quality comparable to 4DAnyone-Base. See [Inference performance](docs/inference_performance.md) for GPU memory, inference speed, and generation quality benchmarks.
4DAnyone supports flexible target-view counts, pitch layers, and yaw coverage. Here are several common camera configurations:
@@ -115,21 +125,13 @@ data/
└── dense/00.mp4 ... <N-1>.mp4 # generated target views
```
-### Inference Efficiency
-
-For faster inference on supported GPUs, optionally install [FlashAttention-3](https://github.com/Dao-AILab/flash-attention/tree/main/hopper) or [SageAttention](https://github.com/thu-ml/SageAttention). Runtime selection follows one fixed order: FlashAttention-3, SageAttention, then PyTorch SDPA.
-
-See [Inference performance](docs/inference_performance.md) for measured 6-view runtimes and peak GPU memory usage on H20-3E, H200, RTX 5880 Ada, and RTX A6000 GPUs.
-
### Custom Data
-Use an input video that:
+Use an input video with:
-- is 720p or higher, with 1080p recommended.
-- uses a 9:16 portrait aspect ratio.
-- shows one person in a full-body or upper-body shot.
-- has at least 121 frames.
-- contains only mild camera motion.
+- a single person in a full-body or upper-body shot.
+- no large camera movements, clear footage.
+- 1080p or higher, 9:16 portrait aspect ratio, at least 121 frames.
## 3DGS Reconstruction
@@ -139,12 +141,13 @@ See the [nerfstudio guide](docs/nerfstudio.md) for details.
### Peak Memory Optimization
-- [x] Reduce peak GPU memory below 32 GB through pose precomputation and memory-efficient operators.
+- [x] Reduce peak GPU memory below 32 GB.
+- [x] Further reduce peak GPU memory below 24 GB.
### Inference Acceleration
-- [x] Achieve up to 1.42× end-to-end speedup by parallelizing pose encoding and VAE processing across GPUs.
-- [x] Achieve a **5.58×** denoising speedup with 4DAnyone-Turbo.
+- [x] Optimize inference speed through multi-GPU parallelism.
+- [x] Accelerate inference via few-step model distillation.
### Reconstruction
diff --git a/docs/inference_performance.md b/docs/inference_performance.md
index 9ad08ef..1a74eda 100644
--- a/docs/inference_performance.md
+++ b/docs/inference_performance.md
@@ -1,21 +1,45 @@
-# Inference performance
+# Inference Performance
-We compare **4DAnyone-Turbo**, the default accelerated model, with **4DAnyone-Base**, the original model described in the paper.
+This report provides practical runtime and GPU-memory references for running 4DAnyone on common GPUs.
-## Quality comparison
+## Turbo vs. Base
-We evaluate 32 validation scenes: 16 from DNA-Rendering, 8 from MVGameHuman, and 8 from SynCamVideo. Both models use identical target cameras, 25-frame clips, seed 42, and foreground-masked metrics; 4DAnyone-Turbo uses 4 denoising steps with scheduler shift 5, while 4DAnyone-Base uses 50 steps, serving as an upper-bound quality reference for our method.
+### Generation Quality
+
+The 4-step Turbo model achieves quality comparable to a 50-step Base reference:
| Model | Denoising steps | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
| --- | ---: | ---: | ---: | ---: |
-| 4DAnyone-Turbo | 4 | 25.48 | 0.895 | **0.072** |
-| 4DAnyone-Base | 50 | **25.69** | **0.899** | 0.073 |
+| 4DAnyone-Turbo | 4 | 25.48 | 0.895 | 0.072 |
+| 4DAnyone-Base | 50 | 25.69 | 0.899 | 0.073 |
+
+### Denoising Speedup
-The two models achieve comparable quality across PSNR, SSIM, and LPIPS.
+For the 6-view example, Turbo uses 4 denoising steps and Base uses 24. The values show Turbo's speedup over Base on the same GPU and attention backend.
-## Denoising performance
+| GPU | SDPA | FlashAttention-3 | SageAttention |
+| --- | ---: | ---: | ---: |
+| H200 | 5.49× | 5.35× | 5.12× |
+| H20-3E | 5.83× | 5.78× | 5.73× |
+| RTX 5880 Ada | 5.79× | — | 5.80× |
+| RTX A6000 | 5.77× | — | 5.75× |
+| RTX 4090 | 5.85× | — | 5.78× |
-We measure 4DAnyone-Turbo with 4 denoising steps and 4DAnyone-Base with 24 steps on the [6-view full-orbit](../README.md#6-view-full-orbit) example:
+## 4DAnyone-Turbo Performance
+
+The performance tables below use the following terms:
+
+- `SDPA`, `FlashAttention-3`, and `SageAttention`: denoising time for each attention backend.
+- `Peak CUDA allocated`: maximum live tensor memory during generation.
+- `GPU memory`: total memory capacity per GPU.
+- `—`: backend or workload unavailable on that GPU.
+
+> [!note]
+> SageAttention results use the official `sageattn` entry point, which automatically dispatches an architecture-specific kernel for each GPU.
+
+### 6-View Full Orbit
+
+The [6-view full-orbit](../README.md#6-view-full-orbit) benchmark uses the following command:
```bash
python inference.py \
@@ -23,57 +47,64 @@ python inference.py \
--views_per_layer 6
```
-Each reported denoising time is the median of three complete runs from the same source revision.
-
-These tables compare denoising time across attention backends for both models.
+| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GB) | GPU memory (GB) |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| H200 | 0.97 | 0.76 | 0.80 | 24.06 | 139.81 |
+| H20-3E | 3.21 | 2.54 | 2.09 | 24.06 | 139.81 |
+| RTX 5880 Ada | 3.47 | — | 2.99 | 24.06 | 47.37 |
+| RTX A6000 | 3.78 | — | 3.69 | 24.06 | 47.53 |
+| RTX 4090 | 2.71 | — | 2.13 | 22.18 | 23.52 |
-- `SDPA`, `FlashAttention-3`, and `SageAttention`: denoising time using each attention backend.
-- `Peak CUDA allocated`: maximum live tensor allocation across all runs and available attention backends.
-- `GPU memory`: total device memory capacity.
-- `—`: backend unavailable on that GPU.
+### 24-View Full Orbit
-### 4DAnyone-Turbo
+The [24-view full-orbit](../README.md#24-view-full-orbit) benchmark runs 4DAnyone-Turbo on a single GPU:
-| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GiB) | GPU memory (GiB) |
-| --- | ---: | ---: | ---: | ---: | ---: |
-| H200 | 0.98 | 0.77 | 0.78 | 26.63 | 139.81 |
-| H20-3E | 3.22 | 2.55 | 2.10 | 26.63 | 139.81 |
-| RTX 5880 Ada | 3.51 | — | 3.04 | 25.47 | 47.37 |
-| RTX A6000 | 3.80 | — | 3.71 | 24.91 | 47.53 |
+```bash
+python inference.py \
+ --video_path "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4" \
+ --views_per_layer 24
+```
-### 4DAnyone-Base
+For 24-view generation, the pipeline first denoises six RCP views to use as references and then denoises the 24 target views. The table reports the total time for both stages.
-| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GiB) | GPU memory (GiB) |
+| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GB) | GPU memory (GB) |
| --- | ---: | ---: | ---: | ---: | ---: |
-| H200 | 5.35 | 4.03 | 4.09 | 26.63 | 139.81 |
-| H20-3E | 18.71 | 14.73 | 11.98 | 26.63 | 139.81 |
-| RTX 5880 Ada | 20.40 | — | 17.58 | 25.47 | 47.37 |
-| RTX A6000 | 22.02 | — | 21.42 | 24.91 | 47.53 |
+| H200 | 5.11 | 3.90 | 3.90 | 24.06 | 139.81 |
+| H20-3E | 17.66 | 13.86 | 11.24 | 24.06 | 139.81 |
+| RTX 5880 Ada | 18.84 | — | 16.14 | 24.06 | 47.37 |
+| RTX A6000 | 20.55 | — | 19.98 | 24.06 | 47.53 |
+| RTX 4090 | 16.20 | — | — | 22.18 | 23.52 |
-### Speedup
+## 4DAnyone-Base Performance
-Speedup is calculated by dividing the 4DAnyone-Base median by the 4DAnyone-Turbo median for the same GPU and attention backend.
+The 6-view 4DAnyone-Base benchmark uses 24 denoising steps:
-| GPU | SDPA | FlashAttention-3 | SageAttention |
-| --- | ---: | ---: | ---: |
-| H200 | 5.45× | 5.26× | 5.23× |
-| H20-3E | 5.81× | 5.78× | 5.69× |
-| RTX 5880 Ada | 5.81× | — | 5.79× |
-| RTX A6000 | 5.79× | — | 5.77× |
+```bash
+python inference.py \
+ --video_path "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4" \
+ --views_per_layer 6 \
+ --enable_turbo=False
+```
-> [!NOTE]
-> SageAttention results use the official `sageattn` entry point, which automatically dispatches an architecture-specific kernel for each GPU.
+| GPU | SDPA (min) | FlashAttention-3 (min) | SageAttention (min) | Peak CUDA allocated (GB) | GPU memory (GB) |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| H200 | 5.33 | 4.05 | 4.09 | 24.06 | 139.81 |
+| H20-3E | 18.68 | 14.70 | 11.98 | 24.06 | 139.81 |
+| RTX 5880 Ada | 20.10 | — | 17.36 | 24.06 | 47.37 |
+| RTX A6000 | 21.82 | — | 21.20 | 24.06 | 47.53 |
+| RTX 4090 | 15.86 | — | 12.29 | 22.18 | 23.52 |
-## Preprocessing performance
+## Preprocessing Performance
-Preprocessing is identical for both models and therefore reported once.
+GVHMR and skeleton extraction run before denoising and are shared by both models. These 6-view timings can vary with CPU and storage performance.
- `GVHMR`: tracking, ViTPose, image-feature extraction, and motion prediction.
- `Skeleton extraction`: target-view skeleton-condition rendering.
| GPU | GVHMR (min) | Skeleton extraction (min) |
| --- | ---: | ---: |
-| H200 | 0.94 | 0.57 |
-| H20-3E | 0.97 | 0.55 |
-| RTX 5880 Ada | 0.94 | 1.00 |
-| RTX A6000 | 1.70 | 1.06 |
+| H200 | 1.20 | 1.25 |
+| H20-3E | 1.08 | 1.27 |
+| RTX 5880 Ada | 0.98 | 1.00 |
+| RTX A6000 | 1.57 | 2.12 |
+| RTX 4090 | 0.72 | 0.73 |
diff --git a/docs/nerfstudio.md b/docs/nerfstudio.md
index 6b52c35..14037c7 100644
--- a/docs/nerfstudio.md
+++ b/docs/nerfstudio.md
@@ -73,7 +73,7 @@ Example 3DGS reconstruction in the Nerfstudio viewer:
![Example 3DGS reconstruction in the Nerfstudio viewer](assets/nerfstudio-example.jpg)
-> [!NOTE]
+> [!note]
> This guide reconstructs a static 3DGS from a single synchronized timestamp and cannot reproduce the 4DGS results shown in our work.
>
> The FreeTimeGS implementation used in the paper is not publicly available. We are evaluating open-source alternatives for a reproducible 4DGS pipeline.