[SIGGRAPH Asia 2026] 4DAnyone: Create Anyone in 4D from a Casual Monocular Video
Add camera layout examples
krahetsae9bdcc15563
diff --git a/README.md b/README.mdindex 3c4a3fb..4c1b36a 100644--- a/README.md+++ b/README.md@@ -32,39 +32,58 @@ python scripts/download_example.py## Inference-### Generate target views+### 6-view full orbit++A lightweight 360° layout for quick iteration.```bash-# Generate 6 evenly spaced views around the subjectpython inference.py \--video_path "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4" \--views_per_layer 6+```++<p align="left"><img src="docs/assets/inference-6-views.jpg" width="70%" alt="Six evenly spaced target cameras on one full orbit"></p>++### 24-view full orbit++A denser 360° layout for smoother angular coverage.-# Generate 24 evenly spaced views around the subject+```bashpython inference.py \--video_path "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4" \--views_per_layer 24+```++<p align="left"><img src="docs/assets/inference-24-views.jpg" width="70%" alt="Twenty-four evenly spaced target cameras on one full orbit"></p>++### 48-view, three pitch layers++Three pitch rings add vertical coverage around the subject.-# Generate 48 views with 3 pitch layers and 16 views per layer+```bashpython inference.py \--video_path "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4" \--views_per_layer 16 --layer_pitches '[-10,15,35]'+```-# Generate 8 views over the frontal 180-degree arc-python inference.py \- --video_path "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4" \- --views_per_layer 8 --start_yaw -90 --yaw_span 180+<p align="left"><img src="docs/assets/inference-48-views-3-layers.jpg" width="70%" alt="Forty-eight target cameras arranged over three pitch layers"></p>-# Generate one view at pitch 15° and yaw 60°+### 8-view frontal arc++A frontal-only layout that concentrates views within a 180° arc.++```bashpython inference.py \--video_path "data/source/pexels/2785536-uhd_2160_3840_25fps.mp4" \- --views_per_layer 1 --layer_pitches '[15]' --start_yaw 60+ --views_per_layer 8 --start_yaw -90 --yaw_span 180```-Camera layout options:+<p align="left"><img src="docs/assets/inference-8-views-front-180.jpg" width="70%" alt="Eight target cameras distributed over the frontal 180-degree arc"></p>++### Camera layout options- `layer_pitches`: pitch angle of each camera layer, in degrees; positive values place the cameras above the person.-- `views_per_layer`: number of evenly spaced views in each camera layer. Total views are `views_per_layer × len(layer_pitches)`.+- `views_per_layer`: number of evenly spaced views in each camera layer; it must be divisible by 4 or 6. Total views are `views_per_layer × len(layer_pitches)`.- `start_yaw`: horizontal angle of the first view, in degrees; yaw `0` is the front view.- `yaw_span`: horizontal range covered by each camera layer, in degrees.diff --git a/docs/assets/inference-24-views.jpg b/docs/assets/inference-24-views.jpgnew file mode 100644index 0000000..2c37efeBinary files /dev/null and b/docs/assets/inference-24-views.jpg differdiff --git a/docs/assets/inference-48-views-3-layers.jpg b/docs/assets/inference-48-views-3-layers.jpgnew file mode 100644index 0000000..a4e9735Binary files /dev/null and b/docs/assets/inference-48-views-3-layers.jpg differdiff --git a/docs/assets/inference-6-views.jpg b/docs/assets/inference-6-views.jpgnew file mode 100644index 0000000..e4cb651Binary files /dev/null and b/docs/assets/inference-6-views.jpg differdiff --git a/docs/assets/inference-8-views-front-180.jpg b/docs/assets/inference-8-views-front-180.jpgnew file mode 100644index 0000000..5dfd5baBinary files /dev/null and b/docs/assets/inference-8-views-front-180.jpg differdiff --git a/docs/output.md b/docs/output.mdindex 5e5043a..9a7ba20 100644--- a/docs/output.md+++ b/docs/output.md@@ -26,7 +26,7 @@ data/## 4DAnyone videos-`videos/dense/` contains `views_per_layer × len(layer_pitches)` final views. Camera IDs are layer-major: every pitch layer contains the requested evenly spaced yaw views before the next layer begins. Yaw `0` faces the person's front, positive pitch places the camera above the subject, and the default camera `00` is therefore frontal. Runs with at most six targets generate them directly; larger runs use RCP by default, with one, four, or six proposal videos following `views_per_group` and stored in `videos/sparse/`. At most the first four proposals are used as target-stage references. `skeletons/` contains one synchronized Goliath40 conditioning video per final camera, with IDs matching `videos/dense/`. All videos contain 121 frames at 704×1280 and use the selected input clip FPS. In `auto` mode, the source FPS is preserved unless it can be evenly reduced to 24, 25, or 30 FPS. Audio is not copied.+`videos/dense/` contains `views_per_layer × len(layer_pitches)` final views. Camera IDs are layer-major: every pitch layer contains the requested evenly spaced yaw views before the next layer begins. Yaw `0` faces the person's front, positive pitch places the camera above the subject, and the default camera `00` is therefore frontal. `views_per_layer` must be divisible by 4 or 6. Runs with at most six targets generate them directly; larger runs use RCP by default, with four or six proposal videos following `views_per_group` and stored in `videos/sparse/`. The first four proposals are used as target-stage references. `skeletons/` contains one synchronized Goliath40 conditioning video per final camera, with IDs matching `videos/dense/`. All videos contain 121 frames at 704×1280 and use the selected input clip FPS. In `auto` mode, the source FPS is preserved unless it can be evenly reduced to 24, 25, or 30 FPS. Audio is not copied.Before generation, the canonical source frames are segmented by the pinned BiRefNet revision at its standard 1024×1024 inference resolution, using FP16 batches of four. Their union mask is cropped with a 4% margin and a center-contain policy. Projected input-camera anatomy, foreground support, and ViTPose agreement classify the source as full-body, half-body, or close-up. The generated views then share one sequence-level camera solve:diff --git a/fdanyone/model/inference.py b/fdanyone/model/inference.pyindex 1a3e362..8861e2e 100644--- a/fdanyone/model/inference.py+++ b/fdanyone/model/inference.py@@ -250,17 +250,6 @@ def _load_rcp_reference_videos(frame_dirs: Iterable[Path], num_frames: int):return _rcp_reference_video_layout(frame_first_video)-def _fill_rcp_reference_slots(latents):- """Match the checkpoint's fixed four-reference ViewPack input."""-- count = latents.shape[0]- if count == 4:- return latents- if count == 1:- return latents.expand(4, *latents.shape[1:])- raise FourDAnyoneError(f"RCP must provide one or four reference latents, got {count}.")--def _decode_rcp(pipe,latents,@@ -470,10 +459,8 @@ def generate_views(# Match the reference JPEG-backed RCP boundary before the second VAE# encode; backing layout changes can otherwise select different BF16# CUDA kernels even when tensor values are identical.- reference_count = min(4, len(view_plan.rcp_camera_ids))- rcp_reference_videos = _load_rcp_reference_videos(frame_dirs[:reference_count], INFERENCE.num_frames)+ rcp_reference_videos = _load_rcp_reference_videos(frame_dirs[:4], INFERENCE.num_frames)rcp_reference_latents = _encode_videos(pipe, rcp_reference_videos, device)- rcp_reference_latents = _fill_rcp_reference_slots(rcp_reference_latents)# VAE38 encodes batch elements independently, so the existing source# encoding is identical to re-encoding source plus the RCP references.target_sources = torch.cat([source_latents, rcp_reference_latents], dim=0)diff --git a/fdanyone/views.py b/fdanyone/views.pyindex 7e9e2f0..bc260d4 100644--- a/fdanyone/views.py+++ b/fdanyone/views.py@@ -8,7 +8,7 @@ from dataclasses import dataclassfrom fdanyone.config import CAMERAfrom fdanyone.errors import ConfigurationError-VALID_VIEWS_PER_GROUP = (1, 4, 6)+VALID_VIEWS_PER_GROUP = (4, 6)MIN_PITCH = -15MAX_PITCH = 45@@ -61,9 +61,7 @@ class ViewPlan:@propertydef tcr_active(self) -> bool:- # A one-view group has no cross-view context to rearrange. All other- # layouts honor enable_tcr, including partial yaw spans and one group.- return self.enable_tcr and self.views_per_group > 1+ return self.enable_tcr@propertydef target_views(self) -> tuple[TargetView, ...]:@@ -148,7 +146,10 @@ def _layer_pitches(value: object) -> tuple[int, ...]:def _group_size(value: int | str, views_per_layer: int) -> int:if isinstance(value, str):if value.lower() == "auto":- return max(size for size in VALID_VIEWS_PER_GROUP if views_per_layer % size == 0)+ divisors = tuple(size for size in VALID_VIEWS_PER_GROUP if views_per_layer % size == 0)+ if not divisors:+ raise ConfigurationError(f"views_per_layer ({views_per_layer}) must be divisible by 4 or 6.")+ return max(divisors)try:value = int(value)except ValueError:diff --git a/inference.py b/inference.pyindex 1c4e702..c44bf21 100644--- a/inference.py+++ b/inference.py@@ -32,7 +32,8 @@ def inference(Args:video_path: Input video; it must contain at least 121 usable frames.- views_per_layer: Number of evenly spaced yaw views at each pitch.+ views_per_layer: Number of evenly spaced yaw views at each pitch. It+ must be divisible by 4 or 6.layer_pitches: Camera pitch for each layer in degrees, for example[-10,15,35]. Positive values place the camera above the subject;each value must be between -15 and 45.@@ -40,8 +41,8 @@ def inference(yaw_span: Angular range sampled by each layer, from 1 to 360 degrees.The end angle is excluded so a full ring never duplicates a view.views_per_group: Maximum target views generated together. auto chooses- the largest divisor among 6, 4, and 1; a manual value must be 1,- 4, or 6 and divide views_per_layer.+ 6 when possible and otherwise 4; a manual value must be 4 or 6 and+ divide views_per_layer.enable_rcp: Use proposal views before generating more than six targets.The proposal count follows views_per_group.enable_tcr: Shift view groups between denoising steps. Partial yaw