Skip to content

[SIGGRAPH Asia 2026] 4DAnyone: Create Anyone in 4D from a Casual Monocular Video

Simplify output documentation

krahetsf7da3ae3cdb7
diff --git a/docs/output.md b/docs/output.md
index 9a7ba20..553ebea 100644
--- a/docs/output.md
+++ b/docs/output.md
@@ -1,70 +1,51 @@
# Output
-With the default `--data_dir data`, one inference run writes:
+With the default `--data_dir data`, each inference run writes:
-```bash
+```text
data/
-├── gvhmr/
-│ └── results/<clip>/
-│ ├── motion.json
-│ └── motion.safetensors
-└── fdanyone/
- └── <clip>/
- ├── metadata.json
- ├── cameras.json
- ├── skeletons/00.mp4 ... <N-1>.mp4
- └── videos/
- ├── sparse/{00,04,09,12,14,19}.mp4 # default 24-view RCP proposals
- └── dense/00.mp4 ... <N-1>.mp4
+├── gvhmr/results/<clip>/
+│ ├── motion.json
+│ └── motion.safetensors
+└── fdanyone/<clip>/
+ ├── metadata.json
+ ├── cameras.json
+ ├── skeletons/00.mp4 ... <N-1>.mp4
+ └── videos/
+ ├── sparse/*.mp4 # RCP proposal views, when used
+ └── dense/00.mp4 ... <N-1>.mp4 # final target views
```
-`<clip>` is the input filename without its extension. Results are written atomically; a run never overwrites an existing result, and temporary intermediates are removed after a successful run.
-
-## GVHMR motion
-
-`motion.safetensors` contains the global and input-camera SMPL parameters, per-frame camera intrinsics, and observed ViTPose keypoints recovered from the input. `motion.json` records the GVHMR revision, timeline, frame identity, raster, and coordinate convention. When generation is retried with the same input clip, the validated motion result is reused instead of recomputed.
+`<clip>` is the input filename without its extension, and `N` is the number of target views.
-## 4DAnyone videos
+## Files
-`videos/dense/` contains `views_per_layer × len(layer_pitches)` final views. Camera IDs are layer-major: every pitch layer contains the requested evenly spaced yaw views before the next layer begins. Yaw `0` faces the person's front, positive pitch places the camera above the subject, and the default camera `00` is therefore frontal. `views_per_layer` must be divisible by 4 or 6. Runs with at most six targets generate them directly; larger runs use RCP by default, with four or six proposal videos following `views_per_group` and stored in `videos/sparse/`. The first four proposals are used as target-stage references. `skeletons/` contains one synchronized Goliath40 conditioning video per final camera, with IDs matching `videos/dense/`. All videos contain 121 frames at 704×1280 and use the selected input clip FPS. In `auto` mode, the source FPS is preserved unless it can be evenly reduced to 24, 25, or 30 FPS. Audio is not copied.
+- `videos/dense/` contains the generated target-view videos.
+- `videos/sparse/` contains intermediate proposal views for larger camera layouts.
+- `skeletons/` contains pose-conditioning videos aligned with the target views.
+- `cameras.json` contains camera intrinsics and poses for every target view.
+- `metadata.json` records the input, generation settings, model versions, and timings.
+- `gvhmr/results/` caches the recovered input motion for reuse.
-Before generation, the canonical source frames are segmented by the pinned BiRefNet revision at its standard 1024×1024 inference resolution, using FP16 batches of four. Their union mask is cropped with a 4% margin and a center-contain policy. Projected input-camera anatomy, foreground support, and ViTPose agreement classify the source as full-body, half-body, or close-up. The generated views then share one sequence-level camera solve:
-
-- the base radius protects the 95th-percentile body height and 80th-percentile body width around the full camera ring;
-- confident half-body and close-up inputs additionally adapt target height and focal length;
-- target and RCP conditioning use a center-aspect crop, while skeleton line width follows projected 3D body scale.
-
-`metadata.json` records this preprocessing solve alongside the resolved layer pitches, yaw range, grouping, input timeline, motion and checkpoint identity, timings, and resource measurements. `cameras.json` stores each camera's layer, pitch, yaw, intrinsics, and pose in OpenCV camera coordinates and the canonical 4DAnyone Y-up human world.
+All output videos contain 121 frames at 704×1280, use the selected output FPS, and contain no audio.
## Nerfstudio export
-Nerfstudio consumes images rather than MP4 files. Export one synchronized timestamp after inference:
+Export one synchronized frame after inference:
```bash
python scripts/export_nerfstudio.py \
- --result_dir data/fdanyone/<clip> \
- --frame_index 60
+ --result_dir data/fdanyone/<clip> \
+ --frame_index 60
```
-This creates:
+This creates a Nerfstudio-ready dataset:
-```bash
+```text
data/nerfstudio/<clip>/frame_060/
├── transforms.json
├── images/00.jpg ... <N-1>.jpg
└── masks/00.png ... <N-1>.png
```
-The exporter runs the same pinned standard BiRefNet model used by inference, at 1024×1024 in FP16 batches of four. Missing BiRefNet files are downloaded automatically into `--model_dir` on first use. Its soft predictions are thresholded at 0.5 and written as single-channel binary PNGs: white pixels are foreground and black pixels are excluded from training.
-
-`transforms.json` follows the [Nerfstudio `OPENCV` dataset schema](https://docs.nerf.studio/quickstart/data_conventions.html). Every frame contains both `file_path` and `mask_path`; Nerfstudio loads those masks automatically. Intrinsics are stored per image so each exported frame remains self-contained. Camera poses are converted from 4DAnyone's OpenCV/Y-up convention to Nerfstudio's OpenGL/Z-up camera-to-world convention. The directory can be passed directly to Splatfacto:
-
-```bash
-ns-train splatfacto \
- --data data/nerfstudio/<clip>/frame_060 \
- --pipeline.model.background-color random
-```
-
-The masks remove generated background pixels from the image loss. The explicit random rendering background also discourages transparent Gaussians from relying on one fixed composite color.
-
-Each export is one static multi-view timestamp. Exporting all 121 frames into one vanilla Nerfstudio dataset would mix a moving person across time; dynamic methods should instead use a method-specific temporal data parser.
+Each export represents one static timestamp and can be passed directly to Nerfstudio.