Skip to content

[SIGGRAPH Asia 2026] 4DAnyone: Create Anyone in 4D from a Casual Monocular Video

Add Nerfstudio 3DGS reconstruction

krahetsb1de25f97aa8
diff --git a/README.md b/README.md
index 3acd1ec..789106b 100644
--- a/README.md
+++ b/README.md
@@ -119,19 +119,7 @@ Use an input video that:
## 3DGS Reconstruction
-Install [Nerfstudio](https://docs.nerf.studio/quickstart/installation.html) before running `ns-train`.
-
-```bash
-# Export frame 0 across all generated views
-python scripts/export_nerfstudio.py \
- --result_dir data/fdanyone/<clip> \
- --frame_index 0
-
-# Train foreground-only 3DGS
-ns-train splatfacto \
- --data data/nerfstudio/<clip>/frame_000 \
- --pipeline.model.background-color random
-```
+See the [nerfstudio guide](docs/nerfstudio.md) for details.
## Todos
diff --git a/docs/THIRD_PARTY_NOTICES.md b/docs/THIRD_PARTY_NOTICES.md
index 2cb5751..1b8d0f2 100644
--- a/docs/THIRD_PARTY_NOTICES.md
+++ b/docs/THIRD_PARTY_NOTICES.md
@@ -78,3 +78,12 @@ The frozen base-model assets come from the official [Wan2.2-TI2V-5B](https://hug
The MHR70 asset is a sanitized inference artifact containing only the sparse support/weight tensors, keypoint names, and a path-free structural schema. The first-party regression tensors are redistributed under Apache-2.0; the embedded name/order metadata remains covered by the schema notice above.
That regressor was trained by the 4DAnyone project on DNA-Rendering frames, using outputs from [SAM 3D Body](https://github.com/facebookresearch/sam-3d-body) and the [Momentum Human Rig](https://github.com/facebookresearch/MHR) conversion tools. SAM 3D Body is governed by the SAM License, MHR source is Apache-2.0, and DNA-Rendering access requires a separate signed dataset agreement. The regressor is not bundled in the source repository; its Hugging Face model card must retain these source-model and dataset citations alongside the split tensor/schema license declaration in this notice.
+
+## VGG-19 perceptual reconstruction
+
+- Loss reference: [Crowdsampling the Plenoptic Function](https://github.com/zhengqili/Crowdsampling-the-Plenoptic-Function/tree/f5216f312cf82d77f8d20454b5eeb3930324630a), released under MIT in its upstream README
+- VGG-19 model: [Oxford Visual Geometry Group](https://www.robots.ox.ac.uk/~vgg/research/very_deep/)
+- MatConvNet distribution: [imagenet-vgg-verydeep-19](https://www.vlfeat.org/matconvnet/pretrained/), source MD5 `106118b7cf60435e6d8e04f6a6dc3657`
+- Model license: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/)
+
+The optional `splatfacto-perceptual` method uses a custom pixel-plus-feature objective rather than standard LPIPS. The companion Hugging Face repository contains a modified copy of the VGG-19 model: MatConvNet filters are transposed into PyTorch layout and serialized as safetensors, while the unused classifier and final two convolution layers are omitted. The retained tensor values are otherwise unchanged. Credit remains with Karen Simonyan and Andrew Zisserman; users must preserve that attribution and the CC BY 4.0 license when redistributing the converted weights.
diff --git a/docs/assets/nerfstudio-example.jpg b/docs/assets/nerfstudio-example.jpg
new file mode 100644
index 0000000..4baac0e
Binary files /dev/null and b/docs/assets/nerfstudio-example.jpg differ
diff --git a/docs/nerfstudio.md b/docs/nerfstudio.md
new file mode 100644
index 0000000..6b52c35
--- /dev/null
+++ b/docs/nerfstudio.md
@@ -0,0 +1,79 @@
+# 3DGS reconstruction with Nerfstudio
+
+Export one synchronized multi-view frame from 4DAnyone and reconstruct it as a 3D Gaussian Splatting (3DGS) scene with Nerfstudio Splatfacto.
+
+## Installation
+
+Install Nerfstudio in a new environment by following the [official installation guide](https://docs.nerf.studio/quickstart/installation.html), then install:
+
+```bash
+pip install huggingface-hub safetensors
+```
+
+## Export
+
+Run the exporter in the 4DAnyone inference environment:
+
+```bash
+conda activate 4danyone
+python scripts/export_nerfstudio.py \
+ --result_dir data/fdanyone/<clip> \
+ --frame_index 0
+```
+
+The exported dataset is written to:
+
+```bash
+data/nerfstudio/<clip>/frame_000/
+├── transforms.json
+├── sparse_pcd.ply # visual-hull initialization
+├── images/00.png ... <N-1>.png
+└── masks/00.png ... <N-1>.png
+```
+
+## Train
+
+Standard Splatfacto:
+
+```bash
+ns-train splatfacto \
+ --data data/nerfstudio/<clip>/frame_000 \
+ --pipeline.model.background-color random
+```
+
+Splatfacto with perceptual loss:
+
+```bash
+python scripts/train_nerfstudio.py splatfacto-perceptual \
+ --data data/nerfstudio/<clip>/frame_000 \
+ --pipeline.model.background-color random \
+ --pipeline.model.perceptual-loss-weight 0.4 \
+ --pipeline.model.perceptual-compute-dtype bfloat16
+```
+
+If the GPU supports `bfloat16`, we recommend enabling it to accelerate training.
+
+## View
+
+Use the wrapper so the custom method is registered before Nerfstudio loads its saved config:
+
+```bash
+python scripts/view_nerfstudio.py \
+ --load-config outputs/<clip>/splatfacto-perceptual/<timestamp>/config.yml \
+ --viewer.websocket-port 7007
+```
+
+For a remote viewer, forward the same port from the workstation:
+
+```bash
+ssh -N -L 7007:127.0.0.1:7007 <server>
+```
+
+Example 3DGS reconstruction in the Nerfstudio viewer:
+
+![Example 3DGS reconstruction in the Nerfstudio viewer](assets/nerfstudio-example.jpg)
+
+> [!NOTE]
+> This guide reconstructs a static 3DGS from a single synchronized timestamp and cannot reproduce the 4DGS results shown in our work.
+>
+> The FreeTimeGS implementation used in the paper is not publicly available. We are evaluating open-source alternatives for a reproducible 4DGS pipeline.
diff --git a/docs/output.md b/docs/output.md
index 553ebea..89ead1a 100644
--- a/docs/output.md
+++ b/docs/output.md
@@ -31,21 +31,4 @@ All output videos contain 121 frames at 704×1280, use the selected output FPS,
## Nerfstudio export
-Export one synchronized frame after inference:
-
-```bash
-python scripts/export_nerfstudio.py \
- --result_dir data/fdanyone/<clip> \
- --frame_index 60
-```
-
-This creates a Nerfstudio-ready dataset:
-
-```text
-data/nerfstudio/<clip>/frame_060/
-├── transforms.json
-├── images/00.jpg ... <N-1>.jpg
-└── masks/00.png ... <N-1>.png
-```
-
-Each export represents one static timestamp and can be passed directly to Nerfstudio.
+See the [3DGS reconstruction guide](nerfstudio.md) for details.
diff --git a/fdanyone/assets.py b/fdanyone/assets.py
index 4f63abd..ef8a097 100644
--- a/fdanyone/assets.py
+++ b/fdanyone/assets.py
@@ -13,7 +13,7 @@ from pathlib import Path
from fdanyone.errors import AssetError
HF_REPO_ID = "AntResearch/4DAnyone"
-HF_REVISION = "eab457d2761b15c7fe7b7b05484ae75a94119677"
+HF_REVISION = "7850985888b56aabf09e69480b73248f1a76bcbe"
BIREFNET_REPO_ID = "ZhengPeng7/BiRefNet"
BIREFNET_REVISION = "e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4"
@@ -39,6 +39,7 @@ GVHMR_CHECKPOINT = "gvhmr/gvhmr_siga24_release.ckpt"
HMR2_CHECKPOINT = "gvhmr/epoch=10-step=25000.ckpt"
VITPOSE_CHECKPOINT = "gvhmr/vitpose-h-multi-coco.pth"
YOLO_CHECKPOINT = "gvhmr/yolov8x.pt"
+PERCEPTUAL_VGG19 = "perceptual/imagenet-vgg-verydeep-19-conv.safetensors"
SMPLX_MODEL = "body_models/smplx/SMPLX_NEUTRAL.npz"
@@ -52,6 +53,7 @@ MODEL_FILES = (
HMR2_CHECKPOINT,
VITPOSE_CHECKPOINT,
YOLO_CHECKPOINT,
+ PERCEPTUAL_VGG19,
)
EXAMPLE_FILES = (
@@ -115,6 +117,16 @@ def resolve_foreground_model(model_dir: str | Path = "models") -> Path:
return root.resolve()
+def resolve_perceptual_vgg19(model_dir: str | Path = "models") -> Path:
+ """Resolve the converted VGG-19 weights used by perceptual reconstruction."""
+
+ return _require_file(
+ Path(model_dir) / PERCEPTUAL_VGG19,
+ "Perceptual VGG-19 weights",
+ "scripts/download_model.py",
+ )
+
+
def resolve_base_assets(model_dir: str | Path = "models") -> BaseAssets:
"""Resolve the local VAE, T5 encoder, and tokenizer."""
diff --git a/fdanyone/download.py b/fdanyone/download.py
index e51be2a..bcb790b 100644
--- a/fdanyone/download.py
+++ b/fdanyone/download.py
@@ -31,7 +31,9 @@ from fdanyone.assets import (
HF_REPO_ID,
HF_REVISION,
MODEL_FILES,
+ PERCEPTUAL_VGG19,
SMPLX_MODEL,
+ resolve_perceptual_vgg19,
)
from fdanyone.errors import AssetError
@@ -153,6 +155,17 @@ def ensure_models(
return models
+def ensure_perceptual_vgg19(model_dir: str | Path = "models") -> Path:
+ """Download only the optional VGG-19 reconstruction asset when missing."""
+
+ models = Path(model_dir).expanduser().resolve()
+ destination = models / PERCEPTUAL_VGG19
+ if not destination.is_file():
+ LOGGER.info("Downloading the perceptual VGG-19 model (first use only)")
+ _snapshot([PERCEPTUAL_VGG19], models)
+ return resolve_perceptual_vgg19(models)
+
+
def download_model(
model_dir: str = "models",
gvhmr_root: str = "third_party/GVHMR",
diff --git a/fdanyone/nerfstudio/__init__.py b/fdanyone/nerfstudio/__init__.py
new file mode 100644
index 0000000..d0b30cb
--- /dev/null
+++ b/fdanyone/nerfstudio/__init__.py
@@ -0,0 +1,38 @@
+"""Nerfstudio dataset export and reconstruction helpers.
+
+Exports stay lazy so the training plugin can run inside Nerfstudio's own
+environment without importing the separate 4DAnyone inference dependency
+closure.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from fdanyone.nerfstudio.cameras import camera_to_nerfstudio
+ from fdanyone.nerfstudio.exporter import NERFSTUDIO_MASK_THRESHOLD, export_nerfstudio
+ from fdanyone.nerfstudio.visual_hull import NERFSTUDIO_POINT_CLOUD
+
+__all__ = [
+ "NERFSTUDIO_MASK_THRESHOLD",
+ "NERFSTUDIO_POINT_CLOUD",
+ "camera_to_nerfstudio",
+ "export_nerfstudio",
+]
+
+
+def __getattr__(name: str):
+ if name == "camera_to_nerfstudio":
+ from fdanyone.nerfstudio.cameras import camera_to_nerfstudio
+
+ return camera_to_nerfstudio
+ if name in {"NERFSTUDIO_MASK_THRESHOLD", "export_nerfstudio"}:
+ from fdanyone.nerfstudio import exporter
+
+ return getattr(exporter, name)
+ if name == "NERFSTUDIO_POINT_CLOUD":
+ from fdanyone.nerfstudio.visual_hull import NERFSTUDIO_POINT_CLOUD
+
+ return NERFSTUDIO_POINT_CLOUD
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
diff --git a/fdanyone/nerfstudio/cameras.py b/fdanyone/nerfstudio/cameras.py
new file mode 100644
index 0000000..6c9f1b8
--- /dev/null
+++ b/fdanyone/nerfstudio/cameras.py
@@ -0,0 +1,86 @@
+"""Camera validation and coordinate conversion for Nerfstudio exports."""
+
+from __future__ import annotations
+
+import numpy as np
+
+from fdanyone.errors import FourDAnyoneError
+
+# 4DAnyone uses a right-handed Y-up world. Nerfstudio uses a right-handed
+# Z-up world, so rotate +Y onto +Z while keeping +X fixed.
+_Y_UP_TO_Z_UP = np.array(
+ [
+ [1.0, 0.0, 0.0, 0.0],
+ [0.0, 0.0, -1.0, 0.0],
+ [0.0, 1.0, 0.0, 0.0],
+ [0.0, 0.0, 0.0, 1.0],
+ ],
+ dtype=np.float64,
+)
+
+# OpenCV camera axes are +X right, +Y down, +Z forward. Nerfstudio follows
+# OpenGL/Blender: +X right, +Y up, +Z back.
+_OPENCV_TO_OPENGL = np.diag([1.0, -1.0, -1.0, 1.0])
+
+
+def camera_to_nerfstudio(camera_to_world: object) -> list[list[float]]:
+ """Convert an OpenCV/Y-up camera-to-world matrix to OpenGL/Z-up."""
+
+ matrix = np.asarray(camera_to_world, dtype=np.float64)
+ if matrix.shape != (4, 4) or not np.isfinite(matrix).all():
+ raise FourDAnyoneError("Camera-to-world must be a finite 4x4 matrix.")
+ if not np.allclose(matrix[3], [0.0, 0.0, 0.0, 1.0]):
+ raise FourDAnyoneError("Camera-to-world must be a homogeneous transform.")
+ converted = _Y_UP_TO_Z_UP @ matrix @ _OPENCV_TO_OPENGL
+ return converted.tolist()
+
+
+def camera_geometry(cameras: list[dict]) -> tuple[np.ndarray, np.ndarray]:
+ """Validate a rig and return camera-to-world and projection matrices."""
+
+ intrinsics = []
+ camera_to_worlds = []
+ for camera in cameras:
+ camera_id = int(camera["camera_id"])
+ intrinsic = np.asarray(camera.get("K"), dtype=np.float64)
+ camera_to_world = np.asarray(camera.get("camera_to_world"), dtype=np.float64)
+ if intrinsic.shape != (3, 3) or not np.isfinite(intrinsic).all():
+ raise FourDAnyoneError(f"Camera {camera_id:02d} has an invalid intrinsic matrix.")
+ if camera_to_world.shape != (4, 4) or not np.isfinite(camera_to_world).all():
+ raise FourDAnyoneError(f"Camera {camera_id:02d} has an invalid camera-to-world matrix.")
+ if not np.allclose(camera_to_world[3], [0.0, 0.0, 0.0, 1.0]):
+ raise FourDAnyoneError(f"Camera {camera_id:02d} has a non-homogeneous camera-to-world matrix.")
+ intrinsics.append(intrinsic)
+ camera_to_worlds.append(camera_to_world)
+
+ intrinsics_array = np.stack(intrinsics)
+ camera_to_worlds_array = np.stack(camera_to_worlds)
+ world_to_cameras = np.linalg.inv(camera_to_worlds_array)
+ projections = intrinsics_array @ world_to_cameras[:, :3]
+ return camera_to_worlds_array, projections
+
+
+def visual_hull_center(camera_to_worlds: np.ndarray) -> np.ndarray:
+ """Return the common look-at target of the generated camera rig."""
+
+ centers = camera_to_worlds[:, :3, 3]
+ directions = camera_to_worlds[:, :3, 2]
+ norms = np.linalg.norm(directions, axis=1, keepdims=True)
+ if np.any(norms <= 1e-12):
+ raise FourDAnyoneError("Camera rig contains a zero-length optical axis.")
+ directions = directions / norms
+ projectors = np.eye(3)[None] - directions[:, :, None] * directions[:, None, :]
+ system = projectors.sum(axis=0)
+ if np.linalg.matrix_rank(system) < 3:
+ raise FourDAnyoneError("Camera rig does not constrain a bounded visual-hull center.")
+ target = np.linalg.solve(system, np.einsum("bij,bj->i", projectors, centers))
+ if not np.isfinite(target).all():
+ raise FourDAnyoneError("Camera rig produced a non-finite visual-hull center.")
+ return target
+
+
+def points_to_nerfstudio(points: np.ndarray) -> np.ndarray:
+ """Convert 4DAnyone Y-up XYZ points into Nerfstudio Z-up coordinates."""
+
+ rotation = _Y_UP_TO_Z_UP[:3, :3].astype(np.float32)
+ return points @ rotation.T
diff --git a/fdanyone/nerfstudio.py b/fdanyone/nerfstudio/exporter.py
similarity index 76%
rename from fdanyone/nerfstudio.py
rename to fdanyone/nerfstudio/exporter.py
index 3970244..68ab974 100644
--- a/fdanyone/nerfstudio.py
+++ b/fdanyone/nerfstudio/exporter.py
@@ -1,4 +1,4 @@
-"""Export one synchronized generated frame and its foreground masks for Nerfstudio."""
+"""Export synchronized 4DAnyone frames as directly trainable Nerfstudio data."""
from __future__ import annotations
@@ -17,40 +17,17 @@ from fdanyone.download import ensure_foreground_model
from fdanyone.errors import FourDAnyoneError
from fdanyone.foreground import predict_foreground_masks
from fdanyone.io import AtomicResultDirectory, write_json
+from fdanyone.nerfstudio.cameras import camera_to_nerfstudio
+from fdanyone.nerfstudio.visual_hull import (
+ NERFSTUDIO_POINT_CLOUD,
+ build_sparse_point_cloud,
+ write_sparse_point_cloud,
+)
-NERFSTUDIO_JPEG_QUALITY = 85
# Nerfstudio casts mask pixels directly to bool, so soft BiRefNet predictions
# must cross a real decision boundary before serialization.
NERFSTUDIO_MASK_THRESHOLD = 128
-# 4DAnyone uses a right-handed Y-up world. Nerfstudio uses a right-handed
-# Z-up world, so rotate +Y onto +Z while keeping +X fixed.
-_Y_UP_TO_Z_UP = np.array(
- [
- [1.0, 0.0, 0.0, 0.0],
- [0.0, 0.0, -1.0, 0.0],
- [0.0, 1.0, 0.0, 0.0],
- [0.0, 0.0, 0.0, 1.0],
- ],
- dtype=np.float64,
-)
-
-# OpenCV camera axes are +X right, +Y down, +Z forward. Nerfstudio follows
-# OpenGL/Blender: +X right, +Y up, +Z back.
-_OPENCV_TO_OPENGL = np.diag([1.0, -1.0, -1.0, 1.0])
-
-
-def camera_to_nerfstudio(camera_to_world: object) -> list[list[float]]:
- """Convert an OpenCV/Y-up camera-to-world matrix to OpenGL/Z-up."""
-
- matrix = np.asarray(camera_to_world, dtype=np.float64)
- if matrix.shape != (4, 4) or not np.isfinite(matrix).all():
- raise FourDAnyoneError("Camera-to-world must be a finite 4x4 matrix.")
- if not np.allclose(matrix[3], [0.0, 0.0, 0.0, 1.0]):
- raise FourDAnyoneError("Camera-to-world must be a homogeneous transform.")
- converted = _Y_UP_TO_Z_UP @ matrix @ _OPENCV_TO_OPENGL
- return converted.tolist()
-
def _read_cameras(result: Path) -> dict:
path = result / "cameras.json"
@@ -116,13 +93,26 @@ def _validate_raster(image: np.ndarray, camera: dict) -> None:
)
-def _write_images(images: tuple[np.ndarray, ...], root: Path) -> None:
+def _write_images(images: tuple[np.ndarray, ...], binary_masks: np.ndarray, root: Path) -> None:
+ """Write RGBA inputs so Splatfacto supervises the background through alpha.
+
+ A separate Nerfstudio ``mask_path`` excludes masked pixels from the loss. It
+ therefore does not constrain Gaussians outside the silhouette. Keeping the
+ foreground mask as alpha instead lets Splatfacto composite a fresh training
+ background and penalize stray geometry everywhere in the image.
+ """
+
+ expected_shape = (len(images), *(images[0].shape[:2]))
+ if binary_masks.dtype != np.bool_ or binary_masks.shape != expected_shape:
+ raise FourDAnyoneError(f"RGBA alpha masks must be boolean with shape {expected_shape}.")
root.mkdir()
for camera_id, image in enumerate(images):
- Image.fromarray(image).save(root / f"{camera_id:02d}.jpg", format="JPEG", quality=NERFSTUDIO_JPEG_QUALITY)
+ alpha = (binary_masks[camera_id] * 255).astype(np.uint8)[..., None]
+ rgba = np.concatenate([image, alpha], axis=2)
+ Image.fromarray(rgba, mode="RGBA").save(root / f"{camera_id:02d}.png", format="PNG")
-def _write_masks(masks: np.ndarray, images: tuple[np.ndarray, ...], root: Path) -> None:
+def _write_masks(masks: np.ndarray, images: tuple[np.ndarray, ...], root: Path) -> np.ndarray:
expected_shape = (len(images), *(images[0].shape[:2]))
if masks.dtype != np.uint8 or masks.shape != expected_shape:
raise FourDAnyoneError(
@@ -131,11 +121,13 @@ def _write_masks(masks: np.ndarray, images: tuple[np.ndarray, ...], root: Path)
)
root.mkdir()
- for camera_id, mask in enumerate(masks):
- binary = np.where(mask >= NERFSTUDIO_MASK_THRESHOLD, 255, 0).astype(np.uint8)
+ binary_masks = masks >= NERFSTUDIO_MASK_THRESHOLD
+ for camera_id in range(len(masks)):
+ binary = binary_masks[camera_id]
if not np.any(binary):
raise FourDAnyoneError(f"BiRefNet found no foreground in camera {camera_id:02d}.")
- Image.fromarray(binary).save(root / f"{camera_id:02d}.png", format="PNG")
+ Image.fromarray((binary * 255).astype(np.uint8)).save(root / f"{camera_id:02d}.png", format="PNG")
+ return binary_masks
def _transforms(cameras: list[dict]) -> dict:
@@ -152,8 +144,7 @@ def _transforms(cameras: list[dict]) -> dict:
raise FourDAnyoneError(f"Camera {camera_id:02d} has invalid image dimensions.") from exc
frames.append(
{
- "file_path": f"images/{camera_id:02d}.jpg",
- "mask_path": f"masks/{camera_id:02d}.png",
+ "file_path": f"images/{camera_id:02d}.png",
"fl_x": float(intrinsic[0, 0]),
"fl_y": float(intrinsic[1, 1]),
"cx": float(intrinsic[0, 2]),
@@ -163,7 +154,11 @@ def _transforms(cameras: list[dict]) -> dict:
"transform_matrix": camera_to_nerfstudio(camera.get("camera_to_world")),
}
)
- return {"camera_model": "OPENCV", "frames": frames}
+ return {
+ "camera_model": "OPENCV",
+ "ply_file_path": NERFSTUDIO_POINT_CLOUD,
+ "frames": frames,
+ }
def export_nerfstudio(
@@ -173,7 +168,7 @@ def export_nerfstudio(
model_dir: str = "models",
device: str = "cuda:0",
) -> dict:
- """Export one masked multi-view timestamp for direct Nerfstudio training.
+ """Export one masked multi-view timestamp and visual hull for Nerfstudio.
Args:
result_dir: A completed data/fdanyone/<clip> result.
@@ -205,13 +200,16 @@ def export_nerfstudio(
images = tuple(_extract_frame(video, frame_index) for video in videos)
for image, camera in zip(images, cameras, strict=True):
_validate_raster(image, camera)
- _write_images(images, work / "images")
device, _ = select_cuda_device(device)
ensure_foreground_model(model_dir)
foreground_model = resolve_foreground_model(model_dir)
masks = predict_foreground_masks(images, foreground_model, device)
- _write_masks(masks, images, work / "masks")
+ binary_masks = _write_masks(masks, images, work / "masks")
+ _write_images(images, binary_masks, work / "images")
+ del foreground_model
+ points, colors = build_sparse_point_cloud(images, binary_masks, cameras, device)
+ write_sparse_point_cloud(work / NERFSTUDIO_POINT_CLOUD, points, colors)
write_json(work / "transforms.json", transforms, sort_keys=False)
return {
@@ -219,4 +217,5 @@ def export_nerfstudio(
"frame_index": frame_index,
"num_images": len(cameras),
"num_masks": len(cameras),
+ "num_points": len(points),
}
diff --git a/fdanyone/nerfstudio/perceptual.py b/fdanyone/nerfstudio/perceptual.py
new file mode 100644
index 0000000..91fc601
--- /dev/null
+++ b/fdanyone/nerfstudio/perceptual.py
@@ -0,0 +1,173 @@
+"""VGG-19 perceptual loss for Nerfstudio reconstruction."""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from contextlib import AbstractContextManager, nullcontext
+from pathlib import Path
+
+import torch
+import torch.nn.functional as F
+from safetensors.torch import load_file
+from torch import nn
+
+from fdanyone.errors import AssetError
+
+# The reference objective reads these layers from the MatConvNet VGG-19
+# checkpoint and returns features after conv{1,2,3,4,5}_2. The final two VGG
+# convolutions and classifier are not evaluated by its forward pass.
+VGG19_CONV_CHANNELS = (
+ (3, 64),
+ (64, 64),
+ (64, 128),
+ (128, 128),
+ (128, 256),
+ (256, 256),
+ (256, 256),
+ (256, 256),
+ (256, 512),
+ (512, 512),
+ (512, 512),
+ (512, 512),
+ (512, 512),
+ (512, 512),
+)
+VGG19_FEATURE_LAYERS = (2, 4, 6, 10, 14)
+VGG19_POOL_AFTER = frozenset((2, 4, 8, 12))
+PERCEPTUAL_FEATURE_DIVISORS = (2.6, 4.8, 3.7, 5.6, 0.15)
+PERCEPTUAL_COMPUTE_DTYPES = ("float32", "bfloat16")
+
+
+def _autocast_context(compute_dtype: str, device: torch.device) -> AbstractContextManager:
+ if compute_dtype not in PERCEPTUAL_COMPUTE_DTYPES:
+ choices = ", ".join(PERCEPTUAL_COMPUTE_DTYPES)
+ raise ValueError(f"perceptual compute dtype must be one of: {choices}.")
+ if compute_dtype == "float32":
+ return nullcontext()
+ if device.type != "cuda":
+ raise RuntimeError("bfloat16 perceptual computation currently requires CUDA.")
+ if not torch.cuda.is_bf16_supported():
+ raise RuntimeError("This CUDA device does not support bfloat16 convolution.")
+ return torch.autocast(device_type="cuda", dtype=torch.bfloat16)
+
+
+def _validated_weights(path: Path) -> Mapping[str, torch.Tensor]:
+ try:
+ state = load_file(str(path), device="cpu")
+ except (OSError, ValueError) as exc:
+ raise AssetError(f"Cannot load perceptual VGG-19 weights at {path}: {exc}") from exc
+
+ expected = set()
+ for index, (in_channels, out_channels) in enumerate(VGG19_CONV_CHANNELS, start=1):
+ weight_name = f"conv{index}.weight"
+ bias_name = f"conv{index}.bias"
+ expected.update((weight_name, bias_name))
+ weight = state.get(weight_name)
+ bias = state.get(bias_name)
+ expected_weight_shape = (out_channels, in_channels, 3, 3)
+ if weight is None or tuple(weight.shape) != expected_weight_shape:
+ shape = None if weight is None else tuple(weight.shape)
+ raise AssetError(
+ f"Perceptual VGG-19 tensor {weight_name} has shape {shape}; expected {expected_weight_shape}."
+ )
+ if bias is None or tuple(bias.shape) != (out_channels,):
+ shape = None if bias is None else tuple(bias.shape)
+ raise AssetError(f"Perceptual VGG-19 tensor {bias_name} has shape {shape}; expected {(out_channels,)}.")
+ unexpected = set(state) - expected
+ if unexpected:
+ raise AssetError(f"Perceptual VGG-19 weights contain unexpected tensors: {sorted(unexpected)}")
+ return state
+
+
+class MatConvNetVGG19Features(nn.Module):
+ """Frozen MatConvNet VGG-19 features used by the perceptual objective.
+
+ The weights are non-persistent buffers: they move with the Nerfstudio model
+ but remain in the separately downloaded asset instead of being duplicated
+ into every Splatfacto checkpoint.
+ """
+
+ def __init__(self, weights_path: str | Path) -> None:
+ super().__init__()
+ path = Path(weights_path).expanduser().resolve()
+ if not path.is_file():
+ raise AssetError(
+ f"Perceptual VGG-19 weights do not exist: {path}. "
+ "Run `python scripts/download_model.py` to install them."
+ )
+ state = _validated_weights(path)
+ for index in range(1, len(VGG19_CONV_CHANNELS) + 1):
+ self.register_buffer(
+ f"conv{index}_weight",
+ state[f"conv{index}.weight"].float().contiguous(),
+ persistent=False,
+ )
+ self.register_buffer(
+ f"conv{index}_bias",
+ state[f"conv{index}.bias"].float().contiguous(),
+ persistent=False,
+ )
+
+ def forward(self, image: torch.Tensor) -> tuple[torch.Tensor, ...]:
+ features = []
+ value = image.float().contiguous()
+ for index in range(1, len(VGG19_CONV_CHANNELS) + 1):
+ weight = getattr(self, f"conv{index}_weight")
+ bias = getattr(self, f"conv{index}_bias")
+ value = F.relu(F.conv2d(value, weight, bias, padding=1), inplace=False)
+ if index in VGG19_FEATURE_LAYERS:
+ features.append(value)
+ if index in VGG19_POOL_AFTER:
+ value = F.avg_pool2d(value, kernel_size=2, stride=2)
+ return tuple(features)
+
+
+def perceptual_distance(
+ prediction: torch.Tensor,
+ target: torch.Tensor,
+ prediction_features: tuple[torch.Tensor, ...],
+ target_features: tuple[torch.Tensor, ...],
+) -> torch.Tensor:
+ """Apply the reference pixel/feature weighting to scaled images."""
+
+ if len(prediction_features) != 5 or len(target_features) != 5:
+ raise ValueError("The perceptual loss requires exactly five VGG feature tensors.")
+ terms = [torch.mean(torch.abs(target - prediction))]
+ terms.extend(
+ torch.mean(torch.abs(target_features[index] - prediction_features[index])) / divisor
+ for index, divisor in enumerate(PERCEPTUAL_FEATURE_DIVISORS)
+ )
+ return torch.stack(terms).sum() / 255.0
+
+
+class VGG19PerceptualLoss(nn.Module):
+ """The reconstruction objective's VGG-19 loss, not standard LPIPS."""
+
+ def __init__(self, weights_path: str | Path, compute_dtype: str = "float32") -> None:
+ super().__init__()
+ if compute_dtype not in PERCEPTUAL_COMPUTE_DTYPES:
+ choices = ", ".join(PERCEPTUAL_COMPUTE_DTYPES)
+ raise ValueError(f"perceptual compute dtype must be one of: {choices}.")
+ self.compute_dtype = compute_dtype
+ self.features = MatConvNetVGG19Features(weights_path)
+ self.register_buffer(
+ "imagenet_mean",
+ torch.tensor((123.6800, 116.7790, 103.9390), dtype=torch.float32).reshape(1, 3, 1, 1),
+ persistent=False,
+ )
+
+ def forward(self, prediction: torch.Tensor, target: torch.Tensor) -> torch.Tensor:
+ if prediction.shape != target.shape or prediction.ndim != 4 or prediction.shape[1] != 3:
+ raise ValueError("VGG-19 perceptual loss expects prediction and target with matching [B, 3, H, W] shapes.")
+ prediction_scaled = prediction.float() * 255.0 - self.imagenet_mean
+ target_scaled = target.float() * 255.0 - self.imagenet_mean
+ with _autocast_context(self.compute_dtype, prediction.device):
+ with torch.no_grad():
+ target_features = self.features(target_scaled)
+ prediction_features = self.features(prediction_scaled)
+ return perceptual_distance(
+ prediction_scaled,
+ target_scaled,
+ prediction_features,
+ target_features,
+ )
diff --git a/fdanyone/nerfstudio/plugin.py b/fdanyone/nerfstudio/plugin.py
new file mode 100644
index 0000000..f93c429
--- /dev/null
+++ b/fdanyone/nerfstudio/plugin.py
@@ -0,0 +1,19 @@
+"""Registration helpers for the repository's optional Nerfstudio method."""
+
+from __future__ import annotations
+
+import os
+
+PERCEPTUAL_METHOD_NAME = "splatfacto-perceptual"
+PERCEPTUAL_METHOD_TARGET = "fdanyone.nerfstudio.splatfacto:splatfacto_perceptual_method"
+
+
+def register_perceptual_method() -> None:
+ """Expose the method through Nerfstudio's environment plugin hook."""
+
+ definition = f"{PERCEPTUAL_METHOD_NAME}={PERCEPTUAL_METHOD_TARGET}"
+ configured = [item for item in os.environ.get("NERFSTUDIO_METHOD_CONFIGS", "").split(",") if item]
+ names = {item.partition("=")[0] for item in configured}
+ if PERCEPTUAL_METHOD_NAME not in names:
+ configured.append(definition)
+ os.environ["NERFSTUDIO_METHOD_CONFIGS"] = ",".join(configured)
diff --git a/fdanyone/nerfstudio/splatfacto.py b/fdanyone/nerfstudio/splatfacto.py
new file mode 100644
index 0000000..01f7dc5
--- /dev/null
+++ b/fdanyone/nerfstudio/splatfacto.py
@@ -0,0 +1,84 @@
+"""A Splatfacto method using the reconstruction perceptual objective."""
+
+from __future__ import annotations
+
+from copy import deepcopy
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from nerfstudio.models.splatfacto import SplatfactoModel, SplatfactoModelConfig
+from nerfstudio.plugins.types import MethodSpecification
+
+from fdanyone.assets import PERCEPTUAL_VGG19
+from fdanyone.download import ensure_perceptual_vgg19
+from fdanyone.nerfstudio.perceptual import VGG19PerceptualLoss
+
+
+@dataclass
+class PerceptualSplatfactoModelConfig(SplatfactoModelConfig):
+ """Splatfacto configuration with the custom VGG-19 perceptual loss."""
+
+ _target: type = field(default_factory=lambda: PerceptualSplatfactoModel)
+ perceptual_loss_weight: float = 0.5
+ """Perceptual-loss multiplier used by the reconstruction recipe."""
+ perceptual_weights_path: Path = Path("models") / PERCEPTUAL_VGG19
+ """Safetensors converted from the official MatConvNet VGG-19 model."""
+ perceptual_compute_dtype: str = "float32"
+ """VGG convolution dtype: float32 for parity or bfloat16 for speed."""
+
+
+class PerceptualSplatfactoModel(SplatfactoModel):
+ """Nerfstudio Splatfacto with the custom pixel-plus-feature loss."""
+
+ config: PerceptualSplatfactoModelConfig
+
+ def populate_modules(self) -> None:
+ super().populate_modules()
+ if self.config.perceptual_loss_weight < 0:
+ raise ValueError("perceptual_loss_weight must be non-negative.")
+ weights_path = Path(self.config.perceptual_weights_path).expanduser()
+ if self.config.perceptual_loss_weight > 0 and weights_path == Path("models") / PERCEPTUAL_VGG19:
+ weights_path = ensure_perceptual_vgg19()
+ self.perceptual_loss = (
+ VGG19PerceptualLoss(
+ weights_path,
+ compute_dtype=self.config.perceptual_compute_dtype,
+ )
+ if self.config.perceptual_loss_weight > 0
+ else None
+ )
+
+ def get_loss_dict(self, outputs, batch, metrics_dict=None):
+ loss_dict = super().get_loss_dict(outputs, batch, metrics_dict)
+ if not self.training or self.perceptual_loss is None:
+ return loss_dict
+
+ target = self.composite_with_background(self.get_gt_img(batch["image"]), outputs["background"])
+ prediction = outputs["rgb"]
+ if "mask" in batch:
+ mask = self._downscale_if_required(batch["mask"]).to(self.device)
+ target = target * mask
+ prediction = prediction * mask
+ target = target.permute(2, 0, 1).unsqueeze(0)
+ prediction = prediction.permute(2, 0, 1).unsqueeze(0)
+ loss_dict["perceptual_loss"] = self.config.perceptual_loss_weight * self.perceptual_loss(
+ prediction,
+ target,
+ )
+ return loss_dict
+
+
+def splatfacto_perceptual_method() -> MethodSpecification:
+ """Register the custom method without patching the Nerfstudio checkout."""
+
+ # Start from the installed Nerfstudio version's own Splatfacto recipe so
+ # optimizer defaults remain aligned with that installation.
+ from nerfstudio.configs.method_configs import method_configs
+
+ config = deepcopy(method_configs["splatfacto"])
+ config.method_name = "splatfacto-perceptual"
+ config.pipeline.model = PerceptualSplatfactoModelConfig()
+ return MethodSpecification(
+ config=config,
+ description="Splatfacto with the custom VGG-19 perceptual reconstruction loss.",
+ )
diff --git a/fdanyone/nerfstudio/visual_hull.py b/fdanyone/nerfstudio/visual_hull.py
new file mode 100644
index 0000000..062c37f
--- /dev/null
+++ b/fdanyone/nerfstudio/visual_hull.py
@@ -0,0 +1,181 @@
+"""Visual-hull carving and sparse PLY serialization."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+
+from fdanyone.errors import FourDAnyoneError
+from fdanyone.nerfstudio.cameras import camera_geometry, points_to_nerfstudio, visual_hull_center
+
+NERFSTUDIO_POINT_CLOUD = "sparse_pcd.ply"
+
+# The canonical world is metric and centered on the initial human root. A
+# 2.5-metre cube encloses the generated subject while remaining small enough
+# for 2-cm visual-hull carving to run in bounded batches.
+VISUAL_HULL_HALF_EXTENT_METERS = 1.25
+VISUAL_HULL_VOXEL_SIZE_METERS = 0.02
+VISUAL_HULL_BATCH_SIZE = 250_000
+VISUAL_HULL_MIN_POINTS = 100
+
+
+def _point_colors(
+ points: object,
+ images: tuple[np.ndarray, ...],
+ masks: object,
+ projections: object,
+ torch: object,
+) -> object:
+ """Average foreground observations of each visual-hull point."""
+
+ images_tensor = torch.from_numpy(np.stack(images)).to(device=points.device)
+ homogeneous = torch.cat([points, torch.ones_like(points[:, :1])], dim=1)
+ projected = torch.matmul(projections, homogeneous.T).transpose(1, 2)
+ depth = projected[..., 2]
+ pixels = projected[..., :2] / depth.unsqueeze(-1).clamp_min(1e-8)
+ u = torch.round(pixels[..., 0]).to(torch.long)
+ v = torch.round(pixels[..., 1]).to(torch.long)
+ height, width = masks.shape[1:]
+
+ color_sum = torch.zeros((points.shape[0], 3), device=points.device, dtype=torch.float32)
+ color_count = torch.zeros((points.shape[0], 1), device=points.device, dtype=torch.float32)
+ for camera_id in range(len(images)):
+ valid = (depth[camera_id] > 0) & (u[camera_id] >= 0) & (u[camera_id] < width)
+ valid &= (v[camera_id] >= 0) & (v[camera_id] < height)
+ point_ids = torch.nonzero(valid, as_tuple=False).squeeze(1)
+ if point_ids.numel() == 0:
+ continue
+ sample_u = u[camera_id, point_ids]
+ sample_v = v[camera_id, point_ids]
+ foreground = masks[camera_id, sample_v, sample_u]
+ point_ids = point_ids[foreground]
+ if point_ids.numel() == 0:
+ continue
+ sample_u = u[camera_id, point_ids]
+ sample_v = v[camera_id, point_ids]
+ color_sum[point_ids] += images_tensor[camera_id, sample_v, sample_u].to(torch.float32)
+ color_count[point_ids] += 1.0
+
+ if torch.any(color_count == 0):
+ raise FourDAnyoneError("Visual-hull points have no foreground color observations.")
+ return torch.round(color_sum / color_count).clamp(0, 255).to(torch.uint8)
+
+
+def build_sparse_point_cloud(
+ images: tuple[np.ndarray, ...],
+ binary_masks: np.ndarray,
+ cameras: list[dict],
+ device: str,
+) -> tuple[np.ndarray, np.ndarray]:
+ """Carve a colored visual hull in the canonical 4DAnyone world."""
+
+ try:
+ import torch
+ except ImportError as exc: # pragma: no cover - the inference environment always has torch.
+ raise FourDAnyoneError("PyTorch is required to carve the Nerfstudio visual hull.") from exc
+
+ if binary_masks.dtype != np.bool_:
+ raise FourDAnyoneError("Visual-hull masks must be boolean.")
+ if len(images) != len(cameras) or binary_masks.shape[0] != len(cameras):
+ raise FourDAnyoneError("Visual-hull images, masks, and cameras must have matching counts.")
+ if len({image.shape for image in images}) != 1:
+ raise FourDAnyoneError("Visual-hull images must share one raster size.")
+
+ camera_to_worlds, projection_matrices = camera_geometry(cameras)
+ center = visual_hull_center(camera_to_worlds)
+ lower = center - VISUAL_HULL_HALF_EXTENT_METERS
+ upper = center + VISUAL_HULL_HALF_EXTENT_METERS
+
+ torch_device = torch.device(device)
+ masks = torch.from_numpy(binary_masks).to(device=torch_device)
+ projections = torch.from_numpy(projection_matrices.astype(np.float32)).to(device=torch_device)
+ axes = [
+ torch.arange(float(lower[axis]), float(upper[axis]), VISUAL_HULL_VOXEL_SIZE_METERS, device=torch_device)
+ for axis in range(3)
+ ]
+ nx, ny, nz = (len(axis) for axis in axes)
+ total_points = nx * ny * nz
+ height, width = binary_masks.shape[1:]
+ kept = []
+
+ with torch.no_grad():
+ for start in range(0, total_points, VISUAL_HULL_BATCH_SIZE):
+ indices = torch.arange(
+ start,
+ min(start + VISUAL_HULL_BATCH_SIZE, total_points),
+ device=torch_device,
+ dtype=torch.long,
+ )
+ iz = indices % nz
+ iy = (indices // nz) % ny
+ ix = indices // (ny * nz)
+ points = torch.stack([axes[0][ix], axes[1][iy], axes[2][iz]], dim=1).to(projections.dtype)
+ homogeneous = torch.cat([points, torch.ones_like(points[:, :1])], dim=1)
+ projected = torch.matmul(projections, homogeneous.T).transpose(1, 2)
+ depth = projected[..., 2]
+ pixels = projected[..., :2] / depth.unsqueeze(-1).clamp_min(1e-8)
+ u = torch.round(pixels[..., 0]).to(torch.long)
+ v = torch.round(pixels[..., 1]).to(torch.long)
+ valid = (depth > 0) & (u >= 0) & (u < width) & (v >= 0) & (v < height)
+ inside = torch.zeros_like(valid)
+ if torch.any(valid):
+ camera_ids = torch.arange(len(cameras), device=torch_device).view(-1, 1).expand_as(u)
+ inside[valid] = masks[camera_ids[valid], v[valid], u[valid]]
+ keep = inside.all(dim=0)
+ if torch.any(keep):
+ kept.append(points[keep])
+
+ if not kept:
+ raise FourDAnyoneError("Foreground masks and cameras produced an empty visual hull.")
+ points_world = torch.cat(kept)
+ if points_world.shape[0] < VISUAL_HULL_MIN_POINTS:
+ raise FourDAnyoneError(
+ f"Foreground masks and cameras produced only {points_world.shape[0]} visual-hull points; "
+ f"expected at least {VISUAL_HULL_MIN_POINTS}."
+ )
+ colors = _point_colors(points_world, images, masks, projections, torch)
+
+ points_world = points_world.cpu().numpy().astype(np.float32, copy=False)
+ colors = colors.cpu().numpy()
+ return points_to_nerfstudio(points_world), colors
+
+
+def write_sparse_point_cloud(path: Path, points: np.ndarray, colors: np.ndarray) -> None:
+ """Write colored XYZ vertices as a binary little-endian PLY."""
+
+ if points.dtype != np.float32 or points.ndim != 2 or points.shape[1] != 3 or not np.isfinite(points).all():
+ raise FourDAnyoneError("Sparse point-cloud positions must be finite float32 XYZ values.")
+ if colors.dtype != np.uint8 or colors.shape != points.shape:
+ raise FourDAnyoneError("Sparse point-cloud colors must be uint8 RGB values matching the positions.")
+ vertices = np.empty(
+ len(points),
+ dtype=[
+ ("x", "<f4"),
+ ("y", "<f4"),
+ ("z", "<f4"),
+ ("red", "u1"),
+ ("green", "u1"),
+ ("blue", "u1"),
+ ],
+ )
+ for axis, name in enumerate(("x", "y", "z")):
+ vertices[name] = points[:, axis]
+ for channel, name in enumerate(("red", "green", "blue")):
+ vertices[name] = colors[:, channel]
+ header = (
+ "ply\n"
+ "format binary_little_endian 1.0\n"
+ "comment generated from 4DAnyone foreground masks\n"
+ f"element vertex {len(points)}\n"
+ "property float x\n"
+ "property float y\n"
+ "property float z\n"
+ "property uchar red\n"
+ "property uchar green\n"
+ "property uchar blue\n"
+ "end_header\n"
+ ).encode("ascii")
+ with path.open("wb") as output:
+ output.write(header)
+ output.write(vertices.tobytes())
diff --git a/scripts/train_nerfstudio.py b/scripts/train_nerfstudio.py
new file mode 100644
index 0000000..bef4cfd
--- /dev/null
+++ b/scripts/train_nerfstudio.py
@@ -0,0 +1,26 @@
+"""Train the repository's custom Nerfstudio reconstruction methods."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+
+def main() -> None:
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+ from fdanyone.nerfstudio.plugin import register_perceptual_method
+
+ register_perceptual_method()
+ try:
+ from nerfstudio.scripts.train import entrypoint
+ except ImportError as exc:
+ print(
+ "error: install Nerfstudio before running this training script.",
+ file=sys.stderr,
+ )
+ raise SystemExit(1) from exc
+ entrypoint()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/view_nerfstudio.py b/scripts/view_nerfstudio.py
new file mode 100644
index 0000000..d90b187
--- /dev/null
+++ b/scripts/view_nerfstudio.py
@@ -0,0 +1,26 @@
+"""Open checkpoints produced by the custom method in Nerfstudio Viewer."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+
+def main() -> None:
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+ from fdanyone.nerfstudio.plugin import register_perceptual_method
+
+ register_perceptual_method()
+ try:
+ from nerfstudio.scripts.viewer.run_viewer import entrypoint
+ except ImportError as exc:
+ print(
+ "error: install Nerfstudio before running this Viewer script.",
+ file=sys.stderr,
+ )
+ raise SystemExit(1) from exc
+ entrypoint()
+
+
+if __name__ == "__main__":
+ main()