Skip to content

SUPRA-Tex: 3D Generation with Super-Aligned Texture

10.7 KB
Raw Blame

Two-stage geometry frontend

The optional geometry frontend supports the public native Pixal3D pipeline and the historical paper implementation. By default both keep Pixal3D's fixed mesh scale; --geometry-scale-search adds bounded mesh-scale calibration with the main geometry model. Only the final mesh and its matching canonical camera are passed to SUPRA-Tex. Intermediate meshes are not saved by this adapter.

Requirements

The native geometry source is bundled at deployment/runtime/geometry. The source setup entrypoints install its dependencies and prepare matching weights, DINO and the local NAF cache. No separate Pixal3D source checkout is needed for the default path. Geometry requires CUDA and runs in a separate process using the same interpreter. An external compatible checkout can still be selected explicitly for the historical paper backend.

Native Pixal3D

The adapter automatically selects the native backend when the source checkout contains pixal3d/pipelines/pixal3d_image_to_3d.py. Its model root must contain pipeline.json, the five geometry model stems referenced by args.models, and a local DINOv3 model under dino/. A typical local layout is:

ckpts/ready/geometry/
  pipeline.json
  ckpts/
    ss_flow_img_dit_1_3B_64_bf16.{json,safetensors}
    ss_dec_conv3d_16l8_fp16.{json,safetensors}
    slat_flow_img2shape_dit_1_3B_512_bf16.{json,safetensors}
    slat_flow_img2shape_dit_1_3B_1024_bf16.{json,safetensors}
    shape_dec_next_dc_f16c32_fp16.{json,safetensors}
  dino/
    config.json
    model.safetensors

The five model keys are sparse_structure_flow_model, sparse_structure_decoder, shape_slat_flow_model_512, shape_slat_flow_model_1024, and shape_slat_decoder. Their JSON and weight files must remain inside the model root. DINO may instead use model.safetensors.index.json with all its referenced local shards. The repository ignores ckpts/, so these local checkpoint copies stay outside Git. Only geometry networks are loaded. Model preparation assembles this directory from the Pixal3D and DINOv3 downloads; the unmodified upstream Pixal3D snapshot alone is not the complete model root expected here. NAF requires the Torch Hub source at ${TORCH_HOME}/hub/valeoai_NAF_main and weights at ${TORCH_HOME}/hub/checkpoints/naf_release.pth. For the default prepared layout, set TORCH_HOME="$PWD/ckpts/ready/torch" when running the standalone frontend or geometry CLI from the source root. infer.py --pipeline and scripts/run_example.py set it from the prepared configuration automatically.

The five native networks are constructed on meta and receive validated local checkpoint tensors directly. Loading preserves the original constructor's per-layer dtypes, including BF16 flow blocks and FP32 decoder boundary layers; missing keys and incompatible shapes fail explicitly. This avoids initializing a full set of parameters that would immediately be overwritten. scripts/convert_geometry_weights.py can write *.runtime.safetensors copies already cast to these dtypes beside the originals; when present they are loaded instead, with identical results and about half the bytes read.

Paper Pixal3D

For the historical backend, supply the Pixal3D paper source containing pixal3dpipeline.py, pixal3dpipeline2stage.py, and its pixal3d/ package, without the native source entry above. The adapter disables the source's MoGe-v1 loader and reuses the prepared MoGe-2 camera. This backend requires its own compatible model layout:

geometry-weights/
  dense/
    dit/{config.json,model.safetensors}
    vae/{config.json,model.safetensors}
    conditioner/config.json
    scheduler/config.json
  sparse512/
    dit/{config.json,model.safetensors}
    vae/{config.json,model.safetensors}
    conditioner/config.json
    scheduler/config.json
  sparse1024/
    dit/{config.json,model.safetensors}
    vae/{config.json,model.safetensors}
    conditioner/config.json
    scheduler/config.json

All required configs and weights are checked before geometry launches. Paper scale probes always use the main dense model (use_dense_check=False); optional dense/scale_init files are neither required nor loaded. DINO/NAF dependencies and their pretrained caches follow the external source's loading policy; they are not included in the geometry checkpoint directory by this adapter.

Complete image frontend

After preparing the models with --with-rmbg for RGB inputs, add the two geometry arguments to the material frontend:

TORCH_HOME="$PWD/ckpts/ready/torch" python -m supratex.frontend.pipeline \
  --input inputs/reference.png --output inputs/object \
  --rmbg-model ckpts/ready/external/rmbg2 \
  --moge-model ckpts/ready/external/moge2 \
  --geometry-package deployment/runtime/geometry \
  --geometry-model ckpts/ready/geometry \
  --material-base ckpts/ready/external/qwen-base \
  --material-model ckpts/ready/project/material

The output includes geometry.pt, the final camera.json, condition RGB/RGBA, albedo, and MRA, ready for texturing.py. For a foreground RGBA input, use --use-input-alpha in place of --rmbg-model.

Geometry follows Pixal3D by default: the mesh scale stays at --geometry-initial-mesh-scale 1.0 and no scale probes run. --geometry-scale-search enables the bounded calibration described below, with --geometry-target-padding 3 and --geometry-max-iterations 2 by default. Each probe adds one 64-cell sparse-structure sampling (about 10 s on an RTX 3090; up to three probes). The model source accepts a local directory or a namespace/repo containing the complete model-root layout above; --geometry-revision selects its Hub revision. --cache-dir and --local-files-only also apply to geometry checkpoint resolution. Offline HF/Transformers flags are forwarded to the geometry process; external torch-hub dependencies need their caches prepared separately.

When scale search is enabled, the native adapter reuses sparse-structure image features across scale probes and final sparse-structure conditioning within the same inference. Shape stages sample DINO/NAF features only at occupied coordinates, preserving the original projection arithmetic and coordinate order. The receipt records these operations under conditioning_acceleration.

Geometry only

If the condition images and MoGe camera are already prepared, generate the final mesh without loading the material model:

TORCH_HOME="$PWD/ckpts/ready/torch" python -m supratex.frontend.geometry \
  --package deployment/runtime/geometry --model ckpts/ready/geometry \
  --input-dir inputs/prepared --output-dir outputs/geometry

Use a fresh output directory. It receives geometry.pt, camera.json, geometry_receipt.json, geometry.log, and copies of the condition images. The original inputs are preserved. The receipt records the selected scale, camera actually used for final generation, check-model usage, seed, and mesh counts. Without --geometry-scale-search, its scale_search entry is {"enabled": false, "probes": [], "stop_reason": "disabled"}. With it, the entry records probes, scale bounds, selected padding, convergence, baseline fallback, and the stopping reason; dense_probe_calls counts main-model probes and uses_dense_check_model is false. The standalone geometry command accepts the same option as --scale-search. Native receipts identify Pixal3D native geometry cascade and record final_sparse_structure_resolution = 32 and final resolution = 1024. In the complete frontend these diagnostics live under geometry/; the final mesh and matching camera are copied to the main input directory.

Scale and coordinate contract

With --geometry-scale-search, both backends use the paper scale-update formula with the adapter's bounded selection policy. Without it, they generate directly at the initial scale. The prepared MoGe-2 camera owns both FOV and distance. By default, scale starts at 1.0 and the frontend computes distance = 0.5 / tan(FOV / 2). The native backend uses this camera directly for 64-cell occupancy probes, then generates a fresh 32-cell sparse structure and runs the 512/1024 shape cascade. Its decoded mesh is already canonical and needs no rotation or scale conversion.

The paper backend retains the original outer inference path. The source first computes its own distance using initial scale and its image-padding convention; before any probe, the hook replaces that shared distance tensor with the prepared value. Its receipt retains both upstream_initial_generation_camera and initial_generation_camera and records the substitution. Both backends keep FOV and distance fixed throughout calibration and final generation and record camera_distance_policy = preserve_prepared_demo_camera.

Calibration aims for boundary padding in a 64-cubed occupancy grid. The default initial scale is 1.0 and target padding is 3 cells; padding from 2 through 4 cells is accepted. The baseline is probed first, followed by at most two scale updates (three probes total). --geometry-max-iterations must be positive and can lower the update budget; values above 2 still allow only two updates. Target padding must be an integer from 1 through 30. Proposals are bounded to half through twice the initial scale. Clipped candidates continue shrinking within those bounds.

Only nonempty, finite, integer occupancy coordinates inside the 64-cell grid are valid. A candidate is safe when its padding reaches the lower acceptance bound (2 cells by default). The search retains the safe candidate closest to the target, even if it never reaches the acceptance range. A safe baseline is retained unless a later safe candidate is closer; an unsafe baseline yields to any safe candidate, even if that candidate has a larger target error. An invalid later probe retains the best candidate already found. If no safe candidate exists, calibration falls back to the baseline. This optimizes occupancy padding, not FOV, pose, or image silhouette IoU.

The paper source exports its final mesh after rotating (x,y,z) to (x,-z,y) and scaling by 0.5 / optimized_mesh_scale. The adapter reverses the rotation and multiplies by the selected scale, recovering the canonical mesh in [-0.5,0.5]. The final camera retains the prepared generation distance and selected mesh_scale; SUPRA-Tex projection and DDA both use effective distance distance * mesh_scale. This paired conversion preserves the source projection. The original MoGe camera is retained under moge_initial_camera for provenance. Falling back to scale 1.0 restores the prepared baseline camera parameters. The paper and native models and image preprocessing differ, so identical geometry across the two backends is not promised. A safer occupancy margin also does not guarantee better shape, alignment, or texture quality.