Skip to content

SUPRA-Tex: 3D Generation with Super-Aligned Texture

9.9 KB
Raw Blame

Image-to-Material Translation frontend

SUPRA-Tex begins with foreground isolation, canonical-camera estimation, and material prediction. The default qwen-image backend uses the SUPRA-Tex Material Adapter based on Qwen-Image-2.1; firered selects the bundled factorized modality-axis joint-attention model. Both produce the aligned albedo and MRA maps used by surface-observation encoding.

Follow the fresh-checkout environment commands and CUDA extension instructions in Installation. They install the pinned Python dependencies and bundled MoGe-2/utils3d packages; the frontend extra alone is insufficient for the native Qwen runtime. The Qwen backend uses its native DiffSynth runtime and torch attention. The FireRed backend selects FA3 for its material Transformer on compatible Hopper setups and native Torch attention otherwise; its text encoder selects FA2 or SDPA. The default Qwen backend supports --low-vram CPU offload. See GPU compatibility for device settings and validation status, and Weights for model artifacts.

After the model preparation step, run the frontend using the prepared local paths. For an RGB input, prepare RMBG-2 as well:

python scripts/prepare_models.py --with-rmbg
python -m supratex.frontend.pipeline --input input.png --output work \
  --rmbg-model ckpts/ready/external/rmbg2 \
  --moge-model ckpts/ready/external/moge2 \
  --material-base ckpts/ready/external/qwen-base \
  --material-model ckpts/ready/project/material

These paths assume the default preparation directory and no local model overrides. The standalone frontend does not read models.json; when using a different preparation directory or local overrides, supply the corresponding paths recorded there. Without explicit material paths, this CLI falls back to ckpts/qwen-base and ckpts/qwen-material under the source root. The bundled Qwen runtime is selected automatically and uses packages installed in the active Python environment. --qwen-runtime can select another compatible local runtime; --qwen-deps optionally adds an existing dependency overlay. Base and model sources also accept Hugging Face namespace/repo IDs; --cache-dir and --local-files-only apply to resolution, and --revision selects the material checkpoint revision. Runtime and dependency paths must be local directories. For the same options on infer.py, use --material-revision instead of --revision (which selects the completion model there). Use a fresh output directory. The handoff contains condition_rgba.png, condition_rgb.png, camera.json, image_depth.npy, albedo.png, and mra.png. image_depth.npy stores float32 MoGe-2 camera-z depth on the condition pixel grid, with NaN for invalid pixels. Its dimensions must match condition_rgba.png; completion resizes it to the 1024-square observation grid with nearest-neighbor sampling, preserving invalid pixels. Single-view completion uses it to exclude source pixels whose depth disagrees with the generated geometry near depth edges. The gate is skipped when depth is absent or a valid fit cannot be obtained; malformed depth files are rejected.

The depth pre-filter is enabled by default for single-view infer.py and python -m supratex.frontend.pipeline runs. Pass --no-filter (alias --no-image-depth) to either command to disable it by omitting the depth file. infer.py --dual-view and frontend --record-material-provenance omit the file because dual-view completion does not support the pre-filter and rejects inputs containing image_depth.npy. The standalone camera command exports depth only when given --depth-output, as shown below.

The infer.py work directory's run.json records image_depth_filter.requested, enabled, and disabled_reason (user_disabled or dual_view_unsupported). These fields describe whether the depth input is enabled for the run; a later degenerate fit can still skip the gate. An enabled run has a null disabled reason.

For an existing non-opaque foreground RGBA, omit --with-rmbg during preparation and replace --rmbg-model with --use-input-alpha in the frontend command. Foreground preprocessing writes a straight-color condition_rgba.png and a matching black-premultiplied condition_rgb.png after reducing the longest input dimension to at most 1024 pixels and applying a square foreground crop with 10% padding. Qwen receives the straight RGBA frame, resized once to its 1024-square model canvas; FireRed receives the RGB condition. Do not premultiply Qwen's RGB channels before attaching alpha.

Foreground extraction uses input alpha or RMBG-2. Depth-based rejection occurs later during 3D observation construction: alpha erosion and first-hit depth discontinuity filtering discard unreliable observation pixels. It does not erode the saved foreground image using MoGe depth. See Texturing.

Supply geometry separately in the canonical frame, or set TORCH_HOME="$PWD/ckpts/ready/torch" and add --geometry-package deployment/runtime/geometry --geometry-model ckpts/ready/geometry to generate the final mesh with Pixal3D's fixed mesh scale (add --geometry-scale-search for bounded scale calibration). This uses the NAF cache created by model preparation. See the geometry frontend for source requirements and the matching mesh/camera handoff.

Qwen-Image material backend

With the default preparation layout, the model and runtime paths are:

ckpts/ready/
  external/qwen-base/                # complete Qwen-Image-2.1 base
  project/material/
    adapter_model.safetensors
    adapter_config.json
deployment/runtime/
  qwen/                             # bundled DiffSynth runtime

Keep the bundled runtime tree, including diffsynth/ and examples/qwen_image_21/pbr/{infer,inference_utils,inference_offload}.py. The adapter requires its native JSON config and complete base model, including processor, VAE, text encoder, and Transformer. The safe file includes learned modality embeddings as well as LoRA and uses the SUPRA-Tex native loader. Legacy checkpoint-0100000.pt plus its matching .complete.json also remain supported. Neither adapter format replaces the external base model.

Inference fixes the native prompts, 1024 resolution, seed 42, sigma 1, one endpoint-prediction step, and torch attention. The outputs are albedo and metallic/roughness only; there is no normal or AO prediction. Native RGBA outputs remain under material/native/. Canonical albedo.png and mra.png are RGB: MRA red is metallic, green is roughness, and blue is the input coverage. Native MR blue is a constant 255 placeholder, which the adapter replaces with input coverage for completion. This coverage is not a prediction of material transparency. Metallic and roughness use linear 0–255 encoding; albedo is sRGB.

FireRed material backend

Select FireRed explicitly with its own base and material bundle:

python -m supratex.frontend.pipeline --input input.png --output work \
  --rmbg-model /models/rmbg2 --moge-model /models/moge2 \
  --material-backend firered --material-base /models/firered-base \
  --material-model /models/firered-material

The legacy --firered-base option also supplies the FireRed base path. For this backend only, replace --material-model with all three explicit artifact arguments:

--material-weights full_transformer.safetensors \
--material-spec joint_contract.json \
--material-checkpoint-manifest checkpoint_complete.json

Individual foreground, camera, and FireRed stages remain available:

python -m supratex.frontend.preprocess --input input.png --rmbg-model /models/rmbg2 \
  --output-dir work
python -m supratex.frontend.camera --image work/condition_rgb.png --model /models/moge2 \
  --output work/camera.json --depth-output work/image_depth.npy
python -m supratex.frontend.firered --base-model /models/firered-base \
  --condition work/condition_rgb.png --weights full_transformer.safetensors \
  --spec joint_contract.json --checkpoint-manifest checkpoint_complete.json --output work/material

The FireRed adapter invokes the selected package's infer.py. The bundled package accepts a JSONL record containing the condition image and instructions; the adapter creates this record automatically. Additional fields retained for checkpoint compatibility are not used as material targets during inference. After decoding, the adapter fixes the blue channel of <key>_mra.png to 255 (normalized opacity 1) everywhere, preserving metallic in red, roughness in green, and the albedo pixels. The checkpoint's native material-alpha prompt is retained; its predicted blue channel is replaced before the maps reach the 3D material encoder. Foreground coverage continues to come from the separate condition mask. The adapter copies its <key>_albedo.png and <key>_mra.png outputs to the canonical albedo.png and mra.png primary material handoff names. Compatibility copies named firered_albedo.png and firered_mra.png are also written.

The bundled FireRed package is source code only. It retains the factorized ModalityAxisJointAttention modules and the exact modality_joint_blocks.* parameter names used by the checkpoint. The base model, factorized transformer weights, joint contract, and completion manifest must be supplied explicitly; no model weight is included in this repository. The frontend requires a checkpoint that produces 1024-by-1024 RGB maps. A compatible external package may be selected with --firered-package in the complete frontend, or --package in the standalone FireRed command; its own entrypoint contract then applies.

Camera handoff

The camera receipt has camera_angle_x, distance, and mesh_scale. These are the canonical fields consumed by the surface-observation path: camera at (0, 0, distance * mesh_scale), looking at the origin with +Y up. MoGe-2 intrinsics are normalized and its horizontal focal length determines both FOV and canonical distance. The optional image_width, image_height, and model receipt fields are provenance only.