SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution
| Name | Last commit message | Last commit date |
|---|---|---|
| assets | ||
| configs | ||
| data | ||
| examples | ||
| scripts | ||
| vlm_agent | ||
| .gitignore 857 B | ||
| .gitmodules 198 B | ||
| LICENSE 1.0 KB | ||
| README.md 8.6 KB | ||
| requirements-perception.txt 370 B | ||
| requirements.txt 613 B | ||
| sam-3d-objects | ||
| sam3 |
SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution
Xingjian Ran1,* Xiaoye Mo2,* Sihao Liu1 Jianyu Zhang1 Li Luo1 Bo Dai1,†
1 The University of Hong Kong 2 University of Electronic Science and Technology of China
* Equal contribution. † Corresponding author.

SceneMosaic turns a single indoor image into a complete, editable 3D scene. The
pipeline runs perception, promptable segmentation, per-object 3D reconstruction,
inter-object relation extraction, and scene evolution, and publishes a set of
representative scene variants as .glb files.
1. Clone with the upstream submodules
sam3 and sam-3d-objects are submodules pinned to the exact upstream commits
this project was validated against:
git clone --recurse-submodules https://github.com/rxjfighting/SceneMosaic.git
cd SceneMosaic
Already cloned without --recurse-submodules? Fetch them in place:
git submodule update --init
sam3/ must contain pyproject.toml and sam-3d-objects/ must contain
environments/default.yml. Upstream SAM3 main has since moved on to SAM 3.1,
whose inference code differs, so do not run git submodule update --remote
unless you intend to revalidate.
2. Prerequisites
- Linux x86-64 with Conda or Mamba.
- An NVIDIA GPU with at least 32 GB of VRAM for SAM 3D Objects inference (observed peak: about 38 GB for a single run).
- A system EGL runtime for headless rendering. On Ubuntu/Debian install
libegl1,libglvnd0, andlibegl-mesa0before creating the environments. - Access approval for the gated SAM3 and SAM 3D Objects model repositories on Hugging Face.
- An API key for an OpenAI-compatible chat endpoint.
3. Create the environments
SceneMosaic runs from one environment, scenemosaic, and delegates segmentation
to a second, sam3. SAM3 and SAM 3D Objects have incompatible Python, PyTorch,
and CUDA dependency stacks, so they cannot share an interpreter. One installer
per environment:
bash scripts/install_scenemosaic.sh
bash scripts/install_sam3.sh
scenemosaic is the project environment; every command in the rest of this
document runs inside it. sam3 holds nothing but the segmentation model — the
pipeline invokes its interpreter directly, and you never activate it by hand.
install_scenemosaic.sh also probes that PyOpenGL and pyrender can load EGL; if
that probe fails, install the system EGL packages from step 2 and rerun it.
Both names are defaults. SCENEMOSAIC_ENV_NAME and SAM3_ENV_NAME override
them, SCENEMOSAIC_ENV_PREFIX and SAM3_ENV_PREFIX install to an explicit Conda
prefix instead; pass whatever you choose to the configurator in step 6.
Behind a mirror or proxy, every package index is overridable:
SCENEMOSAIC_PIP_INDEX=https://mirror.example.com/simple bash scripts/install_sam3.sh
SCENEMOSAIC_TORCH_INDEX (install_sam3.sh), SCENEMOSAIC_PIP_EXTRA_INDEX, and
SCENEMOSAIC_KAOLIN_FIND_LINKS (install_scenemosaic.sh) work the same way.
4. Download model checkpoints
Authenticate once, after both gated repositories have granted access:
conda run -n scenemosaic python -m pip install "huggingface-hub[cli]<1.0"
conda run --no-capture-output -n scenemosaic hf auth login
mkdir -p checkpoints
SAM3 segmentation weights — listing the files explicitly avoids also pulling the unused PyTorch checkpoint from the same repository:
conda run -n scenemosaic hf download facebook/sam3 \
config.json merges.txt model.safetensors processor_config.json \
special_tokens_map.json tokenizer.json tokenizer_config.json vocab.json \
--repo-type model --local-dir checkpoints/sam3
SAM 3D Objects bundle — the staging directory preserves the upstream layout, so
move its nested checkpoints/ into place:
conda run -n scenemosaic hf download facebook/sam-3d-objects \
--repo-type model --local-dir checkpoints/sam3d-download \
--max-workers 1 --include "checkpoints/*"
mv checkpoints/sam3d-download/checkpoints checkpoints/sam3d
The MoGe depth model and the DINOv2 Torch Hub directory:
conda run -n scenemosaic hf download Ruicheng/moge-vitl
conda run -n scenemosaic python -c \
"import torch; torch.hub.set_dir(\"checkpoints/dinov2\"); torch.hub.load(\"facebookresearch/dinov2\", \"dinov2_vitl14_reg\", pretrained=True, trust_repo=True)"
Resulting layout:
checkpoints/
sam3/ config.json model.safetensors processor_config.json ...
sam3d/ pipeline.yaml ss_generator.ckpt slat_generator.ckpt ...
dinov2/ facebookresearch_dinov2_main/ checkpoints/*.pth
Checkpoint directories are untracked; never commit model weights or Hugging Face tokens.
5. PBR materials (optional)
Skipping this step is safe: room surfaces then fall back to the single bundled
material in data/materials/Marble001/. Complete it to enable automatic
text-to-material retrieval, which picks a different material per surface.
conda run -n scenemosaic python scripts/download_ambientcg.py \
--resolution 2K --format PNG --dry-run # add --limit 5 for a smoke test
conda run -n scenemosaic python scripts/download_ambientcg.py \
--resolution 2K --format PNG
conda run -n scenemosaic python scripts/compute_ambientcg_embeddings.py
ambientCG encodes resolution and format in every texture filename
(<asset>_2K-PNG_Color.png), so the whole pipeline must agree on one pair.
2K/PNG is the default everywhere; to use another variant, set it once so the
tools the pipeline invokes internally agree too:
export SCENEMOSAIC_MATERIAL_FORMAT=JPG # also SCENEMOSAIC_MATERIAL_RESOLUTION
Pass --overwrite to regenerate existing embeddings.
6. Configure the runtime and API access
The configurator locates both named Conda environments and writes their resolved interpreter paths, so no Conda path is ever copied by hand:
mkdir -p workspace/perception_attempts
python scripts/configure_runtime.py
One API key serves perception, relation extraction, and evolution. A custom
OpenAI-compatible gateway is optional; when OPENAI_BASE_URL is unset all three
stages use the official OpenAI endpoint:
export OPENAI_API_KEY="replace-with-your-key"
# Optional, for an OpenAI-compatible gateway:
# export OPENAI_BASE_URL="https://gateway.example/v1"
conda run -n scenemosaic python -m vlm_agent.cli verify
verify exits non-zero and names the failing check if anything is missing.
configs/runtime.local.json is machine-local and untracked; it holds resolved
interpreter paths and the names of the environment variables the pipeline
reads, never credential values. Rerun the configurator with --overwrite after
replacing an environment.
7. Run the end-to-end demo
Four input images ship in examples/:
conda run --no-capture-output -n scenemosaic \
python examples/run_end_to_end.py \
--example scene_092 \
--scene-name scene_092_demo
Selectors: scene_018, scene_031, scene_092, gaminglounge. For your own
image, replace --example <name> with --input-image path/to/image.png; an
explicit --scene-name is always required.
Results land in workspace/runs/<scene-name>/, with the published variants at
outputs/representative_scenes/scene_01..06.glb.
Inspect any run:
conda run -n scenemosaic python -m vlm_agent.cli \
--runtime-config configs/runtime.local.json \
status --scene-root workspace/runs/scene_092_demo
8. Resume an interrupted run
Recovery is stage-level. Point run at the existing scene root:
conda run --no-capture-output -n scenemosaic \
python -m vlm_agent.cli \
--runtime-config configs/runtime.local.json \
run --scene-root workspace/runs/<scene-name>
A valid perception publication, SAM3D output, relation publication, or completed scene is reused instead of recomputed; failed attempts are retained for inspection and never overwritten. An evolution failure therefore reuses perception, SAM3D, and relations but reruns evolution from the start.
If perception stopped after a reusable Pass A / SAM3-A prefix, select that attempt explicitly:
conda run --no-capture-output -n scenemosaic \
python -m vlm_agent.cli \
--runtime-config configs/runtime.local.json \
run --scene-root workspace/runs/<scene-name> \
--perception-reuse-local-pre-pass-b-cache-root \
workspace/perception_attempts/<perception-attempt>