Skip to content

SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

Python

Add SceneMosaic pipeline, examples, and setup documentation

Xingjian Ran9321c6a697ef
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..40dea27
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,23 @@
+__pycache__/
+*.py[cod]
+*.orig
+
+# /sam3/ and /sam-3d-objects/ are Git submodules pinning the upstream model
+# sources; they must NOT be ignored here, or `git submodule` would refuse them.
+
+# Machine-local SceneMosaic deployment profile.
+/configs/runtime.local.json
+
+# Server runtime assets; never commit checkpoints, model caches, or material corpora.
+# The material corpus is ignored wholesale, except the Marble001 fixture that the
+# legacy fixed-material path (generate_room_mesh.py) hard-requires. Negation must
+# sit at this level (one pattern below /data/materials/*); a nested .gitignore in
+# the ignored parent cannot re-include a child that a containing pattern matched.
+/checkpoints/
+/vlm_agent/evolution/cache/open_clip/
+/data/materials/*
+!/data/materials/Marble001/
+/data/embeddings/
+
+# Default workspace from the public setup guide.
+/workspace/
diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 0000000..d6638ce
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,6 @@
+[submodule "sam3"]
+ path = sam3
+ url = https://github.com/facebookresearch/sam3.git
+[submodule "sam-3d-objects"]
+ path = sam-3d-objects
+ url = https://github.com/facebookresearch/sam-3d-objects.git
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..e8364ce
--- /dev/null
+++ b/README.md
@@ -0,0 +1,242 @@
+# SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution
+
+Xingjian Ran<sup>1,&ast;</sup>&nbsp;&nbsp;
+Xiaoye Mo<sup>2,&ast;</sup>&nbsp;&nbsp;
+Sihao Liu<sup>1</sup>&nbsp;&nbsp;
+Jianyu Zhang<sup>1</sup>&nbsp;&nbsp;
+Li Luo<sup>1</sup>&nbsp;&nbsp;
+Bo Dai<sup>1,&dagger;</sup>
+
+<sup>1</sup> The University of Hong Kong&nbsp;&nbsp;
+<sup>2</sup> University of Electronic Science and Technology of China
+
+<sup>&ast;</sup> Equal contribution.&nbsp;&nbsp;<sup>&dagger;</sup> Corresponding author.
+
+![SceneMosaic teaser](assets/teaser.png)
+
+SceneMosaic turns a single indoor image into a complete, editable 3D scene. The
+pipeline runs perception, promptable segmentation, per-object 3D reconstruction,
+inter-object relation extraction, and scene evolution, and publishes a set of
+representative scene variants as `.glb` files.
+
+## 1. Clone with the upstream submodules
+
+`sam3` and `sam-3d-objects` are submodules pinned to the exact upstream commits
+this project was validated against:
+
+```bash
+git clone --recurse-submodules https://github.com/rxjfighting/SceneMosaic.git
+cd SceneMosaic
+```
+
+Already cloned without `--recurse-submodules`? Fetch them in place:
+
+```bash
+git submodule update --init
+```
+
+`sam3/` must contain `pyproject.toml` and `sam-3d-objects/` must contain
+`environments/default.yml`. Upstream SAM3 `main` has since moved on to SAM 3.1,
+whose inference code differs, so do not run `git submodule update --remote`
+unless you intend to revalidate.
+
+## 2. Prerequisites
+
+- Linux x86-64 with Conda or Mamba.
+- An NVIDIA GPU with at least 32 GB of VRAM for SAM 3D Objects inference
+ (observed peak: about 38 GB for a single run).
+- A system EGL runtime for headless rendering. On Ubuntu/Debian install
+ `libegl1`, `libglvnd0`, and `libegl-mesa0` *before* creating the environments.
+- Access approval for the gated
+ [SAM3](https://huggingface.co/facebook/sam3) and
+ [SAM 3D Objects](https://huggingface.co/facebook/sam-3d-objects)
+ model repositories on Hugging Face.
+- An API key for an OpenAI-compatible chat endpoint.
+
+## 3. Create the environments
+
+SceneMosaic runs from one environment, `scenemosaic`, and delegates segmentation
+to a second, `sam3`. SAM3 and SAM 3D Objects have incompatible Python, PyTorch,
+and CUDA dependency stacks, so they cannot share an interpreter. One installer
+per environment:
+
+```bash
+bash scripts/install_scenemosaic.sh
+bash scripts/install_sam3.sh
+```
+
+`scenemosaic` is the project environment; every command in the rest of this
+document runs inside it. `sam3` holds nothing but the segmentation model — the
+pipeline invokes its interpreter directly, and you never activate it by hand.
+`install_scenemosaic.sh` also probes that PyOpenGL and pyrender can load EGL; if
+that probe fails, install the system EGL packages from step 2 and rerun it.
+
+Both names are defaults. `SCENEMOSAIC_ENV_NAME` and `SAM3_ENV_NAME` override
+them, `SCENEMOSAIC_ENV_PREFIX` and `SAM3_ENV_PREFIX` install to an explicit Conda
+prefix instead; pass whatever you choose to the configurator in step 6.
+
+Behind a mirror or proxy, every package index is overridable:
+
+```bash
+SCENEMOSAIC_PIP_INDEX=https://mirror.example.com/simple bash scripts/install_sam3.sh
+```
+
+`SCENEMOSAIC_TORCH_INDEX` (`install_sam3.sh`), `SCENEMOSAIC_PIP_EXTRA_INDEX`, and
+`SCENEMOSAIC_KAOLIN_FIND_LINKS` (`install_scenemosaic.sh`) work the same way.
+
+## 4. Download model checkpoints
+
+Authenticate once, after both gated repositories have granted access:
+
+```bash
+conda run -n scenemosaic python -m pip install "huggingface-hub[cli]<1.0"
+conda run --no-capture-output -n scenemosaic hf auth login
+mkdir -p checkpoints
+```
+
+SAM3 segmentation weights — listing the files explicitly avoids also pulling the
+unused PyTorch checkpoint from the same repository:
+
+```bash
+conda run -n scenemosaic hf download facebook/sam3 \
+ config.json merges.txt model.safetensors processor_config.json \
+ special_tokens_map.json tokenizer.json tokenizer_config.json vocab.json \
+ --repo-type model --local-dir checkpoints/sam3
+```
+
+SAM 3D Objects bundle — the staging directory preserves the upstream layout, so
+move its nested `checkpoints/` into place:
+
+```bash
+conda run -n scenemosaic hf download facebook/sam-3d-objects \
+ --repo-type model --local-dir checkpoints/sam3d-download \
+ --max-workers 1 --include "checkpoints/*"
+mv checkpoints/sam3d-download/checkpoints checkpoints/sam3d
+```
+
+The MoGe depth model and the DINOv2 Torch Hub directory:
+
+```bash
+conda run -n scenemosaic hf download Ruicheng/moge-vitl
+conda run -n scenemosaic python -c \
+ "import torch; torch.hub.set_dir(\"checkpoints/dinov2\"); torch.hub.load(\"facebookresearch/dinov2\", \"dinov2_vitl14_reg\", pretrained=True, trust_repo=True)"
+```
+
+Resulting layout:
+
+```text
+checkpoints/
+ sam3/ config.json model.safetensors processor_config.json ...
+ sam3d/ pipeline.yaml ss_generator.ckpt slat_generator.ckpt ...
+ dinov2/ facebookresearch_dinov2_main/ checkpoints/*.pth
+```
+
+Checkpoint directories are untracked; never commit model weights or Hugging Face
+tokens.
+
+## 5. PBR materials (optional)
+
+Skipping this step is safe: room surfaces then fall back to the single bundled
+material in `data/materials/Marble001/`. Complete it to enable automatic
+text-to-material retrieval, which picks a different material per surface.
+
+```bash
+conda run -n scenemosaic python scripts/download_ambientcg.py \
+ --resolution 2K --format PNG --dry-run # add --limit 5 for a smoke test
+conda run -n scenemosaic python scripts/download_ambientcg.py \
+ --resolution 2K --format PNG
+conda run -n scenemosaic python scripts/compute_ambientcg_embeddings.py
+```
+
+ambientCG encodes resolution and format in every texture filename
+(`<asset>_2K-PNG_Color.png`), so the whole pipeline must agree on one pair.
+`2K`/`PNG` is the default everywhere; to use another variant, set it once so the
+tools the pipeline invokes internally agree too:
+
+```bash
+export SCENEMOSAIC_MATERIAL_FORMAT=JPG # also SCENEMOSAIC_MATERIAL_RESOLUTION
+```
+
+Pass `--overwrite` to regenerate existing embeddings.
+
+## 6. Configure the runtime and API access
+
+The configurator locates both named Conda environments and writes their resolved
+interpreter paths, so no Conda path is ever copied by hand:
+
+```bash
+mkdir -p workspace/perception_attempts
+python scripts/configure_runtime.py
+```
+
+One API key serves perception, relation extraction, and evolution. A custom
+OpenAI-compatible gateway is optional; when `OPENAI_BASE_URL` is unset all three
+stages use the official OpenAI endpoint:
+
+```bash
+export OPENAI_API_KEY="replace-with-your-key"
+# Optional, for an OpenAI-compatible gateway:
+# export OPENAI_BASE_URL="https://gateway.example/v1"
+
+conda run -n scenemosaic python -m vlm_agent.cli verify
+```
+
+`verify` exits non-zero and names the failing check if anything is missing.
+`configs/runtime.local.json` is machine-local and untracked; it holds resolved
+interpreter paths and the *names* of the environment variables the pipeline
+reads, never credential values. Rerun the configurator with `--overwrite` after
+replacing an environment.
+
+## 7. Run the end-to-end demo
+
+Four input images ship in [examples/](examples/README.md):
+
+```bash
+conda run --no-capture-output -n scenemosaic \
+ python examples/run_end_to_end.py \
+ --example scene_092 \
+ --scene-name scene_092_demo
+```
+
+Selectors: `scene_018`, `scene_031`, `scene_092`, `gaminglounge`. For your own
+image, replace `--example <name>` with `--input-image path/to/image.png`; an
+explicit `--scene-name` is always required.
+
+Results land in `workspace/runs/<scene-name>/`, with the published variants at
+`outputs/representative_scenes/scene_01..06.glb`.
+
+Inspect any run:
+
+```bash
+conda run -n scenemosaic python -m vlm_agent.cli \
+ --runtime-config configs/runtime.local.json \
+ status --scene-root workspace/runs/scene_092_demo
+```
+
+## 8. Resume an interrupted run
+
+Recovery is stage-level. Point `run` at the existing scene root:
+
+```bash
+conda run --no-capture-output -n scenemosaic \
+ python -m vlm_agent.cli \
+ --runtime-config configs/runtime.local.json \
+ run --scene-root workspace/runs/<scene-name>
+```
+
+A valid perception publication, SAM3D output, relation publication, or completed
+scene is reused instead of recomputed; failed attempts are retained for
+inspection and never overwritten. An evolution failure therefore reuses
+perception, SAM3D, and relations but reruns evolution from the start.
+
+If perception stopped after a reusable Pass A / SAM3-A prefix, select that
+attempt explicitly:
+
+```bash
+conda run --no-capture-output -n scenemosaic \
+ python -m vlm_agent.cli \
+ --runtime-config configs/runtime.local.json \
+ run --scene-root workspace/runs/<scene-name> \
+ --perception-reuse-local-pre-pass-b-cache-root \
+ workspace/perception_attempts/<perception-attempt>
+```
diff --git a/assets/teaser.png b/assets/teaser.png
new file mode 100644
index 0000000..d092c80
Binary files /dev/null and b/assets/teaser.png differ
diff --git a/configs/runtime.example.json b/configs/runtime.example.json
new file mode 100644
index 0000000..63b738e
--- /dev/null
+++ b/configs/runtime.example.json
@@ -0,0 +1,19 @@
+{
+ "schema_version": "scenemosaic_runtime_profile",
+ "conda_envs": {
+ "perception": "sam3",
+ "sam3d": "scenemosaic"
+ },
+ "workspace_root": "../workspace",
+ "perception_attempts_root": "../workspace/perception_attempts",
+ "perception_model_dir": "../checkpoints/sam3",
+ "sam3d_checkpoint": "../checkpoints/sam3d/pipeline.yaml",
+ "dino_hub_cache": "../checkpoints/dinov2",
+ "perception_api_key_env": "OPENAI_API_KEY",
+ "perception_api_base_env": "OPENAI_BASE_URL",
+ "relation_api_key_env": "OPENAI_API_KEY",
+ "relation_api_base_env": "OPENAI_BASE_URL",
+ "scenemosaic_api_key_env": "OPENAI_API_KEY",
+ "scenemosaic_api_base_env": "OPENAI_BASE_URL",
+ "vlm_model": "gpt-5.5"
+}
diff --git a/data/materials/Marble001/Marble001_2K-PNG_Color.png b/data/materials/Marble001/Marble001_2K-PNG_Color.png
new file mode 100644
index 0000000..58c15fa
Binary files /dev/null and b/data/materials/Marble001/Marble001_2K-PNG_Color.png differ
diff --git a/data/materials/Marble001/Marble001_2K-PNG_NormalGL.png b/data/materials/Marble001/Marble001_2K-PNG_NormalGL.png
new file mode 100644
index 0000000..3492eb1
Binary files /dev/null and b/data/materials/Marble001/Marble001_2K-PNG_NormalGL.png differ
diff --git a/data/materials/Marble001/Marble001_2K-PNG_Roughness.png b/data/materials/Marble001/Marble001_2K-PNG_Roughness.png
new file mode 100644
index 0000000..2e46ccd
Binary files /dev/null and b/data/materials/Marble001/Marble001_2K-PNG_Roughness.png differ
diff --git a/examples/README.md b/examples/README.md
new file mode 100644
index 0000000..441ef24
--- /dev/null
+++ b/examples/README.md
@@ -0,0 +1,67 @@
+# Public examples
+
+images/ contains four image inputs used to exercise SceneMosaic's complete
+public pipeline. They are inputs only: no model checkpoint, API credential,
+cache, intermediate artifact, or generated scene is included here.
+
+| Name | Image | Command selector |
+| --- | --- | --- |
+| Scene 018 | images/scene_018.png | scene_018 |
+| Scene 031 | images/scene_031.png | scene_031 |
+| Scene 092 | images/scene_092.png | scene_092 |
+| Gaming lounge | images/gaminglounge.png | gaminglounge |
+
+After completing the repository setup, create the workspace and machine-local
+runtime profile. The configurator resolves the default named Conda
+environments, so no interpreter path needs to be edited:
+
+~~~bash
+mkdir -p workspace/perception_attempts
+python scripts/configure_runtime.py
+export OPENAI_API_KEY="replace-with-your-key"
+# Optional for an OpenAI-compatible gateway:
+# export OPENAI_BASE_URL="https://gateway.example/v1"
+conda run -n scenemosaic python -m vlm_agent.cli verify
+~~~
+
+When OPENAI_BASE_URL is unset, perception, relation extraction, and evolution
+all use the official OpenAI gateway.
+
+Run a new complete scene with a fresh name:
+
+~~~bash
+conda run --no-capture-output -n scenemosaic \
+ python examples/run_end_to_end.py \
+ --example scene_092 \
+ --scene-name scene_092_demo
+~~~
+
+run_end_to_end.py delegates directly to python -m vlm_agent.cli run; it does
+not reimplement any pipeline stage. The run performs image inventory, SAM3
+segmentation, repository-local SAM 3D Objects reconstruction, relation
+extraction, and the native scene-evolution stage. It uses the configured GPU
+and the VLM services named in the local profile.
+
+The configured workspace receives runs/<scene-name>/. Its reconstruction/
+directory contains the perception and SAM3D artifacts, relations/ contains the
+accepted relation publication, and the scene root contains the final evolution
+publication plus outputs/representative_scenes/.
+
+`configs/runtime.local.json` is intentionally left untracked. The
+configurator creates it from `configs/runtime.example.json`, resolves the
+installed Conda interpreters, and leaves repository asset paths relative.
+Keeping it untracked allows machine-local environment locations without
+publishing them. The status command below assumes that setup step has already
+been completed.
+
+To inspect a completed or resumable run, use:
+
+~~~bash
+conda run -n scenemosaic python -m vlm_agent.cli \
+ --runtime-config configs/runtime.local.json \
+ status --scene-root workspace/runs/scene_092_demo
+~~~
+
+For a custom image, replace --example scene_092 with
+--input-image path/to/image.png. The command intentionally requires an
+explicit scene name and never stores credentials in command-line arguments.
diff --git a/examples/images/gaminglounge.png b/examples/images/gaminglounge.png
new file mode 100644
index 0000000..e8964ba
Binary files /dev/null and b/examples/images/gaminglounge.png differ
diff --git a/examples/images/scene_018.png b/examples/images/scene_018.png
new file mode 100644
index 0000000..0bfe3f5
Binary files /dev/null and b/examples/images/scene_018.png differ
diff --git a/examples/images/scene_031.png b/examples/images/scene_031.png
new file mode 100644
index 0000000..709fab0
Binary files /dev/null and b/examples/images/scene_031.png differ
diff --git a/examples/images/scene_092.png b/examples/images/scene_092.png
new file mode 100644
index 0000000..42e771f
Binary files /dev/null and b/examples/images/scene_092.png differ
diff --git a/examples/run_end_to_end.py b/examples/run_end_to_end.py
new file mode 100644
index 0000000..f119479
--- /dev/null
+++ b/examples/run_end_to_end.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""Run the public SceneMosaic pipeline from one image to final evolution."""
+
+from __future__ import annotations
+
+import argparse
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+
+REPOSITORY_ROOT = Path(__file__).resolve().parents[1]
+EXAMPLE_IMAGES = {
+ "scene_018": "scene_018.png",
+ "scene_031": "scene_031.png",
+ "scene_092": "scene_092.png",
+ "gaminglounge": "gaminglounge.png",
+}
+SCENE_NAME_RE = re.compile(r"^[A-Za-z0-9_-]+$")
+
+
+def _parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(
+ description=(
+ "Run SceneMosaic from an input image through perception, "
+ "reconstruction, relation extraction, and scene evolution."
+ )
+ )
+ source = parser.add_mutually_exclusive_group(required=True)
+ source.add_argument(
+ "--example",
+ choices=sorted(EXAMPLE_IMAGES),
+ help="Use one of the repository's public example images.",
+ )
+ source.add_argument(
+ "--input-image",
+ type=Path,
+ help="Use a custom input image instead of a bundled example.",
+ )
+ parser.add_argument(
+ "--scene-name",
+ required=True,
+ help="New scene identifier; use a fresh value to start a new run.",
+ )
+ parser.add_argument(
+ "--runtime-config",
+ type=Path,
+ default=REPOSITORY_ROOT / "configs" / "runtime.local.json",
+ help=(
+ "Secret-free runtime profile (default: "
+ "configs/runtime.local.json)."
+ ),
+ )
+ return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = _parser().parse_args(argv)
+ if not SCENE_NAME_RE.fullmatch(args.scene_name):
+ raise SystemExit(
+ "error: --scene-name may contain only letters, digits, '_' and '-'."
+ )
+
+ image = (
+ REPOSITORY_ROOT / "examples" / "images" / EXAMPLE_IMAGES[args.example]
+ if args.example
+ else args.input_image
+ )
+ assert image is not None
+ runtime_config = args.runtime_config
+ if not image.is_file():
+ raise SystemExit(f"error: input image is missing: {image}")
+ if not runtime_config.is_file():
+ raise SystemExit(
+ "error: runtime configuration is missing: "
+ f"{runtime_config}. Run python scripts/configure_runtime.py first."
+ )
+
+ command = [
+ sys.executable,
+ "-m",
+ "vlm_agent.cli",
+ "--runtime-config",
+ str(runtime_config),
+ "run",
+ "--input-image",
+ str(image),
+ "--scene-name",
+ args.scene_name,
+ ]
+ return subprocess.run(command, cwd=REPOSITORY_ROOT, check=False).returncode
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/requirements-perception.txt b/requirements-perception.txt
new file mode 100644
index 0000000..c53365b
--- /dev/null
+++ b/requirements-perception.txt
@@ -0,0 +1,11 @@
+# Direct dependencies of the SceneMosaic perception pipeline.
+#
+# Upstream SAM3 dependencies are installed from sam3/pyproject.toml. PyTorch,
+# torchvision, and torchaudio remain pinned by scripts/install_sam3.sh so the
+# CUDA build cannot be replaced by a transitive dependency.
+
+Pillow>=10.0,<13
+opencv-python>=4.8,<5
+scipy>=1.10,<2
+transformers==5.5.4
+openai>=2.0,<3
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..3040290
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,26 @@
+# SceneMosaic SAM3D, scene-evolution, and material-retrieval dependencies.
+#
+# Install this file in the SAM3D environment only. The separate SAM3
+# environment is reserved for image segmentation and uses SAM3's own
+# dependencies. PyTorch, PyTorch3D, and SAM3D remain managed by their
+# upstream, CUDA-specific installation procedure.
+
+numpy>=1.26,<2
+opencv-python>=4.8
+Pillow>=10.0
+PyYAML>=6.0
+requests>=2.31
+scipy>=1.10
+tqdm>=4.66
+trimesh>=4.0
+pyrender>=0.1.45
+
+# Scene evolution and ambientCG material retrieval.
+sapien
+fast-simplification
+coacd
+shapely
+openai
+pybullet
+PyOpenGL==3.1.0
+open_clip_torch==2.29.0
diff --git a/sam-3d-objects b/sam-3d-objects
new file mode 160000
index 0000000..81a8237
--- /dev/null
+++ b/sam-3d-objects
@@ -0,0 +1 @@
+Subproject commit 81a82373a3a7f4cbb00bd5b32aaf6b4d0f659ddd
diff --git a/sam3 b/sam3
new file mode 160000
index 0000000..86ed770
--- /dev/null
+++ b/sam3
@@ -0,0 +1 @@
+Subproject commit 86ed77094094e5cabb16b0414ec60c5ba9ce0a0f
diff --git a/scripts/compute_ambientcg_embeddings.py b/scripts/compute_ambientcg_embeddings.py
new file mode 100755
index 0000000..90f37fc
--- /dev/null
+++ b/scripts/compute_ambientcg_embeddings.py
@@ -0,0 +1,261 @@
+#!/usr/bin/env python3
+"""Compute CLIP image embeddings for locally downloaded ambientCG materials."""
+
+from __future__ import annotations
+
+import argparse
+import os
+from pathlib import Path
+
+import numpy as np
+import torch
+import yaml
+from PIL import Image
+from tqdm import tqdm
+
+
+PROJECT_ROOT = Path(__file__).resolve().parents[1]
+DEFAULT_MATERIALS_DIR = PROJECT_ROOT / "data" / "materials"
+DEFAULT_EMBEDDINGS_DIR = PROJECT_ROOT / "data" / "embeddings"
+DEFAULT_CACHE_DIR = PROJECT_ROOT / "vlm_agent" / "evolution" / "cache" / "open_clip"
+MODEL_NAME = "ViT-H-14-378-quickgelu"
+PRETRAINED = "dfn5b"
+EXPECTED_DIMENSION = 1024
+DEFAULT_TEXTURE_RESOLUTION = "2K"
+DEFAULT_TEXTURE_FORMAT = "PNG"
+TEXTURE_RESOLUTIONS = ("1K", "2K", "4K", "8K", "12K", "16K")
+TEXTURE_FORMATS = ("PNG", "JPG")
+
+
+def env_texture_resolution() -> str:
+ """Resolve the texture resolution from the environment (default 2K)."""
+ value = os.environ.get("SCENEMOSAIC_MATERIAL_RESOLUTION")
+ return value if value in TEXTURE_RESOLUTIONS else DEFAULT_TEXTURE_RESOLUTION
+
+
+def env_texture_format() -> str:
+ """Resolve the texture format from the environment (default PNG)."""
+ value = os.environ.get("SCENEMOSAIC_MATERIAL_FORMAT")
+ return value if value in TEXTURE_FORMATS else DEFAULT_TEXTURE_FORMAT
+
+
+def texture_path(
+ material_dir: Path,
+ channel: str,
+ *,
+ resolution: str | None = None,
+ file_format: str | None = None,
+) -> Path:
+ """Return the path of one ambientCG texture of ``material_dir``.
+
+ ambientCG names textures ``<material>_<res>-<fmt>_<channel>.<ext>`` where
+ the filename label uses the format in uppercase (e.g. ``2K-PNG_Color.png``
+ for PNG, ``2K-JPG_Color.jpg`` for JPG). ``channel`` is the map suffix such
+ as ``Color``, ``NormalGL`` or ``Roughness``. Unset arguments fall back to
+ the environment overrides, then to 2K PNG.
+ """
+ resolution = resolution or env_texture_resolution()
+ file_format = file_format or env_texture_format()
+ extension = file_format.lower()
+ return material_dir / (
+ f"{material_dir.name}_{resolution}-{file_format}_{channel}.{extension}"
+ )
+
+
+def discover_materials(
+ materials_dir: Path,
+ *,
+ resolution: str | None = None,
+ file_format: str | None = None,
+) -> list[tuple[str, Path]]:
+ """Return deterministic material IDs and their base-color textures."""
+
+ resolution = resolution or env_texture_resolution()
+ file_format = file_format or env_texture_format()
+ discovered: list[tuple[str, Path]] = []
+ if not materials_dir.is_dir():
+ raise FileNotFoundError(f"Material directory not found: {materials_dir}")
+
+ for material_dir in sorted(materials_dir.iterdir(), key=lambda path: path.name):
+ if not material_dir.is_dir() or material_dir.is_symlink():
+ continue
+ color_path = texture_path(
+ material_dir, "Color", resolution=resolution, file_format=file_format
+ )
+ if color_path.is_file() and not color_path.is_symlink():
+ discovered.append((material_dir.name, color_path))
+
+ if not discovered:
+ raise FileNotFoundError(
+ f"No ambientCG {resolution}-{file_format} Color textures found below {materials_dir}"
+ )
+ return discovered
+
+
+def encode_materials(
+ materials: list[tuple[str, Path]],
+ *,
+ device: str,
+ batch_size: int,
+ cache_dir: Path,
+) -> np.ndarray:
+ """Encode base-color textures with the retrieval model used at runtime."""
+
+ import open_clip
+ cache_dir.mkdir(parents=True, exist_ok=True)
+ model, _, preprocess = open_clip.create_model_and_transforms(
+ MODEL_NAME,
+ pretrained=PRETRAINED,
+ device=device,
+ cache_dir=str(cache_dir),
+ )
+ model.eval()
+
+ encoded_batches: list[np.ndarray] = []
+ with torch.inference_mode():
+ for start in tqdm(
+ range(0, len(materials), batch_size),
+ desc="Computing ambientCG embeddings",
+ ):
+ batch_items = materials[start : start + batch_size]
+ tensors = []
+ for _, color_path in batch_items:
+ with Image.open(color_path) as image:
+ tensors.append(preprocess(image.convert("RGB")))
+ image_batch = torch.stack(tensors).to(device)
+ features = model.encode_image(image_batch)
+ features = features / features.norm(dim=-1, keepdim=True)
+ encoded_batches.append(
+ features.detach().cpu().numpy().astype(np.float32, copy=False)
+ )
+
+ embeddings = np.concatenate(encoded_batches, axis=0)
+ if embeddings.shape != (len(materials), EXPECTED_DIMENSION):
+ raise ValueError(
+ "Unexpected embedding shape: "
+ f"received {embeddings.shape}, expected "
+ f"({len(materials)}, {EXPECTED_DIMENSION})"
+ )
+ if not np.isfinite(embeddings).all():
+ raise ValueError("Computed embeddings contain non-finite values")
+ return embeddings
+
+
+def write_outputs(
+ output_dir: Path,
+ material_ids: list[str],
+ embeddings: np.ndarray,
+ *,
+ overwrite: bool,
+) -> None:
+ """Write the matrix and row index without exposing partial final files."""
+
+ output_dir.mkdir(parents=True, exist_ok=True)
+ embeddings_path = output_dir / "clip_embeddings.npy"
+ index_path = output_dir / "embedding_index.yaml"
+ existing = [path for path in (embeddings_path, index_path) if path.exists()]
+ if existing and not overwrite:
+ names = ", ".join(str(path) for path in existing)
+ raise FileExistsError(f"Output already exists: {names}; pass --overwrite")
+
+ embeddings_tmp = output_dir / ".clip_embeddings.npy.tmp"
+ index_tmp = output_dir / ".embedding_index.yaml.tmp"
+ with embeddings_tmp.open("wb") as handle:
+ np.save(handle, embeddings, allow_pickle=False)
+ index_tmp.write_text(
+ yaml.safe_dump(material_ids, sort_keys=False),
+ encoding="utf-8",
+ )
+ embeddings_tmp.replace(embeddings_path)
+ index_tmp.replace(index_path)
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(
+ description="Compute CLIP embeddings for downloaded ambientCG materials"
+ )
+ parser.add_argument(
+ "--materials-dir",
+ type=Path,
+ default=DEFAULT_MATERIALS_DIR,
+ help="ambientCG material directory (default: <project>/data/materials)",
+ )
+ parser.add_argument(
+ "--output-dir",
+ type=Path,
+ default=DEFAULT_EMBEDDINGS_DIR,
+ help="embedding output directory (default: <project>/data/embeddings)",
+ )
+ parser.add_argument(
+ "--cache-dir",
+ type=Path,
+ default=DEFAULT_CACHE_DIR,
+ help=(
+ "OpenCLIP model cache "
+ "(default: <project>/vlm_agent/evolution/cache/open_clip)"
+ ),
+ )
+ parser.add_argument(
+ "--device",
+ default=None,
+ help="Torch device; defaults to CUDA when available, otherwise CPU",
+ )
+ parser.add_argument(
+ "--resolution",
+ "-r",
+ type=str,
+ default=env_texture_resolution(),
+ choices=list(TEXTURE_RESOLUTIONS),
+ help=(
+ "Texture resolution (default: $SCENEMOSAIC_MATERIAL_RESOLUTION "
+ f"or {DEFAULT_TEXTURE_RESOLUTION})"
+ ),
+ )
+ parser.add_argument(
+ "--format",
+ "-f",
+ type=str,
+ default=env_texture_format(),
+ choices=list(TEXTURE_FORMATS),
+ help=(
+ "Texture file format (default: $SCENEMOSAIC_MATERIAL_FORMAT "
+ f"or {DEFAULT_TEXTURE_FORMAT})"
+ ),
+ )
+ parser.add_argument("--batch-size", type=int, default=16)
+ parser.add_argument("--overwrite", action="store_true")
+ args = parser.parse_args()
+
+ if args.batch_size <= 0:
+ parser.error("--batch-size must be positive")
+
+ materials_dir = args.materials_dir.expanduser().resolve()
+ output_dir = args.output_dir.expanduser().resolve()
+ cache_dir = args.cache_dir.expanduser().resolve()
+ device = args.device or ("cuda" if torch.cuda.is_available() else "cpu")
+
+ materials = discover_materials(
+ materials_dir,
+ resolution=args.resolution,
+ file_format=args.format,
+ )
+ embeddings = encode_materials(
+ materials,
+ device=device,
+ batch_size=args.batch_size,
+ cache_dir=cache_dir,
+ )
+ write_outputs(
+ output_dir,
+ [material_id for material_id, _ in materials],
+ embeddings,
+ overwrite=args.overwrite,
+ )
+ print(
+ f"Wrote {len(materials)} {EXPECTED_DIMENSION}D embeddings to {output_dir}"
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/configure_runtime.py b/scripts/configure_runtime.py
new file mode 100755
index 0000000..50e92f8
--- /dev/null
+++ b/scripts/configure_runtime.py
@@ -0,0 +1,195 @@
+#!/usr/bin/env python3
+"""Create the machine-local runtime profile from installed Conda environments."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import uuid
+from pathlib import Path
+from typing import Any
+
+
+PROJECT_ROOT = Path(__file__).resolve().parents[1]
+DEFAULT_TEMPLATE = PROJECT_ROOT / "configs" / "runtime.example.json"
+DEFAULT_OUTPUT = PROJECT_ROOT / "configs" / "runtime.local.json"
+
+
+class ConfigurationError(RuntimeError):
+ """Raised when an installed environment cannot be resolved safely."""
+
+
+def _read_object(path: Path, label: str) -> dict[str, Any]:
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise ConfigurationError(f"cannot read {label}: {path}") from exc
+ if not isinstance(payload, dict):
+ raise ConfigurationError(f"{label} must contain a JSON object: {path}")
+ return payload
+
+
+def _find_conda_executable() -> Path:
+ candidates = [
+ os.environ.get("CONDA_EXE"),
+ shutil.which("conda"),
+ shutil.which("mamba"),
+ ]
+ for candidate in candidates:
+ if not candidate:
+ continue
+ path = Path(candidate).expanduser().resolve()
+ if path.is_file() and os.access(path, os.X_OK):
+ return path
+ raise ConfigurationError("conda or mamba executable was not found")
+
+
+def _list_environment_prefixes(executable: Path) -> tuple[Path, ...]:
+ try:
+ completed = subprocess.run(
+ [str(executable), "env", "list", "--json"],
+ capture_output=True,
+ text=True,
+ timeout=30,
+ check=False,
+ )
+ except (OSError, subprocess.SubprocessError) as exc:
+ raise ConfigurationError("cannot query Conda environments") from exc
+ if completed.returncode != 0:
+ raise ConfigurationError("conda env list --json failed")
+ try:
+ payload = json.loads(completed.stdout)
+ except json.JSONDecodeError as exc:
+ raise ConfigurationError("conda env list returned invalid JSON") from exc
+ raw_prefixes = payload.get("envs") if isinstance(payload, dict) else None
+ if not isinstance(raw_prefixes, list):
+ raise ConfigurationError("conda env list returned no environment list")
+ return tuple(
+ Path(item).expanduser().resolve()
+ for item in raw_prefixes
+ if isinstance(item, str) and item
+ )
+
+
+def _python_from_prefix(prefix: Path, label: str) -> Path:
+ resolved_prefix = prefix.expanduser().resolve()
+ python = (resolved_prefix / "bin" / "python").resolve()
+ if (
+ not resolved_prefix.is_dir()
+ or not python.is_file()
+ or not os.access(python, os.X_OK)
+ ):
+ raise ConfigurationError(
+ f"{label} is not a usable Conda environment: {resolved_prefix}"
+ )
+ return python
+
+
+def _resolve_named_environment(
+ prefixes: tuple[Path, ...],
+ name: str,
+ label: str,
+) -> Path:
+ normalized = name.strip()
+ if not normalized or "/" in normalized or "\\" in normalized:
+ raise ConfigurationError(f"{label} environment name is invalid")
+ matches = [prefix for prefix in prefixes if prefix.name == normalized]
+ if not matches:
+ raise ConfigurationError(
+ f"Conda environment {normalized!r} was not found for {label}"
+ )
+ if len(matches) > 1:
+ raise ConfigurationError(
+ f"multiple Conda environments named {normalized!r}; use --{label}-prefix"
+ )
+ return _python_from_prefix(matches[0], label)
+
+
+def _atomic_write(path: Path, payload: dict[str, Any], overwrite: bool) -> None:
+ path = path.expanduser().resolve()
+ if path.parent != (PROJECT_ROOT / "configs").resolve():
+ raise ConfigurationError("runtime profile must be written below configs/")
+ if path.exists() or path.is_symlink():
+ if not overwrite:
+ raise ConfigurationError(
+ f"runtime profile already exists: {path}; pass --overwrite"
+ )
+ if path.is_symlink() or not path.is_file():
+ raise ConfigurationError(
+ f"refusing to overwrite unsafe runtime profile path: {path}"
+ )
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.parent / f".{path.name}.{uuid.uuid4().hex}.tmp"
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def build_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(
+ description="Resolve Conda environments and create configs/runtime.local.json"
+ )
+ parser.add_argument("--template", type=Path, default=DEFAULT_TEMPLATE)
+ parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT)
+ sam3 = parser.add_mutually_exclusive_group()
+ sam3.add_argument("--sam3-env")
+ sam3.add_argument("--sam3-prefix", type=Path)
+ scenemosaic = parser.add_mutually_exclusive_group()
+ scenemosaic.add_argument("--scenemosaic-env")
+ scenemosaic.add_argument("--scenemosaic-prefix", type=Path)
+ parser.add_argument("--overwrite", action="store_true")
+ return parser
+
+
+def main() -> int:
+ args = build_parser().parse_args()
+ template = _read_object(args.template.expanduser().resolve(), "runtime template")
+ conda_envs = template.pop("conda_envs", None)
+ if not isinstance(conda_envs, dict):
+ raise ConfigurationError("runtime template is missing conda_envs")
+
+ sam3_name = args.sam3_env or conda_envs.get("perception")
+ sam3d_name = args.scenemosaic_env or conda_envs.get("sam3d")
+ prefixes: tuple[Path, ...] = ()
+ if args.sam3_prefix is None or args.scenemosaic_prefix is None:
+ prefixes = _list_environment_prefixes(_find_conda_executable())
+
+ perception_python = (
+ _python_from_prefix(args.sam3_prefix, "sam3")
+ if args.sam3_prefix is not None
+ else _resolve_named_environment(prefixes, str(sam3_name or ""), "sam3")
+ )
+ sam3d_python = (
+ _python_from_prefix(args.scenemosaic_prefix, "scenemosaic")
+ if args.scenemosaic_prefix is not None
+ else _resolve_named_environment(
+ prefixes, str(sam3d_name or ""), "scenemosaic"
+ )
+ )
+
+ template.update(
+ {
+ "perception_python": str(perception_python),
+ "sam3d_python": str(sam3d_python),
+ "relation_precompute_python": str(sam3d_python),
+ "relation_runner_python": str(sam3d_python),
+ "scenemosaic_python": str(sam3d_python),
+ }
+ )
+ _atomic_write(args.output, template, args.overwrite)
+ print(f"Wrote runtime profile: {args.output.expanduser().resolve()}")
+ print(f"sam3 Python: {perception_python}")
+ print(f"scenemosaic Python: {sam3d_python}")
+ return 0
+
+
+if __name__ == "__main__":
+ try:
+ raise SystemExit(main())
+ except ConfigurationError as exc:
+ raise SystemExit(f"configuration failed: {exc}") from None
diff --git a/scripts/download_ambientcg.py b/scripts/download_ambientcg.py
new file mode 100755
index 0000000..2870d76
--- /dev/null
+++ b/scripts/download_ambientcg.py
@@ -0,0 +1,417 @@
+"""Download PBR materials from AmbientCG.
+
+AmbientCG provides free CC0 PBR materials for 3D rendering. This script
+downloads materials in bulk with configurable resolution and format.
+
+Usage:
+ python scripts/download_ambientcg.py --resolution 2K --format PNG
+ python scripts/download_ambientcg.py -r 4K -f JPG -o data/materials -c 8
+ python scripts/download_ambientcg.py --dry-run # List without downloading
+
+API Documentation: https://docs.ambientcg.com/
+"""
+
+import argparse
+import logging
+import sys
+import time
+import zipfile
+
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from dataclasses import dataclass
+from pathlib import Path
+
+import requests
+
+from tqdm import tqdm
+
+console_logger = logging.getLogger(__name__)
+
+PROJECT_ROOT = Path(__file__).resolve().parents[1]
+DEFAULT_MATERIALS_DIR = PROJECT_ROOT / "data" / "materials"
+
+# API configuration.
+API_BASE_URL = "https://ambientcg.com/api/v2/full_json"
+DOWNLOAD_BASE_URL = "https://ambientcg.com/get"
+PAGE_SIZE = 250
+MAX_RETRIES = 3
+RETRY_DELAY = 2.0 # seconds
+
+
+@dataclass
+class MaterialDownload:
+ """Represents a material to download."""
+
+ asset_id: str
+ download_url: str
+ file_name: str
+ size_bytes: int
+
+
+def fetch_materials_page(
+ offset: int, session: requests.Session
+) -> tuple[list[dict], int]:
+ """Fetch a single page of materials from the API.
+
+ Args:
+ offset: Pagination offset.
+ session: Requests session for connection pooling.
+
+ Returns:
+ Tuple of (list of asset dicts, total number of results).
+
+ Raises:
+ requests.RequestException: If the API request fails.
+ """
+ params = {
+ "type": "Material",
+ "include": "downloadData",
+ "limit": PAGE_SIZE,
+ "offset": offset,
+ }
+
+ response = session.get(url=API_BASE_URL, params=params, timeout=30)
+ response.raise_for_status()
+
+ data = response.json()
+ return data.get("foundAssets", []), data.get("numberOfResults", 0)
+
+
+def fetch_all_materials(session: requests.Session) -> list[dict]:
+ """Fetch all materials from the API with pagination.
+
+ Args:
+ session: Requests session for connection pooling.
+
+ Returns:
+ List of all material asset dicts.
+ """
+ all_assets = []
+ offset = 0
+ total = None
+
+ console_logger.info("Fetching material list from AmbientCG API...")
+
+ while True:
+ assets, total_results = fetch_materials_page(offset=offset, session=session)
+
+ if total is None:
+ total = total_results
+ console_logger.info(f"Found {total} materials total")
+
+ all_assets.extend(assets)
+ console_logger.info(f"Fetched {len(all_assets)}/{total} materials")
+
+ if len(assets) < PAGE_SIZE or len(all_assets) >= total:
+ break
+
+ offset += PAGE_SIZE
+ time.sleep(0.1) # Small delay to be respectful.
+
+ return all_assets
+
+
+def filter_downloads(
+ assets: list[dict], resolution: str, file_format: str
+) -> list[MaterialDownload]:
+ """Filter assets to get download URLs for the specified resolution/format.
+
+ Args:
+ assets: List of asset dicts from the API.
+ resolution: Desired resolution (e.g., "2K").
+ file_format: Desired format (e.g., "PNG").
+
+ Returns:
+ List of MaterialDownload objects.
+ """
+ target_attribute = f"{resolution}-{file_format}"
+ downloads = []
+
+ for asset in assets:
+ asset_id = asset.get("assetId", "")
+
+ # Navigate the nested download structure.
+ download_folders = asset.get("downloadFolders", {})
+ default_folder = download_folders.get("default", {})
+ filetype_categories = default_folder.get("downloadFiletypeCategories", {})
+
+ if isinstance(filetype_categories, dict):
+ zip_category = filetype_categories.get("zip", {})
+ elif isinstance(filetype_categories, list):
+ zip_category = next(
+ (
+ category
+ for category in filetype_categories
+ if isinstance(category, dict) and category.get("title") == "zip"
+ ),
+ {},
+ )
+ else:
+ zip_category = {}
+
+ download_list = (
+ zip_category.get("downloads", []) if isinstance(zip_category, dict) else []
+ )
+ if not isinstance(download_list, list):
+ continue
+
+ # Find matching download.
+ for download in download_list:
+ if not isinstance(download, dict):
+ continue
+ if download.get("attribute") == target_attribute:
+ downloads.append(
+ MaterialDownload(
+ asset_id=asset_id,
+ download_url=download.get("downloadLink", ""),
+ file_name=download.get("fileName", ""),
+ size_bytes=download.get("size", 0),
+ )
+ )
+ break
+
+ return downloads
+
+
+def download_material(
+ material: MaterialDownload,
+ output_dir: Path,
+ session: requests.Session,
+ extract: bool = True,
+) -> bool:
+ """Download and optionally extract a single material.
+
+ Args:
+ material: MaterialDownload object with download info.
+ output_dir: Base output directory.
+ session: Requests session for connection pooling.
+ extract: Whether to extract the ZIP file.
+
+ Returns:
+ True if successful, False otherwise.
+ """
+ material_dir = output_dir / material.asset_id
+
+ # Skip if already downloaded.
+ if material_dir.exists() and any(material_dir.iterdir()):
+ return True
+
+ zip_path = output_dir / material.file_name
+
+ # Download with retries.
+ for attempt in range(MAX_RETRIES):
+ try:
+ response = session.get(
+ url=material.download_url,
+ stream=True,
+ timeout=120,
+ )
+ response.raise_for_status()
+
+ # Write to file.
+ with open(zip_path, "wb") as f:
+ for chunk in response.iter_content(chunk_size=8192):
+ f.write(chunk)
+
+ break
+
+ except requests.RequestException as e:
+ if attempt < MAX_RETRIES - 1:
+ console_logger.warning(
+ f"Retry {attempt + 1}/{MAX_RETRIES} for {material.asset_id}: {e}"
+ )
+ time.sleep(RETRY_DELAY)
+ else:
+ console_logger.error(f"Failed to download {material.asset_id}: {e}")
+ return False
+
+ # Extract if requested.
+ if extract:
+ try:
+ material_dir.mkdir(parents=True, exist_ok=True)
+ with zipfile.ZipFile(zip_path, "r") as zf:
+ zf.extractall(material_dir)
+ zip_path.unlink() # Remove ZIP after extraction.
+ except zipfile.BadZipFile as e:
+ console_logger.error(f"Failed to extract {material.asset_id}: {e}")
+ zip_path.unlink(missing_ok=True)
+ return False
+
+ return True
+
+
+def download_all_materials(
+ downloads: list[MaterialDownload],
+ output_dir: Path,
+ concurrent: int,
+ extract: bool,
+) -> tuple[int, int]:
+ """Download all materials with concurrent workers.
+
+ Args:
+ downloads: List of MaterialDownload objects.
+ output_dir: Base output directory.
+ concurrent: Number of concurrent download workers.
+ extract: Whether to extract ZIP files.
+
+ Returns:
+ Tuple of (successful count, failed count).
+ """
+ output_dir.mkdir(parents=True, exist_ok=True)
+
+ successful = 0
+ failed = 0
+
+ # Use session for connection pooling.
+ session = requests.Session()
+
+ with ThreadPoolExecutor(max_workers=concurrent) as executor:
+ futures = {
+ executor.submit(
+ download_material,
+ material=material,
+ output_dir=output_dir,
+ session=session,
+ extract=extract,
+ ): material
+ for material in downloads
+ }
+
+ with tqdm(total=len(downloads), desc="Downloading materials") as pbar:
+ for future in as_completed(futures):
+ material = futures[future]
+ try:
+ if future.result():
+ successful += 1
+ else:
+ failed += 1
+ except Exception as e:
+ console_logger.error(f"Error downloading {material.asset_id}: {e}")
+ failed += 1
+ pbar.update(1)
+
+ return successful, failed
+
+
+def format_size(size_bytes: int) -> str:
+ """Format byte size as human-readable string."""
+ for unit in ["B", "KB", "MB", "GB", "TB"]:
+ if size_bytes < 1024:
+ return f"{size_bytes:.1f} {unit}"
+ size_bytes /= 1024
+ return f"{size_bytes:.1f} PB"
+
+
+def main() -> int:
+ """Main entry point."""
+ parser = argparse.ArgumentParser(
+ description="Download PBR materials from AmbientCG",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog=__doc__,
+ )
+ parser.add_argument(
+ "--resolution",
+ "-r",
+ type=str,
+ default="2K",
+ choices=["1K", "2K", "4K", "8K", "12K", "16K"],
+ help="Resolution to download (default: 2K)",
+ )
+ parser.add_argument(
+ "--format",
+ "-f",
+ type=str,
+ default="PNG",
+ choices=["PNG", "JPG"],
+ help="File format (default: PNG)",
+ )
+ parser.add_argument(
+ "--output",
+ "-o",
+ type=Path,
+ default=DEFAULT_MATERIALS_DIR,
+ help="Output directory (default: <project>/data/materials)",
+ )
+ parser.add_argument(
+ "--limit",
+ "-l",
+ type=int,
+ default=0,
+ help="Max materials to download, 0 for all (default: 0)",
+ )
+ parser.add_argument(
+ "--concurrent",
+ "-c",
+ type=int,
+ default=4,
+ help="Concurrent downloads (default: 4)",
+ )
+ parser.add_argument(
+ "--no-extract",
+ action="store_true",
+ help="Keep ZIP files without extracting",
+ )
+ parser.add_argument(
+ "--dry-run",
+ action="store_true",
+ help="List materials without downloading",
+ )
+
+ args = parser.parse_args()
+
+ # Set up logging.
+ logging.basicConfig(
+ level=logging.INFO,
+ format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
+ )
+
+ # Fetch all materials.
+ session = requests.Session()
+ assets = fetch_all_materials(session=session)
+
+ # Filter to matching downloads.
+ downloads = filter_downloads(
+ assets=assets, resolution=args.resolution, file_format=args.format
+ )
+
+ if not downloads:
+ console_logger.error(
+ f"No materials found with {args.resolution}-{args.format} format"
+ )
+ return 1
+
+ # Apply limit.
+ if args.limit > 0:
+ downloads = downloads[: args.limit]
+
+ # Calculate total size.
+ total_size = sum(d.size_bytes for d in downloads)
+ console_logger.info(
+ f"Found {len(downloads)} materials "
+ f"({format_size(total_size)} total) matching {args.resolution}-{args.format}"
+ )
+
+ # Dry run - just list materials.
+ if args.dry_run:
+ console_logger.info("Dry run - listing materials:")
+ for d in downloads[:20]: # Show first 20.
+ console_logger.info(f" {d.asset_id}: {format_size(d.size_bytes)}")
+ if len(downloads) > 20:
+ console_logger.info(f" ... and {len(downloads) - 20} more")
+ return 0
+
+ # Download materials.
+ console_logger.info(f"Downloading to {args.output.resolve()}")
+ successful, failed = download_all_materials(
+ downloads=downloads,
+ output_dir=args.output,
+ concurrent=args.concurrent,
+ extract=not args.no_extract,
+ )
+
+ console_logger.info(f"Download complete: {successful} successful, {failed} failed")
+ return 0 if failed == 0 else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
\ No newline at end of file
diff --git a/scripts/install_sam3.sh b/scripts/install_sam3.sh
new file mode 100755
index 0000000..7b2117a
--- /dev/null
+++ b/scripts/install_sam3.sh
@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+SAM3_DIR="$REPO_ROOT/sam3"
+
+# pip package index used for PyPI dependency resolution; override it (e.g. a
+# mirror or an internal proxy) before running this script. The default keeps
+# the original PyPI behavior. SCENEMOSAIC_TORCH_INDEX serves the same purpose
+# for the CUDA-specific PyTorch wheel index.
+export SCENEMOSAIC_PIP_INDEX="${SCENEMOSAIC_PIP_INDEX:-https://pypi.org/simple}"
+export SCENEMOSAIC_TORCH_INDEX="${SCENEMOSAIC_TORCH_INDEX:-https://download.pytorch.org/whl/cu126}"
+
+ENV_NAME="$(printenv SAM3_ENV_NAME || true)"
+ENV_PREFIX="$(printenv SAM3_ENV_PREFIX || true)"
+if [[ -z "$ENV_NAME" && -z "$ENV_PREFIX" ]]; then
+ ENV_NAME="sam3"
+elif [[ -n "$ENV_NAME" && -n "$ENV_PREFIX" ]]; then
+ echo "Set only one of SAM3_ENV_NAME or SAM3_ENV_PREFIX" >&2
+ exit 1
+fi
+
+if command -v mamba >/dev/null 2>&1; then
+ ENV_MANAGER=mamba
+elif command -v conda >/dev/null 2>&1; then
+ ENV_MANAGER=conda
+else
+ echo "mamba or conda is required" >&2
+ exit 1
+fi
+
+if [[ ! -f "$SAM3_DIR/pyproject.toml" ]]; then
+ echo "SAM3 source not found at $SAM3_DIR" >&2
+ exit 1
+fi
+
+if [[ -n "$ENV_PREFIX" ]]; then
+ ENV_LABEL="$ENV_PREFIX"
+ if [[ -x "$ENV_PREFIX/bin/python" ]]; then
+ echo "Using existing environment prefix: $ENV_PREFIX"
+ else
+ "$ENV_MANAGER" create --prefix "$ENV_PREFIX" python=3.12 --yes
+ fi
+else
+ ENV_LABEL="$ENV_NAME"
+ if "$ENV_MANAGER" env list \
+ | awk 'NF && $1 !~ /^#/ {print $1}' \
+ | grep -Fxq "$ENV_NAME"; then
+ echo "Using existing environment: $ENV_NAME"
+ else
+ "$ENV_MANAGER" create --name "$ENV_NAME" python=3.12 --yes
+ fi
+fi
+
+run_in_env() {
+ if [[ -n "$ENV_PREFIX" ]]; then
+ "$ENV_MANAGER" run --prefix "$ENV_PREFIX" "$@"
+ else
+ "$ENV_MANAGER" run --name "$ENV_NAME" "$@"
+ fi
+}
+
+run_in_env \
+ python -m pip install --upgrade pip \
+ --index-url "$SCENEMOSAIC_PIP_INDEX"
+
+if run_in_env \
+ python -c 'import torch; assert tuple(map(int, torch.__version__.split("+")[0].split(".")[:2])) >= (2, 7); assert torch.version.cuda and tuple(map(int, torch.version.cuda.split(".")[:2])) >= (12, 6)' \
+ >/dev/null 2>&1; then
+ echo "Using compatible PyTorch already installed in $ENV_LABEL"
+else
+ run_in_env \
+ python -m pip install \
+ --resume-retries 20 \
+ torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 \
+ --index-url "$SCENEMOSAIC_TORCH_INDEX"
+fi
+
+run_in_env \
+ python -m pip install --editable "$SAM3_DIR"
+
+run_in_env \
+ python -m pip install \
+ --requirement "$REPO_ROOT/requirements-perception.txt" \
+ --index-url "$SCENEMOSAIC_PIP_INDEX"
+
+for MODULE_NAME in \
+ PIL cv2 scipy transformers openai huggingface_hub typing_extensions; do
+ run_in_env python -c "import $MODULE_NAME"
+done
+run_in_env python -m pip check
+echo "SceneMosaic perception dependencies verified"
+
+echo "SAM3 environment ready: $ENV_LABEL"
+run_in_env python -c 'import sys; print(sys.executable)'
diff --git a/scripts/install_scenemosaic.sh b/scripts/install_scenemosaic.sh
new file mode 100755
index 0000000..c7af2c8
--- /dev/null
+++ b/scripts/install_scenemosaic.sh
@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+SAM3D_DIR="$REPO_ROOT/sam-3d-objects"
+
+# pip package indexes used for dependency resolution; override them (e.g. a
+# mirror or an internal proxy) before running this script. The defaults keep
+# the original upstream behavior.
+export SCENEMOSAIC_PIP_INDEX="${SCENEMOSAIC_PIP_INDEX:-https://pypi.org/simple}"
+export SCENEMOSAIC_PIP_EXTRA_INDEX="${SCENEMOSAIC_PIP_EXTRA_INDEX:-https://pypi.ngc.nvidia.com https://download.pytorch.org/whl/cu121}"
+export SCENEMOSAIC_KAOLIN_FIND_LINKS="${SCENEMOSAIC_KAOLIN_FIND_LINKS:-https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.5.1_cu121.html}"
+
+ENV_NAME="$(printenv SCENEMOSAIC_ENV_NAME || true)"
+ENV_PREFIX="$(printenv SCENEMOSAIC_ENV_PREFIX || true)"
+if [[ -z "$ENV_NAME" && -z "$ENV_PREFIX" ]]; then
+ ENV_NAME="scenemosaic"
+elif [[ -n "$ENV_NAME" && -n "$ENV_PREFIX" ]]; then
+ echo "Set only one of SCENEMOSAIC_ENV_NAME or SCENEMOSAIC_ENV_PREFIX" >&2
+ exit 1
+fi
+
+if command -v mamba >/dev/null 2>&1; then
+ ENV_MANAGER=mamba
+elif command -v conda >/dev/null 2>&1; then
+ ENV_MANAGER=conda
+else
+ echo "mamba or conda is required" >&2
+ exit 1
+fi
+
+if [[ ! -f "$SAM3D_DIR/environments/default.yml" ]]; then
+ echo "SAM 3D Objects source not found at $SAM3D_DIR" >&2
+ exit 1
+fi
+
+if [[ -n "$ENV_PREFIX" ]]; then
+ ENV_LABEL="$ENV_PREFIX"
+ if [[ -x "$ENV_PREFIX/bin/python" ]]; then
+ echo "Using existing environment prefix: $ENV_PREFIX"
+ else
+ "$ENV_MANAGER" env create --prefix "$ENV_PREFIX" --file "$SAM3D_DIR/environments/default.yml"
+ fi
+else
+ ENV_LABEL="$ENV_NAME"
+ if "$ENV_MANAGER" env list | awk 'NF && $1 !~ /^#/ {print $1}' | grep -Fxq "$ENV_NAME"; then
+ echo "Using existing environment: $ENV_NAME"
+ else
+ "$ENV_MANAGER" env create --name "$ENV_NAME" --file "$SAM3D_DIR/environments/default.yml"
+ fi
+fi
+
+run_in_env() {
+ if [[ -n "$ENV_PREFIX" ]]; then
+ "$ENV_MANAGER" run --prefix "$ENV_PREFIX" "$@"
+ else
+ "$ENV_MANAGER" run --name "$ENV_NAME" "$@"
+ fi
+}
+
+PIP_EXTRA_INDEX_URL="$SCENEMOSAIC_PIP_EXTRA_INDEX" run_in_env python -m pip install --editable "$SAM3D_DIR[dev]"
+
+PIP_EXTRA_INDEX_URL="$SCENEMOSAIC_PIP_EXTRA_INDEX" run_in_env python -m pip install --editable "$SAM3D_DIR[p3d]"
+
+PIP_EXTRA_INDEX_URL="$SCENEMOSAIC_PIP_EXTRA_INDEX" PIP_FIND_LINKS="$SCENEMOSAIC_KAOLIN_FIND_LINKS" run_in_env python -m pip install --editable "$SAM3D_DIR[inference]"
+
+run_in_env python "$SAM3D_DIR/patching/hydra"
+
+run_in_env python -m pip install --requirement "$REPO_ROOT/requirements.txt" --index-url "$SCENEMOSAIC_PIP_INDEX"
+
+for MODULE_NAME in OpenGL coacd fast_simplification open_clip openai pybullet sapien shapely; do
+ run_in_env python -c "import $MODULE_NAME"
+done
+echo "SceneMosaic dependencies verified"
+
+if ! PYOPENGL_PLATFORM=egl run_in_env python -c 'import ctypes, ctypes.util; library = ctypes.util.find_library("EGL"); assert library, "system dynamic linker cannot find EGL"; ctypes.CDLL(library); import pyrender'; then
+ echo "SceneMosaic requires a system EGL runtime for headless rendering." >&2
+ echo "On Ubuntu/Debian install: sudo apt-get install libegl1 libglvnd0 libegl-mesa0" >&2
+ exit 1
+fi
+echo "SceneMosaic EGL runtime verified"
+
+echo "SceneMosaic environment ready: $ENV_LABEL"
+run_in_env python -c 'import sys; print(sys.executable)'
diff --git a/vlm_agent/__init__.py b/vlm_agent/__init__.py
new file mode 100644
index 0000000..a8887bd
--- /dev/null
+++ b/vlm_agent/__init__.py
@@ -0,0 +1 @@
+"""SceneMosaic single-image to simulation-ready scene pipeline."""
diff --git a/vlm_agent/artifacts.py b/vlm_agent/artifacts.py
new file mode 100644
index 0000000..559c238
--- /dev/null
+++ b/vlm_agent/artifacts.py
@@ -0,0 +1,269 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+import shutil
+import uuid
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from .config import RuntimeConfig
+from .run_paths import SEMANTIC_STAGES, ScenePathError, ScenePaths
+from .state import RunState
+
+
+class RunContractError(ValueError):
+ """Raised when a formal scene run does not satisfy the path contract."""
+
+
+@dataclass(frozen=True)
+class RunContract:
+ """Resolved, read-only paths from a formal ``input/scene_config.json``."""
+
+ scene_name: str
+ scene_config_path: Path
+ run_root: Path
+ input_image: Path
+ config: dict[str, Any]
+
+ @classmethod
+ def from_scene_config(cls, scene_config_path: Path) -> "RunContract":
+ config_path = Path(scene_config_path).expanduser().resolve()
+ if not config_path.is_file():
+ raise RunContractError(f"scene_config is not a file: {config_path}")
+
+ try:
+ config = json.loads(config_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RunContractError(f"cannot parse scene_config: {config_path}") from exc
+ if not isinstance(config, dict):
+ raise RunContractError("scene_config must contain a JSON object")
+
+ scene_name = str(config.get("scene_name") or "").strip()
+ if not scene_name:
+ raise RunContractError("scene_config.scene_name must not be empty")
+
+ raw_run_root = config.get("run_root")
+ if not isinstance(raw_run_root, str) or not raw_run_root.strip():
+ raise RunContractError("scene_config.run_root must be a non-empty path")
+ run_root = Path(raw_run_root).expanduser().resolve()
+ expected_config = run_root / "input" / "scene_config.json"
+ if config_path != expected_config:
+ raise RunContractError(
+ f"scene_config must be the formal run config: expected {expected_config}, got {config_path}"
+ )
+
+ raw_input_image = config.get("input_image")
+ if not isinstance(raw_input_image, str) or not raw_input_image.strip():
+ raise RunContractError("scene_config.input_image must be a non-empty path")
+ input_image = Path(raw_input_image).expanduser().resolve()
+ expected_image = run_root / "input" / "image.png"
+ if input_image != expected_image:
+ raise RunContractError(
+ f"scene_config.input_image must be the formal input image: expected {expected_image}, got {input_image}"
+ )
+ if not input_image.is_file():
+ raise RunContractError(f"formal input image is missing: {input_image}")
+ try:
+ ScenePaths.from_scene_config(config_path)
+ except ScenePathError as exc:
+ raise RunContractError(str(exc)) from exc
+
+ return cls(
+ scene_name=scene_name,
+ scene_config_path=config_path,
+ run_root=run_root,
+ input_image=input_image,
+ config=config,
+ )
+
+ @property
+ def stage03_dir(self) -> Path:
+ return self.output_paths.perception_dir
+
+ @property
+ def stage04_dir(self) -> Path:
+ return self.output_paths.sam3d_input_dir
+
+ @property
+ def stage05_dir(self) -> Path:
+ return self.output_paths.sam3d_output_dir
+
+ @property
+ def sam3d_attempts_dir(self) -> Path:
+ return self.output_paths.sam3d_attempts_dir
+
+ @property
+ def relations_dir(self) -> Path:
+ return self.output_paths.relations_dir
+
+ @property
+ def relation_candidates_dir(self) -> Path:
+ return self.output_paths.relation_candidates_dir
+
+ @property
+ def evolution_dir(self) -> Path:
+ return self.output_paths.evolution_dir
+
+ @property
+ def output_paths(self) -> ScenePaths:
+ try:
+ return ScenePaths.from_scene_config(self.scene_config_path)
+ except ScenePathError as exc:
+ raise RunContractError(str(exc)) from exc
+
+
+SCENE_NAME_RE = re.compile(r"^[A-Za-z0-9_-]+$")
+STAGES = dict(SEMANTIC_STAGES)
+COORDINATE_SYSTEM = {
+ "name": "OpenGL_glTF_global",
+ "up_axis": "+Y",
+ "down_axis": "-Y",
+ "front_axis": "-Z",
+ "back_axis": "+Z",
+ "left_axis": "-X",
+ "right_axis": "+X",
+ "ground_plane": "XZ",
+}
+
+
+class SceneRunError(RuntimeError):
+ """Raised when a formal scene run cannot be initialized safely."""
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _is_within(path: Path, root: Path) -> bool:
+ try:
+ path.relative_to(root)
+ return True
+ except ValueError:
+ return False
+
+
+@dataclass(frozen=True)
+class SceneRun:
+ contract: RunContract
+ state: RunState
+ input_sha256: str
+
+
+def initialize_scene_run(
+ runtime: RuntimeConfig,
+ *,
+ scene_name: str,
+ input_image: Path,
+ run_root: Path | None = None,
+) -> SceneRun:
+ name = str(scene_name or "").strip()
+ if not SCENE_NAME_RE.fullmatch(name):
+ raise SceneRunError(
+ "scene_name may contain only letters, numbers, underscores and hyphens"
+ )
+ source = Path(input_image).expanduser().resolve()
+ if not source.is_file() or source.is_symlink():
+ raise SceneRunError(f"input_image must be a regular file: {source}")
+
+ workspace_runs = (runtime.workspace_root / "runs").resolve()
+ final_root = (
+ Path(run_root).expanduser().resolve()
+ if run_root is not None
+ else workspace_runs / name
+ )
+ if not _is_within(final_root, workspace_runs):
+ raise SceneRunError(
+ f"run_root must stay below the configured workspace runs root: {workspace_runs}"
+ )
+ if final_root.is_symlink() or (final_root.exists() and not final_root.is_dir()):
+ raise SceneRunError(f"run_root is not a regular directory: {final_root}")
+ if final_root.exists() and any(final_root.iterdir()):
+ raise SceneRunError(f"run_root already exists and is non-empty: {final_root}")
+
+ final_root.parent.mkdir(parents=True, exist_ok=True)
+ temporary = final_root.parent / f".{final_root.name}.init-{uuid.uuid4().hex}.tmp"
+ if temporary.exists():
+ raise SceneRunError(f"temporary initialization path already exists: {temporary}")
+
+ input_hash = _sha256(source)
+ try:
+ (temporary / "input").mkdir(parents=True, exist_ok=False)
+ (temporary / ".pipeline.lock").touch(exist_ok=False)
+ copied_image = temporary / "input" / "image.png"
+ shutil.copy2(source, copied_image)
+
+ final_image = final_root / "input" / "image.png"
+ config: dict[str, Any] = {
+ "schema_version": "scenemosaic_scene_config",
+ "scene_name": name,
+ "run_root": str(final_root),
+ "input_image": str(final_image),
+ "source_input_image": str(source),
+ "source_input_sha256": input_hash,
+ "workspace_root": str(runtime.workspace_root),
+ "vlm_model": runtime.vlm_model,
+ "object_id_base": 1,
+ "object_id_format": "object_{index:03d}",
+ "wall_id_format": "wall_{index:03d}",
+ "coordinate_system": COORDINATE_SYSTEM,
+ "conda_envs": {
+ "sam3": str(runtime.perception_python.parent.parent),
+ "sam3d": str(runtime.sam3d_python.parent.parent),
+ },
+ "stage_envs": {
+ "perception": "sam3",
+ "sam3d_input": "sam3",
+ "sam3d_output": "sam3d",
+ "relations": "relation",
+ "evolution": "scenemosaic",
+ },
+ "stages": STAGES,
+ "keep_virtual_surfaces": ["virtual_floor", "virtual_ceiling"],
+ "created_at_utc": datetime.now(timezone.utc).isoformat(),
+ }
+ config_path = temporary / "input" / "scene_config.json"
+ config_path.write_text(
+ json.dumps(config, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ if final_root.exists():
+ final_root.rmdir()
+ os.replace(temporary, final_root)
+ except Exception:
+ if temporary.exists():
+ shutil.rmtree(temporary)
+ raise
+
+ contract = RunContract.from_scene_config(final_root / "input" / "scene_config.json")
+ state = RunState.create(final_root, scene_name=name)
+ state.record_stage(
+ "input",
+ status="published",
+ successful_state="initialized",
+ facts={
+ "scene_config": str(contract.scene_config_path),
+ "input_image": str(contract.input_image),
+ "input_sha256": input_hash,
+ },
+ )
+ return SceneRun(contract=contract, state=state, input_sha256=input_hash)
+
+
+def load_scene_run(scene_root: Path) -> SceneRun:
+ root = Path(scene_root).expanduser().resolve()
+ contract = RunContract.from_scene_config(root / "input" / "scene_config.json")
+ state = RunState.load(root)
+ input_hash = _sha256(contract.input_image)
+ expected = contract.config.get("source_input_sha256")
+ if expected and input_hash != expected:
+ raise SceneRunError("formal input image hash no longer matches scene_config")
+ return SceneRun(contract=contract, state=state, input_sha256=input_hash)
diff --git a/vlm_agent/cli.py b/vlm_agent/cli.py
new file mode 100644
index 0000000..9db80fa
--- /dev/null
+++ b/vlm_agent/cli.py
@@ -0,0 +1,149 @@
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import sys
+from pathlib import Path
+from typing import Sequence
+
+from .pipeline import (
+ FullPipelineError,
+ FullRunOptions,
+ FullSceneOrchestrator,
+)
+from .config import RuntimeConfig, RuntimeConfigError
+
+
+SECRET_ENV_RE = re.compile(
+ r"(?:API_KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL|AUTHORIZATION)", re.I
+)
+
+
+def build_arg_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(
+ description=(
+ "SceneMosaic pipeline: image -> perception -> SAM3D -> "
+ "relation structuring -> scene evolution"
+ )
+ )
+ parser.add_argument(
+ "--runtime-config",
+ type=Path,
+ default=None,
+ help=(
+ "Secret-free deployment profile; defaults to "
+ "SCENEMOSAIC_RUNTIME_CONFIG or configs/runtime.local.json"
+ ),
+ )
+ commands = parser.add_subparsers(dest="command", required=True)
+
+ verify = commands.add_parser(
+ "verify", help="Read-only release, path and renderer verification"
+ )
+ verify.add_argument(
+ "--require-gpu",
+ action="store_true",
+ help="Also verify that an NVIDIA GPU is visible; no model is loaded",
+ )
+
+ status = commands.add_parser(
+ "status", help="Read-only status and immutable fact validation"
+ )
+ status.add_argument("--scene-root", type=Path, required=True)
+
+ run = commands.add_parser(
+ "run", help="Start from one image or resume a validated scene run"
+ )
+ source = run.add_mutually_exclusive_group(required=True)
+ source.add_argument("--input-image", type=Path)
+ source.add_argument("--scene-root", type=Path)
+ run.add_argument(
+ "--scene-name",
+ help="Required with --input-image; ignored values are never inferred",
+ )
+ run.add_argument(
+ "--run-root",
+ type=Path,
+ help="Optional new run root below configured workspace/runs",
+ )
+ run.add_argument(
+ "--perception-reuse-local-pre-pass-b-cache-root",
+ type=Path,
+ help=(
+ "Reuse one validated failed perception attempt below the trusted "
+ "attempts root without reissuing Pass A or SAM3-A"
+ ),
+ )
+ run.add_argument(
+ "--perception-pass-b-reasoning-effort",
+ choices=("none", "low", "medium", "high", "xhigh"),
+ default="none",
+ )
+ return parser
+
+
+def _redact(value: str) -> str:
+ result = str(value)
+ for name, secret in os.environ.items():
+ if SECRET_ENV_RE.search(name) and len(secret) >= 4:
+ result = result.replace(secret, "<redacted>")
+ return result
+
+
+def _print(payload: object) -> None:
+ print(json.dumps(payload, ensure_ascii=False, indent=2))
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ parser = build_arg_parser()
+ raw_argv = list(sys.argv[1:] if argv is None else argv)
+ args = parser.parse_args(raw_argv)
+ try:
+ runtime = RuntimeConfig.load(args.runtime_config)
+ orchestrator = FullSceneOrchestrator(runtime)
+ if args.command == "verify":
+ report = orchestrator.verify(require_gpu=args.require_gpu)
+ _print(report)
+ return 0 if report.get("valid") else 4
+ if args.command == "status":
+ report = orchestrator.status(args.scene_root)
+ _print(report)
+ return 0 if report.get("facts_valid") else 4
+ if args.input_image is not None and not args.scene_name:
+ raise FullPipelineError(
+ "--input-image requires an explicit --scene-name"
+ )
+ if args.scene_root is not None and args.scene_name is not None:
+ raise FullPipelineError(
+ "--scene-name is valid only with --input-image"
+ )
+ report = orchestrator.run(
+ FullRunOptions(
+ input_image=args.input_image,
+ scene_name=args.scene_name,
+ scene_root=args.scene_root,
+ run_root=args.run_root,
+ perception_reuse_local_pre_pass_b_cache_root=(
+ args.perception_reuse_local_pre_pass_b_cache_root
+ ),
+ perception_pass_b_reasoning_effort=(
+ args.perception_pass_b_reasoning_effort
+ ),
+ ),
+ sanitized_argv=raw_argv,
+ )
+ _print(report)
+ return 0
+ except RuntimeConfigError as exc:
+ parser.exit(2, f"error: {_redact(str(exc))}\n")
+ except FullPipelineError as exc:
+ parser.exit(exc.exit_code, f"error: {_redact(str(exc))}\n")
+ except Exception as exc:
+ parser.exit(4, f"error: {_redact(str(exc))}\n")
+ return 4
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/vlm_agent/config.py b/vlm_agent/config.py
new file mode 100644
index 0000000..ce239e1
--- /dev/null
+++ b/vlm_agent/config.py
@@ -0,0 +1,716 @@
+from __future__ import annotations
+
+import ast
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Mapping
+
+from .reconstruction.sam3_runtime import (
+ verify_perception_dependencies,
+ verify_sam3_runtime_provider,
+)
+
+
+RUNTIME_CONFIG_ENV = "SCENEMOSAIC_RUNTIME_CONFIG"
+DEFAULT_VLM_MODEL = "gpt-5.5"
+ENV_NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
+SENSITIVE_ENV_NAME_RE = re.compile(
+ r"(?:KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL|AUTHORIZATION|BASE_URL)", re.I
+)
+RENDERER_FUNCTIONS: Mapping[str, tuple[str, ...]] = {
+ "compose_scene": ("layout_dir", "unposed_dir"),
+ "denormalize_point": ("point_normalized", "transform_info"),
+ "generate_2d_grid_on_plane": ("posed_meshes", "camera_direction", "grid_size"),
+ "render_scene_with_grid": ("meshes", "camera_direction", "radius", "image_size"),
+ "save_grid_mapping": ("mapping", "output_path", "transform_info"),
+ "resolve_orthographic_camera_frame": ("meshes", "camera_direction", "radius"),
+ "build_camera_pose": ("camera_direction", "radius"),
+ "get_camera_matrix": ("yfov_deg", "image_size"),
+ "project_3d_to_2d": ("point_3d", "camera_pose", "camera_matrix", "image_size"),
+}
+
+
+class RuntimeConfigError(ValueError):
+ """Raised when the unified runtime profile is unsafe or incomplete."""
+
+
+def _read_json(path: Path, label: str) -> dict[str, Any]:
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RuntimeConfigError(f"cannot parse {label}: {path}") from exc
+ if not isinstance(payload, dict):
+ raise RuntimeConfigError(f"{label} must contain a JSON object: {path}")
+ return payload
+
+
+def _path(value: Any, *, base: Path, label: str) -> Path:
+ if not isinstance(value, str) or not value.strip():
+ raise RuntimeConfigError(f"{label} must be a non-empty path")
+ candidate = Path(value).expanduser()
+ if not candidate.is_absolute():
+ candidate = base / candidate
+ return candidate.resolve()
+
+
+def _env_name(value: Any, *, label: str) -> str:
+ if not isinstance(value, str) or not ENV_NAME_RE.fullmatch(value):
+ raise RuntimeConfigError(f"{label} must be an environment variable name")
+ return value
+
+
+@dataclass(frozen=True)
+class RuntimeConfig:
+ """One secret-free deployment profile for the complete SceneMosaic pipeline."""
+
+ project_root: Path
+ profile_path: Path
+ workspace_root: Path
+ perception_attempts_root: Path
+ perception_python: Path
+ perception_model_dir: Path
+ sam3d_python: Path
+ sam3d_checkpoint: Path
+ dino_hub_cache: Path
+ relation_precompute_python: Path
+ relation_runner_python: Path
+ scenemosaic_python: Path
+ perception_api_key_env: str = "OPENAI_API_KEY"
+ perception_api_base_env: str = "OPENAI_BASE_URL"
+ relation_api_key_env: str = "OPENAI_API_KEY"
+ relation_api_base_env: str = "OPENAI_BASE_URL"
+ scenemosaic_api_key_env: str = "OPENAI_API_KEY"
+ scenemosaic_api_base_env: str = "OPENAI_BASE_URL"
+ vlm_model: str = DEFAULT_VLM_MODEL
+
+ @classmethod
+ def load(
+ cls,
+ profile_path: Path | str | None = None,
+ *,
+ environ: Mapping[str, str] | None = None,
+ project_root: Path | None = None,
+ ) -> "RuntimeConfig":
+ env = os.environ if environ is None else environ
+ root = (
+ Path(project_root).expanduser().resolve()
+ if project_root is not None
+ else Path(__file__).resolve().parents[1]
+ )
+ selected = profile_path or env.get(RUNTIME_CONFIG_ENV)
+ profile = (
+ Path(selected).expanduser().resolve()
+ if selected
+ else root / "configs" / "runtime.local.json"
+ )
+ payload = _read_json(profile, "runtime profile")
+
+ forbidden = {
+ "api_key",
+ "perception_api_key",
+ "relation_api_key",
+ "scenemosaic_api_key",
+ "authorization",
+ }
+ leaked = sorted(forbidden.intersection(payload))
+ if leaked:
+ raise RuntimeConfigError(
+ "runtime profile may contain environment variable names only; "
+ f"forbidden secret fields: {leaked}"
+ )
+
+ return cls(
+ project_root=root,
+ profile_path=profile,
+ workspace_root=_path(
+ payload.get("workspace_root"),
+ base=profile.parent,
+ label="workspace_root",
+ ),
+ perception_attempts_root=_path(
+ payload.get("perception_attempts_root"),
+ base=profile.parent,
+ label="perception_attempts_root",
+ ),
+ perception_python=_path(
+ payload.get("perception_python"),
+ base=profile.parent,
+ label="perception_python",
+ ),
+ perception_model_dir=_path(
+ payload.get("perception_model_dir"),
+ base=profile.parent,
+ label="perception_model_dir",
+ ),
+ sam3d_python=_path(
+ payload.get("sam3d_python"), base=profile.parent, label="sam3d_python"
+ ),
+ sam3d_checkpoint=_path(
+ payload.get("sam3d_checkpoint"),
+ base=profile.parent,
+ label="sam3d_checkpoint",
+ ),
+ dino_hub_cache=_path(
+ payload.get("dino_hub_cache"),
+ base=profile.parent,
+ label="dino_hub_cache",
+ ),
+ relation_precompute_python=_path(
+ payload.get("relation_precompute_python"),
+ base=profile.parent,
+ label="relation_precompute_python",
+ ),
+ relation_runner_python=_path(
+ payload.get("relation_runner_python"),
+ base=profile.parent,
+ label="relation_runner_python",
+ ),
+ scenemosaic_python=_path(
+ payload.get("scenemosaic_python", sys.executable),
+ base=profile.parent,
+ label="scenemosaic_python",
+ ),
+ perception_api_key_env=_env_name(
+ payload.get("perception_api_key_env", "OPENAI_API_KEY"),
+ label="perception_api_key_env",
+ ),
+ perception_api_base_env=_env_name(
+ payload.get("perception_api_base_env", "OPENAI_BASE_URL"),
+ label="perception_api_base_env",
+ ),
+ relation_api_key_env=_env_name(
+ payload.get("relation_api_key_env", "OPENAI_API_KEY"),
+ label="relation_api_key_env",
+ ),
+ relation_api_base_env=_env_name(
+ payload.get("relation_api_base_env", "OPENAI_BASE_URL"),
+ label="relation_api_base_env",
+ ),
+ scenemosaic_api_key_env=_env_name(
+ payload.get("scenemosaic_api_key_env", "OPENAI_API_KEY"),
+ label="scenemosaic_api_key_env",
+ ),
+ scenemosaic_api_base_env=_env_name(
+ payload.get("scenemosaic_api_base_env", "OPENAI_BASE_URL"),
+ label="scenemosaic_api_base_env",
+ ),
+ vlm_model=str(payload.get("vlm_model") or DEFAULT_VLM_MODEL),
+ )
+
+ @property
+ def agent_root(self) -> Path:
+ return self.project_root / "vlm_agent"
+
+ @property
+ def reconstruction_root(self) -> Path:
+ return self.agent_root / "reconstruction"
+
+ @property
+ def perception_root(self) -> Path:
+ return self.reconstruction_root / "perception"
+
+ @property
+ def perception_entrypoint(self) -> Path:
+ return self.perception_root / "pipeline" / "runner.py"
+
+ @property
+ def relation_package_root(self) -> Path:
+ return self.reconstruction_root / "relations"
+
+ @property
+ def relation_wrapper(self) -> Path:
+ return (
+ self.relation_package_root
+ / "tools"
+ / "benchmark_relation_async_planning.py"
+ )
+
+ @property
+ def sam3_source_root(self) -> Path:
+ return self.project_root / "sam3"
+
+ @property
+ def sam3d_source_root(self) -> Path:
+ return self.project_root / "sam-3d-objects"
+
+ @property
+ def renderer_path(self) -> Path:
+ return self.agent_root / "evolution" / "tools" / "render_scene_from_layout.py"
+
+ def child_environment(
+ self,
+ stage: str,
+ *,
+ environ: Mapping[str, str] | None = None,
+ ) -> dict[str, str]:
+ """Build a child environment without ever serializing secret values."""
+
+ source = os.environ if environ is None else environ
+ if stage == "perception":
+ key_name = self.perception_api_key_env
+ base_name = self.perception_api_base_env
+ elif stage == "relation":
+ key_name = self.relation_api_key_env
+ base_name = self.relation_api_base_env
+ elif stage == "scenemosaic":
+ key_name = self.scenemosaic_api_key_env
+ base_name = self.scenemosaic_api_base_env
+ else:
+ raise RuntimeConfigError(f"unknown API stage: {stage}")
+ child = {
+ name: value
+ for name, value in source.items()
+ if not SENSITIVE_ENV_NAME_RE.search(name)
+ }
+ key_value = source.get(key_name)
+ base_value = source.get(base_name)
+ if key_value:
+ child[key_name] = key_value
+ child["OPENAI_API_KEY"] = key_value
+ if base_value:
+ child[base_name] = base_value
+ child["OPEN_BASE_URL"] = base_value
+ existing_pythonpath = child.get("PYTHONPATH")
+ python_paths: list[str] = []
+ if stage == "perception":
+ python_paths.append(str(self.sam3_source_root))
+ child.update(
+ {
+ "SCENEMOSAIC_RUNTIME_ROOT": str(self.workspace_root),
+ "SCENEMOSAIC_PERCEPTION_ATTEMPTS_ROOT": str(
+ self.perception_attempts_root
+ ),
+ "SCENEMOSAIC_PERCEPTION_MODEL_DIR": str(
+ self.perception_model_dir
+ ),
+ "SCENEMOSAIC_API_METERING_ROOT": str(
+ self.workspace_root / "api_metering"
+ ),
+ "SCENEMOSAIC_SAM3_ENV": (
+ self.perception_python.parent.parent.name or "sam3"
+ ),
+ }
+ )
+ elif stage == "relation":
+ # Relation subprocesses intentionally use their standalone
+ # relations.* imports without importing SceneMosaic's heavy
+ # package initializer. The reconstruction package root supplies
+ # that stable top-level namespace in the isolated environment.
+ python_paths.append(str(self.reconstruction_root))
+ if existing_pythonpath:
+ python_paths.append(existing_pythonpath)
+ if python_paths:
+ child["PYTHONPATH"] = os.pathsep.join(python_paths)
+ child["PYTHONDONTWRITEBYTECODE"] = "1"
+ child["PYTHONNOUSERSITE"] = "1"
+ return child
+
+ def has_api_credentials(
+ self,
+ stage: str,
+ *,
+ environ: Mapping[str, str] | None = None,
+ ) -> bool:
+ source = os.environ if environ is None else environ
+ names = {
+ "perception": self.perception_api_key_env,
+ "relation": self.relation_api_key_env,
+ "scenemosaic": self.scenemosaic_api_key_env,
+ }
+ try:
+ return bool(source.get(names[stage]))
+ except KeyError as exc:
+ raise RuntimeConfigError(f"unknown API stage: {stage}") from exc
+
+ def has_api_base(
+ self,
+ stage: str,
+ *,
+ environ: Mapping[str, str] | None = None,
+ ) -> bool:
+ source = os.environ if environ is None else environ
+ names = {
+ "perception": self.perception_api_base_env,
+ "relation": self.relation_api_base_env,
+ "scenemosaic": self.scenemosaic_api_base_env,
+ }
+ try:
+ value = source.get(names[stage])
+ except KeyError as exc:
+ raise RuntimeConfigError(f"unknown API stage: {stage}") from exc
+ return bool(value and value.strip() and value.strip() != "None")
+
+ @staticmethod
+ def verify_gpu() -> dict[str, Any]:
+ """Verify GPU visibility without importing CUDA or starting a model."""
+
+ base = {
+ "schema_version": "scenemosaic_gpu_preflight",
+ "gpu_used": False,
+ "api_calls": 0,
+ }
+ visible = os.environ.get("CUDA_VISIBLE_DEVICES")
+ if visible is not None and visible.strip().lower() in {
+ "",
+ "-1",
+ "none",
+ "void",
+ }:
+ return {
+ **base,
+ "valid": False,
+ "devices": [],
+ "reason": "CUDA_VISIBLE_DEVICES disables all GPUs",
+ }
+ executable = shutil.which("nvidia-smi")
+ if not executable:
+ return {
+ **base,
+ "valid": False,
+ "devices": [],
+ "reason": "nvidia-smi not found",
+ }
+ try:
+ completed = subprocess.run(
+ [
+ executable,
+ "--query-gpu=index,name,uuid,memory.total",
+ "--format=csv,noheader,nounits",
+ ],
+ capture_output=True,
+ text=True,
+ timeout=10,
+ check=False,
+ )
+ except (OSError, subprocess.SubprocessError) as exc:
+ return {
+ **base,
+ "valid": False,
+ "devices": [],
+ "reason": type(exc).__name__,
+ }
+ devices: list[dict[str, Any]] = []
+ if completed.returncode == 0:
+ for line in completed.stdout.splitlines():
+ fields = [item.strip() for item in line.split(",", 3)]
+ if len(fields) != 4:
+ continue
+ try:
+ memory_mib = int(fields[3])
+ except ValueError:
+ continue
+ devices.append(
+ {
+ "index": fields[0],
+ "name": fields[1],
+ "uuid": fields[2],
+ "memory_total_mib": memory_mib,
+ }
+ )
+ return {
+ **base,
+ "valid": completed.returncode == 0 and bool(devices),
+ "devices": devices,
+ "reason": None if devices else "no visible NVIDIA GPU",
+ }
+
+ def verify(self, *, require_gpu: bool = False) -> dict[str, Any]:
+ checks: list[dict[str, Any]] = []
+
+ def record(
+ name: str,
+ valid: bool,
+ detail: Any = None,
+ *,
+ required: bool = True,
+ ) -> None:
+ checks.append(
+ {
+ "name": name,
+ "valid": bool(valid),
+ "required": required,
+ "detail": detail,
+ }
+ )
+
+ record("project_root", self.project_root.is_dir(), str(self.project_root))
+ # The VLM model name is intentionally not validated here: the profile
+ # accepts any non-empty model id so that OpenAI-compatible gateways
+ # whose id differs from DEFAULT_VLM_MODEL (e.g. casing) can be used.
+ # DEFAULT_VLM_MODEL is still the fallback when the profile omits it.
+ expected_attempts_root = (self.workspace_root / "perception_attempts").resolve()
+ record(
+ "perception_attempts_root_contract",
+ self.perception_attempts_root == expected_attempts_root,
+ {
+ "actual": str(self.perception_attempts_root),
+ "expected": str(expected_attempts_root),
+ },
+ )
+ for name, path, kind in (
+ ("workspace_root", self.workspace_root, "dir"),
+ ("perception_attempts_root", self.perception_attempts_root, "dir"),
+ ("perception_python", self.perception_python, "file"),
+ ("perception_model_dir", self.perception_model_dir, "dir"),
+ ("sam3d_python", self.sam3d_python, "file"),
+ ("sam3d_checkpoint", self.sam3d_checkpoint, "file"),
+ ("dino_hub_cache", self.dino_hub_cache, "dir"),
+ ("relation_precompute_python", self.relation_precompute_python, "file"),
+ ("relation_runner_python", self.relation_runner_python, "file"),
+ ("scenemosaic_python", self.scenemosaic_python, "file"),
+ ("sam3_source_root", self.sam3_source_root, "dir"),
+ ("sam3d_source_root", self.sam3d_source_root, "dir"),
+ ("renderer", self.renderer_path, "file"),
+ ):
+ valid = path.is_dir() if kind == "dir" else path.is_file()
+ record(name, valid, str(path))
+
+ perception = verify_perception_runtime(self.perception_root)
+ record("perception_runtime", perception["valid"], perception)
+ relation = verify_relation_runtime(self.relation_package_root)
+ record("relation_runtime", relation["valid"], relation)
+ sam3_provider = verify_sam3_runtime_provider(
+ self.perception_python,
+ self.sam3_source_root,
+ )
+ record(
+ "sam3_runtime_provider",
+ sam3_provider["valid"],
+ sam3_provider,
+ )
+ perception_dependencies = verify_perception_dependencies(
+ self.perception_python
+ )
+ record(
+ "perception_python_dependencies",
+ perception_dependencies["valid"],
+ perception_dependencies,
+ )
+ sam3d = verify_sam3d_interface(self.sam3d_source_root)
+ record("sam3d_runtime_interface", sam3d["valid"], sam3d)
+ renderer_runtime = verify_headless_renderer_runtime(
+ self.relation_precompute_python
+ )
+ record(
+ "renderer_egl_runtime",
+ renderer_runtime["valid"],
+ renderer_runtime,
+ )
+ renderer = verify_renderer_contract(self.renderer_path)
+ record("renderer_nine_functions", renderer["valid"], renderer)
+
+ if require_gpu:
+ gpu = self.verify_gpu()
+ record("gpu", bool(gpu.get("valid")), gpu)
+
+ return {
+ "schema_version": "scenemosaic_runtime_verification",
+ "profile_path": str(self.profile_path),
+ "project_root": str(self.project_root),
+ "api_calls": 0,
+ "gpu_used": False,
+ "valid": all(item["valid"] for item in checks if item["required"]),
+ "scenemosaic_ready": all(item["valid"] for item in checks),
+ "checks": checks,
+ }
+
+
+def _verify_required_files(
+ package_root: Path,
+ *,
+ component: str,
+ release_id: str,
+ required_files: tuple[str, ...],
+) -> dict[str, Any]:
+ root = Path(package_root).expanduser().resolve()
+ failures: list[dict[str, str]] = []
+ checked = 0
+ for relative in required_files:
+ path = root / relative
+ if not path.is_file() or path.is_symlink():
+ failures.append({"path": relative, "reason": "missing_or_symlink"})
+ else:
+ checked += 1
+ return {
+ "component": component,
+ "release": release_id,
+ "release_id": release_id,
+ "package_root": str(root),
+ "required_file_count": len(required_files),
+ "checked_file_count": checked,
+ "valid": root.is_dir() and not root.is_symlink() and not failures,
+ "failures": failures,
+ }
+
+
+def verify_perception_runtime(package_root: Path) -> dict[str, Any]:
+ return _verify_required_files(
+ package_root,
+ component="perception",
+ release_id="scenemosaic-perception",
+ required_files=(
+ "adoption.py",
+ "stage.py",
+ "inventory/pipeline.py",
+ "pipeline/runner.py",
+ "segmentation/engine.py",
+ "segmentation/runtime.py",
+ ),
+ )
+
+
+def verify_relation_runtime(package_root: Path) -> dict[str, Any]:
+ return _verify_required_files(
+ package_root,
+ component="relations",
+ release_id="scenemosaic-relations",
+ required_files=(
+ "stage.py",
+ "graph.py",
+ "agents/run_scene_delivery_multi_agent.py",
+ "agents/final_assembler.py",
+ "agents/schema_guard.py",
+ "object_physics/run_sidecar.py",
+ "policies/multi_agent_policy.md",
+ "prompts/multi_agent/delivery_schema_guard_system.txt",
+ "prompts/multi_agent/dynamic_task_agent_compact_system.txt",
+ "prompts/multi_agent/dynamic_task_agent_system.txt",
+ "prompts/multi_agent/execution_supervisor_system.txt",
+ "prompts/multi_agent/scene_delivery_manager_compact_system.txt",
+ "prompts/multi_agent/wall_final_adjudication_task.txt",
+ "tools/benchmark_relation_async_planning.py",
+ ),
+ )
+
+
+def verify_sam3d_interface(source_root: Path) -> dict[str, Any]:
+ root = Path(source_root).expanduser().resolve()
+ required = (
+ "notebook/inference.py",
+ "sam3d_objects/__init__.py",
+ "sam3d_objects/pipeline/__init__.py",
+ )
+ report = _verify_required_files(
+ root,
+ component="sam3d",
+ release_id="repository-local-sam-3d-objects",
+ required_files=required,
+ )
+ entrypoint = root / "notebook" / "inference.py"
+ interface_valid = False
+ if entrypoint.is_file() and not entrypoint.is_symlink():
+ try:
+ tree = ast.parse(entrypoint.read_text(encoding="utf-8"))
+ interface_valid = any(
+ isinstance(node, ast.ClassDef) and node.name == "Inference"
+ for node in tree.body
+ )
+ except (OSError, SyntaxError, UnicodeError):
+ interface_valid = False
+ if not interface_valid:
+ report["failures"].append(
+ {"path": "notebook/inference.py", "reason": "Inference interface"}
+ )
+ report["interface"] = "notebook.inference.Inference"
+ report["valid"] = bool(report["valid"] and interface_valid)
+ return report
+
+
+def verify_headless_renderer_runtime(python_path: Path) -> dict[str, Any]:
+ """Check the native EGL loader used by pyrender without creating a GPU context."""
+
+ executable = Path(python_path).expanduser().resolve()
+ report: dict[str, Any] = {
+ "valid": False,
+ "python": str(executable),
+ "platform": "egl",
+ "gpu_used": False,
+ "api_calls": 0,
+ }
+ if not executable.is_file():
+ report["reason"] = "Python interpreter is missing"
+ return report
+
+ probe = """
+import ctypes
+import ctypes.util
+import json
+
+library = ctypes.util.find_library("EGL")
+if not library:
+ raise RuntimeError("the system dynamic linker cannot find EGL")
+ctypes.CDLL(library)
+import pyrender
+print(json.dumps({"egl_library": library, "pyrender": pyrender.__version__}))
+"""
+ environment = {
+ name: value
+ for name, value in os.environ.items()
+ if not SENSITIVE_ENV_NAME_RE.search(name)
+ }
+ environment.update(
+ {
+ "PYOPENGL_PLATFORM": "egl",
+ "PYTHONDONTWRITEBYTECODE": "1",
+ "PYTHONNOUSERSITE": "1",
+ "CUDA_VISIBLE_DEVICES": "",
+ }
+ )
+ try:
+ completed = subprocess.run(
+ [str(executable), "-c", probe],
+ capture_output=True,
+ text=True,
+ timeout=30,
+ check=False,
+ env=environment,
+ )
+ except (OSError, subprocess.SubprocessError) as exc:
+ report["reason"] = type(exc).__name__
+ return report
+ if completed.returncode != 0:
+ stderr = completed.stderr.strip().splitlines()
+ report["reason"] = stderr[-1] if stderr else "EGL import probe failed"
+ return report
+ try:
+ detail = json.loads(completed.stdout.strip().splitlines()[-1])
+ except (IndexError, json.JSONDecodeError):
+ report["reason"] = "EGL import probe returned invalid output"
+ return report
+ report.update({"valid": True, **detail})
+ return report
+
+
+def verify_renderer_contract(renderer_path: Path) -> dict[str, Any]:
+ if not renderer_path.is_file():
+ return {"valid": False, "missing": list(RENDERER_FUNCTIONS)}
+ try:
+ tree = ast.parse(renderer_path.read_text(encoding="utf-8"))
+ except (OSError, SyntaxError):
+ return {"valid": False, "missing": list(RENDERER_FUNCTIONS)}
+ signatures = {
+ node.name: tuple(
+ argument.arg for argument in (*node.args.posonlyargs, *node.args.args)
+ )
+ for node in tree.body
+ if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
+ }
+ missing = [
+ name
+ for name, expected in RENDERER_FUNCTIONS.items()
+ if name not in signatures or signatures[name][: len(expected)] != expected
+ ]
+ return {
+ "valid": not missing,
+ "missing_or_incompatible": missing,
+ "required_signatures": {
+ name: list(arguments) for name, arguments in RENDERER_FUNCTIONS.items()
+ },
+ "signatures": {name: list(arguments) for name, arguments in signatures.items()},
+ }
diff --git a/vlm_agent/evolution/__init__.py b/vlm_agent/evolution/__init__.py
new file mode 100644
index 0000000..be7243d
--- /dev/null
+++ b/vlm_agent/evolution/__init__.py
@@ -0,0 +1,19 @@
+from .orchestrator import PipelineOrchestrator
+from .paths import ProjectPaths
+from .scene_generator import SceneGenerator
+from .tool_runner import ToolRunner
+from .types import OptimizationAction, OptimizationReport, PipelineReport, SceneArtifacts
+from .vlm_client import DummyVLMClient, VLMClient
+
+__all__ = [
+ "DummyVLMClient",
+ "OptimizationAction",
+ "OptimizationReport",
+ "PipelineOrchestrator",
+ "PipelineReport",
+ "ProjectPaths",
+ "SceneArtifacts",
+ "SceneGenerator",
+ "ToolRunner",
+ "VLMClient",
+]
diff --git a/vlm_agent/evolution/cli.py b/vlm_agent/evolution/cli.py
new file mode 100644
index 0000000..48dc9f7
--- /dev/null
+++ b/vlm_agent/evolution/cli.py
@@ -0,0 +1,59 @@
+from __future__ import annotations
+
+import argparse
+import os
+from pathlib import Path
+
+from .orchestrator import PipelineOrchestrator
+from .paths import ProjectPaths
+from .tool_runner import ToolRunner
+from .vlm_client import VLMClient
+
+
+def build_arg_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(description="VLM-based scene optimization framework")
+ parser.add_argument("--project_root", type=Path, default=Path.cwd(), help="Project root directory")
+ parser.add_argument("--output_dir", type=Path, default=None, help="Optional output directory name or path")
+ parser.add_argument("--max_rounds", type=int, default=3, help="How many VLM optimization rounds to run")
+ parser.add_argument("--log_path", type=Path, default=None, help="Optional path for JSON output")
+ parser.add_argument("--model", type=str, default="gpt-5.5", help="VLM model name")
+ parser.add_argument("--api_key", type=str, default=os.getenv("OPENAI_API_KEY"), help="VLM API key")
+ parser.add_argument("--base_url", type=str, default="None", help="VLM base URL (optional)")
+ parser.add_argument(
+ "--collision_policy",
+ type=str,
+ choices=["strict", "repair"],
+ default="strict",
+ help="Policy for handling illegal objects in original scene: 'strict' to discard, 'repair' to attempt recovery"
+ )
+ return parser
+
+
+def main_full_pipeline(args) -> int:
+ """Optimize, generate, and compose an existing scene."""
+ if args.api_key is None:
+ raise ValueError("API key must be provided via --api_key or OPENAI_API_KEY environment variable.")
+ paths = ProjectPaths(args.project_root, output_dir=args.output_dir)
+ tool_runner = ToolRunner(args.project_root)
+ orchestrator = PipelineOrchestrator(
+ paths=paths,
+ tool_runner=tool_runner,
+ vlm_client=VLMClient(
+ url=args.base_url,
+ model=args.model,
+ api_key=args.api_key,
+ ),
+ max_rounds=args.max_rounds,
+ collision_policy=args.collision_policy,
+ )
+
+ report = orchestrator.run_full_pipeline()
+ report_path = orchestrator.save_report(report, args.log_path)
+ print(f"Pipeline completed. Report saved to: {report_path}")
+ return 0
+
+
+if __name__ == "__main__":
+ parser = build_arg_parser()
+ args = parser.parse_args()
+ raise SystemExit(main_full_pipeline(args))
\ No newline at end of file
diff --git a/vlm_agent/evolution/evolution_stage.py b/vlm_agent/evolution/evolution_stage.py
new file mode 100644
index 0000000..70b3dd4
--- /dev/null
+++ b/vlm_agent/evolution/evolution_stage.py
@@ -0,0 +1,1755 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+import shutil
+import uuid
+from dataclasses import dataclass, replace
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+import trimesh
+
+from .layout_combination import (
+ LayoutCombination,
+ LayoutVariantCandidate,
+ _layout_payload_to_matrix,
+)
+from .orchestrator import PipelineOrchestrator
+from ..reconstruction.relations.graph import RelationGraphError, RelationGraphView
+from ..reconstruction.relations.stage import RelationArtifacts
+from ..artifacts import RunContract
+from .tool_runner import ToolRunner, ToolRunResult
+from .types import PipelineReport, SceneArtifacts
+
+SAFE_STAGE_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
+
+
+class SceneMosaicArtifactError(RuntimeError):
+ """Raised when an immutable Stage 08 attempt cannot be prepared."""
+
+
+@dataclass(frozen=True)
+class SceneMosaicStageArtifacts:
+ stage_id: str
+ workspace_dir: Path
+ integration_manifest_path: Path
+ relation_artifacts: RelationArtifacts
+
+
+def _utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _write_json(path: Path, value: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(
+ json.dumps(value, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+
+
+def _atomic_write_json(path: Path, value: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.parent / f".{path.name}.{uuid.uuid4().hex}.tmp"
+ _write_json(temporary, value)
+ os.replace(temporary, path)
+
+
+class SceneMosaicStageAdapter:
+ """Prepare a writable Stage 08 copy from immutable upstream facts."""
+
+ def __init__(
+ self,
+ *,
+ scene_config_path: Path,
+ relation_artifacts: RelationArtifacts,
+ ) -> None:
+ self.contract = RunContract.from_scene_config(scene_config_path)
+ self.relation_artifacts = relation_artifacts
+
+ @property
+ def stage08_root(self) -> Path:
+ return self.contract.evolution_dir
+
+ @property
+ def attempts_dir(self) -> Path:
+ return self.stage08_root / ".stage08_attempts"
+
+ @property
+ def publications_dir(self) -> Path:
+ return self.stage08_root / ".stage08_publications"
+
+ def attempt_workspace(self, stage_id: str) -> Path:
+ if not SAFE_STAGE_ID_RE.fullmatch(stage_id):
+ raise SceneMosaicArtifactError("Stage 08 stage_id is unsafe")
+ return self.attempts_dir / stage_id
+
+ def prepare(self, stage_id: str) -> SceneMosaicStageArtifacts:
+ if not SAFE_STAGE_ID_RE.fullmatch(stage_id):
+ raise SceneMosaicArtifactError("Stage 08 stage_id is unsafe")
+ target = self.attempt_workspace(stage_id)
+ if target.exists():
+ raise SceneMosaicArtifactError(
+ f"Stage 08 attempt already exists: {target}"
+ )
+ temporary = (
+ target.parent / f".{stage_id}.{uuid.uuid4().hex}.tmp"
+ )
+ temporary.mkdir(parents=True, exist_ok=False)
+ try:
+ source_records, formal_records = self._materialize(
+ temporary, target
+ )
+ manifest = {
+ "schema_version": "scenemosaic_stage08_integration",
+ "stage_id": stage_id,
+ "created_at_utc": _utc_now(),
+ "scene_name": self.contract.scene_name,
+ "coordinate_system": "OpenGL_glTF_global",
+ "ground_plane": "XZ",
+ "relation_release_id": self.relation_artifacts.release_id,
+ "relation_publication_id": (
+ self.relation_artifacts.publication_id
+ ),
+ "relation_publication_manifest": str(
+ self.relation_artifacts.publication_manifest_path
+ ),
+ "relation_publication_manifest_sha256": _sha256(
+ self.relation_artifacts.publication_manifest_path
+ ),
+ "source_artifacts": source_records,
+ "formal_working_copies": formal_records,
+ "relation_metadata": dict(
+ self.relation_artifacts.metadata
+ ),
+ "wall_padding": 0.0,
+ "full_relations_preserved": True,
+ "per_object_physics_consumed": False,
+ "upstream_stage05_read_only": True,
+ "relation_publication_read_only": True,
+ "api_calls": 0,
+ }
+ _write_json(temporary / "integration_manifest.json", manifest)
+ target.parent.mkdir(parents=True, exist_ok=True)
+ os.replace(temporary, target)
+ except Exception:
+ if temporary.exists():
+ shutil.rmtree(temporary)
+ raise
+
+ return SceneMosaicStageArtifacts(
+ stage_id=stage_id,
+ workspace_dir=target,
+ integration_manifest_path=target / "integration_manifest.json",
+ relation_artifacts=self.relation_artifacts,
+ )
+
+ def _materialize(
+ self,
+ temporary: Path,
+ final_target: Path,
+ ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ stage05 = self.contract.stage05_dir
+ layout_source = stage05 / "layout_params"
+ unposed_source = stage05 / "unposed_objects"
+ if not layout_source.is_dir() or not unposed_source.is_dir():
+ raise SceneMosaicArtifactError(
+ "Stage 05 layout_params and unposed_objects are required"
+ )
+
+ layout_target = temporary / "layout_params"
+ unposed_target = temporary / "unposed_objects"
+ collision_target = temporary / "collision_hulls"
+ layout_target.mkdir()
+ unposed_target.mkdir()
+ collision_target.mkdir()
+
+ copies: list[tuple[str, Path, Path, Path]] = [
+ (
+ "input_image",
+ self.contract.input_image,
+ temporary / "image.png",
+ final_target / "image.png",
+ ),
+ (
+ "relation_objects",
+ self.relation_artifacts.objects_path,
+ temporary / "objects.json",
+ final_target / "objects.json",
+ ),
+ (
+ "relation_relations",
+ self.relation_artifacts.relations_path,
+ temporary / "relations.json",
+ final_target / "relations.json",
+ ),
+ ]
+ for source in sorted(layout_source.glob("object_*.json")):
+ copies.append(
+ (
+ "layout_params",
+ source,
+ layout_target / source.name,
+ final_target / "layout_params" / source.name,
+ )
+ )
+ for source in sorted(unposed_source.glob("object_*.glb")):
+ copies.append(
+ (
+ "unposed_object",
+ source,
+ unposed_target / source.name,
+ final_target / "unposed_objects" / source.name,
+ )
+ )
+
+ source_records: list[dict[str, Any]] = []
+ formal_records: list[dict[str, Any]] = []
+ for role, source, temporary_path, final_path in copies:
+ source = source.expanduser().resolve()
+ if (
+ not source.is_file()
+ or source.is_symlink()
+ or source.stat().st_size == 0
+ ):
+ raise SceneMosaicArtifactError(
+ f"source must be a real non-empty file: {source}"
+ )
+ source_sha = _sha256(source)
+ shutil.copy2(source, temporary_path)
+ formal_sha = _sha256(temporary_path)
+ if source_sha != formal_sha:
+ raise SceneMosaicArtifactError(
+ f"Stage 08 copy hash mismatch: {source}"
+ )
+ source_records.append(
+ {
+ "role": role,
+ "path": str(source),
+ "sha256": source_sha,
+ "bytes": source.stat().st_size,
+ }
+ )
+ formal_records.append(
+ {
+ "role": role,
+ "path": str(final_path),
+ "sha256": formal_sha,
+ "bytes": temporary_path.stat().st_size,
+ "identical_to_source": True,
+ }
+ )
+
+ layout_ids = tuple(
+ record["path"].rsplit("/", 1)[-1].removesuffix(".json")
+ for record in formal_records
+ if record["role"] == "layout_params"
+ )
+ unposed_ids = tuple(
+ record["path"].rsplit("/", 1)[-1].removesuffix(".glb")
+ for record in formal_records
+ if record["role"] == "unposed_object"
+ )
+ try:
+ objects = json.loads(
+ (temporary / "objects.json").read_text(encoding="utf-8")
+ )
+ relations = json.loads(
+ (temporary / "relations.json").read_text(encoding="utf-8")
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "copied relation artifacts are not valid JSON"
+ ) from exc
+ object_items = (
+ objects.get("objects", []) if isinstance(objects, dict) else []
+ )
+ object_ids = tuple(
+ str(item.get("id") or item.get("object_id") or "")
+ for item in object_items
+ if isinstance(item, dict)
+ )
+ if not (
+ object_ids
+ and object_ids == layout_ids == unposed_ids
+ and isinstance(relations, list)
+ ):
+ raise SceneMosaicArtifactError(
+ "Stage 08 object, layout and asset IDs are not closed"
+ )
+ graph = RelationGraphView.from_relations(relations)
+ if set(graph.primary_parent_by_target) != set(object_ids):
+ raise SceneMosaicArtifactError(
+ "Stage 08 relations do not cover the object ID closure"
+ )
+ _write_json(
+ temporary / "relation_graph_audit.json",
+ graph.audit_summary(),
+ )
+ return source_records, formal_records
+
+ @staticmethod
+ def verify_upstream_unchanged(
+ integration_manifest_path: Path,
+ ) -> dict[str, Any]:
+ try:
+ manifest = json.loads(
+ Path(integration_manifest_path).read_text(encoding="utf-8")
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "cannot parse Stage 08 integration manifest"
+ ) from exc
+ records = manifest.get("source_artifacts")
+ if not isinstance(records, list):
+ raise SceneMosaicArtifactError(
+ "integration manifest lacks source hashes"
+ )
+ failures: list[dict[str, Any]] = []
+ for record in records:
+ path = Path(str(record.get("path") or "")).expanduser().resolve()
+ if not path.is_file():
+ failures.append(
+ {"path": str(path), "reason": "missing"}
+ )
+ continue
+ actual = _sha256(path)
+ if actual != record.get("sha256"):
+ failures.append(
+ {
+ "path": str(path),
+ "reason": "sha256_mismatch",
+ "actual": actual,
+ "expected": record.get("sha256"),
+ }
+ )
+ return {
+ "valid": not failures,
+ "checked": len(records),
+ "failures": failures,
+ }
+
+ @staticmethod
+ def verify_prepared_working_copies(
+ integration_manifest_path: Path,
+ ) -> dict[str, Any]:
+ manifest_path = Path(
+ integration_manifest_path
+ ).expanduser().resolve()
+ try:
+ manifest = json.loads(
+ manifest_path.read_text(encoding="utf-8")
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "cannot parse Stage 08 integration manifest"
+ ) from exc
+ workspace = manifest_path.parent.resolve()
+ records = manifest.get("formal_working_copies")
+ if (
+ manifest.get("schema_version")
+ != "scenemosaic_stage08_integration"
+ or manifest.get("wall_padding") != 0.0
+ or manifest.get("full_relations_preserved") is not True
+ or manifest.get("per_object_physics_consumed") is not False
+ or not isinstance(records, list)
+ or not records
+ ):
+ raise SceneMosaicArtifactError(
+ "prepared Stage 08 integration facts are invalid"
+ )
+ failures: list[dict[str, Any]] = []
+ seen: set[Path] = set()
+ for record in records:
+ if not isinstance(record, dict):
+ failures.append({"reason": "invalid_record"})
+ continue
+ path = Path(
+ str(record.get("path") or "")
+ ).expanduser().resolve()
+ if (
+ path in seen
+ or not path.is_relative_to(workspace)
+ or not path.is_file()
+ or path.is_symlink()
+ or record.get("identical_to_source") is not True
+ ):
+ failures.append(
+ {"path": str(path), "reason": "missing_or_unsafe"}
+ )
+ continue
+ seen.add(path)
+ actual = _sha256(path)
+ if (
+ actual != record.get("sha256")
+ or path.stat().st_size != record.get("bytes")
+ ):
+ failures.append(
+ {
+ "path": str(path),
+ "reason": "hash_or_size_mismatch",
+ "actual": actual,
+ "expected": record.get("sha256"),
+ }
+ )
+ return {
+ "valid": not failures,
+ "checked": len(records),
+ "failures": failures,
+ }
+
+ def _validate_filtered_relation_view(
+ self,
+ workspace: Path,
+ ) -> dict[str, Any]:
+ relations_path = workspace / "relations_filtered.json"
+ objects_path = workspace / "objects_filtered.json"
+ if not relations_path.is_file() or not objects_path.is_file():
+ raise SceneMosaicArtifactError(
+ "final Stage 08 output requires filtered objects and relations"
+ )
+ try:
+ filtered_relations = json.loads(
+ relations_path.read_text(encoding="utf-8")
+ )
+ filtered_objects = json.loads(
+ objects_path.read_text(encoding="utf-8")
+ )
+ full_relations = json.loads(
+ (workspace / "relations.json").read_text(encoding="utf-8")
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "Stage 08 filtered relation artifacts are not valid JSON"
+ ) from exc
+ if (
+ not isinstance(filtered_relations, list)
+ or not isinstance(full_relations, list)
+ or not isinstance(filtered_objects, dict)
+ or not isinstance(filtered_objects.get("objects"), list)
+ ):
+ raise SceneMosaicArtifactError(
+ "Stage 08 filtered relation artifacts have invalid shapes"
+ )
+ object_ids = {
+ str(item.get("id") or item.get("object_id") or "")
+ for item in filtered_objects["objects"]
+ if isinstance(item, dict)
+ }
+ if "" in object_ids:
+ raise SceneMosaicArtifactError(
+ "objects_filtered.json contains an empty object ID"
+ )
+ try:
+ expected = RelationGraphView.from_relations(
+ full_relations
+ ).filtered_to_object_ids(object_ids)
+ actual = RelationGraphView.from_relations(filtered_relations)
+ except RelationGraphError as exc:
+ raise SceneMosaicArtifactError(
+ f"filtered relation graph is invalid: {exc}"
+ ) from exc
+ if actual.full_relations_list() != expected.full_relations_list():
+ raise SceneMosaicArtifactError(
+ "relations_filtered.json is not the exact surviving full graph"
+ )
+ return {
+ "surviving_object_ids": sorted(object_ids),
+ "graph": actual.audit_summary(),
+ "secondary_wall_constraints_preserved": True,
+ "primary_ownership_only": True,
+ }
+
+ def _validate_final_outputs(
+ self,
+ artifacts: SceneMosaicStageArtifacts,
+ report_path: Path,
+ ) -> dict[str, Any]:
+ workspace = artifacts.workspace_dir.resolve()
+ expected_workspace = self.attempt_workspace(artifacts.stage_id).resolve()
+ if (
+ workspace != expected_workspace
+ or not workspace.is_dir()
+ or artifacts.relation_artifacts.publication_id
+ != self.relation_artifacts.publication_id
+ ):
+ raise SceneMosaicArtifactError(
+ "Stage 08 artifacts do not belong to this adapter"
+ )
+ if (
+ not report_path.is_file()
+ or report_path.is_symlink()
+ or report_path.stat().st_size == 0
+ or not report_path.is_relative_to(workspace)
+ ):
+ raise SceneMosaicArtifactError(
+ "pipeline report must be a real file inside Stage 08"
+ )
+ try:
+ report = json.loads(report_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "pipeline report is not valid JSON"
+ ) from exc
+ if not isinstance(report, dict) or report.get("stage") != "composition":
+ raise SceneMosaicArtifactError(
+ "pipeline report must describe the composition stage"
+ )
+
+ integration_path = workspace / "integration_manifest.json"
+ try:
+ integration = json.loads(
+ integration_path.read_text(encoding="utf-8")
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "Stage 08 integration manifest is invalid"
+ ) from exc
+ if (
+ integration.get("stage_id") != artifacts.stage_id
+ or integration.get("relation_publication_id")
+ != self.relation_artifacts.publication_id
+ or integration.get("wall_padding") != 0.0
+ or integration.get("full_relations_preserved") is not True
+ or integration.get("per_object_physics_consumed") is not False
+ ):
+ raise SceneMosaicArtifactError(
+ "Stage 08 integration invariants are not satisfied"
+ )
+
+ combined_root = workspace / "generated" / "combined_scenes"
+ combo_index_path = combined_root / "combo_index.json"
+ try:
+ combo_index = json.loads(
+ combo_index_path.read_text(encoding="utf-8")
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "final Stage 08 combo_index.json is missing or invalid"
+ ) from exc
+ entries = (
+ combo_index.get("combined_scenes")
+ if isinstance(combo_index, dict)
+ else None
+ )
+ count = (
+ combo_index.get("combined_scene_count")
+ if isinstance(combo_index, dict)
+ else None
+ )
+ if (
+ not isinstance(entries, list)
+ or not isinstance(count, int)
+ or count <= 0
+ or count != len(entries)
+ ):
+ raise SceneMosaicArtifactError(
+ "final combo index must contain at least one scene"
+ )
+
+ glb_paths: list[Path] = []
+ for path in sorted(combined_root.rglob("*")):
+ if path.is_symlink():
+ raise SceneMosaicArtifactError(
+ f"final combined output cannot be a symlink: {path}"
+ )
+ if path.is_file() and path.suffix.lower() == ".glb":
+ if path.stat().st_size == 0:
+ raise SceneMosaicArtifactError(
+ f"final combined GLB is empty: {path}"
+ )
+ glb_paths.append(path)
+ if not glb_paths:
+ raise SceneMosaicArtifactError(
+ "final Stage 08 output has no non-empty GLB"
+ )
+
+ return {
+ "workspace": workspace,
+ "report": report_path,
+ "integration": integration_path,
+ "combined_root": combined_root,
+ "combo_index": combo_index_path,
+ "combo_index_payload": combo_index,
+ "glb_paths": glb_paths,
+ "filtered_relations": self._validate_filtered_relation_view(
+ workspace
+ ),
+ }
+
+ @staticmethod
+ def _copy_publication_file(
+ *,
+ source: Path,
+ workspace: Path,
+ temporary: Path,
+ publication_dir: Path,
+ ) -> dict[str, Any]:
+ if (
+ not source.is_file()
+ or source.is_symlink()
+ or source.stat().st_size == 0
+ or not source.resolve().is_relative_to(workspace)
+ ):
+ raise SceneMosaicArtifactError(
+ f"publication source must be a real workspace file: {source}"
+ )
+ relative = source.relative_to(workspace)
+ destination = temporary / "outputs" / relative
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ before = _sha256(source)
+ shutil.copy2(source, destination)
+ copied = _sha256(destination)
+ after = _sha256(source)
+ if before != copied or before != after:
+ raise SceneMosaicArtifactError(
+ f"Stage 08 output changed while publishing: {source}"
+ )
+ return {
+ "source_path": str(source),
+ "path": str(publication_dir / "outputs" / relative),
+ "relative_path": str(Path("outputs") / relative),
+ "sha256": copied,
+ "bytes": destination.stat().st_size,
+ }
+
+ def publish(
+ self,
+ artifacts: SceneMosaicStageArtifacts,
+ *,
+ pipeline_report_path: Path,
+ ) -> Path:
+ report_path = Path(pipeline_report_path).expanduser().resolve()
+ final = self._validate_final_outputs(artifacts, report_path)
+ upstream = self.verify_upstream_unchanged(
+ artifacts.integration_manifest_path
+ )
+ if upstream["valid"] is not True:
+ raise SceneMosaicArtifactError(
+ "upstream facts changed during Stage 08"
+ )
+
+ publication_dir = self.publications_dir / artifacts.stage_id
+ if publication_dir.exists():
+ raise SceneMosaicArtifactError(
+ f"Stage 08 publication already exists: {publication_dir}"
+ )
+ temporary = (
+ publication_dir.parent
+ / f".{artifacts.stage_id}.{uuid.uuid4().hex}.tmp"
+ )
+ temporary.mkdir(parents=True, exist_ok=False)
+ try:
+ workspace = final["workspace"]
+ selected: set[Path] = {
+ final["report"],
+ final["integration"],
+ workspace / "relation_graph_audit.json",
+ workspace / "objects.json",
+ workspace / "relations.json",
+ workspace / "objects_filtered.json",
+ workspace / "relations_filtered.json",
+ }
+ optional = workspace / "cross_anchor_constraints.json"
+ if optional.is_file():
+ selected.add(optional)
+ for path in final["combined_root"].rglob("*"):
+ if path.is_symlink():
+ raise SceneMosaicArtifactError(
+ f"final combined output cannot be a symlink: {path}"
+ )
+ if path.is_file():
+ selected.add(path)
+
+ output_records = [
+ self._copy_publication_file(
+ source=path,
+ workspace=workspace,
+ temporary=temporary,
+ publication_dir=publication_dir,
+ )
+ for path in sorted(selected)
+ ]
+ glb_relative_paths = {
+ str(path.relative_to(workspace))
+ for path in final["glb_paths"]
+ }
+ published_glbs = [
+ {
+ "relative_path": record["relative_path"],
+ "sha256": record["sha256"],
+ "bytes": record["bytes"],
+ }
+ for record in output_records
+ if record["relative_path"].removeprefix("outputs/")
+ in glb_relative_paths
+ ]
+ if not published_glbs:
+ raise SceneMosaicArtifactError(
+ "no final GLB was copied into the publication"
+ )
+ representative_records = self._materialize_representative_scenes(
+ temporary=temporary,
+ publication_dir=publication_dir,
+ combined_root=final["combined_root"],
+ published_glbs=published_glbs,
+ )
+ output_records.extend(representative_records)
+ publication_index = {
+ "schema_version": "scenemosaic_stage08_publication_index",
+ "source_combo_index": str(
+ Path("outputs")
+ / final["combo_index"].relative_to(workspace)
+ ),
+ "combined_scene_count": final["combo_index_payload"][
+ "combined_scene_count"
+ ],
+ "published_glbs": published_glbs,
+ }
+ _write_json(
+ temporary / "publication_combo_index.json",
+ publication_index,
+ )
+ manifest = {
+ "schema_version": "scenemosaic_stage08_publication",
+ "stage_id": artifacts.stage_id,
+ "published_at_utc": _utc_now(),
+ "immutable_copy": True,
+ "source_workspace_dir": str(workspace),
+ "pipeline_report": str(
+ Path("outputs") / report_path.relative_to(workspace)
+ ),
+ "relation_publication_id": (
+ artifacts.relation_artifacts.publication_id
+ ),
+ "relation_publication_manifest_sha256": _sha256(
+ artifacts.relation_artifacts.publication_manifest_path
+ ),
+ "combo_index": publication_index["source_combo_index"],
+ "publication_combo_index": "publication_combo_index.json",
+ "glb_count": len(published_glbs),
+ "outputs": output_records,
+ "filtered_relations": final["filtered_relations"],
+ "upstream_unchanged": upstream,
+ }
+ _write_json(
+ temporary / "publication_manifest.json", manifest
+ )
+ publication_dir.parent.mkdir(parents=True, exist_ok=True)
+ os.replace(temporary, publication_dir)
+ except Exception:
+ if temporary.exists():
+ shutil.rmtree(temporary)
+ raise
+
+ current = {
+ "schema_version": "scenemosaic_stage08_current",
+ "active": True,
+ "stage_id": artifacts.stage_id,
+ "publication_manifest": str(
+ Path(".stage08_publications")
+ / artifacts.stage_id
+ / "publication_manifest.json"
+ ),
+ "updated_at_utc": _utc_now(),
+ }
+ _atomic_write_json(
+ self.publications_dir / "current.json",
+ current,
+ )
+ self._publish_native_outputs(publication_dir)
+ return publication_dir / "publication_manifest.json"
+
+ @staticmethod
+ def _materialize_representative_scenes(
+ *,
+ temporary: Path,
+ publication_dir: Path,
+ combined_root: Path,
+ published_glbs: list[dict[str, Any]],
+ ) -> list[dict[str, Any]]:
+ """Expose ranked final GLBs in the native evolution output tree."""
+
+ summary_path = combined_root / "selection_summary.json"
+ try:
+ summary = json.loads(summary_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "final Stage 08 selection summary is missing or invalid"
+ ) from exc
+ selected = summary.get("selected_scenes")
+ if not isinstance(selected, list) or len(selected) != len(published_glbs):
+ raise SceneMosaicArtifactError(
+ "final Stage 08 selected scenes do not match published GLBs"
+ )
+ ranks: dict[str, int] = {}
+ for item in selected:
+ if not isinstance(item, dict):
+ raise SceneMosaicArtifactError("final Stage 08 selected scene is invalid")
+ name = str(item.get("combo_name") or "")
+ rank = item.get("rank")
+ if not name or not isinstance(rank, int) or rank < 1 or name in ranks:
+ raise SceneMosaicArtifactError("final Stage 08 selected scene rank is invalid")
+ ranks[name] = rank
+ if set(ranks.values()) != set(range(1, len(published_glbs) + 1)):
+ raise SceneMosaicArtifactError("final Stage 08 selected scene ranks are incomplete")
+
+ result_dir = temporary / "outputs" / "representative_scenes"
+ result_dir.mkdir(parents=True, exist_ok=False)
+ records: list[dict[str, Any]] = []
+ for glb in published_glbs:
+ relative = Path(str(glb["relative_path"]))
+ source = temporary / relative
+ combo_name = source.parent.name
+ rank = ranks.get(combo_name)
+ if rank is None or not source.is_file() or source.is_symlink():
+ raise SceneMosaicArtifactError(
+ "final Stage 08 representative GLB is invalid"
+ )
+ destination = result_dir / f"scene_{rank:02d}.glb"
+ os.link(source, destination)
+ copied = _sha256(destination)
+ if copied != glb["sha256"] or destination.stat().st_size != glb["bytes"]:
+ raise SceneMosaicArtifactError(
+ "final Stage 08 representative GLB hash binding failed"
+ )
+ representative_relative = (
+ Path("outputs") / "representative_scenes" / destination.name
+ )
+ records.append(
+ {
+ "source_path": str(publication_dir / relative),
+ "path": str(publication_dir / representative_relative),
+ "relative_path": str(representative_relative),
+ "sha256": copied,
+ "bytes": destination.stat().st_size,
+ }
+ )
+ return records
+
+ def _publish_native_outputs(self, publication_dir: Path) -> None:
+ """Atomically materialize the original evolution ``outputs/`` layout.
+
+ Every regular file is hard-linked to the hash-bound immutable
+ publication. The public view is therefore a normal directory that
+ costs no additional GLB storage and cannot diverge silently.
+ """
+
+ target = self.stage08_root / "outputs"
+ if target.exists() or target.is_symlink():
+ raise SceneMosaicArtifactError(
+ "native evolution outputs already exist and will not be overwritten: "
+ f"{target}"
+ )
+ source = publication_dir / "outputs"
+ if not source.is_dir() or source.is_symlink():
+ raise SceneMosaicArtifactError("Stage 08 publication outputs are missing")
+ temporary = self.stage08_root / f".outputs.{uuid.uuid4().hex}.tmp"
+ try:
+ temporary.mkdir(parents=False, exist_ok=False)
+ for source_path in sorted(source.rglob("*")):
+ if source_path.is_symlink():
+ raise SceneMosaicArtifactError(
+ f"Stage 08 publication output is a symlink: {source_path}"
+ )
+ destination = temporary / source_path.relative_to(source)
+ if source_path.is_dir():
+ destination.mkdir()
+ elif source_path.is_file():
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ os.link(source_path, destination)
+ else:
+ raise SceneMosaicArtifactError(
+ f"Stage 08 publication output has unsupported type: {source_path}"
+ )
+ os.replace(temporary, target)
+ except Exception:
+ if temporary.exists() or temporary.is_symlink():
+ if temporary.is_dir() and not temporary.is_symlink():
+ shutil.rmtree(temporary)
+ else:
+ temporary.unlink()
+ raise
+
+ def validate_publication(
+ self,
+ publication_manifest_path: Path | None = None,
+ ) -> dict[str, Any]:
+ """Revalidate an immutable Stage 08 publication and active pointer."""
+
+ current_path = self.publications_dir / "current.json"
+ try:
+ current = json.loads(
+ current_path.read_text(encoding="utf-8")
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "Stage 08 active publication pointer is missing or invalid"
+ ) from exc
+ stage_id = str(current.get("stage_id") or "")
+ if (
+ current_path.is_symlink()
+ or current.get("schema_version")
+ != "scenemosaic_stage08_current"
+ or current.get("active") is not True
+ or not SAFE_STAGE_ID_RE.fullmatch(stage_id)
+ ):
+ raise SceneMosaicArtifactError(
+ "Stage 08 active publication pointer is unsafe"
+ )
+ declared = Path(
+ str(current.get("publication_manifest") or "")
+ )
+ if declared.is_absolute():
+ raise SceneMosaicArtifactError(
+ "Stage 08 current pointer must use a relative manifest path"
+ )
+ manifest_path = (
+ self.stage08_root / declared
+ ).resolve()
+ expected_manifest = (
+ self.publications_dir / stage_id / "publication_manifest.json"
+ ).resolve()
+ if (
+ manifest_path != expected_manifest
+ or (
+ publication_manifest_path is not None
+ and Path(
+ publication_manifest_path
+ ).expanduser().resolve()
+ != manifest_path
+ )
+ or not manifest_path.is_file()
+ or manifest_path.is_symlink()
+ or manifest_path.stat().st_size == 0
+ ):
+ raise SceneMosaicArtifactError(
+ "Stage 08 current pointer and publication manifest disagree"
+ )
+ try:
+ manifest = json.loads(
+ manifest_path.read_text(encoding="utf-8")
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication manifest is invalid"
+ ) from exc
+ if (
+ manifest.get("schema_version")
+ != "scenemosaic_stage08_publication"
+ or manifest.get("stage_id") != stage_id
+ or manifest.get("immutable_copy") is not True
+ or manifest.get("relation_publication_id")
+ != self.relation_artifacts.publication_id
+ or manifest.get("relation_publication_manifest_sha256")
+ != _sha256(
+ self.relation_artifacts.publication_manifest_path
+ )
+ or not isinstance(manifest.get("upstream_unchanged"), dict)
+ or manifest["upstream_unchanged"].get("valid") is not True
+ ):
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication invariants are invalid"
+ )
+
+ publication_dir = manifest_path.parent.resolve()
+ outputs_root = publication_dir / "outputs"
+ records = manifest.get("outputs")
+ if not isinstance(records, list) or not records:
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication has no output hash closure"
+ )
+ record_by_relative: dict[str, dict[str, Any]] = {}
+ expected_output_paths: set[Path] = set()
+ for record in records:
+ if not isinstance(record, dict):
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication output record is invalid"
+ )
+ relative = str(record.get("relative_path") or "")
+ relative_path = Path(relative)
+ path = Path(
+ str(record.get("path") or "")
+ ).expanduser().resolve()
+ expected = (
+ publication_dir / relative_path
+ ).resolve()
+ if (
+ not relative
+ or relative in record_by_relative
+ or relative_path.is_absolute()
+ or not relative_path.is_relative_to(Path("outputs"))
+ or path != expected
+ or not path.is_relative_to(outputs_root)
+ or not path.is_file()
+ or path.is_symlink()
+ or path.stat().st_size == 0
+ or path.stat().st_size != record.get("bytes")
+ or _sha256(path) != record.get("sha256")
+ ):
+ raise SceneMosaicArtifactError(
+ f"Stage 08 publication output verification failed: {relative}"
+ )
+ record_by_relative[relative] = record
+ expected_output_paths.add(path)
+
+ actual_output_paths: set[Path] = set()
+ if not outputs_root.is_dir() or outputs_root.is_symlink():
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication outputs directory is unsafe"
+ )
+ for path in outputs_root.rglob("*"):
+ if path.is_symlink():
+ raise SceneMosaicArtifactError(
+ f"Stage 08 publication contains a symlink: {path}"
+ )
+ if path.is_file():
+ actual_output_paths.add(path.resolve())
+ if actual_output_paths != expected_output_paths:
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication output closure has missing or extra files"
+ )
+
+ combo_index_name = str(
+ manifest.get("publication_combo_index") or ""
+ )
+ if combo_index_name != "publication_combo_index.json":
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication combo index path is invalid"
+ )
+ publication_combo_path = publication_dir / combo_index_name
+ try:
+ publication_combo = json.loads(
+ publication_combo_path.read_text(encoding="utf-8")
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication combo index is invalid"
+ ) from exc
+ published_glbs = publication_combo.get("published_glbs")
+ combined_count = publication_combo.get("combined_scene_count")
+ source_combo = str(
+ publication_combo.get("source_combo_index") or ""
+ )
+ if (
+ publication_combo_path.is_symlink()
+ or publication_combo.get("schema_version")
+ != "scenemosaic_stage08_publication_index"
+ or source_combo != manifest.get("combo_index")
+ or source_combo not in record_by_relative
+ or not isinstance(combined_count, int)
+ or combined_count <= 0
+ or not isinstance(published_glbs, list)
+ or not published_glbs
+ or len(published_glbs) != manifest.get("glb_count")
+ ):
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication combo facts are inconsistent"
+ )
+ for glb in published_glbs:
+ if not isinstance(glb, dict):
+ raise SceneMosaicArtifactError(
+ "Stage 08 published GLB record is invalid"
+ )
+ relative = str(glb.get("relative_path") or "")
+ output = record_by_relative.get(relative)
+ if (
+ output is None
+ or Path(relative).suffix.lower() != ".glb"
+ or glb.get("sha256") != output.get("sha256")
+ or glb.get("bytes") != output.get("bytes")
+ ):
+ raise SceneMosaicArtifactError(
+ f"Stage 08 published GLB hash binding failed: {relative}"
+ )
+
+ report_relative = str(manifest.get("pipeline_report") or "")
+ report_record = record_by_relative.get(report_relative)
+ if report_record is None:
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication pipeline report is not hash-bound"
+ )
+ try:
+ report = json.loads(
+ Path(str(report_record["path"])).read_text(
+ encoding="utf-8"
+ )
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise SceneMosaicArtifactError(
+ "Stage 08 published pipeline report is invalid"
+ ) from exc
+ filtered = manifest.get("filtered_relations")
+ if (
+ not isinstance(report, dict)
+ or report.get("stage") != "composition"
+ or not isinstance(filtered, dict)
+ or filtered.get(
+ "secondary_wall_constraints_preserved"
+ )
+ is not True
+ or filtered.get("primary_ownership_only") is not True
+ ):
+ raise SceneMosaicArtifactError(
+ "Stage 08 published relation-consumption facts are invalid"
+ )
+
+ allowed_top_level = {
+ manifest_path.resolve(),
+ publication_combo_path.resolve(),
+ }
+ actual_top_level = {
+ path.resolve()
+ for path in publication_dir.iterdir()
+ if path.is_file()
+ }
+ if actual_top_level != allowed_top_level:
+ raise SceneMosaicArtifactError(
+ "Stage 08 publication has unexpected top-level files"
+ )
+ return {
+ "valid": True,
+ "stage_id": stage_id,
+ "publication_manifest": str(manifest_path),
+ "publication_manifest_sha256": _sha256(manifest_path),
+ "current_pointer": str(current_path),
+ "output_count": len(records),
+ "glb_count": len(published_glbs),
+ "combined_scene_count": combined_count,
+ "relation_publication_id": (
+ self.relation_artifacts.publication_id
+ ),
+ }
+
+
+class IntegratedToolRunner(ToolRunner):
+ """Existing SceneMosaic tools with integration invariants enforced."""
+
+ def generate_room_mesh(
+ self,
+ layout_dir: Path,
+ unposed_dir: Path,
+ objects_json: Path,
+ *,
+ material_dir: Path | None = None,
+ thickness: float | None = None,
+ wall_padding: float | None = None,
+ ) -> ToolRunResult:
+ if wall_padding is not None and float(wall_padding) != 0.0:
+ raise ValueError(
+ "published relation walls require wall_padding=0.0"
+ )
+ return super().generate_room_mesh(
+ layout_dir,
+ unposed_dir,
+ objects_json,
+ material_dir=material_dir,
+ thickness=thickness,
+ wall_padding=0.0,
+ )
+
+
+class RelationAwareLayoutCombination(LayoutCombination):
+ """Combination driven by the same filtered primary ownership graph."""
+
+ def _load_relationships_and_objects(
+ self,
+ ) -> tuple[list[dict], list[dict], list[dict]]:
+ filtered_relations = self.paths.outputs_dir / "relations_filtered.json"
+ filtered_objects = self.paths.outputs_dir / "objects_filtered.json"
+ if filtered_relations.exists() != filtered_objects.exists():
+ raise RelationGraphError(
+ "filtered objects and relations must be published together"
+ )
+ suffix = "_filtered" if filtered_relations.is_file() else ""
+ relations_path = self.paths.outputs_dir / f"relations{suffix}.json"
+ objects_path = self.paths.outputs_dir / f"objects{suffix}.json"
+ try:
+ relations = json.loads(
+ relations_path.read_text(encoding="utf-8")
+ )
+ objects_payload = json.loads(
+ objects_path.read_text(encoding="utf-8")
+ )
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RelationGraphError(
+ "combination relation inputs are invalid"
+ ) from exc
+ if not isinstance(relations, list) or not isinstance(
+ objects_payload, dict
+ ):
+ raise RelationGraphError(
+ "combination relation inputs have invalid shapes"
+ )
+ objects = objects_payload.get("objects")
+ walls = objects_payload.get("walls")
+ if not isinstance(objects, list) or not isinstance(walls, list):
+ raise RelationGraphError(
+ "combination objects must preserve objects and walls"
+ )
+ view = RelationGraphView.from_relations(relations)
+ object_ids = {
+ str(item.get("id") or item.get("object_id") or "")
+ for item in objects
+ if isinstance(item, dict)
+ }
+ if set(view.primary_parent_by_target) != object_ids:
+ raise RelationGraphError(
+ "combination relation targets do not match surviving objects"
+ )
+ if suffix:
+ full_path = self.paths.outputs_dir / "relations.json"
+ full = json.loads(full_path.read_text(encoding="utf-8"))
+ expected = RelationGraphView.from_relations(
+ full
+ ).filtered_to_object_ids(object_ids)
+ if view.full_relations_list() != expected.full_relations_list():
+ raise RelationGraphError(
+ "combination must consume the exact filtered relation graph"
+ )
+ return relations, objects, walls
+
+ @staticmethod
+ def _load_relationship_graph(
+ relations: list[dict],
+ ) -> tuple[dict[str, set[str]], dict[str, set[str]]]:
+ view = RelationGraphView.from_relations(relations)
+ parents_by_child = {
+ target: {parent}
+ for target, parent in view.primary_parent_by_target.items()
+ }
+ children_by_parent = {
+ anchor: set(targets)
+ for anchor, targets in view.primary_children_by_anchor.items()
+ }
+ return parents_by_child, children_by_parent
+
+ def _choose_parent_by_child(
+ self, parents_by_child: dict[str, set[str]]
+ ) -> dict[str, str]:
+ selected: dict[str, str] = {}
+ for target, parents in sorted(parents_by_child.items()):
+ if len(parents) != 1:
+ raise RelationGraphError(
+ f"target {target} must have one deterministic primary owner"
+ )
+ selected[target] = next(iter(parents))
+ return selected
+
+ @staticmethod
+ def _mesh_wall_distance(
+ *,
+ object_id: str,
+ layout_dir: Path,
+ unposed_dir: Path,
+ wall: dict[str, Any],
+ ) -> float:
+ layout_path = layout_dir / f"{object_id}.json"
+ mesh_path = unposed_dir / f"{object_id}.glb"
+ try:
+ layout = json.loads(layout_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RelationGraphError(
+ f"cannot validate wall constraint layout for {object_id}"
+ ) from exc
+ if not isinstance(layout, dict) or not mesh_path.is_file():
+ raise RelationGraphError(
+ f"wall constraint artifacts are missing for {object_id}"
+ )
+ try:
+ mesh = trimesh.load(mesh_path, force="mesh", process=False)
+ vertices = np.asarray(mesh.vertices, dtype=np.float64)
+ transform = np.asarray(
+ _layout_payload_to_matrix(layout), dtype=np.float64
+ )
+ except Exception as exc:
+ raise RelationGraphError(
+ f"cannot load wall constraint geometry for {object_id}"
+ ) from exc
+ if (
+ vertices.ndim != 2
+ or vertices.shape[1] != 3
+ or vertices.shape[0] == 0
+ or not np.isfinite(vertices).all()
+ or transform.shape != (4, 4)
+ or not np.isfinite(transform).all()
+ ):
+ raise RelationGraphError(
+ f"wall constraint geometry is invalid for {object_id}"
+ )
+ world = (
+ np.concatenate(
+ [vertices, np.ones((vertices.shape[0], 1), dtype=np.float64)],
+ axis=1,
+ )
+ @ transform.T
+ )[:, :3]
+ first = np.asarray(wall.get("first_point"), dtype=np.float64).reshape(-1)
+ second = np.asarray(
+ wall.get("second_point"), dtype=np.float64
+ ).reshape(-1)
+ if (
+ first.size != 2
+ or second.size != 2
+ or not np.isfinite(first).all()
+ or not np.isfinite(second).all()
+ ):
+ raise RelationGraphError("wall endpoints must contain two finite values")
+ segment = second - first
+ length_squared = float(segment @ segment)
+ if length_squared <= 1e-12:
+ raise RelationGraphError("wall segment must have non-zero length")
+ points = world[:, [0, 2]]
+ projection = np.clip(
+ ((points - first) @ segment) / length_squared,
+ 0.0,
+ 1.0,
+ )
+ closest = first + projection[:, None] * segment
+ distances = np.linalg.norm(points - closest, axis=1)
+ return float(np.min(distances))
+
+ def _secondary_wall_candidate_audit(
+ self, candidate: LayoutVariantCandidate
+ ) -> dict[str, Any]:
+ relations, _, walls = self._load_relationships_and_objects()
+ view = RelationGraphView.from_relations(relations)
+ walls_by_id = {
+ str(wall.get("id")): wall
+ for wall in walls
+ if isinstance(wall, dict) and wall.get("id")
+ }
+ checks: list[dict[str, Any]] = []
+ for target in sorted(view.secondary_wall_constraints_by_target):
+ if view.primary_parent_by_target.get(target) != candidate.anchor_id:
+ continue
+ for relation in view.secondary_wall_constraints_by_target[target]:
+ parameters = relation.get("parameters", {})
+ wall_id = str(parameters.get("object_anchor") or "")
+ wall = walls_by_id.get(wall_id)
+ if wall is None:
+ raise RelationGraphError(
+ f"secondary wall constraint references missing {wall_id}"
+ )
+ baseline_distance = self._mesh_wall_distance(
+ object_id=target,
+ layout_dir=self.paths.layout_params_dir,
+ unposed_dir=self.paths.unposed_objects_dir,
+ wall=wall,
+ )
+ candidate_distance = self._mesh_wall_distance(
+ object_id=target,
+ layout_dir=candidate.layout_dir,
+ unposed_dir=self.paths.unposed_objects_dir,
+ wall=wall,
+ )
+ allowed_distance = max(0.05, baseline_distance + 0.02)
+ checks.append(
+ {
+ "constraint_type": "secondary_wall_attach",
+ "primary_owner": candidate.anchor_id,
+ "secondary_wall": wall_id,
+ "object_target": target,
+ "baseline_distance": baseline_distance,
+ "candidate_distance": candidate_distance,
+ "allowed_distance": allowed_distance,
+ "valid": candidate_distance <= allowed_distance,
+ }
+ )
+ return {
+ "schema_version": "scenemosaic_secondary_wall_candidate_audit",
+ "candidate": candidate.label,
+ "valid": all(item["valid"] for item in checks),
+ "checks": checks,
+ "secondary_constraints_create_ownership": False,
+ }
+
+ def _local_variant_is_valid(
+ self,
+ candidate: LayoutVariantCandidate,
+ artifacts: SceneArtifacts,
+ generator: Any,
+ validation_root: Path,
+ ) -> bool:
+ if not super()._local_variant_is_valid(
+ candidate, artifacts, generator, validation_root
+ ):
+ return False
+ audit = self._secondary_wall_candidate_audit(candidate)
+ _atomic_write_json(
+ validation_root
+ / f"{candidate.label}_secondary_wall_constraints.json",
+ audit,
+ )
+ if not audit["valid"]:
+ print(
+ "Discarding local variant "
+ f"{candidate.label}: secondary wall constraint failed"
+ )
+ return False
+ return True
+
+
+class IntegratedPipelineOrchestrator(PipelineOrchestrator):
+ """SceneMosaic pipeline using one relation interpretation end to end."""
+
+ def _group_objects_by_anchor(
+ self,
+ relationships: list,
+ objects: list,
+ walls: list,
+ ) -> dict[str, dict[str, Any]]:
+ del objects, walls
+ view = RelationGraphView.from_relations(relationships)
+ self._latest_relation_graph_view = view
+ return view.object_groups()
+
+ @staticmethod
+ def _secondary_wall_constraint_records(
+ object_groups: dict[str, dict[str, Any]],
+ ) -> list[dict[str, Any]]:
+ records: list[dict[str, Any]] = []
+ for primary_owner, group in sorted(object_groups.items()):
+ if not isinstance(group, dict):
+ continue
+ relations = group.get("secondary_wall_constraints", [])
+ if not isinstance(relations, list):
+ raise RelationGraphError(
+ "secondary_wall_constraints must be a relation list"
+ )
+ for relation in relations:
+ if not isinstance(relation, dict):
+ raise RelationGraphError(
+ "secondary wall relation must be a JSON object"
+ )
+ parameters = relation.get("parameters")
+ if not isinstance(parameters, dict):
+ raise RelationGraphError(
+ "secondary wall relation parameters are invalid"
+ )
+ wall_id = str(parameters.get("object_anchor") or "")
+ target = str(parameters.get("object_target") or "")
+ if (
+ not wall_id.startswith("wall")
+ or not target
+ or wall_id == primary_owner
+ ):
+ raise RelationGraphError(
+ "secondary wall relation identity is invalid"
+ )
+ records.append(
+ {
+ "constraint_id": (
+ f"secondary_wall_attach:{primary_owner}:"
+ f"{wall_id}:{target}"
+ ),
+ "constraint_type": "secondary_wall_attach",
+ "source": "relation_graph_view",
+ "description": (
+ f"{target} keeps primary ownership under "
+ f"{primary_owner} while remaining attached to "
+ f"{wall_id}."
+ ),
+ "primary_owner": primary_owner,
+ "secondary_wall": wall_id,
+ "object_target": target,
+ "relation": relation,
+ "anchor_a": {
+ "anchor_id": primary_owner,
+ "related_objects": [target],
+ "layout_note": (
+ f"Keep {target} flush with the room boundary "
+ f"defined by {wall_id}; do not create another "
+ "copy or change its primary owner."
+ ),
+ },
+ "anchor_b": {
+ "anchor_id": wall_id,
+ "related_objects": [target],
+ "layout_note": (
+ f"Preserve the existing {target} contact with "
+ f"{wall_id} as a secondary constraint."
+ ),
+ },
+ }
+ )
+ return sorted(
+ records,
+ key=lambda item: str(item["constraint_id"]),
+ )
+
+ def _analyze_cross_anchor_constraints(
+ self, object_groups: dict, objects: list
+ ) -> list[dict[str, Any]]:
+ payload = super()._analyze_cross_anchor_constraints(
+ object_groups, objects
+ )
+ if not isinstance(payload, list) or not all(
+ isinstance(item, dict) for item in payload
+ ):
+ raise RelationGraphError(
+ "cross-anchor analysis must return a JSON array of objects"
+ )
+ if any(
+ item.get("constraint_type") == "secondary_wall_attach"
+ or item.get("source") == "relation_graph_view"
+ for item in payload
+ ):
+ raise RelationGraphError(
+ "VLM cross-anchor output used reserved relation constraint fields"
+ )
+ combined = [
+ *payload,
+ *self._secondary_wall_constraint_records(object_groups),
+ ]
+ _atomic_write_json(
+ self.paths.outputs_dir / "cross_anchor_constraints.json",
+ combined,
+ )
+ return combined
+
+ def _validate_secondary_wall_constraint_file(
+ self,
+ object_groups: dict[str, dict[str, Any]],
+ ) -> dict[str, Any]:
+ path = self.paths.outputs_dir / "cross_anchor_constraints.json"
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RelationGraphError(
+ "cross-anchor constraint file is missing or invalid"
+ ) from exc
+ if not isinstance(payload, list) or not all(
+ isinstance(item, dict) for item in payload
+ ):
+ raise RelationGraphError(
+ "cross-anchor constraint file must contain a JSON array"
+ )
+ expected = self._secondary_wall_constraint_records(object_groups)
+ actual = [
+ item
+ for item in payload
+ if item.get("constraint_type") == "secondary_wall_attach"
+ or item.get("source") == "relation_graph_view"
+ ]
+ if actual != expected:
+ raise RelationGraphError(
+ "secondary wall generation constraints are missing or stale"
+ )
+ identifiers = [
+ str(item["constraint_id"]) for item in expected
+ ]
+ if len(identifiers) != len(set(identifiers)):
+ raise RelationGraphError(
+ "secondary wall generation constraint IDs are not unique"
+ )
+ return {
+ "schema_version": "scenemosaic_secondary_wall_generation_audit",
+ "valid": True,
+ "path": str(path),
+ "sha256": _sha256(path),
+ "constraint_count": len(expected),
+ "constraint_ids": identifiers,
+ "consumed_by_generation": True,
+ "candidate_validation_required": True,
+ "secondary_constraints_create_ownership": False,
+ }
+
+ def _stage_generate_scene(
+ self,
+ artifacts: SceneArtifacts,
+ *args: Any,
+ **kwargs: Any,
+ ) -> None:
+ relationships, objects, walls = (
+ self._load_relationships_and_objects(suffix="_filtered")
+ )
+ object_groups = self._group_objects_by_anchor(
+ relationships, objects, walls
+ )
+ constraints_path = (
+ self.paths.outputs_dir / "cross_anchor_constraints.json"
+ )
+ if constraints_path.is_file():
+ self._validate_secondary_wall_constraint_file(object_groups)
+ else:
+ self._analyze_cross_anchor_constraints(
+ object_groups, objects
+ )
+ super()._stage_generate_scene(artifacts, *args, **kwargs)
+ audit = self._validate_secondary_wall_constraint_file(object_groups)
+ relation_audit_path = (
+ self.paths.outputs_dir / "relation_graph_audit.json"
+ )
+ relation_audit = json.loads(
+ relation_audit_path.read_text(encoding="utf-8")
+ )
+ if not isinstance(relation_audit, dict):
+ raise RelationGraphError("relation graph audit is invalid")
+ relation_audit["secondary_wall_generation"] = audit
+ _atomic_write_json(relation_audit_path, relation_audit)
+
+ def _stage_optimization(
+ self, artifacts: SceneArtifacts
+ ) -> SceneArtifacts:
+ artifacts = super()._stage_optimization(artifacts)
+ self._preserve_filtered_room_materials()
+ self._write_relation_graph_audit()
+ return artifacts
+
+ def _preserve_filtered_room_materials(self) -> None:
+ source_path = self.paths.outputs_dir / "objects.json"
+ filtered_path = self.paths.outputs_dir / "objects_filtered.json"
+ if not source_path.is_file() or not filtered_path.is_file():
+ return
+ source = json.loads(source_path.read_text(encoding="utf-8"))
+ filtered = json.loads(filtered_path.read_text(encoding="utf-8"))
+ room_materials = (
+ source.get("room_materials")
+ if isinstance(source, dict)
+ else None
+ )
+ if not isinstance(filtered, dict) or not isinstance(
+ room_materials, dict
+ ):
+ raise RelationGraphError(
+ "room_materials cannot be preserved in filtered objects"
+ )
+ filtered["room_materials"] = room_materials
+ _write_json(filtered_path, filtered)
+
+ def _write_relation_graph_audit(self) -> None:
+ relations_path = self.paths.outputs_dir / "relations.json"
+ filtered_path = self.paths.outputs_dir / "relations_filtered.json"
+ objects_filtered_path = self.paths.outputs_dir / "objects_filtered.json"
+ if not relations_path.is_file():
+ raise RelationGraphError("full relations.json is missing")
+ full = json.loads(relations_path.read_text(encoding="utf-8"))
+ full_view = RelationGraphView.from_relations(full)
+ payload: dict[str, Any] = {
+ "full_relations": full_view.audit_summary(),
+ "full_relations_preserved": True,
+ "secondary_wall_constraints_create_ownership": False,
+ }
+ if filtered_path.is_file() or objects_filtered_path.is_file():
+ if not filtered_path.is_file() or not objects_filtered_path.is_file():
+ raise RelationGraphError(
+ "filtered objects and relations must be written together"
+ )
+ filtered = json.loads(
+ filtered_path.read_text(encoding="utf-8")
+ )
+ objects_payload = json.loads(
+ objects_filtered_path.read_text(encoding="utf-8")
+ )
+ objects = (
+ objects_payload.get("objects")
+ if isinstance(objects_payload, dict)
+ else None
+ )
+ if not isinstance(filtered, list) or not isinstance(objects, list):
+ raise RelationGraphError(
+ "filtered relation audit inputs are invalid"
+ )
+ object_ids = {
+ str(item.get("id") or item.get("object_id") or "")
+ for item in objects
+ if isinstance(item, dict)
+ }
+ expected = full_view.filtered_to_object_ids(object_ids)
+ actual = RelationGraphView.from_relations(filtered)
+ if actual.full_relations_list() != expected.full_relations_list():
+ raise RelationGraphError(
+ "filtered relation graph differs from surviving objects"
+ )
+ payload["filtered_relations"] = actual.audit_summary()
+ payload["combination_uses_filtered_relations"] = True
+ _write_json(
+ self.paths.outputs_dir / "relation_graph_audit.json",
+ payload,
+ )
+
+ def run_full_pipeline(self) -> PipelineReport:
+ artifacts = SceneArtifacts(
+ unposed_objects_dir=self.paths.unposed_objects_dir,
+ layout_params_dir=self.paths.layout_params_dir,
+ collision_hulls_dir=self.paths.collision_hulls_dir,
+ composed_scene_mesh=self.paths.composed_scene_path,
+ simulated_layout_dir=self.paths.simulated_layout_dir,
+ )
+ artifacts = self._stage_optimization(artifacts)
+ self._stage_generate_scene(
+ artifacts, max_refine_rounds=self.max_rounds
+ )
+ combined_scene_paths = RelationAwareLayoutCombination(
+ self.paths, self.tool_runner
+ ).run(artifacts)
+ combined_scene_index = (
+ self.paths.outputs_dir
+ / "generated"
+ / "combined_scenes"
+ / "combo_index.json"
+ )
+ if not combined_scene_paths:
+ raise RelationGraphError(
+ "Stage 08 produced no composed GLB scenes"
+ )
+ if (
+ not combined_scene_index.is_file()
+ or any(
+ not path.is_file() or path.stat().st_size == 0
+ for path in combined_scene_paths
+ )
+ ):
+ raise RelationGraphError(
+ "Stage 08 final combination outputs are incomplete"
+ )
+ artifacts = replace(
+ artifacts, composed_scene_mesh=combined_scene_index
+ )
+ report = PipelineReport(
+ stage="composition",
+ artifacts=artifacts,
+ tool_outputs={
+ "combined_scene_index": str(combined_scene_index),
+ "combined_scene_count": str(len(combined_scene_paths)),
+ "wall_padding": "0.0",
+ "per_object_physics_consumed": "false",
+ "combination_uses_filtered_relations": "true",
+ },
+ )
+ self.history.append(report)
+ return report
+
+ def save_report(
+ self,
+ report: PipelineReport,
+ output_path: Path | None = None,
+ ) -> Path:
+ path = super().save_report(report, output_path)
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ audit_path = self.paths.outputs_dir / "relation_graph_audit.json"
+ audit = json.loads(audit_path.read_text(encoding="utf-8"))
+ payload["integration"] = {
+ "relation_graph": audit,
+ "full_relations_preserved": True,
+ "wall_padding": 0.0,
+ "per_object_physics_consumed": False,
+ "combination_uses_filtered_relations": True,
+ }
+ _write_json(path, payload)
+ return path
+
diff --git a/vlm_agent/evolution/layout_combination.py b/vlm_agent/evolution/layout_combination.py
new file mode 100644
index 0000000..941576a
--- /dev/null
+++ b/vlm_agent/evolution/layout_combination.py
@@ -0,0 +1,1099 @@
+from __future__ import annotations
+
+import copy
+import itertools
+import json
+import math
+import random
+import re
+import shutil
+import time
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from .paths import ProjectPaths
+from .scene_generator import SceneGenerator
+from .tool_runner import ToolRunner
+from .types import SceneArtifacts
+
+
+def _sanitize_path_component(value: str) -> str:
+ sanitized = re.sub(r"[^A-Za-z0-9_.=-]+", "_", value).strip("_")
+ return sanitized or "item"
+
+
+def _flatten_singleton_list(value: object) -> object:
+ if isinstance(value, list) and len(value) == 1 and isinstance(value[0], list):
+ return value[0]
+ return value
+
+
+def _parse_vector(value: object, *, expected_length: int) -> list[float]:
+ flat_value = _flatten_singleton_list(value)
+ array = np.asarray(flat_value, dtype=np.float32).reshape(-1)
+ if array.size != expected_length:
+ raise ValueError(f"expected {expected_length} values, got {array.size}")
+ return [float(item) for item in array.tolist()]
+
+
+def _parse_scale(value: object) -> list[float]:
+ flat_value = _flatten_singleton_list(value)
+ array = np.asarray(flat_value, dtype=np.float32).reshape(-1)
+ if array.size == 1:
+ return [float(array[0])] * 3
+ if array.size == 3:
+ return [float(item) for item in array.tolist()]
+ raise ValueError(f"scale must have 1 or 3 values, got {array.size}")
+
+
+def _quaternion_to_matrix_numpy(quaternion: object) -> np.ndarray:
+ q = np.asarray(_parse_vector(quaternion, expected_length=4), dtype=np.float32)
+ norm = float(np.linalg.norm(q))
+ if norm <= 1e-8:
+ raise ValueError("quaternion norm is zero")
+ w, x, y, z = q / norm
+ return np.array(
+ [
+ [1 - 2 * (y * y + z * z), 2 * (x * y - z * w), 2 * (x * z + y * w)],
+ [2 * (x * y + z * w), 1 - 2 * (x * x + z * z), 2 * (y * z - x * w)],
+ [2 * (x * z - y * w), 2 * (y * z + x * w), 1 - 2 * (x * x + y * y)],
+ ],
+ dtype=np.float32,
+ )
+
+
+def _matrix_to_quaternion(matrix: np.ndarray) -> list[float]:
+ m = np.asarray(matrix, dtype=np.float32).reshape(3, 3)
+ trace = float(m[0, 0] + m[1, 1] + m[2, 2])
+ if trace > 0.0:
+ s = math.sqrt(trace + 1.0) * 2.0
+ w = 0.25 * s
+ x = (m[2, 1] - m[1, 2]) / s
+ y = (m[0, 2] - m[2, 0]) / s
+ z = (m[1, 0] - m[0, 1]) / s
+ elif m[0, 0] > m[1, 1] and m[0, 0] > m[2, 2]:
+ s = math.sqrt(1.0 + m[0, 0] - m[1, 1] - m[2, 2]) * 2.0
+ w = (m[2, 1] - m[1, 2]) / s
+ x = 0.25 * s
+ y = (m[0, 1] + m[1, 0]) / s
+ z = (m[0, 2] + m[2, 0]) / s
+ elif m[1, 1] > m[2, 2]:
+ s = math.sqrt(1.0 + m[1, 1] - m[0, 0] - m[2, 2]) * 2.0
+ w = (m[0, 2] - m[2, 0]) / s
+ x = (m[0, 1] + m[1, 0]) / s
+ y = 0.25 * s
+ z = (m[1, 2] + m[2, 1]) / s
+ else:
+ s = math.sqrt(1.0 + m[2, 2] - m[0, 0] - m[1, 1]) * 2.0
+ w = (m[1, 0] - m[0, 1]) / s
+ x = (m[0, 2] + m[2, 0]) / s
+ y = (m[1, 2] + m[2, 1]) / s
+ z = 0.25 * s
+ return [float(w), float(x), float(y), float(z)]
+
+
+def _layout_payload_to_matrix(layout: dict) -> np.ndarray:
+ translation = np.asarray(
+ _parse_vector(layout.get("translation", [0.0, 0.0, 0.0]), expected_length=3),
+ dtype=np.float32,
+ )
+ rotation = _quaternion_to_matrix_numpy(layout.get("rotation", [1.0, 0.0, 0.0, 0.0]))
+ scale = np.asarray(_parse_scale(layout.get("scale", [1.0, 1.0, 1.0])), dtype=np.float32)
+
+ matrix = np.eye(4, dtype=np.float32)
+ matrix[:3, :3] = rotation @ np.diag(scale)
+ matrix[:3, 3] = translation
+ return matrix
+
+
+def _matrix_to_layout_payload(matrix: np.ndarray, template: dict | None = None) -> dict:
+ affine = np.asarray(matrix, dtype=np.float32).reshape(4, 4)
+ linear = affine[:3, :3]
+ translation = affine[:3, 3]
+
+ scale = np.linalg.norm(linear, axis=0)
+ safe_scale = np.where(scale < 1e-8, 1.0, scale)
+ rotation = linear @ np.diag(1.0 / safe_scale)
+
+ u, _, vh = np.linalg.svd(rotation)
+ rotation = u @ vh
+ if np.linalg.det(rotation) < 0:
+ u[:, -1] *= -1
+ rotation = u @ vh
+
+ payload = {
+ "translation": [[float(translation[0]), float(translation[1]), float(translation[2])]],
+ "rotation": [_matrix_to_quaternion(rotation)],
+ "scale": [[float(scale[0]), float(scale[1]), float(scale[2])]],
+ }
+ if template:
+ for key, value in template.items():
+ if key not in payload:
+ payload[key] = copy.deepcopy(value)
+ return payload
+
+
+@dataclass(frozen=True)
+class LayoutVariantCandidate:
+ anchor_id: str
+ group_dir: Path
+ variant_dir: Path
+ layout_id: str
+ layout_dir: Path
+ mapping_path: Path
+
+ @property
+ def label(self) -> str:
+ return f"{_sanitize_path_component(self.anchor_id)}={_sanitize_path_component(self.layout_id)}"
+
+
+def discover_layout_variants(generated_root: Path) -> dict[str, list[LayoutVariantCandidate]]:
+ grouped_variants: dict[str, list[LayoutVariantCandidate]] = {}
+ if not generated_root.exists():
+ return grouped_variants
+
+ for group_dir in sorted(p for p in generated_root.glob("group_*") if p.is_dir()):
+ anchor_id = group_dir.name.removeprefix("group_")
+ candidates: list[LayoutVariantCandidate] = []
+
+ for variant_dir in sorted(p for p in group_dir.glob("layout_*") if p.is_dir()):
+ layout_id = variant_dir.name
+ simulated_layout_dir = variant_dir / "layout_params_simulated"
+ layout_params_dir = variant_dir / "layout_params"
+ layout_dir = simulated_layout_dir if simulated_layout_dir.exists() else layout_params_dir
+ if not layout_dir.exists():
+ continue
+
+ if layout_id.endswith("_org"):
+ # For original layouts
+ mapping_path = group_dir / f"grouped_scene_{anchor_id}.json"
+ if not mapping_path.exists():
+ continue
+ else:
+ mapping_path = variant_dir / f"{layout_id}_render.json"
+ if not mapping_path.exists():
+ continue
+
+ candidates.append(
+ LayoutVariantCandidate(
+ anchor_id=anchor_id,
+ group_dir=group_dir,
+ variant_dir=variant_dir,
+ layout_id=layout_id,
+ layout_dir=layout_dir,
+ mapping_path=mapping_path,
+ )
+ )
+
+ if candidates:
+ grouped_variants[anchor_id] = candidates
+
+ return grouped_variants
+
+
+def build_combo_name(variants: tuple[LayoutVariantCandidate, ...] | list[LayoutVariantCandidate]) -> str:
+ parts: list[str] = []
+ for variant in variants:
+ anchor = _sanitize_path_component(variant.anchor_id)
+ layout = _sanitize_path_component(variant.layout_id).removeprefix("layout_")
+ anchor = anchor.replace("_", "")
+ if layout.isdigit():
+ layout = layout.zfill(3)
+ parts.append(f"{anchor}@{layout}")
+ return "__".join(parts)
+
+
+class LayoutCombination:
+ def __init__(self, paths: ProjectPaths, tool_runner: ToolRunner):
+ self.paths = paths
+ self.tool_runner = tool_runner
+
+ @staticmethod
+ def _load_json(path: Path) -> dict:
+ with path.open("r", encoding="utf-8") as f:
+ payload = json.load(f)
+ return payload if isinstance(payload, dict) else {}
+
+ def _load_relationships_and_objects(self) -> tuple[list[dict], list[dict], list[dict]]:
+ relations_path = self.paths.outputs_dir / "relations.json"
+ objects_path = self.paths.outputs_dir / "objects.json"
+
+ relationships: list[dict] = []
+ if relations_path.exists():
+ with relations_path.open("r", encoding="utf-8") as f:
+ payload = json.load(f)
+ if isinstance(payload, list):
+ relationships = [item for item in payload if isinstance(item, dict)]
+
+ objects: list[dict] = []
+ walls: list[dict] = []
+ if objects_path.exists():
+ with objects_path.open("r", encoding="utf-8") as f:
+ payload = json.load(f)
+ if isinstance(payload, dict):
+ raw_objects = payload.get("objects", [])
+ raw_walls = payload.get("walls", [])
+ if isinstance(raw_objects, list):
+ objects = [item for item in raw_objects if isinstance(item, dict)]
+ if isinstance(raw_walls, list):
+ walls = [item for item in raw_walls if isinstance(item, dict)]
+
+ return relationships, objects, walls
+
+ @staticmethod
+ def _has_out_of_bounds(summary: dict | None) -> bool:
+ if not isinstance(summary, dict):
+ return False
+ entries = summary.get("out_of_bounds_objects", [])
+ return isinstance(entries, list) and bool(entries)
+
+ @staticmethod
+ def _is_root_anchor(anchor_id: str) -> bool:
+ return anchor_id == "floor" or anchor_id == "ceiling" or anchor_id.startswith("wall")
+
+ @staticmethod
+ def _root_sort_key(anchor_id: str) -> tuple[int, str]:
+ if anchor_id == "floor":
+ return (0, "")
+ if anchor_id == "ceiling":
+ return (1, "")
+ if anchor_id.startswith("wall"):
+ return (2, anchor_id)
+ return (3, anchor_id)
+
+ def _local_variant_is_valid(
+ self,
+ candidate: LayoutVariantCandidate,
+ artifacts: SceneArtifacts,
+ generator: SceneGenerator,
+ validation_root: Path,
+ ) -> bool:
+ is_org = candidate.layout_id.endswith("_org")
+ if artifacts.collision_hulls_dir is None:
+ return is_org
+
+ validation_root.mkdir(parents=True, exist_ok=True)
+ report_path = validation_root / f"{candidate.label}_local_collision_report.json"
+ collision_result = self.tool_runner.check_object_collisions(
+ layout_dir=candidate.layout_dir,
+ hull_dir=artifacts.collision_hulls_dir,
+ output=report_path,
+ )
+ if collision_result.returncode != 0 or not report_path.exists():
+ msg = f"Collision check failed for {candidate.variant_dir}"
+ if not is_org:
+ print(f"Discarding local variant {candidate.label}: {msg}")
+ return False
+
+ collision_report = self._load_json(report_path)
+ collisions = collision_report.get("collisions", [])
+ if isinstance(collisions, list) and collisions:
+ msg = f"collisions detected ({len(collisions)})"
+ if not is_org:
+ print(f"Discarding local variant {candidate.label}: {msg}")
+ return False
+
+ out_of_bounds_summary = generator.summarize_out_of_bounds_objects(
+ candidate.mapping_path,
+ anchor_id=candidate.anchor_id,
+ )
+ if self._has_out_of_bounds(out_of_bounds_summary):
+ msg = "out-of-bounds objects detected"
+ if not is_org:
+ print(f"Discarding local variant {candidate.label}: {msg}")
+ return False
+
+ return True
+
+ @staticmethod
+ def _load_relationship_graph(relations: list[dict]) -> tuple[dict[str, set[str]], dict[str, set[str]]]:
+ parents_by_child: dict[str, set[str]] = {}
+ children_by_parent: dict[str, set[str]] = {}
+ for relation in relations:
+ if not isinstance(relation, dict) or relation.get("relation") not in ["attach", "contain"]:
+ continue
+ parameters = relation.get("parameters", {})
+ if not isinstance(parameters, dict):
+ continue
+ parent_id = parameters.get("object_anchor")
+ child_id = parameters.get("object_target")
+ if not parent_id or not child_id:
+ continue
+ parent_id = str(parent_id)
+ child_id = str(child_id)
+ parents_by_child.setdefault(child_id, set()).add(parent_id)
+ children_by_parent.setdefault(parent_id, set()).add(child_id)
+ return parents_by_child, children_by_parent
+
+ def _choose_parent_by_child(self, parents_by_child: dict[str, set[str]]) -> dict[str, str]:
+ chosen_parent_by_child: dict[str, str] = {}
+ for child_id, parents in parents_by_child.items():
+ if not parents:
+ continue
+ if "floor" in parents:
+ chosen_parent_by_child[child_id] = "floor"
+ continue
+ chosen_parent_by_child[child_id] = random.choice(sorted(parents))
+ return chosen_parent_by_child
+
+ def _collect_root_nodes(
+ self,
+ grouped_variants: dict[str, list[LayoutVariantCandidate]],
+ chosen_parent_by_child: dict[str, str],
+ ) -> list[str]:
+ root_nodes = [
+ anchor_id
+ for anchor_id in grouped_variants
+ if self._is_root_anchor(anchor_id) or anchor_id not in chosen_parent_by_child
+ ]
+ root_nodes.sort(key=self._root_sort_key)
+ if not root_nodes:
+ return []
+ root_nodes = list(dict.fromkeys(root_nodes))
+ root_nodes.sort(key=self._root_sort_key)
+ return root_nodes
+
+ @staticmethod
+ def _merge_layout_maps(base_layouts: dict[str, dict], extra_layouts: dict[str, dict]) -> dict[str, dict]:
+ merged = copy.deepcopy(base_layouts)
+ for object_id, payload in extra_layouts.items():
+ existing = merged.get(object_id)
+ if existing is not None and existing != payload:
+ raise ValueError(f"conflicting layout for object '{object_id}'")
+ merged[object_id] = payload
+ return merged
+
+ @staticmethod
+ def _write_layout_tree(layouts: dict[str, dict], target_layout_dir: Path) -> None:
+ target_layout_dir.mkdir(parents=True, exist_ok=True)
+ for object_id in sorted(layouts.keys()):
+ output_path = target_layout_dir / f"{object_id}.json"
+ with output_path.open("w", encoding="utf-8") as f:
+ json.dump(layouts[object_id], f, ensure_ascii=False, indent=2)
+
+ def _fill_missing_root_layouts_from_simulated(self, combo_layout_dir: Path) -> None:
+ """Fill missing wall/floor/ceiling layouts from outputs/layout_params_simulated."""
+ simulated_layout_dir = self.paths.outputs_dir / "layout_params_simulated"
+ if not simulated_layout_dir.exists():
+ return
+
+ # Check for missing wall_*, floor, ceiling files
+ for pattern in ["wall_*.json", "floor.json", "ceiling.json"]:
+ for simulated_file in simulated_layout_dir.glob(pattern):
+ target_file = combo_layout_dir / simulated_file.name
+ if not target_file.exists():
+ shutil.copy2(simulated_file, target_file)
+
+ def _compose_variant_into_scene(
+ self,
+ scene_layouts: dict[str, dict],
+ candidate: LayoutVariantCandidate,
+ ) -> bool:
+ local_anchor_path = candidate.layout_dir / f"{candidate.anchor_id}.json"
+ if not local_anchor_path.exists():
+ print(f"Skipping {candidate.label}: missing local anchor layout {local_anchor_path}")
+ return False
+
+ if self._is_root_anchor(candidate.anchor_id):
+ for layout_file in sorted(candidate.layout_dir.glob("*.json")):
+ scene_layouts[layout_file.stem] = self._load_json(layout_file)
+ return True
+
+ parent_payload = scene_layouts.get(candidate.anchor_id)
+ if parent_payload is None:
+ print(
+ f"Skipping {candidate.label}: parent anchor '{candidate.anchor_id}' has not been materialized yet"
+ )
+ return False
+
+ try:
+ parent_matrix = _layout_payload_to_matrix(parent_payload)
+ local_anchor_payload = self._load_json(local_anchor_path)
+ local_anchor_matrix = _layout_payload_to_matrix(local_anchor_payload)
+ transform_matrix = parent_matrix @ np.linalg.inv(local_anchor_matrix)
+ except np.linalg.LinAlgError:
+ print(f"Skipping {candidate.label}: anchor transform is not invertible")
+ return False
+
+ for layout_file in sorted(candidate.layout_dir.glob("*.json")):
+ object_id = layout_file.stem
+ if object_id == candidate.anchor_id:
+ continue
+ local_payload = self._load_json(layout_file)
+ composed_matrix = transform_matrix @ _layout_payload_to_matrix(local_payload)
+ scene_layouts[object_id] = _matrix_to_layout_payload(composed_matrix, local_payload)
+
+ return True
+
+ def _expand_anchor(
+ self,
+ anchor_id: str,
+ base_scene_layouts: dict[str, dict],
+ grouped_variants: dict[str, list[LayoutVariantCandidate]],
+ children_by_parent: dict[str, list[str]],
+ ) -> list[tuple[dict[str, dict], list[LayoutVariantCandidate]]]:
+ candidates = grouped_variants.get(anchor_id, [])
+ if not candidates:
+ return []
+
+ results: list[tuple[dict[str, dict], list[LayoutVariantCandidate]]] = []
+ child_ids = sorted(children_by_parent.get(anchor_id, []))
+
+ for candidate in candidates:
+ composed_scene_layouts = copy.deepcopy(base_scene_layouts)
+ if not self._compose_variant_into_scene(composed_scene_layouts, candidate):
+ continue
+
+ branch_variants = [candidate]
+ if not child_ids:
+ results.append((composed_scene_layouts, branch_variants))
+ continue
+
+ child_result_groups: list[list[tuple[dict[str, dict], list[LayoutVariantCandidate]]]] = []
+ for child_id in child_ids:
+ child_results = self._expand_anchor(
+ child_id,
+ composed_scene_layouts,
+ grouped_variants,
+ children_by_parent,
+ )
+ if child_results:
+ child_result_groups.append(child_results)
+
+ if not child_result_groups:
+ results.append((composed_scene_layouts, branch_variants))
+ continue
+
+ for child_combo in itertools.product(*child_result_groups):
+ merged_layouts = copy.deepcopy(composed_scene_layouts)
+ merged_variants = list(branch_variants)
+ valid_combo = True
+ for child_layouts, child_variants in child_combo:
+ try:
+ merged_layouts = self._merge_layout_maps(merged_layouts, child_layouts)
+ except ValueError:
+ valid_combo = False
+ break
+ merged_variants.extend(child_variants)
+ if valid_combo:
+ results.append((merged_layouts, merged_variants))
+
+ return results
+
+ def _write_combo_manifest(
+ self,
+ output_path: Path,
+ *,
+ combo_name: str,
+ variants: tuple[LayoutVariantCandidate, ...],
+ layout_dir: Path,
+ collision_report_path: Path,
+ novelty_metrics: dict | None = None,
+ ) -> None:
+ payload = {
+ "combo_name": combo_name,
+ "selected_variants": [
+ {
+ "anchor_id": variant.anchor_id,
+ "layout_id": variant.layout_id,
+ "group_dir": str(variant.group_dir),
+ "variant_dir": str(variant.variant_dir),
+ "layout_dir": str(variant.layout_dir),
+ "mapping_path": str(variant.mapping_path),
+ }
+ for variant in variants
+ ],
+ "layout_dir": str(layout_dir),
+ "collision_report_path": str(collision_report_path),
+ }
+ if novelty_metrics is not None:
+ payload["novelty_metrics"] = novelty_metrics
+ with output_path.open("w", encoding="utf-8") as f:
+ json.dump(payload, f, ensure_ascii=False, indent=2)
+
+ @staticmethod
+ def _is_org_variant(variant: LayoutVariantCandidate) -> bool:
+ """Check if variant is the org variant."""
+ return variant.layout_id.endswith("_org")
+
+ def _compose_selected_combo(
+ self,
+ combo_dir: Path,
+ combo_name: str,
+ combo_layout_dir: Path,
+ unposed_objects_dir: Path,
+ ) -> Path | None:
+ """Generate GLB file for a selected combo. Returns glb_path on success, None on failure."""
+ glb_path = combo_dir / f"{combo_name}.glb"
+ compose_result = self.tool_runner.compose_scene(
+ combo_layout_dir,
+ unposed_objects_dir,
+ glb_path,
+ )
+ if compose_result.returncode != 0:
+ print(f"Failed to compose GLB for combo {combo_name}.")
+ shutil.rmtree(combo_dir, ignore_errors=True)
+ return None
+ print(f"Created combined scene GLB for combo {combo_name} at {glb_path}")
+ return glb_path
+
+ def _load_layout_objects(self, layout_dir: Path) -> list[tuple[np.ndarray, np.ndarray, np.ndarray]]:
+ """Load all object positions, rotations (as matrices), and quaternions from layout_params directory."""
+ objects = []
+ for obj_file in sorted(layout_dir.glob("*.json")):
+ layout_data = self._load_json(obj_file)
+ translation = np.asarray(
+ _parse_vector(layout_data.get("translation", [0.0, 0.0, 0.0]), expected_length=3),
+ dtype=np.float32,
+ )
+ # Parse quaternion for rotation matrix
+ rotation = _quaternion_to_matrix_numpy(layout_data.get("rotation", [1.0, 0.0, 0.0, 0.0]))
+ # Also keep raw quaternion for novelty calculation
+ quaternion = np.asarray(
+ _parse_vector(layout_data.get("rotation", [1.0, 0.0, 0.0, 0.0]), expected_length=4),
+ dtype=np.float32,
+ )
+ # Normalize quaternion
+ quaternion = quaternion / np.linalg.norm(quaternion)
+ objects.append((translation, rotation, quaternion))
+ return objects
+
+ def _compute_novelty_metrics(
+ self,
+ combo_layout_dir: Path,
+ org_layout_dir: Path,
+ wp: float = 0.4,
+ wu: float = 0.4,
+ wr: float = 0.2,
+ ) -> dict:
+ """
+ Compute enhanced novelty metrics between a combo layout and the pure org layout.
+
+ Returns a dict with:
+ - D_pos_raw: raw directed position difference
+ - D_dist_raw: raw undirected distance difference
+ - D_rot_raw: raw rotation difference
+ - D_pos_normalized: normalized directed position difference
+ - D_dist_normalized: normalized undirected distance difference
+ - D_rot_normalized: normalized rotation difference
+ - novelty: final novelty score
+ """
+ # Load objects from both layouts
+ combo_objects = self._load_layout_objects(combo_layout_dir)
+ org_objects = self._load_layout_objects(org_layout_dir)
+
+ # Match objects by file name
+ combo_files = {f.stem for f in combo_layout_dir.glob("*.json")}
+ org_files = {f.stem for f in org_layout_dir.glob("*.json")}
+ common_files = sorted(combo_files & org_files)
+
+ if not common_files:
+ return {
+ "D_pos_raw": 0.0,
+ "D_rot_raw": 0.0,
+ "D_pos_normalized": 0.0,
+ "D_rot_normalized": 0.0,
+ "novelty": 0.0,
+ "error": "no common objects",
+ }
+
+ # Build object dictionaries
+ combo_dict = {f.stem: i for i, f in enumerate(sorted(combo_layout_dir.glob("*.json")))}
+ org_dict = {f.stem: i for i, f in enumerate(sorted(org_layout_dir.glob("*.json")))}
+
+ # Part 1: Position and distance differences
+ N = len(common_files)
+ if N < 2:
+ D_pos_raw = 0.0
+ D_dist_raw = 0.0
+ s_pos = 1.0
+ s_dist = 1.0
+ else:
+ # Compute pairwise differences over all ordered pairs i != j
+ # (in the local-coordinate enhanced version, (i,j) and (j,i) are NOT symmetric
+ # because each displacement is projected into object i's own local frame)
+ total_pos_diff = 0.0
+ total_pos_scale = 0.0
+ total_dist_diff = 0.0
+ total_dist_scale = 0.0
+ count = 0
+ for i, obj_i in enumerate(common_files):
+ for j, obj_j in enumerate(common_files):
+ if i == j:
+ continue
+ # Get positions
+ combo_i = combo_objects[combo_dict[obj_i]][0]
+ combo_j = combo_objects[combo_dict[obj_j]][0]
+ org_i = org_objects[org_dict[obj_i]][0]
+ org_j = org_objects[org_dict[obj_j]][0]
+
+ # Global displacement vectors
+ v_ij_org = org_j - org_i
+ v_ij_combo = combo_j - combo_i
+
+ # Local coordinate system (enhanced version)
+ # Project to object i's local frame
+ R_i_org = org_objects[org_dict[obj_i]][1]
+ v_ij_org_local = R_i_org.T @ v_ij_org
+
+ R_i_combo = combo_objects[combo_dict[obj_i]][1]
+ v_ij_combo_local = R_i_combo.T @ v_ij_combo
+
+ # Difference in local coordinates
+ pos_diff = np.linalg.norm(v_ij_combo_local - v_ij_org_local)
+ total_pos_diff += pos_diff ** 2
+
+ # Undirected distance difference
+ dist_diff = abs(np.linalg.norm(v_ij_combo) - np.linalg.norm(v_ij_org))
+ total_dist_diff += dist_diff ** 2
+
+ # Scale factor (original layout distances)
+ total_pos_scale += np.linalg.norm(v_ij_org) ** 2
+ total_dist_scale += np.linalg.norm(v_ij_org) ** 2
+ count += 1
+
+ D_pos_raw = math.sqrt(total_pos_diff / count) if count > 0 else 0.0
+ D_dist_raw = math.sqrt(total_dist_diff / count) if count > 0 else 0.0
+ s_pos = math.sqrt(total_pos_scale / count) if count > 0 else 1.0
+ s_dist = math.sqrt(total_dist_scale / count) if count > 0 else 1.0
+
+ # Normalize position differences
+ D_pos_normalized = D_pos_raw / s_pos if s_pos > 1e-8 else 0.0
+ D_dist_normalized = D_dist_raw / s_dist if s_dist > 1e-8 else 0.0
+
+ # Part 2: Rotation difference (using quaternion dot product as per spec)
+ theta_sum = 0.0
+ for obj_id in common_files:
+ combo_idx = combo_dict[obj_id]
+ org_idx = org_dict[obj_id]
+
+ # Extract quaternions (index 2 in the tuple)
+ q_combo = combo_objects[combo_idx][2]
+ q_org = org_objects[org_idx][2]
+
+ # Compute quaternion dot product: θ = 2 * arccos(|q_combo · q_org|)
+ dot_product = abs(np.dot(q_combo, q_org))
+ dot_product = np.clip(dot_product, 0.0, 1.0)
+ theta = 2.0 * math.acos(dot_product) # in [0, pi]
+ theta_sum += theta
+
+ D_rot_raw = theta_sum / N if N > 0 else 0.0
+ D_rot_normalized = D_rot_raw / math.pi
+
+ # Final novelty score
+ novelty = wp * D_pos_normalized + wu * D_dist_normalized + wr * D_rot_normalized
+
+ return {
+ "D_pos_raw": float(D_pos_raw),
+ "D_dist_raw": float(D_dist_raw),
+ "D_rot_raw": float(D_rot_raw),
+ "D_pos_normalized": float(D_pos_normalized),
+ "D_dist_normalized": float(D_dist_normalized),
+ "D_rot_normalized": float(D_rot_normalized),
+ "novelty": float(novelty),
+ "common_objects_count": N,
+ "weights": {"wp": wp, "wu": wu, "wr": wr},
+ }
+
+ def _select_representative_scenes(
+ self,
+ pending_manifests: list[tuple[Path, dict]],
+ *,
+ org_layout_dir: Path,
+ k: int = 6,
+ alpha: float = 0.1,
+ wp: float = 0.4,
+ wu: float = 0.4,
+ wr: float = 0.2,
+ ) -> list[dict]:
+ """
+ Select diverse representative scenes from candidate combos.
+
+ Algorithm:
+ 0. Seed — the pure-org combo is directly placed into representatives (rank 1).
+ 1. Compute novelty-to-org for every non-org candidate.
+ 2. Iterative Max-Min greedy selection with **dynamic per-round filtering**:
+ In each round, candidates whose min-novelty to *any* already-selected
+ representative is below the current round's ``max_min_distance * alpha`` are
+ removed, then the candidate maximising the minimum distance to all
+ representatives is picked.
+
+ Pairwise novelty scores are cached so each pair of layout dirs is computed
+ at most once.
+
+ Returns the list of selected manifest dicts (with an added ``representative_rank`` key).
+ """
+ # ---- Pairwise novelty cache (layout_dir_pair -> score) -----------------
+ _pair_cache: dict[frozenset, float] = {}
+
+ def _pairwise_novelty(layout_a: Path, layout_b: Path) -> float:
+ key = frozenset((str(layout_a), str(layout_b)))
+ cached = _pair_cache.get(key)
+ if cached is not None:
+ return cached
+ score = self._compute_novelty_metrics(layout_a, layout_b, wp=wp, wu=wu, wr=wr)["novelty"]
+ _pair_cache[key] = score
+ return score
+
+ def _min_novelty_to_reps(
+ layout: Path, reps: list[tuple[dict, float]]
+ ) -> float:
+ """Minimum novelty from *layout* to any representative."""
+ return min(
+ _pairwise_novelty(layout, rep_data["layout_dir"])
+ for rep_data, _ in reps
+ )
+
+ # ---- Step 0: Seed — place pure-org combo as rank 1 ---------------------
+ org_manifest_data: dict | None = None
+ for _manifest_path, manifest_data in pending_manifests:
+ if self._is_pure_org_variant(manifest_data["variants"]):
+ org_manifest_data = manifest_data
+ break
+
+ representatives: list[tuple[dict, float]] = []
+ if org_manifest_data is not None:
+ org_manifest_data["representative_rank"] = 1
+ representatives.append((org_manifest_data, 0.0))
+ print(f" Seed (rank 1): {org_manifest_data['combo_name']} [pure org baseline]")
+ else:
+ print("Warning: no pure org combo found; proceeding without org seed.")
+
+ # ---- Step 1: Compute novelty-to-org for non-org candidates -------------
+ candidates: list[tuple[dict, float]] = [] # (manifest_data, novelty_to_org)
+ for _manifest_path, manifest_data in pending_manifests:
+ if self._is_pure_org_variant(manifest_data["variants"]):
+ continue
+ score = self._compute_novelty_metrics(
+ manifest_data["layout_dir"],
+ org_layout_dir,
+ wp=wp,
+ wu=wu,
+ wr=wr,
+ )["novelty"]
+ candidates.append((manifest_data, score))
+
+ if not candidates:
+ print("No non-org candidates available for scene selection.")
+ return [data for data, _ in representatives]
+
+ # ---- Step 2: Iterative Max-Min greedy with per-round filtering ---------
+ rank = len(representatives) + 1 # next rank number
+
+ while len(representatives) < k and candidates:
+ # Pick the candidate that maximises min-distance to all representatives.
+ best_candidate: tuple[dict, float] | None = None
+ best_idx = -1
+ max_min_distance = -1.0
+
+ for idx, (cand_data, _cand_score) in enumerate(candidates):
+ min_dist = _min_novelty_to_reps(cand_data["layout_dir"], representatives)
+ if min_dist > max_min_distance:
+ max_min_distance = min_dist
+ best_candidate = (cand_data, _cand_score)
+ best_idx = idx
+
+ cand_data, _ = best_candidate # type: ignore[misc]
+ cand_data["representative_rank"] = rank
+ representatives.append(best_candidate) # type: ignore[arg-type]
+ candidates.pop(best_idx)
+ print(
+ f" Selected rank {rank}: {cand_data['combo_name']} "
+ f"(min_dist_to_reps={max_min_distance:.4f})"
+ )
+ rank += 1
+
+ if not candidates:
+ break
+
+ filter_threshold = max_min_distance * alpha
+ before_count = len(candidates)
+ candidates = [
+ (cand_data, cand_score)
+ for cand_data, cand_score in candidates
+ if _min_novelty_to_reps(cand_data["layout_dir"], representatives) >= filter_threshold
+ ]
+ removed = before_count - len(candidates)
+ if removed > 0:
+ print(
+ f" Round {rank - 1}: filtered out {removed} candidate(s) "
+ f"(min-novelty < {filter_threshold:.4f}, alpha={alpha})"
+ )
+
+ print(f"Scene selection complete: {len(representatives)} representative scenes chosen.")
+ return [data for data, _ in representatives]
+
+
+ @staticmethod
+ def _is_pure_org_variant(variants: list[LayoutVariantCandidate]) -> bool:
+ """Check if all variants are org variants."""
+ return all(v.layout_id.endswith("_org") for v in variants)
+
+ def _write_combo_index(self, output_path: Path, combinations: list[dict]) -> None:
+ payload = {
+ "combined_scene_count": len(combinations),
+ "combined_scenes": combinations,
+ }
+ with output_path.open("w", encoding="utf-8") as f:
+ json.dump(payload, f, ensure_ascii=False, indent=2)
+
+ def run(self, artifacts: SceneArtifacts) -> list[Path]:
+ run_t0 = time.time()
+ if artifacts.unposed_objects_dir is None or artifacts.collision_hulls_dir is None:
+ print("Skipping layout combination stage: missing unposed objects or collision hulls.")
+ return []
+
+ generated_root = self.paths.outputs_dir / "generated"
+ t0 = time.time()
+ grouped_variants = discover_layout_variants(generated_root)
+ print(f"[Timing] Discover layout variants: {time.time() - t0:.2f}s ({sum(len(v) for v in grouped_variants.values())} variants in {len(grouped_variants)} groups)")
+ if not grouped_variants:
+ print("Skipping layout combination stage: no generated layout variants found.")
+ return []
+
+ generator = SceneGenerator(artifacts)
+ validation_root = generated_root / "combined_scenes" / "_validation"
+ valid_variants_by_group: dict[str, list[LayoutVariantCandidate]] = {}
+
+ t0 = time.time()
+ for anchor_id in sorted(grouped_variants.keys()):
+ candidates = grouped_variants[anchor_id]
+ valid_candidates: list[LayoutVariantCandidate] = []
+ for candidate in candidates:
+ if self._local_variant_is_valid(candidate, artifacts, generator, validation_root):
+ valid_candidates.append(candidate)
+ if valid_candidates:
+ valid_variants_by_group[anchor_id] = valid_candidates
+ else:
+ print(f"No valid local variants remain for anchor '{anchor_id}'.")
+ print(f"[Timing] Local variant validation: {time.time() - t0:.2f}s ({sum(len(v) for v in valid_variants_by_group.values())} valid)")
+
+ combined_root = generated_root / "combined_scenes"
+ combined_root.mkdir(parents=True, exist_ok=True)
+
+ created_glbs: list[Path] = []
+ combo_index_entries: list[dict] = []
+ # Deferred manifest records: (manifest_path, kwargs_for_manifest)
+ pending_manifests: list[tuple[Path, dict]] = []
+
+ t0 = time.time()
+ relations, _, _ = self._load_relationships_and_objects()
+ parents_by_child, children_by_parent = self._load_relationship_graph(relations)
+ chosen_parent_by_child = self._choose_parent_by_child(parents_by_child)
+ root_nodes = self._collect_root_nodes(valid_variants_by_group, chosen_parent_by_child)
+ print(f"[Timing] Load relations & build graph: {time.time() - t0:.2f}s ({len(root_nodes)} root nodes)")
+
+ if not root_nodes:
+ print("Skipping layout combination stage: no root nodes found in relations.")
+ return []
+
+ root_result_groups: list[list[tuple[dict[str, dict], list[LayoutVariantCandidate]]]] = []
+ t0 = time.time()
+ for root_id in root_nodes:
+ root_results = self._expand_anchor(root_id, {}, valid_variants_by_group, children_by_parent)
+ if not root_results:
+ print(f"No valid local variants remain for root '{root_id}'.")
+ continue
+ root_result_groups.append(root_results)
+ total_combos = sum(len(g) for g in root_result_groups)
+ print(f"[Timing] Expand anchor (combo generation): {time.time() - t0:.2f}s ({total_combos} combos)")
+
+ if not root_result_groups:
+ print("Skipping combination stage: no valid root branches remain.")
+ return []
+
+ t0 = time.time()
+ for root_combo in itertools.product(*root_result_groups):
+ combined_layouts: dict[str, dict] = {}
+ selected_variants: list[LayoutVariantCandidate] = []
+ valid_combo = True
+ for branch_layouts, branch_variants in root_combo:
+ try:
+ combined_layouts = self._merge_layout_maps(combined_layouts, branch_layouts)
+ except ValueError as exc:
+ print(f"Skipping combo during merge: {exc}")
+ valid_combo = False
+ break
+ selected_variants.extend(branch_variants)
+
+ if not valid_combo or not selected_variants:
+ continue
+
+ combo_name = build_combo_name(tuple(selected_variants))
+ combo_dir = combined_root / combo_name
+ if combo_dir.exists():
+ shutil.rmtree(combo_dir)
+ combo_dir.mkdir(parents=True, exist_ok=True)
+
+ combo_layout_dir = combo_dir / "layout_params"
+ combo_layout_dir.mkdir(parents=True, exist_ok=True)
+
+ self._write_layout_tree(combined_layouts, combo_layout_dir)
+
+ # Fill missing wall/floor/ceiling layouts from simulated params if available.
+ self._fill_missing_root_layouts_from_simulated(combo_layout_dir)
+
+ # DEBUG
+ collision_report_path = combo_dir / "collision_report.json"
+ collision_result = self.tool_runner.check_object_collisions(
+ layout_dir=combo_layout_dir,
+ hull_dir=artifacts.collision_hulls_dir,
+ output=collision_report_path,
+ anchor_id=selected_variants[0].anchor_id if selected_variants else None,
+ )
+ if collision_result.returncode != 0 or not collision_report_path.exists():
+ print(f"Skipping combo {combo_name}: collision check failed.")
+ shutil.rmtree(combo_dir, ignore_errors=True)
+ continue
+
+ collision_report = self._load_json(collision_report_path)
+ collisions = collision_report.get("collisions", [])
+ if isinstance(collisions, list) and collisions:
+ print(f"Skipping combo {combo_name}: final collisions detected ({len(collisions)}).")
+ shutil.rmtree(combo_dir, ignore_errors=True)
+ continue
+
+ pending_manifests.append(
+ (
+ combo_dir / "combo_manifest.json",
+ {
+ "combo_name": combo_name,
+ "variants": tuple(selected_variants),
+ "layout_dir": combo_layout_dir,
+ "combo_dir": combo_dir,
+ "collision_report_path": collision_report_path,
+ },
+ )
+ )
+ combo_index_entries.append(
+ {
+ "combo_name": combo_name,
+ "layout_dir": str(combo_layout_dir),
+ "selected_variants": [
+ {
+ "anchor_id": variant.anchor_id,
+ "layout_id": variant.layout_id,
+ "group_dir": str(variant.group_dir),
+ "variant_dir": str(variant.variant_dir),
+ "layout_dir": str(variant.layout_dir),
+ "mapping_path": str(variant.mapping_path),
+ }
+ for variant in selected_variants
+ ],
+ }
+ )
+
+ print(f"[Timing] Combo enumeration & manifest building: {time.time() - t0:.2f}s ({len(pending_manifests)} combos)")
+
+ # ========== Novelty Metrics Computation ==========
+ # Find the pure org combo (all variants use _org layout)
+ org_combo = None
+ org_layout_dir = None
+ for manifest_path, manifest_data in pending_manifests:
+ if self._is_pure_org_variant(manifest_data["variants"]):
+ org_combo = manifest_data
+ org_layout_dir = manifest_data["layout_dir"]
+ break
+
+ if org_layout_dir is None:
+ print("Warning: No pure org combo found. Skipping novelty metrics and scene selection.")
+ # Still write manifests without novelty metrics
+ for manifest_path, manifest_data in pending_manifests:
+ self._write_combo_manifest(
+ manifest_path,
+ combo_name=manifest_data["combo_name"],
+ variants=manifest_data["variants"],
+ layout_dir=manifest_data["layout_dir"],
+ collision_report_path=manifest_data["collision_report_path"],
+ )
+ else:
+ print(f"Found pure org combo. Computing novelty metrics for all combos...")
+ t0 = time.time()
+ # Compute novelty metrics for all combos
+ for manifest_path, manifest_data in pending_manifests:
+ combo_layout_dir = manifest_data["layout_dir"]
+ novelty_metrics = self._compute_novelty_metrics(
+ combo_layout_dir,
+ org_layout_dir,
+ wp=0.4,
+ wu=0.4,
+ wr=0.2,
+ )
+
+ self._write_combo_manifest(
+ manifest_path,
+ combo_name=manifest_data["combo_name"],
+ variants=manifest_data["variants"],
+ layout_dir=manifest_data["layout_dir"],
+ collision_report_path=manifest_data["collision_report_path"],
+ novelty_metrics=novelty_metrics,
+ )
+ print(f" {manifest_data['combo_name']}: novelty={novelty_metrics['novelty']:.4f}")
+ print(f"[Timing] Novelty metrics computation: {time.time() - t0:.2f}s")
+
+ # ========== Representative Scene Selection ==========
+ print(f"\n{'='*60}")
+ print("Starting representative scene selection...")
+ print(f"{'='*60}")
+ t0 = time.time()
+ selected_reps = self._select_representative_scenes(
+ pending_manifests,
+ org_layout_dir=org_layout_dir,
+ k=6,
+ alpha=0.1,
+ wp=0.8,
+ wr=0.2,
+ )
+ print(f"[Timing] Representative scene selection: {time.time() - t0:.2f}s ({len(selected_reps)} selected)")
+
+ # Write a selection summary file
+ if selected_reps:
+ selection_summary_path = combined_root / "selection_summary.json"
+ summary_payload = {
+ "total_combos_evaluated": len(pending_manifests),
+ "selected_count": len(selected_reps),
+ "selected_scenes": [
+ {
+ "rank": rep.get("representative_rank"),
+ "combo_name": rep["combo_name"],
+ "layout_dir": str(rep["layout_dir"]),
+ }
+ for rep in selected_reps
+ ],
+ }
+ with selection_summary_path.open("w", encoding="utf-8") as f:
+ json.dump(summary_payload, f, ensure_ascii=False, indent=2)
+ print(f"Selection summary written to {selection_summary_path}")
+
+ # ========== Compose GLB for Selected Combos (including org seed) ==========
+ print(f"\nComposing GLB files for {len(selected_reps)} selected representative scenes...")
+ t0 = time.time()
+ for rep in selected_reps:
+ combo_name = rep["combo_name"]
+ combo_dir = rep["combo_dir"]
+ combo_layout_dir = rep["layout_dir"]
+ glb_path = self._compose_selected_combo(
+ combo_dir, combo_name, combo_layout_dir, artifacts.unposed_objects_dir
+ )
+ if glb_path is not None:
+ created_glbs.append(glb_path)
+ representative_scenes_dir = self.paths.representative_scenes_dir
+ if representative_scenes_dir.exists():
+ shutil.rmtree(representative_scenes_dir)
+ representative_scenes_dir.mkdir(parents=True, exist_ok=True)
+ for glb_path in created_glbs:
+ link_path = representative_scenes_dir / glb_path.name
+ if link_path.exists() or link_path.is_symlink():
+ link_path.unlink()
+ link_path.symlink_to(glb_path.resolve())
+ print(f"Linked representative GLBs to {representative_scenes_dir}")
+ print(f"[Timing] GLB composition for selected scenes: {time.time() - t0:.2f}s ({len(created_glbs)} GLBs)")
+
+ combo_index_path = combined_root / "combo_index.json"
+ self._write_combo_index(combo_index_path, combo_index_entries)
+
+ print(f"Layout combination stage completed with {len(created_glbs)} composed scenes.")
+ print(f"[Timing] Total layout combination: {time.time() - run_t0:.2f}s")
+ return created_glbs
\ No newline at end of file
diff --git a/vlm_agent/evolution/orchestrator.py b/vlm_agent/evolution/orchestrator.py
new file mode 100644
index 0000000..c9a1db1
--- /dev/null
+++ b/vlm_agent/evolution/orchestrator.py
@@ -0,0 +1,1484 @@
+from __future__ import annotations
+
+import copy
+import json
+import math
+import numpy as np
+import shutil
+import torch
+from dataclasses import dataclass, field, replace
+from pathlib import Path
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pytorch3d.transforms import quaternion_to_matrix
+
+from .scene_optimizer import build_anchor_dependency_graph, iter_subtree_roots, load_layout_snapshot, propagate_subtree_transform, collect_descendant_subtree
+from .layout_combination import LayoutCombination
+from .scene_generator import SceneGenerator
+from .tool_runner import ToolRunner
+from .types import PipelineReport, SceneArtifacts
+from .vlm_client import VLMClient
+from .paths import ProjectPaths
+
+
+@dataclass
+class PipelineOrchestrator:
+ """Pipeline: optimize existing artifacts -> generate -> compose."""
+
+ paths: ProjectPaths
+ tool_runner: ToolRunner
+ vlm_client: VLMClient
+ max_rounds: int = 3
+ collision_policy: str = "strict"
+ history: list[PipelineReport] = field(default_factory=list)
+
+ def _load_prompt(self, prompt_name: str) -> str:
+ """Load a prompt from the prompts directory."""
+ prompt_path = Path(__file__).resolve().parent / "prompts" / f"{prompt_name}.md"
+ if prompt_path.exists():
+ return prompt_path.read_text(encoding="utf-8")
+ return f"[Missing prompt: {prompt_name}]"
+
+ def _load_relationships_and_objects(self, suffix: str = "") -> tuple[list, dict, dict]:
+ """Load relationships and objects from JSON files.
+
+ Returns:
+ (relationships, objects_data, walls_data)
+ """
+ relations_path = self.paths.outputs_dir / f"relations{suffix}.json"
+ objects_path = self.paths.outputs_dir / f"objects{suffix}.json"
+
+ relationships = []
+ if relations_path.exists():
+ with relations_path.open("r", encoding="utf-8") as f:
+ relationships = json.load(f)
+
+ objects_data = {"objects": [], "walls": []}
+ if objects_path.exists():
+ with objects_path.open("r", encoding="utf-8") as f:
+ objects_data = json.load(f)
+
+ return relationships, objects_data.get("objects", []), objects_data.get("walls", [])
+
+ def _analyze_cross_anchor_constraints(self, object_groups: dict, objects: list) -> dict:
+ """Generate and persist cross-anchor constraints with the VLM."""
+ constraints_path = self.paths.outputs_dir / "cross_anchor_constraints.json"
+ constraints_path.parent.mkdir(parents=True, exist_ok=True)
+
+ prompt_template = self._load_prompt("analyze_cross_anchor_constraints")
+ image_path = self.paths.outputs_dir / "image.png"
+ object_categories = {
+ obj.get("id"): obj.get("category")
+ for obj in objects
+ if obj.get("id") and obj.get("category")
+ }
+
+ def _format_object_ref(object_id: str) -> str:
+ category = object_categories.get(object_id)
+ if category:
+ return f"{object_id} ({category})"
+ return object_id
+
+ def _annotate_object_groups(value: object) -> object:
+ if isinstance(value, dict):
+ annotated: dict[str, object] = {}
+ for key, item in value.items():
+ if key == "object_ids" and isinstance(item, list):
+ annotated[key] = [
+ _format_object_ref(obj_id) if isinstance(obj_id, str) else obj_id
+ for obj_id in item
+ ]
+ elif key.startswith("object_") and isinstance(item, dict):
+ new_key = _format_object_ref(key)
+ annotated[new_key] = _annotate_object_groups(item)
+ else:
+ annotated[key] = _annotate_object_groups(item)
+ return annotated
+ if isinstance(value, list):
+ return [_annotate_object_groups(item) for item in value]
+ return value
+
+ prompt = (
+ f"{prompt_template}\n\n"
+ "OBJECT_GROUPS (JSON):\n"
+ f"{json.dumps(_annotate_object_groups(object_groups), ensure_ascii=False, indent=2)}\n"
+ )
+ images = [str(image_path)] if image_path.exists() else None
+ vlm_response = self.vlm_client.plan(prompt, images=images)
+ payload = self.vlm_client.extract_json_payload(vlm_response)
+
+ with constraints_path.open("w", encoding="utf-8") as f:
+ json.dump(payload, f, ensure_ascii=False, indent=2)
+
+ return payload
+
+ def _load_cross_anchor_constraints(self, constraints_path: Path | None = None) -> dict[str, list[dict[str, list[str] | str]]]:
+ """Load cross-anchor constraints from a JSON file.
+
+ Returns:
+ Dict mapping anchor_id -> list of constraints, where each constraint is a dict with keys:
+ - "related_objects": list of object IDs
+ - "layout_note": string describing the layout constraint
+ """
+ if constraints_path is None:
+ constraints_path = self.paths.outputs_dir / "cross_anchor_constraints.json"
+
+ if not constraints_path.exists():
+ return {}
+
+ with constraints_path.open("r", encoding="utf-8") as f:
+ payload = json.load(f)
+
+ grouped_constraints: dict[str, list[dict[str, list[str] | str]]] = {}
+ if not isinstance(payload, list):
+ return grouped_constraints
+
+ for item in payload:
+ if not isinstance(item, dict):
+ continue
+ for anchor_key in ("anchor_a", "anchor_b"):
+ anchor_data = item.get(anchor_key)
+ if not isinstance(anchor_data, dict):
+ continue
+ anchor_id = anchor_data.get("anchor_id")
+ if not anchor_id:
+ continue
+ other_key = "anchor_b" if anchor_key == "anchor_a" else "anchor_a"
+ other_data = item.get(other_key)
+ if isinstance(other_data, dict) and other_data.get("anchor_id") == anchor_id:
+ continue
+ entry = {
+ "related_objects": anchor_data.get("related_objects", []),
+ "layout_note": anchor_data.get("layout_note", ""),
+ }
+ grouped_constraints.setdefault(str(anchor_id), []).append(entry)
+
+ return grouped_constraints
+
+ def _parse_collision_report(self, stdout: str) -> dict | None:
+ text = stdout.strip()
+ if not text:
+ return None
+ start = text.find("{")
+ if start == -1:
+ return None
+ payload_text = text[start:]
+ end = payload_text.rfind("}")
+ if end == -1:
+ return None
+ payload_text = payload_text[: end + 1]
+ try:
+ payload = json.loads(payload_text)
+ except json.JSONDecodeError:
+ return None
+ return payload if isinstance(payload, dict) else None
+
+ def _keep_y_axis_only(self, quaternion: list[float]) -> list[float]:
+ w, _, y, _ = quaternion
+ mag = math.sqrt(w**2 + y**2)
+ if mag < 1e-6:
+ return [1.0, 0.0, 0.0, 0.0]
+ return [w / mag, 0.0, y / mag, 0.0]
+
+ def _normalize_upright_layout_rotations(self, layout_dir: Path, objects: list[dict]) -> None:
+ for obj in objects:
+ if obj.get("upright", False) is not True:
+ continue
+ layout_path = layout_dir / f'{obj["id"]}.json'
+ if not layout_path.exists():
+ continue
+ with layout_path.open("r", encoding="utf-8") as f:
+ layout = json.load(f)
+ rotation = layout.get("rotation")
+ if isinstance(rotation, list) and len(rotation) == 1 and isinstance(rotation[0], list):
+ rotation = rotation[0]
+ if not isinstance(rotation, list) or len(rotation) != 4:
+ continue
+ layout["rotation"] = [self._keep_y_axis_only([float(v) for v in rotation])]
+ with layout_path.open("w", encoding="utf-8") as f:
+ json.dump(layout, f, indent=2)
+
+ def _format_object_with_category(
+ self, object_id: str | None, object_categories: dict[str, str] | None
+ ) -> str | None:
+ if not object_id:
+ return object_id
+ if not object_categories:
+ return object_id
+ category = object_categories.get(object_id)
+ if not category:
+ return object_id
+ return f"{object_id} ({category})"
+
+ def _get_deep_collision_pairs(
+ self,
+ report: dict | None,
+ *,
+ contact_threshold: int,
+ object_categories: dict[str, str] | None = None,
+ ) -> list[dict]:
+ if not report:
+ return []
+ collisions = report.get("collisions", [])
+ if not isinstance(collisions, list):
+ return []
+ results = []
+ for entry in collisions:
+ if not isinstance(entry, dict):
+ continue
+ contacts = entry.get("contacts")
+ if isinstance(contacts, (int, float)) and contacts > contact_threshold:
+ results.append(
+ {
+ "a": self._format_object_with_category(
+ entry.get("a"), object_categories
+ ),
+ "b": self._format_object_with_category(
+ entry.get("b"), object_categories
+ ),
+ "contacts": int(contacts),
+ }
+ )
+ return results
+
+ def _parse_wall_normal(self, wall: dict) -> list:
+ """Compute the normal vector for a wall in 3D space.
+
+ Given two points on the wall (x, z coordinates at y=0), compute the normal vector
+ pointing outward from the wall.
+
+ Returns:
+ Normal vector [x, y, z] in world coordinates
+ """
+ p1 = wall.get("first_point", [0, 0])
+ p2 = wall.get("second_point", [0, 0])
+
+ # Extract x, z coordinates
+ x1, z1 = p1[0], p1[1]
+ x2, z2 = p2[0], p2[1]
+
+ # Direction along the wall
+ wall_dir = np.array([x2 - x1, 0, z2 - z1], dtype=np.float32)
+ wall_dir_norm = np.linalg.norm(wall_dir)
+
+ if wall_dir_norm < 1e-6:
+ print(f"Warning: Wall {wall.get('id')} has zero length, using default normal")
+ return [1.0, 0.0, 0.0]
+
+ wall_dir = wall_dir / wall_dir_norm
+
+ # p1, p2 must be clockwise
+ # Normal perpendicular to wall_dir in XZ plane, pointing outward
+ # For a wall from p1 to p2, the outward normal is rotated 90 degrees
+ # clockwise in XZ plane when viewed from above (+Y)
+ normal = np.array([wall_dir[2], 0.0, -wall_dir[0]], dtype=np.float32)
+
+ return normal.tolist()
+
+ def _group_objects_by_anchor(
+ self, relationships: list, objects: list, walls: list
+ ) -> dict:
+ """Group objects by their anchor point (floor, ceiling, or wall).
+
+ Returns:
+ Dict mapping anchor_id -> {"relation_type": str, "object_ids": list[str]}
+ """
+ groups: dict[str, dict[str, list[str] | str]] = {}
+
+ for rel in relationships:
+ relation_type = rel.get("relation")
+ if relation_type != "attach" and relation_type != "contain":
+ continue
+
+ anchor_id = rel["parameters"]["object_anchor"]
+ target_id = rel["parameters"]["object_target"]
+
+ if anchor_id not in groups:
+ groups[anchor_id] = {"relation_type": relation_type, "object_ids": []}
+ elif groups[anchor_id].get("relation_type") != relation_type:
+ groups[anchor_id]["relation_type"] = "mixed"
+
+ object_ids = groups[anchor_id]["object_ids"]
+ if target_id not in object_ids:
+ object_ids.append(target_id)
+
+ return groups
+
+ def _compute_gravity_for_anchor(self, anchor_id: str, walls_by_id: dict) -> list:
+ """Compute the gravity direction for a given anchor.
+
+ Returns:
+ Gravity vector [x, y, z]
+ """
+ if anchor_id == "floor":
+ # Default gravity pointing downward (-Y)
+ return [0.0, -2.0, 0.0]
+ elif anchor_id == "ceiling":
+ # Reverse gravity pointing upward (+Y)
+ return [0.0, 2.0, 0.0]
+ elif anchor_id.startswith("wall"):
+ # Compute normal from wall geometry
+ if anchor_id in walls_by_id:
+ wall = walls_by_id[anchor_id]
+ normal = self._parse_wall_normal(wall)
+ # Scale to match gravity magnitude
+ magnitude = 2.0
+ gravity = [n * magnitude for n in normal]
+ return gravity
+ else:
+ print(f"Warning: Wall {anchor_id} not found, using default gravity")
+ return [0.0, -2.0, 0.0]
+ else:
+ # Unknown anchor, use default gravity
+ return [0.0, -2.0, 0.0]
+
+ def _get_wall_object_ids(self, layout_dir: Path) -> list[str]:
+ """Get wall object IDs from the layout directory.
+
+ Returns object IDs that match wall patterns (e.g., wall_001, ceiling, floor).
+ """
+ wall_object_ids = []
+ if not layout_dir.exists():
+ return wall_object_ids
+
+ for layout_file in layout_dir.glob("*.json"):
+ if layout_file.stem.startswith("object_"):
+ continue
+ obj_id = layout_file.stem
+ # Check if it matches wall patterns
+ if "wall" in obj_id.lower() or obj_id.lower() in ["ceiling", "floor"]:
+ wall_object_ids.append(obj_id)
+
+ return sorted(wall_object_ids)
+
+ def _compute_camera_direction_for_anchor(self, anchor_id: str, walls_by_id: dict, objects: list,
+ relation_type: str, artifacts: SceneArtifacts) -> tuple[list, str]:
+ """Compute the camera direction for a given anchor.
+
+ Returns:
+ Camera direction vector [x, y, z] (normalized)
+ Render anchor ID
+ """
+ if anchor_id == "floor":
+ # Camera looking downward
+ return [0.0, -1.0, 0.0], anchor_id
+ elif anchor_id == "ceiling":
+ # Camera looking upward
+ return [0.0, 1.0, 0.0], anchor_id
+ elif anchor_id.startswith("wall"):
+ # Use normal from wall geometry (already normalized by _parse_wall_normal)
+ if anchor_id in walls_by_id:
+ wall = walls_by_id[anchor_id]
+ normal = self._parse_wall_normal(wall)
+ return normal, None
+ else:
+ print(f"Warning: Wall {anchor_id} not found, using default camera direction")
+ return [0.0, -1.0, 0.0], None
+ else:
+ if relation_type == "contain" or relation_type == "mixed":
+ if any(obj.get("id") == anchor_id and obj.get("render_direction") == "front" for obj in objects):
+ object_pose_path = artifacts.layout_params_dir / f"{anchor_id}.json"
+ if object_pose_path.exists():
+ with object_pose_path.open("r", encoding="utf-8") as f:
+ pose_data = json.load(f)
+ direction_vector = torch.tensor([0.0, 0.0, -1.0], dtype=torch.float32)
+ q_wxyz = pose_data["rotation"][0]
+ quat_tensor = torch.tensor(q_wxyz, dtype=torch.float32)
+ rot_matrix = quaternion_to_matrix(quat_tensor)
+ rotated_vector = torch.matmul(rot_matrix, direction_vector)
+ camera_direction = -rotated_vector
+ return camera_direction.tolist(), None
+ else:
+ return [0.0, 0.0, 1.0], anchor_id
+ return [0.0, -1.0, 0.0], anchor_id
+
+ def _stage_optimization(self, artifacts: SceneArtifacts) -> SceneArtifacts:
+ """Optimize the existing scene using physics simulation."""
+ if artifacts.unposed_objects_dir and artifacts.collision_hulls_dir:
+ print("Running generate_room_mesh tool.")
+ room_mesh_result = self.tool_runner.generate_room_mesh(
+ artifacts.layout_params_dir,
+ artifacts.unposed_objects_dir,
+ self.paths.outputs_dir / "objects.json",
+ )
+ if room_mesh_result.returncode != 0:
+ print(f"generate_room_mesh failed: {room_mesh_result.stderr}")
+ else:
+ print("Room mesh generated.")
+
+ print("Running build_collision_hulls tool.")
+ manifest = artifacts.collision_hulls_dir / "manifest.json"
+ result = self.tool_runner.build_collision_hulls(
+ artifacts.unposed_objects_dir,
+ artifacts.collision_hulls_dir,
+ manifest,
+ split_mode="none",
+ )
+ if result.failed_object_ids:
+ print(f"Collision hulls built with failed objects: {', '.join(result.failed_object_ids)}")
+ else:
+ print("Collision hulls built.")
+
+ if artifacts.layout_params_dir and artifacts.collision_hulls_dir and artifacts.simulated_layout_dir:
+ print("Running simulate_scene_physics tool with grouped simulation.")
+
+ artifacts.simulated_layout_dir.mkdir(parents=True, exist_ok=True)
+ for layout_file in artifacts.layout_params_dir.glob("*.json"):
+ shutil.copy2(layout_file, artifacts.simulated_layout_dir / layout_file.name)
+
+ # Load relationships and objects
+ relationships, objects, walls = self._load_relationships_and_objects()
+ walls_by_id = {w["id"]: w for w in walls}
+ self._normalize_upright_layout_rotations(artifacts.simulated_layout_dir, objects)
+
+ # Get wall object IDs to be used as ground objects
+ wall_object_ids = self._get_wall_object_ids(artifacts.simulated_layout_dir)
+
+ # Group objects by anchor point
+ object_groups = self._group_objects_by_anchor(relationships, objects, walls)
+
+ if not object_groups:
+ print("No grouped relationships found, running standard simulation.")
+ self.tool_runner.simulate_scene_physics(
+ artifacts.simulated_layout_dir,
+ artifacts.collision_hulls_dir,
+ artifacts.simulated_layout_dir,
+ cpu_only=True,
+ use_ground_object=True,
+ ground_object_ids=wall_object_ids,
+ )
+ else:
+ print(f"Found {len(object_groups)} anchor groups")
+
+ # Fix containment for each group (skip walls and ceiling)
+ for anchor_id, group_info in object_groups.items():
+ object_ids = group_info["object_ids"] if isinstance(group_info, dict) else group_info
+ relation_type = group_info.get("relation_type") if isinstance(group_info, dict) else None
+ if anchor_id.startswith("wall") or anchor_id == "ceiling":
+ continue
+
+ print(
+ f"Fixing containment for {len(object_ids)} objects "
+ f"{'attached to' if relation_type == 'attach' else 'inside'} '{anchor_id}'"
+ )
+
+ combined_object_ids = sorted(list(set(object_ids) | set([anchor_id])))
+ self.tool_runner.fix_layout_containment(
+ artifacts.simulated_layout_dir,
+ artifacts.unposed_objects_dir,
+ artifacts.simulated_layout_dir,
+ object_ids=combined_object_ids,
+ anchor_id=anchor_id,
+ tolerated_out_of_bounds=0.0,
+ )
+
+ parents_by_child, children_by_parent, indegree = build_anchor_dependency_graph(object_groups)
+ ready_queue = [anchor_id for anchor_id, degree in indegree.items() if degree == 0]
+ ready_queue.sort(key=lambda anchor: (0 if anchor.startswith("wall") else 1 if anchor == "ceiling" else 2 if anchor == "floor" else 3, anchor))
+ processed_anchors: set[str] = set()
+ anchors_with_simulation: set[str] = set()
+
+ pending_layout_dir = artifacts.simulated_layout_dir.parent / f"{artifacts.simulated_layout_dir.name}_pending"
+ if pending_layout_dir.exists():
+ shutil.rmtree(pending_layout_dir)
+ if artifacts.simulated_layout_dir.exists():
+ shutil.copytree(artifacts.simulated_layout_dir, pending_layout_dir)
+ else:
+ pending_layout_dir.mkdir(parents=True, exist_ok=True)
+
+ illegal_objects = set()
+ while ready_queue:
+ anchor_id = ready_queue.pop(0)
+ if anchor_id in processed_anchors:
+ continue
+ processed_anchors.add(anchor_id)
+
+ object_ids = object_groups.get(anchor_id, {}).get("object_ids", [])
+ relation_type = object_groups.get(anchor_id, {}).get("relation_type", None)
+ if not object_ids:
+ for child_id in sorted(children_by_parent.get(anchor_id, [])):
+ indegree[child_id] = max(indegree.get(child_id, 0) - 1, 0)
+ if indegree[child_id] == 0:
+ ready_queue.append(child_id)
+ ready_queue.sort(key=lambda anchor: (0 if anchor.startswith("wall") else 1 if anchor == "ceiling" else 2 if anchor == "floor" else 3, anchor))
+ continue
+
+ anchors_with_simulation.add(anchor_id)
+ gravity = self._compute_gravity_for_anchor(anchor_id, walls_by_id)
+ print(f"Simulating {len(object_ids)} objects attached to or inside '{anchor_id}' with gravity {gravity}")
+
+ pending_layout_dir.mkdir(parents=True, exist_ok=True)
+ ground_object_ids = list(set(wall_object_ids) | {anchor_id})
+ combined_object_ids = sorted(list(set(object_ids) | set(ground_object_ids)))
+ before_layouts = load_layout_snapshot(pending_layout_dir)
+ init_non_ground_y_offset = True if anchor_id not in wall_object_ids and relation_type == "attach" else False
+
+ sim_result = self.tool_runner.simulate_scene_physics(
+ pending_layout_dir,
+ artifacts.collision_hulls_dir,
+ pending_layout_dir,
+ gravity=gravity,
+ object_ids=combined_object_ids,
+ cpu_only=True,
+ use_ground_object=True,
+ ground_object_ids=ground_object_ids,
+ init_non_ground_y_offset=init_non_ground_y_offset,
+ )
+
+ if sim_result.returncode != 0:
+ print(f"Simulation failed for anchor '{anchor_id}': {sim_result.stderr}")
+ continue
+
+ if not (anchor_id.startswith("wall") or anchor_id == "ceiling"):
+ detect_only = (self.collision_policy == "strict")
+ print(f"{'Fixing' if not detect_only else 'Detecting'} containment for {len(object_ids)} objects attached to or inside '{anchor_id}'")
+
+ combined_object_ids = sorted(list(set(object_ids) | set([anchor_id])))
+ fix_result = self.tool_runner.fix_layout_containment(
+ pending_layout_dir,
+ artifacts.unposed_objects_dir,
+ pending_layout_dir,
+ object_ids=combined_object_ids,
+ anchor_id=anchor_id,
+ tolerated_out_of_bounds=0.5 if anchor_id != "floor" else 0.0,
+ detect_only=detect_only,
+ )
+
+ # If strict policy, collect all out-of-bounds objects and their descendants
+ if self.collision_policy == "strict" and fix_result.out_of_bounds_object_ids:
+ out_ids = set(fix_result.out_of_bounds_object_ids)
+ for obj_id in fix_result.out_of_bounds_object_ids:
+ subtree_ids = collect_descendant_subtree(obj_id, children_by_parent)
+ out_ids.update(subtree_ids)
+ illegal_objects.update(out_ids)
+ print(f"Anchor '{anchor_id}' has illegal objects: {', '.join(x for x in out_ids)}")
+
+ after_layouts = load_layout_snapshot(pending_layout_dir)
+ propagated_subtrees: list[str] = []
+ for subtree_root_id in iter_subtree_roots(anchor_id, children_by_parent):
+ propagate_subtree_transform(
+ self,
+ pending_layout_dir,
+ children_by_parent,
+ anchor_id,
+ subtree_root_id,
+ before_layouts,
+ after_layouts,
+ )
+ propagated_subtrees.append(subtree_root_id)
+
+ if propagated_subtrees:
+ print(
+ f"Anchor '{anchor_id}' propagated subtrees: {', '.join(propagated_subtrees)}"
+ )
+
+ for child_id in sorted(children_by_parent.get(anchor_id, [])):
+ indegree[child_id] = max(indegree.get(child_id, 0) - 1, 0)
+ if indegree[child_id] == 0:
+ ready_queue.append(child_id)
+ ready_queue.sort(key=lambda anchor: (0 if anchor.startswith("wall") else 1 if anchor == "ceiling" else 2 if anchor == "floor" else 3, anchor))
+
+ if self.collision_policy == "strict":
+ # If strict policy, check for deep collisions
+ collision_output_path = self.paths.outputs_dir / "collision_orig_scene.json"
+ collision_result = self.tool_runner.check_object_collisions(
+ layout_dir=pending_layout_dir,
+ hull_dir=artifacts.collision_hulls_dir,
+ output=collision_output_path,
+ )
+ if collision_result.returncode != 0:
+ print("Collision check failed for anchor "
+ f"{anchor_id}: {collision_result.stderr}"
+ )
+ else:
+ collision_report = self._parse_collision_report(collision_result.stdout)
+ collision_pairs = collision_report.get("collisions", []) if collision_report else []
+ anchor_object_ids = set(object_groups.keys())
+
+ def _subtree_size(object_id: str) -> int:
+ return len(collect_descendant_subtree(object_id, children_by_parent))
+
+ def _object_volume(pair_entry: dict, object_id: str) -> float:
+ size_key = "size_a" if pair_entry.get("a") == object_id else "size_b"
+ sizes = pair_entry.get(size_key, [])
+ if not isinstance(sizes, list) or len(sizes) != 3:
+ return float("inf")
+ return float(sizes[0]) * float(sizes[1]) * float(sizes[2])
+
+ for pair_entry in collision_pairs:
+ if not isinstance(pair_entry, dict):
+ continue
+ obj0 = str(pair_entry.get("a", ""))
+ obj1 = str(pair_entry.get("b", ""))
+ if not obj0 or not obj1:
+ continue
+ if obj0 in illegal_objects or obj1 in illegal_objects:
+ continue
+
+ obj0_is_anchor = obj0 in anchor_object_ids
+ obj1_is_anchor = obj1 in anchor_object_ids
+
+ if not obj0_is_anchor and not obj1_is_anchor:
+ vol0 = _object_volume(pair_entry, obj0)
+ vol1 = _object_volume(pair_entry, obj1)
+ illegal_objects.add(obj0 if vol0 <= vol1 else obj1)
+ print(f"Deep collision detected between non-anchor objects '{obj0}' and '{obj1}', "
+ f"marking smaller volume object '{obj0 if vol0 <= vol1 else obj1}' as illegal.")
+ continue
+
+ if obj0_is_anchor ^ obj1_is_anchor:
+ illegal_objects.add(obj0 if not obj0_is_anchor else obj1)
+ print(f"Collision detected between anchor object '{obj0 if obj0_is_anchor else obj1}' and non-anchor object "
+ f"'{obj1 if obj0_is_anchor else obj0}', marking non-anchor object as illegal.")
+ continue
+
+ def _anchor_priority(object_id: str) -> tuple[int, int, float]:
+ if object_id == "wall" or object_id.startswith("wall"):
+ category_priority = 0
+ elif object_id == "floor":
+ category_priority = 0
+ elif object_id == "ceiling":
+ category_priority = 0
+ else:
+ category_priority = 1
+ subtree_priority = -_subtree_size(object_id)
+ volume_priority = -_object_volume(pair_entry, object_id)
+ return (category_priority, subtree_priority, volume_priority)
+
+ if _anchor_priority(obj0) <= _anchor_priority(obj1):
+ illegal_objects.update(collect_descendant_subtree(obj1, children_by_parent))
+ print(f"Collision detected between anchor objects '{obj0}' and '{obj1}', marking '{obj1}' and its descendants as illegal.")
+ else:
+ illegal_objects.update(collect_descendant_subtree(obj0, children_by_parent))
+ print(f"Collision detected between anchor objects '{obj0}' and '{obj1}', marking '{obj0}' and its descendants as illegal.")
+
+ print(f"Illegal objects after simulation: {illegal_objects if illegal_objects else 'None'}")
+
+ if illegal_objects:
+ filtered_relationships = []
+ for relation in relationships:
+ if not isinstance(relation, dict):
+ continue
+ parameters = relation.get("parameters", {})
+ if not isinstance(parameters, dict):
+ continue
+ if parameters.get("object_anchor") in illegal_objects:
+ continue
+ if parameters.get("object_target") in illegal_objects:
+ continue
+ filtered_relationships.append(relation)
+
+ filtered_objects = [
+ obj for obj in objects
+ if isinstance(obj, dict) and obj.get("id") not in illegal_objects
+ ]
+
+ filtered_relationships = filtered_relationships if illegal_objects else relationships
+ filtered_objects = filtered_objects if illegal_objects else objects
+ with (self.paths.outputs_dir / "relations_filtered.json").open("w", encoding="utf-8") as f:
+ json.dump(filtered_relationships, f, ensure_ascii=False, indent=2)
+ with (self.paths.outputs_dir / "objects_filtered.json").open("w", encoding="utf-8") as f:
+ json.dump({"walls": walls, "objects": filtered_objects}, f, ensure_ascii=False, indent=2)
+
+ if pending_layout_dir.exists():
+ if artifacts.simulated_layout_dir.exists():
+ if illegal_objects:
+ for obj_id in illegal_objects:
+ obj_file = pending_layout_dir / f"{obj_id}.json"
+ if obj_file.exists():
+ obj_file.unlink()
+ shutil.rmtree(artifacts.simulated_layout_dir)
+ shutil.move(str(pending_layout_dir), str(artifacts.simulated_layout_dir))
+
+ if not anchors_with_simulation and object_groups:
+ print("No anchors were processed by queue-based grouped simulation.")
+
+ return artifacts
+
+ def _stage_generate_scene(
+ self, artifacts: SceneArtifacts,
+ deep_collision_contact_threshold: int = 8,
+ max_refine_rounds: int = 3,
+ enable_memory: bool = True,
+ refine_movement_threshold: float = 0.03,
+ refine_rotation_tolerance: float = 1e-2,
+ draw_object_directions: bool = True,
+ draw_object_bboxes: bool = True,
+ ) -> None:
+ """Generate new scene variants based on grouped renders."""
+ generator = SceneGenerator(artifacts)
+ prompt_template = self._load_prompt("generate_layout_description")
+ pose_prompt_template = self._load_prompt("generate_2d_pose")
+ pose_prompt_fix_rot_template = self._load_prompt("generate_2d_pose_fixed_rotation")
+ evaluate_prompt_template = self._load_prompt("evaluate_layout_modification")
+ revise_prompt_template = self._load_prompt("apply_layout_modification")
+
+ relationships_raw, objects, walls = self._load_relationships_and_objects(suffix="_filtered")
+ object_categories = {
+ obj.get("id"): obj.get("category")
+ for obj in objects
+ if obj.get("id") and obj.get("category")
+ }
+
+ layout_dir = artifacts.simulated_layout_dir or artifacts.layout_params_dir
+ unposed_dir = artifacts.unposed_objects_dir
+
+ generation_output_dir = self.paths.outputs_dir / "generated"
+ if generation_output_dir.exists():
+ shutil.rmtree(generation_output_dir)
+ generation_output_dir.mkdir(parents=True, exist_ok=True)
+
+ walls_by_id = {w.get("id"): w for w in walls}
+ object_groups = self._group_objects_by_anchor(relationships_raw, objects, walls)
+ if not object_groups:
+ all_ids = []
+ for layout_file in layout_dir.glob("*.json"):
+ all_ids.append(layout_file.stem)
+ if all_ids:
+ object_groups = {"floor": {"object_ids": sorted(all_ids), "relation_type": "attach"}}
+
+ # Ensure each object has a unique anchor: remove objects from wall anchors if they are already anchored by non-wall anchors
+ non_wall_anchors_object_ids = set()
+ for anchor_id, group_info in object_groups.items():
+ if not anchor_id.startswith("wall"):
+ object_ids = group_info.get("object_ids", []) if isinstance(group_info, dict) else group_info
+ non_wall_anchors_object_ids.update(object_ids)
+
+ for anchor_id, group_info in list(object_groups.items()):
+ if anchor_id.startswith("wall"):
+ object_ids = group_info.get("object_ids", []) if isinstance(group_info, dict) else group_info
+ filtered_object_ids = [obj_id for obj_id in object_ids if obj_id not in non_wall_anchors_object_ids]
+ if not filtered_object_ids:
+ object_groups.pop(anchor_id, None)
+ continue
+ if isinstance(group_info, dict):
+ group_info["object_ids"] = filtered_object_ids
+ else:
+ object_groups[anchor_id] = filtered_object_ids
+
+ if not (self.paths.outputs_dir / "cross_anchor_constraints.json").exists():
+ self._analyze_cross_anchor_constraints(object_groups, objects)
+ cross_anchor_constraints = self._load_cross_anchor_constraints()
+
+ def _process_group(anchor_id: str, group_info: dict | list) -> dict | None:
+ group_logs: list[str] = []
+
+ def group_log(message: str) -> None:
+ line = f"[{anchor_id}] {message}"
+ print(line, flush=True)
+ group_logs.append(line)
+
+ object_ids = group_info.get("object_ids", []) if isinstance(group_info, dict) else group_info
+ relation_type = group_info.get("relation_type") if isinstance(group_info, dict) else None
+
+ group_dir = generation_output_dir / f"group_{anchor_id}"
+ group_dir.mkdir(parents=True, exist_ok=True)
+
+ camera_direction, render_anchor_id = self._compute_camera_direction_for_anchor(
+ anchor_id,
+ walls_by_id,
+ objects,
+ relation_type,
+ artifacts,
+ )
+ output_path = group_dir / f"grouped_scene_{anchor_id}.png"
+ mapping_path = output_path.with_suffix(".json")
+
+ render_object_ids = list(object_ids)
+ if anchor_id not in render_object_ids:
+ render_object_ids.append(anchor_id)
+
+ group_log(f"Rendering grouped scene with camera direction {camera_direction}.")
+ render_result = self.tool_runner.render_scene(
+ layout_dir=layout_dir,
+ unposed_dir=unposed_dir,
+ output=output_path,
+ object_ids=render_object_ids,
+ camera_direction=camera_direction,
+ radius=0.0,
+ image_size=1024,
+ add_grid=True,
+ draw_object_directions=draw_object_directions if relation_type == "attach" else False,
+ draw_object_bboxes=draw_object_bboxes,
+ anchor_id=render_anchor_id,
+ hide_object_ids=[anchor_id],
+ use_aabb_for_object_bboxes=True if relation_type != "attach" else False,
+ )
+
+ if render_result.returncode != 0:
+ group_log(f"Failed to render group '{anchor_id}': {render_result.stderr}")
+ return None
+
+ cross_anchor_constraint = cross_anchor_constraints.get(anchor_id)
+ cross_anchor_constraint_text = ""
+ if cross_anchor_constraint:
+ cross_anchor_constraint_text = (
+ "CROSS_ANCHOR_CONSTRAINTS:\n"
+ f"{json.dumps(cross_anchor_constraint, ensure_ascii=False, indent=2)}\n\n"
+ )
+
+ # group_log(f"Cross-anchor constraints: {cross_anchor_constraint_text}")
+
+ # Generate similar layout description using VLM based on the rendered image and scene context
+ summary, anchor_category = generator.generate_similar_layout_context(
+ mapping_path=mapping_path,
+ objects=objects,
+ anchor_id=anchor_id,
+ )
+ description_prompt = (
+ f"{prompt_template}\n\n"
+ f"ANCHOR_CONSTRAINT: The anchor is {anchor_category or anchor_id} and must not change in the new layout.\n\n"
+ f"{cross_anchor_constraint_text}"
+ "CURRENT_LAYOUT_DATA (\"bbox_2d\": [min_u, min_v, max_u, max_v]):\n"
+ f"{json.dumps(summary, ensure_ascii=False, indent=2)}\n"
+ )
+
+ group_log(f"Generating new layout description.")
+ image_inputs = [str(output_path)] if output_path.exists() else None
+ vlm_response = self.vlm_client.plan(description_prompt, images=image_inputs)
+ response_payload = self.vlm_client.extract_json_payload(vlm_response)
+ response_payload["usage"] = self.vlm_client.last_usage or {}
+
+ output_json = group_dir / "generated_layout_description.json"
+ with output_json.open("w", encoding="utf-8") as f:
+ json.dump(response_payload, f, ensure_ascii=False, indent=2)
+
+ # For each generated layout description, also generate a 2D pose estimation
+ layout_entries = response_payload.get("alternative_layouts", [])
+
+ # Insert the original layout as layout_id='org' at the beginning
+ layout_entries.insert(0, {"layout_id": "org"})
+
+ group_log(f"Processing {len(layout_entries)} layout entries for anchor '{anchor_id}'.")
+
+ # Helper function to check collisions and render a layout round
+ def _check_collisions_and_render_round(
+ source_layout_dir: Path,
+ current_round_dir: Path,
+ current_round_index: int,
+ layout_suffix: str,
+ ) -> tuple[Path, Path, Path, list[dict]]:
+ ground_object_ids = []
+ anchor_layout_path = source_layout_dir / f"{anchor_id}.json"
+ if anchor_layout_path.exists():
+ ground_object_ids = [anchor_id]
+
+ if ground_object_ids:
+ group_log(f"Checking collisions for round {current_round_index} with ground objects {ground_object_ids}")
+ else:
+ group_log(f"Checking collisions for round {current_round_index}")
+
+ deep_collision_pairs: list[dict] = []
+ collision_output_path = (
+ current_round_dir
+ / f"layout_{layout_suffix}_collisions_r{current_round_index}.json"
+ )
+ collision_result = self.tool_runner.check_object_collisions(
+ layout_dir=source_layout_dir,
+ hull_dir=artifacts.collision_hulls_dir,
+ output=collision_output_path,
+ )
+ if collision_result.returncode == 0:
+ collision_report = self._parse_collision_report(collision_result.stdout)
+ deep_collision_pairs = self._get_deep_collision_pairs(
+ collision_report,
+ contact_threshold=deep_collision_contact_threshold,
+ object_categories=object_categories,
+ )
+ else:
+ group_log(
+ "Collision check failed for layout "
+ f"{layout_suffix} round {current_round_index}: {collision_result.stderr}"
+ )
+
+ group_log(f"Rendering layout {layout_suffix} round {current_round_index}")
+ render_output_path = (
+ current_round_dir / f"layout_{layout_suffix}_render_r{current_round_index}.png"
+ )
+ self.tool_runner.render_scene(
+ layout_dir=source_layout_dir,
+ unposed_dir=unposed_dir,
+ output=render_output_path,
+ object_ids=render_object_ids,
+ camera_direction=camera_direction,
+ radius=0.0,
+ image_size=1024,
+ add_grid=True,
+ draw_object_directions=draw_object_directions if relation_type == "attach" else False,
+ draw_object_bboxes=draw_object_bboxes,
+ anchor_id=render_anchor_id,
+ hide_object_ids=[anchor_id],
+ grid_mapping=mapping_path,
+ use_aabb_for_object_bboxes=True if relation_type != "attach" else False,
+ )
+ return (
+ source_layout_dir,
+ render_output_path,
+ render_output_path.with_suffix(".json"),
+ deep_collision_pairs,
+ )
+
+ # Function to process a single layout entry
+ def _process_layout_entry(layout_entry: dict, anchor_id: str, output_path: Path, mapping_path: Path) -> dict | None:
+ layout_logs: list[str] = []
+
+ def layout_log(message: str) -> None:
+ line = f"[{anchor_id}][layout:{layout_entry.get('layout_id', '')}] {message}"
+ layout_logs.append(line)
+
+ layout_id = layout_entry.get("layout_id")
+
+ layout_suffix = str(layout_id) if layout_id is not None else "unknown"
+ base_variant_dir = group_dir / f"layout_{layout_suffix}"
+ base_variant_dir.mkdir(parents=True, exist_ok=True)
+
+ memory = None
+ if enable_memory:
+ memory_path = base_variant_dir / "memory.md"
+ memory = generator.create_layout_memory(
+ output_path=memory_path,
+ anchor_id=anchor_id,
+ layout_id=layout_suffix,
+ )
+
+ # Get the initial layout summary and anchor category for the current layout entry
+ summary, anchor_category = generator.generate_similar_layout_context(
+ mapping_path=mapping_path,
+ objects=objects,
+ anchor_id=anchor_id,
+ )
+
+ # For the original layout (org), skip VLM pose generation and use existing layout directly
+ if layout_id == "org":
+ layout_log(f"Processing original layout as round 0 for group '{anchor_id}'")
+ round_records: list[dict] = []
+ round_index = 0
+ round_dir = base_variant_dir / f"round_{round_index}"
+ round_dir.mkdir(parents=True, exist_ok=True)
+
+ # Use the original layout as the starting point (no VLM generation needed)
+ source_layout_dir = artifacts.simulated_layout_dir or artifacts.layout_params_dir
+ current_layout_dir = round_dir / "layout_params"
+ current_layout_dir.mkdir(parents=True, exist_ok=True)
+
+ # Copy only the relevant objects for this anchor group
+ org_object_ids = set(object_ids) | {anchor_id}
+ for layout_file in source_layout_dir.glob("*.json"):
+ if layout_file.stem in org_object_ids:
+ shutil.copy2(layout_file, current_layout_dir / layout_file.name)
+
+ # Jump to the while True optimization loop
+ # The rest of the optimization loop will be executed below
+
+ else: # layout_id != "org"
+ # Generate 2D pose for the layout entry using VLM
+ pose_prompt = (
+ f"{pose_prompt_template if relation_type == 'attach' else pose_prompt_fix_rot_template}\n\n"
+ f"ANCHOR_CONSTRAINT: The anchor is {anchor_category or anchor_id} and must not change in the new layout.\n\n"
+ f"{cross_anchor_constraint_text}"
+ f"LAYOUT_DESCRIPTION:\n{json.dumps(layout_entry.get('layout_reasoning'), ensure_ascii=False, indent=2)}\n\n"
+ "CURRENT_LAYOUT_DATA (\"bbox_2d\": [min_u, min_v, max_u, max_v]):\n"
+ f"{json.dumps(summary, ensure_ascii=False, indent=2)}\n"
+ )
+
+ layout_log(f"Generating 2D pose for layout variant '{layout_entry.get('layout_id')}' of group '{anchor_id}'")
+ pose_response = self.vlm_client.plan(
+ pose_prompt,
+ images=[str(output_path)] if output_path.exists() else None,
+ )
+ pose_payload = self.vlm_client.extract_json_payload(pose_response)
+ pose_payload_output = {
+ "vlm_output": pose_payload,
+ "usage": self.vlm_client.last_usage or {},
+ }
+
+ if memory is not None:
+ initial_pose = generator.extract_2d_pose_with_rotation(pose_payload)
+ memory.record_initial_pose(initial_pose)
+
+ round_records = []
+ round_index = 0
+ round_dir = base_variant_dir / f"round_{round_index}"
+ pose_output_path = round_dir / f"layout_{layout_suffix}_2d_pose_r{round_index}.json"
+ pose_output_path.parent.mkdir(parents=True, exist_ok=True)
+
+ # Compute 3D pose from 2D pose and mapping data
+ vlm_output = generator.convert_2d_pose_to_3d(mapping_path, pose_payload_output.get("vlm_output", {}))
+ pose_payload_output["vlm_output"] = vlm_output
+ with pose_output_path.open("w", encoding="utf-8") as f:
+ json.dump(pose_payload_output, f, ensure_ascii=False, indent=2)
+
+ generator.generate_posed_layouts(
+ pose_output_path=pose_output_path,
+ pose_payload=vlm_output,
+ render_object_ids=render_object_ids,
+ camera_direction=camera_direction,
+ layout_dir=layout_dir,
+ anchor_id=anchor_id,
+ render_transform_path=mapping_path,
+ )
+
+ current_layout_dir = round_dir / "layout_params"
+ if not current_layout_dir.exists():
+ layout_log(f"Warning: Generated layout params not found for {layout_suffix} round {round_index}")
+ return {
+ "layout_suffix": None,
+ "logs": layout_logs,
+ }
+
+ source_layout_dir = current_layout_dir
+
+ # Check collisions and render the layout for the first round
+ layout_log(
+ f"Checking collisions and rendering layout variant {layout_entry.get('layout_id', '')} "
+ f"round {round_index} for group {anchor_id}"
+ )
+ (
+ current_layout_dir,
+ current_render_path,
+ current_mapping_path,
+ current_collision_pairs,
+ ) = _check_collisions_and_render_round(
+ current_layout_dir,
+ round_dir,
+ round_index,
+ layout_suffix,
+ )
+
+ round_records.append(
+ {
+ "round_index": round_index,
+ "source_layout_dir": source_layout_dir,
+ "render_path": current_render_path,
+ "mapping_path": current_mapping_path,
+ "collision_pairs": current_collision_pairs,
+ "out_of_bounds_summary": generator.summarize_out_of_bounds_objects(
+ current_mapping_path,
+ anchor_id=anchor_id,
+ ),
+ }
+ )
+
+ # Initially detect out-of-bounds objects
+ current_out_of_bounds_summary = round_records[-1]["out_of_bounds_summary"]
+ current_out_of_bounds_notes = generator.format_out_of_bounds_modification_notes(
+ current_out_of_bounds_summary
+ )
+
+ if memory is not None:
+ layout_log(f"Recording initial pose and collisions for round {round_index}")
+ memory.record_round(
+ round_index,
+ collisions=current_collision_pairs,
+ )
+
+ # Initialize memory context for VLM agents
+ memory_context = ""
+ if memory is not None and memory.output_path.exists():
+ memory_context = (
+ "LAYOUT_MEMORY:\n"
+ f"{memory.output_path.read_text(encoding='utf-8')}\n\n"
+ )
+
+ while True:
+ if round_index + 1 >= max_refine_rounds:
+ layout_log(f"Reached max refinement rounds for layout {layout_suffix}")
+ break
+
+ # Evaluation step
+ eval_summary, _ = generator.generate_similar_layout_context(
+ mapping_path=current_mapping_path,
+ objects=objects,
+ anchor_id=anchor_id,
+ )
+ collision_context = ""
+ if current_collision_pairs:
+ collision_context = (
+ "POTENTIAL_DEEP_COLLISIONS:\n"
+ f"{json.dumps(current_collision_pairs, ensure_ascii=False, indent=2)}\n"
+ "These pairs may be in collision. Inspect the image and explicitly compute their 2D boundaries "
+ "to confirm deep collision (ignore planar objects like walls, rugs or paper sheets)."
+ "If confirmed, include them in modification_notes with separation guidance.\n\n"
+ )
+ eval_prompt = (
+ f"{evaluate_prompt_template}\n\n"
+ f"ANCHOR_CONSTRAINT: The anchor is {anchor_category or anchor_id} and must not change in the new layout.\n\n"
+ f"ROTATION_CONSTRAINT: Do not change the rotation of any object.\n\n" if relation_type != "attach" else ""
+ f"{cross_anchor_constraint_text}"
+ f"{collision_context}"
+ "CURRENT_LAYOUT_DATA (\"bbox_2d\": [min_u, min_v, max_u, max_v]):\n"
+ f"{json.dumps(eval_summary, ensure_ascii=False, indent=2)}\n\n"
+ f"{memory_context}"
+ )
+
+ layout_log(f"Evaluating layout variant {layout_entry.get('layout_id', '')} round {round_index}")
+ eval_response = self.vlm_client.plan(
+ eval_prompt,
+ images=[str(current_render_path)] if current_render_path.exists() else None,
+ )
+ eval_payload = self.vlm_client.extract_json_payload(eval_response)
+ eval_payload_output = {
+ "vlm_output": eval_payload,
+ "usage": self.vlm_client.last_usage or {},
+ }
+ eval_output_path = round_dir / f"layout_{layout_suffix}_evaluation_r{round_index}.json"
+ with eval_output_path.open("w", encoding="utf-8") as f:
+ json.dump(eval_payload_output, f, ensure_ascii=False, indent=2)
+
+ should_modify = eval_payload.get("should_modify")
+ if isinstance(should_modify, str):
+ should_modify = should_modify.strip().lower() in {"true", "yes", "y", "modify"}
+ if not isinstance(should_modify, bool):
+ decision = str(eval_payload.get("decision", "")).strip().lower()
+ should_modify = decision in {"modify", "yes", "true"}
+
+ modification_notes = eval_payload.get("modification_notes", [])
+ if isinstance(modification_notes, str):
+ modification_notes = [modification_notes]
+ if not isinstance(modification_notes, list):
+ modification_notes = []
+ modification_notes = [str(note) for note in modification_notes if str(note).strip()]
+ modification_notes = generator.merge_modification_notes(
+ modification_notes,
+ current_out_of_bounds_notes,
+ )
+
+ if current_out_of_bounds_notes:
+ should_modify = True
+
+ if memory is not None:
+ layout_log(f"Recording evaluation feedback for round {round_index}")
+ memory.record_round(round_index, modification_notes=modification_notes)
+
+ if not should_modify:
+ layout_log(f"No further modification requested for layout {layout_suffix}")
+ break
+
+ next_round_index = round_index + 1
+ next_round_dir = base_variant_dir / f"round_{next_round_index}"
+ next_round_dir.mkdir(parents=True, exist_ok=True)
+
+ revise_prompt = (
+ f"{revise_prompt_template}\n\n"
+ f"ANCHOR_CONSTRAINT: The anchor is {anchor_category or anchor_id} and must not change in the new layout.\n\n"
+ "CURRENT_LAYOUT_DATA (\"bbox_2d\": [min_u, min_v, max_u, max_v]):\n"
+ f"{json.dumps(eval_summary, ensure_ascii=False, indent=2)}\n\n"
+ "MODIFICATION_GUIDANCE:\n"
+ f"{json.dumps(modification_notes, ensure_ascii=False, indent=2)}\n\n"
+ f"{memory_context}"
+ )
+
+ layout_log(
+ f"Applying modification for layout variant {layout_entry.get('layout_id', '')} round {next_round_index}"
+ )
+ revise_response = self.vlm_client.plan(
+ revise_prompt,
+ images=[str(current_render_path)] if current_render_path.exists() else None,
+ )
+ revise_payload = self.vlm_client.extract_json_payload(revise_response)
+ revise_payload = generator.resolve_pose_expressions(
+ current_mapping_path,
+ revise_payload,
+ anchor_id=anchor_id,
+ )
+ revise_payload_output = {
+ "vlm_output": revise_payload,
+ "usage": self.vlm_client.last_usage or {},
+ }
+
+ if memory is not None:
+ layout_log(f"Recording revised pose for round {next_round_index}")
+ original_pose_expressions = copy.deepcopy(revise_payload.get("pose_expressions"))
+ revised_pose = generator.extract_2d_pose_with_rotation(revise_payload)
+ memory.record_round(
+ next_round_index,
+ revised_pose=revised_pose,
+ pose_expressions=original_pose_expressions,
+ )
+ revise_pose_path = (
+ next_round_dir / f"layout_{layout_suffix}_2d_pose_r{next_round_index}.json"
+ )
+
+ # Convert 2D pose into 3D using the latest grid mapping
+ revise_vlm_output = generator.convert_2d_pose_to_3d(
+ current_mapping_path, revise_payload_output.get("vlm_output", {})
+ )
+ revise_payload_output["vlm_output"] = revise_vlm_output
+ with revise_pose_path.open("w", encoding="utf-8") as f:
+ json.dump(revise_payload_output, f, ensure_ascii=False, indent=2)
+
+ generator.generate_posed_layouts(
+ pose_output_path=revise_pose_path,
+ pose_payload=revise_vlm_output,
+ render_object_ids=render_object_ids,
+ camera_direction=camera_direction,
+ layout_dir=current_layout_dir,
+ anchor_id=anchor_id,
+ render_transform_path=current_mapping_path,
+ )
+
+ current_layout_dir = next_round_dir / "layout_params"
+ if not current_layout_dir.exists():
+ layout_log(
+ f"Warning: Revised layout params not found for {layout_suffix} round {next_round_index}"
+ )
+ break
+
+ source_layout_dir = current_layout_dir
+ previous_mapping_path = current_mapping_path
+ round_index = next_round_index
+ round_dir = next_round_dir
+ (
+ current_layout_dir,
+ current_render_path,
+ current_mapping_path,
+ current_collision_pairs,
+ ) = _check_collisions_and_render_round(
+ current_layout_dir,
+ round_dir,
+ round_index,
+ layout_suffix,
+ )
+
+ if memory is not None:
+ layout_log(f"Recording simulated pose and collisions for round {round_index}")
+ memory.record_round(
+ round_index,
+ collisions=current_collision_pairs,
+ )
+
+ # Evaluate whether to early stop refinement based on collisions and out-of-bounds objects
+ current_out_of_bounds_summary = generator.summarize_out_of_bounds_objects(
+ current_mapping_path,
+ anchor_id=anchor_id,
+ )
+ current_out_of_bounds_notes = generator.format_out_of_bounds_modification_notes(
+ current_out_of_bounds_summary
+ )
+
+ round_records.append(
+ {
+ "round_index": round_index,
+ "source_layout_dir": source_layout_dir,
+ "render_path": current_render_path,
+ "mapping_path": current_mapping_path,
+ "collision_pairs": current_collision_pairs,
+ "out_of_bounds_summary": current_out_of_bounds_summary,
+ }
+ )
+
+ early_stop = generator.should_early_stop_refinement(
+ previous_mapping_path,
+ current_mapping_path,
+ anchor_id=anchor_id,
+ collision_pairs=current_collision_pairs,
+ out_of_bounds_summary=current_out_of_bounds_summary,
+ movement_threshold=refine_movement_threshold,
+ rotation_tolerance=refine_rotation_tolerance,
+ )
+ if early_stop.get("should_stop"):
+ layout_log(
+ "Early stop refinement: "
+ f"max_translation={early_stop.get('max_translation')}, "
+ f"max_rotation_delta={early_stop.get('max_rotation_delta')}, "
+ "no collisions and no out-of-bound objects."
+ )
+ break
+
+ if not round_records:
+ layout_log(f"No valid round records generated for layout {layout_suffix}")
+ return {
+ "layout_suffix": None,
+ "logs": layout_logs,
+ }
+
+ final_round_record = round_records[-1]
+ if not generator._has_layout_issues(
+ final_round_record["collision_pairs"],
+ final_round_record["out_of_bounds_summary"],
+ ):
+ selected_round_record = final_round_record
+ else:
+ candidate_records = round_records[:-1]
+ if not candidate_records:
+ selected_round_record = final_round_record
+ else:
+ selected_round_record = max(
+ candidate_records,
+ key=lambda record: (
+ generator._score_layout_quality(
+ record["collision_pairs"],
+ record["out_of_bounds_summary"],
+ ),
+ record["round_index"],
+ ),
+ )
+
+ selected_layout_dir = group_dir / f"layout_{layout_suffix}" / "layout_params"
+ selected_layout_dir.mkdir(parents=True, exist_ok=True)
+ shutil.copytree(
+ selected_round_record["source_layout_dir"],
+ selected_layout_dir,
+ dirs_exist_ok=True,
+ )
+ layout_log(f"selected_round_record: {selected_round_record['source_layout_dir']}")
+
+ selected_simulated_layout_dir = selected_layout_dir.parent / "layout_params_simulated"
+ selected_simulated_layout_dir.mkdir(parents=True, exist_ok=True)
+
+ gravity = self._compute_gravity_for_anchor(anchor_id, walls_by_id)
+ anchor_layout_path = selected_layout_dir / f"{anchor_id}.json"
+ ground_object_ids = [anchor_id] if anchor_layout_path.exists() else []
+ layout_log(
+ f"Simulating final selected layout {layout_suffix} for group {anchor_id} "
+ f"with gravity {gravity} and ground objects {ground_object_ids}"
+ )
+ self.tool_runner.simulate_scene_physics(
+ selected_layout_dir,
+ artifacts.collision_hulls_dir,
+ selected_simulated_layout_dir,
+ gravity=gravity,
+ object_ids=render_object_ids,
+ cpu_only=True,
+ use_ground_object=bool(ground_object_ids),
+ ground_object_ids=ground_object_ids or None,
+ )
+
+ selected_render_output_path = selected_layout_dir.parent / f"layout_{layout_suffix}_render.png"
+ self.tool_runner.render_scene(
+ layout_dir=selected_simulated_layout_dir,
+ unposed_dir=unposed_dir,
+ output=selected_render_output_path,
+ object_ids=render_object_ids,
+ camera_direction=camera_direction,
+ radius=0.0,
+ image_size=1024,
+ add_grid=True,
+ draw_object_directions=draw_object_directions if relation_type == "attach" else False,
+ draw_object_bboxes=draw_object_bboxes,
+ anchor_id=render_anchor_id,
+ hide_object_ids=[anchor_id],
+ grid_mapping=mapping_path,
+ use_aabb_for_object_bboxes=True if relation_type != "attach" else False
+ )
+
+ return {
+ "layout_suffix": layout_suffix,
+ "logs": layout_logs,
+ }
+
+ valid_layout_entries = [layout_entry for layout_entry in layout_entries if isinstance(layout_entry, dict)]
+ if valid_layout_entries:
+ layout_workers = min(4, len(valid_layout_entries))
+ layout_results: dict[int, dict | None] = {}
+ with ThreadPoolExecutor(max_workers=layout_workers) as layout_executor:
+ layout_futures = {
+ layout_executor.submit(_process_layout_entry, layout_entry, anchor_id, output_path, mapping_path): index
+ for index, layout_entry in enumerate(valid_layout_entries)
+ }
+ for future in as_completed(layout_futures):
+ layout_index = layout_futures[future]
+ try:
+ layout_results[layout_index] = future.result()
+ except Exception as exc:
+ layout_entry = valid_layout_entries[layout_index]
+ layout_results[layout_index] = {
+ "layout_suffix": None,
+ "logs": [
+ f"[{anchor_id}][layout:{layout_entry.get('layout_id', '')}] Failed to process layout for group '{anchor_id}': {exc}"
+ ],
+ }
+
+ for layout_index, layout_entry in enumerate(valid_layout_entries):
+ layout_result = layout_results.get(layout_index) or {}
+ layout_logs = layout_result.get("logs", []) if isinstance(layout_result, dict) else []
+ for line in layout_logs:
+ group_logs.append(line)
+
+ layout_suffix = layout_result.get("layout_suffix") if isinstance(layout_result, dict) else None
+ if layout_suffix:
+ group_log(f"Completed processing layout {layout_suffix} for group '{anchor_id}'")
+ else:
+ group_log(f"Failed to process layout {layout_entry.get('layout_id', '')} for group '{anchor_id}'")
+
+ return {
+ "anchor_id": anchor_id,
+ "num_layout_entries": len(layout_entries),
+ "logs": group_logs,
+ }
+
+ group_outputs = []
+ group_results: dict[str, dict | None] = {}
+ with ThreadPoolExecutor(max_workers=min(8, max(1, len(object_groups)))) as executor:
+ futures = {
+ executor.submit(_process_group, anchor_id, group_info): anchor_id
+ for anchor_id, group_info in object_groups.items()
+ }
+ for future in as_completed(futures):
+ anchor_id = futures[future]
+ group_results[anchor_id] = future.result()
+
+ group_logs_path = generation_output_dir / "group_logs.txt"
+ with group_logs_path.open("w", encoding="utf-8") as log_file:
+ for anchor_id, _group_info in object_groups.items():
+ result = group_results.get(anchor_id)
+ if not result:
+ continue
+ log_file.write(f"=== group {anchor_id} ===\n")
+ for line in result.get("logs", []):
+ log_file.write(f"{line}\n")
+ log_file.write("\n")
+ if result.get("failed"):
+ continue
+ group_outputs.append(
+ {
+ "anchor_id": anchor_id,
+ "num_layout_entries": result.get("num_layout_entries", 0),
+ }
+ )
+
+ print("Scene generation completed for all groups.")
+ return group_outputs
+
+ def run_full_pipeline(self) -> PipelineReport:
+ """Execute optimization, generation, and composition for existing artifacts."""
+ artifacts = SceneArtifacts(
+ unposed_objects_dir=self.paths.unposed_objects_dir,
+ layout_params_dir=self.paths.layout_params_dir,
+ collision_hulls_dir=self.paths.collision_hulls_dir,
+ composed_scene_mesh=self.paths.composed_scene_path,
+ simulated_layout_dir=self.paths.simulated_layout_dir,
+ )
+ artifacts = self._stage_optimization(artifacts)
+ _ = self._stage_generate_scene(artifacts, max_refine_rounds=self.max_rounds)
+
+ combined_scene_paths = LayoutCombination(self.paths, self.tool_runner).run(artifacts)
+ combined_scene_index = self.paths.outputs_dir / "generated" / "combined_scenes" / "combo_index.json"
+ if combined_scene_index.exists():
+ artifacts = replace(artifacts, composed_scene_mesh=combined_scene_index)
+ report = PipelineReport(
+ stage="composition",
+ artifacts=artifacts,
+ tool_outputs={
+ "combined_scene_index": str(combined_scene_index) if combined_scene_index.exists() else "",
+ "combined_scene_count": str(len(combined_scene_paths)),
+ },
+ )
+ self.history.append(report)
+ return report
+
+ def save_report(self, report: PipelineReport, output_path: Path | None = None) -> Path:
+ """Serialize the complete pipeline report to JSON."""
+ if output_path is None:
+ output_path = self.paths.outputs_dir / "pipeline_report.json"
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ payload = {
+ "stage": report.stage,
+ "artifacts": {
+ "layout_params_dir": str(report.artifacts.layout_params_dir),
+ "composed_scene_mesh": str(report.artifacts.composed_scene_mesh),
+ "simulated_layout_dir": str(report.artifacts.simulated_layout_dir),
+ },
+ }
+ with output_path.open("w", encoding="utf-8") as f:
+ json.dump(payload, f, ensure_ascii=False, indent=2)
+ return output_path
diff --git a/vlm_agent/evolution/paths.py b/vlm_agent/evolution/paths.py
new file mode 100644
index 0000000..66900f9
--- /dev/null
+++ b/vlm_agent/evolution/paths.py
@@ -0,0 +1,46 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+
+@dataclass(frozen=True)
+class ProjectPaths:
+ root: Path
+ output_dir: Path | None = None
+
+ @property
+ def tools_dir(self) -> Path:
+ return self.root / "tools"
+
+ @property
+ def outputs_dir(self) -> Path:
+ return self.root / self.output_dir if self.output_dir is not None else self.root / "outputs"
+
+ @property
+ def layout_params_dir(self) -> Path:
+ return self.outputs_dir / "layout_params"
+
+ @property
+ def unposed_objects_dir(self) -> Path:
+ return self.outputs_dir / "unposed_objects"
+
+ @property
+ def collision_hulls_dir(self) -> Path:
+ return self.outputs_dir / "collision_hulls"
+
+ @property
+ def composed_scene_path(self) -> Path:
+ return self.outputs_dir / "composed_scene.glb"
+
+ @property
+ def simulated_layout_dir(self) -> Path:
+ return self.outputs_dir / "layout_params_simulated"
+
+ @property
+ def optimization_log_path(self) -> Path:
+ return self.outputs_dir / "vlm_agent_log.json"
+
+ @property
+ def representative_scenes_dir(self) -> Path:
+ return self.outputs_dir / "representative_scenes"
diff --git a/vlm_agent/evolution/prompts/analyze_cross_anchor_constraints.md b/vlm_agent/evolution/prompts/analyze_cross_anchor_constraints.md
new file mode 100644
index 0000000..dae5134
--- /dev/null
+++ b/vlm_agent/evolution/prompts/analyze_cross_anchor_constraints.md
@@ -0,0 +1,65 @@
+# Cross-Anchor Spatial Constraint Generation Task
+
+You are an intelligent spatial harmony agent specializing in defining the rules and constraints necessary for establishing global coherence in a combined scene layout. Your task is to generate **critical cross-anchor constraints** for objects belonging to different anchors in a given scene template or description, ensuring they align or orient correctly when combined.
+
+### What is an Anchor:
+Each **anchor** represents a physical surface or container in the scene where objects can be placed. Each anchor defines a localized environment containing a specific set of target objects, within which the model remains completely oblivious to any external objects.
+
+### Context:
+Scene layouts are often generated independently for each anchor and then programmatically combined. To prevent global disharmony when these anchors merge, you must generate strict relationship rules (constraints) between objects residing on **different anchors**.
+
+### Input Data:
+You will be provided with three key inputs to analyze the environment:
+1. **Anchor-Based Relations**: JSON data detailing existing parent-child placement dependencies.
+2. **Rendered Visual Map**: A layout visualization showing the spatial arrangement of objects.
+
+### Objective:
+Generate a list of critical cross-anchor constraints. Focus exclusively on two primary relationship types:
+
+1. **Orientation Alignment**: Defining how objects on different anchors must face each other (e.g., chairs on the floor must face the opposite direction of a desk; monitors on a desk must face the same way as the desk).
+2. **Spatial/Semantic Placement Alignment**: Defining positional or proximity dependencies between different anchors (e.g., a whiteboard on a wall anchor must align centrally with a podium anchor on the floor).
+
+### Constraint Generation Guidelines:
+- **Strong Association Only**: Only define a constraint if the two cross-anchor objects have a direct, functional, or strong semantic relationship. Crucially, an alignment constraint should only be included if it represents the single, uniquely reasonable layout configuration. To better illustrate this principle, consider the following specific examples:
+ * *Exclude (Weak associations):* A painting/panel on a wall and a sofa on the floor.
+ * *Exclude (Multi-solution/Ambiguous scenarios):* A wall-mounted display screen and a conference table in the middle of a meeting room (where multiple positioning layouts are acceptable).
+- **Valid References**: Your `layout_note` instructions can *only* use the anchor itself or target objects belonging to its *own* anchor as reference points for positioning and orientation.
+ *(Example: Instead of saying "chairs should face the monitors on the desk", you must state "chairs should face the opposite direction of the desk", because the monitors are invisible when generating the floor anchor.)*
+- **Phrasing and Language Standards**: Keep instructions simple, direct, and avoid overly complex commands. Use the following standardized phrasing patterns:
+ * *For Orientation/Rotation:* Express clearly using active commands to direct facing angles (e.g., "make Object A face the same/opposite direction as Object B", or "orient Object A to Object B").
+ * *For Positioning:* Use explicit spatial relations, such as "keeping Object A stationary", or "positioning Object A at the center / to the left of Object B".
+- **Related Object Formatting**: Like the following examples, when mentioning `related_objects` in `layout_note`, place each object id immediately after its category name.
+- **Distinct Anchors Only**: For each output relationship constraint, `anchor_id` in `anchor_a` and `anchor_id` in `anchor_b` **must not** be the same.
+
+### Output Requirements:
+Return a JSON array containing the identified cross-anchor constraints using the exact structure below:
+
+```json
+[
+ {
+ "description": "The orientation of the chairs on the floor and the monitors on the desk need to face each other. Therefore, using the desk as the intermediate object, the chairs on the floor should face the desk, and the monitors on the desk should share the same orientation as the desk.",
+ "anchor_a": {
+ "anchor_id": "floor",
+ "related_objects": ["object_001", "object_007"],
+ "layout_note": "The desk (object_001) and chair (object_007) must always face each other."
+ },
+ "anchor_b": {
+ "anchor_id": "object_001",
+ "related_objects": ["object_002", "object_003"],
+ "layout_note": "The monitors (object_002, object_003) must always maintain the same facing direction as the desk."
+ }
+ },
+ {
+ "description": "The Wall TV and the TV stand on the floor must align. Since the TV is on the wall anchor and the TV stand is on the floor anchor, they use the physical wall surface as an intermediate reference. Both the TV stand and the Wall TV must be centered horizontally on the same wall.",
+ "anchor_a": {
+ "anchor_id": "floor",
+ "related_objects": ["object_010"],
+ "layout_note": "Place the TV stand (object_010) flat against the floor-to-wall intersection edge, centered horizontally along its length."
+ },
+ "anchor_b": {
+ "anchor_id": "wall_001",
+ "related_objects": ["object_011"],
+ "layout_note": "Mount the TV (object_011) on wall_001, centered horizontally relative to the wall surface."
+ }
+ }
+]
\ No newline at end of file
diff --git a/vlm_agent/evolution/prompts/apply_layout_modification.md b/vlm_agent/evolution/prompts/apply_layout_modification.md
new file mode 100644
index 0000000..114d4a8
--- /dev/null
+++ b/vlm_agent/evolution/prompts/apply_layout_modification.md
@@ -0,0 +1,36 @@
+# Layout Modification Task
+
+You are an intelligent spatial planning agent. Your task is to apply modification guidance to update object poses based on the rendered image, layout description, current layout data, and modification notes.
+
+### Input Data:
+1. **Rendered Scene Image**: An orthogonal camera rendering of the current layout with a coordinate grid. Blue arrows in the rendered image indicate the facing orientation of each object.
+2. **Current Layout Data (JSON)**: Object centers, orientations, and bounding boxes in grid coordinates.
+3. **Modification Guidance**: Qualitative suggestions for how to adjust objects.
+
+### Objective:
+Generate updated 2D center coordinates and rotation angles for objects that need adjustment. Use symbolic expressions and raw numeric values flexibly, referencing existing 2D pose values (center, size_2d, bbox_2d) from CURRENT_LAYOUT_DATA when appropriate. Keep objects within the anchor surface. Keep the anchor object unchanged.
+
+### Output Requirements:
+Return a JSON object with reasoning and symbolic pose updates. Only include objects that should change.
+Example with comments (actual output should be valid JSON without comments):
+```jsonc
+{
+ "reasoning": "Short explanation tied to the modification guidance.",
+ "pose_expressions": [
+ "object_001.center.u = object_000.center.u", // Align object_001 with object_000 along the u axis
+ "object_002.center.u = object_002.center.u + 0.1", // Move object_002 slightly to the right
+ "object_003.center.u = 0.2" // Set object_003's u directly to 0.2
+ "object_004.rotation_degrees = 90" // Orient object_004 to the left
+ ]
+}
+```
+
+Notes:
+- Coordinate convention: u increases to the right, v increases upward; rotation directions use 0 for facing down, 90 for facing left, 180 for facing up, and -90 for facing right.
+- Do not change the anchor object.
+- Only modify objects explicitly mentioned in the modification guidance; keep all other objects unchanged.
+- Use expressions or raw numeric values as appropriate.
+- Allowed references: `object_id.center.u`, `object_id.center.v`, `object_id.size.u`, `object_id.size.v`,
+ `object_id.bbox.min_u`, `object_id.bbox.min_v`, `object_id.bbox.max_u`, `object_id.bbox.max_v`,
+ `object_id.rotation_degrees`. Aliases: `center_2d`, `size_2d`, `bbox_2d`.
+- Please explicitly compute object boundary extents to avoid overlap when adjusting poses.
\ No newline at end of file
diff --git a/vlm_agent/evolution/prompts/evaluate_layout_modification.md b/vlm_agent/evolution/prompts/evaluate_layout_modification.md
new file mode 100644
index 0000000..b4c2ff8
--- /dev/null
+++ b/vlm_agent/evolution/prompts/evaluate_layout_modification.md
@@ -0,0 +1,42 @@
+# Layout Evaluation Task
+
+You are an intelligent spatial evaluation agent. Your task is to judge whether a rendered layout has any hard errors in physical collisions or semantic (unreasonable) placement based on the rendered image and current layout. If there are any errors, please provide suggestions for corrections.
+
+### Input Data:
+1. **Rendered Scene Image**: An orthogonal camera rendering of the current layout with grid overlay. Blue arrows in the rendered image indicate the facing orientation of each object.
+2. **Current Layout Data (JSON)**: Object centers, orientations, and bounding boxes in grid coordinates.
+3. **Potential Deep Collisions (optional)**: Pairs flagged by physics contact checks with contact counts.
+4. **Layout Memory (optional)**: Previous rounds of modification notes, rejected suggestions, revised poses, and collision history for this same layout group.
+5. **Cross Anchor Constraints (optional)**: Additional layout constraints grouped by anchor_id.
+
+### Objective:
+1. Decide whether the current layout has any hard physical collision errors or semantic (unreasonable) placement errors. If a modification is needed, provide high-level guidance without any exact numeric coordinates. Use the direction guidance with the coordinate convention: u increases to the right, v increases upward; rotation directions use 0 for facing down, 90 for facing left, 180 for facing up, and -90 for facing right.
+2. Also provide an overall layout quality score in `quality_score` as an integer between 0 and 100 (higher is better). The score should capture: physical collisions severity, semantic placement reasonableness, spacing and accessibility, adherence to anchor constraints, and surface-boundary compliance. When computing the score, weigh hard physical collisions and semantic errors highest (they should strongly reduce the score); minor spacing or alignment issues reduce the score modestly.
+3. Explicitly inspect both object rotation degrees and the blue arrows in the rendered image. For floor anchors in particular, check whether edge objects are facing inward. Also re-check whether any rotation-related constraints mentioned in `cross anchor constraints` are being satisfied.
+
+### Constraints:
+- Do not change the anchor object.
+- Avoid precise numeric values; use qualitative directions only (e.g., "slightly left", "closer to the wall", "facing down").
+- Keep objects within the anchor surface boundaries and avoid overlaps.
+- If cross anchor constraints are provided, check whether they are satisfied. If any constraint is not satisfied, include it in `modification_notes`.
+- Use one of the following two options, not both:
+ - If modifications are needed for a single object, you may freely change its position and optionally its rotation.
+ - If modifications are needed for a type of object, you may rotate that type in place or apply one uniform translation to all instances of that type, but you must not specify different movements per object; for example, make all chairs face the table, or move all chairs slightly upward by the same amount.
+- When you mention any object in a suggestion, include its object ID in parentheses.
+- Inspect `LAYOUT_MEMORY` before proposing any correction: If the memory reveals a repeated failure pattern (for example, left-right oscillation on one axis, or repeated boundary-violating moves) for the same object or object pair, explicitly state this reasoning and avoid repeating the failed fix or making a symmetric reversal; instead, pivot to a qualitatively different remedy, such as adjusting a different axis, applying a larger or more decisive move, targeting a different object, or making no modification if no safe alternative exists.
+
+### Output Requirements:
+Return a JSON object with the following structure:
+```json
+{
+ "should_modify": true or false,
+ "reasoning": "Explanation of why changes are needed.",
+ "modification_notes": [
+ "Move the chair (object_000) slightly closer to the table (object_001)."
+ ],
+ "quality_basis": "Explanation of basis for the score.",
+ "quality_score": 0-100
+}
+```
+
+If no changes are needed, set `should_modify` to `false` and return an empty `modification_notes` list.
diff --git a/vlm_agent/evolution/prompts/generate_2d_pose.md b/vlm_agent/evolution/prompts/generate_2d_pose.md
new file mode 100644
index 0000000..da65def
--- /dev/null
+++ b/vlm_agent/evolution/prompts/generate_2d_pose.md
@@ -0,0 +1,45 @@
+# Object Placement and Orientation Task
+
+You are an intelligent spatial planning agent. Your task is to determine the optimal placement for a new object within a given 2D orthogonal scene based on the provided visual and textual information.
+
+### Input Data:
+1. **Scene Image**: An orthogonal camera rendering of the current layout with a coordinate grid or reference markers. Blue arrows in the rendered image indicate the facing orientation of each object.
+2. **Layout Description**: A textual description of where each object is placed in this layout and explanation of why they are placed there.
+3. **Current Layout Data (JSON)**: Object centers, orientations, and bounding boxes in grid coordinates.
+4. **Cross Anchor Constraints (optional)**: Additional layout constraints grouped by anchor_id.
+
+### Objective:
+Based on the new layout description and existing layout examples, generate a layout that is different from the existing one, identifying the precise 2D center coordinates (u, v) and the clockwise rotation angle (in degrees) around the axis perpendicular to the screen for the object.
+Coordinate convention: u increases to the right, v increases upward; rotation directions use 0 for facing down, 90 for facing left, 180 for facing up, and -90 for facing right.
+
+### Placement Constraint:
+All objects must be realistically supported by the anchor's surface. Ensure accurate bounding box calculations to prevent objects from overlapping or extending beyond the anchor's physical edges.
+The rotation angle represents the object's facing direction, so it must follow common sense and the scene context: for example, chairs should face toward the center of the table they are placed around, and objects near the edge of a floor should face inward toward the interior of the floor.
+If cross anchor constraints are provided, the generated pose should strictly satisfy them.
+
+### Output Requirements:
+Return a JSON object with the following structure:
+```json
+{
+ "poses": [
+ {
+ "object_id": "string",
+ "reasoning": "Placed based on text description. Coordinates and rotation are calculated using bounding boxes to fit within the anchor's physical edges and avoid overlapping.",
+ "center_coordinates": {
+ "u": float,
+ "v": float
+ },
+ "rotation_degrees": float
+ },
+ {
+ "object_id": "string",
+ ...
+ }
+ ]
+}
+
+Notes:
+- The rotation angle is absolute. Set this value to its original angle if no orientation change is required.
+- Do not change the anchor object.
+- Use precise numeric values for `u`, `v`, and `rotation_degrees`.
+- When generating layouts, explicitly compute the boundaries of each object to avoid collisions.
\ No newline at end of file
diff --git a/vlm_agent/evolution/prompts/generate_2d_pose_fixed_rotation.md b/vlm_agent/evolution/prompts/generate_2d_pose_fixed_rotation.md
new file mode 100644
index 0000000..8732711
--- /dev/null
+++ b/vlm_agent/evolution/prompts/generate_2d_pose_fixed_rotation.md
@@ -0,0 +1,44 @@
+# Object Placement and Orientation Task
+
+You are an intelligent spatial planning agent. Your task is to determine the optimal placement for a new object within a given 2D orthogonal scene based on the provided visual and textual information.
+
+### Input Data:
+1. **Scene Image**: An orthogonal camera rendering of the current layout with a coordinate grid or reference markers. Blue arrows in the rendered image indicate the facing orientation of each object.
+2. **Layout Description**: A textual description of where each object is placed in this layout and explanation of why they are placed there.
+3. **Current Layout Data (JSON)**: Object centers, orientations, and bounding boxes in grid coordinates.
+4. **Cross Anchor Constraints (optional)**: Additional layout constraints grouped by anchor_id.
+
+### Objective:
+Based on the new layout description and existing layout examples, output exactly one new center point for the object being adjusted. Only change the object's center coordinates `(u, v)`; keep `rotation_degrees` unchanged by copying it exactly from the corresponding object in **Current Layout Data**.
+Coordinate convention: u increases to the right, v increases upward; rotation directions use 0 for facing down, 90 for facing left, 180 for facing up, and -90 for facing right. Do not infer, recalculate, or modify the rotation.
+
+### Placement Constraint:
+Only the center coordinates may change. Keep the adjusted object on the anchor's surface and avoid overlap or extending beyond the anchor's physical edges. Do not change the anchor object or any other object's pose. If cross anchor constraints are provided, the new center must strictly satisfy them.
+
+### Output Requirements:
+Return a JSON object with the following structure:
+```json
+{
+ "poses": [
+ {
+ "object_id": "string",
+ "reasoning": "Placed based on text description. Moved only the object center according to the layout constraints; rotation_degrees is copied unchanged from Current Layout Data.",
+ "center_coordinates": {
+ "u": float,
+ "v": float
+ },
+ "rotation_degrees": float
+ },
+ {
+ "object_id": "string",
+ ...
+ }
+ ]
+}
+
+Notes:
+- Change only `center_coordinates.u` and `center_coordinates.v`.
+- `rotation_degrees` must be copied exactly from the same object's entry in **Current Layout Data**. Never change, normalize, infer, or recalculate it.
+- Do not change the anchor object.
+- Use precise numeric values for `u`, `v`, and `rotation_degrees`.
+- When generating layouts, explicitly compute the boundaries of each object to avoid collisions.
\ No newline at end of file
diff --git a/vlm_agent/evolution/prompts/generate_layout_description.md b/vlm_agent/evolution/prompts/generate_layout_description.md
new file mode 100644
index 0000000..6206b1a
--- /dev/null
+++ b/vlm_agent/evolution/prompts/generate_layout_description.md
@@ -0,0 +1,43 @@
+# Alternative Scene Layout Generation Task
+
+You are an intelligent spatial design and architectural planning agent. Your task is to conceptualize and generate alternative, highly distinct layout configurations for a 2D orthogonal scene based on an existing environment layout.
+
+Instead of just shifting a single object or rotating the entire room, you must generate entirely new, structurally diverse layout archetypes that represent different functional or stylistic design philosophies.
+
+### Input Data:
+1. **Current Scene Image**: An orthogonal camera rendering of the current environment with a coordinate grid or reference markers.
+2. **Current Layout Data (JSON)**: 2D center coordinates (u, v) and orientations of all existing objects in the scene.
+3. **Cross Anchor Constraints (optional)**: Additional layout constraints grouped by anchor_id.
+
+### Objective:
+Generate a set of distinctly different, highly representative alternative layouts. Each layout must match a specific spatial design pattern.
+
+### Placement Constraint:
+All objects must be realistically supported by the anchor's surface. Ensure accurate bounding box calculations to prevent objects from overlapping or extending beyond the anchor's physical edges.
+If cross anchor constraints are provided, the generated alternative layouts should strictly satisfy them.
+
+### Output Requirements:
+Return a JSON object containing alternative layout variations. The number of alternative layouts to generate depends on the total number of objects in the current layout:
+- **1-5 objects**: Generate 0 to 1 alternative layout
+- **6-10 objects**: Generate 0 to 2 alternative layouts
+- **More than 10 objects**: Generate 0 to 3 alternative layouts
+
+Do not include coordinate values in the descriptions of the layout variations. **If the existing layout is determined to be the only representative layout, return the alternative layouts array as empty.**
+
+The output must strictly follow this structure (with an example):
+
+```json
+{
+ "alternative_layouts": [
+ {
+ "layout_id": 1,
+ "layout_style_name": "Centralized Focal Point",
+ "layout_reasoning": "The main sofa is positioned flush against the wall, directly facing the TV stand anchored to the opposite side. The coffee table is centered between them, leaving the perimeter walkways completely clear for traffic."
+ },
+ {
+ "layout_id": 2,
+ "layout_style_name": "string",
+ "layout_reasoning": "brief description"
+ }
+ ]
+}
\ No newline at end of file
diff --git a/vlm_agent/evolution/sam3d_utils.py b/vlm_agent/evolution/sam3d_utils.py
new file mode 100644
index 0000000..d4513fa
--- /dev/null
+++ b/vlm_agent/evolution/sam3d_utils.py
@@ -0,0 +1,107 @@
+from __future__ import annotations
+
+import importlib
+from copy import deepcopy
+
+import numpy as np
+import torch
+from pytorch3d.transforms import matrix_to_quaternion, quaternion_to_matrix
+
+_R_ZUP_TO_YUP = np.array([[1, 0, 0], [0, 0, -1], [0, 1, 0]], dtype=np.float32)
+_R_YUP_TO_ZUP = _R_ZUP_TO_YUP.T
+_R_ZUP_TO_P3D = np.array([[-1, 0, 0], [0, 0, 1], [0, 1, 0]], dtype=np.float32)
+_R_P3D_TO_ZUP = _R_ZUP_TO_P3D
+_R_P3D_TO_OPENGL = np.array([[-1, 0, 0], [0, 1, 0], [0, 0, -1]], dtype=np.float32)
+_R_OPENGL_TO_P3D = _R_P3D_TO_OPENGL.T
+
+
+def make_scene_untextured_mesh(*outputs, in_place=False, return_all=False):
+ import trimesh
+ compose_transform = importlib.import_module(
+ "sam3d_objects.data.dataset.tdfy.transforms_3d"
+ ).compose_transform
+
+ if not in_place:
+ outputs = [deepcopy(output) for output in outputs]
+
+ all_meshes = []
+ all_unposed_meshes = []
+ all_layout_params = []
+ for output in outputs:
+ mesh = output["glb"]
+ if mesh is None:
+ continue
+
+ R_l2c = quaternion_to_matrix(output["rotation"])
+ r_p3d_to_zup = torch.from_numpy(_R_P3D_TO_ZUP).to(device=R_l2c.device, dtype=R_l2c.dtype)
+ stored_rotation_matrix = torch.matmul(r_p3d_to_zup, R_l2c)
+ stored_rotation = matrix_to_quaternion(stored_rotation_matrix.transpose(-1, -2))
+
+ all_unposed_meshes.append(deepcopy(mesh))
+
+ all_layout_params.append(
+ {
+ "translation": output["translation"].detach().cpu().tolist(),
+ "rotation": stored_rotation.detach().cpu().tolist(),
+ "scale": output["scale"].detach().cpu().tolist(),
+ }
+ )
+
+ vertices = mesh.vertices.astype(np.float32) @ _R_YUP_TO_ZUP
+ vertices_tensor = torch.from_numpy(vertices).float().to(output["rotation"].device)
+ l2c_transform = compose_transform(
+ scale=output["scale"],
+ rotation=R_l2c,
+ translation=output["translation"],
+ )
+ vertices = l2c_transform.transform_points(vertices_tensor.unsqueeze(0))
+ mesh.vertices = vertices.squeeze(0).cpu().numpy() @ _R_P3D_TO_OPENGL
+ all_meshes.append(mesh)
+
+ if not all_meshes:
+ if return_all:
+ return None, [], [], []
+ return None
+
+ if len(all_meshes) == 1:
+ scene_mesh = all_meshes[0]
+ else:
+ scene_mesh = trimesh.util.concatenate(all_meshes)
+
+ if return_all:
+ return scene_mesh, all_meshes, all_unposed_meshes, all_layout_params
+ return scene_mesh
+
+
+def normalize_scene_to_canonical_cube(scene_mesh, posed_meshes, layout_params, cube_size=1.0, in_place=False):
+ if scene_mesh is None:
+ return scene_mesh, posed_meshes, layout_params
+
+ if not in_place:
+ scene_mesh = deepcopy(scene_mesh)
+ posed_meshes = [deepcopy(mesh) for mesh in posed_meshes]
+ layout_params = deepcopy(layout_params)
+
+ scene_vertices_p3d = scene_mesh.vertices.astype(np.float32) @ _R_OPENGL_TO_P3D
+ bbox_min = scene_vertices_p3d.min(axis=0)
+ bbox_max = scene_vertices_p3d.max(axis=0)
+ center = (bbox_min + bbox_max) / 2.0
+ extent = bbox_max - bbox_min
+ max_extent = float(np.max(extent))
+ if max_extent <= 0:
+ return scene_mesh, posed_meshes, layout_params
+
+ normalize_scale = float(cube_size) / max_extent
+
+ scene_mesh.vertices = ((scene_vertices_p3d - center) * normalize_scale) @ _R_P3D_TO_OPENGL
+ for mesh in posed_meshes:
+ vertices_p3d = mesh.vertices.astype(np.float32) @ _R_OPENGL_TO_P3D
+ mesh.vertices = ((vertices_p3d - center) * normalize_scale) @ _R_P3D_TO_OPENGL
+
+ for params in layout_params:
+ translation = np.asarray(params["translation"], dtype=np.float32)
+ scale = np.asarray(params["scale"], dtype=np.float32)
+ params["translation"] = ((translation - center) * normalize_scale).tolist()
+ params["scale"] = (scale * normalize_scale).tolist()
+
+ return scene_mesh, posed_meshes, layout_params
diff --git a/vlm_agent/evolution/scene_generator.py b/vlm_agent/evolution/scene_generator.py
new file mode 100644
index 0000000..bd22cd4
--- /dev/null
+++ b/vlm_agent/evolution/scene_generator.py
@@ -0,0 +1,1340 @@
+from __future__ import annotations
+
+import ast
+import json
+import math
+import random
+import re
+from dataclasses import dataclass, field
+from pathlib import Path
+
+import numpy as np
+
+from .types import SceneArtifacts
+
+
+@dataclass
+class LayoutMemory:
+ output_path: Path
+ anchor_id: str
+ layout_id: str | None = None
+ initial_pose: list[dict] = field(default_factory=list)
+ rounds: list[dict] = field(default_factory=list)
+
+ def record_initial_pose(self, pose: list[dict] | None) -> None:
+ self.initial_pose = pose or []
+ self.write()
+
+ def record_round(
+ self,
+ round_index: int,
+ simulated_pose: list[dict] | None = None,
+ collisions: list[dict] | None = None,
+ modification_notes: list[str] | None = None,
+ revised_pose: list[dict] | None = None,
+ pose_expressions: list | None = None,
+ ) -> None:
+ entry = None
+ for candidate in self.rounds:
+ if candidate.get("round_index") == round_index:
+ entry = candidate
+ break
+ if entry is None:
+ entry = {
+ "round_index": round_index,
+ "simulated_pose": None,
+ "collisions": None,
+ "modification_notes": None,
+ "revised_pose": None,
+ "pose_expressions": None,
+ }
+ self.rounds.append(entry)
+
+ if simulated_pose is not None:
+ entry["simulated_pose"] = simulated_pose
+ if collisions is not None:
+ entry["collisions"] = collisions
+ if modification_notes is not None:
+ entry["modification_notes"] = modification_notes
+ if revised_pose is not None:
+ entry["revised_pose"] = revised_pose
+ if pose_expressions is not None:
+ entry["pose_expressions"] = pose_expressions
+
+ self.write()
+
+ def _format_number(self, value: float | int | None) -> str:
+ if value is None:
+ return ""
+ if isinstance(value, (int, float)):
+ return str(round(float(value), 3))
+ return str(value)
+
+ def _format_pose_list(self, pose_list: list[dict] | None, include_rotation: bool) -> list[str]:
+ if not pose_list:
+ return ["- (none)"]
+ lines = []
+ for pose in pose_list:
+ if not isinstance(pose, dict):
+ continue
+ object_id = pose.get("object_id")
+ center = pose.get("center_2d")
+ if not isinstance(center, list) or len(center) < 2:
+ continue
+ u_val, v_val = center[0], center[1]
+ line = f"- {object_id}: u={self._format_number(u_val)}, v={self._format_number(v_val)}"
+ if include_rotation:
+ rotation = pose.get("rotation_degrees")
+ if rotation is not None:
+ line += f", rotation_degrees={self._format_number(rotation)}"
+ lines.append(line)
+ return lines or ["- (none)"]
+
+ def _format_collisions(self, collisions: list[dict] | None) -> list[str]:
+ if not collisions:
+ return ["- (none)"]
+ lines = []
+ for item in collisions:
+ if not isinstance(item, dict):
+ continue
+ a_id = item.get("a")
+ b_id = item.get("b")
+ contacts = item.get("contacts")
+ line = f"- {a_id} vs {b_id} (contacts={self._format_number(contacts)})"
+ lines.append(line)
+ return lines or ["- (none)"]
+
+ def _format_notes(self, notes: list[str] | None) -> list[str]:
+ if not notes:
+ return ["- (none)"]
+ return [f"- {note}" for note in notes]
+
+ def _format_pose_expressions(self, expressions: list | None) -> list[str]:
+ if not expressions:
+ return ["- (none)"]
+ lines = []
+ for expression in expressions:
+ if isinstance(expression, str):
+ lines.append(f"- {expression}")
+ else:
+ lines.append(f"- {json.dumps(expression, ensure_ascii=False)}")
+ return lines or ["- (none)"]
+
+ def write(self) -> None:
+ lines: list[str] = ["# Layout Memory", ""]
+ lines.append(f"- anchor_id: {self.anchor_id}")
+ if self.layout_id is not None:
+ lines.append(f"- layout_id: {self.layout_id}")
+ lines.append("")
+ lines.append("## Initial 2D Pose")
+ lines.extend(self._format_pose_list(self.initial_pose, include_rotation=True))
+
+ sorted_rounds = sorted(self.rounds, key=lambda item: item.get("round_index", 0))
+ for entry in sorted_rounds:
+ round_index = entry.get("round_index")
+ lines.append("")
+ lines.append(f"## Round {round_index}")
+
+ if entry.get("pose_expressions") is not None:
+ lines.append("- Original Pose Expressions")
+ lines.extend(self._format_pose_expressions(entry.get("pose_expressions")))
+
+ if entry.get("revised_pose") is not None:
+ lines.append("- Revised 2D Pose")
+ lines.extend(self._format_pose_list(entry.get("revised_pose"), include_rotation=True))
+
+ if entry.get("simulated_pose") is not None:
+ lines.append("- 2D Pose after Simulation ")
+ lines.extend(self._format_pose_list(entry.get("simulated_pose"), include_rotation=True))
+
+ if entry.get("collisions") is not None:
+ lines.append("- Possible Deep Collisions")
+ lines.extend(self._format_collisions(entry.get("collisions")))
+
+ if entry.get("modification_notes") is not None:
+ lines.append("- Modification Notes for Next Round")
+ lines.extend(self._format_notes(entry.get("modification_notes")))
+
+ self.output_path.parent.mkdir(parents=True, exist_ok=True)
+ self.output_path.write_text("\n".join(lines).rstrip() + "\n", encoding="utf-8")
+
+
+class SceneGenerator:
+ """Generate new scenes based on learned relationships and original artifacts."""
+
+ def __init__(self, original_artifacts: SceneArtifacts):
+ self.original_artifacts = original_artifacts
+
+ def _score_layout_quality(
+ self,
+ collision_pairs: list[dict] | None,
+ out_of_bounds_summary: dict | None,
+ ) -> int:
+ collision_count = len(collision_pairs or [])
+ out_of_bounds_count = 0
+ if isinstance(out_of_bounds_summary, dict):
+ entries = out_of_bounds_summary.get("out_of_bounds_objects", [])
+ if isinstance(entries, list):
+ out_of_bounds_count = len(entries)
+ return max(0, 100 - 20 * (collision_count + out_of_bounds_count))
+
+ def _has_layout_issues(
+ self,
+ collision_pairs: list[dict] | None,
+ out_of_bounds_summary: dict | None,
+ ) -> bool:
+ if collision_pairs:
+ return True
+ if not isinstance(out_of_bounds_summary, dict):
+ return False
+ entries = out_of_bounds_summary.get("out_of_bounds_objects", [])
+ return isinstance(entries, list) and bool(entries)
+
+ def generate_similar_layout_context(
+ self,
+ mapping_path: Path | None = None,
+ objects: list[dict] | None = None,
+ anchor_id: str | None = None,
+ ) -> dict:
+ """Create the layout context for generate_layout_description prompt."""
+ objects_by_id = {}
+ if objects:
+ for obj in objects:
+ if not isinstance(obj, dict):
+ continue
+ object_id = str(obj.get("id", ""))
+ objects_by_id[object_id] = {
+ "category": obj.get("category", ""),
+ "description": obj.get("description", ""),
+ }
+
+ objects_context = {
+ "objects": [],
+ }
+
+ def _round_numbers(value):
+ if isinstance(value, float | int):
+ return round(value, 2)
+ if isinstance(value, list):
+ return [_round_numbers(item) for item in value]
+ if isinstance(value, dict):
+ return {k: _round_numbers(v) for k, v in value.items()}
+ return value
+
+ anchor_category = None
+ if mapping_path and mapping_path.exists():
+ with mapping_path.open("r", encoding="utf-8") as f:
+ mapping_payload = json.load(f)
+
+ mapping_objects = mapping_payload.get("objects", [])
+ if isinstance(mapping_objects, list):
+ for obj_info in mapping_objects:
+ if not isinstance(obj_info, dict):
+ continue
+ object_id = str(obj_info.get("object_id", ""))
+ meta = objects_by_id.get(object_id)
+ if meta is None and object_id.startswith("object_"):
+ alt_id = f"obj_{object_id.split('_', 1)[1]}"
+ meta = objects_by_id.get(alt_id)
+ category = (meta or {}).get("category", object_id)
+ description = (meta or {}).get("description", object_id)
+
+ if anchor_id and object_id == anchor_id:
+ anchor_category = category
+ description += " (This object is the anchor and should not change in the new layout)"
+
+ obj_info = {k: v for k, v in obj_info.items() if k != "signed_distance_to_plane"}
+ obj_info["category"] = category
+ obj_info["description"] = description
+ objects_context["objects"].append(_round_numbers(obj_info))
+
+ return {"objects_context": objects_context}, anchor_category
+
+ def create_layout_memory(
+ self,
+ output_path: Path,
+ anchor_id: str,
+ layout_id: str | None = None,
+ ) -> LayoutMemory:
+ return LayoutMemory(output_path=output_path, anchor_id=anchor_id, layout_id=layout_id)
+
+ def extract_2d_pose_centers(self, pose_payload: dict) -> list[dict]:
+ return self._extract_2d_pose_entries(pose_payload, include_rotation=False)
+
+ def extract_2d_pose_with_rotation(self, pose_payload: dict) -> list[dict]:
+ return self._extract_2d_pose_entries(pose_payload, include_rotation=True)
+
+ def extract_2d_pose_from_mapping(self, mapping_path: Path | None) -> list[dict]:
+ if not mapping_path or not mapping_path.exists():
+ return []
+ with mapping_path.open("r", encoding="utf-8") as f:
+ mapping_payload = json.load(f)
+
+ objects = mapping_payload.get("objects", [])
+ if not isinstance(objects, list):
+ return []
+ results = []
+ for obj in objects:
+ if not isinstance(obj, dict):
+ continue
+ center = obj.get("center_2d")
+ if not isinstance(center, list) or len(center) < 2:
+ continue
+ object_id = obj.get("object_id")
+ entry = {
+ "object_id": object_id,
+ "center_2d": [center[0], center[1]],
+ }
+ rotation = obj.get("rotation_degrees")
+ if rotation is not None:
+ entry["rotation_degrees"] = rotation
+ results.append(entry)
+ return results
+
+ def _parse_bbox_2d(self, obj: dict) -> list[float] | None:
+ bbox = obj.get("bbox_2d")
+ if not isinstance(bbox, list) or len(bbox) < 4:
+ return None
+ try:
+ return [float(bbox[0]), float(bbox[1]), float(bbox[2]), float(bbox[3])]
+ except (TypeError, ValueError):
+ return None
+
+ def summarize_out_of_bounds_objects(
+ self,
+ mapping_path: Path | None,
+ anchor_id: str | None,
+ ) -> dict:
+ if not mapping_path or not mapping_path.exists():
+ return {
+ "anchor_id": anchor_id,
+ "anchor_bbox_2d": None,
+ "out_of_bounds_objects": [],
+ }
+
+ with mapping_path.open("r", encoding="utf-8") as f:
+ mapping_payload = json.load(f)
+
+ objects = mapping_payload.get("objects", [])
+ if not isinstance(objects, list):
+ return {
+ "anchor_id": anchor_id,
+ "anchor_bbox_2d": None,
+ "out_of_bounds_objects": [],
+ }
+
+ anchor_bbox = None
+ for obj in objects:
+ if not isinstance(obj, dict):
+ continue
+ if str(obj.get("object_id")) != str(anchor_id):
+ continue
+ anchor_bbox = self._parse_bbox_2d(obj)
+ break
+
+ summary = {
+ "anchor_id": anchor_id,
+ "anchor_bbox_2d": anchor_bbox,
+ "out_of_bounds_objects": [],
+ }
+ if anchor_bbox is None:
+ return summary
+
+ anchor_min_u, anchor_min_v, anchor_max_u, anchor_max_v = anchor_bbox
+ for obj in objects:
+ if not isinstance(obj, dict):
+ continue
+ object_id = obj.get("object_id")
+ if object_id is None or str(object_id) == str(anchor_id):
+ continue
+
+ bbox = self._parse_bbox_2d(obj)
+ if bbox is None:
+ continue
+
+ min_u, min_v, max_u, max_v = bbox
+ u_buffer = (max_u - min_u) / 2.0 if anchor_id.startswith("object_") else 0.0
+ v_buffer = (max_v - min_v) / 2.0 if anchor_id.startswith("object_") else 0.0
+ exceeded_sides = []
+ if min_u < anchor_min_u - u_buffer - 1e-2:
+ exceeded_sides.append("left")
+ if min_v < anchor_min_v - v_buffer - 1e-2:
+ exceeded_sides.append("bottom")
+ if max_u > anchor_max_u + u_buffer + 1e-2:
+ exceeded_sides.append("right")
+ if max_v > anchor_max_v + v_buffer + 1e-2:
+ exceeded_sides.append("top")
+
+ if exceeded_sides:
+ summary["out_of_bounds_objects"].append(
+ {
+ "object_id": object_id,
+ "bbox_2d": [min_u, min_v, max_u, max_v],
+ "exceeded_sides": exceeded_sides,
+ }
+ )
+
+ return summary
+
+ def format_out_of_bounds_modification_notes(self, out_of_bounds_summary: dict | None) -> list[str]:
+ if not isinstance(out_of_bounds_summary, dict):
+ return []
+ entries = out_of_bounds_summary.get("out_of_bounds_objects", [])
+ if not isinstance(entries, list):
+ return []
+
+ notes: list[str] = []
+ for entry in entries:
+ if not isinstance(entry, dict):
+ continue
+ object_id = entry.get("object_id")
+ if not object_id:
+ continue
+ exceeded_sides = entry.get("exceeded_sides", [])
+ if isinstance(exceeded_sides, list) and exceeded_sides:
+ side_text = ", ".join(str(side) for side in exceeded_sides)
+ notes.append(
+ f"Move ({object_id}) inward so its bbox stays inside the anchor boundary (exceeded: {side_text})."
+ )
+ else:
+ notes.append(f"Move ({object_id}) inward so it stays inside the anchor boundary.")
+ return notes
+
+ def merge_modification_notes(
+ self,
+ base_notes: list[str] | None,
+ out_of_bounds_notes: list[str] | None,
+ ) -> list[str]:
+ merged: list[str] = []
+ for note_list in (base_notes or [], out_of_bounds_notes or []):
+ for note in note_list:
+ text = str(note).strip()
+ if not text:
+ continue
+ if text not in merged:
+ merged.append(text)
+ return merged
+
+ def _extract_mapping_pose_by_id(
+ self,
+ mapping_path: Path | None,
+ anchor_id: str | None = None,
+ ) -> dict[str, dict[str, float | None]]:
+ if not mapping_path or not mapping_path.exists():
+ return {}
+ with mapping_path.open("r", encoding="utf-8") as f:
+ mapping_payload = json.load(f)
+
+ objects = mapping_payload.get("objects", [])
+ if not isinstance(objects, list):
+ return {}
+
+ pose_by_id: dict[str, dict[str, float | None]] = {}
+ for obj in objects:
+ if not isinstance(obj, dict):
+ continue
+ object_id = obj.get("object_id")
+ if object_id is None:
+ continue
+ object_id = str(object_id)
+ if anchor_id is not None and object_id == str(anchor_id):
+ continue
+
+ center = obj.get("center_2d")
+ if not isinstance(center, list) or len(center) < 2:
+ continue
+ try:
+ u_val = float(center[0])
+ v_val = float(center[1])
+ except (TypeError, ValueError):
+ continue
+
+ rotation = obj.get("rotation_degrees")
+ rotation_value = None
+ if isinstance(rotation, (int, float)):
+ rotation_value = float(rotation)
+
+ pose_by_id[object_id] = {
+ "u": u_val,
+ "v": v_val,
+ "rotation_degrees": rotation_value,
+ }
+
+ return pose_by_id
+
+ def summarize_pose_change(
+ self,
+ previous_mapping_path: Path | None,
+ current_mapping_path: Path | None,
+ *,
+ anchor_id: str | None = None,
+ ) -> dict:
+ prev_pose = self._extract_mapping_pose_by_id(previous_mapping_path, anchor_id=anchor_id)
+ curr_pose = self._extract_mapping_pose_by_id(current_mapping_path, anchor_id=anchor_id)
+ shared_ids = sorted(set(prev_pose.keys()) & set(curr_pose.keys()))
+
+ deltas: list[dict] = []
+ max_translation = 0.0
+ max_rotation_delta = 0.0
+ for object_id in shared_ids:
+ prev_entry = prev_pose[object_id]
+ curr_entry = curr_pose[object_id]
+
+ dx = float(curr_entry["u"]) - float(prev_entry["u"])
+ dy = float(curr_entry["v"]) - float(prev_entry["v"])
+ translation = math.sqrt(dx * dx + dy * dy)
+ max_translation = max(max_translation, translation)
+
+ prev_rot = prev_entry.get("rotation_degrees")
+ curr_rot = curr_entry.get("rotation_degrees")
+ rotation_delta = 0.0
+ if isinstance(prev_rot, (int, float)) and isinstance(curr_rot, (int, float)):
+ rotation_delta = abs(float(curr_rot) - float(prev_rot))
+ max_rotation_delta = max(max_rotation_delta, rotation_delta)
+
+ deltas.append(
+ {
+ "object_id": object_id,
+ "translation": translation,
+ "rotation_delta": rotation_delta,
+ }
+ )
+
+ return {
+ "object_count": len(shared_ids),
+ "max_translation": max_translation,
+ "max_rotation_delta": max_rotation_delta,
+ "object_deltas": deltas,
+ }
+
+ def should_early_stop_refinement(
+ self,
+ previous_mapping_path: Path | None,
+ current_mapping_path: Path | None,
+ *,
+ anchor_id: str | None,
+ collision_pairs: list[dict] | None,
+ out_of_bounds_summary: dict | None,
+ movement_threshold: float,
+ rotation_tolerance: float,
+ ) -> dict:
+ change_summary = self.summarize_pose_change(
+ previous_mapping_path,
+ current_mapping_path,
+ anchor_id=anchor_id,
+ )
+ object_count = int(change_summary.get("object_count", 0))
+ max_translation = float(change_summary.get("max_translation", 0.0))
+ max_rotation_delta = float(change_summary.get("max_rotation_delta", 0.0))
+
+ has_collision = bool(collision_pairs)
+ out_of_bounds_objects = []
+ if isinstance(out_of_bounds_summary, dict):
+ raw = out_of_bounds_summary.get("out_of_bounds_objects", [])
+ if isinstance(raw, list):
+ out_of_bounds_objects = raw
+ has_out_of_bounds = bool(out_of_bounds_objects)
+
+ should_stop = (
+ object_count > 0
+ and max_rotation_delta <= float(rotation_tolerance)
+ and max_translation < float(movement_threshold)
+ and not has_collision
+ and not has_out_of_bounds
+ )
+
+ return {
+ "should_stop": should_stop,
+ "object_count": object_count,
+ "max_translation": max_translation,
+ "max_rotation_delta": max_rotation_delta,
+ "has_collision": has_collision,
+ "has_out_of_bounds": has_out_of_bounds,
+ }
+
+ def _extract_2d_pose_entries(self, pose_payload: dict, include_rotation: bool) -> list[dict]:
+ if not isinstance(pose_payload, dict):
+ return []
+ poses = pose_payload.get("poses", [])
+ if not isinstance(poses, list):
+ return []
+ results = []
+ for pose in poses:
+ if not isinstance(pose, dict):
+ continue
+ coords = pose.get("center_coordinates", {})
+ if not isinstance(coords, dict):
+ continue
+ u_val = coords.get("u")
+ v_val = coords.get("v")
+ if u_val is None or v_val is None:
+ continue
+ object_id = pose.get("object_id") or pose.get("id")
+ entry = {
+ "object_id": object_id,
+ "center_2d": [u_val, v_val],
+ }
+ if include_rotation:
+ rotation = pose.get("rotation_degrees")
+ if rotation is not None:
+ entry["rotation_degrees"] = rotation
+ results.append(entry)
+ return results
+
+ def _load_grid_mapping(self, mapping_path: Path) -> tuple[int | None, dict, dict]:
+ grid_size = None
+ grid_map = {}
+ with mapping_path.open("r", encoding="utf-8") as f:
+ mapping_payload = json.load(f)
+
+ raw_grid_size = mapping_payload.get("grid_size")
+ if isinstance(raw_grid_size, int):
+ grid_size = raw_grid_size
+
+ grid_points = mapping_payload.get("grid_points", [])
+ if isinstance(grid_points, list):
+ for point in grid_points:
+ if not isinstance(point, dict):
+ continue
+ i = point.get("grid_i")
+ j = point.get("grid_j")
+ p3d = point.get("point_3d_original")
+ if isinstance(i, int) and isinstance(j, int) and isinstance(p3d, list) and len(p3d) == 3:
+ grid_map[(i, j)] = p3d
+
+ return grid_size, grid_map, mapping_payload
+
+ def _nearest_grid_point(self, grid_map: dict, u: float, v: float) -> list[float] | None:
+ best = None
+ best_dist = None
+ for (i, j), point in grid_map.items():
+ dist = (i - u) ** 2 + (j - v) ** 2
+ if best_dist is None or dist < best_dist:
+ best_dist = dist
+ best = point
+ return best
+
+ def _interpolate_grid_point(
+ self,
+ grid_map: dict,
+ u: float,
+ v: float,
+ max_index: int,
+ ) -> list[float] | None:
+ if max_index < 0:
+ return None
+ u_clamped = max(0.0, min(float(max_index), u))
+ v_clamped = max(0.0, min(float(max_index), v))
+
+ i0 = int(math.floor(u_clamped))
+ j0 = int(math.floor(v_clamped))
+ i1 = min(i0 + 1, max_index)
+ j1 = min(j0 + 1, max_index)
+
+ p00 = grid_map.get((i0, j0))
+ p10 = grid_map.get((i1, j0))
+ p01 = grid_map.get((i0, j1))
+ p11 = grid_map.get((i1, j1))
+
+ if i0 == i1 and j0 == j1:
+ if p00:
+ return p00
+ return self._nearest_grid_point(grid_map, u_clamped, v_clamped)
+
+ if j0 == j1 and p00 and p10:
+ t = u_clamped - i0
+ return [
+ (1 - t) * p00[0] + t * p10[0],
+ (1 - t) * p00[1] + t * p10[1],
+ (1 - t) * p00[2] + t * p10[2],
+ ]
+
+ if i0 == i1 and p00 and p01:
+ s = v_clamped - j0
+ return [
+ (1 - s) * p00[0] + s * p01[0],
+ (1 - s) * p00[1] + s * p01[1],
+ (1 - s) * p00[2] + s * p01[2],
+ ]
+
+ if not (p00 and p10 and p01 and p11):
+ return self._nearest_grid_point(grid_map, u_clamped, v_clamped)
+
+ t = u_clamped - i0
+ s = v_clamped - j0
+
+ return [
+ (1 - t) * (1 - s) * p00[0] + t * (1 - s) * p10[0] + (1 - t) * s * p01[0] + t * s * p11[0],
+ (1 - t) * (1 - s) * p00[1] + t * (1 - s) * p10[1] + (1 - t) * s * p01[1] + t * s * p11[1],
+ (1 - t) * (1 - s) * p00[2] + t * (1 - s) * p10[2] + (1 - t) * s * p01[2] + t * s * p11[2],
+ ]
+
+ def _quaternion_from_axis_angle(self, axis: list, angle_degrees: float) -> list:
+ angle_rad = math.radians(angle_degrees)
+ half_angle = angle_rad / 2.0
+ sin_half = math.sin(half_angle)
+ cos_half = math.cos(half_angle)
+
+ w = cos_half
+ x = axis[0] * sin_half
+ y = axis[1] * sin_half
+ z = axis[2] * sin_half
+
+ return [w, x, y, z]
+
+ def _camera_axes(self, camera_direction: list) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ forward = np.asarray(camera_direction, dtype=np.float32).reshape(3)
+ norm = np.linalg.norm(forward)
+ if norm > 1e-6:
+ forward = forward / norm
+ up_hint = np.array([0.0, 1.0, 0.0], dtype=np.float32)
+ right = np.cross(up_hint, forward)
+ if np.linalg.norm(right) <= 1e-8:
+ up_hint = np.array([0.0, 0.0, 1.0], dtype=np.float32)
+ right = np.cross(up_hint, forward)
+ right = right / np.linalg.norm(right)
+ up = np.cross(forward, right)
+ up = up / np.linalg.norm(up)
+ return right, up, forward
+
+ def _quaternion_to_matrix(self, quat: list | np.ndarray) -> np.ndarray:
+ q = np.asarray(quat, dtype=np.float32).reshape(4)
+ norm = np.linalg.norm(q)
+ if norm > 1e-8:
+ q = q / norm
+ w, x, y, z = q
+ return np.array(
+ [
+ [1 - 2 * (y * y + z * z), 2 * (x * y - z * w), 2 * (x * z + y * w)],
+ [2 * (x * y + z * w), 1 - 2 * (x * x + z * z), 2 * (y * z - x * w)],
+ [2 * (x * z - y * w), 2 * (y * z + x * w), 1 - 2 * (x * x + y * y)],
+ ],
+ dtype=np.float32,
+ )
+
+ def _matrix_to_quaternion(self, matrix: np.ndarray) -> list:
+ m = np.asarray(matrix, dtype=np.float32)
+ trace = float(m[0, 0] + m[1, 1] + m[2, 2])
+ if trace > 0.0:
+ s = math.sqrt(trace + 1.0) * 2.0
+ w = 0.25 * s
+ x = (m[2, 1] - m[1, 2]) / s
+ y = (m[0, 2] - m[2, 0]) / s
+ z = (m[1, 0] - m[0, 1]) / s
+ elif m[0, 0] > m[1, 1] and m[0, 0] > m[2, 2]:
+ s = math.sqrt(1.0 + m[0, 0] - m[1, 1] - m[2, 2]) * 2.0
+ w = (m[2, 1] - m[1, 2]) / s
+ x = 0.25 * s
+ y = (m[0, 1] + m[1, 0]) / s
+ z = (m[0, 2] + m[2, 0]) / s
+ elif m[1, 1] > m[2, 2]:
+ s = math.sqrt(1.0 + m[1, 1] - m[0, 0] - m[2, 2]) * 2.0
+ w = (m[0, 2] - m[2, 0]) / s
+ x = (m[0, 1] + m[1, 0]) / s
+ y = 0.25 * s
+ z = (m[1, 2] + m[2, 1]) / s
+ else:
+ s = math.sqrt(1.0 + m[2, 2] - m[0, 0] - m[1, 1]) * 2.0
+ w = (m[1, 0] - m[0, 1]) / s
+ x = (m[0, 2] + m[2, 0]) / s
+ y = (m[1, 2] + m[2, 1]) / s
+ z = 0.25 * s
+ return [float(w), float(x), float(y), float(z)]
+
+ def _euler_from_matrix_xyz(self, matrix: np.ndarray) -> tuple[float, float, float]:
+ r00, r10, r20 = matrix[0, 0], matrix[1, 0], matrix[2, 0]
+ r21, r22 = matrix[2, 1], matrix[2, 2]
+ if r20 < -0.995:
+ x = 0.0
+ y = math.pi / 2
+ z = math.atan2(matrix[0, 1], matrix[1, 1])
+ elif r20 > 0.995:
+ x = 0.0
+ y = -math.pi / 2
+ z = -math.atan2(matrix[0, 1], matrix[1, 1])
+ else:
+ x = math.atan2(r21, r22)
+ y = math.asin(-r20)
+ z = math.atan2(r10, r00)
+ return x, y, z
+
+ def _matrix_from_euler_xyz(self, x: float, y: float, z: float) -> np.ndarray:
+ cx, sx = math.cos(x), math.sin(x)
+ cy, sy = math.cos(y), math.sin(y)
+ cz, sz = math.cos(z), math.sin(z)
+
+ return np.array(
+ [
+ [cz * cy, cz * sy * sx - sz * cx, cz * sy * cx + sz * sx],
+ [sz * cy, sz * sy * sx + cz * cx, sz * sy * cx - cz * sx],
+ [-sy, cy * sx, cy * cx],
+ ],
+ dtype=np.float32,
+ )
+
+ def update_rotation_about_camera_direction(
+ self,
+ rotation_degrees: float,
+ original_quaternion: list,
+ camera_direction: list,
+ ) -> list:
+ """Update the Euler angle around camera_direction and return a new quaternion."""
+ quat = np.asarray(original_quaternion, dtype=np.float32).reshape(-1)
+ if quat.size != 4:
+ raise ValueError("Original rotation must have 4 elements.")
+
+ right, up, forward = self._camera_axes(camera_direction)
+ basis = np.stack([right, up, forward], axis=1)
+ rotation_world = self._quaternion_to_matrix(quat)
+ rotation_local = basis.T @ rotation_world @ basis
+ euler_x, euler_y, _ = self._euler_from_matrix_xyz(rotation_local)
+ euler_z = math.radians(float(rotation_degrees))
+ rotation_local = self._matrix_from_euler_xyz(euler_x, euler_y, euler_z)
+ rotation_world = basis @ rotation_local @ basis.T
+ return self._matrix_to_quaternion(rotation_world)
+
+ def generate_posed_layouts(
+ self,
+ pose_output_path: Path,
+ pose_payload: dict,
+ render_object_ids: list,
+ camera_direction: list,
+ layout_dir: Path,
+ anchor_id: str,
+ render_transform_path: Path | None = None,
+ ) -> None:
+ """Generate layout JSON files from 2D pose estimates.
+
+ Args:
+ pose_output_path: Path to the 2D pose output JSON file
+ pose_payload: Parsed pose payload dict
+ render_object_ids: List of object IDs that were rendered
+ camera_direction: Camera direction vector for this anchor
+ layout_dir: Source layout directory (layout_params_simulated)
+ anchor_id: ID of the anchor object
+ """
+ layout_output_dir = pose_output_path.parent / "layout_params"
+ layout_output_dir.mkdir(parents=True, exist_ok=True)
+
+ # Extract pose list from payload
+ poses = pose_payload.get("poses", [])
+ if not isinstance(poses, list):
+ return
+
+ # Build a mapping of pose object_id to pose data
+ pose_map = {}
+ for pose in poses:
+ if isinstance(pose, dict):
+ obj_id = pose.get("object_id")
+ if obj_id:
+ pose_map[str(obj_id)] = pose
+
+ # Normalize camera direction
+ camera_dir_array = np.array(camera_direction, dtype=np.float32)
+ camera_dir_norm = np.linalg.norm(camera_dir_array)
+ if camera_dir_norm > 1e-6:
+ camera_direction_normalized = camera_dir_array / camera_dir_norm
+ else:
+ camera_direction_normalized = camera_dir_array
+
+ # Align object rotation with the normalized scene coordinate system in the rendering script
+ rot_fix = 0.0
+ if abs(camera_direction_normalized[0]) < 1e-6 and abs(camera_direction_normalized[2]) < 1e-6:
+ if render_transform_path and render_transform_path.exists():
+ with render_transform_path.open("r", encoding="utf-8") as f:
+ render_transform = json.load(f)
+
+ transform_info = render_transform.get("transform_info", {})
+ rotation_matrix = transform_info.get("rotation_matrix_col_major")
+ if isinstance(rotation_matrix, list) and len(rotation_matrix) == 3:
+ rot_mat = np.asarray(rotation_matrix, dtype=np.float32).reshape(3, 3)
+ forward_vec = rot_mat @ np.array([0.0, 0.0, 1.0], dtype=np.float32)
+ rot_fix = math.degrees(math.atan2(float(forward_vec[0]), float(forward_vec[2])))
+
+ # Process each object in render_object_ids
+ for obj_id in render_object_ids:
+ obj_id_str = str(obj_id)
+
+ # Load source layout from layout_dir
+ source_layout_path = layout_dir / f"{obj_id_str}.json"
+ if not source_layout_path.exists():
+ print(f"Warning: Source layout not found for {obj_id_str}")
+ continue
+
+ with source_layout_path.open("r", encoding="utf-8") as f:
+ source_layout = json.load(f)
+
+ # If object not in pose_map, it means it wasn't moved (anchor case)
+ if obj_id_str not in pose_map or obj_id_str == anchor_id:
+ # Copy anchor layout as-is
+ output_layout_path = layout_output_dir / f"{obj_id_str}.json"
+ with output_layout_path.open("w", encoding="utf-8") as f:
+ json.dump(source_layout, f, ensure_ascii=False, indent=2)
+ continue
+
+ pose_data = pose_map[obj_id_str]
+
+ # Extract 3D coordinates from pose
+ coords_3d = pose_data.get("center_coordinates_3d", {})
+ if not isinstance(coords_3d, dict):
+ print(f"Warning: Invalid 3D coordinates for {obj_id_str}")
+ continue
+
+ x = coords_3d.get("x")
+ y = coords_3d.get("y")
+ z = coords_3d.get("z")
+
+ if x is None or y is None or z is None:
+ print(f"Warning: Incomplete 3D coordinates for {obj_id_str}")
+ continue
+
+ # Extract rotation in degrees
+ rotation_degrees = pose_data.get("rotation_degrees", None)
+
+ # Create new layout with source rotation/scale
+ original_rotation = source_layout.get("rotation", [[1, 0, 0, 0]])
+ new_layout = {
+ "translation": [[x, y, z]],
+ "rotation": original_rotation,
+ "scale": source_layout.get("scale", [[1, 1, 1]]),
+ }
+
+ # Apply rotation
+ if rotation_degrees is not None:
+ # Get original rotation quaternion
+ if isinstance(original_rotation, list) and len(original_rotation) > 0:
+ if len(original_rotation) == 4 and all(
+ isinstance(val, (int, float)) for val in original_rotation
+ ):
+ orig_quat = original_rotation
+ else:
+ orig_quat = original_rotation[0]
+ else:
+ orig_quat = [1, 0, 0, 0]
+
+ try:
+ new_quat = self.update_rotation_about_camera_direction(
+ rotation_degrees + rot_fix,
+ orig_quat,
+ camera_direction_normalized.tolist(),
+ )
+ except ValueError:
+ print(f"Warning: Invalid original quaternion for {obj_id_str}, skipping rotation")
+ new_quat = orig_quat
+
+ new_layout["rotation"] = [new_quat]
+ else:
+ new_layout["rotation"] = original_rotation
+
+ # Save new layout
+ output_layout_path = layout_output_dir / f"{obj_id_str}.json"
+ with output_layout_path.open("w", encoding="utf-8") as f:
+ json.dump(new_layout, f, ensure_ascii=False, indent=2)
+
+ def convert_2d_pose_to_3d(self, mapping_path: Path | None, pose_payload: dict) -> dict:
+ if not mapping_path or not mapping_path.exists():
+ return pose_payload
+ if not isinstance(pose_payload, dict):
+ return pose_payload
+
+ poses = pose_payload.get("poses")
+ if not isinstance(poses, list) or not poses:
+ return pose_payload
+
+ grid_size, grid_map, mapping_payload = self._load_grid_mapping(mapping_path)
+ if not grid_map:
+ return pose_payload
+
+ if isinstance(grid_size, int) and grid_size > 1:
+ max_index = grid_size - 1
+ else:
+ max_i = max(i for i, _ in grid_map.keys())
+ max_j = max(j for _, j in grid_map.keys())
+ max_index = max(max_i, max_j)
+
+ signed_distance_by_id = {}
+ mapping_objects = mapping_payload.get("objects", [])
+ if isinstance(mapping_objects, list):
+ for obj in mapping_objects:
+ if not isinstance(obj, dict):
+ continue
+ object_id = obj.get("object_id")
+ signed_distance = obj.get("signed_distance_to_plane")
+ if object_id is not None and isinstance(signed_distance, (int, float)):
+ signed_distance_by_id[str(object_id)] = float(signed_distance)
+
+ plane_normal = None
+ plane_info = mapping_payload.get("plane", {})
+ if isinstance(plane_info, dict):
+ normal = plane_info.get("normal")
+ if isinstance(normal, list) and len(normal) == 3:
+ plane_normal = np.asarray(normal, dtype=np.float32)
+ norm = np.linalg.norm(plane_normal)
+ if norm > 1e-6:
+ plane_normal = plane_normal / norm
+ else:
+ plane_normal = None
+
+ for pose in poses:
+ if not isinstance(pose, dict):
+ continue
+ coords = pose.get("center_coordinates", {})
+ if not isinstance(coords, dict):
+ continue
+ u = coords.get("u")
+ v = coords.get("v")
+ try:
+ u_val = float(u)
+ v_val = float(v)
+ except (TypeError, ValueError):
+ continue
+
+ point_3d = self._interpolate_grid_point(grid_map, u_val, v_val, max_index)
+ if not point_3d:
+ continue
+
+ object_id = pose.get("object_id") or pose.get("id")
+ signed_distance = None
+ if object_id is not None:
+ signed_distance = signed_distance_by_id.get(str(object_id))
+
+ if signed_distance is not None and plane_normal is not None:
+ point_3d = [
+ point_3d[0] + plane_normal[0] * signed_distance,
+ point_3d[1] + plane_normal[1] * signed_distance,
+ point_3d[2] + plane_normal[2] * signed_distance,
+ ]
+
+ pose["center_coordinates_3d"] = {
+ "x": point_3d[0],
+ "y": point_3d[1],
+ "z": point_3d[2],
+ }
+
+ return pose_payload
+
+ _POSE_REF_PATTERN = re.compile(r"[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)+")
+
+ def _build_pose_context(self, mapping_payload: dict) -> dict:
+ context = {}
+ objects = mapping_payload.get("objects", [])
+ if not isinstance(objects, list):
+ return context
+ for obj in objects:
+ if not isinstance(obj, dict):
+ continue
+ object_id = obj.get("object_id")
+ if object_id is None:
+ continue
+ object_id = str(object_id)
+ entry = {}
+
+ center = obj.get("center_2d")
+ if isinstance(center, list) and len(center) >= 2:
+ entry["center"] = {
+ "u": float(center[0]),
+ "v": float(center[1]),
+ }
+
+ size = obj.get("size_2d")
+ if isinstance(size, list) and len(size) >= 2:
+ entry["size"] = {
+ "u": float(size[0]),
+ "v": float(size[1]),
+ }
+
+ bbox = obj.get("bbox_2d")
+ if isinstance(bbox, list) and len(bbox) >= 4:
+ entry["bbox"] = {
+ "min_u": float(bbox[0]),
+ "min_v": float(bbox[1]),
+ "max_u": float(bbox[2]),
+ "max_v": float(bbox[3]),
+ }
+
+ rotation_degrees = obj.get("rotation_degrees")
+ if isinstance(rotation_degrees, (int, float)):
+ entry["rotation_degrees"] = float(rotation_degrees)
+
+ if entry:
+ context[object_id] = entry
+
+ return context
+
+ def _normalize_pose_section(self, section: str) -> str:
+ section = section.strip().lower()
+ if section in {"center", "center_2d", "center2d"}:
+ return "center"
+ if section in {"size", "size_2d", "size2d"}:
+ return "size"
+ if section in {"bbox", "bbox_2d", "bbox2d"}:
+ return "bbox"
+ if section in {"rotation", "rotation_degrees"}:
+ return "rotation_degrees"
+ return section
+
+ def _lookup_pose_reference(self, reference: str, pose_context: dict) -> float | None:
+ parts = [part.strip() for part in reference.split(".") if part.strip()]
+ if len(parts) < 2:
+ return None
+ object_id = parts[0]
+ obj = pose_context.get(object_id)
+ if not isinstance(obj, dict):
+ return None
+
+ if len(parts) == 2:
+ key = self._normalize_pose_section(parts[1])
+ if key == "rotation_degrees":
+ value = obj.get("rotation_degrees")
+ return float(value) if isinstance(value, (int, float)) else None
+ return None
+
+ section = self._normalize_pose_section(parts[1])
+ key = parts[2]
+ if section == "center":
+ value = (obj.get("center") or {}).get(key)
+ return float(value) if isinstance(value, (int, float)) else None
+ if section == "size":
+ value = (obj.get("size") or {}).get(key)
+ return float(value) if isinstance(value, (int, float)) else None
+ if section == "bbox":
+ value = (obj.get("bbox") or {}).get(key)
+ return float(value) if isinstance(value, (int, float)) else None
+ if section == "rotation_degrees":
+ value = obj.get("rotation_degrees")
+ return float(value) if isinstance(value, (int, float)) else None
+ return None
+
+ def _parse_pose_target(self, target: str) -> tuple[str, str, str | None] | None:
+ if not target:
+ return None
+ parts = [part.strip() for part in target.split(".") if part.strip()]
+ if len(parts) < 2:
+ return None
+ object_id = parts[0]
+
+ if len(parts) == 2:
+ section = self._normalize_pose_section(parts[1])
+ if section == "rotation_degrees":
+ return object_id, "rotation_degrees", None
+ return None
+
+ section = self._normalize_pose_section(parts[1])
+ if section == "center":
+ axis = parts[2]
+ if axis in {"u", "v"}:
+ return object_id, "center", axis
+ if section == "rotation_degrees":
+ return object_id, "rotation_degrees", None
+ return None
+
+ def _parse_pose_assignment(self, statement: object) -> tuple[str, str] | None:
+ target = None
+ expression = None
+
+ if isinstance(statement, dict):
+ target = statement.get("target") or statement.get("lhs") or statement.get("left")
+ expression = statement.get("expression") or statement.get("rhs") or statement.get("value")
+ elif isinstance(statement, str):
+ raw = statement.strip().rstrip(";")
+ if "+=" in raw:
+ left, right = raw.split("+=", 1)
+ target = left.strip()
+ expression = f"{target} + ({right.strip()})"
+ elif "-=" in raw:
+ left, right = raw.split("-=", 1)
+ target = left.strip()
+ expression = f"{target} - ({right.strip()})"
+ elif "=" in raw:
+ left, right = raw.split("=", 1)
+ target = left.strip()
+ expression = right.strip()
+
+ if not target or not expression:
+ return None
+ return str(target), str(expression)
+
+ def _rewrite_pose_expression(self, expression: str, pose_context: dict) -> tuple[str | None, dict | None]:
+ variables: dict[str, float] = {}
+ missing_reference = False
+
+ def _replace(match: re.Match) -> str:
+ nonlocal missing_reference
+ reference = match.group(0)
+ value = self._lookup_pose_reference(reference, pose_context)
+ if value is None:
+ missing_reference = True
+ return "0"
+ key = f"ref_{len(variables)}"
+ variables[key] = float(value)
+ return key
+
+ rewritten = self._POSE_REF_PATTERN.sub(_replace, expression)
+ if missing_reference:
+ return None, None
+ return rewritten, variables
+
+ def _safe_eval_expression(self, expression: str, variables: dict[str, float]) -> float | None:
+ try:
+ tree = ast.parse(expression, mode="eval")
+ except SyntaxError:
+ return None
+
+ def _eval(node: ast.AST) -> float:
+ if isinstance(node, ast.Expression):
+ return _eval(node.body)
+ if isinstance(node, ast.Constant):
+ if isinstance(node.value, (int, float)):
+ return float(node.value)
+ raise ValueError("Unsupported constant")
+ if isinstance(node, ast.Name):
+ if node.id in variables:
+ return float(variables[node.id])
+ raise ValueError("Unknown variable")
+ if isinstance(node, ast.UnaryOp) and isinstance(node.op, (ast.UAdd, ast.USub)):
+ operand = _eval(node.operand)
+ return operand if isinstance(node.op, ast.UAdd) else -operand
+ if isinstance(node, ast.BinOp) and isinstance(
+ node.op, (ast.Add, ast.Sub, ast.Mult, ast.Div)
+ ):
+ left = _eval(node.left)
+ right = _eval(node.right)
+ if isinstance(node.op, ast.Add):
+ return left + right
+ if isinstance(node.op, ast.Sub):
+ return left - right
+ if isinstance(node.op, ast.Mult):
+ return left * right
+ if isinstance(node.op, ast.Div):
+ return left / right
+ raise ValueError("Unsupported expression")
+
+ try:
+ return float(_eval(tree))
+ except Exception:
+ return None
+
+ def _update_center(self, obj_entry: dict, axis: str, value: float) -> None:
+ center = obj_entry.get("center") or {}
+ previous_u = center.get("u")
+ previous_v = center.get("v")
+ if axis == "u":
+ center["u"] = float(value)
+ elif axis == "v":
+ center["v"] = float(value)
+ obj_entry["center"] = center
+
+ if previous_u is None or previous_v is None:
+ return
+ new_u = center.get("u")
+ new_v = center.get("v")
+ if not isinstance(new_u, (int, float)) or not isinstance(new_v, (int, float)):
+ return
+
+ dx = float(new_u) - float(previous_u)
+ dy = float(new_v) - float(previous_v)
+ bbox = obj_entry.get("bbox")
+ if isinstance(bbox, dict):
+ for key, delta in (("min_u", dx), ("max_u", dx), ("min_v", dy), ("max_v", dy)):
+ if key in bbox and isinstance(bbox[key], (int, float)):
+ bbox[key] = float(bbox[key]) + delta
+ obj_entry["bbox"] = bbox
+ elif isinstance(obj_entry.get("size"), dict):
+ size = obj_entry["size"]
+ half_u = float(size.get("u", 0.0)) / 2.0
+ half_v = float(size.get("v", 0.0)) / 2.0
+ obj_entry["bbox"] = {
+ "min_u": float(new_u) - half_u,
+ "min_v": float(new_v) - half_v,
+ "max_u": float(new_u) + half_u,
+ "max_v": float(new_v) + half_v,
+ }
+
+ def resolve_pose_expressions(
+ self,
+ mapping_path: Path | None,
+ pose_payload: dict,
+ anchor_id: str | None = None,
+ ) -> dict:
+ if not mapping_path or not mapping_path.exists():
+ return pose_payload
+ if not isinstance(pose_payload, dict):
+ return pose_payload
+
+ pose_expressions = pose_payload.get("pose_expressions")
+ if not pose_expressions:
+ if "poses" in pose_payload:
+ pose_payload["poses"] = []
+ return pose_payload
+
+ with mapping_path.open("r", encoding="utf-8") as f:
+ mapping_payload = json.load(f)
+
+ pose_context = self._build_pose_context(mapping_payload)
+ original_rotation_by_object = {
+ object_id: obj_entry.get("rotation_degrees")
+ for object_id, obj_entry in pose_context.items()
+ if isinstance(obj_entry, dict)
+ }
+ updated_objects: set[str] = set()
+ rotation_updates: set[str] = set()
+
+ if isinstance(pose_expressions, list):
+ for statement in pose_expressions:
+ parsed = self._parse_pose_assignment(statement)
+ if not parsed:
+ continue
+ target, expression = parsed
+ target_info = self._parse_pose_target(target)
+ if not target_info:
+ continue
+ object_id, field, axis = target_info
+ if anchor_id and object_id == anchor_id:
+ continue
+ if object_id not in pose_context:
+ continue
+
+ rewritten, variables = self._rewrite_pose_expression(expression, pose_context)
+ if rewritten is None or variables is None:
+ continue
+ value = self._safe_eval_expression(rewritten, variables)
+ if value is None:
+ continue
+
+ obj_entry = pose_context.get(object_id)
+ if not isinstance(obj_entry, dict):
+ continue
+ if field == "center" and axis in {"u", "v"}:
+ self._update_center(obj_entry, axis, float(value))
+ updated_objects.add(object_id)
+ elif field == "rotation_degrees":
+ obj_entry["rotation_degrees"] = float(value)
+ updated_objects.add(object_id)
+ rotation_updates.add(object_id)
+
+ if not updated_objects:
+ return pose_payload
+
+ resolved_poses = []
+ for object_id in sorted(updated_objects):
+ obj_entry = pose_context.get(object_id) or {}
+ center = obj_entry.get("center") or {}
+ if not isinstance(center.get("u"), (int, float)) or not isinstance(center.get("v"), (int, float)):
+ continue
+ entry = {
+ "object_id": object_id,
+ "center_coordinates": {
+ "u": float(center["u"]),
+ "v": float(center["v"]),
+ },
+ }
+ if object_id in rotation_updates:
+ rotation = obj_entry.get("rotation_degrees")
+ if isinstance(rotation, (int, float)):
+ entry["rotation_degrees"] = float(rotation)
+ elif isinstance(original_rotation_by_object.get(object_id), (int, float)):
+ entry["rotation_degrees"] = float(original_rotation_by_object[object_id])
+ resolved_poses.append(entry)
+
+ pose_payload["poses"] = resolved_poses
+ return pose_payload
diff --git a/vlm_agent/evolution/scene_optimizer.py b/vlm_agent/evolution/scene_optimizer.py
new file mode 100644
index 0000000..da0de7e
--- /dev/null
+++ b/vlm_agent/evolution/scene_optimizer.py
@@ -0,0 +1,111 @@
+import json
+from pathlib import Path
+import numpy as np
+
+from .layout_combination import _layout_payload_to_matrix, _matrix_to_layout_payload
+
+
+def build_anchor_dependency_graph(object_groups: dict) -> tuple[dict[str, set[str]], dict[str, set[str]], dict[str, int]]:
+ parents_by_child: dict[str, set[str]] = {}
+ children_by_parent: dict[str, set[str]] = {}
+ indegree: dict[str, int] = {}
+
+ for anchor_id, group_info in object_groups.items():
+ if isinstance(group_info, dict):
+ object_ids = group_info.get("object_ids", [])
+ else:
+ object_ids = group_info if isinstance(group_info, list) else []
+ indegree.setdefault(anchor_id, 0)
+ for object_id in object_ids:
+ parents_by_child.setdefault(object_id, set()).add(anchor_id)
+ children_by_parent.setdefault(anchor_id, set()).add(object_id)
+ indegree.setdefault(object_id, 0)
+ indegree[object_id] += 1
+
+ return parents_by_child, children_by_parent, indegree
+
+def collect_descendant_subtree(
+ root_id: str,
+ children_by_parent: dict[str, set[str]],
+) -> list[str]:
+ subtree_nodes: list[str] = []
+ stack = [root_id]
+ visited: set[str] = set()
+
+ while stack:
+ node_id = stack.pop()
+ if node_id in visited:
+ continue
+ visited.add(node_id)
+ subtree_nodes.append(node_id)
+ for child_id in sorted(children_by_parent.get(node_id, []), reverse=True):
+ if child_id not in visited:
+ stack.append(child_id)
+
+ return subtree_nodes
+
+def iter_subtree_roots(
+ anchor_id: str,
+ children_by_parent: dict[str, set[str]],
+) -> list[str]:
+ subtree_roots: list[str] = []
+ for child_id in sorted(children_by_parent.get(anchor_id, [])):
+ subtree_nodes = collect_descendant_subtree(child_id, children_by_parent)
+ if len(subtree_nodes) > 1:
+ subtree_roots.append(child_id)
+ return subtree_roots
+
+def load_layout_snapshot(layout_dir: Path) -> dict[str, dict]:
+ layouts: dict[str, dict] = {}
+ for layout_file in layout_dir.glob("*.json"):
+ try:
+ with layout_file.open("r", encoding="utf-8") as f:
+ payload = json.load(f)
+ except json.JSONDecodeError:
+ continue
+ if isinstance(payload, dict):
+ layouts[layout_file.stem] = payload
+ return layouts
+
+def _write_layout_payload(layout_dir: Path, object_id: str, payload: dict) -> None:
+ layout_path = layout_dir / f"{object_id}.json"
+ with layout_path.open("w", encoding="utf-8") as f:
+ json.dump(payload, f, ensure_ascii=False, indent=2)
+
+def propagate_subtree_transform(
+ self,
+ pending_layout_dir: Path,
+ children_by_parent: dict[str, set[str]],
+ anchor_id: str,
+ subtree_root_id: str,
+ before_layouts: dict[str, dict],
+ after_layouts: dict[str, dict],
+) -> None:
+ subtree_nodes = collect_descendant_subtree(subtree_root_id, children_by_parent)
+ if len(subtree_nodes) <= 1:
+ return
+
+ before_root_payload = before_layouts.get(subtree_root_id)
+ after_root_payload = after_layouts.get(subtree_root_id)
+ if before_root_payload is None or after_root_payload is None:
+ return
+
+ try:
+ before_root_matrix = _layout_payload_to_matrix(before_root_payload)
+ after_root_matrix = _layout_payload_to_matrix(after_root_payload)
+ delta_matrix = after_root_matrix @ np.linalg.inv(before_root_matrix)
+ except np.linalg.LinAlgError:
+ return
+
+ print(f"Propagating anchor '{anchor_id}' through subtree rooted at '{subtree_root_id}'")
+
+ for node_id in subtree_nodes:
+ if node_id == subtree_root_id:
+ continue
+ original_payload = before_layouts.get(node_id)
+ if original_payload is None:
+ continue
+ composed_matrix = delta_matrix @ _layout_payload_to_matrix(original_payload)
+ updated_payload = _matrix_to_layout_payload(composed_matrix, original_payload)
+ after_layouts[node_id] = updated_payload
+ _write_layout_payload(pending_layout_dir, node_id, updated_payload)
diff --git a/vlm_agent/evolution/tool_runner.py b/vlm_agent/evolution/tool_runner.py
new file mode 100644
index 0000000..a05c4da
--- /dev/null
+++ b/vlm_agent/evolution/tool_runner.py
@@ -0,0 +1,433 @@
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+
+
+@dataclass(frozen=True)
+class ToolRunResult:
+ command: list[str]
+ returncode: int
+ stdout: str
+ stderr: str
+ failed_object_ids: tuple[str, ...] = ()
+ out_of_bounds_object_ids: tuple[str, ...] = ()
+
+
+class ToolRunner:
+ def __init__(self, project_root: Path):
+ self.project_root = project_root
+ self.tools_dir = Path(__file__).resolve().parent / "tools"
+ self._build_collision_hulls_summary_prefix = "BUILD_COLLISION_HULLS_SUMMARY:"
+ self._fix_layout_containment_summary_prefix = "FIX_LAYOUT_CONTAINMENT_SUMMARY:"
+ self._retrieve_room_materials_summary_prefix = "ROOM_MATERIAL_RETRIEVAL_SUMMARY:"
+
+ @staticmethod
+ def _load_object_ids(unposed_dir: Path) -> list[str]:
+ return sorted(
+ p.stem.split("_")[-1]
+ for p in unposed_dir.glob("object_*.glb")
+ if p.stem.startswith("object_")
+ )
+
+ @staticmethod
+ def _load_collision_object_ids(output_dir: Path) -> list[str]:
+ return sorted(
+ p.stem.split("_")[-1]
+ for p in output_dir.glob("object_*.obj")
+ if p.stem.startswith("object_")
+ )
+
+ def _missing_collision_object_ids(self, unposed_dir: Path, output_dir: Path) -> list[str]:
+ expected_ids = self._load_object_ids(unposed_dir)
+ existing_ids = set(self._load_collision_object_ids(output_dir))
+ return [object_id for object_id in expected_ids if object_id not in existing_ids]
+
+ def _run_script(self, script_name: str, args: list[str]) -> ToolRunResult:
+ script_path = self.tools_dir / script_name
+ command = [sys.executable, str(script_path), *args]
+ try:
+ completed = subprocess.run(command, capture_output=True, text=True, check=False)
+ except Exception as exc:
+ stderr = f"Failed to run {script_name}: {exc}"
+ print(stderr, file=sys.stderr)
+ return ToolRunResult(command=command, returncode=1, stdout="", stderr=stderr)
+
+ result = ToolRunResult(
+ command=command,
+ returncode=completed.returncode,
+ stdout=completed.stdout,
+ stderr=completed.stderr,
+ )
+ if result.returncode != 0:
+ print(f"[{script_name}] failed with exit code {result.returncode}", file=sys.stderr)
+ if result.stdout:
+ print(result.stdout, file=sys.stderr, end="" if result.stdout.endswith("\n") else "\n")
+ if result.stderr:
+ print(result.stderr, file=sys.stderr, end="" if result.stderr.endswith("\n") else "\n")
+ return result
+
+ def _extract_build_collision_hulls_failed_object_ids(self, stdout: str) -> tuple[str, ...]:
+ for line in reversed(stdout.splitlines()):
+ if not line.startswith(self._build_collision_hulls_summary_prefix):
+ continue
+ payload_text = line[len(self._build_collision_hulls_summary_prefix) :].strip()
+ if not payload_text:
+ return ()
+ try:
+ payload = json.loads(payload_text)
+ except json.JSONDecodeError:
+ return ()
+ failed_object_ids = payload.get("failed_object_ids", []) if isinstance(payload, dict) else []
+ if not isinstance(failed_object_ids, list):
+ return ()
+ return tuple(str(object_id) for object_id in failed_object_ids)
+ return ()
+
+ def _extract_fix_layout_containment_out_of_bounds_object_ids(self, stdout: str) -> tuple[str, ...]:
+ for line in reversed(stdout.splitlines()):
+ if not line.startswith(self._fix_layout_containment_summary_prefix):
+ continue
+ payload_text = line[len(self._fix_layout_containment_summary_prefix) :].strip()
+ if not payload_text:
+ return ()
+ try:
+ payload = json.loads(payload_text)
+ except json.JSONDecodeError:
+ return ()
+ out_of_bounds_object_ids = payload.get("out_of_bounds_object_ids", []) if isinstance(payload, dict) else []
+ if not isinstance(out_of_bounds_object_ids, list):
+ return ()
+ return tuple(str(object_id) for object_id in out_of_bounds_object_ids)
+ return ()
+
+ def _run_build_collision_hulls_once(
+ self,
+ unposed_dir: Path,
+ output_dir: Path,
+ manifest: Path,
+ *,
+ object_ids: list[str] | None = None,
+ workers: int | None = None,
+ target_faces_per_part: int | None = None,
+ split_mode: str | None = "none",
+ ) -> ToolRunResult:
+ args = ["--unposed_dir", str(unposed_dir), "--output_dir", str(output_dir), "--manifest", str(manifest)]
+ if workers is not None:
+ args.extend(["--workers", str(workers)])
+ if target_faces_per_part is not None:
+ args.extend(["--target_faces_per_part", str(target_faces_per_part)])
+ if split_mode is not None:
+ args.extend(["--split_mode", str(split_mode)])
+ if object_ids:
+ args.append("--object_ids")
+ args.extend(str(object_id) for object_id in object_ids)
+ result = self._run_script("build_collision_hulls.py", args)
+ return ToolRunResult(
+ command=result.command,
+ returncode=result.returncode,
+ stdout=result.stdout,
+ stderr=result.stderr,
+ failed_object_ids=self._extract_build_collision_hulls_failed_object_ids(result.stdout),
+ )
+
+ def compose_scene(self, layout_dir: Path, unposed_dir: Path, output: Path) -> ToolRunResult:
+ return self._run_script(
+ "compose_scene_from_layout.py",
+ ["--layout_dir", str(layout_dir), "--unposed_dir", str(unposed_dir), "--output", str(output)],
+ )
+
+ def _extract_room_material_dirs(self, stdout: str) -> tuple[Path, Path, Path] | None:
+ for line in reversed(stdout.splitlines()):
+ if not line.startswith(self._retrieve_room_materials_summary_prefix):
+ continue
+ payload_text = line[len(self._retrieve_room_materials_summary_prefix) :].strip()
+ try:
+ payload = json.loads(payload_text)
+ except json.JSONDecodeError:
+ return None
+ if not isinstance(payload, dict) or payload.get("status") != "resolved":
+ return None
+ paths = (
+ payload.get("wall_material_dir"),
+ payload.get("floor_material_dir"),
+ payload.get("ceiling_material_dir"),
+ )
+ if not all(isinstance(path, str) and path for path in paths):
+ return None
+ return tuple(Path(path) for path in paths)
+ return None
+
+ def retrieve_room_materials(self, objects_json: Path) -> tuple[Path, Path, Path] | None:
+ result = self._run_script(
+ "retrieve_room_materials.py",
+ [
+ "--objects_json",
+ str(objects_json),
+ "--materials_dir",
+ str(self.project_root / "data" / "materials"),
+ "--embeddings_dir",
+ str(self.project_root / "data" / "embeddings"),
+ ],
+ )
+ material_dirs = self._extract_room_material_dirs(result.stdout)
+ if material_dirs is not None:
+ print("Resolved room materials for walls, floor, and ceiling.")
+ return material_dirs
+ if result.returncode == 0:
+ print("No room_materials found; using legacy room material.")
+ else:
+ print("Room material retrieval failed; using legacy room material.", file=sys.stderr)
+ return None
+
+ def generate_room_mesh(
+ self,
+ layout_dir: Path,
+ unposed_dir: Path,
+ objects_json: Path,
+ *,
+ material_dir: Path | None = None,
+ thickness: float | None = None,
+ wall_padding: float | None = None,
+ ) -> ToolRunResult:
+ args = [
+ "--layout_dir",
+ str(layout_dir),
+ "--unposed_dir",
+ str(unposed_dir),
+ "--objects_json",
+ str(objects_json),
+ ]
+ material_dirs = self.retrieve_room_materials(objects_json)
+ if material_dirs is not None:
+ wall_material_dir, floor_material_dir, ceiling_material_dir = material_dirs
+ args.extend(["--wall_material_dir", str(wall_material_dir)])
+ args.extend(["--floor_material_dir", str(floor_material_dir)])
+ args.extend(["--ceiling_material_dir", str(ceiling_material_dir)])
+ if material_dir is not None:
+ args.extend(["--material_dir", str(material_dir)])
+ if thickness is not None:
+ args.extend(["--thickness", str(thickness)])
+ if wall_padding is not None:
+ args.extend(["--wall_padding", str(wall_padding)])
+ return self._run_script("generate_room_mesh.py", args)
+
+ def render_scene(
+ self,
+ layout_dir: Path,
+ unposed_dir: Path,
+ output: Path,
+ *,
+ object_ids: list[str] | None = None,
+ camera_direction=None,
+ radius: float | None = None,
+ image_size: int | None = None,
+ add_grid: bool = False,
+ grid_size: int | None = None,
+ draw_grid_lines: bool = False,
+ draw_object_directions: bool = False,
+ draw_object_bboxes: bool = False,
+ anchor_id: str | None = None,
+ grid_mapping: str | None = None,
+ hide_object_ids: list[str] | None = None,
+ use_aabb_for_object_bboxes: bool = False,
+ ) -> ToolRunResult:
+ args = ["--layout_dir", str(layout_dir), "--unposed_dir", str(unposed_dir), "--output", str(output)]
+ if object_ids:
+ args.append("--object_ids")
+ args.extend(str(object_id) for object_id in object_ids)
+ if camera_direction is not None:
+ args.append("--camera_direction")
+ if isinstance(camera_direction, (list, tuple)):
+ args.extend(str(value) for value in camera_direction)
+ else:
+ args.append(str(camera_direction))
+ if radius is not None:
+ args.extend(["--radius", str(radius)])
+ if image_size is not None:
+ args.extend(["--image_size", str(image_size)])
+ if add_grid:
+ args.append("--add_grid")
+ if grid_size is not None:
+ args.extend(["--grid_size", str(grid_size)])
+ if draw_grid_lines:
+ args.append("--draw_grid_lines")
+ if draw_object_directions:
+ args.append("--draw_object_directions")
+ if draw_object_bboxes:
+ args.append("--draw_object_bboxes")
+ if anchor_id is not None:
+ args.append("--anchor_id")
+ args.append(str(anchor_id))
+ if grid_mapping is not None:
+ args.append("--grid_mapping")
+ args.append(str(grid_mapping))
+ if hide_object_ids:
+ args.append("--hide_object_ids")
+ args.extend(str(object_id) for object_id in hide_object_ids)
+ if use_aabb_for_object_bboxes:
+ args.append("--use_aabb_for_object_bboxes")
+ return self._run_script("render_scene_from_layout.py", args)
+
+ def build_collision_hulls(
+ self,
+ unposed_dir: Path,
+ output_dir: Path,
+ manifest: Path,
+ *,
+ workers: int | None = None,
+ target_faces_per_part: int | None = None,
+ max_retries: int = 4,
+ split_mode: str | None = "none",
+ ) -> ToolRunResult:
+ if max_retries < 1:
+ raise ValueError("max_retries must be >= 1")
+
+ attempt_faces_per_part: list[int | None] = [target_faces_per_part, 5000, 10000, 20000]
+ result = self._run_build_collision_hulls_once(
+ unposed_dir,
+ output_dir,
+ manifest,
+ workers=workers,
+ target_faces_per_part=attempt_faces_per_part[0],
+ split_mode=split_mode,
+ )
+ if result.returncode != 0 or not result.failed_object_ids:
+ return result
+
+ retry_index = 1
+ while retry_index < min(max_retries, len(attempt_faces_per_part)):
+ current_faces_per_part = attempt_faces_per_part[retry_index]
+ retry_object_ids = list(result.failed_object_ids)
+ print(
+ "Retrying build_collision_hulls "
+ f"(attempt {retry_index + 1}/{max_retries}) with failed_object_ids={list(result.failed_object_ids)}, "
+ f"workers={workers}, target_faces_per_part={current_faces_per_part}..."
+ )
+ retry_result = self._run_build_collision_hulls_once(
+ unposed_dir,
+ output_dir,
+ manifest,
+ object_ids=retry_object_ids,
+ workers=workers,
+ target_faces_per_part=current_faces_per_part,
+ split_mode=split_mode,
+ )
+ result = retry_result
+ if retry_result.returncode != 0 or not retry_result.failed_object_ids:
+ return retry_result
+ retry_index += 1
+
+ return result
+
+ def simulate_scene_physics(
+ self,
+ layout_dir: Path,
+ hull_dir: Path,
+ output_layout_dir: Path,
+ *,
+ gravity=None,
+ object_ids: list[str] | None = None,
+ verify_only: bool = False,
+ cpu_only: bool = True,
+ use_ground_object: bool = False,
+ ground_object_ids: list[str] | None = None,
+ lock_rotation_all: bool = False,
+ init_non_ground_y_offset: bool = False,
+ ) -> ToolRunResult:
+ args = [
+ "--layout_dir",
+ str(layout_dir),
+ "--hull_dir",
+ str(hull_dir),
+ "--output_layout_dir",
+ str(output_layout_dir),
+ ]
+ if gravity is not None:
+ args.append("--gravity")
+ if isinstance(gravity, (list, tuple)):
+ args.extend(str(value) for value in gravity)
+ else:
+ args.append(str(gravity))
+ if object_ids:
+ args.append("--object_ids")
+ args.extend(str(object_id) for object_id in object_ids)
+ if verify_only:
+ args.append("--verify_only")
+ if cpu_only:
+ args.append("--cpu_only")
+ if lock_rotation_all:
+ args.append("--lock_rotation_all")
+ if use_ground_object:
+ args.append("--use_ground_object")
+ if ground_object_ids:
+ args.append("--ground_object_id")
+ args.extend(str(object_id) for object_id in ground_object_ids)
+ if init_non_ground_y_offset:
+ args.append("--init_non_ground_y_offset")
+ return self._run_script("simulate_scene_physics.py", args)
+
+ def fix_layout_containment(
+ self,
+ layout_dir: Path,
+ unposed_dir: Path,
+ output_layout_dir: Path,
+ *,
+ object_ids: list[str] | None = None,
+ anchor_id: str,
+ tolerated_out_of_bounds: float = 0.0,
+ detect_only: bool = False,
+ ) -> ToolRunResult:
+ args = [
+ "--layout_dir",
+ str(layout_dir),
+ "--unposed_dir",
+ str(unposed_dir),
+ "--output_layout_dir",
+ str(output_layout_dir),
+ "--anchor_id",
+ str(anchor_id),
+ "--tolerated_out_of_bounds",
+ str(tolerated_out_of_bounds),
+ ]
+ if object_ids:
+ args.append("--object_ids")
+ args.extend(str(object_id) for object_id in object_ids)
+ if detect_only:
+ args.append("--detect_only")
+ result = self._run_script("fix_layout_containment.py", args)
+ return ToolRunResult(
+ command=result.command,
+ returncode=result.returncode,
+ stdout=result.stdout,
+ stderr=result.stderr,
+ out_of_bounds_object_ids=self._extract_fix_layout_containment_out_of_bounds_object_ids(result.stdout),
+ )
+
+ def check_object_collisions(
+ self,
+ layout_dir: Path,
+ hull_dir: Path,
+ *,
+ output: Path | None = None,
+ object_ids: list[str] | None = None,
+ steps: int | None = None,
+ mass: float | None = None,
+ anchor_id: str | None = None,
+ ) -> ToolRunResult:
+ args = ["--layout_dir", str(layout_dir), "--hull_dir", str(hull_dir)]
+ if output is not None:
+ args.extend(["--output", str(output)])
+ if object_ids:
+ args.append("--object_ids")
+ args.extend(str(object_id) for object_id in object_ids)
+ if steps is not None:
+ args.extend(["--steps", str(steps)])
+ if mass is not None:
+ args.extend(["--mass", str(mass)])
+ if anchor_id is not None:
+ args.extend(["--anchor_id", str(anchor_id)])
+ return self._run_script("check_object_collisions.py", args)
diff --git a/vlm_agent/evolution/tools/__init__.py b/vlm_agent/evolution/tools/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/vlm_agent/evolution/tools/build_collision_hulls.py b/vlm_agent/evolution/tools/build_collision_hulls.py
new file mode 100644
index 0000000..0969329
--- /dev/null
+++ b/vlm_agent/evolution/tools/build_collision_hulls.py
@@ -0,0 +1,507 @@
+import argparse
+import os
+import json
+import time
+import traceback
+from concurrent.futures import FIRST_COMPLETED, ProcessPoolExecutor, wait
+from multiprocessing import Manager, get_context
+from pathlib import Path
+from queue import Empty
+from typing import Iterable, Optional, Set
+
+import numpy as np
+import trimesh
+import coacd
+
+
+TASK_TIMEOUT_SECONDS = 180
+R_OPENGL_TO_P3D = np.array([[-1, 0, 0], [0, 1, 0], [0, 0, -1]], dtype=np.float32)
+SUMMARY_PREFIX = "BUILD_COLLISION_HULLS_SUMMARY:"
+
+
+def load_object_ids(unposed_dir: Path):
+ # Return all .glb stems in the directory (support arbitrary filenames)
+ return sorted(p.stem for p in unposed_dir.glob("*.glb"))
+
+
+def resolve_requested_object_ids(requested_object_ids, available_ids: list[str]):
+ """Resolve requested ids (which may be numeric like '001' or stems)
+ to actual available stems from `available_ids`.
+
+ - If `requested_object_ids` is None -> returns None
+ - If a requested id matches an available stem exactly -> use it
+ - If a requested id is numeric (all digits), match any available stem that endswith that numeric suffix.
+ Raises FileNotFoundError if any requested id cannot be resolved.
+ """
+ if requested_object_ids is None:
+ return None
+ if isinstance(requested_object_ids, (str, int)):
+ requested_object_ids = [requested_object_ids]
+
+ available_set = set(available_ids)
+ resolved = []
+ for req in requested_object_ids:
+ reqs = str(req)
+ # numeric id -> match stems that end with this suffix
+ if reqs.isdigit():
+ reqs = f"object_{reqs}"
+ if reqs in available_set:
+ resolved.append(reqs)
+ continue
+ # try common pattern: strip leading 'object_' and check
+ if reqs.startswith("object_"):
+ alt = reqs[len("object_"):]
+ if alt in available_set:
+ resolved.append(alt)
+ continue
+ raise FileNotFoundError(
+ f"Requested object id not found or ambiguous: {reqs}. Available ids: {available_ids}"
+ )
+
+ # Deduplicate while preserving order
+ seen = set()
+ out = []
+ for x in resolved:
+ if x not in seen:
+ seen.add(x)
+ out.append(x)
+ return out
+
+
+def load_split_object_ids(path: Optional[Path], available_ids: list[str]) -> Set[str]:
+ """Load split list JSON and resolve entries against available ids.
+
+ The JSON may be a list of ids or a dict of id->bool. Entries may be
+ stems or numeric suffixes; resolution uses `resolve_requested_object_ids`.
+ """
+ if path is None:
+ return set()
+ if not path.exists():
+ raise FileNotFoundError(f"Split list not found: {path}")
+
+ with path.open("r", encoding="utf-8") as f:
+ payload = json.load(f)
+
+ entries = []
+ if isinstance(payload, list):
+ entries = payload
+ elif isinstance(payload, dict):
+ entries = [k for k, v in payload.items() if bool(v)]
+ else:
+ raise ValueError("Split list JSON must be a list or a dict")
+
+ resolved = resolve_requested_object_ids(entries, available_ids)
+ return set(resolved)
+
+
+def load_requested_object_ids(object_ids: Optional[list[str]]) -> Optional[list[str]]:
+ if object_ids is None:
+ return None
+ normalized = [str(object_id) for object_id in object_ids]
+ return sorted(dict.fromkeys(normalized))
+
+
+def run_coacd_parts(mesh: trimesh.Trimesh, threshold: float, preprocess_mode: str):
+ mesh_coacd = coacd.Mesh(mesh.vertices.astype(np.float32), mesh.faces)
+ return coacd.run_coacd(
+ mesh_coacd,
+ threshold=threshold,
+ preprocess_mode=preprocess_mode,
+ )
+
+
+def simplify_mesh_faces(mesh: trimesh.Trimesh, target_faces: int) -> trimesh.Trimesh:
+ if target_faces <= 0:
+ return mesh
+ if len(mesh.faces) <= target_faces:
+ return mesh
+
+ # Use trimesh decimation when available; fallback to original mesh on failure.
+ try:
+ simplified = mesh.simplify_quadric_decimation(face_count=target_faces)
+ if simplified is not None and len(simplified.faces) > 0:
+ return simplified
+ except Exception as e:
+ print(f"Error: {e}")
+ print(f"Warning: Failed to simplify mesh with {len(mesh.faces)} faces to target {target_faces} faces. Using original mesh.")
+ pass
+ return mesh
+
+
+def split_mesh_for_decomposition(mesh: trimesh.Trimesh) -> Iterable[trimesh.Trimesh]:
+ center = mesh.bounding_box.centroid
+ cut_meshes = [
+ mesh.slice_plane(np.array([0, center[1], 0]), np.array([1, 0, 0])).slice_plane(
+ np.array([center[0], 0, 0]), np.array([0, 1, 0])
+ ),
+ mesh.slice_plane(np.array([0, center[1], 0]), np.array([-1, 0, 0])).slice_plane(
+ np.array([center[0], 0, 0]), np.array([0, 1, 0])
+ ),
+ mesh.slice_plane(np.array([0, center[1], 0]), np.array([1, 0, 0])).slice_plane(
+ np.array([center[0], 0, 0]), np.array([0, -1, 0])
+ ),
+ mesh.slice_plane(np.array([0, center[1], 0]), np.array([-1, 0, 0])).slice_plane(
+ np.array([center[0], 0, 0]), np.array([0, -1, 0])
+ ),
+ ]
+ return [m for m in cut_meshes if len(m.vertices) > 0 and len(m.faces) > 0]
+
+
+def scale_parts_to_origin(parts: Iterable[trimesh.Trimesh], origin_mesh: trimesh.Trimesh):
+ part_meshes = [m.copy() for m in parts]
+ if not part_meshes:
+ return []
+
+ combined_collision_mesh = trimesh.util.concatenate(part_meshes)
+ collision_size = combined_collision_mesh.bounding_box.extents
+ origin_size = origin_mesh.bounding_box.extents
+
+ safe_origin_size = np.where(np.abs(origin_size) < 1e-8, 1.0, origin_size)
+ scale_factor = collision_size / safe_origin_size
+ safe_scale_factor = np.where(np.abs(scale_factor) < 1e-8, 1.0, scale_factor)
+
+ collision_center = combined_collision_mesh.bounding_box.centroid
+ for mesh in part_meshes:
+ mesh.apply_translation(-collision_center)
+ mesh.apply_scale(1.0 / safe_scale_factor)
+ mesh.apply_translation(collision_center)
+ return part_meshes
+
+
+def build_coacd_hull(
+ mesh_path: Path,
+ split_before_decompose: bool,
+ threshold: float,
+ preprocess_mode: str,
+ target_faces_per_part: int,
+):
+ mesh = trimesh.load(mesh_path, force="mesh")
+
+ mesh_zup = mesh.copy()
+ mesh_zup.vertices = mesh.vertices.astype(np.float32) @ R_OPENGL_TO_P3D
+
+ parts = []
+ if split_before_decompose:
+ for cut_mesh in split_mesh_for_decomposition(mesh_zup):
+ cut_mesh = simplify_mesh_faces(cut_mesh, target_faces_per_part)
+ result = run_coacd_parts(cut_mesh, threshold, preprocess_mode)
+ parts.extend(result)
+ else:
+ mesh_zup = simplify_mesh_faces(mesh_zup, target_faces_per_part)
+ parts = run_coacd_parts(mesh_zup, threshold, preprocess_mode)
+
+ hulls = [trimesh.Trimesh(vertices=v, faces=f) for v, f in parts]
+ if not hulls:
+ hulls = [mesh_zup.convex_hull]
+
+ scaled_hulls = scale_parts_to_origin(hulls, mesh_zup)
+ combined_hull = trimesh.util.concatenate(scaled_hulls) if scaled_hulls else mesh_zup.convex_hull
+ return combined_hull, scaled_hulls
+
+
+def save_part_meshes(part_meshes: Iterable[trimesh.Trimesh], parts_dir: Path):
+ parts_dir.mkdir(parents=True, exist_ok=True)
+ for idx, part in enumerate(part_meshes):
+ part.export(parts_dir / f"collision_part_{idx:03d}.obj")
+
+
+def process_object(
+ object_id: str,
+ unposed_dir: Path,
+ output_dir: Path,
+ split_mode: str,
+ split_object_ids: Set[str],
+ save_parts: bool,
+ threshold: float,
+ preprocess_mode: str,
+ target_faces_per_part: int,
+):
+ # object_id is the file stem (without extension). Support arbitrary stems.
+ in_path = unposed_dir / f"{object_id}.glb"
+ out_path = output_dir / f"{object_id}.obj"
+
+ split_before_decompose = split_mode == "all" or (
+ split_mode == "list" and object_id in split_object_ids
+ )
+
+ hull, part_meshes = build_coacd_hull(
+ in_path,
+ split_before_decompose=split_before_decompose,
+ threshold=threshold,
+ preprocess_mode=preprocess_mode,
+ target_faces_per_part=target_faces_per_part,
+ )
+ if part_meshes:
+ scene = trimesh.Scene(part_meshes)
+ export_content = scene.export(file_type='obj')
+ with open(out_path, 'w', encoding='utf-8') as f:
+ f.write(export_content)
+ else:
+ hull.export(out_path)
+
+ parts_dir = output_dir / f"object_{object_id}_parts"
+ if save_parts:
+ save_part_meshes(part_meshes, parts_dir)
+
+ item_key = object_id
+ item_value = {
+ "source_mesh": str(in_path),
+ "collision_hull": str(out_path),
+ "split_before_decompose": bool(split_before_decompose),
+ "num_parts": int(len(part_meshes)),
+ "num_vertices": int(len(hull.vertices)),
+ "num_faces": int(len(hull.faces)),
+ "volume": float(hull.volume),
+ "coacd": {
+ "threshold": float(threshold),
+ "preprocess_mode": preprocess_mode,
+ "target_faces_per_part": int(target_faces_per_part),
+ },
+ }
+ if save_parts:
+ item_value["parts_dir"] = str(parts_dir)
+ return item_key, item_value, in_path.name, out_path.name
+
+
+def process_object_with_error_context(
+ object_id: str,
+ unposed_dir: Path,
+ output_dir: Path,
+ split_mode: str,
+ split_object_ids: Set[str],
+ save_parts: bool,
+ threshold: float,
+ preprocess_mode: str,
+ target_faces_per_part: int,
+ worker_start_queue=None,
+):
+ try:
+ if worker_start_queue is not None:
+ worker_start_queue.put((object_id, os.getpid()))
+ return process_object(
+ object_id,
+ unposed_dir,
+ output_dir,
+ split_mode,
+ split_object_ids,
+ save_parts,
+ threshold,
+ preprocess_mode,
+ target_faces_per_part,
+ )
+ except Exception as exc:
+ print(f"[ERROR] Failed while processing {object_id}.glb", flush=True)
+ traceback.print_exc()
+ raise RuntimeError(f"Failed while processing {object_id}.glb") from exc
+
+
+def terminate_executor_worker(executor: ProcessPoolExecutor, pid: int):
+ process = getattr(executor, "_processes", {}).get(pid)
+ if process is None:
+ return
+ if process.is_alive():
+ process.terminate()
+ process.join(timeout=1)
+
+
+def terminate_executor_workers(executor: ProcessPoolExecutor):
+ processes = getattr(executor, "_processes", None)
+ executor.shutdown(wait=False, cancel_futures=True)
+ if not processes:
+ return
+ for process in processes.values():
+ if process.is_alive():
+ process.terminate()
+ for process in processes.values():
+ process.join(timeout=1)
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Build convex hulls for object_XXX.glb files as collision meshes."
+ )
+ parser.add_argument(
+ "--unposed_dir",
+ type=Path,
+ default=Path("outputs/unposed_objects"),
+ help="Directory containing object_XXX.glb files.",
+ )
+ parser.add_argument(
+ "--output_dir",
+ type=Path,
+ default=Path("outputs/collision_hulls"),
+ help="Directory to save object_XXX.obj convex hulls.",
+ )
+ parser.add_argument(
+ "--manifest",
+ type=Path,
+ default=Path("outputs/collision_hulls/manifest.json"),
+ help="Path to save hull metadata manifest.",
+ )
+ parser.add_argument(
+ "--workers",
+ type=int,
+ default=max(1, min(32, (os.cpu_count() or 1) + 4)),
+ help="Number of worker threads for parallel hull generation.",
+ )
+ parser.add_argument(
+ "--object_ids",
+ nargs="+",
+ default=None,
+ help="Optional list of object ids to process. When omitted, all objects are processed.",
+ )
+ parser.add_argument(
+ "--split_mode",
+ type=str,
+ default="none",
+ choices=["none", "all", "list"],
+ help="Whether to split mesh into 4 slices before COACD decomposition.",
+ )
+ parser.add_argument(
+ "--split_list",
+ type=Path,
+ default=None,
+ help="Optional JSON list/dict specifying object ids to split when split_mode=list.",
+ )
+ parser.add_argument(
+ "--save_parts",
+ action="store_true",
+ help="Save each decomposed collision part as object_XXX_parts/collision_part_YYY.obj.",
+ )
+ parser.add_argument(
+ "--coacd_threshold",
+ type=float,
+ default=0.1,
+ help="COACD threshold.",
+ )
+ parser.add_argument(
+ "--coacd_preprocess_mode",
+ type=str,
+ default="off",
+ help="COACD preprocess mode.",
+ )
+ parser.add_argument(
+ "--target_faces_per_part",
+ type=int,
+ default=1000,
+ help="Target number of faces for each part before COACD. <=0 disables simplification.",
+ )
+ args = parser.parse_args()
+
+ if not args.unposed_dir.exists():
+ raise FileNotFoundError(f"Unposed mesh directory not found: {args.unposed_dir}")
+
+ object_ids = load_object_ids(args.unposed_dir)
+ if not object_ids:
+ raise FileNotFoundError(f"No .glb files found in: {args.unposed_dir}")
+
+ # Resolve requested ids (may be numeric suffixes or stems)
+ requested_object_ids = resolve_requested_object_ids(args.object_ids, object_ids)
+ if requested_object_ids is not None:
+ requested_object_ids_set = set(requested_object_ids)
+ missing_requested = sorted(object_id for object_id in requested_object_ids_set if object_id not in set(object_ids))
+ if missing_requested:
+ raise FileNotFoundError(
+ "Requested object ids were not found in the unposed directory: "
+ + ", ".join(missing_requested)
+ )
+ object_ids = [object_id for object_id in object_ids if object_id in requested_object_ids_set]
+ if not object_ids:
+ raise FileNotFoundError("No matching object ids remained after applying --object_ids")
+
+ args.output_dir.mkdir(parents=True, exist_ok=True)
+ args.manifest.parent.mkdir(parents=True, exist_ok=True)
+
+ if args.workers < 1:
+ raise ValueError("--workers must be >= 1")
+ if args.split_mode == "list" and args.split_list is None:
+ raise ValueError("--split_list is required when --split_mode=list")
+
+ split_object_ids = load_split_object_ids(args.split_list, object_ids)
+
+ manifest = {}
+ failed_object_ids: list[str] = []
+ ctx = get_context("spawn")
+ manager = Manager()
+ worker_start_queue = manager.Queue()
+ executor = ProcessPoolExecutor(max_workers=args.workers, mp_context=ctx)
+ futures = {
+ executor.submit(
+ process_object_with_error_context,
+ object_id,
+ args.unposed_dir,
+ args.output_dir,
+ args.split_mode,
+ split_object_ids,
+ args.save_parts,
+ args.coacd_threshold,
+ args.coacd_preprocess_mode,
+ args.target_faces_per_part,
+ worker_start_queue,
+ ): object_id
+ for object_id in object_ids
+ }
+ submitted_at = {future: time.monotonic() for future in futures}
+ pending = set(futures)
+ timed_out_futures = set()
+ running_worker_pids: dict[str, int] = {}
+ try:
+ while pending:
+ while True:
+ try:
+ object_id, pid = worker_start_queue.get_nowait()
+ except Empty:
+ break
+ running_worker_pids[object_id] = pid
+
+ done, pending = wait(pending, timeout=1, return_when=FIRST_COMPLETED)
+ for future in done:
+ object_id = futures[future]
+ if future in timed_out_futures:
+ continue
+ try:
+ item_key, item_value, in_name, out_name = future.result()
+ except Exception as exc:
+ failed_object_ids.append(object_id)
+ print(f"[FAILED] {object_id}.glb -> {exc}", flush=True)
+ continue
+ manifest[item_key] = item_value
+ print(f"[OK] {in_name} -> {out_name}")
+
+ now = time.monotonic()
+ overdue = [future for future in pending if now - submitted_at[future] > TASK_TIMEOUT_SECONDS]
+ if overdue:
+ for future in overdue:
+ object_id = futures[future]
+ failed_object_ids.append(object_id)
+ timed_out_futures.add(future)
+ pending.remove(future)
+ print(f"[TIMEOUT] {object_id}.glb", flush=True)
+ pid = running_worker_pids.get(object_id)
+ if pid is not None:
+ terminate_executor_worker(executor, pid)
+ finally:
+ manager.shutdown()
+ executor.shutdown(wait=False, cancel_futures=True)
+
+ # Keep manifest entries stable by object id order.
+ manifest = dict(sorted(manifest.items(), key=lambda item: item[0]))
+
+ with args.manifest.open("w", encoding="utf-8") as f:
+ json.dump(manifest, f, ensure_ascii=False, indent=2)
+
+ print(f"Saved hull manifest to: {args.manifest}")
+ print(
+ f"{SUMMARY_PREFIX} "
+ + json.dumps(
+ {
+ "failed_object_ids": sorted(dict.fromkeys(failed_object_ids)),
+ "succeeded_object_ids": sorted(manifest.keys()),
+ },
+ ensure_ascii=False,
+ )
+ )
+
+
+if __name__ == "__main__":
+ main()
\ No newline at end of file
diff --git a/vlm_agent/evolution/tools/check_object_collisions.py b/vlm_agent/evolution/tools/check_object_collisions.py
new file mode 100644
index 0000000..fe3adff
--- /dev/null
+++ b/vlm_agent/evolution/tools/check_object_collisions.py
@@ -0,0 +1,388 @@
+import argparse
+import json
+from pathlib import Path
+
+import numpy as np
+import math
+
+try:
+ import pybullet as p
+except ImportError as e:
+ raise ImportError(
+ "pybullet is required for collision checking. Install it with: pip install pybullet"
+ ) from e
+
+
+def as_vector(value, expected_sizes):
+ arr = np.asarray(value, dtype=np.float32).reshape(-1)
+ if arr.size not in expected_sizes:
+ raise ValueError(f"Expected sizes {expected_sizes}, but got shape {arr.shape}")
+ return arr
+
+
+def parse_scale(value):
+ arr = as_vector(value, {1, 3})
+ if arr.size == 1:
+ return np.repeat(arr, 3)
+ return arr
+
+
+def parse_translation(value):
+ return as_vector(value, {3})
+
+
+def parse_rotation_wxyz(value):
+ quat = as_vector(value, {4})
+ norm = np.linalg.norm(quat)
+ if norm <= 1e-8:
+ raise ValueError("Quaternion norm is zero.")
+ return quat / norm
+
+
+def quat_wxyz_to_xyzw(value):
+ quat = as_vector(value, {4})
+ return np.array([quat[1], quat[2], quat[3], quat[0]], dtype=np.float32)
+
+
+def format_object_id(object_id):
+ object_id = str(object_id)
+ if object_id and object_id[0].isdigit():
+ return f"object_{object_id}"
+ return object_id
+
+
+def resolve_requested_object_ids(requested_object_ids):
+ if requested_object_ids is None:
+ return None
+ if isinstance(requested_object_ids, (str, int)):
+ requested_object_ids = [requested_object_ids]
+ return [format_object_id(object_id) for object_id in requested_object_ids]
+
+
+def load_layout(layout_path: Path):
+ with layout_path.open("r", encoding="utf-8") as f:
+ raw = json.load(f)
+
+ translation = parse_translation(raw["translation"])
+ rotation_wxyz = parse_rotation_wxyz(raw["rotation"])
+ scale = parse_scale(raw["scale"])
+
+ return raw, translation, rotation_wxyz, scale
+
+
+def load_obj_extents(obj_path: Path):
+ vertices = []
+ with obj_path.open("r", encoding="utf-8") as f:
+ for line in f:
+ if not line.startswith("v "):
+ continue
+ parts = line.split()
+ if len(parts) < 4:
+ continue
+ vertices.append([float(parts[1]), float(parts[2]), float(parts[3])])
+
+ if not vertices:
+ raise ValueError(f"No vertices found in collision hull: {obj_path}")
+
+ vertices = np.asarray(vertices, dtype=np.float32)
+ return vertices.max(axis=0) - vertices.min(axis=0)
+
+
+def should_skip_thin_object(hull_path: Path, scale, thin_ratio_threshold: float):
+ extents = load_obj_extents(hull_path)
+ final_sizes = extents * np.asarray(scale, dtype=np.float32).reshape(3)
+ smallest_edge = float(np.min(final_sizes))
+ if smallest_edge <= 1e-8:
+ return True, float("inf"), final_sizes
+
+ ratio = float(np.max(final_sizes) / smallest_edge)
+ return ratio > thin_ratio_threshold, ratio, final_sizes
+
+
+def load_object_ids(layout_dir: Path, hull_dir: Path, requested_object_ids=None):
+ layout_ids = {p.stem for p in layout_dir.glob("*.json")}
+ hull_ids = {p.stem for p in hull_dir.glob("*.obj")}
+ matched_ids = sorted(layout_ids & hull_ids)
+
+ normalized_requested_ids = resolve_requested_object_ids(requested_object_ids)
+ if normalized_requested_ids is None:
+ return matched_ids
+
+ missing_ids = [object_id for object_id in normalized_requested_ids if object_id not in matched_ids]
+ if missing_ids:
+ raise FileNotFoundError(
+ f"Requested object ids not found in both layout and hull directories: {missing_ids}. "
+ f"Matched ids: {matched_ids}"
+ )
+ return normalized_requested_ids
+
+
+def extract_contact_bodies(contact):
+ if hasattr(contact, "bodyUniqueIdA") and hasattr(contact, "bodyUniqueIdB"):
+ return contact.bodyUniqueIdA, contact.bodyUniqueIdB
+ if isinstance(contact, (list, tuple)):
+ if len(contact) >= 14:
+ return contact[1], contact[2]
+ if len(contact) >= 2:
+ return contact[0], contact[1]
+ return None, None
+
+
+def sanitize_for_json(obj):
+ """Recursively replace non-finite floats with None so JSON is valid."""
+ if isinstance(obj, float):
+ return obj if math.isfinite(obj) else "inf"
+ if isinstance(obj, dict):
+ return {k: sanitize_for_json(v) for k, v in obj.items()}
+ if isinstance(obj, list):
+ return [sanitize_for_json(v) for v in obj]
+ return obj
+
+
+def run_collision_check(
+ layout_dir: Path,
+ hull_dir: Path,
+ output_path: Path | None,
+ object_ids=None,
+ steps: int = 1,
+ mass: float = 10.0,
+ thin_ratio_threshold: float = 15.0,
+ anchor_id: str | None = None,
+ collision_distance_threshold: float = 6e-3,
+):
+ object_ids = load_object_ids(layout_dir, hull_dir, requested_object_ids=object_ids)
+ if not object_ids:
+ raise FileNotFoundError(
+ f"No matching object files in {layout_dir} and {hull_dir}."
+ )
+
+ client_id = p.connect(p.DIRECT)
+ if client_id < 0:
+ raise RuntimeError("Failed to connect to PyBullet in DIRECT mode.")
+
+ try:
+ p.resetSimulation(physicsClientId=client_id)
+ p.setGravity(0.0, 0.0, 0.0, physicsClientId=client_id)
+ p.setPhysicsEngineParameter(enableFileCaching=0, physicsClientId=client_id)
+
+ body_to_object_id = {}
+ object_sizes = {}
+ skipped_objects = []
+ flags = 0
+ if hasattr(p, "GEOM_FORCE_CONVEX_MESH"):
+ flags |= p.GEOM_FORCE_CONVEX_MESH
+
+ for object_id in object_ids:
+ layout_path = layout_dir / f"{format_object_id(object_id)}.json"
+ hull_path = hull_dir / f"{format_object_id(object_id)}.obj"
+
+ _, translation, rotation_wxyz, scale = load_layout(layout_path)
+ should_skip, ratio, final_sizes = should_skip_thin_object(
+ hull_path,
+ scale,
+ thin_ratio_threshold,
+ )
+ object_sizes[format_object_id(object_id)] = final_sizes.tolist()
+ if should_skip:
+ skipped_objects.append(
+ {
+ "object_id": format_object_id(object_id),
+ "ratio": ratio,
+ }
+ )
+ continue
+
+ quat_xyzw = quat_wxyz_to_xyzw(rotation_wxyz).tolist()
+
+ shape_id = p.createCollisionShape(
+ shapeType=p.GEOM_MESH,
+ fileName=str(hull_path),
+ meshScale=scale.tolist(),
+ flags=flags,
+ physicsClientId=client_id,
+ )
+ if shape_id < 0:
+ raise RuntimeError(f"Failed to create collision shape: {hull_path}")
+
+ body_id = p.createMultiBody(
+ baseMass=float(mass),
+ baseCollisionShapeIndex=shape_id,
+ basePosition=translation.tolist(),
+ baseOrientation=quat_xyzw,
+ physicsClientId=client_id,
+ )
+ if body_id < 0:
+ raise RuntimeError(f"Failed to create body for: {format_object_id(object_id)}")
+
+ body_to_object_id[body_id] = format_object_id(object_id)
+
+ for _ in range(max(int(steps), 1)):
+ p.stepSimulation(physicsClientId=client_id)
+
+ if hasattr(p, "performCollisionDetection"):
+ p.performCollisionDetection(physicsClientId=client_id)
+
+ body_ids = list(body_to_object_id.keys())
+
+ pair_stats = {}
+
+ penetration_eps = -collision_distance_threshold
+
+ for i in range(len(body_ids)):
+ for j in range(i + 1, len(body_ids)):
+
+ body0 = body_ids[i]
+ body1 = body_ids[j]
+
+ obj0 = body_to_object_id[body0]
+ obj1 = body_to_object_id[body1]
+
+ if anchor_id and (obj0 == anchor_id or obj1 == anchor_id):
+ continue
+
+ pts = p.getClosestPoints(
+ bodyA=body0,
+ bodyB=body1,
+ distance=collision_distance_threshold,
+ physicsClientId=client_id,
+ )
+
+ if not pts:
+ continue
+
+ min_distance = min(pt[8] for pt in pts)
+
+ if min_distance > collision_distance_threshold:
+ continue
+
+ if min_distance >= penetration_eps:
+ continue
+
+ key = tuple(sorted([obj0, obj1]))
+
+ pair_stats[key] = {
+ "a": key[0],
+ "b": key[1],
+ "contacts": len(pts),
+ "min_distance": float(min_distance),
+ "size_a": list(object_sizes.get(obj0, [])),
+ "size_b": list(object_sizes.get(obj1, [])),
+ }
+ finally:
+ p.disconnect(physicsClientId=client_id)
+
+ report = {
+ "num_objects": len(object_ids),
+ "objects": list(object_ids),
+ "num_skipped_objects": len(skipped_objects),
+ "skipped_objects": skipped_objects,
+ "thin_ratio_threshold": float(thin_ratio_threshold),
+ "num_steps": int(max(int(steps), 1)),
+ "collision_distance_threshold": float(collision_distance_threshold),
+ "num_pairs": len(pair_stats),
+ "collisions": list(pair_stats.values()),
+ }
+ # Sanitize report so JSON doesn't contain Infinity/NaN which is not valid JSON
+ sanitized_report = sanitize_for_json(report)
+
+ if output_path is not None:
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ with output_path.open("w", encoding="utf-8") as f:
+ json.dump(sanitized_report, f, indent=2)
+ print(f"[OK] Saved collision report: {output_path}")
+
+ print(json.dumps(sanitized_report, indent=2))
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description=(
+ "Check pairwise collisions between objects using collision hulls and layout poses."
+ )
+ )
+ parser.add_argument(
+ "--layout_dir",
+ type=Path,
+ default=Path("outputs/generated/group_object_020/layout_org/round_1/layout_params"),
+ help="Directory containing object_XXX.json layout files.",
+ )
+ parser.add_argument(
+ "--hull_dir",
+ type=Path,
+ default=Path("outputs/collision_hulls"),
+ help="Directory containing object_XXX.obj collision hull files.",
+ )
+ parser.add_argument(
+ "--output",
+ type=Path,
+ default=None,
+ help="Optional JSON output path for collision report.",
+ )
+ parser.add_argument(
+ "--object_ids",
+ nargs="+",
+ default=None,
+ help=(
+ "Optional subset of object ids to check (XXX in object_XXX). "
+ "Defaults to all matched objects."
+ ),
+ )
+ parser.add_argument(
+ "--steps",
+ type=int,
+ default=1,
+ help="Number of simulation steps to run before checking contacts.",
+ )
+ parser.add_argument(
+ "--mass",
+ type=float,
+ default=10.0,
+ help="Density proxy for collision shape loading.",
+ )
+ parser.add_argument(
+ "--anchor_id",
+ type=str,
+ default=None,
+ help="Optional anchor object id whose collisions with all other objects are ignored.",
+ )
+ parser.add_argument(
+ "--thin_ratio_threshold",
+ type=float,
+ default=15.0,
+ help="Skip objects whose final size aspect ratio exceeds this value.",
+ )
+ parser.add_argument(
+ "--collision_distance_threshold",
+ type=float,
+ default=6e-3,
+ help="Penetration threshold (positive value, collision when min_distance < -threshold).",
+ )
+
+ args = parser.parse_args()
+
+ if not args.layout_dir.exists():
+ raise FileNotFoundError(f"Layout directory not found: {args.layout_dir}")
+ if not args.hull_dir.exists():
+ raise FileNotFoundError(f"Hull directory not found: {args.hull_dir}")
+ if args.mass <= 0:
+ raise ValueError("--mass must be positive.")
+ if args.steps <= 0:
+ raise ValueError("--steps must be positive.")
+ if args.thin_ratio_threshold <= 0:
+ raise ValueError("--thin_ratio_threshold must be positive.")
+
+ run_collision_check(
+ layout_dir=args.layout_dir,
+ hull_dir=args.hull_dir,
+ output_path=args.output,
+ object_ids=args.object_ids,
+ steps=args.steps,
+ mass=args.mass,
+ thin_ratio_threshold=args.thin_ratio_threshold,
+ anchor_id=args.anchor_id,
+ collision_distance_threshold=args.collision_distance_threshold,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/evolution/tools/compose_scene_from_layout.py b/vlm_agent/evolution/tools/compose_scene_from_layout.py
new file mode 100644
index 0000000..7214c5a
--- /dev/null
+++ b/vlm_agent/evolution/tools/compose_scene_from_layout.py
@@ -0,0 +1,162 @@
+import argparse
+import json
+import os
+from pathlib import Path
+
+import numpy as np
+import torch
+import trimesh
+from pytorch3d.transforms import quaternion_to_matrix
+
+os.environ["CUDA_HOME"] = os.environ["CONDA_PREFIX"]
+os.environ["LIDRA_SKIP_INIT"] = "true"
+
+from sam3d_objects.data.dataset.tdfy.transforms_3d import compose_transform
+
+R_OPENGL_TO_P3D = np.array([[-1, 0, 0], [0, 1, 0], [0, 0, -1]], dtype=np.float32)
+R_P3D_TO_OPENGL = R_OPENGL_TO_P3D
+
+
+def quaternion_to_matrix_numpy(quat):
+ """Convert quaternion (w, x, y, z) to a 3x3 rotation matrix."""
+ q = np.asarray(quat, dtype=np.float32).reshape(-1)
+ if q.shape[0] != 4:
+ raise ValueError(f"Quaternion must have 4 elements, got shape {q.shape}.")
+
+ norm = np.linalg.norm(q)
+ if norm <= 0:
+ raise ValueError("Quaternion norm is zero.")
+ w, x, y, z = q / norm
+
+ return np.array(
+ [
+ [1 - 2 * (y * y + z * z), 2 * (x * y - z * w), 2 * (x * z + y * w)],
+ [2 * (x * y + z * w), 1 - 2 * (x * x + z * z), 2 * (y * z - x * w)],
+ [2 * (x * z - y * w), 2 * (y * z + x * w), 1 - 2 * (x * x + y * y)],
+ ],
+ dtype=np.float32,
+ )
+
+
+def parse_scale(scale_value):
+ scale = np.asarray(scale_value, dtype=np.float32).reshape(-1)
+ if scale.size == 1:
+ return np.repeat(scale, 3)
+ if scale.size == 3:
+ return scale
+ raise ValueError(f"Scale must be scalar or 3D vector, got shape {scale.shape}.")
+
+
+def apply_layout_to_unposed_mesh(mesh, layout):
+ """Apply layout transform to an unposed mesh using the demo.py pipeline logic."""
+ translation = torch.tensor(layout["translation"], dtype=torch.float32)
+ rotation_q = torch.tensor(layout["rotation"], dtype=torch.float32)
+ scale = torch.tensor(layout["scale"], dtype=torch.float32)
+
+ vertices = mesh.vertices.astype(np.float32) @ R_OPENGL_TO_P3D
+ vertices_tensor = torch.from_numpy(vertices).float().to(rotation_q.device)
+ rotation_m = quaternion_to_matrix(rotation_q).transpose(-1, -2) # Transpose to apply to row vectors
+ l2c_transform = compose_transform(
+ scale=scale,
+ rotation=rotation_m,
+ translation=translation,
+ )
+ vertices = l2c_transform.transform_points(vertices_tensor.unsqueeze(0))
+ mesh.vertices = vertices.squeeze(0).cpu().numpy() @ R_P3D_TO_OPENGL
+ return mesh
+
+
+def load_object_ids(layout_dir, unposed_dir):
+ # Collect stems of all layout json files and stems of unposed mesh files
+ layout_stems = {p.stem for p in layout_dir.glob("*.json")}
+ mesh_stems = set()
+ for pattern in ("*.glb", "*.gltf", "*.obj", "*.ply"):
+ mesh_stems.update({p.stem for p in unposed_dir.glob(pattern)})
+
+ # Return the sorted list of stems that appear in both layout and unposed dirs
+ return sorted(layout_stems & mesh_stems)
+
+
+def compose_scene(layout_dir, unposed_dir):
+ object_ids = load_object_ids(layout_dir, unposed_dir)
+ if not object_ids:
+ raise FileNotFoundError(
+ f"No matching object files found in {layout_dir} and {unposed_dir}."
+ )
+
+ posed_meshes = []
+ for object_id in object_ids:
+ # layout files are matched by stem (e.g. "chair_01.json" -> stem "chair_01")
+ layout_path = layout_dir / f"{object_id}.json"
+
+ # find a matching mesh file with a supported extension
+ mesh_path = None
+ for ext in (".glb", ".gltf", ".obj", ".ply"):
+ candidate = unposed_dir / f"{object_id}{ext}"
+ if candidate.exists():
+ mesh_path = candidate
+ break
+ if mesh_path is None:
+ raise FileNotFoundError(f"No unposed mesh found for object '{object_id}' in {unposed_dir}")
+
+ with layout_path.open("r", encoding="utf-8") as f:
+ layout = json.load(f)
+
+ mesh = trimesh.load(mesh_path, force="mesh")
+ posed_mesh = apply_layout_to_unposed_mesh(mesh, layout)
+ posed_meshes.append(posed_mesh)
+
+ if len(posed_meshes) == 1:
+ return posed_meshes[0]
+
+ # When multiple posed meshes exist, build a Scene and add each mesh as a separate geometry.
+ # Exporting a Scene preserves per-mesh textures and PBR materials; concatenating meshes can lose texture/material
+ # information because textures would need to be atlased.
+ scene = trimesh.Scene()
+ for id, pm in zip(object_ids, posed_meshes):
+ scene.add_geometry(pm, node_name=id)
+ return scene
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Rebuild scene mesh from unposed meshes and layout parameters."
+ )
+ parser.add_argument(
+ "--layout_dir",
+ type=Path,
+ default=Path("outputs/layout_params"),
+ help="Directory containing object_XXX.json layout files.",
+ )
+ parser.add_argument(
+ "--unposed_dir",
+ type=Path,
+ default=Path("outputs/unposed_objects"),
+ help="Directory containing object_XXX.glb unposed meshes.",
+ )
+ parser.add_argument(
+ "--output",
+ type=Path,
+ default=Path("outputs/composed_scene.glb"),
+ help="Output path for the composed scene mesh.",
+ )
+ args = parser.parse_args()
+
+ layout_dir = args.layout_dir
+ unposed_dir = args.unposed_dir
+ output_path = args.output
+
+ if not layout_dir.exists():
+ raise FileNotFoundError(f"Layout directory not found: {layout_dir}")
+ if not unposed_dir.exists():
+ raise FileNotFoundError(f"Unposed mesh directory not found: {unposed_dir}")
+
+ scene_mesh = compose_scene(layout_dir, unposed_dir)
+
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ scene_mesh.export(output_path)
+ print(f"Saved recomposed scene to: {output_path}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/evolution/tools/fix_layout_containment.py b/vlm_agent/evolution/tools/fix_layout_containment.py
new file mode 100644
index 0000000..736917c
--- /dev/null
+++ b/vlm_agent/evolution/tools/fix_layout_containment.py
@@ -0,0 +1,280 @@
+import argparse
+import json
+from copy import deepcopy
+from pathlib import Path
+import sys
+
+sys.path.append(str(Path(__file__).parent.parent))
+
+import numpy as np
+import torch
+import trimesh
+
+from pytorch3d.transforms import quaternion_to_matrix
+
+from render_scene_from_layout import (
+ apply_layout_to_unposed_mesh,
+ load_object_ids,
+ load_layout,
+ normalize_object_id
+)
+
+
+def fix_containment_and_save(
+ layout_dir: Path,
+ unposed_dir: Path,
+ output_layout_dir: Path,
+ requested_object_ids=None,
+ anchor_id=None,
+ tolerated_out_of_bounds: float = 0.0,
+ detect_only: bool = False,
+):
+ """
+ Main pipeline to fix the containment relationship between the container (anchor)
+ and other objects in BEV perspective, then output updated layout JSONs.
+ """
+ if anchor_id is None:
+ raise ValueError("An anchor_id (container object) must be specified for this containment correction.")
+ if tolerated_out_of_bounds < 0:
+ raise ValueError("tolerated_out_of_bounds must be non-negative.")
+
+ # 1. Load object IDs and filter matched ones
+ object_ids = load_object_ids(layout_dir, unposed_dir, requested_object_ids=requested_object_ids)
+ normalized_anchor_id = normalize_object_id(anchor_id)
+
+ if normalized_anchor_id not in object_ids:
+ raise FileNotFoundError(f"Container anchor_id {normalized_anchor_id} not found in the matched object IDs.")
+
+ # 2. Load all meshes and their initial layout parameters
+ posed_meshes = []
+ layout_params = []
+ anchor_index = -1
+
+ for idx, object_id in enumerate(object_ids):
+ if object_id == normalized_anchor_id:
+ anchor_index = idx
+ layout_path = layout_dir / f"{object_id}.json"
+ mesh_path = unposed_dir / f"{object_id}.glb"
+
+ layout, _, _, _ = load_layout(layout_path)
+ mesh = trimesh.load(mesh_path, force="mesh")
+ posed_mesh = apply_layout_to_unposed_mesh(mesh, layout)
+
+ posed_meshes.append(posed_mesh)
+ layout_params.append(deepcopy(layout))
+
+ # 3. Retrieve container (anchor) details in World Space
+ anchor_layout = layout_params[anchor_index]
+ anchor_rotation = np.asarray(anchor_layout["rotation"], dtype=np.float32).reshape(4)
+
+ # Convert quaternion to row-major matrix to align with the render script's logic
+ anchor_rot_mat = (
+ quaternion_to_matrix(torch.from_numpy(anchor_rotation))
+ .transpose(-1, -2)
+ .cpu()
+ .numpy()
+ )
+ # Rotation fix transforms coordinates from World Space into Container Local Space
+ rotation_fix = anchor_rot_mat.T.astype(np.float32)
+
+ # 4. Compute Container BEV extents (using X and Z axes as BEV plane)
+ # Re-align container mesh to local space to find precise 3D/2D bounding box
+ anchor_local_vertices = posed_meshes[anchor_index].vertices.astype(np.float32) @ rotation_fix
+ c_min = anchor_local_vertices.min(axis=0)
+ c_max = anchor_local_vertices.max(axis=0)
+
+ # We use X and Z dimensions for BEV (Birds-Eye-View) plane bounding box
+ # as common in 3D scene representation pipelines
+ c_bev_min_x, c_bev_max_x = c_min[0], c_max[0]
+ c_bev_min_z, c_bev_max_z = c_min[2], c_max[2]
+
+ container_extent_x = c_bev_max_x - c_bev_min_x
+ container_extent_z = c_bev_max_z - c_bev_min_z
+ container_local_center_x = (c_bev_min_x + c_bev_max_x) / 2.0
+ container_local_center_z = (c_bev_min_z + c_bev_max_z) / 2.0
+ out_of_bounds_tolerance_ratio = float(tolerated_out_of_bounds)
+
+ print(
+ f"Container BEV local bounds -> X: [{c_bev_min_x:.3f}, {c_bev_max_x:.3f}], "
+ f"Z: [{c_bev_min_z:.3f}, {c_bev_max_z:.3f}]"
+ )
+
+ # 5. Process each contained object
+ output_layout_dir.mkdir(parents=True, exist_ok=True)
+ out_of_bounds_object_ids: list[str] = []
+ fixed_object_ids: list[str] = []
+ written_object_ids: list[str] = []
+
+ for idx, object_id in enumerate(object_ids):
+ current_layout = deepcopy(layout_params[idx])
+
+ # Skip correcting the container itself
+ if object_id == normalized_anchor_id:
+ output_path = output_layout_dir / f"{object_id}.json"
+ with open(output_path, "w", encoding="utf-8") as f:
+ json.dump(current_layout, f, ensure_ascii=False, indent=2)
+ written_object_ids.append(object_id)
+ continue
+
+ # Transform current object's vertices into Container Local Space
+ obj_local_vertices = posed_meshes[idx].vertices.astype(np.float32) @ rotation_fix
+
+ o_min = obj_local_vertices.min(axis=0)
+ o_max = obj_local_vertices.max(axis=0)
+
+ o_bev_min_x, o_bev_max_x = o_min[0], o_max[0]
+ o_bev_min_z, o_bev_max_z = o_min[2], o_max[2]
+
+ obj_occupied_x = o_bev_max_x - o_bev_min_x
+ obj_occupied_z = o_bev_max_z - o_bev_min_z
+ out_of_bounds_tolerance_x = out_of_bounds_tolerance_ratio * obj_occupied_x
+ out_of_bounds_tolerance_z = out_of_bounds_tolerance_ratio * obj_occupied_z
+
+ # Calculate current object local translation components
+ obj_world_trans = np.asarray(current_layout["translation"], dtype=np.float32).reshape(3)
+ obj_local_trans = obj_world_trans @ rotation_fix
+
+ fixed_local_x = obj_local_trans[0]
+ fixed_local_z = obj_local_trans[2]
+ object_out_of_bounds = False
+
+ # --- BEV Dimension X Correction ---
+ if obj_occupied_x > container_extent_x + out_of_bounds_tolerance_x * 2:
+ # Case 1: Object is larger than container in X axis -> Center it
+ object_out_of_bounds = True
+ fixed_local_x = container_local_center_x
+ print(f"[{object_id}] X-axis size exceeds container. Force Centered.")
+ else:
+ # Case 2: Standard bounded box constraint check (1-2 borders out of bounds)
+ offset_x_max = o_bev_max_x - c_bev_max_x
+ offset_x_min = c_bev_min_x - o_bev_min_x
+
+ if offset_x_max > out_of_bounds_tolerance_x: # Out of right bound -> push left
+ object_out_of_bounds = True
+ fixed_local_x -= offset_x_max
+ print(f"[{object_id}] Right border out of bounds by {offset_x_max:.3f}. Shifted left.")
+ if offset_x_min > out_of_bounds_tolerance_x: # Out of left bound -> push right
+ object_out_of_bounds = True
+ fixed_local_x += offset_x_min
+ print(f"[{object_id}] Left border out of bounds by {offset_x_min:.3f}. Shifted right.")
+
+ # --- BEV Dimension Z Correction ---
+ if obj_occupied_z > container_extent_z + out_of_bounds_tolerance_z * 2:
+ # Case 1: Object is larger than container in Z axis -> Center it
+ object_out_of_bounds = True
+ fixed_local_z = container_local_center_z
+ print(f"[{object_id}] Z-axis size exceeds container. Force Centered.")
+ else:
+ # Case 2: Standard bounded box constraint check
+ offset_z_max = o_bev_max_z - c_bev_max_z
+ offset_z_min = c_bev_min_z - o_bev_min_z
+
+ if offset_z_max > out_of_bounds_tolerance_z: # Out of top bound -> push bottom
+ object_out_of_bounds = True
+ fixed_local_z -= offset_z_max
+ print(f"[{object_id}] Top border out of bounds by {offset_z_max:.3f}. Shifted down.")
+ elif offset_z_min > out_of_bounds_tolerance_z: # Out of bottom bound -> push top
+ object_out_of_bounds = True
+ fixed_local_z += offset_z_min
+ print(f"[{object_id}] Bottom border out of bounds by {offset_z_min:.3f}. Shifted up.")
+
+ if object_out_of_bounds:
+ out_of_bounds_object_ids.append(object_id)
+ if detect_only:
+ continue
+
+ # 6. Transform the updated local translation back to World Coordinate System
+ updated_local_trans = np.array([fixed_local_x, obj_local_trans[1], fixed_local_z], dtype=np.float32)
+
+ # Reverse the rotation fix mapping: World = Local @ rotation_fix.T
+ updated_world_trans = updated_local_trans @ rotation_fix.T
+
+ # Update layout parameters
+ current_layout["translation"] = [updated_world_trans.tolist()]
+ fixed_object_ids.append(object_id)
+
+ # Save layout to target directory
+ if not detect_only:
+ output_path = output_layout_dir / f"{object_id}.json"
+ with open(output_path, "w", encoding="utf-8") as f:
+ json.dump(current_layout, f, ensure_ascii=False, indent=2)
+ written_object_ids.append(object_id)
+
+ report = {
+ "anchor_id": normalized_anchor_id,
+ "requested_object_ids": list(object_ids),
+ "out_of_bounds_object_ids": out_of_bounds_object_ids,
+ "fixed_object_ids": fixed_object_ids,
+ "written_object_ids": written_object_ids,
+ "detect_only": detect_only,
+ "output_layout_dir": str(output_layout_dir),
+ }
+ print(f"FIX_LAYOUT_CONTAINMENT_SUMMARY: {json.dumps(report, ensure_ascii=False)}")
+ return report
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Fix object containment relationship by adjusting layout centers in Container BEV coordinates."
+ )
+ parser.add_argument(
+ "--layout_dir",
+ type=Path,
+ default=Path("outputs/layout_params"),
+ help="Directory containing original object_XXX.json layout files.",
+ )
+ parser.add_argument(
+ "--unposed_dir",
+ type=Path,
+ default=Path("outputs/unposed_objects"),
+ help="Directory containing object_XXX.glb unposed meshes.",
+ )
+ parser.add_argument(
+ "--output_layout_dir",
+ type=Path,
+ default=Path("outputs/layout_params_test"),
+ help="Directory where fixed object layout parameters will be saved.",
+ )
+ parser.add_argument(
+ "--object_ids",
+ nargs="+",
+ default=None,
+ help="Optional subset of object ids to process. Defaults to all matched objects.",
+ )
+ parser.add_argument(
+ "--anchor_id",
+ type=str,
+ required=True,
+ help="The object ID of the container (anchor) that defines the BEV bounds for containment correction.",
+ )
+ parser.add_argument(
+ "--tolerated_out_of_bounds",
+ type=float,
+ default=0.0,
+ help="Tolerance ratio for out-of-bounds checks in BEV units; the anchor BEV extent is not scaled.",
+ )
+ parser.add_argument(
+ "--detect_only",
+ action="store_true",
+ help="Only detect out-of-bounds objects and do not modify their layouts.",
+ )
+ args = parser.parse_args()
+
+ if not args.layout_dir.exists():
+ raise FileNotFoundError(f"Layout directory not found: {args.layout_dir}")
+ if not args.unposed_dir.exists():
+ raise FileNotFoundError(f"Unposed mesh directory not found: {args.unposed_dir}")
+
+ fix_containment_and_save(
+ layout_dir=args.layout_dir,
+ unposed_dir=args.unposed_dir,
+ output_layout_dir=args.output_layout_dir,
+ requested_object_ids=args.object_ids,
+ anchor_id=args.anchor_id,
+ tolerated_out_of_bounds=args.tolerated_out_of_bounds,
+ detect_only=args.detect_only,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/evolution/tools/generate_room_mesh.py b/vlm_agent/evolution/tools/generate_room_mesh.py
new file mode 100644
index 0000000..c03cc5b
--- /dev/null
+++ b/vlm_agent/evolution/tools/generate_room_mesh.py
@@ -0,0 +1,539 @@
+import argparse
+import json
+import os
+from pathlib import Path
+import numpy as np
+import trimesh
+from PIL import Image
+
+from trimesh.visual.material import PBRMaterial
+
+R_OPENGL_TO_P3D = np.array([[-1, 0, 0], [0, 1, 0], [0, 0, -1]], dtype=np.float32)
+R_P3D_TO_OPENGL = R_OPENGL_TO_P3D
+
+DEFAULT_TEXTURE_RESOLUTION = "2K"
+DEFAULT_TEXTURE_FORMAT = "PNG"
+TEXTURE_RESOLUTIONS = ("1K", "2K", "4K", "8K", "12K", "16K")
+TEXTURE_FORMATS = ("PNG", "JPG")
+
+
+def env_texture_resolution() -> str:
+ """Resolve the texture resolution from the environment (default 2K)."""
+ value = os.environ.get("SCENEMOSAIC_MATERIAL_RESOLUTION")
+ return value if value in TEXTURE_RESOLUTIONS else DEFAULT_TEXTURE_RESOLUTION
+
+
+def env_texture_format() -> str:
+ """Resolve the texture format from the environment (default PNG)."""
+ value = os.environ.get("SCENEMOSAIC_MATERIAL_FORMAT")
+ return value if value in TEXTURE_FORMATS else DEFAULT_TEXTURE_FORMAT
+
+
+def texture_path(
+ material_dir: Path,
+ channel: str,
+ *,
+ resolution: str | None = None,
+ file_format: str | None = None,
+) -> Path:
+ """Return the path of one ambientCG texture of ``material_dir``.
+
+ ambientCG names textures ``<material>_<res>-<fmt>_<channel>.<ext>`` where
+ the filename label uses the format in uppercase (e.g. ``2K-PNG_Color.png``
+ for PNG, ``2K-JPG_Color.jpg`` for JPG). ``channel`` is the map suffix such
+ as ``Color``, ``NormalGL`` or ``Roughness``. Unset arguments fall back to
+ the environment overrides, then to 2K PNG.
+ """
+ resolution = resolution or env_texture_resolution()
+ file_format = file_format or env_texture_format()
+ extension = file_format.lower()
+ return material_dir / (
+ f"{material_dir.name}_{resolution}-{file_format}_{channel}.{extension}"
+ )
+
+
+def load_image(path: Path, mode: str = "RGB"):
+ with Image.open(path) as image:
+ return image.convert(mode)
+
+
+def quaternion_to_matrix_numpy(quat):
+ q = np.asarray(quat, dtype=np.float32).reshape(-1)
+ if q.shape[0] != 4:
+ raise ValueError("Quaternion must have 4 elements.")
+ norm = np.linalg.norm(q)
+ if norm <= 0:
+ raise ValueError("Quaternion norm is zero.")
+ w, x, y, z = q / norm
+ return np.array(
+ [
+ [1 - 2 * (y * y + z * z), 2 * (x * y - z * w), 2 * (x * z + y * w)],
+ [2 * (x * y + z * w), 1 - 2 * (x * x + z * z), 2 * (y * z - x * w)],
+ [2 * (x * z - y * w), 2 * (y * z + x * w), 1 - 2 * (x * x + y * y)],
+ ],
+ dtype=np.float32,
+ )
+
+
+def parse_scale(scale_value):
+ scale = np.asarray(scale_value, dtype=np.float32).reshape(-1)
+ if scale.size == 1:
+ return np.repeat(scale, 3)
+ if scale.size == 3:
+ return scale
+ raise ValueError("Scale must be scalar or 3D vector.")
+
+
+def yaw_to_quaternion(yaw: float) -> list:
+ half = -0.5 * yaw
+ return [float(np.cos(half)), 0.0, float(np.sin(half)), 0.0]
+
+
+def rotate_points_xz(points: np.ndarray, yaw: float) -> np.ndarray:
+ pts = np.asarray(points, dtype=np.float32)
+ c = float(np.cos(yaw))
+ s = float(np.sin(yaw))
+ rot = np.array([[c, -s], [s, c]], dtype=np.float32)
+ return pts @ rot.T
+
+
+def _signed_area_xz(points: np.ndarray) -> float:
+ pts = np.asarray(points, dtype=np.float32).reshape(-1, 2)
+ if pts.shape[0] < 3:
+ return 0.0
+ x = pts[:, 0]
+ z = pts[:, 1]
+ return float(0.5 * np.sum(x * np.roll(z, -1) - np.roll(x, -1) * z))
+
+
+def normalize_wall_loop_winding(walls: list[dict]) -> list[dict]:
+ if len(walls) < 3:
+ return list(walls)
+
+ walls.sort(key=lambda w: w.get("id", ""))
+ loop_points = np.asarray([wall["first_point"] for wall in walls], dtype=np.float32)
+ # Use clockwise winding for a consistent outward-normal convention.
+ if _signed_area_xz(loop_points) >= 0.0:
+ print("Wall loop is already clockwise, no need to reverse.")
+ return list(walls)
+
+ print("Reversing wall loop winding to be clockwise for consistent outward normals.")
+
+ normalized = []
+ for wall in reversed(walls):
+ normalized.append(
+ {
+ **wall,
+ "first_point": wall["second_point"],
+ "second_point": wall["first_point"],
+ }
+ )
+ return normalized
+
+
+def load_room_material(
+ material_dir: Path,
+ *,
+ resolution: str | None = None,
+ file_format: str | None = None,
+) -> PBRMaterial:
+ base_color_path = texture_path(
+ material_dir, "Color", resolution=resolution, file_format=file_format
+ )
+ normal_path = texture_path(
+ material_dir, "NormalGL", resolution=resolution, file_format=file_format
+ )
+ roughness_path = texture_path(
+ material_dir, "Roughness", resolution=resolution, file_format=file_format
+ )
+
+ if not base_color_path.exists():
+ raise FileNotFoundError(f"Base color texture not found: {base_color_path}")
+ if not normal_path.exists():
+ raise FileNotFoundError(f"Normal texture not found: {normal_path}")
+ if not roughness_path.exists():
+ raise FileNotFoundError(f"Roughness texture not found: {roughness_path}")
+
+ base_color = load_image(base_color_path, mode="RGBA")
+ normal_texture = load_image(normal_path, mode="RGB")
+
+ roughness_image = load_image(roughness_path, mode="L")
+ roughness_array = np.array(roughness_image, dtype=np.uint8)
+ metallic_roughness_array = np.zeros((roughness_array.shape[0], roughness_array.shape[1], 3), dtype=np.uint8)
+ metallic_roughness_array[..., 1] = roughness_array
+ metallic_roughness = Image.fromarray(metallic_roughness_array, mode="RGB")
+
+ return PBRMaterial(
+ roughnessFactor=1.0,
+ metallicFactor=0.0,
+ baseColorFactor=np.array([255, 255, 255, 255], dtype=np.uint8),
+ baseColorTexture=base_color,
+ normalTexture=normal_texture,
+ metallicRoughnessTexture=metallic_roughness,
+ doubleSided=True,
+ )
+
+
+def apply_layout_to_unposed_mesh(mesh: trimesh.Trimesh, layout: dict) -> trimesh.Trimesh:
+ translation = np.asarray(layout.get("translation", [0.0, 0.0, 0.0]), dtype=np.float32)
+ rotation = np.asarray(layout.get("rotation", [1.0, 0.0, 0.0, 0.0]), dtype=np.float32).reshape(4)
+ scale = parse_scale(layout.get("scale", [1.0]))
+
+ vertices = mesh.vertices.astype(np.float32) @ R_OPENGL_TO_P3D
+ # apply scale, rotation, translation
+ scaled = vertices * scale.reshape(1, 3)
+ R = quaternion_to_matrix_numpy(rotation)
+ rotated = scaled @ R.T
+ translated = rotated + translation.reshape(1, 3)
+ mesh.vertices = translated
+ return mesh
+
+
+def make_textured_mesh(mesh: trimesh.Trimesh, uv: np.ndarray, material: PBRMaterial) -> trimesh.Trimesh:
+ textured_mesh = mesh.copy()
+ textured_mesh.visual = trimesh.visual.TextureVisuals(uv=uv.astype(np.float32), material=material)
+ return textured_mesh
+
+
+def load_posed_meshes(layout_dir: Path, unposed_dir: Path):
+ layout_files = sorted(list(layout_dir.glob("object_*.json")))
+ posed = []
+ for lf in layout_files:
+ obj_id = lf.stem.split("object_", 1)[1]
+ mesh_path = unposed_dir / f"object_{obj_id}.glb"
+ if not mesh_path.exists():
+ continue
+ with lf.open("r", encoding="utf-8") as f:
+ layout = json.load(f)
+ mesh = trimesh.load(mesh_path, force="mesh")
+ posed_mesh = apply_layout_to_unposed_mesh(mesh, layout)
+ posed.append(posed_mesh)
+ return posed
+
+
+def find_floor_ceiling_heights(posed_meshes, objects_json_path: Path):
+ all_vertices = []
+ for m in posed_meshes:
+ all_vertices.append(m.vertices.astype(np.float32))
+ if all_vertices:
+ all_vertices = np.vstack(all_vertices)
+ y_min = float(np.min(all_vertices[:, 1]))
+ y_max = float(np.max(all_vertices[:, 1]))
+ else:
+ y_min, y_max = 0.0, 2.5
+
+ ceiling_h = None
+ if objects_json_path.exists():
+ with objects_json_path.open("r", encoding="utf-8") as f:
+ objs = json.load(f)
+ if "ceiling" in objs:
+ try:
+ ceiling_h = float(objs["ceiling"]["height"])
+ except Exception:
+ ceiling_h = None
+
+ if ceiling_h is None:
+ ceiling_h = y_max
+
+ return y_min, ceiling_h
+
+
+def compute_wall_planes_from_objects(objects_json_path: Path):
+ if not objects_json_path.exists():
+ raise FileNotFoundError(f"objects.json not found: {objects_json_path}")
+ with objects_json_path.open("r", encoding="utf-8") as f:
+ objs = json.load(f)
+ walls = normalize_wall_loop_winding(objs.get("walls", []))
+ # each wall: has first_point [x,z] and second_point [x,z]
+ planes = []
+ for w in walls:
+ p1 = np.asarray(w["first_point"], dtype=np.float32)
+ p2 = np.asarray(w["second_point"], dtype=np.float32)
+ dir2 = p2 - p1
+ if np.linalg.norm(dir2) <= 1e-8:
+ continue
+ dir2 = dir2 / np.linalg.norm(dir2)
+ # normal in xz plane for clockwise winding.
+ normal_xz = np.array([dir2[1], -dir2[0]], dtype=np.float32)
+ d = float(np.dot(normal_xz, p1))
+ planes.append({"normal_xz": normal_xz, "d": d, "p1": p1, "p2": p2, "wall_id": w.get("id")})
+ return sorted(planes, key=lambda pl: pl["wall_id"].split("_")[-1])
+
+
+def choose_wall_plane_extent(planes, posed_meshes, padding=0.00):
+ # For each plane, compute candidate plane at farthest vertex along normal
+ all_vertices = []
+ for m in posed_meshes:
+ all_vertices.append(m.vertices.astype(np.float32)[:, [0, 2]])
+ if all_vertices:
+ all_vertices = np.vstack(all_vertices)
+ else:
+ all_vertices = np.zeros((0, 2), dtype=np.float32)
+
+ chosen = []
+ for pl in planes:
+ nx = pl["normal_xz"]
+ d_original = pl["d"]
+ if all_vertices.size == 0:
+ d_use = d_original
+ else:
+ projections = all_vertices @ nx
+ max_proj = float(np.max(projections))
+ # choose farther along normal direction
+ d_use = max(d_original, max_proj)
+ # Apply padding to push wall further out
+ d_use = d_use + padding
+ chosen.append({**pl, "d_use": d_use})
+ print(f"wall {pl.get('wall_id')} normal={nx.tolist()} d_original={d_original:.6f} max_proj={max_proj:.6f} d_use={d_use:.6f}")
+ return chosen
+
+
+def intersect_two_planes_xz(n1, d1, n2, d2):
+ A = np.array([n1, n2], dtype=np.float32)
+ b = np.array([d1, d2], dtype=np.float32)
+ det = np.linalg.det(A)
+ if abs(det) <= 1e-8:
+ return None
+ sol = np.linalg.solve(A, b)
+ return sol
+
+
+def build_wall_corners(chosen_planes):
+ # assume ordered walls; compute 4 intersections
+ corners = []
+ N = len(chosen_planes)
+ for i in range(N):
+ pl1 = chosen_planes[i]
+ pl2 = chosen_planes[(i + 1) % N]
+ p = intersect_two_planes_xz(pl1["normal_xz"], pl1["d_use"], pl2["normal_xz"], pl2["d_use"])
+ if p is None:
+ # fallback to midpoint of segment points
+ point = np.asarray(pl1["p2"], dtype=np.float32)
+ else:
+ point = np.asarray(p, dtype=np.float32)
+ corners.append({"point": point, "wall_id": pl1.get("wall_id")})
+ print(f"Chosen plane p1: {pl1['wall_id']} p2: {pl2['wall_id']}: corner={point.tolist()}, d_use={pl1['d_use']:.3f}")
+ return corners
+
+
+def create_wall_mesh(wall_length, wall_height, thickness=0.02):
+ # Local wall mesh: length along +X, height along +Y, thickness along +Z, centered at origin.
+ half_l = 0.5 * float(wall_length)
+ half_h = 0.5 * float(wall_height)
+ half_t = 0.5 * float(thickness)
+
+ bl = np.array([-half_l, -half_h, -half_t], dtype=np.float32)
+ br = np.array([half_l, -half_h, -half_t], dtype=np.float32)
+ tr = np.array([half_l, half_h, -half_t], dtype=np.float32)
+ tl = np.array([-half_l, half_h, -half_t], dtype=np.float32)
+
+ bl2 = np.array([-half_l, -half_h, half_t], dtype=np.float32)
+ br2 = np.array([half_l, -half_h, half_t], dtype=np.float32)
+ tr2 = np.array([half_l, half_h, half_t], dtype=np.float32)
+ tl2 = np.array([-half_l, half_h, half_t], dtype=np.float32)
+
+ verts = np.vstack([
+ bl, br, tr, tl,
+ bl2, br2, tr2, tl2,
+ bl, br, br2, bl2,
+ tl, tr, tr2, tl2,
+ br, br2, tr2, tr,
+ bl, tl, tl2, bl2,
+ ])
+ faces = np.array([
+ [0, 1, 2], [0, 2, 3],
+ [4, 7, 6], [4, 6, 5],
+ [8, 9, 10], [8, 10, 11],
+ [12, 13, 14], [12, 14, 15],
+ [16, 17, 18], [16, 18, 19],
+ [20, 21, 22], [20, 22, 23],
+ ], dtype=np.int64)
+ mesh = trimesh.Trimesh(vertices=verts, faces=faces, process=False)
+
+ uv = np.array([
+ [0.0, 0.0], [wall_length, 0.0], [wall_length, wall_height], [0.0, wall_height],
+ [0.0, 0.0], [wall_length, 0.0], [wall_length, wall_height], [0.0, wall_height],
+ [0.0, 0.0], [wall_length, 0.0], [wall_length, thickness], [0.0, thickness],
+ [0.0, 0.0], [wall_length, 0.0], [wall_length, thickness], [0.0, thickness],
+ [0.0, 0.0], [thickness, 0.0], [thickness, wall_height], [0.0, wall_height],
+ [0.0, 0.0], [thickness, 0.0], [thickness, wall_height], [0.0, wall_height],
+ ], dtype=np.float32)
+ return mesh, uv
+
+
+def create_floor_or_ceiling_mesh(corners_xz, y_height):
+ # corners_xz is Nx2 in order (convex). Create vertices and fan triangulation from centroid.
+ pts3 = [np.array([p[0], y_height, p[1]], dtype=np.float32) for p in corners_xz]
+ centroid = np.mean(np.vstack(pts3), axis=0)
+ verts = np.vstack(pts3 + [centroid])
+ N = len(pts3)
+ faces = []
+ for i in range(N):
+ a = i
+ b = (i + 1) % N
+ c = N # centroid
+ faces.append([a, b, c])
+ mesh = trimesh.Trimesh(vertices=verts, faces=np.array(faces, dtype=np.int64), process=False)
+
+ corners_2d = np.asarray(corners_xz, dtype=np.float32)
+ uv_origin = corners_2d.min(axis=0)
+ uv = np.vstack([
+ corners_2d - uv_origin,
+ np.mean(corners_2d - uv_origin, axis=0, keepdims=True),
+ ])
+ return mesh, uv
+
+
+def recenter_mesh(mesh: trimesh.Trimesh) -> np.ndarray:
+ # Center mesh at origin and return the translation offset for layout.
+ bounds = mesh.bounds
+ center = bounds.mean(axis=0)
+ mesh.vertices = mesh.vertices - center.reshape(1, 3)
+ return center.astype(np.float32)
+
+
+def save_mesh_and_layout(
+ mesh: trimesh.Trimesh,
+ name: str,
+ unposed_dir: Path,
+ layout_dir: Path,
+ layout: dict | None = None,
+):
+ unposed_dir.mkdir(parents=True, exist_ok=True)
+ layout_dir.mkdir(parents=True, exist_ok=True)
+ mesh_path = unposed_dir / f"{name}.glb"
+ # transform mesh back to OPENGL coordinate system for saving
+ mesh.vertices = (mesh.vertices.astype(np.float32) @ R_P3D_TO_OPENGL).astype(np.float32)
+ mesh.export(mesh_path)
+ if layout is None:
+ layout = {"translation": [[0.0, 0.0, 0.0]], "rotation": [[1.0, 0.0, 0.0, 0.0]], "scale": [[1.0, 1.0, 1.0]]}
+ with (layout_dir / f"{name}.json").open("w", encoding="utf-8") as f:
+ json.dump(layout, f, indent=2)
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Generate simple wall/floor/ceiling meshes from layout and objects.json")
+ parser.add_argument("--layout_dir", type=Path, default=Path("outputs/layout_params"))
+ parser.add_argument("--unposed_dir", type=Path, default=Path("outputs/unposed_objects"))
+ parser.add_argument("--objects_json", type=Path, default=Path("outputs/objects.json"))
+ parser.add_argument(
+ "--material_dir",
+ type=Path,
+ default=Path("data/materials/Marble001"),
+ help="Legacy fallback material used when a surface-specific material is not supplied.",
+ )
+ parser.add_argument("--wall_material_dir", type=Path, default=None, help="PBR material directory for walls.")
+ parser.add_argument("--floor_material_dir", type=Path, default=None, help="PBR material directory for the floor.")
+ parser.add_argument("--ceiling_material_dir", type=Path, default=None, help="PBR material directory for the ceiling.")
+ parser.add_argument("--thickness", type=float, default=0.02)
+ parser.add_argument("--wall_padding", type=float, default=0.05, help="Padding distance to push walls outward")
+ parser.add_argument(
+ "--resolution",
+ "-r",
+ type=str,
+ default=env_texture_resolution(),
+ choices=list(TEXTURE_RESOLUTIONS),
+ help=(
+ "Material texture resolution (default: "
+ f"$SCENEMOSAIC_MATERIAL_RESOLUTION or {DEFAULT_TEXTURE_RESOLUTION})"
+ ),
+ )
+ parser.add_argument(
+ "--format",
+ "-f",
+ type=str,
+ default=env_texture_format(),
+ choices=list(TEXTURE_FORMATS),
+ help=(
+ "Material texture file format (default: "
+ f"$SCENEMOSAIC_MATERIAL_FORMAT or {DEFAULT_TEXTURE_FORMAT})"
+ ),
+ )
+ args = parser.parse_args()
+
+ posed = load_posed_meshes(args.layout_dir, args.unposed_dir)
+ floor_y, ceiling_y = find_floor_ceiling_heights(posed, args.objects_json)
+ wall_material = load_room_material(
+ args.wall_material_dir or args.material_dir,
+ resolution=args.resolution,
+ file_format=args.format,
+ )
+ floor_material = load_room_material(
+ args.floor_material_dir or args.material_dir,
+ resolution=args.resolution,
+ file_format=args.format,
+ )
+ ceiling_material = load_room_material(
+ args.ceiling_material_dir or args.material_dir,
+ resolution=args.resolution,
+ file_format=args.format,
+ )
+
+ planes = compute_wall_planes_from_objects(args.objects_json)
+ chosen = choose_wall_plane_extent(planes, posed, padding=args.wall_padding)
+ corners = build_wall_corners(chosen)
+ if len(corners) == 0:
+ raise RuntimeError("No wall corners computed; check objects.json walls entries.")
+
+ corner_points = np.array([corner["point"] for corner in corners], dtype=np.float32)
+
+ # Create walls
+ N = corner_points.shape[0]
+ for i in range(N):
+ a = corner_points[(i - 1) % N]
+ b = corner_points[i]
+ edge_xz = b - a
+ wall_length = float(np.linalg.norm(edge_xz))
+ if wall_length <= 1e-8:
+ wall_length = 1.0
+ wall_height = float(abs(ceiling_y - floor_y))
+ yaw = float(np.arctan2(edge_xz[1], edge_xz[0]))
+ print(f"Wall {corners[i]['wall_id']}: a={a.tolist()[:2]}, b={b.tolist()[:2]}, yaw={np.degrees(yaw):.1f} deg")
+ wall_mesh, wall_uv = create_wall_mesh(wall_length, wall_height, thickness=args.thickness)
+ wall_mesh = make_textured_mesh(wall_mesh, wall_uv, wall_material)
+ wall_center = np.array([
+ 0.5 * (a[0] + b[0]),
+ 0.5 * (floor_y + ceiling_y),
+ 0.5 * (a[1] + b[1]),
+ ], dtype=np.float32)
+ wall_layout = {
+ "translation": [wall_center.tolist()],
+ "rotation": [yaw_to_quaternion(yaw)],
+ "scale": [[1.0, 1.0, 1.0]],
+ }
+ name = corners[i].get("wall_id")
+ if not name:
+ raise RuntimeError(f"Missing wall_id for corner {i}")
+ save_mesh_and_layout(wall_mesh, name, args.unposed_dir, args.layout_dir, layout=wall_layout)
+
+ # Floor and ceiling
+ edge0 = corner_points[0] - corner_points[-1]
+ room_yaw = float(np.arctan2(edge0[1], edge0[0]))
+ corners_local = rotate_points_xz(corner_points, -room_yaw)
+ floor_mesh, floor_uv = create_floor_or_ceiling_mesh(corners_local, 0.0)
+ floor_mesh = make_textured_mesh(floor_mesh, floor_uv, floor_material)
+ floor_center = recenter_mesh(floor_mesh)
+ floor_center_xz = rotate_points_xz(floor_center[[0, 2]], room_yaw)
+ floor_layout = {
+ "translation": [[float(floor_center_xz[0]), float(floor_y + floor_center[1]), float(floor_center_xz[1])]],
+ "rotation": [yaw_to_quaternion(room_yaw)],
+ "scale": [[1.0, 1.0, 1.0]],
+ }
+ save_mesh_and_layout(floor_mesh, "floor", args.unposed_dir, args.layout_dir, layout=floor_layout)
+
+ ceiling_mesh, ceiling_uv = create_floor_or_ceiling_mesh(corners_local, 0.0)
+ ceiling_mesh = make_textured_mesh(ceiling_mesh, ceiling_uv, ceiling_material)
+ ceiling_center = recenter_mesh(ceiling_mesh)
+ ceiling_center_xz = rotate_points_xz(ceiling_center[[0, 2]], room_yaw)
+ ceiling_layout = {
+ "translation": [[float(ceiling_center_xz[0]), float(ceiling_y + ceiling_center[1]), float(ceiling_center_xz[1])]],
+ "rotation": [yaw_to_quaternion(room_yaw)],
+ "scale": [[1.0, 1.0, 1.0]],
+ }
+ save_mesh_and_layout(ceiling_mesh, "ceiling", args.unposed_dir, args.layout_dir, layout=ceiling_layout)
+
+ print(f"Generated {N} walls, floor and ceiling to {args.unposed_dir} and {args.layout_dir}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/evolution/tools/render_scene_from_layout.py b/vlm_agent/evolution/tools/render_scene_from_layout.py
new file mode 100644
index 0000000..1333f7e
--- /dev/null
+++ b/vlm_agent/evolution/tools/render_scene_from_layout.py
@@ -0,0 +1,1433 @@
+import argparse
+import math
+import json
+import os
+from copy import deepcopy
+from pathlib import Path
+import sys
+sys.path.append(str(Path(__file__).parent.parent))
+
+os.environ.setdefault("PYOPENGL_PLATFORM", "egl")
+
+import cv2
+import numpy as np
+import pyrender
+import trimesh
+import torch
+from pytorch3d.transforms import matrix_to_quaternion, quaternion_to_matrix
+
+os.environ["CUDA_HOME"] = os.environ.get("CONDA_PREFIX", os.environ.get("CUDA_HOME", ""))
+os.environ["LIDRA_SKIP_INIT"] = "true"
+
+from sam3d_objects.data.dataset.tdfy.transforms_3d import compose_transform
+
+from simulate_scene_physics import (
+ load_layout,
+ normalize_object_id
+)
+
+R_OPENGL_TO_P3D = np.array([[-1, 0, 0], [0, 1, 0], [0, 0, -1]], dtype=np.float32)
+R_P3D_TO_OPENGL = R_OPENGL_TO_P3D
+
+
+def load_object_ids(layout_dir, unposed_dir, requested_object_ids=None):
+ layout_ids = {p.stem for p in layout_dir.glob("*.json")}
+ mesh_ids = {p.stem for p in unposed_dir.glob("*.glb")}
+ matched_ids = sorted(layout_ids & mesh_ids)
+
+ normalized_requested_ids = resolve_requested_object_ids(requested_object_ids)
+ if normalized_requested_ids is None:
+ return matched_ids
+
+ missing_ids = [object_id for object_id in normalized_requested_ids if object_id not in matched_ids]
+ if missing_ids:
+ normalized_requested_ids = [obj_id for obj_id in normalized_requested_ids if obj_id not in missing_ids]
+ if not normalized_requested_ids:
+ raise FileNotFoundError(
+ f"Requested object ids not found in both layout and unposed directories: {missing_ids}. Matched ids: {matched_ids}"
+ )
+ else:
+ print(f"Warning: Requested object ids not found in both layout and unposed directories: {missing_ids}. Matched ids: {matched_ids}")
+ return normalized_requested_ids
+
+
+def resolve_requested_object_ids(requested_object_ids):
+ if requested_object_ids is None:
+ return None
+ if isinstance(requested_object_ids, (str, int)):
+ requested_object_ids = [requested_object_ids]
+ return [normalize_object_id(object_id) for object_id in requested_object_ids]
+
+
+def parse_camera_direction(value):
+ direction = np.asarray(value, dtype=np.float32).reshape(-1)
+ if direction.size != 3:
+ raise ValueError(f"Camera direction must have 3 elements, got shape {direction.shape}.")
+ norm = np.linalg.norm(direction)
+ if norm <= 1e-8:
+ raise ValueError("Camera direction norm is zero.")
+ return direction / norm
+
+
+def parse_scale(scale_value):
+ scale = np.asarray(scale_value, dtype=np.float32).reshape(-1)
+ if scale.size == 1:
+ return np.repeat(scale, 3)
+ if scale.size == 3:
+ return scale
+ raise ValueError(f"Scale must be scalar or 3D vector, got shape {scale.shape}.")
+
+
+def apply_layout_to_unposed_mesh(mesh, layout):
+ """Apply layout transform to an unposed mesh using the demo.py pipeline logic."""
+ translation = torch.tensor(layout["translation"], dtype=torch.float32)
+ rotation_q = torch.tensor(layout["rotation"], dtype=torch.float32)
+ scale = torch.tensor(layout["scale"], dtype=torch.float32)
+
+ vertices = mesh.vertices.astype(np.float32) @ R_OPENGL_TO_P3D
+ vertices_tensor = torch.from_numpy(vertices).float().to(rotation_q.device)
+ rotation_m = quaternion_to_matrix(rotation_q).transpose(-1, -2)
+ l2c_transform = compose_transform(
+ scale=scale,
+ rotation=rotation_m,
+ translation=translation,
+ )
+ vertices = l2c_transform.transform_points(vertices_tensor.unsqueeze(0))
+ mesh.vertices = vertices.squeeze(0).cpu().numpy()
+ # mesh.vertices = vertices.squeeze(0).cpu().numpy() @ R_P3D_TO_OPENGL
+ return mesh
+
+
+def compose_scene(layout_dir, unposed_dir, requested_object_ids=None, anchor_id=None):
+ object_ids = load_object_ids(layout_dir, unposed_dir, requested_object_ids=requested_object_ids)
+ if not object_ids:
+ raise FileNotFoundError(
+ f"No matching object files found in {layout_dir} and {unposed_dir}."
+ )
+
+ posed_meshes = []
+ layout_params = []
+ for object_id in object_ids:
+ layout_path = layout_dir / f"{object_id}.json"
+ mesh_path = unposed_dir / f"{object_id}.glb"
+
+ layout, _, _, _ = load_layout(layout_path)
+ mesh = trimesh.load(mesh_path, force="mesh")
+ posed_mesh = apply_layout_to_unposed_mesh(mesh, layout)
+ posed_meshes.append(posed_mesh)
+ layout_params.append(deepcopy(layout))
+
+ if len(posed_meshes) == 1:
+ scene_mesh = posed_meshes[0]
+ else:
+ scene_mesh = trimesh.Scene()
+ for idx, posed_mesh in enumerate(posed_meshes):
+ scene_mesh.add_geometry(posed_mesh, node_name=f"object_{idx:03d}")
+
+ scene_mesh, posed_meshes, layout_params, transform_info = normalize_scene_to_canonical_cube_with_info(
+ scene_mesh,
+ posed_meshes,
+ layout_params,
+ cube_size=1.0,
+ in_place=False,
+ anchor_id=anchor_id,
+ object_ids=object_ids,
+ )
+ return scene_mesh, posed_meshes, layout_params, object_ids, transform_info
+
+
+def normalize_scene_to_canonical_cube_with_info(
+ scene_mesh,
+ posed_meshes,
+ layout_params,
+ cube_size=1.0,
+ in_place=False,
+ anchor_id=None,
+ object_ids=None,
+):
+ """Normalize scene to canonical cube and return transform info."""
+ if scene_mesh is None:
+ return scene_mesh, posed_meshes, layout_params, {}
+
+ if not in_place:
+ scene_mesh = deepcopy(scene_mesh)
+ posed_meshes = [deepcopy(mesh) for mesh in posed_meshes]
+ layout_params = deepcopy(layout_params)
+
+ rotation_fix = None
+ if anchor_id is not None:
+ if not object_ids:
+ raise ValueError("anchor_id requires object_ids to be provided.")
+ normalized_anchor_id = normalize_object_id(anchor_id)
+ if normalized_anchor_id not in object_ids:
+ raise FileNotFoundError(
+ f"anchor_id {normalized_anchor_id} not found in object ids: {object_ids}"
+ )
+ anchor_index = object_ids.index(normalized_anchor_id)
+ anchor_rotation = layout_params[anchor_index].get("rotation")
+ if anchor_rotation is None:
+ raise ValueError(f"anchor_id {normalized_anchor_id} has no rotation in layout params.")
+
+ anchor_rotation = np.asarray(anchor_rotation, dtype=np.float32).reshape(-1)
+ if anchor_rotation.size != 4:
+ raise ValueError(
+ f"anchor_id {normalized_anchor_id} rotation must have 4 elements, got shape {anchor_rotation.shape}."
+ )
+
+ # Convert to row-major order for correct application to vertices and translations.
+ anchor_rot = (
+ quaternion_to_matrix(torch.from_numpy(anchor_rotation))
+ .transpose(-1, -2)
+ .cpu()
+ .numpy()
+ )
+ rotation_fix = anchor_rot.T.astype(np.float32)
+
+ for mesh in posed_meshes:
+ if mesh.vertices.size:
+ mesh.vertices = mesh.vertices.astype(np.float32) @ rotation_fix
+
+ if isinstance(scene_mesh, trimesh.Trimesh):
+ scene_mesh.vertices = scene_mesh.vertices.astype(np.float32) @ rotation_fix
+
+ rotation_fix_torch = torch.from_numpy(rotation_fix).float()
+ for params in layout_params:
+ if "translation" in params:
+ translation = np.asarray(params["translation"], dtype=np.float32).reshape(3)
+ params["translation"] = (translation @ rotation_fix).tolist()
+ if "rotation" in params:
+ rot_q = np.asarray(params["rotation"], dtype=np.float32).reshape(-1)
+ if rot_q.size != 4:
+ raise ValueError(
+ f"rotation must have 4 elements, got shape {rot_q.shape}."
+ )
+ rot_mat = quaternion_to_matrix(torch.from_numpy(rot_q)).transpose(-1, -2)
+ rot_mat = torch.matmul(rot_mat, rotation_fix_torch)
+ rot_q = matrix_to_quaternion(rot_mat.transpose(-1, -2))
+ params["rotation"] = rot_q.cpu().numpy().reshape(-1).tolist()
+
+ all_vertices_p3d = []
+ for mesh in posed_meshes:
+ # all_vertices_p3d.append(mesh.vertices.astype(np.float32) @ R_OPENGL_TO_P3D)
+ all_vertices_p3d.append(mesh.vertices.astype(np.float32))
+ if not all_vertices_p3d:
+ return scene_mesh, posed_meshes, layout_params, {}
+
+ scene_vertices_p3d = np.vstack(all_vertices_p3d)
+ bbox_min = scene_vertices_p3d.min(axis=0)
+ bbox_max = scene_vertices_p3d.max(axis=0)
+ center = (bbox_min + bbox_max) / 2.0
+ extent = bbox_max - bbox_min
+ max_extent = float(np.max(extent))
+ if max_extent <= 0:
+ return scene_mesh, posed_meshes, layout_params, {}
+
+ normalize_scale = float(cube_size) / max_extent
+
+ for mesh in posed_meshes:
+ # vertices_p3d = mesh.vertices.astype(np.float32) @ R_OPENGL_TO_P3D
+ # mesh.vertices = ((vertices_p3d - center) * normalize_scale) @ R_P3D_TO_OPENGL
+ vertices_p3d = mesh.vertices.astype(np.float32)
+ mesh.vertices = ((vertices_p3d - center) * normalize_scale)
+
+ if isinstance(scene_mesh, trimesh.Trimesh):
+ # scene_mesh.vertices = ((scene_vertices_p3d - center) * normalize_scale) @ R_P3D_TO_OPENGL
+ scene_mesh.vertices = ((scene_vertices_p3d - center) * normalize_scale)
+
+ for params in layout_params:
+ translation = np.asarray(params["translation"], dtype=np.float32)
+ scale = np.asarray(params["scale"], dtype=np.float32)
+ params["translation"] = ((translation - center) * normalize_scale).tolist()
+ params["scale"] = (scale * normalize_scale).tolist()
+
+ transform_info = {
+ "center": center.tolist(),
+ "normalize_scale": float(normalize_scale),
+ "bbox_min": bbox_min.tolist(),
+ "bbox_max": bbox_max.tolist(),
+ "max_extent": float(max_extent),
+ }
+ if rotation_fix is not None:
+ transform_info["rotation_matrix"] = rotation_fix.tolist()
+ transform_info["anchor_id"] = normalize_object_id(anchor_id)
+ return scene_mesh, posed_meshes, layout_params, transform_info
+
+
+def denormalize_point(point_normalized, transform_info):
+ """Convert a normalized point back to original 3D coordinates."""
+ center = np.asarray(transform_info["center"], dtype=np.float32)
+ normalize_scale = transform_info["normalize_scale"]
+ point = np.asarray(point_normalized, dtype=np.float32).reshape(3)
+ # Reverse the normalization: point_original = (point_normalized / scale + center)
+ original_point = (point / normalize_scale) + center
+ rotation = transform_info.get("rotation_matrix")
+ if rotation is not None:
+ rotation = np.asarray(rotation, dtype=np.float32).reshape(3, 3)
+ original_point = original_point @ rotation.T
+ return original_point
+
+
+def normalize_point(point_original, transform_info):
+ """Convert an original point to the normalized canonical-cube coordinates."""
+ center = np.asarray(transform_info["center"], dtype=np.float32)
+ normalize_scale = float(transform_info["normalize_scale"])
+ point = np.asarray(point_original, dtype=np.float32).reshape(3)
+ rotation = transform_info.get("rotation_matrix")
+ if rotation is not None:
+ rotation = np.asarray(rotation, dtype=np.float32).reshape(3, 3)
+ point = point @ rotation
+ return (point - center) * normalize_scale
+
+
+def estimate_grid_axes_from_mapping(mapping, grid_size):
+ """Estimate grid right/up axes from grid mapping points."""
+ axis_right = None
+ axis_up = None
+ for i in range(grid_size):
+ for j in range(grid_size):
+ key = (i, j)
+ if key not in mapping:
+ continue
+ if axis_right is None and (i + 1, j) in mapping:
+ axis_right = np.asarray(mapping[(i + 1, j)], dtype=np.float32) - np.asarray(
+ mapping[key], dtype=np.float32
+ )
+ if np.linalg.norm(axis_right) <= 1e-8:
+ axis_right = None
+ else:
+ axis_right = axis_right / np.linalg.norm(axis_right)
+ if axis_up is None and (i, j + 1) in mapping:
+ axis_up = np.asarray(mapping[(i, j + 1)], dtype=np.float32) - np.asarray(
+ mapping[key], dtype=np.float32
+ )
+ if np.linalg.norm(axis_up) <= 1e-8:
+ axis_up = None
+ else:
+ axis_up = axis_up / np.linalg.norm(axis_up)
+ if axis_right is not None and axis_up is not None:
+ return axis_right, axis_up
+ return axis_right, axis_up
+
+
+def load_grid_mapping(mapping_path, transform_info):
+ """Load a grid mapping JSON and convert points into current normalized space."""
+ with open(mapping_path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+
+ grid_points = data.get("grid_points", [])
+ if not grid_points:
+ raise ValueError(f"Grid mapping file has no grid_points: {mapping_path}")
+
+ mapping = {}
+ max_i = -1
+ max_j = -1
+ for entry in grid_points:
+ if "grid_i" not in entry or "grid_j" not in entry:
+ raise ValueError("Grid mapping entry is missing grid_i/grid_j.")
+ grid_i = int(entry["grid_i"])
+ grid_j = int(entry["grid_j"])
+ if "point_3d_original" in entry:
+ point_normalized = normalize_point(entry["point_3d_original"], transform_info)
+ elif "point_3d_normalized" in entry:
+ point_normalized = np.asarray(entry["point_3d_normalized"], dtype=np.float32).reshape(3)
+ else:
+ raise ValueError("Grid mapping entry is missing point_3d_original/point_3d_normalized.")
+
+ mapping[(grid_i, grid_j)] = point_normalized.tolist()
+ max_i = max(max_i, grid_i)
+ max_j = max(max_j, grid_j)
+
+ grid_size = int(data.get("grid_size", max(max_i, max_j) + 1))
+ axis_right, axis_up = estimate_grid_axes_from_mapping(mapping, grid_size)
+ if axis_right is not None:
+ axis_right = -np.asarray(axis_right, dtype=np.float32)
+ return mapping, grid_size, axis_right, axis_up
+
+
+def get_plane_from_camera(camera_direction, posed_meshes, plane_side="farthest"):
+ """
+ Find the plane perpendicular to camera direction.
+ Use the farthest or nearest point from camera along the camera direction.
+ Returns (plane_normal, plane_d) in the form: normal·p + d = 0
+ """
+ camera_dir = parse_camera_direction(camera_direction)
+
+ # Find the farthest point along camera direction
+ all_vertices = []
+ for mesh in posed_meshes:
+ all_vertices.append(mesh.vertices.astype(np.float32))
+
+ if all_vertices:
+ all_vertices = np.vstack(all_vertices)
+ distances_along_dir = np.dot(all_vertices, camera_dir)
+ if plane_side == "nearest":
+ plane_distance = np.min(distances_along_dir)
+ else:
+ plane_distance = np.max(distances_along_dir)
+ # Plane perpendicular to camera direction at the selected extreme point
+ plane_normal = camera_dir
+ plane_d = -plane_distance
+ else:
+ plane_normal = camera_dir
+ plane_d = 0.0
+
+ return plane_normal, plane_d
+
+
+def generate_2d_grid_on_plane(posed_meshes, camera_direction, grid_size, plane_normal=None, plane_d=None, plane_side="farthest"):
+ """
+ Generate a 2D grid on the plane perpendicular to camera direction.
+ Returns: grid_2d, mapping (dict), plane_center, local_x, local_y
+
+ grid_2d: (grid_size, grid_size, 2) array of 2D coordinates
+ mapping: dict of {(i, j): original_3d_coords}
+ """
+ if plane_normal is None or plane_d is None:
+ plane_normal, plane_d = get_plane_from_camera(camera_direction, posed_meshes, plane_side=plane_side)
+
+ camera_dir = parse_camera_direction(camera_direction)
+ plane_normal = plane_normal / np.linalg.norm(plane_normal)
+
+ # Find scene center and bounds
+ all_vertices = []
+ for mesh in posed_meshes:
+ all_vertices.append(mesh.vertices.astype(np.float32))
+
+ if not all_vertices:
+ return None, {}
+
+ all_vertices = np.vstack(all_vertices)
+
+ # Create local coordinate system on the plane
+ # plane normal is the "forward" direction
+ up_hint = np.array([0.0, 1.0, 0.0], dtype=np.float32)
+ right = np.cross(up_hint, plane_normal)
+ if np.linalg.norm(right) <= 1e-8:
+ up_hint = np.array([0.0, 0.0, 1.0], dtype=np.float32)
+ right = np.cross(up_hint, plane_normal)
+ right = right / np.linalg.norm(right)
+ up = np.cross(plane_normal, right)
+ up = up / np.linalg.norm(up)
+
+ # Project all vertices onto the plane and get bounds
+ projected_on_right = np.dot(all_vertices, right)
+ projected_on_up = np.dot(all_vertices, up)
+
+ min_right = np.min(projected_on_right)
+ max_right = np.max(projected_on_right)
+ min_up = np.min(projected_on_up)
+ max_up = np.max(projected_on_up)
+
+ # Grid origin at the center of the projection bounds
+ center_right = (min_right + max_right) / 2.0
+ center_up = (min_up + max_up) / 2.0
+
+ # Find a point on the plane passing through scene center
+ scene_center = np.mean(all_vertices, axis=0)
+ # Point on plane closest to scene center
+ dist_to_plane = np.dot(scene_center, plane_normal) + plane_d
+ plane_center = scene_center - dist_to_plane * plane_normal
+
+ # Adjust plane center to use the projection center
+ plane_center = plane_center + (center_right - np.dot(plane_center, right)) * right
+ plane_center = plane_center + (center_up - np.dot(plane_center, up)) * up
+
+ # Generate grid
+ grid_2d = np.zeros((grid_size, grid_size, 2), dtype=np.float32)
+ mapping = {}
+
+ extent_right = max_right - min_right
+ extent_up = max_up - min_up
+
+ # Add padding to grid
+ padding = 0.1
+ grid_min_right = min_right - padding * extent_right
+ grid_max_right = max_right + padding * extent_right
+ grid_min_up = min_up - padding * extent_up
+ grid_max_up = max_up + padding * extent_up
+
+ for i in range(grid_size):
+ for j in range(grid_size):
+ # Normalized grid coordinates [0, 1]
+ u = i / max(grid_size - 1, 1)
+ v = j / max(grid_size - 1, 1)
+
+ # Flip the horizontal index so (0, 0) appears at the lower-left in the rendered view.
+ display_i = grid_size - 1 - i
+
+ # Grid 2D coordinates
+ grid_r = grid_min_right + u * (grid_max_right - grid_min_right)
+ grid_u = grid_min_up + v * (grid_max_up - grid_min_up)
+
+ grid_2d[display_i, j] = [grid_r, grid_u]
+
+ # 3D point on plane
+ point_3d = plane_center + grid_r * right + grid_u * up
+ mapping[(display_i, j)] = point_3d.tolist()
+
+ return grid_2d, mapping, plane_center, right, up
+
+
+def get_camera_basis(camera_direction):
+ """Return forward, right, and up axes for the given camera direction."""
+ forward = parse_camera_direction(camera_direction)
+ up_hint = np.array([0.0, 1.0, 0.0], dtype=np.float32)
+ right = np.cross(forward, up_hint)
+
+ if np.linalg.norm(right) <= 1e-8:
+ up_hint = np.array([0.0, 0.0, 1.0], dtype=np.float32)
+ right = np.cross(forward, up_hint)
+
+ right = right / np.linalg.norm(right)
+ up = np.cross(right, forward)
+ up = up / np.linalg.norm(up)
+ return forward, right, up
+
+
+def get_camera_axes(camera_direction, axis_right=None, axis_up=None):
+ """Resolve orthonormal axes with camera_direction as the last axis."""
+ if axis_right is not None and axis_up is not None:
+ right = np.asarray(axis_right, dtype=np.float32).reshape(3)
+ up = np.asarray(axis_up, dtype=np.float32).reshape(3)
+ right_norm = np.linalg.norm(right)
+ up_norm = np.linalg.norm(up)
+ if right_norm > 1e-8:
+ right = right / right_norm
+ if up_norm > 1e-8:
+ up = up / up_norm
+ forward = np.cross(right, up)
+ forward_norm = np.linalg.norm(forward)
+ if forward_norm > 1e-8:
+ forward = forward / forward_norm
+ else:
+ forward, right, up = get_camera_basis(camera_direction)
+ else:
+ forward, right, up = get_camera_basis(camera_direction)
+ return right, up, forward
+
+
+def quaternion_to_matrix_np(quat):
+ quat = np.asarray(quat, dtype=np.float32).reshape(4)
+ norm = np.linalg.norm(quat)
+ if norm > 1e-8:
+ quat = quat / norm
+ w, x, y, z = quat
+ return np.array(
+ [
+ [1 - 2 * (y * y + z * z), 2 * (x * y - z * w), 2 * (x * z + y * w)],
+ [2 * (x * y + z * w), 1 - 2 * (x * x + z * z), 2 * (y * z - x * w)],
+ [2 * (x * z - y * w), 2 * (y * z + x * w), 1 - 2 * (x * x + y * y)],
+ ],
+ dtype=np.float32,
+ )
+
+
+def euler_from_matrix_xyz(matrix):
+ """Return XYZ Euler angles from rotation matrix (R = Rz @ Ry @ Rx)."""
+ r00, r10, r20 = matrix[0, 0], matrix[1, 0], matrix[2, 0]
+ r21, r22 = matrix[2, 1], matrix[2, 2]
+ # Check for Gimbal Lock: sin(y) around -1 or 1
+ if r20 < -0.995:
+ x = 0.0
+ y = math.pi / 2
+ z = math.atan2(matrix[0, 1], matrix[1, 1])
+ elif r20 > 0.995:
+ x = 0.0
+ y = -math.pi / 2
+ z = -math.atan2(matrix[0, 1], matrix[1, 1])
+ else:
+ # Standard non-locked case
+ x = math.atan2(r21, r22)
+ y = math.asin(-r20)
+ z = math.atan2(r10, r00)
+ return x, y, z
+
+
+def extract_layout_quaternion(layout):
+ rotation = layout.get("rotation")
+ if rotation is None:
+ raise ValueError("Layout rotation is missing.")
+ rotation = np.asarray(rotation, dtype=np.float32).reshape(-1)
+ if rotation.size == 4:
+ return rotation
+ if rotation.size >= 4:
+ return rotation[:4]
+ raise ValueError(f"Rotation must have 4 elements, got shape {rotation.shape}.")
+
+
+def resolve_orthographic_camera_frame(meshes, camera_direction, radius, yfov_deg=60.0, padding=0.35):
+ """Resolve orthographic camera distance and half-extent, auto-fitting when radius is 0."""
+ if radius is not None and float(radius) > 0:
+ yfov_rad = np.deg2rad(yfov_deg)
+ ymag = float(radius) * np.tan(yfov_rad / 2.0)
+ return float(radius), ymag, ymag
+
+ all_vertices = []
+ for mesh in meshes:
+ all_vertices.append(mesh.vertices.astype(np.float32))
+
+ if not all_vertices:
+ raise ValueError("Cannot auto-fit camera frame without any mesh vertices.")
+
+ vertices = np.vstack(all_vertices)
+ forward, right, up = get_camera_basis(camera_direction)
+
+ projected_right = np.dot(vertices, right)
+ projected_up = np.dot(vertices, up)
+ projected_forward = np.dot(vertices, forward)
+
+ half_width = float((np.max(projected_right) - np.min(projected_right)) / 2.0)
+ half_height = float((np.max(projected_up) - np.min(projected_up)) / 2.0)
+ half_depth = float((np.max(projected_forward) - np.min(projected_forward)) / 2.0)
+
+ half_extent = max(half_width, half_height)
+ if half_extent <= 0:
+ half_extent = 1e-6
+
+ xmag = half_extent * (1.0 + padding)
+ ymag = xmag
+ camera_radius = max(half_depth, half_extent) * (1.0 + padding)
+ camera_radius = max(camera_radius, half_depth + 1e-6)
+ return camera_radius, xmag, ymag
+
+
+def compute_grid_bounds_from_mapping(mapping, axis_right, axis_up):
+ """Compute grid bounds in the plane using the saved grid points mapping."""
+ if not mapping:
+ raise ValueError("Grid mapping is empty; cannot compute grid bounds.")
+
+ points = np.asarray(list(mapping.values()), dtype=np.float32)
+ projected_right = np.dot(points, axis_right)
+ projected_up = np.dot(points, axis_up)
+
+ min_right = float(np.min(projected_right))
+ max_right = float(np.max(projected_right))
+ min_up = float(np.min(projected_up))
+ max_up = float(np.max(projected_up))
+ grid_size = max(int(np.sqrt(len(mapping))), 1)
+ return grid_size, min_right, max_right, min_up, max_up
+
+
+def compute_object_2d_stats(mesh, axis_right, axis_up, mapping, rotation_degrees=None, use_aabb_for_object_bboxes=False):
+ """Compute 2D center/extent/bbox in grid index coordinates using grid mapping."""
+ grid_size, min_right, max_right, min_up, max_up = compute_grid_bounds_from_mapping(
+ mapping, axis_right, axis_up
+ )
+ vertices = mesh.vertices.astype(np.float32)
+ projected_right = np.dot(vertices, axis_right)
+ projected_up = np.dot(vertices, axis_up)
+
+ denom_right = max(max_right - min_right, 1e-8)
+ denom_up = max(max_up - min_up, 1e-8)
+ scale = float(max(grid_size - 1, 0))
+
+ grid_i = (max_right - projected_right) / denom_right * scale
+ grid_j = (projected_up - min_up) / denom_up * scale
+
+ min_i = float(np.min(grid_i))
+ max_i = float(np.max(grid_i))
+ min_j = float(np.min(grid_j))
+ max_j = float(np.max(grid_j))
+
+ center_i = (min_i + max_i) / 2.0
+ center_j = (min_j + max_j) / 2.0
+ size_i = max_i - min_i
+ size_j = max_j - min_j
+
+ stats = {
+ "center_2d": [center_i, center_j],
+ "size_2d": [size_i, size_j],
+ "bbox_2d": [min_i, min_j, max_i, max_j],
+ }
+
+
+ if use_aabb_for_object_bboxes:
+ stats["obb_corners_2d"] = [
+ [min_i, min_j],
+ [max_i, min_j],
+ [max_i, max_j],
+ [min_i, max_j],
+ ]
+ elif rotation_degrees is not None:
+ theta = np.deg2rad(float(rotation_degrees))
+
+ # 1. Compute object vertices in physical space relative to center
+ verts_phys = np.stack([projected_right, projected_up], axis=1).astype(np.float32)
+ center_phys = np.array([(np.min(projected_right) + np.max(projected_right)) / 2.0,
+ (np.min(projected_up) + np.max(projected_up)) / 2.0], dtype=np.float32)
+
+ rel_phys = verts_phys - center_phys
+
+ # 2. Compute heading and perpendicular vectors in physical space
+ heading = np.array([np.sin(theta), -np.cos(theta)], dtype=np.float32)
+ perp = np.array([-np.cos(theta), -np.sin(theta)], dtype=np.float32)
+
+ heading_proj = rel_phys @ heading
+ perp_proj = rel_phys @ perp
+
+ min_h, max_h = np.min(heading_proj), np.max(heading_proj)
+ min_p, max_p = np.min(perp_proj), np.max(perp_proj)
+
+ # 3. Compute OBB corners in physical space
+ corners_phys = [
+ center_phys + min_h * heading + min_p * perp,
+ center_phys + max_h * heading + min_p * perp,
+ center_phys + max_h * heading + max_p * perp,
+ center_phys + min_h * heading + max_p * perp,
+ ]
+
+ # 4. Convert OBB corners back to grid coordinates
+ obb_corners_2d = []
+ for pt in corners_phys:
+ pt_i = (max_right - pt[0]) / denom_right * scale
+ pt_j = (pt[1] - min_up) / denom_up * scale
+ obb_corners_2d.append([float(pt_i), float(pt_j)])
+
+ stats["obb_corners_2d"] = obb_corners_2d
+
+ return stats
+
+
+def estimate_arrow_length_to_bbox(bbox_2d):
+ """Estimate arrow length in pixels until it hits the bbox boundary."""
+ if bbox_2d is not None and len(bbox_2d) >= 2:
+ pts = np.asarray(bbox_2d, dtype=np.float32)
+ if pts.ndim >= 2 and len(pts) >= 2:
+ return float(np.linalg.norm(pts[1] - pts[0]))
+ return None
+
+
+def build_objects_info(
+ posed_meshes,
+ object_ids,
+ layout_params,
+ axis_right,
+ axis_up,
+ mapping,
+ camera_direction=None,
+ plane_normal=None,
+ plane_d=None,
+ transform_info=None,
+ draw_object_bboxes=False,
+ use_aabb_for_object_bboxes=False,
+):
+ """Build per-object 2D stats and optional orientation/plane distance info."""
+ objects_info = []
+
+ if camera_direction is not None:
+ basis_right, basis_up, basis_forward = get_camera_axes(
+ camera_direction, axis_right=axis_right, axis_up=axis_up
+ )
+ basis_matrix = np.stack([basis_right, basis_up, basis_forward], axis=1)
+ else:
+ basis_matrix = None
+
+ for mesh, object_id, layout in zip(posed_meshes, object_ids, layout_params):
+ rotation_degrees = None
+ if basis_matrix is not None:
+ try:
+ rotation_q = extract_layout_quaternion(layout)
+ rotation_world = quaternion_to_matrix_np(rotation_q)
+ rotation_local = basis_matrix.T @ rotation_world @ basis_matrix
+ x, y, angle_z = euler_from_matrix_xyz(rotation_local)
+ angle_deg = math.degrees(angle_z)
+ rotation_degrees = (angle_deg + 180.0) % 360.0 - 180.0
+ except ValueError:
+ rotation_degrees = 0.0
+
+ if draw_object_bboxes and rotation_degrees is not None:
+ stats = compute_object_2d_stats(
+ mesh,
+ axis_right,
+ axis_up,
+ mapping,
+ rotation_degrees=rotation_degrees,
+ use_aabb_for_object_bboxes=use_aabb_for_object_bboxes
+ )
+ else:
+ stats = compute_object_2d_stats(mesh, axis_right, axis_up, mapping)
+
+ obj_info = {
+ "object_id": object_id,
+ **stats,
+ }
+
+ if rotation_degrees is not None:
+ obj_info["rotation_degrees"] = float(rotation_degrees)
+
+ if (
+ plane_normal is not None
+ and plane_d is not None
+ and transform_info is not None
+ and "translation" in layout
+ ):
+ center = np.asarray(layout["translation"], dtype=np.float32).reshape(-1)
+ if center.size >= 3:
+ center = denormalize_point(center[:3], transform_info)
+ signed_distance = float(np.dot(plane_normal, center) + plane_d)
+ obj_info["signed_distance_to_plane"] = signed_distance
+
+ objects_info.append(obj_info)
+
+ return objects_info
+
+
+def save_grid_mapping(mapping, output_path, transform_info, posed_meshes=None, object_ids=None,
+ axis_right=None, axis_up=None, camera_direction=None, layout_params=None,
+ objects_info=None, use_aabb_for_object_bboxes=False):
+ """Save the 2D grid to 3D coordinate mapping."""
+ # Convert mapping to a format that's easy to serialize
+ # Include both normalized and original (denormalized) coordinates
+
+ # Keep column-major rotation matrix for correct application to vertices and translations in other tools.
+ rotation_matrix = np.asarray(
+ transform_info.get("rotation_matrix", np.eye(3)), dtype=np.float32
+ ).reshape(3, 3)
+ transform_info["rotation_matrix_col_major"] = rotation_matrix.T.tolist()
+ mapping_data = {
+ "grid_size": int(np.sqrt(len(mapping))),
+ "grid_points": [],
+ "transform_info": transform_info,
+ }
+
+ for (i, j), point_3d_normalized in mapping.items():
+ point_original = denormalize_point(point_3d_normalized, transform_info)
+ mapping_data["grid_points"].append({
+ "grid_i": int(i),
+ "grid_j": int(j),
+ "point_3d_normalized": point_3d_normalized,
+ "point_3d_original": point_original.tolist(),
+ })
+
+ plane_normal = None
+ plane_d = None
+ if axis_right is not None and axis_up is not None and mapping:
+ plane_normal = np.cross(axis_right, axis_up)
+ norm = np.linalg.norm(plane_normal)
+ if norm > 1e-8:
+ plane_normal = plane_normal / norm
+ rotation = transform_info.get("rotation_matrix")
+ if rotation is not None:
+ rotation = np.asarray(rotation, dtype=np.float32).reshape(3, 3)
+ plane_normal = plane_normal @ rotation.T
+ norm = np.linalg.norm(plane_normal)
+ if norm > 1e-8:
+ plane_normal = plane_normal / norm
+ sample_point = np.asarray(next(iter(mapping.values())), dtype=np.float32)
+ sample_point = denormalize_point(sample_point, transform_info)
+ plane_d = -float(np.dot(plane_normal, sample_point))
+ mapping_data["plane"] = {
+ "normal": plane_normal.tolist(),
+ "d": plane_d,
+ }
+
+ if objects_info is not None:
+ objects_info = deepcopy(objects_info)
+ for obj in objects_info:
+ obj.pop("obb_corners_2d", None)
+ mapping_data["objects"] = objects_info
+ elif posed_meshes and object_ids and axis_right is not None and axis_up is not None and layout_params:
+ if plane_normal is None and camera_direction is not None:
+ plane_normal, plane_d = get_plane_from_camera(camera_direction, posed_meshes)
+ plane_normal = plane_normal / np.linalg.norm(plane_normal)
+ sample_point = (-float(plane_d)) * plane_normal
+ sample_point = denormalize_point(sample_point, transform_info)
+ rotation = transform_info.get("rotation_matrix")
+ if rotation is not None:
+ rotation = np.asarray(rotation, dtype=np.float32).reshape(3, 3)
+ plane_normal = plane_normal @ rotation.T
+ norm = np.linalg.norm(plane_normal)
+ if norm > 1e-8:
+ plane_normal = plane_normal / norm
+ plane_d = -float(np.dot(plane_normal, sample_point))
+ mapping_data.setdefault(
+ "plane",
+ {"normal": plane_normal.tolist(), "d": float(plane_d)},
+ )
+
+ mapping_data["objects"] = build_objects_info(
+ posed_meshes=posed_meshes,
+ object_ids=object_ids,
+ layout_params=layout_params,
+ axis_right=axis_right,
+ axis_up=axis_up,
+ mapping=mapping,
+ camera_direction=camera_direction,
+ plane_normal=plane_normal,
+ plane_d=plane_d,
+ transform_info=transform_info,
+ )
+
+ with open(output_path, "w", encoding="utf-8") as f:
+ json.dump(mapping_data, f, ensure_ascii=False, indent=2)
+
+
+def render_scene_with_grid(
+ meshes,
+ camera_direction,
+ radius,
+ image_size,
+ grid_2d=None,
+ mapping=None,
+ draw_grid_lines=False,
+ draw_object_directions=False,
+ draw_object_bboxes=False,
+ object_infos=None,
+ hide_object_ids=None,
+):
+ """Render scene with optional 2D grid overlay on the image."""
+ scene = pyrender.Scene(bg_color=[0.60, 0.60, 0.60, 1.0], ambient_light=[1.0, 1.0, 1.0])
+
+ for mesh in meshes:
+ scene.add(mesh_to_pyrender_mesh(mesh))
+
+ # Use an orthographic camera that either matches the requested radius or auto-fits the scene.
+ yfov_deg = 60.0
+ camera_radius, xmag, ymag = resolve_orthographic_camera_frame(meshes, camera_direction, radius, yfov_deg=yfov_deg)
+ camera = pyrender.OrthographicCamera(xmag=xmag, ymag=ymag)
+ camera_pose = build_camera_pose(camera_direction, camera_radius)
+ scene.add(camera, pose=camera_pose)
+
+ renderer = pyrender.OffscreenRenderer(viewport_width=image_size, viewport_height=image_size)
+ try:
+ color, _ = renderer.render(scene, flags=pyrender.RenderFlags.RGBA)
+ finally:
+ renderer.delete()
+
+ if color.shape[-1] == 4:
+ alpha = color[:, :, 3:4].astype(np.float32) / 255.0
+ rgb = color[:, :, :3].astype(np.float32)
+ background = np.full_like(rgb, 235.0)
+ color = (rgb * alpha + background * (1.0 - alpha)).astype(np.uint8)
+
+ # Overlay grid if provided
+ if grid_2d is not None and mapping is not None:
+ camera_matrix = get_camera_matrix(yfov_deg, image_size, ortho=True, xmag=xmag, ymag=ymag)
+ color = overlay_2d_grid(
+ color,
+ grid_2d,
+ mapping,
+ camera_pose,
+ image_size,
+ camera_matrix=camera_matrix,
+ draw_grid_lines=draw_grid_lines,
+ draw_object_directions=draw_object_directions,
+ draw_object_bboxes=draw_object_bboxes,
+ object_infos=object_infos,
+ hide_object_ids=hide_object_ids,
+ )
+
+ return color
+
+
+def project_3d_to_2d(point_3d, camera_pose, camera_matrix, image_size):
+ """
+ Project a 3D point to 2D image coordinates.
+
+ Args:
+ point_3d: 3D point in world coordinates [x, y, z]
+ camera_pose: 4x4 camera-to-world transformation matrix
+ camera_matrix: 3x3 camera intrinsic matrix
+ image_size: size of the image (square)
+
+ Returns:
+ (x, y) in image coordinates, or None if point is behind camera
+ """
+ point_3d = np.asarray(point_3d, dtype=np.float32).reshape(3, 1)
+
+ # camera_pose is camera_T_world: camera position and axes in world frame
+ # We need world_T_camera to transform world points to camera coordinates
+ camera_pose_inv = np.linalg.inv(camera_pose.astype(np.float32))
+
+ # Transform from world to camera coordinates
+ point_camera = camera_pose_inv[:3, :3] @ point_3d + camera_pose_inv[:3, 3:4]
+
+ # Check if point is in front of camera (not behind)
+ # In OpenGL/pyrender: camera looks along -Z, so Z >= 0 means behind camera (not visible)
+ if point_camera[2, 0] >= 0:
+ return None
+
+ # Support both perspective and orthographic camera_matrix usages.
+ # If camera_matrix is a tuple (matrix, meta) where meta contains 'ortho', handle orthographic mapping.
+ ortho_meta = None
+ if isinstance(camera_matrix, tuple) or isinstance(camera_matrix, list):
+ mat = camera_matrix[0]
+ ortho_meta = camera_matrix[1]
+ camera_matrix = mat
+
+ if ortho_meta is not None and ortho_meta.get("ortho", False):
+ # Orthographic projection mapping
+ xmag = float(ortho_meta["xmag"])
+ ymag = float(ortho_meta["ymag"])
+
+ x_camera = float(point_camera[0, 0])
+ y_camera = float(point_camera[1, 0])
+
+ # Map from camera coordinates to image pixels
+ u = (x_camera / (2.0 * xmag) + 0.5) * image_size
+ v = (0.5 - y_camera / (2.0 * ymag)) * image_size
+
+ if u < 0 or u >= image_size or v < 0 or v >= image_size:
+ return None
+ return (int(round(u)), int(round(v)))
+ else:
+ # Perspective projection (existing behavior)
+ point_2d_homo = camera_matrix @ point_camera
+ x = point_2d_homo[0, 0] / point_2d_homo[2, 0]
+ y = point_2d_homo[1, 0] / point_2d_homo[2, 0]
+
+ # Check if point is within image bounds
+ if x < 0 or x >= image_size or y < 0 or y >= image_size:
+ return None
+ return (int(round(x)), int(round(y)))
+
+
+def get_camera_matrix(yfov_deg, image_size, ortho=False, radius=None, xmag=None, ymag=None):
+ """
+ Compute camera intrinsic matrix. If ortho=True, return a tuple (matrix, meta)
+ where meta contains orthographic params 'ortho', 'xmag', 'ymag'.
+ """
+ yfov_rad = np.deg2rad(yfov_deg)
+ cx = image_size / 2.0
+ cy = image_size / 2.0
+
+ if ortho:
+ if xmag is None or ymag is None:
+ if radius is None:
+ raise ValueError("radius or xmag/ymag must be provided for orthographic camera matrix calculation")
+ # compute half-height in world units that matches perspective view at distance `radius`
+ ymag = float(radius) * np.tan(yfov_rad / 2.0)
+ xmag = ymag
+ else:
+ xmag = float(xmag)
+ ymag = float(ymag)
+ # scale to pixels: s = image_size / (2 * xmag)
+ s = (image_size / (2.0 * xmag)) if xmag != 0 else 1.0
+ # Use negative scale on y to map world-up to image-down
+ camera_matrix = np.array([
+ [s, 0, cx],
+ [0, -s, cy],
+ [0, 0, 1]
+ ], dtype=np.float32)
+ meta = {"ortho": True, "xmag": xmag, "ymag": ymag}
+ return (camera_matrix, meta)
+
+ # fallback to perspective intrinsics
+ f = (image_size / 2.0) / np.tan(yfov_rad / 2.0)
+ camera_matrix = np.array([
+ [f, 0, cx],
+ [0, f, cy],
+ [0, 0, 1]
+ ], dtype=np.float32)
+ return camera_matrix
+
+
+def overlay_2d_grid(image, grid_2d, mapping, camera_pose, image_size, grid_line_color=(0, 0, 255),
+ grid_point_color=(0, 0, 255), grid_point_radius=5, radius=None, camera_matrix=None,
+ draw_grid_lines=False, draw_object_directions=False, draw_object_bboxes=False,
+ object_infos=None, hide_object_ids=None):
+ """
+ Overlay 2D grid points on the rendered image.
+ """
+ # Convert RGB to BGR for cv2 drawing operations
+ image_bgr = cv2.cvtColor(image.copy(), cv2.COLOR_RGB2BGR)
+ # Use a provided camera matrix when available; otherwise fall back to the existing behavior.
+ if camera_matrix is None and radius is not None:
+ camera_matrix = get_camera_matrix(60.0, image_size, ortho=True, radius=radius)
+ elif camera_matrix is None:
+ camera_matrix = get_camera_matrix(60.0, image_size)
+
+ # Project all grid points to 2D
+ grid_size = grid_2d.shape[0]
+ projected_points = {}
+
+ for (i, j), point_3d in mapping.items():
+ proj = project_3d_to_2d(point_3d, camera_pose, camera_matrix, image_size)
+ if proj is not None:
+ projected_points[(i, j)] = proj
+
+ if not projected_points:
+ print("[WARN] No grid points were successfully projected!")
+ return image
+
+ def grid_coord_to_image(grid_i, grid_j):
+ if grid_size <= 1:
+ return None
+ required = [(0, 0), (grid_size - 1, 0), (0, grid_size - 1)]
+ if any(key not in projected_points for key in required):
+ return None
+ p00 = np.asarray(projected_points[(0, 0)], dtype=np.float32)
+ p10 = np.asarray(projected_points[(grid_size - 1, 0)], dtype=np.float32)
+ p01 = np.asarray(projected_points[(0, grid_size - 1)], dtype=np.float32)
+ u = float(grid_i) / float(grid_size - 1)
+ v = float(grid_j) / float(grid_size - 1)
+ point = p00 + u * (p10 - p00) + v * (p01 - p00)
+ return int(round(float(point[0]))), int(round(float(point[1])))
+
+ # Draw grid lines if enabled
+ if draw_grid_lines:
+ for i in range(grid_size):
+ # Horizontal lines (along j direction)
+ line_points = []
+ for j in range(grid_size):
+ if (i, j) in projected_points:
+ line_points.append(projected_points[(i, j)])
+
+ # Draw connected line segments
+ for k in range(len(line_points) - 1):
+ cv2.line(image_bgr, line_points[k], line_points[k + 1], grid_line_color, 1)
+
+ for j in range(grid_size):
+ # Vertical lines (along i direction)
+ line_points = []
+ for i in range(grid_size):
+ if (i, j) in projected_points:
+ line_points.append(projected_points[(i, j)])
+
+ # Draw connected line segments
+ for k in range(len(line_points) - 1):
+ cv2.line(image_bgr, line_points[k], line_points[k + 1], grid_line_color, 1)
+
+ # Draw grid points and coordinates
+ for (i, j), (x, y) in projected_points.items():
+ cv2.circle(image_bgr, (x, y), grid_point_radius, grid_point_color, -1)
+ # Draw coordinate text below the point
+ text = f"({i},{j})"
+ font = cv2.FONT_HERSHEY_SIMPLEX
+ font_scale = 0.7
+ font_thickness = 2
+ text_size = cv2.getTextSize(text, font, font_scale, font_thickness)[0]
+ text_x = x - text_size[0] // 2
+ text_y = y + grid_point_radius + 20
+ cv2.putText(image_bgr, text, (text_x, text_y), font, font_scale, grid_point_color, font_thickness)
+
+ # Draw per-object overlays in image space.
+ if object_infos:
+ normalized_hide_object_ids = set(resolve_requested_object_ids(hide_object_ids) or [])
+ required = [(0, 0), (grid_size - 1, 0), (0, grid_size - 1)]
+ if all(key in projected_points for key in required) and grid_size > 1:
+ for obj_info in object_infos:
+ object_id = normalize_object_id(obj_info.get("object_id", ""))
+ hide_object = object_id in normalized_hide_object_ids
+
+ center_2d = obj_info.get("center_2d")
+ size_2d = obj_info.get("size_2d")
+ angle_deg = float(obj_info.get("rotation_degrees", 0.0))
+ if center_2d is None or size_2d is None:
+ continue
+
+ center_px = grid_coord_to_image(center_2d[0], center_2d[1])
+ if center_px is None:
+ continue
+
+ corners = obj_info.get("obb_corners_2d")
+ pts = [grid_coord_to_image(pt[0], pt[1]) for pt in corners]
+ if draw_object_bboxes and not hide_object:
+ if all(pt is not None for pt in pts):
+ pts = np.asarray(pts, dtype=np.int32)
+ cv2.polylines(image_bgr, [pts], True, (139, 0, 0), 4)
+
+ label_text = object_id.replace("object", "obj")
+ label_font = cv2.FONT_HERSHEY_SIMPLEX
+ label_font_scale = 0.7
+ label_thickness = 2
+ label_size = cv2.getTextSize(label_text, label_font, label_font_scale, label_thickness)[0]
+ label_margin = 6
+ label_x = int(np.max(pts[:, 0]) - label_size[0])
+ label_y = int(np.min(pts[:, 1]) - label_margin)
+ if label_y < label_size[1] + label_margin:
+ label_y = int(np.max(pts[:, 1]) + label_size[1] + label_margin)
+ label_x = max(0, min(label_x, image_bgr.shape[1] - label_size[0] - 1))
+ label_y = max(label_size[1] + label_margin, min(label_y, image_bgr.shape[0] - label_margin))
+ cv2.putText(image_bgr, label_text, (label_x, label_y), label_font, label_font_scale, (0, 0, 0), label_thickness)
+
+ if not draw_object_directions or object_id.lower().split("_")[0] in ["floor", "wall", "ceiling"]:
+ continue
+
+ arrow_length = estimate_arrow_length_to_bbox(bbox_2d=pts) * 1.25
+ if arrow_length is None:
+ continue
+ else:
+ arrow_length = min(max(arrow_length, 40.0), 400.0)
+
+ # 0 degrees points downward (-v in image), positive angle rotates clockwise.
+ theta = np.deg2rad(angle_deg)
+ fixed_head_px = 12.0
+ dx = -arrow_length * np.sin(theta)
+ dy = arrow_length * np.cos(theta)
+
+ start = (int(round(center_px[0])), int(round(center_px[1])))
+ end = (
+ int(round(center_px[0] + dx)),
+ int(round(center_px[1] + dy)),
+ )
+
+ cv2.arrowedLine(
+ image_bgr,
+ start,
+ end,
+ (255, 128, 0),
+ 4,
+ tipLength=fixed_head_px / arrow_length,
+ )
+
+ # Convert back to RGB for consistency
+ result = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
+ return result
+
+
+def build_camera_pose(camera_direction, radius):
+ """Build camera pose matrix for PyRender."""
+ forward = parse_camera_direction(camera_direction)
+ scene_center = np.zeros(3, dtype=np.float32)
+ camera_position = scene_center - forward * float(radius)
+
+ up_hint = np.array([0.0, 1.0, 0.0], dtype=np.float32)
+ right = np.cross(forward, up_hint) # forward × up_hint (correct order)
+
+ if np.linalg.norm(right) <= 1e-8:
+ up_hint = np.array([0.0, 0.0, 1.0], dtype=np.float32)
+ right = np.cross(forward, up_hint) # same order
+
+ right = right / np.linalg.norm(right)
+ true_up = np.cross(right, forward) # right × forward
+ true_up = true_up / np.linalg.norm(true_up)
+
+ camera_pose = np.eye(4, dtype=np.float32)
+ camera_pose[:3, 0] = right
+ camera_pose[:3, 1] = true_up
+ camera_pose[:3, 2] = -forward
+ camera_pose[:3, 3] = camera_position
+
+ return camera_pose
+
+
+def mesh_to_pyrender_mesh(mesh):
+ return pyrender.Mesh.from_trimesh(mesh, smooth=False)
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Render a canonical-cube normalized scene from unposed meshes and layout parameters."
+ )
+ parser.add_argument(
+ "--layout_dir",
+ type=Path,
+ default=Path("outputs/generated/group_object_001/layout_org/layout_params_simulated"),
+ help="Directory containing object_XXX.json layout files.",
+ )
+ parser.add_argument(
+ "--unposed_dir",
+ type=Path,
+ default=Path("outputs/unposed_objects"),
+ help="Directory containing object_XXX.glb unposed meshes.",
+ )
+ parser.add_argument(
+ "--object_ids",
+ nargs="+",
+ default=None,
+ help="Optional subset of object ids to render (XXX in object_XXX). Defaults to all matched objects.",
+ )
+ parser.add_argument(
+ "--anchor_id",
+ type=str,
+ default=None,
+ help="Optional anchor object id to zero its rotation during normalization.",
+ )
+ parser.add_argument(
+ "--hide_object_ids",
+ nargs="+",
+ default=None,
+ help="Optional object ids to hide bbox overlays on the render.",
+ )
+ parser.add_argument(
+ "--camera_direction",
+ type=float,
+ nargs=3,
+ default=[0.0, -1.0, 0.0],
+ help="Camera look direction in world XYZ. Default is straight down.",
+ )
+ parser.add_argument(
+ "--radius",
+ type=float,
+ default=0.0,
+ help="Camera distance from the canonical-cube scene center. Set 0 to auto-fit the scene.",
+ )
+ parser.add_argument(
+ "--output",
+ type=Path,
+ default=Path("outputs/rendered_scene.png"),
+ help="Output path for the rendered image.",
+ )
+ parser.add_argument(
+ "--image_size",
+ type=int,
+ default=1024,
+ help="Square output image size in pixels.",
+ )
+ parser.add_argument(
+ "--add_grid",
+ action="store_true",
+ help="If set, generate 2D grid points and save mapping to JSON.",
+ )
+ parser.add_argument(
+ "--grid_mapping",
+ type=Path,
+ default=None,
+ help="Optional JSON grid mapping from a previous render to reuse grid points.",
+ )
+ parser.add_argument(
+ "--grid_size",
+ type=int,
+ default=5,
+ help="Number of grid points per side (grid_size x grid_size).",
+ )
+ parser.add_argument(
+ "--draw_grid_lines",
+ action="store_true",
+ help="If set, draw lines connecting grid points.",
+ )
+ parser.add_argument(
+ "--draw_object_directions",
+ action="store_true",
+ help="If set, draw non-anchor object direction arrows using obj_info center/size/rotation.",
+ )
+ parser.add_argument(
+ "--draw_object_bboxes",
+ action="store_true",
+ help="If set, draw 2D oriented bounding boxes for objects on the overlay.",
+ )
+ parser.add_argument(
+ "--use_aabb_for_object_bboxes",
+ action="store_true",
+ help="If set, use axis-aligned bounding boxes instead of oriented bounding boxes for object overlays.",
+ )
+ parser.add_argument(
+ "--plane_side",
+ choices=["farthest", "nearest"],
+ default="nearest",
+ help="Choose whether grid/mapping uses the farthest or nearest plane along the camera direction.",
+ )
+ args = parser.parse_args()
+
+ layout_dir = args.layout_dir
+ unposed_dir = args.unposed_dir
+ output_path = args.output
+
+ if not layout_dir.exists():
+ raise FileNotFoundError(f"Layout directory not found: {layout_dir}")
+ if not unposed_dir.exists():
+ raise FileNotFoundError(f"Unposed mesh directory not found: {unposed_dir}")
+ if args.radius < 0:
+ raise ValueError("--radius must be non-negative.")
+ if args.image_size <= 0:
+ raise ValueError("--image_size must be positive.")
+ if args.grid_size <= 0:
+ raise ValueError("--grid_size must be positive.")
+ if args.draw_object_directions and not args.add_grid:
+ raise ValueError("--draw_object_directions requires --add_grid.")
+ if args.draw_object_bboxes and not args.add_grid:
+ raise ValueError("--draw_object_bboxes requires --add_grid.")
+
+ _, posed_meshes, layout_params, object_ids, transform_info = compose_scene(
+ layout_dir=layout_dir,
+ unposed_dir=unposed_dir,
+ requested_object_ids=args.object_ids,
+ anchor_id=args.anchor_id,
+ )
+
+ # Generate grid if requested
+ grid_2d = None
+ mapping = None
+ plane_right = None
+ plane_up = None
+ object_infos_for_overlay = None
+ if args.add_grid:
+ if args.grid_mapping is not None:
+ if not args.grid_mapping.exists():
+ raise FileNotFoundError(f"Grid mapping file not found: {args.grid_mapping}")
+ mapping, grid_size, plane_right, plane_up = load_grid_mapping(
+ args.grid_mapping, transform_info
+ )
+ grid_2d = np.zeros((grid_size, grid_size, 2), dtype=np.float32)
+ else:
+ grid_2d, mapping, _, plane_right, plane_up = generate_2d_grid_on_plane(
+ posed_meshes, args.camera_direction, args.grid_size, plane_side=args.plane_side
+ )
+
+ if args.draw_object_directions or args.draw_object_bboxes:
+ if mapping is None or plane_right is None or plane_up is None:
+ raise ValueError("Object overlays require grid mapping and plane axes.")
+ # Compute plane_normal and plane_d in original (denormalized) space so that
+ # build_objects_info can populate signed_distance_to_plane.
+ computed_plane_normal = np.cross(plane_right, plane_up)
+ computed_plane_d = None
+ norm = np.linalg.norm(computed_plane_normal)
+ if norm > 1e-8:
+ computed_plane_normal = computed_plane_normal / norm
+ rotation = transform_info.get("rotation_matrix")
+ if rotation is not None:
+ rotation = np.asarray(rotation, dtype=np.float32).reshape(3, 3)
+ computed_plane_normal = computed_plane_normal @ rotation.T
+ norm = np.linalg.norm(computed_plane_normal)
+ if norm > 1e-8:
+ computed_plane_normal = computed_plane_normal / norm
+ sample_point = np.asarray(next(iter(mapping.values())), dtype=np.float32)
+ sample_point = denormalize_point(sample_point, transform_info)
+ computed_plane_d = -float(np.dot(computed_plane_normal, sample_point))
+
+ object_infos_for_overlay = build_objects_info(
+ posed_meshes=posed_meshes,
+ object_ids=object_ids,
+ layout_params=layout_params,
+ axis_right=plane_right,
+ axis_up=plane_up,
+ mapping=mapping,
+ camera_direction=args.camera_direction,
+ plane_normal=computed_plane_normal,
+ plane_d=computed_plane_d,
+ transform_info=transform_info,
+ draw_object_bboxes=args.draw_object_bboxes,
+ use_aabb_for_object_bboxes=args.use_aabb_for_object_bboxes,
+ )
+
+ rendered = render_scene_with_grid(
+ meshes=posed_meshes,
+ camera_direction=args.camera_direction,
+ radius=args.radius,
+ image_size=args.image_size,
+ grid_2d=grid_2d,
+ mapping=mapping,
+ draw_grid_lines=args.draw_grid_lines,
+ draw_object_directions=args.draw_object_directions,
+ draw_object_bboxes=args.draw_object_bboxes,
+ object_infos=object_infos_for_overlay,
+ hide_object_ids=args.hide_object_ids,
+ )
+
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ if not cv2.imwrite(str(output_path), cv2.cvtColor(rendered, cv2.COLOR_RGB2BGR)):
+ raise RuntimeError(f"Failed to write rendered image to {output_path}")
+ print(f"Saved rendered scene to: {output_path} (objects: {object_ids})")
+
+ # Save grid mapping if generated
+ if args.add_grid and mapping is not None:
+ mapping_output = output_path.with_suffix(".json")
+ save_grid_mapping(
+ mapping,
+ mapping_output,
+ transform_info,
+ posed_meshes=posed_meshes,
+ object_ids=object_ids,
+ axis_right=plane_right,
+ axis_up=plane_up,
+ camera_direction=args.camera_direction,
+ layout_params=layout_params,
+ objects_info=object_infos_for_overlay,
+ use_aabb_for_object_bboxes=args.use_aabb_for_object_bboxes,
+ )
+ print(f"Saved grid mapping to: {mapping_output}")
+
+
+if __name__ == "__main__":
+ main()
\ No newline at end of file
diff --git a/vlm_agent/evolution/tools/retrieve_room_materials.py b/vlm_agent/evolution/tools/retrieve_room_materials.py
new file mode 100644
index 0000000..5dfac20
--- /dev/null
+++ b/vlm_agent/evolution/tools/retrieve_room_materials.py
@@ -0,0 +1,228 @@
+#!/usr/bin/env python3
+"""Resolve room surface descriptions to local PBR material directories."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+
+import numpy as np
+import open_clip
+import torch
+import yaml
+
+SUMMARY_PREFIX = "ROOM_MATERIAL_RETRIEVAL_SUMMARY:"
+SURFACE_KEYS = ("walls", "floor", "ceiling")
+REQUIRED_CHANNELS = ("Color", "NormalGL", "Roughness")
+DEFAULT_TEXTURE_RESOLUTION = "2K"
+DEFAULT_TEXTURE_FORMAT = "PNG"
+TEXTURE_RESOLUTIONS = ("1K", "2K", "4K", "8K", "12K", "16K")
+TEXTURE_FORMATS = ("PNG", "JPG")
+
+
+def env_texture_resolution() -> str:
+ """Resolve the texture resolution from the environment (default 2K)."""
+ value = os.environ.get("SCENEMOSAIC_MATERIAL_RESOLUTION")
+ return value if value in TEXTURE_RESOLUTIONS else DEFAULT_TEXTURE_RESOLUTION
+
+
+def env_texture_format() -> str:
+ """Resolve the texture format from the environment (default PNG)."""
+ value = os.environ.get("SCENEMOSAIC_MATERIAL_FORMAT")
+ return value if value in TEXTURE_FORMATS else DEFAULT_TEXTURE_FORMAT
+
+_cached_model = None
+_cached_tokenizer = None
+_cached_device = None
+
+
+def print_summary(payload: dict) -> None:
+ print(f"{SUMMARY_PREFIX} {json.dumps(payload, ensure_ascii=False)}")
+
+
+def load_room_material_queries(objects_json: Path) -> dict[str, str] | None:
+ with objects_json.open("r", encoding="utf-8") as f:
+ objects_data = json.load(f)
+
+ room_materials = objects_data.get("room_materials")
+ if room_materials is None:
+ return None
+ if not isinstance(room_materials, dict):
+ raise ValueError("room_materials must be an object.")
+
+ queries = {}
+ for key in SURFACE_KEYS:
+ query = room_materials.get(key)
+ if not isinstance(query, str) or not query.strip():
+ raise ValueError(f"room_materials.{key} must be a non-empty string.")
+ queries[key] = query.strip()
+ return queries
+
+
+def load_material_index(embeddings_dir: Path) -> tuple[np.ndarray, list[str]]:
+ embeddings_path = embeddings_dir / "clip_embeddings.npy"
+ index_path = embeddings_dir / "embedding_index.yaml"
+ embeddings = np.load(embeddings_path)
+ with index_path.open("r", encoding="utf-8") as f:
+ material_ids = yaml.safe_load(f)
+
+ if not isinstance(material_ids, list) or not all(isinstance(item, str) for item in material_ids):
+ raise ValueError("embedding_index.yaml must contain a list of material IDs.")
+ if embeddings.ndim != 2 or embeddings.shape[0] != len(material_ids):
+ raise ValueError("Embedding rows do not match the number of material IDs.")
+ if not len(material_ids):
+ raise ValueError("Material embedding index is empty.")
+ return embeddings.astype(np.float32, copy=False), material_ids
+
+
+def get_clip_model(device: str | None) -> tuple[object, object, str]:
+ global _cached_model, _cached_tokenizer, _cached_device
+
+ target_device = device or ("cuda" if torch.cuda.is_available() else "cpu")
+ if _cached_model is None or _cached_device != target_device:
+ cache_dir = Path(__file__).resolve().parents[1] / "cache" / "open_clip"
+ _cached_model, _, _ = open_clip.create_model_and_transforms(
+ "ViT-H-14-378-quickgelu",
+ pretrained="dfn5b",
+ device=target_device,
+ cache_dir=cache_dir,
+ )
+ _cached_model.eval()
+ _cached_tokenizer = open_clip.get_tokenizer("ViT-H-14-378-quickgelu")
+ _cached_device = target_device
+ return _cached_model, _cached_tokenizer, target_device
+
+
+def retrieve_material_id(query: str, embeddings: np.ndarray, material_ids: list[str], device: str | None) -> str:
+ model, tokenizer, target_device = get_clip_model(device)
+ text_tokens = tokenizer([query]).to(target_device)
+ with torch.no_grad():
+ text_features = model.encode_text(text_tokens)
+ text_features = text_features / text_features.norm(dim=-1, keepdim=True)
+ query_embedding = text_features.detach().cpu().numpy()[0].astype(np.float32, copy=False)
+
+ embedding_norms = np.linalg.norm(embeddings, axis=1)
+ query_norm = np.linalg.norm(query_embedding)
+ if query_norm <= 0 or np.any(embedding_norms <= 0):
+ raise ValueError("Encountered a zero-norm CLIP embedding.")
+ similarities = (embeddings / embedding_norms[:, None]) @ (query_embedding / query_norm)
+ return material_ids[int(np.argmax(similarities))]
+
+
+def texture_path(
+ material_dir: Path,
+ channel: str,
+ *,
+ resolution: str | None = None,
+ file_format: str | None = None,
+) -> Path:
+ """Return the path of one ambientCG texture of ``material_dir``.
+
+ ambientCG names textures ``<material>_<res>-<fmt>_<channel>.<ext>`` where
+ the filename label uses the format in uppercase (e.g. ``2K-PNG_Color.png``
+ for PNG, ``2K-JPG_Color.jpg`` for JPG). ``channel`` is the map suffix such
+ as ``Color``, ``NormalGL`` or ``Roughness``. Unset arguments fall back to
+ the environment overrides, then to 2K PNG.
+ """
+ resolution = resolution or env_texture_resolution()
+ file_format = file_format or env_texture_format()
+ extension = file_format.lower()
+ return material_dir / (
+ f"{material_dir.name}_{resolution}-{file_format}_{channel}.{extension}"
+ )
+
+
+def validate_material_dir(
+ materials_dir: Path,
+ material_id: str,
+ *,
+ resolution: str | None = None,
+ file_format: str | None = None,
+) -> Path:
+ material_dir = materials_dir / material_id
+ required_paths = [
+ texture_path(
+ material_dir, channel, resolution=resolution, file_format=file_format
+ )
+ for channel in REQUIRED_CHANNELS
+ ]
+ if not material_dir.is_dir():
+ raise FileNotFoundError(f"Material directory not found: {material_dir}")
+ missing_paths = [str(path) for path in required_paths if not path.is_file()]
+ if missing_paths:
+ raise FileNotFoundError(f"Material textures not found: {', '.join(missing_paths)}")
+ return material_dir
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="Retrieve PBR materials for room surfaces.")
+ parser.add_argument("--objects_json", type=Path, required=True)
+ parser.add_argument("--materials_dir", type=Path, required=True)
+ parser.add_argument("--embeddings_dir", type=Path, required=True)
+ parser.add_argument("--device", type=str, default=None)
+ parser.add_argument(
+ "--resolution",
+ "-r",
+ type=str,
+ default=env_texture_resolution(),
+ choices=list(TEXTURE_RESOLUTIONS),
+ help=(
+ "Material texture resolution (default: "
+ f"$SCENEMOSAIC_MATERIAL_RESOLUTION or {DEFAULT_TEXTURE_RESOLUTION})"
+ ),
+ )
+ parser.add_argument(
+ "--format",
+ "-f",
+ type=str,
+ default=env_texture_format(),
+ choices=list(TEXTURE_FORMATS),
+ help=(
+ "Material texture file format (default: "
+ f"$SCENEMOSAIC_MATERIAL_FORMAT or {DEFAULT_TEXTURE_FORMAT})"
+ ),
+ )
+ args = parser.parse_args()
+
+ try:
+ queries = load_room_material_queries(args.objects_json)
+ if queries is None:
+ print_summary({"status": "legacy", "reason": "room_materials_absent"})
+ return 0
+
+ embeddings, material_ids = load_material_index(args.embeddings_dir)
+ selected_ids = {
+ surface: retrieve_material_id(query, embeddings, material_ids, args.device)
+ for surface, query in queries.items()
+ }
+ selected_dirs = {
+ surface: validate_material_dir(
+ args.materials_dir,
+ material_id,
+ resolution=args.resolution,
+ file_format=args.format,
+ )
+ for surface, material_id in selected_ids.items()
+ }
+ print_summary(
+ {
+ "status": "resolved",
+ "wall_material_dir": str(selected_dirs["walls"]),
+ "floor_material_dir": str(selected_dirs["floor"]),
+ "ceiling_material_dir": str(selected_dirs["ceiling"]),
+ "material_ids": selected_ids,
+ }
+ )
+ return 0
+ except Exception as exc:
+ message = str(exc)
+ print(f"Room material retrieval failed: {message}", file=sys.stderr)
+ print_summary({"status": "failed", "reason": type(exc).__name__, "error": message})
+ return 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/vlm_agent/evolution/tools/simulate_scene_physics.py b/vlm_agent/evolution/tools/simulate_scene_physics.py
new file mode 100644
index 0000000..4db4f7b
--- /dev/null
+++ b/vlm_agent/evolution/tools/simulate_scene_physics.py
@@ -0,0 +1,1228 @@
+import argparse
+import json
+from pathlib import Path
+
+import numpy as np
+
+try:
+ import sapien
+except ImportError as e:
+ try:
+ import sapien.core as sapien
+ except ImportError as inner_e:
+ raise ImportError(
+ "sapien is required for simulation. Install it with: pip install sapien"
+ ) from inner_e
+
+try:
+ from sapien.utils.viewer import Viewer as SapienViewer
+except Exception:
+ SapienViewer = None
+
+
+def as_vector(value, expected_sizes):
+ arr = np.asarray(value, dtype=np.float32).reshape(-1)
+ if arr.size not in expected_sizes:
+ raise ValueError(f"Expected sizes {expected_sizes}, but got shape {arr.shape}")
+ return arr
+
+
+def parse_scale(value):
+ arr = as_vector(value, {1, 3})
+ if arr.size == 1:
+ return np.repeat(arr, 3)
+ return arr
+
+
+def parse_gravity_vector(value):
+ gravity = as_vector(value, {1, 3})
+ if gravity.size == 1:
+ gravity = np.array([0.0, -float(gravity[0]), 0.0], dtype=np.float32)
+ norm = np.linalg.norm(gravity)
+ if norm <= 1e-8:
+ raise ValueError("Gravity vector norm is zero.")
+ return gravity.astype(np.float32)
+
+
+def parse_translation(value):
+ return as_vector(value, {3})
+
+
+def parse_rotation_wxyz(value):
+ quat = as_vector(value, {4})
+ norm = np.linalg.norm(quat)
+ if norm <= 1e-8:
+ raise ValueError("Quaternion norm is zero.")
+ return quat / norm
+
+
+def quat_wxyz_to_rotmat(quat_wxyz):
+ q = parse_rotation_wxyz(quat_wxyz)
+ w, x, y, z = q.tolist()
+ xx, yy, zz = x * x, y * y, z * z
+ xy, xz, yz = x * y, x * z, y * z
+ wx, wy, wz = w * x, w * y, w * z
+ return np.array(
+ [
+ [1.0 - 2.0 * (yy + zz), 2.0 * (xy - wz), 2.0 * (xz + wy)],
+ [2.0 * (xy + wz), 1.0 - 2.0 * (xx + zz), 2.0 * (yz - wx)],
+ [2.0 * (xz - wy), 2.0 * (yz + wx), 1.0 - 2.0 * (xx + yy)],
+ ],
+ dtype=np.float32,
+ )
+
+
+def quat_wxyz_conjugate(quat_wxyz):
+ q = parse_rotation_wxyz(quat_wxyz)
+ return np.array([q[0], -q[1], -q[2], -q[3]], dtype=np.float32)
+
+
+def quat_wxyz_multiply(lhs_wxyz, rhs_wxyz):
+ lw, lx, ly, lz = parse_rotation_wxyz(lhs_wxyz).tolist()
+ rw, rx, ry, rz = parse_rotation_wxyz(rhs_wxyz).tolist()
+ out = np.array(
+ [
+ lw * rw - lx * rx - ly * ry - lz * rz,
+ lw * rx + lx * rw + ly * rz - lz * ry,
+ lw * ry - lx * rz + ly * rw + lz * rx,
+ lw * rz + lx * ry - ly * rx + lz * rw,
+ ],
+ dtype=np.float32,
+ )
+ return parse_rotation_wxyz(out)
+
+
+def quat_delta_wxyz(prev_wxyz, curr_wxyz):
+ return quat_wxyz_multiply(curr_wxyz, quat_wxyz_conjugate(prev_wxyz))
+
+
+def quat_delta_angle_deg(prev_wxyz, curr_wxyz):
+ delta = quat_delta_wxyz(prev_wxyz, curr_wxyz)
+ angle_rad = 2.0 * np.arctan2(np.linalg.norm(delta[1:]), max(float(delta[0]), 1e-8))
+ return float(np.degrees(abs(angle_rad)))
+
+
+def rotmat_to_quat_wxyz(rotmat):
+ """Convert a 3x3 rotation matrix to a normalized wxyz quaternion."""
+ m = np.asarray(rotmat, dtype=np.float32).reshape(3, 3)
+ trace = float(np.trace(m))
+ if trace > 0.0:
+ s = np.sqrt(trace + 1.0) * 2.0
+ w = 0.25 * s
+ x = (m[2, 1] - m[1, 2]) / s
+ y = (m[0, 2] - m[2, 0]) / s
+ z = (m[1, 0] - m[0, 1]) / s
+ elif m[0, 0] > m[1, 1] and m[0, 0] > m[2, 2]:
+ s = np.sqrt(1.0 + m[0, 0] - m[1, 1] - m[2, 2]) * 2.0
+ w = (m[2, 1] - m[1, 2]) / s
+ x = 0.25 * s
+ y = (m[0, 1] + m[1, 0]) / s
+ z = (m[0, 2] + m[2, 0]) / s
+ elif m[1, 1] > m[2, 2]:
+ s = np.sqrt(1.0 + m[1, 1] - m[0, 0] - m[2, 2]) * 2.0
+ w = (m[0, 2] - m[2, 0]) / s
+ x = (m[0, 1] + m[1, 0]) / s
+ y = 0.25 * s
+ z = (m[1, 2] + m[2, 1]) / s
+ else:
+ s = np.sqrt(1.0 + m[2, 2] - m[0, 0] - m[1, 1]) * 2.0
+ w = (m[1, 0] - m[0, 1]) / s
+ x = (m[0, 2] + m[2, 0]) / s
+ y = (m[1, 2] + m[2, 1]) / s
+ z = 0.25 * s
+
+ return parse_rotation_wxyz([w, x, y, z])
+
+
+def normalized_vector(value):
+ vec = np.asarray(value, dtype=np.float32).reshape(3)
+ norm = np.linalg.norm(vec)
+ if norm <= 1e-8:
+ raise ValueError("Vector norm is zero.")
+ return vec / norm
+
+
+def normalize_object_id(object_id):
+ object_id = str(object_id)
+ if object_id and object_id[0].isdigit():
+ return f"object_{object_id}"
+ return object_id
+
+
+def resolve_requested_object_ids(requested_object_ids):
+ if requested_object_ids is None:
+ return None
+ if isinstance(requested_object_ids, (str, int)):
+ requested_object_ids = [requested_object_ids]
+ return [normalize_object_id(object_id) for object_id in requested_object_ids]
+
+
+def layout_pose_to_sapien(translation, rotation_wxyz):
+ """Return a SAPIEN pose in the same P3D frame used by layout JSON files."""
+ return translation.tolist(), rotation_wxyz.tolist()
+
+
+def sapien_pose_to_layout(translation_xyz, rotation_wxyz):
+ """Convert a SAPIEN pose back to P3D layout JSON fields.
+
+ SAPIEN runs in the same world frame as the layout data here, so fields can
+ be written back directly in translation + wxyz format.
+ """
+ quat = parse_rotation_wxyz(rotation_wxyz)
+ return (
+ [list(map(float, translation_xyz))],
+ [list(map(float, quat))],
+ )
+
+
+def load_object_ids(layout_dir: Path, hull_dir: Path, requested_object_ids=None):
+ layout_ids = {p.stem for p in layout_dir.glob("*.json")}
+ hull_ids = {p.stem for p in hull_dir.glob("*.obj")}
+ matched_ids = sorted(layout_ids & hull_ids)
+
+ normalized_requested_ids = resolve_requested_object_ids(requested_object_ids)
+ if normalized_requested_ids is None:
+ return matched_ids
+
+ missing_ids = [object_id for object_id in normalized_requested_ids if object_id not in matched_ids]
+ if missing_ids:
+ normalized_requested_ids = [obj_id for obj_id in normalized_requested_ids if obj_id not in missing_ids]
+ if not normalized_requested_ids:
+ raise FileNotFoundError(
+ f"Requested object ids not found in both layout and unposed directories: {missing_ids}. Matched ids: {matched_ids}"
+ )
+ else:
+ print(f"Warning: Requested object ids not found in both layout and unposed directories: {missing_ids}. Matched ids: {matched_ids}")
+ return normalized_requested_ids
+
+
+def load_layout(layout_path: Path):
+ with layout_path.open("r", encoding="utf-8") as f:
+ raw = json.load(f)
+
+ translation = parse_translation(raw["translation"])
+ rotation_wxyz = parse_rotation_wxyz(raw["rotation"])
+ scale = parse_scale(raw["scale"])
+
+ return raw, translation, rotation_wxyz, scale
+
+
+def load_obj_vertices(obj_path: Path):
+ vertices = []
+ with obj_path.open("r", encoding="utf-8") as f:
+ for line in f:
+ if line.startswith("v "):
+ parts = line.strip().split()
+ if len(parts) >= 4:
+ vertices.append([float(parts[1]), float(parts[2]), float(parts[3])])
+
+ if not vertices:
+ raise ValueError(f"No vertex lines found in OBJ: {obj_path}")
+
+ return np.asarray(vertices, dtype=np.float32)
+
+
+def compute_scene_ground_plane_from_hulls(layout_dir: Path, hull_dir: Path, object_ids, gravity_xyz):
+ up_direction = normalized_vector(-np.asarray(gravity_xyz, dtype=np.float32).reshape(3))
+ surface_level = None
+
+ for object_id in object_ids:
+ layout_path = layout_dir / f"{normalize_object_id(object_id)}.json"
+ hull_path = hull_dir / f"{normalize_object_id(object_id)}.obj"
+
+ _, translation, rotation_wxyz, scale = load_layout(layout_path)
+ local_vertices = load_obj_vertices(hull_path)
+
+ rotation = quat_wxyz_to_rotmat(rotation_wxyz)
+ scaled_vertices = local_vertices * scale.reshape(1, 3)
+ world_vertices = scaled_vertices @ rotation.T + translation.reshape(1, 3)
+
+ obj_min_level = float(np.min(world_vertices @ up_direction))
+ if surface_level is None or obj_min_level < surface_level:
+ surface_level = obj_min_level
+
+ if surface_level is None:
+ raise RuntimeError("Failed to compute scene ground plane from hulls.")
+
+ return up_direction, float(surface_level)
+
+
+def compute_initial_y_offset(layout_dir: Path, hull_dir: Path, object_ids, selected_ground_object_ids):
+ max_extent = None
+ selected_ground_object_ids = set(selected_ground_object_ids or [])
+
+ for object_id in object_ids:
+ if object_id in selected_ground_object_ids:
+ continue
+
+ layout_path = layout_dir / f"{normalize_object_id(object_id)}.json"
+ hull_path = hull_dir / f"{normalize_object_id(object_id)}.obj"
+
+ _, _, _, scale = load_layout(layout_path)
+ local_vertices = load_obj_vertices(hull_path)
+ bbox_extents = np.ptp(local_vertices, axis=0)
+ scaled_extents = bbox_extents * scale.reshape(3)
+ object_max_extent = float(np.max(scaled_extents))
+ if max_extent is None or object_max_extent > max_extent:
+ max_extent = object_max_extent
+
+ if max_extent is None:
+ return 0.0
+
+ return float(min(max_extent / 2.0, 0.4))
+
+
+def choose_ground_object_id(
+ layout_dir: Path,
+ object_ids,
+ use_ground_object: bool,
+ ground_object_ids=None,
+):
+ if not use_ground_object:
+ return set()
+
+ if ground_object_ids:
+ normalized_ids = []
+ for object_id in ground_object_ids:
+ normalized_id = normalize_object_id(object_id)
+ if normalized_id not in object_ids:
+ raise ValueError(
+ f"Requested --ground_object_id={normalized_id}, "
+ f"but it is not in matched objects: {object_ids}"
+ )
+ normalized_ids.append(normalized_id)
+ return set(normalized_ids)
+
+ min_y = None
+ selected_id = None
+ for object_id in object_ids:
+ layout_path = layout_dir / f"{normalize_object_id(object_id)}.json"
+ _, translation, _, _ = load_layout(layout_path)
+ y = float(translation[1])
+ if min_y is None or y < min_y:
+ min_y = y
+ selected_id = object_id
+ return {selected_id} if selected_id is not None else set()
+
+
+def create_scene_compatible(engine, scene_config, cpu_only: bool):
+ """Create a SAPIEN scene with optional CPU-only mode.
+
+ Newer SAPIEN wrappers build a render system by default when using
+ Engine.create_scene(), which fails on machines without a rendering device.
+ This helper provides a pure-physics fallback scene.
+ """
+ if cpu_only:
+ if hasattr(sapien, "physx") and hasattr(sapien.physx, "set_scene_config"):
+ sapien.physx.set_scene_config(scene_config)
+ if hasattr(sapien, "Scene") and hasattr(sapien.physx, "PhysxCpuSystem"):
+ return sapien.Scene([sapien.physx.PhysxCpuSystem()])
+ raise RuntimeError(
+ "--cpu_only was set, but this SAPIEN build does not expose "
+ "sapien.Scene([sapien.physx.PhysxCpuSystem()])."
+ )
+
+ try:
+ return engine.create_scene(scene_config)
+ except RuntimeError as e:
+ msg = str(e).lower()
+ if "render" in msg and "device" in msg:
+ print(
+ "[WARN] Failed to create scene with renderer (no rendering device). "
+ "Falling back to CPU-only PhysX scene."
+ )
+ if hasattr(sapien, "physx") and hasattr(sapien.physx, "set_scene_config"):
+ sapien.physx.set_scene_config(scene_config)
+ if hasattr(sapien, "Scene") and hasattr(sapien.physx, "PhysxCpuSystem"):
+ return sapien.Scene([sapien.physx.PhysxCpuSystem()])
+ raise RuntimeError(
+ "SAPIEN cannot create a render scene, and CPU-only scene API is unavailable."
+ ) from e
+ raise
+
+
+def create_renderer_compatible():
+ """Create a renderer across SAPIEN API variants."""
+ if hasattr(sapien, "SapienRenderer"):
+ return sapien.SapienRenderer()
+ if hasattr(sapien, "VulkanRenderer"):
+ return sapien.VulkanRenderer()
+ raise RuntimeError(
+ "No supported SAPIEN renderer found (SapienRenderer/VulkanRenderer)."
+ )
+
+
+def add_visual_mesh_compatible(builder, mesh_path: Path, scale):
+ """Attach visual mesh if the SAPIEN API supports it."""
+ if not hasattr(builder, "add_visual_from_file"):
+ return False
+
+ try:
+ builder.add_visual_from_file(
+ filename=str(mesh_path),
+ pose=sapien.Pose(),
+ scale=scale.tolist(),
+ )
+ return True
+ except TypeError:
+ try:
+ builder.add_visual_from_file(str(mesh_path))
+ return True
+ except Exception:
+ return False
+ except Exception:
+ return False
+
+
+def setup_scene_lighting(scene):
+ """Add robust default lights for GUI rendering."""
+ if hasattr(scene, "set_ambient_light"):
+ scene.set_ambient_light([0.35, 0.35, 0.35])
+ if hasattr(scene, "add_directional_light"):
+ scene.add_directional_light(
+ direction=[0.5, -1.0, -0.3],
+ color=[1.0, 1.0, 1.0],
+ shadow=True,
+ )
+ if hasattr(scene, "add_point_light"):
+ scene.add_point_light([3.0, 3.0, 3.0], [1.0, 1.0, 1.0], shadow=False)
+ scene.add_point_light([-3.0, 3.0, -3.0], [0.7, 0.7, 0.7], shadow=False)
+
+
+def _poll_viewer_key(window, key_name: str):
+ """Best-effort key polling across viewer API variants."""
+ candidates = ["key_down", "is_key_down", "key_press", "is_key_pressed"]
+ key_variants = [key_name, key_name.lower(), key_name.upper()]
+ for attr in candidates:
+ if not hasattr(window, attr):
+ continue
+ fn = getattr(window, attr)
+ for key in key_variants:
+ try:
+ if bool(fn(key)):
+ return True
+ except Exception:
+ continue
+ return False
+
+
+def quaternion_wxyz_to_rpy(quat_wxyz):
+ """Convert a wxyz quaternion to roll/pitch/yaw in radians."""
+ w, x, y, z = parse_rotation_wxyz(quat_wxyz).tolist()
+
+ sinr_cosp = 2.0 * (w * x + y * z)
+ cosr_cosp = 1.0 - 2.0 * (x * x + y * y)
+ roll = float(np.arctan2(sinr_cosp, cosr_cosp))
+
+ sinp = 2.0 * (w * y - z * x)
+ if abs(sinp) >= 1.0:
+ pitch = float(np.sign(sinp) * (np.pi / 2.0))
+ else:
+ pitch = float(np.arcsin(sinp))
+
+ siny_cosp = 2.0 * (w * z + x * y)
+ cosy_cosp = 1.0 - 2.0 * (y * y + z * z)
+ yaw = float(np.arctan2(siny_cosp, cosy_cosp))
+
+ return roll, pitch, yaw
+
+
+def limit_dynamic_actor_velocities(
+ actor_to_data,
+ selected_ground_object_ids,
+ max_abs_velocity_xyz,
+ max_abs_angular_velocity_xyz,
+):
+ """Clamp dynamic object linear/angular velocities component-wise in world XYZ."""
+ max_abs_velocity_xyz = np.asarray(max_abs_velocity_xyz, dtype=np.float32).reshape(3)
+ max_abs_angular_velocity_xyz = np.asarray(
+ max_abs_angular_velocity_xyz, dtype=np.float32
+ ).reshape(3)
+
+ def _get_set_velocity_accessors(entity):
+ # Old-style Actor API
+ linear_getter = None
+ linear_setter = None
+ angular_getter = None
+ angular_setter = None
+
+ if hasattr(entity, "get_velocity") and hasattr(entity, "set_velocity"):
+ linear_getter = entity.get_velocity
+ linear_setter = entity.set_velocity
+ if hasattr(entity, "get_angular_velocity") and hasattr(entity, "set_angular_velocity"):
+ angular_getter = entity.get_angular_velocity
+ angular_setter = entity.set_angular_velocity
+
+ if (
+ linear_getter is not None
+ and linear_setter is not None
+ and angular_getter is not None
+ and angular_setter is not None
+ ):
+ return linear_getter, linear_setter, angular_getter, angular_setter
+
+ # Newer Entity + PhysX component API
+ physx = getattr(sapien, "physx", None)
+ dyn_type = getattr(physx, "PhysxRigidDynamicComponent", None)
+ if dyn_type is not None and hasattr(entity, "find_component_by_type"):
+ try:
+ comp = entity.find_component_by_type(dyn_type)
+ except Exception:
+ comp = None
+ if comp is not None:
+ if hasattr(comp, "get_linear_velocity") and hasattr(comp, "set_linear_velocity"):
+ linear_getter = comp.get_linear_velocity
+ linear_setter = comp.set_linear_velocity
+ elif hasattr(comp, "linear_velocity"):
+
+ def _linear_getter():
+ return comp.linear_velocity
+
+ def _linear_setter(v):
+ comp.linear_velocity = v
+
+ linear_getter = _linear_getter
+ linear_setter = _linear_setter
+
+ if hasattr(comp, "get_angular_velocity") and hasattr(comp, "set_angular_velocity"):
+ angular_getter = comp.get_angular_velocity
+ angular_setter = comp.set_angular_velocity
+ elif hasattr(comp, "angular_velocity"):
+
+ def _angular_getter():
+ return comp.angular_velocity
+
+ def _angular_setter(v):
+ comp.angular_velocity = v
+
+ angular_getter = _angular_getter
+ angular_setter = _angular_setter
+
+ return linear_getter, linear_setter, angular_getter, angular_setter
+
+ for actor, data in actor_to_data.items():
+ if data["object_id"] in selected_ground_object_ids:
+ continue
+
+ (
+ velocity_getter,
+ velocity_setter,
+ angular_velocity_getter,
+ angular_velocity_setter,
+ ) = _get_set_velocity_accessors(actor)
+ if (
+ velocity_getter is None
+ or velocity_setter is None
+ or angular_velocity_getter is None
+ or angular_velocity_setter is None
+ ):
+ continue
+
+ velocity = np.asarray(velocity_getter(), dtype=np.float32)
+ clipped_velocity = np.clip(velocity, -max_abs_velocity_xyz, max_abs_velocity_xyz)
+ if not np.allclose(velocity, clipped_velocity):
+ velocity_setter(clipped_velocity)
+
+ angular_velocity = np.asarray(angular_velocity_getter(), dtype=np.float32)
+ clipped_angular_velocity = np.clip(
+ angular_velocity,
+ -max_abs_angular_velocity_xyz,
+ max_abs_angular_velocity_xyz,
+ )
+ if not np.allclose(angular_velocity, clipped_angular_velocity):
+ angular_velocity_setter(clipped_angular_velocity)
+
+
+def run_simulation(
+ layout_dir: Path,
+ hull_dir: Path,
+ output_layout_dir: Path,
+ gravity,
+ timestep: float,
+ depenetration_steps: int,
+ simulation_steps: int,
+ mass: float,
+ linear_damping: float,
+ angular_damping: float,
+ solver_iterations: int,
+ use_ground_object: bool,
+ ground_object_ids,
+ ground_plane_mode: str,
+ gui: bool,
+ gui_manual_control: bool,
+ pose_delta_output: Path | None,
+ object_ids=None,
+ verify_only: bool = False,
+ lock_rotation_all: bool = False,
+ cpu_only: bool = False,
+ init_non_ground_y_offset: bool = False,
+):
+ gravity_xyz = parse_gravity_vector(gravity)
+ requested_object_ids = resolve_requested_object_ids(object_ids)
+ object_ids = load_object_ids(layout_dir, hull_dir, requested_object_ids=requested_object_ids)
+ if not object_ids:
+ raise FileNotFoundError(
+ f"No matching object files in {layout_dir} and {hull_dir}."
+ )
+
+ if verify_only:
+ print("[INFO] Running in verify-only mode (no simulation)")
+ print(f"[INFO] Found {len(object_ids)} matched objects: {object_ids}")
+ for object_id in object_ids:
+ layout_path = layout_dir / f"{normalize_object_id(object_id)}.json"
+ hull_path = hull_dir / f"{normalize_object_id(object_id)}.obj"
+ raw_layout, translation, rotation_wxyz, scale = load_layout(layout_path)
+ print(f"[OK] object_{object_id}:")
+ print(f" Layout file: {layout_path}")
+ print(f" Hull file: {hull_path} (exists: {hull_path.exists()})")
+ print(f" Translation: {translation}")
+ print(f" Rotation (wxyz): {rotation_wxyz}")
+ print(f" Scale: {scale}")
+ print("[INFO] All objects loaded successfully!")
+ return
+
+ selected_ground_object_ids = choose_ground_object_id(
+ layout_dir=layout_dir,
+ object_ids=object_ids,
+ use_ground_object=use_ground_object,
+ ground_object_ids=ground_object_ids,
+ )
+ if selected_ground_object_ids:
+ selected_ground_object_ids = sorted(selected_ground_object_ids)
+ print(
+ "[INFO] Using static ground objects: "
+ + ", ".join(normalize_object_id(object_id) for object_id in selected_ground_object_ids)
+ )
+
+ initial_y_offset = 0.0
+ if init_non_ground_y_offset:
+ initial_y_offset = compute_initial_y_offset(
+ layout_dir=layout_dir,
+ hull_dir=hull_dir,
+ object_ids=object_ids,
+ selected_ground_object_ids=selected_ground_object_ids,
+ )
+ print(f"[INFO] Initial non-ground Y offset enabled: {initial_y_offset:.6f}")
+
+ if cpu_only:
+ print("[INFO] CPU-only mode enabled: scene will be created without RenderSystem.")
+ if gui:
+ print("[WARN] --gui is incompatible with --cpu_only. Falling back to headless mode.")
+
+ engine = sapien.Engine()
+ renderer = None
+ gui_enabled = bool(gui and (not cpu_only))
+ gui_manual_enabled = False
+
+ if not cpu_only:
+ try:
+ renderer = create_renderer_compatible()
+ if hasattr(engine, "set_renderer"):
+ engine.set_renderer(renderer)
+ except Exception as e:
+ print(f"[WARN] Failed to initialize SAPIEN renderer: {e}")
+ if gui_enabled:
+ print("[WARN] GUI requested, but renderer init failed. Falling back to headless mode.")
+ gui_enabled = False
+
+ viewer = None
+ if gui_enabled:
+ if SapienViewer is None:
+ print(
+ "[WARN] sapien.utils.viewer.Viewer is unavailable in this environment. "
+ "Falling back to headless mode."
+ )
+ gui_enabled = False
+ else:
+ try:
+ if renderer is not None:
+ viewer = SapienViewer(renderer)
+ else:
+ viewer = SapienViewer()
+ except TypeError:
+ try:
+ viewer = SapienViewer()
+ except Exception as e:
+ print(f"[WARN] Failed to initialize Viewer: {e}")
+ gui_enabled = False
+ except Exception as e:
+ print(f"[WARN] Failed to initialize Viewer: {e}")
+ gui_enabled = False
+
+ if viewer is not None:
+ print("[INFO] GUI mode enabled.")
+ gui_manual_enabled = bool(gui_manual_control)
+ if gui_manual_enabled:
+ print(
+ "[INFO] GUI manual mode enabled: starts paused at first frame. "
+ "Controls: [N] single-step, [Space] run/pause, [R] reset pause, [Q/Esc] quit."
+ )
+
+ def bind_viewer_scene(scene):
+ if viewer is None:
+ return
+ if hasattr(viewer, "set_scene"):
+ viewer.set_scene(scene)
+ camera_translation = [-0.02, 0.05, -0.68]
+
+ if hasattr(viewer, "set_camera_xyz"):
+ viewer.set_camera_xyz(*camera_translation)
+
+ if ground_plane_mode == "bbox":
+ ground_up_direction, ground_surface_level = compute_scene_ground_plane_from_hulls(
+ layout_dir,
+ hull_dir,
+ object_ids,
+ gravity_xyz,
+ )
+ print(
+ f"[INFO] Ground plane mode=bbox, up_direction={ground_up_direction.tolist()}, "
+ f"surface_level={ground_surface_level:.6f}"
+ )
+ else:
+ ground_up_direction = np.array([0.0, 1.0, 0.0], dtype=np.float32)
+ ground_surface_level = 0.0
+ print("[INFO] Ground plane mode=y0, top surface at y=0.0")
+
+ def build_ground_plane_pose(up_direction, surface_level, half_thickness=0.01):
+ up_direction = normalized_vector(up_direction)
+ reference = np.array([1.0, 0.0, 0.0], dtype=np.float32)
+ if abs(float(np.dot(up_direction, reference))) > 0.9:
+ reference = np.array([0.0, 0.0, 1.0], dtype=np.float32)
+ x_axis = normalized_vector(np.cross(reference, up_direction))
+ z_axis = normalized_vector(np.cross(x_axis, up_direction))
+ rotation = np.column_stack([x_axis, up_direction, z_axis])
+ return up_direction * (float(surface_level) - float(half_thickness)), rotmat_to_quat_wxyz(rotation)
+
+ # Freeze ground plane pose at initialization so it stays fixed for all steps/phases.
+ fixed_ground_pose_p, fixed_ground_pose_q = build_ground_plane_pose(
+ ground_up_direction,
+ ground_surface_level,
+ )
+
+ def build_scene_with_actors(gravity_xyz, pose_overrides=None, lock_rotation=False):
+ scene_config = sapien.SceneConfig()
+ scene_config.gravity = list(map(float, np.asarray(gravity_xyz, dtype=np.float32).tolist()))
+ scene = create_scene_compatible(engine=engine, scene_config=scene_config, cpu_only=cpu_only)
+ scene.set_timestep(float(timestep))
+ if viewer is not None:
+ setup_scene_lighting(scene)
+ bind_viewer_scene(scene)
+
+ # Keep contacts stable and make the behavior closer to the old backend.
+ if hasattr(scene, "set_solver_iterations"):
+ scene.set_solver_iterations(int(solver_iterations))
+
+ physical_material = scene.create_physical_material(
+ static_friction=0.3,
+ dynamic_friction=0.3,
+ restitution=0.0,
+ )
+
+ # Ground plane aligned with the chosen up direction using a thin kinematic box.
+ ground_builder = scene.create_actor_builder()
+ ground_builder.add_box_collision(
+ half_size=[1000.0, 0.01, 1000.0],
+ pose=sapien.Pose(p=fixed_ground_pose_p.tolist(), q=fixed_ground_pose_q.tolist()),
+ material=physical_material,
+ )
+ if viewer is not None and hasattr(ground_builder, "add_box_visual"):
+ try:
+ ground_builder.add_box_visual(
+ half_size=[1000.0, 0.01, 1000.0],
+ pose=sapien.Pose(p=fixed_ground_pose_p.tolist(), q=fixed_ground_pose_q.tolist()),
+ color=[0.2, 0.24, 0.28],
+ )
+ except TypeError:
+ try:
+ ground_builder.add_box_visual(
+ half_size=[1000.0, 0.01, 1000.0],
+ pose=sapien.Pose(p=fixed_ground_pose_p.tolist(), q=fixed_ground_pose_q.tolist()),
+ )
+ except TypeError:
+ ground_builder.add_box_visual([1000.0, 0.01, 1000.0])
+ ground_builder.build_kinematic(name="ground_plane")
+
+ actor_to_data = {}
+ prev_pose_by_actor = {}
+ for object_id in object_ids:
+ layout_path = layout_dir / f"{normalize_object_id(object_id)}.json"
+ hull_path = hull_dir / f"{normalize_object_id(object_id)}.obj"
+
+ raw_layout, translation, rotation_wxyz, scale = load_layout(layout_path)
+ if pose_overrides is not None and object_id in pose_overrides:
+ translation = np.asarray(pose_overrides[object_id]["p"], dtype=np.float32)
+ rotation_wxyz = parse_rotation_wxyz(pose_overrides[object_id]["q"])
+
+ position_xyz, orientation_wxyz = layout_pose_to_sapien(
+ translation=translation,
+ rotation_wxyz=rotation_wxyz,
+ )
+ if init_non_ground_y_offset and object_id not in selected_ground_object_ids:
+ position_xyz = np.asarray(position_xyz, dtype=np.float32)
+ position_xyz[1] += initial_y_offset
+
+ builder = scene.create_actor_builder()
+ if viewer is not None:
+ add_visual_mesh_compatible(builder, hull_path, scale)
+ if hasattr(builder, "add_multiple_convex_collisions_from_file"):
+ builder.add_multiple_convex_collisions_from_file(
+ filename=str(hull_path),
+ pose=sapien.Pose(),
+ scale=scale.tolist(),
+ material=physical_material,
+ density=max(float(mass), 1e-6),
+ )
+ else:
+ raise RuntimeError(
+ "No supported mesh collision API found on ActorBuilder."
+ )
+
+ if object_id in selected_ground_object_ids:
+ actor = builder.build_kinematic(name=normalize_object_id(object_id))
+ else:
+ actor = builder.build(name=normalize_object_id(object_id))
+ if hasattr(actor, "set_damping"):
+ actor.set_damping(float(linear_damping), float(angular_damping))
+
+ actor.set_pose(sapien.Pose(p=position_xyz, q=orientation_wxyz))
+ if lock_rotation:
+ physx_component = actor.find_component_by_type(sapien.physx.PhysxRigidDynamicComponent)
+ if physx_component:
+ physx_component.set_locked_motion_axes([False, False, False, True, True, True])
+
+ actor_to_data[actor] = {
+ "object_id": object_id,
+ "layout": raw_layout,
+ "scale": scale,
+ }
+ prev_pose_by_actor[actor] = {
+ "p": np.asarray(position_xyz, dtype=np.float32),
+ "q": parse_rotation_wxyz(orientation_wxyz),
+ }
+
+ return scene, actor_to_data, prev_pose_by_actor
+
+ pose_delta_records = []
+
+ def record_pose_deltas(phase: str, step_index: int, actor_to_data, prev_pose_by_actor):
+ if pose_delta_output is None:
+ return
+
+ frame_records = []
+ for actor, data in actor_to_data.items():
+ pose = actor.get_pose()
+ curr_p = np.asarray(pose.p, dtype=np.float32)
+ curr_q = parse_rotation_wxyz(pose.q)
+ prev_state = prev_pose_by_actor[actor]
+
+ delta_p = curr_p - prev_state["p"]
+ delta_q = quat_delta_wxyz(prev_state["q"], curr_q)
+ delta_angle_deg = quat_delta_angle_deg(prev_state["q"], curr_q)
+
+ frame_records.append(
+ {
+ "object_id": data["object_id"],
+ "translation_xyz": list(map(float, curr_p.tolist())),
+ "rotation_wxyz": list(map(float, curr_q.tolist())),
+ "delta_translation_xyz": list(map(float, delta_p.tolist())),
+ "delta_rotation_wxyz": list(map(float, delta_q.tolist())),
+ "delta_rotation_angle_deg": delta_angle_deg,
+ }
+ )
+
+ prev_state["p"] = curr_p
+ prev_state["q"] = curr_q
+
+ pose_delta_records.append(
+ {
+ "phase": phase,
+ "step": int(step_index),
+ "objects": frame_records,
+ }
+ )
+
+ zero_gravity = np.array([0.0, 0.0, 0.0], dtype=np.float32)
+ simulation_gravity = np.asarray(gravity_xyz, dtype=np.float32).reshape(3)
+ # Clamp velocity across the whole pipeline. Y is allowed to be faster than X/Z in the legacy default,
+ # but the clip is component-wise so custom gravity directions still work.
+ max_abs_velocity_xyz = np.array([5e-5, 2e-2, 5e-5], dtype=np.float32)
+ # Clamp angular velocity (rad/s) across the whole pipeline.
+ max_abs_angular_velocity_xyz = np.array([5e-5, 5e-5, 5e-5], dtype=np.float32)
+
+ actor_to_data = {}
+ prev_pose_by_actor = {}
+ stop_requested = False
+ run_continuous = not gui_manual_enabled
+ manual_key_supported = False
+
+ if viewer is not None and gui_manual_enabled and hasattr(viewer, "window"):
+ window = viewer.window
+ manual_key_supported = any(
+ hasattr(window, name)
+ for name in ["key_down", "is_key_down", "key_press", "is_key_pressed"]
+ )
+ if not manual_key_supported:
+ print(
+ "[WARN] Viewer key polling API is unavailable in this SAPIEN build; "
+ "falling back to continuous stepping."
+ )
+ run_continuous = True
+
+ def after_step_render(scene):
+ nonlocal stop_requested
+ if viewer is None:
+ return
+ if hasattr(scene, "update_render"):
+ scene.update_render()
+ if hasattr(viewer, "render"):
+ viewer.render()
+ if hasattr(viewer, "closed") and viewer.closed:
+ stop_requested = True
+
+ def process_manual_controls():
+ nonlocal run_continuous, stop_requested
+ if (viewer is None) or (not gui_manual_enabled) or (not manual_key_supported):
+ return False
+
+ window = viewer.window
+ if _poll_viewer_key(window, "q") or _poll_viewer_key(window, "esc"):
+ stop_requested = True
+ return False
+
+ if _poll_viewer_key(window, "space"):
+ run_continuous = not run_continuous
+
+ if _poll_viewer_key(window, "r"):
+ run_continuous = False
+
+ if run_continuous:
+ return True
+
+ return _poll_viewer_key(window, "n")
+
+ def step_scene_with_gui(scene, phase: str, steps: int, actor_to_data, prev_pose_by_actor):
+ nonlocal stop_requested
+ if steps / 50 == 0:
+ print(f"[INFO] {phase} step 0/{max(steps, 0)}")
+ record_pose_deltas(phase, 0, actor_to_data, prev_pose_by_actor)
+
+ applied_steps = 0
+ while applied_steps < max(steps, 0):
+ do_step = True
+ if gui_manual_enabled:
+ do_step = process_manual_controls()
+ if stop_requested:
+ break
+
+ if do_step:
+ scene.step()
+ limit_dynamic_actor_velocities(
+ actor_to_data=actor_to_data,
+ selected_ground_object_ids=selected_ground_object_ids,
+ max_abs_velocity_xyz=max_abs_velocity_xyz,
+ max_abs_angular_velocity_xyz=max_abs_angular_velocity_xyz,
+ )
+ applied_steps += 1
+ print(f"[INFO] {phase} step {applied_steps}/{max(steps, 0)}")
+ record_pose_deltas(
+ phase,
+ applied_steps,
+ actor_to_data,
+ prev_pose_by_actor,
+ )
+
+ after_step_render(scene)
+
+ if stop_requested:
+ break
+
+ return applied_steps
+
+ if depenetration_steps > 0:
+ print(f"[INFO] Depenetration gravity = {zero_gravity.tolist()}")
+ scene, actor_to_data, prev_pose_by_actor = build_scene_with_actors(
+ zero_gravity,
+ lock_rotation=True,
+ )
+ step_scene_with_gui(
+ scene,
+ phase="depenetration",
+ steps=depenetration_steps,
+ actor_to_data=actor_to_data,
+ prev_pose_by_actor=prev_pose_by_actor,
+ )
+ if stop_requested:
+ print("[INFO] Viewer closed by user during depenetration.")
+
+ if (simulation_steps > 0) and (not stop_requested):
+ start_poses = {}
+ for actor, data in actor_to_data.items():
+ pose = actor.get_pose()
+ start_poses[data["object_id"]] = {
+ "p": np.asarray(pose.p, dtype=np.float32),
+ "q": parse_rotation_wxyz(pose.q),
+ }
+
+ print(f"[INFO] Simulation gravity = {simulation_gravity.tolist()}")
+ scene, actor_to_data, prev_pose_by_actor = build_scene_with_actors(
+ simulation_gravity,
+ pose_overrides=start_poses,
+ lock_rotation=lock_rotation_all,
+ )
+ step_scene_with_gui(
+ scene,
+ phase="simulation",
+ steps=simulation_steps,
+ actor_to_data=actor_to_data,
+ prev_pose_by_actor=prev_pose_by_actor,
+ )
+ if stop_requested:
+ print("[INFO] Viewer closed by user during simulation.")
+ else:
+ print(f"[INFO] Simulation gravity = {simulation_gravity.tolist()}")
+ scene, actor_to_data, prev_pose_by_actor = build_scene_with_actors(
+ simulation_gravity,
+ lock_rotation=lock_rotation_all,
+ )
+ step_scene_with_gui(
+ scene,
+ phase="simulation",
+ steps=simulation_steps,
+ actor_to_data=actor_to_data,
+ prev_pose_by_actor=prev_pose_by_actor,
+ )
+ if stop_requested:
+ print("[INFO] Viewer closed by user during simulation.")
+
+ output_layout_dir.mkdir(parents=True, exist_ok=True)
+ for actor, data in actor_to_data.items():
+ pose = actor.get_pose()
+
+ updated_layout = dict(data["layout"])
+ translation_p3d, rotation_p3d = sapien_pose_to_layout(
+ translation_xyz=pose.p,
+ rotation_wxyz=pose.q,
+ )
+ updated_layout["translation"] = translation_p3d
+ if lock_rotation_all:
+ updated_layout["rotation"] = data["layout"]["rotation"]
+ else:
+ updated_layout["rotation"] = rotation_p3d
+ updated_layout["scale"] = [list(map(float, data["scale"]))]
+
+ out_path = output_layout_dir / f"{normalize_object_id(data['object_id'])}.json"
+ with out_path.open("w", encoding="utf-8") as f:
+ json.dump(updated_layout, f, ensure_ascii=False, indent=2)
+ print(f"[OK] Saved simulated pose: {out_path}")
+
+ if pose_delta_output is not None:
+ pose_delta_output.parent.mkdir(parents=True, exist_ok=True)
+ with pose_delta_output.open("w", encoding="utf-8") as f:
+ json.dump(pose_delta_records, f, ensure_ascii=False, indent=2)
+ print(f"[OK] Saved per-step pose deltas: {pose_delta_output}")
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description=(
+ "Run simple rigid-body simulation with collision hulls and save updated "
+ "object_XXX pose JSON files."
+ )
+ )
+ parser.add_argument(
+ "--layout_dir",
+ type=Path,
+ default=Path("outputs/layout_params"),
+ help="Directory containing object_XXX.json layout files.",
+ )
+ parser.add_argument(
+ "--hull_dir",
+ type=Path,
+ default=Path("outputs/collision_hulls"),
+ help="Directory containing object_XXX.obj collision hull files.",
+ )
+ parser.add_argument(
+ "--output_layout_dir",
+ type=Path,
+ default=Path("outputs/layout_params_simulated"),
+ help="Directory to save updated object_XXX.json files.",
+ )
+ parser.add_argument(
+ "--gravity",
+ type=float,
+ nargs="+",
+ default=[2.0],
+ help=(
+ "Gravity magnitude or vector. Pass one value for legacy -Y gravity, "
+ "or three values for an arbitrary gravity vector in world XYZ."
+ ),
+ )
+ parser.add_argument(
+ "--timestep",
+ type=float,
+ default=1.0 / 100.0,
+ help="Simulation timestep in seconds.",
+ )
+ parser.add_argument(
+ "--depenetration_steps",
+ type=int,
+ default=200,
+ help="Steps with zero gravity to resolve initial penetrations.",
+ )
+ parser.add_argument(
+ "--simulation_steps",
+ type=int,
+ default=2000,
+ help="Steps with gravity enabled.",
+ )
+ parser.add_argument(
+ "--mass",
+ type=float,
+ default=10.0,
+ help="Density proxy for each object body in SAPIEN collision loading.",
+ )
+ parser.add_argument(
+ "--linear_damping",
+ type=float,
+ default=5.0,
+ help="Linear damping for each object.",
+ )
+ parser.add_argument(
+ "--angular_damping",
+ type=float,
+ default=5.0,
+ help="Angular damping for each object.",
+ )
+ parser.add_argument(
+ "--solver_iterations",
+ type=int,
+ default=100,
+ help="Constraint solver iterations.",
+ )
+ parser.add_argument(
+ "--use_ground_object",
+ action="store_true",
+ help=(
+ "If set, use a static object as ground anchor to reduce global drift. "
+ "By default, the lowest-Y object is chosen (PyTorch3D coordinate system)."
+ ),
+ )
+ parser.add_argument(
+ "--ground_object_id",
+ type=str,
+ nargs="+",
+ default=None,
+ help=(
+ "One or more object ids (XXX in object_XXX) to use as static ground objects. "
+ "Only valid when --use_ground_object is enabled."
+ ),
+ )
+ parser.add_argument(
+ "--ground_plane_mode",
+ type=str,
+ choices=["bbox", "y0"],
+ default="bbox",
+ help=(
+ "Ground plane placement mode: bbox aligns the plane to the gravity direction "
+ "and uses the transformed collision-hull bbox; y0 keeps the legacy fixed y=0 plane."
+ ),
+ )
+ parser.add_argument(
+ "--cpu_only",
+ action="store_true",
+ help=(
+ "Create a pure-physics SAPIEN scene without RenderSystem. "
+ "Useful on machines without graphics/rendering devices."
+ ),
+ )
+ parser.add_argument(
+ "--gui",
+ action="store_true",
+ help=(
+ "Enable SAPIEN GUI viewer when renderer/device is available; "
+ "otherwise automatically fall back to headless mode."
+ ),
+ )
+ parser.add_argument(
+ "--gui_manual_control",
+ action="store_true",
+ help=(
+ "GUI only: start paused on first frame and manually control stepping "
+ "with keyboard (N single-step, Space run/pause, R pause, Q/Esc quit)."
+ ),
+ )
+ parser.add_argument(
+ "--pose_delta_output",
+ type=Path,
+ default=None,
+ help=(
+ "If set, record per-step translation/rotation changes for each object "
+ "and save to this JSON file."
+ ),
+ )
+ parser.add_argument(
+ "--lock_rotation_all",
+ action="store_true",
+ help="Lock rotation for all objects and only update translations in output.",
+ )
+ parser.add_argument(
+ "--object_ids",
+ nargs="+",
+ default=None,
+ help=(
+ "Optional subset of object ids to simulate (XXX in object_XXX). "
+ "Defaults to all matched objects."
+ ),
+ )
+ parser.add_argument(
+ "--verify_only",
+ action="store_true",
+ help="Only verify loading of layout and hull files, without running simulation.",
+ )
+ parser.add_argument(
+ "--init_non_ground_y_offset",
+ action="store_true",
+ help="Shift non-ground objects upward before simulation by a hull-size-based offset.",
+ )
+ args = parser.parse_args()
+
+ if not args.layout_dir.exists():
+ raise FileNotFoundError(f"Layout directory not found: {args.layout_dir}")
+ if not args.hull_dir.exists():
+ raise FileNotFoundError(f"Hull directory not found: {args.hull_dir}")
+ if args.mass <= 0:
+ raise ValueError("--mass must be positive.")
+ if args.timestep <= 0:
+ raise ValueError("--timestep must be positive.")
+ if args.ground_object_id is not None and not args.use_ground_object:
+ raise ValueError("--ground_object_id requires --use_ground_object.")
+
+ run_simulation(
+ layout_dir=args.layout_dir,
+ hull_dir=args.hull_dir,
+ output_layout_dir=args.output_layout_dir,
+ gravity=args.gravity,
+ timestep=args.timestep,
+ depenetration_steps=args.depenetration_steps,
+ simulation_steps=args.simulation_steps,
+ mass=args.mass,
+ object_ids=args.object_ids,
+ linear_damping=args.linear_damping,
+ angular_damping=args.angular_damping,
+ solver_iterations=args.solver_iterations,
+ use_ground_object=args.use_ground_object,
+ ground_object_ids=args.ground_object_id,
+ ground_plane_mode=args.ground_plane_mode,
+ gui=args.gui,
+ gui_manual_control=args.gui_manual_control,
+ pose_delta_output=args.pose_delta_output,
+ verify_only=args.verify_only,
+ lock_rotation_all=args.lock_rotation_all,
+ cpu_only=args.cpu_only,
+ init_non_ground_y_offset=args.init_non_ground_y_offset,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/evolution/types.py b/vlm_agent/evolution/types.py
new file mode 100644
index 0000000..d313c56
--- /dev/null
+++ b/vlm_agent/evolution/types.py
@@ -0,0 +1,49 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Literal
+
+
+@dataclass(frozen=True)
+class SceneArtifacts:
+ unposed_objects_dir: Path | None = None
+ layout_params_dir: Path | None = None
+ collision_hulls_dir: Path | None = None
+ composed_scene_mesh: Path | None = None
+ simulated_layout_dir: Path | None = None
+ metadata: dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass(frozen=True)
+class OptimizationAction:
+ action_type: Literal[
+ "compose_scene",
+ "build_collision_hulls",
+ "simulate_scene_physics",
+ "adjust_layout",
+ "request_visual_feedback",
+ "export_artifacts",
+ ]
+ target_object_id: str | None = None
+ parameters: dict[str, Any] = field(default_factory=dict)
+ rationale: str = ""
+
+
+@dataclass(frozen=True)
+class OptimizationReport:
+ round_index: int
+ prompt: str
+ model_response: str
+ actions: list[OptimizationAction]
+ tool_outputs: dict[str, str] = field(default_factory=dict)
+
+
+@dataclass(frozen=True)
+class PipelineReport:
+ """Overall report for the active pipeline."""
+ stage: Literal["optimization", "generation", "composition"]
+ artifacts: SceneArtifacts
+ vlm_prompts: list[str] = field(default_factory=list)
+ vlm_responses: list[str] = field(default_factory=list)
+ tool_outputs: dict[str, str] = field(default_factory=dict)
+ timestamp: str = ""
diff --git a/vlm_agent/evolution/vlm_client.py b/vlm_agent/evolution/vlm_client.py
new file mode 100644
index 0000000..fb53fd9
--- /dev/null
+++ b/vlm_agent/evolution/vlm_client.py
@@ -0,0 +1,187 @@
+from __future__ import annotations
+
+import base64
+import json
+import mimetypes
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Sequence
+
+from openai import OpenAI
+
+
+class VLMClient:
+ def __init__(
+ self,
+ url: str | None,
+ model: str,
+ api_key: str,
+ timeout_s: float = 60.0,
+ headers: dict[str, str] | None = None,
+ ) -> None:
+ self.url = url
+ self.model = model
+ self.timeout_s = timeout_s
+ self.headers = headers
+ client_kwargs: dict[str, object] = {
+ "api_key": api_key,
+ "timeout": timeout_s,
+ }
+ if url:
+ client_kwargs["base_url"] = url
+ if headers:
+ client_kwargs["default_headers"] = headers
+ self.client = OpenAI(**client_kwargs)
+ self.last_usage: dict[str, int] | None = None
+
+ def plan(self, prompt: str, images: Sequence[str] | None = None) -> str:
+ content: list[dict[str, object]] = [{"type": "text", "text": prompt}]
+ for data_url in self._encode_images(images):
+ content.append({"type": "image_url", "image_url": {"url": data_url}})
+
+ response = self.client.chat.completions.create(
+ model=self.model,
+ messages=[{"role": "user", "content": content}],
+ )
+ self.last_usage = self._extract_usage(response)
+ return self._extract_response_text(response)
+
+ def extract_json_payload(self, text: str) -> dict:
+ return self._extract_json_payload(text)
+
+ @staticmethod
+ def _encode_images(images: Sequence[str] | None) -> list[str]:
+ if not images:
+ return []
+ encoded: list[str] = []
+ for image in images:
+ image_path = Path(image)
+ if not image_path.exists():
+ continue
+ mime_type, _ = mimetypes.guess_type(image_path.name)
+ if mime_type is None:
+ mime_type = "application/octet-stream"
+ data = base64.b64encode(image_path.read_bytes()).decode("ascii")
+ encoded.append(f"data:{mime_type};base64,{data}")
+ return encoded
+
+ @staticmethod
+ def _extract_response_text(response: object) -> str:
+ output_text = getattr(response, "output_text", None)
+ if isinstance(output_text, str) and output_text:
+ return output_text
+
+ choices = getattr(response, "choices", None)
+ if choices:
+ first_choice = choices[0]
+ message = getattr(first_choice, "message", None)
+ if message is not None:
+ content = getattr(message, "content", None)
+ if isinstance(content, str):
+ return content
+
+ if hasattr(response, "model_dump"):
+ return json.dumps(response.model_dump(), ensure_ascii=False)
+ return str(response)
+
+ @staticmethod
+ def _extract_usage(response: object) -> dict[str, int]:
+ usage = getattr(response, "usage", None)
+ if usage is not None:
+ if hasattr(usage, "model_dump"):
+ usage_dict = usage.model_dump()
+ elif isinstance(usage, dict):
+ usage_dict = usage
+ else:
+ usage_dict = {
+ "prompt_tokens": getattr(usage, "prompt_tokens", None),
+ "completion_tokens": getattr(usage, "completion_tokens", None),
+ "total_tokens": getattr(usage, "total_tokens", None),
+ }
+ normalized_usage: dict[str, int] = {}
+ for key, value in usage_dict.items():
+ if value is None:
+ continue
+ if isinstance(value, dict):
+ nested_value = value.get("value")
+ if nested_value is None:
+ nested_value = value.get("total")
+ if nested_value is None:
+ continue
+ value = nested_value
+ try:
+ normalized_usage[key] = int(value)
+ except (TypeError, ValueError):
+ continue
+ return normalized_usage
+
+ if hasattr(response, "model_dump"):
+ response_dict = response.model_dump()
+ usage_dict = response_dict.get("usage")
+ if isinstance(usage_dict, dict):
+ return {
+ key: int(value)
+ for key, value in usage_dict.items()
+ if value is not None
+ }
+
+ return {}
+
+ @staticmethod
+ def _extract_json_payload(text: str) -> dict:
+ if not text:
+ return {}
+ try:
+ return json.loads(text)
+ except json.JSONDecodeError:
+ pass
+
+ fenced_match = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", text, re.DOTALL)
+ if fenced_match:
+ candidate = fenced_match.group(1)
+ try:
+ return json.loads(candidate)
+ except json.JSONDecodeError:
+ pass
+
+ start = text.find("{")
+ if start != -1:
+ depth = 0
+ for idx in range(start, len(text)):
+ char = text[idx]
+ if char == "{":
+ depth += 1
+ elif char == "}":
+ depth -= 1
+ if depth == 0:
+ candidate = text[start:idx + 1]
+ try:
+ return json.loads(candidate)
+ except json.JSONDecodeError:
+ break
+
+ return {"raw_response": text}
+
+
+@dataclass(frozen=True)
+class DummyVLMClient:
+ default_action: str = "{\"actions\": []}"
+
+ def plan(self, prompt: str, images: Sequence[str] | None = None) -> str:
+ del prompt, images
+ return self.default_action
+
+ def extract_json_payload(self, text: str) -> dict:
+ return VLMClient._extract_json_payload(text)
+
+
+def parse_vlm_json(text: str) -> dict:
+ try:
+ payload = json.loads(text)
+ except json.JSONDecodeError:
+ return {"actions": [], "raw_response": text}
+ if not isinstance(payload, dict):
+ return {"actions": [], "raw_response": text}
+ payload.setdefault("actions", [])
+ return payload
diff --git a/vlm_agent/pipeline.py b/vlm_agent/pipeline.py
new file mode 100644
index 0000000..6e2a2ed
--- /dev/null
+++ b/vlm_agent/pipeline.py
@@ -0,0 +1,1073 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import uuid
+from dataclasses import dataclass, replace
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+from .evolution.vlm_client import VLMClient
+from .reconstruction.perception.stage import (
+ PerceptionStage,
+ PerceptionStageError,
+)
+from .evolution.evolution_stage import (
+ IntegratedPipelineOrchestrator,
+ IntegratedToolRunner,
+ SceneMosaicArtifactError,
+ SceneMosaicStageAdapter,
+ SceneMosaicStageArtifacts,
+)
+from .evolution.paths import ProjectPaths
+from .reconstruction.relations.stage import (
+ RelationArtifacts,
+ RelationStageAdapter,
+ RelationStageError,
+)
+from .state import RunState, RunStateError
+from .config import RuntimeConfig
+from .reconstruction.sam3d_stage import (
+ Sam3DStage,
+ Sam3DStageError,
+ validate_stage05,
+)
+from .artifacts import SceneRun, initialize_scene_run, load_scene_run
+
+
+class FullPipelineError(RuntimeError):
+ exit_code = 2
+
+
+class FullPipelineChildError(FullPipelineError):
+ exit_code = 3
+
+
+class FullPipelineIntegrityError(FullPipelineError):
+ exit_code = 4
+
+
+@dataclass(frozen=True)
+class FullRunOptions:
+ input_image: Path | None = None
+ scene_name: str | None = None
+ scene_root: Path | None = None
+ run_root: Path | None = None
+ perception_reuse_local_pre_pass_b_cache_root: Path | None = None
+ perception_pass_b_reasoning_effort: str = "none"
+
+
+def _utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _atomic_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_name(f".{path.name}.{uuid.uuid4().hex}.tmp")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _optional_url(value: str | None) -> str | None:
+ if value is None:
+ return None
+ normalized = value.strip()
+ return normalized if normalized and normalized != "None" else None
+
+
+class FullSceneOrchestrator:
+
+ def __init__(self, runtime: RuntimeConfig) -> None:
+ self.runtime = runtime
+
+ def verify(self, *, require_gpu: bool = False) -> dict[str, Any]:
+ return self.runtime.verify(require_gpu=require_gpu)
+
+ def _scene(self, options: FullRunOptions) -> SceneRun:
+ starting = options.input_image is not None or options.scene_name is not None
+ resuming = options.scene_root is not None
+ if starting == resuming:
+ raise FullPipelineError(
+ "run requires either --input-image with --scene-name, or --scene-root"
+ )
+ if starting:
+ if options.input_image is None or not options.scene_name:
+ raise FullPipelineError(
+ "new run requires both input_image and scene_name"
+ )
+ return initialize_scene_run(
+ self.runtime,
+ scene_name=options.scene_name,
+ input_image=options.input_image,
+ run_root=options.run_root,
+ )
+ assert options.scene_root is not None
+ if options.run_root is not None:
+ raise FullPipelineError("run_root is valid only for a new scene")
+ return load_scene_run(options.scene_root)
+
+ def _relation_adapter(self, scene: SceneRun) -> RelationStageAdapter:
+ return RelationStageAdapter(
+ scene_config_path=scene.contract.scene_config_path,
+ relation_package_root=self.runtime.relation_package_root,
+ renderer_script=self.runtime.renderer_path,
+ )
+
+ def _aggregate_audit(self, scene: SceneRun) -> Path:
+ state = scene.state.public_status()
+ stages = state.get("stages", {})
+ payload: dict[str, Any] = {
+ "schema_version": "scenemosaic_full_pipeline_audit",
+ "generated_at_utc": _utc_now(),
+ "scene_name": scene.contract.scene_name,
+ "scene_root": str(scene.contract.run_root),
+ "vlm_model": self.runtime.vlm_model,
+ "current_state": state.get("current_state"),
+ "last_successful_state": state.get("last_successful_state"),
+ "next_action": state.get("next_action"),
+ "failure": state.get("failure"),
+ "releases": {
+ "perception": "scenemosaic-perception",
+ "relation": "scenemosaic-relations",
+ "sam3d_source": "SceneMosaic/sam-3d-objects",
+ },
+ "stages": stages,
+ "stage_attempts": state.get("stage_attempts", {}),
+ "invocations": state.get("invocations", []),
+ }
+
+ path = scene.contract.run_root / "audit" / "pipeline_audit.json"
+ _atomic_json(path, payload)
+ return path
+
+ def _prepared_stage08(
+ self,
+ scene: SceneRun,
+ relation: RelationArtifacts,
+ adapter: SceneMosaicStageAdapter,
+ ) -> SceneMosaicStageArtifacts | None:
+ record = scene.state.data.get("stages", {}).get("stage08")
+ if not isinstance(record, dict) or record.get("status") != "prepared":
+ return None
+ facts = record.get("facts")
+ if not isinstance(facts, dict):
+ raise FullPipelineIntegrityError("Stage 08 state facts are missing")
+ stage_id = str(facts.get("stage_id") or "")
+ workspace = Path(str(facts.get("workspace_dir") or "")).resolve()
+ manifest = Path(str(facts.get("integration_manifest") or "")).resolve()
+ expected = adapter.attempt_workspace(stage_id)
+ if workspace != expected or manifest != workspace / "integration_manifest.json":
+ raise FullPipelineIntegrityError("Stage 08 state paths are inconsistent")
+ if not workspace.is_dir() or not manifest.is_file():
+ raise FullPipelineIntegrityError("prepared Stage 08 workspace is missing")
+ manifest_payload = json.loads(manifest.read_text(encoding="utf-8"))
+ if manifest_payload.get("relation_publication_id") != relation.publication_id:
+ raise FullPipelineIntegrityError(
+ "prepared Stage 08 uses a different relation publication"
+ )
+ verification = adapter.verify_upstream_unchanged(manifest)
+ if verification.get("valid") is not True:
+ raise FullPipelineIntegrityError(
+ "prepared Stage 08 upstream hashes changed"
+ )
+ working_copies = adapter.verify_prepared_working_copies(manifest)
+ if working_copies.get("valid") is not True:
+ raise FullPipelineIntegrityError(
+ "prepared Stage 08 working-copy hashes changed"
+ )
+ if (workspace / "pipeline_report.json").exists():
+ raise FullPipelineIntegrityError(
+ "prepared Stage 08 already started and cannot be resumed in place"
+ )
+ return SceneMosaicStageArtifacts(
+ stage_id=stage_id,
+ workspace_dir=workspace,
+ integration_manifest_path=manifest,
+ relation_artifacts=relation,
+ )
+
+ def _validate_completed_scene(
+ self,
+ scene: SceneRun,
+ ) -> dict[str, Any]:
+ runtime_report = self.verify(require_gpu=False)
+ if runtime_report.get("valid") is not True:
+ raise FullPipelineIntegrityError(
+ "completed run no longer passes runtime verification"
+ )
+ perception = PerceptionStage(
+ self.runtime, scene.contract.scene_config_path
+ ).reuse_formal()
+ if perception is None:
+ raise FullPipelineIntegrityError(
+ "completed run has no valid formal perception publication"
+ )
+ sam3d = validate_stage05(
+ scene.contract.scene_config_path,
+ project_root=self.runtime.project_root,
+ )
+ relation = self._relation_adapter(scene).reuse()
+ adapter = SceneMosaicStageAdapter(
+ scene_config_path=scene.contract.scene_config_path,
+ relation_artifacts=relation,
+ )
+ record = scene.state.data.get("stages", {}).get("stage08")
+ if (
+ not isinstance(record, dict)
+ or record.get("status") != "published"
+ or not isinstance(record.get("facts"), dict)
+ ):
+ raise FullPipelineIntegrityError(
+ "completed run has no published Stage 08 state facts"
+ )
+ facts = record["facts"]
+ stage_id = str(facts.get("stage_id") or "")
+ if not stage_id:
+ raise FullPipelineIntegrityError("completed Stage 08 state has no stage_id")
+ workspace = Path(str(facts.get("workspace_dir") or "")).expanduser().resolve()
+ integration = (
+ Path(str(facts.get("integration_manifest") or "")).expanduser().resolve()
+ )
+ publication = (
+ Path(str(facts.get("publication_manifest") or "")).expanduser().resolve()
+ )
+ expected_workspace = adapter.attempt_workspace(stage_id).resolve()
+ if (
+ workspace != expected_workspace
+ or integration != workspace / "integration_manifest.json"
+ or not integration.is_file()
+ or integration.is_symlink()
+ or facts.get("relation_publication_id") != relation.publication_id
+ or facts.get("wall_padding") != 0.0
+ ):
+ raise FullPipelineIntegrityError(
+ "completed Stage 08 state paths or invariants are inconsistent"
+ )
+ upstream = adapter.verify_upstream_unchanged(integration)
+ if upstream.get("valid") is not True:
+ raise FullPipelineIntegrityError(
+ "completed Stage 08 upstream facts changed"
+ )
+ publication_facts = adapter.validate_publication(publication)
+ expected_sha = facts.get("publication_manifest_sha256")
+ if (
+ publication_facts.get("stage_id") != stage_id
+ or not isinstance(expected_sha, str)
+ or publication_facts.get("publication_manifest_sha256") != expected_sha
+ ):
+ raise FullPipelineIntegrityError(
+ "completed Stage 08 publication no longer matches pipeline state"
+ )
+ return {
+ "runtime": runtime_report,
+ "perception": {
+ "delivery_status": (perception.artifacts.delivery_status),
+ "object_ids": list(perception.artifacts.object_ids),
+ "run_manifest_sha256": (perception.run_manifest_sha256),
+ },
+ "sam3d": {
+ "object_ids": list(sam3d.object_ids),
+ "source_manifest_sha256": (sam3d.source_manifest_sha256),
+ },
+ "relation": {
+ "publication_id": relation.publication_id,
+ "publication_manifest_sha256": _sha256(
+ relation.publication_manifest_path
+ ),
+ },
+ "stage08": publication_facts,
+ }
+
+ def run(
+ self,
+ options: FullRunOptions,
+ *,
+ sanitized_argv: Sequence[str] = (),
+ environ: Mapping[str, str] | None = None,
+ ) -> dict[str, Any]:
+ scene = self._scene(options)
+ state = scene.state
+ try:
+ scene_lock = state.acquire_lock(exclusive=True, blocking=False)
+ except RunStateError as exc:
+ raise FullPipelineIntegrityError(str(exc)) from exc
+ state.close_stale_invocations()
+ invocation_id = state.begin_invocation(
+ command="run", sanitized_argv=list(sanitized_argv)
+ )
+ try:
+ if state.current_state == "scenemosaic_published":
+ verification = self._validate_completed_scene(scene)
+ publication = verification["stage08"]["publication_manifest"]
+ state.finish_invocation(invocation_id, result="complete_reuse")
+ return {
+ "complete": True,
+ "status": "scenemosaic_published",
+ "scene_root": str(scene.contract.run_root),
+ "publication_manifest": publication,
+ "reused": True,
+ "fact_verification": verification,
+ "pipeline_audit": str(self._aggregate_audit(scene)),
+ }
+ runtime_report = self.verify(require_gpu=False)
+ if not runtime_report.get("valid"):
+ raise FullPipelineIntegrityError(
+ "runtime preflight failed; run python -m vlm_agent.cli verify for details"
+ )
+
+ perception = PerceptionStage(self.runtime, scene.contract.scene_config_path)
+
+ perception_result = perception.run(
+ approve_gpu=True,
+ approve_api=True,
+ approve_third_party_upload=True,
+ reuse_local_pre_pass_b_cache_root=(
+ options.perception_reuse_local_pre_pass_b_cache_root
+ ),
+ pass_b_reasoning_effort=(
+ options.perception_pass_b_reasoning_effort
+ ),
+ environ=environ,
+ )
+ state.record_stage(
+ "perception",
+ status="published",
+ successful_state="stage04_published",
+ facts={
+ "delivery_status": perception_result.artifacts.delivery_status,
+ "object_ids": list(perception_result.artifacts.object_ids),
+ "run_manifest": str(perception_result.run_manifest_path),
+ "run_manifest_sha256": perception_result.run_manifest_sha256,
+ "stage03": str(perception_result.artifacts.stage03_dir),
+ "stage04": str(perception_result.artifacts.stage04_dir),
+ "subprocess_started": perception_result.subprocess_started,
+ "audit": str(perception_result.audit_path)
+ if perception_result.audit_path
+ else None,
+ },
+ )
+
+ sam3d = Sam3DStage(
+ scene.contract.scene_config_path,
+ runtime_config=self.runtime,
+ )
+ try:
+ sam3d_artifacts = validate_stage05(
+ scene.contract.scene_config_path,
+ project_root=self.runtime.project_root,
+ )
+ sam3d_artifacts = replace(sam3d_artifacts, reused=True)
+ except Sam3DStageError:
+ dry_run = sam3d.dry_run()
+ state.record_stage("sam3d_dry_run", status="passed", facts=dry_run)
+ sam3d_artifacts = sam3d.run(approve_gpu=True)
+ state.record_stage(
+ "sam3d",
+ status="published",
+ successful_state="sam3d_published",
+ facts={
+ "stage05_dir": str(sam3d_artifacts.stage05_dir),
+ "object_ids": list(sam3d_artifacts.object_ids),
+ "source_manifest_sha256": sam3d_artifacts.source_manifest_sha256,
+ "artifact_hashes": sam3d_artifacts.artifact_hashes,
+ "attempt_id": sam3d_artifacts.attempt_id,
+ "reused": sam3d_artifacts.reused,
+ },
+ )
+
+ relation_adapter = self._relation_adapter(scene)
+ relation: RelationArtifacts | None = None
+ try:
+ relation = relation_adapter.reuse()
+ except RelationStageError:
+ relation_record = state.data.get("stages", {}).get("relation", {})
+ relation_facts = (
+ relation_record.get("facts", {})
+ if isinstance(relation_record, dict)
+ else {}
+ )
+ relation_status = (
+ relation_record.get("status")
+ if isinstance(relation_record, dict)
+ else None
+ )
+ candidate_raw = (
+ relation_facts.get("candidate_dir")
+ if relation_status
+ in {
+ "candidate_running",
+ "candidate_valid",
+ "candidate_interrupted",
+ }
+ else None
+ )
+ recovery_candidate = (
+ Path(candidate_raw).expanduser().resolve()
+ if isinstance(candidate_raw, str) and candidate_raw.strip()
+ else None
+ )
+ if recovery_candidate is not None and recovery_candidate.is_dir():
+ try:
+ recovered = relation_adapter.validate_candidate(
+ recovery_candidate
+ )
+ relation = relation_adapter.publish_candidate(
+ candidate_dir=recovered.candidate_dir,
+ publication_id=(
+ f"publication_{recovered.candidate_dir.name}"
+ ),
+ )
+ except RelationStageError as exc:
+ state.record_attempt(
+ "relation",
+ attempt_id=recovery_candidate.name,
+ status="recovery_failed",
+ facts={
+ "candidate_dir": str(recovery_candidate),
+ "error": str(exc),
+ },
+ )
+ else:
+ state.record_attempt(
+ "relation",
+ attempt_id=recovery_candidate.name,
+ status="recovered_and_published",
+ facts={
+ "candidate_dir": str(recovery_candidate),
+ "validation": str(recovered.validation_path),
+ "publication_id": relation.publication_id,
+ },
+ )
+
+ if relation is None:
+ if not self.runtime.has_api_credentials(
+ "relation", environ=environ
+ ):
+ raise FullPipelineError(
+ f"environment {self.runtime.relation_api_key_env} "
+ "is required for Relation execution"
+ )
+ timestamp = datetime.now(timezone.utc).strftime(
+ "%Y%m%dT%H%M%SZ"
+ )
+ benchmark_id = (
+ f"relation_{scene.contract.scene_name}_"
+ f"{timestamp}_{uuid.uuid4().hex[:8]}"
+ )
+ candidate_dir = (
+ scene.contract.relation_candidates_dir / benchmark_id
+ )
+ state.record_stage(
+ "relation",
+ status="candidate_running",
+ facts={
+ "candidate_id": benchmark_id,
+ "candidate_dir": str(candidate_dir),
+ "invocation_id": invocation_id,
+ },
+ )
+ state.record_attempt(
+ "relation",
+ attempt_id=benchmark_id,
+ status="started",
+ facts={
+ "candidate_dir": str(candidate_dir),
+ "invocation_id": invocation_id,
+ },
+ )
+ source_env = os.environ if environ is None else environ
+ api_base = _optional_url(
+ source_env.get(self.runtime.relation_api_base_env)
+ )
+ candidate = relation_adapter.run_candidate(
+ benchmark_id=benchmark_id,
+ api_key_env=self.runtime.relation_api_key_env,
+ model=self.runtime.vlm_model,
+ precompute_python=(
+ self.runtime.relation_precompute_python
+ ),
+ runner_python=self.runtime.relation_runner_python,
+ api_base=api_base,
+ environ=self.runtime.child_environment(
+ "relation", environ=environ
+ ),
+ )
+ validated = relation_adapter.validate_candidate(
+ candidate.candidate_dir
+ )
+ state.record_attempt(
+ "relation",
+ attempt_id=benchmark_id,
+ status="candidate_valid",
+ facts={
+ "candidate_dir": str(candidate.candidate_dir),
+ "validation": str(validated.validation_path),
+ },
+ )
+ relation = relation_adapter.publish_candidate(
+ candidate_dir=validated.candidate_dir,
+ publication_id=f"publication_{benchmark_id}",
+ )
+ state.record_attempt(
+ "relation",
+ attempt_id=benchmark_id,
+ status="published",
+ facts={
+ "candidate_dir": str(candidate.candidate_dir),
+ "validation": str(validated.validation_path),
+ "publication_id": relation.publication_id,
+ },
+ )
+
+ assert relation is not None
+ state.record_stage(
+ "relation",
+ status="published",
+ successful_state="relation_published",
+ facts={
+ "publication_id": relation.publication_id,
+ "publication_manifest": str(
+ relation.publication_manifest_path
+ ),
+ "publication_manifest_sha256": _sha256(
+ relation.publication_manifest_path
+ ),
+ "objects": str(relation.objects_path),
+ "relations": str(relation.relations_path),
+ "candidate_validated": True,
+ },
+ )
+
+ stage08_adapter = SceneMosaicStageAdapter(
+ scene_config_path=scene.contract.scene_config_path,
+ relation_artifacts=relation,
+ )
+ stage08 = self._prepared_stage08(scene, relation, stage08_adapter)
+ if stage08 is None:
+ stage_id = (
+ "stage08_"
+ + datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ_")
+ + uuid.uuid4().hex[:8]
+ )
+ stage08 = stage08_adapter.prepare(stage_id)
+ state.record_stage(
+ "stage08",
+ status="prepared",
+ successful_state="stage08_prepared",
+ facts={
+ "stage_id": stage08.stage_id,
+ "workspace_dir": str(stage08.workspace_dir),
+ "integration_manifest": str(stage08.integration_manifest_path),
+ "relation_publication_id": relation.publication_id,
+ "wall_padding": 0.0,
+ "per_object_physics_consumed": False,
+ },
+ )
+ state.record_attempt(
+ "stage08",
+ attempt_id=stage08.stage_id,
+ status="prepared",
+ facts={
+ "workspace_dir": str(stage08.workspace_dir),
+ "integration_manifest": str(stage08.integration_manifest_path),
+ },
+ )
+
+ source_env = os.environ if environ is None else environ
+ secret = source_env.get(self.runtime.scenemosaic_api_key_env)
+ if not secret:
+ raise FullPipelineError(
+ f"environment {self.runtime.scenemosaic_api_key_env} "
+ "is required for scene evolution"
+ )
+ state.record_attempt(
+ "stage08",
+ attempt_id=stage08.stage_id,
+ status="started",
+ facts={
+ "workspace_dir": str(stage08.workspace_dir),
+ "invocation_id": invocation_id,
+ },
+ )
+ state.record_stage(
+ "stage08",
+ status="running",
+ facts={
+ "stage_id": stage08.stage_id,
+ "workspace_dir": str(stage08.workspace_dir),
+ "integration_manifest": str(stage08.integration_manifest_path),
+ },
+ )
+ api_base = _optional_url(
+ source_env.get(self.runtime.scenemosaic_api_base_env)
+ )
+ client = VLMClient(
+ url=api_base,
+ model=self.runtime.vlm_model,
+ api_key=secret,
+ )
+ paths = ProjectPaths(
+ self.runtime.project_root, output_dir=stage08.workspace_dir
+ )
+ downstream = IntegratedPipelineOrchestrator(
+ paths=paths,
+ tool_runner=IntegratedToolRunner(self.runtime.project_root),
+ vlm_client=client,
+ max_rounds=3,
+ collision_policy="strict",
+ )
+ try:
+ report = downstream.run_full_pipeline()
+ report_path = downstream.save_report(
+ report, stage08.workspace_dir / "pipeline_report.json"
+ )
+ except Exception as exc:
+ state.record_attempt(
+ "stage08",
+ attempt_id=stage08.stage_id,
+ status="failed",
+ facts={
+ "workspace_dir": str(stage08.workspace_dir),
+ "error_type": type(exc).__name__,
+ },
+ )
+ state.fail(
+ stage="stage08",
+ error_type=type(exc).__name__,
+ message="SceneMosaic downstream failed; inspect the retained workspace",
+ resumable=False,
+ next_action=(
+ "inspect the retained Stage 08 attempt; the next run will create a new stage08_id"
+ ),
+ )
+ state.finish_invocation(invocation_id, result="failed")
+ self._aggregate_audit(scene)
+ raise FullPipelineChildError("SceneMosaic downstream failed") from None
+ publication = stage08_adapter.publish(
+ stage08, pipeline_report_path=report_path
+ )
+ state.record_attempt(
+ "stage08",
+ attempt_id=stage08.stage_id,
+ status="published",
+ facts={
+ "workspace_dir": str(stage08.workspace_dir),
+ "publication_manifest": str(publication),
+ "publication_manifest_sha256": _sha256(publication),
+ },
+ )
+ state.record_stage(
+ "stage08",
+ status="published",
+ successful_state="scenemosaic_published",
+ facts={
+ "stage_id": stage08.stage_id,
+ "workspace_dir": str(stage08.workspace_dir),
+ "integration_manifest": str(stage08.integration_manifest_path),
+ "pipeline_report": str(report_path),
+ "publication_manifest": str(publication),
+ "publication_manifest_sha256": _sha256(publication),
+ "relation_publication_id": relation.publication_id,
+ "per_object_physics_consumed": False,
+ "wall_padding": 0.0,
+ },
+ )
+ state.complete()
+ state.finish_invocation(invocation_id, result="complete")
+ audit_path = self._aggregate_audit(scene)
+ return {
+ "complete": True,
+ "status": "scenemosaic_published",
+ "scene_root": str(scene.contract.run_root),
+ "stage08_id": stage08.stage_id,
+ "publication_manifest": str(publication),
+ "pipeline_audit": str(audit_path),
+ }
+ except FullPipelineError as exc:
+ if state.current_state != "failed":
+ state.fail(
+ stage="integrity",
+ error_type=type(exc).__name__,
+ message=str(exc),
+ resumable=False,
+ next_action="run python -m vlm_agent.cli verify and inspect immutable facts",
+ )
+ if any(
+ item.get("invocation_id") == invocation_id
+ and item.get("result") == "running"
+ for item in state.data.get("invocations", [])
+ ):
+ state.finish_invocation(invocation_id, result="failed")
+ self._aggregate_audit(scene)
+ raise
+ except (
+ PerceptionStageError,
+ Sam3DStageError,
+ RelationStageError,
+ SceneMosaicArtifactError,
+ ) as exc:
+ stage = (
+ "perception"
+ if isinstance(exc, PerceptionStageError)
+ else "sam3d"
+ if isinstance(exc, Sam3DStageError)
+ else "relation"
+ if isinstance(exc, RelationStageError)
+ else "stage08"
+ )
+ state.fail(
+ stage=stage,
+ error_type=type(exc).__name__,
+ message=str(exc),
+ resumable=stage != "stage08",
+ next_action="inspect the retained audit and resume after correcting the failure",
+ )
+ state.finish_invocation(invocation_id, result="failed")
+ self._aggregate_audit(scene)
+ child = "failed with exit code" in str(exc) or "runner failed" in str(exc)
+ error_type = FullPipelineChildError if child else FullPipelineIntegrityError
+ raise error_type(str(exc)) from None
+ except Exception as exc:
+ state.fail(
+ stage="orchestrator",
+ error_type=type(exc).__name__,
+ message=str(exc),
+ resumable=False,
+ next_action="inspect pipeline state and facts before retrying",
+ )
+ state.finish_invocation(invocation_id, result="failed")
+ self._aggregate_audit(scene)
+ raise FullPipelineIntegrityError(str(exc)) from None
+ finally:
+ scene_lock.release()
+
+ def status(self, scene_root: Path) -> dict[str, Any]:
+ scene = load_scene_run(scene_root)
+ try:
+ scene_lock = scene.state.acquire_lock(exclusive=False, blocking=False)
+ except RunStateError as exc:
+ raise FullPipelineIntegrityError(str(exc)) from exc
+ try:
+ scene = load_scene_run(scene_root)
+ state = scene.state.public_status()
+ stages = state.get("stages", {})
+ checks: list[dict[str, Any]] = []
+
+ def check(name: str, function: Any) -> None:
+ try:
+ value = function()
+ except Exception as exc:
+ checks.append(
+ {
+ "name": name,
+ "valid": False,
+ "error": str(exc),
+ }
+ )
+ else:
+ checks.append(
+ {
+ "name": name,
+ "valid": True,
+ "facts": value,
+ }
+ )
+
+ check(
+ "runtime",
+ lambda: self.verify(require_gpu=False),
+ )
+
+ perception_record = stages.get("perception")
+ perception_expected = (
+ isinstance(perception_record, dict)
+ or (scene.contract.stage03_dir / "perception_adoption.json").exists()
+ )
+ if perception_expected:
+
+ def perception_facts() -> dict[str, Any]:
+ result = PerceptionStage(
+ self.runtime,
+ scene.contract.scene_config_path,
+ ).reuse_formal()
+ if result is None:
+ raise FullPipelineIntegrityError(
+ "formal perception publication is missing"
+ )
+ recorded = (
+ perception_record.get("facts", {})
+ if isinstance(perception_record, dict)
+ else {}
+ )
+ object_ids = list(result.artifacts.object_ids)
+ if recorded and (
+ recorded.get("object_ids") != object_ids
+ or recorded.get("run_manifest_sha256")
+ != result.run_manifest_sha256
+ ):
+ raise FullPipelineIntegrityError(
+ "perception facts disagree with pipeline state"
+ )
+ return {
+ "delivery_status": (result.artifacts.delivery_status),
+ "object_ids": object_ids,
+ "run_manifest_sha256": (result.run_manifest_sha256),
+ }
+
+ check("perception", perception_facts)
+
+ sam3d_record = stages.get("sam3d")
+ sam3d_expected = (
+ isinstance(sam3d_record, dict)
+ or (scene.contract.stage05_dir / "object_index_map.json").exists()
+ )
+ if sam3d_expected:
+
+ def sam3d_facts() -> dict[str, Any]:
+ result = validate_stage05(
+ scene.contract.scene_config_path,
+ project_root=self.runtime.project_root,
+ )
+ recorded = (
+ sam3d_record.get("facts", {})
+ if isinstance(sam3d_record, dict)
+ else {}
+ )
+ object_ids = list(result.object_ids)
+ if recorded and (
+ recorded.get("object_ids") != object_ids
+ or recorded.get("source_manifest_sha256")
+ != result.source_manifest_sha256
+ ):
+ raise FullPipelineIntegrityError(
+ "SAM3D facts disagree with pipeline state"
+ )
+ return {
+ "object_ids": object_ids,
+ "source_manifest_sha256": (result.source_manifest_sha256),
+ "artifact_hashes": result.artifact_hashes,
+ }
+
+ check("sam3d", sam3d_facts)
+
+ relation_record = stages.get("relation")
+ relation_current = scene.contract.relations_dir / "current.json"
+ relation: RelationArtifacts | None = None
+ if relation_current.is_file() or (
+ isinstance(relation_record, dict)
+ and relation_record.get("status") == "published"
+ ):
+
+ def relation_facts() -> dict[str, Any]:
+ nonlocal relation
+ relation = self._relation_adapter(scene).reuse()
+ recorded = (
+ relation_record.get("facts", {})
+ if isinstance(relation_record, dict)
+ else {}
+ )
+ manifest_sha = _sha256(relation.publication_manifest_path)
+ if recorded and (
+ recorded.get("publication_id") != relation.publication_id
+ or recorded.get("publication_manifest_sha256") != manifest_sha
+ ):
+ raise FullPipelineIntegrityError(
+ "Relation facts disagree with pipeline state"
+ )
+ return {
+ "publication_id": relation.publication_id,
+ "publication_manifest": str(relation.publication_manifest_path),
+ "publication_manifest_sha256": manifest_sha,
+ "candidate_validated": True,
+ }
+
+ check("relation", relation_facts)
+ elif isinstance(relation_record, dict) and relation_record.get(
+ "status"
+ ) in {
+ "candidate_running",
+ "candidate_valid",
+ "candidate_interrupted",
+ }:
+
+ def relation_candidate_facts() -> dict[str, Any]:
+ facts = relation_record.get("facts", {})
+ candidate_raw = (
+ facts.get("candidate_dir") if isinstance(facts, dict) else None
+ )
+ if not isinstance(candidate_raw, str) or not candidate_raw.strip():
+ raise FullPipelineIntegrityError(
+ "Relation candidate path is missing"
+ )
+ candidate = Path(candidate_raw).expanduser().resolve()
+ expected_root = scene.contract.relation_candidates_dir.resolve()
+ if (
+ candidate.parent != expected_root
+ or not candidate.is_dir()
+ or candidate.is_symlink()
+ ):
+ raise FullPipelineIntegrityError(
+ "Relation candidate path is unsafe"
+ )
+ validation = candidate / ("scenemosaic_candidate_validation.json")
+ if relation_record.get("status") == "candidate_valid" and (
+ not validation.is_file() or validation.is_symlink()
+ ):
+ raise FullPipelineIntegrityError(
+ "validated Relation candidate record is missing"
+ )
+ return {
+ "status": relation_record.get("status"),
+ "candidate_id": candidate.name,
+ "candidate_dir": str(candidate),
+ "validation": (
+ str(validation) if validation.is_file() else None
+ ),
+ }
+
+ check(
+ "relation_candidate",
+ relation_candidate_facts,
+ )
+
+ stage08_record = stages.get("stage08")
+ if isinstance(stage08_record, dict):
+ stage08_status = stage08_record.get("status")
+ if stage08_status == "published":
+ check(
+ "stage08",
+ lambda: self._validate_completed_scene(scene)["stage08"],
+ )
+ elif stage08_status == "prepared":
+
+ def prepared_stage08_facts() -> dict[str, Any]:
+ nonlocal relation
+ if relation is None:
+ relation = self._relation_adapter(scene).reuse()
+ adapter = SceneMosaicStageAdapter(
+ scene_config_path=(scene.contract.scene_config_path),
+ relation_artifacts=relation,
+ )
+ prepared = self._prepared_stage08(scene, relation, adapter)
+ if prepared is None:
+ raise FullPipelineIntegrityError(
+ "prepared Stage 08 state is missing"
+ )
+ return {
+ "stage_id": prepared.stage_id,
+ "workspace_dir": str(prepared.workspace_dir),
+ "integration_manifest": str(
+ prepared.integration_manifest_path
+ ),
+ "upstream": (
+ adapter.verify_upstream_unchanged(
+ prepared.integration_manifest_path
+ )
+ ),
+ "working_copies": (
+ adapter.verify_prepared_working_copies(
+ prepared.integration_manifest_path
+ )
+ ),
+ }
+
+ check("stage08_prepared", prepared_stage08_facts)
+ elif stage08_status == "running":
+
+ def running_stage08_facts() -> dict[str, Any]:
+ facts = stage08_record.get("facts", {})
+ workspace = (
+ Path(str(facts.get("workspace_dir") or ""))
+ .expanduser()
+ .resolve()
+ )
+ integration = (
+ Path(str(facts.get("integration_manifest") or ""))
+ .expanduser()
+ .resolve()
+ )
+ stage_id = str(facts.get("stage_id") or "")
+ expected = (
+ scene.contract.evolution_dir
+ / ".stage08_attempts"
+ / stage_id
+ ).resolve()
+ if (
+ workspace != expected
+ or integration != workspace / "integration_manifest.json"
+ or not integration.is_file()
+ or integration.is_symlink()
+ ):
+ raise FullPipelineIntegrityError(
+ "running Stage 08 attempt paths are unsafe"
+ )
+ return {
+ "stage_id": stage_id,
+ "workspace_dir": str(workspace),
+ "upstream": (
+ SceneMosaicStageAdapter.verify_upstream_unchanged(
+ integration
+ )
+ ),
+ "resumable_in_place": False,
+ }
+
+ check("stage08_running", running_stage08_facts)
+
+ complete = state.get("current_state") == "scenemosaic_published"
+ if complete and not any(item["name"] == "stage08" for item in checks):
+ checks.append(
+ {
+ "name": "stage08",
+ "valid": False,
+ "error": ("completed state has no published Stage 08 facts"),
+ }
+ )
+ runtime_checks = next(
+ (
+ item.get("facts", {}).get("checks", [])
+ for item in checks
+ if item["name"] == "runtime" and item["valid"]
+ ),
+ [],
+ )
+ if runtime_checks and not all(
+ item.get("valid") is True
+ for item in runtime_checks
+ if item.get("required", True)
+ ):
+ for item in checks:
+ if item["name"] == "runtime":
+ item["valid"] = False
+ item["error"] = "one or more runtime checks failed"
+ break
+ return {
+ "complete": complete,
+ "scene_root": str(scene.contract.run_root),
+ "current_state": state.get("current_state"),
+ "last_successful_state": state.get("last_successful_state"),
+ "next_action": state.get("next_action"),
+ "failure": state.get("failure"),
+ "fact_checks": checks,
+ "facts_valid": all(item["valid"] for item in checks),
+ "pipeline_state": str(scene.state.path),
+ "pipeline_audit": str(
+ scene.contract.run_root / "audit" / "pipeline_audit.json"
+ ),
+ "gpu_used": False,
+ }
+ finally:
+ scene_lock.release()
diff --git a/vlm_agent/reconstruction/__init__.py b/vlm_agent/reconstruction/__init__.py
new file mode 100644
index 0000000..d0b1a4d
--- /dev/null
+++ b/vlm_agent/reconstruction/__init__.py
@@ -0,0 +1 @@
+"""SAM3 and SAM 3D reconstruction stage controllers."""
diff --git a/vlm_agent/reconstruction/perception/__init__.py b/vlm_agent/reconstruction/perception/__init__.py
new file mode 100644
index 0000000..5d10a0d
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/__init__.py
@@ -0,0 +1,6 @@
+"""SceneMosaic single-image perception subsystem."""
+
+from .adoption import PerceptionAdoptionAdapter
+from .stage import PerceptionStage
+
+__all__ = ["PerceptionAdoptionAdapter", "PerceptionStage"]
diff --git a/vlm_agent/reconstruction/perception/adoption.py b/vlm_agent/reconstruction/perception/adoption.py
new file mode 100644
index 0000000..52af7ac
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/adoption.py
@@ -0,0 +1,1014 @@
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+import os
+import re
+import shutil
+import uuid
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from PIL import Image
+
+from ...artifacts import RunContract
+
+
+READY_STATUSES = frozenset({"sam3d_ready", "sam3d_ready_degraded"})
+OBJECT_ID_RE = re.compile(r"^object_(\d{3})$")
+
+
+class PerceptionAdoptionError(RuntimeError):
+ """Raised when a protected perception delivery cannot be safely adopted."""
+
+
+@dataclass(frozen=True)
+class PerceptionAdoptionArtifacts:
+ stage03_dir: Path
+ stage04_dir: Path
+ reviewed_objects_path: Path
+ sam3d_manifest_path: Path
+ adoption_report_path: Path
+ delivery_status: str
+ object_ids: tuple[str, ...]
+ reused: bool = False
+
+
+@dataclass(frozen=True)
+class _ValidatedDelivery:
+ run_manifest_path: Path
+ run_manifest: dict[str, Any]
+ run_manifest_sha256: str
+ output_dir: Path
+ source_manifest_path: Path
+ source_manifest: dict[str, Any]
+ delivery_report_path: Path
+ delivery_report: dict[str, Any]
+ run_info_path: Path | None
+ quality_report_path: Path
+ image_path: Path
+ image_size: tuple[int, int]
+ mask_paths: tuple[Path, ...]
+ object_ids: tuple[str, ...]
+ status: str
+
+
+def _utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _read_json(path: Path, label: str) -> dict[str, Any]:
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise PerceptionAdoptionError(f"cannot parse {label}: {path}") from exc
+ if not isinstance(payload, dict):
+ raise PerceptionAdoptionError(f"{label} must contain a JSON object: {path}")
+ return payload
+
+
+def _write_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+
+
+def _is_within(path: Path, root: Path) -> bool:
+ try:
+ path.relative_to(root)
+ return True
+ except ValueError:
+ return False
+
+
+def _resolve_confined(
+ raw_path: Any,
+ *,
+ base_dir: Path,
+ trusted_root: Path,
+ label: str,
+) -> Path:
+ if not isinstance(raw_path, str) or not raw_path.strip():
+ raise PerceptionAdoptionError(f"{label} must be a non-empty path")
+ candidate = Path(raw_path).expanduser()
+ if not candidate.is_absolute():
+ candidate = base_dir / candidate
+ resolved = candidate.resolve()
+ if not _is_within(resolved, trusted_root):
+ raise PerceptionAdoptionError(
+ f"{label} escapes perception attempts root: {resolved}"
+ )
+ return resolved
+
+
+def _directory_has_files(path: Path) -> bool:
+ if not path.exists():
+ return False
+ if not path.is_dir():
+ return True
+ return any(item.is_file() or item.is_symlink() for item in path.rglob("*"))
+
+
+def _empty_tree_snapshot(path: Path) -> tuple[bool, tuple[Path, ...]]:
+ if not path.exists():
+ return False, ()
+ if not path.is_dir():
+ raise PerceptionAdoptionError(f"stage target is not a directory: {path}")
+ entries = tuple(path.rglob("*"))
+ if any(item.is_file() or item.is_symlink() for item in entries):
+ raise PerceptionAdoptionError(f"stage target already contains files: {path}")
+ directories = tuple(
+ sorted(
+ (item.relative_to(path) for item in entries if item.is_dir()),
+ key=lambda value: (len(value.parts), str(value)),
+ )
+ )
+ return True, directories
+
+
+def _remove_empty_tree(path: Path) -> None:
+ if not path.exists():
+ return
+ directories = sorted(
+ (item for item in path.rglob("*") if item.is_dir()),
+ key=lambda item: len(item.parts),
+ reverse=True,
+ )
+ for directory in directories:
+ directory.rmdir()
+ path.rmdir()
+
+
+def _restore_empty_tree(path: Path, snapshot: tuple[bool, tuple[Path, ...]]) -> None:
+ existed, directories = snapshot
+ if not existed:
+ return
+ path.mkdir(parents=True, exist_ok=True)
+ for relative in directories:
+ (path / relative).mkdir(parents=True, exist_ok=True)
+
+
+class PerceptionAdoptionAdapter:
+ """Adopt a isolated perception delivery into formal Stage 03 and Stage 04.
+
+ The adapter only validates, copies and publishes existing delivery artifacts.
+ It never invokes segmentation, changes masks, or rebuilds object numbering.
+ """
+
+ def __init__(
+ self,
+ *,
+ run_manifest_path: Path,
+ scene_config_path: Path,
+ perception_attempts_root: Path | None = None,
+ ) -> None:
+ self.run_manifest_path = Path(run_manifest_path).expanduser().resolve()
+ self.perception_attempts_root = Path(
+ perception_attempts_root or self.run_manifest_path.parent
+ ).expanduser().resolve()
+ self.contract = RunContract.from_scene_config(scene_config_path)
+
+ def adopt(self) -> PerceptionAdoptionArtifacts:
+ delivery = self._validate_delivery()
+ existing = self._reuse_existing(delivery)
+ if existing is not None:
+ return existing
+
+ stage03 = self.contract.stage03_dir
+ stage04 = self.contract.stage04_dir
+ stage03_tmp = stage03.parent / f".{stage03.name}.adopt-{uuid.uuid4().hex}.tmp"
+ stage04_tmp = stage04.parent / f".{stage04.name}.adopt-{uuid.uuid4().hex}.tmp"
+ stage03_tmp.mkdir(parents=True, exist_ok=False)
+ stage04_tmp.mkdir(parents=True, exist_ok=False)
+
+ try:
+ object_ids = self._materialize(delivery, stage03_tmp, stage04_tmp)
+ self._publish_pair(stage03_tmp, stage04_tmp, stage03, stage04)
+ except Exception:
+ if stage03_tmp.exists():
+ shutil.rmtree(stage03_tmp)
+ if stage04_tmp.exists():
+ shutil.rmtree(stage04_tmp)
+ raise
+
+ return PerceptionAdoptionArtifacts(
+ stage03_dir=stage03,
+ stage04_dir=stage04,
+ reviewed_objects_path=stage03 / "reviewed_objects.json",
+ sam3d_manifest_path=stage04 / "sam3d_input_manifest.json",
+ adoption_report_path=stage03 / "perception_adoption.json",
+ delivery_status=delivery.status,
+ object_ids=object_ids,
+ reused=False,
+ )
+
+ @classmethod
+ def validate_formal(
+ cls,
+ scene_config_path: Path,
+ ) -> PerceptionAdoptionArtifacts:
+ """Validate an already-published Stage 03/04 without attempt-source access.
+
+ The attempt delivery is a source used during adoption, not a lifetime
+ dependency of the formal run. Every fact needed for reuse is bound by
+ perception_adoption.json and rechecked against the formal copies.
+ """
+
+ contract = RunContract.from_scene_config(scene_config_path)
+ stage03 = contract.stage03_dir.resolve()
+ stage04 = contract.stage04_dir.resolve()
+ adoption_path = stage03 / "perception_adoption.json"
+ adoption = _read_json(adoption_path, "formal perception adoption")
+ status = str(adoption.get("delivery_status") or "")
+ if (
+ adoption.get("schema_version")
+ != "scenemosaic_perception_adoption"
+ or adoption.get("scene_name") != contract.scene_name
+ or status not in READY_STATUSES
+ or adoption.get("sam3d_execution_allowed") is not True
+ or adoption.get("human_review_required") is not False
+ or adoption.get("renumbered") is not False
+ ):
+ raise PerceptionAdoptionError(
+ "formal perception adoption invariants are invalid"
+ )
+
+ formal_records = adoption.get("formal_artifacts")
+ if not isinstance(formal_records, list) or not formal_records:
+ raise PerceptionAdoptionError(
+ "formal perception adoption has no artifact hash closure"
+ )
+ declared: dict[Path, str] = {}
+ for index, record in enumerate(formal_records):
+ if not isinstance(record, dict):
+ raise PerceptionAdoptionError(
+ f"formal artifact record {index} is invalid"
+ )
+ path = Path(
+ str(record.get("path") or "")
+ ).expanduser().resolve()
+ digest = record.get("sha256")
+ if (
+ path in declared
+ or not isinstance(digest, str)
+ or not (
+ _is_within(path, stage03)
+ or _is_within(path, stage04)
+ )
+ or path == adoption_path
+ ):
+ raise PerceptionAdoptionError(
+ f"formal artifact record is unsafe: {path}"
+ )
+ if (
+ not path.is_file()
+ or path.is_symlink()
+ or path.stat().st_size == 0
+ or _sha256(path) != digest
+ ):
+ raise PerceptionAdoptionError(
+ f"formal perception artifact hash mismatch: {path}"
+ )
+ declared[path] = digest
+
+ actual: set[Path] = set()
+ for root in (stage03, stage04):
+ if not root.is_dir() or root.is_symlink():
+ raise PerceptionAdoptionError(
+ f"formal perception stage is missing or unsafe: {root}"
+ )
+ for path in root.rglob("*"):
+ if path.is_symlink():
+ raise PerceptionAdoptionError(
+ f"formal perception stage contains a symlink: {path}"
+ )
+ if path.is_file() and path != adoption_path:
+ actual.add(path.resolve())
+ if actual != set(declared):
+ missing = sorted(str(path) for path in set(declared) - actual)
+ extra = sorted(str(path) for path in actual - set(declared))
+ raise PerceptionAdoptionError(
+ "formal perception artifact closure mismatch: "
+ f"missing={missing}, extra={extra}"
+ )
+
+ reviewed_path = stage03 / "reviewed_objects.json"
+ quality_path = stage03 / "quality_report.json"
+ manifest_path = stage04 / "sam3d_input_manifest.json"
+ delivery_report_path = stage04 / "delivery_report.json"
+ run_info_path = stage04 / "sam3d_input_run_info.json"
+ for path in (
+ reviewed_path,
+ quality_path,
+ manifest_path,
+ delivery_report_path,
+ run_info_path,
+ ):
+ if path not in declared:
+ raise PerceptionAdoptionError(
+ "required formal perception artifact is not "
+ f"hash-bound: {path}"
+ )
+ reviewed = _read_json(
+ reviewed_path, "formal reviewed objects"
+ )
+ _read_json(quality_path, "formal quality report")
+ manifest = _read_json(
+ manifest_path, "formal Stage 04 manifest"
+ )
+ delivery_report = _read_json(
+ delivery_report_path, "formal delivery report"
+ )
+ run_info = _read_json(
+ run_info_path, "formal Stage 04 run info"
+ )
+ if (
+ manifest.get("delivery_status") != status
+ or manifest.get("sam3d_execution_allowed") is not True
+ or run_info.get("status") != status
+ or run_info.get("sam3d_execution_allowed") is not True
+ ):
+ raise PerceptionAdoptionError(
+ "formal Stage 04 delivery status is inconsistent"
+ )
+
+ object_items = manifest.get("objects")
+ reviewed_items = reviewed.get("objects")
+ report_items = delivery_report.get("objects")
+ if not all(
+ isinstance(items, list)
+ for items in (object_items, reviewed_items, report_items)
+ ):
+ raise PerceptionAdoptionError(
+ "formal perception object lists are invalid"
+ )
+ assert isinstance(object_items, list)
+ assert isinstance(reviewed_items, list)
+ assert isinstance(report_items, list)
+ object_count = manifest.get("object_count")
+ if not isinstance(object_count, int) or object_count <= 0:
+ raise PerceptionAdoptionError(
+ "formal Stage 04 object_count is invalid"
+ )
+ expected_ids = tuple(
+ f"object_{index:03d}"
+ for index in range(1, object_count + 1)
+ )
+
+ def ids(items: list[Any], label: str) -> tuple[str, ...]:
+ values: list[str] = []
+ for index, item in enumerate(items):
+ if not isinstance(item, dict):
+ raise PerceptionAdoptionError(
+ f"{label} object {index} is invalid"
+ )
+ object_id = str(item.get("object_id") or "")
+ if (
+ not OBJECT_ID_RE.fullmatch(object_id)
+ or item.get("sam3d_internal_name") != object_id
+ ):
+ raise PerceptionAdoptionError(
+ f"{label} identity mismatch for "
+ f"{object_id or index}"
+ )
+ values.append(object_id)
+ return tuple(values)
+
+ manifest_ids = ids(object_items, "Stage 04")
+ reviewed_ids = ids(reviewed_items, "reviewed")
+ report_ids = ids(report_items, "delivery report")
+ closure = adoption.get("object_id_closure")
+ if (
+ manifest_ids != expected_ids
+ or reviewed_ids != expected_ids
+ or report_ids != expected_ids
+ or reviewed.get("object_count") != object_count
+ or run_info.get("object_count") != object_count
+ or not isinstance(closure, dict)
+ or closure.get("closed") is not True
+ or tuple(closure.get("manifest", ())) != expected_ids
+ or tuple(closure.get("reviewed_objects", ()))
+ != expected_ids
+ or tuple(closure.get("delivery_report", ()))
+ != expected_ids
+ ):
+ raise PerceptionAdoptionError(
+ "formal perception object ID closure is invalid"
+ )
+
+ image_path = Path(
+ str(manifest.get("image_path") or "")
+ ).expanduser().resolve()
+ expected_image = stage04 / "image.png"
+ if (
+ image_path != expected_image
+ or image_path not in declared
+ or _sha256(image_path) != _sha256(contract.input_image)
+ ):
+ raise PerceptionAdoptionError(
+ "formal Stage 04 image does not match the formal input"
+ )
+ try:
+ with Image.open(image_path) as image:
+ image.load()
+ image_size = image.size
+ except (OSError, ValueError) as exc:
+ raise PerceptionAdoptionError(
+ "formal Stage 04 image is unreadable"
+ ) from exc
+
+ masks_dir = stage04 / "masks"
+ actual_masks = (
+ {
+ path.resolve()
+ for path in masks_dir.iterdir()
+ if path.is_file() and not path.is_symlink()
+ }
+ if masks_dir.is_dir() and not masks_dir.is_symlink()
+ else set()
+ )
+ declared_masks: set[Path] = set()
+ mask_hashes: dict[str, str] = {}
+ for index, item in enumerate(object_items):
+ assert isinstance(item, dict)
+ object_id = expected_ids[index]
+ if item.get("order_index") != index:
+ raise PerceptionAdoptionError(
+ "formal Stage 04 order_index mismatch for "
+ f"{object_id}"
+ )
+ mask_path = Path(
+ str(item.get("mask_path") or "")
+ ).expanduser().resolve()
+ if (
+ not _is_within(mask_path, masks_dir)
+ or mask_path not in declared
+ or mask_path in declared_masks
+ ):
+ raise PerceptionAdoptionError(
+ "formal Stage 04 mask path is invalid for "
+ f"{object_id}"
+ )
+ digest = _sha256(mask_path)
+ if item.get("mask_sha256") != digest:
+ raise PerceptionAdoptionError(
+ "formal Stage 04 mask hash mismatch for "
+ f"{object_id}"
+ )
+ try:
+ with Image.open(mask_path) as mask:
+ mask.load()
+ if (
+ mask.size != image_size
+ or mask.convert("L").getbbox() is None
+ ):
+ raise PerceptionAdoptionError(
+ "formal Stage 04 mask content is invalid "
+ f"for {object_id}"
+ )
+ except PerceptionAdoptionError:
+ raise
+ except (OSError, ValueError) as exc:
+ raise PerceptionAdoptionError(
+ f"formal Stage 04 mask is unreadable for {object_id}"
+ ) from exc
+ declared_masks.add(mask_path)
+ mask_hashes[object_id] = digest
+ if actual_masks != declared_masks:
+ raise PerceptionAdoptionError(
+ "formal Stage 04 mask file closure is invalid"
+ )
+
+ comparisons = adoption.get("mask_hash_comparison")
+ if (
+ not isinstance(comparisons, list)
+ or len(comparisons) != object_count
+ ):
+ raise PerceptionAdoptionError(
+ "formal perception mask hash comparison is invalid"
+ )
+ for comparison, object_id in zip(
+ comparisons, expected_ids
+ ):
+ if (
+ not isinstance(comparison, dict)
+ or comparison.get("object_id") != object_id
+ or Path(
+ str(comparison.get("formal_path") or "")
+ ).expanduser().resolve()
+ not in declared_masks
+ or comparison.get("formal_sha256")
+ != mask_hashes[object_id]
+ or comparison.get("identical") is not True
+ ):
+ raise PerceptionAdoptionError(
+ f"formal mask hash evidence mismatch for {object_id}"
+ )
+
+ return PerceptionAdoptionArtifacts(
+ stage03_dir=stage03,
+ stage04_dir=stage04,
+ reviewed_objects_path=reviewed_path,
+ sam3d_manifest_path=manifest_path,
+ adoption_report_path=adoption_path,
+ delivery_status=status,
+ object_ids=expected_ids,
+ reused=True,
+ )
+
+ def _validate_delivery(self) -> _ValidatedDelivery:
+ if not self.run_manifest_path.is_file():
+ raise PerceptionAdoptionError(
+ f"run_manifest is not a file: {self.run_manifest_path}"
+ )
+ if not _is_within(self.run_manifest_path, self.perception_attempts_root):
+ raise PerceptionAdoptionError(
+ f"run_manifest escapes perception attempts root: {self.run_manifest_path}"
+ )
+
+ run_manifest = _read_json(self.run_manifest_path, "run_manifest")
+ if str(run_manifest.get("quality_status") or "") != "pass":
+ raise PerceptionAdoptionError(
+ "perception attempt quality_status must be pass before adoption"
+ )
+ delivery = run_manifest.get("sam3d_delivery")
+ if not isinstance(delivery, dict):
+ raise PerceptionAdoptionError("run_manifest.sam3d_delivery is missing")
+ status = str(delivery.get("status") or "")
+ if status not in READY_STATUSES:
+ raise PerceptionAdoptionError(
+ f"delivery status does not allow adoption: {status or 'missing'}"
+ )
+ if delivery.get("sam3d_execution_allowed") is not True:
+ raise PerceptionAdoptionError("sam3d_execution_allowed must be true")
+
+ base_dir = self.run_manifest_path.parent
+ output_dir = _resolve_confined(
+ delivery.get("output_dir"),
+ base_dir=base_dir,
+ trusted_root=self.perception_attempts_root,
+ label="sam3d_delivery.output_dir",
+ )
+ manifest_path = _resolve_confined(
+ delivery.get("manifest_path"),
+ base_dir=base_dir,
+ trusted_root=self.perception_attempts_root,
+ label="sam3d_delivery.manifest_path",
+ )
+ report_path = _resolve_confined(
+ delivery.get("delivery_report_path"),
+ base_dir=base_dir,
+ trusted_root=self.perception_attempts_root,
+ label="sam3d_delivery.delivery_report_path",
+ )
+ if not output_dir.is_dir():
+ raise PerceptionAdoptionError(f"delivery output_dir is missing: {output_dir}")
+ if manifest_path.parent != output_dir or report_path.parent != output_dir:
+ raise PerceptionAdoptionError(
+ "delivery manifest and report must be direct children of output_dir"
+ )
+ if not manifest_path.is_file() or not report_path.is_file():
+ raise PerceptionAdoptionError("delivery manifest or report is missing")
+
+ manifest = _read_json(manifest_path, "sam3d_input_manifest")
+ report = _read_json(report_path, "delivery_report")
+ if str(report.get("source_quality_status") or "") != "pass":
+ raise PerceptionAdoptionError(
+ "SAM3D delivery source_quality_status must be pass before adoption"
+ )
+ for label, payload in (("manifest", manifest), ("delivery_report", report)):
+ payload_status = payload.get("delivery_status") if label == "manifest" else payload.get("status")
+ if payload_status != status:
+ raise PerceptionAdoptionError(
+ f"{label} status mismatch: expected {status}, got {payload_status}"
+ )
+ if payload.get("sam3d_execution_allowed") is not True:
+ raise PerceptionAdoptionError(
+ f"{label}.sam3d_execution_allowed must be true"
+ )
+
+ manifest_objects = manifest.get("objects")
+ report_objects = report.get("objects")
+ if not isinstance(manifest_objects, list) or not isinstance(report_objects, list):
+ raise PerceptionAdoptionError("delivery objects must be arrays")
+ object_count = manifest.get("object_count")
+ counts = {
+ "run_manifest": delivery.get("object_count"),
+ "manifest": object_count,
+ "delivery_report": report.get("delivery_object_count"),
+ "manifest_objects": len(manifest_objects),
+ "report_objects": len(report_objects),
+ }
+ if not isinstance(object_count, int) or object_count <= 0:
+ raise PerceptionAdoptionError("manifest.object_count must be a positive integer")
+ if any(value != object_count for value in counts.values()):
+ raise PerceptionAdoptionError(f"delivery object_count mismatch: {counts}")
+
+ expected_ids = tuple(f"object_{index:03d}" for index in range(1, object_count + 1))
+ manifest_ids: list[str] = []
+ report_ids: list[str] = []
+ mask_paths: list[Path] = []
+ seen_mask_paths: set[Path] = set()
+ for index, item in enumerate(manifest_objects):
+ if not isinstance(item, dict):
+ raise PerceptionAdoptionError(f"manifest object {index} is not an object")
+ object_id = str(item.get("object_id") or "")
+ if not OBJECT_ID_RE.fullmatch(object_id):
+ raise PerceptionAdoptionError(f"invalid object_id: {object_id}")
+ if item.get("sam3d_internal_name") != object_id:
+ raise PerceptionAdoptionError(
+ f"sam3d_internal_name mismatch for {object_id}"
+ )
+ if item.get("order_index") != index:
+ raise PerceptionAdoptionError(
+ f"order_index mismatch for {object_id}: expected {index}"
+ )
+ mask_path = _resolve_confined(
+ item.get("mask_path"),
+ base_dir=manifest_path.parent,
+ trusted_root=self.perception_attempts_root,
+ label=f"{object_id}.mask_path",
+ )
+ if mask_path in seen_mask_paths:
+ raise PerceptionAdoptionError(f"duplicate mask path: {mask_path}")
+ seen_mask_paths.add(mask_path)
+ manifest_ids.append(object_id)
+ mask_paths.append(mask_path)
+
+ for index, item in enumerate(report_objects):
+ if not isinstance(item, dict):
+ raise PerceptionAdoptionError(f"delivery report object {index} is not an object")
+ object_id = str(item.get("object_id") or "")
+ if item.get("sam3d_internal_name") != object_id:
+ raise PerceptionAdoptionError(
+ f"delivery report sam3d_internal_name mismatch for {object_id}"
+ )
+ report_ids.append(object_id)
+
+ if tuple(manifest_ids) != expected_ids or tuple(report_ids) != expected_ids:
+ raise PerceptionAdoptionError(
+ "object IDs must be unique and continuous from object_001 to object_N"
+ )
+
+ image_path = _resolve_confined(
+ manifest.get("image_path"),
+ base_dir=manifest_path.parent,
+ trusted_root=self.perception_attempts_root,
+ label="manifest.image_path",
+ )
+ if not image_path.is_file():
+ raise PerceptionAdoptionError(f"delivery image is missing: {image_path}")
+ try:
+ with Image.open(image_path) as image:
+ image.load()
+ image_size = image.size
+ except (OSError, ValueError) as exc:
+ raise PerceptionAdoptionError(f"delivery image is unreadable: {image_path}") from exc
+ if list(image_size) != report.get("image_size"):
+ raise PerceptionAdoptionError(
+ f"delivery image size mismatch: {image_size} vs {report.get('image_size')}"
+ )
+ if _sha256(image_path) != _sha256(self.contract.input_image):
+ raise PerceptionAdoptionError(
+ "delivery image does not match formal input/image.png"
+ )
+
+ masks_dir = output_dir / "masks"
+ if not masks_dir.is_dir():
+ raise PerceptionAdoptionError(f"delivery masks directory is missing: {masks_dir}")
+ actual_mask_files = tuple(
+ sorted(item.resolve() for item in masks_dir.iterdir() if item.is_file())
+ )
+ if len(actual_mask_files) != object_count or set(actual_mask_files) != set(mask_paths):
+ raise PerceptionAdoptionError(
+ "actual mask files must exactly match manifest objects"
+ )
+
+ report_by_id = {str(item.get("object_id")): item for item in report_objects}
+ for item, mask_path in zip(manifest_objects, mask_paths):
+ object_id = str(item["object_id"])
+ if not mask_path.is_file() or mask_path.stat().st_size == 0:
+ raise PerceptionAdoptionError(f"mask is missing or empty: {mask_path}")
+ try:
+ with Image.open(mask_path) as mask:
+ mask.load()
+ if mask.size != image_size:
+ raise PerceptionAdoptionError(
+ f"mask size mismatch for {object_id}: {mask.size} vs {image_size}"
+ )
+ if mask.convert("L").getbbox() is None:
+ raise PerceptionAdoptionError(f"mask has no foreground: {object_id}")
+ except PerceptionAdoptionError:
+ raise
+ except (OSError, ValueError) as exc:
+ raise PerceptionAdoptionError(f"mask is unreadable: {mask_path}") from exc
+ actual_sha = _sha256(mask_path)
+ report_item = report_by_id[object_id]
+ if item.get("mask_sha256") != actual_sha or report_item.get("mask_sha256") != actual_sha:
+ raise PerceptionAdoptionError(f"mask SHA-256 mismatch for {object_id}")
+
+ quality_report_path = _resolve_confined(
+ manifest.get("source_quality_report"),
+ base_dir=manifest_path.parent,
+ trusted_root=self.perception_attempts_root,
+ label="manifest.source_quality_report",
+ )
+ if not quality_report_path.is_file():
+ raise PerceptionAdoptionError(
+ f"source quality report is missing: {quality_report_path}"
+ )
+ _read_json(quality_report_path, "source quality report")
+
+ run_info_path = output_dir / "sam3d_input_run_info.json"
+ if not run_info_path.is_file():
+ run_info_path = None
+
+ return _ValidatedDelivery(
+ run_manifest_path=self.run_manifest_path,
+ run_manifest=run_manifest,
+ run_manifest_sha256=_sha256(self.run_manifest_path),
+ output_dir=output_dir,
+ source_manifest_path=manifest_path,
+ source_manifest=manifest,
+ delivery_report_path=report_path,
+ delivery_report=report,
+ run_info_path=run_info_path,
+ quality_report_path=quality_report_path,
+ image_path=image_path,
+ image_size=image_size,
+ mask_paths=tuple(mask_paths),
+ object_ids=expected_ids,
+ status=status,
+ )
+
+ def _reuse_existing(
+ self, delivery: _ValidatedDelivery
+ ) -> PerceptionAdoptionArtifacts | None:
+ stage03 = self.contract.stage03_dir
+ stage04 = self.contract.stage04_dir
+ stage03_has_files = _directory_has_files(stage03)
+ stage04_has_files = _directory_has_files(stage04)
+ if not stage03_has_files and not stage04_has_files:
+ return None
+ if stage03_has_files != stage04_has_files:
+ raise PerceptionAdoptionError(
+ "formal Stage 03/04 are partially populated; refusing implicit recovery"
+ )
+
+ adoption_path = stage03 / "perception_adoption.json"
+ adoption = _read_json(adoption_path, "existing perception adoption")
+ source = adoption.get("source")
+ if not isinstance(source, dict) or source.get("run_manifest_sha256") != delivery.run_manifest_sha256:
+ raise PerceptionAdoptionError(
+ "formal Stage 03/04 already contain a different adoption"
+ )
+ formal_artifacts = adoption.get("formal_artifacts")
+ if not isinstance(formal_artifacts, list):
+ raise PerceptionAdoptionError("existing adoption lacks formal artifact hashes")
+ for record in formal_artifacts:
+ if not isinstance(record, dict):
+ raise PerceptionAdoptionError("existing formal artifact record is invalid")
+ path = Path(str(record.get("path") or "")).expanduser().resolve()
+ expected_sha = record.get("sha256")
+ if not path.is_file() or not isinstance(expected_sha, str) or _sha256(path) != expected_sha:
+ raise PerceptionAdoptionError(
+ f"existing formal artifact hash mismatch: {path}"
+ )
+
+ manifest = _read_json(stage04 / "sam3d_input_manifest.json", "formal Stage 04 manifest")
+ existing_ids = tuple(str(item.get("object_id") or "") for item in manifest.get("objects", []))
+ if existing_ids != delivery.object_ids:
+ raise PerceptionAdoptionError("existing adoption object ID closure mismatch")
+
+ return PerceptionAdoptionArtifacts(
+ stage03_dir=stage03,
+ stage04_dir=stage04,
+ reviewed_objects_path=stage03 / "reviewed_objects.json",
+ sam3d_manifest_path=stage04 / "sam3d_input_manifest.json",
+ adoption_report_path=adoption_path,
+ delivery_status=delivery.status,
+ object_ids=delivery.object_ids,
+ reused=True,
+ )
+
+ def _materialize(
+ self,
+ delivery: _ValidatedDelivery,
+ stage03_tmp: Path,
+ stage04_tmp: Path,
+ ) -> tuple[str, ...]:
+ final_stage03 = self.contract.stage03_dir
+ final_stage04 = self.contract.stage04_dir
+ stage03_masks = stage03_tmp / "reviewed_masks"
+ stage03_overlays = stage03_tmp / "reviewed_overlays"
+ stage04_masks = stage04_tmp / "masks"
+ stage03_masks.mkdir(parents=True)
+ stage03_overlays.mkdir(parents=True)
+ stage04_masks.mkdir(parents=True)
+
+ image_tmp = stage04_tmp / "image.png"
+ shutil.copy2(delivery.image_path, image_tmp)
+ source_image_sha = _sha256(delivery.image_path)
+ if _sha256(image_tmp) != source_image_sha:
+ raise PerceptionAdoptionError("Stage 04 image copy hash mismatch")
+
+ source_objects = delivery.source_manifest["objects"]
+ reviewed_objects: list[dict[str, Any]] = []
+ formal_manifest = copy.deepcopy(delivery.source_manifest)
+ formal_manifest["image_path"] = str(final_stage04 / "image.png")
+ mask_hash_comparison: list[dict[str, Any]] = []
+
+ with Image.open(delivery.image_path) as source_image:
+ base_rgba = source_image.convert("RGBA")
+ for index, (source_object, source_mask) in enumerate(
+ zip(source_objects, delivery.mask_paths)
+ ):
+ object_id = delivery.object_ids[index]
+ formal_mask_tmp = stage04_masks / source_mask.name
+ formal_mask_path = final_stage04 / "masks" / source_mask.name
+ shutil.copy2(source_mask, formal_mask_tmp)
+ source_mask_sha = _sha256(source_mask)
+ formal_mask_sha = _sha256(formal_mask_tmp)
+ if formal_mask_sha != source_mask_sha:
+ raise PerceptionAdoptionError(
+ f"Stage 04 mask copy hash mismatch for {object_id}"
+ )
+
+ category = re.sub(
+ r"[^0-9A-Za-z]+",
+ "_",
+ str(source_object.get("category") or "object"),
+ ).strip("_").lower() or "object"
+ reviewed_mask_name = f"{object_id}_{category}_mask.png"
+ reviewed_overlay_name = f"{object_id}_{category}_overlay.png"
+ reviewed_mask_tmp = stage03_masks / reviewed_mask_name
+ reviewed_overlay_tmp = stage03_overlays / reviewed_overlay_name
+ shutil.copy2(source_mask, reviewed_mask_tmp)
+
+ with Image.open(source_mask) as mask_image:
+ mask_l = mask_image.convert("L")
+ alpha = mask_l.point(lambda value: 112 if value else 0)
+ tint = Image.new("RGBA", delivery.image_size, (255, 32, 32, 0))
+ tint.putalpha(alpha)
+ overlay = Image.alpha_composite(base_rgba, tint).convert("RGB")
+ overlay.save(
+ reviewed_overlay_tmp,
+ format="PNG",
+ optimize=False,
+ compress_level=1,
+ )
+
+ formal_manifest["objects"][index]["mask_path"] = str(formal_mask_path)
+ reviewed = copy.deepcopy(source_object)
+ sam3 = reviewed.get("sam3")
+ if not isinstance(sam3, dict):
+ sam3 = {}
+ sam3 = dict(sam3)
+ sam3["mask_path"] = str(formal_mask_path)
+ sam3["overlay_path"] = str(
+ final_stage03 / "reviewed_overlays" / reviewed_overlay_name
+ )
+ reviewed["sam3"] = sam3
+ reviewed["review_mode"] = "automatic_perception_delivery"
+ reviewed["human_review_required"] = False
+ reviewed_objects.append(reviewed)
+ mask_hash_comparison.append(
+ {
+ "object_id": object_id,
+ "source_path": str(source_mask),
+ "source_sha256": source_mask_sha,
+ "formal_path": str(formal_mask_path),
+ "formal_sha256": formal_mask_sha,
+ "identical": source_mask_sha == formal_mask_sha,
+ }
+ )
+
+ reviewed_payload = {
+ "schema_version": "scenemosaic_reviewed_objects",
+ "scene_name": self.contract.scene_name,
+ "review_mode": "automatic_perception_delivery",
+ "human_review_required": False,
+ "source_delivery_status": delivery.status,
+ "object_count": len(reviewed_objects),
+ "objects": reviewed_objects,
+ }
+ _write_json(stage03_tmp / "reviewed_objects.json", reviewed_payload)
+ _write_json(stage04_tmp / "sam3d_input_manifest.json", formal_manifest)
+ shutil.copy2(delivery.quality_report_path, stage03_tmp / "quality_report.json")
+ shutil.copy2(delivery.delivery_report_path, stage04_tmp / "delivery_report.json")
+ if delivery.run_info_path is not None:
+ shutil.copy2(delivery.run_info_path, stage04_tmp / "sam3d_input_run_info.json")
+ else:
+ _write_json(
+ stage04_tmp / "sam3d_input_run_info.json",
+ {
+ "schema_version": "scenemosaic_sam3d_delivery_run_info",
+ "scene_name": self.contract.scene_name,
+ "status": delivery.status,
+ "sam3d_execution_allowed": True,
+ "object_count": len(reviewed_objects),
+ "created_at": _utc_now(),
+ "source_quality_status": delivery.delivery_report.get("source_quality_status"),
+ "excluded_object_count": len(delivery.delivery_report.get("excluded_objects", [])),
+ "duplicate_collapse_count": len(delivery.delivery_report.get("duplicate_collapses", [])),
+ "infrastructure_error_count": len(delivery.delivery_report.get("infrastructure_errors", [])),
+ },
+ )
+
+ formal_records: list[dict[str, str]] = []
+ for temporary_root, final_root in (
+ (stage03_tmp, final_stage03),
+ (stage04_tmp, final_stage04),
+ ):
+ for path in sorted(item for item in temporary_root.rglob("*") if item.is_file()):
+ formal_records.append(
+ {
+ "path": str(final_root / path.relative_to(temporary_root)),
+ "sha256": _sha256(path),
+ }
+ )
+
+ adoption_report = {
+ "schema_version": "scenemosaic_perception_adoption",
+ "created_at_utc": _utc_now(),
+ "scene_name": self.contract.scene_name,
+ "delivery_status": delivery.status,
+ "sam3d_execution_allowed": True,
+ "review_mode": "automatic_perception_delivery",
+ "human_review_required": False,
+ "renumbered": False,
+ "source": {
+ "perception_attempts_root": str(self.perception_attempts_root),
+ "run_manifest_path": str(delivery.run_manifest_path),
+ "run_manifest_sha256": delivery.run_manifest_sha256,
+ "sam3d_input_manifest_path": str(delivery.source_manifest_path),
+ "sam3d_input_manifest_sha256": _sha256(delivery.source_manifest_path),
+ "delivery_report_path": str(delivery.delivery_report_path),
+ "delivery_report_sha256": _sha256(delivery.delivery_report_path),
+ "quality_report_path": str(delivery.quality_report_path),
+ "quality_report_sha256": _sha256(delivery.quality_report_path),
+ "image_path": str(delivery.image_path),
+ "image_sha256": source_image_sha,
+ },
+ "source_models": {
+ "pass_a": delivery.run_manifest.get("pass_a_model"),
+ "pass_b": delivery.run_manifest.get("pass_b_model"),
+ },
+ "object_id_closure": {
+ "manifest": list(delivery.object_ids),
+ "delivery_report": list(delivery.object_ids),
+ "reviewed_objects": list(delivery.object_ids),
+ "closed": True,
+ },
+ "mask_hash_comparison": mask_hash_comparison,
+ "excluded_objects": delivery.delivery_report.get("excluded_objects", []),
+ "duplicate_collapses": delivery.delivery_report.get("duplicate_collapses", []),
+ "infrastructure_errors": delivery.delivery_report.get("infrastructure_errors", []),
+ "source_warning_count": delivery.delivery_report.get("source_warning_count", 0),
+ "per_object_physics_consumed": False,
+ "formal_artifacts": formal_records,
+ }
+ _write_json(stage03_tmp / "perception_adoption.json", adoption_report)
+ return delivery.object_ids
+
+ @staticmethod
+ def _publish_pair(
+ stage03_tmp: Path,
+ stage04_tmp: Path,
+ stage03: Path,
+ stage04: Path,
+ ) -> None:
+ stage03_snapshot = _empty_tree_snapshot(stage03)
+ stage04_snapshot = _empty_tree_snapshot(stage04)
+ _remove_empty_tree(stage03)
+ _remove_empty_tree(stage04)
+ stage03_published = False
+ stage04_published = False
+ try:
+ os.replace(stage03_tmp, stage03)
+ stage03_published = True
+ os.replace(stage04_tmp, stage04)
+ stage04_published = True
+ except Exception as exc:
+ if stage04_published and stage04.exists():
+ os.replace(stage04, stage04_tmp)
+ if stage03_published and stage03.exists():
+ os.replace(stage03, stage03_tmp)
+ _restore_empty_tree(stage03, stage03_snapshot)
+ _restore_empty_tree(stage04, stage04_snapshot)
+ raise PerceptionAdoptionError(
+ "atomic Stage 03/04 publication failed and was rolled back"
+ ) from exc
diff --git a/vlm_agent/reconstruction/perception/inventory/__init__.py b/vlm_agent/reconstruction/perception/inventory/__init__.py
new file mode 100644
index 0000000..86c522e
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/inventory/__init__.py
@@ -0,0 +1,10 @@
+"""Two-pass visible-object inventory used by SceneMosaic perception."""
+
+from .normalization import normalize_objects, parse_vlm_response_text
+from .review_assets import render_bbox_review_assets
+
+__all__ = [
+ "normalize_objects",
+ "parse_vlm_response_text",
+ "render_bbox_review_assets",
+]
diff --git a/vlm_agent/reconstruction/perception/inventory/image_assets.py b/vlm_agent/reconstruction/perception/inventory/image_assets.py
new file mode 100644
index 0000000..84fcf37
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/inventory/image_assets.py
@@ -0,0 +1,86 @@
+"""Small, reproducible image assets used only for VLM API requests.
+
+The reconstruction pipeline keeps its original images and PNG masks unchanged.
+These helpers create smaller JPEG derivatives so the same full-resolution image
+is not uploaded repeatedly during object review.
+"""
+
+import base64
+from pathlib import Path
+
+
+def create_vlm_jpeg(
+ source_path,
+ output_path,
+ max_long_edge=1280,
+ quality=85,
+ coordinate_grid=False,
+ grid_divisions=10,
+):
+ """Create a bounded RGB JPEG, optionally with a lightweight coordinate ruler."""
+ from PIL import Image
+
+ source_path = Path(source_path)
+ output_path = Path(output_path)
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+
+ with Image.open(source_path) as source:
+ image = source.convert("RGB")
+ width, height = image.size
+ longest = max(width, height)
+ if longest > max_long_edge:
+ scale = max_long_edge / longest
+ image = image.resize(
+ (round(width * scale), round(height * scale)),
+ Image.Resampling.LANCZOS,
+ )
+ if coordinate_grid:
+ from PIL import ImageDraw, ImageFont
+
+ if grid_divisions < 2:
+ raise ValueError("grid_divisions must be at least 2")
+ width_out, height_out = image.size
+ overlay = Image.new("RGBA", image.size, (0, 0, 0, 0))
+ draw = ImageDraw.Draw(overlay)
+ try:
+ font = ImageFont.truetype("DejaVuSans.ttf", max(12, width_out // 100))
+ except OSError:
+ font = ImageFont.load_default()
+ for index in range(grid_divisions + 1):
+ x = round(index * (width_out - 1) / grid_divisions)
+ y = round(index * (height_out - 1) / grid_divisions)
+ alpha = 100 if index % 2 == 0 else 55
+ draw.line((x, 0, x, height_out - 1), fill=(0, 205, 255, alpha), width=1)
+ draw.line((0, y, width_out - 1, y), fill=(0, 205, 255, alpha), width=1)
+ if index < grid_divisions:
+ label = f"{index / grid_divisions:.1f}"
+ top_box = draw.textbbox((x + 3, 1), label, font=font)
+ left_box = draw.textbbox((3, y + 1), label, font=font)
+ draw.rectangle(top_box, fill=(0, 0, 0, 145))
+ draw.rectangle(left_box, fill=(0, 0, 0, 145))
+ draw.text((x + 3, 1), label, fill=(255, 255, 255, 235), font=font)
+ draw.text((3, y + 1), label, fill=(255, 255, 255, 235), font=font)
+ image = Image.alpha_composite(image.convert("RGBA"), overlay).convert("RGB")
+ image.save(output_path, format="JPEG", quality=quality, optimize=True)
+
+ return {
+ "path": str(output_path),
+ "mime_type": "image/jpeg",
+ "source_path": str(source_path),
+ "source_bytes": source_path.stat().st_size,
+ "output_bytes": output_path.stat().st_size,
+ "source_size": [width, height],
+ "output_size": list(image.size),
+ "max_long_edge": max_long_edge,
+ "quality": quality,
+ "coordinate_grid": {
+ "enabled": bool(coordinate_grid),
+ "divisions": int(grid_divisions) if coordinate_grid else None,
+ },
+ }
+
+
+def encode_data_url(path, mime_type="image/jpeg"):
+ with open(path, "rb") as handle:
+ image_b64 = base64.b64encode(handle.read()).decode("utf-8")
+ return f"data:{mime_type};base64,{image_b64}"
diff --git a/vlm_agent/reconstruction/perception/inventory/normalization.py b/vlm_agent/reconstruction/perception/inventory/normalization.py
new file mode 100644
index 0000000..c099369
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/inventory/normalization.py
@@ -0,0 +1,157 @@
+"""Parse and normalize VLM object-inventory responses."""
+
+from __future__ import annotations
+
+import json
+import re
+from typing import Any
+
+
+def parse_vlm_response_text(
+ text: str,
+ *,
+ finish_reason: str | None = None,
+) -> Any:
+ """Parse a JSON response and distinguish truncation from malformed output."""
+
+ text = text.strip()
+ if text.startswith("```"):
+ text = text.removeprefix("```json").removeprefix("```").removesuffix("```").strip()
+ try:
+ return json.loads(text)
+ except json.JSONDecodeError as exc:
+ failure_kind = "provider-truncated" if finish_reason == "length" else "malformed"
+ raise RuntimeError(
+ "VLM returned "
+ f"{failure_kind} JSON: finish_reason={finish_reason!r}, "
+ f"json_error={exc}"
+ ) from exc
+
+
+def _format_object_id(index: int, object_id_format: str) -> str:
+ try:
+ object_id = object_id_format.format(index=index)
+ except Exception as exc:
+ raise ValueError(
+ f"Invalid object_id_format {object_id_format!r}: {exc}"
+ ) from exc
+
+ if not object_id:
+ raise ValueError("object_id_format produced an empty object_id.")
+ return object_id
+
+
+def _grid_location_from_anchor(anchor: list[float]) -> tuple[str, dict[str, int]]:
+ x, y = anchor
+ row = min(10, max(1, int(y * 10) + 1))
+ col = min(10, max(1, int(x * 10) + 1))
+ return f"grid_r{row}_c{col}", {"row": row, "col": col}
+
+
+def _normalize_anchor(
+ anchor: Any,
+ bbox: list[float],
+ object_id: str,
+) -> tuple[list[float], str]:
+ if anchor is None:
+ return [
+ round((float(bbox[0]) + float(bbox[2])) / 2, 3),
+ round((float(bbox[1]) + float(bbox[3])) / 2, 3),
+ ], "derived_bbox_center"
+ if not isinstance(anchor, list) or len(anchor) != 2:
+ raise ValueError(f"Invalid anchor for {object_id}: {anchor}")
+ x, y = anchor
+ if not (0 <= x <= 1 and 0 <= y <= 1):
+ raise ValueError(f"Anchor out of range for {object_id}: {anchor}")
+ if not (bbox[0] <= x <= bbox[2] and bbox[1] <= y <= bbox[3]):
+ raise ValueError(f"Anchor lies outside bbox for {object_id}: {anchor}")
+ return [round(float(x), 3), round(float(y), 3)], "vlm"
+
+
+def normalize_objects(
+ objects: list[dict[str, Any]],
+ object_id_base: int,
+ object_id_format: str,
+) -> list[dict[str, Any]]:
+ """Assign stable IDs and validate inventory geometry without changing meaning."""
+
+ normalized = []
+ seen_ids = set()
+
+ for offset, obj in enumerate(objects):
+ object_id = _format_object_id(object_id_base + offset, object_id_format)
+ if object_id in seen_ids:
+ raise ValueError(f"Duplicate object_id generated: {object_id}")
+ seen_ids.add(object_id)
+
+ normalized_obj = dict(obj)
+ normalized_obj.pop("image_object_id", None)
+ normalized_obj["object_id"] = object_id
+
+ semantic_tokens = set(
+ re.findall(
+ r"[a-z0-9]+",
+ " ".join(
+ str(normalized_obj.get(key) or "")
+ for key in (
+ "category",
+ "sam3_prompt",
+ "instance_name",
+ "description",
+ )
+ ).lower(),
+ )
+ )
+ if (
+ {"water", "dispenser"} <= semantic_tokens
+ and semantic_tokens & {"bottle", "bottled", "jug", "reservoir"}
+ ):
+ previous_prompt = str(normalized_obj.get("sam3_prompt") or "")
+ normalized_obj["sam3_prompt"] = (
+ "complete freestanding water dispenser with water bottle on top"
+ )
+ normalized_obj["sam3_prompt_policy"] = {
+ "rule": "complete_mounted_water_bottle",
+ "previous_prompt": previous_prompt,
+ }
+
+ bbox = normalized_obj.get("bbox_2d_norm_xyxy")
+ if not isinstance(bbox, list) or len(bbox) != 4:
+ raise ValueError(f"Invalid bbox for {object_id}: {bbox}")
+
+ x1, y1, x2, y2 = bbox
+ if not (0 <= x1 <= 1 and 0 <= y1 <= 1 and 0 <= x2 <= 1 and 0 <= y2 <= 1):
+ raise ValueError(f"BBox out of range for {object_id}: {bbox}")
+ if x2 <= x1 or y2 <= y1:
+ raise ValueError(f"Invalid bbox order for {object_id}: {bbox}")
+ normalized_obj["bbox_2d_norm_xyxy"] = [round(float(value), 3) for value in bbox]
+
+ anchor, anchor_source = _normalize_anchor(
+ normalized_obj.get("anchor_2d_norm_xy"),
+ normalized_obj["bbox_2d_norm_xyxy"],
+ object_id,
+ )
+ normalized_obj["anchor_2d_norm_xy"] = anchor
+ normalized_obj["anchor_source"] = anchor_source
+
+ derived_location, derived_order = _grid_location_from_anchor(anchor)
+ if not isinstance(normalized_obj.get("coarse_location"), str) or not normalized_obj.get("coarse_location", "").strip():
+ normalized_obj["coarse_location"] = derived_location
+ if not isinstance(normalized_obj.get("visual_order"), dict):
+ normalized_obj["visual_order"] = derived_order
+
+ # Technical SAM3 candidate assignment belongs to the segmentation stage.
+ normalized_obj.pop("segmentation_mode", None)
+
+ fallback_prompts = normalized_obj.get("fallback_prompts", [])
+ if fallback_prompts is None:
+ fallback_prompts = []
+ if not isinstance(fallback_prompts, list):
+ raise ValueError(f"Invalid fallback_prompts for {object_id}: {fallback_prompts}")
+ normalized_obj["fallback_prompts"] = [
+ str(prompt) for prompt in fallback_prompts[:1] if str(prompt).strip()
+ ]
+
+ normalized.append(normalized_obj)
+
+ return normalized
diff --git a/vlm_agent/reconstruction/perception/inventory/pipeline.py b/vlm_agent/reconstruction/perception/inventory/pipeline.py
new file mode 100644
index 0000000..6e80bed
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/inventory/pipeline.py
@@ -0,0 +1,2044 @@
+"""Production two-pass VLM object inventory.
+
+Pass A inventories primary reconstruction units in the whole image. Pass B
+receives the same image, compact Pass-A registry, and optional segmentation
+evidence, then reports missing units, corrections, and removals. Local code
+validates and fuses both responses into provisional reconstruction objects.
+"""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+import math
+import re
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from .image_assets import create_vlm_jpeg, encode_data_url
+from .normalization import normalize_objects, parse_vlm_response_text
+from .review_assets import render_bbox_review_assets
+
+
+ALLOWED_TYPES = {
+ "furniture",
+ "decor",
+ "small_object",
+ "structural_surface",
+ "opening",
+ "textile",
+ "lighting",
+ "plant",
+ "appliance",
+ "unknown",
+}
+
+LOCAL_CANDIDATE_ACTIONS = {
+ "bind_to_addition",
+ "replace_existing",
+ "merge_extend_existing",
+ "duplicate_of_existing",
+ "part_of_existing",
+ "reject",
+ "uncertain",
+}
+LOCAL_CANDIDATE_EXISTING_TARGET_ACTIONS = {
+ "replace_existing",
+ "merge_extend_existing",
+ "duplicate_of_existing",
+ "part_of_existing",
+}
+LOCAL_CANDIDATE_NULL_TARGET_ACTIONS = {"reject", "uncertain"}
+
+ASSIGNED_EVIDENCE_DECISIONS = {
+ "pass",
+ "update_existing",
+ "split_instances",
+ "remove_existing",
+ "uncertain",
+}
+
+_COMPACT_NUMBER = r"[-+]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][-+]?\d+)?"
+_COMPACT_BARE_BBOX_AFTER_PROMPT = re.compile(
+ rf'("p"\s*:\s*"(?:[^"\\]|\\.)*")\s*,\s*('
+ rf'\[\s*{_COMPACT_NUMBER}\s*,\s*{_COMPACT_NUMBER}\s*,\s*'
+ rf'{_COMPACT_NUMBER}\s*,\s*{_COMPACT_NUMBER}\s*\])'
+)
+
+
+def _repair_compact_bare_bbox_after_prompt(text: str) -> tuple[str, int]:
+ """Restore the only unambiguous omitted key seen in compact additions."""
+
+ return _COMPACT_BARE_BBOX_AFTER_PROMPT.subn(r'\1,"b":\2', text)
+
+
+PASS_A_PROMPT = r"""
+You are Pass A of a visible-object registry for SAM3 + SAM3D reconstruction.
+Inspect the entire image, which has a cyan global coordinate grid. This pass
+builds the PRIMARY inventory, not the final exhaustive inventory.
+
+Register each visible primary physical reconstruction unit once:
+- furniture, supports, storage, large textiles and wall carriers;
+- visible appliances, plants, lighting, fixtures and salient medium objects;
+- functional assemblies that would be incomplete if separated.
+
+Do not spend the response on truly tiny loose surface details. Register each visible
+open-storage carrier once; its small movable contents are reviewed in Pass B with
+high-resolution content-host evidence. This exception does not apply to objects on
+ordinary tables, counters or other support surfaces.
+Always enumerate every visible recessed ceiling light. Include a small object
+when it is visually distinct or is itself a primary scene unit. Exclude bare
+wall, floor, ceiling, windows, doors, curtains,
+background, and objects depicted inside paintings, photos, mirrors, televisions
+or screens. This is not
+relationship extraction; output no relations, scene-graph edges or 3D pose.
+
+Object-unit rules:
+- Mere contact does not merge independent objects.
+- A visible gap, different shelf/cubby/container, or different pose starts a
+ different unit.
+- A bed includes frame, mattress, fitted sheet and visible duvet/blanket, while
+ each loose pillow is separate.
+- A potted plant is one assembly: include both foliage and its visible pot or
+ planter. Use c=potted_plant and make p name the plant together with its pot.
+- Keep fitted or dedicated parts with their assembly when separation would
+ produce an incomplete reconstruction: join a sink, faucet and dedicated
+ supporting vanity/counter; a fitted undercounter dishwasher belongs to that
+ dedicated sink-counter assembly rather than becoming a separate unit. A visible
+ range/stove interrupts a counter run: never widen one counter bbox across it;
+ register the independent counter section on the other side separately. Join an
+ attached water bottle with its dispenser; join connected lamp heads/rods; join a
+ coat rack, towel ladder/rack, drying
+ rack, roll holder/rack or pegboard with its hanging
+ or dedicated contents. Do not absorb independent loose objects merely due to
+ contact.
+
+For each unit provide one coarse bbox. Identity and coverage matter more than
+exact bbox edges; never widen a bbox to hedge across neighboring units. Category
+c is a generic machine identifier for the visible physical unit: use lowercase
+snake_case only (for example vase, book_stack, ceiling_light), with no spaces,
+hyphens, color, material, size or location terms. Field p is the SAM3 semantic
+text prompt: it must name the visible object itself, optionally with color,
+material or form. Never put a location, support, direction, relation, grid cell
+or nearby object in c or p.
+
+Return ONLY one minified JSON object:
+{"o":[{"c":"snake_case category","p":"2-5 object-only English words","t":"furniture|decor|small_object|structural_surface|opening|textile|lighting|plant|appliance|unknown","b":[0.000,0.000,0.000,0.000]}]}
+
+Use normalized whole-image coordinates. ``b`` is exactly
+``[left, top, right, bottom]`` (also called ``[x1, y1, x2, y2]``): require
+``left < right`` and ``top < bottom``. Never use ``[x, y, width, height]``.
+Do not return a name, description or anchor: local code deterministically
+derives those from category and bbox. Do not output any other keys or prose.
+""".strip()
+
+
+
+
+COMPACT_PASS_B_PROTOCOL = "compact_pass_b"
+
+
+
+
+
+
+def utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def read_json(path: Path) -> dict[str, Any]:
+ value = json.loads(path.expanduser().resolve().read_text(encoding="utf-8"))
+ if not isinstance(value, dict):
+ raise ValueError(f"expected JSON object: {path}")
+ return value
+
+
+def write_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+
+
+def sha256_file(path: Path) -> str:
+ return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def load_scene_config(path: Path) -> dict[str, Any]:
+ config_path = path.expanduser().resolve()
+ config = read_json(config_path)
+ image_path = Path(str(config["input_image"])).expanduser().resolve()
+ run_root = Path(str(config["run_root"])).expanduser().resolve()
+ if not image_path.is_file():
+ raise FileNotFoundError(image_path)
+ return {
+ "scene_name": str(config.get("scene_name") or run_root.name),
+ "run_root": run_root,
+ "image_path": image_path,
+ "config_path": config_path,
+ }
+
+
+def ensure_attempt_output(output_dir: Path, run_root: Path) -> None:
+ resolved = output_dir.expanduser().resolve()
+ if "perception_attempts" not in resolved.parts:
+ raise ValueError("output-dir must be inside a perception_attempts directory")
+ formal_stages = {
+ (run_root / name).resolve()
+ for name in ("input", "reconstruction", "evolution")
+ }
+ if any(resolved == stage or stage in resolved.parents for stage in formal_stages):
+ raise ValueError("refusing to write inside a formal pipeline stage")
+ if resolved.exists() and any(resolved.iterdir()):
+ raise FileExistsError(f"perception attempt output is not empty: {resolved}")
+ resolved.mkdir(parents=True, exist_ok=True)
+
+
+def valid_bbox(value: Any) -> bool:
+ return (
+ isinstance(value, list)
+ and len(value) == 4
+ and all(isinstance(item, (int, float)) and math.isfinite(item) for item in value)
+ and 0 <= value[0] < value[2] <= 1
+ and 0 <= value[1] < value[3] <= 1
+ )
+
+
+def valid_anchor(value: Any, bbox: list[float]) -> bool:
+ return (
+ isinstance(value, list)
+ and len(value) == 2
+ and all(isinstance(item, (int, float)) and math.isfinite(item) for item in value)
+ and bbox[0] <= value[0] <= bbox[2]
+ and bbox[1] <= value[1] <= bbox[3]
+ )
+
+
+def _normalize_category_identifier(value: Any, *, field_name: str) -> str:
+ """Return a stable category identifier without touching SAM3 prompt text."""
+
+ if not isinstance(value, str):
+ raise ValueError(f"{field_name} must be a string")
+ # VLMs occasionally express an otherwise valid multi-word category as
+ # prose (for example, "dark vase"). Normalize only harmless spelling
+ # variants; punctuation other than hyphens remains a schema error.
+ category = re.sub(r"[\s-]+", "_", value.strip().lower())
+ category = re.sub(r"_+", "_", category).strip("_")
+ if not re.fullmatch(r"[a-z][a-z0-9_]{0,63}", category):
+ raise ValueError(f"invalid {field_name}: {value!r}")
+ return category
+
+
+def validate_semantic_record(record: dict[str, Any], *, anchor_key: str = "a") -> None:
+ required = {"c", "p", "n", "t", "b", anchor_key, "d"}
+ if set(record) != required:
+ raise ValueError(
+ f"object keys must be exactly {sorted(required)}, got {sorted(record)}"
+ )
+ record["c"] = _normalize_category_identifier(record["c"], field_name="category")
+ if str(record["t"]) not in ALLOWED_TYPES:
+ raise ValueError(f"invalid object type: {record['t']!r}")
+ for key in ("p", "n", "d"):
+ if not str(record[key]).strip():
+ raise ValueError(f"empty semantic field: {key}")
+ if not valid_bbox(record["b"]) or not valid_anchor(record[anchor_key], record["b"]):
+ raise ValueError("invalid bbox or anchor")
+
+
+def expand_record(record: dict[str, Any], *, anchor_key: str = "a") -> dict[str, Any]:
+ record = dict(record)
+ if str(record.get("t")) not in ALLOWED_TYPES:
+ record["t"] = "unknown"
+ validate_semantic_record(record, anchor_key=anchor_key)
+ return {
+ "category": str(record["c"]).strip(),
+ "sam3_prompt": semantic_sam3_prompt(record["c"], record["p"]),
+ "instance_name": str(record["n"]).strip(),
+ "object_type": str(record["t"]).strip(),
+ "bbox_2d_norm_xyxy": [round(float(value), 3) for value in record["b"]],
+ "anchor_2d_norm_xy": [
+ round(float(value), 3) for value in record[anchor_key]
+ ],
+ "description": str(record["d"]).strip(),
+ "fallback_prompts": [],
+ }
+
+
+def _validate_compact_category(value: Any) -> str:
+ return _normalize_category_identifier(value, field_name="compact category")
+
+
+def _validate_compact_prompt(value: Any) -> str:
+ if not isinstance(value, str):
+ raise ValueError("compact SAM3 prompt must be a string")
+ prompt = " ".join(value.split())
+ if not prompt or len(prompt) > 80:
+ raise ValueError("compact SAM3 prompt must contain 1-80 characters")
+ return prompt
+
+
+_SAM3_LOCATION_TOKENS = {
+ "above",
+ "across",
+ "against",
+ "at",
+ "behind",
+ "beneath",
+ "beside",
+ "between",
+ "center",
+ "central",
+ "corner",
+ "far",
+ "front",
+ "inside",
+ "left",
+ "middle",
+ "near",
+ "on",
+ "outside",
+ "rear",
+ "right",
+ "under",
+}
+
+
+def semantic_sam3_prompt(category: Any, prompt: Any) -> str:
+ """Return an instance-level SAM3 prompt with required assembly semantics.
+
+ The prompting contract asks models for object-only text, but retaining an
+ occasional positional qualifier is safer than silently replacing a
+ specific instance or group with its broad category. SAM3 assignment still
+ uses the corresponding local bounding box to distinguish repeated items.
+ """
+
+ normalized_category = _validate_compact_category(category)
+ normalized_prompt = _validate_compact_prompt(prompt)
+ if normalized_category != "potted_plant":
+ return normalized_prompt
+ prompt_tokens = set(re.findall(r"[a-z0-9]+", normalized_prompt.lower()))
+ if prompt_tokens & {"pot", "planter", "container"}:
+ return normalized_prompt
+ return f"{normalized_prompt} with pot"
+
+
+def _bbox_center(bbox: list[float]) -> list[float]:
+ return [
+ round((float(bbox[0]) + float(bbox[2])) / 2.0, 6),
+ round((float(bbox[1]) + float(bbox[3])) / 2.0, 6),
+ ]
+
+
+def _minimal_semantic_record(
+ *,
+ category: Any,
+ prompt: Any,
+ bbox: Any,
+ anchor: Any | None = None,
+ object_type: Any = "unknown",
+) -> dict[str, Any]:
+ normalized_category = _validate_compact_category(category)
+ normalized_prompt = semantic_sam3_prompt(normalized_category, prompt)
+ normalized_object_type = str(object_type).strip()
+ if normalized_object_type not in ALLOWED_TYPES:
+ raise ValueError(f"invalid compact object type: {object_type!r}")
+ if not valid_bbox(bbox):
+ raise ValueError("invalid compact bbox")
+ normalized_bbox = [round(float(value), 6) for value in bbox]
+ normalized_anchor = _bbox_center(normalized_bbox) if anchor is None else [
+ round(float(value), 6) for value in anchor
+ ]
+ if not valid_anchor(normalized_anchor, normalized_bbox):
+ raise ValueError("invalid locally derived compact anchor")
+ center_x, center_y = normalized_anchor
+ row = min(10, max(1, int(center_y * 10) + 1))
+ column = min(10, max(1, int(center_x * 10) + 1))
+ return {
+ "category": normalized_category,
+ "sam3_prompt": normalized_prompt,
+ "instance_name": (
+ f"{normalized_category.replace('_', ' ')} r{row}c{column}"
+ ),
+ "object_type": normalized_object_type,
+ "bbox_2d_norm_xyxy": normalized_bbox,
+ "anchor_2d_norm_xy": normalized_anchor,
+ "description": normalized_prompt,
+ "fallback_prompts": [],
+ }
+
+
+def _semantic_record_copy(record: dict[str, Any]) -> dict[str, Any]:
+ keys = (
+ "category",
+ "sam3_prompt",
+ "instance_name",
+ "object_type",
+ "bbox_2d_norm_xyxy",
+ "anchor_2d_norm_xy",
+ "description",
+ "fallback_prompts",
+ )
+ missing = [key for key in keys if key not in record]
+ if missing:
+ raise ValueError(f"Pass-A object lacks semantic fields: {missing}")
+ return {key: copy.deepcopy(record[key]) for key in keys}
+
+
+def _local_candidate_geometry(candidate: dict[str, Any]) -> dict[str, Any]:
+ candidate_id = str(candidate.get("candidate_id") or "")
+ mask_path = Path(str(candidate.get("mask_path") or "")).expanduser().resolve()
+ if not mask_path.is_file():
+ raise ValueError(f"compact LC mask is missing: {candidate_id}")
+
+ from PIL import Image
+
+ with Image.open(mask_path) as source:
+ mask = source.convert("L")
+ width, height = mask.size
+ if width <= 0 or height <= 0:
+ raise ValueError(f"compact LC mask has invalid shape: {candidate_id}")
+ absolute_bbox = mask.getbbox()
+ if absolute_bbox is None:
+ raise ValueError(f"compact LC mask is empty: {candidate_id}")
+ x1, y1, x2, y2 = absolute_bbox
+ pixels = mask.load()
+ center_x = (x1 + x2 - 1) / 2.0
+ center_y = (y1 + y2 - 1) / 2.0
+ best_pixel = None
+ best_distance = None
+ area_pixels = 0
+ for y in range(y1, y2):
+ for x in range(x1, x2):
+ if pixels[x, y] <= 0:
+ continue
+ area_pixels += 1
+ distance = (x - center_x) ** 2 + (y - center_y) ** 2
+ if best_distance is None or distance < best_distance:
+ best_distance = distance
+ best_pixel = (x, y)
+ if best_pixel is None or area_pixels <= 0:
+ raise ValueError(f"compact LC mask has no foreground: {candidate_id}")
+
+ bbox = [
+ round(x1 / width, 6),
+ round(y1 / height, 6),
+ round(x2 / width, 6),
+ round(y2 / height, 6),
+ ]
+ anchor = [
+ round((best_pixel[0] + 0.5) / width, 6),
+ round((best_pixel[1] + 0.5) / height, 6),
+ ]
+ if not valid_bbox(bbox) or not valid_anchor(anchor, bbox):
+ raise ValueError(f"compact LC mask geometry is invalid: {candidate_id}")
+ reported_bbox = candidate.get("bbox_norm_xyxy")
+ if not valid_bbox(reported_bbox):
+ raise ValueError(f"compact LC reported bbox is invalid: {candidate_id}")
+ maximum_bbox_error = max(
+ abs(float(left) - float(right))
+ for left, right in zip(bbox, reported_bbox)
+ )
+ if maximum_bbox_error > 0.01:
+ raise ValueError(
+ f"compact LC mask and registry bbox disagree: {candidate_id}"
+ )
+ return {
+ "bbox": bbox,
+ "anchor": anchor,
+ "area_pixels": area_pixels,
+ "mask_path": str(mask_path),
+ "mask_sha256": sha256_file(mask_path),
+ "mask_size": [width, height],
+ "maximum_registry_bbox_error": round(maximum_bbox_error, 6),
+ }
+
+
+
+def _validate_assigned_evidence_decisions(
+ raw_decisions: Any,
+ *,
+ assigned_evidence_metadata: list[dict[str, Any]],
+ additions: list[dict[str, Any]],
+ updates: list[dict[str, Any]],
+ removals: list[str],
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Require one grounded semantic verdict for every supplied SAM3 mask tile."""
+
+ expected: dict[str, dict[str, Any]] = {}
+ for raw in assigned_evidence_metadata:
+ item = dict(raw)
+ evidence_id = str(item.get("evidence_id") or "")
+ owner_id = str(item.get("source_object_id") or "")
+ if not re.fullmatch(r"E_\d{3}", evidence_id):
+ raise ValueError(f"invalid assigned evidence ID: {evidence_id!r}")
+ if not re.fullmatch(r"A_\d{3}", owner_id):
+ raise ValueError(f"invalid assigned evidence owner: {owner_id!r}")
+ if evidence_id in expected:
+ raise ValueError(f"duplicate assigned evidence ID: {evidence_id}")
+ bbox = item.get("bbox_norm_xyxy")
+ if not valid_bbox(bbox):
+ raise ValueError(f"invalid assigned evidence bbox: {evidence_id}")
+ expected[evidence_id] = item
+ continuous = [f"E_{index:03d}" for index in range(1, len(expected) + 1)]
+ if list(expected) != continuous:
+ raise ValueError("assigned evidence IDs must be continuous and ordered")
+ if not isinstance(raw_decisions, list):
+ raise ValueError("assigned evidence decisions must be a list")
+
+ update_ids = {str(item.get("pass_a_id") or "") for item in updates}
+ removal_ids = {str(value) for value in removals}
+ addition_boxes = [
+ (f"B_{index:03d}", item.get("bbox_2d_norm_xyxy"))
+ for index, item in enumerate(additions, start=1)
+ if valid_bbox(item.get("bbox_2d_norm_xyxy"))
+ ]
+ decisions: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ unresolved: list[str] = []
+ normalized_action_conflicts: list[dict[str, Any]] = []
+ normalized_unmaterialized_repair_assertions: list[dict[str, Any]] = []
+ for raw in raw_decisions:
+ if not isinstance(raw, dict) or set(raw) != {"i", "d", "r"}:
+ raise ValueError("assigned evidence decision must contain i/d/r")
+ evidence_id = str(raw["i"])
+ decision = str(raw["d"])
+ reason = str(raw["r"]).strip()
+ if evidence_id not in expected:
+ raise ValueError(f"unknown assigned evidence ID: {evidence_id}")
+ if evidence_id in seen:
+ raise ValueError(f"duplicate assigned evidence decision: {evidence_id}")
+ if decision not in ASSIGNED_EVIDENCE_DECISIONS:
+ raise ValueError(f"invalid assigned evidence decision: {decision}")
+ if not reason:
+ raise ValueError("assigned evidence reason must be non-empty")
+ seen.add(evidence_id)
+ item = expected[evidence_id]
+ owner_id = str(item["source_object_id"])
+ has_update = owner_id in update_ids
+ has_removal = owner_id in removal_ids
+ if decision == "pass" and (has_update or has_removal):
+ # The explicit action is the materializable instruction. Preserve it
+ # deterministically instead of discarding an otherwise valid Pass-B
+ # response merely because its explanatory E verdict says ``pass``.
+ # Removal takes precedence if an invalid response somehow contains both.
+ normalized_decision = (
+ "remove_existing" if has_removal else "update_existing"
+ )
+ normalized_action_conflicts.append(
+ {
+ "evidence_id": evidence_id,
+ "source_object_id": owner_id,
+ "reported_decision": "pass",
+ "effective_decision": normalized_decision,
+ "action": "removal" if has_removal else "update",
+ "reported_reason": reason,
+ "rule": "materialized_owner_action_overrides_conflicting_pass",
+ }
+ )
+ decision = normalized_decision
+ reason = (
+ "Pass B supplied a materialized owner repair action; normalized "
+ f"conflicting pass verdict. Original reason: {reason}"
+ )
+ if decision == "pass" and str(item.get("status") or "") == "failed":
+ normalized_unmaterialized_repair_assertions.append(
+ {
+ "evidence_id": evidence_id,
+ "source_object_id": owner_id,
+ "reported_decision": "pass",
+ "effective_decision": "uncertain",
+ "reported_reason": reason,
+ "rule": "failed_sam3_evidence_cannot_be_accepted_without_repair",
+ }
+ )
+ decision = "uncertain"
+ reason = (
+ "SAM3 supplied no mask for this claim; a pass verdict cannot "
+ f"accept it. Original reason: {reason}"
+ )
+ if decision == "update_existing" and not has_update:
+ # Preserve the source object and mark just this tile uncertain when
+ # Pass B notices a defect but omits the canonical repair patch.
+ normalized_unmaterialized_repair_assertions.append(
+ {
+ "evidence_id": evidence_id,
+ "source_object_id": owner_id,
+ "reported_decision": "update_existing",
+ "effective_decision": "uncertain",
+ "reported_reason": reason,
+ "rule": "unmaterialized_evidence_update_becomes_uncertain",
+ }
+ )
+ decision = "uncertain"
+ reason = (
+ "Pass B requested an update without a materialized canonical "
+ f"patch. Original reason: {reason}"
+ )
+ if decision == "remove_existing" and not has_removal:
+ normalized_unmaterialized_repair_assertions.append(
+ {
+ "evidence_id": evidence_id,
+ "source_object_id": owner_id,
+ "reported_decision": "remove_existing",
+ "effective_decision": "uncertain",
+ "reported_reason": reason,
+ "rule": "unmaterialized_evidence_removal_becomes_uncertain",
+ }
+ )
+ decision = "uncertain"
+ reason = (
+ "Pass B requested removal without a materialized canonical "
+ f"action. Original reason: {reason}"
+ )
+ supporting_addition_ids: list[str] = []
+ if decision == "split_instances":
+ owner_bbox = item["bbox_norm_xyxy"]
+ supporting_addition_ids = [
+ addition_id
+ for addition_id, box in addition_boxes
+ if max(owner_bbox[0], box[0]) < min(owner_bbox[2], box[2])
+ and max(owner_bbox[1], box[1]) < min(owner_bbox[3], box[3])
+ ]
+ if not has_update or not supporting_addition_ids:
+ raise ValueError(
+ f"evidence {evidence_id} split requires owner update and "
+ "an overlapping addition"
+ )
+ if decision == "uncertain":
+ unresolved.append(evidence_id)
+ decisions.append(
+ {
+ "evidence_id": evidence_id,
+ "source_object_id": owner_id,
+ "decision": decision,
+ "reason": reason,
+ "supporting_addition_ids": supporting_addition_ids,
+ }
+ )
+ missing = sorted(set(expected) - seen)
+ if missing:
+ raise ValueError(f"missing assigned evidence decisions: {missing}")
+ audit = {
+ "schema_version": "scenemosaic_assigned_sam3_semantic_gate",
+ "expected_count": len(expected),
+ "decision_count": len(decisions),
+ "complete_partition": len(decisions) == len(expected),
+ "unresolved_evidence_ids": unresolved,
+ "normalized_action_conflicts": normalized_action_conflicts,
+ "normalized_unmaterialized_repair_assertions": (
+ normalized_unmaterialized_repair_assertions
+ ),
+ "status": "blocked" if unresolved else "pass",
+ }
+ return decisions, audit
+
+
+def parse_compact_pass_b(
+ text: str,
+ *,
+ pass_a_objects: list[dict[str, Any]],
+ local_candidate_metadata: list[dict[str, Any]] | None = None,
+ assigned_evidence_metadata: list[dict[str, Any]] | None = None,
+ **parse_context: Any,
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ """Expand compact Pass-B into the unchanged canonical action schema."""
+
+ repaired_text, input_repair_count = _repair_compact_bare_bbox_after_prompt(text)
+ payload = parse_vlm_response_text(repaired_text, **parse_context)
+ expected_keys = {"v", "a", "l", "u", "x", "r"}
+ if assigned_evidence_metadata is not None:
+ expected_keys.add("e")
+ if not isinstance(payload, dict) or set(payload) != expected_keys:
+ raise ValueError(
+ f"compact Pass B must contain exactly {sorted(expected_keys)}"
+ )
+ if payload["v"] != 2:
+ raise ValueError("compact Pass B protocol version must be 2")
+
+ ordinary_additions = payload["a"]
+ local_accepts = payload["l"]
+ updates = payload["u"]
+ removals = payload["x"]
+ rejections = payload["r"]
+ if not isinstance(ordinary_additions, list):
+ raise ValueError("compact Pass B additions must be a list")
+ if not isinstance(local_accepts, list):
+ raise ValueError("compact Pass B LC accepts must be a list")
+ if len(ordinary_additions) + len(local_accepts) > 36:
+ raise ValueError("compact Pass B addition budget exceeded")
+ if not isinstance(updates, list) or len(updates) > 8:
+ raise ValueError("invalid compact Pass B updates")
+ if not isinstance(removals, list) or len(removals) > 8:
+ raise ValueError("invalid compact Pass B removals")
+ if not isinstance(rejections, list):
+ raise ValueError("compact Pass B LC rejections must be a list")
+
+ pass_a_by_id: dict[str, dict[str, Any]] = {}
+ for raw_object in pass_a_objects:
+ object_id = str(raw_object.get("object_id") or "")
+ if not re.fullmatch(r"A_\d{3}", object_id):
+ raise ValueError(f"invalid Pass-A object ID: {object_id!r}")
+ if object_id in pass_a_by_id:
+ raise ValueError(f"duplicate Pass-A object ID: {object_id}")
+ pass_a_by_id[object_id] = dict(raw_object)
+
+ candidate_by_id: dict[str, dict[str, Any]] = {}
+ for raw_candidate in local_candidate_metadata or []:
+ candidate = dict(raw_candidate)
+ candidate_id = str(candidate.get("candidate_id") or "")
+ if not re.fullmatch(r"LC_\d{3}", candidate_id):
+ raise ValueError(f"invalid compact local candidate ID: {candidate_id!r}")
+ if candidate_id in candidate_by_id:
+ raise ValueError(f"duplicate compact local candidate ID: {candidate_id}")
+ _validate_compact_prompt(candidate.get("prompt"))
+ if not valid_bbox(candidate.get("bbox_norm_xyxy")):
+ raise ValueError(f"invalid compact local candidate bbox: {candidate_id}")
+ candidate_by_id[candidate_id] = candidate
+
+ ignored_unavailable_local_candidate_protocol = None
+ if not candidate_by_id and (local_accepts or rejections):
+ # Without an LC panel, these fields cannot refer to any real mask or
+ # object. They are occasionally copied from the response template, so
+ # record and discard them rather than turning template text into a
+ # pipeline failure.
+ ignored_unavailable_local_candidate_protocol = {
+ "accepted_entry_count": len(local_accepts),
+ "rejection_ids": [str(value) for value in rejections],
+ }
+ local_accepts = []
+ rejections = []
+
+ additions = []
+ for raw in ordinary_additions:
+ if not isinstance(raw, dict) or set(raw) != {"c", "p", "b"}:
+ raise ValueError("compact ordinary addition must contain c/p/b")
+ additions.append(
+ _minimal_semantic_record(
+ category=raw["c"], prompt=raw["p"], bbox=raw["b"]
+ )
+ )
+
+ accepted_ids = []
+ accepted_categories: dict[str, str] = {}
+ for raw in local_accepts:
+ if not isinstance(raw, dict) or set(raw) != {"i", "c"}:
+ raise ValueError("compact LC accept must contain i/c")
+ candidate_id = str(raw["i"])
+ if candidate_id not in candidate_by_id:
+ raise ValueError(f"unknown compact LC accept ID: {candidate_id}")
+ if candidate_id in accepted_categories:
+ raise ValueError(f"duplicate compact LC accept ID: {candidate_id}")
+ accepted_ids.append(candidate_id)
+ accepted_categories[candidate_id] = _validate_compact_category(raw["c"])
+
+ normalized_rejections = [str(value) for value in rejections]
+ if len(normalized_rejections) != len(set(normalized_rejections)):
+ raise ValueError("duplicate compact LC rejection ID")
+ unknown_rejections = set(normalized_rejections) - set(candidate_by_id)
+ if unknown_rejections:
+ raise ValueError(
+ f"unknown compact LC rejection IDs: {sorted(unknown_rejections)}"
+ )
+ overlap = set(accepted_ids) & set(normalized_rejections)
+ if overlap:
+ raise ValueError(f"compact LC accepted and rejected: {sorted(overlap)}")
+ decided = set(accepted_ids) | set(normalized_rejections)
+ expected_candidates = set(candidate_by_id)
+ if decided != expected_candidates:
+ missing = sorted(expected_candidates - decided)
+ extra = sorted(decided - expected_candidates)
+ raise ValueError(
+ f"compact LC partition is incomplete: missing={missing}, extra={extra}"
+ )
+
+ candidate_evidence = []
+ local_decisions = []
+ locally_suppressed_duplicate_ids = []
+ for candidate_id in accepted_ids:
+ candidate = candidate_by_id[candidate_id]
+ geometry = _local_candidate_geometry(candidate)
+ accepted_category = accepted_categories[candidate_id]
+ semantic_prompt = str(candidate["prompt"])
+ addition = _minimal_semantic_record(
+ category=accepted_category,
+ prompt=semantic_prompt,
+ bbox=geometry["bbox"],
+ anchor=geometry["anchor"],
+ )
+ duplicate_pass_a_id = None
+ for existing in pass_a_objects:
+ duplicate, _ = likely_same_object(dict(existing), addition)
+ if duplicate:
+ duplicate_pass_a_id = existing.get("object_id")
+ break
+ if duplicate_pass_a_id is not None:
+ locally_suppressed_duplicate_ids.append(candidate_id)
+ local_decisions.append(
+ {
+ "candidate_id": candidate_id,
+ "decision": "reject",
+ "target_object_id": None,
+ "reason": (
+ "VLM-accepted compact candidate suppressed by local "
+ "semantic-and-geometry duplicate check"
+ ),
+ }
+ )
+ continue
+
+ duplicate_addition_index = None
+ for existing_index, existing in enumerate(additions, start=1):
+ duplicate, _ = likely_same_object(dict(existing), addition)
+ if duplicate:
+ duplicate_addition_index = existing_index
+ break
+ if duplicate_addition_index is not None:
+ addition_index = duplicate_addition_index
+ local_decisions.append(
+ {
+ "candidate_id": candidate_id,
+ "decision": "bind_to_addition",
+ "target_object_id": f"B_{addition_index:03d}",
+ "reason": (
+ "accepted compact candidate bound to matching ordinary "
+ "addition"
+ ),
+ }
+ )
+ candidate_evidence.append(
+ {
+ "candidate_id": candidate_id,
+ "category": accepted_category,
+ **geometry,
+ }
+ )
+ else:
+ addition_index = len(additions) + 1
+ additions.append(addition)
+ local_decisions.append(
+ {
+ "candidate_id": candidate_id,
+ "decision": "bind_to_addition",
+ "target_object_id": f"B_{addition_index:03d}",
+ "reason": "accepted compact candidate",
+ }
+ )
+ candidate_evidence.append(
+ {
+ "candidate_id": candidate_id,
+ "category": accepted_category,
+ **geometry,
+ }
+ )
+
+ for candidate_id in normalized_rejections:
+ local_decisions.append(
+ {
+ "candidate_id": candidate_id,
+ "decision": "reject",
+ "target_object_id": None,
+ "reason": "rejected by compact Pass B",
+ }
+ )
+
+ expanded_updates = []
+ locally_suppressed_noop_update_ids = []
+ seen_update_ids = set()
+ for raw in updates:
+ if not isinstance(raw, dict) or set(raw) != {"i", "v"}:
+ raise ValueError("compact update must contain i/v")
+ object_id = str(raw["i"])
+ if object_id not in pass_a_by_id:
+ raise ValueError(f"unknown compact update ID: {object_id}")
+ if object_id in seen_update_ids:
+ raise ValueError(f"duplicate compact update ID: {object_id}")
+ seen_update_ids.add(object_id)
+ patch = raw["v"]
+ if not isinstance(patch, dict) or set(patch) not in (
+ {"b"},
+ {"c", "p"},
+ {"c", "p", "b"},
+ ):
+ raise ValueError(
+ "compact update patch must be b, c+p, or c+p+b"
+ )
+ replacement = _semantic_record_copy(pass_a_by_id[object_id])
+ if "c" in patch:
+ category = _validate_compact_category(patch["c"])
+ prompt = semantic_sam3_prompt(category, patch["p"])
+ replacement["category"] = category
+ replacement["sam3_prompt"] = prompt
+ replacement["instance_name"] = category.replace("_", " ")
+ replacement["object_type"] = "unknown"
+ replacement["description"] = prompt
+ replacement["fallback_prompts"] = []
+ if "b" in patch:
+ if not valid_bbox(patch["b"]):
+ raise ValueError(f"invalid compact update bbox: {object_id}")
+ bbox = [round(float(value), 6) for value in patch["b"]]
+ replacement["bbox_2d_norm_xyxy"] = bbox
+ replacement["anchor_2d_norm_xy"] = _bbox_center(bbox)
+ original = pass_a_by_id[object_id]
+ semantic_changed = "c" in patch and (
+ replacement.get("category") != original.get("category")
+ or replacement.get("sam3_prompt") != original.get("sam3_prompt")
+ )
+ bbox_changed = "b" in patch and any(
+ abs(float(left) - float(right)) > 0.000500001
+ for left, right in zip(
+ replacement["bbox_2d_norm_xyxy"],
+ original["bbox_2d_norm_xyxy"],
+ )
+ )
+ if not semantic_changed and not bbox_changed:
+ # Compact context rounds coordinates to three decimals. Treat a
+ # model echo of that rounded record as no action, so an evidence
+ # verdict of ``pass`` remains consistent with the action graph.
+ locally_suppressed_noop_update_ids.append(object_id)
+ continue
+ expanded_updates.append(
+ {"pass_a_id": object_id, "replacement": replacement}
+ )
+
+ normalized_removals = [str(value) for value in removals]
+ locally_materialized_evidence_removal_ids: list[str] = []
+ locally_materialized_evidence_removals: list[dict[str, str]] = []
+ if assigned_evidence_metadata is not None and isinstance(payload.get("e"), list):
+ evidence_owner_by_id = {
+ str(item.get("evidence_id") or ""): str(
+ item.get("source_object_id") or ""
+ )
+ for item in assigned_evidence_metadata
+ if isinstance(item, dict)
+ }
+ for raw_decision in payload["e"]:
+ if (
+ not isinstance(raw_decision, dict)
+ or set(raw_decision) != {"i", "d", "r"}
+ or raw_decision.get("d") != "remove_existing"
+ ):
+ continue
+ evidence_id = str(raw_decision.get("i") or "")
+ owner_id = evidence_owner_by_id.get(evidence_id, "")
+ if (
+ re.fullmatch(r"A_\d{3}", owner_id)
+ and owner_id in pass_a_by_id
+ and owner_id not in normalized_removals
+ ):
+ normalized_removals.append(owner_id)
+ locally_materialized_evidence_removal_ids.append(owner_id)
+ locally_materialized_evidence_removals.append(
+ {"evidence_id": evidence_id, "source_object_id": owner_id}
+ )
+ if len(normalized_removals) != len(set(normalized_removals)):
+ raise ValueError("duplicate compact removal ID")
+ unknown_removals = set(normalized_removals) - set(pass_a_by_id)
+ if unknown_removals:
+ raise ValueError(
+ f"unknown compact removal IDs: {sorted(unknown_removals)}"
+ )
+ update_remove_conflicts = seen_update_ids & set(normalized_removals)
+ locally_resolved_update_removal_conflicts: list[dict[str, Any]] = []
+ if update_remove_conflicts:
+ # Retaining an existing object (and any materialized repair) is safer
+ # than silently deleting it when one compact response contradicts
+ # itself. The conflict remains auditable and matching evidence removals
+ # are rewritten below.
+ materialized_update_ids = {
+ str(item.get("pass_a_id") or "") for item in expanded_updates
+ }
+ normalized_removals = [
+ object_id
+ for object_id in normalized_removals
+ if object_id not in update_remove_conflicts
+ ]
+ locally_materialized_evidence_removal_ids = [
+ object_id
+ for object_id in locally_materialized_evidence_removal_ids
+ if object_id not in update_remove_conflicts
+ ]
+ locally_materialized_evidence_removals = [
+ item
+ for item in locally_materialized_evidence_removals
+ if str(item.get("source_object_id") or "")
+ not in update_remove_conflicts
+ ]
+ for object_id in sorted(update_remove_conflicts):
+ locally_resolved_update_removal_conflicts.append(
+ {
+ "source_object_id": object_id,
+ "effective_action": (
+ "retain_materialized_update"
+ if object_id in materialized_update_ids
+ else "retain_existing_noop_update"
+ ),
+ "rule": "compact_update_over_conflicting_remove",
+ }
+ )
+
+ actions = {
+ "additions": additions,
+ "updates": expanded_updates,
+ "removals": normalized_removals,
+ }
+ evidence_audit = None
+ effective_evidence_payload = payload.get("e")
+ locally_suppressed_unmaterialized_boundary_updates = []
+ locally_rewritten_conflicting_evidence_removals: list[dict[str, Any]] = []
+ if assigned_evidence_metadata is not None:
+ effective_evidence_payload = copy.deepcopy(payload["e"])
+ metadata_by_evidence_id = {
+ str(item.get("evidence_id") or ""): dict(item)
+ for item in assigned_evidence_metadata
+ if isinstance(item, dict)
+ }
+ materialized_update_ids = {
+ str(item.get("pass_a_id") or "") for item in expanded_updates
+ }
+ materialized_removal_ids = set(normalized_removals)
+ for raw_decision in effective_evidence_payload:
+ if not isinstance(raw_decision, dict):
+ continue
+ evidence_id = str(raw_decision.get("i") or "")
+ owner_id = str(
+ metadata_by_evidence_id.get(evidence_id, {}).get(
+ "source_object_id"
+ )
+ or ""
+ )
+ if (
+ owner_id in update_remove_conflicts
+ and raw_decision.get("d") == "remove_existing"
+ ):
+ effective_decision = (
+ "update_existing"
+ if owner_id in materialized_update_ids
+ else "pass"
+ )
+ locally_rewritten_conflicting_evidence_removals.append(
+ {
+ "evidence_id": evidence_id,
+ "source_object_id": owner_id,
+ "reported_decision": "remove_existing",
+ "effective_decision": effective_decision,
+ "rule": "compact_update_over_conflicting_remove",
+ }
+ )
+ raw_decision["d"] = effective_decision
+ raw_decision["r"] = (
+ "Conflicting compact removal was suppressed to preserve "
+ "the existing verified object."
+ )
+ for raw_decision in effective_evidence_payload:
+ if (
+ not isinstance(raw_decision, dict)
+ or raw_decision.get("d") != "update_existing"
+ ):
+ continue
+ evidence_id = str(raw_decision.get("i") or "")
+ metadata = metadata_by_evidence_id.get(evidence_id, {})
+ owner_id = str(metadata.get("source_object_id") or "")
+ if (
+ owner_id in materialized_update_ids
+ or owner_id in materialized_removal_ids
+ ):
+ continue
+ reason = str(raw_decision.get("r") or "").strip()
+ try:
+ coverage = float(
+ metadata.get("claimed_bbox_coverage_by_mask_bbox")
+ )
+ component_count = int(metadata.get("component_count"))
+ except (TypeError, ValueError):
+ continue
+ is_minor_boundary_tightening = bool(
+ re.search(
+ r"(?:thin\s+(?:(?:left|right|top|bottom|side)\s+){0,2}edge|"
+ r"boundary\s+slightly|slightly\s+loose|tighten.{0,24}rim|"
+ r"minor\s+boundary)",
+ reason.lower(),
+ )
+ )
+ if not (
+ metadata.get("status") == "success"
+ and not bool(metadata.get("merge_review_required"))
+ and component_count <= 2
+ and coverage >= 0.93
+ and is_minor_boundary_tightening
+ ):
+ continue
+ raw_decision["d"] = "pass"
+ raw_decision["r"] = (
+ "local high-coverage boundary policy retained the verified mask"
+ )
+ locally_suppressed_unmaterialized_boundary_updates.append(
+ {
+ "evidence_id": evidence_id,
+ "source_object_id": owner_id,
+ "previous_decision": "update_existing",
+ "effective_decision": "pass",
+ "previous_reason": reason,
+ "claimed_bbox_coverage_by_mask_bbox": round(coverage, 6),
+ "component_count": component_count,
+ "rule": (
+ "unmaterialized_minor_boundary_update_requires_no_action_"
+ "when_verified_coverage_ge_0_93"
+ ),
+ }
+ )
+ evidence_decisions, evidence_audit = (
+ _validate_assigned_evidence_decisions(
+ effective_evidence_payload,
+ assigned_evidence_metadata=assigned_evidence_metadata,
+ additions=additions,
+ updates=expanded_updates,
+ removals=normalized_removals,
+ )
+ )
+ actions["assigned_evidence_decisions"] = evidence_decisions
+ actions["assigned_evidence_semantic_gate"] = evidence_audit
+ if candidate_by_id:
+ actions["local_candidate_decisions"] = local_decisions
+ audit = {
+ "schema_version": "scenemosaic_compact_pass_b_expansion",
+ "protocol": COMPACT_PASS_B_PROTOCOL,
+ "input_repair_count": input_repair_count,
+ "ignored_unavailable_local_candidate_protocol": (
+ ignored_unavailable_local_candidate_protocol
+ ),
+ "pass_a_object_count": len(pass_a_by_id),
+ "ordinary_addition_count": len(ordinary_additions),
+ "accepted_local_candidate_count": len(accepted_ids),
+ "rejected_local_candidate_count": len(normalized_rejections),
+ "locally_suppressed_duplicate_count": len(
+ locally_suppressed_duplicate_ids
+ ),
+ "locally_suppressed_duplicate_ids": locally_suppressed_duplicate_ids,
+ "update_count": len(expanded_updates),
+ "locally_suppressed_noop_update_count": len(
+ locally_suppressed_noop_update_ids
+ ),
+ "locally_suppressed_noop_update_ids": (
+ locally_suppressed_noop_update_ids
+ ),
+ "removal_count": len(normalized_removals),
+ "locally_resolved_update_removal_conflict_count": len(
+ locally_resolved_update_removal_conflicts
+ ),
+ "locally_resolved_update_removal_conflicts": (
+ locally_resolved_update_removal_conflicts
+ ),
+ "locally_rewritten_conflicting_evidence_removal_count": len(
+ locally_rewritten_conflicting_evidence_removals
+ ),
+ "locally_rewritten_conflicting_evidence_removals": (
+ locally_rewritten_conflicting_evidence_removals
+ ),
+ "locally_materialized_evidence_removal_count": len(
+ locally_materialized_evidence_removal_ids
+ ),
+ "locally_materialized_evidence_removal_ids": (
+ locally_materialized_evidence_removal_ids
+ ),
+ "locally_materialized_evidence_removals": locally_materialized_evidence_removals,
+ "locally_suppressed_unmaterialized_boundary_update_count": len(
+ locally_suppressed_unmaterialized_boundary_updates
+ ),
+ "locally_suppressed_unmaterialized_boundary_updates": (
+ locally_suppressed_unmaterialized_boundary_updates
+ ),
+ "canonical_addition_count": len(additions),
+ "candidate_partition_complete": decided == expected_candidates,
+ "candidate_evidence": candidate_evidence,
+ "assigned_evidence_semantic_gate": evidence_audit,
+ "locally_derived_fields": [
+ "anchor_2d_norm_xy",
+ "instance_name",
+ "object_type",
+ "description",
+ "fallback_prompts",
+ ],
+ "canonical_action_schema_unchanged": True,
+ }
+ return actions, audit
+
+
+def parse_pass_a(text: str, **parse_context: Any) -> list[dict[str, Any]]:
+ payload = parse_vlm_response_text(text, **parse_context)
+ if not isinstance(payload, dict) or set(payload) != {"o"}:
+ raise ValueError("Pass A response must contain only key 'o'")
+ records = payload["o"]
+ if not isinstance(records, list) or not records:
+ raise ValueError("Pass A must return a non-empty object list")
+ if len(records) > 80:
+ raise ValueError("Pass A object budget exceeded")
+ expanded = []
+ for record in records:
+ if not isinstance(record, dict):
+ raise ValueError("Pass A object record must be an object")
+ compact_keys = set(record)
+ if compact_keys == {"c", "p", "t", "b"}:
+ expanded.append(
+ _minimal_semantic_record(
+ category=record["c"],
+ prompt=record["p"],
+ object_type=record["t"],
+ bbox=record["b"],
+ )
+ )
+ elif compact_keys == {"c", "p", "b"}:
+ expanded.append(
+ _minimal_semantic_record(
+ category=record["c"],
+ prompt=record["p"],
+ bbox=record["b"],
+ )
+ )
+ else:
+ expanded.append(expand_record(record))
+ return expanded
+
+
+
+
+
+def normalize_pass_a(records: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ return normalize_objects(
+ records,
+ object_id_base=1,
+ object_id_format="A_{index:03d}",
+ )
+
+
+def compact_pass_a_context(objects: list[dict[str, Any]]) -> str:
+ """Encode only the Pass-A fields that Pass B must inspect."""
+
+ records = []
+ seen = set()
+ for item in objects:
+ object_id = str(item.get("object_id") or "")
+ if not re.fullmatch(r"A_\d{3}", object_id):
+ raise ValueError(f"invalid Pass-A ID for compact protocol: {object_id!r}")
+ if object_id in seen:
+ raise ValueError(f"duplicate Pass-A ID for compact protocol: {object_id}")
+ seen.add(object_id)
+ sam3_prompt = _validate_compact_prompt(item.get("sam3_prompt"))
+ bbox = item.get("bbox_2d_norm_xyxy")
+ if not valid_bbox(bbox):
+ raise ValueError(f"invalid Pass-A bbox for compact protocol: {object_id}")
+ records.append(
+ [
+ object_id,
+ sam3_prompt,
+ [round(float(value), 3) for value in bbox],
+ ]
+ )
+ return json.dumps(records, ensure_ascii=False, separators=(",", ":"))
+
+
+def compact_local_candidate_context(
+ candidates: list[dict[str, Any]],
+) -> tuple[str, list[str]]:
+ """Drop scores and repeated semantics while retaining visual identity."""
+
+ records = []
+ seen = set()
+ for candidate in candidates:
+ candidate_id = str(candidate.get("candidate_id") or "")
+ if not re.fullmatch(r"LC_\d{3}", candidate_id):
+ raise ValueError(f"invalid compact protocol local candidate ID: {candidate_id!r}")
+ if candidate_id in seen:
+ raise ValueError(f"duplicate compact protocol local candidate ID: {candidate_id}")
+ seen.add(candidate_id)
+ prompt = _validate_compact_prompt(candidate.get("prompt"))
+ namespace = str(candidate.get("source_namespace") or "other")
+ if namespace not in {
+ "top_zone",
+ "sam3_a_unassigned",
+ "other",
+ }:
+ raise ValueError(
+ f"invalid compact protocol candidate namespace: {namespace!r}"
+ )
+ bbox = candidate.get("bbox_norm_xyxy")
+ if not valid_bbox(bbox):
+ raise ValueError(f"invalid compact protocol local candidate bbox: {candidate_id}")
+ record = [
+ candidate_id,
+ namespace,
+ prompt,
+ [round(float(value), 3) for value in bbox],
+ ]
+ alternate_prompts = []
+ for evidence in candidate.get("prompt_evidence") or []:
+ alternate = _validate_compact_prompt(evidence.get("prompt"))
+ if alternate != prompt and alternate not in alternate_prompts:
+ alternate_prompts.append(alternate)
+ if alternate_prompts:
+ record.append(alternate_prompts[:3])
+ records.append(record)
+ return (
+ json.dumps(records, ensure_ascii=False, separators=(",", ":")),
+ [item[0] for item in records],
+ )
+
+
+def compact_content_host_context(hosts: list[dict[str, Any]]) -> str:
+ """Encode H tiles without repeating their owner category."""
+
+ records = []
+ seen = set()
+ for host in hosts:
+ host_id = str(host.get("host_id") or "")
+ source_id = str(host.get("source_object_id") or "")
+ if not re.fullmatch(r"H_\d{3}", host_id):
+ raise ValueError(f"invalid compact protocol content-host ID: {host_id!r}")
+ if host_id in seen:
+ raise ValueError(f"duplicate compact protocol content-host ID: {host_id}")
+ if not re.fullmatch(r"A_\d{3}", source_id):
+ raise ValueError(f"invalid compact protocol content-host owner: {source_id!r}")
+ mode = str(host.get("host_mode") or "")
+ if mode not in {"open_storage", "support_surface"}:
+ raise ValueError(f"invalid compact protocol content-host mode: {host_id}")
+ bbox = host.get("crop_norm_xyxy")
+ if not valid_bbox(bbox):
+ raise ValueError(f"invalid compact protocol content-host bbox: {host_id}")
+ seen.add(host_id)
+ records.append(
+ [
+ host_id,
+ source_id,
+ mode,
+ [round(float(value), 3) for value in bbox],
+ ]
+ )
+ expected = [f"H_{index:03d}" for index in range(1, len(records) + 1)]
+ if [item[0] for item in records] != expected:
+ raise ValueError("compact protocol content-host IDs must be continuous")
+ return json.dumps(records, ensure_ascii=False, separators=(",", ":"))
+
+
+def build_compact_pass_b_prompt(
+ registry: str,
+ local_candidate_registry: str | None = None,
+ *,
+ content_host_registry: str | None = None,
+ assigned_evidence_registry: str | None = None,
+ same_mask_conflict_registry: str | None = None,
+ content_closure_only: bool = False,
+) -> str:
+ """Build the bounded, low-output Pass-B request."""
+
+ if local_candidate_registry is None:
+ local_response_shape = '"l":[],"r":[]'
+ local_candidate_instruction = (
+ "No LC candidate panel is supplied. Return l=[] and r=[] exactly; "
+ "never invent an LC identifier. "
+ )
+ else:
+ local_response_shape = (
+ '"l":[{"i":"LC_001","c":"category"}],'
+ '"r":["LC_002"]'
+ )
+ local_candidate_instruction = "l=accepted LC and r=rejected LC IDs. "
+ response_shape = (
+ '{"v":2,"a":[{"c":"category","p":"SAM3 prompt","b":[0,0,0,0]}],'
+ + local_response_shape
+ + ',"u":[{"i":"A_001","v":{"b":[0,0,0,0}}],'
+ + '"x":[]}'
+ )
+ if assigned_evidence_registry is not None:
+ response_shape = response_shape[:-1] + (
+ ',"e":[{"i":"E_001","d":"pass|update_existing|split_instances|'
+ 'remove_existing|uncertain","r":"short reason"}]}'
+ )
+
+ audit_scope = (
+ "Audit the whole image; do not repeat correct claims. Find visible "
+ "missing physical units, especially small items on supports/shelves/"
+ "cubbies/media furniture, loose pillows or cushions on sofas, chairs, "
+ "or beds, edge or high fixtures, plants, lighting, "
+ "repeated or symmetric locations. "
+ if not content_closure_only
+ else
+ "Perform content closure only. The current inventory already includes "
+ "the main Pass-B additions: preserve their selected object granularity. "
+ "Inspect every supplied H open-storage window, each exposed level and "
+ "both lateral edges only for a visually independent content group not "
+ "already claimed there. Never subdivide, duplicate, or enumerate the "
+ "members of a current group. Return one grouped unit when touching or "
+ "tightly adjacent contents form one shelf-level unit; an empty gap is "
+ "not itself an object. Do not correct, remove, split, rename, or resize "
+ "any current owner. "
+ )
+ sections = [
+ "You are Pass B, the coverage/correction audit for visible-object "
+ "SAM3+SAM3D reconstruction. Image 1 is the whole scene with a cyan "
+ "coordinate grid. Current inventory claims are [ID,sam3_prompt,bbox]:",
+ registry,
+ "Each registry label is an instance-level SAM3 prompt, not a broad "
+ "category. Before emitting a, compare its p and bbox to every current "
+ "claim: if it names the same visible physical unit, retain that claim "
+ "or correct its A ID through u; do not add a second entry. Emit a only "
+ "when pixels show a distinct physical unit.",
+ audit_scope
+ + (
+ "Also correct materially wrong category/bbox and remove duplicate "
+ "or nonphysical claims. "
+ if not content_closure_only
+ else
+ "Use a only for those independent missing content groups and leave "
+ "l, u, x, and r empty. Give a group an object-only p such as \'upright "
+ "book group\' or \'stacked book group\', never an item-by-item list "
+ "or a location phrase. "
+ )
+ + "Require visible pixels; symmetry is only a search cue. Exclude wall, floor, "
+ "ceiling, windows, doors, curtains, background, glare, trim, and content "
+ "depicted in art, mirrors, televisions or screens. Keep fitted assembly "
+ "parts together: bed plus duvet; every visually independent loose pillow "
+ "or cushion on a sofa, chair, or bed is a separate movable object. Sink plus "
+ "faucet, dedicated support and fitted undercounter dishwasher. A visible "
+ "range/stove splits counter sections; never keep one counter claim across "
+ "both sides. Attached water bottle plus dispenser; "
+ "a verified potted plant keeps its pot and foliage as one assembly, never "
+ "replace it with only a vase or planter and never add its foliage as a "
+ "second object. "
+ "connected lamp heads/rods; dedicated coat/towel/drying/roll/pegboard "
+ "racks plus attached contents. Mere "
+ "contact does not merge independent objects. Field p is an "
+ "object-only SAM3 semantic prompt: name the visible object itself, "
+ "optionally with color/material/form; never use a location, support, "
+ "direction, relation, grid cell or nearby object in p.",
+ ]
+ if (
+ local_candidate_registry is not None
+ or content_host_registry is not None
+ or assigned_evidence_registry is not None
+ ):
+ sections.append(
+ "One or more additional numbered review panels follow. All "
+ "output bboxes remain normalized whole-image coordinates and use "
+ "[left, top, right, bottom] (x1, y1, x2, y2), never "
+ "[x, y, width, height]."
+ )
+ if local_candidate_registry is not None:
+ sections.extend(
+ [
+ "LC claims [ID,source,prompt,bbox,optional aliases]:",
+ local_candidate_registry,
+ "Each LC mask is evidence; prompts are fallible hints. Relabel visually. Decide every LC "
+ "exactly once: put a visible new reconstruction unit in l as "
+ "{i,c}; put background, false, duplicate, part, or already "
+ "registered candidates in r by ID. An accepted LC inherits its "
+ "true mask and bbox; never duplicate it in a.",
+ ]
+ )
+ if content_host_registry is not None:
+ sections.extend(
+ [
+ "H tiles are [H_ID,owner_A_ID,open_storage|support_surface,"
+ "crop_bbox]:",
+ content_host_registry,
+ "H tiles are high-resolution attention windows, not masks "
+ "or objects. Scan every exposed level and both lateral "
+ "edges. On fixed shelves/racks, report independent movable "
+ "contents as separate units. On a mobile cart, group "
+ "contents once per visible shelf level. Do not re-add the "
+ "carrier. Report a missing physical unit from H normally "
+ "in a. An H crop bbox is never an object bbox: do not copy "
+ "it into u, and never update its owner because of H. Never "
+ "emit an H ID.",
+ ]
+ )
+ if assigned_evidence_registry is not None:
+ sections.extend(
+ [
+ "E tiles are [E_ID,owner_A_ID,sam3_prompt,SAM3_status,mask_or_crop_bbox]:",
+ assigned_evidence_registry,
+ "Each E tile shows original crop left and the actual assigned "
+ "SAM3 mask overlay right. The crop is the union of the claimed "
+ "bbox and actual-mask bbox: any visible fixed assembly part "
+ "inside the claim but outside the colored mask requires "
+ "update_existing or uncertain, never pass. Decide every E "
+ "exactly once in e: "
+ "pass, update_existing, split_instances, remove_existing, or "
+ "uncertain, with a short visible-pixel reason. If the original "
+ "crop visibly contains no instance matching an E claim, choose "
+ "remove_existing and put its owner in x; do not call visible "
+ "absence uncertain. Use uncertain only when a likely visible "
+ "instance remains but pixels cannot determine its correct mask "
+ "or object unit. For an E with SAM3_status=failed, use "
+ "update_existing only when the original claimed physical object "
+ "is visibly present and the replacement preserves its object "
+ "semantics. Never relabel a failed owner into a nearby different "
+ "object; if its claimed object is absent or semantically mismatched, "
+ "choose remove_existing. update_existing requires the owner A ID in u; "
+ "remove_existing requires it in x; split_instances requires the "
+ "owner in u plus the other physical "
+ "unit in a. Tighten the existing bbox and add the missing object "
+ "for true two-object merges. Do not split a single occluded object "
+ "merely because visible pixels are disconnected.",
+ ]
+ )
+ if same_mask_conflict_registry is not None:
+ sections.extend(
+ [
+ "SMC records are [SMC_ID,A_ID_1,A_ID_2,sam3_prompt_1,sam3_prompt_2,"
+ "bbox_1,bbox_2,mask_iou]. Their separate Pass-A claims received "
+ "nearly identical SAM3 masks. This is ambiguous: they may be one "
+ "physical unit or two adjacent units merged by SAM3. Inspect Image 1 "
+ "and decide conservatively. If exactly one physical instance is "
+ "visible, put exactly one non-retained A ID in x. If two distinct "
+ "adjacent physical instances are visible, keep both out of x; a "
+ "strict local bbox split will then be attempted. If pixels do not "
+ "decide, keep both out of x. Never remove both SMC owners and never "
+ "add a duplicate replacement for an SMC owner.",
+ same_mask_conflict_registry,
+ ]
+ )
+ sections.extend(
+ [
+ "Return ONLY this minified JSON shape with exact keys:",
+ response_shape,
+ "a=ordinary missing objects. " + local_candidate_instruction,
+ "u=material A updates; each v must be exactly {b}, {c,p}, or {c,p,b}. "
+ "x=removed A IDs. e=complete assigned-mask decisions when E "
+ "tiles are supplied. c is a generic machine category identifier: "
+ "use lowercase snake_case only (for example vase, book_stack, "
+ "ceiling_light), never spaces, hyphens, color, material, size, or "
+ "location terms. Put all instance-level visual detail in a short "
+ "object-only English SAM3 prompt p. Use a tight normalized bbox "
+ "[left, top, right, bottom] "
+ "with left < right and top < bottom, and [] when empty. Combined a+l <=36; "
+ "u<=8; x<=8. Output no prose and no other fields.",
+ ]
+ )
+ return "\n\n".join(sections)
+
+
+
+
+def compact_same_mask_conflict_context(
+ conflicts: list[dict[str, Any]],
+) -> str:
+ """Return compact, auditable context for ambiguous Pass-A mask pairs."""
+
+ compact = []
+ for conflict in conflicts:
+ conflict_id = str(conflict.get("conflict_id") or "")
+ owners = [str(value) for value in (conflict.get("object_ids") or [])]
+ objects = list(conflict.get("objects") or [])
+ if not re.fullmatch(r"SMC_\d{3}", conflict_id):
+ raise ValueError(f"invalid same-mask conflict ID: {conflict_id!r}")
+ if len(owners) != 2 or len(set(owners)) != 2:
+ raise ValueError(f"same-mask conflict must have two distinct owners: {conflict_id}")
+ by_id = {str(item.get("object_id") or ""): item for item in objects}
+ if set(by_id) != set(owners):
+ raise ValueError(f"same-mask conflict owner metadata mismatch: {conflict_id}")
+ iou = float(conflict.get("mask_iou") or 0.0)
+ if not math.isfinite(iou) or not 0.0 <= iou <= 1.0:
+ raise ValueError(f"invalid same-mask conflict IoU: {conflict_id}")
+ compact.append(
+ [
+ conflict_id,
+ owners[0],
+ owners[1],
+ _validate_compact_prompt(by_id[owners[0]].get("sam3_prompt")),
+ _validate_compact_prompt(by_id[owners[1]].get("sam3_prompt")),
+ [round(float(value), 3) for value in by_id[owners[0]]["vlm_bbox_norm_xyxy"]],
+ [round(float(value), 3) for value in by_id[owners[1]]["vlm_bbox_norm_xyxy"]],
+ round(iou, 3),
+ ]
+ )
+ expected = [f"SMC_{index:03d}" for index in range(1, len(compact) + 1)]
+ if [item[0] for item in compact] != expected:
+ raise ValueError("same-mask conflict IDs must be continuous")
+ return json.dumps(compact, ensure_ascii=False, separators=(",", ":"))
+
+
+
+
+
+
+
+
+
+def bbox_metrics(left: list[float], right: list[float]) -> dict[str, float]:
+ x1, y1 = max(left[0], right[0]), max(left[1], right[1])
+ x2, y2 = min(left[2], right[2]), min(left[3], right[3])
+ intersection = max(0.0, x2 - x1) * max(0.0, y2 - y1)
+ left_area = (left[2] - left[0]) * (left[3] - left[1])
+ right_area = (right[2] - right[0]) * (right[3] - right[1])
+ union = left_area + right_area - intersection
+ return {
+ "iou": intersection / union if union else 0.0,
+ "left_coverage": intersection / left_area if left_area else 0.0,
+ "right_coverage": intersection / right_area if right_area else 0.0,
+ }
+
+
+def category_key(value: Any) -> str:
+ return "_".join(
+ str(value or "").lower().replace("-", " ").replace("_", " ").split()
+ )
+
+
+def _semantic_label_tokens(value: Any) -> frozenset[str]:
+ return frozenset(re.findall(r"[a-z0-9]+", str(value or "").lower()))
+
+
+def _compatible_object_semantics(
+ left: dict[str, Any], right: dict[str, Any]
+) -> bool:
+ """Match only prompt-level object identities without a category ontology."""
+
+ left_prompt = _semantic_label_tokens(left.get("sam3_prompt"))
+ right_prompt = _semantic_label_tokens(right.get("sam3_prompt"))
+ if not left_prompt or not right_prompt:
+ return False
+ if "unknown" in left_prompt or "unknown" in right_prompt:
+ return False
+ return left_prompt <= right_prompt or right_prompt <= left_prompt
+
+
+def likely_same_object(left: dict[str, Any], right: dict[str, Any]) -> tuple[bool, dict[str, float]]:
+ metrics = bbox_metrics(
+ left["bbox_2d_norm_xyxy"],
+ right["bbox_2d_norm_xyxy"],
+ )
+ left_anchor = left["anchor_2d_norm_xy"]
+ right_anchor = right["anchor_2d_norm_xy"]
+ anchor_distance = math.dist(left_anchor, right_anchor)
+ left_box = left["bbox_2d_norm_xyxy"]
+ right_box = right["bbox_2d_norm_xyxy"]
+ scale = max(
+ 0.02,
+ min(
+ math.hypot(left_box[2] - left_box[0], left_box[3] - left_box[1]),
+ math.hypot(right_box[2] - right_box[0], right_box[3] - right_box[1]),
+ ),
+ )
+ metrics["anchor_distance"] = anchor_distance
+ metrics["relative_anchor_distance"] = anchor_distance / scale
+ left_prompt = category_key(left.get("sam3_prompt"))
+ right_prompt = category_key(right.get("sam3_prompt"))
+ same_sam3_prompt = bool(left_prompt and left_prompt == right_prompt)
+ metrics["same_sam3_prompt"] = float(same_sam3_prompt)
+ if same_sam3_prompt:
+ duplicate = (
+ metrics["iou"] >= 0.55
+ or (
+ min(metrics["left_coverage"], metrics["right_coverage"]) >= 0.75
+ and metrics["relative_anchor_distance"] <= 0.45
+ )
+ )
+ else:
+ duplicate = _compatible_object_semantics(left, right) and (
+ metrics["iou"] >= 0.85
+ or (
+ min(metrics["left_coverage"], metrics["right_coverage"]) >= 0.90
+ and metrics["relative_anchor_distance"] <= 0.15
+ )
+ )
+ return duplicate, metrics
+
+
+def fuse_passes(
+ pass_a_objects: list[dict[str, Any]],
+ pass_b: dict[str, Any],
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ by_id = {str(item["object_id"]): dict(item) for item in pass_a_objects}
+ known_ids = set(by_id)
+ local_decisions = list(pass_b.get("local_candidate_decisions") or [])
+ assigned_evidence_decisions = list(
+ pass_b.get("assigned_evidence_decisions") or []
+ )
+ for decision in local_decisions:
+ target = decision.get("target_object_id")
+ action = decision.get("decision")
+ if action in LOCAL_CANDIDATE_EXISTING_TARGET_ACTIONS and target not in known_ids:
+ raise ValueError(f"local candidate references unknown Pass-A ID: {target}")
+ if action == "bind_to_addition":
+ addition_index = int(str(target).split("_")[1])
+ if not 1 <= addition_index <= len(pass_b["additions"]):
+ raise ValueError(f"local candidate references unknown Pass-B addition: {target}")
+ update_ids = [item["pass_a_id"] for item in pass_b["updates"]]
+ if len(update_ids) != len(set(update_ids)):
+ raise ValueError("Pass B updates one Pass-A ID more than once")
+ unknown = (set(update_ids) | set(pass_b["removals"])) - known_ids
+ if unknown:
+ raise ValueError(f"Pass B references unknown Pass-A IDs: {sorted(unknown)}")
+ conflict = set(update_ids) & set(pass_b["removals"])
+ if conflict:
+ raise ValueError(f"Pass B both updates and removes IDs: {sorted(conflict)}")
+
+ provenance: dict[str, list[dict[str, Any]]] = {
+ object_id: [{"pass": "A", "source_id": object_id, "action": "created"}]
+ for object_id in by_id
+ }
+ for update in pass_b["updates"]:
+ object_id = update["pass_a_id"]
+ replacement = dict(update["replacement"])
+ replacement["object_id"] = object_id
+ replacement = normalize_objects(
+ [replacement],
+ object_id_base=int(object_id.split("_")[1]),
+ object_id_format="A_{index:03d}",
+ )[0]
+ by_id[object_id] = replacement
+ provenance[object_id].append(
+ {"pass": "B", "source_id": object_id, "action": "updated"}
+ )
+ for object_id in pass_b["removals"]:
+ by_id.pop(object_id)
+ provenance.pop(object_id)
+
+ accepted_additions: list[tuple[str, dict[str, Any]]] = []
+ rejected_additions: list[dict[str, Any]] = []
+ for index, addition in enumerate(pass_b["additions"], start=1):
+ source_id = f"B_{index:03d}"
+ conflicts = []
+ for existing_id, existing in [
+ *by_id.items(),
+ *accepted_additions,
+ ]:
+ duplicate, metrics = likely_same_object(existing, addition)
+ if duplicate:
+ conflicts.append(
+ {
+ "existing_id": existing_id,
+ **{key: round(value, 4) for key, value in metrics.items()},
+ }
+ )
+ if conflicts:
+ rejected_additions.append(
+ {
+ "source_id": source_id,
+ "category": addition["category"],
+ "reason": "semantic_and_geometry_duplicate",
+ "conflicts": conflicts,
+ }
+ )
+ continue
+ accepted_additions.append((source_id, dict(addition)))
+
+ candidates: list[tuple[list[dict[str, Any]], dict[str, Any]]] = []
+ for object_id, item in by_id.items():
+ candidates.append((provenance[object_id], item))
+ for source_id, item in accepted_additions:
+ candidates.append(
+ ([{"pass": "B", "source_id": source_id, "action": "added"}], item)
+ )
+ candidates.sort(
+ key=lambda pair: (
+ pair[1]["anchor_2d_norm_xy"][1],
+ pair[1]["anchor_2d_norm_xy"][0],
+ category_key(pair[1]["category"]),
+ )
+ )
+ normalized = normalize_objects(
+ [item for _, item in candidates],
+ object_id_base=1,
+ object_id_format="proposal_{index:03d}",
+ )
+ identity_records = []
+ for output, (sources, _) in zip(normalized, candidates):
+ output["proposal_sources"] = sources
+ identity_records.append(
+ {
+ "proposal_id": output["object_id"],
+ "category": output["category"],
+ "description": output["description"],
+ "instance_name": output["instance_name"],
+ "anchor_2d_norm_xy": output["anchor_2d_norm_xy"],
+ "bbox_2d_norm_xyxy": output["bbox_2d_norm_xyxy"],
+ "sources": sources,
+ "binding_status": "provisional_bbox_anchor_only",
+ }
+ )
+ audit = {
+ "pass_a_input_count": len(pass_a_objects),
+ "pass_b_addition_count": len(pass_b["additions"]),
+ "pass_b_update_count": len(pass_b["updates"]),
+ "pass_b_removal_count": len(pass_b["removals"]),
+ "accepted_addition_count": len(accepted_additions),
+ "ordinary_addition_duplicate_rejection_count": len(rejected_additions),
+ "rejected_additions": rejected_additions,
+ "local_candidate_decision_count": len(local_decisions),
+ "local_candidate_decisions": local_decisions,
+ "assigned_evidence_decision_count": len(
+ assigned_evidence_decisions
+ ),
+ "assigned_evidence_decisions": assigned_evidence_decisions,
+ "assigned_evidence_semantic_gate": pass_b.get(
+ "assigned_evidence_semantic_gate"
+ ),
+ "merged_proposal_count": len(normalized),
+ "identity_records": identity_records,
+ }
+ return normalized, audit
+
+
+def temperature_request_kwargs(model: str) -> dict[str, int]:
+ """Use provider defaults for the repository-wide GPT-5.5 route."""
+
+ canonical = str(model or "").lower().rsplit("/", 1)[-1]
+ if canonical != "gpt-5.5":
+ raise ValueError("SceneMosaic perception requires model gpt-5.5")
+ return {}
+
+
+def run_call(
+ *,
+ client: Any,
+ prompt: str,
+ image_data_url: str,
+ model: str,
+ reasoning_effort: str,
+ output_dir: Path,
+ additional_image_data_urls: list[str] | None = None,
+) -> tuple[str, dict[str, Any]]:
+ content = [
+ {"type": "text", "text": prompt},
+ {
+ "type": "image_url",
+ "image_url": {"url": image_data_url, "detail": "original"},
+ },
+ ]
+ for additional_url in additional_image_data_urls or []:
+ content.append(
+ {
+ "type": "image_url",
+ "image_url": {"url": additional_url, "detail": "original"},
+ }
+ )
+ response = client.chat.completions.create(
+ model=model,
+ messages=[{"role": "user", "content": content}],
+ reasoning_effort=reasoning_effort,
+ **temperature_request_kwargs(model),
+ )
+ text = str(response.choices[0].message.content or "").strip()
+ (output_dir / "response_raw.txt").write_text(text + "\n", encoding="utf-8")
+ return text, {
+ "finish_reason": getattr(response.choices[0], "finish_reason", None),
+ }
+
+
+def require_clean_finish(call_info: dict[str, Any], stage: str) -> None:
+ """Reject any non-stop model finish before parsing a freeze-eligible response."""
+
+ finish_reason = call_info.get("finish_reason")
+ if finish_reason != "stop":
+ raise RuntimeError(
+ f"{stage} response did not finish cleanly: "
+ f"finish_reason={finish_reason!r}"
+ )
+
+
+def run_pass_a(
+ *,
+ client: Any,
+ image_data_url: str,
+ model: str,
+ reasoning_effort: str,
+ output_dir: Path,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Call, parse and normalize Pass A without running Pass B or fusion."""
+
+ output_dir = Path(output_dir).expanduser().resolve()
+ output_dir.mkdir(parents=True, exist_ok=True)
+ (output_dir / "request_prompt.txt").write_text(
+ PASS_A_PROMPT + "\n",
+ encoding="utf-8",
+ )
+ response_text, call_info = run_call(
+ client=client,
+ prompt=PASS_A_PROMPT,
+ image_data_url=image_data_url,
+ model=model,
+ reasoning_effort=reasoning_effort,
+ output_dir=output_dir,
+ )
+ require_clean_finish(call_info, "Pass A")
+ records = parse_pass_a(
+ response_text,
+ **parse_context(call_info),
+ )
+ return normalize_pass_a(records), call_info
+
+
+def compact_assigned_sam3_evidence_context(
+ records: list[dict[str, Any]],
+) -> str:
+ compact = []
+ seen = set()
+ for record in records:
+ evidence_id = str(record.get("evidence_id") or "")
+ source_id = str(record.get("source_object_id") or "")
+ if not re.fullmatch(r"E_\d{3}", evidence_id):
+ raise ValueError(f"invalid assigned evidence ID: {evidence_id!r}")
+ if evidence_id in seen:
+ raise ValueError(f"duplicate assigned evidence ID: {evidence_id}")
+ if not re.fullmatch(r"A_\d{3}", source_id):
+ raise ValueError(f"invalid assigned evidence owner: {source_id!r}")
+ bbox = record.get("bbox_norm_xyxy") or record.get("crop_bbox_norm_xyxy")
+ if not valid_bbox(bbox):
+ raise ValueError(f"invalid assigned evidence bbox: {evidence_id}")
+ seen.add(evidence_id)
+ compact.append([
+ evidence_id,
+ source_id,
+ _validate_compact_prompt(record.get("sam3_prompt")),
+ str(record.get("status") or "unknown"),
+ [round(float(value), 3) for value in bbox],
+ ])
+ expected = [f"E_{index:03d}" for index in range(1, len(compact) + 1)]
+ if [item[0] for item in compact] != expected:
+ raise ValueError("assigned evidence IDs must be continuous")
+ return json.dumps(compact, ensure_ascii=False, separators=(",", ":"))
+
+
+def run_pass_b(
+ *,
+ client: Any,
+ image_data_url: str,
+ pass_a_objects: list[dict[str, Any]],
+ model: str,
+ reasoning_effort: str,
+ output_dir: Path,
+ local_candidate_image_data_url: str | None = None,
+ local_candidate_metadata: list[dict[str, Any]] | None = None,
+ content_host_metadata: list[dict[str, Any]] | None = None,
+ assigned_evidence_image_data_url: str | None = None,
+ assigned_evidence_metadata: list[dict[str, Any]] | None = None,
+ same_mask_conflicts: list[dict[str, Any]] | None = None,
+ content_closure_only: bool = False,
+) -> tuple[dict[str, Any], dict[str, Any], str]:
+ """Call Pass B with at most one bounded local review panel."""
+
+ panel_metadata_present = (
+ local_candidate_metadata is not None
+ or content_host_metadata is not None
+ )
+ if (local_candidate_image_data_url is not None) != panel_metadata_present:
+ raise ValueError(
+ "review panel image and candidate/host metadata must be supplied together"
+ )
+ if (assigned_evidence_image_data_url is not None) != (
+ assigned_evidence_metadata is not None
+ ):
+ raise ValueError(
+ "assigned evidence image and metadata must be supplied together"
+ )
+ output_dir = Path(output_dir).expanduser().resolve()
+ output_dir.mkdir(parents=True, exist_ok=True)
+ candidate_context = None
+ host_context = None
+ evidence_context = None
+ same_mask_conflict_context = None
+ additional_images = None
+ if local_candidate_metadata is not None:
+ if not local_candidate_metadata:
+ raise ValueError("local candidate metadata must not be empty")
+ candidate_context, _ = compact_local_candidate_context(
+ local_candidate_metadata
+ )
+ if content_host_metadata is not None:
+ if not content_host_metadata:
+ raise ValueError("content-host metadata must not be empty")
+ pass_a_by_id = {
+ str(item.get("object_id") or ""): item for item in pass_a_objects
+ }
+ for host in content_host_metadata:
+ source_id = str(host.get("source_object_id") or "")
+ if source_id not in pass_a_by_id:
+ raise ValueError(
+ f"content-host references unknown Pass-A ID: {source_id}"
+ )
+ if category_key(host.get("category")) != category_key(
+ pass_a_by_id[source_id].get("category")
+ ):
+ raise ValueError(
+ f"content-host category does not match {source_id}"
+ )
+ host_context = compact_content_host_context(
+ content_host_metadata
+ )
+ if assigned_evidence_metadata is not None:
+ if not assigned_evidence_metadata:
+ raise ValueError("assigned evidence metadata must not be empty")
+ pass_a_ids = {str(item.get("object_id") or "") for item in pass_a_objects}
+ unknown = sorted(
+ str(item.get("source_object_id") or "")
+ for item in assigned_evidence_metadata
+ if str(item.get("source_object_id") or "") not in pass_a_ids
+ )
+ if unknown:
+ raise ValueError(f"assigned evidence references unknown Pass-A IDs: {unknown}")
+ evidence_context = compact_assigned_sam3_evidence_context(
+ assigned_evidence_metadata
+ )
+ if same_mask_conflicts is not None:
+ if not same_mask_conflicts:
+ raise ValueError("same-mask conflicts must be omitted when empty")
+ pass_a_ids = {str(item.get("object_id") or "") for item in pass_a_objects}
+ unknown = sorted(
+ str(owner_id)
+ for conflict in same_mask_conflicts
+ for owner_id in (conflict.get("object_ids") or [])
+ if str(owner_id) not in pass_a_ids
+ )
+ if unknown:
+ raise ValueError(
+ f"same-mask conflict references unknown Pass-A IDs: {unknown}"
+ )
+ same_mask_conflict_context = compact_same_mask_conflict_context(
+ same_mask_conflicts
+ )
+ additional_images = []
+ if panel_metadata_present:
+ additional_images.append(str(local_candidate_image_data_url))
+ if assigned_evidence_metadata is not None:
+ additional_images.append(str(assigned_evidence_image_data_url))
+ if not additional_images:
+ additional_images = None
+ prompt = build_compact_pass_b_prompt(
+ compact_pass_a_context(pass_a_objects),
+ candidate_context,
+ content_host_registry=host_context,
+ assigned_evidence_registry=evidence_context,
+ same_mask_conflict_registry=same_mask_conflict_context,
+ content_closure_only=content_closure_only,
+ )
+ (output_dir / "request_prompt.txt").write_text(
+ prompt + "\n",
+ encoding="utf-8",
+ )
+ response_text, call_info = run_call(
+ client=client,
+ prompt=prompt,
+ image_data_url=image_data_url,
+ model=model,
+ reasoning_effort=reasoning_effort,
+ output_dir=output_dir,
+ additional_image_data_urls=additional_images,
+ )
+ require_clean_finish(call_info, "Pass B")
+ actions, protocol_audit = parse_compact_pass_b(
+ response_text,
+ pass_a_objects=pass_a_objects,
+ local_candidate_metadata=local_candidate_metadata,
+ assigned_evidence_metadata=assigned_evidence_metadata,
+ **parse_context(call_info),
+ )
+ call_info["response_protocol"] = COMPACT_PASS_B_PROTOCOL
+ call_info["compact_protocol_audit"] = protocol_audit
+ write_json(output_dir / "compact_protocol_audit.json", protocol_audit)
+ return actions, call_info, prompt
+
+def parse_context(call_info: dict[str, Any]) -> dict[str, Any]:
+ return {"finish_reason": call_info.get("finish_reason")}
diff --git a/vlm_agent/reconstruction/perception/inventory/review_assets.py b/vlm_agent/reconstruction/perception/inventory/review_assets.py
new file mode 100644
index 0000000..f8b86f6
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/inventory/review_assets.py
@@ -0,0 +1,132 @@
+"""Human-review assets for normalized object inventories."""
+
+from __future__ import annotations
+
+import json
+import math
+import shutil
+from pathlib import Path
+from typing import Any
+
+from PIL import Image, ImageDraw, ImageFont
+
+
+def _safe_asset_component(value: Any) -> str:
+ value = str(value or "object").strip().lower()
+ cleaned = "".join(ch if (ch.isalnum() or ch in "_-") else "_" for ch in value)
+ return cleaned.strip("_") or "object"
+
+
+def _write_json(path: Path, data: Any) -> None:
+ path.write_text(json.dumps(data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+
+
+def render_bbox_review_assets(
+ image_path: str | Path,
+ objects: list[dict[str, Any]],
+ output_dir: str | Path,
+) -> dict[str, Any]:
+ """Render one context crop per VLM bbox and a compact contact sheet."""
+
+ image_path = Path(image_path)
+ output_dir = Path(output_dir)
+ if output_dir.exists():
+ shutil.rmtree(output_dir)
+ output_dir.mkdir(parents=True, exist_ok=True)
+
+ with Image.open(image_path) as source:
+ image = source.convert("RGB")
+ image_w, image_h = image.size
+ try:
+ font = ImageFont.truetype("DejaVuSans.ttf", 18)
+ small_font = ImageFont.truetype("DejaVuSans.ttf", 14)
+ except OSError:
+ font = ImageFont.load_default()
+ small_font = ImageFont.load_default()
+
+ records = []
+ thumbnails = []
+ for obj in objects:
+ object_id = str(obj["object_id"])
+ category = _safe_asset_component(obj.get("category"))
+ x1, y1, x2, y2 = obj["bbox_2d_norm_xyxy"]
+ bx1, by1, bx2, by2 = (
+ round(x1 * image_w), round(y1 * image_h),
+ round(x2 * image_w), round(y2 * image_h),
+ )
+ box_w = max(1, bx2 - bx1)
+ box_h = max(1, by2 - by1)
+ pad = max(16, round(max(box_w, box_h) * 0.35))
+ crop_x1 = max(0, bx1 - pad)
+ crop_y1 = max(0, by1 - pad)
+ crop_x2 = min(image_w, bx2 + pad)
+ crop_y2 = min(image_h, by2 + pad)
+ crop = image.crop((crop_x1, crop_y1, crop_x2, crop_y2)).copy()
+ draw = ImageDraw.Draw(crop, "RGBA")
+ local_bbox = (bx1 - crop_x1, by1 - crop_y1, bx2 - crop_x1, by2 - crop_y1)
+ line_width = max(2, round(max(crop.size) / 220))
+ draw.rectangle(local_bbox, outline=(255, 40, 40, 255), width=line_width)
+ anchor = obj.get("anchor_2d_norm_xy")
+ if anchor:
+ ax = round(anchor[0] * image_w) - crop_x1
+ ay = round(anchor[1] * image_h) - crop_y1
+ radius = max(4, line_width * 2)
+ draw.ellipse(
+ (ax - radius, ay - radius, ax + radius, ay + radius),
+ fill=(255, 40, 40, 255), outline=(255, 255, 255, 255), width=1,
+ )
+ label = f"{object_id} | {category}"
+ label_box = draw.textbbox((0, 0), label, font=font)
+ draw.rectangle(
+ (0, 0, label_box[2] + 12, label_box[3] + 10),
+ fill=(0, 0, 0, 185),
+ )
+ draw.text((6, 4), label, font=font, fill=(255, 255, 255, 255))
+ crop.thumbnail((640, 640), Image.Resampling.LANCZOS)
+ filename = f"{object_id}_{category}_bbox_review.jpg"
+ asset_path = output_dir / filename
+ crop.save(asset_path, format="JPEG", quality=88, optimize=True)
+ records.append(
+ {
+ "object_id": object_id,
+ "category": obj.get("category"),
+ "bbox_2d_norm_xyxy": obj["bbox_2d_norm_xyxy"],
+ "anchor_2d_norm_xy": anchor,
+ "crop_xyxy_px": [crop_x1, crop_y1, crop_x2, crop_y2],
+ "image": str(asset_path),
+ }
+ )
+ thumbnail = crop.copy()
+ thumbnail.thumbnail((300, 230), Image.Resampling.LANCZOS)
+ thumbnails.append((label, thumbnail))
+
+ cols = 4
+ cell_w, cell_h, label_h = 320, 270, 28
+ rows = max(1, math.ceil(len(thumbnails) / cols))
+ sheet = Image.new("RGB", (cols * cell_w, rows * cell_h), (28, 28, 28))
+ sheet_draw = ImageDraw.Draw(sheet)
+ for index, (label, thumbnail) in enumerate(thumbnails):
+ col, row = index % cols, index // cols
+ left, top = col * cell_w, row * cell_h
+ thumb_x = left + (cell_w - thumbnail.width) // 2
+ thumb_y = top + label_h + (cell_h - label_h - thumbnail.height) // 2
+ sheet.paste(thumbnail, (thumb_x, thumb_y))
+ sheet_draw.text((left + 6, top + 5), label, font=small_font, fill=(255, 255, 255))
+ contact_sheet = output_dir / "bbox_review_contact_sheet.jpg"
+ sheet.save(contact_sheet, format="JPEG", quality=86, optimize=True)
+ manifest_path = output_dir / "bbox_review_manifest.json"
+ _write_json(
+ manifest_path,
+ {
+ "source_image": str(image_path),
+ "source_size": [image_w, image_h],
+ "contact_sheet": str(contact_sheet),
+ "objects": records,
+ },
+ )
+ return {
+ "directory": str(output_dir),
+ "manifest_json": str(manifest_path),
+ "contact_sheet": str(contact_sheet),
+ "object_count": len(records),
+ }
diff --git a/vlm_agent/reconstruction/perception/pipeline/__init__.py b/vlm_agent/reconstruction/perception/pipeline/__init__.py
new file mode 100644
index 0000000..b74d0ea
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/__init__.py
@@ -0,0 +1 @@
+"""Production perception pipeline and mask-refinement components."""
diff --git a/vlm_agent/reconstruction/perception/pipeline/core/__init__.py b/vlm_agent/reconstruction/perception/pipeline/core/__init__.py
new file mode 100644
index 0000000..438d234
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/core/__init__.py
@@ -0,0 +1 @@
+"""Core services for the production perception pipeline."""
diff --git a/vlm_agent/reconstruction/perception/pipeline/core/candidate_review.py b/vlm_agent/reconstruction/perception/pipeline/core/candidate_review.py
new file mode 100644
index 0000000..d9fbfc7
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/core/candidate_review.py
@@ -0,0 +1,1508 @@
+from __future__ import annotations
+
+import copy
+import math
+import re
+from collections import defaultdict
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+from inventory import pipeline as VLM
+from pipeline.refinements import top_zone_candidate_recall as TOP_ZONE
+from pipeline.core.runtime_io import atomic_json
+
+BOUNDED_POST_SAM3A_REVIEW_CONFIG = {
+ "maximum_candidates": 24,
+ "minimum_unassigned_score": 0.60,
+ "minimum_area_pixels": 32,
+ "maximum_area_fraction": 0.04,
+ "successful_mask_exclusion_iou": 0.50,
+ "successful_mask_exclusion_candidate_coverage": 0.80,
+ "duplicate_iou": 0.65,
+ "duplicate_containment": 0.85,
+ "reserved_unassigned_slots": 4,
+}
+
+def select_bounded_post_sam3a_candidates(
+ *,
+ image_path: str | Path,
+ top_candidates: Sequence[Mapping[str, Any]],
+ sam3_registry: Mapping[str, Any],
+ config: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ """Select one bounded mixed review set without creating semantic objects."""
+
+ from PIL import Image
+
+ cfg = dict(BOUNDED_POST_SAM3A_REVIEW_CONFIG)
+ if config:
+ unknown = set(config) - set(cfg)
+ if unknown:
+ raise ValueError(f"unknown bounded candidate config keys: {sorted(unknown)}")
+ cfg.update(config)
+ maximum = int(cfg["maximum_candidates"])
+ if maximum < 1:
+ raise ValueError("maximum_candidates must be positive")
+ if int(cfg["reserved_unassigned_slots"]) < 0:
+ raise ValueError("reserved_unassigned_slots must be non-negative")
+
+ resolved_image = Path(image_path).expanduser().resolve()
+ with Image.open(resolved_image) as source_image:
+ width, height = source_image.size
+ expected_shape = (height, width)
+ image_area = width * height
+ minimum_area = int(cfg["minimum_area_pixels"])
+ maximum_area = int(math.floor(float(cfg["maximum_area_fraction"]) * image_area))
+ if minimum_area < 1 or maximum_area < minimum_area:
+ raise ValueError("invalid bounded candidate area limits")
+
+ successful_masks: list[tuple[str, Any]] = []
+ for obj in sam3_registry.get("objects") or []:
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "success":
+ continue
+ object_id = str(obj.get("object_id") or "")
+ try:
+ mask = TOP_ZONE.load_mask(sam3["mask_path"], expected_shape)
+ except Exception as exc:
+ raise ValueError(
+ f"cannot validate successful SAM3 mask for {object_id}: "
+ f"{type(exc).__name__}: {exc}"
+ ) from exc
+ if not mask.any():
+ raise ValueError(f"successful SAM3 mask is empty: {object_id}")
+ successful_masks.append((object_id, mask))
+
+ eligible: list[dict[str, Any]] = []
+ semantic_aliases: list[dict[str, Any]] = []
+ validation_rejections: list[dict[str, Any]] = []
+
+ def validate_candidate(raw_value: Mapping[str, Any], namespace: str) -> None:
+ raw = dict(raw_value)
+ review_id = str(raw.get("candidate_id") or "")
+ upstream_id = str(raw.get("source_candidate_id") or review_id)
+ prompt = str(
+ raw.get("prompt")
+ or raw.get("sam3_prompt")
+ or raw.get("category_hint")
+ or ""
+ ).strip()
+ rejection = {
+ "source_namespace": namespace,
+ "source_candidate_id": review_id,
+ "source_upstream_candidate_id": upstream_id,
+ "prompt": prompt,
+ "score": None,
+ "area_pixels": None,
+ "reasons": [],
+ }
+ try:
+ score = float(raw.get("score"))
+ except (TypeError, ValueError):
+ score = math.nan
+ if not math.isfinite(score) or not 0.0 <= score <= 1.0:
+ rejection["reasons"].append("invalid_score")
+ else:
+ rejection["score"] = score
+ if (
+ namespace == "sam3_a_unassigned"
+ and score < float(cfg["minimum_unassigned_score"])
+ ):
+ rejection["reasons"].append(
+ "unassigned_score_below_threshold"
+ )
+ if not review_id:
+ rejection["reasons"].append("missing_source_candidate_id")
+ if not prompt:
+ rejection["reasons"].append("missing_prompt")
+
+ mask_path = Path(str(raw.get("mask_path") or "")).expanduser().resolve()
+ overlay_path = Path(str(raw.get("overlay_path") or "")).expanduser().resolve()
+ mask = None
+ try:
+ mask = TOP_ZONE.load_mask(mask_path, expected_shape)
+ if not mask.any():
+ rejection["reasons"].append("empty_mask")
+ except Exception as exc:
+ rejection["reasons"].append(
+ f"invalid_mask:{type(exc).__name__}"
+ )
+ try:
+ if not overlay_path.is_file():
+ raise FileNotFoundError(overlay_path)
+ with Image.open(overlay_path) as overlay:
+ if overlay.size != (width, height):
+ raise ValueError("shape_mismatch")
+ except Exception as exc:
+ rejection["reasons"].append(
+ f"invalid_overlay:{type(exc).__name__}"
+ )
+
+ if mask is None or not mask.any():
+ validation_rejections.append(rejection)
+ return
+ area = int(mask.sum())
+ rejection["area_pixels"] = area
+ declared_area = raw.get("area_pixels")
+ if declared_area is not None and int(declared_area) != area:
+ rejection["reasons"].append("declared_area_mismatch")
+ if area < minimum_area:
+ rejection["reasons"].append("area_too_small")
+ if area > maximum_area:
+ rejection["reasons"].append("area_too_large")
+ bbox = TOP_ZONE.mask_bbox(mask)
+ successful_overlaps = []
+ for object_id, successful_mask in successful_masks:
+ metrics = TOP_ZONE.mask_overlap(mask, successful_mask)
+ if (
+ metrics["iou"] >= float(cfg["successful_mask_exclusion_iou"])
+ or metrics["left_coverage"]
+ >= float(cfg["successful_mask_exclusion_candidate_coverage"])
+ ):
+ successful_overlaps.append(
+ {
+ "object_id": object_id,
+ "iou": round(metrics["iou"], 6),
+ "candidate_coverage": round(metrics["left_coverage"], 6),
+ }
+ )
+ if successful_overlaps:
+ rejection["reasons"].append("overlaps_successful_object")
+ rejection["successful_object_overlaps"] = successful_overlaps
+ if rejection["reasons"]:
+ if rejection["reasons"] == ["unassigned_score_below_threshold"]:
+ semantic_aliases.append({
+ "source_namespace": namespace,
+ "source_candidate_id": review_id,
+ "prompt": prompt,
+ "score": score,
+ "area_pixels": area,
+ "bbox_abs_xyxy": bbox,
+ "mask": mask,
+ })
+ validation_rejections.append(rejection)
+ return
+ eligible.append(
+ {
+ "source_namespace": namespace,
+ "source_candidate_id": review_id,
+ "source_upstream_candidate_id": upstream_id,
+ "prompt": prompt,
+ "score": score,
+ "prompt_evidence": [{
+ "prompt": prompt,
+ "score": score,
+ "source_namespace": namespace,
+ "source_candidate_id": review_id,
+ }],
+ "area_pixels": area,
+ "area_fraction": area / image_area,
+ "bbox_abs_xyxy": bbox,
+ "bbox_norm_xyxy": [
+ bbox[0] / width,
+ bbox[1] / height,
+ bbox[2] / width,
+ bbox[3] / height,
+ ],
+ "source_mask_path": str(mask_path),
+ "source_overlay_path": str(overlay_path),
+ "category_hint": str(raw.get("category_hint") or ""),
+ "target_object_ids": [
+ str(value) for value in raw.get("target_object_ids") or []
+ ],
+ "recall_sources": [
+ str(value) for value in raw.get("recall_sources") or []
+ ],
+ "local_instance_evidence": bool(
+ raw.get("local_instance_evidence")
+ ),
+ "host_source_object_id": str(
+ raw.get("host_source_object_id") or ""
+ ),
+ "grouped_component_count": int(
+ raw.get("grouped_component_count") or 0
+ ),
+ "mask": mask,
+ }
+ )
+
+ for candidate in top_candidates:
+ validate_candidate(candidate, "top_zone")
+ for candidate in sam3_registry.get("unassigned_sam3_candidates") or []:
+ validate_candidate(candidate, "sam3_a_unassigned")
+
+ eligible.sort(
+ key=lambda item: (
+ {
+ "top_zone": 0,
+ "sam3_a_unassigned": 1,
+ }.get(item["source_namespace"], 3),
+ -float(item["score"]),
+ item["bbox_abs_xyxy"][1],
+ item["bbox_abs_xyxy"][0],
+ item["source_candidate_id"],
+ )
+ )
+ deduplicated: list[dict[str, Any]] = []
+ duplicate_rejections: list[dict[str, Any]] = []
+ for candidate in eligible:
+ duplicate = None
+ for existing in deduplicated:
+ metrics = TOP_ZONE.mask_overlap(candidate["mask"], existing["mask"])
+ if (
+ metrics["iou"] >= float(cfg["duplicate_iou"])
+ or max(metrics["left_coverage"], metrics["right_coverage"])
+ >= float(cfg["duplicate_containment"])
+ ):
+ duplicate = (existing, metrics)
+ break
+ if duplicate is None:
+ deduplicated.append(candidate)
+ else:
+ duplicate[0]["prompt_evidence"].extend(candidate["prompt_evidence"])
+ duplicate_rejections.append(
+ {
+ "source_namespace": candidate["source_namespace"],
+ "source_candidate_id": candidate["source_candidate_id"],
+ "duplicate_of_namespace": duplicate[0]["source_namespace"],
+ "duplicate_of_candidate_id": duplicate[0]["source_candidate_id"],
+ "overlap": {
+ key: round(value, 6) for key, value in duplicate[1].items()
+ },
+ }
+ )
+ for alias in semantic_aliases:
+ for existing in deduplicated:
+ metrics = TOP_ZONE.mask_overlap(alias["mask"], existing["mask"])
+ if not (
+ metrics["iou"] >= float(cfg["duplicate_iou"])
+ or max(metrics["left_coverage"], metrics["right_coverage"])
+ >= float(cfg["duplicate_containment"])
+ ):
+ continue
+ evidence = {
+ "prompt": alias["prompt"],
+ "score": alias["score"],
+ "source_namespace": alias["source_namespace"],
+ "source_candidate_id": alias["source_candidate_id"],
+ }
+ if evidence["prompt"] not in {
+ item["prompt"] for item in existing["prompt_evidence"]
+ }:
+ existing["prompt_evidence"].append(evidence)
+ break
+
+
+ top_pool = [
+ item for item in deduplicated if item["source_namespace"] == "top_zone"
+ ]
+ unassigned_pool = [
+ item
+ for item in deduplicated
+ if item["source_namespace"] == "sam3_a_unassigned"
+ ]
+ unassigned_reserved = min(
+ int(cfg["reserved_unassigned_slots"]),
+ len(unassigned_pool),
+ maximum,
+ )
+ top_take = min(
+ len(top_pool),
+ maximum - unassigned_reserved,
+ )
+ selected = list(top_pool[:top_take])
+ selected.extend(unassigned_pool[:unassigned_reserved])
+ for pool, start_index in (
+ (unassigned_pool, unassigned_reserved),
+ (top_pool, top_take),
+ ):
+ if len(selected) >= maximum:
+ break
+ selected.extend(pool[start_index : start_index + maximum - len(selected)])
+ selected_keys = {
+ (item["source_namespace"], item["source_candidate_id"])
+ for item in selected
+ }
+ overflow_rejections = [
+ {
+ **{key: value for key, value in item.items() if key != "mask"},
+ "reason": "bounded_panel_capacity",
+ }
+ for item in deduplicated
+ if (item["source_namespace"], item["source_candidate_id"])
+ not in selected_keys
+ ]
+ hard_errors: list[dict[str, Any]] = []
+ if overflow_rejections:
+ hard_errors.append(
+ {
+ "type": "bounded_candidate_capacity_overflow",
+ "count": len(overflow_rejections),
+ "source_candidate_ids": [
+ item["source_candidate_id"]
+ for item in overflow_rejections
+ ],
+ }
+ )
+ nonblocking_validation_reasons = {
+ "invalid_score",
+ "unassigned_score_below_threshold",
+ "empty_mask",
+ "area_too_small",
+ "area_too_large",
+ "overlaps_successful_object",
+ }
+ blocking_validation_rejections = []
+ for rejection in validation_rejections:
+ score_value = rejection.get("score")
+ area_value = rejection.get("area_pixels")
+ unresolved_reasons = [
+ reason
+ for reason in rejection["reasons"]
+ if reason not in nonblocking_validation_reasons
+ and not str(reason).startswith("invalid_mask:")
+ ]
+ if (
+ isinstance(score_value, (int, float))
+ and float(score_value) >= float(cfg["minimum_unassigned_score"])
+ and isinstance(area_value, int)
+ and minimum_area <= area_value <= maximum_area
+ and unresolved_reasons
+ ):
+ blocking_validation_rejections.append(
+ {
+ **rejection,
+ "unresolved_reasons": unresolved_reasons,
+ }
+ )
+ if blocking_validation_rejections:
+ hard_errors.append(
+ {
+ "type": "bounded_candidate_validation_unresolved",
+ "count": len(blocking_validation_rejections),
+ "source_candidate_ids": [
+ item["source_candidate_id"]
+ for item in blocking_validation_rejections
+ ],
+ }
+ )
+ for rank, candidate in enumerate(selected, start=1):
+ candidate["selection_priority_rank"] = rank
+ selected.sort(
+ key=lambda item: (
+ item["bbox_abs_xyxy"][1],
+ item["bbox_abs_xyxy"][0],
+ item["selection_priority_rank"],
+ )
+ )
+ for index, candidate in enumerate(selected, start=1):
+ candidate["candidate_id"] = f"LC_{index:03d}"
+
+ selected_public = [
+ {key: value for key, value in item.items() if key != "mask"}
+ for item in selected
+ ]
+ selected_counts = {
+ namespace: sum(
+ item["source_namespace"] == namespace for item in selected
+ )
+ for namespace in (
+ "top_zone",
+ "sam3_a_unassigned",
+ )
+ }
+ audit = {
+ "schema_version": "scenemosaic_bounded_post_sam3a_candidate_selection",
+ "status": "candidates_found" if selected else "no_candidates",
+ "source_image": str(resolved_image),
+ "image_size": [width, height],
+ "config": cfg,
+ "input_counts": {
+ "top_zone": len(top_candidates),
+ "sam3_a_unassigned": len(
+ sam3_registry.get("unassigned_sam3_candidates") or []
+ ),
+ "successful_object_masks": len(successful_masks),
+ },
+ "eligible_before_deduplication": len(eligible),
+ "deduplicated_count": len(deduplicated),
+ "selected_count": len(selected),
+ "selected_counts_by_namespace": selected_counts,
+ "unassigned_slot_reserved": unassigned_reserved,
+ "all_eligible_unassigned_silently_crowded_out": bool(
+ unassigned_pool and selected_counts["sam3_a_unassigned"] == 0
+ ),
+ "selected": selected_public,
+ "validation_rejections": validation_rejections,
+ "blocking_validation_rejections": blocking_validation_rejections,
+ "duplicate_rejections": duplicate_rejections,
+ "overflow_rejections": overflow_rejections,
+ "hard_errors": hard_errors,
+ "additional_vlm_calls": 0,
+ "logical_pipeline_vlm_api_calls": 2,
+ "api_calls_unchanged": True,
+ }
+ if audit["all_eligible_unassigned_silently_crowded_out"]:
+ raise AssertionError("eligible SAM3-A candidates were silently crowded out")
+ return {
+ "filtered": {
+ "status": audit["status"],
+ "image_size": [width, height],
+ "config": cfg,
+ "selected": selected,
+ "geometric_rejections": validation_rejections,
+ "duplicate_rejections": duplicate_rejections,
+ "overflow_rejections": overflow_rejections,
+ },
+ "audit": audit,
+ }
+
+
+def materialize_bounded_post_sam3a_review(
+ *,
+ image_path: str | Path,
+ top_candidates: Sequence[Mapping[str, Any]],
+ sam3_registry: Mapping[str, Any],
+ output_dir: str | Path,
+ config: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ selection = select_bounded_post_sam3a_candidates(
+ image_path=image_path,
+ top_candidates=top_candidates,
+ sam3_registry=sam3_registry,
+ config=config,
+ )
+ review = TOP_ZONE.materialize_top_zone_review(
+ image_path,
+ selection["filtered"],
+ output_dir,
+ panel_title=(
+ "MERGED TOP-ZONE + "
+ "SAM3-A UNASSIGNED CANDIDATES"
+ ),
+ schema_version="scenemosaic_bounded_post_sam3a_candidate_review",
+ )
+ audit = {
+ **selection["audit"],
+ "review_manifest": review["manifest_path"],
+ "numbered_candidate_panel": review["panel_path"],
+ }
+ audit_path = Path(output_dir).expanduser().resolve() / "candidate_selection_audit.json"
+ atomic_json(audit_path, audit)
+ return {
+ **review,
+ "selection_audit": audit,
+ "selection_audit_path": str(audit_path),
+ }
+
+
+LOCAL_MATCH_THRESHOLDS = {
+ "minimum_candidate_score": 0.60,
+ "minimum_semantic_score": 0.85,
+ "minimum_iou": 0.65,
+ "minimum_candidate_coverage": 0.80,
+ "minimum_addition_coverage": 0.75,
+ "minimum_area_ratio": 0.50,
+ "maximum_area_ratio": 2.00,
+ "maximum_relative_center_distance": 0.25,
+ "minimum_total_score": 0.82,
+ "minimum_runner_up_margin": 0.10,
+}
+
+
+def _semantic_tokens(value: Any) -> set[str]:
+ return set(re.findall(r"[a-z0-9]+", str(value or "").lower()))
+
+
+def _local_candidate_semantic_score(
+ candidate: Mapping[str, Any],
+ addition: Mapping[str, Any],
+) -> float:
+ # The SAM3 prompt is the sole instance-level binding label. Category is
+ # retained in the formal object schema, but never used for mask binding.
+ candidate_key = VLM.category_key(candidate.get("prompt"))
+ if not candidate_key:
+ return 0.0
+ prompt_key = VLM.category_key(addition.get("sam3_prompt"))
+ if prompt_key and candidate_key == prompt_key:
+ return 1.0
+
+ candidate_tokens = _semantic_tokens(candidate.get("prompt"))
+ primary_tokens = _semantic_tokens(addition.get("sam3_prompt"))
+ all_tokens = set(primary_tokens)
+ for key in ("instance_name", "object_type", "description"):
+ all_tokens.update(_semantic_tokens(addition.get(key)))
+ if (
+ candidate_tokens
+ and candidate_tokens.issubset(all_tokens)
+ and candidate_tokens.intersection(primary_tokens)
+ ):
+ return 0.85
+ return 0.0
+
+
+def _local_candidate_pair_metrics(
+ candidate: Mapping[str, Any],
+ addition: Mapping[str, Any],
+) -> dict[str, float]:
+ candidate_bbox = [float(value) for value in candidate["bbox_norm_xyxy"]]
+ addition_bbox = [float(value) for value in addition["bbox_2d_norm_xyxy"]]
+ overlap = VLM.bbox_metrics(candidate_bbox, addition_bbox)
+ candidate_area = (
+ (candidate_bbox[2] - candidate_bbox[0])
+ * (candidate_bbox[3] - candidate_bbox[1])
+ )
+ addition_area = (
+ (addition_bbox[2] - addition_bbox[0])
+ * (addition_bbox[3] - addition_bbox[1])
+ )
+ candidate_center = (
+ (candidate_bbox[0] + candidate_bbox[2]) / 2.0,
+ (candidate_bbox[1] + candidate_bbox[3]) / 2.0,
+ )
+ addition_center = (
+ (addition_bbox[0] + addition_bbox[2]) / 2.0,
+ (addition_bbox[1] + addition_bbox[3]) / 2.0,
+ )
+ scale = max(
+ math.hypot(
+ candidate_bbox[2] - candidate_bbox[0],
+ candidate_bbox[3] - candidate_bbox[1],
+ ),
+ math.hypot(
+ addition_bbox[2] - addition_bbox[0],
+ addition_bbox[3] - addition_bbox[1],
+ ),
+ )
+ relative_center_distance = (
+ math.dist(candidate_center, addition_center) / scale if scale else math.inf
+ )
+ semantic_score = _local_candidate_semantic_score(candidate, addition)
+ geometry_score = (
+ 0.50 * overlap["iou"]
+ + 0.25 * overlap["left_coverage"]
+ + 0.25 * overlap["right_coverage"]
+ )
+ total_score = 0.75 * geometry_score + 0.25 * semantic_score
+ return {
+ "iou": overlap["iou"],
+ "candidate_coverage": overlap["left_coverage"],
+ "addition_coverage": overlap["right_coverage"],
+ "area_ratio": addition_area / candidate_area if candidate_area else math.inf,
+ "relative_center_distance": relative_center_distance,
+ "semantic_score": semantic_score,
+ "geometry_score": geometry_score,
+ "total_score": total_score,
+ }
+
+
+def match_local_candidates_to_b_additions(
+ fused_objects: Sequence[Mapping[str, Any]],
+ local_candidates: Sequence[Mapping[str, Any]],
+ *,
+ explicitly_rejected_candidate_ids: Sequence[str] = (),
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ # Only accepted B additions may receive local evidence. The matcher never
+ # creates a new object from an LC candidate.
+ thresholds = dict(LOCAL_MATCH_THRESHOLDS)
+ additions: list[tuple[str, dict[str, Any]]] = []
+ seen_addition_ids: set[str] = set()
+ for raw_object in fused_objects:
+ obj = dict(raw_object)
+ sources = [
+ source
+ for source in obj.get("proposal_sources") or []
+ if source.get("pass") == "B" and source.get("action") == "added"
+ ]
+ if not sources:
+ continue
+ if len(sources) != 1:
+ raise ValueError("a fused proposal has multiple B addition sources")
+ source_id = str(sources[0].get("source_id") or "")
+ if not re.fullmatch(r"B_\d{3}", source_id):
+ raise ValueError(f"invalid B addition source ID: {source_id!r}")
+ if source_id in seen_addition_ids:
+ raise ValueError(f"duplicate accepted B addition source: {source_id}")
+ seen_addition_ids.add(source_id)
+ if not VLM.valid_bbox(obj.get("bbox_2d_norm_xyxy")):
+ raise ValueError(f"invalid accepted B addition bbox: {source_id}")
+ additions.append((source_id, obj))
+
+ candidates: list[dict[str, Any]] = []
+ seen_candidate_ids: set[str] = set()
+ for raw_candidate in local_candidates:
+ candidate = dict(raw_candidate)
+ candidate_id = str(candidate.get("candidate_id") or "")
+ if not re.fullmatch(r"LC_\d{3}", candidate_id):
+ raise ValueError(f"invalid local candidate ID: {candidate_id!r}")
+ if candidate_id in seen_candidate_ids:
+ raise ValueError(f"duplicate local candidate ID: {candidate_id}")
+ seen_candidate_ids.add(candidate_id)
+ if not VLM.valid_bbox(candidate.get("bbox_norm_xyxy")):
+ raise ValueError(f"invalid local candidate bbox: {candidate_id}")
+ score = float(candidate.get("score") or 0.0)
+ if not math.isfinite(score) or not 0.0 <= score <= 1.0:
+ raise ValueError(f"invalid local candidate score: {candidate_id}")
+ candidates.append(candidate)
+
+ if isinstance(explicitly_rejected_candidate_ids, (str, bytes)):
+ raise ValueError("explicit local candidate rejections must be a sequence")
+ rejected_candidate_ids = [
+ str(value) for value in explicitly_rejected_candidate_ids
+ ]
+ if len(rejected_candidate_ids) != len(set(rejected_candidate_ids)):
+ raise ValueError("duplicate explicit local candidate rejection ID")
+ if not all(
+ re.fullmatch(r"LC_\d{3}", value) for value in rejected_candidate_ids
+ ):
+ raise ValueError("invalid explicit local candidate rejection ID")
+ unknown_rejections = sorted(set(rejected_candidate_ids) - seen_candidate_ids)
+ if unknown_rejections:
+ raise ValueError(
+ f"unknown explicit local candidate rejection IDs: {unknown_rejections}"
+ )
+ rejected_candidate_id_set = set(rejected_candidate_ids)
+
+ pair_records: list[dict[str, Any]] = []
+ eligible_by_candidate: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ eligible_by_addition: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ for candidate in candidates:
+ candidate_id = str(candidate["candidate_id"])
+ candidate_score = float(candidate.get("score") or 0.0)
+ for addition_id, addition in additions:
+ metrics = _local_candidate_pair_metrics(candidate, addition)
+ rejection_reasons = []
+ if candidate_score < thresholds["minimum_candidate_score"]:
+ rejection_reasons.append("candidate_score")
+ if metrics["semantic_score"] < thresholds["minimum_semantic_score"]:
+ rejection_reasons.append("semantic_score")
+ if metrics["iou"] < thresholds["minimum_iou"]:
+ rejection_reasons.append("iou")
+ if metrics["candidate_coverage"] < thresholds["minimum_candidate_coverage"]:
+ rejection_reasons.append("candidate_coverage")
+ if metrics["addition_coverage"] < thresholds["minimum_addition_coverage"]:
+ rejection_reasons.append("addition_coverage")
+ if not (
+ thresholds["minimum_area_ratio"]
+ <= metrics["area_ratio"]
+ <= thresholds["maximum_area_ratio"]
+ ):
+ rejection_reasons.append("area_ratio")
+ if (
+ metrics["relative_center_distance"]
+ > thresholds["maximum_relative_center_distance"]
+ ):
+ rejection_reasons.append("relative_center_distance")
+ if metrics["total_score"] < thresholds["minimum_total_score"]:
+ rejection_reasons.append("total_score")
+ record = {
+ "candidate_id": candidate_id,
+ "addition_id": addition_id,
+ "proposal_id": addition.get("object_id"),
+ "eligible": not rejection_reasons,
+ "rejection_reasons": rejection_reasons,
+ **{key: round(value, 6) for key, value in metrics.items()},
+ }
+ pair_records.append(record)
+ if record["eligible"]:
+ eligible_by_candidate[candidate_id].append(record)
+ if candidate_id not in rejected_candidate_id_set:
+ eligible_by_addition[addition_id].append(record)
+
+ def ranked(records: Sequence[Mapping[str, Any]], other_key: str) -> list[dict[str, Any]]:
+ return sorted(
+ (dict(record) for record in records),
+ key=lambda record: (-float(record["total_score"]), str(record[other_key])),
+ )
+
+ def unique_best(records: Sequence[Mapping[str, Any]], other_key: str) -> dict[str, Any] | None:
+ ordered = ranked(records, other_key)
+ if not ordered:
+ return None
+ if len(ordered) > 1 and (
+ float(ordered[0]["total_score"])
+ - float(ordered[1]["total_score"])
+ < thresholds["minimum_runner_up_margin"]
+ ):
+ return None
+ return ordered[0]
+
+ decisions: list[dict[str, Any]] = [
+ {
+ "candidate_id": candidate_id,
+ "decision": "reject",
+ "target_object_id": None,
+ "reason": "explicit Pass-B not-new-object disposition",
+ }
+ for candidate_id in rejected_candidate_ids
+ ]
+ matches: list[dict[str, Any]] = []
+ matched_candidate_ids: set[str] = set()
+ matched_addition_ids: set[str] = set()
+ for candidate in candidates:
+ candidate_id = str(candidate["candidate_id"])
+ if candidate_id in rejected_candidate_id_set:
+ continue
+ best = unique_best(eligible_by_candidate.get(candidate_id, []), "addition_id")
+ if best is None:
+ continue
+ addition_id = str(best["addition_id"])
+ reverse_best = unique_best(
+ eligible_by_addition.get(addition_id, []), "candidate_id"
+ )
+ if reverse_best is None or str(reverse_best["candidate_id"]) != candidate_id:
+ continue
+ if addition_id in matched_addition_ids:
+ raise AssertionError("local candidate matcher violated one-to-one assignment")
+ matched_candidate_ids.add(candidate_id)
+ matched_addition_ids.add(addition_id)
+ decisions.append(
+ {
+ "candidate_id": candidate_id,
+ "decision": "bind_to_addition",
+ "target_object_id": addition_id,
+ "reason": "unique local bbox semantic match",
+ }
+ )
+ matches.append(
+ {
+ key: best[key]
+ for key in (
+ "candidate_id",
+ "addition_id",
+ "proposal_id",
+ "iou",
+ "candidate_coverage",
+ "addition_coverage",
+ "area_ratio",
+ "relative_center_distance",
+ "semantic_score",
+ "geometry_score",
+ "total_score",
+ )
+ }
+ )
+
+ hard_errors: list[dict[str, Any]] = []
+ conflicting_rejected_candidate_ids = []
+ for candidate_id in rejected_candidate_ids:
+ eligible = eligible_by_candidate.get(candidate_id, [])
+ if not eligible:
+ continue
+ conflicting_rejected_candidate_ids.append(candidate_id)
+ hard_errors.append(
+ {
+ "type": "conflicting_local_candidate_disposition",
+ "candidate_id": candidate_id,
+ "eligible_addition_ids": sorted(
+ str(record["addition_id"]) for record in eligible
+ ),
+ "reason": (
+ "candidate was explicitly rejected but also has a "
+ "high-confidence B addition match"
+ ),
+ }
+ )
+
+ unresolved_candidate_ids = []
+ for candidate in candidates:
+ candidate_id = str(candidate["candidate_id"])
+ if candidate_id in rejected_candidate_id_set:
+ continue
+ if candidate_id in matched_candidate_ids:
+ continue
+ unresolved_candidate_ids.append(candidate_id)
+ eligible = eligible_by_candidate.get(candidate_id, [])
+ hard_errors.append(
+ {
+ "type": (
+ "ambiguous_local_candidate_match"
+ if eligible
+ else "unmatched_local_candidate"
+ ),
+ "candidate_id": candidate_id,
+ "eligible_addition_ids": sorted(
+ str(record["addition_id"]) for record in eligible
+ ),
+ "reason": (
+ "no unique mutual high-confidence B addition"
+ if eligible
+ else "no high-confidence B addition"
+ ),
+ }
+ )
+
+ if len(decisions) + len(unresolved_candidate_ids) != len(candidates):
+ raise AssertionError("local candidate accounting invariant failed")
+ partition_sets = [
+ set(rejected_candidate_ids),
+ set(matched_candidate_ids),
+ set(unresolved_candidate_ids),
+ ]
+ partition_disjoint = all(
+ not partition_sets[left] & partition_sets[right]
+ for left in range(len(partition_sets))
+ for right in range(left + 1, len(partition_sets))
+ )
+ complete_partition = (
+ partition_disjoint
+ and set().union(*partition_sets) == seen_candidate_ids
+ )
+ if not complete_partition:
+ raise AssertionError("local candidate partition invariant failed")
+ return decisions, {
+ "schema_version": "scenemosaic_local_candidate_match",
+ "mode": "deterministic_bbox_semantic_one_to_one",
+ "thresholds": thresholds,
+ "candidate_count": len(candidates),
+ "accepted_b_addition_count": len(additions),
+ "evaluated_pair_count": len(pair_records),
+ "eligible_pair_count": sum(
+ 1 for record in pair_records if record["eligible"]
+ ),
+ "explicit_reject_count": len(rejected_candidate_ids),
+ "explicit_rejected_candidate_ids": rejected_candidate_ids,
+ "conflicting_rejected_candidate_ids": (
+ conflicting_rejected_candidate_ids
+ ),
+ "match_count": len(matches),
+ "matched_candidate_ids": sorted(matched_candidate_ids),
+ "matches": matches,
+ "unresolved_candidate_ids": unresolved_candidate_ids,
+ "complete_partition": complete_partition,
+ "partition_disjoint": partition_disjoint,
+ "pair_evaluations": pair_records,
+ "hard_errors": hard_errors,
+ "status": "pass" if not hard_errors else "fail",
+ "creates_objects_from_local_candidates": False,
+ }
+
+
+def promote_repeated_top_fixture_candidates(
+ *,
+ pass_a_objects: Sequence[Mapping[str, Any]],
+ pass_b_actions: Mapping[str, Any],
+ local_candidates: Sequence[Mapping[str, Any]],
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ """Recover corroborated top-edge fixtures rejected by compact review.
+ """
+ actions = copy.deepcopy(dict(pass_b_actions))
+ had_local_candidate_decisions = "local_candidate_decisions" in actions
+ additions = list(actions.get("additions") or [])
+ removal_ids = {str(value) for value in actions.get("removals") or []}
+
+ def box(item: Mapping[str, Any]) -> list[float] | None:
+ raw = item.get("bbox_2d_norm_xyxy") or item.get("bbox_norm_xyxy")
+ try:
+ values = [float(value) for value in raw]
+ except (TypeError, ValueError):
+ return None
+ if (
+ len(values) != 4
+ or not (0.0 <= values[0] < values[2] <= 1.0)
+ or not (0.0 <= values[1] < values[3] <= 1.0)
+ ):
+ return None
+ return values
+
+ def tokens(item: Mapping[str, Any]) -> set[str]:
+ return set(
+ re.findall(
+ r"[a-z0-9]+",
+ " ".join(
+ str(item.get(key) or "")
+ for key in ("category", "sam3_prompt", "description", "prompt")
+ ).lower(),
+ )
+ )
+
+ def is_top_fixture(item: Mapping[str, Any]) -> bool:
+ candidate_box = box(item)
+ item_tokens = tokens(item)
+ return bool(
+ candidate_box
+ and candidate_box[3] <= 0.18
+ and item_tokens & {"light", "lamp", "lighting"}
+ )
+
+ def overlap_metrics(
+ left: Sequence[float], right: Sequence[float]
+ ) -> tuple[float, float, float]:
+ inter_left = max(left[0], right[0])
+ inter_top = max(left[1], right[1])
+ inter_right = min(left[2], right[2])
+ inter_bottom = min(left[3], right[3])
+ inter_area = max(0.0, inter_right - inter_left) * max(
+ 0.0, inter_bottom - inter_top
+ )
+ left_area = max(0.0, left[2] - left[0]) * max(
+ 0.0, left[3] - left[1]
+ )
+ right_area = max(0.0, right[2] - right[0]) * max(
+ 0.0, right[3] - right[1]
+ )
+ union_area = left_area + right_area - inter_area
+ return (
+ inter_area / union_area if union_area else 0.0,
+ inter_area / left_area if left_area else 0.0,
+ inter_area / right_area if right_area else 0.0,
+ )
+
+ def matching_addition_id(
+ candidate: Mapping[str, Any], candidate_box: Sequence[float]
+ ) -> str | None:
+ """Find the unique ordinary addition representing an LC candidate.
+
+ A compact response can incorrectly reject an LC while emitting the
+ same physical object as an ordinary addition. This is a protocol
+ inconsistency, not evidence for a second object.
+ """
+ candidate_tokens = tokens(candidate)
+ matches: list[str] = []
+ for index, addition in enumerate(additions, start=1):
+ addition_box = box(addition)
+ shared_tokens = candidate_tokens & tokens(addition)
+ if addition_box is None or not shared_tokens:
+ continue
+ iou, candidate_coverage, addition_coverage = overlap_metrics(
+ candidate_box, addition_box
+ )
+ if (
+ iou >= 0.55
+ or candidate_coverage >= 0.80
+ or addition_coverage >= 0.80
+ ):
+ matches.append(f"B_{index:03d}")
+ return matches[0] if len(matches) == 1 else None
+
+ repeated_peer_count = sum(
+ 1 for item in pass_a_objects if is_top_fixture(item)
+ )
+ source_boxes = [
+ (str(item.get("object_id") or ""), box(item))
+ for item in pass_a_objects
+ if is_top_fixture(item)
+ ]
+ promoted: list[dict[str, Any]] = []
+ retained_decisions: list[dict[str, Any]] = []
+ for raw_decision in actions.get("local_candidate_decisions") or []:
+ decision = copy.deepcopy(dict(raw_decision))
+ candidate_id = str(decision.get("candidate_id") or "")
+ candidate = next(
+ (
+ item
+ for item in local_candidates
+ if str(item.get("candidate_id") or "") == candidate_id
+ ),
+ None,
+ )
+ candidate_box = box(candidate or {})
+ evidence = list((candidate or {}).get("prompt_evidence") or [])
+ evidence_sources = {
+ str(item.get("source_namespace") or "")
+ for item in evidence
+ if isinstance(item, Mapping)
+ }
+ evidence_scores = [
+ float(item.get("score") or 0.0)
+ for item in evidence
+ if isinstance(item, Mapping)
+ ]
+ candidate_tokens = tokens(candidate or {})
+ candidate_center = (
+ ((candidate_box[0] + candidate_box[2]) / 2.0,
+ (candidate_box[1] + candidate_box[3]) / 2.0)
+ if candidate_box is not None
+ else None
+ )
+ matching_retained_source = False
+ if candidate_center is not None:
+ matching_retained_source = any(
+ source_id not in removal_ids
+ and source_box is not None
+ and source_box[0] <= candidate_center[0] <= source_box[2]
+ and source_box[1] <= candidate_center[1] <= source_box[3]
+ for source_id, source_box in source_boxes
+ )
+ qualifies = (
+ decision.get("decision") == "reject"
+ and candidate is not None
+ and str(candidate.get("source_namespace") or "") == "top_zone"
+ and candidate_box is not None
+ and candidate_box[3] <= 0.18
+ and bool(candidate_tokens & {"light", "lamp", "lighting"})
+ and repeated_peer_count >= 2
+ and len(evidence_sources) >= 2
+ and max(evidence_scores, default=0.0) >= 0.55
+ and not matching_retained_source
+ )
+ if not qualifies:
+ retained_decisions.append(decision)
+ continue
+ existing_addition_id = matching_addition_id(candidate, candidate_box)
+ if existing_addition_id is not None:
+ decision.update(
+ {
+ "decision": "bind_to_addition",
+ "target_object_id": existing_addition_id,
+ "reason": (
+ "compact Pass-B rejected the local candidate but "
+ "emitted the same corroborated fixture as an "
+ "ordinary addition"
+ ),
+ }
+ )
+ retained_decisions.append(decision)
+ promoted.append(
+ {
+ "candidate_id": candidate_id,
+ "target_object_id": existing_addition_id,
+ "mode": "bound_to_existing_addition",
+ }
+ )
+ continue
+ semantic_prompt = next(
+ (
+ str(item.get("prompt") or "").strip()
+ for item in sorted(
+ evidence,
+ key=lambda item: float(item.get("score") or 0.0),
+ reverse=True,
+ )
+ if "light" in str(item.get("prompt") or "").lower()
+ ),
+ str(candidate.get("prompt") or "ceiling light").strip(),
+ )
+ category = "_".join(re.findall(r"[a-z0-9]+", semantic_prompt.lower()))
+ if not category:
+ retained_decisions.append(decision)
+ continue
+ anchor = [
+ (candidate_box[0] + candidate_box[2]) / 2.0,
+ (candidate_box[1] + candidate_box[3]) / 2.0,
+ ]
+ additions.append(
+ VLM._minimal_semantic_record(
+ category=category,
+ prompt=semantic_prompt,
+ bbox=candidate_box,
+ anchor=anchor,
+ )
+ )
+ addition_id = f"B_{len(additions):03d}"
+ decision.update(
+ {
+ "decision": "bind_to_addition",
+ "target_object_id": addition_id,
+ "reason": "corroborated repeated top-zone fixture retained",
+ }
+ )
+ retained_decisions.append(decision)
+ promoted.append(
+ {
+ "candidate_id": candidate_id,
+ "target_object_id": addition_id,
+ "category": category,
+ "prompt": semantic_prompt,
+ "evidence_source_count": len(evidence_sources),
+ "maximum_evidence_score": max(evidence_scores, default=0.0),
+ }
+ )
+ actions["additions"] = additions
+ if had_local_candidate_decisions or retained_decisions:
+ actions["local_candidate_decisions"] = retained_decisions
+ audit = {
+ "schema_version": "scenemosaic_repeated_top_fixture_recall",
+ "status": "pass" if promoted else "noop",
+ "repeated_top_fixture_peer_count": repeated_peer_count,
+ "promoted": promoted,
+ "additional_vlm_calls": 0,
+ "additional_sam3_inference_calls": 0,
+ }
+ return actions, audit
+
+
+FAILED_EXISTING_OBJECT_REBIND_THRESHOLDS = {
+ "minimum_candidate_score": 0.60,
+ "minimum_semantic_score": 0.85,
+ "minimum_iou": 0.55,
+ "minimum_candidate_coverage": 0.95,
+ "minimum_existing_coverage": 0.50,
+ "maximum_relative_center_distance": 0.15,
+ "minimum_evidence_source_count": 2,
+ "minimum_evidence_score": 0.60,
+}
+
+
+def rebind_verified_local_candidates_to_failed_existing_objects(
+ *,
+ pass_a_objects: Sequence[Mapping[str, Any]],
+ pass_b_actions: Mapping[str, Any],
+ local_candidates: Sequence[Mapping[str, Any]],
+ sam3_registry: Mapping[str, Any],
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ """Materialize a verified local mask for a failed existing object.
+
+ Compact Pass B distinguishes only new local-candidate additions from
+ rejections. A candidate can therefore be rejected as *not new* while its
+ matching Pass-A object has no SAM3 mask. This deterministic recovery is
+ deliberately narrower than semantic acceptance: it requires a Pass-B
+ semantic pass, a failed SAM3 source, two independent local-evidence
+ sources, and a unique high-agreement geometry match. It never creates an
+ object or turns an unreviewed candidate into a mask.
+ """
+
+ actions = copy.deepcopy(dict(pass_b_actions))
+ thresholds = dict(FAILED_EXISTING_OBJECT_REBIND_THRESHOLDS)
+ pass_a_by_id = {
+ str(item.get("object_id") or ""): dict(item)
+ for item in pass_a_objects
+ if str(item.get("object_id") or "")
+ }
+ registry_by_id = {
+ str(item.get("object_id") or ""): dict(item)
+ for item in sam3_registry.get("objects") or []
+ if isinstance(item, Mapping) and str(item.get("object_id") or "")
+ }
+ candidates_by_id = {
+ str(item.get("candidate_id") or ""): dict(item)
+ for item in local_candidates
+ if str(item.get("candidate_id") or "")
+ }
+ removed_ids = {str(value) for value in actions.get("removals") or []}
+ updated_ids = {
+ str(item.get("pass_a_id") or "")
+ for item in actions.get("updates") or []
+ if isinstance(item, Mapping)
+ }
+
+ gate = actions.get("assigned_evidence_semantic_gate")
+ assertions_by_source: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ if isinstance(gate, Mapping):
+ for raw_assertion in gate.get(
+ "normalized_unmaterialized_repair_assertions"
+ ) or []:
+ if not isinstance(raw_assertion, Mapping):
+ continue
+ assertion = dict(raw_assertion)
+ if (
+ assertion.get("reported_decision") == "pass"
+ and assertion.get("effective_decision") == "uncertain"
+ and assertion.get("rule")
+ == "failed_sam3_evidence_cannot_be_accepted_without_repair"
+ ):
+ source_id = str(assertion.get("source_object_id") or "")
+ if source_id:
+ assertions_by_source[source_id].append(assertion)
+
+ eligible_pairs: list[dict[str, Any]] = []
+ for raw_decision in actions.get("local_candidate_decisions") or []:
+ if not isinstance(raw_decision, Mapping):
+ continue
+ decision = dict(raw_decision)
+ if decision.get("decision") != "reject":
+ continue
+ candidate_id = str(decision.get("candidate_id") or "")
+ candidate = candidates_by_id.get(candidate_id)
+ if candidate is None:
+ continue
+ try:
+ candidate_score = float(candidate.get("score") or 0.0)
+ except (TypeError, ValueError):
+ continue
+ if candidate_score < thresholds["minimum_candidate_score"]:
+ continue
+ evidence = [
+ dict(item)
+ for item in candidate.get("prompt_evidence") or []
+ if isinstance(item, Mapping)
+ ]
+ evidence_sources = {
+ str(item.get("source_namespace") or "")
+ for item in evidence
+ if str(item.get("source_namespace") or "")
+ }
+ try:
+ evidence_scores = [
+ float(item.get("score") or 0.0) for item in evidence
+ ]
+ except (TypeError, ValueError):
+ continue
+ if (
+ len(evidence_sources) < thresholds["minimum_evidence_source_count"]
+ or max(evidence_scores, default=0.0)
+ < thresholds["minimum_evidence_score"]
+ ):
+ continue
+ for source_id, assertions in assertions_by_source.items():
+ source = pass_a_by_id.get(source_id)
+ registry_item = registry_by_id.get(source_id) or {}
+ if (
+ source is None
+ or source_id in removed_ids
+ or source_id in updated_ids
+ or (registry_item.get("sam3") or {}).get("status") != "failed"
+ ):
+ continue
+ metrics = _local_candidate_pair_metrics(candidate, source)
+ if (
+ metrics["semantic_score"] < thresholds["minimum_semantic_score"]
+ or metrics["iou"] < thresholds["minimum_iou"]
+ or metrics["candidate_coverage"]
+ < thresholds["minimum_candidate_coverage"]
+ or metrics["addition_coverage"]
+ < thresholds["minimum_existing_coverage"]
+ or metrics["relative_center_distance"]
+ > thresholds["maximum_relative_center_distance"]
+ ):
+ continue
+ eligible_pairs.append(
+ {
+ "candidate_id": candidate_id,
+ "source_object_id": source_id,
+ "evidence_ids": sorted(
+ str(item.get("evidence_id") or "")
+ for item in assertions
+ if str(item.get("evidence_id") or "")
+ ),
+ "metrics": {
+ key: round(float(value), 6)
+ for key, value in metrics.items()
+ },
+ "evidence_source_count": len(evidence_sources),
+ "maximum_evidence_score": round(
+ max(evidence_scores, default=0.0), 6
+ ),
+ }
+ )
+
+ pairs_by_candidate: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ pairs_by_source: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ for pair in eligible_pairs:
+ pairs_by_candidate[pair["candidate_id"]].append(pair)
+ pairs_by_source[pair["source_object_id"]].append(pair)
+ selected_pairs = [
+ pair
+ for pair in eligible_pairs
+ if len(pairs_by_candidate[pair["candidate_id"]]) == 1
+ and len(pairs_by_source[pair["source_object_id"]]) == 1
+ ]
+ selected_by_candidate = {
+ pair["candidate_id"]: pair for pair in selected_pairs
+ }
+ selected_by_source = {
+ pair["source_object_id"]: pair for pair in selected_pairs
+ }
+
+ rebound: list[dict[str, Any]] = []
+ had_local_candidate_decisions = "local_candidate_decisions" in actions
+ rewritten_decisions: list[dict[str, Any]] = []
+ for raw_decision in actions.get("local_candidate_decisions") or []:
+ decision = copy.deepcopy(dict(raw_decision))
+ pair = selected_by_candidate.get(str(decision.get("candidate_id") or ""))
+ if pair is not None:
+ decision.update(
+ {
+ "decision": "replace_existing",
+ "target_object_id": pair["source_object_id"],
+ "reason": (
+ "one-to-one verified local candidate rebound to a "
+ "failed existing object after Pass-B semantic pass"
+ ),
+ }
+ )
+ rebound.append(pair)
+ rewritten_decisions.append(decision)
+ if had_local_candidate_decisions:
+ actions["local_candidate_decisions"] = rewritten_decisions
+
+ recovered_evidence_ids = {
+ evidence_id
+ for pair in selected_by_source.values()
+ for evidence_id in pair["evidence_ids"]
+ }
+ rewritten_evidence = []
+ for raw_decision in actions.get("assigned_evidence_decisions") or []:
+ decision = copy.deepcopy(dict(raw_decision))
+ if str(decision.get("evidence_id") or "") in recovered_evidence_ids:
+ original_reason = str(decision.get("reason") or "")
+ decision["decision"] = "pass"
+ decision["reason"] = (
+ "one-to-one verified local candidate mask was bound to this "
+ f"failed object. Original reason: {original_reason}"
+ )
+ rewritten_evidence.append(decision)
+ if "assigned_evidence_decisions" in actions:
+ actions["assigned_evidence_decisions"] = rewritten_evidence
+
+ if isinstance(gate, Mapping):
+ rewritten_gate = copy.deepcopy(dict(gate))
+ unresolved = [
+ str(value)
+ for value in rewritten_gate.get("unresolved_evidence_ids") or []
+ if str(value) not in recovered_evidence_ids
+ ]
+ rewritten_gate["unresolved_evidence_ids"] = unresolved
+ rewritten_gate["status"] = "pass" if not unresolved else "blocked"
+ rewritten_gate["local_candidate_rebinding_resolutions"] = rebound
+ actions["assigned_evidence_semantic_gate"] = rewritten_gate
+
+ ambiguous_pairs = [
+ pair
+ for pair in eligible_pairs
+ if pair["candidate_id"] not in selected_by_candidate
+ or pair["source_object_id"] not in selected_by_source
+ ]
+ audit = {
+ "schema_version": (
+ "scenemosaic_failed_existing_object_local_candidate_rebinding"
+ ),
+ "status": "pass" if rebound else "noop",
+ "thresholds": thresholds,
+ "eligible_pair_count": len(eligible_pairs),
+ "ambiguous_pairs": ambiguous_pairs,
+ "rebound": rebound,
+ "additional_vlm_calls": 0,
+ "additional_sam3_inference_calls": 0,
+ "formal_outputs_modified": False,
+ }
+ return actions, audit
+
+
+def compose_local_candidate_binding_actions(
+ pass_b_actions: Mapping[str, Any],
+ local_decisions: Sequence[Mapping[str, Any]],
+) -> dict[str, Any]:
+ """Convert compact rejection output into the binding action shape."""
+
+ binding_actions = copy.deepcopy(dict(pass_b_actions))
+ binding_actions.pop("local_candidate_rejections", None)
+ binding_actions["local_candidate_decisions"] = [
+ copy.deepcopy(dict(item)) for item in local_decisions
+ ]
+ if "local_candidate_rejections" in binding_actions:
+ raise AssertionError("binding actions retain compact rejection protocol")
+ return binding_actions
+
+
+def attach_local_candidate_bindings(
+ fused_objects: Sequence[Mapping[str, Any]],
+ pass_b_actions: Mapping[str, Any],
+ local_candidates: Sequence[Mapping[str, Any]],
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Bind reviewed local masks to fused A/B object identities."""
+
+ objects = [copy.deepcopy(dict(item)) for item in fused_objects]
+ candidates = {str(item["candidate_id"]): dict(item) for item in local_candidates}
+ source_to_object: dict[tuple[str, str], dict[str, Any]] = {}
+ for obj in objects:
+ for source in obj.get("proposal_sources") or []:
+ source_to_object[(str(source.get("pass")), str(source.get("source_id")))] = obj
+
+ hard_errors: list[dict[str, Any]] = []
+ resolved: list[dict[str, Any]] = []
+ decision_counts: dict[str, int] = defaultdict(int)
+ attached_object_count = 0
+ bound_by_object: dict[str, list[tuple[dict[str, Any], dict[str, Any]]]] = defaultdict(list)
+ for decision in pass_b_actions.get("local_candidate_decisions") or []:
+ candidate_id = str(decision.get("candidate_id"))
+ action = str(decision.get("decision"))
+ decision_counts[action] += 1
+ target = decision.get("target_object_id")
+ candidate = candidates.get(candidate_id)
+ if candidate is None:
+ hard_errors.append(
+ {"type": "unknown_local_candidate", "candidate_id": candidate_id}
+ )
+ continue
+ if action == "uncertain":
+ hard_errors.append(
+ {
+ "type": "unresolved_local_candidate",
+ "candidate_id": candidate_id,
+ "reason": decision.get("reason"),
+ }
+ )
+ continue
+ if action in {"reject", "duplicate_of_existing", "part_of_existing"}:
+ resolved.append(dict(decision))
+ continue
+ source_pass = "B" if action == "bind_to_addition" else "A"
+ target_obj = source_to_object.get((source_pass, str(target)))
+ if target_obj is None:
+ hard_errors.append(
+ {
+ "type": "unbound_local_candidate_target",
+ "candidate_id": candidate_id,
+ "decision": action,
+ "target_object_id": target,
+ }
+ )
+ continue
+ bound_by_object[str(target_obj["object_id"])].append((dict(decision), candidate))
+ resolved.append(dict(decision))
+
+ for object_id, records in bound_by_object.items():
+ target_obj = next(obj for obj in objects if str(obj["object_id"]) == object_id)
+ operations = {
+ "union_existing" if decision["decision"] == "merge_extend_existing" else "replace"
+ for decision, _ in records
+ }
+ if len(operations) != 1:
+ hard_errors.append(
+ {
+ "type": "conflicting_local_candidate_operations",
+ "object_id": object_id,
+ "operations": sorted(operations),
+ }
+ )
+ continue
+ binding_candidates = []
+ for decision, candidate in records:
+ candidate_id = str(candidate["candidate_id"])
+ binding_candidates.append(
+ {
+ "candidate_id": candidate_id,
+ "prompt": candidate.get("prompt"),
+ "score": candidate.get("score"),
+ "bbox_norm_xyxy": candidate.get("bbox_norm_xyxy"),
+ "mask_path": candidate.get("mask_path"),
+ "overlay_path": candidate.get("overlay_path"),
+ "decision": decision["decision"],
+ "reason": decision.get("reason"),
+ }
+ )
+ target_obj.setdefault("proposal_sources", []).append(
+ {
+ "pass": "local_candidate",
+ "source_id": candidate_id,
+ "action": decision["decision"],
+ }
+ )
+ target_obj["local_candidate_binding"] = {
+ "operation": next(iter(operations)),
+ "candidates": binding_candidates,
+ }
+ attached_object_count += 1
+
+ return objects, {
+ "schema_version": "scenemosaic_local_candidate_binding_audit",
+ "candidate_count": len(candidates),
+ "decision_count": len(pass_b_actions.get("local_candidate_decisions") or []),
+ "decision_counts": dict(sorted(decision_counts.items())),
+ "bound_object_count": attached_object_count,
+ "binding_candidate_count": sum(
+ len(
+ (item.get("local_candidate_binding") or {}).get("candidates")
+ or []
+ )
+ for item in objects
+ if item.get("local_candidate_binding")
+ ),
+ "resolved_decisions": resolved,
+ "hard_errors": hard_errors,
+ "status": "pass" if not hard_errors else "fail",
+ }
diff --git a/vlm_agent/reconstruction/perception/pipeline/core/inventory_fusion.py b/vlm_agent/reconstruction/perception/pipeline/core/inventory_fusion.py
new file mode 100644
index 0000000..eb41395
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/core/inventory_fusion.py
@@ -0,0 +1,218 @@
+from __future__ import annotations
+
+import copy
+from collections import defaultdict
+from typing import Any, Mapping, Sequence
+
+from pipeline.core.pass_b_policy import validate_b_actions
+from segmentation import runtime as SAM3_SESSION
+
+def _prompt_groups(
+ objects: Sequence[Mapping[str, Any]],
+) -> dict[str, list[dict[str, Any]]]:
+ """Mirror the production grouping without importing the GPU runtime."""
+
+ groups: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ for raw_object in objects:
+ obj = dict(raw_object)
+ prompt = str(obj.get("sam3_prompt") or "").strip()
+ if not prompt:
+ prompt = str(obj.get("category") or "").strip().replace("_", " ")
+ groups[prompt or "object"].append(obj)
+ return dict(groups)
+
+
+def _object_prompt_map(
+ objects: Sequence[Mapping[str, Any]],
+) -> dict[str, str]:
+ result: dict[str, str] = {}
+ for prompt, grouped_objects in _prompt_groups(objects).items():
+ for obj in grouped_objects:
+ result[str(obj["object_id"])] = prompt
+ return result
+
+
+def validate_prompt_group_parity(
+ objects: Sequence[Mapping[str, Any]],
+) -> None:
+ """Ensure orchestration and the reusable SAM3 session group identically."""
+
+ local = {
+ prompt: [str(obj["object_id"]) for obj in grouped]
+ for prompt, grouped in _prompt_groups(objects).items()
+ }
+ production = {
+ prompt: [str(obj["object_id"]) for obj in grouped]
+ for prompt, grouped in SAM3_SESSION.prompt_groups(objects).items()
+ }
+ if local != production:
+ raise RuntimeError(
+ f"SAM3 prompt grouping parity failure: local={local}, production={production}"
+ )
+
+
+def compute_affected_prompt_audit(
+ pass_a_objects: Sequence[Mapping[str, Any]],
+ pass_b_actions: Mapping[str, Any],
+ fused_objects: Sequence[Mapping[str, Any]],
+) -> dict[str, Any]:
+ """Determine complete old/new prompt groups invalidated by Pass B."""
+
+ validate_b_actions(pass_a_objects, pass_b_actions)
+ old_prompt_by_id = _object_prompt_map(pass_a_objects)
+ new_prompt_by_id = _object_prompt_map(fused_objects)
+ final_by_a: dict[str, Mapping[str, Any]] = {}
+ additions: list[Mapping[str, Any]] = []
+ for obj in fused_objects:
+ sources = list(obj.get("proposal_sources") or [])
+ for source in sources:
+ if source.get("pass") == "A":
+ final_by_a[str(source["source_id"])] = obj
+ if (
+ source.get("pass") == "B"
+ and source.get("action") == "added"
+ ):
+ additions.append(obj)
+
+ updated_ids = {
+ str(item["pass_a_id"]) for item in pass_b_actions["updates"]
+ }
+ removed_ids = {str(item) for item in pass_b_actions["removals"]}
+ old_prompts: set[str] = set()
+ new_prompts: set[str] = set()
+ reasons: list[dict[str, Any]] = []
+
+ for old_obj in pass_a_objects:
+ object_id = str(old_obj["object_id"])
+ old_prompt = old_prompt_by_id[object_id]
+ if object_id in removed_ids:
+ old_prompts.add(old_prompt)
+ reasons.append(
+ {
+ "action": "removed",
+ "source_id": object_id,
+ "old_prompt": old_prompt,
+ }
+ )
+ continue
+ final_obj = final_by_a.get(object_id)
+ if final_obj is None:
+ raise ValueError(f"surviving A object lacks final identity: {object_id}")
+ final_id = str(final_obj["object_id"])
+ new_prompt = new_prompt_by_id[final_id]
+ changed = object_id in updated_ids or old_prompt != new_prompt
+ if changed:
+ old_prompts.add(old_prompt)
+ new_prompts.add(new_prompt)
+ reasons.append(
+ {
+ "action": (
+ "updated" if object_id in updated_ids else "regrouped"
+ ),
+ "source_id": object_id,
+ "proposal_id": final_id,
+ "old_prompt": old_prompt,
+ "new_prompt": new_prompt,
+ }
+ )
+
+ for final_obj in additions:
+ final_id = str(final_obj["object_id"])
+ new_prompt = new_prompt_by_id[final_id]
+ binding = final_obj.get("local_candidate_binding") or {}
+ if binding.get("operation") == "replace":
+ reasons.append(
+ {
+ "action": "added_with_reviewed_local_mask",
+ "proposal_id": final_id,
+ "new_prompt": new_prompt,
+ "candidate_ids": [
+ item.get("candidate_id")
+ for item in binding.get("candidates") or []
+ ],
+ }
+ )
+ continue
+ new_prompts.add(new_prompt)
+ sources = [
+ source
+ for source in final_obj.get("proposal_sources") or []
+ if source.get("pass") == "B"
+ and source.get("action") == "added"
+ ]
+ reasons.append(
+ {
+ "action": "added",
+ "source_id": sources[0]["source_id"] if sources else None,
+ "proposal_id": final_id,
+ "new_prompt": new_prompt,
+ }
+ )
+
+ # If an old group was invalidated and surviving members retain the same
+ # prompt, that final prompt must also be recomputed after a removal.
+ for object_id, final_obj in final_by_a.items():
+ old_prompt = old_prompt_by_id[object_id]
+ if old_prompt in old_prompts:
+ new_prompts.add(new_prompt_by_id[str(final_obj["object_id"])])
+
+ return {
+ "old_prompts": sorted(old_prompts),
+ "new_prompts": sorted(new_prompts),
+ "reasons": reasons,
+ }
+
+
+def make_final_identity(
+ fused_objects: Sequence[Mapping[str, Any]],
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Assign continuous final IDs after any deterministic removal stage.
+
+ Intermediate removals (for example a rejected carrier-boundary fragment)
+ naturally create gaps. The final published inventory must nevertheless use
+ continuous IDs, so retain the existing visual order and renumber only at
+ this identity boundary. The manifest records every old-to-new mapping for
+ auditability; duplicate source identities remain a hard error.
+ """
+ objects = [copy.deepcopy(dict(item)) for item in fused_objects]
+ prior_ids = [str(item.get("object_id") or "") for item in objects]
+ if len(prior_ids) != len(set(prior_ids)):
+ raise ValueError(f"duplicate intermediate proposal IDs: {prior_ids}")
+ renumbered = []
+ for index, obj in enumerate(objects, start=1):
+ previous_id = str(obj.get("object_id") or "")
+ if not previous_id:
+ raise ValueError("proposal is missing an intermediate object_id")
+ proposal_id = f"proposal_{index:03d}"
+ obj["object_id"] = proposal_id
+ if previous_id != proposal_id:
+ renumbered.append({
+ "previous_proposal_id": previous_id,
+ "proposal_id": proposal_id,
+ })
+ records = []
+ source_to_proposal: dict[str, str] = {}
+ for obj, previous_id in zip(objects, prior_ids):
+ sources = list(obj.get("proposal_sources") or [])
+ if not sources:
+ raise ValueError(f"proposal has no source identity: {obj['object_id']}")
+ for source in sources:
+ source_key = f"{source.get('pass')}:{source.get('source_id')}"
+ if source_key in source_to_proposal:
+ raise ValueError(f"source identity reused: {source_key}")
+ source_to_proposal[source_key] = obj["object_id"]
+ record = {
+ "proposal_id": obj["object_id"],
+ "category": obj.get("category"),
+ "sources": sources,
+ }
+ if previous_id != obj["object_id"]:
+ record["previous_proposal_id"] = previous_id
+ records.append(record)
+ return objects, {
+ "schema_version": "scenemosaic_attempt_identity",
+ "object_id_format": "proposal_{index:03d}",
+ "records": records,
+ "source_to_proposal": source_to_proposal,
+ "renumbered": renumbered,
+ }
diff --git a/vlm_agent/reconstruction/perception/pipeline/core/pass_b_policy.py b/vlm_agent/reconstruction/perception/pipeline/core/pass_b_policy.py
new file mode 100644
index 0000000..9e9a8a9
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/core/pass_b_policy.py
@@ -0,0 +1,1014 @@
+from __future__ import annotations
+
+import copy
+import math
+import re
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+def validate_b_actions(
+ pass_a_objects: Sequence[Mapping[str, Any]],
+ actions: Mapping[str, Any],
+ *,
+ expected_local_candidate_ids: Sequence[str] | None = None,
+) -> None:
+ known_ids = {str(item["object_id"]) for item in pass_a_objects}
+ required = {"additions", "updates", "removals"}
+ allowed = required | {
+ "local_candidate_decisions",
+ "local_candidate_rejections",
+ "assigned_evidence_decisions",
+ "assigned_evidence_semantic_gate",
+ }
+ if not required.issubset(actions) or not set(actions).issubset(allowed):
+ raise ValueError(
+ f"Pass B action keys must contain {sorted(required)} and only optional "
+ "candidate or assigned-evidence audit fields"
+ )
+ update_ids = [str(item["pass_a_id"]) for item in actions["updates"]]
+ removal_ids = [str(item) for item in actions["removals"]]
+ if len(update_ids) != len(set(update_ids)):
+ raise ValueError("Pass B updates one Pass-A ID more than once")
+ if len(removal_ids) != len(set(removal_ids)):
+ raise ValueError("Pass B removes one Pass-A ID more than once")
+ unknown = (set(update_ids) | set(removal_ids)) - known_ids
+ if unknown:
+ raise ValueError(f"Pass B references unknown Pass-A IDs: {sorted(unknown)}")
+ conflict = set(update_ids) & set(removal_ids)
+ if conflict:
+ raise ValueError(
+ f"Pass B both updates and removes IDs: {sorted(conflict)}"
+ )
+ if (
+ "local_candidate_decisions" in actions
+ and "local_candidate_rejections" in actions
+ ):
+ raise ValueError("Pass B candidate decision protocols conflict")
+ if "local_candidate_rejections" in actions:
+ raw_rejections = actions["local_candidate_rejections"]
+ if not isinstance(raw_rejections, list):
+ raise ValueError("Pass B local candidate rejections must be a list")
+ rejection_ids = [str(value) for value in raw_rejections]
+ if not all(re.fullmatch(r"LC_\d{3}", value) for value in rejection_ids):
+ raise ValueError("invalid Pass B local candidate rejection ID")
+ if len(rejection_ids) != len(set(rejection_ids)):
+ raise ValueError("duplicate Pass B local candidate rejection ID")
+
+ if expected_local_candidate_ids is not None:
+ if isinstance(expected_local_candidate_ids, (str, bytes)):
+ raise ValueError("expected local candidate IDs must be a sequence")
+ expected_ids = [str(value) for value in expected_local_candidate_ids]
+ if len(expected_ids) != len(set(expected_ids)) or not all(
+ re.fullmatch(r"LC_\d{3}", value) for value in expected_ids
+ ):
+ raise ValueError(
+ "expected local candidate IDs must be unique LC_NNN values"
+ )
+ protocol_keys = {
+ key
+ for key in ("local_candidate_decisions", "local_candidate_rejections")
+ if key in actions
+ }
+ if not expected_ids:
+ if protocol_keys:
+ raise ValueError(
+ "Pass B returned a local candidate protocol without a panel"
+ )
+ elif len(protocol_keys) != 1:
+ raise ValueError(
+ "Pass B must use exactly one local candidate protocol for a panel"
+ )
+ elif "local_candidate_rejections" in protocol_keys:
+ unknown_rejections = sorted(
+ set(rejection_ids) - set(expected_ids)
+ )
+ if unknown_rejections:
+ raise ValueError(
+ "Pass B rejects unknown local candidate IDs: "
+ f"{unknown_rejections}"
+ )
+ else:
+ decisions = actions["local_candidate_decisions"]
+ if not isinstance(decisions, list):
+ raise ValueError("Pass B local candidate decisions must be a list")
+ decision_ids = [
+ str(item.get("candidate_id") or "")
+ for item in decisions
+ if isinstance(item, Mapping)
+ ]
+ if len(decision_ids) != len(decisions):
+ raise ValueError("invalid Pass B local candidate decision")
+ if len(decision_ids) != len(set(decision_ids)):
+ raise ValueError("duplicate Pass B local candidate decision ID")
+ if set(decision_ids) != set(expected_ids):
+ raise ValueError(
+ "Pass B local candidate decisions do not cover the panel"
+ )
+
+ evidence_keys = {
+ key
+ for key in (
+ "assigned_evidence_decisions",
+ "assigned_evidence_semantic_gate",
+ )
+ if key in actions
+ }
+ if evidence_keys and len(evidence_keys) != 2:
+ raise ValueError(
+ "assigned evidence decisions and semantic gate must travel together"
+ )
+ if evidence_keys:
+ decisions = actions["assigned_evidence_decisions"]
+ gate = actions["assigned_evidence_semantic_gate"]
+ if not isinstance(decisions, list) or not isinstance(gate, Mapping):
+ raise ValueError("invalid assigned evidence semantic gate")
+ evidence_ids = []
+ for decision in decisions:
+ if not isinstance(decision, Mapping):
+ raise ValueError("invalid assigned evidence decision")
+ evidence_id = str(decision.get("evidence_id") or "")
+ owner_id = str(decision.get("source_object_id") or "")
+ if not re.fullmatch(r"E_\d{3}", evidence_id):
+ raise ValueError("invalid assigned evidence decision ID")
+ if owner_id not in known_ids:
+ raise ValueError(
+ f"assigned evidence references unknown object: {owner_id}"
+ )
+ evidence_ids.append(evidence_id)
+ if len(evidence_ids) != len(set(evidence_ids)):
+ raise ValueError("duplicate assigned evidence decision ID")
+ if int(gate.get("decision_count", -1)) != len(decisions):
+ raise ValueError("assigned evidence gate count mismatch")
+ if not bool(gate.get("complete_partition")):
+ raise ValueError("assigned evidence gate partition is incomplete")
+
+
+
+
+
+def apply_post_pass_b_policy_guard(
+ pass_a_objects: Sequence[Mapping[str, Any]],
+ actions: Mapping[str, Any],
+ sam3_registry: Mapping[str, Any] | None = None,
+ *,
+ content_hosts: Sequence[Mapping[str, Any]] | None = None,
+) -> dict[str, Any]:
+ "Suppress bounded Pass-B semantic and geometry overcorrections."
+
+ effective = copy.deepcopy(dict(actions))
+
+ def tokens(item: Mapping[str, Any]) -> set[str]:
+ text = " ".join(
+ str(item.get(key) or "")
+ for key in (
+ "category",
+ "sam3_prompt",
+ "instance_name",
+ "description",
+ )
+ ).lower()
+ return set(re.findall(r"[a-z0-9]+", text))
+
+ def box(item: Mapping[str, Any]) -> list[float] | None:
+ try:
+ value = [
+ float(component)
+ for component in item.get("bbox_2d_norm_xyxy")
+ ]
+ except (TypeError, ValueError):
+ return None
+ if (
+ len(value) != 4
+ or not all(math.isfinite(component) for component in value)
+ or not (0.0 <= value[0] < value[2] <= 1.0)
+ or not (0.0 <= value[1] < value[3] <= 1.0)
+ ):
+ return None
+ return value
+
+ def center_x(value: Sequence[float]) -> float:
+ return (float(value[0]) + float(value[2])) / 2.0
+
+ def is_bed(item: Mapping[str, Any]) -> bool:
+ category_tokens = set(
+ re.findall(
+ r"[a-z0-9]+",
+ str(item.get("category") or "").lower().replace("_", " "),
+ )
+ )
+ return "bed" in category_tokens and not bool(
+ category_tokens & {"bench", "bedside", "headboard"}
+ )
+
+ def is_complete_potted_plant(item: Mapping[str, Any]) -> bool:
+ """Identify an inventory unit whose pot and foliage form one object."""
+
+ return {"potted", "plant"} <= tokens(item)
+
+ def is_container_only_botanical_downgrade(
+ source: Mapping[str, Any], replacement: Mapping[str, Any]
+ ) -> bool:
+ """Reject a semantic split that turns one verified plant into its pot."""
+
+ replacement_tokens = tokens(replacement)
+ return (
+ is_complete_potted_plant(source)
+ and "plant" not in replacement_tokens
+ and bool(replacement_tokens & {"pot", "planter", "vase"})
+ )
+
+ def center_y(value: Sequence[float]) -> float:
+ return (float(value[1]) + float(value[3])) / 2.0
+
+ def normalized_box(value: Any) -> list[float] | None:
+ try:
+ candidate = [float(component) for component in value]
+ except (TypeError, ValueError):
+ return None
+ if (
+ len(candidate) != 4
+ or not all(math.isfinite(component) for component in candidate)
+ or not (0.0 <= candidate[0] < candidate[2] <= 1.0)
+ or not (0.0 <= candidate[1] < candidate[3] <= 1.0)
+ ):
+ return None
+ return candidate
+
+ def bbox_iou(left: Sequence[float], right: Sequence[float]) -> float:
+ x1 = max(float(left[0]), float(right[0]))
+ y1 = max(float(left[1]), float(right[1]))
+ x2 = min(float(left[2]), float(right[2]))
+ y2 = min(float(left[3]), float(right[3]))
+ intersection = max(0.0, x2 - x1) * max(0.0, y2 - y1)
+ left_area = max(0.0, float(left[2]) - float(left[0])) * max(
+ 0.0, float(left[3]) - float(left[1])
+ )
+ right_area = max(0.0, float(right[2]) - float(right[0])) * max(
+ 0.0, float(right[3]) - float(right[1])
+ )
+ union = left_area + right_area - intersection
+ return intersection / union if union else 0.0
+
+ # H panels are attention context only. A returned update may legitimately
+ # describe an A owner, but it must never replace that owner with the H crop
+ # itself. This geometry check is independent of host category and remains
+ # effective even if the VLM ignores the textual H contract.
+ attention_crop_by_owner: dict[str, list[float]] = {}
+ for raw_host in content_hosts or []:
+ if not isinstance(raw_host, Mapping):
+ continue
+ owner_id = str(raw_host.get("source_object_id") or "")
+ crop = normalized_box(raw_host.get("crop_norm_xyxy"))
+ if owner_id and crop is not None:
+ attention_crop_by_owner[owner_id] = crop
+
+ live = [dict(item) for item in pass_a_objects]
+ live_by_id = {str(item.get("object_id") or ""): item for item in live}
+ registry_by_id = {
+ str(item.get("object_id") or ""): item
+ for item in (sam3_registry or {}).get("objects") or []
+ if isinstance(item, Mapping)
+ }
+
+ def has_successful_sam3_mask(object_id: str) -> bool:
+ registry_item = registry_by_id.get(object_id, {})
+ return str((registry_item.get("sam3") or {}).get("status") or "") == "success"
+
+ def preserves_failed_owner_semantics(
+ source_item: Mapping[str, Any], replacement: Mapping[str, Any]
+ ) -> bool:
+ """Allow failed-mask updates only when their object class remains local."""
+ ignored = {
+ "unknown",
+ "object",
+ "item",
+ "thing",
+ "furniture",
+ "appliance",
+ "fixture",
+ "device",
+ "decor",
+ "small",
+ }
+ source_terms = {
+ token
+ for token in re.findall(
+ r"[a-z0-9]+", str(source_item.get("category") or "").lower()
+ )
+ if token not in ignored
+ }
+ replacement_terms = {
+ token
+ for token in re.findall(
+ r"[a-z0-9]+", str(replacement.get("category") or "").lower()
+ )
+ if token not in ignored
+ }
+ # A generic/unknown label gives no reliable semantic signal. Preserve
+ # the model action and let its geometry/evidence be audited elsewhere.
+ return (
+ not source_terms
+ or not replacement_terms
+ or bool(source_terms & replacement_terms)
+ )
+
+ sinks = [
+ item
+ for item in live
+ if "sink" in tokens(item)
+ and bool({"counter", "assembly", "vanity", "cabinet"} & tokens(item))
+ and box(item) is not None
+ ]
+ counters = [
+ item
+ for item in live
+ if "counter" in tokens(item)
+ and "sink" not in tokens(item)
+ and "island" not in tokens(item)
+ and box(item) is not None
+ ]
+ separators = [
+ item
+ for item in live
+ if bool({"range", "stove"} & tokens(item))
+ and box(item) is not None
+ ]
+ partitions = []
+ protected_roles: dict[str, str] = {}
+ # A successful functional ROI refinement has already probed this complete
+ # vanity (including fixed faucets) with the same local SAM3 session. A
+ # Pass-B wording-only update must not trigger a new prompt-group inference
+ # and silently discard that validated fixture extent.
+ for item in live:
+ item_tokens = tokens(item)
+ if (
+ "vanity" in item_tokens
+ and bool(item_tokens & {"sink", "sinks", "basin", "basins"})
+ and bool(item_tokens & {"faucet", "faucets"})
+ ):
+ protected_roles[str(item.get("object_id") or "")] = (
+ "complete_sink_vanity_assembly"
+ )
+ for sink in sinks:
+ sink_box = box(sink)
+ assert sink_box is not None
+ for counter in counters:
+ counter_box = box(counter)
+ assert counter_box is not None
+ for separator in separators:
+ separator_box = box(separator)
+ assert separator_box is not None
+ sx = center_x(sink_box)
+ cx = center_x(counter_box)
+ rx = center_x(separator_box)
+ if sx < rx < cx:
+ sink_side = "left"
+ sink_gap = max(0.0, separator_box[0] - sink_box[2])
+ counter_gap = max(0.0, counter_box[0] - separator_box[2])
+ elif cx < rx < sx:
+ sink_side = "right"
+ sink_gap = max(0.0, sink_box[0] - separator_box[2])
+ counter_gap = max(0.0, separator_box[0] - counter_box[2])
+ else:
+ continue
+ if max(sink_gap, counter_gap) > 0.12:
+ continue
+ if max(
+ abs(center_y(sink_box) - center_y(separator_box)),
+ abs(center_y(counter_box) - center_y(separator_box)),
+ ) > 0.22:
+ continue
+ sink_id = str(sink.get("object_id") or "")
+ counter_id = str(counter.get("object_id") or "")
+ separator_id = str(separator.get("object_id") or "")
+ protected_roles[sink_id] = "sink_counter_assembly"
+ protected_roles[counter_id] = "adjacent_counter_cabinet"
+ partitions.append(
+ {
+ "sink_object_id": sink_id,
+ "counter_object_id": counter_id,
+ "separator_object_id": separator_id,
+ "sink_side": sink_side,
+ }
+ )
+
+ kept_updates = []
+ suppressed_updates = []
+ failed_mask_semantic_rejections = []
+ for update in effective.get("updates") or []:
+ source_id = str(update.get("pass_a_id") or "")
+ source_item = live_by_id.get(source_id, {})
+ source_tokens = tokens(source_item)
+ replacement = update.get("replacement") or {}
+ replacement_tokens = (
+ tokens(replacement) if isinstance(replacement, Mapping) else set()
+ )
+ source_box = box(source_item)
+ replacement_box = (
+ box(replacement) if isinstance(replacement, Mapping) else None
+ )
+ replacement_is_sink = "sink" in replacement_tokens
+ role = protected_roles.get(source_id)
+ reason = None
+ if (
+ isinstance(replacement, Mapping)
+ and not has_successful_sam3_mask(source_id)
+ and not preserves_failed_owner_semantics(source_item, replacement)
+ ):
+ failed_mask_semantic_rejections.append(
+ {
+ "pass_a_id": source_id,
+ "source_category": str(source_item.get("category") or ""),
+ "replacement_category": str(replacement.get("category") or ""),
+ "reason": "failed_mask_update_changes_object_semantics",
+ }
+ )
+ continue
+ attention_crop = attention_crop_by_owner.get(source_id)
+ if (
+ attention_crop is not None
+ and replacement_box is not None
+ and bbox_iou(attention_crop, replacement_box) >= 0.98
+ ):
+ reason = "content_attention_crop_cannot_replace_owner_bbox"
+ elif (
+ is_container_only_botanical_downgrade(source_item, replacement)
+ and has_successful_sam3_mask(source_id)
+ ):
+ reason = (
+ "verified_potted_plant_assembly_cannot_be_relabelled_as_container"
+ )
+ elif (
+ is_bed(source_item)
+ and isinstance(replacement, Mapping)
+ and is_bed(replacement)
+ and source_box is not None
+ and replacement_box is not None
+ and replacement_box[1] > source_box[1] + 0.015
+ ):
+ reason = "complete_bed_assembly_cannot_drop_headboard_or_bedding"
+ elif role == "adjacent_counter_cabinet" and replacement_is_sink:
+ reason = "counter_across_range_cannot_become_sink_assembly"
+ elif (
+ role == "sink_counter_assembly"
+ and "counter" in replacement_tokens
+ and not replacement_is_sink
+ ):
+ reason = "sink_assembly_cannot_become_plain_counter"
+ elif (
+ bool(source_tokens & {"pillow", "cushion"})
+ and bool(replacement_tokens & {"bed", "bunk", "loft"})
+ ):
+ reason = "independent_pillow_cannot_become_bed"
+ elif (
+ role == "complete_sink_vanity_assembly"
+ and "vanity" in replacement_tokens
+ and bool(replacement_tokens & {"sink", "basin"})
+ ):
+ reason = "roi_refined_sink_vanity_preserves_fixed_fixture_mask"
+ elif (
+ role in {"sink_counter_assembly", "adjacent_counter_cabinet"}
+ and str(replacement.get("category") or "")
+ == str(live_by_id.get(source_id, {}).get("category") or "")
+ and str(replacement.get("sam3_prompt") or "")
+ == str(live_by_id.get(source_id, {}).get("sam3_prompt") or "")
+ ):
+ reason = "roi_refined_functional_role_preserves_assigned_mask"
+ if reason is None:
+ kept_updates.append(update)
+ else:
+ suppressed_updates.append(
+ {
+ "pass_a_id": source_id,
+ "reason": reason,
+ "replacement_category": str(
+ replacement.get("category") or ""
+ ),
+ "attention_crop_norm_xyxy": attention_crop,
+ }
+ )
+ effective["updates"] = kept_updates
+ failed_mask_semantic_rejection_ids = {
+ str(item["pass_a_id"]) for item in failed_mask_semantic_rejections
+ }
+ effective["removals"] = list(
+ dict.fromkeys(
+ [
+ *[str(item) for item in effective.get("removals") or []],
+ *sorted(failed_mask_semantic_rejection_ids),
+ ]
+ )
+ )
+
+ suppressed_update_ids = {
+ str(item["pass_a_id"]) for item in suppressed_updates
+ }
+ rewritten_evidence_decisions = []
+ for decision in effective.get("assigned_evidence_decisions") or []:
+ source_id = str(decision.get("source_object_id") or "")
+ if (
+ source_id in failed_mask_semantic_rejection_ids
+ and decision.get("decision") == "update_existing"
+ ):
+ previous_reason = str(decision.get("reason") or "")
+ decision["decision"] = "remove_existing"
+ decision["reason"] = (
+ "failed SAM3 owner changed to a semantically different object"
+ )
+ rewritten_evidence_decisions.append(
+ {
+ "source_object_id": source_id,
+ "previous_decision": "update_existing",
+ "effective_decision": "remove_existing",
+ "previous_reason": previous_reason,
+ "effective_reason": decision["reason"],
+ }
+ )
+ elif (
+ source_id in suppressed_update_ids
+ and decision.get("decision") == "update_existing"
+ ):
+ previous_reason = str(decision.get("reason") or "")
+ decision["decision"] = "pass"
+ decision["reason"] = (
+ "local object-unit policy retained the verified source role"
+ )
+ rewritten_evidence_decisions.append(
+ {
+ "source_object_id": source_id,
+ "previous_decision": "update_existing",
+ "effective_decision": "pass",
+ "previous_reason": previous_reason,
+ "effective_reason": decision["reason"],
+ }
+ )
+
+ raw_removal_ids = [str(value) for value in effective.get("removals") or []]
+ raw_removal_id_set = set(raw_removal_ids)
+ kept_removals = []
+ suppressed_removals = []
+ registry_by_id = {
+ str(item.get("object_id") or ""): item
+ for item in (sam3_registry or {}).get("objects") or []
+ if isinstance(item, Mapping)
+ }
+
+ def category_key(item: Mapping[str, Any]) -> str:
+ category = str(item.get("category") or "").lower()
+ category_tokens = set(re.findall(r"[a-z0-9]+", category))
+ if category_tokens & {"pillow", "cushion"}:
+ return "pillow"
+ return "_".join(sorted(category_tokens))
+
+ def registry_mask(object_id: str):
+ registry_item = registry_by_id.get(object_id, {})
+ sam3 = registry_item.get("sam3") or {}
+ if sam3.get("status") != "success":
+ return None, "missing_successful_sam3_mask"
+ path = Path(str(sam3.get("mask_path") or "")).expanduser().resolve()
+ if not path.is_file():
+ return None, "sam3_mask_path_missing"
+ try:
+ import numpy as np
+ from PIL import Image
+
+ mask = np.array(Image.open(path).convert("L")) > 0
+ except Exception as exc:
+ return None, f"sam3_mask_load_failed:{type(exc).__name__}"
+ if mask.ndim != 2 or not mask.any():
+ return None, "sam3_mask_invalid_or_empty"
+ return mask, None
+
+ decisions_by_source = {
+ str(item.get("source_object_id") or ""): item
+ for item in effective.get("assigned_evidence_decisions") or []
+ if isinstance(item, Mapping)
+ }
+ has_complete_bed = any(is_bed(item) for item in live)
+ for source_id in raw_removal_ids:
+ decision = decisions_by_source.get(source_id)
+ decision_reason = str((decision or {}).get("reason") or "").lower()
+ is_duplicate_claim = bool(
+ re.search(r"\bduplicate\b|same\s+visible\s+.+\s+region", decision_reason)
+ )
+ source_item = live_by_id.get(source_id, {})
+ source_category = category_key(source_item)
+ is_bed_assembly_exclusion_claim = bool(
+ source_category == "pillow"
+ and has_complete_bed
+ and re.search(r"\bexclud(?:e|ed|ing)\b", decision_reason)
+ and re.search(r"\b(?:bed|bedding|assembly)\b", decision_reason)
+ )
+ if is_bed_assembly_exclusion_claim and not is_duplicate_claim:
+ source_mask, source_mask_error = registry_mask(source_id)
+ if source_mask is not None:
+ suppressed_removals.append(
+ {
+ "source_object_id": source_id,
+ "reason": (
+ "bed_mask_exclusion_preserves_independent_pillow"
+ ),
+ "source_mask_error": source_mask_error,
+ "retained_area_pixels": int(source_mask.sum()),
+ }
+ )
+ if (
+ decision is not None
+ and decision.get("decision") == "remove_existing"
+ ):
+ previous_reason = str(decision.get("reason") or "")
+ decision["decision"] = "pass"
+ decision["reason"] = (
+ "local assembly policy keeps a pillow excluded from "
+ "the bed mask as an independent inventory instance"
+ )
+ rewritten_evidence_decisions.append(
+ {
+ "source_object_id": source_id,
+ "previous_decision": "remove_existing",
+ "effective_decision": "pass",
+ "previous_reason": previous_reason,
+ "effective_reason": decision["reason"],
+ }
+ )
+ continue
+ if not is_duplicate_claim:
+ kept_removals.append(source_id)
+ continue
+ source_mask, source_mask_error = registry_mask(source_id)
+ best_geometry = None
+ if source_mask is not None and source_category:
+ source_area = int(source_mask.sum())
+ for candidate_id, candidate_item in live_by_id.items():
+ if (
+ candidate_id == source_id
+ or candidate_id in raw_removal_id_set
+ or category_key(candidate_item) != source_category
+ ):
+ continue
+ candidate_mask, _ = registry_mask(candidate_id)
+ if candidate_mask is None or candidate_mask.shape != source_mask.shape:
+ continue
+ intersection = int((source_mask & candidate_mask).sum())
+ if intersection < 30:
+ continue
+ candidate_area = int(candidate_mask.sum())
+ union = source_area + candidate_area - intersection
+ geometry = {
+ "candidate_object_id": candidate_id,
+ "intersection_pixels": intersection,
+ "mask_iou": round(intersection / max(union, 1), 6),
+ "source_containment": round(
+ intersection / max(source_area, 1), 6
+ ),
+ "candidate_containment": round(
+ intersection / max(candidate_area, 1), 6
+ ),
+ }
+ if best_geometry is None or geometry["mask_iou"] > best_geometry["mask_iou"]:
+ best_geometry = geometry
+ duplicate_proven = bool(
+ best_geometry
+ and (
+ best_geometry["mask_iou"] >= 0.80
+ or best_geometry["source_containment"] >= 0.95
+ or best_geometry["candidate_containment"] >= 0.95
+ )
+ )
+ if duplicate_proven:
+ kept_removals.append(source_id)
+ continue
+ suppressed_removals.append(
+ {
+ "source_object_id": source_id,
+ "reason": "duplicate_removal_requires_same_category_mask_geometry",
+ "source_mask_error": source_mask_error,
+ "best_same_category_geometry": best_geometry,
+ }
+ )
+ if decision is not None and decision.get("decision") == "remove_existing":
+ previous_reason = str(decision.get("reason") or "")
+ decision["decision"] = "pass"
+ decision["reason"] = (
+ "local geometry policy retained a distinct same-category instance"
+ )
+ rewritten_evidence_decisions.append(
+ {
+ "source_object_id": source_id,
+ "previous_decision": "remove_existing",
+ "effective_decision": "pass",
+ "previous_reason": previous_reason,
+ "effective_reason": decision["reason"],
+ }
+ )
+ effective["removals"] = kept_removals
+
+ hoods = [
+ item
+ for item in live
+ if "hood" in tokens(item) and box(item) is not None
+ ]
+ shelves = [
+ item
+ for item in live
+ if "shelf" in tokens(item) and box(item) is not None
+ ]
+ beds = [item for item in live if is_bed(item) and box(item) is not None]
+ updated_ids = {
+ str(item.get("pass_a_id") or "")
+ for item in effective.get("updates") or []
+ if isinstance(item, Mapping)
+ }
+ verified_potted_plant_assemblies = [
+ item
+ for item in live
+ if (
+ is_complete_potted_plant(item)
+ and str(item.get("object_id") or "") not in updated_ids
+ and str(item.get("object_id") or "") not in kept_removals
+ and has_successful_sam3_mask(str(item.get("object_id") or ""))
+ and box(item) is not None
+ )
+ ]
+ kept_additions = []
+ suppressed_additions = []
+ addition_id_remap = {}
+ for index, addition in enumerate(effective.get("additions") or [], start=1):
+ addition_tokens = tokens(addition)
+ addition_box = box(addition)
+ matched_hood = None
+ matched_shelf = None
+ matched_bed = None
+ matched_potted_plant = None
+ if addition_box is not None and "headboard" in addition_tokens:
+ addition_area = (
+ (addition_box[2] - addition_box[0])
+ * (addition_box[3] - addition_box[1])
+ )
+ for bed in beds:
+ bed_box = box(bed)
+ assert bed_box is not None
+ intersection = (
+ max(
+ 0.0,
+ min(addition_box[2], bed_box[2])
+ - max(addition_box[0], bed_box[0]),
+ )
+ * max(
+ 0.0,
+ min(addition_box[3], bed_box[3])
+ - max(addition_box[1], bed_box[1]),
+ )
+ )
+ if intersection / max(addition_area, 1e-9) >= 0.85:
+ matched_bed = bed
+ break
+ if (
+ addition_box is not None
+ and bool({"light", "lighting", "downlight"} & addition_tokens)
+ and bool({"ceiling", "recessed", "downlight"} & addition_tokens)
+ ):
+ addition_center_x = center_x(addition_box)
+ for hood in hoods:
+ hood_box = box(hood)
+ assert hood_box is not None
+ if (
+ hood_box[0] - 0.01
+ <= addition_center_x
+ <= hood_box[2] + 0.01
+ and hood_box[1] <= addition_box[1] <= hood_box[3] + 0.04
+ ):
+ matched_hood = hood
+ break
+ if (
+ matched_hood is None
+ and addition_box is not None
+ and bool({"light", "lighting"} & addition_tokens)
+ and bool({"under", "undercabinet", "linear", "strip"} & addition_tokens)
+ ):
+ addition_width = addition_box[2] - addition_box[0]
+ addition_height = addition_box[3] - addition_box[1]
+ for shelf in shelves:
+ shelf_box = box(shelf)
+ assert shelf_box is not None
+ shelf_width = shelf_box[2] - shelf_box[0]
+ horizontal_overlap = max(
+ 0.0,
+ min(addition_box[2], shelf_box[2])
+ - max(addition_box[0], shelf_box[0]),
+ )
+ overlap_ratio = horizontal_overlap / max(
+ min(addition_width, shelf_width), 1e-9
+ )
+ if (
+ addition_width >= 0.15
+ and addition_height <= 0.08
+ and addition_width / max(addition_height, 1e-9) >= 4.0
+ and overlap_ratio >= 0.90
+ and shelf_box[1] - 0.01 <= addition_box[1] <= shelf_box[3] + 0.02
+ and addition_box[3] <= shelf_box[3] + 0.08
+ ):
+ matched_shelf = shelf
+ break
+ if (
+ matched_hood is None
+ and matched_shelf is None
+ and matched_bed is None
+ and addition_box is not None
+ and bool(
+ addition_tokens
+ & {
+ "plant",
+ "foliage",
+ "leaf",
+ "leaves",
+ "stem",
+ "stems",
+ "flower",
+ "flowers",
+ "branch",
+ "branches",
+ }
+ )
+ ):
+ addition_area = (
+ (addition_box[2] - addition_box[0])
+ * (addition_box[3] - addition_box[1])
+ )
+ for assembly in verified_potted_plant_assemblies:
+ assembly_box = box(assembly)
+ assert assembly_box is not None
+ intersection = (
+ max(
+ 0.0,
+ min(addition_box[2], assembly_box[2])
+ - max(addition_box[0], assembly_box[0]),
+ )
+ * max(
+ 0.0,
+ min(addition_box[3], assembly_box[3])
+ - max(addition_box[1], assembly_box[1]),
+ )
+ )
+ if intersection / max(addition_area, 1e-9) >= 0.98:
+ matched_potted_plant = assembly
+ break
+ if (
+ matched_hood is None
+ and matched_shelf is None
+ and matched_bed is None
+ and matched_potted_plant is None
+ ):
+ kept_additions.append(addition)
+ addition_id_remap[f"B_{index:03d}"] = (
+ f"B_{len(kept_additions):03d}"
+ )
+ else:
+ suppression = {
+ "raw_addition_index": index,
+ "category": str(addition.get("category") or ""),
+ "bbox_2d_norm_xyxy": list(
+ addition.get("bbox_2d_norm_xyxy") or []
+ ),
+ }
+ if matched_potted_plant is not None:
+ suppression.update(
+ {
+ "matched_potted_plant_object_id": str(
+ matched_potted_plant.get("object_id") or ""
+ ),
+ "reason": (
+ "foliage_inside_verified_potted_plant_is_not_a_second_object"
+ ),
+ }
+ )
+ elif matched_bed is not None:
+ suppression.update(
+ {
+ "matched_bed_object_id": str(
+ matched_bed.get("object_id") or ""
+ ),
+ "reason": (
+ "physical_headboard_is_part_of_complete_bed_assembly"
+ ),
+ }
+ )
+ elif matched_shelf is not None:
+ suppression.update(
+ {
+ "matched_shelf_object_id": str(
+ matched_shelf.get("object_id") or ""
+ ),
+ "reason": (
+ "thin_under_shelf_light_is_attached_host_assembly"
+ ),
+ }
+ )
+ else:
+ suppression.update(
+ {
+ "matched_hood_object_id": str(
+ matched_hood.get("object_id") or ""
+ ),
+ "reason": "recessed_light_inside_range_hood_extent",
+ }
+ )
+ suppressed_additions.append(suppression)
+ effective["additions"] = kept_additions
+ suppressed_bed_addition_ids = {
+ f"B_{int(item['raw_addition_index']):03d}"
+ for item in suppressed_additions
+ if item.get("matched_bed_object_id")
+ }
+ binding_errors = []
+ for decision in effective.get("local_candidate_decisions") or []:
+ if decision.get("decision") != "bind_to_addition":
+ continue
+ old_target = str(decision.get("target_object_id") or "")
+ new_target = addition_id_remap.get(old_target)
+ if new_target is None:
+ binding_errors.append(
+ {
+ "type": "candidate_bound_to_suppressed_addition",
+ "candidate_id": str(decision.get("candidate_id") or ""),
+ "target_object_id": old_target,
+ }
+ )
+ decision["decision"] = "reject"
+ decision["target_object_id"] = None
+ decision["reason"] = "target addition suppressed by local policy"
+ else:
+ decision["target_object_id"] = new_target
+ for decision in effective.get("assigned_evidence_decisions") or []:
+ supporting = list(decision.get("supporting_addition_ids") or [])
+ remapped = [
+ addition_id_remap[value]
+ for value in supporting
+ if value in addition_id_remap
+ ]
+ missing_supporting = [
+ value for value in supporting if value not in addition_id_remap
+ ]
+ source_id = str(decision.get("source_object_id") or "")
+ bed_split_absorbed = bool(
+ missing_supporting
+ and decision.get("decision") == "split_instances"
+ and is_bed(live_by_id.get(source_id, {}))
+ and set(missing_supporting) <= suppressed_bed_addition_ids
+ )
+ if bed_split_absorbed:
+ previous_reason = str(decision.get("reason") or "")
+ effective_decision = (
+ "update_existing"
+ if any(
+ str(item.get("pass_a_id") or "") == source_id
+ for item in kept_updates
+ )
+ else "pass"
+ )
+ decision["decision"] = effective_decision
+ decision["reason"] = (
+ "local complete-bed policy retained the physical headboard"
+ )
+ rewritten_evidence_decisions.append(
+ {
+ "source_object_id": source_id,
+ "previous_decision": "split_instances",
+ "effective_decision": effective_decision,
+ "previous_reason": previous_reason,
+ "effective_reason": decision["reason"],
+ }
+ )
+ elif missing_supporting:
+ binding_errors.append(
+ {
+ "type": "assigned_evidence_references_suppressed_addition",
+ "evidence_id": str(decision.get("evidence_id") or ""),
+ "supporting_addition_ids": supporting,
+ }
+ )
+ decision["supporting_addition_ids"] = remapped
+ audit = {
+ "schema_version": "scenemosaic_post_pass_b_policy_guard",
+ "status": "pass" if not binding_errors else "blocked",
+ "partition_count": len(partitions),
+ "partitions": partitions,
+ "raw_update_count": len(actions.get("updates") or []),
+ "effective_update_count": len(kept_updates),
+ "suppressed_updates": suppressed_updates,
+ "failed_mask_semantic_rejections": failed_mask_semantic_rejections,
+ "rewritten_evidence_decisions": rewritten_evidence_decisions,
+ "raw_removal_count": len(actions.get("removals") or []),
+ "effective_removal_count": len(kept_removals),
+ "suppressed_removals": suppressed_removals,
+ "raw_addition_count": len(actions.get("additions") or []),
+ "effective_addition_count": len(kept_additions),
+ "addition_id_remap": addition_id_remap,
+ "suppressed_additions": suppressed_additions,
+ "hard_errors": binding_errors,
+ "formal_outputs_modified": False,
+ }
+ return {"actions": effective, "audit": audit}
diff --git a/vlm_agent/reconstruction/perception/pipeline/core/publication.py b/vlm_agent/reconstruction/perception/pipeline/core/publication.py
new file mode 100644
index 0000000..10f7a49
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/core/publication.py
@@ -0,0 +1,135 @@
+from __future__ import annotations
+
+import copy
+import os
+import re
+import shutil
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+def atomic_publish(staging: str | Path, final: str | Path) -> None:
+ staging_path = Path(staging).expanduser().resolve()
+ final_path = Path(final).expanduser().resolve()
+ if not staging_path.is_dir():
+ raise FileNotFoundError(staging_path)
+ if final_path.exists():
+ raise FileExistsError(final_path)
+ final_path.parent.mkdir(parents=True, exist_ok=True)
+ os.replace(staging_path, final_path)
+
+
+def safe_name(value: Any) -> str:
+ return re.sub(r"[^A-Za-z0-9_.-]+", "_", str(value)).strip("_") or "object"
+
+
+def canonicalize_final_object_assets(
+ *,
+ objects: Sequence[Mapping[str, Any]],
+ staging_dir: str | Path,
+ image_np: np.ndarray,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ # Removal and replacement stages may renumber objects after their masks
+ # were materialized. Build a clean two-phase asset tree so published
+ # filenames match the final registry and superseded files cannot leak
+ # into manual review.
+ copied = [copy.deepcopy(dict(item)) for item in objects]
+ staging = Path(staging_dir).expanduser().resolve()
+ masks_dir = staging / "masks"
+ overlays_dir = staging / "overlays"
+ temporary_root = staging / ".canonical_final_assets"
+ if temporary_root.exists():
+ raise FileExistsError(
+ f"canonical final asset directory already exists: {temporary_root}"
+ )
+ temporary_masks = temporary_root / "masks"
+ temporary_overlays = temporary_root / "overlays"
+ temporary_masks.mkdir(parents=True)
+ temporary_overlays.mkdir(parents=True)
+
+ old_mask_names = {
+ path.name for path in masks_dir.glob("*") if path.is_file()
+ } if masks_dir.is_dir() else set()
+ old_overlay_names = {
+ path.name for path in overlays_dir.glob("*") if path.is_file()
+ } if overlays_dir.is_dir() else set()
+ target_mask_names: set[str] = set()
+ target_overlay_names: set[str] = set()
+ path_updates: list[tuple[dict[str, Any], str, str]] = []
+ external_source_count = 0
+
+ for obj in copied:
+ sam3 = copy.deepcopy(dict(obj.get("sam3") or {}))
+ obj["sam3"] = sam3
+ if sam3.get("status") != "success":
+ continue
+ object_id = str(obj.get("object_id") or "")
+ if not re.fullmatch(r"proposal_\d{3}", object_id):
+ raise ValueError(f"invalid final object ID for asset publication: {object_id}")
+ category = safe_name(str(obj.get("category") or "object").lower())
+ mask_name = f"{object_id}_{category}_mask.png"
+ overlay_name = f"{object_id}_{category}_overlay.png"
+ if mask_name in target_mask_names or overlay_name in target_overlay_names:
+ raise ValueError(f"duplicate final asset name for {object_id}")
+ source_mask = Path(str(sam3.get("mask_path") or "")).expanduser().resolve()
+ if not source_mask.is_file():
+ raise FileNotFoundError(
+ f"missing final source mask for {object_id}: {source_mask}"
+ )
+ try:
+ source_mask.relative_to(staging)
+ except ValueError:
+ external_source_count += 1
+ with Image.open(source_mask) as source:
+ mask = np.asarray(source.convert("L")) > 0
+ if tuple(mask.shape) != tuple(image_np.shape[:2]):
+ raise ValueError(
+ f"final mask shape mismatch for {object_id}: "
+ f"{tuple(mask.shape)} != {tuple(image_np.shape[:2])}"
+ )
+ Image.fromarray(mask.astype(np.uint8) * 255).save(
+ temporary_masks / mask_name
+ )
+ overlay = np.asarray(image_np, dtype=np.uint8).copy()
+ overlay[mask] = (
+ 0.55 * overlay[mask]
+ + 0.45 * np.array((255, 0, 0), dtype=np.uint8)
+ ).astype(np.uint8)
+ Image.fromarray(overlay).save(temporary_overlays / overlay_name)
+ target_mask_names.add(mask_name)
+ target_overlay_names.add(overlay_name)
+ path_updates.append((sam3, mask_name, overlay_name))
+
+ if masks_dir.exists():
+ shutil.rmtree(masks_dir)
+ if overlays_dir.exists():
+ shutil.rmtree(overlays_dir)
+ os.replace(temporary_masks, masks_dir)
+ os.replace(temporary_overlays, overlays_dir)
+ temporary_root.rmdir()
+
+ for sam3, mask_name, overlay_name in path_updates:
+ sam3["mask_path"] = str(masks_dir / mask_name)
+ sam3["overlay_path"] = str(overlays_dir / overlay_name)
+
+ audit = {
+ "schema_version": "scenemosaic_canonical_final_asset_publication",
+ "status": "pass",
+ "successful_object_count": len(path_updates),
+ "published_mask_count": len(target_mask_names),
+ "published_overlay_count": len(target_overlay_names),
+ "external_source_count": external_source_count,
+ "removed_stale_mask_names": sorted(old_mask_names - target_mask_names),
+ "removed_stale_overlay_names": sorted(
+ old_overlay_names - target_overlay_names
+ ),
+ "final_mask_names": sorted(target_mask_names),
+ "final_overlay_names": sorted(target_overlay_names),
+ "hard_errors": [],
+ "additional_vlm_calls": 0,
+ "additional_sam3_inference_calls": 0,
+ "formal_outputs_modified": False,
+ }
+ return copied, audit
diff --git a/vlm_agent/reconstruction/perception/pipeline/core/quality_gates.py b/vlm_agent/reconstruction/perception/pipeline/core/quality_gates.py
new file mode 100644
index 0000000..d45bebd
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/core/quality_gates.py
@@ -0,0 +1,2071 @@
+from __future__ import annotations
+
+import copy
+import math
+import re
+import shutil
+from collections import defaultdict
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+from pipeline.core.inventory_fusion import _object_prompt_map
+from pipeline.core.publication import safe_name
+from pipeline.core.runtime_io import sha256_file
+
+def _mask_sha256(path: Path) -> str:
+ return sha256_file(path)
+
+
+def build_bounded_candidate_semantic_recall_gate(
+ selection_audit: Mapping[str, Any],
+ pass_b_actions: Mapping[str, Any],
+ final_objects: Sequence[Mapping[str, Any]] | None = None,
+) -> dict[str, Any]:
+ """Fail closed unless strong bounded recall evidence reaches final output."""
+
+ decisions = {
+ str(item.get("candidate_id") or ""): dict(item)
+ for item in pass_b_actions.get("local_candidate_decisions") or []
+ if isinstance(item, Mapping)
+ }
+ expected: list[dict[str, Any]] = []
+ for raw in selection_audit.get("selected") or []:
+ if not isinstance(raw, Mapping):
+ continue
+ candidate_id = str(raw.get("candidate_id") or "")
+ namespace = str(raw.get("source_namespace") or "")
+ prompt_terms = {
+ token
+ for token in re.sub(
+ r"[^a-z0-9]+", " ", str(raw.get("prompt") or "").lower()
+ ).split()
+ if token
+ }
+ bbox = raw.get("bbox_norm_xyxy")
+ try:
+ score = float(raw.get("score"))
+ top_edge_limit_met = (
+ isinstance(bbox, Sequence)
+ and not isinstance(bbox, (str, bytes))
+ and len(bbox) == 4
+ and float(bbox[3]) <= 0.10
+ )
+ except (TypeError, ValueError):
+ score = -1.0
+ top_edge_limit_met = False
+ reason = None
+ if (
+ namespace == "top_zone"
+ and bool(prompt_terms & {"light", "lighting", "lamp"})
+ and score >= 0.65
+ and top_edge_limit_met
+ ):
+ reason = "high_confidence_top_edge_light"
+ if reason:
+ expected.append(
+ {
+ "candidate_id": candidate_id,
+ "source_namespace": namespace,
+ "reason": reason,
+ "score": score,
+ "bbox_norm_xyxy": list(bbox),
+ "source_mask_path": str(raw.get("source_mask_path") or ""),
+ }
+ )
+ accepted_ids = sorted(
+ item["candidate_id"]
+ for item in expected
+ if decisions.get(item["candidate_id"], {}).get("decision")
+ == "bind_to_addition"
+ )
+ rejected_ids = sorted(
+ item["candidate_id"]
+ for item in expected
+ if decisions.get(item["candidate_id"], {}).get("decision")
+ == "reject"
+ )
+ # A bounded top-zone candidate may duplicate an existing Pass-A object
+ # whose failed mask was repaired through a Pass-B update. In that case a
+ # `reject` means "already represented", not "omit this light". Accept it
+ # only with direct mask evidence, not bbox proximity: the candidate must
+ # be nearly contained by one successful final lighting mask.
+ resolved_existing_ids: list[str] = []
+ resolved_existing_matches: list[dict[str, Any]] = []
+ for item in expected:
+ candidate_id = item["candidate_id"]
+ decision = decisions.get(candidate_id, {})
+ action = str(decision.get("decision") or "")
+ if (
+ item["reason"] != "high_confidence_top_edge_light"
+ or action not in {"reject", "replace_existing"}
+ or not final_objects
+ ):
+ continue
+ candidate_path = Path(str(item.get("source_mask_path") or ""))
+ if not candidate_path.is_file():
+ continue
+ try:
+ candidate_mask = (
+ np.array(Image.open(candidate_path).convert("L")) > 0
+ )
+ except (OSError, ValueError):
+ continue
+ candidate_area = int(candidate_mask.sum())
+ if candidate_area < 32:
+ continue
+ for final_obj in final_objects:
+ if not isinstance(final_obj, Mapping):
+ continue
+ sam3 = final_obj.get("sam3") or {}
+ final_path = Path(str(sam3.get("mask_path") or ""))
+ label = " ".join(
+ str(final_obj.get(field) or "")
+ for field in (
+ "category",
+ "sam3_prompt",
+ "instance_name",
+ "description",
+ "object_type",
+ )
+ ).lower()
+ if (
+ str(sam3.get("status") or "") != "success"
+ or not final_path.is_file()
+ or not ({"light", "lighting", "lamp"} & set(label.split()))
+ ):
+ continue
+ if action == "replace_existing":
+ target_source_id = str(decision.get("target_object_id") or "")
+ has_target_source = any(
+ isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "A"
+ and str(source.get("source_id") or "") == target_source_id
+ for source in final_obj.get("proposal_sources") or []
+ )
+ binding_candidate_ids = {
+ str(candidate.get("candidate_id") or "")
+ for candidate in (
+ (final_obj.get("local_candidate_binding") or {})
+ .get("candidates")
+ or []
+ )
+ if isinstance(candidate, Mapping)
+ }
+ if (
+ not has_target_source
+ or candidate_id not in binding_candidate_ids
+ ):
+ continue
+ try:
+ final_mask = np.array(Image.open(final_path).convert("L")) > 0
+ except (OSError, ValueError):
+ continue
+ if final_mask.shape != candidate_mask.shape:
+ continue
+ intersection = int(np.logical_and(candidate_mask, final_mask).sum())
+ union = int(np.logical_or(candidate_mask, final_mask).sum())
+ candidate_coverage = intersection / candidate_area
+ iou = intersection / union if union else 0.0
+ if candidate_coverage < 0.98 or iou < 0.50:
+ continue
+ resolved_existing_ids.append(candidate_id)
+ resolved_existing_matches.append(
+ {
+ "candidate_id": candidate_id,
+ "final_object_id": str(final_obj.get("object_id") or ""),
+ "candidate_coverage": round(candidate_coverage, 6),
+ "mask_iou": round(iou, 6),
+ "resolution": (
+ "reviewed_local_candidate_bound_to_existing_object"
+ if action == "replace_existing"
+ else "already_represented_by_final_lighting_mask"
+ ),
+ }
+ )
+ break
+ resolved_ids = set(accepted_ids) | set(resolved_existing_ids)
+ unresolved = [
+ {
+ **item,
+ "actual_decision": decisions.get(item["candidate_id"], {}).get(
+ "decision", "missing"
+ ),
+ }
+ for item in expected
+ if item["candidate_id"] not in resolved_ids
+ ]
+ hard_errors = []
+ if unresolved:
+ hard_errors.append(
+ {
+ "type": "bounded_candidate_semantic_recall_unresolved",
+ "candidate_ids": [
+ item["candidate_id"] for item in unresolved
+ ],
+ "details": unresolved,
+ }
+ )
+ return {
+ "schema_version": "scenemosaic_bounded_candidate_semantic_recall_gate",
+ "status": "pass" if not hard_errors else "blocked",
+ "expected_accept_count": len(expected),
+ "accepted_count": len(accepted_ids),
+ "accepted_candidate_ids": accepted_ids,
+ "rejected_candidate_ids": rejected_ids,
+ "resolved_existing_candidate_ids": sorted(resolved_existing_ids),
+ "resolved_existing_matches": resolved_existing_matches,
+ "unresolved": unresolved,
+ "hard_errors": hard_errors,
+ }
+
+
+
+
+def build_container_content_completeness_gate(
+ recall_audit: Mapping[str, Any] | None,
+) -> dict[str, Any]:
+ """Require every explicit fruit plan to complete its container mask."""
+
+ if recall_audit is None:
+ return {
+ "schema_version": "scenemosaic_fruit_container_completeness_gate",
+ "status": "disabled",
+ "plan_count": 0,
+ "completed_source_object_ids": [],
+ "unresolved": [],
+ "hard_errors": [],
+ }
+ completed_ids = {
+ str(value)
+ for value in recall_audit.get("completed_source_object_ids") or []
+ }
+ unresolved = []
+ for plan in recall_audit.get("plans") or []:
+ if not isinstance(plan, Mapping):
+ continue
+ source_id = str(plan.get("target_source_object_id") or "")
+ if source_id not in completed_ids:
+ unresolved.append(
+ {
+ "source_object_id": source_id,
+ "prompt": str(plan.get("prompt") or ""),
+ "resolution": "fruit_not_recalled_or_container_not_completed",
+ }
+ )
+ hard_errors = []
+ if unresolved:
+ hard_errors.append(
+ {
+ "type": "explicit_fruit_container_completion_unresolved",
+ "source_object_ids": [
+ item["source_object_id"] for item in unresolved
+ ],
+ "details": unresolved,
+ }
+ )
+ return {
+ "schema_version": "scenemosaic_fruit_container_completeness_gate",
+ "status": "pass" if not hard_errors else "blocked",
+ "plan_count": len(recall_audit.get("plans") or []),
+ "completed_source_object_ids": sorted(completed_ids),
+ "unresolved": unresolved,
+ "hard_errors": hard_errors,
+ }
+
+def _record_category_text(record: Mapping[str, Any]) -> str:
+ """Return semantic category text, including the old-manifest prompt fallback."""
+
+ return " ".join(
+ str(record.get(key) or "")
+ for key in ("category", "sam3_prompt")
+ )
+
+
+def _tiny_ceiling_light_extent_tolerance(
+ record: Mapping[str, Any],
+ decision: Mapping[str, Any] | None = None,
+) -> dict[str, Any] | None:
+ """Accept coarse VLM localization only for a unique tiny top light."""
+
+ category_tokens = set(
+ re.sub(
+ r"[^a-z0-9]+", " ", _record_category_text(record).lower()
+ ).split()
+ )
+ if not (
+ "ceiling" in category_tokens
+ and bool(category_tokens & {"light", "lighting", "downlight"})
+ ):
+ return None
+ decision = decision or {}
+ reason = str(decision.get("reason") or "").lower()
+ visually_verified_top_edge = (
+ str(decision.get("decision") or "") == "pass"
+ and "cover" in reason
+ and "ceiling light" in reason
+ and "top edge" in reason
+ )
+ if (
+ str(record.get("status") or "") != "success"
+ or (
+ float(record.get("score") or 0.0) < 0.65
+ and not visually_verified_top_edge
+ )
+ or bool(record.get("merge_review_required"))
+ or int(record.get("component_count") or 0) != 1
+ ):
+ return None
+ try:
+ norm = [float(value) for value in record.get("bbox_norm_xyxy")]
+ claimed_box = [
+ float(value) for value in record.get("claimed_bbox_abs_xyxy")
+ ]
+ actual_box = [
+ float(value) for value in record.get("actual_mask_bbox_abs_xyxy")
+ ]
+ except (TypeError, ValueError):
+ return None
+ if len(norm) != 4 or len(claimed_box) != 4 or len(actual_box) != 4:
+ return None
+ claimed_width = claimed_box[2] - claimed_box[0]
+ claimed_height = claimed_box[3] - claimed_box[1]
+ actual_width = actual_box[2] - actual_box[0]
+ actual_height = actual_box[3] - actual_box[1]
+ if (
+ norm[3] > 0.17
+ or max(claimed_width, claimed_height) > 64.0
+ or max(actual_width, actual_height) > 64.0
+ or min(claimed_width, claimed_height, actual_width, actual_height)
+ <= 0.0
+ ):
+ return None
+ area_ratio = (actual_width * actual_height) / (
+ claimed_width * claimed_height
+ )
+ claimed_center = (
+ (claimed_box[0] + claimed_box[2]) / 2.0,
+ (claimed_box[1] + claimed_box[3]) / 2.0,
+ )
+ actual_center = (
+ (actual_box[0] + actual_box[2]) / 2.0,
+ (actual_box[1] + actual_box[3]) / 2.0,
+ )
+ center_scale = math.dist(claimed_center, actual_center) / max(
+ math.hypot(claimed_width, claimed_height),
+ math.hypot(actual_width, actual_height),
+ 1.0,
+ )
+ if not (0.10 <= area_ratio <= 4.0 and center_scale <= 1.25):
+ return None
+ return {
+ "policy": (
+ "visually_verified_tiny_top_ceiling_light"
+ if visually_verified_top_edge
+ else "unique_tiny_top_ceiling_light"
+ ),
+ "minimum_score": 0.50 if visually_verified_top_edge else 0.65,
+ "maximum_bottom_normalized_y": 0.17,
+ "maximum_side_pixels": 64,
+ "maximum_center_distance_scale": 1.25,
+ "area_ratio": round(area_ratio, 6),
+ "center_distance_scale": round(center_scale, 6),
+ }
+
+
+def _repeated_recessed_ceiling_light_row_extent_tolerance(
+ record: Mapping[str, Any],
+ peer_records: Sequence[Mapping[str, Any]],
+ decision: Mapping[str, Any],
+) -> dict[str, Any] | None:
+ """Allow deeper perspective rows only with a matching recessed-light peer."""
+
+ category_tokens = set(
+ re.findall(
+ r"[a-z0-9]+",
+ _record_category_text(record).lower().replace("_", " "),
+ )
+ )
+ reason = str(decision.get("reason") or "").lower()
+ if not (
+ str(decision.get("decision") or "") == "pass"
+ and {"ceiling", "light"} <= category_tokens
+ and bool(category_tokens & {"recessed", "downlight"})
+ and "light" in reason
+ and bool(re.search(r"\b(?:match|matches|matched|cover|covered|visible)\b", reason))
+ and not re.search(
+ r"\b(?:missing|misses|incomplete|partial|incorrect|extra|leak)\w*\b",
+ reason,
+ )
+ and str(record.get("status") or "") == "success"
+ and float(record.get("score") or 0.0) >= 0.75
+ and not bool(record.get("merge_review_required"))
+ and int(record.get("component_count") or 0) == 1
+ ):
+ return None
+
+ def geometry(item: Mapping[str, Any]):
+ try:
+ norm = [float(value) for value in item["bbox_norm_xyxy"]]
+ claimed = [
+ float(value) for value in item["claimed_bbox_abs_xyxy"]
+ ]
+ actual = [
+ float(value) for value in item["actual_mask_bbox_abs_xyxy"]
+ ]
+ except (KeyError, TypeError, ValueError):
+ return None
+ if not (
+ len(norm) == 4
+ and len(claimed) == 4
+ and len(actual) == 4
+ and 0.0 <= norm[0] < norm[2] <= 1.0
+ and 0.0 <= norm[1] < norm[3] <= 1.0
+ and claimed[0] < claimed[2]
+ and claimed[1] < claimed[3]
+ and actual[0] < actual[2]
+ and actual[1] < actual[3]
+ ):
+ return None
+ claimed_width = claimed[2] - claimed[0]
+ claimed_height = claimed[3] - claimed[1]
+ actual_width = actual[2] - actual[0]
+ actual_height = actual[3] - actual[1]
+ if (
+ norm[3] > 0.25
+ or max(
+ claimed_width,
+ claimed_height,
+ actual_width,
+ actual_height,
+ )
+ > 64.0
+ ):
+ return None
+ claimed_center = (
+ (claimed[0] + claimed[2]) / 2.0,
+ (claimed[1] + claimed[3]) / 2.0,
+ )
+ actual_center = (
+ (actual[0] + actual[2]) / 2.0,
+ (actual[1] + actual[3]) / 2.0,
+ )
+ scale = max(
+ math.hypot(claimed_width, claimed_height),
+ math.hypot(actual_width, actual_height),
+ 1.0,
+ )
+ area_ratio = (actual_width * actual_height) / (
+ claimed_width * claimed_height
+ )
+ center_scale = math.dist(claimed_center, actual_center) / scale
+ if not (0.25 <= area_ratio <= 1.50 and center_scale <= 0.35):
+ return None
+ return {
+ "norm_center_x": (norm[0] + norm[2]) / 2.0,
+ "norm_center_y": (norm[1] + norm[3]) / 2.0,
+ "actual_width": actual_width,
+ "actual_height": actual_height,
+ "area_ratio": area_ratio,
+ "center_distance_scale": center_scale,
+ }
+
+ current = geometry(record)
+ if current is None:
+ return None
+ peer_ids = []
+ for peer in peer_records:
+ if str(peer.get("evidence_id") or "") == str(
+ record.get("evidence_id") or ""
+ ):
+ continue
+ peer_tokens = set(
+ re.findall(
+ r"[a-z0-9]+",
+ str(peer.get("category") or "").lower().replace("_", " "),
+ )
+ )
+ if not (
+ {"ceiling", "light"} <= peer_tokens
+ and bool(peer_tokens & {"recessed", "downlight"})
+ and str(peer.get("status") or "") == "success"
+ and float(peer.get("score") or 0.0) >= 0.75
+ and not bool(peer.get("merge_review_required"))
+ and int(peer.get("component_count") or 0) == 1
+ ):
+ continue
+ peer_geometry = geometry(peer)
+ if peer_geometry is None:
+ continue
+ width_similarity = min(
+ current["actual_width"], peer_geometry["actual_width"]
+ ) / max(current["actual_width"], peer_geometry["actual_width"])
+ height_similarity = min(
+ current["actual_height"], peer_geometry["actual_height"]
+ ) / max(current["actual_height"], peer_geometry["actual_height"])
+ if not (
+ abs(
+ current["norm_center_y"]
+ - peer_geometry["norm_center_y"]
+ )
+ <= 0.015
+ and abs(
+ current["norm_center_x"]
+ - peer_geometry["norm_center_x"]
+ )
+ >= 0.10
+ and width_similarity >= 0.65
+ and height_similarity >= 0.65
+ ):
+ continue
+ peer_ids.append(str(peer.get("evidence_id") or ""))
+
+ if not peer_ids:
+ return None
+ return {
+ "policy": "repeated_recessed_ceiling_light_row",
+ "minimum_score": 0.75,
+ "maximum_bottom_normalized_y": 0.25,
+ "maximum_vertical_center_delta_normalized": 0.015,
+ "minimum_horizontal_center_separation_normalized": 0.10,
+ "minimum_dimension_similarity": 0.65,
+ "maximum_side_pixels": 64,
+ "maximum_center_distance_scale": 0.35,
+ "area_ratio": round(current["area_ratio"], 6),
+ "center_distance_scale": round(
+ current["center_distance_scale"], 6
+ ),
+ "peer_evidence_ids": sorted(peer_ids),
+ }
+
+
+def _small_flat_media_extent_tolerance(
+ record: Mapping[str, Any],
+) -> dict[str, Any] | None:
+ """Accept a centered, unique thin book/notebook mask inside a coarse box."""
+
+ category_tokens = set(
+ re.sub(
+ r"[^a-z0-9]+", " ", _record_category_text(record).lower()
+ ).split()
+ )
+ if not bool(
+ category_tokens & {"book", "notebook", "paper", "document", "magazine"}
+ ):
+ return None
+ if (
+ str(record.get("status") or "") != "success"
+ or float(record.get("score") or 0.0) < 0.70
+ or bool(record.get("merge_review_required"))
+ or int(record.get("component_count") or 0) != 1
+ ):
+ return None
+ try:
+ claimed = [
+ float(value) for value in record.get("claimed_bbox_abs_xyxy")
+ ]
+ actual = [
+ float(value) for value in record.get("actual_mask_bbox_abs_xyxy")
+ ]
+ except (TypeError, ValueError):
+ return None
+ if len(claimed) != 4 or len(actual) != 4:
+ return None
+ claimed_width = claimed[2] - claimed[0]
+ claimed_height = claimed[3] - claimed[1]
+ actual_width = actual[2] - actual[0]
+ actual_height = actual[3] - actual[1]
+ dimensions = (claimed_width, claimed_height, actual_width, actual_height)
+ if (
+ min(dimensions) <= 0.0
+ or max(claimed_width, actual_width) > 192.0
+ or max(claimed_height, actual_height) > 96.0
+ or min(claimed_width, claimed_height) > 48.0
+ or min(actual_width, actual_height) > 48.0
+ ):
+ return None
+ width_ratio = actual_width / claimed_width
+ height_ratio = actual_height / claimed_height
+ area_ratio = width_ratio * height_ratio
+ claimed_center = (
+ (claimed[0] + claimed[2]) / 2.0,
+ (claimed[1] + claimed[3]) / 2.0,
+ )
+ actual_center = (
+ (actual[0] + actual[2]) / 2.0,
+ (actual[1] + actual[3]) / 2.0,
+ )
+ center_scale = math.dist(claimed_center, actual_center) / max(
+ math.hypot(claimed_width, claimed_height),
+ math.hypot(actual_width, actual_height),
+ 1.0,
+ )
+ if not (
+ 0.40 <= area_ratio <= 2.50
+ and 0.50 <= width_ratio <= 2.00
+ and 0.50 <= height_ratio <= 2.00
+ and center_scale <= 0.20
+ ):
+ return None
+ return {
+ "policy": "unique_small_flat_media_localization",
+ "minimum_score": 0.70,
+ "maximum_width_pixels": 192,
+ "maximum_height_pixels": 96,
+ "maximum_thin_side_pixels": 48,
+ "maximum_center_distance_scale": 0.20,
+ "width_ratio": round(width_ratio, 6),
+ "height_ratio": round(height_ratio, 6),
+ "area_ratio": round(area_ratio, 6),
+ "center_distance_scale": round(center_scale, 6),
+ }
+
+
+def _thin_attached_shelf_extent_tolerance(
+ record: Mapping[str, Any],
+) -> dict[str, Any] | None:
+ """Allow a thin shelf board mask only when an attached light was suppressed."""
+
+ category_tokens = set(
+ re.findall(r"[a-z0-9]+", _record_category_text(record).lower())
+ )
+ if "shelf" not in category_tokens:
+ return None
+ try:
+ claimed = [float(value) for value in record["claimed_bbox_abs_xyxy"]]
+ actual = [float(value) for value in record["actual_mask_bbox_abs_xyxy"]]
+ except (KeyError, TypeError, ValueError):
+ return None
+ if len(claimed) != 4 or len(actual) != 4:
+ return None
+ claimed_width = claimed[2] - claimed[0]
+ claimed_height = claimed[3] - claimed[1]
+ actual_width = actual[2] - actual[0]
+ actual_height = actual[3] - actual[1]
+ if min(claimed_width, claimed_height, actual_width, actual_height) <= 0:
+ return None
+ width_ratio = actual_width / claimed_width
+ height_ratio = actual_height / claimed_height
+ center_x_delta_scale = abs(
+ (actual[0] + actual[2] - claimed[0] - claimed[2]) / 2.0
+ ) / claimed_width
+ if not (
+ claimed_width >= 160
+ and claimed_height <= 64
+ and claimed_width / claimed_height >= 6.0
+ and 0.90 <= width_ratio <= 1.10
+ and 0.65 <= height_ratio <= 1.10
+ and center_x_delta_scale <= 0.05
+ ):
+ return None
+ return {
+ "policy": "thin_shelf_with_locally_suppressed_attached_light",
+ "minimum_height_ratio": 0.65,
+ "minimum_width_ratio": 0.90,
+ "maximum_width_ratio": 1.10,
+ "minimum_claimed_width_pixels": 160,
+ "maximum_claimed_height_pixels": 64,
+ "minimum_aspect_ratio": 6.0,
+ "width_ratio": round(width_ratio, 6),
+ "height_ratio": round(height_ratio, 6),
+ "center_x_delta_scale": round(center_x_delta_scale, 6),
+ }
+
+
+def _thin_wall_shelf_span_extent_tolerance(
+ record: Mapping[str, Any],
+ decision: Mapping[str, Any],
+) -> dict[str, Any] | None:
+ """Accept a verified, very thin wall-shelf span despite a loose VLM height.
+
+ This is deliberately limited to a one-piece wall shelf whose SAM3 mask
+ spans the claimed width. It does not relax cabinets, open shelving, or
+ shelves with contents.
+ """
+
+ if (
+ str(decision.get("decision") or "") != "pass"
+ or str(record.get("status") or "") != "success"
+ or bool(record.get("merge_review_required"))
+ ):
+ return None
+ category_tokens = set(
+ re.findall(r"[a-z0-9]+", _record_category_text(record).lower())
+ )
+ if not {"wall", "shelf"} <= category_tokens:
+ return None
+ try:
+ score = float(record.get("score") or 0.0)
+ component_count = int(record.get("component_count"))
+ claimed = [float(value) for value in record["claimed_bbox_abs_xyxy"]]
+ actual = [float(value) for value in record["actual_mask_bbox_abs_xyxy"]]
+ except (KeyError, TypeError, ValueError):
+ return None
+ if len(claimed) != 4 or len(actual) != 4 or component_count != 1:
+ return None
+ claimed_width = claimed[2] - claimed[0]
+ claimed_height = claimed[3] - claimed[1]
+ actual_width = actual[2] - actual[0]
+ actual_height = actual[3] - actual[1]
+ if min(claimed_width, claimed_height, actual_width, actual_height) <= 0:
+ return None
+ width_ratio = actual_width / claimed_width
+ center_x_delta_scale = abs(
+ (actual[0] + actual[2] - claimed[0] - claimed[2]) / 2.0
+ ) / claimed_width
+ decision_reason = str(decision.get("reason") or "").lower()
+ if not (
+ score >= 0.80
+ and claimed_width >= 160
+ and claimed_height <= 96
+ and claimed_width / claimed_height >= 4.0
+ and actual_height <= min(32.0, 0.35 * claimed_height)
+ and 0.90 <= width_ratio <= 1.10
+ and center_x_delta_scale <= 0.05
+ and ("shelf" in decision_reason or "span" in decision_reason)
+ ):
+ return None
+ return {
+ "policy": "verified_thin_wall_shelf_span",
+ "minimum_score": 0.80,
+ "minimum_claimed_width_pixels": 160,
+ "maximum_claimed_height_pixels": 96,
+ "maximum_actual_height_pixels": 32,
+ "minimum_width_ratio": 0.90,
+ "maximum_width_ratio": 1.10,
+ "width_ratio": round(width_ratio, 6),
+ "center_x_delta_scale": round(center_x_delta_scale, 6),
+ }
+
+
+def _simple_visible_object_extent_tolerance(
+ record: Mapping[str, Any],
+ decision: Mapping[str, Any],
+) -> dict[str, Any] | None:
+ """Allow a loose claim box only for explicitly verified simple objects."""
+
+ if (
+ str(decision.get("decision") or "") != "pass"
+ or str(record.get("status") or "") != "success"
+ ):
+ return None
+ try:
+ coverage = float(record["claimed_bbox_coverage_by_mask_bbox"])
+ component_count = int(record["component_count"])
+ claimed = [float(value) for value in record["claimed_bbox_abs_xyxy"]]
+ actual = [float(value) for value in record["actual_mask_bbox_abs_xyxy"]]
+ except (KeyError, TypeError, ValueError):
+ return None
+ if coverage < 0.35 or component_count != 1:
+ return None
+ if not (
+ len(claimed) == 4
+ and len(actual) == 4
+ and claimed[0] < claimed[2]
+ and claimed[1] < claimed[3]
+ and actual[0] < actual[2]
+ and actual[1] < actual[3]
+ ):
+ return None
+ claimed_width = claimed[2] - claimed[0]
+ claimed_height = claimed[3] - claimed[1]
+ actual_width = actual[2] - actual[0]
+ actual_height = actual[3] - actual[1]
+ actual_center = (
+ (actual[0] + actual[2]) / 2.0,
+ (actual[1] + actual[3]) / 2.0,
+ )
+ pad_x = max(4.0, 0.05 * claimed_width)
+ pad_y = max(4.0, 0.05 * claimed_height)
+ if not (
+ claimed[0] - pad_x <= actual_center[0] <= claimed[2] + pad_x
+ and claimed[1] - pad_y <= actual_center[1] <= claimed[3] + pad_y
+ and actual_width <= 1.15 * claimed_width
+ and actual_height <= 1.15 * claimed_height
+ ):
+ return None
+
+ category_tokens = set(
+ re.findall(r"[a-z0-9]+", _record_category_text(record).lower())
+ )
+ reason = str(decision.get("reason") or "").lower()
+ if re.search(
+ r"\b(?:missing|misses|omits?|omitted|incomplete|partial|partially|"
+ r"incorrect|leaks?|leaking|extra)\b|"
+ r"\bnot\s+(?:fully\s+)?covered\b",
+ reason,
+ ):
+ return None
+ merge_review_required = bool(record.get("merge_review_required"))
+ verified_separate_pillow = (
+ bool(category_tokens & {"pillow", "cushion"})
+ and float(record.get("score") or 0.0) >= 0.80
+ and coverage >= 0.55
+ and "mask cover" in reason
+ and bool(
+ re.search(
+ r"(?:one|visible|separate|individual)",
+ reason,
+ )
+ )
+ and bool(re.search(r"(?:pillow|cushion)", reason))
+ )
+ if merge_review_required and not verified_separate_pillow:
+ return None
+ policy = None
+ if verified_separate_pillow:
+ policy = "visually_verified_separate_pillow_mask"
+ elif (
+ category_tokens & {"pillow", "cushion"}
+ and reason
+ == (
+ "local assembly policy keeps a pillow excluded from the bed "
+ "mask as an independent inventory instance"
+ )
+ and float(record.get("score") or 0.0) >= 0.90
+ and coverage >= 0.55
+ ):
+ policy = "high_confidence_independent_pillow_preserved_from_bed_assembly"
+ elif (
+ category_tokens & {"pillow", "cushion"}
+ and re.search(r"(?:pillow|cushion).{0,32}cover", reason)
+ ):
+ policy = "simple_pillow_explicitly_covered"
+ elif (
+ "lamp" in category_tokens
+ and all(token in reason for token in ("base", "shade", "cover"))
+ ):
+ policy = "lamp_base_and_shade_explicitly_covered"
+ elif (
+ {"soap", "dispenser"} <= category_tokens
+ and reason == "mask matches visible soap dispenser"
+ and float(record.get("score") or 0.0) >= 0.90
+ and coverage >= 0.65
+ ):
+ policy = "high_confidence_visible_soap_dispenser"
+ elif (
+ ({"bedside", "table"} <= category_tokens or "nightstand" in category_tokens)
+ and "visible" in reason
+ and "cover" in reason
+ and bool({"top", "front", "body"} & set(re.findall(r"[a-z0-9]+", reason)))
+ ):
+ policy = "occluded_bedside_table_visible_surfaces_explicitly_covered"
+ if policy is None:
+ return None
+ return {
+ "policy": f"{policy}",
+ "minimum_coverage": 0.35,
+ "component_count": component_count,
+ "actual_to_claimed_width_ratio": round(actual_width / claimed_width, 6),
+ "actual_to_claimed_height_ratio": round(actual_height / claimed_height, 6),
+ }
+
+
+def _verified_lighting_fixture_extent_tolerance(
+ record: Mapping[str, Any],
+ decision: Mapping[str, Any],
+) -> dict[str, Any] | None:
+ """Bounded tolerance for a verified sconce or floor lamp with a loose VLM box.
+
+ This permits only the two transparent, one-piece lighting cases observed in
+ the blind-scene regression. It remains stricter than the ordinary
+ simple-object tolerance: category, score, coverage, component count, and
+ the Pass-B visual justification must all agree. It is not a general
+ relaxation for lights, assemblies, or incomplete masks.
+ """
+
+ if (
+ str(decision.get("decision") or "") != "pass"
+ or str(record.get("status") or "") != "success"
+ or bool(record.get("merge_review_required"))
+ ):
+ return None
+ try:
+ score = float(record.get("score") or 0.0)
+ coverage = float(record["claimed_bbox_coverage_by_mask_bbox"])
+ component_count = int(record["component_count"])
+ claimed = [float(value) for value in record["claimed_bbox_abs_xyxy"]]
+ actual = [float(value) for value in record["actual_mask_bbox_abs_xyxy"]]
+ except (KeyError, TypeError, ValueError):
+ return None
+ if not (score >= 0.90 and 0.60 <= coverage < 0.75 and component_count == 1):
+ return None
+ if not (
+ len(claimed) == 4
+ and len(actual) == 4
+ and claimed[0] < claimed[2]
+ and claimed[1] < claimed[3]
+ and actual[0] < actual[2]
+ and actual[1] < actual[3]
+ ):
+ return None
+ claimed_width = claimed[2] - claimed[0]
+ claimed_height = claimed[3] - claimed[1]
+ actual_width = actual[2] - actual[0]
+ actual_height = actual[3] - actual[1]
+ actual_center = (
+ (actual[0] + actual[2]) / 2.0,
+ (actual[1] + actual[3]) / 2.0,
+ )
+ pad_x = max(4.0, 0.05 * claimed_width)
+ pad_y = max(4.0, 0.05 * claimed_height)
+ if not (
+ claimed[0] - pad_x <= actual_center[0] <= claimed[2] + pad_x
+ and claimed[1] - pad_y <= actual_center[1] <= claimed[3] + pad_y
+ and actual_width <= 1.15 * claimed_width
+ and actual_height <= 1.15 * claimed_height
+ ):
+ return None
+ category_tokens = set(
+ re.findall(r"[a-z0-9]+", _record_category_text(record).lower())
+ )
+ reason = str(decision.get("reason") or "").lower()
+ if re.search(
+ r"\b(?:missing|misses|omits?|omitted|incomplete|partial|partially|"
+ r"incorrect|leaks?|leaking|extra)\b|"
+ r"\bnot\s+(?:fully\s+)?covered\b",
+ reason,
+ ):
+ return None
+ policy = None
+ if {"wall", "sconce"} <= category_tokens and (
+ "wall sconce" in reason
+ and "body" in reason
+ and ("match" in reason or "covered" in reason)
+ ):
+ policy = "high_confidence_wall_sconce_body_verified"
+ elif {"floor", "lamp"} <= category_tokens and all(
+ token in reason for token in ("floor lamp", "shade", "legs", "cover")
+ ):
+ policy = "high_confidence_floor_lamp_shade_and_legs_verified"
+ if policy is None:
+ return None
+ return {
+ "policy": f"{policy}",
+ "minimum_score": 0.90,
+ "minimum_coverage": 0.60,
+ "component_count": component_count,
+ "actual_to_claimed_width_ratio": round(actual_width / claimed_width, 6),
+ "actual_to_claimed_height_ratio": round(actual_height / claimed_height, 6),
+ }
+
+
+def build_assigned_evidence_extent_gate(
+ assigned_manifest: Mapping[str, Any],
+ evidence_decisions: Sequence[Mapping[str, Any]],
+ *,
+ minimum_claimed_bbox_coverage: float = 0.75,
+ completed_container_content_source_ids: Sequence[str] = (),
+ completed_dedicated_attachment_source_ids: Sequence[str] = (),
+ attached_lighting_host_source_ids: Sequence[str] = (),
+ downstream_spatial_rescue_source_ids: Sequence[str] = (),
+ completed_fixed_component_source_ids: Sequence[str] = (),
+ locally_rebound_source_ids: Sequence[str] = (),
+) -> dict[str, Any]:
+ """Do not accept a mask whose extent materially misses its own claim."""
+
+ if not 0.0 < minimum_claimed_bbox_coverage <= 1.0:
+ raise ValueError("minimum claimed-bbox coverage must be in (0, 1]")
+ decisions = {
+ str(item.get("evidence_id") or ""): dict(item)
+ for item in evidence_decisions
+ if isinstance(item, Mapping)
+ }
+ records = [
+ item
+ for item in assigned_manifest.get("records") or []
+ if isinstance(item, Mapping)
+ ]
+ low_coverage = []
+ tolerated_tiny_localization = []
+ tolerated_repeated_ceiling_light_row_localization = []
+ tolerated_small_flat_media_localization = []
+ tolerated_thin_attached_shelf_localization = []
+ tolerated_simple_visible_object_localization = []
+ tolerated_verified_lighting_fixture_localization = []
+ downstream_spatial_rescues = []
+ completed_container_assemblies = []
+ completed_dedicated_attachment_assemblies = []
+ completed_fixed_component_assemblies = []
+ locally_rebound_masks = []
+ completed_container_sources = {
+ str(value) for value in completed_container_content_source_ids
+ }
+ completed_dedicated_attachment_sources = {
+ str(value) for value in completed_dedicated_attachment_source_ids
+ }
+ downstream_spatial_rescue_sources = {
+ str(value) for value in downstream_spatial_rescue_source_ids
+ }
+ completed_fixed_component_sources = {
+ str(value) for value in completed_fixed_component_source_ids
+ }
+ locally_rebound_sources = {
+ str(value) for value in locally_rebound_source_ids
+ }
+ attached_lighting_hosts = {
+ str(value) for value in attached_lighting_host_source_ids
+ }
+ unresolved = []
+ for record in records:
+ value = record.get("claimed_bbox_coverage_by_mask_bbox")
+ if value is None:
+ continue
+ coverage = float(value)
+ if coverage >= minimum_claimed_bbox_coverage:
+ continue
+ evidence_id = str(record.get("evidence_id") or "")
+ decision_record = decisions.get(evidence_id, {})
+ item = {
+ "evidence_id": evidence_id,
+ "source_object_id": str(record.get("source_object_id") or ""),
+ "category": str(record.get("category") or record.get("sam3_prompt") or ""),
+ "coverage": coverage,
+ "decision": str(decision_record.get("decision") or "missing"),
+ }
+ tiny_tolerance = _tiny_ceiling_light_extent_tolerance(
+ record, decision_record
+ )
+ repeated_ceiling_row_tolerance = (
+ _repeated_recessed_ceiling_light_row_extent_tolerance(
+ record, records, decision_record
+ )
+ )
+ flat_media_tolerance = _small_flat_media_extent_tolerance(record)
+ attached_shelf_tolerance = (
+ _thin_attached_shelf_extent_tolerance(record)
+ if item["source_object_id"] in attached_lighting_hosts
+ else None
+ )
+ simple_visible_tolerance = _simple_visible_object_extent_tolerance(
+ record, decision_record
+ )
+ verified_lighting_fixture_tolerance = (
+ _verified_lighting_fixture_extent_tolerance(record, decision_record)
+ )
+ thin_wall_shelf_tolerance = _thin_wall_shelf_span_extent_tolerance(
+ record, decision_record
+ )
+ if tiny_tolerance is not None:
+ item["tiny_localization_tolerance"] = tiny_tolerance
+ if repeated_ceiling_row_tolerance is not None:
+ item["repeated_ceiling_light_row_tolerance"] = (
+ repeated_ceiling_row_tolerance
+ )
+ if flat_media_tolerance is not None:
+ item["small_flat_media_localization_tolerance"] = (
+ flat_media_tolerance
+ )
+ if attached_shelf_tolerance is not None:
+ item["thin_attached_shelf_localization_tolerance"] = (
+ attached_shelf_tolerance
+ )
+ if simple_visible_tolerance is not None:
+ item["simple_visible_object_localization_tolerance"] = (
+ simple_visible_tolerance
+ )
+ if thin_wall_shelf_tolerance is not None:
+ item["thin_wall_shelf_span_tolerance"] = thin_wall_shelf_tolerance
+ if verified_lighting_fixture_tolerance is not None:
+ item["verified_lighting_fixture_localization_tolerance"] = (
+ verified_lighting_fixture_tolerance
+ )
+ low_coverage.append(item)
+ if item["decision"] == "pass":
+ if item["source_object_id"] in downstream_spatial_rescue_sources:
+ item["downstream_resolution"] = (
+ "strict_bbox_positive_sam3_spatial_rescue"
+ )
+ downstream_spatial_rescues.append(item)
+ elif item["source_object_id"] in locally_rebound_sources:
+ item["local_candidate_rebinding_resolution"] = (
+ "verified_final_local_candidate_replace"
+ )
+ locally_rebound_masks.append(item)
+ elif item["source_object_id"] in completed_fixed_component_sources:
+ item["fixed_component_resolution"] = (
+ "fixed_component_absorbed_into_functional_host_assembly"
+ )
+ completed_fixed_component_assemblies.append(item)
+ elif item["source_object_id"] in completed_container_sources:
+ item["container_content_resolution"] = (
+ "fruit_merged_into_completed_container_mask"
+ )
+ completed_container_assemblies.append(item)
+ elif item["source_object_id"] in completed_dedicated_attachment_sources:
+ item["dedicated_attachment_resolution"] = (
+ "attached_content_absorbed_into_explicit_host_assembly"
+ )
+ completed_dedicated_attachment_assemblies.append(item)
+ elif attached_shelf_tolerance is not None:
+ tolerated_thin_attached_shelf_localization.append(item)
+ elif tiny_tolerance is not None:
+ tolerated_tiny_localization.append(item)
+ elif repeated_ceiling_row_tolerance is not None:
+ tolerated_repeated_ceiling_light_row_localization.append(
+ item
+ )
+ elif flat_media_tolerance is not None:
+ tolerated_small_flat_media_localization.append(item)
+ elif thin_wall_shelf_tolerance is not None:
+ tolerated_thin_attached_shelf_localization.append(item)
+ elif verified_lighting_fixture_tolerance is not None:
+ tolerated_verified_lighting_fixture_localization.append(item)
+ elif simple_visible_tolerance is not None:
+ tolerated_simple_visible_object_localization.append(item)
+ else:
+ unresolved.append(item)
+ hard_errors = []
+ if unresolved:
+ hard_errors.append(
+ {
+ "type": "assigned_mask_claimed_extent_mismatch_passed",
+ "minimum_claimed_bbox_coverage": minimum_claimed_bbox_coverage,
+ "evidence_ids": [
+ item["evidence_id"] for item in unresolved
+ ],
+ "details": unresolved,
+ }
+ )
+ return {
+ "schema_version": "scenemosaic_assigned_evidence_extent_gate",
+ "status": "pass" if not hard_errors else "blocked",
+ "minimum_claimed_bbox_coverage": minimum_claimed_bbox_coverage,
+ "low_coverage": low_coverage,
+ "downstream_spatial_rescues": downstream_spatial_rescues,
+ "locally_rebound_masks": locally_rebound_masks,
+ "tolerated_tiny_localization": tolerated_tiny_localization,
+ "tolerated_repeated_ceiling_light_row_localization": (
+ tolerated_repeated_ceiling_light_row_localization
+ ),
+ "tolerated_small_flat_media_localization": (
+ tolerated_small_flat_media_localization
+ ),
+ "tolerated_thin_attached_shelf_localization": (
+ tolerated_thin_attached_shelf_localization
+ ),
+ "tolerated_localization": (
+ tolerated_tiny_localization
+ + tolerated_repeated_ceiling_light_row_localization
+ + tolerated_small_flat_media_localization
+ + tolerated_thin_attached_shelf_localization
+ + tolerated_verified_lighting_fixture_localization
+ + tolerated_simple_visible_object_localization
+ + downstream_spatial_rescues
+ + locally_rebound_masks
+ ),
+ "tolerated_simple_visible_object_localization": (
+ tolerated_simple_visible_object_localization
+ ),
+ "tolerated_verified_lighting_fixture_localization": (
+ tolerated_verified_lighting_fixture_localization
+ ),
+ "completed_container_content_assemblies": (
+ completed_container_assemblies
+ ),
+ "completed_fixed_component_assemblies": (
+ completed_fixed_component_assemblies
+ ),
+ "completed_dedicated_attachment_assemblies": (
+ completed_dedicated_attachment_assemblies
+ ),
+ "unresolved_pass_count": len(unresolved),
+ "hard_errors": hard_errors,
+ }
+
+
+def build_locally_validated_assigned_evidence_overflow(
+ assigned_manifest: Mapping[str, Any],
+ final_objects: Sequence[Mapping[str, Any]],
+) -> dict[str, Any]:
+ """Resolve only high-confidence, source-traceable evidence-panel overflow.
+
+ A panel capacity limit must not reject an object that is already locally
+ validated by SAM3 with a tight VLM-box match. The bypass is intentionally
+ unavailable for low-score, loose, merged, or missing-mask objects.
+ A 0.80 bbox IoU remains conservative here because the object must also
+ have score >= 0.90, a present mask, and no merge-review ambiguity.
+ """
+
+ suspicious = [
+ str(value)
+ for value in assigned_manifest.get("suspicious_overflow_object_ids") or []
+ if str(value)
+ ]
+ by_source: dict[str, Mapping[str, Any]] = {}
+ for obj in final_objects:
+ for source in obj.get("proposal_sources") or []:
+ if (
+ isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "A"
+ and str(source.get("source_id") or "")
+ ):
+ by_source[str(source["source_id"])] = obj
+ accepted = []
+ unresolved = []
+ for source_id in suspicious:
+ obj = by_source.get(source_id)
+ sam3 = dict((obj or {}).get("sam3") or {})
+ merge_review = sam3.get("merge_review") or {}
+ try:
+ score = float(sam3.get("score") or 0.0)
+ bbox_iou = float(sam3.get("vlm_bbox_iou") or 0.0)
+ except (TypeError, ValueError):
+ score, bbox_iou = 0.0, 0.0
+ mask_path = Path(str(sam3.get("mask_path") or ""))
+ valid = (
+ obj is not None
+ and sam3.get("status") == "success"
+ and score >= 0.90
+ and bbox_iou >= 0.80
+ and not bool(merge_review.get("required"))
+ and mask_path.is_file()
+ )
+ record = {
+ "source_object_id": source_id,
+ "proposal_id": str((obj or {}).get("object_id") or "") or None,
+ "score": round(score, 6),
+ "vlm_bbox_iou": round(bbox_iou, 6),
+ "merge_review_required": bool(merge_review.get("required")),
+ "mask_present": mask_path.is_file(),
+ }
+ if valid:
+ record["resolution"] = "local_sam3_high_confidence_bbox_match"
+ accepted.append(record)
+ else:
+ unresolved.append(record)
+ return {
+ "schema_version": "scenemosaic_assigned_evidence_overflow_local_validation",
+ "status": "pass" if not unresolved else "blocked",
+ "minimum_score": 0.90,
+ "minimum_vlm_bbox_iou": 0.80,
+ "accepted": accepted,
+ "unresolved": unresolved,
+ "hard_errors": ([] if not unresolved else [{
+ "type": "assigned_sam3_evidence_budget_overflow",
+ "object_ids": [item["source_object_id"] for item in unresolved],
+ }]),
+ }
+
+
+def resolve_assigned_evidence_semantic_gate_after_spatial_rescue(
+ assigned_manifest: Mapping[str, Any],
+ semantic_gate: Mapping[str, Any] | None,
+ final_objects: Sequence[Mapping[str, Any]],
+ downstream_spatial_rescue_source_ids: Sequence[str],
+ excluded_unmaterialized_source_ids: Sequence[str] = (),
+) -> dict[str, Any]:
+ """Clear an earlier uncertain evidence item only after a final spatial rescue.
+
+ The Pass-B decision remains in the audit trail. This is deliberately
+ limited to source objects whose later bbox-first rescue actually produced
+ a final SAM3 mask; it never converts an arbitrary uncertain decision into
+ a pass.
+ """
+
+ unresolved_ids = [
+ str(value)
+ for value in ((semantic_gate or {}).get("unresolved_evidence_ids") or [])
+ if str(value)
+ ]
+ evidence_source = {
+ str(item.get("evidence_id") or ""): str(item.get("source_object_id") or "")
+ for item in assigned_manifest.get("selected") or []
+ if isinstance(item, Mapping)
+ }
+ rescued_sources = {str(value) for value in downstream_spatial_rescue_source_ids if str(value)}
+ excluded_sources = {
+ str(value) for value in excluded_unmaterialized_source_ids if str(value)
+ }
+ final_by_source: dict[str, Mapping[str, Any]] = {}
+ for obj in final_objects:
+ for source in obj.get("proposal_sources") or []:
+ if (
+ isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "A"
+ and str(source.get("source_id") or "")
+ ):
+ final_by_source[str(source["source_id"])] = obj
+ resolved = []
+ still_unresolved = []
+ for evidence_id in unresolved_ids:
+ source_id = evidence_source.get(evidence_id, "")
+ obj = final_by_source.get(source_id)
+ sam3 = (obj or {}).get("sam3") or {}
+ mask_path = Path(str(sam3.get("mask_path") or ""))
+ if source_id in excluded_sources:
+ resolved.append({
+ "evidence_id": evidence_id,
+ "source_object_id": source_id,
+ "proposal_id": None,
+ "resolution": "unmaterialized_claim_excluded_after_failed_recovery",
+ })
+ elif (
+ source_id in rescued_sources
+ and sam3.get("status") == "success"
+ and mask_path.is_file()
+ ):
+ resolved.append({
+ "evidence_id": evidence_id,
+ "source_object_id": source_id,
+ "proposal_id": str((obj or {}).get("object_id") or ""),
+ "resolution": "downstream_bbox_spatial_rescue",
+ })
+ else:
+ still_unresolved.append(evidence_id)
+ return {
+ "schema_version": "scenemosaic_assigned_evidence_spatial_recovery",
+ "status": "pass" if not still_unresolved else "blocked",
+ "resolved": resolved,
+ "unresolved_evidence_ids": still_unresolved,
+ }
+
+
+
+
+def build_functional_assembly_gate(
+ objects: Sequence[Mapping[str, Any]],
+) -> dict[str, Any]:
+ """Detect sink/faucet or fitted-dishwasher pieces split from support."""
+
+ standalone_categories = {
+ "sink",
+ "kitchen_sink",
+ "vessel_sink",
+ "washbasin",
+ "basin",
+ "faucet",
+ "kitchen_faucet",
+ "bathroom_faucet",
+ }
+ support_terms = {"counter", "countertop", "vanity", "cabinet", "worktable"}
+ live = [
+ dict(item)
+ for item in objects
+ if (item.get("sam3") or {}).get("status") == "success"
+ ]
+ components = [
+ item
+ for item in live
+ if str(item.get("category") or "").lower() in standalone_categories
+ ]
+ supports = [
+ item
+ for item in live
+ if (
+ {
+ token
+ for token in re.sub(
+ r"[^a-z0-9]+",
+ " ",
+ str(item.get("category") or "").lower(),
+ ).split()
+ if token
+ }
+ & support_terms
+ )
+ ]
+ dishwashers = [
+ item
+ for item in live
+ if str(item.get("category") or "").lower()
+ in {"dishwasher", "built_in_dishwasher", "fitted_dishwasher"}
+ ]
+ sink_supports = [
+ item
+ for item in supports
+ if "sink"
+ in {
+ token
+ for token in re.sub(
+ r"[^a-z0-9]+",
+ " ",
+ " ".join(
+ str(item.get(key) or "").lower()
+ for key in ("category", "sam3_prompt", "instance_name")
+ ),
+ ).split()
+ if token
+ }
+ ]
+
+ def related(left: Mapping[str, Any], right: Mapping[str, Any]) -> bool:
+ left_box = _bbox(left)
+ right_box = _bbox(right)
+ ix = max(
+ 0.0,
+ min(left_box[2], right_box[2])
+ - max(left_box[0], right_box[0]),
+ )
+ iy = max(
+ 0.0,
+ min(left_box[3], right_box[3])
+ - max(left_box[1], right_box[1]),
+ )
+ if ix > 0.0 and iy > 0.0:
+ return True
+ return math.dist(_anchor(left), _anchor(right)) <= 0.12
+
+ unresolved = []
+ seen_pairs = set()
+ for component in components:
+ neighbors = [
+ other
+ for other in [*components, *supports]
+ if other.get("object_id") != component.get("object_id")
+ and related(component, other)
+ ]
+ for other in neighbors:
+ pair = tuple(
+ sorted(
+ (
+ str(component.get("object_id") or ""),
+ str(other.get("object_id") or ""),
+ )
+ )
+ )
+ if pair in seen_pairs:
+ continue
+ seen_pairs.add(pair)
+ unresolved.append(
+ {
+ "component_object_id": str(
+ component.get("object_id") or ""
+ ),
+ "component_category": str(
+ component.get("category") or ""
+ ),
+ "nearby_object_id": str(other.get("object_id") or ""),
+ "nearby_category": str(other.get("category") or ""),
+ }
+ )
+ hard_errors = []
+ if unresolved:
+ hard_errors.append(
+ {
+ "type": "functional_sink_assembly_split",
+ "pairs": unresolved,
+ }
+ )
+ dishwasher_splits = []
+ for dishwasher in dishwashers:
+ for support in sink_supports:
+ if not related(dishwasher, support):
+ continue
+ dishwasher_splits.append(
+ {
+ "dishwasher_object_id": str(
+ dishwasher.get("object_id") or ""
+ ),
+ "sink_support_object_id": str(
+ support.get("object_id") or ""
+ ),
+ "sink_support_category": str(
+ support.get("category") or ""
+ ),
+ }
+ )
+ if dishwasher_splits:
+ hard_errors.append(
+ {
+ "type": "functional_fitted_dishwasher_split",
+ "pairs": dishwasher_splits,
+ }
+ )
+ owner_overlap_pairs = []
+ checked_pairs = 0
+ if sink_supports:
+ import numpy as np
+ from PIL import Image
+
+ def semantic_tokens(item: Mapping[str, Any]) -> set[str]:
+ return set(
+ re.sub(
+ r"[^a-z0-9]+",
+ " ",
+ " ".join(
+ str(item.get(key) or "").lower()
+ for key in (
+ "category",
+ "sam3_prompt",
+ "instance_name",
+ )
+ ),
+ ).split()
+ )
+
+ def load_mask(item: Mapping[str, Any]):
+ mask_path = Path(
+ str((item.get("sam3") or {}).get("mask_path") or "")
+ )
+ if not mask_path.is_file():
+ return None
+ mask = np.array(Image.open(mask_path).convert("L")) > 0
+ return mask if mask.any() else None
+
+ rivals = [
+ item
+ for item in live
+ if not semantic_tokens(item) & {"sink", "faucet", "basin"}
+ and (
+ bool(semantic_tokens(item) & {"range", "stove"})
+ or "counter" in semantic_tokens(item)
+ )
+ ]
+ for sink_support in sink_supports:
+ sink_mask = load_mask(sink_support)
+ if sink_mask is None:
+ continue
+ for rival in rivals:
+ if rival.get("object_id") == sink_support.get("object_id"):
+ continue
+ rival_mask = load_mask(rival)
+ if rival_mask is None or rival_mask.shape != sink_mask.shape:
+ continue
+ checked_pairs += 1
+ intersection = int(np.logical_and(sink_mask, rival_mask).sum())
+ if intersection < 30:
+ continue
+ rival_area = int(rival_mask.sum())
+ sink_area = int(sink_mask.sum())
+ rival_coverage = intersection / rival_area
+ if rival_coverage < 0.50:
+ continue
+ owner_overlap_pairs.append(
+ {
+ "sink_support_object_id": str(
+ sink_support.get("object_id") or ""
+ ),
+ "rival_object_id": str(
+ rival.get("object_id") or ""
+ ),
+ "rival_category": str(rival.get("category") or ""),
+ "intersection_pixels": intersection,
+ "rival_coverage": round(rival_coverage, 6),
+ "sink_coverage": round(
+ intersection / sink_area,
+ 6,
+ ),
+ }
+ )
+ if owner_overlap_pairs:
+ hard_errors.append(
+ {
+ "type": "functional_sink_assembly_owner_overlap",
+ "minimum_rival_coverage": 0.50,
+ "pairs": owner_overlap_pairs,
+ }
+ )
+ return {
+ "schema_version": "scenemosaic_functional_assembly_gate",
+ "status": "pass" if not hard_errors else "blocked",
+ "checked_standalone_component_count": len(components),
+ "checked_dishwasher_count": len(dishwashers),
+ "unresolved_pairs": unresolved,
+ "dishwasher_splits": dishwasher_splits,
+ "checked_owner_overlap_pair_count": checked_pairs,
+ "owner_overlap_pairs": owner_overlap_pairs,
+ "hard_errors": hard_errors,
+ }
+
+
+# These errors mean a Stage 04 package cannot be trusted or materialized.
+# All other quality findings remain visible in best-effort delivery metadata,
+# but do not prevent reconstruction from proceeding with its own exclusions.
+INTEGRITY_ERROR_TYPES = frozenset(
+ {
+ "formal_outputs_modified",
+ "identity_manifest_mismatch",
+ "identity_mismatch",
+ "invalid_mask_file",
+ "local_candidate_evidence_hash_mismatch",
+ "mask_shape_mismatch",
+ "missing_local_candidate_evidence",
+ "missing_output_file",
+ }
+)
+
+
+def apply_delivery_quality_policy(
+ quality: Mapping[str, Any], *, strict: bool = False
+) -> dict[str, Any]:
+ """Classify observed findings for strict or best-effort publication.
+
+ Best-effort delivery retains every observed error in the audit. It blocks
+ only broken identity or file-integrity facts; SAM3D delivery separately
+ excludes objects without usable masks and labels the package degraded.
+ """
+
+ result = dict(quality)
+ observed = [
+ dict(item)
+ for item in result.get("hard_errors") or []
+ if isinstance(item, Mapping)
+ ]
+ blocking = (
+ list(observed)
+ if strict
+ else [
+ item
+ for item in observed
+ if str(item.get("type") or "") in INTEGRITY_ERROR_TYPES
+ ]
+ )
+ blocking_ids = {id(item) for item in blocking}
+ advisory = [
+ item for item in observed if id(item) not in blocking_ids
+ ]
+ result["observed_hard_errors"] = observed
+ result["hard_errors"] = blocking
+ result["blocking_errors"] = blocking
+ result["advisory_errors"] = advisory
+ result["quality_policy"] = {
+ "mode": "strict" if strict else "best_effort",
+ "integrity_error_types": sorted(INTEGRITY_ERROR_TYPES),
+ "observed_hard_error_count": len(observed),
+ "blocking_error_count": len(blocking),
+ "advisory_error_count": len(advisory),
+ }
+ result["status"] = "pass" if not blocking else "fail"
+ return result
+
+
+def build_quality_audit(
+ final_objects: Sequence[Mapping[str, Any]],
+ final_sam3_objects: Sequence[Mapping[str, Any]] | Mapping[str, Any],
+ mapping: Mapping[str, Any],
+) -> dict[str, Any]:
+ """Validate final identity and mask evidence without semantic guessing."""
+
+ import numpy as np
+ from PIL import Image
+
+ if isinstance(final_sam3_objects, Mapping):
+ sam3_objects = list(final_sam3_objects.get("objects") or [])
+ else:
+ sam3_objects = list(final_sam3_objects)
+ expected_ids = [str(item["object_id"]) for item in final_objects]
+ actual_ids = [str(item.get("object_id")) for item in sam3_objects]
+ hard_errors: list[dict[str, Any]] = []
+ warnings: list[dict[str, Any]] = []
+ if actual_ids != expected_ids:
+ hard_errors.append(
+ {
+ "type": "identity_mismatch",
+ "expected": expected_ids,
+ "actual": actual_ids,
+ }
+ )
+ mapping_ids = [
+ str(item.get("proposal_id")) for item in mapping.get("records") or []
+ ]
+ if mapping_ids != expected_ids:
+ hard_errors.append(
+ {
+ "type": "identity_manifest_mismatch",
+ "expected": expected_ids,
+ "actual": mapping_ids,
+ }
+ )
+
+ mask_hashes: dict[str, list[str]] = defaultdict(list)
+ mask_digests: dict[str, str] = {}
+ mask_arrays: dict[str, Any] = {}
+ success_count = 0
+ for obj in sam3_objects:
+ object_id = str(obj.get("object_id"))
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "success":
+ hard_errors.append(
+ {
+ "type": "sam3_failed",
+ "object_id": object_id,
+ "reason": sam3.get("reason"),
+ }
+ )
+ continue
+ success_count += 1
+ for key in ("mask_path", "overlay_path"):
+ raw_path = sam3.get(key)
+ if not raw_path or not Path(raw_path).is_file():
+ hard_errors.append(
+ {
+ "type": "missing_output_file",
+ "object_id": object_id,
+ "field": key,
+ "path": raw_path,
+ }
+ )
+ mask_path = sam3.get("mask_path")
+ if mask_path and Path(mask_path).is_file():
+ resolved_mask_path = Path(mask_path)
+ digest = _mask_sha256(resolved_mask_path)
+ mask_hashes[digest].append(object_id)
+ mask_digests[object_id] = digest
+ try:
+ with Image.open(resolved_mask_path) as source_mask:
+ mask_arrays[object_id] = np.array(
+ source_mask.convert("L")
+ ) > 0
+ except Exception as exc:
+ hard_errors.append(
+ {
+ "type": "invalid_mask_file",
+ "object_id": object_id,
+ "path": str(resolved_mask_path),
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ local_evidence = sam3.get("local_candidate_evidence") or {}
+ for evidence in local_evidence.get("candidates") or []:
+ evidence_path = Path(str(evidence.get("evidence_mask_path") or ""))
+ if not evidence_path.is_file():
+ hard_errors.append(
+ {
+ "type": "missing_local_candidate_evidence",
+ "object_id": object_id,
+ "candidate_id": evidence.get("candidate_id"),
+ "path": str(evidence_path),
+ }
+ )
+ continue
+ expected_digest = str(evidence.get("mask_sha256") or "")
+ actual_digest = _mask_sha256(evidence_path)
+ if not expected_digest or actual_digest != expected_digest:
+ hard_errors.append(
+ {
+ "type": "local_candidate_evidence_hash_mismatch",
+ "object_id": object_id,
+ "candidate_id": evidence.get("candidate_id"),
+ "expected": expected_digest or None,
+ "actual": actual_digest,
+ }
+ )
+ if sam3.get("warning") == "low_iou_with_vlm_bbox":
+ warnings.append(
+ {
+ "type": "low_iou_with_vlm_bbox",
+ "object_id": object_id,
+ "iou": sam3.get("vlm_bbox_iou"),
+ }
+ )
+ merge_review = sam3.get("merge_review") or {}
+ if merge_review.get("required"):
+ hard_errors.append(
+ {
+ "type": "unresolved_merge_review",
+ "object_id": object_id,
+ "details": merge_review,
+ }
+ )
+ duplicate_masks = [
+ {"sha256": digest, "object_ids": ids}
+ for digest, ids in mask_hashes.items()
+ if len(ids) > 1
+ ]
+ for duplicate in duplicate_masks:
+ hard_errors.append({"type": "identical_masks", **duplicate})
+
+ overlap_pairs: list[dict[str, Any]] = []
+ object_ids = list(mask_arrays)
+ for left_index, left_id in enumerate(object_ids):
+ left_mask = mask_arrays[left_id]
+ for right_id in object_ids[left_index + 1 :]:
+ right_mask = mask_arrays[right_id]
+ if left_mask.shape != right_mask.shape:
+ hard_errors.append(
+ {
+ "type": "mask_shape_mismatch",
+ "left_object_id": left_id,
+ "right_object_id": right_id,
+ "left_shape": list(left_mask.shape),
+ "right_shape": list(right_mask.shape),
+ }
+ )
+ continue
+ intersection = int(np.logical_and(left_mask, right_mask).sum())
+ if intersection < 30:
+ continue
+ left_area = int(left_mask.sum())
+ right_area = int(right_mask.sum())
+ union = left_area + right_area - intersection
+ iou = intersection / union if union else 0.0
+ left_coverage = intersection / left_area if left_area else 0.0
+ right_coverage = intersection / right_area if right_area else 0.0
+ containment = max(left_coverage, right_coverage)
+ pair = {
+ "left_object_id": left_id,
+ "right_object_id": right_id,
+ "intersection_pixels": intersection,
+ "iou": round(iou, 6),
+ "left_coverage": round(left_coverage, 6),
+ "right_coverage": round(right_coverage, 6),
+ }
+ if mask_digests.get(left_id) == mask_digests.get(right_id):
+ pair["severity"] = "identical_mask"
+ elif iou >= 0.55 or containment >= 0.90:
+ pair["severity"] = "hard_error"
+ hard_errors.append({"type": "high_mask_overlap", **pair})
+ elif iou >= 0.15 or containment >= 0.50:
+ pair["severity"] = "warning"
+ warnings.append({"type": "moderate_mask_overlap", **pair})
+ else:
+ pair["severity"] = "informational"
+ overlap_pairs.append(pair)
+ return {
+ "object_count": len(sam3_objects),
+ "success_count": success_count,
+ "hard_errors": hard_errors,
+ "warnings": warnings,
+ "duplicate_masks": duplicate_masks,
+ "mask_overlap_pairs": overlap_pairs,
+ "mask_overlap_thresholds": {
+ "hard_iou": 0.55,
+ "hard_containment": 0.90,
+ "warning_iou": 0.15,
+ "warning_containment": 0.50,
+ "minimum_intersection_pixels": 30,
+ },
+ }
+
+
+
+
+def _create_contact_sheet(
+ objects: Sequence[Mapping[str, Any]],
+ output_path: Path,
+ columns: int = 4,
+) -> None:
+ from PIL import Image, ImageDraw, ImageFont
+
+ cells = []
+ for obj in objects:
+ overlay_path = Path(str((obj.get("sam3") or {}).get("overlay_path") or ""))
+ if not overlay_path.is_file():
+ continue
+ with Image.open(overlay_path) as source:
+ image = source.convert("RGB")
+ image.thumbnail((360, 240), Image.Resampling.LANCZOS)
+ cell = Image.new("RGB", (380, 290), (245, 245, 245))
+ cell.paste(image, ((380 - image.width) // 2, 34))
+ draw = ImageDraw.Draw(cell)
+ try:
+ font = ImageFont.truetype("DejaVuSans.ttf", 17)
+ except OSError:
+ font = ImageFont.load_default()
+ label = f"{obj['object_id']} {obj.get('category', 'unknown')}"
+ draw.text((8, 7), label, fill=(0, 0, 0), font=font)
+ cells.append(cell)
+ if not cells:
+ raise RuntimeError("cannot build an empty SAM3 contact sheet")
+ rows = math.ceil(len(cells) / columns)
+ sheet = Image.new("RGB", (columns * 380, rows * 290), "white")
+ for index, cell in enumerate(cells):
+ sheet.paste(cell, ((index % columns) * 380, (index // columns) * 290))
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ sheet.save(output_path, quality=90)
+
+
+def _copy_final_masks(
+ *,
+ fused_objects: Sequence[Mapping[str, Any]],
+ affected_prompts: set[str],
+ pass_a_registry: Mapping[str, Any],
+ delta_registry: Mapping[str, Any] | None,
+ staging_dir: Path,
+ image_path: Path,
+) -> list[dict[str, Any]]:
+ from PIL import Image
+ import numpy as np
+
+ staging_masks = staging_dir / "masks"
+ staging_overlays = staging_dir / "overlays"
+ candidate_evidence_dir = staging_dir / "evidence" / "local_candidates"
+ staging_masks.mkdir(parents=True, exist_ok=True)
+ staging_overlays.mkdir(parents=True, exist_ok=True)
+ candidate_evidence_dir.mkdir(parents=True, exist_ok=True)
+ with Image.open(image_path) as source:
+ image_np = np.array(source.convert("RGB")).astype(np.uint8)
+ a_by_id = {
+ str(obj["object_id"]): obj
+ for obj in pass_a_registry.get("objects") or []
+ }
+ delta_by_id = {
+ str(obj["object_id"]): obj
+ for obj in (delta_registry or {}).get("objects") or []
+ }
+ final_prompt_by_id = _object_prompt_map(fused_objects)
+ final_objects: list[dict[str, Any]] = []
+
+ for raw_obj in fused_objects:
+ obj = copy.deepcopy(dict(raw_obj))
+ object_id = str(obj["object_id"])
+ prompt = final_prompt_by_id[object_id]
+ binding = obj.get("local_candidate_binding") or {}
+ binding_operation = binding.get("operation")
+ source_obj = None
+ source_kind = None
+ if binding_operation != "replace":
+ if prompt in affected_prompts:
+ source_obj = delta_by_id.get(object_id)
+ source_kind = "sam3_delta_affected_prompt_group"
+ else:
+ a_sources = [
+ str(source["source_id"])
+ for source in obj.get("proposal_sources") or []
+ if source.get("pass") == "A"
+ ]
+ source_obj = a_by_id.get(a_sources[0]) if len(a_sources) == 1 else None
+ source_kind = "sam3_a_reused_prompt_group"
+
+ mask = None
+ if binding_operation == "replace":
+ source_kind = "vlm_reviewed_local_candidate_replace"
+ source_sam3 = {
+ "status": "success",
+ "warning": None,
+ "selection_method": source_kind,
+ "prompt_used": prompt,
+ "score": max(
+ (float(item.get("score") or 0.0) for item in binding.get("candidates") or []),
+ default=0.0,
+ ),
+ "merge_review": {
+ "required": False,
+ "resolution_stage": "pass_b_local_candidate_review",
+ },
+ }
+ mask = np.zeros(image_np.shape[:2], dtype=bool)
+ elif source_obj is None:
+ obj["sam3"] = {
+ "status": "failed",
+ "reason": "missing_source_segmentation",
+ "selection_method": source_kind,
+ }
+ final_objects.append(obj)
+ continue
+ else:
+ source_sam3 = copy.deepcopy(dict(source_obj.get("sam3") or {}))
+ source_sam3["attempt_reuse_source"] = {
+ "kind": source_kind,
+ "source_object_id": source_obj.get("object_id"),
+ "final_prompt_group": prompt,
+ }
+ if source_sam3.get("status") == "success":
+ source_mask = Path(str(source_sam3.get("mask_path") or ""))
+ if source_mask.is_file():
+ mask = np.array(Image.open(source_mask).convert("L")) > 0
+ else:
+ source_sam3 = {
+ "status": "failed",
+ "reason": "source_mask_missing",
+ "selection_method": source_kind,
+ }
+
+ if binding_operation in {"replace", "union_existing"}:
+ if mask is None:
+ source_sam3 = {
+ "status": "failed",
+ "reason": "local_candidate_base_mask_missing",
+ "selection_method": source_kind,
+ }
+ else:
+ candidate_ids = []
+ candidate_evidence = []
+ for candidate in binding.get("candidates") or []:
+ candidate_id = str(candidate.get("candidate_id"))
+ candidate_path = Path(str(candidate.get("mask_path") or ""))
+ if not candidate_path.is_file():
+ source_sam3 = {
+ "status": "failed",
+ "reason": "local_candidate_mask_missing",
+ "candidate_id": candidate_id,
+ "selection_method": source_kind,
+ }
+ mask = None
+ break
+ candidate_mask = np.array(
+ Image.open(candidate_path).convert("L")
+ ) > 0
+ if candidate_mask.shape != mask.shape:
+ source_sam3 = {
+ "status": "failed",
+ "reason": "local_candidate_mask_shape_mismatch",
+ "candidate_id": candidate_id,
+ "selection_method": source_kind,
+ }
+ mask = None
+ break
+ mask |= candidate_mask
+ candidate_ids.append(candidate_id)
+ evidence_base = safe_name(candidate_id)
+ evidence_mask_path = (
+ candidate_evidence_dir / f"{evidence_base}_mask.png"
+ )
+ shutil.copy2(candidate_path, evidence_mask_path)
+ source_overlay_path = Path(
+ str(candidate.get("overlay_path") or "")
+ )
+ evidence_overlay_path = None
+ if source_overlay_path.is_file():
+ evidence_overlay_path = (
+ candidate_evidence_dir / f"{evidence_base}_overlay.png"
+ )
+ shutil.copy2(source_overlay_path, evidence_overlay_path)
+ candidate_evidence.append(
+ {
+ "candidate_id": candidate_id,
+ "source_mask_path": str(candidate_path),
+ "evidence_mask_path": str(evidence_mask_path),
+ "mask_sha256": _mask_sha256(evidence_mask_path),
+ "source_overlay_path": (
+ str(source_overlay_path)
+ if source_overlay_path.is_file()
+ else None
+ ),
+ "evidence_overlay_path": (
+ str(evidence_overlay_path)
+ if evidence_overlay_path is not None
+ else None
+ ),
+ }
+ )
+ if mask is not None:
+ source_sam3["selection_method"] = (
+ "vlm_reviewed_local_candidate_replace"
+ if binding_operation == "replace"
+ else "vlm_reviewed_local_candidate_union"
+ )
+ source_sam3["local_candidate_evidence"] = {
+ "operation": binding_operation,
+ "candidate_ids": candidate_ids,
+ "candidates": candidate_evidence,
+ }
+ source_sam3["merge_review"] = {
+ "required": False,
+ "resolution_stage": "pass_b_local_candidate_review",
+ }
+
+ if source_sam3.get("status") == "success" and mask is not None:
+ base = f"{safe_name(object_id)}_{safe_name(obj.get('category', 'unknown'))}"
+ mask_path = staging_masks / f"{base}_mask.png"
+ overlay_path = staging_overlays / f"{base}_overlay.png"
+ Image.fromarray(mask.astype(np.uint8) * 255).save(mask_path)
+ overlay = image_np.copy()
+ overlay[mask] = (
+ 0.55 * overlay[mask]
+ + 0.45 * np.array((255, 0, 0), dtype=np.uint8)
+ ).astype(np.uint8)
+ Image.fromarray(overlay).save(overlay_path)
+ source_sam3["mask_path"] = str(mask_path)
+ source_sam3["overlay_path"] = str(overlay_path)
+ source_sam3["area_pixels"] = int(mask.sum())
+ ys, xs = np.nonzero(mask)
+ source_sam3["mask_bbox_abs_xyxy"] = [
+ int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1
+ ]
+ obj["sam3"] = source_sam3
+ final_objects.append(obj)
+ return final_objects
diff --git a/vlm_agent/reconstruction/perception/pipeline/core/runtime_io.py b/vlm_agent/reconstruction/perception/pipeline/core/runtime_io.py
new file mode 100644
index 0000000..f52fab6
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/core/runtime_io.py
@@ -0,0 +1,73 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+def utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def read_json(path: str | Path) -> dict[str, Any]:
+ value = json.loads(Path(path).expanduser().resolve().read_text(encoding="utf-8"))
+ if not isinstance(value, dict):
+ raise ValueError(f"expected a JSON object: {path}")
+ return value
+
+
+def atomic_json(path: str | Path, payload: Any) -> None:
+ target = Path(path)
+ target.parent.mkdir(parents=True, exist_ok=True)
+ temporary = target.with_suffix(target.suffix + ".tmp")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, target)
+
+
+def sha256_file(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def snapshot_tree(paths: Sequence[str | Path]) -> dict[str, str]:
+ """Hash every regular file below the supplied paths."""
+
+ snapshot: dict[str, str] = {}
+ for raw_root in paths:
+ root = Path(raw_root).expanduser().resolve()
+ if not root.exists():
+ snapshot[str(root)] = "<missing>"
+ continue
+ if root.is_file():
+ snapshot[str(root)] = sha256_file(root)
+ continue
+ for path in sorted(item for item in root.rglob("*") if item.is_file()):
+ snapshot[str(path.resolve())] = sha256_file(path)
+ return snapshot
+
+
+def compare_snapshots(
+ before: Mapping[str, str],
+ after: Mapping[str, str],
+) -> dict[str, Any]:
+ before_keys = set(before)
+ after_keys = set(after)
+ changed = sorted(
+ key for key in before_keys & after_keys if before[key] != after[key]
+ )
+ return {
+ "unchanged": before == after,
+ "added": sorted(after_keys - before_keys),
+ "removed": sorted(before_keys - after_keys),
+ "changed": changed,
+ "before_file_count": len(before),
+ "after_file_count": len(after),
+ }
diff --git a/vlm_agent/reconstruction/perception/pipeline/core/sam3_review_evidence.py b/vlm_agent/reconstruction/perception/pipeline/core/sam3_review_evidence.py
new file mode 100644
index 0000000..e654276
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/core/sam3_review_evidence.py
@@ -0,0 +1,440 @@
+#!/usr/bin/env python3
+"""Build a bounded review panel from actual assigned SAM3 masks.
+
+The panel is evidence for the existing Pass-B semantic audit. It does not
+create, modify, or approve any mask and never writes to formal pipeline stages.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import os
+import re
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import cv2
+import numpy as np
+from PIL import Image, ImageDraw, ImageFont
+
+
+@dataclass(frozen=True)
+class AssignedEvidenceConfig:
+ maximum_objects: int = 12
+ low_score_threshold: float = 0.55
+ fragmented_component_count: int = 4
+ minimum_component_pixels: int = 16
+ crop_padding_fraction: float = 0.12
+ tile_width: int = 360
+ tile_height: int = 250
+ panel_columns: int = 3
+
+
+COMPLETENESS_TOKENS = {
+ "bathtub", "bed", "bench", "cabinet", "cart", "counter", "desk", "faucet",
+ "hutch", "island", "nightstand", "rack", "shelf", "shelves", "shelving",
+ "sink", "table", "vanity", "workbench",
+}
+
+FIXED_COMPONENT_ASSEMBLY_TOKEN_GROUPS = (
+ frozenset({"water", "dispenser"}),
+)
+
+
+def _font(size: int):
+ try:
+ return ImageFont.truetype("DejaVuSans.ttf", size)
+ except OSError:
+ return ImageFont.load_default()
+
+
+def _tokens(value: Any) -> set[str]:
+ return set(re.findall(r"[a-z0-9]+", str(value or "").lower().replace("_", " ")))
+
+
+def _object_text(obj: Mapping[str, Any]) -> str:
+ return " ".join(
+ str(obj.get(key) or "")
+ for key in ("category", "sam3_prompt", "instance_name", "description")
+ )
+
+
+def _mask_bbox(mask: np.ndarray) -> list[int] | None:
+ ys, xs = np.where(mask)
+ if not len(xs):
+ return None
+ return [int(xs.min()), int(ys.min()), int(xs.max()), int(ys.max())]
+
+
+def _normalized_bbox(box: Sequence[int], width: int, height: int) -> list[float]:
+ return [
+ round(float(box[0]) / width, 6),
+ round(float(box[1]) / height, 6),
+ round(float(box[2] + 1) / width, 6),
+ round(float(box[3] + 1) / height, 6),
+ ]
+
+
+def _bbox_from_object(obj: Mapping[str, Any], width: int, height: int) -> list[int]:
+ bbox = obj.get("bbox_2d_norm_xyxy")
+ if isinstance(bbox, list) and len(bbox) == 4:
+ return [
+ max(0, min(width - 1, int(math.floor(float(bbox[0]) * width)))),
+ max(0, min(height - 1, int(math.floor(float(bbox[1]) * height)))),
+ max(0, min(width - 1, int(math.ceil(float(bbox[2]) * width)) - 1)),
+ max(0, min(height - 1, int(math.ceil(float(bbox[3]) * height)) - 1)),
+ ]
+ return [0, 0, width - 1, height - 1]
+
+
+def _expand_bbox(
+ box: Sequence[int],
+ width: int,
+ height: int,
+ fraction: float,
+) -> list[int]:
+ x1, y1, x2, y2 = [int(value) for value in box]
+ pad_x = max(8, int(round((x2 - x1 + 1) * fraction)))
+ pad_y = max(8, int(round((y2 - y1 + 1) * fraction)))
+ return [
+ max(0, x1 - pad_x),
+ max(0, y1 - pad_y),
+ min(width - 1, x2 + pad_x),
+ min(height - 1, y2 + pad_y),
+ ]
+
+
+def _component_count(mask: np.ndarray, minimum_pixels: int) -> int:
+ count, _labels, stats, _centroids = cv2.connectedComponentsWithStats(
+ mask.astype(np.uint8), 8
+ )
+ return sum(
+ int(stats[index, cv2.CC_STAT_AREA]) >= minimum_pixels
+ for index in range(1, count)
+ )
+
+
+def select_assigned_sam3_evidence(
+ pass_a_objects: Sequence[Mapping[str, Any]],
+ sam3_registry: Mapping[str, Any],
+ *,
+ config: AssignedEvidenceConfig | None = None,
+) -> dict[str, Any]:
+ cfg = config or AssignedEvidenceConfig()
+ if not 1 <= cfg.maximum_objects <= 24:
+ raise ValueError("maximum_objects must be in [1, 24]")
+ pass_a_by_id = {
+ str(item.get("object_id") or ""): dict(item) for item in pass_a_objects
+ }
+ ranked: list[dict[str, Any]] = []
+ hard_errors: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for raw in sam3_registry.get("objects") or []:
+ sam3_obj = dict(raw)
+ object_id = str(sam3_obj.get("object_id") or "")
+ if not object_id or object_id in seen:
+ hard_errors.append(
+ {"type": "invalid_or_duplicate_assigned_object_id", "object_id": object_id}
+ )
+ continue
+ seen.add(object_id)
+ base = pass_a_by_id.get(object_id)
+ if base is None:
+ hard_errors.append(
+ {"type": "assigned_object_missing_from_pass_a", "object_id": object_id}
+ )
+ continue
+ sam3 = dict(sam3_obj.get("sam3") or {})
+ status = str(sam3.get("status") or "unknown")
+ merge_required = bool((sam3.get("merge_review") or {}).get("required"))
+ try:
+ score = float(sam3.get("score"))
+ except (TypeError, ValueError):
+ score = math.nan
+ low_score = not math.isfinite(score) or score < cfg.low_score_threshold
+ sam3_prompt = " ".join(str(base.get("sam3_prompt") or "").split())
+ if not sam3_prompt:
+ hard_errors.append(
+ {
+ "type": "assigned_object_missing_sam3_prompt",
+ "object_id": object_id,
+ }
+ )
+ continue
+ semantic_tokens = _tokens(_object_text(base))
+ completeness_sensitive = bool(semantic_tokens & COMPLETENESS_TOKENS)
+ fixed_component_assembly = any(
+ group <= semantic_tokens
+ for group in FIXED_COMPONENT_ASSEMBLY_TOKEN_GROUPS
+ )
+ component_count = None
+ mask_path = Path(str(sam3.get("mask_path") or "")).expanduser()
+ if mask_path.is_file():
+ try:
+ mask = np.array(Image.open(mask_path).convert("L")) > 0
+ component_count = _component_count(mask, cfg.minimum_component_pixels)
+ except Exception as exc:
+ hard_errors.append(
+ {
+ "type": "invalid_assigned_mask",
+ "object_id": object_id,
+ "path": str(mask_path),
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ fragmented = (
+ component_count is not None
+ and component_count > cfg.fragmented_component_count
+ )
+ reasons = []
+ priority = 0
+ if status != "success":
+ reasons.append("sam3_failed")
+ priority += 100
+ if merge_required:
+ reasons.append("merge_review_required")
+ priority += 90
+ if fragmented:
+ reasons.append("fragmented_mask")
+ priority += 70
+ if low_score:
+ reasons.append("low_or_missing_score")
+ priority += 50
+ if fixed_component_assembly:
+ reasons.append("fixed_component_assembly")
+ priority += 40
+ if completeness_sensitive:
+ reasons.append("completeness_sensitive_category")
+ priority += 30
+ if not reasons:
+ reasons.append("bounded_registry_coverage")
+ bbox = base.get("bbox_2d_norm_xyxy")
+ ranked.append(
+ {
+ "source_object_id": object_id,
+ "sam3_prompt": sam3_prompt,
+ "category": str(base.get("category") or ""),
+ "status": status,
+ "score": None if not math.isfinite(score) else round(score, 6),
+ "merge_review_required": merge_required,
+ "component_count": component_count,
+ "selection_reasons": reasons,
+ "priority": priority,
+ "bbox_norm_xyxy": bbox,
+ "mask_path": str(mask_path.resolve()) if mask_path.is_file() else None,
+ "overlay_path": str(Path(str(sam3.get("overlay_path") or "")).expanduser()),
+ }
+ )
+ ranked.sort(
+ key=lambda item: (
+ -int(item["priority"]),
+ item["bbox_norm_xyxy"][1]
+ if isinstance(item.get("bbox_norm_xyxy"), list)
+ else 2.0,
+ item["bbox_norm_xyxy"][0]
+ if isinstance(item.get("bbox_norm_xyxy"), list)
+ else 2.0,
+ item["source_object_id"],
+ )
+ )
+ selected = [dict(item) for item in ranked[: cfg.maximum_objects]]
+ for index, item in enumerate(selected, start=1):
+ item["evidence_id"] = f"E_{index:03d}"
+ overflow = ranked[cfg.maximum_objects :]
+ suspicious_overflow = [
+ item for item in overflow if int(item["priority"]) > 0
+ ]
+ return {
+ "schema_version": "scenemosaic_assigned_sam3_evidence_selection",
+ "status": "blocked" if hard_errors or suspicious_overflow else "pass",
+ "config": asdict(cfg),
+ "selected": selected,
+ "selected_count": len(selected),
+ "overflow_count": len(overflow),
+ "suspicious_overflow_count": len(suspicious_overflow),
+ "suspicious_overflow_object_ids": [
+ item["source_object_id"] for item in suspicious_overflow
+ ],
+ "hard_errors": hard_errors,
+ }
+
+
+def _fit(image: Image.Image, width: int, height: int) -> Image.Image:
+ result = image.copy()
+ result.thumbnail((width, height), Image.Resampling.LANCZOS)
+ canvas = Image.new("RGB", (width, height), "white")
+ canvas.paste(
+ result,
+ ((width - result.width) // 2, (height - result.height) // 2),
+ )
+ return canvas
+
+
+def materialize_assigned_sam3_review(
+ *,
+ image_path: str | Path,
+ pass_a_objects: Sequence[Mapping[str, Any]],
+ sam3_registry: Mapping[str, Any],
+ output_dir: str | Path,
+ config: AssignedEvidenceConfig | None = None,
+) -> dict[str, Any]:
+ cfg = config or AssignedEvidenceConfig()
+ output = Path(output_dir).expanduser().resolve()
+ if output.exists() and any(output.iterdir()):
+ raise FileExistsError(f"assigned evidence output is not empty: {output}")
+ output.mkdir(parents=True, exist_ok=True)
+ selection = select_assigned_sam3_evidence(
+ pass_a_objects, sam3_registry, config=cfg
+ )
+ source = Image.open(image_path).convert("RGB")
+ width, height = source.size
+ tile_width, tile_height = cfg.tile_width, cfg.tile_height
+ header_height = 44
+ visual_height = tile_height - header_height
+ half_width = tile_width // 2
+ tiles: list[Image.Image] = []
+ records: list[dict[str, Any]] = []
+ for item in selection["selected"]:
+ mask = None
+ mask_path = Path(str(item.get("mask_path") or ""))
+ if mask_path.is_file():
+ mask = np.array(Image.open(mask_path).convert("L")) > 0
+ if mask.shape != (height, width):
+ selection["hard_errors"].append(
+ {
+ "type": "assigned_mask_shape_mismatch",
+ "object_id": item["source_object_id"],
+ "actual": list(mask.shape),
+ "expected": [height, width],
+ }
+ )
+ mask = None
+ base = next(
+ obj for obj in pass_a_objects
+ if str(obj.get("object_id") or "") == item["source_object_id"]
+ )
+ claimed_box = _bbox_from_object(base, width, height)
+ mask_box = _mask_bbox(mask) if mask is not None else None
+ if mask_box is None:
+ evidence_box = claimed_box
+ claimed_bbox_coverage_by_mask_bbox = 0.0
+ else:
+ evidence_box = [
+ min(claimed_box[0], mask_box[0]),
+ min(claimed_box[1], mask_box[1]),
+ max(claimed_box[2], mask_box[2]),
+ max(claimed_box[3], mask_box[3]),
+ ]
+ ix1 = max(claimed_box[0], mask_box[0])
+ iy1 = max(claimed_box[1], mask_box[1])
+ ix2 = min(claimed_box[2], mask_box[2])
+ iy2 = min(claimed_box[3], mask_box[3])
+ intersection = (
+ max(0, ix2 - ix1 + 1) * max(0, iy2 - iy1 + 1)
+ )
+ claimed_area = (
+ (claimed_box[2] - claimed_box[0] + 1)
+ * (claimed_box[3] - claimed_box[1] + 1)
+ )
+ claimed_bbox_coverage_by_mask_bbox = intersection / claimed_area
+ crop_box = _expand_bbox(
+ evidence_box, width, height, cfg.crop_padding_fraction
+ )
+ x1, y1, x2, y2 = crop_box
+ original_crop = source.crop((x1, y1, x2 + 1, y2 + 1))
+ overlay_path = Path(str(item.get("overlay_path") or "")).expanduser()
+ if overlay_path.is_file():
+ overlay = Image.open(overlay_path).convert("RGB")
+ else:
+ overlay_np = np.array(source).copy()
+ if mask is not None:
+ overlay_np[mask] = (
+ 0.55 * overlay_np[mask]
+ + 0.45 * np.array((255, 0, 0), dtype=np.float32)
+ ).astype(np.uint8)
+ overlay = Image.fromarray(overlay_np)
+ overlay_crop = overlay.crop((x1, y1, x2 + 1, y2 + 1))
+ tile = Image.new("RGB", (tile_width, tile_height), "white")
+ tile.paste(_fit(original_crop, half_width, visual_height), (0, header_height))
+ tile.paste(
+ _fit(overlay_crop, tile_width - half_width, visual_height),
+ (half_width, header_height),
+ )
+ draw = ImageDraw.Draw(tile)
+ title = (
+ f"{item['evidence_id']} {item['source_object_id']} "
+ f"{item['sam3_prompt']} | original / mask"
+ )
+ draw.text((8, 5), title[:56], fill=(0, 0, 0), font=_font(15))
+ draw.text(
+ (8, 23),
+ ",".join(item["selection_reasons"])[:70],
+ fill=(90, 20, 20),
+ font=_font(12),
+ )
+ tiles.append(tile)
+ record = {
+ key: value
+ for key, value in item.items()
+ if key not in {"priority", "mask_path", "overlay_path"}
+ }
+ record["claimed_bbox_abs_xyxy"] = claimed_box
+ record["actual_mask_bbox_abs_xyxy"] = mask_box
+ record["claimed_bbox_coverage_by_mask_bbox"] = round(
+ claimed_bbox_coverage_by_mask_bbox, 6
+ )
+ record["crop_basis"] = "union_of_claimed_and_actual_mask_bbox"
+ record["crop_bbox_abs_xyxy"] = crop_box
+ record["crop_bbox_norm_xyxy"] = _normalized_bbox(
+ crop_box, width, height
+ )
+ record["source_mask_path"] = str(mask_path.resolve()) if mask_path.is_file() else None
+ records.append(record)
+
+ columns = max(1, cfg.panel_columns)
+ rows = max(1, math.ceil(len(tiles) / columns))
+ panel = Image.new(
+ "RGB",
+ (columns * tile_width, rows * tile_height + 34),
+ (238, 238, 238),
+ )
+ draw = ImageDraw.Draw(panel)
+ draw.text(
+ (8, 6),
+ "Assigned SAM3 evidence: each tile shows original crop then actual mask overlay",
+ fill=(0, 0, 0),
+ font=_font(16),
+ )
+ for index, tile in enumerate(tiles):
+ x = (index % columns) * tile_width
+ y = 34 + (index // columns) * tile_height
+ panel.paste(tile, (x, y))
+ panel_path = output / "assigned_sam3_evidence_panel.jpg"
+ panel.save(panel_path, quality=84, optimize=True)
+ manifest = {
+ **selection,
+ "status": (
+ "blocked"
+ if selection["hard_errors"]
+ or selection["suspicious_overflow_count"]
+ else "pass"
+ ),
+ "image_path": str(Path(image_path).expanduser().resolve()),
+ "records": records,
+ "panel_path": str(panel_path),
+ "formal_outputs_modified": False,
+ }
+ manifest_path = output / "assigned_sam3_evidence_manifest.json"
+ temporary = manifest_path.with_suffix(".json.tmp")
+ temporary.write_text(
+ json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, manifest_path)
+ return {
+ "manifest": manifest,
+ "manifest_path": str(manifest_path),
+ "panel_path": str(panel_path),
+ }
diff --git a/vlm_agent/reconstruction/perception/pipeline/core/sam3d_delivery_policy.py b/vlm_agent/reconstruction/perception/pipeline/core/sam3d_delivery_policy.py
new file mode 100644
index 0000000..f66273e
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/core/sam3d_delivery_policy.py
@@ -0,0 +1,773 @@
+#!/usr/bin/env python3
+"""Always materialize the best usable SAM3D input package for a scene.
+
+This module is deliberately downstream of perception. It preserves source
+perception facts and materializes derived, reconstruction-only masks when a
+high-overlap conflict must be assigned to one owner. It converts a published
+isolated-attempt result into one of four delivery outcomes:
+
+* ``sam3d_ready``: all source quality checks passed;
+* ``sam3d_ready_degraded``: source quality passed but a usable object mask
+ was excluded or ownership-derived;
+* ``quality_failed``: a draft package was retained for audit, but source
+ hard quality checks did not pass and SAM3D execution is forbidden;
+* ``infrastructure_failed``: no technically consumable scene package can be
+ produced (for example, unreadable image or zero valid masks).
+"""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+import os
+import re
+import shutil
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+
+SAFE_NAME_RE = re.compile(r"[^A-Za-z0-9_-]+")
+READY_STATUSES = {"sam3d_ready", "sam3d_ready_degraded"}
+DUPLICATE_MINIMUM_INTERSECTION_PIXELS = 30
+DUPLICATE_MINIMUM_IOU = 0.995
+DUPLICATE_MINIMUM_COVERAGE = 0.995
+RECONSTRUCTION_CONFLICT_MINIMUM_INTERSECTION_PIXELS = 30
+RECONSTRUCTION_CONFLICT_MINIMUM_IOU = 0.55
+RECONSTRUCTION_CONFLICT_MINIMUM_CONTAINMENT = 0.90
+RECONSTRUCTION_MINIMUM_REMAINING_PIXELS = 64
+RECONSTRUCTION_MINIMUM_REMAINING_FRACTION = 0.02
+
+
+def _utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def _safe_name(value: Any) -> str:
+ text = str(value or "unknown").strip().lower().replace(" ", "_")
+ text = SAFE_NAME_RE.sub("_", text)
+ return re.sub(r"_+", "_", text).strip("_") or "unknown"
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for block in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(block)
+ return digest.hexdigest()
+
+
+def _atomic_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _load_binary_mask(path: Path, expected_shape: tuple[int, int]) -> np.ndarray:
+ with Image.open(path) as source:
+ mask = np.array(source.convert("L")) > 0
+ if tuple(mask.shape) != tuple(expected_shape):
+ raise ValueError(
+ f"mask shape {tuple(mask.shape)} does not match image {expected_shape}"
+ )
+ if not bool(mask.any()):
+ raise ValueError("mask is empty")
+ return mask
+
+
+def _score(obj: Mapping[str, Any]) -> tuple[float, float]:
+ sam3 = obj.get("sam3") or {}
+ try:
+ score = float(sam3.get("score") or 0.0)
+ except (TypeError, ValueError):
+ score = 0.0
+ try:
+ bbox_iou = float(sam3.get("vlm_bbox_iou") or 0.0)
+ except (TypeError, ValueError):
+ bbox_iou = 0.0
+ return score, bbox_iou
+
+
+def _mask_metrics(left: np.ndarray, right: np.ndarray) -> dict[str, float | int]:
+ intersection = int(np.logical_and(left, right).sum())
+ left_area = int(left.sum())
+ right_area = int(right.sum())
+ union = left_area + right_area - intersection
+ return {
+ "intersection_pixels": intersection,
+ "iou": intersection / max(union, 1),
+ "left_coverage": intersection / max(left_area, 1),
+ "right_coverage": intersection / max(right_area, 1),
+ }
+
+
+def _has_pass_a_provenance(obj: Mapping[str, Any]) -> bool:
+ sources = obj.get("proposal_sources") or []
+ return any(
+ isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "A"
+ for source in sources
+ )
+
+
+def _duplicate_winner_key(
+ item: Mapping[str, Any],
+) -> tuple[int, float, float, int]:
+ score, bbox_iou = item["rank"]
+ return (
+ int(_has_pass_a_provenance(item["source_object"])),
+ score,
+ bbox_iou,
+ -int(item["source_index"]),
+ )
+
+
+def _is_reconstruction_conflict(metrics: Mapping[str, float | int]) -> bool:
+ return (
+ int(metrics["intersection_pixels"])
+ >= RECONSTRUCTION_CONFLICT_MINIMUM_INTERSECTION_PIXELS
+ and (
+ float(metrics["iou"]) >= RECONSTRUCTION_CONFLICT_MINIMUM_IOU
+ or max(
+ float(metrics["left_coverage"]),
+ float(metrics["right_coverage"]),
+ )
+ >= RECONSTRUCTION_CONFLICT_MINIMUM_CONTAINMENT
+ )
+ )
+
+
+def _derive_reconstruction_ownership(
+ items: Sequence[Mapping[str, Any]],
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Derive non-overlapping SAM3D masks without changing perception masks.
+
+ Only pairs already severe enough for the global quality gate are modified.
+ Earlier, better-evidenced objects own shared pixels; a later object with no
+ meaningful residual is excluded as a proved reconstruction duplicate.
+ """
+
+ ordered = sorted(
+ [copy.deepcopy(dict(item)) for item in items],
+ key=_duplicate_winner_key,
+ reverse=True,
+ )
+ derived_by_source: dict[str, np.ndarray] = {}
+ kept: list[dict[str, Any]] = []
+ allocations: list[dict[str, Any]] = []
+ excluded: list[dict[str, Any]] = []
+
+ for priority_index, item in enumerate(ordered):
+ source_id = str(item["source_object_id"])
+ source_mask = np.asarray(item["mask"], dtype=bool)
+ original_area = int(source_mask.sum())
+ owned_by: list[dict[str, Any]] = []
+ delivery_mask = source_mask.copy()
+ for owner in kept:
+ owner_id = str(owner["source_object_id"])
+ metrics = _mask_metrics(source_mask, owner["mask"])
+ if not _is_reconstruction_conflict(metrics):
+ continue
+ shared = np.logical_and(delivery_mask, derived_by_source[owner_id])
+ shared_pixels = int(shared.sum())
+ if not shared_pixels:
+ continue
+ delivery_mask &= ~derived_by_source[owner_id]
+ owned_by.append(
+ {
+ "source_object_id": owner_id,
+ "shared_pixels_removed": shared_pixels,
+ **{
+ key: round(float(value), 6)
+ for key, value in metrics.items()
+ },
+ }
+ )
+
+ delivery_area = int(delivery_mask.sum())
+ remaining_fraction = delivery_area / max(1, original_area)
+ ownership = {
+ "priority_index": priority_index,
+ "mode": "copy" if not owned_by else "shared_pixels_subtracted",
+ "owner_source_object_ids": [
+ str(record["source_object_id"]) for record in owned_by
+ ],
+ "original_area_pixels": original_area,
+ "delivery_area_pixels": delivery_area,
+ "shared_pixels_removed": original_area - delivery_area,
+ "remaining_fraction": round(remaining_fraction, 6),
+ "conflicts": owned_by,
+ }
+ if (
+ owned_by
+ and (
+ delivery_area < RECONSTRUCTION_MINIMUM_REMAINING_PIXELS
+ or remaining_fraction < RECONSTRUCTION_MINIMUM_REMAINING_FRACTION
+ )
+ ):
+ excluded.append(
+ {
+ "source_object_id": source_id,
+ "reason": "high_overlap_fully_owned_by_higher_priority_object",
+ "ownership": ownership,
+ }
+ )
+ continue
+
+ item["reconstruction_mask_ownership"] = ownership
+ item["delivery_mask"] = delivery_mask
+ derived_by_source[source_id] = delivery_mask
+ kept.append(item)
+ if owned_by:
+ allocations.append(
+ {
+ "source_object_id": source_id,
+ **ownership,
+ }
+ )
+
+ kept.sort(key=lambda item: int(item["source_index"]))
+ audit = {
+ "enabled": True,
+ "policy": (
+ "higher_evidence_owner_subtracts_only_global_high_overlap_pixels"
+ ),
+ "thresholds": {
+ "minimum_intersection_pixels": (
+ RECONSTRUCTION_CONFLICT_MINIMUM_INTERSECTION_PIXELS
+ ),
+ "minimum_iou": RECONSTRUCTION_CONFLICT_MINIMUM_IOU,
+ "minimum_containment": RECONSTRUCTION_CONFLICT_MINIMUM_CONTAINMENT,
+ "minimum_remaining_pixels": RECONSTRUCTION_MINIMUM_REMAINING_PIXELS,
+ "minimum_remaining_fraction": RECONSTRUCTION_MINIMUM_REMAINING_FRACTION,
+ },
+ "allocation_count": len(allocations),
+ "allocations": allocations,
+ "excluded_count": len(excluded),
+ "excluded_objects": excluded,
+ }
+ return kept, audit
+
+
+def _materialize_derived_delivery_masks(
+ objects: Sequence[Mapping[str, Any]],
+ expected_shape: tuple[int, int],
+) -> dict[str, np.ndarray]:
+ """Rebuild the audited derived masks from immutable source masks."""
+
+ by_source: dict[str, Mapping[str, Any]] = {}
+ raw_masks: dict[str, np.ndarray] = {}
+ for item in objects:
+ source_id = str(item.get("source_object_id") or "")
+ if not source_id or source_id in by_source:
+ raise ValueError("delivery source object IDs must be unique")
+ source_path = Path(str(item.get("source_mask_path") or "")).resolve()
+ if not source_path.is_file():
+ raise FileNotFoundError(source_path)
+ expected_hash = str(item.get("source_mask_sha256") or "")
+ if len(expected_hash) != 64 or _sha256(source_path) != expected_hash:
+ raise ValueError(f"source mask hash mismatch: {source_id}")
+ by_source[source_id] = item
+ raw_masks[source_id] = _load_binary_mask(source_path, expected_shape)
+
+ def priority(item: Mapping[str, Any]) -> int:
+ ownership = item.get("reconstruction_mask_ownership") or {}
+ value = ownership.get("priority_index")
+ if not isinstance(value, int) or value < 0:
+ raise ValueError("delivery ownership priority is invalid")
+ return value
+
+ ordered = sorted(objects, key=priority)
+ derived: dict[str, np.ndarray] = {}
+ for item in ordered:
+ source_id = str(item["source_object_id"])
+ ownership = item.get("reconstruction_mask_ownership") or {}
+ owners = ownership.get("owner_source_object_ids") or []
+ if not isinstance(owners, list):
+ raise ValueError("delivery ownership owners must be a list")
+ mask = raw_masks[source_id].copy()
+ for owner_id in owners:
+ owner_key = str(owner_id)
+ if owner_key not in derived:
+ raise ValueError(
+ f"delivery ownership owner is unavailable: {owner_key}"
+ )
+ mask &= ~derived[owner_key]
+ expected_area = ownership.get("delivery_area_pixels")
+ if not isinstance(expected_area, int) or int(mask.sum()) != expected_area:
+ raise ValueError(
+ f"derived delivery mask area mismatch for {source_id}"
+ )
+ if not bool(mask.any()):
+ raise ValueError(f"derived delivery mask is empty: {source_id}")
+ derived[source_id] = mask
+ return derived
+
+
+def _contains_value(value: Any, target: str) -> bool:
+ if isinstance(value, Mapping):
+ return any(_contains_value(item, target) for item in value.values())
+ if isinstance(value, Sequence) and not isinstance(value, (str, bytes)):
+ return any(_contains_value(item, target) for item in value)
+ return str(value) == target
+
+
+def _object_annotations(
+ quality: Mapping[str, Any], source_object_id: str
+) -> dict[str, list[dict[str, Any]]]:
+ return {
+ "hard_errors": [
+ copy.deepcopy(dict(item))
+ for item in quality.get("hard_errors") or []
+ if isinstance(item, Mapping)
+ and _contains_value(item, source_object_id)
+ ],
+ "warnings": [
+ copy.deepcopy(dict(item))
+ for item in quality.get("warnings") or []
+ if isinstance(item, Mapping)
+ and _contains_value(item, source_object_id)
+ ],
+ }
+
+
+def build_delivery_plan(
+ *,
+ scene_name: str,
+ image_path: str | Path,
+ objects: Sequence[Mapping[str, Any]],
+ quality: Mapping[str, Any],
+) -> dict[str, Any]:
+ """Classify usable objects and assign continuous downstream identities."""
+
+ resolved_image = Path(image_path).expanduser().resolve()
+ infrastructure_errors: list[dict[str, Any]] = []
+ try:
+ with Image.open(resolved_image) as source:
+ source.verify()
+ with Image.open(resolved_image) as source:
+ width, height = source.size
+ except (OSError, ValueError) as exc:
+ width, height = 0, 0
+ infrastructure_errors.append(
+ {
+ "type": "unreadable_source_image",
+ "path": str(resolved_image),
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+
+ usable: list[dict[str, Any]] = []
+ excluded: list[dict[str, Any]] = []
+ for source_index, raw in enumerate(objects):
+ obj = copy.deepcopy(dict(raw))
+ source_id = str(obj.get("object_id") or f"source_{source_index:03d}")
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "success":
+ excluded.append(
+ {
+ "source_object_id": source_id,
+ "reason": "sam3_not_success",
+ "details": sam3.get("reason"),
+ }
+ )
+ continue
+ mask_path = Path(str(sam3.get("mask_path") or "")).expanduser().resolve()
+ if not mask_path.is_file():
+ excluded.append(
+ {
+ "source_object_id": source_id,
+ "reason": "missing_mask_file",
+ "path": str(mask_path),
+ }
+ )
+ continue
+ if width <= 0 or height <= 0:
+ excluded.append(
+ {
+ "source_object_id": source_id,
+ "reason": "source_image_unavailable_for_mask_validation",
+ }
+ )
+ continue
+ try:
+ mask = _load_binary_mask(mask_path, (height, width))
+ except (OSError, ValueError) as exc:
+ excluded.append(
+ {
+ "source_object_id": source_id,
+ "reason": "invalid_mask",
+ "path": str(mask_path),
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ continue
+ usable.append(
+ {
+ "source_index": source_index,
+ "source_object_id": source_id,
+ "source_object": obj,
+ "source_mask_path": str(mask_path),
+ "mask_sha256": _sha256(mask_path),
+ "area_pixels": int(mask.sum()),
+ "mask": mask,
+ "rank": _score(obj),
+ }
+ )
+
+ parents = list(range(len(usable)))
+
+ def find(index: int) -> int:
+ while parents[index] != index:
+ parents[index] = parents[parents[index]]
+ index = parents[index]
+ return index
+
+ def union(left: int, right: int) -> None:
+ left_root = find(left)
+ right_root = find(right)
+ if left_root != right_root:
+ parents[right_root] = left_root
+
+ for left_index, left in enumerate(usable):
+ for right_index in range(left_index + 1, len(usable)):
+ right = usable[right_index]
+ metrics = _mask_metrics(left["mask"], right["mask"])
+ identical_mask = left["mask_sha256"] == right["mask_sha256"]
+ if identical_mask or (
+ int(metrics["intersection_pixels"])
+ >= DUPLICATE_MINIMUM_INTERSECTION_PIXELS
+ and float(metrics["iou"]) >= DUPLICATE_MINIMUM_IOU
+ and float(metrics["left_coverage"])
+ >= DUPLICATE_MINIMUM_COVERAGE
+ and float(metrics["right_coverage"])
+ >= DUPLICATE_MINIMUM_COVERAGE
+ ):
+ union(left_index, right_index)
+
+ components: dict[int, list[dict[str, Any]]] = {}
+ for index, item in enumerate(usable):
+ components.setdefault(find(index), []).append(item)
+ duplicate_collapses: list[dict[str, Any]] = []
+ duplicate_losers: set[str] = set()
+ for group in components.values():
+ if len(group) < 2:
+ continue
+ winner = max(group, key=_duplicate_winner_key)
+ loser_items = [item for item in group if item is not winner]
+ loser_ids = [item["source_object_id"] for item in loser_items]
+ duplicate_losers.update(loser_ids)
+ identical = len({item["mask_sha256"] for item in group}) == 1
+ collapse = {
+ "kind": "identical_mask" if identical else "near_identical_mask",
+ "kept_source_object_id": winner["source_object_id"],
+ "excluded_source_object_ids": loser_ids,
+ "resolution": (
+ "prefer_pass_a_provenance_then_score_then_bbox_iou_then_"
+ "visual_order"
+ ),
+ "pair_metrics": [
+ {
+ "excluded_source_object_id": item["source_object_id"],
+ **{
+ key: round(float(value), 6)
+ for key, value in _mask_metrics(
+ winner["mask"], item["mask"]
+ ).items()
+ },
+ }
+ for item in loser_items
+ ],
+ }
+ if identical:
+ collapse["mask_sha256"] = winner["mask_sha256"]
+ duplicate_collapses.append(collapse)
+ duplicate_collapses.sort(
+ key=lambda item: str(item["kept_source_object_id"])
+ )
+ selected = [
+ item
+ for item in usable
+ if item["source_object_id"] not in duplicate_losers
+ ]
+ selected.sort(key=lambda item: item["source_index"])
+ collapse_by_loser = {
+ source_id: collapse
+ for collapse in duplicate_collapses
+ for source_id in collapse["excluded_source_object_ids"]
+ }
+ for source_id, collapse in collapse_by_loser.items():
+ excluded.append(
+ {
+ "source_object_id": source_id,
+ "reason": (
+ "identical_mask_duplicate_collapsed"
+ if collapse["kind"] == "identical_mask"
+ else "near_identical_mask_duplicate_collapsed"
+ ),
+ "kept_source_object_id": collapse["kept_source_object_id"],
+ }
+ )
+
+ selected, reconstruction_mask_ownership = (
+ _derive_reconstruction_ownership(selected)
+ )
+ excluded.extend(reconstruction_mask_ownership["excluded_objects"])
+
+ delivery_objects = []
+ for order_index, item in enumerate(selected):
+ source = item["source_object"]
+ sam3 = source.get("sam3") or {}
+ object_id = f"object_{order_index + 1:03d}"
+ category = str(source.get("category") or "unknown")
+ description = str(
+ source.get("description")
+ or source.get("instance_name")
+ or source.get("sam3_prompt")
+ or category
+ ).strip()
+ delivery_objects.append(
+ {
+ "order_index": order_index,
+ "object_id": object_id,
+ "sam3d_internal_name": object_id,
+ "source_object_id": item["source_object_id"],
+ "category": category,
+ "description": description,
+ "source_mask_path": item["source_mask_path"],
+ "source_mask_sha256": item["mask_sha256"],
+ # Materialization replaces this with the hash of the derived
+ # Stage 04 mask; it is never a claim about the source mask.
+ "mask_sha256": item["mask_sha256"],
+ "area_pixels": int(item["delivery_mask"].sum()),
+ "source_area_pixels": item["area_pixels"],
+ "reconstruction_mask_ownership": item[
+ "reconstruction_mask_ownership"
+ ],
+ "bbox_2d_norm_xyxy": source.get("bbox_2d_norm_xyxy"),
+ "mask_bbox_abs_xyxy": sam3.get("mask_bbox_abs_xyxy"),
+ "sam3": {
+ "status": "success",
+ "score": sam3.get("score"),
+ "selection_method": sam3.get("selection_method"),
+ },
+ "quality_annotations": _object_annotations(
+ quality, item["source_object_id"]
+ ),
+ }
+ )
+
+ if not delivery_objects:
+ infrastructure_errors.append(
+ {
+ "type": "no_usable_object_masks",
+ "source_object_count": len(objects),
+ "excluded_object_count": len(excluded),
+ }
+ )
+ source_quality_passed = str(quality.get("status") or "") == "pass"
+ advisory_errors = list(quality.get("advisory_errors") or [])
+ if infrastructure_errors:
+ status = "infrastructure_failed"
+ elif not source_quality_passed:
+ status = "quality_failed"
+ elif not excluded and not duplicate_collapses and not advisory_errors:
+ status = "sam3d_ready"
+ else:
+ status = "sam3d_ready_degraded"
+
+ return {
+ "schema_version": "scenemosaic_sam3d_delivery_plan",
+ "scene_name": scene_name,
+ "status": status,
+ "sam3d_execution_allowed": (
+ source_quality_passed and status in READY_STATUSES
+ ),
+ "source_quality_status": quality.get("status"),
+ "source_hard_error_count": len(quality.get("hard_errors") or []),
+ "source_advisory_error_count": len(advisory_errors),
+ "source_warning_count": len(quality.get("warnings") or []),
+ "source_object_count": len(objects),
+ "usable_source_object_count": len(usable),
+ "delivery_object_count": len(delivery_objects),
+ "excluded_objects": excluded,
+ "duplicate_collapses": duplicate_collapses,
+ "duplicate_collapse_thresholds": {
+ "minimum_intersection_pixels": (
+ DUPLICATE_MINIMUM_INTERSECTION_PIXELS
+ ),
+ "minimum_iou": DUPLICATE_MINIMUM_IOU,
+ "minimum_coverage": DUPLICATE_MINIMUM_COVERAGE,
+ },
+ "reconstruction_mask_ownership": reconstruction_mask_ownership,
+ "infrastructure_errors": infrastructure_errors,
+ "image_path": str(resolved_image),
+ "image_size": [width, height] if width and height else None,
+ "objects": delivery_objects,
+ }
+
+
+def materialize_delivery_package(
+ *,
+ publish_dir: str | Path,
+ output_name: str = "sam3d_input",
+) -> dict[str, Any]:
+ """Build an atomic, standard SAM3D package inside a published result."""
+
+ published = Path(publish_dir).expanduser().resolve()
+ registry_path = published / "sam3_objects.json"
+ quality_path = published / "quality_report.json"
+ if not registry_path.is_file() or not quality_path.is_file():
+ raise FileNotFoundError(
+ f"published result lacks registry or quality report: {published}"
+ )
+ registry = json.loads(registry_path.read_text(encoding="utf-8"))
+ quality = json.loads(quality_path.read_text(encoding="utf-8"))
+ plan = build_delivery_plan(
+ scene_name=str(registry.get("scene_name") or published.parent.name),
+ image_path=str(registry.get("image_path") or ""),
+ objects=registry.get("objects") or [],
+ quality=quality,
+ )
+
+ output_dir = published / output_name
+ temporary = published / f".{output_name}.tmp"
+ if output_dir.exists():
+ raise FileExistsError(f"SAM3D delivery already exists: {output_dir}")
+ if temporary.exists():
+ raise FileExistsError(f"stale SAM3D delivery staging exists: {temporary}")
+ temporary.mkdir()
+ masks_dir = temporary / "masks"
+ masks_dir.mkdir()
+ final_masks_dir = output_dir / "masks"
+ image_output = output_dir / "image.png"
+
+ delivery_masks: dict[str, np.ndarray] = {}
+ if not plan["infrastructure_errors"]:
+ shutil.copy2(plan["image_path"], temporary / "image.png")
+ with Image.open(temporary / "image.png") as source_image:
+ expected_shape = (source_image.height, source_image.width)
+ delivery_masks = _materialize_derived_delivery_masks(
+ plan["objects"], expected_shape
+ )
+ manifest_objects = []
+ for item in plan["objects"]:
+ copied = copy.deepcopy(item)
+ filename = (
+ f"{item['order_index']:03d}_{item['object_id']}_"
+ f"{_safe_name(item['category'])}_mask.png"
+ )
+ mask_path = masks_dir / filename
+ mask = delivery_masks[str(item["source_object_id"])]
+ Image.fromarray(mask.astype(np.uint8) * 255).save(mask_path)
+ delivery_hash = _sha256(mask_path)
+ ownership = copied.get("reconstruction_mask_ownership") or {}
+ copied["mask_mode"] = str(ownership.get("mode") or "copy")
+ copied["mask_path"] = str(final_masks_dir / filename)
+ copied["mask_sha256"] = delivery_hash
+ item["mask_sha256"] = delivery_hash
+ item["delivery_mask_path"] = str(final_masks_dir / filename)
+ manifest_objects.append(copied)
+
+ manifest = {
+ "schema_version": "scenemosaic_sam3d_input_manifest",
+ "scene_name": plan["scene_name"],
+ "stage": "sam3d_input",
+ "delivery_status": plan["status"],
+ "sam3d_execution_allowed": plan["sam3d_execution_allowed"],
+ "mask_mode": "derived_reconstruction_ownership",
+ "image_path": str(image_output),
+ "source_image_path": plan["image_path"],
+ "source_registry": str(registry_path),
+ "source_quality_report": str(quality_path),
+ "object_count": len(manifest_objects),
+ "objects": manifest_objects,
+ }
+ run_info = {
+ "schema_version": "scenemosaic_sam3d_delivery_run_info",
+ "scene_name": plan["scene_name"],
+ "status": plan["status"],
+ "sam3d_execution_allowed": plan["sam3d_execution_allowed"],
+ "object_count": len(manifest_objects),
+ "created_at": _utc_now(),
+ "source_quality_status": plan["source_quality_status"],
+ "excluded_object_count": len(plan["excluded_objects"]),
+ "duplicate_collapse_count": len(plan["duplicate_collapses"]),
+ "reconstruction_mask_ownership_allocation_count": (
+ plan["reconstruction_mask_ownership"]["allocation_count"]
+ ),
+ "reconstruction_mask_ownership_excluded_count": (
+ plan["reconstruction_mask_ownership"]["excluded_count"]
+ ),
+ "infrastructure_error_count": len(plan["infrastructure_errors"]),
+ }
+ scene_config = {
+ "schema_version": "scenemosaic_sam3d_delivery_scene_config",
+ "scene_name": plan["scene_name"],
+ "run_root": str(published),
+ "input_image": str(image_output),
+ "source_input_image": plan["image_path"],
+ "object_id_base": 1,
+ "object_id_format": "object_{index:03d}",
+ "conda_envs": {
+ "sam3": os.environ.get("SCENEMOSAIC_SAM3_ENV", "sam3"),
+ "sam3d": "sam3d-objects",
+ },
+ "stage_envs": {
+ output_name: "sam3",
+ "sam3d_output": "sam3d",
+ },
+ "stages": {
+ "sam3d_input": output_name,
+ "sam3d_output": "sam3d_output",
+ },
+ "delivery_status": plan["status"],
+ "sam3d_execution_allowed": plan["sam3d_execution_allowed"],
+ "created_at_utc": _utc_now(),
+ }
+ _atomic_json(temporary / "sam3d_input_manifest.json", manifest)
+ _atomic_json(temporary / "sam3d_input_run_info.json", run_info)
+ _atomic_json(temporary / "delivery_report.json", plan)
+ _atomic_json(temporary / "sam3d_scene_config.json", scene_config)
+
+ expected_ids = [
+ f"object_{index + 1:03d}" for index in range(len(manifest_objects))
+ ]
+ actual_ids = [item["object_id"] for item in manifest_objects]
+ if actual_ids != expected_ids:
+ raise AssertionError(
+ f"non-continuous delivery identities: {actual_ids} != {expected_ids}"
+ )
+ if plan["sam3d_execution_allowed"]:
+ if not (temporary / "image.png").is_file() or not manifest_objects:
+ raise AssertionError("ready delivery lacks image or objects")
+ if len(list(masks_dir.glob("*.png"))) != len(manifest_objects):
+ raise AssertionError("delivery mask count does not match manifest")
+ if any(not item["description"] for item in manifest_objects):
+ raise AssertionError("delivery object has empty description")
+ os.replace(temporary, output_dir)
+ return {
+ "status": plan["status"],
+ "sam3d_execution_allowed": plan["sam3d_execution_allowed"],
+ "object_count": len(manifest_objects),
+ "excluded_object_count": len(plan["excluded_objects"]),
+ "duplicate_collapse_count": len(plan["duplicate_collapses"]),
+ "reconstruction_mask_ownership_allocation_count": (
+ plan["reconstruction_mask_ownership"]["allocation_count"]
+ ),
+ "reconstruction_mask_ownership_excluded_count": (
+ plan["reconstruction_mask_ownership"]["excluded_count"]
+ ),
+ "output_dir": str(output_dir),
+ "manifest_path": str(output_dir / "sam3d_input_manifest.json"),
+ "scene_config_path": str(output_dir / "sam3d_scene_config.json"),
+ "delivery_report_path": str(output_dir / "delivery_report.json"),
+ }
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/__init__.py b/vlm_agent/reconstruction/perception/pipeline/refinements/__init__.py
new file mode 100644
index 0000000..01c3695
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/__init__.py
@@ -0,0 +1 @@
+"""Deterministic perception refinement algorithms."""
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/ambiguous_same_prompt_component_completion.py b/vlm_agent/reconstruction/perception/pipeline/refinements/ambiguous_same_prompt_component_completion.py
new file mode 100644
index 0000000..38baaf1
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/ambiguous_same_prompt_component_completion.py
@@ -0,0 +1,284 @@
+"""Fail-closed completion of an already ambiguous same-prompt SAM3 component.
+
+SAM3-A can deliberately defer a touching candidate when it has negligible IoU
+with the assigned main mask. If both candidates came from the *same exact
+semantic prompt*, the deferred candidate is retained locally, lies inside the
+same VLM bbox, touches the main mask and does not leak into another final
+object, it is safe to union as a completion. The resulting visible mask may
+have several disconnected components because of occlusion; requiring a single
+pixel-connected component would incorrectly reject real objects such as a
+pottery wheel with separated legs. If no candidate satisfies every safety
+condition, the existing successful mask remains unchanged and the completion is
+recorded as a no-op.
+"""
+
+from __future__ import annotations
+
+import copy
+import json
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+from scipy import ndimage
+
+
+@dataclass(frozen=True)
+class AmbiguousSamePromptCompletionConfig:
+ enabled: bool = True
+ isolated_attempt_only: bool = True
+ minimum_candidate_score: float = 0.60
+ maximum_candidate_overlap_with_other: float = 0.15
+ contact_dilation_pixels: int = 1
+ maximum_candidate_area_ratio: float = 1.00
+ minimum_existing_owner_coverage: float = 0.99
+
+
+def _mask(path_value: Any, shape: tuple[int, int]) -> np.ndarray:
+ path = Path(str(path_value or '')).expanduser().resolve()
+ if not path.is_file():
+ raise FileNotFoundError(path)
+ result = np.asarray(Image.open(path).convert('L')) > 0
+ if result.shape != shape or not result.any():
+ raise ValueError(f'invalid mask: {path}')
+ return result
+
+
+def _bbox(mask: np.ndarray) -> list[int]:
+ ys, xs = np.nonzero(mask)
+ if not len(xs):
+ raise ValueError('empty mask')
+ return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
+
+
+def _norm_bbox(raw: Any, width: int, height: int) -> list[int] | None:
+ try:
+ x1, y1, x2, y2 = (float(value) for value in raw)
+ except (TypeError, ValueError):
+ return None
+ if not (0 <= x1 < x2 <= 1 and 0 <= y1 < y2 <= 1):
+ return None
+ return [round(x1 * width), round(y1 * height), round(x2 * width), round(y2 * height)]
+
+
+def _center_in(box: Sequence[int], target: Sequence[int]) -> bool:
+ cx = (int(box[0]) + int(box[2])) / 2
+ cy = (int(box[1]) + int(box[3])) / 2
+ return int(target[0]) <= cx <= int(target[2]) and int(target[1]) <= cy <= int(target[3])
+
+
+def _touches(left: np.ndarray, right: np.ndarray, pixels: int) -> bool:
+ return bool(np.logical_and(
+ ndimage.binary_dilation(left, structure=np.ones((3, 3), bool), iterations=max(0, pixels)),
+ right,
+ ).any())
+
+
+def _one_component(mask: np.ndarray) -> bool:
+ _, count = ndimage.label(mask, structure=np.ones((3, 3), bool))
+ return int(count) == 1
+
+
+def _sources(obj: Mapping[str, Any], pass_name: str) -> list[str]:
+ return [
+ str(row.get('source_id') or '')
+ for row in (obj.get('proposal_sources') or [])
+ if isinstance(row, Mapping) and str(row.get('pass') or '').upper() == pass_name
+ ]
+
+
+def _save(path: Path, mask: np.ndarray, image: np.ndarray) -> tuple[str, str]:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ Image.fromarray(mask.astype(np.uint8) * 255).save(path)
+ overlay = image.copy()
+ overlay[mask] = (0.55 * overlay[mask] + 0.45 * np.array((76, 175, 80), dtype=np.uint8)).astype(np.uint8)
+ overlay_path = path.parent.parent / 'overlays' / path.name.replace('_mask.png', '_overlay.png')
+ overlay_path.parent.mkdir(parents=True, exist_ok=True)
+ Image.fromarray(overlay).save(overlay_path)
+ return str(path), str(overlay_path)
+
+
+def run_ambiguous_same_prompt_component_completion(
+ *,
+ image_path: str | Path,
+ objects: Sequence[Mapping[str, Any]],
+ sam3_a_registry: Mapping[str, Any],
+ staging_dir: str | Path,
+ work_dir: str | Path,
+ config: AmbiguousSamePromptCompletionConfig | None = None,
+) -> dict[str, Any]:
+ cfg = config or AmbiguousSamePromptCompletionConfig()
+ if not cfg.isolated_attempt_only:
+ raise ValueError('ambiguous same-prompt completion is isolated-attempt only')
+ out = Path(work_dir).expanduser().resolve()
+ if out.exists() and any(out.iterdir()):
+ raise FileExistsError(f'completion work directory is not empty: {out}')
+ out.mkdir(parents=True)
+ image = np.asarray(Image.open(Path(image_path).expanduser().resolve()).convert('RGB'))
+ shape = image.shape[:2]
+ height, width = shape
+ copied = [copy.deepcopy(dict(item)) for item in objects]
+ source_by_id = {
+ str(item.get('object_id') or ''): dict(item)
+ for item in (sam3_a_registry.get('objects') or [])
+ if isinstance(item, Mapping)
+ }
+ candidates = {
+ str(item.get('candidate_id') or ''): dict(item)
+ for item in (sam3_a_registry.get('unassigned_sam3_candidates') or [])
+ if isinstance(item, Mapping)
+ }
+ masks: dict[str, np.ndarray] = {}
+ for obj in copied:
+ if (obj.get('sam3') or {}).get('status') != 'success':
+ continue
+ try:
+ masks[str(obj.get('object_id') or '')] = _mask((obj.get('sam3') or {}).get('mask_path'), shape)
+ except (OSError, ValueError):
+ pass
+
+ audit: dict[str, Any] = {
+ 'schema_version': 'scenemosaic_ambiguous_same_prompt_component_completion',
+ 'config': asdict(cfg),
+ 'target_count': 0,
+ 'completed': [],
+ 'unresolved': [],
+ 'skipped': [],
+ 'hard_errors': [],
+ 'additional_vlm_calls': 0,
+ 'formal_outputs_modified': False,
+ }
+ for obj in copied:
+ object_id = str(obj.get('object_id') or '')
+ source_ids = [item for item in _sources(obj, 'A') if item in source_by_id]
+ if len(source_ids) != 1 or object_id not in masks:
+ continue
+ source = source_by_id[source_ids[0]]
+ source_sam3 = source.get('sam3') or {}
+ review = source_sam3.get('merge_review') or {}
+ ambiguous_ids = [str(value) for value in (review.get('ambiguous_candidate_ids') or [])]
+ # A later review may mark the primary selection as resolved while
+ # retaining the original ambiguous IDs. Their geometry is still
+ # evidence for this deterministic completion stage, so do not discard
+ # them solely because that bookkeeping flag became false.
+ if not ambiguous_ids:
+ continue
+ audit['target_count'] += 1
+ target_bbox = _norm_bbox(source.get('bbox_2d_norm_xyxy'), width, height)
+ base = masks[object_id]
+ if target_bbox is None:
+ audit['unresolved'].append({'object_id': object_id, 'reason': 'invalid_source_bbox'})
+ continue
+ viable: list[dict[str, Any]] = []
+ explained_by_existing: list[dict[str, Any]] = []
+ unresolved_candidate_seen = False
+ for candidate_id in ambiguous_ids:
+ raw = candidates.get(candidate_id)
+ if raw is None:
+ continue
+ if str(raw.get('sam3_prompt') or '') != str(source.get('sam3_prompt') or ''):
+ continue
+ if float(raw.get('score') or 0.0) < cfg.minimum_candidate_score:
+ continue
+ try:
+ candidate = _mask(raw.get('mask_path'), shape)
+ except (OSError, ValueError):
+ continue
+ box = _bbox(candidate)
+ existing_owners: list[dict[str, Any]] = []
+ for other_id, other_mask in masks.items():
+ if other_id == object_id:
+ continue
+ coverage = int(np.logical_and(candidate, other_mask).sum()) / max(
+ 1, int(candidate.sum())
+ )
+ if coverage >= cfg.minimum_existing_owner_coverage:
+ existing_owners.append(
+ {
+ "object_id": other_id,
+ "candidate_coverage": round(coverage, 6),
+ }
+ )
+ # A deferred same-prompt mask can describe an already-declared
+ # neighbouring instance. It is evidence of neither a missing
+ # component nor a semantic merge when an existing mask completely
+ # owns it, so preserve both objects and record the explanation.
+ if existing_owners:
+ explained_by_existing.append(
+ {
+ "candidate_id": candidate_id,
+ "existing_owners": existing_owners,
+ }
+ )
+ continue
+ unresolved_candidate_seen = True
+ if not _center_in(box, target_bbox) or not _touches(
+ base, candidate, cfg.contact_dilation_pixels
+ ):
+ continue
+ if int(candidate.sum()) > int(base.sum()) * cfg.maximum_candidate_area_ratio:
+ continue
+ conflicts: list[dict[str, Any]] = []
+ for other_id, other_mask in masks.items():
+ if other_id == object_id:
+ continue
+ overlap = int(np.logical_and(candidate, other_mask).sum())
+ if overlap / max(1, int(candidate.sum())) > cfg.maximum_candidate_overlap_with_other:
+ conflicts.append({'object_id': other_id, 'candidate_coverage': round(overlap / int(candidate.sum()), 6)})
+ if conflicts:
+ continue
+ union = base | candidate
+ viable.append({'candidate_id': candidate_id, 'mask': candidate, 'score': float(raw.get('score') or 0.0), 'union': union})
+ if not viable:
+ if explained_by_existing and not unresolved_candidate_seen:
+ audit['skipped'].append(
+ {
+ "object_id": object_id,
+ "source_object_id": source_ids[0],
+ "reason": "same_prompt_candidate_already_owned_by_existing_object",
+ "explained_candidates": explained_by_existing,
+ }
+ )
+ continue
+ # This refinement is optional: the lack of a safe extra component
+ # is not evidence that the already successful owner mask is invalid.
+ # Preserve it and retain the deferred candidates as an auditable no-op.
+ audit['skipped'].append(
+ {
+ 'object_id': object_id,
+ 'source_object_id': source_ids[0],
+ 'reason': 'no_safe_touching_same_prompt_component',
+ 'ambiguous_candidate_ids': ambiguous_ids,
+ }
+ )
+ continue
+ selected = max(viable, key=lambda row: (row['score'], int(row['mask'].sum()), row['candidate_id']))
+ output = Path(staging_dir).expanduser().resolve() / 'masks' / f"{object_id}_{str(obj.get('category') or 'object')}_mask.png"
+ mask_path, overlay_path = _save(output, selected['union'], image)
+ obj['sam3'] = copy.deepcopy(dict(obj.get('sam3') or {}))
+ obj['sam3'].update({
+ 'mask_path': mask_path,
+ 'overlay_path': overlay_path,
+ 'area_pixels': int(selected['union'].sum()),
+ 'mask_bbox_abs_xyxy': _bbox(selected['union']),
+ 'selection_method': 'ambiguous_same_prompt_component_completion',
+ 'ambiguous_same_prompt_completion': {
+ 'source_object_id': source_ids[0],
+ 'candidate_id': selected['candidate_id'],
+ 'candidate_score': selected['score'],
+ 'previous_area_pixels': int(base.sum()),
+ 'completed_area_pixels': int(selected['union'].sum()),
+ },
+ })
+ masks[object_id] = selected['union']
+ audit['completed'].append({'object_id': object_id, **obj['sam3']['ambiguous_same_prompt_completion']})
+ if audit['unresolved']:
+ audit['hard_errors'] = [{'type': 'ambiguous_same_prompt_component_unresolved', **item} for item in audit['unresolved']]
+ audit['status'] = 'pass' if not audit['hard_errors'] else 'blocked'
+ (out / 'ambiguous_same_prompt_component_completion_audit.json').write_text(json.dumps(audit, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
+ return {'objects': copied, 'audit': audit}
+
+
+__all__ = ['AmbiguousSamePromptCompletionConfig', 'run_ambiguous_same_prompt_component_completion']
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/bbox_failed_mask_rescue.py b/vlm_agent/reconstruction/perception/pipeline/refinements/bbox_failed_mask_rescue.py
new file mode 100644
index 0000000..8497db9
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/bbox_failed_mask_rescue.py
@@ -0,0 +1,1191 @@
+#!/usr/bin/env python3
+"""Bounded bbox-first spatial rescue for VLM objects whose text mask failed."""
+
+from __future__ import annotations
+
+import copy
+import json
+import math
+import os
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Callable, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+from scipy import ndimage
+
+
+@dataclass(frozen=True)
+class BboxFailedMaskRescueConfig:
+ enabled: bool = True
+ isolated_attempt_only: bool = True
+ maximum_targets: int = 5
+ maximum_candidates_per_target: int = 4
+ maximum_recessed_candidates_per_target: int = 12
+ minimum_score: float = 0.20
+ minimum_recessed_ceiling_light_score: float = 0.45
+ minimum_target_bbox_coverage: float = 0.35
+ minimum_candidate_bbox_coverage: float = 0.50
+ minimum_mask_inside_expanded_roi: float = 0.90
+ minimum_bbox_area_ratio: float = 0.25
+ maximum_bbox_area_ratio: float = 3.00
+ minimum_rank_margin: float = 0.15
+ roi_expansion_fraction: float = 0.35
+ maximum_existing_iou: float = 0.35
+ maximum_existing_candidate_coverage: float = 0.50
+ minimum_conflict_pixels: int = 30
+ minimum_successful_replacement_foreign_overlap_pixels: int = 64
+ minimum_successful_replacement_other_object_coverage: float = 0.05
+ # Bbox rescue is a fallback for failed or missing masks only. A valid
+ # successful SAM3 mask is never replaced merely because its extent differs
+ # from a VLM box.
+ repair_successful_low_extent: bool = False
+ maximum_successful_vlm_bbox_iou: float = 0.35
+ minimum_successful_target_bbox_coverage: float = 0.75
+ minimum_successful_replacement_score_gain: float = 0.02
+ minimum_successful_replacement_target_coverage_gain: float = 0.05
+ repair_successful_low_extent_lighting: bool = False
+ repair_failed_lighting: bool = False
+ repair_failed_recessed_ceiling_lights: bool = True
+ # Source IDs shown in Pass-B's bounded evidence panel. These are the
+ # only successful-low-extent objects whose declared completeness has been
+ # explicitly reviewed by the VLM, so they outrank incidental loose boxes.
+ priority_source_object_ids: frozenset[str] = frozenset()
+ reject_disconnected_generic_successful_replacement: bool = True
+ disconnected_component_min_area_fraction: float = 0.20
+ disconnected_component_min_pixels: int = 128
+ prefer_compact_nested_isolated_vase: bool = True
+ compact_vase_minimum_containment: float = 0.95
+ compact_vase_minimum_area_ratio: float = 0.75
+ compact_vase_minimum_target_coverage: float = 0.50
+ compact_vase_maximum_width_ratio: float = 0.85
+ compact_vase_minimum_height_ratio: float = 0.80
+ compact_vase_minimum_removed_pixels: int = 128
+ preserve_compact_isolated_vase_success: bool = True
+ compact_vase_minimum_existing_bbox_iou: float = 0.45
+ compact_vase_minimum_existing_target_coverage: float = 0.50
+
+
+ELIGIBLE_REASONS = {
+ "no_candidates_from_text_prompt",
+ "no_local_candidate",
+}
+GENERIC_TYPES = {
+ "",
+ "unknown",
+ "object",
+ "thing",
+}
+
+
+def normalize_prompt(value: Any) -> str:
+ return " ".join(re.sub(r"[^a-z0-9]+", " ", str(value or "").lower()).split())
+
+
+def is_lighting_object(obj: Mapping[str, Any]) -> bool:
+ tokens = set(
+ (
+ normalize_prompt(obj.get("category"))
+ + " "
+ + normalize_prompt(obj.get("object_type"))
+ ).split()
+ )
+ return bool(tokens & {"light", "lighting", "lamp", "downlight"})
+
+
+def is_recessed_ceiling_light(obj: Mapping[str, Any]) -> bool:
+ tokens = set(
+ (
+ normalize_prompt(obj.get("category"))
+ + " "
+ + normalize_prompt(obj.get("sam3_prompt"))
+ + " "
+ + normalize_prompt(obj.get("description"))
+ ).split()
+ )
+ return "recessed" in tokens and "ceiling" in tokens and "light" in tokens
+
+
+def _vase_tokens(obj: Mapping[str, Any]) -> set[str]:
+ return set(
+ (
+ normalize_prompt(obj.get("category"))
+ + " "
+ + normalize_prompt(obj.get("sam3_prompt"))
+ + " "
+ + normalize_prompt(obj.get("description"))
+ ).split()
+ )
+
+
+def is_vase_botanical_assembly(obj: Mapping[str, Any]) -> bool:
+ """Whether one declaration names both a vase and its botanical content."""
+
+ tokens = _vase_tokens(obj)
+ return "vase" in tokens and bool(
+ tokens
+ & {
+ "branch",
+ "branches",
+ "flower",
+ "flowers",
+ "floral",
+ "foliage",
+ "greenery",
+ "plant",
+ "arrangement",
+ }
+ )
+
+
+def is_isolated_vase(obj: Mapping[str, Any]) -> bool:
+ """Return true only for a vase declaration without botanical contents."""
+
+ return "vase" in _vase_tokens(obj) and not is_vase_botanical_assembly(obj)
+
+
+def _bbox_axis_gap(left: int, right: int, other_left: int, other_right: int) -> int:
+ return max(0, other_left - right, left - other_right)
+
+
+def rescue_prompt(obj: Mapping[str, Any]) -> str | None:
+ """Prefer a broad VLM object type so a possibly wrong category is not repeated."""
+
+ if is_recessed_ceiling_light(obj):
+ return "recessed ceiling light"
+ if is_vase_botanical_assembly(obj):
+ # Preserve both the vessel and botanical terms. This is intentionally
+ # not an ``only`` prompt: the declared instance is one assembly.
+ return (
+ normalize_prompt(obj.get("category"))
+ or normalize_prompt(obj.get("description"))
+ or normalize_prompt(obj.get("sam3_prompt"))
+ )
+ if is_isolated_vase(obj):
+ description = (
+ normalize_prompt(obj.get("description"))
+ or normalize_prompt(obj.get("sam3_prompt"))
+ or normalize_prompt(obj.get("category"))
+ )
+ return description if description.endswith(" only") else f"{description} only"
+ object_type = normalize_prompt(obj.get("object_type"))
+ if object_type not in GENERIC_TYPES:
+ return object_type
+ category = normalize_prompt(obj.get("category"))
+ return category or None
+
+
+def is_broad_object_type_prompt(obj: Mapping[str, Any], prompt: str) -> bool:
+ """Whether rescue fell back from an instance prompt to a generic type."""
+
+ object_type = normalize_prompt(obj.get("object_type"))
+ category = normalize_prompt(obj.get("category"))
+ return (
+ object_type not in GENERIC_TYPES
+ and normalize_prompt(prompt) == object_type
+ and object_type != category
+ )
+
+
+def disconnected_generic_replacement_evidence(
+ candidate: np.ndarray,
+ existing: np.ndarray,
+ cfg: BboxFailedMaskRescueConfig,
+) -> dict[str, Any] | None:
+ """Detect a second ordinary instance in a broad-prompt rescue candidate.
+
+ This guard is deliberately restricted to a successful existing mask plus a
+ generic object-type fallback. It does not treat ordinary disconnected parts
+ of a known functional assembly as evidence to reject a specific semantic
+ prompt.
+ """
+
+ labels, count = ndimage.label(
+ np.asarray(candidate, dtype=bool),
+ structure=np.ones((3, 3), dtype=bool),
+ )
+ if int(count) < 2:
+ return None
+ candidate_area = int(candidate.sum())
+ existing_area = max(1, int(existing.sum()))
+ rows: list[dict[str, Any]] = []
+ for index in range(1, int(count) + 1):
+ component = labels == index
+ area = int(component.sum())
+ overlap = int(np.logical_and(component, existing).sum())
+ rows.append({
+ "component_index": index,
+ "area_pixels": area,
+ "candidate_area_fraction": round(area / max(1, candidate_area), 6),
+ "existing_coverage": round(overlap / existing_area, 6),
+ "existing_component_coverage": round(overlap / max(1, area), 6),
+ })
+ matched = max(rows, key=lambda item: float(item["existing_coverage"]))
+ if float(matched["existing_coverage"]) < 0.35:
+ return None
+ minimum_area = max(
+ int(cfg.disconnected_component_min_pixels),
+ int(round(candidate_area * cfg.disconnected_component_min_area_fraction)),
+ )
+ foreign = [
+ item
+ for item in rows
+ if item["component_index"] != matched["component_index"]
+ and int(item["area_pixels"]) >= minimum_area
+ and float(item["existing_component_coverage"]) < 0.10
+ ]
+ if not foreign:
+ return None
+ return {
+ "matched_existing_component": matched,
+ "foreign_components": foreign,
+ "minimum_foreign_area_pixels": minimum_area,
+ }
+
+
+def norm_bbox_to_abs(raw: Any, width: int, height: int) -> list[int] | None:
+ try:
+ values = [float(value) for value in raw]
+ except (TypeError, ValueError):
+ return None
+ if (
+ len(values) != 4
+ or not all(math.isfinite(value) for value in values)
+ or not (0.0 <= values[0] < values[2] <= 1.0)
+ or not (0.0 <= values[1] < values[3] <= 1.0)
+ ):
+ return None
+ result = [
+ int(round(values[0] * width)),
+ int(round(values[1] * height)),
+ int(round(values[2] * width)),
+ int(round(values[3] * height)),
+ ]
+ result[0] = max(0, min(width - 1, result[0]))
+ result[1] = max(0, min(height - 1, result[1]))
+ result[2] = max(result[0] + 1, min(width, result[2]))
+ result[3] = max(result[1] + 1, min(height, result[3]))
+ return result
+
+
+def expand_bbox(box: Sequence[int], width: int, height: int, fraction: float) -> list[int]:
+ x1, y1, x2, y2 = [int(value) for value in box]
+ px = int(round((x2 - x1) * fraction))
+ py = int(round((y2 - y1) * fraction))
+ return [max(0, x1 - px), max(0, y1 - py), min(width, x2 + px), min(height, y2 + py)]
+
+
+def bbox_metrics(target: Sequence[int], candidate: Sequence[int]) -> dict[str, float]:
+ tx1, ty1, tx2, ty2 = [int(value) for value in target]
+ cx1, cy1, cx2, cy2 = [int(value) for value in candidate]
+ ix1, iy1, ix2, iy2 = max(tx1, cx1), max(ty1, cy1), min(tx2, cx2), min(ty2, cy2)
+ intersection = max(0, ix2 - ix1) * max(0, iy2 - iy1)
+ target_area = max(1, (tx2 - tx1) * (ty2 - ty1))
+ candidate_area = max(1, (cx2 - cx1) * (cy2 - cy1))
+ union = target_area + candidate_area - intersection
+ return {
+ "iou": intersection / union if union else 0.0,
+ "target_coverage": intersection / target_area,
+ "candidate_coverage": intersection / candidate_area,
+ "bbox_area_ratio": candidate_area / target_area,
+ }
+
+
+def candidate_evidence(
+ obj: Mapping[str, Any],
+ mask: np.ndarray,
+ score: float,
+ target_bbox: Sequence[int],
+ cfg: BboxFailedMaskRescueConfig,
+) -> dict[str, Any]:
+ height, width = mask.shape
+ ys, xs = np.nonzero(mask)
+ if not len(xs):
+ return {"status": "reject", "reason": "empty_mask"}
+ candidate_bbox = [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
+ metrics = bbox_metrics(target_bbox, candidate_bbox)
+ expanded = expand_bbox(target_bbox, width, height, cfg.roi_expansion_fraction)
+ inside = int(mask[expanded[1] : expanded[3], expanded[0] : expanded[2]].sum())
+ mask_inside_roi = inside / max(1, int(mask.sum()))
+ anchor_in_mask = False
+ anchor = obj.get("anchor_2d_norm_xy")
+ if isinstance(anchor, (list, tuple)) and len(anchor) == 2:
+ try:
+ ax = max(0, min(width - 1, int(round(float(anchor[0]) * width))))
+ ay = max(0, min(height - 1, int(round(float(anchor[1]) * height))))
+ anchor_in_mask = bool(mask[ay, ax])
+ except (TypeError, ValueError):
+ anchor_in_mask = False
+ strong_bbox_without_anchor = (
+ (obj.get("sam3") or {}).get("status") == "success"
+ and score >= 0.85
+ and metrics["target_coverage"] >= 0.75
+ and metrics["candidate_coverage"] >= 0.75
+ and mask_inside_roi >= cfg.minimum_mask_inside_expanded_roi
+ and cfg.minimum_bbox_area_ratio <= metrics["bbox_area_ratio"] <= cfg.maximum_bbox_area_ratio
+ )
+ target_width = max(1, int(target_bbox[2]) - int(target_bbox[0]))
+ target_height = max(1, int(target_bbox[3]) - int(target_bbox[1]))
+ nearby_recessed_ceiling_light = (
+ is_recessed_ceiling_light(obj)
+ and score >= 0.65
+ and _bbox_axis_gap(
+ int(target_bbox[0]), int(target_bbox[2]),
+ candidate_bbox[0], candidate_bbox[2],
+ ) <= max(6, int(round(0.75 * target_width)))
+ and _bbox_axis_gap(
+ int(target_bbox[1]), int(target_bbox[3]),
+ candidate_bbox[1], candidate_bbox[3],
+ ) <= max(4, int(round(0.75 * target_height)))
+ and 0.20 <= metrics["bbox_area_ratio"] <= 3.00
+ )
+ # A tiny recessed light often occupies only the bright center inside an
+ # intentionally loose VLM box. It is safe to recover only when the local
+ # mask is almost wholly contained by a *micro* target box; this does not
+ # relax generic lighting or permit a nearby row member to substitute.
+ micro_contained_recessed_ceiling_light = (
+ is_recessed_ceiling_light(obj)
+ and score >= cfg.minimum_score
+ and target_width * target_height <= 400
+ and metrics["target_coverage"] >= 0.15
+ and metrics["candidate_coverage"] >= 0.95
+ and mask_inside_roi >= cfg.minimum_mask_inside_expanded_roi
+ and 0.12 <= metrics["bbox_area_ratio"] <= 0.35
+ )
+ recessed_score_ok = (
+ not is_recessed_ceiling_light(obj)
+ or score >= cfg.minimum_recessed_ceiling_light_score
+ or nearby_recessed_ceiling_light
+ or micro_contained_recessed_ceiling_light
+ )
+ accepted = (
+ score >= cfg.minimum_score
+ and recessed_score_ok
+ and (
+ (
+ anchor_in_mask
+ and metrics["target_coverage"] >= cfg.minimum_target_bbox_coverage
+ and metrics["candidate_coverage"] >= cfg.minimum_candidate_bbox_coverage
+ and mask_inside_roi >= cfg.minimum_mask_inside_expanded_roi
+ and cfg.minimum_bbox_area_ratio <= metrics["bbox_area_ratio"] <= cfg.maximum_bbox_area_ratio
+ )
+ or strong_bbox_without_anchor
+ or nearby_recessed_ceiling_light
+ or micro_contained_recessed_ceiling_light
+ )
+ )
+ rank = (
+ 2.0 * metrics["iou"]
+ + 1.5 * metrics["target_coverage"]
+ + metrics["candidate_coverage"]
+ + mask_inside_roi
+ + 0.25 * score
+ )
+ return {
+ "status": "accept" if accepted else "reject",
+ "reason": (
+ None
+ if accepted
+ else (
+ "recessed_semantic_score_below_threshold"
+ if not recessed_score_ok
+ else "spatial_gate_failed"
+ )
+ ),
+ "target_bbox_abs_xyxy": list(target_bbox),
+ "candidate_bbox_abs_xyxy": candidate_bbox,
+ "score": round(float(score), 6),
+ "anchor_in_mask": anchor_in_mask,
+ "strong_bbox_without_anchor": strong_bbox_without_anchor,
+ "nearby_recessed_ceiling_light": nearby_recessed_ceiling_light,
+ "micro_contained_recessed_ceiling_light": micro_contained_recessed_ceiling_light,
+ "mask_inside_expanded_roi": round(mask_inside_roi, 6),
+ "rank": round(rank, 6),
+ **{key: round(value, 6) for key, value in metrics.items()},
+ }
+
+
+def overlap_conflict(
+ candidate: np.ndarray,
+ existing: np.ndarray,
+ cfg: BboxFailedMaskRescueConfig,
+) -> dict[str, Any] | None:
+ intersection = int(np.logical_and(candidate, existing).sum())
+ if intersection < cfg.minimum_conflict_pixels:
+ return None
+ ca, ea = int(candidate.sum()), int(existing.sum())
+ union = ca + ea - intersection
+ iou = intersection / union if union else 0.0
+ coverage = intersection / ca if ca else 0.0
+ if iou < cfg.maximum_existing_iou and coverage < cfg.maximum_existing_candidate_coverage:
+ return None
+ return {"intersection_pixels": intersection, "iou": round(iou, 6), "candidate_coverage": round(coverage, 6)}
+
+
+def successful_replacement_foreign_overlap(
+ candidate: np.ndarray,
+ existing: np.ndarray,
+ cfg: BboxFailedMaskRescueConfig,
+) -> dict[str, Any] | None:
+ """Reject a broad successful-mask replacement that steals a named peer."""
+
+ intersection = int(np.logical_and(candidate, existing).sum())
+ if intersection < cfg.minimum_successful_replacement_foreign_overlap_pixels:
+ return None
+ candidate_area = max(1, int(candidate.sum()))
+ existing_area = max(1, int(existing.sum()))
+ existing_coverage = intersection / existing_area
+ if existing_coverage < cfg.minimum_successful_replacement_other_object_coverage:
+ return None
+ return {
+ "intersection_pixels": intersection,
+ "candidate_coverage": round(intersection / candidate_area, 6),
+ "existing_object_coverage": round(existing_coverage, 6),
+ }
+
+
+def _load_mask(path_value: Any, shape: tuple[int, int], required_root: Path | None = None) -> tuple[np.ndarray, Path]:
+ path = Path(str(path_value or "")).expanduser().resolve()
+ if required_root is not None and not path.is_relative_to(required_root.resolve()):
+ raise ValueError("candidate mask escapes bbox rescue root")
+ mask = np.array(Image.open(path).convert("L")) > 0
+ if mask.shape != shape or not mask.any():
+ raise ValueError("invalid candidate mask")
+ return mask, path
+
+
+def _atomic_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ temporary.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ os.replace(temporary, path)
+
+
+def run_bbox_failed_mask_rescue(
+ *,
+ session: Any,
+ image_path: str | Path,
+ objects: Sequence[Mapping[str, Any]],
+ work_dir: str | Path,
+ staging_dir: str | Path,
+ infer_candidates_fn: Callable[..., Mapping[str, Any]],
+ config: BboxFailedMaskRescueConfig | None = None,
+ unresolved_source_object_ids: Sequence[str] | None = None,
+) -> dict[str, Any]:
+ cfg = config or BboxFailedMaskRescueConfig()
+ if not cfg.isolated_attempt_only:
+ raise ValueError("bbox rescue is isolated-attempt only")
+ copied = [copy.deepcopy(dict(item)) for item in objects]
+ work = Path(work_dir).expanduser().resolve()
+ if work.exists() and any(work.iterdir()):
+ raise FileExistsError(f"bbox rescue work directory is not empty: {work}")
+ work.mkdir(parents=True, exist_ok=True)
+ audit = {
+ "schema_version": "scenemosaic_bbox_failed_mask_rescue",
+ "enabled": cfg.enabled,
+ "status": "disabled" if not cfg.enabled else "pending",
+ "maximum_targets": cfg.maximum_targets,
+ "target_count": 0,
+ "inference_call_count": 0,
+ "accepted_count": 0,
+ "accepted": [],
+ "rejected": [],
+ "skipped": [],
+ "resolved_prior_retry_error_object_ids": [],
+ "excluded_unmaterialized": [],
+ "excluded_unmaterialized_source_object_ids": [],
+ "nonblocking_errors": [],
+ "hard_errors": [],
+ "missing_success_mask_object_ids": [],
+ "additional_vlm_calls": 0,
+ }
+ audit_path = work / "bbox_failed_mask_rescue_audit.json"
+ if not cfg.enabled:
+ _atomic_json(audit_path, audit)
+ return {"objects": copied, "audit": audit, "audit_path": str(audit_path)}
+
+ image = Path(image_path).expanduser().resolve()
+ with Image.open(image) as source:
+ image_np = np.array(source.convert("RGB"))
+ height, width = image_np.shape[:2]
+ shape = (height, width)
+ existing_masks: dict[str, np.ndarray] = {}
+ for obj in copied:
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "success":
+ continue
+ try:
+ existing_masks[str(obj.get("object_id") or "")], _ = _load_mask(sam3.get("mask_path"), shape)
+ except (OSError, ValueError):
+ # Treat a stale or invalid mask path as a missing mask. It is a
+ # rescue target, not a terminal error, provided the local rescue
+ # can materialize a valid replacement.
+ audit["missing_success_mask_object_ids"].append(
+ str(obj.get("object_id") or "")
+ )
+
+ eligible = []
+ objects_by_id = {
+ str(obj.get("object_id") or ""): obj for obj in copied
+ }
+ suppressed_duplicate_addition_object_ids: set[str] = set()
+ suppressed_failed_original_object_ids: set[str] = set()
+ for obj in copied:
+ sam3 = obj.get("sam3") or {}
+ object_id = str(obj.get("object_id") or "")
+ target_bbox = norm_bbox_to_abs(obj.get("bbox_2d_norm_xyxy"), width, height)
+ prompt = rescue_prompt(obj)
+ existing_mask = existing_masks.get(object_id)
+ has_valid_success_mask = (
+ sam3.get("status") == "success" and existing_mask is not None
+ )
+ failed_or_missing_mask = not has_valid_success_mask
+ try:
+ existing_bbox_iou = float(sam3.get("vlm_bbox_iou"))
+ except (TypeError, ValueError):
+ existing_bbox_iou = math.nan
+ existing_target_bbox_coverage = math.nan
+ if existing_mask is not None and target_bbox is not None:
+ ys, xs = np.nonzero(existing_mask)
+ if len(xs):
+ existing_bbox = [
+ int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1,
+ ]
+ existing_target_bbox_coverage = bbox_metrics(
+ target_bbox, existing_bbox,
+ )["target_coverage"]
+ successful_low_iou = (
+ math.isfinite(existing_bbox_iou)
+ and existing_bbox_iou < cfg.maximum_successful_vlm_bbox_iou
+ )
+ preserve_compact_isolated_vase_success = (
+ cfg.preserve_compact_isolated_vase_success
+ and sam3.get("status") == "success"
+ and is_isolated_vase(obj)
+ and math.isfinite(existing_bbox_iou)
+ and existing_bbox_iou
+ >= cfg.compact_vase_minimum_existing_bbox_iou
+ and math.isfinite(existing_target_bbox_coverage)
+ and existing_target_bbox_coverage
+ >= cfg.compact_vase_minimum_existing_target_coverage
+ )
+ successful_low_coverage = (
+ math.isfinite(existing_target_bbox_coverage)
+ and existing_target_bbox_coverage
+ < cfg.minimum_successful_target_bbox_coverage
+ and not preserve_compact_isolated_vase_success
+ )
+ successful_low_extent = (
+ cfg.repair_successful_low_extent
+ and sam3.get("status") == "success"
+ and (successful_low_iou or successful_low_coverage)
+ and (
+ cfg.repair_successful_low_extent_lighting
+ or not is_lighting_object(obj)
+ )
+ )
+ failed_recessed_ceiling_light = (
+ failed_or_missing_mask
+ and is_recessed_ceiling_light(obj)
+ and cfg.repair_failed_recessed_ceiling_lights
+ )
+ # A broad type such as "furniture" can locate a nearby object, but it
+ # is not evidence that a successful instance mask is wrong. Keep the
+ # verified mask rather than replacing it with a generic query merely
+ # because the VLM bbox and mask bbox disagree. Failed masks remain
+ # eligible for the bounded spatial rescue.
+ successful_broad_replacement_forbidden = (
+ successful_low_extent and is_broad_object_type_prompt(obj, prompt or "")
+ )
+ if (
+ failed_or_missing_mask
+ and target_bbox
+ and prompt
+ ):
+ source_passes = {
+ str(source.get("pass") or "").upper()
+ for source in (obj.get("proposal_sources") or [])
+ if isinstance(source, Mapping)
+ }
+ source_object_ids = {
+ str(source.get("source_id") or "")
+ for source in (obj.get("proposal_sources") or [])
+ if isinstance(source, Mapping)
+ }
+ evidence_priority = bool(
+ source_object_ids & cfg.priority_source_object_ids
+ )
+ # Prioritize explicitly reviewed evidence, then failed Pass-B
+ # additions, then every other failed-or-missing object. Valid
+ # successful masks never enter this queue.
+ priority = (
+ 0 if evidence_priority
+ else 1 if "B" in source_passes
+ else 2
+ )
+ eligible.append(
+ (
+ priority,
+ obj,
+ object_id,
+ target_bbox,
+ prompt,
+ (
+ "failed_recessed_ceiling_light"
+ if failed_recessed_ceiling_light
+ else (
+ "missing_success_mask"
+ if sam3.get("status") == "success"
+ else "failed_mask"
+ )
+ ),
+ )
+ )
+ else:
+ audit["skipped"].append({
+ "object_id": object_id,
+ "reason": (
+ "preserved_compact_isolated_vase_success"
+ if preserve_compact_isolated_vase_success
+ else "valid_success_mask"
+ ),
+ })
+ eligible.sort(key=lambda item: (item[0], item[2]))
+ audit["target_count"] = min(len(eligible), cfg.maximum_targets)
+ for overflow in eligible[cfg.maximum_targets :]:
+ audit["rejected"].append({"object_id": overflow[2], "reason": "target_budget_exceeded"})
+
+ for _, obj, object_id, target_bbox, prompt, rescue_kind in eligible[: cfg.maximum_targets]:
+ target_work = work / object_id
+ source_passes = {
+ str(source.get("pass") or "").upper()
+ for source in (obj.get("proposal_sources") or [])
+ if isinstance(source, Mapping)
+ }
+ is_pass_b_addition = "B" in source_passes and "A" not in source_passes
+ duplicate_conflicts = []
+ replacement_conflicts = []
+ prior_target_mask = existing_masks.get(object_id)
+ audit["inference_call_count"] += 1
+ try:
+ result = infer_candidates_fn(
+ session,
+ image,
+ prompt,
+ target_work,
+ score_threshold=cfg.minimum_score,
+ positive_bbox_abs_xyxy=target_bbox,
+ maximum_candidates=(
+ max(
+ cfg.maximum_candidates_per_target,
+ cfg.maximum_recessed_candidates_per_target,
+ )
+ if rescue_kind == "failed_recessed_ceiling_light"
+ else cfg.maximum_candidates_per_target
+ ),
+ )
+ registry = result.get("registry") or {}
+ records = registry.get("candidates") or []
+ except Exception as exc:
+ audit["hard_errors"].append({"type": "bbox_rescue_inference_exception", "object_id": object_id, "error": f"{type(exc).__name__}: {exc}"})
+ continue
+ viable = []
+ for record in records:
+ try:
+ mask, mask_path = _load_mask(record.get("mask_path"), shape, target_work)
+ evidence = candidate_evidence(obj, mask, float(record.get("score") or 0.0), target_bbox, cfg)
+ except (OSError, ValueError) as exc:
+ audit["rejected"].append({"object_id": object_id, "candidate_id": record.get("candidate_id"), "reason": f"invalid_candidate:{exc}"})
+ continue
+ if evidence["status"] != "accept":
+ audit["rejected"].append({"object_id": object_id, "candidate_id": record.get("candidate_id"), "reason": evidence["reason"], "evidence": evidence})
+ continue
+ if rescue_kind == "successful_low_extent":
+ prior_sam3 = dict(obj.get("sam3") or {})
+ try:
+ prior_score = float(prior_sam3.get("score"))
+ except (TypeError, ValueError):
+ prior_score = math.nan
+ prior_bbox = prior_sam3.get("mask_bbox_abs_xyxy")
+ prior_metrics = None
+ if (
+ isinstance(prior_bbox, (list, tuple))
+ and len(prior_bbox) == 4
+ ):
+ try:
+ prior_metrics = bbox_metrics(target_bbox, prior_bbox)
+ except (TypeError, ValueError):
+ prior_metrics = None
+ score_improved = (
+ math.isfinite(prior_score)
+ and float(evidence["score"])
+ >= prior_score + cfg.minimum_successful_replacement_score_gain
+ )
+ coverage_improved = (
+ prior_metrics is not None
+ and float(evidence["target_coverage"])
+ >= float(prior_metrics["target_coverage"])
+ + cfg.minimum_successful_replacement_target_coverage_gain
+ )
+ if not (score_improved and coverage_improved):
+ audit["rejected"].append({
+ "object_id": object_id,
+ "candidate_id": record.get("candidate_id"),
+ "reason": "successful_mask_replacement_not_proven_better",
+ "evidence": evidence,
+ "prior_score": (
+ round(prior_score, 6)
+ if math.isfinite(prior_score) else None
+ ),
+ "prior_target_bbox_coverage": (
+ round(float(prior_metrics["target_coverage"]), 6)
+ if prior_metrics is not None else None
+ ),
+ })
+ continue
+ disconnected = None
+ if (
+ cfg.reject_disconnected_generic_successful_replacement
+ and prior_target_mask is not None
+ and is_broad_object_type_prompt(obj, prompt)
+ ):
+ disconnected = disconnected_generic_replacement_evidence(
+ mask,
+ prior_target_mask,
+ cfg,
+ )
+ if disconnected is not None:
+ audit["rejected"].append({
+ "object_id": object_id,
+ "candidate_id": record.get("candidate_id"),
+ "reason": "generic_rescue_contains_significant_disconnected_foreign_component",
+ "evidence": evidence,
+ "disconnected_component_evidence": disconnected,
+ })
+ continue
+ conflicts = []
+ strict_foreign_conflicts = []
+ for existing_id, existing_mask in existing_masks.items():
+ if existing_id == object_id:
+ continue
+ conflict = overlap_conflict(mask, existing_mask, cfg)
+ if conflict is not None:
+ conflicts.append({"existing_object_id": existing_id, **conflict})
+ if (
+ rescue_kind == "successful_low_extent"
+ and prior_target_mask is not None
+ and is_broad_object_type_prompt(obj, prompt)
+ ):
+ strict_conflict = successful_replacement_foreign_overlap(
+ mask, existing_mask, cfg
+ )
+ if strict_conflict is not None:
+ strict_foreign_conflicts.append({
+ "existing_object_id": existing_id,
+ **strict_conflict,
+ })
+ if conflicts or strict_foreign_conflicts:
+ rejection = {
+ "object_id": object_id,
+ "candidate_id": record.get("candidate_id"),
+ "reason": (
+ "generic_rescue_overlaps_existing_success_object"
+ if strict_foreign_conflicts
+ else "overlaps_existing_success"
+ ),
+ "conflicts": conflicts,
+ "strict_foreign_conflicts": strict_foreign_conflicts,
+ }
+ audit["rejected"].append(rejection)
+ if (
+ rescue_kind == "failed_recessed_ceiling_light"
+ and bool(evidence.get("nearby_recessed_ceiling_light"))
+ ):
+ for item in conflicts:
+ replacement = objects_by_id.get(
+ str(item.get("existing_object_id") or "")
+ )
+ replacement_passes = {
+ str(source.get("pass") or "").upper()
+ for source in (replacement or {}).get("proposal_sources") or []
+ if isinstance(source, Mapping)
+ }
+ if (
+ replacement is not None
+ and is_recessed_ceiling_light(replacement)
+ and "B" in replacement_passes
+ and float(item.get("candidate_coverage") or 0.0) >= 0.90
+ ):
+ replacement_conflicts.append({
+ "replacement_object_id": replacement.get("object_id"),
+ "candidate_id": record.get("candidate_id"),
+ "candidate_coverage": item.get("candidate_coverage"),
+ "evidence": evidence,
+ })
+ if (
+ is_pass_b_addition
+ and any(
+ float(item.get("candidate_coverage") or 0.0) >= 0.90
+ for item in conflicts
+ )
+ ):
+ duplicate_conflicts.append(rejection)
+ continue
+ viable.append({"record": record, "mask": mask, "mask_path": mask_path, "evidence": evidence})
+ viable.sort(key=lambda item: float(item["evidence"]["rank"]), reverse=True)
+ if not viable:
+ if replacement_conflicts:
+ replacement_ids = sorted({
+ str(item.get("replacement_object_id") or "")
+ for item in replacement_conflicts
+ if str(item.get("replacement_object_id") or "")
+ })
+ if len(replacement_ids) != 1:
+ audit["hard_errors"].append({
+ "type": "bbox_spatial_replacement_ambiguous",
+ "object_id": object_id,
+ "replacement_object_ids": replacement_ids,
+ })
+ continue
+ replacement_id = replacement_ids[0]
+ replacement = objects_by_id[replacement_id]
+ replacement_sources = list(replacement.get("proposal_sources") or [])
+ source_keys = {
+ (str(source.get("pass") or ""), str(source.get("source_id") or ""))
+ for source in replacement_sources
+ if isinstance(source, Mapping)
+ }
+ for source in obj.get("proposal_sources") or []:
+ if not isinstance(source, Mapping):
+ continue
+ key = (str(source.get("pass") or ""), str(source.get("source_id") or ""))
+ if key in source_keys:
+ continue
+ carried = copy.deepcopy(dict(source))
+ carried["action"] = "nearby_spatial_replacement"
+ replacement_sources.append(carried)
+ source_keys.add(key)
+ replacement["proposal_sources"] = replacement_sources
+ replacement_sam3 = copy.deepcopy(dict(replacement.get("sam3") or {}))
+ replacement_sam3["prior_selection_method"] = replacement_sam3.get(
+ "selection_method"
+ )
+ replacement_sam3["selection_method"] = "bbox_first_spatial_rescue"
+ replacement_sam3["nearby_failed_original_replacement"] = {
+ "failed_object_id": object_id,
+ "replacement_object_id": replacement_id,
+ "prompt": prompt,
+ "candidate_id": replacement_conflicts[0].get("candidate_id"),
+ "evidence": replacement_conflicts[0].get("evidence"),
+ }
+ replacement["sam3"] = replacement_sam3
+ suppressed_failed_original_object_ids.add(object_id)
+ audit["accepted"].append({
+ "object_id": object_id,
+ "replacement_object_id": replacement_id,
+ "category_preserved": obj.get("category"),
+ "rescue_kind": "failed_recessed_light_reused_nearby_pass_b_success",
+ "evidence": replacement_conflicts[0].get("evidence"),
+ })
+ audit["resolved_prior_retry_error_object_ids"].append(object_id)
+ continue
+ if duplicate_conflicts:
+ if prior_target_mask is not None:
+ audit["skipped"].append({
+ "object_id": object_id,
+ "reason": "successful_mask_preserved_after_conflicting_rescue",
+ "prompt": prompt,
+ "conflicts": duplicate_conflicts,
+ })
+ continue
+ suppressed_duplicate_addition_object_ids.add(object_id)
+ audit.setdefault("suppressed_duplicate_additions", []).append({
+ "object_id": object_id,
+ "reason": "failed_pass_b_addition_fully_covered_by_existing_success",
+ "conflicts": duplicate_conflicts,
+ })
+ continue
+ if prior_target_mask is not None:
+ audit["skipped"].append({
+ "object_id": object_id,
+ "reason": "successful_mask_preserved_after_unsuccessful_rescue",
+ "prompt": prompt,
+ })
+ continue
+ unsafe_generic_rejection = any(
+ item.get("object_id") == object_id
+ and item.get("reason") in {
+ "generic_rescue_contains_significant_disconnected_foreign_component",
+ "generic_rescue_overlaps_existing_success_object",
+ }
+ for item in audit["rejected"]
+ if isinstance(item, Mapping)
+ )
+ if prior_target_mask is not None and unsafe_generic_rejection:
+ audit["skipped"].append({
+ "object_id": object_id,
+ "reason": "unsafe_generic_rescue_preserved_existing_success",
+ "prompt": prompt,
+ })
+ continue
+ audit["hard_errors"].append({"type": "bbox_spatial_rescue_failed", "object_id": object_id, "prompt": prompt})
+ continue
+ if len(viable) > 1 and float(viable[0]["evidence"]["rank"]) - float(viable[1]["evidence"]["rank"]) < cfg.minimum_rank_margin:
+ if prior_target_mask is not None:
+ audit["skipped"].append({
+ "object_id": object_id,
+ "reason": "successful_mask_preserved_after_ambiguous_rescue",
+ "prompt": prompt,
+ "remaining_candidate_count": len(viable),
+ "remaining_rank_margin": round(
+ float(viable[0]["evidence"]["rank"])
+ - float(viable[1]["evidence"]["rank"]),
+ 6,
+ ),
+ })
+ continue
+ unsafe_generic_rejection = any(
+ item.get("object_id") == object_id
+ and item.get("reason") in {
+ "generic_rescue_contains_significant_disconnected_foreign_component",
+ "generic_rescue_overlaps_existing_success_object",
+ }
+ for item in audit["rejected"]
+ if isinstance(item, Mapping)
+ )
+ if prior_target_mask is not None and unsafe_generic_rejection:
+ audit["skipped"].append({
+ "object_id": object_id,
+ "reason": "unsafe_generic_rescue_preserved_existing_success",
+ "prompt": prompt,
+ "remaining_candidate_count": len(viable),
+ "remaining_rank_margin": round(
+ float(viable[0]["evidence"]["rank"])
+ - float(viable[1]["evidence"]["rank"]),
+ 6,
+ ),
+ })
+ continue
+ audit["hard_errors"].append({"type": "bbox_spatial_rescue_ambiguous", "object_id": object_id, "rank_margin": round(float(viable[0]["evidence"]["rank"]) - float(viable[1]["evidence"]["rank"]), 6)})
+ continue
+ selected = viable[0]
+ compact_vase_selection = None
+ if (
+ cfg.prefer_compact_nested_isolated_vase
+ and is_isolated_vase(obj)
+ and len(viable) > 1
+ ):
+ broad = viable[0]
+ broad_mask = np.asarray(broad["mask"], dtype=bool)
+ broad_box = broad["evidence"]["candidate_bbox_abs_xyxy"]
+ broad_area = max(1, int(broad_mask.sum()))
+ broad_width = max(1, int(broad_box[2]) - int(broad_box[0]))
+ broad_height = max(1, int(broad_box[3]) - int(broad_box[1]))
+ compact_candidates = []
+ for alternative in viable[1:]:
+ compact_mask = np.asarray(alternative["mask"], dtype=bool)
+ compact_area = int(compact_mask.sum())
+ compact_box = alternative["evidence"]["candidate_bbox_abs_xyxy"]
+ containment = int(
+ np.logical_and(compact_mask, broad_mask).sum()
+ ) / max(1, compact_area)
+ area_ratio = compact_area / broad_area
+ width_ratio = (
+ int(compact_box[2]) - int(compact_box[0])
+ ) / broad_width
+ height_ratio = (
+ int(compact_box[3]) - int(compact_box[1])
+ ) / broad_height
+ removed_pixels = broad_area - int(
+ np.logical_and(compact_mask, broad_mask).sum()
+ )
+ if (
+ containment >= cfg.compact_vase_minimum_containment
+ and area_ratio >= cfg.compact_vase_minimum_area_ratio
+ and float(alternative["evidence"]["target_coverage"])
+ >= cfg.compact_vase_minimum_target_coverage
+ and width_ratio <= cfg.compact_vase_maximum_width_ratio
+ and height_ratio >= cfg.compact_vase_minimum_height_ratio
+ and removed_pixels >= cfg.compact_vase_minimum_removed_pixels
+ ):
+ compact_candidates.append((
+ float(alternative["evidence"]["rank"]),
+ alternative,
+ {
+ "containment_by_broad_candidate": round(containment, 6),
+ "area_ratio_to_broad_candidate": round(area_ratio, 6),
+ "width_ratio_to_broad_candidate": round(width_ratio, 6),
+ "height_ratio_to_broad_candidate": round(height_ratio, 6),
+ "removed_broad_pixels": removed_pixels,
+ "broad_candidate_id": broad["record"].get("candidate_id"),
+ },
+ ))
+ if compact_candidates:
+ _, selected, compact_vase_selection = max(
+ compact_candidates,
+ key=lambda item: item[0],
+ )
+ staging = Path(staging_dir).expanduser().resolve()
+ masks_dir, overlays_dir = staging / "masks", staging / "overlays"
+ masks_dir.mkdir(parents=True, exist_ok=True)
+ overlays_dir.mkdir(parents=True, exist_ok=True)
+ base = f"{object_id}_{normalize_prompt(obj.get('category')).replace(' ', '_') or 'object'}"
+ mask_path = masks_dir / f"{base}_mask.png"
+ overlay_path = overlays_dir / f"{base}_overlay.png"
+ Image.fromarray(selected["mask"].astype(np.uint8) * 255).save(mask_path)
+ overlay = image_np.copy()
+ overlay[selected["mask"]] = (0.55 * overlay[selected["mask"]] + 0.45 * np.array((255, 0, 0), dtype=np.uint8)).astype(np.uint8)
+ Image.fromarray(overlay).save(overlay_path)
+ prior = copy.deepcopy(dict(obj.get("sam3") or {}))
+ candidate = selected["record"]
+ obj["sam3"] = {
+ "status": "success",
+ "warning": "semantic_label_provisional_after_bbox_spatial_rescue",
+ "prompt_used": prompt,
+ "score": float(candidate.get("score") or 0.0),
+ "area_pixels": int(selected["mask"].sum()),
+ "vlm_bbox_abs_xyxy": list(target_bbox),
+ "mask_bbox_abs_xyxy": list(selected["evidence"]["candidate_bbox_abs_xyxy"]),
+ "mask_path": str(mask_path),
+ "overlay_path": str(overlay_path),
+ "selection_method": "bbox_first_spatial_rescue",
+ "bbox_spatial_rescue": {
+ "semantic_label_status": "provisional",
+ "original_category": obj.get("category"),
+ "original_prompt": prior.get("prompt_used") or obj.get("sam3_prompt"),
+ "broad_prompt": prompt,
+ "positive_bbox_abs_xyxy": list(target_bbox),
+ "candidate_id": candidate.get("candidate_id"),
+ "evidence": selected["evidence"],
+ "rescue_kind": rescue_kind,
+ "compact_nested_vase_selection": compact_vase_selection,
+ },
+ }
+ existing_masks[object_id] = selected["mask"]
+ accepted = {
+ "object_id": object_id,
+ "category_preserved": obj.get("category"),
+ "semantic_label_status": "provisional",
+ "broad_prompt": prompt,
+ "candidate_id": candidate.get("candidate_id"),
+ "mask_path": str(mask_path),
+ "overlay_path": str(overlay_path),
+ "prior_sam3": prior,
+ "rescue_kind": rescue_kind,
+ "evidence": selected["evidence"],
+ }
+ audit["accepted"].append(accepted)
+ audit["resolved_prior_retry_error_object_ids"].append(object_id)
+
+ suppressed_object_ids = (
+ suppressed_duplicate_addition_object_ids
+ | suppressed_failed_original_object_ids
+ )
+ unresolved_sources = {
+ str(source_id)
+ for source_id in unresolved_source_object_ids or []
+ if str(source_id)
+ }
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ sam3 = obj.get("sam3") or {}
+ source_passes = {
+ str(source.get("pass") or "").upper()
+ for source in obj.get("proposal_sources") or []
+ if isinstance(source, Mapping)
+ }
+ source_ids = {
+ str(source.get("source_id") or "")
+ for source in obj.get("proposal_sources") or []
+ if isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "A"
+ and str(source.get("source_id") or "")
+ }
+ pass_b_source_ids = sorted(
+ {
+ str(source.get("source_id") or "")
+ for source in obj.get("proposal_sources") or []
+ if isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "B"
+ and str(source.get("source_id") or "")
+ }
+ )
+ unresolved_existing_claim = bool(source_ids & unresolved_sources)
+ unmaterialized_pass_b_addition = (
+ "B" in source_passes and "A" not in source_passes
+ )
+ if sam3.get("status") != "success" and (
+ unresolved_existing_claim or unmaterialized_pass_b_addition
+ ):
+ suppressed_object_ids.add(object_id)
+ audit["excluded_unmaterialized"].append(
+ {
+ "object_id": object_id,
+ "source_object_ids": sorted(source_ids & unresolved_sources),
+ "pass_b_source_ids": pass_b_source_ids,
+ "reason": (
+ "failed_sam3_and_spatial_rescue_after_unresolved_review"
+ if unresolved_existing_claim
+ else "unmaterialized_pass_b_addition_excluded"
+ ),
+ }
+ )
+ if suppressed_object_ids:
+ copied = [
+ obj for obj in copied
+ if str(obj.get("object_id") or "") not in suppressed_object_ids
+ ]
+ audit["suppressed_duplicate_addition_object_ids"] = sorted(
+ suppressed_duplicate_addition_object_ids
+ )
+ audit["suppressed_failed_original_object_ids"] = sorted(
+ suppressed_failed_original_object_ids
+ )
+ excluded_object_ids = {
+ str(item.get("object_id") or "")
+ for item in audit["excluded_unmaterialized"]
+ }
+ audit["excluded_unmaterialized_source_object_ids"] = sorted(
+ {
+ source_id
+ for item in audit["excluded_unmaterialized"]
+ for source_id in item["source_object_ids"]
+ }
+ )
+ successful_object_ids = {
+ str(obj.get("object_id") or "")
+ for obj in copied
+ if str((obj.get("sam3") or {}).get("status") or "") == "success"
+ }
+ rescue_error_types = {
+ "bbox_rescue_inference_exception",
+ "bbox_spatial_rescue_ambiguous",
+ "bbox_spatial_rescue_failed",
+ }
+ blocking_errors = []
+ for error in audit["hard_errors"]:
+ object_id = str(error.get("object_id") or "")
+ if object_id in excluded_object_ids:
+ audit["nonblocking_errors"].append(
+ {**error, "resolution": "unmaterialized_object_excluded"}
+ )
+ elif object_id in successful_object_ids and error.get("type") in rescue_error_types:
+ audit["nonblocking_errors"].append(
+ {**error, "resolution": "successful_object_retained"}
+ )
+ else:
+ blocking_errors.append(error)
+ audit["hard_errors"] = blocking_errors
+ audit["accepted_count"] = len(audit["accepted"])
+ if audit["hard_errors"]:
+ audit["status"] = "failed"
+ elif audit["target_count"] == 0:
+ audit["status"] = "noop"
+ elif audit["accepted_count"]:
+ audit["status"] = "pass"
+ else:
+ audit["status"] = "no_rescue"
+ _atomic_json(audit_path, audit)
+ return {"objects": copied, "audit": audit, "audit_path": str(audit_path)}
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/content_host_review.py b/vlm_agent/reconstruction/perception/pipeline/refinements/content_host_review.py
new file mode 100644
index 0000000..679f781
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/content_host_review.py
@@ -0,0 +1,891 @@
+#!/usr/bin/env python3
+"""Build a bounded high-resolution review panel for content-bearing objects.
+
+This module is intentionally semantic-light. It selects only objects whose
+existing Pass-A names consistently describe an exposed support surface or open
+storage carrier, then renders local crops for the existing Pass-B call. The
+crops are attention aids, not segmentation evidence and never create objects.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import os
+import re
+import shutil
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+from PIL import Image, ImageDraw, ImageFont
+
+from .repair_scope_router import classify_object_role
+
+
+@dataclass(frozen=True)
+class ContentHostConfig:
+ maximum_hosts: int = 8
+ minimum_score: float = 0.68
+ minimum_bbox_area_fraction: float = 0.0015
+ maximum_bbox_area_fraction: float = 0.35
+ duplicate_iou: float = 0.65
+ duplicate_containment: float = 0.85
+ maximum_crop_iou: float = 0.55
+ fail_on_budget_overflow: bool = True
+
+
+OPEN_STORAGE_HEADS = {
+ "bookcase",
+ "bookshelf",
+ "cubby",
+ "hutch",
+ "rack",
+ "shelf",
+ "shelving",
+}
+SUPPORT_SURFACE_HEADS = {
+ "cabinet",
+ "cart",
+ "console",
+ "counter",
+ "countertop",
+ "desk",
+ "dresser",
+ "island",
+ "nightstand",
+ "sideboard",
+ "table",
+ "vanity",
+}
+GENERIC_HOST_SUFFIXES = {"fixture", "furniture", "storage", "system", "unit"}
+SUPPORTED_TYPES = {"furniture", "structural_surface"}
+CONFLICT_TYPES = {
+ "appliance",
+ "lighting",
+ "opening",
+ "plant",
+ "small_object",
+ "textile",
+}
+
+
+def _font(size: int):
+ try:
+ return ImageFont.truetype("DejaVuSans.ttf", size)
+ except OSError:
+ return ImageFont.load_default()
+
+
+def _tokens(value: Any) -> list[str]:
+ return re.findall(r"[a-z0-9]+", str(value or "").lower())
+
+
+def _field_host_mode(value: Any) -> str | None:
+ tokens = _tokens(value)
+ while len(tokens) > 1 and tokens[-1] in GENERIC_HOST_SUFFIXES:
+ tokens.pop()
+ if not tokens:
+ return None
+ head = tokens[-1]
+ if head in OPEN_STORAGE_HEADS:
+ return "open_storage"
+ if head in SUPPORT_SURFACE_HEADS:
+ return "support_surface"
+ return None
+
+
+def _valid_bbox(value: Any) -> bool:
+ return (
+ isinstance(value, list)
+ and len(value) == 4
+ and all(isinstance(item, (int, float)) and math.isfinite(item) for item in value)
+ and 0 <= float(value[0]) < float(value[2]) <= 1
+ and 0 <= float(value[1]) < float(value[3]) <= 1
+ )
+
+
+def _bbox_metrics(left: Sequence[float], right: Sequence[float]) -> dict[str, float]:
+ x1 = max(float(left[0]), float(right[0]))
+ y1 = max(float(left[1]), float(right[1]))
+ x2 = min(float(left[2]), float(right[2]))
+ y2 = min(float(left[3]), float(right[3]))
+ intersection = max(0.0, x2 - x1) * max(0.0, y2 - y1)
+ left_area = (float(left[2]) - float(left[0])) * (
+ float(left[3]) - float(left[1])
+ )
+ right_area = (float(right[2]) - float(right[0])) * (
+ float(right[3]) - float(right[1])
+ )
+ union = left_area + right_area - intersection
+ return {
+ "iou": intersection / union if union else 0.0,
+ "left_coverage": intersection / left_area if left_area else 0.0,
+ "right_coverage": intersection / right_area if right_area else 0.0,
+ }
+
+
+def _clamp(value: float, minimum: float, maximum: float) -> float:
+ return max(minimum, min(maximum, value))
+
+
+def _expand_interval(
+ start: float,
+ end: float,
+ minimum_size: float,
+) -> tuple[float, float]:
+ size = end - start
+ if size >= minimum_size:
+ return max(0.0, start), min(1.0, end)
+ center = (start + end) / 2
+ start = center - minimum_size / 2
+ end = center + minimum_size / 2
+ if start < 0:
+ end -= start
+ start = 0.0
+ if end > 1:
+ start -= end - 1
+ end = 1.0
+ return max(0.0, start), min(1.0, end)
+
+
+def _initial_crop(host: Mapping[str, Any]) -> list[float]:
+ x1, y1, x2, y2 = [float(value) for value in host["bbox_norm_xyxy"]]
+ width = x2 - x1
+ height = y2 - y1
+ if host["host_mode"] == "open_storage":
+ x_pad = min(0.04, max(0.015, 0.06 * width))
+ y_pad = min(0.04, max(0.015, 0.06 * height))
+ crop = [x1 - x_pad, y1 - y_pad, x2 + x_pad, y2 + y_pad]
+ min_width, min_height = 0.14, 0.20
+ else:
+ x_pad = _clamp(0.15 * width, 0.02, 0.06)
+ text = " ".join(
+ str(host.get(key) or "")
+ for key in ("category", "sam3_prompt", "instance_name", "description")
+ ).lower()
+ partial_at_edge = (
+ any(token in text for token in ("partial", "cropped", "truncated"))
+ and (x1 <= 0.01 or x2 >= 0.99 or y2 >= 0.99)
+ )
+ top_pad = (
+ _clamp(1.75 * height, 0.12, 0.30)
+ if partial_at_edge
+ else _clamp(0.75 * height, 0.05, 0.18)
+ )
+ crop = [x1 - x_pad, y1 - top_pad, x2 + x_pad, y1 + 0.55 * height]
+ min_width, min_height = 0.14, 0.18
+ crop[0], crop[2] = _expand_interval(crop[0], crop[2], min_width)
+ crop[1], crop[3] = _expand_interval(crop[1], crop[3], min_height)
+ return [round(_clamp(value, 0.0, 1.0), 6) for value in crop]
+
+
+def _adjust_neighbor_padding(hosts: list[dict[str, Any]]) -> None:
+ """Shrink only padding between disjoint neighboring host bboxes."""
+
+ for left_index in range(len(hosts)):
+ for right_index in range(left_index + 1, len(hosts)):
+ left = hosts[left_index]
+ right = hosts[right_index]
+ lb = left["bbox_norm_xyxy"]
+ rb = right["bbox_norm_xyxy"]
+ lc = left["crop_norm_xyxy"]
+ rc = right["crop_norm_xyxy"]
+ vertical_overlap = min(lb[3], rb[3]) - max(lb[1], rb[1])
+ horizontal_overlap = min(lb[2], rb[2]) - max(lb[0], rb[0])
+ if lb[2] <= rb[0] and vertical_overlap > 0:
+ split = (lb[2] + rb[0]) / 2
+ if lc[2] > split and rc[0] < split:
+ lc[2] = split
+ rc[0] = split
+ elif rb[2] <= lb[0] and vertical_overlap > 0:
+ split = (rb[2] + lb[0]) / 2
+ if rc[2] > split and lc[0] < split:
+ rc[2] = split
+ lc[0] = split
+ if lb[3] <= rb[1] and horizontal_overlap > 0:
+ split = (lb[3] + rb[1]) / 2
+ if lc[3] > split and rc[1] < split:
+ lc[3] = split
+ rc[1] = split
+ elif rb[3] <= lb[1] and horizontal_overlap > 0:
+ split = (rb[3] + lb[1]) / 2
+ if rc[3] > split and lc[1] < split:
+ rc[3] = split
+ lc[1] = split
+ for host in hosts:
+ host["crop_norm_xyxy"] = [
+ round(float(value), 6) for value in host["crop_norm_xyxy"]
+ ]
+
+
+def select_content_hosts(
+ objects: Sequence[Mapping[str, Any]],
+ *,
+ config: ContentHostConfig | None = None,
+) -> dict[str, Any]:
+ """Select a bounded set of content-bearing regions from Pass-A objects."""
+
+ cfg = config or ContentHostConfig()
+ if cfg.maximum_hosts < 1:
+ raise ValueError("maximum_hosts must be positive")
+ accepted: list[dict[str, Any]] = []
+ rejections: list[dict[str, Any]] = []
+ hard_errors: list[dict[str, Any]] = []
+ seen_ids: set[str] = set()
+
+ for raw in objects:
+ item = dict(raw)
+ object_id = str(item.get("object_id") or "")
+ if not object_id or object_id in seen_ids:
+ hard_errors.append(
+ {"type": "invalid_or_duplicate_object_id", "object_id": object_id}
+ )
+ continue
+ seen_ids.add(object_id)
+ bbox = item.get("bbox_2d_norm_xyxy")
+ if not _valid_bbox(bbox):
+ modes = [
+ _field_host_mode(item.get(key))
+ for key in ("category", "sam3_prompt", "instance_name")
+ ]
+ if any(modes):
+ hard_errors.append(
+ {"type": "strong_host_invalid_bbox", "object_id": object_id}
+ )
+ else:
+ rejections.append(
+ {"object_id": object_id, "reason": "invalid_bbox_nonhost"}
+ )
+ continue
+
+ modes = [
+ _field_host_mode(item.get(key))
+ for key in ("category", "sam3_prompt", "instance_name")
+ ]
+ mode_counts = {
+ mode: modes.count(mode)
+ for mode in ("open_storage", "support_surface")
+ }
+ host_mode = max(mode_counts, key=mode_counts.get)
+ votes = mode_counts[host_mode]
+ scope_audit = classify_object_role(item)
+ scope_router_override = False
+ object_type = str(item.get("object_type") or "unknown")
+ if (
+ object_type not in CONFLICT_TYPES
+ and scope_audit.get("host_mode") != host_mode
+ ):
+ routed_mode = scope_audit.get("host_mode")
+ confidence = float(scope_audit.get("confidence") or 0.0)
+ if (
+ routed_mode in {"open_storage", "support_surface"}
+ and confidence >= 0.90
+ ):
+ # Explicit category semantics outrank generic end words.
+ host_mode = str(routed_mode)
+ votes = max(2, mode_counts[host_mode])
+ scope_router_override = True
+ else:
+ rejections.append(
+ {
+ "object_id": object_id,
+ "reason": "scope_router_disagreement",
+ "vote_host_mode": host_mode,
+ "scope_router": scope_audit,
+ }
+ )
+ continue
+ if votes < 2:
+ rejections.append(
+ {
+ "object_id": object_id,
+ "reason": "insufficient_host_head_votes",
+ "field_modes": modes,
+ }
+ )
+ continue
+
+ object_type = str(item.get("object_type") or "unknown")
+ if object_type in CONFLICT_TYPES:
+ rejections.append(
+ {
+ "object_id": object_id,
+ "reason": "semantic_type_conflict",
+ "host_mode": host_mode,
+ "object_type": object_type,
+ }
+ )
+ continue
+ if object_type in SUPPORTED_TYPES:
+ type_support = 1.0
+ elif object_type == "unknown":
+ type_support = 0.5 if votes == 3 else 0.0
+ elif object_type == "decor" and host_mode == "open_storage":
+ type_support = 0.5
+ else:
+ rejections.append(
+ {
+ "object_id": object_id,
+ "reason": "unsupported_host_type",
+ "host_mode": host_mode,
+ "object_type": object_type,
+ }
+ )
+ continue
+
+ x1, y1, x2, y2 = [float(value) for value in bbox]
+ area = (x2 - x1) * (y2 - y1)
+ if not cfg.minimum_bbox_area_fraction <= area <= cfg.maximum_bbox_area_fraction:
+ rejections.append(
+ {
+ "object_id": object_id,
+ "reason": "host_bbox_area_out_of_range",
+ "host_mode": host_mode,
+ "area_fraction": round(area, 8),
+ }
+ )
+ continue
+ field_agreement = votes / 3
+ affordance = 1.0 if host_mode == "open_storage" else 0.8
+ score = (
+ 0.40 * min(votes / 2, 1.0)
+ + 0.20 * field_agreement
+ + 0.20 * type_support
+ + 0.10
+ + 0.10 * affordance
+ )
+ if score < cfg.minimum_score:
+ rejections.append(
+ {
+ "object_id": object_id,
+ "reason": "host_score_below_threshold",
+ "score": round(score, 6),
+ }
+ )
+ continue
+ accepted.append(
+ {
+ "source_object_id": object_id,
+ "category": str(item.get("category") or ""),
+ "sam3_prompt": str(item.get("sam3_prompt") or ""),
+ "instance_name": str(item.get("instance_name") or ""),
+ "description": str(item.get("description") or ""),
+ "object_type": object_type,
+ "host_mode": host_mode,
+ "head_votes": votes,
+ "scope_router_override": scope_router_override,
+ "scope_router_audit": scope_audit,
+ "score": round(score, 6),
+ "bbox_norm_xyxy": [round(float(value), 6) for value in bbox],
+ "bbox_area_fraction": round(area, 8),
+ }
+ )
+
+ accepted.sort(
+ key=lambda item: (
+ -item["score"],
+ -item["head_votes"],
+ 0 if item["host_mode"] == "open_storage" else 1,
+ item["bbox_norm_xyxy"][1],
+ item["bbox_norm_xyxy"][0],
+ item["source_object_id"],
+ )
+ )
+ deduplicated: list[dict[str, Any]] = []
+ duplicate_rejections: list[dict[str, Any]] = []
+ for candidate in accepted:
+ duplicate = None
+ for existing in deduplicated:
+ metrics = _bbox_metrics(
+ candidate["bbox_norm_xyxy"], existing["bbox_norm_xyxy"]
+ )
+ if (
+ candidate["host_mode"] == existing["host_mode"]
+ and (
+ metrics["iou"] >= cfg.duplicate_iou
+ or max(metrics["left_coverage"], metrics["right_coverage"])
+ >= cfg.duplicate_containment
+ )
+ ):
+ duplicate = (existing, metrics)
+ break
+ if duplicate is None:
+ deduplicated.append(candidate)
+ else:
+ duplicate_rejections.append(
+ {
+ "source_object_id": candidate["source_object_id"],
+ "duplicate_of_object_id": duplicate[0]["source_object_id"],
+ "overlap": {
+ key: round(value, 6) for key, value in duplicate[1].items()
+ },
+ }
+ )
+
+ overflow = deduplicated[cfg.maximum_hosts :]
+ selected = [dict(item) for item in deduplicated[: cfg.maximum_hosts]]
+ if overflow and cfg.fail_on_budget_overflow:
+ hard_errors.append(
+ {
+ "type": "content_host_budget_overflow",
+ "count": len(overflow),
+ "source_object_ids": [
+ item["source_object_id"] for item in overflow
+ ],
+ }
+ )
+ for host in selected:
+ host["crop_norm_xyxy"] = _initial_crop(host)
+ _adjust_neighbor_padding(selected)
+ for left_index, left in enumerate(selected):
+ for right in selected[left_index + 1 :]:
+ metrics = _bbox_metrics(
+ left["crop_norm_xyxy"], right["crop_norm_xyxy"]
+ )
+ if metrics["iou"] > cfg.maximum_crop_iou:
+ hard_errors.append(
+ {
+ "type": "content_host_crop_ambiguity",
+ "left_object_id": left["source_object_id"],
+ "right_object_id": right["source_object_id"],
+ "crop_iou": round(metrics["iou"], 6),
+ }
+ )
+
+ selected.sort(
+ key=lambda item: (
+ item["bbox_norm_xyxy"][1],
+ item["bbox_norm_xyxy"][0],
+ item["source_object_id"],
+ )
+ )
+ for index, host in enumerate(selected, start=1):
+ host["host_id"] = f"H_{index:03d}"
+ return {
+ "schema_version": "scenemosaic_content_host_selection",
+ "status": "pass" if not hard_errors else "blocked",
+ "config": asdict(cfg),
+ "input_object_count": len(objects),
+ "eligible_before_deduplication": len(accepted),
+ "selected_count": len(selected),
+ "selected": selected,
+ "rejections": rejections,
+ "duplicate_rejections": duplicate_rejections,
+ "overflow_rejections": overflow,
+ "hard_errors": hard_errors,
+ "additional_vlm_calls": 0,
+ }
+
+
+def _norm_to_abs(
+ bbox: Sequence[float], width: int, height: int
+) -> list[int]:
+ x1 = max(0, min(width - 1, int(math.floor(float(bbox[0]) * width))))
+ y1 = max(0, min(height - 1, int(math.floor(float(bbox[1]) * height))))
+ x2 = max(x1 + 1, min(width, int(math.ceil(float(bbox[2]) * width))))
+ y2 = max(y1 + 1, min(height, int(math.ceil(float(bbox[3]) * height))))
+ return [x1, y1, x2, y2]
+
+
+def _draw_global_grid(
+ crop: Image.Image,
+ crop_norm: Sequence[float],
+) -> Image.Image:
+ result = crop.copy()
+ draw = ImageDraw.Draw(result)
+ width, height = result.size
+ font = _font(max(13, min(20, width // 24)))
+ x1, y1, x2, y2 = [float(value) for value in crop_norm]
+ for step in range(1, 4):
+ fraction = step / 4
+ x = int(round(fraction * width))
+ y = int(round(fraction * height))
+ gx = x1 + fraction * (x2 - x1)
+ gy = y1 + fraction * (y2 - y1)
+ draw.line((x, 0, x, height), fill=(0, 210, 230), width=1)
+ draw.line((0, y, width, y), fill=(0, 210, 230), width=1)
+ draw.text((x + 2, 2), f"x={gx:.3f}", fill=(0, 255, 255), font=font)
+ draw.text((2, y + 2), f"y={gy:.3f}", fill=(0, 255, 255), font=font)
+ return result
+
+
+def materialize_content_host_review(
+ *,
+ image_path: str | Path,
+ objects: Sequence[Mapping[str, Any]],
+ output_dir: str | Path,
+ config: ContentHostConfig | None = None,
+ overwrite: bool = False,
+) -> dict[str, Any]:
+ """Render local crops and one numbered content-host panel."""
+
+ output = Path(output_dir).expanduser().resolve()
+ if output.exists() and any(output.iterdir()):
+ if not overwrite:
+ raise FileExistsError(output)
+ shutil.rmtree(output)
+ output.mkdir(parents=True, exist_ok=True)
+ crops_dir = output / "crops"
+ crops_dir.mkdir()
+ selection = select_content_hosts(objects, config=config)
+ image = Image.open(image_path).convert("RGB")
+ width, height = image.size
+ records = []
+ for host in selection["selected"]:
+ record = dict(host)
+ crop_abs = _norm_to_abs(record["crop_norm_xyxy"], width, height)
+ crop = image.crop(tuple(crop_abs))
+ crop = _draw_global_grid(crop, record["crop_norm_xyxy"])
+ crop_path = crops_dir / f"{record['host_id']}_{record['source_object_id']}.jpg"
+ crop.save(crop_path, quality=92)
+ record.update(
+ {
+ "crop_abs_xyxy": crop_abs,
+ "crop_path": str(crop_path),
+ "crop_to_global": {
+ "x_offset_norm": record["crop_norm_xyxy"][0],
+ "y_offset_norm": record["crop_norm_xyxy"][1],
+ "x_scale_norm": round(
+ record["crop_norm_xyxy"][2] - record["crop_norm_xyxy"][0],
+ 6,
+ ),
+ "y_scale_norm": round(
+ record["crop_norm_xyxy"][3] - record["crop_norm_xyxy"][1],
+ 6,
+ ),
+ },
+ }
+ )
+ records.append(record)
+
+ columns = 2
+ cell_width = 640
+ rows = max(1, math.ceil(len(records) / columns))
+ cell_height = max(250, min(360, 1240 // rows))
+ header_height = 40
+ panel = Image.new(
+ "RGB",
+ (columns * cell_width, header_height + rows * cell_height),
+ "white",
+ )
+ ImageDraw.Draw(panel).text(
+ (10, 8),
+ "CONTENT-HOST HIGH-RES REVIEW (H IDs are context only)",
+ fill="black",
+ font=_font(22),
+ )
+ for index, record in enumerate(records):
+ crop = Image.open(record["crop_path"]).convert("RGB")
+ crop.thumbnail(
+ (cell_width - 16, cell_height - 62), Image.Resampling.LANCZOS
+ )
+ x0 = (index % columns) * cell_width
+ y0 = header_height + (index // columns) * cell_height
+ panel.paste(
+ crop,
+ (
+ x0 + (cell_width - crop.width) // 2,
+ y0 + 50 + (cell_height - 58 - crop.height) // 2,
+ ),
+ )
+ label = (
+ f"{record['host_id']} {record['source_object_id']} "
+ f"{record['category']} g={record['crop_norm_xyxy']}"
+ )
+ ImageDraw.Draw(panel).text(
+ (x0 + 8, y0 + 8), label, fill="black", font=_font(17)
+ )
+ panel_path = output / "content_host_panel.jpg"
+ panel.save(panel_path, quality=90)
+ manifest = {
+ **selection,
+ "source_image": str(Path(image_path).expanduser().resolve()),
+ "image_size": [width, height],
+ "selected": records,
+ "content_host_panel": str(panel_path),
+ }
+ manifest_path = output / "content_host_manifest.json"
+ temporary = manifest_path.with_suffix(".json.tmp")
+ temporary.write_text(
+ json.dumps(manifest, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, manifest_path)
+ return {
+ "manifest": manifest,
+ "manifest_path": str(manifest_path),
+ "panel_path": str(panel_path),
+ }
+
+
+def _expanded_abs_crop(
+ bbox: Sequence[int],
+ *,
+ image_size: tuple[int, int],
+ padding_fraction: float = 0.75,
+) -> list[int]:
+ width, height = image_size
+ x1, y1, x2, y2 = [int(value) for value in bbox]
+ box_width = max(1, x2 - x1)
+ box_height = max(1, y2 - y1)
+ target_width = max(
+ int(math.ceil(box_width * (1 + 2 * padding_fraction))),
+ int(math.ceil(width * 0.10)),
+ )
+ target_height = max(
+ int(math.ceil(box_height * (1 + 2 * padding_fraction))),
+ int(math.ceil(height * 0.10)),
+ )
+ center_x = (x1 + x2) / 2
+ center_y = (y1 + y2) / 2
+ left = int(math.floor(center_x - target_width / 2))
+ top = int(math.floor(center_y - target_height / 2))
+ right = left + target_width
+ bottom = top + target_height
+ if left < 0:
+ right -= left
+ left = 0
+ if right > width:
+ left -= right - width
+ right = width
+ if top < 0:
+ bottom -= top
+ top = 0
+ if bottom > height:
+ top -= bottom - height
+ bottom = height
+ return [
+ max(0, left),
+ max(0, top),
+ min(width, right),
+ min(height, bottom),
+ ]
+
+
+def materialize_combined_review_panel(
+ *,
+ image_path: str | Path,
+ local_candidates: Sequence[Mapping[str, Any]],
+ content_hosts: Sequence[Mapping[str, Any]],
+ output_path: str | Path,
+) -> dict[str, Any]:
+ """Pack LC evidence and H attention crops into one bounded JPEG.
+
+ H tiles use two grid slots because they must retain enough resolution for
+ small shelf or tabletop contents. LC tiles use one slot and remain the
+ only entries eligible for the local-candidate accept/reject protocol.
+ """
+
+ candidates = [dict(item) for item in local_candidates]
+ hosts = [dict(item) for item in content_hosts]
+ if not candidates and not hosts:
+ raise ValueError("combined review panel requires candidates or hosts")
+ if len(candidates) > 24:
+ raise ValueError("combined review panel supports at most 24 LC candidates")
+ if len(hosts) > 8:
+ raise ValueError("combined review panel supports at most 8 content hosts")
+ expected_lc_ids = [
+ f"LC_{index:03d}" for index in range(1, len(candidates) + 1)
+ ]
+ actual_lc_ids = [
+ str(item.get("candidate_id") or "") for item in candidates
+ ]
+ if actual_lc_ids != expected_lc_ids:
+ raise ValueError("LC IDs must be continuous and visually ordered")
+ expected_h_ids = [
+ f"H_{index:03d}" for index in range(1, len(hosts) + 1)
+ ]
+ actual_h_ids = [str(item.get("host_id") or "") for item in hosts]
+ if actual_h_ids != expected_h_ids:
+ raise ValueError("content-host IDs must be continuous and visually ordered")
+
+ source = Image.open(image_path).convert("RGB")
+ source_size = source.size
+ items: list[dict[str, Any]] = []
+ for host in hosts:
+ crop_path = Path(str(host.get("crop_path") or "")).expanduser().resolve()
+ if not crop_path.is_file():
+ raise FileNotFoundError(crop_path)
+ items.append(
+ {
+ "kind": "host",
+ "span": (
+ 5 if host.get("host_mode") == "open_storage" else 2
+ ),
+ "host_mode": host.get("host_mode"),
+ "id": host["host_id"],
+ "label": (
+ f"{host['host_id']} {host['source_object_id']} "
+ f"{host['category']} g={host['crop_norm_xyxy']}"
+ ),
+ "image": Image.open(crop_path).convert("RGB"),
+ }
+ )
+ for candidate in candidates:
+ overlay_path = Path(
+ str(candidate.get("overlay_path") or "")
+ ).expanduser().resolve()
+ if not overlay_path.is_file():
+ raise FileNotFoundError(overlay_path)
+ overlay = Image.open(overlay_path).convert("RGB")
+ if overlay.size != source_size:
+ raise ValueError(
+ f"candidate overlay size mismatch: {candidate['candidate_id']}"
+ )
+ bbox = candidate.get("bbox_abs_xyxy")
+ if (
+ not isinstance(bbox, list)
+ or len(bbox) != 4
+ or not all(isinstance(value, int) for value in bbox)
+ ):
+ raise ValueError(
+ f"candidate bbox must be integer xyxy: {candidate['candidate_id']}"
+ )
+ crop_abs = _expanded_abs_crop(bbox, image_size=source_size)
+ prompt = str(
+ candidate.get("prompt")
+ or candidate.get("sam3_prompt")
+ or candidate.get("category_hint")
+ or "object"
+ ).strip()
+ items.append(
+ {
+ "kind": "candidate",
+ "span": 1,
+ "id": candidate["candidate_id"],
+ "label": (
+ f"{candidate['candidate_id']} {prompt} "
+ f"s={float(candidate.get('score') or 0):.3f}"
+ ),
+ "image": overlay.crop(tuple(crop_abs)),
+ }
+ )
+
+ rows: list[list[dict[str, Any]]] = []
+ remaining_hosts = [item for item in items if item["kind"] == "host"]
+ remaining_candidates = [
+ item for item in items if item["kind"] == "candidate"
+ ]
+ while remaining_hosts or remaining_candidates:
+ row: list[dict[str, Any]] = []
+ capacity = 5
+ while remaining_candidates and capacity >= 1:
+ row.append(remaining_candidates.pop(0))
+ capacity -= 1
+ while remaining_hosts:
+ fitting_index = next(
+ (
+ index
+ for index, item in enumerate(remaining_hosts)
+ if int(item["span"]) <= capacity
+ ),
+ None,
+ )
+ if fitting_index is None:
+ break
+ item = remaining_hosts.pop(fitting_index)
+ row.append(item)
+ capacity -= int(item["span"])
+ rows.append(row)
+
+ panel_width = 1280
+ header_height = 40
+ slot_width = panel_width // 5
+ row_weights = [
+ (
+ 4.0
+ if any(
+ item.get("kind") == "host"
+ and item.get("host_mode") == "open_storage"
+ for item in row
+ )
+ else (
+ 1.5
+ if any(item.get("kind") == "candidate" for item in row)
+ else 1.0
+ )
+ )
+ for row in rows
+ ]
+ height_unit = 1240.0 / sum(row_weights)
+ row_heights = [max(72, int(round(height_unit * weight))) for weight in row_weights]
+ row_heights[-1] += 1240 - sum(row_heights)
+ panel = Image.new(
+ "RGB",
+ (panel_width, header_height + sum(row_heights)),
+ "white",
+ )
+ draw = ImageDraw.Draw(panel)
+ draw.text(
+ (10, 8),
+ "PASS-B REVIEW: H=context only; LC=SAM3 mask evidence",
+ fill="black",
+ font=_font(21),
+ )
+ y_offsets = []
+ running_y = header_height
+ for row_height in row_heights:
+ y_offsets.append(running_y)
+ running_y += row_height
+ for row_index, row in enumerate(rows):
+ row_height = row_heights[row_index]
+ slot_index = 0
+ for item in row:
+ cell_width = slot_width * int(item["span"])
+ x0 = slot_index * slot_width
+ y0 = y_offsets[row_index]
+ draw.rectangle(
+ (x0, y0, x0 + cell_width - 1, y0 + row_height - 1),
+ outline=(205, 205, 205),
+ width=1,
+ )
+ draw.text(
+ (x0 + 5, y0 + 5),
+ item["label"],
+ fill="black",
+ font=_font(14 if item["span"] == 1 else 15),
+ )
+ tile = item["image"].copy()
+ tile.thumbnail(
+ (cell_width - 10, row_height - 42),
+ Image.Resampling.LANCZOS,
+ )
+ panel.paste(
+ tile,
+ (
+ x0 + (cell_width - tile.width) // 2,
+ y0 + 36 + (row_height - 38 - tile.height) // 2,
+ ),
+ )
+ slot_index += int(item["span"])
+
+ target = Path(output_path).expanduser().resolve()
+ target.parent.mkdir(parents=True, exist_ok=True)
+ panel.save(target, quality=88)
+ return {
+ "schema_version": "scenemosaic_combined_pass_b_review_panel",
+ "panel_path": str(target),
+ "panel_size": list(panel.size),
+ "row_count": len(rows),
+ "local_candidate_count": len(candidates),
+ "content_host_count": len(hosts),
+ "tile_priority": "local_candidates_then_full_width_open_storage",
+ "row_heights": row_heights,
+ "open_storage_full_width_count": sum(
+ item.get("kind") == "host"
+ and item.get("host_mode") == "open_storage"
+ and int(item.get("span") or 0) == 5
+ for row in rows
+ for item in row
+ ),
+ "request_additional_image_count": 1,
+ }
+
+
+__all__ = [
+ "ContentHostConfig",
+ "materialize_combined_review_panel",
+ "materialize_content_host_review",
+ "select_content_hosts",
+]
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/functional_assembly_refinement.py b/vlm_agent/reconstruction/perception/pipeline/refinements/functional_assembly_refinement.py
new file mode 100644
index 0000000..10e3592
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/functional_assembly_refinement.py
@@ -0,0 +1,2290 @@
+#!/usr/bin/env python3
+"""Bounded functional-assembly normalization and local ROI refinement.
+
+This stage handles two explicit geometry patterns: either a broad rear counter
+claim crosses a visible range/stove, or Pass A already separates a sink
+assembly and an adjacent counter on opposite sides of the range but their SAM3
+prompts still need bounded ROI refinement. It also completes explicit bed
+objects as one physical bed assembly (headboard, frame, mattress, and duvet)
+while subtracting independently inventoried pillow masks. It never publishes
+formal outputs or calls a VLM API.
+"""
+
+from __future__ import annotations
+
+import copy
+import json
+import math
+import os
+import re
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+
+COMPLETE_BED_PROMPT = (
+ "complete bed including frame headboard mattress and duvet excluding pillows"
+)
+COMPLETE_SINK_VANITY_PROMPT = (
+ "complete sink vanity including cabinet countertop basin and fixed faucet"
+)
+COMPLETE_DOUBLE_SINK_VANITY_PROMPT = (
+ "complete double sink vanity including cabinet countertop two basins and fixed faucets"
+)
+COMPLETE_POTTED_PLANT_PROMPT = (
+ "complete potted plant including foliage and pot"
+)
+
+
+@dataclass(frozen=True)
+class AssemblyConfig:
+ maximum_assemblies: int = 2
+ maximum_targets: int = 6
+ maximum_candidates_per_target: int = 16
+ score_threshold: float = 0.20
+ minimum_selected_score: float = 0.45
+ minimum_target_bbox_coverage: float = 0.55
+ minimum_candidate_bbox_coverage: float = 0.35
+ maximum_center_distance_scale: float = 0.75
+ ambiguity_quality_gap: float = 0.12
+ ambiguity_mask_iou: float = 0.20
+ minimum_side_width: float = 0.05
+ maximum_sink_vanity_assemblies: int = 2
+ maximum_fixed_faucet_unions: int = 2
+ minimum_source_mask_coverage: float = 0.85
+ maximum_protected_object_removal_fraction: float = 0.35
+
+
+@dataclass(frozen=True)
+class PlantContainerMaskCompletionConfig:
+ maximum_assemblies: int = 2
+ minimum_candidate_score: float = 0.60
+ minimum_candidate_overlap: float = 0.35
+ maximum_candidate_overlap: float = 0.92
+ minimum_new_pixels: int = 64
+ minimum_new_pixel_bbox_fraction: float = 0.75
+ minimum_candidate_bbox_fraction: float = 0.70
+ minimum_downward_extension_pixels: int = 8
+ maximum_other_object_overlap: float = 0.05
+ ambiguity_score_gap: float = 0.05
+
+
+def _tokens(obj: Mapping[str, Any]) -> set[str]:
+ text = " ".join(
+ str(obj.get(key) or "")
+ for key in ("category", "sam3_prompt", "instance_name", "description")
+ ).lower()
+ return set(re.findall(r"[a-z0-9]+", text))
+
+
+def _category_tokens(obj: Mapping[str, Any]) -> set[str]:
+ return set(
+ re.findall(
+ r"[a-z0-9]+",
+ str(obj.get("category") or "").lower().replace("_", " "),
+ )
+ )
+
+
+def _bbox(obj: Mapping[str, Any]) -> list[float]:
+ try:
+ box = [float(value) for value in obj.get("bbox_2d_norm_xyxy")]
+ except (TypeError, ValueError):
+ raise ValueError("invalid bbox") from None
+ if (
+ len(box) != 4
+ or not all(math.isfinite(value) for value in box)
+ or not (0.0 <= box[0] < box[2] <= 1.0)
+ or not (0.0 <= box[1] < box[3] <= 1.0)
+ ):
+ raise ValueError("invalid bbox")
+ return box
+
+
+def _round_box(box: Sequence[float]) -> list[float]:
+ return [round(float(value), 6) for value in box]
+
+
+def _center(box: Sequence[float]) -> tuple[float, float]:
+ return ((box[0] + box[2]) / 2.0, (box[1] + box[3]) / 2.0)
+
+
+def _overlap_1d(a1: float, a2: float, b1: float, b2: float) -> float:
+ return max(0.0, min(a2, b2) - max(a1, b1))
+
+
+def _is_sink(obj: Mapping[str, Any]) -> bool:
+ return bool(_tokens(obj) & {"sink", "washbasin", "basin"})
+
+
+def _is_counter_support(obj: Mapping[str, Any]) -> bool:
+ tokens = _category_tokens(obj)
+ if _is_sink(obj) or tokens & {"island", "pantry", "upper", "wall"}:
+ return False
+ return bool(tokens & {"counter", "countertop", "worktable"}) or (
+ {"base", "cabinet"} <= tokens
+ )
+
+
+def _is_separator(obj: Mapping[str, Any]) -> bool:
+ tokens = _category_tokens(obj)
+ return bool(tokens & {"range", "stove"}) and not bool(
+ tokens & {"hood", "vent"}
+ )
+
+
+def _is_bed(obj: Mapping[str, Any]) -> bool:
+ category = _category_tokens(obj)
+ return "bed" in category and not bool(
+ category & {"bench", "bedside", "headboard"}
+ )
+
+
+def _is_pillow(obj: Mapping[str, Any]) -> bool:
+ category = _category_tokens(obj)
+ return bool(category & {"pillow", "cushion"}) and not bool(
+ category & {"bench", "chair", "sofa"}
+ )
+
+
+def _is_plant(obj: Mapping[str, Any]) -> bool:
+ return "plant" in _category_tokens(obj)
+
+
+def _is_vase_like(obj: Mapping[str, Any]) -> bool:
+ return bool(_category_tokens(obj) & {"pot", "planter", "vase"})
+
+
+def _match_potted_plant_pattern(
+ plant: Mapping[str, Any],
+ vessel: Mapping[str, Any],
+) -> dict[str, Any] | None:
+ """Prove a foliage/vessel pair is one potted-plant assembly from boxes."""
+
+ plant_box = _bbox(plant)
+ vessel_box = _bbox(vessel)
+ vessel_width = vessel_box[2] - vessel_box[0]
+ vessel_height = vessel_box[3] - vessel_box[1]
+ plant_area = (plant_box[2] - plant_box[0]) * (plant_box[3] - plant_box[1])
+ vessel_area = vessel_width * vessel_height
+ horizontal_coverage = _overlap_1d(
+ plant_box[0], plant_box[2], vessel_box[0], vessel_box[2]
+ ) / max(vessel_width, 1e-9)
+ vertical_coverage = _overlap_1d(
+ plant_box[1], plant_box[3], vessel_box[1], vessel_box[3]
+ ) / max(vessel_height, 1e-9)
+ if (
+ horizontal_coverage < 0.90
+ or vertical_coverage < 0.40
+ or vessel_area > 0.75 * plant_area
+ or _center(vessel_box)[1] < _center(plant_box)[1]
+ ):
+ return None
+ union = [
+ min(plant_box[0], vessel_box[0]),
+ min(plant_box[1], vessel_box[1]),
+ max(plant_box[2], vessel_box[2]),
+ max(plant_box[3], vessel_box[3]),
+ ]
+ return {
+ "plant": plant,
+ "vessel": vessel,
+ "target_bbox_norm_xyxy": _round_box(union),
+ "vessel_horizontal_coverage": round(horizontal_coverage, 6),
+ "vessel_vertical_coverage": round(vertical_coverage, 6),
+ "vessel_to_plant_bbox_area_ratio": round(
+ vessel_area / max(plant_area, 1e-9), 6
+ ),
+ }
+
+
+def _is_sink_vanity(obj: Mapping[str, Any]) -> bool:
+ """A cabinet-mounted sink whose fixed faucet belongs to the same instance."""
+
+ category = _category_tokens(obj)
+ return (
+ "vanity" in category
+ and bool(category & {"sink", "basin"})
+ and "kitchen" not in _tokens(obj)
+ )
+
+
+def _is_bathtub(obj: Mapping[str, Any]) -> bool:
+ category = _category_tokens(obj)
+ return "bathtub" in category and "faucet" not in category
+
+
+def _is_fixed_bathtub_faucet(obj: Mapping[str, Any]) -> bool:
+ category = _category_tokens(obj)
+ return "bathtub" in category and "faucet" in category
+
+
+def _is_fixed_faucet(obj: Mapping[str, Any]) -> bool:
+ return "faucet" in _category_tokens(obj)
+
+
+def _expand_box_upward(box: Sequence[float], fraction: float = 0.18) -> list[float]:
+ """Reserve only the shallow band where fixed taps sit above a vanity."""
+
+ height = box[3] - box[1]
+ return _round_box(
+ [
+ box[0],
+ max(0.0, box[1] - max(0.04, min(0.12, height * fraction))),
+ box[2],
+ box[3],
+ ]
+ )
+
+
+def _is_absorbable_bed_component(obj: Mapping[str, Any]) -> bool:
+ """Keep physical bed parts in the assembly while protecting other objects."""
+
+ category = _category_tokens(obj)
+ return bool(
+ category
+ & {
+ "bedframe",
+ "bedding",
+ "bedspread",
+ "blanket",
+ "comforter",
+ "duvet",
+ "frame",
+ "headboard",
+ "mattress",
+ "quilt",
+ "sheet",
+ }
+ )
+
+
+def _is_absorbable_component(obj: Mapping[str, Any]) -> bool:
+ tokens = _tokens(obj)
+ return "dishwasher" in tokens or (
+ {"under", "counter", "appliance"} <= tokens
+ )
+
+
+def _match_pattern(
+ sink: Mapping[str, Any],
+ support: Mapping[str, Any],
+ separator: Mapping[str, Any],
+ cfg: AssemblyConfig,
+) -> dict[str, Any] | None:
+ sink_box = _bbox(sink)
+ support_box = _bbox(support)
+ separator_box = _bbox(separator)
+ if not (
+ support_box[0] + cfg.minimum_side_width < separator_box[0]
+ and separator_box[2] + cfg.minimum_side_width < support_box[2]
+ ):
+ return None
+ if _overlap_1d(
+ support_box[1], support_box[3], separator_box[1], separator_box[3]
+ ) <= 0.0:
+ return None
+ sink_center_x, sink_center_y = _center(sink_box)
+ if not (
+ support_box[0] <= sink_center_x <= support_box[2]
+ and support_box[1] - 0.12 <= sink_center_y <= support_box[3] + 0.12
+ ):
+ return None
+ separator_width = separator_box[2] - separator_box[0]
+ if (
+ sink_center_x < separator_box[0]
+ and sink_box[2] <= separator_box[0] + 0.25 * separator_width
+ ):
+ side = "left"
+ elif (
+ sink_center_x > separator_box[2]
+ and sink_box[0] >= separator_box[2] - 0.25 * separator_width
+ ):
+ side = "right"
+ else:
+ return None
+ return {
+ "pattern_mode": "broad_counter_crosses_separator",
+ "side": side,
+ "distance": min(
+ abs(sink_center_x - separator_box[0]),
+ abs(sink_center_x - separator_box[2]),
+ ),
+ "sink_box": sink_box,
+ "support_box": support_box,
+ "separator_box": separator_box,
+ "sink": sink,
+ "support": support,
+ "separator": separator,
+ }
+
+
+def _match_partitioned_pattern(
+ sink: Mapping[str, Any],
+ support: Mapping[str, Any],
+ separator: Mapping[str, Any],
+ cfg: AssemblyConfig,
+) -> dict[str, Any] | None:
+ "Match an already-correct sink/range/counter left-to-right partition."
+
+ sink_tokens = _tokens(sink)
+ if not ({"counter", "assembly", "vanity", "cabinet"} & sink_tokens):
+ return None
+ sink_box = _bbox(sink)
+ support_box = _bbox(support)
+ separator_box = _bbox(separator)
+ if (
+ _overlap_1d(
+ sink_box[1], sink_box[3], separator_box[1], separator_box[3]
+ )
+ <= 0.0
+ or _overlap_1d(
+ support_box[1],
+ support_box[3],
+ separator_box[1],
+ separator_box[3],
+ )
+ <= 0.0
+ ):
+ return None
+ sink_center_x, sink_center_y = _center(sink_box)
+ support_center_x, support_center_y = _center(support_box)
+ separator_center_x, separator_center_y = _center(separator_box)
+ vertical_scale = max(
+ sink_box[3] - sink_box[1],
+ support_box[3] - support_box[1],
+ separator_box[3] - separator_box[1],
+ 0.01,
+ )
+ if max(
+ abs(sink_center_y - separator_center_y),
+ abs(support_center_y - separator_center_y),
+ ) > max(0.18, 1.25 * vertical_scale):
+ return None
+ if sink_center_x < separator_center_x < support_center_x:
+ side = "left"
+ sink_gap = max(0.0, separator_box[0] - sink_box[2])
+ support_gap = max(0.0, support_box[0] - separator_box[2])
+ elif support_center_x < separator_center_x < sink_center_x:
+ side = "right"
+ sink_gap = max(0.0, sink_box[0] - separator_box[2])
+ support_gap = max(0.0, separator_box[0] - support_box[2])
+ else:
+ return None
+ if max(sink_gap, support_gap) > 0.12:
+ return None
+ if min(
+ sink_box[2] - sink_box[0],
+ support_box[2] - support_box[0],
+ ) < cfg.minimum_side_width:
+ return None
+ return {
+ "pattern_mode": "already_partitioned_needs_roi",
+ "side": side,
+ "distance": sink_gap + support_gap,
+ "sink_box": sink_box,
+ "support_box": support_box,
+ "separator_box": separator_box,
+ "sink": sink,
+ "support": support,
+ "separator": separator,
+ }
+
+
+def _blocked_normalization(
+ objects: Sequence[Mapping[str, Any]],
+ cfg: AssemblyConfig,
+ hard_errors: list[dict[str, Any]],
+) -> dict[str, Any]:
+ copied = [copy.deepcopy(dict(item)) for item in objects]
+ return {
+ "objects": copied,
+ "audit": {
+ "schema_version": "scenemosaic_functional_assembly_normalization",
+ "status": "blocked",
+ "config": asdict(cfg),
+ "raw_object_count": len(copied),
+ "normalized_object_count": len(copied),
+ "record_count": 0,
+ "records": [],
+ "targets": [],
+ "id_remap": {},
+ "removed_source_object_ids": [],
+ "hard_errors": hard_errors,
+ "formal_outputs_modified": False,
+ },
+ }
+
+
+def normalize_pass_a_functional_assemblies(
+ objects: Sequence[Mapping[str, Any]],
+ config: AssemblyConfig | None = None,
+) -> dict[str, Any]:
+ """Normalize explicit sink/counter and complete-bed assemblies."""
+
+ cfg = config or AssemblyConfig()
+ if not 1 <= cfg.maximum_assemblies <= 4:
+ raise ValueError("maximum_assemblies must be in [1, 4]")
+ if not 0 <= cfg.maximum_sink_vanity_assemblies <= 4:
+ raise ValueError("maximum_sink_vanity_assemblies must be in [0, 4]")
+ if not 0 <= cfg.maximum_fixed_faucet_unions <= 4:
+ raise ValueError("maximum_fixed_faucet_unions must be in [0, 4]")
+ if not 0.0 < cfg.minimum_source_mask_coverage <= 1.0:
+ raise ValueError("minimum_source_mask_coverage must be in (0, 1]")
+ if not 0.0 <= cfg.maximum_protected_object_removal_fraction < 1.0:
+ raise ValueError("maximum_protected_object_removal_fraction must be in [0, 1)")
+ copied = [copy.deepcopy(dict(item)) for item in objects]
+ ids = [str(item.get("object_id") or "") for item in copied]
+ errors: list[dict[str, Any]] = []
+ if not all(ids) or len(set(ids)) != len(ids):
+ errors.append({"type": "invalid_or_duplicate_pass_a_object_id"})
+ for item in copied:
+ try:
+ _bbox(item)
+ except ValueError:
+ errors.append(
+ {
+ "type": "invalid_functional_assembly_bbox",
+ "object_id": str(item.get("object_id") or ""),
+ }
+ )
+ if errors:
+ return _blocked_normalization(objects, cfg, errors)
+
+ sinks = [item for item in copied if _is_sink(item)]
+ supports = [item for item in copied if _is_counter_support(item)]
+ separators = [item for item in copied if _is_separator(item)]
+ matches = []
+ for sink in sinks:
+ candidates = [
+ evidence
+ for support in supports
+ for separator in separators
+ if (
+ evidence := _match_pattern(
+ sink, support, separator, cfg
+ )
+ )
+ is not None
+ ]
+ candidates.sort(
+ key=lambda item: (
+ float(item["distance"]),
+ str(item["support"].get("object_id") or ""),
+ str(item["separator"].get("object_id") or ""),
+ )
+ )
+ if len(candidates) > 1 and abs(
+ float(candidates[0]["distance"])
+ - float(candidates[1]["distance"])
+ ) <= 0.02:
+ errors.append(
+ {
+ "type": "ambiguous_functional_assembly_pattern",
+ "sink_object_id": str(sink.get("object_id") or ""),
+ "candidate_count": len(candidates),
+ }
+ )
+ elif candidates:
+ matches.append(candidates[0])
+ matched_sink_ids = {
+ str(item["sink"].get("object_id") or "") for item in matches
+ }
+ for sink in sinks:
+ if str(sink.get("object_id") or "") in matched_sink_ids:
+ continue
+ candidates = [
+ evidence
+ for support in supports
+ for separator in separators
+ if (
+ evidence := _match_partitioned_pattern(
+ sink, support, separator, cfg
+ )
+ )
+ is not None
+ ]
+ candidates.sort(
+ key=lambda item: (
+ float(item["distance"]),
+ str(item["support"].get("object_id") or ""),
+ str(item["separator"].get("object_id") or ""),
+ )
+ )
+ if len(candidates) > 1 and abs(
+ float(candidates[0]["distance"])
+ - float(candidates[1]["distance"])
+ ) <= 0.02:
+ errors.append(
+ {
+ "type": "ambiguous_partitioned_functional_assembly",
+ "sink_object_id": str(sink.get("object_id") or ""),
+ "candidate_count": len(candidates),
+ }
+ )
+ elif candidates:
+ matches.append(candidates[0])
+ if len(matches) > cfg.maximum_assemblies:
+ errors.append(
+ {
+ "type": "functional_assembly_capacity_overflow",
+ "actual": len(matches),
+ "maximum": cfg.maximum_assemblies,
+ }
+ )
+ used_ids: set[str] = set()
+ for match in matches:
+ pair = {
+ str(match["sink"]["object_id"]),
+ str(match["support"]["object_id"]),
+ }
+ if used_ids & pair:
+ errors.append(
+ {
+ "type": "functional_assembly_object_reused",
+ "object_ids": sorted(used_ids & pair),
+ }
+ )
+ used_ids |= pair
+ if errors:
+ return _blocked_normalization(objects, cfg, errors)
+
+ by_id = {str(item["object_id"]): item for item in copied}
+ removed_ids: set[str] = set()
+ raw_records = []
+ raw_targets = []
+ for match in matches:
+ sink = by_id[str(match["sink"]["object_id"])]
+ support = by_id[str(match["support"]["object_id"])]
+ separator = by_id[str(match["separator"]["object_id"])]
+ sink_box = list(match["sink_box"])
+ support_box = list(match["support_box"])
+ separator_box = list(match["separator_box"])
+ side = str(match["side"])
+ pattern_mode = str(match.get("pattern_mode") or "")
+ sep_w = separator_box[2] - separator_box[0]
+ sep_h = separator_box[3] - separator_box[1]
+ if side == "left":
+ edge = separator_box[0] + 0.24 * sep_w
+ if pattern_mode == "already_partitioned_needs_roi":
+ sink_target = [
+ sink_box[0],
+ max(
+ 0.0,
+ min(
+ sink_box[1],
+ separator_box[1] - 0.30 * sep_h,
+ ),
+ ),
+ max(sink_box[2], edge),
+ max(sink_box[3], separator_box[3]),
+ ]
+ else:
+ sink_target = [
+ min(sink_box[0], support_box[0]),
+ min(sink_box[1], support_box[1]),
+ max(sink_box[2], edge),
+ max(sink_box[3], support_box[3]),
+ ]
+ counter_target = [
+ separator_box[2] - 0.075 * sep_w,
+ min(support_box[1], separator_box[1] - 0.10 * sep_h),
+ support_box[2],
+ min(support_box[3], separator_box[3] - 0.20 * sep_h),
+ ]
+ counter_relation = "right of gas range"
+ sink_relation = "left"
+ sink_limit = {"maximum_x_norm": round(edge, 6)}
+ counter_limit = {
+ "minimum_x_norm": round(separator_box[2] - 0.10 * sep_w, 6)
+ }
+ else:
+ edge = separator_box[2] - 0.24 * sep_w
+ if pattern_mode == "already_partitioned_needs_roi":
+ sink_target = [
+ min(sink_box[0], edge),
+ max(
+ 0.0,
+ min(
+ sink_box[1],
+ separator_box[1] - 0.30 * sep_h,
+ ),
+ ),
+ sink_box[2],
+ max(sink_box[3], separator_box[3]),
+ ]
+ else:
+ sink_target = [
+ min(sink_box[0], edge),
+ min(sink_box[1], support_box[1]),
+ max(sink_box[2], support_box[2]),
+ max(sink_box[3], support_box[3]),
+ ]
+ counter_target = [
+ support_box[0],
+ min(support_box[1], separator_box[1] - 0.10 * sep_h),
+ separator_box[0] + 0.075 * sep_w,
+ min(support_box[3], separator_box[3] - 0.20 * sep_h),
+ ]
+ counter_relation = "left of gas range"
+ sink_relation = "right"
+ sink_limit = {"minimum_x_norm": round(edge, 6)}
+ counter_limit = {
+ "maximum_x_norm": round(separator_box[0] + 0.10 * sep_w, 6)
+ }
+ if (
+ counter_target[2] - counter_target[0] < cfg.minimum_side_width
+ or counter_target[3] - counter_target[1] < cfg.minimum_side_width
+ ):
+ errors.append(
+ {
+ "type": "functional_assembly_partition_too_small",
+ "sink_object_id": str(sink["object_id"]),
+ "support_object_id": str(support["object_id"]),
+ }
+ )
+ continue
+
+ absorbed = []
+ for component in copied:
+ component_id = str(component.get("object_id") or "")
+ if component_id in {
+ str(sink["object_id"]),
+ str(support["object_id"]),
+ str(separator["object_id"]),
+ } or not _is_absorbable_component(component):
+ continue
+ cx, cy = _center(_bbox(component))
+ if (
+ sink_target[0] <= cx <= sink_target[2]
+ and sink_target[1] <= cy <= sink_target[3]
+ ):
+ absorbed.append(component_id)
+ removed_ids.add(component_id)
+
+ old_sink_id = str(sink["object_id"])
+ old_support_id = str(support["object_id"])
+ old_separator_id = str(separator["object_id"])
+ sink.update(
+ {
+ "category": "kitchen_sink_counter_assembly",
+ "sam3_prompt": (
+ "complete kitchen sink counter assembly including countertop "
+ "base cabinets dishwasher sink and faucet"
+ ),
+ "instance_name": f"{sink_relation} kitchen sink counter assembly",
+ "description": (
+ "complete sink faucet countertop base cabinets and fitted dishwasher"
+ ),
+ "object_type": "furniture",
+ "bbox_2d_norm_xyxy": _round_box(sink_target),
+ "anchor_2d_norm_xy": [
+ round((sink_target[0] + sink_target[2]) / 2.0, 6),
+ round((sink_target[1] + sink_target[3]) / 2.0, 6),
+ ],
+ }
+ )
+ support.update(
+ {
+ "category": "kitchen_counter_cabinet",
+ "sam3_prompt": (
+ "wooden kitchen base cabinet with white countertop "
+ f"{counter_relation}"
+ ),
+ "instance_name": f"counter cabinet {counter_relation}",
+ "description": f"rear counter cabinet immediately {counter_relation}",
+ "object_type": "furniture",
+ "bbox_2d_norm_xyxy": _round_box(counter_target),
+ "anchor_2d_norm_xy": [
+ round((counter_target[0] + counter_target[2]) / 2.0, 6),
+ round((counter_target[1] + counter_target[3]) / 2.0, 6),
+ ],
+ }
+ )
+ raw_targets.extend(
+ [
+ {
+ "old_source_object_id": old_sink_id,
+ "role": "sink_counter_assembly",
+ "prompt": sink["sam3_prompt"],
+ "bbox_norm_xyxy": list(sink["bbox_2d_norm_xyxy"]),
+ "hard_ownership_limit": sink_limit,
+ },
+ {
+ "old_source_object_id": old_support_id,
+ "role": "adjacent_counter_cabinet",
+ "prompt": support["sam3_prompt"],
+ "bbox_norm_xyxy": list(support["bbox_2d_norm_xyxy"]),
+ "hard_ownership_limit": counter_limit,
+ },
+ ]
+ )
+ raw_records.append(
+ {
+ "pattern_mode": pattern_mode,
+ "sink_source_object_id_before": old_sink_id,
+ "support_source_object_id_before": old_support_id,
+ "separator_source_object_id_before": old_separator_id,
+ "sink_side": side,
+ "absorbed_component_ids_before": sorted(absorbed),
+ "sink_target_bbox_norm_xyxy": list(sink["bbox_2d_norm_xyxy"]),
+ "counter_target_bbox_norm_xyxy": list(
+ support["bbox_2d_norm_xyxy"]
+ ),
+ }
+ )
+ if errors:
+ return _blocked_normalization(objects, cfg, errors)
+
+ raw_bed_records = []
+ raw_bed_targets = []
+ for bed in copied:
+ old_bed_id = str(bed.get("object_id") or "")
+ if old_bed_id in removed_ids or not _is_bed(bed):
+ continue
+ original_prompt = str(bed.get("sam3_prompt") or "")
+ original_description = str(bed.get("description") or "")
+ bed.update(
+ {
+ "sam3_prompt": COMPLETE_BED_PROMPT,
+ "description": (
+ "complete physical bed including headboard frame mattress "
+ "and duvet; independent pillows excluded"
+ ),
+ "object_type": "furniture",
+ }
+ )
+ raw_bed_targets.append(
+ {
+ "old_source_object_id": old_bed_id,
+ "role": "complete_bed_assembly",
+ "prompt": COMPLETE_BED_PROMPT,
+ "bbox_norm_xyxy": list(bed["bbox_2d_norm_xyxy"]),
+ "hard_ownership_limit": {},
+ }
+ )
+ raw_bed_records.append(
+ {
+ "pattern_mode": "explicit_complete_bed_assembly",
+ "bed_source_object_id_before": old_bed_id,
+ "original_prompt": original_prompt,
+ "original_description": original_description,
+ "normalized_prompt": COMPLETE_BED_PROMPT,
+ }
+ )
+
+ raw_vanity_records = []
+ raw_vanity_targets = []
+ vanities = [
+ item
+ for item in copied
+ if str(item.get("object_id") or "") not in removed_ids
+ and _is_sink_vanity(item)
+ ]
+ if len(vanities) > cfg.maximum_sink_vanity_assemblies:
+ return _blocked_normalization(
+ objects,
+ cfg,
+ [
+ {
+ "type": "sink_vanity_assembly_capacity_overflow",
+ "actual": len(vanities),
+ "maximum": cfg.maximum_sink_vanity_assemblies,
+ }
+ ],
+ )
+ for vanity in vanities:
+ old_vanity_id = str(vanity.get("object_id") or "")
+ original_prompt = str(vanity.get("sam3_prompt") or "")
+ original_description = str(vanity.get("description") or "")
+ prompt = (
+ COMPLETE_DOUBLE_SINK_VANITY_PROMPT
+ if "double" in _category_tokens(vanity)
+ else COMPLETE_SINK_VANITY_PROMPT
+ )
+ target_box = _expand_box_upward(_bbox(vanity))
+ vanity.update(
+ {
+ "sam3_prompt": prompt,
+ "description": (
+ "complete sink vanity including cabinet countertop basin "
+ "and fixed faucet"
+ ),
+ "object_type": "furniture",
+ "bbox_2d_norm_xyxy": target_box,
+ "anchor_2d_norm_xy": [
+ round((target_box[0] + target_box[2]) / 2.0, 6),
+ round((target_box[1] + target_box[3]) / 2.0, 6),
+ ],
+ }
+ )
+ raw_vanity_targets.append(
+ {
+ "old_source_object_id": old_vanity_id,
+ "role": "complete_sink_vanity_assembly",
+ "prompt": prompt,
+ "bbox_norm_xyxy": list(target_box),
+ "hard_ownership_limit": {},
+ "require_source_mask_coverage": cfg.minimum_source_mask_coverage,
+ "protect_independent_objects": True,
+ }
+ )
+ raw_vanity_records.append(
+ {
+ "pattern_mode": "explicit_complete_sink_vanity_assembly",
+ "vanity_source_object_id_before": old_vanity_id,
+ "original_prompt": original_prompt,
+ "original_description": original_description,
+ "normalized_prompt": prompt,
+ "target_bbox_norm_xyxy": list(target_box),
+ }
+ )
+
+ raw_botanical_records = []
+ plants = [
+ item
+ for item in copied
+ if str(item.get("object_id") or "") not in removed_ids and _is_plant(item)
+ ]
+ vessels = [
+ item
+ for item in copied
+ if (
+ str(item.get("object_id") or "") not in removed_ids
+ and _is_vase_like(item)
+ )
+ ]
+ botanical_candidates = [
+ match
+ for plant in plants
+ for vessel in vessels
+ if (
+ match := _match_potted_plant_pattern(plant, vessel)
+ ) is not None
+ ]
+ candidates_by_plant: dict[str, list[dict[str, Any]]] = {}
+ candidates_by_vessel: dict[str, list[dict[str, Any]]] = {}
+ for match in botanical_candidates:
+ plant_id = str(match["plant"].get("object_id") or "")
+ vessel_id = str(match["vessel"].get("object_id") or "")
+ candidates_by_plant.setdefault(plant_id, []).append(match)
+ candidates_by_vessel.setdefault(vessel_id, []).append(match)
+ for match in sorted(
+ botanical_candidates,
+ key=lambda item: (
+ str(item["plant"].get("object_id") or ""),
+ str(item["vessel"].get("object_id") or ""),
+ ),
+ ):
+ plant = match["plant"]
+ vessel = match["vessel"]
+ plant_id = str(plant.get("object_id") or "")
+ vessel_id = str(vessel.get("object_id") or "")
+ if (
+ len(candidates_by_plant[plant_id]) != 1
+ or len(candidates_by_vessel[vessel_id]) != 1
+ ):
+ continue
+ target_box = list(match["target_bbox_norm_xyxy"])
+ plant.update(
+ {
+ "category": "potted_plant",
+ "sam3_prompt": COMPLETE_POTTED_PLANT_PROMPT,
+ "instance_name": "potted plant",
+ "description": "complete potted plant including foliage and pot",
+ "object_type": "plant",
+ "bbox_2d_norm_xyxy": target_box,
+ "anchor_2d_norm_xy": [
+ round((target_box[0] + target_box[2]) / 2.0, 6),
+ round((target_box[1] + target_box[3]) / 2.0, 6),
+ ],
+ }
+ )
+ removed_ids.add(vessel_id)
+ raw_botanical_records.append(
+ {
+ "pattern_mode": "plant_vessel_potted_plant_assembly",
+ "plant_source_object_id_before": plant_id,
+ "vessel_source_object_id_before": vessel_id,
+ "normalized_prompt": COMPLETE_POTTED_PLANT_PROMPT,
+ "target_bbox_norm_xyxy": target_box,
+ "vessel_horizontal_coverage": match[
+ "vessel_horizontal_coverage"
+ ],
+ "vessel_vertical_coverage": match["vessel_vertical_coverage"],
+ "vessel_to_plant_bbox_area_ratio": match[
+ "vessel_to_plant_bbox_area_ratio"
+ ],
+ }
+ )
+
+ retained = [
+ item for item in copied if str(item["object_id"]) not in removed_ids
+ ]
+ id_remap = {}
+ for index, item in enumerate(retained, start=1):
+ old_id = str(item["object_id"])
+ new_id = f"A_{index:03d}"
+ id_remap[old_id] = new_id
+ item["object_id"] = new_id
+ records = [
+ {
+ **raw,
+ "sink_source_object_id_after": id_remap[
+ raw["sink_source_object_id_before"]
+ ],
+ "support_source_object_id_after": id_remap[
+ raw["support_source_object_id_before"]
+ ],
+ "separator_source_object_id_after": id_remap[
+ raw["separator_source_object_id_before"]
+ ],
+ }
+ for raw in raw_records
+ ]
+ records.extend(
+ {
+ **raw,
+ "bed_source_object_id_after": id_remap[
+ raw["bed_source_object_id_before"]
+ ],
+ }
+ for raw in raw_bed_records
+ )
+ records.extend(
+ {
+ **raw,
+ "vanity_source_object_id_after": id_remap[
+ raw["vanity_source_object_id_before"]
+ ],
+ }
+ for raw in raw_vanity_records
+ )
+ records.extend(
+ {
+ **raw,
+ "potted_plant_source_object_id_after": id_remap[
+ raw["plant_source_object_id_before"]
+ ],
+ }
+ for raw in raw_botanical_records
+ )
+ targets = [
+ {
+ **target,
+ "source_object_id": id_remap[target["old_source_object_id"]],
+ }
+ for target in raw_targets
+ ]
+ targets.extend(
+ {
+ **target,
+ "source_object_id": id_remap[target["old_source_object_id"]],
+ }
+ for target in raw_bed_targets
+ )
+ targets.extend(
+ {
+ **target,
+ "source_object_id": id_remap[target["old_source_object_id"]],
+ }
+ for target in raw_vanity_targets
+ )
+ audit = {
+ "schema_version": "scenemosaic_functional_assembly_normalization",
+ "status": "pass" if records else "noop",
+ "config": asdict(cfg),
+ "raw_object_count": len(objects),
+ "normalized_object_count": len(retained),
+ "record_count": len(records),
+ "records": records,
+ "targets": targets,
+ "id_remap": id_remap,
+ "removed_source_object_ids": sorted(removed_ids),
+ "hard_errors": [],
+ "formal_outputs_modified": False,
+ }
+ return {"objects": retained, "audit": audit}
+
+
+def _load_mask(path_value: Any, shape: tuple[int, int]) -> np.ndarray:
+ path = Path(str(path_value or "")).expanduser().resolve()
+ if not path.is_file():
+ raise FileNotFoundError(path)
+ mask = np.array(Image.open(path).convert("L")) > 0
+ if mask.shape != shape or not mask.any():
+ raise ValueError(f"invalid mask: {path}")
+ return mask
+
+
+def _mask_bbox(mask: np.ndarray) -> list[int]:
+ ys, xs = np.nonzero(mask)
+ if not len(xs):
+ raise ValueError("empty mask")
+ return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
+
+
+def _bbox_metrics(
+ target: Sequence[int], candidate: Sequence[int]
+) -> dict[str, float]:
+ ix = max(0, min(target[2], candidate[2]) - max(target[0], candidate[0]))
+ iy = max(0, min(target[3], candidate[3]) - max(target[1], candidate[1]))
+ intersection = ix * iy
+ target_area = max(1, (target[2] - target[0]) * (target[3] - target[1]))
+ candidate_area = max(
+ 1, (candidate[2] - candidate[0]) * (candidate[3] - candidate[1])
+ )
+ target_center = _center(target)
+ candidate_center = _center(candidate)
+ scale = max(
+ math.hypot(target[2] - target[0], target[3] - target[1]),
+ math.hypot(candidate[2] - candidate[0], candidate[3] - candidate[1]),
+ 1.0,
+ )
+ return {
+ "target_bbox_coverage": intersection / target_area,
+ "candidate_bbox_coverage": intersection / candidate_area,
+ "center_distance_scale": math.dist(target_center, candidate_center) / scale,
+ }
+
+
+def _mask_iou(left: np.ndarray, right: np.ndarray) -> float:
+ union = int(np.logical_or(left, right).sum())
+ return int(np.logical_and(left, right).sum()) / union if union else 0.0
+
+
+def _atomic_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _is_plant_container_assembly(obj: Mapping[str, Any]) -> bool:
+ """Return whether one declaration explicitly includes foliage and a vessel."""
+
+ tokens = _tokens(obj)
+ has_container = bool(
+ tokens & {"pot", "potted", "planter", "vase", "arrangement"}
+ )
+ has_botanical_content = bool(
+ tokens
+ & {
+ "plant",
+ "greenery",
+ "foliage",
+ "branch",
+ "branches",
+ "flower",
+ "flowers",
+ "floral",
+ }
+ )
+ return has_container and has_botanical_content
+
+
+def _absolute_bbox_from_normalized(
+ box: Sequence[float], width: int, height: int
+) -> list[int]:
+ return [
+ max(0, min(width - 1, int(round(float(box[0]) * width)))),
+ max(0, min(height - 1, int(round(float(box[1]) * height)))),
+ max(1, min(width, int(round(float(box[2]) * width)))),
+ max(1, min(height, int(round(float(box[3]) * height)))),
+ ]
+
+
+def _mask_fraction_within_bbox(mask: np.ndarray, box: Sequence[int]) -> float:
+ """Return the fraction of one mask whose pixels lie inside box."""
+
+ x1, y1, x2, y2 = [int(value) for value in box]
+ return float(mask[y1:y2, x1:x2].sum()) / float(max(1, mask.sum()))
+
+
+def complete_plant_container_masks(
+ *,
+ image_path: str | Path,
+ registry: Mapping[str, Any],
+ output_dir: str | Path,
+ config: PlantContainerMaskCompletionConfig | None = None,
+) -> dict[str, Any]:
+ """Union a proven foliage-plus-vessel candidate into one plant assembly.
+
+ This is intentionally bounded to a pre-existing plant-container declaration.
+ It never creates an object, never queries SAM3 or a VLM, and only uses an
+ unassigned high-confidence candidate that mostly lies in the declaration's
+ bbox while contributing a clear lower extension. That makes the completion
+ suitable for a vase, pot, or planter that SAM3 separated from its foliage.
+ """
+
+ cfg = config or PlantContainerMaskCompletionConfig()
+ output = Path(output_dir).expanduser().resolve()
+ if output.exists() and any(output.iterdir()):
+ raise FileExistsError(
+ f"plant-container completion output is not empty: {output}"
+ )
+ output.mkdir(parents=True, exist_ok=True)
+
+ copied = copy.deepcopy(dict(registry))
+ objects = copied.get("objects")
+ if not isinstance(objects, list):
+ raise ValueError("registry must contain objects")
+
+ resolved_image = Path(image_path).expanduser().resolve()
+ image = np.array(Image.open(resolved_image).convert("RGB"), dtype=np.uint8)
+ height, width = image.shape[:2]
+ shape = (height, width)
+
+ successes: dict[str, tuple[Mapping[str, Any], np.ndarray]] = {}
+ errors: list[dict[str, Any]] = []
+ for obj in objects:
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "success":
+ continue
+ object_id = str(obj.get("object_id") or "")
+ if not object_id:
+ errors.append({"type": "missing_successful_object_id"})
+ continue
+ try:
+ successes[object_id] = (obj, _load_mask(sam3.get("mask_path"), shape))
+ except Exception as exc:
+ errors.append(
+ {
+ "type": "invalid_successful_plant_container_mask",
+ "object_id": object_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+
+ candidates = list(copied.get("unassigned_sam3_candidates") or [])
+ records: list[dict[str, Any]] = []
+ accepted_count = 0
+ for object_id, (obj, source_mask) in successes.items():
+ if not _is_plant_container_assembly(obj):
+ continue
+ try:
+ target_box = _absolute_bbox_from_normalized(_bbox(obj), width, height)
+ except ValueError as exc:
+ errors.append(
+ {
+ "type": "invalid_plant_container_bbox",
+ "object_id": object_id,
+ "error": str(exc),
+ }
+ )
+ continue
+ source_box = _mask_bbox(source_mask)
+ eligible: list[dict[str, Any]] = []
+ rejected: list[dict[str, Any]] = []
+ for raw in candidates:
+ candidate_id = str(raw.get("candidate_id") or "")
+ prompt = str(
+ raw.get("prompt")
+ or raw.get("sam3_prompt")
+ or raw.get("category_hint")
+ or ""
+ )
+ candidate_tokens = set(re.findall(r"[a-z0-9]+", prompt.lower()))
+ if not candidate_tokens & {
+ "plant",
+ "potted",
+ "pot",
+ "planter",
+ "vase",
+ "greenery",
+ "foliage",
+ "branch",
+ "branches",
+ "flower",
+ "flowers",
+ "floral",
+ }:
+ continue
+ try:
+ score = float(raw.get("score"))
+ candidate_mask = _load_mask(raw.get("mask_path"), shape)
+ except (TypeError, ValueError, OSError) as exc:
+ rejected.append(
+ {
+ "candidate_id": candidate_id,
+ "reason": f"invalid_candidate:{type(exc).__name__}",
+ }
+ )
+ continue
+ if score < cfg.minimum_candidate_score:
+ rejected.append(
+ {
+ "candidate_id": candidate_id,
+ "reason": "score_below_threshold",
+ "score": round(score, 6),
+ }
+ )
+ continue
+ intersection = int(np.logical_and(source_mask, candidate_mask).sum())
+ candidate_area = int(candidate_mask.sum())
+ candidate_overlap = intersection / max(1, candidate_area)
+ if not (
+ cfg.minimum_candidate_overlap
+ <= candidate_overlap
+ <= cfg.maximum_candidate_overlap
+ ):
+ rejected.append(
+ {
+ "candidate_id": candidate_id,
+ "reason": "overlap_outside_completion_band",
+ "candidate_overlap": round(candidate_overlap, 6),
+ }
+ )
+ continue
+ candidate_bbox_fraction = _mask_fraction_within_bbox(
+ candidate_mask, target_box
+ )
+ if candidate_bbox_fraction < cfg.minimum_candidate_bbox_fraction:
+ rejected.append(
+ {
+ "candidate_id": candidate_id,
+ "reason": "candidate_outside_declared_bbox",
+ "candidate_bbox_fraction": round(candidate_bbox_fraction, 6),
+ }
+ )
+ continue
+ new_pixels = np.logical_and(candidate_mask, ~source_mask)
+ new_pixel_count = int(new_pixels.sum())
+ if new_pixel_count < cfg.minimum_new_pixels:
+ rejected.append(
+ {
+ "candidate_id": candidate_id,
+ "reason": "insufficient_new_pixels",
+ "new_pixel_count": new_pixel_count,
+ }
+ )
+ continue
+ new_pixel_bbox_fraction = (
+ float(new_pixels[target_box[1]:target_box[3], target_box[0]:target_box[2]].sum())
+ / float(max(1, new_pixel_count))
+ )
+ if new_pixel_bbox_fraction < cfg.minimum_new_pixel_bbox_fraction:
+ rejected.append(
+ {
+ "candidate_id": candidate_id,
+ "reason": "new_pixels_outside_declared_bbox",
+ "new_pixel_bbox_fraction": round(new_pixel_bbox_fraction, 6),
+ }
+ )
+ continue
+ candidate_box = _mask_bbox(candidate_mask)
+ if candidate_box[3] < (
+ source_box[3] + cfg.minimum_downward_extension_pixels
+ ):
+ rejected.append(
+ {
+ "candidate_id": candidate_id,
+ "reason": "no_clear_lower_container_extension",
+ "source_bottom": source_box[3],
+ "candidate_bottom": candidate_box[3],
+ }
+ )
+ continue
+ conflict = None
+ for other_id, (_, other_mask) in successes.items():
+ if other_id == object_id:
+ continue
+ overlap = int(np.logical_and(new_pixels, other_mask).sum()) / max(
+ 1, new_pixel_count
+ )
+ if overlap > cfg.maximum_other_object_overlap:
+ conflict = (other_id, overlap)
+ break
+ if conflict is not None:
+ rejected.append(
+ {
+ "candidate_id": candidate_id,
+ "reason": "new_pixels_overlap_other_object",
+ "other_object_id": conflict[0],
+ "overlap": round(conflict[1], 6),
+ }
+ )
+ continue
+ eligible.append(
+ {
+ "candidate_id": candidate_id,
+ "prompt": prompt,
+ "score": score,
+ "mask": candidate_mask,
+ "candidate_overlap": candidate_overlap,
+ "candidate_bbox_fraction": candidate_bbox_fraction,
+ "new_pixel_count": new_pixel_count,
+ "new_pixel_bbox_fraction": new_pixel_bbox_fraction,
+ }
+ )
+
+ eligible.sort(
+ key=lambda item: (
+ -float(item["score"]),
+ -int(item["new_pixel_count"]),
+ str(item["candidate_id"]),
+ )
+ )
+ if len(eligible) > 1 and (
+ float(eligible[0]["score"]) - float(eligible[1]["score"])
+ <= cfg.ambiguity_score_gap
+ ):
+ records.append(
+ {
+ "object_id": object_id,
+ "status": "ambiguous",
+ "eligible_candidate_ids": [
+ str(item["candidate_id"]) for item in eligible
+ ],
+ "rejected_candidates": rejected,
+ }
+ )
+ continue
+ if not eligible:
+ records.append(
+ {
+ "object_id": object_id,
+ "status": "noop",
+ "rejected_candidates": rejected,
+ }
+ )
+ continue
+
+ winner = eligible[0]
+ union_mask = np.logical_or(source_mask, winner["mask"])
+ selected_dir = output / "selected"
+ selected_dir.mkdir(parents=True, exist_ok=True)
+ mask_path = selected_dir / f"{object_id.lower()}_mask.png"
+ overlay_path = selected_dir / f"{object_id.lower()}_overlay.png"
+ Image.fromarray(union_mask.astype(np.uint8) * 255).save(mask_path)
+ canvas = image.astype(np.float32).copy()
+ canvas[union_mask] = (
+ 0.55 * canvas[union_mask]
+ + 0.45 * np.array((255, 0, 0), dtype=np.float32)
+ )
+ Image.fromarray(np.clip(canvas, 0, 255).astype(np.uint8)).save(
+ overlay_path
+ )
+ union_box = _mask_bbox(union_mask)
+ sam3 = dict(obj.get("sam3") or {})
+ sam3.update(
+ {
+ "area_pixels": int(union_mask.sum()),
+ "mask_bbox_abs_xyxy": union_box,
+ "vlm_bbox_iou": _bbox_metrics(target_box, union_box)[
+ "candidate_bbox_coverage"
+ ],
+ "mask_path": str(mask_path),
+ "overlay_path": str(overlay_path),
+ "selection_method": "plant_container_candidate_union",
+ "plant_container_mask_completion": {
+ "candidate_id": str(winner["candidate_id"]),
+ "candidate_prompt": str(winner["prompt"]),
+ "candidate_score": round(float(winner["score"]), 6),
+ "candidate_overlap": round(
+ float(winner["candidate_overlap"]), 6
+ ),
+ "candidate_bbox_fraction": round(
+ float(winner["candidate_bbox_fraction"]), 6
+ ),
+ "new_pixel_count": int(winner["new_pixel_count"]),
+ "new_pixel_bbox_fraction": round(
+ float(winner["new_pixel_bbox_fraction"]), 6
+ ),
+ },
+ }
+ )
+ obj["sam3"] = sam3
+ accepted_count += 1
+ records.append(
+ {
+ "object_id": object_id,
+ "status": "merged",
+ "selected_candidate_id": str(winner["candidate_id"]),
+ "selected_mask_path": str(mask_path),
+ "selected_overlay_path": str(overlay_path),
+ "selected_area_pixels": int(union_mask.sum()),
+ "selected_mask_bbox_abs_xyxy": union_box,
+ "candidate_score": round(float(winner["score"]), 6),
+ "candidate_overlap": round(
+ float(winner["candidate_overlap"]), 6
+ ),
+ "candidate_bbox_fraction": round(
+ float(winner["candidate_bbox_fraction"]), 6
+ ),
+ "new_pixel_count": int(winner["new_pixel_count"]),
+ "new_pixel_bbox_fraction": round(
+ float(winner["new_pixel_bbox_fraction"]), 6
+ ),
+ "rejected_candidates": rejected,
+ }
+ )
+
+ audit = {
+ "schema_version": "scenemosaic_plant_container_mask_completion",
+ "status": "blocked" if errors else ("pass" if accepted_count else "noop"),
+ "config": asdict(cfg),
+ "candidate_count": len(candidates),
+ "accepted_count": accepted_count,
+ "records": records,
+ "additional_model_load_count": 0,
+ "additional_vlm_api_calls": 0,
+ "formal_outputs_modified": False,
+ "hard_errors": errors,
+ }
+ audit_path = output / "plant_container_mask_completion_audit.json"
+ _atomic_json(audit_path, audit)
+ return {"registry": copied, "audit": audit, "audit_path": str(audit_path)}
+
+
+def run_functional_assembly_roi_refinement(
+ *,
+ session: Any,
+ infer_text_candidates_fn: Any,
+ image_path: str | Path,
+ registry: Mapping[str, Any],
+ normalization_audit: Mapping[str, Any],
+ output_dir: str | Path,
+ config: AssemblyConfig | None = None,
+) -> dict[str, Any]:
+ """Refine normalized targets with bounded positive-box probes."""
+
+ cfg = config or AssemblyConfig()
+ output = Path(output_dir).expanduser().resolve()
+ if output.exists() and any(output.iterdir()):
+ raise FileExistsError(f"functional assembly output is not empty: {output}")
+ output.mkdir(parents=True, exist_ok=True)
+ copied = copy.deepcopy(dict(registry))
+ objects = copied.get("objects")
+ if not isinstance(objects, list):
+ raise ValueError("registry must contain objects")
+ targets = list(normalization_audit.get("targets") or [])
+ errors: list[dict[str, Any]] = []
+ if len(targets) > cfg.maximum_targets:
+ errors.append(
+ {
+ "type": "functional_assembly_target_capacity_overflow",
+ "actual": len(targets),
+ "maximum": cfg.maximum_targets,
+ }
+ )
+ targets = targets[: cfg.maximum_targets]
+ resolved_image = Path(image_path).expanduser().resolve()
+ image = np.array(Image.open(resolved_image).convert("RGB"), dtype=np.uint8)
+ height, width = image.shape[:2]
+ shape = (height, width)
+ by_id = {str(item.get("object_id") or ""): item for item in objects}
+ records = []
+ inference_before = int(getattr(session, "inference_run_count", 0))
+ pillow_masks = []
+ independent_bed_protection_masks = []
+ if any(
+ str(target.get("role") or "") == "complete_bed_assembly"
+ for target in targets
+ ):
+ for independent in objects:
+ sam3 = independent.get("sam3") or {}
+ if sam3.get("status") != "success":
+ continue
+ independent_id = str(independent.get("object_id") or "")
+ if _is_absorbable_bed_component(independent):
+ continue
+ try:
+ independent_mask = _load_mask(sam3.get("mask_path"), shape)
+ except Exception as exc:
+ errors.append(
+ {
+ "type": (
+ "invalid_independent_pillow_mask"
+ if _is_pillow(independent)
+ else "invalid_independent_bed_protection_mask"
+ ),
+ "object_id": independent_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ continue
+ if _is_pillow(independent):
+ pillow_masks.append((independent_id, independent_mask))
+ else:
+ independent_bed_protection_masks.append(
+ (
+ independent_id,
+ str(independent.get("category") or ""),
+ independent_mask,
+ )
+ )
+
+ for index, target in enumerate(targets, start=1):
+ object_id = str(target.get("source_object_id") or "")
+ obj = by_id.get(object_id)
+ if obj is None:
+ errors.append(
+ {"type": "missing_functional_assembly_target", "object_id": object_id}
+ )
+ continue
+ norm_box = [float(value) for value in target["bbox_norm_xyxy"]]
+ target_box = [
+ max(0, min(width - 1, int(round(norm_box[0] * width)))),
+ max(0, min(height - 1, int(round(norm_box[1] * height)))),
+ max(1, min(width, int(round(norm_box[2] * width)))),
+ max(1, min(height, int(round(norm_box[3] * height)))),
+ ]
+ span_x = target_box[2] - target_box[0]
+ span_y = target_box[3] - target_box[1]
+ pad_x = max(16, int(round(span_x * 0.10)))
+ pad_y = max(16, int(round(span_y * 0.10)))
+ positive_box = [
+ max(0, target_box[0] - pad_x),
+ max(0, target_box[1] - pad_y),
+ min(width, target_box[2] + pad_x),
+ min(height, target_box[3] + pad_y),
+ ]
+ role = str(target.get("role") or "")
+ source_mask = None
+ required_source_coverage = float(
+ target.get("require_source_mask_coverage") or 0.0
+ )
+ if required_source_coverage:
+ try:
+ source_mask = _load_mask(
+ (obj.get("sam3") or {}).get("mask_path"), shape
+ )
+ except Exception as exc:
+ errors.append(
+ {
+ "type": "functional_assembly_missing_source_mask",
+ "object_id": object_id,
+ "role": role,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ records.append(
+ {
+ "object_id": object_id,
+ "role": role,
+ "status": "blocked",
+ "reason": "missing_source_mask",
+ }
+ )
+ continue
+ protected_masks = []
+ if bool(target.get("protect_independent_objects")):
+ for independent in objects:
+ independent_id = str(independent.get("object_id") or "")
+ if independent_id == object_id or _is_fixed_faucet(independent):
+ continue
+ independent_sam3 = independent.get("sam3") or {}
+ if independent_sam3.get("status") != "success":
+ continue
+ try:
+ independent_mask = _load_mask(
+ independent_sam3.get("mask_path"), shape
+ )
+ except Exception as exc:
+ errors.append(
+ {
+ "type": "invalid_protected_independent_mask",
+ "object_id": independent_id,
+ "protected_for_object_id": object_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ continue
+ if independent_mask[
+ target_box[1] : target_box[3],
+ target_box[0] : target_box[2],
+ ].any():
+ protected_masks.append(
+ (
+ independent_id,
+ str(independent.get("category") or ""),
+ independent_mask,
+ )
+ )
+ raw_dir = (
+ output / "raw_candidates" / f"target_{index:02d}_{object_id.lower()}"
+ )
+ result = infer_text_candidates_fn(
+ session,
+ resolved_image,
+ str(target["prompt"]),
+ raw_dir,
+ score_threshold=cfg.score_threshold,
+ positive_bbox_abs_xyxy=positive_box,
+ maximum_candidates=cfg.maximum_candidates_per_target,
+ overwrite=False,
+ )
+ evaluations = []
+ for candidate in (result.get("registry") or {}).get("candidates") or []:
+ try:
+ mask = _load_mask(candidate.get("mask_path"), shape)
+ ownership = dict(target.get("hard_ownership_limit") or {})
+ if "maximum_x_norm" in ownership:
+ cutoff = int(round(float(ownership["maximum_x_norm"]) * width))
+ mask[:, cutoff:] = False
+ if "minimum_x_norm" in ownership:
+ cutoff = int(round(float(ownership["minimum_x_norm"]) * width))
+ mask[:, :cutoff] = False
+ candidate_box = _mask_bbox(mask)
+ except Exception as exc:
+ evaluations.append(
+ {
+ "candidate_id": candidate.get("candidate_id"),
+ "eligible": False,
+ "reason": (
+ f"invalid_candidate: {type(exc).__name__}: {exc}"
+ ),
+ }
+ )
+ continue
+ metrics = _bbox_metrics(target_box, candidate_box)
+ score = float(candidate.get("score") or 0.0)
+ source_mask_coverage = (
+ int(np.logical_and(mask, source_mask).sum())
+ / max(int(source_mask.sum()), 1)
+ if source_mask is not None
+ else None
+ )
+ eligible = (
+ score >= cfg.minimum_selected_score
+ and metrics["target_bbox_coverage"]
+ >= cfg.minimum_target_bbox_coverage
+ and metrics["candidate_bbox_coverage"]
+ >= cfg.minimum_candidate_bbox_coverage
+ and metrics["center_distance_scale"]
+ <= cfg.maximum_center_distance_scale
+ and (
+ source_mask_coverage is None
+ or source_mask_coverage >= required_source_coverage
+ )
+ )
+ quality = (
+ score
+ + 0.50 * metrics["target_bbox_coverage"]
+ + 0.25 * metrics["candidate_bbox_coverage"]
+ + (0.20 * source_mask_coverage if source_mask_coverage is not None else 0.0)
+ - 0.20 * metrics["center_distance_scale"]
+ )
+ evaluations.append(
+ {
+ "candidate_id": str(candidate.get("candidate_id") or ""),
+ "source_mask_path": str(candidate.get("mask_path") or ""),
+ "score": score,
+ "mask": mask,
+ "mask_bbox_abs_xyxy": candidate_box,
+ "target_bbox_coverage": round(
+ metrics["target_bbox_coverage"], 6
+ ),
+ "candidate_bbox_coverage": round(
+ metrics["candidate_bbox_coverage"], 6
+ ),
+ "center_distance_scale": round(
+ metrics["center_distance_scale"], 6
+ ),
+ "source_mask_coverage": (
+ round(source_mask_coverage, 6)
+ if source_mask_coverage is not None
+ else None
+ ),
+ "quality": quality,
+ "eligible": eligible,
+ "reason": None if eligible else "bounded_geometry_or_score_gate",
+ }
+ )
+ eligible = sorted(
+ [item for item in evaluations if item.get("eligible")],
+ key=lambda item: (
+ -float(item["quality"]),
+ str(item["candidate_id"]),
+ ),
+ )
+ public_evaluations = [
+ {key: value for key, value in item.items() if key != "mask"}
+ for item in evaluations
+ ]
+ if not eligible:
+ errors.append(
+ {
+ "type": "functional_assembly_no_eligible_roi_candidate",
+ "object_id": object_id,
+ }
+ )
+ records.append(
+ {
+ "object_id": object_id,
+ "status": "blocked",
+ "target_bbox_abs_xyxy": target_box,
+ "positive_bbox_abs_xyxy": positive_box,
+ "evaluations": public_evaluations,
+ }
+ )
+ continue
+ winner = eligible[0]
+ if len(eligible) > 1:
+ runner_up = eligible[1]
+ gap = float(winner["quality"]) - float(runner_up["quality"])
+ overlap = _mask_iou(winner["mask"], runner_up["mask"])
+ if (
+ gap <= cfg.ambiguity_quality_gap
+ and overlap < cfg.ambiguity_mask_iou
+ ):
+ errors.append(
+ {
+ "type": "functional_assembly_ambiguous_roi_candidates",
+ "object_id": object_id,
+ "candidate_ids": [
+ winner["candidate_id"],
+ runner_up["candidate_id"],
+ ],
+ "quality_gap": round(gap, 6),
+ "mask_iou": round(overlap, 6),
+ }
+ )
+ continue
+
+ pillow_exclusions = []
+ independent_object_exclusions = []
+ if str(target.get("role") or "") == "complete_bed_assembly":
+ area_before_exclusion = int(winner["mask"].sum())
+ for pillow_id, pillow_mask in pillow_masks:
+ overlap_pixels = int(
+ np.logical_and(winner["mask"], pillow_mask).sum()
+ )
+ if overlap_pixels <= 0:
+ continue
+ winner["mask"] = np.logical_and(
+ winner["mask"], np.logical_not(pillow_mask)
+ )
+ pillow_exclusions.append(
+ {
+ "pillow_object_id": pillow_id,
+ "overlap_pixels_removed": overlap_pixels,
+ }
+ )
+ area_after_pillow_exclusion = int(winner["mask"].sum())
+ for (
+ independent_id,
+ independent_category,
+ independent_mask,
+ ) in independent_bed_protection_masks:
+ if independent_id == object_id:
+ continue
+ overlap_pixels = int(
+ np.logical_and(winner["mask"], independent_mask).sum()
+ )
+ if overlap_pixels <= 0:
+ continue
+ winner["mask"] = np.logical_and(
+ winner["mask"], np.logical_not(independent_mask)
+ )
+ independent_object_exclusions.append(
+ {
+ "object_id": independent_id,
+ "category": independent_category,
+ "overlap_pixels_removed": overlap_pixels,
+ }
+ )
+ area_after_exclusion = int(winner["mask"].sum())
+ pillow_removed_fraction = (
+ (area_before_exclusion - area_after_pillow_exclusion)
+ / max(area_before_exclusion, 1)
+ )
+ independent_removed_fraction = (
+ (area_after_pillow_exclusion - area_after_exclusion)
+ / max(area_before_exclusion, 1)
+ )
+ removed_fraction = (
+ (area_before_exclusion - area_after_exclusion)
+ / max(area_before_exclusion, 1)
+ )
+ if not winner["mask"].any() or removed_fraction > 0.40:
+ errors.append(
+ {
+ "type": "complete_bed_protected_object_exclusion_invalid",
+ "object_id": object_id,
+ "area_before_exclusion": area_before_exclusion,
+ "area_after_pillow_exclusion": (
+ area_after_pillow_exclusion
+ ),
+ "area_after_exclusion": area_after_exclusion,
+ "removed_fraction": round(removed_fraction, 6),
+ }
+ )
+ records.append(
+ {
+ "object_id": object_id,
+ "role": target.get("role"),
+ "status": "blocked",
+ "reason": "independent_object_subtraction_gate",
+ "pillow_exclusions": pillow_exclusions,
+ "independent_object_exclusions": (
+ independent_object_exclusions
+ ),
+ }
+ )
+ continue
+ winner["mask_bbox_abs_xyxy"] = _mask_bbox(winner["mask"])
+ elif role == "complete_sink_vanity_assembly":
+ area_before_exclusion = int(winner["mask"].sum())
+ pillow_exclusions = []
+ independent_object_exclusions = []
+ for independent_id, independent_category, independent_mask in protected_masks:
+ overlap_pixels = int(
+ np.logical_and(winner["mask"], independent_mask).sum()
+ )
+ if overlap_pixels <= 0:
+ continue
+ winner["mask"] = np.logical_and(
+ winner["mask"], np.logical_not(independent_mask)
+ )
+ independent_object_exclusions.append(
+ {
+ "object_id": independent_id,
+ "category": independent_category,
+ "overlap_pixels_removed": overlap_pixels,
+ }
+ )
+ area_after_pillow_exclusion = area_before_exclusion
+ area_after_exclusion = int(winner["mask"].sum())
+ pillow_removed_fraction = 0.0
+ independent_removed_fraction = (
+ (area_before_exclusion - area_after_exclusion)
+ / max(area_before_exclusion, 1)
+ )
+ removed_fraction = independent_removed_fraction
+ final_source_coverage = (
+ int(np.logical_and(winner["mask"], source_mask).sum())
+ / max(int(source_mask.sum()), 1)
+ if source_mask is not None
+ else None
+ )
+ final_metrics = _bbox_metrics(target_box, _mask_bbox(winner["mask"])) if winner["mask"].any() else None
+ if (
+ not winner["mask"].any()
+ or removed_fraction > cfg.maximum_protected_object_removal_fraction
+ or (
+ final_source_coverage is not None
+ and final_source_coverage < required_source_coverage
+ )
+ or final_metrics is None
+ or final_metrics["target_bbox_coverage"]
+ < cfg.minimum_target_bbox_coverage
+ ):
+ errors.append(
+ {
+ "type": "complete_sink_vanity_protected_object_exclusion_invalid",
+ "object_id": object_id,
+ "area_before_exclusion": area_before_exclusion,
+ "area_after_exclusion": area_after_exclusion,
+ "removed_fraction": round(removed_fraction, 6),
+ "source_mask_coverage": (
+ round(final_source_coverage, 6)
+ if final_source_coverage is not None
+ else None
+ ),
+ }
+ )
+ records.append(
+ {
+ "object_id": object_id,
+ "role": role,
+ "status": "blocked",
+ "reason": "protected_object_or_source_coverage_gate",
+ "independent_object_exclusions": independent_object_exclusions,
+ }
+ )
+ continue
+ winner["mask_bbox_abs_xyxy"] = _mask_bbox(winner["mask"])
+ else:
+ area_before_exclusion = int(winner["mask"].sum())
+ area_after_pillow_exclusion = area_before_exclusion
+ area_after_exclusion = area_before_exclusion
+ pillow_removed_fraction = 0.0
+ independent_removed_fraction = 0.0
+ removed_fraction = 0.0
+
+ selected_dir = output / "selected"
+ selected_dir.mkdir(parents=True, exist_ok=True)
+ mask_path = selected_dir / f"{object_id.lower()}_mask.png"
+ overlay_path = selected_dir / f"{object_id.lower()}_overlay.png"
+ Image.fromarray(winner["mask"].astype(np.uint8) * 255).save(mask_path)
+ canvas = image.astype(np.float32).copy()
+ canvas[winner["mask"]] = (
+ 0.55 * canvas[winner["mask"]]
+ + 0.45 * np.array((255, 0, 0), dtype=np.float32)
+ )
+ Image.fromarray(
+ np.clip(canvas, 0, 255).astype(np.uint8)
+ ).save(overlay_path)
+ candidate_box = list(winner["mask_bbox_abs_xyxy"])
+ obj["sam3"] = {
+ "status": "success",
+ "warning": None,
+ "prompt_used": str(target["prompt"]),
+ "score": float(winner["score"]),
+ "area_pixels": int(winner["mask"].sum()),
+ "vlm_bbox_abs_xyxy": target_box,
+ "mask_bbox_abs_xyxy": candidate_box,
+ "vlm_bbox_iou": _bbox_metrics(target_box, candidate_box)[
+ "candidate_bbox_coverage"
+ ],
+ "merged_candidate_indices": [],
+ "mask_path": str(mask_path),
+ "overlay_path": str(overlay_path),
+ "selection_method": "functional_assembly_roi_refinement",
+ "merge_review": {
+ "required": False,
+ "ambiguous_candidate_indices": [],
+ "ambiguous_candidate_ids": [],
+ "protected_shared_candidate_indices": [],
+ "protected_shared_candidate_ids": [],
+ "resolution_stage": None,
+ },
+ "functional_assembly_roi_refinement": {
+ "candidate_id": winner["candidate_id"],
+ "positive_bbox_abs_xyxy": positive_box,
+ "target_bbox_coverage": winner["target_bbox_coverage"],
+ "candidate_bbox_coverage": winner[
+ "candidate_bbox_coverage"
+ ],
+ "center_distance_scale": winner["center_distance_scale"],
+ "hard_ownership_limit": target.get("hard_ownership_limit"),
+ "independent_pillow_exclusion": {
+ "applied": bool(pillow_exclusions),
+ "area_before_exclusion": area_before_exclusion,
+ "area_after_exclusion": area_after_pillow_exclusion,
+ "removed_fraction": round(pillow_removed_fraction, 6),
+ "records": pillow_exclusions,
+ },
+ "independent_object_exclusion": {
+ "applied": bool(independent_object_exclusions),
+ "area_before_exclusion": area_after_pillow_exclusion,
+ "area_after_exclusion": area_after_exclusion,
+ "removed_fraction_of_raw_candidate": round(
+ independent_removed_fraction, 6
+ ),
+ "total_removed_fraction": round(removed_fraction, 6),
+ "records": independent_object_exclusions,
+ },
+ },
+ }
+ records.append(
+ {
+ "object_id": object_id,
+ "role": target.get("role"),
+ "status": "refined",
+ "prompt": target["prompt"],
+ "target_bbox_abs_xyxy": target_box,
+ "positive_bbox_abs_xyxy": positive_box,
+ "selected_candidate_id": winner["candidate_id"],
+ "selected_score": winner["score"],
+ "selected_mask_path": str(mask_path),
+ "selected_area_pixels": int(winner["mask"].sum()),
+ "selected_mask_bbox_abs_xyxy": candidate_box,
+ "area_before_pillow_exclusion": area_before_exclusion,
+ "area_after_pillow_exclusion": area_after_pillow_exclusion,
+ "pillow_exclusion_fraction": round(
+ pillow_removed_fraction, 6
+ ),
+ "pillow_exclusions": pillow_exclusions,
+ "area_after_independent_object_exclusion": (
+ area_after_exclusion
+ ),
+ "independent_object_exclusion_fraction": round(
+ independent_removed_fraction, 6
+ ),
+ "total_protected_object_exclusion_fraction": round(
+ removed_fraction, 6
+ ),
+ "independent_object_exclusions": (
+ independent_object_exclusions
+ ),
+ "evaluations": public_evaluations,
+ }
+ )
+
+ inference_after = int(getattr(session, "inference_run_count", 0))
+ audit = {
+ "schema_version": "scenemosaic_functional_assembly_roi_refinement",
+ "status": "blocked" if errors else ("pass" if targets else "noop"),
+ "config": asdict(cfg),
+ "target_count": len(targets),
+ "refined_count": sum(
+ item.get("status") == "refined" for item in records
+ ),
+ "records": records,
+ "additional_model_load_count": 0,
+ "inference_count": inference_after - inference_before,
+ "additional_vlm_api_calls": 0,
+ "formal_outputs_modified": False,
+ "hard_errors": errors,
+ }
+ audit_path = output / "functional_assembly_roi_refinement_audit.json"
+ _atomic_json(audit_path, audit)
+ return {
+ "registry": copied,
+ "audit": audit,
+ "audit_path": str(audit_path),
+ }
+
+
+def _pair_is_geometrically_fixed(
+ bathtub: Mapping[str, Any], faucet: Mapping[str, Any]
+) -> bool:
+ """Require the tap to be anchored at the bathtub rim, not merely nearby."""
+
+ bathtub_box = _bbox(bathtub)
+ faucet_box = _bbox(faucet)
+ bathtub_width = bathtub_box[2] - bathtub_box[0]
+ bathtub_height = bathtub_box[3] - bathtub_box[1]
+ horizontal_overlap = _overlap_1d(
+ bathtub_box[0], bathtub_box[2], faucet_box[0], faucet_box[2]
+ )
+ if horizontal_overlap < 0.55 * (faucet_box[2] - faucet_box[0]):
+ return False
+ return bool(
+ faucet_box[2] >= bathtub_box[0] - 0.08 * bathtub_width
+ and faucet_box[0] <= bathtub_box[2] + 0.08 * bathtub_width
+ and faucet_box[1] <= bathtub_box[1] + 0.20 * bathtub_height
+ and faucet_box[3] >= bathtub_box[1] - 0.28 * bathtub_height
+ )
+
+
+def union_fixed_bathtub_faucet_masks(
+ *,
+ image_path: str | Path,
+ registry: Mapping[str, Any],
+ output_dir: str | Path,
+ config: AssemblyConfig | None = None,
+) -> dict[str, Any]:
+ """Union only an explicitly typed, uniquely matched fixed bathtub faucet.
+
+ The source faucet remains in the temporary Pass-A registry so that Pass B can
+ retain its source identity; forced_removals removes that duplicate from
+ the final fused proposal deterministically. No IDs are renumbered here.
+ """
+
+ cfg = config or AssemblyConfig()
+ output = Path(output_dir).expanduser().resolve()
+ if output.exists() and any(output.iterdir()):
+ raise FileExistsError(f"fixed component output is not empty: {output}")
+ output.mkdir(parents=True, exist_ok=True)
+ copied = copy.deepcopy(dict(registry))
+ objects = copied.get("objects")
+ if not isinstance(objects, list):
+ raise ValueError("registry must contain objects")
+ image = np.array(
+ Image.open(Path(image_path).expanduser().resolve()).convert("RGB"),
+ dtype=np.uint8,
+ )
+ shape = tuple(image.shape[:2])
+ bathtubs = [item for item in objects if _is_bathtub(item)]
+ faucets = [item for item in objects if _is_fixed_bathtub_faucet(item)]
+ records = []
+ hard_errors = []
+ forced_removals = []
+ object_updates = {}
+ accepted_count = 0
+ for bathtub in bathtubs:
+ bathtub_id = str(bathtub.get("object_id") or "")
+ matches = [
+ faucet
+ for faucet in faucets
+ if _pair_is_geometrically_fixed(bathtub, faucet)
+ ]
+ if len(matches) != 1:
+ records.append(
+ {
+ "bathtub_object_id": bathtub_id,
+ "status": "skipped",
+ "reason": (
+ "no_unique_fixed_bathtub_faucet"
+ if not matches
+ else "ambiguous_fixed_bathtub_faucet"
+ ),
+ "candidate_faucet_object_ids": [
+ str(item.get("object_id") or "") for item in matches
+ ],
+ }
+ )
+ continue
+ if accepted_count >= cfg.maximum_fixed_faucet_unions:
+ hard_errors.append(
+ {
+ "type": "fixed_bathtub_faucet_union_capacity_overflow",
+ "maximum": cfg.maximum_fixed_faucet_unions,
+ }
+ )
+ break
+ faucet = matches[0]
+ faucet_id = str(faucet.get("object_id") or "")
+ try:
+ bathtub_mask = _load_mask(
+ (bathtub.get("sam3") or {}).get("mask_path"), shape
+ )
+ faucet_mask = _load_mask(
+ (faucet.get("sam3") or {}).get("mask_path"), shape
+ )
+ except Exception as exc:
+ hard_errors.append(
+ {
+ "type": "fixed_bathtub_faucet_union_invalid_source_mask",
+ "bathtub_object_id": bathtub_id,
+ "faucet_object_id": faucet_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ continue
+ conflicting = []
+ for other in objects:
+ other_id = str(other.get("object_id") or "")
+ if other_id in {bathtub_id, faucet_id}:
+ continue
+ other_sam3 = other.get("sam3") or {}
+ if other_sam3.get("status") != "success":
+ continue
+ try:
+ other_mask = _load_mask(other_sam3.get("mask_path"), shape)
+ except Exception as exc:
+ hard_errors.append(
+ {
+ "type": "fixed_bathtub_faucet_union_invalid_peer_mask",
+ "bathtub_object_id": bathtub_id,
+ "peer_object_id": other_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ continue
+ overlap_fraction = int(np.logical_and(faucet_mask, other_mask).sum()) / max(
+ int(faucet_mask.sum()), 1
+ )
+ if overlap_fraction > 0.10:
+ conflicting.append(
+ {
+ "object_id": other_id,
+ "overlap_fraction_of_faucet": round(overlap_fraction, 6),
+ }
+ )
+ if conflicting:
+ records.append(
+ {
+ "bathtub_object_id": bathtub_id,
+ "faucet_object_id": faucet_id,
+ "status": "skipped",
+ "reason": "faucet_overlaps_independent_object",
+ "conflicts": conflicting,
+ }
+ )
+ continue
+ union_mask = np.logical_or(bathtub_mask, faucet_mask)
+ selected = output / "selected"
+ selected.mkdir(parents=True, exist_ok=True)
+ mask_path = selected / f"{bathtub_id.lower()}_mask.png"
+ overlay_path = selected / f"{bathtub_id.lower()}_overlay.png"
+ Image.fromarray(union_mask.astype(np.uint8) * 255).save(mask_path)
+ canvas = image.astype(np.float32).copy()
+ canvas[union_mask] = (
+ 0.55 * canvas[union_mask]
+ + 0.45 * np.array((255, 0, 0), dtype=np.float32)
+ )
+ Image.fromarray(np.clip(canvas, 0, 255).astype(np.uint8)).save(
+ overlay_path
+ )
+ previous = copy.deepcopy(dict(bathtub.get("sam3") or {}))
+ bathtub["sam3"] = {
+ **previous,
+ "status": "success",
+ "mask_path": str(mask_path),
+ "overlay_path": str(overlay_path),
+ "area_pixels": int(union_mask.sum()),
+ "mask_bbox_abs_xyxy": _mask_bbox(union_mask),
+ "selection_method": "fixed_bathtub_faucet_mask_union",
+ "fixed_component_mask_union": {
+ "component_object_id": faucet_id,
+ "host_area_pixels_before": int(bathtub_mask.sum()),
+ "component_area_pixels": int(faucet_mask.sum()),
+ "union_area_pixels": int(union_mask.sum()),
+ },
+ }
+ faucet_sam3 = dict(faucet.get("sam3") or {})
+ faucet_sam3["fixed_component_absorbed_by"] = bathtub_id
+ faucet["sam3"] = faucet_sam3
+ object_updates[bathtub_id] = {
+ "sam3_prompt": "complete bathtub including fixed faucet",
+ "description": "complete bathtub including its fixed faucet",
+ }
+ forced_removals.append(faucet_id)
+ accepted_count += 1
+ records.append(
+ {
+ "bathtub_object_id": bathtub_id,
+ "faucet_object_id": faucet_id,
+ "status": "unioned",
+ "selected_mask_path": str(mask_path),
+ "forced_final_removal": faucet_id,
+ }
+ )
+ audit = {
+ "schema_version": "scenemosaic_fixed_bathtub_faucet_union",
+ "status": "blocked" if hard_errors else ("pass" if records and accepted_count else "noop"),
+ "bathtub_count": len(bathtubs),
+ "typed_faucet_count": len(faucets),
+ "accepted_count": accepted_count,
+ "records": records,
+ "forced_removals": forced_removals,
+ "object_updates": object_updates,
+ "additional_vlm_api_calls": 0,
+ "formal_outputs_modified": False,
+ "hard_errors": hard_errors,
+ }
+ audit_path = output / "fixed_bathtub_faucet_union_audit.json"
+ _atomic_json(audit_path, audit)
+ return {"registry": copied, "audit": audit, "audit_path": str(audit_path)}
+
+
+def apply_fixed_component_absorption_actions(
+ actions: Mapping[str, Any], union_audit: Mapping[str, Any]
+) -> dict[str, Any]:
+ """Force only deterministic component removals after Pass-B validation."""
+
+ copied = copy.deepcopy(dict(actions))
+ forced = sorted({str(item) for item in union_audit.get("forced_removals") or []})
+ updates = list(copied.get("updates") or [])
+ removed_updates = [
+ item for item in updates if str(item.get("pass_a_id") or "") in forced
+ ]
+ copied["updates"] = [
+ item for item in updates if str(item.get("pass_a_id") or "") not in forced
+ ]
+ original_removals = [str(item) for item in copied.get("removals") or []]
+ copied["removals"] = sorted(set(original_removals) | set(forced))
+ return {
+ "actions": copied,
+ "audit": {
+ "schema_version": "scenemosaic_fixed_component_action_policy",
+ "status": "pass" if forced else "noop",
+ "forced_removals": forced,
+ "suppressed_component_updates": removed_updates,
+ "additional_vlm_api_calls": 0,
+ "formal_outputs_modified": False,
+ "hard_errors": [],
+ },
+ }
+
+
+__all__ = [
+ "AssemblyConfig",
+ "PlantContainerMaskCompletionConfig",
+ "normalize_pass_a_functional_assemblies",
+ "complete_plant_container_masks",
+ "run_functional_assembly_roi_refinement",
+ "union_fixed_bathtub_faucet_masks",
+ "apply_fixed_component_absorption_actions",
+]
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/host_part_mask_fusion.py b/vlm_agent/reconstruction/perception/pipeline/refinements/host_part_mask_fusion.py
new file mode 100644
index 0000000..2701984
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/host_part_mask_fusion.py
@@ -0,0 +1,929 @@
+#!/usr/bin/env python3
+"""Fuse explicitly claimed component masks into one host assembly.
+
+This is deliberately narrower than generic overlap removal. It only joins a
+Pass-B addition to a successful Pass-A attachment host when the host's own
+Pass-A wording explicitly claims that kind of attached content (for example,
+``wall coat rack robes``). A second failed duplicate addition may be removed
+only when the accepted sibling mask already covers its declared box. The
+stage is isolated-attempt only, has no VLM calls, and materializes a fresh union mask in
+the final staging directory.
+"""
+
+from __future__ import annotations
+
+import copy
+import json
+import os
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Callable, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+
+@dataclass(frozen=True)
+class HostPartMaskFusionConfig:
+ enabled: bool = True
+ isolated_attempt_only: bool = True
+ minimum_child_box_coverage_by_host: float = 0.75
+ minimum_failed_sibling_box_coverage: float = 0.75
+ host_bbox_expansion_fraction: float = 0.06
+ maximum_positive_bbox_duplicate_confirmations: int = 1
+ minimum_confirmation_score: float = 0.85
+ minimum_confirmation_target_coverage: float = 0.25
+ minimum_confirmation_covered_by_accepted_mask: float = 0.85
+ maximum_complete_host_probes: int = 2
+ maximum_candidates_per_complete_host_probe: int = 8
+ minimum_complete_host_score: float = 0.45
+ minimum_complete_host_original_coverage: float = 0.85
+ minimum_complete_host_added_fraction: float = 0.02
+ maximum_complete_host_area_ratio: float = 3.0
+ minimum_idempotent_anchor_area_ratio: float = 0.90
+ maximum_idempotent_anchor_area_ratio: float = 1.10
+ minimum_supplemental_attachment_score: float = 0.20
+ maximum_supplemental_attachment_area_ratio: float = 0.75
+ minimum_supplemental_added_fraction: float = 0.02
+ supplemental_bbox_expansion_fraction: float = 0.20
+ enable_semantic_component_absorption: bool = True
+ minimum_semantic_component_mask_containment: float = 0.90
+ maximum_semantic_component_area_ratio: float = 0.50
+
+
+def _atomic_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _tokens(value: Any) -> set[str]:
+ raw = re.findall(r"[a-z0-9]+", str(value or "").lower())
+ normalized = set(raw)
+ for token in raw:
+ if len(token) > 3 and token.endswith("ies"):
+ normalized.add(token[:-3] + "y")
+ elif len(token) > 3 and token.endswith("s"):
+ normalized.add(token[:-1])
+ return normalized
+
+
+def _object_tokens(obj: Mapping[str, Any]) -> set[str]:
+ return _tokens(
+ " ".join(
+ str(obj.get(key) or "")
+ for key in ("category", "sam3_prompt", "instance_name", "description")
+ )
+ )
+
+
+SEMANTIC_COMPONENT_TOKENS = frozenset(
+ {
+ "arm",
+ "armrest",
+ "back",
+ "backrest",
+ "base",
+ "cushion",
+ "door",
+ "drawer",
+ "footboard",
+ "frame",
+ "handle",
+ "headboard",
+ "leg",
+ "panel",
+ "seat",
+ "shelf",
+ }
+)
+
+SEMANTIC_ANCHOR_STOP_TOKENS = frozenset(
+ {
+ "a",
+ "an",
+ "and",
+ "black",
+ "blue",
+ "brown",
+ "dark",
+ "decorative",
+ "gray",
+ "green",
+ "large",
+ "light",
+ "metal",
+ "orange",
+ "red",
+ "small",
+ "the",
+ "white",
+ "wood",
+ "wooden",
+ }
+)
+
+
+def _semantic_component_anchor_tokens(
+ host: Mapping[str, Any], child: Mapping[str, Any]
+) -> list[str]:
+ """Return a meaningful host name shared by an intrinsic component."""
+ child_tokens = _object_tokens(child)
+ if not (child_tokens & SEMANTIC_COMPONENT_TOKENS):
+ return []
+ # Anchor on the host category, not shared colour/material adjectives in
+ # free-form prompts. A child may mention its host only in its prompt
+ # (for example, category="drawer", prompt="sideboard drawer").
+ host_name_tokens = (
+ _tokens(host.get("category"))
+ - SEMANTIC_COMPONENT_TOKENS
+ - SEMANTIC_ANCHOR_STOP_TOKENS
+ )
+ shared = host_name_tokens & child_tokens
+ return sorted(shared)
+
+
+def _has_pass_source(obj: Mapping[str, Any], pass_name: str) -> bool:
+ return any(
+ str(source.get("pass") or "").upper() == pass_name
+ for source in (obj.get("proposal_sources") or [])
+ if isinstance(source, Mapping)
+ )
+
+
+def _pass_a_source_ids(obj: Mapping[str, Any]) -> list[str]:
+ return [
+ str(source.get("source_id") or "")
+ for source in (obj.get("proposal_sources") or [])
+ if isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "A"
+ and str(source.get("source_id") or "")
+ ]
+
+
+def _bbox(obj: Mapping[str, Any]) -> list[float] | None:
+ try:
+ value = [float(item) for item in obj.get("bbox_2d_norm_xyxy")]
+ except (TypeError, ValueError):
+ return None
+ if len(value) != 4 or not (0.0 <= value[0] < value[2] <= 1.0 and 0.0 <= value[1] < value[3] <= 1.0):
+ return None
+ return value
+
+
+def _coverage_by_expanded_host(child: Sequence[float], host: Sequence[float], expansion: float) -> float:
+ hx1, hy1, hx2, hy2 = [float(value) for value in host]
+ pad_x = (hx2 - hx1) * expansion
+ pad_y = (hy2 - hy1) * expansion
+ hx1, hy1 = max(0.0, hx1 - pad_x), max(0.0, hy1 - pad_y)
+ hx2, hy2 = min(1.0, hx2 + pad_x), min(1.0, hy2 + pad_y)
+ cx1, cy1, cx2, cy2 = [float(value) for value in child]
+ intersection = max(0.0, min(hx2, cx2) - max(hx1, cx1)) * max(0.0, min(hy2, cy2) - max(hy1, cy1))
+ return intersection / max(1e-9, (cx2 - cx1) * (cy2 - cy1))
+
+
+def _mask_bbox_coverage(mask: np.ndarray, bbox_norm: Sequence[float]) -> float:
+ height, width = mask.shape
+ x1 = max(0, min(width - 1, int(round(float(bbox_norm[0]) * width))))
+ y1 = max(0, min(height - 1, int(round(float(bbox_norm[1]) * height))))
+ x2 = max(x1 + 1, min(width, int(round(float(bbox_norm[2]) * width))))
+ y2 = max(y1 + 1, min(height, int(round(float(bbox_norm[3]) * height))))
+ return int(mask[y1:y2, x1:x2].sum()) / max(1, (x2 - x1) * (y2 - y1))
+
+
+def _confirm_failed_duplicate_with_positive_bbox(
+ *,
+ child: Mapping[str, Any],
+ accepted_mask: np.ndarray,
+ image_path: str | Path | None,
+ session: Any,
+ infer_candidates_fn: Callable[..., Mapping[str, Any]] | None,
+ work_dir: Path,
+ cfg: HostPartMaskFusionConfig,
+) -> dict[str, Any]:
+ """Confirm that a failed sibling is already covered by one SAM3 component.
+
+ This is not a generic failed-mask bypass. It is available only to the
+ explicit host-assembly caller and uses the failed VLM box as a positive
+ SAM3 input.
+ """
+
+ box = _bbox(child)
+ if box is None or image_path is None or session is None or infer_candidates_fn is None:
+ return {"status": "not_run", "reason": "missing_local_probe_inputs"}
+ height, width = accepted_mask.shape
+ target = [
+ max(0, min(width - 1, int(round(box[0] * width)))),
+ max(0, min(height - 1, int(round(box[1] * height)))),
+ max(1, min(width, int(round(box[2] * width)))),
+ max(1, min(height, int(round(box[3] * height)))),
+ ]
+ target[2] = max(target[0] + 1, target[2])
+ target[3] = max(target[1] + 1, target[3])
+ prompt = str((child.get("sam3") or {}).get("prompt_used") or child.get("sam3_prompt") or "").strip()
+ if not prompt:
+ return {"status": "not_run", "reason": "missing_child_prompt"}
+ try:
+ result = infer_candidates_fn(
+ session, image_path, prompt, work_dir, score_threshold=0.20,
+ positive_bbox_abs_xyxy=target, maximum_candidates=4,
+ )
+ except Exception as exc:
+ return {"status": "error", "reason": f"{type(exc).__name__}: {exc}"}
+ records = list((result.get("registry") or {}).get("candidates") or [])
+ viable = []
+ for record in records:
+ try:
+ score = float(record.get("score") or 0.0)
+ path = Path(str(record.get("mask_path") or "")).expanduser().resolve()
+ if not path.is_file() or not path.is_relative_to(work_dir.resolve()):
+ continue
+ candidate = np.array(Image.open(path).convert("L")) > 0
+ if candidate.shape != accepted_mask.shape or not bool(candidate.any()):
+ continue
+ target_coverage = _mask_bbox_coverage(candidate, box)
+ covered_by_accepted = int(np.logical_and(candidate, accepted_mask).sum()) / max(1, int(candidate.sum()))
+ except (OSError, ValueError):
+ continue
+ if (
+ score >= cfg.minimum_confirmation_score
+ and target_coverage >= cfg.minimum_confirmation_target_coverage
+ and covered_by_accepted >= cfg.minimum_confirmation_covered_by_accepted_mask
+ ):
+ viable.append({
+ "candidate_id": str(record.get("candidate_id") or ""),
+ "score": round(score, 6),
+ "target_coverage": round(target_coverage, 6),
+ "covered_by_accepted_mask": round(covered_by_accepted, 6),
+ })
+ viable.sort(key=lambda item: (item["score"], item["target_coverage"], item["covered_by_accepted_mask"]), reverse=True)
+ if not viable:
+ return {"status": "not_confirmed", "positive_bbox_abs_xyxy": target, "candidate_count": len(records)}
+ return {
+ "status": "confirmed",
+ "positive_bbox_abs_xyxy": target,
+ "candidate_count": len(records),
+ "selected": viable[0],
+ }
+
+
+def _host_role(obj: Mapping[str, Any]) -> str | None:
+ tokens = _object_tokens(obj)
+ if {"coat", "rack"} <= tokens:
+ return "coat_rack"
+ if "pegboard" in tokens and ("rack" in tokens or "tool" in tokens):
+ return "pegboard"
+ if tokens & {"crate", "box", "caddy"} and tokens & {"tool", "utensil"}:
+ return "tool_crate"
+ if "towel" in tokens and ("ladder" in tokens or "rack" in tokens):
+ return "towel_support"
+ if "drying" in tokens and "rack" in tokens:
+ return "drying_rack"
+ if "roll" in tokens and ("holder" in tokens or "rack" in tokens):
+ return "roll_holder"
+ return None
+
+
+def _child_family(obj: Mapping[str, Any]) -> str | None:
+ tokens = _object_tokens(obj)
+ if tokens & {"bathrobe", "robe", "coat", "apron", "garment", "clothes", "clothing"}:
+ return "garment"
+ if "towel" in tokens:
+ return "towel"
+ if tokens & {"paper", "roll", "tube"}:
+ return "roll"
+ if tokens & {"tool", "brush", "hammer", "chisel", "scissors", "spatula", "wrench"}:
+ return "tool"
+ return None
+
+
+def _explicitly_claims(host: Mapping[str, Any], family: str) -> bool:
+ tokens = _object_tokens(host)
+ expected = {
+ "garment": {"bathrobe", "robe", "apron", "garment", "clothes", "clothing"},
+ "towel": {"towel"},
+ "roll": {"roll", "tube", "paper"},
+ "tool": {"tool", "brush", "hammer", "chisel", "scissors", "spatula", "wrench"},
+ }[family]
+ return bool(tokens & expected)
+
+
+def _role_allows(role: str, family: str) -> bool:
+ return {
+ "coat_rack": {"garment"},
+ "pegboard": {"garment", "tool"},
+ "tool_crate": {"tool"},
+ "towel_support": {"towel"},
+ "drying_rack": {"towel", "garment"},
+ "roll_holder": {"roll"},
+ }.get(role, set()) >= {family}
+
+
+def _complete_host_prompts(
+ role: str,
+ host: Mapping[str, Any] | None = None,
+) -> tuple[str, ...]:
+ # These are physical assemblies, not open-storage interiors: their
+ # attached/contained pixels belong to the host reconstruction object.
+ if role == "pegboard":
+ return (
+ "complete pegboard tool rack with every attached hanging garment and hand tool",
+ "complete pegboard including all visible hanging tools and clothing",
+ )
+ if role == "tool_crate":
+ container_name = (
+ "tool caddy"
+ if "caddy" in _object_tokens(host or {})
+ else "tool crate"
+ )
+ return (
+ f"complete {container_name} together with every visible tool contained in or resting on it",
+ f"cluster of all visible hand tools held inside and protruding above the {container_name}",
+ )
+ return ()
+
+
+def _probe_complete_functional_host(
+ *,
+ role: str,
+ host: Mapping[str, Any],
+ host_mask: np.ndarray,
+ image_path: str | Path | None,
+ session: Any,
+ infer_candidates_fn: Callable[..., Mapping[str, Any]] | None,
+ work_dir: Path,
+ cfg: HostPartMaskFusionConfig,
+) -> dict[str, Any]:
+ prompts = _complete_host_prompts(role, host)
+ if not prompts:
+ return {"status": "not_applicable", "inference_count": 0}
+ box = _bbox(host)
+ if box is None or image_path is None or session is None or infer_candidates_fn is None:
+ return {
+ "status": "not_run",
+ "reason": "missing_local_probe_inputs",
+ "inference_count": 0,
+ }
+ height, width = host_mask.shape
+ positive_box = [
+ max(0, min(width - 1, int(round(box[0] * width)))),
+ max(0, min(height - 1, int(round(box[1] * height)))),
+ max(1, min(width, int(round(box[2] * width)))),
+ max(1, min(height, int(round(box[3] * height)))),
+ ]
+ positive_box[2] = max(positive_box[0] + 1, positive_box[2])
+ positive_box[3] = max(positive_box[1] + 1, positive_box[3])
+ if role == "tool_crate":
+ # VLM boxes for caddies/crates commonly stop at the container rim and
+ # omit tools protruding above it. Expand only this functional role's
+ # local SAM3 support box upward; generic host boxes remain unchanged.
+ box_width = positive_box[2] - positive_box[0]
+ box_height = positive_box[3] - positive_box[1]
+ pad_x = max(2, int(round(box_width * 0.08)))
+ positive_box = [
+ max(0, positive_box[0] - pad_x),
+ max(0, positive_box[1] - box_height),
+ min(width, positive_box[2] + pad_x),
+ min(height, positive_box[3] + max(1, int(round(box_height * 0.10)))),
+ ]
+ host_area = max(1, int(host_mask.sum()))
+ viable: list[dict[str, Any]] = []
+ all_candidates: list[dict[str, Any]] = []
+ records: list[dict[str, Any]] = []
+ for index, prompt in enumerate(prompts[: cfg.maximum_complete_host_probes], 1):
+ probe_dir = work_dir / "complete_functional_host" / str(host.get("object_id") or "host") / f"probe_{index:02d}"
+ try:
+ result = infer_candidates_fn(
+ session,
+ image_path,
+ prompt,
+ probe_dir,
+ score_threshold=0.20,
+ positive_bbox_abs_xyxy=positive_box,
+ maximum_candidates=cfg.maximum_candidates_per_complete_host_probe,
+ )
+ except Exception as exc:
+ records.append({"prompt": prompt, "status": "error", "error": f"{type(exc).__name__}: {exc}"})
+ continue
+ candidate_records = list((result.get("registry") or {}).get("candidates") or [])
+ records.append({"prompt": prompt, "status": "completed", "candidate_count": len(candidate_records)})
+ for candidate in candidate_records:
+ try:
+ score = float(candidate.get("score") or 0.0)
+ path = Path(str(candidate.get("mask_path") or "")).expanduser().resolve()
+ mask = np.array(Image.open(path).convert("L")) > 0
+ except (OSError, ValueError):
+ continue
+ if mask.shape != host_mask.shape or not bool(mask.any()):
+ continue
+ host_coverage = int(np.logical_and(mask, host_mask).sum()) / host_area
+ area_ratio = int(mask.sum()) / host_area
+ added_fraction = int(np.logical_and(mask, ~host_mask).sum()) / host_area
+ all_candidates.append({
+ "mask": mask,
+ "candidate_id": str(candidate.get("candidate_id") or ""),
+ "prompt": prompt,
+ "score": score,
+ "host_coverage": host_coverage,
+ "added_fraction": added_fraction,
+ "area_ratio": area_ratio,
+ })
+ if (
+ score < cfg.minimum_complete_host_score
+ or host_coverage < cfg.minimum_complete_host_original_coverage
+ or (
+ added_fraction < cfg.minimum_complete_host_added_fraction
+ and not cfg.minimum_idempotent_anchor_area_ratio
+ <= area_ratio <= cfg.maximum_idempotent_anchor_area_ratio
+ )
+ or area_ratio > cfg.maximum_complete_host_area_ratio
+ ):
+ continue
+ viable.append({
+ "mask": mask,
+ "candidate_id": str(candidate.get("candidate_id") or ""),
+ "prompt": prompt,
+ "score": score,
+ "host_coverage": host_coverage,
+ "added_fraction": added_fraction,
+ "area_ratio": area_ratio,
+ })
+ if not viable:
+ return {
+ "status": "not_confirmed",
+ "inference_count": min(len(prompts), cfg.maximum_complete_host_probes),
+ "records": records,
+ }
+ selected = sorted(
+ viable,
+ key=lambda item: (item["score"], item["host_coverage"], item["added_fraction"], -item["area_ratio"]),
+ reverse=True,
+ )[0]
+ combined = selected["mask"].copy()
+ supplements: list[dict[str, Any]] = []
+ host_bbox = [float(value) for value in box]
+ for item in sorted(
+ all_candidates,
+ key=lambda value: (value["score"], value["added_fraction"]),
+ reverse=True,
+ ):
+ if item is selected or np.array_equal(item["mask"], selected["mask"]):
+ continue
+ candidate_mask = item["mask"]
+ ys, xs = np.nonzero(candidate_mask)
+ candidate_box = [
+ float(xs.min()) / width,
+ float(ys.min()) / height,
+ float(xs.max() + 1) / width,
+ float(ys.max() + 1) / height,
+ ]
+ bbox_coverage = _coverage_by_expanded_host(
+ candidate_box, host_bbox, cfg.supplemental_bbox_expansion_fraction
+ )
+ new_pixels = int(np.logical_and(candidate_mask, ~combined).sum())
+ new_fraction = new_pixels / host_area
+ if not (
+ item["score"] >= cfg.minimum_supplemental_attachment_score
+ and item["area_ratio"] <= cfg.maximum_supplemental_attachment_area_ratio
+ and new_fraction >= cfg.minimum_supplemental_added_fraction
+ and bbox_coverage >= 0.70
+ ):
+ continue
+ combined |= candidate_mask
+ supplements.append({
+ "candidate_id": item["candidate_id"],
+ "prompt": item["prompt"],
+ "score": round(float(item["score"]), 6),
+ "bbox_coverage_by_expanded_host": round(bbox_coverage, 6),
+ "new_pixels": new_pixels,
+ "new_fraction": round(new_fraction, 6),
+ })
+ if (
+ not supplements
+ and selected["added_fraction"] < cfg.minimum_complete_host_added_fraction
+ ):
+ return {
+ "status": "not_confirmed",
+ "reason": "idempotent_anchor_without_supplement",
+ "inference_count": min(len(prompts), cfg.maximum_complete_host_probes),
+ "records": records,
+ "selected": {key: value for key, value in selected.items() if key != "mask"},
+ }
+ return {
+ "status": "confirmed",
+ "inference_count": min(len(prompts), cfg.maximum_complete_host_probes),
+ "records": records,
+ "selected": {key: value for key, value in selected.items() if key != "mask"},
+ "supplemental_attachments": supplements,
+ "mask": combined,
+ }
+
+
+def _success_mask(obj: Mapping[str, Any], shape: tuple[int, int], staging: Path) -> np.ndarray | None:
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "success":
+ return None
+ path = Path(str(sam3.get("mask_path") or "")).expanduser().resolve()
+ if not path.is_file() or not path.is_relative_to(staging):
+ return None
+ mask = np.array(Image.open(path).convert("L")) > 0
+ return mask if mask.shape == shape and bool(mask.any()) else None
+
+
+def _write_mask_and_overlay(
+ obj: dict[str, Any],
+ mask: np.ndarray,
+ image_np: np.ndarray,
+ staging: Path,
+ *,
+ selection_method: str = "dedicated_attachment_host_union",
+) -> None:
+ object_id = str(obj.get("object_id") or "object")
+ category = re.sub(r"[^a-z0-9]+", "_", str(obj.get("category") or "object").lower()).strip("_") or "object"
+ mask_path = staging / "masks" / f"{object_id}_{category}_mask.png"
+ overlay_path = staging / "overlays" / f"{object_id}_{category}_overlay.png"
+ mask_path.parent.mkdir(parents=True, exist_ok=True)
+ overlay_path.parent.mkdir(parents=True, exist_ok=True)
+ Image.fromarray(mask.astype(np.uint8) * 255).save(mask_path)
+ overlay = image_np.copy()
+ overlay[mask] = (0.55 * overlay[mask] + 0.45 * np.array((255, 0, 0), dtype=np.uint8)).astype(np.uint8)
+ Image.fromarray(overlay).save(overlay_path)
+ ys, xs = np.nonzero(mask)
+ sam3 = copy.deepcopy(dict(obj.get("sam3") or {}))
+ sam3.update({
+ "mask_path": str(mask_path), "overlay_path": str(overlay_path),
+ "area_pixels": int(mask.sum()),
+ "mask_bbox_abs_xyxy": [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1],
+ "selection_method": selection_method,
+ })
+ obj["sam3"] = sam3
+
+
+def _renumber(objects: list[dict[str, Any]], staging: Path) -> dict[str, str]:
+ remap: dict[str, str] = {}
+ for index, obj in enumerate(objects, start=1):
+ old_id = str(obj.get("object_id") or "")
+ new_id = f"proposal_{index:03d}"
+ remap[old_id] = new_id
+ obj["object_id"] = new_id
+ for obj in objects:
+ old_id = next(key for key, value in remap.items() if value == obj["object_id"])
+ if old_id == obj["object_id"]:
+ continue
+ sam3 = obj.get("sam3") or {}
+ for key in ("mask_path", "overlay_path"):
+ path = Path(str(sam3.get(key) or "")).expanduser().resolve()
+ if not path.is_file() or not path.is_relative_to(staging) or not path.name.startswith(old_id + "_"):
+ continue
+ target = path.with_name(obj["object_id"] + path.name[len(old_id):])
+ if target.exists():
+ target.unlink()
+ path.rename(target)
+ sam3[key] = str(target)
+ return remap
+
+
+def run_host_part_mask_fusion(
+ *, image_np: np.ndarray, objects: Sequence[Mapping[str, Any]], staging_dir: str | Path,
+ work_dir: str | Path, config: HostPartMaskFusionConfig | None = None,
+ image_path: str | Path | None = None, session: Any = None,
+ infer_candidates_fn: Callable[..., Mapping[str, Any]] | None = None,
+) -> dict[str, Any]:
+ cfg = config or HostPartMaskFusionConfig()
+ if not cfg.isolated_attempt_only:
+ raise ValueError("host part mask fusion is isolated-attempt only")
+ copied = [copy.deepcopy(dict(item)) for item in objects]
+ staging = Path(staging_dir).expanduser().resolve()
+ work = Path(work_dir).expanduser().resolve()
+ work.mkdir(parents=True, exist_ok=True)
+ audit: dict[str, Any] = {
+ "schema_version": "scenemosaic_dedicated_attachment_absorption",
+ "enabled": cfg.enabled, "status": "disabled" if not cfg.enabled else "noop",
+ "absorbed_hosts": [], "removed_duplicate_additions": [],
+ "removed_object_ids": [], "completed_host_source_ids": [],
+ "positive_bbox_duplicate_confirmations": [],
+ "functional_host_completions": [],
+ "semantic_component_absorptions": [],
+ "semantic_component_rejections": [],
+ "id_remap": {}, "hard_errors": [],
+ "additional_sam3_inference_calls": 0, "additional_vlm_calls": 0,
+ }
+ if not cfg.enabled:
+ _atomic_json(work / "dedicated_attachment_absorption_audit.json", audit)
+ return {"objects": copied, "audit": audit}
+ shape = tuple(int(value) for value in image_np.shape[:2])
+ remove_ids: set[str] = set()
+ positive_confirmation_count = 0
+ for host in copied:
+ role = _host_role(host)
+ host_box = _bbox(host)
+ host_mask = _success_mask(host, shape, staging)
+ if role is None or host_box is None or host_mask is None or not _has_pass_source(host, "A"):
+ continue
+ completion = _probe_complete_functional_host(
+ role=role,
+ host=host,
+ host_mask=host_mask,
+ image_path=image_path,
+ session=session,
+ infer_candidates_fn=infer_candidates_fn,
+ work_dir=work,
+ cfg=cfg,
+ )
+ completion_mask = completion.pop("mask", None)
+ audit["additional_sam3_inference_calls"] += int(
+ completion.get("inference_count") or 0
+ )
+ audit["functional_host_completions"].append(
+ {
+ "host_object_id": str(host.get("object_id") or ""),
+ "host_role": role,
+ **completion,
+ }
+ )
+ completed_by_probe = completion_mask is not None
+ accepted: list[tuple[dict[str, Any], np.ndarray, str, float]] = []
+ for child in copied:
+ if child is host or not _has_pass_source(child, "B"):
+ continue
+ family = _child_family(child)
+ child_box = _bbox(child)
+ child_mask = _success_mask(child, shape, staging)
+ if family is None or child_box is None or child_mask is None:
+ continue
+ coverage = _coverage_by_expanded_host(
+ child_box, host_box, cfg.host_bbox_expansion_fraction
+ )
+ explicit_or_intrinsic = _explicitly_claims(host, family) or (
+ role == "pegboard" and family in {"garment", "tool"}
+ ) or (role == "tool_crate" and family == "tool")
+ if (
+ _role_allows(role, family)
+ and explicit_or_intrinsic
+ and coverage >= cfg.minimum_child_box_coverage_by_host
+ ):
+ accepted.append((child, child_mask, family, coverage))
+ if not accepted and not completed_by_probe:
+ continue
+ merged = (
+ np.asarray(completion_mask, dtype=bool).copy()
+ if completed_by_probe
+ else host_mask.copy()
+ )
+ host_remove_ids: set[str] = set()
+ for child, child_mask, _, _ in accepted:
+ merged |= child_mask
+ host_remove_ids.add(str(child.get("object_id") or ""))
+ removed_failed = []
+ for child in copied:
+ child_id = str(child.get("object_id") or "")
+ if (
+ child is host
+ or child_id in host_remove_ids
+ or not _has_pass_source(child, "B")
+ ):
+ continue
+ family = _child_family(child)
+ child_box = _bbox(child)
+ status = str((child.get("sam3") or {}).get("status") or "")
+ if family is None or child_box is None or status != "failed":
+ continue
+ explicit_or_intrinsic = _explicitly_claims(host, family) or (
+ role == "pegboard" and family in {"garment", "tool"}
+ ) or (role == "tool_crate" and family == "tool")
+ if not (_role_allows(role, family) and explicit_or_intrinsic):
+ continue
+ host_coverage = _coverage_by_expanded_host(
+ child_box, host_box, cfg.host_bbox_expansion_fraction
+ )
+ mask_coverage = _mask_bbox_coverage(merged, child_box)
+ if host_coverage < cfg.minimum_child_box_coverage_by_host:
+ continue
+ if mask_coverage >= cfg.minimum_failed_sibling_box_coverage:
+ host_remove_ids.add(child_id)
+ removed_failed.append({
+ "object_id": child_id, "family": family,
+ "accepted_mask_bbox_coverage": round(mask_coverage, 6),
+ "resolution": "direct_accepted_mask_coverage",
+ })
+ continue
+ if positive_confirmation_count >= cfg.maximum_positive_bbox_duplicate_confirmations:
+ continue
+ confirmation = _confirm_failed_duplicate_with_positive_bbox(
+ child=child, accepted_mask=merged, image_path=image_path,
+ session=session, infer_candidates_fn=infer_candidates_fn,
+ work_dir=work / "positive_bbox_duplicate_confirmation" / child_id,
+ cfg=cfg,
+ )
+ positive_confirmation_count += 1
+ audit["additional_sam3_inference_calls"] += 1
+ audit["positive_bbox_duplicate_confirmations"].append({
+ "host_object_id": str(host.get("object_id") or ""),
+ "failed_child_object_id": child_id, "family": family,
+ "accepted_mask_bbox_coverage": round(mask_coverage, 6),
+ **confirmation,
+ })
+ if confirmation.get("status") == "confirmed":
+ host_remove_ids.add(child_id)
+ removed_failed.append({
+ "object_id": child_id, "family": family,
+ "accepted_mask_bbox_coverage": round(mask_coverage, 6),
+ "resolution": "positive_bbox_duplicate_confirmation",
+ "confirmation": confirmation,
+ })
+ if not host_remove_ids and not completed_by_probe:
+ continue
+ remove_ids.update(host_remove_ids)
+ _write_mask_and_overlay(
+ host,
+ merged,
+ image_np,
+ staging,
+ selection_method=(
+ "dedicated_attachment_host_completion_union"
+ if completed_by_probe
+ else "dedicated_attachment_host_union"
+ ),
+ )
+ host_sources = _pass_a_source_ids(host)
+ audit["completed_host_source_ids"].extend(host_sources)
+ audit["absorbed_hosts"].append({
+ "host_object_id": str(host.get("object_id") or ""),
+ "host_source_object_ids": host_sources,
+ "host_role": role,
+ "completion_probe_status": completion.get("status"),
+ "accepted_children": [
+ {
+ "object_id": str(child.get("object_id") or ""),
+ "family": family,
+ "bbox_coverage_by_expanded_host": round(coverage, 6),
+ }
+ for child, _, family, coverage in accepted
+ ],
+ "removed_failed_duplicates": removed_failed,
+ })
+ audit["removed_duplicate_additions"].extend(removed_failed)
+
+ # A Pass-B item such as "sofa cushion" can be a named intrinsic part of
+ # one pre-existing furniture object. Unlike a loose "throw pillow", the
+ # shared host noun makes this ownership explicit. Absorb only when mask
+ # and bbox containment prove a unique host; otherwise leave the quality
+ # gate to report the ambiguity rather than guessing.
+ if cfg.enable_semantic_component_absorption:
+ semantic_children_by_host: dict[str, list[dict[str, Any]]] = {}
+ for child in copied:
+ child_id = str(child.get("object_id") or "")
+ if (
+ not child_id
+ or child_id in remove_ids
+ or not _has_pass_source(child, "B")
+ ):
+ continue
+ child_box = _bbox(child)
+ child_mask = _success_mask(child, shape, staging)
+ if child_box is None or child_mask is None:
+ continue
+ child_area = max(1, int(child_mask.sum()))
+ matches: list[dict[str, Any]] = []
+ for host in copied:
+ host_id = str(host.get("object_id") or "")
+ if (
+ not host_id
+ or host_id == child_id
+ or host_id in remove_ids
+ or not _has_pass_source(host, "A")
+ ):
+ continue
+ anchors = _semantic_component_anchor_tokens(host, child)
+ if not anchors:
+ continue
+ host_box = _bbox(host)
+ host_mask = _success_mask(host, shape, staging)
+ if host_box is None or host_mask is None:
+ continue
+ host_area = max(1, int(host_mask.sum()))
+ contained_pixels = int(np.logical_and(child_mask, host_mask).sum())
+ child_containment = contained_pixels / child_area
+ child_to_host_area_ratio = child_area / host_area
+ bbox_coverage = _coverage_by_expanded_host(
+ child_box, host_box, cfg.host_bbox_expansion_fraction
+ )
+ if (
+ child_containment
+ < cfg.minimum_semantic_component_mask_containment
+ or child_to_host_area_ratio
+ > cfg.maximum_semantic_component_area_ratio
+ or bbox_coverage
+ < cfg.minimum_child_box_coverage_by_host
+ ):
+ continue
+ matches.append(
+ {
+ "host": host,
+ "host_id": host_id,
+ "child_containment": child_containment,
+ "child_to_host_area_ratio": child_to_host_area_ratio,
+ "bbox_coverage_by_expanded_host": bbox_coverage,
+ "shared_host_tokens": anchors,
+ }
+ )
+ if len(matches) != 1:
+ if len(matches) > 1:
+ audit["semantic_component_rejections"].append(
+ {
+ "child_object_id": child_id,
+ "reason": "ambiguous_semantic_component_host",
+ "candidate_host_object_ids": sorted(
+ str(match["host_id"]) for match in matches
+ ),
+ }
+ )
+ continue
+ match = matches[0]
+ semantic_children_by_host.setdefault(
+ str(match["host_id"]), []
+ ).append(
+ {
+ "child": child,
+ "child_mask": child_mask,
+ **{key: value for key, value in match.items() if key != "host"},
+ }
+ )
+
+ for host_id, records in semantic_children_by_host.items():
+ host = next(
+ (
+ item
+ for item in copied
+ if str(item.get("object_id") or "") == host_id
+ ),
+ None,
+ )
+ if host is None:
+ continue
+ host_mask = _success_mask(host, shape, staging)
+ if host_mask is None:
+ continue
+ merged = host_mask.copy()
+ child_records: list[dict[str, Any]] = []
+ for record in records:
+ child = record["child"]
+ child_id = str(child.get("object_id") or "")
+ added_pixels = int(
+ np.logical_and(record["child_mask"], ~merged).sum()
+ )
+ merged |= record["child_mask"]
+ remove_ids.add(child_id)
+ child_records.append(
+ {
+ "object_id": child_id,
+ "shared_host_tokens": record["shared_host_tokens"],
+ "child_mask_containment": round(
+ float(record["child_containment"]), 6
+ ),
+ "child_to_host_area_ratio": round(
+ float(record["child_to_host_area_ratio"]), 6
+ ),
+ "bbox_coverage_by_expanded_host": round(
+ float(record["bbox_coverage_by_expanded_host"]), 6
+ ),
+ "new_host_pixels": added_pixels,
+ }
+ )
+ if not np.array_equal(merged, host_mask):
+ _write_mask_and_overlay(
+ host,
+ merged,
+ image_np,
+ staging,
+ selection_method="semantic_host_component_absorption_union",
+ )
+ audit["semantic_component_absorptions"].append(
+ {
+ "host_object_id": host_id,
+ "host_source_object_ids": _pass_a_source_ids(host),
+ "resolution": "unique_named_intrinsic_component",
+ "absorbed_children": child_records,
+ }
+ )
+ if remove_ids or audit["completed_host_source_ids"]:
+ for item in copied:
+ if str(item.get("object_id") or "") not in remove_ids:
+ continue
+ for key in ("mask_path", "overlay_path"):
+ path = Path(str((item.get("sam3") or {}).get(key) or "")).expanduser().resolve()
+ if path.is_file() and path.is_relative_to(staging):
+ path.unlink()
+ copied = [item for item in copied if str(item.get("object_id") or "") not in remove_ids]
+ audit["removed_object_ids"] = sorted(remove_ids)
+ if remove_ids:
+ audit["id_remap"] = _renumber(copied, staging)
+ audit["completed_host_source_ids"] = sorted(set(audit["completed_host_source_ids"]))
+ audit["status"] = "pass"
+ _atomic_json(work / "dedicated_attachment_absorption_audit.json", audit)
+ return {"objects": copied, "audit": audit}
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/low_profile_accessory_suppression.py b/vlm_agent/reconstruction/perception/pipeline/refinements/low_profile_accessory_suppression.py
new file mode 100644
index 0000000..517f40d
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/low_profile_accessory_suppression.py
@@ -0,0 +1,199 @@
+#!/usr/bin/env python3
+"""Suppress a proved low-profile accessory that is not reconstructable alone.
+
+This is intentionally geometric rather than category-specific. It removes a
+very small, thin mask only when a larger, vertically extended retained object
+directly overlaps it as a local dependent. Typical examples are a saucer or
+tray beneath an already-separated candle or cup. It never edits source masks
+and keeps an audit explaining every suppression.
+"""
+
+from __future__ import annotations
+
+import copy
+import json
+import os
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+
+@dataclass(frozen=True)
+class LowProfileAccessoryConfig:
+ enabled: bool = True
+ isolated_attempt_only: bool = True
+ maximum_area_fraction: float = 0.0005
+ maximum_height_fraction: float = 0.03
+ minimum_width_to_height_ratio: float = 1.5
+ minimum_dependent_area_ratio: float = 1.5
+ minimum_dependent_height_ratio: float = 1.5
+ minimum_dependent_horizontal_coverage: float = 0.75
+ minimum_mask_intersection_pixels: int = 20
+ vertical_tolerance_fraction: float = 0.004
+
+
+def _atomic_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _load_mask(path_value: Any, expected_shape: tuple[int, int]) -> np.ndarray:
+ path = Path(str(path_value or "")).expanduser().resolve()
+ if not path.is_file():
+ raise FileNotFoundError(path)
+ mask = np.array(Image.open(path).convert("L")) > 0
+ if mask.shape != expected_shape or not bool(mask.any()):
+ raise ValueError("invalid_mask_shape_or_empty")
+ return mask
+
+
+def _bbox(mask: np.ndarray) -> list[int]:
+ ys, xs = np.where(mask)
+ if not len(xs):
+ raise ValueError("empty_mask")
+ return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
+
+
+def _horizontal_coverage(base: Sequence[int], dependent: Sequence[int]) -> float:
+ overlap = max(
+ 0,
+ min(int(base[2]), int(dependent[2]))
+ - max(int(base[0]), int(dependent[0])),
+ )
+ dependent_width = max(1, int(dependent[2]) - int(dependent[0]))
+ return overlap / dependent_width
+
+
+def run_low_profile_accessory_suppression(
+ *,
+ objects: Sequence[Mapping[str, Any]],
+ image_shape: tuple[int, int],
+ output_dir: str | Path,
+ config: LowProfileAccessoryConfig | None = None,
+) -> dict[str, Any]:
+ """Remove only a small horizontal accessory proved to support a child."""
+
+ cfg = config or LowProfileAccessoryConfig()
+ if not cfg.isolated_attempt_only:
+ raise ValueError("low-profile accessory suppression is isolated-attempt only")
+ work = Path(output_dir).expanduser().resolve()
+ work.mkdir(parents=True, exist_ok=True)
+ copied = [
+ copy.deepcopy(dict(obj)) for obj in objects if isinstance(obj, Mapping)
+ ]
+ height, width = image_shape
+ audit: dict[str, Any] = {
+ "schema_version": "scenemosaic_low_profile_accessory_suppression",
+ "enabled": cfg.enabled,
+ "isolated_attempt_only": cfg.isolated_attempt_only,
+ "status": "disabled" if not cfg.enabled else "noop",
+ "input_object_count": len(copied),
+ "suppressed": [],
+ "rejected": [],
+ "hard_errors": [],
+ "thresholds": asdict(cfg),
+ }
+ if not cfg.enabled:
+ _atomic_json(work / "low_profile_accessory_suppression_audit.json", audit)
+ return {"objects": copied, "audit": audit}
+
+ masks: dict[str, np.ndarray] = {}
+ boxes: dict[str, list[int]] = {}
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "success":
+ continue
+ try:
+ mask = _load_mask(sam3.get("mask_path"), image_shape)
+ masks[object_id] = mask
+ boxes[object_id] = _bbox(mask)
+ except (OSError, ValueError) as exc:
+ audit["hard_errors"].append(
+ {
+ "type": "invalid_success_mask_for_low_profile_accessory",
+ "object_id": object_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+
+ removal_ids: set[str] = set()
+ tolerance = max(1, int(round(height * cfg.vertical_tolerance_fraction)))
+ for base in copied:
+ base_id = str(base.get("object_id") or "")
+ if base_id not in masks:
+ continue
+ base_mask = masks[base_id]
+ base_area = int(base_mask.sum())
+ x1, y1, x2, y2 = boxes[base_id]
+ base_width, base_height = x2 - x1, y2 - y1
+ if (
+ base_area / max(1, height * width) > cfg.maximum_area_fraction
+ or base_height / max(1, height) > cfg.maximum_height_fraction
+ or base_width / max(1, base_height) < cfg.minimum_width_to_height_ratio
+ ):
+ continue
+
+ dependents = []
+ for child in copied:
+ child_id = str(child.get("object_id") or "")
+ if child_id == base_id or child_id not in masks:
+ continue
+ child_mask = masks[child_id]
+ child_area = int(child_mask.sum())
+ cx1, cy1, cx2, cy2 = boxes[child_id]
+ child_height = cy2 - cy1
+ intersection = int(np.logical_and(base_mask, child_mask).sum())
+ horizontal_coverage = _horizontal_coverage(
+ boxes[base_id], boxes[child_id]
+ )
+ directly_supported = cy1 < y2 and cy2 >= y1 - tolerance
+ if (
+ child_area >= base_area * cfg.minimum_dependent_area_ratio
+ and child_height >= base_height * cfg.minimum_dependent_height_ratio
+ and horizontal_coverage >= cfg.minimum_dependent_horizontal_coverage
+ and directly_supported
+ and intersection >= cfg.minimum_mask_intersection_pixels
+ ):
+ dependents.append(
+ {
+ "object_id": child_id,
+ "category": str(child.get("category") or ""),
+ "area_pixels": child_area,
+ "mask_intersection_pixels": intersection,
+ "horizontal_coverage": round(horizontal_coverage, 6),
+ "bbox_abs_xyxy": [cx1, cy1, cx2, cy2],
+ }
+ )
+ if not dependents:
+ continue
+ removal_ids.add(base_id)
+ audit["suppressed"].append(
+ {
+ "removed_object_id": base_id,
+ "removed_category": str(base.get("category") or ""),
+ "policy": "low_profile_accessory_with_local_dependent",
+ "area_pixels": base_area,
+ "bbox_abs_xyxy": [x1, y1, x2, y2],
+ "dependent_objects": dependents,
+ }
+ )
+
+ result = [
+ obj for obj in copied if str(obj.get("object_id") or "") not in removal_ids
+ ]
+ audit["status"] = "pass" if removal_ids else "noop"
+ audit["output_object_count"] = len(result)
+ _atomic_json(work / "low_profile_accessory_suppression_audit.json", audit)
+ return {"objects": result, "audit": audit}
+
+
+__all__ = ["LowProfileAccessoryConfig", "run_low_profile_accessory_suppression"]
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/mask_ownership_resolution.py b/vlm_agent/reconstruction/perception/pipeline/refinements/mask_ownership_resolution.py
new file mode 100644
index 0000000..563b10c
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/mask_ownership_resolution.py
@@ -0,0 +1,251 @@
+#!/usr/bin/env python3
+"""Deterministic SEG-008 ownership cleanup for verified open-storage hosts."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+import sys
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+MODULE_DIR = Path(__file__).resolve().parent
+if str(MODULE_DIR) not in sys.path:
+ sys.path.insert(0, str(MODULE_DIR))
+
+from . import repair_scope_router as SCOPE
+
+
+def _sha256(path: Path) -> str:
+ return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def _load_mask(path: Any, shape: tuple[int, int]) -> tuple[Path, np.ndarray]:
+ resolved = Path(str(path or "")).expanduser().resolve()
+ if not resolved.is_file():
+ raise FileNotFoundError(resolved)
+ mask = np.array(Image.open(resolved).convert("L")) > 0
+ if mask.shape != shape:
+ raise ValueError(f"mask shape mismatch: {mask.shape} != {shape}")
+ return resolved, mask
+
+
+def _bbox(mask: np.ndarray) -> list[int]:
+ ys, xs = np.where(mask)
+ if not len(xs):
+ raise ValueError("empty mask")
+ return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
+
+
+def _safe(value: Any) -> str:
+ return re.sub(r"[^a-z0-9]+", "_", str(value or "").lower()).strip("_") or "object"
+
+
+def resolve_open_storage_ownership(
+ *,
+ image_path: str | Path,
+ objects: Sequence[Mapping[str, Any]],
+ output_dir: str | Path,
+) -> dict[str, Any]:
+ """Subtract verified nested child pixels from open-storage structure masks."""
+
+ resolved_image = Path(image_path).expanduser().resolve()
+ image = np.array(Image.open(resolved_image).convert("RGB"), dtype=np.uint8)
+ shape = tuple(image.shape[:2])
+ output = Path(output_dir).expanduser().resolve()
+ if output.exists() and any(output.iterdir()):
+ raise FileExistsError(f"ownership output is not empty: {output}")
+ output.mkdir(parents=True, exist_ok=True)
+
+ copied = [dict(item) for item in objects]
+ masks: dict[str, np.ndarray] = {}
+ source_paths: dict[str, Path] = {}
+ source_hashes: dict[str, str] = {}
+ hard_errors: list[dict[str, Any]] = []
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ sam3 = dict(obj.get("sam3") or {})
+ if sam3.get("status") != "success":
+ continue
+ try:
+ path, mask = _load_mask(sam3.get("mask_path"), shape)
+ if not mask.any():
+ raise ValueError("empty mask")
+ except Exception as exc:
+ hard_errors.append(
+ {
+ "type": "invalid_mask_for_ownership",
+ "object_id": object_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ continue
+ masks[object_id] = mask
+ source_paths[object_id] = path
+ source_hashes[object_id] = _sha256(path)
+
+ records: list[dict[str, Any]] = []
+ changed_ids: list[str] = []
+ for host in copied:
+ host_id = str(host.get("object_id") or "")
+ if host_id not in masks:
+ continue
+ scope = SCOPE.classify_object_role(host)
+ if scope.get("object_role") != SCOPE.OPEN_STORAGE_ROLE:
+ continue
+ host_mask = masks[host_id]
+ host_box = _bbox(host_mask)
+ hx1, hy1, hx2, hy2 = host_box
+ child_ids: list[str] = []
+ intersection_before: dict[str, int] = {}
+ for child in copied:
+ child_id = str(child.get("object_id") or "")
+ if child_id == host_id or child_id not in masks:
+ continue
+ child_mask = masks[child_id]
+ child_area = int(child_mask.sum())
+ if not child_area:
+ continue
+ inside_fraction = (
+ int(child_mask[hy1:hy2, hx1:hx2].sum()) / child_area
+ )
+ intersection = int(np.logical_and(host_mask, child_mask).sum())
+ if inside_fraction < 0.70 or intersection <= 0:
+ continue
+ child_scope = SCOPE.classify_object_role(child)
+ if (
+ child_scope.get("object_role")
+ in {SCOPE.OPEN_STORAGE_ROLE, SCOPE.SUPPORT_SURFACE_ROLE}
+ and child_area > 0.25 * int(host_mask.sum())
+ ):
+ continue
+ child_ids.append(child_id)
+ intersection_before[child_id] = intersection
+
+ if not child_ids:
+ records.append(
+ {
+ "host_object_id": host_id,
+ "repair_scope": scope,
+ "status": "no_overlapping_nested_contents",
+ "child_object_ids": [],
+ "removed_pixels": 0,
+ }
+ )
+ continue
+ child_union = np.zeros(shape, dtype=bool)
+ for child_id in child_ids:
+ child_union |= masks[child_id]
+ resolved = host_mask & ~child_union
+ if not resolved.any():
+ hard_errors.append(
+ {
+ "type": "host_subtraction_emptied_mask",
+ "host_object_id": host_id,
+ "child_object_ids": child_ids,
+ }
+ )
+ continue
+ removed = int(host_mask.sum()) - int(resolved.sum())
+ masks_dir = output / "masks"
+ overlays_dir = output / "overlays"
+ masks_dir.mkdir(parents=True, exist_ok=True)
+ overlays_dir.mkdir(parents=True, exist_ok=True)
+ stem = f"{_safe(host_id)}_{_safe(host.get('category'))}"
+ mask_path = masks_dir / f"{stem}_mask.png"
+ overlay_path = overlays_dir / f"{stem}_overlay.png"
+ Image.fromarray(resolved.astype(np.uint8) * 255).save(mask_path)
+ canvas = image.astype(np.float32).copy()
+ canvas[resolved] = (
+ 0.55 * canvas[resolved]
+ + 0.45 * np.array((255, 0, 0), dtype=np.float32)
+ )
+ Image.fromarray(np.clip(canvas, 0, 255).astype(np.uint8)).save(
+ overlay_path
+ )
+ sam3 = dict(host.get("sam3") or {})
+ sam3.update(
+ {
+ "mask_path": str(mask_path),
+ "overlay_path": str(overlay_path),
+ "area_pixels": int(resolved.sum()),
+ "mask_bbox_abs_xyxy": _bbox(resolved),
+ "pixel_ownership_resolution": {
+ "action": "host_minus_contents",
+ "source_mask_path": str(source_paths[host_id]),
+ "child_object_ids": child_ids,
+ "removed_pixels": removed,
+ },
+ }
+ )
+ host["sam3"] = sam3
+ changed_ids.append(host_id)
+ masks[host_id] = resolved
+ after = {
+ child_id: int(np.logical_and(resolved, masks[child_id]).sum())
+ for child_id in child_ids
+ }
+ if any(after.values()):
+ hard_errors.append(
+ {
+ "type": "host_content_overlap_remains",
+ "host_object_id": host_id,
+ "intersection_after": after,
+ }
+ )
+ records.append(
+ {
+ "host_object_id": host_id,
+ "repair_scope": scope,
+ "status": "resolved",
+ "child_object_ids": child_ids,
+ "intersection_before": intersection_before,
+ "intersection_after": after,
+ "removed_pixels": removed,
+ "output_mask_path": str(mask_path),
+ "output_overlay_path": str(overlay_path),
+ }
+ )
+
+ source_mutations = []
+ for object_id, source_path in source_paths.items():
+ current = _sha256(source_path)
+ if current != source_hashes[object_id]:
+ source_mutations.append(object_id)
+ if source_mutations:
+ hard_errors.append(
+ {
+ "type": "source_mask_mutated",
+ "object_ids": source_mutations,
+ }
+ )
+ audit = {
+ "schema_version": "scenemosaic_open_storage_ownership_resolution",
+ "status": "fail" if hard_errors else "pass",
+ "changed_host_count": len(changed_ids),
+ "changed_host_object_ids": changed_ids,
+ "host_records": records,
+ "source_masks_unchanged": not source_mutations,
+ "formal_outputs_modified": False,
+ "hard_errors": hard_errors,
+ }
+ audit_path = output / "ownership_resolution_audit.json"
+ temporary = audit_path.with_suffix(".json.tmp")
+ temporary.write_text(
+ json.dumps(audit, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, audit_path)
+ return {
+ "objects": copied,
+ "audit": audit,
+ "audit_path": str(audit_path),
+ }
+
+
+__all__ = ["resolve_open_storage_ownership"]
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/mobile_tiered_carrier_completion.py b/vlm_agent/reconstruction/perception/pipeline/refinements/mobile_tiered_carrier_completion.py
new file mode 100644
index 0000000..dc84088
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/mobile_tiered_carrier_completion.py
@@ -0,0 +1,612 @@
+"""Post-Pass-B completion for small mobile tiered carriers."""
+
+from __future__ import annotations
+
+import copy
+import json
+import re
+from collections import Counter
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Callable, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+
+@dataclass(frozen=True)
+class MobileTieredCarrierCompletionConfig:
+ enabled: bool = True
+ isolated_attempt_only: bool = True
+ maximum_hosts: int = 2
+ minimum_named_local_children: int = 2
+ minimum_dominant_category_count: int = 2
+ minimum_dominant_category_fraction: float = 0.66
+ maximum_candidates_per_top_probe: int = 6
+ minimum_top_candidate_score: float = 0.80
+ minimum_candidate_inside_top_roi: float = 0.90
+ maximum_top_candidate_host_overlap: float = 0.15
+ maximum_other_object_candidate_coverage: float = 0.15
+ minimum_candidate_area_pixels: int = 64
+ top_roi_horizontal_padding_fraction: float = 0.10
+ top_roi_upward_padding_fraction: float = 0.40
+ top_roi_downward_fraction: float = 0.18
+ top_child_center_fraction: float = 0.18
+ minimum_same_band_vertical_overlap: float = 0.50
+ maximum_same_band_gap_fraction: float = 0.03
+ maximum_supported_tiers: int = 4
+
+
+def _tokens(value: Any) -> set[str]:
+ return set(re.findall(r"[a-z0-9]+", str(value or "").lower()))
+
+
+def is_mobile_tiered_host(obj: Mapping[str, Any]) -> bool:
+ words: set[str] = set()
+ for key in ("category", "description", "instance_name", "sam3_prompt"):
+ words |= _tokens(obj.get(key))
+ return bool(words & {"cart", "trolley"}) and bool(
+ words & {"rolling", "storage", "serving", "utility"}
+ )
+
+
+def _mask(path_value: Any, shape: tuple[int, int]) -> np.ndarray:
+ path = Path(str(path_value or "")).expanduser().resolve()
+ if not path.is_file():
+ raise FileNotFoundError(path)
+ value = np.asarray(Image.open(path).convert("L")) > 0
+ if value.shape != shape or not value.any():
+ raise ValueError(f"invalid mask: {path}")
+ return value
+
+
+def _bbox(mask: np.ndarray) -> list[int]:
+ ys, xs = np.nonzero(mask)
+ if not len(xs):
+ raise ValueError("empty mask")
+ return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
+
+
+def _claimed_host_box(
+ host_record: Mapping[str, Any],
+ fallback: Sequence[int],
+ shape: tuple[int, int],
+) -> list[int]:
+ """Prefer the original VLM claim for tier geometry, not an expanded mask."""
+
+ height, width = shape
+ raw = (host_record.get("sam3") or {}).get("vlm_bbox_abs_xyxy")
+ try:
+ box = [int(round(float(value))) for value in raw]
+ except (TypeError, ValueError):
+ return [int(value) for value in fallback]
+ if len(box) != 4 or not (
+ 0 <= box[0] < box[2] <= width and 0 <= box[1] < box[3] <= height
+ ):
+ return [int(value) for value in fallback]
+ return box
+
+
+def _inside(mask: np.ndarray, box: Sequence[int]) -> float:
+ x1, y1, x2, y2 = (int(value) for value in box)
+ return int(mask[y1:y2, x1:x2].sum()) / max(1, int(mask.sum()))
+
+
+def _local_ids(obj: Mapping[str, Any]) -> list[str]:
+ return [
+ str(source.get("source_id") or "")
+ for source in (obj.get("proposal_sources") or [])
+ if isinstance(source, Mapping)
+ and str(source.get("pass") or "") == "local_candidate"
+ and str(source.get("source_id") or "")
+ ]
+
+
+def _has_source(obj: Mapping[str, Any], source_id: str) -> bool:
+ return any(
+ isinstance(source, Mapping)
+ and str(source.get("source_id") or "") == source_id
+ for source in (obj.get("proposal_sources") or [])
+ )
+
+
+def _same_band(
+ left: Sequence[int],
+ right: Sequence[int],
+ host_height: int,
+ cfg: MobileTieredCarrierCompletionConfig,
+) -> bool:
+ ly1, ly2 = int(left[1]), int(left[3])
+ ry1, ry2 = int(right[1]), int(right[3])
+ overlap = max(0, min(ly2, ry2) - max(ly1, ry1))
+ minimum_height = max(1, min(ly2 - ly1, ry2 - ry1))
+ if overlap / minimum_height >= cfg.minimum_same_band_vertical_overlap:
+ return True
+ gap = max(0, max(ly1, ry1) - min(ly2, ry2))
+ return gap <= max(2, round(cfg.maximum_same_band_gap_fraction * host_height))
+
+
+def _group_bands(
+ rows: Sequence[Mapping[str, Any]],
+ host_height: int,
+ cfg: MobileTieredCarrierCompletionConfig,
+) -> list[list[dict[str, Any]]]:
+ pending = [dict(row) for row in rows]
+ groups: list[list[dict[str, Any]]] = []
+ while pending:
+ group = [pending.pop(0)]
+ changed = True
+ while changed:
+ changed = False
+ retained = []
+ for row in pending:
+ if any(
+ _same_band(row["bbox"], member["bbox"], host_height, cfg)
+ for member in group
+ ):
+ group.append(row)
+ changed = True
+ else:
+ retained.append(row)
+ pending = retained
+ groups.append(group)
+ groups.sort(key=lambda group: min(int(row["bbox"][1]) for row in group))
+ return groups
+
+
+def _plural_category(category: str) -> str:
+ words = str(category or "object").replace("_", " ").strip().split()
+ if not words:
+ return "objects"
+ last = words[-1]
+ if last.endswith("y") and not last.endswith(("ay", "ey", "iy", "oy", "uy")):
+ words[-1] = last[:-1] + "ies"
+ elif not last.endswith("s"):
+ words[-1] = last + "s"
+ return " ".join(words)
+
+
+def _top_roi(
+ host_box: Sequence[int],
+ shape: tuple[int, int],
+ cfg: MobileTieredCarrierCompletionConfig,
+) -> list[int]:
+ height, width = shape
+ x1, y1, x2, y2 = (int(value) for value in host_box)
+ host_width = max(1, x2 - x1)
+ host_height = max(1, y2 - y1)
+ return [
+ max(0, x1 - round(cfg.top_roi_horizontal_padding_fraction * host_width)),
+ max(0, y1 - round(cfg.top_roi_upward_padding_fraction * host_height)),
+ min(width, x2),
+ min(height, y1 + round(cfg.top_roi_downward_fraction * host_height)),
+ ]
+
+
+def _save(
+ mask: np.ndarray,
+ image: np.ndarray,
+ mask_path: Path,
+) -> tuple[str, str]:
+ mask_path.parent.mkdir(parents=True, exist_ok=True)
+ Image.fromarray(mask.astype(np.uint8) * 255).save(mask_path)
+ overlay = image.copy()
+ overlay[mask] = (
+ 0.55 * overlay[mask]
+ + 0.45 * np.array((36, 170, 235), dtype=np.uint8)
+ ).astype(np.uint8)
+ overlay_path = (
+ mask_path.parent.parent
+ / "overlays"
+ / mask_path.name.replace("_mask.png", "_overlay.png")
+ )
+ overlay_path.parent.mkdir(parents=True, exist_ok=True)
+ Image.fromarray(overlay).save(overlay_path)
+ return str(mask_path), str(overlay_path)
+
+
+def _tier_names(count: int, has_verified_top: bool) -> list[str]:
+ if count <= 0:
+ return []
+ if not has_verified_top:
+ return [f"tier_{index}" for index in range(1, count + 1)]
+ if count == 1:
+ return ["top"]
+ if count == 2:
+ return ["top", "bottom"]
+ if count == 3:
+ return ["top", "middle", "bottom"]
+ return ["top"] + [f"tier_{index}" for index in range(2, count)] + ["bottom"]
+
+
+def run_mobile_tiered_carrier_completion(
+ *,
+ session: Any,
+ image_path: str | Path,
+ objects: Sequence[Mapping[str, Any]],
+ local_candidate_metadata: Sequence[Mapping[str, Any]],
+ sam3_a_registry: Mapping[str, Any],
+ staging_dir: str | Path,
+ work_dir: str | Path,
+ infer_candidates_fn: Callable[..., Mapping[str, Any]],
+ config: MobileTieredCarrierCompletionConfig | None = None,
+) -> dict[str, Any]:
+ cfg = config or MobileTieredCarrierCompletionConfig()
+ if not cfg.isolated_attempt_only:
+ raise ValueError("mobile tiered carrier completion is isolated-attempt only")
+ copied = [copy.deepcopy(dict(obj)) for obj in objects]
+ work = Path(work_dir).expanduser().resolve()
+ if work.exists() and any(work.iterdir()):
+ raise FileExistsError(f"mobile tier work directory is not empty: {work}")
+ work.mkdir(parents=True, exist_ok=True)
+ staging = Path(staging_dir).expanduser().resolve()
+ image_file = Path(image_path).expanduser().resolve()
+ image = np.asarray(Image.open(image_file).convert("RGB"), dtype=np.uint8)
+ shape = image.shape[:2]
+ height, width = shape
+ audit: dict[str, Any] = {
+ "schema_version": "scenemosaic_mobile_tiered_carrier_completion",
+ "config": asdict(cfg),
+ "status": "disabled" if not cfg.enabled else "pending",
+ "target_host_count": 0,
+ "processed_host_count": 0,
+ "inference_call_count": 0,
+ "replaced_atomic_object_ids": [],
+ "created_tier_objects": [],
+ "safe_abstentions": [],
+ "hard_errors": [],
+ "additional_vlm_calls": 0,
+ "formal_runs_modified": False,
+ }
+ if not cfg.enabled:
+ (work / "mobile_tiered_carrier_completion_audit.json").write_text(
+ json.dumps(audit, indent=2)
+ )
+ return {"objects": copied, "audit": audit}
+
+ metadata_by_id = {
+ str(row.get("candidate_id") or ""): dict(row)
+ for row in local_candidate_metadata
+ if isinstance(row, Mapping) and str(row.get("candidate_id") or "")
+ }
+ host_registry = {
+ str(obj.get("object_id") or ""): dict(obj)
+ for obj in (sam3_a_registry.get("objects") or [])
+ if isinstance(obj, Mapping) and is_mobile_tiered_host(obj)
+ }
+ masks: dict[str, np.ndarray] = {}
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ if (obj.get("sam3") or {}).get("status") != "success":
+ continue
+ try:
+ masks[object_id] = _mask((obj.get("sam3") or {}).get("mask_path"), shape)
+ except (OSError, ValueError):
+ audit["hard_errors"].append({
+ "type": "invalid_success_mask",
+ "object_id": object_id,
+ })
+
+ host_children: dict[str, list[dict[str, Any]]] = {}
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ local_ids = _local_ids(obj)
+ if not local_ids or object_id not in masks:
+ continue
+ host_ids = {
+ str((metadata_by_id.get(local_id) or {}).get("host_source_object_id") or "")
+ for local_id in local_ids
+ }
+ host_ids.discard("")
+ if len(host_ids) != 1:
+ continue
+ host_id = next(iter(host_ids))
+ if host_id not in host_registry:
+ continue
+ host_children.setdefault(host_id, []).append({
+ "object": obj,
+ "object_id": object_id,
+ "local_ids": local_ids,
+ "mask": masks[object_id],
+ "bbox": _bbox(masks[object_id]),
+ })
+
+ eligible_hosts = [
+ host_id
+ for host_id, children in host_children.items()
+ if len(children) >= cfg.minimum_named_local_children
+ ]
+ eligible_hosts.sort()
+ audit["target_host_count"] = min(len(eligible_hosts), cfg.maximum_hosts)
+ for host_id in eligible_hosts[cfg.maximum_hosts:]:
+ audit["safe_abstentions"].append({
+ "host_source_object_id": host_id,
+ "reason": "host_budget_exceeded",
+ })
+
+ replacements: dict[str, list[dict[str, Any]]] = {}
+ suppressed: set[str] = set()
+ for host_id in eligible_hosts[:cfg.maximum_hosts]:
+ children = host_children[host_id]
+ categories = [str(row["object"].get("category") or "unknown") for row in children]
+ dominant_category, dominant_count = Counter(categories).most_common(1)[0]
+ if (
+ dominant_count < cfg.minimum_dominant_category_count
+ or dominant_count / len(categories) < cfg.minimum_dominant_category_fraction
+ ):
+ audit["safe_abstentions"].append({
+ "host_source_object_id": host_id,
+ "reason": "no_dominant_named_child_category",
+ "categories": categories,
+ })
+ continue
+ host_record = host_registry[host_id]
+ try:
+ host_mask = _mask((host_record.get("sam3") or {}).get("mask_path"), shape)
+ except (OSError, ValueError):
+ audit["hard_errors"].append({
+ "type": "invalid_mobile_host_mask",
+ "host_source_object_id": host_id,
+ })
+ continue
+ host_mask_box = _bbox(host_mask)
+ host_box = _claimed_host_box(host_record, host_mask_box, shape)
+ host_height = max(1, host_box[3] - host_box[1])
+ host_description = str(
+ host_record.get("description")
+ or host_record.get("sam3_prompt")
+ or host_record.get("category")
+ or "mobile storage cart"
+ ).replace("_", " ").strip()
+ top_prompt = (
+ f"all {_plural_category(dominant_category)} on the top tier of the "
+ f"{host_description}"
+ )
+ top_roi = _top_roi(host_box, shape, cfg)
+ audit["inference_call_count"] += 1
+ try:
+ response = infer_candidates_fn(
+ session,
+ image_file,
+ top_prompt,
+ work / f"host_{host_id.lower()}" / "top_probe",
+ score_threshold=0.20,
+ positive_bbox_abs_xyxy=top_roi,
+ maximum_candidates=cfg.maximum_candidates_per_top_probe,
+ )
+ raw_candidates = (response.get("registry") or {}).get("candidates") or []
+ except Exception as exc:
+ audit["safe_abstentions"].append({
+ "host_source_object_id": host_id,
+ "reason": f"top_probe_exception:{type(exc).__name__}",
+ })
+ raw_candidates = []
+
+ child_ids = {row["object_id"] for row in children}
+ final_host_ids = {
+ str(obj.get("object_id") or "")
+ for obj in copied
+ if _has_source(obj, host_id)
+ }
+ viable = []
+ for raw in raw_candidates:
+ try:
+ candidate = _mask(raw.get("mask_path"), shape)
+ except (OSError, ValueError):
+ continue
+ score = float(raw.get("score") or 0.0)
+ area = int(candidate.sum())
+ inside = _inside(candidate, top_roi)
+ host_overlap = int(np.logical_and(candidate, host_mask).sum()) / max(1, area)
+ conflicts = []
+ for other_id, other_mask in masks.items():
+ if other_id in child_ids or other_id in final_host_ids:
+ continue
+ coverage = int(np.logical_and(candidate, other_mask).sum()) / max(1, area)
+ if coverage > cfg.maximum_other_object_candidate_coverage:
+ conflicts.append({
+ "object_id": other_id,
+ "candidate_coverage": round(coverage, 6),
+ })
+ if (
+ score < cfg.minimum_top_candidate_score
+ or area < cfg.minimum_candidate_area_pixels
+ or inside < cfg.minimum_candidate_inside_top_roi
+ or host_overlap > cfg.maximum_top_candidate_host_overlap
+ or conflicts
+ ):
+ continue
+ viable.append({
+ "candidate_id": str(raw.get("candidate_id") or ""),
+ "mask": candidate,
+ "score": score,
+ "inside_top_roi": inside,
+ "host_overlap": host_overlap,
+ "area_pixels": area,
+ })
+ viable.sort(
+ key=lambda row: (float(row["score"]), -float(row["host_overlap"])),
+ reverse=True,
+ )
+ top_candidate = viable[0] if viable else None
+ top_limit = host_box[1] + round(cfg.top_child_center_fraction * host_height)
+ top_children = [
+ row
+ for row in children
+ if (int(row["bbox"][1]) + int(row["bbox"][3])) / 2 <= top_limit
+ ]
+ lower_children = [row for row in children if row not in top_children]
+ lower_groups = _group_bands(lower_children, host_height, cfg)
+ tier_masks: list[np.ndarray] = []
+ tier_members: list[list[dict[str, Any]]] = []
+ if top_candidate is not None:
+ tier_masks.append(np.asarray(top_candidate["mask"], dtype=bool))
+ tier_members.append(top_children)
+ elif top_children:
+ tier_masks.append(np.logical_or.reduce([row["mask"] for row in top_children]))
+ tier_members.append(top_children)
+ audit["safe_abstentions"].append({
+ "host_source_object_id": host_id,
+ "reason": "top_semantic_probe_failed_preserved_named_top_children",
+ })
+ for group in lower_groups:
+ tier_masks.append(np.logical_or.reduce([row["mask"] for row in group]))
+ tier_members.append(group)
+ if not tier_masks:
+ audit["safe_abstentions"].append({
+ "host_source_object_id": host_id,
+ "reason": "no_tier_masks_after_grouping",
+ })
+ continue
+ if len(tier_masks) > cfg.maximum_supported_tiers:
+ audit["hard_errors"].append({
+ "type": "mobile_tier_count_exceeds_bound",
+ "host_source_object_id": host_id,
+ "actual": len(tier_masks),
+ "maximum": cfg.maximum_supported_tiers,
+ })
+ continue
+
+ tier_names = _tier_names(len(tier_masks), top_candidate is not None)
+ created = []
+ all_child_sources: list[dict[str, Any]] = []
+ for row in children:
+ all_child_sources.extend(copy.deepcopy(row["object"].get("proposal_sources") or []))
+ used_source_keys: set[tuple[str, str]] = set()
+ for tier_index, (tier_name, tier_mask, members) in enumerate(
+ zip(tier_names, tier_masks, tier_members), 1
+ ):
+ member_sources: list[dict[str, Any]] = []
+ for row in members:
+ for source in copy.deepcopy(row["object"].get("proposal_sources") or []):
+ key = (str(source.get("pass") or ""), str(source.get("source_id") or ""))
+ if key not in used_source_keys:
+ member_sources.append(source)
+ used_source_keys.add(key)
+ source_id = f"{host_id}:mobile_tier:{tier_name}"
+ member_sources.append({
+ "pass": "local_tier_completion",
+ "source_id": source_id,
+ "action": "replace_atomic_children_with_tier",
+ })
+ category = f"rolling_cart_{tier_name}_tier_contents"
+ object_id = min(
+ (str(row["object_id"]) for row in members),
+ default=f"proposal_mobile_{host_id.lower()}_{tier_index:02d}",
+ )
+ mask_path, overlay_path = _save(
+ tier_mask,
+ image,
+ staging / "masks" / f"{object_id}_{category}_mask.png",
+ )
+ box = _bbox(tier_mask)
+ base = copy.deepcopy(members[0]["object"]) if members else {}
+ base.update({
+ "object_id": object_id,
+ "category": category,
+ "sam3_prompt": top_prompt if tier_name == "top" else dominant_category.replace("_", " "),
+ "instance_name": f"{host_description} {tier_name} tier contents",
+ "object_type": "small_object_group",
+ "bbox_2d_norm_xyxy": [
+ box[0] / width,
+ box[1] / height,
+ box[2] / width,
+ box[3] / height,
+ ],
+ "anchor_2d_norm_xy": [
+ (box[0] + box[2]) / (2 * width),
+ (box[1] + box[3]) / (2 * height),
+ ],
+ "description": (
+ f"all visible {dominant_category.replace('_', ' ')} contents "
+ f"on the {tier_name} tier of the {host_description}, grouped "
+ "as one reconstruction object"
+ ),
+ "fallback_prompts": [],
+ "proposal_sources": member_sources,
+ "sam3": {
+ "status": "success",
+ "score": (
+ float(top_candidate["score"])
+ if tier_name == "top" and top_candidate is not None
+ else max(
+ float((row["object"].get("sam3") or {}).get("score") or 0.0)
+ for row in members
+ )
+ ),
+ "area_pixels": int(tier_mask.sum()),
+ "mask_bbox_abs_xyxy": box,
+ "vlm_bbox_abs_xyxy": box,
+ "vlm_bbox_iou": 1.0,
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ "selection_method": "mobile_tiered_carrier_semantic_top_and_named_band_union",
+ "mobile_tiered_carrier_completion": {
+ "host_source_object_id": host_id,
+ "tier_name": tier_name,
+ "dominant_child_category": dominant_category,
+ "replaced_object_ids": [row["object_id"] for row in members],
+ "top_semantic_candidate_id": (
+ top_candidate["candidate_id"]
+ if tier_name == "top" and top_candidate is not None
+ else None
+ ),
+ },
+ },
+ })
+ created.append(base)
+ audit["created_tier_objects"].append({
+ "host_source_object_id": host_id,
+ "intermediate_object_id": object_id,
+ "tier_name": tier_name,
+ "category": category,
+ "area_pixels": int(tier_mask.sum()),
+ "bbox_abs_xyxy": box,
+ "replaced_object_ids": [row["object_id"] for row in members],
+ "top_semantic_candidate_id": (
+ top_candidate["candidate_id"]
+ if tier_name == "top" and top_candidate is not None
+ else None
+ ),
+ })
+ replacements[host_id] = created
+ suppressed |= child_ids
+ audit["replaced_atomic_object_ids"].extend(sorted(child_ids))
+ audit["processed_host_count"] += 1
+
+ if replacements:
+ first_child_index: dict[str, int] = {}
+ for index, obj in enumerate(copied):
+ object_id = str(obj.get("object_id") or "")
+ for host_id, children in host_children.items():
+ if any(row["object_id"] == object_id for row in children):
+ first_child_index.setdefault(host_id, index)
+ output_objects: list[dict[str, Any]] = []
+ for index, obj in enumerate(copied):
+ for host_id, insertion_index in first_child_index.items():
+ if index == insertion_index and host_id in replacements:
+ output_objects.extend(replacements[host_id])
+ if str(obj.get("object_id") or "") not in suppressed:
+ output_objects.append(obj)
+ copied = output_objects
+
+ audit["replaced_atomic_object_ids"] = sorted(set(audit["replaced_atomic_object_ids"]))
+ audit["status"] = (
+ "blocked"
+ if audit["hard_errors"]
+ else "pass"
+ if audit["processed_host_count"]
+ else "pass_with_safe_abstentions"
+ if audit["safe_abstentions"]
+ else "noop"
+ )
+ audit_path = work / "mobile_tiered_carrier_completion_audit.json"
+ audit_path.write_text(json.dumps(audit, indent=2))
+ return {"objects": copied, "audit": audit, "audit_path": str(audit_path)}
+
+
+__all__ = [
+ "MobileTieredCarrierCompletionConfig",
+ "is_mobile_tiered_host",
+ "run_mobile_tiered_carrier_completion",
+]
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/named_local_instance_semantic_completion.py b/vlm_agent/reconstruction/perception/pipeline/refinements/named_local_instance_semantic_completion.py
new file mode 100644
index 0000000..cd6e4c5
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/named_local_instance_semantic_completion.py
@@ -0,0 +1,297 @@
+"""Bounded safety check after Pass-B names a local carrier candidate."""
+
+from __future__ import annotations
+
+import copy
+import json
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Callable, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+from scipy import ndimage
+
+
+@dataclass(frozen=True)
+class NamedLocalSemanticCompletionConfig:
+ enabled: bool = True
+ isolated_attempt_only: bool = True
+ maximum_targets: int = 24
+ maximum_candidates_per_target: int = 4
+ minimum_score: float = 0.20
+ minimum_existing_coverage: float = 0.65
+ minimum_mask_inside_host_bbox: float = 0.90
+ maximum_candidate_area_ratio: float = 3.0
+ maximum_other_object_candidate_coverage: float = 0.15
+ minimum_rank_margin: float = 0.10
+ roi_padding_fraction: float = 0.15
+ outer_boundary_band_pixels: int = 1
+ verify_nonboundary_candidates: bool = False
+
+
+def _mask(path_value: Any, shape: tuple[int, int]) -> np.ndarray:
+ path = Path(str(path_value or '')).expanduser().resolve()
+ if not path.is_file():
+ raise FileNotFoundError(path)
+ value = np.asarray(Image.open(path).convert('L')) > 0
+ if value.shape != shape or not value.any():
+ raise ValueError(f'invalid mask: {path}')
+ return value
+
+
+def _bbox(mask: np.ndarray) -> list[int]:
+ ys, xs = np.nonzero(mask)
+ if not len(xs):
+ raise ValueError('empty mask')
+ return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
+
+
+def _expand_and_clip(box: Sequence[int], host: Sequence[int], fraction: float) -> list[int]:
+ x1, y1, x2, y2 = (int(value) for value in box)
+ hx1, hy1, hx2, hy2 = (int(value) for value in host)
+ px = max(2, round((x2 - x1) * fraction))
+ py = max(2, round((y2 - y1) * fraction))
+ return [max(hx1, x1 - px), max(hy1, y1 - py), min(hx2, x2 + px), min(hy2, y2 + py)]
+
+
+def _inside_bbox(mask: np.ndarray, box: Sequence[int]) -> float:
+ x1, y1, x2, y2 = (int(value) for value in box)
+ return int(mask[y1:y2, x1:x2].sum()) / max(1, int(mask.sum()))
+
+
+def _outer_edges(box: Sequence[int], host: Sequence[int], band: int) -> list[str]:
+ x1, y1, x2, y2 = (int(value) for value in box)
+ hx1, hy1, hx2, hy2 = (int(value) for value in host)
+ result = []
+ if x1 <= hx1 + band:
+ result.append('left')
+ if y1 <= hy1 + band:
+ result.append('top')
+ if x2 >= hx2 - band:
+ result.append('right')
+ if y2 >= hy2 - band:
+ result.append('bottom')
+ return result
+
+
+def _one_component(mask: np.ndarray) -> bool:
+ _, count = ndimage.label(mask, structure=np.ones((3, 3), bool))
+ return int(count) == 1
+
+
+def _local_candidate_ids(obj: Mapping[str, Any]) -> list[str]:
+ return [
+ str(item.get('source_id') or '')
+ for item in (obj.get('proposal_sources') or [])
+ if isinstance(item, Mapping) and str(item.get('pass') or '') == 'local_candidate'
+ ]
+
+
+def _save(mask: np.ndarray, image: np.ndarray, path: Path) -> tuple[str, str]:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ Image.fromarray(mask.astype(np.uint8) * 255).save(path)
+ overlay = image.copy()
+ overlay[mask] = (0.55 * overlay[mask] + 0.45 * np.array((58, 145, 220), dtype=np.uint8)).astype(np.uint8)
+ overlay_path = path.parent.parent / 'overlays' / path.name.replace('_mask.png', '_overlay.png')
+ overlay_path.parent.mkdir(parents=True, exist_ok=True)
+ Image.fromarray(overlay).save(overlay_path)
+ return str(path), str(overlay_path)
+
+
+def run_named_local_instance_semantic_completion(
+ *,
+ session: Any,
+ image_path: str | Path,
+ objects: Sequence[Mapping[str, Any]],
+ local_candidate_metadata: Sequence[Mapping[str, Any]],
+ sam3_a_registry: Mapping[str, Any],
+ staging_dir: str | Path,
+ work_dir: str | Path,
+ infer_candidates_fn: Callable[..., Mapping[str, Any]],
+ config: NamedLocalSemanticCompletionConfig | None = None,
+) -> dict[str, Any]:
+ cfg = config or NamedLocalSemanticCompletionConfig()
+ if not cfg.isolated_attempt_only:
+ raise ValueError('named local semantic completion is isolated-attempt only')
+ out = Path(work_dir).expanduser().resolve()
+ if out.exists() and any(out.iterdir()):
+ raise FileExistsError(f'named local completion work directory is not empty: {out}')
+ out.mkdir(parents=True)
+ image_file = Path(image_path).expanduser().resolve()
+ image = np.asarray(Image.open(image_file).convert('RGB'))
+ shape = image.shape[:2]
+ copied = [copy.deepcopy(dict(item)) for item in objects]
+ metadata_by_id = {
+ str(item.get('candidate_id') or ''): dict(item)
+ for item in local_candidate_metadata
+ if isinstance(item, Mapping)
+ }
+ host_by_id = {
+ str(item.get('object_id') or ''): dict(item)
+ for item in (sam3_a_registry.get('objects') or [])
+ if isinstance(item, Mapping)
+ }
+ masks: dict[str, np.ndarray] = {}
+ for obj in copied:
+ if (obj.get('sam3') or {}).get('status') != 'success':
+ continue
+ try:
+ masks[str(obj.get('object_id') or '')] = _mask((obj.get('sam3') or {}).get('mask_path'), shape)
+ except (OSError, ValueError):
+ continue
+ targets = []
+ for obj in copied:
+ local_ids = _local_candidate_ids(obj)
+ if len(local_ids) != 1:
+ continue
+ metadata = metadata_by_id.get(local_ids[0])
+ host_id = str((metadata or {}).get('host_source_object_id') or '')
+ if not metadata or not host_id or host_id not in host_by_id:
+ continue
+ if str(obj.get('object_id') or '') not in masks:
+ continue
+ targets.append((str(obj.get('object_id') or ''), local_ids[0], host_id))
+ targets.sort()
+ audit: dict[str, Any] = {
+ 'schema_version': 'scenemosaic_named_local_instance_semantic_completion',
+ 'config': asdict(cfg),
+ 'target_count': min(len(targets), cfg.maximum_targets),
+ 'inference_call_count': 0,
+ 'completed': [],
+ 'verified': [],
+ 'removed_boundary_artifacts': [],
+ 'unresolved': [],
+ 'skipped': [],
+ 'hard_errors': [],
+ 'additional_vlm_calls': 0,
+ 'formal_outputs_modified': False,
+ }
+ retained = []
+ target_ids = {item[0] for item in targets[:cfg.maximum_targets]}
+ for object_id, local_id, host_id in targets[cfg.maximum_targets:]:
+ audit['unresolved'].append({'object_id': object_id, 'candidate_id': local_id, 'host_source_object_id': host_id, 'reason': 'target_budget_exceeded'})
+ for obj in copied:
+ object_id = str(obj.get('object_id') or '')
+ if object_id not in target_ids:
+ retained.append(obj)
+ continue
+ local_id = _local_candidate_ids(obj)[0]
+ metadata = metadata_by_id[local_id]
+ host_id = str(metadata['host_source_object_id'])
+ base = masks[object_id]
+ host_mask = _mask((host_by_id[host_id].get('sam3') or {}).get('mask_path'), shape)
+ host_bbox = _bbox(host_mask)
+ base_bbox = _bbox(base)
+ prompt = str(obj.get('sam3_prompt') or '').strip()
+ edges = _outer_edges(base_bbox, host_bbox, cfg.outer_boundary_band_pixels)
+ if not prompt:
+ audit['unresolved'].append({'object_id': object_id, 'candidate_id': local_id, 'host_source_object_id': host_id, 'reason': 'missing_semantic_prompt'})
+ retained.append(obj)
+ continue
+ # A named, interior local candidate is already semantically bound by
+ # Pass B. Do not make publication depend on a second SAM3 prompt:
+ # small/occluded contents often cannot be recovered by that redundant
+ # query even when the original local mask is correct. Boundary-touching
+ # candidates remain subject to the stricter probe below.
+ if not edges and not cfg.verify_nonboundary_candidates:
+ audit['verified'].append({
+ 'object_id': object_id,
+ 'candidate_id': local_id,
+ 'host_source_object_id': host_id,
+ 'verification': 'pass_b_named_interior_local_candidate',
+ 'semantic_probe': 'not_required_for_interior_candidate',
+ })
+ retained.append(obj)
+ continue
+ positive_bbox = _expand_and_clip(base_bbox, host_bbox, cfg.roi_padding_fraction)
+ audit['inference_call_count'] += 1
+ target_work = out / object_id
+ try:
+ response = infer_candidates_fn(
+ session, image_file, prompt, target_work,
+ score_threshold=cfg.minimum_score,
+ positive_bbox_abs_xyxy=positive_bbox,
+ maximum_candidates=cfg.maximum_candidates_per_target,
+ )
+ raw_candidates = (response.get('registry') or {}).get('candidates') or []
+ except Exception as exc:
+ audit['unresolved'].append({'object_id': object_id, 'candidate_id': local_id, 'host_source_object_id': host_id, 'reason': f'inference_exception:{type(exc).__name__}'})
+ retained.append(obj)
+ continue
+ viable = []
+ for raw in raw_candidates:
+ try:
+ candidate = _mask(raw.get('mask_path'), shape)
+ except (OSError, ValueError):
+ continue
+ score = float(raw.get('score') or 0.0)
+ existing_coverage = int(np.logical_and(candidate, base).sum()) / max(1, int(base.sum()))
+ inside = _inside_bbox(candidate, host_bbox)
+ area_ratio = int(candidate.sum()) / max(1, int(base.sum()))
+ if score < cfg.minimum_score or existing_coverage < cfg.minimum_existing_coverage or inside < cfg.minimum_mask_inside_host_bbox or area_ratio > cfg.maximum_candidate_area_ratio:
+ continue
+ conflicts = []
+ for other_id, other_mask in masks.items():
+ if other_id == object_id:
+ continue
+ # The open carrier itself is expected to overlap a contained
+ # object before ownership resolution; do not treat it as a peer.
+ other_sources = next((item.get('proposal_sources') or [] for item in copied if str(item.get('object_id') or '') == other_id), [])
+ if any(isinstance(row, Mapping) and str(row.get('source_id') or '') == host_id for row in other_sources):
+ continue
+ overlap = int(np.logical_and(candidate, other_mask).sum()) / max(1, int(candidate.sum()))
+ if overlap > cfg.maximum_other_object_candidate_coverage:
+ conflicts.append({'object_id': other_id, 'candidate_coverage': round(overlap, 6)})
+ if conflicts:
+ continue
+ union = base | candidate
+ if not _one_component(union):
+ continue
+ rank = 2.0 * existing_coverage + min(1.0, area_ratio) + 0.25 * score
+ viable.append({'candidate_id': str(raw.get('candidate_id') or ''), 'mask': candidate, 'score': score, 'existing_coverage': existing_coverage, 'inside': inside, 'area_ratio': area_ratio, 'union': union, 'rank': rank})
+ viable.sort(key=lambda row: row['rank'], reverse=True)
+ if not viable:
+ if edges:
+ masks.pop(object_id, None)
+ audit['removed_boundary_artifacts'].append({'object_id': object_id, 'candidate_id': local_id, 'host_source_object_id': host_id, 'outer_anchor_edges': edges, 'reason': 'no_exact_semantic_candidate_inside_boundary_artifact'})
+ continue
+ audit['unresolved'].append({'object_id': object_id, 'candidate_id': local_id, 'host_source_object_id': host_id, 'reason': 'no_safe_exact_semantic_candidate'})
+ retained.append(obj)
+ continue
+ if len(viable) > 1 and viable[0]['rank'] - viable[1]['rank'] < cfg.minimum_rank_margin:
+ audit['unresolved'].append({'object_id': object_id, 'candidate_id': local_id, 'host_source_object_id': host_id, 'reason': 'ambiguous_semantic_candidate_rank'})
+ retained.append(obj)
+ continue
+ chosen = viable[0]
+ union = chosen['union']
+ if int(union.sum()) > int(base.sum()) * 1.03:
+ output = Path(staging_dir).expanduser().resolve() / 'masks' / f"{object_id}_{str(obj.get('category') or 'object')}_mask.png"
+ mask_path, overlay_path = _save(union, image, output)
+ obj['sam3'] = copy.deepcopy(dict(obj.get('sam3') or {}))
+ obj['sam3'].update({
+ 'mask_path': mask_path,
+ 'overlay_path': overlay_path,
+ 'area_pixels': int(union.sum()),
+ 'mask_bbox_abs_xyxy': _bbox(union),
+ 'selection_method': 'named_local_semantic_completion',
+ 'named_local_semantic_completion': {
+ 'candidate_id': chosen['candidate_id'], 'candidate_score': round(chosen['score'], 6),
+ 'host_source_object_id': host_id, 'local_candidate_id': local_id,
+ 'previous_area_pixels': int(base.sum()), 'completed_area_pixels': int(union.sum()),
+ 'existing_coverage': round(chosen['existing_coverage'], 6),
+ },
+ })
+ masks[object_id] = union
+ audit['completed'].append({'object_id': object_id, **obj['sam3']['named_local_semantic_completion']})
+ else:
+ audit['verified'].append({'object_id': object_id, 'candidate_id': chosen['candidate_id'], 'host_source_object_id': host_id, 'existing_coverage': round(chosen['existing_coverage'], 6)})
+ retained.append(obj)
+ if audit['unresolved']:
+ audit['hard_errors'] = [{'type': 'named_local_instance_semantic_completion_unresolved', **item} for item in audit['unresolved']]
+ audit['status'] = 'pass' if not audit['hard_errors'] else 'blocked'
+ (out / 'named_local_instance_semantic_completion_audit.json').write_text(json.dumps(audit, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
+ return {'objects': retained, 'audit': audit}
+
+
+__all__ = ['NamedLocalSemanticCompletionConfig', 'run_named_local_instance_semantic_completion']
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/nested_part_mask_suppression.py b/vlm_agent/reconstruction/perception/pipeline/refinements/nested_part_mask_suppression.py
new file mode 100644
index 0000000..cb8baf4
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/nested_part_mask_suppression.py
@@ -0,0 +1,370 @@
+#!/usr/bin/env python3
+"""Fail-closed suppression of a proved part masquerading as an inventory item."""
+
+from __future__ import annotations
+
+import copy
+import json
+import os
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+from . import repair_scope_router as SCOPE
+
+
+@dataclass(frozen=True)
+class NestedPartMaskSuppressionConfig:
+ enabled: bool = True
+ isolated_attempt_only: bool = True
+ minimum_child_area_pixels: int = 32
+ minimum_child_containment: float = 0.95
+ maximum_child_to_parent_area_ratio: float = 0.85
+ minimum_explicit_assembly_containment: float = 0.98
+
+
+ASSEMBLY_LINK_TOKENS = {"with", "including", "featuring"}
+ATTACHED_BOTANICAL_TOKENS = {
+ "blossom", "blossoms", "branch", "branches", "flower", "flowers",
+ "foliage", "leaf", "leaves", "petal", "petals", "stem", "stems",
+}
+INDEPENDENT_CONTENT_HOST_TOKENS = {
+ "basket", "bowl", "container", "cup", "dish", "glass", "mug",
+ "plate", "platter", "receptacle", "tray",
+}
+
+
+def _atomic_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _tokens(value: Any) -> set[str]:
+ return set(re.findall(r"[a-z0-9]+", str(value or "").lower()))
+
+
+def _is_complete_potted_plant(obj: Mapping[str, Any]) -> bool:
+ return {"potted", "plant"} <= _tokens(obj.get("category"))
+
+
+def _is_vase_like_component(obj: Mapping[str, Any]) -> bool:
+ return bool(_tokens(obj.get("category")) & {"vase", "planter"})
+
+
+def _semantic_tokens(obj: Mapping[str, Any]) -> set[str]:
+ result: set[str] = set()
+ for field in ("category", "sam3_prompt", "instance_name", "description"):
+ result.update(_tokens(obj.get(field)))
+ return result
+
+
+def _explicit_attached_botanical_tokens(obj: Mapping[str, Any]) -> set[str]:
+ result: set[str] = set()
+ for field in ("sam3_prompt", "description"):
+ words = re.findall(r"[a-z0-9]+", str(obj.get(field) or "").lower())
+ for index, word in enumerate(words):
+ if word in ASSEMBLY_LINK_TOKENS:
+ result.update(words[index + 1 :])
+ return result & ATTACHED_BOTANICAL_TOKENS
+
+
+def _can_own_attached_botanical_component(
+ parent: Mapping[str, Any],
+ child: Mapping[str, Any],
+) -> bool:
+ parent_components = _explicit_attached_botanical_tokens(parent)
+ child_components = _semantic_tokens(child) & ATTACHED_BOTANICAL_TOKENS
+ if not parent_components or not parent_components & child_components:
+ return False
+ if str(parent.get("object_type") or "").strip().lower() == "furniture":
+ return False
+ if _semantic_tokens(parent) & INDEPENDENT_CONTENT_HOST_TOKENS:
+ return False
+ role = SCOPE.classify_object_role(parent).get("object_role")
+ return role not in {
+ SCOPE.OPEN_STORAGE_ROLE,
+ SCOPE.SUPPORT_SURFACE_ROLE,
+ SCOPE.AMBIGUOUS_ROLE,
+ }
+
+
+def _contained_metrics(
+ child_mask: np.ndarray,
+ parent_mask: np.ndarray,
+) -> tuple[int, int, float, float]:
+ child_area = int(child_mask.sum())
+ parent_area = int(parent_mask.sum())
+ intersection = int(np.logical_and(child_mask, parent_mask).sum())
+ return (
+ child_area,
+ parent_area,
+ intersection / max(child_area, 1),
+ child_area / max(parent_area, 1),
+ )
+
+
+def _load_mask(path_value: Any, expected_shape: tuple[int, int]) -> np.ndarray:
+ path = Path(str(path_value or "")).expanduser().resolve()
+ if not path.is_file():
+ raise FileNotFoundError(path)
+ mask = np.array(Image.open(path).convert("L")) > 0
+ if mask.shape != expected_shape or not mask.any():
+ raise ValueError("invalid_mask_shape_or_empty")
+ return mask
+
+
+def _remap_exact_proposal_ids(value: Any, mapping: Mapping[str, str]) -> Any:
+ if isinstance(value, str):
+ return mapping.get(value, value)
+ if isinstance(value, list):
+ return [_remap_exact_proposal_ids(item, mapping) for item in value]
+ if isinstance(value, Mapping):
+ return {
+ key: _remap_exact_proposal_ids(item, mapping)
+ for key, item in value.items()
+ }
+ return value
+
+
+def run_nested_part_mask_suppression(
+ *,
+ objects: Sequence[Mapping[str, Any]],
+ image_shape: tuple[int, int],
+ output_dir: str | Path,
+ config: NestedPartMaskSuppressionConfig | None = None,
+) -> dict[str, Any]:
+ """Remove only a uniquely proved component already owned by an assembly."""
+
+ cfg = config or NestedPartMaskSuppressionConfig()
+ if not cfg.isolated_attempt_only:
+ raise ValueError("nested part mask suppression is isolated-attempt only")
+ work = Path(output_dir).expanduser().resolve()
+ work.mkdir(parents=True, exist_ok=True)
+ copied = [copy.deepcopy(dict(obj)) for obj in objects if isinstance(obj, Mapping)]
+ audit: dict[str, Any] = {
+ "schema_version": "scenemosaic_contained_component_suppression",
+ "enabled": cfg.enabled,
+ "isolated_attempt_only": cfg.isolated_attempt_only,
+ "status": "disabled" if not cfg.enabled else "noop",
+ "input_object_count": len(copied),
+ "suppressed": [],
+ "rejected": [],
+ "hard_errors": [],
+ "thresholds": {
+ "minimum_child_area_pixels": cfg.minimum_child_area_pixels,
+ "minimum_child_containment": cfg.minimum_child_containment,
+ "maximum_child_to_parent_area_ratio": cfg.maximum_child_to_parent_area_ratio,
+ "minimum_explicit_assembly_containment": (
+ cfg.minimum_explicit_assembly_containment
+ ),
+ },
+ }
+ if not cfg.enabled:
+ _atomic_json(work / "contained_component_suppression_audit.json", audit)
+ return {"objects": copied, "audit": audit}
+
+ masks: dict[str, np.ndarray] = {}
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ sam3 = obj.get("sam3") or {}
+ if str(sam3.get("status") or "") != "success":
+ continue
+ try:
+ masks[object_id] = _load_mask(sam3.get("mask_path"), image_shape)
+ except (OSError, ValueError) as exc:
+ audit["hard_errors"].append(
+ {
+ "type": "invalid_success_mask_for_contained_component_suppression",
+ "object_id": object_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+
+ parent_objects = [
+ obj for obj in copied
+ if _is_complete_potted_plant(obj)
+ and str(obj.get("object_id") or "") in masks
+ ]
+ removal_ids: set[str] = set()
+ for child in copied:
+ child_id = str(child.get("object_id") or "")
+ if (
+ not _is_vase_like_component(child)
+ or child_id not in masks
+ or _is_complete_potted_plant(child)
+ ):
+ continue
+ child_mask = masks[child_id]
+ child_area = int(child_mask.sum())
+ if child_area < cfg.minimum_child_area_pixels:
+ audit["rejected"].append(
+ {
+ "object_id": child_id,
+ "reason": "child_area_below_minimum",
+ "child_area_pixels": child_area,
+ }
+ )
+ continue
+ matches: list[dict[str, Any]] = []
+ for parent in parent_objects:
+ parent_id = str(parent.get("object_id") or "")
+ if parent_id == child_id:
+ continue
+ parent_mask = masks[parent_id]
+ _, parent_area, child_containment, child_to_parent_area_ratio = (
+ _contained_metrics(child_mask, parent_mask)
+ )
+ if (
+ child_containment >= cfg.minimum_child_containment
+ and child_to_parent_area_ratio <= cfg.maximum_child_to_parent_area_ratio
+ ):
+ matches.append(
+ {
+ "parent_object_id": parent_id,
+ "parent_category": str(parent.get("category") or ""),
+ "parent_area_pixels": parent_area,
+ "child_containment": child_containment,
+ "child_to_parent_area_ratio": child_to_parent_area_ratio,
+ }
+ )
+ if len(matches) != 1:
+ audit["rejected"].append(
+ {
+ "object_id": child_id,
+ "reason": (
+ "no_unique_complete_potted_plant_container"
+ if not matches
+ else "ambiguous_complete_potted_plant_container"
+ ),
+ "matching_parent_count": len(matches),
+ }
+ )
+ continue
+ match = matches[0]
+ removal_ids.add(child_id)
+ audit["suppressed"].append(
+ {
+ "removed_object_id": child_id,
+ "removed_category": str(child.get("category") or ""),
+ "kept_object_id": match["parent_object_id"],
+ "kept_category": match["parent_category"],
+ "policy": "complete_potted_plant_contains_vase_like_submask",
+ "child_area_pixels": child_area,
+ "parent_area_pixels": match["parent_area_pixels"],
+ "child_containment": round(float(match["child_containment"]), 6),
+ "child_to_parent_area_ratio": round(
+ float(match["child_to_parent_area_ratio"]), 6
+ ),
+ }
+ )
+
+ for child in copied:
+ child_id = str(child.get("object_id") or "")
+ if (
+ child_id in removal_ids
+ or child_id not in masks
+ or not (_semantic_tokens(child) & ATTACHED_BOTANICAL_TOKENS)
+ ):
+ continue
+ semantic_parents = [
+ parent
+ for parent in copied
+ if str(parent.get("object_id") or "") != child_id
+ and str(parent.get("object_id") or "") in masks
+ and _can_own_attached_botanical_component(parent, child)
+ ]
+ if not semantic_parents:
+ continue
+ matches = []
+ for parent in semantic_parents:
+ parent_id = str(parent.get("object_id") or "")
+ (
+ child_area,
+ parent_area,
+ containment,
+ area_ratio,
+ ) = _contained_metrics(masks[child_id], masks[parent_id])
+ if (
+ child_area >= cfg.minimum_child_area_pixels
+ and containment >= cfg.minimum_explicit_assembly_containment
+ and area_ratio <= cfg.maximum_child_to_parent_area_ratio
+ ):
+ matches.append(
+ {
+ "parent_object_id": parent_id,
+ "parent_category": str(parent.get("category") or ""),
+ "parent_area_pixels": parent_area,
+ "child_area_pixels": child_area,
+ "child_containment": containment,
+ "child_to_parent_area_ratio": area_ratio,
+ }
+ )
+ if len(matches) != 1:
+ audit["rejected"].append(
+ {
+ "object_id": child_id,
+ "reason": (
+ "no_unique_explicit_assembly_owner"
+ if not matches
+ else "ambiguous_explicit_assembly_owner"
+ ),
+ "semantic_parent_count": len(semantic_parents),
+ "matching_parent_count": len(matches),
+ }
+ )
+ continue
+ match = matches[0]
+ removal_ids.add(child_id)
+ audit["suppressed"].append(
+ {
+ "removed_object_id": child_id,
+ "removed_category": str(child.get("category") or ""),
+ "kept_object_id": match["parent_object_id"],
+ "kept_category": match["parent_category"],
+ "policy": (
+ "explicit_complete_assembly_contains_attached_component"
+ ),
+ "child_area_pixels": match["child_area_pixels"],
+ "parent_area_pixels": match["parent_area_pixels"],
+ "child_containment": round(
+ float(match["child_containment"]), 6
+ ),
+ "child_to_parent_area_ratio": round(
+ float(match["child_to_parent_area_ratio"]), 6
+ ),
+ }
+ )
+
+ result = [
+ obj for obj in copied if str(obj.get("object_id") or "") not in removal_ids
+ ]
+ object_id_map = {
+ str(obj.get("object_id") or ""): f"proposal_{index:03d}"
+ for index, obj in enumerate(result, start=1)
+ }
+ result = [_remap_exact_proposal_ids(obj, object_id_map) for obj in result]
+ audit["object_id_map"] = [
+ {"old_object_id": old_id, "new_object_id": new_id}
+ for old_id, new_id in object_id_map.items()
+ if old_id != new_id
+ ]
+ for record in audit["suppressed"]:
+ record["kept_final_object_id"] = object_id_map.get(
+ str(record["kept_object_id"]), str(record["kept_object_id"])
+ )
+ audit["output_object_count"] = len(result)
+ audit["suppressed_count"] = len(audit["suppressed"])
+ audit["status"] = (
+ "fail" if audit["hard_errors"] else ("pass" if removal_ids else "noop")
+ )
+ _atomic_json(work / "contained_component_suppression_audit.json", audit)
+ return {"objects": result, "audit": audit}
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/open_storage_content_prompt.py b/vlm_agent/reconstruction/perception/pipeline/refinements/open_storage_content_prompt.py
new file mode 100644
index 0000000..9e2ce24
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/open_storage_content_prompt.py
@@ -0,0 +1,191 @@
+#!/usr/bin/env python3
+"""Audit repeated content prompts inside verified open-storage hosts."""
+
+from __future__ import annotations
+
+import copy
+import re
+from collections import defaultdict
+from dataclasses import asdict, dataclass
+from typing import Any, Mapping, Sequence
+
+
+@dataclass(frozen=True)
+class OpenStorageContentPromptConfig:
+ minimum_repeated_instances: int = 2
+ minimum_bbox_coverage_by_host: float = 0.80
+
+
+GENERIC_CATEGORY_WORDS = {"item", "object", "thing", "unknown"}
+
+
+def _words(value: Any) -> list[str]:
+ return re.findall(r"[a-z0-9]+", str(value or "").lower().replace("_", " "))
+
+
+def _canonical_category(value: Any) -> str:
+ return " ".join(_words(value))
+
+
+def _valid_bbox(value: Any) -> list[float] | None:
+ if (
+ not isinstance(value, Sequence)
+ or isinstance(value, (str, bytes))
+ or len(value) != 4
+ ):
+ return None
+ try:
+ box = [float(item) for item in value]
+ except (TypeError, ValueError):
+ return None
+ if not (0.0 <= box[0] < box[2] <= 1.0 and 0.0 <= box[1] < box[3] <= 1.0):
+ return None
+ return box
+
+
+def _bbox_coverage(inner: Sequence[float], outer: Sequence[float]) -> float:
+ x1 = max(float(inner[0]), float(outer[0]))
+ y1 = max(float(inner[1]), float(outer[1]))
+ x2 = min(float(inner[2]), float(outer[2]))
+ y2 = min(float(inner[3]), float(outer[3]))
+ intersection = max(0.0, x2 - x1) * max(0.0, y2 - y1)
+ area = (float(inner[2]) - float(inner[0])) * (
+ float(inner[3]) - float(inner[1])
+ )
+ return intersection / area if area else 0.0
+
+
+def canonicalize_open_storage_content_prompts(
+ *,
+ objects: Sequence[Mapping[str, Any]],
+ content_hosts: Sequence[Mapping[str, Any]],
+ config: OpenStorageContentPromptConfig | None = None,
+) -> dict[str, Any]:
+ """Record repeated host contents without rewriting instance prompts.
+
+ A repeated category may contain visually distinct objects or deliberately
+ grouped units. Its declared sam3_prompt is therefore an input contract
+ for segmentation, not a field this audit may normalize.
+ """
+
+ cfg = config or OpenStorageContentPromptConfig()
+ if cfg.minimum_repeated_instances < 2:
+ raise ValueError("minimum_repeated_instances must be at least two")
+ if not 0.0 < cfg.minimum_bbox_coverage_by_host <= 1.0:
+ raise ValueError("minimum_bbox_coverage_by_host must be in (0, 1]")
+ copied = [
+ copy.deepcopy(dict(item))
+ for item in objects
+ if isinstance(item, Mapping)
+ ]
+ hosts = []
+ for raw in content_hosts:
+ box = _valid_bbox(raw.get("bbox_norm_xyxy"))
+ if str(raw.get("host_mode") or "") != "open_storage" or box is None:
+ continue
+ hosts.append(
+ {
+ "source_object_id": str(raw.get("source_object_id") or ""),
+ "bbox_norm_xyxy": box,
+ "area": (box[2] - box[0]) * (box[3] - box[1]),
+ }
+ )
+
+ groups: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
+ rejections = []
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ box = _valid_bbox(obj.get("bbox_2d_norm_xyxy"))
+ category = _canonical_category(obj.get("category"))
+ category_words = set(_words(category))
+ prompt = " ".join(_words(obj.get("sam3_prompt")))
+ if (
+ not object_id
+ or box is None
+ or not category
+ or category_words <= GENERIC_CATEGORY_WORDS
+ or not prompt
+ ):
+ continue
+ matching_hosts = [
+ host
+ for host in hosts
+ if _bbox_coverage(box, host["bbox_norm_xyxy"])
+ >= cfg.minimum_bbox_coverage_by_host
+ ]
+ if not matching_hosts:
+ continue
+ host = min(matching_hosts, key=lambda item: item["area"])
+ if not category_words.issubset(set(_words(prompt))):
+ rejections.append(
+ {
+ "object_id": object_id,
+ "category": category,
+ "sam3_prompt": prompt,
+ "host_source_object_id": host["source_object_id"],
+ "reason": "category_words_not_preserved_by_prompt",
+ }
+ )
+ continue
+ groups[(host["source_object_id"], category)].append(
+ {
+ "object": obj,
+ "object_id": object_id,
+ "old_prompt": prompt,
+ "bbox_coverage_by_host": _bbox_coverage(
+ box, host["bbox_norm_xyxy"]
+ ),
+ }
+ )
+
+ changes = []
+ preserved = []
+ checked_groups = []
+ for (host_id, category), members in sorted(groups.items()):
+ checked_groups.append(
+ {
+ "host_source_object_id": host_id,
+ "category": category,
+ "object_ids": [item["object_id"] for item in members],
+ "input_prompts": sorted({item["old_prompt"] for item in members}),
+ }
+ )
+ if len(members) < cfg.minimum_repeated_instances:
+ continue
+ for member in members:
+ preserved.append(
+ {
+ "object_id": member["object_id"],
+ "host_source_object_id": host_id,
+ "category": category,
+ "sam3_prompt": member["old_prompt"],
+ "bbox_coverage_by_host": round(
+ member["bbox_coverage_by_host"], 6
+ ),
+ }
+ )
+
+ audit = {
+ "schema_version": "scenemosaic_open_storage_content_prompt_audit",
+ "status": "noop",
+ "config": asdict(cfg),
+ "verified_open_storage_host_count": len(hosts),
+ "checked_groups": checked_groups,
+ "changed_count": len(changes),
+ "changes": changes,
+ "preserved_count": len(preserved),
+ "preserved": preserved,
+ "rejections": rejections,
+ "semantic_categories_modified": False,
+ "object_count_modified": False,
+ "additional_sam3_inference_calls": 0,
+ "additional_vlm_calls": 0,
+ "hard_errors": [],
+ }
+ return {"objects": copied, "audit": audit}
+
+
+__all__ = [
+ "OpenStorageContentPromptConfig",
+ "canonicalize_open_storage_content_prompts",
+]
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/pass_a_same_mask_conflict.py b/vlm_agent/reconstruction/perception/pipeline/refinements/pass_a_same_mask_conflict.py
new file mode 100644
index 0000000..21927fc
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/pass_a_same_mask_conflict.py
@@ -0,0 +1,428 @@
+#!/usr/bin/env python3
+"""Fail-closed closure for distinct Pass-A claims sharing one SAM3 mask.
+
+SAM3 occasionally returns one broad region for two nearby Pass-A claims. A
+near-identical mask alone is not proof that the claims are duplicates: the
+scene may contain two adjacent physical units. This helper therefore only
+detects and reports the conflict before Pass B. After Pass B, it accepts an
+explicit removal as a same-instance decision; otherwise it attempts two
+bounded positive-box SAM3 probes and accepts them only when they are clearly
+separate. An unresolved pair remains unchanged for the normal quality gates
+to reject, rather than silently deleting an object.
+"""
+
+from __future__ import annotations
+
+import copy
+import json
+import math
+import os
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+
+@dataclass(frozen=True)
+class SameMaskConflictConfig:
+ maximum_conflicts: int = 4
+ minimum_intersection_pixels: int = 32
+ minimum_mask_iou: float = 0.995
+ minimum_mask_coverage: float = 0.995
+ maximum_bbox_gap_norm: float = 0.04
+ minimum_orthogonal_overlap_fraction: float = 0.20
+ split_score_threshold: float = 0.20
+ split_minimum_selected_score: float = 0.45
+ split_minimum_bbox_coverage: float = 0.45
+ split_maximum_center_distance_scale: float = 0.85
+ split_maximum_pair_iou: float = 0.18
+ split_maximum_candidates_per_claim: int = 8
+
+
+def _atomic_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _load_mask(path_value: Any, shape: tuple[int, int]) -> np.ndarray:
+ path = Path(str(path_value or "")).expanduser()
+ if not path.is_file():
+ raise FileNotFoundError(f"mask is missing: {path}")
+ mask = np.array(Image.open(path).convert("L")) > 0
+ if tuple(mask.shape) != shape:
+ raise ValueError(f"mask has unexpected shape: {path}")
+ return mask
+
+
+def _mask_bbox(mask: np.ndarray) -> list[int]:
+ ys, xs = np.where(mask)
+ if not len(xs):
+ raise ValueError("mask is empty")
+ return [int(xs.min()), int(ys.min()), int(xs.max()), int(ys.max())]
+
+
+def _norm_bbox(obj: Mapping[str, Any]) -> list[float]:
+ try:
+ box = [float(value) for value in obj.get("bbox_2d_norm_xyxy")]
+ except (TypeError, ValueError):
+ raise ValueError("invalid normalized bbox") from None
+ if (
+ len(box) != 4
+ or not all(math.isfinite(value) for value in box)
+ or not (0.0 <= box[0] < box[2] <= 1.0)
+ or not (0.0 <= box[1] < box[3] <= 1.0)
+ ):
+ raise ValueError("invalid normalized bbox")
+ return box
+
+
+def _bbox_metrics(target: Sequence[float], candidate: Sequence[float]) -> dict[str, float]:
+ tx1, ty1, tx2, ty2 = [float(value) for value in target]
+ cx1, cy1, cx2, cy2 = [float(value) for value in candidate]
+ ix1, iy1 = max(tx1, cx1), max(ty1, cy1)
+ ix2, iy2 = min(tx2, cx2), min(ty2, cy2)
+ intersection = max(0.0, ix2 - ix1 + 1.0) * max(0.0, iy2 - iy1 + 1.0)
+ target_area = max(1.0, (tx2 - tx1 + 1.0) * (ty2 - ty1 + 1.0))
+ candidate_area = max(1.0, (cx2 - cx1 + 1.0) * (cy2 - cy1 + 1.0))
+ union = target_area + candidate_area - intersection
+ target_center = ((tx1 + tx2) / 2.0, (ty1 + ty2) / 2.0)
+ candidate_center = ((cx1 + cx2) / 2.0, (cy1 + cy2) / 2.0)
+ scale = max(math.hypot(tx2 - tx1, ty2 - ty1), 1.0)
+ return {
+ "target_bbox_coverage": intersection / target_area,
+ "candidate_bbox_coverage": intersection / candidate_area,
+ "bbox_iou": intersection / union if union else 0.0,
+ "center_distance_scale": math.dist(target_center, candidate_center) / scale,
+ }
+
+
+def _mask_iou(left: np.ndarray, right: np.ndarray) -> float:
+ union = int(np.logical_or(left, right).sum())
+ return int(np.logical_and(left, right).sum()) / union if union else 0.0
+
+
+def _spatially_related(left: Sequence[float], right: Sequence[float], cfg: SameMaskConflictConfig) -> bool:
+ lx1, ly1, lx2, ly2 = left
+ rx1, ry1, rx2, ry2 = right
+ horizontal_gap = max(0.0, max(lx1, rx1) - min(lx2, rx2))
+ vertical_gap = max(0.0, max(ly1, ry1) - min(ly2, ry2))
+ horizontal_overlap = max(0.0, min(lx2, rx2) - max(lx1, rx1))
+ vertical_overlap = max(0.0, min(ly2, ry2) - max(ly1, ry1))
+ left_width, right_width = lx2 - lx1, rx2 - rx1
+ left_height, right_height = ly2 - ly1, ry2 - ry1
+ if horizontal_gap <= cfg.maximum_bbox_gap_norm:
+ return vertical_overlap >= cfg.minimum_orthogonal_overlap_fraction * min(left_height, right_height)
+ if vertical_gap <= cfg.maximum_bbox_gap_norm:
+ return horizontal_overlap >= cfg.minimum_orthogonal_overlap_fraction * min(left_width, right_width)
+ return False
+
+
+def _object_is_eligible(obj: Mapping[str, Any]) -> bool:
+ """Exclude structural claims and known fixed-component assembly inputs."""
+
+ text = " ".join(
+ str(obj.get(key) or "")
+ for key in ("category", "sam3_prompt", "instance_name")
+ ).lower()
+ excluded = {
+ "wall", "floor", "ceiling", "window", "door", "curtain",
+ "faucet", "sconce", "lamp", "light", "handle", "knob",
+ }
+ return not any(token in text for token in excluded)
+
+
+def detect_pass_a_same_mask_conflicts(
+ *,
+ image_path: str | Path,
+ registry: Mapping[str, Any],
+ output_dir: str | Path,
+ config: SameMaskConflictConfig | None = None,
+) -> dict[str, Any]:
+ """Detect only ambiguous same-mask pairs; this does not change registry."""
+
+ cfg = config or SameMaskConflictConfig()
+ output = Path(output_dir).expanduser().resolve()
+ if output.exists() and any(output.iterdir()):
+ raise FileExistsError(f"same-mask conflict output is not empty: {output}")
+ output.mkdir(parents=True, exist_ok=True)
+ with Image.open(Path(image_path).expanduser().resolve()) as source:
+ shape = (source.height, source.width)
+ objects = list(registry.get("objects") or [])
+ records: list[dict[str, Any]] = []
+ errors: list[dict[str, Any]] = []
+ candidates: list[tuple[Mapping[str, Any], np.ndarray, list[float]]] = []
+ for obj in objects:
+ object_id = str(obj.get("object_id") or "")
+ sam3 = obj.get("sam3") or {}
+ if not object_id or sam3.get("status") != "success" or not _object_is_eligible(obj):
+ continue
+ try:
+ candidates.append((obj, _load_mask(sam3.get("mask_path"), shape), _norm_bbox(obj)))
+ except Exception as exc:
+ errors.append({"type": "same_mask_conflict_invalid_source_mask", "object_id": object_id, "error": f"{type(exc).__name__}: {exc}"})
+ for left_index, (left, left_mask, left_box) in enumerate(candidates):
+ for right, right_mask, right_box in candidates[left_index + 1 :]:
+ if not _spatially_related(left_box, right_box, cfg):
+ continue
+ intersection = int(np.logical_and(left_mask, right_mask).sum())
+ if intersection < cfg.minimum_intersection_pixels:
+ continue
+ left_area, right_area = int(left_mask.sum()), int(right_mask.sum())
+ coverage_left = intersection / max(left_area, 1)
+ coverage_right = intersection / max(right_area, 1)
+ iou = _mask_iou(left_mask, right_mask)
+ if (
+ iou < cfg.minimum_mask_iou
+ or coverage_left < cfg.minimum_mask_coverage
+ or coverage_right < cfg.minimum_mask_coverage
+ ):
+ continue
+ owner_ids = sorted((str(left["object_id"]), str(right["object_id"])))
+ left_by_id = {str(left["object_id"]): (left, left_box), str(right["object_id"]): (right, right_box)}
+ records.append({
+ "conflict_id": "",
+ "object_ids": owner_ids,
+ "objects": [
+ {
+ "object_id": object_id,
+ "category": str(left_by_id[object_id][0].get("category") or "unknown"),
+ "sam3_prompt": str(left_by_id[object_id][0].get("sam3_prompt") or "object"),
+ "vlm_bbox_norm_xyxy": [round(value, 6) for value in left_by_id[object_id][1]],
+ }
+ for object_id in owner_ids
+ ],
+ "mask_iou": round(iou, 6),
+ "intersection_pixels": intersection,
+ "coverage_by_object_id": {
+ str(left["object_id"]): round(coverage_left, 6),
+ str(right["object_id"]): round(coverage_right, 6),
+ },
+ "resolution_policy": "pass_b_remove_one_or_local_bbox_split",
+ })
+ records.sort(key=lambda item: (-float(item["mask_iou"]), item["object_ids"]))
+ if len(records) > cfg.maximum_conflicts:
+ errors.append({"type": "same_mask_conflict_capacity_overflow", "actual": len(records), "maximum": cfg.maximum_conflicts})
+ records = records[: cfg.maximum_conflicts]
+ for index, record in enumerate(records, start=1):
+ record["conflict_id"] = f"SMC_{index:03d}"
+ audit = {
+ "schema_version": "scenemosaic_pass_a_same_mask_conflict",
+ "status": "blocked" if errors else ("pass" if records else "noop"),
+ "config": asdict(cfg),
+ "conflict_count": len(records),
+ "records": records,
+ "additional_vlm_calls": 0,
+ "additional_sam3_inference_count": 0,
+ "formal_outputs_modified": False,
+ "hard_errors": errors,
+ }
+ _atomic_json(output / "pass_a_same_mask_conflicts_audit.json", audit)
+ return {"conflicts": records, "audit": audit}
+
+
+def _abs_box(box: Sequence[float], width: int, height: int) -> list[int]:
+ x1 = max(0, min(width - 1, int(round(float(box[0]) * width))))
+ y1 = max(0, min(height - 1, int(round(float(box[1]) * height))))
+ x2 = max(x1 + 1, min(width, int(round(float(box[2]) * width))))
+ y2 = max(y1 + 1, min(height, int(round(float(box[3]) * height))))
+ return [x1, y1, x2, y2]
+
+
+def _select_bbox_candidate(
+ *,
+ candidates: Sequence[Mapping[str, Any]],
+ target_box: Sequence[int],
+ shape: tuple[int, int],
+ cfg: SameMaskConflictConfig,
+) -> tuple[dict[str, Any] | None, list[dict[str, Any]]]:
+ evaluations = []
+ for candidate in candidates:
+ try:
+ mask = _load_mask(candidate.get("mask_path"), shape)
+ candidate_box = _mask_bbox(mask)
+ except Exception as exc:
+ evaluations.append({"candidate_id": candidate.get("candidate_id"), "eligible": False, "reason": f"invalid_candidate: {type(exc).__name__}: {exc}"})
+ continue
+ metrics = _bbox_metrics(target_box, candidate_box)
+ score = float(candidate.get("score") or 0.0)
+ eligible = (
+ score >= cfg.split_minimum_selected_score
+ and metrics["target_bbox_coverage"] >= cfg.split_minimum_bbox_coverage
+ and metrics["candidate_bbox_coverage"] >= cfg.split_minimum_bbox_coverage
+ and metrics["center_distance_scale"] <= cfg.split_maximum_center_distance_scale
+ )
+ quality = score + metrics["target_bbox_coverage"] + 0.25 * metrics["candidate_bbox_coverage"] - 0.20 * metrics["center_distance_scale"]
+ evaluations.append({
+ "candidate_id": str(candidate.get("candidate_id") or ""),
+ "mask_path": str(candidate.get("mask_path") or ""),
+ "score": score,
+ "mask": mask,
+ "mask_bbox_abs_xyxy": candidate_box,
+ "target_bbox_coverage": round(metrics["target_bbox_coverage"], 6),
+ "candidate_bbox_coverage": round(metrics["candidate_bbox_coverage"], 6),
+ "center_distance_scale": round(metrics["center_distance_scale"], 6),
+ "quality": quality,
+ "eligible": eligible,
+ "reason": None if eligible else "bounded_split_geometry_or_score_gate",
+ })
+ eligible = sorted((item for item in evaluations if item["eligible"]), key=lambda item: (-float(item["quality"]), item["candidate_id"]))
+ public = [{key: value for key, value in item.items() if key != "mask"} for item in evaluations]
+ return (eligible[0] if eligible else None), public
+
+
+def resolve_pass_a_same_mask_conflicts(
+ *,
+ session: Any,
+ infer_text_candidates_fn: Any,
+ image_path: str | Path,
+ registry: Mapping[str, Any],
+ actions: Mapping[str, Any],
+ conflicts: Sequence[Mapping[str, Any]],
+ output_dir: str | Path,
+ config: SameMaskConflictConfig | None = None,
+) -> dict[str, Any]:
+ """Apply Pass-B decision or bounded local split without changing actions."""
+
+ cfg = config or SameMaskConflictConfig()
+ output = Path(output_dir).expanduser().resolve()
+ if output.exists() and any(output.iterdir()):
+ raise FileExistsError(f"same-mask resolution output is not empty: {output}")
+ output.mkdir(parents=True, exist_ok=True)
+ copied = copy.deepcopy(dict(registry))
+ objects = list(copied.get("objects") or [])
+ by_id = {str(obj.get("object_id") or ""): obj for obj in objects}
+ removed = {str(value) for value in (actions.get("removals") or [])}
+ updated = {
+ str(item.get("pass_a_id") or "")
+ for item in (actions.get("updates") or [])
+ if isinstance(item, Mapping)
+ }
+ resolved_image = Path(image_path).expanduser().resolve()
+ image = np.array(Image.open(resolved_image).convert("RGB"), dtype=np.uint8)
+ shape = tuple(image.shape[:2])
+ height, width = shape
+ records = []
+ errors: list[dict[str, Any]] = []
+ inference_before = int(getattr(session, "inference_run_count", 0))
+ for conflict in conflicts:
+ conflict_id = str(conflict.get("conflict_id") or "")
+ owner_ids = [str(value) for value in (conflict.get("object_ids") or [])]
+ if len(owner_ids) != 2 or any(owner_id not in by_id for owner_id in owner_ids):
+ errors.append({"type": "same_mask_conflict_invalid_owners", "conflict_id": conflict_id, "object_ids": owner_ids})
+ continue
+ selected_removals = sorted(set(owner_ids) & removed)
+ selected_updates = sorted(set(owner_ids) & updated)
+ if selected_updates:
+ errors.append({
+ "type": "same_mask_conflict_local_split_blocked_by_pass_b_update",
+ "conflict_id": conflict_id,
+ "object_ids": owner_ids,
+ "updated_object_ids": selected_updates,
+ })
+ records.append({
+ "conflict_id": conflict_id,
+ "object_ids": owner_ids,
+ "status": "unresolved",
+ "reason": "pass_b_update_requires_delta_resegmentation",
+ "updated_object_ids": selected_updates,
+ })
+ continue
+ if len(selected_removals) == 1:
+ records.append({"conflict_id": conflict_id, "object_ids": owner_ids, "status": "same_instance_removed", "removed_object_id": selected_removals[0], "additional_sam3_inference_count": 0})
+ continue
+ if len(selected_removals) == 2:
+ errors.append({"type": "same_mask_conflict_removed_both", "conflict_id": conflict_id, "object_ids": owner_ids})
+ records.append({"conflict_id": conflict_id, "object_ids": owner_ids, "status": "blocked", "reason": "both_claims_removed"})
+ continue
+ winners: dict[str, dict[str, Any]] = {}
+ evaluations_by_owner: dict[str, list[dict[str, Any]]] = {}
+ for index, owner_id in enumerate(owner_ids, start=1):
+ obj = by_id[owner_id]
+ target_box = _abs_box(_norm_bbox(obj), width, height)
+ raw_dir = output / "raw_candidates" / conflict_id.lower() / owner_id.lower()
+ result = infer_text_candidates_fn(
+ session,
+ resolved_image,
+ str(obj.get("sam3_prompt") or obj.get("category") or "object"),
+ raw_dir,
+ score_threshold=cfg.split_score_threshold,
+ positive_bbox_abs_xyxy=target_box,
+ maximum_candidates=cfg.split_maximum_candidates_per_claim,
+ overwrite=False,
+ )
+ winner, public_evaluations = _select_bbox_candidate(
+ candidates=list((result.get("registry") or {}).get("candidates") or []),
+ target_box=target_box,
+ shape=shape,
+ cfg=cfg,
+ )
+ evaluations_by_owner[owner_id] = public_evaluations
+ if winner is not None:
+ winners[owner_id] = winner
+ if len(winners) != 2:
+ errors.append({"type": "same_mask_conflict_local_split_failed", "conflict_id": conflict_id, "object_ids": owner_ids, "reason": "missing_eligible_bounded_candidate"})
+ records.append({"conflict_id": conflict_id, "object_ids": owner_ids, "status": "unresolved", "reason": "missing_eligible_bounded_candidate", "evaluations": evaluations_by_owner})
+ continue
+ pair_iou = _mask_iou(winners[owner_ids[0]]["mask"], winners[owner_ids[1]]["mask"])
+ if pair_iou > cfg.split_maximum_pair_iou:
+ errors.append({"type": "same_mask_conflict_local_split_failed", "conflict_id": conflict_id, "object_ids": owner_ids, "reason": "local_candidates_still_overlap", "pair_mask_iou": round(pair_iou, 6)})
+ records.append({"conflict_id": conflict_id, "object_ids": owner_ids, "status": "unresolved", "reason": "local_candidates_still_overlap", "pair_mask_iou": round(pair_iou, 6), "evaluations": evaluations_by_owner})
+ continue
+ selected_dir = output / "selected" / conflict_id.lower()
+ selected_dir.mkdir(parents=True, exist_ok=True)
+ selected_objects = []
+ for owner_id in owner_ids:
+ winner = winners[owner_id]
+ mask_path = selected_dir / f"{owner_id.lower()}_mask.png"
+ overlay_path = selected_dir / f"{owner_id.lower()}_overlay.png"
+ Image.fromarray(winner["mask"].astype(np.uint8) * 255).save(mask_path)
+ canvas = image.astype(np.float32).copy()
+ canvas[winner["mask"]] = 0.55 * canvas[winner["mask"]] + 0.45 * np.array((255, 0, 0), dtype=np.float32)
+ Image.fromarray(np.clip(canvas, 0, 255).astype(np.uint8)).save(overlay_path)
+ previous = copy.deepcopy(dict(by_id[owner_id].get("sam3") or {}))
+ by_id[owner_id]["sam3"] = {
+ **previous,
+ "status": "success",
+ "mask_path": str(mask_path),
+ "overlay_path": str(overlay_path),
+ "area_pixels": int(winner["mask"].sum()),
+ "mask_bbox_abs_xyxy": list(winner["mask_bbox_abs_xyxy"]),
+ "score": float(winner["score"]),
+ "selection_method": "pass_a_same_mask_local_bbox_split",
+ "pass_a_same_mask_local_bbox_split": {
+ "conflict_id": conflict_id,
+ "candidate_id": winner["candidate_id"],
+ "target_bbox_coverage": winner["target_bbox_coverage"],
+ "candidate_bbox_coverage": winner["candidate_bbox_coverage"],
+ "center_distance_scale": winner["center_distance_scale"],
+ "pair_mask_iou": round(pair_iou, 6),
+ },
+ }
+ selected_objects.append(
+ {
+ "object_id": owner_id,
+ "selected_mask_path": str(mask_path),
+ "candidate_id": winner["candidate_id"],
+ }
+ )
+ records.append({"conflict_id": conflict_id, "object_ids": owner_ids, "status": "locally_split", "pair_mask_iou": round(pair_iou, 6), "selected": selected_objects, "evaluations": evaluations_by_owner})
+ inference_after = int(getattr(session, "inference_run_count", inference_before))
+ audit = {
+ "schema_version": "scenemosaic_pass_a_same_mask_conflict_resolution",
+ "status": "blocked" if errors else ("pass" if records else "noop"),
+ "conflict_count": len(conflicts),
+ "records": records,
+ "additional_vlm_calls": 0,
+ "additional_sam3_inference_count": max(0, inference_after - inference_before),
+ "formal_outputs_modified": False,
+ "hard_errors": errors,
+ }
+ _atomic_json(output / "pass_a_same_mask_conflict_resolution_audit.json", audit)
+ return {"registry": copied, "audit": audit}
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/post_delta_multiplicity_closure.py b/vlm_agent/reconstruction/perception/pipeline/refinements/post_delta_multiplicity_closure.py
new file mode 100644
index 0000000..9823710
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/post_delta_multiplicity_closure.py
@@ -0,0 +1,1196 @@
+#!/usr/bin/env python3
+"""Recover repeated instances when VLM count is lower than delta SAM3 evidence."""
+
+from __future__ import annotations
+
+import copy
+import json
+import os
+import re
+from collections import defaultdict
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+from scipy import ndimage
+
+
+@dataclass(frozen=True)
+class MultiplicityClosureConfig:
+ enabled: bool = True
+ isolated_attempt_only: bool = True
+ minimum_peer_count: int = 2
+ minimum_candidate_score: float = 0.85
+ minimum_single_peer_score: float = 0.88
+ minimum_single_peer_area_ratio: float = 0.35
+ minimum_single_peer_dimension_ratio: float = 0.35
+ minimum_single_peer_suspended_light_score: float = 0.50
+ minimum_single_peer_suspended_light_area_ratio: float = 0.15
+ minimum_single_peer_suspended_light_dimension_ratio: float = 0.30
+ minimum_recessed_ceiling_light_score: float = 0.80
+ maximum_recessed_ceiling_light_y_fraction: float = 0.25
+ minimum_area_ratio: float = 0.55
+ maximum_area_ratio: float = 1.80
+ minimum_dimension_ratio: float = 0.50
+ maximum_dimension_ratio: float = 2.00
+ maximum_existing_iou: float = 0.10
+ maximum_existing_candidate_coverage: float = 0.20
+ minimum_conflict_pixels: int = 20
+ minimum_completion_candidate_score: float = 0.30
+ minimum_superset_existing_coverage: float = 0.95
+ maximum_superset_area_ratio: float = 3.00
+ minimum_touching_candidate_score: float = 0.60
+ minimum_touching_candidate_area_ratio: float = 1.05
+ maximum_touching_candidate_area_ratio: float = 2.00
+ minimum_superset_union_bbox_coverage: float = 0.70
+ minimum_completion_union_bbox_coverage: float = 0.90
+ maximum_completion_overlap_with_other: float = 0.15
+ minimum_adjacent_component_score: float = 0.60
+ minimum_adjacent_component_area_ratio: float = 0.20
+ maximum_adjacent_component_area_ratio: float = 1.50
+ adjacent_component_contact_dilation_pixels: int = 1
+ maximum_adjacent_completion_components: int = 4
+ minimum_failed_object_candidate_score: float = 0.30
+ maximum_additions_per_group: int = 2
+ maximum_total_additions: int = 4
+
+
+HOST_WORDS = {
+ "cart",
+ "shelf",
+ "shelving",
+ "rack",
+ "cabinet",
+ "bookcase",
+ "storage",
+ "counter",
+ "worktable",
+ "workbench",
+}
+OPEN_STORAGE_WORDS = {
+ "bookcase",
+ "bookshelf",
+ "cubby",
+ "hutch",
+ "rack",
+ "shelf",
+ "shelving",
+ "storage",
+}
+SINGLE_PEER_REPEAT_WORDS = {"chair", "stool"}
+SINGLE_PEER_SUSPENDED_LIGHT_WORDS = {"pendant", "chandelier"}
+
+
+
+
+def normalize(value: Any) -> str:
+ return " ".join(re.sub(r"[^a-z0-9]+", " ", str(value or "").lower()).split())
+
+
+def sam3_query_key(item: Mapping[str, Any]) -> str:
+ """Return the text query that actually produced an item's SAM3 mask.
+
+ Repeated inventory entries can retain descriptive VLM prompts such as
+ ``books`` and ``stacked books`` while the segmentation stage shares one
+ category-level SAM3 query, ``book``. Candidate closure must compare that
+ executed query, not the descriptive inventory wording.
+ """
+
+ sam3 = item.get("sam3") or {}
+ return normalize(sam3.get("prompt_used") or item.get("sam3_prompt"))
+
+
+def is_pass_b_addition(obj: Mapping[str, Any]) -> bool:
+ return any(
+ str(item.get("pass") or "").upper() == "B"
+ and str(item.get("action") or "").lower() == "added"
+ for item in (obj.get("proposal_sources") or [])
+ if isinstance(item, Mapping)
+ )
+
+
+def is_host(obj: Mapping[str, Any]) -> bool:
+ text = " ".join(
+ normalize(obj.get(key))
+ for key in ("category", "sam3_prompt", "instance_name", "description")
+ )
+ return any(re.search(rf"\b{re.escape(word)}\b", text) for word in HOST_WORDS)
+
+
+def is_open_storage_host(obj: Mapping[str, Any]) -> bool:
+ text = " ".join(
+ normalize(obj.get(key))
+ for key in ("category", "sam3_prompt", "instance_name", "description")
+ )
+ return any(
+ re.search(rf"\b{re.escape(word)}\b", text)
+ for word in OPEN_STORAGE_WORDS
+ )
+
+
+def masks_touch(
+ left: np.ndarray,
+ right: np.ndarray,
+ *,
+ dilation_pixels: int,
+) -> bool:
+ if np.logical_and(left, right).any():
+ return True
+ return bool(
+ np.logical_and(
+ ndimage.binary_dilation(
+ left,
+ structure=np.ones((3, 3), dtype=bool),
+ iterations=max(1, dilation_pixels),
+ ),
+ right,
+ ).any()
+ )
+
+
+def retain_completion_components_connected_to_base(
+ candidate: np.ndarray,
+ base: np.ndarray,
+ *,
+ dilation_pixels: int,
+) -> tuple[np.ndarray, int]:
+ """Keep only candidate components connected to an existing instance mask.
+
+ SAM3 may return one candidate mask containing a valid whole-instance
+ component plus an unrelated, disconnected instance of the same prompt.
+ Multiplicity completion must never turn those separate instances into one
+ object merely because their enclosing bounding box is target-local.
+ """
+
+ labels, component_count = ndimage.label(candidate)
+ if component_count <= 1:
+ return candidate, 0
+ contact_region = ndimage.binary_dilation(
+ base,
+ structure=np.ones((3, 3), dtype=bool),
+ iterations=max(1, dilation_pixels),
+ )
+ connected_labels = {
+ int(label)
+ for label in np.unique(labels[contact_region])
+ if int(label) > 0
+ }
+ if not connected_labels:
+ return np.zeros_like(candidate, dtype=bool), int(component_count)
+ retained = np.isin(labels, list(connected_labels))
+ return retained, int(component_count - len(connected_labels))
+
+
+def bbox_from_mask(mask: np.ndarray) -> list[int] | None:
+ ys, xs = np.nonzero(mask)
+ if not len(xs):
+ return None
+ return [
+ int(xs.min()),
+ int(ys.min()),
+ int(xs.max()) + 1,
+ int(ys.max()) + 1,
+ ]
+
+
+def valid_bbox(raw: Any) -> list[int] | None:
+ try:
+ values = [int(round(float(value))) for value in raw]
+ except (TypeError, ValueError):
+ return None
+ if len(values) != 4 or not (values[0] < values[2] and values[1] < values[3]):
+ return None
+ return values
+
+
+def center(box: Sequence[int]) -> tuple[float, float]:
+ return ((box[0] + box[2]) / 2.0, (box[1] + box[3]) / 2.0)
+
+
+def contains(box: Sequence[int], point: Sequence[float]) -> bool:
+ return box[0] <= point[0] <= box[2] and box[1] <= point[1] <= box[3]
+
+
+def overlap(a: np.ndarray, b: np.ndarray) -> dict[str, float | int]:
+ intersection = int(np.logical_and(a, b).sum())
+ aa, ba = int(a.sum()), int(b.sum())
+ union = aa + ba - intersection
+ return {
+ "intersection_pixels": intersection,
+ "iou": intersection / union if union else 0.0,
+ "candidate_coverage": intersection / aa if aa else 0.0,
+ }
+
+
+def bbox_coverage(outer: Sequence[int], inner: Sequence[int]) -> float:
+ x1 = max(int(outer[0]), int(inner[0]))
+ y1 = max(int(outer[1]), int(inner[1]))
+ x2 = min(int(outer[2]), int(inner[2]))
+ y2 = min(int(outer[3]), int(inner[3]))
+ intersection = max(0, x2 - x1) * max(0, y2 - y1)
+ inner_area = max(
+ 1,
+ (int(inner[2]) - int(inner[0]))
+ * (int(inner[3]) - int(inner[1])),
+ )
+ return intersection / inner_area
+
+
+def expanded_object_bbox(obj: Mapping[str, Any]) -> list[int] | None:
+ sam3 = obj.get("sam3") or {}
+ locality = sam3.get("scale_aware_locality") or {}
+ return (
+ valid_bbox(locality.get("expanded_vlm_bbox_abs_xyxy"))
+ or valid_bbox(sam3.get("vlm_bbox_abs_xyxy"))
+ )
+
+
+def candidate_key(candidate: Mapping[str, Any]) -> str:
+ return str(
+ candidate.get("mask_path")
+ or candidate.get("candidate_id")
+ or ""
+ )
+
+
+def load_mask(path_value: Any, shape: tuple[int, int], root: Path | None = None) -> tuple[np.ndarray, Path]:
+ path = Path(str(path_value or "")).expanduser().resolve()
+ if root is not None and not path.is_relative_to(root.resolve()):
+ raise ValueError("candidate path escapes delta unassigned root")
+ mask = np.array(Image.open(path).convert("L")) > 0
+ if mask.shape != shape or not mask.any():
+ raise ValueError("invalid mask")
+ return mask, path
+
+
+def next_object_id(objects: Sequence[Mapping[str, Any]]) -> str:
+ maximum = 0
+ for obj in objects:
+ match = re.fullmatch(r"proposal_(\d+)", str(obj.get("object_id") or ""))
+ if match:
+ maximum = max(maximum, int(match.group(1)))
+ return f"proposal_{maximum + 1:03d}"
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ temporary.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ os.replace(temporary, path)
+
+
+def run_post_delta_multiplicity_closure(
+ *,
+ image_path: str | Path,
+ objects: Sequence[Mapping[str, Any]],
+ delta_registry: Mapping[str, Any] | None,
+ fallback_registry: Mapping[str, Any] | None = None,
+ work_dir: str | Path,
+ staging_dir: str | Path,
+ config: MultiplicityClosureConfig | None = None,
+) -> dict[str, Any]:
+ cfg = config or MultiplicityClosureConfig()
+ if not cfg.isolated_attempt_only:
+ raise ValueError("multiplicity closure is isolated-attempt only")
+ copied = [copy.deepcopy(dict(item)) for item in objects]
+ work = Path(work_dir).expanduser().resolve()
+ if work.exists() and any(work.iterdir()):
+ raise FileExistsError(f"multiplicity work directory is not empty: {work}")
+ work.mkdir(parents=True, exist_ok=True)
+ audit = {
+ "schema_version": "scenemosaic_post_delta_multiplicity_closure",
+ "enabled": cfg.enabled,
+ "status": "disabled" if not cfg.enabled else "pending",
+ "groups_checked": [],
+ "accepted": [],
+ "completed": [],
+ "rescued_failed": [],
+ "rejected": [],
+ "filtered_unconnected_components": [],
+ "accepted_count": 0,
+ "completed_count": 0,
+ "rescued_failed_count": 0,
+ "additional_vlm_calls": 0,
+ "additional_sam3_inference_calls": 0,
+ "hard_errors": [],
+ }
+ audit_path = work / "post_delta_multiplicity_closure_audit.json"
+ if not cfg.enabled:
+ atomic_json(audit_path, audit)
+ return {"objects": copied, "audit": audit, "audit_path": str(audit_path)}
+
+ candidates = []
+ seen_candidate_masks = set()
+ for registry_value in (delta_registry, fallback_registry):
+ for candidate in (registry_value or {}).get("unassigned_sam3_candidates") or []:
+ key = str(candidate.get("mask_path") or "")
+ if key and key not in seen_candidate_masks:
+ candidates.append(candidate)
+ seen_candidate_masks.add(key)
+ if not candidates:
+ audit["status"] = "noop"
+ atomic_json(audit_path, audit)
+ return {"objects": copied, "audit": audit, "audit_path": str(audit_path)}
+
+ with Image.open(Path(image_path).expanduser().resolve()) as source:
+ image_np = np.array(source.convert("RGB"))
+ height, width = image_np.shape[:2]
+ shape = (height, width)
+
+ existing_masks: dict[str, np.ndarray] = {}
+ for obj in copied:
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "success":
+ continue
+ try:
+ existing_masks[str(obj.get("object_id") or "")], _ = load_mask(sam3.get("mask_path"), shape)
+ except (OSError, ValueError):
+ audit["hard_errors"].append(
+ {"type": "invalid_existing_success_mask", "object_id": obj.get("object_id")}
+ )
+
+ staging = Path(staging_dir).expanduser().resolve()
+ candidate_masks: dict[str, np.ndarray] = {}
+ used_candidate_keys: set[str] = set()
+
+ def candidate_mask(candidate: Mapping[str, Any]) -> np.ndarray:
+ key = candidate_key(candidate)
+ if not key:
+ raise ValueError("candidate has no stable key")
+ if key not in candidate_masks:
+ candidate_masks[key], _ = load_mask(candidate.get("mask_path"), shape)
+ return candidate_masks[key]
+
+ def write_object_mask(
+ obj: Mapping[str, Any],
+ mask: np.ndarray,
+ ) -> tuple[str, str]:
+ object_id = str(obj.get("object_id") or "")
+ category_slug = (
+ normalize(obj.get("category")).replace(" ", "_")
+ or "object"
+ )
+ mask_path = (
+ staging
+ / "masks"
+ / f"{object_id}_{category_slug}_mask.png"
+ )
+ overlay_path = (
+ staging
+ / "overlays"
+ / f"{object_id}_{category_slug}_overlay.png"
+ )
+ mask_path.parent.mkdir(parents=True, exist_ok=True)
+ overlay_path.parent.mkdir(parents=True, exist_ok=True)
+ Image.fromarray(mask.astype(np.uint8) * 255).save(mask_path)
+ overlay_image = image_np.copy()
+ overlay_image[mask] = (
+ 0.55 * overlay_image[mask]
+ + 0.45 * np.array((255, 0, 0), dtype=np.uint8)
+ ).astype(np.uint8)
+ Image.fromarray(overlay_image).save(overlay_path)
+ return str(mask_path), str(overlay_path)
+
+ def shared_host(
+ points: Sequence[Sequence[float]],
+ excluded_object_ids: set[str],
+ ) -> tuple[dict[str, Any] | None, list[int] | None]:
+ hosts = []
+ for host in copied:
+ object_id = str(host.get("object_id") or "")
+ host_box = valid_bbox(
+ (host.get("sam3") or {}).get("vlm_bbox_abs_xyxy")
+ )
+ if (
+ object_id not in excluded_object_ids
+ and host_box
+ and is_host(host)
+ and (host.get("sam3") or {}).get("status") == "success"
+ and all(contains(host_box, point) for point in points)
+ ):
+ area = max(
+ 1,
+ (host_box[2] - host_box[0])
+ * (host_box[3] - host_box[1]),
+ )
+ hosts.append((area, host, host_box))
+ if not hosts:
+ return None, None
+ _, host, host_box = min(hosts, key=lambda item: item[0])
+ return host, host_box
+
+ def mask_conflicts(
+ mask: np.ndarray,
+ *,
+ object_id: str,
+ exempt_object_ids: set[str],
+ maximum_coverage: float,
+ ) -> list[dict[str, Any]]:
+ area = int(mask.sum())
+ conflicts = []
+ if not area:
+ return conflicts
+ for other_id, other_mask in existing_masks.items():
+ if other_id == object_id or other_id in exempt_object_ids:
+ continue
+ intersection = int(np.logical_and(mask, other_mask).sum())
+ coverage = intersection / area
+ if (
+ intersection >= cfg.minimum_conflict_pixels
+ and coverage > maximum_coverage
+ ):
+ conflicts.append(
+ {
+ "object_id": other_id,
+ "intersection_pixels": intersection,
+ "candidate_coverage": round(coverage, 6),
+ }
+ )
+ return conflicts
+
+ prompt_members: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ for obj in copied:
+ prompt = sam3_query_key(obj)
+ if prompt and not is_host(obj):
+ prompt_members[prompt].append(obj)
+
+ # SAM3 commonly emits a high-confidence part mask beside a lower-score
+ # whole-instance mask. Complete an already assigned repeated instance only
+ # when the extra candidate is geometrically local, touches or contains the
+ # current mask, and contributes no unsafe pixels to another object.
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ sam3 = obj.get("sam3") or {}
+ prompt = sam3_query_key(obj)
+ if (
+ sam3.get("status") != "success"
+ or len(prompt_members.get(prompt, [])) < cfg.minimum_peer_count
+ or object_id not in existing_masks
+ ):
+ continue
+ target_box = valid_bbox(sam3.get("vlm_bbox_abs_xyxy"))
+ expanded_box = expanded_object_bbox(obj)
+ if target_box is None or expanded_box is None:
+ continue
+ base = existing_masks[object_id]
+ base_area = int(base.sum())
+ base_box = valid_bbox(sam3.get("mask_bbox_abs_xyxy"))
+ base_score = float(sam3.get("score") or 0.0)
+ if not base_area or base_box is None:
+ continue
+ viable = []
+ for candidate in candidates:
+ key = candidate_key(candidate)
+ if (
+ not key
+ or key in used_candidate_keys
+ or sam3_query_key(candidate) != prompt
+ ):
+ continue
+ score = float(candidate.get("score") or 0.0)
+ if score < cfg.minimum_completion_candidate_score:
+ continue
+ try:
+ raw_candidate_value = candidate_mask(candidate)
+ except (OSError, ValueError) as exc:
+ audit["hard_errors"].append(
+ {
+ "type": "invalid_completion_candidate",
+ "candidate_id": candidate.get("candidate_id"),
+ "error": str(exc),
+ }
+ )
+ continue
+ candidate_value, discarded_component_count = (
+ retain_completion_components_connected_to_base(
+ raw_candidate_value,
+ base,
+ dilation_pixels=cfg.adjacent_component_contact_dilation_pixels,
+ )
+ )
+ if discarded_component_count:
+ audit["filtered_unconnected_components"].append(
+ {
+ "object_id": object_id,
+ "candidate_id": candidate.get("candidate_id"),
+ "discarded_component_count": discarded_component_count,
+ }
+ )
+ candidate_box = bbox_from_mask(candidate_value)
+ if candidate_box is None:
+ continue
+ candidate_area = int(candidate_value.sum())
+ intersection = int(np.logical_and(base, candidate_value).sum())
+ if not candidate_area:
+ continue
+ existing_coverage = intersection / base_area
+ area_ratio = candidate_area / base_area
+ host, _ = shared_host(
+ [center(base_box), center(candidate_box)],
+ {object_id},
+ )
+ adjacent_same_object_component = (
+ is_pass_b_addition(obj)
+ and host is not None
+ and is_open_storage_host(host)
+ and score >= cfg.minimum_adjacent_component_score
+ and cfg.minimum_adjacent_component_area_ratio
+ <= area_ratio
+ <= cfg.maximum_adjacent_component_area_ratio
+ and contains(target_box, center(candidate_box))
+ and masks_touch(
+ base,
+ candidate_value,
+ dilation_pixels=cfg.adjacent_component_contact_dilation_pixels,
+ )
+ )
+ if not intersection and not adjacent_same_object_component:
+ continue
+ union_box = [
+ min(base_box[0], candidate_box[0]),
+ min(base_box[1], candidate_box[1]),
+ max(base_box[2], candidate_box[2]),
+ max(base_box[3], candidate_box[3]),
+ ]
+ union_bbox_coverage = bbox_coverage(expanded_box, union_box)
+ mode = None
+ if (
+ existing_coverage
+ >= cfg.minimum_superset_existing_coverage
+ and 1.05 <= area_ratio <= cfg.maximum_superset_area_ratio
+ and contains(expanded_box, center(candidate_box))
+ and union_bbox_coverage
+ >= cfg.minimum_superset_union_bbox_coverage
+ ):
+ mode = "safe_superset"
+ elif (
+ score >= cfg.minimum_touching_candidate_score
+ and score > base_score
+ and cfg.minimum_touching_candidate_area_ratio
+ <= area_ratio
+ <= cfg.maximum_touching_candidate_area_ratio
+ and contains(target_box, center(candidate_box))
+ and union_bbox_coverage
+ >= cfg.minimum_completion_union_bbox_coverage
+ ):
+ mode = "better_local_candidate"
+ elif (
+ adjacent_same_object_component
+ and union_bbox_coverage
+ >= cfg.minimum_completion_union_bbox_coverage
+ ):
+ mode = "adjacent_component_on_open_storage_host"
+ if mode is None:
+ continue
+ extra = np.logical_and(candidate_value, np.logical_not(base))
+ extra_area = int(extra.sum())
+ if extra_area < max(cfg.minimum_conflict_pixels, int(base_area * 0.05)):
+ continue
+ exempt_ids = (
+ {str(host.get("object_id") or "")}
+ if host is not None
+ else set()
+ )
+ conflicts = mask_conflicts(
+ extra,
+ object_id=object_id,
+ exempt_object_ids=exempt_ids,
+ maximum_coverage=cfg.maximum_completion_overlap_with_other,
+ )
+ if conflicts:
+ continue
+ viable.append(
+ {
+ "candidate": candidate,
+ "key": key,
+ "mask": candidate_value,
+ "mode": mode,
+ "score": score,
+ "area_ratio": area_ratio,
+ "previous_area": base_area,
+ "completed_mask": (
+ base | candidate_value
+ if mode == "adjacent_component_on_open_storage_host"
+ else candidate_value
+ ),
+ "completed_area": int(
+ (
+ base | candidate_value
+ if mode == "adjacent_component_on_open_storage_host"
+ else candidate_value
+ ).sum()
+ ),
+ "host_object_id": (
+ str(host.get("object_id") or "")
+ if host is not None
+ else None
+ ),
+ "discarded_unconnected_component_count": (
+ discarded_component_count
+ ),
+ }
+ )
+ if not viable:
+ continue
+ selected = max(
+ viable,
+ key=lambda item: (
+ item["mode"] == "safe_superset",
+ item["completed_area"],
+ item["score"],
+ str(item["candidate"].get("candidate_id") or ""),
+ ),
+ )
+ completed = selected["completed_mask"]
+ selected_keys = {selected["key"]}
+ extensions: list[dict[str, Any]] = []
+
+ # A collection can be represented by several adjacent SAM3 components
+ # (for example individual book spines). Once a safe host-local
+ # component is selected, extend it only through directly connected,
+ # target-local components. This is deliberately bounded and never
+ # crosses into another final object.
+ if selected["mode"] == "adjacent_component_on_open_storage_host":
+ for _ in range(max(0, cfg.maximum_adjacent_completion_components - 1)):
+ current_box = [
+ int(np.nonzero(completed)[1].min()),
+ int(np.nonzero(completed)[0].min()),
+ int(np.nonzero(completed)[1].max()) + 1,
+ int(np.nonzero(completed)[0].max()) + 1,
+ ]
+ extension_choices = []
+ for candidate in candidates:
+ key = candidate_key(candidate)
+ if (
+ not key
+ or key in used_candidate_keys
+ or key in selected_keys
+ or sam3_query_key(candidate) != prompt
+ ):
+ continue
+ score = float(candidate.get("score") or 0.0)
+ candidate_box = valid_bbox(candidate.get("mask_bbox_abs_xyxy"))
+ if (
+ score < cfg.minimum_adjacent_component_score
+ or candidate_box is None
+ or not contains(target_box, center(candidate_box))
+ ):
+ continue
+ try:
+ candidate_value = candidate_mask(candidate)
+ except (OSError, ValueError):
+ continue
+ candidate_area = int(candidate_value.sum())
+ area_ratio = candidate_area / base_area if base_area else 0.0
+ if not (
+ cfg.minimum_adjacent_component_area_ratio
+ <= area_ratio
+ <= cfg.maximum_adjacent_component_area_ratio
+ ):
+ continue
+ if not masks_touch(
+ completed,
+ candidate_value,
+ dilation_pixels=cfg.adjacent_component_contact_dilation_pixels,
+ ):
+ continue
+ host, _ = shared_host(
+ [center(current_box), center(candidate_box)],
+ {object_id},
+ )
+ if host is None or not is_open_storage_host(host):
+ continue
+ extra = np.logical_and(candidate_value, np.logical_not(completed))
+ extra_area = int(extra.sum())
+ if extra_area < max(cfg.minimum_conflict_pixels, int(base_area * 0.05)):
+ continue
+ conflicts = mask_conflicts(
+ extra,
+ object_id=object_id,
+ exempt_object_ids={str(host.get("object_id") or "")},
+ maximum_coverage=cfg.maximum_completion_overlap_with_other,
+ )
+ if conflicts:
+ continue
+ extension_choices.append(
+ {
+ "candidate": candidate,
+ "key": key,
+ "mask": candidate_value,
+ "score": score,
+ "extra_area": extra_area,
+ }
+ )
+ if not extension_choices:
+ break
+ extension = max(
+ extension_choices,
+ key=lambda item: (
+ item["extra_area"],
+ item["score"],
+ str(item["candidate"].get("candidate_id") or ""),
+ ),
+ )
+ completed |= extension["mask"]
+ selected_keys.add(extension["key"])
+ extensions.append(
+ {
+ "candidate_id": extension["candidate"].get("candidate_id"),
+ "candidate_score": extension["score"],
+ "added_area_pixels": extension["extra_area"],
+ }
+ )
+
+ completed_box = [
+ int(np.nonzero(completed)[1].min()),
+ int(np.nonzero(completed)[0].min()),
+ int(np.nonzero(completed)[1].max()) + 1,
+ int(np.nonzero(completed)[0].max()) + 1,
+ ]
+ mask_path, overlay_path = write_object_mask(obj, completed)
+ prior_sam3 = copy.deepcopy(dict(sam3))
+ obj["sam3"] = prior_sam3
+ obj["sam3"].update(
+ {
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ "area_pixels": int(completed.sum()),
+ "mask_bbox_abs_xyxy": completed_box,
+ "selection_method": "post_delta_same_prompt_mask_completion",
+ "multiplicity_completion": {
+ "candidate_id": selected["candidate"].get("candidate_id"),
+ "mode": selected["mode"],
+ "candidate_score": selected["score"],
+ "previous_area_pixels": selected["previous_area"],
+ "completed_area_pixels": int(completed.sum()),
+ "host_object_id": selected["host_object_id"],
+ "extension_candidates": extensions,
+ },
+ }
+ )
+ existing_masks[object_id] = completed
+ used_candidate_keys.update(selected_keys)
+ audit["completed"].append(
+ {
+ "object_id": object_id,
+ **obj["sam3"]["multiplicity_completion"],
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ }
+ )
+
+ # A cross-object containment decision may leave a real VLM instance
+ # without a mask even though a distinct same-prompt SAM3 candidate remains.
+ # Reassign only a peer-sized, target-local candidate on the same verified
+ # host and reject any candidate that conflicts with an existing object.
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ sam3 = obj.get("sam3") or {}
+ prompt = sam3_query_key(obj)
+ if (
+ sam3.get("status") == "success"
+ or len(prompt_members.get(prompt, [])) < cfg.minimum_peer_count
+ ):
+ continue
+ target_box = valid_bbox(sam3.get("vlm_bbox_abs_xyxy"))
+ peers = [
+ peer
+ for peer in prompt_members.get(prompt, [])
+ if (peer.get("sam3") or {}).get("status") == "success"
+ and str(peer.get("object_id") or "") in existing_masks
+ ]
+ peer_boxes = [
+ valid_bbox((peer.get("sam3") or {}).get("mask_bbox_abs_xyxy"))
+ for peer in peers
+ ]
+ peer_areas = [
+ int((peer.get("sam3") or {}).get("area_pixels") or 0)
+ for peer in peers
+ ]
+ if (
+ target_box is None
+ or len(peers) < cfg.minimum_peer_count
+ or any(box is None for box in peer_boxes)
+ or not all(peer_areas)
+ ):
+ continue
+ peer_boxes = [box for box in peer_boxes if box is not None]
+ median_area = float(np.median(peer_areas))
+ median_width = float(
+ np.median([box[2] - box[0] for box in peer_boxes])
+ )
+ median_height = float(
+ np.median([box[3] - box[1] for box in peer_boxes])
+ )
+ prompt_words = set(prompt.split())
+ host_not_required = (
+ "recessed" in prompt_words
+ and bool(prompt_words & {"light", "lamp"})
+ )
+ viable = []
+ for candidate in candidates:
+ key = candidate_key(candidate)
+ if (
+ not key
+ or key in used_candidate_keys
+ or sam3_query_key(candidate) != prompt
+ ):
+ continue
+ score = float(candidate.get("score") or 0.0)
+ candidate_box = valid_bbox(candidate.get("mask_bbox_abs_xyxy"))
+ area = int(candidate.get("area_pixels") or 0)
+ if (
+ score < cfg.minimum_failed_object_candidate_score
+ or candidate_box is None
+ or not area
+ or not contains(target_box, center(candidate_box))
+ ):
+ continue
+ area_ratio = area / median_area if median_area else 0.0
+ width_ratio = (
+ (candidate_box[2] - candidate_box[0]) / median_width
+ if median_width
+ else 0.0
+ )
+ height_ratio = (
+ (candidate_box[3] - candidate_box[1]) / median_height
+ if median_height
+ else 0.0
+ )
+ if not (
+ cfg.minimum_area_ratio
+ <= area_ratio
+ <= cfg.maximum_area_ratio
+ and cfg.minimum_dimension_ratio
+ <= width_ratio
+ <= cfg.maximum_dimension_ratio
+ and cfg.minimum_dimension_ratio
+ <= height_ratio
+ <= cfg.maximum_dimension_ratio
+ ):
+ continue
+ host, host_box = shared_host(
+ [center(box) for box in peer_boxes] + [center(candidate_box)],
+ {str(peer.get("object_id") or "") for peer in peers}
+ | {object_id},
+ )
+ if host is None and not host_not_required:
+ continue
+ try:
+ candidate_value = candidate_mask(candidate)
+ except (OSError, ValueError) as exc:
+ audit["hard_errors"].append(
+ {
+ "type": "invalid_failed_object_candidate",
+ "candidate_id": candidate.get("candidate_id"),
+ "error": str(exc),
+ }
+ )
+ continue
+ exempt_ids = (
+ {str(host.get("object_id") or "")}
+ if host is not None
+ else set()
+ )
+ conflicts = mask_conflicts(
+ candidate_value,
+ object_id=object_id,
+ exempt_object_ids=exempt_ids,
+ maximum_coverage=cfg.maximum_existing_candidate_coverage,
+ )
+ if conflicts:
+ continue
+ viable.append(
+ {
+ "candidate": candidate,
+ "key": key,
+ "mask": candidate_value,
+ "score": score,
+ "area_ratio": area_ratio,
+ "width_ratio": width_ratio,
+ "height_ratio": height_ratio,
+ "host": host,
+ "host_box": host_box,
+ }
+ )
+ if not viable:
+ continue
+ selected = max(
+ viable,
+ key=lambda item: (
+ -abs(1.0 - item["area_ratio"]),
+ -abs(1.0 - item["width_ratio"]),
+ -abs(1.0 - item["height_ratio"]),
+ item["score"],
+ str(item["candidate"].get("candidate_id") or ""),
+ ),
+ )
+ rescued = selected["mask"]
+ rescued_box = valid_bbox(
+ selected["candidate"].get("mask_bbox_abs_xyxy")
+ )
+ mask_path, overlay_path = write_object_mask(obj, rescued)
+ prior_failure = copy.deepcopy(dict(sam3))
+ obj["sam3"] = {
+ "status": "success",
+ "warning": None,
+ "prompt_used": selected["candidate"].get("sam3_prompt"),
+ "score": selected["score"],
+ "area_pixels": int(rescued.sum()),
+ "vlm_bbox_abs_xyxy": target_box,
+ "mask_bbox_abs_xyxy": rescued_box,
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ "selection_method": "post_delta_failed_object_reassignment",
+ "multiplicity_reassignment": {
+ "candidate_id": selected["candidate"].get("candidate_id"),
+ "previous_failure": prior_failure,
+ "host_object_id": (
+ str(selected["host"].get("object_id") or "")
+ if selected["host"] is not None
+ else None
+ ),
+ "host_bbox_abs_xyxy": selected["host_box"],
+ "area_ratio_to_peer_median": round(
+ selected["area_ratio"], 6
+ ),
+ "width_ratio_to_peer_median": round(
+ selected["width_ratio"], 6
+ ),
+ "height_ratio_to_peer_median": round(
+ selected["height_ratio"], 6
+ ),
+ },
+ }
+ existing_masks[object_id] = rescued
+ used_candidate_keys.add(selected["key"])
+ audit["rescued_failed"].append(
+ {
+ "object_id": object_id,
+ "candidate_id": selected["candidate"].get("candidate_id"),
+ "prompt": prompt,
+ "host_object_id": (
+ str(selected["host"].get("object_id") or "")
+ if selected["host"] is not None
+ else None
+ ),
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ }
+ )
+
+ groups: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ for obj in copied:
+ prompt = sam3_query_key(obj)
+ if prompt and not is_host(obj) and (obj.get("sam3") or {}).get("status") == "success":
+ groups[prompt].append(obj)
+
+ total_added = 0
+ for prompt, peers in sorted(groups.items()):
+ prompt_words = set(prompt.split())
+ single_peer_repeat = (
+ len(peers) == 1
+ and bool(prompt_words & SINGLE_PEER_REPEAT_WORDS)
+ )
+ single_peer_suspended_light = (
+ len(peers) == 1
+ and bool(prompt_words & SINGLE_PEER_SUSPENDED_LIGHT_WORDS)
+ and bool(prompt_words & {"light", "lamp"})
+ )
+ single_peer_exception = single_peer_repeat or single_peer_suspended_light
+ repeated_recessed_ceiling_light = (
+ len(peers) >= cfg.minimum_peer_count
+ and "recessed" in prompt_words
+ and bool(prompt_words & {"light", "lamp"})
+ )
+ host_not_required = (
+ single_peer_exception or repeated_recessed_ceiling_light
+ )
+ if len(peers) < cfg.minimum_peer_count and not single_peer_exception:
+ continue
+ peer_boxes = [valid_bbox((peer.get("sam3") or {}).get("mask_bbox_abs_xyxy")) for peer in peers]
+ peer_boxes = [box for box in peer_boxes if box]
+ peer_areas = [int((peer.get("sam3") or {}).get("area_pixels") or 0) for peer in peers]
+ if len(peer_boxes) != len(peers) or not all(peer_areas):
+ audit["rejected"].append({"prompt": prompt, "reason": "invalid_peer_geometry"})
+ continue
+ group_candidates = [
+ candidate for candidate in candidates
+ if candidate_key(candidate) not in used_candidate_keys
+ if sam3_query_key(candidate) == prompt
+ ]
+ audit["groups_checked"].append(
+ {"prompt": prompt, "peer_object_ids": [p.get("object_id") for p in peers], "candidate_count": len(group_candidates)}
+ )
+ for candidate in sorted(group_candidates, key=lambda item: float(item.get("score") or 0.0), reverse=True):
+ if total_added >= cfg.maximum_total_additions:
+ audit["rejected"].append({"candidate_id": candidate.get("candidate_id"), "reason": "total_budget_exceeded"})
+ continue
+ already_for_group = sum(1 for item in audit["accepted"] if item.get("prompt") == prompt)
+ if already_for_group >= cfg.maximum_additions_per_group:
+ audit["rejected"].append({"candidate_id": candidate.get("candidate_id"), "reason": "group_budget_exceeded"})
+ continue
+ score = float(candidate.get("score") or 0.0)
+ box = valid_bbox(candidate.get("mask_bbox_abs_xyxy"))
+ minimum_score = cfg.minimum_candidate_score
+ if single_peer_repeat:
+ minimum_score = cfg.minimum_single_peer_score
+ elif single_peer_suspended_light:
+ minimum_score = cfg.minimum_single_peer_suspended_light_score
+ elif repeated_recessed_ceiling_light:
+ minimum_score = cfg.minimum_recessed_ceiling_light_score
+ if score < minimum_score or not box:
+ audit["rejected"].append({"candidate_id": candidate.get("candidate_id"), "reason": "score_or_bbox_gate"})
+ continue
+ area = int(candidate.get("area_pixels") or 0)
+ median_area = float(np.median(peer_areas))
+ area_ratio = area / median_area if median_area else 0.0
+ widths = [b[2] - b[0] for b in peer_boxes]
+ heights = [b[3] - b[1] for b in peer_boxes]
+ width_ratio = (box[2] - box[0]) / float(np.median(widths))
+ height_ratio = (box[3] - box[1]) / float(np.median(heights))
+ minimum_area_ratio = cfg.minimum_area_ratio
+ minimum_dimension_ratio = cfg.minimum_dimension_ratio
+ if single_peer_repeat:
+ minimum_area_ratio = cfg.minimum_single_peer_area_ratio
+ minimum_dimension_ratio = cfg.minimum_single_peer_dimension_ratio
+ elif single_peer_suspended_light:
+ minimum_area_ratio = cfg.minimum_single_peer_suspended_light_area_ratio
+ minimum_dimension_ratio = cfg.minimum_single_peer_suspended_light_dimension_ratio
+ if not (
+ minimum_area_ratio <= area_ratio <= cfg.maximum_area_ratio
+ and minimum_dimension_ratio <= width_ratio <= cfg.maximum_dimension_ratio
+ and minimum_dimension_ratio <= height_ratio <= cfg.maximum_dimension_ratio
+ ):
+ audit["rejected"].append({"candidate_id": candidate.get("candidate_id"), "reason": "peer_geometry_gate", "area_ratio": area_ratio, "width_ratio": width_ratio, "height_ratio": height_ratio})
+ continue
+ if (
+ repeated_recessed_ceiling_light
+ and center(box)[1] / height
+ > cfg.maximum_recessed_ceiling_light_y_fraction
+ ):
+ audit["rejected"].append({
+ "candidate_id": candidate.get("candidate_id"),
+ "reason": "recessed_ceiling_light_outside_top_zone",
+ })
+ continue
+
+ # Repeated units may live on different shelves. The candidate only
+ # needs one verified local host; global peers provide the repeated
+ # category geometry and must not all fit into that same host.
+ host, host_box = shared_host(
+ [center(box)],
+ {str(peer.get("object_id") or "") for peer in peers},
+ )
+ if host is None and not host_not_required:
+ audit["rejected"].append(
+ {
+ "candidate_id": candidate.get("candidate_id"),
+ "reason": "no_local_verified_host",
+ }
+ )
+ continue
+ try:
+ mask, source_mask = load_mask(candidate.get("mask_path"), shape)
+ except (OSError, ValueError) as exc:
+ audit["hard_errors"].append({"type": "invalid_unassigned_candidate", "candidate_id": candidate.get("candidate_id"), "error": str(exc)})
+ continue
+ conflicts = []
+ exempt_ids = {str(host.get("object_id") or "")} if host else set()
+ for object_id, existing in existing_masks.items():
+ if object_id in exempt_ids:
+ continue
+ metrics = overlap(mask, existing)
+ if (
+ metrics["intersection_pixels"] >= cfg.minimum_conflict_pixels
+ and (
+ metrics["iou"] >= cfg.maximum_existing_iou
+ or metrics["candidate_coverage"] >= cfg.maximum_existing_candidate_coverage
+ )
+ ):
+ conflicts.append({"object_id": object_id, **metrics})
+ if conflicts:
+ audit["rejected"].append({"candidate_id": candidate.get("candidate_id"), "reason": "overlaps_existing_nonhost", "conflicts": conflicts})
+ continue
+
+ template = copy.deepcopy(peers[0])
+ object_id = next_object_id(copied)
+ category_slug = normalize(template.get("category")).replace(" ", "_") or "object"
+ staging = Path(staging_dir).expanduser().resolve()
+ mask_path = staging / "masks" / f"{object_id}_{category_slug}_mask.png"
+ overlay_path = staging / "overlays" / f"{object_id}_{category_slug}_overlay.png"
+ mask_path.parent.mkdir(parents=True, exist_ok=True)
+ overlay_path.parent.mkdir(parents=True, exist_ok=True)
+ Image.fromarray(mask.astype(np.uint8) * 255).save(mask_path)
+ overlay_image = image_np.copy()
+ overlay_image[mask] = (0.55 * overlay_image[mask] + 0.45 * np.array((255, 0, 0), dtype=np.uint8)).astype(np.uint8)
+ Image.fromarray(overlay_image).save(overlay_path)
+ cx, cy = center(box)
+ row = max(1, min(10, int(cy / height * 10) + 1))
+ col = max(1, min(10, int(cx / width * 10) + 1))
+ template.update({
+ "object_id": object_id,
+ "instance_name": f"{normalize(template.get('category')) or 'object'} recovered r{row}c{col}",
+ "bbox_2d_norm_xyxy": [box[0] / width, box[1] / height, box[2] / width, box[3] / height],
+ "anchor_2d_norm_xy": [cx / width, cy / height],
+ "anchor_source": "sam3_candidate_centroid",
+ "coarse_location": f"grid_r{row}_c{col}",
+ "visual_order": {"row": row, "col": col},
+ "proposal_sources": [{
+ "pass": "local",
+ "source_id": candidate.get("candidate_id"),
+ "action": "post_delta_multiplicity_closure",
+ "host_object_id": host.get("object_id") if host else None,
+ }],
+ "sam3": {
+ "status": "success",
+ "warning": None,
+ "prompt_used": candidate.get("sam3_prompt"),
+ "score": score,
+ "area_pixels": area,
+ "vlm_bbox_abs_xyxy": None,
+ "mask_bbox_abs_xyxy": box,
+ "mask_path": str(mask_path),
+ "overlay_path": str(overlay_path),
+ "selection_method": "post_delta_same_prompt_multiplicity_closure",
+ "multiplicity_closure": {
+ "candidate_id": candidate.get("candidate_id"),
+ "peer_object_ids": [peer.get("object_id") for peer in peers],
+ "host_object_id": host.get("object_id") if host else None,
+ "host_bbox_abs_xyxy": host_box,
+ "area_ratio_to_peer_median": round(area_ratio, 6),
+ "width_ratio_to_peer_median": round(width_ratio, 6),
+ "height_ratio_to_peer_median": round(height_ratio, 6),
+ },
+ },
+ })
+ copied.append(template)
+ existing_masks[object_id] = mask
+ total_added += 1
+ audit["accepted"].append({
+ "object_id": object_id,
+ "candidate_id": candidate.get("candidate_id"),
+ "prompt": prompt,
+ "peer_object_ids": [peer.get("object_id") for peer in peers],
+ "host_object_id": host.get("object_id") if host else None,
+ "mask_path": str(mask_path),
+ "overlay_path": str(overlay_path),
+ })
+
+ audit["accepted_count"] = len(audit["accepted"])
+ audit["completed_count"] = len(audit["completed"])
+ audit["rescued_failed_count"] = len(audit["rescued_failed"])
+ changed = bool(
+ audit["accepted"]
+ or audit["completed"]
+ or audit["rescued_failed"]
+ )
+ audit["status"] = (
+ "failed"
+ if audit["hard_errors"]
+ else ("pass" if changed else "noop")
+ )
+ atomic_json(audit_path, audit)
+ return {"objects": copied, "audit": audit, "audit_path": str(audit_path)}
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/receptacle_content_mask_completion.py b/vlm_agent/reconstruction/perception/pipeline/refinements/receptacle_content_mask_completion.py
new file mode 100644
index 0000000..08df68e
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/receptacle_content_mask_completion.py
@@ -0,0 +1,488 @@
+#!/usr/bin/env python3
+
+
+from __future__ import annotations
+
+import copy
+import json
+import math
+import os
+import re
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+from scipy import ndimage
+
+
+CONTAINER_TOKENS = {"bowl", "basket", "dish", "tray", "container"}
+FRUIT_TOKENS = {
+ "fruit", "fruits", "apple", "apples", "banana", "bananas",
+ "orange", "oranges", "pear", "pears", "grape", "grapes",
+ "lemon", "lemons", "peach", "peaches", "mango", "mangoes",
+ "berry", "berries", "strawberry", "strawberries", "kiwi",
+ "melon", "melons",
+}
+
+
+@dataclass(frozen=True)
+class ReceptacleContentMaskCompletionConfig:
+ maximum_targets: int = 2
+ maximum_candidates_per_target: int = 8
+ minimum_score: float = 0.45
+ minimum_area_pixels: int = 24
+ minimum_candidate_inside_fraction: float = 0.65
+ minimum_retained_fraction: float = 0.25
+ minimum_already_covered_fraction: float = 0.75
+ bbox_expansion_fraction: float = 0.08
+
+
+def _tokens(*values: Any) -> set[str]:
+ return {
+ token
+ for value in values
+ for token in re.sub(r"[^a-z0-9]+", " ", str(value or "").lower()).split()
+ if token
+ }
+
+
+def _valid_norm_bbox(value: Any) -> bool:
+ if not isinstance(value, Sequence) or isinstance(value, (str, bytes)):
+ return False
+ if len(value) != 4:
+ return False
+ try:
+ x1, y1, x2, y2 = (float(item) for item in value)
+ except (TypeError, ValueError):
+ return False
+ return 0.0 <= x1 < x2 <= 1.0 and 0.0 <= y1 < y2 <= 1.0
+
+
+def _expanded_bbox(bbox: Sequence[float], fraction: float) -> list[float]:
+ x1, y1, x2, y2 = (float(value) for value in bbox)
+ dx, dy = (x2 - x1) * fraction, (y2 - y1) * fraction
+ return [
+ max(0.0, x1 - dx), max(0.0, y1 - dy),
+ min(1.0, x2 + dx), min(1.0, y2 + dy),
+ ]
+
+
+def _center_in_bbox(candidate: Sequence[float], host: Sequence[float]) -> bool:
+ cx = (float(candidate[0]) + float(candidate[2])) / 2.0
+ cy = (float(candidate[1]) + float(candidate[3])) / 2.0
+ return float(host[0]) <= cx <= float(host[2]) and float(host[1]) <= cy <= float(host[3])
+
+
+def plan_receptacle_content_mask_completion(
+ pass_a_objects: Sequence[Mapping[str, Any]],
+ sam3_registry: Mapping[str, Any],
+ *,
+ config: ReceptacleContentMaskCompletionConfig | None = None,
+) -> list[dict[str, Any]]:
+ """Plan explicit missing fruit only; plants are deliberately out of scope."""
+
+ cfg = config or ReceptacleContentMaskCompletionConfig()
+ if cfg.maximum_targets < 0:
+ raise ValueError("maximum_targets must be non-negative")
+ registry_by_id = {
+ str(item.get("object_id") or ""): item
+ for item in sam3_registry.get("objects") or []
+ if isinstance(item, Mapping)
+ }
+ objects = [dict(item) for item in pass_a_objects if isinstance(item, Mapping)]
+ plans = []
+ for obj in objects:
+ object_id = str(obj.get("object_id") or "")
+ sam3 = (registry_by_id.get(object_id) or {}).get("sam3") or {}
+ bbox = obj.get("bbox_2d_norm_xyxy")
+ if (
+ not object_id
+ or str(sam3.get("status") or "") != "success"
+ or not _valid_norm_bbox(bbox)
+ or not (_tokens(obj.get("category"), obj.get("sam3_prompt")) & CONTAINER_TOKENS)
+ or not (
+ _tokens(
+ obj.get("description"),
+ obj.get("category"),
+ obj.get("sam3_prompt"),
+ obj.get("instance_name"),
+ )
+ & FRUIT_TOKENS
+ )
+ ):
+ continue
+ expanded = _expanded_bbox(bbox, cfg.bbox_expansion_fraction)
+ reliable_existing = False
+ for other in objects:
+ other_id = str(other.get("object_id") or "")
+ other_bbox = other.get("bbox_2d_norm_xyxy")
+ other_sam3 = (registry_by_id.get(other_id) or {}).get("sam3") or {}
+ if (
+ other_id != object_id
+ and _valid_norm_bbox(other_bbox)
+ and str(other_sam3.get("status") or "") == "success"
+ and _tokens(
+ other.get("category"),
+ other.get("sam3_prompt"),
+ other.get("instance_name"),
+ ) & FRUIT_TOKENS
+ and _center_in_bbox(other_bbox, expanded)
+ ):
+ reliable_existing = True
+ break
+ if reliable_existing:
+ continue
+ plans.append(
+ {
+ "target_source_object_id": object_id,
+ "target_category": str(obj.get("category") or ""),
+ "target_bbox_norm_xyxy": [float(value) for value in bbox],
+ "probe_bbox_norm_xyxy": expanded,
+ "prompt": "fruit",
+ "category_hint": "fruit",
+ "trigger_description": str(obj.get("description") or ""),
+ "target_mask_path": str(sam3.get("mask_path") or ""),
+ }
+ )
+ if len(plans) >= cfg.maximum_targets:
+ break
+ return plans
+
+
+def _load_mask(path: str | Path, expected_shape: tuple[int, int]) -> np.ndarray:
+ resolved = Path(path).expanduser().resolve()
+ if not resolved.is_file():
+ raise FileNotFoundError(resolved)
+ mask = np.array(Image.open(resolved).convert("L")) > 0
+ if mask.shape != expected_shape:
+ raise ValueError(f"mask shape mismatch: {mask.shape} != {expected_shape}")
+ return mask
+
+
+def _mask_bbox(mask: np.ndarray) -> list[int]:
+ ys, xs = np.nonzero(mask)
+ if not len(xs):
+ raise ValueError("empty mask")
+ return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
+
+
+def _save_mask_overlay(
+ image_np: np.ndarray,
+ mask: np.ndarray,
+ output_dir: Path,
+ stem: str,
+ *,
+ color: tuple[int, int, int],
+) -> tuple[str, str]:
+ masks_dir, overlays_dir = output_dir / "masks", output_dir / "overlays"
+ masks_dir.mkdir(parents=True, exist_ok=True)
+ overlays_dir.mkdir(parents=True, exist_ok=True)
+ mask_path = masks_dir / f"{stem}_mask.png"
+ overlay_path = overlays_dir / f"{stem}_overlay.png"
+ Image.fromarray(mask.astype(np.uint8) * 255).save(mask_path)
+ overlay = image_np.copy()
+ overlay_color = np.array(color, dtype=np.uint8)
+ overlay[mask] = (0.45 * overlay[mask] + 0.55 * overlay_color).astype(np.uint8)
+ Image.fromarray(overlay).save(overlay_path)
+ return str(mask_path), str(overlay_path)
+
+
+def _bbox_iou(left: Sequence[int], right: Sequence[int]) -> float:
+ x1, y1 = max(int(left[0]), int(right[0])), max(int(left[1]), int(right[1]))
+ x2, y2 = min(int(left[2]), int(right[2])), min(int(left[3]), int(right[3]))
+ intersection = max(0, x2 - x1) * max(0, y2 - y1)
+ left_area = max(0, int(left[2]) - int(left[0])) * max(0, int(left[3]) - int(left[1]))
+ right_area = max(0, int(right[2]) - int(right[0])) * max(0, int(right[3]) - int(right[1]))
+ union = left_area + right_area - intersection
+ return float(intersection / union) if union else 0.0
+
+
+def run_bounded_receptacle_content_mask_completion(
+ *,
+ session: Any,
+ infer_text_candidates_fn: Any,
+ image_path: str | Path,
+ pass_a_objects: Sequence[Mapping[str, Any]],
+ sam3_registry: Mapping[str, Any],
+ output_dir: str | Path,
+ config: ReceptacleContentMaskCompletionConfig | None = None,
+) -> dict[str, Any]:
+ """Run bounded fruit probes and complete target container registry masks."""
+
+ cfg = config or ReceptacleContentMaskCompletionConfig()
+ if not 0.0 <= cfg.minimum_score <= 1.0:
+ raise ValueError("minimum_score must be in [0, 1]")
+ output = Path(output_dir).expanduser().resolve()
+ if output.exists() and any(output.iterdir()):
+ raise FileExistsError(output)
+ output.mkdir(parents=True, exist_ok=True)
+ image = Image.open(image_path).convert("RGB")
+ image_np = np.array(image).astype(np.uint8)
+ width, height = image.size
+ shape = (height, width)
+ plans = plan_receptacle_content_mask_completion(pass_a_objects, sam3_registry, config=cfg)
+
+ successful_union = np.zeros(shape, dtype=bool)
+ mask_errors = []
+ for obj in sam3_registry.get("objects") or []:
+ sam3 = (obj or {}).get("sam3") or {}
+ if str(sam3.get("status") or "") != "success":
+ continue
+ try:
+ successful_union |= _load_mask(sam3.get("mask_path"), shape)
+ except Exception as exc:
+ mask_errors.append(
+ {"object_id": str((obj or {}).get("object_id") or ""),
+ "error": f"{type(exc).__name__}: {exc}"}
+ )
+ if mask_errors:
+ raise ValueError(f"cannot build successful-mask registry: {mask_errors}")
+
+ completed_registry = copy.deepcopy(sam3_registry)
+ completed_by_id = {
+ str(item.get("object_id") or ""): item
+ for item in completed_registry.get("objects") or []
+ if isinstance(item, Mapping)
+ }
+ completed_objects, probe_audits = [], []
+ for plan_index, plan in enumerate(plans, start=1):
+ bx = plan["probe_bbox_norm_xyxy"]
+ positive_bbox = [
+ float(bx[0]) * width, float(bx[1]) * height,
+ float(bx[2]) * width, float(bx[3]) * height,
+ ]
+ raw = infer_text_candidates_fn(
+ session,
+ image_path,
+ "fruit",
+ output / f"probe_{plan_index:03d}_fruit",
+ score_threshold=cfg.minimum_score,
+ min_area=cfg.minimum_area_pixels,
+ positive_bbox_abs_xyxy=positive_bbox,
+ maximum_candidates=cfg.maximum_candidates_per_target,
+ )
+ x1, y1 = max(0, int(math.floor(positive_bbox[0]))), max(0, int(math.floor(positive_bbox[1])))
+ x2, y2 = min(width, int(math.ceil(positive_bbox[2]))), min(height, int(math.ceil(positive_bbox[3])))
+ probe_region = np.zeros(shape, dtype=bool)
+ probe_region[y1:y2, x1:x2] = True
+ target_mask = _load_mask(plan["target_mask_path"], shape)
+ retained_masks, retained_records, rejected = [], [], []
+ already_covered_records = []
+ for record in raw["registry"].get("candidates") or []:
+ reasons = []
+ score = float(record.get("score") or 0.0)
+ source_mask = _load_mask(record.get("mask_path"), shape)
+ source_area = int(source_mask.sum())
+ inside_fraction = float((source_mask & probe_region).sum() / source_area) if source_area else 0.0
+ target_overlap_fraction = (
+ float((source_mask & target_mask).sum() / source_area)
+ if source_area else 0.0
+ )
+ retained = source_mask & probe_region & ~successful_union
+ retained_area = int(retained.sum())
+ retained_fraction = retained_area / source_area if source_area else 0.0
+ if score < cfg.minimum_score:
+ reasons.append("score_below_threshold")
+ if inside_fraction < cfg.minimum_candidate_inside_fraction:
+ reasons.append("insufficient_probe_bbox_coverage")
+ if retained_area < cfg.minimum_area_pixels:
+ reasons.append("retained_area_too_small")
+ if retained_fraction < cfg.minimum_retained_fraction:
+ reasons.append("mostly_existing_registry_pixels")
+ record_audit = {
+ "raw_candidate_id": str(record.get("candidate_id") or ""),
+ "score": score,
+ "area_pixels": source_area,
+ "inside_fraction": round(inside_fraction, 6),
+ "retained_fraction": round(retained_fraction, 6),
+ "target_mask_overlap_fraction": round(target_overlap_fraction, 6),
+ "source_mask_path": str(record.get("mask_path") or ""),
+ }
+ already_covered = (
+ score >= cfg.minimum_score
+ and source_area >= cfg.minimum_area_pixels
+ and inside_fraction >= cfg.minimum_candidate_inside_fraction
+ and target_overlap_fraction >= cfg.minimum_already_covered_fraction
+ )
+ if already_covered:
+ already_covered_records.append(record_audit)
+ elif reasons:
+ rejected.append({**record_audit, "reasons": reasons})
+ else:
+ retained_masks.append(retained)
+ retained_records.append(record_audit)
+
+ grouped_component_count = 0
+ grouped = np.logical_or.reduce(retained_masks) if retained_masks else np.zeros(shape, dtype=bool)
+ if grouped.any():
+ labels, count = ndimage.label(grouped, structure=np.ones((3, 3), dtype=bool))
+ filtered = np.zeros(shape, dtype=bool)
+ for label_id in range(1, count + 1):
+ component = labels == label_id
+ if int(component.sum()) >= cfg.minimum_area_pixels:
+ filtered |= component
+ grouped_component_count += 1
+ grouped = filtered
+
+ fruit_evidence_id = f"FRUIT_{plan_index:03d}"
+ completion = None
+ if grouped.any():
+ fruit_bbox = _mask_bbox(grouped)
+ fruit_mask_path, fruit_overlay_path = _save_mask_overlay(
+ image_np, grouped, output / "fruit_evidence",
+ fruit_evidence_id, color=(255, 75, 35),
+ )
+ source_id = plan["target_source_object_id"]
+ target = completed_by_id.get(source_id)
+ if target is None:
+ raise KeyError(f"planned container missing from registry: {source_id}")
+ target_sam3 = dict(target.get("sam3") or {})
+ original_mask_path = str(target_sam3.get("mask_path") or "")
+ original_mask = _load_mask(original_mask_path, shape)
+ completed_mask = original_mask | grouped
+ completed_bbox = _mask_bbox(completed_mask)
+ completed_mask_path, completed_overlay_path = _save_mask_overlay(
+ image_np, completed_mask, output / "completed_containers",
+ f"{source_id}_fruit_container_complete", color=(50, 220, 95),
+ )
+ completion = {
+ "policy": "fruit_only_merge_into_container",
+ "source_object_id": source_id,
+ "original_mask_path": original_mask_path,
+ "original_area_pixels": int(original_mask.sum()),
+ "fruit_evidence_id": fruit_evidence_id,
+ "fruit_evidence_mask_path": fruit_mask_path,
+ "fruit_evidence_overlay_path": fruit_overlay_path,
+ "fruit_area_pixels": int(grouped.sum()),
+ "fruit_mask_bbox_abs_xyxy": fruit_bbox,
+ "grouped_component_count": grouped_component_count,
+ "completed_mask_path": completed_mask_path,
+ "completed_overlay_path": completed_overlay_path,
+ "completed_area_pixels": int(completed_mask.sum()),
+ "completed_mask_bbox_abs_xyxy": completed_bbox,
+ "raw_candidate_ids": [item["raw_candidate_id"] for item in retained_records],
+ }
+ target_sam3.update(
+ {
+ "mask_path": completed_mask_path,
+ "overlay_path": completed_overlay_path,
+ "area_pixels": int(completed_mask.sum()),
+ "mask_bbox_abs_xyxy": completed_bbox,
+ "container_content_completion": completion,
+ }
+ )
+ vlm_bbox = target_sam3.get("vlm_bbox_abs_xyxy")
+ if isinstance(vlm_bbox, Sequence) and len(vlm_bbox) == 4:
+ target_sam3["vlm_bbox_iou"] = _bbox_iou(vlm_bbox, completed_bbox)
+ target["sam3"] = target_sam3
+ completed_objects.append(completion)
+ elif already_covered_records:
+ source_id = plan["target_source_object_id"]
+ target = completed_by_id.get(source_id)
+ if target is None:
+ raise KeyError(f"planned container missing from registry: {source_id}")
+ target_sam3 = dict(target.get("sam3") or {})
+ original_mask_path = str(target_sam3.get("mask_path") or "")
+ original_mask = _load_mask(original_mask_path, shape)
+ best = max(
+ already_covered_records,
+ key=lambda item: (
+ float(item["target_mask_overlap_fraction"]),
+ float(item["score"]),
+ ),
+ )
+ evidence_mask_path = str(best["source_mask_path"])
+ completion = {
+ "policy": "fruit_already_in_target_container_mask",
+ "source_object_id": source_id,
+ "original_mask_path": original_mask_path,
+ "original_area_pixels": int(original_mask.sum()),
+ "fruit_evidence_id": fruit_evidence_id,
+ "fruit_evidence_mask_path": evidence_mask_path,
+ "fruit_area_pixels": int(best["area_pixels"]),
+ "target_mask_overlap_fraction": float(
+ best["target_mask_overlap_fraction"]
+ ),
+ "completed_mask_path": original_mask_path,
+ "completed_area_pixels": int(original_mask.sum()),
+ "completed_mask_bbox_abs_xyxy": _mask_bbox(original_mask),
+ "raw_candidate_ids": [best["raw_candidate_id"]],
+ "mask_write_required": False,
+ }
+ target_sam3["container_content_completion"] = completion
+ target["sam3"] = target_sam3
+ completed_objects.append(completion)
+
+ probe_audits.append(
+ {
+ **plan,
+ "raw_registry": raw.get("registry_path"),
+ "raw_candidate_count": int(raw["registry"].get("candidate_count") or 0),
+ "retained_raw_candidates": retained_records,
+ "already_covered_raw_candidates": already_covered_records,
+ "rejected_raw_candidates": rejected,
+ "grouped_component_count": grouped_component_count,
+ "fruit_evidence_id": fruit_evidence_id if completion else None,
+ "container_completed": completion is not None,
+ "completion_mode": (
+ completion.get("policy") if completion is not None else None
+ ),
+ }
+ )
+
+ completed_registry_path = output / "completed_sam3_registry.json"
+ registry_temporary = completed_registry_path.with_suffix(".json.tmp")
+ registry_temporary.write_text(
+ json.dumps(completed_registry, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(registry_temporary, completed_registry_path)
+ output_files = [path for path in output.rglob("*") if path.is_file()]
+ audit = {
+ "schema_version": "scenemosaic_bounded_fruit_container_completion",
+ "status": (
+ "not_applicable" if not plans else
+ "containers_completed" if len(completed_objects) == len(plans) else
+ "unresolved"
+ ),
+ "config": asdict(cfg),
+ "plan_count": len(plans),
+ "plans": plans,
+ "probe_count": len(probe_audits),
+ "probes": probe_audits,
+ "completed_count": len(completed_objects),
+ "completed_source_object_ids": [item["source_object_id"] for item in completed_objects],
+ "completed_objects": completed_objects,
+ "completed_registry_path": str(completed_registry_path),
+ "policy": {
+ "content_type": "fruit_only",
+ "preferred_resolution": "merge_into_container_if_missing",
+ "preserve_reliable_independent_fruit": True,
+ "plants_in_scope": False,
+ },
+ "additional_model_load_count": 0,
+ "additional_vlm_calls": 0,
+ "storage_bytes_before_audit": sum(path.stat().st_size for path in output_files),
+ "formal_outputs_modified": False,
+ "hard_errors": [],
+ }
+ audit_path = output / "container_content_candidate_recall_audit.json"
+ temporary = audit_path.with_suffix(".json.tmp")
+ temporary.write_text(json.dumps(audit, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ os.replace(temporary, audit_path)
+ return {
+ "completed_registry": completed_registry,
+ "completed_registry_path": str(completed_registry_path),
+ "completed_objects": completed_objects,
+ "audit": audit,
+ "audit_path": str(audit_path),
+ }
+
+
+__all__ = [
+ "ReceptacleContentMaskCompletionConfig",
+ "plan_receptacle_content_mask_completion",
+ "run_bounded_receptacle_content_mask_completion",
+]
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/repair_scope_router.py b/vlm_agent/reconstruction/perception/pipeline/refinements/repair_scope_router.py
new file mode 100644
index 0000000..4604722
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/repair_scope_router.py
@@ -0,0 +1,151 @@
+#!/usr/bin/env python3
+"""Conservative semantic routing for segmentation refinement actions."""
+
+from __future__ import annotations
+
+import re
+from typing import Any, Mapping, Sequence
+
+
+OPEN_STORAGE_ROLE = "open_storage_interior"
+SUPPORT_SURFACE_ROLE = "support_surface"
+ASSEMBLY_ROLE = "assembly_or_fixture"
+ORDINARY_ROLE = "ordinary_object"
+AMBIGUOUS_ROLE = "ambiguous"
+
+OPEN_STORAGE_TOKENS = {
+ "bookcase", "bookshelf", "cubby", "hutch", "rack", "shelf", "shelves",
+ "shelving",
+}
+SUPPORT_SURFACE_TOKENS = {
+ "console", "counter", "countertop", "desk", "dresser", "island",
+ "nightstand", "sideboard", "table", "vanity", "workbench",
+}
+OPEN_QUALIFIERS = {"display", "open", "rack", "shelf", "shelves", "shelving", "tier", "tiers"}
+CART_STORAGE_QUALIFIERS = OPEN_QUALIFIERS | {"storage"}
+ASSEMBLY_TOKENS = {
+ "bathtub", "bed", "faucet", "lamp", "light", "machine", "sink", "vanity",
+}
+DEDICATED_ASSEMBLY_TOKEN_SETS = (
+ {"coat", "rack"},
+ {"towel", "rack"},
+ {"towel", "ladder"},
+ {"drying", "rack"},
+ {"fabric", "roll", "rack"},
+ {"roll", "holder"},
+ {"pegboard", "rack"},
+)
+NON_HOST_TYPES = {"appliance", "lighting", "opening", "plant", "small_object", "textile"}
+
+
+def _tokens(value: Any) -> list[str]:
+ return re.findall(r"[a-z0-9]+", str(value or "").lower().replace("_", " "))
+
+
+def _fields(obj: Mapping[str, Any]) -> list[list[str]]:
+ return [
+ _tokens(obj.get(key))
+ for key in ("category", "sam3_prompt", "instance_name")
+ ]
+
+
+def _field_vote(tokens: Sequence[str]) -> str | None:
+ token_set = set(tokens)
+ if token_set & SUPPORT_SURFACE_TOKENS:
+ return SUPPORT_SURFACE_ROLE
+ # A storage/tier/shelf cart is a small mobile open carrier. A plain
+ # utility/serving cart remains a support surface.
+ if "cart" in token_set:
+ return (
+ OPEN_STORAGE_ROLE
+ if token_set & CART_STORAGE_QUALIFIERS
+ else SUPPORT_SURFACE_ROLE
+ )
+ if token_set & OPEN_STORAGE_TOKENS:
+ return OPEN_STORAGE_ROLE
+ if "cabinet" in token_set:
+ return (
+ OPEN_STORAGE_ROLE
+ if token_set & {"display", "open", "shelf", "shelves", "shelving"}
+ else SUPPORT_SURFACE_ROLE
+ )
+ return None
+
+
+def classify_object_role(obj: Mapping[str, Any]) -> dict[str, Any]:
+ """Return a fail-closed role audit for one existing object."""
+
+ fields = _fields(obj)
+ votes = [_field_vote(tokens) for tokens in fields]
+ object_type = "_".join(_tokens(obj.get("object_type"))) or "unknown"
+ category_tokens = set(fields[0])
+ all_tokens = set().union(*(set(tokens) for tokens in fields))
+ dedicated_assembly = any(
+ required.issubset(all_tokens)
+ for required in DEDICATED_ASSEMBLY_TOKEN_SETS
+ )
+ open_votes = votes.count(OPEN_STORAGE_ROLE)
+ support_votes = votes.count(SUPPORT_SURFACE_ROLE)
+
+ if object_type in NON_HOST_TYPES:
+ role = ORDINARY_ROLE
+ confidence = 0.98
+ reason = "semantic_type_is_not_a_furniture_host"
+ elif dedicated_assembly:
+ role = ASSEMBLY_ROLE
+ confidence = 0.96
+ reason = "dedicated_holder_and_attached_parts_form_one_assembly"
+ elif "cart" in category_tokens and category_tokens & CART_STORAGE_QUALIFIERS:
+ role = OPEN_STORAGE_ROLE
+ confidence = 0.96
+ reason = "explicit_storage_cart_category"
+ elif category_tokens & SUPPORT_SURFACE_TOKENS:
+ role = SUPPORT_SURFACE_ROLE
+ confidence = 0.98 if support_votes >= 2 else 0.90
+ reason = "support_surface_category_has_priority_over_modifier_text"
+ elif category_tokens & OPEN_STORAGE_TOKENS:
+ role = OPEN_STORAGE_ROLE
+ confidence = 0.98 if open_votes >= 2 else 0.90
+ reason = "explicit_open_storage_category"
+ elif open_votes >= 2 and support_votes == 0:
+ role = OPEN_STORAGE_ROLE
+ confidence = 0.92
+ reason = "two_semantic_fields_agree_on_open_storage"
+ elif support_votes >= 2 and open_votes == 0:
+ role = SUPPORT_SURFACE_ROLE
+ confidence = 0.92
+ reason = "two_semantic_fields_agree_on_support_surface"
+ elif open_votes and support_votes:
+ role = AMBIGUOUS_ROLE
+ confidence = 0.50
+ reason = "open_storage_and_support_surface_fields_conflict"
+ elif open_votes == 1 or support_votes == 1:
+ role = AMBIGUOUS_ROLE
+ confidence = 0.45
+ reason = "only_one_semantic_field_supports_a_host_role"
+ elif category_tokens & ASSEMBLY_TOKENS:
+ role = ASSEMBLY_ROLE
+ confidence = 0.80
+ reason = "category_requires_assembly_completeness_policy"
+ else:
+ role = ORDINARY_ROLE
+ confidence = 0.80
+ reason = "no_supported_host_semantics"
+
+ host_mode = {
+ OPEN_STORAGE_ROLE: "open_storage",
+ SUPPORT_SURFACE_ROLE: "support_surface",
+ }.get(role)
+ return {
+ "object_role": role,
+ "host_mode": host_mode,
+ "confidence": round(confidence, 6),
+ "reason": reason,
+ "field_votes": votes,
+ "open_storage_votes": open_votes,
+ "support_surface_votes": support_votes,
+ "object_type": object_type,
+ "host_minus_contents_allowed": role == OPEN_STORAGE_ROLE,
+ "support_surface_review_allowed": role == SUPPORT_SURFACE_ROLE,
+ "semantic_verification_required": role == AMBIGUOUS_ROLE,
+ }
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/strict_same_mask_duplicate_collapse.py b/vlm_agent/reconstruction/perception/pipeline/refinements/strict_same_mask_duplicate_collapse.py
new file mode 100644
index 0000000..ce37224
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/strict_same_mask_duplicate_collapse.py
@@ -0,0 +1,413 @@
+#!/usr/bin/env python3
+"""Fail-closed removal of a Pass-B addition proved to duplicate an object."""
+
+from __future__ import annotations
+
+import copy
+import json
+import os
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+
+@dataclass(frozen=True)
+class StrictSameMaskDuplicateCollapseConfig:
+ enabled: bool = True
+ isolated_attempt_only: bool = True
+ minimum_intersection_pixels: int = 30
+ minimum_iou: float = 0.995
+ minimum_coverage: float = 0.995
+ minimum_aggregate_member_count: int = 2
+ minimum_aggregate_coverage_by_members: float = 0.85
+ minimum_member_coverage_by_aggregate: float = 0.95
+ # Cross-category Pass-B additions need slightly more tolerance than the
+ # same-prompt retry path: prompt-specific SAM3 masks can differ by a thin
+ # boundary while still referring to exactly the same existing instance.
+ minimum_pass_b_preexisting_iou: float = 0.99
+ minimum_pass_b_preexisting_coverage: float = 0.995
+
+
+def _atomic_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _category_key(obj: Mapping[str, Any]) -> str:
+ return " ".join(
+ re.findall(r"[a-z0-9]+", str(obj.get("category") or "").lower())
+ )
+
+
+def _category_tokens(obj: Mapping[str, Any]) -> set[str]:
+ return set(re.findall(r"[a-z0-9]+", str(obj.get("category") or "").lower()))
+
+
+def _aggregate_member_tokens(obj: Mapping[str, Any]) -> set[str]:
+ aggregate_markers = {"bundle", "cluster", "collection", "group", "pair", "set"}
+ return _category_tokens(obj) - aggregate_markers
+
+
+def _pass_b_only_addition(obj: Mapping[str, Any]) -> bool:
+ sources = obj.get("proposal_sources") or []
+ if not isinstance(sources, list) or not sources:
+ return False
+ return all(
+ isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "B"
+ and str(source.get("action") or "").lower() == "added"
+ for source in sources
+ )
+
+
+def _load_mask(path_value: Any, expected_shape: tuple[int, int]) -> np.ndarray:
+ path = Path(str(path_value or "")).expanduser().resolve()
+ if not path.is_file():
+ raise FileNotFoundError(path)
+ mask = np.array(Image.open(path).convert("L")) > 0
+ if mask.shape != expected_shape or not mask.any():
+ raise ValueError("invalid_mask_shape_or_empty")
+ return mask
+
+
+def _mask_metrics(left: np.ndarray, right: np.ndarray) -> dict[str, float | int]:
+ intersection = int(np.logical_and(left, right).sum())
+ left_area = int(left.sum())
+ right_area = int(right.sum())
+ union = left_area + right_area - intersection
+ return {
+ "intersection_pixels": intersection,
+ "iou": intersection / max(union, 1),
+ "left_coverage": intersection / max(left_area, 1),
+ "right_coverage": intersection / max(right_area, 1),
+ }
+
+
+def _retry_object_by_id(
+ retry_registry: Mapping[str, Any] | None,
+) -> dict[str, Mapping[str, Any]]:
+ objects = (retry_registry or {}).get("objects") or []
+ return {
+ str(obj.get("object_id") or ""): obj
+ for obj in objects
+ if isinstance(obj, Mapping) and str(obj.get("object_id") or "")
+ }
+
+
+def _remap_exact_proposal_ids(value: Any, mapping: Mapping[str, str]) -> Any:
+ """Recursively rewrite exact ID values, never path substrings."""
+
+ if isinstance(value, str):
+ return mapping.get(value, value)
+ if isinstance(value, list):
+ return [_remap_exact_proposal_ids(item, mapping) for item in value]
+ if isinstance(value, Mapping):
+ return {
+ key: _remap_exact_proposal_ids(item, mapping)
+ for key, item in value.items()
+ }
+ return value
+
+
+def run_strict_same_mask_duplicate_collapse(
+ *,
+ objects: Sequence[Mapping[str, Any]],
+ retry_registry: Mapping[str, Any] | None,
+ retry_audit: Mapping[str, Any] | None,
+ image_shape: tuple[int, int],
+ output_dir: str | Path,
+ config: StrictSameMaskDuplicateCollapseConfig | None = None,
+) -> dict[str, Any]:
+ """Return copied objects and a strict audit without modifying source data."""
+
+ cfg = config or StrictSameMaskDuplicateCollapseConfig()
+ if not cfg.isolated_attempt_only:
+ raise ValueError("strict same-mask duplicate collapse is isolated-attempt only")
+ work = Path(output_dir).expanduser().resolve()
+ work.mkdir(parents=True, exist_ok=True)
+ copied = [copy.deepcopy(dict(obj)) for obj in objects if isinstance(obj, Mapping)]
+ audit: dict[str, Any] = {
+ "schema_version": "scenemosaic_strict_same_mask_duplicate_collapse",
+ "enabled": cfg.enabled,
+ "isolated_attempt_only": cfg.isolated_attempt_only,
+ "status": "disabled" if not cfg.enabled else "noop",
+ "input_object_count": len(copied),
+ "collapsed": [],
+ "rejected": [],
+ "hard_errors": [],
+ "thresholds": {
+ "minimum_intersection_pixels": cfg.minimum_intersection_pixels,
+ "minimum_iou": cfg.minimum_iou,
+ "minimum_coverage": cfg.minimum_coverage,
+ "minimum_aggregate_member_count": cfg.minimum_aggregate_member_count,
+ "minimum_aggregate_coverage_by_members": (
+ cfg.minimum_aggregate_coverage_by_members
+ ),
+ "minimum_member_coverage_by_aggregate": (
+ cfg.minimum_member_coverage_by_aggregate
+ ),
+ "minimum_pass_b_preexisting_iou": (
+ cfg.minimum_pass_b_preexisting_iou
+ ),
+ "minimum_pass_b_preexisting_coverage": (
+ cfg.minimum_pass_b_preexisting_coverage
+ ),
+ },
+ }
+ if not cfg.enabled:
+ _atomic_json(work / "strict_same_mask_duplicate_collapse_audit.json", audit)
+ return {"objects": copied, "audit": audit}
+
+ retry_by_id = _retry_object_by_id(retry_registry)
+ rejected_by_id = {
+ str(item.get("object_id") or ""): str(item.get("reason") or "")
+ for item in ((retry_audit or {}).get("rejected") or [])
+ if isinstance(item, Mapping)
+ }
+ success_masks: dict[str, np.ndarray] = {}
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "success":
+ continue
+ try:
+ success_masks[object_id] = _load_mask(sam3.get("mask_path"), image_shape)
+ except (OSError, ValueError) as exc:
+ audit["hard_errors"].append(
+ {
+ "type": "invalid_existing_success_mask_for_duplicate_collapse",
+ "object_id": object_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+
+ removal_ids: set[str] = set()
+ for obj in copied:
+ object_id = str(obj.get("object_id") or "")
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "failed" or not _pass_b_only_addition(obj):
+ continue
+ if rejected_by_id.get(object_id) != "mask_overlap_ambiguity":
+ continue
+ retry_obj = retry_by_id.get(object_id)
+ retry_sam3 = (retry_obj or {}).get("sam3") or {}
+ if retry_sam3.get("status") != "success":
+ continue
+ try:
+ retry_mask = _load_mask(retry_sam3.get("mask_path"), image_shape)
+ except (OSError, ValueError) as exc:
+ audit["hard_errors"].append(
+ {
+ "type": "invalid_retry_mask_for_duplicate_collapse",
+ "object_id": object_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ continue
+ category = _category_key(obj)
+ best: tuple[str, dict[str, float | int]] | None = None
+ for existing in copied:
+ existing_id = str(existing.get("object_id") or "")
+ if (
+ existing_id == object_id
+ or existing_id not in success_masks
+ or _category_key(existing) != category
+ ):
+ continue
+ metrics = _mask_metrics(retry_mask, success_masks[existing_id])
+ if best is None or float(metrics["iou"]) > float(best[1]["iou"]):
+ best = (existing_id, metrics)
+ if best is None:
+ audit["rejected"].append(
+ {"object_id": object_id, "reason": "no_same_category_success"}
+ )
+ continue
+ kept_id, metrics = best
+ proven = (
+ int(metrics["intersection_pixels"]) >= cfg.minimum_intersection_pixels
+ and float(metrics["iou"]) >= cfg.minimum_iou
+ and float(metrics["left_coverage"]) >= cfg.minimum_coverage
+ and float(metrics["right_coverage"]) >= cfg.minimum_coverage
+ )
+ if not proven:
+ audit["rejected"].append(
+ {
+ "object_id": object_id,
+ "reason": "same_category_geometry_below_strict_threshold",
+ "kept_object_id": kept_id,
+ **{key: round(float(value), 6) for key, value in metrics.items()},
+ }
+ )
+ continue
+ removal_ids.add(object_id)
+ audit["collapsed"].append(
+ {
+ "removed_object_id": object_id,
+ "kept_object_id": kept_id,
+ "category": category,
+ "source_policy": "pass_b_only_addition_with_category_retry_exact_mask",
+ **{key: round(float(value), 6) for key, value in metrics.items()},
+ }
+ )
+
+ # A successful Pass-B-only addition can also duplicate an already
+ # materialized Pass-A object. If their masks are effectively identical,
+ # provenance provides a deterministic owner even when the VLM assigned
+ # incompatible category names: retain the pre-existing object and remove
+ # the later addition. This is stricter than semantic matching and never
+ # chooses between two independently established objects.
+ for addition in copied:
+ addition_id = str(addition.get("object_id") or "")
+ if (
+ addition_id in removal_ids
+ or addition_id not in success_masks
+ or not _pass_b_only_addition(addition)
+ ):
+ continue
+ best: tuple[str, dict[str, float | int]] | None = None
+ for existing in copied:
+ existing_id = str(existing.get("object_id") or "")
+ if (
+ existing_id == addition_id
+ or existing_id in removal_ids
+ or existing_id not in success_masks
+ or _pass_b_only_addition(existing)
+ ):
+ continue
+ metrics = _mask_metrics(
+ success_masks[addition_id], success_masks[existing_id]
+ )
+ if best is None or float(metrics["iou"]) > float(best[1]["iou"]):
+ best = (existing_id, metrics)
+ if best is None:
+ continue
+ kept_id, metrics = best
+ proven = (
+ int(metrics["intersection_pixels"]) >= cfg.minimum_intersection_pixels
+ and float(metrics["iou"]) >= cfg.minimum_pass_b_preexisting_iou
+ and float(metrics["left_coverage"])
+ >= cfg.minimum_pass_b_preexisting_coverage
+ and float(metrics["right_coverage"])
+ >= cfg.minimum_pass_b_preexisting_coverage
+ )
+ if not proven:
+ continue
+ removal_ids.add(addition_id)
+ audit["collapsed"].append(
+ {
+ "removed_object_id": addition_id,
+ "kept_object_id": kept_id,
+ "removed_category": _category_key(addition),
+ "kept_category": _category_key(
+ next(
+ item
+ for item in copied
+ if str(item.get("object_id") or "") == kept_id
+ )
+ ),
+ "source_policy": (
+ "pass_b_only_addition_exact_mask_prefer_preexisting_object"
+ ),
+ **{
+ key: round(float(value), 6)
+ for key, value in metrics.items()
+ },
+ }
+ )
+
+ aggregate_markers = {"bundle", "cluster", "collection", "group", "pair", "set"}
+ for aggregate in copied:
+ aggregate_id = str(aggregate.get("object_id") or "")
+ aggregate_tokens = _category_tokens(aggregate)
+ member_tokens = _aggregate_member_tokens(aggregate)
+ if (
+ aggregate_id in removal_ids
+ or aggregate_id not in success_masks
+ or not (aggregate_tokens & aggregate_markers)
+ or not member_tokens
+ ):
+ continue
+ aggregate_mask = success_masks[aggregate_id]
+ members = []
+ for candidate in copied:
+ candidate_id = str(candidate.get("object_id") or "")
+ if (
+ candidate_id == aggregate_id
+ or candidate_id in removal_ids
+ or candidate_id not in success_masks
+ or not (_category_tokens(candidate) & member_tokens)
+ or bool(_category_tokens(candidate) & aggregate_markers)
+ ):
+ continue
+ metrics = _mask_metrics(success_masks[candidate_id], aggregate_mask)
+ if (
+ int(metrics["intersection_pixels"]) >= cfg.minimum_intersection_pixels
+ and float(metrics["left_coverage"])
+ >= cfg.minimum_member_coverage_by_aggregate
+ ):
+ members.append((candidate_id, metrics))
+ if len(members) < cfg.minimum_aggregate_member_count:
+ continue
+ member_union = np.logical_or.reduce(
+ [success_masks[candidate_id] for candidate_id, _ in members]
+ )
+ aggregate_coverage = float(
+ np.logical_and(aggregate_mask, member_union).sum()
+ / max(int(aggregate_mask.sum()), 1)
+ )
+ if aggregate_coverage < cfg.minimum_aggregate_coverage_by_members:
+ continue
+ removal_ids.add(aggregate_id)
+ audit["collapsed"].append(
+ {
+ "removed_object_id": aggregate_id,
+ "kept_object_ids": [candidate_id for candidate_id, _ in members],
+ "category": _category_key(aggregate),
+ "source_policy": "aggregate_mask_explained_by_individual_members",
+ "aggregate_coverage_by_members": round(aggregate_coverage, 6),
+ "member_coverages_by_aggregate": {
+ candidate_id: round(float(metrics["left_coverage"]), 6)
+ for candidate_id, metrics in members
+ },
+ }
+ )
+
+ result = [obj for obj in copied if str(obj.get("object_id") or "") not in removal_ids]
+ object_id_map = {
+ str(obj.get("object_id") or ""): f"proposal_{index:03d}"
+ for index, obj in enumerate(result, start=1)
+ }
+ result = [
+ _remap_exact_proposal_ids(obj, object_id_map)
+ for obj in result
+ ]
+ audit["object_id_map"] = [
+ {"old_object_id": old_id, "new_object_id": new_id}
+ for old_id, new_id in object_id_map.items()
+ if old_id != new_id
+ ]
+ for record in audit["collapsed"]:
+ if record.get("kept_object_id") is not None:
+ kept_old_id = str(record.get("kept_object_id") or "")
+ record["kept_final_object_id"] = object_id_map.get(
+ kept_old_id, kept_old_id
+ )
+ else:
+ record["kept_final_object_ids"] = [
+ object_id_map.get(str(value), str(value))
+ for value in record.get("kept_object_ids") or []
+ ]
+ audit["output_object_count"] = len(result)
+ audit["collapsed_count"] = len(audit["collapsed"])
+ audit["status"] = "fail" if audit["hard_errors"] else ("pass" if removal_ids else "noop")
+ _atomic_json(work / "strict_same_mask_duplicate_collapse_audit.json", audit)
+ return {"objects": result, "audit": audit}
diff --git a/vlm_agent/reconstruction/perception/pipeline/refinements/top_zone_candidate_recall.py b/vlm_agent/reconstruction/perception/pipeline/refinements/top_zone_candidate_recall.py
new file mode 100644
index 0000000..200bbab
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/refinements/top_zone_candidate_recall.py
@@ -0,0 +1,316 @@
+#!/usr/bin/env python3
+"""Filter raw SAM3 text masks into a bounded, numbered top-zone review set."""
+
+from __future__ import annotations
+
+import json
+import math
+import os
+import re
+import shutil
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image, ImageDraw, ImageFont
+from scipy import ndimage
+
+
+@dataclass(frozen=True)
+class TopZoneConfig:
+ top_boundary_fraction: float = 0.32
+ minimum_top_zone_coverage: float = 0.60
+ minimum_score: float = 0.30
+ minimum_component_pixels: int = 30
+ minimum_component_source_fraction: float = 0.05
+ maximum_area_fraction: float = 0.03
+ duplicate_iou: float = 0.65
+ duplicate_containment: float = 0.85
+ maximum_candidates: int = 8
+
+
+def _safe_name(value: Any) -> str:
+ return re.sub(r"[^A-Za-z0-9_.-]+", "_", str(value)).strip("_") or "item"
+
+
+def _bool_mask(value: np.ndarray) -> np.ndarray:
+ mask = np.asarray(value)
+ if mask.ndim != 2:
+ raise ValueError(f"mask must be 2D, got {mask.shape}")
+ return mask.astype(bool, copy=False)
+
+
+def load_mask(path: str | Path, expected_shape: tuple[int, int]) -> np.ndarray:
+ resolved = Path(path).expanduser().resolve()
+ if not resolved.is_file():
+ raise FileNotFoundError(resolved)
+ mask = np.array(Image.open(resolved).convert("L")) > 0
+ if mask.shape != expected_shape:
+ raise ValueError(f"mask shape mismatch: {resolved}: {mask.shape} != {expected_shape}")
+ return mask
+
+
+def mask_bbox(mask: np.ndarray) -> list[int]:
+ ys, xs = np.nonzero(_bool_mask(mask))
+ if not len(xs):
+ raise ValueError("empty mask")
+ return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
+
+
+def mask_overlap(left: np.ndarray, right: np.ndarray) -> dict[str, float]:
+ left = _bool_mask(left)
+ right = _bool_mask(right)
+ intersection = int((left & right).sum())
+ left_area = int(left.sum())
+ right_area = int(right.sum())
+ union = left_area + right_area - intersection
+ return {
+ "iou": intersection / union if union else 0.0,
+ "left_coverage": intersection / left_area if left_area else 0.0,
+ "right_coverage": intersection / right_area if right_area else 0.0,
+ }
+
+
+def _split_components(mask: np.ndarray, config: TopZoneConfig) -> list[np.ndarray]:
+ labels, count = ndimage.label(_bool_mask(mask), structure=np.ones((3, 3), bool))
+ source_area = int(mask.sum())
+ minimum = max(
+ config.minimum_component_pixels,
+ int(math.ceil(config.minimum_component_source_fraction * source_area)),
+ )
+ components = []
+ for label_id in range(1, count + 1):
+ component = labels == label_id
+ if int(component.sum()) >= minimum:
+ components.append(component)
+ return components
+
+
+def filter_top_zone_candidates(
+ raw_candidates: Sequence[Mapping[str, Any]],
+ *,
+ image_size: tuple[int, int],
+ config: TopZoneConfig | None = None,
+) -> dict[str, Any]:
+ """Split disconnected masks, filter geometrically and remove duplicates."""
+
+ cfg = config or TopZoneConfig()
+ width, height = int(image_size[0]), int(image_size[1])
+ if width <= 0 or height <= 0:
+ raise ValueError("image_size must be positive")
+ image_area = width * height
+ top_y = int(round(cfg.top_boundary_fraction * height))
+ expanded: list[dict[str, Any]] = []
+ rejected: list[dict[str, Any]] = []
+
+ for raw in raw_candidates:
+ source_id = str(raw.get("candidate_id") or "")
+ score = float(raw.get("score") or 0.0)
+ source_mask = load_mask(raw["mask_path"], (height, width))
+ components = _split_components(source_mask, cfg)
+ for component_index, component in enumerate(components, start=1):
+ area = int(component.sum())
+ bbox = mask_bbox(component)
+ top_coverage = float(component[:top_y].sum() / area)
+ center_y = (bbox[1] + bbox[3]) / (2 * height)
+ area_fraction = area / image_area
+ reasons = []
+ if score < cfg.minimum_score:
+ reasons.append("low_score")
+ if top_coverage < cfg.minimum_top_zone_coverage:
+ reasons.append("insufficient_top_zone_coverage")
+ if center_y > cfg.top_boundary_fraction:
+ reasons.append("center_below_top_zone")
+ if area_fraction > cfg.maximum_area_fraction:
+ reasons.append("area_too_large")
+ record = {
+ "source_candidate_id": source_id,
+ "source_component_index": component_index,
+ "prompt": str(raw.get("prompt") or ""),
+ "score": score,
+ "area_pixels": area,
+ "area_fraction": area_fraction,
+ "top_zone_coverage": top_coverage,
+ "bbox_abs_xyxy": bbox,
+ "bbox_norm_xyxy": [
+ bbox[0] / width,
+ bbox[1] / height,
+ bbox[2] / width,
+ bbox[3] / height,
+ ],
+ "mask": component,
+ }
+ if reasons:
+ rejected.append({**{k: v for k, v in record.items() if k != "mask"}, "reasons": reasons})
+ else:
+ expanded.append(record)
+
+ expanded.sort(key=lambda item: (-item["score"], item["area_pixels"]))
+ kept: list[dict[str, Any]] = []
+ duplicate_rejections: list[dict[str, Any]] = []
+ for candidate in expanded:
+ duplicate = None
+ for existing in kept:
+ metrics = mask_overlap(candidate["mask"], existing["mask"])
+ if (
+ metrics["iou"] >= cfg.duplicate_iou
+ or max(metrics["left_coverage"], metrics["right_coverage"])
+ >= cfg.duplicate_containment
+ ):
+ duplicate = (existing, metrics)
+ break
+ if duplicate is None:
+ kept.append(candidate)
+ else:
+ duplicate_rejections.append(
+ {
+ "source_candidate_id": candidate["source_candidate_id"],
+ "source_component_index": candidate["source_component_index"],
+ "duplicate_of_source_candidate_id": duplicate[0]["source_candidate_id"],
+ "overlap": {k: round(v, 6) for k, v in duplicate[1].items()},
+ }
+ )
+
+ overflow = kept[cfg.maximum_candidates :]
+ kept = kept[: cfg.maximum_candidates]
+ kept.sort(key=lambda item: (item["bbox_abs_xyxy"][1], item["bbox_abs_xyxy"][0]))
+ for index, candidate in enumerate(kept, start=1):
+ candidate["candidate_id"] = f"LC_{index:03d}"
+ return {
+ "status": "candidates_found" if kept else "no_candidates",
+ "image_size": [width, height],
+ "config": cfg.__dict__,
+ "selected": kept,
+ "geometric_rejections": rejected,
+ "duplicate_rejections": duplicate_rejections,
+ "overflow_rejections": [
+ {k: v for k, v in item.items() if k != "mask"} for item in overflow
+ ],
+ }
+
+
+def _font(size: int):
+ try:
+ return ImageFont.truetype("DejaVuSans.ttf", size)
+ except OSError:
+ return ImageFont.load_default()
+
+
+def materialize_top_zone_review(
+ image_path: str | Path,
+ filtered: Mapping[str, Any],
+ output_dir: str | Path,
+ *,
+ overwrite: bool = False,
+ panel_title: str = "TOP-ZONE LOCAL CANDIDATES",
+ schema_version: str = "scenemosaic_top_zone_candidate_review",
+) -> dict[str, Any]:
+ """Save bounded candidate masks, overlays, a numbered panel and manifest."""
+
+ output = Path(output_dir).expanduser().resolve()
+ if output.exists() and any(output.iterdir()):
+ if not overwrite:
+ raise FileExistsError(output)
+ shutil.rmtree(output)
+ output.mkdir(parents=True, exist_ok=True)
+ masks_dir = output / "masks"
+ overlays_dir = output / "overlays"
+ masks_dir.mkdir()
+ overlays_dir.mkdir()
+ image = Image.open(image_path).convert("RGB")
+ image_np = np.array(image).astype(np.uint8)
+ records = []
+ colors = [
+ (240, 55, 55), (40, 150, 245), (40, 190, 90), (235, 155, 30),
+ (150, 75, 235), (245, 80, 175), (25, 185, 185), (185, 180, 35),
+ ]
+ overview = image.copy()
+ overview_draw = ImageDraw.Draw(overview)
+ label_font = _font(22)
+ cells = []
+
+ for index, raw in enumerate(filtered.get("selected") or []):
+ candidate = dict(raw)
+ mask = _bool_mask(candidate.pop("mask"))
+ candidate_id = candidate["candidate_id"]
+ base = _safe_name(candidate_id)
+ mask_path = masks_dir / f"{base}_mask.png"
+ overlay_path = overlays_dir / f"{base}_overlay.png"
+ Image.fromarray(mask.astype(np.uint8) * 255).save(mask_path)
+ overlay_np = image_np.copy()
+ color = np.array(colors[index % len(colors)], dtype=np.uint8)
+ overlay_np[mask] = (0.45 * overlay_np[mask] + 0.55 * color).astype(np.uint8)
+ overlay = Image.fromarray(overlay_np)
+ bbox = candidate["bbox_abs_xyxy"]
+ draw = ImageDraw.Draw(overlay)
+ draw.rectangle(tuple(bbox), outline=tuple(color.tolist()), width=4)
+ draw.text((bbox[0] + 3, max(0, bbox[1] - 25)), candidate_id, fill=tuple(color.tolist()), font=label_font)
+ overlay.save(overlay_path)
+
+ overview_draw.rectangle(tuple(bbox), outline=tuple(color.tolist()), width=4)
+ overview_draw.text((bbox[0] + 3, max(0, bbox[1] - 25)), candidate_id, fill=tuple(color.tolist()), font=label_font)
+ cell = overlay.copy()
+ cell.thumbnail((440, 260), Image.Resampling.LANCZOS)
+ canvas = Image.new("RGB", (460, 310), "white")
+ canvas.paste(cell, ((460 - cell.width) // 2, 38))
+ ImageDraw.Draw(canvas).text(
+ (8, 7),
+ f"{candidate_id} score={candidate['score']:.3f}",
+ fill="black",
+ font=_font(18),
+ )
+ cells.append(canvas)
+ records.append(
+ {
+ **candidate,
+ "mask_path": str(mask_path),
+ "overlay_path": str(overlay_path),
+ "review_status": "pending",
+ }
+ )
+
+ overview.thumbnail((1380, 780), Image.Resampling.LANCZOS)
+ overview_cell = Image.new("RGB", (1380, overview.height + 42), "white")
+ overview_cell.paste(overview, (0, 42))
+ ImageDraw.Draw(overview_cell).text(
+ (8, 8), panel_title, fill="black", font=_font(22)
+ )
+ columns = 3
+ rows = math.ceil(len(cells) / columns) if cells else 0
+ panel = Image.new(
+ "RGB",
+ (1380, overview_cell.height + rows * 310),
+ "white",
+ )
+ panel.paste(overview_cell, (0, 0))
+ for index, cell in enumerate(cells):
+ panel.paste(cell, ((index % columns) * 460, overview_cell.height + (index // columns) * 310))
+ panel_path = output / "numbered_candidate_panel.jpg"
+ panel.save(panel_path, quality=88)
+
+ manifest = {
+ "schema_version": schema_version,
+ "status": filtered.get("status"),
+ "source_image": str(Path(image_path).expanduser().resolve()),
+ "candidate_count": len(records),
+ "candidates": records,
+ "numbered_candidate_panel": str(panel_path),
+ "geometric_rejections": filtered.get("geometric_rejections") or [],
+ "duplicate_rejections": filtered.get("duplicate_rejections") or [],
+ "overflow_rejections": filtered.get("overflow_rejections") or [],
+ "config": filtered.get("config") or {},
+ }
+ manifest_path = output / "candidate_manifest.json"
+ temporary = manifest_path.with_suffix(".json.tmp")
+ temporary.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ os.replace(temporary, manifest_path)
+ return {"manifest": manifest, "manifest_path": str(manifest_path), "panel_path": str(panel_path)}
+
+
+__all__ = [
+ "TopZoneConfig",
+ "filter_top_zone_candidates",
+ "materialize_top_zone_review",
+ "mask_overlap",
+]
diff --git a/vlm_agent/reconstruction/perception/pipeline/runner.py b/vlm_agent/reconstruction/perception/pipeline/runner.py
new file mode 100644
index 0000000..5a87072
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/pipeline/runner.py
@@ -0,0 +1,3521 @@
+#!/usr/bin/env python3
+"""Run a two-pass VLM inventory overlapped with one reusable SAM3 session.
+
+All intermediate writes stay inside an attempt-local staging tree; formal stages are
+published only by the adoption layer. Pass A overlaps one local SAM3 top-zone recall probe;
+Pass B then reviews the numbered local candidates while SAM3 segments Pass-A
+objects with the same loaded session. After fusion, only complete prompt
+groups affected by Pass-B actions are recomputed. Unaffected groups reuse
+Pass-A masks and reviewed local candidates provide direct mask evidence.
+"""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+import math
+import os
+import re
+import shutil
+import sys
+import time
+from collections import defaultdict
+from concurrent.futures import ThreadPoolExecutor
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+
+CODE_ROOT = Path(__file__).resolve().parents[1]
+for directory in (CODE_ROOT,):
+ if str(directory) not in sys.path:
+ sys.path.insert(0, str(directory))
+
+from pipeline.core.candidate_review import (
+ BOUNDED_POST_SAM3A_REVIEW_CONFIG,
+ select_bounded_post_sam3a_candidates,
+ materialize_bounded_post_sam3a_review,
+ _semantic_tokens,
+ _local_candidate_semantic_score,
+ _local_candidate_pair_metrics,
+ match_local_candidates_to_b_additions,
+ compose_local_candidate_binding_actions,
+ attach_local_candidate_bindings,
+ promote_repeated_top_fixture_candidates,
+ rebind_verified_local_candidates_to_failed_existing_objects,
+)
+from pipeline.core.inventory_fusion import (
+ _prompt_groups,
+ _object_prompt_map,
+ validate_prompt_group_parity,
+ compute_affected_prompt_audit,
+ make_final_identity,
+)
+from pipeline.core.quality_gates import (
+ _mask_sha256,
+ build_bounded_candidate_semantic_recall_gate,
+ build_container_content_completeness_gate,
+ _tiny_ceiling_light_extent_tolerance,
+ _repeated_recessed_ceiling_light_row_extent_tolerance,
+ _small_flat_media_extent_tolerance,
+ _thin_attached_shelf_extent_tolerance,
+ _thin_wall_shelf_span_extent_tolerance,
+ _simple_visible_object_extent_tolerance,
+ _verified_lighting_fixture_extent_tolerance,
+ build_locally_validated_assigned_evidence_overflow,
+ resolve_assigned_evidence_semantic_gate_after_spatial_rescue,
+ build_functional_assembly_gate,
+ build_quality_audit,
+ apply_delivery_quality_policy,
+ _create_contact_sheet,
+ _copy_final_masks,
+)
+from pipeline.core.publication import (
+ atomic_publish,
+ safe_name,
+ canonicalize_final_object_assets,
+)
+from pipeline.core.pass_b_policy import (
+ validate_b_actions,
+ apply_post_pass_b_policy_guard,
+)
+from pipeline.core.runtime_io import (
+ utc_now,
+ read_json,
+ atomic_json,
+ sha256_file,
+ snapshot_tree,
+ compare_snapshots,
+)
+from pipeline.refinements import ambiguous_same_prompt_component_completion as AMBIGUOUS_SAME_PROMPT_COMPONENT_COMPLETION
+from pipeline.refinements import bbox_failed_mask_rescue as BBOX_FAILED_MASK_RESCUE
+from pipeline.refinements import nested_part_mask_suppression as NESTED_PART_MASK_SUPPRESSION
+from pipeline.refinements import low_profile_accessory_suppression as LOW_PROFILE_ACCESSORY_SUPPRESSION
+from pipeline.refinements import receptacle_content_mask_completion as RECEPTACLE_CONTENT_MASK_COMPLETION
+from pipeline.refinements import content_host_review as CONTENT_HOST_REVIEW
+from pipeline.refinements import host_part_mask_fusion as HOST_PART_MASK_FUSION
+from pipeline.refinements import functional_assembly_refinement as FUNCTIONAL_ASSEMBLY_REFINEMENT
+from pipeline.refinements import mask_ownership_resolution as MASK_OWNERSHIP_RESOLUTION
+from pipeline.refinements import mobile_tiered_carrier_completion as MOBILE_TIERED_CARRIER_COMPLETION
+from pipeline.refinements import named_local_instance_semantic_completion as NAMED_LOCAL_INSTANCE_SEMANTIC_COMPLETION
+from pipeline.refinements import open_storage_content_prompt as OPEN_STORAGE_CONTENT_PROMPT
+from pipeline.refinements import pass_a_same_mask_conflict as PASS_A_SAME_MASK_CONFLICT
+from pipeline.refinements import post_delta_multiplicity_closure as POST_DELTA_MULTIPLICITY_CLOSURE
+from pipeline.core import sam3d_delivery_policy as SAM3D_DELIVERY_POLICY
+from segmentation import runtime as SAM3_SESSION
+from pipeline.core import sam3_review_evidence as SAM3_REVIEW_EVIDENCE
+from pipeline.refinements import strict_same_mask_duplicate_collapse as STRICT_SAME_MASK_DUPLICATE_COLLAPSE
+from pipeline.refinements import top_zone_candidate_recall as TOP_ZONE
+
+from inventory import pipeline as VLM
+
+
+_DEFAULT_RUNTIME_ROOT = Path(
+ os.environ.get(
+ "SCENEMOSAIC_RUNTIME_ROOT",
+ Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
+ / "scenemosaic",
+ )
+).expanduser().resolve()
+PERCEPTION_ATTEMPTS_ROOT = Path(
+ os.environ.get(
+ "SCENEMOSAIC_PERCEPTION_ATTEMPTS_ROOT",
+ _DEFAULT_RUNTIME_ROOT / "perception_attempts",
+ )
+).expanduser().resolve()
+DEFAULT_OUTPUT = PERCEPTION_ATTEMPTS_ROOT / "perception_run"
+FORMAL_STAGE_NAMES = ("01_vlm_first", "02_sam3", "02_sam3_recall")
+ASSIGNMENT_POLICY = "adaptive_local"
+def resolve_merge_reviews_from_assigned_evidence(
+ *,
+ objects: Sequence[Mapping[str, Any]],
+ assigned_evidence_decisions: Sequence[Mapping[str, Any]],
+) -> dict[str, Any]:
+ """Close a merge review without inventing a new semantic object.
+
+ Only affirmative decisions (``pass`` or ``update_existing``) on the same
+ Pass-A source are accepted, and the successor must have a final SAM3 mask.
+ For a Pass-B-only addition, an extra adjacent SAM3 component has no prior
+ semantic owner. Keep the declared primary component and defer the extra
+ component as audit evidence rather than merging it or materializing an
+ undeclared object. This avoids a second VLM call while retaining a
+ source-traceable decision for a geometric ambiguity.
+ """
+
+ resolved_objects = [copy.deepcopy(dict(item)) for item in objects]
+ decisions_by_source = {
+ str(item.get("source_object_id") or ""): dict(item)
+ for item in assigned_evidence_decisions
+ if isinstance(item, Mapping)
+ and str(item.get("decision") or "") in {"pass", "update_existing"}
+ and str(item.get("source_object_id") or "")
+ }
+ records = []
+ for obj in resolved_objects:
+ sam3 = obj.get("sam3") or {}
+ review = sam3.get("merge_review") or {}
+ if not review.get("required") or sam3.get("status") != "success":
+ continue
+ mask_path = Path(str(sam3.get("mask_path") or ""))
+ if not mask_path.is_file():
+ continue
+ source_passes = {
+ str(source.get("pass") or "").upper()
+ for source in obj.get("proposal_sources") or []
+ if isinstance(source, Mapping)
+ }
+ if "B" in source_passes and "A" not in source_passes:
+ review["required"] = False
+ review["resolution_stage"] = (
+ "pass_b_unclaimed_adjacent_component_deferred"
+ )
+ review["deferred_unclaimed_candidate_ids"] = list(
+ review.get("ambiguous_candidate_ids") or []
+ )
+ sam3["merge_review"] = review
+ obj["sam3"] = sam3
+ records.append(
+ {
+ "proposal_id": str(obj.get("object_id") or ""),
+ "resolution": review["resolution_stage"],
+ "deferred_candidate_ids": review[
+ "deferred_unclaimed_candidate_ids"
+ ],
+ }
+ )
+ continue
+ matched_source_id = next(
+ (
+ str(source.get("source_id") or "")
+ for source in obj.get("proposal_sources") or []
+ if isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "A"
+ and str(source.get("source_id") or "") in decisions_by_source
+ ),
+ "",
+ )
+ if not matched_source_id:
+ continue
+ decision = decisions_by_source[matched_source_id]
+ review["required"] = False
+ review["resolution_stage"] = "assigned_sam3_evidence_semantic_confirmation"
+ review["assigned_evidence_resolution"] = {
+ "source_object_id": matched_source_id,
+ "evidence_id": str(decision.get("evidence_id") or ""),
+ "decision": str(decision.get("decision") or ""),
+ }
+ sam3["merge_review"] = review
+ obj["sam3"] = sam3
+ records.append({
+ "proposal_id": str(obj.get("object_id") or ""),
+ **review["assigned_evidence_resolution"],
+ })
+ return {
+ "schema_version": "scenemosaic_assigned_evidence_merge_resolution",
+ "status": "pass",
+ "resolved_count": len(records),
+ "records": records,
+ "hard_errors": [],
+ }, resolved_objects
+
+
+def resolve_cross_object_ambiguous_candidate_ownership(
+ *,
+ objects: Sequence[Mapping[str, Any]],
+ unassigned_candidates: Sequence[Mapping[str, Any]],
+ candidate_root: Path,
+) -> dict[str, Any]:
+ """Resolve merge-review candidates already owned by another final object."""
+
+ from PIL import Image
+ import numpy as np
+
+ resolved_objects = [copy.deepcopy(dict(item)) for item in objects]
+ candidates = {
+ str(item.get("candidate_id") or ""): copy.deepcopy(dict(item))
+ for item in unassigned_candidates
+ if isinstance(item, Mapping)
+ }
+ root = candidate_root.expanduser().resolve()
+ successful = [
+ item
+ for item in resolved_objects
+ if str((item.get("sam3") or {}).get("status") or "") == "success"
+ ]
+ records: list[dict[str, Any]] = []
+ hard_errors: list[dict[str, Any]] = []
+ mask_cache: dict[str, np.ndarray] = {}
+
+ def load_mask(path_value: Any, *, require_candidate_root: bool) -> np.ndarray:
+ path = Path(str(path_value or "")).expanduser().resolve()
+ if require_candidate_root and not path.is_relative_to(root):
+ raise ValueError(f"candidate mask escapes root: {path}")
+ key = str(path)
+ if key not in mask_cache:
+ if not path.is_file():
+ raise FileNotFoundError(path)
+ mask = np.array(Image.open(path).convert("L")) > 0
+ if not mask.any():
+ raise ValueError(f"empty mask: {path}")
+ mask_cache[key] = mask
+ return mask_cache[key]
+
+ for obj in resolved_objects:
+ sam3 = obj.get("sam3") or {}
+ review = sam3.get("merge_review") or {}
+ if not review.get("required"):
+ continue
+ ambiguous_ids = [
+ str(value) for value in review.get("ambiguous_candidate_ids") or []
+ ]
+ ownership_records = []
+ all_resolved = bool(ambiguous_ids)
+ for candidate_id in ambiguous_ids:
+ candidate = candidates.get(candidate_id)
+ if candidate is None:
+ all_resolved = False
+ continue
+ try:
+ candidate_mask = load_mask(
+ candidate.get("mask_path"), require_candidate_root=True
+ )
+ except (OSError, ValueError) as exc:
+ hard_errors.append(
+ {
+ "type": "invalid_ambiguous_candidate_mask",
+ "candidate_id": candidate_id,
+ "error": f"{type(exc).__name__}: {exc}",
+ }
+ )
+ all_resolved = False
+ continue
+ matches = []
+ for owner in successful:
+ if owner is obj:
+ continue
+ try:
+ owner_mask = load_mask(
+ (owner.get("sam3") or {}).get("mask_path"),
+ require_candidate_root=False,
+ )
+ except (OSError, ValueError):
+ continue
+ if owner_mask.shape != candidate_mask.shape:
+ continue
+ intersection = int(np.logical_and(candidate_mask, owner_mask).sum())
+ if intersection == 0:
+ continue
+ candidate_area = int(candidate_mask.sum())
+ owner_area = int(owner_mask.sum())
+ union = candidate_area + owner_area - intersection
+ metrics = {
+ "iou": intersection / union if union else 0.0,
+ "candidate_coverage_by_owner": intersection / candidate_area,
+ "owner_coverage_by_candidate": intersection / owner_area,
+ }
+ if (
+ metrics["iou"] >= 0.90
+ and metrics["candidate_coverage_by_owner"] >= 0.95
+ and metrics["owner_coverage_by_candidate"] >= 0.90
+ ):
+ matches.append((metrics["iou"], owner, metrics))
+ matches.sort(key=lambda item: item[0], reverse=True)
+ if not matches or (
+ len(matches) > 1 and matches[0][0] - matches[1][0] < 0.02
+ ):
+ all_resolved = False
+ continue
+ _, owner, metrics = matches[0]
+ ownership = {
+ "candidate_id": candidate_id,
+ "source_object_id": str(obj.get("object_id") or ""),
+ "owned_by_object_id": str(owner.get("object_id") or ""),
+ "iou": round(metrics["iou"], 6),
+ "candidate_coverage_by_owner": round(
+ metrics["candidate_coverage_by_owner"], 6
+ ),
+ "owner_coverage_by_candidate": round(
+ metrics["owner_coverage_by_candidate"], 6
+ ),
+ "thresholds": {
+ "minimum_iou": 0.90,
+ "minimum_candidate_coverage_by_owner": 0.95,
+ "minimum_owner_coverage_by_candidate": 0.90,
+ "minimum_best_match_gap": 0.02,
+ },
+ }
+ ownership_records.append(ownership)
+ records.append(ownership)
+ if all_resolved and len(ownership_records) == len(ambiguous_ids):
+ review["required"] = False
+ review["resolution_stage"] = "cross_object_candidate_mask_ownership"
+ review["ownership_records"] = ownership_records
+ sam3["merge_review"] = review
+ obj["sam3"] = sam3
+
+ audit = {
+ "schema_version": "scenemosaic_cross_object_candidate_ownership",
+ "status": "fail" if hard_errors else "pass",
+ "resolved_candidate_count": len(records),
+ "resolved_source_object_ids": sorted(
+ {item["source_object_id"] for item in records}
+ ),
+ "records": records,
+ "additional_sam3_inference_calls": 0,
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ "hard_errors": hard_errors,
+ }
+ return {"objects": resolved_objects, "audit": audit}
+
+
+
+def _rewrite_paths(value: Any, old_root: Path, new_root: Path) -> Any:
+ if isinstance(value, dict):
+ return {
+ key: _rewrite_paths(item, old_root, new_root)
+ for key, item in value.items()
+ }
+ if isinstance(value, list):
+ return [_rewrite_paths(item, old_root, new_root) for item in value]
+ if isinstance(value, str):
+ old_prefix = str(old_root)
+ if value == old_prefix or value.startswith(old_prefix + os.sep):
+ return str(new_root) + value[len(old_prefix):]
+ return value
+
+
+def remap_exact_proposal_id_values(
+ value: Any, object_id_map: Mapping[str, str]
+) -> Any:
+ """Update exact proposal-ID fields while preserving evidence paths."""
+
+ if isinstance(value, str):
+ return object_id_map.get(value, value)
+ if isinstance(value, Mapping):
+ return {
+ key: remap_exact_proposal_id_values(item, object_id_map)
+ for key, item in value.items()
+ }
+ if isinstance(value, list):
+ return [
+ remap_exact_proposal_id_values(item, object_id_map)
+ for item in value
+ ]
+ return value
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--scene-config", required=True, type=Path)
+ parser.add_argument("--output-dir", type=Path, default=DEFAULT_OUTPUT)
+ parser.add_argument("--pass-a-model", default="gpt-5.5")
+ parser.add_argument("--pass-b-model", default="gpt-5.5")
+ parser.add_argument(
+ "--reasoning-effort",
+ choices=("low", "medium", "high"),
+ default="low",
+ help="Pass-A reasoning effort.",
+ )
+ parser.add_argument(
+ "--pass-b-reasoning-effort",
+ choices=("none", "low", "medium", "high", "xhigh"),
+ default="none",
+ help=(
+ "Pass-B reasoning effort; none is the production default for the "
+ "compact structured protocol."
+ ),
+ )
+ parser.add_argument("--max-image-long-edge", type=int, default=1280)
+ parser.add_argument("--top-zone-prompt", default="ceiling light")
+ parser.add_argument("--top-zone-max-candidates", type=int, default=8)
+ parser.add_argument("--top-zone-panel-long-edge", type=int, default=1024)
+ parser.add_argument(
+ "--model-dir",
+ type=Path,
+ default=SAM3_SESSION.DEFAULT_MODEL_DIR,
+ )
+ parser.add_argument(
+ "--reuse-local-pre-pass-b-cache-root",
+ type=Path,
+ default=None,
+ help=(
+ "Reuse one validated failed isolated attempt's canonical local SAM3-A "
+ "registry and candidate evidence, then continue at bounded Pass B "
+ "in a new empty attempt output. This never reissues Pass A or "
+ "recomputes SAM3-A."
+ ),
+ )
+ parser.add_argument("--content-host-max-hosts", type=int, default=8)
+ parser.add_argument(
+ "--content-host-panel-long-edge",
+ type=int,
+ default=1280,
+ )
+ parser.add_argument("--assigned-evidence-max-objects", type=int, default=12)
+ parser.add_argument("--assigned-evidence-panel-long-edge", type=int, default=1280)
+ parser.add_argument("--allow-third-party-upload", action="store_true")
+ parser.add_argument(
+ "--strict-quality",
+ action="store_true",
+ help=(
+ "Block delivery on every observed quality finding. The default "
+ "best_effort policy blocks only identity and file-integrity errors."
+ ),
+ )
+ return parser.parse_args()
+
+
+def _validate_args(args: argparse.Namespace) -> None:
+ if not 1 <= args.content_host_max_hosts <= 8:
+ raise ValueError("--content-host-max-hosts must be in [1, 8]")
+ if not 768 <= args.content_host_panel_long_edge <= 1280:
+ raise ValueError("--content-host-panel-long-edge must be in [768, 1280]")
+ if not 1 <= args.assigned_evidence_max_objects <= 24:
+ raise ValueError("--assigned-evidence-max-objects must be in [1, 24]")
+ if not 768 <= args.assigned_evidence_panel_long_edge <= 1280:
+ raise ValueError("--assigned-evidence-panel-long-edge must be in [768, 1280]")
+ if not args.allow_third_party_upload:
+ raise PermissionError(
+ "--allow-third-party-upload is required for the external VLM calls"
+ )
+ if not str(args.top_zone_prompt).strip():
+ raise ValueError("--top-zone-prompt must be non-empty")
+ if not 1 <= args.top_zone_max_candidates <= 8:
+ raise ValueError("--top-zone-max-candidates must be in [1, 8]")
+ if not 512 <= args.top_zone_panel_long_edge <= 1280:
+ raise ValueError("--top-zone-panel-long-edge must be in [512, 1280]")
+
+ target = args.output_dir.expanduser().resolve()
+ if (
+ target != PERCEPTION_ATTEMPTS_ROOT
+ and PERCEPTION_ATTEMPTS_ROOT not in target.parents
+ ):
+ raise ValueError(
+ f"output directory must be below {PERCEPTION_ATTEMPTS_ROOT}"
+ )
+ if args.reuse_local_pre_pass_b_cache_root is not None:
+ source = args.reuse_local_pre_pass_b_cache_root.expanduser().resolve()
+ if source == target:
+ raise ValueError(
+ "cached pre-Pass-B source and output directory must differ"
+ )
+ if (
+ source == PERCEPTION_ATTEMPTS_ROOT
+ or PERCEPTION_ATTEMPTS_ROOT not in source.parents
+ ):
+ raise ValueError(
+ "cached pre-Pass-B source must be below the perception attempts root"
+ )
+ if source.is_symlink() or not source.is_dir():
+ raise ValueError(
+ "cached pre-Pass-B source must be a real attempt directory"
+ )
+ if not os.environ.get("OPENAI_API_KEY"):
+ raise RuntimeError("OPENAI_API_KEY must be set")
+
+
+def _prepare_output(output_dir: Path, run_root: Path) -> Path:
+ output_dir = output_dir.expanduser().resolve()
+ VLM.ensure_attempt_output(output_dir, run_root)
+ return output_dir
+
+
+
+def _openai_client() -> Any:
+ """Construct a VLM client, using OpenAI's default gateway when unset."""
+
+ from openai import OpenAI
+
+ kwargs: dict[str, Any] = {
+ "api_key": os.environ["OPENAI_API_KEY"],
+ "timeout": 300.0,
+ "max_retries": 0,
+ }
+ base_url = (
+ os.environ.get("OPEN_BASE_URL")
+ or os.environ.get("OPENAI_BASE_URL")
+ )
+ if base_url and base_url.strip() and base_url.strip() != "None":
+ kwargs["base_url"] = base_url
+ return OpenAI(**kwargs)
+
+
+def _timed_pass_a(
+ *,
+ image_data_url: str,
+ model: str,
+ reasoning_effort: str,
+ output_dir: Path,
+) -> dict[str, Any]:
+ started = time.monotonic()
+ client = _openai_client()
+ objects, call_info = VLM.run_pass_a(
+ client=client,
+ image_data_url=image_data_url,
+ model=model,
+ reasoning_effort=reasoning_effort,
+ output_dir=output_dir,
+ )
+ ended = time.monotonic()
+ return {
+ "objects": objects,
+ "call_info": call_info,
+ "start": started,
+ "end": ended,
+ }
+
+
+def _valid_norm_bbox(value: Any) -> list[float] | None:
+ try:
+ box = [float(component) for component in value]
+ except (TypeError, ValueError):
+ return None
+ if (
+ len(box) != 4
+ or not all(math.isfinite(component) for component in box)
+ or not (0.0 <= box[0] < box[2] <= 1.0)
+ or not (0.0 <= box[1] < box[3] <= 1.0)
+ ):
+ return None
+ return box
+
+
+def _bbox_center_is_inside(candidate_bbox: Any, container_bbox: Any) -> bool:
+ candidate = _valid_norm_bbox(candidate_bbox)
+ container = _valid_norm_bbox(container_bbox)
+ if candidate is None or container is None:
+ return False
+ center_x = (candidate[0] + candidate[2]) / 2.0
+ center_y = (candidate[1] + candidate[3]) / 2.0
+ return (
+ container[0] <= center_x <= container[2]
+ and container[1] <= center_y <= container[3]
+ )
+
+
+def _category_key(value: Any) -> str:
+ return "_".join(re.findall(r"[a-z0-9]+", str(value or "").lower()))
+
+
+def _bbox_coverage(inner_bbox: Any, outer_bbox: Any) -> float:
+ inner = _valid_norm_bbox(inner_bbox)
+ outer = _valid_norm_bbox(outer_bbox)
+ if inner is None or outer is None:
+ return 0.0
+ left = max(inner[0], outer[0])
+ top = max(inner[1], outer[1])
+ right = min(inner[2], outer[2])
+ bottom = min(inner[3], outer[3])
+ intersection = max(0.0, right - left) * max(0.0, bottom - top)
+ inner_area = (inner[2] - inner[0]) * (inner[3] - inner[1])
+ return intersection / inner_area if inner_area else 0.0
+
+
+def _prompt_declares_group(prompt: Any, category: Any) -> bool:
+ tokens = set(re.findall(r"[a-z0-9]+", str(prompt or "").lower()))
+ if tokens & {
+ "group", "set", "collection", "cluster", "bundle",
+ "pair", "row", "stack", "stacked", "pile",
+ }:
+ return True
+ category_words = re.findall(r"[a-z0-9]+", str(category or "").lower())
+ return any(f"{word}s" in tokens for word in category_words if word)
+
+
+def _share_open_storage_host(
+ left_bbox: Any,
+ right_bbox: Any,
+ hosts: Sequence[Mapping[str, Any]],
+) -> bool:
+ usable_hosts = [
+ item for item in hosts
+ if _valid_norm_bbox(item.get("bbox_norm_xyxy")) is not None
+ ]
+ if not usable_hosts:
+ return True
+ return any(
+ _bbox_center_is_inside(left_bbox, host.get("bbox_norm_xyxy"))
+ and _bbox_center_is_inside(right_bbox, host.get("bbox_norm_xyxy"))
+ for host in usable_hosts
+ )
+
+
+def _filter_content_closure_additions(
+ *,
+ primary_actions: Mapping[str, Any],
+ closure_actions: Mapping[str, Any],
+ open_storage_hosts: Sequence[Mapping[str, Any]] = (),
+ coverage_threshold: float = 0.80,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Keep only independent closure groups missed by main Pass B.
+
+ A closure pass may see individual members inside a group already selected
+ by main Pass B. Such members are neither missing nor valid new
+ reconstruction units. The prompt is asked to emit groups, and this gate
+ independently rejects a singular closure item that would subdivide a
+ same-category main group in the same verified open-storage host. It also
+ rejects any closure group substantially covered by a main-pass group.
+ """
+
+ if not 0.0 < coverage_threshold <= 1.0:
+ raise ValueError("coverage_threshold must be in (0, 1]")
+ primary = [
+ dict(item)
+ for item in (primary_actions.get("additions") or [])
+ if isinstance(item, Mapping)
+ and _valid_norm_bbox(item.get("bbox_2d_norm_xyxy")) is not None
+ ]
+ accepted: list[dict[str, Any]] = []
+ suppressed: list[dict[str, Any]] = []
+ invalid: list[dict[str, Any]] = []
+ for index, raw in enumerate(closure_actions.get("additions") or [], start=1):
+ if not isinstance(raw, Mapping):
+ invalid.append({"closure_addition_index": index, "reason": "not_mapping"})
+ continue
+ addition = dict(raw)
+ bbox = _valid_norm_bbox(addition.get("bbox_2d_norm_xyxy"))
+ category = _category_key(addition.get("category"))
+ if bbox is None or not category:
+ invalid.append(
+ {
+ "closure_addition_index": index,
+ "category": category,
+ "reason": "invalid_bbox_or_category",
+ }
+ )
+ continue
+ same_host_primary = [
+ existing for existing in primary
+ if _category_key(existing.get("category")) == category
+ and _share_open_storage_host(
+ bbox,
+ existing.get("bbox_2d_norm_xyxy"),
+ open_storage_hosts,
+ )
+ ]
+ group_blockers = [
+ existing for existing in same_host_primary
+ if _prompt_declares_group(
+ existing.get("sam3_prompt"), existing.get("category")
+ )
+ ]
+ if group_blockers and not _prompt_declares_group(
+ addition.get("sam3_prompt"), addition.get("category")
+ ):
+ suppressed.append(
+ {
+ "closure_addition_index": index,
+ "category": category,
+ "sam3_prompt": str(addition.get("sam3_prompt") or ""),
+ "reason": "would_subdivide_main_pass_b_group",
+ "blockers": [
+ {
+ "sam3_prompt": str(existing.get("sam3_prompt") or ""),
+ "bbox": existing.get("bbox_2d_norm_xyxy"),
+ }
+ for existing in group_blockers
+ ],
+ }
+ )
+ continue
+ blockers = []
+ for primary_index, existing in enumerate(primary, start=1):
+ if existing not in same_host_primary:
+ continue
+ coverage = _bbox_coverage(bbox, existing.get("bbox_2d_norm_xyxy"))
+ if coverage >= coverage_threshold:
+ blockers.append(
+ {
+ "primary_addition_index": primary_index,
+ "sam3_prompt": str(existing.get("sam3_prompt") or ""),
+ "coverage": round(coverage, 6),
+ }
+ )
+ if blockers:
+ suppressed.append(
+ {
+ "closure_addition_index": index,
+ "category": category,
+ "sam3_prompt": str(addition.get("sam3_prompt") or ""),
+ "reason": "covered_by_main_pass_b_group",
+ "blockers": blockers,
+ }
+ )
+ continue
+ accepted.append(addition)
+ audit = {
+ "schema_version": "scenemosaic_content_closure_deduplication",
+ "status": "pass",
+ "coverage_threshold": coverage_threshold,
+ "raw_addition_count": len(closure_actions.get("additions") or []),
+ "accepted_addition_count": len(accepted),
+ "suppressed_addition_count": len(suppressed),
+ "suppressed": suppressed,
+ "invalid": invalid,
+ "hard_errors": [],
+ }
+ return accepted, audit
+
+
+def _open_storage_hosts_for_content_closure(
+ content_hosts: Sequence[Mapping[str, Any]],
+) -> list[dict[str, Any]]:
+ """Return every verified open-storage host for a second content audit.
+
+ A crop is only attention context, not the physical interior of its owner.
+ Using crop occupancy as a proxy for content lets nearby lamps or furniture
+ hide an otherwise empty shelf. Every verified open-storage interior is
+ therefore rechecked once against the current inventory. The review prompt
+ asks for visible missing units only, so an empty compartment stays empty.
+ """
+ return [
+ dict(item)
+ for item in content_hosts
+ if isinstance(item, Mapping)
+ and str(item.get("host_mode") or "") == "open_storage"
+ ]
+
+
+def _build_content_closure_context(
+ *,
+ pass_a_objects: Sequence[Mapping[str, Any]],
+ actions: Mapping[str, Any],
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """Expose the current inventory to a closure-only Pass-B call.
+
+ Compact Pass B uses temporary A-numbered identifiers. First-pass additions
+ do not have formal IDs yet, so the closure context assigns temporary,
+ attempt-local IDs after the Pass-A range. Closure actions may add objects
+ only; update/removal output is discarded and cannot target those IDs.
+ """
+ current = [copy.deepcopy(dict(item)) for item in pass_a_objects]
+ seen_ids = {str(item.get("object_id") or "") for item in current}
+ numeric_ids = [
+ int(match.group(1))
+ for object_id in seen_ids
+ if (match := re.fullmatch(r"A_(\d{3})", object_id))
+ ]
+ next_index = max(numeric_ids, default=0) + 1
+ aliases: list[dict[str, Any]] = []
+ for addition_index, raw_addition in enumerate(
+ actions.get("additions") or [], start=1
+ ):
+ if not isinstance(raw_addition, Mapping):
+ continue
+ addition = copy.deepcopy(dict(raw_addition))
+ if _valid_norm_bbox(addition.get("bbox_2d_norm_xyxy")) is None:
+ continue
+ temporary_id = f"A_{next_index:03d}"
+ next_index += 1
+ addition["object_id"] = temporary_id
+ current.append(addition)
+ aliases.append(
+ {
+ "temporary_object_id": temporary_id,
+ "initial_pass_b_addition_index": addition_index,
+ "category": str(addition.get("category") or ""),
+ "bbox_2d_norm_xyxy": list(addition["bbox_2d_norm_xyxy"]),
+ }
+ )
+ return current, aliases
+
+
+def _renumber_content_hosts(
+ hosts: Sequence[Mapping[str, Any]],
+) -> list[dict[str, Any]]:
+ result = []
+ for index, raw_host in enumerate(hosts, start=1):
+ host = dict(raw_host)
+ host["host_id"] = f"H_{index:03d}"
+ result.append(host)
+ return result
+
+
+def _timed_pass_b(
+ *,
+ image_data_url: str,
+ pass_a_objects: list[dict[str, Any]],
+ model: str,
+ reasoning_effort: str,
+ output_dir: Path,
+ local_candidate_image_data_url: str | None = None,
+ local_candidate_metadata: list[dict[str, Any]] | None = None,
+ content_host_metadata: list[dict[str, Any]] | None = None,
+ assigned_evidence_image_data_url: str | None = None,
+ assigned_evidence_metadata: list[dict[str, Any]] | None = None,
+ same_mask_conflicts: list[dict[str, Any]] | None = None,
+ content_closure_only: bool = False,
+) -> dict[str, Any]:
+ started = time.monotonic()
+ client = _openai_client()
+ actions, call_info, prompt = VLM.run_pass_b(
+ client=client,
+ image_data_url=image_data_url,
+ pass_a_objects=pass_a_objects,
+ model=model,
+ reasoning_effort=reasoning_effort,
+ output_dir=output_dir,
+ local_candidate_image_data_url=local_candidate_image_data_url,
+ local_candidate_metadata=local_candidate_metadata,
+ content_host_metadata=content_host_metadata,
+ assigned_evidence_image_data_url=assigned_evidence_image_data_url,
+ assigned_evidence_metadata=assigned_evidence_metadata,
+ same_mask_conflicts=same_mask_conflicts,
+ content_closure_only=content_closure_only,
+ )
+ ended = time.monotonic()
+ return {
+ "actions": actions,
+ "call_info": call_info,
+ "prompt": prompt,
+ "start": started,
+ "end": ended,
+ }
+
+
+
+
+
+
+
+
+
+
+
+
+
+def _run_bounded_post_sam3a_pass_b(
+ *,
+ image_path: Path,
+ image_data_url: str,
+ pass_a_objects: list[dict[str, Any]],
+ sam3_a_registry: Mapping[str, Any],
+ top_candidates: Sequence[Mapping[str, Any]],
+ same_mask_conflicts: Sequence[Mapping[str, Any]] | None = None,
+ model: str,
+ reasoning_effort: str,
+ output_dir: Path,
+ review_dir: Path,
+ assets_dir: Path,
+ panel_long_edge: int,
+ session: Any | None = None,
+ content_host_max_hosts: int = 8,
+ assigned_evidence_max_objects: int = 12,
+ assigned_evidence_panel_long_edge: int = 1280,
+) -> dict[str, Any]:
+ """Materialize one bounded review panel, then call compact Pass B."""
+
+ selection_start = time.monotonic()
+ preselected_hosts: list[dict[str, Any]] = []
+ container_content_recall = None
+ working_sam3_registry = sam3_a_registry
+ host_selection = CONTENT_HOST_REVIEW.select_content_hosts(
+ pass_a_objects,
+ config=CONTENT_HOST_REVIEW.ContentHostConfig(
+ maximum_hosts=content_host_max_hosts
+ ),
+ )
+ host_errors = list(host_selection.get("hard_errors") or [])
+ if host_errors:
+ raise RuntimeError(
+ "content-host selection blocked before Pass B: "
+ + json.dumps(host_errors, ensure_ascii=False)
+ )
+ preselected_hosts = list(host_selection.get("selected") or [])
+ if session is None:
+ raise ValueError(
+ "fruit container completion requires the shared SAM3 session"
+ )
+ container_content_recall = (
+ RECEPTACLE_CONTENT_MASK_COMPLETION
+ .run_bounded_receptacle_content_mask_completion(
+ session=session,
+ infer_text_candidates_fn=SAM3_SESSION.infer_text_candidates,
+ image_path=image_path,
+ pass_a_objects=pass_a_objects,
+ sam3_registry=working_sam3_registry,
+ output_dir=review_dir.parent / "container_content_recall",
+ )
+ )
+ working_sam3_registry = container_content_recall[
+ "completed_registry"
+ ]
+ review = materialize_bounded_post_sam3a_review(
+ image_path=image_path,
+ top_candidates=top_candidates,
+ sam3_registry=working_sam3_registry,
+ output_dir=review_dir,
+ )
+ candidates = list(review["manifest"].get("candidates") or [])
+ host_review = None
+ hosts: list[dict[str, Any]] = []
+ host_review = CONTENT_HOST_REVIEW.materialize_content_host_review(
+ image_path=image_path,
+ objects=pass_a_objects,
+ output_dir=review_dir / "content_hosts",
+ config=CONTENT_HOST_REVIEW.ContentHostConfig(
+ maximum_hosts=content_host_max_hosts
+ ),
+ )
+ hosts = list(host_review["manifest"].get("selected") or [])
+ preselected_signature = [
+ (item.get("source_object_id"), item.get("host_mode"))
+ for item in preselected_hosts
+ ]
+ materialized_signature = [
+ (item.get("source_object_id"), item.get("host_mode"))
+ for item in hosts
+ ]
+ if materialized_signature != preselected_signature:
+ raise RuntimeError(
+ "content-host selection changed during materialization"
+ )
+ host_errors = list(host_review["manifest"].get("hard_errors") or [])
+ if host_errors:
+ raise RuntimeError(
+ "content-host selection blocked before Pass B: "
+ + json.dumps(host_errors, ensure_ascii=False)
+ )
+
+ assigned_evidence_review = None
+ assigned_evidence_records: list[dict[str, Any]] = []
+ assigned_evidence_asset = None
+ assigned_evidence_data_url = None
+ assigned_evidence_review = (
+ SAM3_REVIEW_EVIDENCE.materialize_assigned_sam3_review(
+ image_path=image_path,
+ pass_a_objects=pass_a_objects,
+ sam3_registry=working_sam3_registry,
+ output_dir=review_dir / "assigned_sam3_evidence",
+ config=SAM3_REVIEW_EVIDENCE.AssignedEvidenceConfig(
+ maximum_objects=assigned_evidence_max_objects
+ ),
+ )
+ )
+ assigned_manifest = assigned_evidence_review["manifest"]
+ assigned_errors = list(assigned_manifest.get("hard_errors") or [])
+ if assigned_errors:
+ raise RuntimeError(
+ "assigned SAM3 evidence blocked before Pass B: "
+ + json.dumps(assigned_errors, ensure_ascii=False)
+ )
+ assigned_evidence_records = list(
+ assigned_manifest.get("records") or []
+ )
+ if assigned_evidence_records:
+ assigned_evidence_asset = VLM.create_vlm_jpeg(
+ assigned_evidence_review["panel_path"],
+ assets_dir / "assigned_sam3_evidence_panel.jpg",
+ max_long_edge=assigned_evidence_panel_long_edge,
+ quality=84,
+ coordinate_grid=False,
+ )
+ assigned_evidence_data_url = VLM.encode_data_url(
+ Path(assigned_evidence_asset["path"]),
+ assigned_evidence_asset["mime_type"],
+ )
+
+ panel_asset = None
+ panel_data_url = None
+ combined_panel = None
+ panel_source = None
+ if hosts:
+ combined_panel = CONTENT_HOST_REVIEW.materialize_combined_review_panel(
+ image_path=image_path,
+ local_candidates=candidates,
+ content_hosts=hosts,
+ output_path=review_dir / "combined_review_panel.jpg",
+ )
+ panel_source = combined_panel["panel_path"]
+ elif candidates:
+ panel_source = review["panel_path"]
+ if panel_source is not None:
+ panel_asset = VLM.create_vlm_jpeg(
+ panel_source,
+ assets_dir / "merged_local_candidate_panel.jpg",
+ max_long_edge=panel_long_edge,
+ quality=84,
+ coordinate_grid=False,
+ )
+ panel_data_url = VLM.encode_data_url(
+ Path(panel_asset["path"]), panel_asset["mime_type"]
+ )
+ selection_end = time.monotonic()
+ b_result = _timed_pass_b(
+ image_data_url=image_data_url,
+ pass_a_objects=pass_a_objects,
+ model=model,
+ reasoning_effort=reasoning_effort,
+ output_dir=output_dir,
+ local_candidate_image_data_url=panel_data_url,
+ local_candidate_metadata=candidates if candidates else None,
+ content_host_metadata=hosts if hosts else None,
+ assigned_evidence_image_data_url=assigned_evidence_data_url,
+ assigned_evidence_metadata=(
+ assigned_evidence_records if assigned_evidence_records else None
+ ),
+ same_mask_conflicts=(
+ [dict(item) for item in same_mask_conflicts]
+ if same_mask_conflicts else None
+ ),
+ )
+ closure_hosts = _renumber_content_hosts(
+ _open_storage_hosts_for_content_closure(hosts)
+ )
+ content_closure = {
+ "status": "not_needed",
+ "requested_host_source_ids": [],
+ "addition_count": 0,
+ "discarded_update_count": 0,
+ "discarded_removal_count": 0,
+ "current_inventory_aliases": [],
+ "panel_asset": None,
+ }
+ if closure_hosts:
+ closure_panel = CONTENT_HOST_REVIEW.materialize_combined_review_panel(
+ image_path=image_path,
+ local_candidates=[],
+ content_hosts=closure_hosts,
+ output_path=review_dir / "content_closure_panel.jpg",
+ )
+ closure_asset = VLM.create_vlm_jpeg(
+ closure_panel["panel_path"],
+ assets_dir / "content_closure_panel.jpg",
+ max_long_edge=panel_long_edge,
+ quality=84,
+ coordinate_grid=False,
+ )
+ closure_data_url = VLM.encode_data_url(
+ Path(closure_asset["path"]), closure_asset["mime_type"]
+ )
+ closure_context, closure_aliases = _build_content_closure_context(
+ pass_a_objects=pass_a_objects,
+ actions=b_result["actions"],
+ )
+ closure_result = _timed_pass_b(
+ image_data_url=image_data_url,
+ pass_a_objects=closure_context,
+ model=model,
+ reasoning_effort=reasoning_effort,
+ output_dir=output_dir / "content_closure",
+ local_candidate_image_data_url=closure_data_url,
+ content_host_metadata=closure_hosts,
+ content_closure_only=True,
+ )
+ closure_actions = closure_result["actions"]
+ accepted_closure_additions, closure_deduplication_audit = (
+ _filter_content_closure_additions(
+ primary_actions=b_result["actions"],
+ closure_actions=closure_actions,
+ open_storage_hosts=closure_hosts,
+ )
+ )
+ atomic_json(
+ output_dir / "content_closure_deduplication_audit.json",
+ closure_deduplication_audit,
+ )
+ merged_actions = copy.deepcopy(b_result["actions"])
+ merged_actions["additions"].extend(
+ copy.deepcopy(accepted_closure_additions)
+ )
+ b_result["actions"] = merged_actions
+ b_result["call_info"]["content_closure"] = closure_result["call_info"]
+ content_closure = {
+ "status": "completed",
+ "requested_host_source_ids": [
+ str(host.get("source_object_id") or "")
+ for host in closure_hosts
+ ],
+ "raw_addition_count": len(closure_actions.get("additions") or []),
+ "addition_count": len(accepted_closure_additions),
+ "discarded_update_count": len(closure_actions.get("updates") or []),
+ "discarded_removal_count": len(closure_actions.get("removals") or []),
+ "current_inventory_aliases": closure_aliases,
+ "panel_asset": closure_asset["path"],
+ "panel_manifest": closure_panel,
+ "deduplication_audit": closure_deduplication_audit,
+ }
+
+ panel_count = 1 if panel_source is not None else 0
+ assigned_evidence_image_count = (
+ 1 if assigned_evidence_data_url is not None else 0
+ )
+ return {
+ "b_result": b_result,
+ "review": review,
+ "content_host_review": host_review,
+ "container_content_recall": container_content_recall,
+ "sam3_a_registry": working_sam3_registry,
+ "assigned_evidence_review": assigned_evidence_review,
+ "assigned_evidence_asset": assigned_evidence_asset,
+ "combined_panel": combined_panel,
+ "content_host_metadata": hosts,
+ "local_candidate_metadata": candidates,
+ "local_candidate_panel_asset": panel_asset,
+ "selection_interval": {
+ "start": selection_start,
+ "end": selection_end,
+ },
+ "api_call_audit": {
+ "mode": "bounded_post_sam3a",
+ "pass_b_calls_made": 1 + (
+ 1 if content_closure["status"] == "completed" else 0
+ ),
+ "inherited_pass_b_calls": 0,
+ "current_process_request_image_count": (
+ 1
+ + panel_count
+ + assigned_evidence_image_count
+ + (2 if content_closure["status"] == "completed" else 0)
+ ),
+ "additional_vlm_calls": (
+ 1 if content_closure["status"] == "completed" else 0
+ ),
+ "logical_pipeline_vlm_api_calls": 2 + (
+ 1 if content_closure["status"] == "completed" else 0
+ ),
+ "api_calls_unchanged": content_closure["status"] != "completed",
+ "full_scene_image_included": True,
+ "total_request_image_count": (
+ 1
+ + panel_count
+ + assigned_evidence_image_count
+ + (2 if content_closure["status"] == "completed" else 0)
+ ),
+ "review_panel_count": (
+ panel_count
+ + assigned_evidence_image_count
+ + (1 if content_closure["status"] == "completed" else 0)
+ ),
+ "candidate_panel_count": panel_count if candidates else 0,
+ "local_candidate_count": len(candidates),
+ "content_host_tile_count": len(hosts),
+ "container_content_completed_count": (
+ container_content_recall["audit"].get("completed_count", 0)
+ if container_content_recall is not None
+ else 0
+ ),
+ "container_content_additional_sam3_inference_count": (
+ container_content_recall["audit"].get("probe_count", 0)
+ if container_content_recall is not None
+ else 0
+ ),
+ "assigned_sam3_evidence_count": len(assigned_evidence_records),
+ "assigned_sam3_evidence_image_count": assigned_evidence_image_count,
+ "assigned_sam3_evidence_manifest": (
+ assigned_evidence_review["manifest_path"]
+ if assigned_evidence_review is not None else None
+ ),
+ "content_host_context_only": True,
+ "content_closure": content_closure,
+ "combined_review_panel": combined_panel,
+ "candidate_sources": dict(
+ review["selection_audit"].get(
+ "selected_counts_by_namespace", {}
+ )
+ ),
+ "sam3_a_unassigned_included": bool(
+ review["selection_audit"]
+ .get("selected_counts_by_namespace", {})
+ .get("sam3_a_unassigned", 0)
+ ),
+ "parallel_with_sam3_a": False,
+ "original_top_only_panel_sent": False,
+ },
+ }
+
+
+
+
+
+
+def _load_cached_pre_pass_b_bundle(
+ *,
+ cache_root: Path,
+ config: Mapping[str, Any],
+ top_zone_prompt: str,
+ top_zone_max_candidates: int,
+ allow_missing_unassigned_evidence: bool = False,
+) -> dict[str, Any]:
+ """Validate a stopped attempt cache for a no-SAM3-A continuation.
+
+ Accept either a run stopped before Pass B or a run whose sole Pass-B
+ response was unusable because it ended with ``finish_reason=length``.
+ Canonical SAM3-A masks and unassigned candidate masks remain in the source
+ attempt tree and are read-only evidence for a new, empty continuation tree.
+ """
+
+ source = cache_root.expanduser().resolve()
+ if not source.is_dir():
+ raise FileNotFoundError(source)
+ if source == Path(config["run_root"]).resolve():
+ raise ValueError("cached pre-Pass-B source may not be a formal run root")
+
+ manifest_path = source / "run_manifest.json"
+ manifest = read_json(manifest_path)
+ if manifest.get("status") != "failed":
+ raise ValueError("cached pre-Pass-B source must be a failed isolated attempt")
+ if manifest.get("scene_name") != config["scene_name"]:
+ raise ValueError("cached pre-Pass-B scene mismatch")
+ if manifest.get("input_image") != str(Path(config["image_path"]).resolve()):
+ raise ValueError("cached pre-Pass-B input image mismatch")
+ pass_b_response_path = source / "pass_b" / "response_raw.txt"
+ failed_length_pass_b = pass_b_response_path.is_file()
+ if failed_length_pass_b:
+ source_error = dict(manifest.get("error") or {})
+ expected_error = "Pass B response did not finish cleanly: finish_reason='length'"
+ if (
+ source_error.get("type") != "RuntimeError"
+ or source_error.get("message") != expected_error
+ ):
+ raise ValueError(
+ "cached pre-Pass-B source made a non-recoverable Pass-B call"
+ )
+
+ required = {
+ "asset": source / "assets" / "whole_scene_global_grid.jpg",
+ "pass_a_objects": source / "pass_a" / "normalized_objects.json",
+ "pass_a_call_info": source / "pass_a" / "call_info.json",
+ "pass_a_bbox_review": source / "pass_a" / "bbox_review_assets" / "bbox_review_manifest.json",
+ "top_raw": source / "work" / "top_zone_raw_candidates" / "text_candidates.json",
+ "top_review": source / "work" / "top_zone_review" / "candidate_manifest.json",
+ "sam3_a": source / "work" / "sam3_a" / "sam3_objects.json",
+ "unassigned": source / "work" / "sam3_a" / "unassigned_candidates",
+ }
+ if allow_missing_unassigned_evidence:
+ required.pop("unassigned")
+ missing = [str(path) for path in required.values() if not path.exists()]
+ if missing:
+ raise FileNotFoundError(f"cached pre-Pass-B bundle is incomplete: {missing}")
+ if any(path.is_symlink() for path in required.values()):
+ raise ValueError("cached pre-Pass-B bundle may not use symlinked roots")
+
+ def checked_evidence_path(raw_value: Any, label: str) -> Path:
+ path = Path(str(raw_value or "")).expanduser().resolve()
+ if source not in path.parents or not path.is_file():
+ raise ValueError(f"cached {label} evidence escapes or is missing: {path}")
+ return path
+
+ objects_payload = read_json(required["pass_a_objects"])
+ pass_a_objects = list(objects_payload.get("objects") or [])
+ if not pass_a_objects:
+ raise ValueError("cached pre-Pass-B Pass-A objects are empty")
+ pass_a_ids = [str(item.get("object_id") or "") for item in pass_a_objects]
+ if any(not item for item in pass_a_ids) or len(set(pass_a_ids)) != len(pass_a_ids):
+ raise ValueError("cached pre-Pass-B Pass-A IDs are invalid")
+
+ pass_a_info = read_json(required["pass_a_call_info"])
+ raw_registry = read_json(required["top_raw"])
+ if raw_registry.get("prompt") != top_zone_prompt:
+ raise ValueError("cached pre-Pass-B top-zone prompt mismatch")
+ review_manifest = read_json(required["top_review"])
+ if review_manifest.get("source_image") != str(Path(config["image_path"]).resolve()):
+ raise ValueError("cached pre-Pass-B top-zone image mismatch")
+ local_candidates = list(review_manifest.get("candidates") or [])
+ if len(local_candidates) > top_zone_max_candidates:
+ raise ValueError("cached pre-Pass-B top-zone candidate count exceeds limit")
+ expected_lc_ids = [f"LC_{index:03d}" for index in range(1, len(local_candidates) + 1)]
+ if [str(item.get("candidate_id") or "") for item in local_candidates] != expected_lc_ids:
+ raise ValueError("cached pre-Pass-B top-zone candidate IDs are invalid")
+ for candidate in local_candidates:
+ checked_evidence_path(candidate.get("mask_path"), "top-zone mask")
+ checked_evidence_path(candidate.get("overlay_path"), "top-zone overlay")
+
+ sam3_a_registry = read_json(required["sam3_a"])
+ registry_objects = list(sam3_a_registry.get("objects") or [])
+ registry_ids = [str(item.get("object_id") or "") for item in registry_objects]
+ if set(registry_ids) != set(pass_a_ids):
+ raise ValueError("cached pre-Pass-B SAM3-A registry IDs do not match Pass A")
+ for item in registry_objects:
+ sam3 = dict(item.get("sam3") or {})
+ if sam3.get("status") == "success":
+ checked_evidence_path(sam3.get("mask_path"), "SAM3-A mask")
+ checked_evidence_path(sam3.get("overlay_path"), "SAM3-A overlay")
+ for candidate in sam3_a_registry.get("unassigned_sam3_candidates") or []:
+ if not allow_missing_unassigned_evidence:
+ checked_evidence_path(candidate.get("mask_path"), "unassigned SAM3-A mask")
+ checked_evidence_path(candidate.get("overlay_path"), "unassigned SAM3-A overlay")
+
+
+ asset_path = checked_evidence_path(required["asset"], "whole-scene asset")
+ pass_a_assets = read_json(required["pass_a_bbox_review"])
+ return {
+ "asset": {
+ "path": str(asset_path),
+ "mime_type": "image/jpeg",
+ "reused_from": str(asset_path),
+ },
+ "pass_a_objects": pass_a_objects,
+ "pass_a_info": {**pass_a_info, "inherited": True, "inherited_from": str(source)},
+ "pass_a_assets": pass_a_assets,
+ "top_raw_result": {"registry": raw_registry, "registry_path": str(required["top_raw"])},
+ "top_review": {
+ "manifest": review_manifest,
+ "manifest_path": str(required["top_review"]),
+ "panel_path": str(source / "work" / "top_zone_review" / "numbered_candidate_panel.jpg"),
+ },
+ "local_candidate_metadata": local_candidates,
+ "sam3_a_registry": sam3_a_registry,
+ "cache_provenance": {
+ "schema_version": "scenemosaic_cached_pre_pass_b_resume",
+ "source_root": str(source),
+ "source_status": manifest.get("status"),
+ "source_failed_pass_b_finish_reason": (
+ "length" if failed_length_pass_b else None
+ ),
+ "reused_local_sam3_a": True,
+ "source_sam3_a_object_count": len(registry_objects),
+ "source_unassigned_candidate_count": len(sam3_a_registry.get("unassigned_sam3_candidates") or []),
+ "formal_outputs_modified": False,
+ },
+ }
+
+
+
+
+
+
+def _select_result_directory_name(quality_status: str) -> str:
+ return "final" if quality_status == "pass" else "draft_final"
+
+
+def main() -> int:
+ args = parse_args()
+ _validate_args(args)
+ pass_b_reasoning_effort = (
+ args.pass_b_reasoning_effort or args.reasoning_effort
+ )
+ cached_pre_pass_b_resume = args.reuse_local_pre_pass_b_cache_root is not None
+ cached_pre_pass_b_bundle = None
+ config = VLM.load_scene_config(args.scene_config)
+ output_dir = _prepare_output(args.output_dir, config["run_root"])
+ work_dir = output_dir / "work"
+ pass_a_dir = output_dir / "pass_a"
+ pass_b_dir = output_dir / "pass_b"
+ assets_dir = output_dir / "assets"
+ for directory in (work_dir, pass_a_dir, pass_b_dir, assets_dir):
+ directory.mkdir(parents=True, exist_ok=True)
+
+ formal_paths = [
+ config["run_root"] / name for name in FORMAL_STAGE_NAMES
+ ]
+ formal_before = snapshot_tree(formal_paths)
+ atomic_json(output_dir / "formal_hashes_before.json", formal_before)
+ manifest: dict[str, Any] = {
+ "schema_version": "scenemosaic_perception_attempt",
+ "status": "running",
+ "created_at_utc": utc_now(),
+ "scene_name": config["scene_name"],
+ "scene_config": str(config["config_path"]),
+ "input_image": str(config["image_path"]),
+ "output_dir": str(output_dir),
+ "formal_outputs_modified": False,
+ "pass_a_model": args.pass_a_model,
+ "pass_b_model": args.pass_b_model,
+ "pass_b_response_protocol": VLM.COMPACT_PASS_B_PROTOCOL,
+ "reasoning_effort": args.reasoning_effort,
+ "pass_a_reasoning_effort": args.reasoning_effort,
+ "pass_b_reasoning_effort": pass_b_reasoning_effort,
+ "top_zone_recall_config": {
+ "prompt": args.top_zone_prompt,
+ "maximum_candidates": args.top_zone_max_candidates,
+ "panel_long_edge": args.top_zone_panel_long_edge,
+ "additional_vlm_calls": 0,
+ },
+ "bounded_post_sam3a_candidate_review_config": {
+ "enabled": True,
+ "mode": "bounded_post_sam3a",
+ **BOUNDED_POST_SAM3A_REVIEW_CONFIG,
+ "additional_vlm_calls": 0,
+ "logical_pipeline_vlm_api_calls": 2,
+ "api_calls_unchanged": True,
+ },
+ "fruit_container_completion_config": {
+ "enabled": True,
+ "default_enabled": True,
+ "scope": "explicit_fruit_in_bowl_basket_tray_only",
+ "plants_in_scope": False,
+ "maximum_targets": 2,
+ "resolution": "merge_recalled_fruit_into_new_container_mask",
+ "preserve_successful_independent_fruit": True,
+ "source_masks_overwritten": False,
+ "additional_vlm_calls": 0,
+ },
+ "assigned_sam3_evidence_config": {
+ "enabled": True,
+ "maximum_objects": args.assigned_evidence_max_objects,
+ "panel_long_edge": args.assigned_evidence_panel_long_edge,
+ "additional_vlm_calls": 0,
+ "maximum_additional_request_images": 1,
+ "formal_outputs_modified": False,
+ },
+ "content_host_review_config": {
+ "enabled": True,
+ "maximum_hosts": args.content_host_max_hosts,
+ "panel_long_edge": args.content_host_panel_long_edge,
+ "context_only": True,
+ "additional_vlm_calls": 0,
+ "selected_mode": "bounded_post_sam3a",
+ "candidate_source_policy": (
+ "top_zone_plus_bounded_sam3a_unassigned"
+ ),
+ "supported_modes": ["bounded_post_sam3a"],
+ },
+ "open_storage_ownership_resolution_config": {
+ "enabled": True,
+ "default_enabled": True,
+ "scope": "verified_open_storage_only",
+ "source_masks_overwritten": False,
+ "additional_sam3_inference_calls": 0,
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ },
+ "ambiguous_same_prompt_component_completion_config": {
+ "enabled": True,
+ "scope": "saved_same_prompt_touching_ambiguous_sam3_components_only",
+ "additional_sam3_inference_calls": 0,
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ },
+ "named_local_instance_semantic_completion_config": {
+ "enabled": True,
+ "scope": "pass_b_named_carrier_local_instances_only",
+ "maximum_targets": 24,
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ },
+ "bbox_failed_mask_rescue_config": {
+ "enabled": True,
+ "default_enabled": True,
+ "mode": "isolated_attempt_bbox_positive_prompt",
+ "maximum_targets": (
+ BBOX_FAILED_MASK_RESCUE.BboxFailedMaskRescueConfig.maximum_targets
+ ),
+ "maximum_candidates_per_target": (
+ BBOX_FAILED_MASK_RESCUE.BboxFailedMaskRescueConfig
+ .maximum_candidates_per_target
+ ),
+ "maximum_recessed_candidates_per_target": (
+ BBOX_FAILED_MASK_RESCUE.BboxFailedMaskRescueConfig
+ .maximum_recessed_candidates_per_target
+ ),
+ "semantic_label_status_after_rescue": "provisional",
+ "maximum_additional_sam3_inference_calls": (
+ BBOX_FAILED_MASK_RESCUE.BboxFailedMaskRescueConfig.maximum_targets
+ ),
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ },
+ "post_delta_multiplicity_closure_config": {
+ "enabled": True,
+ "default_enabled": True,
+ "minimum_pass_b_peer_count": 2,
+ "minimum_candidate_score": 0.85,
+ "requires_shared_verified_host": True,
+ "maximum_total_additions": 4,
+ "additional_sam3_inference_calls": 0,
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ },
+ "assignment_policy": ASSIGNMENT_POLICY,
+ "model_dir": str(args.model_dir.expanduser().resolve()),
+ "manual_object_additions": 0,
+ }
+ atomic_json(output_dir / "run_manifest.json", manifest)
+
+ try:
+ if cached_pre_pass_b_resume:
+ cached_pre_pass_b_bundle = _load_cached_pre_pass_b_bundle(
+ cache_root=args.reuse_local_pre_pass_b_cache_root,
+ config=config,
+ top_zone_prompt=args.top_zone_prompt,
+ top_zone_max_candidates=args.top_zone_max_candidates,
+ allow_missing_unassigned_evidence=False,
+ )
+ asset = cached_pre_pass_b_bundle["asset"]
+ pass_a_objects = cached_pre_pass_b_bundle["pass_a_objects"]
+ pass_a_info = cached_pre_pass_b_bundle["pass_a_info"]
+ pass_a_assets = cached_pre_pass_b_bundle["pass_a_assets"]
+ top_raw_result = cached_pre_pass_b_bundle["top_raw_result"]
+ top_review = cached_pre_pass_b_bundle["top_review"]
+ local_candidate_metadata = cached_pre_pass_b_bundle[
+ "local_candidate_metadata"
+ ]
+ local_candidate_panel_asset = None
+ image_data_url = VLM.encode_data_url(
+ Path(asset["path"]), asset["mime_type"]
+ )
+ local_candidate_image_data_url = None
+ manifest["upload_asset"] = asset
+ manifest["cached_pre_pass_b_provenance"] = cached_pre_pass_b_bundle[
+ "cache_provenance"
+ ]
+ intervals = {}
+ model_load_start = time.monotonic()
+ session = SAM3_SESSION.load_session(
+ model_dir=args.model_dir,
+ device="cuda",
+ )
+ model_load_end = time.monotonic()
+ intervals["model_load"] = {
+ "start": model_load_start,
+ "end": model_load_end,
+ }
+ atomic_json(
+ output_dir / "top_zone_recall.json",
+ {
+ "schema_version": "scenemosaic_top_zone_recall_orchestration",
+ "mode": "cached_pre_pass_b_resume",
+ "prompt": args.top_zone_prompt,
+ "raw_candidate_count": top_raw_result["registry"].get(
+ "candidate_count", 0
+ ),
+ "review_candidate_count": len(local_candidate_metadata),
+ "raw_registry": top_raw_result["registry_path"],
+ "review_manifest": top_review["manifest_path"],
+ "candidate_panel_asset": None,
+ "current_process_model_load_count": session.model_load_count,
+ "current_process_top_zone_inference_count": 0,
+ "current_process_sam3_a_inference_count": 0,
+ "vlm_api_calls_added_before_pass_b": 0,
+ },
+ )
+ atomic_json(output_dir / "run_manifest.json", manifest)
+ else:
+ asset = VLM.create_vlm_jpeg(
+ config["image_path"],
+ assets_dir / "whole_scene_global_grid.jpg",
+ max_long_edge=args.max_image_long_edge,
+ quality=85,
+ coordinate_grid=True,
+ grid_divisions=10,
+ )
+ image_data_url = VLM.encode_data_url(
+ Path(asset["path"]), asset["mime_type"]
+ )
+ manifest["upload_asset"] = asset
+
+ intervals: dict[str, dict[str, float]] = {}
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ a_future = executor.submit(
+ _timed_pass_a,
+ image_data_url=image_data_url,
+ model=args.pass_a_model,
+ reasoning_effort=args.reasoning_effort,
+ output_dir=pass_a_dir,
+ )
+ local_recall_start = time.monotonic()
+ model_load_start = time.monotonic()
+ session = SAM3_SESSION.load_session(
+ model_dir=args.model_dir,
+ device="cuda",
+ )
+ model_load_end = time.monotonic()
+ top_raw_start = time.monotonic()
+ top_raw_result = SAM3_SESSION.infer_text_candidates(
+ session,
+ config["image_path"],
+ args.top_zone_prompt,
+ work_dir / "top_zone_raw_candidates",
+ )
+ top_raw_end = time.monotonic()
+ top_filtered = TOP_ZONE.filter_top_zone_candidates(
+ top_raw_result["registry"].get("candidates") or [],
+ image_size=tuple(top_raw_result["registry"]["image_size"]),
+ config=TOP_ZONE.TopZoneConfig(
+ maximum_candidates=args.top_zone_max_candidates
+ ),
+ )
+ top_review = TOP_ZONE.materialize_top_zone_review(
+ config["image_path"],
+ top_filtered,
+ work_dir / "top_zone_review",
+ )
+ local_candidate_metadata = list(
+ top_review["manifest"].get("candidates") or []
+ )
+ local_candidate_panel_asset = None
+ local_candidate_image_data_url = None
+ local_recall_end = time.monotonic()
+ a_result = a_future.result()
+
+ pass_a_objects = a_result["objects"]
+ pass_a_info = a_result["call_info"]
+ pass_a_start = a_result["start"]
+ pass_a_end = a_result["end"]
+ atomic_json(
+ pass_a_dir / "objects.json",
+ {
+ "schema_version": "scenemosaic_pipelined_pass_a",
+ "objects": pass_a_objects,
+ },
+ )
+ pass_a_assets = VLM.render_bbox_review_assets(
+ config["image_path"],
+ pass_a_objects,
+ pass_a_dir / "bbox_review_assets",
+ )
+ atomic_json(pass_a_dir / "call_info.json", pass_a_info)
+ atomic_json(
+ output_dir / "top_zone_recall.json",
+ {
+ "schema_version": "scenemosaic_top_zone_recall_orchestration",
+ "mode": "fresh_parallel_run",
+ "prompt": args.top_zone_prompt,
+ "raw_candidate_count": top_raw_result["registry"].get(
+ "candidate_count", 0
+ ),
+ "review_candidate_count": len(local_candidate_metadata),
+ "raw_registry": top_raw_result["registry_path"],
+ "review_manifest": top_review["manifest_path"],
+ "candidate_panel_asset": local_candidate_panel_asset,
+ "model_load_count": session.model_load_count,
+ "vlm_api_calls_added": 0,
+ },
+ )
+ intervals["pass_a"] = {"start": pass_a_start, "end": pass_a_end}
+ intervals["model_load"] = {
+ "start": model_load_start,
+ "end": model_load_end,
+ }
+ intervals["top_zone_raw_sam3"] = {
+ "start": top_raw_start,
+ "end": top_raw_end,
+ }
+ intervals["top_zone_local_recall"] = {
+ "start": local_recall_start,
+ "end": local_recall_end,
+ }
+
+ normalization_result = (
+ FUNCTIONAL_ASSEMBLY_REFINEMENT.normalize_pass_a_functional_assemblies(
+ pass_a_objects
+ )
+ )
+ functional_assembly_normalization = normalization_result["audit"]
+ pass_a_objects = normalization_result["objects"]
+ if functional_assembly_normalization.get("hard_errors"):
+ atomic_json(
+ output_dir / "functional_assembly_normalization_audit.json",
+ functional_assembly_normalization,
+ )
+ raise RuntimeError(
+ "functional assembly normalization failed closed before Pass B"
+ )
+ atomic_json(
+ output_dir / "functional_assembly_normalization_audit.json",
+ functional_assembly_normalization,
+ )
+ atomic_json(
+ pass_a_dir / "normalized_objects.json",
+ {
+ "schema_version": "scenemosaic_normalized_pass_a",
+ "objects": pass_a_objects,
+ },
+ )
+ manifest["functional_assembly_normalization"] = {
+ "status": functional_assembly_normalization.get("status"),
+ "raw_object_count": functional_assembly_normalization.get(
+ "raw_object_count", len(pass_a_objects)
+ ),
+ "normalized_object_count": len(pass_a_objects),
+ "target_count": len(
+ functional_assembly_normalization.get("targets") or []
+ ),
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ }
+ validate_prompt_group_parity(pass_a_objects)
+ bounded_candidate_review = None
+ bounded_candidate_api_audit = None
+ pass_b_review_api_audit = None
+ content_host_review = None
+ container_content_candidate_recall = None
+ functional_assembly_refinement = None
+ fixed_bathtub_faucet_union = None
+ fixed_component_action_policy = None
+ pass_a_same_mask_conflict_detection = None
+ pass_a_same_mask_conflict_resolution = None
+ assigned_evidence_review = None
+ assigned_evidence_asset = None
+ pass_b_review_panel_asset = local_candidate_panel_asset
+ pass_b_review_image_data_url = local_candidate_image_data_url
+ local_branch_start = None
+ local_branch_start = time.monotonic()
+ top_candidate_metadata = list(local_candidate_metadata)
+ sam3_a_start = time.monotonic()
+ if cached_pre_pass_b_resume:
+ if cached_pre_pass_b_bundle is None:
+ raise AssertionError("cached pre-Pass-B bundle was not loaded")
+ sam3_a_result = {
+ "registry": dict(cached_pre_pass_b_bundle["sam3_a_registry"])
+ }
+ intervals["cached_pre_pass_b_sam3_a"] = {
+ "start": sam3_a_start,
+ "end": sam3_a_start,
+ "source_root": cached_pre_pass_b_bundle["cache_provenance"][
+ "source_root"
+ ],
+ "source_object_count": cached_pre_pass_b_bundle[
+ "cache_provenance"
+ ]["source_sam3_a_object_count"],
+ "current_process_inference_count": 0,
+ "formal_outputs_modified": False,
+ }
+ else:
+ sam3_a_result = SAM3_SESSION.segment_registry(
+ session,
+ config["image_path"],
+ pass_a_objects,
+ work_dir / "sam3_a",
+ assignment_policy=ASSIGNMENT_POLICY,
+ scene_name=config["scene_name"],
+ registry_metadata={
+ "source": "pipelined_pass_a_before_bounded_pass_b",
+ "image_object_registry_version": "pipelined_attempt_a",
+ },
+ )
+ sam3_a_end = time.monotonic()
+ local_branch_end = sam3_a_end
+ functional_assembly_refinement_start = time.monotonic()
+ functional_assembly_refinement = (
+ FUNCTIONAL_ASSEMBLY_REFINEMENT.run_functional_assembly_roi_refinement(
+ session=session,
+ infer_text_candidates_fn=SAM3_SESSION.infer_text_candidates,
+ image_path=config["image_path"],
+ registry=sam3_a_result["registry"],
+ normalization_audit=functional_assembly_normalization,
+ output_dir=work_dir / "functional_assembly_roi_refinement",
+ )
+ )
+ functional_assembly_refinement_end = time.monotonic()
+ intervals["functional_assembly_roi_refinement"] = {
+ "start": functional_assembly_refinement_start,
+ "end": functional_assembly_refinement_end,
+ }
+ sam3_a_result["registry"] = functional_assembly_refinement[
+ "registry"
+ ]
+ atomic_json(
+ output_dir / "functional_assembly_roi_refinement_audit.json",
+ functional_assembly_refinement["audit"],
+ )
+ if functional_assembly_refinement["audit"].get("hard_errors"):
+ raise RuntimeError(
+ "functional assembly ROI refinement failed closed before Pass B"
+ )
+ manifest["functional_assembly_roi_refinement"] = {
+ "status": functional_assembly_refinement["audit"].get("status"),
+ "target_count": functional_assembly_refinement["audit"].get(
+ "target_count", 0
+ ),
+ "refined_count": functional_assembly_refinement["audit"].get(
+ "refined_count", 0
+ ),
+ "inference_count": functional_assembly_refinement["audit"].get(
+ "inference_count", 0
+ ),
+ "additional_model_load_count": 0,
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ }
+ fixed_union_start = time.monotonic()
+ fixed_bathtub_faucet_union = (
+ FUNCTIONAL_ASSEMBLY_REFINEMENT.union_fixed_bathtub_faucet_masks(
+ image_path=config["image_path"],
+ registry=sam3_a_result["registry"],
+ output_dir=work_dir / "fixed_bathtub_faucet_union",
+ )
+ )
+ fixed_union_end = time.monotonic()
+ intervals["fixed_bathtub_faucet_union"] = {
+ "start": fixed_union_start,
+ "end": fixed_union_end,
+ }
+ sam3_a_result["registry"] = fixed_bathtub_faucet_union["registry"]
+ atomic_json(
+ output_dir / "fixed_bathtub_faucet_union_audit.json",
+ fixed_bathtub_faucet_union["audit"],
+ )
+ if fixed_bathtub_faucet_union["audit"].get("hard_errors"):
+ raise RuntimeError(
+ "fixed bathtub faucet union failed closed before Pass B"
+ )
+ for pass_a_object in pass_a_objects:
+ object_update = (
+ fixed_bathtub_faucet_union["audit"]
+ .get("object_updates", {})
+ .get(str(pass_a_object.get("object_id") or ""))
+ )
+ if object_update:
+ pass_a_object.update(dict(object_update))
+ validate_prompt_group_parity(pass_a_objects)
+ manifest["fixed_bathtub_faucet_union"] = {
+ "status": fixed_bathtub_faucet_union["audit"].get("status"),
+ "accepted_count": fixed_bathtub_faucet_union["audit"].get(
+ "accepted_count", 0
+ ),
+ "forced_removals": fixed_bathtub_faucet_union["audit"].get(
+ "forced_removals", []
+ ),
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ }
+
+ plant_container_completion_start = time.monotonic()
+ plant_container_mask_completion = (
+ FUNCTIONAL_ASSEMBLY_REFINEMENT.complete_plant_container_masks(
+ image_path=config["image_path"],
+ registry=sam3_a_result["registry"],
+ output_dir=work_dir / "plant_container_mask_completion",
+ )
+ )
+ plant_container_completion_end = time.monotonic()
+ intervals["plant_container_mask_completion"] = {
+ "start": plant_container_completion_start,
+ "end": plant_container_completion_end,
+ }
+ sam3_a_result["registry"] = plant_container_mask_completion["registry"]
+ atomic_json(
+ output_dir / "plant_container_mask_completion_audit.json",
+ plant_container_mask_completion["audit"],
+ )
+ if plant_container_mask_completion["audit"].get("hard_errors"):
+ raise RuntimeError(
+ "plant-container mask completion failed closed before Pass B"
+ )
+ manifest["plant_container_mask_completion"] = {
+ "status": plant_container_mask_completion["audit"].get("status"),
+ "accepted_count": plant_container_mask_completion["audit"].get(
+ "accepted_count", 0
+ ),
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ }
+
+ same_mask_detection_start = time.monotonic()
+ pass_a_same_mask_conflict_detection = (
+ PASS_A_SAME_MASK_CONFLICT.detect_pass_a_same_mask_conflicts(
+ image_path=config["image_path"],
+ registry=sam3_a_result["registry"],
+ output_dir=work_dir / "pass_a_same_mask_conflicts",
+ )
+ )
+ same_mask_detection_end = time.monotonic()
+ intervals["pass_a_same_mask_conflict_detection"] = {
+ "start": same_mask_detection_start,
+ "end": same_mask_detection_end,
+ }
+ atomic_json(
+ output_dir / "pass_a_same_mask_conflicts_audit.json",
+ pass_a_same_mask_conflict_detection["audit"],
+ )
+ if pass_a_same_mask_conflict_detection["audit"].get("hard_errors"):
+ raise RuntimeError(
+ "Pass-A same-mask conflict detection failed closed before Pass B"
+ )
+ manifest["pass_a_same_mask_conflict_detection"] = {
+ "status": pass_a_same_mask_conflict_detection["audit"].get(
+ "status"
+ ),
+ "conflict_count": pass_a_same_mask_conflict_detection["audit"].get(
+ "conflict_count", 0
+ ),
+ "additional_vlm_calls": 0,
+ "additional_sam3_inference_count": 0,
+ "formal_outputs_modified": False,
+ }
+ bounded_result = _run_bounded_post_sam3a_pass_b(
+ image_path=config["image_path"],
+ image_data_url=image_data_url,
+ pass_a_objects=pass_a_objects,
+ sam3_a_registry=sam3_a_result["registry"],
+ top_candidates=top_candidate_metadata,
+ same_mask_conflicts=(
+ pass_a_same_mask_conflict_detection["conflicts"]
+ if pass_a_same_mask_conflict_detection is not None
+ else None
+ ),
+ model=args.pass_b_model,
+ reasoning_effort=pass_b_reasoning_effort,
+ output_dir=pass_b_dir,
+ review_dir=work_dir / "bounded_post_sam3a_review",
+ assets_dir=assets_dir,
+ panel_long_edge=args.content_host_panel_long_edge,
+ session=session,
+ content_host_max_hosts=args.content_host_max_hosts,
+ assigned_evidence_max_objects=(
+ args.assigned_evidence_max_objects
+ ),
+ assigned_evidence_panel_long_edge=(
+ args.assigned_evidence_panel_long_edge
+ ),
+ )
+ bounded_candidate_review = bounded_result["review"]
+ content_host_review = bounded_result["content_host_review"]
+ container_content_candidate_recall = bounded_result[
+ "container_content_recall"
+ ]
+ sam3_a_result["registry"] = bounded_result["sam3_a_registry"]
+ if container_content_candidate_recall is not None:
+ atomic_json(
+ output_dir
+ / "container_content_candidate_recall_audit.json",
+ container_content_candidate_recall["audit"],
+ )
+ assigned_evidence_review = bounded_result[
+ "assigned_evidence_review"
+ ]
+ assigned_evidence_asset = bounded_result[
+ "assigned_evidence_asset"
+ ]
+ bounded_candidate_api_audit = bounded_result["api_call_audit"]
+ pass_b_review_api_audit = bounded_candidate_api_audit
+ local_candidate_metadata = bounded_result["local_candidate_metadata"]
+ local_candidate_panel_asset = bounded_result[
+ "local_candidate_panel_asset"
+ ]
+ pass_b_review_panel_asset = local_candidate_panel_asset
+ intervals["bounded_candidate_selection"] = bounded_result[
+ "selection_interval"
+ ]
+ b_result = bounded_result["b_result"]
+ intervals["pass_b"] = {
+ "start": b_result["start"],
+ "end": b_result["end"],
+ }
+ pass_b_actions = b_result["actions"]
+ pass_b_info = b_result["call_info"]
+ atomic_json(pass_b_dir / "actions.json", pass_b_actions)
+ atomic_json(
+ output_dir / "bounded_candidate_api_call_audit.json",
+ bounded_candidate_api_audit,
+ )
+ atomic_json(
+ output_dir / "bounded_candidate_selection_audit.json",
+ bounded_candidate_review["selection_audit"],
+ )
+ if content_host_review is not None:
+ atomic_json(
+ output_dir / "content_host_selection_audit.json",
+ content_host_review["manifest"],
+ )
+ if assigned_evidence_review is not None:
+ atomic_json(
+ output_dir / "assigned_sam3_evidence_audit.json",
+ assigned_evidence_review["manifest"],
+ )
+
+ intervals["model_load"] = {
+ "start": model_load_start,
+ "end": model_load_end,
+ }
+ intervals["sam3_a"] = {"start": sam3_a_start, "end": sam3_a_end}
+ intervals["sam3_local_branch"] = {
+ "start": local_branch_start,
+ "end": local_branch_end,
+ }
+ validate_b_actions(
+ pass_a_objects,
+ pass_b_actions,
+ expected_local_candidate_ids=[
+ str(item["candidate_id"])
+ for item in local_candidate_metadata
+ ],
+ )
+ if fixed_bathtub_faucet_union is not None:
+ fixed_component_action_policy = (
+ FUNCTIONAL_ASSEMBLY_REFINEMENT.apply_fixed_component_absorption_actions(
+ pass_b_actions, fixed_bathtub_faucet_union["audit"]
+ )
+ )
+ pass_b_actions = fixed_component_action_policy["actions"]
+ atomic_json(
+ output_dir / "fixed_component_action_policy_audit.json",
+ fixed_component_action_policy["audit"],
+ )
+ validate_b_actions(
+ pass_a_objects,
+ pass_b_actions,
+ expected_local_candidate_ids=[
+ str(item["candidate_id"])
+ for item in local_candidate_metadata
+ ],
+ )
+ content_hosts = (
+ list(
+ (content_host_review.get("manifest") or {}).get("selected")
+ or []
+ )
+ if content_host_review is not None
+ else []
+ )
+ pass_b_action_guard_result = apply_post_pass_b_policy_guard(
+ pass_a_objects,
+ pass_b_actions,
+ sam3_a_result["registry"],
+ content_hosts=content_hosts,
+ )
+ pass_b_actions = pass_b_action_guard_result["actions"]
+ pass_b_action_guard_audit = pass_b_action_guard_result["audit"]
+ pass_b_actions, top_fixture_recall_audit = (
+ promote_repeated_top_fixture_candidates(
+ pass_a_objects=pass_a_objects,
+ pass_b_actions=pass_b_actions,
+ local_candidates=local_candidate_metadata,
+ )
+ )
+ (
+ pass_b_actions,
+ failed_existing_object_rebinding_audit,
+ ) = rebind_verified_local_candidates_to_failed_existing_objects(
+ pass_a_objects=pass_a_objects,
+ pass_b_actions=pass_b_actions,
+ local_candidates=local_candidate_metadata,
+ sam3_registry=sam3_a_result["registry"],
+ )
+ validate_b_actions(
+ pass_a_objects,
+ pass_b_actions,
+ expected_local_candidate_ids=[
+ str(item["candidate_id"])
+ for item in local_candidate_metadata
+ ],
+ )
+ atomic_json(
+ output_dir / "pass_b_action_guard_audit.json",
+ pass_b_action_guard_audit,
+ )
+ atomic_json(
+ output_dir / "repeated_top_fixture_recall_audit.json",
+ top_fixture_recall_audit,
+ )
+ atomic_json(
+ output_dir / "failed_existing_object_local_candidate_rebinding_audit.json",
+ failed_existing_object_rebinding_audit,
+ )
+ atomic_json(pass_b_dir / "effective_actions.json", pass_b_actions)
+ if pass_a_same_mask_conflict_detection is not None:
+ same_mask_resolution_start = time.monotonic()
+ pass_a_same_mask_conflict_resolution = (
+ PASS_A_SAME_MASK_CONFLICT.resolve_pass_a_same_mask_conflicts(
+ session=session,
+ infer_text_candidates_fn=SAM3_SESSION.infer_text_candidates,
+ image_path=config["image_path"],
+ registry=sam3_a_result["registry"],
+ actions=pass_b_actions,
+ conflicts=pass_a_same_mask_conflict_detection["conflicts"],
+ output_dir=work_dir / "pass_a_same_mask_conflict_resolution",
+ )
+ )
+ same_mask_resolution_end = time.monotonic()
+ intervals["pass_a_same_mask_conflict_resolution"] = {
+ "start": same_mask_resolution_start,
+ "end": same_mask_resolution_end,
+ }
+ sam3_a_result["registry"] = pass_a_same_mask_conflict_resolution[
+ "registry"
+ ]
+ atomic_json(
+ output_dir / "pass_a_same_mask_conflict_resolution_audit.json",
+ pass_a_same_mask_conflict_resolution["audit"],
+ )
+
+ fused_objects, fusion_audit = VLM.fuse_passes(
+ pass_a_objects, pass_b_actions
+ )
+ binding_actions = pass_b_actions
+ local_match_audit = None
+ if "local_candidate_decisions" not in pass_b_actions:
+ local_decisions, local_match_audit = (
+ match_local_candidates_to_b_additions(
+ fused_objects,
+ local_candidate_metadata,
+ explicitly_rejected_candidate_ids=(
+ pass_b_actions.get("local_candidate_rejections") or []
+ ),
+ )
+ )
+ binding_actions = compose_local_candidate_binding_actions(
+ pass_b_actions,
+ local_decisions,
+ )
+ fused_objects, local_binding_audit = attach_local_candidate_bindings(
+ fused_objects,
+ binding_actions,
+ local_candidate_metadata,
+ )
+ if local_match_audit is not None:
+ local_binding_audit["local_candidate_match"] = local_match_audit
+ local_binding_audit["hard_errors"].extend(
+ local_match_audit["hard_errors"]
+ )
+ if local_match_audit["hard_errors"]:
+ local_binding_audit["status"] = "fail"
+ prompt_normalization_result = (
+ OPEN_STORAGE_CONTENT_PROMPT
+ .canonicalize_open_storage_content_prompts(
+ objects=fused_objects,
+ content_hosts=content_hosts,
+ )
+ )
+ fused_objects = prompt_normalization_result["objects"]
+ open_storage_content_prompt_audit = (
+ prompt_normalization_result["audit"]
+ )
+ atomic_json(
+ output_dir
+ / "open_storage_content_prompt_normalization_audit.json",
+ open_storage_content_prompt_audit,
+ )
+ fused_objects, identity = make_final_identity(fused_objects)
+ affected_audit = compute_affected_prompt_audit(
+ pass_a_objects,
+ pass_b_actions,
+ fused_objects,
+ )
+ affected_prompts = set(affected_audit["new_prompts"])
+ validate_prompt_group_parity(fused_objects)
+ atomic_json(output_dir / "fusion_audit.json", fusion_audit)
+ if local_match_audit is not None:
+ atomic_json(
+ output_dir / "local_candidate_match_audit.json",
+ local_match_audit,
+ )
+ atomic_json(
+ output_dir / "local_candidate_binding_audit.json",
+ local_binding_audit,
+ )
+ atomic_json(output_dir / "identity_manifest_provisional.json", identity)
+ atomic_json(output_dir / "affected_prompt_groups.json", affected_audit)
+ atomic_json(
+ output_dir / "fused_proposals.json",
+ {
+ "schema_version": "scenemosaic_pipelined_proposals",
+ "objects": fused_objects,
+ },
+ )
+ fused_assets = VLM.render_bbox_review_assets(
+ config["image_path"],
+ fused_objects,
+ output_dir / "fused_bbox_review_assets",
+ )
+
+ delta_result = None
+ delta_start = time.monotonic()
+ if affected_prompts:
+ delta_result = SAM3_SESSION.segment_registry(
+ session,
+ config["image_path"],
+ fused_objects,
+ work_dir / "sam3_delta",
+ assignment_policy=ASSIGNMENT_POLICY,
+ prompt_filter=affected_prompts,
+ scene_name=config["scene_name"],
+ registry_metadata={
+ "source": "pipelined_pass_b_affected_groups",
+ "image_object_registry_version": "pipelined_attempt_delta",
+ "raw_candidate_evidence_storage": (
+ "ambiguity_and_repeated_instance_delta_candidates"
+ ),
+ },
+ capture_debug_candidates=False,
+ capture_unassigned_candidates=False,
+ capture_ambiguous_candidate_evidence=True,
+ capture_post_delta_multiplicity_candidates=True,
+ )
+ delta_end = time.monotonic()
+ intervals["sam3_delta"] = {"start": delta_start, "end": delta_end}
+
+ staging_prefix = ".final_staging"
+ staging_dir = output_dir / f"{staging_prefix}_{os.getpid()}"
+ if staging_dir.exists():
+ shutil.rmtree(staging_dir)
+ staging_dir.mkdir(parents=True)
+ finalize_start = time.monotonic()
+ final_objects = _copy_final_masks(
+ fused_objects=fused_objects,
+ affected_prompts=affected_prompts,
+ pass_a_registry=sam3_a_result["registry"],
+ delta_registry=delta_result["registry"] if delta_result else None,
+ staging_dir=staging_dir,
+ image_path=config["image_path"],
+ )
+ with SAM3_SESSION._load_production_module().Image.open(
+ config["image_path"]
+ ) as source:
+ image_np = SAM3_SESSION._load_production_module().np.array(
+ source.convert("RGB")
+ ).astype(SAM3_SESSION._load_production_module().np.uint8)
+ multiplicity_start = time.monotonic()
+ multiplicity_result = (
+ POST_DELTA_MULTIPLICITY_CLOSURE.run_post_delta_multiplicity_closure(
+ image_path=config["image_path"],
+ objects=final_objects,
+ delta_registry=delta_result["registry"] if delta_result else None,
+ fallback_registry=sam3_a_result["registry"],
+ work_dir=work_dir / "post_delta_multiplicity_closure",
+ staging_dir=staging_dir,
+ config=POST_DELTA_MULTIPLICITY_CLOSURE.MultiplicityClosureConfig(
+ enabled=True,
+ isolated_attempt_only=True,
+ ),
+ )
+ )
+ final_objects = multiplicity_result["objects"]
+ multiplicity_audit = multiplicity_result["audit"]
+ final_objects, identity = make_final_identity(final_objects)
+ multiplicity_end = time.monotonic()
+ intervals["post_delta_multiplicity_closure"] = {
+ "start": multiplicity_start,
+ "end": multiplicity_end,
+ }
+ multiplicity_audit_path = (
+ output_dir / "post_delta_multiplicity_closure_audit.json"
+ )
+ atomic_json(multiplicity_audit_path, multiplicity_audit)
+ contained_component_start = time.monotonic()
+ contained_component_result = (
+ NESTED_PART_MASK_SUPPRESSION.run_nested_part_mask_suppression(
+ objects=final_objects,
+ image_shape=tuple(image_np.shape[:2]),
+ output_dir=work_dir / "contained_component_suppression",
+ config=(
+ NESTED_PART_MASK_SUPPRESSION
+ .NestedPartMaskSuppressionConfig(
+ enabled=True,
+ isolated_attempt_only=True,
+ )
+ ),
+ )
+ )
+ final_objects = contained_component_result["objects"]
+ contained_component_audit = contained_component_result["audit"]
+ final_objects, identity = make_final_identity(final_objects)
+ contained_component_end = time.monotonic()
+ intervals["contained_component_suppression"] = {
+ "start": contained_component_start,
+ "end": contained_component_end,
+ }
+ contained_component_audit_path = (
+ output_dir / "contained_component_suppression_audit.json"
+ )
+ atomic_json(contained_component_audit_path, contained_component_audit)
+
+ low_profile_accessory_start = time.monotonic()
+ low_profile_accessory_result = (
+ LOW_PROFILE_ACCESSORY_SUPPRESSION
+ .run_low_profile_accessory_suppression(
+ objects=final_objects,
+ image_shape=tuple(image_np.shape[:2]),
+ output_dir=work_dir / "low_profile_accessory_suppression",
+ config=(
+ LOW_PROFILE_ACCESSORY_SUPPRESSION
+ .LowProfileAccessoryConfig(
+ enabled=True,
+ isolated_attempt_only=True,
+ )
+ ),
+ )
+ )
+ final_objects = low_profile_accessory_result["objects"]
+ low_profile_accessory_audit = low_profile_accessory_result["audit"]
+ final_objects, identity = make_final_identity(final_objects)
+ low_profile_accessory_end = time.monotonic()
+ intervals["low_profile_accessory_suppression"] = {
+ "start": low_profile_accessory_start,
+ "end": low_profile_accessory_end,
+ }
+ atomic_json(
+ output_dir / "low_profile_accessory_suppression_audit.json",
+ low_profile_accessory_audit,
+ )
+
+ strict_duplicate_start = time.monotonic()
+ strict_duplicate_result = (
+ STRICT_SAME_MASK_DUPLICATE_COLLAPSE.run_strict_same_mask_duplicate_collapse(
+ objects=final_objects,
+ retry_registry=None,
+ retry_audit=None,
+ image_shape=tuple(image_np.shape[:2]),
+ output_dir=(
+ work_dir / "strict_same_mask_duplicate_collapse"
+ ),
+ config=(
+ STRICT_SAME_MASK_DUPLICATE_COLLAPSE
+ .StrictSameMaskDuplicateCollapseConfig(
+ enabled=True,
+ isolated_attempt_only=True,
+ )
+ ),
+ )
+ )
+ final_objects = strict_duplicate_result["objects"]
+ strict_duplicate_audit = strict_duplicate_result["audit"]
+
+ strict_duplicate_end = time.monotonic()
+ intervals["strict_same_mask_duplicate_collapse"] = {
+ "start": strict_duplicate_start,
+ "end": strict_duplicate_end,
+ }
+ atomic_json(
+ output_dir / "strict_same_mask_duplicate_collapse_audit.json",
+ strict_duplicate_audit,
+ )
+ bbox_rescue_start = time.monotonic()
+ bbox_rescue_result = BBOX_FAILED_MASK_RESCUE.run_bbox_failed_mask_rescue(
+ session=session,
+ image_path=config["image_path"],
+ objects=final_objects,
+ work_dir=work_dir / "bbox_failed_mask_rescue",
+ staging_dir=staging_dir,
+ infer_candidates_fn=SAM3_SESSION.infer_text_candidates,
+ config=BBOX_FAILED_MASK_RESCUE.BboxFailedMaskRescueConfig(
+ enabled=True,
+ isolated_attempt_only=True,
+ priority_source_object_ids=frozenset(
+ str(item.get("source_object_id") or "")
+ for item in (
+ pass_b_actions.get("assigned_evidence_decisions") or []
+ )
+ if isinstance(item, Mapping)
+ and str(item.get("source_object_id") or "")
+ ),
+ ),
+ unresolved_source_object_ids=frozenset(
+ str(item.get("source_object_id") or "")
+ for item in (
+ pass_b_actions.get("assigned_evidence_decisions") or []
+ )
+ if isinstance(item, Mapping)
+ and item.get("decision") == "uncertain"
+ and str(item.get("source_object_id") or "")
+ ),
+ )
+ final_objects = bbox_rescue_result["objects"]
+ bbox_rescue_audit = bbox_rescue_result["audit"]
+
+ bbox_rescue_end = time.monotonic()
+ intervals["bbox_failed_mask_rescue"] = {
+ "start": bbox_rescue_start,
+ "end": bbox_rescue_end,
+ }
+ bbox_rescue_audit_path = output_dir / "bbox_failed_mask_rescue_audit.json"
+ atomic_json(bbox_rescue_audit_path, bbox_rescue_audit)
+
+ # Text-mask failures are repaired only here, after the pre-Pass-B
+ # completion has already run. Re-run the same deterministic
+ # plant-container check against the rescued objects so that this route
+ # cannot bypass a declared foliage-plus-vessel assembly.
+ post_bbox_plant_container_start = time.monotonic()
+ post_bbox_plant_container_registry = copy.deepcopy(
+ sam3_a_result["registry"]
+ )
+ post_bbox_plant_container_registry["objects"] = copy.deepcopy(
+ final_objects
+ )
+ post_bbox_plant_container_result = (
+ FUNCTIONAL_ASSEMBLY_REFINEMENT.complete_plant_container_masks(
+ image_path=config["image_path"],
+ registry=post_bbox_plant_container_registry,
+ output_dir=(
+ work_dir / "plant_container_mask_completion_after_bbox_rescue"
+ ),
+ )
+ )
+ post_bbox_plant_container_end = time.monotonic()
+ intervals["plant_container_mask_completion_after_bbox_rescue"] = {
+ "start": post_bbox_plant_container_start,
+ "end": post_bbox_plant_container_end,
+ }
+ final_objects = post_bbox_plant_container_result["registry"]["objects"]
+ post_bbox_plant_container_audit = post_bbox_plant_container_result[
+ "audit"
+ ]
+ post_bbox_plant_container_audit_path = (
+ output_dir / "plant_container_mask_completion_after_bbox_rescue_audit.json"
+ )
+ atomic_json(
+ post_bbox_plant_container_audit_path,
+ post_bbox_plant_container_audit,
+ )
+ if post_bbox_plant_container_audit.get("hard_errors"):
+ raise RuntimeError(
+ "plant-container mask completion failed closed after bbox rescue"
+ )
+ manifest["plant_container_mask_completion_after_bbox_rescue"] = {
+ "status": post_bbox_plant_container_audit.get("status"),
+ "accepted_count": post_bbox_plant_container_audit.get(
+ "accepted_count", 0
+ ),
+ "additional_vlm_calls": 0,
+ "formal_outputs_modified": False,
+ }
+ ambiguous_completion_start = time.monotonic()
+ ambiguous_completion_result = (
+ AMBIGUOUS_SAME_PROMPT_COMPONENT_COMPLETION
+ .run_ambiguous_same_prompt_component_completion(
+ image_path=config["image_path"],
+ objects=final_objects,
+ sam3_a_registry=sam3_a_result["registry"],
+ staging_dir=staging_dir,
+ work_dir=work_dir / "ambiguous_same_prompt_component_completion",
+ config=(
+ AMBIGUOUS_SAME_PROMPT_COMPONENT_COMPLETION
+ .AmbiguousSamePromptCompletionConfig(
+ enabled=True,
+ isolated_attempt_only=True,
+ )
+ ),
+ )
+ )
+ final_objects = ambiguous_completion_result["objects"]
+ ambiguous_completion_audit = ambiguous_completion_result["audit"]
+ final_objects, identity = make_final_identity(final_objects)
+ ambiguous_completion_end = time.monotonic()
+ intervals["ambiguous_same_prompt_component_completion"] = {
+ "start": ambiguous_completion_start,
+ "end": ambiguous_completion_end,
+ }
+ atomic_json(
+ output_dir / "ambiguous_same_prompt_component_completion_audit.json",
+ ambiguous_completion_audit,
+ )
+ named_local_completion_start = time.monotonic()
+ named_local_completion_result = (
+ NAMED_LOCAL_INSTANCE_SEMANTIC_COMPLETION
+ .run_named_local_instance_semantic_completion(
+ session=session,
+ image_path=config["image_path"],
+ objects=final_objects,
+ local_candidate_metadata=local_candidate_metadata,
+ sam3_a_registry=sam3_a_result["registry"],
+ staging_dir=staging_dir,
+ work_dir=work_dir / "named_local_instance_semantic_completion",
+ infer_candidates_fn=SAM3_SESSION.infer_text_candidates,
+ config=(
+ NAMED_LOCAL_INSTANCE_SEMANTIC_COMPLETION
+ .NamedLocalSemanticCompletionConfig(
+ enabled=True,
+ isolated_attempt_only=True,
+ )
+ ),
+ )
+ )
+ final_objects = named_local_completion_result["objects"]
+ named_local_completion_audit = named_local_completion_result["audit"]
+ final_objects, identity = make_final_identity(final_objects)
+ named_local_completion_end = time.monotonic()
+ intervals["named_local_instance_semantic_completion"] = {
+ "start": named_local_completion_start,
+ "end": named_local_completion_end,
+ }
+ atomic_json(
+ output_dir / "named_local_instance_semantic_completion_audit.json",
+ named_local_completion_audit,
+ )
+ mobile_tier_start = time.monotonic()
+ mobile_tier_result = (
+ MOBILE_TIERED_CARRIER_COMPLETION
+ .run_mobile_tiered_carrier_completion(
+ session=session,
+ image_path=config["image_path"],
+ objects=final_objects,
+ local_candidate_metadata=local_candidate_metadata,
+ sam3_a_registry=sam3_a_result["registry"],
+ staging_dir=staging_dir,
+ work_dir=work_dir / "mobile_tiered_carrier_completion",
+ infer_candidates_fn=SAM3_SESSION.infer_text_candidates,
+ config=(
+ MOBILE_TIERED_CARRIER_COMPLETION
+ .MobileTieredCarrierCompletionConfig(
+ enabled=True,
+ isolated_attempt_only=True,
+ )
+ ),
+ )
+ )
+ final_objects = mobile_tier_result["objects"]
+ mobile_tier_audit = mobile_tier_result["audit"]
+ final_objects, identity = make_final_identity(final_objects)
+ mobile_tier_end = time.monotonic()
+ intervals["mobile_tiered_carrier_completion"] = {
+ "start": mobile_tier_start,
+ "end": mobile_tier_end,
+ }
+ atomic_json(
+ output_dir / "mobile_tiered_carrier_completion_audit.json",
+ mobile_tier_audit,
+ )
+ production = SAM3_SESSION._load_production_module()
+ bed_pillow_edits = production.exclude_independent_pillows_from_beds(
+ final_objects, image_np
+ )
+ ownership_start = time.monotonic()
+ ownership_resolution_audit = {
+ "schema_version": "scenemosaic_open_storage_ownership_resolution",
+ "status": "disabled",
+ "changed_host_count": 0,
+ "changed_host_object_ids": [],
+ "formal_outputs_modified": False,
+ "hard_errors": [],
+ }
+ ownership_result = (
+ MASK_OWNERSHIP_RESOLUTION.resolve_open_storage_ownership(
+ image_path=config["image_path"],
+ objects=final_objects,
+ output_dir=work_dir / "open_storage_ownership_resolution",
+ )
+ )
+ final_objects = ownership_result["objects"]
+ ownership_resolution_audit = ownership_result["audit"]
+ ownership_end = time.monotonic()
+ intervals["open_storage_ownership_resolution"] = {
+ "start": ownership_start,
+ "end": ownership_end,
+ }
+ atomic_json(
+ output_dir / "open_storage_ownership_resolution_audit.json",
+ ownership_resolution_audit,
+ )
+ assigned_evidence_merge_resolution, final_objects = (
+ resolve_merge_reviews_from_assigned_evidence(
+ objects=final_objects,
+ assigned_evidence_decisions=(
+ pass_b_actions.get("assigned_evidence_decisions") or []
+ ),
+ )
+ )
+ atomic_json(
+ output_dir / "assigned_evidence_merge_resolution_audit.json",
+ assigned_evidence_merge_resolution,
+ )
+ cross_object_candidate_ownership = (
+ resolve_cross_object_ambiguous_candidate_ownership(
+ objects=final_objects,
+ unassigned_candidates=(
+ sam3_a_result["registry"].get(
+ "unassigned_sam3_candidates", []
+ )
+ ),
+ candidate_root=work_dir / "sam3_a" / "unassigned_candidates",
+ )
+ )
+ final_objects = cross_object_candidate_ownership["objects"]
+ cross_object_candidate_ownership_audit = (
+ cross_object_candidate_ownership["audit"]
+ )
+ atomic_json(
+ output_dir / "cross_object_candidate_ownership_audit.json",
+ cross_object_candidate_ownership_audit,
+ )
+ dedicated_attachment_start = time.monotonic()
+ dedicated_attachment_result = (
+ HOST_PART_MASK_FUSION.run_host_part_mask_fusion(
+ image_np=image_np,
+ image_path=config["image_path"],
+ session=session,
+ infer_candidates_fn=SAM3_SESSION.infer_text_candidates,
+ objects=final_objects,
+ staging_dir=staging_dir,
+ work_dir=work_dir / "dedicated_attachment_absorption",
+ config=(
+ HOST_PART_MASK_FUSION
+ .HostPartMaskFusionConfig(
+ enabled=True,
+ isolated_attempt_only=True,
+ )
+ ),
+ )
+ )
+ final_objects = dedicated_attachment_result["objects"]
+ dedicated_attachment_audit = dedicated_attachment_result["audit"]
+ final_objects, identity = make_final_identity(final_objects)
+ (
+ final_objects,
+ canonical_final_asset_publication_audit,
+ ) = canonicalize_final_object_assets(
+ objects=final_objects,
+ staging_dir=staging_dir,
+ image_np=image_np,
+ )
+ atomic_json(
+ output_dir / "canonical_final_asset_publication_audit.json",
+ canonical_final_asset_publication_audit,
+ )
+ dedicated_attachment_end = time.monotonic()
+ intervals["dedicated_attachment_absorption"] = {
+ "start": dedicated_attachment_start,
+ "end": dedicated_attachment_end,
+ }
+ atomic_json(
+ output_dir / "dedicated_attachment_absorption_audit.json",
+ dedicated_attachment_audit,
+ )
+ production.Image.fromarray(
+ production.rebuild_scene_overlay(final_objects, image_np)
+ ).save(staging_dir / "sam3_overlay.png")
+ _create_contact_sheet(
+ final_objects,
+ staging_dir / "sam3_overlay_contact_sheet.jpg",
+ )
+
+ quality = build_quality_audit(final_objects, final_objects, identity)
+ quality["canonical_final_asset_publication"] = (
+ canonical_final_asset_publication_audit
+ )
+ quality["hard_errors"].extend(
+ canonical_final_asset_publication_audit.get("hard_errors") or []
+ )
+ quality["assigned_evidence_merge_resolution"] = (
+ assigned_evidence_merge_resolution
+ )
+ quality["local_candidate_binding_audit"] = local_binding_audit
+ quality["hard_errors"].extend(local_binding_audit["hard_errors"])
+ quality["failed_existing_object_local_candidate_rebinding"] = (
+ failed_existing_object_rebinding_audit
+ )
+ quality["pass_b_action_guard"] = pass_b_action_guard_audit
+ quality["hard_errors"].extend(
+ pass_b_action_guard_audit.get("hard_errors") or []
+ )
+ quality["cross_object_candidate_ownership"] = (
+ cross_object_candidate_ownership_audit
+ )
+ quality["hard_errors"].extend(
+ cross_object_candidate_ownership_audit.get("hard_errors") or []
+ )
+ quality["dedicated_attachment_absorption"] = (
+ dedicated_attachment_audit
+ )
+ quality["hard_errors"].extend(
+ dedicated_attachment_audit.get("hard_errors") or []
+ )
+ quality["functional_assembly_normalization"] = (
+ functional_assembly_normalization
+ )
+ quality["hard_errors"].extend(
+ functional_assembly_normalization.get("hard_errors") or []
+ )
+ quality["open_storage_content_prompt_normalization"] = (
+ open_storage_content_prompt_audit
+ )
+ quality["hard_errors"].extend(
+ open_storage_content_prompt_audit.get("hard_errors") or []
+ )
+ if functional_assembly_refinement is not None:
+ quality["functional_assembly_roi_refinement"] = (
+ functional_assembly_refinement["audit"]
+ )
+ quality["hard_errors"].extend(
+ functional_assembly_refinement["audit"].get("hard_errors") or []
+ )
+ else:
+ quality["functional_assembly_roi_refinement"] = {
+ "status": "disabled_or_not_applicable",
+ "hard_errors": [],
+ }
+
+ quality["strict_same_mask_duplicate_collapse"] = strict_duplicate_audit
+ quality["hard_errors"].extend(
+ strict_duplicate_audit.get("hard_errors") or []
+ )
+ quality["contained_component_suppression"] = contained_component_audit
+ quality["hard_errors"].extend(
+ contained_component_audit.get("hard_errors") or []
+ )
+ quality["low_profile_accessory_suppression"] = (
+ low_profile_accessory_audit
+ )
+ quality["hard_errors"].extend(
+ low_profile_accessory_audit.get("hard_errors") or []
+ )
+ quality["bbox_failed_mask_rescue"] = bbox_rescue_audit
+ quality["hard_errors"].extend(
+ bbox_rescue_audit.get("hard_errors") or []
+ )
+ quality["plant_container_mask_completion_after_bbox_rescue"] = (
+ post_bbox_plant_container_audit
+ )
+ quality["hard_errors"].extend(
+ post_bbox_plant_container_audit.get("hard_errors") or []
+ )
+ quality["ambiguous_same_prompt_component_completion"] = (
+ ambiguous_completion_audit
+ )
+ quality["hard_errors"].extend(
+ ambiguous_completion_audit.get("hard_errors") or []
+ )
+ quality["named_local_instance_semantic_completion"] = (
+ named_local_completion_audit
+ )
+ quality["hard_errors"].extend(
+ named_local_completion_audit.get("hard_errors") or []
+ )
+ quality["mobile_tiered_carrier_completion"] = mobile_tier_audit
+ quality["hard_errors"].extend(
+ mobile_tier_audit.get("hard_errors") or []
+ )
+ quality["post_delta_multiplicity_closure"] = multiplicity_audit
+ quality["hard_errors"].extend(
+ multiplicity_audit.get("hard_errors") or []
+ )
+ if pass_a_same_mask_conflict_detection is not None:
+ quality["pass_a_same_mask_conflict_detection"] = (
+ pass_a_same_mask_conflict_detection["audit"]
+ )
+ quality["hard_errors"].extend(
+ pass_a_same_mask_conflict_detection["audit"].get("hard_errors")
+ or []
+ )
+ else:
+ quality["pass_a_same_mask_conflict_detection"] = {
+ "status": "disabled_or_not_bounded_review",
+ "hard_errors": [],
+ }
+ if pass_a_same_mask_conflict_resolution is not None:
+ quality["pass_a_same_mask_conflict_resolution"] = (
+ pass_a_same_mask_conflict_resolution["audit"]
+ )
+ quality["hard_errors"].extend(
+ pass_a_same_mask_conflict_resolution["audit"].get("hard_errors")
+ or []
+ )
+ else:
+ quality["pass_a_same_mask_conflict_resolution"] = {
+ "status": "not_applicable",
+ "hard_errors": [],
+ }
+ quality["open_storage_ownership_resolution"] = (
+ ownership_resolution_audit
+ )
+ quality["hard_errors"].extend(
+ ownership_resolution_audit.get("hard_errors") or []
+ )
+ if container_content_candidate_recall is not None:
+ container_content_gate = build_container_content_completeness_gate(
+ container_content_candidate_recall["audit"]
+ )
+ quality["container_content_candidate_recall"] = (
+ container_content_candidate_recall["audit"]
+ )
+ else:
+ container_content_gate = (
+ build_container_content_completeness_gate(None)
+ )
+ quality["container_content_candidate_recall"] = {
+ "status": "disabled",
+ "hard_errors": [],
+ }
+ quality["container_content_completeness_gate"] = container_content_gate
+ quality["hard_errors"].extend(
+ container_content_gate.get("hard_errors") or []
+ )
+ if assigned_evidence_review is None:
+ quality["hard_errors"].append(
+ {"type": "missing_assigned_sam3_evidence_review"}
+ )
+ quality["assigned_sam3_evidence"] = {"status": "missing"}
+ else:
+ assigned_manifest = assigned_evidence_review["manifest"]
+ quality["assigned_sam3_evidence"] = {
+ "status": assigned_manifest.get("status"),
+ "selected_count": assigned_manifest.get("selected_count", 0),
+ "overflow_count": assigned_manifest.get("overflow_count", 0),
+ "suspicious_overflow_count": assigned_manifest.get(
+ "suspicious_overflow_count", 0
+ ),
+ "audit_path": str(
+ output_dir / "assigned_sam3_evidence_audit.json"
+ ),
+ "additional_vlm_calls": 0,
+ }
+ quality["hard_errors"].extend(
+ assigned_manifest.get("hard_errors") or []
+ )
+ semantic_gate = pass_b_actions.get(
+ "assigned_evidence_semantic_gate"
+ )
+ evidence_decisions = list(
+ pass_b_actions.get("assigned_evidence_decisions") or []
+ )
+ quality["assigned_sam3_evidence"][
+ "semantic_gate"
+ ] = semantic_gate
+ quality["assigned_sam3_evidence"][
+ "decision_count"
+ ] = len(evidence_decisions)
+ expected_evidence_count = int(
+ assigned_manifest.get("selected_count", 0)
+ )
+ if (
+ not isinstance(semantic_gate, Mapping)
+ or len(evidence_decisions) != expected_evidence_count
+ or not semantic_gate.get("complete_partition")
+ ):
+ quality["hard_errors"].append(
+ {
+ "type": "assigned_sam3_semantic_gate_incomplete",
+ "expected_count": expected_evidence_count,
+ "actual_count": len(evidence_decisions),
+ }
+ )
+ downstream_spatial_rescue_source_ids = sorted(
+ {
+ str(source.get("source_id") or "")
+ for obj in final_objects
+ if (
+ (obj.get("sam3") or {}).get("selection_method")
+ == "bbox_first_spatial_rescue"
+ )
+ for source in (obj.get("proposal_sources") or [])
+ if isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "A"
+ and str(source.get("source_id") or "")
+ }
+ )
+ semantic_gate_recovery = (
+ resolve_assigned_evidence_semantic_gate_after_spatial_rescue(
+ assigned_manifest,
+ semantic_gate if isinstance(semantic_gate, Mapping) else None,
+ final_objects,
+ downstream_spatial_rescue_source_ids,
+ bbox_rescue_audit.get(
+ "excluded_unmaterialized_source_object_ids", []
+ ),
+ )
+ )
+ quality["assigned_sam3_evidence"][
+ "semantic_gate_recovery"
+ ] = semantic_gate_recovery
+ unresolved_evidence_ids = list(
+ semantic_gate_recovery.get("unresolved_evidence_ids") or []
+ )
+ if unresolved_evidence_ids:
+ quality["hard_errors"].append(
+ {
+ "type": "assigned_sam3_semantic_gate_unresolved",
+ "evidence_ids": unresolved_evidence_ids,
+ }
+ )
+ locally_rebound_source_ids = sorted(
+ {
+ str(source.get("source_id") or "")
+ for obj in final_objects
+ if (
+ (obj.get("sam3") or {}).get("status") == "success"
+ and (obj.get("sam3") or {}).get("selection_method")
+ == "vlm_reviewed_local_candidate_replace"
+ )
+ for source in (obj.get("proposal_sources") or [])
+ if isinstance(source, Mapping)
+ and str(source.get("pass") or "").upper() == "A"
+ and str(source.get("source_id") or "")
+ }
+ )
+ overflow_local_validation = (
+ build_locally_validated_assigned_evidence_overflow(
+ assigned_manifest, final_objects
+ )
+ )
+ quality["assigned_sam3_evidence"][
+ "overflow_local_validation"
+ ] = overflow_local_validation
+ quality["hard_errors"].extend(
+ overflow_local_validation.get("hard_errors") or []
+ )
+ extent_gate = {
+ "status": "not_enforced",
+ "reason": (
+ "assigned-mask claimed-bbox extent is recorded by SAM3 "
+ "but is not a delivery gate"
+ ),
+ }
+ quality["assigned_sam3_evidence_extent_gate"] = extent_gate
+ quality["assigned_sam3_evidence"]["extent_gate"] = extent_gate
+ if "assigned_sam3_evidence_extent_gate" not in quality:
+ quality["assigned_sam3_evidence_extent_gate"] = {
+ "status": "disabled_or_missing"
+ }
+
+ if bounded_candidate_review is not None:
+ candidate_semantic_gate = (
+ build_bounded_candidate_semantic_recall_gate(
+ bounded_candidate_review["selection_audit"],
+ pass_b_actions,
+ final_objects,
+ )
+ )
+ else:
+ candidate_semantic_gate = {
+ "status": "disabled",
+ "hard_errors": [],
+ }
+ quality["bounded_candidate_semantic_recall_gate"] = (
+ candidate_semantic_gate
+ )
+ quality["hard_errors"].extend(
+ candidate_semantic_gate.get("hard_errors") or []
+ )
+ functional_assembly_gate = build_functional_assembly_gate(
+ final_objects
+ )
+ quality["functional_assembly_gate"] = functional_assembly_gate
+ quality["hard_errors"].extend(
+ functional_assembly_gate.get("hard_errors") or []
+ )
+ if fixed_bathtub_faucet_union is not None:
+ quality["fixed_bathtub_faucet_union"] = (
+ fixed_bathtub_faucet_union["audit"]
+ )
+ quality["hard_errors"].extend(
+ fixed_bathtub_faucet_union["audit"].get("hard_errors") or []
+ )
+ else:
+ quality["fixed_bathtub_faucet_union"] = {
+ "status": "disabled_or_not_applicable",
+ "hard_errors": [],
+ }
+ if fixed_component_action_policy is not None:
+ quality["fixed_component_action_policy"] = (
+ fixed_component_action_policy["audit"]
+ )
+
+
+ formal_after = snapshot_tree(formal_paths)
+ atomic_json(output_dir / "formal_hashes_after.json", formal_after)
+ formal_comparison = compare_snapshots(formal_before, formal_after)
+ if not formal_comparison["unchanged"]:
+ quality["hard_errors"].append(
+ {
+ "type": "formal_outputs_modified",
+ "details": formal_comparison,
+ }
+ )
+ if session.model_load_count != 1:
+ quality["hard_errors"].append(
+ {
+ "type": "model_loaded_more_than_once",
+ "count": session.model_load_count,
+ }
+ )
+
+ review_mode = "bounded_post_sam3a"
+ quality["content_host_review"] = (
+ {
+ "status": content_host_review["manifest"]["status"],
+ "mode": review_mode,
+ "selected_host_count": content_host_review["manifest"][
+ "selected_count"
+ ],
+ "context_only": True,
+ "selection_audit": str(
+ output_dir / "content_host_selection_audit.json"
+ ),
+ "additional_vlm_calls": 0,
+ "api_call_audit": pass_b_review_api_audit,
+ }
+ if content_host_review is not None
+ else {
+ "status": "disabled",
+ "mode": review_mode,
+ "selected_host_count": 0,
+ "context_only": True,
+ "additional_vlm_calls": 0,
+ "api_call_audit": pass_b_review_api_audit,
+ }
+ )
+ if pass_b_review_api_audit is None:
+ quality["hard_errors"].append(
+ {"type": "missing_pass_b_review_api_audit"}
+ )
+ else:
+ expected_mode = review_mode
+ if pass_b_review_api_audit.get("mode") != expected_mode:
+ quality["hard_errors"].append(
+ {
+ "type": "pass_b_review_mode_mismatch",
+ "expected": expected_mode,
+ "actual": pass_b_review_api_audit.get("mode"),
+ }
+ )
+ closure = pass_b_review_api_audit.get("content_closure") or {}
+ expected_current_calls = 1 + (
+ 1 if closure.get("status") == "completed" else 0
+ )
+ if (
+ pass_b_review_api_audit.get("pass_b_calls_made")
+ != expected_current_calls
+ ):
+ quality["hard_errors"].append(
+ {"type": "pass_b_review_call_count_mismatch"}
+ )
+ quality = apply_delivery_quality_policy(
+ quality,
+ strict=bool(args.strict_quality),
+ )
+ quality["hard_error_count"] = len(quality["hard_errors"])
+ quality["warning_count"] = len(quality["warnings"])
+ publish_name = _select_result_directory_name(quality["status"])
+ publish_dir = output_dir / publish_name
+ final_objects = _rewrite_paths(final_objects, staging_dir, publish_dir)
+ quality = _rewrite_paths(quality, staging_dir, publish_dir)
+ atomic_json(bbox_rescue_audit_path, bbox_rescue_audit)
+ atomic_json(multiplicity_audit_path, multiplicity_audit)
+ bbox_rescue_summary = {
+ "enabled": bool(bbox_rescue_audit.get("enabled")),
+ "status": bbox_rescue_audit.get("status"),
+ "target_count": int(bbox_rescue_audit.get("target_count") or 0),
+ "accepted_count": int(bbox_rescue_audit.get("accepted_count") or 0),
+ "inference_call_count": int(
+ bbox_rescue_audit.get("inference_call_count") or 0
+ ),
+ "semantic_label_status": "provisional",
+ "additional_vlm_calls": 0,
+ "audit_path": str(bbox_rescue_audit_path),
+ }
+ multiplicity_summary = {
+ "enabled": bool(multiplicity_audit.get("enabled")),
+ "status": multiplicity_audit.get("status"),
+ "groups_checked": len(multiplicity_audit.get("groups_checked") or []),
+ "accepted_count": int(multiplicity_audit.get("accepted_count") or 0),
+ "additional_sam3_inference_calls": 0,
+ "additional_vlm_calls": 0,
+ "audit_path": str(multiplicity_audit_path),
+ }
+ open_storage_content_prompt_summary = {
+ "status": open_storage_content_prompt_audit.get("status"),
+ "verified_open_storage_host_count": int(
+ open_storage_content_prompt_audit.get(
+ "verified_open_storage_host_count"
+ ) or 0
+ ),
+ "changed_count": int(
+ open_storage_content_prompt_audit.get("changed_count") or 0
+ ),
+ "audit_path": str(
+ output_dir
+ / "open_storage_content_prompt_normalization_audit.json"
+ ),
+ }
+ mobile_tier_summary = {
+ "status": mobile_tier_audit.get("status"),
+ "target_host_count": int(mobile_tier_audit.get("target_host_count") or 0),
+ "processed_host_count": int(mobile_tier_audit.get("processed_host_count") or 0),
+ "inference_call_count": int(mobile_tier_audit.get("inference_call_count") or 0),
+ "created_tier_object_count": len(
+ mobile_tier_audit.get("created_tier_objects") or []
+ ),
+ "additional_vlm_calls": 0,
+ "audit_path": str(output_dir / "mobile_tiered_carrier_completion_audit.json"),
+ }
+
+ identity["records"] = [
+ {
+ **record,
+ "binding_status": (
+ "sam3_mask_bound"
+ if any(
+ obj["object_id"] == record["proposal_id"]
+ and (obj.get("sam3") or {}).get("status") == "success"
+ for obj in final_objects
+ )
+ else "sam3_failed"
+ ),
+ }
+ for record in identity["records"]
+ ]
+ top_review_candidate_count = len(
+ top_review["manifest"].get("candidates") or []
+ )
+ bounded_candidate_summary = {
+ "enabled": True,
+ "status": (
+ bounded_candidate_review["selection_audit"]["status"]
+ if bounded_candidate_review is not None
+ else "disabled"
+ ),
+ "selected_candidate_count": (
+ bounded_candidate_review["selection_audit"]["selected_count"]
+ if bounded_candidate_review is not None
+ else 0
+ ),
+ "selected_counts_by_namespace": (
+ bounded_candidate_review["selection_audit"][
+ "selected_counts_by_namespace"
+ ]
+ if bounded_candidate_review is not None
+ else {}
+ ),
+ "candidate_panel": (
+ local_candidate_panel_asset["path"]
+ if bounded_candidate_review is not None
+ and local_candidate_panel_asset is not None
+ else None
+ ),
+ "selection_audit": (
+ str(output_dir / "bounded_candidate_selection_audit.json")
+ if bounded_candidate_review is not None
+ else None
+ ),
+ "api_call_audit": bounded_candidate_api_audit,
+ "additional_vlm_calls": int(
+ (bounded_candidate_api_audit or {}).get("additional_vlm_calls")
+ or 0
+ ),
+ "logical_pipeline_vlm_api_calls": int(
+ (bounded_candidate_api_audit or {}).get(
+ "logical_pipeline_vlm_api_calls"
+ ) or 2
+ ),
+ "api_calls_unchanged": bool(
+ (bounded_candidate_api_audit or {}).get(
+ "api_calls_unchanged", True
+ )
+ ),
+ }
+ content_host_summary = {
+ "enabled": True,
+ "status": (
+ content_host_review["manifest"]["status"]
+ if content_host_review is not None
+ else "disabled"
+ ),
+ "selected_host_count": (
+ content_host_review["manifest"]["selected_count"]
+ if content_host_review is not None
+ else 0
+ ),
+ "content_host_panel": (
+ content_host_review["panel_path"]
+ if content_host_review is not None
+ else None
+ ),
+ "mode": (pass_b_review_api_audit or {}).get("mode"),
+ "combined_review_panel": (pass_b_review_api_audit or {}).get(
+ "combined_review_panel"
+ ),
+ "pass_b_review_panel": (
+ pass_b_review_panel_asset["path"]
+ if pass_b_review_panel_asset is not None
+ else None
+ ),
+ "candidate_sources": (pass_b_review_api_audit or {}).get(
+ "candidate_sources", {}
+ ),
+ "pass_b_request_image_count": (pass_b_review_api_audit or {}).get(
+ "total_request_image_count", 1
+ ),
+ "parallel_with_sam3_a": (pass_b_review_api_audit or {}).get(
+ "parallel_with_sam3_a", False
+ ),
+ "selection_audit": (
+ str(output_dir / "content_host_selection_audit.json")
+ if content_host_review is not None
+ else None
+ ),
+ "context_only": True,
+ "content_closure": (pass_b_review_api_audit or {}).get(
+ "content_closure"
+ ),
+ "additional_vlm_calls": int(
+ (pass_b_review_api_audit or {}).get("additional_vlm_calls")
+ or 0
+ ),
+ "logical_pipeline_vlm_api_calls": int(
+ (pass_b_review_api_audit or {}).get(
+ "logical_pipeline_vlm_api_calls"
+ ) or 2
+ ),
+ "api_calls_unchanged": bool(
+ (pass_b_review_api_audit or {}).get(
+ "api_calls_unchanged", True
+ )
+ ),
+ }
+ post_bbox_plant_container_summary = {
+ "status": post_bbox_plant_container_audit.get("status"),
+ "accepted_count": post_bbox_plant_container_audit.get(
+ "accepted_count", 0
+ ),
+ "additional_vlm_calls": 0,
+ "audit_path": str(post_bbox_plant_container_audit_path),
+ }
+ final_registry = {
+ "schema_version": "scenemosaic_perception_registry",
+ "scene_name": config["scene_name"],
+ "image_path": str(config["image_path"]),
+ "objects": final_objects,
+ "canonical_final_asset_publication": (
+ canonical_final_asset_publication_audit
+ ),
+ "manual_bed_pillow_exclusions": bed_pillow_edits,
+ "sam3_assignment_policy": ASSIGNMENT_POLICY,
+ "affected_prompt_groups": affected_audit,
+ "open_storage_content_prompt_normalization": (
+ open_storage_content_prompt_summary
+ ),
+ "bbox_failed_mask_rescue": bbox_rescue_summary,
+ "plant_container_mask_completion_after_bbox_rescue": (
+ post_bbox_plant_container_summary
+ ),
+ "post_delta_multiplicity_closure": multiplicity_summary,
+ "mobile_tiered_carrier_completion": mobile_tier_summary,
+
+ "top_zone_recall": {
+ "prompt": args.top_zone_prompt,
+ "raw_candidate_count": top_raw_result["registry"].get(
+ "candidate_count", 0
+ ),
+ "review_candidate_count": top_review_candidate_count,
+ "binding_status": local_binding_audit["status"],
+ "binding_audit": str(
+ output_dir / "local_candidate_binding_audit.json"
+ ),
+ "decision_counts": local_binding_audit["decision_counts"],
+ "raw_registry": top_raw_result["registry_path"],
+ "review_manifest": top_review["manifest_path"],
+ },
+ "bounded_post_sam3a_candidate_review": bounded_candidate_summary,
+ "content_host_review": content_host_summary,
+ "model_load_count": session.model_load_count,
+ "inference_run_count": session.inference_run_count,
+ }
+ atomic_json(staging_dir / "sam3_objects.json", final_registry)
+ atomic_json(staging_dir / "identity_manifest.json", identity)
+ atomic_json(staging_dir / "quality_report.json", quality)
+
+ atomic_json(output_dir / "quality_report.json", quality)
+ atomic_json(output_dir / "formal_hash_comparison.json", formal_comparison)
+
+ manifest.update(
+ {
+ "status": (
+ "success" if quality["status"] == "pass" else "quality_failed"
+ ),
+ "finished_at_utc": utc_now(),
+ "pass_a": {
+ "object_count": len(pass_a_objects),
+ "call": pass_a_info,
+ "bbox_review_assets": pass_a_assets,
+ },
+ "pass_b": {
+ "addition_count": len(pass_b_actions["additions"]),
+ "update_count": len(pass_b_actions["updates"]),
+ "removal_count": len(pass_b_actions["removals"]),
+ "local_candidate_decision_count": len(
+ pass_b_actions.get("local_candidate_decisions") or []
+ ),
+ "local_candidate_rejection_count": len(
+ pass_b_actions.get("local_candidate_rejections") or []
+ ),
+ "local_candidate_match_count": (
+ local_match_audit.get("match_count", 0)
+ if local_match_audit is not None
+ else 0
+ ),
+ "local_candidate_unresolved_count": (
+ len(local_match_audit.get("unresolved_candidate_ids") or [])
+ if local_match_audit is not None
+ else 0
+ ),
+ "call": pass_b_info,
+ },
+ "pass_b_action_guard": {
+ "status": pass_b_action_guard_audit.get("status"),
+ "partition_count": pass_b_action_guard_audit.get(
+ "partition_count", 0
+ ),
+ "suppressed_update_count": len(
+ pass_b_action_guard_audit.get("suppressed_updates") or []
+ ),
+ "suppressed_addition_count": len(
+ pass_b_action_guard_audit.get("suppressed_additions") or []
+ ),
+ "audit_path": str(
+ output_dir / "pass_b_action_guard_audit.json"
+ ),
+ },
+ "top_zone_recall": {
+ "prompt": args.top_zone_prompt,
+ "raw_candidate_count": top_raw_result["registry"].get(
+ "candidate_count", 0
+ ),
+ "review_candidate_count": top_review_candidate_count,
+ "candidate_panel": top_review["panel_path"],
+ "binding_status": local_binding_audit["status"],
+ "bound_object_count": local_binding_audit["bound_object_count"],
+ "decision_counts": local_binding_audit["decision_counts"],
+ "raw_registry": top_raw_result["registry_path"],
+ "review_manifest": top_review["manifest_path"],
+ "binding_audit": str(
+ output_dir / "local_candidate_binding_audit.json"
+ ),
+ },
+ "bounded_post_sam3a_candidate_review": bounded_candidate_summary,
+ "content_host_review": content_host_summary,
+ "open_storage_content_prompt_normalization": (
+ open_storage_content_prompt_summary
+ ),
+ "fruit_container_completion": (
+ {
+ "enabled": True,
+ "status": container_content_candidate_recall["audit"].get(
+ "status"
+ ),
+ "probe_count": container_content_candidate_recall[
+ "audit"
+ ].get("probe_count", 0),
+ "completed_count": container_content_candidate_recall[
+ "audit"
+ ].get("completed_count", 0),
+ "completed_source_object_ids": (
+ container_content_candidate_recall["audit"].get(
+ "completed_source_object_ids", []
+ )
+ ),
+ "plants_in_scope": False,
+ "additional_vlm_calls": 0,
+ "audit_path": str(
+ output_dir
+ / "container_content_candidate_recall_audit.json"
+ ),
+ }
+ if container_content_candidate_recall is not None
+ else {
+ "enabled": False,
+ "status": "disabled",
+ "probe_count": 0,
+ "completed_count": 0,
+ "completed_source_object_ids": [],
+ "plants_in_scope": False,
+ "additional_vlm_calls": 0,
+ }
+ ),
+
+ "bbox_failed_mask_rescue": bbox_rescue_summary,
+ "post_delta_multiplicity_closure": multiplicity_summary,
+ "mobile_tiered_carrier_completion": mobile_tier_summary,
+ "fused_object_count": len(fused_objects),
+ "final_object_count": len(final_objects),
+ "fused_bbox_review_assets": fused_assets,
+ "affected_prompt_count": len(affected_prompts),
+ "sam3_model_load_count": session.model_load_count,
+ "sam3_inference_run_count": session.inference_run_count,
+ "published_result": str(publish_dir),
+ "quality_status": quality["status"],
+ "quality_policy": quality["quality_policy"],
+ "formal_outputs_modified": not formal_comparison["unchanged"],
+ "pass_a_inherited": cached_pre_pass_b_resume,
+ "pass_b_inherited": False,
+ "eligible_for_stage04": quality["status"] == "pass",
+ "sam3_model_load_scope": (
+ "one_in_cache_continuation_process; cached top-zone evidence "
+ "came from one earlier source-process load"
+ if cached_pre_pass_b_resume
+ else "one_in_fresh_process"
+ ),
+ }
+ )
+ atomic_json(output_dir / "run_manifest.json", manifest)
+ atomic_publish(staging_dir, publish_dir)
+
+ try:
+ sam3d_delivery = (
+ SAM3D_DELIVERY_POLICY.materialize_delivery_package(
+ publish_dir=publish_dir,
+ )
+ )
+ except Exception as delivery_exc:
+ sam3d_delivery = {
+ "status": "infrastructure_failed",
+ "sam3d_execution_allowed": False,
+ "object_count": 0,
+ "error": {
+ "type": type(delivery_exc).__name__,
+ "message": str(delivery_exc),
+ },
+ }
+ atomic_json(
+ output_dir / "sam3d_delivery_failure.json",
+ sam3d_delivery,
+ )
+ manifest["sam3d_delivery"] = sam3d_delivery
+ source_quality_passed = quality["status"] == "pass"
+ manifest["eligible_for_stage04"] = bool(
+ source_quality_passed
+ and sam3d_delivery.get("sam3d_execution_allowed")
+ )
+ if not source_quality_passed:
+ manifest["status"] = "quality_failed"
+ elif sam3d_delivery.get("status") == "sam3d_ready_degraded":
+ manifest["status"] = "success_degraded"
+ elif sam3d_delivery.get("status") == "sam3d_ready":
+ manifest["status"] = "success"
+ else:
+ manifest["status"] = "infrastructure_failed"
+ atomic_json(output_dir / "run_manifest.json", manifest)
+
+ print("status:", manifest["status"])
+ print("scene_name:", config["scene_name"])
+ print("pass_a_objects:", len(pass_a_objects))
+ print("pass_b_additions:", len(pass_b_actions["additions"]))
+ print("top_zone_raw_candidates:", top_raw_result["registry"].get("candidate_count", 0))
+ print("top_zone_review_candidates:", len(local_candidate_metadata))
+ print("top_zone_bound_objects:", local_binding_audit["bound_object_count"])
+ print("fused_objects:", len(fused_objects))
+ print("affected_prompts:", len(affected_prompts))
+ print("sam3_model_load_count:", session.model_load_count)
+ print("sam3_inference_run_count:", session.inference_run_count)
+ print(
+ "bbox_failed_mask_rescue:",
+ bbox_rescue_summary["status"],
+ f"accepted={bbox_rescue_summary['accepted_count']}",
+ f"inferences={bbox_rescue_summary['inference_call_count']}",
+ )
+ print(
+ "post_delta_multiplicity_closure:",
+ multiplicity_summary["status"],
+ f"accepted={multiplicity_summary['accepted_count']}",
+ "additional_inferences=0",
+ )
+ print("quality_status:", quality["status"])
+ print("formal_outputs_modified:", not formal_comparison["unchanged"])
+ print("result_dir:", publish_dir)
+ print("sam3d_delivery_status:", sam3d_delivery.get("status"))
+ print(
+ "sam3d_execution_allowed:",
+ sam3d_delivery.get("sam3d_execution_allowed"),
+ )
+ print(
+ "contact_sheet:",
+ publish_dir / "sam3_overlay_contact_sheet.jpg",
+ )
+ return 0 if sam3d_delivery.get("sam3d_execution_allowed") else 3
+ except Exception as exc:
+ formal_after = snapshot_tree(formal_paths)
+ formal_comparison = compare_snapshots(formal_before, formal_after)
+ atomic_json(output_dir / "formal_hashes_after.json", formal_after)
+ atomic_json(output_dir / "formal_hash_comparison.json", formal_comparison)
+ manifest.update(
+ {
+ "status": "failed",
+ "finished_at_utc": utc_now(),
+ "error": {
+ "type": type(exc).__name__,
+ "message": str(exc),
+ },
+ "formal_outputs_modified": not formal_comparison["unchanged"],
+ }
+ )
+ atomic_json(output_dir / "run_manifest.json", manifest)
+ raise
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/vlm_agent/reconstruction/perception/segmentation/__init__.py b/vlm_agent/reconstruction/perception/segmentation/__init__.py
new file mode 100644
index 0000000..c8e3866
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/segmentation/__init__.py
@@ -0,0 +1,21 @@
+"""Production SAM3 segmentation runtime used by SceneMosaic perception."""
+
+from .runtime import (
+ ASSIGNMENT_POLICIES,
+ DEFAULT_MODEL_DIR,
+ Sam3Session,
+ infer_text_candidates,
+ load_session,
+ prompt_groups,
+ segment_registry,
+)
+
+__all__ = [
+ "ASSIGNMENT_POLICIES",
+ "DEFAULT_MODEL_DIR",
+ "Sam3Session",
+ "infer_text_candidates",
+ "load_session",
+ "prompt_groups",
+ "segment_registry",
+]
diff --git a/vlm_agent/reconstruction/perception/segmentation/conservative_assignment.py b/vlm_agent/reconstruction/perception/segmentation/conservative_assignment.py
new file mode 100644
index 0000000..7540b3e
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/segmentation/conservative_assignment.py
@@ -0,0 +1,848 @@
+#!/usr/bin/env python3
+"""Conservative post-assignment recovery for production SAM3 segmentation.
+
+The primary SAM3 assignment remains authoritative. This module only plans
+one-to-one reuse of candidates that production left unassigned. It never
+steals a successful candidate and never performs global category-order
+matching.
+"""
+
+from __future__ import annotations
+
+import copy
+import math
+import re
+from collections import defaultdict
+from functools import lru_cache
+from typing import Any, Callable, Mapping, Sequence
+
+import numpy as np
+from PIL import Image
+
+
+CARRIER_ORDER_MAXIMUM_CENTER_OBJECT_SCALE = 1.25
+CARRIER_ORDER_MAXIMUM_CENTER_CARRIER_SCALE = 0.20
+CARRIER_ORDER_MAXIMUM_ANCHOR_GAP_OBJECT_SCALE = 1.00
+
+
+def bbox_area(bbox):
+ if bbox is None:
+ return 0
+ return max(0, bbox[2] - bbox[0]) * max(0, bbox[3] - bbox[1])
+
+
+def bbox_intersection(left, right):
+ if left is None or right is None:
+ return 0
+ return max(0, min(left[2], right[2]) - max(left[0], right[0])) * max(
+ 0, min(left[3], right[3]) - max(left[1], right[1])
+ )
+
+
+def bbox_iou(left, right):
+ intersection = bbox_intersection(left, right)
+ union = bbox_area(left) + bbox_area(right) - intersection
+ return intersection / union if union else 0.0
+
+
+def bbox_center(bbox):
+ if bbox is None:
+ return None
+ return ((bbox[0] + bbox[2]) / 2.0, (bbox[1] + bbox[3]) / 2.0)
+
+
+def normalized_bbox_to_abs(bbox, width, height):
+ if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
+ return None
+ return [
+ max(0, min(width, int(round(float(bbox[0]) * width)))),
+ max(0, min(height, int(round(float(bbox[1]) * height)))),
+ max(0, min(width, int(round(float(bbox[2]) * width)))),
+ max(0, min(height, int(round(float(bbox[3]) * height)))),
+ ]
+
+
+def object_point(obj, width, height):
+ anchor = obj.get("anchor_2d_norm_xy")
+ if (
+ isinstance(anchor, (list, tuple))
+ and len(anchor) == 2
+ and all(isinstance(value, (int, float)) for value in anchor)
+ ):
+ return (float(anchor[0]) * width, float(anchor[1]) * height)
+ return bbox_center(
+ normalized_bbox_to_abs(obj.get("bbox_2d_norm_xyxy"), width, height)
+ )
+
+
+def contains_point(bbox, point):
+ return bool(
+ bbox is not None
+ and point is not None
+ and bbox[0] <= point[0] <= bbox[2]
+ and bbox[1] <= point[1] <= bbox[3]
+ )
+
+
+def point_bbox_distance(point, bbox):
+ if point is None or bbox is None:
+ return math.inf
+ dx = max(float(bbox[0]) - point[0], 0.0, point[0] - float(bbox[2]))
+ dy = max(float(bbox[1]) - point[1], 0.0, point[1] - float(bbox[3]))
+ return math.hypot(dx, dy)
+
+
+def carrier_order_geometry_evidence(
+ obj,
+ candidate_bbox,
+ carrier_bbox,
+ width,
+ height,
+):
+ target_bbox = normalized_bbox_to_abs(
+ obj.get("bbox_2d_norm_xyxy"), width, height
+ )
+ anchor = object_point(obj, width, height)
+ target_center = bbox_center(target_bbox)
+ candidate_center = bbox_center(candidate_bbox)
+ if (
+ target_bbox is None
+ or candidate_bbox is None
+ or carrier_bbox is None
+ or anchor is None
+ or target_center is None
+ or candidate_center is None
+ or bbox_area(target_bbox) <= 0
+ or bbox_area(candidate_bbox) <= 0
+ or bbox_area(carrier_bbox) <= 0
+ ):
+ return {
+ "status": "reject",
+ "reason": "invalid_geometry",
+ }
+ target_diagonal = math.hypot(
+ target_bbox[2] - target_bbox[0],
+ target_bbox[3] - target_bbox[1],
+ )
+ candidate_diagonal = math.hypot(
+ candidate_bbox[2] - candidate_bbox[0],
+ candidate_bbox[3] - candidate_bbox[1],
+ )
+ carrier_diagonal = math.hypot(
+ carrier_bbox[2] - carrier_bbox[0],
+ carrier_bbox[3] - carrier_bbox[1],
+ )
+ object_scale = max(target_diagonal, candidate_diagonal, 1.0)
+ carrier_scale = max(carrier_diagonal, 1.0)
+ center_distance = math.dist(target_center, candidate_center)
+ center_object_scale = center_distance / object_scale
+ center_carrier_scale = center_distance / carrier_scale
+ anchor_gap_object_scale = (
+ point_bbox_distance(anchor, candidate_bbox) / object_scale
+ )
+ accepted = (
+ center_object_scale <= CARRIER_ORDER_MAXIMUM_CENTER_OBJECT_SCALE
+ and center_carrier_scale <= CARRIER_ORDER_MAXIMUM_CENTER_CARRIER_SCALE
+ and anchor_gap_object_scale
+ <= CARRIER_ORDER_MAXIMUM_ANCHOR_GAP_OBJECT_SCALE
+ )
+ return {
+ "status": "accept" if accepted else "reject",
+ "reason": None if accepted else "candidate_object_geometry_infeasible",
+ "target_bbox_abs_xyxy": target_bbox,
+ "candidate_bbox_abs_xyxy": list(candidate_bbox),
+ "carrier_bbox_abs_xyxy": list(carrier_bbox),
+ "bbox_iou": round(float(bbox_iou(target_bbox, candidate_bbox)), 6),
+ "anchor_in_candidate_bbox": contains_point(candidate_bbox, anchor),
+ "center_distance_object_scale": round(center_object_scale, 6),
+ "center_distance_carrier_scale": round(center_carrier_scale, 6),
+ "anchor_gap_object_scale": round(anchor_gap_object_scale, 6),
+ "thresholds": {
+ "maximum_center_object_scale": (
+ CARRIER_ORDER_MAXIMUM_CENTER_OBJECT_SCALE
+ ),
+ "maximum_center_carrier_scale": (
+ CARRIER_ORDER_MAXIMUM_CENTER_CARRIER_SCALE
+ ),
+ "maximum_anchor_gap_object_scale": (
+ CARRIER_ORDER_MAXIMUM_ANCHOR_GAP_OBJECT_SCALE
+ ),
+ },
+ }
+
+
+def strictly_contains(outer, inner):
+ return bool(
+ outer is not None
+ and inner is not None
+ and bbox_area(outer) > bbox_area(inner)
+ and outer[0] <= inner[0]
+ and outer[1] <= inner[1]
+ and outer[2] >= inner[2]
+ and outer[3] >= inner[3]
+ )
+
+
+def candidate_index(record):
+ explicit = record.get("candidate_index")
+ if isinstance(explicit, int) and explicit >= 0:
+ return explicit
+ match = re.search(r"_(\d{3})$", str(record.get("candidate_id") or ""))
+ return int(match.group(1)) if match else None
+
+
+def unique_minimal_carrier(
+ point,
+ unit_bbox,
+ carrier_objects,
+ width,
+ height,
+ excluded_ids,
+):
+ """Find one minimal enclosing object, or abstain on overlap ambiguity."""
+ if point is None or unit_bbox is None:
+ return None, []
+ unit_area = bbox_area(unit_bbox)
+ records = []
+ for obj in carrier_objects:
+ object_id = str(obj.get("object_id") or "")
+ if not object_id or object_id in excluded_ids:
+ continue
+ carrier_bbox = normalized_bbox_to_abs(
+ obj.get("bbox_2d_norm_xyxy"), width, height
+ )
+ if (
+ carrier_bbox is None
+ or not contains_point(carrier_bbox, point)
+ or not strictly_contains(carrier_bbox, unit_bbox)
+ ):
+ continue
+ records.append((object_id, carrier_bbox))
+ minimal = [
+ record
+ for record in records
+ if not any(
+ other[0] != record[0] and strictly_contains(record[1], other[1])
+ for other in records
+ )
+ ]
+ if len(minimal) != 1:
+ return None, sorted(object_id for object_id, _ in minimal)
+ return minimal[0][0], [minimal[0][0]]
+
+
+def principal_axis_orders(object_points, candidate_points):
+ labels = [
+ ("object", key, point) for key, point in object_points.items()
+ ] + [
+ ("candidate", key, point) for key, point in candidate_points.items()
+ ]
+ if len(labels) < 2 or any(point is None for _, _, point in labels):
+ return None
+ coordinates = np.asarray([point for _, _, point in labels], dtype=float)
+ centered = coordinates - coordinates.mean(axis=0, keepdims=True)
+ covariance = centered.T @ centered
+ eigenvalues, eigenvectors = np.linalg.eigh(covariance)
+ if float(eigenvalues[-1]) <= 0:
+ return None
+ axis = eigenvectors[:, -1]
+ dominant = int(np.argmax(np.abs(axis)))
+ if axis[dominant] < 0:
+ axis = -axis
+ projections = coordinates @ axis
+ object_projection = {}
+ candidate_projection = {}
+ for (kind, key, _), projection in zip(labels, projections):
+ target = object_projection if kind == "object" else candidate_projection
+ target[key] = float(projection)
+ if len(set(object_projection.values())) != len(object_projection):
+ return None
+ if len(set(candidate_projection.values())) != len(candidate_projection):
+ return None
+ object_order = sorted(object_projection, key=object_projection.get)
+ candidate_order = sorted(candidate_projection, key=candidate_projection.get)
+ return {
+ "axis": [round(float(value), 9) for value in axis],
+ "object_order": object_order,
+ "candidate_order": candidate_order,
+ "object_ranks": {key: rank for rank, key in enumerate(object_order)},
+ "candidate_ranks": {
+ key: rank for rank, key in enumerate(candidate_order)
+ },
+ }
+
+
+def unique_monotone_path(
+ object_ranks,
+ candidate_ranks,
+ object_count,
+ candidate_count,
+):
+ """Return a unique ordinal optimum, independent of image pixels."""
+ object_ranks = tuple(object_ranks)
+ candidate_ranks = tuple(candidate_ranks)
+ if len(candidate_ranks) < len(object_ranks):
+ return None
+ object_denominator = max(1, object_count - 1)
+ candidate_denominator = max(1, candidate_count - 1)
+
+ @lru_cache(maxsize=None)
+ def solve(object_offset, candidate_offset):
+ remaining_objects = len(object_ranks) - object_offset
+ remaining_candidates = len(candidate_ranks) - candidate_offset
+ if remaining_objects == 0:
+ return (0, 1, ())
+ if remaining_candidates < remaining_objects:
+ return (None, 0, ())
+ options = []
+ skipped = solve(object_offset, candidate_offset + 1)
+ if skipped[0] is not None:
+ options.append(skipped)
+ matched = solve(object_offset + 1, candidate_offset + 1)
+ if matched[0] is not None:
+ ordinal_cost = abs(
+ object_ranks[object_offset] * candidate_denominator
+ - candidate_ranks[candidate_offset] * object_denominator
+ )
+ options.append(
+ (
+ ordinal_cost + matched[0],
+ matched[1],
+ (candidate_ranks[candidate_offset],) + matched[2],
+ )
+ )
+ if not options:
+ return (None, 0, ())
+ minimum = min(option[0] for option in options)
+ best = [option for option in options if option[0] == minimum]
+ ways = min(2, sum(option[1] for option in best))
+ return (minimum, ways, best[0][2])
+
+ cost, ways, path = solve(0, 0)
+ if cost is None or ways != 1:
+ return None
+ return {"ordinal_cost": int(cost), "candidate_ranks": list(path)}
+
+
+def default_mask_loader(path):
+ return np.asarray(Image.open(path).convert("L")) > 0
+
+
+def masks_conflict(left_path, right_path, mask_loader):
+ """Reject material overlap while ignoring a few anti-aliased edge pixels."""
+ if not left_path or not right_path:
+ return True
+ try:
+ left = np.asarray(mask_loader(left_path), dtype=bool)
+ right = np.asarray(mask_loader(right_path), dtype=bool)
+ except (OSError, ValueError):
+ return True
+ if left.shape != right.shape:
+ return True
+ overlap = int(np.logical_and(left, right).sum())
+ if overlap == 0:
+ return False
+ smaller_area = min(int(left.sum()), int(right.sum()))
+ if smaller_area <= 0:
+ return True
+ return overlap >= 64 or overlap / smaller_area >= 0.01
+
+
+def _prompt_groups(objects, unassigned):
+ groups = defaultdict(lambda: {"objects": [], "unassigned": []})
+ for obj in objects:
+ sam3 = obj.get("sam3") or {}
+ prompt = str(sam3.get("prompt_used") or "").strip()
+ if prompt:
+ groups[prompt]["objects"].append(obj)
+ for candidate in unassigned:
+ prompt = str(candidate.get("sam3_prompt") or "").strip()
+ if prompt:
+ groups[prompt]["unassigned"].append(candidate)
+ return groups
+
+
+def plan_registry_fallbacks(
+ registry: Mapping[str, Any],
+ *,
+ carrier_objects: Sequence[Mapping[str, Any]] | None = None,
+ mask_loader: Callable[[str], np.ndarray] = default_mask_loader,
+):
+ """Plan conservative reuse without mutating the SAM3 registry.
+
+ carrier_objects may contain the full fused registry when registry
+ is a prompt-filtered delta run. Only registry objects can receive a
+ fallback; the extra objects are used solely as enclosing carrier evidence.
+ """
+ objects = copy.deepcopy(list(registry.get("objects") or []))
+ carrier_context = copy.deepcopy(
+ list(carrier_objects) if carrier_objects is not None else objects
+ )
+ carrier_by_id = {
+ str(item.get("object_id")): item
+ for item in carrier_context
+ if str(item.get("object_id") or "")
+ }
+ unassigned = copy.deepcopy(
+ list(registry.get("unassigned_sam3_candidates") or [])
+ )
+ width, height = [int(value) for value in registry.get("image_size") or []]
+ if width <= 0 or height <= 0:
+ raise ValueError("registry image_size must be [width, height]")
+
+ matches = []
+ abstentions = []
+ successful_mask_paths = [
+ (obj.get("sam3") or {}).get("mask_path")
+ for obj in objects
+ if (obj.get("sam3") or {}).get("status") == "success"
+ ]
+ for prompt, group in _prompt_groups(objects, unassigned).items():
+ target_objects = group["objects"]
+ failed = [
+ obj for obj in target_objects
+ if (obj.get("sam3") or {}).get("status") != "success"
+ ]
+ if not failed or not group["unassigned"]:
+ continue
+
+ candidates = {}
+ locked = {}
+ for obj in target_objects:
+ sam3 = obj.get("sam3") or {}
+ if sam3.get("status") != "success":
+ continue
+ indices = list(sam3.get("merged_candidate_indices") or [])
+ if not indices and isinstance(
+ sam3.get("candidate_index_in_prompt_group"), int
+ ):
+ indices = [sam3["candidate_index_in_prompt_group"]]
+ if len(indices) != 1:
+ locked[str(obj["object_id"])] = None
+ continue
+ index = int(indices[0])
+ candidates[index] = {
+ "candidate_index": index,
+ "candidate_id": f"locked_{prompt}_{index:03d}",
+ "bbox": list(sam3.get("mask_bbox_abs_xyxy") or []),
+ "mask_path": sam3.get("mask_path"),
+ "assigned_object_id": str(obj["object_id"]),
+ "record": None,
+ }
+ locked[str(obj["object_id"])] = index
+ for record in group["unassigned"]:
+ index = candidate_index(record)
+ bbox = record.get("mask_bbox_abs_xyxy")
+ if index is None or not isinstance(bbox, list) or len(bbox) != 4:
+ continue
+ if index in candidates:
+ abstentions.append(
+ {
+ "prompt": prompt,
+ "candidate_id": record.get("candidate_id"),
+ "reason": "duplicate_candidate_index",
+ }
+ )
+ continue
+ candidates[index] = {
+ "candidate_index": index,
+ "candidate_id": str(record["candidate_id"]),
+ "bbox": list(bbox),
+ "mask_path": record.get("mask_path"),
+ "assigned_object_id": None,
+ "record": record,
+ }
+
+ used_indices = {
+ index for index in locked.values() if isinstance(index, int)
+ }
+ available = {
+ index: value
+ for index, value in candidates.items()
+ if index not in used_indices and value["record"] is not None
+ }
+
+ # Independent image-edge containment fallback. This is intentionally
+ # not generalized to ordinary containment.
+ edge_options = defaultdict(list)
+ edge_owners = defaultdict(list)
+ locked_mask_paths = list(successful_mask_paths)
+ for obj in failed:
+ object_id = str(obj["object_id"])
+ object_bbox = normalized_bbox_to_abs(
+ obj.get("bbox_2d_norm_xyxy"), width, height
+ )
+ if object_bbox is None or bbox_area(object_bbox) <= 0:
+ continue
+ object_edges = set()
+ if object_bbox[0] == 0:
+ object_edges.add("left")
+ if object_bbox[1] == 0:
+ object_edges.add("top")
+ if object_bbox[2] == width:
+ object_edges.add("right")
+ if object_bbox[3] == height:
+ object_edges.add("bottom")
+ for index, candidate in available.items():
+ candidate_bbox = candidate["bbox"]
+ candidate_edges = set()
+ if candidate_bbox[0] == 0:
+ candidate_edges.add("left")
+ if candidate_bbox[1] == 0:
+ candidate_edges.add("top")
+ if candidate_bbox[2] == width:
+ candidate_edges.add("right")
+ if candidate_bbox[3] == height:
+ candidate_edges.add("bottom")
+ shared_edges = object_edges & candidate_edges
+ object_area = bbox_area(object_bbox)
+ coverage = (
+ bbox_intersection(candidate_bbox, object_bbox) / object_area
+ if object_area
+ else 0.0
+ )
+ if (
+ not shared_edges
+ or bbox_area(candidate_bbox) < object_area
+ or coverage < 0.90
+ ):
+ continue
+ if any(
+ masks_conflict(
+ candidate["mask_path"], locked_path, mask_loader
+ )
+ for locked_path in locked_mask_paths
+ ):
+ continue
+ option = {
+ "candidate_index": index,
+ "candidate_id": candidate["candidate_id"],
+ "shared_image_edges": sorted(shared_edges),
+ "object_bbox_coverage": round(float(coverage), 6),
+ }
+ edge_options[object_id].append(option)
+ edge_owners[index].append(object_id)
+ for obj in failed:
+ object_id = str(obj["object_id"])
+ options = [
+ option
+ for option in edge_options[object_id]
+ if len(edge_owners[option["candidate_index"]]) == 1
+ ]
+ if len(options) != 1:
+ if edge_options[object_id]:
+ abstentions.append(
+ {
+ "object_id": object_id,
+ "prompt": prompt,
+ "method": "shared_image_edge_object_bbox_containment",
+ "reason": "non_unique_edge_containment",
+ "candidate_ids": sorted(
+ option["candidate_id"]
+ for option in edge_options[object_id]
+ ),
+ }
+ )
+ continue
+ option = options[0]
+ matches.append(
+ {
+ "object_id": object_id,
+ "candidate_id": option["candidate_id"],
+ "candidate_index": option["candidate_index"],
+ "prompt": prompt,
+ "method": "shared_image_edge_object_bbox_containment",
+ "audit": option,
+ }
+ )
+ used_indices.add(option["candidate_index"])
+ available.pop(option["candidate_index"], None)
+ locked[object_id] = option["candidate_index"]
+
+ remaining_failed = [
+ obj for obj in failed if str(obj["object_id"]) not in locked
+ ]
+ if not remaining_failed or len(target_objects) < 2:
+ continue
+
+ excluded_ids = {str(obj["object_id"]) for obj in target_objects}
+ target_carriers = {}
+ target_carrier_alternatives = {}
+ for obj in target_objects:
+ object_id = str(obj["object_id"])
+ bbox = normalized_bbox_to_abs(
+ obj.get("bbox_2d_norm_xyxy"), width, height
+ )
+ carrier, alternatives = unique_minimal_carrier(
+ object_point(obj, width, height),
+ bbox,
+ carrier_context,
+ width,
+ height,
+ excluded_ids,
+ )
+ target_carriers[object_id] = carrier
+ target_carrier_alternatives[object_id] = alternatives
+
+ candidate_carriers = {}
+ for index, candidate in candidates.items():
+ carrier, _ = unique_minimal_carrier(
+ bbox_center(candidate["bbox"]),
+ candidate["bbox"],
+ carrier_context,
+ width,
+ height,
+ excluded_ids,
+ )
+ candidate_carriers[index] = carrier
+
+ carrier_ids = sorted(
+ {
+ target_carriers[str(obj["object_id"])]
+ for obj in remaining_failed
+ if target_carriers[str(obj["object_id"])] is not None
+ }
+ )
+ for obj in remaining_failed:
+ object_id = str(obj["object_id"])
+ if target_carriers[object_id] is None:
+ abstentions.append(
+ {
+ "object_id": object_id,
+ "prompt": prompt,
+ "method": "carrier_order",
+ "reason": "no_unique_carrier",
+ "minimal_carrier_candidates": (
+ target_carrier_alternatives[object_id]
+ ),
+ }
+ )
+
+ for carrier_id in carrier_ids:
+ carrier_record = carrier_by_id.get(carrier_id)
+ carrier_bbox = normalized_bbox_to_abs(
+ (carrier_record or {}).get("bbox_2d_norm_xyxy"),
+ width,
+ height,
+ )
+ if carrier_bbox is None or bbox_area(carrier_bbox) <= 0:
+ abstentions.extend(
+ {
+ "object_id": str(obj["object_id"]),
+ "prompt": prompt,
+ "method": "carrier_order",
+ "reason": "invalid_carrier_geometry",
+ "carrier_object_id": carrier_id,
+ }
+ for obj in remaining_failed
+ if target_carriers[str(obj["object_id"])] == carrier_id
+ )
+ continue
+ carrier_object_ids = [
+ str(obj["object_id"])
+ for obj in target_objects
+ if target_carriers[str(obj["object_id"])] == carrier_id
+ ]
+ carrier_candidate_indices = [
+ index
+ for index, candidate_carrier in candidate_carriers.items()
+ if candidate_carrier == carrier_id
+ ]
+ carrier_failed_ids = [
+ object_id
+ for object_id in carrier_object_ids
+ if object_id not in locked
+ ]
+ if not carrier_failed_ids:
+ continue
+ carrier_locked = [
+ (object_id, locked[object_id])
+ for object_id in carrier_object_ids
+ if isinstance(locked.get(object_id), int)
+ ]
+ if not carrier_locked or any(
+ candidate_carriers.get(index) != carrier_id
+ for _, index in carrier_locked
+ ):
+ abstentions.extend(
+ {
+ "object_id": object_id,
+ "prompt": prompt,
+ "method": "carrier_order",
+ "reason": "no_consistent_locked_pair",
+ "carrier_object_id": carrier_id,
+ }
+ for object_id in carrier_failed_ids
+ )
+ continue
+
+ object_by_id = {
+ str(obj["object_id"]): obj for obj in target_objects
+ }
+ order = principal_axis_orders(
+ {
+ object_id: object_point(
+ object_by_id[object_id], width, height
+ )
+ for object_id in carrier_object_ids
+ },
+ {
+ index: bbox_center(candidates[index]["bbox"])
+ for index in carrier_candidate_indices
+ },
+ )
+ if order is None:
+ abstentions.extend(
+ {
+ "object_id": object_id,
+ "prompt": prompt,
+ "method": "carrier_order",
+ "reason": "non_unique_principal_axis_order",
+ "carrier_object_id": carrier_id,
+ }
+ for object_id in carrier_failed_ids
+ )
+ continue
+ object_ranks = order["object_ranks"]
+ candidate_ranks = order["candidate_ranks"]
+ locked_ranks = sorted(
+ (
+ object_ranks[object_id],
+ candidate_ranks[index],
+ object_id,
+ index,
+ )
+ for object_id, index in carrier_locked
+ )
+ if any(
+ left[1] >= right[1]
+ for left, right in zip(locked_ranks, locked_ranks[1:])
+ ):
+ abstentions.extend(
+ {
+ "object_id": object_id,
+ "prompt": prompt,
+ "method": "carrier_order",
+ "reason": "locked_pairs_cross_in_order",
+ "carrier_object_id": carrier_id,
+ }
+ for object_id in carrier_failed_ids
+ )
+ continue
+
+ object_by_rank = {
+ rank: object_id for object_id, rank in object_ranks.items()
+ }
+ candidate_by_rank = {
+ rank: index for index, rank in candidate_ranks.items()
+ }
+ boundaries = [
+ (-1, -1, None, None),
+ *locked_ranks,
+ (
+ len(carrier_object_ids),
+ len(carrier_candidate_indices),
+ None,
+ None,
+ ),
+ ]
+ for lower, upper in zip(boundaries, boundaries[1:]):
+ gap_object_ranks = [
+ rank
+ for rank in range(lower[0] + 1, upper[0])
+ if object_by_rank[rank] not in locked
+ ]
+ if not gap_object_ranks:
+ continue
+ gap_candidate_ranks = [
+ rank
+ for rank in range(lower[1] + 1, upper[1])
+ if candidate_by_rank[rank] not in used_indices
+ ]
+ alignment = unique_monotone_path(
+ gap_object_ranks,
+ gap_candidate_ranks,
+ len(carrier_object_ids),
+ len(carrier_candidate_indices),
+ )
+ if alignment is None:
+ abstentions.extend(
+ {
+ "object_id": object_by_rank[rank],
+ "prompt": prompt,
+ "method": "carrier_order",
+ "reason": "no_unique_monotone_optimum",
+ "carrier_object_id": carrier_id,
+ "candidate_rank_options": gap_candidate_ranks,
+ }
+ for rank in gap_object_ranks
+ )
+ continue
+ for object_rank, candidate_rank in zip(
+ gap_object_ranks, alignment["candidate_ranks"]
+ ):
+ object_id = object_by_rank[object_rank]
+ index = candidate_by_rank[candidate_rank]
+ candidate = candidates[index]
+ geometry = carrier_order_geometry_evidence(
+ object_by_id[object_id],
+ candidate["bbox"],
+ carrier_bbox,
+ width,
+ height,
+ )
+ if geometry["status"] != "accept":
+ abstentions.append(
+ {
+ "object_id": object_id,
+ "prompt": prompt,
+ "method": "carrier_order",
+ "reason": "candidate_object_geometry_infeasible",
+ "candidate_id": candidate["candidate_id"],
+ "candidate_index": index,
+ "carrier_object_id": carrier_id,
+ "geometry": geometry,
+ }
+ )
+ continue
+ matches.append(
+ {
+ "object_id": object_id,
+ "candidate_id": candidate["candidate_id"],
+ "candidate_index": index,
+ "prompt": prompt,
+ "method": (
+ "carrier_partitioned_order_preserving_one_to_one"
+ ),
+ "audit": {
+ "carrier_object_id": carrier_id,
+ "principal_axis": order["axis"],
+ "object_rank": object_rank,
+ "candidate_rank": candidate_rank,
+ "ordinal_cost": alignment["ordinal_cost"],
+ "geometry": geometry,
+ "locked_pairs": [
+ {
+ "object_id": locked_object_id,
+ "candidate_index": locked_index,
+ }
+ for _, _, locked_object_id, locked_index
+ in locked_ranks
+ ],
+ },
+ }
+ )
+ used_indices.add(index)
+ locked[object_id] = index
+
+ matched_objects = [match["object_id"] for match in matches]
+ matched_candidates = [match["candidate_id"] for match in matches]
+ if len(matched_objects) != len(set(matched_objects)):
+ raise RuntimeError("fallback planned one object more than once")
+ if len(matched_candidates) != len(set(matched_candidates)):
+ raise RuntimeError("fallback planned one candidate more than once")
+ return {
+ "schema_version": "scenemosaic_conservative_assignment_plan",
+ "matches": matches,
+ "abstentions": abstentions,
+ }
diff --git a/vlm_agent/reconstruction/perception/segmentation/engine.py b/vlm_agent/reconstruction/perception/segmentation/engine.py
new file mode 100755
index 0000000..5bfa03d
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/segmentation/engine.py
@@ -0,0 +1,1414 @@
+#!/usr/bin/env python3
+import argparse
+import json
+import os
+import sys
+from collections import defaultdict
+from datetime import datetime, timezone
+from pathlib import Path
+
+
+np = None
+torch = None
+Image = None
+Sam3Model = None
+Sam3Processor = None
+
+_DEFAULT_RUNTIME_ROOT = Path(
+ os.environ.get(
+ "SCENEMOSAIC_RUNTIME_ROOT",
+ Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
+ / "scenemosaic",
+ )
+).expanduser().resolve()
+DEFAULT_MODEL_DIR = str(
+ Path(
+ os.environ.get(
+ "SCENEMOSAIC_PERCEPTION_MODEL_DIR",
+ _DEFAULT_RUNTIME_ROOT / "models" / "sam3",
+ )
+ ).expanduser().resolve()
+)
+SCORE_THRESHOLD = 0.30
+MASK_THRESHOLD = 0.50
+MIN_AREA = 30
+
+LOW_IOU_WARNING_THRESHOLD = 0.01
+OFF_BBOX_LOW_CONFIDENCE_SCORE = 0.70
+IOU_WEIGHT = 10.0
+SCORE_WEIGHT = 1.0
+CENTER_WEIGHT = 0.5
+
+# Merge candidates must stay local to one coarse VLM box. These guards are
+# intentionally category-independent so prompt text does not grow with every
+# new failure case.
+MERGE_BBOX_PAD_RATIO = 0.10
+MERGE_MIN_PAD_PIXELS = 12
+LOCAL_MIN_PAD_IMAGE_RATIO = 0.06
+MERGE_MIN_CANDIDATE_MASK_COVERAGE = 0.55
+MERGE_MAX_CLUSTER_GAP_RATIO = 0.008
+MERGE_MIN_CLUSTER_GAP_PIXELS = 6
+MERGE_MAX_CLUSTER_GAP_PIXELS = 18
+MERGE_MAX_UNION_BBOX_SPILL = 0.25
+MERGE_MAX_CROSS_OBJECT_CONTAINMENT = 0.80
+
+# The adaptive policy never uses a global pixel distance to decide semantic composition.
+# Distances are normalized by the current VLM object's own scale. Nearby
+# candidates are still not merged unless their masks substantially overlap:
+# low-overlap neighbors remain atomic evidence for the second VLM review.
+ADAPTIVE_LOCAL_MAX_NORMALIZED_GAP = 0.08
+ADAPTIVE_LOCAL_MAX_REPEATED_OBJECT_CENTER_DISTANCE = 1.35
+ADAPTIVE_LOCAL_MIN_CANDIDATE_COVERAGE_BY_CURRENT = 0.85
+ADAPTIVE_LOCAL_MIN_MASK_IOU_FOR_REDUNDANT_MERGE = 0.50
+
+
+def assignment_has_minimum_local_evidence(iou, score):
+ """Reject a weak text mask that does not touch its VLM localization box."""
+ return float(iou) > 0.0 or float(score) >= OFF_BBOX_LOW_CONFIDENCE_SCORE
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Run SAM3 segmentation for the no-wall-floor pipeline."
+ )
+ parser.add_argument(
+ "--scene_config",
+ help="Path to the scene_config.json for the current SceneMosaic run.",
+ )
+ parser.add_argument(
+ "--dry_run_config",
+ action="store_true",
+ help="Resolve input/output paths and exit without loading the SAM3 model.",
+ )
+ parser.set_defaults(assignment_policy="adaptive_local")
+ return parser.parse_args()
+
+
+def load_runtime_modules():
+ global np, torch, Image, Sam3Model, Sam3Processor
+
+ if np is not None:
+ return
+
+ import numpy as _np
+ import torch as _torch
+ from PIL import Image as _Image
+ from transformers import Sam3Model as _Sam3Model, Sam3Processor as _Sam3Processor
+
+ np = _np
+ torch = _torch
+ Image = _Image
+ Sam3Model = _Sam3Model
+ Sam3Processor = _Sam3Processor
+
+
+def safe_name(s):
+ return "".join(c if c.isalnum() or c in ["_", "-"] else "_" for c in str(s))
+
+
+def candidate_output_id(prompt, candidate_index):
+ return f"candidate_{safe_name(prompt)}_{candidate_index:03d}"
+
+
+def has_repeated_pass_b_additions(group_objects):
+ # Current fusion writes added; created remains accepted in previously materialized caches.
+ return sum(
+ any(
+ isinstance(source, dict)
+ and str(source.get("pass") or "").upper() == "B"
+ and str(source.get("action") or "").lower() in {"added", "created"}
+ for source in (obj.get("proposal_sources") or [])
+ )
+ for obj in group_objects
+ ) >= 2
+
+
+def get_object_id(obj):
+ return obj.get("object_id") or obj.get("image_object_id")
+
+
+def canonicalize_objects(objects):
+ canonical = []
+ seen = set()
+
+ for obj in objects:
+ object_id = get_object_id(obj)
+ if not object_id:
+ raise ValueError(f"Object is missing object_id/image_object_id: {obj}")
+ if object_id in seen:
+ raise ValueError(f"Duplicate object_id detected in VLM input: {object_id}")
+ seen.add(object_id)
+
+ normalized_obj = dict(obj)
+ normalized_obj.pop("image_object_id", None)
+ normalized_obj["object_id"] = object_id
+ canonical.append(normalized_obj)
+
+ return canonical
+
+
+def xyxy_norm_to_abs(bbox, w, h):
+ if bbox is None:
+ return None
+ x1, y1, x2, y2 = bbox
+ return [
+ max(0, min(w, int(round(x1 * w)))),
+ max(0, min(h, int(round(y1 * h)))),
+ max(0, min(w, int(round(x2 * w)))),
+ max(0, min(h, int(round(y2 * h)))),
+ ]
+
+
+def abs_to_norm_bbox(bbox, w, h):
+ x1, y1, x2, y2 = bbox
+ return [x1 / w, y1 / h, x2 / w, y2 / h]
+
+
+def mask_to_bbox_xyxy(mask_bool):
+ ys, xs = np.where(mask_bool)
+ if len(xs) == 0 or len(ys) == 0:
+ return None
+ return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
+
+
+def retain_target_local_components(mask_bool, target_bbox):
+ """Discard significant disconnected components with no support in one claim.
+
+ SAM3 can return one text candidate containing two separate instances of the
+ same prompt. A single inventory claim must not inherit a remote component
+ when another substantial component actually covers its own VLM bbox. The
+ guard is geometry-only: components touching the claim remain intact, so it
+ does not impose category-specific assembly rules.
+ """
+ if mask_bool is None or target_bbox is None or not mask_bool.any():
+ return mask_bool, {"status": "noop", "reason": "missing_mask_or_bbox"}
+ from scipy import ndimage
+
+ labels, count = ndimage.label(
+ mask_bool, structure=np.ones((3, 3), dtype=bool)
+ )
+ if int(count) <= 1:
+ return mask_bool, {"status": "noop", "component_count": int(count)}
+
+ x1, y1, x2, y2 = (int(value) for value in target_bbox)
+ total_area = int(mask_bool.sum())
+ minimum_significant_area = max(MIN_AREA, int(round(total_area * 0.08)))
+ supported_labels = set()
+ components = []
+ for label in range(1, int(count) + 1):
+ component = labels == label
+ area = int(component.sum())
+ if area == 0:
+ continue
+ bbox = mask_to_bbox_xyxy(component)
+ in_target = int(component[y1:y2, x1:x2].sum())
+ center_x = (bbox[0] + bbox[2]) / 2.0
+ center_y = (bbox[1] + bbox[3]) / 2.0
+ target_centered = x1 <= center_x <= x2 and y1 <= center_y <= y2
+ if in_target > 0 or target_centered:
+ supported_labels.add(label)
+ components.append(
+ {
+ "label": label,
+ "area_pixels": area,
+ "bbox_abs_xyxy": bbox,
+ "pixels_in_vlm_bbox": in_target,
+ "target_centered": target_centered,
+ }
+ )
+ if not supported_labels:
+ return mask_bool, {
+ "status": "noop",
+ "reason": "no_component_supported_by_vlm_bbox",
+ "component_count": len(components),
+ }
+ retained_labels = set(supported_labels)
+ excluded = []
+ for component in components:
+ label = int(component["label"])
+ if label in supported_labels:
+ continue
+ if int(component["area_pixels"]) >= minimum_significant_area:
+ excluded.append(component)
+ continue
+ retained_labels.add(label)
+ if not excluded:
+ return mask_bool, {
+ "status": "noop",
+ "component_count": len(components),
+ "supported_component_count": len(supported_labels),
+ }
+ retained = np.isin(labels, list(retained_labels))
+ return retained, {
+ "status": "filtered",
+ "component_count": len(components),
+ "supported_component_count": len(supported_labels),
+ "minimum_significant_area_pixels": minimum_significant_area,
+ "excluded_components": excluded,
+ }
+
+
+def bbox_area(b):
+ if b is None:
+ return 0
+ x1, y1, x2, y2 = b
+ return max(0, x2 - x1) * max(0, y2 - y1)
+
+
+def bbox_intersection(a, b):
+ if a is None or b is None:
+ return 0
+ ax1, ay1, ax2, ay2 = a
+ bx1, by1, bx2, by2 = b
+ ix1 = max(ax1, bx1)
+ iy1 = max(ay1, by1)
+ ix2 = min(ax2, bx2)
+ iy2 = min(ay2, by2)
+ return max(0, ix2 - ix1) * max(0, iy2 - iy1)
+
+
+def bbox_iou(a, b):
+ inter = bbox_intersection(a, b)
+ area_a = bbox_area(a)
+ area_b = bbox_area(b)
+ union = area_a + area_b - inter
+ if union <= 0:
+ return 0.0
+ return inter / union
+
+
+def bbox_center(b):
+ if b is None:
+ return None
+ x1, y1, x2, y2 = b
+ return ((x1 + x2) / 2, (y1 + y2) / 2)
+
+
+def center_similarity(a, b, w, h):
+ ca = bbox_center(a)
+ cb = bbox_center(b)
+ if ca is None or cb is None:
+ return 0.0
+ dx = (ca[0] - cb[0]) / w
+ dy = (ca[1] - cb[1]) / h
+ dist = (dx * dx + dy * dy) ** 0.5
+ return max(0.0, 1.0 - dist)
+
+
+def expand_bbox(bbox, w, h, pad_ratio=MERGE_BBOX_PAD_RATIO):
+ if bbox is None:
+ return None
+ x1, y1, x2, y2 = bbox
+ # First-pass VLM boxes are coarse hints. A small image-relative floor
+ # absorbs ordinary localization error without permitting remote fallback.
+ pad_x = max(
+ MERGE_MIN_PAD_PIXELS,
+ int(round((x2 - x1) * pad_ratio)),
+ int(round(w * LOCAL_MIN_PAD_IMAGE_RATIO)),
+ )
+ pad_y = max(
+ MERGE_MIN_PAD_PIXELS,
+ int(round((y2 - y1) * pad_ratio)),
+ int(round(h * LOCAL_MIN_PAD_IMAGE_RATIO)),
+ )
+ return [
+ max(0, x1 - pad_x),
+ max(0, y1 - pad_y),
+ min(w, x2 + pad_x),
+ min(h, y2 + pad_y),
+ ]
+
+
+def bbox_contains_point(bbox, point):
+ if bbox is None or point is None:
+ return False
+ x1, y1, x2, y2 = bbox
+ x, y = point
+ return x1 <= x <= x2 and y1 <= y <= y2
+
+
+def bbox_coverage(container_bbox, item_bbox):
+ area = bbox_area(item_bbox)
+ if area <= 0:
+ return 0.0
+ return bbox_intersection(container_bbox, item_bbox) / area
+
+
+def bbox_union(a, b):
+ if a is None:
+ return b
+ if b is None:
+ return a
+ return [
+ min(a[0], b[0]),
+ min(a[1], b[1]),
+ max(a[2], b[2]),
+ max(a[3], b[3]),
+ ]
+
+
+def bbox_gap(a, b):
+ if a is None or b is None:
+ return float("inf")
+ dx = max(0, max(a[0], b[0]) - min(a[2], b[2]))
+ dy = max(0, max(a[1], b[1]) - min(a[3], b[3]))
+ return (dx * dx + dy * dy) ** 0.5
+
+
+def mask_coverage_in_bbox(mask_bool, bbox):
+ if bbox is None:
+ return 0.0
+ total = int(mask_bool.sum())
+ if total <= 0:
+ return 0.0
+ x1, y1, x2, y2 = bbox
+ inside = int(mask_bool[y1:y2, x1:x2].sum())
+ return inside / total
+
+
+def mask_overlap_metrics(left_mask, right_mask):
+ left_area = int(left_mask.sum())
+ right_area = int(right_mask.sum())
+ intersection = int(np.logical_and(left_mask, right_mask).sum())
+ union = left_area + right_area - intersection
+ return {
+ "intersection_pixels": intersection,
+ "iou": intersection / union if union > 0 else 0.0,
+ "left_coverage": intersection / left_area if left_area > 0 else 0.0,
+ "right_coverage": intersection / right_area if right_area > 0 else 0.0,
+ }
+
+
+def object_scale_pixels(vlm_bbox):
+ """Return a resolution-independent area scale for merge-gap checks."""
+ return max(1.0, bbox_area(vlm_bbox) ** 0.5)
+
+
+def anchor_scale_pixels(vlm_bbox):
+ """Use the longer bbox side so thin/clipped instances are not over-penalized."""
+ if not vlm_bbox:
+ return 1.0
+ return max(
+ 1.0,
+ float(vlm_bbox[2] - vlm_bbox[0]),
+ float(vlm_bbox[3] - vlm_bbox[1]),
+ )
+
+
+def merge_candidate_metrics(obj, cand, w, h):
+ vlm_bbox = xyxy_norm_to_abs(obj.get("bbox_2d_norm_xyxy"), w, h)
+ expanded_bbox = expand_bbox(vlm_bbox, w, h)
+ cand_bbox = cand["mask_bbox_abs_xyxy"]
+ center_inside = bbox_contains_point(expanded_bbox, bbox_center(cand_bbox))
+ mask_coverage = mask_coverage_in_bbox(cand["mask_bool"], expanded_bbox)
+ return {
+ "vlm_bbox_abs_xyxy": vlm_bbox,
+ "expanded_vlm_bbox_abs_xyxy": expanded_bbox,
+ "candidate_bbox_coverage": bbox_coverage(expanded_bbox, cand_bbox),
+ "candidate_mask_coverage": mask_coverage,
+ "bbox_iou": bbox_iou(vlm_bbox, cand_bbox),
+ "center_similarity": center_similarity(vlm_bbox, cand_bbox, w, h),
+ "center_inside_expanded_bbox": center_inside,
+ "eligible": bool(
+ center_inside
+ and mask_coverage >= MERGE_MIN_CANDIDATE_MASK_COVERAGE
+ ),
+ }
+
+
+def allows_additional_candidates(obj, prompt_group_size):
+ """Only collections or a singleton may absorb extra connected masks."""
+ if prompt_group_size == 1:
+ return True
+ category = str(obj.get("category") or "").lower()
+ return category.endswith("_group") or category.endswith("_collection")
+
+
+def maximum_cardinality_local_anchors(eligible_by_candidate, object_count):
+ """Return a maximum-cardinality one-to-one local object/candidate match."""
+ edges_by_object = {oi: [] for oi in range(object_count)}
+ for ci, eligible in eligible_by_candidate.items():
+ for quality, oi, metrics in eligible:
+ edges_by_object[oi].append((quality, ci, metrics))
+ for edges in edges_by_object.values():
+ edges.sort(key=lambda item: item[0], reverse=True)
+
+ candidate_match = {}
+ object_match = {}
+
+ def augment(oi, seen_candidates):
+ for _, ci, metrics in edges_by_object[oi]:
+ if ci in seen_candidates:
+ continue
+ seen_candidates.add(ci)
+ previous = candidate_match.get(ci)
+ if previous is None or augment(previous[0], seen_candidates):
+ candidate_match[ci] = (oi, metrics)
+ object_match[oi] = (ci, metrics)
+ return True
+ return False
+
+ object_order = sorted(
+ range(object_count),
+ key=lambda oi: (len(edges_by_object[oi]) or 10**9, oi),
+ )
+ for oi in object_order:
+ if edges_by_object[oi]:
+ augment(oi, set())
+ return object_match
+
+
+def assign_scale_aware_candidates_uniquely(target_objs, candidates, w, h):
+ """Conservative assignment with object-relative geometry.
+
+ One local anchor is selected for every object whenever possible. Extra
+ candidates are automatically unioned only when they are largely redundant
+ with the current mask. Merely touching or nearby candidates remain
+ unclaimed so Stage 03 can decide whether they are a missing object, an
+ extension, a duplicate, or a component.
+ """
+ assignments = {oi: [] for oi in range(len(target_objs))}
+ diagnostics = {}
+
+ for oi, obj in enumerate(target_objs):
+ vlm_bbox = xyxy_norm_to_abs(obj.get("bbox_2d_norm_xyxy"), w, h)
+ diagnostics[oi] = {
+ "vlm_bbox_abs_xyxy": vlm_bbox,
+ "expanded_vlm_bbox_abs_xyxy": expand_bbox(vlm_bbox, w, h),
+ "object_scale_pixels": round(object_scale_pixels(vlm_bbox), 3),
+ "anchor_scale_pixels": round(anchor_scale_pixels(vlm_bbox), 3),
+ "max_normalized_gap": ADAPTIVE_LOCAL_MAX_NORMALIZED_GAP,
+ "eligible_candidate_indices": [],
+ "accepted_candidate_indices": [],
+ "ambiguous_candidate_indices": [],
+ "protected_shared_candidate_indices": [],
+ "rejected_far_candidate_indices": [],
+ "rejected_disconnected_candidate_indices": [],
+ "candidate_decisions": [],
+ "rejected_cross_object_overlap": None,
+ "forced_fallback_used": False,
+ }
+
+ eligible_by_candidate = {}
+ for ci, cand in enumerate(candidates):
+ eligible = []
+ for oi, obj in enumerate(target_objs):
+ metrics = merge_candidate_metrics(obj, cand, w, h)
+ if not metrics["eligible"]:
+ continue
+ vlm_bbox = diagnostics[oi]["vlm_bbox_abs_xyxy"]
+ vlm_center = bbox_center(vlm_bbox)
+ candidate_center = bbox_center(cand["mask_bbox_abs_xyxy"])
+ center_distance = (
+ (
+ (vlm_center[0] - candidate_center[0]) ** 2
+ + (vlm_center[1] - candidate_center[1]) ** 2
+ ) ** 0.5
+ )
+ normalized_center_distance = (
+ center_distance / anchor_scale_pixels(vlm_bbox)
+ )
+ metrics["normalized_center_distance"] = normalized_center_distance
+ if (
+ len(target_objs) > 1
+ and normalized_center_distance
+ > ADAPTIVE_LOCAL_MAX_REPEATED_OBJECT_CENTER_DISTANCE
+ ):
+ diagnostics[oi]["rejected_far_candidate_indices"].append(ci)
+ continue
+ quality = (
+ metrics["bbox_iou"],
+ metrics["candidate_mask_coverage"],
+ metrics["candidate_bbox_coverage"],
+ metrics["center_similarity"],
+ cand["score"],
+ )
+ eligible.append((quality, oi, metrics))
+ diagnostics[oi]["eligible_candidate_indices"].append(ci)
+ if eligible:
+ eligible_by_candidate[ci] = eligible
+
+ anchor_matches = maximum_cardinality_local_anchors(
+ eligible_by_candidate, len(target_objs)
+ )
+ anchored_candidates = set()
+ owned = {oi: [] for oi in range(len(target_objs))}
+ for oi, (ci, metrics) in anchor_matches.items():
+ owned[oi].append((ci, metrics))
+ anchored_candidates.add(ci)
+
+ # Shared unanchored candidates have uncertain ownership. Do not let the
+ # largest or coarsest object bbox consume them before Stage 03 review.
+ for ci, eligible in eligible_by_candidate.items():
+ if ci in anchored_candidates:
+ continue
+ eligible_for_addition = [
+ item for item in eligible
+ if allows_additional_candidates(target_objs[item[1]], len(target_objs))
+ ]
+ if not eligible_for_addition:
+ continue
+ if len(eligible_for_addition) > 1:
+ for _, oi, _ in eligible_for_addition:
+ diagnostics[oi]["protected_shared_candidate_indices"].append(ci)
+ continue
+ _, oi, metrics = eligible_for_addition[0]
+ owned[oi].append((ci, metrics))
+
+ for oi, records in owned.items():
+ if not records:
+ continue
+
+ vlm_bbox = diagnostics[oi]["vlm_bbox_abs_xyxy"]
+ scale = object_scale_pixels(vlm_bbox)
+ anchor_ci, _ = max(
+ records,
+ key=lambda item: (
+ bbox_iou(vlm_bbox, candidates[item[0]]["mask_bbox_abs_xyxy"]),
+ item[1]["candidate_mask_coverage"],
+ item[1]["center_similarity"],
+ candidates[item[0]]["score"],
+ ),
+ )
+ accepted = [anchor_ci]
+ current_mask = candidates[anchor_ci]["mask_bool"].copy()
+ current_bbox = candidates[anchor_ci]["mask_bbox_abs_xyxy"]
+ diagnostics[oi]["candidate_decisions"].append({
+ "candidate_index": anchor_ci,
+ "decision": "accepted_anchor",
+ })
+
+ pending = sorted(
+ (item for item in records if item[0] != anchor_ci),
+ key=lambda item: (
+ bbox_gap(current_bbox, candidates[item[0]]["mask_bbox_abs_xyxy"])
+ / scale,
+ -item[1]["candidate_mask_coverage"],
+ -candidates[item[0]]["score"],
+ ),
+ )
+ ambiguous = []
+ disconnected = []
+
+ for ci, _ in pending:
+ cand = candidates[ci]
+ gap_pixels = bbox_gap(current_bbox, cand["mask_bbox_abs_xyxy"])
+ normalized_gap = gap_pixels / scale
+ overlap = mask_overlap_metrics(current_mask, cand["mask_bool"])
+ decision = {
+ "candidate_index": ci,
+ "gap_pixels": round(float(gap_pixels), 3),
+ "normalized_gap": round(float(normalized_gap), 6),
+ "mask_iou": round(float(overlap["iou"]), 6),
+ "candidate_coverage_by_current": round(
+ float(overlap["right_coverage"]), 6
+ ),
+ "current_coverage_by_candidate": round(
+ float(overlap["left_coverage"]), 6
+ ),
+ }
+
+ if normalized_gap > ADAPTIVE_LOCAL_MAX_NORMALIZED_GAP:
+ decision["decision"] = "defer_scale_gap"
+ disconnected.append(ci)
+ diagnostics[oi]["candidate_decisions"].append(decision)
+ continue
+
+ redundant_overlap = (
+ overlap["right_coverage"]
+ >= ADAPTIVE_LOCAL_MIN_CANDIDATE_COVERAGE_BY_CURRENT
+ or overlap["iou"] >= ADAPTIVE_LOCAL_MIN_MASK_IOU_FOR_REDUNDANT_MERGE
+ )
+ if not redundant_overlap:
+ decision["decision"] = "defer_ambiguous_low_overlap"
+ ambiguous.append(ci)
+ diagnostics[oi]["candidate_decisions"].append(decision)
+ continue
+
+ proposed_bbox = bbox_union(current_bbox, cand["mask_bbox_abs_xyxy"])
+ spill = 1.0 - bbox_coverage(
+ diagnostics[oi]["expanded_vlm_bbox_abs_xyxy"], proposed_bbox
+ )
+ if spill > MERGE_MAX_UNION_BBOX_SPILL:
+ decision["decision"] = "defer_union_spill"
+ decision["union_bbox_spill"] = round(float(spill), 6)
+ ambiguous.append(ci)
+ diagnostics[oi]["candidate_decisions"].append(decision)
+ continue
+
+ decision["decision"] = "accepted_redundant_overlap"
+ diagnostics[oi]["candidate_decisions"].append(decision)
+ accepted.append(ci)
+ current_mask |= cand["mask_bool"]
+ current_bbox = proposed_bbox
+
+ assignments[oi] = sorted(accepted)
+ diagnostics[oi]["accepted_candidate_indices"] = sorted(accepted)
+ diagnostics[oi]["ambiguous_candidate_indices"] = sorted(ambiguous)
+ diagnostics[oi]["rejected_disconnected_candidate_indices"] = sorted(
+ disconnected
+ )
+
+ # Retain the cross-object containment safety gate for broad SAM3 masks.
+ def assignment_mask(indices):
+ merged = np.zeros((h, w), dtype=bool)
+ for ci in indices:
+ merged |= candidates[ci]["mask_bool"]
+ return merged
+
+ for left in range(len(target_objs)):
+ if not assignments[left]:
+ continue
+ for right in range(left + 1, len(target_objs)):
+ if not assignments[right]:
+ continue
+ left_mask = assignment_mask(assignments[left])
+ right_mask = assignment_mask(assignments[right])
+ overlap = mask_overlap_metrics(left_mask, right_mask)
+ containment = max(overlap["left_coverage"], overlap["right_coverage"])
+ if containment < MERGE_MAX_CROSS_OBJECT_CONTAINMENT:
+ continue
+
+ def locality_quality(oi, mask):
+ bbox = mask_to_bbox_xyxy(mask)
+ vlm_bbox = diagnostics[oi]["vlm_bbox_abs_xyxy"]
+ expanded_bbox = diagnostics[oi]["expanded_vlm_bbox_abs_xyxy"]
+ return (
+ mask_coverage_in_bbox(mask, expanded_bbox),
+ bbox_iou(vlm_bbox, bbox),
+ center_similarity(vlm_bbox, bbox, w, h),
+ -int(mask.sum()),
+ )
+
+ left_quality = locality_quality(left, left_mask)
+ right_quality = locality_quality(right, right_mask)
+ winner, loser = (
+ (left, right) if left_quality >= right_quality else (right, left)
+ )
+ diagnostics[loser]["rejected_cross_object_overlap"] = {
+ "winner_object_id": target_objs[winner]["object_id"],
+ "containment": round(float(containment), 6),
+ }
+ diagnostics[loser]["accepted_candidate_indices"] = []
+ assignments[loser] = []
+
+ claimed = {
+ ci for indices in assignments.values() for ci in indices
+ }
+ return assignments, claimed, diagnostics
+
+
+def tensor_or_array_to_numpy(x):
+ if torch.is_tensor(x):
+ return x.detach().cpu().numpy()
+ return np.array(x)
+
+
+def draw_overlay(image_np, mask_bool, color=(255, 0, 0)):
+ overlay = image_np.copy()
+ overlay[mask_bool] = (
+ 0.55 * overlay[mask_bool] + 0.45 * np.array(color)
+ ).astype(np.uint8)
+ return overlay
+
+
+def save_mask_overlay(mask_bool, image_np, base, mask_dir, overlays_dir, color=(255, 0, 0)):
+ mask_path = mask_dir / f"{base}_mask.png"
+ overlay_path = overlays_dir / f"{base}_overlay.png"
+ Image.fromarray(mask_bool.astype(np.uint8) * 255, mode="L").save(mask_path)
+ Image.fromarray(draw_overlay(image_np, mask_bool, color=color)).save(overlay_path)
+ return str(mask_path), str(overlay_path)
+
+
+def category_has_token(category, token):
+ """Match semantic category tokens without confusing e.g. bed and bedside."""
+ normalized = "".join(
+ character.lower() if character.isalnum() else "_"
+ for character in str(category or "")
+ )
+ return token.lower() in [part for part in normalized.split("_") if part]
+
+
+def exclude_independent_pillows_from_beds(objects, image_np):
+ """Make separately reconstructed pillows exclusive of their bed masks."""
+ load_runtime_modules()
+ successful_pillows = [
+ obj for obj in objects
+ if category_has_token(obj.get("category"), "pillow")
+ and obj.get("sam3", {}).get("status") == "success"
+ and obj.get("sam3", {}).get("mask_path")
+ and Path(obj["sam3"]["mask_path"]).is_file()
+ ]
+ edits = []
+ for bed in objects:
+ if (
+ not category_has_token(bed.get("category"), "bed")
+ # Categories such as bed_pillow name an independent pillow, not a
+ # bed. The more specific pillow role must take precedence.
+ or category_has_token(bed.get("category"), "pillow")
+ or bed.get("sam3", {}).get("status") != "success"
+ ):
+ continue
+ sam3 = dict(bed["sam3"])
+ mask_path = Path(sam3["mask_path"])
+ if not mask_path.is_file():
+ continue
+ before = np.array(Image.open(mask_path).convert("L")) > 0
+ after = before.copy()
+ excluded = []
+ for pillow in successful_pillows:
+ if pillow is bed or pillow.get("object_id") == bed.get("object_id"):
+ continue
+ pillow_mask = np.array(Image.open(pillow["sam3"]["mask_path"]).convert("L")) > 0
+ overlap = int(np.logical_and(after, pillow_mask).sum())
+ if overlap:
+ after[pillow_mask] = False
+ excluded.append({"pillow_object_id": pillow["object_id"], "pixels_removed": overlap})
+ if not excluded:
+ continue
+ if not after.any():
+ raise RuntimeError(f"pillow exclusion emptied bed mask: {bed['object_id']}")
+ Image.fromarray(after.astype(np.uint8) * 255, mode="L").save(mask_path)
+ overlay_path = Path(sam3["overlay_path"])
+ Image.fromarray(draw_overlay(image_np, after)).save(overlay_path)
+ before_area = int(before.sum())
+ sam3["area_pixels"] = int(after.sum())
+ sam3["mask_bbox_abs_xyxy"] = mask_to_bbox_xyxy(after)
+ sam3["manual_mask_edit"] = {
+ "action": "exclude_independent_pillow_masks",
+ "reason": "Independent pillows must not be reconstructed as part of a bed.",
+ "excluded_pillows": excluded,
+ }
+ bed["sam3"] = sam3
+ edits.append({
+ "bed_object_id": bed["object_id"],
+ "before_area_pixels": before_area,
+ "after_area_pixels": int(after.sum()),
+ "excluded_pillows": excluded,
+ })
+ return edits
+
+
+def rebuild_scene_overlay(objects, image_np):
+ """Render the combined overlay from final post-processed mask files."""
+ combined = image_np.copy()
+ rng = np.random.default_rng(42)
+ for obj in objects:
+ sam3 = obj.get("sam3", {})
+ mask_path = sam3.get("mask_path")
+ if sam3.get("status") != "success" or not mask_path or not Path(mask_path).is_file():
+ continue
+ mask = np.array(Image.open(mask_path).convert("L")) > 0
+ color = rng.integers(0, 255, size=3, dtype=np.uint8)
+ combined[mask] = (0.55 * combined[mask] + 0.45 * color).astype(np.uint8)
+ return combined
+
+
+def candidate_sort_key(cand):
+ b = cand["mask_bbox_abs_xyxy"]
+ c = bbox_center(b)
+ if c is None:
+ return (999999, 999999)
+ return (c[1], c[0])
+
+
+def object_sort_key(obj):
+ order = obj.get("visual_order") or {}
+ return (order.get("row", 999), order.get("col", 999), obj.get("object_id", ""))
+
+
+def category_level_prompt(category):
+ """Convert a repeated registry category into one reusable SAM3 prompt."""
+ prompt = str(category or "").strip().replace("_", " ")
+ for suffix in (" group", " collection"):
+ if prompt.endswith(suffix):
+ prompt = prompt[:-len(suffix)].strip()
+ break
+ return prompt
+
+
+def build_prompt_groups(objects):
+ """Group only identical declared SAM3 queries.
+
+ sam3_prompt is the semantic contract produced by the inventory stage.
+ Different prompts under one category can deliberately distinguish a
+ collection (for example books) from a compound instance (for example
+ stacked books). Collapsing them to a category word changes the object
+ requested from SAM3 and can turn a collection into unrelated part masks.
+ Exact duplicate prompts still share one call; the category is only a
+ fallback for incomplete inventory records.
+ """
+ groups = defaultdict(list)
+ for obj in objects:
+ prompt = str(obj.get("sam3_prompt") or "").strip()
+ if not prompt:
+ prompt = category_level_prompt(obj.get("category")) or "object"
+ groups[prompt].append(obj)
+ return groups
+
+
+def load_scene_config(scene_config_path):
+ config_path = Path(scene_config_path).expanduser().resolve()
+ if not config_path.exists():
+ raise FileNotFoundError(f"scene_config not found: {config_path}")
+
+ data = json.loads(config_path.read_text(encoding="utf-8"))
+ scene_name = data.get("scene_name") or config_path.parent.parent.name
+ run_root = Path(data["run_root"]).expanduser().resolve()
+ input_image = Path(data["input_image"]).expanduser().resolve()
+ stages = data.get("stages") or {}
+ vlm_first_stage = stages.get("vlm_first", "01_vlm_first")
+ sam3_stage = stages.get("sam3", "02_sam3")
+
+ output_dir = run_root / sam3_stage
+ return {
+ "scene_name": scene_name,
+ "run_root": run_root,
+ "input_image": input_image,
+ "vlm_objects_json": run_root / vlm_first_stage / "vlm_objects.json",
+ "vlm_run_info_json": run_root / vlm_first_stage / "vlm_first_run_info.json",
+ "output_dir": output_dir,
+ "masks_dir": output_dir / "masks",
+ "overlays_dir": output_dir / "overlays",
+ "unassigned_candidates_dir": output_dir / "unassigned_candidates",
+ "debug_dir": output_dir / "debug_prompt_candidates",
+ "sam3_objects_json": output_dir / "sam3_objects.json",
+ "sam3_overlay": output_dir / "sam3_overlay.png",
+ "run_info_json": output_dir / "sam3_run_info.json",
+ "scene_config_path": config_path,
+ }
+
+
+def load_runtime_config(args):
+ if args.scene_config:
+ return load_scene_config(args.scene_config)
+ raise ValueError("--scene_config must be provided.")
+
+
+def ensure_runtime_dirs(config):
+ for path in [
+ config["output_dir"],
+ config["masks_dir"],
+ config["overlays_dir"],
+ config["unassigned_candidates_dir"],
+ config["debug_dir"],
+ ]:
+ path.mkdir(parents=True, exist_ok=True)
+
+
+def write_json(path, data):
+ path.write_text(json.dumps(data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+
+
+def build_run_info(config, object_count_input, object_count_output):
+ return {
+ "stage": "02_sam3",
+ "scene_name": config["scene_name"],
+ "run_root": str(config["run_root"]),
+ "input_image": str(config["input_image"]),
+ "vlm_objects_json": str(config["vlm_objects_json"]),
+ "vlm_first_run_info": str(config["vlm_run_info_json"]),
+ "output_dir": str(config["output_dir"]),
+ "sam3_objects_json": str(config["sam3_objects_json"]),
+ "sam3_overlay": str(config["sam3_overlay"]),
+ "object_count_input": object_count_input,
+ "object_count_output": object_count_output,
+ "created_at": datetime.now(timezone.utc).isoformat(),
+ }
+
+
+def print_dry_run(config):
+ print("scene_name:", config["scene_name"])
+ print("input_image:", config["input_image"])
+ print("vlm_objects_json:", config["vlm_objects_json"])
+ print("output_dir:", config["output_dir"])
+ print("masks_dir:", config["masks_dir"])
+ print("overlays_dir:", config["overlays_dir"])
+ print("unassigned_candidates_dir:", config["unassigned_candidates_dir"])
+ print("sam3_objects_json:", config["sam3_objects_json"])
+ print("sam3_overlay:", config["sam3_overlay"])
+ print("run_info_json:", config["run_info_json"])
+
+
+def main():
+ args = parse_args()
+ try:
+ config = load_runtime_config(args)
+ except Exception as exc:
+ print(f"ERROR: {exc}", file=sys.stderr)
+ sys.exit(1)
+
+ if args.dry_run_config:
+ print_dry_run(config)
+ return
+
+ load_runtime_modules()
+ ensure_runtime_dirs(config)
+
+ image = Image.open(config["input_image"]).convert("RGB")
+ img_np = np.array(image).astype(np.uint8)
+ h, w = img_np.shape[:2]
+
+ registry = json.loads(config["vlm_objects_json"].read_text(encoding="utf-8"))
+ objects = canonicalize_objects(registry.get("objects", []))
+ assignment_policy = "adaptive_local"
+
+ print("image:", config["input_image"])
+ print("image size:", (w, h))
+ print("vlm json:", config["vlm_objects_json"])
+ print("num VLM objects:", len(objects))
+ print("assignment policy:", assignment_policy)
+
+ device = "cuda" if torch.cuda.is_available() else "cpu"
+ print("device =", device)
+ if device == "cpu":
+ print("WARNING: 当前是 CPU,建议切到有卡模式。")
+
+ processor = Sam3Processor.from_pretrained(DEFAULT_MODEL_DIR)
+ model = Sam3Model.from_pretrained(DEFAULT_MODEL_DIR).to(device)
+ model.eval()
+
+ groups = build_prompt_groups(objects)
+
+ print("\n=== PROMPT GROUPS ===")
+ for prompt, group_objects in groups.items():
+ print(f"{prompt!r}: {len(group_objects)} -> {[o['object_id'] for o in group_objects]}")
+
+ updated_by_id = {}
+ all_overlay = img_np.copy()
+ unassigned_candidates = []
+ capture_debug_candidates = bool(config.get("capture_debug_candidates", True))
+ capture_unassigned_candidates = bool(
+ config.get("capture_unassigned_candidates", True)
+ )
+ # Bulk raw candidates are expensive. Ambiguous candidates are a small
+ # exception because a later ownership resolver needs their exact masks.
+ capture_ambiguous_candidate_evidence = bool(
+ config.get("capture_ambiguous_candidate_evidence", True)
+ )
+ # A same-prompt group with two or more Pass-B additions is the only
+ # place where an extra unassigned candidate can close a repeated-instance
+ # count (for example a third stack of plates on one cart).
+ capture_post_delta_multiplicity_candidates = bool(
+ config.get("capture_post_delta_multiplicity_candidates", True)
+ )
+ rng = np.random.default_rng(42)
+
+ for prompt, group_objs in groups.items():
+ print("\n==============================")
+ print(f"PROMPT GROUP: {prompt!r}, objects={len(group_objs)}")
+ print("==============================")
+
+ inputs = processor(images=image, text=prompt, return_tensors="pt")
+ inputs = {k: v.to(device) if hasattr(v, "to") else v for k, v in inputs.items()}
+
+ with torch.no_grad():
+ outputs = model(**inputs)
+
+ results = processor.post_process_instance_segmentation(
+ outputs,
+ threshold=SCORE_THRESHOLD,
+ mask_threshold=MASK_THRESHOLD,
+ target_sizes=[(h, w)],
+ )[0]
+
+ masks = results.get("masks", [])
+ scores = results.get("scores", [])
+
+ print(f"raw SAM3 candidates: masks={len(masks)}, scores={len(scores)}")
+
+ candidates = []
+ for mi, mask in enumerate(masks):
+ mask_np = tensor_or_array_to_numpy(mask)
+ mask_bool = mask_np > 0
+ area = int(mask_bool.sum())
+ if area < MIN_AREA:
+ continue
+
+ score = float(scores[mi].detach().cpu()) if torch.is_tensor(scores[mi]) else float(scores[mi])
+ mask_bbox = mask_to_bbox_xyxy(mask_bool)
+
+ candidates.append({
+ "candidate_id": mi,
+ "mask_bool": mask_bool,
+ "score": score,
+ "area": area,
+ "mask_bbox_abs_xyxy": mask_bbox,
+ })
+
+ candidates = sorted(candidates, key=candidate_sort_key)
+ print("valid candidates:", len(candidates))
+
+ for ci, cand in enumerate(candidates):
+ if capture_debug_candidates:
+ dbg_base = f"{safe_name(prompt)}_cand_{ci:03d}"
+ save_mask_overlay(
+ cand["mask_bool"],
+ img_np,
+ dbg_base,
+ config["debug_dir"],
+ config["debug_dir"],
+ )
+ print(
+ f" cand {ci:03d}: score={cand['score']:.4f}, "
+ f"area={cand['area']}, bbox={cand['mask_bbox_abs_xyxy']}"
+ )
+
+ if len(candidates) == 0:
+ for obj in group_objs:
+ obj_out = dict(obj)
+ obj_out["sam3"] = {
+ "status": "failed",
+ "reason": "no_candidates_from_text_prompt",
+ "prompt_used": prompt,
+ "selection_method": f"sam3_prompt_group_{assignment_policy}",
+ }
+ updated_by_id[obj["object_id"]] = obj_out
+ print(f" {obj['object_id']} failed: no candidates")
+ continue
+
+ instance_objs = []
+ merge_objs = sorted(group_objs, key=object_sort_key)
+ unit_label = "SCALE-AWARE"
+ failure_reason = "no_local_candidate"
+ locality_field = "scale_aware_locality"
+ selection_method = "scale_aware_conservative_candidate_assignment"
+ merge_assignments, assigned_cands, merge_diagnostics = (
+ assign_scale_aware_candidates_uniquely(
+ merge_objs, candidates, w, h
+ )
+ )
+
+ ambiguity_evidence_indices = {
+ int(candidate_index)
+ for diagnostic in merge_diagnostics.values()
+ for candidate_index in (
+ diagnostic.get("ambiguous_candidate_indices", [])
+ or diagnostic.get("protected_shared_candidate_indices", [])
+ )
+ }
+ repeated_pass_b_addition_group = (
+ capture_post_delta_multiplicity_candidates
+ and has_repeated_pass_b_additions(group_objs)
+ )
+
+ for oi, obj in enumerate(merge_objs):
+ obj_id = obj["object_id"]
+ vlm_bbox_abs = xyxy_norm_to_abs(obj.get("bbox_2d_norm_xyxy"), w, h)
+ selected = merge_assignments[oi]
+
+ if not selected:
+ obj_out = dict(obj)
+ obj_out["sam3"] = {
+ "status": "failed",
+ "reason": failure_reason,
+ "prompt_used": prompt,
+ "vlm_bbox_abs_xyxy": vlm_bbox_abs,
+ locality_field: merge_diagnostics[oi],
+ "selection_method": selection_method,
+ }
+ updated_by_id[obj_id] = obj_out
+ print(f" {unit_label} {obj_id} failed: no local candidate")
+ continue
+
+ merged_mask = np.zeros((h, w), dtype=bool)
+ for ci in selected:
+ merged_mask |= candidates[ci]["mask_bool"]
+
+ merged_mask, component_locality = retain_target_local_components(
+ merged_mask, vlm_bbox_abs
+ )
+ merged_bbox = mask_to_bbox_xyxy(merged_mask)
+ merged_area = int(merged_mask.sum())
+ max_score = max(candidates[ci]["score"] for ci in selected)
+ iou = bbox_iou(vlm_bbox_abs, merged_bbox)
+ if not assignment_has_minimum_local_evidence(iou, max_score):
+ for ci in selected:
+ assigned_cands.discard(ci)
+ obj_out = dict(obj)
+ obj_out["sam3"] = {
+ "status": "failed",
+ "reason": "low_confidence_candidate_outside_vlm_bbox",
+ "prompt_used": prompt,
+ "score": max_score,
+ "vlm_bbox_abs_xyxy": vlm_bbox_abs,
+ "mask_bbox_abs_xyxy": merged_bbox,
+ "vlm_bbox_iou": iou,
+ locality_field: merge_diagnostics[oi],
+ "selection_method": selection_method,
+ }
+ updated_by_id[obj_id] = obj_out
+ print(
+ f" {unit_label} {obj_id} failed: weak candidate outside VLM bbox "
+ f"(score={max_score:.4f}, IoU={iou:.4f})"
+ )
+ continue
+
+ base = f"{safe_name(obj_id)}_{safe_name(obj.get('category', 'unknown'))}"
+ mask_path, overlay_path = save_mask_overlay(
+ merged_mask,
+ img_np,
+ base,
+ config["masks_dir"],
+ config["overlays_dir"],
+ )
+
+ color = rng.integers(0, 255, size=3, dtype=np.uint8)
+ all_overlay[merged_mask] = (
+ 0.55 * all_overlay[merged_mask] + 0.45 * color
+ ).astype(np.uint8)
+
+ obj_out = dict(obj)
+ obj_out["sam3"] = {
+ "status": "success",
+ "warning": None if iou >= LOW_IOU_WARNING_THRESHOLD else "low_iou_with_vlm_bbox",
+ "prompt_used": prompt,
+ "score": max_score,
+ "area_pixels": merged_area,
+ "vlm_bbox_abs_xyxy": vlm_bbox_abs,
+ "mask_bbox_abs_xyxy": merged_bbox,
+ "vlm_bbox_iou": iou,
+ "merged_candidate_indices": selected,
+ "component_locality": component_locality,
+ locality_field: merge_diagnostics[oi],
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ "selection_method": selection_method,
+ }
+ ambiguous = merge_diagnostics[oi].get(
+ "ambiguous_candidate_indices", []
+ )
+ protected = merge_diagnostics[oi].get(
+ "protected_shared_candidate_indices", []
+ )
+ obj_out["sam3"]["merge_review"] = {
+ "required": bool(ambiguous or protected),
+ "ambiguous_candidate_indices": ambiguous,
+ "ambiguous_candidate_ids": [
+ candidate_output_id(prompt, ci) for ci in ambiguous
+ ],
+ "protected_shared_candidate_indices": protected,
+ "protected_shared_candidate_ids": [
+ candidate_output_id(prompt, ci) for ci in protected
+ ],
+ "resolution_stage": (
+ "03_vlm_check" if ambiguous or protected else None
+ ),
+ }
+ updated_by_id[obj_id] = obj_out
+
+ print(
+ f" {unit_label} {obj_id} -> unique cands {selected}, "
+ f"area={merged_area}, IoU={iou:.4f}, bbox={merged_bbox}"
+ )
+
+ instance_objs = sorted(instance_objs, key=object_sort_key)
+ pairs = []
+ for oi, obj in enumerate(instance_objs):
+ obj_id = obj["object_id"]
+ vlm_bbox_abs = xyxy_norm_to_abs(obj.get("bbox_2d_norm_xyxy"), w, h)
+
+ for ci, cand in enumerate(candidates):
+ if ci in assigned_cands:
+ continue
+
+ cand_bbox = cand["mask_bbox_abs_xyxy"]
+ iou = bbox_iou(vlm_bbox_abs, cand_bbox)
+ center_sim = center_similarity(vlm_bbox_abs, cand_bbox, w, h)
+ rank_score = IOU_WEIGHT * iou + CENTER_WEIGHT * center_sim + SCORE_WEIGHT * cand["score"]
+
+ pairs.append({
+ "obj_index": oi,
+ "cand_index": ci,
+ "obj_id": obj_id,
+ "iou": iou,
+ "center_sim": center_sim,
+ "rank_score": rank_score,
+ })
+
+ pairs.sort(key=lambda x: x["rank_score"], reverse=True)
+
+ assigned_objs = set()
+ assignments = {}
+
+ for pair in pairs:
+ oi = pair["obj_index"]
+ ci = pair["cand_index"]
+
+ if oi in assigned_objs:
+ continue
+ if ci in assigned_cands:
+ continue
+
+ assigned_objs.add(oi)
+ assigned_cands.add(ci)
+ assignments[oi] = pair
+
+ for oi, obj in enumerate(instance_objs):
+ obj_id = obj["object_id"]
+ vlm_bbox_abs = xyxy_norm_to_abs(obj.get("bbox_2d_norm_xyxy"), w, h)
+ obj_out = dict(obj)
+
+ if oi not in assignments:
+ obj_out["sam3"] = {
+ "status": "failed",
+ "reason": "no_unassigned_candidate_available",
+ "prompt_used": prompt,
+ "vlm_bbox_abs_xyxy": vlm_bbox_abs,
+ "selection_method": "instance_mode_one_to_one_assignment",
+ }
+ updated_by_id[obj_id] = obj_out
+ print(f" INSTANCE {obj_id} failed: no candidate")
+ continue
+
+ assign = assignments[oi]
+ ci = assign["cand_index"]
+ cand = candidates[ci]
+
+ mask_bool = cand["mask_bool"]
+ base = f"{safe_name(obj_id)}_{safe_name(obj.get('category', 'unknown'))}"
+ mask_path, overlay_path = save_mask_overlay(
+ mask_bool,
+ img_np,
+ base,
+ config["masks_dir"],
+ config["overlays_dir"],
+ )
+
+ color = rng.integers(0, 255, size=3, dtype=np.uint8)
+ all_overlay[mask_bool] = (
+ 0.55 * all_overlay[mask_bool] + 0.45 * color
+ ).astype(np.uint8)
+
+ warning = None if assign["iou"] >= LOW_IOU_WARNING_THRESHOLD else "low_iou_with_vlm_bbox"
+
+ obj_out["sam3"] = {
+ "status": "success",
+ "warning": warning,
+ "prompt_used": prompt,
+ "score": cand["score"],
+ "area_pixels": cand["area"],
+ "vlm_bbox_abs_xyxy": vlm_bbox_abs,
+ "mask_bbox_abs_xyxy": cand["mask_bbox_abs_xyxy"],
+ "vlm_bbox_iou": assign["iou"],
+ "center_similarity": assign["center_sim"],
+ "candidate_index_in_prompt_group": ci,
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ "selection_method": "instance_mode_one_to_one_assignment",
+ }
+ updated_by_id[obj_id] = obj_out
+
+ print(
+ f" INSTANCE {obj_id} -> cand {ci:03d}, "
+ f"score={cand['score']:.4f}, IoU={assign['iou']:.4f}, "
+ f"center={assign['center_sim']:.4f}, warning={warning}"
+ )
+
+ for ci, cand in enumerate(candidates):
+ if ci in assigned_cands:
+ continue
+ retained_for_ambiguity = (
+ capture_ambiguous_candidate_evidence
+ and ci in ambiguity_evidence_indices
+ )
+ retained_for_multiplicity = repeated_pass_b_addition_group
+ if (
+ not capture_unassigned_candidates
+ and not retained_for_ambiguity
+ and not retained_for_multiplicity
+ ):
+ print(f" UNASSIGNED candidate {ci:03d}: not persisted by storage policy")
+ continue
+
+ cand_id = candidate_output_id(prompt, ci)
+ base = safe_name(cand_id)
+ mask_path, overlay_path = save_mask_overlay(
+ cand["mask_bool"],
+ img_np,
+ base,
+ config["unassigned_candidates_dir"],
+ config["unassigned_candidates_dir"],
+ color=(255, 0, 0),
+ )
+
+ candidate_record = {
+ "candidate_id": cand_id,
+ "source": "sam3_unassigned_candidate",
+ "category_hint": prompt.replace(" ", "_"),
+ "sam3_prompt": prompt,
+ "score": cand["score"],
+ "area_pixels": cand["area"],
+ "mask_bbox_abs_xyxy": cand["mask_bbox_abs_xyxy"],
+ "mask_bbox_norm_xyxy": abs_to_norm_bbox(cand["mask_bbox_abs_xyxy"], w, h),
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ "review_status": "pending",
+ "retention_reason": (
+ "merge_review_ambiguity"
+ if retained_for_ambiguity and not capture_unassigned_candidates
+ else "post_delta_repeated_instance_closure"
+ if retained_for_multiplicity and not capture_unassigned_candidates
+ else "unassigned_candidate_capture"
+ ),
+ "note": "SAM3 produced this candidate, but no VLM object was assigned to it. It may be a missed object instance or a duplicate/partial mask.",
+ }
+ unassigned_candidates.append(candidate_record)
+
+ print(f" UNASSIGNED {cand_id}: bbox={cand['mask_bbox_abs_xyxy']}, score={cand['score']:.4f}")
+
+ updated_objects = []
+ for obj in objects:
+ obj_id = obj["object_id"]
+ if obj_id in updated_by_id:
+ updated_objects.append(updated_by_id[obj_id])
+ else:
+ obj_out = dict(obj)
+ obj_out["sam3"] = {
+ "status": "failed",
+ "reason": "not_processed",
+ }
+ updated_objects.append(obj_out)
+
+ bed_pillow_edits = exclude_independent_pillows_from_beds(updated_objects, img_np)
+
+ out_registry = dict(registry)
+ out_registry["scene_name"] = config["scene_name"]
+ out_registry["image_path"] = str(config["input_image"])
+ out_registry["image_size"] = [w, h]
+ out_registry["objects"] = updated_objects
+ out_registry["unassigned_sam3_candidates"] = unassigned_candidates
+ out_registry["manual_bed_pillow_exclusions"] = bed_pillow_edits
+ out_registry["sam3_assignment_policy"] = assignment_policy
+
+ write_json(config["sam3_objects_json"], out_registry)
+ Image.fromarray(rebuild_scene_overlay(updated_objects, img_np)).save(config["sam3_overlay"])
+
+ run_info = build_run_info(
+ config=config,
+ object_count_input=len(objects),
+ object_count_output=len(updated_objects),
+ )
+ run_info["assignment_policy"] = assignment_policy
+ run_info["registry_version"] = registry.get("image_object_registry_version")
+ run_info["manual_bed_pillow_exclusions"] = bed_pillow_edits
+ write_json(config["run_info_json"], run_info)
+
+ success_count = sum(1 for o in updated_objects if o.get("sam3", {}).get("status") == "success")
+ failed_count = len(updated_objects) - success_count
+ low_iou_count = sum(1 for o in updated_objects if o.get("sam3", {}).get("warning") == "low_iou_with_vlm_bbox")
+
+ print("\n=== DONE ===")
+ print("success:", success_count)
+ print("failed :", failed_count)
+ print("low_iou warnings:", low_iou_count)
+ print("unassigned candidates:", len(unassigned_candidates))
+ print("saved json:", config["sam3_objects_json"])
+ print("sam3 overlay:", config["sam3_overlay"])
+ print("run info:", config["run_info_json"])
+ print("candidate dir:", config["unassigned_candidates_dir"])
+ print("debug candidates:", config["debug_dir"])
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/perception/segmentation/runtime.py b/vlm_agent/reconstruction/perception/segmentation/runtime.py
new file mode 100644
index 0000000..53cbcb1
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/segmentation/runtime.py
@@ -0,0 +1,804 @@
+#!/usr/bin/env python3
+"""Reusable, in-process SAM3 runtime for production perception attempts.
+
+The runtime delegates segmentation and output generation to :mod:`.engine`.
+It adds lifecycle management around that implementation: one model load,
+isolated attempt configuration, complete-prompt-group filtering, and a stable
+Python API.
+"""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+import os
+import shutil
+import threading
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Iterable, Mapping, Sequence
+
+from . import conservative_assignment as ASSIGNMENT_FALLBACK
+from . import engine as SEGMENTATION_ENGINE
+
+
+_DEFAULT_RUNTIME_ROOT = Path(
+ os.environ.get(
+ "SCENEMOSAIC_RUNTIME_ROOT",
+ Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
+ / "scenemosaic",
+ )
+).expanduser().resolve()
+DEFAULT_MODEL_DIR = Path(
+ os.environ.get(
+ "SCENEMOSAIC_PERCEPTION_MODEL_DIR",
+ _DEFAULT_RUNTIME_ROOT / "models" / "sam3",
+ )
+).expanduser().resolve()
+ASSIGNMENT_POLICIES = {"adaptive_local"}
+
+_production_module = None
+_production_module_lock = threading.Lock()
+
+
+def _utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def _load_production_module():
+ """Return the production segmentation engine exactly once."""
+ global _production_module
+ if _production_module is not None:
+ return _production_module
+
+ with _production_module_lock:
+ if _production_module is not None:
+ return _production_module
+ _production_module = SEGMENTATION_ENGINE
+ return _production_module
+
+
+def _registry_from_input(
+ objects: Sequence[Mapping[str, Any]] | Mapping[str, Any],
+ registry_metadata: Mapping[str, Any] | None,
+) -> dict[str, Any]:
+ if isinstance(objects, Mapping):
+ registry = copy.deepcopy(dict(objects))
+ if "objects" not in registry:
+ raise ValueError("Registry input must contain an 'objects' list.")
+ else:
+ registry = {"objects": copy.deepcopy(list(objects))}
+
+ if registry_metadata:
+ for key, value in registry_metadata.items():
+ if key == "objects":
+ continue
+ registry[key] = copy.deepcopy(value)
+ if not isinstance(registry.get("objects"), list):
+ raise TypeError("Registry 'objects' must be a list.")
+ return registry
+
+
+def prompt_groups(
+ objects: Sequence[Mapping[str, Any]] | Mapping[str, Any],
+) -> dict[str, list[dict[str, Any]]]:
+ """Return the exact prompt grouping used by the production SAM3 stage.
+
+ The returned dictionaries are copies. Callers may use the keys to choose
+ affected groups without mutating their source registry.
+ """
+ production = _load_production_module()
+ registry = _registry_from_input(objects, registry_metadata=None)
+ canonical = production.canonicalize_objects(registry["objects"])
+ groups = production.build_prompt_groups(canonical)
+ return {
+ str(prompt): copy.deepcopy(list(group_objects))
+ for prompt, group_objects in groups.items()
+ }
+
+
+def _select_complete_prompt_groups(
+ production,
+ registry: Mapping[str, Any],
+ prompt_filter: Iterable[str] | None,
+) -> tuple[dict[str, Any], list[str], list[str]]:
+ source_registry = copy.deepcopy(dict(registry))
+ canonical = production.canonicalize_objects(
+ source_registry.get("objects", [])
+ )
+ groups = production.build_prompt_groups(canonical)
+ available_prompts = list(groups)
+
+ if prompt_filter is None:
+ selected_prompts = available_prompts
+ else:
+ requested = []
+ seen = set()
+ for raw_prompt in prompt_filter:
+ prompt = str(raw_prompt)
+ if prompt not in seen:
+ requested.append(prompt)
+ seen.add(prompt)
+ unknown = [prompt for prompt in requested if prompt not in groups]
+ if unknown:
+ raise ValueError(
+ "Unknown SAM3 prompt group(s): "
+ + ", ".join(repr(prompt) for prompt in unknown)
+ + ". Available groups: "
+ + ", ".join(repr(prompt) for prompt in available_prompts)
+ )
+ requested_set = set(requested)
+ # Preserve production group order, not caller set order.
+ selected_prompts = [
+ prompt for prompt in available_prompts
+ if prompt in requested_set
+ ]
+
+ selected_ids = {
+ obj["object_id"]
+ for prompt in selected_prompts
+ for obj in groups[prompt]
+ }
+ source_registry["objects"] = [
+ obj for obj in canonical if obj["object_id"] in selected_ids
+ ]
+ source_registry["sam3_session_selection"] = {
+ "source_object_count": len(canonical),
+ "selected_object_count": len(source_registry["objects"]),
+ "available_prompts": available_prompts,
+ "selected_prompts": selected_prompts,
+ "complete_prompt_groups_only": True,
+ }
+ return source_registry, available_prompts, selected_prompts
+
+
+@dataclass
+class Sam3Session:
+ """One loaded SAM3 processor/model pair.
+
+ ``run_lock`` protects the temporary segmentation-engine bindings used by
+ ``segment_registry``. A session is intentionally serial; orchestration may
+ overlap a remote VLM request with this local GPU work, but must not issue
+ two GPU segmentation calls through the same session simultaneously.
+ """
+
+ processor: Any
+ model: Any
+ device: str
+ model_dir: Path
+ production: Any = field(repr=False)
+ loaded_at: str = field(default_factory=_utc_now)
+ model_load_count: int = 1
+ inference_run_count: int = 0
+ run_lock: threading.RLock = field(
+ default_factory=threading.RLock, repr=False
+ )
+
+
+def load_session(
+ model_dir: str | Path = DEFAULT_MODEL_DIR,
+ device: str | None = None,
+) -> Sam3Session:
+ """Load SAM3 once and return a reusable in-process session."""
+ production = _load_production_module()
+ production.load_runtime_modules()
+
+ resolved_model_dir = Path(model_dir).expanduser().resolve()
+ if not resolved_model_dir.exists():
+ raise FileNotFoundError(f"SAM3 model directory not found: {resolved_model_dir}")
+
+ auto_device = "cuda" if production.torch.cuda.is_available() else "cpu"
+ resolved_device = str(device or auto_device)
+ if resolved_device.startswith("cuda") and not production.torch.cuda.is_available():
+ raise RuntimeError(
+ f"CUDA device requested but torch.cuda.is_available() is false: {resolved_device}"
+ )
+
+ processor = production.Sam3Processor.from_pretrained(
+ str(resolved_model_dir)
+ )
+ model = production.Sam3Model.from_pretrained(
+ str(resolved_model_dir)
+ ).to(resolved_device)
+ model.eval()
+ return Sam3Session(
+ processor=processor,
+ model=model,
+ device=resolved_device,
+ model_dir=resolved_model_dir,
+ production=production,
+ )
+
+
+class _CachedProcessorProvider:
+ def __init__(self, session: Sam3Session):
+ self._session = session
+
+ def from_pretrained(self, _model_dir):
+ return self._session.processor
+
+
+class _CachedModelHandle:
+ def __init__(self, session: Sam3Session):
+ self._session = session
+
+ def to(self, requested_device):
+ requested = str(requested_device)
+ actual = str(self._session.device)
+ if requested.split(":", 1)[0] != actual.split(":", 1)[0]:
+ raise RuntimeError(
+ "Production SAM3 device selection differs from loaded session: "
+ f"production={requested!r}, session={actual!r}"
+ )
+ return self._session.model
+
+
+class _CachedModelProvider:
+ def __init__(self, session: Sam3Session):
+ self._session = session
+
+ def from_pretrained(self, _model_dir):
+ return _CachedModelHandle(self._session)
+
+
+def _build_runtime_config(
+ image_path: Path,
+ output_dir: Path,
+ input_registry_path: Path,
+ scene_name: str,
+) -> dict[str, Any]:
+ return {
+ "scene_name": scene_name,
+ "run_root": output_dir.parent,
+ "input_image": image_path,
+ "vlm_objects_json": input_registry_path,
+ "vlm_run_info_json": input_registry_path.parent / "vlm_first_run_info.json",
+ "output_dir": output_dir,
+ "masks_dir": output_dir / "masks",
+ "overlays_dir": output_dir / "overlays",
+ "unassigned_candidates_dir": output_dir / "unassigned_candidates",
+ "debug_dir": output_dir / "debug_prompt_candidates",
+ "sam3_objects_json": output_dir / "sam3_objects.json",
+ "sam3_overlay": output_dir / "sam3_overlay.png",
+ "run_info_json": output_dir / "sam3_run_info.json",
+ "scene_config_path": None,
+ }
+
+
+def _prepare_output_dir(output_dir: Path, overwrite: bool) -> None:
+ if output_dir.exists() and any(output_dir.iterdir()):
+ if not overwrite:
+ raise FileExistsError(
+ f"SAM3 session output directory is not empty: {output_dir}"
+ )
+ shutil.rmtree(output_dir)
+ output_dir.mkdir(parents=True, exist_ok=True)
+
+
+def _apply_post_assignment_fallbacks(
+ production,
+ output_registry: dict[str, Any],
+ carrier_objects: Sequence[Mapping[str, Any]],
+ config: Mapping[str, Any],
+ assignment_policy: str,
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ """Materialize only conservative adaptive fallback matches."""
+ if assignment_policy != "adaptive_local":
+ return output_registry, {
+ "schema_version": "scenemosaic_conservative_assignment_plan",
+ "status": "disabled_for_assignment_policy",
+ "matches": [],
+ "abstentions": [],
+ }
+ if not output_registry.get("unassigned_sam3_candidates"):
+ return output_registry, {
+ "schema_version": "scenemosaic_conservative_assignment_plan",
+ "status": "no_unassigned_candidates",
+ "matches": [],
+ "abstentions": [],
+ }
+
+ audit = ASSIGNMENT_FALLBACK.plan_registry_fallbacks(
+ output_registry,
+ carrier_objects=carrier_objects,
+ )
+ audit["status"] = "applied" if audit["matches"] else "abstained"
+ if not audit["matches"]:
+ output_registry["post_assignment_fallback_audit"] = audit
+ temporary = config["sam3_objects_json"].with_suffix(".json.tmp")
+ temporary.write_text(
+ json.dumps(output_registry, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, config["sam3_objects_json"])
+ return output_registry, audit
+
+ production.load_runtime_modules()
+ with production.Image.open(config["input_image"]) as source:
+ image_np = production.np.asarray(source.convert("RGB")).astype(
+ production.np.uint8
+ )
+ objects_by_id = {
+ str(obj["object_id"]): obj
+ for obj in output_registry.get("objects") or []
+ }
+ candidates_by_id = {
+ str(candidate["candidate_id"]): candidate
+ for candidate in output_registry.get("unassigned_sam3_candidates") or []
+ }
+ applied_candidate_ids = set()
+ for match in audit["matches"]:
+ object_id = str(match["object_id"])
+ candidate_id = str(match["candidate_id"])
+ obj = objects_by_id.get(object_id)
+ candidate = candidates_by_id.get(candidate_id)
+ if obj is None or candidate is None:
+ raise RuntimeError(
+ f"fallback plan references missing pair: {object_id}, {candidate_id}"
+ )
+ prior_sam3 = copy.deepcopy(dict(obj.get("sam3") or {}))
+ if prior_sam3.get("status") == "success":
+ raise RuntimeError(f"fallback attempted to replace success: {object_id}")
+ candidate_path = Path(str(candidate.get("mask_path") or ""))
+ if not candidate_path.is_file():
+ raise FileNotFoundError(
+ f"fallback candidate mask missing: {candidate_path}"
+ )
+ mask = production.np.asarray(
+ production.Image.open(candidate_path).convert("L")
+ ) > 0
+ if mask.shape != image_np.shape[:2] or not bool(mask.any()):
+ raise RuntimeError(
+ f"fallback candidate mask invalid for {candidate_id}: {mask.shape}"
+ )
+ base = (
+ f"{production.safe_name(object_id)}_"
+ f"{production.safe_name(obj.get('category', 'unknown'))}"
+ )
+ mask_path, overlay_path = production.save_mask_overlay(
+ mask,
+ image_np,
+ base,
+ config["masks_dir"],
+ config["overlays_dir"],
+ )
+ vlm_bbox = prior_sam3.get("vlm_bbox_abs_xyxy")
+ if not isinstance(vlm_bbox, list) or len(vlm_bbox) != 4:
+ vlm_bbox = production.xyxy_norm_to_abs(
+ obj.get("bbox_2d_norm_xyxy"),
+ image_np.shape[1],
+ image_np.shape[0],
+ )
+ mask_bbox = production.mask_to_bbox_xyxy(mask)
+ iou = production.bbox_iou(vlm_bbox, mask_bbox)
+ locality = copy.deepcopy(
+ prior_sam3.get("scale_aware_locality") or {}
+ )
+ locality["accepted_candidate_indices"] = [
+ int(match["candidate_index"])
+ ]
+ locality["forced_fallback_used"] = True
+ locality["post_assignment_fallback"] = copy.deepcopy(match)
+ obj["sam3"] = {
+ "status": "success",
+ "warning": (
+ None
+ if iou >= production.LOW_IOU_WARNING_THRESHOLD
+ else "low_iou_with_vlm_bbox"
+ ),
+ "prompt_used": str(match["prompt"]),
+ "score": float(candidate.get("score") or 0.0),
+ "area_pixels": int(mask.sum()),
+ "vlm_bbox_abs_xyxy": vlm_bbox,
+ "mask_bbox_abs_xyxy": mask_bbox,
+ "vlm_bbox_iou": float(iou),
+ "merged_candidate_indices": [int(match["candidate_index"])],
+ "scale_aware_locality": locality,
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ "selection_method": str(match["method"]),
+ "post_assignment_fallback": copy.deepcopy(match),
+ "merge_review": {
+ "required": False,
+ "resolution_stage": "post_assignment_conservative_fallback",
+ },
+ }
+ applied_candidate_ids.add(candidate_id)
+
+ output_registry["unassigned_sam3_candidates"] = [
+ candidate
+ for candidate in output_registry.get("unassigned_sam3_candidates") or []
+ if str(candidate.get("candidate_id")) not in applied_candidate_ids
+ ]
+ output_registry["post_assignment_fallback_audit"] = audit
+ temporary = config["sam3_objects_json"].with_suffix(".json.tmp")
+ temporary.write_text(
+ json.dumps(output_registry, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, config["sam3_objects_json"])
+ production.Image.fromarray(
+ production.rebuild_scene_overlay(
+ output_registry.get("objects") or [], image_np
+ )
+ ).save(config["sam3_overlay"])
+ return output_registry, audit
+
+
+def segment_registry(
+ session: Sam3Session,
+ image_path: str | Path,
+ objects: Sequence[Mapping[str, Any]] | Mapping[str, Any],
+ output_dir: str | Path,
+ assignment_policy: str = "adaptive_local",
+ prompt_filter: Iterable[str] | None = None,
+ *,
+ scene_name: str | None = None,
+ registry_metadata: Mapping[str, Any] | None = None,
+ capture_debug_candidates: bool = True,
+ capture_unassigned_candidates: bool = True,
+ capture_ambiguous_candidate_evidence: bool = True,
+ capture_post_delta_multiplicity_candidates: bool = True,
+ overwrite: bool = False,
+) -> dict[str, Any]:
+ """Segment one registry using an already-loaded SAM3 session.
+
+ ``prompt_filter`` contains exact keys returned by :func:`prompt_groups`.
+ Filtering happens only after production grouping, so all objects sharing a
+ selected prompt are always processed together.
+ """
+ if not isinstance(session, Sam3Session):
+ raise TypeError("session must be a Sam3Session returned by load_session().")
+ if assignment_policy not in ASSIGNMENT_POLICIES:
+ raise ValueError(
+ f"Unsupported assignment_policy={assignment_policy!r}; "
+ f"expected one of {sorted(ASSIGNMENT_POLICIES)}"
+ )
+
+ resolved_image = Path(image_path).expanduser().resolve()
+ if not resolved_image.is_file():
+ raise FileNotFoundError(f"SAM3 input image not found: {resolved_image}")
+ resolved_output = Path(output_dir).expanduser().resolve()
+ inferred_scene_name = scene_name or resolved_image.stem
+
+ registry = _registry_from_input(objects, registry_metadata)
+ filtered_registry, available_prompts, selected_prompts = (
+ _select_complete_prompt_groups(
+ session.production, registry, prompt_filter
+ )
+ )
+ if not filtered_registry["objects"]:
+ raise ValueError("No objects remain after applying prompt_filter.")
+
+ with session.run_lock:
+ _prepare_output_dir(resolved_output, overwrite=overwrite)
+ input_dir = resolved_output / "_session_input"
+ input_dir.mkdir(parents=True, exist_ok=True)
+ input_registry_path = input_dir / "vlm_objects.json"
+ input_registry_path.write_text(
+ json.dumps(filtered_registry, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ (input_dir / "vlm_first_run_info.json").write_text(
+ json.dumps(
+ {
+ "source": "sam3_session",
+ "scene_name": inferred_scene_name,
+ "created_at": _utc_now(),
+ },
+ ensure_ascii=False,
+ indent=2,
+ )
+ + "\n",
+ encoding="utf-8",
+ )
+ config = _build_runtime_config(
+ resolved_image,
+ resolved_output,
+ input_registry_path,
+ inferred_scene_name,
+ )
+ config["capture_debug_candidates"] = bool(capture_debug_candidates)
+ config["capture_unassigned_candidates"] = bool(
+ capture_unassigned_candidates
+ )
+ # Keep only masks that are required to resolve an adaptive-local
+ # ambiguity when bulk unassigned-candidate capture is disabled.
+ config["capture_ambiguous_candidate_evidence"] = bool(
+ capture_ambiguous_candidate_evidence
+ )
+ config["capture_post_delta_multiplicity_candidates"] = bool(
+ capture_post_delta_multiplicity_candidates
+ )
+
+ production = session.production
+ original_parse_args = production.parse_args
+ original_load_runtime_config = production.load_runtime_config
+ original_processor_class = production.Sam3Processor
+ original_model_class = production.Sam3Model
+ run_index = session.inference_run_count + 1
+
+ try:
+ production.parse_args = lambda: argparse.Namespace(
+ scene_config=None,
+ dry_run_config=False,
+ assignment_policy=assignment_policy,
+ )
+ production.load_runtime_config = lambda _args: config
+ production.Sam3Processor = _CachedProcessorProvider(session)
+ production.Sam3Model = _CachedModelProvider(session)
+ production.main()
+ finally:
+ production.parse_args = original_parse_args
+ production.load_runtime_config = original_load_runtime_config
+ production.Sam3Processor = original_processor_class
+ production.Sam3Model = original_model_class
+
+ session.inference_run_count = run_index
+
+ output_registry = json.loads(
+ config["sam3_objects_json"].read_text(encoding="utf-8")
+ )
+ output_registry, fallback_audit = _apply_post_assignment_fallbacks(
+ production,
+ output_registry,
+ registry.get("objects") or [],
+ config,
+ assignment_policy,
+ )
+ run_info = json.loads(
+ config["run_info_json"].read_text(encoding="utf-8")
+ )
+ session_info = {
+ "api": "sam3_session",
+ "model_dir": str(session.model_dir),
+ "device": session.device,
+ "loaded_at": session.loaded_at,
+ "model_load_count": session.model_load_count,
+ "inference_run_index": run_index,
+ "available_prompts": available_prompts,
+ "selected_prompts": selected_prompts,
+ "complete_prompt_groups_only": True,
+ }
+ run_info["sam3_session"] = session_info
+ run_info["post_assignment_fallback_audit"] = fallback_audit
+ config["run_info_json"].write_text(
+ json.dumps(run_info, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+
+ return {
+ "registry": output_registry,
+ "run_info": run_info,
+ "output_dir": str(resolved_output),
+ "sam3_objects_json": str(config["sam3_objects_json"]),
+ "sam3_overlay": str(config["sam3_overlay"]),
+ "run_info_json": str(config["run_info_json"]),
+ "available_prompts": available_prompts,
+ "selected_prompts": selected_prompts,
+ "model_load_count": session.model_load_count,
+ "inference_run_index": run_index,
+ }
+
+
+
+def infer_text_candidates(
+ session: Sam3Session,
+ image_path: str | Path,
+ prompt: str,
+ output_dir: str | Path,
+ *,
+ score_threshold: float | None = None,
+ mask_threshold: float | None = None,
+ min_area: int | None = None,
+ positive_bbox_abs_xyxy: Sequence[float] | None = None,
+ maximum_candidates: int | None = None,
+ overwrite: bool = False,
+) -> dict[str, Any]:
+ """Return all instance masks for one text concept without bbox assignment."""
+
+ if not isinstance(session, Sam3Session):
+ raise TypeError("session must be a Sam3Session returned by load_session().")
+ resolved_image = Path(image_path).expanduser().resolve()
+ if not resolved_image.is_file():
+ raise FileNotFoundError(f"SAM3 input image not found: {resolved_image}")
+ normalized_prompt = " ".join(str(prompt).split())
+ if not normalized_prompt:
+ raise ValueError("prompt must be non-empty")
+ positive_box = None
+ if positive_bbox_abs_xyxy is not None:
+ if (
+ len(positive_bbox_abs_xyxy) != 4
+ or not all(isinstance(value, (int, float)) for value in positive_bbox_abs_xyxy)
+ ):
+ raise ValueError("positive_bbox_abs_xyxy must contain four numbers")
+ positive_box = [float(value) for value in positive_bbox_abs_xyxy]
+ if not (
+ 0 <= positive_box[0] < positive_box[2]
+ and 0 <= positive_box[1] < positive_box[3]
+ ):
+ raise ValueError("positive_bbox_abs_xyxy must be a non-empty XYXY box")
+ resolved_output = Path(output_dir).expanduser().resolve()
+ production = session.production
+ production.load_runtime_modules()
+ resolved_score = float(
+ production.SCORE_THRESHOLD if score_threshold is None else score_threshold
+ )
+ resolved_mask = float(
+ production.MASK_THRESHOLD if mask_threshold is None else mask_threshold
+ )
+ resolved_min_area = int(production.MIN_AREA if min_area is None else min_area)
+ if not 0 <= resolved_score <= 1 or not 0 <= resolved_mask <= 1:
+ raise ValueError("score and mask thresholds must be in [0, 1]")
+ if resolved_min_area <= 0:
+ raise ValueError("min_area must be positive")
+ if maximum_candidates is not None and int(maximum_candidates) <= 0:
+ raise ValueError("maximum_candidates must be positive")
+
+ with session.run_lock:
+ _prepare_output_dir(resolved_output, overwrite=overwrite)
+ masks_dir = resolved_output / "masks"
+ overlays_dir = resolved_output / "overlays"
+ masks_dir.mkdir()
+ overlays_dir.mkdir()
+
+ image = production.Image.open(resolved_image).convert("RGB")
+ image_np = production.np.array(image).astype(production.np.uint8)
+ height, width = image_np.shape[:2]
+ processor_kwargs = {
+ "images": image,
+ "text": normalized_prompt,
+ "return_tensors": "pt",
+ }
+ if positive_box is not None:
+ processor_kwargs["input_boxes"] = [[positive_box]]
+ processor_kwargs["input_boxes_labels"] = [[1]]
+ inputs = session.processor(**processor_kwargs)
+ inputs = {
+ key: value.to(session.device) if hasattr(value, "to") else value
+ for key, value in inputs.items()
+ }
+ with production.torch.no_grad():
+ outputs = session.model(**inputs)
+ result = session.processor.post_process_instance_segmentation(
+ outputs,
+ threshold=resolved_score,
+ mask_threshold=resolved_mask,
+ target_sizes=[(height, width)],
+ )[0]
+ masks = result.get("masks", [])
+ scores = result.get("scores", [])
+ candidates = []
+ for raw_index, raw_mask in enumerate(masks):
+ mask_np = production.tensor_or_array_to_numpy(raw_mask)
+ mask_bool = mask_np > 0
+ area = int(mask_bool.sum())
+ if area < resolved_min_area:
+ continue
+ raw_score = scores[raw_index]
+ score = (
+ float(raw_score.detach().cpu())
+ if production.torch.is_tensor(raw_score)
+ else float(raw_score)
+ )
+ candidates.append(
+ {
+ "raw_candidate_index": raw_index,
+ "mask_bool": mask_bool,
+ "score": score,
+ "area": area,
+ "mask_bbox_abs_xyxy": production.mask_to_bbox_xyxy(mask_bool),
+ }
+ )
+ raw_candidate_count = len(candidates)
+ if positive_box is not None:
+ def spatial_rank(candidate):
+ box = candidate["mask_bbox_abs_xyxy"]
+ ix1 = max(float(positive_box[0]), float(box[0]))
+ iy1 = max(float(positive_box[1]), float(box[1]))
+ ix2 = min(float(positive_box[2]), float(box[2]))
+ iy2 = min(float(positive_box[3]), float(box[3]))
+ intersection = max(0.0, ix2 - ix1 + 1) * max(0.0, iy2 - iy1 + 1)
+ target_area = max(1.0, (positive_box[2] - positive_box[0] + 1) * (positive_box[3] - positive_box[1] + 1))
+ candidate_area = max(1.0, (box[2] - box[0] + 1) * (box[3] - box[1] + 1))
+ union = target_area + candidate_area - intersection
+ iou = intersection / union if union else 0.0
+ target_coverage = intersection / target_area
+ target_center = ((positive_box[0] + positive_box[2]) / 2, (positive_box[1] + positive_box[3]) / 2)
+ candidate_center = ((box[0] + box[2]) / 2, (box[1] + box[3]) / 2)
+ scale = max(((positive_box[2] - positive_box[0]) ** 2 + (positive_box[3] - positive_box[1]) ** 2) ** 0.5, 1.0)
+ center_distance = ((target_center[0] - candidate_center[0]) ** 2 + (target_center[1] - candidate_center[1]) ** 2) ** 0.5 / scale
+ return (
+ 2.0 * iou
+ + 2.0 * target_coverage
+ + max(0.0, 1.0 - center_distance)
+ + 0.25 * float(candidate["score"])
+ )
+ candidates = sorted(candidates, key=spatial_rank, reverse=True)
+ else:
+ candidates = sorted(candidates, key=production.candidate_sort_key)
+ candidate_count_before_limit = len(candidates)
+ if maximum_candidates is not None:
+ candidates = candidates[: int(maximum_candidates)]
+
+ records = []
+ for candidate_index, candidate in enumerate(candidates, start=1):
+ candidate_id = f"local_candidate_{candidate_index:03d}"
+ base = f"{candidate_id}_{production.safe_name(normalized_prompt)}"
+ mask_path, overlay_path = production.save_mask_overlay(
+ candidate["mask_bool"],
+ image_np,
+ base,
+ masks_dir,
+ overlays_dir,
+ color=(255, 0, 0),
+ )
+ records.append(
+ {
+ "candidate_id": candidate_id,
+ "prompt": normalized_prompt,
+ "score": candidate["score"],
+ "area_pixels": candidate["area"],
+ "area_fraction": candidate["area"] / (height * width),
+ "mask_bbox_abs_xyxy": candidate["mask_bbox_abs_xyxy"],
+ "mask_bbox_norm_xyxy": production.abs_to_norm_bbox(
+ candidate["mask_bbox_abs_xyxy"], width, height
+ ),
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ "raw_candidate_index": candidate["raw_candidate_index"],
+ }
+ )
+
+ run_index = session.inference_run_count + 1
+ session.inference_run_count = run_index
+ registry = {
+ "schema_version": "scenemosaic_sam3_text_candidates",
+ "image_path": str(resolved_image),
+ "image_size": [width, height],
+ "prompt": normalized_prompt,
+ "positive_bbox_abs_xyxy": positive_box,
+ "thresholds": {
+ "score": resolved_score,
+ "mask": resolved_mask,
+ "min_area": resolved_min_area,
+ },
+ "candidate_count": len(records),
+ "raw_candidate_count": raw_candidate_count,
+ "candidate_count_before_limit": candidate_count_before_limit,
+ "maximum_candidates": maximum_candidates,
+ "candidates": records,
+ "sam3_session": {
+ "model_dir": str(session.model_dir),
+ "device": session.device,
+ "model_load_count": session.model_load_count,
+ "inference_run_index": run_index,
+ },
+ }
+ registry_path = resolved_output / "text_candidates.json"
+ temporary = registry_path.with_suffix(".json.tmp")
+ temporary.write_text(
+ json.dumps(registry, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, registry_path)
+ return {
+ "registry": registry,
+ "output_dir": str(resolved_output),
+ "registry_path": str(registry_path),
+ "model_load_count": session.model_load_count,
+ "inference_run_index": run_index,
+ }
+
+
+__all__ = [
+ "ASSIGNMENT_POLICIES",
+ "DEFAULT_MODEL_DIR",
+ "Sam3Session",
+ "load_session",
+ "infer_text_candidates",
+ "prompt_groups",
+ "segment_registry",
+]
diff --git a/vlm_agent/reconstruction/perception/stage.py b/vlm_agent/reconstruction/perception/stage.py
new file mode 100644
index 0000000..01897d2
--- /dev/null
+++ b/vlm_agent/reconstruction/perception/stage.py
@@ -0,0 +1,300 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import re
+import subprocess
+import uuid
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable, Mapping
+
+from .adoption import (
+ PerceptionAdoptionAdapter,
+ PerceptionAdoptionArtifacts,
+ PerceptionAdoptionError,
+)
+from ...artifacts import RunContract
+from ...config import RuntimeConfig, verify_perception_runtime
+
+
+SECRET_NAME_RE = re.compile(
+ r"(?:KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL|AUTHORIZATION|BASE_URL)", re.I
+)
+
+
+class PerceptionStageError(RuntimeError):
+ """Raised when the perception pipeline cannot run or be adopted safely."""
+
+
+class PerceptionApprovalRequired(PerceptionStageError):
+ """Raised before any paid/upload/GPU operation when approval is absent."""
+
+
+@dataclass(frozen=True)
+class PerceptionRunResult:
+ artifacts: PerceptionAdoptionArtifacts
+ run_manifest_path: Path
+ attempt_output_dir: Path
+ subprocess_started: bool
+ audit_path: Path | None
+ run_manifest_sha256: str
+
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _write_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_name(f".{path.name}.{uuid.uuid4().hex}.tmp")
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+class PerceptionStage:
+ """Run the perception pipeline and adopt only its delivery."""
+
+ def __init__(self, runtime: RuntimeConfig, scene_config_path: Path) -> None:
+ self.runtime = runtime
+ self.contract = RunContract.from_scene_config(scene_config_path)
+
+ def verify_runtime(self) -> dict[str, Any]:
+ report = verify_perception_runtime(self.runtime.perception_root)
+ if not report.get("valid"):
+ raise PerceptionStageError(
+ f"perception runtime verification failed: {report}"
+ )
+ return report
+
+ def adopt(self, run_manifest_path: Path) -> PerceptionRunResult:
+ self.verify_runtime()
+ manifest = Path(run_manifest_path).expanduser().resolve()
+ try:
+ artifacts = PerceptionAdoptionAdapter(
+ run_manifest_path=manifest,
+ scene_config_path=self.contract.scene_config_path,
+ perception_attempts_root=self.runtime.perception_attempts_root,
+ ).adopt()
+ except PerceptionAdoptionError as exc:
+ raise PerceptionStageError(str(exc)) from exc
+ return PerceptionRunResult(
+ artifacts=artifacts,
+ run_manifest_path=manifest,
+ attempt_output_dir=manifest.parent,
+ subprocess_started=False,
+ audit_path=None,
+ run_manifest_sha256=_sha256(manifest),
+ )
+
+ def reuse_formal(self) -> PerceptionRunResult | None:
+ adoption_path = self.contract.stage03_dir / "perception_adoption.json"
+ if not adoption_path.is_file():
+ return None
+ self.verify_runtime()
+ try:
+ adoption = json.loads(adoption_path.read_text(encoding="utf-8"))
+ source = adoption["source"]
+ manifest = Path(str(source["run_manifest_path"]))
+ manifest_sha256 = str(source["run_manifest_sha256"])
+ except (OSError, json.JSONDecodeError, KeyError, TypeError) as exc:
+ raise PerceptionStageError(
+ "existing perception adoption is invalid"
+ ) from exc
+ try:
+ artifacts = PerceptionAdoptionAdapter.validate_formal(
+ self.contract.scene_config_path
+ )
+ except PerceptionAdoptionError as exc:
+ raise PerceptionStageError(str(exc)) from exc
+ return PerceptionRunResult(
+ artifacts=artifacts,
+ run_manifest_path=manifest,
+ attempt_output_dir=manifest.parent,
+ subprocess_started=False,
+ audit_path=None,
+ run_manifest_sha256=manifest_sha256,
+ )
+
+ def run(
+ self,
+ *,
+ approve_gpu: bool,
+ approve_api: bool,
+ approve_third_party_upload: bool,
+ reuse_local_pre_pass_b_cache_root: Path | None = None,
+ pass_b_reasoning_effort: str = "none",
+ environ: Mapping[str, str] | None = None,
+ on_subprocess_start: Callable[[], None] | None = None,
+ ) -> PerceptionRunResult:
+ existing = self.reuse_formal()
+ if existing is not None:
+ return existing
+ self.verify_runtime()
+ missing: list[str] = []
+ if not approve_gpu:
+ missing.append("GPU approval")
+ if not approve_api:
+ missing.append("perception API approval")
+ if not approve_third_party_upload:
+ missing.append("third-party image upload approval")
+ if missing:
+ raise PerceptionApprovalRequired(
+ "perception was not started; missing " + ", ".join(missing)
+ )
+ recovery_root: Path | None = None
+ if reuse_local_pre_pass_b_cache_root is not None:
+ trusted_root = self.runtime.perception_attempts_root.resolve(strict=True)
+ candidate = Path(reuse_local_pre_pass_b_cache_root).expanduser()
+ if candidate.is_symlink():
+ raise PerceptionStageError(
+ "perception recovery root must not be a symlink"
+ )
+ try:
+ recovery_root = candidate.resolve(strict=True)
+ except OSError as exc:
+ raise PerceptionStageError(
+ "perception recovery root does not exist"
+ ) from exc
+ try:
+ recovery_root.relative_to(trusted_root)
+ except ValueError as exc:
+ raise PerceptionStageError(
+ "perception recovery root is outside perception attempts root"
+ ) from exc
+ if not recovery_root.is_dir():
+ raise PerceptionStageError(
+ "perception recovery root is not a directory"
+ )
+ if pass_b_reasoning_effort not in {
+ None,
+ "none",
+ "low",
+ "medium",
+ "high",
+ "xhigh",
+ }:
+ raise PerceptionStageError("invalid Pass-B reasoning effort")
+ if not self.runtime.has_api_credentials("perception", environ=environ):
+ raise PerceptionApprovalRequired(
+ f"perception was not started; environment {self.runtime.perception_api_key_env} is unset"
+ )
+ gpu_preflight = self.runtime.verify_gpu()
+ if not gpu_preflight.get("valid"):
+ raise PerceptionApprovalRequired(
+ "perception was not started; GPU preflight failed"
+ )
+
+ attempt_id = f"{self.contract.scene_name}_{datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ')}_{uuid.uuid4().hex[:8]}"
+ output_dir = self.runtime.perception_attempts_root / attempt_id
+ if output_dir.exists():
+ raise PerceptionStageError(
+ f"isolated perception attempt already exists: {output_dir}"
+ )
+ output_dir.parent.mkdir(parents=True, exist_ok=True)
+ command = [
+ str(self.runtime.perception_python),
+ str(self.runtime.perception_entrypoint),
+ "--scene-config",
+ str(self.contract.scene_config_path),
+ "--output-dir",
+ str(output_dir),
+ "--model-dir",
+ str(self.runtime.perception_model_dir),
+ "--pass-a-model",
+ self.runtime.vlm_model,
+ "--pass-b-model",
+ self.runtime.vlm_model,
+ "--allow-third-party-upload",
+ ]
+ if recovery_root is not None:
+ command.extend(
+ ["--reuse-local-pre-pass-b-cache-root", str(recovery_root)]
+ )
+ if pass_b_reasoning_effort is not None:
+ command.extend(
+ ["--pass-b-reasoning-effort", pass_b_reasoning_effort]
+ )
+ source_environment = os.environ if environ is None else environ
+ secret_values = tuple(
+ value
+ for name, value in source_environment.items()
+ if value and SECRET_NAME_RE.search(name)
+ )
+
+ def redact(value: str) -> str:
+ result = value
+ for secret_value in secret_values:
+ result = result.replace(secret_value, "<redacted>")
+ if len(result) > 2 * 1024 * 1024:
+ result = result[: 2 * 1024 * 1024] + "\n<truncated>\n"
+ return result
+
+ if on_subprocess_start is not None:
+ on_subprocess_start()
+ completed = subprocess.run(
+ command,
+ cwd=self.runtime.perception_root,
+ env=self.runtime.child_environment("perception", environ=environ),
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ audit_path = self.contract.run_root / "audit" / f"perception_{attempt_id}.json"
+ audit = {
+ "schema_version": "scenemosaic_perception_subprocess_audit",
+ "attempt_id": attempt_id,
+ "command": command,
+ "cwd": str(self.runtime.perception_root),
+ "returncode": completed.returncode,
+ "stdout": redact(completed.stdout),
+ "stderr": redact(completed.stderr),
+ "release_verification": self.verify_runtime(),
+ "gpu_preflight": gpu_preflight,
+ "approvals": {
+ "gpu": True,
+ "api": True,
+ "third_party_image_upload": True,
+ },
+ "recovery": {
+ "reuse_local_pre_pass_b_cache_root": (
+ str(recovery_root) if recovery_root is not None else None
+ ),
+ "pass_b_reasoning_effort": pass_b_reasoning_effort,
+ "compact_pass_b_protocol": True,
+ },
+ }
+ _write_json(audit_path, audit)
+ if completed.returncode != 0:
+ raise PerceptionStageError(
+ f"perception failed with return code {completed.returncode}; audit: {audit_path}"
+ )
+ manifest = output_dir / "run_manifest.json"
+ if not manifest.is_file():
+ raise PerceptionStageError(
+ f"perception returned success without run_manifest.json: {output_dir}"
+ )
+ try:
+ adopted = self.adopt(manifest)
+ except PerceptionStageError as exc:
+ raise PerceptionStageError(
+ f"perception output failed adoption; audit: {audit_path}; {exc}"
+ ) from exc
+ return PerceptionRunResult(
+ artifacts=adopted.artifacts,
+ run_manifest_path=manifest,
+ attempt_output_dir=output_dir,
+ subprocess_started=True,
+ audit_path=audit_path,
+ run_manifest_sha256=adopted.run_manifest_sha256,
+ )
diff --git a/vlm_agent/reconstruction/relations/__init__.py b/vlm_agent/reconstruction/relations/__init__.py
new file mode 100644
index 0000000..3ab1aa7
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/__init__.py
@@ -0,0 +1 @@
+"""Relation extraction and relation-publication support for SceneMosaic."""
diff --git a/vlm_agent/reconstruction/relations/agents/__init__.py b/vlm_agent/reconstruction/relations/agents/__init__.py
new file mode 100644
index 0000000..31e2b97
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/__init__.py
@@ -0,0 +1 @@
+"""Multi-agent scene delivery package."""
diff --git a/vlm_agent/reconstruction/relations/agents/api_client.py b/vlm_agent/reconstruction/relations/agents/api_client.py
new file mode 100644
index 0000000..2ef2a5f
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/api_client.py
@@ -0,0 +1,279 @@
+from __future__ import annotations
+
+import base64
+import json
+import http.client
+import mimetypes
+import os
+import time
+import urllib.error
+import urllib.request
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+DEFAULT_API_BASE = "https://api.openai.com/v1"
+
+
+@dataclass
+class APIConfig:
+ model: str
+ vision_model: str | None = None
+ api_base: str | None = None
+ api_key_env: str | None = "OPENAI_API_KEY"
+ dry_run: bool = True
+ image_detail: str | None = None
+ default_api_base: str = DEFAULT_API_BASE
+ resolved_api_base: str = field(init=False)
+ api_base_source: str = field(init=False)
+ resolved_api_key_env: str = field(init=False)
+
+ def __post_init__(self) -> None:
+ self.resolved_api_key_env = self.api_key_env or "OPENAI_API_KEY"
+ if self.api_base:
+ self.resolved_api_base = self.api_base
+ self.api_base_source = "command_line"
+ elif os.environ.get("OPEN_BASE_URL"):
+ self.resolved_api_base = os.environ["OPEN_BASE_URL"]
+ self.api_base_source = "OPEN_BASE_URL"
+ elif os.environ.get("OPENAI_BASE_URL"):
+ self.resolved_api_base = os.environ["OPENAI_BASE_URL"]
+ self.api_base_source = "OPENAI_BASE_URL"
+ elif os.environ.get("OPENAI_API_BASE"):
+ self.resolved_api_base = os.environ["OPENAI_API_BASE"]
+ self.api_base_source = "OPENAI_API_BASE"
+ else:
+ self.resolved_api_base = self.default_api_base
+ self.api_base_source = "default"
+
+
+class APIClientError(RuntimeError):
+ def __init__(self, message: str, raw_response: str | None = None, retry_events: list[dict[str, Any]] | None = None):
+ super().__init__(message)
+ self.raw_response = raw_response
+ self.retry_events = retry_events or []
+
+
+class SceneDeliveryAPIClient:
+ def __init__(self, config: APIConfig):
+ self.config = config
+ self.retry_events: list[dict[str, Any]] = []
+ self.last_retry_events: list[dict[str, Any]] = []
+ self.last_response_model: str | None = None
+ self.last_finish_reason: str | None = None
+
+ @property
+ def api_base(self) -> str:
+ return self.config.resolved_api_base
+
+ @property
+ def api_base_source(self) -> str:
+ return self.config.api_base_source
+
+ @property
+ def api_key_env(self) -> str:
+ return self.config.resolved_api_key_env
+
+ def require_api_key(self) -> str:
+ api_key = os.environ.get(self.api_key_env)
+ if not api_key:
+ raise APIClientError(f"API key missing: environment variable {self.api_key_env} is not set.")
+ return api_key
+
+ def usage_metadata(self) -> dict[str, Any]:
+ """Return non-accounting response identity metadata."""
+
+ return {
+ "response_model": self.last_response_model,
+ "finish_reason": self.last_finish_reason,
+ }
+
+ def call_json(
+ self,
+ role_name: str,
+ system_prompt: str,
+ user_payload: dict[str, Any],
+ use_vision_model: bool = False,
+ max_retries: int = 3,
+ retry_base_sleep: float = 2.0,
+ image_paths: list[str] | None = None,
+ ) -> dict[str, Any]:
+ if self.config.dry_run:
+ return self._dry_run_response(role_name, user_payload)
+
+ api_key = self.require_api_key()
+ model = self.config.vision_model if (use_vision_model or image_paths) and self.config.vision_model else self.config.model
+ body = self.build_chat_completion_body(
+ model=model,
+ system_prompt=system_prompt,
+ user_payload=user_payload,
+ image_paths=image_paths,
+ )
+ request_body = json.dumps(body).encode("utf-8")
+ delays = [2.0, 5.0, 10.0]
+ retry_events: list[dict[str, Any]] = []
+ self.last_retry_events = []
+ self.last_response_model = None
+ self.last_finish_reason = None
+ last_raw_response: str | None = None
+ total_attempts = max(1, int(max_retries) + 1)
+
+ for attempt in range(1, total_attempts + 1):
+ req = urllib.request.Request(
+ self.api_base.rstrip("/") + "/chat/completions",
+ data=request_body,
+ headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
+ method="POST",
+ )
+ try:
+ with urllib.request.urlopen(req, timeout=180) as resp:
+ raw_response = resp.read().decode("utf-8")
+ data = json.loads(raw_response)
+ choice = data["choices"][0]
+ content = choice["message"]["content"]
+ response_model = data.get("model")
+ self.last_response_model = str(response_model) if response_model is not None else None
+ finish_reason = choice.get("finish_reason")
+ self.last_finish_reason = str(finish_reason) if finish_reason is not None else None
+ last_raw_response = raw_response
+ parsed = json.loads(content)
+ if retry_events:
+ retry_events.append({
+ "role_name": role_name,
+ "model": model,
+ "attempt": attempt,
+ "status": "success_after_retry",
+ })
+ self.retry_events.extend(retry_events)
+ self.last_retry_events = retry_events
+ return parsed
+ except urllib.error.HTTPError as exc:
+ detail = exc.read().decode("utf-8", errors="replace")
+ last_raw_response = detail
+ error_message = f"{role_name} API call failed with HTTP {exc.code}: {detail}"
+ event = self._retry_event(role_name, model, attempt, "HTTPError", error_message, max_retries)
+ except urllib.error.URLError as exc:
+ error_message = f"{role_name} API call failed: {exc}"
+ event = self._retry_event(role_name, model, attempt, "URLError", error_message, max_retries)
+ except http.client.RemoteDisconnected as exc:
+ error_message = f"{role_name} API call disconnected without response: {exc}"
+ event = self._retry_event(role_name, model, attempt, "RemoteDisconnected", error_message, max_retries)
+ except (ConnectionResetError, TimeoutError) as exc:
+ error_message = f"{role_name} API call connection failed: {exc}"
+ event = self._retry_event(role_name, model, attempt, type(exc).__name__, error_message, max_retries)
+ except json.JSONDecodeError as exc:
+ error_message = f"{role_name} API call returned incomplete or non-JSON response: {exc}"
+ event = self._retry_event(role_name, model, attempt, "JSONDecodeError", error_message, max_retries)
+ except (KeyError, IndexError, TypeError) as exc:
+ error_message = f"{role_name} API response did not match expected chat completion schema: {exc}"
+ event = self._retry_event(role_name, model, attempt, type(exc).__name__, error_message, max_retries)
+
+ retry_events.append(event)
+ if attempt < total_attempts:
+ sleep_seconds = delays[min(attempt - 1, len(delays) - 1)] if retry_base_sleep == 2.0 else retry_base_sleep
+ retry_events[-1]["retry_sleep_seconds"] = sleep_seconds
+ time.sleep(sleep_seconds)
+ continue
+
+ self.retry_events.extend(retry_events)
+ self.last_retry_events = retry_events
+ raise APIClientError(error_message, raw_response=last_raw_response, retry_events=retry_events)
+
+ raise APIClientError(f"{role_name} API call failed unexpectedly.", retry_events=retry_events)
+
+
+ def build_chat_completion_body(
+ self,
+ model: str,
+ system_prompt: str,
+ user_payload: dict[str, Any],
+ image_paths: list[str] | None = None,
+ ) -> dict[str, Any]:
+ user_text = json.dumps(user_payload, ensure_ascii=False)
+ if image_paths:
+ content: str | list[dict[str, Any]] = [{"type": "text", "text": user_text}]
+ for image_path in image_paths:
+ image_url = {"url": self._image_data_url(image_path)}
+ if self.config.image_detail:
+ image_url["detail"] = self.config.image_detail
+ content.append({"type": "image_url", "image_url": image_url})
+ else:
+ content = user_text
+ return {
+ "model": model,
+ "messages": [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": content},
+ ],
+ "response_format": {"type": "json_object"},
+ }
+
+ def _image_data_url(self, image_path: str) -> str:
+ path = Path(image_path)
+ if not path.exists():
+ raise APIClientError(f"Vision image path does not exist: {path}")
+ mime_type = mimetypes.guess_type(path.name)[0] or "image/png"
+ encoded = base64.b64encode(path.read_bytes()).decode("ascii")
+ return f"data:{mime_type};base64,{encoded}"
+
+ def _retry_event(
+ self,
+ role_name: str,
+ model: str,
+ attempt: int,
+ exception_type: str,
+ message: str,
+ max_retries: int,
+ ) -> dict[str, Any]:
+ return {
+ "role_name": role_name,
+ "model": model,
+ "attempt": attempt,
+ "max_retries": max_retries,
+ "exception_type": exception_type,
+ "message": message,
+ "will_retry": attempt <= max_retries,
+ }
+
+ def _dry_run_response(self, role_name: str, user_payload: dict[str, Any]) -> dict[str, Any]:
+ if role_name == "SceneDeliveryManager":
+ return {
+ "dry_run": True,
+ "plan_id": "manager_plan_000",
+ "objective": "Prepare scene delivery evidence tasks without calling the API.",
+ "tasks": [
+ {
+ "task_id": "task_0001",
+ "agent_role": "DynamicTaskAgent",
+ "task_type": "inventory_and_schema_preview",
+ "needs_tool": False,
+ "policy_snippets": ["objects keep SAM3D ids", "relations allow attach and contain", "walls require top-down evidence"],
+ "instructions": "Summarize available evidence and identify missing final-output judgements.",
+ }
+ ],
+ }
+ if role_name == "DynamicTaskAgent":
+ return {
+ "dry_run": True,
+ "task_id": user_payload.get("task", {}).get("task_id", "task_0001"),
+ "status": "mock_completed",
+ "findings": [
+ "Object candidates were collected from current SAM3D/layout evidence.",
+ "No final wall, ceiling, or attach judgement was made in dry_run.",
+ ],
+ "proposed_state_updates": {},
+ }
+ if role_name == "ExecutionSupervisorAgent":
+ return {
+ "dry_run": True,
+ "decision": "stop",
+ "reason": "Dry run generated scaffolding and mock task logs; further API calls are disabled.",
+ "ready_for_schema_guard": False,
+ }
+ if role_name == "DeliverySchemaGuard":
+ return {
+ "dry_run": True,
+ "status": "skipped",
+ "reason": "Dry run does not produce real final objects.json or relations.json.",
+ }
+ return {"dry_run": True, "status": "mock"}
diff --git a/vlm_agent/reconstruction/relations/agents/assembly/__init__.py b/vlm_agent/reconstruction/relations/agents/assembly/__init__.py
new file mode 100644
index 0000000..aeece84
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/assembly/__init__.py
@@ -0,0 +1 @@
+"""Deterministic relation assembly components."""
diff --git a/vlm_agent/reconstruction/relations/agents/assembly/containment.py b/vlm_agent/reconstruction/relations/agents/assembly/containment.py
new file mode 100644
index 0000000..8cace8c
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/assembly/containment.py
@@ -0,0 +1,1094 @@
+from __future__ import annotations
+
+import json
+import math
+import re
+from pathlib import Path
+from typing import Any
+
+from relations.agents.assembly.object_properties import _walk_with_path
+from relations.agents.assembly.relation_candidates import collect_accepted_attach_candidates
+from relations.agents.state import sorted_object_id_key
+
+def _target_coverage(object_ids: list[str], relations: list[dict[str, Any]]) -> dict[str, Any]:
+ covered = {
+ rel.get("parameters", {}).get("object_target")
+ for rel in relations
+ if isinstance(rel, dict)
+ and rel.get("relation") in {"attach", "contain"}
+ and isinstance(rel.get("parameters"), dict)
+ }
+ missing = sorted(set(object_ids) - covered, key=sorted_object_id_key)
+ return {
+ "object_ids": object_ids,
+ "covered_object_targets": sorted([target for target in covered if target in object_ids], key=sorted_object_id_key),
+ "missing_object_targets": missing,
+ "complete": not missing,
+ }
+
+
+
+def _contain_geometry_index(scene_state: dict[str, Any] | None) -> tuple[dict[str, dict[str, Any]], str | None, str | None]:
+ by_id: dict[str, dict[str, Any]] = {}
+ if isinstance(scene_state, dict):
+ candidates = scene_state.get("object_candidates", {})
+ for item in candidates.get("items", []) if isinstance(candidates, dict) else []:
+ if not (isinstance(item, dict) and item.get("object_id")):
+ continue
+ by_id[str(item["object_id"])] = {
+ "object_id": str(item["object_id"]),
+ "bbox_global": item.get("bbox_3d") if isinstance(item.get("bbox_3d"), dict) else None,
+ "orientation_for_agent": (
+ item.get("layout_reference", {}).get("orientation_for_agent")
+ if isinstance(item.get("layout_reference"), dict)
+ else None
+ ),
+ }
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state, dict) else {}
+ relation_precompute = evidence.get("relation_precompute", {}) if isinstance(evidence, dict) else {}
+ outputs = relation_precompute.get("outputs", {}) if isinstance(relation_precompute, dict) else {}
+ raw_path = outputs.get("object_footprints_path") if isinstance(outputs, dict) else None
+ if not raw_path:
+ return by_id, None, "object_footprints_path unavailable"
+ path = Path(str(raw_path))
+ if not path.exists():
+ return by_id, str(path), "object_footprints_path does not exist"
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ return by_id, str(path), str(exc)
+ items = payload.get("items", []) if isinstance(payload, dict) else []
+ for item in items:
+ if not (isinstance(item, dict) and item.get("object_id")):
+ continue
+ object_id = str(item["object_id"])
+ merged = dict(by_id.get(object_id, {"object_id": object_id}))
+ for key in [
+ "bbox_global",
+ "center_xz",
+ "extent_xz",
+ "bottom_y",
+ "top_y",
+ "height_y",
+ "mesh_floor_footprint_hull_xz",
+ "footprint_polygon_xz",
+ "topdown_grid_stats",
+ ]:
+ if item.get(key) is not None:
+ merged[key] = item.get(key)
+ by_id[object_id] = merged
+ return by_id, str(path), None
+
+
+def _contain_point_xz(record: dict[str, Any]) -> tuple[float, float] | None:
+ center_xz = record.get("center_xz")
+ if isinstance(center_xz, list) and len(center_xz) >= 2:
+ try:
+ return float(center_xz[0]), float(center_xz[1])
+ except (TypeError, ValueError):
+ pass
+ bbox = record.get("bbox_global")
+ center = bbox.get("center") if isinstance(bbox, dict) else None
+ if isinstance(center, list) and len(center) >= 3:
+ try:
+ return float(center[0]), float(center[2])
+ except (TypeError, ValueError):
+ return None
+ return None
+
+
+def _contain_center_y(record: dict[str, Any]) -> float | None:
+ bbox = record.get("bbox_global")
+ center = bbox.get("center") if isinstance(bbox, dict) else None
+ if isinstance(center, list) and len(center) >= 3:
+ try:
+ return float(center[1])
+ except (TypeError, ValueError):
+ pass
+ bottom = record.get("bottom_y")
+ top = record.get("top_y")
+ try:
+ if bottom is not None and top is not None:
+ return (float(bottom) + float(top)) * 0.5
+ except (TypeError, ValueError):
+ pass
+ return None
+
+
+def _contain_y_bounds(record: dict[str, Any]) -> tuple[float, float] | None:
+ bbox = record.get("bbox_global")
+ minimum = bbox.get("min") if isinstance(bbox, dict) else None
+ maximum = bbox.get("max") if isinstance(bbox, dict) else None
+ if isinstance(minimum, list) and isinstance(maximum, list) and len(minimum) >= 3 and len(maximum) >= 3:
+ try:
+ return min(float(minimum[1]), float(maximum[1])), max(float(minimum[1]), float(maximum[1]))
+ except (TypeError, ValueError):
+ pass
+ try:
+ bottom = float(record["bottom_y"])
+ top = float(record["top_y"])
+ return min(bottom, top), max(bottom, top)
+ except (KeyError, TypeError, ValueError):
+ return None
+
+
+def _contain_hull_xz(record: dict[str, Any]) -> list[tuple[float, float]]:
+ for key in ["mesh_floor_footprint_hull_xz", "footprint_polygon_xz"]:
+ raw = record.get(key)
+ if not isinstance(raw, list):
+ continue
+ points: list[tuple[float, float]] = []
+ for point in raw:
+ if not (isinstance(point, (list, tuple)) and len(point) >= 2):
+ continue
+ try:
+ points.append((float(point[0]), float(point[1])))
+ except (TypeError, ValueError):
+ continue
+ if len(points) >= 3:
+ return points
+ return []
+
+
+def _contain_rotate_xz(point: tuple[float, float], angle: float) -> tuple[float, float]:
+ cosine = math.cos(angle)
+ sine = math.sin(angle)
+ x, z = point
+ return x * cosine + z * sine, -x * sine + z * cosine
+
+
+def _contain_min_area_angle(points: list[tuple[float, float]]) -> float:
+ angles = {
+ round(math.atan2(second[1] - first[1], second[0] - first[0]) % (math.pi / 2.0), 12)
+ for first, second in zip(points, points[1:] + points[:1])
+ if math.dist(first, second) > 1e-9
+ }
+ if not angles:
+ return 0.0
+ best_angle = 0.0
+ best_area = float("inf")
+ for angle in sorted(angles):
+ rotated = [_contain_rotate_xz(point, angle) for point in points]
+ area = (
+ (max(point[0] for point in rotated) - min(point[0] for point in rotated))
+ * (max(point[1] for point in rotated) - min(point[1] for point in rotated))
+ )
+ if area < best_area:
+ best_area = area
+ best_angle = angle
+ return best_angle
+
+
+def _axis_outside_distance(value: float, minimum: float, maximum: float) -> float:
+ if value < minimum:
+ return minimum - value
+ if value > maximum:
+ return value - maximum
+ return 0.0
+
+
+def _contain_geometry_decision(
+ anchor: dict[str, Any] | None,
+ target: dict[str, Any] | None,
+) -> dict[str, Any]:
+ if not isinstance(anchor, dict) or not isinstance(target, dict):
+ return {
+ "decision": "uncertain",
+ "reason": "anchor or target geometry is unavailable",
+ "geometry_source": "unavailable",
+ }
+ target_xz = _contain_point_xz(target)
+ if target_xz is None:
+ return {
+ "decision": "uncertain",
+ "reason": "target center_xz is unavailable",
+ "geometry_source": "unavailable",
+ }
+
+ horizontal_decision = "uncertain"
+ horizontal_metrics: dict[str, Any] = {}
+ geometry_source = "bbox_global_aabb"
+ hull = _contain_hull_xz(anchor)
+ if hull:
+ angle = _contain_min_area_angle(hull)
+ rotated = [_contain_rotate_xz(point, angle) for point in hull]
+ target_uv = _contain_rotate_xz(target_xz, angle)
+ min_u = min(point[0] for point in rotated)
+ max_u = max(point[0] for point in rotated)
+ min_v = min(point[1] for point in rotated)
+ max_v = max(point[1] for point in rotated)
+ extent_u = max(max_u - min_u, 1e-6)
+ extent_v = max(max_v - min_v, 1e-6)
+ outside_u = _axis_outside_distance(target_uv[0], min_u, max_u)
+ outside_v = _axis_outside_distance(target_uv[1], min_v, max_v)
+ relaxed_u = max(extent_u * 0.08, 0.015)
+ relaxed_v = max(extent_v * 0.08, 0.015)
+ clear_u = max(extent_u * 0.18, 0.03)
+ clear_v = max(extent_v * 0.18, 0.03)
+ if outside_u > clear_u or outside_v > clear_v:
+ horizontal_decision = "reject"
+ elif outside_u <= relaxed_u and outside_v <= relaxed_v:
+ horizontal_decision = "pass"
+ geometry_source = "mesh_footprint_min_area_obb"
+ horizontal_metrics = {
+ "target_center_xz": [round(target_xz[0], 6), round(target_xz[1], 6)],
+ "target_center_local_uv": [round(target_uv[0], 6), round(target_uv[1], 6)],
+ "anchor_local_bounds_uv": [round(min_u, 6), round(min_v, 6), round(max_u, 6), round(max_v, 6)],
+ "outside_distance_uv": [round(outside_u, 6), round(outside_v, 6)],
+ "relaxed_margin_uv": [round(relaxed_u, 6), round(relaxed_v, 6)],
+ "clear_reject_margin_uv": [round(clear_u, 6), round(clear_v, 6)],
+ "obb_angle_degrees": round(math.degrees(angle), 6),
+ }
+ else:
+ bbox = anchor.get("bbox_global")
+ minimum = bbox.get("min") if isinstance(bbox, dict) else None
+ maximum = bbox.get("max") if isinstance(bbox, dict) else None
+ if isinstance(minimum, list) and isinstance(maximum, list) and len(minimum) >= 3 and len(maximum) >= 3:
+ try:
+ min_x, max_x = sorted([float(minimum[0]), float(maximum[0])])
+ min_z, max_z = sorted([float(minimum[2]), float(maximum[2])])
+ extent_x = max(max_x - min_x, 1e-6)
+ extent_z = max(max_z - min_z, 1e-6)
+ outside_x = _axis_outside_distance(target_xz[0], min_x, max_x)
+ outside_z = _axis_outside_distance(target_xz[1], min_z, max_z)
+ clear_x = max(extent_x * 0.18, 0.03)
+ clear_z = max(extent_z * 0.18, 0.03)
+ if outside_x > clear_x or outside_z > clear_z:
+ horizontal_decision = "reject"
+ horizontal_metrics = {
+ "target_center_xz": [round(target_xz[0], 6), round(target_xz[1], 6)],
+ "anchor_aabb_xz": [round(min_x, 6), round(min_z, 6), round(max_x, 6), round(max_z, 6)],
+ "outside_distance_xz": [round(outside_x, 6), round(outside_z, 6)],
+ "clear_reject_margin_xz": [round(clear_x, 6), round(clear_z, 6)],
+ }
+ except (TypeError, ValueError):
+ pass
+
+ vertical_decision = "uncertain"
+ vertical_metrics: dict[str, Any] = {}
+ y_bounds = _contain_y_bounds(anchor)
+ target_y = _contain_center_y(target)
+ if y_bounds is not None and target_y is not None:
+ min_y, max_y = y_bounds
+ extent_y = max(max_y - min_y, 1e-6)
+ outside_y = _axis_outside_distance(target_y, min_y, max_y)
+ relaxed_y = max(extent_y * 0.08, 0.02)
+ clear_y = max(extent_y * 0.15, 0.04)
+ if outside_y > clear_y:
+ vertical_decision = "reject"
+ elif outside_y <= relaxed_y:
+ vertical_decision = "pass"
+ vertical_metrics = {
+ "target_center_y": round(target_y, 6),
+ "anchor_bounds_y": [round(min_y, 6), round(max_y, 6)],
+ "outside_distance_y": round(outside_y, 6),
+ "relaxed_margin_y": round(relaxed_y, 6),
+ "clear_reject_margin_y": round(clear_y, 6),
+ }
+
+ if horizontal_decision == "reject" or vertical_decision == "reject":
+ decision = "reject"
+ reason = "target center is clearly outside the container OBB or vertical range"
+ elif horizontal_decision == "pass" and vertical_decision in {"pass", "uncertain"}:
+ decision = "pass"
+ reason = "target center is consistent with the container OBB and available vertical range"
+ else:
+ decision = "uncertain"
+ reason = "geometry is incomplete or close to the container boundary; preserve Agent judgement"
+ return {
+ "decision": decision,
+ "reason": reason,
+ "geometry_source": geometry_source,
+ "horizontal_decision": horizontal_decision,
+ "vertical_decision": vertical_decision,
+ "metrics": {**horizontal_metrics, **vertical_metrics},
+ }
+
+
+def filter_contain_relations_by_geometry(
+ relations: list[dict[str, Any]],
+ scene_state: dict[str, Any] | None,
+ trusted_agent_pairs: set[tuple[str, str]] | None = None,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ geometry_by_id, footprint_path, read_error = _contain_geometry_index(scene_state)
+ trusted_agent_pairs = trusted_agent_pairs or set()
+ kept: list[dict[str, Any]] = []
+ items: list[dict[str, Any]] = []
+ for rel in relations:
+ if not (isinstance(rel, dict) and rel.get("relation") == "contain" and isinstance(rel.get("parameters"), dict)):
+ kept.append(rel)
+ continue
+ anchor = str(rel["parameters"].get("object_anchor") or "")
+ target = str(rel["parameters"].get("object_target") or "")
+ result = _contain_geometry_decision(geometry_by_id.get(anchor), geometry_by_id.get(target))
+ rejected = result.get("decision") == "reject"
+ if rejected and (anchor, target) in trusted_agent_pairs:
+ result = {
+ **result,
+ "local_geometry_decision": "reject",
+ "decision": "uncertain",
+ "reason": (
+ "Explicit original-image semantic evidence places the target on an internal multi-level tier; "
+ "preserve it when SAM3D containment geometry disagrees."
+ ),
+ "trusted_agent_override": True,
+ }
+ rejected = False
+ item = {
+ "relation": rel,
+ "object_anchor": anchor,
+ "object_target": target,
+ **result,
+ "final_action": "remove" if rejected else "keep",
+ }
+ items.append(item)
+ if not rejected:
+ kept.append(rel)
+ counts = {
+ decision: len([item for item in items if item.get("decision") == decision])
+ for decision in ["pass", "uncertain", "reject"]
+ }
+ audit = {
+ "schema_version": "contain_geometry_audit",
+ "status": "completed",
+ "object_footprints_path": footprint_path,
+ "object_footprints_read_error": read_error,
+ "counts": counts,
+ "items": items,
+ "policy": (
+ "Agent remains responsible for open-world contain discovery. Local geometry removes only clear contradictions: "
+ "a target center materially outside the anchor mesh-footprint OBB or vertical range. Boundary or missing-data "
+ "cases remain uncertain. An explicit accepted semantic verdict for an internal multi-level tier may "
+ "override a contradictory SAM3D footprint because reconstruction placement is advisory rather than image truth."
+ ),
+ }
+ return kept, audit
+
+
+# Local contain candidates remain high-recall review selectors, not semantic
+# evidence. Conflicting attach/contain verdicts, including a strong local
+# selector that disagrees with Agent semantics, are escalated to the existing
+# DeliverySchemaGuard; normal scenes add no task or API call.
+CONTAIN_CONFLICT_MIN_LOCAL_CONFIDENCE = 0.90
+CONTAIN_INTERNAL_SURFACE_ROLE = "internal_shelf"
+CONTAIN_EXTERNAL_SURFACE_ROLES = {"outer_top", "external_surface"}
+
+
+def _normalized_contain_surface_role(
+ resolution: dict[str, Any],
+ anchor_category: Any = None,
+) -> tuple[str, str]:
+ """Normalize the Guard's structured role, with a narrow text-reason fallback."""
+ raw_role = resolution.get("support_surface_role") or resolution.get("surface_role")
+ if raw_role is not None:
+ normalized = re.sub(r"[^a-z0-9]+", "_", str(raw_role).strip().lower()).strip("_")
+ normalized_anchor = re.sub(
+ r"[^a-z0-9]+",
+ "_",
+ str(anchor_category or "").strip().lower(),
+ ).strip("_")
+ reason = " ".join(
+ str(resolution.get(key) or "")
+ for key in ("reason_code", "reason", "explanation")
+ ).strip().lower().replace("_", " ")
+ singular_exposed_shelf = normalized_anchor in {
+ "shelf",
+ "shelf_board",
+ "wall_shelf",
+ "floating_shelf",
+ "single_shelf",
+ }
+ bench_top = (
+ normalized_anchor.rsplit("_", 1)[-1] in {"bench", "workbench"}
+ and re.search(r"\b(?:top|topmost|uppermost)(?: level| surface)?\b", reason)
+ )
+ if normalized in {"multi_level_tier", "multilevel_tier"} and (
+ singular_exposed_shelf or bench_top
+ ):
+ return "external_surface", "structured_role_corrected_by_anchor_category"
+ aliases = {
+ "internal": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "inside": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "inside_shelf": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "internal_shelf_board": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "bounded_tier": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "multi_level_tier": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "multilevel_tier": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "bounded_tray": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "rail_enclosed_deck": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "framed_bunk": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "cubby": CONTAIN_INTERNAL_SURFACE_ROLE,
+ "outermost_top": "outer_top",
+ "outer_top_surface": "outer_top",
+ "top": "outer_top",
+ "sleep_surface": "external_surface",
+ "sleeping_surface": "external_surface",
+ "bed_surface": "external_surface",
+ "work_surface": "external_surface",
+ "exposed_single_shelf": "external_surface",
+ "single_shelf_top": "external_surface",
+ "floating_shelf_top": "external_surface",
+ "exterior": "external_surface",
+ "exterior_surface": "external_surface",
+ "unknown": "unknown",
+ "uncertain": "unknown",
+ }
+ normalized = aliases.get(normalized, normalized)
+ if normalized in {CONTAIN_INTERNAL_SURFACE_ROLE, *CONTAIN_EXTERNAL_SURFACE_ROLES, "unknown"}:
+ return normalized, "structured_support_surface_role"
+ return "unknown", "invalid_structured_support_surface_role"
+
+ reason = " ".join(
+ str(resolution.get(key) or "")
+ for key in ("reason_code", "reason", "explanation")
+ ).strip().lower().replace("_", "-")
+ reason = re.sub(r"[-\s]+", " ", reason)
+ if re.search(r"\b(?:outermost|outer|exterior) top\b", reason) or re.search(
+ r"\b(?:top of (?:the )?(?:cabinet|fridge|refrigerator|bookshelf|bookcase|shelving unit)|(?:cabinet|fridge|refrigerator|bookshelf|bookcase|shelving unit) top)\b",
+ reason,
+ ):
+ return "outer_top", "text_reason_fallback"
+ if re.search(
+ r"\b(?:sleep(?:ing)?|bed|bench|work) surface\b|\b(?:exposed|single|floating) shelf (?:top|surface)\b",
+ reason,
+ ):
+ return "external_surface", "text_reason_fallback"
+ if re.search(r"\b(?:internal|inside|lower|middle|upper)(?: open)? shelf(?: board| surface| bay)?\b", reason) or any(
+ marker in reason
+ for marker in ("display cubby", "shelf bay", "shelf compartment", "cabinet compartment")
+ ):
+ return CONTAIN_INTERNAL_SURFACE_ROLE, "text_reason_fallback"
+ if any(marker in reason for marker in ("external surface", "exterior surface", "outside the container")):
+ return "external_surface", "text_reason_fallback"
+ return "unknown", "surface_role_missing"
+
+
+def _contain_conflict_id(anchor: str, target: str) -> str:
+ return f"contain_conflict:{anchor}:{target}"
+
+
+def _contain_candidate_pairs_from_scene_state(
+ scene_state: dict[str, Any] | None,
+) -> tuple[list[dict[str, Any]], str | None, str | None]:
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state, dict) else {}
+ precompute = evidence.get("relation_precompute", {}) if isinstance(evidence, dict) else {}
+ outputs = precompute.get("outputs", {}) if isinstance(precompute, dict) else {}
+ raw_path = outputs.get("contain_candidates_path") if isinstance(outputs, dict) else None
+ if not raw_path and isinstance(precompute, dict) and precompute.get("precompute_dir"):
+ raw_path = str(Path(str(precompute["precompute_dir"])) / "contain_candidates.json")
+ if not raw_path:
+ return [], None, "contain_candidates_path unavailable"
+ path = Path(str(raw_path))
+ if not path.exists():
+ return [], str(path), "contain_candidates_path does not exist"
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ return [], str(path), str(exc)
+ pairs = payload.get("candidate_pairs", []) if isinstance(payload, dict) else []
+ return [item for item in pairs if isinstance(item, dict)], str(path), None
+
+
+def _positive_containment_hypothesis(candidate: dict[str, Any]) -> dict[str, Any] | None:
+ decision = str(candidate.get("decision") or "").lower()
+ mode = str(candidate.get("support_mode") or "").lower()
+ anchor, target = candidate.get("object_anchor"), candidate.get("object_target")
+ hypothesis = candidate.get("containment_hypothesis")
+ hypothesis_decision = ""
+ if isinstance(hypothesis, dict):
+ hypothesis_decision = str(hypothesis.get("decision") or hypothesis.get("status") or "").lower()
+ anchor = hypothesis.get("object_anchor") or hypothesis.get("anchor") or anchor
+ target = hypothesis.get("object_target") or hypothesis.get("target") or target
+ positive_decisions = {"accept", "accepted", "true", "yes", "positive"}
+ positive_hypotheses = {"inside", "accept", "accepted", "true", "yes", "positive"}
+ accepted_positive = decision in positive_decisions and (
+ mode in {"contain", "inside", "container"} or hypothesis_decision in positive_hypotheses
+ )
+ deferred_inside = (
+ decision in {"defer", "deferred"}
+ and mode == "defer_to_contain"
+ and hypothesis_decision in positive_hypotheses
+ )
+ if not (accepted_positive or deferred_inside) or not (
+ isinstance(anchor, str) and isinstance(target, str)
+ ):
+ return None
+ return {
+ "object_anchor": anchor,
+ "object_target": target,
+ "reason_code": candidate.get("reason_code") or candidate.get("reason"),
+ "support_mode": mode,
+ }
+
+
+def _contain_rejections_from_task_results(
+ task_results: list[dict[str, Any]],
+) -> dict[tuple[str, str], dict[str, Any]]:
+ rejected: dict[tuple[str, str], dict[str, Any]] = {}
+ for root in task_results:
+ if not isinstance(root, dict):
+ continue
+ kind = str(root.get("canonical_task_kind") or root.get("task_type") or "").lower()
+ capabilities = root.get("provided_capabilities", [])
+ if "contain" not in kind and not (isinstance(capabilities, list) and "contain_relations_ready" in capabilities):
+ continue
+ for path, node in _walk_with_path(root, str(root.get("task_id") or "contain_task")):
+ values = node.get("contain_candidates") if isinstance(node, dict) else None
+ if not isinstance(values, list):
+ continue
+ for candidate in values:
+ if not isinstance(candidate, dict) or str(candidate.get("decision") or "").lower() not in {"reject", "rejected", "false", "no"}:
+ continue
+ anchor, target = candidate.get("object_anchor"), candidate.get("object_target")
+ if isinstance(anchor, str) and isinstance(target, str):
+ rejected.setdefault(
+ (anchor, target),
+ {
+ "source_task_id": root.get("task_id"),
+ "source_path": f"{path}.contain_candidates",
+ "reason_code": candidate.get("reason_code") or candidate.get("reason"),
+ "evidence_codes": candidate.get("evidence_codes") or candidate.get("evidence") or [],
+ },
+ )
+ return rejected
+
+
+def collect_multi_level_support_contain_promotions(
+ task_results: list[dict[str, Any]],
+) -> dict[str, Any]:
+ """Promote only explicit Support Agent multi-level containment verdicts."""
+ rejected = _contain_rejections_from_task_results(task_results)
+ accepted: list[dict[str, Any]] = []
+ skipped: list[dict[str, Any]] = []
+ seen: set[tuple[str, str]] = set()
+ positive = {"inside", "accept", "accepted", "true", "yes", "positive"}
+ accepted_decisions = {"accept", "accepted", "true", "yes"}
+ tier_roles = {"multi_level_tier", "multilevel_tier", "internal_shelf", "bounded_tier"}
+ for root in task_results:
+ if not isinstance(root, dict):
+ continue
+ kind = str(root.get("canonical_task_kind") or root.get("task_type") or "").lower()
+ capabilities = root.get("provided_capabilities", [])
+ if "support" not in kind and not (
+ isinstance(capabilities, list) and "support_relations_ready" in capabilities
+ ):
+ continue
+ for path, node in _walk_with_path(root, str(root.get("task_id") or "support_task")):
+ values = node.get("support_contact_candidates") if isinstance(node, dict) else None
+ if not isinstance(values, list):
+ continue
+ for candidate in values:
+ if not isinstance(candidate, dict):
+ continue
+ hypothesis = candidate.get("containment_hypothesis")
+ hypothesis = hypothesis if isinstance(hypothesis, dict) else {}
+ anchor = hypothesis.get("object_anchor") or hypothesis.get("anchor") or candidate.get("object_anchor")
+ target = hypothesis.get("object_target") or hypothesis.get("target") or candidate.get("object_target")
+ pair = (str(anchor or ""), str(target or ""))
+ role = re.sub(
+ r"[^a-z0-9]+",
+ "_",
+ str(candidate.get("support_surface_role") or "").strip().lower(),
+ ).strip("_")
+ qualifies = (
+ str(candidate.get("decision") or "").lower() in accepted_decisions
+ and str(candidate.get("support_mode") or "").lower() == "defer_to_contain"
+ and str(hypothesis.get("decision") or hypothesis.get("status") or "").lower() in positive
+ and role in tier_roles
+ and pair[0].startswith("object_")
+ and pair[1].startswith("object_")
+ and pair[0] != pair[1]
+ )
+ if not qualifies or pair in seen:
+ continue
+ base = {
+ "object_anchor": pair[0],
+ "object_target": pair[1],
+ "source_task_id": root.get("task_id"),
+ "source_path": f"{path}.support_contact_candidates",
+ "support_surface_role": role,
+ "reason_code": candidate.get("reason_code") or candidate.get("reason"),
+ "raw_candidate": candidate,
+ }
+ if pair in rejected:
+ skipped.append({
+ **base,
+ "reason": "explicit_contain_specialist_rejection_prevents_support_promotion",
+ "contain_rejection": rejected[pair],
+ })
+ continue
+ seen.add(pair)
+ accepted.append({
+ **base,
+ "relation": {
+ "relation": "contain",
+ "parameters": {"object_anchor": pair[0], "object_target": pair[1]},
+ },
+ "decision": "accept",
+ })
+ return {
+ "schema_version": "multi_level_support_contain_promotion",
+ "accepted": accepted,
+ "skipped": skipped,
+ "policy": (
+ "Promote only an explicit accepted Support Agent defer_to_contain verdict with an inside "
+ "hypothesis and a multi-level/internal tier role; an explicit contain-specialist rejection wins."
+ ),
+ }
+
+
+def _contain_uncertainties_from_task_results(
+ task_results: list[dict[str, Any]],
+) -> dict[tuple[str, str], dict[str, Any]]:
+ """Collect explicit specialist uncertainty for later visual SchemaGuard arbitration."""
+ uncertain: dict[tuple[str, str], dict[str, Any]] = {}
+ for root in task_results:
+ if not isinstance(root, dict):
+ continue
+ kind = str(root.get("canonical_task_kind") or root.get("task_type") or "").lower()
+ capabilities = root.get("provided_capabilities", [])
+ if "contain" not in kind and not (isinstance(capabilities, list) and "contain_relations_ready" in capabilities):
+ continue
+ for path, node in _walk_with_path(root, str(root.get("task_id") or "contain_task")):
+ values = node.get("contain_candidates") if isinstance(node, dict) else None
+ if not isinstance(values, list):
+ continue
+ for candidate in values:
+ if not isinstance(candidate, dict) or str(candidate.get("decision") or "").lower() not in {"uncertain", "unknown", "ambiguous"}:
+ continue
+ anchor, target = candidate.get("object_anchor"), candidate.get("object_target")
+ if isinstance(anchor, str) and isinstance(target, str) and anchor != target:
+ uncertain.setdefault(
+ (anchor, target),
+ {
+ "source_task_id": root.get("task_id"),
+ "source_path": f"{path}.contain_candidates",
+ "reason_code": candidate.get("reason_code") or candidate.get("reason"),
+ "evidence_codes": candidate.get("evidence_codes") or candidate.get("evidence") or [],
+ },
+ )
+ return uncertain
+
+
+def _accepted_object_relation_hypotheses(
+ task_results: list[dict[str, Any]],
+) -> tuple[dict[tuple[str, str], dict[str, Any]], dict[tuple[str, str], dict[str, Any]]]:
+ """Collect independent accepted object attach/contain verdicts."""
+ accepted_attach: dict[tuple[str, str], dict[str, Any]] = {}
+ accepted_contain: dict[tuple[str, str], dict[str, Any]] = {}
+ collection = collect_accepted_attach_candidates(task_results)
+ for item in collection.get("accepted", []):
+ relation = item.get("relation") if isinstance(item, dict) else None
+ params = relation.get("parameters") if isinstance(relation, dict) and isinstance(relation.get("parameters"), dict) else {}
+ relation_type = relation.get("relation") if isinstance(relation, dict) else None
+ anchor, target = params.get("object_anchor"), params.get("object_target")
+ if not (
+ relation_type in {"attach", "contain"}
+ and isinstance(anchor, str)
+ and anchor.startswith("object_")
+ and isinstance(target, str)
+ and target.startswith("object_")
+ and anchor != target
+ ):
+ continue
+ raw = item.get("raw_candidate") if isinstance(item.get("raw_candidate"), dict) else {}
+ evidence = raw.get("evidence_codes") or raw.get("evidence") or []
+ if not isinstance(evidence, list):
+ evidence = [evidence] if evidence else []
+ record = {
+ "object_anchor": anchor,
+ "object_target": target,
+ "source_path": item.get("source_path"),
+ "reason_code": raw.get("reason_code") or raw.get("reason") or raw.get("rationale"),
+ "evidence_codes": [str(value)[:120] for value in evidence][:8],
+ }
+ destination = accepted_attach if relation_type == "attach" else accepted_contain
+ destination.setdefault((anchor, target), record)
+ return accepted_attach, accepted_contain
+
+
+def _contain_conflict_object_metadata(
+ scene_state: dict[str, Any] | None,
+) -> dict[str, dict[str, Any]]:
+ candidates = scene_state.get("object_candidates", {}) if isinstance(scene_state, dict) else {}
+ items = candidates.get("items", []) if isinstance(candidates, dict) else []
+ result: dict[str, dict[str, Any]] = {}
+ for item in items if isinstance(items, list) else []:
+ if not isinstance(item, dict):
+ continue
+ object_id = item.get("object_id") or item.get("id")
+ if not object_id:
+ continue
+ result[str(object_id)] = {
+ "category": item.get("category"),
+ "image_bbox": item.get("bbox_2d") or item.get("image_bbox"),
+ }
+ return result
+
+
+def collect_contain_conflicts(
+ scene_state: dict[str, Any] | None,
+ task_results: list[dict[str, Any]],
+ minimum_local_confidence: float = CONTAIN_CONFLICT_MIN_LOCAL_CONFIDENCE,
+) -> dict[str, Any]:
+ """Return semantic attach/contain disagreements for the existing guard call."""
+ candidates, candidate_path, read_error = _contain_candidate_pairs_from_scene_state(scene_state)
+ local: dict[tuple[str, str], dict[str, Any]] = {}
+ for candidate in candidates:
+ params = candidate.get("parameters") if isinstance(candidate.get("parameters"), dict) else {}
+ anchor, target = params.get("object_anchor"), params.get("object_target")
+ try:
+ confidence = float(candidate.get("confidence_hint", 0.0))
+ except (TypeError, ValueError):
+ confidence = 0.0
+ evidence = candidate.get("evidence") if isinstance(candidate.get("evidence"), list) else []
+ evidence_text = " ".join(str(item).lower() for item in evidence)
+ if not (isinstance(anchor, str) and isinstance(target, str) and confidence >= minimum_local_confidence):
+ continue
+ if "outside_anchor" in evidence_text or "outside_container" in evidence_text:
+ continue
+ anchor_data = candidate.get("anchor") if isinstance(candidate.get("anchor"), dict) else {}
+ target_data = candidate.get("target") if isinstance(candidate.get("target"), dict) else {}
+ local.setdefault(
+ (anchor, target),
+ {
+ "confidence_hint": round(confidence, 4),
+ "evidence": [str(item)[:120] for item in evidence][:8],
+ "anchor_category": anchor_data.get("category"),
+ "target_category": target_data.get("category"),
+ "anchor_image_bbox": anchor_data.get("image_bbox"),
+ "target_image_bbox": target_data.get("image_bbox"),
+ },
+ )
+
+ support_inside: dict[tuple[str, str], dict[str, Any]] = {}
+ for root in task_results:
+ if not isinstance(root, dict) or "support_contact" not in str(root.get("canonical_task_kind") or root.get("task_type") or "").lower():
+ continue
+ for path, node in _walk_with_path(root, str(root.get("task_id") or "support_task")):
+ values = node.get("support_contact_candidates") if isinstance(node, dict) else None
+ if not isinstance(values, list):
+ continue
+ for candidate in values:
+ hypothesis = _positive_containment_hypothesis(candidate) if isinstance(candidate, dict) else None
+ if hypothesis:
+ key = (hypothesis["object_anchor"], hypothesis["object_target"])
+ support_inside.setdefault(
+ key,
+ {
+ "source_task_id": root.get("task_id"),
+ "source_path": f"{path}.support_contact_candidates",
+ **hypothesis,
+ },
+ )
+
+ rejected = _contain_rejections_from_task_results(task_results)
+ uncertain = _contain_uncertainties_from_task_results(task_results)
+ accepted_attach, accepted_contain = _accepted_object_relation_hypotheses(task_results)
+ # Two independent semantic agents explicitly disagree here. Do not require
+ # the local high-recall selector to exceed a geometry-derived confidence
+ # threshold before asking the existing SchemaGuard to arbitrate the image.
+ support_reject_conflict_pairs = set(support_inside) & set(rejected)
+ accepted_conflict_pairs = set(accepted_attach) & set(accepted_contain)
+ localized_disagreement_pairs = set(local) & set(accepted_attach) & set(rejected)
+ uncertain_high_recall_pairs = (set(local) & set(uncertain)) - set(rejected) - set(accepted_contain)
+ conflict_pairs = support_reject_conflict_pairs | accepted_conflict_pairs | localized_disagreement_pairs | uncertain_high_recall_pairs
+ object_metadata = _contain_conflict_object_metadata(scene_state)
+
+ conflicts: list[dict[str, Any]] = []
+ for anchor, target in sorted(
+ conflict_pairs,
+ key=lambda pair: (sorted_object_id_key(pair[0]), sorted_object_id_key(pair[1])),
+ ):
+ local_item = local.get((anchor, target), {})
+ anchor_metadata = object_metadata.get(anchor, {})
+ target_metadata = object_metadata.get(target, {})
+ conflict_types: list[str] = []
+ if (anchor, target) in accepted_conflict_pairs:
+ conflict_types.append("accepted_attach_vs_accepted_contain")
+ if (anchor, target) in support_reject_conflict_pairs:
+ conflict_types.append("support_inside_vs_contain_reject")
+ if (anchor, target) in localized_disagreement_pairs:
+ conflict_types.append("accepted_attach_vs_high_recall_contain_reject")
+ if (anchor, target) in uncertain_high_recall_pairs:
+ conflict_types.append("high_recall_contain_uncertain")
+ conflict: dict[str, Any] = {
+ "conflict_id": _contain_conflict_id(anchor, target),
+ "conflict_type": conflict_types[0],
+ "conflict_types": conflict_types,
+ "object_anchor": anchor,
+ "object_target": target,
+ "anchor_category": local_item.get("anchor_category") or anchor_metadata.get("category"),
+ "target_category": local_item.get("target_category") or target_metadata.get("category"),
+ "anchor_image_bbox": local_item.get("anchor_image_bbox") or anchor_metadata.get("image_bbox"),
+ "target_image_bbox": local_item.get("target_image_bbox") or target_metadata.get("image_bbox"),
+ "geometry_status": "semantic_image_tie_break_required",
+ }
+ if local_item:
+ conflict["local_candidate_confidence"] = local_item.get("confidence_hint")
+ conflict["local_evidence"] = local_item.get("evidence", [])
+ if (anchor, target) in accepted_conflict_pairs:
+ conflict["accepted_attach"] = accepted_attach[(anchor, target)]
+ conflict["accepted_contain"] = accepted_contain[(anchor, target)]
+ if (anchor, target) in localized_disagreement_pairs:
+ conflict["accepted_attach"] = accepted_attach[(anchor, target)]
+ conflict["contain_rejection"] = rejected[(anchor, target)]
+ if (anchor, target) in support_reject_conflict_pairs:
+ conflict["support_hypothesis"] = support_inside[(anchor, target)]
+ conflict["contain_rejection"] = rejected[(anchor, target)]
+ if (anchor, target) in uncertain_high_recall_pairs:
+ conflict["contain_uncertainty"] = uncertain[(anchor, target)]
+ if (anchor, target) in support_inside:
+ conflict["support_hypothesis"] = support_inside[(anchor, target)]
+ conflicts.append(conflict)
+
+ return {
+ "schema_version": "contain_conflict_candidates",
+ "status": "available" if conflicts or not read_error else "unavailable",
+ "contain_candidates_path": candidate_path,
+ "read_error": read_error,
+ "minimum_local_confidence": minimum_local_confidence,
+ "high_confidence_candidate_pair_count": len(local),
+ "positive_support_hypothesis_count": len(support_inside),
+ "contain_rejection_count": len(rejected),
+ "contain_uncertain_pair_count": len(uncertain),
+ "uncertain_high_recall_conflict_count": len(uncertain_high_recall_pairs),
+ "accepted_attach_pair_count": len(accepted_attach),
+ "accepted_contain_pair_count": len(accepted_contain),
+ "accepted_pair_conflict_count": len(accepted_conflict_pairs),
+ "support_contain_reject_conflict_count": len(support_reject_conflict_pairs),
+ "localized_attach_reject_conflict_count": len(localized_disagreement_pairs),
+ "conflicts": conflicts,
+ }
+
+
+def compact_contain_conflicts_for_guard(conflict_report: dict[str, Any] | None) -> list[dict[str, Any]]:
+ """Keep semantic verdicts; local recall scores and bbox arithmetic stay local."""
+ compact: list[dict[str, Any]] = []
+ for item in (conflict_report or {}).get("conflicts", []):
+ if not isinstance(item, dict):
+ continue
+ support = item.get("support_hypothesis") if isinstance(item.get("support_hypothesis"), dict) else {}
+ rejected = item.get("contain_rejection") if isinstance(item.get("contain_rejection"), dict) else {}
+ accepted_attach = item.get("accepted_attach") if isinstance(item.get("accepted_attach"), dict) else {}
+ accepted_contain = item.get("accepted_contain") if isinstance(item.get("accepted_contain"), dict) else {}
+ uncertain = item.get("contain_uncertainty") if isinstance(item.get("contain_uncertainty"), dict) else {}
+ compact.append(
+ {
+ "conflict_id": item.get("conflict_id"),
+ "conflict_type": item.get("conflict_type"),
+ "conflict_types": item.get("conflict_types", []),
+ "object_anchor": item.get("object_anchor"),
+ "object_target": item.get("object_target"),
+ "anchor_category": item.get("anchor_category"),
+ "target_category": item.get("target_category"),
+ "accepted_attach": {
+ "reason_code": accepted_attach.get("reason_code"),
+ "evidence_codes": accepted_attach.get("evidence_codes", []),
+ } if accepted_attach else None,
+ "accepted_contain": {
+ "reason_code": accepted_contain.get("reason_code"),
+ "evidence_codes": accepted_contain.get("evidence_codes", []),
+ } if accepted_contain else None,
+ "support_hypothesis": {
+ "reason_code": support.get("reason_code"),
+ "support_mode": support.get("support_mode"),
+ } if support else None,
+ "contain_rejection": {
+ "reason_code": rejected.get("reason_code"),
+ "evidence_codes": rejected.get("evidence_codes", []),
+ } if rejected else None,
+ "contain_uncertainty": {
+ "reason_code": uncertain.get("reason_code"),
+ "evidence_codes": uncertain.get("evidence_codes", []),
+ } if uncertain else None,
+ "geometry_status": item.get("geometry_status"),
+ }
+ )
+ return compact
+
+def collect_contain_conflict_resolutions(guard_result: dict[str, Any] | None) -> dict[str, dict[str, Any]]:
+ resolutions: dict[str, dict[str, Any]] = {}
+ if not isinstance(guard_result, dict):
+ return resolutions
+ for _, node in _walk_with_path(guard_result, "DeliverySchemaGuard"):
+ values = node.get("contain_conflict_resolutions") if isinstance(node, dict) else None
+ if not isinstance(values, list):
+ continue
+ for item in values:
+ if not isinstance(item, dict):
+ continue
+ conflict_id = item.get("conflict_id")
+ if not isinstance(conflict_id, str) or not conflict_id:
+ anchor, target = item.get("object_anchor"), item.get("object_target")
+ if isinstance(anchor, str) and isinstance(target, str):
+ conflict_id = _contain_conflict_id(anchor, target)
+ if isinstance(conflict_id, str) and conflict_id:
+ normalized = dict(item)
+ if not normalized.get("decision") and normalized.get("verdict"):
+ normalized["decision"] = normalized["verdict"]
+ if not normalized.get("reason") and not normalized.get("reason_code") and normalized.get("explanation"):
+ normalized["reason"] = normalized["explanation"]
+ resolutions.setdefault(conflict_id, normalized)
+ return resolutions
+
+
+def resolve_attach_contain_conflicts(
+ relations: list[dict[str, Any]],
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """Remove unresolved same-pair conflicts and cross-furniture support contradictions."""
+ attach_pairs: set[tuple[Any, Any]] = set()
+ contain_pairs: set[tuple[Any, Any]] = set()
+ for relation in relations:
+ if not isinstance(relation, dict):
+ continue
+ relation_type = relation.get("relation")
+ if relation_type not in {"attach", "contain"}:
+ continue
+ params = relation.get("parameters")
+ params = params if isinstance(params, dict) else {}
+ pair = (params.get("object_anchor"), params.get("object_target"))
+ if relation_type == "attach":
+ attach_pairs.add(pair)
+ else:
+ contain_pairs.add(pair)
+
+ if not attach_pairs or not contain_pairs:
+ return relations, []
+ conflict_pairs = attach_pairs & contain_pairs
+ contain_anchors_by_target: dict[Any, set[Any]] = {}
+ for anchor, target in contain_pairs:
+ contain_anchors_by_target.setdefault(target, set()).add(anchor)
+ cross_furniture_attach_pairs = {
+ pair
+ for pair in attach_pairs
+ if isinstance(pair[0], str)
+ and pair[0].startswith("object_")
+ and pair[1] in contain_anchors_by_target
+ and pair[0] not in contain_anchors_by_target[pair[1]]
+ }
+ if not conflict_pairs and not cross_furniture_attach_pairs:
+ return relations, []
+
+ kept: list[dict[str, Any]] = []
+ removed: list[dict[str, Any]] = []
+ for relation in relations:
+ params = relation.get("parameters", {}) if isinstance(relation, dict) and isinstance(relation.get("parameters"), dict) else {}
+ pair = (params.get("object_anchor"), params.get("object_target"))
+ if pair in conflict_pairs and relation.get("relation") in {"attach", "contain"}:
+ removed.append(
+ {
+ "relation": relation,
+ "reason": "unresolved_attach_contain_same_pair_requires_schema_guard",
+ "policy": (
+ "Attach and contain are mutually exclusive for one anchor-target pair, "
+ "but neither relation type has a fixed priority. Both are removed when "
+ "the earlier SchemaGuard arbitration did not produce exactly one verdict."
+ ),
+ }
+ )
+ continue
+ if pair in cross_furniture_attach_pairs and relation.get("relation") == "attach":
+ removed.append(
+ {
+ "relation": relation,
+ "reason": "object_attach_conflicts_with_other_contain_anchor",
+ "policy": (
+ "A target retained inside one object container cannot also use a different object as "
+ "its physical support. Structural floor, wall, and ceiling relations are unaffected."
+ ),
+ "contain_anchors": sorted(str(value) for value in contain_anchors_by_target[pair[1]]),
+ }
+ )
+ continue
+ kept.append(relation)
+ return kept, removed
+
+def filter_redundant_floor_attaches_for_object_supported_targets(
+ relations: list[dict[str, Any]],
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """Keep object support instead of a contradictory floor fallback."""
+ object_supported_targets = {
+ str(params.get("object_target"))
+ for relation in relations
+ if isinstance(relation, dict)
+ and relation.get("relation") == "attach"
+ and isinstance((params := relation.get("parameters")), dict)
+ and isinstance(params.get("object_anchor"), str)
+ and str(params.get("object_anchor")).startswith("object_")
+ and params.get("object_target")
+ }
+ kept: list[dict[str, Any]] = []
+ removed: list[dict[str, Any]] = []
+ for relation in relations:
+ params = relation.get("parameters", {}) if isinstance(relation, dict) else {}
+ if (
+ relation.get("relation") == "attach"
+ and params.get("object_anchor") == "floor"
+ and str(params.get("object_target")) in object_supported_targets
+ ):
+ removed.append(
+ {
+ "relation": relation,
+ "reason": "object_supported_target_redundant_floor_attach",
+ "policy": "One target has one gravity support path; object support outranks a generic floor fallback.",
+ }
+ )
+ continue
+ kept.append(relation)
+ return kept, removed
+
+
+def normalize_floor_covering_relations(
+ relations: list[dict[str, Any]],
+ object_properties: dict[str, Any] | None,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ floor_covering_ids = set((object_properties or {}).get("floor_covering_object_ids", []))
+ if not floor_covering_ids:
+ return relations, {
+ "schema_version": "relation_normalization_audit",
+ "floor_covering_object_ids": [],
+ "rewritten_count": 0,
+ "items": [],
+ }
+ normalized: list[dict[str, Any]] = []
+ seen: set[tuple[Any, Any, Any]] = set()
+ items: list[dict[str, Any]] = []
+ for rel in relations:
+ if not isinstance(rel, dict) or not isinstance(rel.get("parameters"), dict):
+ normalized.append(rel)
+ continue
+ relation_type = rel.get("relation")
+ params = rel.get("parameters", {})
+ anchor = params.get("object_anchor")
+ target = params.get("object_target")
+ new_rel = rel
+ if relation_type == "attach" and anchor in floor_covering_ids and target != anchor:
+ new_rel = {"relation": "attach", "parameters": {"object_anchor": "floor", "object_target": target}}
+ items.append(
+ {
+ "original_relation": rel,
+ "normalized_relation": new_rel,
+ "reason": "floor_covering_anchor_normalized_to_floor",
+ "floor_covering_anchor": anchor,
+ }
+ )
+ key = (new_rel.get("relation"), new_rel.get("parameters", {}).get("object_anchor"), new_rel.get("parameters", {}).get("object_target"))
+ if key in seen:
+ items.append(
+ {
+ "original_relation": rel,
+ "normalized_relation": new_rel,
+ "reason": "duplicate_after_floor_covering_normalization_dropped",
+ }
+ )
+ continue
+ seen.add(key)
+ normalized.append(new_rel)
+ return normalized, {
+ "schema_version": "relation_normalization_audit",
+ "floor_covering_object_ids": sorted(floor_covering_ids, key=sorted_object_id_key),
+ "rewritten_count": len([item for item in items if item.get("reason") == "floor_covering_anchor_normalized_to_floor"]),
+ "items": items,
+ "policy": "Objects marked as floor covering are not final support anchors; their supported targets are normalized to floor attach.",
+ }
diff --git a/vlm_agent/reconstruction/relations/agents/assembly/object_properties.py b/vlm_agent/reconstruction/relations/agents/assembly/object_properties.py
new file mode 100644
index 0000000..af88b39
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/assembly/object_properties.py
@@ -0,0 +1,998 @@
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+from typing import Any
+
+from relations.agents.state import sorted_object_id_key, write_audit_pair
+from relations.object_physics.core import local_upright_semantic_override
+
+ROOM_MATERIAL_KEYS = ("walls", "floor", "ceiling")
+
+
+def _is_ceiling_light_object(obj: dict[str, Any]) -> bool:
+ category = str(obj.get("category") or "").lower()
+ description = str(obj.get("description") or "").lower()
+ return category == "ceiling_light" or ("ceiling" in description and any(token in description for token in ["light", "spotlight", "fixture"]))
+
+def _snake_case(value: str | None) -> str:
+ value = (value or "unknown_object").strip()
+ value = re.sub(r"[^0-9A-Za-z]+", "_", value)
+ value = re.sub(r"_+", "_", value).strip("_").lower()
+ return value or "unknown_object"
+
+
+def candidate_objects(scene_state: dict[str, Any]) -> list[dict[str, str]]:
+ items = scene_state.get("object_candidates", {}).get("items", [])
+ objects = []
+ for item in items:
+ object_id = item.get("object_id")
+ if not object_id:
+ continue
+ objects.append(
+ {
+ "id": object_id,
+ "category": _snake_case(item.get("category")),
+ "description": item.get("description") or _snake_case(item.get("category")).replace("_", " "),
+ }
+ )
+ objects.sort(key=lambda obj: sorted_object_id_key(obj["id"]))
+ return objects
+
+
+def _walk(value: Any):
+ if isinstance(value, dict):
+ yield value
+ for child in value.values():
+ yield from _walk(child)
+ elif isinstance(value, list):
+ for child in value:
+ yield from _walk(child)
+
+
+def _walk_with_path(value: Any, path: str = "$"):
+ yield path, value
+ if isinstance(value, dict):
+ for key, child in value.items():
+ yield from _walk_with_path(child, f"{path}.{key}")
+ elif isinstance(value, list):
+ for idx, child in enumerate(value):
+ yield from _walk_with_path(child, f"{path}[{idx}]")
+
+
+def normalize_room_materials(value: Any) -> tuple[dict[str, str] | None, list[str]]:
+ if not isinstance(value, dict):
+ return None, ["room_materials must be an object."]
+
+ errors: list[str] = []
+ keys = set(value.keys())
+ missing = [key for key in ROOM_MATERIAL_KEYS if key not in keys]
+ extra = sorted(keys - set(ROOM_MATERIAL_KEYS))
+ if missing:
+ errors.append(f"room_materials is missing required keys: {missing}.")
+ if extra:
+ errors.append(f"room_materials has unexpected keys: {extra}.")
+
+ normalized: dict[str, str] = {}
+ for key in ROOM_MATERIAL_KEYS:
+ raw = value.get(key)
+ if not isinstance(raw, str) or not raw.strip():
+ errors.append(f"room_materials.{key} must be a non-empty string.")
+ continue
+ normalized[key] = re.sub(r"\s+", " ", raw).strip()
+
+ return (normalized if not errors else None), errors
+
+
+def collect_room_materials(
+ task_results: list[dict[str, Any]],
+ guard_result: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+ roots: list[Any] = list(task_results)
+ if guard_result:
+ roots.append(guard_result)
+
+ valid_candidates: list[dict[str, Any]] = []
+ invalid_candidates: list[dict[str, Any]] = []
+ for order, root in enumerate(roots):
+ if not isinstance(root, dict) or "room_materials" not in root:
+ continue
+ provided = root.get("provided_capabilities")
+ if isinstance(provided, str):
+ provided_capabilities = [provided]
+ elif isinstance(provided, list):
+ provided_capabilities = [str(item) for item in provided]
+ else:
+ provided_capabilities = []
+ record = {
+ "source_task_id": root.get("task_id"),
+ "source_task_type": root.get("canonical_task_kind") or root.get("task_type"),
+ "declared_room_materials_provider": "room_materials_ready" in provided_capabilities,
+ "source_order": order,
+ }
+ normalized, errors = normalize_room_materials(root.get("room_materials"))
+ if normalized is None:
+ invalid_candidates.append({
+ **record,
+ "errors": errors,
+ "observed_keys": sorted(root["room_materials"].keys()) if isinstance(root.get("room_materials"), dict) else [],
+ })
+ continue
+ valid_candidates.append({**record, "room_materials": normalized})
+
+ declared = [item for item in valid_candidates if item["declared_room_materials_provider"]]
+ selected = (declared or valid_candidates)[-1] if (declared or valid_candidates) else None
+ return {
+ "status": "ready" if selected else "missing_or_invalid",
+ "room_materials": selected.get("room_materials") if selected else None,
+ "source_task_id": selected.get("source_task_id") if selected else None,
+ "source_task_type": selected.get("source_task_type") if selected else None,
+ "selection_policy": "latest_declared_provider_else_latest_valid_candidate",
+ "valid_candidates": valid_candidates,
+ "invalid_candidates": invalid_candidates,
+ }
+
+
+_OBJECT_REVISION_CONFIDENCE_SCORES = {"low": 1, "medium": 2, "high": 3}
+_OBJECT_REVISION_SOURCE_PRIORITIES = {
+ "object_revision_check": 3,
+ "schema_guard": 3,
+ "support_contact_extraction": 2,
+ "contain_relation_extraction": 2,
+}
+
+
+def _normalize_object_revision_confidence(value: Any) -> str:
+ normalized = str(value or "medium").strip().lower()
+ return normalized if normalized in _OBJECT_REVISION_CONFIDENCE_SCORES else "medium"
+
+
+def _resolve_object_revision_patches(
+ patches: list[dict[str, Any]],
+ warnings: list[str],
+) -> list[dict[str, Any]]:
+ grouped: dict[tuple[str, str], list[dict[str, Any]]] = {}
+ for patch in patches:
+ for field_name in ("new_category", "new_description"):
+ if patch.get(field_name) is not None:
+ grouped.setdefault((patch["object_id"], field_name), []).append(patch)
+
+ resolved_by_object: dict[str, dict[str, Any]] = {}
+ for (object_id, field_name), candidates in grouped.items():
+ best_score = max(
+ (
+ _OBJECT_REVISION_CONFIDENCE_SCORES[item["confidence"]],
+ int(item.get("source_priority", 1)),
+ )
+ for item in candidates
+ )
+ top = [
+ item
+ for item in candidates
+ if (
+ _OBJECT_REVISION_CONFIDENCE_SCORES[item["confidence"]],
+ int(item.get("source_priority", 1)),
+ ) == best_score
+ ]
+ top_values = {str(item[field_name]) for item in top}
+ if len(top_values) != 1:
+ warnings.append(
+ f"Skipped ambiguous object revision for {object_id}.{field_name}: "
+ f"equally ranked tasks proposed {sorted(top_values)}."
+ )
+ continue
+ chosen = top[0]
+ discarded_values = {
+ str(item[field_name])
+ for item in candidates
+ if str(item[field_name]) != str(chosen[field_name])
+ }
+ if discarded_values:
+ warnings.append(
+ f"Resolved object revision conflict for {object_id}.{field_name} in favor of "
+ f"{chosen[field_name]!r} from {chosen.get('source_task_id') or chosen.get('source_task_kind')}; "
+ f"discarded lower-ranked values {sorted(discarded_values)}."
+ )
+ resolved = resolved_by_object.setdefault(
+ object_id,
+ {
+ "object_id": object_id,
+ "new_category": None,
+ "new_description": None,
+ "reasons": [],
+ "source_task_ids": [],
+ "confidence": chosen["confidence"],
+ },
+ )
+ resolved[field_name] = chosen[field_name]
+ resolved["reasons"].append(chosen["reason"])
+ source_id = chosen.get("source_task_id") or chosen.get("source_task_kind")
+ if source_id:
+ resolved["source_task_ids"].append(source_id)
+ if _OBJECT_REVISION_CONFIDENCE_SCORES[chosen["confidence"]] > _OBJECT_REVISION_CONFIDENCE_SCORES[resolved["confidence"]]:
+ resolved["confidence"] = chosen["confidence"]
+
+ resolved_patches: list[dict[str, Any]] = []
+ for object_id in sorted(resolved_by_object):
+ item = resolved_by_object[object_id]
+ reasons = list(dict.fromkeys(item.pop("reasons")))
+ source_task_ids = list(dict.fromkeys(item.pop("source_task_ids")))
+ item["reason"] = "; ".join(reasons)
+ item["source_task_id"] = ",".join(source_task_ids) if source_task_ids else None
+ resolved_patches.append(item)
+ return resolved_patches
+
+
+def collect_object_revision_patches(task_results: list[dict[str, Any]], guard_result: dict[str, Any] | None = None) -> tuple[list[dict[str, Any]], list[str]]:
+ roots: list[tuple[Any, bool]] = [(item, False) for item in task_results]
+ if guard_result:
+ roots.append((guard_result, True))
+ patches: list[dict[str, Any]] = []
+ warnings: list[str] = []
+ for root, is_guard_result in roots:
+ root_kind = (
+ "schema_guard"
+ if is_guard_result
+ else str(root.get("canonical_task_kind") or root.get("task_type") or "unknown")
+ if isinstance(root, dict)
+ else "unknown"
+ )
+ source_priority = _OBJECT_REVISION_SOURCE_PRIORITIES.get(root_kind, 1)
+ source_task_id = root.get("task_id") if isinstance(root, dict) else None
+ for node in _walk(root):
+ candidates = []
+ for key in ["object_revisions", "object_revision_candidates", "revision_patches", "object_revision_patches", "accepted_object_revision_patches"]:
+ if isinstance(node.get(key), list):
+ candidates.extend(node[key])
+ for patch in candidates:
+ if not isinstance(patch, dict):
+ continue
+ object_id = patch.get("object_id") or patch.get("id")
+ attach_relevant_evidence = (
+ patch.get("image_or_topdown_evidence")
+ or patch.get("image_evidence")
+ or patch.get("mask_evidence")
+ or patch.get("topdown_evidence")
+ or patch.get("attach_relevant_conflict")
+ )
+ reason = patch.get("reason") or patch.get("rationale") or patch.get("evidence") or patch.get("reason_code")
+ new_category = patch.get("new_category") or patch.get("final_category")
+ new_description = patch.get("new_description") or patch.get("final_description")
+ if new_category is None and patch.get("normalized_category") and patch.get("category"):
+ warnings.append(
+ f"Low-priority category alias normalization skipped for {object_id or 'unknown_object'}: "
+ f"{patch.get('category')} -> {patch.get('normalized_category')}."
+ )
+ continue
+ if new_category is None and patch.get("category") and attach_relevant_evidence:
+ new_category = patch.get("category")
+ if new_description is None and patch.get("description") and attach_relevant_evidence:
+ new_description = f"{patch.get('description')} [revision needed: {attach_relevant_evidence}]"
+ if not reason and attach_relevant_evidence:
+ reason = f"Attach-relevant image/mask/topdown conflict: {attach_relevant_evidence}"
+ if not object_id:
+ warnings.append(f"Skipped object revision patch without object_id: {patch}")
+ continue
+ if not reason:
+ warnings.append(f"Skipped object revision patch for {object_id} because reason is missing.")
+ continue
+ if new_category is None and new_description is None:
+ warnings.append(f"Skipped object revision patch for {object_id} because no category/description change was provided.")
+ continue
+ patches.append(
+ {
+ "object_id": object_id,
+ "new_category": _snake_case(new_category) if new_category is not None else None,
+ "new_description": new_description,
+ "reason": str(reason),
+ "confidence": _normalize_object_revision_confidence(patch.get("confidence")),
+ "source_priority": source_priority,
+ "source_task_kind": root_kind,
+ "source_task_id": source_task_id,
+ }
+ )
+ return _resolve_object_revision_patches(patches, warnings), warnings
+
+def apply_object_revision_patches(objects: list[dict[str, str]], patches: list[dict[str, Any]]) -> tuple[list[dict[str, str]], list[dict[str, Any]], list[str]]:
+ by_id = {obj["id"]: obj for obj in objects}
+ applied: list[dict[str, Any]] = []
+ warnings: list[str] = []
+ for patch in patches:
+ object_id = patch["object_id"]
+ if object_id not in by_id:
+ warnings.append(f"Skipped object revision patch for unknown object_id {object_id}.")
+ continue
+ obj = by_id[object_id]
+ changes: dict[str, dict[str, str]] = {}
+ if patch.get("new_category") and patch["new_category"] != obj["category"]:
+ changes["category"] = {"old": obj["category"], "new": patch["new_category"]}
+ obj["category"] = patch["new_category"]
+ if patch.get("new_description") and patch["new_description"] != obj["description"]:
+ changes["description"] = {"old": obj["description"], "new": patch["new_description"]}
+ obj["description"] = patch["new_description"]
+ if changes:
+ applied.append(
+ {
+ "object_id": object_id,
+ "fields": list(changes.keys()),
+ "changes": changes,
+ "reason": patch["reason"],
+ "source_task_id": patch.get("source_task_id"),
+ }
+ )
+ return objects, applied, warnings
+
+
+def write_object_revision_audit(output_dir: Path, applied: list[dict[str, Any]], warnings: list[str], dry_run: bool) -> None:
+ data = {
+ "dry_run": dry_run,
+ "status": "completed" if not warnings else "completed_with_warnings",
+ "items": applied,
+ "warnings": warnings,
+ "description_policy": (
+ "description is weak evidence; object revision should focus on attach-relevant semantic or positional "
+ "conflicts grounded by image/mask evidence, with layout/topdown used only as coarse support. Minor category "
+ "alias normalization is low priority and should not override attach-relevant conflicts."
+ ),
+ }
+ write_audit_pair(output_dir, "objects", "object_revision_audit", data, "Object Revision Audit")
+
+
+FLOOR_COVERING_TEXT_HINTS = {
+ "area rug",
+ "carpet",
+ "floor covering",
+ "floor mat",
+ "mat",
+ "rug",
+ "tatami",
+ "woven mat",
+}
+
+UPRIGHT_FALSE_TEXT_HINTS = {
+ "artwork",
+ "bean_bag",
+ "box",
+ "book",
+ "carpet",
+ "controller",
+ "cooling_rack",
+ "cushion",
+ "floor_covering",
+ "framed_art",
+ "mat",
+ "painting",
+ "pillow",
+ "rug",
+ "slab",
+ "wall_art",
+ "wall_clock",
+ "wall_decoration",
+ "wall_mirror",
+ "wall_panel",
+ "wall_sign",
+ "whiteboard",
+}
+
+UPRIGHT_IMAGE_AXIS_TEXT_HINTS = {
+ "bottle",
+ "canister",
+ "cylinder",
+ "paper_towel_roll",
+ "roll",
+ "thermos",
+ "tumbler",
+}
+
+UPRIGHT_TRUE_TEXT_HINTS = {
+ "appliance",
+ "armchair",
+ "basin",
+ "basket",
+ "bed",
+ "bench",
+ "book_shelf",
+ "bookcase",
+ "bookshelf",
+ "bowl",
+ "cabinet",
+ "chair",
+ "couch",
+ "counter",
+ "cup",
+ "desk",
+ "display_screen",
+ "dresser",
+ "figure",
+ "figurine",
+ "floor_lamp",
+ "fridge",
+ "lamp",
+ "media_console",
+ "monitor",
+ "mug",
+ "nightstand",
+ "oven",
+ "plate",
+ "plant",
+ "refrigerator",
+ "shelf",
+ "side_table",
+ "sideboard",
+ "sofa",
+ "stand",
+ "stool",
+ "table",
+ "television",
+ "tray",
+ "tv",
+ "vase",
+ "vessel",
+ "wardrobe",
+}
+
+CONTAIN_UP_DIRECTION_HINTS = {
+ "basket", "bin", "bowl", "box", "crate", "drawer", "open_top", "tray",
+}
+CONTAIN_FRONT_DIRECTION_HINTS = {
+ "bookcase", "bookshelf", "bunk_bed", "cabinet", "case", "closet", "cubby", "display",
+ "freezer", "fridge", "loft_bed", "multi_tier", "open_shelf", "rack", "refrigerator",
+ "shelf", "shelving", "tiered", "wardrobe",
+}
+
+
+def _bool_or_none(value: Any) -> bool | None:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ text = value.strip().lower()
+ if text in {"true", "yes", "1", "y", "positive", "accepted"}:
+ return True
+ if text in {"false", "no", "0", "n", "negative", "rejected"}:
+ return False
+ return None
+
+
+def _scene_object_context(scene_state: dict[str, Any] | None) -> dict[str, dict[str, Any]]:
+ result: dict[str, dict[str, Any]] = {}
+ if not isinstance(scene_state, dict):
+ return result
+ items = scene_state.get("object_candidates", {}).get("items", []) if isinstance(scene_state.get("object_candidates"), dict) else []
+ for item in items:
+ if isinstance(item, dict) and item.get("object_id"):
+ result[str(item["object_id"])] = item
+ return result
+
+
+def _object_text(obj: dict[str, Any], scene_obj: dict[str, Any] | None = None) -> str:
+ scene_obj = scene_obj or {}
+ return " ".join(
+ str(value or "")
+ for value in [
+ obj.get("category"),
+ obj.get("description"),
+ scene_obj.get("category"),
+ scene_obj.get("description"),
+ ]
+ ).lower()
+
+
+def _bbox_extent_xyz(scene_obj: dict[str, Any] | None) -> list[float] | None:
+ bbox = scene_obj.get("bbox_3d") if isinstance(scene_obj, dict) else None
+ extent = bbox.get("extent") if isinstance(bbox, dict) else None
+ if not (isinstance(extent, list) and len(extent) >= 3):
+ return None
+ values = []
+ for item in extent[:3]:
+ try:
+ values.append(abs(float(item)))
+ except Exception:
+ return None
+ return values
+
+
+def _bbox_2d_height_width_ratio(scene_obj: dict[str, Any] | None) -> float | None:
+ bbox = scene_obj.get("bbox_2d") if isinstance(scene_obj, dict) else None
+ if not (isinstance(bbox, list) and len(bbox) >= 4):
+ return None
+ try:
+ width = abs(float(bbox[2]) - float(bbox[0]))
+ height = abs(float(bbox[3]) - float(bbox[1]))
+ except (TypeError, ValueError):
+ return None
+ if width <= 1e-6 or height <= 1e-6:
+ return None
+ return height / width
+
+
+def _local_floor_covering_hint(obj: dict[str, Any], scene_obj: dict[str, Any] | None = None) -> tuple[bool, str, str]:
+ text = _object_text(obj, scene_obj)
+ if _matches_semantic_hint(text, FLOOR_COVERING_TEXT_HINTS):
+ return True, "high", "local_text_fallback: category/description indicates rug/carpet/mat/floor covering"
+ extent = _bbox_extent_xyz(scene_obj)
+ if extent:
+ x, y, z = extent
+ horizontal_area = x * z
+ if y <= 0.035 and horizontal_area >= 0.04 and max(x, z) >= 0.25:
+ return True, "medium", "local_geometry_fallback: object is very flat, broad, and floor-like"
+ return False, "low", "no floor-covering evidence"
+
+
+def _matches_semantic_hint(value: Any, hints: set[str]) -> bool:
+ normalized = f"_{_snake_case(str(value or ''))}_"
+ return any(f"_{_snake_case(hint)}_" in normalized for hint in hints)
+
+
+def _local_upright_semantic_hint(
+ obj: dict[str, Any],
+ scene_obj: dict[str, Any] | None = None,
+ is_floor_covering: bool = False,
+) -> tuple[bool, str, str] | None:
+ scene_obj = scene_obj or {}
+ category_values = [str(value or "") for value in [obj.get("category"), scene_obj.get("category")]]
+ category = " ".join(category_values)
+ text = _object_text(obj, scene_obj)
+ if is_floor_covering or _matches_semantic_hint(text, FLOOR_COVERING_TEXT_HINTS):
+ return False, "local_semantic_policy", "floor covering is not an upright reconstruction constraint"
+ for category_value in category_values:
+ stable_category = local_upright_semantic_override({"category": category_value})
+ if stable_category is not None:
+ upright, reason = stable_category
+ return upright, "local_semantic_policy", reason
+ image_ratio = _bbox_2d_height_width_ratio(scene_obj)
+ if (
+ image_ratio is not None
+ and image_ratio >= 1.35
+ and _matches_semantic_hint(text, UPRIGHT_IMAGE_AXIS_TEXT_HINTS)
+ ):
+ return True, "local_image_geometry_policy", "axis-sensitive object is visibly vertical in the original-image bbox"
+ if _matches_semantic_hint(category, UPRIGHT_TRUE_TEXT_HINTS):
+ return True, "local_semantic_policy", "canonical category requires a stable local-up direction under gravity"
+ if _matches_semantic_hint(category, UPRIGHT_FALSE_TEXT_HINTS):
+ return False, "local_semantic_policy", "canonical category does not require a fixed local-up direction under gravity"
+ if _matches_semantic_hint(text, UPRIGHT_TRUE_TEXT_HINTS):
+ return True, "local_semantic_policy", "description indicates a stable local-up direction is required under gravity"
+ if _matches_semantic_hint(text, UPRIGHT_FALSE_TEXT_HINTS):
+ return False, "local_semantic_policy", "description indicates no fixed local-up direction is required under gravity"
+ return None
+
+
+def _local_upright_fallback(
+ obj: dict[str, Any],
+ scene_obj: dict[str, Any] | None = None,
+ is_floor_covering: bool = False,
+) -> tuple[bool, str, str]:
+ semantic = _local_upright_semantic_hint(obj, scene_obj, is_floor_covering)
+ if semantic is not None:
+ return semantic
+ extent = _bbox_extent_xyz(scene_obj)
+ if extent:
+ x, y, z = extent
+ horizontal = max(x, z, 1e-6)
+ flat = y <= min(x, z, 1e-6) * 0.45 or y <= 0.04
+ tall = y >= horizontal * 0.65 and y >= 0.12
+ if flat:
+ return False, "local_geometry_fallback", "unknown-category object is geometrically flat/low"
+ if tall:
+ return True, "local_geometry_fallback", "unknown-category object is tall relative to footprint"
+ return True, "local_default_fallback", "defaulting unknown non-wall object to upright for stable gravity-based reconstruction"
+
+def collect_object_physical_properties(
+ task_results: list[dict[str, Any]],
+ guard_result: dict[str, Any] | None,
+ objects: list[dict[str, Any]],
+ scene_state: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+ scene_by_id = _scene_object_context(scene_state)
+ by_id: dict[str, dict[str, Any]] = {
+ str(obj["id"]): {
+ "object_id": str(obj["id"]),
+ "agent": {},
+ "fallback": {},
+ }
+ for obj in objects
+ if isinstance(obj, dict) and obj.get("id")
+ }
+ roots: list[Any] = list(task_results)
+ if guard_result:
+ roots.append(guard_result)
+
+ def apply_property(raw: dict[str, Any], source_task_id: str | None = None) -> None:
+ object_id = raw.get("object_id") or raw.get("id") or raw.get("target_object_id")
+ if not object_id or str(object_id) not in by_id:
+ return
+ entry = by_id[str(object_id)]
+ agent = entry.setdefault("agent", {})
+ floor_covering = _bool_or_none(
+ raw.get("is_floor_covering")
+ if "is_floor_covering" in raw
+ else raw.get("floor_covering")
+ if "floor_covering" in raw
+ else raw.get("is_rug_or_floor_covering")
+ )
+ if floor_covering is not None:
+ agent["is_floor_covering"] = floor_covering
+ agent["floor_covering_confidence"] = raw.get("floor_covering_confidence") or raw.get("confidence") or "medium"
+ agent["floor_covering_reason"] = raw.get("floor_covering_reason") or raw.get("reason") or raw.get("rationale")
+ agent["floor_covering_source_task_id"] = source_task_id
+
+ property_list_keys = [
+ "object_physical_properties",
+ "object_properties",
+ "physical_properties",
+ "object_semantic_properties",
+ "object_property_judgements",
+ "object_physical_property_reasoning",
+ ]
+ for root in roots:
+ source_task_id = root.get("task_id") if isinstance(root, dict) else None
+ for node in _walk(root):
+ if not isinstance(node, dict):
+ continue
+ for key in property_list_keys:
+ value = node.get(key)
+ if isinstance(value, list):
+ for item in value:
+ if isinstance(item, dict):
+ apply_property(item, source_task_id)
+ elif isinstance(value, dict):
+ if isinstance(value.get("items"), list):
+ for item in value["items"]:
+ if isinstance(item, dict):
+ apply_property(item, source_task_id)
+ else:
+ apply_property(value, source_task_id)
+ if any(key in node for key in ["is_floor_covering", "floor_covering", "is_rug_or_floor_covering"]):
+ apply_property(node, source_task_id)
+
+ items = []
+ floor_covering_ids: set[str] = set()
+ for obj in objects:
+ object_id = str(obj.get("id"))
+ if object_id not in by_id:
+ continue
+ entry = by_id[object_id]
+ agent = entry.get("agent", {})
+ local_fc, local_fc_conf, local_fc_reason = _local_floor_covering_hint(obj, scene_by_id.get(object_id))
+ is_floor_covering = bool(agent.get("is_floor_covering")) if "is_floor_covering" in agent else local_fc
+ fc_source = "agent_property" if "is_floor_covering" in agent else ("local_text_or_geometry_fallback" if local_fc else "none")
+ if is_floor_covering:
+ floor_covering_ids.add(object_id)
+ entry["resolved"] = {
+ "object_id": object_id,
+ "is_floor_covering": is_floor_covering,
+ "floor_covering_confidence": agent.get("floor_covering_confidence") or local_fc_conf,
+ "floor_covering_reason": agent.get("floor_covering_reason") or local_fc_reason,
+ "floor_covering_source": fc_source,
+ }
+ items.append(entry["resolved"])
+ return {
+ "schema_version": "object_physical_properties",
+ "items": items,
+ "by_id": {item["object_id"]: item for item in items},
+ "floor_covering_object_ids": sorted(floor_covering_ids, key=sorted_object_id_key),
+ }
+
+
+def write_object_physical_properties_audit(output_dir: Path, properties: dict[str, Any], dry_run: bool) -> None:
+ data = {
+ "dry_run": dry_run,
+ "status": "completed",
+ "schema_version": properties.get("schema_version"),
+ "floor_covering_object_ids": properties.get("floor_covering_object_ids", []),
+ "items": properties.get("items", []),
+ "policy": "Relation Agents may supply floor-covering decisions only. Upright is owned by the independent Physics Sidecar and relation-aware final assembly.",
+ }
+ write_audit_pair(output_dir, "objects", "object_physical_properties_audit", data, "Object Physical Properties Audit")
+
+
+def load_async_sidecar_upright_priors(
+ output_dir: Path,
+ object_ids: list[str],
+) -> dict[str, Any]:
+ sidecar_path = output_dir / "physics_sidecar" / "integrated_async" / "object_physics.json"
+ audit_path = output_dir / "physics_sidecar" / "integrated_async" / "physics_audit.json"
+ result: dict[str, Any] = {
+ "schema_version": "sidecar_upright_priors",
+ "sidecar_path": str(sidecar_path),
+ "available": sidecar_path.exists(),
+ "by_id": {},
+ "missing_object_ids": list(object_ids),
+ "invalid_items": [],
+ "read_error": None,
+ }
+ if not sidecar_path.exists():
+ return result
+ try:
+ payload = json.loads(sidecar_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ result["read_error"] = str(exc)
+ return result
+
+ provenance: dict[str, str] = {}
+ if audit_path.exists():
+ try:
+ audit_payload = json.loads(audit_path.read_text(encoding="utf-8"))
+ provenance = {
+ str(item.get("object_id")): str(item.get("upright_source") or "unknown")
+ for item in audit_payload.get("items", [])
+ if isinstance(item, dict) and item.get("object_id")
+ }
+ except (OSError, json.JSONDecodeError):
+ provenance = {}
+
+ expected = set(object_ids)
+ for item in payload.get("objects", []) if isinstance(payload, dict) else []:
+ if not isinstance(item, dict):
+ continue
+ object_id = str(item.get("object_id") or "")
+ if object_id not in expected:
+ continue
+ upright = item.get("upright")
+ if isinstance(upright, bool):
+ result["by_id"][object_id] = {
+ "upright": upright,
+ "sidecar_source": provenance.get(object_id, "agent"),
+ }
+ else:
+ result["invalid_items"].append({
+ "object_id": object_id,
+ "upright": upright,
+ "reason": "sidecar upright is missing or is not a JSON boolean",
+ })
+ result["missing_object_ids"] = [
+ object_id for object_id in object_ids if object_id not in result["by_id"]
+ ]
+ return result
+
+
+def apply_upright_properties(
+ objects: list[dict[str, Any]],
+ relations: list[dict[str, Any]],
+ properties: dict[str, Any],
+ scene_state: dict[str, Any] | None,
+ output_dir: Path,
+ dry_run: bool,
+ upright_priors: dict[str, Any] | None = None,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ scene_by_id = _scene_object_context(scene_state)
+ props_by_id = properties.get("by_id", {}) if isinstance(properties, dict) else {}
+ sidecar_by_id = upright_priors.get("by_id", {}) if isinstance(upright_priors, dict) else {}
+ target_relation_anchors: dict[str, list[str]] = {}
+ for rel in relations:
+ if not (isinstance(rel, dict) and isinstance(rel.get("parameters"), dict)):
+ continue
+ target = rel.get("parameters", {}).get("object_target")
+ anchor = rel.get("parameters", {}).get("object_anchor")
+ if not target:
+ continue
+ target_relation_anchors.setdefault(str(target), []).append(str(anchor or ""))
+ wall_attach_only_targets = {
+ target
+ for target, anchors in target_relation_anchors.items()
+ if anchors and all(anchor.startswith("wall_") for anchor in anchors)
+ }
+ ceiling_attach_targets = {
+ rel.get("parameters", {}).get("object_target")
+ for rel in relations
+ if isinstance(rel, dict)
+ and rel.get("relation") == "attach"
+ and isinstance(rel.get("parameters"), dict)
+ and str(rel.get("parameters", {}).get("object_anchor", "")).lower() == "ceiling"
+ }
+ updated: list[dict[str, Any]] = []
+ audit_items: list[dict[str, Any]] = []
+ for obj in objects:
+ new_obj = dict(obj)
+ object_id = str(new_obj.get("id"))
+ prop = props_by_id.get(object_id, {}) if isinstance(props_by_id, dict) else {}
+ if object_id in wall_attach_only_targets:
+ upright = False
+ source = "wall_attach_only_override"
+ reason = "object is covered only by wall attach relations; it is treated as wall-constrained rather than gravity-upright"
+ elif object_id in ceiling_attach_targets:
+ upright = False
+ source = "ceiling_attach_override"
+ reason = "object has final ceiling attach; ceiling-mounted objects are not gravity-upright reconstruction constraints"
+ elif _is_ceiling_light_object(new_obj):
+ upright = False
+ source = "local_text_fallback"
+ reason = "category/description indicates a ceiling light fixture, which is not a gravity-upright reconstruction constraint"
+ elif bool(prop.get("is_floor_covering")):
+ upright = False
+ source = "floor_covering_override"
+ reason = "floor covering is not an upright reconstruction constraint"
+ elif isinstance(sidecar_by_id.get(object_id, {}).get("upright"), bool):
+ upright = bool(sidecar_by_id[object_id]["upright"])
+ sidecar_source = str(sidecar_by_id[object_id].get("sidecar_source") or "agent")
+ source = "async_physics_sidecar_agent" if sidecar_source == "agent" else "async_physics_sidecar_prior"
+ reason = "independent Physics Sidecar supplied the relation-agnostic canonical upright prior"
+ else:
+ upright, source, reason = _local_upright_fallback(new_obj, scene_by_id.get(object_id), bool(prop.get("is_floor_covering")))
+ new_obj["upright"] = bool(upright)
+ updated.append(new_obj)
+ audit_items.append(
+ {
+ "object_id": object_id,
+ "upright": bool(upright),
+ "source": source,
+ "reason": reason,
+ "is_floor_covering": bool(prop.get("is_floor_covering")),
+ "sidecar_upright": sidecar_by_id.get(object_id, {}).get("upright"),
+ "sidecar_upright_source": sidecar_by_id.get(object_id, {}).get("sidecar_source"),
+ }
+ )
+ audit = {
+ "dry_run": dry_run,
+ "status": "completed",
+ "schema_version": "upright_audit",
+ "items": audit_items,
+ "sidecar_upright_priors": {
+ "available": bool((upright_priors or {}).get("available")),
+ "sidecar_path": (upright_priors or {}).get("sidecar_path"),
+ "missing_object_ids": (upright_priors or {}).get("missing_object_ids", []),
+ "invalid_items": (upright_priors or {}).get("invalid_items", []),
+ "read_error": (upright_priors or {}).get("read_error"),
+ },
+ "policy": "wall-only, ceiling, and floor-covering relation overrides come first; the independent Physics Sidecar owns the general upright prior; local semantic/image/geometry fallback is used only when the sidecar prior is unavailable.",
+ }
+ write_audit_pair(output_dir, "objects", "upright_audit", audit, "Upright Audit")
+ return updated, audit
+
+
+def _normalize_contain_render_direction(value: Any) -> str | None:
+ raw = str(value or "").strip().lower()
+ if raw == "正面":
+ return "front"
+ if raw == "上面":
+ return "up"
+ normalized = _snake_case(raw)
+ if normalized in {"front", "forward", "front_view"}:
+ return "front"
+ if normalized in {"up", "top", "top_view", "above"}:
+ return "up"
+ return None
+
+
+def _semantic_contain_render_direction(obj: dict[str, Any], scene_obj: dict[str, Any] | None = None) -> str:
+ text = _object_text(obj, scene_obj)
+ front = _matches_semantic_hint(text, CONTAIN_FRONT_DIRECTION_HINTS)
+ up = _matches_semantic_hint(text, CONTAIN_UP_DIRECTION_HINTS)
+ if front:
+ return "front"
+ if up:
+ return "up"
+ extent = _bbox_extent_xyz(scene_obj)
+ if extent:
+ x, y, z = extent
+ if y <= max(x, z, 1e-6) * 0.45:
+ return "up"
+ return "front"
+
+
+def _contain_candidate_anchor_directions(scene_state: dict[str, Any] | None) -> tuple[dict[str, str], str | None, str | None]:
+ if not isinstance(scene_state, dict):
+ return {}, None, None
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state.get("evidence"), dict) else {}
+ relation_precompute = evidence.get("relation_precompute", {}) if isinstance(evidence.get("relation_precompute"), dict) else {}
+ outputs = relation_precompute.get("outputs", {}) if isinstance(relation_precompute.get("outputs"), dict) else {}
+ raw_path = outputs.get("contain_candidates_path")
+ if not raw_path:
+ return {}, None, None
+ path = Path(str(raw_path))
+ if not path.exists():
+ return {}, str(path), "contain_candidates_path does not exist"
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ return {}, str(path), str(exc)
+ directions: dict[str, str] = {}
+ for item in payload.get("possible_containers", []) if isinstance(payload, dict) else []:
+ if not isinstance(item, dict):
+ continue
+ object_id = item.get("object_id") or item.get("id")
+ direction = _normalize_contain_render_direction(
+ item.get("render_direction")
+ or item.get("anchor_render_direction")
+ or item.get("渲染方向")
+ )
+ if object_id and direction:
+ directions[str(object_id)] = direction
+ return directions, str(path), None
+
+
+def _agent_contain_anchor_directions(
+ task_results: list[dict[str, Any]],
+ guard_result: dict[str, Any] | None,
+) -> dict[str, dict[str, str]]:
+ result: dict[str, dict[str, str]] = {}
+ roots: list[Any] = list(task_results)
+ if guard_result:
+ roots.append(guard_result)
+ for root in roots:
+ source_task_id = str(root.get("task_id") or "schema_guard") if isinstance(root, dict) else "unknown"
+ for node in _walk(root):
+ if not isinstance(node, dict):
+ continue
+ parameters = node.get("parameters") if isinstance(node.get("parameters"), dict) else {}
+ anchor = (
+ node.get("object_anchor")
+ or node.get("anchor_object_id")
+ or parameters.get("object_anchor")
+ or (node.get("object_id") if _bool_or_none(node.get("contain_anchor")) else None)
+ )
+ direction = _normalize_contain_render_direction(
+ node.get("render_direction")
+ or node.get("anchor_render_direction")
+ or node.get("渲染方向")
+ )
+ if anchor and str(anchor).startswith("object_") and direction:
+ result[str(anchor)] = {"render_direction": direction, "source_task_id": source_task_id}
+ return result
+
+
+def apply_contain_anchor_properties(
+ objects: list[dict[str, Any]],
+ relations: list[dict[str, Any]],
+ task_results: list[dict[str, Any]],
+ guard_result: dict[str, Any] | None,
+ scene_state: dict[str, Any] | None,
+ output_dir: Path,
+ dry_run: bool,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ anchor_ids = {
+ str(rel.get("parameters", {}).get("object_anchor"))
+ for rel in relations
+ if isinstance(rel, dict)
+ and rel.get("relation") == "contain"
+ and isinstance(rel.get("parameters"), dict)
+ and str(rel.get("parameters", {}).get("object_anchor", "")).startswith("object_")
+ }
+ candidate_directions, candidate_path, candidate_read_error = _contain_candidate_anchor_directions(scene_state)
+ agent_directions = _agent_contain_anchor_directions(task_results, guard_result)
+ scene_by_id = _scene_object_context(scene_state)
+ updated: list[dict[str, Any]] = []
+ audit_items: list[dict[str, Any]] = []
+ for obj in objects:
+ new_obj = dict(obj)
+ object_id = str(new_obj.get("id"))
+ is_anchor = object_id in anchor_ids
+ direction = None
+ source = "not_a_contain_anchor"
+ if is_anchor:
+ if object_id in agent_directions:
+ direction = agent_directions[object_id]["render_direction"]
+ source = "agent_contain_property"
+ elif object_id in candidate_directions:
+ direction = candidate_directions[object_id]
+ source = "local_contain_candidate"
+ else:
+ direction = _semantic_contain_render_direction(new_obj, scene_by_id.get(object_id))
+ source = "local_semantic_or_geometry_fallback"
+ new_obj.pop("contain_anchor", None)
+ new_obj.pop("render_direction", None)
+ if is_anchor:
+ new_obj["contain_anchor"] = True
+ new_obj["render_direction"] = direction
+ updated.append(new_obj)
+ audit_items.append({
+ "object_id": object_id,
+ "contain_anchor": is_anchor,
+ "render_direction": direction,
+ "source": source,
+ "agent_direction": agent_directions.get(object_id),
+ "candidate_direction": candidate_directions.get(object_id),
+ })
+ audit = {
+ "dry_run": dry_run,
+ "status": "completed",
+ "schema_version": "contain_anchor_audit",
+ "contain_anchor_object_ids": sorted(anchor_ids, key=sorted_object_id_key),
+ "contain_candidates_path": candidate_path,
+ "contain_candidates_read_error": candidate_read_error,
+ "items": audit_items,
+ "policy": "Only final contain-relation anchors receive contain_anchor=true and render_direction=front|up; non-anchors omit both fields. Direction priority is Agent visual judgement, local candidate hint, then semantic/geometry fallback.",
+ }
+ write_audit_pair(output_dir, "objects", "contain_anchor_audit", audit, "Contain Anchor Audit")
+ return updated, audit
diff --git a/vlm_agent/reconstruction/relations/agents/assembly/relation_candidates.py b/vlm_agent/reconstruction/relations/agents/assembly/relation_candidates.py
new file mode 100644
index 0000000..4dbee7e
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/assembly/relation_candidates.py
@@ -0,0 +1,342 @@
+from __future__ import annotations
+
+import re
+from typing import Any
+
+from relations.agents.assembly.object_properties import _walk_with_path
+
+def _candidate_decision(candidate: dict[str, Any]) -> str | None:
+ decision = (
+ candidate.get("decision")
+ or candidate.get("status")
+ or candidate.get("result")
+ or candidate.get("relation_recommendation")
+ or candidate.get("recommendation")
+ )
+ return str(decision).lower() if decision is not None else None
+
+
+def _relation_from_candidate(candidate: dict[str, Any], source_path: str | None = None) -> dict[str, Any] | None:
+ relation_type = str(candidate.get("relation") or candidate.get("relation_type") or "").strip().lower()
+ source_path_l = str(source_path or "").lower()
+ contains_contain_fields = bool(
+ candidate.get("container")
+ or candidate.get("container_id")
+ or candidate.get("container_anchor")
+ or candidate.get("contained_object")
+ or candidate.get("contained_object_id")
+ )
+ if not relation_type and ("contain" in source_path_l or contains_contain_fields):
+ relation_type = "contain"
+ if relation_type not in {"attach", "contain"}:
+ relation_type = "attach"
+ if isinstance(candidate.get("parameters"), dict):
+ params = candidate["parameters"]
+ anchor = (
+ params.get("object_anchor")
+ or params.get("anchor")
+ or params.get("container")
+ or params.get("container_anchor")
+ or params.get("container_id")
+ or candidate.get("candidate_wall_id")
+ )
+ target = (
+ params.get("object_target")
+ or params.get("target")
+ or params.get("contained_object")
+ or params.get("contained_object_id")
+ or params.get("object_id")
+ or candidate.get("object_target")
+ )
+ else:
+ anchor = (
+ candidate.get("object_anchor")
+ or candidate.get("container")
+ or candidate.get("container_anchor")
+ or candidate.get("container_id")
+ or candidate.get("support_object_id")
+ or candidate.get("support_anchor_id")
+ or candidate.get("candidate_wall_id")
+ or candidate.get("wall_id")
+ or candidate.get("anchor")
+ or candidate.get("anchor_id")
+ or candidate.get("subject_id")
+ )
+ target = (
+ candidate.get("object_target")
+ or candidate.get("object_target_id")
+ or candidate.get("contained_object")
+ or candidate.get("contained_object_id")
+ or candidate.get("target_object_id")
+ or candidate.get("object_id")
+ )
+ if not anchor or not target:
+ return None
+ return {"relation": relation_type, "parameters": {"object_anchor": anchor, "object_target": target}}
+
+
+def _repair_inconsistent_support_candidate(candidate: dict[str, Any]) -> dict[str, Any] | None:
+ """Repair a structural contradiction using only fields the Agent supplied."""
+ if _candidate_decision(candidate) not in {"accept", "accepted", "true", "yes"}:
+ return None
+ if str(candidate.get("support_mode") or "").lower() != "exclusive_wall_or_ceiling":
+ return None
+ hypothesis = candidate.get("containment_hypothesis")
+ if not isinstance(hypothesis, dict):
+ return None
+ anchor = hypothesis.get("object_anchor") or hypothesis.get("anchor")
+ target = hypothesis.get("object_target") or hypothesis.get("target") or candidate.get("object_target")
+ reason = str(candidate.get("reason_code") or candidate.get("reason") or "").strip().lower()
+ surface_reason = bool(re.fullmatch(r"on_[a-z0-9_]+_surface", reason))
+ if not (
+ surface_reason
+ and "wall" not in reason
+ and "ceiling" not in reason
+ and isinstance(anchor, str)
+ and anchor.startswith("object_")
+ and isinstance(target, str)
+ and target.startswith("object_")
+ and anchor != target
+ ):
+ return None
+ return {
+ "decision": "accept",
+ "relation": "attach",
+ "parameters": {"object_anchor": anchor, "object_target": target},
+ "reason_code": reason,
+ "support_candidate_consistency_repair": {
+ "original_support_mode": "exclusive_wall_or_ceiling",
+ "repaired_support_mode": "object",
+ "source_anchor_field": "containment_hypothesis.object_anchor",
+ "reason": "accepted object-surface reason contradicted exclusive wall/ceiling mode",
+ },
+ }
+
+
+
+def _relation_from_accepted_support_candidate(candidate: dict[str, Any]) -> dict[str, Any] | None:
+ """Compile one accepted gravity-support verdict into the final relation schema."""
+ if _candidate_decision(candidate) not in {"accept", "accepted", "true", "yes"}:
+ return None
+ support_mode = str(candidate.get("support_mode") or "").strip().lower()
+ target = candidate.get("object_target") or candidate.get("target") or candidate.get("object_id")
+ anchor = candidate.get("object_anchor") or candidate.get("anchor")
+ if not (isinstance(target, str) and target.startswith("object_")):
+ return None
+ if support_mode == "floor":
+ anchor = "floor"
+ elif support_mode == "object":
+ if not (isinstance(anchor, str) and anchor.startswith("object_") and anchor != target):
+ return None
+ else:
+ return None
+ return {
+ "decision": "accept",
+ "relation": "attach",
+ "parameters": {"object_anchor": anchor, "object_target": target},
+ "reason_code": candidate.get("reason_code") or candidate.get("reason"),
+ "compiled_from_support_contact_candidate": True,
+ }
+
+def collect_accepted_attach_candidates(task_results: list[dict[str, Any]], guard_result: dict[str, Any] | None = None) -> dict[str, Any]:
+ roots: list[Any] = list(task_results)
+ if guard_result:
+ roots.append(guard_result)
+ accepted: list[dict[str, Any]] = []
+ skipped: list[dict[str, Any]] = []
+ intermediate_decisions = {
+ "needs_consistency_check",
+ "needs_wall_resolution",
+ "not_final_yet",
+ "pending",
+ "possible_wall_contact",
+ "needs_more_evidence",
+ "candidate_pending_wall_resolution",
+ }
+
+ def add_candidate(candidate: dict[str, Any], source_path: str, acceptance_mode: str) -> None:
+ relation = _relation_from_candidate(candidate, source_path)
+ decision = _candidate_decision(candidate)
+ if decision in intermediate_decisions or (decision and ("pending" in decision or decision.startswith("needs_"))):
+ skipped.append({"candidate": candidate, "source_path": source_path, "reason": "intermediate_wall_attach_candidate_not_final"})
+ return
+ if acceptance_mode == "requires_accept" and decision not in {"accept", "accepted", "true", "yes"}:
+ skipped.append({"candidate": candidate, "source_path": source_path, "reason": "decision_not_accept"})
+ return
+ if acceptance_mode == "implicit_unless_reject" and decision in {"reject", "rejected", "false", "no"}:
+ skipped.append({"candidate": candidate, "source_path": source_path, "reason": "decision_reject"})
+ return
+ if not relation:
+ skipped.append({"candidate": candidate, "source_path": source_path, "reason": "could_not_normalize_relation"})
+ return
+ accepted.append({"relation": relation, "source_path": source_path, "raw_candidate": candidate, "decision": decision})
+
+ def add_nested_candidates(node: dict[str, Any], source_path: str, acceptance_mode: str) -> bool:
+ candidates = node.get("candidates")
+ if not isinstance(candidates, list):
+ return False
+ parent_target = node.get("object_target") or node.get("object_id")
+ any_child = False
+ for idx, child in enumerate(candidates):
+ if not isinstance(child, dict):
+ continue
+ candidate = dict(child)
+ if parent_target and not candidate.get("object_target"):
+ candidate["object_target"] = parent_target
+ if candidate.get("anchor_id") and not candidate.get("object_anchor"):
+ candidate["object_anchor"] = candidate.get("anchor_id")
+ add_candidate(candidate, f"{source_path}.candidates[{idx}]", acceptance_mode)
+ any_child = True
+ return any_child
+
+ def add_candidate_or_nested(candidate: dict[str, Any], source_path: str, acceptance_mode: str) -> None:
+ if add_nested_candidates(candidate, source_path, acceptance_mode):
+ return
+ add_candidate(candidate, source_path, acceptance_mode)
+
+ for root_idx, root in enumerate(roots):
+ root_name = root.get("task_id", f"root_{root_idx}") if isinstance(root, dict) else f"root_{root_idx}"
+ for path, node in _walk_with_path(root, root_name):
+ if not isinstance(node, dict):
+ continue
+ support_values = node.get("support_contact_candidates")
+ if isinstance(support_values, list):
+ for idx, item in enumerate(support_values):
+ if not isinstance(item, dict):
+ continue
+ compiled = _relation_from_accepted_support_candidate(item)
+ if compiled:
+ add_candidate(
+ compiled,
+ f"{path}.support_contact_candidates[{idx}].compiled_support_verdict",
+ "requires_accept",
+ )
+ repaired = _repair_inconsistent_support_candidate(item)
+ if repaired:
+ add_candidate(
+ repaired,
+ f"{path}.support_contact_candidates[{idx}].consistency_repair",
+ "requires_accept",
+ )
+ for key in ["accepted_relations", "accepted_attach_candidates", "accepted_contain_relations", "accepted_contain_candidates", "accepted_contain_findings", "verified_wall_attach_relations", "accepted_wall_attach_relations", "verified_wall_attach_candidates"]:
+ value = node.get(key)
+ if isinstance(value, list):
+ for item in value:
+ if isinstance(item, dict):
+ add_candidate_or_nested(item, f"{path}.{key}", "implicit_unless_reject")
+ psu = node.get("proposed_state_updates")
+ if isinstance(psu, dict):
+ for key in ["wall_attach_candidates", "attach_candidates", "contain_candidates", "contain_relations", "contain_findings", "accepted_relations", "accepted_contain_relations", "accepted_contain_candidates", "accepted_contain_findings", "verified_wall_attach_relations", "accepted_wall_attach_relations", "verified_wall_attach_candidates", "final_relation_candidates", "final_attach_relation_candidates", "final_contain_relation_candidates", "proposed_attach_relations", "proposed_contain_relations"]:
+ value = psu.get(key)
+ if isinstance(value, list):
+ for item in value:
+ if isinstance(item, dict):
+ add_candidate_or_nested(item, f"{path}.proposed_state_updates.{key}", "implicit_unless_reject")
+ findings = node.get("findings")
+ if isinstance(findings, dict):
+ for key in ["wall_attach_candidates", "attach_candidates", "contain_candidates", "contain_relations", "contain_findings", "relation_candidates", "relations"]:
+ value = findings.get(key)
+ if isinstance(value, list):
+ for item in value:
+ if isinstance(item, dict):
+ add_candidate_or_nested(item, f"{path}.findings.{key}", "requires_accept")
+ for key in ["accepted_relations", "accepted_attach_candidates", "accepted_contain_relations", "accepted_contain_candidates", "accepted_contain_findings", "final_relation_candidates", "final_attach_relation_candidates", "final_contain_relation_candidates", "proposed_attach_relations", "proposed_contain_relations"]:
+ value = findings.get(key)
+ if isinstance(value, list):
+ for item in value:
+ if isinstance(item, dict):
+ add_candidate_or_nested(item, f"{path}.findings.{key}", "implicit_unless_reject")
+ for key in ["relations", "relation_candidates", "wall_attach_candidates", "wall_attach_recall", "attach_candidates", "contain_candidates", "contain_relations", "contain_findings"]:
+ value = node.get(key)
+ if isinstance(value, list):
+ for item in value:
+ if isinstance(item, dict):
+ add_candidate_or_nested(item, f"{path}.{key}", "requires_accept")
+ if isinstance(node.get("relations_json"), list):
+ for item in node["relations_json"]:
+ if isinstance(item, dict):
+ add_candidate_or_nested(item, f"{path}.relations_json", "implicit_unless_reject")
+ return {"accepted": accepted, "skipped": skipped}
+
+def collect_coverage_repair_candidates(task_results: list[dict[str, Any]], guard_result: dict[str, Any] | None = None) -> dict[str, Any]:
+ roots: list[Any] = list(task_results)
+ if guard_result:
+ roots.append(guard_result)
+ accepted: list[dict[str, Any]] = []
+ skipped: list[dict[str, Any]] = []
+
+ def add_candidate(candidate: dict[str, Any], source_path: str, implicit: bool = False) -> None:
+ if not isinstance(candidate, dict):
+ return
+ if not implicit and candidate.get("coverage_repair") is not True:
+ skipped.append({"candidate": candidate, "source_path": source_path, "reason": "not_marked_coverage_repair"})
+ return
+ relation = _relation_from_candidate(candidate)
+ if not relation:
+ skipped.append({"candidate": candidate, "source_path": source_path, "reason": "could_not_normalize_relation"})
+ return
+ accepted.append(
+ {
+ "relation": relation,
+ "source_path": source_path,
+ "raw_candidate": candidate,
+ "reason": candidate.get("repair_reason") or candidate.get("reason") or candidate.get("rationale"),
+ "evidence_used": candidate.get("evidence_used") or candidate.get("evidence") or [],
+ "coverage_repair": True,
+ }
+ )
+
+ for root_idx, root in enumerate(roots):
+ root_name = root.get("task_id", f"root_{root_idx}") if isinstance(root, dict) else f"root_{root_idx}"
+ for path, node in _walk_with_path(root, root_name):
+ if not isinstance(node, dict):
+ continue
+ psu = node.get("proposed_state_updates")
+ if isinstance(psu, dict):
+ for key in ["coverage_repair_relations", "coverage_repair_candidates"]:
+ value = psu.get(key)
+ if isinstance(value, list):
+ for item in value:
+ add_candidate(item, f"{path}.proposed_state_updates.{key}", implicit=True)
+ findings = node.get("findings")
+ if isinstance(findings, dict):
+ for key in ["coverage_repair_relations", "coverage_repair_candidates"]:
+ value = findings.get(key)
+ if isinstance(value, list):
+ for item in value:
+ add_candidate(item, f"{path}.findings.{key}", implicit=True)
+ for key in ["coverage_repair_relations", "coverage_repair_candidates"]:
+ value = node.get(key)
+ if isinstance(value, list):
+ for item in value:
+ add_candidate(item, f"{path}.{key}", implicit=True)
+ for key in ["accepted_relations", "relation_candidates"]:
+ value = node.get(key)
+ if isinstance(value, list):
+ for item in value:
+ add_candidate(item, f"{path}.{key}", implicit=False)
+ return {"accepted": accepted, "skipped": skipped}
+
+
+
+WALL_MOUNTED_KEEP_KEYWORDS = {
+ "attached to wall",
+ "embedded",
+ "mounted",
+ "wall mounted",
+ "wall-mounted",
+ "wall-hung",
+}
+
+# Exact semantic categories that are wall-constrained even when the free-text
+# description omits words such as "mounted" or "hanging".
+WALL_MOUNTED_CATEGORIES = {
+ "acoustic_panel",
+ "art_panel",
+ "bulletin_board",
+ "painting",
+ "projection_screen",
+ "wall_art",
+ "wall_panel",
+ "whiteboard",
+}
diff --git a/vlm_agent/reconstruction/relations/agents/assembly/wall_assembly.py b/vlm_agent/reconstruction/relations/agents/assembly/wall_assembly.py
new file mode 100644
index 0000000..bbbcef3
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/assembly/wall_assembly.py
@@ -0,0 +1,1383 @@
+from __future__ import annotations
+
+import json
+import math
+from pathlib import Path
+from typing import Any
+
+from relations.agents.assembly.object_properties import _walk
+from relations.agents.schema_guard import WALL_IDS, validate_wall_geometry
+from relations.agents.state import write_audit_pair
+from relations.agents.wall_attach_prefilter import wall_geometry_key
+from relations.tools.wall_canonical_naming import canonical_wall_naming_policy, final_wall_naming_audit
+
+FORMAL_OBB_WALL_METHOD = "obb_from_mesh_footprint_min_rect"
+FORMAL_OBB_WALL_SOURCE = "obb_from_mesh_footprint_min_rect"
+
+ALLOWED_FINAL_WALL_SOURCES = {
+ FORMAL_OBB_WALL_SOURCE,
+}
+INTERMEDIATE_WALL_SOURCES = {
+ "layout_xz_boundary_points",
+}
+HINT_ONLY_WALL_SOURCES = {
+ "layout_bounds_hint",
+ "object_footprint_bounds_hint",
+ "aggregate_bbox_hint",
+ "grid_extent_hint",
+ "candidate_room_polygon_hint",
+ "canonical_grid_mapping.grid_points_extent_hint",
+ "canonical_grid_mapping.x_range_z_range_hint",
+ "wall_attached_object_distribution",
+ "topdown_camera_config.scene_bbox_global_xz_hint",
+ "topdown_camera_config.x_range_z_range_hint",
+ "scene_state.object_candidates.bbox_3d_aggregate_hint",
+}
+DEFAULT_WALL_PROMOTION_EVIDENCE = [
+ "contact_sheet",
+ "candidate_walls_overlay",
+ "canonical_topdown",
+ "fixed_layout_views",
+ "original_image",
+ "mask_overlay",
+ "object_distribution",
+]
+
+def _wall_source_from_context(wall: dict[str, Any], node: dict[str, Any], root: dict[str, Any]) -> str | None:
+ return (
+ wall.get("source")
+ or wall.get("wall_source")
+ or node.get("source")
+ or node.get("wall_source")
+ or root.get("source")
+ or root.get("wall_source")
+ )
+
+
+def _wall_verification_status(wall: dict[str, Any], node: dict[str, Any], root: dict[str, Any]) -> str | None:
+ raw = wall.get("topdown_verification") or wall.get("topdown_verification_status") or node.get("topdown_verification") or root.get("topdown_verification")
+ if isinstance(raw, dict):
+ raw = raw.get("status") or raw.get("result")
+ return str(raw).lower() if raw is not None else None
+
+
+def _truthy(value: Any) -> bool:
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ return value.strip().lower() in {"true", "yes", "1", "promoted", "candidate_promotion"}
+ return bool(value)
+
+
+def _listify_strings(value: Any) -> list[str]:
+ if isinstance(value, list):
+ return [str(item) for item in value if item]
+ if isinstance(value, str) and value.strip():
+ return [value.strip()]
+ return []
+
+
+def _candidate_promotion_audit(
+ wall: dict[str, Any],
+ node: dict[str, Any],
+ root: dict[str, Any],
+ original_source: str | None,
+ normalized_source: str | None,
+ topdown_verification: str | None,
+ evidence: Any,
+ source_normalized: bool,
+) -> dict[str, Any]:
+ evidence_text = str(evidence or "")
+ evidence_l = evidence_text.lower()
+ candidate_original_source = (
+ wall.get("candidate_original_source")
+ or wall.get("promoted_from_source")
+ or wall.get("promotion_original_source")
+ or node.get("candidate_original_source")
+ or node.get("promoted_from_source")
+ or root.get("candidate_original_source")
+ or root.get("promoted_from_source")
+ )
+ if not candidate_original_source:
+ if original_source == "layout_xz_boundary_points":
+ candidate_original_source = "layout_xz_boundary_points"
+ elif "layout_xz_boundary_points" in evidence_l:
+ candidate_original_source = "layout_xz_boundary_points"
+ explicit_promotion = any(
+ _truthy(value)
+ for value in [
+ wall.get("promotion_from_candidate"),
+ wall.get("promoted_from_candidate"),
+ node.get("promotion_from_candidate"),
+ node.get("promoted_from_candidate"),
+ root.get("promotion_from_candidate"),
+ root.get("promoted_from_candidate"),
+ ]
+ )
+ text_promotion = "promot" in evidence_l and ("candidate" in evidence_l or "layout_xz_boundary_points" in evidence_l)
+ source_promotion = source_normalized and original_source == "layout_xz_boundary_points"
+ promotion = bool(
+ normalized_source == "obb_from_mesh_footprint_min_rect"
+ and topdown_verification == "passed"
+ and (explicit_promotion or text_promotion or source_promotion or candidate_original_source)
+ )
+ verification_evidence_used = (
+ _listify_strings(wall.get("verification_evidence_used"))
+ or _listify_strings(wall.get("visual_evidence_used"))
+ or _listify_strings(node.get("verification_evidence_used"))
+ or _listify_strings(root.get("verification_evidence_used"))
+ or (DEFAULT_WALL_PROMOTION_EVIDENCE if promotion else [])
+ )
+ promotion_reason = (
+ wall.get("promotion_reason")
+ or node.get("promotion_reason")
+ or root.get("promotion_reason")
+ or evidence_text
+ or "layout-XZ candidate wall coordinates were visually verified before promotion"
+ )
+ candidate_modified = any(
+ _truthy(value)
+ for value in [
+ wall.get("candidate_modified"),
+ node.get("candidate_modified"),
+ root.get("candidate_modified"),
+ ]
+ )
+ revision_reason = wall.get("revision_reason") or node.get("revision_reason") or root.get("revision_reason")
+ if candidate_modified:
+ promotion = bool(normalized_source == "obb_from_mesh_footprint_min_rect" and topdown_verification == "passed")
+ if not candidate_original_source:
+ candidate_original_source = "layout_xz_boundary_points"
+ if not revision_reason:
+ revision_reason = evidence_text or "candidate wall coordinates were revised using layout/visual evidence"
+ return {
+ "promotion_from_candidate": promotion,
+ "candidate_modified": candidate_modified,
+ "revision_reason": revision_reason,
+ "candidate_original_source": str(candidate_original_source) if candidate_original_source else None,
+ "normalized_source": normalized_source,
+ "verification_evidence_used": verification_evidence_used,
+ "promotion_reason": promotion_reason if promotion else None,
+ }
+
+
+def _normalize_wall_source(source: str | None, topdown_verification: str | None) -> tuple[str | None, bool, str | None]:
+ return source, False, None
+
+
+def _wall_source_allowed(source: str | None, emergency_fallback: bool = False) -> bool:
+ if source in ALLOWED_FINAL_WALL_SOURCES:
+ return True
+ if emergency_fallback:
+ return True
+ return False
+
+
+def _normalize_wall_set(walls: list[Any], node: dict[str, Any], root: dict[str, Any]) -> tuple[list[dict[str, Any]], list[dict[str, Any]], list[str]]:
+ warnings: list[str] = []
+ normalized: list[dict[str, Any]] = []
+ records: list[dict[str, Any]] = []
+ for wall in walls:
+ if not isinstance(wall, dict):
+ continue
+ original_source = _wall_source_from_context(wall, node, root)
+ topdown_verification = _wall_verification_status(wall, node, root)
+ source, source_normalized, normalization_reason = _normalize_wall_source(original_source, topdown_verification)
+ emergency = bool(wall.get("emergency_fallback") or node.get("emergency_fallback") or root.get("emergency_fallback"))
+ evidence = wall.get("evidence") or wall.get("audit_evidence") or node.get("evidence") or node.get("evidence_used")
+ promotion_audit = _candidate_promotion_audit(wall, node, root, original_source, source, topdown_verification, evidence, source_normalized)
+ evidence_strength = wall.get("evidence_strength") or node.get("evidence_strength") or root.get("evidence_strength")
+ completion_method = wall.get("completion_method") or node.get("completion_method") or root.get("completion_method")
+ completion_reason = wall.get("completion_reason") or node.get("completion_reason") or root.get("completion_reason")
+ canonical_naming = wall.get("canonical_naming") if isinstance(wall.get("canonical_naming"), dict) else {}
+ candidate_or_geometry_wall_id = (
+ wall.get("candidate_or_geometry_wall_id")
+ or wall.get("candidate_wall_id")
+ or wall.get("geometry_wall_id")
+ or wall.get("provisional_wall_id")
+ or canonical_naming.get("candidate_or_geometry_wall_id")
+ or canonical_naming.get("geometry_wall_id")
+ )
+ original_image_visibility = (
+ wall.get("original_image_visibility")
+ or wall.get("visibility")
+ or canonical_naming.get("original_image_visibility")
+ or node.get("original_image_visibility")
+ or root.get("original_image_visibility")
+ )
+ visible_role = wall.get("visible_role") or canonical_naming.get("visible_role") or wall.get("wall_visible_role")
+ original_visible_wall_count = (
+ wall.get("original_visible_wall_count")
+ or canonical_naming.get("original_visible_wall_count")
+ or node.get("original_visible_wall_count")
+ or root.get("original_visible_wall_count")
+ )
+ visibility_text = str(original_image_visibility or "").lower()
+ closure_only = _truthy(wall.get("closure_only") or node.get("closure_only") or root.get("closure_only"))
+ weak_wall = bool(
+ _truthy(wall.get("weak_wall") or node.get("weak_wall") or root.get("weak_wall"))
+ or closure_only
+ or str(evidence_strength or "").lower() == "weak"
+ or str(completion_method or "").lower() == "inferred_by_room_closure"
+ or visibility_text in {"hidden", "not_visible", "not directly visible", "inferred", "occluded"}
+ )
+ record = {
+ "wall_id": wall.get("id"),
+ "id": wall.get("id"),
+ "candidate_or_geometry_wall_id": candidate_or_geometry_wall_id,
+ "candidate_wall_id": wall.get("candidate_wall_id"),
+ "geometry_wall_id": wall.get("geometry_wall_id"),
+ "provisional_wall_id": wall.get("provisional_wall_id"),
+ "first_point": wall.get("first_point"),
+ "second_point": wall.get("second_point"),
+ "original_source": original_source,
+ "normalized_source": source,
+ "source": source,
+ "source_normalized": source_normalized,
+ "evidence": evidence,
+ "topdown_verification": topdown_verification,
+ "evidence_strength": evidence_strength,
+ "completion_method": completion_method,
+ "completion_reason": completion_reason,
+ "weak_wall": weak_wall,
+ "closure_only": closure_only,
+ "original_image_visibility": original_image_visibility,
+ "visible_role": visible_role,
+ "original_visible_wall_count": original_visible_wall_count,
+ "canonical_naming": canonical_naming,
+ "fixed_views_used": wall.get("fixed_views_used") or node.get("fixed_views_used") or [],
+ "promotion_from_candidate": promotion_audit["promotion_from_candidate"],
+ "candidate_modified": promotion_audit["candidate_modified"],
+ "revision_reason": promotion_audit["revision_reason"],
+ "candidate_original_source": promotion_audit["candidate_original_source"],
+ "verification_evidence_used": promotion_audit["verification_evidence_used"],
+ "promotion_reason": promotion_audit["promotion_reason"],
+ "emergency_fallback": emergency,
+ "accepted": False,
+ "accepted_as_final_wall": False,
+ "reason": normalization_reason,
+ }
+ if normalization_reason:
+ warnings.append(f"{normalization_reason}: `{wall.get('id')}`.")
+ if not _wall_source_allowed(source, emergency_fallback=emergency):
+ reason = "missing_source" if not source else "hint_only_or_untrusted_source"
+ if original_source in HINT_ONLY_WALL_SOURCES or source in HINT_ONLY_WALL_SOURCES:
+ reason = "hint_only_source_rejected"
+ if original_source == "layout_xz_boundary_points" and topdown_verification != "passed":
+ reason = "layout_xz_boundary_points_requires_topdown_verification_passed"
+ record["rejection_reason"] = reason
+ record["reason"] = record.get("reason") or reason
+ warnings.append(f"Rejected wall `{wall.get('id')}` as final wall because source `{original_source}` normalized to `{source}` is not final-eligible.")
+ else:
+ record["accepted"] = True
+ record["accepted_as_final_wall"] = True
+ record["reason"] = record.get("reason") or "final wall source is layout XZ coordinates with topdown verification or manual layout XZ walls"
+ normalized.append({"id": wall.get("id"), "first_point": wall.get("first_point"), "second_point": wall.get("second_point")})
+ if emergency:
+ warnings.append(f"Accepted wall `{wall.get('id')}` only as emergency fallback; final validation should treat this as warning.")
+ records.append(record)
+ return normalized, records, warnings
+
+
+def _is_wall_final_adjudication_node(node: dict[str, Any], root: dict[str, Any]) -> bool:
+ task_type = str(node.get("task_type") or "").lower()
+ task_id = str(node.get("task_id") or "").lower()
+ decision = str(node.get("decision") or "").lower()
+ if node is root:
+ task_type = task_type or str(root.get("task_type") or "").lower()
+ task_id = task_id or str(root.get("task_id") or "").lower()
+ decision = decision or str(root.get("decision") or "").lower()
+ return (
+ task_type == "wall_final_adjudication"
+ or "wall_final_adjudication" in task_id
+ or decision in {"verified_final_walls", "insufficient_layout_xz_wall_evidence"}
+ )
+
+
+def _wall_final_adjudication_decision(node: dict[str, Any], root: dict[str, Any]) -> str | None:
+ decision = node.get("decision") or root.get("decision")
+ if decision is None:
+ return None
+ decision_s = str(decision).lower()
+ if decision_s in {"accepted_candidate", "accept_candidate", "accepted_relation"}:
+ return "accept"
+ return decision_s
+
+
+def summarize_wall_final_adjudication(task_results: list[dict[str, Any]], guard_result: dict[str, Any] | None = None, accepted_walls: list[dict[str, Any]] | None = None) -> dict[str, Any]:
+ roots: list[Any] = list(task_results)
+ if guard_result:
+ roots.append(guard_result)
+ summary = {
+ "executed": False,
+ "decision": None,
+ "accepted_final_walls": bool(accepted_walls and len(accepted_walls) == 4),
+ "invalid_intermediate_output": False,
+ "items": [],
+ }
+ for root in roots:
+ if not isinstance(root, dict):
+ continue
+ for node in _walk(root):
+ if not isinstance(node, dict) or not _is_wall_final_adjudication_node(node, root):
+ continue
+ decision = _wall_final_adjudication_decision(node, root)
+ walls = node.get("walls") or node.get("accepted_walls") or node.get("wall_candidates") or []
+ wall_count = len(walls) if isinstance(walls, list) else 0
+ accepted_count = 0
+ if isinstance(walls, list) and wall_count:
+ normalized, _, _ = _normalize_wall_set(walls, node, root)
+ accepted_ids = {wall.get("id") for wall in normalized if isinstance(wall, dict)}
+ if set(WALL_IDS).issubset(accepted_ids):
+ accepted_count = 4
+ else:
+ accepted_count = len(accepted_ids.intersection(WALL_IDS))
+ candidate_modified = any(isinstance(wall, dict) and _truthy(wall.get("candidate_modified")) for wall in walls) if isinstance(walls, list) else False
+ revision_reasons = [str(wall.get("revision_reason")) for wall in walls if isinstance(wall, dict) and wall.get("revision_reason")] if isinstance(walls, list) else []
+ item = {
+ "task_id": node.get("task_id") or root.get("task_id"),
+ "task_type": node.get("task_type") or root.get("task_type"),
+ "decision": decision,
+ "wall_count": wall_count,
+ "accepted_final_wall_count": accepted_count,
+ "candidate_modified": candidate_modified,
+ "revision_used": candidate_modified,
+ "revision_reasons": revision_reasons,
+ "reason": node.get("reason") or root.get("reason"),
+ }
+ summary["executed"] = True
+ summary["items"].append(item)
+ if decision == "verified_final_walls":
+ if accepted_count == 4:
+ summary["decision"] = "verified_final_walls"
+ else:
+ summary["decision"] = "invalid_intermediate_output"
+ summary["invalid_intermediate_output"] = True
+ elif decision == "insufficient_layout_xz_wall_evidence":
+ summary["decision"] = "insufficient_layout_xz_wall_evidence"
+ elif wall_count:
+ summary["decision"] = summary.get("decision") or "invalid_intermediate_output"
+ summary["invalid_intermediate_output"] = True
+ summary["candidate_modified"] = any(item.get("candidate_modified") for item in summary.get("items", []))
+ summary["revision_used"] = bool(summary["candidate_modified"])
+ summary["revision_reasons"] = [reason for item in summary.get("items", []) for reason in item.get("revision_reasons", [])]
+ if summary["accepted_final_walls"] and summary.get("decision") in {None, "verified_final_walls"}:
+ summary["decision"] = "verified_final_walls"
+ return summary
+
+def collect_walls(
+ task_results: list[dict[str, Any]],
+ guard_result: dict[str, Any] | None = None,
+ wall_evidence: dict[str, Any] | None = None,
+) -> tuple[list[dict[str, Any]], list[str], list[dict[str, Any]]]:
+ roots: list[Any] = list(task_results)
+ if guard_result:
+ roots.append(guard_result)
+ warnings: list[str] = []
+ all_records: list[dict[str, Any]] = []
+ for root in roots:
+ if not isinstance(root, dict):
+ continue
+ for node in _walk(root):
+ if not isinstance(node, dict):
+ continue
+ walls = node.get("accepted_walls") or node.get("walls") or node.get("wall_candidates")
+ visibility_review = node.get("wall_visibility_review")
+ if isinstance(visibility_review, list):
+ for item in visibility_review:
+ if not isinstance(item, dict):
+ continue
+ wall_id = item.get("wall_id") or item.get("id")
+ if wall_id:
+ all_records.append(
+ {
+ "id": wall_id,
+ "wall_id": wall_id,
+ "accepted_as_final_wall": False,
+ "visibility_review_only": True,
+ "original_image_visibility": item.get("original_image_visibility") or item.get("visibility"),
+ "visible_role": item.get("visible_role"),
+ "weak_wall": _truthy(item.get("weak_wall")),
+ "closure_only": _truthy(item.get("closure_only")),
+ "reason": item.get("reason"),
+ "source_task_id": root.get("task_id"),
+ }
+ )
+ is_final_adjudication = _is_wall_final_adjudication_node(node, root)
+ decision = _wall_final_adjudication_decision(node, root)
+ if is_final_adjudication and decision == "insufficient_layout_xz_wall_evidence":
+ reason = node.get("reason") or root.get("reason") or "wall final adjudication returned insufficient evidence"
+ warnings.append(f"wall_final_adjudication returned insufficient_layout_xz_wall_evidence: {reason}")
+ all_records.append({
+ "task_id": node.get("task_id") or root.get("task_id"),
+ "task_type": node.get("task_type") or root.get("task_type"),
+ "decision": decision,
+ "accepted_as_final_wall": False,
+ "rejection_reason": "wall_final_adjudication_insufficient",
+ "reason": reason,
+ })
+ continue
+ if isinstance(walls, list) and (len(walls) >= 4 or is_final_adjudication):
+ normalized, records, wall_warnings = _normalize_wall_set(walls, node, root)
+ warnings.extend(wall_warnings)
+ all_records.extend(records)
+ wall_ids = {wall.get("id") for wall in normalized}
+ if set(WALL_IDS).issubset(wall_ids):
+ normalized = sorted([w for w in normalized if w.get("id") in WALL_IDS], key=lambda w: WALL_IDS.index(w["id"]))
+ return normalized[:4], warnings, all_records
+ if is_final_adjudication:
+ warnings.append("wall_final_adjudication_returned_intermediate_state")
+ all_records.append({
+ "task_id": node.get("task_id") or root.get("task_id"),
+ "task_type": node.get("task_type") or root.get("task_type"),
+ "decision": decision or "invalid_intermediate_output",
+ "accepted_as_final_wall": False,
+ "rejection_reason": "wall_final_adjudication_returned_intermediate_state",
+ })
+ if isinstance(wall_evidence, dict) and wall_evidence.get("candidate_walls"):
+ warnings.append("Structured wall_evidence.candidate_walls are hint_only and are not accepted as final walls without a final-eligible source.")
+ for wall in wall_evidence.get("candidate_walls", []):
+ if isinstance(wall, dict):
+ all_records.append(
+ {
+ "id": wall.get("id"),
+ "first_point": wall.get("first_point"),
+ "second_point": wall.get("second_point"),
+ "source": wall.get("source"),
+ "trust_level": wall.get("trust_level", "hint_only"),
+ "may_be_used_as_final_wall": wall.get("may_be_used_as_final_wall", False),
+ "accepted_as_final_wall": False,
+ "rejection_reason": "wall_evidence_hint_only",
+ }
+ )
+ warnings.append("No complete wall_001..wall_004 set with final-eligible source was found.")
+ return [], warnings, all_records
+
+
+WALL_SEMANTIC_CONFLICT_RESOLUTION_PRIORITY = 350
+
+
+def _wall_semantic_source_priority(root: dict[str, Any]) -> int:
+ """Prefer candidate-panel semantics once local code binds them to final geometry."""
+ text = " ".join(
+ str(root.get(key) or "")
+ for key in ("task_id", "task_type", "canonical_task_kind", "semantic_authority")
+ ).lower()
+ if "selected_candidate_provisional_visibility" in text or "local_provisional_wall_visibility_binding" in text:
+ return 300
+ if "wall_final_adjudication" in text:
+ return 200
+ provided = root.get("provided_capabilities") if isinstance(root.get("provided_capabilities"), list) else []
+ if "wall_visibility_semantics_ready" in provided:
+ return 100
+ if "wall_attach" in text:
+ return 50
+ return 25
+
+
+def collect_wall_final_semantics(
+ task_results: list[dict[str, Any]],
+ guard_result: dict[str, Any] | None = None,
+) -> dict[str, dict[str, Any]]:
+ """Read visibility metadata with explicit source priority.
+
+ The local provisional binding normally has the highest priority because it is
+ Agent semantic evidence tied to the selected finite candidate by immutable
+ geometry keys. The only higher-priority exception is one wall that the later
+ recall Agent both reclassifies as visible and accepts for an exact locally
+ flagged semantic/geometry conflict pair.
+ """
+ semantics: dict[str, dict[str, Any]] = {}
+ roots = list(task_results) + ([guard_result] if isinstance(guard_result, dict) else [])
+ semantic_keys = [
+ "original_image_visibility",
+ "visible_role",
+ "original_visible_wall_count",
+ "weak_wall",
+ "inferred",
+ "closure_only",
+ "can_anchor_wall_attach",
+ "attach_eligible",
+ "evidence_strength",
+ "completion_method",
+ "completion_reason",
+ "reason",
+ "reason_code",
+ "evidence_code",
+ "semantic_visibility_geometry_conflict_resolved",
+ "semantic_visibility_geometry_conflict_object_ids",
+ ]
+ for root in roots:
+ if not isinstance(root, dict):
+ continue
+ task_text = " ".join(str(root.get(key) or "") for key in ["task_id", "task_type", "canonical_task_kind"]).lower()
+ provided_capabilities = root.get("provided_capabilities") if isinstance(root.get("provided_capabilities"), list) else []
+ if "wall_final_adjudication" not in task_text and "wall_visibility_semantics_ready" not in provided_capabilities:
+ continue
+ source_priority = _wall_semantic_source_priority(root)
+ numbered_semantic_sets = [
+ node.get("wall_visibility_semantics")
+ for node in _walk(root)
+ if isinstance(node, dict) and isinstance(node.get("wall_visibility_semantics"), list)
+ ]
+ has_complete_geometry_bound_semantics = any(
+ {str(item.get("wall_id") or item.get("id") or "") for item in items if isinstance(item, dict)} == set(WALL_IDS)
+ and all(
+ isinstance(item, dict) and bool(item.get("geometry_key"))
+ for item in items
+ )
+ for items in numbered_semantic_sets
+ )
+ for node in _walk(root):
+ if not isinstance(node, dict):
+ continue
+ for list_key in ["walls", "accepted_walls", "wall_visibility_review", "wall_visibility_semantics"]:
+ items = node.get(list_key)
+ if not isinstance(items, list):
+ continue
+ for item in items:
+ if not isinstance(item, dict):
+ continue
+ wall_id = item.get("wall_id") or item.get("id")
+ if wall_id not in WALL_IDS:
+ continue
+ item_priority = (
+ WALL_SEMANTIC_CONFLICT_RESOLUTION_PRIORITY
+ if (
+ item.get("semantic_visibility_geometry_conflict_resolved") is True
+ or (list_key == "wall_visibility_semantics" and has_complete_geometry_bound_semantics)
+ )
+ else source_priority
+ )
+ record = semantics.setdefault(str(wall_id), {"wall_id": str(wall_id), "source_lists": [], "_field_priorities": {}})
+ priorities = record["_field_priorities"]
+ geometry_key = item.get("geometry_key") or wall_geometry_key(item)
+ if geometry_key and item_priority >= int(priorities.get("geometry_key", -1)):
+ record["geometry_key"] = geometry_key
+ priorities["geometry_key"] = item_priority
+ if item.get("first_point") is not None and item.get("second_point") is not None and item_priority >= int(priorities.get("reported_geometry", -1)):
+ record["reported_geometry"] = {"first_point": item.get("first_point"), "second_point": item.get("second_point")}
+ priorities["reported_geometry"] = item_priority
+ if list_key not in record["source_lists"]:
+ record["source_lists"].append(list_key)
+ for key in semantic_keys:
+ if key not in item or item.get(key) is None:
+ continue
+ target_key = {
+ "reason": "semantic_reason",
+ "evidence_code": "reason_code",
+ "attach_eligible": "can_anchor_wall_attach",
+ }.get(key, key)
+ if item_priority >= int(priorities.get(target_key, -1)):
+ record[target_key] = item.get(key)
+ priorities[target_key] = item_priority
+ record["semantic_source_priority"] = max(int(record.get("semantic_source_priority", -1)), item_priority)
+ for record in semantics.values():
+ record.pop("_field_priorities", None)
+ return semantics
+
+def apply_wall_final_semantics(
+ wall_records: list[dict[str, Any]],
+ semantics: dict[str, dict[str, Any]],
+ wall_id_remap: dict[str, str] | None = None,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Overlay Agent semantics only when wall id and immutable geometry agree."""
+ wall_id_remap = wall_id_remap or {}
+ final_by_id = {
+ str(record.get("id") or record.get("wall_id")): record
+ for record in wall_records
+ if isinstance(record, dict) and record.get("accepted_as_final_wall")
+ }
+ remapped: dict[str, dict[str, Any]] = {}
+ by_geometry_key: dict[str, dict[str, Any]] = {}
+ rejected_records: list[dict[str, Any]] = []
+ for source_wall_id, metadata in semantics.items():
+ if not isinstance(metadata, dict):
+ continue
+ final_id = str(wall_id_remap.get(source_wall_id, source_wall_id))
+ final_record = final_by_id.get(final_id)
+ if final_id not in WALL_IDS or final_record is None:
+ rejected_records.append({
+ "reported_wall_id": source_wall_id,
+ "remapped_wall_id": final_id,
+ "geometry_key": metadata.get("geometry_key"),
+ "reason": "semantic_wall_id_not_in_final_walls",
+ })
+ continue
+ expected_key = wall_geometry_key(final_record)
+ reported_key = metadata.get("geometry_key")
+ if reported_key and expected_key and str(reported_key) != str(expected_key):
+ rejected_records.append({
+ "reported_wall_id": source_wall_id,
+ "remapped_wall_id": final_id,
+ "geometry_key": reported_key,
+ "expected_geometry_key": expected_key,
+ "reason": "wall_id_geometry_key_mismatch",
+ })
+ continue
+ remapped[final_id] = metadata
+ if expected_key:
+ by_geometry_key[str(expected_key)] = metadata
+
+ updated_records = []
+ applied_ids = []
+ for record in wall_records:
+ updated = dict(record)
+ wall_id = str(record.get("id") or record.get("wall_id") or "")
+ geometry_key = wall_geometry_key(record)
+ metadata = None
+ if record.get("accepted_as_final_wall"):
+ metadata = by_geometry_key.get(str(geometry_key)) if geometry_key else None
+ if metadata is None:
+ metadata = remapped.get(wall_id)
+ if geometry_key:
+ updated["geometry_key"] = geometry_key
+ if metadata:
+ for key in [
+ "original_image_visibility", "visible_role", "original_visible_wall_count",
+ "weak_wall", "closure_only", "can_anchor_wall_attach", "evidence_strength", "completion_method", "completion_reason",
+ ]:
+ if key in metadata:
+ updated[key] = metadata[key]
+ updated["agent_wall_semantics_applied"] = True
+ updated["wall_semantic_reason"] = metadata.get("semantic_reason")
+ updated["wall_semantic_source_lists"] = metadata.get("source_lists", [])
+ applied_ids.append(wall_id)
+ updated_records.append(updated)
+ return updated_records, {
+ "applied": bool(applied_ids),
+ "source": "prioritized Agent wall semantics with strict canonical id and immutable geometry binding",
+ "coordinates_accepted_from_agent": False,
+ "binding_policy": "geometry_key_and_remapped_wall_id_must_agree_when_both_are_available",
+ "applied_wall_ids": sorted(set(applied_ids)),
+ "missing_wall_ids": [wall_id for wall_id in WALL_IDS if wall_id not in set(applied_ids)],
+ "rejected_semantic_records": rejected_records,
+ }
+
+
+def summarize_candidate_wall_promotion(wall_records: list[dict[str, Any]]) -> dict[str, Any]:
+ accepted = [record for record in wall_records if record.get("accepted_as_final_wall")]
+ promoted = [record for record in accepted if record.get("promotion_from_candidate")]
+ modified = [record for record in accepted if record.get("candidate_modified")]
+ evidence_used = sorted({item for record in promoted + modified for item in record.get("verification_evidence_used", []) if item})
+ return {
+ "promotion_from_candidate": bool(promoted or modified),
+ "candidate_modified": bool(modified),
+ "revision_used": bool(modified),
+ "promoted_wall_ids": [record.get("id") for record in promoted or modified],
+ "modified_wall_ids": [record.get("id") for record in modified],
+ "candidate_original_sources": sorted({str(record.get("candidate_original_source")) for record in promoted + modified if record.get("candidate_original_source")}),
+ "normalized_source": "obb_from_mesh_footprint_min_rect" if (promoted or modified) else None,
+ "verification_evidence_used": evidence_used,
+ "revision_reasons": [record.get("revision_reason") for record in modified if record.get("revision_reason")],
+ "items": [
+ {
+ "wall_id": record.get("id"),
+ "promotion_from_candidate": bool(record.get("promotion_from_candidate") or record.get("candidate_modified")),
+ "candidate_modified": bool(record.get("candidate_modified")),
+ "revision_reason": record.get("revision_reason"),
+ "candidate_original_source": record.get("candidate_original_source"),
+ "normalized_source": record.get("normalized_source"),
+ "verification_evidence_used": record.get("verification_evidence_used", []),
+ "promotion_reason": record.get("promotion_reason"),
+ }
+ for record in promoted + [record for record in modified if record not in promoted]
+ ],
+ }
+
+
+def summarize_weak_wall_completion(wall_records: list[dict[str, Any]]) -> dict[str, Any]:
+ weak_records = [record for record in wall_records if record.get("accepted_as_final_wall") and record.get("weak_wall")]
+ return {
+ "used": bool(weak_records),
+ "weak_wall_ids": [record.get("id") or record.get("wall_id") for record in weak_records],
+ "completion_methods": sorted({str(record.get("completion_method")) for record in weak_records if record.get("completion_method")}),
+ "evidence_strengths": sorted({str(record.get("evidence_strength")) for record in weak_records if record.get("evidence_strength")}),
+ "completion_reasons": [record.get("completion_reason") for record in weak_records if record.get("completion_reason")],
+ "items": [
+ {
+ "wall_id": record.get("id") or record.get("wall_id"),
+ "evidence_strength": record.get("evidence_strength"),
+ "completion_method": record.get("completion_method"),
+ "completion_reason": record.get("completion_reason"),
+ "weak_wall": bool(record.get("weak_wall")),
+ "original_image_visibility": record.get("original_image_visibility"),
+ "visible_role": record.get("visible_role"),
+ }
+ for record in weak_records
+ ],
+ "rule": "Weak or hidden walls may be completed by room closure only when other walls, room footprint, and layout evidence support a closed boundary.",
+ }
+
+def weak_or_inferred_wall_attach_blocks(wall_records: list[dict[str, Any]]) -> dict[str, str]:
+ """Final walls may be weakly completed for room closure, but weak/hidden walls should not anchor attach relations."""
+ blocked: dict[str, str] = {}
+ for record in wall_records:
+ if not isinstance(record, dict) or not record.get("accepted_as_final_wall"):
+ continue
+ wall_id = record.get("id") or record.get("wall_id")
+ if not wall_id:
+ continue
+ evidence_strength = str(record.get("evidence_strength") or "").lower()
+ completion_method = str(record.get("completion_method") or "").lower()
+ completion_reason = str(record.get("completion_reason") or record.get("evidence") or "").lower()
+ visibility = str(record.get("original_image_visibility") or record.get("visibility") or "").lower()
+ weak_or_inferred = bool(
+ record.get("can_anchor_wall_attach") is False
+ or
+ record.get("weak_wall")
+ or record.get("closure_only")
+ or evidence_strength == "weak"
+ or completion_method == "inferred_by_room_closure"
+ or visibility in {"hidden", "not_visible", "not directly visible", "inferred", "occluded"}
+ or "hidden" in completion_reason
+ or "not directly visible" in completion_reason
+ )
+ if weak_or_inferred:
+ blocked[str(wall_id)] = (
+ "Weak/hidden/inferred/closure-only wall or agent-disallowed attach anchor is not allowed as an attach anchor; "
+ "attach requires a wall surface or wall-footline/contact visible in source evidence."
+ )
+ return blocked
+
+def shift_walls_from_attach_policy(
+ walls: list[dict[str, Any]],
+ wall_records: list[dict[str, Any]],
+ merge_report: dict[str, Any],
+ relations: list[dict[str, Any]],
+ offset_ratio: float = 0.05,
+ max_area_growth_ratio: float = 0.25,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]:
+ """Move only unattached virtual walls outward after final relation decisions."""
+ filtered = [
+ item for item in merge_report.get("skipped_candidates", [])
+ if isinstance(item, dict) and item.get("reason") == "weak_or_inferred_wall_anchor_not_allowed_for_attach"
+ ]
+ triggered_targets: dict[str, set[str]] = {}
+ for item in filtered:
+ relation = item.get("relation") if isinstance(item.get("relation"), dict) else {}
+ parameters = relation.get("parameters") if isinstance(relation.get("parameters"), dict) else {}
+ wall_id = parameters.get("object_anchor")
+ target = parameters.get("object_target")
+ if wall_id in WALL_IDS:
+ triggered_targets.setdefault(str(wall_id), set()).add(str(target or "unknown"))
+ attached_wall_ids: set[str] = set()
+ for relation in relations:
+ if not isinstance(relation, dict) or relation.get("relation") != "attach":
+ continue
+ parameters = relation.get("parameters") if isinstance(relation.get("parameters"), dict) else {}
+ anchor = parameters.get("object_anchor")
+ if anchor in WALL_IDS:
+ attached_wall_ids.add(str(anchor))
+ unattached_wall_ids = [wall_id for wall_id in WALL_IDS if wall_id not in attached_wall_ids]
+ virtual_wall_ids = set(weak_or_inferred_wall_attach_blocks(wall_records))
+ # Run only after the complete relation set is final. An empty visible wall
+ # is not evidence that its OBB boundary is too tight.
+ triggered_wall_ids = [
+ wall_id for wall_id in WALL_IDS
+ if wall_id in virtual_wall_ids and wall_id in unattached_wall_ids
+ ]
+ trigger_reasons = {
+ wall_id: [
+ reason
+ for condition, reason in [
+ (wall_id in unattached_wall_ids, "virtual_wall_without_retained_wall_attach"),
+ (wall_id in triggered_targets, "filtered_weak_wall_attach"),
+ ]
+ if condition
+ ]
+ for wall_id in triggered_wall_ids
+ }
+ base_audit = {
+ "schema_version": "wall_attach_policy_outward_shift",
+ "decision_source": "local_final_wall_attach_policy",
+ "offset_ratio": offset_ratio,
+ "attached_wall_ids": sorted(attached_wall_ids),
+ "unattached_wall_ids": unattached_wall_ids,
+ "virtual_wall_ids": sorted(virtual_wall_ids),
+ "triggered_wall_ids": triggered_wall_ids,
+ "trigger_reasons": trigger_reasons,
+ "triggered_object_targets": {key: sorted(value) for key, value in triggered_targets.items()},
+ "filtered_candidate_count": len(filtered),
+ }
+ if not triggered_wall_ids:
+ return walls, wall_records, {
+ **base_audit,
+ "status": "no_change",
+ "reason": "no_unattached_virtual_wall_requires_outward_shift",
+ }
+ by_id = {str(wall.get("id")): wall for wall in walls if isinstance(wall, dict)}
+ if set(by_id) != set(WALL_IDS):
+ return walls, wall_records, {**base_audit, "status": "rejected", "reason": "requires_four_canonical_walls"}
+ ordered = [by_id[wall_id] for wall_id in WALL_IDS]
+ points = [_safe_float_pair(wall.get("first_point")) for wall in ordered]
+ second_points = [_safe_float_pair(wall.get("second_point")) for wall in ordered]
+ if any(point is None for point in points + second_points):
+ return walls, wall_records, {**base_audit, "status": "rejected", "reason": "invalid_wall_endpoint"}
+ assert all(point is not None for point in points + second_points)
+ if any(math.dist(second_points[index], points[(index + 1) % 4]) > 1e-5 for index in range(4)):
+ return walls, wall_records, {**base_audit, "status": "rejected", "reason": "wall_segments_not_connected"}
+
+ def area(vertices: list[list[float]]) -> float:
+ return abs(sum(
+ vertices[index][0] * vertices[(index + 1) % 4][1]
+ - vertices[(index + 1) % 4][0] * vertices[index][1]
+ for index in range(4)
+ )) * 0.5
+
+ center = [sum(point[axis] for point in points) / 4.0 for axis in range(2)]
+ lines: list[dict[str, Any]] = []
+ for index, first in enumerate(points):
+ second = points[(index + 1) % 4]
+ dx, dz = second[0] - first[0], second[1] - first[1]
+ length = math.hypot(dx, dz)
+ if length <= 1e-12:
+ return walls, wall_records, {**base_audit, "status": "rejected", "reason": "degenerate_wall_segment"}
+ midpoint = [(first[0] + second[0]) * 0.5, (first[1] + second[1]) * 0.5]
+ away = [midpoint[0] - center[0], midpoint[1] - center[1]]
+ normals = [(-dz / length, dx / length), (dz / length, -dx / length)]
+ normal = max(normals, key=lambda value: value[0] * away[0] + value[1] * away[1])
+ lines.append({"normal": normal, "constant": normal[0] * first[0] + normal[1] * first[1], "length": length})
+ short_side = min(line["length"] for line in lines)
+ offset = short_side * offset_ratio
+ revised = [dict(line) for line in lines]
+ for index, wall_id in enumerate(WALL_IDS):
+ if wall_id in triggered_wall_ids:
+ revised[index]["constant"] += offset
+
+ def intersection(first: dict[str, Any], second: dict[str, Any]) -> list[float]:
+ a1, b1 = first["normal"]
+ a2, b2 = second["normal"]
+ determinant = a1 * b2 - a2 * b1
+ if abs(determinant) <= 1e-10:
+ raise ValueError("adjacent wall support lines are parallel")
+ c1, c2 = first["constant"], second["constant"]
+ return [(c1 * b2 - c2 * b1) / determinant, (a1 * c2 - a2 * c1) / determinant]
+
+ try:
+ after_points = [intersection(revised[index - 1], revised[index]) for index in range(4)]
+ except ValueError as exc:
+ return walls, wall_records, {**base_audit, "status": "rejected", "reason": str(exc)}
+ before_area, after_area = area(points), area(after_points)
+ growth = after_area / before_area - 1.0 if before_area > 1e-12 else float("inf")
+ original_contained = all(
+ all(line["normal"][0] * point[0] + line["normal"][1] * point[1] <= line["constant"] + 1e-7 for line in revised)
+ for point in points
+ )
+ errors = []
+ if after_area <= before_area + 1e-10:
+ errors.append("area_did_not_increase")
+ if not original_contained:
+ errors.append("original_obb_not_contained")
+ if growth > max_area_growth_ratio + 1e-9:
+ errors.append("area_growth_exceeds_limit")
+ if not all(math.isfinite(value) for point in after_points for value in point):
+ errors.append("non_finite_intersection")
+ if errors:
+ return walls, wall_records, {**base_audit, "status": "rejected", "validation_errors": errors}
+
+ revised_by_id: dict[str, dict[str, Any]] = {}
+ for index, wall_id in enumerate(WALL_IDS):
+ revised_wall = dict(by_id[wall_id])
+ revised_wall["first_point"] = [round(value, 6) for value in after_points[index]]
+ revised_wall["second_point"] = [round(value, 6) for value in after_points[(index + 1) % 4]]
+ revised_by_id[wall_id] = revised_wall
+ updated_walls = [revised_by_id.get(str(wall.get("id")), wall) for wall in walls]
+ updated_records = []
+ for record in wall_records:
+ updated = dict(record)
+ wall_id = str(record.get("id") or record.get("wall_id") or "")
+ if wall_id in revised_by_id:
+ updated["first_point"] = revised_by_id[wall_id]["first_point"]
+ updated["second_point"] = revised_by_id[wall_id]["second_point"]
+ updated["virtual_wall_shift"] = {
+ "triggered_by_filtered_attach": wall_id in triggered_targets,
+ "triggered_by_no_retained_attach": wall_id in virtual_wall_ids and wall_id in unattached_wall_ids,
+ "trigger_reasons": trigger_reasons.get(wall_id, []),
+ "support_line_offset_xz_units": round(offset if wall_id in triggered_wall_ids else 0.0, 6),
+ }
+ updated_records.append(updated)
+ return updated_walls, updated_records, {
+ **base_audit,
+ "status": "applied",
+ "shifted_support_wall_ids": triggered_wall_ids,
+ "offset_xz_units": round(offset, 6),
+ "before_area_xz": round(before_area, 8),
+ "after_area_xz": round(after_area, 8),
+ "area_growth_ratio": round(growth, 8),
+ "original_obb_contained": original_contained,
+ "policy": "After all relation decisions are finalized, only canonical weak/inferred/closure-only walls without a retained wall attach move outward once. Visible walls remain fixed even when no object attaches to them.",
+ }
+
+
+def _safe_float_pair(value: Any) -> list[float] | None:
+ if not isinstance(value, list) or len(value) != 2:
+ return None
+ try:
+ return [float(value[0]), float(value[1])]
+ except Exception:
+ return None
+
+
+def _load_xz_to_pixel_affine_from_wall_evidence(wall_evidence: dict[str, Any] | None) -> list[list[float]] | None:
+ if not isinstance(wall_evidence, dict):
+ return None
+ candidates: list[Any] = []
+ precompute = wall_evidence.get("render_grid_wall_candidates")
+ if isinstance(precompute, dict):
+ candidates.append(precompute.get("render_grid_affine_path"))
+ evidence = wall_evidence.get("topdown_verification")
+ if isinstance(evidence, dict):
+ candidates.append(evidence.get("render_grid_affine_path"))
+ topdown = wall_evidence.get("topdown")
+ if isinstance(topdown, dict):
+ candidates.append(topdown.get("render_grid_affine_path"))
+ for raw in candidates:
+ if not raw:
+ continue
+ path = Path(str(raw))
+ if not path.exists():
+ continue
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except Exception:
+ continue
+ affine = data.get("xz_to_pixel_affine")
+ if isinstance(affine, list) and len(affine) == 3:
+ return affine
+ return None
+
+
+def _xz_to_screen_xy(point: list[float], affine: list[list[float]] | None) -> list[float]:
+ x, z = float(point[0]), float(point[1])
+ if affine:
+ return [
+ float(affine[0][0]) * x + float(affine[1][0]) * z + float(affine[2][0]),
+ float(affine[0][1]) * x + float(affine[1][1]) * z + float(affine[2][1]),
+ ]
+ # PNG/screen coordinates grow downward, so use -z as a conservative fallback.
+ return [x, -z]
+
+
+def _signed_area_xy(points: list[list[float]]) -> float:
+ total = 0.0
+ for idx, point in enumerate(points):
+ nxt = points[(idx + 1) % len(points)]
+ total += point[0] * nxt[1] - nxt[0] * point[1]
+ return total / 2.0
+
+
+def _segment_midpoint_xy(first: list[float], second: list[float], affine: list[list[float]] | None) -> list[float]:
+ a = _xz_to_screen_xy(first, affine)
+ b = _xz_to_screen_xy(second, affine)
+ return [(a[0] + b[0]) / 2.0, (a[1] + b[1]) / 2.0]
+
+
+def _point_close(a: list[float], b: list[float], eps: float = 1e-6) -> bool:
+ return abs(float(a[0]) - float(b[0])) <= eps and abs(float(a[1]) - float(b[1])) <= eps
+
+
+def _match_old_wall_id(first: list[float], second: list[float], original_walls: list[dict[str, Any]]) -> str | None:
+ for wall in original_walls:
+ old_first = _safe_float_pair(wall.get("first_point"))
+ old_second = _safe_float_pair(wall.get("second_point"))
+ if not old_first or not old_second:
+ continue
+ if (_point_close(first, old_first) and _point_close(second, old_second)) or (_point_close(first, old_second) and _point_close(second, old_first)):
+ return str(wall.get("id"))
+ return None
+
+
+def _record_by_old_wall_id(wall_records: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
+ result: dict[str, dict[str, Any]] = {}
+ for record in wall_records:
+ if isinstance(record, dict) and record.get("accepted_as_final_wall") and record.get("id"):
+ result[str(record["id"])] = record
+ return result
+
+
+def _wall_record_aliases(record: dict[str, Any]) -> list[str]:
+ aliases: list[str] = []
+
+ def add(value: Any) -> None:
+ if value is None:
+ return
+ if isinstance(value, str):
+ value_s = value.strip()
+ if value_s and value_s not in aliases:
+ aliases.append(value_s)
+ elif isinstance(value, (int, float)):
+ value_s = str(value)
+ if value_s not in aliases:
+ aliases.append(value_s)
+
+ for key in [
+ "candidate_or_geometry_wall_id",
+ "candidate_wall_id",
+ "geometry_wall_id",
+ "provisional_wall_id",
+ "original_wall_id_before_canonicalization",
+ "id",
+ "wall_id",
+ ]:
+ add(record.get(key))
+ formal = record.get("formal_obb_segment")
+ if isinstance(formal, dict):
+ for key in ["geometry_wall_id", "provisional_wall_id"]:
+ add(formal.get(key))
+ return aliases
+
+
+def _wall_geometry_aliases(record: dict[str, Any]) -> list[str]:
+ aliases: list[str] = []
+
+ def add(value: Any) -> None:
+ if value is None:
+ return
+ value_s = str(value).strip()
+ if value_s and value_s not in aliases:
+ aliases.append(value_s)
+
+ for key in [
+ "candidate_or_geometry_wall_id",
+ "candidate_wall_id",
+ "geometry_wall_id",
+ "provisional_wall_id",
+ ]:
+ add(record.get(key))
+ formal = record.get("formal_obb_segment")
+ if isinstance(formal, dict):
+ for key in ["geometry_wall_id", "provisional_wall_id"]:
+ add(formal.get(key))
+ return aliases
+
+
+def _record_by_wall_geometry_alias(wall_records: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
+ result: dict[str, dict[str, Any]] = {}
+ for record in wall_records:
+ if not isinstance(record, dict) or not record.get("accepted_as_final_wall"):
+ continue
+ for alias in _wall_geometry_aliases(record):
+ result.setdefault(alias, record)
+ return result
+
+
+def build_wall_relation_id_remap(wall_records: list[dict[str, Any]], wall_canonicalization_gate: dict[str, Any] | None = None) -> dict[str, str]:
+ remap: dict[str, str] = {}
+ gate_map = wall_canonicalization_gate.get("old_to_new_wall_id") if isinstance(wall_canonicalization_gate, dict) else None
+ if isinstance(gate_map, dict):
+ for old_id, new_id in gate_map.items():
+ if old_id and new_id:
+ remap[str(old_id)] = str(new_id)
+ for record in wall_records:
+ if not isinstance(record, dict) or not record.get("accepted_as_final_wall"):
+ continue
+ final_id = record.get("id") or record.get("wall_id")
+ if final_id not in WALL_IDS:
+ continue
+ for alias in _wall_record_aliases(record):
+ if alias in WALL_IDS:
+ remap.setdefault(alias, str(final_id))
+ else:
+ remap[alias] = str(final_id)
+ return remap
+
+
+def _visible_count_from_records(records_by_id: dict[str, dict[str, Any]]) -> int | None:
+ counts: list[int] = []
+ for record in records_by_id.values():
+ raw = record.get("original_visible_wall_count")
+ try:
+ counts.append(int(raw))
+ except Exception:
+ continue
+ if not counts:
+ return None
+ return max(set(counts), key=counts.count)
+
+
+def _formal_obb_primary_candidate(wall_evidence: dict[str, Any] | None) -> dict[str, Any] | None:
+ if not isinstance(wall_evidence, dict):
+ return None
+ for key in ("render_grid_wall_candidates",):
+ source = wall_evidence.get(key)
+ if not isinstance(source, dict):
+ continue
+ primary = source.get("primary_candidate")
+ if isinstance(primary, dict) and primary.get("method") == FORMAL_OBB_WALL_METHOD:
+ return primary
+ return None
+
+
+def _authoritative_obb_walls_from_evidence(
+ walls: list[dict[str, Any]],
+ wall_records: list[dict[str, Any]],
+ wall_evidence: dict[str, Any] | None,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]:
+ """Replace final wall coordinates with the formal OBB candidate.
+
+ Agents may still adjudicate visibility and weak/hidden wall metadata, but
+ the coordinate source for final wall_001..wall_004 is fixed locally to
+ ``obb_from_mesh_footprint_min_rect``. This prevents old layout/mask/LSD
+ wall coordinates from re-entering final objects.json.
+ """
+ primary = _formal_obb_primary_candidate(wall_evidence)
+ if not primary:
+ return walls, wall_records, {
+ "applied": False,
+ "required_method": FORMAL_OBB_WALL_METHOD,
+ "reason": "formal_obb_primary_candidate_missing",
+ }
+ selected_orientation_candidate_id = primary.get("candidate_id") or "formal_obb_primary"
+ segments = primary.get("wall_segments")
+ if not isinstance(segments, list) or len(segments) < 4:
+ return walls, wall_records, {
+ "applied": False,
+ "required_method": FORMAL_OBB_WALL_METHOD,
+ "reason": "formal_obb_primary_candidate_has_no_four_segments",
+ }
+ normalized_segments: list[dict[str, Any]] = []
+ for idx, segment in enumerate(segments[:4]):
+ if not isinstance(segment, dict):
+ continue
+ first = _safe_float_pair(segment.get("first_point"))
+ second = _safe_float_pair(segment.get("second_point"))
+ if not first or not second:
+ continue
+ normalized_segments.append(
+ {
+ "id": segment.get("id") if segment.get("id") in WALL_IDS else WALL_IDS[idx],
+ "first_point": [round(float(first[0]), 6), round(float(first[1]), 6)],
+ "second_point": [round(float(second[0]), 6), round(float(second[1]), 6)],
+ "segment": segment,
+ }
+ )
+ if len(normalized_segments) != 4:
+ return walls, wall_records, {
+ "applied": False,
+ "required_method": FORMAL_OBB_WALL_METHOD,
+ "reason": "formal_obb_primary_candidate_segments_invalid",
+ "valid_segment_count": len(normalized_segments),
+ }
+
+ # When formal OBB is authoritative, inherit only geometry/candidate metadata
+ # from the agent output. Agent final wall ids are unstable across serial and
+ # skeleton-async runs, so matching by id/wall_id can copy weak/visibility
+ # metadata onto the wrong OBB segment.
+ previous_records_by_alias = _record_by_wall_geometry_alias(wall_records)
+ authoritative_source_path = "wall_evidence.render_grid_wall_candidates.primary_candidate.wall_segments"
+
+ new_walls: list[dict[str, Any]] = []
+ new_records: list[dict[str, Any]] = []
+ matched_aliases: dict[str, str | None] = {}
+ for idx, item in enumerate(normalized_segments):
+ wall_id = item["id"]
+ segment = item["segment"]
+ segment_aliases = [
+ segment.get("geometry_wall_id"),
+ segment.get("provisional_wall_id"),
+ segment.get("id"),
+ wall_id,
+ ]
+ old_record: dict[str, Any] = {}
+ matched_alias: str | None = None
+ for alias in segment_aliases:
+ if alias and str(alias) in previous_records_by_alias:
+ old_record = previous_records_by_alias[str(alias)]
+ matched_alias = str(alias)
+ break
+ matched_aliases[wall_id] = matched_alias
+ new_walls.append({"id": wall_id, "first_point": item["first_point"], "second_point": item["second_point"]})
+ record = dict(old_record) if isinstance(old_record, dict) else {}
+ previous_evidence_used = record.get("verification_evidence_used")
+ if isinstance(previous_evidence_used, list):
+ verification_evidence_used = [str(item) for item in previous_evidence_used if item]
+ elif previous_evidence_used:
+ verification_evidence_used = [str(previous_evidence_used)]
+ else:
+ verification_evidence_used = []
+ verification_evidence_used.extend([
+ "render_grid_wall_candidates.primary_candidate",
+ FORMAL_OBB_WALL_METHOD,
+ f"wall_orientation_candidate:{selected_orientation_candidate_id}",
+ ])
+ record.update(
+ {
+ "id": wall_id,
+ "wall_id": wall_id,
+ "first_point": item["first_point"],
+ "second_point": item["second_point"],
+ "source": FORMAL_OBB_WALL_SOURCE,
+ "original_source": FORMAL_OBB_WALL_SOURCE,
+ "normalized_source": FORMAL_OBB_WALL_SOURCE,
+ "topdown_verification": "local_formal_obb_gate",
+ "evidence": "Final wall coordinates are locally overridden from render_grid_wall_candidates.primary_candidate using obb_from_mesh_footprint_min_rect.",
+ "accepted": True,
+ "accepted_as_final_wall": True,
+ "reason": "formal OBB wall coordinate source is authoritative for final wall geometry",
+ "candidate_original_source": FORMAL_OBB_WALL_SOURCE,
+ "verification_evidence_used": sorted(set(verification_evidence_used)),
+ "formal_wall_orientation_candidate_id": selected_orientation_candidate_id,
+ "formal_wall_orientation": primary.get("orientation", {}),
+ "formal_obb_segment": {
+ "provisional_wall_id": segment.get("provisional_wall_id"),
+ "geometry_wall_id": segment.get("geometry_wall_id"),
+ "clockwise_order_index": segment.get("clockwise_order_index"),
+ "source_method": segment.get("source_method"),
+ "matched_agent_wall_alias": matched_alias,
+ "matched_agent_wall_id": old_record.get("id") if isinstance(old_record, dict) else None,
+ },
+ }
+ )
+ record.setdefault("canonical_naming", segment.get("canonical_naming") or {})
+ new_records.append(record)
+
+ return new_walls, new_records, {
+ "applied": True,
+ "method": FORMAL_OBB_WALL_METHOD,
+ "selected_orientation_candidate_id": selected_orientation_candidate_id,
+ "selected_orientation": primary.get("orientation", {}),
+ "coordinate_frame": "OpenGL original XZ; Y is vertical",
+ "source": authoritative_source_path,
+ "previous_wall_count": len(walls),
+ "previous_accepted_record_count": len([r for r in wall_records if isinstance(r, dict) and r.get("accepted_as_final_wall")]),
+ "new_wall_count": len(new_walls),
+ "matched_agent_record_aliases": matched_aliases,
+ "policy": "local OBB coordinate authority before screen-clockwise wall-id canonicalization",
+ "note": "Agent wall coordinates are never used. Local code selects one finite orientation using Agent-identified wall-following objects; final coordinates come from that selected candidate and only semantic metadata is preserved.",
+ }
+
+
+def canonicalize_walls_for_screen_clockwise(
+ walls: list[dict[str, Any]],
+ wall_records: list[dict[str, Any]],
+ wall_evidence: dict[str, Any] | None,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]:
+ """Make final wall ids follow the human top-down clockwise convention.
+
+ The model may output a geometrically valid wall_001..wall_004 set that is
+ clockwise in OpenGL XZ but visually counter-clockwise in the rendered PNG.
+ This gate keeps the coordinates, but renumbers/reverses the final wall
+ records so downstream attach relations use the canonical human-facing wall
+ ids. Candidate/provisional ids are allowed to be wrong here; final ids are
+ not.
+ """
+ if len(walls) != 4 or any(wall.get("id") not in WALL_IDS for wall in walls):
+ return walls, wall_records, {"applied": False, "reason": "requires_complete_wall_001_to_wall_004"}
+ original_walls = [dict(wall) for wall in sorted(walls, key=lambda wall: WALL_IDS.index(wall["id"]))]
+ vertices = []
+ for wall in original_walls:
+ point = _safe_float_pair(wall.get("first_point"))
+ if not point:
+ return walls, wall_records, {"applied": False, "reason": "invalid_wall_first_point"}
+ vertices.append(point)
+ affine = _load_xz_to_pixel_affine_from_wall_evidence(wall_evidence)
+ screen_vertices = [_xz_to_screen_xy(point, affine) for point in vertices]
+ screen_area_before = _signed_area_xy(screen_vertices)
+ # In image/screen coordinates y grows downward; positive shoelace area is
+ # the human-visible clockwise order.
+ if screen_area_before < 0:
+ vertices = list(reversed(vertices))
+ screen_vertices = [_xz_to_screen_xy(point, affine) for point in vertices]
+ screen_area_after = _signed_area_xy(screen_vertices)
+ segments: list[dict[str, Any]] = []
+ for idx, first in enumerate(vertices):
+ second = vertices[(idx + 1) % len(vertices)]
+ old_id = _match_old_wall_id(first, second, original_walls)
+ segments.append(
+ {
+ "old_wall_id": old_id,
+ "first_point": [round(float(first[0]), 6), round(float(first[1]), 6)],
+ "second_point": [round(float(second[0]), 6), round(float(second[1]), 6)],
+ "midpoint_screen_xy": _segment_midpoint_xy(first, second, affine),
+ }
+ )
+ records_by_id = _record_by_old_wall_id(wall_records)
+ visible_count = _visible_count_from_records(records_by_id)
+ # Keep wall ids deterministic across serial/async runs. Earlier versions
+ # rotated the screen-clockwise segment list according to agent-provided
+ # visible_role metadata. That made wall ids vary when two API calls described
+ # the same OBB edges differently. We now preserve the local OBB/screen order
+ # and use agent visibility only as metadata, not as the numbering anchor.
+ start_idx = 0
+ ordered_segments = segments
+ old_to_new: dict[str, str] = {}
+ new_walls: list[dict[str, Any]] = []
+ for idx, segment in enumerate(ordered_segments):
+ new_id = WALL_IDS[idx]
+ old_id = segment.get("old_wall_id")
+ if old_id:
+ old_to_new[str(old_id)] = new_id
+ new_walls.append({"id": new_id, "first_point": segment["first_point"], "second_point": segment["second_point"]})
+ # Fill any unmatched old ids conservatively to keep remap total for audits.
+ for old_id in WALL_IDS:
+ old_to_new.setdefault(old_id, old_id)
+ by_new_id = {wall["id"]: wall for wall in new_walls}
+ new_records: list[dict[str, Any]] = []
+ for record in wall_records:
+ if not isinstance(record, dict) or not record.get("id"):
+ new_records.append(record)
+ continue
+ old_id = str(record.get("id"))
+ new_record = dict(record)
+ new_id = old_to_new.get(old_id, old_id)
+ if old_id in WALL_IDS:
+ new_record["original_wall_id_before_canonicalization"] = old_id
+ new_record["id"] = new_id
+ new_record["wall_id"] = new_id
+ if new_id in by_new_id:
+ new_record["first_point"] = by_new_id[new_id]["first_point"]
+ new_record["second_point"] = by_new_id[new_id]["second_point"]
+ new_records.append(new_record)
+ return new_walls, new_records, {
+ "applied": True,
+ "policy": "screen_clockwise_wall_ids_deterministic_obb_order",
+ "screen_coordinate_source": "render_grid_affine.xz_to_pixel_affine" if affine else "fallback_x_neg_z",
+ "screen_signed_area_before": screen_area_before,
+ "screen_signed_area_after": screen_area_after,
+ "screen_order_interpretation": "clockwise" if screen_area_after > 0 else "counter_clockwise_or_degenerate",
+ "original_visible_wall_count_used": visible_count,
+ "start_segment_old_wall_id": ordered_segments[0].get("old_wall_id") if ordered_segments else None,
+ "start_segment_policy": "deterministic_first_screen_clockwise_segment_no_agent_visible_role_rotation",
+ "old_to_new_wall_id": old_to_new,
+ "changed_wall_ids": {old: new for old, new in old_to_new.items() if old != new},
+ "note": "Final wall ids are canonicalized for the rendered top-down image, not for raw XZ signed-area order.",
+ }
+
+
+
+def write_wall_audit(
+ output_dir: Path,
+ walls: list[dict[str, Any]],
+ wall_records: list[dict[str, Any]],
+ warnings: list[str],
+ dry_run: bool,
+ wall_final_adjudication: dict[str, Any] | None = None,
+ wall_canonicalization_gate: dict[str, Any] | None = None,
+ authoritative_wall_geometry_gate: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+ geometry = validate_wall_geometry(walls) if walls else {"valid": False, "errors": ["No complete final-eligible wall set available."], "warnings": warnings, "details": {}}
+ accepted_wall_records = [record for record in wall_records if record.get("accepted_as_final_wall")]
+ naming_audit_source = accepted_wall_records if accepted_wall_records else walls
+ data = {
+ "dry_run": dry_run,
+ "status": "completed" if geometry.get("valid") else "failed",
+ "walls": walls,
+ "wall_source_records": wall_records,
+ "coordinate_source_definition": "final wall coordinates are layout XZ coordinates; topdown/grid/fixed views are verification and auxiliary evidence",
+ "orientation_rule": "Final walls may be axis-aligned or oblique; final assembly requires final-eligible source, closure, adjacent endpoint consistency, and clockwise geometry, not x-constant/z-constant edges.",
+ "wall_canonical_naming_policy": canonical_wall_naming_policy(),
+ "wall_canonical_naming_audit": final_wall_naming_audit(naming_audit_source),
+ "allowed_final_wall_sources": sorted(ALLOWED_FINAL_WALL_SOURCES),
+ "intermediate_wall_sources": sorted(INTERMEDIATE_WALL_SOURCES),
+ "hint_only_sources": sorted(HINT_ONLY_WALL_SOURCES),
+ "geometry": geometry,
+ "warnings": warnings,
+ "wall_final_adjudication": wall_final_adjudication or {"executed": False, "decision": None, "accepted_final_walls": bool(walls and len(walls) == 4)},
+ "authoritative_wall_geometry_gate": authoritative_wall_geometry_gate or {"applied": False},
+ "wall_canonicalization_gate": wall_canonicalization_gate or {"applied": False},
+ "candidate_wall_promotion": summarize_candidate_wall_promotion(wall_records),
+ "weak_wall_completion": summarize_weak_wall_completion(wall_records),
+ }
+ write_audit_pair(output_dir, "walls", "wall_audit", data, "Wall Audit")
+ return geometry
diff --git a/vlm_agent/reconstruction/relations/agents/final_assembler.py b/vlm_agent/reconstruction/relations/agents/final_assembler.py
new file mode 100644
index 0000000..6985b1b
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/final_assembler.py
@@ -0,0 +1,3226 @@
+from __future__ import annotations
+
+import json
+import math
+import re
+from pathlib import Path
+from typing import Any
+
+from relations.agents.schema_guard import WALL_IDS
+from relations.agents.assembly.containment import (
+ CONTAIN_CONFLICT_MIN_LOCAL_CONFIDENCE,
+ CONTAIN_INTERNAL_SURFACE_ROLE,
+ CONTAIN_EXTERNAL_SURFACE_ROLES,
+ _target_coverage,
+ _contain_geometry_index,
+ _contain_point_xz,
+ _contain_center_y,
+ _contain_y_bounds,
+ _contain_hull_xz,
+ _contain_rotate_xz,
+ _contain_min_area_angle,
+ _axis_outside_distance,
+ _contain_geometry_decision,
+ filter_contain_relations_by_geometry,
+ _normalized_contain_surface_role,
+ _contain_conflict_id,
+ _contain_candidate_pairs_from_scene_state,
+ _positive_containment_hypothesis,
+ _contain_rejections_from_task_results,
+ collect_multi_level_support_contain_promotions,
+ _contain_uncertainties_from_task_results,
+ _accepted_object_relation_hypotheses,
+ _contain_conflict_object_metadata,
+ collect_contain_conflicts,
+ compact_contain_conflicts_for_guard,
+ collect_contain_conflict_resolutions,
+ resolve_attach_contain_conflicts,
+ filter_redundant_floor_attaches_for_object_supported_targets,
+ normalize_floor_covering_relations,
+)
+from relations.agents.assembly.relation_candidates import (
+ _candidate_decision,
+ _relation_from_candidate,
+ _repair_inconsistent_support_candidate,
+ _relation_from_accepted_support_candidate,
+ collect_accepted_attach_candidates,
+ collect_coverage_repair_candidates,
+)
+from relations.agents.assembly.wall_assembly import (
+ FORMAL_OBB_WALL_METHOD,
+ FORMAL_OBB_WALL_SOURCE,
+ ALLOWED_FINAL_WALL_SOURCES,
+ INTERMEDIATE_WALL_SOURCES,
+ HINT_ONLY_WALL_SOURCES,
+ DEFAULT_WALL_PROMOTION_EVIDENCE,
+ WALL_SEMANTIC_CONFLICT_RESOLUTION_PRIORITY,
+ _wall_source_from_context,
+ _wall_verification_status,
+ _truthy,
+ _listify_strings,
+ _candidate_promotion_audit,
+ _normalize_wall_source,
+ _wall_source_allowed,
+ _normalize_wall_set,
+ _is_wall_final_adjudication_node,
+ _wall_final_adjudication_decision,
+ summarize_wall_final_adjudication,
+ collect_walls,
+ _wall_semantic_source_priority,
+ collect_wall_final_semantics,
+ apply_wall_final_semantics,
+ summarize_candidate_wall_promotion,
+ summarize_weak_wall_completion,
+ weak_or_inferred_wall_attach_blocks,
+ shift_walls_from_attach_policy,
+ _safe_float_pair,
+ _load_xz_to_pixel_affine_from_wall_evidence,
+ _xz_to_screen_xy,
+ _signed_area_xy,
+ _segment_midpoint_xy,
+ _point_close,
+ _match_old_wall_id,
+ _record_by_old_wall_id,
+ _wall_record_aliases,
+ _wall_geometry_aliases,
+ _record_by_wall_geometry_alias,
+ build_wall_relation_id_remap,
+ _visible_count_from_records,
+ _formal_obb_primary_candidate,
+ _authoritative_obb_walls_from_evidence,
+ canonicalize_walls_for_screen_clockwise,
+ write_wall_audit,
+)
+from relations.agents.assembly.object_properties import (
+ ROOM_MATERIAL_KEYS,
+ _OBJECT_REVISION_CONFIDENCE_SCORES,
+ _OBJECT_REVISION_SOURCE_PRIORITIES,
+ FLOOR_COVERING_TEXT_HINTS,
+ UPRIGHT_FALSE_TEXT_HINTS,
+ UPRIGHT_IMAGE_AXIS_TEXT_HINTS,
+ UPRIGHT_TRUE_TEXT_HINTS,
+ CONTAIN_UP_DIRECTION_HINTS,
+ CONTAIN_FRONT_DIRECTION_HINTS,
+ _snake_case,
+ candidate_objects,
+ _walk,
+ _walk_with_path,
+ normalize_room_materials,
+ collect_room_materials,
+ _normalize_object_revision_confidence,
+ _resolve_object_revision_patches,
+ collect_object_revision_patches,
+ apply_object_revision_patches,
+ write_object_revision_audit,
+ _bool_or_none,
+ _scene_object_context,
+ _object_text,
+ _bbox_extent_xyz,
+ _bbox_2d_height_width_ratio,
+ _local_floor_covering_hint,
+ _matches_semantic_hint,
+ _local_upright_semantic_hint,
+ _local_upright_fallback,
+ collect_object_physical_properties,
+ write_object_physical_properties_audit,
+ load_async_sidecar_upright_priors,
+ apply_upright_properties,
+ _normalize_contain_render_direction,
+ _semantic_contain_render_direction,
+ _contain_candidate_anchor_directions,
+ _agent_contain_anchor_directions,
+ apply_contain_anchor_properties,
+ _is_ceiling_light_object,
+)
+from relations.agents.state import sorted_object_id_key, write_audit_pair, write_json, write_md
+from relations.object_physics.core import resolve_object_physics, validate_object_physics
+
+WALL_MOUNTED_KEEP_KEYWORDS = {
+ "attached to wall",
+ "embedded",
+ "mounted",
+ "wall mounted",
+ "wall-mounted",
+ "wall-hung",
+}
+
+WALL_MOUNTED_CATEGORIES = {
+ "acoustic_panel",
+ "art_panel",
+ "bulletin_board",
+ "painting",
+ "projection_screen",
+ "wall_art",
+ "wall_panel",
+ "whiteboard",
+}
+
+
+
+
+
+
+
+
+
+
+def _object_context_by_id(objects: list[dict[str, Any]] | None) -> dict[str, dict[str, Any]]:
+ result: dict[str, dict[str, Any]] = {}
+ for obj in objects or []:
+ if isinstance(obj, dict) and obj.get("id"):
+ result[str(obj["id"])] = obj
+ return result
+
+
+def _alias_key(value: Any) -> str:
+ return re.sub(r"[^0-9a-z]+", "_", str(value or "").lower()).strip("_")
+
+
+def _object_anchor_alias_map(objects: list[dict[str, Any]] | None) -> dict[str, str]:
+ alias_to_ids: dict[str, set[str]] = {}
+
+ def add(alias: Any, object_id: str) -> None:
+ key = _alias_key(alias)
+ if not key or key.startswith("object_") or key in {"floor", "ceiling", *WALL_IDS}:
+ return
+ alias_to_ids.setdefault(key, set()).add(object_id)
+
+ for obj in objects or []:
+ if not isinstance(obj, dict) or not obj.get("id"):
+ continue
+ object_id = str(obj["id"])
+ category = str(obj.get("category") or "")
+ description = str(obj.get("description") or "")
+ add(category, object_id)
+ add(category.replace("_", " "), object_id)
+ parts = [part for part in _alias_key(category).split("_") if part]
+ if len(parts) > 1:
+ add(parts[-1], object_id)
+ for phrase in ["coffee table", "headphone stand", "built in cabinet", "built-in cabinet", "office chair", "mini fridge"]:
+ if phrase in description.lower() or phrase.replace(" ", "_") == _alias_key(category):
+ add(phrase, object_id)
+ return {alias: next(iter(ids)) for alias, ids in alias_to_ids.items() if len(ids) == 1}
+
+
+
+
+def _ceiling_light_object_ids(objects: list[dict[str, Any]] | None) -> list[str]:
+ return [str(obj["id"]) for obj in objects or [] if isinstance(obj, dict) and obj.get("id") and _is_ceiling_light_object(obj)]
+
+
+def _ceiling_confirmed_from_objects(objects: list[dict[str, Any]] | None) -> bool:
+ return bool(_ceiling_light_object_ids(objects))
+
+
+EXPLICIT_AGENT_WALL_ROLES = {"wall_mounted", "wall_embedded", "mounted", "embedded", "hung"}
+FLOOR_AGENT_WALL_ROLES = EXPLICIT_AGENT_WALL_ROLES | {"against_wall"}
+
+
+def _collect_high_confidence_wall_roles(
+ task_results: list[dict[str, Any]],
+ allowed_roles: set[str],
+) -> dict[str, dict[str, Any]]:
+ result: dict[str, dict[str, Any]] = {}
+ for root in task_results:
+ for source_path, node in _walk_with_path(root, str(root.get("task_id") or "task") if isinstance(root, dict) else "task"):
+ if not isinstance(node, dict):
+ continue
+ object_id = node.get("object_id")
+ role = str(node.get("role") or "").strip().lower()
+ confidence = str(node.get("confidence") or "").strip().lower()
+ if (
+ isinstance(object_id, str)
+ and role in allowed_roles
+ and confidence in {"high", "very_high", "very high"}
+ ):
+ result[object_id] = {
+ "object_id": object_id,
+ "role": role,
+ "confidence": confidence,
+ "source_path": source_path,
+ }
+ return result
+
+
+def collect_high_confidence_explicit_wall_roles(
+ task_results: list[dict[str, Any]],
+) -> dict[str, dict[str, Any]]:
+ """Collect roles that justify wall constraint independently of floor or object support."""
+ return _collect_high_confidence_wall_roles(task_results, EXPLICIT_AGENT_WALL_ROLES)
+
+
+def collect_high_confidence_floor_wall_roles(
+ task_results: list[dict[str, Any]],
+) -> dict[str, dict[str, Any]]:
+ """Include high-confidence against-wall evidence only for the floor+wall filter."""
+ return _collect_high_confidence_wall_roles(task_results, FLOOR_AGENT_WALL_ROLES)
+
+
+SEMANTIC_WALL_CONTACT_MARKERS = (
+ "built_in",
+ "embedded",
+ "fastened",
+ "fixed_to_wall",
+ "flush",
+ "installed",
+ "integrated",
+ "leaning",
+ "mounted",
+ "rests_against_wall",
+ "touching_wall",
+ "visible_back_contact",
+ "visible_rear_contact",
+ "visible_wall_contact",
+ "wall_contact",
+ "wall_hung",
+ "wall_mounted",
+)
+SEMANTIC_WALL_SEPARATION_MARKERS = (
+ "clear_gap",
+ "gap_from_wall",
+ "no_contact",
+ "not_in_contact",
+ "not_touching",
+ "separate_from_wall",
+ "visible_gap",
+)
+SEMANTIC_WALL_PLACEMENT_ONLY_MARKERS = (
+ "near_only",
+ "position_only",
+ "positional_only",
+)
+
+
+def _semantic_wall_text(node: dict[str, Any]) -> str:
+ values: list[str] = []
+ for key in (
+ "evidence_code",
+ "reason_code",
+ "reason",
+ "rationale",
+ "wall_contact_type",
+ "geometry_classification",
+ ):
+ if node.get(key) is not None:
+ values.append(str(node.get(key)))
+ for key in ("evidence_codes", "evidence"):
+ value = node.get(key)
+ if isinstance(value, list):
+ values.extend(str(item) for item in value if item is not None)
+ elif value is not None:
+ values.append(str(value))
+ return _alias_key(" ".join(values))
+
+
+def _semantic_wall_signal_flags(text: str) -> dict[str, bool]:
+ explicit_separation = any(
+ marker in text for marker in SEMANTIC_WALL_SEPARATION_MARKERS
+ )
+ explicit_contact = (
+ not explicit_separation
+ and any(marker in text for marker in SEMANTIC_WALL_CONTACT_MARKERS)
+ )
+ placement_only = any(
+ marker in text for marker in SEMANTIC_WALL_PLACEMENT_ONLY_MARKERS
+ )
+ return {
+ "explicit_contact": explicit_contact,
+ "explicit_separation": explicit_separation,
+ "placement_only": placement_only,
+ }
+
+
+def _selected_wall_orientation_candidate_id(
+ wall_evidence: dict[str, Any] | None,
+) -> str | None:
+ if not isinstance(wall_evidence, dict):
+ return None
+ containers = [wall_evidence]
+ render_grid = wall_evidence.get("render_grid_wall_candidates")
+ if isinstance(render_grid, dict):
+ containers.append(render_grid)
+ for container in containers:
+ value = (
+ container.get("selected_orientation_candidate_id")
+ or container.get("selected_candidate_id")
+ )
+ if isinstance(value, str) and value:
+ return value
+ return None
+
+
+def collect_semantic_wall_bindings(
+ task_results: list[dict[str, Any]],
+ wall_evidence: dict[str, Any] | None,
+ wall_id_remap: dict[str, str] | None = None,
+) -> dict[str, Any]:
+ """Resolve one medium-or-higher semantic wall binding per target.
+
+ Only the already-selected orientation candidate is used when that id is
+ available. Without a selected id, every qualifying binding must agree on
+ one remapped wall. High confidence is still required for relation creation;
+ medium confidence is retained only to protect an already-accepted pair.
+ """
+ wall_id_remap = wall_id_remap or {}
+ selected_candidate_id = _selected_wall_orientation_candidate_id(wall_evidence)
+ confidence_rank = {
+ "medium": 1,
+ "high": 2,
+ "very_high": 3,
+ "very high": 3,
+ }
+ role_rank = {role: 2 for role in EXPLICIT_AGENT_WALL_ROLES}
+ role_rank["against_wall"] = 1
+ role_records: dict[str, list[dict[str, Any]]] = {}
+ binding_records: dict[str, list[dict[str, Any]]] = {}
+ support_semantics: dict[str, list[str]] = {}
+
+ for root in task_results:
+ root_name = (
+ str(root.get("task_id") or "task")
+ if isinstance(root, dict)
+ else "task"
+ )
+ for source_path, node in _walk_with_path(root, root_name):
+ if not isinstance(node, dict):
+ continue
+ object_id = node.get("object_id")
+ target_id = node.get("object_target")
+ role = str(node.get("role") or "").strip().lower()
+ confidence = str(node.get("confidence") or "").strip().lower()
+ if (
+ isinstance(object_id, str)
+ and role in FLOOR_AGENT_WALL_ROLES
+ and confidence in confidence_rank
+ ):
+ role_records.setdefault(object_id, []).append(
+ {
+ "object_id": object_id,
+ "role": role,
+ "confidence": confidence,
+ "evidence_text": _semantic_wall_text(node),
+ "source_path": source_path,
+ }
+ )
+ if (
+ isinstance(target_id, str)
+ and node.get("support_mode") is not None
+ ):
+ support_text = _semantic_wall_text(node)
+ if support_text:
+ support_semantics.setdefault(target_id, []).append(
+ support_text
+ )
+
+ bindings = node.get("candidate_wall_bindings")
+ if not isinstance(bindings, list):
+ continue
+ for index, binding in enumerate(bindings):
+ if not isinstance(binding, dict):
+ continue
+ target = binding.get("object_id")
+ candidate_id = str(binding.get("candidate_id") or "")
+ if (
+ selected_candidate_id
+ and candidate_id
+ and candidate_id != selected_candidate_id
+ ):
+ continue
+ binding_role = str(binding.get("role") or "").strip().lower()
+ binding_confidence = str(
+ binding.get("confidence") or ""
+ ).strip().lower()
+ provisional_wall = str(
+ binding.get("final_wall_id")
+ or binding.get("candidate_wall_id")
+ or binding.get("provisional_wall_id")
+ or binding.get("wall_id")
+ or ""
+ )
+ wall_id = wall_id_remap.get(provisional_wall, provisional_wall)
+ if (
+ not isinstance(target, str)
+ or binding_role not in FLOOR_AGENT_WALL_ROLES
+ or binding_confidence not in confidence_rank
+ or wall_id not in WALL_IDS
+ ):
+ continue
+ binding_records.setdefault(target, []).append(
+ {
+ "object_id": target,
+ "wall_id": wall_id,
+ "provisional_wall_id": provisional_wall,
+ "candidate_id": candidate_id or None,
+ "role": binding_role,
+ "confidence": binding_confidence,
+ "evidence_text": _semantic_wall_text(binding),
+ "source_path": (
+ f"{source_path}.candidate_wall_bindings[{index}]"
+ ),
+ }
+ )
+
+ resolved: dict[str, dict[str, Any]] = {}
+ ambiguous: list[dict[str, Any]] = []
+ for target, records in binding_records.items():
+ wall_ids = sorted({str(item["wall_id"]) for item in records})
+ if len(wall_ids) != 1:
+ ambiguous.append(
+ {
+ "object_target": target,
+ "wall_ids": wall_ids,
+ "reason": "multiple_semantic_wall_bindings",
+ "bindings": records,
+ }
+ )
+ continue
+ all_role_records = records + role_records.get(target, [])
+ best_role = max(
+ all_role_records,
+ key=lambda item: (
+ confidence_rank.get(str(item.get("confidence") or ""), 0),
+ role_rank.get(str(item.get("role") or ""), 0),
+ ),
+ )
+ evidence_text = _alias_key(
+ " ".join(
+ [
+ *[
+ str(item.get("evidence_text") or "")
+ for item in all_role_records
+ ],
+ *support_semantics.get(target, []),
+ ]
+ )
+ )
+ resolved[target] = {
+ "object_target": target,
+ "wall_id": wall_ids[0],
+ "role": best_role.get("role"),
+ "confidence": best_role.get("confidence"),
+ "evidence_text": evidence_text,
+ "signal_flags": _semantic_wall_signal_flags(evidence_text),
+ "selected_candidate_id": selected_candidate_id,
+ "bindings": records,
+ "role_evidence": role_records.get(target, []),
+ "support_semantics": support_semantics.get(target, []),
+ }
+ return {
+ "selected_candidate_id": selected_candidate_id,
+ "bindings": resolved,
+ "ambiguous": ambiguous,
+ }
+
+
+def reconcile_semantic_wall_contact_relations(
+ relations: list[dict[str, Any]],
+ accepted_candidates: list[dict[str, Any]],
+ objects: list[dict[str, Any]] | None,
+ task_results: list[dict[str, Any]],
+ wall_evidence: dict[str, Any] | None,
+ coverage_repairs: list[dict[str, Any]] | None = None,
+ blocked_wall_attach_anchors: dict[str, str] | None = None,
+ wall_id_remap: dict[str, str] | None = None,
+) -> tuple[
+ list[dict[str, Any]],
+ dict[str, Any],
+ dict[tuple[str, str], dict[str, Any]],
+]:
+ """Reconcile saved semantic wall binding with saved contact evidence.
+
+ The rule is category-agnostic and conservative. A missing floor+wall
+ relation is restored only when the Wall Agent's sole negative is the
+ structured soft geometry outcome floor_supported_proximity_only with
+ no flat-contact run, and independent high-confidence semantic plus contact
+ evidence points to one wall. Non-soft rejection remains authoritative.
+ """
+ completed = list(relations)
+ object_by_id = _object_context_by_id(objects)
+ coverage_repairs = coverage_repairs or []
+ blocked_wall_attach_anchors = blocked_wall_attach_anchors or {}
+ wall_id_remap = wall_id_remap or {}
+ binding_report = collect_semantic_wall_bindings(
+ task_results,
+ wall_evidence,
+ wall_id_remap,
+ )
+ geometry = _geometry_by_object_wall(wall_evidence)
+ floor_targets = {
+ str(rel.get("parameters", {}).get("object_target"))
+ for rel in completed
+ if isinstance(rel, dict)
+ and rel.get("relation") == "attach"
+ and isinstance(rel.get("parameters"), dict)
+ and rel.get("parameters", {}).get("object_anchor") == "floor"
+ }
+
+ accepted_wall_pairs: set[tuple[str, str]] = set()
+ for item in accepted_candidates:
+ rel = item.get("relation") if isinstance(item, dict) else None
+ params = rel.get("parameters", {}) if isinstance(rel, dict) else {}
+ anchor = str(params.get("object_anchor") or "")
+ target = str(params.get("object_target") or "")
+ mapped_anchor = wall_id_remap.get(anchor, anchor)
+ if (
+ isinstance(rel, dict)
+ and rel.get("relation") == "attach"
+ and mapped_anchor.startswith("wall_")
+ ):
+ accepted_wall_pairs.add((target, mapped_anchor))
+
+ coverage_wall_pairs: set[tuple[str, str]] = set()
+ for item in coverage_repairs:
+ rel = item.get("relation") if isinstance(item, dict) else None
+ params = rel.get("parameters", {}) if isinstance(rel, dict) else {}
+ anchor = str(params.get("object_anchor") or "")
+ target = str(params.get("object_target") or "")
+ if item.get("coverage_repair") is True and anchor.startswith("wall_"):
+ coverage_wall_pairs.add((target, anchor))
+
+ pair_signals: dict[tuple[str, str], list[dict[str, Any]]] = {}
+ reject_decisions = {"reject", "rejected", "false", "no"}
+ for root in task_results:
+ root_name = (
+ str(root.get("task_id") or "task")
+ if isinstance(root, dict)
+ else "task"
+ )
+ for source_path, node in _walk_with_path(root, root_name):
+ if not isinstance(node, dict) or node.get("decision") is None:
+ continue
+ target = str(node.get("object_target") or "")
+ params = (
+ node.get("parameters")
+ if isinstance(node.get("parameters"), dict)
+ else {}
+ )
+ provisional_wall = str(
+ node.get("candidate_wall_id")
+ or node.get("wall_id")
+ or params.get("object_anchor")
+ or ""
+ )
+ wall_id = wall_id_remap.get(provisional_wall, provisional_wall)
+ if not target.startswith("object_") or wall_id not in WALL_IDS:
+ continue
+ text = _semantic_wall_text(node)
+ flags = _semantic_wall_signal_flags(text)
+ decision = str(node.get("decision") or "").lower()
+ reason_code = _alias_key(
+ node.get("reason_code") or node.get("reason")
+ )
+ evidence_values = (
+ node.get("evidence_codes")
+ if isinstance(node.get("evidence_codes"), list)
+ else []
+ )
+ evidence_codes = {
+ _alias_key(value)
+ for value in evidence_values
+ if value is not None
+ }
+ explicit_position_evidence = any(
+ "position" in value for value in evidence_codes
+ )
+ strong_verified_contact = any(
+ marker in text
+ for marker in (
+ "valid_flat_contact",
+ "validated_flat_contact",
+ "flat_contact_run",
+ "flat_edge_contact",
+ "flush",
+ "mounted",
+ "wall_hung",
+ "wall_mounted",
+ )
+ )
+ soft_geometry_rejection = (
+ decision in reject_decisions
+ and reason_code == "floor_supported_proximity_only"
+ and "floor_supported" in evidence_codes
+ and any(
+ "no_flat_contact" in value
+ for value in evidence_codes
+ )
+ )
+ pair_signals.setdefault((target, wall_id), []).append(
+ {
+ "decision": decision,
+ "reason_code": reason_code,
+ "source_path": source_path,
+ "evidence_text": text,
+ "explicit_separation": flags["explicit_separation"],
+ "placement_only": flags["placement_only"]
+ or (
+ explicit_position_evidence
+ and not strong_verified_contact
+ ),
+ "soft_geometry_rejection": soft_geometry_rejection,
+ "hard_rejection": (
+ decision in reject_decisions
+ and not soft_geometry_rejection
+ ),
+ "candidate_mount_surface_rejected": (
+ decision in reject_decisions
+ and reason_code
+ == "candidate_wall_not_verified_mount_surface"
+ ),
+ }
+ )
+
+ added: list[dict[str, Any]] = []
+ remapped: list[dict[str, Any]] = []
+ retained: list[dict[str, Any]] = []
+ skipped: list[dict[str, Any]] = []
+ trusted: dict[tuple[str, str], dict[str, Any]] = {}
+ high_confidences = {"high", "very_high", "very high"}
+ for target, binding in binding_report["bindings"].items():
+ wall_id = str(binding["wall_id"])
+ role = str(binding.get("role") or "")
+ confidence = str(binding.get("confidence") or "")
+ metric = geometry.get((target, wall_id), {})
+ distance = metric.get("footprint_min_distance_to_wall")
+ local_possible = (
+ metric.get("possible_against_wall_contact") is True
+ or (
+ isinstance(distance, (int, float))
+ and float(distance) <= 0.035
+ )
+ )
+ semantic_flags = binding.get("signal_flags", {})
+ support_semantics = [
+ _alias_key(value)
+ for value in binding.get("support_semantics", [])
+ ]
+ independent_floor_support = any(
+ marker in text
+ for text in support_semantics
+ for marker in (
+ "appliance_on_floor",
+ "floor_only",
+ "free_standing",
+ "freestanding",
+ "standalone",
+ )
+ )
+ signals = pair_signals.get((target, wall_id), [])
+ soft_geometry_rejection = any(
+ item.get("soft_geometry_rejection") for item in signals
+ )
+ hard_rejection = any(
+ item.get("hard_rejection") for item in signals
+ )
+ explicit_separation = bool(
+ semantic_flags.get("explicit_separation")
+ or any(item.get("explicit_separation") for item in signals)
+ )
+ placement_only = any(item.get("placement_only") for item in signals)
+ obj = object_by_id.get(target, {})
+ object_text = _alias_key(
+ f"{obj.get('category', '')} {obj.get('description', '')}"
+ )
+ freestanding = any(
+ marker in object_text
+ for marker in ("free_standing", "freestanding", "standalone")
+ )
+ existing_walls = {
+ str(rel.get("parameters", {}).get("object_anchor") or "")
+ for rel in completed
+ if isinstance(rel, dict)
+ and rel.get("relation") == "attach"
+ and isinstance(rel.get("parameters"), dict)
+ and str(
+ rel.get("parameters", {}).get("object_target") or ""
+ ) == target
+ and str(
+ rel.get("parameters", {}).get("object_anchor") or ""
+ ).startswith("wall_")
+ }
+ pair = (target, wall_id)
+ high_confidence = confidence in high_confidences
+ explicit_role = role in EXPLICIT_AGENT_WALL_ROLES
+ unreviewed_explicit_contact = (
+ role == "against_wall"
+ and target in floor_targets
+ and not signals
+ and semantic_flags.get("explicit_contact") is True
+ )
+ against_wall_contact = (
+ role == "against_wall"
+ and target in floor_targets
+ and (
+ (
+ high_confidence
+ and soft_geometry_rejection
+ and (
+ local_possible
+ or semantic_flags.get("explicit_contact") is True
+ )
+ )
+ or unreviewed_explicit_contact
+ )
+ )
+ explicit_wall_contact = explicit_role and high_confidence
+ already_accepted_contact = (
+ pair in accepted_wall_pairs
+ and wall_id in existing_walls
+ and role == "against_wall"
+ and semantic_flags.get("explicit_contact") is True
+ )
+ positive = (
+ explicit_wall_contact
+ or against_wall_contact
+ or already_accepted_contact
+ )
+ base_record = {
+ "object_target": target,
+ "wall_id": wall_id,
+ "semantic_wall_binding": binding,
+ "local_possible_against_wall_contact": local_possible,
+ "footprint_min_distance_to_wall": distance,
+ "soft_geometry_rejection": soft_geometry_rejection,
+ "unreviewed_explicit_contact": unreviewed_explicit_contact,
+ "independent_floor_support": independent_floor_support,
+ "support_semantics": support_semantics,
+ "pair_signals": signals,
+ }
+ if wall_id in blocked_wall_attach_anchors:
+ skipped.append(
+ {
+ **base_record,
+ "reason": "semantic_wall_binding_is_weak_or_inferred",
+ "blocked_wall_anchor_reason": (
+ blocked_wall_attach_anchors[wall_id]
+ ),
+ }
+ )
+ continue
+ if (
+ freestanding
+ or independent_floor_support
+ or explicit_separation
+ or placement_only
+ or hard_rejection
+ ):
+ skipped.append(
+ {
+ **base_record,
+ "reason": (
+ "freestanding_semantics_veto_wall_contact"
+ if freestanding
+ else "independent_floor_support_semantics_veto_wall_contact"
+ if independent_floor_support
+ else "explicit_wall_separation_veto"
+ if explicit_separation
+ else "placement_only_wall_evidence_veto"
+ if placement_only
+ else "non_soft_wall_agent_rejection_veto"
+ ),
+ }
+ )
+ continue
+ if not positive:
+ skipped.append(
+ {
+ **base_record,
+ "reason": "insufficient_independent_contact_evidence",
+ }
+ )
+ continue
+
+ if wall_id in existing_walls:
+ record = {
+ **base_record,
+ "reason": (
+ "accepted_semantic_contact_relation_retained"
+ if already_accepted_contact
+ else "existing_relation_matches_semantic_wall_contact"
+ ),
+ }
+ retained.append(record)
+ trusted[pair] = record
+ continue
+ if existing_walls:
+ wrong_pairs = {
+ (target, existing_wall) for existing_wall in existing_walls
+ }
+ coverage_only = all(
+ wrong_pair in coverage_wall_pairs
+ and wrong_pair not in accepted_wall_pairs
+ for wrong_pair in wrong_pairs
+ )
+ wrong_coverage_rejected = all(
+ any(
+ signal.get("candidate_mount_surface_rejected")
+ for signal in pair_signals.get(wrong_pair, [])
+ )
+ for wrong_pair in wrong_pairs
+ )
+ can_remap_coverage = (
+ explicit_wall_contact
+ and coverage_only
+ and (local_possible or wrong_coverage_rejected)
+ )
+ if not can_remap_coverage:
+ skipped.append(
+ {
+ **base_record,
+ "reason": (
+ "existing_non_coverage_or_unverified_wall_"
+ "relation_not_overridden"
+ ),
+ "existing_wall_ids": sorted(existing_walls),
+ "coverage_only": coverage_only,
+ "wrong_coverage_wall_rejected": (
+ wrong_coverage_rejected
+ ),
+ }
+ )
+ continue
+ original_relations = [
+ rel
+ for rel in completed
+ if isinstance(rel, dict)
+ and rel.get("relation") == "attach"
+ and isinstance(rel.get("parameters"), dict)
+ and str(
+ rel.get("parameters", {}).get("object_target") or ""
+ ) == target
+ and str(
+ rel.get("parameters", {}).get("object_anchor") or ""
+ ) in existing_walls
+ ]
+ completed = [
+ rel for rel in completed if rel not in original_relations
+ ]
+ remap_record = {
+ **base_record,
+ "reason": (
+ "coverage_only_wall_choice_reconciled_to_unique_"
+ "semantic_binding"
+ ),
+ "original_wall_ids": sorted(existing_walls),
+ "wrong_coverage_wall_rejected": wrong_coverage_rejected,
+ }
+ remapped.append(remap_record)
+ for repair in coverage_repairs:
+ repair_rel = (
+ repair.get("relation")
+ if isinstance(repair, dict)
+ else None
+ )
+ repair_params = (
+ repair_rel.get("parameters", {})
+ if isinstance(repair_rel, dict)
+ else {}
+ )
+ if (
+ str(repair_params.get("object_target") or "") == target
+ and str(
+ repair_params.get("object_anchor") or ""
+ ) in existing_walls
+ ):
+ original_relation = json.loads(
+ json.dumps(repair_rel, ensure_ascii=False)
+ )
+ repair[
+ "original_relation_before_semantic_reconciliation"
+ ] = original_relation
+ repair["original_selected_anchor"] = (
+ repair_params.get("object_anchor")
+ )
+ repair["selected_anchor"] = wall_id
+ repair["relation"] = {
+ "relation": "attach",
+ "parameters": {
+ "object_anchor": wall_id,
+ "object_target": target,
+ },
+ }
+ repair["semantic_wall_contact_reconciliation"] = (
+ remap_record["reason"]
+ )
+
+ relation = {
+ "relation": "attach",
+ "parameters": {
+ "object_anchor": wall_id,
+ "object_target": target,
+ },
+ }
+ completed.append(relation)
+ action = {
+ **base_record,
+ "relation": relation,
+ "reason": (
+ "coverage_only_wall_choice_reconciled_to_unique_semantic_binding"
+ if existing_walls
+ else "missing_wall_contact_restored_from_unique_semantic_binding"
+ ),
+ }
+ added.append(action)
+ trusted[pair] = action
+
+ return completed, {
+ "added_count": len(added),
+ "added": added,
+ "retained_count": len(retained),
+ "retained": retained,
+ "coverage_remap_count": len(remapped),
+ "coverage_remaps": remapped,
+ "skipped": skipped,
+ "ambiguous_bindings": binding_report["ambiguous"],
+ "selected_candidate_id": binding_report["selected_candidate_id"],
+ "policy": (
+ "A unique high-confidence semantic wall binding may restore a "
+ "missing floor+wall contact only when the Wall Agent recorded the "
+ "structured soft geometry rejection floor_supported_proximity_only "
+ "with no flat-contact run, or when medium-or-higher semantic evidence "
+ "explicitly states contact and that exact wall pair was never reviewed. "
+ "Other rejections, blocked or ambiguous "
+ "walls, freestanding or independently floor-supported objects, "
+ "placement-only evidence, and explicit separation are vetoes. "
+ "Coverage-only wall choices may be remapped "
+ "only for explicit wall roles when the old anchor was itself "
+ "rejected or saved local geometry supports the bound wall."
+ ),
+ "additional_api_calls": 0,
+ "additional_geometry_or_render_passes": 0,
+ "category_specific_gates": False,
+ }, trusted
+
+
+def _is_explicit_wall_mounted_object(obj: dict[str, Any]) -> bool:
+ category = str(obj.get("category") or "").lower().strip()
+ if category in WALL_MOUNTED_CATEGORIES:
+ return True
+ text = f"{obj.get('category', '')} {obj.get('description', '')}".lower()
+ return any(keyword in text for keyword in WALL_MOUNTED_KEEP_KEYWORDS)
+
+
+def filter_surface_supported_wall_attaches(
+ relations: list[dict[str, Any]],
+ objects: list[dict[str, Any]] | None,
+ semantic_wall_roles: dict[str, dict[str, Any]] | None = None,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """Drop incidental wall proximity for targets already supported by another object.
+
+ Object support and wall attachment coexist only when independent high-confidence
+ wall-mount semantics exist. A flat rear edge or ordinary proximity is not enough.
+ """
+ object_by_id = _object_context_by_id(objects)
+ semantic_wall_roles = semantic_wall_roles or {}
+ object_supported_targets = {
+ rel.get("parameters", {}).get("object_target")
+ for rel in relations
+ if isinstance(rel, dict)
+ and rel.get("relation") == "attach"
+ and isinstance(rel.get("parameters"), dict)
+ and str(rel.get("parameters", {}).get("object_anchor", "")).startswith("object_")
+ }
+ if not object_supported_targets:
+ return relations, []
+ kept: list[dict[str, Any]] = []
+ skipped: list[dict[str, Any]] = []
+ for rel in relations:
+ params = rel.get("parameters", {}) if isinstance(rel.get("parameters"), dict) else {}
+ anchor = params.get("object_anchor")
+ target = params.get("object_target")
+ if not (str(anchor).startswith("wall_") and target in object_supported_targets):
+ kept.append(rel)
+ continue
+ obj = object_by_id.get(str(target), {})
+ explicit_agent_wall_mount = str(target) in semantic_wall_roles
+ if _is_explicit_wall_mounted_object(obj) or explicit_agent_wall_mount:
+ kept.append(rel)
+ continue
+ skipped.append(
+ {
+ "relation": rel,
+ "reason": "surface_supported_object_wall_attach_filtered",
+ "policy": "A target already supported by another object keeps wall attach only with explicit high-confidence wall-mounted/embedded/hung/rack/bracket evidence; proximity or a flat edge alone is insufficient.",
+ "object": obj,
+ "semantic_wall_role": semantic_wall_roles.get(str(target)),
+ }
+ )
+ return kept, skipped
+
+
+def collect_visual_reconstruction_wall_contact_overrides(
+ task_results: list[dict[str, Any]],
+) -> dict[tuple[str, str], dict[str, Any]]:
+ """Collect exact Agent-approved wall pairs where image evidence overrides noisy SAM3D geometry."""
+ overrides: dict[tuple[str, str], dict[str, Any]] = {}
+ for root in task_results:
+ if not isinstance(root, dict):
+ continue
+ kind = " ".join(
+ str(root.get(key) or "")
+ for key in ("task_id", "task_type", "canonical_task_kind")
+ ).lower()
+ capabilities = root.get("provided_capabilities", [])
+ if "wall_attach" not in kind and not (
+ isinstance(capabilities, list) and "wall_attach_decisions_ready" in capabilities
+ ):
+ continue
+ for path, node in _walk_with_path(root, str(root.get("task_id") or "wall_attach_task")):
+ values = node.get("wall_attach_candidates") if isinstance(node, dict) else None
+ if not isinstance(values, list):
+ continue
+ for candidate in values:
+ if not isinstance(candidate, dict):
+ continue
+ target = str(candidate.get("object_target") or "")
+ wall_id = str(candidate.get("candidate_wall_id") or candidate.get("wall_id") or "")
+ geometry = candidate.get("geometry_evidence")
+ geometry = geometry if isinstance(geometry, dict) else {}
+ geometry_text = " ".join(
+ str(geometry.get(key) or "")
+ for key in ("geometry_reliability", "conflict", "reason")
+ ).lower()
+ structured_reconstruction_uncertainty = (
+ geometry.get("reconstruction_uncertainty_used") is True
+ and bool(str(geometry.get("conflict") or geometry.get("reason") or "").strip())
+ )
+ contact_type = re.sub(
+ r"[^a-z0-9]+",
+ "_",
+ str(candidate.get("wall_contact_type") or "").strip().lower(),
+ ).strip("_")
+ flat_back_or_side_contact = (
+ contact_type == "against_wall_horizontal_back_contact"
+ or (
+ contact_type.startswith("against_wall_")
+ and any(marker in contact_type for marker in ("back", "rear", "side"))
+ and any(marker in contact_type for marker in ("contact", "surface", "panel"))
+ and not any(marker in contact_type for marker in ("point", "corner", "tangent"))
+ )
+ )
+ qualifies = (
+ str(candidate.get("decision") or "").lower() in {"accept", "accepted", "true", "yes"}
+ and candidate.get("decision_is_final") is True
+ and flat_back_or_side_contact
+ and bool(str(candidate.get("visual_evidence") or "").strip())
+ and bool(str(candidate.get("mask_evidence") or "").strip())
+ and (
+ structured_reconstruction_uncertainty
+ or "reconstruction_override" in geometry_text
+ or "conflicts_with_reconstructed" in geometry_text
+ or "reconstructed_footprint" in geometry_text
+ )
+ and target.startswith("object_")
+ and wall_id.startswith("wall_")
+ )
+ if qualifies:
+ overrides[(target, wall_id)] = {
+ "object_id": target,
+ "wall_id": wall_id,
+ "source_task_id": root.get("task_id"),
+ "source_path": f"{path}.wall_attach_candidates",
+ "visual_evidence": candidate.get("visual_evidence"),
+ "mask_evidence": candidate.get("mask_evidence"),
+ "geometry_evidence": geometry,
+ "reason": candidate.get("reason"),
+ }
+ return overrides
+
+
+def collect_floor_wall_relation_policy_overrides(
+ accepted_candidates: list[dict[str, Any]],
+ objects: list[dict[str, Any]] | None,
+ wall_evidence: dict[str, Any] | None,
+ semantic_wall_roles: dict[str, dict[str, Any]] | None = None,
+ wall_id_remap: dict[str, str] | None = None,
+) -> dict[str, dict[tuple[str, str], dict[str, Any]]]:
+ """Separate verified contact from proximity-only Agent language."""
+ object_by_id = _object_context_by_id(objects)
+ semantic_wall_roles = semantic_wall_roles or {}
+ wall_id_remap = wall_id_remap or {}
+ geometry = _geometry_by_object_wall(wall_evidence)
+ trusted: dict[tuple[str, str], dict[str, Any]] = {}
+ proximity_only: dict[tuple[str, str], dict[str, Any]] = {}
+ strong_contact_markers = (
+ "valid_flat_contact",
+ "validated_flat_contact",
+ "flat_contact_run",
+ "flat_edge_contact",
+ "flush",
+ )
+ installed_markers = (
+ "built_in",
+ "builtin",
+ "embedded",
+ "fastened",
+ "fixed_to_wall",
+ "installed",
+ "integrated",
+ "mounted",
+ "wall_hung",
+ "wall_mounted",
+ )
+ for item in accepted_candidates:
+ rel = item.get("relation") if isinstance(item.get("relation"), dict) else {}
+ params = rel.get("parameters", {}) if isinstance(rel.get("parameters"), dict) else {}
+ target = str(params.get("object_target") or "")
+ provisional_wall = str(params.get("object_anchor") or "")
+ wall_id = wall_id_remap.get(provisional_wall, provisional_wall)
+ if rel.get("relation") != "attach" or not wall_id.startswith("wall_"):
+ continue
+ obj = object_by_id.get(target, {})
+ category_text = _alias_key(obj.get("category"))
+ category_tokens = set(category_text.split("_"))
+ object_text = _alias_key(f"{obj.get('category', '')} {obj.get('description', '')}")
+ raw = item.get("raw_candidate") if isinstance(item.get("raw_candidate"), dict) else {}
+ evidence_values = raw.get("evidence_codes")
+ if not isinstance(evidence_values, list):
+ evidence_values = []
+ evidence_codes = [_alias_key(value) for value in evidence_values if value is not None]
+ reason_text = _alias_key(
+ " ".join(
+ str(value or "")
+ for value in (
+ raw.get("reason_code"),
+ raw.get("reason"),
+ item.get("reason"),
+ )
+ )
+ )
+ evidence_text = "_".join(evidence_codes + [reason_text])
+ explicitly_installed = (
+ _is_explicit_wall_mounted_object(obj)
+ or any(marker in object_text or marker in evidence_text for marker in installed_markers)
+ )
+ freestanding_cooling = (
+ bool({"refrigerator", "fridge", "freezer"}.intersection(category_tokens))
+ and not explicitly_installed
+ )
+ strong_flat_contact = any(
+ marker in evidence_text for marker in strong_contact_markers
+ )
+ positional_only = (
+ any("position" in code for code in evidence_codes)
+ and not strong_flat_contact
+ and not explicitly_installed
+ )
+ pair = (target, wall_id)
+ if freestanding_cooling or positional_only:
+ proximity_only[pair] = {
+ "object_id": target,
+ "wall_id": wall_id,
+ "reason": (
+ "freestanding_cooling_appliance_proximity_is_not_attachment"
+ if freestanding_cooling
+ else "position_only_wall_evidence_is_not_attachment"
+ ),
+ "category": obj.get("category"),
+ "evidence_codes": evidence_values,
+ "source_path": item.get("source_path"),
+ }
+ continue
+
+ metric = geometry.get(pair, {})
+ distance = metric.get("footprint_min_distance_to_wall")
+ local_possible = (
+ metric.get("possible_against_wall_contact") is True
+ or (
+ isinstance(distance, (int, float))
+ and float(distance) <= 0.035
+ )
+ )
+ role = semantic_wall_roles.get(target, {})
+ high_against_wall = (
+ str(role.get("role") or "") == "against_wall"
+ and str(role.get("confidence") or "") in {"high", "very_high", "very high"}
+ )
+ tall_storage = (
+ "tall" in object_text
+ and any(
+ marker in object_text
+ for marker in ("armoire", "bookcase", "bookshelf", "cabinet", "shelving", "storage", "wardrobe")
+ )
+ )
+ visible_rear_contact = any(
+ marker in evidence_text
+ for marker in ("back_wall", "rear_contact", "rear_wall", "visible_rear_contact")
+ )
+ if (
+ (strong_flat_contact and local_possible)
+ or (tall_storage and visible_rear_contact and high_against_wall)
+ ):
+ trusted[pair] = {
+ "object_id": target,
+ "wall_id": wall_id,
+ "reason": (
+ "accepted_agent_flat_contact_with_local_possible_contact"
+ if strong_flat_contact and local_possible
+ else "high_confidence_tall_storage_visible_rear_contact"
+ ),
+ "category": obj.get("category"),
+ "evidence_codes": evidence_values,
+ "semantic_wall_role": role or None,
+ "local_possible_against_wall_contact": local_possible,
+ "source_path": item.get("source_path"),
+ }
+ for pair in trusted:
+ proximity_only.pop(pair, None)
+ return {
+ "trusted_contact": trusted,
+ "proximity_only": proximity_only,
+ }
+
+
+def filter_floor_supported_wall_attaches_without_flat_contact_edge(
+ relations: list[dict[str, Any]],
+ objects: list[dict[str, Any]] | None,
+ wall_evidence: dict[str, Any] | None,
+ semantic_wall_roles: dict[str, dict[str, Any]] | None = None,
+ visual_reconstruction_overrides: dict[tuple[str, str], dict[str, Any]] | None = None,
+ trusted_agent_contact_overrides: dict[tuple[str, str], dict[str, Any]] | None = None,
+ proximity_only_wall_pairs: dict[tuple[str, str], dict[str, Any]] | None = None,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]], list[dict[str, Any]]]:
+ """Require real flat-edge contact, not point/corner proximity, for ordinary floor+wall contact."""
+ object_by_id = _object_context_by_id(objects)
+ semantic_wall_roles = semantic_wall_roles or {}
+ visual_reconstruction_overrides = visual_reconstruction_overrides or {}
+ trusted_agent_contact_overrides = trusted_agent_contact_overrides or {}
+ proximity_only_wall_pairs = proximity_only_wall_pairs or {}
+ floor_targets = {
+ rel.get("parameters", {}).get("object_target")
+ for rel in relations
+ if isinstance(rel, dict)
+ and rel.get("relation") == "attach"
+ and isinstance(rel.get("parameters"), dict)
+ and rel.get("parameters", {}).get("object_anchor") == "floor"
+ }
+ geometry: dict[tuple[str, str], dict[str, Any]] = {}
+ if isinstance(wall_evidence, dict):
+ for item in wall_evidence.get("wall_attach_geometry", []):
+ if isinstance(item, dict) and item.get("object_id") and item.get("wall_id"):
+ geometry[(str(item["object_id"]), str(item["wall_id"]))] = item
+
+ kept: list[dict[str, Any]] = []
+ filtered: list[dict[str, Any]] = []
+ evaluated: list[dict[str, Any]] = []
+ for rel in relations:
+ params = rel.get("parameters", {}) if isinstance(rel, dict) and isinstance(rel.get("parameters"), dict) else {}
+ anchor = str(params.get("object_anchor") or "")
+ target = str(params.get("object_target") or "")
+ if rel.get("relation") != "attach" or not anchor.startswith("wall_") or target not in floor_targets:
+ kept.append(rel)
+ continue
+ obj = object_by_id.get(target, {})
+ metric = geometry.get((target, anchor), {})
+ analysis = metric.get("flat_contact_edge_analysis") if isinstance(metric.get("flat_contact_edge_analysis"), dict) else {}
+ record = {
+ "relation": rel,
+ "object_id": target,
+ "wall_id": anchor,
+ "is_floor_supported": metric.get("is_floor_supported"),
+ "analysis_available": analysis.get("analysis_available"),
+ "has_valid_flat_contact_edge": analysis.get("has_valid_flat_contact_edge"),
+ "flat_contact_edge_analysis": analysis,
+ }
+ visual_override = visual_reconstruction_overrides.get((target, anchor))
+ if visual_override:
+ record["decision"] = "preserve"
+ record["reason"] = "accepted_visual_wall_contact_with_explicit_reconstruction_override"
+ record["visual_reconstruction_override"] = visual_override
+ evaluated.append(record)
+ kept.append(rel)
+ continue
+ proximity_veto = proximity_only_wall_pairs.get((target, anchor))
+ if proximity_veto:
+ record.update({
+ "decision": "filter",
+ "reason": "floor_supported_wall_attach_proximity_only_evidence",
+ "proximity_only_evidence": proximity_veto,
+ "object": obj,
+ "policy": (
+ "A freestanding cooling appliance or position-only wall candidate "
+ "does not gain attach from ordinary proximity, wall-run alignment, "
+ "or an against_wall label."
+ ),
+ })
+ evaluated.append(record)
+ filtered.append(record)
+ continue
+ trusted_contact = trusted_agent_contact_overrides.get((target, anchor))
+ if trusted_contact:
+ record["decision"] = "preserve"
+ record["reason"] = "accepted_trusted_agent_floor_wall_contact"
+ record["trusted_agent_contact_override"] = trusted_contact
+ evaluated.append(record)
+ kept.append(rel)
+ continue
+ semantic_role = semantic_wall_roles.get(target)
+ semantic_role_has_geometry = bool(
+ semantic_role
+ and metric.get("possible_against_wall_contact") is True
+ and isinstance(metric.get("footprint_min_distance_to_wall"), (int, float))
+ and float(metric["footprint_min_distance_to_wall"]) <= 0.035
+ )
+ if _is_explicit_wall_mounted_object(obj) or semantic_role_has_geometry or metric.get("is_floor_supported") is not True:
+ record["decision"] = "not_applicable"
+ if semantic_role_has_geometry and str(semantic_role.get("role") or "") == "against_wall":
+ record["reason"] = "high_confidence_against_wall_agent_verdict_within_reconstruction_uncertainty"
+ elif semantic_role_has_geometry:
+ record["reason"] = "high_confidence_wall_mounted_or_embedded_with_near_wall_geometry"
+ else:
+ record["reason"] = "explicit_wall_mounted_or_not_floor_supported"
+ if semantic_role:
+ record["semantic_wall_role"] = semantic_role
+ evaluated.append(record)
+ kept.append(rel)
+ continue
+ if analysis.get("analysis_available") is not True:
+ record["decision"] = "preserve"
+ record["reason"] = "mesh_floor_footprint_hull_unavailable"
+ evaluated.append(record)
+ kept.append(rel)
+ continue
+ if analysis.get("has_valid_flat_contact_edge") is True:
+ record["decision"] = "preserve"
+ record["reason"] = "valid_flat_mesh_hull_edge_contact_not_point_or_corner"
+ evaluated.append(record)
+ kept.append(rel)
+ continue
+ record.update({
+ "decision": "filter",
+ "reason": "floor_supported_wall_attach_without_valid_flat_contact_edge",
+ "policy": "An ordinary floor-supported object may keep an additional wall attach when its real lower-mesh XZ hull has a finite wall-parallel flat contact edge. Consecutive near-collinear mesh segments are merged; isolated point/corner proximity and curved tangent fragments do not qualify.",
+ "object": obj,
+ })
+ evaluated.append(record)
+ filtered.append(record)
+ return kept, filtered, evaluated
+
+
+def enforce_single_wall_attach_per_target(
+ relations: list[dict[str, Any]],
+ preferred_wall_by_target: dict[str, str] | None = None,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """Keep at most one wall attach for each object target.
+
+ Gravity/support relations such as floor->object and object->object are not
+ affected. A wall selected by accepted Agent evidence is preferred; when no
+ preference is available, canonical wall id order provides a deterministic
+ fallback.
+ """
+ preferred_wall_by_target = preferred_wall_by_target or {}
+ wall_anchors_by_target: dict[str, list[str]] = {}
+ for relation in relations:
+ parameters = relation.get("parameters", {}) if isinstance(relation, dict) else {}
+ anchor = parameters.get("object_anchor")
+ target = parameters.get("object_target")
+ if relation.get("relation") != "attach" or not isinstance(anchor, str) or not anchor.startswith("wall_"):
+ continue
+ wall_anchors_by_target.setdefault(str(target), []).append(anchor)
+
+ selected_wall_by_target: dict[str, str] = {}
+ for target, anchors in wall_anchors_by_target.items():
+ unique_anchors = sorted(set(anchors))
+ if len(unique_anchors) <= 1:
+ continue
+ preferred = preferred_wall_by_target.get(target)
+ selected_wall_by_target[target] = preferred if preferred in unique_anchors else unique_anchors[0]
+
+ if not selected_wall_by_target:
+ return relations, []
+
+ kept: list[dict[str, Any]] = []
+ removed: list[dict[str, Any]] = []
+ for relation in relations:
+ parameters = relation.get("parameters", {}) if isinstance(relation, dict) else {}
+ anchor = parameters.get("object_anchor")
+ target = str(parameters.get("object_target"))
+ selected_wall = selected_wall_by_target.get(target)
+ if (
+ relation.get("relation") == "attach"
+ and isinstance(anchor, str)
+ and anchor.startswith("wall_")
+ and selected_wall is not None
+ and anchor != selected_wall
+ ):
+ removed.append(
+ {
+ "relation": relation,
+ "reason": "multiple_wall_attach_same_target_deduped",
+ "object_target": target,
+ "removed_anchor": anchor,
+ "kept_anchor": selected_wall,
+ }
+ )
+ continue
+ kept.append(relation)
+ return kept, removed
+
+
+def filter_wall_mounted_floor_attaches(
+ relations: list[dict[str, Any]],
+ objects: list[dict[str, Any]] | None,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ """Remove contradictory non-wall support for explicitly wall-mounted objects.
+
+ A wall-mounted panel/painting/screen is physically covered by its wall
+ attach. Keeping an additional floor or furniture attach for the same target
+ creates a contradictory support path. This is intentionally limited to
+ explicit wall-mounted semantics; leaning or freestanding furniture may still
+ have both floor and wall contact.
+ """
+ object_by_id = _object_context_by_id(objects)
+ wall_attached_targets = {
+ str(rel.get("parameters", {}).get("object_target"))
+ for rel in relations
+ if isinstance(rel, dict)
+ and rel.get("relation") == "attach"
+ and isinstance(rel.get("parameters"), dict)
+ and str(rel.get("parameters", {}).get("object_anchor", "")).startswith("wall_")
+ }
+ if not wall_attached_targets:
+ return relations, []
+
+ kept: list[dict[str, Any]] = []
+ skipped: list[dict[str, Any]] = []
+ for rel in relations:
+ params = rel.get("parameters", {}) if isinstance(rel.get("parameters"), dict) else {}
+ anchor = str(params.get("object_anchor") or "")
+ target = str(params.get("object_target") or "")
+ obj = object_by_id.get(target, {})
+ contradictory_support = anchor == "floor" or anchor.startswith("object_")
+ if contradictory_support and target in wall_attached_targets and _is_explicit_wall_mounted_object(obj):
+ skipped.append(
+ {
+ "relation": rel,
+ "reason": "explicit_wall_mounted_object_with_wall_attach_nonwall_support_filtered",
+ "policy": "An explicitly wall-mounted object with an accepted wall attach is not assigned a second floor or furniture support path.",
+ "object": obj,
+ }
+ )
+ continue
+ kept.append(rel)
+ return kept, skipped
+
+
+def _median_float(values: list[float]) -> float | None:
+ if not values:
+ return None
+ ordered = sorted(values)
+ middle = len(ordered) // 2
+ if len(ordered) % 2:
+ return ordered[middle]
+ return (ordered[middle - 1] + ordered[middle]) / 2.0
+
+
+def _object_bottom_y_from_wall_evidence(wall_evidence: dict[str, Any] | None) -> dict[str, float]:
+ if not isinstance(wall_evidence, dict):
+ return {}
+ result: dict[str, float] = {}
+ for key in ("render_grid_wall_attach_geometry", "wall_attach_geometry"):
+ rows = wall_evidence.get(key)
+ if not isinstance(rows, list):
+ continue
+ for item in rows:
+ if not isinstance(item, dict) or not item.get("object_id"):
+ continue
+ try:
+ bottom_y = float(item.get("bottom_y"))
+ except (TypeError, ValueError):
+ continue
+ result.setdefault(str(item["object_id"]), bottom_y)
+ return result
+
+
+def complete_floor_support_for_structural_anchors(
+ relations: list[dict[str, Any]],
+ objects: list[dict[str, Any]] | None,
+ wall_evidence: dict[str, Any] | None,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Close omitted gravity support only for floor-level structural objects.
+
+ The Agent remains responsible for semantic support selection. This local
+ safety net addresses a narrower invariant: an object used as a support or
+ contain anchor, or a freestanding object with a wall attach, cannot be left
+ with no gravity-direction relation when its mesh bottom belongs to the
+ scene's observed floor-height cluster.
+ """
+ object_by_id = _object_context_by_id(objects)
+ object_ids = set(object_by_id)
+ bottom_y_by_id = _object_bottom_y_from_wall_evidence(wall_evidence)
+ relation_keys = {
+ (
+ rel.get("relation"),
+ rel.get("parameters", {}).get("object_anchor"),
+ rel.get("parameters", {}).get("object_target"),
+ )
+ for rel in relations
+ if isinstance(rel, dict) and isinstance(rel.get("parameters"), dict)
+ }
+
+ candidate_sources: dict[str, set[str]] = {}
+ gravity_resolved_targets: set[str] = set()
+ floor_targets: set[str] = set()
+ for rel in relations:
+ if not isinstance(rel, dict) or not isinstance(rel.get("parameters"), dict):
+ continue
+ relation_type = rel.get("relation")
+ params = rel.get("parameters", {})
+ anchor = str(params.get("object_anchor") or "")
+ target = str(params.get("object_target") or "")
+ if anchor in object_ids and relation_type in {"attach", "contain"}:
+ candidate_sources.setdefault(anchor, set()).add("object_relation_anchor")
+ if relation_type == "attach" and anchor.startswith("wall_") and target in object_ids:
+ candidate_sources.setdefault(target, set()).add("wall_attached_target")
+ if target in object_ids and relation_type == "attach" and not anchor.startswith("wall_"):
+ gravity_resolved_targets.add(target)
+ if anchor == "floor":
+ floor_targets.add(target)
+ elif target in object_ids and relation_type == "contain":
+ gravity_resolved_targets.add(target)
+
+ all_bottoms = list(bottom_y_by_id.values())
+ known_floor_bottoms = [
+ bottom_y_by_id[target]
+ for target in floor_targets
+ if target in bottom_y_by_id
+ ]
+ floor_reference_y = _median_float(known_floor_bottoms)
+ floor_reference_source = "median_existing_floor_attach_targets"
+ if floor_reference_y is None and all_bottoms:
+ ordered = sorted(all_bottoms)
+ floor_reference_y = ordered[int((len(ordered) - 1) * 0.15)]
+ floor_reference_source = "lower_15_percentile_object_bottoms"
+ vertical_span = (max(all_bottoms) - min(all_bottoms)) if len(all_bottoms) >= 2 else 0.0
+ floor_tolerance_y = max(0.025, min(0.060, vertical_span * 0.10))
+ reconstruction_floor_tolerance_y = min(0.090, floor_tolerance_y * 1.5)
+
+ completed = list(relations)
+ added: list[dict[str, Any]] = []
+ skipped: list[dict[str, Any]] = []
+ for object_id in sorted(candidate_sources, key=sorted_object_id_key):
+ sources = sorted(candidate_sources[object_id])
+ obj = object_by_id.get(object_id, {})
+ if object_id in gravity_resolved_targets:
+ skipped.append({
+ "object_id": object_id,
+ "candidate_sources": sources,
+ "reason": "gravity_or_containment_relation_already_present",
+ })
+ continue
+ if _is_ceiling_light_object(obj) or _is_explicit_wall_mounted_object(obj):
+ skipped.append({
+ "object_id": object_id,
+ "candidate_sources": sources,
+ "reason": "explicit_wall_or_ceiling_mounted_object",
+ })
+ continue
+ bottom_y = bottom_y_by_id.get(object_id)
+ if bottom_y is None or floor_reference_y is None:
+ skipped.append({
+ "object_id": object_id,
+ "candidate_sources": sources,
+ "reason": "insufficient_mesh_bottom_evidence",
+ })
+ continue
+ object_floor_tolerance_y = (
+ reconstruction_floor_tolerance_y
+ if "wall_attached_target" in sources
+ else floor_tolerance_y
+ )
+ floor_threshold_y = floor_reference_y + object_floor_tolerance_y
+ if bottom_y > floor_threshold_y:
+ skipped.append({
+ "object_id": object_id,
+ "candidate_sources": sources,
+ "bottom_y": bottom_y,
+ "floor_threshold_y": floor_threshold_y,
+ "floor_tolerance_y": object_floor_tolerance_y,
+ "reconstruction_uncertainty_relaxation_used": "wall_attached_target" in sources,
+ "reason": "object_bottom_above_observed_floor_cluster",
+ })
+ continue
+ relation = {
+ "relation": "attach",
+ "parameters": {"object_anchor": "floor", "object_target": object_id},
+ }
+ key = ("attach", "floor", object_id)
+ if key in relation_keys:
+ continue
+ completed.append(relation)
+ relation_keys.add(key)
+ added.append({
+ "relation": relation,
+ "object_id": object_id,
+ "candidate_sources": sources,
+ "bottom_y": bottom_y,
+ "floor_reference_y": floor_reference_y,
+ "floor_tolerance_y": object_floor_tolerance_y,
+ "reconstruction_uncertainty_relaxation_used": "wall_attached_target" in sources,
+ "reason": "floor_level_structural_anchor_missing_gravity_support",
+ })
+
+ return completed, {
+ "schema_version": "gravity_support_completion_audit",
+ "status": "completed",
+ "policy": "Manager/Agent choose support semantics. Local completion adds floor attach only when an object is a structural relation anchor or wall-attached target, lacks any gravity/containment relation, is not explicitly wall/ceiling mounted, and its mesh bottom lies in the observed floor-height cluster.",
+ "floor_reference_y": floor_reference_y,
+ "floor_reference_source": floor_reference_source if floor_reference_y is not None else "unavailable",
+ "floor_tolerance_y": floor_tolerance_y,
+ "wall_attached_reconstruction_floor_tolerance_y": reconstruction_floor_tolerance_y,
+ "candidate_object_ids": sorted(candidate_sources, key=sorted_object_id_key),
+ "added_count": len(added),
+ "added_relations": added,
+ "skipped_items": skipped,
+ }
+
+
+
+
+def write_relation_normalization_audit(output_dir: Path, audit: dict[str, Any], dry_run: bool) -> dict[str, Any]:
+ data = {"dry_run": dry_run, "status": "completed", **audit}
+ write_audit_pair(output_dir, "relations", "relation_normalization_audit", data, "Relation Normalization Audit")
+ return data
+
+
+def write_flat_contact_edge_wall_attach_audit(
+ output_dir: Path,
+ merge_report: dict[str, Any],
+ dry_run: bool,
+) -> dict[str, Any]:
+ report = merge_report.get("flat_contact_edge_wall_attach_filter", {})
+ data = {
+ "dry_run": dry_run,
+ "status": "completed",
+ "filtered_count": report.get("filtered_count", 0),
+ "items": report.get("items", []),
+ "evaluations": report.get("evaluations", []),
+ "policy": report.get("policy"),
+ }
+ write_audit_pair(
+ output_dir,
+ "relations",
+ "flat_contact_edge_wall_attach_audit",
+ data,
+ "Flat Contact Edge Wall Attach Audit",
+ )
+ return data
+
+
+def write_semantic_wall_contact_reconciliation_audit(
+ output_dir: Path,
+ merge_report: dict[str, Any],
+ dry_run: bool,
+) -> dict[str, Any]:
+ report = merge_report.get("semantic_wall_contact_reconciliation", {})
+ data = {
+ "dry_run": dry_run,
+ "status": "completed",
+ **report,
+ }
+ write_audit_pair(
+ output_dir,
+ "relations",
+ "semantic_wall_contact_reconciliation_audit",
+ data,
+ "Semantic Wall Contact Reconciliation Audit",
+ )
+ return data
+
+
+def write_weak_wall_attach_audit(
+ output_dir: Path,
+ merge_report: dict[str, Any],
+ dry_run: bool,
+ virtual_wall_shift_audit: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+ items = [
+ item
+ for item in merge_report.get("skipped_candidates", [])
+ if isinstance(item, dict) and item.get("reason") == "weak_or_inferred_wall_anchor_not_allowed_for_attach"
+ ]
+ data = {
+ "dry_run": dry_run,
+ "status": "completed",
+ "filtered_count": len(items),
+ "items": items,
+ "blocked_wall_attach_anchors": merge_report.get("blocked_wall_attach_anchors", {}),
+ "wall_id_remap_applied_to_relations": merge_report.get("wall_id_remap_applied_to_relations", {}),
+ "wall_outward_shift": virtual_wall_shift_audit or {"status": "not_run"},
+ "virtual_wall_outward_shift": virtual_wall_shift_audit or {"status": "not_run"},
+ "policy": "Weak/hidden/inferred walls cannot anchor ordinary wall attach. After final relation filtering, every canonical wall without a retained wall attach moves outward once; a filtered weak-wall attach remains an explicit audit trigger.",
+ }
+ write_audit_pair(output_dir, "relations", "weak_wall_attach_audit", data, "Weak Wall Attach Audit")
+ return data
+
+
+def write_gravity_support_completion_audit(
+ output_dir: Path,
+ report: dict[str, Any],
+ dry_run: bool,
+) -> dict[str, Any]:
+ data = {"dry_run": dry_run, **report}
+ write_audit_pair(
+ output_dir,
+ "attach",
+ "gravity_support_completion_audit",
+ data,
+ "Gravity Support Completion Audit",
+ )
+ return data
+
+
+def write_coverage_repair_audit(output_dir: Path, coverage_report: dict[str, Any], dry_run: bool) -> dict[str, Any]:
+ data = {
+ "dry_run": dry_run,
+ "status": "completed" if not coverage_report.get("missing_after") else "needs_repair",
+ "missing_before": coverage_report.get("missing_before", []),
+ "missing_after": coverage_report.get("missing_after", []),
+ "coverage_repair_used": bool(coverage_report.get("used_repairs")),
+ "coverage_repair_count": len(coverage_report.get("used_repairs", [])),
+ "items": coverage_report.get("used_repairs", []),
+ "skipped_candidates": coverage_report.get("skipped_candidates", []),
+ "policy": "Coverage repair must choose the most plausible physical anchor and should prefer local physical contact over generic floor fallback.",
+ }
+ write_audit_pair(output_dir, "attach", "coverage_repair_audit", data, "Coverage Repair Audit")
+ return data
+
+
+def _ceiling_confirmed_from_guard(guard_result: dict[str, Any] | None) -> bool:
+ if not isinstance(guard_result, dict):
+ return False
+ if guard_result.get("ceiling_confirmed") is True or guard_result.get("ceiling_anchor_allowed") is True:
+ return True
+ for node in _walk(guard_result):
+ if isinstance(node, dict) and (node.get("ceiling_confirmed") is True or node.get("ceiling_anchor_allowed") is True or node.get("ceiling_present_for_attach") is True):
+ return True
+ return False
+
+
+def _ceiling_confirmed_from_task_results(task_results: list[dict[str, Any]]) -> bool:
+ accept_decisions = {"accept", "accepted", "true", "yes", "confirmed", "positive"}
+ for root in task_results:
+ if not isinstance(root, dict):
+ continue
+ for node in _walk(root):
+ if not isinstance(node, dict):
+ continue
+ if node.get("relation") == "attach" and isinstance(node.get("parameters"), dict):
+ if node["parameters"].get("object_anchor") == "ceiling":
+ decision_text = str(node.get("decision") or node.get("status") or node.get("result") or "accepted").lower()
+ if decision_text in accept_decisions or decision_text == "":
+ return True
+ if node.get("object_anchor") == "ceiling" or node.get("anchor") == "ceiling":
+ decision_text = str(node.get("decision") or node.get("status") or node.get("result") or "").lower()
+ if decision_text in accept_decisions:
+ return True
+ decision = node.get("ceiling_decision")
+ if isinstance(decision, dict) and (
+ decision.get("allow_ceiling_anchor") is True
+ or decision.get("ceiling_confirmed") is True
+ or decision.get("ceiling_anchor_allowed") is True
+ or str(decision.get("status") or "").lower() in {"positive", "confirmed", "accepted", "yes", "true"}
+ ):
+ return True
+ scene_level = node.get("scene_level_ceiling_present")
+ if isinstance(scene_level, dict) and str(scene_level.get("judgement") or scene_level.get("status") or "").lower() in {"positive", "confirmed", "accepted", "yes", "true"}:
+ return True
+ if str(node.get("suggested_anchor") or "").lower() == "ceiling" and str(node.get("judgement") or node.get("status") or "").lower() in {"positive", "confirmed", "accepted", "yes", "true"}:
+ return True
+ judgements = node.get("ceiling_anchor_judgements")
+ if isinstance(judgements, list):
+ for item in judgements:
+ if isinstance(item, dict) and str(item.get("suggested_anchor") or "").lower() == "ceiling" and str(item.get("judgement") or item.get("status") or "").lower() in {"positive", "confirmed", "accepted", "yes", "true"}:
+ return True
+ if (
+ node.get("allow_ceiling_anchor") is True
+ or node.get("allow_ceiling_attach") is True
+ or node.get("ceiling_confirmed") is True
+ or node.get("ceiling_anchor_allowed") is True
+ ):
+ return True
+ return False
+
+
+def collect_relations(
+ guard_result: dict[str, Any] | None,
+ task_results: list[dict[str, Any]],
+ object_ids: list[str],
+ blocked_wall_attach_anchors: dict[str, str] | None = None,
+ wall_id_remap: dict[str, str] | None = None,
+ objects: list[dict[str, Any]] | None = None,
+ object_properties: dict[str, Any] | None = None,
+ wall_evidence: dict[str, Any] | None = None,
+ scene_state: dict[str, Any] | None = None,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ collection = collect_accepted_attach_candidates(task_results, guard_result)
+ coverage_collection = collect_coverage_repair_candidates(task_results, guard_result)
+ ceiling_confirmed = (
+ _ceiling_confirmed_from_guard(guard_result)
+ or _ceiling_confirmed_from_task_results(task_results)
+ or _ceiling_confirmed_from_objects(objects)
+ )
+ allowed_attach_anchors = {"floor", *WALL_IDS, *object_ids}
+ if ceiling_confirmed:
+ allowed_attach_anchors.add("ceiling")
+ anchor_alias_map = _object_anchor_alias_map(objects)
+ ceiling_light_ids = set(_ceiling_light_object_ids(objects))
+ normalized: list[dict[str, Any]] = []
+ skipped = list(collection["skipped"])
+ blocked_wall_attach_anchors = blocked_wall_attach_anchors or {}
+ wall_id_remap = wall_id_remap or {}
+ seen = set()
+
+ def target_already_covered(target_id: str) -> bool:
+ return target_id in {
+ rel.get("parameters", {}).get("object_target")
+ for rel in normalized
+ if rel.get("relation") in {"attach", "contain"}
+ }
+
+ def should_remap_wall_relation(item: dict[str, Any]) -> bool:
+ source_path = str(item.get("source_path") or "").lower()
+ raw = item.get("raw_candidate") if isinstance(item.get("raw_candidate"), dict) else {}
+ raw_params = raw.get("parameters") if isinstance(raw.get("parameters"), dict) else {}
+ # Wall attach recall runs before final wall canonicalization in the DAG, so
+ # wall ids from that task are provisional candidate ids. Schema/supervisor
+ # outputs, however, are written after final walls exist and may already
+ # use canonical ids; remapping those again can turn a correct final wall id
+ # back into a different provisional/final alias.
+ if "wall_attach_candidate_recall" in source_path or "wall_attach_candidates" in source_path:
+ return True
+ if raw.get("candidate_wall_id") or raw.get("provisional_wall_id") or raw.get("geometry_wall_id"):
+ return True
+ if raw_params.get("candidate_wall_id") or raw_params.get("provisional_wall_id") or raw_params.get("geometry_wall_id"):
+ return True
+ return False
+
+ def relation_remap(item: dict[str, Any], anchor: Any, target: Any) -> tuple[Any, Any]:
+ if not should_remap_wall_relation(item):
+ return anchor, target
+ if anchor in wall_id_remap:
+ anchor = wall_id_remap[anchor]
+ if target in wall_id_remap:
+ target = wall_id_remap[target]
+ return anchor, target
+
+ def raw_wall_aliases_for_candidate(item: dict[str, Any]) -> list[str]:
+ raw = item.get("raw_candidate") if isinstance(item.get("raw_candidate"), dict) else {}
+ aliases: list[str] = []
+
+ def add(value: Any) -> None:
+ if value is None:
+ return
+ if isinstance(value, str):
+ for match in re.findall(r"wall_\d{3}", value):
+ if match not in aliases:
+ aliases.append(match)
+ elif isinstance(value, dict):
+ for child in value.values():
+ add(child)
+ elif isinstance(value, list):
+ for child in value:
+ add(child)
+
+ for key in [
+ "provisional_wall_id",
+ "candidate_wall_id",
+ "geometry_wall_id",
+ "nearest_wall_id_by_footprint",
+ "footprint_nearest_wall",
+ "nearest_wall",
+ ]:
+ add(raw.get(key))
+ geom = raw.get("geometry_evidence")
+ if isinstance(geom, dict):
+ for key in ["provisional_wall_id", "nearest_wall_id_by_footprint", "footprint_nearest_wall", "nearest_wall"]:
+ add(geom.get(key))
+ return aliases
+
+ def remap_blocked_wall_anchor_from_raw_candidate(item: dict[str, Any], anchor: Any) -> Any:
+ if not (isinstance(anchor, str) and anchor.startswith("wall_") and anchor in blocked_wall_attach_anchors):
+ return anchor
+ for alias in raw_wall_aliases_for_candidate(item):
+ remapped = wall_id_remap.get(alias, alias)
+ if (
+ isinstance(remapped, str)
+ and remapped.startswith("wall_")
+ and remapped in allowed_attach_anchors
+ and remapped not in blocked_wall_attach_anchors
+ ):
+ item.setdefault("wall_anchor_repair", {
+ "original_anchor": anchor,
+ "raw_wall_alias": alias,
+ "remapped_anchor": remapped,
+ "reason": "raw candidate geometry alias remapped away from weak/inferred wall anchor",
+ })
+ return remapped
+ return anchor
+
+ def try_add(item: dict[str, Any], source: str, require_missing_target: bool = False) -> bool:
+ rel = json.loads(json.dumps(item["relation"], ensure_ascii=False))
+ params = rel.get("parameters", {})
+ anchor = params.get("object_anchor")
+ target = params.get("object_target")
+ anchor, target = relation_remap(item, anchor, target)
+ params["object_anchor"] = anchor
+ params["object_target"] = target
+ relation_type = rel.get("relation")
+ if relation_type == "attach" and isinstance(anchor, str) and anchor not in allowed_attach_anchors:
+ mapped_anchor = anchor_alias_map.get(_alias_key(anchor))
+ if mapped_anchor:
+ params["object_anchor"] = mapped_anchor
+ anchor = mapped_anchor
+ if relation_type not in {"attach", "contain"}:
+ skipped.append({**item, "reason": "relation_type_not_allowed", "merge_source": source})
+ return False
+ if target not in object_ids:
+ skipped.append({**item, "reason": "object_target_not_in_final_objects", "merge_source": source})
+ return False
+ if relation_type == "attach":
+ if anchor not in allowed_attach_anchors:
+ skipped.append({**item, "reason": "object_anchor_not_allowed", "merge_source": source})
+ return False
+ repaired_anchor = remap_blocked_wall_anchor_from_raw_candidate(item, anchor)
+ if repaired_anchor != anchor:
+ params["object_anchor"] = repaired_anchor
+ anchor = repaired_anchor
+ if anchor in blocked_wall_attach_anchors:
+ skipped.append(
+ {
+ **item,
+ "reason": "weak_or_inferred_wall_anchor_not_allowed_for_attach",
+ "blocked_wall_anchor_reason": blocked_wall_attach_anchors[anchor],
+ "merge_source": source,
+ }
+ )
+ return False
+ elif relation_type == "contain":
+ if anchor not in object_ids:
+ skipped.append({**item, "reason": "contain_anchor_must_be_object", "merge_source": source})
+ return False
+ if anchor == target:
+ skipped.append({**item, "reason": "contain_anchor_target_same_object", "merge_source": source})
+ return False
+ if require_missing_target and target_already_covered(target):
+ skipped.append({**item, "reason": "target_already_covered", "merge_source": source})
+ return False
+ key = (relation_type, anchor, target)
+ if key in seen:
+ skipped.append({**item, "reason": "duplicate_relation", "merge_source": source})
+ return False
+ normalized.append({"relation": relation_type, "parameters": {"object_anchor": anchor, "object_target": target}})
+ seen.add(key)
+ return True
+
+ for item in collection["accepted"]:
+ try_add(item, "accepted_relation")
+
+ contain_conflict_report = collect_contain_conflicts(scene_state, task_results)
+ contain_conflict_resolutions = collect_contain_conflict_resolutions(guard_result)
+ contain_conflict_applied: list[dict[str, Any]] = []
+ contain_conflict_unresolved: list[dict[str, Any]] = []
+ contain_conflict_candidate_removals: list[dict[str, Any]] = []
+ schema_guard_support_pairs: set[tuple[str, str]] = set()
+
+ # Remove both provisional verdicts before applying the single SchemaGuard
+ # decision. This prevents insertion order or relation type from becoming a
+ # hidden semantic priority.
+ for conflict in contain_conflict_report.get("conflicts", []):
+ if not isinstance(conflict, dict):
+ continue
+ conflict_pair = (conflict.get("object_anchor"), conflict.get("object_target"))
+ kept_relations: list[dict[str, Any]] = []
+ for relation in normalized:
+ params = relation.get("parameters", {}) if isinstance(relation.get("parameters"), dict) else {}
+ pair = (params.get("object_anchor"), params.get("object_target"))
+ if pair == conflict_pair and relation.get("relation") in {"attach", "contain"}:
+ removal = {
+ "relation": relation,
+ "reason": "schema_guard_attach_contain_arbitration",
+ "conflict_id": conflict.get("conflict_id"),
+ "merge_source": "contain_conflict_pre_arbitration",
+ }
+ contain_conflict_candidate_removals.append(removal)
+ skipped.append(removal)
+ continue
+ kept_relations.append(relation)
+ normalized = kept_relations
+ seen = {
+ (
+ relation.get("relation"),
+ relation.get("parameters", {}).get("object_anchor"),
+ relation.get("parameters", {}).get("object_target"),
+ )
+ for relation in normalized
+ if isinstance(relation, dict) and isinstance(relation.get("parameters"), dict)
+ }
+ trusted_multi_level_pairs: set[tuple[str, str]] = set()
+
+ for conflict in contain_conflict_report.get("conflicts", []):
+ if not isinstance(conflict, dict):
+ continue
+ conflict_id = str(conflict.get("conflict_id") or "")
+ anchor = conflict.get("object_anchor")
+ target = conflict.get("object_target")
+ resolution = contain_conflict_resolutions.get(conflict_id)
+ if not isinstance(resolution, dict):
+ contain_conflict_unresolved.append({**conflict, "reason": "schema_guard_resolution_missing"})
+ continue
+ decision = str(resolution.get("decision") or resolution.get("verdict") or "").lower()
+ accepted_attach = conflict.get("accepted_attach") if isinstance(conflict.get("accepted_attach"), dict) else {}
+ accepted_contain = conflict.get("accepted_contain") if isinstance(conflict.get("accepted_contain"), dict) else {}
+ support_anchor = (
+ resolution.get("support_anchor")
+ or resolution.get("object_support_anchor")
+ or accepted_attach.get("object_anchor")
+ )
+ surface_role, surface_role_source = _normalized_contain_surface_role(
+ resolution,
+ conflict.get("anchor_category"),
+ )
+ effective_decision = decision
+ policy_reason = None
+ if surface_role == CONTAIN_INTERNAL_SURFACE_ROLE:
+ effective_decision = "accept_contain"
+ if decision not in {"accept_contain", "accept", "accepted"}:
+ policy_reason = "internal shelf/cubby membership is contain under the shared ontology"
+ elif surface_role in CONTAIN_EXTERNAL_SURFACE_ROLES:
+ effective_decision = "reject_contain_use_support"
+ if decision not in {"reject_contain_use_support", "reject_with_support", "use_support"}:
+ policy_reason = "outermost top or exterior support is attach under the shared ontology"
+ elif decision in {"reject_contain_use_support", "reject_with_support", "use_support"} and accepted_contain:
+ effective_decision = "accept_contain"
+ policy_reason = "unsubstantiated Guard rejection cannot override an accepted contain specialist verdict"
+
+ if effective_decision in {"accept_contain", "accept", "accepted"}:
+ item = {
+ "relation": {"relation": "contain", "parameters": {"object_anchor": anchor, "object_target": target}},
+ "source_path": "DeliverySchemaGuard.contain_conflict_resolutions",
+ "raw_candidate": resolution,
+ "decision": "accept",
+ }
+ if try_add(item, "contain_conflict_schema_guard"):
+ if surface_role == CONTAIN_INTERNAL_SURFACE_ROLE:
+ trusted_multi_level_pairs.add((str(anchor), str(target)))
+ contain_conflict_applied.append(
+ {
+ "conflict_id": conflict_id,
+ "decision": "accept_contain",
+ "schema_guard_decision": decision,
+ "support_surface_role": surface_role,
+ "support_surface_role_source": surface_role_source,
+ "policy_reason": policy_reason,
+ "relation": item["relation"],
+ "reason": resolution.get("reason") or resolution.get("reason_code") or resolution.get("explanation"),
+ }
+ )
+ continue
+ if effective_decision in {"reject_contain_use_support", "reject_with_support", "use_support"} and support_anchor:
+ schema_guard_support_pairs.add((str(support_anchor), str(target)))
+ item = {
+ "relation": {"relation": "attach", "parameters": {"object_anchor": support_anchor, "object_target": target}},
+ "source_path": "DeliverySchemaGuard.contain_conflict_resolutions",
+ "raw_candidate": resolution,
+ "decision": "accept",
+ }
+ if try_add(item, "contain_conflict_schema_guard"):
+ contain_conflict_applied.append(
+ {
+ "conflict_id": conflict_id,
+ "decision": "reject_contain_use_support",
+ "schema_guard_decision": decision,
+ "support_surface_role": surface_role,
+ "support_surface_role_source": surface_role_source,
+ "policy_reason": policy_reason,
+ "relation": item["relation"],
+ "reason": resolution.get("reason") or resolution.get("reason_code") or resolution.get("explanation"),
+ }
+ )
+ continue
+ contain_conflict_unresolved.append(
+ {
+ **conflict,
+ "reason": "schema_guard_resolution_not_actionable",
+ "schema_guard_resolution": resolution,
+ "support_surface_role": surface_role,
+ "support_surface_role_source": surface_role_source,
+ }
+ )
+
+ # A specific conflict resolution is more authoritative than the Guard's
+ # provisional aggregate relation list. When support is selected, discard any
+ # provisional contain for that exact support pair before later promotion.
+ if schema_guard_support_pairs:
+ kept_relations: list[dict[str, Any]] = []
+ for relation in normalized:
+ params = relation.get("parameters", {}) if isinstance(relation.get("parameters"), dict) else {}
+ pair = (str(params.get("object_anchor") or ""), str(params.get("object_target") or ""))
+ if pair in schema_guard_support_pairs and relation.get("relation") == "contain":
+ removal = {
+ "relation": relation,
+ "reason": "schema_guard_selected_support_for_same_pair",
+ "merge_source": "contain_conflict_post_arbitration",
+ }
+ contain_conflict_candidate_removals.append(removal)
+ skipped.append(removal)
+ continue
+ kept_relations.append(relation)
+ normalized = kept_relations
+ seen = {
+ (
+ relation.get("relation"),
+ relation.get("parameters", {}).get("object_anchor"),
+ relation.get("parameters", {}).get("object_target"),
+ )
+ for relation in normalized
+ if isinstance(relation, dict) and isinstance(relation.get("parameters"), dict)
+ }
+
+ multi_level_contain_promotion = collect_multi_level_support_contain_promotions(task_results)
+ conflict_pairs = {
+ (str(item.get("object_anchor") or ""), str(item.get("object_target") or ""))
+ for item in contain_conflict_report.get("conflicts", [])
+ if isinstance(item, dict)
+ }
+ conflict_pairs.update(schema_guard_support_pairs)
+ promotion_applied: list[dict[str, Any]] = []
+ for item in multi_level_contain_promotion.get("accepted", []):
+ pair = (str(item.get("object_anchor") or ""), str(item.get("object_target") or ""))
+ if pair in conflict_pairs:
+ multi_level_contain_promotion["skipped"].append({
+ **item,
+ "reason": (
+ "schema_guard_selected_support_for_same_pair"
+ if pair in schema_guard_support_pairs
+ else "attach_contain_conflict_already_delegated_to_schema_guard"
+ ),
+ })
+ continue
+ trusted_multi_level_pairs.add(pair)
+ if try_add(item, "support_multi_level_contain_promotion"):
+ promotion_applied.append({
+ "relation": item.get("relation"),
+ "source_task_id": item.get("source_task_id"),
+ "source_path": item.get("source_path"),
+ "support_surface_role": item.get("support_surface_role"),
+ "reason_code": item.get("reason_code"),
+ })
+ multi_level_contain_promotion["applied"] = promotion_applied
+
+ unresolved_contain_conflict_targets = {
+ str(item.get("object_target"))
+ for item in contain_conflict_unresolved
+ if isinstance(item, dict) and item.get("object_target")
+ }
+
+ normalized, contain_geometry_validation = filter_contain_relations_by_geometry(
+ normalized,
+ scene_state,
+ trusted_agent_pairs=trusted_multi_level_pairs,
+ )
+ for item in contain_geometry_validation.get("items", []):
+ if item.get("decision") == "reject":
+ skipped.append({
+ "relation": item.get("relation"),
+ "reason": "contain_target_clearly_outside_anchor_obb",
+ "geometry_decision": item,
+ "merge_source": "contain_geometry_policy",
+ })
+
+ coverage_before = _target_coverage(object_ids, normalized)
+ used_repairs: list[dict[str, Any]] = []
+ coverage_skipped = list(coverage_collection["skipped"])
+ for target in coverage_before["missing_object_targets"]:
+ if target not in ceiling_light_ids or not ceiling_confirmed:
+ continue
+ item = {
+ "relation": {"relation": "attach", "parameters": {"object_anchor": "ceiling", "object_target": target}},
+ "source_path": "local_ceiling_light_category_coverage",
+ "reason": "object category/description indicates a ceiling light fixture; ceiling anchor is locally confirmed for coverage",
+ "evidence_used": ["scene_state.object_candidates.category", "scene_state.object_candidates.description"],
+ "coverage_repair": True,
+ }
+ if try_add(item, "local_ceiling_light_category_coverage", require_missing_target=True):
+ used_repairs.append(
+ {
+ "object_target": target,
+ "selected_anchor": "ceiling",
+ "relation": item["relation"],
+ "reason": item["reason"],
+ "evidence_used": item["evidence_used"],
+ "coverage_repair": True,
+ "source_path": item["source_path"],
+ }
+ )
+ for item in coverage_collection["accepted"]:
+ target = item["relation"].get("parameters", {}).get("object_target")
+ if target in unresolved_contain_conflict_targets:
+ coverage_skipped.append({**item, "reason": "contain_conflict_guard_missing_prevents_coverage_repair"})
+ continue
+ if target not in coverage_before["missing_object_targets"]:
+ coverage_skipped.append({**item, "reason": "target_not_missing"})
+ continue
+ repair_anchor = str(item["relation"].get("parameters", {}).get("object_anchor") or "").lower()
+ if repair_anchor == "ceiling" and target not in ceiling_light_ids:
+ coverage_skipped.append(
+ {
+ **item,
+ "reason": "non_fixture_ceiling_coverage_repair_requires_dedicated_ceiling_evidence",
+ }
+ )
+ continue
+ if try_add(item, "coverage_repair", require_missing_target=True):
+ params = item["relation"].get("parameters", {})
+ used_repairs.append(
+ {
+ "object_target": params.get("object_target"),
+ "selected_anchor": params.get("object_anchor"),
+ "relation": item["relation"],
+ "reason": item.get("reason"),
+ "evidence_used": item.get("evidence_used"),
+ "coverage_repair": True,
+ "source_path": item.get("source_path"),
+ }
+ )
+ else:
+ coverage_skipped.append({**item, "reason": "coverage_relation_not_merged"})
+
+ # Last-resort local coverage for freestanding / floor-supported objects.
+ # This keeps fast async runs from failing schema when a support agent omits
+ # obvious floor contacts. It deliberately skips ceiling fixtures and objects
+ # explicitly described as wall-mounted/hung/embedded.
+ object_by_id = _object_context_by_id(objects)
+ explicit_semantic_wall_roles = collect_high_confidence_explicit_wall_roles(task_results)
+ coverage_after_agent_repairs = _target_coverage(object_ids, normalized)
+ for target in coverage_after_agent_repairs["missing_object_targets"]:
+ if target in unresolved_contain_conflict_targets:
+ coverage_skipped.append(
+ {
+ "relation": {"relation": "attach", "parameters": {"object_anchor": "floor", "object_target": target}},
+ "reason": "contain_conflict_guard_missing_prevents_floor_fallback",
+ "coverage_repair": True,
+ }
+ )
+ continue
+ obj = object_by_id.get(str(target), {})
+ if (
+ target in ceiling_light_ids
+ or _is_explicit_wall_mounted_object(obj)
+ or target in explicit_semantic_wall_roles
+ ):
+ if target in explicit_semantic_wall_roles:
+ coverage_skipped.append({
+ "relation": {
+ "relation": "attach",
+ "parameters": {"object_anchor": "floor", "object_target": target},
+ },
+ "reason": "high_confidence_wall_only_object_cannot_use_floor_coverage_fallback",
+ "semantic_wall_role": explicit_semantic_wall_roles[target],
+ "coverage_repair": True,
+ })
+ continue
+ item = {
+ "relation": {"relation": "attach", "parameters": {"object_anchor": "floor", "object_target": target}},
+ "source_path": "local_floor_fallback_coverage",
+ "reason": "No accepted attach/contain target coverage remained after relation merge; defaulting freestanding object to floor support for schema coverage.",
+ "evidence_used": ["scene_state.object_candidates", "local_schema_coverage_fallback"],
+ "coverage_repair": True,
+ }
+ if try_add(item, "local_floor_fallback_coverage", require_missing_target=True):
+ used_repairs.append(
+ {
+ "object_target": target,
+ "selected_anchor": "floor",
+ "relation": item["relation"],
+ "reason": item["reason"],
+ "evidence_used": item["evidence_used"],
+ "coverage_repair": True,
+ "source_path": item["source_path"],
+ }
+ )
+
+ normalized, gravity_support_completion = complete_floor_support_for_structural_anchors(
+ normalized,
+ objects,
+ wall_evidence,
+ )
+
+ floor_semantic_wall_roles = collect_high_confidence_floor_wall_roles(task_results)
+ floor_wall_policy = collect_floor_wall_relation_policy_overrides(
+ collection["accepted"],
+ objects,
+ wall_evidence,
+ floor_semantic_wall_roles,
+ wall_id_remap,
+ )
+ normalized, semantic_wall_contact_reconciliation, reconciled_contact_overrides = (
+ reconcile_semantic_wall_contact_relations(
+ normalized,
+ collection["accepted"],
+ objects,
+ task_results,
+ wall_evidence,
+ used_repairs,
+ blocked_wall_attach_anchors,
+ wall_id_remap,
+ )
+ )
+ trusted_agent_contact_overrides = dict(floor_wall_policy["trusted_contact"])
+ trusted_agent_contact_overrides.update(reconciled_contact_overrides)
+ raw_visual_reconstruction_overrides = collect_visual_reconstruction_wall_contact_overrides(task_results)
+ visual_reconstruction_overrides: dict[tuple[str, str], dict[str, Any]] = {}
+ for (target, wall_id), item in raw_visual_reconstruction_overrides.items():
+ visual_reconstruction_overrides[(target, wall_id)] = item
+ visual_reconstruction_overrides[(target, wall_id_remap.get(wall_id, wall_id))] = item
+ normalized, flat_edge_filtered_wall_attaches, flat_edge_evaluations = filter_floor_supported_wall_attaches_without_flat_contact_edge(
+ normalized,
+ objects,
+ wall_evidence,
+ floor_semantic_wall_roles,
+ visual_reconstruction_overrides,
+ trusted_agent_contact_overrides,
+ floor_wall_policy["proximity_only"],
+ )
+ for item in flat_edge_filtered_wall_attaches:
+ skipped.append({**item, "merge_source": "flat_contact_edge_wall_attach_policy"})
+
+ normalized, attach_contain_conflicts = resolve_attach_contain_conflicts(normalized)
+ for item in attach_contain_conflicts:
+ skipped.append({**item, "merge_source": "attach_contain_pair_policy"})
+
+ filtered_surface_wall_attaches: list[dict[str, Any]] = []
+ normalized, filtered_surface_wall_attaches = filter_surface_supported_wall_attaches(
+ normalized,
+ objects,
+ explicit_semantic_wall_roles,
+ )
+ for item in filtered_surface_wall_attaches:
+ skipped.append({**item, "merge_source": "surface_supported_wall_attach_policy"})
+
+ preferred_wall_by_target: dict[str, str] = {}
+ for item in collection["accepted"]:
+ rel = item.get("relation") if isinstance(item.get("relation"), dict) else None
+ params = rel.get("parameters", {}) if isinstance(rel, dict) else {}
+ anchor = params.get("object_anchor")
+ target = params.get("object_target")
+ if not (isinstance(anchor, str) and anchor.startswith("wall_") and isinstance(target, str)):
+ continue
+ remapped_anchor, _ = relation_remap(item, anchor, target)
+ if isinstance(remapped_anchor, str) and remapped_anchor.startswith("wall_"):
+ preferred_wall_by_target.setdefault(target, remapped_anchor)
+
+ normalized, removed_multi_wall_attaches = enforce_single_wall_attach_per_target(
+ normalized,
+ preferred_wall_by_target,
+ )
+ for item in removed_multi_wall_attaches:
+ skipped.append({**item, "merge_source": "single_wall_attach_per_target_policy"})
+
+ filtered_wall_mounted_floor_attaches: list[dict[str, Any]] = []
+ normalized, filtered_wall_mounted_floor_attaches = filter_wall_mounted_floor_attaches(normalized, objects)
+ for item in filtered_wall_mounted_floor_attaches:
+ skipped.append({**item, "merge_source": "wall_mounted_floor_attach_policy"})
+
+ normalized, relation_normalization_audit = normalize_floor_covering_relations(normalized, object_properties)
+ for item in relation_normalization_audit.get("items", []):
+ if isinstance(item, dict) and item.get("original_relation"):
+ skipped.append({
+ "relation": item.get("original_relation"),
+ "normalized_relation": item.get("normalized_relation"),
+ "reason": item.get("reason"),
+ "merge_source": "floor_covering_relation_normalization",
+ "policy": relation_normalization_audit.get("policy"),
+ })
+
+ normalized, redundant_floor_attaches = filter_redundant_floor_attaches_for_object_supported_targets(normalized)
+ for item in redundant_floor_attaches:
+ skipped.append({**item, "merge_source": "single_gravity_support_path_policy"})
+
+ coverage_after = _target_coverage(object_ids, normalized)
+ normalized.sort(key=lambda item: (sorted_object_id_key(item["parameters"]["object_target"]), item.get("relation"), item["parameters"]["object_anchor"]))
+ return normalized, {
+ "accepted_candidates": collection["accepted"],
+ "skipped_candidates": skipped,
+ "attach_contain_conflict_resolution": {
+ "defensive_removed_relation_count": len(attach_contain_conflicts),
+ "items": attach_contain_conflicts,
+ "policy": "No relation type has fixed priority; any pair reaching this late guard with both attach and contain is removed as unresolved.",
+ },
+ "contain_relation_count": len([rel for rel in normalized if rel.get("relation") == "contain"]),
+ "contain_conflict_resolution": {
+ "schema_version": "contain_conflict_resolution_audit",
+ "detected_count": len(contain_conflict_report.get("conflicts", [])),
+ "conflicts": contain_conflict_report.get("conflicts", []),
+ "pre_arbitration_removed_candidates": contain_conflict_candidate_removals,
+ "schema_guard_resolutions": [
+ contain_conflict_resolutions[conflict_id]
+ for conflict_id in sorted(contain_conflict_resolutions)
+ if any(item.get("conflict_id") == conflict_id for item in contain_conflict_report.get("conflicts", []))
+ ],
+ "applied": contain_conflict_applied,
+ "unresolved": contain_conflict_unresolved,
+ "policy": (
+ "Independent accepted attach/contain verdicts, support-inside versus contain-reject disagreements, "
+ "and accepted-attach versus high-recall-contain-reject disagreements are escalated to the existing "
+ "DeliverySchemaGuard call. Neither relation type wins by default. Generic coverage fallback runs only "
+ "after arbitration."
+ ),
+ },
+ "contain_geometry_validation": contain_geometry_validation,
+ "multi_level_support_contain_promotion": multi_level_contain_promotion,
+ "gravity_attach_normalization": {
+ "removed_floor_attach_count": len(redundant_floor_attaches),
+ "items": redundant_floor_attaches,
+ "policy": "Object support and floor support are mutually exclusive gravity paths; keep object support and remove generic floor fallback.",
+ },
+ "floor_wall_semantic_policy": {
+ "trusted_contact_overrides": list(floor_wall_policy["trusted_contact"].values()),
+ "proximity_only_vetoes": list(floor_wall_policy["proximity_only"].values()),
+ "policy": (
+ "The accepted-candidate preservation policy is unchanged: "
+ "verified flat contact plus local possible-contact evidence, or its "
+ "existing high-confidence tall-storage rear-contact exception, may "
+ "survive noisy lower-mesh edges. New missing-pair completion is handled "
+ "only by semantic_wall_contact_reconciliation."
+ ),
+ "additional_api_calls": 0,
+ "additional_geometry_or_render_passes": 0,
+ "accepted_candidate_preservation_policy_unchanged": True,
+ },
+ "semantic_wall_contact_reconciliation": semantic_wall_contact_reconciliation,
+ "flat_contact_edge_wall_attach_filter": {
+ "filtered_count": len(flat_edge_filtered_wall_attaches),
+ "items": flat_edge_filtered_wall_attaches,
+ "evaluations": flat_edge_evaluations,
+ "visual_reconstruction_overrides": list(raw_visual_reconstruction_overrides.values()),
+ "policy": "Ordinary floor-supported wall attach requires a finite wall-parallel flat contact edge from the real lower-mesh XZ hull. Consecutive collinear fragments may be merged; wrapper bbox/OBB edges and isolated point/corner contact are insufficient. An exact pair may be preserved only when the wall-attach Agent marks its decision final and supplies both visual and mask evidence for a reconstruction conflict.",
+ },
+ "surface_supported_wall_attach_filter": {
+ "filtered_count": len(filtered_surface_wall_attaches),
+ "items": filtered_surface_wall_attaches,
+ "policy": "Object-supported targets keep wall attach only with explicit high-confidence wall-mounted/embedded/hung/rack/bracket evidence.",
+ },
+ "wall_mounted_floor_attach_filter": {
+ "filtered_count": len(filtered_wall_mounted_floor_attaches),
+ "items": filtered_wall_mounted_floor_attaches,
+ "policy": "Explicitly wall-mounted objects retain their accepted wall attach and do not also receive floor attach solely for coverage.",
+ },
+ "single_wall_attach_per_target": {
+ "removed_count": len(removed_multi_wall_attaches),
+ "items": removed_multi_wall_attaches,
+ "policy": "Each object target may have at most one wall attach; floor/object support relations remain independent.",
+ },
+ "gravity_support_completion": gravity_support_completion,
+ "relation_normalization": relation_normalization_audit,
+ "ceiling_confirmed": ceiling_confirmed,
+ "blocked_wall_attach_anchors": blocked_wall_attach_anchors,
+ "wall_id_remap_applied_to_relations": wall_id_remap,
+ "coverage_repair": {
+ "missing_before": coverage_before["missing_object_targets"],
+ "missing_after": coverage_after["missing_object_targets"],
+ "used_repairs": used_repairs,
+ "skipped_candidates": coverage_skipped,
+ },
+ }
+
+
+def _geometry_by_object_wall(wall_evidence: dict[str, Any] | None) -> dict[tuple[str, str], dict[str, Any]]:
+ if not isinstance(wall_evidence, dict):
+ return {}
+ result = {}
+ for item in wall_evidence.get("wall_attach_geometry", []) if isinstance(wall_evidence.get("wall_attach_geometry"), list) else []:
+ if isinstance(item, dict) and item.get("object_id") and item.get("wall_id"):
+ result[(item["object_id"], item["wall_id"])] = item
+ return result
+
+
+def _first_non_none(*values: Any) -> Any:
+ for value in values:
+ if value is not None:
+ return value
+ return None
+
+
+def normalize_geometry_evidence(raw: Any) -> dict[str, Any]:
+ """Normalize unstable agent geometry audit fields without dropping raw evidence."""
+ normalized: dict[str, Any] = {
+ "center_to_wall_distance": None,
+ "footprint_min_distance_to_wall": None,
+ "bbox_side_contact": None,
+ "possible_against_wall_contact": None,
+ "is_floor_supported": None,
+ "geometry_classification": None,
+ "geometry_reliability": None,
+ "raw_geometry_evidence": raw,
+ "geometry_evidence_items": [],
+ }
+
+ def merge_dict(item: dict[str, Any]) -> None:
+ updates = {
+ "center_to_wall_distance": item.get("center_to_wall_distance"),
+ "footprint_min_distance_to_wall": item.get("footprint_min_distance_to_wall"),
+ "bbox_side_contact": _first_non_none(item.get("bbox_side_contact"), item.get("bbox_side_near_wall")),
+ "possible_against_wall_contact": item.get("possible_against_wall_contact"),
+ "is_floor_supported": item.get("is_floor_supported"),
+ "geometry_classification": item.get("geometry_classification"),
+ "geometry_reliability": item.get("geometry_reliability"),
+ }
+ for key, value in updates.items():
+ if normalized.get(key) is None and value is not None:
+ normalized[key] = value
+
+ if isinstance(raw, dict):
+ merge_dict(raw)
+ elif isinstance(raw, list):
+ for item in raw:
+ if isinstance(item, dict):
+ merge_dict(item)
+ elif isinstance(item, str):
+ normalized["geometry_evidence_items"].append(item)
+ elif item is not None:
+ normalized["geometry_evidence_items"].append(str(item))
+ elif isinstance(raw, str):
+ normalized["geometry_evidence_items"].append(raw)
+ elif raw is not None:
+ normalized["geometry_evidence_items"].append(str(raw))
+
+ return normalized
+
+
+def _raw_geometry_from_candidate(item: dict[str, Any], fallback: Any) -> Any:
+ for key in ["geometry_evidence", "geometry", "geometric_evidence", "geometry_evidence_items"]:
+ if key in item and item.get(key) is not None:
+ return item.get(key)
+ return fallback
+
+
+def collect_wall_attach_recall(task_results: list[dict[str, Any]], guard_result: dict[str, Any] | None = None, wall_evidence: dict[str, Any] | None = None) -> dict[str, Any]:
+ roots: list[Any] = list(task_results)
+ if guard_result:
+ roots.append(guard_result)
+ candidates: list[dict[str, Any]] = []
+ geometry_lookup = _geometry_by_object_wall(wall_evidence)
+ for root in roots:
+ for node in _walk(root):
+ if not isinstance(node, dict):
+ continue
+ for key in ["wall_attach_candidates", "wall_attach_recall", "wall_attach_decisions", "attach_decisions"]:
+ value = node.get(key)
+ if isinstance(value, list):
+ for item in value:
+ if isinstance(item, dict):
+ candidates.append(item)
+ elif isinstance(value, dict) and isinstance(value.get("candidates"), list):
+ for item in value["candidates"]:
+ if isinstance(item, dict):
+ candidates.append(item)
+ accepted = []
+ rejected = []
+ intermediate = []
+ warnings = []
+ accept_decisions = {"accept", "accepted", "true", "yes"}
+ reject_decisions = {"reject", "rejected", "false", "no"}
+ intermediate_decisions = {"needs_consistency_check", "needs_wall_resolution", "not_final_yet", "pending", "possible_wall_contact", "needs_more_evidence"}
+ for item in candidates:
+ decision = _candidate_decision(item)
+ anchor = item.get("candidate_wall_id") or item.get("wall_id") or item.get("object_anchor")
+ if not anchor and isinstance(item.get("parameters"), dict):
+ anchor = item["parameters"].get("object_anchor")
+ target = item.get("object_target") or item.get("object_id")
+ if not target and isinstance(item.get("parameters"), dict):
+ target = item["parameters"].get("object_target")
+ raw_geometry = _raw_geometry_from_candidate(item, geometry_lookup.get((target, anchor)))
+ geometry = normalize_geometry_evidence(raw_geometry)
+ visual = item.get("visual_evidence") or item.get("evidence") or item.get("evidence_used")
+ mask_evidence = item.get("mask_evidence") or item.get("mask_overlay_evidence")
+ topdown_evidence = item.get("topdown_evidence") or item.get("top_down_evidence")
+ geometry_reliability = item.get("geometry_reliability") or geometry.get("geometry_reliability") or "weak"
+ footprint_distance = geometry.get("footprint_min_distance_to_wall")
+ possible_contact = geometry.get("possible_against_wall_contact") is True or (isinstance(footprint_distance, (int, float)) and footprint_distance <= 0.03)
+ visual_suggests_contact = isinstance(visual, str) and any(token in visual.lower() for token in ["against wall", "touch", "contact", "mounted", "embedded", "wall"])
+ if decision is None and (possible_contact or visual_suggests_contact):
+ decision = "needs_consistency_check"
+ decision_is_final = bool(item.get("decision_is_final")) if "decision_is_final" in item else decision in (accept_decisions | reject_decisions)
+ if geometry_reliability == "depends_on_unstable_wall" or decision in intermediate_decisions:
+ decision_is_final = False
+ normalized = {
+ "decision": decision or None,
+ "decision_is_final": decision_is_final,
+ "object_target": target,
+ "candidate_wall_id": anchor,
+ "wall_contact_type": item.get("wall_contact_type") or geometry.get("geometry_classification"),
+ "visual_evidence": visual,
+ "mask_evidence": mask_evidence,
+ "topdown_evidence": topdown_evidence,
+ "geometry_evidence": {
+ "center_to_wall_distance": geometry.get("center_to_wall_distance"),
+ "footprint_min_distance_to_wall": geometry.get("footprint_min_distance_to_wall"),
+ "bbox_side_contact": geometry.get("bbox_side_contact"),
+ "possible_against_wall_contact": geometry.get("possible_against_wall_contact"),
+ "is_floor_supported": geometry.get("is_floor_supported"),
+ "geometry_reliability": geometry_reliability,
+ "raw_geometry_evidence": geometry.get("raw_geometry_evidence"),
+ "geometry_evidence_items": geometry.get("geometry_evidence_items", []),
+ },
+ "description_used_as": item.get("description_used_as") or "weak_support",
+ "reason": item.get("reason") or item.get("rationale"),
+ "selective_topdown_used": item.get("selective_topdown_used"),
+ "raw": item,
+ }
+ if decision in accept_decisions:
+ accepted.append(normalized)
+ elif decision in intermediate_decisions:
+ intermediate.append(normalized)
+ else:
+ reason_text = " ".join(
+ str(value or "")
+ for value in [
+ normalized.get("reason"),
+ normalized.get("description_used_as"),
+ normalized.get("wall_contact_type"),
+ normalized["geometry_evidence"].get("geometry_reliability"),
+ ]
+ ).lower()
+ weak_reject = (
+ decision in reject_decisions
+ and (
+ normalized["geometry_evidence"].get("footprint_min_distance_to_wall") is None
+ or visual_suggests_contact
+ or possible_contact
+ or any(token in reason_text for token in ["center", "distance", "description", "category", "floor", "non-final", "unstable"])
+ )
+ )
+ if weak_reject or (normalized["geometry_evidence"].get("footprint_min_distance_to_wall") is None and not visual):
+ normalized["warning"] = "weak_or_unstable_rejection_reason"
+ warnings.append({"object_target": target, "candidate_wall_id": anchor, "warning": "weak_or_unstable_rejection_reason"})
+ rejected.append(normalized)
+ return {"candidate_count": len(candidates), "accepted": accepted, "rejected": rejected, "intermediate": intermediate, "warnings": warnings}
+
+
+def write_accepted_attach_merge_check(
+ output_dir: Path,
+ accepted_candidates: list[dict[str, Any]],
+ final_relations: list[dict[str, Any]],
+ final_outputs_blocked_reason: str | None = None,
+ skipped_candidates: list[dict[str, Any]] | None = None,
+ wall_id_remap: dict[str, str] | None = None,
+) -> dict[str, Any]:
+ wall_id_remap = wall_id_remap or {}
+
+ def remapped_key(rel: dict[str, Any]) -> tuple[Any, Any, Any]:
+ params = rel.get("parameters", {}) if isinstance(rel.get("parameters"), dict) else {}
+ anchor = params.get("object_anchor")
+ target = params.get("object_target")
+ if anchor in wall_id_remap:
+ anchor = wall_id_remap[anchor]
+ if target in wall_id_remap:
+ target = wall_id_remap[target]
+ return (rel.get("relation"), anchor, target)
+
+ # Final relations have already been canonicalized during collect_relations;
+ # only accepted candidate keys need remapping for comparison.
+ final_keys = {
+ (rel.get("relation"), rel.get("parameters", {}).get("object_anchor"), rel.get("parameters", {}).get("object_target"))
+ for rel in final_relations
+ if isinstance(rel, dict)
+ }
+ policy_filtered_keys: dict[tuple[Any, Any, Any], dict[str, Any]] = {}
+ for skipped in skipped_candidates or []:
+ if not isinstance(skipped, dict):
+ continue
+ if skipped.get("reason") not in {
+ "relation_type_not_allowed",
+ "object_target_not_in_final_objects",
+ "object_anchor_not_allowed",
+ "contain_anchor_must_be_object",
+ "contain_anchor_target_same_object",
+ "weak_or_inferred_wall_anchor_not_allowed_for_attach",
+ "surface_supported_object_wall_attach_filtered",
+ "floor_supported_wall_attach_without_valid_flat_contact_edge",
+ "schema_guard_attach_contain_arbitration",
+ "unresolved_attach_contain_same_pair_requires_schema_guard",
+ "contain_target_clearly_outside_anchor_obb",
+ "duplicate_relation",
+ "multiple_wall_attach_same_target_deduped",
+ "target_already_covered",
+ "floor_covering_anchor_normalized_to_floor",
+ "duplicate_after_floor_covering_normalization_dropped",
+ }:
+ continue
+ rel = skipped.get("relation") or skipped.get("relation_candidate")
+ if not rel and isinstance(skipped.get("raw_candidate"), dict):
+ rel = _relation_from_candidate(skipped["raw_candidate"])
+ if not rel and isinstance(skipped.get("candidate"), dict):
+ rel = _relation_from_candidate(skipped["candidate"])
+ if isinstance(rel, dict):
+ key = remapped_key(rel)
+ policy_filtered_keys[key] = skipped
+ items = []
+ errors = []
+ for candidate in accepted_candidates:
+ rel = candidate.get("relation") or candidate.get("relation_candidate") or candidate.get("relation")
+ if not rel and isinstance(candidate.get("raw_candidate"), dict):
+ rel = _relation_from_candidate(candidate["raw_candidate"])
+ if not rel and isinstance(candidate.get("relation"), dict):
+ rel = candidate["relation"]
+ if isinstance(rel, dict):
+ source_path = str(candidate.get("source_path") or "")
+ source_path_l = source_path.lower()
+ # Schema/supervisor roots may provide a whole relations_json snapshot.
+ # That snapshot is useful as a merge hint, but it is not an original
+ # task-level accepted candidate. Treating it as a hard requirement can
+ # falsely fail finalization when local canonical wall remapping or
+ # policy filters choose the authoritative final relation set.
+ if source_path_l.startswith("root_") and source_path_l.endswith(".relations_json"):
+ items.append({
+ "accepted_candidate": rel,
+ "status": "schema_snapshot_not_hard_checked",
+ "source_path": source_path,
+ "severity": "informational",
+ })
+ continue
+ key = remapped_key(rel)
+ if key in final_keys:
+ status = "present_in_final_relations"
+ elif key in policy_filtered_keys:
+ status = "intentionally_filtered_by_policy"
+ elif final_outputs_blocked_reason:
+ status = "blocked_by_missing_final_outputs"
+ else:
+ status = "missing_from_final_relations"
+ item = {"accepted_candidate": rel, "status": status, "source_path": candidate.get("source_path")}
+ if status == "intentionally_filtered_by_policy":
+ item["policy_filter_reason"] = policy_filtered_keys[key].get("reason")
+ item["blocked_wall_anchor_reason"] = policy_filtered_keys[key].get("blocked_wall_anchor_reason")
+ item["policy"] = policy_filtered_keys[key].get("policy")
+ item["severity"] = "policy_filtered"
+ item["note"] = "Accepted candidate was intentionally skipped by local merge policy and is not required in final relations."
+ elif final_outputs_blocked_reason and status == "blocked_by_missing_final_outputs":
+ item["blocked_reason"] = final_outputs_blocked_reason
+ item["severity"] = "blocked"
+ elif status != "present_in_final_relations":
+ item["severity"] = "error"
+ errors.append(item)
+ items.append(item)
+ data = {
+ "valid": not errors,
+ "items": items,
+ "errors": errors,
+ "final_outputs_blocked_reason": final_outputs_blocked_reason,
+ "wall_id_remap_used_for_check": wall_id_remap,
+ }
+ write_audit_pair(output_dir, "attach", "accepted_attach_merge_check", data, "Accepted Attach Merge Check")
+ return data
+
+
+def write_attach_audit(
+ output_dir: Path,
+ task_results: list[dict[str, Any]],
+ guard_result: dict[str, Any] | None,
+ relations: list[dict[str, Any]],
+ merge_report: dict[str, Any],
+ dry_run: bool,
+ wall_evidence: dict[str, Any] | None = None,
+ final_outputs_blocked_reason: str | None = None,
+) -> dict[str, Any]:
+ recall = collect_wall_attach_recall(task_results, guard_result, wall_evidence=wall_evidence)
+ wall_relations = [rel for rel in relations if rel.get("relation") == "attach" and str(rel.get("parameters", {}).get("object_anchor", "")).startswith("wall_")]
+ merge_check = write_accepted_attach_merge_check(
+ output_dir,
+ merge_report.get("accepted_candidates", []),
+ relations,
+ final_outputs_blocked_reason=final_outputs_blocked_reason,
+ skipped_candidates=merge_report.get("skipped_candidates", []),
+ wall_id_remap=merge_report.get("wall_id_remap_applied_to_relations", {}),
+ )
+ data = {
+ "dry_run": dry_run,
+ "status": "completed" if merge_check.get("valid") else "failed",
+ "wall_attach_candidate_recall": recall,
+ "accepted_wall_attach_relations": wall_relations,
+ "wall_attach_relation_count": len(wall_relations),
+ "accepted_attach_merge": merge_check,
+ "skipped_candidates": merge_report.get("skipped_candidates", []),
+ }
+ write_audit_pair(output_dir, "attach", "attach_decision_audit", data, "Attach Decision Audit")
+ return merge_check
+
+
+def write_partial_output(output_dir: Path, name: str, payload: Any) -> Path:
+ path = output_dir / "debug" / "partial_outputs" / name
+ write_json(path, payload)
+ return path
+
+
+def apply_simulation_physics(
+ objects: list[dict[str, Any]],
+ output_dir: Path,
+ dry_run: bool = False,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ sidecar_path = output_dir / "physics_sidecar" / "integrated_async" / "object_physics.json"
+ agent_payload: dict[str, Any] = {}
+ source = "local_fallback_sidecar_not_ready"
+ read_error = None
+ if sidecar_path.exists():
+ try:
+ loaded = json.loads(sidecar_path.read_text(encoding="utf-8"))
+ if isinstance(loaded, dict):
+ agent_payload = loaded
+ source = "async_physics_sidecar"
+ else:
+ read_error = "object_physics.json top level is not an object"
+ except (OSError, json.JSONDecodeError) as exc:
+ read_error = str(exc)
+
+ base_payload = {"walls": [], "objects": objects}
+ sidecar, enriched, audit = resolve_object_physics(base_payload, agent_payload)
+ expected_ids = [str(obj.get("id")) for obj in objects]
+ validation_errors = validate_object_physics(sidecar, expected_ids)
+ if validation_errors:
+ sidecar, enriched, audit = resolve_object_physics(base_payload, {})
+ fallback_errors = validate_object_physics(sidecar, expected_ids)
+ source = "local_fallback_invalid_sidecar"
+ validation_errors.extend(fallback_errors)
+
+ # apply_upright_properties has already combined the Sidecar prior with
+ # relation-aware wall, ceiling, and floor-covering overrides. Physics
+ # enrichment may add numeric fields, but must not replace that final value.
+ final_upright_by_id = {
+ str(obj.get("id")): obj.get("upright")
+ for obj in objects
+ if isinstance(obj, dict) and isinstance(obj.get("upright"), bool)
+ }
+ for obj in enriched.get("objects", []):
+ object_id = str(obj.get("id") or "")
+ if object_id in final_upright_by_id:
+ obj["upright"] = final_upright_by_id[object_id]
+
+ audit.update({
+ "integration_source": source,
+ "sidecar_path": str(sidecar_path),
+ "sidecar_available_at_final_assembly": sidecar_path.exists(),
+ "relation_flow_waited_for_sidecar": False,
+ "relation_aware_upright_preserved": True,
+ "read_error": read_error,
+ "validation_errors": validation_errors,
+ })
+ write_audit_pair(
+ output_dir,
+ "objects",
+ "simulation_physics_audit",
+ audit,
+ "Simulation Physics Audit",
+ )
+ if not dry_run:
+ write_json(output_dir / "debug" / "objects" / "resolved_object_physics.json", sidecar)
+ return enriched["objects"], audit
+
+
+def assemble_final_payloads(
+ scene_state: dict[str, Any],
+ task_results: list[dict[str, Any]],
+ guard_result: dict[str, Any] | None,
+ output_dir: Path,
+ dry_run: bool = False,
+ wall_evidence: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+ base_objects = candidate_objects(scene_state)
+ patches, patch_warnings = collect_object_revision_patches(task_results, guard_result)
+ final_objects, applied_revisions, apply_warnings = apply_object_revision_patches(base_objects, patches)
+ object_warnings = patch_warnings + apply_warnings
+ write_object_revision_audit(output_dir, applied_revisions, object_warnings, dry_run=dry_run)
+
+ walls, wall_warnings, wall_records = collect_walls(task_results, guard_result, wall_evidence=wall_evidence)
+ room_materials_audit = collect_room_materials(task_results, guard_result)
+ room_materials_audit["dry_run"] = dry_run
+ write_audit_pair(output_dir, "objects", "room_materials_audit", room_materials_audit, "Room Materials Audit")
+
+ wall_final_adjudication = summarize_wall_final_adjudication(task_results, guard_result, accepted_walls=walls)
+ walls, wall_records, authoritative_wall_geometry_gate = _authoritative_obb_walls_from_evidence(walls, wall_records, wall_evidence)
+ if authoritative_wall_geometry_gate.get("applied"):
+ wall_warnings.append("Final wall coordinates were overridden by local obb_from_mesh_footprint_min_rect authoritative geometry gate.")
+ wall_final_adjudication["accepted_final_walls"] = True
+ wall_final_adjudication["coordinate_source_overridden_by"] = FORMAL_OBB_WALL_METHOD
+ else:
+ wall_warnings.append(f"Formal OBB wall geometry gate did not apply: {authoritative_wall_geometry_gate.get('reason')}.")
+ walls, wall_records, wall_canonicalization_gate = canonicalize_walls_for_screen_clockwise(walls, wall_records, wall_evidence)
+ if wall_canonicalization_gate.get("changed_wall_ids"):
+ wall_warnings.append("Final wall ids were remapped by the local screen-clockwise canonicalization gate; relation anchors are remapped during merge.")
+ wall_final_semantics = collect_wall_final_semantics(task_results, guard_result)
+ wall_semantic_remap = (
+ wall_canonicalization_gate.get("old_to_new_wall_id", {})
+ if isinstance(wall_canonicalization_gate, dict)
+ else {}
+ )
+ wall_records, wall_semantic_overlay_gate = apply_wall_final_semantics(
+ wall_records,
+ wall_final_semantics,
+ wall_semantic_remap,
+ )
+ authoritative_wall_geometry_gate["semantic_metadata_overlay"] = wall_semantic_overlay_gate
+ wall_geometry = write_wall_audit(
+ output_dir,
+ walls,
+ wall_records,
+ wall_warnings,
+ dry_run=dry_run,
+ wall_final_adjudication=wall_final_adjudication,
+ wall_canonicalization_gate=wall_canonicalization_gate,
+ authoritative_wall_geometry_gate=authoritative_wall_geometry_gate,
+ )
+ wall_blocked_reason = None
+ if not walls or len(walls) != 4 or not wall_geometry.get("valid"):
+ wall_blocked_reason = "missing_or_invalid_final_eligible_walls"
+ object_ids = [obj["id"] for obj in final_objects]
+ blocked_wall_attach_anchors = weak_or_inferred_wall_attach_blocks(wall_records)
+ relation_wall_id_remap = build_wall_relation_id_remap(
+ wall_records,
+ wall_canonicalization_gate if isinstance(wall_canonicalization_gate, dict) else None,
+ )
+ object_physical_properties = collect_object_physical_properties(task_results, guard_result, final_objects, scene_state)
+ write_object_physical_properties_audit(output_dir, object_physical_properties, dry_run=dry_run)
+ relations, merge_report = collect_relations(
+ guard_result,
+ task_results,
+ object_ids,
+ blocked_wall_attach_anchors=blocked_wall_attach_anchors,
+ wall_id_remap=relation_wall_id_remap,
+ objects=final_objects,
+ object_properties=object_physical_properties,
+ wall_evidence=wall_evidence,
+ scene_state=scene_state,
+ )
+ contain_geometry_audit = dict(merge_report.get("contain_geometry_validation", {}))
+ contain_geometry_audit["dry_run"] = dry_run
+ write_audit_pair(output_dir, "relations", "contain_geometry_audit", contain_geometry_audit, "Contain Geometry Audit")
+ contain_conflict_audit = dict(merge_report.get("contain_conflict_resolution", {}))
+ contain_conflict_audit["dry_run"] = dry_run
+ write_audit_pair(output_dir, "relations", "contain_conflict_resolution_audit", contain_conflict_audit, "Contain Conflict Resolution Audit")
+ relation_normalization_audit = write_relation_normalization_audit(output_dir, merge_report.get("relation_normalization", {}), dry_run=dry_run)
+ flat_contact_edge_wall_attach_audit = write_flat_contact_edge_wall_attach_audit(
+ output_dir,
+ merge_report,
+ dry_run=dry_run,
+ )
+ semantic_wall_contact_reconciliation_audit = (
+ write_semantic_wall_contact_reconciliation_audit(
+ output_dir,
+ merge_report,
+ dry_run=dry_run,
+ )
+ )
+ sidecar_upright_priors = load_async_sidecar_upright_priors(output_dir, object_ids)
+ final_objects, upright_audit = apply_upright_properties(
+ final_objects,
+ relations,
+ object_physical_properties,
+ scene_state,
+ output_dir,
+ dry_run=dry_run,
+ upright_priors=sidecar_upright_priors,
+ )
+ final_objects, contain_anchor_audit = apply_contain_anchor_properties(
+ final_objects,
+ relations,
+ task_results,
+ guard_result,
+ scene_state,
+ output_dir,
+ dry_run=dry_run,
+ )
+ final_objects, simulation_physics_audit = apply_simulation_physics(final_objects, output_dir, dry_run=dry_run)
+ gravity_support_completion_audit = write_gravity_support_completion_audit(
+ output_dir,
+ merge_report.get("gravity_support_completion", {}),
+ dry_run=dry_run,
+ )
+ coverage_repair_audit = write_coverage_repair_audit(output_dir, merge_report.get("coverage_repair", {}), dry_run=dry_run)
+ coverage_missing = merge_report.get("coverage_repair", {}).get("missing_after", [])
+ merge_check = write_attach_audit(
+ output_dir,
+ task_results,
+ guard_result,
+ relations,
+ merge_report,
+ dry_run=dry_run,
+ wall_evidence=wall_evidence,
+ final_outputs_blocked_reason=wall_blocked_reason,
+ )
+
+ # This is deliberately the final geometry postprocess: all Agent decisions,
+ # relation normalization, coverage repair, and merge auditing are complete.
+ walls, wall_records, virtual_wall_shift_audit = shift_walls_from_attach_policy(
+ walls,
+ wall_records,
+ merge_report,
+ relations=relations,
+ offset_ratio=0.05,
+ )
+ if virtual_wall_shift_audit.get("status") == "applied":
+ wall_warnings.append(
+ "Only unattached weak/inferred/closure-only wall support lines were moved outward after final relation assembly."
+ )
+ wall_final_adjudication["coordinate_postprocess"] = "wall_attach_policy_outward_shift"
+ wall_geometry = write_wall_audit(
+ output_dir,
+ walls,
+ wall_records,
+ wall_warnings,
+ dry_run=dry_run,
+ wall_final_adjudication=wall_final_adjudication,
+ wall_canonicalization_gate=wall_canonicalization_gate,
+ authoritative_wall_geometry_gate=authoritative_wall_geometry_gate,
+ )
+ if not wall_geometry.get("valid"):
+ wall_blocked_reason = "invalid_walls_after_wall_attach_policy_shift"
+ weak_wall_attach_audit = write_weak_wall_attach_audit(
+ output_dir,
+ merge_report,
+ dry_run=dry_run,
+ virtual_wall_shift_audit=virtual_wall_shift_audit,
+ )
+ partial_objects_payload = {
+ "walls": walls,
+ "room_materials": room_materials_audit.get("room_materials") or {},
+ "objects": final_objects,
+ }
+
+ errors: list[str] = []
+ repair_tasks: list[dict[str, Any]] = []
+ if room_materials_audit.get("status") != "ready":
+ errors.append("Final objects cannot be written because room_materials is missing or invalid.")
+ repair_tasks.append(
+ {
+ "task_type": "room_material_classification",
+ "reason": "objects.json requires image-grounded room_materials with non-empty walls, floor, and ceiling strings",
+ "inputs": ["original_image_path"],
+ "output_schema": {
+ "room_materials": {
+ "walls": "concise visible wall material/finish",
+ "floor": "concise visible floor material/finish",
+ "ceiling": "concise visible ceiling material/finish",
+ }
+ },
+ }
+ )
+ if wall_blocked_reason:
+ errors.append("Final objects cannot be written because wall_001..wall_004 are missing, geometrically invalid, or lack final-eligible layout XZ source with topdown verification.")
+ repair_tasks.append(
+ {
+ "task_type": "wall_final_adjudication",
+ "reason": "objects.json requires final adjudication: wall_001..wall_004 must be verified layout XZ coordinates before finalization",
+ "original_repair_task_type": "infer_four_clockwise_walls_from_topdown",
+ "inputs": [
+ "wall_evidence.visual_wall_extraction",
+ "canonical_topdown.png",
+ "canonical_topdown_grid_mapping.json",
+ "wall_evidence.object_footprints",
+ ],
+ "output_schema": {
+ "walls": [
+ {
+ "id": "wall_001",
+ "first_point": ["x", "z"],
+ "second_point": ["x", "z"],
+ "source": "obb_from_mesh_footprint_min_rect",
+ "evidence": "layout XZ coordinate evidence verified by topdown/fixed views",
+ "topdown_verification": "passed",
+ }
+ ]
+ },
+ }
+ )
+ if coverage_missing:
+ errors.append(f"Final relations are missing attach/contain target coverage for objects: {coverage_missing}.")
+ repair_tasks.append(
+ {
+ "task_type": "repair_missing_relation_targets",
+ "missing_object_targets": coverage_missing,
+ "instruction": "Choose the most plausible physical anchor or container for each missing object target. Prefer local physical contact or interior containment over generic floor fallback.",
+ "inputs": [
+ "missing_object_targets",
+ "scene_state.object_candidates",
+ "current_accepted_relations",
+ "candidate_anchor_objects",
+ "original_image_path",
+ "mask_overlay_path",
+ "canonical_topdown",
+ "fixed_layout_views_manifest",
+ "layout_bbox_or_footprint_evidence",
+ "selective_render_tool_capability",
+ ],
+ "output_schema": {
+ "coverage_repair_relations": [
+ {
+ "relation": "attach",
+ "parameters": {"object_anchor": "...", "object_target": "object_xxx"},
+ "repair_reason": "physical anchor reasoning",
+ "evidence_used": ["mask_overlay", "layout_bbox", "fixed_layout_views"],
+ "coverage_repair": True,
+ }
+ ]
+ },
+ }
+ )
+ if not merge_check.get("valid"):
+ errors.append("Accepted attach candidates were missing from final relations after merge.")
+
+ partial_path = None
+ if errors:
+ partial_path = write_partial_output(output_dir, "objects_or_relations_validation_failed.json", {"objects": partial_objects_payload, "relations": relations})
+
+ if repair_tasks:
+ write_json(output_dir / "debug" / "agents" / "schema_repair_tasks.json", {"tasks": repair_tasks, "errors": errors})
+ write_md(
+ output_dir / "debug" / "agents" / "schema_repair_tasks.md",
+ "Schema Repair Tasks",
+ ["```json", json.dumps({"tasks": repair_tasks, "errors": errors}, indent=2, ensure_ascii=False), "```"],
+ )
+
+ objects_payload = None if errors else partial_objects_payload
+ objects_path = output_dir / "final_outputs" / "objects.json"
+ relations_path = output_dir / "final_outputs" / "relations.json"
+ final_objects_written = False
+ final_relations_written = False
+ if not errors:
+ write_json(objects_path, partial_objects_payload)
+ write_json(relations_path, relations)
+ final_objects_written = True
+ final_relations_written = True
+
+ return {
+ "objects_payload": objects_payload,
+ "relations_payload": relations if not errors else None,
+ "objects_path": str(objects_path) if final_objects_written else None,
+ "relations_path": str(relations_path) if final_relations_written else None,
+ "partial_objects_payload": partial_objects_payload,
+ "simulation_physics_audit": simulation_physics_audit,
+ "partial_objects_path": str(partial_path) if partial_path else None,
+ "object_revision_warnings": object_warnings,
+ "applied_object_revisions": applied_revisions,
+ "wall_geometry": wall_geometry,
+ "wall_source_records": wall_records,
+ "wall_final_adjudication": wall_final_adjudication,
+ "wall_errors": wall_geometry.get("errors", []),
+ "attach_merge_check": merge_check,
+ "ceiling_confirmed": merge_report.get("ceiling_confirmed"),
+ "coverage_repair_audit": coverage_repair_audit,
+ "semantic_wall_contact_reconciliation_audit": (
+ semantic_wall_contact_reconciliation_audit
+ ),
+ "gravity_support_completion_audit": gravity_support_completion_audit,
+ "relation_normalization_audit": relation_normalization_audit,
+ "weak_wall_attach_audit": weak_wall_attach_audit,
+ "virtual_wall_shift_audit": virtual_wall_shift_audit,
+ "upright_audit": upright_audit,
+ "contain_anchor_audit": contain_anchor_audit,
+ "contain_geometry_audit": contain_geometry_audit,
+ "contain_conflict_audit": contain_conflict_audit,
+ "object_physical_properties": object_physical_properties,
+ "room_materials_audit": room_materials_audit,
+ "missing_object_targets": coverage_missing,
+ "errors": errors,
+ "repair_tasks": repair_tasks,
+ "final_objects_written": final_objects_written,
+ "final_relations_written": final_relations_written,
+ }
diff --git a/vlm_agent/reconstruction/relations/agents/orchestration/__init__.py b/vlm_agent/reconstruction/relations/agents/orchestration/__init__.py
new file mode 100644
index 0000000..9b51c6e
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/orchestration/__init__.py
@@ -0,0 +1 @@
+"""Agent planning and delivery orchestration helpers."""
diff --git a/vlm_agent/reconstruction/relations/agents/orchestration/delivery_io.py b/vlm_agent/reconstruction/relations/agents/orchestration/delivery_io.py
new file mode 100644
index 0000000..d8d196a
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/orchestration/delivery_io.py
@@ -0,0 +1,311 @@
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shutil
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+from relations.agents.state import write_audit_pair, write_json, write_md
+
+REPOSITORY_ROOT = Path(__file__).resolve().parents[5]
+if str(REPOSITORY_ROOT) not in sys.path:
+ sys.path.insert(0, str(REPOSITORY_ROOT))
+
+from vlm_agent.run_paths import ScenePaths
+
+RELATIONS_ROOT = Path(__file__).resolve().parents[2]
+PROMPT_DIR = RELATIONS_ROOT / "prompts" / "multi_agent"
+POLICY_PATH = RELATIONS_ROOT / "policies" / "multi_agent_policy.md"
+RENDER_GRID_WALL_PRECOMPUTE_PATH_ENV = "RENDER_GRID_WALL_PRECOMPUTE_PATH"
+RELATION_LAYOUT_DIR_ENV = "RELATION_LAYOUT_DIR"
+
+
+def configure_wall_candidate_environment(args: argparse.Namespace) -> dict[str, Any]:
+ mode = str(getattr(args, "wall_candidate_mode", "render_grid_only") or "render_grid_only").strip().lower()
+ if mode != "render_grid_only":
+ raise ValueError(f"unsupported wall candidate mode for the formal pipeline: {mode}")
+ return {
+ "wall_candidate_mode": mode,
+ "render_grid_wall_precompute_path_env": RENDER_GRID_WALL_PRECOMPUTE_PATH_ENV,
+ }
+
+def load_prompt(name: str) -> str:
+ return (PROMPT_DIR / name).read_text(encoding="utf-8")
+
+
+def write_manager_plan(output_dir: Path, plan: dict[str, Any]) -> None:
+ write_json(output_dir / "manager_plans" / "manager_plan_000.json", plan)
+ lines = [
+ f"- dry_run: `{plan.get('dry_run', False)}`",
+ f"- objective: {plan.get('objective', '')}",
+ f"- task_count: `{len(plan.get('tasks', []))}`",
+ "",
+ "## Tasks",
+ ]
+ for task in plan.get("tasks", []):
+ lines.append(f"- `{task.get('task_id')}`: {task.get('task_type')} ({task.get('agent_role')})")
+ write_md(output_dir / "manager_plans" / "manager_plan_000.md", "Manager Plan 000", lines)
+
+
+def write_task_output(output_dir: Path, task: dict[str, Any], response: dict[str, Any]) -> None:
+ task_dir = output_dir / "agent_outputs" / task.get("task_id", "task_0001")
+ task_dir.mkdir(parents=True, exist_ok=True)
+ (task_dir / "prompt.txt").write_text(json.dumps(task, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
+ write_json(task_dir / "response.json", response)
+ lines = [
+ f"- task_id: `{response.get('task_id', task.get('task_id'))}`",
+ f"- status: `{response.get('status')}`",
+ f"- dry_run: `{response.get('dry_run', False)}`",
+ "",
+ "## Findings",
+ ]
+ for item in response.get("findings", []):
+ lines.append(f"- {item}")
+ write_md(task_dir / "response.md", "Dynamic Task Response", lines)
+
+
+def load_static_policy() -> dict[str, Any]:
+ if not POLICY_PATH.exists():
+ raise FileNotFoundError(f"production multi-agent policy is missing: {POLICY_PATH}")
+ return {
+ "status": "static_production_policy",
+ "output_path": str(POLICY_PATH),
+ "source": "relations/policies/multi_agent_policy.md",
+ }
+
+
+def copy_policy_to_run(output_dir: Path) -> None:
+ target = output_dir / "policy" / "multi_agent_policy.md"
+ target.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copyfile(POLICY_PATH, target)
+
+
+def write_dry_run_final_marker(output_dir: Path) -> None:
+ marker = output_dir / "final_outputs" / "DRY_RUN_NO_FINAL_OUTPUTS.md"
+ marker.write_text(
+ "# Dry Run Final Outputs\n\n"
+ "No real `objects.json` or `relations.json` was generated because `--dry_run` was enabled.\n"
+ "Remove `--dry_run` and provide a valid API key environment variable to request real agent output.\n"
+ "Final real outputs must preserve exactly the object ids from scene_state.object_candidates; object numbering is inherited from SAM3D/layout and may be 0-based or 1-based.\n",
+ encoding="utf-8",
+ )
+
+
+def update_agent_audits(output_dir: Path, plan: dict[str, Any], task_results: list[dict[str, Any]], supervisor_result: dict[str, Any]) -> None:
+ write_audit_pair(
+ output_dir,
+ "agents",
+ "agent_task_log",
+ {
+ "dry_run": plan.get("dry_run", False),
+ "manager_plan_id": plan.get("plan_id"),
+ "tasks": plan.get("tasks", []),
+ "task_results": task_results,
+ },
+ "Agent Task Log",
+ )
+ write_audit_pair(output_dir, "supervisor", "supervisor_log", supervisor_result, "Supervisor Log")
+
+
+def write_render_audit(output_dir: Path, render_capabilities: dict[str, Any], dry_run: bool) -> None:
+ write_audit_pair(
+ output_dir,
+ "rendering",
+ "topdown_render_log",
+ {
+ "dry_run": dry_run,
+ "status": "capability_scan_completed",
+ "render_capabilities": render_capabilities,
+ "render_calls": [],
+ },
+ "Topdown Render Log",
+ )
+
+
+def _load_json_file(path: Path | None) -> dict[str, Any] | None:
+ if not path or not path.exists():
+ return None
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except Exception:
+ return None
+ return data if isinstance(data, dict) else None
+
+
+def relation_layout_manifest_candidates(run_root: Path | None) -> list[Path]:
+ explicit_dir = os.environ.get(RELATION_LAYOUT_DIR_ENV)
+ if explicit_dir:
+ return [Path(explicit_dir).expanduser().resolve() / "manifest.json"]
+ if not run_root:
+ return []
+ default_dir = ScenePaths.from_scene_root(run_root).relations_dir / "relation_layout"
+ return [default_dir / "manifest.json"]
+
+
+def relation_layout_expected_manifest_path(run_root: Path | None) -> Path | None:
+ explicit_dir = os.environ.get(RELATION_LAYOUT_DIR_ENV)
+ if explicit_dir:
+ return Path(explicit_dir).expanduser().resolve() / "expected_manifest.json"
+ if not run_root:
+ return None
+ default_dir = ScenePaths.from_scene_root(run_root).relations_dir / "relation_layout"
+ return default_dir / "expected_manifest.json"
+
+
+def load_relation_layout_manifest(run_root: Path | None) -> dict[str, Any] | None:
+ for manifest_path in relation_layout_manifest_candidates(run_root):
+ manifest = _load_json_file(manifest_path)
+ if isinstance(manifest, dict):
+ manifest = dict(manifest)
+ manifest["manifest_path"] = str(manifest_path)
+ manifest["manifest_source"] = "relation_layout"
+ return manifest
+ return None
+
+
+def _relation_precompute_outputs_from_run_root(run_root: Path | None) -> dict[str, Any]:
+ manifest = load_relation_layout_manifest(run_root)
+ if not isinstance(manifest, dict):
+ return {}
+ outputs = manifest.get("outputs")
+ return outputs if isinstance(outputs, dict) else {}
+
+
+def wait_for_relation_precompute(args: argparse.Namespace, *, poll_seconds: float = 1.0) -> dict[str, Any] | None:
+ wait_seconds = float(getattr(args, "relation_precompute_wait_seconds", 0) or 0)
+ if wait_seconds <= 0:
+ return load_render_grid_wall_precompute(args)
+ deadline = time.time() + wait_seconds
+ latest: dict[str, Any] | None = None
+ while time.time() <= deadline:
+ latest = load_render_grid_wall_precompute(args)
+ if isinstance(latest, dict) and latest.get("status") == "available":
+ return latest
+ time.sleep(max(0.1, poll_seconds))
+ return latest
+
+
+def load_render_grid_wall_precompute(args: argparse.Namespace) -> dict[str, Any] | None:
+ raw_path = getattr(args, "render_grid_wall_precompute_path", None)
+ raw_dir = getattr(args, "render_grid_wall_precompute_dir", None)
+ run_root = Path(getattr(args, "run_root", "")).resolve() if getattr(args, "run_root", None) else None
+ relation_outputs = _relation_precompute_outputs_from_run_root(run_root)
+ precompute_dir: Path | None = None
+ method_candidates_path: Path | None = None
+ if raw_path:
+ path = Path(raw_path).resolve()
+ if path.is_dir():
+ precompute_dir = path
+ method_candidates_path = path / "method_candidates.json"
+ else:
+ method_candidates_path = path
+ precompute_dir = path.parent
+ elif raw_dir:
+ precompute_dir = Path(raw_dir).resolve()
+ method_candidates_path = precompute_dir / "method_candidates.json"
+ elif relation_outputs.get("method_candidates_path"):
+ method_candidates_path = Path(str(relation_outputs["method_candidates_path"])).resolve()
+ precompute_dir = Path(str(relation_outputs.get("render_grid_wall_precompute_dir") or method_candidates_path.parent)).resolve()
+ if not method_candidates_path:
+ return None
+
+ manifest_path = precompute_dir / "render_grid_wall_precompute_manifest.json" if precompute_dir else None
+ manifest = _load_json_file(manifest_path) or {}
+ method_data = _load_json_file(method_candidates_path)
+ if not isinstance(method_data, dict) or not isinstance(method_data.get("candidates"), list):
+ return {
+ "status": "missing",
+ "reason": f"method_candidates.json missing or invalid: {method_candidates_path}",
+ "precompute_dir": str(precompute_dir) if precompute_dir else None,
+ "method_candidates_path": str(method_candidates_path),
+ "manifest_path": str(manifest_path) if manifest_path else None,
+ "loaded_from_external_precompute": True,
+ }
+
+ def pick(key: str, filename: str | None = None) -> str | None:
+ value = manifest.get(key)
+ if value:
+ return str(value)
+ if precompute_dir and filename:
+ path = precompute_dir / filename
+ return str(path) if path.exists() else str(path)
+ return None
+
+ return {
+ "status": "available",
+ "scene_name": manifest.get("scene_name") or method_data.get("scene"),
+ "precompute_dir": str(precompute_dir) if precompute_dir else None,
+ "method_candidates_path": str(method_candidates_path),
+ "manifest_path": str(manifest_path) if manifest_path else None,
+ "canonical_topdown_image_path": pick("canonical_topdown_image_path", "canonical_topdown.png"),
+ "canonical_mapping_path": pick("canonical_mapping_path", "canonical_topdown_grid_mapping.json"),
+ "render_grid_affine_path": pick("render_grid_affine_path", "render_grid_affine.json"),
+ "render_foreground_mask_path": pick("render_foreground_mask_path", "render_foreground_mask.png"),
+ "candidate_overlay_path": pick("candidate_overlay_path", "candidate_overlay.png"),
+ "wall_orientation_candidates_overlay_path": pick("wall_orientation_candidates_overlay_path", "wall_orientation_candidates_overlay.png"),
+ "primary_candidate_overlay_path": pick("primary_candidate_overlay_path", "primary_candidate_overlay.png"),
+ "primary_method": manifest.get("primary_method") or method_data.get("method_policy", {}).get("primary_method"),
+ "supporting_methods": manifest.get("supporting_methods") or method_data.get("method_policy", {}).get("supporting_methods"),
+ "same_grid_policy": manifest.get("same_grid_policy") or method_data.get("renderer", {}).get("same_grid_policy"),
+ "loaded_from_external_precompute": True,
+ "reused_existing": True,
+ "relation_precompute_manifest_path": (load_relation_layout_manifest(run_root) or {}).get("manifest_path") if run_root else None,
+ "loaded_from_relation_precompute": bool(relation_outputs),
+ }
+
+
+def apply_render_grid_precompute_to_scene_state(scene_state: dict[str, Any], precompute: dict[str, Any] | None) -> dict[str, Any]:
+ if not isinstance(precompute, dict) or precompute.get("status") not in {"available", "missing_primary"}:
+ return scene_state
+ method_candidates_path = precompute.get("method_candidates_path")
+ canonical_mapping = precompute.get("canonical_mapping_path")
+ canonical_topdown = precompute.get("canonical_topdown_image_path")
+ if method_candidates_path:
+ os.environ[RENDER_GRID_WALL_PRECOMPUTE_PATH_ENV] = str(method_candidates_path)
+ if canonical_mapping:
+ scene_state["topdown_grid_mapping"] = {
+ "canonical_mapping_path": str(canonical_mapping),
+ "canonical_topdown_image_path": str(canonical_topdown) if canonical_topdown else None,
+ "source": "render_grid_wall_precompute",
+ "fallback_used": False,
+ "status": "available",
+ "render_grid_affine_path": precompute.get("render_grid_affine_path"),
+ "method_candidates_path": method_candidates_path,
+ }
+ scene_state.setdefault("evidence", {})["render_grid_wall_precompute"] = precompute
+ return scene_state
+
+
+def fixed_views_precompute_dir_arg(args: argparse.Namespace) -> Path | None:
+ explicit = getattr(args, "fixed_views_precompute_dir", None)
+ if explicit:
+ return Path(explicit).resolve()
+ run_root = Path(getattr(args, "run_root", "")).resolve() if getattr(args, "run_root", None) else None
+ relation_outputs = _relation_precompute_outputs_from_run_root(run_root)
+ fixed_dir = relation_outputs.get("fixed_views_dir") if isinstance(relation_outputs, dict) else None
+ fixed_manifest = relation_outputs.get("fixed_views_manifest_path") if isinstance(relation_outputs, dict) else None
+ if fixed_dir and fixed_manifest and Path(str(fixed_manifest)).exists():
+ return Path(str(fixed_dir)).resolve()
+ return None
+
+
+
+
+def wait_for_fixed_views_precompute_dir(args: argparse.Namespace, *, poll_seconds: float = 1.0) -> Path | None:
+ initial = fixed_views_precompute_dir_arg(args)
+ if initial:
+ return initial
+ wait_seconds = float(getattr(args, "fixed_views_precompute_wait_seconds", getattr(args, "relation_precompute_wait_seconds", 0)) or 0)
+ if wait_seconds <= 0:
+ return None
+ deadline = time.time() + wait_seconds
+ while time.time() <= deadline:
+ current = fixed_views_precompute_dir_arg(args)
+ if current:
+ return current
+ time.sleep(max(0.1, poll_seconds))
+ return fixed_views_precompute_dir_arg(args)
diff --git a/vlm_agent/reconstruction/relations/agents/orchestration/planning.py b/vlm_agent/reconstruction/relations/agents/orchestration/planning.py
new file mode 100644
index 0000000..8450906
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/orchestration/planning.py
@@ -0,0 +1,1431 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from relations.agents.state import sorted_object_id_key, write_json, write_md
+from relations.agents.wall_attach_prefilter import formal_wall_references
+from relations.tools.wall_orientation_candidates import axis_angle_distance_degrees
+
+CANONICAL_WALL_IDS = ("wall_001", "wall_002", "wall_003", "wall_004")
+PROVISIONAL_WALL_VISIBILITY_CAPABILITY = "provisional_wall_visibility_ready"
+CANONICAL_WALL_NAMING_INSTRUCTION = (
+ "Preserve the provided canonical screen-clockwise wall ids exactly. The formal OBB preprocessing and local final assembler own wall coordinates and id order. "
+ "Judge original-image visibility and weak/inferred status for each existing wall, but do not rotate, renumber, or replace wall ids based on visibility. "
+ "Candidate wall ids are provisional only until the local canonical remap is applied. Weak/hidden/inferred closure walls may complete room geometry, "
+ "but mark them weak_wall=true and original_image_visibility=not_visible or inferred; do not use them as ordinary attach anchors without visible wall contact."
+)
+
+def _as_instruction_list(value: Any) -> list[str]:
+ if isinstance(value, list):
+ return [str(item) for item in value]
+ if isinstance(value, str) and value.strip():
+ return [value]
+ return []
+
+
+def append_task_instruction(task: dict[str, Any], instruction: str) -> None:
+ instructions = _as_instruction_list(task.get("instructions"))
+ if instruction not in instructions:
+ instructions.append(instruction)
+ task["instructions"] = instructions
+
+
+def task_identifier(task: dict[str, Any]) -> str:
+ return str(task.get("task_id") or task.get("id") or "unknown_task")
+
+
+def task_capabilities(task: dict[str, Any], field: str) -> list[str]:
+ value = task.get(field)
+ if isinstance(value, list):
+ return [str(item) for item in value]
+ if isinstance(value, str) and value.strip():
+ return [value.strip()]
+ return []
+
+
+def task_provides_capability(task: dict[str, Any], capability: str) -> bool:
+ return capability in task_capabilities(task, "provides_capabilities")
+
+
+def task_requires_capability(task: dict[str, Any], capability: str) -> bool:
+ return capability in task_capabilities(task, "requires_capabilities")
+
+
+def is_wall_attach_recall_task(task: dict[str, Any]) -> bool:
+ canonical_kind = str(task.get("canonical_task_kind") or "").lower()
+ task_type = str(task.get("task_type") or "").lower()
+ task_id = task_identifier(task).lower()
+ return bool(
+ canonical_kind == "wall_attach_candidate_recall"
+ or "wall_attach_candidate_recall" in task_type
+ or "wall_attach_candidate_recall" in task_id
+ )
+
+
+def hydrate_wall_attach_candidate_scope(
+ task: dict[str, Any],
+ prefilter: dict[str, Any],
+) -> dict[str, Any]:
+ """Expand a Manager-selected scope with candidates created after planning."""
+ manager_scope = [
+ str(object_id)
+ for object_id in task.get("object_scope", [])
+ if isinstance(object_id, str) and object_id.strip()
+ ]
+ candidate_scope: list[str] = []
+ for pair in prefilter.get("candidate_pairs", []):
+ if not isinstance(pair, dict):
+ continue
+ object_id = pair.get("object_target")
+ if isinstance(object_id, str) and object_id.strip() and object_id not in candidate_scope:
+ candidate_scope.append(object_id)
+
+ effective_scope = list(dict.fromkeys([*manager_scope, *candidate_scope]))
+ added_candidate_object_ids = [object_id for object_id in candidate_scope if object_id not in manager_scope]
+ task["object_scope"] = effective_scope
+ scope_audit = {
+ "policy": "manager_scope_union_local_prefilter_candidates",
+ "manager_object_scope": manager_scope,
+ "prefilter_candidate_object_ids": candidate_scope,
+ "added_candidate_object_ids": added_candidate_object_ids,
+ "effective_object_scope": effective_scope,
+ "candidate_pairs_require_explicit_decision": True,
+ }
+ task["wall_attach_scope_hydration"] = scope_audit
+ append_task_instruction(
+ task,
+ "The effective object_scope is the union of the Manager-selected scope and objects produced later by the local wall-attach prefilter. "
+ "Return an explicit accept or reject decision for every pair in wall_attach_prefilter.candidate_pairs, including candidates outside the Manager's original scope. "
+ "Do not silently omit a prefiltered pair.",
+ )
+ if prefilter.get("semantic_visibility_geometry_conflicts"):
+ append_task_instruction(
+ task,
+ "Some candidate pairs carry semantic_visibility_geometry_conflict because "
+ "a high-confidence wall-following object has substantially stronger local "
+ "contact geometry on an adjacent wall that an earlier visibility record blocked. "
+ "For those pairs, re-check the original image against the numbered final-wall "
+ "overlay instead of treating the earlier visibility label as authoritative. "
+ "If the wall-id binding was wrong, correct that wall's wall_visibility_review "
+ "record and then judge attach; geometry alone is never sufficient for acceptance.",
+ )
+ return scope_audit
+
+
+def annotate_semantic_visibility_geometry_conflict_resolutions(
+ task: dict[str, Any],
+ response: dict[str, Any],
+) -> dict[str, Any]:
+ """Grant correction authority only to an explicitly accepted conflict pair."""
+ prefilter = task.get("wall_attach_prefilter") if isinstance(task.get("wall_attach_prefilter"), dict) else {}
+ conflicts = prefilter.get("semantic_visibility_geometry_conflicts", [])
+ if not isinstance(conflicts, list) or not conflicts:
+ return response
+
+ accepted_pairs: set[tuple[str, str]] = set()
+ for item in response.get("wall_attach_candidates", []):
+ if not isinstance(item, dict) or str(item.get("decision") or "").lower() != "accept":
+ continue
+ object_id = item.get("object_target")
+ wall_id = item.get("candidate_wall_id")
+ if isinstance(object_id, str) and wall_id in CANONICAL_WALL_IDS:
+ accepted_pairs.add((object_id, str(wall_id)))
+ for item in response.get("accepted_attach_candidates", []):
+ if not isinstance(item, dict):
+ continue
+ relation = item.get("relation") if isinstance(item.get("relation"), dict) else item
+ parameters = relation.get("parameters") if isinstance(relation.get("parameters"), dict) else {}
+ object_id = parameters.get("object_target")
+ wall_id = parameters.get("object_anchor")
+ if isinstance(object_id, str) and wall_id in CANONICAL_WALL_IDS:
+ accepted_pairs.add((object_id, str(wall_id)))
+
+ reviews = response.get("wall_visibility_review", [])
+ review_by_wall = {
+ str(item.get("wall_id") or item.get("id")): item
+ for item in reviews
+ if isinstance(item, dict) and (item.get("wall_id") or item.get("id")) in CANONICAL_WALL_IDS
+ } if isinstance(reviews, list) else {}
+ resolutions = []
+ for conflict in conflicts:
+ if not isinstance(conflict, dict):
+ continue
+ object_id = str(conflict.get("object_id") or "")
+ wall_id = str(conflict.get("wall_id") or "")
+ review = review_by_wall.get(wall_id)
+ if (object_id, wall_id) not in accepted_pairs or not isinstance(review, dict):
+ continue
+ visibility = str(review.get("original_image_visibility") or "").strip().lower()
+ if (
+ visibility not in {"visible", "partially_visible"}
+ or review.get("can_anchor_wall_attach") is not True
+ or bool(review.get("weak_wall"))
+ or bool(review.get("closure_only"))
+ ):
+ continue
+ review["semantic_visibility_geometry_conflict_resolved"] = True
+ object_ids = review.setdefault("semantic_visibility_geometry_conflict_object_ids", [])
+ if isinstance(object_ids, list) and object_id not in object_ids:
+ object_ids.append(object_id)
+ resolutions.append({
+ "object_id": object_id,
+ "wall_id": wall_id,
+ "decision": "accepted_pair_and_visible_wall_correct_prior_binding",
+ "prior_block_reason": conflict.get("prior_block_reason"),
+ })
+ if resolutions:
+ response["semantic_visibility_geometry_conflict_resolutions"] = resolutions
+ return response
+
+
+CONTAIN_SCOPE_CONFIDENCE_THRESHOLD = 0.90
+CONTAIN_SCOPE_MAX_HIGH_CONFIDENCE_PAIRS_PER_ANCHOR = 32
+CONTAIN_TASK_MAX_REVIEW_PAIRS = 80
+
+
+def _normalized_contain_pair(item: Any) -> dict[str, Any] | None:
+ if not isinstance(item, dict):
+ return None
+ parameters = item.get("parameters") if isinstance(item.get("parameters"), dict) else item
+ anchor = parameters.get("object_anchor")
+ target = parameters.get("object_target")
+ if not (
+ isinstance(anchor, str)
+ and anchor.startswith("object_")
+ and isinstance(target, str)
+ and target.startswith("object_")
+ and anchor != target
+ ):
+ return None
+ try:
+ confidence = float(item.get("confidence_hint", 0.0))
+ except (TypeError, ValueError):
+ confidence = 0.0
+ return {
+ "object_anchor": anchor,
+ "object_target": target,
+ "confidence_hint": confidence,
+ "source": item,
+ }
+
+
+def select_contain_review_pairs(
+ candidates: dict[str, Any],
+ max_pairs: int = CONTAIN_TASK_MAX_REVIEW_PAIRS,
+) -> dict[str, Any]:
+ """Select one deterministic, anchor-balanced pair set for scope and API payload."""
+ by_anchor: dict[str, list[dict[str, Any]]] = {}
+ seen: set[tuple[str, str]] = set()
+ invalid_pair_count = 0
+ for item in candidates.get("candidate_pairs", []):
+ pair = _normalized_contain_pair(item)
+ if pair is None:
+ invalid_pair_count += 1
+ continue
+ key = (pair["object_anchor"], pair["object_target"])
+ if key in seen:
+ continue
+ seen.add(key)
+ by_anchor.setdefault(pair["object_anchor"], []).append(pair)
+ per_anchor_overflow: list[dict[str, Any]] = []
+ for anchor, anchor_pairs in by_anchor.items():
+ anchor_pairs.sort(key=lambda pair: (-float(pair["confidence_hint"]), pair["object_target"]))
+ per_anchor_overflow.extend(anchor_pairs[CONTAIN_SCOPE_MAX_HIGH_CONFIDENCE_PAIRS_PER_ANCHOR:])
+ by_anchor[anchor] = anchor_pairs[:CONTAIN_SCOPE_MAX_HIGH_CONFIDENCE_PAIRS_PER_ANCHOR]
+
+ selected: list[dict[str, Any]] = []
+ anchor_ids = sorted(by_anchor, key=sorted_object_id_key)
+ round_index = 0
+ limit = max(0, int(max_pairs))
+ while len(selected) < limit:
+ added = False
+ for anchor in anchor_ids:
+ anchor_pairs = by_anchor[anchor]
+ if round_index >= len(anchor_pairs):
+ continue
+ selected.append(anchor_pairs[round_index])
+ added = True
+ if len(selected) >= limit:
+ break
+ if not added:
+ break
+ round_index += 1
+
+ selected_keys = {(pair["object_anchor"], pair["object_target"]) for pair in selected}
+ dropped = [
+ *per_anchor_overflow,
+ *[
+ pair
+ for anchor in anchor_ids
+ for pair in by_anchor[anchor]
+ if (pair["object_anchor"], pair["object_target"]) not in selected_keys
+ ],
+ ]
+ return {
+ "selected_pairs": selected,
+ "dropped_pairs": dropped,
+ "valid_pair_count": len(seen),
+ "invalid_pair_count": invalid_pair_count,
+ "max_pairs": limit,
+ "selection_policy": "deterministic_round_robin_by_anchor_then_confidence",
+ }
+
+
+def hydrate_contain_candidate_scope(
+ task: dict[str, Any],
+ candidates: dict[str, Any],
+ confidence_threshold: float = CONTAIN_SCOPE_CONFIDENCE_THRESHOLD,
+) -> dict[str, Any]:
+ """Use the exact transmitted contain pair set to hydrate the review scope."""
+ manager_scope = [
+ str(object_id)
+ for object_id in task.get("object_scope", [])
+ if isinstance(object_id, str) and object_id.strip()
+ ]
+ selection = select_contain_review_pairs(candidates)
+ review_pairs = [
+ {
+ "object_anchor": pair["object_anchor"],
+ "object_target": pair["object_target"],
+ "confidence_hint": pair["confidence_hint"],
+ }
+ for pair in selection["selected_pairs"]
+ ]
+ dropped_review_pairs = [
+ {
+ "object_anchor": pair["object_anchor"],
+ "object_target": pair["object_target"],
+ "confidence_hint": pair["confidence_hint"],
+ }
+ for pair in selection["dropped_pairs"]
+ ]
+ candidate_scope: list[str] = []
+ for pair in review_pairs:
+ for object_id in (pair["object_anchor"], pair["object_target"]):
+ if object_id not in candidate_scope:
+ candidate_scope.append(object_id)
+
+ high_confidence_pairs = [
+ pair for pair in review_pairs
+ if float(pair["confidence_hint"]) >= confidence_threshold
+ ]
+ dropped_high_confidence_pairs = [
+ pair for pair in dropped_review_pairs
+ if float(pair["confidence_hint"]) >= confidence_threshold
+ ]
+ effective_scope = list(dict.fromkeys([*manager_scope, *candidate_scope]))
+ task["object_scope"] = effective_scope
+ scope_audit = {
+ "policy": "manager_priority_scope_union_exact_transmitted_local_contain_candidates",
+ "selection_policy": selection["selection_policy"],
+ "confidence_threshold": confidence_threshold,
+ "max_transmitted_candidate_pairs": selection["max_pairs"],
+ "manager_object_scope": manager_scope,
+ "transmitted_candidate_pairs": review_pairs,
+ "dropped_candidate_pairs": dropped_review_pairs,
+ "high_confidence_candidate_pairs": high_confidence_pairs,
+ "dropped_high_confidence_candidate_pairs": dropped_high_confidence_pairs,
+ "candidate_object_ids": candidate_scope,
+ "added_candidate_object_ids": [object_id for object_id in candidate_scope if object_id not in manager_scope],
+ "effective_object_scope": effective_scope,
+ "candidate_pairs_require_explicit_decision": True,
+ }
+ task["contain_scope_hydration"] = scope_audit
+ append_task_instruction(
+ task,
+ "The Manager-selected object_scope and scene_focus are priorities, not a closed contain search set. Review every retained local recall pair, including pairs outside the Manager original scope, and return an explicit accept, reject, or uncertain decision for each one. The local selection score is not semantic evidence and is not exposed to the Agent.",
+ )
+ return scope_audit
+
+
+HIGH_CONFIDENCE_EXCLUSIVE_MODES = {"ceiling_only", "wall_only"}
+HIGH_CONFIDENCE_NON_WALL_MODES = {"ceiling_only", "support_only", "contain_candidate"}
+
+
+def _object_anchor_routing_by_id(value: Any) -> dict[str, dict[str, str]]:
+ result: dict[str, dict[str, str]] = {}
+ for item in value if isinstance(value, list) else []:
+ if not isinstance(item, dict) or not item.get("object_id"):
+ continue
+ result[str(item["object_id"])] = {
+ "object_id": str(item["object_id"]),
+ "anchor_mode": str(item.get("anchor_mode") or "uncertain").lower(),
+ "confidence": str(item.get("confidence") or "low").lower(),
+ }
+ return result
+
+
+def _scene_state_object_ids(scene_state: dict[str, Any]) -> list[str]:
+ candidates = scene_state.get("object_candidates", {})
+ items = candidates.get("items", []) if isinstance(candidates, dict) else []
+ return list(dict.fromkeys(
+ str(item.get("object_id") or item.get("id"))
+ for item in items
+ if isinstance(item, dict) and (item.get("object_id") or item.get("id"))
+ ))
+
+
+def hydrate_task_scope_from_anchor_routing(
+ task: dict[str, Any],
+ object_ids: list[str],
+ object_anchor_routing: list[dict[str, Any]] | None,
+ available_anchor_modes: set[str] | None = None,
+) -> dict[str, Any] | None:
+ """Compile Manager semantic routes into non-exclusive task scopes."""
+ if is_support_contact_task(task):
+ return None
+ routing = _object_anchor_routing_by_id(object_anchor_routing)
+ if not routing:
+ return None
+ available_anchor_modes = available_anchor_modes or {
+ "ceiling_only", "wall_only", "support_only", "contain_candidate",
+ }
+ manager_scope = [
+ str(object_id)
+ for object_id in task.get("manager_priority_object_scope", task.get("object_scope", []))
+ if isinstance(object_id, str) and object_id.strip()
+ ]
+ task.setdefault("manager_priority_object_scope", list(manager_scope))
+ capabilities = set(task_capabilities(task, "provides_capabilities"))
+ kind = str(task.get("canonical_task_kind") or task.get("task_type") or "").lower()
+ is_wall = any(capability.startswith("wall_") for capability in capabilities) or kind.startswith("wall_")
+ is_ceiling = "ceiling_judgement_ready" in capabilities or kind == "ceiling_judgement"
+ is_contain = "contain_relations_ready" in capabilities or kind == "contain_relation_extraction"
+ if not (is_wall or is_ceiling or is_contain):
+ return None
+
+ routed: list[str] = []
+ excluded: list[str] = []
+ for object_id in object_ids:
+ record = routing.get(object_id)
+ if record is None:
+ continue
+ mode = record["anchor_mode"]
+ confidence = record["confidence"]
+ high = confidence == "high"
+ if is_wall:
+ if high and mode in HIGH_CONFIDENCE_NON_WALL_MODES and mode in available_anchor_modes:
+ excluded.append(object_id)
+ continue
+ if mode in {"wall_only", "floor_wall", "uncertain"} or not high:
+ routed.append(object_id)
+ elif is_ceiling:
+ if mode == "ceiling_only":
+ routed.append(object_id)
+ elif is_contain:
+ if mode == "contain_candidate":
+ routed.append(object_id)
+
+ effective_scope = [
+ object_id
+ for object_id in dict.fromkeys([*manager_scope, *routed])
+ if object_id not in excluded
+ ]
+ task["object_scope"] = effective_scope
+ task["object_anchor_routing"] = [
+ routing[object_id] for object_id in effective_scope if object_id in routing
+ ]
+ audit = {
+ "policy": "manager_priority_scope_plus_nonexclusive_anchor_routing",
+ "manager_priority_object_scope": manager_scope,
+ "routed_object_ids": routed,
+ "high_confidence_excluded_object_ids": excluded,
+ "effective_object_scope": effective_scope,
+ }
+ task["anchor_routing_scope_hydration"] = audit
+ return audit
+
+
+def hydrate_support_capability_scope(
+ task: dict[str, Any],
+ object_ids: list[str] | None = None,
+ object_anchor_routing: list[dict[str, Any]] | None = None,
+ available_anchor_modes: set[str] | None = None,
+) -> dict[str, Any]:
+ """Give Support a complete gravity baseline while preserving Manager priorities."""
+ manager_scope = [
+ str(object_id)
+ for object_id in task.get("manager_priority_object_scope", task.get("object_scope", []))
+ if isinstance(object_id, str) and object_id.strip()
+ ]
+ task.setdefault("manager_priority_object_scope", list(manager_scope))
+ all_object_ids = list(dict.fromkeys(object_ids or manager_scope))
+ routing = _object_anchor_routing_by_id(object_anchor_routing)
+ available_anchor_modes = available_anchor_modes or HIGH_CONFIDENCE_EXCLUSIVE_MODES
+ excluded: list[str] = []
+ gravity_scope: list[str] = []
+ for object_id in all_object_ids:
+ record = routing.get(object_id)
+ if (
+ record
+ and record.get("confidence") == "high"
+ and record.get("anchor_mode") in HIGH_CONFIDENCE_EXCLUSIVE_MODES
+ and record.get("anchor_mode") in available_anchor_modes
+ ):
+ excluded.append(object_id)
+ continue
+ gravity_scope.append(object_id)
+
+ task["object_scope"] = gravity_scope
+ task["gravity_coverage_scope"] = gravity_scope
+ task["object_anchor_routing"] = [
+ routing[object_id] for object_id in all_object_ids if object_id in routing
+ ]
+ audit = {
+ "policy": "complete_gravity_baseline_plus_manager_priority_scope",
+ "manager_priority_object_scope": manager_scope,
+ "gravity_coverage_scope": gravity_scope,
+ "high_confidence_exclusive_object_ids": excluded,
+ "support_anchor_requires_own_gravity_verdict": True,
+ "every_gravity_scope_object_requires_explicit_verdict": True,
+ }
+ task["support_scope_hydration"] = audit
+ append_task_instruction(
+ task,
+ "Gravity-direction support is the first relation pass. Return exactly one concise support_contact_candidates verdict for every object in gravity_coverage_scope; do not silently omit obvious floor-standing furniture. The Manager priority scope is not a closed support set and only identifies difficult cases; when an object supports another object, also decide its own gravity-direction support. Floor support may coexist with a separate wall attach for floor_wall objects.",
+ )
+ return audit
+
+
+def task_control_text(task: dict[str, Any]) -> str:
+ keys = [
+ "task_id",
+ "id",
+ "task_type",
+ "canonical_task_kind",
+ "runner_action",
+ "runtime_class",
+ "runtime_policy",
+ "execution_stage_hint",
+ "instructions",
+ "reason",
+ ]
+ return " ".join(str(task.get(key, "")) for key in keys).lower()
+
+
+def is_support_contact_task(task: dict[str, Any]) -> bool:
+ capabilities = task_capabilities(task, "provides_capabilities")
+ if capabilities:
+ return "support_relations_ready" in capabilities
+ identity = " ".join(
+ str(task.get(key) or "")
+ for key in ("task_id", "id", "task_type", "canonical_task_kind")
+ ).lower()
+ return any(marker in identity for marker in ("support_contact", "support contact", "support_relation", "support relation"))
+
+
+
+def _as_float_pair(value: Any) -> list[float] | None:
+ if not isinstance(value, (list, tuple)) or len(value) < 2:
+ return None
+ try:
+ return [float(value[0]), float(value[1])]
+ except (TypeError, ValueError):
+ return None
+
+
+def _point_in_polygon_xz(point: list[float], polygon: list[list[float]]) -> bool:
+ x, z = point
+ inside = False
+ n = len(polygon)
+ if n < 3:
+ return False
+ j = n - 1
+ for i in range(n):
+ xi, zi = polygon[i]
+ xj, zj = polygon[j]
+ intersects = (zi > z) != (zj > z)
+ if intersects:
+ x_at_z = (xj - xi) * (z - zi) / ((zj - zi) or 1e-12) + xi
+ if x < x_at_z:
+ inside = not inside
+ j = i
+ return inside
+
+
+def _polygon_area_abs_xz(polygon: list[list[float]]) -> float:
+ if len(polygon) < 3:
+ return 0.0
+ area = 0.0
+ for idx, point in enumerate(polygon):
+ nxt = polygon[(idx + 1) % len(polygon)]
+ area += point[0] * nxt[1] - nxt[0] * point[1]
+ return abs(area) * 0.5
+
+
+def _obb_polygon_from_wall_evidence(wall_evidence: dict[str, Any]) -> list[list[float]]:
+ render_grid = wall_evidence.get("render_grid_wall_candidates") if isinstance(wall_evidence, dict) else None
+ primary = render_grid.get("primary_candidate") if isinstance(render_grid, dict) else None
+ segments = primary.get("wall_segments") if isinstance(primary, dict) else None
+ polygon: list[list[float]] = []
+ if isinstance(segments, list):
+ for segment in segments[:4]:
+ if not isinstance(segment, dict):
+ continue
+ point = _as_float_pair(segment.get("first_point"))
+ if point:
+ polygon.append(point)
+ if len(polygon) == 4:
+ return polygon
+ candidate_walls = wall_evidence.get("candidate_walls") if isinstance(wall_evidence, dict) else None
+ if isinstance(candidate_walls, list):
+ polygon = []
+ for wall in candidate_walls[:4]:
+ if not isinstance(wall, dict):
+ continue
+ point = _as_float_pair(wall.get("first_point"))
+ if point:
+ polygon.append(point)
+ if len(polygon) == 4:
+ return polygon
+ return []
+
+
+def build_obb_wall_risk_audit(wall_evidence: dict[str, Any]) -> dict[str, Any]:
+ """Route formal OBB wall review to light or heavy visual adjudication."""
+ render_grid = wall_evidence.get("render_grid_wall_candidates", {}) if isinstance(wall_evidence, dict) else {}
+ primary = render_grid.get("primary_candidate") if isinstance(render_grid, dict) else None
+ method = None
+ if isinstance(primary, dict):
+ method = primary.get("method") or primary.get("source_method")
+ policy_method = render_grid.get("method_policy", {}).get("primary_method") if isinstance(render_grid, dict) else None
+ polygon = _obb_polygon_from_wall_evidence(wall_evidence if isinstance(wall_evidence, dict) else {})
+ object_footprints = wall_evidence.get("object_footprints", []) if isinstance(wall_evidence, dict) else []
+ risk_score = 0
+ triggered: list[dict[str, Any]] = []
+
+ def trigger(check: str, severity: int, reason: str, details: dict[str, Any] | None = None) -> None:
+ nonlocal risk_score
+ risk_score += severity
+ triggered.append({"check": check, "severity": severity, "reason": reason, "details": details or {}})
+
+ if not isinstance(primary, dict):
+ trigger("primary_candidate_missing", 5, "No formal OBB primary candidate is available.")
+ elif (method or policy_method) != "obb_from_mesh_footprint_min_rect":
+ trigger(
+ "unexpected_primary_method",
+ 4,
+ "Formal wall flow expects obb_from_mesh_footprint_min_rect as the primary method.",
+ {"method": method, "policy_primary_method": policy_method},
+ )
+ if len(polygon) != 4:
+ trigger("invalid_polygon", 5, "Formal OBB candidate does not expose four valid wall corners.", {"corner_count": len(polygon)})
+
+ inside_count = 0
+ outside_objects: list[str] = []
+ if len(polygon) == 4 and isinstance(object_footprints, list):
+ for obj in object_footprints:
+ if not isinstance(obj, dict):
+ continue
+ center = _as_float_pair(obj.get("center_xz"))
+ if not center:
+ continue
+ if _point_in_polygon_xz(center, polygon):
+ inside_count += 1
+ else:
+ outside_objects.append(str(obj.get("object_id")))
+ total = inside_count + len(outside_objects)
+ outside_ratio = (len(outside_objects) / total) if total else 0.0
+ if total and outside_ratio > 0.15:
+ trigger(
+ "object_centers_outside_obb",
+ 4 if outside_ratio > 0.3 else 2,
+ "Many object footprint centers fall outside the formal OBB candidate.",
+ {"outside_ratio": round(outside_ratio, 4), "outside_objects": outside_objects[:12], "object_count": total},
+ )
+
+ xs: list[float] = []
+ zs: list[float] = []
+ for obj in object_footprints:
+ if not isinstance(obj, dict):
+ continue
+ center = _as_float_pair(obj.get("center_xz"))
+ if center:
+ xs.append(center[0])
+ zs.append(center[1])
+ if xs and zs:
+ object_extent_area = max((max(xs) - min(xs)) * (max(zs) - min(zs)), 1e-8)
+ obb_area = _polygon_area_abs_xz(polygon)
+ area_ratio = obb_area / object_extent_area
+ if area_ratio > 8.0 or area_ratio < 0.6:
+ trigger(
+ "obb_area_ratio_unusual",
+ 2,
+ "OBB area is unusually large or small relative to object center spread.",
+ {"obb_area": round(obb_area, 6), "object_center_extent_area": round(object_extent_area, 6), "area_ratio": round(area_ratio, 3)},
+ )
+
+ segments = primary.get("wall_segments") if isinstance(primary, dict) else None
+ lengths: list[float] = []
+ if isinstance(segments, list):
+ for segment in segments[:4]:
+ if not isinstance(segment, dict):
+ continue
+ first = _as_float_pair(segment.get("first_point"))
+ second = _as_float_pair(segment.get("second_point"))
+ if first and second:
+ lengths.append(((first[0] - second[0]) ** 2 + (first[1] - second[1]) ** 2) ** 0.5)
+ if len(lengths) == 4 and min(lengths) > 1e-8:
+ length_ratio = max(lengths) / min(lengths)
+ if length_ratio > 8.0:
+ trigger("extreme_wall_length_ratio", 3, "OBB wall length ratio is extreme.", {"length_ratio": round(length_ratio, 3), "lengths": [round(v, 6) for v in lengths]})
+
+ near_contact_count = 0
+ near_geometry = wall_evidence.get("render_grid_wall_attach_geometry") or wall_evidence.get("wall_attach_geometry") if isinstance(wall_evidence, dict) else None
+ if isinstance(near_geometry, list):
+ for item in near_geometry:
+ if not isinstance(item, dict):
+ continue
+ try:
+ if float(item.get("footprint_min_distance_to_wall")) <= 0.035:
+ near_contact_count += 1
+ except (TypeError, ValueError):
+ pass
+ if isinstance(object_footprints, list) and object_footprints and near_contact_count == 0:
+ trigger("no_near_wall_object_support", 2, "No object footprints are near any OBB wall; wall placement may be suspect.")
+
+ orientation_disagreement = wall_orientation_disagreement_signal(wall_evidence)
+ if orientation_disagreement.get("requires_agent_wall_alignment_objects"):
+ trigger(
+ "mesh_min_area_disagrees_with_stable_object_axis",
+ 3,
+ "The minimum-area OBB direction differs materially from a dominant multi-object orientation consensus.",
+ orientation_disagreement,
+ )
+
+ if risk_score >= 5:
+ risk_level = "high"
+ recommendation = "heavy_review"
+ elif risk_score >= 3:
+ risk_level = "medium"
+ recommendation = "heavy_review"
+ else:
+ risk_level = "low"
+ recommendation = "light_review"
+ return {
+ "schema_version": "obb_wall_risk_audit",
+ "status": "available",
+ "formal_method": "obb_from_mesh_footprint_min_rect",
+ "risk_score": risk_score,
+ "risk_level": risk_level,
+ "recommendation": recommendation,
+ "triggered_checks": triggered,
+ "metrics": {
+ "corner_count": len(polygon),
+ "object_center_inside_count": inside_count,
+ "object_center_outside_count": len(outside_objects),
+ "near_wall_geometry_count": near_contact_count,
+ "wall_lengths": [round(v, 6) for v in lengths],
+ "orientation_disagreement": orientation_disagreement,
+ },
+ "policy": {
+ "low_risk": "Use light wall visibility / weak-wall review.",
+ "medium_or_high_risk": "Use heavy wall final adjudication with richer visual evidence.",
+ "coordinates": "Final coordinates are still locally generated by obb_from_mesh_footprint_min_rect; this audit only routes review effort.",
+ },
+ }
+
+
+def write_obb_wall_risk_audit(output_dir: Path, audit: dict[str, Any]) -> None:
+ write_json(output_dir / "debug" / "walls" / "obb_wall_risk_audit.json", audit)
+ lines = [
+ f"- risk_level: `{audit.get('risk_level')}`",
+ f"- risk_score: `{audit.get('risk_score')}`",
+ f"- recommendation: `{audit.get('recommendation')}`",
+ f"- formal_method: `{audit.get('formal_method')}`",
+ "",
+ "## Triggered Checks",
+ ]
+ for item in audit.get("triggered_checks", []):
+ lines.append(f"- `{item.get('check')}` severity={item.get('severity')}: {item.get('reason')}")
+ if not audit.get("triggered_checks"):
+ lines.append("- none")
+ write_md(output_dir / "debug" / "walls" / "obb_wall_risk_audit.md", "OBB Wall Risk Audit", lines)
+
+
+def resolve_wall_review_mode(requested: str, audit: dict[str, Any] | None) -> str:
+ mode = str(requested or "auto").strip().lower()
+ if mode in {"light", "heavy"}:
+ return mode
+ if isinstance(audit, dict) and audit.get("recommendation") == "light_review":
+ return "light"
+ return "heavy"
+
+
+def local_wall_reasoning_signal(wall_evidence: dict[str, Any]) -> dict[str, Any]:
+ """Return geometry-only evidence that justifies repairing a missing wall reasoning chain."""
+ raw_geometry = wall_evidence.get("render_grid_wall_attach_geometry")
+ if not isinstance(raw_geometry, list):
+ raw_geometry = wall_evidence.get("wall_attach_geometry")
+ candidates: list[dict[str, Any]] = []
+ seen: set[tuple[str, str]] = set()
+ for item in raw_geometry if isinstance(raw_geometry, list) else []:
+ if not isinstance(item, dict):
+ continue
+ object_id = str(item.get("object_id") or "")
+ wall_id = str(item.get("wall_id") or "")
+ if not object_id or not wall_id:
+ continue
+ try:
+ distance = float(item.get("footprint_min_distance_to_wall"))
+ except (TypeError, ValueError):
+ continue
+ floor_supported = bool(item.get("is_floor_supported"))
+ flat_analysis = item.get("flat_contact_edge_analysis")
+ has_flat_edge = bool(item.get("has_valid_flat_contact_edge"))
+ if isinstance(flat_analysis, dict):
+ has_flat_edge = has_flat_edge or bool(flat_analysis.get("has_valid_flat_contact_edge"))
+ thin_planar = bool(item.get("is_thin_planar_candidate"))
+ close_to_wall = distance <= 0.035
+ reason = None
+ if close_to_wall and not floor_supported:
+ reason = "non_floor_supported_object_near_formal_wall"
+ elif close_to_wall and floor_supported and has_flat_edge:
+ reason = "floor_supported_object_has_valid_flat_contact_edge"
+ elif close_to_wall and thin_planar:
+ reason = "thin_planar_object_near_formal_wall"
+ if not reason:
+ continue
+ key = (object_id, wall_id)
+ if key in seen:
+ continue
+ seen.add(key)
+ candidates.append({
+ "object_id": object_id,
+ "wall_id": wall_id,
+ "category": item.get("category"),
+ "footprint_min_distance_to_wall": distance,
+ "is_floor_supported": floor_supported,
+ "is_thin_planar_candidate": thin_planar,
+ "has_valid_flat_contact_edge": has_flat_edge,
+ "reason": reason,
+ })
+ return {
+ "status": "candidate_signal" if candidates else "no_candidate_signal",
+ "candidate_count": len(candidates),
+ "candidates": candidates,
+ "policy": "Geometry is a high-recall trigger only. Agent wall semantics and attach review still decide the relation.",
+ }
+
+
+def wall_orientation_disagreement_signal(wall_evidence: dict[str, Any]) -> dict[str, Any]:
+ """Detect when minimum-area geometry and a stable object-axis consensus disagree."""
+ render_grid = wall_evidence.get("render_grid_wall_candidates") if isinstance(wall_evidence, dict) else {}
+ review = render_grid.get("wall_orientation_review") if isinstance(render_grid, dict) else {}
+ candidates = render_grid.get("orientation_candidates") if isinstance(render_grid, dict) else []
+ candidate_by_id = {
+ str(item.get("candidate_id")): item
+ for item in candidates if isinstance(item, dict) and item.get("candidate_id")
+ }
+ baseline_id = str(review.get("baseline_candidate_id") or "")
+ baseline = candidate_by_id.get(baseline_id)
+ baseline_orientation = baseline.get("orientation") if isinstance(baseline, dict) and isinstance(baseline.get("orientation"), dict) else {}
+ baseline_angle = baseline_orientation.get("angle_degrees_xz")
+ clusters = []
+ for item in candidates if isinstance(candidates, list) else []:
+ if not isinstance(item, dict):
+ continue
+ orientation = item.get("orientation") if isinstance(item.get("orientation"), dict) else {}
+ if orientation.get("source") != "robust_object_rotation_consensus":
+ continue
+ clusters.append({
+ "candidate_id": item.get("candidate_id"),
+ "angle_degrees_xz": orientation.get("angle_degrees_xz"),
+ "member_count": int(orientation.get("member_count") or 0),
+ "total_weight": float(orientation.get("total_weight") or 0.0),
+ })
+ clusters.sort(key=lambda item: (-item["total_weight"], -item["member_count"], str(item["candidate_id"])))
+ dominant = clusters[0] if clusters else None
+ total_cluster_weight = sum(item["total_weight"] for item in clusters)
+ dominance = (
+ dominant["total_weight"] / total_cluster_weight
+ if dominant and total_cluster_weight > 0 else 0.0
+ )
+ angle_difference = None
+ if dominant and isinstance(baseline_angle, (int, float)) and isinstance(dominant.get("angle_degrees_xz"), (int, float)):
+ angle_difference = axis_angle_distance_degrees(float(baseline_angle), float(dominant["angle_degrees_xz"]))
+ requires_review = bool(
+ len(candidate_by_id) > 1
+ and dominant
+ and dominant["member_count"] >= 2
+ and dominance >= 0.55
+ and angle_difference is not None
+ and angle_difference >= 10.0
+ )
+ return {
+ "status": "orientation_review_required" if requires_review else "no_strong_orientation_disagreement",
+ "requires_agent_wall_alignment_objects": requires_review,
+ "baseline_candidate_id": baseline_id or None,
+ "baseline_angle_degrees_xz": baseline_angle,
+ "dominant_object_axis_candidate_id": dominant.get("candidate_id") if dominant else None,
+ "dominant_object_axis_angle_degrees_xz": dominant.get("angle_degrees_xz") if dominant else None,
+ "dominant_member_count": dominant.get("member_count") if dominant else 0,
+ "dominant_weight_share": round(dominance, 6),
+ "axis_difference_degrees": round(angle_difference, 6) if angle_difference is not None else None,
+ "policy": "A strong disagreement triggers semantic object selection; it never directly selects a wall candidate.",
+ }
+
+
+def configure_wall_alignment_task(task: dict[str, Any]) -> dict[str, Any]:
+ """Convert a pre-orientation wall node into the compact alignment contract.
+
+ A Manager may merge ``provisional_wall_visibility_ready`` into this node.
+ That optional capability judges labelled edges in finite candidate panels;
+ it remains distinct from final wall-id semantics and therefore stays valid
+ when local code selects a different finite orientation after this task.
+ """
+ previous_kind = str(task.get("canonical_task_kind") or task.get("task_type") or "")
+ focus = str(task.get("scene_focus") or "").strip()
+ scope = [str(item) for item in task.get("object_scope", []) if item]
+ preserve_provisional = task_provides_capability(task, PROVISIONAL_WALL_VISIBILITY_CAPABILITY)
+ task["task_type"] = "wall_alignment_object_review"
+ task["canonical_task_kind"] = "wall_alignment_object_review"
+ task["runner_action"] = "api_dynamic_task"
+ task["runtime_class"] = "required_reasoning"
+ task["requires_artifacts"] = ["object_list", "original_image", "mask_overlay", "wall_orientation_candidates_overlay"]
+ task["requires_capabilities"] = []
+ task["provides_capabilities"] = ["wall_alignment_objects_ready", "wall_object_hypotheses_ready"]
+ task["image_artifact_profile"] = ["original_image", "mask_overlay", "wall_orientation_candidates_overlay"]
+ if preserve_provisional:
+ task["provides_capabilities"].append(PROVISIONAL_WALL_VISIBILITY_CAPABILITY)
+ task["needs_tool"] = False
+ task["use_vision_model"] = True
+ instructions = [
+ "Identify only objects whose real pose reliably reveals a room-wall axis: wall-mounted, wall-embedded, physically against a wall, or clearly parallel/aligned with a wall.",
+ "Use the original image for semantics and the mask overlay for object ids. Treat object_scope as a priority, not a closed set.",
+ "Exclude central furniture, round objects, small clutter, and uncertain headings. Return high or medium confidence only.",
+ "Do not choose the winning room-orientation candidate, edit coordinates/angles, emit final wall ids, or emit attach relations. Local code scores the finite OBB axes from each selected object's canonical rotation_degrees_topdown.",
+ "For each likely wall object, use the isolated labelled candidate panels to return candidate_wall_bindings for every panel where the corresponding real wall edge is identifiable. Copy candidate_id and provisional_wall_id exactly. These bindings are semantic review evidence, not final wall selection or attach acceptance.",
+ ]
+ expected_output_schema: dict[str, Any] = {
+ "likely_wall_objects": [{
+ "object_id": "object_###",
+ "role": "wall_mounted|wall_embedded|against_wall|wall_aligned",
+ "confidence": "high|medium",
+ "evidence_code": "short_visual_evidence_code",
+ }],
+ "candidate_wall_bindings": [{
+ "object_id": "object_###",
+ "candidate_id": "copy exactly from candidate panel",
+ "provisional_wall_id": "wall_001|wall_002|wall_003|wall_004",
+ "role": "wall_mounted|wall_embedded|against_wall|wall_aligned",
+ "confidence": "high|medium",
+ "evidence_code": "short_visual_binding_code",
+ }],
+ }
+ if preserve_provisional:
+ instructions.extend([
+ "The isolated wall-orientation candidate panels are labelled independently. For each panel edge, you may judge only whether it is a directly visible real wall versus inferred, weak, or closure-only geometry.",
+ "Return candidate_id plus provisional_wall_id exactly as printed on the panel. Do not select the winning candidate or emit final wall ids/relations; local code binds records only for the selected candidate after orientation scoring.",
+ ])
+ expected_output_schema["provisional_wall_visibility_review"] = [{
+ "candidate_id": "copy exactly from candidate panel",
+ "provisional_wall_id": "wall_001|wall_002|wall_003|wall_004",
+ "original_image_visibility": "visible|partially_visible|not_visible|inferred",
+ "weak_wall": False,
+ "closure_only": False,
+ "can_anchor_wall_attach": True,
+ "reason_code": "short_reason_code",
+ }]
+ task["provisional_wall_visibility_merged_with_alignment"] = True
+ if focus:
+ instructions.append(f"Manager scene focus: {focus}")
+ if scope:
+ instructions.append("Manager priority object scope: " + ", ".join(scope))
+ task["instructions"] = instructions
+ task["expected_output_schema"] = expected_output_schema
+ task["wall_alignment_contract_normalized_by_runner"] = True
+ task.setdefault("wall_alignment_previous_task_kind", previous_kind)
+ return task
+
+
+def configure_provisional_wall_visibility_task(task: dict[str, Any]) -> dict[str, Any]:
+ """Normalize a standalone pre-orientation candidate-edge semantic node."""
+ previous_kind = str(task.get("canonical_task_kind") or task.get("task_type") or "")
+ focus = str(task.get("scene_focus") or "").strip()
+ scope = [str(item) for item in task.get("object_scope", []) if item]
+ task["task_type"] = "provisional_wall_visibility_review"
+ task["canonical_task_kind"] = "provisional_wall_visibility_review"
+ task["runner_action"] = "api_dynamic_task"
+ task["runtime_class"] = "required_reasoning"
+ task["requires_artifacts"] = ["original_image", "mask_overlay", "wall_orientation_candidates_overlay"]
+ task["provides_capabilities"] = [PROVISIONAL_WALL_VISIBILITY_CAPABILITY]
+ task["image_artifact_profile"] = ["original_image", "mask_overlay", "wall_orientation_candidates_overlay"]
+ task["needs_tool"] = False
+ task["use_vision_model"] = True
+ instructions = [
+ "Use the isolated finite wall-orientation candidate panels to judge directly visible real-wall edges versus inferred, weak, or closure-only room boundaries.",
+ "Return compact candidate_id plus provisional_wall_id records only. Do not select an orientation candidate, create coordinates, alter edges, emit final wall ids, or emit attach relations.",
+ "Mark visible or partially_visible only with direct source-image wall-surface or wall-junction evidence. If uncertain, use inferred/not_visible and can_anchor_wall_attach=false.",
+ ]
+ if focus:
+ instructions.append(f"Manager scene focus: {focus}")
+ if scope:
+ instructions.append("Manager priority object scope: " + ", ".join(scope))
+ task["instructions"] = instructions
+ task["expected_output_schema"] = {
+ "provisional_wall_visibility_review": [{
+ "candidate_id": "copy exactly from candidate panel",
+ "provisional_wall_id": "wall_001|wall_002|wall_003|wall_004",
+ "original_image_visibility": "visible|partially_visible|not_visible|inferred",
+ "weak_wall": False,
+ "closure_only": False,
+ "can_anchor_wall_attach": True,
+ "reason_code": "short_reason_code",
+ }]
+ }
+ task["provisional_wall_visibility_contract_normalized_by_runner"] = True
+ task.setdefault("provisional_wall_visibility_previous_task_kind", previous_kind)
+ return task
+
+def ensure_wall_alignment_object_review_task(
+ manager_plan: dict[str, Any],
+ *,
+ reason: str,
+) -> dict[str, Any]:
+ tasks = manager_plan.setdefault("tasks", [])
+ for task in tasks:
+ if isinstance(task, dict) and task_provides_capability(task, "wall_alignment_objects_ready"):
+ configure_wall_alignment_task(task)
+ return manager_plan
+ task = {
+ "task_id": "task_wall_alignment_object_review",
+ "reason": reason,
+ "object_scope": [],
+ "scene_focus": "Identify only reliable wall-following objects for local finite-axis scoring.",
+ }
+ tasks.append(configure_wall_alignment_task(task))
+ manager_plan["wall_alignment_task_injected_by_runner"] = True
+ return manager_plan
+
+
+def ensure_post_orientation_wall_visibility_task(
+ manager_plan: dict[str, Any],
+) -> dict[str, Any]:
+ tasks = manager_plan.setdefault("tasks", [])
+ if any(
+ isinstance(task, dict)
+ and task_provides_capability(task, "wall_visibility_semantics_ready")
+ and not task_provides_capability(task, "wall_alignment_objects_ready")
+ for task in tasks
+ ):
+ return manager_plan
+ tasks.append({
+ "task_id": "task_wall_visibility_semantics_review",
+ "task_type": "wall_visibility_semantics_review",
+ "canonical_task_kind": "wall_visibility_semantics_review",
+ "agent_role": "DynamicTaskAgent",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "required_reasoning",
+ "requires_artifacts": ["formal_obb_walls", "original_image", "mask_overlay"],
+ "requires_capabilities": ["wall_alignment_objects_ready"],
+ "provides_capabilities": ["wall_visibility_semantics_ready"],
+ "image_artifact_profile": ["original_image", "mask_overlay"],
+ "needs_tool": False,
+ "use_vision_model": True,
+ "instructions": [
+ "The local scorer has already selected and applied the formal OBB orientation. Judge visibility, weak/inferred or closure-only status, and attach eligibility for all four final wall references.",
+ "Copy wall_id and geometry_key exactly. Do not choose another candidate or edit coordinates, angles, ids, or methods.",
+ ],
+ "expected_output_schema": {
+ "wall_visibility_review": [{
+ "wall_id": "wall_001",
+ "geometry_key": "copy supplied wall_edge_... exactly",
+ "original_image_visibility": "visible|partially_visible|not_visible|inferred",
+ "weak_wall": False,
+ "closure_only": False,
+ "can_anchor_wall_attach": True,
+ "reason_code": "short_reason_code",
+ }]
+ },
+ })
+ manager_plan["post_orientation_visibility_task_injected_by_runner"] = True
+ return manager_plan
+
+
+def apply_wall_task_policy(
+ manager_plan: dict[str, Any],
+ wall_evidence: dict[str, Any],
+ wall_candidate_mode: str,
+ wall_review_mode: str = "heavy",
+ obb_risk_audit: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+
+ tasks = [task for task in manager_plan.get("tasks", []) if isinstance(task, dict)]
+ manager_plan["tasks"] = tasks
+ local_signal = local_wall_reasoning_signal(wall_evidence)
+ orientation_signal = wall_orientation_disagreement_signal(wall_evidence)
+ repairs: list[dict[str, Any]] = []
+ manager_wall_task_ids = [
+ task_identifier(task)
+ for task in tasks
+ if any(capability.startswith("wall_") for capability in task_capabilities(task, "provides_capabilities"))
+ ]
+ visibility_requested_by_manager = any(
+ task_provides_capability(task, "wall_visibility_semantics_ready")
+ for task in tasks
+ )
+ provisional_visibility_requested_by_manager = any(
+ task_provides_capability(task, PROVISIONAL_WALL_VISIBILITY_CAPABILITY)
+ for task in tasks
+ )
+ split_attach_scopes: list[str] = []
+
+ # A pre-orientation node cannot also provide semantics keyed to final wall
+ # ids. Split that incompatible merge while preserving Manager scope/focus.
+ for task in tasks:
+ caps = task_capabilities(task, "provides_capabilities")
+ if (
+ "wall_object_hypotheses_ready" in caps
+ and "wall_alignment_objects_ready" not in caps
+ ):
+ caps.append("wall_alignment_objects_ready")
+ task["provides_capabilities"] = caps
+ if (
+ task_provides_capability(task, "wall_alignment_objects_ready")
+ and task_provides_capability(task, "wall_attach_decisions_ready")
+ ):
+ split_attach_scopes.extend(str(item) for item in task.get("object_scope", []) if item)
+ configure_wall_alignment_task(task)
+ repairs.append({
+ "repair_reason": "pre_orientation_alignment_and_post_orientation_wall_attach_must_be_separate",
+ "converted_task_id": task_identifier(task),
+ })
+ elif (
+ task_provides_capability(task, "wall_alignment_objects_ready")
+ and task_provides_capability(task, "wall_visibility_semantics_ready")
+ ):
+ configure_wall_alignment_task(task)
+ repairs.append({
+ "repair_reason": "pre_orientation_alignment_and_post_orientation_visibility_must_be_separate",
+ "converted_task_id": task_identifier(task),
+ })
+
+ if split_attach_scopes:
+ before = {task_identifier(task) for task in manager_plan.get("tasks", [])}
+ manager_plan = ensure_wall_attach_recall_task(manager_plan, wall_evidence)
+ for task in manager_plan.get("tasks", []):
+ if task_identifier(task) not in before and is_wall_attach_recall_task(task):
+ task["object_scope"] = sorted(set(split_attach_scopes))
+ repairs.append({
+ "repair_reason": "split_combined_alignment_and_wall_attach_node",
+ "injected_task_ids": [
+ task_identifier(task)
+ for task in manager_plan.get("tasks", [])
+ if task_identifier(task) not in before
+ ],
+ })
+
+ def providers(capability: str) -> list[dict[str, Any]]:
+ return [task for task in manager_plan.get("tasks", []) if task_provides_capability(task, capability)]
+
+ # Candidate-edge visibility is pre-orientation evidence. It cannot be
+ # produced by the downstream attach node because the local prefilter needs
+ # it before that node is executed. When a compact Manager accidentally
+ # groups both capabilities there, preserve its semantic intent by moving
+ # only the provisional capability into an existing alignment node.
+ misplaced_provisional_tasks = [
+ task for task in providers(PROVISIONAL_WALL_VISIBILITY_CAPABILITY)
+ if task_provides_capability(task, "wall_attach_decisions_ready")
+ ]
+ if misplaced_provisional_tasks:
+ for task in misplaced_provisional_tasks:
+ task["provides_capabilities"] = [
+ capability
+ for capability in task_capabilities(task, "provides_capabilities")
+ if capability != PROVISIONAL_WALL_VISIBILITY_CAPABILITY
+ ]
+ if not providers("wall_alignment_objects_ready"):
+ before = {task_identifier(task) for task in manager_plan.get("tasks", [])}
+ manager_plan = ensure_wall_alignment_object_review_task(
+ manager_plan,
+ reason="provisional_candidate_visibility_requires_preorientation_alignment_context",
+ )
+ repairs.append({
+ "repair_reason": "moved_provisional_visibility_from_downstream_attach_to_alignment",
+ "injected_task_ids": [
+ task_identifier(task)
+ for task in manager_plan.get("tasks", [])
+ if task_identifier(task) not in before
+ ],
+ })
+ for alignment_task in providers("wall_alignment_objects_ready"):
+ provided = task_capabilities(alignment_task, "provides_capabilities")
+ if PROVISIONAL_WALL_VISIBILITY_CAPABILITY not in provided:
+ provided.append(PROVISIONAL_WALL_VISIBILITY_CAPABILITY)
+ alignment_task["provides_capabilities"] = provided
+ configure_wall_alignment_task(alignment_task)
+ repairs.append({
+ "repair_reason": "provisional_visibility_must_precede_wall_attach_prefilter",
+ "moved_from_task_ids": [task_identifier(task) for task in misplaced_provisional_tasks],
+ "target_task_ids": [task_identifier(task) for task in providers("wall_alignment_objects_ready")],
+ })
+
+ has_wall_intent = bool(manager_wall_task_ids)
+ attach_requested = bool(providers("wall_attach_decisions_ready"))
+ needs_attach_chain = attach_requested or bool(local_signal.get("candidate_count"))
+
+ if bool(local_signal.get("candidate_count")) and not attach_requested:
+ before = {task_identifier(task) for task in manager_plan.get("tasks", [])}
+ manager_plan = ensure_wall_attach_recall_task(manager_plan, wall_evidence)
+ added = [
+ task_identifier(task)
+ for task in manager_plan.get("tasks", [])
+ if task_identifier(task) not in before
+ ]
+ repairs.append({
+ "repair_reason": "local_geometry_candidates_require_wall_attach_decision",
+ "injected_task_ids": added,
+ "trigger": "local_geometry_candidate_signal",
+ })
+ attach_requested = bool(providers("wall_attach_decisions_ready"))
+ needs_attach_chain = True
+
+ if orientation_signal.get("requires_agent_wall_alignment_objects") and not attach_requested:
+ before = {task_identifier(task) for task in manager_plan.get("tasks", [])}
+ manager_plan = ensure_wall_attach_recall_task(manager_plan, wall_evidence)
+ added = [
+ task_identifier(task)
+ for task in manager_plan.get("tasks", [])
+ if task_identifier(task) not in before
+ ]
+ repairs.append({
+ "repair_reason": "orientation_risk_review_requires_post_orientation_wall_attach_and_visibility_decision",
+ "injected_task_ids": added,
+ "trigger": orientation_signal.get("status"),
+ })
+ attach_requested = bool(providers("wall_attach_decisions_ready"))
+ needs_attach_chain = True
+
+ needs_alignment = bool(
+ has_wall_intent
+ or needs_attach_chain
+ or bool(providers(PROVISIONAL_WALL_VISIBILITY_CAPABILITY))
+ or orientation_signal.get("requires_agent_wall_alignment_objects")
+ )
+ if needs_alignment and not providers("wall_alignment_objects_ready"):
+ before = {task_identifier(task) for task in manager_plan.get("tasks", [])}
+ manager_plan = ensure_wall_alignment_object_review_task(
+ manager_plan,
+ reason=(
+ "strong_local_orientation_disagreement"
+ if orientation_signal.get("requires_agent_wall_alignment_objects")
+ else "wall_reasoning_requires_semantic_alignment_objects"
+ ),
+ )
+ added = [
+ task_identifier(task)
+ for task in manager_plan.get("tasks", [])
+ if task_identifier(task) not in before
+ ]
+ repairs.append({
+ "repair_reason": "formal_wall_orientation_requires_semantic_alignment_objects",
+ "injected_task_ids": added,
+ "trigger": orientation_signal.get("status"),
+ })
+
+ # In the normal path the downstream attach task also performs final wall
+ # visibility review, avoiding a third wall API call.
+ for task in providers("wall_attach_decisions_ready"):
+ provided = task_capabilities(task, "provides_capabilities")
+ if "wall_visibility_semantics_ready" not in provided:
+ provided.append("wall_visibility_semantics_ready")
+ task["provides_capabilities"] = provided
+ required = task_capabilities(task, "requires_capabilities")
+ for capability in ("wall_alignment_objects_ready", "wall_attach_candidates_ready"):
+ if capability not in required:
+ required.append(capability)
+ if providers(PROVISIONAL_WALL_VISIBILITY_CAPABILITY) and PROVISIONAL_WALL_VISIBILITY_CAPABILITY not in required:
+ required.append(PROVISIONAL_WALL_VISIBILITY_CAPABILITY)
+ task["requires_capabilities"] = required
+
+ if visibility_requested_by_manager and not providers("wall_attach_decisions_ready") and not providers("wall_visibility_semantics_ready"):
+ manager_plan = ensure_post_orientation_wall_visibility_task(manager_plan)
+ repairs.append({
+ "repair_reason": "manager_requested_visibility_moved_after_local_orientation_selection",
+ "injected_task_ids": ["task_wall_visibility_semantics_review"],
+ })
+
+ for task in manager_plan.get("tasks", []):
+ if not isinstance(task, dict):
+ continue
+ wall_caps = {
+ capability
+ for capability in task_capabilities(task, "provides_capabilities")
+ if capability.startswith("wall_")
+ }
+ if not wall_caps:
+ continue
+ if "wall_alignment_objects_ready" in wall_caps:
+ configure_wall_alignment_task(task)
+ continue
+ if PROVISIONAL_WALL_VISIBILITY_CAPABILITY in wall_caps:
+ configure_provisional_wall_visibility_task(task)
+ continue
+ task.setdefault("formal_wall_references", formal_wall_references(wall_evidence))
+ task.setdefault("formal_wall_geometry_source", "obb_from_mesh_footprint_min_rect")
+ append_task_instruction(task, CANONICAL_WALL_NAMING_INSTRUCTION)
+ append_task_instruction(
+ task,
+ "The final formal OBB orientation is selected by local scoring from Agent-identified wall-following objects before this task. Do not choose another candidate or edit wall coordinates, ids, angles, or methods.",
+ )
+ if "wall_visibility_semantics_ready" in wall_caps:
+ task.setdefault("wall_review_mode", wall_review_mode)
+ task.setdefault("obb_wall_risk_audit", obb_risk_audit or {})
+ append_task_instruction(
+ task,
+ "Return visibility/weak/inferred/closure-only attach eligibility for all four current formal_wall_references, copying each geometry_key exactly.",
+ )
+ if "wall_attach_decisions_ready" in wall_caps:
+ task["needs_tool"] = True
+ append_task_instruction(
+ task,
+ "Return one explicit accept or reject verdict for every local wall_attach_prefilter candidate pair after reviewing final-wall visibility.",
+ )
+
+ manager_plan["wall_task_policy"] = "compact_manager_semantic_objects_then_local_orientation"
+ manager_plan["manager_plan_repaired"] = bool(repairs)
+ manager_plan["manager_plan_repairs"] = repairs
+ manager_plan["formal_wall_task_control"] = {
+ "mode": wall_candidate_mode,
+ "manager_plan_format": "compact",
+ "manager_selected_wall_task_ids": manager_wall_task_ids,
+ "final_wall_task_ids": [
+ task_identifier(task)
+ for task in manager_plan.get("tasks", [])
+ if any(capability.startswith("wall_") for capability in task_capabilities(task, "provides_capabilities"))
+ ],
+ "manager_had_wall_intent": has_wall_intent,
+ "manager_requested_provisional_wall_visibility": provisional_visibility_requested_by_manager,
+ "provisional_wall_visibility_provider_task_ids": [
+ task_identifier(task) for task in providers(PROVISIONAL_WALL_VISIBILITY_CAPABILITY)
+ ],
+ "local_wall_reasoning_signal": local_signal,
+ "local_wall_orientation_disagreement_signal": orientation_signal,
+ "repairs": repairs,
+ "note": "Manager selects wall capabilities and grouping. When requested, provisional candidate-edge visibility is bound locally only after finite orientation selection; the runner enforces data dependencies, not a fixed wall-task roster.",
+ "wall_review_mode": wall_review_mode,
+ "obb_wall_risk_level": obb_risk_audit.get("risk_level") if isinstance(obb_risk_audit, dict) else None,
+ "obb_wall_risk_recommendation": obb_risk_audit.get("recommendation") if isinstance(obb_risk_audit, dict) else None,
+ }
+ return manager_plan
+
+
+
+def ensure_wall_attach_recall_task(manager_plan: dict[str, Any], wall_evidence: dict[str, Any]) -> dict[str, Any]:
+ tasks = manager_plan.setdefault("tasks", [])
+ has_recall_task = any(is_wall_attach_recall_task(task) for task in tasks if isinstance(task, dict))
+ if has_recall_task:
+ for task in tasks:
+ if isinstance(task, dict) and is_wall_attach_recall_task(task):
+ required = task_capabilities(task, "requires_capabilities")
+ for capability in ("wall_alignment_objects_ready", "wall_attach_candidates_ready"):
+ if capability not in required:
+ required.append(capability)
+ task["requires_capabilities"] = required
+ provided = task_capabilities(task, "provides_capabilities")
+ for capability in ("wall_visibility_semantics_ready", "wall_attach_decisions_ready"):
+ if capability not in provided:
+ provided.append(capability)
+ task["provides_capabilities"] = provided
+ return manager_plan
+ tasks.append(
+ {
+ "task_id": "task_wall_attach_candidate_recall",
+ "agent_role": "DynamicTaskAgent",
+ "task_type": "wall_attach_candidate_recall",
+ "canonical_task_kind": "wall_attach_candidate_recall",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "required_reasoning",
+ "requires_capabilities": ["wall_alignment_objects_ready", "wall_attach_candidates_ready"],
+ "provides_capabilities": ["wall_visibility_semantics_ready", "wall_attach_decisions_ready"],
+ "needs_tool": True,
+ "use_vision_model": True,
+ "candidate_categories_include_but_not_limited_to": [
+ "picture",
+ "painting",
+ "glass_door",
+ "window",
+ "whiteboard",
+ "wall_shelf",
+ "wall_panel",
+ "mounted_screen",
+ "wall_lamp",
+ ],
+ "requires_artifacts": [
+ "object_list",
+ "object_footprints",
+ "formal_obb_walls",
+ "canonical_topdown",
+ "canonical_grid_mapping",
+ "wall_evidence",
+ "original_image",
+ "mask_overlay",
+ ],
+ "evidence_needed": [
+ "original image and mask overlay paths from scene_state",
+ "object masks and object footprints from wall_evidence.object_footprints",
+ "wall_evidence.wall_attach_geometry footprint-to-wall metrics",
+ "wall_evidence.render_grid_wall_candidates and render_grid_wall_attach_geometry from obb_from_mesh_footprint_min_rect",
+ "canonical topdown image and grid mapping",
+ "final wall coordinates will be locally overridden from obb_from_mesh_footprint_min_rect; candidate_walls are formal OBB geometry evidence",
+ "selective topdown render if wall id or contact is ambiguous",
+ ],
+ "instructions": (
+ "The local scorer has already selected and applied the final formal OBB orientation. First return visibility/weak/inferred/closure-only semantics for all four supplied formal_wall_references, copying wall_id and geometry_key exactly. "
+ "Recall and judge possible wall-mounted, wall-embedded, or verified against-wall horizontal contact attach relations. Do not rely on category or description alone. "
+ "Do not reject using object center distance alone: thick objects can touch a wall with their back/side while their center is away from it. "
+ "Floor-supported or object-supported objects may also be wall-attached; gravity-direction support and horizontal wall contact are different physical relations. A floor/object support relation must not replace a second wall attach when the object back/side is visibly or geometrically in contact with a verified non-weak wall. "
+ "For each candidate, combine image/overlay/mask evidence, canonical topdown, "
+ "object footprint/bbox side contact, wall_attach_geometry footprint_min_distance_to_wall, render_grid_wall_attach_geometry from the formal OBB wall candidate, and final canonical wall coordinates. "
+ "Generated wall candidates are provisional geometry evidence; their candidate/provisional ids may be renamed by wall_final_adjudication and the local canonicalization gate. Do not choose between wall extraction algorithms. "
+ "Return wall_attach_candidates with decision accept or reject, object_target, candidate_wall_id using the final canonical wall id after wall_final_adjudication when available, wall_contact_type, visual_evidence, "
+ "geometry_evidence, description_used_as, reason, and selective_topdown_used. Reject decisions must cite concrete visual or footprint evidence. "
+ "Accepted wall attaches must use wall_001..wall_004 only; never unknown_wall or unresolved_anchor. "
+ "Do not accept attach to a weak, hidden, or inferred-by-room-closure wall when the original image cannot show the actual wall surface/contact; such walls may close the room geometry but cannot anchor attach relations."
+ " Consume the local wall_attach_prefilter candidate pairs as the primary review set. Do not exhaustively rescan every object-wall combination. "
+ "The prefilter has already applied deterministic lower-mesh flat-edge checks for ordinary floor-supported objects. It may deliberately retain candidates on walls whose final visibility is not known yet. "
+ "You may add a missing relation only when the original image directly shows an obvious wall-mounted or wall-embedded object omitted by the local candidates; explain that supplement explicitly."
+ ),
+ "wall_evidence_summary": {
+ "canonical_topdown_image_path": wall_evidence.get("topdown", {}).get("canonical_topdown_image_path"),
+ "canonical_grid_mapping_path": wall_evidence.get("topdown", {}).get("canonical_grid_mapping_path"),
+ "candidate_walls_count": len(wall_evidence.get("candidate_walls", [])),
+ "candidate_walls_trust_level": wall_evidence.get("candidate_room_polygon_hint", {}).get("trust_level"),
+ "object_footprints_count": len(wall_evidence.get("object_footprints", [])),
+ "wall_attach_geometry_count": len(wall_evidence.get("wall_attach_geometry", [])),
+ "render_grid_wall_status": wall_evidence.get("render_grid_wall_candidates", {}).get("status"),
+ "render_grid_wall_primary_method": wall_evidence.get("render_grid_wall_candidates", {}).get("method_policy", {}).get("primary_method"),
+ "render_grid_wall_attach_geometry_count": len(wall_evidence.get("render_grid_wall_attach_geometry", [])),
+ },
+ "output_schema": {
+ "wall_visibility_review": [
+ {
+ "wall_id": "wall_001",
+ "geometry_key": "copy supplied wall_edge_... exactly",
+ "original_image_visibility": "visible|partially_visible|not_visible|inferred",
+ "weak_wall": False,
+ "closure_only": False,
+ "can_anchor_wall_attach": True,
+ "reason_code": "short_reason_code",
+ }
+ ],
+ "wall_attach_candidates": [
+ {
+ "decision": "accept|reject",
+ "object_target": "object_###",
+ "candidate_wall_id": "wall_001|wall_002|wall_003|wall_004",
+ "evidence": ["..."],
+ "reason": "...",
+ "selective_topdown_used": False,
+ }
+ ],
+ "accepted_attach_candidates": [
+ {"relation": "attach", "parameters": {"object_anchor": "wall_001", "object_target": "object_###"}}
+ ],
+ },
+ }
+ )
+ manager_plan["wall_attach_recall_task_injected_by_runner"] = True
+ return manager_plan
diff --git a/vlm_agent/reconstruction/relations/agents/run_scene_delivery_multi_agent.py b/vlm_agent/reconstruction/relations/agents/run_scene_delivery_multi_agent.py
new file mode 100644
index 0000000..7118473
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/run_scene_delivery_multi_agent.py
@@ -0,0 +1,4134 @@
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from contextlib import nullcontext
+from pathlib import Path
+from typing import Any
+
+CODE_ROOT = Path(__file__).resolve().parents[2]
+RELATIONS_ROOT = Path(__file__).resolve().parents[1]
+if str(CODE_ROOT) not in sys.path:
+ sys.path.insert(0, str(CODE_ROOT))
+
+from relations.agents.api_client import APIConfig, APIClientError, SceneDeliveryAPIClient
+from relations.agents.final_assembler import (
+ assemble_final_payloads,
+ collect_contain_conflicts,
+ compact_contain_conflicts_for_guard,
+)
+from relations.agents.schema_guard import validate_files
+from relations.agents.state import ensure_output_tree, initialize_empty_audits, write_audit_pair, write_json, write_md
+from relations.agents.orchestration.delivery_io import (
+ POLICY_PATH,
+ RELATION_LAYOUT_DIR_ENV,
+ _load_json_file,
+ apply_render_grid_precompute_to_scene_state,
+ configure_wall_candidate_environment,
+ copy_policy_to_run,
+ load_prompt,
+ load_relation_layout_manifest,
+ load_static_policy,
+ relation_layout_expected_manifest_path,
+ update_agent_audits,
+ wait_for_fixed_views_precompute_dir,
+ wait_for_relation_precompute,
+ write_dry_run_final_marker,
+ write_manager_plan,
+ write_render_audit,
+ write_task_output,
+)
+from relations.agents.supervisor import supervise_execution
+from relations.agents.tool_runner import ensure_canonical_topdown_mapping, scan_render_script_capabilities
+from relations.agents.task_registry import (
+ compact_manager_capability_catalog,
+ compile_compact_manager_plan,
+ task_image_artifact_profiles,
+)
+from relations.tools.build_scene_state import build_scene_state, write_scene_state
+from relations.tools.extract_wall_evidence import build_wall_evidence, write_wall_evidence
+from relations.tools.wall_orientation_candidates import resolve_orientation_candidate_from_wall_objects
+from relations.tools.generate_fixed_layout_views import reuse_fixed_layout_views_cache
+from relations.tools.build_wall_visual_evidence import ensure_wall_visual_evidence
+from relations.tools.visualize_final_walls_on_topdown import visualize as visualize_final_walls_on_topdown
+from relations.tools.write_agent_process_report import write_final_process_report
+from relations.tools.runtime_tracker import RuntimeTracker
+from relations.tools.build_wall_object_hypotheses import build_and_write_wall_object_hypotheses, extract_wall_object_hypotheses, write_wall_object_hypotheses
+from relations.agents.wall_attach_prefilter import (
+ build_wall_attach_prefilter,
+ compact_wall_evidence_for_recall,
+ formal_wall_references,
+ render_wall_attach_prefilter_overlay,
+ wall_geometry_key,
+)
+
+from relations.agents.orchestration.planning import (
+ CANONICAL_WALL_IDS,
+ PROVISIONAL_WALL_VISIBILITY_CAPABILITY,
+ CANONICAL_WALL_NAMING_INSTRUCTION,
+ CONTAIN_SCOPE_CONFIDENCE_THRESHOLD,
+ CONTAIN_SCOPE_MAX_HIGH_CONFIDENCE_PAIRS_PER_ANCHOR,
+ CONTAIN_TASK_MAX_REVIEW_PAIRS,
+ HIGH_CONFIDENCE_EXCLUSIVE_MODES,
+ HIGH_CONFIDENCE_NON_WALL_MODES,
+ _as_instruction_list,
+ append_task_instruction,
+ task_identifier,
+ task_capabilities,
+ task_provides_capability,
+ task_requires_capability,
+ is_wall_attach_recall_task,
+ hydrate_wall_attach_candidate_scope,
+ annotate_semantic_visibility_geometry_conflict_resolutions,
+ _normalized_contain_pair,
+ select_contain_review_pairs,
+ hydrate_contain_candidate_scope,
+ _object_anchor_routing_by_id,
+ _scene_state_object_ids,
+ hydrate_task_scope_from_anchor_routing,
+ hydrate_support_capability_scope,
+ task_control_text,
+ is_support_contact_task,
+ _as_float_pair,
+ _point_in_polygon_xz,
+ _polygon_area_abs_xz,
+ _obb_polygon_from_wall_evidence,
+ build_obb_wall_risk_audit,
+ write_obb_wall_risk_audit,
+ resolve_wall_review_mode,
+ local_wall_reasoning_signal,
+ wall_orientation_disagreement_signal,
+ configure_wall_alignment_task,
+ configure_provisional_wall_visibility_task,
+ ensure_wall_alignment_object_review_task,
+ ensure_post_orientation_wall_visibility_task,
+ apply_wall_task_policy,
+ ensure_wall_attach_recall_task,
+)
+
+def extract_wall_attach_candidate_evidence(task_results: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ evidence: list[dict[str, Any]] = []
+
+ def walk(value: Any):
+ if isinstance(value, dict):
+ yield value
+ for child in value.values():
+ yield from walk(child)
+ elif isinstance(value, list):
+ for child in value:
+ yield from walk(child)
+
+ for root in task_results:
+ if not isinstance(root, dict):
+ continue
+ for node in walk(root):
+ if not isinstance(node, dict):
+ continue
+ for key in ["wall_attach_candidates", "wall_attach_recall", "wall_attach_decisions", "attach_decisions", "accepted_attach_candidates"]:
+ value = node.get(key)
+ if isinstance(value, list):
+ for item in value:
+ if isinstance(item, dict):
+ evidence.append({
+ "source_task_id": root.get("task_id"),
+ "source_key": key,
+ "candidate": item,
+ })
+ elif isinstance(value, dict) and isinstance(value.get("candidates"), list):
+ for item in value["candidates"]:
+ if isinstance(item, dict):
+ evidence.append({
+ "source_task_id": root.get("task_id"),
+ "source_key": f"{key}.candidates",
+ "candidate": item,
+ })
+ return evidence
+
+
+def build_and_write_wall_evidence(scene_state: dict[str, Any], output_dir: Path) -> dict[str, Any]:
+ wall_evidence = build_wall_evidence(scene_state)
+ write_wall_evidence(wall_evidence, output_dir / "debug" / "walls")
+ return wall_evidence
+
+
+def compact_formal_wall_selection(wall_evidence: dict[str, Any]) -> dict[str, Any]:
+ source = wall_evidence.get("render_grid_wall_candidates") if isinstance(wall_evidence, dict) else {}
+ source = source if isinstance(source, dict) else {}
+ audit = wall_evidence.get("wall_orientation_selection_audit") if isinstance(wall_evidence, dict) else {}
+ audit = audit if isinstance(audit, dict) else {}
+ selected_id = (
+ audit.get("selected_candidate_id")
+ or source.get("selected_orientation_candidate_id")
+ )
+ return {
+ "selected_orientation_candidate_id": selected_id,
+ "selection_source": (
+ audit.get("selection_source")
+ or source.get("wall_orientation_selection_source")
+ ),
+ "selection_is_final": bool(selected_id),
+ "agent_may_select_candidate": False,
+ "agent_may_change_geometry": False,
+ }
+
+
+def compact_provisional_wall_visibility_catalog(wall_evidence: dict[str, Any]) -> dict[str, Any]:
+ """Expose candidate-edge identities without forwarding local coordinates."""
+ source = wall_evidence.get("render_grid_wall_candidates") if isinstance(wall_evidence, dict) else {}
+ source = source if isinstance(source, dict) else {}
+ candidates = []
+ for candidate in source.get("orientation_candidates", []):
+ if not isinstance(candidate, dict):
+ continue
+ candidate_id = str(candidate.get("candidate_id") or "").strip()
+ if not candidate_id:
+ continue
+ edges = []
+ for index, wall in enumerate(candidate.get("wall_segments", []), start=1):
+ if not isinstance(wall, dict):
+ continue
+ provisional_wall_id = str(wall.get("id") or wall.get("provisional_wall_id") or "")
+ if provisional_wall_id not in CANONICAL_WALL_IDS:
+ continue
+ edges.append({
+ "provisional_wall_id": provisional_wall_id,
+ "geometry_key": wall_geometry_key(wall),
+ "screen_clockwise_order_index": wall.get("screen_clockwise_order_index") or index,
+ })
+ candidates.append({
+ "candidate_id": candidate_id,
+ "orientation_source": (candidate.get("orientation") or {}).get("source") if isinstance(candidate.get("orientation"), dict) else None,
+ "provisional_wall_ids": [item["provisional_wall_id"] for item in edges],
+ "edges": edges,
+ })
+ return {
+ "candidate_count": len(candidates),
+ "candidates": candidates,
+ "agent_contract": "Use the isolated labelled candidate panels. Return candidate_id plus provisional_wall_id semantics only; local code maps the selected candidate by immutable edge geometry after orientation scoring.",
+ }
+
+
+def _walk_mapping_nodes(value: Any):
+ if isinstance(value, dict):
+ yield value
+ for child in value.values():
+ yield from _walk_mapping_nodes(child)
+ elif isinstance(value, list):
+ for child in value:
+ yield from _walk_mapping_nodes(child)
+
+
+def materialize_selected_provisional_wall_visibility(
+ wall_evidence: dict[str, Any],
+ selected_candidate_id: str | None,
+ task_results: list[dict[str, Any]],
+) -> tuple[dict[str, Any] | None, dict[str, Any]]:
+ """Bind candidate-panel semantics to immutable final OBB edges after selection."""
+ catalog = compact_provisional_wall_visibility_catalog(wall_evidence)
+ selected_candidate = next(
+ (item for item in catalog.get("candidates", []) if item.get("candidate_id") == selected_candidate_id),
+ None,
+ )
+ audit: dict[str, Any] = {
+ "schema_version": "provisional_wall_visibility_binding",
+ "selected_candidate_id": selected_candidate_id,
+ "candidate_panel_count": catalog.get("candidate_count", 0),
+ "binding_policy": "candidate_id_exact_then_geometry_key_exact",
+ "status": "not_requested",
+ "source_task_ids": [],
+ "matched_provisional_records": [],
+ "unmatched_provisional_records": [],
+ "missing_final_wall_ids": list(CANONICAL_WALL_IDS),
+ }
+ if not selected_candidate_id or not isinstance(selected_candidate, dict):
+ audit["status"] = "selected_candidate_unavailable"
+ return None, audit
+
+ candidate_edges = {
+ str(edge.get("provisional_wall_id")): edge
+ for edge in selected_candidate.get("edges", [])
+ if isinstance(edge, dict) and edge.get("provisional_wall_id")
+ }
+ final_references = formal_wall_references(wall_evidence)
+ final_by_geometry = {
+ str(reference.get("geometry_key")): reference
+ for reference in final_references if reference.get("geometry_key")
+ }
+ seen: set[tuple[str, str]] = set()
+ materialized: list[dict[str, Any]] = []
+ source_ids: list[str] = []
+ for root in task_results:
+ if not isinstance(root, dict):
+ continue
+ provided = root.get("provided_capabilities") if isinstance(root.get("provided_capabilities"), list) else []
+ if PROVISIONAL_WALL_VISIBILITY_CAPABILITY not in provided:
+ continue
+ source_task_id = str(root.get("task_id") or "unknown_task")
+ if source_task_id not in source_ids:
+ source_ids.append(source_task_id)
+ for node in _walk_mapping_nodes(root):
+ items = node.get("provisional_wall_visibility_review") if isinstance(node, dict) else None
+ if not isinstance(items, list):
+ continue
+ for item in items:
+ if not isinstance(item, dict):
+ continue
+ candidate_id = str(item.get("candidate_id") or "").strip()
+ provisional_wall_id = str(item.get("provisional_wall_id") or item.get("wall_id") or item.get("id") or "").strip()
+ if candidate_id != selected_candidate_id:
+ continue
+ edge = candidate_edges.get(provisional_wall_id)
+ final_reference = final_by_geometry.get(str(edge.get("geometry_key"))) if isinstance(edge, dict) and edge.get("geometry_key") else None
+ if not final_reference:
+ audit["unmatched_provisional_records"].append({
+ "source_task_id": source_task_id,
+ "candidate_id": candidate_id,
+ "provisional_wall_id": provisional_wall_id or None,
+ "reason": "candidate_edge_or_final_geometry_key_unavailable",
+ })
+ continue
+ key = (source_task_id, provisional_wall_id)
+ if key in seen:
+ continue
+ seen.add(key)
+ record = {
+ "wall_id": final_reference.get("wall_id"),
+ "geometry_key": final_reference.get("geometry_key"),
+ "original_image_visibility": item.get("original_image_visibility"),
+ "weak_wall": item.get("weak_wall"),
+ "closure_only": item.get("closure_only"),
+ "can_anchor_wall_attach": item.get("can_anchor_wall_attach"),
+ "reason": item.get("reason") or item.get("reason_code"),
+ "reason_code": item.get("reason_code"),
+ "candidate_id": candidate_id,
+ "provisional_wall_id": provisional_wall_id,
+ "semantic_binding_source_task_id": source_task_id,
+ }
+ record = {key: value for key, value in record.items() if value is not None}
+ materialized.append(record)
+ audit["matched_provisional_records"].append({
+ "source_task_id": source_task_id,
+ "candidate_id": candidate_id,
+ "provisional_wall_id": provisional_wall_id,
+ "final_wall_id": final_reference.get("wall_id"),
+ "geometry_key": final_reference.get("geometry_key"),
+ })
+ audit["source_task_ids"] = source_ids
+ matched_final_ids = {str(record.get("wall_id")) for record in materialized if record.get("wall_id")}
+ audit["missing_final_wall_ids"] = [wall_id for wall_id in CANONICAL_WALL_IDS if wall_id not in matched_final_ids]
+ if not materialized:
+ audit["status"] = "no_matching_selected_candidate_semantics"
+ return None, audit
+ audit["status"] = "applied" if len(matched_final_ids) == len(CANONICAL_WALL_IDS) else "partially_applied"
+ result = {
+ "task_id": "local_provisional_wall_visibility_binding",
+ "task_type": "local_provisional_wall_visibility_binding",
+ "canonical_task_kind": "local_provisional_wall_visibility_binding",
+ "provided_capabilities": ["wall_visibility_semantics_ready"],
+ "semantic_authority": "selected_candidate_provisional_visibility",
+ "wall_semantic_priority": 300,
+ "selected_candidate_id": selected_candidate_id,
+ "wall_visibility_review": materialized,
+ }
+ return result, audit
+
+
+def materialize_selected_candidate_object_wall_bindings(
+ wall_evidence: dict[str, Any],
+ selected_candidate_id: str | None,
+ task_results: list[dict[str, Any]],
+) -> tuple[dict[str, Any] | None, dict[str, Any]]:
+ """Map Agent candidate-panel object bindings onto final canonical wall ids."""
+ catalog = compact_provisional_wall_visibility_catalog(wall_evidence)
+ selected_candidate = next(
+ (item for item in catalog.get("candidates", []) if item.get("candidate_id") == selected_candidate_id),
+ None,
+ )
+ audit: dict[str, Any] = {
+ "schema_version": "selected_candidate_object_wall_binding",
+ "selected_candidate_id": selected_candidate_id,
+ "binding_policy": "candidate_id_exact_then_provisional_edge_geometry_key_exact",
+ "status": "not_requested",
+ "matched_bindings": [],
+ "unmatched_bindings": [],
+ }
+ if not selected_candidate_id or not isinstance(selected_candidate, dict):
+ audit["status"] = "selected_candidate_unavailable"
+ return None, audit
+
+ candidate_edges = {
+ str(edge.get("provisional_wall_id")): edge
+ for edge in selected_candidate.get("edges", [])
+ if isinstance(edge, dict) and edge.get("provisional_wall_id")
+ }
+ final_by_geometry = {
+ str(reference.get("geometry_key")): reference
+ for reference in formal_wall_references(wall_evidence)
+ if reference.get("geometry_key")
+ }
+ hypotheses: list[dict[str, Any]] = []
+ seen: set[tuple[str, str]] = set()
+ source_task_ids: list[str] = []
+ for root in task_results:
+ if not isinstance(root, dict):
+ continue
+ source_task_id = str(root.get("task_id") or "unknown_task")
+ for node in _walk_mapping_nodes(root):
+ items = node.get("candidate_wall_bindings") if isinstance(node, dict) else None
+ if not isinstance(items, list):
+ continue
+ if source_task_id not in source_task_ids:
+ source_task_ids.append(source_task_id)
+ for item in items:
+ if not isinstance(item, dict):
+ continue
+ candidate_id = str(item.get("candidate_id") or "").strip()
+ provisional_wall_id = str(item.get("provisional_wall_id") or "").strip()
+ object_id = str(item.get("object_id") or "").strip()
+ confidence = str(item.get("confidence") or "").strip().lower()
+ if candidate_id != selected_candidate_id or not object_id or confidence not in {"high", "medium"}:
+ continue
+ edge = candidate_edges.get(provisional_wall_id)
+ reference = final_by_geometry.get(str(edge.get("geometry_key"))) if isinstance(edge, dict) else None
+ if not reference:
+ audit["unmatched_bindings"].append({
+ "source_task_id": source_task_id,
+ "object_id": object_id,
+ "candidate_id": candidate_id,
+ "provisional_wall_id": provisional_wall_id,
+ })
+ continue
+ final_wall_id = str(reference.get("wall_id"))
+ key = (object_id, final_wall_id)
+ if key in seen:
+ continue
+ seen.add(key)
+ hypothesis = {
+ "hypothesis_id": f"selected_panel_binding:{object_id}:{final_wall_id}",
+ "object_id": object_id,
+ "supporting_objects": [object_id],
+ "candidate_wall_id": final_wall_id,
+ "geometry_key": reference.get("geometry_key"),
+ "role": item.get("role"),
+ "confidence": confidence,
+ "evidence_code": item.get("evidence_code"),
+ "source": "selected_candidate_agent_semantic_binding",
+ "candidate_id": candidate_id,
+ "provisional_wall_id": provisional_wall_id,
+ "may_be_used_as_final_wall": False,
+ }
+ hypotheses.append(hypothesis)
+ audit["matched_bindings"].append({
+ "source_task_id": source_task_id,
+ "object_id": object_id,
+ "candidate_id": candidate_id,
+ "provisional_wall_id": provisional_wall_id,
+ "final_wall_id": final_wall_id,
+ "geometry_key": reference.get("geometry_key"),
+ })
+ audit["source_task_ids"] = source_task_ids
+ if not hypotheses:
+ audit["status"] = "no_matching_selected_candidate_bindings"
+ return None, audit
+ audit["status"] = "applied"
+ return {
+ "task_id": "local_selected_candidate_object_wall_binding",
+ "task_type": "local_selected_candidate_object_wall_binding",
+ "canonical_task_kind": "local_selected_candidate_object_wall_binding",
+ "provided_capabilities": ["wall_object_hypotheses_ready"],
+ "selected_candidate_id": selected_candidate_id,
+ "wall_line_hypotheses": hypotheses,
+ }, audit
+
+
+def scene_object_topdown_rotations(scene_state: dict[str, Any]) -> dict[str, float]:
+ candidates = scene_state.get("object_candidates") if isinstance(scene_state, dict) else {}
+ items = candidates.get("items") if isinstance(candidates, dict) else []
+ rotations: dict[str, float] = {}
+ for item in items if isinstance(items, list) else []:
+ if not isinstance(item, dict):
+ continue
+ object_id = str(item.get("object_id") or item.get("id") or "").strip()
+ layout = item.get("layout_reference") if isinstance(item.get("layout_reference"), dict) else {}
+ orientation = layout.get("orientation_for_agent") if isinstance(layout.get("orientation_for_agent"), dict) else {}
+ rotation = orientation.get("rotation_degrees_topdown")
+ if object_id and isinstance(rotation, (int, float)) and not isinstance(rotation, bool):
+ rotations[object_id] = float(rotation)
+ return rotations
+
+
+def _discard_pre_orientation_wall_semantics(results: list[dict[str, Any]]) -> list[str]:
+ """Remove semantic labels that were keyed to a superseded wall candidate."""
+ removed: list[str] = []
+ for result in results:
+ if not isinstance(result, dict):
+ continue
+ for key in ("wall_visibility_review", "walls", "accepted_walls"):
+ if key in result:
+ result.pop(key, None)
+ removed.append(f"{result.get('task_id', 'unknown')}:{key}")
+ return removed
+
+
+def apply_wall_orientation_selection(
+ scene_state: dict[str, Any],
+ semantic_results: list[dict[str, Any]],
+ output_dir: Path,
+) -> dict[str, Any]:
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state.get("evidence"), dict) else {}
+ precompute = evidence.get("render_grid_wall_precompute") if isinstance(evidence.get("render_grid_wall_precompute"), dict) else {}
+ method_path = Path(str(precompute.get("method_candidates_path") or ""))
+ method_data = _load_json_file(method_path) if method_path.exists() else None
+ raw_candidates = [
+ item for item in (method_data.get("candidates", []) if isinstance(method_data, dict) else [])
+ if isinstance(item, dict) and item.get("method") == "obb_from_mesh_footprint_min_rect"
+ ]
+ candidate_ids = [
+ str(item.get("candidate_id") or f"candidate_{index:02d}")
+ for index, item in enumerate(raw_candidates, start=1)
+ ]
+ review = method_data.get("wall_orientation_review") if isinstance(method_data, dict) and isinstance(method_data.get("wall_orientation_review"), dict) else {}
+ baseline_id = review.get("baseline_candidate_id")
+ if baseline_id not in candidate_ids:
+ baseline_id = candidate_ids[0] if candidate_ids else None
+
+ object_score_resolution = resolve_orientation_candidate_from_wall_objects(
+ semantic_results,
+ raw_candidates,
+ scene_object_topdown_rotations(scene_state),
+ )
+ local_choice = object_score_resolution.get("resolved_candidate_id")
+ selected_id = local_choice or baseline_id
+ previous_id = precompute.get("selected_orientation_candidate_id")
+ if selected_id:
+ precompute["selected_orientation_candidate_id"] = selected_id
+ if local_choice:
+ precompute["wall_orientation_selection_source"] = "local_agent_object_rotation_score"
+ else:
+ precompute["wall_orientation_selection_source"] = "precompute_baseline_fallback"
+
+ invalidated_semantics: list[str] = []
+ if (
+ local_choice
+ and previous_id
+ and selected_id != previous_id
+ ):
+ invalidated_semantics = _discard_pre_orientation_wall_semantics(semantic_results)
+
+ audit = {
+ "schema_version": "wall_orientation_selection_audit",
+ "status": "applied" if selected_id else "unavailable",
+ "method_candidates_path": str(method_path) if method_path else None,
+ "available_candidate_ids": candidate_ids,
+ "selection_required": len(candidate_ids) > 1,
+ "selected_candidate_id": selected_id,
+ "previous_selected_candidate_id": previous_id,
+ "selection_source": precompute.get("wall_orientation_selection_source"),
+ "local_object_rotation_scoring": object_score_resolution,
+ "agent_selected_candidate_id_is_authoritative": False,
+ "pre_orientation_wall_semantics_invalidated": invalidated_semantics,
+ "geometry_policy": "candidate extent comes from all posed mesh footprints; Agent selects semantic wall-following objects and local code scores only finite precomputed axes",
+ "wall_attach_dependency": "selected geometry is applied and wall evidence is rebuilt before visibility and wall-attach review",
+ "orientation_support_does_not_imply_attach": True,
+ }
+ write_audit_pair(output_dir, "walls", "wall_orientation_selection", audit, "Wall Orientation Selection")
+ return audit
+
+
+TASK_IMAGE_ARTIFACT_PROFILES = task_image_artifact_profiles()
+
+CAPABILITY_IMAGE_ARTIFACT_PROFILES = {
+ "wall_alignment_objects_ready": ("original_image", "mask_overlay"),
+ "provisional_wall_visibility_ready": ("wall_orientation_overlay",),
+ "wall_visibility_semantics_ready": ("original_image", "wall_attach_prefilter_overlay"),
+ "wall_attach_decisions_ready": ("original_image", "mask_overlay", "wall_attach_prefilter_overlay"),
+ "object_physical_properties_ready": ("original_image", "mask_overlay"),
+ "room_materials_ready": ("original_image",),
+}
+
+NO_IMAGE_TASK_KINDS = {"relation_reconciliation", "final_validation", "final_consistency_validation"}
+
+IMAGE_ARTIFACT_ALIASES = {
+ "original_image_path": "original_image",
+ "reviewed_overlay": "mask_overlay",
+ "reviewed_overlay_path": "mask_overlay",
+ "canonical_topdown_path": "canonical_topdown",
+ "wall_orientation_candidates_overlay": "wall_orientation_overlay",
+ "wall_orientation_candidates_overlay_path": "wall_orientation_overlay",
+}
+
+
+def is_wall_visual_task(task: dict[str, Any]) -> bool:
+ text = task_control_text(task)
+ return any(token in text for token in ["wall", "infer_four_clockwise_walls", "repair_missing_final_walls"])
+
+
+def is_contain_relation_task(task: dict[str, Any]) -> bool:
+ capabilities = task_capabilities(task, "provides_capabilities")
+ if capabilities:
+ return "contain_relations_ready" in capabilities
+ identity = " ".join(
+ str(task.get(key) or "")
+ for key in ("task_id", "id", "task_type", "canonical_task_kind")
+ ).lower()
+ return any(marker in identity for marker in ("contain_relation", "contain relation", "containment", "container"))
+
+
+def compact_image_paths(paths: list[Any], limit: int) -> list[str]:
+ seen: set[str] = set()
+ result: list[str] = []
+ for path in paths:
+ if not path:
+ continue
+ path_s = str(path)
+ if path_s in seen:
+ continue
+ if not Path(path_s).exists():
+ continue
+ seen.add(path_s)
+ result.append(path_s)
+ if len(result) >= limit:
+ break
+ return result
+
+
+def _task_binding_path(task: dict[str, Any], key: str) -> str | None:
+ binding = task.get("evidence_binding")
+ if isinstance(binding, dict) and binding.get(key):
+ return str(binding[key])
+ return None
+
+
+def _wall_image_artifact_paths(task: dict[str, Any], evidence: dict[str, Any]) -> dict[str, Any]:
+ prefilter = task.get("wall_attach_prefilter") if isinstance(task.get("wall_attach_prefilter"), dict) else {}
+ prefilter_overlay = prefilter.get("overlay_path") if prefilter.get("overlay_status") == "available" else None
+ prefilter_overlay = (
+ prefilter_overlay
+ or evidence.get("wall_attach_prefilter_overlay_path")
+ or evidence.get("primary_candidate_overlay_path")
+ or evidence.get("candidate_walls_overlay_path")
+ )
+ return {
+ "original_image": _task_binding_path(task, "original_image") or evidence.get("original_image_path"),
+ "mask_overlay": _task_binding_path(task, "mask_overlay") or evidence.get("mask_overlay_path") or evidence.get("reviewed_overlay_path"),
+ "wall_contact_sheet": evidence.get("contact_sheet_path"),
+ "candidate_wall_overlay": evidence.get("candidate_walls_overlay_path"),
+ "primary_wall_overlay": evidence.get("primary_candidate_overlay_path") or evidence.get("candidate_walls_overlay_path"),
+ "wall_orientation_overlay": evidence.get("render_grid_wall_orientation_candidates_overlay_path") or evidence.get("primary_candidate_overlay_path") or evidence.get("candidate_walls_overlay_path"),
+ "canonical_topdown": _task_binding_path(task, "canonical_topdown") or evidence.get("canonical_topdown_path"),
+ "top_view_clean": evidence.get("top_view_clean_path"),
+ "top_view_grid": evidence.get("top_view_grid_path"),
+ "wall_attach_prefilter_overlay": prefilter_overlay,
+ }
+
+
+def resolve_task_image_evidence(task: dict[str, Any], wall_visual_evidence: dict[str, Any] | None) -> dict[str, Any]:
+ evidence = wall_visual_evidence if isinstance(wall_visual_evidence, dict) else {}
+ available = _wall_image_artifact_paths(task, evidence)
+ canonical_kind = str(task.get("canonical_task_kind") or task.get("task_type") or "").strip().lower()
+ text = task_control_text(task)
+ no_image_markers = ["reconciliation", "coverage_and_repair", "merge_and_coverage", "coverage_repair", "final_validation", "consistency_validation", "relations_validation_and_repair"]
+ if canonical_kind in NO_IMAGE_TASK_KINDS or any(marker in text for marker in no_image_markers):
+ return {"matched": True, "image_paths": [], "selected_artifacts": [], "routing_source": "structured_no_images", "keyword_fallback_used": False}
+
+ artifact_names: list[str] = []
+ routing_sources: list[str] = []
+
+ def add_artifacts(names: Any, source: str) -> None:
+ for name in names or ():
+ normalized = IMAGE_ARTIFACT_ALIASES.get(str(name).strip().lower().replace("-", "_").replace(" ", "_"), str(name).strip().lower().replace("-", "_").replace(" ", "_"))
+ if normalized in available and normalized not in artifact_names:
+ artifact_names.append(normalized)
+ if source not in routing_sources:
+ routing_sources.append(source)
+
+ if canonical_kind == "wall_final_adjudication":
+ if str(task.get("wall_review_mode", "heavy")).lower() == "light":
+ add_artifacts(("original_image", "wall_orientation_overlay", "mask_overlay"), "canonical_task_kind")
+ else:
+ add_artifacts(("wall_contact_sheet", "candidate_wall_overlay", "canonical_topdown", "top_view_clean", "top_view_grid", "original_image", "mask_overlay"), "canonical_task_kind")
+ elif canonical_kind in TASK_IMAGE_ARTIFACT_PROFILES:
+ add_artifacts(TASK_IMAGE_ARTIFACT_PROFILES[canonical_kind], "canonical_task_kind")
+
+ required_artifacts = task_capabilities(task, "requires_artifacts")
+ recognized_required = []
+ for item in required_artifacts:
+ normalized = str(item).strip().lower().replace("-", "_").replace(" ", "_")
+ normalized = IMAGE_ARTIFACT_ALIASES.get(normalized, normalized)
+ if normalized in available:
+ recognized_required.append(normalized)
+ if recognized_required:
+ known_profile = canonical_kind in TASK_IMAGE_ARTIFACT_PROFILES or canonical_kind == "wall_final_adjudication"
+ permitted_required = (
+ [name for name in recognized_required if name in artifact_names] if known_profile else recognized_required
+ )
+ if permitted_required:
+ add_artifacts(permitted_required, "requires_artifacts")
+
+ for capability in task_capabilities(task, "provides_capabilities"):
+ profile = CAPABILITY_IMAGE_ARTIFACT_PROFILES.get(str(capability).strip().lower())
+ if profile:
+ add_artifacts(profile, "provides_capabilities")
+
+ matched = bool(routing_sources) or canonical_kind in TASK_IMAGE_ARTIFACT_PROFILES
+ selected_artifacts = [name for name in artifact_names if available.get(name)]
+ return {
+ "matched": matched,
+ "image_paths": [str(available[name]) for name in selected_artifacts],
+ "selected_artifacts": selected_artifacts,
+ "routing_source": "+".join(routing_sources) if routing_sources else "task_keyword_fallback",
+ "keyword_fallback_used": not matched,
+ }
+
+
+def contain_task_image_paths(task: dict[str, Any], scene_state: dict[str, Any]) -> list[str]:
+ if not is_contain_relation_task(task):
+ return []
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state.get("evidence"), dict) else {}
+ return compact_image_paths(
+ [
+ _task_binding_path(task, "original_image") or evidence.get("original_image_path"),
+ _task_binding_path(task, "mask_overlay") or evidence.get("mask_overlay_path"),
+ ],
+ 2,
+ )
+
+
+def load_contain_candidates_for_task(
+ task: dict[str, Any],
+ scene_state: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+ """Project recall pairs with image localization, never heuristic evidence."""
+ path = _task_binding_path(task, "contain_candidates")
+ data = _load_json_file(Path(path)) if path else None
+ if not isinstance(data, dict):
+ return {"available": False}
+
+ bbox_by_id: dict[str, list[float]] = {}
+ candidates = (
+ scene_state.get("object_candidates", {})
+ if isinstance(scene_state, dict) and isinstance(scene_state.get("object_candidates"), dict)
+ else {}
+ )
+ for item in candidates.get("items", []) if isinstance(candidates.get("items"), list) else []:
+ if not isinstance(item, dict):
+ continue
+ object_id = item.get("object_id") or item.get("id")
+ bbox = item.get("bbox_2d")
+ if not (object_id and isinstance(bbox, list) and len(bbox) >= 4):
+ continue
+ try:
+ bbox_by_id[str(object_id)] = [round(float(value), 6) for value in bbox[:4]]
+ except (TypeError, ValueError):
+ continue
+
+ raw_pairs = data.get("candidate_pairs") if isinstance(data.get("candidate_pairs"), list) else []
+ selection = select_contain_review_pairs(data)
+ pairs: list[dict[str, Any]] = []
+ for selected in selection["selected_pairs"]:
+ item = selected["source"]
+ anchor = selected["object_anchor"]
+ target = selected["object_target"]
+ pair = {
+ "object_anchor": anchor,
+ "object_target": target,
+ "anchor_render_direction": item.get("anchor_render_direction"),
+ "candidate_origin": "local_high_recall_scan",
+ "candidate_is_hint_only": True,
+ "image_bboxes_are_localization_only": True,
+ }
+ if anchor in bbox_by_id:
+ pair["anchor_bbox_2d"] = bbox_by_id[anchor]
+ if target in bbox_by_id:
+ pair["target_bbox_2d"] = bbox_by_id[target]
+ pairs.append(pair)
+
+ possible_containers: list[dict[str, Any]] = []
+ raw_containers = data.get("possible_containers") if isinstance(data.get("possible_containers"), list) else []
+ for item in raw_containers:
+ if not isinstance(item, dict):
+ continue
+ possible_containers.append({
+ key: item.get(key)
+ for key in (
+ "object_id",
+ "category",
+ "description",
+ "render_direction",
+ "anchor_render_direction",
+ )
+ if item.get(key) is not None
+ })
+
+ return {
+ "available": True,
+ "possible_containers": possible_containers,
+ "candidate_pairs": pairs,
+ "candidate_pair_count": selection["valid_pair_count"],
+ "retained_candidate_pair_count": len(pairs),
+ "dropped_candidate_pair_count": len(selection["dropped_pairs"]),
+ "candidate_selection_policy": selection["selection_policy"],
+ "note": (
+ "These pairs only select what to inspect. Per-pair bbox_2d values locate object ids in the original image and "
+ "carry zero positive evidence for contain. Accept only from positive original-image/mask evidence of interior "
+ "membership; add directly visible omissions."
+ ),
+ }
+
+def wall_visual_image_paths(task: dict[str, Any], wall_visual_evidence: dict[str, Any] | None) -> list[str]:
+ routing = resolve_task_image_evidence(task, wall_visual_evidence)
+ if routing.get("matched"):
+ return list(routing.get("image_paths", []))
+ if not isinstance(wall_visual_evidence, dict) or not is_wall_visual_task(task):
+ return []
+ task_type = str(task.get("task_type", "")).lower()
+ task_id = task_identifier(task).lower()
+ text = task_control_text(task)
+
+ def compact(paths: list[Any], limit: int) -> list[str]:
+ seen: set[str] = set()
+ result: list[str] = []
+ for path in paths:
+ if not path:
+ continue
+ path_s = str(path)
+ if path_s in seen:
+ continue
+ seen.add(path_s)
+ result.append(path_s)
+ if len(result) >= limit:
+ break
+ return result
+
+ original = wall_visual_evidence.get("original_image_path")
+ mask = wall_visual_evidence.get("mask_overlay_path") or wall_visual_evidence.get("reviewed_overlay_path")
+ contact = wall_visual_evidence.get("contact_sheet_path")
+ candidate_overlay = wall_visual_evidence.get("candidate_walls_overlay_path")
+ canonical = wall_visual_evidence.get("canonical_topdown_path")
+ top_clean = wall_visual_evidence.get("top_view_clean_path")
+ top_grid = wall_visual_evidence.get("top_view_grid_path")
+
+ # These are structured merge/check tasks. Sending images again is expensive and usually redundant.
+ if any(marker in text for marker in ["reconciliation", "coverage_and_repair", "merge_and_coverage", "coverage_repair", "final_validation", "final_scene_delivery_validation", "final_consistency_validation", "consistency_validation", "relations_validation_and_repair"]):
+ return []
+ if "support_contact" in text:
+ return compact([original, mask, top_clean, canonical, candidate_overlay], 4)
+ if task_type == "wall_object_hypothesis":
+ return compact([original, mask, contact, canonical, top_clean, candidate_overlay], 6)
+ if task_type == "wall_final_adjudication":
+ if str(task.get("wall_review_mode", "heavy")).lower() == "light":
+ primary_overlay = wall_visual_evidence.get("primary_candidate_overlay_path")
+ return compact([original, primary_overlay or candidate_overlay, canonical, mask], 4)
+ return compact([contact, candidate_overlay, canonical, top_clean, top_grid, original, mask], 7)
+ if "wall_attach" in task_type or "wall_attach" in task_id:
+ prefilter = task.get("wall_attach_prefilter") if isinstance(task.get("wall_attach_prefilter"), dict) else {}
+ prefilter_overlay = prefilter.get("overlay_path") if prefilter.get("overlay_status") == "available" else None
+ if prefilter_overlay:
+ return compact([original, mask, prefilter_overlay], 3)
+ return compact([original, mask, contact, canonical, candidate_overlay], 5)
+
+ paths = wall_visual_evidence.get("image_paths_for_wall_repair") or []
+ if not isinstance(paths, list):
+ return []
+ return compact(paths, 5)
+
+def wall_visual_payload(wall_visual_evidence: dict[str, Any] | None) -> dict[str, Any]:
+ if not isinstance(wall_visual_evidence, dict):
+ return {"available": False}
+ keys = [
+ "manifest_path",
+ "contact_sheet_path",
+ "candidate_walls_overlay_path",
+ "canonical_topdown_path",
+ "canonical_grid_mapping_path",
+ "render_grid_method_candidates_path",
+ "render_grid_affine_path",
+ "render_grid_topdown_path",
+ "render_grid_topdown_grid_path",
+ "render_grid_foreground_mask_path",
+ "render_grid_candidate_overlay_path",
+ "render_grid_wall_orientation_candidates_overlay_path",
+ "render_grid_primary_candidate_overlay_path",
+ "top_view_clean_path",
+ "top_view_grid_path",
+ "front_view_path",
+ "back_view_path",
+ "left_view_path",
+ "right_view_path",
+ "original_image_path",
+ "mask_overlay_path",
+ "reviewed_overlay_path",
+ "image_paths_for_wall_repair",
+ "wall_evidence_path",
+ "candidate_walls_count",
+ "candidate_walls_source",
+ "candidate_walls_trust_level",
+ "candidate_walls_may_be_final",
+ "candidate_room_polygon_hint",
+ "failure_summary",
+ "next_suggestions",
+ ]
+ return {key: wall_visual_evidence.get(key) for key in keys}
+
+
+def compact_object_summary(scene_state: dict[str, Any]) -> dict[str, Any]:
+ items = scene_state.get("object_candidates", {}).get("items", []) if isinstance(scene_state.get("object_candidates"), dict) else []
+ compact_items = []
+ for item in items if isinstance(items, list) else []:
+ if not isinstance(item, dict):
+ continue
+ compact_items.append(
+ {
+ "object_id": item.get("object_id"),
+ "category": item.get("category"),
+ "description": item.get("description"),
+ "has_mask": bool(item.get("mask_path")),
+ "has_layout_bbox": bool(item.get("bbox_3d")),
+ "bbox_3d": item.get("bbox_3d"),
+ }
+ )
+ return {"count": len(compact_items), "items": compact_items}
+
+
+def build_manager_planning_payload(
+ args: argparse.Namespace,
+ scene_state: dict[str, Any],
+ render_capabilities: dict[str, Any],
+ wall_evidence: dict[str, Any],
+ wall_visual_evidence: dict[str, Any],
+) -> dict[str, Any]:
+ fixed = scene_state.get("evidence", {}).get("fixed_layout_views", {}) if isinstance(scene_state.get("evidence"), dict) else {}
+ topdown = scene_state.get("topdown_grid_mapping", {}) if isinstance(scene_state.get("topdown_grid_mapping"), dict) else {}
+ return {
+ "planning_payload_mode": "light",
+ "dry_run": args.dry_run,
+ "scene_summary": {
+ "scene_name": scene_state.get("scene_name"),
+ "run_root": scene_state.get("run_root"),
+ "objects": compact_object_summary(scene_state),
+ "topdown_status": topdown.get("status"),
+ "topdown_mapping_path": topdown.get("canonical_mapping_path"),
+ "fixed_layout_views_status": fixed.get("status"),
+ "fixed_layout_views_manifest_path": fixed.get("manifest_path"),
+ },
+ "available_evidence_summary": {
+ "original_image_path": scene_state.get("evidence", {}).get("original_image_path"),
+ "mask_overlay_path": scene_state.get("evidence", {}).get("mask_overlay_path"),
+ "reviewed_objects_path": scene_state.get("evidence", {}).get("reviewed_objects_path"),
+ "wall_visual_manifest_path": wall_visual_evidence.get("manifest_path") if isinstance(wall_visual_evidence, dict) else None,
+ "candidate_walls_overlay_path": wall_visual_evidence.get("candidate_walls_overlay_path") if isinstance(wall_visual_evidence, dict) else None,
+ "image_paths_for_wall_repair_count": len(wall_visual_evidence.get("image_paths_for_wall_repair", [])) if isinstance(wall_visual_evidence, dict) else 0,
+ },
+ "wall_policy_summary": {
+ "wall_candidate_mode": wall_evidence.get("wall_candidate_mode"),
+ "active_wall_candidate_source": wall_evidence.get("active_wall_candidate_source"),
+ "candidate_walls_count": len(wall_evidence.get("candidate_walls", [])),
+ "render_grid_wall_status": wall_evidence.get("render_grid_wall_candidates", {}).get("status"),
+ "runner_controls_wall_tasks": True,
+ "manager_should_not_generate_wall_extraction_tasks": True,
+ "runner_injected_wall_tasks": [
+ "task_wall_object_hypothesis",
+ "task_wall_final_adjudication",
+ "task_wall_attach_candidate_recall",
+ ],
+ },
+ "relation_policy_summary": {
+ "allowed_relation_types": ["attach", "contain"],
+ "contain_is_same_level_as_attach": True,
+ "same_anchor_target_pair_attach_and_contain_are_mutually_exclusive": True,
+ "multi_attach_allowed": True,
+ "floor_support_is_primary_but_not_exclusive": True,
+ "floor_supported_against_wall_objects_may_have_both_floor_and_wall_attach": True,
+ "weak_or_inferred_hidden_walls_cannot_anchor_attach": True,
+ "each_object_must_appear_at_least_once_as_object_target": True,
+ "ceiling_anchor_requires_positive_judgement": True,
+ },
+ "render_capabilities_summary": {
+ "select_partial_objects": render_capabilities.get("capabilities", {}).get("select_partial_objects"),
+ "topdown_render": render_capabilities.get("capabilities", {}).get("topdown_render"),
+ "reuse_grid_mapping": render_capabilities.get("capabilities", {}).get("reuse_grid_mapping"),
+ },
+ "manager_instructions": [
+ "Plan only non-wall support tasks that are useful for final attach extraction; the runner will inject and control A-line wall tasks.",
+ "Preserve multi-agent autonomy: decide whether ceiling judgement, object revision, support-contact extraction, and validation/repair tasks are needed for this scene.",
+ "For every planned task, include depends_on when the task needs outputs from earlier tasks. Use [] for independent tasks.",
+ "Prefer support_contact_extraction as an independent first-layer task when object_list and object_footprints are sufficient. Add an object-revision dependency only when the support decision genuinely requires a corrected object identity.",
+ "For every planned task, include stable routing fields: canonical_task_kind, runner_action, and runtime_class. Use runner_action=local_schema_validation and runtime_class=local_redundant_validation for final validation/coverage/merge checks that the local assembler or schema guard can perform instead of an API task.",
+ "Do not use old wall extraction methods or request layout_walls_from_06_layout as a final wall source.",
+ "Allow an object to have both floor attach and wall attach when floor support and wall contact are both physically supported.",
+ "Plan contain_relation_extraction when the original image or object list positively shows a concrete target inside an ordinary container or on/within any functional level of multi-level furniture. For bunk beds, bookcases/shelving units, multi-tier service carts, and multi-level racks, every occupied level including the top is contain and the same pair must not emit attach. For ordinary single-level furniture, a plain exterior top remains attach. Anchor existence alone is insufficient when empty.",
+ ],
+ "policy_excerpt": {
+ "final_objects_keep_sam3d_object_ids": True,
+ "final_relations_allow_attach_and_contain": True,
+ "category_description_changes_require_object_revision_audit": True,
+ },
+ }
+
+
+
+
+def load_expected_relation_manifest(run_root: Path | None) -> dict[str, Any] | None:
+ path = relation_layout_expected_manifest_path(run_root)
+ if not path:
+ return None
+ manifest = _load_json_file(path)
+ if isinstance(manifest, dict):
+ manifest = dict(manifest)
+ manifest["expected_manifest_path"] = str(path)
+ return manifest
+ return None
+
+
+def expected_relation_manifest_readiness(
+ manifest: dict[str, Any] | None,
+) -> tuple[bool, str]:
+ """Require a real object index before asynchronous Manager planning starts."""
+ if not isinstance(manifest, dict):
+ return False, "expected_manifest_missing_or_invalid"
+ objects = manifest.get("objects")
+ if not isinstance(objects, list) or not objects:
+ return False, "expected_manifest_objects_empty"
+ object_ids = [
+ str(item.get("object_id") or "").strip()
+ for item in objects
+ if isinstance(item, dict)
+ ]
+ if len(object_ids) != len(objects) or any(not object_id for object_id in object_ids):
+ return False, "expected_manifest_object_id_missing"
+ try:
+ object_count = int(manifest.get("object_count"))
+ except (TypeError, ValueError):
+ return False, "expected_manifest_object_count_invalid"
+ if object_count <= 0 or object_count != len(objects):
+ return False, "expected_manifest_object_count_mismatch"
+ return True, "planning_ready_with_nonempty_object_index"
+
+
+def wait_for_expected_relation_manifest(
+ args: argparse.Namespace,
+ run_root: Path | None,
+ *,
+ poll_seconds: float = 1.0,
+) -> tuple[dict[str, Any] | None, dict[str, Any]]:
+ """Poll the lightweight manifest only; full relation preprocessing stays async."""
+ wait_seconds = max(0.0, float(getattr(args, "relation_precompute_wait_seconds", 0) or 0))
+ deadline = time.time() + wait_seconds
+ attempts = 0
+ latest: dict[str, Any] | None = None
+ ready = False
+ reason = "expected_manifest_not_checked"
+ while True:
+ attempts += 1
+ latest = load_expected_relation_manifest(run_root)
+ ready, reason = expected_relation_manifest_readiness(latest)
+ if ready or time.time() >= deadline:
+ break
+ time.sleep(max(0.1, poll_seconds))
+ return latest, {
+ "schema_version": "expected_manifest_readiness",
+ "status": "ready" if ready else "not_ready",
+ "reason": reason,
+ "poll_interval_seconds": max(0.1, poll_seconds),
+ "wait_budget_seconds": wait_seconds,
+ "poll_attempts": attempts,
+ "object_count": latest.get("object_count") if isinstance(latest, dict) else None,
+ "expected_manifest_path": latest.get("expected_manifest_path") if isinstance(latest, dict) else None,
+ "policy": "Manager planning and Physics Sidecar may start before full preprocessing, but never before a nonempty, internally consistent object index exists.",
+ }
+
+
+def launch_async_physics_sidecar(
+ args: argparse.Namespace,
+ expected_manifest: dict[str, Any] | None,
+ output_dir: Path,
+) -> tuple[subprocess.Popen[Any] | None, dict[str, Any]]:
+ sidecar_dir = output_dir / "physics_sidecar" / "integrated_async"
+ manifest_path = (
+ Path(expected_manifest["expected_manifest_path"])
+ if isinstance(expected_manifest, dict) and expected_manifest.get("expected_manifest_path")
+ else None
+ )
+ metadata: dict[str, Any] = {
+ "mode": "async",
+ "relation_critical_path_dependency": False,
+ "output_dir": str(sidecar_dir),
+ "object_physics_path": str(sidecar_dir / "object_physics.json"),
+ "started": False,
+ }
+ if not manifest_path or not manifest_path.exists():
+ metadata["status"] = "skipped_missing_expected_manifest"
+ return None, metadata
+
+ sidecar_dir.mkdir(parents=True, exist_ok=True)
+ command = [
+ sys.executable,
+ str(RELATIONS_ROOT / "object_physics" / "run_sidecar.py"),
+ "--scene_name", args.scene_name,
+ "--run_root", str(args.run_root.resolve()),
+ "--expected_manifest", str(manifest_path),
+ "--sidecar_id", "integrated_async",
+ "--output_dir", str(sidecar_dir),
+ "--model", str(getattr(args, "physics_model", None) or args.model),
+ "--vision_model", str(getattr(args, "physics_vision_model", None) or args.vision_model),
+ "--api_key_env", args.api_key_env,
+ "--vision_image_detail", str(getattr(args, "physics_vision_image_detail", "low")),
+ ]
+ if args.api_base:
+ command.extend(["--api_base", args.api_base])
+ if getattr(args, "physics_include_original", False):
+ command.append("--include_original")
+ if args.dry_run:
+ command.append("--dry_run")
+
+ stdout_path = sidecar_dir / "process_stdout.log"
+ stderr_path = sidecar_dir / "process_stderr.log"
+ environment = os.environ.copy()
+ environment["PYTHONPATH"] = str(CODE_ROOT) + (
+ os.pathsep + environment["PYTHONPATH"] if environment.get("PYTHONPATH") else ""
+ )
+ with stdout_path.open("w", encoding="utf-8") as stdout_file, stderr_path.open("w", encoding="utf-8") as stderr_file:
+ process = subprocess.Popen(
+ command,
+ stdout=stdout_file,
+ stderr=stderr_file,
+ env=environment,
+ start_new_session=True,
+ )
+ metadata.update({
+ "status": "running",
+ "started": True,
+ "pid": process.pid,
+ "expected_manifest_path": str(manifest_path),
+ "stdout_path": str(stdout_path),
+ "stderr_path": str(stderr_path),
+ "command": command,
+ })
+ write_json(sidecar_dir / "launch_manifest.json", metadata)
+ return process, metadata
+
+
+def snapshot_physics_sidecar(
+ process: subprocess.Popen[Any] | None,
+ metadata: dict[str, Any],
+) -> dict[str, Any]:
+ result = dict(metadata)
+ if process is not None:
+ return_code = process.poll()
+ result["return_code"] = return_code
+ result["status"] = "running" if return_code is None else ("completed" if return_code == 0 else "failed")
+ physics_path = Path(result["object_physics_path"]) if result.get("object_physics_path") else None
+ result["object_physics_available"] = bool(physics_path and physics_path.exists())
+ if result.get("output_dir"):
+ output_dir = Path(result["output_dir"])
+ runtime_path = output_dir / "runtime.json"
+ result["runtime_path"] = str(runtime_path)
+ if runtime_path.exists():
+ try:
+ result["sidecar_runtime"] = json.loads(runtime_path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ result["runtime_read_error"] = str(exc)
+ write_json(output_dir / "integration_status.json", result)
+ return result
+
+
+def build_compact_manager_skeleton_payload(
+ args: argparse.Namespace,
+ expected_manifest: dict[str, Any] | None,
+ scene_state: dict[str, Any],
+ render_capabilities: dict[str, Any],
+) -> dict[str, Any]:
+ objects = []
+ if isinstance(expected_manifest, dict) and isinstance(expected_manifest.get("objects"), list):
+ objects = expected_manifest.get("objects", [])
+ elif isinstance(scene_state.get("object_candidates"), dict):
+ objects = compact_object_summary(scene_state).get("items", [])
+ artifacts = expected_manifest.get("artifacts", {}) if isinstance(expected_manifest, dict) and isinstance(expected_manifest.get("artifacts"), dict) else {}
+ compact_artifacts = {
+ str(key): {
+ "status": value.get("status"),
+ "description": value.get("description"),
+ }
+ for key, value in artifacts.items()
+ if isinstance(value, dict)
+ }
+ return {
+ "manager_plan_format": "compact",
+ "planning_payload_mode": "compact_skeleton_async",
+ "scene_summary": {
+ "scene_name": args.scene_name,
+ "objects": {"count": len(objects), "items": objects},
+ },
+ "expected_artifact_contract": compact_artifacts,
+ "capability_catalog": compact_manager_capability_catalog(),
+ "runner_contract": {
+ "formal_wall_method": "obb_from_mesh_footprint_min_rect",
+ "runner_controlled_wall_capabilities": [
+ "wall_object_hypotheses_ready",
+ "wall_visibility_semantics_ready",
+ "wall_attach_decisions_ready",
+ ],
+ "local_capabilities": [
+ "wall_attach_candidate_prefilter",
+ "final_assembly",
+ "schema_validation",
+ "coverage_fallback",
+ ],
+ "manager_must_not_plan_local_validation": True,
+ },
+ "relation_contract": {
+ "allowed_relation_types": ["attach", "contain"],
+ "same_pair_attach_contain_requires_schema_guard_arbitration": True,
+ "floor_support_and_wall_contact_may_coexist": True,
+ "task_object_scopes_are_not_mutually_exclusive": True,
+ "object_anchor_routing_modes": [
+ "ceiling_only", "wall_only", "floor_wall",
+ "support_only", "contain_candidate", "uncertain",
+ ],
+ "high_confidence_exclusive_routes_only": True,
+ "floor_wall_routes_to_support_and_wall": True,
+ "every_object_requires_target_coverage": True,
+ "ceiling_requires_positive_judgement": True,
+ },
+ "compact_task_contract": {
+ "required_fields": [
+ "node_id",
+ "canonical_task_kind",
+ "provides_capabilities",
+ "depends_on",
+ "reason",
+ "object_scope",
+ "scene_focus",
+ ],
+ "registered_tasks_must_not_repeat_execution_boilerplate": True,
+ "custom_tasks_require": ["custom_instructions", "expected_output_schema"],
+ },
+ "render_capabilities_summary": {
+ "select_partial_objects": render_capabilities.get("capabilities", {}).get("select_partial_objects"),
+ "topdown_render": render_capabilities.get("capabilities", {}).get("topdown_render"),
+ "reuse_grid_mapping": render_capabilities.get("capabilities", {}).get("reuse_grid_mapping"),
+ },
+ }
+
+
+
+def build_manager_skeleton_payload(
+ args: argparse.Namespace,
+ expected_manifest: dict[str, Any] | None,
+ scene_state: dict[str, Any],
+ render_capabilities: dict[str, Any],
+) -> dict[str, Any]:
+ payload = build_compact_manager_skeleton_payload(
+ args,
+ expected_manifest,
+ scene_state,
+ render_capabilities,
+ )
+ payload["planning_mode"] = "skeleton_async"
+ payload["available_evidence_summary"] = {
+ "original_image_available": bool(
+ scene_state.get("evidence", {}).get("original_image_path")
+ ),
+ "mask_overlay_available": bool(
+ scene_state.get("evidence", {}).get("mask_overlay_path")
+ ),
+ "full_evidence_available": bool(
+ scene_state.get("evidence", {}).get("wall_visual_evidence")
+ ),
+ }
+ return payload
+
+
+def manager_skeleton_image_paths(args: argparse.Namespace, expected_manifest: dict[str, Any] | None) -> list[str]:
+ artifact_path = None
+ if isinstance(expected_manifest, dict):
+ artifacts = expected_manifest.get("artifacts")
+ if isinstance(artifacts, dict) and isinstance(artifacts.get("original_image"), dict):
+ artifact_path = artifacts["original_image"].get("path")
+ fallback = Path(args.run_root) / "input" / "image.png" if getattr(args, "run_root", None) else None
+ return compact_image_paths([artifact_path, str(fallback) if fallback else None], 1)
+
+
+def write_manager_skeleton_plan(output_dir: Path, plan: dict[str, Any]) -> None:
+ write_json(output_dir / "manager_plans" / "manager_plan_skeleton_000.json", plan)
+ tasks = plan.get("tasks", []) if isinstance(plan.get("tasks"), list) else []
+ lines = [
+ "- planning_payload_mode: `skeleton_async`",
+ f"- task_count: `{len(tasks)}`",
+ "",
+ "## Skeleton Tasks",
+ ]
+ for task in tasks:
+ if isinstance(task, dict):
+ lines.append(f"- `{task.get('task_id')}`: {task.get('canonical_task_kind') or task.get('task_type')} requires={task.get('requires_artifacts', [])}")
+ write_md(output_dir / "manager_plans" / "manager_plan_skeleton_000.md", "Manager Plan Skeleton 000", lines)
+
+
+def compile_manager_plan_response(
+ args: argparse.Namespace,
+ raw_plan: dict[str, Any],
+ output_dir: Path,
+) -> tuple[dict[str, Any], dict[str, Any] | None]:
+ write_json(output_dir / "manager_plans" / "manager_plan_compact_000.json", raw_plan)
+ compiled, audit = compile_compact_manager_plan(raw_plan)
+ audit["source_path"] = str(output_dir / "manager_plans" / "manager_plan_compact_000.json")
+ audit["compiled_plan_path"] = str(output_dir / "manager_plans" / "manager_plan_000.json")
+ write_json(output_dir / "manager_plans" / "manager_plan_compilation_audit.json", audit)
+ return compiled, audit
+
+
+def update_manager_compilation_audit_with_wall_policy(
+ output_dir: Path,
+ manager_plan: dict[str, Any],
+) -> None:
+ path = output_dir / "manager_plans" / "manager_plan_compilation_audit.json"
+ if not path.exists():
+ return
+ try:
+ audit = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError):
+ audit = {}
+ if not isinstance(audit, dict):
+ audit = {}
+ audit["wall_capability_policy"] = {
+ "wall_task_policy": manager_plan.get("wall_task_policy"),
+ "manager_plan_repaired": bool(manager_plan.get("manager_plan_repaired")),
+ "manager_plan_repairs": manager_plan.get("manager_plan_repairs", []),
+ "formal_wall_task_control": manager_plan.get("formal_wall_task_control", {}),
+ "field_source": "manager_then_runner_policy",
+ }
+ write_json(path, audit)
+
+
+def manager_prompt_name() -> str:
+ return "scene_delivery_manager_compact_system.txt"
+
+
+def annotate_manager_api_call(
+ runtime_tracker: RuntimeTracker,
+ manager_client: SceneDeliveryAPIClient,
+ payload: dict[str, Any],
+ prompt_text: str,
+ args: argparse.Namespace,
+ response_plan: dict[str, Any] | None = None,
+) -> None:
+ """Attach compact-manager size/routing metadata to the latest API call."""
+ if not runtime_tracker.api_calls:
+ return
+ item = runtime_tracker.api_calls[-1]
+ item.update({
+ **manager_client.usage_metadata(),
+ "manager_plan_format": "compact",
+ "manager_image_detail": getattr(manager_client.config, "image_detail", None),
+ "manager_output_task_count": (
+ len(response_plan.get("tasks", []))
+ if isinstance(response_plan, dict) and isinstance(response_plan.get("tasks"), list)
+ else None
+ ),
+ })
+
+
+
+def available_relation_artifacts(run_root: Path | None, scene_state: dict[str, Any]) -> dict[str, dict[str, Any]]:
+ expected = load_expected_relation_manifest(run_root)
+ manifest = load_relation_layout_manifest(run_root)
+ outputs = manifest.get("outputs", {}) if isinstance(manifest, dict) and isinstance(manifest.get("outputs"), dict) else {}
+ artifacts: dict[str, dict[str, Any]] = {}
+ if isinstance(expected, dict) and isinstance(expected.get("artifacts"), dict):
+ for key, raw in expected["artifacts"].items():
+ if isinstance(raw, dict):
+ artifacts[str(key)] = dict(raw)
+ path_by_key = {
+ "object_list": outputs.get("objects_path") or outputs.get("layout_for_relation_path"),
+ "object_footprints": outputs.get("object_footprints_path"),
+ "formal_obb_walls": outputs.get("obb_walls_path"),
+ "canonical_topdown": outputs.get("canonical_topdown_image_path"),
+ "render_grid_affine": outputs.get("render_grid_affine_path"),
+ "wall_evidence": outputs.get("wall_evidence_path"),
+ "contain_candidates": outputs.get("contain_candidates_path"),
+ "contain_visual_evidence": outputs.get("contain_visual_evidence_path"),
+ "fixed_views": outputs.get("fixed_views_manifest_path"),
+ "scene_state": outputs.get("scene_state_path"),
+ }
+ for key, path in path_by_key.items():
+ artifact = artifacts.setdefault(key, {"description": f"Relation artifact {key}"})
+ if path:
+ artifact["path"] = str(path)
+ artifact["status"] = "available" if Path(str(path)).exists() else "pending"
+ else:
+ artifact.setdefault("status", "missing")
+ topdown = scene_state.get("topdown_grid_mapping", {}) if isinstance(scene_state.get("topdown_grid_mapping"), dict) else {}
+ if topdown.get("canonical_mapping_path"):
+ artifacts.setdefault("canonical_grid_mapping", {"description": "Canonical top-down grid mapping"})
+ artifacts["canonical_grid_mapping"].update({
+ "path": topdown.get("canonical_mapping_path"),
+ "status": "available" if Path(str(topdown.get("canonical_mapping_path"))).exists() else "pending",
+ })
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state.get("evidence"), dict) else {}
+ visual = evidence.get("wall_visual_evidence", {}) if isinstance(evidence.get("wall_visual_evidence"), dict) else {}
+ fixed = evidence.get("fixed_layout_views", {}) if isinstance(evidence.get("fixed_layout_views"), dict) else {}
+ alias_paths = {
+ "original_image": evidence.get("original_image_path"),
+ "mask_overlay": evidence.get("mask_overlay_path"),
+ "fixed_views": fixed.get("manifest_path"),
+ "wall_visual_evidence": visual.get("manifest_path"),
+ "wall_visual_manifest": visual.get("manifest_path"),
+ "contact_sheet": visual.get("contact_sheet_path"),
+ "candidate_walls_overlay": visual.get("candidate_walls_overlay_path"),
+ "primary_candidate_overlay": visual.get("primary_candidate_overlay_path"),
+ }
+ for key, path in alias_paths.items():
+ if path:
+ artifacts.setdefault(key, {"description": f"Hydrated evidence alias {key}"})
+ artifacts[key].update({
+ "path": str(path),
+ "status": "available" if Path(str(path)).exists() else "pending",
+ })
+ return artifacts
+
+
+def hydrate_manager_plan(manager_plan: dict[str, Any], run_root: Path | None, scene_state: dict[str, Any]) -> dict[str, Any]:
+ plan = json.loads(json.dumps(manager_plan))
+ artifacts = available_relation_artifacts(run_root, scene_state)
+ tasks = [task for task in plan.get("tasks", []) if isinstance(task, dict)]
+ object_ids = _scene_state_object_ids(scene_state)
+ object_anchor_routing = plan.get("object_anchor_routing", [])
+ planned_anchor_modes: set[str] = set()
+ if any(is_support_contact_task(task) for task in tasks):
+ planned_anchor_modes.add("support_only")
+ if any(
+ any(capability.startswith("wall_") for capability in task_capabilities(task, "provides_capabilities"))
+ or str(task.get("canonical_task_kind") or task.get("task_type") or "").lower().startswith("wall_")
+ for task in tasks
+ ):
+ planned_anchor_modes.add("wall_only")
+ if any(
+ "ceiling_judgement_ready" in task_capabilities(task, "provides_capabilities")
+ or str(task.get("canonical_task_kind") or task.get("task_type") or "").lower() == "ceiling_judgement"
+ for task in tasks
+ ):
+ planned_anchor_modes.add("ceiling_only")
+ if any(is_contain_relation_task(task) for task in tasks):
+ planned_anchor_modes.add("contain_candidate")
+ contain_scope_hydrations: list[dict[str, Any]] = []
+ support_scope_hydrations: list[dict[str, Any]] = []
+ anchor_routing_scope_hydrations: list[dict[str, Any]] = []
+ for task in tasks:
+ routing_scope_audit = hydrate_task_scope_from_anchor_routing(
+ task,
+ object_ids,
+ object_anchor_routing,
+ planned_anchor_modes,
+ )
+ if isinstance(routing_scope_audit, dict):
+ anchor_routing_scope_hydrations.append({
+ "task_id": task_identifier(task),
+ **routing_scope_audit,
+ })
+ required = task.get("requires_artifacts")
+ if not isinstance(required, list):
+ required = task.get("evidence_needed") if isinstance(task.get("evidence_needed"), list) else []
+ normalized_required = [str(item) for item in required]
+ bindings: dict[str, Any] = {}
+ missing: list[str] = []
+ for key in normalized_required:
+ artifact = artifacts.get(key)
+ if isinstance(artifact, dict) and artifact.get("status") == "available" and artifact.get("path"):
+ bindings[key] = artifact.get("path")
+ else:
+ missing.append(key)
+ task["requires_artifacts"] = normalized_required
+ task["evidence_binding"] = bindings
+ task["evidence_readiness"] = {
+ "status": "ready" if not missing else "missing_optional_or_pending",
+ "missing_or_pending_artifacts": missing,
+ }
+ if bindings:
+ append_task_instruction(task, "Use the hydrated evidence_binding paths as the concrete artifact locations for this task.")
+ if is_contain_relation_task(task):
+ candidate_path = bindings.get("contain_candidates")
+ candidate_data = _load_json_file(Path(str(candidate_path))) if candidate_path else None
+ if isinstance(candidate_data, dict):
+ contain_scope_hydrations.append({
+ "task_id": task_identifier(task),
+ **hydrate_contain_candidate_scope(task, candidate_data),
+ })
+ if is_support_contact_task(task):
+ support_scope_hydrations.append({
+ "task_id": task_identifier(task),
+ **hydrate_support_capability_scope(
+ task,
+ object_ids,
+ object_anchor_routing,
+ planned_anchor_modes,
+ ),
+ })
+ plan["tasks"] = tasks
+ plan["manager_planning_mode"] = "skeleton_async_hydrated"
+ plan["hydration"] = {
+ "status": "applied",
+ "artifact_count": len(artifacts),
+ "available_artifacts": sorted(key for key, value in artifacts.items() if isinstance(value, dict) and value.get("status") == "available"),
+ "artifacts": artifacts,
+ "object_anchor_routing": object_anchor_routing,
+ "planned_anchor_modes": sorted(planned_anchor_modes),
+ "anchor_routing_scope_hydrations": anchor_routing_scope_hydrations,
+ "contain_scope_hydrations": contain_scope_hydrations,
+ "support_scope_hydrations": support_scope_hydrations,
+ }
+ return plan
+
+
+def _unique_runner_task_id(tasks: list[dict[str, Any]], base_id: str) -> str:
+ used = {task_identifier(task) for task in tasks}
+ if base_id not in used:
+ return base_id
+ suffix = 2
+ while f"{base_id}_{suffix}" in used:
+ suffix += 1
+ return f"{base_id}_{suffix}"
+
+
+def _compile_runner_capability_task(
+ tasks: list[dict[str, Any]],
+ canonical_task_kind: str,
+ base_task_id: str,
+ reason: str,
+) -> dict[str, Any]:
+ node_id = _unique_runner_task_id(tasks, base_task_id)
+ compiled, _ = compile_compact_manager_plan({
+ "plan_id": "runner_capability_completion",
+ "objective": "Complete missing required relation capabilities without replacing the Manager DAG.",
+ "tasks": [{
+ "node_id": node_id,
+ "canonical_task_kind": canonical_task_kind,
+ "depends_on": [],
+ "reason": reason,
+ "object_scope": [],
+ "scene_focus": reason,
+ }],
+ })
+ task = compiled["tasks"][0]
+ task["manager_plan_repair"] = {
+ "source": "runner_capability_completion",
+ "reason": reason,
+ }
+ return task
+
+
+def ensure_required_relation_capabilities(
+ manager_plan: dict[str, Any],
+ run_root: Path | None,
+ scene_state: dict[str, Any],
+) -> dict[str, Any]:
+ """Repair only missing relation capabilities, leaving Manager grouping intact."""
+ plan = json.loads(json.dumps(manager_plan))
+ tasks = [task for task in plan.get("tasks", []) if isinstance(task, dict)]
+ object_ids = _scene_state_object_ids(scene_state)
+ repairs: list[dict[str, Any]] = []
+
+ if not any(task_provides_capability(task, "room_materials_ready") for task in tasks):
+ task = _compile_runner_capability_task(
+ tasks,
+ "room_material_classification",
+ "task_room_materials_capability_completion",
+ "Final objects.json requires image-grounded walls, floor, and ceiling material descriptions; the Manager omitted a provider for room_materials_ready.",
+ )
+ tasks.append(task)
+ repairs.append({
+ "repair_reason": "final_output_requires_room_materials_capability",
+ "capability": "room_materials_ready",
+ "injected_task_ids": [task_identifier(task)],
+ })
+
+ if object_ids and not any(task_provides_capability(task, "support_relations_ready") for task in tasks):
+ task = _compile_runner_capability_task(
+ tasks,
+ "support_contact_extraction",
+ "task_support_contact_capability_completion",
+ "Every reconstructed object requires an explicit gravity-support verdict; the Manager omitted a provider for support_relations_ready.",
+ )
+ tasks.append(task)
+ repairs.append({
+ "repair_reason": "nonempty_scene_requires_gravity_support_capability",
+ "capability": "support_relations_ready",
+ "injected_task_ids": [task_identifier(task)],
+ })
+
+ artifacts = available_relation_artifacts(run_root, scene_state)
+ contain_artifact = artifacts.get("contain_candidates") if isinstance(artifacts.get("contain_candidates"), dict) else {}
+ contain_path = contain_artifact.get("path") if contain_artifact.get("status") == "available" else None
+ contain_data = _load_json_file(Path(str(contain_path))) if contain_path else None
+ selection = select_contain_review_pairs(contain_data) if isinstance(contain_data, dict) else {
+ "selected_pairs": [],
+ "valid_pair_count": 0,
+ "selection_policy": "no_available_candidates",
+ }
+ strong_pairs = [
+ pair for pair in selection.get("selected_pairs", [])
+ if float(pair.get("confidence_hint", 0.0)) >= CONTAIN_SCOPE_CONFIDENCE_THRESHOLD
+ ]
+ routing = plan.get("object_anchor_routing") if isinstance(plan.get("object_anchor_routing"), list) else []
+ routing_object_ids = [
+ str(item.get("object_id"))
+ for item in routing
+ if isinstance(item, dict)
+ and str(item.get("anchor_mode") or "").lower() == "contain_candidate"
+ and str(item.get("confidence") or "low").lower() in {"high", "medium"}
+ and item.get("object_id")
+ ]
+ contain_signal = bool(strong_pairs or routing_object_ids)
+ if contain_signal and not any(task_provides_capability(task, "contain_relations_ready") for task in tasks):
+ task = _compile_runner_capability_task(
+ tasks,
+ "contain_relation_extraction",
+ "task_contain_relation_capability_completion",
+ "Strong local contain recall or Manager anchor routing requires semantic contain review; candidates remain hints and require original-image verification.",
+ )
+ tasks.append(task)
+ repairs.append({
+ "repair_reason": "strong_contain_signal_requires_semantic_review_capability",
+ "capability": "contain_relations_ready",
+ "injected_task_ids": [task_identifier(task)],
+ "strong_candidate_pair_count": len(strong_pairs),
+ "routing_object_ids": routing_object_ids,
+ })
+
+ plan["tasks"] = tasks
+ plan["relation_capability_completion"] = {
+ "policy": "capability_floor_only_manager_retains_task_grouping_and_scene_reasoning",
+ "nonempty_object_count": len(object_ids),
+ "contain_candidate_path": str(contain_path) if contain_path else None,
+ "contain_valid_pair_count": int(selection.get("valid_pair_count", 0)),
+ "strong_contain_pair_count": len(strong_pairs),
+ "contain_routing_object_ids": routing_object_ids,
+ "repairs": repairs,
+ }
+ if repairs:
+ plan["manager_plan_repaired"] = True
+ existing_repairs = plan.get("manager_plan_repairs") if isinstance(plan.get("manager_plan_repairs"), list) else []
+ plan["manager_plan_repairs"] = [*existing_repairs, *repairs]
+ return plan
+
+
+def _add_matching_task_ids(deps: list[str], known_task_ids: set[str], current_task_id: str, *markers: str) -> None:
+ for candidate in sorted(known_task_ids):
+ candidate_l = candidate.lower()
+ if candidate != current_task_id and any(marker in candidate_l for marker in markers) and candidate not in deps:
+ deps.append(candidate)
+
+
+def infer_task_dependencies(
+ task: dict[str, Any],
+ known_task_ids: set[str],
+ capability_providers: dict[str, list[str]] | None = None,
+) -> list[str]:
+ raw = task.get("depends_on")
+ task_id = task_identifier(task)
+ task_type = str(task.get("task_type") or "").lower()
+ text = task_control_text(task)
+ is_support_task = is_support_contact_task(task)
+ is_wall_final_task = task_id == "task_wall_final_adjudication" or task_type == "wall_final_adjudication"
+ provides_alignment = task_provides_capability(task, "wall_alignment_objects_ready")
+ provides_provisional_visibility = task_provides_capability(task, PROVISIONAL_WALL_VISIBILITY_CAPABILITY)
+ provides_visibility = task_provides_capability(task, "wall_visibility_semantics_ready")
+ provides_wall_attach = task_provides_capability(task, "wall_attach_decisions_ready")
+ deps: list[str] = []
+ capability_providers = capability_providers or {}
+
+ def add(dep: str) -> None:
+ if dep in known_task_ids and dep != task_id and dep not in deps:
+ deps.append(dep)
+
+ if isinstance(raw, list):
+ for item in raw:
+ dep = str(item)
+ dep_l = dep.lower()
+ # Support/contact extraction is the gravity/local-contact branch. It does not need wall-attach recall.
+ # Keeping manager's wall-attach dependency here serializes an otherwise independent task by ~40s.
+ if is_support_task and ("wall_attach" in dep_l or dep_l == "task_wall_final_adjudication"):
+ continue
+ if is_wall_final_task and "wall_attach" in dep_l:
+ continue
+ add(dep)
+ if (is_wall_final_task or provides_visibility or provides_wall_attach) and not provides_alignment:
+ alignment_providers = capability_providers.get("wall_alignment_objects_ready", [])
+ for provider_id in alignment_providers:
+ add(provider_id)
+ if not alignment_providers:
+ add("task_wall_alignment_object_review")
+ if task_id == "task_wall_attach_candidate_recall" or "wall_attach_candidate_recall" in task_type or "wall_attach_candidate_recall" in text or provides_wall_attach:
+ # Candidate-panel visibility is optional Manager-selected evidence. If
+ # present, it must finish before local prefiltering/attach review so
+ # weak or inferred candidate edges can be excluded deterministically.
+ provisional_providers = capability_providers.get(PROVISIONAL_WALL_VISIBILITY_CAPABILITY, [])
+ for provider_id in provisional_providers:
+ add(provider_id)
+ # Recall tasks that do not provide final visibility depend on a
+ # separate visibility provider.
+ if not provides_visibility:
+ semantic_providers = capability_providers.get("wall_visibility_semantics_ready", [])
+ for provider_id in semantic_providers:
+ add(provider_id)
+ if not semantic_providers:
+ add("task_wall_final_adjudication")
+ elif is_support_task:
+ # Gravity/local-contact extraction is independently useful from the original
+ # object list and footprints. Do not infer object-revision, ceiling-review,
+ # or wall dependencies; preserve them only when the Manager explicitly plans them.
+ pass
+ elif (
+ "relations_validation" in task_id
+ or "validation_and_repair" in task_id
+ or "final_validation" in text
+ or ("validation" in text and "relation" in text)
+ ):
+ add("task_wall_attach_candidate_recall")
+ _add_matching_task_ids(deps, known_task_ids, task_id, "ceiling")
+ _add_matching_task_ids(deps, known_task_ids, task_id, "object_revision")
+ _add_matching_task_ids(deps, known_task_ids, task_id, "support_contact", "support_relation")
+ return deps
+
+def apply_relation_policy_and_dependencies(manager_plan: dict[str, Any]) -> dict[str, Any]:
+ tasks = [task for task in manager_plan.get("tasks", []) if isinstance(task, dict)]
+ known_ids = {task_identifier(task) for task in tasks}
+ capability_providers: dict[str, list[str]] = {}
+ for provider_task in tasks:
+ for capability in task_capabilities(provider_task, "provides_capabilities"):
+ capability_providers.setdefault(capability, []).append(task_identifier(provider_task))
+ dual_attach_instruction = (
+ "Allow multi-attach: floor support is usually the primary support relation, but it does not exclude an additional wall attach when a floor-supported object's back/side is physically against a verified wall."
+ )
+ weak_wall_attach_instruction = (
+ "Do not attach objects to weak, hidden, not_visible, closure_only, or inferred-by-room-closure walls when the source image cannot show the actual wall surface/contact; those walls may be valid room-closure geometry but not valid attach anchors. Use the canonical final wall ids after wall_final_adjudication, not provisional candidate ids if they were renamed."
+ )
+ wall_visibility_evidence_instruction = (
+ "An image crop boundary, furniture side panel, occlusion contour, or room-closure edge is not evidence that a wall is visible. Mark visible or partially_visible only when the actual wall surface or a wall-floor/wall-ceiling junction is directly visible in the source image."
+ )
+ for task in tasks:
+ task_id = task_identifier(task)
+ task_type = str(task.get("task_type") or "").lower()
+ text = " ".join(_as_instruction_list(task.get("instructions"))).lower()
+ if "wall_attach" in task_type or "support_contact" in task_id or "support contact" in text or "reconciliation" in task_id:
+ append_task_instruction(task, dual_attach_instruction)
+ append_task_instruction(task, weak_wall_attach_instruction)
+ if (
+ "wall_attach" in task_type
+ or task_provides_capability(task, "wall_attach_decisions_ready")
+ or task_provides_capability(task, "wall_visibility_semantics_ready")
+ or task_provides_capability(task, PROVISIONAL_WALL_VISIBILITY_CAPABILITY)
+ ):
+ append_task_instruction(task, wall_visibility_evidence_instruction)
+ deps = infer_task_dependencies(task, known_ids, capability_providers)
+ task["depends_on"] = deps
+ manager_plan["tasks"] = tasks
+ manager_plan["task_dependency_policy"] = {
+ "mode": "runner_inferred_dag_dependencies",
+ "multi_attach_allowed": True,
+ "floor_support_is_primary_but_not_exclusive": True,
+ "wall_attach_requires_verification": True,
+ "support_contact_default_independent_of_object_revision": True,
+ }
+ return manager_plan
+
+
+FAST_PRESET_PRUNE_MARKERS = (
+ "wall_attach_reconciliation",
+ "relation_reconciliation",
+ "reconciliation_after_wall_tasks",
+ "relation_coverage_and_repair",
+ "relation_merge_and_coverage_repair",
+ "wall_aware_attach_merge_and_repair",
+ "attach_merge_and_repair",
+ "merge_and_repair",
+ "merge_and_coverage",
+ "coverage_repair",
+ "final_validation",
+ "final_scene_delivery_validation",
+ "final_consistency_validation",
+ "consistency_validation",
+ "relations_validation_and_repair",
+)
+FAST_PRESET_LOCAL_RUNNER_ACTIONS = {
+ "local_attach_merge_check",
+ "local_final_assembly_validation",
+ "local_final_validation",
+ "local_relation_coverage_check",
+ "local_schema_validation",
+}
+FAST_PRESET_PRUNE_RUNTIME_CLASSES = {
+ "coverage_repair",
+ "final_validation",
+ "local_redundant_validation",
+ "merge_check",
+ "optional_review",
+ "reconciliation",
+}
+
+
+def apply_runtime_preset(manager_plan: dict[str, Any]) -> dict[str, Any]:
+ tasks = [task for task in manager_plan.get("tasks", []) if isinstance(task, dict)]
+ retained: list[dict[str, Any]] = []
+ pruned: list[dict[str, Any]] = []
+ for task in tasks:
+ task_id = task_identifier(task).lower()
+ task_type = str(task.get("task_type") or "").lower()
+ text = task_control_text(task)
+ runner_action = str(task.get("runner_action") or "").strip().lower()
+ runtime_class = str(task.get("runtime_class") or "").strip().lower()
+ if (
+ runner_action in FAST_PRESET_LOCAL_RUNNER_ACTIONS
+ or runtime_class in FAST_PRESET_PRUNE_RUNTIME_CLASSES
+ or any(marker in text for marker in FAST_PRESET_PRUNE_MARKERS)
+ ):
+ pruned.append(task)
+ else:
+ retained.append(task)
+ retained_ids = {task_identifier(task) for task in retained}
+ for task in retained:
+ deps = task.get("depends_on", []) if isinstance(task.get("depends_on"), list) else []
+ task["depends_on"] = [dep for dep in deps if dep in retained_ids]
+ manager_plan["tasks"] = retained
+ manager_plan["task_runtime_preset"] = {
+ "mode": "fast",
+ "target_seconds": 240,
+ "pruned_task_ids": [task_identifier(task) for task in pruned],
+ "pruned_task_types": [task.get("task_type") for task in pruned],
+ "pruned_runner_actions": [task.get("runner_action") for task in pruned],
+ "pruned_runtime_classes": [task.get("runtime_class") for task in pruned],
+ "policy": "Preserve manager planning, but prune optional reconciliation/coverage/final-validation DynamicTaskAgent calls that local assembler, schema guard, and validation can cover under the 4-minute runtime target.",
+ }
+ return manager_plan
+
+
+def task_dependency_layers(tasks: list[dict[str, Any]]) -> list[list[dict[str, Any]]]:
+ pending = {task_identifier(task): task for task in tasks if isinstance(task, dict)}
+ completed: set[str] = set()
+ layers: list[list[dict[str, Any]]] = []
+ while pending:
+ ready = [task for task_id, task in pending.items() if all(dep in completed for dep in task.get("depends_on", []))]
+ if not ready:
+ # Break dependency cycles conservatively by running the remaining tasks in their current order.
+ ready = list(pending.values())
+ ready_ids = {task_identifier(task) for task in ready}
+ layers.append(ready)
+ completed.update(ready_ids)
+ pending = {task_id: task for task_id, task in pending.items() if task_id not in ready_ids}
+ return layers
+
+
+
+def _compact_object_record(item: dict[str, Any]) -> dict[str, Any]:
+ """Keep relation-relevant geometry, discard local paths and mesh details."""
+ bbox_3d = item.get("bbox_3d") if isinstance(item.get("bbox_3d"), dict) else {}
+ layout = item.get("layout_reference", {}) if isinstance(item.get("layout_reference"), dict) else {}
+ orientation = layout.get("orientation_for_agent", {}) if isinstance(layout.get("orientation_for_agent"), dict) else {}
+ return {
+ "object_id": item.get("object_id") or item.get("id"),
+ "category": item.get("category"),
+ "description": item.get("description"),
+ "bbox_2d": item.get("bbox_2d"),
+ "bbox_3d": {
+ "center": bbox_3d.get("center"),
+ "extent": bbox_3d.get("extent"),
+ "min": bbox_3d.get("min"),
+ "max": bbox_3d.get("max"),
+ },
+ "rotation_degrees_topdown": orientation.get("rotation_degrees_topdown"),
+ }
+
+
+def compact_object_index(scene_state: dict[str, Any], object_ids: set[str] | None = None) -> dict[str, Any]:
+ candidates = scene_state.get("object_candidates", {}) if isinstance(scene_state.get("object_candidates"), dict) else {}
+ items = candidates.get("items", []) if isinstance(candidates.get("items"), list) else []
+ compact_items = []
+ for item in items:
+ if not isinstance(item, dict):
+ continue
+ object_id = str(item.get("object_id") or item.get("id") or "")
+ if object_ids is not None and object_id not in object_ids:
+ continue
+ compact_items.append(_compact_object_record(item))
+ return {
+ "coordinate_system": "OpenGL original XZ; Y is vertical",
+ "items": compact_items,
+ "full_object_count": len(items),
+ "included_object_count": len(compact_items),
+ }
+
+
+def compact_wall_metrics(records: Any, object_ids: set[str] | None = None) -> list[dict[str, Any]]:
+ result: list[dict[str, Any]] = []
+ for item in records if isinstance(records, list) else []:
+ if not isinstance(item, dict):
+ continue
+ object_id = str(item.get("object_id") or "")
+ if object_ids is not None and object_id not in object_ids:
+ continue
+ flat = item.get("flat_contact_edge_analysis") if isinstance(item.get("flat_contact_edge_analysis"), dict) else {}
+ best_edge = flat.get("best_edge") if isinstance(flat.get("best_edge"), dict) else {}
+ result.append({
+ "object_id": object_id,
+ "category": item.get("category"),
+ "wall_id": item.get("wall_id"),
+ "footprint_min_distance_to_wall": item.get("footprint_min_distance_to_wall"),
+ "bbox_side_near_wall": item.get("bbox_side_near_wall"),
+ "possible_against_wall_contact": item.get("possible_against_wall_contact"),
+ "is_floor_supported": item.get("is_floor_supported"),
+ "is_thin_planar_candidate": item.get("is_thin_planar_candidate"),
+ "flat_contact_edge": {
+ "analysis_available": flat.get("analysis_available"),
+ "has_valid_flat_contact_edge": flat.get("has_valid_flat_contact_edge"),
+ "support_type": best_edge.get("support_type"),
+ },
+ })
+ return result
+
+
+def compact_prior_task_results(task: dict[str, Any], prior_task_results: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ """Only keep dependency conclusions that a later task can consume."""
+ canonical_kind = str(task.get("canonical_task_kind") or task.get("task_type") or "").lower()
+ selected_keys = {
+ "wall_final_adjudication": {"wall_line_hypotheses", "likely_wall_objects", "wall_visibility_review", "walls", "decision"},
+ "wall_attach_candidate_recall": {"wall_visibility_review", "walls", "decision", "wall_line_hypotheses"},
+ }.get(canonical_kind, set())
+ if not selected_keys:
+ return []
+ compact: list[dict[str, Any]] = []
+ for result in prior_task_results:
+ if not isinstance(result, dict):
+ continue
+ payload = {key: result.get(key) for key in selected_keys if result.get(key) is not None}
+ if payload:
+ compact.append({"task_id": result.get("task_id"), "task_type": result.get("task_type"), "result": payload})
+ return compact
+
+
+def compact_output_schema(value: Any) -> Any:
+ """Replace verbose audit prose fields in the transport schema only."""
+ if isinstance(value, dict):
+ compact = {key: compact_output_schema(item) for key, item in value.items() if key not in {"reason", "evidence"}}
+ if "reason" in value:
+ compact["reason_code"] = "short_snake_case_code"
+ if "evidence" in value:
+ compact["evidence_codes"] = ["short_snake_case_code"]
+ return compact
+ if isinstance(value, list):
+ return [compact_output_schema(item) for item in value]
+ return value
+
+
+def compact_task_definition(task: dict[str, Any]) -> dict[str, Any]:
+ """The Agent needs task intent, not local artifact paths or hydration audits."""
+ return {
+ "task_id": task.get("task_id"),
+ "task_type": task.get("task_type"),
+ "canonical_task_kind": task.get("canonical_task_kind"),
+ "depends_on": task.get("depends_on", []),
+ "provides_capabilities": task_capabilities(task, "provides_capabilities"),
+ "object_scope": task.get("object_scope", []),
+ "scene_focus": task.get("scene_focus"),
+ "scene_focus_evidence_status": "planning_hypothesis_only" if task.get("scene_focus") else None,
+ "instructions": _as_instruction_list(task.get("instructions")),
+ "expected_output_schema": compact_output_schema(task.get("expected_output_schema", {})),
+ }
+
+
+def compact_task_api_context(
+ task: dict[str, Any],
+ prior_task_results: list[dict[str, Any]],
+ scene_state: dict[str, Any],
+ wall_evidence: dict[str, Any],
+) -> dict[str, Any]:
+ canonical_kind = str(task.get("canonical_task_kind") or task.get("task_type") or "").lower()
+ prefilter = task.get("wall_attach_prefilter") if isinstance(task.get("wall_attach_prefilter"), dict) else {}
+ candidate_ids = {
+ str(item.get("object_target"))
+ for item in prefilter.get("candidate_pairs", [])
+ if isinstance(item, dict) and item.get("object_target")
+ }
+ candidate_ids.update(str(item) for item in prefilter.get("supplemental_visual_review_object_ids", []) if item)
+ object_ids = candidate_ids if canonical_kind == "wall_attach_candidate_recall" and candidate_ids else None
+ context: dict[str, Any] = {
+ "schema_version": "task_api_context_balanced",
+ "scene_name": scene_state.get("scene_name"),
+ "task": compact_task_definition(task),
+ "object_index": compact_object_index(scene_state, object_ids),
+ "prior_task_results": compact_prior_task_results(task, prior_task_results),
+ }
+ provides_provisional_visibility = task_provides_capability(task, PROVISIONAL_WALL_VISIBILITY_CAPABILITY)
+ if canonical_kind in {"wall_alignment_object_review", "wall_object_hypothesis", "provisional_wall_visibility_review"}:
+ for item in context["object_index"].get("items", []):
+ if isinstance(item, dict):
+ item.pop("rotation_degrees_topdown", None)
+ if canonical_kind == "contain_relation_extraction":
+ context["contain_candidates"] = load_contain_candidates_for_task(task, scene_state)
+ if canonical_kind in WALL_CONTEXT_TASK_KINDS:
+ if canonical_kind in {"wall_alignment_object_review", "wall_object_hypothesis"}:
+ context["wall_alignment_contract"] = {
+ "agent_output": "semantic object ids plus candidate-panel object-to-provisional-wall bindings",
+ "local_inputs_not_sent": "finite candidate coordinates and rotation_degrees_topdown remain local",
+ "local_next_step": "score finite OBB axes, then bind only the selected panel's object-wall records by immutable geometry",
+ "candidate_id_selection_allowed": False,
+ "final_wall_id_or_attach_decision_allowed": False,
+ }
+ context["provisional_wall_candidate_catalog"] = compact_provisional_wall_visibility_catalog(wall_evidence)
+ if provides_provisional_visibility:
+ context["provisional_wall_visibility_contract"] = {
+ "agent_output": "candidate_id plus provisional_wall_id visibility semantics only",
+ "final_wall_id_selection_allowed": False,
+ "local_next_step": "bind only records belonging to the locally selected finite candidate by immutable edge geometry",
+ }
+ context["provisional_wall_candidate_catalog"] = compact_provisional_wall_visibility_catalog(wall_evidence)
+ elif canonical_kind == "provisional_wall_visibility_review" or provides_provisional_visibility:
+ context["provisional_wall_visibility_contract"] = {
+ "agent_output": "candidate_id plus provisional_wall_id visibility semantics only",
+ "final_wall_id_selection_allowed": False,
+ "local_next_step": "bind only records belonging to the locally selected finite candidate by immutable edge geometry",
+ }
+ context["provisional_wall_candidate_catalog"] = compact_provisional_wall_visibility_catalog(wall_evidence)
+ elif canonical_kind == "wall_attach_candidate_recall" and prefilter:
+ context["wall_context"] = compact_wall_evidence_for_recall(wall_evidence, prefilter)
+ context["wall_attach_prefilter"] = {
+ "candidate_pairs": prefilter.get("candidate_pairs", []),
+ "supplemental_visual_review_object_ids": prefilter.get("supplemental_visual_review_object_ids", []),
+ "candidate_pair_count": prefilter.get("candidate_pair_count", len(prefilter.get("candidate_pairs", []))),
+ "raw_object_wall_pair_count": prefilter.get("raw_object_wall_pair_count"),
+ "blocked_wall_ids": prefilter.get("blocked_wall_ids", {}),
+ }
+ else:
+ context["wall_context"] = {
+ "coordinate_source": "obb_from_mesh_footprint_min_rect_selected_orientation",
+ "formal_wall_references": formal_wall_references(wall_evidence),
+ "wall_orientation_selection": compact_formal_wall_selection(wall_evidence),
+ "wall_attach_metric_summary": compact_wall_metrics(wall_evidence.get("wall_attach_geometry")),
+ }
+ return context
+
+
+SCHEMA_GUARD_WALL_FIELDS = (
+ "wall_id",
+ "original_image_visibility",
+ "weak_wall",
+ "closure_only",
+ "can_anchor_wall_attach",
+ "can_anchor_attach",
+ "wall_attach_allowed",
+ "decision",
+ "reason_code",
+ "evidence_codes",
+)
+SCHEMA_GUARD_OBJECT_PATCH_KEYS = (
+ "object_revisions",
+ "object_revision_candidates",
+ "revision_patches",
+ "object_revision_patches",
+ "accepted_object_revision_patches",
+)
+SCHEMA_GUARD_PHYSICAL_KEYS = (
+ "object_physical_properties",
+ "physical_properties",
+ "object_properties",
+ "object_property_assessments",
+)
+
+
+def _short_codes(value: Any, limit: int = 6) -> list[str]:
+ if not isinstance(value, list):
+ return []
+ return [str(item)[:96] for item in value if item is not None][:limit]
+
+
+def _guard_relation_from_node(node: dict[str, Any], task_result: dict[str, Any], source_path: str) -> dict[str, Any] | None:
+ parameters = node.get("parameters") if isinstance(node.get("parameters"), dict) else {}
+ anchor = (
+ parameters.get("object_anchor")
+ or parameters.get("anchor")
+ or node.get("object_anchor")
+ or node.get("anchor")
+ or node.get("candidate_wall_id")
+ )
+ target = parameters.get("object_target") or parameters.get("target") or node.get("object_target") or node.get("target")
+ if not anchor or not target:
+ return None
+ relation = str(node.get("relation") or "").lower()
+ if relation not in {"attach", "contain"}:
+ task_kind = str(task_result.get("canonical_task_kind") or task_result.get("task_type") or "").lower()
+ relation = "contain" if "contain" in task_kind else "attach"
+ decision = node.get("decision") or node.get("verdict") or node.get("status") or "proposed"
+ reason_code = node.get("reason_code") or node.get("reason") or node.get("rationale")
+ evidence_codes = node.get("evidence_codes") or node.get("evidence") or []
+ return {
+ "relation": relation,
+ "object_anchor": str(anchor),
+ "object_target": str(target),
+ "decision": str(decision)[:48],
+ "reason_code": str(reason_code)[:120] if reason_code else None,
+ "evidence_codes": _short_codes(evidence_codes),
+ "source_task_id": task_result.get("task_id"),
+ "source_task_kind": task_result.get("canonical_task_kind") or task_result.get("task_type"),
+ "source_path": source_path,
+ }
+
+
+def _walk_guard_nodes(value: Any, path: str = "$"):
+ if isinstance(value, dict):
+ yield path, value
+ for key, child in value.items():
+ if key in {"mesh_floor_footprint_hull_xz", "object_footprints", "raw_geometry_evidence", "vertices", "faces"}:
+ continue
+ yield from _walk_guard_nodes(child, f"{path}.{key}")
+ elif isinstance(value, list):
+ for index, child in enumerate(value):
+ yield from _walk_guard_nodes(child, f"{path}[{index}]")
+
+
+def compact_schema_guard_relation_catalog(task_results: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ """Keep every task-produced relation decision without forwarding task geometry."""
+ catalog: list[dict[str, Any]] = []
+ seen: set[tuple[str, str, str, str, str]] = set()
+ for task_result in task_results:
+ if not isinstance(task_result, dict):
+ continue
+ for path, node in _walk_guard_nodes(task_result):
+ if not isinstance(node, dict):
+ continue
+ candidate = _guard_relation_from_node(node, task_result, path)
+ if candidate is None:
+ continue
+ key = (
+ candidate["relation"],
+ candidate["object_anchor"],
+ candidate["object_target"],
+ candidate["decision"],
+ str(candidate["source_task_id"]),
+ )
+ if key not in seen:
+ seen.add(key)
+ catalog.append(candidate)
+ return catalog
+
+
+def _compact_wall_semantic_records(task_results: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ records: list[dict[str, Any]] = []
+ seen: set[tuple[str, str]] = set()
+ for task_result in task_results:
+ if not isinstance(task_result, dict):
+ continue
+ for path, node in _walk_guard_nodes(task_result):
+ if not isinstance(node, dict) or not node.get("wall_id"):
+ continue
+ if not any(field in node for field in SCHEMA_GUARD_WALL_FIELDS[1:]):
+ continue
+ record = {field: node.get(field) for field in SCHEMA_GUARD_WALL_FIELDS if node.get(field) is not None}
+ record["source_task_id"] = task_result.get("task_id")
+ record["source_path"] = path
+ key = (str(record.get("wall_id")), str(record.get("source_task_id")))
+ if key not in seen:
+ seen.add(key)
+ records.append(record)
+ return records
+
+
+def _compact_named_task_values(task_result: dict[str, Any], keys: tuple[str, ...]) -> dict[str, Any]:
+ compact: dict[str, Any] = {}
+ for key in keys:
+ value = task_result.get(key)
+ if value is None:
+ continue
+ if isinstance(value, list):
+ compact[key] = [
+ {
+ item_key: item.get(item_key)
+ for item_key in (
+ "object_id", "id", "new_category", "new_description",
+ "is_floor_covering", "mass", "linear_damping", "angular_damping",
+ "static_friction", "dynamic_friction", "decision", "reason_code",
+ )
+ if item.get(item_key) is not None
+ }
+ for item in value if isinstance(item, dict)
+ ]
+ elif isinstance(value, (str, bool, int, float)):
+ compact[key] = value
+ return compact
+
+
+def compact_schema_guard_context(
+ scene_state: dict[str, Any],
+ task_results: list[dict[str, Any]],
+ supervisor_result: dict[str, Any],
+ wall_evidence: dict[str, Any],
+ contain_conflict_report: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+ """Build the API projection for DeliverySchemaGuard.
+
+ Complete evidence remains local for final assembly, geometry checks, and
+ audit. SchemaGuard receives the conclusions it needs to reconcile rather
+ than megabytes of mesh-footprint samples.
+ """
+ task_summaries: list[dict[str, Any]] = []
+ for task_result in task_results:
+ if not isinstance(task_result, dict):
+ continue
+ summary = {
+ "task_id": task_result.get("task_id"),
+ "task_type": task_result.get("task_type"),
+ "canonical_task_kind": task_result.get("canonical_task_kind"),
+ "provided_capabilities": task_result.get("provided_capabilities", []),
+ }
+ summary.update(_compact_named_task_values(task_result, SCHEMA_GUARD_OBJECT_PATCH_KEYS))
+ summary.update(_compact_named_task_values(task_result, SCHEMA_GUARD_PHYSICAL_KEYS))
+ for key in ("ceiling_confirmed", "ceiling_anchor_allowed", "ceiling_present_for_attach", "decision"):
+ if task_result.get(key) is not None:
+ summary[key] = task_result.get(key)
+ task_summaries.append(summary)
+
+ object_items = compact_object_index(scene_state).get("items", [])
+ object_index = [
+ {
+ "object_id": item.get("object_id"),
+ "category": item.get("category"),
+ "description": item.get("description"),
+ }
+ for item in object_items
+ ]
+ supervisor = supervisor_result if isinstance(supervisor_result, dict) else {}
+ contain_conflicts = compact_contain_conflicts_for_guard(contain_conflict_report)
+ instructions = (
+ "Reconcile only the supplied candidate catalog and conclusion summaries. "
+ "Do not infer a new object-wall pair or geometry not present in this context. "
+ "The local final assembler owns formal OBB geometry and deterministic schema validation."
+ )
+ if contain_conflicts:
+ instructions += (
+ " For every contain_conflicts item, including a high-confidence local pair that the contain specialist marked uncertain, return one explicit contain_conflict_resolutions verdict: "
+ "accept_contain with support_surface_role=internal_shelf when the original-image crop positively shows "
+ "membership in an open or closed shelf/cubby/interior volume, or reject_contain_use_support with "
+ "support_surface_role=outer_top|external_surface and a concrete support_anchor (floor or object_###). "
+ "For multi-level furniture, every occupied functional level remains contain even though it supplies gravity support. "
+ "This includes both bunks and every level of a bookcase/shelving unit, multi-tier service cart, or rack, including "
+ "a plain top level. Do not preserve attach for the same pair; use support_surface_role=multi_level_tier. For an "
+ "ordinary single-level anchor, a plain exterior top remains attach. Use unknown when tier membership is ambiguous. "
+ "Neither relation type has an unconditional priority, and local candidate scores or bbox overlap are not proof. "
+ "The attached crop is limited to these listed pairs; do not add new pairs."
+ )
+ return {
+ "schema_version": "delivery_schema_guard_context_balanced",
+ "scene_name": scene_state.get("scene_name"),
+ "object_index": object_index,
+ "canonical_walls": formal_wall_references(wall_evidence),
+ "wall_semantic_records": _compact_wall_semantic_records(task_results),
+ "relation_candidate_catalog": compact_schema_guard_relation_catalog(task_results),
+ "contain_conflicts": contain_conflicts,
+ "task_conclusion_summaries": task_summaries,
+ "supervisor": {
+ "decision": supervisor.get("decision"),
+ "ready_for_schema_guard": supervisor.get("ready_for_schema_guard"),
+ "reason": str(supervisor.get("reason") or "")[:320] or None,
+ "requested_next_tasks": [str(item)[:180] for item in supervisor.get("requested_next_tasks", []) if item][:8],
+ },
+ "instructions": instructions,
+ }
+
+
+def _normalized_image_bbox(value: Any) -> tuple[float, float, float, float] | None:
+ if not isinstance(value, (list, tuple)) or len(value) < 4:
+ return None
+ try:
+ x0, y0, x1, y1 = (float(value[index]) for index in range(4))
+ except (TypeError, ValueError):
+ return None
+ if not all(0.0 <= item <= 1.0 for item in (x0, y0, x1, y1)):
+ return None
+ return min(x0, x1), min(y0, y1), max(x0, x1), max(y0, y1)
+
+
+def build_contain_conflict_contact_sheet(
+ scene_state: dict[str, Any],
+ contain_conflict_report: dict[str, Any] | None,
+ output_dir: Path,
+) -> dict[str, Any]:
+ """Create one small original-image crop sheet only when a tie-break is needed."""
+ conflicts = (contain_conflict_report or {}).get("conflicts", [])
+ if not isinstance(conflicts, list) or not conflicts:
+ audit = {
+ "schema_version": "contain_conflict_review_manifest",
+ "status": "not_needed",
+ "conflict_count": 0,
+ "contact_sheet_path": None,
+ "items": [],
+ }
+ write_audit_pair(output_dir, "relations", "contain_conflict_review_manifest", audit, "Contain Conflict Review Manifest")
+ return audit
+
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state, dict) else {}
+ original_path = evidence.get("original_image_path") if isinstance(evidence, dict) else None
+ if not original_path or not Path(str(original_path)).exists():
+ audit = {
+ "schema_version": "contain_conflict_review_manifest",
+ "status": "unavailable",
+ "reason": "original_image_path unavailable",
+ "conflict_count": len(conflicts),
+ "contact_sheet_path": None,
+ "items": [],
+ }
+ write_audit_pair(output_dir, "relations", "contain_conflict_review_manifest", audit, "Contain Conflict Review Manifest")
+ return audit
+
+ try:
+ from PIL import Image, ImageDraw
+ original = Image.open(str(original_path)).convert("RGB")
+ except Exception as exc: # Pillow/image decode is optional evidence only.
+ audit = {
+ "schema_version": "contain_conflict_review_manifest",
+ "status": "unavailable",
+ "reason": f"could not render local crop: {type(exc).__name__}: {exc}",
+ "conflict_count": len(conflicts),
+ "contact_sheet_path": None,
+ "items": [],
+ }
+ write_audit_pair(output_dir, "relations", "contain_conflict_review_manifest", audit, "Contain Conflict Review Manifest")
+ return audit
+
+ width, height = original.size
+ tiles: list[tuple[Any, str]] = []
+ items: list[dict[str, Any]] = []
+ for index, conflict in enumerate(conflicts[:4], start=1):
+ if not isinstance(conflict, dict):
+ continue
+ anchor_bbox = _normalized_image_bbox(conflict.get("anchor_image_bbox"))
+ target_bbox = _normalized_image_bbox(conflict.get("target_image_bbox"))
+ if not anchor_bbox or not target_bbox:
+ items.append(
+ {
+ "conflict_id": conflict.get("conflict_id"),
+ "status": "skipped",
+ "reason": "normalized anchor/target image bbox unavailable",
+ }
+ )
+ continue
+ x0 = max(0.0, min(anchor_bbox[0], target_bbox[0]) - 0.08)
+ y0 = max(0.0, min(anchor_bbox[1], target_bbox[1]) - 0.08)
+ x1 = min(1.0, max(anchor_bbox[2], target_bbox[2]) + 0.08)
+ y1 = min(1.0, max(anchor_bbox[3], target_bbox[3]) + 0.08)
+ crop_box = (
+ int(x0 * width),
+ int(y0 * height),
+ max(int(x1 * width), int(x0 * width) + 1),
+ max(int(y1 * height), int(y0 * height) + 1),
+ )
+ crop = original.crop(crop_box)
+ draw = ImageDraw.Draw(crop)
+ for bbox, color, label in ((anchor_bbox, "#2f7df6", "A"), (target_bbox, "#ee4b3c", "T")):
+ left = int((bbox[0] - x0) / max(x1 - x0, 1e-6) * crop.width)
+ top = int((bbox[1] - y0) / max(y1 - y0, 1e-6) * crop.height)
+ right = int((bbox[2] - x0) / max(x1 - x0, 1e-6) * crop.width)
+ bottom = int((bbox[3] - y0) / max(y1 - y0, 1e-6) * crop.height)
+ draw.rectangle((left, top, right, bottom), outline=color, width=max(2, crop.width // 180))
+ draw.text((left + 3, max(0, top - 14)), label, fill=color)
+ crop.thumbnail((720, 560))
+ label = f"{index}. A={conflict.get('object_anchor')} T={conflict.get('object_target')}"
+ tiles.append((crop, label))
+ items.append(
+ {
+ "conflict_id": conflict.get("conflict_id"),
+ "status": "rendered",
+ "crop_box_normalized": [round(x0, 4), round(y0, 4), round(x1, 4), round(y1, 4)],
+ "label": label,
+ }
+ )
+
+ if not tiles:
+ audit = {
+ "schema_version": "contain_conflict_review_manifest",
+ "status": "unavailable",
+ "reason": "no conflict carried usable normalized image boxes",
+ "conflict_count": len(conflicts),
+ "contact_sheet_path": None,
+ "items": items,
+ }
+ write_audit_pair(output_dir, "relations", "contain_conflict_review_manifest", audit, "Contain Conflict Review Manifest")
+ return audit
+
+ sheet_width = max(tile.width for tile, _ in tiles)
+ sheet_height = sum(tile.height + 28 for tile, _ in tiles)
+ sheet = Image.new("RGB", (sheet_width, sheet_height), "white")
+ draw = ImageDraw.Draw(sheet)
+ cursor = 0
+ for tile, label in tiles:
+ sheet.paste(tile, (0, cursor))
+ draw.rectangle((0, cursor + tile.height, sheet_width, cursor + tile.height + 28), fill="white")
+ draw.text((4, cursor + tile.height + 6), label, fill="black")
+ cursor += tile.height + 28
+ contact_sheet_path = output_dir / "debug" / "relations" / "contain_conflict_contact_sheet.png"
+ contact_sheet_path.parent.mkdir(parents=True, exist_ok=True)
+ sheet.save(contact_sheet_path)
+ audit = {
+ "schema_version": "contain_conflict_review_manifest",
+ "status": "available",
+ "conflict_count": len(conflicts),
+ "rendered_crop_count": len(tiles),
+ "original_image_path": str(original_path),
+ "contact_sheet_path": str(contact_sheet_path),
+ "items": items,
+ "policy": "The crop is attached only to the existing DeliverySchemaGuard request when a high-confidence support/contain conflict exists.",
+ }
+ write_audit_pair(output_dir, "relations", "contain_conflict_review_manifest", audit, "Contain Conflict Review Manifest")
+ return audit
+
+def dynamic_task_payload(
+ task: dict[str, Any],
+ prior_task_results: list[dict[str, Any]],
+ scene_state: dict[str, Any],
+ wall_evidence: dict[str, Any],
+ wall_visual_evidence: dict[str, Any],
+ image_paths: list[str],
+) -> dict[str, Any]:
+ """Serialize only the evidence needed by this Agent request.
+
+ Complete scene and wall evidence remains on disk for local assembly and
+ audit; API requests receive task-specific compact context.
+ """
+ task_type_l = str(task.get("task_type", "")).lower()
+ canonical_kind = str(task.get("canonical_task_kind") or task.get("task_type") or "").lower()
+ is_recall = is_wall_attach_recall_task(task)
+ prefilter = task.get("wall_attach_prefilter") if isinstance(task.get("wall_attach_prefilter"), dict) else {}
+
+ context = compact_task_api_context(task, prior_task_results, scene_state, wall_evidence)
+ task_view = context.pop("task")
+ return {
+ "task": task_view,
+ "scene_context": context,
+ "relation_policy": {
+ "allowed_relation_types": ["attach", "contain"],
+ "floor_support_is_primary_but_not_exclusive": True,
+ "wall_contact_can_be_additional_to_floor_support": True,
+ "same_anchor_target_pair_attach_and_contain_are_mutually_exclusive": True,
+ "weak_or_inferred_hidden_walls_cannot_anchor_attach": True,
+ },
+ "wall_review_mode": task.get("wall_review_mode") if canonical_kind == "wall_final_adjudication" else None,
+ "wall_visual_image_paths": image_paths,
+ "response_policy": {
+ "format": "compact_structured_json",
+ "reason_field": "Use a short snake_case reason_code or evidence_code instead of a prose explanation.",
+ "avoid": ["long visual narration", "duplicate geometry already supplied by local context", "repeating accepted relations in multiple fields"],
+ },
+ }
+
+
+def normalize_compact_task_response(response: dict[str, Any]) -> dict[str, Any]:
+ """Make compact reason/evidence codes compatible with existing audits."""
+ def visit(value: Any) -> None:
+ if isinstance(value, dict):
+ if value.get("reason") is None and value.get("reason_code"):
+ value["reason"] = str(value["reason_code"])
+ if value.get("evidence") is None and isinstance(value.get("evidence_codes"), list):
+ value["evidence"] = [str(item) for item in value["evidence_codes"]]
+ for child in value.values():
+ visit(child)
+ elif isinstance(value, list):
+ for child in value:
+ visit(child)
+ visit(response)
+ return response
+
+DEFAULT_BASE_MODEL = "gpt-5.5"
+DEFAULT_WALL_SPECIALIST_MODEL = "gpt-5.5"
+
+WALL_SPECIALIST_TASK_KINDS = {
+ "wall_alignment_object_review",
+ "provisional_wall_visibility_review",
+ "wall_visibility_semantics_review",
+ "wall_final_adjudication",
+ "wall_attach_candidate_recall",
+}
+
+
+# `balanced` is deliberately task-aware. It keeps high-detail vision only for
+# decisions that depend on fine wall/contact or containment evidence.
+BALANCED_TASK_IMAGE_DETAILS = {
+ "support_contact_extraction": "low",
+ "contain_relation_extraction": "high",
+ "object_revision_check": "low",
+ "object_physical_property_reasoning": "low",
+ "room_material_classification": "low",
+ "ceiling_judgement": "low",
+ "wall_alignment_object_review": "high",
+ "provisional_wall_visibility_review": "high",
+ "wall_object_hypothesis": "high",
+ "wall_visibility_semantics_review": "high",
+ "wall_final_adjudication": "high",
+ "wall_attach_candidate_recall": "high",
+}
+
+WALL_CONTEXT_TASK_KINDS = {
+ "wall_alignment_object_review",
+ "wall_object_hypothesis",
+ "provisional_wall_visibility_review",
+ "wall_visibility_semantics_review",
+ "wall_final_adjudication",
+ "wall_attach_candidate_recall",
+}
+
+def configured_wall_specialist_task_kinds(args: argparse.Namespace) -> set[str]:
+ configured = getattr(args, "wall_specialist_task_kinds", None)
+ if configured is None:
+ return set(WALL_SPECIALIST_TASK_KINDS)
+ values = [configured] if isinstance(configured, str) else list(configured)
+ return {
+ part.strip().lower()
+ for item in values for part in str(item).split(",") if part.strip()
+ }
+
+
+def resolve_dynamic_task_model_route(args: argparse.Namespace, task: dict[str, Any]) -> dict[str, str]:
+ canonical_kind = str(task.get("canonical_task_kind") or task.get("task_type") or "").lower()
+ task_id = task_identifier(task).lower()
+ specialist_task_kinds = configured_wall_specialist_task_kinds(args)
+ specialist_enabled = bool(
+ getattr(args, "wall_specialist_model", None)
+ or getattr(args, "wall_specialist_vision_model", None)
+ )
+ is_wall_specialist_task = (
+ canonical_kind in specialist_task_kinds
+ or (task_provides_capability(task, PROVISIONAL_WALL_VISIBILITY_CAPABILITY) and (
+ "provisional_wall_visibility_review" in specialist_task_kinds
+ or "wall_alignment_object_review" in specialist_task_kinds
+ ))
+ or (task_provides_capability(task, "wall_visibility_semantics_ready") and "wall_visibility_semantics_review" in specialist_task_kinds)
+ or (task_id == "task_wall_final_adjudication" and "wall_final_adjudication" in specialist_task_kinds)
+ or ("wall_attach_candidate_recall" in task_id and "wall_attach_candidate_recall" in specialist_task_kinds)
+ )
+ if specialist_enabled and is_wall_specialist_task:
+ specialist_model = getattr(args, "wall_specialist_model", None) or args.model
+ specialist_vision_model = (
+ getattr(args, "wall_specialist_vision_model", None)
+ or getattr(args, "wall_specialist_model", None)
+ or args.vision_model
+ )
+ return {
+ "model": specialist_model,
+ "vision_model": specialist_vision_model,
+ "model_route": "wall_specialist",
+ "canonical_task_kind": canonical_kind,
+ }
+ return {
+ "model": args.model,
+ "vision_model": args.vision_model,
+ "model_route": "base",
+ "canonical_task_kind": canonical_kind,
+ }
+
+
+
+def task_vision_detail_overrides(args: argparse.Namespace) -> dict[str, str]:
+ raw = str(getattr(args, "task_vision_detail_overrides", "") or "").strip()
+ overrides: dict[str, str] = {}
+ if not raw:
+ return overrides
+ for item in raw.split(","):
+ key, separator, value = item.partition("=")
+ key, value = key.strip().lower(), value.strip().lower()
+ if separator and key and value in {"low", "high", "auto"}:
+ overrides[key] = value
+ return overrides
+
+
+def resolve_task_image_detail(args: argparse.Namespace, task: dict[str, Any]) -> str | None:
+ # A global detail override takes precedence over task-aware defaults.
+ if getattr(args, "vision_image_detail", None) is not None:
+ return args.vision_image_detail
+ canonical_kind = str(task.get("canonical_task_kind") or task.get("task_type") or "").lower()
+ return task_vision_detail_overrides(args).get(canonical_kind, BALANCED_TASK_IMAGE_DETAILS.get(canonical_kind, "low"))
+
+def execute_dynamic_task_once(
+ args: argparse.Namespace,
+ task: dict[str, Any],
+ prior_task_results: list[dict[str, Any]],
+ scene_state: dict[str, Any],
+ wall_evidence: dict[str, Any],
+ wall_visual_evidence: dict[str, Any],
+) -> dict[str, Any]:
+ image_routing = resolve_task_image_evidence(task, wall_visual_evidence)
+ wall_image_paths = list(image_routing.get("image_paths", [])) if image_routing.get("matched") else wall_visual_image_paths(task, wall_visual_evidence)
+ contain_image_paths = contain_task_image_paths(task, scene_state)
+ image_paths = compact_image_paths(wall_image_paths + contain_image_paths, 8)
+ if contain_image_paths:
+ current_source = str(image_routing.get("routing_source") or "")
+ image_routing["routing_source"] = "+".join(item for item in [current_source, "contain_task_profile"] if item)
+ image_routing["selected_artifacts"] = list(image_routing.get("selected_artifacts", [])) + ["contain_task_images"]
+ task_type_l = str(task.get("task_type", "")).lower()
+ task_id = task_identifier(task)
+ execution_stage = "wall_final_adjudication" if task_type_l == "wall_final_adjudication" else "dynamic_task_api"
+ use_vision_model = bool(task.get("use_vision_model") or image_paths)
+ payload = dynamic_task_payload(
+ task, prior_task_results, scene_state, wall_evidence, wall_visual_evidence, image_paths,
+ )
+ model_route = resolve_dynamic_task_model_route(args, task)
+ local_client = SceneDeliveryAPIClient(
+ APIConfig(
+ model=model_route["model"],
+ vision_model=model_route["vision_model"],
+ api_base=args.api_base,
+ api_key_env=args.api_key_env,
+ dry_run=args.dry_run,
+ image_detail=resolve_task_image_detail(args, task),
+ )
+ )
+ response = local_client.call_json(
+ "DynamicTaskAgent",
+ load_prompt("dynamic_task_agent_compact_system.txt" if task.get("compiled_from_compact_manager_plan") else "dynamic_task_agent_system.txt"),
+ payload,
+ use_vision_model=use_vision_model,
+ image_paths=image_paths,
+ )
+ if isinstance(response, dict):
+ response = normalize_compact_task_response(response)
+ return {
+ "task": task,
+ "task_id": task_id,
+ "task_type": task_type_l,
+ "execution_stage": execution_stage,
+ "response": response,
+ "use_vision_model": use_vision_model,
+ "image_count": len(image_paths),
+ "image_routing_source": image_routing.get("routing_source"),
+ "selected_image_artifacts": image_routing.get("selected_artifacts", []),
+ "keyword_image_fallback_used": bool(image_routing.get("keyword_fallback_used")),
+ "image_detail": local_client.config.image_detail,
+ **local_client.usage_metadata(),
+ "retry_events": [{**event, "task_id": task_id, "canonical_task_kind": model_route["canonical_task_kind"]} for event in local_client.retry_events],
+ "model": model_route["vision_model"] if use_vision_model else model_route["model"],
+ "model_route": model_route["model_route"],
+ "canonical_task_kind": model_route["canonical_task_kind"],
+ }
+
+
+def prior_results_for_task(task: dict[str, Any], task_results_by_id: dict[str, dict[str, Any]]) -> list[dict[str, Any]]:
+ deps = task.get("depends_on", []) if isinstance(task.get("depends_on"), list) else []
+ return [task_results_by_id[dep] for dep in deps if dep in task_results_by_id]
+
+
+def write_final_wall_overlay_or_skip(output_dir: Path, scene_state: dict[str, Any], final_outputs_generated: bool) -> tuple[dict[str, Any] | None, str | None]:
+ overlay_path = output_dir / "debug" / "walls" / "final_walls_overlay_on_canonical_topdown.png"
+ check_json = output_dir / "debug" / "walls" / "final_walls_overlay_check.json"
+ check_md = output_dir / "debug" / "walls" / "final_walls_overlay_check.md"
+ objects_json = output_dir / "final_outputs" / "objects.json"
+ relations_json = output_dir / "final_outputs" / "relations.json"
+ stale_final_outputs_detected = bool(
+ not final_outputs_generated
+ and (objects_json.exists() or relations_json.exists() or overlay_path.exists())
+ )
+ topdown = scene_state.get("topdown_grid_mapping", {}) if isinstance(scene_state.get("topdown_grid_mapping"), dict) else {}
+ topdown_png = Path(str(topdown.get("canonical_topdown_image_path", ""))) if topdown.get("canonical_topdown_image_path") else None
+ grid_mapping = Path(str(topdown.get("canonical_mapping_path", ""))) if topdown.get("canonical_mapping_path") else None
+ if not final_outputs_generated or not objects_json.exists():
+ reason = "final outputs not generated in current run"
+ elif not topdown_png or not topdown_png.exists():
+ reason = "final wall overlay skipped because canonical top-down image was not available"
+ elif not grid_mapping or not grid_mapping.exists():
+ reason = "final wall overlay skipped because canonical grid mapping was not available"
+ else:
+ report = visualize_final_walls_on_topdown(topdown_png, grid_mapping, objects_json, overlay_path)
+ return report, None
+ data = {
+ "status": "skipped",
+ "reason": reason,
+ "output_overlay_png": str(overlay_path),
+ "coordinate_source": "layout_xz",
+ "topdown_usage": "visual_audit_only",
+ "candidate_modified": False,
+ "revision_used": False,
+ "stale_final_outputs_detected": stale_final_outputs_detected,
+ "stale_paths_checked": {
+ "objects_json_exists": objects_json.exists(),
+ "relations_json_exists": relations_json.exists(),
+ "overlay_png_exists": overlay_path.exists(),
+ },
+ "note": "This overlay is for human verification and is generated only after final walls exist in the current run.",
+ }
+ write_json(check_json, data)
+ write_md(
+ check_md,
+ "Final Walls Overlay Check",
+ [
+ "- status: `skipped`",
+ f"- reason: {reason}",
+ f"- output_overlay_png: `{overlay_path}`",
+ "- coordinate_source: `layout_xz`",
+ "- topdown_usage: `visual_audit_only`",
+ "- candidate_modified: `False`",
+ "- revision_used: `False`",
+ f"- stale_final_outputs_detected: `{stale_final_outputs_detected}`",
+ "- this overlay is for human verification",
+ ],
+ )
+ return None, reason
+
+
+REPAIRABLE_TASK_TYPES = {
+ "infer_four_clockwise_walls_from_topdown",
+ "repair_missing_final_walls",
+ "wall_final_adjudication",
+ "accepted_attach_merge_repair",
+ "repair_missing_attach_targets",
+ "repair_missing_relation_targets",
+ "room_material_classification",
+}
+
+
+def repair_task_signature(task: dict[str, Any]) -> str:
+ return json.dumps(
+ {
+ "task_type": task.get("task_type"),
+ "reason": task.get("reason"),
+ "inputs": task.get("inputs"),
+ },
+ sort_keys=True,
+ ensure_ascii=False,
+ )
+
+
+def strengthen_schema_repair_task(task: dict[str, Any], index: int, wall_evidence: dict[str, Any]) -> dict[str, Any]:
+ repair_task = dict(task)
+ task_type = repair_task.get("task_type", "schema_repair_task")
+ repair_task.setdefault("task_id", f"schema_repair_{index:02d}_{task_type}")
+ repair_task.setdefault("agent_role", "DynamicTaskAgent")
+ repair_task.setdefault("needs_tool", False)
+ if task_type in {"infer_four_clockwise_walls_from_topdown", "repair_missing_final_walls", "wall_final_adjudication"}:
+ repair_task["original_task_type"] = task_type
+ repair_task["task_type"] = "wall_final_adjudication"
+ repair_task["task_id"] = repair_task.get("task_id") or f"schema_repair_{index:02d}_wall_final_adjudication"
+ if "wall_final_adjudication" not in str(repair_task.get("task_id", "")):
+ repair_task["task_id"] = f"{repair_task['task_id']}_wall_final_adjudication"
+ repair_task["use_vision_model"] = True
+ repair_task["schema_repair_prompt"] = (
+ "You are executing the final wall adjudication step. This is not a candidate generation task. "
+ "Decide whether available layout-XZ wall candidates can be promoted to final wall_001 through wall_004. "
+ "Output exactly one of: A. {\"decision\":\"verified_final_walls\",\"walls\":[...]} with source obb_from_mesh_footprint_min_rect and topdown_verification passed for every wall; "
+ "or B. {\"decision\":\"missing_or_invalid_formal_obb_wall_evidence\",\"reason\":\"...\"}. "
+ "A layout-XZ candidate wall may be promoted or revised only when visual/layout evidence verifies the final room boundary; the visual evidence does not create coordinates. "
+ "If revising candidates, compare grid/camera-like candidates against wall_line_hypotheses from likely wall-attached or against-wall objects when provided, prefer an oblique/oriented quadrilateral when it better follows the true room footprint, then return candidate_modified=true and revision_reason. "
+ "Final walls may be oblique and are not required to be x-constant or z-constant. A weak/hidden wall may be completed by room closure when other walls and footprint evidence support closure; audit evidence_strength=weak, completion_method=inferred_by_room_closure, weak_wall=true, original_image_visibility=not_visible or inferred, and completion_reason. "
+ + CANONICAL_WALL_NAMING_INSTRUCTION
+ + " Wall object hypotheses are support evidence only; source wall_attached_object_distribution is not a final wall source. "
+ "Every final wall must be a segment with first_point [x,z] and second_point [x,z]. Do not output needs_check walls, single-coordinate summaries, candidate-only walls, grid_extent_hint, layout_bounds_hint, object_footprint_bounds_hint, or candidate_room_polygon_hint as final walls."
+ )
+ repair_task["requirements"] = {
+ "final_wall_ids": ["wall_001", "wall_002", "wall_003", "wall_004"],
+ "coordinate_format": "[x, z]",
+ "coordinate_source": "layout_xz",
+ "must_be_clockwise": True,
+ "adjacent_endpoints_must_match": True,
+ "final_eligible_source": "obb_from_mesh_footprint_min_rect",
+ "allowed_decisions": ["verified_final_walls", "missing_or_invalid_formal_obb_wall_evidence"],
+ "canonical_naming_policy": {
+ "wall_001_when_three_visible_walls": "middle visible wall",
+ "wall_001_when_two_visible_walls": "left visible wall",
+ "remaining_walls": "clockwise from wall_001 in top-down XZ",
+ "required_audit_fields": ["original_visible_wall_count", "visible_role", "original_image_visibility", "clockwise_order_index", "wall_001_anchor_reason"],
+ },
+ }
+ repair_task["wall_output_schema"] = {
+ "verified_final_walls": {
+ "decision": "verified_final_walls",
+ "walls": [
+ {"id": "wall_001", "first_point": ["x", "z"], "second_point": ["x", "z"], "source": "obb_from_mesh_footprint_min_rect", "evidence": "visual/layout-XZ verification", "topdown_verification": "passed", "original_visible_wall_count": 2, "visible_role": "left_visible|middle_visible|right_visible|hidden_closure", "original_image_visibility": "visible|partly_visible|not_visible|inferred", "canonical_naming": {"wall_001_anchor_reason": "middle visible wall or left visible wall according to original-image visibility", "clockwise_order_index": 1}}
+ ],
+ },
+ "insufficient": {"decision": "missing_or_invalid_formal_obb_wall_evidence", "reason": "why verification failed"},
+ }
+ repair_task["wall_evidence_payload_for_task"] = {
+ "candidate_walls": wall_evidence.get("candidate_walls", []),
+ "candidate_room_polygon_hint": wall_evidence.get("candidate_room_polygon_hint", {}),
+ "canonical_topdown": wall_evidence.get("canonical_topdown", {}),
+ "grid_mapping_summary": wall_evidence.get("grid_mapping_summary", {}),
+ "hint_only_sources": wall_evidence.get("hint_only_sources", {}),
+ "visual_wall_extraction": wall_evidence.get("visual_wall_extraction", {}),
+ "wall_output_requirements": wall_evidence.get("wall_output_requirements", {}),
+ "wall_canonical_naming_policy": wall_evidence.get("wall_canonical_naming_policy", {}),
+ }
+ elif task_type in {"repair_missing_attach_targets", "repair_missing_relation_targets"}:
+ repair_task["use_vision_model"] = True
+ repair_task["schema_repair_prompt"] = (
+ "You are repairing missing attach target coverage. Every final object must appear at least once as object_target. "
+ "Choose the most plausible physical anchor for each missing object. Prefer local physical contact over generic floor fallback. "
+ "Use image/mask, canonical top-down, fixed layout views, layout bbox/footprint evidence, prior attach candidates, and selective render evidence if available. "
+ "Return coverage_repair_relations or coverage_repair_candidates with clean attach relation/parameters plus repair_reason, evidence_used, and coverage_repair=true."
+ )
+ repair_task["coverage_repair_output_schema"] = {
+ "coverage_repair_relations": [
+ {
+ "relation": "attach",
+ "parameters": {"object_anchor": "floor/object/wall/ceiling", "object_target": "object_xxx"},
+ "repair_reason": "physical anchor reasoning",
+ "evidence_used": ["mask_overlay", "layout_bbox", "fixed_layout_views"],
+ "coverage_repair": True,
+ }
+ ]
+ }
+ elif task_type == "room_material_classification":
+ repair_task["use_vision_model"] = True
+ repair_task["canonical_task_kind"] = "room_material_classification"
+ repair_task["provides_capabilities"] = ["room_materials_ready"]
+ repair_task["requires_artifacts"] = ["original_image"]
+ repair_task["schema_repair_prompt"] = (
+ "Inspect the supplied original scene image and return exactly one room_materials object. "
+ "It must contain the keys walls, floor, and ceiling, each mapped to a concise non-empty English material/finish phrase suitable for rendering. "
+ "Do not copy examples unless they visually match; do not return relations, wall coordinates, objects, or unsupported hidden construction claims."
+ )
+ repair_task["expected_output_schema"] = {
+ "room_materials": {
+ "walls": "concise visible wall material/finish",
+ "floor": "concise visible floor material/finish",
+ "ceiling": "concise visible ceiling material/finish",
+ }
+ }
+ return repair_task
+
+
+def compact_missing_relation_repair_payload(
+ repair_task: dict[str, Any],
+ scene_state: dict[str, Any],
+ task_results: list[dict[str, Any]],
+ wall_evidence: dict[str, Any],
+ assembly: dict[str, Any],
+) -> dict[str, Any]:
+ """Project missing-target repair evidence without forwarding raw geometry."""
+ missing_ids = {
+ str(object_id)
+ for object_id in (repair_task.get("missing_object_targets") or assembly.get("missing_object_targets") or [])
+ if object_id
+ }
+ relation_catalog = compact_schema_guard_relation_catalog(task_results)
+ relevant_catalog = [
+ item
+ for item in relation_catalog
+ if str(item.get("object_target")) in missing_ids
+ or str(item.get("object_anchor")) in missing_ids
+ ]
+ current_relations = assembly.get("relations_payload") or []
+ related_object_ids = set(missing_ids)
+ for item in relevant_catalog:
+ for field in ("object_anchor", "object_target"):
+ value = str(item.get(field) or "")
+ if value.startswith("object_"):
+ related_object_ids.add(value)
+ for relation in current_relations:
+ parameters = relation.get("parameters", {}) if isinstance(relation, dict) else {}
+ for field in ("object_anchor", "object_target"):
+ value = str(parameters.get(field) or "")
+ if value in missing_ids or (field == "object_anchor" and str(parameters.get("object_target") or "") in missing_ids):
+ if value.startswith("object_"):
+ related_object_ids.add(value)
+
+ return {
+ "schema_version": "missing_relation_repair_context_compact",
+ "scene_name": scene_state.get("scene_name"),
+ "schema_repair": True,
+ "task": {
+ "task_id": repair_task.get("task_id"),
+ "task_type": repair_task.get("task_type"),
+ "missing_object_targets": sorted(missing_ids),
+ "instruction": repair_task.get("instruction"),
+ },
+ "schema_repair_prompt": repair_task.get("schema_repair_prompt"),
+ "missing_object_index": compact_object_index(scene_state, missing_ids),
+ "related_object_index": compact_object_index(scene_state, related_object_ids),
+ "canonical_walls": formal_wall_references(wall_evidence),
+ "wall_semantic_records": _compact_wall_semantic_records(task_results),
+ "relation_candidate_catalog": relevant_catalog,
+ "current_accepted_relations": current_relations,
+ "coverage_repair_output_schema": repair_task.get("coverage_repair_output_schema", {}),
+ "policy": [
+ "Return a relation only for each listed missing object target.",
+ "Prefer an existing supported candidate and preserve canonical wall ids.",
+ "Do not invent geometry or return objects/walls.",
+ "Use floor only when no stronger physical object, wall, ceiling, or contain anchor is supported.",
+ ],
+ }
+
+
+def build_schema_repair_payload(
+ repair_task: dict[str, Any],
+ scene_state: dict[str, Any],
+ task_results: list[dict[str, Any]],
+ wall_evidence: dict[str, Any],
+ assembly: dict[str, Any],
+ wall_visual_evidence: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+ return {
+ "task": repair_task,
+ "task_type": repair_task.get("task_type"),
+ "requirements": repair_task.get("requirements", {}),
+ "schema_repair": True,
+ "schema_repair_prompt": repair_task.get("schema_repair_prompt"),
+ "previous_finalization_errors": assembly.get("errors", []),
+ "previous_wall_errors": assembly.get("wall_errors", []),
+ "previous_repair_tasks": assembly.get("repair_tasks", []),
+ "scene_state": scene_state,
+ "wall_evidence": wall_evidence,
+ "candidate_walls": wall_evidence.get("candidate_walls", []),
+ "candidate_room_polygon_hint": wall_evidence.get("candidate_room_polygon_hint", {}),
+ "wall_final_adjudication_prompt": load_prompt("wall_final_adjudication_task.txt") if repair_task.get("task_type") == "wall_final_adjudication" else None,
+ "wall_object_hypotheses": extract_wall_object_hypotheses(task_results),
+ "candidate_wall_attach_evidence": extract_wall_attach_candidate_evidence(task_results),
+ "canonical_topdown": wall_evidence.get("canonical_topdown", {}),
+ "object_candidates": scene_state.get("object_candidates", {}),
+ "object_footprints": wall_evidence.get("object_footprints", []),
+ "current_accepted_relations": assembly.get("relations_payload") or [],
+ "missing_object_targets": repair_task.get("missing_object_targets") or assembly.get("missing_object_targets", []),
+ "candidate_anchor_objects": scene_state.get("object_candidates", {}).get("items", []),
+ "original_image_path": scene_state.get("evidence", {}).get("original_image_path"),
+ "mask_overlay_path": scene_state.get("evidence", {}).get("mask_overlay_path"),
+ "fixed_layout_views_manifest": scene_state.get("evidence", {}).get("fixed_layout_views", {}).get("manifest_path"),
+ "wall_visual_evidence": wall_visual_payload(wall_visual_evidence),
+ "wall_visual_image_paths": wall_visual_image_paths(repair_task, wall_visual_evidence),
+ "selective_render_tool_capability": "target object plus candidate anchor objects; top/front/back/left/right evidence may be requested by task",
+ "wall_output_schema": repair_task.get("wall_output_schema", {}),
+ "coverage_repair_output_schema": repair_task.get("coverage_repair_output_schema", {}),
+ "prior_task_results": task_results,
+ "policy_snippets": [
+ "This is a schema repair task, not general planning.",
+ "Return exactly four final-eligible layout-XZ walls or missing_or_invalid_formal_obb_wall_evidence when repairing walls.",
+ "For missing attach target coverage, return clean attach proposals with coverage_repair=true and repair_reason/evidence_used.",
+ "Do not cite hint-only sources as final wall source and do not default to floor when local physical contact evidence suggests an object anchor.",
+ "Wall object hypotheses may support wall_final_adjudication, but wall_attached_object_distribution is not final-eligible.",
+ "Final walls may be oblique; infer the oriented room footprint rather than accepting a loose grid/camera extent rectangle.",
+ "A hidden/weak wall may be completed by room closure when the room footprint is otherwise well supported; audit weak_wall completion fields.",
+ "Every final wall must have first_point and second_point layout-XZ endpoints; single-coordinate summaries are invalid.",
+ ],
+ }
+
+
+def write_schema_repair_task_results(output_dir: Path, data: dict[str, Any]) -> None:
+ write_json(output_dir / "debug" / "agents" / "schema_repair_task_results.json", data)
+ lines = [
+ f"- triggered: `{data.get('triggered')}`",
+ f"- executed_count: `{data.get('executed_count')}`",
+ f"- skipped_count: `{data.get('skipped_count')}`",
+ f"- reassembled: `{data.get('reassembled')}`",
+ f"- final_objects_after_repair: `{data.get('final_objects_after_repair')}`",
+ f"- final_relations_after_repair: `{data.get('final_relations_after_repair')}`",
+ "",
+ "## Results",
+ ]
+ for item in data.get("items", []):
+ lines.append(f"- `{item.get('task_id')}` ({item.get('task_type')}): {item.get('status')} final_eligible_walls=`{item.get('final_eligible_walls')}`")
+ write_md(output_dir / "debug" / "agents" / "schema_repair_task_results.md", "Schema Repair Task Results", lines)
+
+
+def write_api_retry_log(output_dir: Path, client: SceneDeliveryAPIClient) -> None:
+ events = list(getattr(client, "retry_events", []) or [])
+ if not events:
+ return
+ data = {
+ "status": "retry_events_recorded",
+ "event_count": len(events),
+ "events": events,
+ "api_key_redacted": True,
+ }
+ write_audit_pair(output_dir, "agents", "api_retry_log", data, "API Retry Log")
+
+
+def write_working_state_snapshot(
+ output_dir: Path,
+ stage: str,
+ scene_state: dict[str, Any],
+ manager_plan: dict[str, Any] | None,
+ task_results: list[dict[str, Any]],
+ supervisor_result: dict[str, Any] | None,
+ api_call_count: int,
+) -> None:
+ data = {
+ "stage": stage,
+ "api_call_count_so_far": api_call_count,
+ "scene_name": scene_state.get("scene_name"),
+ "object_candidate_count": len(scene_state.get("object_candidates", {}).get("items", [])),
+ "manager_plan_task_count": len(manager_plan.get("tasks", [])) if isinstance(manager_plan, dict) else None,
+ "task_result_count": len(task_results),
+ "task_result_ids": [result.get("task_id") for result in task_results if isinstance(result, dict)],
+ "supervisor_result": supervisor_result,
+ }
+ write_json(output_dir / "debug" / "agents" / "working_state_latest.json", data)
+ write_md(
+ output_dir / "debug" / "agents" / "working_state_latest.md",
+ "Working State Snapshot",
+ [
+ f"- stage: `{stage}`",
+ f"- api_call_count_so_far: `{api_call_count}`",
+ f"- object_candidate_count: `{data['object_candidate_count']}`",
+ f"- task_result_count: `{len(task_results)}`",
+ ],
+ )
+
+
+def write_api_failure_report(
+ output_dir: Path,
+ failed_stage: str,
+ exc: BaseException,
+ api_call_count_so_far: int,
+ local_schema_fallback_attempted: bool,
+ next_suggested_command: str,
+) -> dict[str, Any]:
+ data = {
+ "failed_stage": failed_stage,
+ "exception_type": type(exc).__name__,
+ "exception_message": str(exc),
+ "api_call_count_so_far": api_call_count_so_far,
+ "final_objects_existed": (output_dir / "final_outputs" / "objects.json").exists(),
+ "final_relations_existed": (output_dir / "final_outputs" / "relations.json").exists(),
+ "local_schema_fallback_attempted": local_schema_fallback_attempted,
+ "api_retry_log_path": str(output_dir / "debug" / "agents" / "api_retry_log.json"),
+ "next_suggested_command": next_suggested_command,
+ "api_key_redacted": True,
+ }
+ if isinstance(exc, APIClientError):
+ data["retry_events"] = exc.retry_events
+ data["raw_response_saved"] = bool(exc.raw_response)
+ write_audit_pair(output_dir, "agents", "api_failure_report", data, "API Failure Report")
+ return data
+
+
+def response_has_final_eligible_walls(response: dict[str, Any]) -> bool:
+ walls = response.get("walls") or response.get("accepted_walls") or response.get("wall_candidates")
+ if not isinstance(walls, list):
+ return False
+ required = {"wall_001", "wall_002", "wall_003", "wall_004"}
+ ids = {wall.get("id") for wall in walls if isinstance(wall, dict)}
+ if not required.issubset(ids):
+ return False
+ return all(
+ isinstance(wall, dict)
+ and wall.get("id") in required
+ and wall.get("source") in {"obb_from_mesh_footprint_min_rect", "obb_from_mesh_footprint_min_rect"}
+ and bool(wall.get("evidence"))
+ for wall in walls
+ if isinstance(wall, dict) and wall.get("id") in required
+ )
+
+
+def execute_schema_repair_loop(
+ client: SceneDeliveryAPIClient,
+ output_dir: Path,
+ scene_state: dict[str, Any],
+ task_results: list[dict[str, Any]],
+ guard_result: dict[str, Any],
+ wall_evidence: dict[str, Any],
+ assembly: dict[str, Any],
+ wall_visual_evidence: dict[str, Any] | None = None,
+ runtime_tracker: RuntimeTracker | None = None,
+) -> tuple[dict[str, Any], int]:
+ executed_signatures: set[str] = set()
+ repair_log: dict[str, Any] = {
+ "triggered": bool(assembly.get("repair_tasks")),
+ "items": [],
+ "executed_count": 0,
+ "skipped_count": 0,
+ "reassembled": False,
+ "initial_errors": assembly.get("errors", []),
+ }
+ current_assembly = assembly
+ while current_assembly.get("repair_tasks"):
+ runnable = []
+ for index, task in enumerate(current_assembly.get("repair_tasks", []), start=1):
+ if not isinstance(task, dict):
+ continue
+ task_type = task.get("task_type")
+ signature = repair_task_signature(task)
+ if task_type not in REPAIRABLE_TASK_TYPES:
+ repair_log["items"].append({"task_type": task_type, "status": "skipped_unrepairable_task_type"})
+ repair_log["skipped_count"] += 1
+ continue
+ if signature in executed_signatures:
+ repair_log["items"].append({"task_type": task_type, "status": "skipped_no_new_repair_signature"})
+ repair_log["skipped_count"] += 1
+ continue
+ runnable.append((index, task, signature))
+ if not runnable:
+ break
+ for index, task, signature in runnable:
+ executed_signatures.add(signature)
+ repair_task = strengthen_schema_repair_task(task, index, wall_evidence)
+ if repair_task.get("task_type") in {"repair_missing_attach_targets", "repair_missing_relation_targets"}:
+ payload = compact_missing_relation_repair_payload(
+ repair_task, scene_state, task_results, wall_evidence, current_assembly
+ )
+ image_paths = []
+ else:
+ payload = build_schema_repair_payload(
+ repair_task, scene_state, task_results, wall_evidence, current_assembly, wall_visual_evidence
+ )
+ image_paths = wall_visual_image_paths(repair_task, wall_visual_evidence)
+ if runtime_tracker is not None and not client.config.dry_run:
+ with runtime_tracker.api_call(
+ "schema_repair_loop",
+ role="DynamicTaskAgent",
+ task_id=str(repair_task.get("task_id") or f"schema_repair_{index:02d}"),
+ task_type=str(repair_task.get("task_type") or "schema_repair"),
+ use_vision_model=bool(repair_task.get("use_vision_model") or image_paths),
+ image_count=len(image_paths),
+ ):
+ response = client.call_json(
+ "DynamicTaskAgent",
+ load_prompt("dynamic_task_agent_system.txt"),
+ payload,
+ use_vision_model=bool(repair_task.get("use_vision_model") or image_paths),
+ image_paths=image_paths,
+ )
+ else:
+ with runtime_tracker.stage("schema_repair_loop") if runtime_tracker is not None else nullcontext():
+ response = client.call_json(
+ "DynamicTaskAgent",
+ load_prompt("dynamic_task_agent_system.txt"),
+ payload,
+ use_vision_model=bool(repair_task.get("use_vision_model") or image_paths),
+ image_paths=image_paths,
+ )
+ if runtime_tracker is not None and not client.config.dry_run and runtime_tracker.api_calls:
+ runtime_tracker.api_calls[-1].update(
+ client.usage_metadata(), )
+ if isinstance(response, dict):
+ response.setdefault("task_id", repair_task.get("task_id"))
+ response.setdefault("task_type", repair_task.get("task_type"))
+ response.setdefault("canonical_task_kind", repair_task.get("canonical_task_kind") or repair_task.get("task_type"))
+ response.setdefault("provided_capabilities", task_capabilities(repair_task, "provides_capabilities"))
+ write_task_output(output_dir, repair_task, response)
+ task_results.append(response)
+ repair_log["items"].append(
+ {
+ "task_id": repair_task.get("task_id"),
+ "task_type": repair_task.get("task_type"),
+ "status": response.get("status"),
+ "prompt_payload": payload,
+ "response": response,
+ "final_eligible_walls": response_has_final_eligible_walls(response),
+ }
+ )
+ repair_log["executed_count"] += 1
+ next_assembly = assemble_final_payloads(
+ scene_state=scene_state,
+ task_results=task_results,
+ guard_result=guard_result,
+ output_dir=output_dir,
+ dry_run=False,
+ wall_evidence=wall_evidence,
+ )
+ repair_log["reassembled"] = True
+ if next_assembly.get("objects_payload") is not None or next_assembly.get("errors") != current_assembly.get("errors"):
+ current_assembly = next_assembly
+ if current_assembly.get("objects_payload") is not None:
+ break
+ continue
+ current_assembly = next_assembly
+ break
+ repair_log["final_errors"] = current_assembly.get("errors", [])
+ repair_log["final_objects_after_repair"] = current_assembly.get("objects_payload") is not None
+ repair_log["final_relations_after_repair"] = current_assembly.get("relations_payload") is not None
+ write_schema_repair_task_results(output_dir, repair_log)
+ return current_assembly, repair_log.get("executed_count", 0)
+
+
+def run_delivery(args: argparse.Namespace) -> dict[str, Any]:
+ wall_candidate_config = configure_wall_candidate_environment(args)
+ output_dir = args.output_dir.resolve()
+ run_root = args.run_root.resolve()
+ render_script = args.render_script.resolve()
+ if getattr(args, "relation_layout_dir", None):
+ os.environ[RELATION_LAYOUT_DIR_ENV] = str(args.relation_layout_dir.resolve())
+ else:
+ os.environ.pop(RELATION_LAYOUT_DIR_ENV, None)
+ ensure_output_tree(output_dir)
+ runtime_tracker = RuntimeTracker()
+
+ with runtime_tracker.stage("policy_generation"):
+ policy_result = load_static_policy()
+ copy_policy_to_run(output_dir)
+
+ with runtime_tracker.stage("scene_state_build"):
+ scene_state = build_scene_state(args.scene_name, run_root, render_script)
+ write_scene_state(scene_state, output_dir)
+
+ initialize_empty_audits(output_dir, dry_run=args.dry_run)
+ render_capabilities = scan_render_script_capabilities(render_script)
+ write_render_audit(output_dir, render_capabilities, args.dry_run)
+
+ client = SceneDeliveryAPIClient(
+ APIConfig(
+ model=args.model,
+ vision_model=args.vision_model,
+ api_base=args.api_base,
+ api_key_env=args.api_key_env,
+ dry_run=args.dry_run,
+ image_detail=args.vision_image_detail,
+ )
+ )
+ api_call_count = 0
+ manager_client = SceneDeliveryAPIClient(
+ APIConfig(
+ model=getattr(args, "manager_model", None) or args.model,
+ vision_model=getattr(args, "manager_vision_model", None) or args.vision_model,
+ api_base=args.api_base,
+ api_key_env=args.api_key_env,
+ dry_run=args.dry_run,
+ image_detail=getattr(args, "manager_vision_image_detail", "low"),
+ )
+ )
+ policy_text = POLICY_PATH.read_text(encoding="utf-8")
+ manager_plan_skeleton: dict[str, Any] | None = None
+ physics_sidecar_process: subprocess.Popen[Any] | None = None
+ physics_sidecar: dict[str, Any] = {
+ "mode": "async",
+ "status": "not_started",
+ "relation_critical_path_dependency": False,
+ }
+ with runtime_tracker.stage("expected_manifest_readiness"):
+ expected_manifest, expected_manifest_readiness_audit = wait_for_expected_relation_manifest(
+ args,
+ run_root,
+ )
+ write_json(
+ output_dir / "debug" / "agents" / "expected_manifest_readiness.json",
+ expected_manifest_readiness_audit,
+ )
+ if expected_manifest_readiness_audit.get("status") != "ready":
+ raise RuntimeError(
+ "Manager planning refused an incomplete expected_manifest: "
+ f"{expected_manifest_readiness_audit.get('reason')}"
+ )
+ with runtime_tracker.stage("physics_sidecar_launch"):
+ physics_sidecar_process, physics_sidecar = launch_async_physics_sidecar(
+ args,
+ expected_manifest,
+ output_dir,
+ )
+ skeleton_payload = build_manager_skeleton_payload(args, expected_manifest, scene_state, render_capabilities)
+ skeleton_image_paths = manager_skeleton_image_paths(args, expected_manifest)
+ if skeleton_image_paths:
+ skeleton_payload["manager_planning_images"] = {
+ "original_image_path": skeleton_image_paths[0],
+ "purpose": "Use this single original image with the object list to decide whether contain_relation_extraction is needed before slower precompute artifacts finish.",
+ }
+ if args.dry_run:
+ with runtime_tracker.stage("manager_planning_skeleton"):
+ manager_plan_skeleton = manager_client.call_json(
+ "SceneDeliveryManager",
+ load_prompt(manager_prompt_name()),
+ skeleton_payload,
+ use_vision_model=bool(skeleton_image_paths),
+ image_paths=skeleton_image_paths,
+ )
+ else:
+ with runtime_tracker.api_call("manager_planning_skeleton", role="SceneDeliveryManager", task_id="manager_plan_skeleton", model=(getattr(args, "manager_model", None) or args.model)):
+ manager_plan_skeleton = manager_client.call_json(
+ "SceneDeliveryManager",
+ load_prompt(manager_prompt_name()),
+ skeleton_payload,
+ use_vision_model=bool(skeleton_image_paths),
+ image_paths=skeleton_image_paths,
+ )
+ api_call_count += 1
+ annotate_manager_api_call(
+ runtime_tracker,
+ manager_client,
+ skeleton_payload,
+ load_prompt(manager_prompt_name()),
+ args,
+ manager_plan_skeleton,
+ )
+ if isinstance(manager_plan_skeleton, dict):
+ manager_plan_skeleton, _manager_compilation_audit = compile_manager_plan_response(args, manager_plan_skeleton, output_dir)
+ manager_plan_skeleton["manager_planning_mode"] = "skeleton_async"
+ manager_plan_skeleton["relation_layout_expected_manifest_path"] = (expected_manifest or {}).get("expected_manifest_path") if isinstance(expected_manifest, dict) else None
+ write_manager_skeleton_plan(output_dir, manager_plan_skeleton)
+ client.retry_events.extend(manager_client.retry_events)
+ write_api_retry_log(output_dir, client)
+
+ render_grid_precompute_result: dict[str, Any] | None = None
+ with runtime_tracker.stage("render_grid_wall_precompute_load"):
+ render_grid_precompute_result = wait_for_relation_precompute(args)
+ if isinstance(render_grid_precompute_result, dict) and render_grid_precompute_result.get("status") == "available":
+ scene_state = apply_render_grid_precompute_to_scene_state(scene_state, render_grid_precompute_result)
+ wall_candidate_config["render_grid_wall_precompute_path"] = render_grid_precompute_result.get("method_candidates_path")
+ wall_candidate_config["render_grid_wall_precompute_dir"] = render_grid_precompute_result.get("precompute_dir")
+ wall_candidate_config["render_grid_wall_precompute_status"] = render_grid_precompute_result.get("status")
+ wall_candidate_config["render_grid_wall_precompute_source"] = "external_precompute"
+ write_scene_state(scene_state, output_dir)
+ else:
+ runtime_tracker.mark_skipped("render_grid_wall_precompute", "formal external precompute is required")
+ raise RuntimeError(
+ "Formal relation extraction requires a candidate-local relation layout manifest, "
+ "--render_grid_wall_precompute_dir, or --render_grid_wall_precompute_path. "
+ "Generate it first with "
+ "vlm_agent.reconstruction.relations.tools.build_relation_layout_from_sam3d."
+ )
+
+ runtime_tracker.start("canonical_topdown_generation")
+ if isinstance(render_grid_precompute_result, dict) and render_grid_precompute_result.get("status") in {"available", "missing_primary"} and render_grid_precompute_result.get("canonical_mapping_path"):
+ canonical_result = {
+ "status": "available",
+ "generated": False,
+ "canonical_mapping_path": render_grid_precompute_result.get("canonical_mapping_path"),
+ "canonical_topdown_image_path": render_grid_precompute_result.get("canonical_topdown_image_path"),
+ "source": "render_grid_wall_precompute",
+ "fallback_used": False,
+ "reused_from_render_grid_wall_precompute": True,
+ "precompute_manifest_path": render_grid_precompute_result.get("manifest_path"),
+ }
+ else:
+ canonical_result = ensure_canonical_topdown_mapping(
+ scene_state,
+ dry_run=args.dry_run,
+ )
+ runtime_tracker.end("canonical_topdown_generation", status="success" if canonical_result.get("status") == "available" or args.dry_run else "failed", result_status=canonical_result.get("status"))
+ runtime_tracker.record_render_call("canonical_topdown_generation", success=canonical_result.get("status") == "available" or args.dry_run, result_status=canonical_result.get("status"), cache_status="reused_from_render_grid_wall_precompute" if canonical_result.get("reused_from_render_grid_wall_precompute") else None)
+ if not args.dry_run and canonical_result.get("status") != "available":
+ validation = {
+ "dry_run": False,
+ "valid": False,
+ "finalization_status": "canonical_topdown_regeneration_failed",
+ "canonical_result": canonical_result,
+ }
+ write_audit_pair(output_dir, "final_validation", "schema_validation", validation, "Schema Validation")
+ runtime_tracker.mark_skipped("fixed_layout_views_generation", "canonical topdown regeneration failed")
+ runtime_tracker.mark_skipped("wall_evidence_build", "canonical topdown regeneration failed")
+ runtime_tracker.mark_skipped("wall_visual_evidence_build", "canonical topdown regeneration failed")
+ runtime_summary = runtime_tracker.write(output_dir)
+ final_process_report = write_final_process_report(
+ output_dir=output_dir,
+ scene_state=scene_state,
+ manager_plan={},
+ task_results=[],
+ validation=validation,
+ wall_evidence={},
+ wall_visual_evidence={},
+ api_called=False,
+ api_call_count=0,
+ runtime_summary=runtime_summary,
+ finalization_status="canonical_topdown_regeneration_failed",
+ )
+ return {
+ "scene_name": args.scene_name,
+ "output_dir": str(output_dir),
+ "dry_run": args.dry_run,
+ "api_called": False,
+ "api_call_count": 0,
+ "final_outputs_generated": False,
+ "canonical_topdown_regeneration": canonical_result,
+ "schema_validation": validation,
+ "runtime_summary_path": runtime_summary.get("json_path"),
+ "final_process_report_path": final_process_report.get("md_path"),
+ }
+ with runtime_tracker.stage("scene_state_build"):
+ scene_state = build_scene_state(args.scene_name, run_root, render_script)
+ scene_state = apply_render_grid_precompute_to_scene_state(scene_state, render_grid_precompute_result)
+ write_scene_state(scene_state, output_dir)
+ fixed_views_output_dir = output_dir / "generated_evidence" / "views"
+ fixed_views_result: dict[str, Any] | None = None
+ fixed_views_source_dir = wait_for_fixed_views_precompute_dir(args)
+ if fixed_views_source_dir and not args.disable_evidence_cache:
+ runtime_tracker.start("fixed_layout_views_load")
+ fixed_views_result = reuse_fixed_layout_views_cache(
+ scene_state,
+ fixed_views_source_dir,
+ fixed_views_output_dir,
+ )
+ runtime_tracker.end(
+ "fixed_layout_views_load",
+ status="success" if fixed_views_result.get("status") == "available" else "failed",
+ result_status=fixed_views_result.get("status"),
+ cache_status=fixed_views_result.get("cache_status"),
+ source_dir=str(fixed_views_source_dir),
+ )
+ if fixed_views_result.get("status") != "available":
+ fixed_views_result = None
+ elif fixed_views_source_dir and args.disable_evidence_cache:
+ runtime_tracker.mark_skipped("fixed_layout_views_load", "--disable_evidence_cache ignored explicit fixed views precompute")
+ else:
+ runtime_tracker.mark_skipped("fixed_layout_views_load", "no external fixed views precompute configured")
+
+ if fixed_views_result is not None:
+ runtime_tracker.mark_skipped("fixed_layout_views_generation", "external fixed views precompute loaded")
+ else:
+ runtime_tracker.mark_skipped(
+ "fixed_layout_views_generation",
+ "optional fixed views were not precomputed; the formal runner does not render evidence in-run",
+ )
+ with runtime_tracker.stage("scene_state_build"):
+ scene_state = build_scene_state(args.scene_name, run_root, render_script)
+ scene_state = apply_render_grid_precompute_to_scene_state(scene_state, render_grid_precompute_result)
+ if isinstance(fixed_views_result, dict) and fixed_views_result.get("status") == "available":
+ scene_state.setdefault("evidence", {})["fixed_layout_views"] = fixed_views_result
+ write_scene_state(scene_state, output_dir)
+ with runtime_tracker.stage("wall_evidence_build"):
+ wall_evidence = build_and_write_wall_evidence(scene_state, output_dir)
+ with runtime_tracker.stage("obb_wall_risk_audit"):
+ obb_wall_risk_audit = build_obb_wall_risk_audit(wall_evidence)
+ write_obb_wall_risk_audit(output_dir, obb_wall_risk_audit)
+ wall_evidence["formal_obb_risk_audit"] = obb_wall_risk_audit
+ resolved_wall_review_mode = resolve_wall_review_mode(args.wall_review_mode, obb_wall_risk_audit)
+ with runtime_tracker.stage("wall_visual_evidence_build"):
+ wall_visual_evidence = ensure_wall_visual_evidence(
+ scene_state,
+ output_dir / "debug" / "walls" / "visual_review",
+ )
+ scene_state.setdefault("evidence", {})["wall_visual_evidence"] = wall_visual_payload(wall_visual_evidence)
+ write_scene_state(scene_state, output_dir)
+
+ if isinstance(manager_plan_skeleton, dict):
+ with runtime_tracker.stage("manager_plan_hydration"):
+ manager_plan = hydrate_manager_plan(manager_plan_skeleton, run_root, scene_state)
+ elif args.dry_run:
+ with runtime_tracker.stage("manager_planning"):
+ manager_plan = manager_client.call_json(
+ "SceneDeliveryManager",
+ load_prompt(manager_prompt_name()),
+ build_manager_planning_payload(args, scene_state, render_capabilities, wall_evidence, wall_visual_evidence),
+ )
+ else:
+ with runtime_tracker.api_call("manager_planning", role="SceneDeliveryManager", task_id="manager_plan", model=(getattr(args, "manager_model", None) or args.model)):
+ manager_plan = manager_client.call_json(
+ "SceneDeliveryManager",
+ load_prompt(manager_prompt_name()),
+ build_manager_planning_payload(args, scene_state, render_capabilities, wall_evidence, wall_visual_evidence),
+ )
+ api_call_count += 1
+ manager_payload_for_metrics = build_manager_planning_payload(
+ args, scene_state, render_capabilities, wall_evidence, wall_visual_evidence
+ )
+ annotate_manager_api_call(
+ runtime_tracker,
+ manager_client,
+ manager_payload_for_metrics,
+ load_prompt(manager_prompt_name()),
+ args,
+ manager_plan,
+ )
+ if manager_plan_skeleton is None:
+ manager_plan, _manager_compilation_audit = compile_manager_plan_response(args, manager_plan, output_dir)
+ client.retry_events.extend(manager_client.retry_events)
+ write_api_retry_log(output_dir, client)
+ manager_plan = apply_wall_task_policy(
+ manager_plan,
+ wall_evidence,
+ wall_candidate_config["wall_candidate_mode"],
+ wall_review_mode=resolved_wall_review_mode,
+ obb_risk_audit=obb_wall_risk_audit,
+ )
+ manager_plan = ensure_required_relation_capabilities(manager_plan, run_root, scene_state)
+ update_manager_compilation_audit_with_wall_policy(output_dir, manager_plan)
+ manager_plan["wall_review_routing"] = {
+ "requested_mode": args.wall_review_mode,
+ "resolved_mode": resolved_wall_review_mode,
+ "risk_level": obb_wall_risk_audit.get("risk_level"),
+ "risk_score": obb_wall_risk_audit.get("risk_score"),
+ "recommendation": obb_wall_risk_audit.get("recommendation"),
+ "audit_path": str(output_dir / "debug" / "walls" / "obb_wall_risk_audit.json"),
+ }
+ manager_plan = apply_relation_policy_and_dependencies(manager_plan)
+ manager_plan = apply_runtime_preset(manager_plan)
+ manager_plan = hydrate_manager_plan(manager_plan, run_root, scene_state)
+ dependency_layers = task_dependency_layers(manager_plan.get("tasks", []))
+ manager_plan["task_model_routing"] = {
+ "base_model": args.model,
+ "base_vision_model": args.vision_model,
+ "manager_plan_format": "compact",
+ "manager_model": getattr(args, "manager_model", None) or args.model,
+ "manager_vision_model": getattr(args, "manager_vision_model", None) or args.vision_model,
+ "manager_vision_image_detail": getattr(args, "manager_vision_image_detail", "low"),
+ "wall_specialist_enabled": bool(args.wall_specialist_model or args.wall_specialist_vision_model),
+ "wall_specialist_model": args.wall_specialist_model,
+ "wall_specialist_vision_model": args.wall_specialist_vision_model or args.wall_specialist_model,
+ "wall_specialist_task_kinds": sorted(configured_wall_specialist_task_kinds(args)),
+ "policy": "Manager DAG is unchanged; only the listed dynamic task kinds use the specialist route.",
+ }
+ manager_plan["task_execution_policy"] = {
+ "mode": "dependency_layers",
+ "max_parallel_tasks": args.max_parallel_tasks,
+ "layer_count": len(dependency_layers),
+ "layers": [[task_identifier(task) for task in layer] for layer in dependency_layers],
+ }
+ manager_plan["wall_candidate_config"] = wall_candidate_config
+ write_manager_plan(output_dir, manager_plan)
+
+ task_results: list[dict[str, Any]] = []
+ task_results_by_id: dict[str, dict[str, Any]] = {}
+ runtime_tracker.start("dynamic_tasks_total")
+ task_order = {
+ task_identifier(task): index
+ for index, task in enumerate(manager_plan.get("tasks", []))
+ if isinstance(task, dict)
+ }
+ wall_orientation_selection_done = False
+ selected_orientation_candidate_id: str | None = None
+ provisional_wall_visibility_binding_result: dict[str, Any] | None = None
+ provisional_wall_visibility_binding_audit: dict[str, Any] | None = None
+ for layer_index, layer in enumerate(dependency_layers, start=1):
+ layer_stage = f"dynamic_task_layer_{layer_index:02d}"
+ recall_tasks = [
+ task for task in layer
+ if is_wall_attach_recall_task(task)
+ or task_requires_capability(task, "wall_attach_candidates_ready")
+ ]
+ if recall_tasks:
+ runtime_tracker.start("wall_attach_local_prefilter")
+ semantic_provider_ids = [
+ task_identifier(task)
+ for task in manager_plan.get("tasks", [])
+ if isinstance(task, dict)
+ and task_provides_capability(task, "wall_visibility_semantics_ready")
+ ]
+ semantic_provider_results = [
+ task_results_by_id[task_id]
+ for task_id in semantic_provider_ids
+ if task_id in task_results_by_id
+ ]
+ if provisional_wall_visibility_binding_result is not None:
+ semantic_provider_results.append(provisional_wall_visibility_binding_result)
+ prefilter = build_wall_attach_prefilter(
+ scene_state,
+ wall_evidence,
+ semantic_provider_results,
+ extract_wall_object_hypotheses(list(task_results_by_id.values())),
+ object_anchor_routing=manager_plan.get("object_anchor_routing", []),
+ )
+ prefilter_json_path = output_dir / "debug" / "relations" / "wall_attach_prefilter.json"
+ prefilter_overlay_path = output_dir / "debug" / "walls" / "wall_attach_prefilter_overlay.png"
+ overlay_result = render_wall_attach_prefilter_overlay(
+ wall_visual_evidence.get("canonical_topdown_path"),
+ wall_visual_evidence.get("render_grid_affine_path"),
+ prefilter_overlay_path,
+ wall_evidence,
+ prefilter,
+ )
+ prefilter["artifact_path"] = str(prefilter_json_path)
+ prefilter["overlay_status"] = overlay_result.get("status")
+ prefilter["overlay_path"] = overlay_result.get("output_path")
+ prefilter["overlay_result"] = overlay_result
+ if overlay_result.get("status") == "available" and overlay_result.get("output_path"):
+ wall_visual_evidence["wall_attach_prefilter_overlay_path"] = overlay_result["output_path"]
+ compact_prefilter = {
+ key: prefilter.get(key)
+ for key in [
+ "schema_version", "status", "capability_contract", "policy",
+ "formal_wall_references", "wall_semantics", "semantic_binding_audit",
+ "blocked_wall_ids", "raw_object_wall_pair_count", "candidate_pair_count",
+ "candidate_object_count", "candidate_pairs",
+ "supplemental_visual_review_object_ids", "artifact_path",
+ "semantic_visibility_geometry_conflicts",
+ "overlay_status", "overlay_path",
+ ]
+ }
+ scope_hydration_audits: list[dict[str, Any]] = []
+ for recall_task in recall_tasks:
+ recall_task["wall_attach_prefilter"] = compact_prefilter
+ recall_task.setdefault("evidence_binding", {})["wall_attach_prefilter"] = str(prefilter_json_path)
+ scope_hydration_audits.append({
+ "task_id": task_identifier(recall_task),
+ **hydrate_wall_attach_candidate_scope(recall_task, prefilter),
+ })
+ prefilter["scope_hydration_audits"] = scope_hydration_audits
+ write_audit_pair(output_dir, "relations", "wall_attach_prefilter", prefilter, "Wall Attach Local Prefilter")
+ write_manager_plan(output_dir, manager_plan)
+ runtime_tracker.end(
+ "wall_attach_local_prefilter",
+ status="success",
+ raw_pair_count=prefilter.get("raw_object_wall_pair_count", 0),
+ candidate_pair_count=prefilter.get("candidate_pair_count", 0),
+ scope_added_object_count=sum(
+ len(item.get("added_candidate_object_ids", []))
+ for item in scope_hydration_audits
+ ),
+ blocked_wall_ids=sorted(prefilter.get("blocked_wall_ids", {})),
+ overlay_status=overlay_result.get("status"),
+ )
+ runtime_tracker.start(layer_stage)
+ layer_results: list[dict[str, Any]] = []
+ future_to_task: dict[Any, dict[str, Any]] = {}
+ failed_task_id = "unknown_task"
+
+ def run_one_layer_task(layer_task: dict[str, Any]) -> dict[str, Any]:
+ return execute_dynamic_task_once(
+ args,
+ layer_task,
+ prior_results_for_task(layer_task, task_results_by_id),
+ scene_state,
+ wall_evidence,
+ wall_visual_evidence,
+ )
+
+ try:
+ if args.max_parallel_tasks > 1 and len(layer) > 1:
+ with ThreadPoolExecutor(max_workers=min(args.max_parallel_tasks, len(layer))) as executor:
+ future_to_task = {executor.submit(run_one_layer_task, task): task for task in layer}
+ for future in as_completed(future_to_task):
+ failed_task_id = task_identifier(future_to_task[future])
+ layer_results.append(future.result())
+ else:
+ for task in layer:
+ failed_task_id = task_identifier(task)
+ layer_results.append(run_one_layer_task(task))
+ except APIClientError as exc:
+ retry_events = list(getattr(exc, "retry_events", []) or [])
+ if retry_events:
+ client.retry_events.extend(retry_events)
+ write_api_retry_log(output_dir, client)
+ write_working_state_snapshot(
+ output_dir,
+ f"api_failed_{failed_task_id}",
+ scene_state,
+ manager_plan,
+ task_results,
+ None,
+ api_call_count,
+ )
+ write_api_failure_report(
+ output_dir=output_dir,
+ failed_stage=f"DynamicTaskAgent:{failed_task_id}",
+ exc=exc,
+ api_call_count_so_far=api_call_count,
+ local_schema_fallback_attempted=False,
+ next_suggested_command="Rerun through the SceneMosaic pipeline after checking network/API gateway stability; each attempt uses its own output directory.",
+ )
+ runtime_tracker.end(layer_stage, status="failed", failed_task_id=failed_task_id)
+ runtime_tracker.end("dynamic_tasks_total", status="failed", failed_task_id=failed_task_id)
+ runtime_tracker.write(output_dir)
+ raise
+
+ layer_results.sort(key=lambda item: task_order.get(item.get("task_id"), 10_000))
+ for item in layer_results:
+ task = item["task"]
+ response = item["response"]
+ if isinstance(response, dict):
+ response.setdefault("task_id", item["task_id"])
+ response.setdefault("task_type", item["task_type"])
+ response.setdefault("canonical_task_kind", item["canonical_task_kind"])
+ response.setdefault("provided_capabilities", task_capabilities(task, "provides_capabilities"))
+ annotate_semantic_visibility_geometry_conflict_resolutions(task, response)
+ if item.get("retry_events"):
+ client.retry_events.extend(item["retry_events"])
+ if args.dry_run:
+ runtime_tracker.record_stage_result(
+ item["execution_stage"],
+ status="success",
+ task_id=item["task_id"],
+ task_type=item["task_type"],
+ )
+ else:
+ runtime_tracker.record_api_call(
+ item["execution_stage"],
+ success=True,
+ role="DynamicTaskAgent",
+ task_id=item["task_id"],
+ task_type=item["task_type"],
+ model=item["model"],
+ model_route=item["model_route"],
+ canonical_task_kind=item["canonical_task_kind"],
+ use_vision_model=item["use_vision_model"],
+ image_count=item["image_count"],
+ image_routing_source=item.get("image_routing_source"),
+ selected_image_artifacts=item.get("selected_image_artifacts", []),
+ keyword_image_fallback_used=item.get("keyword_image_fallback_used", False),
+ image_detail=item.get("image_detail"),
+ response_model=item.get("response_model"),
+ finish_reason=item.get("finish_reason"),
+ )
+ write_api_retry_log(output_dir, client)
+ api_call_count += 0 if args.dry_run else 1
+ write_task_output(output_dir, task, response)
+ task_results_by_id[item["task_id"]] = response
+ task_results.append(response)
+ orientation_provider_capability = "wall_alignment_objects_ready"
+ if (
+ not wall_orientation_selection_done
+ and any(
+ task_provides_capability(item.get("task", {}), orientation_provider_capability)
+ for item in layer_results
+ )
+ ):
+ runtime_tracker.start("wall_orientation_selection_apply")
+ semantic_provider_results = [
+ task_results_by_id[task_identifier(task)]
+ for task in manager_plan.get("tasks", [])
+ if isinstance(task, dict)
+ and task_provides_capability(task, orientation_provider_capability)
+ and task_identifier(task) in task_results_by_id
+ ]
+ selection_audit = apply_wall_orientation_selection(
+ scene_state,
+ semantic_provider_results,
+ output_dir,
+ )
+ if selection_audit.get("status") == "applied":
+ wall_evidence = build_and_write_wall_evidence(scene_state, output_dir)
+ wall_evidence["formal_obb_risk_audit"] = obb_wall_risk_audit
+ wall_evidence["wall_orientation_selection_audit"] = selection_audit
+ wall_reference_overlay = render_wall_attach_prefilter_overlay(
+ wall_visual_evidence.get("canonical_topdown_path"),
+ wall_visual_evidence.get("render_grid_affine_path"),
+ output_dir / "debug" / "walls" / "formal_wall_reference_overlay.png",
+ wall_evidence,
+ {
+ "blocked_wall_ids": {},
+ "wall_semantics": {},
+ "candidate_pairs": [],
+ "candidate_pair_count": 0,
+ },
+ )
+ if wall_reference_overlay.get("status") == "available":
+ wall_visual_evidence["wall_attach_prefilter_overlay_path"] = wall_reference_overlay.get("output_path")
+ current_wall_references = formal_wall_references(wall_evidence)
+ for planned_task in manager_plan.get("tasks", []):
+ if not isinstance(planned_task, dict):
+ continue
+ if any(
+ capability.startswith("wall_")
+ for capability in task_capabilities(planned_task, "provides_capabilities")
+ ):
+ planned_task["formal_wall_references"] = current_wall_references
+ scene_state.setdefault("evidence", {})["wall_orientation_selection"] = selection_audit
+ write_scene_state(scene_state, output_dir)
+ write_manager_plan(output_dir, manager_plan)
+ wall_orientation_selection_done = True
+ selected_orientation_candidate_id = str(selection_audit.get("selected_candidate_id") or "") or None
+ runtime_tracker.end(
+ "wall_orientation_selection_apply",
+ status=selection_audit.get("status"),
+ selected_candidate_id=selection_audit.get("selected_candidate_id"),
+ selection_source=selection_audit.get("selection_source"),
+ )
+ if any(
+ item.get("task_type") in {"wall_object_hypothesis", "wall_alignment_object_review"}
+ or task_provides_capability(item.get("task", {}), "wall_alignment_objects_ready")
+ for item in layer_results
+ ):
+ build_and_write_wall_object_hypotheses(output_dir, task_results)
+
+ # This local binding is deliberately after orientation selection and
+ # before the next recall layer. It never changes coordinates or ids:
+ # it binds only the selected candidate panel's Agent semantics to
+ # matching immutable final edge geometry keys.
+ if wall_orientation_selection_done:
+ bound_result, binding_audit = materialize_selected_provisional_wall_visibility(
+ wall_evidence,
+ selected_orientation_candidate_id,
+ task_results,
+ )
+ provisional_wall_visibility_binding_audit = binding_audit
+ if bound_result is not None:
+ binding_task_id = str(bound_result["task_id"])
+ task_results = [
+ result for result in task_results
+ if not (isinstance(result, dict) and result.get("task_id") == binding_task_id)
+ ]
+ task_results.append(bound_result)
+ task_results_by_id[binding_task_id] = bound_result
+ provisional_wall_visibility_binding_result = bound_result
+ else:
+ task_results_by_id.pop("local_provisional_wall_visibility_binding", None)
+ provisional_wall_visibility_binding_result = None
+ write_audit_pair(
+ output_dir,
+ "walls",
+ "provisional_wall_visibility_binding",
+ binding_audit,
+ "Provisional Wall Visibility Binding",
+ )
+ object_wall_binding_result, object_wall_binding_audit = materialize_selected_candidate_object_wall_bindings(
+ wall_evidence,
+ selected_orientation_candidate_id,
+ task_results,
+ )
+ object_binding_task_id = "local_selected_candidate_object_wall_binding"
+ task_results = [
+ result for result in task_results
+ if not (isinstance(result, dict) and result.get("task_id") == object_binding_task_id)
+ ]
+ task_results_by_id.pop(object_binding_task_id, None)
+ if object_wall_binding_result is not None:
+ task_results.append(object_wall_binding_result)
+ task_results_by_id[object_binding_task_id] = object_wall_binding_result
+ write_audit_pair(
+ output_dir,
+ "walls",
+ "selected_candidate_object_wall_binding",
+ object_wall_binding_audit,
+ "Selected Candidate Object Wall Binding",
+ )
+ build_and_write_wall_object_hypotheses(output_dir, task_results)
+ runtime_tracker.end(
+ layer_stage,
+ status="success",
+ task_count=len(layer_results),
+ task_ids=[item.get("task_id") for item in layer_results],
+ )
+ if not any(isinstance(result, dict) and (result.get("task_type") == "wall_object_hypothesis" or "wall_object_hypothesis" in str(result.get("task_id", ""))) for result in task_results):
+ write_wall_object_hypotheses(output_dir, extract_wall_object_hypotheses(task_results))
+ runtime_tracker.end("dynamic_tasks_total", status="success", layer_count=len(dependency_layers), max_parallel_tasks=args.max_parallel_tasks)
+
+ if args.dry_run:
+ with runtime_tracker.stage("execution_supervisor"):
+ supervisor_result = supervise_execution(
+ client,
+ load_prompt("execution_supervisor_system.txt"),
+ scene_state,
+ task_results,
+ )
+ else:
+ with runtime_tracker.api_call("execution_supervisor", role="ExecutionSupervisor", task_id="execution_supervisor", model=args.model):
+ supervisor_result = supervise_execution(
+ client,
+ load_prompt("execution_supervisor_system.txt"),
+ scene_state,
+ task_results,
+ )
+ if not args.dry_run and runtime_tracker.api_calls:
+ runtime_tracker.api_calls[-1].update(
+ client.usage_metadata(), )
+ write_api_retry_log(output_dir, client)
+ api_call_count += 0 if args.dry_run else 1
+ update_agent_audits(output_dir, manager_plan, task_results, supervisor_result)
+
+ if args.dry_run:
+ write_dry_run_final_marker(output_dir)
+ runtime_tracker.mark_skipped("schema_guard", "dry_run")
+ runtime_tracker.mark_skipped("final_assembly", "dry_run")
+ runtime_tracker.mark_skipped("schema_repair_loop", "dry_run")
+ with runtime_tracker.stage("schema_validation"):
+ validation = validate_files(
+ output_dir / "final_outputs" / "objects.json",
+ output_dir / "final_outputs" / "relations.json",
+ output_dir,
+ ceiling_confirmed=False,
+ dry_run=True,
+ scene_state=scene_state,
+ scene_state_path=output_dir / "scene_states" / "scene_state.json",
+ )
+ else:
+ write_working_state_snapshot(
+ output_dir,
+ "before_delivery_schema_guard",
+ scene_state,
+ manager_plan,
+ task_results,
+ supervisor_result,
+ api_call_count,
+ )
+ guard_result: dict[str, Any]
+ guard_api_failed = False
+ api_failure_report: dict[str, Any] | None = None
+ contain_conflict_report = collect_contain_conflicts(scene_state, task_results)
+ contain_conflict_visual = build_contain_conflict_contact_sheet(
+ scene_state,
+ contain_conflict_report,
+ output_dir,
+ )
+ guard_image_paths = compact_image_paths(
+ [contain_conflict_visual.get("contact_sheet_path")],
+ 1,
+ )
+ guard_payload = compact_schema_guard_context(
+ scene_state,
+ task_results,
+ supervisor_result,
+ wall_evidence,
+ contain_conflict_report=contain_conflict_report,
+ )
+ guard_payload["contain_conflict_review_image_attached"] = bool(guard_image_paths)
+ try:
+ with runtime_tracker.api_call(
+ "schema_guard",
+ role="DeliverySchemaGuard",
+ task_id="schema_guard",
+ model=args.model,
+ use_vision_model=bool(guard_image_paths),
+ image_count=len(guard_image_paths),
+ image_routing_source="contain_conflict_only" if guard_image_paths else "none",
+ ):
+ guard_result = client.call_json(
+ "DeliverySchemaGuard",
+ load_prompt("delivery_schema_guard_system.txt"),
+ guard_payload,
+ use_vision_model=bool(guard_image_paths),
+ image_paths=guard_image_paths,
+ )
+ if runtime_tracker.api_calls:
+ runtime_tracker.api_calls[-1].update(
+ client.usage_metadata(), contain_conflict_count=len(contain_conflict_report.get("conflicts", [])),
+ contain_conflict_contact_sheet_path=contain_conflict_visual.get("contact_sheet_path"),
+ )
+ write_api_retry_log(output_dir, client)
+ api_call_count += 1
+ except APIClientError as exc:
+ guard_api_failed = True
+ write_api_retry_log(output_dir, client)
+ guard_result = {
+ "status": "api_failed_local_schema_fallback",
+ "ceiling_confirmed": False,
+ "api_error": str(exc),
+ "failed_stage": "DeliverySchemaGuard",
+ "local_schema_fallback": True,
+ }
+ api_failure_report = write_api_failure_report(
+ output_dir=output_dir,
+ failed_stage="DeliverySchemaGuard",
+ exc=exc,
+ api_call_count_so_far=api_call_count,
+ local_schema_fallback_attempted=True,
+ next_suggested_command=(
+ "python -m vlm_agent.reconstruction.relations.agents.run_scene_delivery_multi_agent "
+ f"--scene_name {args.scene_name} --run_root {run_root} --model {args.model} --vision_model {args.vision_model} "
+ f"--api_key_env {args.api_key_env} --api_base <OPEN_BASE_URL> --render_script {render_script} --output_dir {output_dir} "
+ f"--relation_layout_dir {getattr(args, 'relation_layout_dir', None) or relation_layout_expected_manifest_path(run_root).parent}"
+ ),
+ )
+ guard_output_dir = output_dir / "agent_outputs" / "delivery_schema_guard"
+ write_json(guard_output_dir / "response.json", guard_result)
+ write_md(
+ guard_output_dir / "response.md",
+ "Delivery Schema Guard Response",
+ ["```json", json.dumps(guard_result, ensure_ascii=False, indent=2), "```"],
+ )
+ write_working_state_snapshot(
+ output_dir,
+ "after_delivery_schema_guard_or_local_fallback",
+ scene_state,
+ manager_plan,
+ task_results,
+ supervisor_result,
+ api_call_count,
+ )
+ with runtime_tracker.stage("final_assembly"):
+ assembly = assemble_final_payloads(
+ scene_state=scene_state,
+ task_results=task_results,
+ guard_result=guard_result,
+ output_dir=output_dir,
+ dry_run=False,
+ wall_evidence=wall_evidence,
+ )
+ if assembly.get("repair_tasks") and not assembly.get("objects_payload"):
+ assembly, repair_api_calls = execute_schema_repair_loop(
+ client=client,
+ output_dir=output_dir,
+ scene_state=scene_state,
+ task_results=task_results,
+ guard_result=guard_result,
+ wall_evidence=wall_evidence,
+ assembly=assembly,
+ wall_visual_evidence=wall_visual_evidence,
+ runtime_tracker=runtime_tracker,
+ )
+ api_call_count += repair_api_calls
+ else:
+ runtime_tracker.mark_skipped("schema_repair_loop", "no repair tasks were executed")
+ if assembly["objects_payload"] is not None and assembly["relations_payload"] is not None:
+ write_json(output_dir / "final_outputs" / "objects.json", assembly["objects_payload"])
+ assembly["final_objects_written"] = True
+ write_json(output_dir / "final_outputs" / "relations.json", assembly["relations_payload"])
+ assembly["final_relations_written"] = True
+ with runtime_tracker.stage("schema_validation"):
+ validation = validate_files(
+ output_dir / "final_outputs" / "objects.json",
+ output_dir / "final_outputs" / "relations.json",
+ output_dir,
+ ceiling_confirmed=bool(guard_result.get("ceiling_confirmed") or assembly.get("ceiling_confirmed")),
+ dry_run=False,
+ scene_state=scene_state,
+ scene_state_path=output_dir / "scene_states" / "scene_state.json",
+ )
+ if guard_api_failed:
+ validation["local_schema_fallback"] = True
+ validation["api_failure_report_path"] = str(output_dir / "debug" / "agents" / "api_failure_report.json")
+ write_audit_pair(output_dir, "final_validation", "schema_validation", validation, "Schema Validation")
+ else:
+ with runtime_tracker.stage("schema_validation"):
+ validation = {
+ "dry_run": False,
+ "valid": False,
+ "finalization_status": "finalization_failed_with_partial_outputs",
+ "errors": assembly.get("errors", []),
+ "wall_errors": assembly.get("wall_errors", []),
+ "wall_final_adjudication": assembly.get("wall_final_adjudication"),
+ "partial_objects_path": assembly.get("partial_objects_path"),
+ "repair_tasks": assembly.get("repair_tasks", []),
+ "objects": None,
+ "relations": None,
+ "local_schema_fallback": bool(guard_api_failed),
+ "api_failure_report_path": str(output_dir / "debug" / "agents" / "api_failure_report.json") if guard_api_failed else None,
+ }
+ write_audit_pair(output_dir, "final_validation", "schema_validation", validation, "Schema Validation")
+
+ current_final_outputs_generated = bool(
+ (not args.dry_run)
+ and validation.get("valid")
+ and (output_dir / "final_outputs" / "objects.json").exists()
+ and (output_dir / "final_outputs" / "relations.json").exists()
+ )
+ stale_final_outputs_detected = bool(
+ not current_final_outputs_generated
+ and (
+ (output_dir / "final_outputs" / "objects.json").exists()
+ or (output_dir / "final_outputs" / "relations.json").exists()
+ or (output_dir / "debug" / "walls" / "final_walls_overlay_on_canonical_topdown.png").exists()
+ )
+ )
+ with runtime_tracker.stage("final_wall_overlay"):
+ final_wall_overlay, final_wall_overlay_skipped_reason = write_final_wall_overlay_or_skip(
+ output_dir,
+ scene_state,
+ final_outputs_generated=current_final_outputs_generated,
+ )
+ runtime_summary_for_report = runtime_tracker.to_json()
+ with runtime_tracker.stage("final_process_report"):
+ final_process_report = write_final_process_report(
+ output_dir=output_dir,
+ scene_state=scene_state,
+ manager_plan=manager_plan,
+ task_results=task_results,
+ validation=validation,
+ wall_evidence=wall_evidence,
+ wall_visual_evidence=wall_visual_evidence,
+ final_wall_overlay=final_wall_overlay,
+ final_wall_overlay_skipped_reason=final_wall_overlay_skipped_reason,
+ api_called=not args.dry_run,
+ api_call_count=api_call_count,
+ finalization_status=validation.get("finalization_status") or ("completed" if validation.get("valid") else "failed"),
+ runtime_summary=runtime_summary_for_report,
+ )
+ runtime_summary = runtime_tracker.write(output_dir)
+ final_process_report = write_final_process_report(
+ output_dir=output_dir,
+ scene_state=scene_state,
+ manager_plan=manager_plan,
+ task_results=task_results,
+ validation=validation,
+ wall_evidence=wall_evidence,
+ wall_visual_evidence=wall_visual_evidence,
+ final_wall_overlay=final_wall_overlay,
+ final_wall_overlay_skipped_reason=final_wall_overlay_skipped_reason,
+ api_called=not args.dry_run,
+ api_call_count=api_call_count,
+ finalization_status=validation.get("finalization_status") or ("completed" if validation.get("valid") else "failed"),
+ runtime_summary=runtime_summary,
+ )
+
+ physics_sidecar = snapshot_physics_sidecar(physics_sidecar_process, physics_sidecar)
+
+ return {
+ "scene_name": args.scene_name,
+ "output_dir": str(output_dir),
+ "dry_run": args.dry_run,
+ "policy": policy_result,
+ "scene_state_path": str(output_dir / "scene_states" / "scene_state.json"),
+ "object_candidate_count": len(scene_state["object_candidates"]["items"]),
+ "render_capabilities": render_capabilities,
+ "manager_plan_path": str(output_dir / "manager_plans" / "manager_plan_000.json"),
+ "final_outputs_generated": current_final_outputs_generated,
+ "stale_final_outputs_detected": stale_final_outputs_detected,
+ "schema_validation": validation,
+ "api_called": not args.dry_run,
+ "api_call_count": api_call_count,
+ "guard_api_failed": bool(locals().get("guard_api_failed", False)),
+ "api_failure_report": locals().get("api_failure_report"),
+ "wall_evidence_path": str(output_dir / "debug" / "walls" / "wall_evidence.json"),
+ "wall_visual_evidence_manifest_path": str(output_dir / "debug" / "walls" / "visual_review" / f"{args.scene_name}_wall_visual_review_manifest.json"),
+ "wall_candidate_config": wall_candidate_config,
+ "wall_evidence_candidate_walls": len(wall_evidence.get("candidate_walls", [])),
+ "final_wall_overlay": final_wall_overlay,
+ "final_wall_overlay_skipped_reason": final_wall_overlay_skipped_reason,
+ "final_process_report_path": final_process_report.get("md_path"),
+ "runtime_summary_path": runtime_summary.get("json_path"),
+ "canonical_topdown_regeneration": canonical_result,
+ "render_grid_wall_precompute": render_grid_precompute_result,
+ "fixed_layout_views": fixed_views_result,
+ "physics_sidecar": physics_sidecar,
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Run multi-agent scene delivery for final objects/relations outputs.")
+ parser.add_argument("--scene_name", required=True)
+ parser.add_argument("--run_root", required=True, type=Path)
+ parser.add_argument("--model", default=DEFAULT_BASE_MODEL, help=f"Base text model for ordinary tasks; default: {DEFAULT_BASE_MODEL}.")
+ parser.add_argument("--vision_model", default=DEFAULT_BASE_MODEL, help=f"Base vision model for ordinary tasks; default: {DEFAULT_BASE_MODEL}.")
+ parser.add_argument(
+ "--wall_specialist_model",
+ default=DEFAULT_WALL_SPECIALIST_MODEL,
+ help=f"Model for configured wall semantic/attach task kinds; default: {DEFAULT_WALL_SPECIALIST_MODEL}.",
+ )
+ parser.add_argument(
+ "--wall_specialist_vision_model",
+ default=None,
+ help="Vision model for wall specialist tasks; defaults to --wall_specialist_model.",
+ )
+ parser.add_argument(
+ "--wall_specialist_task_kinds",
+ nargs="+",
+ default=sorted(WALL_SPECIALIST_TASK_KINDS),
+ help=(
+ "Canonical task kinds routed to the wall specialist when it is enabled. "
+ "Accepts space- or comma-separated values. The default covers wall visibility, "
+ "final wall adjudication, and wall attach recall."
+ ),
+ )
+ parser.add_argument("--api_base", default=None)
+ parser.add_argument("--api_key_env", default="OPENAI_API_KEY")
+ parser.add_argument("--render_script", required=True, type=Path)
+ parser.add_argument("--output_dir", required=True, type=Path)
+ parser.add_argument("--dry_run", action="store_true")
+ parser.add_argument("--disable_evidence_cache", action="store_true", help="Disable reuse of evidence caches. Formal render_grid_only runs should normally pass explicit precompute dirs instead.")
+ parser.add_argument("--fixed_views_precompute_dir", type=Path, default=None, help="Directory containing a compatible precomputed fixed-view cache. Formal render_grid_only runs consume this evidence instead of rendering fixed views inside the multi-agent run.")
+ parser.add_argument("--relation_precompute_wait_seconds", type=float, default=0.0, help="When relation precompute is running concurrently, poll its manifest for up to this many seconds before failing.")
+ parser.add_argument("--fixed_views_precompute_wait_seconds", type=float, default=None, help="Poll for precomputed fixed views for this many seconds. Defaults to --relation_precompute_wait_seconds. Use 0 for dry-run smoke tests that intentionally skip fixed views.")
+ parser.add_argument("--relation_layout_dir", type=Path, default=None, help="Explicit candidate-local relation layout directory to consume.")
+ parser.add_argument("--manager_model", default=None, help="Optional model for Manager planning; defaults to --model.")
+ parser.add_argument("--manager_vision_model", default=None, help="Optional vision model for Manager planning; defaults to --vision_model.")
+ parser.add_argument("--manager_vision_image_detail", choices=["low", "high", "auto"], default="low", help="Image detail used by compact Manager planning.")
+ parser.add_argument("--vision_image_detail", choices=["low", "high", "auto"], default=None, help="Optional image_url.detail for vision inputs; low uses a smaller image representation when fine visual detail is not needed.")
+ parser.add_argument("--task_vision_detail_overrides", default="", help="Optional comma-separated task-kind overrides, e.g. support_contact_extraction=high,contain_relation_extraction=low. An explicit --vision_image_detail still overrides every task.")
+ parser.add_argument("--max_parallel_tasks", type=int, default=4, help="Maximum DynamicTaskAgent calls to run concurrently within the same dependency layer.")
+ parser.add_argument("--physics_model", default=None, help="Optional text model for the independent Physics Sidecar; defaults to --model.")
+ parser.add_argument("--physics_vision_model", default=None, help="Optional vision model for the independent Physics Sidecar; defaults to --vision_model.")
+ parser.add_argument("--physics_include_original", action="store_true", help="Also provide the original image to Physics Sidecar. Object-list-only mode is the faster default.")
+ parser.add_argument("--physics_vision_image_detail", choices=["low", "high", "auto"], default="low")
+ parser.add_argument(
+ "--wall_candidate_mode",
+ choices=["render_grid_only"],
+ default="render_grid_only",
+ help="Formal path: consume the precomputed OBB render-grid package as the only wall coordinate source.",
+ )
+ parser.add_argument(
+ "--wall_review_mode",
+ choices=["auto", "light", "heavy"],
+ default="auto",
+ help="auto routes formal OBB wall review through local risk audit: low risk uses light visibility/weak-wall review, medium/high risk keeps heavy final wall adjudication. Use heavy to always run the comprehensive multi-image adjudication.",
+ )
+ parser.add_argument("--render_grid_wall_precompute_dir", type=Path, default=None, help="Directory produced by the SceneMosaic relation render-grid precompute. Formal render_grid_only runs consume this precomputed evidence instead of rendering inside the multi-agent run.")
+ parser.add_argument("--render_grid_wall_precompute_path", type=Path, default=None, help="Path to a precomputed method_candidates.json produced by the SceneMosaic relation render-grid precompute.")
+ args = parser.parse_args()
+ if args.max_parallel_tasks < 1:
+ parser.error("--max_parallel_tasks must be >= 1")
+
+ try:
+ result = run_delivery(args)
+ except APIClientError as exc:
+ print(f"ERROR: {exc}", file=sys.stderr)
+ raise SystemExit(2)
+
+ print(json.dumps(result, indent=2, ensure_ascii=False))
+ if (
+ not args.dry_run
+ and not result.get("final_outputs_generated")
+ ):
+ raise SystemExit(1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/relations/agents/schema_guard.py b/vlm_agent/reconstruction/relations/agents/schema_guard.py
new file mode 100644
index 0000000..641d2f9
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/schema_guard.py
@@ -0,0 +1,709 @@
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import re
+import sys
+from pathlib import Path
+from typing import Any
+
+ROOT = Path(__file__).resolve().parents[2]
+if str(ROOT) not in sys.path:
+ sys.path.insert(0, str(ROOT))
+
+from relations.agents.state import sorted_object_id_key, write_audit_pair
+from relations.object_physics.core import FIELD_LIMITS, PHYSICS_FIELDS
+
+FORBIDDEN_PARAM_KEYS = {"obj_anchor", "obj_tar", "obj_target", "object_tar"}
+WALL_IDS = ["wall_001", "wall_002", "wall_003", "wall_004"]
+ROOM_MATERIAL_KEYS = ("walls", "floor", "ceiling")
+OBJECT_ID_RE = re.compile(r"^object_\d+$")
+WALL_EPSILON = 1e-5
+CLOCKWISE_SIGN_CONVENTION = "top_down_xz_standard_math: negative signed_area is interpreted as clockwise"
+
+
+def is_number_pair(value: Any) -> bool:
+ return isinstance(value, list) and len(value) == 2 and all(isinstance(x, (int, float)) and not isinstance(x, bool) for x in value)
+
+
+def is_snake_case(value: Any) -> bool:
+ return isinstance(value, str) and re.fullmatch(r"[a-z0-9]+(?:_[a-z0-9]+)*", value) is not None
+
+
+def _point_distance(a: list[float], b: list[float]) -> float:
+ return math.sqrt((float(a[0]) - float(b[0])) ** 2 + (float(a[1]) - float(b[1])) ** 2)
+
+
+def _polygon_signed_area(points: list[list[float]]) -> float:
+ area2 = 0.0
+ for idx, point in enumerate(points):
+ nxt = points[(idx + 1) % len(points)]
+ area2 += float(point[0]) * float(nxt[1]) - float(nxt[0]) * float(point[1])
+ return area2 / 2.0
+
+
+def wall_axis_alignment_summary(walls: list[Any], epsilon: float = WALL_EPSILON) -> dict[str, Any]:
+ wall_by_id = {wall.get("id"): wall for wall in walls if isinstance(wall, dict)}
+ edge_orientations: list[dict[str, Any]] = []
+ usable_edges = 0
+ all_axis_aligned = True
+ for wall_id in WALL_IDS:
+ wall = wall_by_id.get(wall_id)
+ if not isinstance(wall, dict):
+ all_axis_aligned = False
+ edge_orientations.append({"id": wall_id, "orientation": "missing", "axis_aligned": False})
+ continue
+ first = wall.get("first_point")
+ second = wall.get("second_point")
+ if not is_number_pair(first) or not is_number_pair(second):
+ all_axis_aligned = False
+ edge_orientations.append({"id": wall_id, "orientation": "invalid", "axis_aligned": False})
+ continue
+ usable_edges += 1
+ dx = abs(float(second[0]) - float(first[0]))
+ dz = abs(float(second[1]) - float(first[1]))
+ x_constant = dx <= epsilon
+ z_constant = dz <= epsilon
+ edge_axis_aligned = x_constant or z_constant
+ if x_constant and z_constant:
+ orientation = "degenerate"
+ elif x_constant:
+ orientation = "x_constant"
+ elif z_constant:
+ orientation = "z_constant"
+ else:
+ orientation = "oblique"
+ all_axis_aligned = all_axis_aligned and edge_axis_aligned
+ edge_orientations.append(
+ {
+ "id": wall_id,
+ "orientation": orientation,
+ "axis_aligned": edge_axis_aligned,
+ "delta_x": dx,
+ "delta_z": dz,
+ }
+ )
+ complete = usable_edges == len(WALL_IDS)
+ axis_aligned = bool(complete and all_axis_aligned)
+ return {
+ "complete": complete,
+ "axis_aligned": axis_aligned,
+ "oriented_boundary": bool(complete and not axis_aligned),
+ "edge_orientations": edge_orientations,
+ "definition": "axis_aligned means every wall edge is x-constant or z-constant within epsilon; oblique walls are allowed when the room footprint is oriented.",
+ }
+
+
+def extract_scene_state_candidates(scene_state: dict[str, Any] | None) -> dict[str, dict[str, Any]]:
+ if not isinstance(scene_state, dict):
+ return {}
+ items = scene_state.get("object_candidates", {}).get("items", [])
+ candidates = {}
+ for item in items:
+ if isinstance(item, dict) and item.get("object_id"):
+ candidates[str(item["object_id"])] = item
+ return candidates
+
+
+def load_revision_audit(path: Path | None) -> dict[str, Any] | None:
+ if path is None or not path.exists():
+ return None
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except Exception:
+ return None
+ return data if isinstance(data, dict) else None
+
+
+def load_weak_wall_completion(output_dir: Path) -> dict[str, Any]:
+ wall_audit_path = output_dir / "debug" / "walls" / "wall_audit.json"
+ default = {
+ "weak_wall_completion_used": False,
+ "weak_wall_ids": [],
+ "completion_methods": [],
+ "completion_reasons": [],
+ "wall_audit_path": str(wall_audit_path),
+ }
+ if not wall_audit_path.exists():
+ return default
+ try:
+ wall_audit = json.loads(wall_audit_path.read_text(encoding="utf-8"))
+ except Exception as exc:
+ default["read_error"] = str(exc)
+ return default
+ completion = wall_audit.get("weak_wall_completion") if isinstance(wall_audit, dict) else None
+ if not isinstance(completion, dict):
+ return default
+ return {
+ "weak_wall_completion_used": bool(completion.get("used")),
+ "weak_wall_ids": completion.get("weak_wall_ids", []),
+ "completion_methods": completion.get("completion_methods", []),
+ "completion_reasons": completion.get("completion_reasons", []),
+ "items": completion.get("items", []),
+ "wall_audit_path": str(wall_audit_path),
+ }
+
+
+def load_wall_final_adjudication(output_dir: Path) -> dict[str, Any]:
+ wall_audit_path = output_dir / "debug" / "walls" / "wall_audit.json"
+ default = {"executed": False, "decision": None, "accepted_final_walls": False, "wall_audit_path": str(wall_audit_path)}
+ if not wall_audit_path.exists():
+ return default
+ try:
+ wall_audit = json.loads(wall_audit_path.read_text(encoding="utf-8"))
+ except Exception as exc:
+ default["read_error"] = str(exc)
+ return default
+ adjudication = wall_audit.get("wall_final_adjudication") if isinstance(wall_audit, dict) else None
+ if not isinstance(adjudication, dict):
+ return default
+ result = dict(adjudication)
+ result.setdefault("executed", False)
+ result.setdefault("decision", None)
+ result.setdefault("accepted_final_walls", False)
+ promotion = wall_audit.get("candidate_wall_promotion") if isinstance(wall_audit, dict) else None
+ if isinstance(promotion, dict):
+ result["candidate_modified"] = bool(promotion.get("candidate_modified"))
+ result["revision_used"] = bool(promotion.get("revision_used") or promotion.get("candidate_modified"))
+ result["revision_reasons"] = promotion.get("revision_reasons", [])
+ result["candidate_wall_promotion"] = promotion
+ else:
+ result.setdefault("candidate_modified", False)
+ result.setdefault("revision_used", False)
+ result["wall_audit_path"] = str(wall_audit_path)
+ if wall_audit.get("warnings"):
+ result["wall_audit_warnings"] = wall_audit.get("warnings")
+ return result
+
+
+def _audit_items(data: dict[str, Any] | None) -> list[dict[str, Any]]:
+ if not isinstance(data, dict):
+ return []
+ items = data.get("items")
+ if isinstance(items, list):
+ return [item for item in items if isinstance(item, dict)]
+ revisions = data.get("revisions")
+ if isinstance(revisions, list):
+ return [item for item in revisions if isinstance(item, dict)]
+ return []
+
+
+def has_revision_reason(audit_data: dict[str, Any] | None, object_id: str, field_name: str) -> bool:
+ for item in _audit_items(audit_data):
+ item_object_id = item.get("object_id") or item.get("id")
+ if item_object_id != object_id:
+ continue
+ fields = item.get("fields") or item.get("changed_fields") or item.get("changes") or []
+ if isinstance(fields, dict):
+ field_matches = field_name in fields
+ elif isinstance(fields, list):
+ field_matches = field_name in fields or any(isinstance(x, dict) and x.get("field") == field_name for x in fields)
+ else:
+ field_matches = False
+ reason = item.get("reason") or item.get("rationale") or item.get("evidence") or item.get("audit_note")
+ if field_matches and isinstance(reason, str) and reason.strip():
+ return True
+ return False
+
+
+def validate_wall_geometry(walls: list[Any], epsilon: float = WALL_EPSILON) -> dict[str, Any]:
+ errors: list[str] = []
+ warnings: list[str] = []
+ wall_by_id = {wall.get("id"): wall for wall in walls if isinstance(wall, dict)}
+ details: dict[str, Any] = {
+ "epsilon": epsilon,
+ "coordinate_format": "[x, z]",
+ "coordinate_source_definition": "Wall geometry is validated in layout XZ coordinates; topdown/fixed views are verification and visual audit only.",
+ "clockwise_sign_convention": CLOCKWISE_SIGN_CONVENTION,
+ "signed_area": None,
+ "interpreted_order": "unknown",
+ "edge_connections": [],
+ "axis_alignment_epsilon": epsilon,
+ "axis_aligned": None,
+ "oriented_boundary": None,
+ "edge_orientations": [],
+ "orientation_rule": "Final walls may be axis-aligned or oblique; geometry validation does not require x-constant/z-constant edges.",
+ }
+
+ if any(wall_id not in wall_by_id for wall_id in WALL_IDS):
+ errors.append("Cannot validate wall geometry until wall_001 through wall_004 are all present.")
+ return {"valid": False, "errors": errors, "warnings": warnings, "details": details}
+
+ points: list[list[float]] = []
+ for idx, wall_id in enumerate(WALL_IDS):
+ wall = wall_by_id[wall_id]
+ first_point = wall.get("first_point")
+ second_point = wall.get("second_point")
+ if not is_number_pair(first_point) or not is_number_pair(second_point):
+ errors.append(f"Cannot validate geometry for `{wall_id}` because first_point/second_point is invalid.")
+ continue
+ points.append([float(first_point[0]), float(first_point[1])])
+ next_wall = wall_by_id[WALL_IDS[(idx + 1) % len(WALL_IDS)]]
+ next_first = next_wall.get("first_point")
+ if is_number_pair(next_first):
+ distance = _point_distance(second_point, next_first)
+ connection = {
+ "from": f"{wall_id}.second_point",
+ "to": f"{WALL_IDS[(idx + 1) % len(WALL_IDS)]}.first_point",
+ "distance": distance,
+ "within_epsilon": distance <= epsilon,
+ }
+ details["edge_connections"].append(connection)
+ if distance > epsilon:
+ errors.append(
+ f"{connection['from']} does not connect to {connection['to']} within epsilon {epsilon}; distance={distance}."
+ )
+
+ alignment = wall_axis_alignment_summary(walls, epsilon=epsilon)
+ details["axis_aligned"] = alignment["axis_aligned"]
+ details["oriented_boundary"] = alignment["oriented_boundary"]
+ details["edge_orientations"] = alignment["edge_orientations"]
+
+ if len(points) == 4:
+ signed_area = _polygon_signed_area(points)
+ # In raw OpenGL XZ math, negative signed area is clockwise under the old
+ # schema convention. In rendered PNG/screen coordinates the vertical axis
+ # is mirrored, so the human-visible order is the opposite sign. Keep both
+ # in the audit and let the canonical wall naming gate enforce screen order.
+ screen_signed_area_y_down = -signed_area
+ details["signed_area"] = signed_area
+ details["signed_area_xz"] = signed_area
+ details["signed_area_screen_y_down_fallback"] = screen_signed_area_y_down
+ if abs(signed_area) <= epsilon:
+ details["interpreted_order"] = "degenerate_or_too_small"
+ details["interpreted_order_xz"] = "degenerate_or_too_small"
+ details["interpreted_order_screen_y_down_fallback"] = "degenerate_or_too_small"
+ errors.append("Wall polygon signed area is too small; cannot confirm a valid closed room polygon.")
+ else:
+ details["interpreted_order_xz"] = "clockwise" if signed_area < 0 else "counter_clockwise"
+ details["interpreted_order_screen_y_down_fallback"] = "clockwise" if screen_signed_area_y_down > 0 else "counter_clockwise"
+ details["interpreted_order"] = details["interpreted_order_xz"]
+ if signed_area > 0:
+ warnings.append(
+ "Wall polygon is counter-clockwise in raw layout XZ, but this may be intentional after screen-clockwise canonical wall naming; "
+ f"signed_area_xz={signed_area}."
+ )
+ else:
+ errors.append("Wall polygon does not have four usable first_point vertices.")
+
+ return {"valid": not errors, "errors": errors, "warnings": warnings, "details": details}
+
+
+def validate_objects_payload(
+ payload: Any,
+ scene_state: dict[str, Any] | None = None,
+ object_revision_audit: dict[str, Any] | None = None,
+ wall_epsilon: float = WALL_EPSILON,
+) -> dict[str, Any]:
+ errors: list[str] = []
+ warnings: list[str] = []
+ object_ids: list[str] = []
+ if not isinstance(payload, dict):
+ return {"valid": False, "errors": ["objects.json top-level must be an object."], "warnings": [], "object_ids": []}
+
+ if "walls" not in payload:
+ errors.append("objects.json is missing `walls`.")
+ if "room_materials" not in payload:
+ errors.append("objects.json is missing `room_materials`.")
+ if "objects" not in payload:
+ errors.append("objects.json is missing `objects`.")
+ extra_top = sorted(set(payload.keys()) - {"walls", "room_materials", "objects"})
+ if extra_top:
+ warnings.append(f"objects.json has extra top-level keys: {extra_top}")
+
+ walls = payload.get("walls")
+ objects = payload.get("objects")
+ if not isinstance(walls, list):
+ errors.append("`walls` must be a list.")
+ walls = []
+ room_materials = payload.get("room_materials")
+ if "room_materials" in payload:
+ if not isinstance(room_materials, dict):
+ errors.append("`room_materials` must be an object.")
+ else:
+ material_keys = set(room_materials.keys())
+ missing_material_keys = [key for key in ROOM_MATERIAL_KEYS if key not in material_keys]
+ extra_material_keys = sorted(material_keys - set(ROOM_MATERIAL_KEYS))
+ if missing_material_keys:
+ errors.append(f"`room_materials` is missing required keys: {missing_material_keys}.")
+ if extra_material_keys:
+ errors.append(f"`room_materials` has unexpected keys: {extra_material_keys}.")
+ for key in ROOM_MATERIAL_KEYS:
+ value = room_materials.get(key)
+ if not isinstance(value, str) or not value.strip():
+ errors.append(f"`room_materials.{key}` must be a non-empty string.")
+ if not isinstance(objects, list):
+ errors.append("`objects` must be a list.")
+ objects = []
+
+ wall_id_set = {wall.get("id") for wall in walls if isinstance(wall, dict)}
+ for wall_id in WALL_IDS:
+ if wall_id not in wall_id_set:
+ errors.append(f"Missing required wall id `{wall_id}`.")
+ for wall in walls:
+ if not isinstance(wall, dict):
+ errors.append("Each wall must be an object.")
+ continue
+ keys = set(wall.keys())
+ if keys != {"id", "first_point", "second_point"}:
+ errors.append(f"Wall `{wall.get('id')}` must contain only id/first_point/second_point.")
+ if wall.get("id") not in WALL_IDS:
+ errors.append(f"Unexpected wall id `{wall.get('id')}`.")
+ if not is_number_pair(wall.get("first_point")):
+ errors.append(f"Wall `{wall.get('id')}` first_point must be a numeric [x, z] pair.")
+ if not is_number_pair(wall.get("second_point")):
+ errors.append(f"Wall `{wall.get('id')}` second_point must be a numeric [x, z] pair.")
+
+ wall_geometry = validate_wall_geometry(walls, epsilon=wall_epsilon)
+ errors.extend(wall_geometry["errors"])
+ warnings.extend(wall_geometry["warnings"])
+
+ previous_key = None
+ seen_object_ids = set()
+ final_by_id: dict[str, dict[str, Any]] = {}
+ for obj in objects:
+ if not isinstance(obj, dict):
+ errors.append("Each object must be an object.")
+ continue
+ keys = set(obj.keys())
+ base_object_keys = {"id", "category", "description", "upright", "physics"}
+ contain_anchor_keys = {"contain_anchor", "render_direction"}
+ if frozenset(keys) not in {frozenset(base_object_keys), frozenset(base_object_keys | contain_anchor_keys)}:
+ errors.append(
+ f"Object `{obj.get('id')}` must contain id/category/description/upright/physics, "
+ "with contain_anchor/render_direction present together only for contain anchors."
+ )
+ object_id = obj.get("id")
+ object_ids.append(object_id)
+ if not isinstance(object_id, str) or not OBJECT_ID_RE.fullmatch(object_id):
+ errors.append(f"Object id `{object_id}` must match object_<number>, for example object_001.")
+ if object_id in seen_object_ids:
+ errors.append(f"Duplicate object id `{object_id}` in final objects.json.")
+ seen_object_ids.add(object_id)
+ final_by_id[object_id] = obj
+ key = sorted_object_id_key(object_id)
+ if previous_key and key < previous_key:
+ errors.append("Objects must be sorted by the numeric suffix of object id ascending.")
+ previous_key = key
+ if not is_snake_case(obj.get("category")):
+ errors.append(f"Object `{object_id}` category must be snake_case.")
+ if not isinstance(obj.get("description"), str):
+ errors.append(f"Object `{object_id}` description must be a string.")
+ if not isinstance(obj.get("upright"), bool):
+ errors.append(f"Object `{object_id}` upright must be a boolean.")
+ has_contain_fields = "contain_anchor" in obj or "render_direction" in obj
+ if has_contain_fields:
+ contain_anchor = obj.get("contain_anchor")
+ render_direction = obj.get("render_direction")
+ if contain_anchor is not True:
+ errors.append(f"Object `{object_id}` may include contain_anchor only with value true.")
+ if render_direction not in {"front", "up"}:
+ errors.append(f"Contain anchor `{object_id}` render_direction must be `front` or `up`.")
+ physics = obj.get("physics")
+ if not isinstance(physics, dict):
+ errors.append(f"Object `{object_id}` physics must be an object.")
+ else:
+ if set(physics.keys()) != set(PHYSICS_FIELDS):
+ errors.append(
+ f"Object `{object_id}` physics must contain only: {list(PHYSICS_FIELDS)}."
+ )
+ for field in PHYSICS_FIELDS:
+ value = physics.get(field)
+ lower, upper = FIELD_LIMITS[field]
+ if (
+ isinstance(value, bool)
+ or not isinstance(value, (int, float))
+ or not math.isfinite(float(value))
+ or not lower <= float(value) <= upper
+ ):
+ errors.append(
+ f"Object `{object_id}` physics.{field} must be finite within [{lower}, {upper}]."
+ )
+ static_friction = physics.get("static_friction")
+ dynamic_friction = physics.get("dynamic_friction")
+ if (
+ isinstance(static_friction, (int, float))
+ and not isinstance(static_friction, bool)
+ and isinstance(dynamic_friction, (int, float))
+ and not isinstance(dynamic_friction, bool)
+ and float(dynamic_friction) > float(static_friction)
+ ):
+ errors.append(
+ f"Object `{object_id}` physics.dynamic_friction cannot exceed static_friction."
+ )
+
+ scene_candidates = extract_scene_state_candidates(scene_state)
+ scene_candidate_ids = sorted(scene_candidates.keys(), key=sorted_object_id_key)
+ if scene_candidates:
+ final_id_set = set(object_ids)
+ scene_id_set = set(scene_candidates.keys())
+ unexpected = sorted(final_id_set - scene_id_set, key=sorted_object_id_key)
+ missing = sorted(scene_id_set - final_id_set, key=sorted_object_id_key)
+ if unexpected:
+ errors.append(f"Final objects contain ids not present in scene_state candidates: {unexpected}.")
+ if missing:
+ errors.append(f"Final objects are missing scene_state candidate ids: {missing}.")
+ if not unexpected and not missing:
+ expected_order = scene_candidate_ids
+ if object_ids != expected_order:
+ errors.append(
+ "Final objects id set matches scene_state but order is not numeric-suffix ascending as inherited from scene_state."
+ )
+ for object_id in sorted(final_id_set & scene_id_set, key=sorted_object_id_key):
+ final_obj = final_by_id.get(object_id, {})
+ candidate = scene_candidates[object_id]
+ for field_name in ["category", "description"]:
+ if final_obj.get(field_name) != candidate.get(field_name):
+ if not has_revision_reason(object_revision_audit, object_id, field_name):
+ warnings.append(
+ f"Object `{object_id}` changed `{field_name}` from scene_state candidate but no reason was found in object_revision_audit."
+ )
+
+ return {
+ "valid": not errors,
+ "errors": errors,
+ "warnings": warnings,
+ "object_ids": object_ids,
+ "scene_state_candidate_ids": scene_candidate_ids,
+ "wall_geometry": wall_geometry,
+ }
+
+
+def validate_relations_payload(payload: Any, object_ids: list[str], ceiling_confirmed: bool = False) -> dict[str, Any]:
+ errors: list[str] = []
+ warnings: list[str] = []
+ if not isinstance(payload, list):
+ return {"valid": False, "errors": ["relations.json top-level must be a list."], "warnings": []}
+
+ allowed_attach_anchors = {"floor", *WALL_IDS, *object_ids}
+ if ceiling_confirmed:
+ allowed_attach_anchors.add("ceiling")
+
+ seen = set()
+ seen_targets = set()
+ last_target = None
+ closed_targets = set()
+ pair_relation_types: dict[tuple[Any, Any], set[str]] = {}
+ for idx, rel in enumerate(payload):
+ if not isinstance(rel, dict):
+ errors.append(f"Relation at index {idx} must be an object.")
+ continue
+ relation_type = rel.get("relation")
+ if relation_type not in {"attach", "contain"}:
+ errors.append(f"Relation at index {idx} must have relation=`attach` or relation=`contain`.")
+ params = rel.get("parameters")
+ if not isinstance(params, dict):
+ errors.append(f"Relation at index {idx} parameters must be an object.")
+ continue
+ forbidden = sorted(FORBIDDEN_PARAM_KEYS & set(params.keys()))
+ if forbidden:
+ errors.append(f"Relation at index {idx} uses forbidden parameter keys: {forbidden}.")
+ if set(params.keys()) != {"object_anchor", "object_target"}:
+ errors.append(f"Relation at index {idx} parameters must contain exactly object_anchor/object_target.")
+ anchor = params.get("object_anchor")
+ target = params.get("object_target")
+ if target not in object_ids:
+ errors.append(f"Relation at index {idx} object_target `{target}` is not in final objects.")
+ else:
+ seen_targets.add(target)
+ if relation_type == "attach":
+ if anchor == "ceiling" and not ceiling_confirmed:
+ errors.append("Ceiling anchor is forbidden because ceiling was not confirmed present.")
+ if anchor not in allowed_attach_anchors:
+ errors.append(f"Relation at index {idx} object_anchor `{anchor}` is not allowed for attach.")
+ elif relation_type == "contain":
+ if anchor not in object_ids:
+ errors.append(f"Relation at index {idx} contain object_anchor `{anchor}` must be a final object id.")
+ if anchor == target:
+ errors.append(f"Relation at index {idx} contain anchor and target cannot be the same object `{anchor}`.")
+ triple = (relation_type, anchor, target)
+ if triple in seen:
+ errors.append(f"Duplicate relation triple found at index {idx}: {triple}.")
+ seen.add(triple)
+ pair_relation_types.setdefault((anchor, target), set()).add(str(relation_type))
+ if target != last_target:
+ if target in closed_targets:
+ warnings.append(f"Relations for target `{target}` are not consecutive.")
+ if last_target is not None:
+ closed_targets.add(last_target)
+ last_target = target
+
+ for pair, relation_types in sorted(pair_relation_types.items(), key=lambda item: (str(item[0][1]), str(item[0][0]))):
+ if "attach" in relation_types and "contain" in relation_types:
+ errors.append(f"Pair {pair} has both attach and contain; resolve the pair to exactly one relation without a fixed relation-type priority.")
+
+ missing_targets = sorted(set(object_ids) - seen_targets, key=sorted_object_id_key)
+ for object_id in missing_targets:
+ errors.append(f"Object {object_id} has no attach or contain relation as object_target.")
+
+ return {
+ "valid": not errors,
+ "errors": errors,
+ "warnings": warnings,
+ "target_coverage": {
+ "required_object_ids": object_ids,
+ "covered_object_targets": sorted(seen_targets, key=sorted_object_id_key),
+ "missing_object_targets": missing_targets,
+ "complete": not missing_targets,
+ },
+ }
+
+
+def validate_contain_anchor_annotations(objects_payload: Any, relations_payload: Any) -> dict[str, Any]:
+ errors: list[str] = []
+ objects = objects_payload.get("objects", []) if isinstance(objects_payload, dict) else []
+ relations = relations_payload if isinstance(relations_payload, list) else []
+ object_by_id = {
+ str(obj.get("id")): obj
+ for obj in objects
+ if isinstance(obj, dict) and obj.get("id")
+ }
+ relation_anchor_ids = {
+ str(rel.get("parameters", {}).get("object_anchor"))
+ for rel in relations
+ if isinstance(rel, dict)
+ and rel.get("relation") == "contain"
+ and isinstance(rel.get("parameters"), dict)
+ and str(rel.get("parameters", {}).get("object_anchor", "")).startswith("object_")
+ }
+ annotated_ids = {
+ object_id
+ for object_id, obj in object_by_id.items()
+ if "contain_anchor" in obj or "render_direction" in obj
+ }
+ for object_id in sorted(relation_anchor_ids):
+ obj = object_by_id.get(object_id)
+ if obj is None:
+ continue
+ if obj.get("contain_anchor") is not True or obj.get("render_direction") not in {"front", "up"}:
+ errors.append(
+ f"Contain relation anchor `{object_id}` must include contain_anchor=true and render_direction=front|up."
+ )
+ for object_id in sorted(annotated_ids - relation_anchor_ids):
+ errors.append(
+ f"Non-contain anchor `{object_id}` must omit contain_anchor and render_direction."
+ )
+ return {
+ "valid": not errors,
+ "errors": errors,
+ "contain_relation_anchor_ids": sorted(relation_anchor_ids),
+ "annotated_object_ids": sorted(annotated_ids),
+ }
+
+
+def validate_files(
+ objects_path: Path,
+ relations_path: Path,
+ output_dir: Path,
+ ceiling_confirmed: bool = False,
+ dry_run: bool = False,
+ scene_state: dict[str, Any] | None = None,
+ scene_state_path: Path | None = None,
+ object_revision_audit_path: Path | None = None,
+ wall_epsilon: float = WALL_EPSILON,
+) -> dict[str, Any]:
+ result: dict[str, Any] = {
+ "dry_run": dry_run,
+ "objects_path": str(objects_path),
+ "relations_path": str(relations_path),
+ "ceiling_confirmed": ceiling_confirmed,
+ "scene_state_path": str(scene_state_path) if scene_state_path else None,
+ "objects": None,
+ "relations": None,
+ "valid": False,
+ }
+ if scene_state is None and scene_state_path and scene_state_path.exists():
+ try:
+ scene_state = json.loads(scene_state_path.read_text(encoding="utf-8"))
+ except Exception as exc:
+ result.setdefault("warnings", []).append(f"Failed to read scene_state for strict object id validation: {exc}")
+ if object_revision_audit_path is None:
+ object_revision_audit_path = output_dir / "debug" / "objects" / "object_revision_audit.json"
+ object_revision_audit = load_revision_audit(object_revision_audit_path)
+ result["object_revision_audit_path"] = str(object_revision_audit_path)
+ result["wall_final_adjudication"] = load_wall_final_adjudication(output_dir)
+ result["weak_wall_completion"] = load_weak_wall_completion(output_dir)
+
+ if dry_run:
+ result.update({"valid": False, "skipped": True, "reason": "Dry run does not validate or generate real final objects.json/relations.json."})
+ write_audit_pair(output_dir, "final_validation", "schema_validation", result, "Schema Validation")
+ return result
+
+ try:
+ objects_payload = json.loads(objects_path.read_text(encoding="utf-8"))
+ except Exception as exc:
+ result["objects"] = {"valid": False, "errors": [f"Failed to read objects.json: {exc}"], "warnings": [], "object_ids": []}
+ objects_payload = None
+ if objects_payload is not None:
+ result["objects"] = validate_objects_payload(
+ objects_payload,
+ scene_state=scene_state,
+ object_revision_audit=object_revision_audit,
+ wall_epsilon=wall_epsilon,
+ )
+ wall_geometry = result.get("objects", {}).get("wall_geometry") if isinstance(result.get("objects"), dict) else None
+ details = wall_geometry.get("details") if isinstance(wall_geometry, dict) else None
+ if isinstance(details, dict):
+ weak_completion = result.get("weak_wall_completion", {})
+ details["weak_wall_completion_used"] = bool(weak_completion.get("weak_wall_completion_used"))
+ details["weak_wall_ids"] = weak_completion.get("weak_wall_ids", [])
+ details["completion_methods"] = weak_completion.get("completion_methods", [])
+ details["completion_reasons"] = weak_completion.get("completion_reasons", [])
+
+ try:
+ relations_payload = json.loads(relations_path.read_text(encoding="utf-8"))
+ except Exception as exc:
+ result["relations"] = {"valid": False, "errors": [f"Failed to read relations.json: {exc}"], "warnings": []}
+ relations_payload = None
+ if relations_payload is not None:
+ object_ids = result["objects"].get("object_ids", []) if result.get("objects") else []
+ result["relations"] = validate_relations_payload(relations_payload, object_ids, ceiling_confirmed=ceiling_confirmed)
+ coverage_audit_path = output_dir / "debug" / "attach" / "coverage_repair_audit.json"
+ if coverage_audit_path.exists():
+ try:
+ coverage_audit = json.loads(coverage_audit_path.read_text(encoding="utf-8"))
+ except Exception:
+ coverage_audit = {}
+ result["relations"]["coverage_repair_used"] = bool(coverage_audit.get("coverage_repair_used"))
+ result["relations"]["coverage_repair_count"] = int(coverage_audit.get("coverage_repair_count") or 0)
+ result["relations"]["coverage_repair_audit_path"] = str(coverage_audit_path)
+
+ contain_anchor_validation = validate_contain_anchor_annotations(objects_payload, relations_payload)
+ result["contain_anchor_annotations"] = contain_anchor_validation
+ if not contain_anchor_validation["valid"]:
+ if isinstance(result.get("objects"), dict):
+ result["objects"].setdefault("errors", []).extend(contain_anchor_validation["errors"])
+ result["objects"]["valid"] = False
+
+ result["valid"] = bool(result.get("objects", {}).get("valid") and result.get("relations", {}).get("valid"))
+ write_audit_pair(output_dir, "final_validation", "schema_validation", result, "Schema Validation")
+ return result
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Validate multi-agent scene delivery final outputs.")
+ parser.add_argument("--objects_json", required=True, type=Path)
+ parser.add_argument("--relations_json", required=True, type=Path)
+ parser.add_argument("--output_dir", required=True, type=Path)
+ parser.add_argument("--scene_state", type=Path, default=None)
+ parser.add_argument("--object_revision_audit", type=Path, default=None)
+ parser.add_argument("--wall_epsilon", type=float, default=WALL_EPSILON)
+ parser.add_argument("--ceiling_confirmed", action="store_true")
+ parser.add_argument("--dry_run", action="store_true")
+ args = parser.parse_args()
+ result = validate_files(
+ args.objects_json,
+ args.relations_json,
+ args.output_dir,
+ args.ceiling_confirmed,
+ args.dry_run,
+ scene_state_path=args.scene_state,
+ object_revision_audit_path=args.object_revision_audit,
+ wall_epsilon=args.wall_epsilon,
+ )
+ print(json.dumps(result, indent=2, ensure_ascii=False))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/relations/agents/state.py b/vlm_agent/reconstruction/relations/agents/state.py
new file mode 100644
index 0000000..e44f899
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/state.py
@@ -0,0 +1,87 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+AUDIT_FILES = {
+ "walls": ["wall_audit"],
+ "ceiling": ["ceiling_audit"],
+ "objects": ["object_grounding_audit", "object_revision_audit"],
+ "attach": ["attach_decision_audit"],
+ "rendering": ["topdown_render_log"],
+ "agents": ["agent_task_log"],
+ "supervisor": ["supervisor_log"],
+ "final_validation": ["schema_validation"],
+}
+
+
+def write_json(path: Path, data: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
+
+
+def read_json(path: Path) -> Any:
+ return json.loads(path.read_text(encoding="utf-8"))
+
+
+def write_md(path: Path, title: str, lines: list[str]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ body = [f"# {title}", ""]
+ body.extend(lines)
+ path.write_text("\n".join(body).rstrip() + "\n", encoding="utf-8")
+
+
+def ensure_output_tree(output_dir: Path) -> None:
+ dirs = [
+ "scene_states",
+ "policy",
+ "manager_plans",
+ "agent_outputs/task_0001",
+ "generated_evidence/topdown",
+ "final_outputs",
+ ]
+ for rel in dirs:
+ (output_dir / rel).mkdir(parents=True, exist_ok=True)
+ for group in AUDIT_FILES:
+ (output_dir / "debug" / group).mkdir(parents=True, exist_ok=True)
+
+
+def write_audit_pair(output_dir: Path, group: str, name: str, data: dict[str, Any], title: str | None = None) -> None:
+ json_path = output_dir / "debug" / group / f"{name}.json"
+ md_path = output_dir / "debug" / group / f"{name}.md"
+ write_json(json_path, data)
+ lines = []
+ for key, value in data.items():
+ if isinstance(value, (dict, list)):
+ rendered = json.dumps(value, indent=2, ensure_ascii=False)
+ lines.append(f"## {key}")
+ lines.append("")
+ lines.append("```json")
+ lines.append(rendered)
+ lines.append("```")
+ else:
+ lines.append(f"- **{key}**: {value}")
+ write_md(md_path, title or name.replace("_", " ").title(), lines)
+
+
+def initialize_empty_audits(output_dir: Path, dry_run: bool) -> None:
+ base = {
+ "dry_run": dry_run,
+ "status": "initialized",
+ "note": "Placeholder audit created by the multi-agent scene delivery scaffold.",
+ "items": [],
+ }
+ for group, names in AUDIT_FILES.items():
+ for name in names:
+ if group == "final_validation":
+ continue
+ write_audit_pair(output_dir, group, name, dict(base), title=name.replace("_", " ").title())
+
+
+def sorted_object_id_key(object_id: str) -> tuple[int, str]:
+ try:
+ return (int(str(object_id).split("_")[-1]), str(object_id))
+ except Exception:
+ return (10**9, str(object_id))
diff --git a/vlm_agent/reconstruction/relations/agents/supervisor.py b/vlm_agent/reconstruction/relations/agents/supervisor.py
new file mode 100644
index 0000000..f424c6a
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/supervisor.py
@@ -0,0 +1,14 @@
+from __future__ import annotations
+
+from typing import Any
+
+from relations.agents.api_client import SceneDeliveryAPIClient
+
+
+def supervise_execution(client: SceneDeliveryAPIClient, system_prompt: str, scene_state: dict[str, Any], recent_results: list[dict[str, Any]]) -> dict[str, Any]:
+ return client.call_json(
+ "ExecutionSupervisorAgent",
+ system_prompt,
+ {"scene_state": scene_state, "recent_results": recent_results},
+ use_vision_model=False,
+ )
diff --git a/vlm_agent/reconstruction/relations/agents/task_registry.py b/vlm_agent/reconstruction/relations/agents/task_registry.py
new file mode 100644
index 0000000..7c398b4
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/task_registry.py
@@ -0,0 +1,666 @@
+from __future__ import annotations
+
+import copy
+import re
+from typing import Any
+
+
+OBJECT_ANCHOR_MODES = {
+ "ceiling_only",
+ "wall_only",
+ "floor_wall",
+ "support_only",
+ "contain_candidate",
+ "uncertain",
+}
+OBJECT_ANCHOR_CONFIDENCES = {"high", "medium", "low"}
+
+
+TASK_KIND_REGISTRY: dict[str, dict[str, Any]] = {
+ "support_contact_extraction": {
+ "description": "Extract gravity-direction floor/object support and direct contact relations.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "required_reasoning",
+ "requires_artifacts": ["object_list", "object_footprints", "original_image", "mask_overlay", "canonical_topdown"],
+ "provides_capabilities": ["support_relations_ready"],
+ "image_profile": ["original_image", "mask_overlay", "canonical_topdown"],
+ "instructions": [
+ "Gravity-direction support is the first relation pass. Return one explicit support_contact_candidates verdict for every object in the hydrated gravity_coverage_scope before resolving Manager-priority ambiguous cases.",
+ "Extract physically supported attach relations in the gravity direction: floor-to-object and object-to-object support/contact. Judge actual contact from the supplied image, mask, and footprints; do not replace a plausible object support with floor merely for coverage.",
+ "Wall contact is a separate horizontal relation. This task may report floor/object support without deciding wall attachment.",
+ "The Manager priority scope identifies difficult cases; it never replaces gravity_coverage_scope. For every object you accept as a support anchor for another object, also decide that anchor's own gravity support unless it is exclusively wall-mounted, ceiling-mounted, or contained. Floor support and wall attach may coexist for the same freestanding object.",
+ "Use concise reason_code values for obvious floor support. When an object is visibly inside a container or exclusively wall/ceiling mounted, return defer_to_contain or exclusive_wall_or_ceiling rather than inventing floor support. For a clearly visible inside-container case, include containment_hypothesis with decision=inside and the object anchor/target; otherwise use decision=uncertain. Do not use contain as a support_mode.",
+ "An open-front shelf, bookcase bay, display cubby, or cabinet compartment is an interior containment volume. An object resting on an internal shelf board is contained by that furniture even though the board also provides gravity support. For ordinary single-level furniture, a target on a plain exterior top remains object-to-object attach.",
+ "Multi-level furniture uses a simpler ontology: a target placed on or within any functional level of a bunk bed, bookcase/shelving unit, multi-tier service cart, or multi-level rack is contain, including the top level. Do not emit object-to-object attach for the same multi-level-furniture pair. Use defer_to_contain with containment_hypothesis.decision=inside and support_surface_role=multi_level_tier. Decide tier membership from the original image, not SAM3D bbox overlap alone.",
+ "Keep support_mode structurally consistent with the anchor: floor requires object_anchor=floor; object requires object_anchor=object_###; exclusive_wall_or_ceiling is valid only for a genuinely wall- or ceiling-mounted target. Never label an object described as on another object's surface as exclusive_wall_or_ceiling.",
+ "Treat Manager scene focus as a hypothesis, never as contact evidence. Judge gravity support independently from any contain task. Accept exterior object support only when the supplied images show direct surface contact; use defer_to_contain only when the same images positively show that the target is inside an interior volume.",
+ "Before every support verdict, ground the object_id from its image bbox and mask region. Category and description are weak labels and location phrases such as on the bunk, on the desk, or inside the cabinet must never override the grounded mask. When a clear image/mask conflict would change the support anchor, follow the mask for the relation and emit one concise object_revisions patch. Do not spend output on cosmetic naming improvements that cannot change a relation.",
+ ],
+ "expected_output_schema": {
+ "support_contact_candidates": [{
+ "decision": "accept|reject|uncertain",
+ "support_mode": "floor|object|defer_to_contain|exclusive_wall_or_ceiling|uncertain",
+ "object_anchor": "floor|object_###|none",
+ "object_target": "object_###",
+ "reason_code": "short_reason_code",
+ "support_surface_role": "internal_shelf|bounded_tier|multi_level_tier|outer_top|external_surface|not_applicable|uncertain",
+ "containment_hypothesis": {
+ "decision": "inside|uncertain",
+ "object_anchor": "object_###",
+ "object_target": "object_###",
+ },
+ }],
+ "accepted_relations": [{
+ "relation": "attach",
+ "parameters": {"object_anchor": "floor|object_###", "object_target": "object_###"},
+ }],
+ "object_revisions": [{
+ "object_id": "object_###",
+ "new_category": "optional corrected category",
+ "new_description": "optional corrected description",
+ "reason": "image/mask-grounded conflict that changes support",
+ "confidence": "high|medium|low",
+ }],
+ },
+ },
+ "contain_relation_extraction": {
+ "description": "Extract object-in-container relations from positive visual/spatial evidence.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "optional_reasoning",
+ "requires_artifacts": ["object_list", "object_footprints", "original_image", "mask_overlay", "contain_candidates"],
+ "provides_capabilities": ["contain_relations_ready"],
+ "image_profile": ["original_image", "mask_overlay"],
+ "instructions": [
+ "Extract contain only when the target lies inside an anchor object's cabinet, shelf, cubby, box, drawer, basket, tray, or other interior storage/display volume. Local candidates only select pairs to inspect and carry zero positive evidentiary weight. Decide from the original image and mask: reject false positives and add directly evident omissions.",
+ "For ordinary single-level anchors, a usable interior, shelf/cubby, cabinet compartment, drawer, box, basket, tray, or other container volume is contain, while a target merely resting on a plain exterior top is attach. A door or fully enclosed cavity is not required.",
+ "For multi-level furniture, every occupied functional level is contain, regardless of whether it is lower, middle, or top and regardless of rails or rims. This includes both bunks of a bunk bed and every tier of a multi-tier service cart, bookcase/shelving unit, or rack. Do not output attach for the same anchor-target pair. Decide whether the target belongs to a level from the original image and mask, not from local bbox overlap or candidate score.",
+ "For the same anchor-target pair, contain and attach are mutually exclusive; emit contain when interior membership is the correct relation.",
+ "Treat the Manager object scope as a priority rather than a closed set. Return an explicit accept, reject, or uncertain decision for every retained local recall pair supplied at runtime, even when its objects were outside the original Manager scope. Bbox overlap, a local score, or category compatibility never proves containment; when the image does not positively show interior membership, return uncertain or reject.",
+ "For every accepted contain anchor, report render_direction as front for front-open cabinets/shelves/display volumes and front-accessed internal tiers of multi-level furniture such as bunk beds, or up for top-open boxes/baskets/trays/drawers. Final assembly validates and writes this property to objects.json.",
+ "Before every contain verdict, ground the object_id from its image bbox and mask region. Category and description are weak labels and location phrases such as on the bunk, on the desk, or inside the cabinet must never override the grounded mask. When a clear image/mask conflict would change contain versus attach, follow the mask for the relation and emit one concise object_revisions patch. Do not spend output on cosmetic naming improvements that cannot change a relation.",
+ ],
+ "expected_output_schema": {
+ "contain_candidates": [{
+ "decision": "accept|reject|uncertain",
+ "object_anchor": "object_###",
+ "object_target": "object_###",
+ "evidence": ["..."],
+ "reason": "...",
+ }],
+ "accepted_contain_relations": [{
+ "relation": "contain",
+ "parameters": {"object_anchor": "object_###", "object_target": "object_###"},
+ }],
+ "contain_anchor_properties": [{
+ "object_id": "object_###",
+ "contain_anchor": True,
+ "render_direction": "front|up",
+ "reason": "...",
+ }],
+ "object_revisions": [{
+ "object_id": "object_###",
+ "new_category": "optional corrected category",
+ "new_description": "optional corrected description",
+ "reason": "image/mask-grounded conflict that changes contain versus attach",
+ "confidence": "high|medium|low",
+ }],
+ },
+ },
+ "object_revision_check": {
+ "description": "Propose only semantic object revisions that materially affect relation extraction.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "optional_reasoning",
+ "requires_artifacts": ["object_list", "original_image", "mask_overlay"],
+ "provides_capabilities": ["object_revisions_ready"],
+ "image_profile": ["original_image", "mask_overlay"],
+ "instructions": [
+ "Check only category or description errors that materially affect relation extraction. Preserve every SAM3D object id and return revision patches rather than a replacement object list.",
+ "Treat descriptions as weak evidence and ground revisions in the original image and mask.",
+ ],
+ "expected_output_schema": {
+ "object_revisions": [{
+ "object_id": "object_###",
+ "new_category": "optional corrected category",
+ "new_description": "optional corrected description",
+ "reason": "...",
+ "confidence": "high|medium|low",
+ }],
+ },
+ },
+ "object_physical_property_reasoning": {
+ "description": "Judge floor-covering semantics needed by deterministic relation normalization.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "optional_reasoning",
+ "requires_artifacts": ["object_list", "object_footprints", "original_image", "mask_overlay"],
+ "provides_capabilities": ["object_physical_properties_ready"],
+ "image_profile": ["original_image", "mask_overlay"],
+ "instructions": [
+ "Judge only whether an object is a floor covering whose apparent object-to-object support relations should normalize to floor support. Return uncertain judgements explicitly; final assembly has deterministic fallback hints.",
+ "Do not output upright. The independent Physics Sidecar owns the general upright prior, and final assembly applies relation-aware wall/ceiling overrides.",
+ "Do not infer relation changes beyond the floor-covering classification itself.",
+ ],
+ "expected_output_schema": {
+ "object_physical_properties": [{
+ "object_id": "object_###",
+ "is_floor_covering": True,
+ "floor_covering_confidence": "high|medium|low",
+ "floor_covering_reason": "...",
+ }],
+ },
+ },
+ "room_material_classification": {
+ "description": "Describe the dominant visible wall, floor, and ceiling finish materials for final room metadata.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "required_reasoning",
+ "requires_artifacts": ["original_image"],
+ "provides_capabilities": ["room_materials_ready"],
+ "image_profile": ["original_image"],
+ "instructions": [
+ "Inspect the original scene image and describe the dominant visible finish for walls, floor, and ceiling.",
+ "Return exactly one room_materials object with the keys walls, floor, and ceiling. Each value must be a concise non-empty English material/finish phrase suitable for rendering, such as off-white painted wall, light oak wooden floor, or flat white plaster ceiling.",
+ "Do not copy the example phrases unless they match the image. Do not infer materials from SAM3D object categories or wall geometry. When a surface is only partly visible, return the most specific description supported by the image and avoid unsupported brand, construction, or hidden-layer claims.",
+ "This capability may be merged into another task that already receives the original image; do not return relations, wall coordinates, or object revisions solely for this capability.",
+ ],
+ "expected_output_schema": {
+ "room_materials": {
+ "walls": "concise visible wall material/finish",
+ "floor": "concise visible floor material/finish",
+ "ceiling": "concise visible ceiling material/finish",
+ },
+ },
+ },
+ "ceiling_judgement": {
+ "description": "Decide whether ceiling is a valid attach anchor for concrete objects.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "optional_reasoning",
+ "requires_artifacts": ["object_list", "original_image", "mask_overlay"],
+ "provides_capabilities": ["ceiling_judgement_ready"],
+ "image_profile": ["original_image", "mask_overlay"],
+ "instructions": [
+ "Judge whether the scene provides positive evidence that ceiling is a valid anchor, and identify only concrete ceiling-mounted targets. Do not allow ceiling attachment from category text alone when visual evidence contradicts it.",
+ ],
+ "expected_output_schema": {
+ "ceiling_decision": {
+ "allow_ceiling_anchor": True,
+ "ceiling_confirmed": True,
+ "evidence": ["..."],
+ "reason": "...",
+ },
+ "accepted_relations": [{
+ "relation": "attach",
+ "parameters": {"object_anchor": "ceiling", "object_target": "object_###"},
+ }],
+ },
+ },
+ "wall_alignment_object_review": {
+ "description": "Identify real wall-following objects; local code scores finite OBB axes from their canonical top-down rotations.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "required_reasoning",
+ "requires_artifacts": ["object_list", "original_image", "mask_overlay", "wall_orientation_candidates_overlay"],
+ "provides_capabilities": ["wall_alignment_objects_ready", "wall_object_hypotheses_ready"],
+ "image_profile": ["original_image", "mask_overlay", "wall_orientation_candidates_overlay"],
+ "formal_geometry_controlled": True,
+ "instructions": [
+ "Identify only objects whose real pose reliably reveals a room-wall axis: wall-mounted, wall-embedded, physically against a wall, or clearly parallel/aligned with a wall. Use the original image for semantics and the mask overlay for object ids.",
+ "Treat object_scope as a priority, not a closed set. Include directly evident omissions, but exclude central furniture, round objects, small clutter, and objects whose heading does not reliably follow a wall.",
+ "Do not choose the winning room-orientation candidate, final wall id, coordinate, angle, visibility label, or attach relation. Local code reads rotation_degrees_topdown for the selected object ids and scores the finite formal OBB candidates.",
+ "For each likely wall object, use the isolated candidate panels to return candidate_wall_bindings keyed by exact candidate_id and provisional_wall_id. The binding is semantic evidence only and does not select a final candidate or accept attach.",
+ "Keep the response compact. Use high or medium confidence only when the wall-following evidence is concrete; omit uncertain objects instead of returning low-confidence filler.",
+ ],
+ "expected_output_schema": {
+ "likely_wall_objects": [{
+ "object_id": "object_###",
+ "role": "wall_mounted|wall_embedded|against_wall|wall_aligned",
+ "confidence": "high|medium",
+ "evidence_code": "short_visual_evidence_code",
+ }],
+ "candidate_wall_bindings": [{
+ "object_id": "object_###",
+ "candidate_id": "copy exactly from candidate panel",
+ "provisional_wall_id": "wall_001|wall_002|wall_003|wall_004",
+ "role": "wall_mounted|wall_embedded|against_wall|wall_aligned",
+ "confidence": "high|medium",
+ "evidence_code": "short_visual_binding_code",
+ }],
+ },
+ },
+ "wall_object_hypothesis": {
+ "description": "Identify likely wall-following objects as semantic evidence for local formal-wall orientation scoring.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "required_reasoning",
+ "requires_artifacts": ["object_list", "original_image", "mask_overlay"],
+ "provides_capabilities": ["wall_alignment_objects_ready", "wall_object_hypotheses_ready"],
+ "image_profile": ["original_image", "mask_overlay"],
+ "formal_geometry_controlled": True,
+ "instructions": [
+ "Identify only real wall-mounted, wall-embedded, physically against-wall, or clearly wall-aligned objects. These object ids are semantic evidence for local orientation scoring and later wall-contact review.",
+ "Do not output candidate wall ids, coordinates, lines, angles, visibility labels, or relations before local wall orientation is resolved.",
+ "Treat object_scope as a priority rather than a closed set, and omit uncertain central or round objects.",
+ ],
+ "expected_output_schema": {
+ "likely_wall_objects": [{
+ "object_id": "object_###",
+ "role": "wall_mounted|wall_embedded|against_wall|wall_aligned",
+ "confidence": "high|medium",
+ "evidence_code": "short_visual_evidence_code",
+ }],
+ },
+ },
+ "provisional_wall_visibility_review": {
+ "description": "Review original-image visibility semantics for labelled edges in finite pre-orientation OBB candidate panels.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "required_reasoning",
+ "requires_artifacts": ["original_image", "mask_overlay", "wall_orientation_candidates_overlay"],
+ "provides_capabilities": ["provisional_wall_visibility_ready"],
+ "image_profile": ["original_image", "mask_overlay", "wall_orientation_candidates_overlay"],
+ "formal_geometry_controlled": True,
+ "instructions": [
+ "Use the isolated finite candidate panels only to judge which labelled candidate edges correspond to directly visible real walls versus inferred, weak, or closure-only room boundaries. Each panel is independently labelled wall_001 through wall_004.",
+ "Return compact records keyed by candidate_id and provisional_wall_id. You may judge visibility semantics, but do not choose the winning orientation candidate, create coordinates, alter an edge, emit a final wall id, or emit any attach relation.",
+ "Visible or partially_visible requires direct source-image evidence of a wall surface or a wall-floor/wall-ceiling junction. Do not treat a furniture contour, crop border, occlusion edge, or proximity to a mesh rectangle as a visible wall.",
+ "When evidence is absent, use inferred/not_visible and can_anchor_wall_attach=false rather than guessing. These are provisional records; local code binds only the records belonging to the orientation selected later from wall-following objects.",
+ ],
+ "expected_output_schema": {
+ "provisional_wall_visibility_review": [{
+ "candidate_id": "copy exactly from the candidate panel header",
+ "provisional_wall_id": "wall_001|wall_002|wall_003|wall_004",
+ "original_image_visibility": "visible|partially_visible|not_visible|inferred",
+ "weak_wall": False,
+ "closure_only": False,
+ "can_anchor_wall_attach": True,
+ "reason_code": "short_reason_code",
+ }],
+ },
+ },
+ "wall_visibility_semantics_review": {
+ "description": "Review visibility and weak/inferred semantics for the already selected, canonical formal OBB walls.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "required_reasoning",
+ "requires_artifacts": ["formal_obb_walls", "original_image", "mask_overlay"],
+ "requires_capabilities": ["wall_alignment_objects_ready"],
+ "provides_capabilities": ["wall_visibility_semantics_ready"],
+ "image_profile": ["original_image", "wall_attach_prefilter_overlay"],
+ "formal_geometry_controlled": True,
+ "instructions": [
+ "The local scorer has already selected and applied the formal OBB orientation. Judge original-image visibility, weak/inferred or closure-only status, and attach eligibility for the supplied final wall references.",
+ "Copy every supplied wall_id and geometry_key exactly. Do not choose another candidate and do not rotate, edit, renumber, or replace wall geometry.",
+ "Return one compact record for each of wall_001 through wall_004.",
+ ],
+ "expected_output_schema": {
+ "wall_visibility_review": [{
+ "wall_id": "wall_001",
+ "geometry_key": "copy supplied wall_edge_... exactly",
+ "original_image_visibility": "visible|partially_visible|not_visible|inferred",
+ "weak_wall": False,
+ "closure_only": False,
+ "can_anchor_wall_attach": True,
+ "reason_code": "short_reason_code",
+ }],
+ },
+ },
+ "wall_final_adjudication": {
+ "description": "Perform detailed semantic adjudication of already selected canonical formal OBB walls.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "required_reasoning",
+ "requires_artifacts": ["formal_obb_walls", "wall_evidence", "wall_visual_evidence", "original_image", "mask_overlay"],
+ "requires_capabilities": ["wall_alignment_objects_ready"],
+ "provides_capabilities": ["wall_visibility_semantics_ready"],
+ "image_profile": ["original_image", "mask_overlay"],
+ "formal_geometry_controlled": True,
+ "instructions": [
+ "The formal OBB orientation and canonical wall ids have already been selected by local scoring from Agent-identified wall-following objects. Audit visibility, weak/inferred closure status, and attach eligibility only.",
+ "Copy every supplied wall_id and geometry_key exactly. Never choose another candidate or invent/edit coordinates, angles, ids, or methods.",
+ ],
+ "expected_output_schema": {
+ "decision": "verified_final_walls|missing_or_invalid_formal_obb_wall_evidence",
+ "wall_visibility_review": [{
+ "wall_id": "wall_001",
+ "geometry_key": "copy supplied wall_edge_... exactly",
+ "original_image_visibility": "visible|partially_visible|not_visible|inferred",
+ "weak_wall": False,
+ "closure_only": False,
+ "can_anchor_wall_attach": True,
+ "reason_code": "short_reason_code",
+ }],
+ "reason": "required only when final references are invalid",
+ },
+ },
+ "wall_attach_candidate_recall": {
+ "description": "Review final-wall visibility and judge compact locally prefiltered wall-attach candidates in one downstream call.",
+ "runner_action": "api_dynamic_task",
+ "runtime_class": "required_reasoning",
+ "requires_artifacts": ["object_list", "object_footprints", "formal_obb_walls", "wall_evidence", "original_image", "mask_overlay"],
+ "requires_capabilities": ["wall_alignment_objects_ready", "wall_attach_candidates_ready"],
+ "provides_capabilities": ["wall_visibility_semantics_ready", "wall_attach_decisions_ready"],
+ "image_profile": ["original_image", "mask_overlay", "wall_attach_prefilter_overlay"],
+ "formal_geometry_controlled": True,
+ "instructions": [
+ "The local scorer has already selected and applied the formal OBB orientation. First return visibility/weak/inferred/closure-only semantics for all four supplied final wall references, copying wall_id and geometry_key exactly.",
+ "Then review the compact local wall-attach candidate set. One object may have up to two plausible neighboring wall ids; compare every supplied pair on the numbered overlay, accept at most one wall for each object, and reject the other pair explicitly.",
+ "The candidate wall id is a review option, not a pre-decided answer. Choose only among the supplied formal wall ids and never transfer visual evidence from one wall to its opposite or adjacent edge.",
+ "Floor/object support may coexist with one wall attach. Do not confuse proximity, facing, an isolated corner, or orientation alignment with physical wall contact.",
+ "When a target is already supported by another object, reject ordinary wall proximity. Preserve an additional wall attach only with explicit wall-mounted, embedded, hung, rack, or bracket evidence.",
+ "A weak, hidden, inferred, closure-only, or can_anchor_wall_attach=false wall may close room geometry but cannot anchor a final wall attach.",
+ "An image crop boundary, furniture side panel, occlusion contour, or room-closure edge is not evidence that a wall is visible. Mark visible or partially_visible only when the actual wall surface or a wall-floor/wall-ceiling junction is directly visible in the source image.",
+ "Do not choose another orientation candidate and do not edit coordinates, angles, wall ids, or methods.",
+ ],
+ "expected_output_schema": {
+ "wall_visibility_review": [{
+ "wall_id": "wall_001",
+ "geometry_key": "copy supplied wall_edge_... exactly",
+ "original_image_visibility": "visible|partially_visible|not_visible|inferred",
+ "weak_wall": False,
+ "closure_only": False,
+ "can_anchor_wall_attach": True,
+ "reason_code": "short_reason_code",
+ }],
+ "wall_attach_candidates": [{
+ "decision": "accept|reject",
+ "object_target": "object_###",
+ "candidate_wall_id": "wall_001|wall_002|wall_003|wall_004",
+ "evidence_codes": ["short_code"],
+ "reason_code": "short_reason_code",
+ }],
+ "accepted_attach_candidates": [{
+ "relation": "attach",
+ "parameters": {"object_anchor": "wall_001", "object_target": "object_###"},
+ }],
+ },
+ },
+ "local_validation": {
+ "description": "Represent deterministic local validation without an API DynamicTaskAgent call.",
+ "runner_action": "local_schema_validation",
+ "runtime_class": "local_redundant_validation",
+ "requires_artifacts": ["scene_state"],
+ "provides_capabilities": ["local_validation_ready"],
+ "image_profile": [],
+ "instructions": ["Run deterministic local relation/object schema validation."],
+ "expected_output_schema": {"valid": True, "errors": []},
+ },
+}
+
+
+TASK_KIND_ALIASES = {
+ "support_contact": "support_contact_extraction",
+ "support_relation_extraction": "support_contact_extraction",
+ "contain_relations": "contain_relation_extraction",
+ "contain_relation_probe": "contain_relation_extraction",
+ "object_revision": "object_revision_check",
+ "object_revision_audit": "object_revision_check",
+ "object_physical_properties": "object_physical_property_reasoning",
+ "room_materials": "room_material_classification",
+ "room_materials_classification": "room_material_classification",
+ "ceiling_anchor_judgement": "ceiling_judgement",
+ "ceiling_attach_judgement": "ceiling_judgement",
+ "ceiling_anchor_review": "ceiling_judgement",
+ "wall_alignment_review": "wall_alignment_object_review",
+ "wall_orientation_object_review": "wall_alignment_object_review",
+ "provisional_wall_semantic_review": "provisional_wall_visibility_review",
+ "candidate_wall_visibility_review": "provisional_wall_visibility_review",
+ "wall_attach_recall": "wall_attach_candidate_recall",
+ "final_validation": "local_validation",
+ "final_consistency_validation": "local_validation",
+}
+
+
+CAPABILITY_TO_KIND: dict[str, str] = {}
+for _kind, _spec in TASK_KIND_REGISTRY.items():
+ for _capability in _spec.get("provides_capabilities", []):
+ CAPABILITY_TO_KIND.setdefault(str(_capability), _kind)
+
+
+def normalize_task_kind(value: Any) -> str:
+ kind = str(value or "").strip().lower().replace("-", "_").replace(" ", "_")
+ return TASK_KIND_ALIASES.get(kind, kind)
+
+
+def task_image_artifact_profiles() -> dict[str, tuple[str, ...]]:
+ return {
+ kind: tuple(str(item) for item in spec.get("image_profile", []))
+ for kind, spec in TASK_KIND_REGISTRY.items()
+ }
+
+
+def compact_manager_capability_catalog() -> list[dict[str, Any]]:
+ result = []
+ for kind, spec in TASK_KIND_REGISTRY.items():
+ result.append({
+ "canonical_task_kind": kind,
+ "description": spec["description"],
+ "provides_capabilities": list(spec.get("provides_capabilities", [])),
+ "requires_capabilities": list(spec.get("requires_capabilities", [])),
+ "runner_controlled": bool(spec.get("runner_controlled", False)),
+ "formal_geometry_controlled": bool(spec.get("formal_geometry_controlled", False)),
+ })
+ return result
+
+
+def _ordered_union(*collections: Any) -> list[str]:
+ result: list[str] = []
+ for collection in collections:
+ for item in collection or []:
+ value = str(item)
+ if value and value not in result:
+ result.append(value)
+ return result
+
+
+def _safe_task_id(value: Any, index: int) -> str:
+ raw = str(value or f"node_{index:03d}").strip().lower()
+ normalized = re.sub(r"[^a-z0-9_]+", "_", raw).strip("_") or f"node_{index:03d}"
+ return normalized if normalized.startswith("task_") else f"task_{normalized}"
+
+
+def _merge_schema(target: dict[str, Any], source: Any) -> None:
+ if not isinstance(source, dict):
+ return
+ for key, value in source.items():
+ if key not in target:
+ target[key] = copy.deepcopy(value)
+ elif isinstance(target[key], dict) and isinstance(value, dict):
+ _merge_schema(target[key], value)
+
+
+def normalize_object_anchor_routing(value: Any) -> list[dict[str, str]]:
+ """Normalize compact Manager routing without making scene decisions locally."""
+ if not isinstance(value, list):
+ return []
+ confidence_rank = {"low": 1, "medium": 2, "high": 3}
+ by_object: dict[str, dict[str, str]] = {}
+ order: list[str] = []
+ for item in value:
+ if not isinstance(item, dict):
+ continue
+ object_id = str(item.get("object_id") or "").strip()
+ anchor_mode = str(item.get("anchor_mode") or "").strip().lower()
+ confidence = str(item.get("confidence") or "").strip().lower()
+ if not object_id.startswith("object_") or anchor_mode not in OBJECT_ANCHOR_MODES:
+ continue
+ if confidence not in OBJECT_ANCHOR_CONFIDENCES:
+ confidence = "low"
+ record = {
+ "object_id": object_id,
+ "anchor_mode": anchor_mode,
+ "confidence": confidence,
+ }
+ previous = by_object.get(object_id)
+ if previous is None:
+ order.append(object_id)
+ if previous is None or confidence_rank[confidence] > confidence_rank[previous["confidence"]]:
+ by_object[object_id] = record
+ return [by_object[object_id] for object_id in order]
+
+
+def compile_compact_manager_plan(raw_plan: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any]]:
+ raw_tasks = [item for item in raw_plan.get("tasks", []) if isinstance(item, dict)]
+ compiled_tasks: list[dict[str, Any]] = []
+ aliases: dict[str, str] = {}
+ used_ids: set[str] = set()
+ audits: list[dict[str, Any]] = []
+
+ for index, node in enumerate(raw_tasks, start=1):
+ base_id = _safe_task_id(node.get("node_id") or node.get("task_id"), index)
+ task_id = base_id
+ suffix = 2
+ while task_id in used_ids:
+ task_id = f"{base_id}_{suffix}"
+ suffix += 1
+ used_ids.add(task_id)
+ for alias in (node.get("node_id"), node.get("task_id"), task_id):
+ if alias:
+ aliases[str(alias)] = task_id
+
+ primary_kind = normalize_task_kind(node.get("canonical_task_kind") or node.get("task_type") or "custom_reasoning_task")
+ requested_capabilities = _ordered_union(node.get("provides_capabilities"))
+ template_kinds: list[str] = []
+ if primary_kind in TASK_KIND_REGISTRY:
+ template_kinds.append(primary_kind)
+ for capability in requested_capabilities:
+ template_kind = CAPABILITY_TO_KIND.get(capability)
+ if template_kind and template_kind not in template_kinds:
+ template_kinds.append(template_kind)
+
+ specs = [TASK_KIND_REGISTRY[kind] for kind in template_kinds]
+ manager_custom = not specs
+ task_type = primary_kind
+ agent_role = "DynamicTaskAgent"
+ runner_action = "api_dynamic_task"
+ runtime_class = "optional_reasoning" if manager_custom else (
+ "required_reasoning" if any(spec.get("runtime_class") == "required_reasoning" for spec in specs)
+ else str(specs[0].get("runtime_class", "optional_reasoning"))
+ )
+ if specs:
+ runner_action = str(specs[0].get("runner_action", runner_action))
+ if any(spec.get("runner_action") == "api_dynamic_task" for spec in specs):
+ runner_action = "api_dynamic_task"
+
+ required_artifacts = _ordered_union(
+ *(spec.get("requires_artifacts", []) for spec in specs),
+ node.get("requires_artifacts", []),
+ )
+ requires_capabilities = _ordered_union(
+ *(spec.get("requires_capabilities", []) for spec in specs),
+ node.get("requires_capabilities", []),
+ )
+ provides_capabilities = _ordered_union(
+ *(spec.get("provides_capabilities", []) for spec in specs),
+ requested_capabilities,
+ )
+ image_profile = _ordered_union(*(spec.get("image_profile", []) for spec in specs))
+ instructions = _ordered_union(*(spec.get("instructions", []) for spec in specs))
+ scene_focus = str(node.get("scene_focus") or "").strip()
+ object_scope = [str(item) for item in node.get("object_scope", [])] if isinstance(node.get("object_scope"), list) else []
+ reason = str(node.get("reason") or "").strip()
+ custom_instructions = node.get("custom_instructions")
+ if scene_focus:
+ if "support_contact_extraction" in template_kinds:
+ instructions.append(
+ "Manager-proposed scene focus (hypothesis only; verify every claimed surface or contact against supplied evidence): "
+ + scene_focus
+ )
+ elif "contain_relation_extraction" in template_kinds:
+ instructions.append(
+ "Manager-proposed scene focus is a planning hypothesis only, not semantic evidence. "
+ "Verify every claimed inside, outside, or top-surface statement independently from the original image, mask, "
+ "and per-pair image bboxes; explicitly review every hydrated candidate pair even when the Manager omitted it: "
+ + scene_focus
+ )
+ else:
+ instructions.append(f"Scene-specific focus from Manager: {scene_focus}")
+ if object_scope:
+ instructions.append("Manager-selected priority object scope: " + ", ".join(object_scope))
+ if isinstance(custom_instructions, list):
+ instructions.extend(str(item) for item in custom_instructions if str(item).strip())
+ elif isinstance(custom_instructions, str) and custom_instructions.strip():
+ instructions.append(custom_instructions.strip())
+ if manager_custom and not instructions:
+ instructions.append("Complete the custom reasoning task described by the Manager reason and scene focus using only supplied evidence.")
+
+ expected_output_schema: dict[str, Any] = {}
+ for spec in specs:
+ _merge_schema(expected_output_schema, spec.get("expected_output_schema"))
+ _merge_schema(expected_output_schema, node.get("expected_output_schema"))
+ if not expected_output_schema:
+ expected_output_schema = {
+ "status": "completed|insufficient_evidence",
+ "findings": [],
+ "proposed_state_updates": {},
+ "unresolved_items": [],
+ }
+
+ task = {
+ "task_id": task_id,
+ "task_type": task_type,
+ "canonical_task_kind": primary_kind,
+ "agent_role": agent_role,
+ "runner_action": runner_action,
+ "runtime_class": runtime_class,
+ "depends_on": list(node.get("depends_on", [])) if isinstance(node.get("depends_on"), list) else [],
+ "requires_artifacts": required_artifacts,
+ "requires_capabilities": requires_capabilities,
+ "provides_capabilities": provides_capabilities,
+ "needs_tool": False,
+ "use_vision_model": bool(image_profile),
+ "image_artifact_profile": image_profile,
+ "instructions": instructions,
+ "expected_output_schema": expected_output_schema,
+ "reason": reason,
+ "object_scope": object_scope,
+ "scene_focus": scene_focus,
+ "compiled_from_compact_manager_plan": True,
+ }
+ compiled_tasks.append(task)
+ audits.append({
+ "task_id": task_id,
+ "source_node_id": node.get("node_id") or node.get("task_id"),
+ "canonical_task_kind": primary_kind,
+ "template_kinds": template_kinds,
+ "custom_task": manager_custom,
+ "field_sources": {
+ "task_id": "compiler",
+ "task_type": "registry_default" if specs else "manager",
+ "canonical_task_kind": "manager",
+ "agent_role": "registry_default",
+ "runner_action": "registry_default",
+ "runtime_class": "registry_default",
+ "depends_on": "manager_then_compiler_remap",
+ "requires_artifacts": "registry_default+manager",
+ "requires_capabilities": "registry_default+manager",
+ "provides_capabilities": "registry_default+manager",
+ "instructions": "registry_default+manager_scene_brief",
+ "expected_output_schema": "registry_default+manager",
+ "needs_tool": "registry_default",
+ },
+ })
+
+ for task in compiled_tasks:
+ remapped: list[str] = []
+ for dependency in task.get("depends_on", []):
+ mapped = aliases.get(str(dependency), str(dependency))
+ if mapped != task["task_id"] and mapped not in remapped:
+ remapped.append(mapped)
+ task["depends_on"] = remapped
+
+ object_anchor_routing = normalize_object_anchor_routing(raw_plan.get("object_anchor_routing"))
+ compiled_plan = {
+ "plan_id": raw_plan.get("plan_id", "manager_plan_000"),
+ "objective": raw_plan.get("objective", ""),
+ "object_anchor_routing": object_anchor_routing,
+ "tasks": compiled_tasks,
+ "stop_if": raw_plan.get("stop_if", []),
+ "manager_plan_format": "compact_compiled",
+ "compact_source_task_count": len(raw_tasks),
+ }
+ audit = {
+ "schema_version": "compact_manager_plan_compilation",
+ "status": "compiled",
+ "raw_task_count": len(raw_tasks),
+ "compiled_task_count": len(compiled_tasks),
+ "raw_plan_bytes": len(str(raw_plan).encode("utf-8")),
+ "object_anchor_routing_count": len(object_anchor_routing),
+ "object_anchor_routing_source": "manager_semantic_routing_normalized_by_compiler",
+ "tasks": audits,
+ "policy": "Manager selects and groups capabilities; local registry supplies stable execution contracts without changing scene-specific decisions.",
+ }
+ return compiled_plan, audit
+
diff --git a/vlm_agent/reconstruction/relations/agents/tool_runner.py b/vlm_agent/reconstruction/relations/agents/tool_runner.py
new file mode 100644
index 0000000..f164154
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/tool_runner.py
@@ -0,0 +1,85 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+def scan_render_script_capabilities(render_script: Path) -> dict[str, Any]:
+ """Describe the SceneMosaic render adapter without executing it."""
+ text = render_script.read_text(encoding="utf-8") if render_script.exists() else ""
+ arguments = {}
+ for name in [
+ "layout_dir",
+ "unposed_dir",
+ "object_ids",
+ "anchor_id",
+ "camera_direction",
+ "radius",
+ "output",
+ "image_size",
+ "add_grid",
+ "grid_mapping",
+ "grid_size",
+ "draw_grid_lines",
+ "plane_side",
+ ]:
+ arguments[name] = f"--{name}" in text
+ return {
+ "render_script": str(render_script),
+ "exists": render_script.exists(),
+ "supported_arguments": arguments,
+ "capabilities": {
+ "select_partial_objects": arguments["object_ids"],
+ "topdown_render": arguments["camera_direction"],
+ "default_topdown_direction": [0.0, -1.0, 0.0] if "[0.0, -1.0, 0.0]" in text else None,
+ "reuse_grid_mapping": arguments["grid_mapping"],
+ "output_image": arguments["output"],
+ "output_grid_json_when_add_grid": 'with_suffix(".json")' in text or "with_suffix('.json')" in text,
+ "preserve_coordinate_grid": arguments["add_grid"] and arguments["draw_grid_lines"],
+ "include_walls_flag": False,
+ "include_floor_flag": False,
+ },
+ "notes": [
+ "Production relation runs consume canonical top-down and grid mapping from the candidate-local relation layout.",
+ "The multi-agent runner does not regenerate top-down evidence.",
+ ],
+ }
+
+
+def _mapping_is_valid(path: Path) -> bool:
+ if not path.exists():
+ return False
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except Exception:
+ return False
+ return isinstance(payload, dict) and (
+ isinstance(payload.get("grid_points"), list)
+ or isinstance(payload.get("projected_grid_samples"), list)
+ or isinstance(payload.get("xz_to_pixel_affine"), list)
+ )
+
+
+def ensure_canonical_topdown_mapping(
+ scene_state: dict[str, Any],
+ dry_run: bool = False,
+) -> dict[str, Any]:
+ """Validate externally precomputed top-down evidence.
+
+ Regeneration belongs to build_relation_layout_from_sam3d.py.
+ """
+ mapping = scene_state.get("topdown_grid_mapping", {}) if isinstance(scene_state.get("topdown_grid_mapping"), dict) else {}
+ mapping_path = Path(str(mapping.get("canonical_mapping_path"))) if mapping.get("canonical_mapping_path") else None
+ image_path = Path(str(mapping.get("canonical_topdown_image_path"))) if mapping.get("canonical_topdown_image_path") else None
+ available = bool(mapping_path and _mapping_is_valid(mapping_path) and image_path and image_path.exists())
+ return {
+ "status": "available" if available else "missing_required_relation_precompute",
+ "generated": False,
+ "canonical_mapping_path": str(mapping_path) if mapping_path else None,
+ "canonical_topdown_image_path": str(image_path) if image_path else None,
+ "source": mapping.get("source"),
+ "fallback_used": False,
+ "dry_run": dry_run,
+ "reason": None if available else "Run build_relation_layout_from_sam3d.py before multi-agent relation extraction.",
+ }
diff --git a/vlm_agent/reconstruction/relations/agents/wall_attach_prefilter.py b/vlm_agent/reconstruction/relations/agents/wall_attach_prefilter.py
new file mode 100644
index 0000000..f69aed8
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/agents/wall_attach_prefilter.py
@@ -0,0 +1,968 @@
+from __future__ import annotations
+
+import hashlib
+import json
+from pathlib import Path
+from typing import Any
+
+
+WALL_IDS = ("wall_001", "wall_002", "wall_003", "wall_004")
+WALL_SEMANTIC_LIST_KEYS = (
+ "walls",
+ "accepted_walls",
+ "wall_visibility_review",
+ "wall_visibility_semantics",
+)
+BLOCKED_VISIBILITY = {"hidden", "not_visible", "not visible", "inferred", "occluded"}
+WALL_MOUNTED_ROLES = {"wall_mounted", "wall_embedded", "mounted", "embedded", "hung"}
+POINT_GEOMETRY_WALL_MOUNT_ROLES = {"wall_mounted", "mounted", "hung"}
+CONFIDENCE_RANK = {"high": 3, "medium": 2, "low": 1}
+SEMANTIC_RECONSTRUCTION_DISTANCE_MULTIPLIER = 1.5
+SELECTED_PANEL_BINDING_DISTANCE_MULTIPLIER = 2.0
+SEMANTIC_CONFLICT_MIN_EDGE_LENGTH_RATIO = 0.50
+SEMANTIC_CONFLICT_EDGE_RATIO_MARGIN = 0.20
+SEMANTIC_CONFLICT_EDGE_LENGTH_MULTIPLIER = 1.50
+SEMANTIC_CONFLICT_CENTER_DISTANCE_MULTIPLIER = 0.75
+FLOOR_COVERING_TERMS = {
+ "area rug",
+ "area_rug",
+ "carpet",
+ "floor covering",
+ "floor_covering",
+ "floor mat",
+ "floor_mat",
+ "rug",
+}
+
+
+def _pair(value: Any) -> tuple[float, float] | None:
+ if not isinstance(value, (list, tuple)) or len(value) != 2:
+ return None
+ try:
+ return float(value[0]), float(value[1])
+ except (TypeError, ValueError):
+ return None
+
+
+def wall_geometry_key(wall: dict[str, Any]) -> str | None:
+ """Return an orientation-independent identity for one formal wall segment."""
+ first = _pair(wall.get("first_point") or wall.get("first_point_xz"))
+ second = _pair(wall.get("second_point") or wall.get("second_point_xz"))
+ if first is None or second is None:
+ return None
+ endpoints = sorted((first, second))
+ payload = "|".join(f"{x:.6f},{z:.6f}" for x, z in endpoints)
+ return f"wall_edge_{hashlib.sha1(payload.encode('ascii')).hexdigest()[:12]}"
+
+
+def formal_wall_references(wall_evidence: dict[str, Any]) -> list[dict[str, Any]]:
+ by_id = {
+ str(wall.get("id")): wall
+ for wall in wall_evidence.get("candidate_walls", [])
+ if isinstance(wall, dict) and wall.get("id") in WALL_IDS
+ }
+ references = []
+ for wall_id in WALL_IDS:
+ wall = by_id.get(wall_id)
+ if not wall:
+ continue
+ references.append({
+ "wall_id": wall_id,
+ "geometry_key": wall_geometry_key(wall),
+ "first_point": wall.get("first_point"),
+ "second_point": wall.get("second_point"),
+ "coordinate_source": "obb_from_mesh_footprint_min_rect",
+ })
+ return references
+
+
+def _walk(value: Any):
+ if isinstance(value, dict):
+ yield value
+ for child in value.values():
+ yield from _walk(child)
+ elif isinstance(value, list):
+ for child in value:
+ yield from _walk(child)
+
+
+def collect_wall_semantics_by_geometry(
+ provider_results: list[dict[str, Any]],
+ references: list[dict[str, Any]],
+) -> tuple[dict[str, dict[str, Any]], dict[str, Any]]:
+ """Bind Agent visibility semantics to immutable local wall geometry."""
+ reference_by_key = {item.get("geometry_key"): item for item in references if item.get("geometry_key")}
+ reference_by_id = {item.get("wall_id"): item for item in references if item.get("wall_id")}
+ semantics: dict[str, dict[str, Any]] = {}
+ unmatched = []
+ for root in provider_results:
+ if not isinstance(root, dict):
+ continue
+ for node in _walk(root):
+ if not isinstance(node, dict):
+ continue
+ for list_key in WALL_SEMANTIC_LIST_KEYS:
+ items = node.get(list_key)
+ if not isinstance(items, list):
+ continue
+ for item in items:
+ if not isinstance(item, dict):
+ continue
+ explicit_key = item.get("geometry_key")
+ reported_key = explicit_key or wall_geometry_key(item)
+ reported_id = item.get("wall_id") or item.get("id")
+ reference = None
+ mismatch_reason = None
+ if reported_key:
+ reference = reference_by_key.get(reported_key)
+ id_reference = reference_by_id.get(reported_id)
+ if reference is None:
+ mismatch_reason = "geometry_key_not_in_formal_wall_references"
+ elif id_reference and id_reference.get("geometry_key") != reported_key:
+ mismatch_reason = "wall_id_geometry_key_mismatch"
+ reference = None
+ else:
+ reference = reference_by_id.get(reported_id)
+ if not reference:
+ unmatched.append({
+ "reported_wall_id": reported_id,
+ "geometry_key": reported_key,
+ "reason": mismatch_reason or "wall_reference_not_found",
+ })
+ continue
+ wall_id = str(reference["wall_id"])
+ record = semantics.setdefault(wall_id, {
+ "wall_id": wall_id,
+ "geometry_key": reference.get("geometry_key"),
+ "semantic_sources": [],
+ })
+ if list_key not in record["semantic_sources"]:
+ record["semantic_sources"].append(list_key)
+ for key in [
+ "original_image_visibility",
+ "visible_role",
+ "weak_wall",
+ "closure_only",
+ "can_anchor_wall_attach",
+ "evidence_strength",
+ "completion_method",
+ "completion_reason",
+ "reason",
+ ]:
+ if item.get(key) is not None:
+ record[key] = item.get(key)
+ return semantics, {
+ "binding_policy": (
+ "strict_geometry_key_when_present_else_canonical_wall_id; "
+ "conflicting wall_id/geometry_key records are rejected"
+ ),
+ "matched_wall_ids": sorted(semantics),
+ "missing_wall_ids": [wall_id for wall_id in WALL_IDS if wall_id not in semantics],
+ "unmatched_agent_records": unmatched,
+ }
+
+
+def _wall_is_blocked(semantic: dict[str, Any] | None) -> tuple[bool, str | None]:
+ if not semantic:
+ return False, None
+ visibility = str(semantic.get("original_image_visibility") or "").strip().lower()
+ completion_method = str(semantic.get("completion_method") or "").strip().lower()
+ blocked = bool(
+ semantic.get("weak_wall")
+ or semantic.get("closure_only")
+ or semantic.get("can_anchor_wall_attach") is False
+ or visibility in BLOCKED_VISIBILITY
+ or completion_method == "inferred_by_room_closure"
+ )
+ if not blocked:
+ return False, None
+ return True, "agent_visibility_semantics_block_wall_attach_anchor"
+
+
+def _objects_by_id(scene_state: dict[str, Any]) -> dict[str, dict[str, Any]]:
+ result = {}
+ candidates = scene_state.get("object_candidates", {})
+ items = candidates.get("items", []) if isinstance(candidates, dict) else []
+ for item in items:
+ if not isinstance(item, dict):
+ continue
+ object_id = item.get("object_id") or item.get("id")
+ if object_id:
+ result[str(object_id)] = {
+ "object_id": str(object_id),
+ "category": item.get("category"),
+ "description": item.get("description"),
+ }
+ return result
+
+
+def _hypotheses_by_object(wall_object_hypotheses: dict[str, Any] | None) -> dict[str, dict[str, Any]]:
+ result: dict[str, dict[str, Any]] = {}
+ items = wall_object_hypotheses.get("likely_wall_objects", []) if isinstance(wall_object_hypotheses, dict) else []
+ for item in items:
+ if not isinstance(item, dict) or not item.get("object_id"):
+ continue
+ object_id = str(item["object_id"])
+ current = result.get(object_id)
+ score = CONFIDENCE_RANK.get(str(item.get("confidence") or "").lower(), 0)
+ current_score = CONFIDENCE_RANK.get(str(current.get("confidence") or "").lower(), 0) if current else -1
+ if current is None or score > current_score:
+ result[object_id] = dict(item)
+ return result
+
+
+def _wall_line_hypotheses_by_object(
+ wall_object_hypotheses: dict[str, Any] | None,
+) -> dict[str, list[dict[str, Any]]]:
+ result: dict[str, list[dict[str, Any]]] = {}
+ items = wall_object_hypotheses.get("wall_line_hypotheses", []) if isinstance(wall_object_hypotheses, dict) else []
+ for item in items:
+ if not isinstance(item, dict) or item.get("candidate_wall_id") not in WALL_IDS:
+ continue
+ object_ids = item.get("supporting_objects", [])
+ if item.get("object_id"):
+ object_ids = [*object_ids, item["object_id"]] if isinstance(object_ids, list) else [item["object_id"]]
+ if not isinstance(object_ids, list):
+ continue
+ for object_id in object_ids:
+ if object_id:
+ result.setdefault(str(object_id), []).append(dict(item))
+ return result
+
+
+def _has_semantic_near_wall_geometry(metric: dict[str, Any]) -> bool:
+ # Floor-supported semantic matches need an edge, not isolated point contact.
+ if metric.get("possible_against_wall_contact") is not True:
+ return False
+ if metric.get("is_floor_supported") is not True:
+ return True
+ analysis = metric.get("flat_contact_edge_analysis") if isinstance(metric.get("flat_contact_edge_analysis"), dict) else {}
+ if analysis.get("has_valid_flat_contact_edge") is True:
+ return True
+ best_edge = analysis.get("best_edge") if isinstance(analysis.get("best_edge"), dict) else {}
+ return (
+ best_edge.get("flat_segment_candidate") is True
+ and best_edge.get("parallel_to_wall") is True
+ and best_edge.get("whole_edge_near_wall") is True
+ )
+
+
+def _is_semantic_reconstruction_uncertain_edge(metric: dict[str, Any]) -> bool:
+ if metric.get("is_floor_supported") is not True or metric.get("possible_against_wall_contact") is not True:
+ return False
+ analysis = metric.get("flat_contact_edge_analysis") if isinstance(metric.get("flat_contact_edge_analysis"), dict) else {}
+ best_edge = analysis.get("best_edge") if isinstance(analysis.get("best_edge"), dict) else {}
+ if analysis.get("analysis_available") is not True or best_edge.get("parallel_to_wall") is not True:
+ return False
+ try:
+ distance_threshold = float(analysis.get("distance_threshold"))
+ maximum_distance = float(best_edge.get("maximum_distance_to_wall"))
+ except (TypeError, ValueError):
+ return False
+ return (
+ distance_threshold > 0.0
+ and maximum_distance > distance_threshold
+ and maximum_distance <= distance_threshold * SEMANTIC_RECONSTRUCTION_DISTANCE_MULTIPLIER
+ )
+
+
+def _is_selected_panel_semantic_near_edge(metric: dict[str, Any]) -> bool:
+ """Allow a review-only pair when image semantics bind a nearby parallel mesh edge."""
+ if metric.get("is_floor_supported") is not True:
+ return False
+ analysis = metric.get("flat_contact_edge_analysis") if isinstance(metric.get("flat_contact_edge_analysis"), dict) else {}
+ best_edge = analysis.get("best_edge") if isinstance(analysis.get("best_edge"), dict) else {}
+ if analysis.get("analysis_available") is not True or best_edge.get("parallel_to_wall") is not True:
+ return False
+ try:
+ distance_threshold = float(analysis.get("distance_threshold"))
+ maximum_distance = float(best_edge.get("maximum_distance_to_wall"))
+ except (TypeError, ValueError):
+ return False
+ return bool(
+ distance_threshold > 0.0
+ and maximum_distance <= distance_threshold * SELECTED_PANEL_BINDING_DISTANCE_MULTIPLIER
+ )
+
+
+def _is_floor_covering(obj: dict[str, Any]) -> bool:
+ text = " ".join(str(obj.get(key) or "").strip().lower() for key in ("category", "description"))
+ return any(term in text for term in FLOOR_COVERING_TERMS)
+
+
+def _float_or_none(value: Any) -> float | None:
+ try:
+ return float(value)
+ except (TypeError, ValueError):
+ return None
+
+
+def _flat_edge_signature(metric: dict[str, Any]) -> dict[str, Any]:
+ analysis = metric.get("flat_contact_edge_analysis") if isinstance(metric.get("flat_contact_edge_analysis"), dict) else {}
+ best_edge = analysis.get("best_edge") if isinstance(analysis.get("best_edge"), dict) else {}
+ return {
+ "wall_id": str(metric.get("wall_id") or ""),
+ "valid": analysis.get("has_valid_flat_contact_edge") is True,
+ "edge_length": _float_or_none(best_edge.get("length")),
+ "edge_length_ratio": _float_or_none(best_edge.get("length_ratio")),
+ "maximum_distance_to_wall": _float_or_none(best_edge.get("maximum_distance_to_wall")),
+ "distance_threshold": _float_or_none(analysis.get("distance_threshold")),
+ "footprint_min_distance_to_wall": _float_or_none(metric.get("footprint_min_distance_to_wall")),
+ "center_to_wall_distance": _float_or_none(metric.get("center_to_wall_distance")),
+ "possible_against_wall_contact": metric.get("possible_against_wall_contact") is True,
+ "is_floor_supported": metric.get("is_floor_supported") is True,
+ }
+
+
+def _walls_are_adjacent(first: str, second: str) -> bool:
+ if first not in WALL_IDS or second not in WALL_IDS:
+ return False
+ delta = abs(WALL_IDS.index(first) - WALL_IDS.index(second))
+ return delta in {1, len(WALL_IDS) - 1}
+
+
+def _semantic_visibility_geometry_conflicts(
+ object_id: str,
+ role: str,
+ confidence: str,
+ metrics: list[dict[str, Any]],
+ blocked_walls: dict[str, str],
+) -> dict[str, dict[str, Any]]:
+ """Reopen only strong wall-id conflicts for downstream Agent comparison.
+
+ This does not change wall visibility or accept a relation locally. It keeps
+ one blocked candidate in the existing recall call when object geometry
+ strongly contradicts an adjacent visible-wall binding.
+ """
+ if CONFIDENCE_RANK.get(confidence, 0) < CONFIDENCE_RANK["high"]:
+ return {}
+ if role not in {"wall_mounted", "wall_embedded", "against_wall", "wall_aligned", "mounted", "embedded", "hung"}:
+ return {}
+
+ metric_by_wall = {
+ str(metric.get("wall_id")): metric
+ for metric in metrics
+ if isinstance(metric, dict) and metric.get("wall_id") in WALL_IDS
+ }
+ signatures = {
+ wall_id: _flat_edge_signature(metric)
+ for wall_id, metric in metric_by_wall.items()
+ }
+ nearest_wall_id = min(
+ signatures,
+ key=lambda wall_id: (
+ signatures[wall_id].get("footprint_min_distance_to_wall")
+ if signatures[wall_id].get("footprint_min_distance_to_wall") is not None
+ else float("inf"),
+ wall_id,
+ ),
+ default=None,
+ )
+ unblocked_valid = [
+ signature
+ for wall_id, signature in signatures.items()
+ if wall_id not in blocked_walls and signature.get("valid")
+ ]
+ unblocked_valid.sort(key=lambda signature: (
+ -(signature.get("edge_length_ratio") or 0.0),
+ signature.get("center_to_wall_distance")
+ if signature.get("center_to_wall_distance") is not None
+ else float("inf"),
+ signature.get("wall_id") or "",
+ ))
+ best_unblocked = unblocked_valid[0] if unblocked_valid else None
+
+ conflicts: list[dict[str, Any]] = []
+ for wall_id in sorted(blocked_walls):
+ signature = signatures.get(wall_id)
+ if not signature or signature.get("possible_against_wall_contact") is not True:
+ continue
+
+ if role in WALL_MOUNTED_ROLES:
+ if wall_id != nearest_wall_id:
+ continue
+ conflicts.append({
+ "object_id": object_id,
+ "wall_id": wall_id,
+ "reason": "high_confidence_wall_mount_nearest_geometry_conflicts_with_blocked_visibility",
+ "prior_block_reason": blocked_walls[wall_id],
+ "geometry": signature,
+ "competing_unblocked_wall": best_unblocked,
+ "local_action": "reopen_for_existing_wall_attach_agent_review_only",
+ })
+ continue
+
+ if signature.get("valid") is not True:
+ continue
+ edge_length = signature.get("edge_length") or 0.0
+ edge_ratio = signature.get("edge_length_ratio") or 0.0
+ max_distance = signature.get("maximum_distance_to_wall")
+ threshold = signature.get("distance_threshold")
+ within_edge_threshold = bool(
+ max_distance is not None
+ and threshold is not None
+ and threshold > 0.0
+ and max_distance <= threshold * SEMANTIC_RECONSTRUCTION_DISTANCE_MULTIPLIER
+ )
+ if not within_edge_threshold:
+ continue
+
+ if best_unblocked is None:
+ strong_conflict = edge_ratio >= SEMANTIC_CONFLICT_MIN_EDGE_LENGTH_RATIO
+ else:
+ competing_wall_id = str(best_unblocked.get("wall_id") or "")
+ if not _walls_are_adjacent(wall_id, competing_wall_id):
+ continue
+ competing_length = best_unblocked.get("edge_length") or 0.0
+ competing_ratio = best_unblocked.get("edge_length_ratio") or 0.0
+ center_distance = signature.get("center_to_wall_distance")
+ competing_center_distance = best_unblocked.get("center_to_wall_distance")
+ footprint_distance = signature.get("footprint_min_distance_to_wall")
+ competing_footprint_distance = best_unblocked.get("footprint_min_distance_to_wall")
+ stronger_edge = bool(
+ edge_ratio >= max(
+ SEMANTIC_CONFLICT_MIN_EDGE_LENGTH_RATIO,
+ competing_ratio + SEMANTIC_CONFLICT_EDGE_RATIO_MARGIN,
+ )
+ and edge_length >= competing_length * SEMANTIC_CONFLICT_EDGE_LENGTH_MULTIPLIER
+ )
+ closer_center = bool(
+ center_distance is not None
+ and competing_center_distance is not None
+ and center_distance <= competing_center_distance * SEMANTIC_CONFLICT_CENTER_DISTANCE_MULTIPLIER
+ )
+ no_farther_footprint = bool(
+ footprint_distance is not None
+ and competing_footprint_distance is not None
+ and footprint_distance <= competing_footprint_distance
+ )
+ strong_conflict = stronger_edge and closer_center and no_farther_footprint
+ if not strong_conflict:
+ continue
+ conflicts.append({
+ "object_id": object_id,
+ "wall_id": wall_id,
+ "reason": "dominant_adjacent_flat_edge_conflicts_with_blocked_visibility",
+ "prior_block_reason": blocked_walls[wall_id],
+ "geometry": signature,
+ "competing_unblocked_wall": best_unblocked,
+ "local_action": "reopen_for_existing_wall_attach_agent_review_only",
+ })
+
+ # One conflict option exposes a likely adjacent-wall swap without restoring
+ # an exhaustive object-wall scan.
+ conflicts.sort(key=lambda item: (
+ -(item.get("geometry", {}).get("edge_length_ratio") or 0.0),
+ item.get("geometry", {}).get("footprint_min_distance_to_wall")
+ if item.get("geometry", {}).get("footprint_min_distance_to_wall") is not None
+ else float("inf"),
+ str(item.get("wall_id") or ""),
+ ))
+ return {
+ str(item["wall_id"]): item
+ for item in conflicts[:1]
+ if item.get("wall_id") in WALL_IDS
+ }
+
+
+def _compact_metric(metric: dict[str, Any]) -> dict[str, Any]:
+ analysis = metric.get("flat_contact_edge_analysis") if isinstance(metric.get("flat_contact_edge_analysis"), dict) else {}
+ return {
+ "object_id": metric.get("object_id"),
+ "wall_id": metric.get("wall_id"),
+ "footprint_min_distance_to_wall": metric.get("footprint_min_distance_to_wall"),
+ "center_to_wall_distance": metric.get("center_to_wall_distance"),
+ "possible_against_wall_contact": metric.get("possible_against_wall_contact"),
+ "is_floor_supported": metric.get("is_floor_supported"),
+ "is_thin_planar_candidate": metric.get("is_thin_planar_candidate"),
+ "geometry_classification": metric.get("geometry_classification"),
+ "flat_contact_edge_analysis": {
+ "analysis_available": analysis.get("analysis_available"),
+ "has_valid_flat_contact_edge": analysis.get("has_valid_flat_contact_edge"),
+ "distance_threshold": analysis.get("distance_threshold"),
+ "semantic_reconstruction_distance_multiplier": SEMANTIC_RECONSTRUCTION_DISTANCE_MULTIPLIER,
+ "reason": analysis.get("reason"),
+ "best_edge": analysis.get("best_edge"),
+ },
+ }
+
+
+def build_wall_attach_prefilter(
+ scene_state: dict[str, Any],
+ wall_evidence: dict[str, Any],
+ semantic_provider_results: list[dict[str, Any]],
+ wall_object_hypotheses: dict[str, Any] | None,
+ object_anchor_routing: list[dict[str, Any]] | None = None,
+) -> dict[str, Any]:
+ references = formal_wall_references(wall_evidence)
+ semantics, semantic_binding = collect_wall_semantics_by_geometry(semantic_provider_results, references)
+ blocked_walls = {}
+ for reference in references:
+ wall_id = str(reference["wall_id"])
+ blocked, reason = _wall_is_blocked(semantics.get(wall_id))
+ if blocked:
+ blocked_walls[wall_id] = reason
+
+ objects = _objects_by_id(scene_state)
+ hypotheses = _hypotheses_by_object(wall_object_hypotheses)
+ wall_line_hypotheses = _wall_line_hypotheses_by_object(wall_object_hypotheses)
+ routing_by_object = {
+ str(item["object_id"]): {
+ "anchor_mode": str(item.get("anchor_mode") or "uncertain").lower(),
+ "confidence": str(item.get("confidence") or "low").lower(),
+ }
+ for item in object_anchor_routing or []
+ if isinstance(item, dict) and item.get("object_id")
+ }
+ metrics_by_object: dict[str, list[dict[str, Any]]] = {}
+ raw_pair_count = 0
+ for metric in wall_evidence.get("wall_attach_geometry", []):
+ if not isinstance(metric, dict) or metric.get("wall_id") not in WALL_IDS or not metric.get("object_id"):
+ continue
+ raw_pair_count += 1
+ metrics_by_object.setdefault(str(metric["object_id"]), []).append(metric)
+
+ candidates = []
+ object_evaluations = []
+ supplemental_visual_review_object_ids = []
+ semantic_visibility_geometry_conflicts = []
+ for object_id in sorted(objects):
+ if _is_floor_covering(objects[object_id]):
+ object_evaluations.append({
+ "object_id": object_id,
+ "decision": "excluded_floor_covering",
+ "hypothesis_role": None,
+ "hypothesis_confidence": None,
+ "rejected_reasons": [{"reason": "floor_covering_cannot_anchor_or_target_wall_attach"}],
+ })
+ continue
+ hypothesis = hypotheses.get(object_id, {})
+ role = str(hypothesis.get("role") or "").lower()
+ confidence = str(hypothesis.get("confidence") or "").lower()
+ explicit_visual_mount = role in WALL_MOUNTED_ROLES
+ high_confidence_visual_mount = explicit_visual_mount and CONFIDENCE_RANK.get(confidence, 0) >= CONFIDENCE_RANK["high"]
+ high_confidence_hanging_mount = (
+ role in POINT_GEOMETRY_WALL_MOUNT_ROLES
+ and CONFIDENCE_RANK.get(confidence, 0) >= CONFIDENCE_RANK["high"]
+ )
+ routing = routing_by_object.get(object_id, {})
+ routing_mode = str(routing.get("anchor_mode") or "uncertain")
+ routing_confidence = str(routing.get("confidence") or "low")
+ semantic_wall_alignment = (
+ role in {"wall_mounted", "wall_embedded", "against_wall", "wall_aligned"}
+ and CONFIDENCE_RANK.get(confidence, 0) >= CONFIDENCE_RANK["medium"]
+ )
+ hard_ceiling_exclusion = routing_confidence == "high" and routing_mode == "ceiling_only"
+ soft_non_wall_exclusion = (
+ routing_confidence == "high"
+ and routing_mode in {"support_only", "contain_candidate"}
+ and not semantic_wall_alignment
+ )
+ if hard_ceiling_exclusion or soft_non_wall_exclusion:
+ object_evaluations.append({
+ "object_id": object_id,
+ "decision": "excluded_by_high_confidence_anchor_routing",
+ "anchor_routing": routing,
+ "hypothesis_role": role or None,
+ "hypothesis_confidence": confidence or None,
+ "rejected_reasons": [{
+ "reason": (
+ "anchor_routing_ceiling_only_excludes_wall_review"
+ if hard_ceiling_exclusion
+ else f"anchor_routing_{routing_mode}_soft_exclusion_without_agent_wall_alignment"
+ )
+ }],
+ })
+ continue
+ object_wall_lines = wall_line_hypotheses.get(object_id, [])
+ semantic_line_wall_ids = {
+ str(item.get("candidate_wall_id"))
+ for item in object_wall_lines
+ if item.get("candidate_wall_id") in WALL_IDS
+ }
+ conflict_reopenings = _semantic_visibility_geometry_conflicts(
+ object_id,
+ role,
+ confidence,
+ metrics_by_object.get(object_id, []),
+ blocked_walls,
+ )
+ semantic_visibility_geometry_conflicts.extend(conflict_reopenings.values())
+ object_metrics = [
+ metric for metric in metrics_by_object.get(object_id, [])
+ if (
+ str(metric.get("wall_id")) not in blocked_walls
+ or str(metric.get("wall_id")) in conflict_reopenings
+ )
+ ]
+ object_metrics.sort(key=lambda metric: (
+ float(metric.get("footprint_min_distance_to_wall") or 1e9),
+ str(metric.get("wall_id")),
+ ))
+ nearest_final_wall_id = str(object_metrics[0].get("wall_id")) if object_metrics else None
+ nearest_neighbor_wall_ids = {
+ str(metric.get("wall_id")) for metric in object_metrics[:2]
+ }
+ semantic_alignment_object = (
+ role in {"wall_mounted", "wall_embedded", "against_wall", "wall_aligned"}
+ and CONFIDENCE_RANK.get(confidence, 0) >= CONFIDENCE_RANK["medium"]
+ )
+ eligible_metrics = []
+ rejected_reasons = []
+ for metric in metrics_by_object.get(object_id, []):
+ wall_id = str(metric.get("wall_id"))
+ semantic_geometry_conflict = conflict_reopenings.get(wall_id)
+ if wall_id in blocked_walls and semantic_geometry_conflict is None:
+ rejected_reasons.append({"wall_id": wall_id, "reason": blocked_walls[wall_id]})
+ continue
+ analysis = metric.get("flat_contact_edge_analysis") if isinstance(metric.get("flat_contact_edge_analysis"), dict) else {}
+ floor_supported = metric.get("is_floor_supported") is True
+ valid_flat_edge = analysis.get("analysis_available") is True and analysis.get("has_valid_flat_contact_edge") is True
+ possible_contact = metric.get("possible_against_wall_contact") is True
+ thin_planar = metric.get("is_thin_planar_candidate") is True
+ semantic_line_support = wall_id in semantic_line_wall_ids
+ selected_panel_line_support = any(
+ item.get("candidate_wall_id") == wall_id
+ and item.get("source") == "selected_candidate_agent_semantic_binding"
+ and CONFIDENCE_RANK.get(str(item.get("confidence") or "").lower(), 0) >= CONFIDENCE_RANK["medium"]
+ for item in object_wall_lines
+ )
+ selected_panel_high_confidence_support = any(
+ item.get("candidate_wall_id") == wall_id
+ and item.get("source") == "selected_candidate_agent_semantic_binding"
+ and CONFIDENCE_RANK.get(str(item.get("confidence") or "").lower(), 0) >= CONFIDENCE_RANK["high"]
+ for item in object_wall_lines
+ )
+ local_nearest_alignment_support = (
+ semantic_alignment_object
+ and wall_id == nearest_final_wall_id
+ )
+ semantic_reconstruction_uncertain = (
+ (semantic_line_support or local_nearest_alignment_support)
+ and _is_semantic_reconstruction_uncertain_edge(metric)
+ )
+ if floor_supported:
+ if semantic_geometry_conflict is not None and valid_flat_edge:
+ eligible = True
+ reason = "semantic_visibility_geometry_conflict_flat_edge_review"
+ elif selected_panel_line_support and _is_selected_panel_semantic_near_edge(metric):
+ eligible = True
+ reason = "selected_candidate_semantic_wall_binding_with_near_parallel_edge"
+ elif selected_panel_high_confidence_support and semantic_alignment_object:
+ # Exact original-image semantics may survive a missing SAM3D
+ # contact edge. This creates review input, never local acceptance.
+ eligible = True
+ reason = "selected_candidate_semantic_wall_binding_reconstruction_override"
+ elif semantic_reconstruction_uncertain:
+ eligible = True
+ reason = (
+ "agent_wall_line_parallel_edge_within_reconstruction_uncertainty"
+ if semantic_line_support
+ else "agent_alignment_nearest_wall_parallel_edge_within_reconstruction_uncertainty"
+ )
+ elif (
+ high_confidence_hanging_mount
+ and (semantic_line_support or wall_id in nearest_neighbor_wall_ids)
+ ):
+ # A genuinely wall-mounted/hung object can have unreliable SAM3D
+ # contact points on one real wall. Keep at most the two nearest
+ # non-blocked alternatives for visual comparison; this remains
+ # candidate recall only, never a local final acceptance.
+ eligible = True
+ if possible_contact:
+ reason = (
+ "high_confidence_wall_mount_with_semantic_line_and_near_point_geometry"
+ if semantic_line_support
+ else "high_confidence_wall_mount_with_nearest_wall_point_geometry"
+ )
+ else:
+ reason = (
+ "high_confidence_wall_mount_with_semantic_line_geometry_uncertain_second_option"
+ if semantic_line_support
+ else "high_confidence_wall_mount_with_nearest_wall_geometry_uncertain_second_option"
+ )
+ elif high_confidence_visual_mount and possible_contact:
+ eligible = _has_semantic_near_wall_geometry(metric)
+ reason = (
+ "high_confidence_wall_mounted_or_embedded_with_near_wall_edge"
+ if eligible else "high_confidence_wall_hypothesis_with_corner_only_geometry"
+ )
+ else:
+ eligible = valid_flat_edge
+ reason = "floor_supported_real_lower_mesh_flat_edge" if eligible else "floor_supported_without_valid_lower_mesh_flat_edge"
+ else:
+ eligible = possible_contact and (explicit_visual_mount or thin_planar)
+ reason = "visual_wall_mount_or_thin_planar_near_wall" if eligible else "non_floor_object_without_wall_mount_contact_evidence"
+ if eligible:
+ eligible_metrics.append((metric, reason))
+ else:
+ rejected_reasons.append({"wall_id": wall_id, "reason": reason})
+
+ if eligible_metrics:
+ semantic_wall_rank = {
+ str(item["candidate_wall_id"]): index
+ for index, item in enumerate(object_wall_lines)
+ if item.get("candidate_wall_id") in WALL_IDS
+ }
+ semantic_eligible_metrics = [
+ pair for pair in eligible_metrics
+ if str(pair[0].get("wall_id")) in semantic_wall_rank
+ and (
+ _has_semantic_near_wall_geometry(pair[0])
+ or _is_semantic_reconstruction_uncertain_edge(pair[0])
+ or (
+ any(
+ item.get("candidate_wall_id") == str(pair[0].get("wall_id"))
+ and item.get("source") == "selected_candidate_agent_semantic_binding"
+ for item in object_wall_lines
+ )
+ and _is_selected_panel_semantic_near_edge(pair[0])
+ )
+ or pair[1] == "selected_candidate_semantic_wall_binding_reconstruction_override"
+ )
+ ]
+ semantic_preference_applied = bool(semantic_eligible_metrics)
+ selection_pool = semantic_eligible_metrics or eligible_metrics
+ selection_pool.sort(key=lambda pair: (
+ semantic_wall_rank.get(str(pair[0].get("wall_id")), len(semantic_wall_rank)),
+ 0 if pair[0].get("possible_against_wall_contact") is True else 1,
+ float(pair[0].get("footprint_min_distance_to_wall") or 1e9),
+ str(pair[0].get("wall_id")),
+ ))
+ selected_pairs = selection_pool[:2]
+ for option_index, (metric, reason) in enumerate(selected_pairs, start=1):
+ wall_id = str(metric["wall_id"])
+ selected_wall_line = next(
+ (item for item in object_wall_lines if item.get("candidate_wall_id") == wall_id),
+ None,
+ )
+ candidate = {
+ "object_target": object_id,
+ "candidate_wall_id": wall_id,
+ "candidate_group_id": object_id,
+ "candidate_option_index": option_index,
+ "candidate_option_count": len(selected_pairs),
+ "wall_geometry_key": next((item.get("geometry_key") for item in references if item.get("wall_id") == wall_id), None),
+ "candidate_source": (
+ reason
+ if reason in {
+ "agent_wall_line_parallel_edge_within_reconstruction_uncertainty",
+ "agent_alignment_nearest_wall_parallel_edge_within_reconstruction_uncertainty",
+ "selected_candidate_semantic_wall_binding_reconstruction_override",
+ }
+ else "agent_wall_line_hypothesis_with_near_wall_edge"
+ if semantic_preference_applied else reason
+ ),
+ "geometry_eligibility_source": reason,
+ "reconstruction_uncertainty_used": reason in {
+ "agent_wall_line_parallel_edge_within_reconstruction_uncertainty",
+ "agent_alignment_nearest_wall_parallel_edge_within_reconstruction_uncertainty",
+ "selected_candidate_semantic_wall_binding_reconstruction_override",
+ },
+ "object": objects[object_id],
+ "object_anchor_routing": routing or None,
+ "wall_object_hypothesis": hypothesis or None,
+ "wall_line_hypothesis": selected_wall_line,
+ "semantic_visibility_geometry_conflict": conflict_reopenings.get(wall_id),
+ "semantic_wall_preference_applied": semantic_preference_applied,
+ "geometry_evidence": _compact_metric(metric),
+ "agent_action": "compare_wall_options_then_accept_at_most_one",
+ }
+ candidates.append(candidate)
+ decision = "candidate"
+ else:
+ semantic_review_line = next(
+ (
+ item for item in object_wall_lines
+ if item.get("candidate_wall_id") in WALL_IDS
+ and item.get("candidate_wall_id") not in blocked_walls
+ ),
+ None,
+ )
+ if high_confidence_visual_mount and semantic_review_line is not None:
+ wall_id = str(semantic_review_line["candidate_wall_id"])
+ metric = next(
+ (
+ item for item in metrics_by_object.get(object_id, [])
+ if str(item.get("wall_id")) == wall_id
+ ),
+ None,
+ )
+ candidate = {
+ "object_target": object_id,
+ "candidate_wall_id": wall_id,
+ "wall_geometry_key": next(
+ (item.get("geometry_key") for item in references if item.get("wall_id") == wall_id),
+ None,
+ ),
+ "candidate_source": "high_confidence_visual_mount_with_semantic_wall_line_review",
+ "geometry_eligibility_source": "visual_semantic_mount_without_reliable_reconstruction_contact",
+ "reconstruction_uncertainty_used": True,
+ "object": objects[object_id],
+ "wall_object_hypothesis": hypothesis or None,
+ "wall_line_hypothesis": semantic_review_line,
+ "semantic_wall_preference_applied": True,
+ "geometry_evidence": (
+ _compact_metric(metric)
+ if isinstance(metric, dict)
+ else {
+ "object_id": object_id,
+ "wall_id": wall_id,
+ "geometry_classification": "visual_review_required_without_metric",
+ }
+ ),
+ "agent_action": "judge_accept_or_reject",
+ }
+ candidates.append(candidate)
+ decision = "candidate"
+ else:
+ decision = "excluded_from_exhaustive_pair_scan"
+ if explicit_visual_mount and object_id not in supplemental_visual_review_object_ids:
+ supplemental_visual_review_object_ids.append(object_id)
+ object_evaluations.append({
+ "object_id": object_id,
+ "decision": decision,
+ "hypothesis_role": role or None,
+ "hypothesis_confidence": confidence or None,
+ "rejected_reasons": rejected_reasons,
+ })
+
+ candidate_metrics = [candidate["geometry_evidence"] for candidate in candidates]
+ return {
+ "schema_version": "wall_attach_prefilter",
+ "status": "available",
+ "capability_contract": {
+ "requires": ["formal_wall_geometry_ready", "wall_visibility_semantics_ready"],
+ "provides": ["wall_attach_candidates_ready"],
+ "policy": "Capability dependency is fixed; Agent task names and task grouping remain Manager-controlled.",
+ },
+ "policy": {
+ "object_anchor_routing": "High-confidence ceiling_only is a hard wall-review exclusion. support_only and contain_candidate are soft routing hints: a medium/high Agent wall-mounted, embedded, against-wall, or wall-aligned finding may reopen local geometry review. floor_wall enters both Support and Wall.",
+ "candidate_wall_options": "Keep up to two geometrically plausible neighboring walls for one object so the Agent, not nearest-edge ordering, selects the actual contact wall.",
+ "ordinary_floor_supported": "Require a finite wall-parallel flat contact edge from the real lower-mesh XZ hull near a non-blocked wall; merge collinear fragments and reject isolated point/corner contact.",
+ "semantic_wall_line_priority": "Prefer an exact selected-candidate Agent object-wall binding. Medium confidence still requires a nearby parallel reconstructed edge. A high-confidence binding may open one review pair when SAM3D loses that edge, but it never creates local acceptance. Generic nearest-wall/corner evidence cannot use this override.",
+ "semantic_reconstruction_uncertainty": "When SAM3D placement leaves a small gap, either an Agent wall-line hypothesis or a high/medium-confidence Agent wall-alignment object mapped locally to its nearest final wall may enter review up to 1.5 times the adaptive edge-distance threshold, but only with a real parallel hull edge. This creates a review candidate, not a final attach.",
+ "wall_mounted": "A high-confidence wall-mounted/embedded hypothesis with a non-blocked semantic wall line becomes an explicit Agent review pair even when SAM3D contact geometry is weak; local code does not auto-accept it.",
+ "floor_covering": "Exclude rugs, carpets, and floor coverings from wall-attach review.",
+ "weak_wall": "Visibility/weak semantics come from an Agent provider and are bound geometry-key-first. Missing semantics remain reviewable rather than locally blocked.",
+ "semantic_geometry_conflict": "A high-confidence wall-following object may reopen one blocked adjacent wall only when its real geometry strongly contradicts the prior wall-id visibility binding. Reopening adds a pair to the existing Agent review; it never changes wall semantics or accepts attach locally.",
+ "open_supplement": "The recall Agent may add an immediately obvious wall-mounted object, but must not exhaustively rescan every object-wall pair.",
+ },
+ "formal_wall_references": references,
+ "wall_semantics": semantics,
+ "object_anchor_routing": routing_by_object,
+ "semantic_binding_audit": semantic_binding,
+ "blocked_wall_ids": blocked_walls,
+ "raw_object_wall_pair_count": raw_pair_count,
+ "candidate_pair_count": len(candidates),
+ "candidate_object_count": len({item["object_target"] for item in candidates}),
+ "candidate_pairs": candidates,
+ "candidate_wall_attach_geometry": candidate_metrics,
+ "semantic_visibility_geometry_conflicts": semantic_visibility_geometry_conflicts,
+ "supplemental_visual_review_object_ids": supplemental_visual_review_object_ids,
+ "object_evaluations": object_evaluations,
+ }
+
+
+def compact_wall_evidence_for_recall(wall_evidence: dict[str, Any], prefilter: dict[str, Any]) -> dict[str, Any]:
+ return {
+ "scene_name": wall_evidence.get("scene_name"),
+ "coordinate_source": "obb_from_mesh_footprint_min_rect",
+ "canonical_topdown": wall_evidence.get("canonical_topdown", {}),
+ "formal_wall_references": prefilter.get("formal_wall_references", []),
+ "wall_semantics": prefilter.get("wall_semantics", {}),
+ "blocked_wall_ids": prefilter.get("blocked_wall_ids", {}),
+ "wall_attach_geometry": prefilter.get("candidate_wall_attach_geometry", []),
+ "semantic_visibility_geometry_conflicts": prefilter.get(
+ "semantic_visibility_geometry_conflicts", []
+ ),
+ "input_reduction": {
+ "raw_object_wall_pair_count": prefilter.get("raw_object_wall_pair_count", 0),
+ "candidate_pair_count": prefilter.get("candidate_pair_count", len(prefilter.get("candidate_pairs", []))),
+ },
+ }
+
+
+def render_wall_attach_prefilter_overlay(
+ canonical_topdown: str | Path | None,
+ affine_path: str | Path | None,
+ output_path: str | Path,
+ wall_evidence: dict[str, Any],
+ prefilter: dict[str, Any],
+) -> dict[str, Any]:
+ output = Path(output_path)
+ if not canonical_topdown or not Path(canonical_topdown).exists():
+ return {"status": "unavailable", "reason": "canonical_topdown_missing", "output_path": str(output)}
+ if not affine_path or not Path(affine_path).exists():
+ return {"status": "unavailable", "reason": "render_grid_affine_missing", "output_path": str(output)}
+ try:
+ import cv2
+ except ImportError:
+ return {"status": "unavailable", "reason": "opencv_missing", "output_path": str(output)}
+ image = cv2.imread(str(canonical_topdown))
+ if image is None:
+ return {"status": "unavailable", "reason": "canonical_topdown_unreadable", "output_path": str(output)}
+ affine_data = json.loads(Path(affine_path).read_text(encoding="utf-8"))
+ affine = affine_data.get("xz_to_pixel_affine")
+ if not (isinstance(affine, list) and len(affine) == 3):
+ return {"status": "unavailable", "reason": "xz_to_pixel_affine_missing", "output_path": str(output)}
+
+ def project(point: Any) -> tuple[int, int] | None:
+ pair = _pair(point)
+ if pair is None:
+ return None
+ x, z = pair
+ px = x * float(affine[0][0]) + z * float(affine[1][0]) + float(affine[2][0])
+ py = x * float(affine[0][1]) + z * float(affine[1][1]) + float(affine[2][1])
+ return int(round(px)), int(round(py))
+
+ blocked = set(prefilter.get("blocked_wall_ids", {}))
+ semantic_by_wall = prefilter.get("wall_semantics", {})
+ wall_midpoints = {}
+ for wall in wall_evidence.get("candidate_walls", []):
+ if not isinstance(wall, dict) or wall.get("id") not in WALL_IDS:
+ continue
+ wall_id = str(wall["id"])
+ first, second = project(wall.get("first_point")), project(wall.get("second_point"))
+ if first is None or second is None:
+ continue
+ color = (120, 120, 120) if wall_id in blocked else (30, 180, 30)
+ cv2.arrowedLine(image, first, second, color, 4, cv2.LINE_AA, tipLength=0.035)
+ midpoint = ((first[0] + second[0]) // 2, (first[1] + second[1]) // 2)
+ wall_midpoints[wall_id] = midpoint
+ visibility = str(semantic_by_wall.get(wall_id, {}).get("original_image_visibility") or "unknown")
+ label = f"{wall_id} {'BLOCKED' if wall_id in blocked else visibility}"
+ cv2.putText(image, label, midpoint, cv2.FONT_HERSHEY_SIMPLEX, 0.55, color, 2, cv2.LINE_AA)
+
+ footprint_by_id = {
+ str(item.get("object_id")): item
+ for item in wall_evidence.get("object_footprints", [])
+ if isinstance(item, dict) and item.get("object_id")
+ }
+ for index, candidate in enumerate(prefilter.get("candidate_pairs", [])):
+ object_id = str(candidate.get("object_target"))
+ wall_id = str(candidate.get("candidate_wall_id"))
+ footprint = footprint_by_id.get(object_id, {})
+ center = project(footprint.get("center_xz"))
+ if center is None:
+ metric = candidate.get("geometry_evidence", {})
+ bbox_min, bbox_max = _pair(metric.get("bbox_xz_min")), _pair(metric.get("bbox_xz_max"))
+ if bbox_min and bbox_max:
+ center = project([(bbox_min[0] + bbox_max[0]) * 0.5, (bbox_min[1] + bbox_max[1]) * 0.5])
+ if center is None:
+ continue
+ color = (0, 80 + (index * 53) % 160, 255)
+ cv2.circle(image, center, 9, color, -1, cv2.LINE_AA)
+ cv2.putText(image, object_id, (center[0] + 8, center[1] - 8), cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2, cv2.LINE_AA)
+ if wall_id in wall_midpoints:
+ cv2.line(image, center, wall_midpoints[wall_id], color, 1, cv2.LINE_AA)
+
+ output.parent.mkdir(parents=True, exist_ok=True)
+ if not cv2.imwrite(str(output), image):
+ return {"status": "unavailable", "reason": "overlay_write_failed", "output_path": str(output)}
+ return {
+ "status": "available",
+ "output_path": str(output),
+ "candidate_pair_count": prefilter.get("candidate_pair_count", 0),
+ "blocked_wall_ids": sorted(blocked),
+ }
diff --git a/vlm_agent/reconstruction/relations/graph.py b/vlm_agent/reconstruction/relations/graph.py
new file mode 100644
index 0000000..d10fd40
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/graph.py
@@ -0,0 +1,242 @@
+from __future__ import annotations
+
+import copy
+from dataclasses import dataclass
+from types import MappingProxyType
+from typing import Any, Iterable, Mapping
+
+
+class RelationGraphError(ValueError):
+ """Raised when published relations do not have deterministic ownership."""
+
+
+def _is_wall(anchor_id: str) -> bool:
+ return anchor_id.startswith("wall_")
+
+
+@dataclass(frozen=True)
+class RelationGraphView:
+
+ full_relations: tuple[dict[str, Any], ...]
+ primary_relation_by_target: Mapping[str, dict[str, Any]]
+ primary_parent_by_target: Mapping[str, str]
+ primary_children_by_anchor: Mapping[str, tuple[str, ...]]
+ secondary_wall_constraints_by_target: Mapping[
+ str, tuple[dict[str, Any], ...]
+ ]
+
+ @classmethod
+ def from_relations(
+ cls, relations: Iterable[dict[str, Any]]
+ ) -> "RelationGraphView":
+ original: list[dict[str, Any]] = []
+ by_target: dict[str, list[dict[str, Any]]] = {}
+ seen: set[tuple[str, str, str]] = set()
+
+ for index, raw_relation in enumerate(relations):
+ if not isinstance(raw_relation, dict):
+ raise RelationGraphError(f"relation {index} must be an object")
+ relation = str(raw_relation.get("relation") or "")
+ if relation not in {"attach", "contain"}:
+ raise RelationGraphError(
+ f"relation {index} has unsupported type: {relation or 'missing'}"
+ )
+ parameters = raw_relation.get("parameters")
+ if not isinstance(parameters, dict):
+ raise RelationGraphError(
+ f"relation {index}.parameters must be an object"
+ )
+ anchor = str(parameters.get("object_anchor") or "").strip()
+ target = str(parameters.get("object_target") or "").strip()
+ if not anchor or not target:
+ raise RelationGraphError(
+ f"relation {index} must define object_anchor and object_target"
+ )
+ if anchor == target:
+ raise RelationGraphError(
+ f"relation target cannot own itself: {target}"
+ )
+ if _is_wall(anchor) and relation != "attach":
+ raise RelationGraphError(
+ f"wall constraint must use attach: {anchor} -> {target}"
+ )
+ triple = (relation, anchor, target)
+ if triple in seen:
+ raise RelationGraphError(f"duplicate relation: {triple}")
+ seen.add(triple)
+
+ # Keep the publication payload unchanged at the JSON-value level.
+ # A normalized copy is used only for deterministic ownership.
+ original_item = copy.deepcopy(raw_relation)
+ original.append(original_item)
+ normalized_item = copy.deepcopy(raw_relation)
+ normalized_item["relation"] = relation
+ normalized_parameters = copy.deepcopy(parameters)
+ normalized_parameters["object_anchor"] = anchor
+ normalized_parameters["object_target"] = target
+ normalized_item["parameters"] = normalized_parameters
+ by_target.setdefault(target, []).append(normalized_item)
+
+ primary_relation_by_target: dict[str, dict[str, Any]] = {}
+ primary_parent_by_target: dict[str, str] = {}
+ primary_children: dict[str, list[str]] = {}
+ secondary_walls: dict[str, tuple[dict[str, Any], ...]] = {}
+
+ for target in sorted(by_target):
+ target_relations = by_target[target]
+ structural = [
+ relation
+ for relation in target_relations
+ if not _is_wall(
+ str(relation["parameters"]["object_anchor"])
+ )
+ ]
+ wall_attaches = [
+ relation
+ for relation in target_relations
+ if _is_wall(str(relation["parameters"]["object_anchor"]))
+ ]
+
+ if len(structural) > 1:
+ candidates = [
+ (
+ relation["relation"],
+ relation["parameters"]["object_anchor"],
+ )
+ for relation in structural
+ ]
+ raise RelationGraphError(
+ f"target {target} has competing non-wall owners: {candidates}"
+ )
+
+ if structural:
+ primary = structural[0]
+ if wall_attaches:
+ secondary_walls[target] = tuple(
+ copy.deepcopy(item)
+ for item in sorted(
+ wall_attaches,
+ key=lambda item: (
+ item["parameters"]["object_anchor"],
+ item["relation"],
+ ),
+ )
+ )
+ elif wall_attaches:
+ if len(wall_attaches) > 1:
+ candidates = [
+ relation["parameters"]["object_anchor"]
+ for relation in wall_attaches
+ ]
+ raise RelationGraphError(
+ f"target {target} has competing wall owners: {candidates}"
+ )
+ primary = wall_attaches[0]
+ else:
+ raise RelationGraphError(
+ f"target {target} has no consumable ownership relation"
+ )
+
+ primary_copy = copy.deepcopy(primary)
+ anchor = str(primary_copy["parameters"]["object_anchor"])
+ primary_relation_by_target[target] = primary_copy
+ primary_parent_by_target[target] = anchor
+ primary_children.setdefault(anchor, []).append(target)
+
+ return cls(
+ full_relations=tuple(original),
+ primary_relation_by_target=MappingProxyType(
+ primary_relation_by_target
+ ),
+ primary_parent_by_target=MappingProxyType(
+ primary_parent_by_target
+ ),
+ primary_children_by_anchor=MappingProxyType(
+ {
+ anchor: tuple(sorted(targets))
+ for anchor, targets in sorted(primary_children.items())
+ }
+ ),
+ secondary_wall_constraints_by_target=MappingProxyType(
+ secondary_walls
+ ),
+ )
+
+ def object_groups(self) -> dict[str, dict[str, Any]]:
+ """Return primary-only groups accepted by the existing orchestrator."""
+
+ groups: dict[str, dict[str, Any]] = {}
+ for anchor, targets in self.primary_children_by_anchor.items():
+ relation_types = {
+ self.primary_relation_by_target[target]["relation"]
+ for target in targets
+ }
+ relation_type = (
+ next(iter(relation_types))
+ if len(relation_types) == 1
+ else "mixed"
+ )
+ groups[anchor] = {
+ "relation_type": relation_type,
+ "object_ids": list(targets),
+ }
+ secondary_constraints = [
+ copy.deepcopy(relation)
+ for target in targets
+ for relation in self.secondary_wall_constraints_by_target.get(
+ target, ()
+ )
+ ]
+ if secondary_constraints:
+ groups[anchor]["secondary_wall_constraints"] = (
+ secondary_constraints
+ )
+ return groups
+
+ def primary_parent(self, target: str) -> str | None:
+ return self.primary_parent_by_target.get(target)
+
+ def full_relations_list(self) -> list[dict[str, Any]]:
+ return copy.deepcopy(list(self.full_relations))
+
+ def filtered_to_object_ids(
+ self,
+ object_ids: Iterable[str],
+ ) -> "RelationGraphView":
+ """Restrict the graph to objects that survived physics filtering."""
+
+ allowed = {str(object_id) for object_id in object_ids}
+ filtered: list[dict[str, Any]] = []
+ for relation in self.full_relations:
+ parameters = relation.get("parameters", {})
+ anchor = str(parameters.get("object_anchor") or "")
+ target = str(parameters.get("object_target") or "")
+ if target not in allowed:
+ continue
+ if anchor.startswith("object_") and anchor not in allowed:
+ continue
+ filtered.append(copy.deepcopy(relation))
+ filtered_view = type(self).from_relations(filtered)
+ missing = sorted(allowed - set(filtered_view.primary_parent_by_target))
+ if missing:
+ raise RelationGraphError(
+ "surviving objects have no primary ownership after filtering: "
+ f"{missing}"
+ )
+ return filtered_view
+
+ def audit_summary(self) -> dict[str, Any]:
+ return {
+ "full_relation_count": len(self.full_relations),
+ "primary_ownership_count": len(self.primary_parent_by_target),
+ "primary_parent_by_target": dict(self.primary_parent_by_target),
+ "primary_children_by_anchor": {
+ anchor: list(targets)
+ for anchor, targets in self.primary_children_by_anchor.items()
+ },
+ "secondary_wall_constraints_by_target": {
+ target: copy.deepcopy(list(relations))
+ for target, relations
+ in self.secondary_wall_constraints_by_target.items()
+ },
+ }
diff --git a/vlm_agent/reconstruction/relations/object_physics/__init__.py b/vlm_agent/reconstruction/relations/object_physics/__init__.py
new file mode 100644
index 0000000..04c1513
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/object_physics/__init__.py
@@ -0,0 +1,2 @@
+"""Independent object physics enrichment sidecar."""
+
diff --git a/vlm_agent/reconstruction/relations/object_physics/core.py b/vlm_agent/reconstruction/relations/object_physics/core.py
new file mode 100644
index 0000000..eff7f6a
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/object_physics/core.py
@@ -0,0 +1,322 @@
+"""Deterministic validation and fallback for object physics estimates."""
+
+from __future__ import annotations
+
+from functools import lru_cache
+import math
+from typing import Any
+
+
+PHYSICS_FIELDS = (
+ "mass",
+ "linear_damping",
+ "angular_damping",
+ "static_friction",
+ "dynamic_friction",
+)
+
+UPRIGHT_FIELD = "upright"
+
+FIELD_LIMITS = {
+ "mass": (0.01, 10000.0),
+ "linear_damping": (0.0, 1.0),
+ "angular_damping": (0.0, 1.0),
+ "static_friction": (0.0, 2.0),
+ "dynamic_friction": (0.0, 2.0),
+}
+
+DEFAULT_PROFILE = {
+ "mass": 5.0,
+ "linear_damping": 0.08,
+ "angular_damping": 0.12,
+ "static_friction": 0.55,
+ "dynamic_friction": 0.4,
+}
+
+PROFILES = [
+ (("rug", "carpet", "mat", "floor covering"), {"mass": 8.0, "linear_damping": 0.3, "angular_damping": 0.5, "static_friction": 0.9, "dynamic_friction": 0.75}),
+ (("sofa", "couch", "bed"), {"mass": 55.0, "linear_damping": 0.12, "angular_damping": 0.2, "static_friction": 0.75, "dynamic_friction": 0.58}),
+ (("cabinet", "wardrobe", "bookshelf", "refrigerator"), {"mass": 75.0, "linear_damping": 0.1, "angular_damping": 0.18, "static_friction": 0.72, "dynamic_friction": 0.55}),
+ (("desk", "table", "counter", "console"), {"mass": 30.0, "linear_damping": 0.09, "angular_damping": 0.15, "static_friction": 0.68, "dynamic_friction": 0.5}),
+ (("chair", "stool", "ottoman", "bench"), {"mass": 10.0, "linear_damping": 0.08, "angular_damping": 0.14, "static_friction": 0.62, "dynamic_friction": 0.46}),
+ (("lamp", "floor light"), {"mass": 6.0, "linear_damping": 0.08, "angular_damping": 0.22, "static_friction": 0.7, "dynamic_friction": 0.48}),
+ (("television", "projection screen", "painting", "panel", "whiteboard"), {"mass": 7.0, "linear_damping": 0.12, "angular_damping": 0.18, "static_friction": 0.55, "dynamic_friction": 0.4}),
+ (("monitor", "speaker", "computer", "appliance", "equipment"), {"mass": 6.0, "linear_damping": 0.08, "angular_damping": 0.13, "static_friction": 0.58, "dynamic_friction": 0.42}),
+ (("book", "remote", "figurine", "decoration", "box", "vase"), {"mass": 0.8, "linear_damping": 0.06, "angular_damping": 0.1, "static_friction": 0.5, "dynamic_friction": 0.36}),
+ (("ceiling light", "recessed light"), {"mass": 2.5, "linear_damping": 0.12, "angular_damping": 0.18, "static_friction": 0.5, "dynamic_friction": 0.35}),
+]
+
+
+def _object_text(obj: dict[str, Any]) -> str:
+ return f"{obj.get('category', '')} {obj.get('description', '')}".lower().replace("_", " ")
+
+
+WALL_HANGING_UPRIGHT_FALSE_TERMS = {
+ "artwork",
+ "framed art",
+ "hanging art",
+ "painting",
+ "wall art",
+ "wall clock",
+ "wall decoration",
+ "wall mirror",
+ "wall mounted",
+ "wall panel",
+ "wall sign",
+ "whiteboard",
+}
+
+WALL_HANGING_UPRIGHT_PREFILTER_TERMS = ("wall", "art", "hang", "paint", "whiteboard")
+
+STABLE_UPRIGHT_TRUE_CATEGORY_TERMINALS = {
+ "basin",
+ "basket",
+ "stand",
+ "vessel",
+}
+
+STABLE_UPRIGHT_FALSE_CATEGORY_TERMINALS = {
+ "controller",
+ "cushion",
+ "pillow",
+ "slab",
+}
+
+STABLE_UPRIGHT_TRUE_COMPOUND_CATEGORIES = {
+ "slab_roller",
+}
+STABLE_UPRIGHT_TRUE_COMPOUND_SUFFIXES = tuple(f"_{value}" for value in STABLE_UPRIGHT_TRUE_COMPOUND_CATEGORIES)
+
+
+@lru_cache(maxsize=512)
+def _stable_upright_category_override(category_value: str) -> tuple[bool, str] | None:
+ category = category_value.strip().lower().replace("-", "_").replace(" ", "_").strip("_")
+ terminal = category.rsplit("_", 1)[-1]
+ if category in STABLE_UPRIGHT_TRUE_COMPOUND_CATEGORIES or category.endswith(STABLE_UPRIGHT_TRUE_COMPOUND_SUFFIXES):
+ return True, "rigid functional machine category has a gravity-defined functional up direction"
+ if terminal in STABLE_UPRIGHT_TRUE_CATEGORY_TERMINALS:
+ return True, "rigid container/support category has a gravity-defined functional up direction"
+ if terminal == "box":
+ if category == "box":
+ return False, "generic loose box category has no required gravity-upright constraint"
+ return True, "typed rigid box category has a gravity-defined functional up direction"
+ if terminal in STABLE_UPRIGHT_FALSE_CATEGORY_TERMINALS:
+ return False, "soft, handheld, or intentionally flat category has no gravity-upright constraint"
+ if category in {"bean_bag", "cooling_rack"} or category.endswith(("_bean_bag", "_cooling_rack")):
+ return False, "soft, handheld, or intentionally flat category has no gravity-upright constraint"
+ return None
+
+
+def local_upright_semantic_override(obj: dict[str, Any]) -> tuple[bool, str] | None:
+ """Return only category-stable relation-agnostic upright overrides."""
+ category_override = _stable_upright_category_override(str(obj.get("category") or ""))
+ if category_override is not None:
+ return category_override
+ text = _object_text(obj)
+ if any(marker in text for marker in WALL_HANGING_UPRIGHT_PREFILTER_TERMS):
+ if any(term in text for term in WALL_HANGING_UPRIGHT_FALSE_TERMS):
+ return False, "thin wall-hanging decorative/display object is not a gravity-upright constraint"
+ return None
+
+
+def local_physics_fallback(obj: dict[str, Any]) -> tuple[dict[str, float], str]:
+ """Return a broad semantic prior, not a measured physical property."""
+ category = str(obj.get("category") or "").lower().replace("_", " ")
+ text = _object_text(obj)
+ for keywords, profile in PROFILES:
+ if any(keyword in category for keyword in keywords):
+ return dict(profile), f"local semantic category profile matched: {category}"
+ for keywords, profile in PROFILES:
+ if any(keyword in text for keyword in keywords):
+ return dict(profile), f"local semantic description profile matched: {', '.join(keyword for keyword in keywords if keyword in text)}"
+ return dict(DEFAULT_PROFILE), "local generic object profile"
+
+
+def _finite_number(value: Any) -> float | None:
+ if isinstance(value, bool):
+ return None
+ try:
+ number = float(value)
+ except (TypeError, ValueError):
+ return None
+ return number if math.isfinite(number) else None
+
+
+def extract_agent_items(payload: Any) -> list[dict[str, Any]]:
+ if not isinstance(payload, dict):
+ return []
+ for key in ("object_physics", "objects", "physics_properties", "items"):
+ value = payload.get(key)
+ if isinstance(value, list):
+ return [item for item in value if isinstance(item, dict)]
+ if isinstance(value, dict) and isinstance(value.get("items"), list):
+ return [item for item in value["items"] if isinstance(item, dict)]
+ findings = payload.get("findings")
+ if isinstance(findings, dict):
+ return extract_agent_items(findings)
+ return []
+
+
+def resolve_object_physics(
+ objects_payload: dict[str, Any],
+ agent_payload: dict[str, Any] | None = None,
+) -> tuple[dict[str, Any], dict[str, Any], dict[str, Any]]:
+ """Resolve complete physics values and return sidecar, enriched objects, audit."""
+ objects = objects_payload.get("objects") if isinstance(objects_payload, dict) else None
+ if not isinstance(objects, list):
+ raise ValueError("objects payload must contain an objects list")
+ final_ids = {str(obj.get("id")) for obj in objects if isinstance(obj, dict) and obj.get("id")}
+ agent_by_id: dict[str, dict[str, Any]] = {}
+ ignored_agent_items: list[dict[str, Any]] = []
+ for item in extract_agent_items(agent_payload or {}):
+ object_id = str(item.get("object_id") or item.get("id") or "")
+ if object_id in final_ids:
+ agent_by_id[object_id] = item
+ else:
+ ignored_agent_items.append({"item": item, "reason": "object_id_not_in_final_objects"})
+
+ sidecar_items: list[dict[str, Any]] = []
+ enriched_objects: list[dict[str, Any]] = []
+ audit_items: list[dict[str, Any]] = []
+ invalid_agent_upright_object_ids: list[str] = []
+ for obj in objects:
+ if not isinstance(obj, dict) or not obj.get("id"):
+ continue
+ object_id = str(obj["id"])
+ fallback, fallback_reason = local_physics_fallback(obj)
+ agent = agent_by_id.get(object_id, {})
+ nested = agent.get("physics") if isinstance(agent.get("physics"), dict) else {}
+ resolved: dict[str, float] = {}
+ sources: dict[str, str] = {}
+ rejected: list[dict[str, Any]] = []
+ for field in PHYSICS_FIELDS:
+ raw = agent.get(field, nested.get(field))
+ number = _finite_number(raw)
+ lower, upper = FIELD_LIMITS[field]
+ if number is not None and lower <= number <= upper:
+ resolved[field] = round(number, 6)
+ sources[field] = "agent"
+ else:
+ resolved[field] = float(fallback[field])
+ sources[field] = "local_fallback"
+ if raw is not None:
+ rejected.append({"field": field, "value": raw, "reason": f"must be finite and within [{lower}, {upper}]"})
+
+ if resolved["dynamic_friction"] > resolved["static_friction"]:
+ rejected.append({
+ "field": "dynamic_friction",
+ "value": resolved["dynamic_friction"],
+ "reason": "dynamic_friction cannot exceed static_friction; clamped to static_friction",
+ })
+ resolved["dynamic_friction"] = resolved["static_friction"]
+ sources["dynamic_friction"] = f"{sources['dynamic_friction']}_clamped"
+
+ agent_upright = agent.get(UPRIGHT_FIELD)
+ input_upright = obj.get(UPRIGHT_FIELD)
+ resolved_upright: bool | None = None
+ upright_source = "missing"
+ upright_reason = None
+ if agent_upright is not None and not isinstance(agent_upright, bool):
+ invalid_agent_upright_object_ids.append(object_id)
+ rejected.append({
+ "field": UPRIGHT_FIELD,
+ "value": agent_upright,
+ "reason": "upright must be a JSON boolean",
+ })
+ semantic_upright_override = local_upright_semantic_override(obj)
+ if semantic_upright_override is not None:
+ resolved_upright, upright_reason = semantic_upright_override
+ upright_source = "local_semantic_override"
+ elif isinstance(agent_upright, bool):
+ resolved_upright = agent_upright
+ upright_source = "agent"
+ elif isinstance(input_upright, bool):
+ resolved_upright = input_upright
+ upright_source = "input_object_fallback"
+
+ sidecar_item = {
+ "object_id": object_id,
+ **resolved,
+ }
+ enriched_object = {**obj, "physics": resolved}
+ if resolved_upright is not None:
+ sidecar_item[UPRIGHT_FIELD] = resolved_upright
+ enriched_object[UPRIGHT_FIELD] = resolved_upright
+ sidecar_items.append(sidecar_item)
+ enriched_objects.append(enriched_object)
+ audit_items.append({
+ "object_id": object_id,
+ "resolved": resolved,
+ "field_sources": sources,
+ "upright": resolved_upright,
+ "upright_source": upright_source,
+ "upright_reason": upright_reason,
+ "fallback_reason": fallback_reason,
+ "agent_item_present": bool(agent),
+ "rejected_agent_values": rejected,
+ })
+
+ sidecar = {
+ "schema_version": "object_physics",
+ "mass_unit": "kg_estimate",
+ "damping_range": [0.0, 1.0],
+ "friction_model": "dimensionless_coulomb_estimate",
+ "objects": sidecar_items,
+ }
+ enriched = {**objects_payload, "objects": enriched_objects}
+ audit = {
+ "schema_version": "object_physics_audit",
+ "status": "completed",
+ "object_count": len(enriched_objects),
+ "agent_item_count": len(agent_by_id),
+ "fallback_only_object_count": sum(1 for item in audit_items if not item["agent_item_present"]),
+ "invalid_agent_upright_object_ids": invalid_agent_upright_object_ids,
+ "ignored_agent_items": ignored_agent_items,
+ "items": audit_items,
+ "policy": "Agent physics estimates are accepted only inside numeric bounds. Missing/invalid numeric fields use deterministic local semantic priors; dynamic friction cannot exceed static friction. Agent upright is retained only when it is a JSON boolean; relation-aware overrides remain the final assembler's responsibility.",
+ }
+ return sidecar, enriched, audit
+
+
+def validate_object_physics(payload: dict[str, Any], expected_object_ids: list[str]) -> list[str]:
+ errors: list[str] = []
+ items = payload.get("objects") if isinstance(payload, dict) else None
+ if not isinstance(items, list):
+ return ["object_physics.json must contain an objects list"]
+ ids = [str(item.get("object_id")) for item in items if isinstance(item, dict)]
+ if ids != expected_object_ids:
+ errors.append(f"physics object ids/order mismatch: expected {expected_object_ids}, got {ids}")
+ for item in items:
+ if not isinstance(item, dict):
+ errors.append("each physics item must be an object")
+ continue
+ for field in PHYSICS_FIELDS:
+ number = _finite_number(item.get(field))
+ lower, upper = FIELD_LIMITS[field]
+ if number is None or not lower <= number <= upper:
+ errors.append(f"{item.get('object_id')} {field} must be finite within [{lower}, {upper}]")
+ dynamic = _finite_number(item.get("dynamic_friction"))
+ static = _finite_number(item.get("static_friction"))
+ if dynamic is not None and static is not None and dynamic > static:
+ errors.append(f"{item.get('object_id')} dynamic_friction exceeds static_friction")
+ return errors
+
+def validate_object_upright_priors(payload: dict[str, Any], expected_object_ids: list[str]) -> list[str]:
+ """Validate the sidecar's relation-agnostic upright prior for each object."""
+ errors: list[str] = []
+ items = payload.get("objects") if isinstance(payload, dict) else None
+ if not isinstance(items, list):
+ return ["object_physics.json must contain an objects list"]
+ by_id = {
+ str(item.get("object_id")): item
+ for item in items
+ if isinstance(item, dict) and item.get("object_id")
+ }
+ for object_id in expected_object_ids:
+ item = by_id.get(str(object_id))
+ if item is None:
+ errors.append(f"{object_id} is missing from sidecar upright priors")
+ elif not isinstance(item.get(UPRIGHT_FIELD), bool):
+ errors.append(f"{object_id} upright must be a JSON boolean")
+ return errors
+
diff --git a/vlm_agent/reconstruction/relations/object_physics/object_physics_agent_system.txt b/vlm_agent/reconstruction/relations/object_physics/object_physics_agent_system.txt
new file mode 100644
index 0000000..b34e8c0
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/object_physics/object_physics_agent_system.txt
@@ -0,0 +1,34 @@
+You are an Object Physics and Upright Prior Estimation Agent. This task is independent from scene relations and wall geometry.
+
+Estimate plausible reconstruction/simulation priors for every supplied object. These are semantic estimates, not measured laboratory values.
+
+Return one JSON object with this exact top-level shape:
+
+{
+ "object_physics": [
+ {
+ "object_id": "object_001",
+ "upright": true,
+ "mass": 10.0,
+ "linear_damping": 0.08,
+ "angular_damping": 0.12,
+ "static_friction": 0.60,
+ "dynamic_friction": 0.45
+ }
+ ]
+}
+
+Requirements:
+
+- Return exactly one item for every supplied object id, in the supplied order.
+- upright is a relation-agnostic canonical reconstruction prior. Set it to true when the object's local-up direction must remain aligned with gravity for a stable, meaningful pose. This includes ordinary furniture/appliances and objects whose opening, contents, standing pose, or operating orientation defines an up direction, such as bowls, cups, trays, vases, plants, toy houses, stacking toys, and open-top bins. Set it to false for flexible, loose, orientation-insensitive, intentionally flat, or clearly wall-hanging thin decorative/display objects, such as rugs, pillows, cushions, loose books, balls, paintings, framed wall art, wall clocks, wall signs, wall panels, and wall-mounted whiteboards. A thin wall-hanging object is constrained by its mounting plane rather than by gravity uprightness.
+- Judge the object category consistently across scenes. Use the description, compact 3D extent, original-image bbox shape, and optional original image only to resolve category ambiguity.
+- This is not a relation decision. Do not infer attach, contain, walls, wall ids, or anchors. Final assembly will override upright=false for objects whose final relations are wall-only or ceiling-attached.
+- mass is an estimated real-world mass in kilograms and must be greater than zero.
+- linear_damping and angular_damping must be within [0, 1]. They represent velocity damping, not spring stiffness.
+- Friction values are dimensionless Coulomb-friction estimates within [0, 2].
+- dynamic_friction must not exceed static_friction.
+- Infer broad material, size, and mobility from category, description, optional bbox extent, and optional original image.
+- Use conservative values when evidence is weak. Do not claim exact measured physics.
+- For each object return exactly object_id, upright, and the five required numeric fields. Do not return confidence, reason, prose, relation data, or extra keys.
+- Output JSON only.
diff --git a/vlm_agent/reconstruction/relations/object_physics/run_sidecar.py b/vlm_agent/reconstruction/relations/object_physics/run_sidecar.py
new file mode 100644
index 0000000..98783cf
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/object_physics/run_sidecar.py
@@ -0,0 +1,276 @@
+#!/usr/bin/env python3
+"""Run independent object-physics enrichment beside or after relation extraction."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+CODE_ROOT = Path(__file__).resolve().parents[2]
+if str(CODE_ROOT) not in sys.path:
+ sys.path.insert(0, str(CODE_ROOT))
+
+from relations.agents.api_client import APIClientError, APIConfig, SceneDeliveryAPIClient
+from relations.object_physics.core import resolve_object_physics, validate_object_physics, validate_object_upright_priors
+
+
+PROMPT_PATH = Path(__file__).with_name("object_physics_agent_system.txt")
+
+
+def read_json(path: Path) -> Any:
+ return json.loads(path.read_text(encoding="utf-8"))
+
+
+def write_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_suffix(path.suffix + ".tmp")
+ temporary.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ temporary.replace(path)
+
+
+def scene_objects(scene_state: dict[str, Any]) -> dict[str, dict[str, Any]]:
+ candidates = scene_state.get("object_candidates") if isinstance(scene_state, dict) else None
+ items = candidates.get("items", []) if isinstance(candidates, dict) else []
+ return {
+ str(item["object_id"]): item
+ for item in items
+ if isinstance(item, dict) and item.get("object_id")
+ }
+
+
+def original_image_path(
+ run_root: Path,
+ scene_state: dict[str, Any],
+ expected_manifest: dict[str, Any] | None = None,
+) -> Path | None:
+ evidence = scene_state.get("evidence") if isinstance(scene_state, dict) else None
+ candidate = evidence.get("original_image_path") if isinstance(evidence, dict) else None
+ paths = [Path(candidate)] if candidate else []
+ artifacts = expected_manifest.get("artifacts", {}) if isinstance(expected_manifest, dict) else {}
+ original = artifacts.get("original_image") if isinstance(artifacts, dict) else None
+ manifest_candidate = original.get("path") if isinstance(original, dict) else original
+ if manifest_candidate:
+ paths.append(Path(manifest_candidate))
+ paths.extend(run_root / "input" / name for name in ("image.png", "image.jpg", "image.jpeg"))
+ return next((path for path in paths if path.exists()), None)
+
+
+def objects_from_expected_manifest(expected_manifest: dict[str, Any]) -> dict[str, Any]:
+ objects = []
+ for item in expected_manifest.get("objects", []):
+ if not isinstance(item, dict):
+ continue
+ object_id = item.get("object_id") or item.get("id")
+ if not object_id:
+ continue
+ obj = {
+ "id": str(object_id),
+ "category": item.get("category") or "unknown_object",
+ "description": item.get("description") or item.get("category") or "unknown object",
+ }
+ bbox_2d = item.get("vlm_bbox_2d_norm_xyxy") or item.get("bbox_2d")
+ if isinstance(bbox_2d, list) and len(bbox_2d) >= 4:
+ obj["bbox_2d_norm_xyxy"] = bbox_2d[:4]
+ objects.append(obj)
+ return {"walls": [], "objects": objects}
+
+
+def build_agent_payload(scene_name: str, objects_payload: dict[str, Any], scene_state: dict[str, Any]) -> dict[str, Any]:
+ context = scene_objects(scene_state)
+ objects = []
+ for obj in objects_payload.get("objects", []):
+ object_id = str(obj.get("id"))
+ scene_obj = context.get(object_id, {})
+ bbox = scene_obj.get("bbox_3d") if isinstance(scene_obj.get("bbox_3d"), dict) else {}
+ bbox_2d = obj.get("bbox_2d_norm_xyxy") or scene_obj.get("bbox_2d")
+ item = {
+ "object_id": object_id,
+ "category": obj.get("category"),
+ "description": str(obj.get("description") or "")[:240],
+ "bbox_extent_xyz": bbox.get("extent"),
+ }
+ if isinstance(bbox_2d, list) and len(bbox_2d) >= 4:
+ try:
+ width = abs(float(bbox_2d[2]) - float(bbox_2d[0]))
+ height = abs(float(bbox_2d[3]) - float(bbox_2d[1]))
+ item["original_image_bbox_xyxy"] = [round(float(value), 6) for value in bbox_2d[:4]]
+ item["image_bbox_height_width_ratio"] = round(height / width, 6) if width > 1e-6 else None
+ except (TypeError, ValueError):
+ pass
+ objects.append(item)
+ return {
+ "schema_version": "object_physics_request_compact",
+ "scene_name": scene_name,
+ "task": "Estimate independent object simulation-physics priors and one relation-agnostic upright prior. Do not reason about attach, contain, walls, or object identity.",
+ "objects": objects,
+ "required_fields": ["upright", "mass", "linear_damping", "angular_damping", "static_friction", "dynamic_friction"],
+ "response_policy": "Return object_id, one boolean upright field, and exactly the five required numeric fields for every supplied object. Return no reasons, confidence, prose, or relation data.",
+ }
+
+
+def sidecar_publish_eligible(validation_errors: list[str], upright_validation_errors: list[str]) -> bool:
+ """Return whether this sidecar is complete enough for downstream publication."""
+ return not validation_errors and not upright_validation_errors
+
+
+def write_report(output_dir: Path, runtime: dict[str, Any], audit: dict[str, Any], errors: list[str]) -> None:
+ lines = [
+ "# Object Physics Sidecar Report",
+ "",
+ f"- status: `{runtime['status']}`",
+ f"- model: `{runtime['model']}`",
+ f"- relation_critical_path_dependency: `false`",
+ f"- object_count: `{audit.get('object_count', 0)}`",
+ f"- fallback_only_object_count: `{audit.get('fallback_only_object_count', 0)}`",
+ f"- validation_errors: `{len(errors)}`",
+ f"- upright_validation_errors: `{len(runtime.get('upright_validation_errors', []))}`",
+ f"- publish_eligible: `{str(bool(runtime.get('publish_eligible'))).lower()}`",
+ "",
+ "The sidecar subprocess never writes relation final outputs. Final assembly may consume its ready upright/physics priors by object id without waiting; otherwise local fallbacks are used.",
+ ]
+ (output_dir / "scene_report.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--scene_name", required=True)
+ parser.add_argument("--run_root", required=True, type=Path)
+ parser.add_argument("--agent_run_dir", type=Path)
+ parser.add_argument("--expected_manifest", type=Path)
+ parser.add_argument("--sidecar_id", required=True)
+ parser.add_argument("--output_dir", type=Path)
+ parser.add_argument("--model", default="gpt-5.5")
+ parser.add_argument("--vision_model", default="gpt-5.5")
+ parser.add_argument("--api_base")
+ parser.add_argument("--api_key_env", default="OPENAI_API_KEY")
+ parser.add_argument("--vision_image_detail", choices=["low", "high", "auto"], default="low")
+ parser.add_argument("--include_original", action="store_true")
+ parser.add_argument("--decision_json", type=Path)
+ parser.add_argument("--dry_run", action="store_true")
+ return parser.parse_args()
+
+
+def main() -> None:
+ args = parse_args()
+ if not args.agent_run_dir and not args.expected_manifest:
+ raise SystemExit("Provide either --agent_run_dir or --expected_manifest")
+ if args.output_dir:
+ output_dir = args.output_dir
+ elif args.agent_run_dir:
+ output_dir = args.agent_run_dir / "physics_sidecar" / args.sidecar_id
+ else:
+ raise SystemExit("--output_dir is required with --expected_manifest")
+ output_dir.mkdir(parents=True, exist_ok=True)
+ expected_manifest = read_json(args.expected_manifest) if args.expected_manifest else None
+ if args.expected_manifest and not isinstance(expected_manifest, dict):
+ raise SystemExit("--expected_manifest must contain a JSON object")
+ if args.expected_manifest:
+ objects_path = args.expected_manifest
+ relations_path = None
+ scene_state_path = None
+ objects_payload = objects_from_expected_manifest(expected_manifest)
+ scene_state = {}
+ input_mode = "expected_manifest_async"
+ else:
+ objects_path = args.agent_run_dir / "final_outputs" / "objects.json"
+ relations_path = args.agent_run_dir / "final_outputs" / "relations.json"
+ scene_state_path = args.agent_run_dir / "scene_states" / "scene_state.json"
+ if not objects_path.exists() or not relations_path.exists():
+ raise SystemExit("Physics sidecar requires completed relation final_outputs/objects.json and relations.json")
+ objects_payload = read_json(objects_path)
+ scene_state = read_json(scene_state_path) if scene_state_path.exists() else {}
+ input_mode = "completed_relation_outputs"
+ if not objects_payload.get("objects"):
+ raise SystemExit("Physics sidecar input contains no objects")
+ image_path = original_image_path(args.run_root, scene_state, expected_manifest)
+
+ agent_payload: dict[str, Any] = {}
+ status = "fallback_only"
+ api_error = None
+ retry_events: list[dict[str, Any]] = []
+ if args.decision_json:
+ agent_payload = read_json(args.decision_json)
+ status = "decision_json"
+ elif not args.dry_run:
+ client = SceneDeliveryAPIClient(APIConfig(
+ model=args.model,
+ vision_model=args.vision_model,
+ api_base=args.api_base,
+ api_key_env=args.api_key_env,
+ dry_run=False,
+ image_detail=args.vision_image_detail,
+ ))
+ api_request_payload = build_agent_payload(args.scene_name, objects_payload, scene_state)
+ try:
+ images = [str(image_path)] if args.include_original and image_path else None
+ agent_payload = client.call_json(
+ role_name="ObjectPhysicsEstimator",
+ system_prompt=PROMPT_PATH.read_text(encoding="utf-8"),
+ user_payload=api_request_payload,
+ use_vision_model=bool(images),
+ image_paths=images,
+ )
+ status = "agent_completed"
+ except APIClientError as exc:
+ status = "fallback_after_api_error"
+ api_error = str(exc)
+ retry_events = exc.retry_events
+
+ sidecar, enriched, audit = resolve_object_physics(objects_payload, agent_payload)
+ expected_ids = [str(obj.get("id")) for obj in objects_payload.get("objects", [])]
+ validation_errors = validate_object_physics(sidecar, expected_ids)
+ upright_validation_errors = validate_object_upright_priors(sidecar, expected_ids)
+ upright_validation_errors.extend(
+ f"{object_id} agent upright must be a JSON boolean"
+ for object_id in audit.get("invalid_agent_upright_object_ids", [])
+ if f"{object_id} agent upright must be a JSON boolean" not in upright_validation_errors
+ )
+ publish_eligible = sidecar_publish_eligible(validation_errors, upright_validation_errors)
+ if validation_errors:
+ status = "validation_failed"
+ elif upright_validation_errors and status in {"agent_completed", "decision_json"}:
+ status = f"{status}_with_incomplete_upright"
+
+ write_json(output_dir / "agent_response.json", agent_payload)
+ write_json(output_dir / "object_physics.json", sidecar)
+ write_json(output_dir / "objects_with_physics.json", enriched)
+ write_json(output_dir / "physics_audit.json", audit)
+ runtime = {
+ "status": status,
+ "scene_name": args.scene_name,
+ "sidecar_id": args.sidecar_id,
+ "model": args.vision_model if args.include_original else args.model,
+ "include_original": bool(args.include_original and image_path),
+ "relation_critical_path_dependency": False,
+ "original_relation_outputs_modified": False,
+ "input_mode": input_mode,
+ "input_objects_path": str(objects_path),
+ "input_relations_path": str(relations_path) if relations_path else None,
+ "original_image_path": str(image_path) if image_path else None,
+ "api_error": api_error,
+ "retry_events": retry_events,
+ "validation_errors": validation_errors,
+ "upright_validation_errors": upright_validation_errors,
+ "publish_eligible": publish_eligible,
+ }
+ write_json(output_dir / "runtime.json", runtime)
+ write_report(output_dir, runtime, audit, validation_errors)
+ print(json.dumps({
+ "status": status,
+ "output_dir": str(output_dir),
+ "object_count": audit.get("object_count"),
+ "fallback_only_object_count": audit.get("fallback_only_object_count"),
+ "validation_errors": validation_errors,
+ "upright_validation_errors": upright_validation_errors,
+ "publish_eligible": publish_eligible,
+ }, ensure_ascii=False, indent=2))
+ if not publish_eligible:
+ raise SystemExit(2)
+
+
+if __name__ == "__main__":
+ main()
+
diff --git a/vlm_agent/reconstruction/relations/policies/multi_agent_policy.md b/vlm_agent/reconstruction/relations/policies/multi_agent_policy.md
new file mode 100644
index 0000000..3b64a21
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/policies/multi_agent_policy.md
@@ -0,0 +1,64 @@
+# Multi-Agent Relation Delivery Policy
+
+This is the static production policy for relation extraction. It defines output and evidence constraints without prescribing a fixed DAG or fixed task names.
+
+## Delivery scope
+
+- Read scene inputs from the relation-ready package generated from SAM3D outputs.
+- Write final `objects.json` and `relations.json` only inside the requested run output directory.
+- Preserve intermediate evidence, rejected candidates, repairs, and timing under `debug/`.
+- Do not use historical relation outputs as inference inputs.
+
+## Objects and walls
+
+- Final `objects.json` contains exactly four walls and the complete SAM3D object id set.
+- Walls use ids `wall_001` through `wall_004` and OpenGL top-down `[x, z]` coordinates.
+- Formal wall geometry comes from `obb_from_mesh_footprint_min_rect` and is canonicalized to screen-clockwise ids before relation merge.
+- Each final object contains base fields `id`, `category`, `description`, boolean `upright`, and a local-validated `physics` object with `mass`, `linear_damping`, `angular_damping`, `static_friction`, and `dynamic_friction`.
+- Only objects that anchor at least one final `contain` relation additionally contain `contain_anchor=true` and `render_direction=front|up`; non-anchors omit both fields.
+- Upright has one semantic owner: the independent Physics Sidecar returns a relation-agnostic prior together with the five simulation values. Final assembly then applies deterministic relation overrides: wall-only and ceiling attach force `upright=false`; floor/object-supported furniture and displays keep the sidecar prior. Local semantic/image/geometry logic is fallback only when the sidecar prior is unavailable.
+- Agents may propose object revision/property patches; they must not replace, merge, delete, or renumber the object list.
+
+## Relations
+
+- Final relation types are `attach` and `contain`.
+- Both use `parameters.object_anchor` and `parameters.object_target`.
+- Attach anchors may be `floor`, confirmed `ceiling`, a canonical wall id, or an existing object id.
+- Contain anchors and targets must both be existing object ids.
+- The same anchor-target pair cannot be both attach and contain. Neither type wins by default; conflicting independent Agent verdicts are arbitrated from a local original-image crop in the existing DeliverySchemaGuard call.
+- Multiple directional contacts are allowed when physically valid, for example floor attach plus wall attach.
+- Each object target may have at most one wall attach relation; final assembly deterministically removes additional wall anchors without removing floor/object support.
+- Every object should be covered as an attach or contain target unless the run is explicitly left partial for repair.
+
+## Semantics
+
+- Attach means physical support, contact, mounting, hanging, or fixation. Nearness or facing alone is insufficient.
+- Contain means the target is inside an ordinary container or on/within any functional level of multi-level furniture. Every occupied level of a bunk bed, bookcase/shelving unit, multi-tier service cart, or rack, including the top, is contain and the same pair must not use attach. A plain exterior top is attach only for an ordinary single-level anchor.
+- Local contain candidates are high-recall review selectors, not semantic evidence and not a closed set; agents must reject false positives and check for directly visible missed pairs.
+- After Agent acceptance, local mesh-footprint OBB validation removes only clear contain contradictions. Missing geometry or near-boundary cases remain uncertain and preserve Agent judgement.
+- Ceiling anchors require positive evidence.
+- Weak, inferred, closure-only, or not-visible walls do not anchor ordinary freestanding objects without strong wall-mounted evidence.
+
+## Physical properties
+
+- Relation Agents may output `is_floor_covering` with confidence and reasons. They must not output `upright`; that field belongs to the independent Physics Sidecar.
+- Floor-covering anchors are normalized to floor support by final assembly.
+- Objects covered only by wall attach, ceiling-attached objects, and ceiling fixtures are not gravity-upright.
+- Explicitly wall-mounted objects with accepted wall attach must not also receive floor attach solely for coverage.
+
+## Evidence priority
+
+- Preserve mask/object grounding unless strong evidence proves it wrong.
+- Use the canonical top-down image, its matching render-grid affine, object footprints, original image, and wall evidence together.
+- Agent outputs are semantic proposals. Canonical wall-id mapping, schema validation, deduplication, conflict resolution, and coverage accounting are local deterministic responsibilities.
+
+## Agent coordination
+
+- SceneDeliveryManager chooses the task DAG from the capability contract and current scene.
+- Skeleton-async planning may begin from the object list and original image before heavy artifacts finish.
+- Hydration binds logical artifact keys to generated files after preprocessing.
+- Independent tasks in the same DAG layer may run concurrently.
+- DynamicTaskAgent performs scoped evidence/reasoning tasks.
+- ExecutionSupervisor evaluates task completeness.
+- DeliverySchemaGuard checks candidate coverage and schema readiness.
+- FinalAssembler creates the deterministic final payloads and audit files.
diff --git a/vlm_agent/reconstruction/relations/prompts/multi_agent/delivery_schema_guard_system.txt b/vlm_agent/reconstruction/relations/prompts/multi_agent/delivery_schema_guard_system.txt
new file mode 100644
index 0000000..e3645ad
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/prompts/multi_agent/delivery_schema_guard_system.txt
@@ -0,0 +1,22 @@
+You are DeliverySchemaGuard.
+
+Your job is to validate and summarize accepted multi-agent working_state for final delivery. Do not introduce new unsupported judgements at this stage.
+
+Do not generate the full objects list from scratch. The runner assembles final objects deterministically from scene_state.object_candidates. You may return object revision patches only, each with object_id, optional new_category, optional new_description, and reason.
+
+The local final assembler validates that canonical wall_001 through wall_004 are present, use [x, z] points, connect in order, and satisfy the clockwise convention. Treat canonical_walls in the context as immutable local OBB geometry: do not renumber, replace, or infer alternative wall coordinates. If they are missing or inconsistent, report that validation issue rather than inventing a wall repair.
+
+relations_json may contain attach and contain relations. Enforce object_anchor/object_target parameter names and reject forbidden aliases. A contain relation must use object anchors and object targets. For the same anchor-target pair, attach and contain are mutually exclusive, but neither relation type has a fixed priority. When the supplied conflict ledger lists the pair, return exactly one structured arbitration verdict.
+
+The API context is a compact decision ledger, not raw geometry. Reconcile only the supplied relation_candidate_catalog and task conclusion summaries. You may accept, reject, or correct candidates already present in that catalog, but do not invent a new object-wall pair or replacement wall geometry. Formal OBB wall coordinates and deterministic schema checks remain local.
+
+When contain_conflicts is non-empty, resolve every listed conflict exactly once. A conflict may be independent accepted attach versus accepted contain verdicts, a support-inside hypothesis versus an explicit contain rejection, or a high-confidence local recall pair that the contain specialist explicitly marked uncertain. Use the attached original-image crop as the primary semantic evidence; local candidate selection and bbox overlap are not proof. Return:
+- accept_contain with support_surface_role=internal_shelf or multi_level_tier when the target is genuinely inside an ordinary container or placed on/within any functional level of multi-level furniture; or
+- reject_contain_use_support with support_surface_role=outer_top or external_surface and support_anchor=floor or object_### only when the target rests on a plain exposed surface with no rim, rail, recess, frame, or other containing boundary.
+For multi-level furniture, every occupied functional level is contain, including the top level and regardless of rails or rims. This includes both bunks of a bunk bed and every tier of a bookcase/shelving unit, multi-tier service cart, or rack. Do not preserve object-to-object attach for the same pair. For ordinary single-level furniture, a plain exterior top remains attach. Require positive original-image evidence that the target belongs to the stated level.
+Every reject_contain_use_support verdict must include support_surface_role and a concrete reason_code grounded in the crop. If the crop is ambiguous, use support_surface_role=unknown; do not silently override an accepted specialist contain verdict.
+Do not omit the verdict, rename either object, invent a new pair, default to contain, or use a generic explanation in place of the structured decision.
+
+If final outputs cannot be fully validated, record every failure in final_validation audit and report repair tasks; do not claim success.
+
+Return JSON with: relations_json, ceiling_confirmed, wall_candidates, object_revision_patches, validation_notes, repair_notes, contain_conflict_resolutions.
diff --git a/vlm_agent/reconstruction/relations/prompts/multi_agent/dynamic_task_agent_compact_system.txt b/vlm_agent/reconstruction/relations/prompts/multi_agent/dynamic_task_agent_compact_system.txt
new file mode 100644
index 0000000..aa41678
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/prompts/multi_agent/dynamic_task_agent_compact_system.txt
@@ -0,0 +1,21 @@
+FORMAL WALL AUTHORITY: use the supplied formal OBB wall geometry and canonical ids exactly. Do not invent, resize, rotate, renumber, or replace walls.
+
+You are DynamicTaskAgent. Complete the capability task described in the payload using only supplied scene state, evidence, policy, and prior results.
+
+Preserve SAM3D object ids exactly. Do not create a replacement object list or invent object ids, wall ids, anchors, or evidence.
+
+Follow the task's instructions and expected_output_schema. Return JSON only with the requested fields. Use explicit uncertainty when evidence is insufficient; do not turn a weak guess into an accepted relation.
+
+When payload.response_policy.format is compact_structured_json, keep all machine-readable decisions, ids, booleans, confidence, and required schema fields. Use short snake_case reason_code/evidence_codes instead of prose explanations, do not repeat supplied local geometry, and do not duplicate accepted relations across multiple output fields.
+
+Relation rules:
+- Allowed relation types are attach and contain.
+- contain means the target is inside an anchor object's interior storage/display volume.
+- For the same anchor-target pair, contain and attach are mutually exclusive.
+- Floor support and an additional physically supported wall contact may coexist.
+- Weak, hidden, inferred, closure-only, or not-visible walls are not ordinary wall attach anchors.
+
+Evidence rules:
+- Ground object decisions in original image, masks, footprints, and relevant top-down evidence.
+- Treat descriptions and categories as hints, not proof.
+- For wall tasks, images verify formal layout-XZ walls; they do not create replacement coordinates.
diff --git a/vlm_agent/reconstruction/relations/prompts/multi_agent/dynamic_task_agent_system.txt b/vlm_agent/reconstruction/relations/prompts/multi_agent/dynamic_task_agent_system.txt
new file mode 100644
index 0000000..e62dd57
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/prompts/multi_agent/dynamic_task_agent_system.txt
@@ -0,0 +1,24 @@
+FORMAL WALL AUTHORITY: local code owns the selected room-wall geometry and canonical wall ids. Use supplied formal walls exactly. Never choose a wall-orientation candidate, infer replacement coordinates, resize or rotate the room, renumber walls, or promote provisional geometry.
+
+You are DynamicTaskAgent. Complete the task in the payload using only the supplied scene state, evidence, policy, prior task results, and expected output schema. Return JSON only.
+
+Preserve every supplied SAM3D object id exactly. Object ids use the formal continuous `object_001` sequence established upstream. Never invent, delete, merge, or renumber objects, wall ids, anchors, relations, or evidence. Do not return a replacement objects list; object corrections must be revision patches for existing ids only.
+
+Relation rules:
+- The only relation types are `attach` and `contain`.
+- `contain` means that the target occupies an anchor object's interior storage or display volume. It is not ordinary top-surface support.
+- The same anchor-target pair cannot be both `attach` and `contain`.
+- Floor support and an additional physically supported wall contact may coexist.
+- Weak, hidden, inferred, closure-only, or not-visible walls cannot anchor a wall-attach relation.
+- Use visible contact, masks, footprints, and layout evidence. Category and description text are hints, not proof.
+- A non-light object cannot be attached to the ceiling without direct positive visual evidence from the dedicated ceiling judgement.
+
+For a missing-relation repair, select the most plausible physical anchor from supplied evidence. Prefer observed local support or containment over a generic floor fallback. Return only the requested relation proposal and compact audit evidence; do not repeat the full graph.
+
+For a wall-related repair, copy the supplied formal wall ids and geometry keys unchanged. Images may verify visibility, contact, and semantic consistency, but they do not generate coordinates. If the supplied wall fact source is insufficient, report the task as unresolved rather than producing alternative walls.
+
+For a `wall_final_adjudication` repair, review the already selected four formal walls for visibility, closure, and whether each wall may anchor relations. Do not select among orientation candidates. Return either the requested verified-wall review for those exact walls or an explicit insufficient-evidence result.
+
+If object metadata needs correction, emit only the requested revision fields for an existing object id. Do not infer `upright` or convert physical-property estimates to density; the physics sidecar and downstream audit own those values.
+
+Follow `task.instructions`, `expected_output_schema`, and the payload response policy exactly. Keep all required ids, booleans, confidence values, decisions, and machine-readable fields. Use short reason/evidence codes where requested. Express uncertainty explicitly; never turn a weak guess into an accepted relation.
diff --git a/vlm_agent/reconstruction/relations/prompts/multi_agent/execution_supervisor_system.txt b/vlm_agent/reconstruction/relations/prompts/multi_agent/execution_supervisor_system.txt
new file mode 100644
index 0000000..ebb1e7c
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/prompts/multi_agent/execution_supervisor_system.txt
@@ -0,0 +1,28 @@
+You are ExecutionSupervisorAgent.
+
+Your role is not to extract attach relations directly. Your role is to supervise the autonomous multi-agent execution process.
+
+Do not use a fixed hard limit such as a fixed number of rounds, API calls, or renders. Decide adaptively.
+
+Allow more calls when:
+- a new call is likely to produce evidence that can change final objects.json or relations.json;
+- an unresolved attach candidate can be clarified by visual, mask, geometry, or top-down evidence;
+- a generated local top-down view can reduce ambiguity;
+- a schema or consistency issue has a clear repair path.
+
+Do not allow successful finalize unless:
+- final objects have four valid walls wall_001 through wall_004;
+- final object ids exactly match scene_state.object_candidates;
+- every relation object_target exists in final objects;
+- ceiling anchors are absent unless ceiling is positively confirmed.
+
+Stop further calls when:
+- final object ids have been checked against scene_state.object_candidates, four valid walls exist, relation targets are legal, and no task is likely to change valid final deliverables;
+- recent tasks repeat previous reasoning without adding new evidence;
+- additional calls are unlikely to change accepted walls, ceiling decision, object categories, descriptions, or attach relations;
+- remaining unresolved items should be recorded in debug rather than forced into final outputs;
+- final_outputs are ready for schema validation.
+
+When stopping, explain why continued execution is no longer expected to improve the final deliverables. If required final schema pieces are missing, use status finalization_failed_with_partial_outputs rather than successful final output.
+
+Return JSON with: decision ("continue" or "stop"), reason, requested_next_tasks, ready_for_schema_guard.
diff --git a/vlm_agent/reconstruction/relations/prompts/multi_agent/scene_delivery_manager_compact_system.txt b/vlm_agent/reconstruction/relations/prompts/multi_agent/scene_delivery_manager_compact_system.txt
new file mode 100644
index 0000000..1459fd5
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/prompts/multi_agent/scene_delivery_manager_compact_system.txt
@@ -0,0 +1,62 @@
+You are SceneDeliveryManager. Produce a compact capability decision DAG for the relation-extraction stage.
+
+Your responsibility is orchestration, not execution boilerplate. Decide which scene-dependent reasoning capabilities are needed, whether compatible capabilities should be merged into one task, what object scope deserves attention, and which semantic dependencies are necessary. Local code will compile your compact nodes into executable DynamicTaskAgent contracts, bind artifact paths, add fixed schemas/prompts, and enforce physical/schema invariants.
+
+The formal wall coordinate source is the local posed-mesh OBB candidate set. You may autonomously plan wall reasoning capabilities, but wall orientation now follows a strict data contract: an Agent identifies genuinely wall-following objects, then local code scores finite OBB axes from those objects' canonical rotation_degrees_topdown. No Agent chooses a candidate id or creates/edits wall coordinates, angles, ids, or methods. Visibility and weak/inferred semantics must be judged only after local code applies the selected orientation and rebuilds the final wall references.
+
+Use the original image and object list to make scene-specific decisions.
+
+Before defining tasks, assign every object one compact physical anchor route:
+- ceiling_only: visibly ceiling-mounted and not floor/object/wall supported.
+- wall_only: visibly suspended, embedded, hung, rack-mounted, or bracket-mounted on a wall with no gravity support from floor or another object.
+- floor_wall: floor-standing furniture whose back or side may also physically contact a wall. This route must enter both Support and Wall reasoning; task ownership is not mutually exclusive.
+- support_only: floor-supported or object-supported and not explicitly wall-mounted. A television on a console belongs here unless a wall bracket/mount is visible.
+- contain_candidate: positive evidence suggests the object is inside a storage/display volume.
+- uncertain: evidence is insufficient; preserve recall by routing to every physically plausible task.
+
+Use high confidence only when the exclusive route is visually clear. Medium/low routing must not close another plausible reasoning path. Being near or parallel to a wall is not wall_only. Ceiling-only is exclusive from Support and Wall; floor_wall is explicitly shared by both.
+
+Then make scene-specific task decisions:
+- Every nonempty scene must provide support_relations_ready. You may use support_contact_extraction or merge that capability into a compatible reasoning node, but object scopes are priorities rather than mutually exclusive ownership: every object still requires one explicit gravity-support verdict unless it is exclusively wall-mounted, ceiling-mounted, or contained.
+- Every scene must provide room_materials_ready from the original image. Use room_material_classification or merge this capability into another task that already receives the original image. The result must contain concise non-empty English descriptions for walls, floor, and ceiling; do not copy example materials unless they match the image.
+- Provide contain_relations_ready when there is positive evidence for a concrete object inside an ordinary container or placed on/within any level of multi-level furniture. For bunk beds, bookcases/shelving units, multi-tier service carts, and multi-level racks, every occupied functional level including the top is contain; do not route the same pair to object-to-object attach. For ordinary single-level furniture, a plain exterior top remains attach. An empty anchor is not sufficient.
+- Support and contain tasks may emit concise image/mask-grounded object revision patches when a category or description error changes their relation verdict. Plan a separate object_revision_check only when a likely relation-changing error cannot be resolved inline by those tasks; never add it for cosmetic naming cleanup.
+- Plan object_physical_property_reasoning only when Agent judgement of floor-covering semantics is useful; it may be merged with object revision. Do not request upright from relation tasks because the independent Physics Sidecar owns it.
+- Plan ceiling_judgement only when concrete ceiling-mounted targets or ambiguous ceiling support need positive review.
+- Decide whether wall reasoning is needed from the original image and object list. Skip wall tasks when there is no concrete wall-mounted, wall-integrated, or physically against-wall candidate.
+- When wall reasoning is useful, select the minimum capabilities needed from wall_alignment_objects_ready, wall_object_hypotheses_ready, provisional_wall_visibility_ready, wall_visibility_semantics_ready, and wall_attach_decisions_ready.
+- Use wall_alignment_object_review, or the compatible wall_object_hypothesis capability, to identify only genuine wall-mounted, wall-embedded, against-wall, or wall-aligned objects. This stage uses original-image semantics and must not choose candidate ids, wall ids, coordinates, final visibility labels, or relations.
+- When wall attach is planned and isolated finite orientation panels are available, prefer merging provisional_wall_visibility_ready into the pre-orientation wall-alignment task. That compact capability marks candidate_id plus provisional_wall_id as visible/weak/inferred only; it does not choose the winning orientation or emit a final wall id. You may instead plan a separate provisional_wall_visibility_review when the scene warrants it, or omit it when no wall attach is needed.
+- Any task providing final wall_visibility_semantics_ready or wall_attach_decisions_ready must be downstream from wall_alignment_objects_ready so local code can apply the final OBB orientation first. A task providing provisional_wall_visibility_ready may run before or with alignment because local code binds only the selected candidate afterwards.
+- When wall attach review is needed, prefer one downstream wall_attach_candidate_recall task that provides both final wall_visibility_semantics_ready and wall_attach_decisions_ready. Local wall_attach_candidates_ready is generated after orientation selection and before that task, so this merge avoids an extra API call.
+- Use a separate wall_visibility_semantics_review or wall_final_adjudication only when final visibility/closure semantics are needed without wall attach, or when the source image genuinely requires a more detailed semantic audit. These tasks review already selected final walls and may not choose another candidate.
+- Do not plan local validation, reconciliation, coverage, merge, or final-schema tasks; deterministic code handles them.
+- Custom reasoning tasks are allowed when no registered capability expresses a necessary scene-specific judgement.
+
+Preserve autonomy over task grouping and DAG structure. Do not force a fixed roster. Independent tasks should use depends_on=[] so they can run in parallel. Add a dependency only when the downstream semantic judgement genuinely requires the upstream result.
+
+Return one JSON object with exactly these top-level fields:
+- plan_id
+- objective
+- object_anchor_routing
+- tasks
+- stop_if
+
+object_anchor_routing must contain one compact record for every supplied object:
+- object_id
+- anchor_mode: ceiling_only|wall_only|floor_wall|support_only|contain_candidate|uncertain
+- confidence: high|medium|low
+
+Each task must contain:
+- node_id: short stable identifier unique within this plan
+- canonical_task_kind
+- provides_capabilities: zero or more capability names; use multiple values when merging capabilities
+- depends_on: node_id values from this plan
+- reason: concise scene-specific reason for including the task
+- object_scope: object ids that deserve primary attention, or []
+- scene_focus: concise task-specific concern, or an empty string
+
+For a custom canonical_task_kind, also provide custom_instructions and expected_output_schema. For registered task kinds, do not repeat generic instructions, artifact lists, API roles, runner actions, runtime classes, image lists, or output schemas.
+
+Do not invent object ids. Do not output full relations, objects, wall coordinates, file paths, or evidence that is not in the payload. Use English for all generated task text.
+
diff --git a/vlm_agent/reconstruction/relations/prompts/multi_agent/wall_final_adjudication_task.txt b/vlm_agent/reconstruction/relations/prompts/multi_agent/wall_final_adjudication_task.txt
new file mode 100644
index 0000000..1603a8f
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/prompts/multi_agent/wall_final_adjudication_task.txt
@@ -0,0 +1,23 @@
+You are reviewing the already selected formal wall solution.
+
+Local geometry authority:
+- Local code has selected and applied one finite OBB orientation before this task.
+- Treat the supplied wall_001 through wall_004 and each geometry_key as immutable.
+- Do not choose another candidate or invent, edit, resize, shift, rotate, reorder, renumber, or replace coordinates, wall ids, angles, or methods.
+
+Review all four supplied wall references and return:
+- wall_id and its exact supplied geometry_key
+- original_image_visibility: visible|partially_visible|not_visible|inferred
+- weak_wall: true|false
+- closure_only: true|false
+- can_anchor_wall_attach: true|false
+- a short reason_code
+
+Wall attach semantics:
+- Visible or partially visible physical walls may anchor genuinely contacting objects.
+- Weak, inferred, closure-only, or not-visible walls must not anchor ordinary freestanding or floor-supported objects.
+- Being near a room boundary is not sufficient for wall attach.
+
+If all four references are structurally valid, return decision=verified_final_walls and exactly four wall_visibility_review records. If supplied references are missing or inconsistent, return decision=missing_or_invalid_formal_obb_wall_evidence with a concise reason. Do not propose replacement geometry.
+
+Return JSON only.
diff --git a/vlm_agent/reconstruction/relations/stage.py b/vlm_agent/reconstruction/relations/stage.py
new file mode 100644
index 0000000..cf91f28
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/stage.py
@@ -0,0 +1,1243 @@
+from __future__ import annotations
+
+import ast
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import uuid
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Mapping
+
+from .graph import RelationGraphView
+from ...artifacts import RunContract
+from ...config import verify_relation_runtime
+
+
+READY_STATUSES = frozenset({"sam3d_ready", "sam3d_ready_degraded"})
+OBJECT_ID_RE = re.compile(r"^object_\d{3}$")
+SAFE_RUN_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
+ENV_NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
+SECRET_NAME_RE = re.compile(
+ r"(?:KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL|AUTHORIZATION|BASE_URL)", re.I
+)
+
+RENDERER_FUNCTIONS: Mapping[str, tuple[str, ...]] = {
+ "compose_scene": ("layout_dir", "unposed_dir"),
+ "denormalize_point": ("point_normalized", "transform_info"),
+ "generate_2d_grid_on_plane": (
+ "posed_meshes",
+ "camera_direction",
+ "grid_size",
+ ),
+ "render_scene_with_grid": (
+ "meshes",
+ "camera_direction",
+ "radius",
+ "image_size",
+ ),
+ "save_grid_mapping": ("mapping", "output_path", "transform_info"),
+ "resolve_orthographic_camera_frame": (
+ "meshes",
+ "camera_direction",
+ "radius",
+ ),
+ "build_camera_pose": ("camera_direction", "radius"),
+ "get_camera_matrix": ("yfov_deg", "image_size"),
+ "project_3d_to_2d": (
+ "point_3d",
+ "camera_pose",
+ "camera_matrix",
+ "image_size",
+ ),
+}
+
+
+class RelationStageError(RuntimeError):
+ """Raised when relation input, publication or execution gates fail."""
+
+
+@dataclass(frozen=True)
+class RelationArtifacts:
+ objects_path: Path
+ relations_path: Path
+ schema_validation_path: Path
+ publication_manifest_path: Path
+ source_run_dir: Path
+ release_id: str
+ publication_id: str
+ metadata: Mapping[str, Any] = field(default_factory=dict)
+
+
+@dataclass(frozen=True)
+class CandidateRunResult:
+ candidate_dir: Path
+ command: tuple[str, ...]
+ returncode: int
+ stdout: str
+ stderr: str
+
+
+@dataclass(frozen=True)
+class CandidateValidationResult:
+ candidate_dir: Path
+ summary_path: Path
+ objects_path: Path
+ relations_path: Path
+ schema_validation_path: Path
+ validation_path: Path
+ object_ids: tuple[str, ...]
+ source_hashes: Mapping[str, str]
+ graph_audit: Mapping[str, Any]
+ runtime_audit: Mapping[str, Any]
+
+
+def _utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _read_json(path: Path, label: str) -> dict[str, Any]:
+ try:
+ value = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RelationStageError(f"cannot parse {label}: {path}") from exc
+ if not isinstance(value, dict):
+ raise RelationStageError(f"{label} must contain a JSON object: {path}")
+ return value
+
+
+def _read_relations(path: Path) -> list[dict[str, Any]]:
+ try:
+ value = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RelationStageError(f"cannot parse relations: {path}") from exc
+ if not isinstance(value, list) or not all(
+ isinstance(item, dict) for item in value
+ ):
+ raise RelationStageError("relations.json must contain an array of objects")
+ return value
+
+
+def _write_json(path: Path, value: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(
+ json.dumps(value, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+
+
+def _atomic_write_json(path: Path, value: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.parent / f".{path.name}.{uuid.uuid4().hex}.tmp"
+ _write_json(temporary, value)
+ os.replace(temporary, path)
+
+
+def _is_within(path: Path, root: Path) -> bool:
+ try:
+ path.relative_to(root)
+ return True
+ except ValueError:
+ return False
+
+
+def _resolve_confined(
+ raw_path: Any,
+ *,
+ base_dir: Path,
+ root: Path,
+ label: str,
+) -> Path:
+ if not isinstance(raw_path, str) or not raw_path.strip():
+ raise RelationStageError(f"{label} must be a non-empty path")
+ candidate = Path(raw_path).expanduser()
+ if not candidate.is_absolute():
+ candidate = base_dir / candidate
+ resolved = candidate.resolve()
+ if not _is_within(resolved, root):
+ raise RelationStageError(f"{label} escapes its trusted root: {resolved}")
+ return resolved
+
+
+def _object_ids(items: Any, label: str) -> tuple[str, ...]:
+ if not isinstance(items, list):
+ raise RelationStageError(f"{label} must be an array")
+ values: list[str] = []
+ for index, item in enumerate(items):
+ if not isinstance(item, dict):
+ raise RelationStageError(f"{label}[{index}] must be an object")
+ object_id = str(item.get("object_id") or item.get("id") or "")
+ if not OBJECT_ID_RE.fullmatch(object_id):
+ raise RelationStageError(
+ f"{label}[{index}] has invalid object ID: {object_id}"
+ )
+ values.append(object_id)
+ if len(values) != len(set(values)):
+ raise RelationStageError(f"{label} contains duplicate object IDs")
+ return tuple(values)
+
+
+def _relation_runtime_audit(path: Path) -> dict[str, Any]:
+ if not path.is_file() or path.is_symlink() or path.stat().st_size == 0:
+ raise RelationStageError(
+ f"relation execution summary is missing or unsafe: {path}"
+ )
+ summary = _read_json(path, "relation execution summary")
+ api_calls = summary.get("api_calls")
+ if (
+ summary.get("schema_version") != "relation_execution_summary"
+ or not isinstance(api_calls, int)
+ or isinstance(api_calls, bool)
+ or api_calls <= 0
+ or not isinstance(summary.get("stages"), dict)
+ ):
+ raise RelationStageError(
+ "relation execution summary has invalid execution closure"
+ )
+ return {
+ "runtime_summary_path": str(path),
+ "runtime_summary_sha256": _sha256(path),
+ "api_calls": api_calls,
+ }
+
+
+class RelationStageAdapter:
+ """SceneMosaic-facing gate for the verified relation runtime."""
+
+ def __init__(
+ self,
+ *,
+ scene_config_path: Path,
+ relation_package_root: Path,
+ renderer_script: Path,
+ ) -> None:
+ self.contract = RunContract.from_scene_config(scene_config_path)
+ self.package_root = Path(relation_package_root).expanduser().resolve()
+ self.renderer_script = Path(renderer_script).expanduser().resolve()
+
+ @property
+ def relations_root(self) -> Path:
+ return self.contract.relations_dir
+
+ def preflight(self) -> dict[str, Any]:
+ release = self.verify_runtime()
+ renderer = self.verify_renderer_contract()
+ inputs = self.validate_inputs()
+ return {
+ "release": release,
+ "renderer": renderer,
+ "inputs": inputs,
+ "api_calls": 0,
+ }
+
+ def verify_runtime(self) -> dict[str, Any]:
+ report = verify_relation_runtime(self.package_root)
+ if not report.get("valid"):
+ raise RelationStageError(
+ f"relation runtime verification failed: {report}"
+ )
+ release_id = report.get("release_id")
+ if (
+ not isinstance(release_id, str)
+ or not SAFE_RUN_ID_RE.fullmatch(release_id)
+ ):
+ raise RelationStageError("relation runtime release_id is unsafe")
+ return report
+
+ def verify_renderer_contract(self) -> dict[str, Any]:
+ if not self.renderer_script.is_file():
+ raise RelationStageError(
+ f"renderer script is missing: {self.renderer_script}"
+ )
+ try:
+ tree = ast.parse(
+ self.renderer_script.read_text(encoding="utf-8"),
+ filename=str(self.renderer_script),
+ )
+ except (OSError, SyntaxError) as exc:
+ raise RelationStageError(
+ f"cannot parse renderer script: {self.renderer_script}"
+ ) from exc
+ functions = {
+ node.name: tuple(
+ argument.arg
+ for argument in [*node.args.posonlyargs, *node.args.args]
+ )
+ for node in tree.body
+ if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
+ }
+ failures: list[str] = []
+ for name, required in RENDERER_FUNCTIONS.items():
+ actual = functions.get(name)
+ if actual is None:
+ failures.append(f"missing:{name}")
+ continue
+ if actual[: len(required)] != required:
+ failures.append(
+ f"signature:{name}:expected_prefix={required}:actual={actual}"
+ )
+ if failures:
+ raise RelationStageError(
+ f"renderer contract verification failed: {failures}"
+ )
+ return {
+ "renderer_path": str(self.renderer_script),
+ "function_count": len(RENDERER_FUNCTIONS),
+ "functions": {
+ name: list(functions[name]) for name in RENDERER_FUNCTIONS
+ },
+ "coordinate_system": "OpenGL_glTF_global",
+ "ground_plane": "XZ",
+ "valid": True,
+ }
+
+ def validate_inputs(self) -> dict[str, Any]:
+ run_root = self.contract.run_root
+ reviewed_path = self.contract.stage03_dir / "reviewed_objects.json"
+ quality_path = self.contract.stage03_dir / "quality_report.json"
+ adoption_path = self.contract.stage03_dir / "perception_adoption.json"
+ stage04_manifest_path = (
+ self.contract.stage04_dir / "sam3d_input_manifest.json"
+ )
+ stage05 = self.contract.stage05_dir
+ index_path = stage05 / "object_index_map.json"
+ run_info_path = stage05 / "sam3d_run_info.json"
+ required = (
+ reviewed_path,
+ quality_path,
+ adoption_path,
+ stage04_manifest_path,
+ index_path,
+ run_info_path,
+ )
+ missing = [str(path) for path in required if not path.is_file()]
+ if missing:
+ raise RelationStageError(
+ f"relation input gate is missing required files: {missing}"
+ )
+
+ adoption = _read_json(adoption_path, "perception adoption")
+ if (
+ adoption.get("delivery_status") not in READY_STATUSES
+ or adoption.get("sam3d_execution_allowed") is not True
+ or adoption.get("human_review_required") is not False
+ ):
+ raise RelationStageError(
+ "protected perception adoption is not accepted for SAM3D"
+ )
+ formal_artifacts = adoption.get("formal_artifacts")
+ if not isinstance(formal_artifacts, list):
+ raise RelationStageError(
+ "perception adoption has no formal artifact hashes"
+ )
+ for record in formal_artifacts:
+ if not isinstance(record, dict):
+ raise RelationStageError(
+ "perception adoption artifact record is invalid"
+ )
+ artifact = Path(str(record.get("path") or "")).expanduser().resolve()
+ if not _is_within(artifact, run_root):
+ raise RelationStageError(
+ f"perception artifact escapes run root: {artifact}"
+ )
+ if (
+ not artifact.is_file()
+ or _sha256(artifact) != record.get("sha256")
+ ):
+ raise RelationStageError(
+ f"perception artifact hash mismatch: {artifact}"
+ )
+
+ reviewed = _read_json(reviewed_path, "reviewed objects")
+ quality = _read_json(quality_path, "quality report")
+ stage04_manifest = _read_json(
+ stage04_manifest_path, "Stage 04 manifest"
+ )
+ index = _read_json(index_path, "Stage 05 object index")
+ run_info = _read_json(run_info_path, "Stage 05 run info")
+ if run_info.get("status") != "success":
+ raise RelationStageError(
+ f"Stage 05 is not successful: {run_info.get('status')}"
+ )
+ if (
+ stage04_manifest.get("delivery_status") not in READY_STATUSES
+ or stage04_manifest.get("sam3d_execution_allowed") is not True
+ ):
+ raise RelationStageError("Stage 04 delivery is not executable")
+
+ reviewed_ids = _object_ids(reviewed.get("objects"), "reviewed objects")
+ stage04_ids = _object_ids(
+ stage04_manifest.get("objects"), "Stage 04 objects"
+ )
+ stage05_ids = _object_ids(index.get("objects"), "Stage 05 objects")
+ expected_ids = tuple(
+ f"object_{index:03d}"
+ for index in range(1, len(stage04_ids) + 1)
+ )
+ if stage04_ids != expected_ids:
+ raise RelationStageError(
+ "Stage 04 IDs must be continuous object_001 through object_N"
+ )
+
+ layout_dir = stage05 / "layout_params"
+ unposed_dir = stage05 / "unposed_objects"
+ layout_ids = tuple(
+ sorted(path.stem for path in layout_dir.glob("object_*.json"))
+ )
+ unposed_ids = tuple(
+ sorted(path.stem for path in unposed_dir.glob("object_*.glb"))
+ )
+ if not (
+ reviewed_ids
+ == stage04_ids
+ == stage05_ids
+ == layout_ids
+ == unposed_ids
+ ):
+ raise RelationStageError(
+ "reviewed, Stage 04, Stage 05, layout and unposed object IDs "
+ "must be identical"
+ )
+
+ layout_hashes: dict[str, str] = {}
+ mesh_hashes: dict[str, str] = {}
+ for object_id in expected_ids:
+ layout_path = layout_dir / f"{object_id}.json"
+ mesh_path = unposed_dir / f"{object_id}.glb"
+ layout = _read_json(layout_path, f"layout {object_id}")
+ if (
+ layout.get("object_id") != object_id
+ or layout.get("sam3d_internal_name") != object_id
+ ):
+ raise RelationStageError(
+ f"layout identity mismatch for {object_id}"
+ )
+ for key in ("translation", "rotation", "scale"):
+ if not isinstance(layout.get(key), list) or not layout[key]:
+ raise RelationStageError(
+ f"layout {object_id} lacks valid {key}"
+ )
+ if (
+ not mesh_path.is_file()
+ or mesh_path.is_symlink()
+ or mesh_path.stat().st_size == 0
+ ):
+ raise RelationStageError(
+ f"unposed GLB must be a real non-empty file: {mesh_path}"
+ )
+ layout_hashes[object_id] = _sha256(layout_path)
+ mesh_hashes[object_id] = _sha256(mesh_path)
+
+ closure = adoption.get("object_id_closure")
+ if (
+ not isinstance(closure, dict)
+ or closure.get("closed") is not True
+ or tuple(closure.get("manifest", [])) != expected_ids
+ ):
+ raise RelationStageError(
+ "perception adoption object ID closure is invalid"
+ )
+ return {
+ "scene_name": self.contract.scene_name,
+ "run_root": str(run_root),
+ "object_ids": list(expected_ids),
+ "object_count": len(expected_ids),
+ "reviewed_objects_path": str(reviewed_path),
+ "reviewed_objects_sha256": _sha256(reviewed_path),
+ "quality_report_path": str(quality_path),
+ "quality_report_sha256": _sha256(quality_path),
+ "quality_report_keys": sorted(quality),
+ "stage04_manifest_path": str(stage04_manifest_path),
+ "stage04_manifest_sha256": _sha256(stage04_manifest_path),
+ "stage05_index_path": str(index_path),
+ "stage05_index_sha256": _sha256(index_path),
+ "layout_sha256": layout_hashes,
+ "unposed_glb_sha256": mesh_hashes,
+ "segmentation_accepted": True,
+ "sam3d_accepted": True,
+ "id_closure": True,
+ }
+
+ def reuse(self) -> RelationArtifacts:
+ preflight = self.preflight()
+ current_candidates = (
+ self.relations_root / "current.json",
+ self.relations_root / "published" / "current.json",
+ )
+ existing = [path for path in current_candidates if path.is_file()]
+ if not existing:
+ raise RelationStageError(
+ "an active Relation publication is not available"
+ )
+ if len(existing) > 1 and _sha256(existing[0]) != _sha256(existing[1]):
+ raise RelationStageError(
+ "relation current pointers disagree; refusing ambiguous reuse"
+ )
+ pointer_path = existing[0]
+ pointer = _read_json(pointer_path, "relation current pointer")
+ if pointer.get("active") is not True:
+ raise RelationStageError("relation current pointer is not active")
+
+ publication_manifest_path = _resolve_confined(
+ pointer.get("publication_manifest"),
+ base_dir=self.relations_root,
+ root=self.relations_root,
+ label="publication manifest",
+ )
+ artifacts = self._validate_publication(
+ publication_manifest_path,
+ tuple(preflight["inputs"]["object_ids"]),
+ {
+ key: preflight["inputs"][key]
+ for key in (
+ "reviewed_objects_sha256",
+ "quality_report_sha256",
+ "stage04_manifest_sha256",
+ "stage05_index_sha256",
+ )
+ },
+ )
+ metadata = dict(artifacts.metadata)
+ metadata.update(
+ {
+ "reused": True,
+ "api_calls": 0,
+ "preflight": preflight,
+ "current_pointer_path": str(pointer_path),
+ }
+ )
+ return RelationArtifacts(
+ objects_path=artifacts.objects_path,
+ relations_path=artifacts.relations_path,
+ schema_validation_path=artifacts.schema_validation_path,
+ publication_manifest_path=artifacts.publication_manifest_path,
+ source_run_dir=artifacts.source_run_dir,
+ release_id=artifacts.release_id,
+ publication_id=artifacts.publication_id,
+ metadata=metadata,
+ )
+
+ def _validate_publication(
+ self,
+ manifest_path: Path,
+ expected_ids: tuple[str, ...],
+ expected_input_hashes: Mapping[str, str],
+ ) -> RelationArtifacts:
+ publications_root = (self.relations_root / "publications").resolve()
+ if not _is_within(manifest_path, publications_root):
+ raise RelationStageError(
+ f"publication is outside immutable publication root: {manifest_path}"
+ )
+ manifest = _read_json(manifest_path, "publication manifest")
+ publication_id = str(manifest.get("publication_id") or "")
+ if not SAFE_RUN_ID_RE.fullmatch(publication_id):
+ raise RelationStageError("publication_id is missing or unsafe")
+ release_id = str(manifest.get("release_id") or "")
+ if not SAFE_RUN_ID_RE.fullmatch(release_id):
+ raise RelationStageError(
+ "publication release_id is missing or unsafe"
+ )
+ if manifest.get("input_object_ids") != list(expected_ids):
+ raise RelationStageError(
+ "publication input object IDs no longer match this run"
+ )
+ input_hashes = manifest.get("input_hashes")
+ if not isinstance(input_hashes, dict):
+ raise RelationStageError("publication lacks input hash bindings")
+ mismatched_inputs = {
+ key: (input_hashes.get(key), expected)
+ for key, expected in expected_input_hashes.items()
+ if input_hashes.get(key) != expected
+ }
+ if mismatched_inputs:
+ raise RelationStageError(
+ f"publication input hash binding mismatch: {mismatched_inputs}"
+ )
+
+ runtime_audit = manifest.get("runtime_audit")
+ benchmark_evidence = manifest.get("benchmark_evidence")
+ audit_sources = manifest.get("audit_sources")
+ if (
+ not isinstance(runtime_audit, dict)
+ or not isinstance(runtime_audit.get("api_calls"), int)
+ or runtime_audit.get("api_calls") <= 0
+ or not isinstance(
+ runtime_audit.get("runtime_summary_sha256"), str
+ )
+ or len(runtime_audit["runtime_summary_sha256"]) != 64
+ or not isinstance(benchmark_evidence, dict)
+ or benchmark_evidence.get("agent_api_calls")
+ != runtime_audit.get("api_calls")
+ or not isinstance(audit_sources, dict)
+ ):
+ raise RelationStageError(
+ "publication Relation execution evidence is missing or inconsistent"
+ )
+ runtime_summary_path = str(
+ runtime_audit.get("runtime_summary_path") or ""
+ )
+ runtime_source_record = audit_sources.get(
+ runtime_summary_path
+ )
+ if (
+ not isinstance(runtime_source_record, dict)
+ or runtime_source_record.get("sha256")
+ != runtime_audit["runtime_summary_sha256"]
+ ):
+ raise RelationStageError(
+ "publication Relation execution summary hash is not source-bound"
+ )
+
+ publication_dir = manifest_path.parent
+ required_names = (
+ "objects.json",
+ "relations.json",
+ "schema_validation.json",
+ "candidate_validation.json",
+ )
+ file_records = manifest.get("files")
+ if not isinstance(file_records, dict):
+ raise RelationStageError(
+ "publication manifest lacks file hash records"
+ )
+ paths: dict[str, Path] = {}
+ for name in required_names:
+ path = publication_dir / name
+ record = file_records.get(name)
+ if (
+ not path.is_file()
+ or not isinstance(record, dict)
+ or record.get("sha256") != _sha256(path)
+ or record.get("bytes") != path.stat().st_size
+ ):
+ raise RelationStageError(
+ f"publication file verification failed: {path}"
+ )
+ paths[name] = path
+
+ objects = _read_json(paths["objects.json"], "published objects")
+ relations = _read_relations(paths["relations.json"])
+ schema = _read_json(
+ paths["schema_validation.json"], "published schema validation"
+ )
+ if schema.get("valid") is not True:
+ raise RelationStageError("published relation Schema is invalid")
+ graph = self._validate_relation_payloads(
+ objects, relations, expected_ids
+ )
+
+ source_run_dir = Path(
+ str(manifest.get("source_run_dir") or publication_dir)
+ ).expanduser()
+ return RelationArtifacts(
+ objects_path=paths["objects.json"],
+ relations_path=paths["relations.json"],
+ schema_validation_path=paths["schema_validation.json"],
+ publication_manifest_path=manifest_path,
+ source_run_dir=source_run_dir,
+ release_id=release_id,
+ publication_id=publication_id,
+ metadata={
+ "publication_manifest_sha256": _sha256(manifest_path),
+ "candidate_validated": True,
+ "schema_valid": True,
+ "relation_graph": graph.audit_summary(),
+ "historical_api": runtime_audit,
+ "historical_api_calls": runtime_audit["api_calls"],
+ "api_calls_this_invocation": 0,
+ "per_object_physics_consumed": False,
+ },
+ )
+
+ @staticmethod
+ def _validate_relation_payloads(
+ objects: dict[str, Any],
+ relations: list[dict[str, Any]],
+ expected_ids: tuple[str, ...],
+ ) -> RelationGraphView:
+ walls = objects.get("walls")
+ room_materials = objects.get("room_materials")
+ object_items = objects.get("objects")
+ if not isinstance(walls, list) or not walls:
+ raise RelationStageError("objects.json must preserve walls")
+ if not isinstance(room_materials, dict) or any(
+ not isinstance(room_materials.get(key), str)
+ or not room_materials[key].strip()
+ for key in ("walls", "floor", "ceiling")
+ ):
+ raise RelationStageError(
+ "room_materials walls/floor/ceiling must be non-empty"
+ )
+ output_ids = _object_ids(object_items, "published objects")
+ if output_ids != expected_ids:
+ raise RelationStageError(
+ "published objects do not match reviewed and Stage 05 IDs"
+ )
+ physics_keys = {
+ "mass",
+ "linear_damping",
+ "angular_damping",
+ "static_friction",
+ "dynamic_friction",
+ }
+ for item in object_items:
+ physics = item.get("physics")
+ if not isinstance(physics, dict) or not physics_keys.issubset(
+ physics
+ ):
+ raise RelationStageError(
+ f"published object physics is incomplete: {item.get('id')}"
+ )
+
+ wall_ids = {
+ str(item.get("id"))
+ for item in walls
+ if isinstance(item, dict) and item.get("id")
+ }
+ allowed_anchors = set(expected_ids) | wall_ids | {
+ "floor",
+ "ceiling",
+ }
+ targets: set[str] = set()
+ for relation in relations:
+ parameters = relation.get("parameters", {})
+ anchor = str(parameters.get("object_anchor") or "")
+ target = str(parameters.get("object_target") or "")
+ if anchor not in allowed_anchors:
+ raise RelationStageError(
+ f"relation anchor is not published: {anchor}"
+ )
+ if target not in expected_ids:
+ raise RelationStageError(
+ f"relation target is not published: {target}"
+ )
+ targets.add(target)
+ if targets != set(expected_ids):
+ raise RelationStageError(
+ "every published object must be covered as a relation target"
+ )
+ return RelationGraphView.from_relations(relations)
+
+ def _candidate_path(self, candidate_dir: Path) -> Path:
+ candidate = Path(candidate_dir).expanduser().resolve()
+ benchmarks_root = self.contract.relation_candidates_dir.resolve()
+ if (
+ candidate.parent != benchmarks_root
+ or not SAFE_RUN_ID_RE.fullmatch(candidate.name)
+ ):
+ raise RelationStageError(
+ f"candidate is outside the direct benchmark root: {candidate}"
+ )
+ return candidate
+
+ def validate_candidate(
+ self,
+ candidate_dir: Path,
+ ) -> CandidateValidationResult:
+ """Validate a production-wrapper result before any human decision."""
+
+ preflight = self.preflight()
+ candidate = self._candidate_path(candidate_dir)
+ summary_path = candidate / "benchmark_summary.json"
+ final_dir = candidate / "async_agent_run" / "final_outputs"
+ objects_path = final_dir / "objects.json"
+ relations_path = final_dir / "relations.json"
+ schema_path = (
+ candidate
+ / "async_agent_run"
+ / "debug"
+ / "final_validation"
+ / "schema_validation.json"
+ )
+ runtime_summary_path = (
+ candidate
+ / "async_agent_run"
+ / "debug"
+ / "agents"
+ / "runtime_summary.json"
+ )
+ source_paths = {
+ "benchmark_summary.json": summary_path,
+ "objects.json": objects_path,
+ "relations.json": relations_path,
+ "schema_validation.json": schema_path,
+ "runtime_summary.json": runtime_summary_path,
+ }
+ missing = [
+ str(path)
+ for path in source_paths.values()
+ if not path.is_file()
+ ]
+ if missing:
+ raise RelationStageError(
+ f"candidate validation is missing files: {missing}"
+ )
+ unsafe = [
+ str(path)
+ for path in source_paths.values()
+ if path.is_symlink() or path.stat().st_size == 0
+ ]
+ if unsafe:
+ raise RelationStageError(
+ f"candidate files must be real and non-empty: {unsafe}"
+ )
+
+ summary = _read_json(summary_path, "benchmark summary")
+ if summary.get("schema_version") != "relation_async_benchmark":
+ raise RelationStageError("candidate benchmark schema is invalid")
+ if (
+ summary.get("scene_name") != self.contract.scene_name
+ or summary.get("benchmark_id") != candidate.name
+ ):
+ raise RelationStageError(
+ "candidate benchmark identity does not match this scene"
+ )
+ for key, expected in (
+ ("run_root", self.contract.run_root),
+ ("benchmark_root", candidate),
+ ):
+ raw_path = summary.get(key)
+ if (
+ not isinstance(raw_path, str)
+ or Path(raw_path).expanduser().resolve() != expected
+ ):
+ raise RelationStageError(
+ f"candidate benchmark {key} does not match: {raw_path}"
+ )
+ if summary.get("dry_run") is not False:
+ raise RelationStageError(
+ "dry-run relation benchmark cannot become a candidate"
+ )
+ if summary.get("benchmark_success") is not True:
+ raise RelationStageError(
+ "benchmark_summary does not report benchmark_success=true"
+ )
+
+ async_summary = summary.get("async")
+ if not isinstance(async_summary, dict):
+ raise RelationStageError("candidate lacks an async benchmark result")
+ commands = async_summary.get("commands")
+ required_commands = {"expected_only", "precompute", "runner"}
+ if (
+ not isinstance(commands, dict)
+ or not required_commands.issubset(commands)
+ or any(
+ not isinstance(command, dict)
+ or command.get("returncode") != 0
+ for command in commands.values()
+ )
+ ):
+ raise RelationStageError(
+ "benchmark summary does not show all successful async commands"
+ )
+ metrics = async_summary.get("metrics")
+ if (
+ not isinstance(metrics, dict)
+ or metrics.get("final_objects_exists") is not True
+ or metrics.get("final_relations_exists") is not True
+ ):
+ raise RelationStageError(
+ "async metrics do not confirm both final JSON outputs"
+ )
+ for key, expected in (
+ ("final_objects_path", objects_path),
+ ("final_relations_path", relations_path),
+ ):
+ raw_path = metrics.get(key)
+ if (
+ not isinstance(raw_path, str)
+ or Path(raw_path).expanduser().resolve() != expected
+ ):
+ raise RelationStageError(
+ f"async metric {key} does not match candidate outputs"
+ )
+
+ runtime_audit = _relation_runtime_audit(
+ runtime_summary_path
+ )
+ metric_api_calls = metrics.get("agent_api_calls")
+ if (
+ metric_api_calls is not None
+ and metric_api_calls != runtime_audit["api_calls"]
+ ):
+ raise RelationStageError(
+ "benchmark metrics and runtime summary api_calls disagree"
+ )
+
+ benchmark_evidence = {
+ "agent_api_calls": runtime_audit["api_calls"],
+ "command_returncodes": {
+ name: int(commands[name]["returncode"])
+ for name in sorted(required_commands)
+ },
+ "final_objects_path": str(objects_path),
+ "final_relations_path": str(relations_path),
+ }
+
+ objects = _read_json(objects_path, "candidate objects")
+ relations = _read_relations(relations_path)
+ schema = _read_json(schema_path, "candidate schema validation")
+ if schema.get("valid") is not True:
+ raise RelationStageError(
+ "schema-invalid candidate cannot be published"
+ )
+ object_ids = tuple(preflight["inputs"]["object_ids"])
+ graph = self._validate_relation_payloads(
+ objects,
+ relations,
+ object_ids,
+ )
+ source_hashes = {
+ name: _sha256(path) for name, path in source_paths.items()
+ }
+ validation_path = candidate / "scenemosaic_candidate_validation.json"
+ stable_payload = {
+ "schema_version": "scenemosaic_relation_candidate_validation",
+ "status": "candidate_valid",
+ "candidate_dir": str(candidate),
+ "release_id": preflight["release"]["release_id"],
+ "object_ids": list(object_ids),
+ "source_hashes": source_hashes,
+ "relation_graph": graph.audit_summary(),
+ "benchmark_evidence": benchmark_evidence,
+ "runtime_audit": runtime_audit,
+ "per_object_physics_consumed": False,
+ }
+ existing: dict[str, Any] | None = None
+ if validation_path.is_file():
+ existing = _read_json(
+ validation_path, "candidate validation record"
+ )
+ if not isinstance(existing, dict) or any(
+ existing.get(key) != value
+ for key, value in stable_payload.items()
+ ):
+ payload = dict(stable_payload)
+ payload["validated_at_utc"] = _utc_now()
+ _atomic_write_json(validation_path, payload)
+
+ return CandidateValidationResult(
+ candidate_dir=candidate,
+ summary_path=summary_path,
+ objects_path=objects_path,
+ relations_path=relations_path,
+ schema_validation_path=schema_path,
+ validation_path=validation_path,
+ object_ids=object_ids,
+ source_hashes=source_hashes,
+ graph_audit=graph.audit_summary(),
+ runtime_audit=runtime_audit,
+ )
+
+ def publish_candidate(
+ self,
+ *,
+ candidate_dir: Path,
+ publication_id: str,
+ ) -> RelationArtifacts:
+ preflight = self.preflight()
+ if not SAFE_RUN_ID_RE.fullmatch(publication_id):
+ raise RelationStageError("publication_id is unsafe")
+ validation = self.validate_candidate(candidate_dir)
+ candidate = validation.candidate_dir
+ summary_path = validation.summary_path
+ objects_path = validation.objects_path
+ relations_path = validation.relations_path
+ schema_path = validation.schema_validation_path
+ required = (
+ summary_path,
+ objects_path,
+ relations_path,
+ schema_path,
+ validation.validation_path,
+ )
+ missing = [str(path) for path in required if not path.is_file()]
+ if missing:
+ raise RelationStageError(
+ f"candidate publication is missing files: {missing}"
+ )
+
+ summary = _read_json(summary_path, "benchmark summary")
+ async_summary = summary.get("async")
+ commands = (
+ async_summary.get("commands")
+ if isinstance(async_summary, dict)
+ else None
+ )
+ if (
+ not isinstance(commands, dict)
+ or not commands
+ or any(
+ not isinstance(command, dict)
+ or command.get("returncode") != 0
+ for command in commands.values()
+ )
+ ):
+ raise RelationStageError(
+ "benchmark summary does not show successful async commands"
+ )
+
+ objects = _read_json(objects_path, "candidate objects")
+ relations = _read_relations(relations_path)
+ schema = _read_json(schema_path, "candidate schema validation")
+ if schema.get("valid") is not True:
+ raise RelationStageError(
+ "schema-invalid candidate cannot be published"
+ )
+ self._validate_relation_payloads(
+ objects,
+ relations,
+ tuple(preflight["inputs"]["object_ids"]),
+ )
+
+ publications_root = self.relations_root / "publications"
+ publications_root.mkdir(parents=True, exist_ok=True)
+ publication_dir = publications_root / publication_id
+ if publication_dir.exists():
+ raise RelationStageError(
+ f"publication already exists: {publication_dir}"
+ )
+ temporary = (
+ publications_root
+ / f".{publication_id}.{uuid.uuid4().hex}.tmp"
+ )
+ temporary.mkdir(parents=False, exist_ok=False)
+ try:
+ source_files = {
+ "objects.json": objects_path,
+ "relations.json": relations_path,
+ "schema_validation.json": schema_path,
+ "candidate_validation.json": validation.validation_path,
+ }
+ for name, source in source_files.items():
+ shutil.copy2(source, temporary / name)
+ file_records = {
+ name: {
+ "sha256": _sha256(temporary / name),
+ "bytes": (temporary / name).stat().st_size,
+ }
+ for name in source_files
+ }
+ for name in (
+ "objects.json",
+ "relations.json",
+ "schema_validation.json",
+ ):
+ if (
+ file_records[name]["sha256"]
+ != validation.source_hashes[name]
+ ):
+ raise RelationStageError(
+ f"candidate changed while publishing: {name}"
+ )
+ audit_sources: dict[str, dict[str, Any]] = {}
+ audit_candidates = (
+ summary_path,
+ candidate
+ / "async_agent_run"
+ / "debug"
+ / "agents"
+ / "runtime_summary.json",
+ candidate
+ / "async_agent_run"
+ / "debug"
+ / "agents"
+ / "final_process_report.json",
+ )
+ for source in audit_candidates:
+ if source.is_file():
+ audit_sources[str(source)] = {
+ "sha256": _sha256(source),
+ "bytes": source.stat().st_size,
+ }
+ benchmark_evidence = {
+ "agent_api_calls": validation.runtime_audit["api_calls"],
+ "command_returncodes": {
+ name: int(command["returncode"])
+ for name, command in sorted(commands.items())
+ },
+ "final_objects_path": str(validation.objects_path),
+ "final_relations_path": str(validation.relations_path),
+ }
+ manifest = {
+ "schema_version": "scenemosaic_relation_publication",
+ "publication_id": publication_id,
+ "release_id": preflight["release"]["release_id"],
+ "published_at_utc": _utc_now(),
+ "source_run_dir": str(candidate),
+ "files": file_records,
+ "input_object_ids": preflight["inputs"]["object_ids"],
+ "input_hashes": {
+ "reviewed_objects_sha256": preflight["inputs"][
+ "reviewed_objects_sha256"
+ ],
+ "quality_report_sha256": preflight["inputs"][
+ "quality_report_sha256"
+ ],
+ "stage04_manifest_sha256": preflight["inputs"][
+ "stage04_manifest_sha256"
+ ],
+ "stage05_index_sha256": preflight["inputs"][
+ "stage05_index_sha256"
+ ],
+ },
+ "audit_sources": audit_sources,
+ "benchmark_evidence": benchmark_evidence,
+ "runtime_audit": dict(validation.runtime_audit),
+ "candidate_validated": True,
+ "schema_valid": True,
+ "per_object_physics_consumed": False,
+ }
+ _write_json(temporary / "manifest.json", manifest)
+ os.replace(temporary, publication_dir)
+ except Exception:
+ if temporary.exists():
+ shutil.rmtree(temporary)
+ raise
+
+ pointer = {
+ "schema_version": "scenemosaic_relation_current",
+ "active": True,
+ "publication_id": publication_id,
+ "publication_manifest": str(
+ Path("publications") / publication_id / "manifest.json"
+ ),
+ "updated_at_utc": _utc_now(),
+ }
+ _atomic_write_json(self.relations_root / "current.json", pointer)
+ return self.reuse()
+
+ def run_candidate(
+ self,
+ *,
+ benchmark_id: str,
+ api_key_env: str,
+ precompute_python: Path,
+ runner_python: Path,
+ model: str = "gpt-5.5",
+ api_base: str | None = None,
+ environ: Mapping[str, str] | None = None,
+ ) -> CandidateRunResult:
+ self.preflight()
+ if not SAFE_RUN_ID_RE.fullmatch(benchmark_id):
+ raise RelationStageError("benchmark_id is unsafe")
+ if not ENV_NAME_RE.fullmatch(api_key_env):
+ raise RelationStageError("API key environment variable name is invalid")
+ if not isinstance(model, str) or model.strip().lower() != "gpt-5.5":
+ raise RelationStageError("SceneMosaic Relation requires model gpt-5.5")
+ model = "gpt-5.5"
+ source_env = os.environ if environ is None else environ
+ secret = source_env.get(api_key_env)
+ if not secret:
+ raise RelationStageError(
+ f"API key environment variable is not set: {api_key_env}"
+ )
+ precompute = Path(precompute_python).expanduser().resolve()
+ runner = Path(runner_python).expanduser().resolve()
+ if not precompute.is_file() or not runner.is_file():
+ raise RelationStageError(
+ "precompute_python and runner_python must both exist"
+ )
+ candidate_dir = self.contract.relation_candidates_dir / benchmark_id
+ if candidate_dir.exists():
+ raise RelationStageError(
+ f"benchmark directory already exists: {candidate_dir}"
+ )
+ wrapper = (
+ self.package_root
+ / "tools"
+ / "benchmark_relation_async_planning.py"
+ )
+ command = [
+ str(runner),
+ str(wrapper),
+ "--scene_name",
+ self.contract.scene_name,
+ "--run_root",
+ str(self.contract.run_root),
+ "--benchmark_id",
+ benchmark_id,
+ "--render_script",
+ str(self.renderer_script),
+ "--precompute_python",
+ str(precompute),
+ "--runner_python",
+ str(runner),
+ "--api_key_env",
+ api_key_env,
+ "--model",
+ model,
+ "--vision_model",
+ model,
+ "--wall_specialist_model",
+ model,
+ "--wall_specialist_vision_model",
+ model,
+ ]
+ if "--force" in command or "--force_regenerate" in command:
+ raise RelationStageError("destructive wrapper flags are forbidden")
+
+ child_env = {
+ name: value
+ for name, value in source_env.items()
+ if not SECRET_NAME_RE.search(name)
+ }
+ child_env[api_key_env] = secret
+ child_env["PYTHONDONTWRITEBYTECODE"] = "1"
+ child_env["PYTHONNOUSERSITE"] = "1"
+ resolved_base = (
+ api_base.strip()
+ if api_base and api_base.strip() and api_base.strip() != "None"
+ else source_env.get("OPEN_BASE_URL")
+ or source_env.get("OPENAI_BASE_URL")
+ or source_env.get("OPENAI_API_BASE")
+ )
+ if resolved_base:
+ child_env["OPEN_BASE_URL"] = resolved_base
+ secret_values = {
+ value
+ for name, value in source_env.items()
+ if value and SECRET_NAME_RE.search(name)
+ }
+ secret_values.update(value for value in (secret, resolved_base) if value)
+
+ def redact(value: str) -> str:
+ result = value
+ for secret_value in secret_values:
+ result = result.replace(secret_value, "<redacted>")
+ return result[: 2 * 1024 * 1024]
+
+ completed = subprocess.run(
+ command,
+ capture_output=True,
+ text=True,
+ check=False,
+ env=child_env,
+ )
+ redacted_stdout = redact(completed.stdout)
+ redacted_stderr = redact(completed.stderr)
+ if completed.returncode != 0:
+ raise RelationStageError(
+ "relation wrapper failed with return code "
+ f"{completed.returncode}; stdout={redacted_stdout[-2000:]}; "
+ f"stderr={redacted_stderr[-2000:]}"
+ )
+ return CandidateRunResult(
+ candidate_dir=candidate_dir,
+ command=tuple(command),
+ returncode=completed.returncode,
+ stdout=redacted_stdout,
+ stderr=redacted_stderr,
+ )
diff --git a/vlm_agent/reconstruction/relations/tools/__init__.py b/vlm_agent/reconstruction/relations/tools/__init__.py
new file mode 100644
index 0000000..aba86a2
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/__init__.py
@@ -0,0 +1 @@
+"""Tools for multi-agent scene delivery."""
diff --git a/vlm_agent/reconstruction/relations/tools/benchmark_relation_async_planning.py b/vlm_agent/reconstruction/relations/tools/benchmark_relation_async_planning.py
new file mode 100644
index 0000000..f1ce7c3
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/benchmark_relation_async_planning.py
@@ -0,0 +1,330 @@
+#!/usr/bin/env python3
+"""Run asynchronous skeleton manager planning with an isolated layout cache."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shlex
+import subprocess
+import sys
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+from typing import Any
+
+CODE_ROOT = Path(__file__).resolve().parents[4]
+if str(CODE_ROOT) not in sys.path:
+ sys.path.insert(0, str(CODE_ROOT))
+
+from vlm_agent.run_paths import ScenePaths
+
+RELATIONS_ROOT = Path(__file__).resolve().parents[1]
+PRECOMPUTE_SCRIPT = RELATIONS_ROOT / 'tools' / 'build_relation_layout_from_sam3d.py'
+RUNNER_SCRIPT = RELATIONS_ROOT / 'agents' / 'run_scene_delivery_multi_agent.py'
+DEFAULT_RENDER_SCRIPT = CODE_ROOT / 'tools' / 'render_scene_from_layout.py'
+DEFAULT_PRECOMPUTE_PYTHON = Path(os.environ.get('SCENEMOSAIC_RELATION_PRECOMPUTE_PYTHON', sys.executable))
+DEFAULT_BASE_MODEL = 'gpt-5.5'
+DEFAULT_WALL_SPECIALIST_MODEL = 'gpt-5.5'
+DEFAULT_WALL_SPECIALIST_TASK_KINDS = [
+ 'wall_alignment_object_review',
+ 'provisional_wall_visibility_review',
+ 'wall_visibility_semantics_review',
+ 'wall_final_adjudication',
+ 'wall_attach_candidate_recall',
+]
+
+
+def write_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with path.open('w', encoding='utf-8') as f:
+ json.dump(payload, f, ensure_ascii=False, indent=2)
+ f.write('\n')
+
+
+def read_json_if_exists(path: Path) -> Any | None:
+ if not path.exists():
+ return None
+ try:
+ return json.loads(path.read_text(encoding='utf-8'))
+ except Exception:
+ return None
+
+
+def run_logged(name: str, cmd: list[str], log_dir: Path, env: dict[str, str] | None = None) -> dict[str, Any]:
+ log_dir.mkdir(parents=True, exist_ok=True)
+ stdout_path = log_dir / f'{name}.stdout.log'
+ stderr_path = log_dir / f'{name}.stderr.log'
+ cmd_path = log_dir / f'{name}.cmd.txt'
+ cmd_path.write_text(shlex.join(cmd) + '\n', encoding='utf-8')
+ with stdout_path.open('w', encoding='utf-8') as out, stderr_path.open('w', encoding='utf-8') as err:
+ proc = subprocess.run(cmd, stdout=out, stderr=err, text=True, env=env)
+ return {
+ 'name': name,
+ 'cmd': cmd,
+ 'cmd_text': shlex.join(cmd),
+ 'returncode': proc.returncode,
+ 'stdout_path': str(stdout_path),
+ 'stderr_path': str(stderr_path),
+ 'cmd_path': str(cmd_path),
+ }
+
+
+def start_logged(name: str, cmd: list[str], log_dir: Path, env: dict[str, str] | None = None) -> tuple[subprocess.Popen[str], dict[str, Any]]:
+ log_dir.mkdir(parents=True, exist_ok=True)
+ stdout_path = log_dir / f'{name}.stdout.log'
+ stderr_path = log_dir / f'{name}.stderr.log'
+ cmd_path = log_dir / f'{name}.cmd.txt'
+ cmd_path.write_text(shlex.join(cmd) + '\n', encoding='utf-8')
+ out = stdout_path.open('w', encoding='utf-8')
+ err = stderr_path.open('w', encoding='utf-8')
+ proc = subprocess.Popen(cmd, stdout=out, stderr=err, text=True, env=env)
+ meta = {
+ 'name': name,
+ 'cmd': cmd,
+ 'cmd_text': shlex.join(cmd),
+ 'stdout_path': str(stdout_path),
+ 'stderr_path': str(stderr_path),
+ 'cmd_path': str(cmd_path),
+ '_stdout_handle': out,
+ '_stderr_handle': err,
+ }
+ return proc, meta
+
+
+def finish_logged(proc: subprocess.Popen[str], meta: dict[str, Any]) -> dict[str, Any]:
+ returncode = proc.wait()
+ meta['_stdout_handle'].close()
+ meta['_stderr_handle'].close()
+ result = {k: v for k, v in meta.items() if not k.startswith('_')}
+ result.update({'returncode': returncode})
+ return result
+
+
+def precompute_cmd(args: argparse.Namespace, output_dir: Path, *, expected_only: bool = False) -> list[str]:
+ cmd = [
+ str(args.precompute_python),
+ str(PRECOMPUTE_SCRIPT),
+ '--scene_name', args.scene_name,
+ '--run_root', str(args.run_root),
+ '--output_dir', str(output_dir),
+ '--grid_size', str(args.grid_size),
+ '--image_size', str(args.image_size),
+ ]
+ if expected_only:
+ cmd.append('--expected_only')
+ return cmd
+
+
+def runner_cmd(args: argparse.Namespace, output_dir: Path, relation_layout_dir: Path) -> list[str]:
+ cmd = [
+ str(args.runner_python),
+ str(RUNNER_SCRIPT),
+ '--scene_name', args.scene_name,
+ '--run_root', str(args.run_root),
+ '--model', args.model,
+ '--vision_model', args.vision_model,
+ ]
+ if args.wall_specialist_model:
+ cmd.extend(['--wall_specialist_model', args.wall_specialist_model])
+ if args.wall_specialist_vision_model:
+ cmd.extend(['--wall_specialist_vision_model', args.wall_specialist_vision_model])
+ if args.manager_model:
+ cmd.extend(["--manager_model", args.manager_model])
+ if args.manager_vision_model:
+ cmd.extend(["--manager_vision_model", args.manager_vision_model])
+ if args.manager_vision_image_detail:
+ cmd.extend(["--manager_vision_image_detail", args.manager_vision_image_detail])
+ if args.wall_specialist_task_kinds:
+ cmd.extend(['--wall_specialist_task_kinds', *args.wall_specialist_task_kinds])
+ cmd.extend([
+ '--render_script', str(args.render_script),
+ '--output_dir', str(output_dir),
+ '--relation_layout_dir', str(relation_layout_dir),
+ '--wall_candidate_mode', 'render_grid_only',
+ '--max_parallel_tasks', str(args.max_parallel_tasks),
+ '--relation_precompute_wait_seconds', str(args.relation_precompute_wait_seconds),
+ '--fixed_views_precompute_wait_seconds', '0.0',
+ ])
+ if args.api_base:
+ cmd.extend(['--api_base', args.api_base])
+ if args.api_key_env:
+ cmd.extend(['--api_key_env', args.api_key_env])
+ if args.vision_image_detail:
+ cmd.extend(['--vision_image_detail', args.vision_image_detail])
+ if args.physics_model:
+ cmd.extend(['--physics_model', args.physics_model])
+ if args.physics_include_original:
+ cmd.append('--physics_include_original')
+ if args.dry_run:
+ cmd.append('--dry_run')
+ return cmd
+
+
+def collect_run_metrics(agent_dir: Path, relation_layout_dir: Path) -> dict[str, Any]:
+ runtime = read_json_if_exists(agent_dir / 'debug' / 'agents' / 'runtime_summary.json')
+ rel_manifest = read_json_if_exists(relation_layout_dir / 'manifest.json')
+ expected = read_json_if_exists(relation_layout_dir / 'expected_manifest.json')
+ manager_plan = read_json_if_exists(agent_dir / 'manager_plans' / 'manager_plan_000.json')
+ skeleton_plan = read_json_if_exists(agent_dir / 'manager_plans' / 'manager_plan_skeleton_000.json')
+ compact_plan_path = agent_dir / 'manager_plans' / 'manager_plan_compact_000.json'
+ compilation_audit_path = agent_dir / 'manager_plans' / 'manager_plan_compilation_audit.json'
+ return {
+ 'agent_runtime_summary_path': str(agent_dir / 'debug' / 'agents' / 'runtime_summary.json'),
+ 'agent_api_calls': runtime.get('api_calls') if isinstance(runtime, dict) else None,
+ 'relation_layout_manifest_path': str(relation_layout_dir / 'manifest.json'),
+ 'relation_layout_stages': rel_manifest.get('stages') if isinstance(rel_manifest, dict) else None,
+ 'expected_manifest_status': expected.get('status') if isinstance(expected, dict) else None,
+ 'manager_planning_mode': manager_plan.get('manager_planning_mode') if isinstance(manager_plan, dict) else None,
+ 'manager_plan_format': manager_plan.get('task_model_routing', {}).get('manager_plan_format') if isinstance(manager_plan, dict) and isinstance(manager_plan.get('task_model_routing'), dict) else None,
+ 'manager_compact_plan_path': str(compact_plan_path),
+ 'manager_compact_plan_exists': compact_plan_path.exists(),
+ 'manager_compilation_audit_path': str(compilation_audit_path),
+ 'manager_compilation_audit_exists': compilation_audit_path.exists(),
+ 'task_model_routing': manager_plan.get('task_model_routing') if isinstance(manager_plan, dict) else None,
+ 'manager_task_count': len(manager_plan.get('tasks', [])) if isinstance(manager_plan, dict) and isinstance(manager_plan.get('tasks'), list) else None,
+ 'skeleton_task_count': len(skeleton_plan.get('tasks', [])) if isinstance(skeleton_plan, dict) and isinstance(skeleton_plan.get('tasks'), list) else None,
+ 'final_relations_path': str(agent_dir / 'final_outputs' / 'relations.json'),
+ 'final_relations_exists': (agent_dir / 'final_outputs' / 'relations.json').exists(),
+ 'final_objects_path': str(agent_dir / 'final_outputs' / 'objects.json'),
+ 'final_objects_exists': (agent_dir / 'final_outputs' / 'objects.json').exists(),
+ 'physics_sidecar_dir': str(agent_dir / 'physics_sidecar' / 'integrated_async'),
+ 'physics_sidecar_runtime': read_json_if_exists(agent_dir / 'physics_sidecar' / 'integrated_async' / 'runtime.json'),
+ 'physics_sidecar_integration_status': read_json_if_exists(agent_dir / 'physics_sidecar' / 'integrated_async' / 'integration_status.json'),
+ 'simulation_physics_audit': read_json_if_exists(agent_dir / 'debug' / 'objects' / 'simulation_physics_audit.json'),
+ }
+
+
+def write_report(path: Path, summary: dict[str, Any]) -> None:
+ async_ = summary.get('async', {})
+ lines = [
+ '# Relation Async Planning Benchmark',
+ '',
+ f"- scene_name: `{summary.get('scene_name')}`",
+ f"- benchmark_id: `{summary.get('benchmark_id')}`",
+ f"- dry_run: `{summary.get('dry_run')}`",
+ '',
+ '## Wall-clock Results',
+ '',
+ '| Branch | return codes |',
+ '|---|---|',
+ ]
+ commands = async_.get('commands', {})
+ rcs = []
+ for key, item in commands.items():
+ if isinstance(item, dict):
+ rcs.append(f"{key}={item.get('returncode')}")
+ lines.append(f"| `async` | {'; '.join(rcs)} |")
+ lines.extend(['', '## Notes', ''])
+ lines.append('Async writes `expected_manifest.json`, starts full preprocessing and the runner concurrently, then hydrates the manager DAG after artifacts appear. The runner also launches Physics Sidecar beside manager planning; relation tasks never wait for it.')
+ lines.extend(['', '## Output Directories', ''])
+ lines.append(f"- async relation_layout_dir: `{async_.get('relation_layout_dir')}`")
+ lines.append(f"- async agent_dir: `{async_.get('agent_dir')}`")
+ path.write_text('\n'.join(lines) + '\n', encoding='utf-8')
+
+
+def benchmark_has_failed_commands(summary: dict[str, Any]) -> bool:
+ branch = summary.get("async")
+ if not isinstance(branch, dict):
+ return True
+ commands = branch.get("commands", {})
+ if not isinstance(commands, dict):
+ return True
+ for command in commands.values():
+ if not isinstance(command, dict) or command.get("returncode") != 0:
+ return True
+ metrics = branch.get("metrics", {})
+ if isinstance(metrics, dict):
+ if not metrics.get("final_objects_exists") or not metrics.get("final_relations_exists"):
+ return True
+ return False
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description='Run skeleton_async manager planning with a clean relation layout directory.')
+ parser.add_argument('--scene_name', required=True)
+ parser.add_argument('--run_root', required=True, type=Path)
+ parser.add_argument('--benchmark_id', required=True)
+ parser.add_argument('--model', default=DEFAULT_BASE_MODEL)
+ parser.add_argument('--vision_model', default=DEFAULT_BASE_MODEL)
+ parser.add_argument('--manager_model', default=None)
+ parser.add_argument('--manager_vision_model', default=None)
+ parser.add_argument('--manager_vision_image_detail', choices=['low', 'high', 'auto'], default='low')
+ parser.add_argument('--wall_specialist_model', default=DEFAULT_WALL_SPECIALIST_MODEL)
+ parser.add_argument('--wall_specialist_vision_model', default=None)
+ parser.add_argument(
+ '--wall_specialist_task_kinds',
+ nargs='+',
+ default=DEFAULT_WALL_SPECIALIST_TASK_KINDS,
+ help='Canonical task kinds sent to the optional wall specialist; accepts space- or comma-separated values.',
+ )
+ parser.add_argument('--render_script', type=Path, default=DEFAULT_RENDER_SCRIPT)
+ parser.add_argument('--precompute_python', type=Path, default=DEFAULT_PRECOMPUTE_PYTHON if DEFAULT_PRECOMPUTE_PYTHON.exists() else Path(sys.executable))
+ parser.add_argument('--runner_python', type=Path, default=Path(sys.executable))
+ parser.add_argument('--max_parallel_tasks', type=int, default=4)
+ parser.add_argument('--relation_precompute_wait_seconds', type=float, default=900.0)
+ parser.add_argument('--grid_size', type=int, default=33)
+ parser.add_argument('--image_size', type=int, default=768)
+ parser.add_argument('--api_base', default=None)
+ parser.add_argument('--api_key_env', default='OPENAI_API_KEY')
+ parser.add_argument('--vision_image_detail', choices=['low', 'high', 'auto'], default=None)
+ parser.add_argument('--physics_model', default=None, help='Defaults to --model in the relation runner.')
+ parser.add_argument('--physics_include_original', action='store_true')
+ parser.add_argument('--dry_run', action='store_true')
+ args = parser.parse_args()
+
+ args.run_root = args.run_root.resolve()
+ args.render_script = args.render_script.resolve()
+ scene_paths = ScenePaths.from_scene_root(args.run_root)
+ bench_root = scene_paths.relation_candidates_dir / args.benchmark_id
+ if bench_root.exists():
+ raise SystemExit(f'benchmark directory already exists: {bench_root}. Choose a new --benchmark_id.')
+ bench_root.mkdir(parents=True, exist_ok=True)
+ log_dir = bench_root / 'logs'
+
+ env = os.environ.copy()
+
+ summary: dict[str, Any] = {
+ 'schema_version': 'relation_async_benchmark',
+ 'scene_name': args.scene_name,
+ 'benchmark_id': args.benchmark_id,
+ 'run_root': str(args.run_root),
+ 'benchmark_root': str(bench_root),
+ 'dry_run': bool(args.dry_run),
+ 'fixed_views_enabled': False,
+ }
+
+ async_layout = bench_root / 'async_relation_layout'
+ async_agent = bench_root / 'async_agent_run'
+ expected = run_logged('async_expected_only', precompute_cmd(args, async_layout, expected_only=True), log_dir, env)
+ async_commands: dict[str, Any] = {'expected_only': expected}
+ if expected['returncode'] == 0:
+ pre_proc, pre_meta = start_logged('async_precompute', precompute_cmd(args, async_layout), log_dir, env)
+ runner_proc, runner_meta = start_logged('async_runner', runner_cmd(args, async_agent, async_layout), log_dir, env)
+ # Reap both processes concurrently while preserving independent return codes.
+ with ThreadPoolExecutor(max_workers=2) as pool:
+ pre_future = pool.submit(finish_logged, pre_proc, pre_meta)
+ runner_future = pool.submit(finish_logged, runner_proc, runner_meta)
+ pre_result = pre_future.result()
+ runner_result = runner_future.result()
+ async_commands['precompute'] = pre_result
+ async_commands['runner'] = runner_result
+ else:
+ async_commands['precompute'] = {'name': 'async_precompute', 'returncode': None, 'skipped_reason': 'expected_only_failed'}
+ async_commands['runner'] = {'name': 'async_runner', 'returncode': None, 'skipped_reason': 'expected_only_failed'}
+ summary['async'] = {
+ 'relation_layout_dir': str(async_layout),
+ 'agent_dir': str(async_agent),
+ 'commands': async_commands,
+ 'metrics': collect_run_metrics(async_agent, async_layout),
+ }
+ summary['benchmark_success'] = not benchmark_has_failed_commands(summary)
+ write_json(bench_root / 'benchmark_summary.json', summary)
+ write_report(bench_root / 'benchmark_report.md', summary)
+ print(json.dumps(summary, indent=2, ensure_ascii=False))
+ if not summary['benchmark_success']:
+ raise SystemExit(2)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/vlm_agent/reconstruction/relations/tools/build_relation_layout_from_sam3d.py b/vlm_agent/reconstruction/relations/tools/build_relation_layout_from_sam3d.py
new file mode 100644
index 0000000..a986cb8
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/build_relation_layout_from_sam3d.py
@@ -0,0 +1,911 @@
+#!/usr/bin/env python3
+"""Build the relation-ready layout package directly from SAM3D outputs."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+import cv2
+import numpy as np
+
+ROOT = Path(__file__).resolve().parents[2]
+REPOSITORY_ROOT = Path(__file__).resolve().parents[4]
+if str(REPOSITORY_ROOT) not in sys.path:
+ sys.path.insert(0, str(REPOSITORY_ROOT))
+if str(ROOT) not in sys.path:
+ sys.path.insert(0, str(ROOT))
+POLY_TOOLS = REPOSITORY_ROOT / 'vlm_agent' / 'evolution' / 'tools'
+if str(POLY_TOOLS) not in sys.path:
+ sys.path.insert(0, str(POLY_TOOLS))
+
+from relations.agents.state import sorted_object_id_key, write_json, write_md # noqa: E402
+from relations.tools.build_scene_state import build_scene_state # noqa: E402
+from relations.tools.extract_wall_evidence import build_wall_evidence, write_wall_evidence # noqa: E402
+from relations.tools.generate_fixed_layout_views import ensure_fixed_layout_views # noqa: E402
+from relations.tools.wall_geometry import build_obb_walls # noqa: E402
+from vlm_agent.run_paths import ScenePaths # noqa: E402
+
+DEFAULT_RENDER_SCRIPT = POLY_TOOLS / 'render_scene_from_layout.py'
+PRIMARY_METHOD = 'obb_from_mesh_footprint_min_rect'
+FRONT_RENDER_DIRECTION = 'front'
+TOP_RENDER_DIRECTION = 'up'
+MAX_CONTAIN_CANDIDATE_PAIRS_PER_ANCHOR = 16
+MAX_HIGH_CONFIDENCE_CONTAIN_CANDIDATE_PAIRS_PER_ANCHOR = 32
+HIGH_CONFIDENCE_CONTAIN_CANDIDATE_THRESHOLD = 0.95
+
+CONTAINER_KEYWORDS = {
+ 'basket',
+ 'bin',
+ 'bookcase',
+ 'bookshelf',
+ 'bunk bed',
+ 'bowl',
+ 'box',
+ 'cabinet',
+ 'case',
+ 'closet',
+ 'crate',
+ 'cubby',
+ 'drawer',
+ 'display cabinet',
+ 'display case',
+ 'freezer',
+ 'fridge',
+ 'loft bed',
+ 'multi-tier',
+ 'open shelf',
+ 'rack',
+ 'refrigerator',
+ 'shelf',
+ 'shelving',
+ 'storage',
+ 'tiered',
+ 'tray',
+ 'wardrobe',
+}
+
+TOP_VIEW_CONTAINER_KEYWORDS = {
+ 'basket',
+ 'bin',
+ 'bowl',
+ 'box',
+ 'crate',
+ 'drawer',
+ 'open-top',
+ 'tray',
+}
+
+FRONT_VIEW_CONTAINER_KEYWORDS = {
+ 'bookcase',
+ 'bookshelf',
+ 'bunk bed',
+ 'cabinet',
+ 'case',
+ 'closet',
+ 'cubby',
+ 'display',
+ 'freezer',
+ 'fridge',
+ 'loft bed',
+ 'multi-tier',
+ 'open shelf',
+ 'rack',
+ 'refrigerator',
+ 'shelf',
+ 'shelving',
+ 'tiered',
+ 'wardrobe',
+}
+
+
+def load_json(path: Path) -> Any:
+ with path.open('r', encoding='utf-8') as f:
+ return json.load(f)
+
+
+def load_json_if_exists(path: Path) -> Any | None:
+ if not path.exists():
+ return None
+ return load_json(path)
+
+
+def load_layout_param_records(layout_params_dir: Path) -> list[dict[str, Any]]:
+ records: list[dict[str, Any]] = []
+ for path in sorted(layout_params_dir.glob('object_*.json'), key=lambda p: sorted_object_id_key(p.stem)):
+ data = load_json_if_exists(path)
+ if not isinstance(data, dict):
+ continue
+ object_id = str(data.get('object_id') or path.stem)
+ records.append(
+ {
+ 'object_id': object_id,
+ 'category': data.get('category') or 'unknown_object',
+ 'description': data.get('description') or data.get('category') or object_id,
+ 'source': data.get('source'),
+ 'layout_params_path': str(path),
+ 'source_mask_path': data.get('source_mask_path'),
+ 'vlm_bbox_2d_norm_xyxy': data.get('vlm_bbox_2d_norm_xyxy'),
+ 'sam3_mask_bbox_abs_xyxy': data.get('sam3_mask_bbox_abs_xyxy'),
+ 'posed_glb': data.get('posed_glb'),
+ 'unposed_glb': data.get('unposed_glb'),
+ 'raw_layout_params': data,
+ }
+ )
+ records.sort(key=lambda item: sorted_object_id_key(item['object_id']))
+ return records
+
+
+def artifact_contract(scene_name: str, run_root: Path, object_records: list[dict[str, Any]], output_dir: Path, include_fixed_views: bool = False) -> dict[str, Any]:
+ artifacts = {
+ 'original_image': {
+ 'status': 'available' if (run_root / 'input' / 'image.png').exists() else 'missing',
+ 'description': 'Original input image. The async manager may use this single image with the object list to decide whether a contain-relation task is needed before slower relation artifacts finish.',
+ 'path': str(run_root / 'input' / 'image.png'),
+ },
+ 'object_list': {
+ 'status': 'available',
+ 'description': 'SAM3D object ids, categories, short descriptions, original mask references, and image bboxes. This is enough for early manager DAG planning.',
+ 'path': str(output_dir / 'objects.json'),
+ },
+ 'object_footprints': {
+ 'status': 'pending',
+ 'description': 'Per-object OpenGL XZ footprint polygons and 3D bounding boxes computed from posed SAM3D mesh vertices. Useful for floor/support/object contact and wall proximity.',
+ 'path': str(output_dir / 'object_footprints.json'),
+ },
+ 'formal_obb_walls': {
+ 'status': 'pending',
+ 'description': 'Four screen-clockwise wall segments from the formal obb_from_mesh_footprint_min_rect method. This is the only formal wall coordinate source.',
+ 'path': str(output_dir / 'walls' / 'obb_walls.json'),
+ },
+ 'canonical_topdown': {
+ 'status': 'pending',
+ 'description': 'Top-down render and grid mapping generated from the same render grid used by formal wall candidates and downstream wall visual evidence.',
+ 'path': str(output_dir / 'render_grid_wall' / 'canonical_topdown.png'),
+ },
+ 'render_grid_affine': {
+ 'status': 'pending',
+ 'description': 'Pixel-to-OpenGL-XZ and OpenGL-XZ-to-pixel affine transforms for the canonical top-down image.',
+ 'path': str(output_dir / 'render_grid_wall' / 'render_grid_affine.json'),
+ },
+ 'wall_evidence': {
+ 'status': 'pending',
+ 'description': 'Structured wall evidence package consumed by controlled wall tasks in the multi-agent runner.',
+ 'path': str(output_dir / 'walls' / 'wall_evidence.json'),
+ },
+ 'contain_candidates': {
+ 'status': 'pending',
+ 'description': 'High-recall local hints for possible object-in-container relations. These candidates are not a closed set; the contain agent may add or reject pairs.',
+ 'path': str(output_dir / 'contain_candidates.json'),
+ },
+ 'contain_visual_evidence': {
+ 'status': 'pending',
+ 'description': 'Optional original-image overlay/crops for local contain candidates.',
+ 'path': str(output_dir / 'contain_visual_evidence' / 'contain_candidates_overlay.png'),
+ },
+ 'fixed_views': {
+ 'status': 'pending' if include_fixed_views else 'optional_disabled',
+ 'description': 'Optional fixed top/front/back/left/right scene views. Expensive and disabled by default; tasks must not require them unless include_fixed_views is enabled.',
+ 'path': str(output_dir / 'fixed_views' / 'views_manifest.json'),
+ },
+ 'scene_state': {
+ 'status': 'pending',
+ 'description': 'Relation-ready scene state assembled from this candidate-local layout package.',
+ 'path': str(output_dir / 'scene_state.json'),
+ },
+ }
+ return {
+ 'schema_version': 'relation_expected_manifest',
+ 'status': 'planning_ready',
+ 'scene_name': scene_name,
+ 'precompute_dir': str(output_dir),
+ 'created_by': 'vlm_agent.reconstruction.relations.tools.build_relation_layout_from_sam3d',
+ 'coordinate_frame': 'OpenGL original XZ; Y is vertical. Top-down PNG coordinates use x right and y downward.',
+ 'formal_wall_method': PRIMARY_METHOD,
+ 'object_count': len(object_records),
+ 'objects': [
+ {
+ 'object_id': item['object_id'],
+ 'category': item.get('category'),
+ 'description': item.get('description'),
+ 'source_mask_path': item.get('source_mask_path'),
+ 'vlm_bbox_2d_norm_xyxy': item.get('vlm_bbox_2d_norm_xyxy'),
+ }
+ for item in object_records
+ ],
+ 'artifacts': artifacts,
+ 'manager_planning_contract': {
+ 'manager_can_start_before_artifacts_are_available': True,
+ 'manager_should_plan_a_dag_skeleton_only': True,
+ 'manager_should_reference_required_artifacts_by_key': True,
+ 'hydration_step_binds_real_paths_later': True,
+ 'required_task_fields': ['task_id', 'task_type', 'canonical_task_kind', 'runner_action', 'runtime_class', 'depends_on', 'requires_artifacts'],
+ },
+ }
+
+
+def write_expected_manifest(scene_name: str, run_root: Path, output_dir: Path, object_records: list[dict[str, Any]], include_fixed_views: bool = False) -> dict[str, Any]:
+ manifest = artifact_contract(scene_name, run_root, object_records, output_dir, include_fixed_views=include_fixed_views)
+ write_json(output_dir / 'expected_manifest.json', manifest)
+ lines = [
+ f"- status: `{manifest['status']}`",
+ f"- scene_name: `{scene_name}`",
+ f"- object_count: `{manifest['object_count']}`",
+ f"- formal_wall_method: `{PRIMARY_METHOD}`",
+ '',
+ '## Artifact Contract',
+ ]
+ for key, value in manifest['artifacts'].items():
+ lines.append(f"- `{key}`: `{value['status']}` - {value['description']}")
+ write_md(output_dir / 'expected_manifest.md', 'Relation Expected Manifest', lines)
+ return manifest
+
+
+def build_scene_state_from_relation_layout(
+ scene_name: str,
+ run_root: Path,
+ render_script: Path,
+ relation_layout_dir: Path,
+) -> dict[str, Any]:
+ previous = os.environ.get("RELATION_LAYOUT_DIR")
+ os.environ["RELATION_LAYOUT_DIR"] = str(relation_layout_dir.resolve())
+ try:
+ return build_scene_state(scene_name, run_root, render_script)
+ finally:
+ if previous is None:
+ os.environ.pop("RELATION_LAYOUT_DIR", None)
+ else:
+ os.environ["RELATION_LAYOUT_DIR"] = previous
+
+
+def denormalized_vertices(mesh: Any, transform_info: dict[str, Any]) -> np.ndarray:
+ verts = np.asarray(getattr(mesh, 'vertices', []), dtype=np.float32)
+ if verts.size == 0:
+ return np.zeros((0, 3), dtype=np.float32)
+ from render_scene_from_layout import denormalize_point
+
+ return np.asarray([denormalize_point(vertex, transform_info) for vertex in verts], dtype=np.float32)
+
+
+def convex_hull_xz(points: np.ndarray) -> list[list[float]]:
+ if points.size == 0:
+ return []
+ array = np.asarray(points, dtype=np.float32).reshape(-1, 2)
+ if len(array) <= 2:
+ return [[float(point[0]), float(point[1])] for point in array]
+ hull = cv2.convexHull(array.reshape(-1, 1, 2), clockwise=False, returnPoints=True)
+ return [[float(point[0]), float(point[1])] for point in hull.reshape(-1, 2)]
+
+
+def mesh_footprint_payload(vertices: np.ndarray) -> dict[str, Any]:
+ if vertices.size == 0:
+ return {
+ 'mesh_footprint_hull_xz': [],
+ 'mesh_floor_footprint_hull_xz': [],
+ 'mesh_floor_band_max_y': None,
+ 'mesh_floor_footprint_source': 'missing_mesh_vertices',
+ }
+ mn = vertices.min(axis=0)
+ mx = vertices.max(axis=0)
+ height = float(max(mx[1] - mn[1], 0.0))
+ floor_band_height = min(height, max(0.012, height * 0.20))
+ floor_band_max_y = float(mn[1] + floor_band_height)
+ floor_vertices = vertices[vertices[:, 1] <= floor_band_max_y + 1e-8]
+ if len(floor_vertices) < 3:
+ floor_vertices = vertices
+ source = 'all_mesh_vertices_fallback'
+ else:
+ source = 'bottom_20_percent_mesh_vertices'
+ return {
+ 'mesh_footprint_hull_xz': convex_hull_xz(vertices[:, [0, 2]]),
+ 'mesh_floor_footprint_hull_xz': convex_hull_xz(floor_vertices[:, [0, 2]]),
+ 'mesh_floor_band_max_y': floor_band_max_y,
+ 'mesh_floor_footprint_source': source,
+ }
+
+
+def bbox_payload(vertices: np.ndarray) -> dict[str, Any]:
+ if vertices.size == 0:
+ return {'available': False, 'reason': 'mesh_vertices_missing'}
+ mn = vertices.min(axis=0)
+ mx = vertices.max(axis=0)
+ center = (mn + mx) / 2.0
+ extent = mx - mn
+ x0, z0 = float(mn[0]), float(mn[2])
+ x1, z1 = float(mx[0]), float(mx[2])
+ return {
+ 'available': True,
+ 'coordinate_frame': 'OpenGL original XZ; Y is vertical',
+ 'bbox_global': {
+ 'min': [float(v) for v in mn.tolist()],
+ 'max': [float(v) for v in mx.tolist()],
+ 'center': [float(v) for v in center.tolist()],
+ 'extent': [float(v) for v in extent.tolist()],
+ },
+ 'bbox_xz_min': [min(x0, x1), min(z0, z1)],
+ 'bbox_xz_max': [max(x0, x1), max(z0, z1)],
+ 'center_xz': [float(center[0]), float(center[2])],
+ 'footprint_polygon_xz': [[x0, z0], [x1, z0], [x1, z1], [x0, z1]],
+ 'extent_xz': [abs(x1 - x0), abs(z1 - z0)],
+ 'bottom_y': float(min(mn[1], mx[1])),
+ 'top_y': float(max(mn[1], mx[1])),
+ 'height_y': float(abs(mx[1] - mn[1])),
+ **mesh_footprint_payload(vertices),
+ }
+
+
+def build_objects_from_sam3d(run_root: Path, mapping_objects: list[dict[str, Any]] | None = None) -> tuple[dict[str, Any], dict[str, Any]]:
+ sam3d_output_dir = ScenePaths.from_scene_root(run_root).sam3d_output_dir
+ layout_dir = sam3d_output_dir / 'layout_params'
+ unposed_dir = sam3d_output_dir / 'unposed_objects'
+ records = {item['object_id']: item for item in load_layout_param_records(layout_dir)}
+ mapping_by_id = {str(item.get('object_id')): item for item in mapping_objects or [] if isinstance(item, dict) and item.get('object_id')}
+ from render_scene_from_layout import compose_scene
+
+ _, posed_meshes, layout_params, object_ids, transform_info = compose_scene(
+ layout_dir=layout_dir,
+ unposed_dir=unposed_dir,
+ requested_object_ids=None,
+ anchor_id=None,
+ )
+ objects: list[dict[str, Any]] = []
+ footprints: list[dict[str, Any]] = []
+ for mesh, object_id, layout in zip(posed_meshes, object_ids, layout_params):
+ object_id = str(object_id)
+ record = records.get(object_id, {'object_id': object_id})
+ vertices = denormalized_vertices(mesh, transform_info)
+ bbox = bbox_payload(vertices)
+ mapping = mapping_by_id.get(object_id, {})
+ obj = {
+ 'object_id': object_id,
+ 'category': record.get('category') or layout.get('category') or 'unknown_object',
+ 'description': record.get('description') or layout.get('description') or object_id,
+ 'source': record.get('source') or layout.get('source'),
+ 'image_bbox': record.get('vlm_bbox_2d_norm_xyxy'),
+ 'bbox_global': bbox.get('bbox_global'),
+ 'orientation_for_agent': {
+ 'render_direction': None,
+ 'rotation_degrees_topdown': mapping.get('rotation_degrees'),
+ 'source': 'render_scene_from_layout.save_grid_mapping.objects.rotation_degrees' if mapping else None,
+ },
+ 'sam3d_paths': {
+ 'layout_params_path': str(sam3d_output_dir / 'layout_params' / f'{object_id}.json'),
+ 'posed_glb_path': str(sam3d_output_dir / 'posed_objects' / f'{object_id}.glb'),
+ 'unposed_glb_path': str(sam3d_output_dir / 'unposed_objects' / f'{object_id}.glb'),
+ 'source_mask_path': record.get('source_mask_path'),
+ },
+ 'topdown_grid_stats': mapping,
+ }
+ fp = {
+ 'object_id': object_id,
+ 'category': obj['category'],
+ 'description': obj['description'],
+ 'source': 'sam3d_output mesh vertices via compose_scene',
+ **{k: v for k, v in bbox.items() if k != 'bbox_global'},
+ 'bbox_global': bbox.get('bbox_global'),
+ 'topdown_grid_stats': mapping,
+ }
+ objects.append(obj)
+ footprints.append(fp)
+ objects.sort(key=lambda item: sorted_object_id_key(item['object_id']))
+ footprints.sort(key=lambda item: sorted_object_id_key(item['object_id']))
+ layout_for_relation = {
+ 'schema_version': 'relation_layout_objects',
+ 'coordinate_system': 'OpenGL original XZ; Y is vertical',
+ 'source': 'sam3d_output',
+ 'objects': objects,
+ 'walls': [],
+ }
+ object_footprints = {
+ 'schema_version': 'object_footprints',
+ 'coordinate_frame': 'OpenGL original XZ; Y is vertical',
+ 'source': 'sam3d_output mesh vertices via compose_scene',
+ 'object_count': len(footprints),
+ 'items': footprints,
+ }
+ return layout_for_relation, object_footprints
+
+
+
+def _text_for_object(obj: dict[str, Any]) -> str:
+ return f"{obj.get('category', '')} {obj.get('description', '')}".lower()
+
+
+def _bbox3(obj: dict[str, Any]) -> dict[str, Any]:
+ bbox = obj.get('bbox_global')
+ return bbox if isinstance(bbox, dict) else {}
+
+
+def _bbox_center_inside(inner: dict[str, Any], outer: dict[str, Any], margin_ratio: float = 0.08) -> bool:
+ center = inner.get('center')
+ mn = outer.get('min')
+ mx = outer.get('max')
+ extent = outer.get('extent') or []
+ if not (isinstance(center, list) and isinstance(mn, list) and isinstance(mx, list) and len(center) >= 3 and len(mn) >= 3 and len(mx) >= 3):
+ return False
+ margins = []
+ for i in range(3):
+ if isinstance(extent, list) and len(extent) >= 3:
+ margins.append(max(abs(float(extent[i])) * margin_ratio, 0.03))
+ else:
+ margins.append(0.03)
+ return all(float(mn[i]) - margins[i] <= float(center[i]) <= float(mx[i]) + margins[i] for i in range(3))
+
+
+def _bbox_smaller_than(inner: dict[str, Any], outer: dict[str, Any]) -> bool:
+ ie = inner.get('extent')
+ oe = outer.get('extent')
+ if not (isinstance(ie, list) and isinstance(oe, list) and len(ie) >= 3 and len(oe) >= 3):
+ return False
+ # Containers are often thin in one dimension, so use volume/height plus at least
+ # one horizontal axis instead of requiring every axis to be smaller.
+ inner_volume = max(float(ie[0]), 0.001) * max(float(ie[1]), 0.001) * max(float(ie[2]), 0.001)
+ outer_volume = max(float(oe[0]), 0.001) * max(float(oe[1]), 0.001) * max(float(oe[2]), 0.001)
+ horizontal_smaller = float(ie[0]) <= float(oe[0]) * 1.15 or float(ie[2]) <= float(oe[2]) * 1.15
+ return inner_volume <= outer_volume * 0.75 and horizontal_smaller
+
+
+def _norm_image_bbox(obj: dict[str, Any]) -> list[float] | None:
+ bbox = obj.get('image_bbox')
+ if isinstance(bbox, list) and len(bbox) == 4:
+ return [float(v) for v in bbox]
+ if isinstance(bbox, dict):
+ raw = bbox.get('bbox_2d_norm_xyxy') or bbox.get('vlm_bbox_2d_norm_xyxy')
+ if isinstance(raw, list) and len(raw) == 4:
+ return [float(v) for v in raw]
+ return None
+
+
+def _image_center_inside(inner_obj: dict[str, Any], outer_obj: dict[str, Any], margin: float = 0.035) -> bool:
+ inner = _norm_image_bbox(inner_obj)
+ outer = _norm_image_bbox(outer_obj)
+ if not inner or not outer:
+ return False
+ cx = (inner[0] + inner[2]) / 2.0
+ cy = (inner[1] + inner[3]) / 2.0
+ return outer[0] - margin <= cx <= outer[2] + margin and outer[1] - margin <= cy <= outer[3] + margin
+
+
+def _is_container_like(obj: dict[str, Any]) -> bool:
+ text = _text_for_object(obj)
+ return any(keyword in text for keyword in CONTAINER_KEYWORDS)
+
+
+def _contain_render_direction(obj: dict[str, Any]) -> str:
+ text = _text_for_object(obj)
+ if any(keyword in text for keyword in TOP_VIEW_CONTAINER_KEYWORDS) and not any(keyword in text for keyword in FRONT_VIEW_CONTAINER_KEYWORDS):
+ return TOP_RENDER_DIRECTION
+ return FRONT_RENDER_DIRECTION
+
+
+def build_contain_candidates(layout_for_relation: dict[str, Any], object_footprints: dict[str, Any], run_root: Path, output_dir: Path) -> dict[str, Any]:
+ objects = [item for item in layout_for_relation.get('objects', []) if isinstance(item, dict)]
+ footprint_by_id = {str(item.get('object_id')): item for item in object_footprints.get('items', []) if isinstance(item, dict) and item.get('object_id')}
+ containers = []
+ pairs = []
+ for anchor in objects:
+ anchor_id = str(anchor.get('object_id'))
+ if not _is_container_like(anchor):
+ continue
+ direction = _contain_render_direction(anchor)
+ anchor_bbox = _bbox3(anchor)
+ anchor_fp = footprint_by_id.get(anchor_id, {})
+ container_record = {
+ 'object_id': anchor_id,
+ 'category': anchor.get('category'),
+ 'description': anchor.get('description'),
+ '渲染方向': direction,
+ 'render_direction': direction,
+ 'source': 'local_container_keyword_scan',
+ 'candidate_is_hint_only': True,
+ }
+ containers.append(container_record)
+ for target in objects:
+ target_id = str(target.get('object_id'))
+ if target_id == anchor_id:
+ continue
+ target_bbox = _bbox3(target)
+ evidence = []
+ score = 0.0
+ if _image_center_inside(target, anchor):
+ evidence.append('target_image_bbox_center_inside_anchor_image_bbox')
+ score += 0.45
+ if _bbox_center_inside(target_bbox, anchor_bbox):
+ evidence.append('target_3d_bbox_center_inside_anchor_bbox')
+ score += 0.35
+ if _bbox_smaller_than(target_bbox, anchor_bbox):
+ evidence.append('target_bbox_smaller_than_anchor_bbox')
+ score += 0.20
+ target_text = _text_for_object(target)
+ if any(token in target_text for token in ['book', 'box', 'figure', 'figurine', 'toy', 'decor', 'vase', 'bottle', 'plate', 'item', 'object']):
+ evidence.append('target_semantics_are_small_or_storable')
+ score += 0.10
+ # High-recall gate: accept if either image or 3D center suggests inclusion,
+ # and the target is not bigger than the container by volume.
+ if score < 0.45:
+ continue
+ pairs.append(
+ {
+ 'relation': 'contain',
+ 'parameters': {'object_anchor': anchor_id, 'object_target': target_id},
+ 'decision': 'candidate',
+ 'confidence_hint': round(min(score, 0.99), 3),
+ 'candidate_is_hint_only': True,
+ 'agent_may_add_or_reject': True,
+ 'anchor_render_direction': direction,
+ 'evidence': evidence,
+ 'anchor': {
+ 'category': anchor.get('category'),
+ 'description': anchor.get('description'),
+ 'bbox_global': anchor.get('bbox_global'),
+ 'image_bbox': anchor.get('image_bbox'),
+ 'footprint_summary': {
+ 'center_xz': anchor_fp.get('center_xz'),
+ 'extent_xz': anchor_fp.get('extent_xz'),
+ },
+ },
+ 'target': {
+ 'category': target.get('category'),
+ 'description': target.get('description'),
+ 'bbox_global': target.get('bbox_global'),
+ 'image_bbox': target.get('image_bbox'),
+ },
+ }
+ )
+ # Keep the normal prompt budget bounded, but do not truncate tied
+ # high-confidence occupants of one large multi-level anchor.
+ raw_pair_count = len(pairs)
+ pairs_by_anchor: dict[str, list[dict[str, Any]]] = {}
+ for pair in pairs:
+ anchor_id = str(pair['parameters']['object_anchor'])
+ pairs_by_anchor.setdefault(anchor_id, []).append(pair)
+ retained_pairs: list[dict[str, Any]] = []
+ truncated_anchor_pairs: dict[str, int] = {}
+ high_confidence_overflow_retained: dict[str, int] = {}
+ for anchor_id, anchor_pairs in pairs_by_anchor.items():
+ ranked_pairs = sorted(
+ anchor_pairs,
+ key=lambda item: (
+ -float(item.get('confidence_hint', 0.0)),
+ sorted_object_id_key(item['parameters']['object_target']),
+ ),
+ )
+ retained_for_anchor = ranked_pairs[:MAX_CONTAIN_CANDIDATE_PAIRS_PER_ANCHOR]
+ overflow_budget = max(
+ 0,
+ MAX_HIGH_CONFIDENCE_CONTAIN_CANDIDATE_PAIRS_PER_ANCHOR - len(retained_for_anchor),
+ )
+ high_confidence_overflow = [
+ item
+ for item in ranked_pairs[MAX_CONTAIN_CANDIDATE_PAIRS_PER_ANCHOR:]
+ if float(item.get('confidence_hint', 0.0)) >= HIGH_CONFIDENCE_CONTAIN_CANDIDATE_THRESHOLD
+ ][:overflow_budget]
+ retained_for_anchor.extend(high_confidence_overflow)
+ retained_pairs.extend(retained_for_anchor)
+ if high_confidence_overflow:
+ high_confidence_overflow_retained[anchor_id] = len(high_confidence_overflow)
+ omitted_count = len(ranked_pairs) - len(retained_for_anchor)
+ if omitted_count > 0:
+ truncated_anchor_pairs[anchor_id] = omitted_count
+ pairs = retained_pairs
+
+ overlay = write_contain_candidate_overlay(run_root, output_dir, objects, containers, pairs)
+ return {
+ 'schema_version': 'contain_candidates',
+ 'status': 'available',
+ 'source': 'relation_layout local high-recall container scan',
+ 'policy': {
+ 'candidate_set_is_closed': False,
+ 'agent_must_check_for_missing_contain_relations': True,
+ 'same_anchor_target_pair_attach_and_contain_are_mutually_exclusive': True,
+ 'soft_max_candidate_pairs_per_anchor': MAX_CONTAIN_CANDIDATE_PAIRS_PER_ANCHOR,
+ 'high_confidence_overflow_threshold': HIGH_CONFIDENCE_CONTAIN_CANDIDATE_THRESHOLD,
+ 'hard_max_candidate_pairs_per_anchor': MAX_HIGH_CONFIDENCE_CONTAIN_CANDIDATE_PAIRS_PER_ANCHOR,
+ 'contain_definition': 'Target is inside an ordinary container, or is placed on/within any functional level of multi-level furniture such as a bunk bed, bookcase/shelving unit, multi-tier service cart, or rack. Every multi-level tier including the top is contain. A plain exterior top is attach only for an ordinary single-level anchor.',
+ },
+ 'possible_containers': containers,
+ 'candidate_pairs': sorted(pairs, key=lambda item: (sorted_object_id_key(item['parameters']['object_anchor']), sorted_object_id_key(item['parameters']['object_target']))),
+ 'visual_evidence': overlay,
+ 'summary': {
+ 'possible_container_count': len(containers),
+ 'raw_candidate_pair_count': raw_pair_count,
+ 'candidate_pair_count': len(pairs),
+ 'high_confidence_overflow_retained': high_confidence_overflow_retained,
+ 'truncated_anchor_pairs': truncated_anchor_pairs,
+ },
+ }
+
+
+def write_contain_candidate_overlay(run_root: Path, output_dir: Path, objects: list[dict[str, Any]], containers: list[dict[str, Any]], pairs: list[dict[str, Any]]) -> dict[str, Any]:
+ image_path = run_root / 'input' / 'image.png'
+ visual_dir = output_dir / 'contain_visual_evidence'
+ visual_dir.mkdir(parents=True, exist_ok=True)
+ if not image_path.exists():
+ return {'status': 'missing_original_image', 'original_image_path': str(image_path)}
+ try:
+ from PIL import Image, ImageDraw, ImageFont
+ except Exception as exc:
+ return {'status': 'skipped', 'reason': f'PIL unavailable: {exc}', 'original_image_path': str(image_path)}
+ object_by_id = {str(item.get('object_id')): item for item in objects if item.get('object_id')}
+ container_ids = {item['object_id'] for item in containers}
+ target_ids = {item['parameters']['object_target'] for item in pairs}
+ image = Image.open(image_path).convert('RGB')
+ draw = ImageDraw.Draw(image)
+ width, height = image.size
+ font = ImageFont.load_default()
+
+ def abs_box(obj: dict[str, Any]) -> tuple[int, int, int, int] | None:
+ bbox = _norm_image_bbox(obj)
+ if not bbox:
+ return None
+ x0 = int(round(bbox[0] * width))
+ y0 = int(round(bbox[1] * height))
+ x1 = int(round(bbox[2] * width))
+ y1 = int(round(bbox[3] * height))
+ return max(0, x0), max(0, y0), min(width - 1, x1), min(height - 1, y1)
+
+ for object_id in sorted(container_ids, key=sorted_object_id_key):
+ box = abs_box(object_by_id.get(object_id, {}))
+ if not box:
+ continue
+ draw.rectangle(box, outline=(255, 90, 40), width=4)
+ draw.text((box[0] + 4, box[1] + 4), f'{object_id} container', fill=(255, 90, 40), font=font)
+ for object_id in sorted(target_ids, key=sorted_object_id_key):
+ box = abs_box(object_by_id.get(object_id, {}))
+ if not box:
+ continue
+ draw.rectangle(box, outline=(60, 210, 255), width=3)
+ draw.text((box[0] + 3, max(0, box[1] - 12)), object_id, fill=(60, 210, 255), font=font)
+ overlay_path = visual_dir / 'contain_candidates_overlay.png'
+ image.save(overlay_path)
+ return {
+ 'status': 'available',
+ 'original_image_path': str(image_path),
+ 'overlay_path': str(overlay_path),
+ 'container_count': len(container_ids),
+ 'target_count': len(target_ids),
+ }
+
+def load_mapping_objects(mapping_path: Path) -> list[dict[str, Any]]:
+ data = load_json_if_exists(mapping_path)
+ if isinstance(data, dict) and isinstance(data.get('objects'), list):
+ return [item for item in data['objects'] if isinstance(item, dict)]
+ return []
+
+
+def update_expected_status(expected: dict[str, Any], outputs: dict[str, Any], *, fixed_views_status: str) -> dict[str, Any]:
+ expected = json.loads(json.dumps(expected))
+ artifacts = expected.get('artifacts', {}) if isinstance(expected.get('artifacts'), dict) else {}
+ path_by_key = {
+ 'object_list': outputs.get('layout_for_relation_path'),
+ 'object_footprints': outputs.get('object_footprints_path'),
+ 'formal_obb_walls': outputs.get('obb_walls_path'),
+ 'canonical_topdown': outputs.get('canonical_topdown_image_path'),
+ 'render_grid_affine': outputs.get('render_grid_affine_path'),
+ 'wall_evidence': outputs.get('wall_evidence_path'),
+ 'contain_candidates': outputs.get('contain_candidates_path'),
+ 'contain_visual_evidence': outputs.get('contain_visual_evidence_path'),
+ 'fixed_views': outputs.get('fixed_views_manifest_path'),
+ 'scene_state': outputs.get('scene_state_path'),
+ }
+ for key, artifact in artifacts.items():
+ path = path_by_key.get(key) or artifact.get('path')
+ artifact['path'] = path
+ if key == 'fixed_views' and fixed_views_status == 'skipped':
+ artifact['status'] = 'skipped'
+ elif path and Path(str(path)).exists():
+ artifact['status'] = 'available'
+ else:
+ artifact['status'] = 'pending'
+ expected['status'] = 'available'
+ expected['artifacts'] = artifacts
+ return expected
+
+
+def write_manifest_md(path: Path, manifest: dict[str, Any]) -> None:
+ lines = [
+ f"- status: `{manifest.get('status')}`",
+ f"- scene_name: `{manifest.get('scene_name')}`",
+ f"- precompute_dir: `{manifest.get('precompute_dir')}`",
+ f"- formal_wall_method: `{manifest.get('formal_wall_method')}`",
+ '',
+ '## Stages',
+ ]
+ for key, value in (manifest.get('stages') or {}).items():
+ lines.append(f"- `{key}`: status=`{value.get('status')}`")
+ lines.extend(['', '## Outputs'])
+ for key, value in sorted((manifest.get('outputs') or {}).items()):
+ lines.append(f"- `{key}`: `{value}`")
+ write_md(path.with_suffix('.md'), 'Relation Layout Manifest', lines)
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description='Build a candidate-local relation layout from the formal SAM3D output.')
+ parser.add_argument('--scene_name', required=True)
+ parser.add_argument('--run_root', required=True, type=Path)
+ parser.add_argument('--render_script', type=Path, default=DEFAULT_RENDER_SCRIPT)
+ parser.add_argument('--output_dir', type=Path, default=None, help='Default: <run_root>/reconstruction/relations/relation_layout')
+ parser.add_argument('--grid_size', type=int, default=33)
+ parser.add_argument('--image_size', type=int, default=768)
+ parser.add_argument('--include_fixed_views', action='store_true', help='Generate optional top/front/back/left/right views. Disabled by default.')
+ parser.add_argument('--expected_only', action='store_true', help='Write only expected_manifest.json for early manager planning.')
+ parser.add_argument('--dry_run', action='store_true')
+ args = parser.parse_args()
+
+ run_root = args.run_root.resolve()
+ scene_paths = ScenePaths.from_scene_root(run_root)
+ output_dir = (
+ args.output_dir or (scene_paths.relations_dir / 'relation_layout')
+ ).resolve()
+ render_grid_dir = output_dir / 'render_grid_wall'
+ walls_dir = output_dir / 'walls'
+ fixed_views_dir = output_dir / 'fixed_views'
+ for directory in [output_dir, render_grid_dir, walls_dir, fixed_views_dir]:
+ directory.mkdir(parents=True, exist_ok=True)
+
+ stages: dict[str, dict[str, Any]] = {}
+ layout_records = load_layout_param_records(scene_paths.sam3d_output_dir / 'layout_params')
+ expected = write_expected_manifest(args.scene_name, run_root, output_dir, layout_records, include_fixed_views=bool(args.include_fixed_views))
+ if args.expected_only:
+ print(json.dumps(expected, indent=2, ensure_ascii=False))
+ return
+
+ from relations.tools.render_grid_wall_precompute import run_precompute
+
+ render_manifest = run_precompute(
+ scene_name=args.scene_name,
+ run_root=run_root,
+ output_dir=render_grid_dir,
+ grid_size=args.grid_size,
+ image_size=args.image_size,
+ )
+ stages['render_grid_wall_precompute'] = {
+ 'status': render_manifest.get('status'),
+ 'reused_existing': bool(render_manifest.get('reused_existing')),
+ }
+
+ mapping_path = Path(str(render_manifest.get('canonical_mapping_path') or render_grid_dir / 'canonical_topdown_grid_mapping.json'))
+ mapping_objects = load_mapping_objects(mapping_path)
+
+ layout_for_relation, object_footprints = build_objects_from_sam3d(run_root, mapping_objects)
+ layout_path = output_dir / 'layout_for_relation.json'
+ objects_path = output_dir / 'objects.json'
+ footprints_path = output_dir / 'object_footprints.json'
+ write_json(layout_path, layout_for_relation)
+ write_json(objects_path, {'schema_version': 'relation_objects', 'objects': layout_for_relation['objects']})
+ write_json(footprints_path, object_footprints)
+ stages['object_geometry_from_sam3d'] = {
+ 'status': 'available',
+ 'object_count': object_footprints.get('object_count'),
+ }
+
+ contain_candidates = build_contain_candidates(layout_for_relation, object_footprints, run_root, output_dir)
+ contain_candidates_path = output_dir / 'contain_candidates.json'
+ write_json(contain_candidates_path, contain_candidates)
+ stages['contain_candidates'] = {
+ 'status': contain_candidates.get('status'),
+ 'possible_container_count': contain_candidates.get('summary', {}).get('possible_container_count'),
+ 'candidate_pair_count': contain_candidates.get('summary', {}).get('candidate_pair_count'),
+ }
+
+ method_candidates_path = Path(str(render_manifest.get('method_candidates_path') or render_grid_dir / 'method_candidates.json'))
+ affine_path = Path(str(render_manifest.get('render_grid_affine_path') or render_grid_dir / 'render_grid_affine.json'))
+ method_candidates = load_json(method_candidates_path)
+ render_grid_affine = load_json(affine_path)
+ obb_walls = build_obb_walls(method_candidates, render_grid_affine)
+ obb_walls_path = walls_dir / 'obb_walls.json'
+ write_json(obb_walls_path, obb_walls)
+ write_json(walls_dir / 'walls.json', {'schema_version': 'relation_walls', 'walls': obb_walls.get('walls', []), 'source': PRIMARY_METHOD})
+
+ relation_outputs_partial = {
+ 'layout_for_relation_path': str(layout_path),
+ 'objects_path': str(objects_path),
+ 'object_footprints_path': str(footprints_path),
+ 'obb_walls_path': str(obb_walls_path),
+ 'method_candidates_path': str(method_candidates_path),
+ 'render_grid_wall_precompute_dir': str(render_grid_dir),
+ 'render_grid_affine_path': str(affine_path),
+ 'canonical_topdown_image_path': render_manifest.get('canonical_topdown_image_path'),
+ 'canonical_grid_mapping_path': render_manifest.get('canonical_mapping_path'),
+ 'candidate_overlay_path': render_manifest.get('candidate_overlay_path'),
+ 'primary_candidate_overlay_path': render_manifest.get('primary_candidate_overlay_path'),
+ 'fixed_views_dir': str(fixed_views_dir),
+ 'fixed_views_manifest_path': str(fixed_views_dir / 'views_manifest.json'),
+ 'contain_candidates_path': str(contain_candidates_path),
+ 'contain_visual_evidence_path': str(output_dir / 'contain_visual_evidence' / 'contain_candidates_overlay.png'),
+ }
+ early_manifest = {
+ 'schema_version': 'relation_layout_manifest',
+ 'status': 'in_progress',
+ 'scene_name': args.scene_name,
+ 'run_root': str(run_root),
+ 'precompute_dir': str(output_dir),
+ 'created_by': 'vlm_agent.reconstruction.relations.tools.build_relation_layout_from_sam3d',
+ 'coordinate_frame': 'OpenGL original XZ; Y is vertical',
+ 'formal_wall_method': PRIMARY_METHOD,
+ 'outputs': relation_outputs_partial,
+ 'stages': stages,
+ }
+ write_json(output_dir / 'manifest.json', early_manifest)
+
+ scene_state = build_scene_state_from_relation_layout(
+ args.scene_name,
+ run_root,
+ args.render_script.resolve(),
+ output_dir,
+ )
+ fixed_views_result: dict[str, Any]
+ if not args.include_fixed_views:
+ fixed_views_result = {'status': 'skipped', 'reason': 'fixed views disabled by default', 'views_dir': str(fixed_views_dir), 'manifest_path': str(fixed_views_dir / 'views_manifest.json'), 'views': []}
+ stages['fixed_layout_views'] = {'status': 'skipped'}
+ else:
+ fixed_views_result = ensure_fixed_layout_views(scene_state, fixed_views_dir, dry_run=bool(args.dry_run))
+ write_json(fixed_views_dir / 'views_manifest.json', fixed_views_result)
+ stages['fixed_layout_views'] = {
+ 'status': fixed_views_result.get('status'),
+ 'view_count': len(fixed_views_result.get('views', [])) if isinstance(fixed_views_result.get('views'), list) else 0,
+ }
+ if fixed_views_result.get('status') == 'available':
+ scene_state.setdefault('evidence', {})['fixed_layout_views'] = fixed_views_result
+
+ scene_state = build_scene_state_from_relation_layout(
+ args.scene_name,
+ run_root,
+ args.render_script.resolve(),
+ output_dir,
+ )
+ if isinstance(fixed_views_result, dict) and fixed_views_result.get('status') == 'available':
+ scene_state.setdefault('evidence', {})['fixed_layout_views'] = fixed_views_result
+ scene_state.setdefault('evidence', {})['render_grid_wall_precompute'] = {
+ 'status': render_manifest.get('status'),
+ 'method_candidates_path': str(method_candidates_path),
+ 'render_grid_affine_path': str(affine_path),
+ 'canonical_topdown_image_path': render_manifest.get('canonical_topdown_image_path'),
+ 'canonical_grid_mapping_path': render_manifest.get('canonical_mapping_path'),
+ 'source': PRIMARY_METHOD,
+ }
+ wall_evidence = build_wall_evidence(scene_state)
+ wall_evidence_path = walls_dir / 'wall_evidence.json'
+ write_json(wall_evidence_path, wall_evidence)
+ write_wall_evidence(wall_evidence, walls_dir)
+ scene_state_path = output_dir / 'scene_state.json'
+ write_json(scene_state_path, scene_state)
+
+ outputs = {
+ **relation_outputs_partial,
+ 'wall_evidence_path': str(wall_evidence_path),
+ 'contain_candidates_path': str(contain_candidates_path),
+ 'contain_visual_evidence_path': str(output_dir / 'contain_visual_evidence' / 'contain_candidates_overlay.png'),
+ 'scene_state_path': str(scene_state_path),
+ 'expected_manifest_path': str(output_dir / 'expected_manifest.json'),
+ }
+ status = 'available' if render_manifest.get('status') == 'available' and obb_walls.get('status') == 'available' else 'partial'
+ if args.include_fixed_views and fixed_views_result.get('status') != 'available':
+ status = 'partial'
+ manifest = {
+ 'schema_version': 'relation_layout_manifest',
+ 'status': status,
+ 'scene_name': args.scene_name,
+ 'run_root': str(run_root),
+ 'precompute_dir': str(output_dir),
+ 'created_by': 'vlm_agent.reconstruction.relations.tools.build_relation_layout_from_sam3d',
+ 'coordinate_frame': 'OpenGL original XZ; Y is vertical',
+ 'formal_wall_method': PRIMARY_METHOD,
+ 'stages': stages,
+ 'outputs': outputs,
+ 'fixed_views_enabled': bool(args.include_fixed_views),
+ 'dry_run': bool(args.dry_run),
+ }
+ expected_done = update_expected_status(expected, outputs, fixed_views_status=str(fixed_views_result.get('status')))
+ write_json(output_dir / 'expected_manifest.json', expected_done)
+ write_json(output_dir / 'manifest.json', manifest)
+ write_manifest_md(output_dir / 'manifest.json', manifest)
+ print(json.dumps(manifest, indent=2, ensure_ascii=False))
+
+
+if __name__ == '__main__':
+ main()
diff --git a/vlm_agent/reconstruction/relations/tools/build_scene_state.py b/vlm_agent/reconstruction/relations/tools/build_scene_state.py
new file mode 100644
index 0000000..cdfc31e
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/build_scene_state.py
@@ -0,0 +1,396 @@
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import sys
+from pathlib import Path
+from typing import Any
+
+ROOT = Path(__file__).resolve().parents[2]
+REPOSITORY_ROOT = Path(__file__).resolve().parents[4]
+if str(REPOSITORY_ROOT) not in sys.path:
+ sys.path.insert(0, str(REPOSITORY_ROOT))
+if str(ROOT) not in sys.path:
+ sys.path.insert(0, str(ROOT))
+
+from relations.agents.state import sorted_object_id_key, write_json, write_md
+from vlm_agent.run_paths import ScenePaths
+
+
+def _load_json_if_exists(path: Path | None) -> Any | None:
+ if not path or not path.exists():
+ return None
+ try:
+ return json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError):
+ # Async relation precompute may have created the manifest before its
+ # atomic payload is fully visible. Treat that short state as not ready;
+ # the runner's normal readiness polling will load it later.
+ return None
+
+
+def _detect_render_python() -> str:
+ env_python = os.environ.get("SCENEMOSAIC_RENDER_PYTHON")
+ if env_python and Path(env_python).exists():
+ return str(Path(env_python))
+ return sys.executable
+
+
+def _list_files(directory: Path | None, suffixes: tuple[str, ...]) -> list[str]:
+ if not directory or not directory.exists():
+ return []
+ return sorted(str(p) for p in directory.iterdir() if p.is_file() and p.suffix.lower() in suffixes)
+
+
+
+
+def _json_has_grid_points(path: Path) -> bool:
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except Exception:
+ return False
+ return isinstance(data, dict) and isinstance(data.get("grid_points"), list) and bool(data["grid_points"])
+
+
+def _detect_relation_precompute(run_root: Path) -> dict[str, Any]:
+ explicit_dir = os.environ.get("RELATION_LAYOUT_DIR")
+ if explicit_dir:
+ candidates = [(Path(explicit_dir).expanduser().resolve() / "manifest.json", "explicit_relation_layout")]
+ else:
+ default_dir = ScenePaths.from_scene_root(run_root).relations_dir / "relation_layout"
+ candidates = [(default_dir / "manifest.json", "relation_layout")]
+ missing_paths = []
+ for manifest_path, source in candidates:
+ data = _load_json_if_exists(manifest_path)
+ if not isinstance(data, dict):
+ missing_paths.append(str(manifest_path))
+ continue
+ outputs = data.get("outputs") if isinstance(data.get("outputs"), dict) else {}
+ return {
+ "status": data.get("status"),
+ "source": source,
+ "manifest_path": str(manifest_path),
+ "precompute_dir": data.get("precompute_dir"),
+ "formal_wall_method": data.get("formal_wall_method"),
+ "outputs": outputs,
+ "stages": data.get("stages"),
+ }
+ return {"status": "missing", "source": None, "manifest_path": missing_paths[0] if missing_paths else None, "checked_manifest_paths": missing_paths}
+
+
+def _detect_topdown_grid_mapping(run_root: Path, topdown_json: list[str]) -> dict[str, Any]:
+ relation_precompute = _detect_relation_precompute(run_root)
+ outputs = relation_precompute.get("outputs") if isinstance(relation_precompute, dict) else {}
+ if isinstance(outputs, dict):
+ mapping = outputs.get("canonical_grid_mapping_path")
+ image = outputs.get("canonical_topdown_image_path")
+ affine = outputs.get("render_grid_affine_path")
+ method_candidates = outputs.get("method_candidates_path")
+ if mapping and image and Path(str(mapping)).exists() and Path(str(image)).exists() and _json_has_grid_points(Path(str(mapping))):
+ return {
+ "canonical_mapping_path": str(mapping),
+ "canonical_topdown_image_path": str(image),
+ "source": relation_precompute.get("source") or "relation_precompute",
+ "fallback_used": False,
+ "status": "available",
+ "render_grid_affine_path": str(affine) if affine else None,
+ "method_candidates_path": str(method_candidates) if method_candidates else None,
+ "relation_precompute_manifest_path": relation_precompute.get("manifest_path"),
+ }
+ return {
+ "canonical_mapping_path": None,
+ "canonical_topdown_image_path": None,
+ "source": None,
+ "fallback_used": None,
+ "status": "missing_relation_layout_precompute",
+ }
+
+
+def _detect_fixed_layout_views(run_root: Path, topdown_grid_mapping: dict[str, Any]) -> dict[str, Any]:
+ relation_precompute = _detect_relation_precompute(run_root)
+ outputs = relation_precompute.get("outputs") if isinstance(relation_precompute, dict) else {}
+ canonical_mapping = topdown_grid_mapping.get("canonical_mapping_path") if isinstance(topdown_grid_mapping, dict) else None
+ if isinstance(outputs, dict):
+ relation_manifest = outputs.get("fixed_views_manifest_path")
+ relation_views_dir = outputs.get("fixed_views_dir")
+ if relation_manifest and Path(str(relation_manifest)).exists():
+ manifest = _load_json_if_exists(Path(str(relation_manifest)))
+ if isinstance(manifest, dict) and manifest.get("status") == "available":
+ manifest = dict(manifest)
+ manifest["source"] = relation_precompute.get("source") or "relation_precompute"
+ manifest["relation_precompute_manifest_path"] = relation_precompute.get("manifest_path")
+ manifest["views_dir"] = manifest.get("views_dir") or relation_views_dir
+ manifest["manifest_path"] = str(relation_manifest)
+ return manifest
+ views_dir = ScenePaths.from_scene_root(run_root).relations_dir / "relation_layout" / "fixed_views"
+ return {
+ "status": "missing_or_disabled",
+ "views_dir": str(views_dir),
+ "manifest_path": str(views_dir / "views_manifest.json"),
+ "canonical_grid_mapping_path": canonical_mapping,
+ "views": [],
+ }
+
+
+def _snake_case(value: str | None) -> str:
+ value = (value or "unknown_object").strip()
+ value = re.sub(r"[^0-9A-Za-z]+", "_", value)
+ value = re.sub(r"_+", "_", value).strip("_").lower()
+ return value or "unknown_object"
+
+
+def _index_reviewed_objects(reviewed_objects_path: Path | None) -> dict[str, dict[str, Any]]:
+ data = _load_json_if_exists(reviewed_objects_path)
+ if not isinstance(data, dict):
+ return {}
+ indexed = {}
+ for item in data.get("objects", []):
+ object_id = item.get("object_id") or item.get("id")
+ if object_id:
+ indexed[str(object_id)] = item
+ return indexed
+
+
+def _find_mask(mask_dir: Path | None, object_id: str) -> str | None:
+ if not mask_dir or not mask_dir.exists():
+ return None
+ matches = sorted(mask_dir.glob(f"{object_id}_*_mask.png")) + sorted(mask_dir.glob(f"{object_id}.png"))
+ return str(matches[0]) if matches else None
+
+
+def _find_overlay(overlay_dir: Path | None, object_id: str) -> str | None:
+ if not overlay_dir or not overlay_dir.exists():
+ return None
+ matches = sorted(overlay_dir.glob(f"{object_id}_*_overlay.png")) + sorted(overlay_dir.glob(f"{object_id}.png"))
+ return str(matches[0]) if matches else None
+
+
+def build_scene_state(scene_name: str, run_root: Path, render_script: Path) -> dict[str, Any]:
+ run_root = run_root.resolve()
+ scene_paths = ScenePaths.from_scene_root(run_root)
+ default_relation_layout = scene_paths.relations_dir / "relation_layout"
+ relation_precompute = _detect_relation_precompute(run_root)
+ relation_outputs = relation_precompute.get("outputs") if isinstance(relation_precompute.get("outputs"), dict) else {}
+ relation_layout_path = relation_outputs.get("layout_for_relation_path")
+ layout_for_relation_path = Path(str(relation_layout_path)) if relation_layout_path else (default_relation_layout / "layout_for_relation.json")
+ layout_prompt_path = default_relation_layout / "layout_for_relation_prompt.json"
+ reviewed_objects_path = scene_paths.perception_dir / "reviewed_objects.json"
+ reviewed_masks_dir = scene_paths.perception_dir / "reviewed_masks"
+ reviewed_overlays_dir = scene_paths.perception_dir / "reviewed_overlays"
+ canonical_topdown_path = relation_outputs.get("canonical_topdown_image_path")
+ topdown_dir = Path(str(canonical_topdown_path)).parent if canonical_topdown_path else (default_relation_layout / "render_grid_wall")
+ layout_params_dir = scene_paths.sam3d_output_dir / "layout_params"
+ unposed_objects_dir = scene_paths.sam3d_output_dir / "unposed_objects"
+ sam3d_input_masks_dir = scene_paths.sam3d_input_dir / "masks"
+
+ layout_data = _load_json_if_exists(layout_for_relation_path)
+ reviewed_index = _index_reviewed_objects(reviewed_objects_path)
+
+ object_items: list[dict[str, Any]] = []
+ layout_objects = layout_data.get("objects", []) if isinstance(layout_data, dict) else []
+ for raw in layout_objects:
+ object_id = str(raw.get("object_id") or raw.get("id") or "")
+ if not object_id:
+ continue
+ reviewed = reviewed_index.get(object_id, {})
+ category = _snake_case(raw.get("category") or reviewed.get("category"))
+ description = raw.get("description") or reviewed.get("description") or category.replace("_", " ")
+ mask_path = reviewed.get("sam3", {}).get("mask_path") or _find_mask(reviewed_masks_dir, object_id)
+ overlay_path = reviewed.get("sam3", {}).get("overlay_path") or _find_overlay(reviewed_overlays_dir, object_id)
+ layout_params_path = layout_params_dir / f"{object_id}.json"
+ unposed_path = unposed_objects_dir / f"{object_id}.glb"
+ object_items.append(
+ {
+ "object_id": object_id,
+ "category": category,
+ "description": description,
+ "bbox_2d": raw.get("image_bbox") or {"bbox_2d_norm_xyxy": reviewed.get("bbox_2d_norm_xyxy")},
+ "bbox_3d": raw.get("bbox_global"),
+ "layout_reference": {
+ "orientation_for_agent": raw.get("orientation_for_agent"),
+ "coordinate_system": layout_data.get("coordinate_system") if isinstance(layout_data, dict) else None,
+ },
+ "mask_path": mask_path,
+ "overlay_path": overlay_path,
+ "layout_params_path": str(layout_params_path) if layout_params_path.exists() else None,
+ "unposed_object_path": str(unposed_path) if unposed_path.exists() else None,
+ "evidence_status": {
+ "has_mask": bool(mask_path),
+ "has_layout": layout_params_path.exists() and raw.get("bbox_global") is not None,
+ "has_3d_asset": unposed_path.exists(),
+ },
+ }
+ )
+
+ object_items.sort(key=lambda item: sorted_object_id_key(item["object_id"]))
+ grounding_items = [
+ {
+ "object_id": item["object_id"],
+ "primary_grounding": "reviewed mask/overlay when available; VLM only for conflict review",
+ "mask_path": item.get("mask_path"),
+ "overlay_path": item.get("overlay_path"),
+ "status": "grounded_by_existing_labels" if item.get("mask_path") else "missing_mask_needs_review",
+ }
+ for item in object_items
+ ]
+
+ topdown_png = _list_files(topdown_dir, (".png",))
+ topdown_json = _list_files(topdown_dir, (".json",))
+ topdown_grid_mapping = _detect_topdown_grid_mapping(run_root, topdown_json)
+ fixed_layout_views = _detect_fixed_layout_views(run_root, topdown_grid_mapping)
+ missing_inputs = []
+ for label, path in [
+ ("layout_for_relation_path", layout_for_relation_path),
+ ("reviewed_objects_path", reviewed_objects_path),
+ ("layout_params_dir", layout_params_dir),
+ ("unposed_objects_dir", unposed_objects_dir),
+ ("topdown_dir", topdown_dir),
+ ("render_script", render_script),
+ ]:
+ if not path.exists():
+ missing_inputs.append({"name": label, "path": str(path)})
+
+ warnings = []
+ if not object_items:
+ warnings.append(f"No object candidates were extracted from {layout_for_relation_path}.")
+ if not topdown_png:
+ warnings.append(f"No top-down PNG evidence found under {topdown_dir}.")
+
+ return {
+ "scene_name": scene_name,
+ "run_root": str(run_root),
+ "evidence": {
+ "scene_config_path": str(run_root / "input" / "scene_config.json") if (run_root / "input" / "scene_config.json").exists() else None,
+ "original_image_path": str(run_root / "input" / "image.png") if (run_root / "input" / "image.png").exists() else None,
+ "mask_overlay_path": str(scene_paths.perception_dir / "reviewed_overlay.png") if (scene_paths.perception_dir / "reviewed_overlay.png").exists() else None,
+ "reviewed_objects_path": str(reviewed_objects_path) if reviewed_objects_path.exists() else None,
+ "object_masks_dir": str(reviewed_masks_dir) if reviewed_masks_dir.exists() else None,
+ "sam3d_input_masks_dir": str(sam3d_input_masks_dir) if sam3d_input_masks_dir.exists() else None,
+ "layout_for_relation_path": str(layout_for_relation_path) if layout_for_relation_path.exists() else None,
+ "layout_for_relation_prompt_path": str(layout_prompt_path) if layout_prompt_path.exists() else None,
+ "layout_params_dir": str(layout_params_dir) if layout_params_dir.exists() else None,
+ "unposed_objects_dir": str(unposed_objects_dir) if unposed_objects_dir.exists() else None,
+ "topdown_dir": str(topdown_dir) if topdown_dir.exists() else None,
+ "topdown_assets": {"png_files": topdown_png, "json_files": topdown_json},
+ "render_script": str(render_script),
+ "render_python": _detect_render_python(),
+ "fixed_layout_views": fixed_layout_views,
+ "relation_precompute": relation_precompute,
+ },
+ "topdown_grid_mapping": topdown_grid_mapping,
+ "object_candidates": {
+ "source_priority": ["relation_layout", "sam3d_output", "reviewed_masks"],
+ "items": object_items,
+ "status": "available_needs_validation" if object_items else "missing",
+ },
+ "object_grounding": {
+ "policy": "mask_overlay_and_per_object_masks_are_primary",
+ "items": grounding_items,
+ },
+ "structure_targets": {
+ "walls": {
+ "status": "to_be_generated",
+ "required_ids": ["wall_001", "wall_002", "wall_003", "wall_004"],
+ "coordinate_format": "[x, z]",
+ "clockwise_required": True,
+ "source_evidence": ["topdown_with_coordinates"],
+ },
+ "floor": {"status": "assumed_present", "id": "floor"},
+ "ceiling": {"status": "unknown", "requires_agent_judgement": True},
+ },
+ "relation_targets": {
+ "allowed_relation_types": ["attach", "contain"],
+ "attach_schema": {"relation": "attach", "parameters": {"object_anchor": "...", "object_target": "..."}},
+ "contain_schema": {"relation": "contain", "parameters": {"object_anchor": "...", "object_target": "..."}},
+ "contain_policy": "Contain is same-level with attach. For the same object_anchor/object_target pair, emit contain instead of attach when the target is inside the anchor's storage/display/interior volume.",
+ },
+ "working_state": {
+ "tasks": [],
+ "task_results": [],
+ "wall_candidates": [],
+ "accepted_walls": [],
+ "ceiling_decision": None,
+ "object_revision_candidates": [],
+ "attach_candidates": [],
+ "accepted_relations": [],
+ "rejected_relations": [],
+ "generated_evidence": [],
+ "warnings": [],
+ },
+ "deliverables": {
+ "final_outputs_dir": "final_outputs",
+ "objects_json_path": "final_outputs/objects.json",
+ "relations_json_path": "final_outputs/relations.json",
+ },
+ "constraints": {
+ "do_not_use_old_workflow_outputs_for_inference": True,
+ "final_objects_keep_sam3d_object_ids": True,
+ "final_objects_match_scene_state_candidate_ids": True,
+ "do_not_add_delete_merge_or_renumber_objects": True,
+ "object_id_numbering_inherited_from_sam3d_layout": True,
+ "object_id_base_may_be_zero_or_one": True,
+ "object_id_format": "object_<number>",
+ "final_objects_sorted_by_numeric_suffix": True,
+ "category_description_changes_require_object_revision_audit": True,
+ "final_relations_allow_attach_and_contain": True,
+ "ceiling_anchor_requires_positive_judgement": True,
+ },
+ "missing_inputs": missing_inputs,
+ "warnings": warnings,
+ }
+
+
+def write_scene_state(scene_state: dict[str, Any], output_dir: Path) -> None:
+ state_path = output_dir / "scene_states" / "scene_state.json"
+ summary_path = output_dir / "scene_states" / "scene_state_summary.md"
+ write_json(state_path, scene_state)
+ candidates = scene_state["object_candidates"]["items"]
+ lines = [
+ f"- scene_name: `{scene_state['scene_name']}`",
+ f"- run_root: `{scene_state['run_root']}`",
+ f"- object_candidates: `{len(candidates)}`",
+ f"- object_id_range: `{candidates[0]['object_id'] if candidates else 'none'}` to `{candidates[-1]['object_id'] if candidates else 'none'}`",
+ f"- missing_inputs: `{len(scene_state['missing_inputs'])}`",
+ f"- warnings: `{len(scene_state['warnings'])}`",
+ "",
+ "## Evidence",
+ f"- layout_for_relation: `{scene_state['evidence']['layout_for_relation_path']}`",
+ f"- reviewed_objects: `{scene_state['evidence']['reviewed_objects_path']}`",
+ f"- topdown_assets_png: `{len(scene_state['evidence']['topdown_assets']['png_files'])}`",
+ f"- topdown_assets_json: `{len(scene_state['evidence']['topdown_assets']['json_files'])}`",
+ f"- topdown_grid_mapping_status: `{scene_state.get('topdown_grid_mapping', {}).get('status')}`",
+ f"- topdown_grid_mapping_path: `{scene_state.get('topdown_grid_mapping', {}).get('canonical_mapping_path')}`",
+ f"- fixed_layout_views_status: `{scene_state.get('evidence', {}).get('fixed_layout_views', {}).get('status')}`",
+ f"- fixed_layout_views_manifest: `{scene_state.get('evidence', {}).get('fixed_layout_views', {}).get('manifest_path')}`",
+ "",
+ "## Important Constraint",
+ "This scene_state intentionally does not load old final objects.json, old relations.json, old attach pass outputs, or old wall extraction results as inference inputs.",
+ "Final objects must preserve exactly the object ids from scene_state.object_candidates; numbering is inherited from SAM3D/layout and may be 0-based or 1-based.",
+ ]
+ write_md(summary_path, "Scene State Summary", lines)
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Build scene_state.json for multi-agent scene delivery.")
+ parser.add_argument("--scene_name", required=True)
+ parser.add_argument("--run_root", required=True, type=Path)
+ parser.add_argument("--render_script", type=Path, default=REPOSITORY_ROOT / "vlm_agent" / "evolution" / "tools" / "render_scene_from_layout.py")
+ parser.add_argument("--output_dir", type=Path, default=None)
+ parser.add_argument("--output", type=Path, default=None, help="Optional direct scene_state.json output path.")
+ args = parser.parse_args()
+ if args.output_dir is None and args.output is None:
+ parser.error("one of --output_dir or --output is required")
+ scene_state = build_scene_state(args.scene_name, args.run_root, args.render_script)
+ if args.output is not None:
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ write_json(args.output, scene_state)
+ print(args.output)
+ else:
+ write_scene_state(scene_state, args.output_dir)
+ print(args.output_dir / "scene_states" / "scene_state.json")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/relations/tools/build_wall_object_hypotheses.py b/vlm_agent/reconstruction/relations/tools/build_wall_object_hypotheses.py
new file mode 100644
index 0000000..08c2dc1
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/build_wall_object_hypotheses.py
@@ -0,0 +1,135 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+ROOT = Path(__file__).resolve().parents[2]
+if str(ROOT) not in sys.path:
+ sys.path.insert(0, str(ROOT))
+
+from relations.agents.state import write_json, write_md
+
+
+HYPOTHESES_JSON_NAME = "wall_object_hypotheses.json"
+HYPOTHESES_MD_NAME = "wall_object_hypotheses.md"
+
+
+def _walk(value: Any):
+ if isinstance(value, dict):
+ yield value
+ for child in value.values():
+ yield from _walk(child)
+ elif isinstance(value, list):
+ for child in value:
+ yield from _walk(child)
+
+
+def _list_of_dicts(value: Any) -> list[dict[str, Any]]:
+ if not isinstance(value, list):
+ return []
+ return [item for item in value if isinstance(item, dict)]
+
+
+def extract_wall_object_hypotheses(task_results: list[dict[str, Any]]) -> dict[str, Any]:
+ likely_wall_objects: list[dict[str, Any]] = []
+ wall_line_hypotheses: list[dict[str, Any]] = []
+ source_tasks: list[str] = []
+
+ for root in task_results:
+ if not isinstance(root, dict):
+ continue
+ source_task_id = str(root.get("task_id") or "unknown_task")
+ source_task_type = str(root.get("task_type") or "")
+ root_is_hypothesis = source_task_type == "wall_object_hypothesis" or "wall_object_hypothesis" in source_task_id
+ for node in _walk(root):
+ if not isinstance(node, dict):
+ continue
+ object_items = _list_of_dicts(node.get("likely_wall_objects"))
+ line_items = _list_of_dicts(node.get("wall_line_hypotheses"))
+ if not object_items and not line_items and not root_is_hypothesis:
+ continue
+ if object_items or line_items:
+ if source_task_id not in source_tasks:
+ source_tasks.append(source_task_id)
+ for item in object_items:
+ enriched = dict(item)
+ enriched.setdefault("source_task_id", source_task_id)
+ enriched.setdefault("hypothesis_role", "likely_wall_object")
+ likely_wall_objects.append(enriched)
+ for item in line_items:
+ enriched = dict(item)
+ enriched.setdefault("source_task_id", source_task_id)
+ enriched.setdefault("source", item.get("source") or "wall_attached_object_distribution")
+ enriched.setdefault("may_be_used_as_final_wall", False)
+ wall_line_hypotheses.append(enriched)
+
+ return {
+ "available": bool(likely_wall_objects or wall_line_hypotheses),
+ "likely_wall_objects_count": len(likely_wall_objects),
+ "wall_line_hypotheses_count": len(wall_line_hypotheses),
+ "source_task_ids": source_tasks,
+ "likely_wall_objects": likely_wall_objects,
+ "wall_line_hypotheses": wall_line_hypotheses,
+ "final_wall_source_allowed": False,
+ "coordinate_source_definition": "Wall object hypotheses are layout-XZ support evidence for final adjudication only; they are not final wall coordinates.",
+ "notes": "These hypotheses may support visibility and wall-attach review in wall_final_adjudication, but final walls must still use obb_from_mesh_footprint_min_rect or obb_from_mesh_footprint_min_rect.",
+ }
+
+
+def write_wall_object_hypotheses(output_dir: Path, hypotheses: dict[str, Any]) -> dict[str, Any]:
+ wall_debug_dir = output_dir / "debug" / "walls"
+ json_path = wall_debug_dir / HYPOTHESES_JSON_NAME
+ md_path = wall_debug_dir / HYPOTHESES_MD_NAME
+ data = dict(hypotheses)
+ data["json_path"] = str(json_path)
+ data["md_path"] = str(md_path)
+ write_json(json_path, data)
+
+ lines = [
+ f"- available: `{data.get('available')}`",
+ f"- likely_wall_objects_count: `{data.get('likely_wall_objects_count')}`",
+ f"- wall_line_hypotheses_count: `{data.get('wall_line_hypotheses_count')}`",
+ f"- source_task_ids: `{data.get('source_task_ids', [])}`",
+ "- final_wall_source_allowed: `False`",
+ "- coordinate_source_definition: Wall object hypotheses support final adjudication only; they are not final walls.",
+ "",
+ "## Likely Wall Objects",
+ ]
+ for item in data.get("likely_wall_objects", []):
+ lines.append(
+ f"- `{item.get('object_id')}` role=`{item.get('role')}` side=`{item.get('candidate_wall_side')}` confidence=`{item.get('confidence')}`"
+ )
+ lines.extend(["", "## Wall Line Hypotheses"])
+ for item in data.get("wall_line_hypotheses", []):
+ line = item.get("estimated_line", {}) if isinstance(item.get("estimated_line"), dict) else {}
+ lines.append(
+ f"- `{item.get('hypothesis_id')}` candidate_wall_id=`{item.get('candidate_wall_id')}` line_type=`{item.get('line_type')}` "
+ f"point_a=`{line.get('point_a')}` point_b=`{line.get('point_b')}` confidence=`{item.get('confidence')}` final_ok=`{item.get('may_be_used_as_final_wall', False)}`"
+ )
+ write_md(md_path, "Wall Object Hypotheses", lines)
+ return data
+
+
+def build_and_write_wall_object_hypotheses(output_dir: Path, task_results: list[dict[str, Any]]) -> dict[str, Any]:
+ return write_wall_object_hypotheses(output_dir, extract_wall_object_hypotheses(task_results))
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Extract and write wall object hypotheses from task results JSON.")
+ parser.add_argument("--task_results", required=True, type=Path)
+ parser.add_argument("--output_dir", required=True, type=Path)
+ args = parser.parse_args()
+ task_results = json.loads(args.task_results.read_text(encoding="utf-8"))
+ if isinstance(task_results, dict) and isinstance(task_results.get("task_results"), list):
+ task_results = task_results["task_results"]
+ if not isinstance(task_results, list):
+ raise SystemExit("task_results must be a list or an object containing task_results list")
+ data = build_and_write_wall_object_hypotheses(args.output_dir, task_results)
+ print(json.dumps({"json_path": data["json_path"], "wall_line_hypotheses_count": data["wall_line_hypotheses_count"]}, indent=2, ensure_ascii=False))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/relations/tools/build_wall_visual_evidence.py b/vlm_agent/reconstruction/relations/tools/build_wall_visual_evidence.py
new file mode 100644
index 0000000..43a3553
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/build_wall_visual_evidence.py
@@ -0,0 +1,496 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+from PIL import Image, ImageDraw, ImageFont
+
+ROOT = Path(__file__).resolve().parents[2]
+if str(ROOT) not in sys.path:
+ sys.path.insert(0, str(ROOT))
+
+from relations.agents.state import write_json, write_md
+
+CONTACT_SHEET_NAME = "{scene}_wall_evidence_contact_sheet.png"
+CANDIDATE_OVERLAY_NAME = "{scene}_candidate_walls_overlay.png"
+MANIFEST_NAME = "{scene}_wall_visual_review_manifest.json"
+OVERLAY_JSON_NAME = "{scene}_candidate_walls_overlay.json"
+OVERLAY_MD_NAME = "{scene}_candidate_walls_overlay.md"
+MANIFEST_SCHEMA_VERSION = "wall_visual_review"
+
+
+def _load_json(path: Path | None) -> Any:
+ if not path or not path.exists():
+ return None
+ try:
+ return json.loads(path.read_text(encoding="utf-8"))
+ except Exception:
+ return None
+
+
+def _path(value: Any) -> Path | None:
+ return Path(value) if isinstance(value, str) and value else None
+
+
+def _path_status(paths: dict[str, Path | None]) -> dict[str, dict[str, Any]]:
+ return {key: {"path": str(path) if path else None, "exists": bool(path and path.exists())} for key, path in paths.items()}
+
+
+def _fixed_view_paths(scene_state: dict[str, Any]) -> dict[str, Path | None]:
+ fixed = scene_state.get("evidence", {}).get("fixed_layout_views", {})
+ result: dict[str, Path | None] = {
+ "top_view_clean": None,
+ "top_view_grid": None,
+ "front_view": None,
+ "back_view": None,
+ "left_view": None,
+ "right_view": None,
+ }
+ for item in fixed.get("views", []) if isinstance(fixed.get("views"), list) else []:
+ if not isinstance(item, dict):
+ continue
+ view = item.get("view")
+ if view == "top":
+ result["top_view_clean"] = _path(item.get("clean_image_path") or item.get("image_path"))
+ result["top_view_grid"] = _path(item.get("grid_image_path"))
+ elif view in {"front", "back", "left", "right"}:
+ result[f"{view}_view"] = _path(item.get("image_path"))
+ return result
+
+
+def _title(draw: ImageDraw.ImageDraw, x: int, y: int, title: str, path: Path | None, font: ImageFont.ImageFont) -> None:
+ draw.text((x + 12, y + 10), title, fill=(15, 15, 15), font=font)
+ draw.text((x + 12, y + 25), path.name if path else "missing", fill=(80, 80, 80), font=font)
+
+
+def _make_contact_sheet(image_specs: list[tuple[str, Path | None]], output_path: Path) -> None:
+ font = ImageFont.load_default()
+ tile_w = 560
+ tile_h = 420
+ label_h = 44
+ cols = 3
+ rows = max(1, (len(image_specs) + cols - 1) // cols)
+ sheet = Image.new("RGB", (cols * tile_w, rows * tile_h), (246, 246, 246))
+ draw = ImageDraw.Draw(sheet)
+ for idx, (title, path) in enumerate(image_specs):
+ x0 = (idx % cols) * tile_w
+ y0 = (idx // cols) * tile_h
+ draw.rectangle((x0, y0, x0 + tile_w - 1, y0 + tile_h - 1), outline=(205, 205, 205), fill=(250, 250, 250))
+ _title(draw, x0, y0, title, path, font)
+ if not path or not path.exists():
+ draw.text((x0 + 12, y0 + 82), "MISSING", fill=(180, 0, 0), font=font)
+ continue
+ try:
+ im = Image.open(path).convert("RGB")
+ im.thumbnail((tile_w - 28, tile_h - label_h - 24), Image.Resampling.LANCZOS)
+ ix = x0 + (tile_w - im.width) // 2
+ iy = y0 + label_h + (tile_h - label_h - im.height) // 2
+ sheet.paste(im, (ix, iy))
+ except Exception as exc:
+ draw.text((x0 + 12, y0 + 82), f"LOAD FAILED: {exc}", fill=(180, 0, 0), font=font)
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ sheet.save(output_path)
+
+
+def _candidate_fields(wall_evidence: dict[str, Any]) -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
+ candidate_walls = wall_evidence.get("candidate_walls") or []
+ polygon = wall_evidence.get("candidate_room_polygon_hint")
+ if not candidate_walls and isinstance(wall_evidence.get("debug_summary"), dict):
+ candidate_walls = wall_evidence["debug_summary"].get("candidate_walls") or []
+ if not polygon and isinstance(wall_evidence.get("debug_summary"), dict):
+ polygon = wall_evidence["debug_summary"].get("candidate_room_polygon_hint")
+ return [w for w in candidate_walls if isinstance(w, dict)], polygon if isinstance(polygon, dict) else None
+
+
+def _render_grid_assets(wall_evidence: dict[str, Any]) -> dict[str, Path | None]:
+ source = wall_evidence.get("render_grid_wall_candidates")
+ source = source if isinstance(source, dict) else {}
+ assets = source.get("assets") if isinstance(source.get("assets"), dict) else {}
+ source_path = _path(source.get("source_path"))
+ base = source_path.parent if source_path else None
+
+ def path_from_asset(key: str, default_name: str | None = None) -> Path | None:
+ raw = assets.get(key)
+ path = _path(raw)
+ if path and path.exists():
+ return path
+ if base and default_name:
+ fallback = base / default_name
+ if fallback.exists():
+ return fallback
+ return path if path else None
+
+ return {
+ "render_grid_topdown_path": path_from_asset("render_topdown_path", "render_topdown.png"),
+ "render_grid_topdown_grid_path": path_from_asset("render_topdown_grid_path", "render_topdown_grid.png"),
+ "render_grid_affine_path": path_from_asset("render_grid_affine_path", "render_grid_affine.json"),
+ "render_grid_foreground_mask_path": path_from_asset("render_foreground_mask_path", "render_foreground_mask.png"),
+ "render_grid_candidate_overlay_path": path_from_asset("candidate_overlay_path", "candidate_overlay.png"),
+ "render_grid_wall_orientation_candidates_overlay_path": path_from_asset("wall_orientation_candidates_overlay_path", "wall_orientation_candidates_overlay.png"),
+ "render_grid_primary_candidate_overlay_path": path_from_asset("primary_candidate_overlay_path", "primary_candidate_overlay.png"),
+ "render_grid_method_candidates_path": source_path,
+ }
+
+
+def _exact_candidate_overlay(
+ render_grid_assets: dict[str, Path | None],
+ candidate_walls: list[dict[str, Any]],
+ output_png: Path,
+) -> dict[str, Any] | None:
+ source = render_grid_assets.get("render_grid_primary_candidate_overlay_path") or render_grid_assets.get("render_grid_candidate_overlay_path")
+ affine = render_grid_assets.get("render_grid_affine_path")
+ if not source or not source.exists():
+ return None
+ output_png.parent.mkdir(parents=True, exist_ok=True)
+ output_png.write_bytes(source.read_bytes())
+ return {
+ "overlay_path": str(output_png),
+ "generated": True,
+ "mapping_possible": True,
+ "mapping_method": "exact_render_grid_affine_overlay_reused",
+ "mapping_is_approximate": False,
+ "source_overlay_path": str(source),
+ "render_grid_affine_path": str(affine) if affine else None,
+ "reason": "Reused exact current-run render-grid overlay generated with render_grid_affine.json.",
+ "source_warning": "Overlay is current-run generated wall candidate evidence for visual review; it is not a final wall set by itself.",
+ "candidate_walls_source_trust": [
+ {
+ "id": wall.get("id"),
+ "source": wall.get("source"),
+ "trust_level": wall.get("trust_level"),
+ "may_be_used_as_final_wall": wall.get("may_be_used_as_final_wall"),
+ "topdown_verification_status": wall.get("topdown_verification_status"),
+ "first_point": wall.get("first_point"),
+ "second_point": wall.get("second_point"),
+ }
+ for wall in candidate_walls
+ ],
+ }
+
+
+def _grid_xz_bounds(grid_mapping: dict[str, Any] | None) -> dict[str, float] | None:
+ if not isinstance(grid_mapping, dict):
+ return None
+ xs: list[float] = []
+ zs: list[float] = []
+ for pt in grid_mapping.get("grid_points", []) if isinstance(grid_mapping.get("grid_points"), list) else []:
+ if not isinstance(pt, dict):
+ continue
+ raw = pt.get("point_3d_original") or pt.get("point_3d_normalized") or pt.get("point_3d_original_xz")
+ if isinstance(raw, list) and len(raw) >= 3:
+ xs.append(float(raw[0]))
+ zs.append(float(raw[2]))
+ elif isinstance(raw, list) and len(raw) == 2:
+ xs.append(float(raw[0]))
+ zs.append(float(raw[1]))
+ if not xs or not zs or max(xs) - min(xs) <= 1e-12 or max(zs) - min(zs) <= 1e-12:
+ return None
+ return {"x_min": min(xs), "x_max": max(xs), "z_min": min(zs), "z_max": max(zs)}
+
+
+def _draw_candidate_overlay(
+ canonical_topdown: Path | None,
+ grid_mapping_path: Path | None,
+ candidate_walls: list[dict[str, Any]],
+ output_png: Path,
+) -> dict[str, Any]:
+ result: dict[str, Any] = {
+ "overlay_path": str(output_png),
+ "generated": False,
+ "mapping_possible": False,
+ "mapping_method": None,
+ "mapping_is_approximate": True,
+ "reason": None,
+ "source_warning": "Overlay is for human/agent visual review only and does not represent final walls.",
+ "candidate_walls_source_trust": [
+ {
+ "id": wall.get("id"),
+ "source": wall.get("source"),
+ "trust_level": wall.get("trust_level"),
+ "may_be_used_as_final_wall": wall.get("may_be_used_as_final_wall"),
+ "topdown_verification_status": wall.get("topdown_verification_status"),
+ "first_point": wall.get("first_point"),
+ "second_point": wall.get("second_point"),
+ }
+ for wall in candidate_walls
+ ],
+ }
+ if not canonical_topdown or not canonical_topdown.exists():
+ result["reason"] = "canonical top-down image is missing."
+ return result
+ if not grid_mapping_path or not grid_mapping_path.exists():
+ result["reason"] = "canonical grid mapping JSON is missing."
+ return result
+ if not candidate_walls:
+ result["reason"] = "wall_evidence.json does not contain candidate_walls."
+ return result
+ try:
+ base = Image.open(canonical_topdown).convert("RGBA")
+ except Exception as exc:
+ result["reason"] = f"canonical top-down could not be opened: {exc}"
+ return result
+ bounds = _grid_xz_bounds(_load_json(grid_mapping_path))
+ if not bounds:
+ result["reason"] = "canonical grid mapping does not contain usable XZ bounds."
+ return result
+
+ margin = 28
+ width, height = base.size
+
+ def xz_to_px(point: list[float]) -> tuple[float, float]:
+ x, z = float(point[0]), float(point[1])
+ px = margin + (x - bounds["x_min"]) / (bounds["x_max"] - bounds["x_min"]) * (width - 2 * margin)
+ py = height - margin - (z - bounds["z_min"]) / (bounds["z_max"] - bounds["z_min"]) * (height - 2 * margin)
+ return px, py
+
+ font = ImageFont.load_default()
+ canvas = base.copy()
+ draw = ImageDraw.Draw(canvas)
+ colors = [(255, 40, 40, 255), (40, 120, 255, 255), (0, 170, 80, 255), (255, 150, 0, 255)]
+ for idx, wall in enumerate(candidate_walls):
+ p1 = wall.get("first_point")
+ p2 = wall.get("second_point")
+ if not (isinstance(p1, list) and isinstance(p2, list) and len(p1) == 2 and len(p2) == 2):
+ continue
+ xy1, xy2 = xz_to_px(p1), xz_to_px(p2)
+ color = colors[idx % len(colors)]
+ draw.line((xy1, xy2), fill=color, width=6)
+ for xy in (xy1, xy2):
+ draw.ellipse((xy[0] - 6, xy[1] - 6, xy[0] + 6, xy[1] + 6), fill=color)
+ mx, my = (xy1[0] + xy2[0]) / 2, (xy1[1] + xy2[1]) / 2
+ draw.rectangle((mx + 5, my - 8, mx + 80, my + 12), fill=(255, 255, 255, 220))
+ draw.text((mx + 8, my - 7), str(wall.get("id")), fill=color, font=font)
+ draw.rectangle((8, 8, width - 8, 50), fill=(255, 255, 255, 225))
+ draw.text((16, 17), "candidate_walls overlay: approximate grid-extent XZ -> image mapping; visual review only", fill=(180, 0, 0), font=font)
+ output_png.parent.mkdir(parents=True, exist_ok=True)
+ canvas.convert("RGB").save(output_png)
+ result.update(
+ {
+ "generated": True,
+ "mapping_possible": True,
+ "mapping_method": "approximate_linear_xz_to_image_using_canonical_grid_extent_and_image_dimensions",
+ "mapping_bounds_xz": bounds,
+ "image_size": {"width": width, "height": height},
+ "image_margin_px": margin,
+ "reason": "Generated using grid extent because grid mapping does not store explicit pixel coordinates.",
+ }
+ )
+ return result
+
+
+def _write_overlay_md(path: Path, overlay_result: dict[str, Any]) -> None:
+ lines = [
+ f"- overlay_generated: `{overlay_result.get('generated')}`",
+ f"- overlay_path: `{overlay_result.get('overlay_path')}`",
+ f"- mapping_possible: `{overlay_result.get('mapping_possible')}`",
+ f"- mapping_method: `{overlay_result.get('mapping_method')}`",
+ f"- mapping_is_approximate: `{overlay_result.get('mapping_is_approximate')}`",
+ f"- reason: `{overlay_result.get('reason')}`",
+ "",
+ "Overlay is for visual review only and does not represent final walls.",
+ "",
+ "## Candidate Walls",
+ ]
+ for wall in overlay_result.get("candidate_walls_source_trust", []):
+ lines.append(
+ f"- {wall.get('id')}: source=`{wall.get('source')}`, trust_level=`{wall.get('trust_level')}`, "
+ f"may_be_used_as_final_wall=`{wall.get('may_be_used_as_final_wall')}`, verification=`{wall.get('topdown_verification_status')}`"
+ )
+ write_md(path, "Candidate Walls Overlay", lines)
+
+
+def ensure_wall_visual_evidence(scene_state: dict[str, Any], output_dir: str | Path) -> dict[str, Any]:
+ output_dir = Path(output_dir).resolve()
+ scene_name = str(scene_state.get("scene_name") or "scene")
+ output_dir.mkdir(parents=True, exist_ok=True)
+ manifest_path = output_dir / MANIFEST_NAME.format(scene=scene_name)
+ if manifest_path.exists():
+ manifest = _load_json(manifest_path)
+ if (
+ isinstance(manifest, dict)
+ and manifest.get("schema_version") == MANIFEST_SCHEMA_VERSION
+ and Path(str(manifest.get("contact_sheet_path", ""))).exists()
+ ):
+ overlay = manifest.get("candidate_walls_overlay_path")
+ if not overlay or Path(str(overlay)).exists():
+ return manifest
+
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state.get("evidence"), dict) else {}
+ topdown = scene_state.get("topdown_grid_mapping", {}) if isinstance(scene_state.get("topdown_grid_mapping"), dict) else {}
+ run_root = Path(scene_state.get("run_root", "."))
+ attach_root = output_dir.parents[2] if output_dir.name == "visual_review" and output_dir.parent.name == "walls" else output_dir
+ wall_evidence_path = output_dir.parent / "wall_evidence.json" if output_dir.parent.name == "walls" else attach_root / "debug" / "walls" / "wall_evidence.json"
+
+ paths: dict[str, Path | None] = {
+ "canonical_topdown_path": _path(topdown.get("canonical_topdown_image_path")),
+ "canonical_grid_mapping_path": _path(topdown.get("canonical_mapping_path")),
+ "wall_evidence_path": wall_evidence_path,
+ "scene_state_path": output_dir.parents[2] / "scene_states" / "scene_state.json" if len(output_dir.parents) >= 3 else None,
+ "original_image_path": _path(evidence.get("original_image_path")),
+ "mask_overlay_path": _path(evidence.get("mask_overlay_path")),
+ "reviewed_overlay_path": _path(evidence.get("reviewed_overlay_path")),
+ "object_masks_dir": _path(evidence.get("object_masks_dir")),
+ }
+ paths.update(_fixed_view_paths(scene_state))
+
+ wall_evidence = _load_json(paths["wall_evidence_path"]) if paths.get("wall_evidence_path") else {}
+ wall_evidence = wall_evidence if isinstance(wall_evidence, dict) else {}
+ candidate_walls, candidate_polygon = _candidate_fields(wall_evidence)
+ render_grid_assets = _render_grid_assets(wall_evidence)
+ paths.update(render_grid_assets)
+
+ contact_sheet_path = output_dir / CONTACT_SHEET_NAME.format(scene=scene_name)
+ contact_specs = [
+ ("render_grid_wall_orientation_candidates", paths.get("render_grid_wall_orientation_candidates_overlay_path")),
+ ("render_grid_primary_candidate_exact", paths.get("render_grid_primary_candidate_overlay_path")),
+ ("render_grid_candidates_exact", paths.get("render_grid_candidate_overlay_path")),
+ ("render_grid_topdown", paths.get("render_grid_topdown_path")),
+ ("render_grid_topdown_grid", paths.get("render_grid_topdown_grid_path")),
+ ("render_grid_foreground_mask", paths.get("render_grid_foreground_mask_path")),
+ ("canonical_topdown", paths.get("canonical_topdown_path")),
+ ("top_view_clean", paths.get("top_view_clean")),
+ ("top_view_grid", paths.get("top_view_grid")),
+ ("front_view", paths.get("front_view")),
+ ("back_view", paths.get("back_view")),
+ ("left_view", paths.get("left_view")),
+ ("right_view", paths.get("right_view")),
+ ("original_image", paths.get("original_image_path")),
+ ("mask_overlay", paths.get("mask_overlay_path")),
+ ]
+ if paths.get("reviewed_overlay_path"):
+ contact_specs.append(("reviewed_overlay", paths.get("reviewed_overlay_path")))
+ _make_contact_sheet(contact_specs, contact_sheet_path)
+
+ overlay_path = output_dir / CANDIDATE_OVERLAY_NAME.format(scene=scene_name)
+ overlay_result = _exact_candidate_overlay(render_grid_assets, candidate_walls, overlay_path)
+ if overlay_result is None:
+ overlay_result = _draw_candidate_overlay(
+ paths.get("canonical_topdown_path"),
+ paths.get("canonical_grid_mapping_path"),
+ candidate_walls,
+ overlay_path,
+ )
+ overlay_json_path = output_dir / OVERLAY_JSON_NAME.format(scene=scene_name)
+ write_json(overlay_json_path, overlay_result)
+ overlay_md_path = output_dir / OVERLAY_MD_NAME.format(scene=scene_name)
+ _write_overlay_md(overlay_md_path, overlay_result)
+
+ sources = sorted({str(w.get("source")) for w in candidate_walls if w.get("source")})
+ trust_levels = sorted({str(w.get("trust_level")) for w in candidate_walls if w.get("trust_level")})
+ may_final = all(w.get("may_be_used_as_final_wall") is True for w in candidate_walls) if candidate_walls else False
+ image_paths_priority = [
+ str(contact_sheet_path),
+ str(overlay_path) if overlay_result.get("generated") else None,
+ str(paths["render_grid_wall_orientation_candidates_overlay_path"]) if paths.get("render_grid_wall_orientation_candidates_overlay_path") and paths["render_grid_wall_orientation_candidates_overlay_path"].exists() else None,
+ str(paths["render_grid_primary_candidate_overlay_path"]) if paths.get("render_grid_primary_candidate_overlay_path") and paths["render_grid_primary_candidate_overlay_path"].exists() else None,
+ str(paths["render_grid_topdown_path"]) if paths.get("render_grid_topdown_path") and paths["render_grid_topdown_path"].exists() else None,
+ str(paths["render_grid_topdown_grid_path"]) if paths.get("render_grid_topdown_grid_path") and paths["render_grid_topdown_grid_path"].exists() else None,
+ str(paths["render_grid_foreground_mask_path"]) if paths.get("render_grid_foreground_mask_path") and paths["render_grid_foreground_mask_path"].exists() else None,
+ str(paths["canonical_topdown_path"]) if paths.get("canonical_topdown_path") and paths["canonical_topdown_path"].exists() else None,
+ str(paths["top_view_clean"]) if paths.get("top_view_clean") and paths["top_view_clean"].exists() else None,
+ str(paths["top_view_grid"]) if paths.get("top_view_grid") and paths["top_view_grid"].exists() else None,
+ str(paths["original_image_path"]) if paths.get("original_image_path") and paths["original_image_path"].exists() else None,
+ str(paths["mask_overlay_path"]) if paths.get("mask_overlay_path") and paths["mask_overlay_path"].exists() else None,
+ str(paths["front_view"]) if paths.get("front_view") and paths["front_view"].exists() else None,
+ str(paths["back_view"]) if paths.get("back_view") and paths["back_view"].exists() else None,
+ str(paths["left_view"]) if paths.get("left_view") and paths["left_view"].exists() else None,
+ str(paths["right_view"]) if paths.get("right_view") and paths["right_view"].exists() else None,
+ str(paths["reviewed_overlay_path"]) if paths.get("reviewed_overlay_path") and paths["reviewed_overlay_path"].exists() else None,
+ ]
+ image_paths_priority = [p for p in image_paths_priority if p]
+ fallback_image_paths_priority = image_paths_priority[:5]
+
+ manifest = {
+ "scene_name": scene_name,
+ "review_type": "wall_visual_review",
+ "schema_version": MANIFEST_SCHEMA_VERSION,
+ "created_at_utc": datetime.now(timezone.utc).isoformat(),
+ "output_dir": str(output_dir),
+ "manifest_path": str(manifest_path),
+ "contact_sheet_path": str(contact_sheet_path),
+ "candidate_walls_overlay_path": str(overlay_path) if overlay_result.get("generated") else None,
+ "candidate_walls_overlay_json_path": str(overlay_json_path),
+ "candidate_walls_overlay_md_path": str(overlay_md_path),
+ "canonical_topdown_path": str(paths.get("canonical_topdown_path")) if paths.get("canonical_topdown_path") else None,
+ "canonical_grid_mapping_path": str(paths.get("canonical_grid_mapping_path")) if paths.get("canonical_grid_mapping_path") else None,
+ "render_grid_method_candidates_path": str(paths.get("render_grid_method_candidates_path")) if paths.get("render_grid_method_candidates_path") else None,
+ "render_grid_affine_path": str(paths.get("render_grid_affine_path")) if paths.get("render_grid_affine_path") else None,
+ "render_grid_topdown_path": str(paths.get("render_grid_topdown_path")) if paths.get("render_grid_topdown_path") else None,
+ "render_grid_topdown_grid_path": str(paths.get("render_grid_topdown_grid_path")) if paths.get("render_grid_topdown_grid_path") else None,
+ "render_grid_foreground_mask_path": str(paths.get("render_grid_foreground_mask_path")) if paths.get("render_grid_foreground_mask_path") else None,
+ "render_grid_candidate_overlay_path": str(paths.get("render_grid_candidate_overlay_path")) if paths.get("render_grid_candidate_overlay_path") else None,
+ "render_grid_wall_orientation_candidates_overlay_path": str(paths.get("render_grid_wall_orientation_candidates_overlay_path")) if paths.get("render_grid_wall_orientation_candidates_overlay_path") else None,
+ "render_grid_primary_candidate_overlay_path": str(paths.get("render_grid_primary_candidate_overlay_path")) if paths.get("render_grid_primary_candidate_overlay_path") else None,
+ "top_view_clean_path": str(paths.get("top_view_clean")) if paths.get("top_view_clean") else None,
+ "top_view_grid_path": str(paths.get("top_view_grid")) if paths.get("top_view_grid") else None,
+ "front_view_path": str(paths.get("front_view")) if paths.get("front_view") else None,
+ "back_view_path": str(paths.get("back_view")) if paths.get("back_view") else None,
+ "left_view_path": str(paths.get("left_view")) if paths.get("left_view") else None,
+ "right_view_path": str(paths.get("right_view")) if paths.get("right_view") else None,
+ "original_image_path": str(paths.get("original_image_path")) if paths.get("original_image_path") else None,
+ "mask_overlay_path": str(paths.get("mask_overlay_path")) if paths.get("mask_overlay_path") else None,
+ "reviewed_overlay_path": str(paths.get("reviewed_overlay_path")) if paths.get("reviewed_overlay_path") else None,
+ "object_masks_dir": str(paths.get("object_masks_dir")) if paths.get("object_masks_dir") else None,
+ "wall_evidence_path": str(paths.get("wall_evidence_path")) if paths.get("wall_evidence_path") else None,
+ "path_status": _path_status(paths),
+ "candidate_walls_count": len(candidate_walls),
+ "candidate_walls_source": sources,
+ "candidate_walls_trust_level": trust_levels,
+ "candidate_walls_may_be_final": may_final,
+ "candidate_room_polygon_hint": candidate_polygon,
+ "candidate_walls": candidate_walls,
+ "candidate_overlay": overlay_result,
+ "image_paths_for_wall_repair": image_paths_priority,
+ "fallback_image_paths_for_wall_repair": fallback_image_paths_priority,
+ "failure_summary": [
+ "no final-eligible wall set",
+ "candidate walls are hint-only",
+ "final outputs not generated when final walls are missing",
+ ],
+ "next_suggestions": [
+ "If visual evidence is sufficient, the wall review task should preserve obb_from_mesh_footprint_min_rect walls.",
+ "If visual evidence is insufficient, return insufficient_layout_xz_wall_evidence rather than promoting hints.",
+ "If automation remains ambiguous, ask a human to provide four layout XZ corners or wall segments.",
+ ],
+ "api_called": False,
+ "final_outputs_modified": False,
+ }
+ write_json(manifest_path, manifest)
+ lines = [
+ f"- scene_name: `{scene_name}`",
+ f"- contact_sheet_path: `{manifest['contact_sheet_path']}`",
+ f"- candidate_walls_overlay_path: `{manifest['candidate_walls_overlay_path']}`",
+ f"- canonical_topdown_path: `{manifest['canonical_topdown_path']}`",
+ f"- canonical_grid_mapping_path: `{manifest['canonical_grid_mapping_path']}`",
+ f"- wall_evidence_path: `{manifest['wall_evidence_path']}`",
+ f"- candidate_walls_count: `{manifest['candidate_walls_count']}`",
+ f"- candidate_walls_source: `{manifest['candidate_walls_source']}`",
+ f"- candidate_walls_trust_level: `{manifest['candidate_walls_trust_level']}`",
+ f"- candidate_walls_may_be_final: `{manifest['candidate_walls_may_be_final']}`",
+ f"- original_image_path: `{manifest['original_image_path']}`",
+ f"- mask_overlay_path: `{manifest['mask_overlay_path']}`",
+ f"- image_paths_for_wall_repair: `{manifest['image_paths_for_wall_repair']}`",
+ "",
+ "## Failure Summary",
+ ]
+ lines.extend(f"- {item}" for item in manifest["failure_summary"])
+ write_md(manifest_path.with_suffix(".md"), "Wall Visual Review Manifest", lines)
+ return manifest
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Build wall visual evidence for visual wall inference tasks.")
+ parser.add_argument("--scene_state", required=True, type=Path)
+ parser.add_argument("--output_dir", required=True, type=Path)
+ args = parser.parse_args()
+ scene_state = json.loads(args.scene_state.read_text(encoding="utf-8"))
+ manifest = ensure_wall_visual_evidence(scene_state, args.output_dir)
+ print(json.dumps({"manifest_path": str(args.output_dir / MANIFEST_NAME.format(scene=manifest.get("scene_name", "scene"))), "contact_sheet_path": manifest.get("contact_sheet_path"), "candidate_walls_overlay_path": manifest.get("candidate_walls_overlay_path")}, indent=2, ensure_ascii=False))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/relations/tools/extract_wall_evidence.py b/vlm_agent/reconstruction/relations/tools/extract_wall_evidence.py
new file mode 100644
index 0000000..c11426a
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/extract_wall_evidence.py
@@ -0,0 +1,977 @@
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+ROOT = Path(__file__).resolve().parents[2]
+if str(ROOT) not in sys.path:
+ sys.path.insert(0, str(ROOT))
+
+from relations.agents.state import sorted_object_id_key, write_json, write_md
+from relations.tools.wall_canonical_naming import canonical_wall_naming_policy, provisional_segments_from_polygon
+
+WALL_IDS = ["wall_001", "wall_002", "wall_003", "wall_004"]
+WALL_CONTACT_EPSILON = 0.035
+FLAT_CONTACT_EDGE_MIN_LENGTH_RATIO = 0.008
+FLAT_CONTACT_EDGE_MIN_ABSOLUTE_LENGTH = 0.003
+FLAT_CONTACT_EDGE_MAX_ABSOLUTE_LENGTH = 0.008
+FLAT_CONTACT_EDGE_PARALLEL_TOLERANCE_DEGREES = 8.0
+FLAT_CONTACT_EDGE_COLLINEAR_TOLERANCE_DEGREES = 6.0
+FLAT_CONTACT_EDGE_STANDALONE_CORNER_DEGREES = 20.0
+FLAT_CONTACT_EDGE_LONG_RUN_RATIO = 0.25
+RENDER_GRID_WALL_PRECOMPUTE_PATH_ENV = "RENDER_GRID_WALL_PRECOMPUTE_PATH"
+FORMAL_WALL_METHOD = "obb_from_mesh_footprint_min_rect"
+
+
+def _load_json(path: Path | None) -> Any | None:
+ if not path or not path.exists():
+ return None
+ try:
+ return json.loads(path.read_text(encoding="utf-8"))
+ except Exception:
+ return None
+
+
+def _number(value: Any) -> float | None:
+ if isinstance(value, (int, float)) and not isinstance(value, bool):
+ return float(value)
+ return None
+
+
+def _bounds_from_points(points: list[list[float]]) -> dict[str, float] | None:
+ if not points:
+ return None
+ xs = [p[0] for p in points]
+ zs = [p[1] for p in points]
+ return {"x_min": min(xs), "x_max": max(xs), "z_min": min(zs), "z_max": max(zs)}
+
+
+def _bounds_from_mapping(mapping: dict[str, Any] | None) -> dict[str, Any] | None:
+ if not isinstance(mapping, dict):
+ return None
+ points: list[list[float]] = []
+ for point in mapping.get("grid_points", []) if isinstance(mapping.get("grid_points"), list) else []:
+ if not isinstance(point, dict):
+ continue
+ raw = point.get("point_3d_original_xz")
+ if isinstance(raw, list) and len(raw) == 2:
+ x, z = _number(raw[0]), _number(raw[1])
+ if x is not None and z is not None:
+ points.append([x, z])
+ continue
+ raw3 = point.get("point_3d_original") or point.get("point_3d_normalized")
+ if isinstance(raw3, list) and len(raw3) >= 3:
+ x, z = _number(raw3[0]), _number(raw3[2])
+ if x is not None and z is not None:
+ points.append([x, z])
+ bounds = _bounds_from_points(points)
+ if bounds:
+ bounds["source"] = "canonical_grid_mapping.grid_points_extent_hint"
+ return bounds
+ x_range = mapping.get("x_range")
+ z_range = mapping.get("z_range")
+ if isinstance(x_range, list) and isinstance(z_range, list) and len(x_range) == 2 and len(z_range) == 2:
+ vals = [_number(x_range[0]), _number(x_range[1]), _number(z_range[0]), _number(z_range[1])]
+ if all(v is not None for v in vals):
+ return {"x_min": min(vals[0], vals[1]), "x_max": max(vals[0], vals[1]), "z_min": min(vals[2], vals[3]), "z_max": max(vals[2], vals[3]), "source": "canonical_grid_mapping.x_range_z_range_hint"}
+ return None
+
+
+def _bounds_from_camera_config(camera_config: dict[str, Any] | None) -> dict[str, Any] | None:
+ if not isinstance(camera_config, dict):
+ return None
+ scene_bbox = camera_config.get("scene_bbox_global_xz")
+ if isinstance(scene_bbox, dict):
+ vals = {k: _number(scene_bbox.get(k)) for k in ["x_min", "x_max", "z_min", "z_max"]}
+ if all(v is not None for v in vals.values()):
+ vals["source"] = "topdown_camera_config.scene_bbox_global_xz_hint"
+ return vals
+ x_range = camera_config.get("x_range") or camera_config.get("world_to_pixel", {}).get("x_range")
+ z_range = camera_config.get("z_range") or camera_config.get("world_to_pixel", {}).get("z_range")
+ if isinstance(x_range, list) and isinstance(z_range, list) and len(x_range) == 2 and len(z_range) == 2:
+ vals = [_number(x_range[0]), _number(x_range[1]), _number(z_range[0]), _number(z_range[1])]
+ if all(v is not None for v in vals):
+ return {"x_min": min(vals[0], vals[1]), "x_max": max(vals[0], vals[1]), "z_min": min(vals[2], vals[3]), "z_max": max(vals[2], vals[3]), "source": "topdown_camera_config.x_range_z_range_hint"}
+ return None
+
+
+def _bounds_from_objects(items: list[dict[str, Any]]) -> dict[str, Any] | None:
+ points: list[list[float]] = []
+ for item in items:
+ bbox = item.get("bbox_3d") if isinstance(item, dict) else None
+ if not isinstance(bbox, dict):
+ continue
+ mn = bbox.get("min")
+ mx = bbox.get("max")
+ if isinstance(mn, list) and len(mn) >= 3 and isinstance(mx, list) and len(mx) >= 3:
+ x0, z0 = _number(mn[0]), _number(mn[2])
+ x1, z1 = _number(mx[0]), _number(mx[2])
+ if None not in (x0, z0, x1, z1):
+ points.extend([[x0, z0], [x1, z1]])
+ bounds = _bounds_from_points(points)
+ if bounds:
+ bounds["source"] = "scene_state.object_candidates.bbox_3d_aggregate_hint"
+ return bounds
+ return None
+
+
+def _clockwise_walls_from_bounds(bounds: dict[str, Any] | None) -> list[dict[str, Any]]:
+ if not bounds:
+ return []
+ vals = {k: _number(bounds.get(k)) for k in ["x_min", "x_max", "z_min", "z_max"]}
+ if not all(v is not None for v in vals.values()):
+ return []
+ x_min, x_max, z_min, z_max = vals["x_min"], vals["x_max"], vals["z_min"], vals["z_max"]
+ if abs(x_max - x_min) <= 1e-8 or abs(z_max - z_min) <= 1e-8:
+ return []
+ source = str(bounds.get("source") or "bounds_hint")
+ return [
+ {"id": "wall_001", "first_point": [x_min, z_min], "second_point": [x_min, z_max], "source": f"{source}", "trust_level": "hint_only", "may_be_used_as_final_wall": False},
+ {"id": "wall_002", "first_point": [x_min, z_max], "second_point": [x_max, z_max], "source": f"{source}", "trust_level": "hint_only", "may_be_used_as_final_wall": False},
+ {"id": "wall_003", "first_point": [x_max, z_max], "second_point": [x_max, z_min], "source": f"{source}", "trust_level": "hint_only", "may_be_used_as_final_wall": False},
+ {"id": "wall_004", "first_point": [x_max, z_min], "second_point": [x_min, z_min], "source": f"{source}", "trust_level": "hint_only", "may_be_used_as_final_wall": False},
+ ]
+
+
+def _clean_polygon_xz(raw: Any) -> list[list[float]]:
+ points: list[list[float]] = []
+ if not isinstance(raw, list):
+ return points
+ for point in raw:
+ if not isinstance(point, list) or len(point) != 2:
+ return []
+ x, z = _number(point[0]), _number(point[1])
+ if x is None or z is None:
+ return []
+ points.append([x, z])
+ return points
+
+
+def _polygon_signed_area_xz(points: list[list[float]]) -> float | None:
+ if len(points) < 3:
+ return None
+ total = 0.0
+ for idx, point in enumerate(points):
+ nxt = points[(idx + 1) % len(points)]
+ total += point[0] * nxt[1] - nxt[0] * point[1]
+ return total / 2.0
+
+
+def _segments_from_polygon_xz(points: list[list[float]], *, source: str, method: str, role: str, xz_to_pixel_affine: Any = None) -> list[dict[str, Any]]:
+ return provisional_segments_from_polygon(
+ points,
+ xz_to_pixel_affine=xz_to_pixel_affine,
+ source=source,
+ method=method,
+ role=role,
+ trust_level="candidate",
+ may_be_used_as_final_wall=False,
+ )
+
+
+def _candidate_record_from_render_grid(
+ raw: dict[str, Any],
+ rank_label: int,
+ xz_to_pixel_affine: Any = None,
+ *,
+ selected: bool = False,
+) -> dict[str, Any] | None:
+ method = str(raw.get("method") or "")
+ metrics = raw.get("metrics") if isinstance(raw.get("metrics"), dict) else {}
+ polygon = _clean_polygon_xz(metrics.get("polygon_xz"))
+ if len(polygon) < 3:
+ return None
+ candidate_id = str(raw.get("candidate_id") or f"candidate_{rank_label:02d}")
+ role = "primary_wall_candidate" if selected else "wall_orientation_candidate"
+ source = "render_grid_selected_wall_orientation_candidate" if selected else "render_grid_wall_orientation_candidate"
+ signed_area = _polygon_signed_area_xz(polygon)
+ return {
+ "candidate_id": candidate_id,
+ "rank_label": rank_label,
+ "method": method,
+ "role": role,
+ "polygon_xz": polygon,
+ "orientation": raw.get("orientation") if isinstance(raw.get("orientation"), dict) else {},
+ "evidence": raw.get("evidence") if isinstance(raw.get("evidence"), dict) else {},
+ "polygon_orientation_xz": "counter_clockwise_or_xz_positive" if signed_area and signed_area > 0 else "clockwise_or_xz_negative",
+ "signed_area_xz": signed_area,
+ "wall_segments": _segments_from_polygon_xz(
+ polygon,
+ source=source,
+ method=method,
+ role=role,
+ xz_to_pixel_affine=xz_to_pixel_affine,
+ ),
+ "metrics": {
+ "area_xz": metrics.get("area_xz"),
+ "axis_deviation_degrees": metrics.get("axis_deviation_degrees"),
+ "foreground_coverage": metrics.get("foreground_coverage"),
+ "empty_pixel_ratio_in_candidate": metrics.get("empty_pixel_ratio_in_candidate"),
+ },
+ "selected_by_wall_semantic_review": selected,
+ "trust_level": "candidate",
+ "may_be_used_as_final_wall": False,
+ }
+
+
+def _discover_render_grid_method_candidates(scene_state: dict[str, Any]) -> tuple[Path | None, dict[str, Any] | None, str | None]:
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state.get("evidence"), dict) else {}
+ precompute = evidence.get("render_grid_wall_precompute") if isinstance(evidence.get("render_grid_wall_precompute"), dict) else {}
+ candidate_paths = []
+ for raw in [
+ precompute.get("method_candidates_path") if isinstance(precompute, dict) else None,
+ os.environ.get(RENDER_GRID_WALL_PRECOMPUTE_PATH_ENV),
+ ]:
+ if isinstance(raw, str) and raw:
+ candidate_paths.append(Path(raw))
+ for candidate_path in candidate_paths:
+ data = _load_json(candidate_path)
+ if isinstance(data, dict) and isinstance(data.get("candidates"), list):
+ return candidate_path, data, None
+ if candidate_paths:
+ return candidate_paths[0], None, "configured render-grid method_candidates.json is missing or invalid"
+ return None, None, "formal render-grid precompute was not bound to scene_state"
+
+
+def _extract_render_grid_wall_candidates(scene_state: dict[str, Any]) -> dict[str, Any]:
+ source_path, data, error = _discover_render_grid_method_candidates(scene_state)
+ if not isinstance(data, dict):
+ return {
+ "status": "missing",
+ "source_path": str(source_path) if source_path else None,
+ "reason": error,
+ "primary_method": FORMAL_WALL_METHOD,
+ "supporting_methods": [],
+ "may_be_used_as_final_wall": False,
+ }
+ raw_candidates = [
+ candidate
+ for candidate in data.get("candidates", [])
+ if isinstance(candidate, dict) and candidate.get("method") == FORMAL_WALL_METHOD
+ ]
+ assets = data.get("assets") if isinstance(data.get("assets"), dict) else {}
+ affine_data = _load_json(Path(str(assets.get("render_grid_affine_path")))) if assets.get("render_grid_affine_path") else None
+ xz_to_pixel_affine = affine_data.get("xz_to_pixel_affine") if isinstance(affine_data, dict) else None
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state.get("evidence"), dict) else {}
+ precompute = evidence.get("render_grid_wall_precompute") if isinstance(evidence.get("render_grid_wall_precompute"), dict) else {}
+ requested_id = precompute.get("selected_orientation_candidate_id")
+ review = data.get("wall_orientation_review") if isinstance(data.get("wall_orientation_review"), dict) else {}
+ baseline_id = review.get("baseline_candidate_id")
+ available_ids = [
+ str(candidate.get("candidate_id") or f"candidate_{index:02d}")
+ for index, candidate in enumerate(raw_candidates, start=1)
+ ]
+ if requested_id not in available_ids:
+ requested_id = baseline_id if baseline_id in available_ids else (available_ids[0] if available_ids else None)
+
+ orientation_candidates = []
+ primary = None
+ primary_rank = None
+ for index, candidate in enumerate(raw_candidates, start=1):
+ candidate_id = str(candidate.get("candidate_id") or f"candidate_{index:02d}")
+ record = _candidate_record_from_render_grid(
+ candidate,
+ index,
+ xz_to_pixel_affine=xz_to_pixel_affine,
+ selected=candidate_id == requested_id,
+ )
+ if not record:
+ continue
+ orientation_candidates.append(record)
+ if candidate_id == requested_id:
+ primary = record
+ primary_rank = index
+
+ method_policy = data.get("method_policy") if isinstance(data.get("method_policy"), dict) else {}
+ return {
+ "status": "available" if primary else "missing_primary",
+ "source_path": str(source_path),
+ "source_run_id": data.get("run_id"),
+ "source_scene": data.get("scene"),
+ "assets": assets,
+ "render_grid_affine_path": assets.get("render_grid_affine_path"),
+ "wall_id_naming_source": "deterministic_screen_clockwise_wall_naming",
+ "canonical_topdown_image_path": assets.get("canonical_topdown_path"),
+ "canonical_grid_mapping_path": assets.get("canonical_grid_mapping_path"),
+ "primary_candidate_overlay_path": assets.get("primary_candidate_overlay_path"),
+ "candidate_overlay_path": assets.get("candidate_overlay_path"),
+ "wall_orientation_candidates_overlay_path": assets.get("wall_orientation_candidates_overlay_path"),
+ "integration_mode": "formal_render_grid_precompute_with_local_orientation_selection",
+ "method_family": "render_grid_wall_precompute",
+ "method_policy": {
+ **method_policy,
+ "agent_should_not_choose_method": True,
+ "orientation_selected_locally": True,
+ "primary_method": FORMAL_WALL_METHOD,
+ "supporting_methods": [],
+ },
+ "wall_orientation_review": review,
+ "orientation_candidates": orientation_candidates,
+ "selected_orientation_candidate_id": requested_id,
+ "selection_source": (
+ str(precompute.get("wall_orientation_selection_source") or "local_object_rotation_score")
+ if precompute.get("selected_orientation_candidate_id") in available_ids
+ else "precompute_baseline_fallback"
+ ),
+ "primary_candidate": primary,
+ "primary_rank": primary_rank,
+ "supporting_candidates": [
+ candidate for candidate in orientation_candidates
+ if candidate.get("candidate_id") != requested_id
+ ],
+ "may_be_used_as_final_wall": False,
+ }
+
+
+def _point_segment_distance(point: list[float], a: list[float], b: list[float]) -> float:
+ px, pz = point
+ ax, az = a
+ bx, bz = b
+ dx = bx - ax
+ dz = bz - az
+ denom = dx * dx + dz * dz
+ if denom <= 1e-12:
+ return math.hypot(px - ax, pz - az)
+ t = max(0.0, min(1.0, ((px - ax) * dx + (pz - az) * dz) / denom))
+ qx = ax + t * dx
+ qz = az + t * dz
+ return math.hypot(px - qx, pz - qz)
+
+
+def _bbox_wall_distance(min_xz: list[float] | None, max_xz: list[float] | None, wall: dict[str, Any]) -> tuple[float | None, str | None]:
+ if not min_xz or not max_xz:
+ return None, None
+ x0, z0 = min(min_xz[0], max_xz[0]), min(min_xz[1], max_xz[1])
+ x1, z1 = max(min_xz[0], max_xz[0]), max(min_xz[1], max_xz[1])
+ a = wall.get("first_point")
+ b = wall.get("second_point")
+ if not (isinstance(a, list) and isinstance(b, list) and len(a) == 2 and len(b) == 2):
+ return None, None
+ ax, az = float(a[0]), float(a[1])
+ bx, bz = float(b[0]), float(b[1])
+ if abs(ax - bx) <= 1e-8:
+ wall_x = ax
+ overlap = not (z1 < min(az, bz) or z0 > max(az, bz))
+ side = "left" if wall_x <= (x0 + x1) / 2 else "right"
+ if overlap:
+ return min(abs(x0 - wall_x), abs(x1 - wall_x)), side
+ if abs(az - bz) <= 1e-8:
+ wall_z = az
+ overlap = not (x1 < min(ax, bx) or x0 > max(ax, bx))
+ side = "back" if wall_z <= (z0 + z1) / 2 else "front"
+ if overlap:
+ return min(abs(z0 - wall_z), abs(z1 - wall_z)), side
+ corners = [[x0, z0], [x0, z1], [x1, z0], [x1, z1]]
+ return min(_point_segment_distance(corner, [ax, az], [bx, bz]) for corner in corners), None
+
+
+def analyze_flat_contact_edge(
+ hull_xz: list[list[float]] | None,
+ wall: dict[str, Any],
+) -> dict[str, Any]:
+ """Detect a finite flat wall-contact edge while rejecting point/corner contact."""
+ points = [
+ [float(point[0]), float(point[1])]
+ for point in hull_xz or []
+ if isinstance(point, list) and len(point) == 2
+ and all(isinstance(value, (int, float)) and not isinstance(value, bool) for value in point)
+ ]
+ wall_a = wall.get("first_point")
+ wall_b = wall.get("second_point")
+ if len(points) < 3 or not (
+ isinstance(wall_a, list) and len(wall_a) == 2
+ and isinstance(wall_b, list) and len(wall_b) == 2
+ ):
+ return {
+ "analysis_available": False,
+ "has_valid_flat_contact_edge": None,
+ "reason": "mesh_floor_footprint_hull_or_wall_missing",
+ }
+
+ wall_dx = float(wall_b[0]) - float(wall_a[0])
+ wall_dz = float(wall_b[1]) - float(wall_a[1])
+ wall_length = math.hypot(wall_dx, wall_dz)
+ if wall_length <= 1e-12:
+ return {
+ "analysis_available": False,
+ "has_valid_flat_contact_edge": None,
+ "reason": "degenerate_wall_segment",
+ }
+ wall_unit = (wall_dx / wall_length, wall_dz / wall_length)
+
+ def undirected_angle_degrees(first: tuple[float, float], second: tuple[float, float]) -> float:
+ first_length = math.hypot(*first)
+ second_length = math.hypot(*second)
+ if first_length <= 1e-12 or second_length <= 1e-12:
+ return 90.0
+ cosine = abs((first[0] * second[0] + first[1] * second[1]) / (first_length * second_length))
+ return math.degrees(math.acos(max(0.0, min(1.0, cosine))))
+
+ xs = [point[0] for point in points]
+ zs = [point[1] for point in points]
+ footprint_span = max(max(xs) - min(xs), max(zs) - min(zs), 1e-9)
+ minimum_contact_length = max(
+ FLAT_CONTACT_EDGE_MIN_ABSOLUTE_LENGTH,
+ min(FLAT_CONTACT_EDGE_MAX_ABSOLUTE_LENGTH, footprint_span * FLAT_CONTACT_EDGE_MIN_LENGTH_RATIO),
+ )
+ distance_threshold = min(WALL_CONTACT_EPSILON, max(0.006, footprint_span * 0.10))
+ edges: list[dict[str, Any]] = []
+ for index, start_point in enumerate(points):
+ end_point = points[(index + 1) % len(points)]
+ vector = (end_point[0] - start_point[0], end_point[1] - start_point[1])
+ length = math.hypot(*vector)
+ midpoint = [(start_point[0] + end_point[0]) / 2.0, (start_point[1] + end_point[1]) / 2.0]
+ distances = [
+ _point_segment_distance(start_point, wall_a, wall_b),
+ _point_segment_distance(midpoint, wall_a, wall_b),
+ _point_segment_distance(end_point, wall_a, wall_b),
+ ]
+ parallel_error = undirected_angle_degrees(vector, wall_unit)
+ whole_edge_near_wall = max(distances) <= distance_threshold
+ parallel_to_wall = parallel_error <= FLAT_CONTACT_EDGE_PARALLEL_TOLERANCE_DEGREES
+ edges.append({
+ "index": index,
+ "start": start_point,
+ "end": end_point,
+ "vector": vector,
+ "length": length,
+ "length_ratio": length / footprint_span,
+ "endpoint_and_midpoint_distances": distances,
+ "maximum_distance_to_wall": max(distances),
+ "minimum_distance_to_wall": min(distances),
+ "parallel_error_degrees": parallel_error,
+ "parallel_to_wall": parallel_to_wall,
+ "whole_edge_near_wall": whole_edge_near_wall,
+ "flat_segment_candidate": length > 1e-9 and parallel_to_wall and whole_edge_near_wall,
+ })
+
+ groups: list[list[dict[str, Any]]] = []
+ for edge in edges:
+ if not edge["flat_segment_candidate"]:
+ continue
+ if (
+ groups
+ and edge["index"] == groups[-1][-1]["index"] + 1
+ and undirected_angle_degrees(groups[-1][-1]["vector"], edge["vector"])
+ <= FLAT_CONTACT_EDGE_COLLINEAR_TOLERANCE_DEGREES
+ ):
+ groups[-1].append(edge)
+ else:
+ groups.append([edge])
+ if (
+ len(groups) > 1
+ and groups[0][0]["index"] == 0
+ and groups[-1][-1]["index"] == len(edges) - 1
+ and undirected_angle_degrees(groups[-1][-1]["vector"], groups[0][0]["vector"])
+ <= FLAT_CONTACT_EDGE_COLLINEAR_TOLERANCE_DEGREES
+ ):
+ groups = [groups[-1] + groups[0], *groups[1:-1]]
+
+ flat_runs: list[dict[str, Any]] = []
+ for group in groups:
+ projected_length = sum(
+ abs(edge["vector"][0] * wall_unit[0] + edge["vector"][1] * wall_unit[1])
+ for edge in group
+ )
+ standalone_corner_angles = None
+ standalone_shape_supported = True
+ long_contact_run = projected_length / footprint_span >= FLAT_CONTACT_EDGE_LONG_RUN_RATIO
+ if len(group) == 1:
+ edge_index = group[0]["index"]
+ previous_edge = edges[(edge_index - 1) % len(edges)]
+ next_edge = edges[(edge_index + 1) % len(edges)]
+ standalone_corner_angles = [
+ undirected_angle_degrees(previous_edge["vector"], group[0]["vector"]),
+ undirected_angle_degrees(group[0]["vector"], next_edge["vector"]),
+ ]
+ standalone_shape_supported = (
+ min(standalone_corner_angles) >= FLAT_CONTACT_EDGE_STANDALONE_CORNER_DEGREES
+ or long_contact_run
+ )
+ contact_extent_sufficient = projected_length >= minimum_contact_length
+ valid = contact_extent_sufficient and standalone_shape_supported
+ flat_runs.append({
+ "start": group[0]["start"],
+ "end": group[-1]["end"],
+ "length": projected_length,
+ "length_ratio": projected_length / footprint_span,
+ "source_edge_count": len(group),
+ "source_edge_indices": [edge["index"] for edge in group],
+ "maximum_distance_to_wall": max(edge["maximum_distance_to_wall"] for edge in group),
+ "minimum_distance_to_wall": min(edge["minimum_distance_to_wall"] for edge in group),
+ "maximum_parallel_error_degrees": max(edge["parallel_error_degrees"] for edge in group),
+ "standalone_corner_angles_degrees": standalone_corner_angles,
+ "standalone_shape_supported": standalone_shape_supported,
+ "long_contact_run": long_contact_run,
+ "contact_extent_sufficient": contact_extent_sufficient,
+ "whole_edge_near_wall": True,
+ "parallel_to_wall": True,
+ "valid_flat_contact_run": valid,
+ "support_type": "fragmented_collinear_flat_edge" if len(group) > 1 else "standalone_flat_edge",
+ })
+
+ valid_runs = [run for run in flat_runs if run["valid_flat_contact_run"]]
+ nearest = min(edges, key=lambda edge: edge["maximum_distance_to_wall"])
+ nearest_public = {key: value for key, value in nearest.items() if key != "vector"}
+ best = max(valid_runs, key=lambda run: run["length"]) if valid_runs else nearest_public
+ return {
+ "analysis_available": True,
+ "has_valid_flat_contact_edge": bool(valid_runs),
+ "mesh_floor_footprint_vertex_count": len(points),
+ "footprint_span": footprint_span,
+ "minimum_contact_length": minimum_contact_length,
+ "distance_threshold": distance_threshold,
+ "parallel_tolerance_degrees": FLAT_CONTACT_EDGE_PARALLEL_TOLERANCE_DEGREES,
+ "collinear_merge_tolerance_degrees": FLAT_CONTACT_EDGE_COLLINEAR_TOLERANCE_DEGREES,
+ "flat_contact_runs": flat_runs,
+ "best_edge": best,
+ "reason": "valid_flat_edge_contact_not_point_or_corner" if valid_runs else "no_flat_edge_contact_or_only_point_corner_contact",
+ }
+
+def _relation_footprint_records(scene_state: dict[str, Any]) -> dict[str, dict[str, Any]]:
+ evidence = scene_state.get("evidence", {}) if isinstance(scene_state, dict) else {}
+ relation_precompute = evidence.get("relation_precompute", {}) if isinstance(evidence, dict) else {}
+ outputs = relation_precompute.get("outputs", {}) if isinstance(relation_precompute, dict) else {}
+ candidate = outputs.get("object_footprints_path") if isinstance(outputs, dict) else None
+ if not candidate:
+ layout_path = evidence.get("layout_for_relation_path") if isinstance(evidence, dict) else None
+ if layout_path:
+ candidate = str(Path(layout_path).parent / "object_footprints.json")
+ data = _load_json(Path(candidate)) if candidate else None
+ items = data.get("items", []) if isinstance(data, dict) else []
+ return {
+ str(item["object_id"]): item
+ for item in items
+ if isinstance(item, dict) and item.get("object_id")
+ }
+
+
+def classify_wall_contact_geometry(metric: dict[str, Any]) -> str:
+ if metric.get("possible_against_wall_contact") and metric.get("is_floor_supported"):
+ return "possible floor_supported_and_wall_contact"
+ if metric.get("possible_against_wall_contact"):
+ return "needs_more_evidence"
+ return "not_wall_attach_by_geometry_only"
+
+
+def _object_footprints(
+ items: list[dict[str, Any]],
+ walls: list[dict[str, Any]],
+ footprint_records: dict[str, dict[str, Any]] | None = None,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+ footprints = []
+ metrics = []
+ footprint_records = footprint_records or {}
+ for item in sorted(items, key=lambda x: sorted_object_id_key(x.get("object_id", ""))):
+ object_id = str(item.get("object_id") or "")
+ footprint_record = footprint_records.get(object_id, {})
+ floor_hull = footprint_record.get("mesh_floor_footprint_hull_xz")
+ bbox = item.get("bbox_3d") if isinstance(item, dict) else None
+ center_xz = None
+ min_xz = None
+ max_xz = None
+ bottom_y = None
+ extent_xz = None
+ if isinstance(bbox, dict):
+ center = bbox.get("center")
+ mn = bbox.get("min")
+ mx = bbox.get("max")
+ if isinstance(center, list) and len(center) >= 3:
+ x, z = _number(center[0]), _number(center[2])
+ if x is not None and z is not None:
+ center_xz = [x, z]
+ if isinstance(mn, list) and len(mn) >= 3:
+ x, y, z = _number(mn[0]), _number(mn[1]), _number(mn[2])
+ if x is not None and z is not None:
+ min_xz = [x, z]
+ bottom_y = y
+ if isinstance(mx, list) and len(mx) >= 3:
+ x, z = _number(mx[0]), _number(mx[2])
+ if x is not None and z is not None:
+ max_xz = [x, z]
+ if min_xz and max_xz:
+ extent_xz = [abs(max_xz[0] - min_xz[0]), abs(max_xz[1] - min_xz[1])]
+ is_floor_supported = bool(bottom_y is not None and bottom_y <= -0.05)
+ object_metrics = []
+ for wall in walls:
+ wall_id = wall.get("id")
+ if not center_xz:
+ center_distance = None
+ else:
+ center_distance = _point_segment_distance(center_xz, wall.get("first_point"), wall.get("second_point"))
+ footprint_distance, bbox_side = _bbox_wall_distance(min_xz, max_xz, wall)
+ if isinstance(floor_hull, list) and floor_hull:
+ hull_distances = [
+ _point_segment_distance(point, wall.get("first_point"), wall.get("second_point"))
+ for point in floor_hull
+ if isinstance(point, list) and len(point) == 2
+ ]
+ if hull_distances:
+ footprint_distance = min(hull_distances)
+ possible_contact = bool(footprint_distance is not None and footprint_distance <= WALL_CONTACT_EPSILON)
+ flat_edge = analyze_flat_contact_edge(floor_hull, wall)
+ metric = {
+ "object_id": item.get("object_id"),
+ "category": item.get("category"),
+ "wall_id": wall_id,
+ "center_to_wall_distance": center_distance,
+ "footprint_min_distance_to_wall": footprint_distance,
+ "bbox_xz_min": min_xz,
+ "bbox_xz_max": max_xz,
+ "bbox_side_near_wall": possible_contact,
+ "bbox_side": bbox_side,
+ "object_depth_toward_wall": min(extent_xz) if extent_xz else None,
+ "is_thin_planar_candidate": bool(extent_xz and min(extent_xz) <= 0.04),
+ "is_floor_supported": is_floor_supported,
+ "bottom_y": bottom_y,
+ "possible_against_wall_contact": possible_contact,
+ "mesh_floor_footprint_hull_xz": floor_hull,
+ "flat_contact_edge_analysis": flat_edge,
+ "flat_contact_edge_analysis_available": flat_edge.get("analysis_available"),
+ "has_valid_flat_contact_edge": flat_edge.get("has_valid_flat_contact_edge"),
+ }
+ metric["geometry_classification"] = classify_wall_contact_geometry(metric)
+ metrics.append(metric)
+ object_metrics.append(metric)
+ nearest = min(object_metrics, key=lambda m: m["footprint_min_distance_to_wall"] if m["footprint_min_distance_to_wall"] is not None else 10**9, default=None)
+ footprints.append({
+ "object_id": item.get("object_id"),
+ "category": item.get("category"),
+ "center_xz": center_xz,
+ "bbox_xz_min": min_xz,
+ "bbox_xz_max": max_xz,
+ "nearest_wall_id_by_footprint": nearest.get("wall_id") if nearest else None,
+ "nearest_wall_distance_by_footprint": nearest.get("footprint_min_distance_to_wall") if nearest else None,
+ "possible_against_wall_contact": nearest.get("possible_against_wall_contact") if nearest else None,
+ "is_floor_supported": is_floor_supported,
+ "bottom_y": bottom_y,
+ "mesh_floor_footprint_hull_xz": floor_hull,
+ "mesh_floor_footprint_source": footprint_record.get("mesh_floor_footprint_source"),
+ })
+ return footprints, metrics
+
+
+
+def _grid_point_xz(point: dict[str, Any]) -> list[float] | None:
+ raw = point.get("point_3d_original_xz")
+ if isinstance(raw, list) and len(raw) == 2:
+ x, z = _number(raw[0]), _number(raw[1])
+ if x is not None and z is not None:
+ return [x, z]
+ raw3 = point.get("point_3d_original") or point.get("point_3d_normalized")
+ if isinstance(raw3, list) and len(raw3) >= 3:
+ x, z = _number(raw3[0]), _number(raw3[2])
+ if x is not None and z is not None:
+ return [x, z]
+ return None
+
+
+def _grid_mapping_summary(mapping: dict[str, Any] | None) -> dict[str, Any]:
+ if not isinstance(mapping, dict):
+ return {
+ "available": False,
+ "coordinate_range": None,
+ "grid_points_sample": [],
+ "pixel_to_world_or_world_to_pixel_notes": "No canonical grid mapping JSON was available.",
+ }
+ points = mapping.get("grid_points", []) if isinstance(mapping.get("grid_points"), list) else []
+ xz_points = []
+ samples = []
+ for point in points:
+ if not isinstance(point, dict):
+ continue
+ xz = _grid_point_xz(point)
+ if xz is not None:
+ xz_points.append(xz)
+ if len(samples) < 8:
+ samples.append({
+ "grid_i": point.get("grid_i"),
+ "grid_j": point.get("grid_j"),
+ "point_xz": xz,
+ "point_3d_original": point.get("point_3d_original"),
+ "point_3d_normalized": point.get("point_3d_normalized"),
+ })
+ coordinate_range = _bounds_from_points(xz_points) if xz_points else None
+ return {
+ "available": bool(xz_points),
+ "grid_size": mapping.get("grid_size"),
+ "grid_point_count": len(points),
+ "coordinate_range": coordinate_range,
+ "grid_points_sample": samples,
+ "pixel_to_world_or_world_to_pixel_notes": (
+ "Grid points are canonical top-down world [x,z] samples produced with the canonical PNG. "
+ "The extent rectangle is a hint only; final walls are layout XZ coordinates. Canonical top-down/grid mapping is used for verification and visualization, not as the primary coordinate source."
+ ),
+ "source": mapping.get("source"),
+ "fallback_used": mapping.get("fallback_used"),
+ }
+
+def build_wall_evidence(scene_state: dict[str, Any]) -> dict[str, Any]:
+ warnings: list[str] = []
+ evidence = scene_state.get("evidence", {})
+ topdown_assets = evidence.get("topdown_assets", {}) if isinstance(evidence.get("topdown_assets"), dict) else {}
+ topdown_jsons = [Path(p) for p in topdown_assets.get("json_files", []) if isinstance(p, str)]
+ camera_config_path = None
+ for path in topdown_jsons:
+ if path.name == "topdown_camera_config.json" and path.exists():
+ camera_config_path = path
+ break
+ topdown_mapping_state = scene_state.get("topdown_grid_mapping", {}) if isinstance(scene_state.get("topdown_grid_mapping"), dict) else {}
+ mapping_path = topdown_mapping_state.get("canonical_mapping_path")
+ image_path = topdown_mapping_state.get("canonical_topdown_image_path")
+ mapping = _load_json(Path(mapping_path)) if mapping_path else None
+ grid_summary = _grid_mapping_summary(mapping)
+ camera_config = _load_json(camera_config_path)
+ object_items = scene_state.get("object_candidates", {}).get("items", [])
+
+ mapping_bounds = _bounds_from_mapping(mapping)
+ camera_bounds = _bounds_from_camera_config(camera_config)
+ object_bounds = _bounds_from_objects(object_items if isinstance(object_items, list) else [])
+
+ layout_bounds = camera_bounds or object_bounds or mapping_bounds
+ if layout_bounds is None:
+ layout_bounds = {"x_min": None, "x_max": None, "z_min": None, "z_max": None, "source": None}
+ warnings.append("Could not infer layout bounds hints from topdown camera config, object bbox, or grid mapping.")
+
+ polygon_bounds = mapping_bounds or camera_bounds or object_bounds
+ candidate_walls = _clockwise_walls_from_bounds(polygon_bounds)
+ if candidate_walls:
+ for wall in candidate_walls:
+ wall["source"] = "layout_xz_boundary_points"
+ wall["topdown_verification_status"] = "needs_check"
+ wall["trust_level"] = "candidate"
+ wall["may_be_used_as_final_wall"] = False
+ warnings.append("candidate_walls are layout XZ boundary point candidates from grid/camera/object extents; they remain candidate-only until topdown_verification=passed.")
+ else:
+ warnings.append("Could not derive a four-wall clockwise hint from available bounds.")
+ polygon_points = [wall["first_point"] for wall in candidate_walls] if candidate_walls else []
+ relation_footprint_records = _relation_footprint_records(scene_state)
+ footprints, wall_attach_geometry = _object_footprints(
+ object_items if isinstance(object_items, list) else [],
+ candidate_walls,
+ relation_footprint_records,
+ )
+
+ render_grid_wall_candidates = _extract_render_grid_wall_candidates(scene_state)
+ primary_render_grid = render_grid_wall_candidates.get("primary_candidate") if isinstance(render_grid_wall_candidates, dict) else None
+ primary_render_grid_walls = primary_render_grid.get("wall_segments", []) if isinstance(primary_render_grid, dict) else []
+ if primary_render_grid_walls:
+ render_grid_object_footprints, render_grid_wall_attach_geometry = _object_footprints(
+ object_items if isinstance(object_items, list) else [],
+ primary_render_grid_walls,
+ relation_footprint_records,
+ )
+ warnings.append("Formal render-grid wall evidence uses only obb_from_mesh_footprint_min_rect.")
+ else:
+ render_grid_object_footprints, render_grid_wall_attach_geometry = [], []
+ reason = render_grid_wall_candidates.get("reason") if isinstance(render_grid_wall_candidates, dict) else "unknown"
+ warnings.append(f"No usable formal render-grid primary wall candidate was loaded: {reason}")
+
+ active_candidate_walls = primary_render_grid_walls
+ active_polygon_points = (
+ primary_render_grid.get("polygon_xz") or [wall.get("first_point") for wall in primary_render_grid_walls]
+ if isinstance(primary_render_grid, dict)
+ else []
+ )
+ active_polygon_source = (
+ primary_render_grid_walls[0].get("source", "render_grid_generated_primary_wall_candidate")
+ if primary_render_grid_walls
+ else None
+ )
+ active_polygon_trust_level = "candidate" if primary_render_grid_walls else "missing"
+ active_footprints = render_grid_object_footprints
+ active_wall_attach_geometry = render_grid_wall_attach_geometry
+ non_authoritative_wall_hints = {
+ "reason": "Non-authoritative layout/grid/bbox wall hints are retained for audit only; formal tasks require render-grid OBB walls.",
+ "candidate_walls_count": len(candidate_walls),
+ "wall_attach_geometry_count": len(wall_attach_geometry),
+ "candidate_room_polygon_hint": {
+ "available": bool(polygon_points),
+ "points_xz": polygon_points,
+ "source": polygon_bounds.get("source") if isinstance(polygon_bounds, dict) else None,
+ "trust_level": "hint_only",
+ "may_be_used_as_final_walls": False,
+ },
+ }
+ wall_candidate_mode = "render_grid_only"
+
+ fixed_views = evidence.get("fixed_layout_views") if isinstance(evidence.get("fixed_layout_views"), dict) else {}
+ return {
+ "scene_name": scene_state.get("scene_name"),
+ "layout_xz_evidence": {
+ "source_candidates": ["layout_for_relation", "layout_params", "object_footprints", "topdown_camera_config"],
+ "coordinate_system": "layout_xz",
+ "notes": ["Final wall coordinates should be expressed in OpenGL original XZ and generated by obb_from_mesh_footprint_min_rect."],
+ },
+ "topdown_verification": {
+ "canonical_topdown_image_path": image_path,
+ "canonical_grid_mapping_path": mapping_path,
+ "topdown_camera_config_path": str(camera_config_path) if camera_config_path else None,
+ "render_grid_affine_path": render_grid_wall_candidates.get("render_grid_affine_path") if isinstance(render_grid_wall_candidates, dict) else None,
+ "usage": "verification_only",
+ "source": topdown_mapping_state.get("source"),
+ "fallback_used": topdown_mapping_state.get("fallback_used"),
+ "notes": [
+ "Canonical top-down/grid mapping helps verify and visualize layout XZ wall coordinates, but is not the primary source of final wall coordinates."
+ ],
+ },
+ "fixed_layout_views_verification": {
+ "manifest_path": fixed_views.get("manifest_path"),
+ "usage": "auxiliary_visual_verification_only",
+ "views": [item.get("view") for item in fixed_views.get("views", []) if isinstance(item, dict)],
+ "notes": [
+ "Fixed layout views can support wall visibility, wall-id consistency, wall attach reasoning, ceiling/floor evidence, object placement checks, and description revision, but they are not primary wall coordinate sources."
+ ],
+ },
+ "topdown": {
+ "canonical_topdown_image_path": image_path,
+ "canonical_grid_mapping_path": mapping_path,
+ "usage": "verification_only",
+ "source": topdown_mapping_state.get("source"),
+ "fallback_used": topdown_mapping_state.get("fallback_used"),
+ "topdown_camera_config_path": str(camera_config_path) if camera_config_path else None,
+ "render_grid_affine_path": render_grid_wall_candidates.get("render_grid_affine_path") if isinstance(render_grid_wall_candidates, dict) else None,
+ "topdown_assets": topdown_assets.get("png_files", []) + topdown_assets.get("json_files", []),
+ },
+ "canonical_topdown": {
+ "image_path": image_path,
+ "grid_mapping_path": mapping_path,
+ "usage": "verification_only",
+ "source": topdown_mapping_state.get("source"),
+ "fallback_used": topdown_mapping_state.get("fallback_used"),
+ },
+ "grid_mapping_summary": grid_summary,
+ "visual_wall_extraction": {
+ "source": "obb_from_mesh_footprint_min_rect",
+ "status": "formal_obb_wall_coordinates_required",
+ "canonical_topdown_image_path": image_path,
+ "canonical_grid_mapping_path": mapping_path,
+ "fallback_used": topdown_mapping_state.get("fallback_used"),
+ "note": "Final wall coordinates are generated by obb_from_mesh_footprint_min_rect in OpenGL XZ. Top-down/grid/fixed views verify and audit those coordinates; they are not alternate wall coordinate sources.",
+ },
+ "layout_bounds": layout_bounds,
+ "grid_mapping_bounds": mapping_bounds,
+ "camera_config_bounds": camera_bounds,
+ "object_bbox_bounds": object_bounds,
+ "wall_candidate_mode": wall_candidate_mode,
+ "active_wall_candidate_source": active_polygon_source,
+ "non_authoritative_wall_hints": non_authoritative_wall_hints,
+ "object_footprints": active_footprints,
+ "wall_attach_geometry": active_wall_attach_geometry,
+ "render_grid_wall_candidates": render_grid_wall_candidates,
+ "render_grid_object_footprints": render_grid_object_footprints,
+ "render_grid_wall_attach_geometry": render_grid_wall_attach_geometry,
+ "candidate_room_polygon_hint": {
+ "available": bool(active_polygon_points),
+ "points_xz": active_polygon_points,
+ "source": active_polygon_source,
+ "trust_level": active_polygon_trust_level,
+ "may_be_used_as_final_walls": False,
+ },
+ "candidate_walls": active_candidate_walls,
+ "hint_only_sources": {
+ "candidate_room_polygon_hint": {
+ "available": bool(active_polygon_points),
+ "points_xz": active_polygon_points,
+ "source": active_polygon_source,
+ "trust_level": active_polygon_trust_level,
+ "may_be_used_as_final_walls": False,
+ },
+ "candidate_walls": active_candidate_walls,
+ "render_grid_wall_candidates": render_grid_wall_candidates,
+ "layout_bounds": layout_bounds,
+ "object_bbox_bounds": object_bounds,
+ "may_be_used_as_final_walls": False,
+ },
+ "wall_canonical_naming_policy": canonical_wall_naming_policy(),
+ "wall_output_requirements": {
+ "wall_canonical_naming_policy": canonical_wall_naming_policy(),
+ "required_wall_ids": WALL_IDS,
+ "coordinate_format": "[x, z]",
+ "clockwise_required": True,
+ "fields": ["id", "first_point", "second_point"],
+ "adjacent_endpoint_consistency_required": True,
+ "required_fields_for_task_output": ["id", "first_point", "second_point", "source", "evidence", "topdown_verification"],
+ "final_eligible_sources": [FORMAL_WALL_METHOD],
+ "intermediate_sources": ["wall_attached_object_distribution", "agent_visibility_review"],
+ "formal_authoritative_wall_method": FORMAL_WALL_METHOD,
+ "insufficient_evidence_status": "missing_formal_render_grid_wall_evidence",
+ "hint_only_sources_cannot_be_final_source": True,
+ },
+ "warnings": warnings,
+ }
+
+
+def write_wall_evidence(wall_evidence: dict[str, Any], output_dir: Path) -> None:
+ write_json(output_dir / "wall_evidence.json", wall_evidence)
+ lines = [
+ f"- scene_name: `{wall_evidence.get('scene_name')}`",
+ "- coordinate_source_definition: `final wall coordinates come from the formal OBB mesh-footprint method in OpenGL XZ; visual assets provide verification`",
+ f"- layout_xz_coordinate_system: `{wall_evidence.get('layout_xz_evidence', {}).get('coordinate_system')}`",
+ f"- canonical_topdown_image_path: `{wall_evidence.get('topdown_verification', {}).get('canonical_topdown_image_path')}`",
+ f"- canonical_grid_mapping_path: `{wall_evidence.get('topdown_verification', {}).get('canonical_grid_mapping_path')}`",
+ f"- topdown_usage: `{wall_evidence.get('topdown_verification', {}).get('usage')}`",
+ f"- fixed_views_manifest: `{wall_evidence.get('fixed_layout_views_verification', {}).get('manifest_path')}`",
+ f"- fixed_views_usage: `{wall_evidence.get('fixed_layout_views_verification', {}).get('usage')}`",
+ f"- topdown_camera_config_path: `{wall_evidence.get('topdown_verification', {}).get('topdown_camera_config_path')}`",
+ f"- canonical_source: `{wall_evidence.get('topdown_verification', {}).get('source')}`",
+ f"- fallback_used: `{wall_evidence.get('topdown_verification', {}).get('fallback_used')}`",
+ f"- visual_wall_extraction_status: `{wall_evidence.get('visual_wall_extraction', {}).get('status')}`",
+ f"- wall_candidate_mode: `{wall_evidence.get('wall_candidate_mode')}`",
+ f"- active_wall_candidate_source: `{wall_evidence.get('active_wall_candidate_source')}`",
+ f"- non_authoritative_wall_hints: `{bool(wall_evidence.get('non_authoritative_wall_hints'))}`",
+ f"- grid_mapping_available: `{wall_evidence.get('grid_mapping_summary', {}).get('available')}`",
+ f"- grid_mapping_coordinate_range: `{wall_evidence.get('grid_mapping_summary', {}).get('coordinate_range')}`",
+ f"- candidate_polygon_source: `{wall_evidence.get('candidate_room_polygon_hint', {}).get('source')}`",
+ f"- candidate_polygon_trust: `{wall_evidence.get('candidate_room_polygon_hint', {}).get('trust_level')}`",
+ f"- candidate_may_be_final: `{wall_evidence.get('candidate_room_polygon_hint', {}).get('may_be_used_as_final_walls')}`",
+ f"- candidate_walls_count: `{len(wall_evidence.get('candidate_walls', []))}`",
+ f"- object_footprints_count: `{len(wall_evidence.get('object_footprints', []))}`",
+ f"- wall_attach_geometry_count: `{len(wall_evidence.get('wall_attach_geometry', []))}`",
+ f"- wall_canonical_naming_policy: `{wall_evidence.get('wall_canonical_naming_policy', {}).get('schema_version')}`",
+ "- wall_001_final_rule: `3 visible walls -> middle visible wall; 2 visible walls -> left visible wall; remaining walls clockwise`",
+ f"- render_grid_status: `{wall_evidence.get('render_grid_wall_candidates', {}).get('status')}`",
+ f"- render_grid_source_path: `{wall_evidence.get('render_grid_wall_candidates', {}).get('source_path')}`",
+ f"- render_grid_primary_method: `{wall_evidence.get('render_grid_wall_candidates', {}).get('method_policy', {}).get('primary_method')}`",
+ f"- render_grid_attach_geometry_count: `{len(wall_evidence.get('render_grid_wall_attach_geometry', []))}`",
+ "",
+ "## Wall Canonical Naming Policy",
+ "```json",
+ json.dumps(wall_evidence.get("wall_canonical_naming_policy", {}), indent=2, ensure_ascii=False),
+ "```",
+ "",
+ "## Candidate Wall Hints",
+ "```json",
+ json.dumps(wall_evidence.get("candidate_walls", []), indent=2, ensure_ascii=False),
+ "```",
+ "",
+ "## External A-Line Wall Candidates",
+ "```json",
+ json.dumps(wall_evidence.get("render_grid_wall_candidates", {}), indent=2, ensure_ascii=False),
+ "```",
+ "",
+ "## Warnings",
+ ]
+ warnings = wall_evidence.get("warnings") or []
+ lines.extend([f"- {warning}" for warning in warnings] or ["- none"])
+ write_md(output_dir / "wall_evidence.md", "Wall Evidence", lines)
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Extract structured wall evidence for multi-agent scene delivery.")
+ parser.add_argument("--scene_state", required=True, type=Path)
+ parser.add_argument("--output_dir", required=True, type=Path)
+ args = parser.parse_args()
+ scene_state = json.loads(args.scene_state.read_text(encoding="utf-8"))
+ wall_evidence = build_wall_evidence(scene_state)
+ args.output_dir.mkdir(parents=True, exist_ok=True)
+ write_wall_evidence(wall_evidence, args.output_dir)
+ print(json.dumps({"wall_evidence_path": str(args.output_dir / "wall_evidence.json"), "candidate_walls": len(wall_evidence.get("candidate_walls", [])), "warnings": wall_evidence.get("warnings", [])}, indent=2, ensure_ascii=False))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/relations/tools/generate_fixed_layout_views.py b/vlm_agent/reconstruction/relations/tools/generate_fixed_layout_views.py
new file mode 100644
index 0000000..01efeee
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/generate_fixed_layout_views.py
@@ -0,0 +1,478 @@
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+ROOT = Path(__file__).resolve().parents[2]
+if str(ROOT) not in sys.path:
+ sys.path.insert(0, str(ROOT))
+
+from relations.agents.state import write_json, write_md
+
+DEFAULT_IMAGE_SIZE = 512
+
+VIEW_DIRECTIONS: dict[str, list[float]] = {
+ "top": [0.0, -1.0, 0.0],
+ "front": [0.0, 0.0, -1.0],
+ "back": [0.0, 0.0, 1.0],
+ "left": [-1.0, 0.0, 0.0],
+ "right": [1.0, 0.0, 0.0],
+}
+
+SIDE_VIEWS = {"front", "back", "left", "right"}
+
+GRID_MAPPING_USAGE_NOTE = (
+ "canonical top-down [x,z] reference only; side-view pixels should not be interpreted "
+ "as independent [x,z] coordinates"
+)
+TOP_VIEW_USAGE_NOTE = "top grid image may show coordinate overlay; clean image is preferred for visual object inspection"
+
+
+def _load_json(path: Path) -> dict[str, Any]:
+ try:
+ data = json.loads(path.read_text(encoding="utf-8"))
+ except Exception:
+ return {}
+ return data if isinstance(data, dict) else {}
+
+
+def _write_manifest_md(manifest_path: Path, manifest: dict[str, Any]) -> None:
+ lines = [
+ f"- status: `{manifest.get('status')}`",
+ f"- views_dir: `{manifest.get('views_dir')}`",
+ f"- canonical_grid_mapping_path: `{manifest.get('canonical_grid_mapping_path')}`",
+ f"- render_script: `{manifest.get('render_script')}`",
+ f"- render_python: `{manifest.get('render_python')}`",
+ f"- grid_mapping_usage_note: `{manifest.get('grid_mapping_usage_note')}`",
+ "",
+ "## Views",
+ ]
+ for item in manifest.get("views", []):
+ image_path = item.get("clean_image_path") or item.get("image_path")
+ lines.extend(
+ [
+ f"- `{item.get('view')}`: status `{item.get('status')}`, reused `{item.get('reused_existing')}`",
+ f" - image: `{image_path}`",
+ f" - clean_image: `{item.get('clean_image_path')}`",
+ f" - grid_image: `{item.get('grid_image_path')}`",
+ f" - metadata: `{item.get('metadata_path')}`",
+ f" - camera_direction: `{item.get('camera_direction')}`",
+ f" - draw_grid_overlay: `{item.get('draw_grid_overlay')}`",
+ f" - is_clean_view: `{item.get('is_clean_view')}`",
+ ]
+ )
+ write_md(manifest_path.with_suffix(".md"), "Fixed Layout Views Manifest", lines)
+
+
+def _write_render_log(scene_state: dict[str, Any], output_dir: Path, calls: list[dict[str, Any]], manifest: dict[str, Any]) -> None:
+ # Runner output is <run>/generated_evidence/views, while standalone
+ # fixed-views precompute writes directly to its own precompute dir.
+ # Keep logs beside the owner of the generated evidence in both cases.
+ if output_dir.parent.name == "generated_evidence" and len(output_dir.parents) > 1:
+ log_root = output_dir.parents[1]
+ else:
+ log_root = output_dir
+ log_dir = log_root / "debug" / "rendering"
+ log_dir.mkdir(parents=True, exist_ok=True)
+ log_path = log_dir / "fixed_views_render_log.json"
+ existing = _load_json(log_path)
+ log = existing if existing else {"render_calls": []}
+ log.setdefault("render_calls", []).extend(calls)
+ log["latest_manifest_path"] = str(output_dir / "views_manifest.json")
+ log["latest_status"] = manifest.get("status")
+ write_json(log_path, log)
+ lines = [
+ f"- latest_status: `{manifest.get('status')}`",
+ f"- manifest: `{output_dir / 'views_manifest.json'}`",
+ f"- canonical_grid_mapping_path: `{manifest.get('canonical_grid_mapping_path')}`",
+ "",
+ "## Latest Calls",
+ ]
+ for call in calls:
+ lines.extend(
+ [
+ f"- `{call.get('view')}` `{call.get('variant') or 'default'}`: `{call.get('status')}`",
+ f" - output_png: `{call.get('output_png')}`",
+ f" - output_json: `{call.get('output_json')}`",
+ f" - used_grid_mapping: `{call.get('uses_canonical_grid_mapping')}`",
+ f" - passes_grid_mapping_to_renderer: `{call.get('passes_grid_mapping_to_renderer')}`",
+ f" - draw_grid_overlay: `{call.get('draw_grid_overlay')}`",
+ ]
+ )
+ write_md(log_dir / "fixed_views_render_log.md", "Fixed Views Render Log", lines)
+
+
+def _current_manifest_ok(manifest_path: Path, canonical_mapping_path: str, views: dict[str, list[float]]) -> bool:
+ manifest = _load_json(manifest_path)
+ if not manifest or manifest.get("canonical_grid_mapping_path") != canonical_mapping_path:
+ return False
+ items = manifest.get("views")
+ if not isinstance(items, list) or {item.get("view") for item in items} != set(views):
+ return False
+ for item in items:
+ view = item.get("view")
+ metadata_path = item.get("metadata_path")
+ if not metadata_path or not Path(str(metadata_path)).exists():
+ return False
+ if view == "top":
+ if item.get("draw_grid_overlay") is not True:
+ return False
+ if not item.get("clean_image_path") or not Path(str(item.get("clean_image_path"))).exists():
+ return False
+ if not item.get("grid_image_path") or not Path(str(item.get("grid_image_path"))).exists():
+ return False
+ elif view in SIDE_VIEWS:
+ if item.get("draw_grid_overlay") is not False or item.get("is_clean_view") is not True:
+ return False
+ if not item.get("image_path") or not Path(str(item.get("image_path"))).exists():
+ return False
+ else:
+ return False
+ return True
+
+
+def reuse_fixed_layout_views_cache(scene_state: dict[str, Any], cache_dir: Path, output_dir: Path) -> dict[str, Any]:
+ """Reuse a previously generated fixed-views manifest without copying image files.
+
+ The returned manifest is written into the current run's output directory for
+ provenance, while each view item keeps its original image/metadata paths.
+ This makes cached evidence explicit and avoids accidental regeneration when
+ a new run writes to a fresh output directory.
+ """
+ output_dir = output_dir.resolve()
+ output_dir.mkdir(parents=True, exist_ok=True)
+ manifest_path = output_dir / "views_manifest.json"
+ cache_dir = cache_dir.resolve()
+ cache_manifest_path = cache_dir / "views_manifest.json"
+ canonical_mapping = scene_state.get("topdown_grid_mapping", {}).get("canonical_mapping_path")
+ if not canonical_mapping:
+ return {
+ "status": "missing_or_needs_generation",
+ "reason": "scene_state.topdown_grid_mapping.canonical_mapping_path is missing",
+ "views_dir": str(output_dir),
+ "manifest_path": str(manifest_path),
+ "cache_source_dir": str(cache_dir),
+ "views": [],
+ }
+ canonical_mapping_path = Path(canonical_mapping)
+ if not _current_manifest_ok(cache_manifest_path, str(canonical_mapping_path), VIEW_DIRECTIONS):
+ return {
+ "status": "missing_or_needs_generation",
+ "reason": f"fixed views cache is missing, stale, or incomplete: {cache_manifest_path}",
+ "views_dir": str(output_dir),
+ "manifest_path": str(manifest_path),
+ "cache_source_dir": str(cache_dir),
+ "cache_manifest_path": str(cache_manifest_path),
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "views": [],
+ }
+
+ manifest = _load_json(cache_manifest_path)
+ for item in manifest.get("views", []):
+ item["reused_existing"] = True
+ item["cache_source_dir"] = str(cache_dir)
+ manifest["status"] = "available"
+ manifest["reused_existing"] = True
+ manifest["cache_status"] = "reused_from_fixed_views_cache_dir"
+ manifest["cache_source_dir"] = str(cache_dir)
+ manifest["cache_manifest_path"] = str(cache_manifest_path)
+ manifest["manifest_path"] = str(manifest_path)
+ manifest["current_run_views_manifest_path"] = str(manifest_path)
+ write_json(manifest_path, manifest)
+ _write_manifest_md(manifest_path, manifest)
+ _write_render_log(scene_state, output_dir, [], manifest)
+ return manifest
+
+
+def _render_python(scene_state: dict[str, Any]) -> str:
+ configured = scene_state.get("evidence", {}).get("render_python")
+ if configured and Path(configured).exists():
+ return str(configured)
+ return sys.executable
+
+
+def _scene_value(scene_state: dict[str, Any], key: str) -> str:
+ value = scene_state.get("evidence", {}).get(key)
+ if not value:
+ raise ValueError(f"scene_state.evidence.{key} is missing")
+ return str(value)
+
+
+def _direction_args(direction: list[float]) -> list[str]:
+ return [str(int(v)) if isinstance(v, float) and v.is_integer() else str(v) for v in direction]
+
+
+def _base_command(render_python: str, render_script: str, layout_dir: str, unposed_dir: str, direction: list[float], output: Path) -> list[str]:
+ return [
+ render_python,
+ render_script,
+ "--layout_dir",
+ layout_dir,
+ "--unposed_dir",
+ unposed_dir,
+ "--camera_direction",
+ *_direction_args(direction),
+ "--output",
+ str(output),
+ "--image_size",
+ str(DEFAULT_IMAGE_SIZE),
+ ]
+
+
+def _write_view_metadata(path: Path, metadata: dict[str, Any]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ write_json(path, metadata)
+
+
+def _run_command(command: list[str], dry_run: bool) -> dict[str, Any]:
+ if dry_run:
+ return {"status": "skipped_dry_run"}
+ proc = subprocess.run(command, text=True, capture_output=True, check=False)
+ return {
+ "status": "success" if proc.returncode == 0 else "failed",
+ "returncode": proc.returncode,
+ "stdout": proc.stdout,
+ "stderr": proc.stderr,
+ }
+
+
+def ensure_fixed_layout_views(scene_state: dict[str, Any], output_dir: Path, dry_run: bool = False) -> dict[str, Any]:
+ output_dir = output_dir.resolve()
+ output_dir.mkdir(parents=True, exist_ok=True)
+ manifest_path = output_dir / "views_manifest.json"
+ canonical_mapping = scene_state.get("topdown_grid_mapping", {}).get("canonical_mapping_path")
+ if not canonical_mapping:
+ return {
+ "status": "missing_or_needs_generation",
+ "reason": "scene_state.topdown_grid_mapping.canonical_mapping_path is missing",
+ "views_dir": str(output_dir),
+ "manifest_path": str(manifest_path),
+ "views": [],
+ }
+ canonical_mapping_path = Path(canonical_mapping)
+ if not canonical_mapping_path.exists():
+ return {
+ "status": "missing_or_needs_generation",
+ "reason": f"canonical grid mapping does not exist: {canonical_mapping_path}",
+ "views_dir": str(output_dir),
+ "manifest_path": str(manifest_path),
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "views": [],
+ }
+
+ render_script = _scene_value(scene_state, "render_script")
+ layout_dir = _scene_value(scene_state, "layout_params_dir")
+ unposed_dir = _scene_value(scene_state, "unposed_objects_dir")
+ render_python = _render_python(scene_state)
+
+ if _current_manifest_ok(manifest_path, str(canonical_mapping_path), VIEW_DIRECTIONS):
+ manifest = _load_json(manifest_path)
+ for item in manifest.get("views", []):
+ item["reused_existing"] = True
+ manifest["status"] = "available"
+ manifest["reused_existing"] = True
+ write_json(manifest_path, manifest)
+ _write_manifest_md(manifest_path, manifest)
+ _write_render_log(scene_state, output_dir, [], manifest)
+ return manifest
+
+ calls: list[dict[str, Any]] = []
+ view_items: list[dict[str, Any]] = []
+ any_failed = False
+
+ for view, direction in VIEW_DIRECTIONS.items():
+ view_dir = output_dir / view
+ view_dir.mkdir(parents=True, exist_ok=True)
+ metadata_path = view_dir / f"{view}_view.json"
+
+ if view == "top":
+ clean_image_path = view_dir / "top_view_clean.png"
+ grid_image_path = view_dir / "top_view_grid.png"
+ grid_render_metadata_path = grid_image_path.with_suffix(".json")
+ clean_command = _base_command(render_python, render_script, layout_dir, unposed_dir, direction, clean_image_path)
+ grid_command = _base_command(render_python, render_script, layout_dir, unposed_dir, direction, grid_image_path)
+ grid_command.extend(["--add_grid", "--draw_grid_lines", "--grid_mapping", str(canonical_mapping_path)])
+
+ item = {
+ "view": view,
+ "image_path": str(clean_image_path),
+ "clean_image_path": str(clean_image_path),
+ "grid_image_path": str(grid_image_path),
+ "metadata_path": str(metadata_path),
+ "grid_render_metadata_path": str(grid_render_metadata_path),
+ "camera_direction": direction,
+ "render_script": render_script,
+ "render_python": render_python,
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "uses_canonical_grid_mapping": True,
+ "draw_grid_overlay": True,
+ "is_clean_view": False,
+ "image_size": DEFAULT_IMAGE_SIZE,
+ "grid_mapping_usage_note": TOP_VIEW_USAGE_NOTE,
+ "notes": TOP_VIEW_USAGE_NOTE,
+ "reused_existing": False,
+ "status": "skipped_dry_run" if dry_run else "pending",
+ }
+ top_calls = [
+ {
+ "event": "fixed_layout_view_render",
+ "view": view,
+ "variant": "clean",
+ "camera_direction": direction,
+ "output_png": str(clean_image_path),
+ "output_json": str(metadata_path),
+ "uses_canonical_grid_mapping": True,
+ "passes_grid_mapping_to_renderer": False,
+ "draw_grid_overlay": False,
+ "image_size": DEFAULT_IMAGE_SIZE,
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "grid_mapping_usage_note": TOP_VIEW_USAGE_NOTE,
+ "command": clean_command,
+ "status": "skipped_dry_run" if dry_run else "pending",
+ },
+ {
+ "event": "fixed_layout_view_render",
+ "view": view,
+ "variant": "grid",
+ "camera_direction": direction,
+ "output_png": str(grid_image_path),
+ "output_json": str(grid_render_metadata_path),
+ "uses_canonical_grid_mapping": True,
+ "passes_grid_mapping_to_renderer": True,
+ "draw_grid_overlay": True,
+ "image_size": DEFAULT_IMAGE_SIZE,
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "grid_mapping_usage_note": TOP_VIEW_USAGE_NOTE,
+ "command": grid_command,
+ "status": "skipped_dry_run" if dry_run else "pending",
+ },
+ ]
+ if not dry_run:
+ top_failed = False
+ for call in top_calls:
+ result = _run_command(call["command"], dry_run=False)
+ call.update(result)
+ if result.get("status") != "success":
+ top_failed = True
+ _write_view_metadata(
+ metadata_path,
+ {
+ "view": view,
+ "camera_direction": direction,
+ "clean_image_path": str(clean_image_path),
+ "grid_image_path": str(grid_image_path),
+ "grid_render_metadata_path": str(grid_render_metadata_path),
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "uses_canonical_grid_mapping": True,
+ "draw_grid_overlay": True,
+ "clean_draw_grid_overlay": False,
+ "grid_draw_grid_overlay": True,
+ "image_size": DEFAULT_IMAGE_SIZE,
+ "notes": TOP_VIEW_USAGE_NOTE,
+ },
+ )
+ if top_failed or not clean_image_path.exists() or not grid_image_path.exists() or not metadata_path.exists():
+ any_failed = True
+ item["status"] = "failed"
+ else:
+ item["status"] = "success"
+ calls.extend(top_calls)
+ view_items.append(item)
+ continue
+
+ image_path = view_dir / f"{view}_view.png"
+ command = _base_command(render_python, render_script, layout_dir, unposed_dir, direction, image_path)
+ item = {
+ "view": view,
+ "image_path": str(image_path),
+ "metadata_path": str(metadata_path),
+ "camera_direction": direction,
+ "render_script": render_script,
+ "render_python": render_python,
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "uses_canonical_grid_mapping": True,
+ "draw_grid_overlay": False,
+ "is_clean_view": True,
+ "image_size": DEFAULT_IMAGE_SIZE,
+ "grid_mapping_usage_note": GRID_MAPPING_USAGE_NOTE,
+ "reused_existing": False,
+ "status": "skipped_dry_run" if dry_run else "pending",
+ }
+ call = {
+ "event": "fixed_layout_view_render",
+ "view": view,
+ "camera_direction": direction,
+ "output_png": str(image_path),
+ "output_json": str(metadata_path),
+ "uses_canonical_grid_mapping": True,
+ "passes_grid_mapping_to_renderer": False,
+ "draw_grid_overlay": False,
+ "image_size": DEFAULT_IMAGE_SIZE,
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "grid_mapping_usage_note": GRID_MAPPING_USAGE_NOTE,
+ "command": command,
+ "status": "skipped_dry_run" if dry_run else "pending",
+ }
+ if not dry_run:
+ result = _run_command(command, dry_run=False)
+ call.update(result)
+ item["status"] = result["status"]
+ if result["status"] == "success" and image_path.exists():
+ _write_view_metadata(
+ metadata_path,
+ {
+ "view": view,
+ "camera_direction": direction,
+ "image_path": str(image_path),
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "uses_canonical_grid_mapping": True,
+ "draw_grid_overlay": False,
+ "is_clean_view": True,
+ "passes_grid_mapping_to_renderer": False,
+ "grid_mapping_usage_note": GRID_MAPPING_USAGE_NOTE,
+ "image_size": DEFAULT_IMAGE_SIZE,
+ },
+ )
+ if result["status"] != "success" or not image_path.exists() or not metadata_path.exists():
+ any_failed = True
+ calls.append(call)
+ view_items.append(item)
+
+ manifest = {
+ "status": "failed" if any_failed else "available",
+ "scene_name": scene_state.get("scene_name"),
+ "views_dir": str(output_dir),
+ "manifest_path": str(manifest_path),
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "uses_canonical_grid_mapping": True,
+ "render_script": render_script,
+ "render_python": render_python,
+ "view_convention": "front/back/left/right are render camera directions in SceneMosaic world XYZ; canonical grid remains top-down [x,z].",
+ "grid_mapping_usage_note": GRID_MAPPING_USAGE_NOTE,
+ "side_view_policy": "side views record the canonical grid mapping as shared top-down [x,z] reference metadata only; grid mapping is not passed to the side-view renderer and no grid overlay is drawn.",
+ "image_size": DEFAULT_IMAGE_SIZE,
+ "views": view_items,
+ }
+ write_json(manifest_path, manifest)
+ _write_manifest_md(manifest_path, manifest)
+ _write_render_log(scene_state, output_dir, calls, manifest)
+ return manifest
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Generate fixed multi-view layout evidence for multi-agent attach reasoning.")
+ parser.add_argument("--scene_state", required=True, type=Path)
+ parser.add_argument("--output_dir", required=True, type=Path)
+ parser.add_argument("--dry_run", action="store_true")
+ args = parser.parse_args()
+ scene_state = json.loads(args.scene_state.read_text(encoding="utf-8"))
+ manifest = ensure_fixed_layout_views(scene_state, args.output_dir, dry_run=args.dry_run)
+ print(json.dumps(manifest, indent=2, ensure_ascii=False))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/relations/tools/render_grid_wall_precompute.py b/vlm_agent/reconstruction/relations/tools/render_grid_wall_precompute.py
new file mode 100644
index 0000000..afa0c8c
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/render_grid_wall_precompute.py
@@ -0,0 +1,686 @@
+#!/usr/bin/env python3
+"""Generate render-grid wall candidates inside a multi-agent run.
+
+This is the formal OBB render-grid precompute used by relation extraction. It writes all top-down, mask, affine, candidate, and
+overlay assets into the current multi-agent run so later stages consume one
+shared grid.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import sys
+from pathlib import Path
+from typing import Any, Sequence
+
+import cv2
+import numpy as np
+
+CODE_DIR = Path(__file__).resolve().parents[2]
+REPOSITORY_ROOT = Path(__file__).resolve().parents[4]
+if str(REPOSITORY_ROOT) not in sys.path:
+ sys.path.insert(0, str(REPOSITORY_ROOT))
+POLY_TOOLS = REPOSITORY_ROOT / "vlm_agent" / "evolution" / "tools"
+if str(CODE_DIR) not in sys.path:
+ sys.path.insert(0, str(CODE_DIR))
+if str(POLY_TOOLS) not in sys.path:
+ sys.path.insert(0, str(POLY_TOOLS))
+
+from relations.tools.wall_geometry import ( # noqa: E402
+ candidate_metrics,
+ canonical_clockwise_rect,
+ dominant_rect_angle_degrees,
+ layout_wall_polygon,
+ min_area_rect,
+ rect_from_angle,
+ xz_to_pixel,
+)
+from relations.tools.wall_orientation_candidates import ( # noqa: E402
+ axis_angle_distance_degrees,
+ cluster_object_orientation_axes,
+ normalize_axis_angle_degrees,
+)
+from relations.tools.wall_canonical_naming import canonical_screen_clockwise_segments_from_polygon # noqa: E402
+from render_scene_from_layout import ( # noqa: E402
+ build_camera_pose,
+ compose_scene,
+ denormalize_point,
+ generate_2d_grid_on_plane,
+ get_camera_matrix,
+ project_3d_to_2d,
+ render_scene_with_grid,
+ resolve_orthographic_camera_frame,
+ save_grid_mapping,
+)
+
+from vlm_agent.run_paths import ScenePaths # noqa: E402
+
+Point = tuple[float, float]
+# Formal wall coordinates come from posed mesh footprints projected to OpenGL
+# XZ and enclosed by the minimum rotated rectangle. Alternative coordinate
+# estimators are intentionally excluded from the publication path.
+PRIMARY_METHOD = "obb_from_mesh_footprint_min_rect"
+SUPPORTING_METHODS: list[str] = []
+
+
+def read_json(path: Path) -> Any:
+ with path.open("r", encoding="utf-8") as handle:
+ return json.load(handle)
+
+
+def write_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with path.open("w", encoding="utf-8") as handle:
+ json.dump(payload, handle, ensure_ascii=False, indent=2)
+ handle.write("\n")
+
+
+def save_rgb(path: Path, image_rgb: np.ndarray) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ if not cv2.imwrite(str(path), cv2.cvtColor(image_rgb, cv2.COLOR_RGB2BGR)):
+ raise RuntimeError(f"failed to save image: {path}")
+
+
+def pixel_affines(
+ mapping: dict[tuple[int, int], Sequence[float]],
+ transform_info: dict[str, Any],
+ camera_pose: np.ndarray,
+ camera_matrix: Any,
+ image_size: int,
+) -> tuple[np.ndarray, np.ndarray, list[dict[str, Any]]]:
+ pixel_rows = []
+ xz_rows = []
+ samples: list[dict[str, Any]] = []
+ for key, point_normalized in mapping.items():
+ pixel = project_3d_to_2d(point_normalized, camera_pose, camera_matrix, image_size)
+ if pixel is None:
+ continue
+ original = denormalize_point(point_normalized, transform_info)
+ u, v = float(pixel[0]), float(pixel[1])
+ x, z = float(original[0]), float(original[2])
+ pixel_rows.append([u, v, 1.0])
+ xz_rows.append([x, z])
+ samples.append({
+ "grid_i": int(key[0]),
+ "grid_j": int(key[1]),
+ "pixel": [u, v],
+ "point_3d_original": original.tolist(),
+ "point_3d_normalized": list(point_normalized),
+ })
+ if len(samples) < 4:
+ raise RuntimeError("not enough projected grid samples for affine fit")
+ pixel_to_xz, *_ = np.linalg.lstsq(np.asarray(pixel_rows), np.asarray(xz_rows), rcond=None)
+ xz_rows_h = [[x, z, 1.0] for x, z in xz_rows]
+ pixel_output = [[row[0], row[1]] for row in pixel_rows]
+ xz_to_pixel_matrix, *_ = np.linalg.lstsq(np.asarray(xz_rows_h), np.asarray(pixel_output), rcond=None)
+ return pixel_to_xz.astype(np.float64), xz_to_pixel_matrix.astype(np.float64), samples
+
+
+def foreground_mask(image_rgb: np.ndarray) -> np.ndarray:
+ height, width = image_rgb.shape[:2]
+ border_h = max(4, height // 32)
+ border_w = max(4, width // 32)
+ corners = np.vstack([
+ image_rgb[:border_h, :border_w].reshape(-1, 3),
+ image_rgb[-border_h:, :border_w].reshape(-1, 3),
+ image_rgb[:border_h, -border_w:].reshape(-1, 3),
+ image_rgb[-border_h:, -border_w:].reshape(-1, 3),
+ ])
+ background = np.median(corners.astype(np.float32), axis=0)
+ distance = np.linalg.norm(image_rgb.astype(np.float32) - background.reshape(1, 1, 3), axis=2)
+ mask = (distance > 14.0).astype(np.uint8) * 255
+ kernel = np.ones((3, 3), np.uint8)
+ mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)
+ return cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel, iterations=2)
+
+
+def draw_candidates(
+ image_rgb: np.ndarray,
+ candidates: Sequence[dict[str, Any]],
+ xz_to_pixel_matrix: np.ndarray,
+ path: Path,
+) -> None:
+ colors = [(226, 35, 54), (30, 153, 139), (46, 134, 171), (255, 169, 77)]
+ canvas = image_rgb.copy()
+ for index, candidate in enumerate(candidates):
+ polygon = [(float(x), float(z)) for x, z in candidate["metrics"]["polygon_xz"]]
+ points = np.asarray(xz_to_pixel(polygon, xz_to_pixel_matrix), dtype=np.int32)
+ color = colors[index % len(colors)]
+ cv2.polylines(canvas, [points], isClosed=True, color=color, thickness=3)
+ x, y = points[0]
+ label = str(candidate.get("candidate_id") or f'{index + 1}:{candidate["method"]}')
+ cv2.putText(canvas, label, (int(x) + 4, int(y) - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.55, color, 2)
+ save_rgb(path, canvas)
+
+
+def _grid_to_pixel_affine(projected_samples: Sequence[dict[str, Any]]) -> np.ndarray | None:
+ rows = []
+ pixels = []
+ for sample in projected_samples:
+ if not isinstance(sample, dict):
+ continue
+ pixel = sample.get("pixel")
+ if not isinstance(pixel, list) or len(pixel) != 2:
+ continue
+ rows.append([float(sample["grid_i"]), float(sample["grid_j"]), 1.0])
+ pixels.append([float(pixel[0]), float(pixel[1])])
+ if len(rows) < 4:
+ return None
+ matrix, *_ = np.linalg.lstsq(np.asarray(rows), np.asarray(pixels), rcond=None)
+ return matrix.astype(np.float64)
+
+
+def _draw_object_direction_arrows(
+ canvas: np.ndarray,
+ objects_info: Sequence[dict[str, Any]],
+ grid_to_pixel: np.ndarray | None,
+ *,
+ highlighted_object_ids: set[str] | None = None,
+ highlight_color: tuple[int, int, int] = (226, 35, 54),
+) -> None:
+ if grid_to_pixel is None:
+ return
+ for item in objects_info:
+ if not isinstance(item, dict):
+ continue
+ center = item.get("center_2d")
+ size = item.get("size_2d")
+ rotation = item.get("rotation_degrees")
+ object_id = str(item.get("object_id") or "")
+ if not object_id or not isinstance(center, list) or len(center) != 2:
+ continue
+ if not isinstance(size, list) or len(size) != 2 or not isinstance(rotation, (int, float)):
+ continue
+ center_row = np.asarray([float(center[0]), float(center[1]), 1.0]) @ grid_to_pixel
+ scale_row = np.asarray([
+ float(center[0]) + max(float(size[0]), float(size[1])) * 0.35,
+ float(center[1]),
+ 1.0,
+ ]) @ grid_to_pixel
+ arrow_length = max(18.0, min(48.0, float(np.linalg.norm(scale_row - center_row))))
+ theta = math.radians(float(rotation))
+ start = (int(round(center_row[0])), int(round(center_row[1])))
+ end = (
+ int(round(center_row[0] - arrow_length * math.sin(theta))),
+ int(round(center_row[1] + arrow_length * math.cos(theta))),
+ )
+ highlighted = highlighted_object_ids is not None and object_id in highlighted_object_ids
+ color = highlight_color if highlighted else (70, 70, 70)
+ thickness = 3 if highlighted else 1
+ cv2.arrowedLine(canvas, start, end, color, thickness, cv2.LINE_AA, tipLength=0.25)
+ cv2.putText(
+ canvas,
+ object_id.replace("object_", "o"),
+ (start[0] + 4, start[1] - 4),
+ cv2.FONT_HERSHEY_SIMPLEX,
+ 0.42,
+ color,
+ thickness,
+ cv2.LINE_AA,
+ )
+
+
+def draw_orientation_review_overlay(
+ image_rgb: np.ndarray,
+ candidates: Sequence[dict[str, Any]],
+ xz_to_pixel_matrix: np.ndarray,
+ objects_info: Sequence[dict[str, Any]],
+ projected_samples: Sequence[dict[str, Any]],
+ path: Path,
+) -> None:
+ """Draw each finite room-axis choice in an isolated contact-sheet panel."""
+ colors = [(74, 93, 117), (226, 35, 54), (30, 153, 139), (46, 134, 171)]
+ grid_to_pixel = _grid_to_pixel_affine(projected_samples)
+ panel_width = min(640, image_rgb.shape[1])
+ panels: list[np.ndarray] = []
+ for index, candidate in enumerate(candidates):
+ canvas = image_rgb.copy()
+ color = colors[index % len(colors)]
+ candidate_id = str(candidate.get("candidate_id") or f"candidate_{index + 1}")
+ orientation = candidate.get("orientation", {}) if isinstance(candidate.get("orientation"), dict) else {}
+ angle = float(orientation.get("angle_degrees_xz") or 0.0)
+ supports = orientation.get("supporting_object_ids", []) if isinstance(orientation.get("supporting_object_ids"), list) else []
+ support_ids = {str(item) for item in supports}
+ _draw_object_direction_arrows(
+ canvas,
+ objects_info,
+ grid_to_pixel,
+ highlighted_object_ids=support_ids if support_ids else None,
+ highlight_color=color,
+ )
+ polygon = [(float(x), float(z)) for x, z in candidate["metrics"]["polygon_xz"]]
+ points = np.asarray(xz_to_pixel(polygon, xz_to_pixel_matrix), dtype=np.int32)
+ cv2.polylines(canvas, [points], isClosed=True, color=color, thickness=5, lineType=cv2.LINE_AA)
+ segments, _ = canonical_screen_clockwise_segments_from_polygon(
+ polygon,
+ xz_to_pixel_affine=xz_to_pixel_matrix.tolist(),
+ source="orientation_review_overlay",
+ method=str(candidate.get("method") or PRIMARY_METHOD),
+ role="wall_orientation_candidate",
+ )
+ for segment in segments:
+ edge_points = xz_to_pixel(
+ [tuple(segment["first_point"]), tuple(segment["second_point"])],
+ xz_to_pixel_matrix,
+ )
+ midpoint = (
+ (int(edge_points[0][0]) + int(edge_points[1][0])) // 2,
+ (int(edge_points[0][1]) + int(edge_points[1][1])) // 2,
+ )
+ cv2.putText(
+ canvas,
+ str(segment["id"]),
+ (midpoint[0] + 4, midpoint[1] - 4),
+ cv2.FONT_HERSHEY_SIMPLEX,
+ 0.58,
+ color,
+ 2,
+ cv2.LINE_AA,
+ )
+
+ scale = panel_width / float(canvas.shape[1])
+ content_height = max(1, int(round(canvas.shape[0] * scale)))
+ resized = cv2.resize(canvas, (panel_width, content_height), interpolation=cv2.INTER_AREA)
+ header_height = 82
+ panel = np.full((header_height + content_height, panel_width, 3), 248, dtype=np.uint8)
+ panel[header_height:, :, :] = resized
+ source = str(orientation.get("source") or "unknown")
+ cv2.putText(panel, f"{candidate_id} angle={angle:+.1f} deg", (12, 24), cv2.FONT_HERSHEY_SIMPLEX, 0.58, color, 2, cv2.LINE_AA)
+ cv2.putText(panel, f"source: {source}", (12, 47), cv2.FONT_HERSHEY_SIMPLEX, 0.42, (35, 35, 35), 1, cv2.LINE_AA)
+ short_supports = ", ".join(str(item).replace("object_", "o") for item in supports[:8])
+ support_text = f"orientation supports: {short_supports}" if short_supports else "orientation supports: none"
+ cv2.putText(panel, support_text, (12, 69), cv2.FONT_HERSHEY_SIMPLEX, 0.40, (35, 35, 35), 1, cv2.LINE_AA)
+ cv2.rectangle(panel, (0, 0), (panel.shape[1] - 1, panel.shape[0] - 1), color, 2)
+ panels.append(panel)
+
+ if not panels:
+ save_rgb(path, image_rgb)
+ return
+
+ columns = 1 if len(panels) == 1 else 2
+ rows = int(math.ceil(len(panels) / columns))
+ gap = 12
+ title_height = 44
+ panel_height = max(panel.shape[0] for panel in panels)
+ sheet_width = columns * panel_width + (columns + 1) * gap
+ sheet_height = title_height + rows * panel_height + (rows + 1) * gap
+ sheet = np.full((sheet_height, sheet_width, 3), 255, dtype=np.uint8)
+ cv2.putText(
+ sheet,
+ "Wall orientation candidates - compare isolated panels; same mesh extent",
+ (12, 29),
+ cv2.FONT_HERSHEY_SIMPLEX,
+ 0.62,
+ (20, 20, 20),
+ 2,
+ cv2.LINE_AA,
+ )
+ for index, panel in enumerate(panels):
+ row, column = divmod(index, columns)
+ x0 = gap + column * (panel_width + gap)
+ y0 = title_height + gap + row * (panel_height + gap)
+ sheet[y0:y0 + panel.shape[0], x0:x0 + panel.shape[1], :] = panel
+ save_rgb(path, sheet)
+
+def _copy_file(src: Path, dst: Path) -> None:
+ dst.parent.mkdir(parents=True, exist_ok=True)
+ dst.write_bytes(src.read_bytes())
+
+
+def _draw_single_candidate(
+ image_rgb: np.ndarray,
+ candidate: dict[str, Any],
+ xz_to_pixel_matrix: np.ndarray,
+ output_path: Path,
+ label: str,
+) -> None:
+ canvas = image_rgb.copy()
+ polygon = [(float(x), float(z)) for x, z in candidate["metrics"]["polygon_xz"]]
+ pts = np.asarray(xz_to_pixel(polygon, xz_to_pixel_matrix), dtype=np.int32)
+ color = (226, 35, 54)
+ cv2.polylines(canvas, [pts], isClosed=True, color=color, thickness=5)
+ for x, y in pts:
+ cv2.circle(canvas, (int(x), int(y)), 6, color, -1, cv2.LINE_AA)
+ x0, y0 = pts[0]
+ cv2.putText(canvas, label, (int(x0) + 5, int(y0) - 8), cv2.FONT_HERSHEY_SIMPLEX, 0.65, color, 2)
+ save_rgb(output_path, canvas)
+
+
+def sample_mesh_footprint_points(
+ posed_meshes: Sequence[Any],
+ transform_info: dict[str, Any],
+ *,
+ max_points: int = 20000,
+) -> tuple[list[Point], dict[str, Any]]:
+ """Sample all posed meshes once for both extent and orientation variants."""
+ if not posed_meshes:
+ return [], {"error": "no_posed_meshes", "point_count": 0, "object_count": 0}
+ pts: list[Point] = []
+ per_object_budget = max(64, max_points // max(1, len(posed_meshes)))
+ sampled_vertices = 0
+ total_vertices = 0
+ for mesh in posed_meshes:
+ verts = getattr(mesh, "vertices", None)
+ if verts is None:
+ continue
+ verts = np.asarray(verts, dtype=np.float32)
+ if verts.size == 0:
+ continue
+ total_vertices += int(verts.shape[0])
+ if verts.shape[0] > per_object_budget:
+ step = max(1, verts.shape[0] // per_object_budget)
+ verts = verts[::step]
+ sampled_vertices += int(verts.shape[0])
+ for vertex in verts:
+ original = denormalize_point(vertex, transform_info)
+ pts.append((float(original[0]), float(original[2])))
+ evidence = {
+ "source": "posed mesh footprint vertices projected to original OpenGL XZ",
+ "coordinate_frame": "OpenGL original XZ; Y is vertical",
+ "point_count": len(pts),
+ "sampled_vertices": sampled_vertices,
+ "total_vertices": total_vertices,
+ "object_count": len(posed_meshes),
+ "max_points": max_points,
+ }
+ if len(pts) < 4:
+ evidence["error"] = "not_enough_mesh_footprint_points"
+ return pts, evidence
+
+
+def run_precompute(
+ *,
+ scene_name: str,
+ run_root: Path,
+ output_dir: Path,
+ grid_size: int,
+ image_size: int,
+) -> dict[str, Any]:
+ output_dir = output_dir.resolve()
+ manifest_path = output_dir / "render_grid_wall_precompute_manifest.json"
+ method_candidates_path = output_dir / "method_candidates.json"
+ if method_candidates_path.exists() and manifest_path.exists():
+ manifest = read_json(manifest_path)
+ if isinstance(manifest, dict):
+ manifest["reused_existing"] = True
+ return manifest
+
+ scene_paths = ScenePaths.from_scene_root(run_root)
+ layout_dir = scene_paths.sam3d_output_dir / "layout_params"
+ unposed_dir = scene_paths.sam3d_output_dir / "unposed_objects"
+ output_dir.mkdir(parents=True, exist_ok=True)
+
+ layout_polygon = None
+
+ _, posed_meshes, layout_params, object_ids, transform_info = compose_scene(
+ layout_dir=layout_dir,
+ unposed_dir=unposed_dir,
+ requested_object_ids=None,
+ anchor_id=None,
+ )
+ camera_direction = np.asarray([0.0, -1.0, 0.0], dtype=np.float32)
+ grid_2d, mapping, _, plane_right, plane_up = generate_2d_grid_on_plane(
+ posed_meshes,
+ camera_direction,
+ grid_size,
+ plane_side="nearest",
+ )
+ base = render_scene_with_grid(posed_meshes, camera_direction, 0.0, image_size)
+ grid = render_scene_with_grid(
+ posed_meshes,
+ camera_direction,
+ 0.0,
+ image_size,
+ grid_2d=grid_2d,
+ mapping=mapping,
+ draw_grid_lines=True,
+ )
+
+ render_topdown_path = output_dir / "render_topdown.png"
+ render_topdown_grid_path = output_dir / "render_topdown_grid.png"
+ raw_mapping_path = output_dir / "render_topdown_grid_mapping_raw.json"
+ canonical_topdown_path = output_dir / "canonical_topdown.png"
+ canonical_mapping_path = output_dir / "canonical_topdown_grid_mapping.json"
+ save_rgb(render_topdown_path, base)
+ save_rgb(render_topdown_grid_path, grid)
+ _copy_file(render_topdown_path, canonical_topdown_path)
+ save_grid_mapping(
+ mapping,
+ raw_mapping_path,
+ transform_info,
+ posed_meshes=posed_meshes,
+ object_ids=object_ids,
+ axis_right=plane_right,
+ axis_up=plane_up,
+ camera_direction=camera_direction,
+ layout_params=layout_params,
+ )
+ _copy_file(raw_mapping_path, canonical_mapping_path)
+
+ camera_radius, xmag, ymag = resolve_orthographic_camera_frame(posed_meshes, camera_direction, 0.0)
+ camera_pose = build_camera_pose(camera_direction, camera_radius)
+ camera_matrix = get_camera_matrix(60.0, image_size, ortho=True, xmag=xmag, ymag=ymag)
+ pix_to_xz, xz_to_pix, projected_samples = pixel_affines(
+ mapping,
+ transform_info,
+ camera_pose,
+ camera_matrix,
+ image_size,
+ )
+ affine_path = output_dir / "render_grid_affine.json"
+ write_json(
+ affine_path,
+ {
+ "grid_size": grid_size,
+ "image_size": image_size,
+ "same_grid_policy": "current multi-agent run uses this topdown grid and affine for mask, candidates, overlays, and wall visual evidence",
+ "pixel_to_xz_affine": pix_to_xz.tolist(),
+ "xz_to_pixel_affine": xz_to_pix.tolist(),
+ "projected_grid_samples": projected_samples,
+ },
+ )
+
+ fg = foreground_mask(base)
+ mask_path = output_dir / "render_foreground_mask.png"
+ cv2.imwrite(str(mask_path), fg)
+
+ mapping_payload = read_json(raw_mapping_path)
+ objects_info = mapping_payload.get("objects", []) if isinstance(mapping_payload, dict) else []
+ mesh_points, obb_evidence = sample_mesh_footprint_points(posed_meshes, transform_info)
+ obb_polygon = None
+ if len(mesh_points) >= 4:
+ obb_polygon, _, _ = min_area_rect(mesh_points, margin=0.0)
+ obb_polygon = canonical_clockwise_rect(obb_polygon)
+ orientation_clusters = cluster_object_orientation_axes(
+ [item for item in objects_info if isinstance(item, dict)],
+ grid_size=grid_size,
+ )
+
+ candidates: list[dict[str, Any]] = []
+ if obb_polygon and len(obb_polygon) >= 4:
+ metrics = candidate_metrics(obb_polygon, layout_polygon, fg, xz_to_pix)
+ candidates.append(
+ {
+ "candidate_id": "mesh_min_area",
+ "method": PRIMARY_METHOD,
+ "metrics": metrics,
+ "orientation": {
+ "source": "mesh_footprint_minimum_area",
+ "angle_degrees_xz": round(
+ normalize_axis_angle_degrees(dominant_rect_angle_degrees(obb_polygon)),
+ 6,
+ ),
+ "supporting_object_ids": [],
+ },
+ "evidence": {
+ **obb_evidence,
+ "note": "Formal baseline OBB. Alternative mask/LSD/layout wall methods are not exposed.",
+ },
+ }
+ )
+
+ baseline_angle = (
+ float(candidates[0]["orientation"]["angle_degrees_xz"])
+ if candidates
+ else None
+ )
+ for cluster in orientation_clusters.get("clusters", []):
+ angle_degrees = float(cluster["angle_degrees_xz"])
+ if baseline_angle is not None and axis_angle_distance_degrees(angle_degrees, baseline_angle) <= 1.0:
+ continue
+ polygon = rect_from_angle(mesh_points, math.radians(angle_degrees), margin=0.0)
+ metrics = candidate_metrics(polygon, layout_polygon, fg, xz_to_pix)
+ candidates.append(
+ {
+ "candidate_id": str(cluster["cluster_id"]),
+ "method": PRIMARY_METHOD,
+ "metrics": metrics,
+ "orientation": {
+ "source": "robust_object_rotation_consensus",
+ "angle_degrees_xz": round(angle_degrees, 6),
+ "supporting_object_ids": list(cluster.get("supporting_object_ids", [])),
+ "member_count": cluster.get("member_count"),
+ "total_weight": cluster.get("total_weight"),
+ },
+ "evidence": {
+ **obb_evidence,
+ "angle_source": "rotation_degrees from the same canonical top-down render grid",
+ "orientation_cluster": cluster,
+ "note": "The complete posed-mesh footprint fixes the extent; only the rectangle axis is constrained by this object-rotation cluster.",
+ },
+ }
+ )
+
+ for idx, candidate in enumerate(candidates, start=1):
+ candidate["rank_by_policy"] = idx
+ candidate["rank_by_layout_iou_then_coverage"] = idx
+
+ all_overlay_path = output_dir / "candidate_overlay.png"
+ draw_candidates(base, candidates, xz_to_pix, all_overlay_path)
+ orientation_overlay_path = output_dir / "wall_orientation_candidates_overlay.png"
+ draw_orientation_review_overlay(
+ base,
+ candidates,
+ xz_to_pix,
+ objects_info,
+ projected_samples,
+ orientation_overlay_path,
+ )
+ primary = candidates[0] if candidates else None
+ primary_overlay_path = None
+ if primary:
+ primary_overlay_path = output_dir / "primary_candidate_overlay.png"
+ _draw_single_candidate(
+ base,
+ primary,
+ xz_to_pix,
+ primary_overlay_path,
+ f'primary:{primary.get("candidate_id", PRIMARY_METHOD)}',
+ )
+
+ result = {
+ "scene": scene_name,
+ "run_id": output_dir.name,
+ "output_dir": str(output_dir),
+ "method_family": "render_grid_wall_precompute",
+ "renderer": {
+ "layout_dir": str(layout_dir),
+ "unposed_dir": str(unposed_dir),
+ "object_count": len(object_ids),
+ "camera_direction": camera_direction.tolist(),
+ "image_size": image_size,
+ "grid_size": grid_size,
+ "same_grid_policy": "single current-run render grid reused by topdown, mask, candidate generation, overlays, and downstream wall visual evidence",
+ },
+ "method_policy": {
+ "agent_should_not_choose_method": True,
+ "orientation_selected_locally": True,
+ "primary_method": PRIMARY_METHOD,
+ "supporting_methods": SUPPORTING_METHODS,
+ "excluded_methods": [
+ "layout_walls_from_06_layout",
+ "mesh_vertices_min_area_rect",
+ "render_foreground_mask_min_area_rect",
+ "opencv_lsd_rect",
+ "opencv_hough_linesp_rect",
+ "sklearn_ransac_edge_rect",
+ ],
+ "reason": "Every candidate uses the complete posed-mesh footprint and the formal OBB method family. Local code selects one finite orientation using Agent-identified wall-following objects; the Agent cannot choose methods or invent coordinates.",
+ },
+ "wall_orientation_review": {
+ "schema_version": "formal_wall_orientation_review",
+ "local_selection_required": len(candidates) > 1,
+ "baseline_candidate_id": "mesh_min_area" if primary else None,
+ "candidate_ids": [candidate.get("candidate_id") for candidate in candidates],
+ "object_orientation_clusters": orientation_clusters,
+ "review_asset_layout": "isolated_candidate_contact_sheet",
+ "agent_contract": "Identify wall-following objects and their visible orientation evidence from the isolated panels. Local code scores the finite candidate ids. Supporting object ids are orientation evidence only and never imply wall attach.",
+ },
+ "layout_polygon_available": layout_polygon is not None,
+ "layout_polygon_source": None,
+ "layout_polygon_note": (
+ "Optional upstream layout polygon used only for IoU diagnostics; formal wall coordinates do not depend on it."
+ if layout_polygon is not None
+ else "Upstream layout polygon unavailable; formal wall coordinates were generated directly from SAM3D mesh footprints."
+ ),
+ "candidate_count": len(candidates),
+ "candidates": candidates,
+ "assets": {
+ "render_topdown_path": str(render_topdown_path),
+ "render_topdown_grid_path": str(render_topdown_grid_path),
+ "canonical_topdown_path": str(canonical_topdown_path),
+ "canonical_grid_mapping_path": str(canonical_mapping_path),
+ "render_topdown_grid_mapping_raw_path": str(raw_mapping_path),
+ "render_grid_affine_path": str(affine_path),
+ "render_foreground_mask_path": str(mask_path),
+ "candidate_overlay_path": str(all_overlay_path),
+ "wall_orientation_candidates_overlay_path": str(orientation_overlay_path),
+ "primary_candidate_overlay_path": str(primary_overlay_path) if primary_overlay_path else None,
+ },
+ }
+ write_json(method_candidates_path, result)
+
+ manifest = {
+ "status": "available" if primary else "missing_primary",
+ "scene_name": scene_name,
+ "precompute_dir": str(output_dir),
+ "method_candidates_path": str(method_candidates_path),
+ "manifest_path": str(manifest_path),
+ "canonical_topdown_image_path": str(canonical_topdown_path),
+ "canonical_mapping_path": str(canonical_mapping_path),
+ "render_grid_affine_path": str(affine_path),
+ "render_foreground_mask_path": str(mask_path),
+ "candidate_overlay_path": str(all_overlay_path),
+ "wall_orientation_candidates_overlay_path": str(orientation_overlay_path),
+ "primary_candidate_overlay_path": str(primary_overlay_path) if primary_overlay_path else None,
+ "primary_method": PRIMARY_METHOD,
+ "supporting_methods": SUPPORTING_METHODS,
+ "same_grid_policy": result["renderer"]["same_grid_policy"],
+ "reused_existing": False,
+ }
+ write_json(manifest_path, manifest)
+ return manifest
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Generate current-run render-grid wall candidates.")
+ parser.add_argument("--scene_name", required=True)
+ parser.add_argument("--run_root", required=True, type=Path)
+ parser.add_argument("--output_dir", required=True, type=Path)
+ parser.add_argument("--grid_size", type=int, default=33)
+ parser.add_argument("--image_size", type=int, default=768)
+ args = parser.parse_args()
+ result = run_precompute(
+ scene_name=args.scene_name,
+ run_root=args.run_root,
+ output_dir=args.output_dir,
+ grid_size=args.grid_size,
+ image_size=args.image_size,
+ )
+ print(json.dumps(result, indent=2, ensure_ascii=False))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/relations/tools/runtime_tracker.py b/vlm_agent/reconstruction/relations/tools/runtime_tracker.py
new file mode 100644
index 0000000..094c97c
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/runtime_tracker.py
@@ -0,0 +1,132 @@
+from __future__ import annotations
+
+from contextlib import contextmanager
+from pathlib import Path
+from typing import Any, Iterator
+
+from relations.agents.state import write_json, write_md
+
+
+class RuntimeTracker:
+ """Track execution closure without token, cost, or timing accounting."""
+
+ def __init__(self) -> None:
+ self.stages: dict[str, dict[str, Any]] = {}
+ self.api_calls: list[dict[str, Any]] = []
+ self.render_calls: list[dict[str, Any]] = []
+
+
+ def start(self, stage_name: str) -> None:
+ stage = self.stages.setdefault(
+ stage_name, {"status": "running", "count": 0}
+ )
+ stage["status"] = "running"
+
+ def end(
+ self, stage_name: str, status: str = "success", **metadata: Any
+ ) -> None:
+ stage = self.stages.setdefault(
+ stage_name, {"status": status, "count": 0}
+ )
+ stage["status"] = status
+ stage["count"] = int(stage.get("count", 0)) + 1
+ if metadata:
+ stage.setdefault("metadata", {}).update(metadata)
+
+ def mark_skipped(self, stage_name: str, reason: str | None = None) -> None:
+ stage = self.stages.setdefault(
+ stage_name, {"status": "skipped", "count": 0}
+ )
+ stage["status"] = "skipped"
+ if reason:
+ stage["reason"] = reason
+
+
+ @contextmanager
+ def stage(self, stage_name: str, **metadata: Any) -> Iterator[None]:
+ self.start(stage_name)
+ try:
+ yield
+ except Exception as exc:
+ self.end(
+ stage_name, status="failed", error=type(exc).__name__, **metadata
+ )
+ raise
+ else:
+ self.end(stage_name, status="success", **metadata)
+
+
+ @contextmanager
+ def api_call(self, stage_name: str, **metadata: Any) -> Iterator[None]:
+ try:
+ yield
+ except Exception as exc:
+ self.record_api_call(
+ stage_name, success=False,
+ error=type(exc).__name__, **metadata
+ )
+ raise
+ else:
+ self.record_api_call(stage_name, success=True, **metadata)
+
+ def record_api_call(
+ self, stage: str, success: bool = True, **metadata: Any
+ ) -> None:
+ item = {"stage": stage, "success": bool(success)}
+ item.update(metadata)
+ self.api_calls.append(item)
+ self.record_stage_result(
+ stage, status="success" if success else "failed"
+ )
+
+ def record_stage_result(
+ self, stage: str, status: str = "success",
+ **metadata: Any
+ ) -> None:
+ stage_data = self.stages.setdefault(
+ stage, {"status": status, "count": 0}
+ )
+ stage_data["status"] = status
+ stage_data["count"] = int(stage_data.get("count", 0)) + 1
+ if metadata:
+ stage_data.setdefault("metadata", {}).update(metadata)
+
+ def record_render_call(
+ self, stage: str, success: bool = True, **metadata: Any
+ ) -> None:
+ item = {"stage": stage, "success": bool(success)}
+ item.update(metadata)
+ self.render_calls.append(item)
+
+ def to_json(self) -> dict[str, Any]:
+ return {
+ "schema_version": "relation_execution_summary",
+ "api_calls": len(self.api_calls),
+ "render_calls": len(self.render_calls),
+ "stages": self.stages,
+ }
+
+ def to_markdown_lines(self) -> list[str]:
+ data = self.to_json()
+ lines = [
+ f"- api_calls: `{data['api_calls']}`",
+ f"- render_calls: `{data['render_calls']}`",
+ "",
+ "## Stages",
+ ]
+ for name, stage in data["stages"].items():
+ lines.append(
+ f"- `{name}`: status=`{stage.get('status')}` "
+ f"count=`{stage.get('count', 0)}`"
+ )
+ return lines
+
+ def write(self, output_dir: Path) -> dict[str, Any]:
+ data = self.to_json()
+ json_path = output_dir / "debug" / "agents" / "runtime_summary.json"
+ md_path = output_dir / "debug" / "agents" / "runtime_summary.md"
+ write_json(json_path, data)
+ write_md(md_path, "Execution Summary", self.to_markdown_lines())
+ data["json_path"] = str(json_path)
+ data["md_path"] = str(md_path)
+ return data
diff --git a/vlm_agent/reconstruction/relations/tools/visualize_final_walls_on_topdown.py b/vlm_agent/reconstruction/relations/tools/visualize_final_walls_on_topdown.py
new file mode 100644
index 0000000..e5afd72
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/visualize_final_walls_on_topdown.py
@@ -0,0 +1,302 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+ROOT = Path(__file__).resolve().parents[2]
+if str(ROOT) not in sys.path:
+ sys.path.insert(0, str(ROOT))
+
+from relations.agents.schema_guard import WALL_IDS, validate_wall_geometry, wall_axis_alignment_summary
+from relations.agents.state import write_json, write_md
+
+
+def _load_json(path: Path) -> Any:
+ return json.loads(path.read_text(encoding="utf-8"))
+
+
+def _num(value: Any) -> float | None:
+ if isinstance(value, (int, float)) and not isinstance(value, bool):
+ return float(value)
+ return None
+
+
+def _mapping_bounds(mapping: dict[str, Any]) -> dict[str, float]:
+ points: list[list[float]] = []
+ for point in mapping.get("grid_points", []) if isinstance(mapping.get("grid_points"), list) else []:
+ if not isinstance(point, dict):
+ continue
+ raw = point.get("point_3d_original_xz")
+ if isinstance(raw, list) and len(raw) == 2:
+ x, z = _num(raw[0]), _num(raw[1])
+ if x is not None and z is not None:
+ points.append([x, z])
+ continue
+ raw3 = point.get("point_3d_original") or point.get("point_3d_normalized")
+ if isinstance(raw3, list) and len(raw3) >= 3:
+ x, z = _num(raw3[0]), _num(raw3[2])
+ if x is not None and z is not None:
+ points.append([x, z])
+ if not points:
+ raise ValueError("grid mapping does not contain usable world x/z grid points")
+ xs = [p[0] for p in points]
+ zs = [p[1] for p in points]
+ if max(xs) - min(xs) <= 1e-12 or max(zs) - min(zs) <= 1e-12:
+ raise ValueError("grid mapping bounds are degenerate")
+ return {"x_min": min(xs), "x_max": max(xs), "z_min": min(zs), "z_max": max(zs)}
+
+
+def _world_to_pixel(point: list[float], bounds: dict[str, float], width: int, height: int) -> list[int]:
+ x, z = float(point[0]), float(point[1])
+ px = round((x - bounds["x_min"]) / (bounds["x_max"] - bounds["x_min"]) * (width - 1))
+ py = round((bounds["z_max"] - z) / (bounds["z_max"] - bounds["z_min"]) * (height - 1))
+ return [int(px), int(py)]
+
+
+def _load_xz_to_pixel_affine(grid_mapping: Path) -> list[list[float]] | None:
+ candidates = [grid_mapping.with_name("render_grid_affine.json")]
+ for path in candidates:
+ if not path.exists():
+ continue
+ try:
+ data = _load_json(path)
+ except Exception:
+ continue
+ matrix = data.get("xz_to_pixel_affine") if isinstance(data, dict) else None
+ if (
+ isinstance(matrix, list)
+ and len(matrix) == 3
+ and all(isinstance(row, list) and len(row) == 2 for row in matrix)
+ ):
+ return [[float(value) for value in row] for row in matrix]
+ return None
+
+
+def _world_to_pixel_exact(point: list[float], affine: list[list[float]]) -> list[int]:
+ x, z = float(point[0]), float(point[1])
+ px = x * affine[0][0] + z * affine[1][0] + affine[2][0]
+ py = x * affine[0][1] + z * affine[1][1] + affine[2][1]
+ return [int(round(px)), int(round(py))]
+
+
+def _signed_area(points: list[list[float]]) -> float:
+ area2 = 0.0
+ for idx, p in enumerate(points):
+ q = points[(idx + 1) % len(points)]
+ area2 += p[0] * q[1] - q[0] * p[1]
+ return area2 / 2.0
+
+
+def _screen_y_down_polygon_order(points: list[list[float]]) -> dict[str, Any]:
+ if len(points) < 3:
+ return {"signed_area": None, "interpreted_order": "unknown", "screen_clockwise": False}
+ signed_area = _signed_area(points)
+ if abs(signed_area) <= 1e-9:
+ interpreted_order = "degenerate_or_too_small"
+ else:
+ interpreted_order = "clockwise" if signed_area > 0 else "counter_clockwise"
+ return {
+ "signed_area": signed_area,
+ "interpreted_order": interpreted_order,
+ "screen_clockwise": interpreted_order == "clockwise",
+ }
+
+
+def _walls_from_objects(objects_payload: dict[str, Any]) -> list[dict[str, Any]]:
+ walls = objects_payload.get("walls")
+ if not isinstance(walls, list):
+ return []
+ by_id = {wall.get("id"): wall for wall in walls if isinstance(wall, dict)}
+ return [by_id[wall_id] for wall_id in WALL_IDS if wall_id in by_id]
+
+
+def _wall_revision_summary(output: Path, candidate_modified: bool | None = None, revision_reason: str | None = None) -> dict[str, Any]:
+ wall_audit_path = output.with_name("wall_audit.json")
+ summary = {
+ "candidate_modified": bool(candidate_modified),
+ "revision_used": bool(candidate_modified),
+ "revision_reasons": [revision_reason] if revision_reason else [],
+ "weak_wall_completion": {"used": False, "weak_wall_ids": [], "completion_methods": [], "completion_reasons": []},
+ }
+ if not wall_audit_path.exists():
+ return summary
+ try:
+ wall_audit = _load_json(wall_audit_path)
+ except Exception:
+ return summary
+ promotion = wall_audit.get("candidate_wall_promotion") if isinstance(wall_audit, dict) else None
+ if isinstance(promotion, dict):
+ summary["candidate_modified"] = bool(promotion.get("candidate_modified"))
+ summary["revision_used"] = bool(promotion.get("revision_used") or promotion.get("candidate_modified"))
+ summary["revision_reasons"] = promotion.get("revision_reasons", [])
+ summary["candidate_wall_promotion"] = promotion
+ completion = wall_audit.get("weak_wall_completion") if isinstance(wall_audit, dict) else None
+ if isinstance(completion, dict):
+ summary["weak_wall_completion"] = completion
+ return summary
+
+
+def visualize(topdown_png: Path, grid_mapping: Path, objects_json: Path, output: Path, candidate_modified: bool | None = None, revision_reason: str | None = None) -> dict[str, Any]:
+ import cv2
+
+ image = cv2.imread(str(topdown_png))
+ if image is None:
+ raise ValueError(f"failed to read topdown png: {topdown_png}")
+ height, width = image.shape[:2]
+ mapping = _load_json(grid_mapping)
+ bounds = _mapping_bounds(mapping)
+ affine = _load_xz_to_pixel_affine(grid_mapping)
+ projection_method = "exact_render_grid_affine" if affine else "approximate_grid_bounds"
+ objects_payload = _load_json(objects_json)
+ walls = _walls_from_objects(objects_payload)
+ geometry = validate_wall_geometry(walls)
+ axis_summary = wall_axis_alignment_summary(walls)
+
+ colors = {
+ "wall_001": (0, 80, 255),
+ "wall_002": (0, 200, 0),
+ "wall_003": (255, 80, 0),
+ "wall_004": (210, 0, 210),
+ }
+ wall_reports = []
+ for wall in walls:
+ wall_id = wall.get("id")
+ first = wall.get("first_point")
+ second = wall.get("second_point")
+ if not (isinstance(first, list) and isinstance(second, list) and len(first) == 2 and len(second) == 2):
+ wall_reports.append({"id": wall_id, "error": "invalid wall points"})
+ continue
+ if affine:
+ p0 = _world_to_pixel_exact(first, affine)
+ p1 = _world_to_pixel_exact(second, affine)
+ else:
+ p0 = _world_to_pixel(first, bounds, width, height)
+ p1 = _world_to_pixel(second, bounds, width, height)
+ in_range = all(0 <= value < limit for value, limit in [(p0[0], width), (p1[0], width), (p0[1], height), (p1[1], height)])
+ color = colors.get(wall_id, (255, 255, 255))
+ cv2.arrowedLine(image, tuple(p0), tuple(p1), color, 4, cv2.LINE_AA, tipLength=0.08)
+ cv2.circle(image, tuple(p0), 7, color, -1, cv2.LINE_AA)
+ cv2.circle(image, tuple(p1), 5, color, 2, cv2.LINE_AA)
+ cv2.putText(image, "first", (p0[0] + 5, p0[1] - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.45, color, 1, cv2.LINE_AA)
+ cv2.putText(image, "second", (p1[0] + 5, p1[1] - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.45, color, 1, cv2.LINE_AA)
+ mid = [round((p0[0] + p1[0]) / 2), round((p0[1] + p1[1]) / 2)]
+ cv2.putText(image, f"{wall_id} first->second", tuple(mid), cv2.FONT_HERSHEY_SIMPLEX, 0.62, color, 2, cv2.LINE_AA)
+ wall_reports.append({
+ "id": wall_id,
+ "first_point_xz": first,
+ "second_point_xz": second,
+ "first_point_px": p0,
+ "second_point_px": p1,
+ "endpoints_in_image_range": in_range,
+ })
+
+ output.parent.mkdir(parents=True, exist_ok=True)
+ cv2.imwrite(str(output), image)
+ screen_vertices = [item.get("first_point_px") for item in wall_reports if isinstance(item.get("first_point_px"), list)]
+ screen_order = _screen_y_down_polygon_order(screen_vertices)
+ screen_closed = (
+ len(wall_reports) == 4
+ and all(
+ wall_reports[index].get("second_point_px") == wall_reports[(index + 1) % len(wall_reports)].get("first_point_px")
+ for index in range(len(wall_reports))
+ )
+ )
+ first_vertices = [wall.get("first_point") for wall in walls if isinstance(wall.get("first_point"), list)]
+ signed_area = _signed_area(first_vertices) if len(first_vertices) == 4 else None
+ interpreted = "clockwise" if signed_area is not None and signed_area < 0 else "counter_clockwise" if signed_area is not None and signed_area > 0 else "unknown"
+ revision_summary = _wall_revision_summary(output, candidate_modified=candidate_modified, revision_reason=revision_reason)
+ weak_completion = revision_summary.get("weak_wall_completion", {}) if isinstance(revision_summary.get("weak_wall_completion"), dict) else {}
+ report = {
+ "topdown_png": str(topdown_png),
+ "grid_mapping": str(grid_mapping),
+ "objects_json": str(objects_json),
+ "output_overlay_png": str(output),
+ "coordinate_source": "obb_from_mesh_footprint_min_rect_xz",
+ "topdown_usage": "visual_audit_only",
+ "fixed_layout_views_usage": "auxiliary_visual_audit_only",
+ "candidate_modified": revision_summary.get("candidate_modified", False),
+ "revision_used": revision_summary.get("revision_used", False),
+ "revision_reasons": revision_summary.get("revision_reasons", []),
+ "image_size": [width, height],
+ "mapping_bounds_xz": bounds,
+ "projection_method": projection_method,
+ "render_grid_affine_used": bool(affine),
+ "screen_signed_area_y_down": screen_order["signed_area"],
+ "screen_interpreted_order": screen_order["interpreted_order"],
+ "screen_clockwise": screen_order["screen_clockwise"],
+ "screen_closed": screen_closed,
+ "walls": wall_reports,
+ "all_endpoints_in_image_range": all(item.get("endpoints_in_image_range") for item in wall_reports),
+ "geometry": geometry,
+ "axis_aligned": axis_summary.get("axis_aligned"),
+ "oriented_boundary": axis_summary.get("oriented_boundary"),
+ "edge_orientations": axis_summary.get("edge_orientations", []),
+ "weak_wall_completion_used": bool(weak_completion.get("used")),
+ "weak_wall_ids": weak_completion.get("weak_wall_ids", []),
+ "weak_wall_completion": weak_completion,
+ "orientation_note": "Final walls are drawn with first->second arrows; coordinates are expected to come from obb_from_mesh_footprint_min_rect after final assembly.",
+ "signed_area": signed_area,
+ "interpreted_order": interpreted,
+ "note": "This tool projects final OBB XZ wall coordinates onto the canonical top-down image for visual audit. It does not make the top-down image the primary coordinate source.",
+ }
+ if revision_summary.get("candidate_wall_promotion"):
+ report["candidate_wall_promotion"] = revision_summary["candidate_wall_promotion"]
+ write_json(output.with_name("final_walls_overlay_check.json"), report)
+ lines = [
+ f"- topdown_png: `{topdown_png}`",
+ f"- grid_mapping: `{grid_mapping}`",
+ f"- objects_json: `{objects_json}`",
+ f"- output_overlay_png: `{output}`",
+ "- coordinate_source: `obb_from_mesh_footprint_min_rect_xz`",
+ "- topdown_usage: `visual_audit_only`",
+ "- fixed_layout_views_usage: `auxiliary_visual_audit_only`",
+ f"- candidate_modified: `{report['candidate_modified']}`",
+ f"- revision_used: `{report['revision_used']}`",
+ f"- revision_reasons: `{report['revision_reasons']}`",
+ f"- all_endpoints_in_image_range: `{report['all_endpoints_in_image_range']}`",
+ f"- projection_method: `{report['projection_method']}`",
+ f"- render_grid_affine_used: `{report['render_grid_affine_used']}`",
+ f"- screen_signed_area_y_down: `{report['screen_signed_area_y_down']}`",
+ f"- screen_interpreted_order: `{report['screen_interpreted_order']}`",
+ f"- screen_clockwise: `{report['screen_clockwise']}`",
+ f"- screen_closed: `{report['screen_closed']}`",
+ f"- axis_aligned: `{report['axis_aligned']}`",
+ f"- oriented_boundary: `{report['oriented_boundary']}`",
+ f"- weak_wall_completion_used: `{report['weak_wall_completion_used']}`",
+ f"- weak_wall_ids: `{report['weak_wall_ids']}`",
+ f"- signed_area: `{signed_area}`",
+ f"- interpreted_order: `{interpreted}`",
+ "",
+ "## Walls",
+ ]
+ for item in wall_reports:
+ lines.append(f"- `{item.get('id')}` xz {item.get('first_point_xz')} -> {item.get('second_point_xz')}; px {item.get('first_point_px')} -> {item.get('second_point_px')}; in_range `{item.get('endpoints_in_image_range')}`")
+ if report.get("oriented_boundary"):
+ lines.extend(["", "## Oriented Boundary", "- Final walls form an oriented quadrilateral and are not required to be axis-aligned."])
+ if report.get("weak_wall_completion_used"):
+ lines.extend(["", "## Hidden / Weak Wall Completion", "- Some walls were inferred by room closure and should be visually inspected."])
+ if report.get("candidate_modified"):
+ lines.extend(["", "## Revision Notice", "- These walls were revised during wall_final_adjudication; inspect visually before accepting."])
+ lines.extend(["", "## Note", "- This image projects final OBB XZ wall coordinates onto canonical top-down for human visual audit. Arrows show first_point -> second_point. Top-down is not the primary coordinate source."])
+ write_md(output.with_name("final_walls_overlay_check.md"), "Final Walls Overlay Check", lines)
+ return report
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Overlay final wall coordinates on canonical top-down evidence.")
+ parser.add_argument("--topdown_png", required=True, type=Path)
+ parser.add_argument("--grid_mapping", required=True, type=Path)
+ parser.add_argument("--objects_json", required=True, type=Path)
+ parser.add_argument("--output", required=True, type=Path)
+ parser.add_argument("--candidate_modified", action="store_true")
+ parser.add_argument("--revision_reason", default=None)
+ args = parser.parse_args()
+ report = visualize(args.topdown_png, args.grid_mapping, args.objects_json, args.output, candidate_modified=args.candidate_modified, revision_reason=args.revision_reason)
+ print(json.dumps({"output": report["output_overlay_png"], "all_endpoints_in_image_range": report["all_endpoints_in_image_range"], "axis_aligned": report["axis_aligned"], "oriented_boundary": report["oriented_boundary"], "weak_wall_completion_used": report["weak_wall_completion_used"], "weak_wall_ids": report["weak_wall_ids"], "signed_area": report["signed_area"]}, indent=2, ensure_ascii=False))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/relations/tools/wall_canonical_naming.py b/vlm_agent/reconstruction/relations/tools/wall_canonical_naming.py
new file mode 100644
index 0000000..4f510ae
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/wall_canonical_naming.py
@@ -0,0 +1,289 @@
+from __future__ import annotations
+
+from copy import deepcopy
+from typing import Any, Sequence
+
+WALL_IDS = ["wall_001", "wall_002", "wall_003", "wall_004"]
+
+CANONICAL_WALL_NAMING_POLICY: dict[str, Any] = {
+ "schema_version": "wall_canonical_naming",
+ "final_policy": {
+ "if_original_image_shows_three_walls": "wall_001 is the middle visible wall; wall_002..wall_004 continue clockwise in top-down XZ order.",
+ "if_original_image_shows_two_walls": "wall_001 is the left visible wall; wall_002..wall_004 continue clockwise in top-down XZ order.",
+ "remaining_walls": "Assign the remaining room-closure walls clockwise in top-down XZ order from wall_001.",
+ "weak_hidden_wall_rule": "Weak/hidden/inferred walls may close the room geometry, but they must be marked weak/hidden and are not valid attach anchors unless the actual wall surface/contact is visible.",
+ },
+ "candidate_fallback_policy": {
+ "purpose": "Used only for local candidate evidence before the vision agent decides original-image visibility.",
+ "rule": "When original-image visibility is unknown, name the leftmost top-down segment wall_001, then continue clockwise. This is provisional and may be renamed by final wall adjudication.",
+ },
+ "required_final_audit_fields": [
+ "original_visible_wall_count",
+ "wall_001_anchor_reason",
+ "visible_role",
+ "original_image_visibility",
+ "clockwise_order_index",
+ "weak_wall",
+ ],
+}
+
+
+def canonical_wall_naming_policy() -> dict[str, Any]:
+ return deepcopy(CANONICAL_WALL_NAMING_POLICY)
+
+
+def _number(value: Any) -> float | None:
+ if isinstance(value, (int, float)) and not isinstance(value, bool):
+ return float(value)
+ return None
+
+
+def _clean_points(points: Sequence[Any]) -> list[list[float]]:
+ cleaned: list[list[float]] = []
+ for point in points:
+ if not isinstance(point, Sequence) or isinstance(point, (str, bytes)) or len(point) != 2:
+ return []
+ x = _number(point[0])
+ z = _number(point[1])
+ if x is None or z is None:
+ return []
+ if cleaned and abs(cleaned[-1][0] - x) <= 1e-12 and abs(cleaned[-1][1] - z) <= 1e-12:
+ continue
+ cleaned.append([x, z])
+ if len(cleaned) >= 2 and abs(cleaned[0][0] - cleaned[-1][0]) <= 1e-12 and abs(cleaned[0][1] - cleaned[-1][1]) <= 1e-12:
+ cleaned.pop()
+ return cleaned
+
+
+def _centroid(points: Sequence[Sequence[float]]) -> list[float]:
+ return [sum(point[0] for point in points) / len(points), sum(point[1] for point in points) / len(points)]
+
+
+def _bounds(points: Sequence[Sequence[float]]) -> dict[str, float]:
+ xs = [point[0] for point in points]
+ zs = [point[1] for point in points]
+ return {"x_min": min(xs), "x_max": max(xs), "z_min": min(zs), "z_max": max(zs)}
+
+
+def classify_segment_geometry_role(segment: dict[str, Any], center: Sequence[float], bounds: dict[str, float]) -> str:
+ first = segment.get("first_point")
+ second = segment.get("second_point")
+ if not (isinstance(first, list) and isinstance(second, list) and len(first) == 2 and len(second) == 2):
+ return "unknown"
+ mx = (float(first[0]) + float(second[0])) / 2.0
+ mz = (float(first[1]) + float(second[1])) / 2.0
+ span_x = max(abs(bounds["x_max"] - bounds["x_min"]), 1e-9)
+ span_z = max(abs(bounds["z_max"] - bounds["z_min"]), 1e-9)
+ dx = (mx - float(center[0])) / span_x
+ dz = (mz - float(center[1])) / span_z
+ if abs(dx) >= abs(dz):
+ return "left" if dx < 0 else "right"
+ return "back" if dz < 0 else "front"
+
+
+def _segment_midpoint(segment: dict[str, Any]) -> list[float]:
+ first = segment["first_point"]
+ second = segment["second_point"]
+ return [(float(first[0]) + float(second[0])) / 2.0, (float(first[1]) + float(second[1])) / 2.0]
+
+
+def _valid_affine(affine: Any) -> list[list[float]] | None:
+ if not (isinstance(affine, Sequence) and len(affine) == 3):
+ return None
+ rows: list[list[float]] = []
+ for row in affine:
+ if not (isinstance(row, Sequence) and not isinstance(row, (str, bytes)) and len(row) >= 2):
+ return None
+ x = _number(row[0])
+ y = _number(row[1])
+ if x is None or y is None:
+ return None
+ rows.append([x, y])
+ return rows
+
+
+def xz_to_screen_xy(point: Sequence[float], affine: Any = None) -> list[float]:
+ """Project an OpenGL XZ point to top-down screen coordinates.
+
+ When the render-grid affine is available we use it, so wall naming follows
+ the exact image shown to humans/agents. The fallback is only for
+ calls that lack a render grid; it remains deterministic but is less
+ semantically rich.
+ """
+ x = float(point[0])
+ z = float(point[1])
+ matrix = _valid_affine(affine)
+ if matrix:
+ return [
+ matrix[0][0] * x + matrix[1][0] * z + matrix[2][0],
+ matrix[0][1] * x + matrix[1][1] * z + matrix[2][1],
+ ]
+ return [x, -z]
+
+
+def _signed_area_xy(points: Sequence[Sequence[float]]) -> float:
+ total = 0.0
+ for idx, point in enumerate(points):
+ nxt = points[(idx + 1) % len(points)]
+ total += float(point[0]) * float(nxt[1]) - float(nxt[0]) * float(point[1])
+ return total / 2.0
+
+
+def _rotate(items: list[Any], start_idx: int) -> list[Any]:
+ if not items:
+ return items
+ return items[start_idx:] + items[:start_idx]
+
+
+def canonical_screen_clockwise_segments_from_polygon(
+ points: Sequence[Any],
+ *,
+ xz_to_pixel_affine: Any = None,
+ source: str,
+ method: str,
+ role: str,
+ trust_level: str = "candidate",
+ may_be_used_as_final_wall: bool = False,
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Return one deterministic wall-id order for a polygon.
+
+ This is the single local naming rule used by precompute, wall evidence, and
+ final assembly. It first orients the polygon clockwise in rendered top-down
+ screen coordinates, then chooses the first wall by a stable screen midpoint
+ sort. Model-provided visible-role metadata is audit-only and must not rotate
+ the final wall ids.
+ """
+ cleaned = _clean_points(points)
+ if len(cleaned) < 3:
+ return [], {"status": "invalid", "reason": "not_enough_polygon_points"}
+ affine = _valid_affine(xz_to_pixel_affine)
+ screen_points = [xz_to_screen_xy(point, affine) for point in cleaned]
+ screen_area_before = _signed_area_xy(screen_points)
+ clockwise_points = list(cleaned)
+ if screen_area_before < 0:
+ clockwise_points = list(reversed(clockwise_points))
+ screen_points = [xz_to_screen_xy(point, affine) for point in clockwise_points]
+ screen_area_after = _signed_area_xy(screen_points)
+ center = _centroid(clockwise_points)
+ bounds = _bounds(clockwise_points)
+ raw_segments: list[dict[str, Any]] = []
+ for idx, point in enumerate(clockwise_points):
+ nxt = clockwise_points[(idx + 1) % len(clockwise_points)]
+ first_screen = xz_to_screen_xy(point, affine)
+ second_screen = xz_to_screen_xy(nxt, affine)
+ raw_segments.append(
+ {
+ "provisional_original_index": idx,
+ "first_point": [round(float(point[0]), 6), round(float(point[1]), 6)],
+ "second_point": [round(float(nxt[0]), 6), round(float(nxt[1]), 6)],
+ "source": source,
+ "source_method": method,
+ "role": role,
+ "trust_level": trust_level,
+ "topdown_verification_status": "needs_check",
+ "may_be_used_as_final_wall": may_be_used_as_final_wall,
+ "midpoint_screen_xy": [
+ (first_screen[0] + second_screen[0]) / 2.0,
+ (first_screen[1] + second_screen[1]) / 2.0,
+ ],
+ }
+ )
+ for segment in raw_segments:
+ segment["geometry_role"] = classify_segment_geometry_role(segment, center, bounds)
+ segment["midpoint_xz"] = _segment_midpoint(segment)
+ start_idx = min(
+ range(len(raw_segments)),
+ key=lambda idx: (
+ raw_segments[idx]["midpoint_screen_xy"][1],
+ raw_segments[idx]["midpoint_screen_xy"][0],
+ raw_segments[idx]["midpoint_xz"][0],
+ raw_segments[idx]["midpoint_xz"][1],
+ ),
+ )
+ ordered = _rotate(raw_segments, start_idx)
+ audit = {
+ "schema_version": "deterministic_screen_clockwise_wall_naming",
+ "status": "available",
+ "screen_coordinate_source": "render_grid_affine.xz_to_pixel_affine" if affine else "fallback_x_neg_z",
+ "screen_signed_area_before": screen_area_before,
+ "screen_signed_area_after": screen_area_after,
+ "screen_order_interpretation": "clockwise" if screen_area_after > 0 else "counter_clockwise_or_degenerate",
+ "start_segment_policy": "topmost_then_leftmost_screen_midpoint",
+ "model_visible_role_can_rotate_ids": False,
+ }
+ for idx, segment in enumerate(ordered):
+ segment["geometry_wall_id"] = f"geom_wall_{idx + 1:03d}"
+ segment["id"] = WALL_IDS[idx] if idx < len(WALL_IDS) else f"wall_{idx + 1:03d}"
+ segment["provisional_wall_id"] = segment["id"]
+ segment["clockwise_order_index"] = idx + 1
+ segment["screen_clockwise_order_index"] = idx + 1
+ segment["canonical_naming"] = {
+ "schema_version": CANONICAL_WALL_NAMING_POLICY["schema_version"],
+ "status": "deterministic_screen_clockwise_geometry",
+ "candidate_fallback_rule": "topmost_then_leftmost_screen_midpoint, then continue screen-clockwise",
+ "final_policy_required": CANONICAL_WALL_NAMING_POLICY["final_policy"],
+ "wall_001_anchor_reason": "deterministic topmost/leftmost rendered top-down segment; original-image visibility is audit metadata only",
+ "geometry_role": segment["geometry_role"],
+ "original_image_visibility": "unknown_until_final_adjudication",
+ "must_be_confirmed_or_renamed_by_final_adjudication": False,
+ "screen_coordinate_source": audit["screen_coordinate_source"],
+ "start_segment_policy": audit["start_segment_policy"],
+ }
+ return ordered, audit
+
+
+def provisional_segments_from_polygon(
+ points: Sequence[Any],
+ *,
+ xz_to_pixel_affine: Any = None,
+ source: str,
+ method: str,
+ role: str,
+ trust_level: str = "candidate",
+ may_be_used_as_final_wall: bool = False,
+) -> list[dict[str, Any]]:
+ """Return candidate wall segments with stable provisional ids."""
+ ordered, _ = canonical_screen_clockwise_segments_from_polygon(
+ points,
+ xz_to_pixel_affine=xz_to_pixel_affine,
+ source=source,
+ method=method,
+ role=role,
+ trust_level=trust_level,
+ may_be_used_as_final_wall=may_be_used_as_final_wall,
+ )
+ return ordered
+
+def final_wall_naming_audit(walls: Sequence[Any]) -> dict[str, Any]:
+ clean_walls = [wall for wall in walls if isinstance(wall, dict)]
+ points: list[list[float]] = []
+ for wall in clean_walls:
+ first = wall.get("first_point")
+ if isinstance(first, list) and len(first) == 2 and _number(first[0]) is not None and _number(first[1]) is not None:
+ points.append([float(first[0]), float(first[1])])
+ center = _centroid(points) if points else None
+ bounds = _bounds(points) if len(points) >= 2 else None
+ items: list[dict[str, Any]] = []
+ missing_audit_fields: list[dict[str, Any]] = []
+ for idx, wall in enumerate(clean_walls):
+ item = {
+ "id": wall.get("id"),
+ "clockwise_order_index": idx + 1,
+ "original_visible_wall_count": wall.get("original_visible_wall_count") or wall.get("canonical_naming", {}).get("original_visible_wall_count"),
+ "visible_role": wall.get("visible_role") or wall.get("canonical_naming", {}).get("visible_role"),
+ "original_image_visibility": wall.get("original_image_visibility") or wall.get("visibility") or wall.get("canonical_naming", {}).get("original_image_visibility"),
+ "weak_wall": bool(wall.get("weak_wall")),
+ }
+ if center and bounds:
+ item["geometry_role_fallback"] = classify_segment_geometry_role(wall, center, bounds)
+ missing = [field for field in CANONICAL_WALL_NAMING_POLICY["required_final_audit_fields"] if item.get(field) is None and field != "weak_wall"]
+ if missing:
+ missing_audit_fields.append({"id": wall.get("id"), "missing_fields": missing})
+ items.append(item)
+ return {
+ "policy": canonical_wall_naming_policy(),
+ "items": items,
+ "missing_final_audit_fields": missing_audit_fields,
+ "complete_final_visibility_audit": not missing_audit_fields,
+ "note": "Geometry fallback roles are audit hints only; final wall ids are assigned by deterministic rendered top-down screen-clockwise geometry.",
+ }
diff --git a/vlm_agent/reconstruction/relations/tools/wall_geometry.py b/vlm_agent/reconstruction/relations/tools/wall_geometry.py
new file mode 100644
index 0000000..516a6dd
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/wall_geometry.py
@@ -0,0 +1,294 @@
+"""Production 2D wall geometry for relation preprocessing.
+
+This module owns the formal minimum-area OBB implementation and related
+polygon diagnostics. It intentionally has no dependency on exploratory
+exploratory relation packages or SceneMosaic runtime modules.
+"""
+
+from __future__ import annotations
+
+import math
+from typing import Any, Iterable, Sequence
+
+import cv2
+import numpy as np
+
+from relations.tools.wall_canonical_naming import canonical_screen_clockwise_segments_from_polygon
+
+Point = tuple[float, float]
+FORMAL_WALL_METHOD = "obb_from_mesh_footprint_min_rect"
+WALL_IDS = ["wall_001", "wall_002", "wall_003", "wall_004"]
+
+
+def signed_area(points: Sequence[Point]) -> float:
+ total = 0.0
+ for index, (x1, z1) in enumerate(points):
+ x2, z2 = points[(index + 1) % len(points)]
+ total += x1 * z2 - x2 * z1
+ return total * 0.5
+
+
+def polygon_area(points: Sequence[Point]) -> float:
+ return abs(signed_area(points))
+
+
+def convex_hull(points: Iterable[Point]) -> list[Point]:
+ unique = sorted({(round(float(x), 12), round(float(z), 12)) for x, z in points})
+ if len(unique) <= 1:
+ return list(unique)
+
+ def cross(origin: Point, first: Point, second: Point) -> float:
+ return (first[0] - origin[0]) * (second[1] - origin[1]) - (first[1] - origin[1]) * (second[0] - origin[0])
+
+ lower: list[Point] = []
+ for point in unique:
+ while len(lower) >= 2 and cross(lower[-2], lower[-1], point) <= 0:
+ lower.pop()
+ lower.append(point)
+ upper: list[Point] = []
+ for point in reversed(unique):
+ while len(upper) >= 2 and cross(upper[-2], upper[-1], point) <= 0:
+ upper.pop()
+ upper.append(point)
+ return lower[:-1] + upper[:-1]
+
+
+def ensure_clockwise(points: Sequence[Point]) -> list[Point]:
+ result = [(float(x), float(z)) for x, z in points]
+ if signed_area(result) > 0:
+ result.reverse()
+ return result
+
+
+def canonical_clockwise_rect(points: Sequence[Point]) -> list[Point]:
+ if len(points) != 4:
+ raise ValueError("canonical_clockwise_rect requires exactly four points")
+ ordered = ensure_clockwise(points)
+ edges = []
+ for index in range(4):
+ first = ordered[index]
+ second = ordered[(index + 1) % 4]
+ midpoint = ((first[0] + second[0]) * 0.5, (first[1] + second[1]) * 0.5)
+ edges.append((midpoint[1], midpoint[0], -math.dist(first, second), index))
+ start = min(edges)[3]
+ return [ordered[(start + index) % 4] for index in range(4)]
+
+
+def _rotate_point(point: Point, angle: float) -> Point:
+ cosine = math.cos(angle)
+ sine = math.sin(angle)
+ x, z = point
+ return x * cosine + z * sine, -x * sine + z * cosine
+
+
+def _unrotate_point(point: Point, angle: float) -> Point:
+ cosine = math.cos(angle)
+ sine = math.sin(angle)
+ u, v = point
+ return u * cosine - v * sine, u * sine + v * cosine
+
+
+def rect_from_angle(points: Sequence[Point], angle: float, margin: float = 0.0) -> list[Point]:
+ if not points:
+ raise ValueError("rect_from_angle requires at least one point")
+ rotated = [_rotate_point(point, angle) for point in points]
+ us = [point[0] for point in rotated]
+ vs = [point[1] for point in rotated]
+ min_u, max_u = min(us) - margin, max(us) + margin
+ min_v, max_v = min(vs) - margin, max(vs) + margin
+ rectangle = [
+ (min_u, min_v),
+ (max_u, min_v),
+ (max_u, max_v),
+ (min_u, max_v),
+ ]
+ return canonical_clockwise_rect([_unrotate_point(point, angle) for point in rectangle])
+
+
+def min_area_rect(points: Sequence[Point], margin: float = 0.0) -> tuple[list[Point], float, float]:
+ if not points:
+ raise ValueError("min_area_rect requires at least one point")
+ hull = convex_hull(points)
+ if len(hull) < 3:
+ rectangle = rect_from_angle(hull, 0.0, margin=margin)
+ return rectangle, polygon_area(rectangle), 0.0
+ angles = {
+ round(math.atan2(hull[(index + 1) % len(hull)][1] - point[1], hull[(index + 1) % len(hull)][0] - point[0]) % (math.pi / 2.0), 12)
+ for index, point in enumerate(hull)
+ }
+ best_rectangle: list[Point] | None = None
+ best_area = float("inf")
+ best_angle = 0.0
+ for angle in sorted(angles):
+ rectangle = rect_from_angle(hull, angle, margin=margin)
+ area = polygon_area(rectangle)
+ if area < best_area:
+ best_rectangle = rectangle
+ best_area = area
+ best_angle = angle
+ if best_rectangle is None:
+ best_rectangle = rect_from_angle(hull, 0.0, margin=margin)
+ best_area = polygon_area(best_rectangle)
+ return best_rectangle, best_area, best_angle
+
+
+def layout_wall_polygon(layout: dict[str, Any] | None) -> list[Point] | None:
+ if not isinstance(layout, dict):
+ return None
+ points: list[Point] = []
+ for wall in layout.get("walls") or []:
+ if not isinstance(wall, dict):
+ continue
+ first = wall.get("first_point_xz") or wall.get("first_point")
+ second = wall.get("second_point_xz") or wall.get("second_point")
+ if first and second:
+ points.extend([(float(first[0]), float(first[1])), (float(second[0]), float(second[1]))])
+ if len(points) < 3:
+ return None
+ hull = convex_hull(points)
+ return canonical_clockwise_rect(hull) if len(hull) == 4 else hull
+
+
+def dominant_rect_angle_degrees(points: Sequence[Point]) -> float:
+ lengths = [math.dist(points[index], points[(index + 1) % len(points)]) for index in range(len(points))]
+ index = max(range(len(lengths)), key=lengths.__getitem__)
+ first, second = points[index], points[(index + 1) % len(points)]
+ angle = math.degrees(math.atan2(second[1] - first[1], second[0] - first[0])) % 180.0
+ return angle - 90.0 if angle >= 90.0 else angle
+
+
+def axis_deviation_degrees(angle_degrees: float) -> float:
+ value = abs(angle_degrees) % 90.0
+ return min(value, 90.0 - value)
+
+
+def _inside_halfplane(point: Point, first: Point, second: Point, clockwise: bool) -> bool:
+ cross = (second[0] - first[0]) * (point[1] - first[1]) - (second[1] - first[1]) * (point[0] - first[0])
+ return cross <= 1e-9 if clockwise else cross >= -1e-9
+
+
+def _line_intersection(start: Point, end: Point, first: Point, second: Point) -> Point:
+ x1, y1 = start
+ x2, y2 = end
+ x3, y3 = first
+ x4, y4 = second
+ denominator = (x1 - x2) * (y3 - y4) - (y1 - y2) * (x3 - x4)
+ if abs(denominator) < 1e-12:
+ return end
+ px = ((x1 * y2 - y1 * x2) * (x3 - x4) - (x1 - x2) * (x3 * y4 - y3 * x4)) / denominator
+ py = ((x1 * y2 - y1 * x2) * (y3 - y4) - (y1 - y2) * (x3 * y4 - y3 * x4)) / denominator
+ return float(px), float(py)
+
+
+def _clip_intersection_area(subject: Sequence[Point], clipper: Sequence[Point], clockwise: bool) -> float:
+ output = list(subject)
+ for index, first in enumerate(clipper):
+ second = clipper[(index + 1) % len(clipper)]
+ input_points = output
+ output = []
+ if not input_points:
+ break
+ start = input_points[-1]
+ for end in input_points:
+ end_inside = _inside_halfplane(end, first, second, clockwise)
+ start_inside = _inside_halfplane(start, first, second, clockwise)
+ if end_inside:
+ if not start_inside:
+ output.append(_line_intersection(start, end, first, second))
+ output.append(end)
+ elif start_inside:
+ output.append(_line_intersection(start, end, first, second))
+ start = end
+ return polygon_area(output) if len(output) >= 3 else 0.0
+
+
+def polygon_iou(first: Sequence[Point] | None, second: Sequence[Point] | None) -> float | None:
+ if not first or not second:
+ return None
+ intersection = max(
+ _clip_intersection_area(first, second, clockwise=True),
+ _clip_intersection_area(first, second, clockwise=False),
+ )
+ union = polygon_area(first) + polygon_area(second) - intersection
+ return None if union <= 1e-12 else intersection / union
+
+
+def xz_to_pixel(points: Sequence[Point], matrix: np.ndarray) -> list[tuple[int, int]]:
+ rows = np.asarray([[float(x), float(z), 1.0] for x, z in points], dtype=np.float64)
+ projected = rows @ matrix
+ return [(int(round(x)), int(round(y))) for x, y in projected]
+
+
+def candidate_metrics(
+ polygon_xz: Sequence[Point],
+ layout_polygon: Sequence[Point] | None,
+ foreground: np.ndarray,
+ xz_to_pixel_matrix: np.ndarray,
+) -> dict[str, Any]:
+ polygon = canonical_clockwise_rect(polygon_xz) if len(polygon_xz) == 4 else list(polygon_xz)
+ points = np.asarray(xz_to_pixel(polygon, xz_to_pixel_matrix), dtype=np.int32)
+ polygon_mask = np.zeros(foreground.shape, dtype=np.uint8)
+ cv2.fillPoly(polygon_mask, [points], 255)
+ foreground_total = max(1, int(np.count_nonzero(foreground)))
+ foreground_inside = int(np.count_nonzero(cv2.bitwise_and(foreground, foreground, mask=polygon_mask)))
+ polygon_area_pixels = max(1, int(np.count_nonzero(polygon_mask)))
+ axis_deviation = axis_deviation_degrees(dominant_rect_angle_degrees(polygon)) if len(polygon) == 4 else None
+ return {
+ "polygon_xz": [[round(float(x), 6), round(float(z), 6)] for x, z in polygon],
+ "area_xz": round(polygon_area(polygon), 6),
+ "axis_deviation_degrees": None if axis_deviation is None else round(axis_deviation, 3),
+ "iou_vs_layout_walls": None if layout_polygon is None else round(float(polygon_iou(polygon, layout_polygon) or 0.0), 4),
+ "foreground_coverage": round(foreground_inside / foreground_total, 4),
+ "foreground_fill_in_candidate": round(foreground_inside / polygon_area_pixels, 4),
+ "empty_pixel_ratio_in_candidate": round(1.0 - foreground_inside / polygon_area_pixels, 4),
+ }
+
+
+def build_obb_walls(method_candidates: dict[str, Any], render_grid_affine: dict[str, Any]) -> dict[str, Any]:
+ candidates = method_candidates.get("candidates", []) if isinstance(method_candidates.get("candidates"), list) else []
+ primary = next((item for item in candidates if isinstance(item, dict) and item.get("method") == FORMAL_WALL_METHOD), None)
+ if not primary:
+ return {"status": "missing", "method": FORMAL_WALL_METHOD, "walls": [], "reason": "formal candidate missing"}
+ polygon = primary.get("metrics", {}).get("polygon_xz")
+ if not isinstance(polygon, list) or len(polygon) < 4:
+ return {"status": "missing", "method": FORMAL_WALL_METHOD, "walls": [], "reason": "polygon_xz missing"}
+ segments, naming_audit = canonical_screen_clockwise_segments_from_polygon(
+ polygon[:4],
+ xz_to_pixel_affine=render_grid_affine.get("xz_to_pixel_affine"),
+ source=FORMAL_WALL_METHOD,
+ method=FORMAL_WALL_METHOD,
+ role="formal_obb_wall",
+ trust_level="final_coordinate_source",
+ may_be_used_as_final_wall=True,
+ )
+ if len(segments) != 4:
+ return {"status": "invalid", "method": FORMAL_WALL_METHOD, "walls": [], "reason": "canonical segment generation failed", "canonical_naming_audit": naming_audit}
+ walls = []
+ for index, segment in enumerate(segments):
+ walls.append({
+ "id": WALL_IDS[index],
+ "type": "wall",
+ "first_point_xz": segment["first_point"],
+ "second_point_xz": segment["second_point"],
+ "source": FORMAL_WALL_METHOD,
+ "coordinate_frame": "OpenGL original XZ; Y is vertical",
+ "screen_clockwise_order_index": index + 1,
+ "geometry_wall_id": segment.get("geometry_wall_id"),
+ "midpoint_screen_xy": segment.get("midpoint_screen_xy"),
+ "canonical_naming": segment.get("canonical_naming"),
+ })
+ return {
+ "status": "available",
+ "schema_version": "obb_walls",
+ "method": FORMAL_WALL_METHOD,
+ "coordinate_frame": "OpenGL original XZ; Y is vertical",
+ "screen_coordinate_source": naming_audit.get("screen_coordinate_source"),
+ "screen_signed_area_before": naming_audit.get("screen_signed_area_before"),
+ "screen_signed_area_after": naming_audit.get("screen_signed_area_after"),
+ "screen_order_interpretation": naming_audit.get("screen_order_interpretation"),
+ "start_segment_policy": naming_audit.get("start_segment_policy"),
+ "model_visible_role_can_rotate_ids": False,
+ "walls": walls,
+ "primary_candidate": primary,
+ "canonical_naming_audit": naming_audit,
+ }
diff --git a/vlm_agent/reconstruction/relations/tools/wall_orientation_candidates.py b/vlm_agent/reconstruction/relations/tools/wall_orientation_candidates.py
new file mode 100644
index 0000000..df566ec
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/wall_orientation_candidates.py
@@ -0,0 +1,391 @@
+"""Local wall-orientation candidate utilities.
+
+The room extent still comes from the complete posed-mesh footprint. This
+module only proposes robust rectangle axes from object rotations already
+computed in the canonical top-down grid. A vision task may choose one of the
+finite candidates, but it never supplies coordinates or an arbitrary angle.
+"""
+
+from __future__ import annotations
+
+import math
+from typing import Any, Iterable
+
+
+def normalize_axis_angle_degrees(value: float) -> float:
+ """Normalize an undirected rectangular axis to ``[-45, 45)`` degrees."""
+ return (float(value) + 45.0) % 90.0 - 45.0
+
+
+def axis_angle_distance_degrees(first: float, second: float) -> float:
+ """Return the smallest distance between two axes with a 90-degree period."""
+ return abs(normalize_axis_angle_degrees(float(first) - float(second)))
+
+
+def _weighted_axis_mean(records: Iterable[dict[str, Any]]) -> float:
+ sine = 0.0
+ cosine = 0.0
+ for record in records:
+ angle = math.radians(float(record["axis_angle_degrees"]) * 4.0)
+ weight = float(record.get("weight") or 1.0)
+ sine += math.sin(angle) * weight
+ cosine += math.cos(angle) * weight
+ if abs(sine) <= 1e-12 and abs(cosine) <= 1e-12:
+ return 0.0
+ return normalize_axis_angle_degrees(math.degrees(math.atan2(sine, cosine)) / 4.0)
+
+
+def _orientation_record(item: dict[str, Any]) -> dict[str, Any] | None:
+ object_id = str(item.get("object_id") or "").strip()
+ rotation = item.get("rotation_degrees")
+ size = item.get("size_2d")
+ if not object_id or not isinstance(rotation, (int, float)) or isinstance(rotation, bool):
+ return None
+ if not isinstance(size, list) or len(size) != 2:
+ return None
+ try:
+ width = abs(float(size[0]))
+ height = abs(float(size[1]))
+ except (TypeError, ValueError):
+ return None
+ if width <= 1e-9 or height <= 1e-9:
+ return None
+ area = width * height
+ aspect_ratio = max(width, height) / max(min(width, height), 1e-9)
+ # Square-like and tiny footprints have an unstable semantic heading. The
+ # area square root prevents a single large central object from completely
+ # suppressing several agreeing wall-aligned objects.
+ weight = math.sqrt(area) * math.sqrt(min(aspect_ratio, 4.0))
+ return {
+ "object_id": object_id,
+ "rotation_degrees_topdown": round(float(rotation), 6),
+ "axis_angle_degrees": normalize_axis_angle_degrees(float(rotation)),
+ "size_2d": [width, height],
+ "area_2d": area,
+ "aspect_ratio": aspect_ratio,
+ "weight": weight,
+ }
+
+
+def cluster_object_orientation_axes(
+ objects: list[dict[str, Any]],
+ *,
+ grid_size: int,
+ cluster_tolerance_degrees: float = 10.0,
+ max_clusters: int = 3,
+) -> dict[str, Any]:
+ """Cluster reliable object headings into a small set of room-axis hints."""
+ raw_records = [record for item in objects if isinstance(item, dict) if (record := _orientation_record(item))]
+ max_area = max((float(record["area_2d"]) for record in raw_records), default=0.0)
+ grid_area = float(max(1, grid_size - 1) ** 2)
+ minimum_area = max(1.0, max_area * 0.01, grid_area * 0.0025)
+ reliable = [
+ record
+ for record in raw_records
+ if float(record["area_2d"]) >= minimum_area and float(record["aspect_ratio"]) >= 1.2
+ ]
+ if not reliable:
+ reliable = [record for record in raw_records if float(record["aspect_ratio"]) >= 1.05]
+
+ clusters: list[dict[str, Any]] = []
+ for record in sorted(reliable, key=lambda item: (-float(item["weight"]), item["object_id"])):
+ nearest = None
+ nearest_distance = float("inf")
+ for cluster in clusters:
+ distance = axis_angle_distance_degrees(record["axis_angle_degrees"], cluster["angle_degrees_xz"])
+ if distance < nearest_distance:
+ nearest = cluster
+ nearest_distance = distance
+ if nearest is None or nearest_distance > cluster_tolerance_degrees:
+ clusters.append({"members": [record], "angle_degrees_xz": record["axis_angle_degrees"]})
+ else:
+ nearest["members"].append(record)
+ nearest["angle_degrees_xz"] = _weighted_axis_mean(nearest["members"])
+
+ clusters.sort(
+ key=lambda cluster: (
+ -sum(float(item["weight"]) for item in cluster["members"]),
+ -len(cluster["members"]),
+ abs(float(cluster["angle_degrees_xz"])),
+ )
+ )
+ output_clusters = []
+ for index, cluster in enumerate(clusters[:max_clusters], start=1):
+ members = sorted(cluster["members"], key=lambda item: (-float(item["weight"]), item["object_id"]))
+ output_clusters.append({
+ "cluster_id": f"object_axis_cluster_{index:02d}",
+ "angle_degrees_xz": round(_weighted_axis_mean(members), 6),
+ "supporting_object_ids": [item["object_id"] for item in members],
+ "member_count": len(members),
+ "total_weight": round(sum(float(item["weight"]) for item in members), 6),
+ "members": [
+ {
+ "object_id": item["object_id"],
+ "rotation_degrees_topdown": item["rotation_degrees_topdown"],
+ "axis_angle_degrees": round(float(item["axis_angle_degrees"]), 6),
+ "size_2d": [round(float(value), 6) for value in item["size_2d"]],
+ "area_2d": round(float(item["area_2d"]), 6),
+ "aspect_ratio": round(float(item["aspect_ratio"]), 6),
+ "weight": round(float(item["weight"]), 6),
+ }
+ for item in members
+ ],
+ })
+ return {
+ "schema_version": "object_orientation_axis_clusters",
+ "coordinate_frame": "OpenGL original XZ; rectangular axes have a 90-degree period",
+ "grid_size": grid_size,
+ "cluster_tolerance_degrees": cluster_tolerance_degrees,
+ "minimum_reliable_area_2d": round(minimum_area, 6),
+ "raw_object_count": len(raw_records),
+ "reliable_object_count": len(reliable),
+ "clusters": output_clusters,
+ }
+
+
+
+
+
+
+WALL_ALIGNMENT_ROLES = {
+ "wall_mounted": 1.20,
+ "wall_embedded": 1.20,
+ "against_wall": 1.00,
+ "wall_aligned": 0.85,
+}
+WALL_ALIGNMENT_CONFIDENCE = {
+ "high": 1.00,
+ "medium": 0.65,
+}
+
+
+def wall_alignment_object_records(
+ task_results: Iterable[dict[str, Any]],
+) -> list[dict[str, Any]]:
+ """Extract compact Agent judgements about objects that reveal wall axes.
+
+ Candidate ids and wall ids are intentionally ignored here. The Agent's
+ job is semantic: identify objects whose real pose follows a room wall.
+ Local code later reads each selected object's canonical top-down rotation.
+ """
+ best_by_object: dict[str, dict[str, Any]] = {}
+
+ def walk(value: Any):
+ if isinstance(value, dict):
+ yield value
+ for child in value.values():
+ yield from walk(child)
+ elif isinstance(value, list):
+ for child in value:
+ yield from walk(child)
+
+ for result in task_results:
+ for node in walk(result):
+ for key in ("wall_alignment_objects", "likely_wall_objects"):
+ items = node.get(key)
+ if not isinstance(items, list):
+ continue
+ for item in items:
+ if not isinstance(item, dict):
+ continue
+ object_id = str(item.get("object_id") or "").strip()
+ role = str(item.get("role") or "").strip().lower()
+ confidence = str(item.get("confidence") or "").strip().lower()
+ if (
+ not object_id
+ or role not in WALL_ALIGNMENT_ROLES
+ or confidence not in WALL_ALIGNMENT_CONFIDENCE
+ ):
+ continue
+ score = WALL_ALIGNMENT_ROLES[role] * WALL_ALIGNMENT_CONFIDENCE[confidence]
+ current = best_by_object.get(object_id)
+ if current is None or score > float(current["semantic_weight"]):
+ best_by_object[object_id] = {
+ "object_id": object_id,
+ "role": role,
+ "confidence": confidence,
+ "semantic_weight": score,
+ }
+ return [best_by_object[key] for key in sorted(best_by_object)]
+
+
+def resolve_orientation_candidate_from_wall_objects(
+ task_results: Iterable[dict[str, Any]],
+ candidates: list[dict[str, Any]],
+ object_rotations_topdown: dict[str, float],
+ *,
+ agreement_limit_degrees: float = 22.5,
+ minimum_score: float = 0.35,
+ minimum_margin: float = 0.05,
+) -> dict[str, Any]:
+ """Score finite wall axes from Agent-selected objects and local rotations.
+
+ The Agent never chooses a candidate in this path. It only identifies
+ semantically reliable wall-following objects. The returned choice is a
+ deterministic function of their rotation_degrees_topdown and the already
+ generated finite candidate angles.
+ """
+ semantic_records = wall_alignment_object_records(task_results)
+ # Candidate generation records an axis reliability weight for each object.
+ # Use it only to break semantic-score ties among Agent-selected objects.
+ reliability_by_object: dict[str, float] = {}
+ for candidate in candidates:
+ evidence = candidate.get("evidence") if isinstance(candidate, dict) else None
+ cluster = evidence.get("orientation_cluster") if isinstance(evidence, dict) else None
+ members = cluster.get("members") if isinstance(cluster, dict) else None
+ if not isinstance(members, list):
+ continue
+ for member in members:
+ if not isinstance(member, dict) or not member.get("object_id"):
+ continue
+ try:
+ weight = max(float(member.get("weight", 1.0)), 1e-6)
+ except (TypeError, ValueError):
+ weight = 1.0
+ object_id = str(member["object_id"])
+ reliability_by_object[object_id] = max(reliability_by_object.get(object_id, 0.0), weight)
+
+ usable_records: list[dict[str, Any]] = []
+ missing_rotation_ids: list[str] = []
+ for item in semantic_records:
+ object_id = item["object_id"]
+ rotation = object_rotations_topdown.get(object_id)
+ if not isinstance(rotation, (int, float)) or isinstance(rotation, bool):
+ missing_rotation_ids.append(object_id)
+ continue
+ usable_records.append({
+ **item,
+ "rotation_degrees_topdown": round(float(rotation), 6),
+ "axis_angle_degrees": round(normalize_axis_angle_degrees(float(rotation)), 6),
+ "orientation_reliability_weight": round(reliability_by_object.get(object_id, 1.0), 6),
+ })
+
+ candidate_scores: list[dict[str, Any]] = []
+ total_semantic_weight = sum(float(item["semantic_weight"]) for item in usable_records)
+ total_reliability_weight = sum(
+ float(item["semantic_weight"]) * float(item["orientation_reliability_weight"])
+ for item in usable_records
+ )
+ for candidate in candidates:
+ if not isinstance(candidate, dict) or not candidate.get("candidate_id"):
+ continue
+ orientation = candidate.get("orientation") if isinstance(candidate.get("orientation"), dict) else {}
+ angle = orientation.get("angle_degrees_xz")
+ if not isinstance(angle, (int, float)) or isinstance(angle, bool):
+ continue
+ object_scores = []
+ weighted_agreement = 0.0
+ reliability_weighted_agreement = 0.0
+ for item in usable_records:
+ distance = axis_angle_distance_degrees(item["axis_angle_degrees"], float(angle))
+ agreement = max(0.0, 1.0 - distance / max(agreement_limit_degrees, 1e-6))
+ contribution = float(item["semantic_weight"]) * agreement
+ reliability_contribution = (
+ float(item["semantic_weight"])
+ * float(item["orientation_reliability_weight"])
+ * agreement
+ )
+ weighted_agreement += contribution
+ reliability_weighted_agreement += reliability_contribution
+ object_scores.append({
+ "object_id": item["object_id"],
+ "axis_distance_degrees": round(distance, 6),
+ "agreement": round(agreement, 6),
+ "weighted_contribution": round(contribution, 6),
+ "orientation_reliability_weight": item["orientation_reliability_weight"],
+ "reliability_weighted_contribution": round(reliability_contribution, 6),
+ })
+ normalized_score = weighted_agreement / total_semantic_weight if total_semantic_weight > 0 else 0.0
+ reliability_normalized_score = (
+ reliability_weighted_agreement / total_reliability_weight
+ if total_reliability_weight > 0 else normalized_score
+ )
+ candidate_scores.append({
+ "candidate_id": str(candidate["candidate_id"]),
+ "angle_degrees_xz": round(float(angle), 6),
+ "normalized_score": round(normalized_score, 6),
+ "weighted_score": round(weighted_agreement, 6),
+ "reliability_normalized_score": round(reliability_normalized_score, 6),
+ "reliability_weighted_score": round(reliability_weighted_agreement, 6),
+ "object_scores": object_scores,
+ })
+
+ candidate_scores.sort(key=lambda item: (-float(item["normalized_score"]), item["candidate_id"]))
+ best = candidate_scores[0] if candidate_scores else None
+ runner_up = candidate_scores[1] if len(candidate_scores) > 1 else None
+ margin = (
+ float(best["normalized_score"]) - float(runner_up["normalized_score"])
+ if best and runner_up else float(best["normalized_score"]) if best else 0.0
+ )
+ resolved_id = None
+ status = "no_usable_agent_wall_alignment_objects"
+ reliability_tiebreak: dict[str, Any] = {
+ "used": False,
+ "resolved_candidate_id": None,
+ "best_score": None,
+ "runner_up_score": None,
+ "score_margin": None,
+ }
+ if best and usable_records:
+ if float(best["normalized_score"]) < minimum_score:
+ status = "agent_objects_do_not_match_any_finite_candidate"
+ elif runner_up and margin < minimum_margin:
+ reliability_ranked = sorted(
+ candidate_scores,
+ key=lambda item: (-float(item["reliability_normalized_score"]), item["candidate_id"]),
+ )
+ reliability_best = reliability_ranked[0]
+ reliability_runner_up = reliability_ranked[1] if len(reliability_ranked) > 1 else None
+ reliability_margin = (
+ float(reliability_best["reliability_normalized_score"])
+ - float(reliability_runner_up["reliability_normalized_score"])
+ if reliability_runner_up else float(reliability_best["reliability_normalized_score"])
+ )
+ reliability_tiebreak = {
+ "used": True,
+ "resolved_candidate_id": None,
+ "best_candidate_id": reliability_best["candidate_id"],
+ "best_score": float(reliability_best["reliability_normalized_score"]),
+ "runner_up_candidate_id": reliability_runner_up["candidate_id"] if reliability_runner_up else None,
+ "runner_up_score": (
+ float(reliability_runner_up["reliability_normalized_score"])
+ if reliability_runner_up else None
+ ),
+ "score_margin": round(reliability_margin, 6),
+ "minimum_margin": minimum_margin,
+ "policy": "Break semantic-score ties only with local geometric reliability weights for the Agent-selected wall-following objects.",
+ }
+ if (
+ float(reliability_best["reliability_normalized_score"]) >= minimum_score
+ and reliability_margin >= minimum_margin
+ ):
+ resolved_id = str(reliability_best["candidate_id"])
+ reliability_tiebreak["resolved_candidate_id"] = resolved_id
+ status = "selected_by_local_agent_object_rotation_reliability_tiebreak"
+ else:
+ status = "finite_candidates_ambiguous_for_agent_objects"
+ else:
+ resolved_id = str(best["candidate_id"])
+ status = "selected_by_local_agent_object_rotation_score"
+
+ return {
+ "resolved_candidate_id": resolved_id,
+ "resolution_source": (
+ "local_agent_object_rotation_reliability_tiebreak"
+ if resolved_id and reliability_tiebreak.get("resolved_candidate_id")
+ else "local_agent_object_rotation_score" if resolved_id
+ else "no_local_object_score_selection"
+ ),
+ "status": status,
+ "wall_alignment_objects": usable_records,
+ "missing_rotation_object_ids": missing_rotation_ids,
+ "candidate_scores": candidate_scores,
+ "best_score": float(best["normalized_score"]) if best else 0.0,
+ "runner_up_score": float(runner_up["normalized_score"]) if runner_up else None,
+ "score_margin": round(margin, 6),
+ "reliability_tiebreak": reliability_tiebreak,
+ "agreement_limit_degrees": agreement_limit_degrees,
+ "minimum_score": minimum_score,
+ "minimum_margin": minimum_margin,
+ "policy": "Agent selects semantic wall-alignment objects; local canonical rotations score finite formal OBB axes.",
+ }
diff --git a/vlm_agent/reconstruction/relations/tools/write_agent_process_report.py b/vlm_agent/reconstruction/relations/tools/write_agent_process_report.py
new file mode 100644
index 0000000..52335f8
--- /dev/null
+++ b/vlm_agent/reconstruction/relations/tools/write_agent_process_report.py
@@ -0,0 +1,402 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+ROOT = Path(__file__).resolve().parents[2]
+if str(ROOT) not in sys.path:
+ sys.path.insert(0, str(ROOT))
+
+from relations.agents.schema_guard import wall_axis_alignment_summary
+from relations.agents.state import sorted_object_id_key, write_json, write_md
+
+
+def _load_json(path: Path | None) -> Any:
+ if path is None or not path.exists():
+ return None
+ try:
+ return json.loads(path.read_text(encoding="utf-8"))
+ except Exception:
+ return None
+
+
+def _load_or(value: Any, path: Path | None, default: Any) -> Any:
+ if value is not None:
+ return value
+ loaded = _load_json(path)
+ return loaded if loaded is not None else default
+
+
+def _object_ids(scene_state: dict[str, Any]) -> list[str]:
+ items = scene_state.get("object_candidates", {}).get("items", []) if isinstance(scene_state, dict) else []
+ ids = [str(item.get("object_id")) for item in items if isinstance(item, dict) and item.get("object_id")]
+ return sorted(ids, key=sorted_object_id_key)
+
+
+def _task_purpose(task: dict[str, Any]) -> str:
+ for key in ["purpose", "instructions", "reason", "objective"]:
+ value = task.get(key)
+ if isinstance(value, str) and value.strip():
+ return value.strip().replace("\n", " ")[:220]
+ return ""
+
+
+def _task_summary(manager_plan: dict[str, Any], api_called: bool | None) -> list[dict[str, Any]]:
+ tasks = manager_plan.get("tasks", []) if isinstance(manager_plan, dict) else []
+ result = []
+ for task in tasks if isinstance(tasks, list) else []:
+ if not isinstance(task, dict):
+ continue
+ result.append(
+ {
+ "task_id": task.get("task_id"),
+ "task_type": task.get("task_type"),
+ "purpose": _task_purpose(task),
+ "api_used": bool(api_called),
+ "vision_used": bool(task.get("use_vision_model") or task.get("wall_visual_image_paths") or "wall" in str(task.get("task_type", "")).lower()),
+ }
+ )
+ return result
+
+
+def _count_relations(relations: list[dict[str, Any]]) -> dict[str, int]:
+ counts = {"floor": 0, "ceiling": 0, "wall": 0, "object": 0}
+ for rel in relations if isinstance(relations, list) else []:
+ if not isinstance(rel, dict) or rel.get("relation") != "attach":
+ continue
+ anchor = str(rel.get("parameters", {}).get("object_anchor"))
+ if anchor == "floor":
+ counts["floor"] += 1
+ elif anchor == "ceiling":
+ counts["ceiling"] += 1
+ elif anchor.startswith("wall_"):
+ counts["wall"] += 1
+ elif anchor.startswith("object_"):
+ counts["object"] += 1
+ return counts
+
+
+def _final_relations(output_dir: Path) -> list[dict[str, Any]]:
+ relations = _load_json(output_dir / "final_outputs" / "relations.json")
+ return relations if isinstance(relations, list) else []
+
+
+def _final_walls(output_dir: Path) -> list[dict[str, Any]]:
+ payload = _load_json(output_dir / "final_outputs" / "objects.json")
+ if not isinstance(payload, dict) or not isinstance(payload.get("walls"), list):
+ return []
+ return [wall for wall in payload["walls"] if isinstance(wall, dict)]
+
+
+def _glob_first(output_dir: Path, pattern: str) -> Path | None:
+ matches = sorted(output_dir.glob(pattern))
+ return matches[0] if matches else None
+
+
+def write_final_process_report(
+ output_dir: Path,
+ scene_state: dict[str, Any] | None = None,
+ manager_plan: dict[str, Any] | None = None,
+ task_results: list[dict[str, Any]] | None = None,
+ validation: dict[str, Any] | None = None,
+ wall_evidence: dict[str, Any] | None = None,
+ wall_visual_evidence: dict[str, Any] | None = None,
+ final_wall_overlay: dict[str, Any] | None = None,
+ final_wall_overlay_skipped_reason: str | None = None,
+ api_called: bool | None = None,
+ api_call_count: int | None = None,
+ finalization_status: str | None = None,
+ runtime_summary: dict[str, Any] | None = None,
+) -> dict[str, Any]:
+ output_dir = Path(output_dir)
+ scene_state_path = output_dir / "scene_states" / "scene_state.json"
+ manager_plan_path = output_dir / "manager_plans" / "manager_plan_000.json"
+ validation_path = output_dir / "debug" / "final_validation" / "schema_validation.json"
+ wall_evidence_path = output_dir / "debug" / "walls" / "wall_evidence.json"
+ wall_visual_manifest_path = _glob_first(output_dir, "debug/walls/visual_review/*_wall_visual_review_manifest.json")
+ wall_audit_path = output_dir / "debug" / "walls" / "wall_audit.json"
+ wall_object_hypotheses_path = output_dir / "debug" / "walls" / "wall_object_hypotheses.json"
+ attach_audit_path = output_dir / "debug" / "attach" / "attach_decision_audit.json"
+ coverage_audit_path = output_dir / "debug" / "attach" / "coverage_repair_audit.json"
+ accepted_merge_path = output_dir / "debug" / "attach" / "accepted_attach_merge_check.json"
+ runtime_summary_path = output_dir / "debug" / "agents" / "runtime_summary.json"
+
+ scene_state = _load_or(scene_state, scene_state_path, {})
+ manager_plan = _load_or(manager_plan, manager_plan_path, {})
+ validation = _load_or(validation, validation_path, {})
+ wall_evidence = _load_or(wall_evidence, wall_evidence_path, {})
+ wall_visual_evidence = _load_or(wall_visual_evidence, wall_visual_manifest_path, {})
+ wall_audit = _load_json(wall_audit_path) or {}
+ wall_object_hypotheses = _load_json(wall_object_hypotheses_path) or {}
+ attach_audit = _load_json(attach_audit_path) or {}
+ coverage_audit = _load_json(coverage_audit_path) or {}
+ accepted_merge = _load_json(accepted_merge_path) or {}
+ runtime_summary = runtime_summary or _load_json(runtime_summary_path) or {}
+
+ ids = _object_ids(scene_state)
+ relation_counts = _count_relations(_final_relations(output_dir))
+ final_walls = _final_walls(output_dir)
+ tasks = _task_summary(manager_plan, api_called)
+ candidate_walls = wall_evidence.get("candidate_walls", []) if isinstance(wall_evidence, dict) else []
+ candidate_sources = sorted({str(item.get("source")) for item in candidate_walls if isinstance(item, dict) and item.get("source")})
+ candidate_trust = sorted({str(item.get("trust_level")) for item in candidate_walls if isinstance(item, dict) and item.get("trust_level")})
+ wall_adjudication = validation.get("wall_final_adjudication") or wall_audit.get("wall_final_adjudication") or {}
+ promotion = wall_audit.get("candidate_wall_promotion") or {}
+ weak_wall_completion = wall_audit.get("weak_wall_completion") or validation.get("weak_wall_completion") or {}
+ candidate_modified = bool(wall_adjudication.get("candidate_modified") or promotion.get("candidate_modified"))
+ revision_reasons = wall_adjudication.get("revision_reasons") or promotion.get("revision_reasons") or []
+ overlay_png = None
+ if isinstance(final_wall_overlay, dict):
+ overlay_png = final_wall_overlay.get("output_overlay_png") or final_wall_overlay.get("output")
+ overlay_png = overlay_png or str(output_dir / "debug" / "walls" / "final_walls_overlay_on_canonical_topdown.png")
+ overlay_exists = Path(overlay_png).exists() if overlay_png else False
+ if overlay_exists:
+ final_wall_overlay_skipped_reason = None
+ elif not final_wall_overlay_skipped_reason:
+ final_wall_overlay_skipped_reason = "final wall overlay skipped because final walls were not generated"
+
+ objects_path = output_dir / "final_outputs" / "objects.json"
+ relations_path = output_dir / "final_outputs" / "relations.json"
+ objects_valid = validation.get("objects", {}).get("valid") if isinstance(validation.get("objects"), dict) else None
+ relations_valid = validation.get("relations", {}).get("valid") if isinstance(validation.get("relations"), dict) else None
+ target_coverage = validation.get("relations", {}).get("target_coverage", {}) if isinstance(validation.get("relations"), dict) else {}
+ wall_geometry = validation.get("objects", {}).get("wall_geometry") if isinstance(validation.get("objects"), dict) else wall_audit.get("geometry")
+ wall_geometry_details = wall_geometry.get("details", {}) if isinstance(wall_geometry, dict) and isinstance(wall_geometry.get("details"), dict) else {}
+ orientation_summary = wall_axis_alignment_summary(final_walls) if final_walls else {}
+ axis_aligned_final_walls = wall_geometry_details.get("axis_aligned", orientation_summary.get("axis_aligned"))
+ oriented_boundary_inferred = wall_geometry_details.get("oriented_boundary", orientation_summary.get("oriented_boundary"))
+ edge_orientations = wall_geometry_details.get("edge_orientations", orientation_summary.get("edge_orientations", []))
+ wall_object_hypotheses_influenced_orientation = bool(
+ wall_object_hypotheses.get("available") and (oriented_boundary_inferred or candidate_modified)
+ )
+
+ report = {
+ "scene_summary": {
+ "scene_name": scene_state.get("scene_name"),
+ "object_candidate_count": len(ids),
+ "object_id_range": [ids[0], ids[-1]] if ids else [],
+ "output_dir": str(output_dir),
+ },
+ "evidence_preparation": {
+ "scene_state_path": str(scene_state_path),
+ "scene_state_exists": scene_state_path.exists(),
+ "canonical_topdown_path": wall_visual_evidence.get("canonical_topdown_path") or scene_state.get("topdown_grid_mapping", {}).get("canonical_topdown_image_path"),
+ "canonical_grid_mapping_path": wall_visual_evidence.get("canonical_grid_mapping_path") or scene_state.get("topdown_grid_mapping", {}).get("canonical_mapping_path"),
+ "fixed_layout_views_manifest": scene_state.get("evidence", {}).get("fixed_layout_views", {}).get("manifest_path"),
+ "wall_evidence_path": str(wall_evidence_path),
+ "wall_visual_contact_sheet": wall_visual_evidence.get("contact_sheet_path"),
+ "original_image_path": wall_visual_evidence.get("original_image_path") or scene_state.get("evidence", {}).get("original_image_path"),
+ "mask_overlay_path": wall_visual_evidence.get("mask_overlay_path") or scene_state.get("evidence", {}).get("mask_overlay_path"),
+ },
+ "manager_plan": {"task_count": len(tasks), "tasks": tasks},
+ "wall_reasoning": {
+ "candidate_walls_count": len(candidate_walls) if isinstance(candidate_walls, list) else 0,
+ "candidate_walls_source": candidate_sources,
+ "candidate_walls_trust_level": candidate_trust,
+ "promotion_rubric": {
+ "object_occupied_bounds": "interior_occupancy_evidence_not_wall_boundary",
+ "room_wall_boundary": "may_be_larger_than_object_extents_and_enclose_empty_floor_area",
+ "camera_grid_extent": "reject_only_when_no_visual_room_boundary_support_or_excessive_unrelated_padding",
+ },
+ "wall_final_adjudication": wall_adjudication,
+ "decision": wall_adjudication.get("decision"),
+ "candidate_modified": candidate_modified,
+ "revision_reasons": revision_reasons,
+ "candidate_promotion": promotion,
+ "axis_aligned_final_walls": axis_aligned_final_walls,
+ "oriented_boundary_inferred": oriented_boundary_inferred,
+ "edge_orientations": edge_orientations,
+ "wall_object_hypotheses_influenced_orientation": wall_object_hypotheses_influenced_orientation,
+ "grid_camera_extent_usage": "weak_reference_only; reject when it is only a loose extent rectangle and does not follow the actual room footprint",
+ "weak_wall_completion": weak_wall_completion,
+ "weak_wall_completion_used": bool(weak_wall_completion.get("used") or weak_wall_completion.get("weak_wall_completion_used")),
+ "weak_wall_ids": weak_wall_completion.get("weak_wall_ids", []),
+ "weak_wall_completion_methods": weak_wall_completion.get("completion_methods", []),
+ "weak_wall_completion_reasons": weak_wall_completion.get("completion_reasons", []),
+ "oriented_boundary_rule": "Axis-aligned walls remain allowed when they match the true room footprint; oblique walls are allowed and preferred when the footprint/object layout is oblique.",
+ "final_wall_source": "obb_from_mesh_footprint_min_rect" if promotion.get("promotion_from_candidate") or wall_adjudication.get("accepted_final_walls") else None,
+ "overlay_png": overlay_png if overlay_exists else None,
+ "overlay_skipped_reason": final_wall_overlay_skipped_reason,
+ },
+ "wall_object_hypotheses": {
+ "available": bool(wall_object_hypotheses.get("available")),
+ "likely_wall_objects_count": int(wall_object_hypotheses.get("likely_wall_objects_count") or 0),
+ "wall_line_hypotheses_count": int(wall_object_hypotheses.get("wall_line_hypotheses_count") or 0),
+ "source_task_ids": wall_object_hypotheses.get("source_task_ids", []),
+ "used_in_final_adjudication": bool(wall_object_hypotheses.get("available") and wall_adjudication.get("executed")),
+ "likely_wall_objects": wall_object_hypotheses.get("likely_wall_objects", []),
+ "wall_line_hypotheses": wall_object_hypotheses.get("wall_line_hypotheses", []),
+ "json_path": str(wall_object_hypotheses_path),
+ },
+ "attach_reasoning": {
+ "floor_attach_count": relation_counts["floor"],
+ "ceiling_attach_count": relation_counts["ceiling"],
+ "ceiling_confirmed": bool(validation.get("ceiling_confirmed") or attach_audit.get("ceiling_confirmed")),
+ "wall_attach_count": relation_counts["wall"],
+ "object_to_object_attach_count": relation_counts["object"],
+ "accepted_candidate_merge_valid": accepted_merge.get("valid"),
+ "accepted_candidate_merge_items": len(accepted_merge.get("items", [])) if isinstance(accepted_merge.get("items"), list) else 0,
+ "skipped_candidate_count": len(attach_audit.get("skipped_candidates", [])) if isinstance(attach_audit.get("skipped_candidates"), list) else 0,
+ "common_skip_reasons": sorted({str(item.get("reason")) for item in attach_audit.get("skipped_candidates", []) if isinstance(item, dict) and item.get("reason")})[:10],
+ },
+ "coverage_repair": {
+ "missing_before": coverage_audit.get("missing_before", []),
+ "missing_after": coverage_audit.get("missing_after", []),
+ "coverage_repair_used": bool(coverage_audit.get("coverage_repair_used")),
+ "coverage_repair_count": int(coverage_audit.get("coverage_repair_count") or 0),
+ "missing_object_count_before_repair": len(coverage_audit.get("missing_before", [])) if isinstance(coverage_audit.get("missing_before"), list) else 0,
+ "items": coverage_audit.get("items", []),
+ "coverage_repair_audit_path": str(coverage_audit_path),
+ },
+ "final_validation": {
+ "objects_json_generated": objects_path.exists(),
+ "relations_json_generated": relations_path.exists(),
+ "schema_validation_valid": bool(validation.get("valid")),
+ "objects_valid": objects_valid,
+ "relations_valid": relations_valid,
+ "wall_geometry_valid": wall_geometry.get("valid") if isinstance(wall_geometry, dict) else None,
+ "object_target_coverage_complete": target_coverage.get("complete") if isinstance(target_coverage, dict) else None,
+ "accepted_attach_merge_valid": accepted_merge.get("valid"),
+ "finalization_status": finalization_status or validation.get("finalization_status"),
+ "errors": validation.get("errors", []),
+ },
+ "important_debug_files": {
+ "final_objects": str(objects_path),
+ "final_relations": str(relations_path),
+ "schema_validation_md": str(output_dir / "debug" / "final_validation" / "schema_validation.md"),
+ "wall_audit_md": str(output_dir / "debug" / "walls" / "wall_audit.md"),
+ "wall_object_hypotheses_md": str(output_dir / "debug" / "walls" / "wall_object_hypotheses.md"),
+ "final_wall_overlay": str(output_dir / "debug" / "walls" / "final_walls_overlay_on_canonical_topdown.png"),
+ "coverage_repair_audit_md": str(output_dir / "debug" / "attach" / "coverage_repair_audit.md"),
+ "accepted_attach_merge_check_md": str(output_dir / "debug" / "attach" / "accepted_attach_merge_check.md"),
+ "runtime_summary_json": str(runtime_summary_path),
+ "runtime_summary_md": str(output_dir / "debug" / "agents" / "runtime_summary.md"),
+ "manager_plan": str(manager_plan_path),
+ "agent_outputs": str(output_dir / "agent_outputs"),
+ },
+ "human_review_notes": [
+ "Check whether final_walls_overlay_on_canonical_topdown.png fits the true room boundary.",
+ "Review whether wall attach relations are physically plausible after final wall ids are stable.",
+ "Review coverage repair anchors for plausible physical support.",
+ "Review object descriptions for obvious image or mask conflicts.",
+ ],
+ "runtime": {"api_called": bool(api_called), "api_call_count": api_call_count, "summary": runtime_summary},
+ }
+
+ lines = [
+ "## Execution Summary",
+ f"- api_calls: `{runtime_summary.get('api_calls', api_call_count)}`",
+ f"- render_calls: `{runtime_summary.get('render_calls')}`",
+ f"- runtime_summary_json: `{runtime_summary_path}`",
+ "",
+ "## 1. Scene Summary",
+ f"- scene_name: `{report['scene_summary']['scene_name']}`",
+ f"- object_candidate_count: `{report['scene_summary']['object_candidate_count']}`",
+ f"- object_id_range: `{report['scene_summary']['object_id_range']}`",
+ f"- output_dir: `{output_dir}`",
+ "",
+ "## 2. Evidence Preparation",
+ ]
+ for key, value in report["evidence_preparation"].items():
+ lines.append(f"- {key}: `{value}`")
+ lines.extend(["", "## 3. Manager Plan"])
+ for task in tasks:
+ lines.append(f"- `{task.get('task_id')}` {task.get('task_type')}: API=`{task.get('api_used')}` vision=`{task.get('vision_used')}` purpose={task.get('purpose')}")
+ if not tasks:
+ lines.append("- No manager tasks recorded.")
+ lines.extend([
+ "",
+ "## 4. Wall Reasoning",
+ f"- candidate_walls_count: `{report['wall_reasoning']['candidate_walls_count']}`",
+ f"- candidate_walls_source: `{candidate_sources}`",
+ f"- candidate_walls_trust_level: `{candidate_trust}`",
+ "- object_occupied_bounds: `interior_occupancy_evidence_not_wall_boundary`",
+ "- room_wall_boundary: `may_be_larger_than_object_extents_and_enclose_empty_floor_area`",
+ "- camera_grid_extent: `reject_only_when_no_visual_room_boundary_support_or_excessive_unrelated_padding`",
+ "- rubric_note: Candidate wall promotion judges plausible room boundary alignment, not tight fit to object occupied bounds.",
+ "- oriented_boundary_rule: Axis-aligned walls are allowed if they match the true room footprint; oblique walls are allowed and preferred if the room footprint/object layout is oblique.",
+ f"- axis_aligned_final_walls: `{report['wall_reasoning']['axis_aligned_final_walls']}`",
+ f"- oriented_boundary_inferred: `{report['wall_reasoning']['oriented_boundary_inferred']}`",
+ f"- edge_orientations: `{report['wall_reasoning']['edge_orientations']}`",
+ f"- wall_object_hypotheses_influenced_orientation: `{report['wall_reasoning']['wall_object_hypotheses_influenced_orientation']}`",
+ f"- grid_camera_extent_usage: `{report['wall_reasoning']['grid_camera_extent_usage']}`",
+ f"- weak_wall_completion_used: `{report['wall_reasoning']['weak_wall_completion_used']}`",
+ f"- weak_wall_ids: `{report['wall_reasoning']['weak_wall_ids']}`",
+ f"- weak_wall_completion_methods: `{report['wall_reasoning']['weak_wall_completion_methods']}`",
+ f"- wall_final_adjudication: `{wall_adjudication}`",
+ f"- candidate_promotion: `{promotion}`",
+ f"- final_wall_source: `{report['wall_reasoning']['final_wall_source']}`",
+ f"- overlay_png: `{report['wall_reasoning']['overlay_png']}`",
+ f"- overlay_skipped_reason: `{report['wall_reasoning']['overlay_skipped_reason']}`",
+ "",
+ "## Wall Object Hypotheses",
+ f"- available: `{report['wall_object_hypotheses']['available']}`",
+ f"- likely_wall_objects_count: `{report['wall_object_hypotheses']['likely_wall_objects_count']}`",
+ f"- wall_line_hypotheses_count: `{report['wall_object_hypotheses']['wall_line_hypotheses_count']}`",
+ f"- source_task_ids: `{report['wall_object_hypotheses']['source_task_ids']}`",
+ f"- used_in_final_adjudication: `{report['wall_object_hypotheses']['used_in_final_adjudication']}`",
+ f"- json_path: `{wall_object_hypotheses_path}`",
+ "",
+ "## Hidden / Weak Wall Completion",
+ f"- used: `{report['wall_reasoning']['weak_wall_completion_used']}`",
+ f"- weak_wall_ids: `{report['wall_reasoning']['weak_wall_ids']}`",
+ f"- completion_methods: `{report['wall_reasoning']['weak_wall_completion_methods']}`",
+ f"- completion_reasons: `{report['wall_reasoning']['weak_wall_completion_reasons']}`",
+ "",
+ "## Wall Final Adjudication",
+ f"- decision: `{wall_adjudication.get('decision')}`",
+ f"- candidate_modified: `{candidate_modified}`",
+ f"- revision_reasons: `{revision_reasons}`",
+ f"- final_walls_accepted: `{wall_adjudication.get('accepted_final_walls')}`",
+ "",
+ "## 5. Attach Reasoning",
+ f"- floor_attach_count: `{relation_counts['floor']}`",
+ f"- ceiling_confirmed: `{report['attach_reasoning']['ceiling_confirmed']}`",
+ f"- ceiling_attach_count: `{relation_counts['ceiling']}`",
+ f"- wall_attach_count: `{relation_counts['wall']}`",
+ f"- object_to_object_attach_count: `{relation_counts['object']}`",
+ f"- accepted_candidate_merge_valid: `{accepted_merge.get('valid')}`",
+ f"- skipped_candidate_count: `{report['attach_reasoning']['skipped_candidate_count']}`",
+ f"- common_skip_reasons: `{report['attach_reasoning']['common_skip_reasons']}`",
+ "",
+ "## 6. Coverage Repair",
+ "## Attach Target Coverage",
+ f"- missing_object_count_before_repair: `{report['coverage_repair']['missing_object_count_before_repair']}`",
+ f"- coverage_repair_used: `{report['coverage_repair']['coverage_repair_used']}`",
+ f"- coverage_repair_count: `{report['coverage_repair']['coverage_repair_count']}`",
+ f"- missing_before: `{coverage_audit.get('missing_before', [])}`",
+ f"- missing_after: `{coverage_audit.get('missing_after', [])}`",
+ f"- coverage_repair_used: `{coverage_audit.get('coverage_repair_used')}`",
+ f"- coverage_repair_count: `{coverage_audit.get('coverage_repair_count')}`",
+ f"- coverage_repair_audit: `{coverage_audit_path}`",
+ "",
+ "## 7. Final Validation",
+ ])
+ for key, value in report["final_validation"].items():
+ lines.append(f"- {key}: `{value}`")
+ lines.extend(["", "## 8. Important Debug Files"])
+ for key, value in report["important_debug_files"].items():
+ lines.append(f"- {key}: `{value}`")
+ lines.extend(["", "## 9. Human Review Notes"])
+ lines.extend(f"- {item}" for item in report["human_review_notes"])
+
+ json_path = output_dir / "debug" / "agents" / "final_process_report.json"
+ md_path = output_dir / "debug" / "agents" / "final_process_report.md"
+ write_json(json_path, report)
+ write_md(md_path, "Final Process Report", lines)
+ return {"json_path": str(json_path), "md_path": str(md_path), **report}
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Write a human-readable final multi-agent process report.")
+ parser.add_argument("--output_dir", required=True, type=Path)
+ args = parser.parse_args()
+ result = write_final_process_report(args.output_dir)
+ print(json.dumps({"md_path": result["md_path"], "json_path": result["json_path"]}, indent=2, ensure_ascii=False))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/vlm_agent/reconstruction/sam3_runtime.py b/vlm_agent/reconstruction/sam3_runtime.py
new file mode 100644
index 0000000..5dc2cc4
--- /dev/null
+++ b/vlm_agent/reconstruction/sam3_runtime.py
@@ -0,0 +1,165 @@
+from __future__ import annotations
+
+import json
+import os
+import re
+import subprocess
+from pathlib import Path
+from typing import Any
+
+
+EXPECTED_TRANSFORMERS_VERSION = "5.5.4"
+PERCEPTION_DEPENDENCY_MODULES = (
+ "PIL",
+ "cv2",
+ "scipy",
+ "transformers",
+ "openai",
+ "huggingface_hub",
+ "typing_extensions",
+)
+SENSITIVE_ENV_NAME_RE = re.compile(
+ r"(?:KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL|AUTHORIZATION|BASE_URL)", re.I
+)
+
+
+def verify_perception_dependencies(
+ perception_python: Path,
+) -> dict[str, Any]:
+ """Import each perception dependency in an isolated CPU-only process."""
+
+ python = Path(perception_python).expanduser().resolve()
+ base: dict[str, Any] = {
+ "required_modules": list(PERCEPTION_DEPENDENCY_MODULES),
+ "api_calls": 0,
+ "gpu_used": False,
+ }
+ if not python.is_file() or python.is_symlink():
+ return {**base, "valid": False, "failure": "python missing"}
+
+ environment = {
+ name: value
+ for name, value in os.environ.items()
+ if not SENSITIVE_ENV_NAME_RE.search(name)
+ }
+ environment["CUDA_VISIBLE_DEVICES"] = ""
+ environment["PYTHONDONTWRITEBYTECODE"] = "1"
+ environment["PYTHONNOUSERSITE"] = "1"
+ results: list[dict[str, Any]] = []
+ for module_name in PERCEPTION_DEPENDENCY_MODULES:
+ try:
+ completed = subprocess.run(
+ [
+ str(python),
+ "-c",
+ (
+ "import importlib; "
+ f"importlib.import_module({module_name!r})"
+ ),
+ ],
+ capture_output=True,
+ text=True,
+ timeout=30,
+ check=False,
+ env=environment,
+ )
+ results.append(
+ {
+ "module": module_name,
+ "valid": completed.returncode == 0,
+ "returncode": completed.returncode,
+ }
+ )
+ except (OSError, subprocess.SubprocessError) as exc:
+ results.append(
+ {
+ "module": module_name,
+ "valid": False,
+ "failure": type(exc).__name__,
+ }
+ )
+ return {
+ **base,
+ "valid": all(item["valid"] for item in results),
+ "modules": results,
+ }
+
+
+def verify_sam3_runtime_provider(
+ perception_python: Path,
+ source_root: Path,
+) -> dict[str, Any]:
+ # Verify the version and import origin used by the perception runtime.
+ base: dict[str, Any] = {
+ "provider": "transformers",
+ "expected_version": EXPECTED_TRANSFORMERS_VERSION,
+ "api_calls": 0,
+ "gpu_used": False,
+ }
+ python = Path(perception_python).expanduser().resolve()
+ source = Path(source_root).expanduser().resolve()
+ if not python.is_file() or python.is_symlink():
+ return {**base, "valid": False, "failure": "python missing"}
+
+ script = """
+import importlib.metadata
+import importlib.util
+import json
+from pathlib import Path
+
+distribution = importlib.metadata.distribution("transformers")
+spec = importlib.util.find_spec("sam3")
+print(json.dumps({
+ "version": distribution.version,
+ "sam3_origin": (
+ str(Path(spec.origin).resolve())
+ if spec and spec.origin
+ else None
+ ),
+}))
+"""
+ environment = {
+ name: value
+ for name, value in os.environ.items()
+ if not SENSITIVE_ENV_NAME_RE.search(name)
+ }
+ environment["PYTHONPATH"] = str(source)
+ environment["PYTHONDONTWRITEBYTECODE"] = "1"
+ environment["PYTHONNOUSERSITE"] = "1"
+ try:
+ completed = subprocess.run(
+ [str(python), "-c", script],
+ capture_output=True,
+ text=True,
+ timeout=30,
+ check=False,
+ env=environment,
+ )
+ except (OSError, subprocess.SubprocessError) as exc:
+ return {**base, "valid": False, "failure": type(exc).__name__}
+ if completed.returncode != 0:
+ return {
+ **base,
+ "valid": False,
+ "failure": "provider probe failed",
+ "returncode": completed.returncode,
+ }
+ try:
+ result = json.loads(completed.stdout)
+ except json.JSONDecodeError:
+ return {**base, "valid": False, "failure": "invalid provider probe"}
+
+ expected_origin = (source / "sam3" / "__init__.py").resolve()
+ origin = result.get("sam3_origin")
+ valid = (
+ result.get("version") == EXPECTED_TRANSFORMERS_VERSION
+ and origin == str(expected_origin)
+ )
+ return {
+ **base,
+ "valid": valid,
+ "version": result.get("version"),
+ "sam3_origin": origin,
+ "expected_sam3_origin": str(expected_origin),
+ "failures": [] if valid else ["version_or_import_origin"],
+ }
diff --git a/vlm_agent/reconstruction/sam3d_runner.py b/vlm_agent/reconstruction/sam3d_runner.py
new file mode 100644
index 0000000..02896ac
--- /dev/null
+++ b/vlm_agent/reconstruction/sam3d_runner.py
@@ -0,0 +1,830 @@
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib
+import json
+import os
+import re
+import sys
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable
+
+import yaml
+
+REPOSITORY_ROOT = Path(__file__).resolve().parents[2]
+if str(REPOSITORY_ROOT) not in sys.path:
+ sys.path.insert(0, str(REPOSITORY_ROOT))
+
+from vlm_agent.run_paths import ScenePathError, ScenePaths
+
+READY_STATUSES = frozenset({"sam3d_ready", "sam3d_ready_degraded"})
+SEED = 42
+SECRET_NAME_RE = re.compile(r"(?:KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL)", re.I)
+
+
+class Sam3DRunnerError(RuntimeError):
+ """Raised when the repository-local SAM3D runner contract is invalid."""
+
+
+@dataclass(frozen=True)
+class Sam3DInputRecord:
+ order_index: int
+ object_id: str
+ sam3d_internal_name: str
+ category: str | None
+ description: str | None
+ source: Any
+ mask_path: Path
+ mask_sha256: str
+ bbox_2d_norm_xyxy: Any
+ mask_bbox_abs_xyxy: Any
+ sam3: dict[str, Any]
+
+
+@dataclass(frozen=True)
+class Sam3DRunnerJob:
+ scene_name: str
+ run_root: Path
+ scene_config_path: Path
+ stage04_dir: Path
+ input_image: Path
+ source_manifest: Path
+ source_manifest_sha256: str
+ output_dir: Path
+ published_output_dir: Path
+ project_root: Path
+ sam3d_source_root: Path
+ inference_path: Path
+ checkpoint_path: Path
+ checkpoint_assets: tuple[Path, ...]
+ dino_hub_dir: Path
+ dino_repository: Path
+ dino_checkpoints: tuple[Path, ...]
+ records: tuple[Sam3DInputRecord, ...]
+
+
+def _utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _read_json(path: Path, label: str) -> dict[str, Any]:
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise Sam3DRunnerError(f"cannot parse {label}: {path}") from exc
+ if not isinstance(payload, dict):
+ raise Sam3DRunnerError(f"{label} must contain a JSON object: {path}")
+ return payload
+
+
+def _write_json(path: Path, payload: Any) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.parent / f".{path.name}.tmp-{os.getpid()}"
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _redact_secrets(message: str) -> str:
+ result = message
+ for name, value in os.environ.items():
+ if SECRET_NAME_RE.search(name) and len(value) >= 4:
+ result = result.replace(value, "<redacted>")
+ return result
+
+
+def _is_within(path: Path, root: Path) -> bool:
+ try:
+ path.relative_to(root)
+ return True
+ except ValueError:
+ return False
+
+
+def _real_nonempty_file(path: Path, label: str) -> None:
+ if not path.is_file() or path.is_symlink() or path.stat().st_size == 0:
+ raise Sam3DRunnerError(f"{label} must be a real non-empty file: {path}")
+
+
+CHECKPOINT_ASSET_SUFFIXES = frozenset(
+ {".yaml", ".yml", ".ckpt", ".pt", ".pth", ".safetensors"}
+)
+
+
+def _checkpoint_asset_paths(config_path: Path) -> tuple[Path, ...]:
+ """Resolve every local model/config file reachable from a pipeline YAML."""
+
+ root = Path(config_path).expanduser().resolve()
+ pending = [root]
+ assets: set[Path] = set()
+ parsed_yaml: set[Path] = set()
+
+ def visit(value: Any, *, relative_to: Path) -> None:
+ if isinstance(value, dict):
+ for item in value.values():
+ visit(item, relative_to=relative_to)
+ return
+ if isinstance(value, list):
+ for item in value:
+ visit(item, relative_to=relative_to)
+ return
+ if not isinstance(value, str):
+ return
+ candidate = Path(value).expanduser()
+ if candidate.suffix.lower() not in CHECKPOINT_ASSET_SUFFIXES:
+ return
+ if not candidate.is_absolute():
+ candidate = relative_to / candidate
+ candidate = candidate.resolve()
+ _real_nonempty_file(candidate, "SAM3D checkpoint dependency")
+ assets.add(candidate)
+ if (
+ candidate.suffix.lower() in {".yaml", ".yml"}
+ and candidate not in parsed_yaml
+ ):
+ pending.append(candidate)
+
+ while pending:
+ yaml_path = pending.pop()
+ if yaml_path in parsed_yaml:
+ continue
+ _real_nonempty_file(yaml_path, "SAM3D checkpoint YAML")
+ parsed_yaml.add(yaml_path)
+ assets.add(yaml_path)
+ try:
+ payload = yaml.safe_load(yaml_path.read_text(encoding="utf-8"))
+ except (OSError, yaml.YAMLError) as exc:
+ raise Sam3DRunnerError(
+ f"cannot parse SAM3D checkpoint YAML: {yaml_path}"
+ ) from exc
+ visit(payload, relative_to=yaml_path.parent)
+ return tuple(sorted(assets))
+
+
+def _asset_records(paths: tuple[Path, ...]) -> list[dict[str, Any]]:
+ return [
+ {
+ "path": str(path),
+ "size_bytes": path.stat().st_size,
+ "sha256": _sha256(path),
+ }
+ for path in paths
+ ]
+
+
+def _resolve_dino_cache(raw_path: Path) -> tuple[Path, Path, tuple[Path, ...]]:
+ candidate = Path(raw_path).expanduser().resolve()
+ if not candidate.is_dir() or candidate.is_symlink():
+ raise Sam3DRunnerError(
+ f"DINO hub cache must be a real directory: {candidate}"
+ )
+
+ if (candidate / "hubconf.py").is_file():
+ repository = candidate
+ hub_dir = candidate.parent
+ else:
+ repository = candidate / "facebookresearch_dinov2_main"
+ hub_dir = candidate
+ _real_nonempty_file(repository / "hubconf.py", "cached DINO hubconf")
+
+ checkpoint_dirs = (hub_dir / "checkpoints", hub_dir / "hub" / "checkpoints")
+ checkpoints = tuple(
+ sorted(
+ path.resolve()
+ for directory in checkpoint_dirs
+ if directory.is_dir()
+ for path in directory.glob("*.pth")
+ if path.is_file() and not path.is_symlink() and path.stat().st_size > 0
+ )
+ )
+ if not checkpoints:
+ raise Sam3DRunnerError(
+ f"DINO hub cache contains no local .pth checkpoint: {hub_dir}"
+ )
+ return hub_dir, repository.resolve(), checkpoints
+
+
+def _load_scene_contract(scene_config_path: Path) -> tuple[str, Path, Path, Path]:
+ config_path = Path(scene_config_path).expanduser().resolve()
+ _real_nonempty_file(config_path, "scene_config")
+ config = _read_json(config_path, "scene_config")
+ scene_name = str(config.get("scene_name") or "").strip()
+ if not scene_name:
+ raise Sam3DRunnerError("scene_config.scene_name must not be empty")
+
+ raw_run_root = config.get("run_root")
+ if not isinstance(raw_run_root, str) or not raw_run_root.strip():
+ raise Sam3DRunnerError("scene_config.run_root must be a non-empty path")
+ run_root = Path(raw_run_root).expanduser().resolve()
+ expected_config = run_root / "input" / "scene_config.json"
+ if config_path != expected_config:
+ raise Sam3DRunnerError(
+ f"scene_config must be the formal run config: {expected_config}"
+ )
+
+ try:
+ paths = ScenePaths.from_scene_config(config_path)
+ except ScenePathError as exc:
+ raise Sam3DRunnerError(str(exc)) from exc
+ stage04_dir = paths.sam3d_input_dir
+ input_image = stage04_dir / "image.png"
+ manifest_path = stage04_dir / "sam3d_input_manifest.json"
+ _real_nonempty_file(input_image, "Stage 04 image")
+ _real_nonempty_file(manifest_path, "Stage 04 manifest")
+ return scene_name, run_root, input_image, manifest_path
+
+
+def _load_records(
+ manifest_path: Path,
+ *,
+ scene_name: str,
+ input_image: Path,
+) -> tuple[Sam3DInputRecord, ...]:
+ manifest = _read_json(manifest_path, "Stage 04 manifest")
+ if manifest.get("delivery_status") not in READY_STATUSES:
+ raise Sam3DRunnerError(
+ "Stage 04 delivery status is not sam3d_ready or sam3d_ready_degraded"
+ )
+ if manifest.get("sam3d_execution_allowed") is not True:
+ raise Sam3DRunnerError("Stage 04 does not allow SAM3D execution")
+ # Adoption preserves the protected delivery's source scene name. A formal
+ # SceneMosaic run may intentionally use a different run/scene identifier.
+ if not str(manifest.get("scene_name") or "").strip():
+ raise Sam3DRunnerError("Stage 04 manifest.scene_name must not be empty")
+
+ raw_image = manifest.get("image_path")
+ if not isinstance(raw_image, str) or Path(raw_image).expanduser().resolve() != input_image:
+ raise Sam3DRunnerError("Stage 04 manifest image_path is not the formal image")
+
+ objects = manifest.get("objects")
+ object_count = manifest.get("object_count")
+ if (
+ not isinstance(objects, list)
+ or not objects
+ or not isinstance(object_count, int)
+ or object_count != len(objects)
+ ):
+ raise Sam3DRunnerError("Stage 04 object_count must match a non-empty objects list")
+
+ masks_root = manifest_path.parent / "masks"
+ records: list[Sam3DInputRecord] = []
+ seen_masks: set[Path] = set()
+ for index, item in enumerate(objects):
+ if not isinstance(item, dict):
+ raise Sam3DRunnerError(f"Stage 04 object {index} is not a JSON object")
+ expected_id = f"object_{index + 1:03d}"
+ object_id = str(item.get("object_id") or "")
+ internal_name = str(item.get("sam3d_internal_name") or "")
+ if object_id != expected_id or internal_name != expected_id:
+ raise Sam3DRunnerError(
+ "Stage 04 IDs must be continuous object_001 through object_N and "
+ "sam3d_internal_name must equal object_id"
+ )
+ if item.get("order_index") != index:
+ raise Sam3DRunnerError(f"Stage 04 order_index mismatch for {object_id}")
+
+ raw_mask = item.get("mask_path")
+ if not isinstance(raw_mask, str) or not raw_mask.strip():
+ raise Sam3DRunnerError(f"Stage 04 mask_path is missing for {object_id}")
+ mask_path = Path(raw_mask).expanduser().resolve()
+ if not _is_within(mask_path, masks_root.resolve()) or mask_path.parent != masks_root.resolve():
+ raise Sam3DRunnerError(
+ f"Stage 04 mask must be a direct child of the formal masks directory: {mask_path}"
+ )
+ _real_nonempty_file(mask_path, f"Stage 04 mask for {object_id}")
+ if mask_path in seen_masks:
+ raise Sam3DRunnerError(f"duplicate Stage 04 mask: {mask_path}")
+ seen_masks.add(mask_path)
+ mask_sha256 = str(item.get("mask_sha256") or "")
+ if len(mask_sha256) != 64 or _sha256(mask_path) != mask_sha256:
+ raise Sam3DRunnerError(f"Stage 04 mask hash mismatch for {object_id}")
+
+ records.append(
+ Sam3DInputRecord(
+ order_index=index,
+ object_id=object_id,
+ sam3d_internal_name=internal_name,
+ category=(
+ str(item["category"])
+ if item.get("category") is not None
+ else None
+ ),
+ description=(
+ str(item["description"])
+ if item.get("description") is not None
+ else None
+ ),
+ source=item.get("source"),
+ mask_path=mask_path,
+ mask_sha256=mask_sha256,
+ bbox_2d_norm_xyxy=item.get("bbox_2d_norm_xyxy"),
+ mask_bbox_abs_xyxy=item.get("mask_bbox_abs_xyxy"),
+ sam3=item.get("sam3") if isinstance(item.get("sam3"), dict) else {},
+ )
+ )
+ return tuple(records)
+
+
+def load_job(
+ *,
+ scene_config_path: Path,
+ output_dir: Path,
+ published_output_dir: Path,
+ project_root: Path,
+ checkpoint_path: Path,
+ dino_hub_cache: Path,
+) -> Sam3DRunnerJob:
+ project = Path(project_root).expanduser().resolve()
+ expected_project = Path(__file__).resolve().parents[2]
+ if project != expected_project:
+ raise Sam3DRunnerError(
+ "SAM3D project_root must be the SceneMosaic repository containing this runner: "
+ f"{expected_project}"
+ )
+
+ source_root = project / "sam-3d-objects"
+ inference_path = source_root / "notebook" / "inference.py"
+ if not source_root.is_dir() or source_root.is_symlink():
+ raise Sam3DRunnerError(
+ f"repository-local sam-3d-objects is missing: {source_root}"
+ )
+ _real_nonempty_file(inference_path, "repository-local SAM3D inference module")
+
+ checkpoint = Path(checkpoint_path).expanduser().resolve()
+ _real_nonempty_file(checkpoint, "SAM3D checkpoint configuration")
+ if checkpoint.suffix.lower() not in {".yaml", ".yml"}:
+ raise Sam3DRunnerError("SAM3D checkpoint configuration must be YAML")
+ checkpoint_assets = _checkpoint_asset_paths(checkpoint)
+ dino_hub_dir, dino_repository, dino_checkpoints = _resolve_dino_cache(
+ Path(dino_hub_cache)
+ )
+
+ scene_name, run_root, input_image, manifest_path = _load_scene_contract(
+ Path(scene_config_path)
+ )
+ records = _load_records(
+ manifest_path,
+ scene_name=scene_name,
+ input_image=input_image,
+ )
+
+ output = Path(output_dir).expanduser().resolve()
+ published = Path(published_output_dir).expanduser().resolve()
+ expected_published = ScenePaths.from_scene_config(
+ Path(scene_config_path)
+ ).sam3d_output_dir
+ if published != expected_published:
+ raise Sam3DRunnerError(
+ f"published output must be the formal Stage 05 directory: {expected_published}"
+ )
+ if not _is_within(output, run_root) or output == published:
+ raise Sam3DRunnerError(
+ "runner output must be an isolated attempt directory inside the formal run root"
+ )
+
+ return Sam3DRunnerJob(
+ scene_name=scene_name,
+ run_root=run_root,
+ scene_config_path=Path(scene_config_path).expanduser().resolve(),
+ stage04_dir=manifest_path.parent,
+ input_image=input_image,
+ source_manifest=manifest_path,
+ source_manifest_sha256=_sha256(manifest_path),
+ output_dir=output,
+ published_output_dir=published,
+ project_root=project,
+ sam3d_source_root=source_root,
+ inference_path=inference_path,
+ checkpoint_path=checkpoint,
+ dino_hub_dir=dino_hub_dir,
+ checkpoint_assets=checkpoint_assets,
+ dino_repository=dino_repository,
+ dino_checkpoints=dino_checkpoints,
+ records=records,
+ )
+
+
+def dry_run_payload(job: Sam3DRunnerJob) -> dict[str, Any]:
+ return {
+ "status": "dry_run_ok",
+ "scene_name": job.scene_name,
+ "scene_config": str(job.scene_config_path),
+ "input_image": str(job.input_image),
+ "source_manifest": str(job.source_manifest),
+ "source_manifest_sha256": job.source_manifest_sha256,
+ "attempt_output_dir": str(job.output_dir),
+ "published_output_dir": str(job.published_output_dir),
+ "project_root": str(job.project_root),
+ "sam3d_source_root": str(job.sam3d_source_root),
+ "checkpoint_path": str(job.checkpoint_path),
+ "checkpoint_sha256": _sha256(job.checkpoint_path),
+ "checkpoint_assets": _asset_records(job.checkpoint_assets),
+ "dino_hub_dir": str(job.dino_hub_dir),
+ "dino_repository": str(job.dino_repository),
+ "dino_checkpoint_count": len(job.dino_checkpoints),
+ "dino_checkpoints": _asset_records(job.dino_checkpoints),
+ "object_count": len(job.records),
+ "object_ids": [record.object_id for record in job.records],
+ "seed": SEED,
+ "appearance_mode": "vertex_color",
+ "with_texture_baking": False,
+ "gpu_started": False,
+ "api_calls": 0,
+ }
+
+
+def _ensure_empty_attempt(path: Path) -> None:
+ if path.is_symlink():
+ raise Sam3DRunnerError(f"attempt output cannot be a symlink: {path}")
+ if path.exists():
+ if not path.is_dir():
+ raise Sam3DRunnerError(f"attempt output is not a directory: {path}")
+ if any(path.iterdir()):
+ raise Sam3DRunnerError(f"attempt output is not empty: {path}")
+ else:
+ path.mkdir(parents=True, exist_ok=False)
+
+
+def _install_offline_torch_hub(torch_module: Any, job: Sam3DRunnerJob) -> None:
+ torch_module.hub.set_dir(str(job.dino_hub_dir))
+ original_load: Callable[..., Any] = torch_module.hub.load
+
+ def offline_load(
+ repo_or_dir: str,
+ model: str,
+ *args: Any,
+ source: str = "github",
+ **kwargs: Any,
+ ) -> Any:
+ if str(repo_or_dir) != "facebookresearch/dinov2":
+ raise Sam3DRunnerError(
+ f"network-backed torch.hub repository is disabled: {repo_or_dir}"
+ )
+ return original_load(
+ str(job.dino_repository),
+ model,
+ *args,
+ source="local",
+ **kwargs,
+ )
+
+ def block_download(url: str, *args: Any, **kwargs: Any) -> None:
+ raise Sam3DRunnerError(
+ "offline DINO checkpoint is missing; network download is disabled"
+ )
+
+ torch_module.hub.load = offline_load
+ torch_module.hub.download_url_to_file = block_download
+
+
+def _load_runtime_modules(job: Sam3DRunnerJob) -> dict[str, Any]:
+ conda_prefix = Path(sys.executable).resolve().parent.parent
+ os.environ.setdefault("CONDA_PREFIX", str(conda_prefix))
+ os.environ.setdefault("CUDA_HOME", os.environ["CONDA_PREFIX"])
+ os.environ["LIDRA_SKIP_INIT"] = "true"
+ os.environ["TORCH_HOME"] = str(job.dino_hub_dir)
+
+ source_root = str(job.sam3d_source_root)
+ notebook_root = str(job.sam3d_source_root / "notebook")
+ for entry in (source_root, notebook_root, str(job.project_root)):
+ if entry in sys.path:
+ sys.path.remove(entry)
+ sys.path.insert(0, entry)
+
+ import torch
+
+ _install_offline_torch_hub(torch, job)
+ inference_module = importlib.import_module("inference")
+ loaded_inference = Path(inference_module.__file__).resolve()
+ if loaded_inference != job.inference_path.resolve():
+ raise Sam3DRunnerError(
+ f"SAM3D inference resolved outside SceneMosaic: {loaded_inference}"
+ )
+ sam3d_package = importlib.import_module("sam3d_objects")
+ package_path = Path(sam3d_package.__file__).resolve()
+ if not _is_within(package_path, job.sam3d_source_root):
+ raise Sam3DRunnerError(
+ f"sam3d_objects package resolved outside SceneMosaic: {package_path}"
+ )
+
+ utilities = importlib.import_module("vlm_agent.evolution.sam3d_utils")
+ return {
+ "torch": torch,
+ "Inference": inference_module.Inference,
+ "load_image": inference_module.load_image,
+ "load_mask": inference_module.load_mask,
+ "make_scene": utilities.make_scene_untextured_mesh,
+ "normalize_scene": utilities.normalize_scene_to_canonical_cube,
+ }
+
+
+def _mesh_has_vertex_colors(mesh: Any) -> bool:
+ visual = getattr(mesh, "visual", None)
+ colors = getattr(visual, "vertex_colors", None)
+ return (
+ getattr(visual, "kind", None) == "vertex"
+ and colors is not None
+ and len(colors) == len(mesh.vertices)
+ and len(colors) > 0
+ )
+
+
+def _relative_symlink(target: Path, link_path: Path) -> None:
+ relative = os.path.relpath(target, link_path.parent)
+ link_path.symlink_to(relative)
+
+
+def _relative_from_output(job: Sam3DRunnerJob, path: Path) -> str:
+ return os.path.relpath(path, job.output_dir)
+
+
+def _export_outputs(
+ job: Sam3DRunnerJob,
+ outputs: list[dict[str, Any]],
+ modules: dict[str, Any],
+) -> None:
+ import trimesh
+
+ if len(outputs) != len(job.records):
+ raise Sam3DRunnerError(
+ f"SAM3D returned {len(outputs)} outputs for {len(job.records)} objects"
+ )
+ missing_meshes = [
+ job.records[index].object_id
+ for index, output in enumerate(outputs)
+ if not isinstance(output, dict) or output.get("glb") is None
+ ]
+ if missing_meshes:
+ raise Sam3DRunnerError(
+ "SAM3D returned no mesh for: " + ", ".join(missing_meshes)
+ )
+
+ scene_mesh, posed_meshes, unposed_meshes, layout_params = modules[
+ "make_scene"
+ ](*outputs, return_all=True)
+ scene_mesh, posed_meshes, layout_params = modules["normalize_scene"](
+ scene_mesh,
+ posed_meshes,
+ layout_params,
+ cube_size=1.0,
+ in_place=True,
+ )
+ expected_count = len(job.records)
+ actual_counts = {
+ "posed": len(posed_meshes),
+ "unposed": len(unposed_meshes),
+ "layout": len(layout_params),
+ }
+ if scene_mesh is None or any(value != expected_count for value in actual_counts.values()):
+ raise Sam3DRunnerError(
+ f"SAM3D output closure failed: expected {expected_count}, got {actual_counts}"
+ )
+ missing_colors = [
+ record.object_id
+ for record, posed, unposed in zip(job.records, posed_meshes, unposed_meshes)
+ if not _mesh_has_vertex_colors(posed) or not _mesh_has_vertex_colors(unposed)
+ ]
+ if missing_colors:
+ raise Sam3DRunnerError(
+ "vertex color verification failed for: " + ", ".join(missing_colors)
+ )
+
+ posed_dir = job.output_dir / "posed_objects"
+ unposed_dir = job.output_dir / "unposed_objects"
+ layout_dir = job.output_dir / "layout_params"
+ objects_dir = job.output_dir / "objects"
+ for directory in (posed_dir, unposed_dir, layout_dir, objects_dir):
+ directory.mkdir(parents=True, exist_ok=False)
+
+ index_entries: list[dict[str, Any]] = []
+ for index, record in enumerate(job.records):
+ posed_path = posed_dir / f"{record.object_id}.glb"
+ unposed_path = unposed_dir / f"{record.object_id}.glb"
+ layout_path = layout_dir / f"{record.object_id}.json"
+ object_dir = objects_dir / record.object_id
+ object_dir.mkdir()
+
+ posed_meshes[index].export(posed_path)
+ unposed_meshes[index].export(unposed_path)
+ layout = dict(layout_params[index])
+ layout.update(
+ {
+ "sam3d_index": record.order_index,
+ "sam3d_internal_name": record.sam3d_internal_name,
+ "object_id": record.object_id,
+ "category": record.category,
+ "description": record.description,
+ "source": record.source,
+ "source_mask_path": str(record.mask_path),
+ "source_mask_sha256": record.mask_sha256,
+ "vlm_bbox_2d_norm_xyxy": record.bbox_2d_norm_xyxy,
+ "sam3_mask_bbox_abs_xyxy": record.mask_bbox_abs_xyxy,
+ "posed_glb": _relative_from_output(job, posed_path),
+ "unposed_glb": _relative_from_output(job, unposed_path),
+ "layout_json": _relative_from_output(job, layout_path),
+ }
+ )
+ _write_json(layout_path, layout)
+
+ local_mesh = object_dir / "local_mesh.glb"
+ camera_mesh = object_dir / "camera_mesh.glb"
+ full_json = object_dir / f"{record.object_id}_full.json"
+ _relative_symlink(unposed_path, local_mesh)
+ _relative_symlink(posed_path, camera_mesh)
+ _relative_symlink(layout_path, full_json)
+
+ final_object_dir = job.published_output_dir / "objects" / record.object_id
+ index_entries.append(
+ {
+ "order_index": record.order_index,
+ "sam3d_index": record.order_index,
+ "sam3d_internal_name": record.sam3d_internal_name,
+ "object_id": record.object_id,
+ "category": record.category,
+ "description": record.description,
+ "mask_path": str(record.mask_path),
+ "mask_sha256": record.mask_sha256,
+ "output_dir": str(final_object_dir),
+ "local_mesh_path": str(final_object_dir / "local_mesh.glb"),
+ "camera_mesh_path": str(final_object_dir / "camera_mesh.glb"),
+ "full_json_path": str(final_object_dir / f"{record.object_id}_full.json"),
+ "posed_glb_sha256": _sha256(posed_path),
+ "unposed_glb_sha256": _sha256(unposed_path),
+ "layout_sha256": _sha256(layout_path),
+ }
+ )
+
+ export_scene = trimesh.Scene()
+ for record, mesh in zip(job.records, posed_meshes):
+ export_scene.add_geometry(
+ mesh,
+ geom_name=record.object_id,
+ node_name=record.object_id,
+ )
+ scene_path = job.output_dir / "scene.glb"
+ export_scene.export(scene_path)
+
+ index_path = job.output_dir / "object_index_map.json"
+ index_payload = {
+ "schema_version": "scenemosaic_sam3d_object_index",
+ "scene_name": job.scene_name,
+ "run_root": str(job.run_root),
+ "stage": "sam3d_output",
+ "source_manifest": str(job.source_manifest),
+ "source_manifest_sha256": job.source_manifest_sha256,
+ "output_dir": str(job.published_output_dir),
+ "seed": SEED,
+ "with_texture_baking": False,
+ "texture_verified": False,
+ "vertex_color_verified": True,
+ "appearance_mode": "vertex_color",
+ "scene_glb_sha256": _sha256(scene_path),
+ "sam3d_source": {
+ "root": str(job.sam3d_source_root),
+ "inference_path": str(job.inference_path),
+ },
+ "checkpoint": {
+ "assets": _asset_records(job.checkpoint_assets),
+ "path": str(job.checkpoint_path),
+ "sha256": _sha256(job.checkpoint_path),
+ },
+ "dino": {
+ "hub_dir": str(job.dino_hub_dir),
+ "checkpoints": _asset_records(job.dino_checkpoints),
+ "repository": str(job.dino_repository),
+ "offline": True,
+ },
+ "objects": index_entries,
+ }
+ _write_json(index_path, index_payload)
+ _write_json(
+ job.output_dir / "sam3d_run_info.json",
+ {
+ "schema_version": "scenemosaic_sam3d_run_info",
+ "stage": "sam3d_output",
+ "scene_name": job.scene_name,
+ "run_root": str(job.run_root),
+ "input_image": str(job.input_image),
+ "source_manifest": str(job.source_manifest),
+ "source_manifest_sha256": job.source_manifest_sha256,
+ "output_dir": str(job.published_output_dir),
+ "object_index_map": str(job.published_output_dir / index_path.name),
+ "object_index_map_sha256": _sha256(index_path),
+ "object_count": len(job.records),
+ "created_at_utc": _utc_now(),
+ "status": "success",
+ "seed": SEED,
+ "with_texture_baking": False,
+ "texture_verified": False,
+ "vertex_color_verified": True,
+ "appearance_mode": "vertex_color",
+ "api_calls": 0,
+ "source_is_repository_local": True,
+ "dino_offline": True,
+ },
+ )
+
+
+def execute(job: Sam3DRunnerJob) -> dict[str, Any]:
+ _ensure_empty_attempt(job.output_dir)
+ modules = _load_runtime_modules(job)
+ inference = modules["Inference"](str(job.checkpoint_path), compile=False)
+ image = modules["load_image"](str(job.input_image))
+ outputs: list[dict[str, Any]] = []
+ for record in job.records:
+ print(
+ f"[{record.order_index:03d}] reconstruct "
+ f"{record.sam3d_internal_name} | {record.category}",
+ flush=True,
+ )
+ mask = modules["load_mask"](str(record.mask_path))
+ outputs.append(inference(image, mask, seed=SEED))
+ _export_outputs(job, outputs, modules)
+ return {
+ "status": "success",
+ "scene_name": job.scene_name,
+ "object_count": len(job.records),
+ "object_ids": [record.object_id for record in job.records],
+ "output_dir": str(job.output_dir),
+ "published_output_dir": str(job.published_output_dir),
+ "seed": SEED,
+ "appearance_mode": "vertex_color",
+ "api_calls": 0,
+ }
+
+
+def _failed_run_info(job: Sam3DRunnerJob, exc: Exception) -> dict[str, Any]:
+ return {
+ "schema_version": "scenemosaic_sam3d_run_info",
+ "stage": "sam3d_output",
+ "scene_name": job.scene_name,
+ "run_root": str(job.run_root),
+ "source_manifest": str(job.source_manifest),
+ "source_manifest_sha256": job.source_manifest_sha256,
+ "output_dir": str(job.published_output_dir),
+ "object_count": len(job.records),
+ "created_at_utc": _utc_now(),
+ "status": "failed",
+ "seed": SEED,
+ "appearance_mode": "vertex_color",
+ "with_texture_baking": False,
+ "api_calls": 0,
+ "error_type": type(exc).__name__,
+ "error_message": _redact_secrets(str(exc)),
+ }
+
+
+def build_arg_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(
+ description="Run repository-local sam-3d-objects for formal SceneMosaic Stage 05."
+ )
+ parser.add_argument("--scene-config", required=True)
+ parser.add_argument("--output-dir", required=True)
+ parser.add_argument("--published-output-dir", required=True)
+ parser.add_argument("--project-root", required=True)
+ parser.add_argument("--checkpoint-path", required=True)
+ parser.add_argument("--dino-hub-cache", required=True)
+ parser.add_argument("--dry-run", action="store_true")
+ return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = build_arg_parser().parse_args(argv)
+ job: Sam3DRunnerJob | None = None
+ try:
+ job = load_job(
+ scene_config_path=Path(args.scene_config),
+ output_dir=Path(args.output_dir),
+ published_output_dir=Path(args.published_output_dir),
+ project_root=Path(args.project_root),
+ checkpoint_path=Path(args.checkpoint_path),
+ dino_hub_cache=Path(args.dino_hub_cache),
+ )
+ result = dry_run_payload(job) if args.dry_run else execute(job)
+ print(json.dumps(result, ensure_ascii=False, sort_keys=True), flush=True)
+ return 0
+ except Exception as exc:
+ if job is not None and not args.dry_run:
+ try:
+ _write_json(job.output_dir / "sam3d_run_info.json", _failed_run_info(job, exc))
+ except Exception:
+ pass
+ print(f"ERROR: {_redact_secrets(str(exc))}", file=sys.stderr, flush=True)
+ return 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/vlm_agent/reconstruction/sam3d_stage.py b/vlm_agent/reconstruction/sam3d_stage.py
new file mode 100644
index 0000000..5d993d7
--- /dev/null
+++ b/vlm_agent/reconstruction/sam3d_stage.py
@@ -0,0 +1,1234 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import math
+import os
+import re
+import struct
+import subprocess
+import uuid
+from dataclasses import dataclass, replace
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Callable, Mapping
+
+import numpy as np
+import trimesh
+
+from ..config import RuntimeConfig
+from ..run_paths import ScenePaths
+from .sam3d_runner import (
+ SEED,
+ Sam3DInputRecord,
+ Sam3DRunnerError,
+ _checkpoint_asset_paths,
+ _load_records,
+ _load_scene_contract,
+ _resolve_dino_cache,
+ dry_run_payload,
+ load_job,
+)
+
+ATTEMPT_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}$")
+SECRET_NAME_RE = re.compile(r"(?:KEY|TOKEN|SECRET|PASSWORD|CREDENTIAL)", re.I)
+
+
+class Sam3DStageError(RuntimeError):
+ """Raised when Stage 05 cannot be safely run, reused, or published."""
+
+
+@dataclass(frozen=True)
+class Sam3DStageArtifacts:
+ stage05_dir: Path
+ object_index_map_path: Path
+ run_info_path: Path
+ scene_glb_path: Path
+ layout_params_dir: Path
+ posed_objects_dir: Path
+ unposed_objects_dir: Path
+ objects_dir: Path
+ object_ids: tuple[str, ...]
+ source_manifest_sha256: str
+ artifact_hashes: dict[str, str]
+ attempt_id: str | None = None
+ audit_path: Path | None = None
+ reused: bool = False
+
+
+def _utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def _read_json(path: Path, label: str) -> dict[str, Any]:
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise Sam3DStageError(f"cannot parse {label}: {path}") from exc
+ if not isinstance(payload, dict):
+ raise Sam3DStageError(f"{label} must contain a JSON object: {path}")
+ return payload
+
+
+def _write_json(path: Path, payload: Any) -> None:
+ temporary = path.parent / f".{path.name}.tmp-{uuid.uuid4().hex}"
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ os.replace(temporary, path)
+
+
+def _write_text(path: Path, content: str) -> None:
+ temporary = path.parent / f".{path.name}.tmp-{uuid.uuid4().hex}"
+ temporary.write_text(content, encoding="utf-8")
+ os.replace(temporary, path)
+
+
+def _is_within(path: Path, root: Path) -> bool:
+ try:
+ path.relative_to(root)
+ return True
+ except ValueError:
+ return False
+
+
+def _real_nonempty_file(path: Path, label: str) -> None:
+ if not path.is_file() or path.is_symlink() or path.stat().st_size == 0:
+ raise Sam3DStageError(f"{label} must be a real non-empty file: {path}")
+
+
+def _flatten_numbers(value: Any, *, label: str) -> list[float]:
+ flattened: list[float] = []
+
+ def visit(item: Any) -> None:
+ if isinstance(item, list):
+ if not item:
+ raise Sam3DStageError(f"{label} contains an empty list")
+ for child in item:
+ visit(child)
+ return
+ if isinstance(item, bool) or not isinstance(item, (int, float)):
+ raise Sam3DStageError(f"{label} must contain only finite numbers")
+ number = float(item)
+ if not math.isfinite(number):
+ raise Sam3DStageError(f"{label} must contain only finite numbers")
+ flattened.append(number)
+
+ if not isinstance(value, list) or not value:
+ raise Sam3DStageError(f"{label} must be a non-empty list")
+ visit(value)
+ return flattened
+
+
+def _validate_glb(path: Path, label: str) -> str:
+ _real_nonempty_file(path, label)
+ if path.stat().st_size < 12:
+ raise Sam3DStageError(f"{label} is shorter than a GLB header: {path}")
+ with path.open("rb") as handle:
+ header = handle.read(12)
+ magic, version, declared_length = struct.unpack("<4sII", header)
+ if magic != b"glTF" or version != 2 or declared_length != path.stat().st_size:
+ raise Sam3DStageError(f"{label} has an invalid GLB 2.0 header: {path}")
+ return _sha256(path)
+
+
+def _directory_files(directory: Path, suffix: str, label: str) -> dict[str, Path]:
+ if not directory.is_dir() or directory.is_symlink():
+ raise Sam3DStageError(f"{label} must be a real directory: {directory}")
+ result: dict[str, Path] = {}
+ for item in directory.iterdir():
+ if item.is_symlink() or not item.is_file() or item.suffix != suffix:
+ raise Sam3DStageError(f"unexpected entry in {label}: {item}")
+ if item.stem in result:
+ raise Sam3DStageError(f"duplicate object artifact in {label}: {item.stem}")
+ result[item.stem] = item
+ return result
+
+
+def _validate_relative_symlink(link: Path, expected_target: Path, label: str) -> None:
+ if not link.is_symlink():
+ raise Sam3DStageError(f"{label} must be a relative symlink: {link}")
+ raw_target = Path(os.readlink(link))
+ if raw_target.is_absolute():
+ raise Sam3DStageError(f"{label} must not use an absolute symlink: {link}")
+ actual = (link.parent / raw_target).resolve()
+ if actual != expected_target.resolve() or not actual.is_file():
+ raise Sam3DStageError(f"{label} target mismatch: {link} -> {raw_target}")
+
+def _validate_asset_records(
+ records: Any,
+ expected_paths: tuple[Path, ...],
+ *,
+ label: str,
+) -> dict[str, str]:
+ if not isinstance(records, list) or len(records) != len(expected_paths):
+ raise Sam3DStageError(f"{label} asset closure mismatch")
+ hashes: dict[str, str] = {}
+ for record, expected_path in zip(records, expected_paths):
+ if not isinstance(record, dict):
+ raise Sam3DStageError(f"{label} asset record is invalid")
+ path = Path(str(record.get("path") or "")).expanduser().resolve()
+ if path != expected_path.resolve():
+ raise Sam3DStageError(f"{label} asset path mismatch: {path}")
+ _real_nonempty_file(path, f"{label} asset")
+ size = path.stat().st_size
+ digest = _sha256(path)
+ if record.get("size_bytes") != size or record.get("sha256") != digest:
+ raise Sam3DStageError(f"{label} asset hash mismatch: {path}")
+ hashes[str(path)] = digest
+ return hashes
+
+
+def _stage04_records(
+ scene_config_path: Path,
+) -> tuple[str, Path, Path, tuple[Sam3DInputRecord, ...]]:
+ try:
+ scene_name, run_root, input_image, manifest_path = _load_scene_contract(
+ Path(scene_config_path)
+ )
+ records = _load_records(
+ manifest_path,
+ scene_name=scene_name,
+ input_image=input_image,
+ )
+ except Sam3DRunnerError as exc:
+ raise Sam3DStageError(str(exc)) from exc
+ return scene_name, run_root, manifest_path, records
+
+
+def validate_stage05(
+ scene_config_path: Path,
+ stage05_dir: Path | None = None,
+ *,
+ published_output_dir: Path | None = None,
+ project_root: Path | None = None,
+) -> Sam3DStageArtifacts:
+ """Strictly validate a formal Stage 05 or an unpublished attempt.
+
+ ``published_output_dir`` is only different from ``stage05_dir`` while an
+ attempt is being checked before its atomic promotion. Paths recorded in
+ Stage 05 always point at the eventual formal publication.
+ """
+
+ scene_name, run_root, manifest_path, records = _stage04_records(
+ Path(scene_config_path)
+ )
+ paths = ScenePaths.from_scene_config(Path(scene_config_path))
+ root = Path(stage05_dir or paths.sam3d_output_dir).expanduser().resolve()
+ published = Path(
+ published_output_dir or paths.sam3d_output_dir
+ ).expanduser().resolve()
+ expected_published = paths.sam3d_output_dir
+ if published != expected_published:
+ raise Sam3DStageError(
+ f"published Stage 05 path mismatch: expected {expected_published}, got {published}"
+ )
+ if not root.is_dir() or root.is_symlink():
+ raise Sam3DStageError(f"Stage 05 must be a real directory: {root}")
+
+ index_path = root / "object_index_map.json"
+ run_info_path = root / "sam3d_run_info.json"
+ scene_path = root / "scene.glb"
+ _real_nonempty_file(index_path, "Stage 05 object index")
+ _real_nonempty_file(run_info_path, "Stage 05 run info")
+ scene_sha = _validate_glb(scene_path, "Stage 05 scene")
+
+ index = _read_json(index_path, "Stage 05 object index")
+ run_info = _read_json(run_info_path, "Stage 05 run info")
+ manifest_sha = _sha256(manifest_path)
+ expected_ids = tuple(record.object_id for record in records)
+ invariant_pairs = (
+ (index.get("stage"), "sam3d_output", "index.stage"),
+ (run_info.get("stage"), "sam3d_output", "run_info.stage"),
+ (index.get("scene_name"), scene_name, "index.scene_name"),
+ (run_info.get("scene_name"), scene_name, "run_info.scene_name"),
+ (index.get("source_manifest"), str(manifest_path), "index.source_manifest"),
+ (run_info.get("source_manifest"), str(manifest_path), "run_info.source_manifest"),
+ (index.get("source_manifest_sha256"), manifest_sha, "index source hash"),
+ (run_info.get("source_manifest_sha256"), manifest_sha, "run_info source hash"),
+ (index.get("output_dir"), str(published), "index.output_dir"),
+ (run_info.get("output_dir"), str(published), "run_info.output_dir"),
+ (index.get("seed"), SEED, "index.seed"),
+ (run_info.get("seed"), SEED, "run_info.seed"),
+ (index.get("appearance_mode"), "vertex_color", "index appearance"),
+ (run_info.get("appearance_mode"), "vertex_color", "run_info appearance"),
+ )
+ for actual, expected, label in invariant_pairs:
+ if actual != expected:
+ raise Sam3DStageError(
+ f"Stage 05 {label} mismatch: expected {expected!r}, got {actual!r}"
+ )
+ if run_info.get("status") != "success":
+ raise Sam3DStageError(f"Stage 05 is not successful: {run_info.get('status')}")
+ for payload, label in ((index, "index"), (run_info, "run_info")):
+ if (
+ payload.get("with_texture_baking") is not False
+ or payload.get("texture_verified") is not False
+ or payload.get("vertex_color_verified") is not True
+ ):
+ raise Sam3DStageError(
+ f"Stage 05 {label} does not prove vertex-color/no-texture mode"
+ )
+ if run_info.get("object_count") != len(expected_ids):
+ raise Sam3DStageError("Stage 05 run_info object_count mismatch")
+ if run_info.get("object_index_map") != str(published / index_path.name):
+ raise Sam3DStageError("Stage 05 run_info object_index_map path mismatch")
+ if run_info.get("object_index_map_sha256") != _sha256(index_path):
+ raise Sam3DStageError("Stage 05 object index hash mismatch")
+ if index.get("scene_glb_sha256") != scene_sha:
+ raise Sam3DStageError("Stage 05 scene GLB hash mismatch")
+
+ local_project = Path(project_root or Path(__file__).resolve().parents[2]).resolve()
+ source = index.get("sam3d_source")
+ if not isinstance(source, dict):
+ raise Sam3DStageError("Stage 05 has no SAM3D source provenance")
+ expected_source_root = local_project / "sam-3d-objects"
+ expected_inference = expected_source_root / "notebook" / "inference.py"
+ if (
+ source.get("root") != str(expected_source_root)
+ or source.get("inference_path") != str(expected_inference)
+ or not expected_inference.is_file()
+ ):
+ raise Sam3DStageError(
+ "Stage 05 was not produced by the current repository-local sam-3d-objects"
+ )
+ checkpoint = index.get("checkpoint")
+ if not isinstance(checkpoint, dict):
+ raise Sam3DStageError("Stage 05 has no checkpoint provenance")
+ checkpoint_path = Path(
+ str(checkpoint.get("path") or "")
+ ).expanduser().resolve()
+ _real_nonempty_file(checkpoint_path, "SAM3D checkpoint configuration")
+ if checkpoint.get("sha256") != _sha256(checkpoint_path):
+ raise Sam3DStageError("Stage 05 checkpoint hash mismatch")
+ try:
+ checkpoint_assets = _checkpoint_asset_paths(checkpoint_path)
+ except Sam3DRunnerError as exc:
+ raise Sam3DStageError(str(exc)) from exc
+ _validate_asset_records(
+ checkpoint.get("assets"),
+ checkpoint_assets,
+ label="SAM3D checkpoint",
+ )
+
+ dino = index.get("dino")
+ if not isinstance(dino, dict) or dino.get("offline") is not True:
+ raise Sam3DStageError("Stage 05 does not prove offline DINO loading")
+ raw_hub_dir = str(dino.get("hub_dir") or "")
+ if not raw_hub_dir:
+ raise Sam3DStageError("Stage 05 has no DINO hub directory")
+ try:
+ dino_hub, dino_repository, dino_checkpoints = _resolve_dino_cache(
+ Path(raw_hub_dir)
+ )
+ except Sam3DRunnerError as exc:
+ raise Sam3DStageError(str(exc)) from exc
+ if (
+ dino.get("hub_dir") != str(dino_hub)
+ or dino.get("repository") != str(dino_repository)
+ ):
+ raise Sam3DStageError("Stage 05 DINO path provenance mismatch")
+ _validate_asset_records(
+ dino.get("checkpoints"),
+ dino_checkpoints,
+ label="DINO checkpoint",
+ )
+
+ index_objects = index.get("objects")
+ if not isinstance(index_objects, list) or len(index_objects) != len(records):
+ raise Sam3DStageError("Stage 05 object index count mismatch")
+ index_ids = tuple(
+ str(item.get("object_id") or "")
+ for item in index_objects
+ if isinstance(item, dict)
+ )
+ if index_ids != expected_ids or len(index_ids) != len(index_objects):
+ raise Sam3DStageError("Stage 04 and Stage 05 object ID closure failed")
+
+ layout_dir = root / "layout_params"
+ posed_dir = root / "posed_objects"
+ unposed_dir = root / "unposed_objects"
+ objects_dir = root / "objects"
+ layout_files = _directory_files(layout_dir, ".json", "layout_params")
+ posed_files = _directory_files(posed_dir, ".glb", "posed_objects")
+ unposed_files = _directory_files(unposed_dir, ".glb", "unposed_objects")
+ for label, actual in (
+ ("layout_params", tuple(sorted(layout_files))),
+ ("posed_objects", tuple(sorted(posed_files))),
+ ("unposed_objects", tuple(sorted(unposed_files))),
+ ):
+ if actual != expected_ids:
+ raise Sam3DStageError(
+ f"Stage 05 {label} ID closure mismatch: {actual} != {expected_ids}"
+ )
+ if not objects_dir.is_dir() or objects_dir.is_symlink():
+ raise Sam3DStageError(f"Stage 05 objects must be a real directory: {objects_dir}")
+ object_entries = tuple(sorted(item.name for item in objects_dir.iterdir()))
+ if object_entries != expected_ids:
+ raise Sam3DStageError("Stage 05 object compatibility directory closure failed")
+
+ artifact_hashes: dict[str, str] = {
+ "object_index_map.json": _sha256(index_path),
+ "sam3d_run_info.json": _sha256(run_info_path),
+ "scene.glb": scene_sha,
+ }
+ for record, entry in zip(records, index_objects):
+ if not isinstance(entry, dict):
+ raise Sam3DStageError(f"Stage 05 index record is invalid for {record.object_id}")
+ object_id = record.object_id
+ identity_checks = (
+ (entry.get("order_index"), record.order_index, "order_index"),
+ (entry.get("sam3d_index"), record.order_index, "sam3d_index"),
+ (entry.get("sam3d_internal_name"), object_id, "sam3d_internal_name"),
+ (entry.get("mask_path"), str(record.mask_path), "mask_path"),
+ (entry.get("mask_sha256"), record.mask_sha256, "mask_sha256"),
+ )
+ for actual, expected, label in identity_checks:
+ if actual != expected:
+ raise Sam3DStageError(
+ f"Stage 05 {label} mismatch for {object_id}: {actual!r}"
+ )
+
+ posed_path = posed_files[object_id]
+ unposed_path = unposed_files[object_id]
+ layout_path = layout_files[object_id]
+ posed_sha = _validate_glb(posed_path, f"posed GLB for {object_id}")
+ unposed_sha = _validate_glb(unposed_path, f"unposed GLB for {object_id}")
+ layout = _read_json(layout_path, f"layout for {object_id}")
+ if (
+ layout.get("object_id") != object_id
+ or layout.get("sam3d_internal_name") != object_id
+ or layout.get("sam3d_index") != record.order_index
+ or layout.get("source_mask_path") != str(record.mask_path)
+ or layout.get("source_mask_sha256") != record.mask_sha256
+ ):
+ raise Sam3DStageError(f"Stage 05 layout identity mismatch for {object_id}")
+ expected_lengths = {"translation": 3, "rotation": 4, "scale": 3}
+ for key, length in expected_lengths.items():
+ values = _flatten_numbers(layout.get(key), label=f"{object_id}.{key}")
+ if len(values) != length:
+ raise Sam3DStageError(
+ f"{object_id}.{key} must contain exactly {length} numbers"
+ )
+ expected_relative = {
+ "posed_glb": os.path.join("posed_objects", f"{object_id}.glb"),
+ "unposed_glb": os.path.join("unposed_objects", f"{object_id}.glb"),
+ "layout_json": os.path.join("layout_params", f"{object_id}.json"),
+ }
+ for key, expected in expected_relative.items():
+ if layout.get(key) != expected:
+ raise Sam3DStageError(f"Stage 05 layout path mismatch: {object_id}.{key}")
+
+ expected_hashes = {
+ "posed_glb_sha256": posed_sha,
+ "unposed_glb_sha256": unposed_sha,
+ "layout_sha256": _sha256(layout_path),
+ }
+ for key, expected in expected_hashes.items():
+ if entry.get(key) != expected:
+ raise Sam3DStageError(f"Stage 05 {key} mismatch for {object_id}")
+
+ final_object_dir = published / "objects" / object_id
+ expected_paths = {
+ "output_dir": final_object_dir,
+ "local_mesh_path": final_object_dir / "local_mesh.glb",
+ "camera_mesh_path": final_object_dir / "camera_mesh.glb",
+ "full_json_path": final_object_dir / f"{object_id}_full.json",
+ }
+ for key, expected in expected_paths.items():
+ if entry.get(key) != str(expected):
+ raise Sam3DStageError(f"Stage 05 {key} mismatch for {object_id}")
+
+ object_dir = objects_dir / object_id
+ if not object_dir.is_dir() or object_dir.is_symlink():
+ raise Sam3DStageError(f"invalid compatibility directory for {object_id}")
+ expected_link_names = {
+ "local_mesh.glb",
+ "camera_mesh.glb",
+ f"{object_id}_full.json",
+ }
+ if {path.name for path in object_dir.iterdir()} != expected_link_names:
+ raise Sam3DStageError(f"compatibility link closure failed for {object_id}")
+ _validate_relative_symlink(
+ object_dir / "local_mesh.glb", unposed_path, f"{object_id} local mesh"
+ )
+ _validate_relative_symlink(
+ object_dir / "camera_mesh.glb", posed_path, f"{object_id} camera mesh"
+ )
+ _validate_relative_symlink(
+ object_dir / f"{object_id}_full.json",
+ layout_path,
+ f"{object_id} full layout",
+ )
+ artifact_hashes[f"posed_objects/{object_id}.glb"] = posed_sha
+ artifact_hashes[f"unposed_objects/{object_id}.glb"] = unposed_sha
+ artifact_hashes[f"layout_params/{object_id}.json"] = _sha256(layout_path)
+
+ audit_path = root / "sam3d_stage_audit.json"
+ attempt_id = None
+ if audit_path.is_file() and not audit_path.is_symlink():
+ audit = _read_json(audit_path, "SAM3D stage audit")
+ attempt_id = str(audit.get("attempt_id") or "") or None
+ artifact_hashes[audit_path.name] = _sha256(audit_path)
+
+ return Sam3DStageArtifacts(
+ stage05_dir=root,
+ object_index_map_path=index_path,
+ run_info_path=run_info_path,
+ scene_glb_path=scene_path,
+ layout_params_dir=layout_dir,
+ posed_objects_dir=posed_dir,
+ unposed_objects_dir=unposed_dir,
+ objects_dir=objects_dir,
+ object_ids=expected_ids,
+ source_manifest_sha256=manifest_sha,
+ artifact_hashes=artifact_hashes,
+ attempt_id=attempt_id,
+ audit_path=audit_path if audit_path.is_file() else None,
+ )
+
+
+def _equivalence_mesh(
+ path: Path,
+ label: str,
+) -> tuple[Any, np.ndarray, np.ndarray, np.ndarray]:
+ _validate_glb(path, label)
+ try:
+ mesh = trimesh.load(path, force="mesh", process=False)
+ except Exception as exc:
+ raise Sam3DStageError(f"cannot parse {label}: {path}") from exc
+ if not isinstance(mesh, trimesh.Trimesh):
+ raise Sam3DStageError(f"{label} must contain one mesh: {path}")
+ vertices = np.asarray(mesh.vertices, dtype=np.float64)
+ faces = np.asarray(mesh.faces, dtype=np.int64)
+ colors = np.asarray(
+ getattr(getattr(mesh, "visual", None), "vertex_colors", None)
+ )
+ if (
+ vertices.ndim != 2
+ or vertices.shape[1:] != (3,)
+ or vertices.shape[0] == 0
+ or faces.ndim != 2
+ or faces.shape[1:] != (3,)
+ or faces.shape[0] == 0
+ or not np.isfinite(vertices).all()
+ or getattr(mesh.visual, "kind", None) != "vertex"
+ or colors.ndim != 2
+ or colors.shape[0] != vertices.shape[0]
+ or colors.shape[1] not in {3, 4}
+ ):
+ raise Sam3DStageError(
+ f"{label} must contain finite triangle geometry and vertex colors"
+ )
+ return mesh, vertices, faces, colors
+
+
+def _compare_equivalence_meshes(
+ reference_path: Path,
+ candidate_path: Path,
+ *,
+ label: str,
+ rtol: float,
+ atol: float,
+) -> dict[str, Any]:
+ reference, reference_vertices, reference_faces, reference_colors = (
+ _equivalence_mesh(reference_path, f"reference {label}")
+ )
+ candidate, candidate_vertices, candidate_faces, candidate_colors = (
+ _equivalence_mesh(candidate_path, f"candidate {label}")
+ )
+ if (
+ reference_vertices.shape != candidate_vertices.shape
+ or reference_faces.shape != candidate_faces.shape
+ or not np.array_equal(reference_faces, candidate_faces)
+ ):
+ raise Sam3DStageError(f"{label} mesh topology mismatch")
+ if not np.allclose(
+ reference_vertices,
+ candidate_vertices,
+ rtol=rtol,
+ atol=atol,
+ ):
+ difference = float(
+ np.max(np.abs(reference_vertices - candidate_vertices))
+ )
+ raise Sam3DStageError(
+ f"{label} vertex mismatch; max_abs_difference={difference}"
+ )
+ if (
+ reference_colors.shape != candidate_colors.shape
+ or not np.array_equal(reference_colors, candidate_colors)
+ ):
+ raise Sam3DStageError(f"{label} vertex color mismatch")
+ reference_bounds = np.asarray(reference.bounds, dtype=np.float64)
+ candidate_bounds = np.asarray(candidate.bounds, dtype=np.float64)
+ if not np.allclose(
+ reference_bounds,
+ candidate_bounds,
+ rtol=rtol,
+ atol=atol,
+ ):
+ raise Sam3DStageError(f"{label} bounding box mismatch")
+ reference_sha256 = _sha256(reference_path)
+ candidate_sha256 = _sha256(candidate_path)
+ return {
+ "vertex_count": int(reference_vertices.shape[0]),
+ "face_count": int(reference_faces.shape[0]),
+ "vertex_colors_exact": True,
+ "bbox_close": True,
+ "max_abs_vertex_difference": float(
+ np.max(np.abs(reference_vertices - candidate_vertices))
+ ),
+ "reference_sha256": reference_sha256,
+ "candidate_sha256": candidate_sha256,
+ "glb_sha256_equal": reference_sha256 == candidate_sha256,
+ }
+
+
+def _equivalence_index_ids(stage_dir: Path, label: str) -> tuple[str, ...]:
+ index = _read_json(
+ stage_dir / "object_index_map.json",
+ f"{label} object index",
+ )
+ entries = index.get("objects")
+ if not isinstance(entries, list) or not entries:
+ raise Sam3DStageError(f"{label} object index is empty or invalid")
+ identifiers = tuple(
+ str(item.get("object_id") or "")
+ for item in entries
+ if isinstance(item, dict)
+ )
+ if (
+ len(identifiers) != len(entries)
+ or any(not identifier for identifier in identifiers)
+ or len(set(identifiers)) != len(identifiers)
+ ):
+ raise Sam3DStageError(f"{label} object index IDs are invalid")
+ return identifiers
+
+
+def compare_stage05_equivalence(
+ scene_config_path: Path,
+ *,
+ reference_dir: Path,
+ candidate_dir: Path,
+ project_root: Path | None = None,
+ rtol: float = 1e-5,
+ atol: float = 1e-6,
+) -> dict[str, Any]:
+ """Gate a repository-local attempt against an existing formal Stage 05."""
+
+ if (
+ not math.isfinite(rtol)
+ or not math.isfinite(atol)
+ or rtol < 0
+ or atol < 0
+ ):
+ raise Sam3DStageError("SAM3D equivalence tolerances must be finite")
+ _, run_root, _, records = _stage04_records(Path(scene_config_path))
+ reference = Path(reference_dir).expanduser().resolve()
+ candidate = Path(candidate_dir).expanduser().resolve()
+ paths = ScenePaths.from_scene_config(Path(scene_config_path))
+ formal = paths.sam3d_output_dir.resolve()
+ attempts = paths.sam3d_attempts_dir.resolve()
+ if reference != formal:
+ raise Sam3DStageError("equivalence reference must be formal Stage 05")
+ if candidate.parent != attempts or candidate == reference:
+ raise Sam3DStageError(
+ "equivalence candidate must be one isolated Stage 05 attempt"
+ )
+ candidate_artifacts = validate_stage05(
+ scene_config_path,
+ candidate,
+ published_output_dir=reference,
+ project_root=project_root,
+ )
+ expected_ids = tuple(record.object_id for record in records)
+ reference_ids = _equivalence_index_ids(reference, "reference")
+ candidate_ids = _equivalence_index_ids(candidate, "candidate")
+ if not (
+ expected_ids
+ == reference_ids
+ == candidate_ids
+ == candidate_artifacts.object_ids
+ ):
+ raise Sam3DStageError(
+ "SAM3D equivalence object ID/order/count closure failed"
+ )
+
+ _validate_glb(reference / "scene.glb", "reference scene GLB")
+ _validate_glb(candidate / "scene.glb", "candidate scene GLB")
+ objects: dict[str, Any] = {}
+ for object_id in expected_ids:
+ reference_layout = _read_json(
+ reference / "layout_params" / f"{object_id}.json",
+ f"reference layout {object_id}",
+ )
+ candidate_layout = _read_json(
+ candidate / "layout_params" / f"{object_id}.json",
+ f"candidate layout {object_id}",
+ )
+ layout_differences: dict[str, float] = {}
+ for key in ("translation", "rotation", "scale"):
+ reference_values = np.asarray(
+ _flatten_numbers(
+ reference_layout.get(key),
+ label=f"reference {object_id}.{key}",
+ ),
+ dtype=np.float64,
+ )
+ candidate_values = np.asarray(
+ _flatten_numbers(
+ candidate_layout.get(key),
+ label=f"candidate {object_id}.{key}",
+ ),
+ dtype=np.float64,
+ )
+ if (
+ reference_values.shape != candidate_values.shape
+ or not np.allclose(
+ reference_values,
+ candidate_values,
+ rtol=rtol,
+ atol=atol,
+ )
+ ):
+ raise Sam3DStageError(
+ f"{object_id}.{key} layout equivalence mismatch"
+ )
+ layout_differences[key] = float(
+ np.max(np.abs(reference_values - candidate_values))
+ )
+
+ mesh_records = {
+ kind: _compare_equivalence_meshes(
+ reference / f"{kind}_objects" / f"{object_id}.glb",
+ candidate / f"{kind}_objects" / f"{object_id}.glb",
+ label=f"{kind} {object_id}",
+ rtol=rtol,
+ atol=atol,
+ )
+ for kind in ("posed", "unposed")
+ }
+ objects[object_id] = {
+ "layout_close": True,
+ "layout_max_abs_difference": layout_differences,
+ "meshes": mesh_records,
+ }
+
+ reference_scene_sha256 = _sha256(reference / "scene.glb")
+ candidate_scene_sha256 = _sha256(candidate / "scene.glb")
+ return {
+ "schema_version": "scenemosaic_sam3d_equivalence",
+ "valid": True,
+ "scene_config": str(Path(scene_config_path).expanduser().resolve()),
+ "reference_dir": str(reference),
+ "candidate_dir": str(candidate),
+ "object_ids": list(expected_ids),
+ "object_count": len(expected_ids),
+ "rtol": rtol,
+ "atol": atol,
+ "topology_exact": True,
+ "vertices_close": True,
+ "vertex_colors_exact": True,
+ "bbox_close": True,
+ "objects": objects,
+ "scene_glb": {
+ "reference_sha256": reference_scene_sha256,
+ "candidate_sha256": candidate_scene_sha256,
+ "sha256_equal": reference_scene_sha256 == candidate_scene_sha256,
+ "hash_is_audit_only": True,
+ },
+ "api_calls": 0,
+ }
+
+
+def _runtime_lookup(runtime: Any, *names: str) -> Any:
+ if runtime is None:
+ return None
+ containers: list[Any] = [runtime]
+ if isinstance(runtime, Mapping):
+ nested = runtime.get("sam3d")
+ else:
+ nested = getattr(runtime, "sam3d", None)
+ if nested is not None:
+ containers.append(nested)
+ for container in containers:
+ for name in names:
+ if isinstance(container, Mapping):
+ value = container.get(name)
+ else:
+ value = getattr(container, name, None)
+ if value is not None and str(value).strip():
+ return value
+ return None
+
+
+def _snapshot_empty_directory(path: Path) -> tuple[bool, tuple[Path, ...]]:
+ if not path.exists() and not path.is_symlink():
+ return False, ()
+ if path.is_symlink() or not path.is_dir():
+ raise Sam3DStageError(f"Stage 05 target is not a real directory: {path}")
+ directories: list[Path] = []
+ for item in path.rglob("*"):
+ if item.is_symlink() or not item.is_dir():
+ raise Sam3DStageError(
+ f"Stage 05 target contains data and cannot be replaced: {item}"
+ )
+ directories.append(item.relative_to(path))
+ return True, tuple(sorted(directories, key=lambda value: (len(value.parts), str(value))))
+
+
+def _remove_empty_directory(path: Path) -> None:
+ if not path.exists():
+ return
+ directories = sorted(
+ (item for item in path.rglob("*") if item.is_dir()),
+ key=lambda item: len(item.parts),
+ reverse=True,
+ )
+ for directory in directories:
+ directory.rmdir()
+ path.rmdir()
+
+
+def _restore_empty_directory(path: Path, snapshot: tuple[bool, tuple[Path, ...]]) -> None:
+ existed, directories = snapshot
+ if not existed:
+ return
+ path.mkdir(parents=True, exist_ok=True)
+ for relative in directories:
+ (path / relative).mkdir(parents=True, exist_ok=True)
+
+
+def _redact_and_limit(text: str | None, limit: int = 2 * 1024 * 1024) -> str:
+ result = text or ""
+ for name, value in os.environ.items():
+ if SECRET_NAME_RE.search(name) and len(value) >= 4:
+ result = result.replace(value, "<redacted>")
+ if len(result) > limit:
+ result = result[:limit] + "\n<truncated>\n"
+ return result
+
+
+class Sam3DStage:
+ """Run repository-local SAM3D in an isolated attempt and publish Stage 05."""
+
+ def __init__(
+ self,
+ scene_config_path: Path,
+ runtime_config: Any | None = None,
+ *,
+ project_root: Path | None = None,
+ sam3d_python: Path | None = None,
+ checkpoint_path: Path | None = None,
+ dino_hub_cache: Path | None = None,
+ timeout_seconds: float | None = None,
+ gpu_preflight: Callable[[], Mapping[str, Any]] | None = None,
+ with_texture_baking: bool = False,
+ ) -> None:
+ self.scene_config_path = Path(scene_config_path).expanduser().resolve()
+ self.runtime_config = runtime_config
+ self.project_root = Path(
+ project_root
+ or _runtime_lookup(runtime_config, "project_root", "scenemosaic_root")
+ or Path(__file__).resolve().parents[2]
+ ).expanduser().resolve()
+ raw_python = sam3d_python or _runtime_lookup(
+ runtime_config,
+ "sam3d_python",
+ "sam3d_python_path",
+ "sam3d_interpreter",
+ "python",
+ )
+ raw_checkpoint = checkpoint_path or _runtime_lookup(
+ runtime_config,
+ "sam3d_checkpoint_path",
+ "sam3d_checkpoint",
+ "sam3d_config_path",
+ "sam3d_config",
+ "checkpoint_path",
+ "checkpoint",
+ )
+ raw_dino = dino_hub_cache or _runtime_lookup(
+ runtime_config,
+ "dino_hub_cache",
+ "sam3d_dino_hub_cache",
+ "torch_hub_cache",
+ )
+ self.sam3d_python = (
+ Path(raw_python).expanduser().resolve() if raw_python is not None else None
+ )
+ self.checkpoint_path = (
+ Path(raw_checkpoint).expanduser().resolve()
+ if raw_checkpoint is not None
+ else None
+ )
+ self.dino_hub_cache = (
+ Path(raw_dino).expanduser().resolve() if raw_dino is not None else None
+ )
+ raw_timeout = timeout_seconds
+ if raw_timeout is None:
+ raw_timeout = _runtime_lookup(
+ runtime_config, "sam3d_timeout_seconds", "timeout_seconds"
+ )
+ self.timeout_seconds = float(raw_timeout) if raw_timeout is not None else None
+ if self.timeout_seconds is not None and self.timeout_seconds <= 0:
+ raise Sam3DStageError("SAM3D timeout_seconds must be positive")
+ self.with_texture_baking = bool(with_texture_baking)
+ self._gpu_preflight = gpu_preflight or RuntimeConfig.verify_gpu
+ self._gpu_preflight_report: dict[str, Any] | None = None
+
+ _, self.run_root, _, _ = _stage04_records(self.scene_config_path)
+ paths = ScenePaths.from_scene_config(self.scene_config_path)
+ self.stage05_dir = paths.sam3d_output_dir
+ self.attempts_dir = paths.sam3d_attempts_dir
+ self.runner_path = (
+ self.project_root
+ / "vlm_agent"
+ / "reconstruction"
+ / "sam3d_runner.py"
+ )
+
+ def _require_runtime(self) -> None:
+ if self.with_texture_baking:
+ raise Sam3DStageError(
+ "texture baking is unsupported by SceneMosaic's repository-local "
+ "sam-3d-objects; Stage 05 is fixed to vertex colors"
+ )
+ missing = [
+ name
+ for name, value in (
+ ("sam3d_python", self.sam3d_python),
+ ("checkpoint_path", self.checkpoint_path),
+ ("dino_hub_cache", self.dino_hub_cache),
+ )
+ if value is None
+ ]
+ if missing:
+ raise Sam3DStageError(
+ "SAM3D runtime profile is missing: " + ", ".join(missing)
+ )
+ assert self.sam3d_python is not None
+ _real_nonempty_file(self.sam3d_python, "SAM3D Python interpreter")
+ if not os.access(self.sam3d_python, os.X_OK):
+ raise Sam3DStageError(
+ f"SAM3D Python interpreter is not executable: {self.sam3d_python}"
+ )
+ _real_nonempty_file(self.runner_path, "repository-local SAM3D runner")
+
+ def _attempt_path(self, attempt_id: str) -> Path:
+ if (
+ not ATTEMPT_ID_RE.fullmatch(attempt_id)
+ or attempt_id in {".", ".."}
+ or ".." in attempt_id
+ ):
+ raise Sam3DStageError(f"invalid SAM3D attempt ID: {attempt_id!r}")
+ path = (self.attempts_dir / attempt_id).resolve()
+ if not _is_within(path, self.attempts_dir.resolve()):
+ raise Sam3DStageError(f"SAM3D attempt escapes attempts directory: {path}")
+ return path
+
+ def _job(self, attempt_dir: Path):
+ self._require_runtime()
+ assert self.checkpoint_path is not None
+ assert self.dino_hub_cache is not None
+ try:
+ return load_job(
+ scene_config_path=self.scene_config_path,
+ output_dir=attempt_dir,
+ published_output_dir=self.stage05_dir,
+ project_root=self.project_root,
+ checkpoint_path=self.checkpoint_path,
+ dino_hub_cache=self.dino_hub_cache,
+ )
+ except Sam3DRunnerError as exc:
+ raise Sam3DStageError(str(exc)) from exc
+
+ def dry_run(self) -> dict[str, Any]:
+ """Validate all inputs and runtime paths without imports, GPU, or writes."""
+
+ existing: Sam3DStageArtifacts | None = None
+ if self.stage05_dir.exists() or self.stage05_dir.is_symlink():
+ try:
+ snapshot = _snapshot_empty_directory(self.stage05_dir)
+ except Sam3DStageError:
+ existing = validate_stage05(
+ self.scene_config_path,
+ self.stage05_dir,
+ project_root=self.project_root,
+ )
+ else:
+ # Existing directories-only placeholders are safe. They are
+ # removed with rmdir immediately before atomic publication.
+ del snapshot
+
+ hypothetical = self._attempt_path("dry-run")
+ job = self._job(hypothetical)
+ payload = dry_run_payload(job)
+ payload.update(
+ {
+ "sam3d_python": str(self.sam3d_python),
+ "runner_path": str(self.runner_path),
+ "stage05_state": "reusable" if existing is not None else "available",
+ "reused": existing is not None,
+ }
+ )
+ return payload
+
+ def _build_command(self, attempt_dir: Path) -> list[str]:
+ assert self.sam3d_python is not None
+ assert self.checkpoint_path is not None
+ assert self.dino_hub_cache is not None
+ return [
+ str(self.sam3d_python),
+ str(self.runner_path),
+ "--scene-config",
+ str(self.scene_config_path),
+ "--output-dir",
+ str(attempt_dir),
+ "--published-output-dir",
+ str(self.stage05_dir),
+ "--project-root",
+ str(self.project_root),
+ "--checkpoint-path",
+ str(self.checkpoint_path),
+ "--dino-hub-cache",
+ str(self.dino_hub_cache),
+ ]
+
+ def _write_audit(
+ self,
+ attempt_dir: Path,
+ *,
+ attempt_id: str,
+ status: str,
+ command: list[str],
+ started_at_utc: str,
+ returncode: int | None = None,
+ error_type: str | None = None,
+ error_message: str | None = None,
+ ) -> Path:
+ audit = {
+ "schema_version": "scenemosaic_sam3d_stage_audit",
+ "attempt_id": attempt_id,
+ "status": status,
+ "scene_config": str(self.scene_config_path),
+ "attempt_dir": str(attempt_dir),
+ "published_output_dir": str(self.stage05_dir),
+ "started_at_utc": started_at_utc,
+ "updated_at_utc": _utc_now(),
+ "command": command,
+ "shell": False,
+ "approve_gpu": True,
+ "gpu_started": True,
+ "gpu_preflight": self._gpu_preflight_report,
+ "seed": SEED,
+ "appearance_mode": "vertex_color",
+ "with_texture_baking": False,
+ "api_calls": 0,
+ "returncode": returncode,
+ }
+ if error_type:
+ audit["error_type"] = error_type
+ if error_message:
+ audit["error_message"] = _redact_and_limit(error_message, limit=32768)
+ path = attempt_dir / "sam3d_stage_audit.json"
+ _write_json(path, audit)
+ return path
+
+ def run(
+ self,
+ *,
+ approve_gpu: bool = False,
+ attempt_id: str | None = None,
+ ) -> Sam3DStageArtifacts:
+ """Run SAM3D once, retaining failures and atomically publishing success."""
+
+ if self.stage05_dir.exists() or self.stage05_dir.is_symlink():
+ try:
+ existing = validate_stage05(
+ self.scene_config_path,
+ self.stage05_dir,
+ project_root=self.project_root,
+ )
+ except Sam3DStageError:
+ _snapshot_empty_directory(self.stage05_dir)
+ else:
+ return replace(existing, reused=True)
+
+ self.dry_run()
+ if not approve_gpu:
+ raise Sam3DStageError(
+ "SAM3D GPU execution requires explicit approve_gpu=True for this invocation"
+ )
+ gpu_preflight = dict(self._gpu_preflight())
+ if not gpu_preflight.get("valid"):
+ raise Sam3DStageError(
+ "SAM3D GPU execution was not started; GPU preflight failed"
+ )
+ self._gpu_preflight_report = gpu_preflight
+
+ selected_id = attempt_id or (
+ datetime.now(timezone.utc).strftime("attempt_%Y%m%dT%H%M%S_")
+ + uuid.uuid4().hex[:12]
+ )
+ attempt_dir = self._attempt_path(selected_id)
+ if self.attempts_dir.exists() and (
+ self.attempts_dir.is_symlink() or not self.attempts_dir.is_dir()
+ ):
+ raise Sam3DStageError(
+ f"SAM3D attempts root must be a real directory: {self.attempts_dir}"
+ )
+ self.attempts_dir.mkdir(parents=True, exist_ok=True)
+ attempt_dir.mkdir(exist_ok=False)
+ command = self._build_command(attempt_dir)
+ started_at = _utc_now()
+ environment = {
+ name: value
+ for name, value in os.environ.items()
+ if not SECRET_NAME_RE.search(name)
+ }
+ environment["PYTHONDONTWRITEBYTECODE"] = "1"
+ environment["PYTHONNOUSERSITE"] = "1"
+
+ try:
+ kwargs: dict[str, Any] = {
+ "cwd": self.project_root,
+ "env": environment,
+ "text": True,
+ "capture_output": True,
+ "check": False,
+ }
+ if self.timeout_seconds is not None:
+ kwargs["timeout"] = self.timeout_seconds
+ completed = subprocess.run(command, **kwargs)
+ stdout = _redact_and_limit(completed.stdout)
+ stderr = _redact_and_limit(completed.stderr)
+ _write_text(attempt_dir / "runner.stdout.log", stdout)
+ _write_text(attempt_dir / "runner.stderr.log", stderr)
+ if completed.returncode != 0:
+ self._write_audit(
+ attempt_dir,
+ attempt_id=selected_id,
+ status="failed",
+ command=command,
+ started_at_utc=started_at,
+ returncode=completed.returncode,
+ error_type="Sam3DRunnerProcessError",
+ error_message=stderr or stdout or "SAM3D runner failed",
+ )
+ raise Sam3DStageError(
+ f"SAM3D runner failed with exit code {completed.returncode}; "
+ f"attempt retained at {attempt_dir}"
+ )
+
+ validate_stage05(
+ self.scene_config_path,
+ attempt_dir,
+ published_output_dir=self.stage05_dir,
+ project_root=self.project_root,
+ )
+ self._write_audit(
+ attempt_dir,
+ attempt_id=selected_id,
+ status="validated_for_publication",
+ command=command,
+ started_at_utc=started_at,
+ returncode=completed.returncode,
+ )
+ except Sam3DStageError as exc:
+ audit_path = attempt_dir / "sam3d_stage_audit.json"
+ if not audit_path.exists():
+ try:
+ self._write_audit(
+ attempt_dir,
+ attempt_id=selected_id,
+ status="failed_validation",
+ command=command,
+ started_at_utc=started_at,
+ error_type=type(exc).__name__,
+ error_message=str(exc),
+ )
+ except Exception:
+ pass
+ raise
+ except Exception as exc:
+ try:
+ self._write_audit(
+ attempt_dir,
+ attempt_id=selected_id,
+ status="failed",
+ command=command,
+ started_at_utc=started_at,
+ error_type=type(exc).__name__,
+ error_message=str(exc),
+ )
+ except Exception:
+ pass
+ raise Sam3DStageError(
+ f"SAM3D execution failed; attempt retained at {attempt_dir}"
+ ) from exc
+
+ placeholder = _snapshot_empty_directory(self.stage05_dir)
+ _remove_empty_directory(self.stage05_dir)
+ published = False
+ try:
+ os.replace(attempt_dir, self.stage05_dir)
+ published = True
+ artifacts = validate_stage05(
+ self.scene_config_path,
+ self.stage05_dir,
+ project_root=self.project_root,
+ )
+ except Exception as exc:
+ rollback_error: Exception | None = None
+ if published and self.stage05_dir.exists():
+ try:
+ os.replace(self.stage05_dir, attempt_dir)
+ except Exception as rollback_exc:
+ rollback_error = rollback_exc
+ try:
+ _restore_empty_directory(self.stage05_dir, placeholder)
+ except Exception as restore_exc:
+ rollback_error = rollback_error or restore_exc
+ detail = (
+ f"; rollback also failed: {rollback_error}"
+ if rollback_error is not None
+ else ""
+ )
+ raise Sam3DStageError(
+ "atomic Stage 05 publication failed; successful attempt was retained"
+ + detail
+ ) from exc
+
+ try:
+ self.attempts_dir.rmdir()
+ except OSError:
+ # Failed or retained attempts are audit evidence and keep the
+ # directory non-empty. Only a newly empty container is removed.
+ pass
+
+ return replace(
+ artifacts,
+ attempt_id=selected_id,
+ audit_path=self.stage05_dir / "sam3d_stage_audit.json",
+ reused=False,
+ )
+
+
+__all__ = [
+ "Sam3DStage",
+ "Sam3DStageArtifacts",
+ "Sam3DStageError",
+ "validate_stage05",
+]
diff --git a/vlm_agent/run_paths.py b/vlm_agent/run_paths.py
new file mode 100644
index 0000000..6e0474c
--- /dev/null
+++ b/vlm_agent/run_paths.py
@@ -0,0 +1,148 @@
+"""Resolve the versioned on-disk contract for one SceneMosaic run.
+
+The scene configuration is the authority for output locations. Keeping the
+contract here prevents individual stages from guessing paths or inspecting
+directory names.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Mapping
+
+
+SEMANTIC_STAGES: dict[str, str] = {
+ "input": "input",
+ "perception": "reconstruction/perception",
+ "sam3d_input": "reconstruction/sam3d/input",
+ "sam3d_attempts": "reconstruction/sam3d/attempts",
+ "sam3d_output": "reconstruction/sam3d/output",
+ "relations": "reconstruction/relations",
+}
+
+
+class ScenePathError(ValueError):
+ """Raised when a scene configuration declares an unsafe output path."""
+
+
+def _read_config(path: Path) -> dict[str, Any]:
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise ScenePathError(f"cannot parse scene_config: {path}") from exc
+ if not isinstance(payload, dict):
+ raise ScenePathError("scene_config must contain a JSON object")
+ return payload
+
+
+def _is_within(path: Path, root: Path) -> bool:
+ try:
+ path.relative_to(root)
+ return True
+ except ValueError:
+ return False
+
+
+def _resolve_relative(run_root: Path, raw: Any, *, label: str) -> Path:
+ if not isinstance(raw, str) or not raw.strip():
+ raise ScenePathError(f"scene_config.stages.{label} must be a non-empty path")
+ relative = Path(raw)
+ if relative.is_absolute():
+ raise ScenePathError(f"scene_config.stages.{label} must be relative")
+ resolved = (run_root / relative).resolve()
+ if resolved == run_root or not _is_within(resolved, run_root):
+ raise ScenePathError(f"scene_config.stages.{label} escapes the scene root")
+ return resolved
+
+
+@dataclass(frozen=True)
+class ScenePaths:
+ """Resolved formal paths for a scene run."""
+
+ run_root: Path
+ scene_config_path: Path
+ config: Mapping[str, Any]
+ input_dir: Path
+ perception_dir: Path
+ sam3d_input_dir: Path
+ sam3d_attempts_dir: Path
+ sam3d_output_dir: Path
+ relations_dir: Path
+ evolution_dir: Path
+
+ @classmethod
+ def from_scene_root(cls, scene_root: Path) -> "ScenePaths":
+ root = Path(scene_root).expanduser().resolve()
+ return cls.from_scene_config(root / "input" / "scene_config.json")
+
+ @classmethod
+ def from_scene_config(cls, scene_config_path: Path) -> "ScenePaths":
+ config_path = Path(scene_config_path).expanduser().resolve()
+ config = _read_config(config_path)
+ raw_root = config.get("run_root")
+ if not isinstance(raw_root, str) or not raw_root.strip():
+ raise ScenePathError("scene_config.run_root must be a non-empty path")
+ run_root = Path(raw_root).expanduser().resolve()
+ expected_config = run_root / "input" / "scene_config.json"
+ if config_path != expected_config:
+ raise ScenePathError(
+ f"scene_config must be the formal run config: {expected_config}"
+ )
+
+ if config.get("schema_version") != "scenemosaic_scene_config":
+ raise ScenePathError("unsupported scene_config schema; create a new run")
+ stages = config.get("stages")
+ if not isinstance(stages, dict):
+ raise ScenePathError("scene_config.stages must contain a JSON object")
+
+ resolved: dict[str, Path] = {}
+ for logical_name, raw in SEMANTIC_STAGES.items():
+ if stages.get(logical_name) != raw:
+ raise ScenePathError(
+ f"scene_config.stages.{logical_name} must equal {raw!r}"
+ )
+ resolved[logical_name] = _resolve_relative(
+ run_root, raw, label=logical_name
+ )
+
+ if resolved["input"] != run_root / "input":
+ raise ScenePathError("scene_config input stage must resolve to <run_root>/input")
+ values = list(resolved.values())
+ if len(set(values)) != len(values):
+ raise ScenePathError("scene_config stage paths must be distinct")
+
+ return cls(
+ run_root=run_root,
+ scene_config_path=config_path,
+ config=config,
+ input_dir=resolved["input"],
+ perception_dir=resolved["perception"],
+ sam3d_input_dir=resolved["sam3d_input"],
+ sam3d_attempts_dir=resolved["sam3d_attempts"],
+ sam3d_output_dir=resolved["sam3d_output"],
+ relations_dir=resolved["relations"],
+ # The original evolution CLI writes its native ``outputs/`` tree
+ # relative to the project root. For an integrated scene, that
+ # project root is the scene root; Stage 08 bookkeeping stays
+ # in hidden siblings rather than a visible ``evolution/`` wrapper.
+ evolution_dir=run_root,
+ )
+
+ @property
+ def relation_candidates_dir(self) -> Path:
+ return self.relations_dir / "candidates"
+
+ @property
+ def relation_publications_dir(self) -> Path:
+ return self.relations_dir / "publications"
+
+ @property
+ def evolution_runs_dir(self) -> Path:
+ return self.evolution_dir / ".stage08_attempts"
+
+ @property
+ def evolution_publications_dir(self) -> Path:
+ return self.evolution_dir / ".stage08_publications"
+
diff --git a/vlm_agent/state.py b/vlm_agent/state.py
new file mode 100644
index 0000000..de63139
--- /dev/null
+++ b/vlm_agent/state.py
@@ -0,0 +1,272 @@
+from __future__ import annotations
+
+import copy
+import fcntl
+import json
+import os
+import uuid
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, IO, Iterable
+
+
+STATE_SCHEMA = "scenemosaic_full_pipeline_state"
+SECRET_KEYS = frozenset(
+ {"api_key", "apikey", "secret", "authorization", "access_token", "refresh_token"}
+)
+
+
+class RunStateError(RuntimeError):
+ """Raised when the durable pipeline state is invalid or unsafe."""
+
+
+@dataclass
+class SceneRunLock:
+ handle: IO[str]
+ exclusive: bool
+
+ def release(self) -> None:
+ if self.handle.closed:
+ return
+ try:
+ fcntl.flock(self.handle.fileno(), fcntl.LOCK_UN)
+ finally:
+ self.handle.close()
+
+
+def utc_now() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def _sanitize(value: Any, secret_values: Iterable[str] = ()) -> Any:
+ secrets = tuple(item for item in secret_values if item)
+ if isinstance(value, dict):
+ cleaned: dict[str, Any] = {}
+ for key, item in value.items():
+ normalized = str(key).lower().replace("-", "_")
+ if normalized in SECRET_KEYS or normalized.endswith("_api_key"):
+ cleaned[str(key)] = "<redacted>"
+ else:
+ cleaned[str(key)] = _sanitize(item, secrets)
+ return cleaned
+ if isinstance(value, list):
+ return [_sanitize(item, secrets) for item in value]
+ if isinstance(value, tuple):
+ return [_sanitize(item, secrets) for item in value]
+ if isinstance(value, str):
+ result = value
+ for secret in secrets:
+ result = result.replace(secret, "<redacted>")
+ return result
+ return value
+
+
+def _atomic_json(path: Path, payload: dict[str, Any]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ temporary = path.with_name(f".{path.name}.{uuid.uuid4().hex}.tmp")
+ try:
+ temporary.write_text(
+ json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
+ with temporary.open("rb") as handle:
+ os.fsync(handle.fileno())
+ os.replace(temporary, path)
+ finally:
+ if temporary.exists():
+ temporary.unlink()
+
+
+@dataclass
+class RunState:
+ """Atomic index for a fact-validated, resumable full pipeline."""
+
+ path: Path
+ data: dict[str, Any]
+
+ @classmethod
+ def create(cls, scene_root: Path, *, scene_name: str) -> "RunState":
+ root = Path(scene_root).expanduser().resolve()
+ path = root / "pipeline_state.json"
+ if path.exists():
+ raise RunStateError(f"pipeline state already exists: {path}")
+ now = utc_now()
+ state = cls(
+ path=path,
+ data={
+ "schema_version": STATE_SCHEMA,
+ "scene_name": scene_name,
+ "scene_root": str(root),
+ "current_state": "initialized",
+ "last_successful_state": "initialized",
+ "next_action": None,
+ "failure": None,
+ "stages": {},
+ "stage_attempts": {},
+ "invocations": [],
+ "created_at_utc": now,
+ "updated_at_utc": now,
+ },
+ )
+ state.save()
+ return state
+
+ @classmethod
+ def load(cls, scene_root_or_path: Path) -> "RunState":
+ supplied = Path(scene_root_or_path).expanduser().resolve()
+ path = supplied if supplied.name == "pipeline_state.json" else supplied / "pipeline_state.json"
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise RunStateError(f"cannot parse pipeline state: {path}") from exc
+ if not isinstance(payload, dict) or payload.get("schema_version") != STATE_SCHEMA:
+ raise RunStateError(f"unsupported pipeline state: {path}")
+ expected_root = path.parent.resolve()
+ if Path(str(payload.get("scene_root"))).expanduser().resolve() != expected_root:
+ raise RunStateError("pipeline state scene_root does not match its location")
+ return cls(path=path, data=payload)
+
+ @property
+ def current_state(self) -> str:
+ return str(self.data.get("current_state") or "")
+
+ @property
+ def scene_root(self) -> Path:
+ return self.path.parent
+
+ def save(self, *, secret_values: Iterable[str] = ()) -> None:
+ self.data["updated_at_utc"] = utc_now()
+ self.data = _sanitize(self.data, secret_values)
+ _atomic_json(self.path, self.data)
+
+ def acquire_lock(
+ self,
+ *,
+ exclusive: bool = True,
+ blocking: bool = False,
+ ) -> SceneRunLock:
+ lock_path = self.scene_root / ".pipeline.lock"
+ if not lock_path.is_file() or lock_path.is_symlink():
+ raise RunStateError(f"pipeline lock file is missing or unsafe: {lock_path}")
+ handle = lock_path.open("r+", encoding="utf-8")
+ operation = fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH
+ if not blocking:
+ operation |= fcntl.LOCK_NB
+ try:
+ fcntl.flock(handle.fileno(), operation)
+ except BlockingIOError as exc:
+ handle.close()
+ raise RunStateError(
+ "another pipeline invocation currently owns this scene"
+ ) from exc
+ return SceneRunLock(handle=handle, exclusive=exclusive)
+
+ def close_stale_invocations(self) -> int:
+ count = 0
+ for record in self.data.get("invocations", []):
+ if record.get("result") == "running":
+ record["ended_at_utc"] = utc_now()
+ record["result"] = "interrupted"
+ count += 1
+ if count:
+ self.save()
+ return count
+
+ def begin_invocation(self, *, command: str, sanitized_argv: list[str]) -> str:
+ invocation_id = f"invocation_{uuid.uuid4().hex}"
+ self.data.setdefault("invocations", []).append(
+ {
+ "invocation_id": invocation_id,
+ "command": command,
+ "argv": _sanitize(list(sanitized_argv)),
+ "started_at_utc": utc_now(),
+ "ended_at_utc": None,
+ "result": "running",
+ }
+ )
+ self.save()
+ return invocation_id
+
+ def finish_invocation(self, invocation_id: str, *, result: str) -> None:
+ for record in reversed(self.data.get("invocations", [])):
+ if record.get("invocation_id") == invocation_id:
+ record["ended_at_utc"] = utc_now()
+ record["result"] = result
+ self.save()
+ return
+ raise RunStateError(f"unknown invocation: {invocation_id}")
+
+ def record_attempt(
+ self,
+ stage: str,
+ *,
+ attempt_id: str,
+ status: str,
+ facts: dict[str, Any] | None = None,
+ ) -> None:
+ if not str(attempt_id).strip():
+ raise RunStateError("attempt_id must be non-empty")
+ self.data.setdefault("stage_attempts", {}).setdefault(
+ stage, []
+ ).append(
+ {
+ "attempt_id": str(attempt_id),
+ "status": str(status),
+ "facts": _sanitize(
+ copy.deepcopy(facts or {})
+ ),
+ "recorded_at_utc": utc_now(),
+ }
+ )
+ self.save()
+
+ def record_stage(
+ self,
+ stage: str,
+ *,
+ status: str,
+ facts: dict[str, Any] | None = None,
+ successful_state: str | None = None,
+ ) -> None:
+ self.data.setdefault("stages", {})[stage] = {
+ "status": status,
+ "facts": _sanitize(copy.deepcopy(facts or {})),
+ "updated_at_utc": utc_now(),
+ }
+ if successful_state:
+ self.data["current_state"] = successful_state
+ self.data["last_successful_state"] = successful_state
+ self.data["failure"] = None
+ self.data["next_action"] = None
+ self.save()
+
+ def fail(
+ self,
+ *,
+ stage: str,
+ error_type: str,
+ message: str,
+ resumable: bool,
+ next_action: str | None,
+ ) -> None:
+ self.data["current_state"] = "failed"
+ self.data["failure"] = {
+ "stage": stage,
+ "error_type": error_type,
+ "message": _sanitize(message),
+ "resumable": bool(resumable),
+ "failed_at_utc": utc_now(),
+ }
+ self.data["next_action"] = next_action
+ self.save()
+
+ def complete(self) -> None:
+ self.data["current_state"] = "scenemosaic_published"
+ self.data["last_successful_state"] = "scenemosaic_published"
+ self.data["next_action"] = None
+ self.data["failure"] = None
+ self.save()
+
+ def public_status(self) -> dict[str, Any]:
+ return _sanitize(copy.deepcopy(self.data))