Skip to content

Official Codebase of 3DCodeVerse

Python

eval/: evaluating the harness (bench) and evaluating a bare LLM (llm) live next to the harness, not in it

gaoypeng6565487e44b5
diff --git a/.gitignore b/.gitignore
index 9462fb36..776c215a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,7 +10,7 @@ node_modules
runs/
harness/runs/
bench/out/
-harness/bench/out/
+eval/bench/out/
# node drivers run with cwd=harness/runtime_js/, so a relative --out drops files here
harness/runtime_js/artifacts/
harness/logs/
diff --git a/CLAUDE.md b/CLAUDE.md
index 4939a9c1..ad948bc8 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -5,4 +5,8 @@ The harness lives in **`harness/`** — read `harness/CLAUDE.md` +
`pip install -e harness`. Run tests with
`cd harness && python -m pytest tests -q -m "not live"`.
-Other top-level folders are separate components (datasets, web, papers).
+**`eval/`** evaluates things and is never imported by the harness: `eval/bench` evaluates the HARNESS
+(batteries, A/B rigs, reports — `cd eval && python -m pytest`), `eval/llm` evaluates a bare LLM/VLM
+(was `finetune/3dcodeverse_eval`). Read `eval/README.md` first.
+
+Other top-level folders are separate components (`toolkits/` data tooling, `finetune/` training).
diff --git a/README.md b/README.md
index ec198d28..91d90893 100644
--- a/README.md
+++ b/README.md
@@ -16,7 +16,7 @@ training corpora; and a finetuning study that trains open models on that data.
└──────────── better generators / judges ◀───────────────────┘
```
-## [`harness/`](harness/) — the generation & evaluation harness
+## [`harness/`](harness/) — the generation harness
Python dist `3dcodeverse` (import `codeverse`, CLI `3dcodeverse` / `3dcode`).
LLMs write raw 3D code across four tracks — `static_object` ·
@@ -37,7 +37,17 @@ bash harness/setup.sh # python deps + node runtime + doctor (Linux, pytho
```
Start at [`harness/README.md`](harness/README.md); design and operation live in
-`harness/docs/` (ARCHITECTURE, INSTALL, RUNBOOK, EVAL, DECISIONS, COST).
+`harness/docs/` (ARCHITECTURE, INSTALL, RUNBOOK, DECISIONS, COST).
+
+## [`eval/`](eval/) — evaluating the harness, and evaluating a bare LLM
+
+Two evaluations kept apart from what they evaluate. [`eval/bench/`](eval/bench/) measures **the
+harness**: prompt batteries run through it and through one-shot / bare-agent arms under one fixed
+judge, paired A/B rigs for harness switches, offline re-judging and reports. [`eval/llm/`](eval/llm/)
+measures **a bare LLM / VLM** on text → 3D code and image → 3D code (3DCodeBench, held-out dialect
+sets, ten harness batteries): every answer is executed and scored against references. Protocols and
+every recorded comparison: `eval/docs/` (EVAL, COMPLEXITY, PAPER_WRITING). Nothing under `harness/`
+imports anything from `eval/`.
## [`toolkits/`](toolkits/) — raw 3D projects → trainable data
diff --git a/eval/README.md b/eval/README.md
new file mode 100644
index 00000000..e10ade4c
--- /dev/null
+++ b/eval/README.md
@@ -0,0 +1,31 @@
+# eval/ — two evaluations, kept apart from what they evaluate
+
+| folder | what it evaluates | how |
+|---|---|---|
+| [`bench/`](bench/) | **the harness** (`../harness`): does plan → build → gates → judge → refine beat asking a model once? | prompt batteries run through the harness and through one-shot / bare-agent arms, judged by ONE fixed evaluator; paired A/B rigs for harness switches; offline re-judging and reports |
+| [`llm/`](llm/) | **a bare LLM / VLM** on 3D coding — no harness in the loop | text → code and image → code over 3DCodeBench, the held-out dialect sets and ten harness batteries; every answer is executed; geometry and image metrics against references |
+
+`docs/` holds the evaluation write-ups: `EVAL.md` (protocol, judge calibration, every recorded
+comparison), `COMPLEXITY.md` (the complexity vector and the score-vs-complexity study),
+`PAPER_WRITING.md`. `tests/` tests the `bench` scripts.
+
+The dependency is one-way: `bench` and `llm` import `codeverse` (the harness package); nothing under
+`harness/` imports anything here, and the harness test suite passes without this folder. The two
+evaluations share one set of battery files — `bench/prompts/*.yaml` — and `llm` reads the ten it can
+ask one-shot from there (`llm/config.py: HARNESS_BATTERIES`).
+
+```bash
+pip install -e ../harness # once; both evaluations import codeverse
+cd eval
+3dcode bench run bench/prompts/static_objects_v1.yaml --generator gemini-cli:gemini-3.7-flash
+python bench/compare_backends.py --prompts bench/prompts/compare_v1.yaml --arms harness:gemini-cli:gemini-3.7-flash,oneshot:gemini:gemini-3.7-flash --out bench/out/compare_v1
+python -m llm.config # doctor for the LLM evaluation: tools and assets it can / cannot find
+python -m pytest # the bench tests (offline)
+ruff check .
+```
+
+Run data is never in git: `bench/out/` (battery output) and `llm/data/prompts/*.jsonl` (built by
+`python -m llm.build_prompts` from the gated Hub assets) are ignored. Every `bench` script bootstraps
+its own import path — `../harness` for THIS tree's `codeverse`, `.` for the `bench` package — so an
+editable install of a different checkout cannot silently stand in for the tree under test
+(`tests/test_worktree_import.py`).
diff --git a/harness/bench/__init__.py b/eval/bench/__init__.py
similarity index 100%
rename from harness/bench/__init__.py
rename to eval/bench/__init__.py
diff --git a/harness/bench/_ab_report.py b/eval/bench/_ab_report.py
similarity index 100%
rename from harness/bench/_ab_report.py
rename to eval/bench/_ab_report.py
diff --git a/harness/bench/_compare_report.py b/eval/bench/_compare_report.py
similarity index 100%
rename from harness/bench/_compare_report.py
rename to eval/bench/_compare_report.py
diff --git a/harness/bench/_fixed_eval.py b/eval/bench/_fixed_eval.py
similarity index 100%
rename from harness/bench/_fixed_eval.py
rename to eval/bench/_fixed_eval.py
diff --git a/harness/bench/_infra.py b/eval/bench/_infra.py
similarity index 100%
rename from harness/bench/_infra.py
rename to eval/bench/_infra.py
diff --git a/harness/bench/_jsonl.py b/eval/bench/_jsonl.py
similarity index 100%
rename from harness/bench/_jsonl.py
rename to eval/bench/_jsonl.py
diff --git a/harness/bench/_oneshot.py b/eval/bench/_oneshot.py
similarity index 100%
rename from harness/bench/_oneshot.py
rename to eval/bench/_oneshot.py
diff --git a/harness/bench/ab_gate_rates.py b/eval/bench/ab_gate_rates.py
similarity index 100%
rename from harness/bench/ab_gate_rates.py
rename to eval/bench/ab_gate_rates.py
diff --git a/harness/bench/ab_plan.py b/eval/bench/ab_plan.py
similarity index 99%
rename from harness/bench/ab_plan.py
rename to eval/bench/ab_plan.py
index e4aaf4fc..6d366c2c 100644
--- a/harness/bench/ab_plan.py
+++ b/eval/bench/ab_plan.py
@@ -66,9 +66,10 @@ from pydantic import BaseModel, Field
# child of a worktree run the MAIN tree's harness, both arms identically, and the A/B
# measured nothing while looking completely healthy — it happened twice on 2026-08-25
# (`bench/out/plan_loop/C0/invalid_attempt1_maintree_import`, and again to the skills wave).
-REPO = Path(__file__).resolve().parents[1]
-if str(REPO) not in sys.path: # `python bench/ab_plan.py` from anywhere
- sys.path.insert(0, str(REPO))
+REPO = Path(__file__).resolve().parents[2] / "harness" # the harness tree these scripts evaluate
+for _p in (REPO, Path(__file__).resolve().parents[1]): # its codeverse + the `bench` package (eval/)
+ if str(_p) not in sys.path:
+ sys.path.insert(0, str(_p))
from datetime import UTC # noqa: E402
diff --git a/harness/bench/ab_view.py b/eval/bench/ab_view.py
similarity index 100%
rename from harness/bench/ab_view.py
rename to eval/bench/ab_view.py
diff --git a/harness/bench/bestofk_report.py b/eval/bench/bestofk_report.py
similarity index 98%
rename from harness/bench/bestofk_report.py
rename to eval/bench/bestofk_report.py
index b54052a0..3924c4f3 100644
--- a/harness/bench/bestofk_report.py
+++ b/eval/bench/bestofk_report.py
@@ -35,7 +35,8 @@ import sys
from collections import defaultdict
from pathlib import Path
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+for _p in (Path(__file__).resolve().parents[2] / "harness", Path(__file__).resolve().parents[1]):
+ sys.path.insert(0, str(_p)) # this tree's codeverse (harness/) + the `bench` package (eval/)
from codeverse.proc import read_jsonl_lenient # noqa: E402
diff --git a/harness/bench/compare_backends.py b/eval/bench/compare_backends.py
similarity index 99%
rename from harness/bench/compare_backends.py
rename to eval/bench/compare_backends.py
index 4c84fed4..e540c134 100644
--- a/harness/bench/compare_backends.py
+++ b/eval/bench/compare_backends.py
@@ -49,9 +49,10 @@ from pydantic import BaseModel, Field
# sys.path BEFORE any `codeverse` import: this file is also run as a script, and an
# editable install would otherwise resolve `codeverse` to the tree it was installed from
# rather than this one. See the same note in `bench/ab_plan.py`.
-REPO = Path(__file__).resolve().parents[1]
-if str(REPO) not in sys.path: # `python bench/compare_backends.py` from anywhere
- sys.path.insert(0, str(REPO))
+REPO = Path(__file__).resolve().parents[2] / "harness" # the harness tree these scripts evaluate
+for _p in (REPO, Path(__file__).resolve().parents[1]): # its codeverse + the `bench` package (eval/)
+ if str(_p) not in sys.path:
+ sys.path.insert(0, str(_p))
from datetime import UTC # noqa: E402
diff --git a/harness/bench/complexity_report.py b/eval/bench/complexity_report.py
similarity index 98%
rename from harness/bench/complexity_report.py
rename to eval/bench/complexity_report.py
index f1c4ba6d..60f4df2f 100644
--- a/harness/bench/complexity_report.py
+++ b/eval/bench/complexity_report.py
@@ -28,7 +28,8 @@ from collections.abc import Iterable, Sequence
from pathlib import Path
from typing import Any
-sys.path.insert(0, str(Path(__file__).resolve().parents[0].parent))
+for _p in (Path(__file__).resolve().parents[2] / "harness", Path(__file__).resolve().parents[1]):
+ sys.path.insert(0, str(_p)) # this tree's codeverse (harness/) + the `bench` package (eval/)
from codeverse.addons.calibration import _ranks # noqa: E402 — tie-averaged ranks, one copy
from codeverse.contracts.run import RunId # noqa: E402
diff --git a/harness/bench/concurrency_probe.py b/eval/bench/concurrency_probe.py
similarity index 100%
rename from harness/bench/concurrency_probe.py
rename to eval/bench/concurrency_probe.py
diff --git a/harness/bench/coupling_stats.py b/eval/bench/coupling_stats.py
similarity index 97%
rename from harness/bench/coupling_stats.py
rename to eval/bench/coupling_stats.py
index e8450c61..9c1bee64 100644
--- a/harness/bench/coupling_stats.py
+++ b/eval/bench/coupling_stats.py
@@ -21,7 +21,8 @@ import statistics
import sys
from pathlib import Path
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # `python bench/coupling_stats.py` from the repo root
+for _p in (Path(__file__).resolve().parents[2] / "harness", Path(__file__).resolve().parents[1]):
+ sys.path.insert(0, str(_p)) # this tree's codeverse (harness/) + the `bench` package (eval/)
from codeverse.record.record import unique_files # noqa: E402
from codeverse.spatial.joints_model import UrdfError, load_urdf # noqa: E402
diff --git a/harness/bench/data/coupled/README.md b/eval/bench/data/coupled/README.md
similarity index 100%
rename from harness/bench/data/coupled/README.md
rename to eval/bench/data/coupled/README.md
diff --git a/harness/bench/data/coupled/base_mimic.jsonl b/eval/bench/data/coupled/base_mimic.jsonl
similarity index 100%
rename from harness/bench/data/coupled/base_mimic.jsonl
rename to eval/bench/data/coupled/base_mimic.jsonl
diff --git a/harness/bench/data/coupled/base_nomimic.jsonl b/eval/bench/data/coupled/base_nomimic.jsonl
similarity index 100%
rename from harness/bench/data/coupled/base_nomimic.jsonl
rename to eval/bench/data/coupled/base_nomimic.jsonl
diff --git a/harness/bench/data/plan_stage/README.md b/eval/bench/data/plan_stage/README.md
similarity index 100%
rename from harness/bench/data/plan_stage/README.md
rename to eval/bench/data/plan_stage/README.md
diff --git a/harness/bench/data/plan_stage/coupled_plan.jsonl b/eval/bench/data/plan_stage/coupled_plan.jsonl
similarity index 100%
rename from harness/bench/data/plan_stage/coupled_plan.jsonl
rename to eval/bench/data/plan_stage/coupled_plan.jsonl
diff --git a/harness/bench/data/plan_stage/restart_off.jsonl b/eval/bench/data/plan_stage/restart_off.jsonl
similarity index 100%
rename from harness/bench/data/plan_stage/restart_off.jsonl
rename to eval/bench/data/plan_stage/restart_off.jsonl
diff --git a/harness/bench/data/plan_stage/restart_on.jsonl b/eval/bench/data/plan_stage/restart_on.jsonl
similarity index 100%
rename from harness/bench/data/plan_stage/restart_on.jsonl
rename to eval/bench/data/plan_stage/restart_on.jsonl
diff --git a/harness/bench/data/plan_stage/trigger_narrow.jsonl b/eval/bench/data/plan_stage/trigger_narrow.jsonl
similarity index 100%
rename from harness/bench/data/plan_stage/trigger_narrow.jsonl
rename to eval/bench/data/plan_stage/trigger_narrow.jsonl
diff --git a/harness/bench/data/plan_stage/trigger_off.jsonl b/eval/bench/data/plan_stage/trigger_off.jsonl
similarity index 100%
rename from harness/bench/data/plan_stage/trigger_off.jsonl
rename to eval/bench/data/plan_stage/trigger_off.jsonl
diff --git a/harness/bench/data/plan_stage/trigger_wide.jsonl b/eval/bench/data/plan_stage/trigger_wide.jsonl
similarity index 100%
rename from harness/bench/data/plan_stage/trigger_wide.jsonl
rename to eval/bench/data/plan_stage/trigger_wide.jsonl
diff --git a/harness/bench/h2h_glb.py b/eval/bench/h2h_glb.py
similarity index 98%
rename from harness/bench/h2h_glb.py
rename to eval/bench/h2h_glb.py
index f96ea09a..efce132f 100644
--- a/harness/bench/h2h_glb.py
+++ b/eval/bench/h2h_glb.py
@@ -41,7 +41,8 @@ from typing import Any
from pydantic import BaseModel, Field
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # `python bench/h2h_glb.py` from the repo root
+for _p in (Path(__file__).resolve().parents[2] / "harness", Path(__file__).resolve().parents[1]):
+ sys.path.insert(0, str(_p)) # this tree's codeverse (harness/) + the `bench` package (eval/)
from bench._fixed_eval import RUBRIC # noqa: E402
from bench.h2h_scene import sign_test # noqa: E402
diff --git a/harness/bench/h2h_scene.py b/eval/bench/h2h_scene.py
similarity index 98%
rename from harness/bench/h2h_scene.py
rename to eval/bench/h2h_scene.py
index 3a08c205..6a09cd50 100644
--- a/harness/bench/h2h_scene.py
+++ b/eval/bench/h2h_scene.py
@@ -38,7 +38,8 @@ from typing import Any
from PIL import Image, ImageDraw
from pydantic import BaseModel
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # `python bench/h2h_scene.py` from the repo root
+for _p in (Path(__file__).resolve().parents[2] / "harness", Path(__file__).resolve().parents[1]):
+ sys.path.insert(0, str(_p)) # this tree's codeverse (harness/) + the `bench` package (eval/)
from bench.run_bench import Battery, BenchPrompt # noqa: E402
from codeverse.contracts.artifacts import RenderSet, RenderView # noqa: E402
diff --git a/harness/bench/judge_calib_graphics.py b/eval/bench/judge_calib_graphics.py
similarity index 100%
rename from harness/bench/judge_calib_graphics.py
rename to eval/bench/judge_calib_graphics.py
diff --git a/harness/bench/paired_compare.py b/eval/bench/paired_compare.py
similarity index 98%
rename from harness/bench/paired_compare.py
rename to eval/bench/paired_compare.py
index 00c43252..387d9fb3 100644
--- a/harness/bench/paired_compare.py
+++ b/eval/bench/paired_compare.py
@@ -29,9 +29,10 @@ from pathlib import Path
from pydantic import BaseModel, Field
-REPO = Path(__file__).resolve().parents[1]
-if str(REPO) not in sys.path: # `python bench/paired_compare.py` from anywhere
- sys.path.insert(0, str(REPO))
+REPO = Path(__file__).resolve().parents[2] / "harness" # the harness tree these scripts evaluate
+for _p in (REPO, Path(__file__).resolve().parents[1]): # its codeverse + the `bench` package (eval/)
+ if str(_p) not in sys.path:
+ sys.path.insert(0, str(_p))
from bench._ab_report import _fmt as _f # noqa: E402
from bench._ab_report import _sign_test # noqa: E402
diff --git a/harness/bench/penetration_thresholds.py b/eval/bench/penetration_thresholds.py
similarity index 98%
rename from harness/bench/penetration_thresholds.py
rename to eval/bench/penetration_thresholds.py
index be845c61..67a4cff2 100644
--- a/harness/bench/penetration_thresholds.py
+++ b/eval/bench/penetration_thresholds.py
@@ -24,7 +24,8 @@ import sys
from collections import Counter
from pathlib import Path
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+for _p in (Path(__file__).resolve().parents[2] / "harness", Path(__file__).resolve().parents[1]):
+ sys.path.insert(0, str(_p)) # this tree's codeverse (harness/) + the `bench` package (eval/)
from codeverse.languages.urdf import REST_PENETRATION_MAX_M # noqa: E402
from codeverse.record.record import unique_files # noqa: E402
diff --git a/harness/bench/pin_plan.py b/eval/bench/pin_plan.py
similarity index 100%
rename from harness/bench/pin_plan.py
rename to eval/bench/pin_plan.py
diff --git a/harness/bench/plan_stage_bench.py b/eval/bench/plan_stage_bench.py
similarity index 97%
rename from harness/bench/plan_stage_bench.py
rename to eval/bench/plan_stage_bench.py
index 30fdfce8..28eea60e 100644
--- a/harness/bench/plan_stage_bench.py
+++ b/eval/bench/plan_stage_bench.py
@@ -168,7 +168,8 @@ def run_one(battery, item, backends, label: str, rep: int, provenance: dict[str,
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
- ap.add_argument("--tree", type=Path, default=Path.cwd(), help="harness tree to import from")
+ ap.add_argument("--tree", type=Path, default=Path(__file__).resolve().parents[2] / "harness",
+ help="harness tree to import from")
ap.add_argument("--label", required=True, help="arm name, recorded on every row")
ap.add_argument("--reps", type=int, default=20, help="calls per prompt")
ap.add_argument("--out", type=Path, required=True)
@@ -189,7 +190,8 @@ def main(argv: list[str] | None = None) -> int:
key, _, value = kv.partition("=")
os.environ[key.strip()] = value.strip()
print(f"env {key.strip()}={value.strip()}")
- sys.path.insert(0, str(ns.tree.resolve()))
+ sys.path.insert(0, str(Path(__file__).resolve().parents[1])) # the `bench` package (eval/)
+ sys.path.insert(0, str(ns.tree.resolve())) # the harness tree under test wins for `codeverse`
import codeverse
from bench._jsonl import seal_for_append # after the path insert: `python bench/x.py`
diff --git a/harness/bench/plan_stage_report.py b/eval/bench/plan_stage_report.py
similarity index 100%
rename from harness/bench/plan_stage_report.py
rename to eval/bench/plan_stage_report.py
diff --git a/harness/bench/prompts/articulated_v1.yaml b/eval/bench/prompts/articulated_v1.yaml
similarity index 100%
rename from harness/bench/prompts/articulated_v1.yaml
rename to eval/bench/prompts/articulated_v1.yaml
diff --git a/harness/bench/prompts/articulated_v2.yaml b/eval/bench/prompts/articulated_v2.yaml
similarity index 100%
rename from harness/bench/prompts/articulated_v2.yaml
rename to eval/bench/prompts/articulated_v2.yaml
diff --git a/harness/bench/prompts/compare_v1.yaml b/eval/bench/prompts/compare_v1.yaml
similarity index 100%
rename from harness/bench/prompts/compare_v1.yaml
rename to eval/bench/prompts/compare_v1.yaml
diff --git a/harness/bench/prompts/compare_v2.yaml b/eval/bench/prompts/compare_v2.yaml
similarity index 100%
rename from harness/bench/prompts/compare_v2.yaml
rename to eval/bench/prompts/compare_v2.yaml
diff --git a/harness/bench/prompts/compare_v3.yaml b/eval/bench/prompts/compare_v3.yaml
similarity index 100%
rename from harness/bench/prompts/compare_v3.yaml
rename to eval/bench/prompts/compare_v3.yaml
diff --git a/harness/bench/prompts/compare_v4.yaml b/eval/bench/prompts/compare_v4.yaml
similarity index 100%
rename from harness/bench/prompts/compare_v4.yaml
rename to eval/bench/prompts/compare_v4.yaml
diff --git a/harness/bench/prompts/complexity_v3.yaml b/eval/bench/prompts/complexity_v3.yaml
similarity index 100%
rename from harness/bench/prompts/complexity_v3.yaml
rename to eval/bench/prompts/complexity_v3.yaml
diff --git a/harness/bench/prompts/coupled_v1.yaml b/eval/bench/prompts/coupled_v1.yaml
similarity index 100%
rename from harness/bench/prompts/coupled_v1.yaml
rename to eval/bench/prompts/coupled_v1.yaml
diff --git a/harness/bench/prompts/fancy_v1_blender.yaml b/eval/bench/prompts/fancy_v1_blender.yaml
similarity index 100%
rename from harness/bench/prompts/fancy_v1_blender.yaml
rename to eval/bench/prompts/fancy_v1_blender.yaml
diff --git a/harness/bench/prompts/fancy_v1_cadquery.yaml b/eval/bench/prompts/fancy_v1_cadquery.yaml
similarity index 100%
rename from harness/bench/prompts/fancy_v1_cadquery.yaml
rename to eval/bench/prompts/fancy_v1_cadquery.yaml
diff --git a/harness/bench/prompts/fancy_v1_threejs.yaml b/eval/bench/prompts/fancy_v1_threejs.yaml
similarity index 100%
rename from harness/bench/prompts/fancy_v1_threejs.yaml
rename to eval/bench/prompts/fancy_v1_threejs.yaml
diff --git a/harness/bench/prompts/graphics_v1.yaml b/eval/bench/prompts/graphics_v1.yaml
similarity index 100%
rename from harness/bench/prompts/graphics_v1.yaml
rename to eval/bench/prompts/graphics_v1.yaml
diff --git a/harness/bench/prompts/graphics_v2.yaml b/eval/bench/prompts/graphics_v2.yaml
similarity index 100%
rename from harness/bench/prompts/graphics_v2.yaml
rename to eval/bench/prompts/graphics_v2.yaml
diff --git a/harness/bench/prompts/h2h_brilliana_v1_blender.yaml b/eval/bench/prompts/h2h_brilliana_v1_blender.yaml
similarity index 100%
rename from harness/bench/prompts/h2h_brilliana_v1_blender.yaml
rename to eval/bench/prompts/h2h_brilliana_v1_blender.yaml
diff --git a/harness/bench/prompts/h2h_brilliana_v1_cadquery.yaml b/eval/bench/prompts/h2h_brilliana_v1_cadquery.yaml
similarity index 100%
rename from harness/bench/prompts/h2h_brilliana_v1_cadquery.yaml
rename to eval/bench/prompts/h2h_brilliana_v1_cadquery.yaml
diff --git a/harness/bench/prompts/h2h_brilliana_v1_threejs.yaml b/eval/bench/prompts/h2h_brilliana_v1_threejs.yaml
similarity index 100%
rename from harness/bench/prompts/h2h_brilliana_v1_threejs.yaml
rename to eval/bench/prompts/h2h_brilliana_v1_threejs.yaml
diff --git a/harness/bench/prompts/h2h_scene_v1.yaml b/eval/bench/prompts/h2h_scene_v1.yaml
similarity index 100%
rename from harness/bench/prompts/h2h_scene_v1.yaml
rename to eval/bench/prompts/h2h_scene_v1.yaml
diff --git a/harness/bench/prompts/refs_v1_graphics.yaml b/eval/bench/prompts/refs_v1_graphics.yaml
similarity index 100%
rename from harness/bench/prompts/refs_v1_graphics.yaml
rename to eval/bench/prompts/refs_v1_graphics.yaml
diff --git a/harness/bench/prompts/refs_v2_graphics.yaml b/eval/bench/prompts/refs_v2_graphics.yaml
similarity index 100%
rename from harness/bench/prompts/refs_v2_graphics.yaml
rename to eval/bench/prompts/refs_v2_graphics.yaml
diff --git a/harness/bench/prompts/scenes_v1.yaml b/eval/bench/prompts/scenes_v1.yaml
similarity index 100%
rename from harness/bench/prompts/scenes_v1.yaml
rename to eval/bench/prompts/scenes_v1.yaml
diff --git a/harness/bench/prompts/scenes_v2.yaml b/eval/bench/prompts/scenes_v2.yaml
similarity index 100%
rename from harness/bench/prompts/scenes_v2.yaml
rename to eval/bench/prompts/scenes_v2.yaml
diff --git a/harness/bench/prompts/static_objects_v1.yaml b/eval/bench/prompts/static_objects_v1.yaml
similarity index 100%
rename from harness/bench/prompts/static_objects_v1.yaml
rename to eval/bench/prompts/static_objects_v1.yaml
diff --git a/harness/bench/prompts/static_objects_v2.yaml b/eval/bench/prompts/static_objects_v2.yaml
similarity index 100%
rename from harness/bench/prompts/static_objects_v2.yaml
rename to eval/bench/prompts/static_objects_v2.yaml
diff --git a/harness/bench/prompts/teaser_v1_articulated.yaml b/eval/bench/prompts/teaser_v1_articulated.yaml
similarity index 100%
rename from harness/bench/prompts/teaser_v1_articulated.yaml
rename to eval/bench/prompts/teaser_v1_articulated.yaml
diff --git a/harness/bench/prompts/teaser_v1_graphics.yaml b/eval/bench/prompts/teaser_v1_graphics.yaml
similarity index 100%
rename from harness/bench/prompts/teaser_v1_graphics.yaml
rename to eval/bench/prompts/teaser_v1_graphics.yaml
diff --git a/harness/bench/prompts/teaser_v1_scene.yaml b/eval/bench/prompts/teaser_v1_scene.yaml
similarity index 100%
rename from harness/bench/prompts/teaser_v1_scene.yaml
rename to eval/bench/prompts/teaser_v1_scene.yaml
diff --git a/harness/bench/prompts/teaser_v1_static.yaml b/eval/bench/prompts/teaser_v1_static.yaml
similarity index 100%
rename from harness/bench/prompts/teaser_v1_static.yaml
rename to eval/bench/prompts/teaser_v1_static.yaml
diff --git a/harness/bench/prompts/turns_v1.yaml b/eval/bench/prompts/turns_v1.yaml
similarity index 100%
rename from harness/bench/prompts/turns_v1.yaml
rename to eval/bench/prompts/turns_v1.yaml
diff --git a/harness/bench/refs/README.md b/eval/bench/refs/README.md
similarity index 100%
rename from harness/bench/refs/README.md
rename to eval/bench/refs/README.md
diff --git a/harness/bench/refs/tsr_gfx_aurora_ridge/aurora_green_spiral_sea.png b/eval/bench/refs/tsr_gfx_aurora_ridge/aurora_green_spiral_sea.png
similarity index 100%
rename from harness/bench/refs/tsr_gfx_aurora_ridge/aurora_green_spiral_sea.png
rename to eval/bench/refs/tsr_gfx_aurora_ridge/aurora_green_spiral_sea.png
diff --git a/harness/bench/refs/tsr_gfx_aurora_ridge/aurora_purple_green_lake.png b/eval/bench/refs/tsr_gfx_aurora_ridge/aurora_purple_green_lake.png
similarity index 100%
rename from harness/bench/refs/tsr_gfx_aurora_ridge/aurora_purple_green_lake.png
rename to eval/bench/refs/tsr_gfx_aurora_ridge/aurora_purple_green_lake.png
diff --git a/harness/bench/rejudge_offline.py b/eval/bench/rejudge_offline.py
similarity index 100%
rename from harness/bench/rejudge_offline.py
rename to eval/bench/rejudge_offline.py
diff --git a/harness/bench/report.py b/eval/bench/report.py
similarity index 100%
rename from harness/bench/report.py
rename to eval/bench/report.py
diff --git a/harness/bench/run_bench.py b/eval/bench/run_bench.py
similarity index 100%
rename from harness/bench/run_bench.py
rename to eval/bench/run_bench.py
diff --git a/harness/bench/scene_regate.py b/eval/bench/scene_regate.py
similarity index 97%
rename from harness/bench/scene_regate.py
rename to eval/bench/scene_regate.py
index fc4c93cf..0751586b 100644
--- a/harness/bench/scene_regate.py
+++ b/eval/bench/scene_regate.py
@@ -29,7 +29,8 @@ import sys
import tempfile
from pathlib import Path
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # this tree's codeverse
+for _p in (Path(__file__).resolve().parents[2] / "harness", Path(__file__).resolve().parents[1]):
+ sys.path.insert(0, str(_p)) # this tree's codeverse (harness/) + the `bench` package (eval/)
from bench.scene_stats import scene_records # noqa: E402
from codeverse.config import get_settings # noqa: E402
diff --git a/harness/bench/scene_stats.py b/eval/bench/scene_stats.py
similarity index 97%
rename from harness/bench/scene_stats.py
rename to eval/bench/scene_stats.py
index 9203d26c..a59b6dc6 100644
--- a/harness/bench/scene_stats.py
+++ b/eval/bench/scene_stats.py
@@ -24,7 +24,8 @@ import sys
from collections import Counter
from pathlib import Path
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # this tree's codeverse, not the editable install
+for _p in (Path(__file__).resolve().parents[2] / "harness", Path(__file__).resolve().parents[1]):
+ sys.path.insert(0, str(_p)) # this tree's codeverse (harness/) + the `bench` package (eval/)
from codeverse.proc import read_jsonl_lenient # noqa: E402
from codeverse.record.record import unique_files # noqa: E402
diff --git a/harness/bench/session_stats.py b/eval/bench/session_stats.py
similarity index 97%
rename from harness/bench/session_stats.py
rename to eval/bench/session_stats.py
index 354af8fa..897f3f6b 100644
--- a/harness/bench/session_stats.py
+++ b/eval/bench/session_stats.py
@@ -22,7 +22,8 @@ import sys
from collections import Counter
from pathlib import Path
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # `python bench/session_stats.py` from the repo root
+for _p in (Path(__file__).resolve().parents[2] / "harness", Path(__file__).resolve().parents[1]):
+ sys.path.insert(0, str(_p)) # this tree's codeverse (harness/) + the `bench` package (eval/)
from codeverse.record.record import unique_files # noqa: E402
diff --git a/harness/bench/skill_targets.py b/eval/bench/skill_targets.py
similarity index 98%
rename from harness/bench/skill_targets.py
rename to eval/bench/skill_targets.py
index 3ec685d1..e0c6cfb8 100644
--- a/harness/bench/skill_targets.py
+++ b/eval/bench/skill_targets.py
@@ -30,7 +30,8 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Any
-sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+for _p in (Path(__file__).resolve().parents[2] / "harness", Path(__file__).resolve().parents[1]):
+ sys.path.insert(0, str(_p)) # this tree's codeverse (harness/) + the `bench` package (eval/)
from codeverse.addons.skill_targets import ( # noqa: E402
SRC_ARTIFACT,
diff --git a/harness/docs/COMPLEXITY.md b/eval/docs/COMPLEXITY.md
similarity index 98%
rename from harness/docs/COMPLEXITY.md
rename to eval/docs/COMPLEXITY.md
index deaa888c..222c0aca 100644
--- a/harness/docs/COMPLEXITY.md
+++ b/eval/docs/COMPLEXITY.md
@@ -1,5 +1,9 @@
# Complexity — measuring how much artifact the harness can actually build
+> **Paths in this document.** `bench/…` is relative to `eval/` (this folder's parent); `codeverse/…`, `tests/…`,
+> `runtime_js/…` and the other `docs/…` files are relative to `harness/`. Recorded battery output (`bench/out/…`)
+> is run data and is not in git.
+
Every other number in this repo answers *how good is it?* This one answers
*how much of it is there?* — objectively, from the built artifact, with no VLM,
no render and no model call. The pair is the point: a 0.9 on a five-box stool
diff --git a/harness/docs/EVAL.md b/eval/docs/EVAL.md
similarity index 99%
rename from harness/docs/EVAL.md
rename to eval/docs/EVAL.md
index 39abad70..e758373b 100644
--- a/harness/docs/EVAL.md
+++ b/eval/docs/EVAL.md
@@ -1,5 +1,9 @@
# Evaluation protocol
+> **Paths in this document.** `bench/…` is relative to `eval/` (this folder's parent); `codeverse/…`, `tests/…`,
+> `runtime_js/…` and the other `docs/…` files are relative to `harness/`. Recorded battery output (`bench/out/…`)
+> is run data and is not in git.
+
How we decide whether the harness (plan → generate → gate → render → judge →
refine) beats raw generation, and whether one backend beats another, without
fooling ourselves. Tools: `3dcode bench`, `bench/compare_backends.py`,
diff --git a/harness/docs/PAPER_WRITING.md b/eval/docs/PAPER_WRITING.md
similarity index 99%
rename from harness/docs/PAPER_WRITING.md
rename to eval/docs/PAPER_WRITING.md
index 36a31adf..20295a04 100644
--- a/harness/docs/PAPER_WRITING.md
+++ b/eval/docs/PAPER_WRITING.md
@@ -1,5 +1,9 @@
# Paper writing: everything measured, and where it came from
+> **Paths in this document.** `bench/…` is relative to `eval/` (this folder's parent); `codeverse/…`, `tests/…`,
+> `runtime_js/…` and the other `docs/…` files are relative to `harness/`. Recorded battery output (`bench/out/…`)
+> is run data and is not in git.
+
Everything a paper about this harness would need in one file: what the system is, what was
built, every experiment with its setup and numbers, the null results, the mechanism findings,
the methodology, and a claim → evidence index. Written 2026-09-03/04 over the work of
diff --git a/eval/llm/.gitignore b/eval/llm/.gitignore
new file mode 100644
index 00000000..7331ebd0
--- /dev/null
+++ b/eval/llm/.gitignore
@@ -0,0 +1,6 @@
+# large / generated assets live under $CV3D_EVAL_DATA, never here
+__pycache__/
+*.pyc
+out/
+# built by `python -m llm.build_prompts` from the (gated) Hub assets — datasets are kept out of git
+data/prompts/*.jsonl
diff --git a/eval/llm/CLAUDE.md b/eval/llm/CLAUDE.md
new file mode 100644
index 00000000..c85de23d
--- /dev/null
+++ b/eval/llm/CLAUDE.md
@@ -0,0 +1,21 @@
+# eval/llm (component notes for agents; was `3dcodeverse_eval`)
+
+Standalone LLM/VLM evaluation for 3D code (Blender-Python / CadQuery / OpenSCAD / GLSL / three.js). Read
+`README.md`, then `docs/metrics.md` and `docs/protocol.md`. Do not import from `../eval` or `../scripts`;
+this package is the portable re-implementation (provenance in `docs/inventory_*.md`).
+
+- **Paths**: never hardcode; everything goes through `config.py` (`CV3D_EVAL_DATA`, `CV3D_TOOLS`, `CV3D_BLENDER` …).
+ Large assets live under `$CV3D_EVAL_DATA` (hub/, refs/, out/), never in git. `data/prompts/*.jsonl` are committed.
+- **Environment**: conda env `cv3d-eval` (`~/miniconda3/envs/cv3d-eval/bin/python`): vLLM 0.28 + torch cu130
+ (RTX 5090 works), cadquery, playwright, trimesh, transformers. Blender 5.0.1 at `~/.local/bin/blender-5.0`
+ (bundled python has scipy 1.17.1 + shapely), OpenSCAD AppImage + glslang under `~/3dcodeverse_data/tools`.
+ `python -m llm.config` is the doctor.
+- **Protocols**: keep the report protocol (`3dcodebench_text`, `heldout_*`) byte-identical to
+ `finetune/docs/REPORT.md` so numbers stay comparable; the official protocol (`3dcodebench_official_*`)
+ mirrors `~/nips2026_paper/3dcodebench` (prompts, renderer, SigLIP-2/DINOv3, squared unit-sphere Chamfer).
+ Changing a system prompt or metric = a new suite name, not an edit.
+- **Long-output dialects** (OpenSCAD / GLSL / three.js): T=0.7 is the primary protocol (greedy degenerates);
+ always report `truncated`. Compare models only within one batch (noise floor in `docs/metrics.md` §6).
+- **Statuses** are uniform: OK / EMPTY / FAIL / CRASH / TIMEOUT / MISSING; `*_mean_all` uses `n_ref_ok` as the
+ denominator (reference failures are not charged to the model).
+- Tests: `~/miniconda3/envs/cv3d-eval/bin/python -m pytest llm/tests -q` (needs the tools for the executor tests).
diff --git a/eval/llm/README.md b/eval/llm/README.md
new file mode 100644
index 00000000..2c5fd06c
--- /dev/null
+++ b/eval/llm/README.md
@@ -0,0 +1,134 @@
+# eval/llm — one place to evaluate an LLM / VLM on 3D coding (was `finetune/3dcodeverse_eval`)
+
+Self-contained, machine-independent evaluation of *text → 3D code* and *image → 3D code* across the
+3DCodeVerse dialects (Blender-Python, CadQuery, OpenSCAD, GLSL, three.js). It packages the prompt sets,
+the reference assets, the executors, the metrics and the reporting that were previously spread over
+`finetune/eval`, `finetune/scripts`, `harness/bench` and the official `3dcodebench` repo, with every
+path resolved from the environment (`config.py`) instead of `/wekafs/...`.
+
+```
+llm/
+ config.py paths + tool discovery (CV3D_EVAL_DATA, CV3D_TOOLS, CV3D_BLENDER, …); `python -m llm.config` = doctor
+ download.py pull the assets from the Hub (needs a token with access to ilabai/*)
+ build_prompts.py Hub assets → data/prompts/<suite>.jsonl (one schema for every suite)
+ build_refs.py execute the reference programs → $CV3D_EVAL_DATA/refs/<suite>/<id>/ref.{glb,stl}
+ suites.py suite registry: dialect, decoding defaults, which metrics apply, tags
+ generate.py backends: vllm (local, text+image) · openai-compatible (vLLM server / API) · anthropic · reference
+ extract.py dialect-aware code extraction from free-form answers (think blocks, multiple fences)
+ executors/ blender · cadquery · openscad · glsl (compile + WebGL render) · threejs (Playwright)
+ execute.py run a gen dir → exec_results.jsonl (uniform OK/EMPTY/FAIL/CRASH/TIMEOUT/MISSING)
+ render_views.py official 4-view turntable render of a GLB (Cycles, 512 px)
+ metrics.py Chamfer / F@τ / voxel IoU (report protocol) · official squared unit-sphere Chamfer · pass@k
+ image_sim.py SigLIP-2 / DINOv3 view-paired + best-assignment similarity (official image metric)
+ structural.py official structural-integrity metric (floating parts, fragmentation, watertight, stability), vendored
+ judge.py VLM judge: absolute 1-5 grades and pairwise A/B (both orders) from the 4 renders — quality without GT
+ rescore_logs.py replay the paper's frontier-model outputs through this pipeline
+ setup.sh installs the conda env + Blender / OpenSCAD / glslang / Chromium (no sudo)
+ score.py metrics.jsonl + summary.json per (run, suite)
+ report.py report.md / report.json over an output root (+ pass@k over sampled runs)
+ run_eval.py gen → exec → render → score → report in one command
+ data/prompts/ the prompt sets (built by build_prompts.py; NOT in git) harness YAML batteries: ../bench/prompts (config.HARNESS_BATTERIES)
+ data/official_prompts/ the official 3DCodeBench system prompts
+ docs/ metrics.md · protocol.md · survey_synthesis.md · survey_records.md · inventory_*.md · results/
+ tests/ offline smoke tests (`pytest llm/tests`)
+```
+
+## Suites (`python -c "from llm.suites import all_suites; ..."` or `suites.py`)
+
+| suite | n | input → dialect | reference | metrics | decoding default |
+|---|---|---|---|---|---|
+| `3dcodebench_text` | 212 | structured instruction → Blender | GT mesh (re-executed factory, + canonical GLB) | exec, Chamfer, [email protected]/0.1, IoU | greedy, 8k |
+| `3dcodebench_desc` | 212 | short description → Blender | same | same | greedy, 8k |
+| `3dcodebench_img1` / `img4` / `img4text` | 212 | 1 / 4 GT views (+ description) → Blender | same | same (VLM) | greedy, 8k |
+| `3dcodebench_official_text` / `official_img4` | 212 | official system prompt (raw python, no fence) | same | exec, official Chamfer, **SigLIP-2 / DINOv3** view similarity, F/IoU | T=0.7, 16k |
+| `heldout_blender` | 103 | bioinspired3d + non-test factories → Blender | executed reference (96 OK) | exec, geometry | greedy, 8k |
+| `heldout_cadquery` | 200 | DeepCAD / Articraft → CadQuery | executed reference (200 OK) | exec, geometry | greedy, 8k |
+| `heldout_openscad` | 50 | Thingiverse → OpenSCAD | executed reference (50 OK) | exec, geometry | **T=0.7**, 8k |
+| `heldout_glsl` | 200 | Shadertoy captions → GLSL | compile (178/200 refs) | compile rate, WebGL render rate | **T=0.7**, 8k |
+| `heldout_threejs` | 40 | scene briefs → single-file three.js | execution (40/40 refs) | exec | **T=0.7**, 12k |
+| `harness_*` (10) | 3–40 | harness batteries (must_have, dimensions) → blender / cadquery / threejs / glsl | none | exec (+ VLM judge via harness) | greedy / T=0.7 |
+
+Tags: `headline` (3dcodebench_text), `text`, `vlm`, `official`, `heldout`, `long-output`, `rubric`, `all` (= everything but `rubric`).
+
+## Quickstart (a new machine: three commands to a report)
+
+```bash
+# 0. environment: conda env cv3d-eval + Blender 5.0.1 + OpenSCAD + glslang + Chromium, no sudo (~15 min, re-runnable)
+bash llm/setup.sh
+export CV3D_EVAL_DATA=~/3dcodeverse_data/3dcodeverse_eval CV3D_TOOLS=~/3dcodeverse_data/tools # defaults
+python -m llm.config # doctor: every tool and asset it can / cannot find
+
+# 1. assets (token with access to ilabai/*: `hf auth login`)
+python -m llm.download --core --views --logs # ~1 GB; --gt-tar adds the canonical GLBs (2.1 GB)
+python -m llm.build_prompts
+SEED=0 python -m llm.build_refs # executes every reference program (~20 min)
+
+# 2. sanity: the references through the whole pipeline (exec ≈ 100 %, [email protected] ≈ 1)
+python -m llm.run_eval --backend reference --run reference --suites all
+
+# 3. a local open model (vLLM in-process; 8B bf16 fits a 24 GB card)
+python -m llm.run_eval --backend vllm --model ~/3dcodeverse_data/models/Qwen3-8B --run qwen3_8b --suites headline heldout
+python -m llm.run_eval --backend vllm --model ... --run qwen3_8b_t07 --suites long-output --samples 4 # pass@k
+
+# 4. any OpenAI-compatible endpoint (vLLM server, OpenRouter, GPT…) or Claude
+vllm serve ~/3dcodeverse_data/models/Qwen3-8B --port 8000 &
+python -m llm.run_eval --backend openai --base-url http://localhost:8000/v1 --model Qwen3-8B --run q8_server --suites all
+ANTHROPIC_API_KEY=… python -m llm.run_eval --backend anthropic --model claude-sonnet-4-6 --run sonnet --suites headline
+
+# 5. quality without ground truth: VLM judge (any OpenAI-compatible VLM endpoint, or Claude)
+python -m llm.run_eval ... --judge-model gpt-5.4 --judge-base-url https://…/v1 # absolute 1-5 after scoring
+python -m llm.judge pairwise --run-a qwen3_8b --run-b official_gpt-5.5 --suite 3dcodebench_official_text --judge-model …
+
+# 6. tables
+python -m llm.report --root $CV3D_EVAL_DATA/out
+```
+
+What is measured, per suite: execution → geometry vs GT ([email protected], Chamfer, IoU) → structural integrity of the
+mesh itself (floating parts, fragmentation, watertight, stability) → official render similarity (SigLIP-2 /
+DINOv3) → optional VLM judge. `docs/metrics.md` defines each; the answer to "is the object good" is [email protected] /
+SigLIP-2 when a GT exists, and the structural fields + judge when it does not.
+
+Outputs: `$CV3D_EVAL_OUT/<run>/<suite>[/sN]/{<id>/{code.*, raw.txt, exec/…}, gens.jsonl, gen_stats.json,
+exec_results.jsonl, metrics.jsonl, summary.json}` and `$CV3D_EVAL_OUT/report.md`.
+
+## Prompt row schema (`data/prompts/*.jsonl`)
+
+```json
+{"id": "AgaveMonocot_seed0", "suite": "3dcodebench_text", "dialect": "blender",
+ "messages": [{"role": "system", "content": "…"}, {"role": "user", "content": "…"}],
+ "images": ["hub/3DCode/3DCodeBench_ModelLogs/inputs/AgaveMonocot_seed0/images/Image_005.png", …], // VLM suites
+ "reference": {"code": "…", "mesh": "refs/3dcodebench/AgaveMonocot_seed0/ref.glb", "renders": ["…Image_005.png", …]},
+ "meta": {"factory": "AgaveMonocot", "caption_type": "instruction", "source": "YipengGao/3DCode/3DCodeBench"}}
+```
+Paths are relative to `$CV3D_EVAL_DATA`. Adding a suite = adding a JSONL in this schema (+ one line in
+`suites.SUITES` for decoding defaults / metrics).
+
+## What the numbers mean
+
+See `docs/metrics.md` (definitions, formulas, pitfalls) and `docs/protocol.md` (the recommended protocol:
+decoding, budgets, noise floor, reporting). Two protocols coexist on purpose:
+
+* **report protocol** (`3dcodebench_text`, `heldout_*`): the exact prompts, executors and metrics of
+ `finetune/docs/REPORT.md`, so every historical number in `llm_finetune_exps.md` stays comparable.
+* **official protocol** (`3dcodebench_official_*`): the prompts, 4-view renderer, SigLIP-2 / DINOv3 and
+ squared unit-sphere Chamfer of the 3DCodeBench paper (arXiv:2606.01057), so a local model can be placed
+ in the paper's frontier-model table. The paper's raw frontier outputs (`download.py --logs`) can be
+ re-scored here to calibrate.
+
+Reference self-test on this machine (2026-09-08, Blender 5.0.1 + scipy 1.17.1 + shapely, OpenSCAD 2026.09.07,
+glslang 16.5, Chromium 151): 3DCodeBench refs 210/212 (ElkhornCoral empty, FanCoral scipy ValueError),
+held-out Blender 96/103, CadQuery 200/200, OpenSCAD 50/50, GLSL 178/200 compile (133 render non-uniform),
+three.js 40/40 — identical to the report's reference counts.
+
+## Reusing the harness judge
+
+The harness batteries (`../bench/prompts/*.yaml`, the ten in `config.HARNESS_BATTERIES`) have no ground truth; their quality score is the calibrated
+Gemini VLM judge in `harness/codeverse/judges` (14-view rig, rubric `static_object_v1`, σ ≈ 0.03 at n=3).
+`docs/inventory_harness_bench.md` §7 shows the minimal call sequence (`FixedEvaluator`, `h2h_glb.py`) to
+judge a GLB produced here without the agentic pipeline.
+
+## Provenance
+
+* prompts/executors/metrics of the report protocol: `docs/inventory_finetune_eval.md`
+* harness batteries and judge: `docs/inventory_harness_bench.md`
+* public benchmarks and metrics literature (130 records): `docs/survey_records.md`, synthesis in `docs/survey_synthesis.md`
diff --git a/harness/tests/bench_prompts/__init__.py b/eval/llm/__init__.py
similarity index 100%
rename from harness/tests/bench_prompts/__init__.py
rename to eval/llm/__init__.py
diff --git a/eval/llm/build_prompts.py b/eval/llm/build_prompts.py
new file mode 100644
index 00000000..5e40c070
--- /dev/null
+++ b/eval/llm/build_prompts.py
@@ -0,0 +1,227 @@
+"""Build the prompt sets under data/prompts/ from the Hub assets (run once after `download.py`).
+
+One JSONL row per task, one schema for every suite::
+
+ {"id": str, unique within the suite (filesystem-safe)
+ "suite": str, e.g. "3dcodebench_text"
+ "dialect": str, blender | cadquery | openscad | glsl | threejs
+ "messages": [{"role": "system"|"user", "content": str}], the chat prompt sent to the model
+ "images": [str], image paths relative to $CV3D_EVAL_DATA (VLM suites only), one <image> tag
+ per image appears in the user message
+ "reference": {"code": str|null, "mesh": str|null, "renders": [str]}, ground truth (code inline; mesh/renders
+ relative to $CV3D_EVAL_DATA, filled by build_refs.py / download.py)
+ "meta": {...}} source ids, caption type, tier, must_have, ...
+
+Suites built here (sizes are what the assets give):
+ 3dcodebench_text 212 Blender prompt_instruction.txt (the report's headline suite)
+ 3dcodebench_desc 212 Blender prompt_description.txt (short caption variant)
+ 3dcodebench_img1/img4 212 Blender 1 / 4 GT reference views (image→code, VLM)
+ 3dcodebench_img4text 212 Blender 4 views + description
+ heldout_blender 103, heldout_cadquery 200, heldout_openscad 50, heldout_glsl 200, heldout_threejs 40
+ (ilabai/3dcodeverse-llamafactory/test/*.parquet, never trained on)
+ harness_<battery> rubric-judged batteries copied from harness/bench/prompts (no GT; scored by
+ execution + optional VLM judge), one suite per YAML
+
+System prompts are the exact strings used in finetune/docs/REPORT.md so numbers stay comparable; the
+held-out sets carry their own system prompt inside the parquet.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import yaml
+
+from . import config
+
+SYSTEM_BENCH = ("You are an expert in procedural 3D modeling with Blender Python (bpy). Given a description of an object, "
+ "write a complete, standalone Blender 5 Python script that builds it from scratch (clear the default scene "
+ "first, create geometry with bpy/bmesh, assign simple materials when relevant, leave the finished mesh objects "
+ "in the scene). Output ONLY the code in one ```python block.")
+SYSTEM_BENCH_IMG = ("You are an expert in procedural 3D modeling with Blender Python (bpy). Given reference renders of an "
+ "object, write a complete, standalone Blender 5 Python script that reproduces it (clear the default scene "
+ "first). Output ONLY the code in one ```python block.")
+SYSTEM_BENCH_IMGTEXT = ("You are an expert in procedural 3D modeling with Blender Python (bpy). Given reference renders and a "
+ "description of an object, write a complete, standalone Blender 5 Python script that builds the requested "
+ "object from scratch (clear the default scene first). Output ONLY the code in one ```python block.")
+
+# one-shot system prompts for the harness batteries (derived from harness/codeverse/prompts/<lang>/contract.md)
+SYSTEM_BATTERY = {
+ "blender": ("You are an expert Blender (bpy) modeller writing raw code. Write ONE complete, standalone Blender 5 Python "
+ "script that builds the requested object from scratch in an emptied scene. Conventions: Z-up, the object's "
+ "front faces -Y, units are metres, the object rests on z=0. Give every mesh object a descriptive unique name "
+ "and a simple material. Do not render, export, import files or access the network. "
+ "Output ONLY the code in one ```python block."),
+ "cadquery": ("You are an expert CAD programmer. Write ONE complete, standalone CadQuery (Python) script that builds the "
+ "requested part and leaves the final solid in a variable named `result`. Units are millimetres unless the "
+ "brief says otherwise. Output ONLY the code in one ```python block."),
+ "threejs": ("You are an expert three.js developer. Write ONE complete, standalone HTML page that renders the requested "
+ "object with three.js (import three from https://cdn.jsdelivr.net/npm/[email protected]/build/three.module.js "
+ "via an importmap; procedural geometry only, no external assets), with lighting, a camera framing the object "
+ "and OrbitControls. Output ONLY the code in one ```html block."),
+ "glsl_shader": ("You are an expert shader programmer. Write ONE complete Shadertoy-style GLSL ES 3.00 fragment shader "
+ "defining `void mainImage(out vec4 fragColor, in vec2 fragCoord)` that renders the requested scene "
+ "(animate with iTime). Output ONLY the code in one ```glsl block."),
+ "urdf_blender": None, # articulated: two-file contract, not a one-shot suite here
+}
+BATTERY_DIALECT = {"blender": "blender", "cadquery": "cadquery", "threejs": "threejs", "glsl_shader": "glsl"}
+
+
+def rel(p: Path) -> str:
+ return str(Path(p).resolve().relative_to(config.DATA_DIR.resolve()))
+
+
+def write_jsonl(rows: list[dict], name: str) -> Path:
+ out = config.PROMPTS_DIR / f"{name}.jsonl"
+ out.parent.mkdir(parents=True, exist_ok=True)
+ with out.open("w") as f:
+ for r in rows:
+ f.write(json.dumps(r, ensure_ascii=False) + "\n")
+ print(f"[prompts] {name}: {len(rows)} rows -> {out}")
+ return out
+
+
+def bench_tasks() -> list[Path]:
+ d = config.BENCH_TASKS_DIR
+ if not d.exists():
+ raise SystemExit(f"3DCodeBench tasks not found at {d}; run download.py first")
+ return sorted(p for p in d.iterdir() if p.is_dir() and (p / "prompt_instruction.txt").exists())
+
+
+def bench_images(task: str) -> list[Path]:
+ """The official 4 GT views (3DCodeBench_ModelLogs/inputs/<task>/images/Image_0{05,15,25,35}.png)."""
+ d = config.HUB_DIR / "3DCode" / "3DCodeBench_ModelLogs" / "inputs" / task / "images"
+ return sorted(d.glob("*.png")) if d.exists() else []
+
+
+def build_3dcodebench() -> None:
+ text, desc, img1, img4, img4text = [], [], [], [], []
+ n_img = 0
+ for t in bench_tasks():
+ task = t.name
+ instr = (t / "prompt_instruction.txt").read_text().strip()
+ descr = (t / "prompt_description.txt").read_text().strip()
+ code = (t / f"{task}.py").read_text() if (t / f"{task}.py").exists() else None
+ ref_mesh = config.REFS_DIR / "3dcodebench" / task / "ref.glb"
+ imgs = bench_images(task)
+ ref = {"code": code, "mesh": rel(ref_mesh) if ref_mesh.exists() else None, "renders": [rel(p) for p in imgs]}
+ meta = {"factory": task.replace("_seed0", ""), "source": "YipengGao/3DCode/3DCodeBench"}
+ ref_nocode = {**ref, "code": None} # the code is stored once, in 3dcodebench_text (see meta.code_from)
+ text.append({"id": task, "suite": "3dcodebench_text", "dialect": "blender",
+ "messages": [{"role": "system", "content": SYSTEM_BENCH}, {"role": "user", "content": instr}],
+ "reference": ref, "meta": {**meta, "caption_type": "instruction"}})
+ desc.append({"id": task, "suite": "3dcodebench_desc", "dialect": "blender",
+ "messages": [{"role": "system", "content": SYSTEM_BENCH}, {"role": "user", "content": descr}],
+ "reference": ref_nocode, "meta": {**meta, "caption_type": "description", "code_from": "3dcodebench_text"}})
+ if imgs:
+ n_img += 1
+ img1.append({"id": task, "suite": "3dcodebench_img1", "dialect": "blender",
+ "messages": [{"role": "system", "content": SYSTEM_BENCH_IMG},
+ {"role": "user", "content": "<image>\nReproduce this 3D object with Blender Python code."}],
+ "images": [rel(imgs[0])], "reference": ref_nocode, "meta": {**meta, "n_images": 1, "code_from": "3dcodebench_text"}})
+ img4.append({"id": task, "suite": "3dcodebench_img4", "dialect": "blender",
+ "messages": [{"role": "system", "content": SYSTEM_BENCH_IMG},
+ {"role": "user", "content": "<image>" * len(imgs[:4]) + "\nThese are reference renders of one object. "
+ "Reproduce this 3D object with Blender Python code."}],
+ "images": [rel(p) for p in imgs[:4]], "reference": ref_nocode, "meta": {**meta, "n_images": len(imgs[:4]), "code_from": "3dcodebench_text"}})
+ img4text.append({"id": task, "suite": "3dcodebench_img4text", "dialect": "blender",
+ "messages": [{"role": "system", "content": SYSTEM_BENCH_IMGTEXT},
+ {"role": "user", "content": "<image>" * len(imgs[:4]) + "\nThese are reference renders of one object. "
+ "Write the Blender Python code that builds it.\n\nThe object: " + descr}],
+ "images": [rel(p) for p in imgs[:4]], "reference": ref_nocode,
+ "meta": {**meta, "n_images": len(imgs[:4]), "caption_type": "description", "code_from": "3dcodebench_text"}})
+ write_jsonl(text, "3dcodebench_text")
+ write_jsonl(desc, "3dcodebench_desc")
+ # official protocol: description prompt under the official raw-python system prompt (configs default
+ # prompt_type: description); image task = the 4 views alone, no text
+ off_dir = config.PKG_DIR / "data" / "official_prompts"
+ sys_text = (off_dir / "text_to_3d_system_prompt.txt").read_text().strip()
+ sys_img = (off_dir / "image_to_3d_system_prompt.txt").read_text().strip()
+ off_text = [{**r, "suite": "3dcodebench_official_text",
+ "messages": [{"role": "system", "content": sys_text}, {"role": "user", "content": r["messages"][1]["content"]}],
+ "meta": {**r["meta"], "protocol": "official", "source": "3dcodebench/prompts/text_to_3d_system_prompt.txt"}}
+ for r in desc]
+ write_jsonl(off_text, "3dcodebench_official_text")
+ if n_img:
+ off_img = [{**r, "suite": "3dcodebench_official_img4",
+ "messages": [{"role": "system", "content": sys_img},
+ {"role": "user", "content": "<image>" * len(r["images"]) + "\nReconstruct the object shown in the following "
+ "reference image(s) as a Blender 5.0 Python script. Treat all images as views of the SAME object."}],
+ "meta": {**r["meta"], "protocol": "official", "source": "3dcodebench/prompts/image_to_3d_system_prompt.txt"}}
+ for r in img4]
+ write_jsonl(off_img, "3dcodebench_official_img4")
+ if n_img:
+ write_jsonl(img1, "3dcodebench_img1")
+ write_jsonl(img4, "3dcodebench_img4")
+ write_jsonl(img4text, "3dcodebench_img4text")
+ else:
+ print("[prompts] no GT views found under 3DCodeBench_ModelLogs/inputs — image suites skipped")
+
+
+def build_heldout() -> None:
+ import pyarrow.parquet as pq
+
+ d = config.LF_HUB / "test"
+ if not d.exists():
+ raise SystemExit(f"held-out parquets not found at {d}; run download.py first")
+ ext = {"blender": "glb", "cadquery": "stl", "openscad": "stl", "glsl": None, "threejs": None}
+ for dialect in ("blender", "cadquery", "openscad", "glsl", "threejs"):
+ p = d / f"{dialect}.parquet"
+ if not p.exists():
+ print(f"[prompts] missing {p}")
+ continue
+ rows = []
+ for r in pq.read_table(p).to_pylist():
+ tid = r["id"].replace("/", "__")
+ ref_mesh = config.REFS_DIR / f"heldout_{dialect}" / tid / f"ref.{ext[dialect]}" if ext[dialect] else None
+ code = r["reference_code"]
+ rows.append({"id": tid, "suite": f"heldout_{dialect}", "dialect": dialect,
+ "messages": [{"role": "system", "content": r["system"]}, {"role": "user", "content": r["prompt"]}],
+ "reference": {"code": code, "mesh": rel(ref_mesh) if ref_mesh and ref_mesh.exists() else None, "renders": []},
+ "meta": {"source_id": r["id"], "source": "ilabai/3dcodeverse-llamafactory/test"}})
+ write_jsonl(rows, f"heldout_{dialect}")
+
+
+def build_batteries() -> None:
+ for y in (config.BATTERIES_DIR / f"{name}.yaml" for name in config.HARNESS_BATTERIES):
+ bat = yaml.safe_load(y.read_text())
+ lang = bat.get("language", "blender")
+ rows = []
+ for p in bat["prompts"]:
+ pl = p.get("language") or lang
+ sysmsg = SYSTEM_BATTERY.get(pl)
+ if sysmsg is None:
+ continue
+ brief = p["prompt"].strip()
+ extra = []
+ if p.get("dimensions_m"):
+ extra.append("Dimensions (m): " + ", ".join(f"{k}={v}" for k, v in p["dimensions_m"].items()))
+ if p.get("must_have"):
+ extra.append("MUST HAVE:\n" + "\n".join(f"- {m}" for m in p["must_have"]))
+ user = brief + ("\n\n" + "\n\n".join(extra) if extra else "")
+ rows.append({"id": p["id"], "suite": f"harness_{bat['name']}", "dialect": BATTERY_DIALECT[pl],
+ "messages": [{"role": "system", "content": sysmsg}, {"role": "user", "content": user}],
+ "reference": {"code": None, "mesh": None, "renders": []},
+ "meta": {"tier": p.get("tier"), "category": p.get("category"), "must_have": p.get("must_have", []),
+ "dimensions_m": p.get("dimensions_m"), "tags": p.get("tags", []), "track": bat.get("track"),
+ "source": f"harness/bench/prompts/{y.name}"}})
+ if rows:
+ write_jsonl(rows, f"harness_{bat['name']}")
+
+
+def main() -> None:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--only", nargs="*", choices=["bench", "heldout", "batteries"], default=None)
+ a = ap.parse_args()
+ which = set(a.only or ["bench", "heldout", "batteries"])
+ if "bench" in which:
+ build_3dcodebench()
+ if "heldout" in which:
+ build_heldout()
+ if "batteries" in which:
+ build_batteries()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/eval/llm/build_refs.py b/eval/llm/build_refs.py
new file mode 100644
index 00000000..dbf0ca33
--- /dev/null
+++ b/eval/llm/build_refs.py
@@ -0,0 +1,112 @@
+"""Build reference meshes by executing the reference programs of every suite that has them.
+
+Writes $CV3D_EVAL_DATA/refs/<suite>/<id>/ref.{glb|stl} plus refs/<suite>/ref_exec.jsonl, then re-runs
+build_prompts so the prompt rows point at the meshes. Executing the reference through the *same* executor
+as the model outputs is also the executor's self-test: the report's sanity figure for 3DCodeBench is
+Chamfer ≈ 0.016 / [email protected] ≈ 1.0 between a re-executed reference and the shipped GT.
+
+For 3DCodeBench the canonical GT is `renders/object.glb` inside `ilabai/3dcodeverse/3dcodebench/factories_geo/
+samples-000.tar`; when that tar is present (`download.py --gt-tar`) the canonical mesh is stored as
+`ref_canonical.glb` and `ref.glb` is the re-executed one — `score.py --ref canonical` switches between them.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import tarfile
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+
+from . import config, executors
+from .extract import extract
+from .suites import load_prompts
+
+MESH_EXT = {"blender": "glb", "cadquery": "stl", "openscad": "stl"}
+
+
+def ref_dir_name(suite: str) -> str:
+ """All 3dcodebench_* suites share one GT set (refs/3dcodebench); held-out suites own theirs."""
+ return "3dcodebench" if suite.startswith("3dcodebench") else suite
+
+
+def _one(row: dict, suite_dir: Path, force: bool) -> dict:
+ d = suite_dir / row["id"]
+ ext = MESH_EXT[row["dialect"]]
+ target = d / f"ref.{ext}"
+ if target.exists() and not force:
+ return {"id": row["id"], "status": "OK", "cached": True, "mesh": str(target)}
+ code = row["reference"]["code"]
+ if "```" in code: # held-out parquets store the reference inside a fence
+ code = extract(code, row["dialect"])[0]
+ rep = executors.run(row["dialect"], code, str(d))
+ if rep.get("mesh"):
+ Path(rep["mesh"]).replace(target)
+ rep["mesh"] = str(target)
+ return {"id": row["id"], **{k: rep.get(k) for k in ("status", "error", "latency_s", "mesh", "n_verts", "n_faces")}}
+
+
+def build(suite: str, workers: int, force: bool, limit: int | None) -> None:
+ rows = [r for r in load_prompts(suite) if r["reference"].get("code") and r["dialect"] in MESH_EXT]
+ if limit:
+ rows = rows[:limit]
+ suite_dir = config.REFS_DIR / ref_dir_name(suite)
+ suite_dir.mkdir(parents=True, exist_ok=True)
+ print(f"[refs] {suite}: executing {len(rows)} reference programs with {workers} workers")
+ res = []
+ with ThreadPoolExecutor(workers) as ex:
+ futs = {ex.submit(_one, r, suite_dir, force): r["id"] for r in rows}
+ for i, fu in enumerate(as_completed(futs), 1):
+ res.append(fu.result())
+ if i % 25 == 0:
+ print(f"[refs] {suite} {i}/{len(rows)}", flush=True)
+ res.sort(key=lambda r: r["id"])
+ with (suite_dir / "ref_exec.jsonl").open("w") as f:
+ for r in res:
+ f.write(json.dumps(r) + "\n")
+ from collections import Counter
+ c = Counter(r["status"] for r in res)
+ print(f"[refs] {suite}: {dict(c)} OK={c.get('OK', 0)}/{len(res)}")
+
+
+def extract_canonical(tar_path: Path) -> None:
+ """Pull renders/object.glb for every factory out of the 3dcodebench factories_geo tar."""
+ import pyarrow.parquet as pq
+
+ meta = pq.read_table(config.CV3D_HUB / "3dcodebench" / "factories_geo" / "metadata.parquet").to_pylist()
+ want = {m["key"].replace("Factory_geo", "") + "_seed0": m for m in meta} # AgaveMonocotFactory_geo -> AgaveMonocot_seed0
+ tasks = {r["id"] for r in load_prompts("3dcodebench_text")}
+ n = 0
+ with tarfile.open(tar_path) as tf:
+ for member in tf:
+ if not member.name.endswith("object.glb"):
+ continue
+ key = member.name.split("/")[0] if "/" in member.name else ""
+ task = key.replace("Factory_geo", "") + "_seed0"
+ if task not in tasks:
+ continue
+ out = config.REFS_DIR / "3dcodebench" / task / "ref_canonical.glb"
+ out.parent.mkdir(parents=True, exist_ok=True)
+ out.write_bytes(tf.extractfile(member).read())
+ n += 1
+ print(f"[refs] canonical GT extracted for {n}/{len(tasks)} tasks (of {len(want)} factories in the tar)")
+
+
+def main() -> None:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--suites", nargs="*", default=["3dcodebench_text", "heldout_blender", "heldout_cadquery", "heldout_openscad"])
+ ap.add_argument("--workers", type=int, default=8)
+ ap.add_argument("--force", action="store_true")
+ ap.add_argument("--limit", type=int, default=None)
+ ap.add_argument("--canonical-tar", type=Path, default=None, help="3dcodebench/factories_geo/samples-000.tar")
+ a = ap.parse_args()
+ for s in a.suites:
+ build(s, a.workers, a.force, a.limit)
+ if a.canonical_tar:
+ extract_canonical(a.canonical_tar)
+ from .build_prompts import build_3dcodebench, build_heldout
+ build_3dcodebench()
+ build_heldout()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/eval/llm/config.py b/eval/llm/config.py
new file mode 100644
index 00000000..b64b6976
--- /dev/null
+++ b/eval/llm/config.py
@@ -0,0 +1,122 @@
+"""Paths and tool discovery for 3dcodeverse_eval.
+
+Nothing in this package hardcodes a machine. Every location is resolved here, from environment
+variables first and sensible defaults second:
+
+ CV3D_EVAL_DATA root for downloaded assets, reference meshes and run outputs
+ (default ~/3dcodeverse_data/3dcodeverse_eval)
+ CV3D_EVAL_OUT run outputs (default $CV3D_EVAL_DATA/out)
+ CV3D_TOOLS directory holding blender-5.0.1-linux-x64/, openscad/, glslang/
+ (default ~/3dcodeverse_data/tools)
+ CV3D_BLENDER, CV3D_OPENSCAD, CV3D_GLSLANG, CV3D_CQ_PYTHON explicit binaries
+ CV3D_XLIBS extra shared-library dir prepended to LD_LIBRARY_PATH for Blender/OpenSCAD
+ (default ~/.local/xlibs/usr/lib/x86_64-linux-gnu, where libSM/libICE/libOpenGL
+ were unpacked on a box without sudo)
+ PLAYWRIGHT_BROWSERS_PATH honoured by Playwright itself
+
+`python -m llm.config` prints a doctor report.
+"""
+from __future__ import annotations
+
+import os
+import shutil
+import sys
+from pathlib import Path
+
+PKG_DIR = Path(__file__).resolve().parent
+PROMPTS_DIR = PKG_DIR / "data" / "prompts"
+# the harness batteries are ONE set of files, owned by the harness evaluation next door (eval/bench/prompts);
+# these ten are the ones a bare LLM can be asked one-shot (static + graphics, no plan, no tools)
+BATTERIES_DIR = PKG_DIR.parent / "bench" / "prompts"
+HARNESS_BATTERIES = ("articulated_v2", "compare_v4", "complexity_v3", "fancy_v1_blender", "fancy_v1_cadquery",
+ "fancy_v1_threejs", "graphics_v2", "static_objects_v1", "static_objects_v2", "teaser_v1_static")
+
+DATA_DIR = Path(os.environ.get("CV3D_EVAL_DATA", "~/3dcodeverse_data/3dcodeverse_eval")).expanduser()
+HUB_DIR = DATA_DIR / "hub" # snapshot_download targets (ilabai/*, YipengGao/3DCode)
+REFS_DIR = DATA_DIR / "refs" # reference meshes / renders per suite
+OUT_DIR = Path(os.environ.get("CV3D_EVAL_OUT", DATA_DIR / "out")).expanduser()
+TOOLS_DIR = Path(os.environ.get("CV3D_TOOLS", "~/3dcodeverse_data/tools")).expanduser()
+XLIBS = Path(os.environ.get("CV3D_XLIBS", "~/.local/xlibs/usr/lib/x86_64-linux-gnu")).expanduser()
+
+BENCH_TASKS_DIR = HUB_DIR / "3DCode" / "3DCodeBench" # 212 <task>/{<task>.py, prompt_*.txt}
+CV3D_HUB = HUB_DIR / "3dcodeverse" # ilabai/3dcodeverse partial snapshot
+LF_HUB = HUB_DIR / "3dcodeverse-llamafactory" # ilabai/3dcodeverse-llamafactory (test/*.parquet)
+
+
+def _first_existing(*cands: str | Path | None) -> Path | None:
+ for c in cands:
+ if not c:
+ continue
+ p = Path(c).expanduser()
+ if p.exists():
+ return p
+ return None
+
+
+def find_blender() -> Path | None:
+ return _first_existing(
+ os.environ.get("CV3D_BLENDER"),
+ "~/.local/bin/blender-5.0",
+ TOOLS_DIR / "blender-5.0.1-linux-x64" / "blender",
+ shutil.which("blender"),
+ )
+
+
+def find_openscad() -> Path | None:
+ return _first_existing(
+ os.environ.get("CV3D_OPENSCAD"),
+ TOOLS_DIR / "openscad" / "AppRun",
+ TOOLS_DIR / "openscad.AppImage",
+ shutil.which("openscad"),
+ )
+
+
+def find_glslang() -> Path | None:
+ return _first_existing(
+ os.environ.get("CV3D_GLSLANG"),
+ TOOLS_DIR / "glslang" / "bin" / "glslang",
+ TOOLS_DIR / "glslang" / "bin" / "glslangValidator",
+ shutil.which("glslang"),
+ shutil.which("glslangValidator"),
+ )
+
+
+def cadquery_python() -> str:
+ """Interpreter that has `cadquery` importable (default: the current one)."""
+ return os.environ.get("CV3D_CQ_PYTHON") or sys.executable
+
+
+def tool_env(**extra: str) -> dict[str, str]:
+ """Environment for Blender / OpenSCAD subprocesses: private HOME, extra libs, no audio."""
+ env = dict(os.environ)
+ if XLIBS.exists():
+ env["LD_LIBRARY_PATH"] = f"{XLIBS}:{env.get('LD_LIBRARY_PATH', '')}"
+ env.update(extra)
+ return env
+
+
+def doctor(verbose: bool = True) -> dict[str, object]:
+ rep: dict[str, object] = {
+ "data_dir": str(DATA_DIR),
+ "out_dir": str(OUT_DIR),
+ "bench_tasks": BENCH_TASKS_DIR.exists() and len([p for p in BENCH_TASKS_DIR.iterdir() if p.is_dir()]),
+ "heldout_parquets": sorted(p.name for p in (LF_HUB / "test").glob("*.parquet")) if (LF_HUB / "test").exists() else [],
+ "blender": str(find_blender()),
+ "openscad": str(find_openscad()),
+ "glslang": str(find_glslang()),
+ "cadquery_python": cadquery_python(),
+ }
+ for mod in ("cadquery", "trimesh", "scipy", "playwright", "vllm", "openai", "pyarrow", "PIL"):
+ try:
+ __import__(mod)
+ rep[f"py:{mod}"] = "ok"
+ except Exception as e: # noqa: BLE001
+ rep[f"py:{mod}"] = f"MISSING ({type(e).__name__})"
+ if verbose:
+ for k, v in rep.items():
+ print(f"{k:20s} {v}")
+ return rep
+
+
+if __name__ == "__main__":
+ doctor()
diff --git a/eval/llm/data/official_prompts/image_judge.txt b/eval/llm/data/official_prompts/image_judge.txt
new file mode 100644
index 00000000..f5f3bdff
--- /dev/null
+++ b/eval/llm/data/official_prompts/image_judge.txt
@@ -0,0 +1,36 @@
+You are an impartial judge in a 3D-modeling arena. Two anonymous AI systems were
+each given the SAME prompt and asked to produce a 3D object as a Blender Python
+script. We rendered the resulting 3D object from four canonical viewpoints
+(front, side, back, top-ish). Your job is to decide which side better matches
+the prompt, judging only from the renders.
+
+You will see, in this order:
+ 1. The original prompt that was given to both systems.
+ 2. (only for image-to-3D prompts) the reference image(s) that the systems
+ were instructed to reconstruct.
+ 3. Four renders of System A's 3D object.
+ 4. Four renders of System B's 3D object.
+
+Verdict options:
+ "a" — A clearly matches the prompt better than B
+ "b" — B clearly matches the prompt better than A
+ "tie" — Both match the prompt roughly equally well
+ "both_bad" — Neither output is recognizable as the requested object,
+ or both are so broken that the comparison is meaningless
+
+Judge on these criteria, weighted in order:
+ 1. Object identity — is it recognizably the requested object at all?
+ 2. Structural correctness — major parts present, in the right place,
+ in plausible proportion.
+ 3. Geometric detail — shape fidelity, surface plausibility.
+ 4. For image-to-3D: faithfulness to the reference images (silhouette,
+ proportions, distinctive features). Color/material is NOT a criterion —
+ the renders are untextured grey.
+
+Do NOT reward "more polygons" or "more visual flair" if the underlying object
+is wrong. A clean, simple but correct shape beats a busy but unrecognizable mess.
+
+Output STRICTLY a single JSON object on one line, no markdown fences, no extra
+text:
+
+{"winner": "a"|"b"|"tie"|"both_bad", "reasoning": "<one to three sentences>"}
diff --git a/eval/llm/data/official_prompts/image_to_3d_system_prompt.txt b/eval/llm/data/official_prompts/image_to_3d_system_prompt.txt
new file mode 100644
index 00000000..9c16c3c2
--- /dev/null
+++ b/eval/llm/data/official_prompts/image_to_3d_system_prompt.txt
@@ -0,0 +1,52 @@
+You are a procedural 3D modeling expert. Given one or more reference images of a 3D object, you produce a single self-contained Python script that reconstructs the depicted object in Blender 5.0.
+
+# Input format
+
+You will receive one or more reference images of a single target object as part of the user message. The images may be:
+- A single view (front, side, 3/4, etc.) — infer unseen sides by symmetry and category priors.
+- Multiple views of the SAME object (e.g. front + side + back, or turntable frames). Treat them as multi-view evidence of one object, not as separate objects. Cross-reference views to resolve depth, proportions, and occluded structure.
+- A mix of full-object shots and close-up detail crops. Use the close-ups to refine local geometry (handles, vents, ornament) of the same object.
+
+If the images appear to depict different objects, model the most prominent / first-shown object and silently ignore the rest.
+
+Read the images carefully before writing code. Identify the object's category, overall proportions (length:width:height), part decomposition, symmetries (bilateral, radial, none), repeating structures (slats, ribs, teeth, scales, leaves), and any distinctive ornament. Your script should reproduce these as real geometry.
+
+# Output format (HARD constraint — read carefully)
+
+Your entire response will be saved verbatim into a `.py` file and executed by Blender 5.0. ANY non-Python content anywhere in the response will break that file. Treat this as a strict machine-to-machine contract, not a chat reply.
+
+Your response MUST consist of nothing but Python source code. Specifically:
+
+- Do NOT emit Markdown code fences anywhere — no opening ```python, no opening ```py, no opening ```, no closing ``` at the end. Not even one. Not even on a single line by itself.
+- Do NOT prepend any prose ("Here is the code", "I will create", "This script generates", "Sure", "Looking at the image", etc.).
+- Do NOT append any prose ("Hope this helps", "Let me know if", "Note that…", etc.).
+- Do NOT include HTML/XML tags, bullet points, headings, or any formatting other than plain Python.
+- Do NOT describe what you see in the image as prose; if you want to record observations, put them in Python comments inside the script.
+- The very first character of your response must be the first character of valid Python source (typically `i` of `import bpy`, or `#` of a top-level comment, or `"` of a module docstring).
+- The very last character of your response must be the last character of the Python script (a closing parenthesis, a bare newline at end of file, etc.) — never a backtick.
+
+Self-check before answering: if I save your response to `out.py` and run `python -c "import ast; ast.parse(open('out.py').read())"`, it must succeed without modification.
+
+# Target environment
+
+- Blender version: **5.0**. Use the Blender 5.0 Python API (`bpy`, `bmesh`, `mathutils`).
+- The script will be executed via `blender --background --python <file>` or pasted into the Blender Text Editor.
+- Allowed libraries (these are the ONLY libraries you may import):
+ - Blender-bundled: `bpy`, `bmesh`, `mathutils`
+ - Python stdlib: `math`, `random`, `itertools`, `collections`, `functools`, `dataclasses`, `enum`, `typing`
+ - Numerical: `numpy`, `scipy`
+- Do NOT import anything that requires network access, GUI interaction, or external file reads (no `os`, `sys.path` hacks, `requests`, `PIL`, `cv2`, etc.). In particular, the reference images are NOT available to the script at runtime — do not attempt to load, open, or read them from disk; encode everything you inferred from the images directly into the geometry your script builds.
+
+# Code requirements
+
+- Produce ONE final 3D object (or coherent assembly) that corresponds to the object shown in the reference images.
+- Generate ONLY the depicted object — NO ground plane, NO backdrop, NO skybox, NO environmental props, NO decorative context, even if the reference images show a background, a floor, a stand, a hand holding the object, or other surrounding scenery. Model only the object itself, sitting at the origin.
+- Match the reference as faithfully as the geometry allows: overall silhouette, part proportions, count and placement of repeating elements (e.g. number of legs, number of petals, number of slats), and characteristic curvature. When the images give clear quantitative cues (e.g. "8 spokes", "3 drawers"), reproduce those counts exactly rather than approximating.
+- Push for as much geometric detail as the images warrant. If the reference shows ribs, slats, vents, handles, fins, scales, leaves, rivets, pleating, segmentation, or ornament, model them as real geometry rather than as flat surfaces with a label. Use subdivision surface, bevel, array, mirror, and screw modifiers when they are the right tool, and drop into `bmesh` for finer features. Aim for high fidelity but keep meshes within a few hundred thousand vertices — the renderer has a 240s budget per script.
+- Prefer procedural construction: parametric loops, `bmesh` operators, modifiers, and array/mirror operations. Avoid hard-coded long vertex lists.
+- Exploit symmetry when the reference supports it (mirror modifier for bilateral objects, array/spin for radial ones); this both saves vertices and makes the result cleaner.
+- At the start of the script, clear the default scene (delete the default cube, camera, and light if present) so that the output scene contains only your generated geometry.
+- Leave geometry untextured. Do NOT try to reproduce the colors, materials, or surface texture seen in the reference — geometry only. Ignore lighting, shadows, and photographic artifacts in the images; they are not part of the object.
+- No need to save the `.blend` file. Do NOT trigger a render. Do NOT call `sys.exit` or `bpy.ops.wm.quit_blender`.
+- If the images are ambiguous about back-side details, internal structure, or absolute scale, choose reasonable defaults silently (consistent with the visible views and the object category) and proceed. Never ask clarifying questions.
+- The script must terminate normally; final geometry must exist in `bpy.data.objects` when execution completes.
diff --git a/eval/llm/data/official_prompts/text_to_3d_system_prompt.txt b/eval/llm/data/official_prompts/text_to_3d_system_prompt.txt
new file mode 100644
index 00000000..98945f3b
--- /dev/null
+++ b/eval/llm/data/official_prompts/text_to_3d_system_prompt.txt
@@ -0,0 +1,38 @@
+You are a procedural 3D modeling expert. Given a natural-language description of a 3D object, you produce a single self-contained Python script that constructs the described object in Blender 5.0.
+
+# Output format (HARD constraint — read carefully)
+
+Your entire response will be saved verbatim into a `.py` file and executed by Blender 5.0. ANY non-Python content anywhere in the response will break that file. Treat this as a strict machine-to-machine contract, not a chat reply.
+
+Your response MUST consist of nothing but Python source code. Specifically:
+
+- Do NOT emit Markdown code fences anywhere — no opening ```python, no opening ```py, no opening ```, no closing ``` at the end. Not even one. Not even on a single line by itself.
+- Do NOT prepend any prose ("Here is the code", "I will create", "This script generates", "Sure", etc.).
+- Do NOT append any prose ("Hope this helps", "Let me know if", "Note that…", etc.).
+- Do NOT include HTML/XML tags, bullet points, headings, or any formatting other than plain Python.
+- The very first character of your response must be the first character of valid Python source (typically `i` of `import bpy`, or `#` of a top-level comment, or `"` of a module docstring).
+- The very last character of your response must be the last character of the Python script (a closing parenthesis, a bare newline at end of file, etc.) — never a backtick.
+
+Self-check before answering: if I save your response to `out.py` and run `python -c "import ast; ast.parse(open('out.py').read())"`, it must succeed without modification.
+
+# Target environment
+
+- Blender version: **5.0**. Use the Blender 5.0 Python API (`bpy`, `bmesh`, `mathutils`).
+- The script will be executed via `blender --background --python <file>` or pasted into the Blender Text Editor.
+- Allowed libraries (these are the ONLY libraries you may import):
+ - Blender-bundled: `bpy`, `bmesh`, `mathutils`
+ - Python stdlib: `math`, `random`, `itertools`, `collections`, `functools`, `dataclasses`, `enum`, `typing`
+ - Numerical: `numpy`, `scipy`
+- Do NOT import anything that requires network access, GUI interaction, or external file reads (no `os`, `sys.path` hacks, `requests`, `PIL`, `cv2`, etc.).
+
+# Code requirements
+
+- Produce ONE final 3D object (or coherent assembly) that corresponds to the description.
+- Generate ONLY the described object — NO ground plane, NO backdrop, NO skybox, NO environmental props, NO decorative context. No grass under the chair, no pedestal beneath the figurine, no "studio floor" plane. After your script finishes, the scene must contain only your geometry, sitting at the origin.
+- Push for as much geometric detail as the description warrants. If the prompt mentions ribs, slats, vents, handles, fins, scales, leaves, rivets, pleating, segmentation, or ornament, model them as real geometry rather than as flat surfaces with a label. Use subdivision surface, bevel, array, mirror, and screw modifiers when they are the right tool, and drop into `bmesh` for finer features. Aim for high fidelity but keep meshes within a few hundred thousand vertices — the renderer has a 240s budget per script.
+- Prefer procedural construction: parametric loops, `bmesh` operators, modifiers, and array/mirror operations. Avoid hard-coded long vertex lists.
+- At the start of the script, clear the default scene (delete the default cube, camera, and light if present) so that the output scene contains only your generated geometry.
+- Leave geometry untextured.
+- No need to save the `.blend` file. Do NOT trigger a render. Do NOT call `sys.exit` or `bpy.ops.wm.quit_blender`.
+- If the description is ambiguous on dimensions, proportions, or stylistic details, choose reasonable defaults silently and proceed. Never ask clarifying questions.
+- The script must terminate normally; final geometry must exist in `bpy.data.objects` when execution completes.
diff --git a/eval/llm/docs/inventory_finetune_eval.md b/eval/llm/docs/inventory_finetune_eval.md
new file mode 100644
index 00000000..a1842945
--- /dev/null
+++ b/eval/llm/docs/inventory_finetune_eval.md
@@ -0,0 +1,135 @@
+# Inventory: the existing `finetune/eval` + `scripts` evaluation code
+
+Read-only audit of `/home/yipeng/3dcodeverse/finetune/` (2026-09-08). This is the reference for what
+`3dcodeverse_eval/` re-implements portably. All paths relative to `finetune/` unless absolute.
+
+## 1. Evaluation suites
+
+### 1.1 Shared conventions
+
+- Prompt jsonl row (all text suites): `{"task": <str>, "messages": [{"role":"system",...},{"role":"user",...}]}`. Image suites add `"images": [abs paths]` and one `<image>` per image in the user text (`eval/generate_vllm_img.py:3-4`).
+- Task id for held-out sets: `r["id"].replace("/", "__")`, messages truncated to `[:2]` (system + user; the assistant reference is dropped) — `eval/eval_all.sh:47-50`, `eval/run_dialect_eval.sh:11-13`, `scripts/threejs_check.sh:19-21`.
+- Every held-out set row in `data/multidialect/<dialect>/test.jsonl` is `{"messages":[system,user,assistant("```<fence>\n<code>\n```")], "dialect", "subset", "id"}` (`scripts/build_multidialect.py:66-70`).
+- Reference code is recovered from the assistant turn with the **legacy** longest-fenced-block rule (`eval/dialect_eval.py:13-14`, `eval/blender_dialect_eval.py:9-10`).
+- `scripts/build_hf_dataset.py:119-129` exports every `test.jsonl` to `test/<dialect>.parquet` with columns `id, system, prompt, reference_code` — 593 rows total (103+200+50+200+40). This is the cleanest portable source of the five held-out sets (now on the Hub: `ilabai/3dcodeverse-llamafactory/test/*.parquet`).
+
+### 1.2 3DCodeBench (212 tasks, text → Blender-Python)
+
+| item | value | ref |
+|---|---|---|
+| Prompt file | `data/sft_v1/bench_prompts.jsonl` | `eval/run_eval.sh:8-9`, `eval/eval_all.sh:43` |
+| Built by | `for d in sorted(os.listdir(BENCH)): prompt = open(f"{BENCH}/{d}/prompt_instruction.txt").read().strip()` | `scripts/prep_data.py:176-180` |
+| BENCH dir | `/wekafs/ict/hx_624/data/3dcodebench/data/<task>/` with `prompt_instruction.txt`, `prompt_description.txt`, `glb/<task>.glb` (GT), `images/*.png` (GT renders, ≥4 views) | `prep_data.py:69`, `metrics.py:8,47` |
+| Task name | directory name, e.g. `AgaveMonocot_seed0`; factory name = name minus `_seed0` | `prep_data.py:70` |
+| System prompt (exact) | `"You are an expert in procedural 3D modeling with Blender Python (bpy). Given a description of an object, write a complete, standalone Blender 5 Python script that builds it from scratch (clear the default scene first, create geometry with bpy/bmesh, assign simple materials when relevant, leave the finished mesh objects in the scene). Output ONLY the code in one ```python block."` | `prep_data.py:14-17` |
+| User prompt | verbatim `prompt_instruction.txt`. Alternate suite uses `prompt_description.txt` | `prep_data.py:178` |
+| Size | 212 = number of dirs in BENCH; `metrics.py:43` iterates the BENCH listing, so tasks with no generation become `exec: "MISSING"` and count in the denominator | `metrics.py:43-46,65` |
+| Leakage control | training drops factories whose `name.replace("Factory","")` is in the 212 test names (`prep_data.py:85-86`, `build_multidialect.py:33,36`); `3dcodebench/instances_*` excluded entirely (`build_md_max.py:12`) | |
+| Sanity | reference scripts through the harness: Chamfer ≈ 0.016, [email protected] ≈ 1.0 (REPORT §5) | |
+
+### 1.3 Blender held-out (103)
+
+- Source: all `bioinspired3d/*/metadata.parquet` plus `3dcodebench/*/metadata.parquet` filtered to non-test factories (`build_multidialect.py:34-36`).
+- Prompt: `caps.instruction` if present, else `"Create the 3D object described below using Python Blender code: " + caps.detailed` (line 25).
+- System: `"You are an expert in procedural 3D modeling with Blender Python (bpy). Given a description of an object, write a complete, standalone Blender 5 Python script that builds it from scratch (clear the default scene first). Output ONLY the code in one ```python block."` (line 9).
+- QC (`build_multidialect.py:48-56`): drop empty code, `<3` newlines, control chars, prompt `<20` chars, sha1-duplicate code. Seed `random.seed(7)`; `n_test = min(200, max(50, len(keep)//50))`. Reference exec: 96/103 OK.
+
+### 1.4 CadQuery (200)
+
+- Source: `deepcad/*` + `articraft/cadquery_single_tex*`. Prompt: `"Write a CadQuery (Python) script that builds the following CAD model: " + (instruction or detailed)`.
+- System: `"You are an expert CAD programmer. Given a description of a part, write a complete, standalone CadQuery (Python) script that builds it and leaves the final solid in a variable named `result`. Output ONLY the code in one ```python block."` Reference exec 199/200.
+
+### 1.5 OpenSCAD (50)
+
+- Source: `thingiverse/openscad`. Prompt: `detailed or instruction` verbatim. System: `"You are an expert OpenSCAD programmer. Given a request, write a complete, standalone OpenSCAD (.scad) program that produces the described model. Output ONLY the code in one ```openscad block."` Reference exec 50/50.
+
+### 1.6 GLSL (200)
+
+- Source: `shadertoy/*` main image pass only. Prompt: `instruction` if it starts with write/create/implement/make, else `"Write a GLSL fragment shader for Shadertoy that renders: " + (instruction or detailed)`.
+- System: `"You are an expert shader programmer. Given a description, write a complete Shadertoy-style GLSL ES 3.00 fragment shader defining `void mainImage(out vec4 fragColor, in vec2 fragCoord)`. Output ONLY the code in one ```glsl block."` Reference compile 178/200. Metric is compile-only.
+
+### 1.7 three.js (40)
+
+- `data/multidialect/threejs/test.jsonl` (280/20/40 split of `threejs_distill`; the split builder is not in the repo). Fence ```html. Reference exec 40/40. Historical budgets 12288 → 32768.
+
+### 1.8 Image-conditioned suites (3DCodeBench renders → Blender-Python)
+
+| file | images | system | user |
+|---|---|---|---|
+| `_img_bench_prompts.jsonl` | first render | `"...Given reference renders of an object, write a complete, standalone Blender 5 Python script that reproduces it (clear the default scene first). Output ONLY the code in one ```python block."` | `"<image>\nReproduce this 3D object with Blender Python code."` |
+| `_img4_bench_prompts.jsonl` | 4 renders | same | `"<image>"*4 + "\nReproduce this 3D object with Blender Python code."` |
+| `_img4text_bench_prompts.jsonl` | 4 renders + caption | (builder missing; inferred from `build_mm_dataset.py:33,48`) | `"<image>"*4 + "\nThese are reference renders of one object. Write the Blender Python code that builds it.\n\nThe object: " + caption` |
+
+Scoring identical to the text bench, so image and text numbers are directly comparable.
+
+## 2. Executors
+
+### 2.1 Blender (`eval/run_bench.py` + `eval/blender_runner.py`)
+- `BLENDER -b --factory-startup -noaudio --python blender_runner.py -- --script code.py --out out.glb --report exec.json`; env `HOME=/tmp`, `XDG_CONFIG_HOME=/tmp/.bcfg`; workers 16–32, timeout 300 s.
+- Statuses: OK / EMPTY / FAIL (from report) ; CRASH (no report) ; TIMEOUT.
+- Inside Blender: delete every object, purge orphan data (so the default cube cannot pass), `runpy.run_path`, then count depsgraph-evaluated MESH objects → EMPTY if none / no verts; export GLB (`export_yup=True`, `export_apply=True`).
+- Pitfalls: scipy must be installed in Blender's bundled python (15/212 refs import it); only MESH objects count; no sandbox; `blender_dialect_eval.py` uses timeout 180 and a different env.
+
+### 2.2 CadQuery (`eval/dialect_runners.py:6-60`)
+- Subprocess with a runner that monkeypatches `cq.exporters.export`, injects `show_object`/`debug`, finds the result object (captured export → named globals `result, final, model, part, ...` → any), `Assembly→toCompound`, STL export with `tolerance=0.01`. `CQ_REPORT {json}` line in stdout. Timeout 90 s. EMPTY when volume ≤ 1e-9 and no faces.
+
+### 2.3 OpenSCAD — `openscad -o out.stl --export-format binstl model.scad`, `QT_QPA_PLATFORM=offscreen`, timeout 120 s; OK iff rc==0 and STL > 100 bytes.
+
+### 2.4 GLSL compile — strip `#version`/`precision`/uniform/`out vec4` lines, wrap with a Shadertoy prelude (`#version 300 es`, uniforms, `void main`), `glslangValidator -S frag`. Prelude lacks `iFrameRate` (render harness has it).
+
+### 2.5 GLSL render (`eval/glsl_render.py`) — WebGL2 in headless Chromium (swiftshader), 320×240, frames at t=0 and 1.7 s; OK if distinct colours > 3 and std > 2, else STATIC. Not part of any suite historically.
+
+### 2.6 three.js (`eval/threejs_runner.py`) — Playwright Chromium (angle/swiftshader), 640×480, `file://` load, 4 s settle, collect pageerror/console.error/requestfailed; blank decision from a screenshot (`distinct > 6 and std > 4`). Depends on CDN import of three.js (network).
+
+## 3. Metrics (`eval/metrics.py`)
+- Normalise: centre at bbox centre, scale so max extent = 1; `sample_surface(n)` (bench 10 000, held-out 5 000), `np.random.seed(0)`.
+- `chamfer = mean(d_a→b) + mean(d_b→a)` (sum, L2, not squared); `F@τ = 2PR/(P+R)` with `P=frac(d_a→b<τ)`, `R=frac(d_b→a<τ)`, τ ∈ {0.05, 0.1}.
+- Rotation search: gen points rotated by k·90° about Y (glTF up), keep min-Chamfer.
+- `summary.json`: `n_tasks, exec_ok, exec_ok_rate, n_scored, chamfer_mean_scored, chamfer_median_scored, [email protected]_mean_scored, [email protected]_mean_scored, [email protected]_mean_all(fail=0), [email protected]_mean_all(fail=0)`.
+- pass@N / best-of-N (`pass_at_n.py`): pass@1 = mean OK over samples; pass@N = tasks with ≥1 OK; best-of-N F = mean over tasks of max_i F (fail = 0).
+- Extraction stats (`extract.summarize`): `n, with_think, no_fence, from_think_fallback, multi_block, chosen_not_last, syntax_valid, no_dialect_marker, empty`.
+
+## 4. Code extraction (`eval/extract.py`)
+- Strip closed `<think>` blocks; unclosed → truncate at tag, mine code from the think text as fallback.
+- Candidates: every fence + one trailing unterminated fence; no fence → from first code-like line.
+- Score: +3 per dialect marker (max 2), +2 fence-lang match, +2 syntax OK; penalties for shell/pip lines, JSON, tracebacks, < 40 chars. Tie → later block, then longer.
+- Known bug: the shell-prompt penalty `^\s*[$#>]\s+\w` also fires on a python script whose first line is `# comment`.
+
+## 5. Generation layer
+- `generate_vllm.py`: `LLM(dtype=bf16, max_model_len 12288, enable_prefix_caching)`, `llm.chat(messages, chat_template_kwargs={"enable_thinking": False})`, `SamplingParams(temperature, top_p=0.95 if T>0 else 1.0, max_tokens, seed)`; output `<out>/<task>/code.py`, `raw.txt`, `gens_shard0.jsonl` (`task, n_new_tokens, finished, has_code_block, extract`), `extract_stats.json`.
+- `generate_multi.py`: one engine for many suites (`--spec` list of `{name,prompts,out,dialect}`), adds `truncated`.
+- `generate_vllm_img.py`: `limit_mm_per_prompt={"image":4}`, images as JPEG data-URLs (thumbnail 768), thinking always off.
+
+## 6. Zero-shot baselines to reproduce against
+
+| model | suite | decode | exec | notes |
+|---|---|---|---|---|
+| Qwen3.5-9B | 3DCodeBench 212 | greedy, no-think, 6144 | **0/212 = 0.0%** | 165/212 start with hallucinated `bpy.ops.scene.clear()`; lenient patch → 4/212 |
+| Qwen3.5-9B | 3DCodeBench | greedy, thinking | 1/212 | |
+| Qwen3.5-9B | Blender 103 | greedy nt / think | 1% / 14.6% | |
+| Qwen3.5-9B | CadQuery 200 | greedy nt / think | 10% ([email protected] all 0.023) / 33% | |
+| Qwen3.5-9B | OpenSCAD 50 | greedy nt / think | 42–46% / 84% | |
+| Qwen3.5-9B | GLSL 200 | greedy nt / think | 39% / 35% | compile only |
+| Qwen3.5-9B | three.js 40 | greedy nt / T0.7 / think | 77.5% / 67.5% / 87.5% | |
+| Qwen3.8-27B | 3DCodeBench | greedy nt | 61.8%, [email protected](all) 0.257 | thinking@12k 33.5% (truncation) |
+| Qwen2.5-Coder-7B-Instruct | 3DCodeBench | greedy, HF, 6144 | 8.5%, CD 0.312, [email protected](ok) 0.231 | |
+| Qwen3-8B | any suite zero-shot | — | **not measured** | LoRA-v1: 76.4% |
+
+Protocol notes: budgets drifted 6144 → 8192 → 12288 → 32768; REPORT §13.4 mandates reporting greedy **and** T=0.7 for OpenSCAD/GLSL/three.js; within-session noise floor: bench/Blender 0.0 pp, CadQuery 0.5, GLSL 1.5, OpenSCAD 2.0, three.js 5.0 pp; across sessions up to 6–8 pp on OpenSCAD.
+
+## 7. Hardcoded `/wekafs/ict/hx_624/...` (parameterised in the new package)
+Blender binary, OpenSCAD AppRun, glslangValidator, llmft python, ms-playwright cache, 3DCodeBench GT dir, `eval/out`, `runs`, conda activation, secrets, model dirs.
+
+## 8. Bugs / inconsistencies found
+1. Held-out Blender timeout 180 vs bench 300; held-out CRASH has no error text.
+2. `metrics.py:63` comment describes an unimplemented Chamfer penalty.
+3. `[email protected]_mean_all` in the held-out evals counts reference failures against the model.
+4. `GLSL_PRELUDE` lacks `iFrameRate`.
+5. `reextract.py`/`generate.py` use the legacy longest-block extractor.
+6. `metrics.py` derives the task list from the GT dir listing, so `--limit` runs report MISSING.
+7. Blender EMPTY ignores non-MESH objects that evaluate to geometry.
+8. three.js TIMEOUT detected by substring; CDN dependence.
+9. Decoding budget drift; image generator cannot enable thinking.
+10. Three different Blender system prompts across suites.
+11. Missing builders: three.js split, `_img4text` prompts.
+12. Pins: Blender 5.0.1 (+scipy), OpenSCAD ≥ 2026.08 (`--export-format binstl`), cadquery 2.8.0, glslang 16.5, playwright + chromium, trimesh, scipy.
diff --git a/eval/llm/docs/inventory_harness_bench.md b/eval/llm/docs/inventory_harness_bench.md
new file mode 100644
index 00000000..a0b255ee
--- /dev/null
+++ b/eval/llm/docs/inventory_harness_bench.md
@@ -0,0 +1,65 @@
+# Inventory: the `harness/bench` evaluation layer (what is reusable for LLM/VLM eval)
+
+Read-only audit of `/home/yipeng/3dcodeverse/harness/` (2026-09-08). Paths relative to `harness/`.
+
+## 1. Prompt batteries (`bench/prompts/*.yaml`)
+
+Schema (`bench/run_bench.py:49-71`): `Battery{name, track, language, description, prompts[]}`;
+`BenchPrompt{id, prompt, tier (easy|medium|hard), category, must_have[], dimensions_m{}?, tags[], language?, references[]}`.
+
+| battery | track / language | n | tiers | notes |
+|---|---|---|---|---|
+| `static_objects_v1` | static_object / blender | 24 | 8/8/8, 8 categories | must_have + dimensions_m |
+| `static_objects_v2` | blender | 20 | 4 med + 16 hard | tags `axis:decomposition/curved_organic/thin_features/arch_detail/multi_material` |
+| `compare_v1..v3` | blender | 8/8/12 | | |
+| `compare_v4` | blender | 40 | 8/8/24 | union of v1+v2, paired-CI sized |
+| `complexity_v3` | blender | 12 | 3/3/6 | `expected_complexity: [lo,hi]` |
+| `fancy_v1_blender` | blender | 10 | hard | long product-designer briefs |
+| `fancy_v1_cadquery` | cadquery | 3 | hard | |
+| `fancy_v1_threejs` | threejs | 3 | hard | |
+| `teaser_v1_static` | mixed | 9 | | 3 cadquery, 3 threejs rows |
+| `h2h_brilliana_v1_*` | static_object | 5/4/3 | medium | must_have empty on purpose |
+| `articulated_v1/v2` | urdf_blender | 12/14 | | ≥3 DoF in v2 |
+| `scenes_v1/v2`, `teaser_v1_scene`, `h2h_scene_v1` | scene_threejs | 12/10/6/5 | | **needs the agentic harness** (multi-file) |
+| `graphics_v1/v2`, `teaser_v1_graphics` | glsl_shader (some opengl_python) | 6/10/5 | | |
+| `refs_v1/v2_graphics` | glsl_shader | 6 each | | `ref`/`noref` arms with photos in `bench/refs/aurora_ref_*` |
+
+**One-shot appropriate** (single LLM call → code): every static_object battery (blender/cadquery/threejs), every graphics battery, articulated (two-file contract). **Not one-shot**: scene batteries.
+
+## 2. Judging methodology
+
+- `codeverse/judges/vlm_judge.py` `VlmJudge(rubric, model_id, n_samples, temperature=0.2, thinking="low")`; default judge `gemini:gemini-3.1-pro-preview`. Variants `ReferenceJudge` (reference images + silhouette IoU), `LikenessJudge`, `PairwiseJudge` (`judges/pairwise.py`: both orderings, winner only when they agree).
+- Prompt (`judges/prompt_builder.py`): blind judge, observe-then-score, defect checklist; BRIEF + CONSTRAINTS + MEASUREMENTS + GATE FINDINGS; ≤5 labelled 2×2 montages (512 px tiles) + ≤2 detail crops; cross-section slices only on connectivity ERROR (D48).
+- Score (`judges/rubrics.py`, SCORING_VERSION=2): weighted criteria → majority-voted defects with penalties/caps → gate caps → `passed = overall ≥ threshold ∧ no floor ∧ no unverified must item`.
+- Rubrics (`judges/rubrics/*.yaml`): `static_object_v1` (threshold 0.72; intent_fidelity .22, structure .18, geometry_detail .16, proportions .12, assembly_fit .12, materials .10, craftsmanship .10; 11 defects), `articulated_v1` (0.70), `scene_v1` (0.70), `shader_v2` (0.70; shader_v1 had Spearman 0.16 vs human), `asset_v1`, `reference_v1` (measured silhouette IoU criterion, weight .25).
+- View rig (`codeverse/conventions.py:87-102`): 14 views (high ring ×4 @ +30°, eye-level ×4, low ring ×4 @ −30°, top, bottom), 768² studio background; clay montage ×4. Graphics: frames at t = 0, 1, 2.5, 4, 6 s.
+- Calibration (`docs/EVAL.md` §6): pro σ(overall, n=3) **0.030**, flash σ **0.083**; fixed-order re-judge σ 0.035; paired A/A on identical arms: paired sd 0.16–0.20 → generation variance dominates; ±0.02 needs ~500 pairs. ~$0.15–0.20 per pro verdict.
+- Deterministic gates: per-language lint; build (`ok` requires non-empty GLB); `measure_glb` (bbox, tri_count, n_meshes, n_islands, per-part volume/watertight, ground_gap, materials, complexity vector); `check_connectivity` (floating part = ERROR, penetration WARN > 2 mm, ERROR > 10 mm); graphics `gl_frames` (NaN, black/blown, static, flicker, low_detail); `reference_silhouette` IoU; `must_have` → VLM acceptance items.
+- **No CLIP/DINO/Chamfer metric exists in the harness**; silhouette IoU is the only image similarity; geometry is compared to the plan's boxes only.
+
+## 3. One-shot path (`bench/_oneshot.py`)
+- `oneshot_prompt(spec)` = `"Model this object in raw bpy: <prompt>"` + dimensions + MUST HAVE/NOT + minimal contract (Blender 5.x, emptied scene, Z-up, −Y front, metres, on z=0, PascalCase names, material per mesh, no render/export/import/network, <500k tris, <120 s, call `main()`) + "reply with the COMPLETE file as ONE ```python block".
+- Backends: `claude-code[:model]`, `codex[:model]`, `gemini:|anthropic:|openai:<model>` (ApiOneShot: temperature 0.5, thinking medium, max_output 32000). **Local open model works** via `openai:<model>` + `CV3D_OPENAI_BASE_URL=http://host:8000/v1` (vLLM OpenAI server).
+- Contracts per language to reuse verbatim: `codeverse/prompts/{blender,cadquery,threejs,glsl_shader,opengl_python,scene_threejs,urdf}/{system.md,contract.md,cookbook.md}`.
+- Evaluate a workspace: `bench/_fixed_eval.FixedEvaluator(judge_model, n_samples, track, language).evaluate(ws, spec) -> EvalOutcome` (build → gates → 14-view render → judge).
+
+## 4. Renderers / executors (`codeverse/languages/<lang>`)
+| language | build | outputs |
+|---|---|---|
+| blender | `blender -b --factory-startup --python wrappers/run_bpy.py -- --script src/model.py --out artifacts --rlimit-gb 12 --tri-limit 600000 --seed 0 --stl` | `object.glb` (Y-up), `object.stl`, `build.json`, `census.json` |
+| cadquery | `wrappers/run_cq.py --script … --tolerance 0.001` | `object.glb/.step/.stl` |
+| threejs | `node runtime_js/export_glb.mjs` ([email protected]) | `object.glb` |
+| urdf_blender | `run_bpy_links.py` + FK check + fcl sweep | `robot.urdf`, `meshes/*.glb`, pose renders |
+| scene_threejs | puppeteer probe/render over http | renders, `metrics.json` |
+| glsl_shader / opengl_python | moderngl `GlHost` | `frames/*.png`, `preview.gif`, `metrics.json` |
+
+GLB rendering for judging: `codeverse.spatial.render.render_glb(glb, out_dir, views=OBJECT_VIEWS, mode=shaded|clay|…)` via `runtime_js/render_glb.mjs` in headless Chrome.
+
+## 5. Result schema
+`BenchItemResult`: `id, tier, category, score_baseline, score_final, passed, rounds, cost_usd, minutes, status, errors, workspace, generator, judge`. `compare_backends.py` → `CellResult{prompt_id, arm, status (scored|build_failed|no_code|judge_error|infra_failed|budget_exhausted), score, score_std, passed, build_ok, gate_errors[], tris, gen_cost_usd, judge_cost_usd, criteria{}, …}` + `paired.md` (paired Δ, t-CI, sign test).
+
+## 6. Reuse recommendation for `3dcodeverse_eval/`
+1. Batteries are data: copy the YAMLs (`static_objects_v2`, `compare_v4`, `fancy_v1_*`, `graphics_v2`, `articulated_v2`) as a **rubric-judged / no-GT** prompt set.
+2. Judge stack as-is: `codeverse.judges.{rubrics, prompt_builder, vlm_judge, pairwise}` + rubric YAMLs; keep Gemini pro for comparability; n = 1 or 3.
+3. Gates/renderers: `codeverse.spatial.{measure.measure_glb, connectivity.check_connectivity, render.render_glb, complexity.complexity_of_glb, silhouette.best_view_match, frame_stats}`; `codeverse.languages.get_runtime(lang).build`.
+4. Gaps: one-shot contracts exist only for blender/urdf (derive others from `prompts/<lang>/contract.md`); no plan-free `dimensions_m` gate (write one over `measure_glb().extents`); scene batteries need the full harness.
diff --git a/eval/llm/docs/metrics.md b/eval/llm/docs/metrics.md
new file mode 100644
index 00000000..fee1866b
--- /dev/null
+++ b/eval/llm/docs/metrics.md
@@ -0,0 +1,95 @@
+# Metrics — definitions, formulas, pitfalls
+
+Everything `score.py` / `metrics.py` compute, with the exact convention used, and the reason. Numbers in
+`finetune/docs/REPORT.md` and `llm_finetune_exps.md` were produced with the same definitions (§1–§3), so they
+remain comparable; §4–§6 are additions.
+
+## 1. Execution-level
+
+| metric | definition | notes |
+|---|---|---|
+| **exec_rate** | `#OK / n` | OK is *strict*: Blender = ≥1 evaluated MESH object with ≥1 vertex **and** a successful GLB export in an emptied factory scene; CadQuery = a shape with volume > 1e-9 or faces, STL written; OpenSCAD = rc 0 and STL > 100 bytes; GLSL = glslang compiles the `mainImage` body under the Shadertoy prelude; three.js = page loads, zero fatal JS/import/request errors, a `<canvas>` exists, screenshot is non-uniform (`distinct colours > 6`, `std > 4`) |
+| statuses | OK / EMPTY / FAIL / CRASH / TIMEOUT / MISSING | EMPTY = ran but produced nothing; CRASH = host died; MISSING = no generation for the task (counts in the denominator) |
+| **render_rate** (GLSL only) | `#(compiled AND rendered non-uniform image) / n` | rendered in WebGL2 (headless Chromium) at t=0 and t=1.7 s: `distinct colours > 3` and `std > 2`; `STATIC` = compiles but paints a constant colour. Stricter than compile rate; the historical GLSL numbers are compile-only |
+| truncated | `#(finish_reason == length)` | every reasoning/long-output failure shows up here first — report it next to exec_rate |
+| extraction diagnostics | `no_fence`, `multi_block`, `chosen_not_last`, `from_think_fallback`, `syntax_valid`, `no_dialect_marker` | from `extract.py`; separates *format* failures from *capability* failures (REPORT §11.2) |
+
+## 2. Geometry (suites with a reference mesh)
+
+Both meshes are normalised: translate to the bounding-box centre, scale so the largest bbox extent is 1.
+`n_points` surface samples (10 000 for 3DCodeBench, 5 000 for the held-out sets), `seed=0`.
+
+| metric | formula | notes |
+|---|---|---|
+| **Chamfer** | `CD = mean_{a∈G} min_{b∈R} ‖a−b‖₂ + mean_{b∈R} min_{a∈G} ‖a−b‖₂` | **sum** of the two directional means, unsquared L2 (the report's convention). Many CAD papers report `½·CD` or squared distances ×1000 — convert before comparing to them. |
+| **F@τ** | `P = frac(d_{G→R} < τ)`, `R = frac(d_{R→G} < τ)`, `F = 2PR/(P+R)`; τ ∈ {0.05, 0.1} of the unit bbox | [email protected] is the headline geometry metric; [email protected] is lenient |
+| rotation search | generated cloud rotated by k·90° (k=0..3) about the up axis (Y for GLB, Z for STL); keep the rotation with minimal CD | objects may face any cardinal direction; no continuous ICP on purpose (would reward wrong shapes) |
+| **IoU_vox** | solid-voxel IoU on a 64³ grid of the normalised bbox | needs closed meshes; `None` when a mesh cannot be voxelised (open shells) — reported as mean over scored tasks and with fail = 0 |
+| `*_mean_scored` | mean over tasks with `exec == OK` and a reference mesh | quality *given* execution |
+| `*_mean_all` | fail counted as 0, denominator = tasks whose **reference** executed (`n_ref_ok`) | the headline; the old held-out evals divided by `n` and so charged reference failures to the model (bug #3 in `inventory_finetune_eval.md`) |
+
+Sanity anchor: re-executing the 3DCodeBench reference scripts in the same executor and scoring against the
+shipped GT gives CD ≈ 0.016, [email protected] ≈ 1.0 (REPORT §5). `run_eval --backend reference` reproduces this
+self-test for every suite.
+
+## 3. Sampling metrics
+
+| metric | formula |
+|---|---|
+| **pass@k** | unbiased estimator (Chen et al. 2021): `1 − C(n−c, k)/C(n, k)` for n samples with c OK, averaged over tasks |
+| best-of-N [email protected] (all) | mean over tasks of `max_i [email protected]_i` with failures = 0 |
+
+Sampling at T = 0.7, top-p 0.95, seeds 0..N−1 (`run_eval --samples N`).
+
+## 4. Rubric-judged suites (harness batteries, no GT)
+
+Execution rate is always reported. Quality needs a judge: the harness's calibrated VLM judge
+(`harness/codeverse/judges`, Gemini pro, 14-view rig, `static_object_v1` rubric, threshold 0.72, σ ≈ 0.03 at
+n=3) is the reference implementation — see `docs/inventory_harness_bench.md` §2 for how to call it on a
+GLB. No judge is bundled here yet; if one is added it must be pairwise with swapped order and report κ against a
+human-labelled subset (LLM judges have position/length bias — survey §2e).
+
+## 4b. Structural integrity (any executed mesh; no GT needed) — `structural.py`, vendored from the official repo
+
+What Chamfer, F and the image encoders are blind to: is the output *one coherent object* or a pile of parts?
+Mesh normalised to a unit sphere, vertices welded, connected components on the vertex graph; contact tolerance
+ε = 1 % of the bbox diagonal.
+
+| field | meaning |
+|---|---|
+| `n_components`, `abs_log2_component_ratio` | component count and |log₂(gen / ref)| when a reference exists |
+| **`floating_part_rate`**, `floating_area_fraction` | components not within ε of any other component (share of count / of surface area) |
+| `largest_component_volume_fraction` | fragmentation: volume of the largest component / total |
+| `nonmanifold_edge_rate`, `boundary_edge_rate`, `is_watertight` | topology health |
+| `ground_contact`, **`com_over_support`**, `com_support_margin` | has a real base; centre of mass over the support polygon (static stability) |
+
+Reported conditional (executed meshes) and penalized (failures at the worst value, per the official
+`PENALTY` table). Read most fields as a *ratio to the reference*: only 38 % of the 3DCodeBench GT meshes are
+watertight and 25 % are single-component (a fern legitimately has thousands of parts). On this machine the
+CadQuery reference set scores floating 0.027 / watertight 0.93 / stable 0.93.
+
+## 4c. VLM judge (`judge.py`) — the quality layer when there is no GT
+
+Absolute mode: 1–5 on identity / structure / detail (+ reference faithfulness for image tasks) and an overall
+grade, from the 4 canonical renders; reported conditional and penalized (unrendered = 0). Pairwise mode: the
+official 3DCodeBench arena prompt, judged in both orders; a winner counts only when both orders agree, else
+`tie` (order disagreements are reported — they measure the judge's position bias). Any OpenAI-compatible VLM
+endpoint or Claude. Treat absolute grades as triage; rank models with pairwise win-rates.
+
+## 5. Image-conditioned (VLM) suites
+
+Same executor and geometry metrics as the text suites (same GT), so `3dcodebench_img4` vs `3dcodebench_text`
+is directly comparable. Reference views are the official 4 GT renders (`Image_005/015/025/035`), sent as
+768-px JPEG data-URLs, image parts before the text.
+
+## 6. Noise floor and reporting rules (from REPORT §12.7, §13.4, §13.7)
+
+- Within one session with one engine config, greedy runs repeat to 0.0 pp on Blender suites, 0.5 pp CadQuery,
+ 1.5 pp GLSL, 2.0 pp OpenSCAD, 5.0 pp three.js (n = 40). Across sessions / engine configs drift can reach
+ 6–8 pp on OpenSCAD. **Compare models only within one batch.**
+- OpenSCAD / GLSL / three.js: greedy decoding degenerates into runaway loops on boilerplate-heavy dialects;
+ report **T = 0.7 as primary** and greedy as secondary (suite defaults do this).
+- Fix `max_new_tokens` per suite (8192 text/CAD, 12288 three.js) and always report `truncated`.
+- Report `exec_rate` and `[email protected] (all)` as the two headline numbers; `[email protected] (ok)`, `CD (ok)`, `IoU (ok)`
+ describe quality given execution.
+- 212-task binomial 95 % CI at 50 % is ±6.7 pp; at n = 40 it is ±15 pp — do not read third decimals.
diff --git a/eval/llm/docs/protocol.md b/eval/llm/docs/protocol.md
new file mode 100644
index 00000000..e1c61bf4
--- /dev/null
+++ b/eval/llm/docs/protocol.md
@@ -0,0 +1,76 @@
+# Evaluation protocol — what to run, how to decode, how to report
+
+Distilled from `finetune/docs/REPORT.md` (§5, §11–§13), the 3DCodeBench paper protocol
+(`~/nips2026_paper/3dcodebench`, arXiv:2606.01057) and the literature survey (`survey_synthesis.md` §4).
+
+## 1. The standard battery for one model
+
+| tier | suites | why |
+|---|---|---|
+| headline | `3dcodebench_text` | the report's headline; 212 tasks, GT mesh, greedy |
+| dialect coverage | `heldout_blender`, `heldout_cadquery`, `heldout_openscad`, `heldout_glsl`, `heldout_threejs` | never-trained-on prompts per dialect; catches "one-dialect" models |
+| VLM | `3dcodebench_img4` (+ `img1`, `img4text` for the ablation) | image→code with the same GT as text, so the text/image gap is a direct read |
+| paper placement | `3dcodebench_official_text`, `3dcodebench_official_img4` | the official prompts + SigLIP-2/DINOv3 + official Chamfer, comparable to the frontier table |
+| rubric | `harness_static_objects_v2`, `harness_compare_v4`, `harness_graphics_v2` … | brief-style prompts with must-have lists; needs the harness VLM judge for quality |
+
+`run_eval --suites headline heldout vlm official` runs the first four tiers.
+
+## 2. Decoding
+
+| dialect / suite | primary | secondary | max_new_tokens |
+|---|---|---|---|
+| Blender (3dcodebench_*, heldout_blender), CadQuery | greedy (T=0), thinking off | T=0.7 × 4 samples (pass@k) | 8192 |
+| OpenSCAD, GLSL, three.js | **T=0.7, top-p 0.95** | greedy (diagnostic: report the runaway/truncation rate) | 8192 / 8192 / 12288 |
+| official suites | T=0.7 (paper default), thinking "medium" for API models | — | 16384 |
+
+Rationale: greedy decoding on boilerplate-heavy dialects collapses into repetition loops that hit the token
+cap (REPORT §13; ShaderMatch's `incomplete` bucket; survey §2g). Thinking modes raise execution rates on most
+dialects but exhaust the budget on GLSL (REPORT §11.2) — evaluate them as a separate row with a stated budget,
+never mixed with no-think rows. `--think` keeps the model's reasoning on; the default sets
+`enable_thinking=False` through the chat template.
+
+## 3. Sampling and pass@k
+
+`--samples N` (seeds 0..N-1, T forced to 0.7 if the suite default is greedy) → `report.py` computes unbiased
+pass@1/2/4/N and best-of-N [email protected]. Use N = 4 for cost, N = 10 when a 3-point gap matters. pass@1 from N
+samples is the *sampling* pass@1; the greedy row is a different quantity — label both.
+
+## 4. Execution budgets (identical across suites of a dialect)
+
+Blender 300 s, CadQuery 90 s, OpenSCAD 120 s, GLSL 60 s (compile + WebGL render), three.js 60 s page load +
+4 s settle. Official renderer 240 s per GLB (Cycles 64 spp, 512 px, GPU if present).
+
+## 5. What to report (one table row per (model, suite))
+
+`exec_rate` · `[email protected] (all)` · `[email protected] (ok)` · `CD (ok)` · `IoU (ok)` · `render_rate` (GLSL) · `truncated` ·
+`mean_new_tokens` · `no_fence` / `multi_block` — `report.py` emits exactly this. For official suites add
+`siglip2 paired (penalized / conditional)`, `dinov3 paired`, `cd_official_yawmin (cond / pen)`.
+
+Always give: model id + revision, backend (vllm / server / API), temperature, max_new_tokens, thinking on/off,
+tool versions (Blender 5.0.1, OpenSCAD build, glslang, Chromium), and the date — API drift (Blender 4→5) was
+the largest single confound in the paper.
+
+## 6. Noise, comparisons, contamination
+
+- Within one batch and engine config, greedy repeats are exact on Blender suites; OpenSCAD / GLSL / three.js
+ vary 1.5–5 pp; across sessions up to 6–8 pp (REPORT §13.7). **Run the models you compare in one batch**,
+ same `--limit`, same engine.
+- Binomial 95 % CI at n = 212 is ±6.7 pp around 50 %; n = 40 (three.js) ±15 pp. Report per-suite n.
+- Do not read a metric computed only on executed tasks (`*_scored`) as a quality ranking: it rewards models
+ that fail often (survivor bias, survey §2a). The `*_all` / penalized numbers are the headline.
+- Contamination: the 212 test factories and all their seed instances are excluded from every 3DCodeVerse
+ training mix (`build_multidialect.py`, `build_md_max.py` rules); the held-out parquets were built with
+ `random.seed(7)` splits before training. A model trained on the open `ilabai/3dcodeverse` corpus **has
+ seen** DeepCAD / Shadertoy / Thingiverse neighbours of the held-out prompts (same sources, different items) —
+ say so when reporting.
+
+## 7. Extending
+
+- New prompt set: write `data/prompts/<suite>.jsonl` in the row schema (README) and add a `SuiteSpec`.
+- New dialect: add `executors/<dialect>.py` with `run(code, workdir, timeout) -> report`, register in
+ `executors/__init__.py` (`TIMEOUTS`, `CODE_EXT`, `RUNNERS`), add dialect markers to `extract.DIALECTS`.
+- New metric: `metrics.py` primitive + one block in `score.py` gated by the suite's `metrics` tuple + a row in
+ `docs/metrics.md`.
+- Deterministic property tests per prompt (survey §3.3: CADTestBench-style must-have checks, human agreement
+ F1 0.94 vs 0.66 for Chamfer) are the highest-value next addition; the harness batteries already carry
+ `must_have` lists in `meta` to seed them.
diff --git a/eval/llm/docs/reference_numbers.md b/eval/llm/docs/reference_numbers.md
new file mode 100644
index 00000000..289cec76
--- /dev/null
+++ b/eval/llm/docs/reference_numbers.md
@@ -0,0 +1,59 @@
+# Reference numbers — what a new model should be compared against
+
+Three sources, three protocols. Never mix columns across sections without converting (see `metrics.md`).
+
+## A. 3DCodeBench paper, Table 1 (arXiv:2606.01057, `~/paper_refs/3dcodebench/tables_main/main_results.tex`)
+
+Official protocol: official system prompt (raw python), T=0.7, thinking at each model's best level, Blender 5.0,
+4-view Cycles renders. **Each value is the mean over the text-to-3D and image-to-3D tracks; SigLIP-2 / DINOv3 /
+Chamfer are conditional (executed instances only).** Chamfer = squared, unit-sphere, 8 192 pts.
+
+| model | Exec ↑ | SigLIP-2 ↑ | DINOv3 ↑ | Chamfer ↓ | Uni3D 3D–3D ↑ | Elo |
+|---|---|---|---|---|---|---|
+| Gemini 3 Flash | 0.481 | 0.810 | 0.528 | 0.067 | 0.543 | 1039 |
+| Gemini 3.1 Flash Lite | 0.575 | 0.778 | 0.496 | 0.077 | 0.445 | 877 |
+| Gemini 3.1 Pro | 0.725 | 0.824 | 0.569 | 0.069 | 0.567 | 1147 |
+| Gemini 3.5 Flash | 0.464 | 0.824 | 0.563 | 0.068 | 0.519 | 1119 |
+| Gemma 4 26B | 0.517 | 0.786 | 0.483 | 0.077 | 0.435 | 859 |
+| Gemma 4 31B | 0.582 | 0.801 | 0.518 | 0.076 | 0.494 | 952 |
+| Claude Haiku 4.5 | 0.502 | 0.761 | 0.413 | 0.095 | 0.363 | 799 |
+| Claude Sonnet 4.6 | 0.804 | 0.813 | 0.551 | 0.068 | 0.525 | 1015 |
+| Claude Opus 4.7 | 0.910 | 0.814 | 0.545 | 0.067 | 0.490 | 1006 |
+| GPT-5.4 mini | 0.731 | 0.803 | 0.526 | 0.070 | 0.506 | 951 |
+| GPT-5.4 | 0.866 | 0.817 | 0.560 | 0.064 | 0.552 | 1074 |
+| GPT-5.5 | 0.906 | 0.834 | 0.576 | 0.059 | 0.562 | 1163 |
+
+Single-shot (no thinking-level search) appendix table differs by ≤ 0.07 exec (Gemini 3 Flash 0.547, 3.1 Pro 0.698,
+3.5 Flash 0.479; the rest identical). Gemini 2.5 Pro and GPT-5.4 nano were excluded (exec < 10 %).
+
+Paper findings to keep in mind: SigLIP-2 view-paired (cond.) has Pearson r = 0.964 and DINOv3 Spearman ρ = 0.972
+against the human Elo; multi-turn traceback feedback lifts exec from 0.70 to 0.97 without changing conditional
+quality; coding-agent harnesses do the same (0.747 → 0.973, conditional SigLIP-2 −0.010).
+
+The raw outputs behind this table are on the Hub (`3DCodeBench_ModelLogs/data/text_to_3D.parquet` etc.);
+`rescore_logs.py` replays them through this package — see §C for the local re-score.
+
+## B. finetune report protocol (`finetune/docs/REPORT.md`, `llm_finetune_exps.md`)
+
+Fenced-block system prompt, greedy, no-think, 6 144–8 192 new tokens, Blender 5.0.1, exec = non-empty mesh + GLB
+export, F@τ on unit-bbox point clouds (10 k), reported as F(all) with failures = 0.
+
+| model | 3DCodeBench exec | [email protected] all | Blender 103 | CadQuery 200 | OpenSCAD 50 | GLSL 200 | three.js 40 |
+|---|---|---|---|---|---|---|---|
+| Qwen3.5-9B zero-shot (no-think) | 0.0 % | 0.000 | 1 % | 10 % | 42–46 % | 39 % | 77.5 % |
+| Qwen3.5-9B zero-shot (thinking) | 0.5 % | – | 14.6 % | 33 % | 84 % | 35 % | 87.5 % |
+| Qwen2.5-Coder-7B-Instruct zero-shot | 8.5 % | 0.014 | – | – | – | – | – |
+| Qwen3.8-27B zero-shot (no-think) | 61.8 % | 0.257 | 77.7 % | 62.5 % | 88 % | 62 % | 82.5 % |
+| Qwen3.5-9B LoRA v1 (5.2 k Blender) | 75.0 % | 0.285 | – | – | – | – | – |
+| Qwen3-8B LoRA v1 | 76.4 % | 0.277 | – | – | – | – | – |
+| Qwen3.5-9B + exec-feedback DPO r2 | 96.2 % | – | – | – | – | – | – |
+| Qwen3.5-9B md_xl (270 k, 4 dialects) | 90.1 % | – | – | – | – | – | – |
+| Qwen3.8-27B v2 SFT | 93.4 % | best geometry | – | – | – | – | – |
+
+Reference programs through the same executors: 3DCodeBench 210/212 (this machine), Blender held-out 96/103,
+CadQuery 200/200 (report: 199/200), OpenSCAD 50/50, GLSL 178/200 compile, three.js 40/40.
+
+## C. Local re-scores (this machine, `docs/results/`)
+
+Filled in by the runs under `$CV3D_EVAL_OUT` (`report.py`): frontier logs replayed under the official protocol,
+the reference self-test, and the open 8B/9B runs. See `docs/results/README.md`.
diff --git a/eval/llm/docs/results/README.md b/eval/llm/docs/results/README.md
new file mode 100644
index 00000000..8175d25b
--- /dev/null
+++ b/eval/llm/docs/results/README.md
@@ -0,0 +1,10 @@
+# Local results (this machine: WSL2, RTX 5090 24 GB, Blender 5.0.1, OpenSCAD 2026.09.07, glslang 16.5, Chromium 151)
+
+Every table here is regenerated by `python -m llm.report --root $CV3D_EVAL_OUT`; the per-run
+`summary.json` files are the source of truth. Runs are dated because tool versions drift.
+
+| run | what | file |
+|---|---|---|
+| `reference` | the reference programs through the whole pipeline (executor + metric self-test) | `2026-09-08_reference.md` |
+| `official_<model>` | the paper's frontier outputs (text_to_3D logs) re-executed + re-scored here under the official protocol | `2026-09-08_official_rescore.md` |
+| `qwen3_8b*` | Qwen3-8B zero-shot via vLLM (report protocol + official protocol) | `2026-09-08_qwen3_8b.md` |
diff --git a/eval/llm/docs/survey_records.md b/eval/llm/docs/survey_records.md
new file mode 100644
index 00000000..2419567f
--- /dev/null
+++ b/eval/llm/docs/survey_records.md
@@ -0,0 +1,1459 @@
+# Per-benchmark records (raw fact sheets, 2026-09-08)
+
+
+<!-- ===== recent2026.md ===== -->
+
+# 2026-era / cross-cutting entries (collected by lead agent)
+
+## 3DCodeBench (Gao et al., arXiv 2606.01057, May 31 2026)
+- Paper https://arxiv.org/abs/2606.01057 ; HTML https://arxiv.org/html/2606.01057 ; site https://www.3dcodebench.com/ ; code https://github.com/gaoypeng/3dcodebench ; data HF `YipengGao/3DCode`
+- Task: text (object description / procedural instruction / factory-level spec) or 4 canonical multi-view reference images (45/135/225/315 deg azimuth) -> standalone Blender 5.0 Python script
+- Data: 212 Infinigen-derived object categories; 26K prompt/code/mesh triplets; 12,963 fine-tune instances (site says 12,720 factory instances); 212-instance eval set; median script 387 lines, mean 531. GT = Infinigen factories migrated by a VLM agent into standalone bpy + baked GLB. Dedup via "Class Deduplication" in Experience Library. No explicit difficulty tiers.
+- Env: Blender 5.0, 240 s wall-clock timeout, fresh subprocess per run, Cycles renders from 4 canonical views.
+- Metrics: Exec_i = 1[script runs within 240 s and produces >=1 mesh]; SigLIP-2 view-paired cosine similarity (4 views), DINOv3-ViT-L16 same; Chamfer on exported GLB point clouds (normalized), Uni3D 3D-3D cosine and cross-modal (text->pc, image->pc); each reported "conditional" (only successes) or "penalized" (failures = 0). Human: 3DCodeArena pairwise votes (a/b/tie/both bad; tie & both-bad = 0.5/0.5), Bradley-Terry MLE Elo per modality, 1000-resample bootstrap CI; ~3,100 votes at writing.
+- Correlation with human Elo: SigLIP-2 Pearson r=0.964; DINOv3 Spearman rho=0.972; executability weakest correlate.
+- Results (Table A.1, best thinking level, avg text+image): GPT-5.5 exec 0.906 / SigLIP2 0.834 / DINOv3 0.576 / CD 0.059 / Uni3D 0.562 / Elo 1163; Gemini 3.1 Pro 0.725/0.824/0.569/0.069/0.567/1147; Claude Opus 4.7 0.910/0.814/0.545/0.067/0.490/1006; Gemma 4 31B (open) 0.582/0.801/0.518/0.076/0.494/952; Gemma 4 26B 0.517/0.786/0.483/0.077/0.435/859; Claude Haiku 4.5 0.502/0.761/0.413/0.095/0.363/799. Gemini 2.5 Pro (7.1% exec) and GPT-5.4 Nano (6.1%) excluded (<10% exec, Blender 5.0 API cutoff).
+- Findings: 2 retries with traceback lift exec 0.702->0.974 (+27.2 pp) and SigLIP-2 +0.128; native coding-agent harness reaches ~0.995 exec but no conditional quality gain (Uni3D 3D-3D -0.003 matched subset). Failures mostly API mismatches; successful renders suffer disconnected/floating parts. Thinking budget: lightweight models +~19 pp exec minimal->high, frontier plateau.
+- No 7-9B open model; smallest open = Gemma 4 26B. License CC BY 4.0 (arXiv).
+
+## P3D-Bench (Yang et al., arXiv 2606.11152, Jun 9 2026)
+- https://arxiv.org/html/2606.11152v1 ; project https://lucasqaq.github.io/p3d/
+- Tasks: Text-to-3D (descriptive vs parametric spec) -> JSON/OpenSCAD; Image-to-3D (single render) -> CadQuery/OpenSCAD/Three.js; Assembly-3D (image + part annotations) -> CadQuery/OpenSCAD.
+- Data: 400 text cases from Text2CAD v1.1 (176,017 programs), 400 image cases + 203 assembly cases from Fusion 360 Gallery (8,251 assemblies). Filtering: deterministic executable/renderable check, MLLM review (Gemini 3.1 Pro) for ambiguity/quality, DINOv2 near-duplicate removal, complexity-balanced sampling over easy/medium/hard tiers. Prompts: GPT-5.5 writes descriptive + parametric specs; Claude Opus 4.6 labels parts; MLLM consistency check; static validators cross-check every numeric value against GT geometry.
+- Env: format-specific compilers, multiview renders; mesh alignment = normalize, translate, rotate (min CD), bounded scale/position refinement.
+- Metrics: Valid (binary compile+render); Geometry bucket: CD, [email protected], [email protected], Normal Consistency, IoU (CSG for single parts, voxel for assemblies); Topology bucket: NoOE (edges w/ exactly 2 faces), InvN (reversed normals), NM (non-manifold); Judge bucket (Gemini 3.1 Pro): QA-S (4 semantic MCQ/case), QA-P (8 parametric Qs/case), J-Sem/J-Geo/J-Aes on 1-10; Part bucket: Claude Opus 4.6 decomposes prediction, fidelity gate CD<5e-4 & IoU>0.95, PartFS (Hungarian-matched F-score), PartMatchF1. Buckets = equal-weight means in [0,1]; invalid outputs get worst value.
+- Results: Text-to-3D descriptive judge: GPT-5.5 0.875 (valid 0.998), Gemini 3.1 Pro 0.865, Claude Opus 4.6 0.850. Image-to-3D avg: GPT-5.5 Geo 0.549 / Judge 0.562 / Valid 0.979. Assembly: GPT-5.5 Geo 0.586 / Part 0.629. J-Sem ~0.80 but J-Geo ~0.35 on assemblies; PartMatchF1 ~0.50.
+- Error taxonomy: syntax (73% of text-descriptive failures), undefined reference (~16% CadQuery), parameter (55% Image-to-3D CadQuery), geometry (24-27% CadQuery). Three.js meshes non-manifold -> excluded from topology/part scoring. Multi-turn: Gemini +0.03 avg, GPT-5.5 ~0.
+- Open 7-9B: none (domain models Text2CAD 363M, cadrille 2B, CAD-Coder 13B). Availability: project page only; license arXiv non-exclusive.
+
+## Text2CAD-Bench (Wang et al., arXiv 2605.18430, May 18 2026)
+- https://arxiv.org/html/2605.18430v1 ; CC BY 4.0; code/leaderboard "released after accept" (URL placeholder).
+- Task: text -> CadQuery. 600 human-curated examples x 2 prompt styles = 1,200 prompts. Tiers: L1 basic 200, L2 intermediate 200 (booleans), L3 advanced 100 (sweep/loft/shell/freeform), L4 real-world 100. Prompt styles: geometric (global->local->details->summary) vs sequence (command-style); L1 avg 89.9 vs 81.5 words, L3 429.4 vs 173.6. Three engineering annotators verify executability, description-geometry correspondence, complexity, cross-style consistency. Few-shot examples from held-out set.
+- Env: Python 3.10, CadQuery 2.4, trimesh, isolated, 60 s timeout, STEP -> point cloud.
+- Metrics: CD bidirectional mean NN distance, 30,000 points/mesh, x10^3; Invalidity Rate (fail/timeout/degenerate); IoU on 256^3 voxels; L4: GLM-4.6V judge on 8 multi-view renders, 0-10 across Overall/External/Functional/Extended/Detail features + overall similarity; 3 human annotators re-scored stratified L4 subset -> consistent trend, no same-family bias.
+- Results: GPT-5.2 L1 CD 44.31 / IR 11.1% / IoU 0.59 (geometric prompts) -> L3 CD 93.46 / IR 68.0% / IoU 0.23. Text2CAD IR 2-11% but CD 219-267, IoU ~0 (executable but unfaithful). L4: Gemini-3-Flash lowest IR 17% but lowest feature scores. No open 7-9B model reported.
+- Pitfalls: survivorship bias (CD/IoU on executed subset only when IR varies); CadQuery-only; single-turn.
+
+## BenchCAD (Zhang et al., arXiv 2605.10865, May 2026)
+- https://arxiv.org/html/2605.10865v1 ; data HF `BenchCAD/BenchCAD` (CC-BY-4.0); code GitHub `BenchCAD/BenchCAD-main` (MIT); site https://benchcad.github.io/BenchCAD_webpage/
+- Tasks: Vision2Code (4 orthographic views -> CadQuery), Vision QA / Code QA (numeric answers), Code Edit (program + instruction -> CadQuery).
+- Data: 17,900 execution-verified CadQuery programs over 106 industrial part families (52 anchored to ISO/DIN/EN/ASME/IEC tables); 2,400 QA items; 748 edit pairs; easy/medium/hard tiers; 10 families held out OOD; expert visual sign-off.
+- Env: sandboxed subprocess, 30 s budget, degenerate filter volume <= 1e-6 mm^3, error quarantine (parse/import/runtime/timeout).
+- Metrics: voxel IoU (rotation-invariant variant App. M), CD, exec_pct, Essential-Op Recall (helix/loft/sweep/twist-extrude), Feature-F1, Edit normalized accuracy = clip((IoU_gen - IoU_orig)/(1 - IoU_orig),0,1), QA accuracy (+-5% ratios, exact ints).
+- Results: qwen3-VL-2B RL (iid) IoU 0.752 / exec 98.9% / total 0.768; CADEvolve v3 0.750 / 92.7% / 0.601; Claude Opus 4.7 thinking IoU 0.267 / exec 90.4% / 0.378; GPT-5.3 thinking 0.207 / 67.5% / 0.212. Vision QA best Gemini 3.1 Pro 0.587.
+- Findings: advanced ops near-zero recall in pretrained models; ~64% of "successful" edits silently corrupt unrelated features; hard-tier drop >45 pts.
+
+## CADFS (Pyatov et al., arXiv 2605.01925, May 2026)
+- https://arxiv.org/abs/2605.01925 ; site https://voyleg.github.io/cadfs/
+- Task: text or multi-view images -> Onshape FeatureScript program (15 ops). 451k real designs (subset of ABC, superset of DeepCAD/Text2CAD), ~15% discarded by reproduce-original validation. Annotations by Annotator+Reviewer LLM pair with FeatureScript docs.
+- Metrics (App.): CD = symmetric mean squared NN distance, 100k points, median, relative to model size, x10^3; Edge CD (points near edges, r=0.004, |n_i.n_j|<0.2); Normal Consistency; COV/MMD/JSD on 3k shapes x10 repeats with 2k points (DeepCAD protocol); IR = N_inv/N_gen (kernel error or invalid geometry/topology).
+- Table 4 (text-conditioned, their test set): Qwen2-VL-2B CD 0.06 / ECD 8.5 / NC 98.9 / MMD 8.10 / COV 84.9 / JSD 0.59 / IR 10%; Qwen3-8B 0.06 / 7.8 / 98.9 / 8.04 / 85.0 / 0.60 / IR 8%.
+- Table 1 comparison: Fusion360 8,625; DeepCAD 179k; Text2CAD 170k; Omni-CAD 275k; cadrille 170k; CAD-Recode 1M synthetic; CADFS 451k real.
+
+## CADEvolve (Elistratov et al., arXiv 2602.16317, Feb 2026)
+- https://arxiv.org/html/2602.16317v1 ; code https://github.com/zhemdi/CADEvolve ; data https://huggingface.co/datasets/kulibinai/cadevolve ; model https://huggingface.co/kulibinai/cadevolve-rl1 ; CC BY 4.0
+- Task: 8 views (6 ortho + 2 iso, 2x4 grid) -> CadQuery. Data: 7,945 evolved generators -> ~800k scripts -> ~1.3M canonicalized (scale longest extent = 200, 24 rotations).
+- Metrics: CD 8,192 vs 8,192 points x10^3, MEDIAN "to reduce invalidity bias"; volumetric IoU mean; IR = fail to compile or non-watertight/degenerate.
+- Results vs cadrille-RL: DeepCAD CD 0.16 vs 0.17, IoU 91.1 vs 92.2, IR 0.1 vs 0.1; Fusion360 0.16/84.0/0.2 vs 0.17/84.6/0.1; MCB 0.52/55.2/0.4 vs 0.87/47.6/2.5.
+
+## Text-to-CadQuery (Xie & Ju, arXiv 2505.06507, May 2025)
+- https://arxiv.org/abs/2505.06507 ; CC BY 4.0. 170k CadQuery annotations converted from Text2CAD; six open models fine-tuned; top-1 exact-match 58.8% -> 69.3%, CD -48.6% (abstract; per-size numbers not extracted).
+
+## BlenderRAG (Rondelli et al., arXiv 2605.00632, May 2026)
+- https://arxiv.org/abs/2605.00632 ; https://github.com/MaxRondelli/BlenderRAG ; CC BY 4.0
+- Task: text -> Blender Python. Dataset: 500 expert-validated (text, code, image) triplets, 50 categories (25 indoor / 25 outdoor) x 10 variants; code drafted by Claude Opus 4.1 from detailed prompts then expert-validated.
+- Eval: only 30 out-of-distribution prompts (novel objects absent from dataset). Metrics: compilation success % (code executes w/o error); semantic alignment = normalized CLIP cosine between prompt and rendered image. Table 1 (Base -> RAG): Claude Sonnet 4.5 compile 43.3 -> 76.7, CLIP 0.544 -> 0.780; GPT-5 Chat 56.6 -> 66.7, 0.267 -> 0.777; Gemini 3 Flash 53.3 -> 80.0, 0.498 -> 0.770; Mistral Large 10.1 -> 56.7, 0.327 -> 0.769; average 40.8 -> 70.0, 0.409 -> 0.774. No open 7-9B model; no Chamfer or VLM judge in the paper (only compile + CLIP); n=30 means each compile-rate point is 3.3 pp.
+
+## CodeGen-3D (Ji et al., IEEE Access 14:18181-18192, Jan 2026)
+- https://scholarworks.sjsu.edu/faculty_rsca/6835/ ; DOI 10.1109/ACCESS.2026.3654948
+- Task: text -> Blender script; 100 prompts from Objaverse captions (Cap3D / DiffuRank / Llama-based). Metrics: error rate (compile+runtime), multi-view CLIP (avg & max), GPT-4o pairwise VLM judge. Eight general LLMs + BlenderLLM + iterative agent: failure 3-4% (specialized/iterative) vs 21-92% (general); iterative agent 33.3% GPT-4o win rate. Finding: judge preferences diverge from CLIP alignment. No code URL found.
+
+## LL3M (Lu et al., arXiv 2508.08228, Aug 2025)
+- https://arxiv.org/abs/2508.08228 ; https://threedle.github.io/ll3m — multi-agent Blender-script system; abstract gives no quantitative benchmark. Not a benchmark.
+
+## SceneActBench (Zhao et al., arXiv 2607.22393, Jul 2026)
+- https://arxiv.org/abs/2607.22393 — 210 source instances -> 520 task cases, 5 tasks, agent-environment loop, geometric metrics vs hidden GT, 11 proprietary VLM configs score 38.6-50.2. Whether outputs are code: not established from abstract.
+
+## Other hits not yet verified
+- "LLM-based 3D Model Generation of MHE for OpenSCAD" (Procedia CS 2026) https://www.sciencedirect.com/science/article/pii/S1877050926000773 — 403 on fetch; search snippet: functions in simple/intermediate/advanced tiers by number of components.
+- ModelRift "OpenSCAD LLM Benchmark" blog https://modelrift.com/blog/openscad-llm-benchmark/ — fetch timed out.
+
+<!-- ===== cad.md ===== -->
+
+# Survey: benchmarks / datasets / metrics for LLM & VLM generation of CAD code (2021 – Sep 2026)
+
+Compiled 2026-09-08 from primary sources (arXiv abstract + HTML pages, GitHub READMEs, HF dataset cards, project pages). Every number below is copied from the cited source; "not found" means the source I could reach did not state it. Numbers were NOT recomputed or inferred.
+
+Conventions used repeatedly across the field (see the cross-cutting notes at the end):
+- **CD** = Chamfer distance, almost always reported ×10³ on shapes normalised to a unit cube / [-0.5,0.5]³; point counts vary 1k–30k and are frequently unstated.
+- **IR** = invalidity ratio (fraction of generations that do not execute / do not yield a valid solid). **VSR** = valid-shape / valid-syntax rate = 1 − IR (definitions differ).
+- **IoU** = volumetric IoU on voxel grids (resolution varies 96³–256³) or exact CSG.
+
+---
+
+## 0. Foundational datasets (not LLM benchmarks per se, but every later benchmark derives from them)
+
+### DeepCAD (Wu, Xiao, Zheng — ICCV 2021)
+- URLs: paper https://arxiv.org/abs/2105.09492 ; code+data https://github.com/rundiwu/DeepCAD ; parser https://github.com/rundiwu/onshape-cad-parser
+- Task: unconditional/autoencoding generation of sketch-and-extrude command sequences (DeepCAD JSON / 17-d command vectors). Not text-conditioned.
+- Size/construction: 178,238 CAD models "parsed from Onshape public documents with links from ABC dataset" (README). Standard split used by all derivatives: 90/5/5 → test = 8,046 models (count stated by CAD-Recode https://arxiv.org/html/2412.14042 and cadrille https://arxiv.org/html/2505.22914). Sketch+extrude only.
+- Metrics defined here and inherited everywhere: command accuracy, parameter accuracy, Chamfer distance, invalid ratio (autoencoding); COV, MMD, JSD (generation) — README https://github.com/rundiwu/DeepCAD. GenCAD states DeepCAD-style CD is computed on 2,000 sampled points https://arxiv.org/html/2409.16294.
+- Pitfalls reported by derivatives: heavy class imbalance — Text2CAD v1.0 notes ~25% cuboids and ~8% cylinders (HF card https://huggingface.co/datasets/SadilKhan/Text2CAD); CADmium filtered cuboids to 5% https://arxiv.org/html/2507.09792; FlexCAD/SkexGen remove duplicates and invalid sequences https://arxiv.org/html/2411.05823. Fusion360 Gallery is NOT in ABC, so it is the usual "clean" OOD set (CADmium https://arxiv.org/html/2507.09792).
+- License: MIT (repo). Source models fall under Onshape ToS (SGP-Bench notes this https://arxiv.org/html/2408.08313v3).
+
+### Fusion 360 Gallery (Willis et al. — SIGGRAPH 2021)
+- URLs: https://github.com/AutodeskAILab/Fusion360GalleryDataset ; license https://github.com/AutodeskAILab/Fusion360GalleryDataset/blob/master/LICENSE.md ; build123d port https://github.com/zalo/Fusion360GalleryDataset-build123d
+- Subsets: Reconstruction 8,625 sketch+extrude sequences; Segmentation 35,680 parts; Assembly 8,251 assemblies / 154,468 parts (README). Test slice used by CAD-Recode/cadrille = 1,725 models https://arxiv.org/html/2412.14042.
+- Formats: JSON sequences, STEP, meshes. A community port of 7,683 models as executable build123d scripts exists (zalo repo).
+- License: custom "Fusion 360 Gallery Dataset License" — non-commercial research only, no redistribution of the whole dataset, downstream restrictions must be preserved (LICENSE.md). This matters: several 2026 benchmarks (CADBench, P3D-Bench, CADEngBench) build on it.
+
+---
+
+## 1. Understanding / QA benchmarks over CAD programs
+
+### SGP-Bench — "Can Large Language Models Understand Symbolic Graphics Programs?" (Qiu, Liu et al., 2024; ICLR 2025)
+- URLs: paper https://arxiv.org/abs/2408.08313 (HTML v3 https://arxiv.org/html/2408.08313v3) ; code https://github.com/sgp-bench/sgp-bench ; HF `sgp-bench/sgp-bench`, `sgp-bench/sgp-mnist` ; project https://sgp-bench.github.io/
+- Task: program → multiple-choice QA (the model sees ONLY the code, never the render). SVG part: 1,085 programs / 4,340 questions (Kaggle SVG-icons). CAD part: 2,400 programs / 2,400 questions = 1,000 DeepCAD (3D) + 700 Fusion360 Reconstruction (3D complex) + 700 SketchGraphs (2D sketches). "Each of our 3 CAD subsets follows a different language syntax" — i.e. native DeepCAD / Fusion360 / SketchGraphs JSON-style DSLs, NOT CadQuery. Also SGP-MNIST (1,000 digit SVGs) and an invariance test (translation/rotation perturbation of the code).
+- Construction: questions + 4 options generated by GPT-4o from renders; human agreement checked on 500 sampled questions.
+- Metric: accuracy.
+- Results (Table 1, CAD accuracy 3D-DeepCAD | 3D-Fusion360 | 2D-SketchGraphs): Llama-3-8B 0.550|0.633|0.472; Mistral-7B-v0.3 0.495|0.551|0.481; Qwen-1.5-7B 0.486|0.560|0.426; DeepSeek-Coder-V2-16B 0.547|0.611|0.521; Llama-3-70B 0.634|0.694|0.619; Qwen-2-72B 0.692|0.753|0.669; GPT-3.5-Turbo 0.576|0.654|0.530; GPT-4-Turbo 0.716|0.762|0.694; GPT-4o 0.733|0.782|0.711; Claude 3.5 Sonnet 0.742|0.769|0.727. SGP-MNIST: GPT-4o ≈13%.
+- Pitfalls: MNIST result shows models do not "render in their head"; invariance test exposes sensitivity to numeric perturbation.
+- License: code MIT; "data license follows the license of the original data source" (Onshape ToS for DeepCAD, CC for others).
+- Note: CAD-Assistant (below) reports GPT-4o baseline 0.686 (2D) / 0.782 (3D) on SGP-Bench CAD QA.
+
+### MM-CAD (Bharathi et al. — Computer Graphics Forum 2026)
+- URL: https://onlinelibrary.wiley.com/doi/10.1111/cgf.70523 (paywalled; code/HF link not found)
+- 33,816 unique CAD models unified from eleven benchmark datasets with isometric renders, point clouds, human-curated multi-level captions, and 4,376 real hand-drawn sketches; positioned for retrieval / RAG. Output-language, metrics and LLM numbers: not found (abstract only).
+
+---
+
+## 2. Text/image → CAD **command sequence** (DeepCAD JSON style) — the Text2CAD family
+
+### Text2CAD (Khan et al. — NeurIPS 2024 spotlight)
+- URLs: paper https://arxiv.org/abs/2409.17106 (HTML https://arxiv.org/html/2409.17106) ; code https://github.com/SadilKhan/Text2CAD ; data https://huggingface.co/datasets/SadilKhan/Text2CAD ; project https://sadilkhan.github.io/text2cad-project/
+- Task: text (4 levels: L0 abstract, L1 beginner, L2 intermediate, L3 expert with relative values) → DeepCAD-style sequential sketch+extrude parameters (minimal JSON / vector). Built-in model is a 363M transformer, not an LLM.
+- Data: DeepCAD ~170K models; ~150K train / ~8K val / ~8K test; 4 prompts per model → ~600K train / ~32K test+val prompts; total ~660K annotations. Prompts: LLaVA-NeXT captions from multi-view renders + Mixtral (paper) two-stage CoT prompting from "minimal JSON" metadata. v1.1 (HF card) re-annotated with Qwen2-VL-14B + Qwen2.5-72B-Instruct-8bit, adds `description`/`keywords` fields, and filters training set to ~5% basic shapes (v1.0 had ~25% cuboids, ~8% cylinders). Renders: 10 Blender images per model (8 circular side views + top + bottom).
+- Metrics (expert prompts): F1 on lines/arcs/circles/extrusions via Hungarian matching of loops; CD ×10³ (point count not stated in the page I read); IR = share of sequences that are invalid (e.g. identical start/end points, zero extrusion depth). GPT-4V pairwise judge on multi-view renders (1,000 samples/level); user study with 5 CAD designers × 100 samples/level.
+- Baselines (Table 1, expert level): DeepCAD (adapted) Line/Arc/Circle/Extr F1 76.78/20.04/65.14/88.72, median CD 32.82, mean CD 97.93, IR 10.00; Text2CAD 81.13/36.03/74.25/93.31, median CD 0.37, mean CD 26.41, IR 0.93. GPT-4V preference lead grows from −2.8% (beginner) to +27.18% (expert).
+- Pitfalls the paper reports: LLaVA-NeXT sensitive to perspective distortion; DeepCAD imbalance; failures when prompt names an object instead of parameters.
+- License: paper CC BY 4.0; dataset CC BY-NC-SA 4.0 (HF card). Test split public.
+
+### Text2CAD-Bench ("A Benchmark for LLM-based Text-to-Parametric CAD Generation", May 2026)
+- URLs: https://arxiv.org/abs/2605.18430 (HTML https://arxiv.org/html/2605.18430v1). Code/leaderboard "to be released after accept" — no URL yet.
+- Task: text → **CadQuery** → STEP. 600 human-curated examples in 4 tiers: L1 200 (primitives), L2 200 (booleans/standard features), L3 100 (sweep/loft/shell/freeform), L4 100 real-world domains (industrial 40%, consumer 25%, medical 15%, architectural 10%, educational 10%). Each has two prompt styles: "geometric" (global→local→detail) and "sequence" (step-by-step). Authored by three mechanical-engineering contributors, human-in-the-loop.
+- Execution: Python 3.10, CadQuery 2.4, isolated env, 60 s timeout, up to 3 retries with error feedback; official APIs, no hyper-parameter tuning (temperature not stated).
+- Metrics: CD (bidirectional NN, 30,000 points/mesh, unit-bounding-box normalisation, ×10³); IR (fail / timeout / degenerate geometry); IoU at 256³ voxels; L4: GLM-4.6V judge on 8 multi-view renders, 0–10 across 5 feature questions.
+- Results (Table 1, L1→L3 CD / IR / IoU): GPT-5.2 44.31/11.1%/0.59 → 93.46/68%/0.23; Claude-4.5-Sonnet 52.62/20.3%/0.54 → 70.13/70%/0.25; DeepSeek-V3.2 53.15/13.3%/0.54 → 101.23/69%/0.17; Qwen3-max 84.54/21.4%/0.40 → 148.58/92%/0.07; MiniMax-M2.1 65.92/24.8%/0.45 → 114.33/91%/0.26; GLM-4.7 67.50/13.7%/0.44 → 92.70/83%/0.25; Gemini-3-Flash 66.82/17.7%/0.44 → 91.61/12.8%/0.20. Domain models: Text2CAD CD 220+ with IR 2–11%; CADFusion CD 190–242, IR 51–70%; Text2CADQuery CD 227–277, IR 29–90%. L4 IR: GPT-5.2 61%, Claude-4.5 54%, DeepSeek-V3.2 55%, Gemini-3-Flash 17%, MiniMax 81%. No ~7–9B open model reported.
+- Pitfalls: executability, geometric precision and feature understanding are "largely independent" dimensions; domain-specific models have low IR but poor CD (valid-but-wrong); geometric prompts win at L1–L2, sequence prompts at L3.
+- License: CC BY 4.0 (paper).
+
+### CAD-Llama (Fudan, 2025)
+- URLs: https://arxiv.org/abs/2505.04481 (HTML https://arxiv.org/html/2505.04481)
+- Task: text → SPCC ("Structured Parametric CAD Code", Python-like pseudocode with hierarchical descriptions) → DeepCAD sequence. Base LLaMA3-8B full FT + LoRA r=256 instruction tuning on 48K multitask examples.
+- Data: DeepCAD deduplicated 178K→100K; two-stage GPT-4o annotation (component-level from renders+sketches, then global); 70M GPT-4o tokens.
+- Metrics: text-to-CAD ACC_T (command acc + parameter acc + success ratio combined), median CD, MMD, JSD; unconditional COV/MMD/JSD/SR/Novelty; BLEU/ROUGE for captioning.
+- Results (text-to-CAD): GPT-4 ACC_T 20.03% / MCD 25.62; LLaMA3-8B 17.26% / 17.33; Text2CAD 69.91% / 20.64; CAD-Llama-INS 84.72% / 10.53.
+- Pitfalls: parameter errors in complex cases; text–CAD misalignment.
+- License: CC BY 4.0 (paper). Code/data URL: not found.
+
+### CADFusion (Microsoft — ICML 2025)
+- URLs: https://arxiv.org/abs/2501.19054 (HTML https://arxiv.org/html/2501.19054) ; code https://github.com/microsoft/CADFusion
+- Task: text → stringified sketch-and-extrude sequence. LLaMA-3-8B-Instruct + LoRA (r=32); alternates sequential learning (SL) and visual-feedback DPO (VF).
+- Data: DeepCAD (SkexGen-processed) 20K text–CAD pairs (GPT-4o captions, human refined) 90/5/5; ~1,500 preference pairs per VF round from 1,000 prompts.
+- Metrics: F1-Sketch (avg lines/arcs/circles), F1-Extrusion, CD, COV, MMD, JSD, IR (fail to render), LVM score = GPT-4o 0–10 judge, human rank (6 judges).
+- Results (Table 1): GPT-4o F1-S 82.96 / F1-E 85.72 / CD 68.50 / IR 37.93 / LVM 6.60; Text2CAD 63.94 / 92.13 / 30.23 / — / 3.37; CADFusion 85.22 / 92.79 / 19.89 / 6.20 / 8.96.
+- Pitfalls the paper states: GPT-4o used as judge may bias toward GPT-4o; LVMs degrade with multi-image input so single view used; 20K human-annotated beats 170K unannotated; human eval only ~50 unique samples.
+- License: CC BY-NC-SA 4.0.
+
+### FlexCAD (Microsoft — ICLR 2025)
+- URLs: https://arxiv.org/abs/2411.05823 (HTML https://arxiv.org/html/2411.05823) ; code https://github.com/microsoft/FlexCAD
+- Task: controllable generation at sketch/extrusion/face/loop/curve level via hierarchy-aware masking; Llama-3-8B LoRA (~3.4M params). DeepCAD 178,238 seqs 90/5/5, dedup per SkexGen.
+- Metrics: Novel, Unique, PV (prediction validity), JSD, COV, MMD, human "Realism".
+- Results (sketch-level): FlexCAD COV 65.6 / MMD 1.19 / JSD 0.82 / PV 93.4%; Hnc-CAD 62.4/1.21/1.07/72.6%; SkexGen 60.6/1.27/1.51/68.7%; GPT-4o 58.2/1.34/1.43/62.3%.
+- License: arXiv non-exclusive.
+
+### CAD-Editor (Microsoft — ICML 2025)
+- URLs: https://arxiv.org/abs/2502.03997 (HTML https://arxiv.org/html/2502.03997) ; code https://github.com/microsoft/CAD-Editor ; weights https://huggingface.co/microsoft/CAD-Editor
+- Task: (original CAD sequence, edit instruction) → edited sequence. 120K synthetic triplets from DeepCAD via design-variation models + LVLM difference captioning; DeepCAD 90/5/5; test 2,000 examples manually checked, 5 outputs each → 10,000 evaluated; 2,000 rated by 5 crowd workers each. Llama-3-8B-Instruct LoRA r=32.
+- Metrics: VR (valid ratio), JSD, CD, D-CLIP (directional CLIP), H-Eval (binary success). JSD/CD/D-CLIP ×10².
+- Results: GPT-4o-Basic VR 63.2 / JSD 1.10 / CD 2.30 / D-CLIP −1.08 / H-Eval 7.22; GPT-4o-IC 84.5/0.70/1.55/−0.11/15.6; CAD-Editor 95.6/0.65/1.18/0.11/43.2.
+- License: CC BY-NC-SA 4.0.
+
+### CAD-GPT (AAAI 2025)
+- URLs: https://arxiv.org/abs/2412.19663 (HTML https://arxiv.org/html/2412.19663v1) ; code https://github.com/SiyuWang0906/CAD-GPT ; project https://OpenIWIN.github.io/CAD-GPT/
+- Task: single image or text → DeepCAD sequence with spatial-reasoning tokens; LLaVA-1.5-7B.
+- Data: DeepCAD dedup; image→CAD 162K (fixed-angle OpenCascade renders); text→CAD 18K (InstructGPT-generated, GPT-4o filtered, manually curated).
+- Metrics: CD, IR, ACC_cmd, ACC_param.
+- Results: GPT-4 image IR 64.37% / CD 62.64, text IR 76.97% / CD 187.52; DeepCAD image IR 23.16% / CD 23.78; CAD-GPT image IR 1.61% / CD 9.77, text IR 7.43% / CD 28.33.
+- License: arXiv non-exclusive.
+
+### CAD-MLLM / Omni-CAD (2024)
+- URLs: https://arxiv.org/abs/2411.04954 (HTML https://arxiv.org/html/2411.04954) ; code https://github.com/CAD-MLLM/CAD-MLLM ; project https://cad-mllm.github.io/
+- Task: text / multi-view image / point cloud (any combination) → command sequence; Vicuna-7B LoRA.
+- Data: Omni-CAD 453,220 models after augmentation from ABC via Onshape API (DeepCAD methodology); captions by InternVL2-26B from 4 views; 9:1 split → 425,726 train / 27,494 test.
+- Metrics: CD on [−0.5,0.5]³, F-score @0.05, normal consistency, plus topology metrics SegE, DangEL (dangling-edge length), SIR (self-intersection ratio), FluxEE (flux enclosure error).
+- Results: point-cond CD 1.85 vs DeepCAD 4.51 (×100), F 90.88 vs 71.83; image-cond CD 3.77 vs InstantMesh 5.38; text: user-study 4.16/5 vs Michelangelo 1.16/5.
+- License: CC BY 4.0 (paper).
+
+### GenCAD (2024/2025)
+- URLs: https://arxiv.org/abs/2409.16294 (HTML https://arxiv.org/html/2409.16294) ; project https://gencad.github.io/
+- Task: image → DeepCAD command sequence via contrastive CSR + diffusion prior. DeepCAD 178,238 → 168,674 valid solids; 152,530 / 8,515 / 7,629; 5 scale variants per model → 845,105 grayscale 448² renders + sketch versions.
+- Metrics: CD (2,000 points), IR, COV/MMD/JSD, Recall@B. Results: command acc 99.51%; image retrieval R@2048 60.77% vs ResNet 3.91%; conditional COV 81.37 (image). GPT-4-class baselines: not found.
+- License: arXiv non-exclusive. Its command sequences are the source of **GenCAD-Code** (below).
+
+### CADCrafter (CVPR 2025)
+- URLs: https://arxiv.org/abs/2504.04753 (HTML https://arxiv.org/html/2504.04753v1)
+- Task: unconstrained single/multi-view photos → command sequence (latent diffusion + DPO with code-checker feedback). Train on DeepCAD renders (8×4-view sets); RealCAD test = 150 DeepCAD-test models 3D-printed and photographed (4 views, iPhone).
+- Metrics: ACC_cmd, ACC_para, median CD, IR (fail to compile). DeepCAD: 84.62% / 73.31% / 0.026 / 0.036 vs Img2CAD 80.57% / 68.77% / 0.160 / 0.288; RealCAD 83.18% / 66.89%.
+- License: CC BY 4.0. Code: not found.
+
+### Img2CAD (VLM-assisted conditional factorization — SIGGRAPH Asia 2025)
+- URLs: https://arxiv.org/abs/2408.01437 (HTML https://arxiv.org/html/2408.01437) ; code https://github.com/qq456cvb/Img2CAD
+- Task: single image → sketch-extrude program; discrete structure by fine-tuned Llama-3.2, continuous params by regressor; GPT-4V/4o for dataset creation. Data: CAD-ified ShapeNet — 1,026 chairs, 3,243 tables, 305 cabinets (PartNet + GPT-4V, ~30% kept after manual review).
+- Metrics: CD on uniformly sampled surface points; part Seg-Acc, Seg-mIoU. Chair CD: GPT-4o 0.3806; DeepCAD-End2End 0.2346; Img2CAD 0.0984.
+- License: arXiv non-exclusive.
+
+---
+
+## 3. Text/image/point-cloud → **executable CAD code** (CadQuery / PythonOCC / FreeCAD / FeatureScript / OpenSCAD)
+
+### CADPrompt + CADCodeVerify (Alrashedy et al. — ICLR 2025)
+- URLs: https://arxiv.org/abs/2410.05340 (HTML https://arxiv.org/html/2410.05340) ; code+data https://github.com/Kamel773/CAD_Code_Generation ; ICLR page https://iclr.cc/virtual/2025/poster/30576
+- Task: NL prompt → **CadQuery** (chosen over OpenSCAD because Python-native and more concise). 200 objects "selected from a collection of modular CAD objects" of DeepCAD (Wu et al. 2021), stratified by mesh complexity / geometric complexity / compile difficulty, with simple vs complex splits. Prompts: 2 annotators write, a 3rd picks the better, a 4th verifies all 200; ground-truth CadQuery by one CAD expert validated against STL in Blender. Each entry: STL, OBJ, JSON commands, NL prompt, expert Python.
+- Metrics: Point-cloud distance (bidirectional mean NN, 1,000 sampled points, unit-cube normalisation, ICP alignment); Hausdorff; IoGT (intersection over ground-truth bounding volume); compile rate. Refinement ≤2 iterations; compile-error repair loop.
+- Results (few-shot, best refinement, IQR in parentheses): GPT-4 IoGT 0.944 (0.028) / PCD 0.127 (0.135) / HD 0.419 (0.356) / compile 96.5%; Gemini 1.5 0.939 / 0.147 / 0.492 / 85.0%; CodeLlama 0.935 (IQR 0.957) / 0.185 (IQR 1.620) / 0.582 / 73.5%. CADCodeVerify gives 7.30% PCD reduction and +5.5% success on GPT-4 vs 3D-Premise self-refinement; ~9% compile-rate gain on Hard split at first refinement where baselines drop 20%.
+- Pitfalls: geometric-solver baseline "accessed ground truth" (upper bound); huge IQR for CodeLlama means medians hide a bimodal distribution; only 1,000 points for PCD.
+- License: repo has no license file (README). Test set fully public (it IS the dataset) — heavily reused: CADmium, EvoCAD, CADTestBench, IterCAD all evaluate on it.
+
+### CAD-Coder — "An Open-Source VLM for CAD Code Generation" (Doris et al., IDETC 2025) + GenCAD-Code
+- URLs: https://arxiv.org/abs/2505.14646 (HTML https://arxiv.org/html/2505.14646) ; code https://github.com/anniedoris/CAD-Coder ; data https://github.com/anniedoris/GenCAD-Code , https://huggingface.co/datasets/CADCODER/GenCAD-Code , https://huggingface.co/datasets/CADCODER/real_photo_test ; weights https://huggingface.co/CADCODER/CAD-Coder
+- Task: single isometric image → **CadQuery**. GenCAD-Code: 163,671 image–script pairs converted from GenCAD/DeepCAD .h5 vectors by `h5tocadquery.py`; 147,289 train / 7,355 test / 9,027 val; avg script 611 tokens, 99.9% < 3,000. Real-photo test: 400 photos of 3D-printed objects (repo) — paper evaluated 5 printed objects. Base LLaVA-1.5 (Vicuna-13B + CLIP-ViT-L-336).
+- Metrics: VSR = % scripts that execute; IoU_best = IoU after centroid removal, RMS-radius-of-gyration scaling, PCA alignment over all valid rotation combos ("SolidAlign" in repo).
+- Results (100 test samples): CAD-Coder VSR 100% / IoU_best 0.675; GPT-4.5 84% / 0.524; Qwen2.5-VL-72B 94% / 0.352; LLaVA-v1.5-13B 0% / 0.0.
+- Pitfalls: only 100 test images; fillet unseen in training tested via prompting.
+- License: code Apache-2.0; GenCAD-Code repo has no license stated; no dedup/filtering documented.
+
+### Text-to-CadQuery (2025)
+- URLs: https://arxiv.org/abs/2505.06507 (HTML https://arxiv.org/html/2505.06507) ; code https://github.com/Text-to-CadQuery/Text-to-CadQuery
+- Task: Text2CAD prompts → **CadQuery**. ~170K pairs: Text2CAD minimal-JSON translated to CadQuery by Gemini 2.0 Flash with execution-error feedback loop + human validation; 90/5/5.
+- Metrics: CD (10,000 points/mesh, normalised, ×10³, mean and median); IR (execution failure); "evaluation score" = Gemini 2.0 Flash shown Blender renders of generated vs GT STL and asked "same 3D object?" → % Yes. (The abstract's "top-1 exact match 69.3%" is this Gemini score for Qwen2.5-3B, not string match.)
+- Results (Table 3; median CD / mean CD / IR / Gemini): CodeGPT-small-124M 0.234/13.520/9.4%/60.28%; GPT-2-medium 0.223/12.567/15.6%/60.31%; GPT-2-large 0.221/12.175/13.8%/63.61%; Gemma3-1B 0.204/11.609/8.4%/66.86%; Qwen2.5-3B 0.191/10.229/6.5%/69.30%; Mistral-7B (LoRA) 0.219/11.835/1.32%/65.38%. Text2CAD transformer baseline median CD 0.370, 58.80%.
+- Pitfalls: 7B Mistral underperforms 3B (authors blame insufficient data); numerically unstable arcs with near-collinear points. Text2CAD-Bench later reports this model at CD 227–277 with IR 29–90% on its OOD tiers.
+- License: CC BY 4.0.
+
+### CAD-Coder — "Text-to-CAD Generation with Chain-of-Thought and Geometric Reward" (2025; distinct from Doris et al.)
+- URLs: https://arxiv.org/abs/2505.19713 (HTML https://arxiv.org/html/2505.19713). Code: not found.
+- Task: Text2CAD prompts → CadQuery. Data: 110K text–CadQuery–3D triplets built by having DeepSeek-V3 generate candidates, executing, keeping lowest-CD; tiers 8K (CD<1e-4), 70K (CD<1e-3), 32K lower; plus 1.5K CoT samples. Base Qwen2.5-7B-Instruct; GRPO with reward = λ_geo·R_geo + λ_fmt·R_fmt, R_geo piecewise (1.0 if CD<1e-5, 0 if CD>0.5, linear between).
+- Metrics: mean CD, median CD, IR.
+- Results: Text2CAD 29.29 / 0.37 / 3.75%; GPT-4o 133.52 / 45.91 / 93.00%; DeepSeek-V3 186.69 / 107.57 / 51.96%; CAD-Coder (full) 6.54 / 0.17 / 1.45%.
+- Pitfalls stated: thin structures suffer "reward hacking due to sparse point sampling"; multi-component alignment and internal cavities fail; extrude-vs-cut confusion. (Note GPT-4o IR 93% here vs 37.93% in CADFusion vs 11% in Text2CAD-Bench — prompt/format dependence is large.)
+- License: arXiv non-exclusive.
+
+### CAD-Recode (Rukhovich et al. — ICCV 2025)
+- URLs: https://arxiv.org/abs/2412.14042 (HTML https://arxiv.org/html/2412.14042) ; code https://github.com/filaPro/cad-recode ; data https://huggingface.co/datasets/filapro/cad-recode ; project https://cad-recode.github.io/
+- Task: point cloud (256 FPS points) → **CadQuery**; Qwen2-1.5B + linear point projector.
+- Data: 1M procedurally generated CadQuery scripts (random sketch primitives + booleans + extrudes) filtered by Python syntactic checks and BRepCheck_Analyzer; HF card: ~1M train / ~1K val, CC-BY-NC-4.0. Eval: DeepCAD test 8,046; Fusion360 1,725; CC3D 2,973 real scans.
+- Metrics: CD ×10³ with 8,192 points (mean and median); IoU from meshes (%); IR. Test-time: 10 candidates, pick min-CD (so numbers are best-of-10 against GT — an oracle selection).
+- Results: DeepCAD mean CD 0.30 / median 0.16 / IoU 92.0 / IR 0.4%; Fusion360 0.35 / 0.15 / 87.8 / 0.5%. Prior SOTA CAD-SIGNet: DeepCAD median 0.28, IoU 77.6; Fusion360 0.48, 65.6.
+- Pitfalls: best-of-10 selection uses GT; procedural data → domain gap (cadrille reports ~60% CC3D IoU for SFT-only).
+- License: paper arXiv non-exclusive; dataset CC BY-NC 4.0.
+
+### cadrille (ICLR 2026)
+- URLs: https://arxiv.org/abs/2505.22914 (HTML https://arxiv.org/html/2505.22914) ; code https://github.com/col14m/cadrille ; weights https://huggingface.co/maksimko123/cadrille , https://huggingface.co/maksimko123/cadrille-rl
+- Task: point cloud / multi-view image / text → CadQuery in one Qwen2-VL-2B; SFT on CAD-Recode 1M + DeepCAD/Text2CAD ~160K, then online RL (GRPO / Dr.CPPO) with reward = 10·IoU − 10·[invalid]; hard-example mining (avg reward < 7.5).
+- Eval sets: DeepCAD 8,046 (PC/Img/Text), Fusion360 1,725, CC3D 2,973, Omni-CAD 27K+.
+- Metrics: median CD (8,192 pts, ×10³), IoU %, IR.
+- Results (IoU, SFT→RL): PC DeepCAD 87.1→90.2, Fusion360 79.8→85.0, CC3D 61.8→67.9; Image DeepCAD 86.1→92.2, Fusion360 77.6→84.6, CC3D 56.1→65.0. IR ~10% (SFT) → <0.2% (RL).
+- Pitfalls: different datasets have incompatible operation vocabularies so naive mixing fails; RL sidesteps the need for GT sequences.
+- License: Apache-2.0 (repo).
+
+### CADmium (Mila — 2025, v2 Jan 2026)
+- URLs: https://arxiv.org/abs/2507.09792 (HTML https://arxiv.org/html/2507.09792) ; data/models https://huggingface.co/collections/chandar-lab/cadmium-6866b402be81f39321af98d4
+- Task: text → JSON CAD sequence (NOT CadQuery), fine-tuning Qwen2.5-Coder 1.5B/3B/7B/14B with greedy decoding. 170K+ DeepCAD models re-annotated with GPT-4.1 "human-like" descriptions; filtered so cuboids 25%→5%, cylinders 8%→5%; 118,299 / 8,925 / 8,046. Annotation quality judged by Gemma-3-12B and Mistral-Small-3.2-24B on 10K triplets.
+- Metrics: IR; median CD (points not stated); F1 (Hungarian on loops); new: Sphericity Discrepancy Ψ=(6V)^(1/3)/(π^(1/3)·s-ish, see paper), Discrete Mean Curvature Difference (dihedral angles), Exact Euler Characteristic Match (χ=V−E+F); also SegE/DangEL/SIR/FluxEE from CAD-MLLM.
+- Results — CADPrompt OOD (Table 1, Text2CAD vs Qwen2.5-Coder-14B): IR 1.57% vs 6.28%; median CD 127.70 vs 116.75; Arc F1 0.00 vs 0.22; Circle F1 0.31 vs 0.61; DMCD 13.39 vs 8.42; EECM 0.66 vs 0.68. CADmium test by scale 1.5B/3B/7B/14B: IR 11.86/8.51/5.93/3.33%; median CD 0.31/0.24/0.27/0.23; EECM 0.87/0.89/0.88/0.89. Fusion360: IR 22.62/23.38/16.79/12.58%; median CD 110.18/76.34/92.07/76.05.
+- Pitfalls: "simple metrics often fail to reflect quality"; point-cloud metrics ignore internal edges; median-CD chosen because means are outlier-dominated; 7B non-monotone vs 3B.
+- License: dataset+code MIT, models Apache-2.0.
+
+### ProCAD — "Clarify Before You Draw" (ICML 2026)
+- URLs: https://arxiv.org/abs/2602.03045 (HTML https://arxiv.org/html/2602.03045v2) ; code+data https://github.com/BoYuanVisionary/Pro-CAD
+- Task: possibly-ambiguous text → (clarifying questions) → CadQuery; two Qwen2.5-7B-Instruct agents. Test set 2,469 (≈1:1 unambiguous/ambiguous), from DeepCAD shapes with CAD-Recode CadQuery programs; ambiguity injected by GPT-5-mini (under-specified / inconsistent), kept only if original CD<2e-4, perturbed CD>2e-4 and ≥10× degradation.
+- Metrics: mean/median CD ×10³, IR, LLM-judge Efficiency/Resolution.
+- Results: Claude Sonnet 4.5 mean CD 3.10 / median 0.09 / IR 4.8%; GPT-4o-mini 9.98 / 0.14 / 7.5%; ProCAD 0.85 / 0.08 / 0.9%.
+- License: CC BY 4.0.
+
+### CAD-RL / ExeCAD — "From Intent to Execution" (2025)
+- URLs: https://arxiv.org/abs/2508.10118 (HTML https://arxiv.org/html/2508.10118v1). Code/HF: not found.
+- Task: NL + structured design language (+ render) → CadQuery; Qwen2.5-VL with CoT cold start + RL (executability, geometric, GPT-4o "external" rewards). ExeCAD = 16,540 samples refined from GenCAD-Code + Text2CAD with GPT-4o in 4 stages.
+- Metrics: executability (binary), IoU %, mean/median CD.
+- Results (structured input): GPT-4o IoU 47.7 / CD 65.52 / exec 72.72%; Qwen2.5-VL 31.6 / 114.64 / 62.30%; InternVL3 43.5 / 82.72 / 72.03%; CAD-Coder (re-impl.) 70.4 / 29.63 / 95.44%; CAD-RL 78.7 / 15.21 / 98.83%.
+- License: not found.
+
+### CME-CAD / CADExpert (Dec 2025)
+- URL: https://arxiv.org/abs/2512.23333
+- CADExpert: 17,299 instances of orthographic projections with dimension annotations + expert CoT + executable CadQuery + renders; multi-expert SFT + RL. Numbers/license/code: not found (abstract only).
+
+### EvoCAD (Oct 2025)
+- URLs: https://arxiv.org/abs/2510.11631 (HTML https://arxiv.org/html/2510.11631v1) ; code https://github.com/toprei/evo-cad
+- Task: evolutionary search over CadQuery programs with GPT-4V/GPT-4o generation and o3-mini fitness ranking; evaluated on CADPrompt (~80% watertight subset). Population 6, 4 generations.
+- Metrics: PCD, HDD, IoU, Dice, plus Topology Error |χ(O)−χ(Ô)| and Topology Correctness (binary Euler match).
+- Pitfall (explicit): "objects with incorrect topologies can have better spatial metrics than those with correct topologies"; non-watertight outputs cannot be scored on volume/topology at all.
+- License: arXiv non-exclusive.
+
+### Seek-CAD (ICLR 2026)
+- URLs: https://arxiv.org/abs/2505.17702 (HTML https://arxiv.org/html/2505.17702) ; data https://github.com/Sunny-Hack/Seek-CAD
+- Task: text → Python (custom CADShape class over PythonOCC) with DeepSeek-R1 local inference + Gemini-2.0 visual/CoT self-refinement; training-free. Dataset 40K models from ABC via Onshape API/FeatureScript in SSR (Sketch, Sketch-feature, Refinements: extrude/revolve/fillet/chamfer/shell). Eval 500 models held out.
+- Metrics: CD, HD, IoGT, G-Score (Gemini 1–5), Novelty (>80% similarity to corpus), pass@k compile.
+- Results (best of compared 3D-PreMise / CADCodeVerify / CAD-Llama): CD 0.1979, HD 0.5566, IoGT 0.7226, G-Score 3.5185, Novelty 64.04%.
+- License: arXiv non-exclusive; repo license not stated.
+
+### Query2CAD (2024)
+- URLs: https://arxiv.org/abs/2406.00144 (HTML https://arxiv.org/html/2406.00144) ; code+data https://github.com/akshay140601/Query2CAD
+- Task: NL → **FreeCAD Python macros**, run via PyAutoGUI-driven FreeCAD, BLIP-2/VQAScore(≥0.9) feedback loop. 57 hand-written queries (21 easy / 20 medium / 16 hard).
+- Metric: success = exact match to spec by manual inspection ("any deviation … is a failure").
+- Results: GPT-4-Turbo 53.6% → 73.2% (1 refinement) → 76.7%; GPT-3.5-Turbo 32.7% → 53.4% (3 refinements); GPT-4-Turbo by tier easy 95.23% / medium 70% / hard 41.7%; CodeLlama-70B "extremely bad".
+- Pitfalls: 69% of failures are non-executable code; BLIP2 captions weaker than human feedback; tiny set.
+- License: arXiv non-exclusive.
+
+### OpenECAD (Computers & Graphics 2024)
+- URLs: https://arxiv.org/abs/2406.09913 (HTML https://arxiv.org/html/2406.09913)
+- Task: 2D view(s) (default / transparent / orthographic, 640×480) → Python-style DSL over PythonOCC (`add_sketchplane`, `add_line`, `add_extrude` …). TinyLLaVA with OpenELM-450M / Gemma-2B / Phi-2 → 0.55B/0.89B/2.4B/3.1B. Data from DeepCAD: 100K/150K/200K pairs at 1K/2K/3K token limits.
+- Metric: custom 100-point score (executability 10 + curve accuracy 45 + loop count 5 + loop quality 40), 625 test designs.
+- Results (default view): 0.55B 83.89; 0.89B 86.05; 2.4B 88.80; 3.1B 85.84; GPT-4o-mini 39.72.
+- Pitfalls: context-length truncation, concave/convex confusion. Data "available on request" only.
+
+### LLM4CAD (Li, Sun, Sha — IDETC 2024 / JCISE 25(2) 2025)
+- URLs: https://asmedigitalcollection.asme.org/computingengineering/article/25/2/021005/1208543/ ; IDETC https://asmedigitalcollection.asme.org/IDETC-CIE/proceedings-abstract/IDETC-CIE2024/88407/V006T06A015/1208976 ; preprint PDF https://sidilab.net/wp-content/uploads/2025/01/llm4cad_jcise_preprint.pdf (pages 403'd / binary for me — details below from secondary summaries of the paper)
+- Task: text / text+sketch / text+image of mechanical parts → **CadQuery** → STL. Synthetic parts in 5 classes with MTurk NL descriptions incl. dimensions: 621 shafts, 671 nuts, 692 flanges, 680 springs, 661 gears.
+- Metrics: Cap1 = syntactically correct program rate; Cap2 = IoU of parsed 3D shape. Exact GPT-4 / GPT-4V per-class numbers: not found in accessible text. Finding: GPT-4V text-only beats multimodal on average; multimodal helps on springs/gears. Follow-up "LLM4CAD Fine-Tuned: Dataset and Experiments" (JMD preprint https://sidilab.net/wp-content/uploads/2025/01/llm4cad_jmd_preprint.pdf).
+- License / data link: not found.
+
+### 3D-PreMise (Jan 2024)
+- URL: https://arxiv.org/abs/2401.06437
+- LLM program synthesis for industrial parametric shapes with visual self-correction; dataset of test programs with problem descriptions + GT code. Size/API/metrics/numbers: not found on abstract page. Used as the self-refinement baseline in CADPrompt and Seek-CAD.
+
+### "Don't Mesh with Me" (Nov 2024)
+- URLs: https://arxiv.org/abs/2411.15279 (HTML https://arxiv.org/html/2411.15279) ; project mxmws.github.io/dont_mesh_with_me
+- Task: text → CSG as Python over the OpenMC library (half-spaces + cylinders), NOT OpenSCAD. 37,220 ABC parts converted BREP→CSG via GEOUNED; GPT-4o captions; DeepSeek-Coder-1B fine-tuned. Results: 96.5% syntax+logic correct, 82.2% surface utilisation. No external baselines.
+
+### Human-in-the-Loop OpenSCAD evaluation (Sep 2025)
+- URL: https://arxiv.org/abs/2509.07010 (HTML https://arxiv.org/html/2509.07010)
+- Single L-bracket case study, **OpenSCAD**, GPT-4 only, four input modalities. Metrics: triangle count, area/volume ratio, Euler characteristic, volumetric accuracy, surface alignment, dimensional fidelity, Hausdorff, PCA/ICP alignment. Not a benchmark; CC BY 4.0. (An MHE/OpenSCAD paper in Procedia CS 2026 exists https://www.sciencedirect.com/science/article/pii/S1877050926000773 — 403 for me.) **No dedicated multi-model OpenSCAD LLM benchmark was found; the only OpenSCAD-format multi-model evaluation is P3D-Bench (below).**
+
+### CAD2Program (AAAI 2025)
+- URLs: https://arxiv.org/abs/2412.11892 (HTML https://arxiv.org/html/2412.11892) ; project https://manycore-research.github.io/CAD2Program
+- 2D engineering drawing (raster) → Python dataclass shape program of cabinet primitives; Mini-InternVL-1.5-2B; 368K cabinets (364K/2K/2K). Metrics: model-retrieval acc 93.80%, reconstruction F1 (IoU>0.5 TP) 82.76%, parameter acc 97.21%. CC BY 4.0. Domain-specific (furniture), not general CAD.
+
+### Ortho2CAD (Jul 2026)
+- URL: https://arxiv.org/abs/2607.08891
+- Orthographic drawings → CadQuery; >1M synthetic drawings (pythonOCC first-angle projections from STEP) + 100 manually dimensioned; metrics valid-code % and mean IoU; GPT-5.5 best with 100% valid code. Per-model table/license: not found.
+
+### CAD-Assistant (ICCV 2025)
+- URLs: https://arxiv.org/abs/2412.13810 (HTML https://arxiv.org/html/2412.13810) ; code https://github.com/dimitrismallis/CAD-Assistant
+- Tool-augmented GPT-4o agent executing Python on **FreeCAD**. Evaluated on SGP-Bench CAD QA (700 2D + 3D), SketchGraphs autoconstraining (700), hand-drawn parameterization. Results: 2D QA 0.686→0.791; 3D QA 0.782→0.857; autoconstrain PF1 0.693→0.979, CF1 0.274→0.484. License CC BY-NC-ND 4.0.
+
+---
+
+## 4. Large-scale program datasets with new representations (2026)
+
+### CADFS (CVPR 2026)
+- URLs: https://arxiv.org/abs/2605.01925 (HTML https://arxiv.org/html/2605.01925v1) ; data https://huggingface.co/datasets/VladPyatov/CADFS ; project https://voyleg.github.io/cadfs/
+- 451K Onshape public designs (subset of ABC, superset of DeepCAD/Text2CAD) as executable **FeatureScript** across 15 operations (sketch, extrude, revolve, sweep, loft, fillet, chamfer, shell, hole, booleans, patterns…). Text by gpt-oss-120B annotator+reviewer. Qwen2-VL-2B (also Qwen3-8B). Test: 7,278 DeepCAD-test + ~9K new all-operation designs.
+- Metrics: CD, Edge-CD, normal consistency, COV/MMD/JSD, IR. Text-to-CAD on DeepCAD test: −40% CD, −64% ECD vs cadrille. ~15% of reconstructed programs fail validation.
+- License: HF dataset CC BY 4.0 (search result); paper arXiv non-exclusive.
+
+### CADEvolve (Feb 2026)
+- URLs: https://arxiv.org/abs/2602.16317 (HTML https://arxiv.org/html/2602.16317) ; code https://github.com/zhemdi/CADEvolve
+- Program-evolution pipeline (gpt-5-mini) → 7,945 generators → ~8×10⁵ programs → ~1.3M canonicalised CadQuery scripts (SFT set 2,720,481). Qwen2-VL-2B, SFT + Dr.GRPO/CPPO.
+- Image2CAD results (median CD ×10³ / IoU % / IR %): cadrille-RL DeepCAD 0.17/92.2/0.1, Fusion360 0.17/84.6/0.1, MCB 0.87/47.6/2.5; CADEvolve-C big RL1 0.15/92.6/0.2, 0.16/87.2/0.5, 0.62/51.4/2.3; RL2 0.16/91.1/0.1, 0.16/84.0/0.2, 0.52/55.2/0.4. IR here = non-compiling OR non-watertight.
+- License: CC BY 4.0.
+
+### HistCAD (Feb 2026)
+- URLs: https://arxiv.org/abs/2602.19171 (HTML https://arxiv.org/html/2602.19171) ; anon code https://anonymous.4open.science/r/HistCAD-68C2
+- 170,236 executable sequences in a software-independent IL with 19 constraint types + fillet/chamfer/helix: DeepCAD 153,534 + Fusion360 8,609 + Industrial 8,093 (research-agreement only). Execution adapters: JiuShao POWER and Fusion 360 API. Benchmark = constraint-aware editability: ER, cPCSR, OES = ER×cPCSR. Qwen3-8B LoRA on full HistCAD: OES 73.33%, ER 83.67%, cPCSR 87.36%. GPT-5.5 qualitative only.
+
+### UniCAD (Jun 2026)
+- URLs: https://arxiv.org/abs/2606.05058 (HTML https://arxiv.org/html/2606.05058)
+- 1,448,150 models (DeepCAD + Text2CAD + CAD-Recode) with CadQuery, 8-view renders, sketch-style images, point clouds+normals, QA pairs; test = DeepCAD 8,046. Qwen2-VL-2B.
+- Metrics: CD ×10³ (avg squared NN), voxel IoU at 0.02 resolution in unit cube (≈50³), 4-way QA accuracy.
+- Results: UniCAD-MLLM PC 0.17/89.7, text 0.20/86.3, multi-view 0.17/89.8, sketch 0.22/84.2, QA 90.0%; CAD-Recode PC 0.18/87.1; cadrille text 0.20/82.1; Text2CAD 0.37/71.5; GPT-4o QA 0.77. Fusion360 zero-shot PC IoU 84.3 vs CAD-Recode 79.1.
+- Pitfall: sketch-extrude only; no fillet/chamfer/freeform. CC BY 4.0; release promised.
+
+---
+
+## 5. 2026 evaluation-first benchmarks
+
+### CADBench (DeCoDELab, May/Jun 2026) — "A Multimodal Benchmark for AI-Assisted CAD Program Generation"
+- URLs: https://arxiv.org/abs/2605.10873 (HTML https://arxiv.org/html/2605.10873v1) ; data https://huggingface.co/datasets/DeCoDELab/CADBench (78.4 GB parquet, DOI 10.57967/hf/8666)
+- 18,000 samples, six families from DeepCAD, Fusion 360, ABC, MCB, Objaverse; five input modalities (single-view grayscale, multi-view, photorealistic PBR, clean mesh, Gaussian-noised mesh) → CadQuery. 1.4M+ generations over 11 systems.
+- Metrics (all after continuous Procrustes alignment; GT scaled to max bbox edge 1.0): symmetric CD on surface samples; voxel IoU; Surface-IoU (bidirectional coverage, τ = 1% of bbox diagonal); VSR = executes AND valid shape (syntax errors, failed ops, non-manifold all invalid); token count; op count.
+- Results (Table 2 aggregate: IoU / SIoU / CD / VSR): CADFit 0.895/0.685/0.038/1.000; CAD-Recode 0.506/0.462/0.064/0.919; CADEvolve 0.611/0.456/0.071/0.974; cadrille 0.555/0.482/0.062/0.945; Claude Opus 4.7 0.306/0.110/0.106/0.807; Gemini 3.1 Pro 0.295/0.108/0.092/0.737; GPT-5.4 0.124/0.028/0.114/0.517; Kimi K2.6 0.142/0.041/0.116/0.616; Qwen3.5-27B 0.011/0.002/0.179/0.336; Qwen3.5-9B 0.000/0.000/0.199/0.099; CAD-Coder 0.288/0.113/0.159/0.951.
+- Pitfalls: mesh-conditioned ≫ image-conditioned; specialists collapse under modality shift; metric rankings disagree across models; Appendix D notes some source datasets' licences constrain redistribution.
+- License: paper arXiv non-exclusive; HF card license not stated.
+
+### BenchCAD (May 2026) — "A Comprehensive, Industry-Standard Benchmark for Programmatic CAD"
+- URLs: https://arxiv.org/abs/2605.10865 (HTML https://arxiv.org/html/2605.10865v1) ; data HF `BenchCAD/BenchCAD` ; code https://github.com/BenchCAD/BenchCAD-main ; third-party leaderboards https://llm-stats.com/benchmarks/benchcad
+- 17,900 execution-verified CadQuery programs, 106 industrial part families hand-crafted by domain experts, 49% anchored to ISO/DIN/EN/ASME/IEC tables. Tasks: Vision2Code, Code Edit, QA (code- or image-conditioned, 6–12 templates/family). 10 families held out OOD. Sandbox subprocess, 30 s wall clock; parse/runtime/timeout/degenerate volume → excluded.
+- Metrics: voxel IoU at 256³ (+ rotation-invariant over 24 axes), CD, Hausdorff, Feature-F1, essential-op recall, execution rate; Edit accuracy = clip((IoU(gen,tgt) − IoU(orig,tgt)) / (1 − IoU(orig,tgt)), 0, 1); QA accuracy (±5% for ratios, exact for integers). Contamination control: blind baselines (black image / no code) as floors; standard-table parameter sampling.
+- Results Vision2Code (IoU / CD / exec): CADEvolve v3 0.7497/0.0080/92.7%; cadrille-RL 0.0683/0.3507/91.2%; Claude Opus 4.7 (thinking) 0.2670/0.0240/90.4%; Gemini 3.1 Pro (thinking) 0.2790/0.0250/79.8%; GPT-4o 0.1884/0.0623/87.0%; Qwen3-VL-2B RL (IID) 0.7520/0.0041/98.9%, OOD 0.7140/0.0047/99.1%. Code Edit: GPT-5.3 0.865, Claude Opus 4.7 0.853, Gemini 3.1 Pro 0.837. Vision QA: Gemini 0.587, Claude 0.526, GPT-5.3 0.513; Code QA 0.80+.
+- Pitfalls: ~15–20 pt code-vs-image QA gap ("holistic spatial and detailing deficit"); misses twist-extrude/loft/helix; edits break unrelated features.
+- License: data CC BY 4.0, eval code MIT.
+
+### CADTestBench — "Text-to-CAD Evaluation with CADTests" (May 2026)
+- URLs: https://arxiv.org/abs/2605.07807 (HTML https://arxiv.org/html/2605.07807v1) ; code https://github.com/dimitrismallis/CADTestBench ; data https://huggingface.co/datasets/dimitrismallis/CADTestBench
+- Replaces CD/IoU with executable boolean B-Rep predicates (CadQuery API) derived from the prompt, not from the reference: 200 CADPrompt programs → 5,937 tests (3,037 detailed / 2,900 abstract, ~15/sample) in 6 categories, plus 1,275 CAD mutants; tests authored by Claude-Sonnet-4.6 over 4 refinement rounds; mutation score >90%.
+- Metrics: Pass-Rate (all tests pass), Requirement Score, per-category Acc, IR (runtime errors count as failures), Mutation Score. Human agreement: CADTests F1 0.938 vs VLM-judge 0.549 vs CD AUC 0.663.
+- Results (detailed prompts, IR / RS / PR): Text2CAD 0.005/0.405/0.025; CadCodeVerify(GPT-4.1) 0.025/0.794/0.410; DeepSeek-V3 ReAct 0.035/0.752/0.370; Qwen3-Coder-480B ReAct 0.025/0.793/0.425; Claude-4.6-Sonnet ReAct 0/0.874/0.580; +CADTests-log 0/0.897/0.625. Abstract prompts: Claude ReAct 0/0.929/0.715, +log 0.962/0.810.
+- Pitfall stated: CD/IoU "fail to account for the space of valid design variations" unconstrained by the prompt. License CC BY 4.0.
+
+### MUSE (May/Jun 2026) — manufacturable / functional / assemblable text-to-CAD
+- URLs: https://arxiv.org/abs/2605.28579 (HTML https://arxiv.org/html/2605.28579) ; project+leaderboard https://dong7313.github.io/muse-benchmark/ ; code https://github.com/dong7313/muse
+- 106 multi-component design instances (CNC, 3D-print, laser-cut; timber/PLA) with Design Specifications: expert seed models → Claude Opus 4.7 expansion → GPT-5.5 spec synthesis → human review. Output CadQuery. Three-stage: code check (executes) → OCCT geometric checks (watertight, manifold, no self-intersection, no overlap) → rubric VLM judge (Gemini 3.1 Pro; also GPT-5.5/GPT-4o tested) on functionality / manufacturability / assemblability. Human agreement r=0.713 (sub-criteria), ≈0.83 (instance).
+- Results (code / geometry / final %): GPT-5.5 77.36/70.75/52.36; Claude Opus 4.7 76.42/60.38/39.47; Gemini 3.1 Pro 65.09/58.49/43.40; Claude 3.7 Sonnet 46.23/23.58/12.74; GPT-4o 36.79/13.21/5.03; GLM-5.1 31.13/27.36/18.87; Qwen-2.5-72B 50.00/20.75/4.40; Qwen-3.5-122B 27.36/13.21/8.33; Llama-3.1-70B 31.13/23.58/2.20; MiniMax-M2.7 18.87/10.38/5.66.
+- Pitfall: "failure cascade" executable → valid geometry → engineering-ready. License CC BY 4.0 (data), MIT (code).
+
+### P3D-Bench (Jun 2026) — parametric 3D generation in JSON / OpenSCAD / CadQuery / Three.js
+- URLs: https://arxiv.org/abs/2606.11152 (HTML https://arxiv.org/html/2606.11152v1) ; project https://lucasqaq.github.io/p3d/ (code/HF: "future work")
+- 400 text cases + 400 image cases (Text2CAD v1.1 single parts) + 203 annotated Fusion 360 Gallery assemblies. Descriptive vs parametric specs written by an MLLM pipeline and auto-verified against source geometry. Every output executed and rendered; 4-step alignment (normalise, translate, rotate, bounded scale/position refinement minimising bidirectional CD).
+- Metrics: Valid (binary); CD, [email protected], [email protected], normal consistency, IoU (CSG for parts, voxel for assemblies); topology NoOE/InvN/NM; MLLM bucket QA-S/QA-P accuracy + J-Sem/J-Geo/J-Aes 1–10 by Gemini 3.1 Pro; PartFS / PartMatchF1 for assemblies.
+- Results: Text-3D parametric (JSON/OpenSCAD avg, Geo / IoU / Valid): GPT-5.5 0.706/0.812/0.999; Gemini 3.1 Pro 0.699/0.776/1.000; Claude Opus 4.6 0.692/0.788/0.995; DeepSeek V4 Pro 0.647/0.735/0.968; Kimi K2.6 0.679/0.716/0.988; Text2CAD Judge 0.055. Image-3D (CadQuery/OpenSCAD/Three.js avg, Geo / Judge / Valid): GPT-5.5 0.549/0.562/0.979; Gemini 0.552/0.540/0.970; Claude 0.525/0.431/0.975; cadrille 0.235/0.010/0.820. Best PartMatchF1 ≈0.5 on assemblies.
+- Pitfalls (explicit): **OpenSCAD is the most consistent format; CadQuery shows high syntax-failure rates; Three.js yields non-manifold triangle soups**; JSON QA-P>QA-S because prompts were derived from JSON (leakage). License arXiv non-exclusive.
+
+### IterCAD-Bench (Jun 2026)
+- URLs: https://arxiv.org/abs/2606.13368 (HTML https://arxiv.org/html/2606.13368v2)
+- Proprietary 11,000 drawing-to-code + 200 editing tasks, plus "Text2CAD Bench" (8,046 DeepCAD-test parts with text specs) and "CADPrompt Bench" (200). Output CadQuery (`import cadquery as cq`, result in `r`). New metric: CD-Tolerance-Recall curve and AUC-TR ∈[0,1] that counts non-executing samples (survivor-bias-free). Models: GPT-5, Gemini-2.5/3-flash-lite, GLM-4.6v, InternVL3.5-30B/8B, Qwen3.5-35B/4B, Claude-3.7, GPT-4o, DeepSeek-V3, Qwen2.5-72B/7B — numbers not extracted. License CC BY-NC-SA 4.0; code link not found.
+
+### CADSmith (Mar 2026)
+- URL: https://arxiv.org/abs/2603.26512 (HTML https://arxiv.org/html/2603.26512)
+- 100 prompts with hand-written CadQuery references in tiers T1 50 / T2 25 / T3 25 (1–3, 3–8, 5–15 ops). CadQuery 2.6.1. Metrics: CD = avg bidirectional squared NN over 10,000 surface points; F1 at 1.0 mm absolute threshold; voxel IoU at 1.0 mm. Claude Sonnet generator, Claude Opus validator, GPT-4 baseline; numbers not extracted. CC BY 4.0; code link not found.
+
+### CADGenBench (Hugging Face, 2026)
+- URLs: code https://github.com/huggingface/cadgenbench ; data https://huggingface.co/datasets/HuggingAI4Engineering/cadgenbench-data ; leaderboard https://huggingface.co/spaces/HuggingAI4Engineering/CADGenBench ; submissions https://huggingface.co/datasets/HuggingAI4Engineering/cadgenbench-submissions
+- 81 mechanical-part fixtures: 49 Generation (engineering drawing/description → solid) + 32 Editing (change to an existing STEP). Tool-agnostic: submit one STEP per fixture. Ground truth private (`cadgenbench-data-gt`, only the Space reads it). CAD Score = hard validity gate (well-formed watertight B-Rep) then weighted mean of shape similarity (surface-distance F1 + volume IoU), interface match (authored keep-in/keep-out volumes), topology match (Betti numbers b0,b1,b2); weights not stated. Reference agent: LiteLLM (Claude / GPT-5.5 / Gemini), build123d and CadQuery, iterative render-review loop. Leaderboard numbers: dynamic, not captured. License Apache-2.0.
+
+### Parametric CAD Bench (gNucleus, cadbench.ai, 2026)
+- URLs: https://cadbench.ai/ ; https://www.gnucleus.ai/cad-bench/news/parametric-cad-bench ; HF dataset "cad-gen-freecad"
+- Agents author **FreeCAD Part Design** models from NL; three tracks (part generation, assembly, complex workflow); six scoring axes (geometry accuracy, constraint/assembly correctness, parametric correctness incl. rebuild after dimension change, topology, agent workflow success, efficiency); weights not published. Leaderboard (combined score / cost): claude-opus-5 via claude-code 0.906 / $113.24; grok-4.6 mini-swe-agent 0.888 / $119.46; grok-4.6 grok-build 0.884 / $76.60; gpt-5.6-sol codex 0.865 / $97.58; grok-4.5 grok-build 0.837 / $36.59. License: not found.
+
+### CADEngBench (Aug 2026) — "It looks like CAD, but does it work?"
+- URL: https://arxiv.org/abs/2608.09296 (HTML https://arxiv.org/html/2608.09296)
+- CADEngBench-P: 300 parts (159 BenchCAD + 141 Fusion 360 Gallery) → 600 tasks (generate + edit) in layers L0 validity (execute, B-Rep solid, STEP export, STEP re-import — all required), L1 engineering/DFM requirements, L2-Z parametric perturbation, L2-E functional edits with preservation, L3 CalculiX FEA agreement. CADEngBench-A: 150 Fusion360 assembly-joint pairs (Entity@k, Typed@k, MRR, A2 E2E). Output CadQuery with `DEFAULT_PARAMS` / `PARAMETER_MANIFEST` / `build()`.
+- Models: GPT-5.2, Claude Sonnet 4.5, Gemini 3 Flash, GLM-4.6V, Kimi K2.5, Mistral Medium 3.5, Llama 4 Maverick, Qwen3.5-35B (per-model table not extracted).
+- Headline pitfalls: 58.1% of executable CAD violates engineering requirements; editing success 99.6% for isolated features vs 40.2–45.8% under join/cut coupling; FEA agreement uncorrelated with reach (ρ=0.024); generation vs assembly ranks ρ=0.048. License / repo: not captured.
+
+### RealCADBench (Sep 2026)
+- URL: https://arxiv.org/html/2609.03773 (code "to be publicly released")
+- 12,632 tasks in 19 factory-automation categories from an industrial marketplace + purchased CAD repository; reported slice 1,770 (1,745 parts across text 568 / 2D drawing 236 / real picture 568 / rendered image 373 + 25 assemblies). Output **FreeCAD 1.1.1 Python** exporting result.stl (mm). Multi-stage verification by five CAD practitioners; text tasks generated by Gemini 3.0 Pro from photos.
+- Metrics: executability (non-empty export); Solid IoU and Surface IoU after signed-PCA alignment + uniform scale, 96³ voxels with 2% padding; Judge = Kimi K2.6 rubric vs the input evidence (not the reference).
+- Results (Table 4, exec / solid IoU / surface IoU / judge): GPT-5.5 0.931/0.383/0.139/73.79; Claude Opus 4.8 0.892/0.408/0.143/68.31; Gemini 3.1 Pro Preview 0.883/0.429/0.158/73.33; GPT-5.4 0.801/0.380/0.134/61.19; Kimi K3 0.342/0.448/0.166/72.23; Qwen3.8-27B 0.796/0.359/0.113/56.30. Also Qwen3-VL-8B/32B, Doubao Seed 2.0 Pro evaluated (numbers not extracted). Per-regime exec: text 0.565, drawing 0.678, photo 0.799, render 0.812.
+- Pitfalls: "no single model leads all four metrics"; executability and IoU do not co-move (Kimi K3: 34% exec but highest IoU among survivors — survivor bias). License arXiv non-exclusive.
+
+---
+
+## 6. Not found / could not verify
+- "Text2CAD-Edit", "TextCAD-Verify", "MMCAD" (as an LLM benchmark — only MM-CAD CGF dataset found), "CAD-Bench 2026" (as a distinct name — the 2026 items are CADBench, BenchCAD, Text2CAD-Bench, CADEngBench, RealCADBench, P3D-Bench, MUSE, CADGenBench, Parametric CAD Bench): **not found**.
+- A dedicated multi-model **OpenSCAD** LLM benchmark: not found; P3D-Bench is the only one that scores OpenSCAD output across many models. A **build123d** benchmark: only CADGenBench's reference agent uses build123d; zalo's Fusion360→build123d conversion is a dataset, not a benchmark.
+- LLM4CAD per-class GPT-4/GPT-4V numbers, 3D-PreMise dataset size, CME-CAD numbers, Ortho2CAD/IterCAD/CADSmith per-model tables: not extracted (paywall/403 or not in fetched text).
+
+## 7. Cross-cutting observations (all traceable to entries above)
+1. **Chamfer conventions are incompatible across papers**: 1,000 pts + ICP (CADPrompt), 2,000 pts (DeepCAD/GenCAD), 8,192 pts (CAD-Recode/cadrille), 10,000 pts squared (Text-to-CadQuery, CADSmith), 30,000 pts unit-bbox (Text2CAD-Bench); alignment ranges from none to Procrustes (CADBench) to bounded-scale refinement (P3D-Bench) to PCA (RealCADBench, CAD-Coder). Median vs mean matters enormously (Text2CAD: median 0.37 vs mean 26.41).
+2. **IR/VSR definitions differ**: execution only (Text-to-CadQuery), execution+valid solid (CADBench), execution+watertight (CADEvolve), execution+STEP round-trip (CADEngBench), non-empty export (RealCADBench). Several benchmarks explicitly count failures as worst-case rather than excluding them (CADTestBench, IterCAD AUC-TR, MUSE cascade) because excluding them produces survivor bias (RealCADBench Kimi K3 example).
+3. **Best-of-N vs greedy**: CAD-Recode/cadrille pick min-CD of 10 candidates vs GT; CADmium uses greedy; Text2CAD-Bench uses API defaults with 3 error-feedback retries; most papers do not state decoding — the finetune/CLAUDE.md note that greedy collapses on boilerplate-heavy dialects is consistent with P3D-Bench's finding that CadQuery has the highest syntax-failure rate while OpenSCAD is most stable.
+4. **Frontier-model IR on text→CadQuery varies 11%–93% for "GPT-4o-class" depending on prompt format** (Text2CAD-Bench L1 11.1% for GPT-5.2; CADFusion 37.93%; CAD-Coder-CoT 93.00%; CAD-GPT 76.97%) — numbers are not comparable across papers.
+5. **Topology/validity-aware metrics** are the 2025–26 trend: Euler characteristic (CADmium, EvoCAD), SegE/DangEL/SIR/FluxEE (CAD-MLLM), Betti numbers (CADGenBench), OCCT watertight/manifold (MUSE), property-based CADTests, FEA (CADEngBench), rebuild-after-edit (Parametric CAD Bench, HistCAD, BenchCAD edit).
+6. **Contamination**: DeepCAD-derived test sets (8,046) are shared by Text2CAD, CADmium, UniCAD, IterCAD "Text2CAD Bench" and are in the training data of most fine-tuned models; only BenchCAD (blind baselines, OOD families), CADGenBench (private GT), RealCADBench (purchased industrial data), MUSE (expert-authored) and Text2CAD-Bench (hand-authored) attempt contamination control.
+7. **Licensing**: Fusion 360 Gallery is non-commercial with no redistribution; CAD-Recode data CC BY-NC; Text2CAD data CC BY-NC-SA; CAD-Editor/CADFusion CC BY-NC-SA; CADmium MIT; BenchCAD/MUSE/CADTestBench CC BY 4.0; CADGenBench Apache-2.0.
+
+<!-- ===== blender_scene.md ===== -->
+
+# Survey: benchmarks / datasets / metrics for LLM+VLM generation of Blender Python, procedural-3D code, scene-layout programs and animation code
+
+Compiled 2026-09-08 from primary sources (arXiv HTML/abs, GitHub, HF). Every number below was read from the cited page; "not found" means the source I fetched did not state it. Entries are grouped: (A) Blender-code object/editing benchmarks, (B) scene-layout programs, (C) animation / 4D code, (D) procedural materials, (E) CAD-code adjacent, (F) text-to-3D evaluators / judges (non-code but reused as metrics), (G) spatial-reasoning prerequisites, (H) items searched for but not found / out of scope.
+
+---
+
+## A. Blender-Python object generation and graphics-editing benchmarks
+
+### A1. BlenderGym — 2025, CVPR 2025 (Highlight)
+- **URLs**: paper https://arxiv.org/abs/2504.01786 (HTML https://arxiv.org/html/2504.01786); code https://github.com/richard-guyunqi/BlenderGym-Open; data https://huggingface.co/datasets/richard-guyunqi/BG_bench_data; project/leaderboard https://blendergym.github.io/ ; CVPR page https://openaccess.thecvf.com/content/CVPR2025/html/Gu_BlenderGym_Benchmarking_Foundational_Model_Systems_for_Graphics_Editing_CVPR_2025_paper.html
+- **Task**: start-scene `.blend` + start Python script + rendered start/goal image pair → edited Python script that reproduces the goal scene when executed (code-based 3D reconstruction/editing). Five task types: procedural geometry (50), lighting (40), procedural material (40), blend-shape (75), object placement (40) = 245 instances (https://arxiv.org/html/2504.01786).
+- **Dataset construction**: hand-crafted start/goal pairs derived from BlenderAlchemy prototypes, Infinigen and BlenderKit assets; each instance ships base `.blend`, start/goal scripts, renders and language description. No train/test split; human baseline run on ~20% of instances. HF card: 1,391 rows, 1.96 GB, single "train" split, license field empty (https://huggingface.co/datasets/richard-guyunqi/BG_bench_data).
+- **Execution env**: Blender Python API with Infinigen wrapper; ≥3 camera views per scene (≥1 "comprehensive" view); VLM sees 2 views, evaluation uses all views; 8-minute wall-clock limit per instance (matched to the human baseline). Blender version: not found in paper, README or HF card.
+- **Metrics**: Photometric Loss (PL) = pixel-space difference of renders; N-CLIP = (1 − CLIP score), "lower is better"; Chamfer Distance (CD) on 3D geometry (blend-shape task). PL and N-CLIP reported ×10⁻³ for blend-shape/placement and ×10⁻² for geometry/material/lighting; averaged per task (https://arxiv.org/html/2504.01786).
+- **Baselines (Table 1, best-of-1 pipeline)**: GPT-4o blend-shape PL 9.14 / N-CLIP 20.47 / CD 0.904; lighting PL 2.41 / N-CLIP 2.40; material PL 3.65 / N-CLIP 8.94. Claude 3.5 Sonnet blend-shape PL 12.79 / N-CLIP 27.96 / CD 1.962; lighting PL 2.90 / N-CLIP 4.05. Open 7–9B models (Qwen2-VL-7B, InternVL2-8B) "15–20% worse" and marked "–" on geometry/material when >75% of outputs are non-executable. Human baseline (10 Blender users): blend-shape PL 0.934 / N-CLIP 9.12 / CD 0.399; lighting PL 1.24 / N-CLIP 1.63; material PL 0.629 / N-CLIP 3.04. Leaderboard lists 14 systems incl. GPT-4-Turbo best material PL 8.812 (https://blendergym.github.io/).
+- **Inference scaling**: verifier re-selection k 1→64 improves selection; "scaled InternVL2-8B outperform unscaled or slightly scaled GPT-4o and Claude 3.5 Sonnet"; optimal generation/verification compute split shifts toward verification at higher budgets.
+- **Pitfalls reported**: generator hallucinates non-existent visual differences; non-executable scripts dominate for open models; irrelevant edits in 80+ line procedural scripts; verifier position bias ("Qwen consistently favor the second edit candidate in the pair, regardless of permutation"); human–VLM verifier agreement Claude 3.5 Sonnet 0.66 vs inter-human 0.79; verifiers miss physical implausibility.
+- **License/availability**: paper under arXiv non-exclusive license; data public on HF (no license field); site CC BY-SA 4.0 (Nerfies template); leaderboard at project page.
+
+### A2. BlenderLLM / BlendNet / CADBench — Dec 2024, arXiv
+- **URLs**: paper https://arxiv.org/abs/2412.14203 (HTML https://arxiv.org/html/2412.14203); code https://github.com/FreedomIntelligence/BlenderLLM; data HF `FreedomIntelligence/BlendNet`, `FreedomIntelligence/CADBench`, model `FreedomIntelligence/BlenderLLM` (https://huggingface.co/FreedomIntelligence/BlenderLLM).
+- **Task**: text instruction → bpy script → Blender render of a single CAD-style object.
+- **Dataset**: BlendNet — 135 seed instructions × 16 categories, Self-Instruct-expanded to ~50k instructions; 8 instruction types × 5 length levels; scripts by GPT-4o (gpt-4o-2024-08-06); 4 renders/object verified by GPT-4o (89.7% agreement with humans on 10k samples); 2k human-verified (12 annotators, 3 arbitrators, 30% QC) + 6k GPT-verified in paper; GitHub README says 12k pairs (2k manual + 10k GPT-4o). CADBench = 500 simulated (uniform over categories/types) + 200 "Wild" scraped from BlenderArtists/Reddit/Discord (https://arxiv.org/html/2412.14203; https://github.com/FreedomIntelligence/BlenderLLM).
+- **Execution env**: Blender renders 4 views per script; script is "executable" if Blender renders without error; Blender version not found.
+- **Metrics**: three dimensions/eight sub-dimensions — Attributes (shape, color, size, proportion, texture), Spatial (space, contact/distance), Instruction execution; per-sample 12–15 binary checkpoints judged by GPT-4o E(l,I,s,cᵢ)∈{0,1}, image-based for visual props and script-based for numeric props; sub-dimension → dimension → overall averages; E_syntax = N_error/N_total. Judge reliability: human–human κ 0.883, GPT-4o–human κ 0.791 on 200 outputs.
+- **Baselines (CADBench-Sim Avg / E_syntax)**: BlenderLLM (Qwen2.5-Coder-7B-Instruct + 3 self-improvement rounds) 0.748±0.085 / 3.4%; o1-Preview 0.687 / 15.6%; Claude-3.5-Sonnet 0.593 / 15.6%; GPT-4o 0.565 / 21.4%; DeepSeek-V2.5 0.479 / 25.2%; Qwen2.5-Coder-7B 0.353 / 31.4%. CADBench-Wild: BlenderLLM 0.664 / 3.5%; o1-Preview 0.583 / 17.5%; GPT-4o 0.444 / 28.5%; Gemini-1.5-Pro 0.380 / 38.0%.
+- **Pitfalls**: basic CAD only (no advanced materials/internals), single-turn only, Blender only; syntax errors 15–38% for frontier zero-shot.
+- **License**: repo Apache-2.0; paper CC BY 4.0. Test set public; no live leaderboard.
+
+### A3. BlenderAlchemy — 2024, ECCV 2024
+- **URLs**: paper https://arxiv.org/abs/2404.17672 (HTML https://arxiv.org/html/2404.17672); code https://github.com/ianhuang0630/BlenderAlchemyOfficial; project https://ianhuang0630.github.io/BlenderAlchemyWeb/
+- **Task**: text and/or reference image → edits of an existing Blender Python program (procedural material, blend-shape/geometry-node/placement, lighting). Edit-only; cannot build scenes from scratch.
+- **Eval set**: text-material: multiple descriptions on one Infinigen wooden starter material; image-material: 5 Infinigen materials toward shared targets (some DALL-E 3 targets); 32 text prompts in user study. Starter materials from Infinigen (some BlenderKit).
+- **Execution env**: 512×512 renders of a sphere, fixed camera/lighting; d=4 iterations × b=8 hypotheses; VLM GPT-4V. Blender version not found.
+- **Metrics**: CLIP similarity (avg ViT-B/32 and ViT-L/14 vs text); photometric loss; LPIPS; pairwise VLM state evaluator V(w₁,w₂,I); MTurk 592 pairwise comparisons.
+- **Numbers**: text-material CLIP ViT-B/32 28.2 vs BlenderGPT 25.2; ViT-L/14 24.0 vs 21.1; preferred over Paint3D 73%, over TEXTure 56%; human agreement with VLM evaluator 71%; removing vision drops ViT-B/32 to 25.7.
+- **Pitfalls**: "out-of-the-box VLMs have a poor understanding of the visual consequences of Blender program edits"; correct edits are sparse in program space.
+- **License**: paper CC BY 4.0; code on GitHub.
+
+### A4. 3D-PreMise — Jan 2024, arXiv
+- **URLs**: https://arxiv.org/abs/2401.06437 (HTML https://arxiv.org/html/2401.06437v1); code/data: not found (paper gives only an issue-report link).
+- **Task**: fully-specified text description of an industrial-style object → bpy program → mesh; evaluates self-correction with a visual interface.
+- **Dataset**: 57 samples (furniture, toys, decorative items) built for the paper; prompts "almost fully defined, leaving little space of ambiguity".
+- **Metrics**: Pass/fail via normalized point-cloud bidirectional matching within threshold δ; Chamfer distance d_CD = (1/|P_S'|)Σ min‖x−y‖² + (1/|P_T'|)Σ min‖x−y‖².
+- **Numbers**: GPT-4 only: best Pass@1 17.5% (one-shot CoT, greedy); Pass@3 19.3%; Pass@5 19.3%.
+- **Pitfalls**: spatial-precision errors most frequent; GPT-4 fixes programming mistakes from visual feedback but misses common-sense visual errors, hallucinates on visual feedback, struggles with geometric math; "redundant scaling" bug pattern.
+- **License**: not found.
+
+### A5. SceneCraft — 2024, ICML 2024 (oral)
+- **URLs**: https://arxiv.org/abs/2403.01248 (HTML https://arxiv.org/html/2403.01248); PMLR https://proceedings.mlr.press/v235/hu24g.html; code: not found (no repo linked on arXiv/HF paper page).
+- **Task**: text → relational scene graph → Blender Python with numerical layout constraints → render → GPT-4V critique loop; up to ~100 assets retrieved from TurboSquid via CLIP top-10.
+- **Eval**: 40 synthetic queries with ground-truth constraints (20 library-learning / 20 test) + Sintel movie scenes (first half train, second half test).
+- **Metrics**: CLIP similarity; Constraint Score (annotator-written scoring functions per constraint); Layout-matrix cosine similarity (Sintel); FVD; human win-rate on text fidelity / composition / aesthetics.
+- **Numbers**: SceneCraft vs BlenderGPT — CLIP 69.8 vs 24.7; constraint 88.9 vs 5.6; human win text-fidelity 76.8% vs 12.7%, composition 83.6% vs 11.4%, aesthetics 74.5% vs 14.5%; Sintel layout similarity 69.3 vs 27.5, scene CLIP 82.7 vs 41.8, FVD 317 vs 574.
+- **Pitfalls**: initial scripts "do not produce the correct layout outright" (wrong constraints or faulty constraint functions); library learning needs ≥20 GT-constraint examples; asset pool = TurboSquid.
+- **License**: paper CC BY 4.0; Blender version not found; code not released per pages fetched.
+
+### A6. 3D-GPT — Oct 2023, arXiv (also IEEE Xplore 2025)
+- **URLs**: https://arxiv.org/abs/2310.12945 (HTML https://arxiv.org/html/2310.12945); code https://github.com/Chuny1/3DGPT; project https://chuny1.github.io/3DGPT/3dgpt.html
+- **Task**: text → Infinigen function parameters / Python calls (task-dispatch, conceptualization, modeling agents) → Blender scene.
+- **Eval**: 100 ChatGPT-generated nature-scene descriptions; flower-type prompts for fine control; sequential-edit prompts.
+- **Metrics**: CLIP score (cosine in CLIP space); failure rate (datatype/parsing/missing-parameter); Shannon diversity of parameters.
+- **Numbers**: full system CLIP 29.16; w/o task-dispatch agent 22.79; w/o conceptualization agent 21.51 with failure rate 3.6%.
+- **Pitfalls**: limited curve/shading control; quality bounded by procedural functions; text-only.
+- **License**: arXiv non-exclusive; Blender version not found.
+
+### A7. Infinigen / Infinigen Indoors — CVPR 2023 / CVPR 2024 (data source, not an LLM benchmark)
+- **URLs**: https://arxiv.org/abs/2306.09310; https://github.com/princeton-vl/infinigen; https://infinigen.org/
+- **Role**: fully procedural Blender generator (plants, animals, terrain, indoor objects) whose factory scripts are the ground truth for 3DCodeBench, MeshCoder (Infinigen Indoors), Code2Worlds and 3D-GPT. License BSD-3-Clause (https://arxiv.org/abs/2306.09310 search summary; repo). No LLM eval numbers itself.
+
+### A8. 3DCodeBench / 3DCodeData / 3DCodeArena — May/Jun 2026, arXiv
+- **URLs**: https://arxiv.org/abs/2606.01057 (HTML https://arxiv.org/html/2606.01057v1); code https://github.com/gaoypeng/3dcodebench; data https://huggingface.co/datasets/YipengGao/3DCode; site 3dcodebench.com (per paper).
+- **Task**: text (three caption styles: object description / procedural instruction / factory-level spec) or 1–4 reference renders → standalone Blender 5.0 Python that procedurally builds one object; settings: single-shot, multi-turn error feedback, coding-agent harness (Claude Code / Codex CLI / Gemini CLI, 600–900 s budget).
+- **Dataset**: 212 Infinigen categories (one eval instance each, seed-addressable); ground-truth factory scripts median 387 / mean 531 lines; 3DCodeData corpus 212×60 seeds = 12,720 instances (paper says 12,963 for fine-tuning; README says 12,720) with 2 captions, 4 renders, textured + white GLB; captions validated by Gemini 3.1 Pro and humans. Released logs: 81,605 generated scripts / 82,042 trials + 2,767 agent transcripts.
+- **Execution env**: Blender 5.0 Cycles, sandboxed headless subprocess, 240 s timeout, 4 canonical views (azimuth 45/135/225/315°). Multi-turn: up to two stateless retries, each a fresh call with truncated stderr (head-70%/tail-30%, 33K-char cap per HTML; the paper's summary elsewhere says 3K).
+- **Metrics**: Executability 𝕀[ℰ(f)≠∅ ∧ |Mesh(ℰ(f))|≥1] (empty scene = failure); view-paired SigLIP-2 and DINOv3 cosine; Chamfer distance and Uni3D 3D–3D / cross-modal cosine on GLBs; "conditional" (over successes) vs "penalized" (failures = 0) aggregation; 3DCodeArena pairwise human votes (a / b / tie / both-bad; ties count 0.5/0.5) → Bradley–Terry MLE → Elo (400/ln10 per logit), 1000-resample bootstrap CI; ~3,100 votes over 12 models. SigLIP-2 Pearson r=0.964 with Elo, DINOv3 Spearman ρ=0.972. VLM-judge pairwise prompt in Appendix F.2; rubric dimensions not enumerated in the fetched excerpt.
+- **Baselines (exec / SigLIP-2 / Uni3D 3D–3D / Elo)**: GPT-5.5 0.906 / 0.834 / 0.562 / 1163; Claude Opus 4.7 0.910 / 0.814 / 0.490 / 1006; Gemini 3.1 Pro 0.725 / 0.824 / 0.567 / 1147; Claude Sonnet 4.6 0.804 / 0.813 / 0.525 / 1015; GPT-5.4 0.866 / 0.817 / 0.552 / 1074; Gemini 3.5 Flash 0.464 / 0.824 / 0.519 / 1119; Gemma 4 31B 0.582 / 0.801 / 0.494 / 952; Claude Haiku 4.5 0.502 / 0.761 / 0.363 / 799. Excluded (<10% exec): Gemini 2.5 Pro 7.1% text-to-3D, GPT-5.4 Nano 6.1%. Multi-turn: aggregate exec 0.702→0.974; agent harness exec 0.747→0.973 with ΔSigLIP-2 ≈ −0.010 (shape quality unchanged).
+- **Pitfalls**: physical plausibility (disconnected parts, floating primitives) is the bottleneck beyond executability; Blender 4.x→5.0 API deprecation is the dominant failure; thinking budget helps small models 15–19 exec points, frontier ≤5; multi-view input barely helps except the largest models.
+- **License**: paper CC BY 4.0; factory scripts BSD-3-Clause; eval code MIT; test set public; live arena leaderboard.
+
+### A9. VIGA + BlenderBench — Jan 2026, arXiv
+- **URLs**: https://arxiv.org/abs/2601.11109 (HTML https://arxiv.org/html/2601.11109); code https://github.com/Fugtemypt123/VIGA (MIT); data https://huggingface.co/datasets/DietCoke4671/blenderbench; project fugtemypt123.github.io/VIGA-website; overview https://www.emergentmind.com/topics/blenderbench
+- **Task**: target image(s) → iterative code-render-inspect edits of a Blender scene. Three tracks: camera adjustment, fixed-view multi-step editing, exploratory/compositional editing; 10 instances each (HTML says 27 tasks total; emergentmind says 30 episodes). Scenes 3–12 objects (primitives + imported assets), deliberate style/arrow-prompt domain gaps.
+- **Metrics**: PL (pixel-wise L2 to target); N-CLIP (negative CLIP cosine); VLM Score 0–5 by GPT-4o over task completion, spatial accuracy, detail accuracy, visual quality; success rate (executable trajectories); relative improvement. max_iterate_round=10; best-of-N selected by CLIP.
+- **Numbers**: GPT-4o best-of-4 PL — one-shot 48.16 / 7.36 / 30.14 (T1/T2/T3), BlenderAlchemy 14.50 / 1.95 / 20.62, VIGA 5.47 / 2.94 / 12.62; avg improvement +124.7% on BlenderBench, +35.32% BlenderGym (abstract; +26.88% GPT-4o best-of-1 in text), +117.17% SlideBench. VLM score best-of-1: GPT-4o one-shot 0.58/2.75/0.25 → VIGA 1.44/3.58/1.53; Qwen3-VL-8B one-shot 0.28/1.61/1.25 → VIGA 1.31/3.33/2.25.
+- **Pitfalls**: bounded by VLM spatial perception; GPT-4o "struggles with active spatial exploration" vs GPT-5 / Claude Sonnet 4; Blender version not found.
+- **License**: code MIT; paper arXiv non-exclusive.
+
+### A10. SceneActBench — Jul 2026, arXiv
+- **URLs**: https://arxiv.org/abs/2607.22393 (HTML https://arxiv.org/html/2607.22393); code https://github.com/Feinaldo2/SceneActBench; data https://huggingface.co/datasets/FEInaldo/SceneActBench; project https://feinaldo2.github.io/sceneactbench-project-page/
+- **Task**: images (1 or ~11 calibrated views, or 32/144 video frames) + anonymized GLBs → agent drives headless Blender via MCP (`get_scene_info`, `get_object_info`, `execute_blender_code`, `render_scene_view`, + `read_reference_frames`) to output object poses / camera pose JSON / articulated states / furnished-scene GLB / animated GLB. Five tasks: Layout, Camera, Articulated, Reconstruction, Dynamic; step budgets 30/30/60/35/80.
+- **Dataset**: 210 sources → 520 cases: 100 3D-FRONT rooms (3–7 objects, 27 categories), 100 S2O articulated containers (32-frame sequences), 10 Kenney CC0 dynamic scenes (144 frames @24fps); assets re-centered, re-scaled, names anonymized to remove answer leakage.
+- **Metrics** (deterministic, no VLM judge): ADD-S (Hungarian-matched nearest-surface distance, m); camera PE (m) and AE (deg, ignores roll); MPE = max over movable parts of geometry error or unreproduced range; F@5% after ICP + Hungarian; MME (max normalized mover trajectory error, unmatched=1) and LE (static centroid NN distance / 2S); overall = mean of five normalized task scores q↓=100·max(0,1−m/u), u=4 m (layout, camera), 90° (AE), 1 otherwise; seed 0 point sampling.
+- **Numbers (Overall)**: Doubao Seed 2.0 Pro High 50.2; Claude Opus 4.6 High 48.9; GPT-5.4 Medium 48.7; GPT-5.4 High 48.7 (Layout 84.1, Articulated 73.8, Dynamic 46.7); Qwen 3.7 Plus 46.2; Gemini 3.1 Pro 45.4; Claude Sonnet 5 39.5; MiniMax M3 38.6. Reconstruction F@5% 0.088–0.104 for top-3.
+- **Pitfalls**: rankings driven by balance not wins; multi-view helped 9/11 configs, hurt 2; similar scores hide different failure stages (Doubao moves 13/391 parts vs Claude 255/391 at similar MPE); tool-call volume anti-correlated with score (ρ=−0.68); single run per case; only 10 dynamic scenes.
+- **License**: arXiv non-exclusive; data on HF; Blender version not found.
+
+### A11. DreamHouse (physical generative reasoning) — Mar 2026, arXiv (Salesforce)
+- **URLs**: https://arxiv.org/abs/2603.24866v1 (HTML https://arxiv.org/html/2603.24866v1); project https://luluyuyuyang.github.io/dreamhouse
+- **Task**: 5 orthographic 512×512 renders (front/back/left/right/iso; "Frame" or occluded "Facade" condition) → Blender Python that builds a timber-frame house scene graph (foundation, floor, walls, roof members).
+- **Dataset**: 26,543 procedurally generated, human-validated structures, 13 styles, 133–1,548 members each (mean 673); JSON configs → procedural bpy generator; all pass the 10-test suite before release. Eval: 1,200 structures per protocol.
+- **Execution env**: headless Blender + Cycles (version not found); validation runs on the scene graph, no physics sim.
+- **Metrics**: Structural pass = all 10 deterministic tests (load path, IRC span limits, 16"/24" spacing, standard lumber dims, L/360 deflection, roof coverage ≥0.70, gap ≤0.20, cantilever limits, stability index =1, dual-end connection), no partial credit; Visual fidelity S = mean over views of max(0, 1−10·MSE) on alpha-masked renders, threshold 0.6; Topological fidelity T = 0.3·census + 0.4·Hungarian match (δ=0.3 m) + 0.3·voxel IoU; Joint pass = structural ∧ S≥0.6 (primary).
+- **Numbers (Planner-Atomic, Frame/Facade)**: GPT-5 struct 0.792/0.637, visual 0.312/0.287, joint 0.035/0.019; Claude Opus 4.5 struct 0.716/0.779, visual 0.406/0.377, joint 0.071/0.064 (best joint 7.1%); Gemini 3 Flash struct 0.454/0.539, joint 0.031/0.036; Planner-Managed lifts Gemini struct to 0.785. Open models (Qwen3.5-397B-A17B, Qwen3-VL-8B/30B, Kimi K2.5) show "fundamental" failures (wrong rafter orientation, scale). Visual-feedback refinement: Claude +0.033 S, 70.3% improved, 68.5% structural retention.
+- **Pitfalls**: structural validity and visual fidelity orthogonal; protocol choice swings results more than model choice (33 pp); topological score does not separate pass/fail; "a generated house mesh may score well on Chamfer distance while failing every structural test".
+- **License**: CC BY 4.0.
+
+### A12. 3D-CoS (code-synthesis 3D reconstruction) — Jun 2026, arXiv / OpenReview
+- **URLs**: https://arxiv.org/abs/2606.10478 (HTML https://arxiv.org/html/2606.10478); https://openreview.net/forum?id=0TV81QK0l8; code: not found.
+- **Task**: single RGB image → Blender 4.4 Python reconstructing the object; five workflows: single-call (~12k tokens), planning/blueprint (~26k), RAG over 21,102 Blender API functions (~34k), few-shot (~21k), part-wise agent with execution verification (~170k). Editing track on BlendNet-E (55 objects with hand-written edit instructions).
+- **Dataset**: ModelNet10, 100 objects / 10 categories, easy/hard split.
+- **Metrics**: CD (bidirectional NN surface distance), F@5%, Spatial-Balanced Recall@2%, Normal Consistency@5%; input-view silhouette IoU, depth NRMSE, normal MAE; upright-preserving similarity alignment; editing: CLIPsim, CLIPdir, 1–5 user study.
+- **Numbers**: Gemini 3 Pro CD 0.0249 / F@5% 0.8532; o3 0.0318 / 0.7949; Claude Sonnet 4 0.0368 / 0.7394; Qwen2.5-VL-72B 0.0492 / 0.6702; InternVL3.5-38B 0.0642 / 0.5584; LLaVA-OV-72B 0.0674 / 0.5186; InstantMesh 0.0191 / 0.9060; MeshCoder 0.0442 / 0.7053; editing CLIPsim 0.0578 vs point-cloud baseline 0.0142.
+- **Pitfalls**: agent workflow raises part coverage but introduces assembly inconsistencies; still short of feed-forward reconstructors.
+- **License**: CC BY 4.0.
+
+### A13. Thinking in Blender (SEIG, staged executable inverse graphics) — Jun 2026, arXiv (Cornell)
+- **URLs**: https://arxiv.org/abs/2606.02580 (HTML https://arxiv.org/html/2606.02580); code: not found.
+- **Task**: single image → four-stage generator-verifier loop (scene init, geometry, material, composition/lighting) emitting Blender Python; model Claude Opus 4.7.
+- **Data**: NeRF-synthetic (7/8 scenes), VoxHammer (13 object scenes), in-the-wild set.
+- **Metrics**: PSNR, SSIM, LPIPS, DreamSim, DINO, CLIP. NeRF: PSNR 13.58, SSIM 0.6881, LPIPS 0.3493; best on 5/6 metrics.
+- **Pitfalls**: early-stage errors propagate; high compute. License arXiv non-exclusive.
+
+### A14. EZBlender (Plan-and-ReAct agent) — Jan 2026, arXiv
+- **URLs**: https://arxiv.org/abs/2601.07143 (HTML https://arxiv.org/html/2601.07143); code https://github.com/Aztech-Lab/EZ_Blender
+- **Task**: text/visual prompt → Blender edits; evaluated on BlenderGym's 245 scenes plus a 5-scenario / 50-episode multi-task benchmark (shape keys, materials, lighting, background, camera) and 85 text-prompt scenes.
+- **Metrics/numbers**: CLIP 30.21 (shape keys, text); visual-prompt similarity 0.9816 (blend shape); multi-task completion 78.67% (Scene 1); latency 37.35 s; $0.0241/edit (GPT-4o); claims ~7× faster than BlenderAlchemy, 67% fewer tokens.
+- **Pitfalls**: camera posing and lighting remain hard due to VLM 3D perception. License arXiv non-exclusive.
+
+### A15. LL3M (Large Language 3D Modelers) — Aug 2025, arXiv (UChicago)
+- **URLs**: https://arxiv.org/abs/2508.08228 (HTML https://arxiv.org/html/2508.08228v1); project https://threedle.github.io/ll3m
+- **Task**: text → Blender 4.4 Python via planner (GPT-4o), BlenderRAG retrieval over 1,729 doc pages, coder (Claude 3.5 Sonnet), critic/verifier (Gemini 2.0 Flash), user-proxy edits.
+- **Eval**: 17 target objects, qualitative gallery vs BlenderMCP; complex-op count 5.86× with RAG vs 1.20; execution errors 2.43 vs 3.29; ~4 min create + ~6 min refine; 59% of user edits done in one instruction. No formal benchmark/user study.
+- **License**: CC BY 4.0; code availability not stated.
+
+### A16. ShapeCraft — Oct 2025, NeurIPS 2025
+- **URLs**: https://arxiv.org/abs/2510.17603 (HTML https://arxiv.org/html/2510.17603); OpenReview https://openreview.net/pdf?id=skS03tzYNw; project https://sanbingyouyong.github.io/shapecraft/
+- **Task**: text → Graph-based Procedural Shape (GPS) → wrapped bpy scripts; parser/coder Qwen3-235B-A22B, evaluator Qwen-VL-Max.
+- **Eval**: 26 long-form functional prompts from MARVEL-40M+ (Objaverse-derived).
+- **Metrics**: IoGT (point-cloud intersection over GT), Hausdorff, CLIP ViT-B/32 over 10 views, VQA pass rate (5 yes/no questions per prompt judged by Qwen-VL-Max on multi-view renders).
+- **Numbers (IoGT / Hausdorff / CLIP / VQA)**: ShapeCraft 0.471 / 0.415 / 27.27 / 0.44; BlenderLLM 0.455 / 0.511 / 26.99 / 0.43; 3D-PreMise 0.385 / 0.527 / 26.76 / 0.33; CADCodeVerify 0.334 / 0.511 / 25.94 / 0.34; LLaMA-Mesh 0.346 / 0.464 / 25.72 / 0.28; MVDream 0.427 / 0.411 / 26.84 / 0.42.
+- **Pitfalls**: ambiguous/brief/creative prompts break decomposition; complex topology/organic detail weak. License CC BY 4.0; code release not stated.
+
+### A17. MeshCoder — Aug 2025, arXiv (InternRobotics)
+- **URLs**: https://arxiv.org/abs/2508.14879 (HTML https://arxiv.org/html/2508.14879v2); code https://github.com/InternRobotics/MeshCoder; data https://huggingface.co/datasets/InternRobotics/MeshCoderDataset; project https://daibingquan.github.io/MeshCoder/
+- **Task**: point cloud → part-segmented Blender Python using a custom API (primitives, translation sweep, bridge-loop, boolean, array, fill-grid…); Llama-3.2-1B + LoRA with triplane shape tokenizer.
+- **Dataset**: 1M object–code pairs, 41 categories from Infinigen Indoors; ~10M part-level pairs; 70/15/15 split; 100K pairs released per HF search summary.
+- **Metrics**: L2 Chamfer on 100k points; voxel IoU at 32³. MeshCoder CD 0.063×10⁻² / IoU 86.75% vs PLAD 1.87 / 67.62%, Shape2Prog 6.00 / 45.03%.
+- **Pitfalls**: human-made objects only; no executability rate reported. License arXiv non-exclusive.
+
+### A18. Proc3D — Jan 2026, arXiv (USF / Adobe)
+- **URLs**: https://arxiv.org/abs/2601.12234 (HTML https://arxiv.org/html/2601.12234)
+- **Task**: text → Procedural Compact Graph (engine-agnostic; compiles to Blender/Substance/Unity) with editable parameters.
+- **Dataset**: 63k instruction–graph pairs from 21k PartNet models (chairs, tables, trash cans, storage, beds); LLaVA captions + LLaMA-3-70B instructions at 3 lengths; 1,000 hand-annotated.
+- **Metrics/numbers**: compile rate PCG 89% (GPT-4o ICL) vs 45% LLaMA-Mesh vs 30% raw Blender code; fine-tuned LLaMA-3 8B/70B compile 98%; ULIP 0.15 (70B-FT) vs 0.11 SDFusion / 0.08 LLaMA-Mesh; OOD ULIP drops to 0.08; tokens 702 vs 6,048 for geometry-nodes code.
+- **Pitfalls**: cube-primitive only; five categories. License CC BY 4.0; repo not confirmed.
+
+### A19. LLaMA-Mesh — Nov 2024, arXiv (NVIDIA) — mesh-as-text, not code
+- **URLs**: https://arxiv.org/abs/2411.09595 (HTML https://arxiv.org/html/2411.09595v1); https://research.nvidia.com/labs/toronto-ai/LLaMA-Mesh
+- **Task**: text ↔ OBJ text (v/f lines), 64-bin quantization, ≤500 faces; 31k Objaverse meshes ×4 rotations = 125k; Cap3D captions.
+- **Metrics**: qualitative vs MeshXL/Unique3D; language retention MMLU 61.74 vs 66.07, PIQA 79.16 vs 81.01, HellaSwag 77.35 vs 79.19, GSM8K 62.09 vs 77.18. No execution/geometry benchmark. License CC BY 4.0. Used as a baseline in ShapeCraft (A16) and Proc3D (A18).
+
+---
+
+## B. Scene-layout programs / LLM scene synthesis
+
+### B1. Holodeck — CVPR 2024 (AI2)
+- **URLs**: https://arxiv.org/abs/2312.09067 (HTML https://arxiv.org/html/2312.09067); code https://github.com/allenai/Holodeck (Apache-2.0); CVPR https://openaccess.thecvf.com/content/CVPR2024/html/Yang_Holodeck_Language_Guided_Generation_of_3D_Embodied_AI_Environments_CVPR_2024_paper.html
+- **Task**: text → GPT-4 floor plan, asset selection from 51,464 GPT-4V-annotated Objaverse assets, relational constraints (global/distance/position/alignment/rotation) → solver → AI2-THOR scene (Unity render; Blender optional).
+- **Eval**: 120 paired residential scenes vs ProcTHOR (preferred 59.8% asset selection, 56.9% layout, 64.4% overall); 260 diverse scenes/52 types rated 1–5 (Holodeck beats ProcTHOR on 28/52); layout ablation MRR 0.706 for constraint solver; CLIP score = 100×cosine (OpenCLIP ViT-L/14 LAION-2B) of top-down view vs "a top-down view of [scene type]"; ObjectNav on NoveltyTHOR 20.4% SR / 0.127 SPL vs ProcTHOR 4.11% / 0.015.
+- **Pitfalls**: "Absolute method performs no better than Random" when LLM outputs coordinates directly (collisions, OOB); struggles with restaurants, domain-specific assets; GPT-4 cultural bias; ~$0.2/room, ~3 min.
+- **License**: code Apache-2.0; requires GPT-4o-2024-05-13 and Unity 2020.3.25f1.
+
+### B2. LayoutGPT — NeurIPS 2023
+- **URLs**: https://arxiv.org/abs/2305.15393 (HTML https://arxiv.org/html/2305.15393); code https://github.com/weixi-feng/LayoutGPT (also UCSB-AI/LayoutGPT); OpenReview https://openreview.net/forum?id=Xu8aG5Q8M3
+- **Task**: text → CSS-style layout (2D boxes or 3D furniture with size/position/orientation) via in-context exemplars.
+- **Data**: 3D-FRONT bedrooms 3,397/453/423 and living rooms 690/98/53 (rectangular floors only); NSR-1K 2D numerical/spatial benchmark.
+- **Metrics**: OOB rate = % scenes with furniture outside floor plan; KL divergence of category distributions; FID over 4 camera renders. NSR-1K: precision/recall of predicted objects, layout accuracy, GLIP image accuracy.
+- **Numbers (GPT-3.5)**: bedrooms OOB 43.26% (ATISS 49.88%), KL 0.0995 (ATISS 0.0113), FID 28.37 (30.02); living rooms OOB 64.16% (83.02%), FID 76.34 (85.40). NSR-1K numerical P/R 94.81/96.49, layout acc 86.33%; spatial layout acc 82.54%.
+- **Pitfalls**: rare furniture underrepresented; overlaps/OOB persist. License CC BY 4.0.
+
+### B3. I-Design — ECCV 2024 workshops
+- **URLs**: https://arxiv.org/abs/2404.02838 (HTML https://arxiv.org/html/2404.02838); project https://atcelen.github.io/I-Design/
+- **Task**: text → multi-agent LLM dialogue → scene graph → backtracking placement (collision-free, in-bounds) → Objaverse retrieval.
+- **Eval**: 40 prompts (10 each functionality/layout/materials/atmosphere); 20 scenes vs LayoutGPT. GPT-4V grading 0–10 on four criteria; OOB rate; Bounding-Box Loss (overlap volume); object count.
+- **Numbers**: I-Design vs LayoutGPT: objects 18.2 vs 6.2; OOB 0.0% vs 57.6%; BBL 0.33 vs 7.58; GPT-4V avg 5.7 vs 4.8.
+- **Pitfalls**: placement termination failures; retrieved-asset dimension mismatch. License CC BY-SA 4.0.
+
+### B4. LayoutVLM — CVPR 2025 (Stanford/Google)
+- **URLs**: https://arxiv.org/abs/2412.02193 (HTML https://arxiv.org/html/2412.02193); project https://ai.stanford.edu/~sunfanyun/layoutvlm/; CVPR PDF https://openaccess.thecvf.com/content/CVPR2025/papers/Sun_LayoutVLM_Differentiable_Optimization_of_3D_Layout_via_Vision-Language_Models_CVPR_2025_paper.pdf
+- **Task**: instruction + unlabeled assets + floor plan (visually marked images) → initial poses + differentiable spatial relations (distance, on-top-of, align-with, point-towards, against-wall) → optimization.
+- **Eval**: 11 room types × 3 rooms, up to 80 assets; Objaverse assets; GPT-4-authored instructions.
+- **Metrics**: Collision-Free %, In-Boundary %, GPT-4o positional and rotational coherency, PSA = GPT-4o rating weighted by physical plausibility (0–100); user–GPT Kendall τ 0.46–0.61.
+- **Numbers (PSA avg)**: LayoutVLM 58.8; I-Design 40.0; LayoutGPT 16.6; Holodeck 5.6. VLMs: GPT-4o primary; LLaVA-NeXT-Interleave fine-tuned on ~9k 3D-FRONT rooms. No Claude/Gemini/Qwen numbers.
+- **Pitfalls**: "occasional failures … due to suboptimal VLM initializations". License CC BY-NC-ND 4.0.
+
+### B5. SceneEval / SceneEval-500 — Mar 2025 (v3 Mar 2026), arXiv (SFU)
+- **URLs**: https://arxiv.org/abs/2503.14756 (HTML https://arxiv.org/html/2503.14756v3); code https://github.com/3dlg-hcvc/SceneEval; site https://3dlg-hcvc.github.io/SceneEval/
+- **Task**: evaluation framework for text-conditioned indoor scene generators (works on any layout output with meshes).
+- **Data**: 500 descriptions (100 hand-written, 400 o4-mini-generated + manual validation), 10 room types, easy (≤4 furniture) / medium (5–8, ≤3 small) / hard (≥9), annotated with counts (eq/gt/lt/ge/le), attributes, 13 object–object and 10 object–architecture relations.
+- **Metrics**: fidelity — CNT, ATR (GPT-4o-2024-08-06 on renders), OOR/OAR (geometric ray-casting/point tests after VLM relation mapping); plausibility — COL (mesh intersection), SUP (VLM support-type + ray cast), NAV (largest free component / total free), ACC (functional sides unblocked), OOB (<99% of floor points inside → OOB). Qwen2.5-VL-7B tested as open judge (lower, same trend).
+- **Numbers (CNT / ATR / OOR / OAR / COL↓ / SUP)**: Holodeck 32.64 / 28.49 / 11.52 / 37.27 / 15.91 / 63.21; LayoutVLM 35.59 / 20.20 / 6.03 / 19.39 / 32.13 / 76.90; LayoutGPT 11.84 / 8.05 / 1.18 / 4.87 / 11.46 / 30.13 (OOB 72.25%); InstructScene 14.14 / 11.53 / 3.59 / 10.20 / 55.00 / 80.79; DiffuScene 11.99 / 9.28 / 3.20 / 8.21 / 31.81 / 75.40; ATISS 11.18 / 7.40 / 1.07 / 8.03 / 50.36 / 90.90.
+- **Pitfalls**: "even the best method satisfies fewer than 30% of attribute requirements"; retrieval fails on color/material/style; LLM-generated descriptions lose diversity with long history. License arXiv non-exclusive; code on GitHub.
+
+### B6. Open-Universe Indoor Scene Generation (Aguina-Kang et al.) — Mar 2024, arXiv
+- **URLs**: https://arxiv.org/abs/2403.09675 (HTML https://arxiv.org/html/2403.09675)
+- **Task**: text → Python-embedded declarative DSL (`adjacent()`, `facing()`, `aligned()`, `mounted_on_wall()`) → solver; retrieval from un-annotated Objaverse + ULIP renders.
+- **Eval**: 3 closed-universe room types; 59 open-universe prompts / 6 categories; 2AFC human studies (35 and 24 participants); retrieval P/R on Cap3D; orientation accuracy on 450 objects (~94%).
+- **Numbers**: preferred over ATISS 79%, DiffuScene 81%, LayoutGPT 65% (51–76% by prompt type); LLM errors per 1000 objects ~50 → ~31 with staged synthesis (hallucination, misuse, contradiction, unsatisfiability).
+- **License**: paper CC BY 4.0; code "upon publication".
+
+### B7. The Scene Language — Oct 2024, CVPR 2025 Highlight (Stanford)
+- **URLs**: https://arxiv.org/abs/2410.16770 (HTML https://arxiv.org/html/2410.16770v2); code https://github.com/zzyunzhi/scene-language
+- **Task**: text/image → Lisp-style Python DSL (program + words + embeddings) with Claude 3.5 Sonnet; renderers: Mitsuba, Minecraft, 3DGS-SDS, MIGC.
+- **Eval**: 99 numeric + 88 generic prompts; Prolific study 103 participants; CLIP (OpenCLIP ViT-H-14, 120 views); counting accuracy; 4D: dynamic degree via RAFT.
+- **Numbers**: alignment 85.65% vs GraphDreamer 3.56% / MVDream 10.79%; CLIP 0.351 vs 0.297 / 0.312; counting 1.0 vs 0.11; 4D CLIP 0.341 vs 4D-fy 0.352 but motion 5.9% vs 0.2%; image LPIPS 0.681 vs 0.811.
+- **License**: arXiv non-exclusive; code on GitHub.
+
+### B8. HDSL — Jun 2026, arXiv
+- **URLs**: https://arxiv.org/abs/2606.09738 (HTML https://arxiv.org/html/2606.09738)
+- **Task**: text → XML/CSS-style hierarchical DSL for indoor scenes, LLM agents generate + localized edits (HRAG).
+- **Eval**: 6 MIT-Scene categories × 5 runs; metrics NObj, OOB (fraction of objects intersecting room boundary), PIoU (mean pairwise 3D bbox IoU), CLIP, time.
+- **Numbers**: HDSL NObj 58.19 / OOB 0.05 / PIoU 0.07 / CLIP 20.99 / 383 s; Holodeck 43.15 / 0.06 / 2.45 / 19.81 / 552 s; I-Design 13.20 / 0.54 / 0.01 / 19.58 / 849 s; editing 5.22× fewer tokens; 20-user study 82.2% edits match.
+- **Pitfalls**: Objaverse coverage; static only. License arXiv non-exclusive.
+
+### B9. SceneGenAgent / SceneInstruct — Oct 2024, arXiv (THU + Siemens) — industrial scenes, C# not bpy
+- **URLs**: https://arxiv.org/abs/2410.21909 (HTML https://arxiv.org/html/2410.21909); code https://github.com/THUDM/SceneGenAgent
+- **Task**: text → C# for Siemens Process Simulate (Tecnomatix API). Benchmark 40 hand-written descriptions / 5 categories; SceneInstruct 3,002 synthesized descriptions, 21,756 instances (placement 5,704, verification 12,397, reassignment 2,587).
+- **Metric**: Pass@1 judged by 3 experts comparing render to description. GPT-4o 81.0%; GLM-4-Plus 73.5%; Llama3.1-70B 73.0% → 78.5% fine-tuned; CodeLlama-34B tuned 59.5%; "all generated code fails to compile without API guidance".
+- **License**: code Apache-2.0; dataset CC BY 4.0.
+
+---
+
+## C. Animation / 4D / cinematic code
+
+### C1. Code2Worlds / Code4D — Feb 2026, arXiv (PKU)
+- **URLs**: https://arxiv.org/abs/2602.11757 (HTML https://arxiv.org/html/2602.11757); code https://github.com/AIGeeksGroup/Code2Worlds; site https://aigeeksgroup.github.io/Code2Worlds
+- **Task**: text → Blender 4.3 bpy for 4D animated scenes (soft body, fluid, particles, rigid body, atmosphere) on Infinigen factories; Gemini 3 backbone; VLM-Motion Critic loop.
+- **Benchmark**: Code4D — 10 prompts (Appendix D); release "upon acceptance".
+- **Metrics**: O-CLIP / S-CLIP / Style-CLIP; VBench motion smoothness, subject/background consistency, temporal flicker; GPT-4o SGS (scene geometry), HRS (hybrid realism), Richness; manual failure rate (interpenetration, gravity, collision errors).
+- **Numbers**: Code2Worlds O-CLIP 0.2655 / SGS 61.4 / Style-CLIP 0.6734 vs ImmerseGen 0.2417 / 43.5 / 0.5991 vs Infinigen 0.2431 / 35.5 / 0.6671; scene S-CLIP 0.2432, Richness 62.3, HRS 55.4, failure 10%; video motion smoothness 0.9952 vs AnimateDiff failure 70%, Hunyuan 30%.
+- **License**: CC BY-NC-SA 4.0.
+
+### C2. SimWorlds / 4DBuildBench — Jul 2026, arXiv (CMU/Harvard/UC Merced)
+- **URLs**: https://arxiv.org/abs/2607.01766 (HTML https://arxiv.org/html/2607.01766); site https://dynsimworlds.github.io
+- **Task**: text → Blender 5.1 procedural dynamic scenes where physics is solved (cloth, fluid, rigid, particles, soft body) by planner-coder-reviewer agents (Claude Opus 4.7) with bl_rna-derived knowledge base.
+- **Benchmark**: 50 hand-authored prompts: 9 per dynamic category × 5 + 5 static; difficulty D1 single solver / D2 within-category / D3 cross-category.
+- **Metrics**: engine-state audit — MPR (mean fraction of per-actor predicates satisfied; 42 typed predicates incl. anti-cheat for faked keyframes), SPR (fraction of declared spatial relations holding under BVH distance test); itemized VLM judge (GPT-5.5, 5 frames, binary verdicts on objects/relations/actions/quality/aesthetics → mean fraction).
+- **Numbers**: SimWorlds MPR 0.87 / SPR 0.89 / VLM 0.82 vs VIGA 0.67 / 0.70 / 0.78. 70–120 min per scene (bake-dominated).
+- **Pitfalls**: judge is still an LLM/VLM; sampled frames can mask wrong intermediate dynamics; text-only. License CC BY 4.0.
+
+### C3. GPT4Motion — Nov 2023 (v2 Apr 2024), arXiv
+- **URLs**: https://arxiv.org/abs/2311.12631
+- **Task**: text → GPT-4-written Blender physics script (rigid drop/collision, cloth, liquid) → renders → Stable Diffusion video. Quantitative metrics/numbers: not found in abstract; code URL not found; license arXiv non-exclusive.
+
+### C4. Keyframer — Feb 2024, arXiv (Apple) — 2D CSS animation, not 3D
+- **URLs**: https://arxiv.org/abs/2402.06071 (HTML https://arxiv.org/html/2402.06071v1); https://github.com/apple/ml-keyframer; https://machinelearning.apple.com/research/keyframer
+- **Task**: SVG + text → CSS animation code (GPT-4). 13-participant study, two 15-min tasks; 223 designs; 90.4% of generated CSS syntactically clean, 6.7% errors; satisfaction 3.9/5. No benchmark; license CC BY 4.0.
+
+### C5. Animation2Code — Jun 2026, arXiv (UC Berkeley) — web animation, not Blender
+- **URLs**: https://arxiv.org/abs/2606.28593 (HTML https://arxiv.org/html/2606.28593); site anya-ji.github.io/animation2code-website
+- **Task**: video → HTML/CSS/JS reproducing it; 1,069 CodePen animations (80/20 split), 7–10,657 LOC, headless Chromium 1024×768 @30 fps.
+- **Metrics**: appearance = DreamSim + DTW; temporal = CoTracker3 tracklets, direction/speed agreement, Chamfer aggregation; both on animated-region crops; human validation 65 annotators / 600 pairs, ROC-AUC 0.89 appearance / 0.73 temporal.
+- **Numbers (exec / appearance / temporal)**: GPT-5.4 100% / 0.84 / 0.29; Gemini 3 Flash 100% / 0.80 / 0.30; Claude Sonnet 4.6 99.5% / 0.82 / 0.29; Qwen3-VL-8B 80.4% / 0.70 / 0.24; LLaMA 4 Scout 97.7% / 0.62 / 0.21. Temporal capped ~0.31 for all.
+- **License**: dataset CC BY 4.0, sources MIT.
+
+### C6. Cutscene Agent / CutsceneBench — Apr 2026, arXiv (Kuaishou) — Unreal Engine 5, not Blender
+- **URLs**: https://arxiv.org/abs/2604.25318 (HTML https://arxiv.org/html/2604.25318); project https://kuaishou-gamemind.github.io/cutscene_agent/
+- **Task**: script → UE5 Level Sequence via tool calls (animation, dialogue, camera). 65 scenarios, tiers S1–S5.
+- **Metrics**: L1 tool-use (selection accuracy, parameter validity, call completeness, efficiency, dependency compliance); L2 structure (track completeness, camera coverage, temporal consistency); L3 LLM-judge on rendered video, 4 dims × 0–25.
+- **Numbers (CC / CamC / L3)**: Claude Opus 4.6 100% / 96.4% / 50.2; Claude Sonnet 4.6 98.4 / 89.5 / 41.7; GPT-5.4 95.7 / 93.5 / 42.4; Qwen 3.5 Plus 94.5 / 89.3 / 30.0; Qwen2.5-72B 56.6 / 66.2 / excluded.
+- **License**: CC BY 4.0; code availability not stated.
+
+---
+
+## D. Procedural material generation
+
+### D1. VLMaterial — Jan 2025, arXiv (MIT) (ICLR 2025 per proceedings PDF)
+- **URLs**: https://arxiv.org/abs/2501.18623; ICLR PDF https://proceedings.iclr.cc/paper_files/paper/2025/file/5393eaf0b723278738788520ec54e4b4-Paper-Conference.pdf; code/data https://github.com/mit-gfx/VLMaterial
+- **Task**: single image → Blender 3.3 procedural material as Python code; LLaVA-NeXT (LLaMA-3 8B + CLIP ViT-L/14) with LoRA.
+- **Dataset**: 3,663 collected (BlenderKit 2,411, Infinigen 60, packs 1,192) → 1,640 valid → ~550K via GPT-4o-mini graph crossovers (50.4K) + 10× parameter perturbation; filters: ≤2,048 tokens, JPEG ≥12 KB, executes. Test: 44 Blender (ID), 64 Substance (OOD), 64 real photos.
+- **Metrics**: style loss (L1 VGG Gram + 0.1·L1 of 16×16 downsample); SWD; CLIP cosine; program correctness = valid fraction (N=50 tries, K=20 valid, pick lowest style loss).
+- **Numbers (Blender set, style / SWD / CLIP / correctness)**: VLMaterial 0.019 / 1.760 / 0.856 / 0.911; GPT-4o-mini 0.041 / 3.478 / 0.728 / 0.294; BlenderAlchemy 0.027 / 2.381 / 0.777; Cond. MatFormer 0.034 / 2.828 / 0.686. User study 16 participants: 91% visual / 94% graph preference over BlenderAlchemy.
+- **Pitfalls**: fails >~30 nodes or >2,000 tokens; worse on <4-node materials; budget-sensitive. License: paper arXiv non-exclusive; "first public Blender procedural material dataset".
+
+### D2. MaterialApprentice — Jul 2026, arXiv (UCSD)
+- **URLs**: https://arxiv.org/abs/2607.13318 (HTML https://arxiv.org/html/2607.13318v2); site https://materialapprentice.github.io/
+- **Task**: text/image → process traces retrieved from 158 YouTube tutorials (29 categories) + 10 own → compiled Blender 3.0–4.5 shader node graphs.
+- **Eval**: 110 BlenderKit materials + 35 photos; 25 coarse + 25 fine edits; 5 expert artists; 150 MTurk raters.
+- **Metrics**: CLIP cosine, SWD, style loss, GPT-5 forced choice, user preference %, execution rate, node detection mAP50 0.98.
+- **Numbers**: image-to-material CLIP 0.852 vs VLMaterial 0.856 vs BlenderMCP 0.781; preference 92% over BlenderMCP; editing CLIP 0.260 vs BlenderMCP 0.237, 100% preference.
+- **License**: arXiv non-exclusive; code/data "will be available".
+
+---
+
+## E. CAD-code adjacent (CadQuery/OpenSCAD, not bpy) — included because the metric designs transfer
+
+### E1. CadCodeVerify / CADPrompt — Oct 2024 arXiv, ICLR 2025
+- **URLs**: https://arxiv.org/abs/2410.05340; ICLR PDF https://proceedings.iclr.cc/paper_files/paper/2025/file/81a934cd364e18ea6fdeaf57a93c17d4-Paper-Conference.pdf
+- **Task**: text → CadQuery; CADPrompt = 200 expert-annotated prompts+scripts. Metrics Chamfer, IoU, Hausdorff, compile/execute rate. GPT-4 with CadCodeVerify: −7.30% point-cloud distance, +5.0% success. Per-model numbers for Gemini 1.5 Pro / CodeLlama: not found in abstract. Code URL: not found.
+
+### E2. Text2CAD-Bench — May 2026, arXiv
+- **URLs**: https://arxiv.org/abs/2605.18430 (HTML https://arxiv.org/html/2605.18430v1)
+- **Data**: 600 human+AI co-authored CadQuery examples, L1 200 / L2 200 / L3 100 / L4 100 (real-world domains), two prompt styles (geometric / sequence).
+- **Metrics**: CD ×10³ bidirectional; Invalidity Rate (syntax/runtime/60 s timeout/degenerate); IoU at 256³ voxels; L4: GLM-4.6V judge, 8 views, five 0–10 questions + overall.
+- **Numbers (L1-Geo CD / L3-Geo CD / L3 IR)**: GPT-5.2 44.31 / 93.46 / 68%; Claude-4.5-Sonnet 52.62 / 70.13 / 70%; DeepSeek-V3.2 53.15 / 101.23 / 69%; Qwen3-max 84.54 / 148.58 / 92%; Gemini-3-Flash 66.82 / 91.61. Domain models (Text2CAD, CADFusion) IR 2–11% but CD 209–277, IoU ≈0.
+- **Pitfalls**: executability, geometric precision and feature understanding are decoupled. License CC BY 4.0; leaderboard pending.
+
+### E3. BenchCAD — May 2026, arXiv
+- **URLs**: https://arxiv.org/abs/2605.10865 (HTML https://arxiv.org/html/2605.10865v1); HF `BenchCAD/BenchCAD`; site https://benchcad.github.io/BenchCAD_webpage/; leaderboard https://llm-stats.com/benchmarks/benchcad
+- **Data**: 17,900 expert-verified CadQuery programs, 106 industrial families (52 tied to ISO/DIN/ASME…), 49 operations; 30 s timeout, volume check, expert visual sign-off; BenchCAD-QA 2,400; BenchCAD-Edit 748.
+- **Tasks/metrics**: image→code (exec %, voxel IoU incl. rotation-invariant, CD, essential-operation recall), code edit (headroom-normalized accuracy), vision QA, code QA. Vision QA mean 0.587 vs code QA 0.838; edit accuracy GPT-4o 0.615 → GPT-5.3-thinking 0.865; 64% of "successful" edits silently corrupt other features.
+- **License**: data CC BY 4.0; code MIT.
+
+### E4. P3D-Bench — Jun 2026, arXiv (NJU)
+- **URLs**: https://arxiv.org/abs/2606.11152 (HTML https://arxiv.org/html/2606.11152v1); project https://lucasqaq.github.io/p3d/
+- **Data**: 400 text + 400 image + 203 annotated assemblies from Text2CAD v1.1 and Fusion 360 Gallery; outputs JSON / OpenSCAD / CadQuery / Three.js; easy/medium/hard.
+- **Metrics**: geometry (CD, [email protected], [email protected], normal consistency, IoU), topology (open edges, inverted normals, non-manifold), MLLM judge (semantic/geometric/aesthetic + QA banks), part (PartMatchF1, PartFS), Valid.
+- **Numbers (GPT-5.5, Geo / Topo / Judge / Valid)**: text-param 0.71 / 0.997 / 0.81 / 0.999; image 0.55 / 0.91 / 0.56 / 0.98; assembly 0.59 / 0.97 / 0.54 (Part 0.61) / 0.99. Others: Gemini 3.1 Pro, Claude Opus 4.6, Kimi K2.6, Qwen3.6-Plus, DeepSeek V4 Pro, GLM-5.1, MiMo v2.5 Pro (numbers not in fetched excerpt).
+- **Pitfalls**: assemblies hardest; semantic ≫ geometric precision (~0.8 vs ~0.35). License arXiv non-exclusive.
+
+---
+
+## F. Text-to-3D evaluators and judges (non-code; reused as metrics for code-generated assets)
+
+### F1. T3Bench — Oct 2023, arXiv (Tsinghua)
+- **URLs**: https://arxiv.org/abs/2310.02977 (HTML https://arxiv.org/html/2310.02977v1); code https://github.com/THU-LYJ-Lab/T3Bench; site https://t3bench.com
+- **Data**: 300 GPT-4-generated, manually filtered prompts (100 single object / 100 with surroundings / 100 multi-object), ROUGE-L dedup.
+- **Metrics**: Quality = multi-view (161 icosahedron vertices × 5 focal lengths) ImageReward + regional convolution / graph mean pooling to detect Janus inconsistency (Spearman 0.752 with humans); Alignment = BLIP captions from 12 views merged by GPT-4, recall-oriented scoring (Spearman 0.765).
+- **Numbers (avg over quality+alignment, single / surroundings / multi)**: ProlificDreamer 49.4 / 44.8 / 35.8; Magic3D 37.0 / 35.4 / 25.7; DreamFusion 24.4 / 24.6 / 16.1.
+- **License**: arXiv non-exclusive; code public.
+
+### F2. GPTEval3D — Jan 2024, arXiv
+- **URLs**: https://arxiv.org/abs/2401.04092; code https://github.com/3DTopia/GPTEval3D; site https://gpteval3d.github.io/
+- **Protocol**: GPT-4V generates prompts, does pairwise comparisons under user-defined criteria, Elo from comparisons; "strongly align with human preference". Kendall τ numbers, prompt/model counts: not found in abstract.
+
+### F3. 3DGen-Bench / 3DGen-Arena — Mar 2025, arXiv
+- **URLs**: https://arxiv.org/abs/2503.21745 (HTML https://arxiv.org/html/2503.21745v1); site https://zyh482.github.io/3DGen-Bench/; HF dataset (per paper).
+- **Data**: 1,020 prompts (510 text / 510 image), 22 generators (9 T23D, 13 I23D), 68,400 pairwise votes + 56,100 absolute scores over 5 dims (geometry plausibility, geometry details, texture quality, geometry–texture coherence, prompt alignment).
+- **Metrics**: 3DGen-Score (CLIP-based multi-view win-prob) alignment 0.725 T23D / 0.767 I23D vs CLIP 0.661 / 0.531; 3DGen-Eval (MLLM judge).
+- **License**: prompts/annotations MIT; assets per source model.
+
+---
+
+## G. Spatial-reasoning prerequisites for 3D-software agents (not code generation)
+
+### G1. MultiView-Bench — Jul/Aug 2026, arXiv (Yale)
+- **URLs**: https://arxiv.org/abs/2607.08970 (HTML https://arxiv.org/html/2607.08970); site hantaozhangrichard.github.io/MultiView-Bench
+- **Task**: Blender-rendered multi-view images with visible axes (GUI-like) → world-centric object coordinates (±X/0, ±Y/0, ±Z/0); 5 variants × 100 + 20 extended (2,500+ instances). 3-DoF accuracy: GPT-5.6 Sol 63.3%, Claude Opus 4.8 56.3%, Gemini 3.5 Flash 54.3%, GPT-5 49.0%, GPT-4o 2.0%. Pitfalls: axis confusion, rotation brittleness. License CC BY 4.0.
+
+### G2. Open3D-VQA — Mar 2025, ACM MM 2025
+- **URLs**: https://arxiv.org/abs/2503.11094; code https://github.com/EmbodiedCity/Open3D-VQA.code
+- **Task**: aerial-view spatial VQA (73k pairs, 7 tasks, real + simulated); 13 MLLMs; relative relations easier than absolute distances; 3D-LLMs no advantage. Not a code benchmark; license arXiv non-exclusive.
+
+---
+
+## H. Searched for but not found / out of scope
+
+- **3D-Fixup** (SIGGRAPH 2025, https://arxiv.org/abs/2505.10566) — diffusion-based 3D-aware photo editing; no code generation or Blender benchmark.
+- **GenEval-3D / 3D-GenEval** — no such benchmark found; only GenEval (T2I, https://arxiv.org/html/2310.11513) and GenEval 2 (https://arxiv.org/html/2512.16853v1).
+- **BlenderMCP** (https://github.com/ahujasid/blender-mcp) — tool, not a benchmark; no paper with metrics found. It appears as a baseline in LL3M (A15) and MaterialApprentice (D2), and SceneActBench (A10) uses an MCP interface.
+- **"Blender-Bench"** — only Blender's own GPU benchmark (https://opendata.blender.org/); the LLM "BlenderBench" is VIGA's (A9).
+- **MotionScript** (https://arxiv.org/abs/2312.12634) — natural-language motion captioning, not code. **ChatAnimation**, **AniCraft** (UIST 2024 MR prototyping) — not LLM code benchmarks.
+- **MatAgent** — materials-science discovery agents (https://arxiv.org/pdf/2504.00741), not graphics materials.
+- **SceneProg** — no benchmark by that name found; nearest are Scene Language (B7), Open-Universe DSL (B6), HDSL (B8). **SceneScript** (Meta) is reconstruction-to-structured-language, not LLM code generation.
+- **SCOPE** (https://github.com/HindsboNikolaj/SCOPE, HRI 2026) — PTZ-camera agent tasks in Blender (536 tasks, GPT-4o-judged accuracy, best 73.8% Qwen3-30B-A3B+Moondream3, MIT); no code generation.
+- **BlenderGPT** — used as a baseline in SceneCraft/BlenderAlchemy; no standalone benchmark found.
+- **Infinigen-based LLM benchmark** — the only ones found are 3DCodeBench (A8) and 3D-GPT (A6); Code2Worlds (C1) and MeshCoder (A17) also build on Infinigen.
+- **LLM Blender keyframe-animation benchmark** — none with keyframe-level metrics; closest are 4DBuildBench (C2, solver-based), Code4D (C1), SceneActBench Dynamic/Articulated tracks (A10), Animation2Code (C5, web).
+
+<!-- ===== shader_web.md ===== -->
+
+# Survey: benchmarks / datasets / metrics for LLM+VLM generation of shaders, three.js/WebGL scenes, and symbolic graphics programs
+
+Compiled 2026-09-08. Coverage window: 2023 – mid 2026. Every claim carries a URL; "not found" means I could not locate the fact in a primary source (I did not guess). Numbers are copied from the cited page; where a page was gated (HF 401) I say so.
+
+Organisation:
+- Part A — GLSL / Shadertoy (ShaderEval family, shaders21k, 2025–26 shader-LLM papers, public micro-evals)
+- Part B — three.js / WebGL / browser 3D (WorldCoder-Bench, P3D-Bench, ArtifactsBench, Web-Bench, WebGen-Bench, WebDev Arena, Mage/Unity)
+- Part C — Symbolic graphics programs: SGP-Bench, SGP-GenBench, SVG benchmarks (brief), VCode
+- Part D — Analogous "code → render → compare" benchmarks: Design2Code family, Sketch2Code, Interaction2Code, FullFront, WebIGBench, Image2Struct, Plot2Code, ChartMimic, SWE-bench Multimodal, TikZ (DaTikZ/TikZero, vTikZ), Manim (ManiBench, ManimBench, DTVBench), VisPlotBench, Vision2Code, InteractScience
+- Part E — Procedural material / node-graph and Blender-program benchmarks (VLMaterial, MultiMat, BlenderGym, SceneCraft)
+- Part F — Cross-cutting metric definitions and pitfalls (what transfers to a GLSL / three.js eval)
+- Part G — Things searched for and NOT found
+
+---------------------------------------------------------------------------------------------------
+
+## PART A — GLSL / Shadertoy
+
+### A1. ShaderMatch / ShaderEval-2 ("Evaluating Language Models for Computer Graphics Code Completion")
+- **Name; year/venue**: ShaderMatch benchmark + metric (a.k.a. ShaderEval task 2, "shadereval-2: FunctionCompletion"). Kels, Dahou, Mathiak. *2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code)*, co-located with ICSE 2025, pp. 96–103, DOI 10.1109/LLM4Code66737.2025.00017. Author "Vipitis" = Jan Kels (HHU Düsseldorf); co-authors at GESIS.
+ - https://conf.researchr.org/details/icse-2025/llm4code-2025-papers/13/Evaluating-Language-Models-for-Computer-Graphics-Code-Completion
+ - https://ieeexplore.ieee.org/document/11028297/
+- **URLs**: no arXiv version found. Leaderboard + metric space: https://huggingface.co/spaces/Vipitis/Shadermatch ; metric code: https://huggingface.co/spaces/Vipitis/Shadermatch/raw/main/shadermatch.py ; inputs: https://huggingface.co/datasets/Vipitis/Shadereval-inputs ; results: https://huggingface.co/datasets/Vipitis/Shadereval-results ; raw generations (gated): https://huggingface.co/datasets/Vipitis/Shadereval-runs ; dataset builder: https://github.com/Vipitis/shadertoys-dataset ; bigcode-evaluation-harness task PR: https://github.com/bigcode-project/bigcode-evaluation-harness/pull/173
+- **Task**: partial code → GLSL. Zero-shot *function completion*: the model receives `model_inp` = the user-written comment preceding a function + the function header, and must produce the function body; the completed function is spliced back into the full Shadertoy "image" pass and rendered. (Shadereval-inputs card; conf page abstract.)
+- **Dataset size & construction**: paper abstract says **467 function headers**; the current HF card (v0.3) says **394 functions** (v0.2 had 257) — both from https://huggingface.co/datasets/Vipitis/Shadereval-inputs. Single `test` split (~726 kB). Sources: the shaders21k/"shaders20k" Shadertoy subset (years 2013–2021) plus the Shadertoy public API (2022–2023). Pipeline (GitHub README): download via Shadertoy API (API key) → "annotate" (flatten nested render passes; extract license and functions with tree-sitter-glsl v0.1.9; license detection via scancode-toolkit) → "filter". Every retained function was validated with wgpu-shadertoy to confirm it is actually necessary for the shader (removing it changes the frames). Fields: comment, header, body, image_code, id, author, date, license (SPDX identifiers, e.g. MIT, CC0-1.0), func_bytes, functions, model_inp, function_frequency, header_frequency. Licensing: per-shader SPDX tags; builder scripts Apache-2.0; "shader code are under their respective license". Dedup / leakage: the Shadereval-runs dataset is *gated* "to prevent training use"; the santacoder model card explicitly warns of contamination for the earlier Shadertoys-fine training set (see A3). Difficulty tiers: not found.
+- **Execution environment**: wgpu-shadertoy (wgpu-py, i.e. WebGPU/wgpu-native, headless) — from shadertoys-dataset README and the metric code. Resolution **(512, 288)** ("matching thumbnail dimensions"). Rendered at **10 timesteps: [0.0, 0.1, 0.2, 0.5, 1.0, 1.6787, 2.0, 2.31, 3.333, 17]** chosen to avoid periodic patterns (shadermatch.py). Space README admits the timestamps "are not methodologically justified" and could miss shaders with particular periodicity. Timeouts: not found.
+- **Metric — exact definition (from shadermatch.py + README)**: two-step — static code comparison, then frame-render comparison. Checks run in this order and each sample gets exactly one of **eight labels**:
+ 1. `incomplete_generation` — generation contains the marker "// incomplete generation!" (function never closed / cut off).
+ 2. `c0-clone` — exact string match with the reference.
+ 3. `c1-clone` — "lexical similarity": only whitespace/comments differ.
+ 4. `c2-clone` — "syntactic similarity": only identifiers differ.
+ 5. `code_error` — shader fails to compile.
+ 6. `c4-clone` — "semantic similarity": all 10 timesamples render identical images. Image equality is **exact pixel equality** via `PIL.ImageChops.difference(...).getbbox() is None` — no tolerance.
+ 7. `single_color` — all rendered frames are a single uniform colour (treated as failure).
+ 8. `variation` — renders differ from the reference at some timestep ("potentially better or worse than reference").
+ Derived leaderboard aggregates (analysis.py): `error_rate = code_error + incomplete_generation`; `clone_rate = c0+c1+c2+c4`; leaderboard sorted by error_rate ascending. https://huggingface.co/spaces/Vipitis/Shadermatch/raw/main/analysis.py
+- **Generation settings**: Shadereval-results card: temperature 0.2, top_p 0.95, seed 0 (22 models). Number of samples per prompt: not found.
+- **Reported baseline numbers (Shadereval-results, fraction of test set per label)** — https://datasets-server.huggingface.co/rows?dataset=Vipitis/Shadereval-results&config=default&split=train :
+
+| model | incomplete | c0 | c1 | c2 | c4 | code_error | single_color | variation |
+|---|---|---|---|---|---|---|---|---|
+| deepseek-coder-6.7b-base | 0.1285 | 0.0428 | 0.0600 | 0.0043 | 0.0942 | 0.1799 | 0.0171 | 0.4732 |
+| deepseek-coder-7b-base-v1.5 | 0.1328 | 0.0321 | 0.0514 | 0.0043 | 0.0921 | 0.1842 | 0.0236 | 0.4797 |
+| deepseek-coder-5.7bmqa-base | 0.1285 | 0.0321 | 0.0535 | 0.0064 | 0.0814 | 0.1949 | 0.0236 | 0.4797 |
+| deepseek-coder-1.3b-base | 0.1413 | 0.0128 | 0.0471 | 0.0043 | 0.0514 | 0.1799 | 0.0321 | 0.5310 |
+| CodeQwen1.5-7B | 0.1285 | 0.0343 | 0.0428 | 0.0193 | 0.0835 | 0.1906 | 0.0278 | 0.4732 |
+| StarCoder2-15B | 0.096 | 0.021 | 0.090 | 0.011 | 0.094 | 0.231 | 0.024 | 0.433 |
+| StarCoder2-7B | 0.199 | 0.009 | 0.032 | 0.004 | 0.081 | 0.231 | 0.028 | 0.415 |
+| StarCoder2-3B | 0.208 | 0.006 | 0.039 | 0.009 | 0.045 | 0.225 | 0.026 | 0.443 |
+| CodeLlama-13b-hf | 0.1092 | 0.0343 | 0.0450 | 0.0064 | 0.0878 | 0.2056 | 0.0171 | 0.4946 |
+| CodeLlama-7b-hf | 0.1734 | 0.0300 | 0.0407 | 0.0064 | 0.0707 | 0.2441 | 0.0300 | 0.4047 |
+| Llama-3.1-8B | 0.167 | 0.002 | 0.047 | 0.004 | 0.056 | 0.208 | 0.036 | 0.480 |
+| Yi-Coder-9B | 0.150 | 0.024 | 0.049 | 0.013 | 0.088 | 0.225 | 0.017 | 0.435 |
+| Yi-Coder-1.5B | 0.291 | 0.011 | 0.021 | 0.002 | 0.036 | 0.214 | 0.026 | 0.398 |
+| granite-20b-code-base | 0.1413 | 0.0214 | 0.0664 | 0.0064 | 0.0878 | 0.2120 | 0.0150 | 0.4497 |
+| granite-8b-code-base | 0.1328 | 0.0128 | 0.0578 | 0.0086 | 0.0792 | 0.2463 | 0.0171 | 0.4454 |
+| granite-3b-code-base | 0.1991 | 0.0150 | 0.0385 | 0.0128 | 0.0642 | 0.2120 | 0.0107 | 0.4475 |
+| codegemma-7b | 0.1306 | 0.0107 | 0.0407 | 0.0021 | 0.0578 | 0.2570 | 0.0214 | 0.4797 |
+| codegemma-2b | 0.2355 | 0.0086 | 0.0300 | 0.0000 | 0.0471 | 0.2227 | 0.0343 | 0.4218 |
+| stable-code-3b | 0.126 | 0.017 | 0.021 | 0.006 | 0.066 | 0.257 | 0.026 | 0.480 |
+| phi-2 | 0.373 | 0.004 | 0.006 | 0.000 | 0.009 | 0.345 | 0.013 | 0.251 |
+| phi-1_5 | 0.373 | 0.002 | 0.002 | 0.000 | 0.019 | 0.355 | 0.024 | 0.225 |
+| phi-1 | 0.122 | 0.002 | 0.000 | 0.000 | 0.015 | 0.608 | 0.024 | 0.229 |
+
+ Derived (my arithmetic on the table above): deepseek-coder-6.7b-base error_rate = 0.308 (this is the "31 % fail to generate working code" from the abstract), clone_rate = 0.201; StarCoder2-15B error_rate 0.327, clone_rate 0.216; CodeQwen1.5-7B error_rate 0.319. No frontier/closed models appear in the results set (only open code models ≤20B). Model metadata (size, FIM support, #languages, whether GLSL was in pretraining) is in https://huggingface.co/spaces/Vipitis/Shadermatch/raw/main/models.csv (e.g. StarCoder2-15B "glsl-trained: yes"; CodeLlama "unknown"; phi "no").
+- **Pitfalls / observations**: (i) exact-pixel c4 match is strict — tiny float differences push a sample into `variation`, so "variation" (0.22–0.53) is the dominant bucket and is *not* a quality score; (ii) `single_color` is a specific degenerate mode (1–4 % of samples) that a naive image metric would score as "similar" for dark references; (iii) `incomplete_generation` is large for small models (up to 37 %) — a max-token / stop-sequence artefact; (iv) timestamps not principled; (v) GLSL called "a low-resource language rarely found in pretraining datasets"; (vi) raw generations gated to limit contamination.
+- **License / availability**: metric + space open (HF); inputs public with per-shader SPDX licenses; generations gated; results public.
+
+### A2. ShaderEval-1 (2023, "return completion")
+- **Name; year**: ShaderEval task 1 "shadereval-1: ReturnCompletion" (Vipitis, 2023; bigcode-evaluation-harness PR #173). https://github.com/bigcode-project/bigcode-evaluation-harness/pull/173 ; space https://huggingface.co/spaces/Vipitis/ShaderEval (README only holds space config).
+- **Task**: partial GLSL function body up to the `return` → predict the return statement. Subset "return_completion" of Shadertoys-fine; two features: body-before-return and the return statement; evaluator keeps everything before the semicolon, strips whitespace.
+- **Metric**: `exact_match` (HF evaluate).
+- **Numbers**: only found for the fine-tuned santacoder model card: 0.567 on 300 samples (greedy), 0.597 on all samples (greedy). The card states "The model results can't be trusted for this simple benchmark" because of data contamination and "an undefined license" for the training data. https://huggingface.co/Vipitis/santacoder-finetuned-Shadertoys-fine
+- **Pitfall**: superseded by ShaderMatch precisely because exact-match on a return line is contamination-prone and blind to semantics.
+
+### A3. Shadertoys / Shadertoys-fine datasets (Vipitis)
+- **URLs**: https://huggingface.co/datasets/Vipitis/Shadertoys ; https://huggingface.co/datasets/Vipitis/Shadertoys-fine ; builder https://github.com/Vipitis/shadertoys-dataset . The two HF cards returned HTTP 401 (gated) to my fetcher; facts below come from search snippets of the cards and the GitHub README.
+- **Size / fields**: Shadertoys "contains over 44k renderpasses collected from the Shadertoy.com API"; a datapoint is the whole shader code + API info + metadata (fields: num_passes, has_inputs, name, type, code, title, description, tags, license, author, source). Shadertoys-fine is "a finer variant" (function-level rows) used for fine-tuning. Train/test split sizes: not found (gated).
+- **Construction**: Shadertoy API download → annotate (tree-sitter-glsl function extraction, scancode license detection) → filter to an Arrow dataset. Alternative source "shaders20k" (Shadertoy subset of shaders21k). License: shaders keep their own license (many Shadertoy shaders default to CC BY-NC-SA 3.0; the dataset annotates SPDX per shader); builder Apache-2.0.
+- **Leakage**: the author's own note that Shadertoys-fine-trained models are contaminated for ShaderEval-1 (A2).
+
+### A4. shaders21k (Baradad et al., NeurIPS 2022) — corpus, not an LLM benchmark
+- https://github.com/mbaradad/shaders21k ; https://mbaradad.github.io/shaders21k/ ; paper "Procedural Image Programs for Representation Learning" https://openreview.net/pdf?id=wJwHTgIoE0P
+- 21k OpenGL fragment shaders: 1k from TwiGL + 20k from Shadertoy; provides rendered frames (384×384, "979 frames per second" on one GPU) for visual representation learning. Used by ShaderEval as the 2013–2021 source. Predecessor: "Learning to See by Looking at Noise" https://arxiv.org/pdf/2106.05963 .
+
+### A5. AI Co-Artist (GLSL shader evolution with GPT-4), arXiv 2512.08951 (Nov 2025)
+- Yuksel & Sawaf. https://arxiv.org/abs/2512.08951 ; html https://arxiv.org/html/2512.08951v1
+- Task: text/parent-selection → GLSL fragment shaders (Picbreeder-style evolutionary UI; audio-reactive shaders; VLMs as "autonomous aesthetic judges" for closed-loop evolution).
+- Evaluation: user studies only — "novice users created 4.2 shaders on average with AI assistance versus 0.6 without; experts 6.8 vs 2.9; time to first viable output reduced by 60 %" (from search snippet of the HTML). No public benchmark dataset, no compile-rate table found. Code link: not found.
+
+### A6. ShadAR (IEEE ISMAR-Adjunct 2025; arXiv 2602.17481, Feb 2026)
+- Mei, Wendt, Mueller, Gugenheimer. https://arxiv.org/abs/2602.17481
+- Text → **HLSL** shader (per themoonlight review: "an LLM interprets to generate corresponding HLSL code") compiled live to an AR headset viewport. 2-page demo paper; no dataset, no metrics, no numbers. Code: not found.
+
+### A7. Artificial Analysis MicroEval "LLM Ultimate Challenge: Interactive GLSL Shader Art" (2025)
+- https://artificialanalysis.ai/microevals/llm-ultimate-challenge-interactive-glsl-shader-art-1756340323607
+- Single prompt: one self-contained HTML file using **three.js + a GLSL fragment shader** rendering an animated Julia set with mouse-wheel zoom (cursor-centred), drag-pan, mouse-x drives animation, space pauses; a specified palette (indigo→magenta→orange→yellow). Scoring is human thumbs-up voting. 16 models; those with 1 vote: Qwen3 Coder 480B A35B, MiniMax M1 40k, Kimi K2, Mistral Medium 3, Grok 4, DeepSeek V3 0324, Llama 4 Maverick, Claude 4 Sonnet; o3 and Gemini 2.5 Pro among the 0-vote models. Date not shown. Not a rigorous benchmark, but the only public multi-model GLSL+three.js comparison found.
+
+### A8. Blog: "Generative AI for Procedural 2D Shaders: Teaching a Language Model to Draw With Math" (Medium, Jul 2026)
+- https://medium.com/@niloufarmj/generative-ai-for-procedural-2d-shaders-teaching-a-language-model-to-draw-with-math-2647e15679b6 — fetch returned 403; content not verified. Listed only as a pointer.
+
+---------------------------------------------------------------------------------------------------
+
+## PART B — three.js / WebGL / browser 3D / game engines
+
+### B1. WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis (arXiv 2606.01869, Jun 2026)
+- Lu, Xu, Yu, Jiang, Yu, Wang, Yang, Zhang, Wang, He, Liang. https://arxiv.org/abs/2606.01869 ; html https://arxiv.org/html/2606.01869 ; code/data: anonymous.4open.science/r/WorldCoder-Bench/ (per abstract). License CC BY 4.0.
+- **Task**: natural-language spec (+ optional .glb assets) → executable **three.js** scene whose runtime state must satisfy hidden behavioural contracts (spatial/physical constraints, user controls in sync with engine state).
+- **Dataset**: 2,026 expert-curated tasks; Simulation 649 / Rendering 620 / Application 757; 15 fine-grained domains "including physics, shaders, materials, game, animation, architecture". WorldCoder-Core = 205 tasks (primary leaderboard); WorldCoder-Robust = 615 parameter-perturbed variants (3 per Core task). Construction: 5 PhD researchers hand-crafted seeds over 2 months with peer review → LLM-assisted expansion to 10k+ candidates → annotator filtering → runtime validation in headless browser, expert-authored contracts, unstable tasks discarded → anti-contamination via randomised physical constants/object counts/prompt rewrites.
+- **Execution**: headless Chromium via Playwright; three.js version-locked from a local archive; fixed browser config; program must expose a standardised runtime-state interface `window.__3D_STATE__`; deterministic action sequences with state snapshots; outcomes labelled Runtime_Crash / Check_Fail / Check_Pass with auditable traces.
+- **Metrics**: "StateProbe" protocol. Primary **Verification Coverage (V-Cov)** = proportion of hidden assertions that pass; diagnostics: Affordance Coverage (required objects/controls present), State Coverage (intermediate states reachable), Transition Coverage (action-triggered post-conditions). Contracts calibrated by mutation testing. Also "Return on Automation" and "Time Efficiency Multiplier" (cost/time savings adjusted by correctness).
+- **Numbers**: best GPT-5.4 27.8 % V-Cov on Core, 19.9 % on Robust; Gemini 3.1 Pro 26.5 %; Qwen 3.6-Plus 25.3 %; Claude Opus 4.6 17.5 % (Core). 9 frontier models; no ~7–9B models reported (not found).
+- **Pitfalls**: DOM/screenshot-based scoring is essentially uncorrelated with hidden state correctness — Kendall τb = −0.02 over 1,434 model-task pairs; dominant failures are "state-schema drift and broken interaction chains rather than missing scene elements".
+
+### B2. P3D-Bench: Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning (arXiv 2606.11152, Jun 2026)
+- Yang, Hu, Lin, Zhou, Xu, Zhang, Liu, Yao (Nanjing Univ., Envision). https://arxiv.org/html/2606.11152v1 ; project https://lucasqaq.github.io/p3d/ (page body empty at fetch time); code/data links: not found.
+- **Task**: text or single rendered image → program in one of four formats: minimal JSON, OpenSCAD, CadQuery, **Three.js**; executed and rendered by the benchmark. Three sub-tasks: Text-to-3D (single part), Image-to-3D (multi-part), Assembly-3D (part + assembly annotations).
+- **Dataset**: from Text2CAD v1.1 (176,017 single-part programs) + Fusion 360 Gallery (8,251 assemblies) → dedup/filter with DINOv2 embeddings → complexity-balanced sampling → 400 Text-to-3D, 400 Image-to-3D, 203 Assembly-3D cases.
+- **Execution**: Three.js executed headless, producing triangulated meshes; other formats via their native toolchains.
+- **Metrics**: Validity (compiles + renders); Geometry ([email protected]/0.01, Chamfer distance, normal consistency, IoU); Topology (no-open-edge ratio, inverted-normal ratio, non-manifold ratios); MLLM judge (semantic/geometric/aesthetic ratings + QA accuracy on spec constraints); Part-level (Hungarian-matched part F-scores, part-count accuracy).
+- **Numbers**: GPT-5.5 Text-to-3D JSON: Geo 0.696 / Topo 0.997 / Judge 0.773 / Valid 1.000; Image-to-3D OpenSCAD: 0.567 / 1.000 / 0.592 / 1.000; Assembly-3D CadQuery: 0.570 / 0.948 / 0.527 / 0.985; Three.js on Image-to-3D ≈ Geo 0.556 / Topo 0.828 / Judge 0.569. Open ~7–9B numbers: not found in the fetched text.
+- **Pitfalls**: Three.js scores low on topology because triangulated output lacks manifold guarantees; semantic alignment (~0.8) far exceeds geometric precision (~0.35); CadQuery failures are parameter/geometry errors while JSON/OpenSCAD failures are syntax; PartMatchF1 ≈ 0.5 for the best model.
+
+### B3. ArtifactsBench (Tencent Hunyuan, arXiv 2507.04952, Jul 2025)
+- https://arxiv.org/abs/2507.04952 ; html https://arxiv.org/html/2507.04952 ; code https://github.com/Tencent-Hunyuan/ArtifactsBenchmark ; data https://huggingface.co/datasets/tencent/ArtifactsBenchmark (CC-BY-NC-4.0).
+- **Task**: text prompt → interactive visual artifact (HTML/JS/SVG). 1,825 tasks, 9 domains: Game Development, SVG Generation, Web Applications, Simulations, Data Science, Management Systems, Multimedia Editing, Quick Tools, Others; 74 fine classes; dynamics tiers Static Visual / Mild-to-Moderate Dynamics / High Dynamics / Intensive Interactive. Dataset card shows Canvas/HTML5 games and at least one explicit **three.js task ("Rubik's Cube")**; no WebGL/shader-specific tasks visible.
+- **Construction**: expert sourcing + LLM generation → automated filtering of incomplete/non-interactive items → two-stage dedup (MinHash + semantic similarity); difficulty assigned retrospectively from aggregate model performance to a 30/40/30 Easy/Medium/Hard split.
+- **Execution**: sandboxed headless Chromium via Playwright at 1024×768; three staged screenshots (before / during / after interaction).
+- **Metric**: MLLM-as-Judge (Gemini-2.5-Pro; Qwen2.5-VL-72B as second referee) with a per-task 10-item checklist; dimensions split into vision-oriented (layout, fidelity, motion, feedback, UX) and code-oriented (logic correctness, robustness, modularity, scalability, redundancy avoidance). Score reported 0–100.
+- **Numbers**: paper — Gemini-2.5-Pro ≈ 59, Claude 4.0 Sonnet ≈ 57, DeepSeek-R1-0528 ≈ 51, Qwen 7–14B models 25–32. Leaderboard v1.2 (Aug 2025, GitHub README): GPT-5 72.55, Claude Opus 4.1 59.76, GPT-OSS-120B 57.69. Human agreement: 90.95 % pairwise (Gemini judge), 71.34 % (Qwen judge); 94.4 % ranking consistency with WebDev Arena. JanusCoder reports ArtifactsBench task scores of 80 (8B) / 86 (14B) / GPT-4o 85 — https://arxiv.org/html/2510.23538 (note: different scoring version).
+- **Pitfalls**: all models underperform on Intensive-Interactive and project-level tasks; judge quality varies strongly with judge model (91 % vs 71 % agreement).
+
+### B4. Web-Bench (ByteDance, arXiv 2505.07473, May 2025)
+- Xu, Mao, Guan, Feng. https://arxiv.org/abs/2505.07473 ; https://github.com/bytedance/web-bench (Apache-2.0) ; data https://huggingface.co/datasets/bytedance-research/Web-Bench
+- **Task**: 50 projects × 20 sequentially dependent tasks (1,000 tasks) covering Web Standards (HTML, CSS, JS, DOM, SVG, **WebGL**, TypeScript) and frameworks (React, Vue, Redux, Next.js, Vite, Prisma). Search snippet confirms a project that is "a 3D version of the Snake game by three.js with a portal mechanism added" (https://arxiv.org/pdf/2505.07473). Designed by engineers with 5–10 years' experience; 4–8 h per project for a senior engineer.
+- **Execution / metric**: Web-Agent feeds each task + current project files to the LLM, extracts code, runs Playwright E2E tests. Pass@1 (and Error@1 in the repo; exact definition not fetched).
+- **Numbers**: Claude 3.7 Sonnet 25.1 % Pass@1 (best); open-model numbers not fetched (leaderboard on HF Space per README).
+- **Observations**: much harder than SWE-bench Verified (65.4 %) for the same model.
+
+### B5. WebGen-Bench (NeurIPS 2025 D&B; arXiv 2505.03733)
+- Lu et al. https://arxiv.org/abs/2505.03733 ; https://openreview.net/forum?id=q2VpjD7k1V
+- Text instruction → multi-file website from scratch; instructions by humans + GPT-4o; 3 major / 13 minor categories; **647 test cases** (GPT-4o-generated, manually filtered) executed by a web-navigation agent; appearance judged by GPT-4o. Training set WebGen-Instruct 6,667 instructions.
+- Numbers: Bolt.diy + DeepSeek-R1 27.8 % (best off-the-shelf); Qwen2.5-Coder-32B-Instruct fine-tuned on Bolt.diy trajectories 38.2 %. No 3D/WebGL-specific subset found.
+
+### B6. WebDev Arena (LMArena, launched Dec 2024)
+- https://news.lmarena.ai/webdev-arena/ ; https://arena.ai/blog/webdev-arena
+- Live human-preference leaderboard: two anonymous models build a web app in a sandbox, humans vote; Bradley-Terry Arena Score. By May 2026: 288,203 votes across 77 models. Has a fine-grained category "3D Graphics and Shaders" (per DeepWiki/aiwiki summaries of the category taxonomy: https://deepwiki.com/lmarena/lmarena.github.io/2.3-webdev-arena). Per-category scores for the 3D/shader slice: not found in fetched pages. ArtifactsBench uses WebDev Arena as its human-preference gold standard (94.4 % ranking consistency).
+
+### B7. Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate (arXiv 2605.07342, May 2026)
+- Liu & Tatar. https://arxiv.org/abs/2605.07342 (companion: "Grounding Machine Creativity in Game Design Knowledge Representations…" https://arxiv.org/pdf/2603.07101)
+- **Task**: NL / intermediate representation → **Unity C#** scene code. 26 hand-crafted Unity goal patterns; 858 generation attempts; 4 open-weight LLMs (7B–30B); two IR granularities.
+- **Metrics (four axes)**: compile success; runtime success; structural fidelity; mechanism adherence (F1 against the goal-pattern mechanisms).
+- **Numbers**: direct NL→C#: 43 % mean runtime-pass but mechanism F1 ≈ 0.12; IR-conditioned: runtime rate roughly halved but F1 up to 1.00. Compile-pass rate was anti-correlated with functional correctness. Benchmark, replay logs, per-record metrics released (URL not in abstract).
+- **Pitfall**: "compile-correctness divergence" — the same phenomenon as ShaderMatch's `variation`/`single_color`: code that runs but does nothing meaningful.
+
+### B8. Unreal Blueprint / OpenGL C++ LLM benchmarks
+- Not found as benchmarks. Only tooling: NodeToCode (Blueprint→C++ via LLMs) https://github.com/protospatial/NodeToCode ; a Unity unit-test-generation paper https://dl.acm.org/doi/pdf/10.1145/3663532.3664466 . No OpenGL-C++ LLM benchmark located.
+
+---------------------------------------------------------------------------------------------------
+
+## PART C — Symbolic graphics programs and SVG
+
+### C1. SGP-Bench: "Can Large Language Models Understand Symbolic Graphics Programs?" (ICLR 2025 Spotlight; arXiv 2408.08313)
+- Qiu, Liu, Feng, Liu, Xiao, Collins, Tenenbaum, Weller, Black, Schölkopf. https://arxiv.org/abs/2408.08313 ; html https://arxiv.org/html/2408.08313v3 ; https://openreview.net/forum?id=Yk87CwhBDx ; https://github.com/sgp-bench/sgp-bench ; https://sgp-bench.github.io/
+- **Task**: program text (SVG or CAD DSL) → answer multiple-choice questions about the *rendered image* without seeing it (semantic understanding), plus **semantic consistency** (same questions on SE(2)-perturbed programs — translation/rotation — which changes numerics but not semantics; also reduces leakage risk).
+- **Dataset**: SVG 4,340 questions on 1,085 programs across 19 categories (Semantic 1,085 / Color 864 / Shape 1,217 / Count 819 / Reasoning 355); CAD 2,400 questions (3D DeepCAD 1,000; 3D-complex Fusion360 700; 2D SketchGraphs 700); SGP-MNIST 1,000 (100 per digit). Questions generated by rendering + GPT-4o captioning/question generation, then human verification; 4 options, balanced answers. Sources: Kaggle SVG Icons (SVGrepo, permissive), DeepCAD, Fusion360, SketchGraphs; MNIST-SVG CC BY-SA 3.0.
+- **Numbers (accuracy, SVG / CAD)**: Claude 3.5 Sonnet 67.4 / 74.2; GPT-4o 63.3 / 73.3; Llama-3.1-70B 57.4 / 68.8; Mistral-Large2 57.2 / 71.0; Qwen-2-72B 53.7 / 69.2; **Llama-3.1-8B 46.5 / 57.4**. Symbolic Instruction Tuning (72k programs with VLM-written descriptions) lifts a baseline from 46.7 → 51.4.
+- **Pitfalls**: performance tracks general reasoning strength; SGP-MNIST is near-chance for LLMs (per paper's framing); no rendering is needed at eval time, so this is an *understanding*, not generation, benchmark.
+
+### C2. SGP-GenBench: "Symbolic Graphics Programming with Large Language Models" (TMLR; arXiv 2509.05208, Sep 2025)
+- Chen, Zhang, Huang, Qiu, Zhang, Wen, Liu. https://arxiv.org/abs/2509.05208 ; html https://arxiv.org/html/2509.05208 ; https://spherelab.ai/SGP-Gen/ ; OpenReview https://openreview.net/forum?id=JfK8EHJAQn
+- **Task**: text → SVG. Three parts: Scene (COCO-val, 1,024 captions), Object (SGP-Object-val, 930 internet SVGs with AI captions), Compositional (SGP-CompBench, 3,200 prompts: attribute binding colour/shape/texture, spatial 2D/3D/implicit, numeracy 3–10 objects).
+- **Metrics**: SigLIP text–image cosine; DINO image–image vs reference; VQA-Score; HPS v2; compositional: Gemini-2.5-Flash judge 0–100 (numeracy = 20 % total count + 20 % item presence + 60 % count-per-item). RL reward = format validity + SigLIP/DINO alignment.
+- **Numbers**: VQA avg — Claude 3.7 Sonnet Thinking 0.584 (comp 84.8); Gemini 2.5 Pro Preview 0.563 (76.2); **Qwen-2.5-7B base 0.325 (comp 8.8) → +RL 0.596 (comp 60.8)**.
+- **Observation**: RL "induces finer object decomposition and improved scene coherence"; a 7B model matches frontier on VQA after RL — an existence proof that render-and-compare rewards work for symbolic graphics.
+
+### C3. SVG benchmarks (brief, metrics only)
+- **StarVector / SVG-Bench** (arXiv 2312.11556, v3 May 2025, CC BY 4.0): image/text → SVG; SVG-Bench = 10 datasets, 3 tasks (Image-to-SVG, Text-to-SVG, diagram generation); metric emphasis on DinoScore plus SSIM/LPIPS/MSE (SVGenius reports StarVector SSIM 37.60–56.53). https://arxiv.org/abs/2312.11556 ; https://github.com/joanrod/star-vector ; https://starvector.github.io/starvector/
+- **SVGEditBench** (CVPRW 2024) — MSE on rasterised edit results; **SVGEditBench V2** (arXiv 2502.19453): 1,683 (original SVG, edited GT, instruction) triplets from SVG emoji sets; four metrics spanning raster-based, contour-based and description-based comparisons. https://arxiv.org/abs/2502.19453 ; https://github.com/mti-lab/SVGEditBenchV2
+- **VGBench** (arXiv 2407.10972): VGQA 4,279 (SVG 2,228 / TikZ 1,139 / Graphviz 912) + VGen 5,845 (SVG 2,000 / TikZ 2,000 / Graphviz 1,845); generation metrics Long-CLIP score and FID; understanding accuracy (zero-shot, SVG/TikZ/Graphviz): GPT-4o 64.4/79.8/80.8; GPT-4 54.9/81.0/84.5; Llama-3-70B 53.4/71.1/69.5; **Llama-3-8B 40.0/54.5/58.8**; Qwen2-72B 53.9/78.6/79.6. https://arxiv.org/abs/2407.10972 ; https://vgbench.github.io
+- **LLM4SVG** (CVPR 2025, arXiv 2412.11102): SVGX-SFT (580k instruction pairs); learnable semantic tokens; metrics FID / CLIPScore / Aesthetic / HPS (reported e.g. GPT-2-XL variant FID 64.11, CLIPScore 0.3496, Aesthetic 5.9836, HPS 0.2485 per liner review). https://arxiv.org/abs/2412.11102 ; https://ximinng.github.io/LLM4SVGProject/
+- **Reason-SVG** (CVPR 2026, arXiv 2505.24499): "Drawing-with-Thought" SFT + GRPO with hybrid reward (reasoning presence, structural validity, semantic alignment, visual quality); SVGX-DwT-10k. Numbers not in abstract. https://arxiv.org/abs/2505.24499
+- **SVGenius** (ACM MM 2025, arXiv 2506.03139, CC BY 4.0): 2,377 queries, 24 domains, 8 tasks (Perceptual QA, Semantic QA; Bug Fixing, Code Optimization, Style Editing; Text-to-SVG, Image-to-SVG, Style Transfer), 18 metrics (HPS, Aesthetic, SSIM, LPIPS, DINO; CLIP, rCLIP; PSS path-structure similarity, MSE, rMSE; compression ratio, repair accuracy; 5 LLM-scored style-transfer criteria). Complexity tiers by path count (easy 2.14 / medium 9.87 / hard 16.02 paths). 22 models; e.g. Text-to-SVG easy: Claude 21.35 HPS; Image-to-SVG easy: Claude 23.70 PSS, GPT-4o 23.43. Findings: universal degradation with complexity; "<7B models exhibit complete failure on challenging instances"; style transfer unsolved. https://arxiv.org/abs/2506.03139 ; https://github.com/ZJU-REAL/SVGenius-Bench
+- **VCode** (Microsoft/Oxford, arXiv 2511.02778): image → SVG that preserves symbolic meaning; **CodeVQA** metric = a VLM answers the original questions using only the rendered SVG; GPT-5 46.8 vs raw-image upper bound 61.7. https://arxiv.org/abs/2511.02778
+- **Vector-Bench** (arXiv 2607.19056, Jul 2026): 40 SVG surgical-edit tasks, 34 endpoints; binary spec reward with attribute-aware perceptual tolerances; Unintended Change Rate; best model 15.0 % full-spec success despite 43.7 % mean repair progress. https://arxiv.org/abs/2607.19056
+
+---------------------------------------------------------------------------------------------------
+
+## PART D — Analogous code→render→compare benchmarks (web, plots, TikZ, Manim)
+
+### D1. Design2Code (NAACL 2025; arXiv 2403.03163; CC BY 4.0)
+- Si, Zhang, Li, Yang, Liu, Yang. https://arxiv.org/abs/2403.03163 ; html https://arxiv.org/html/2403.03163 ; https://aclanthology.org/2025.naacl-long.199.pdf
+- Screenshot → HTML/CSS. 484 real webpages from C4 validation (from 14k after automatic length/layout screening, then manual filtering); Design2Code-HARD = 80 pages (26 % >500 tags, 19 % non-English).
+- **Metrics (exact)**: high-level *CLIP* = cosine of CLIP-ViT-B/32 embeddings of screenshots with detected text boxes masked; low-level, on text blocks matched by Jonker–Volgenant assignment: *Block-Match* = total size of matched blocks / total size of all blocks (penalises missing and hallucinated); *Text* = character-level Sørensen–Dice (2·overlap / total chars); *Position* = 1 − max(|xq−xp|, |yq−yp|) on normalised coords; *Color* = CIEDE2000 difference; final visual score = average of the five.
+- **Numbers (Block/Text/Position/Color/CLIP)**: GPT-4o 93.0/98.2/85.5/84.1/90.4; GPT-4V 85.8/97.4/80.5/73.3/86.9; Gemini 1.0 Pro Vision 80.2/94.6/72.3/66.2/84.4; fine-tuned Design2Code-18B (CogAgent) 78.5 Block / 96.4 Text; GPT-4o on HARD Block-Match 56.6.
+- **Human eval / pitfalls**: 49 % of GPT-4V outputs judged interchangeable with originals; 64 % preferred AI pages over originals; human judgment *negatively* correlated with Text similarity (people weigh layout/colour); self-revision prompting gave minimal gains; text-augmented prompting lifted LLaVA Block-Match 50.4 → 68.4.
+
+### D2. Sketch2Code (arXiv 2410.16232, Oct 2024)
+- Li, Zhang, Yang. https://arxiv.org/abs/2410.16232 ; html https://arxiv.org/html/2410.16232 ; https://salt-nlp.github.io/Sketch2Code-Project-Page/
+- Hand-drawn wireframe sketch → HTML. 731 sketches over the 484 Design2Code pages, drawn by Prolific annotators with UI expertise (boxes with X = images, curly lines = text; ~18 % pages sketched by 2 designers, 16.5 % by ≥3).
+- Metrics: Design2Code visual similarity + new **Layout Similarity** = weighted average of IoU over 7 component types (text, image, video, nav, form/table, button, divider); 69.2 % agreement with humans. Interactive-agent modes: feedback-following vs question-asking.
+- Numbers (Layout / Text IoU / Image IoU / human satisfaction): Claude 3.5 Sonnet 21.64/22.51/10.47/36 %; GPT-4o 19.20/17.12/16.19/30 %; Gemini 1.5 Pro 18.25/16.44/14.69/22 %; Claude 3 Opus 12.86/10.43/12.67/10 %; open 8B models (InternVL2-8B, LLaVA-1.6-8B) layout similarity < 11.
+- Pitfalls: feedback rounds give ≤7.1 % visual / 2.7 % layout gains; question-asking hurts after 3–4 rounds; practitioners still prefer question-asking.
+
+### D3. Interaction2Code (ASE 2025; arXiv 2411.03292)
+- Xiao et al. https://arxiv.org/abs/2411.03292 ; html https://arxiv.org/html/2411.03292 ; https://github.com/WebPAI/Interaction2Code ; https://huggingface.co/datasets/whale99/Interaction2Code
+- Interactive prototype (screenshots before/after interaction) → HTML/JS. 127 webpages, 374 interactions, 15 page types, 31 interaction categories (CommonCrawl + GitHub).
+- Metrics: CLIP, SSIM, Position = 1 − max(|xo−xg|, |yo−yg|), BLEU text; widget-level Widget Similarity, Widget Match Rate, Implement Rate. Ten failure types: missing element; no interaction; wrong element; wrong element type; wrong element position; wrong post-interaction position; wrong effect type; effect on wrong element; partial implementation; wrong function.
+- Numbers (CLIP / SSIM / Text): Claude-3.5-Sonnet 0.7130/0.5242/0.5835; GPT-4o 0.7059/0.5259/0.5007; Qwen2.5-VL-72B 0.6573/0.4529/0.4456; Gemini-1.5-flash 0.6151/0.4761/0.4801.
+- Pitfalls: interaction generation lags full-page generation; visually subtle interactions fail; single-modality descriptions insufficient.
+
+### D4. FullFront (arXiv 2505.17399, May 2025)
+- Sun, Wang, Gu, Li, Cheng. https://arxiv.org/abs/2505.17399 ; https://github.com/Mikivishy/FullFront
+- Tasks: Webpage Design (50), Perception QA (1,800 MCQ), Code Generation 400 = Image-to-Code 200 + Text-to-Code 50 + Interaction Authoring 100 + Code Refinement 50. Pages are real sites rewritten to clean HTML by GPT-4o then refined by Claude 3.7 Sonnet (avoids copyright/external deps).
+- Metrics: CLIP score; Gemini visual score on 10 criteria; DOM-tag-sequence structural similarity; content-type similarity; human expert baseline.
+- Numbers: Claude 3.7 Sonnet / Gemini 2.5 Pro ≈ 55 % on perception vs 95 %+ humans; top code scores ≈ 0.68–0.75, open models rarely > 0.60.
+
+### D5. WebIGBench (arXiv 2606.00154, Jun 2026)
+- Wu, Dong, Gao, Chen, Huang, Xiao, Liao. https://arxiv.org/abs/2606.00154 ; https://github.com/anoa12159-hue/WebIGBench_eval
+- 103 real complex interactive webpages, 5 interaction action types, 871 interactive actions; metrics: visual fidelity, code structure, interaction consistency (new automated pipeline). Model numbers not in abstract.
+
+### D6. Image2Struct (NeurIPS 2024 D&B; arXiv 2410.22456; CC BY 4.0)
+- Roberts, Lee, Wong, Yasunaga, Mai, Liang. https://arxiv.org/abs/2410.22456 ; https://github.com/stanford-crfm/image2struct/ ; results https://crfm.stanford.edu/helm/image2struct/v1.0.1/
+- Image → LaTeX / HTML / LilyPond, rendered and compared to the input (round-trip). Renewable data stream (arXiv, GitHub, IMSLP). Five image metrics: pixel similarity, Inception-vector cosine, LPIPS, SSIM, earth-mover similarity; plus compilation success. 14 VLMs; best scores 0.830 (LaTeX equations) vs 0.402 (sheet music).
+
+### D7. Plot2Code (NAACL Findings 2025; arXiv 2405.07990)
+- https://arxiv.org/abs/2405.07990 ; https://aclanthology.org/2025.findings-naacl.164/
+- Plot image (+ instruction) → matplotlib/plotly code. 132 matplotlib + 150 Python-plotly + 86 R-plotly = 368 plots, each with source and GPT-4-written instruction. Metrics: code pass rate; text-match ratio (fine-grained text overlap in rendered plot); GPT-4V rating 1–10 on overall visual similarity (appearance, colours, shapes, positions). GPT-4V rating correlates with humans "for datasets of a certain size".
+
+### D8. ChartMimic (ICLR 2025; arXiv 2406.09961)
+- https://arxiv.org/abs/2406.09961 ; https://github.com/ChartMimic/ChartMimic
+- 4,800 human-curated (figure, instruction, code) triplets; 22 chart types (18 regular + 4 advanced), 201 subcategories; tasks Direct Mimic and Customized Mimic. Metrics: execution rate; low-level text/layout/type/color; high-level GPT-4V score. GPT-4o 82.2 Direct / 61.6 Customized; 3 proprietary + 14 open models. JanusCoder-8B/14B report 64.72 / 66.68 on Customized vs GPT-4o 59.4 (https://arxiv.org/html/2510.23538).
+
+### D9. SWE-bench Multimodal (ICLR 2025; arXiv 2410.03859)
+- Yang et al. https://arxiv.org/abs/2410.03859
+- 617–619 issue-resolution instances from 17 user-facing JavaScript repos (web UI, diagramming, data visualisation, syntax highlighting, interactive mapping); each with ≥1 image in issue or tests. Metric: resolve rate via repo tests. SWE-agent 12 %, next best 6 %; 83.5 % of issues need the image. Specific repo names (e.g. three.js) not listed in the abstract page fetched.
+
+### D10. TikZ: DaTikZ / AutomaTikZ / TikZero (ICCV 2025 highlight; arXiv 2503.11509, CC BY 4.0)
+- Belouadi et al. https://arxiv.org/abs/2503.11509 ; html https://arxiv.org/html/2503.11509 ; https://github.com/potamides/DeTikZify ; AutomaTikZ https://arxiv.org/pdf/2310.00367 ; TikZilla (2026) https://arxiv.org/html/2603.03072v2
+- Caption → TikZ, compiled by LaTeX. DaTikZ v3: 456,469 TikZ programs, ~170k captioned; test 1,000 captioned samples.
+- Metrics: CLIPScore (caption↔image), DreamSim (perceptual image similarity vs reference), KID (distribution), cBLEU, TED (TeX edit distance), MTE (mean token efficiency ≈ compile reliability).
+- Numbers (DSim↑ / KID↓ / CLIP↑ / cBLEU↑ / TED↓ / AVG↑): TikZero+ 56.30/1.83/24.18/1.99/59.01/87.04; TikZero(cos) 52.83/5.10/10.05/1.60/65.51/85.60; GPT-4o 56.46/2.84/31.79/0.33/58.51/79.02; Qwen2.5-Coder-32B 54.47/5.49/24.87/0.29/59.86/48.59. Human BWS with 13 annotators on 100 items.
+- **Pitfall (important for CLIP metrics)**: CLIPScore is inflated by models that copy caption text into the drawing; ROT13-redacting text dropped score ratios from 50 % to 34–51 %, exposing "string matching" rather than visual quality.
+
+### D11. vTikZ — "LLM Code Customization with Visual Results: A Benchmark on TikZ" (EASE 2025; arXiv 2505.04670)
+- Reux, Acher, Khelladi, Barais, Quinton. https://arxiv.org/abs/2505.04670 ; https://github.com/IV2C/VTikZ ; https://huggingface.co/datasets/CharlyR/vtikz
+- 100 curated edit scenarios from 45 TikZ programs (50 scientific diagrams, 50 animal drawings; 25 add / 13 remove / 62 update; 41 easy / 36 medium / 23 hard). Metrics: CompileMetric, LocationMetric (correct lines edited), SuccessCustomizationMetric (matches parameterised solution or image), CrystalBLEU similarity, LineMetric. Results (N=5, T=0.7): GPT-4o 28 % success (57.0 sim, 75.3 line); Llama-3.3-70B ≈16 %; DeepSeek-R1-Distill-70B ≈14 %. Failure taxonomy over 225 wrong variants: feature not found 67, wrong edit 65, partial 41, too many features 19, misunderstanding 20, no change 13.
+
+### D12. Manim: ManiBench (arXiv 2603.13251), ManimBench / ManimTrainer (arXiv 2604.18364), DTVBench
+- ManiBench (Oli, CC BY 4.0): 12 problems, 5 difficulty levels; Alignment Score = Σ(wᵢ·pᵢ·tᵢ)/Σwᵢ (presence × timing of required visual events); Coverage Score = weighted 4 dims (math annotation 35 %, visual mapping 30 %, numeric evidence 20 %, structural clarity 15 %); judged automatically by AST/regex heuristics, no human eval. Zero-shot render success: Claude-Sonnet-4 and Kimi-K2.5 8/12 (66.7 %), Gemini-2.5-Pro and Qwen3.5-Plus 33.3 %, Qwen3-235B 25 %, Qwen-2.5-Coder 0 %; mean coverage 0.17. Failure modes named: **syntactic hallucination** (valid Python, non-existent/deprecated Manim APIs) and **visual-logic drift** (renders but the narrative is wrong). https://arxiv.org/html/2603.13251 ; https://github.com/nabin2004/ManiBench
+- ManimBench (Silva, Lotfi, Ihianle, Shahtahmassebi, Bird): 17 open sub-30B LLMs × 9 train/inference strategies; metrics Render Success Rate and Visual Similarity to reference video (similarity function not stated in abstract). Qwen3-Coder-30B + GRPO + renderer-in-the-loop-with-docs: 94 % RSR, 85.7 % VS (+3 pp VS over GPT-4.1). Code: https://github.com/SuienS/manim-trainer ; https://arxiv.org/abs/2604.18364
+- DTVBench (in JanusCoder, arXiv 2510.23538): 102 tasks (52 Manim from 3Blue1Brown segments + 50 Wolfram demonstrations). https://arxiv.org/html/2510.23538v1
+
+### D13. VisPlotBench / VisCoder2 (arXiv 2510.23642) and JanusCoder (arXiv 2510.23538)
+- VisPlotBench: 888 executable tasks across 8 of 12 languages (Python, JavaScript, TypeScript, C++, R, HTML, SVG, LaTeX, Asymptote, Mermaid, LilyPond, Vega-Lite); Jupyter-kernel execution with strict timeouts; metrics Execution Pass Rate, LLM-judge Task Score, Visual Score. VisCoder2-7B 70.9 % (76.4 % with self-debug), 32B 73.1 % (82.4 %), GPT-4.1 63.4 % (82.4 %), Qwen2.5-Coder-7B 51.2 %. https://arxiv.org/html/2510.23642
+- JanusCoder: JanusCode-800K (charts, web UI, SVG, Manim, Mathematica, R/MATLAB, artifacts, scientific demos); evaluated on PandasPlotBench, ArtifactsBench, DTVBench, ChartMimic, DesignBench, WebCode2M, InteractScience; no shader/three.js domain listed. https://arxiv.org/html/2510.23538 ; https://github.com/InternLM/JanusCoder
+
+### D14. Vision2Code (arXiv 2605.11307, May 2026)
+- Periasami, Wang, Dhingra. 2,169 test examples from 15 source datasets: charts, geometry, graphs, scientific imagery, documents, 3D spatial scenes; VLM rater with dataset-specific rubrics + deterministic guardrails + render-success diagnostics; "leading models perform well on regular chart- and graph-like visuals but remain weak on spatial scenes". https://arxiv.org/abs/2605.11307 ; https://image2code.github.io/vision2code/
+
+### D15. InteractScience (arXiv 2510.09724)
+- Interactive scientific-demonstration web code; hybrid eval = programmatic functional tests (simulated user actions) + visually-grounded qualitative tests with reference snapshots and VLM checklists; 30 LLMs; 5 disciplines. Size: not found in fetched snippet. https://arxiv.org/abs/2510.09724 ; https://huggingface.co/datasets/internlm/InteractScience
+
+### D16. Other web design-to-code corpora (for scale reference only)
+- WebSight v0.1 823k / v0.2 1.92M synthetic pages; Web2Code 1,179.7k instruction pairs (NeurIPS 2024 D&B, https://arxiv.org/html/2406.20098v2); WebCode2M 2.56M real pages (https://arxiv.org/pdf/2404.06369); DesignBench (https://arxiv.org/pdf/2506.06251); WebCoderBench (https://arxiv.org/html/2601.02430v1); UI2App (https://arxiv.org/pdf/2607.06306); Cookie-Bench (https://arxiv.org/pdf/2605.30000).
+
+---------------------------------------------------------------------------------------------------
+
+## PART E — Procedural materials / node graphs / Blender programs
+
+### E1. VLMaterial (ICLR 2025 Spotlight; arXiv 2501.18623)
+- Li, Wu, Solar-Lezama, Zheng, Shi, Bickel, Matusik. https://arxiv.org/abs/2501.18623 ; https://proceedings.iclr.cc/paper_files/paper/2025/file/5393eaf0b723278738788520ec54e4b4-Paper-Conference.pdf
+- Image → Python program that builds a Blender procedural material (node graph) whose flat render matches the image. Open-source procedural-material dataset + LLM-driven program-level augmentation; VLM fine-tuning. Metric names/values not in the abstract page (search snippet mentions RL with image-similarity rewards). Code: not found on abstract page.
+
+### E2. MultiMat (ICLR 2026 poster; arXiv 2509.22151)
+- Belouadi, Boubekeur, Kaiser. Multimodal program synthesis for procedural material node graphs (DAG) from image and text; new production-quality material dataset; claims SOTA on unconditional and conditional synthesis. Numbers not in abstract. https://arxiv.org/abs/2509.22151
+
+### E3. MatFormer (SIGGRAPH 2022) — pre-LLM transformer over 2,816 procedural graphs; https://dl.acm.org/doi/10.1145/3528223.3530173 ; "Reflecting Process Expertise in Procedural Material Generation" (2026, retrieval-augmented LLM → Blender shader nodes) https://arxiv.org/html/2607.13318v2 . "Infinite Texture": not found as a benchmark.
+
+### E4. BlenderGym (CVPR 2025; arXiv 2504.01786)
+- Gu, Huang, Je, Yang, Guibas. https://arxiv.org/abs/2504.01786 ; html https://arxiv.org/html/2504.01786
+- Start Blender scene + goal render → edit Python to reproduce the goal. 245 scenes: procedural geometry 50, lighting 40, procedural material 40, blend shapes 75, object placement 40. Metrics: Photometric Loss (pixel), N-CLIP = 1 − CLIP similarity, Chamfer distance (geometry tasks). Human baseline far ahead (blend shapes: human PL 0.934×10⁻³ vs GPT-4o 9.14×10⁻³). VLM-verifier alignment with humans 0.66 (Claude 3.5 Sonnet) vs 0.79 inter-human.
+
+### E5. SceneCraft (ICML 2024; arXiv 2403.01248) — text → Blender Python with up to ~100 assets; scene-graph planning; human assessment; https://arxiv.org/abs/2403.01248 . Related: LL3M https://arxiv.org/pdf/2508.08228 , BlenderRAG https://arxiv.org/pdf/2605.00632 , Procedural Scene Programs https://arxiv.org/pdf/2510.16147 .
+
+### E6. Analogue in audio: "Benchmarking LLM Code Generation for Audio Programming with Visual Dataflow Languages" (arXiv 2409.00856) — LLM writes Max/MSP-style patches via metaprogramming vs raw JSON nodes; finding: metaprogramming yields more semantically correct code when well-formed. https://arxiv.org/abs/2409.00856
+
+---------------------------------------------------------------------------------------------------
+
+## PART F — Cross-cutting metric definitions and pitfalls (what transfers to a GLSL / three.js eval)
+
+1. **Staged outcome labels beat scalar similarity.** ShaderMatch's ordered labels (incomplete → clone levels → compile error → semantic clone → single_color → variation) and Mage's four axes both show that "compiles" and "renders" are weak proxies: Mage found compile-pass anti-correlated with mechanism F1 (≈0.12 at 43 % runtime pass); WorldCoder-Bench found DOM/screenshot scoring uncorrelated with hidden-state correctness (τb = −0.02).
+2. **Multi-timestamp rendering** is the shader-specific idea (10 fixed, non-periodic times at 512×288 in ShaderMatch; ArtifactsBench uses 3 interaction-staged screenshots; ManiBench scores temporal presence×timing). ShaderMatch's own README flags that its timestamps are unjustified.
+3. **Degenerate outputs must be named explicitly**: `single_color` (ShaderMatch, 1–4 % of samples), "structurally vacuous" scenes (Mage), "visual-logic drift" and "syntactic hallucination" of non-existent APIs (ManiBench), copied caption text gaming CLIPScore (TikZero ROT13 test).
+4. **Image metrics used across the field**: exact pixel equality (ShaderMatch), pixel/SSIM/LPIPS/Inception-cosine/earth-mover (Image2Struct), CLIP with text masked (Design2Code), DINO/SigLIP/VQA-Score/HPSv2 (SGP-GenBench), DreamSim/KID (TikZero), Photometric/N-CLIP/Chamfer (BlenderGym), F-score/Chamfer/IoU/normal consistency + mesh topology ratios (P3D-Bench for three.js meshes).
+5. **Judge rubrics**: GPT-4V 1–10 overall similarity (Plot2Code), GPT-4V low/high-level (ChartMimic), Gemini-2.5-Pro per-task 10-item checklist (ArtifactsBench: 90.95 % pairwise human agreement; Qwen2.5-VL-72B only 71.34 %), Gemini-2.5-Flash 0–100 compositional (SGP-GenBench), Gemini 10-criteria visual score (FullFront), MLLM semantic/geometric/aesthetic + spec-QA (P3D-Bench), CodeVQA (VCode). Design2Code shows human preference anti-correlates with text-exactness metrics.
+6. **Sampling**: ShaderEval results were produced at T=0.2/top_p 0.95; vTikZ at T=0.7 with N=5 (success 28 % for GPT-4o even with 5 tries); ManimBench/VisCoder2 report large gains from renderer-in-the-loop self-debug (GPT-4.1 63.4 → 82.4 % exec pass). No paper found reporting the greedy-decoding degeneration effect for GLSL specifically (the finetune/docs note in this repo is a local observation).
+7. **Contamination controls seen**: gated generations (Shadereval-runs), SE(2) perturbation (SGP-Bench), parameter-randomised Robust split (WorldCoder), renewable fresh data stream (Image2Struct), retrospective difficulty assignment + MinHash/semantic dedup (ArtifactsBench), DINOv2 dedup (P3D-Bench).
+8. **Open ~7–9B vs frontier gaps** (headline): ShaderMatch only covers open ≤20B code models (best error_rate ≈ 0.31; no frontier numbers exist). SGP-Bench Llama-3.1-8B 46.5 vs Claude 3.5 Sonnet 67.4 (SVG). VGBench Llama-3-8B 40.0 vs GPT-4o 64.4 (SVG QA). ArtifactsBench Qwen 7–14B 25–32 vs Gemini-2.5-Pro ≈59. Sketch2Code open 8B < 11 layout vs Claude 3.5 Sonnet 21.6. SGP-GenBench shows the gap closes with render-reward RL on Qwen-2.5-7B (0.325 → 0.596 VQA).
+
+---------------------------------------------------------------------------------------------------
+
+## PART G — Searched and NOT found (as of 2026-09-08)
+
+- A dedicated **text-to-GLSL / text-to-shader benchmark** with a public test set and rendered-image metrics (searched "text-to-shader", "ShaderLLM", "ShaderGPT", "ShaderBench", "GLSL LLM evaluation 2026", "shader generation LLM benchmark 2025/2026"). Only ShaderMatch (function completion), the AA MicroEval (single prompt, human votes), and system papers (AI Co-Artist, ShadAR) exist. "ShaderGPT" is a 14islands/Slanted demo tool (https://www.slanted.de/shadergpt/), not a paper.
+- Any **WGSL or HLSL LLM benchmark** (only ShadAR uses HLSL, no eval).
+- A **ThreeJS-Bench** by that name. The closest are WorldCoder-Bench (three.js scenes, state contracts), P3D-Bench (three.js as one of four output formats), Web-Bench (WebGL/three.js among 50 projects), ArtifactsBench (a few three.js tasks), and the WebDev Arena "3D Graphics and Shaders" category (no per-category numbers found).
+- **p5.js / Processing** LLM benchmark: none found; p5.js appears only as one "arena" in a commercial multi-arena coding leaderboard (search snippet, no primary source) and in Spellburst (HCI system). CreativeBench (arXiv 2603.11863) is about creative *code* generation generally, not creative coding.
+- **Unreal Blueprint** and **OpenGL C++** LLM benchmarks: none.
+- "**Infinite Texture**" as an LLM/procedural-shader benchmark: none.
+- Per-model results table from the ShaderMatch *paper* PDF itself (IEEE/ResearchGate blocked); the HF Shadereval-results dataset was used instead and matches the abstract's 31 % figure.
+- Gated HF cards (Shadertoys, Shadertoys-fine): train/test split counts.
+- GPT-4-class or Claude-class numbers on ShaderMatch: none published.
+
+<!-- ===== methodology.md ===== -->
+
+# Methodology survey: code benchmarks, VLM/LLM judges, image metrics, 3D shape metrics
+
+Compiled 2026-09-08 from primary sources (arXiv abstract/HTML pages, GitHub READMEs, project pages).
+Every number below is taken from the cited URL; "not found" means the source I could read did not state it.
+Rendering note: arXiv-HTML math sometimes duplicates digits when fetched (e.g. "1010" for 10, "0.20.2" for 0.2, "80×80×" for 80×). Where that happened I record the de-duplicated value and flag it.
+
+Format per item: **name** — year/venue — URL(s) — what it measures — exact definition — human-correlation / bias numbers — pitfall — transfer to a 3D-code benchmark.
+
+---
+
+## A. Code benchmarks (construction, metric definitions, noise reporting)
+
+### A1. HumanEval / Codex (Chen et al. 2021)
+- URLs: https://arxiv.org/abs/2107.03374 ; full text https://ar5iv.labs.arxiv.org/html/2107.03374
+- Measures: functional correctness of Python functions from docstrings, 164 problems, "an average of 7.7 tests per problem".
+- Hand-written to avoid contamination: "It is important for these tasks to be hand-written, since our models are trained on a large fraction of GitHub, which already contains solutions to problems from a variety of sources."
+- Unbiased pass@k estimator: generate n >= k samples per problem, c = number correct; pass@k = E_problems[ 1 - C(n-c, k) / C(n, k) ]. "One may be tempted to estimate pass@k with 1-(1-p̂)^k where p̂ is the empirical estimate of pass@1, but we show that it is biased in Appendix A." Numerically stable implementation: product form 1 - prod_{i=n-c+1}^{n} (1 - k/i) (np.prod in the paper's code listing).
+- Sampling: n = 200 samples per problem. Temperature: "For a 679M parameter model, the optimal temperature for pass@1 is T*=0.2 and the optimal temperature for pass@100 is T*=0.8."
+- Metric caveat (match-based metrics): "BLEU score may not be a reliable indicator of functional correctness by showing that functionally inequivalent programs generated by our model (which are guaranteed to disagree with the reference solution on some input) often have higher BLEU scores than functionally equivalent ones."
+- Transfer to 3D-code: use the unbiased estimator with n>k samples per prompt; sample at T≈0.2 for pass@1 and higher T for pass@k; hand-author prompts (do not scrape existing OpenSCAD/Blender tutorials verbatim); do not use text similarity to reference code as the score.
+
+### A2. HumanEval+ / EvalPlus (Liu et al. NeurIPS 2023)
+- URLs: https://arxiv.org/abs/2305.01210 ; full text https://arxiv.org/html/2305.01210
+- Measures: same tasks, 80× more tests: "EvalPlus extends the test-cases of the popular HumanEval benchmark by 80× to build HumanEval+" (HumanEval+ has "764.1 tests" per task on average; HumanEval+-mini distils by 47× to "16.1 tests for each task").
+- Pipeline: "EvalPlus first uses ChatGPT to generate a set of high-quality seed inputs ... we perform type-aware mutation to quickly generate numerous new inputs ... We use differential testing as the oracle to cross-check the output."
+- Ground truth found wrong: "18 defects (11% of problems) even in the original ground-truth in HumanEval" — "Unhandled edge-case: five prior ground-truths fail to handle corner-case inputs ... Bad logic: 10 prior ground-truths incorrectly implement the desired functionality; and Performance issue: three inefficient implementations."
+- Effect on scores: "up-to 19.3% (pass@1) / 24.9% (pass@10) / 28.9% (pass@100) reduction over the evaluated models" (HTML rendered as pass@11/1010/100100). Ranking flips: "WizardCoder-CodeLlama and Phind-CodeLlama ... can actually outperform the proprietary ChatGPT" on HumanEval+ but not on HumanEval.
+- Pitfall: thin test suites (7.7 tests) let wrong code pass and even the reference solutions were wrong 11% of the time.
+- Transfer: reference 3D solutions need adversarial checking (re-render/re-mesh at many parameter values, differential testing across two implementations); expect ~10% of hand-written GT to be defective and budget a verification pass; report how many GT were fixed.
+
+### A3. MBPP (Austin et al. 2021)
+- URL: https://ar5iv.labs.arxiv.org/html/2108.07732
+- "The Mostly Basic Programming Problems dataset contains 974 short Python programs constructed by crowd-sourcing to an internal pool of crowdworkers who have basic knowledge of Python." Each item: "a short problem statement, a single self-contained Python function solving the problem specified, and three test cases".
+- Hand-verified subset: "we manually inspected, edited, and pruned a subset of the questions, yielding 426 hand-verified questions, which we refer to as the edited dataset."
+- Sampling: "we use temperature sampling (with temperature 0.5) to generate 80 samples of code and then execute the code contained in the samples against tests". Greedy vs sampling: "lower temperatures (more greedy decoding) perform better with only a single evaluation allowed, but higher temperature, less greedy strategies begin to solve more tasks within a budget of 10 samples."
+- Pitfall: crowd-written prompts were "somewhat ambiguous"; the sanitized subset exists because of that.
+- Transfer: keep a "sanitized" hand-edited subset; three tests per task is too few (see A2).
+
+### A4. LiveCodeBench (Jain et al. 2024)
+- URLs: https://arxiv.org/abs/2403.07974 ; full text https://arxiv.org/html/2403.07974
+- "we have collected 511 problems from contests across three competition platforms – LeetCode, AtCoder, and CodeForces occurring from May 2023 to the present (May 2024)"; 182 Easy / 206 Medium / 123 Hard using "problem difficulty ratings (sourced from the competition websites)".
+- Contamination control by release-date window: "we only consider problems released after the model's cutoff date to ensure that the model has not encountered the exact problem in the training dataset"; evidence: "DeepSeek-Instruct and GPT-4-O perform considerably worse on problems released since September and November 2023 (their release and cutoff dates respectively!)".
+- Four scenarios: code generation, self-repair, code execution, test output prediction.
+- Tests: LLM-generated: "we use a LLM (here GPT-4-Turbo) to generate tests for the problems", about 17 tests per problem on average (HTML rendered "1717").
+- pass@1 protocol: "we generate 10 candidate answers for each problem ... We use nucleus sampling with temperature 0.2 and top_p 0.95" (HTML rendered "1010", "0.20.2", "0.950.95").
+- Overfitting finding: "models cluster into two groups, ones that perform well on both benchmarks and others that perform well on HumanEval but not on LiveCodeBench".
+- Transfer: timestamp every 3D task and report scores per release window; n=10 at T=0.2 is a cheap pass@1 estimator; a self-repair scenario (model sees render/compile error) is directly applicable to 3D code.
+
+### A5. LiveCodeBench Pro (2025)
+- URLs: https://arxiv.org/abs/2506.11928 ; full text https://arxiv.org/html/2506.11928
+- 584 problems "drawn from top-tier contests including Codeforces, ICPC series, and IOI series"; tiers by Elo: Easy "≤2000", Medium "(2000,3000]", Hard ">3000".
+- Olympiad medalists annotate categories (knowledge-heavy / logic-heavy / observation-heavy) and do "a line-by-line analysis of failed model-generated submissions".
+- Result: "without external tools, the best model achieves only 53% pass@1 on medium-difficulty problems and 0% on hard problems"; "o3-mini commits 34 more algorithm logic errors than human contestants".
+- Transfer: expert-annotated failure taxonomy (per-failure line-by-line labelling) is the model for a 3D "defect checklist"; difficulty tiers defined by an external, model-independent rating.
+
+### A6. SWE-bench (Jimenez et al. ICLR 2024)
+- URLs: https://arxiv.org/abs/2310.06770 ; full text https://arxiv.org/html/2310.06770 ; https://www.swebench.com
+- "2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories"; pipeline from ~90,000 PRs → 11,407 post-conversion → 2,294 post-validation (Table 10).
+- FAIL_TO_PASS: "We filter out task instances without at least one test where its status changes from a fail to pass (henceforth referred to as fail-to-pass test)". PASS_TO_PASS: tests that must keep passing so "prior functionality is properly maintained".
+- Resolution rule: "If the patch applies successfully and all of these tests pass we consider the proposed solution to have successfully resolved the issue." Metrics: % Resolved, % Apply.
+- Environment (original paper): conda envs — "Create executable contexts as conda envs. based on latest task instance per version." (Docker came with the Verified harness, A7.)
+- Task ambiguity: the paper does not analyse it (that came from A7).
+- Transfer: FAIL_TO_PASS/PASS_TO_PASS maps to "the new geometry check must go from failing to passing, and the existing scene checks must still pass" for edit-style 3D tasks; containerise Blender/OpenSCAD versions.
+
+### A7. SWE-bench Verified (OpenAI, Aug 2024)
+- URLs: https://openai.com/index/introducing-swe-bench-verified/ (403 to fetch; text mirrored at https://github.com/irthomasthomas/undecidability/issues/933) ; https://www.swebench.com/verified.html
+- "93 software developers experienced in Python" annotated "1,699 random samples from the SWE-bench test set"; each sample reviewed by three annotators; severity labels "[0, 1, 2, 3]"; "Labels 0 and 1 are minor; labels 2 and 3 are severe"; samples with ensemble severity ≥2 on either the problem statement or the FAIL_TO_PASS tests were removed.
+- Fractions filtered: "38.3% of samples were flagged for underspecified problem statements"; "61.1% were flagged for unit tests that may unfairly mark valid solutions as incorrect"; "68.3% of SWE-bench samples being filtered out" overall; 500 kept.
+- Effect: GPT-4o (Agentless) went from 16% to 33.2%; harness moved to "containerized Docker environments".
+- Later note: OpenAI stopped reporting it ( https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ ; details not fetched).
+- Transfer: budget a 3-annotator, 0–3 severity pass over every 3D task for (a) underspecified prompt and (b) over-specific checks; expect to discard a large fraction; report the discard rate.
+
+### A8. SWE-bench Multimodal (Yang et al. 2024)
+- URLs: https://arxiv.org/abs/2410.03859 ; full text https://arxiv.org/html/2410.03859
+- "617 task instances collected from 17 JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping"; "Each SWE-bench M task instance contains at least one image in its problem statement or unit tests." Dev 102 / test 517.
+- Validation: consistency testing (10 runs per task), and "the authors manually inspect each task instance" for image necessity; environments with "Node.js and Chrome, to support JavaScript execution, visual testing, and webpage rendering in-browser"; "Designing environments ... took an average of ten hours of manual labor per repository".
+- Visual tests: "pixel-level visual testing" via screenshot comparison (69 instances), actual/expected image pairs (67 instances from carbon), plus functional unit tests; Puppeteer/Pixelmatch are "commonplace".
+- Result: best 12.2% (SWE-agent M + GPT-4o).
+- Transfer: closest analogue to render-based grading; run each task's checks N times to confirm determinism before admitting it; pixel-diff tests need tolerances.
+
+### A9. BigCodeBench (Zhuo et al. 2024)
+- URLs: https://arxiv.org/abs/2406.15877 ; full text https://arxiv.org/html/2406.15877 ; https://github.com/bigcode-project/bigcodebench ; Hard subset https://huggingface.co/blog/terryyz/bigcodebench-hard
+- "1,140 fine-grained tasks" covering "723 function calls from 139 popular libraries across 7 domains"; "each task encompasses 5.6 test cases with an average branch coverage of 99%".
+- Authoring: "Data Synthesis, Semi-automatic Program Refactoring and Testing Case Generation, and Human Curation"; "20 authors as annotators for one year in total, with 75% of them having more than 5 years of experience".
+- Splits: Complete = "code generation based on the structured docstrings"; Instruct = "natural-language-oriented instructions".
+- Metric: "we report Pass@1 with greedy decoding for the main experiments in the zero-shot setting". Calibrated Pass@1: "models constantly omit the essential code in the generation and hence fail the tests, we calibrate the generation quality by adding the missing setup" (i.e. re-insert the missing imports/setup before executing).
+- Human performance: "97% (32 out of 33) of sampled tasks can pass all test cases" (11 annotators writing solutions).
+- BigCodeBench-Hard: 148 tasks; selection: Stack Overflow query embedding similarity ("a similarity score above 0.7 is a good threshold"), 626 after dedup, then keep tasks that "require more than two libraries", solution length > 426 tokens, and "solve rate below 50%".
+- Pitfall: README: "batch inference results could vary from batch sizes to batch sizes ... if you want to get more deterministic results for greedy decoding, please set --bs to 1".
+- Transfer: LLM-drafted + human-refined tasks with branch-coverage targets; a "calibrated" variant that forgives boilerplate omissions (imports, `import bpy`) isolates modelling ability from prompt-format failures; a Hard subset chosen by real-user-query similarity and solve rate.
+
+### A10. Design2Code (Si et al. 2024)
+- URLs: https://arxiv.org/abs/2403.03163 ; full text https://arxiv.org/html/2403.03163
+- 484 real webpages from C4 validation (127.9k → 14k after auto-filtering → 484 after manual curation "to check the independence from external files, the absence of sensitive content, and proper formatting").
+- Metrics: high-level Visual Similarity = "similarity of their CLIP embedding" with "CLIP-ViT-B/32 after resizing screenshots to squares"; low-level: Block-Match = "total sizes of all matched blocks divided by the total sizes of all blocks ... including unmatched ones"; Text = "character-level Sørensen-Dice similarity"; Position = "1−max(abs(xq−xp),abs(yq−yp))" on normalised coordinates; Color = "CIEDE2000 color difference formula".
+- Human eval: pairwise, "5 human annotators" per pair with majority vote (≥3); "49% of the AI-generated webpages are considered exchangeable with the reference webpages" and "webpages generated by GPT-4V are preferred in 64% cases".
+- Design2Code-HARD: "80 hard examples" (26% >500 HTML tags, 19% non-English).
+- Auto-vs-human: logistic regression on the automatic metrics predicts human pairwise preference with "79.9% accuracy"; Block-Match and Position most positively associated; Text similarity had the "negative and least significant association with human judgment"; simulated vs annotated win rates r = 0.975 (Pearson), τ = 0.931 (Kendall).
+- Transfer: decompose the score into a global embedding similarity plus low-level element matches (part count/position/colour) and validate the decomposition against pairwise human votes; text/code similarity is a poor proxy.
+
+### A11. Web-Bench (2025)
+- URL: https://arxiv.org/html/2505.07473
+- "50 projects, each consisting of 20 tasks with sequential dependencies", "designed by engineers with 5 to 10 years of experience"; "Test: end-to-end (E2E) test with Playwright", 3.6 E2E cases per task, 72.4 per project.
+- Metric: pass_tasks@1 = "number of tasks passed on first attempt before any failure" (sequential; a task that fails twice ends the project's evaluation). "SOTA (Claude 3.7 Sonnet) achieves only 25.1% Pass@1".
+- Transfer: multi-step scene-building tasks with dependency chains; report the first-failure index.
+
+### A12. WebGen-Bench (2025)
+- URL: https://arxiv.org/html/2505.03733
+- "101 instructions", "647 test cases (4–11 per instruction)", test cases reviewed by "Two computer science Ph.D. students".
+- Grader: a UI agent (WebVoyager) executes the test and returns YES/NO/PARTIAL; alignment with manual judgement "Alignment Rate = N_Manual=Agent / N_total × 100%": Claude-3.5-Sonnet 90.3%, DeepSeek-R1 86.1%, DeepSeek-V3 94.4%.
+- Transfer: agentic graders (a VLM that interacts with the scene/viewport) need an explicit alignment-rate audit vs manual grading.
+
+### A13. ArtifactsBench (2025)
+- URLs: https://arxiv.org/abs/2507.04952 ; full text https://arxiv.org/html/2507.04952
+- "1,825 diverse tasks" across nine domains (413 games, 123 SVG, 441 web, 75 simulation, 122 data science, 314 management, 118 multimedia, 179 tools, 40 other); difficulty split 559/611/655.
+- Pipeline: "We standardize execution with headless Chromium (Playwright) at 1024×768 resolution and deterministic seeds. We capture three staged screenshots (before/during/after scripted interaction)"; judge gets "temporal evidence, the original task, the model's full answer, and a fine-grained checklist"; "10-dimension checklist ... five vision-oriented and five code-oriented dimensions", 10-point scale each, absolute (not pairwise). Judges: Gemini-2.5-Pro and Qwen2.5-VL-72B.
+- Human agreement: "94.4% normalized Footrule consistency" with WebDev Arena ranking ("versus 69.4% for WebBench"); expert pairwise agreement on a 280-instance study: Gemini-2.5-pro 90.95% ("For a given query with m model responses, we can form m(m−1)/2 unique pairs ... count the number of pairs for which the MLLM referee and the human judges agree on the rank ordering"); "Incorporating execution screenshots markedly improves agreement". Checklist calibration "achieves Cohen's κ≥0.8 among annotators".
+- Contamination: "MinHash + semantic similarity" and "screenshot perceptual hashing".
+- Pitfall: "For highly complex, long-horizon, or state-dependent interactions ... discrete sampling may not fully capture the fluidity, correctness, and robustness of the entire interactive experience."
+- Transfer: the closest published template for a render-then-VLM-judge 3D benchmark: fixed renderer settings + deterministic seeds, per-task checklist, absolute 10-point sub-scores, agreement reported both as rank-consistency with an arena and pairwise agreement with experts.
+
+### A14. Aider polyglot (Dec 2024)
+- URL: https://aider.chat/docs/leaderboards/
+- "225 challenging Exercism coding exercises across C++, Go, Java, JavaScript, Python, and Rust"; two attempts (second after test feedback); leaderboard reports pass rate and "Percent cases well formed" (edit-format compliance).
+- Transfer: separate "correct edit format" from "correct result" — for 3D code, separate parse/compile from geometric correctness.
+
+### A15. MultiPL-E (Cassano et al. 2022)
+- URL: https://arxiv.org/abs/2208.08227
+- "a system for translating unit test-driven code generation benchmarks to new languages" — HumanEval/MBPP prompts+tests compiled into "18 additional programming languages"; pass@k; "Codex matches or even exceeds its performance on Python for several other languages".
+- Transfer: port one task spec into OpenSCAD / Blender-Python / three.js / GLSL dialects with the same geometric oracle.
+
+### A16. Noise, variance, contamination
+- **Adding Error Bars to Evals** (Miller 2024) — https://arxiv.org/html/2411.00640 — recommendations: "Computing standard errors of the mean using the Central Limit Theorem"; "When questions are drawn in related groups, computing clustered standard errors"; "Reducing variance by resampling answers and by analyzing next-token probabilities"; "When two models are being compared, conducting statistical inference on the question-level paired differences"; "Using power analysis". Resampling: "Var(sᵢ) = σᵢ²/K" with K resamples per question. Power: to detect a 3% difference at 80% power "the eval will need to contain at least n ≈ 969 independent questions". Reporting: "reporting the standard error of the mean alongside (beneath) the mean".
+- **A Sober Look at Progress in LM Reasoning** (2025) — https://arxiv.org/html/2504.07086 — "Pass@1 values show surprisingly high standard deviation—ranging from 5 to 15 percentage points across seeds" (AIME'24); "A change in just one question shifts Pass@1 by 2.5–3.3 percentage points"; "Bootstrapping over 30 runs substantially stabilizes Pass@1 estimates and should be considered a minimal standard"; temperature changes cause "large variations in performance (upto 15%)", top_p "upto 8%"; hardware/cluster differences "up to 8% for OpenRS-1.5B and 6% for DeepSeek-R1-Distill-7B on AIME'24"; "Report the mean and standard deviation".
+- **Non-Determinism of "Deterministic" LLM Settings** (2024) — https://arxiv.org/abs/2408.04667 — "accuracy variations up to 15% across naturally occurring runs" and "a gap of best possible performance to worst possible performance up to 70%" over eight tasks × 10 runs at supposedly deterministic settings.
+- **GSM1k** (Zhang et al. 2024) — https://arxiv.org/abs/2405.00332 — "GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark"; "accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting"; "a positive relationship (Spearman's r^2 = 0.36) between a model's probability of generating an example from GSM8k and its performance gap".
+- **Evaluating LLMs is a minefield** (Narayanan & Kapoor, Oct 2023) — https://www.cs.princeton.edu/~arvindn/talks/evaluating_llms_minefield/ — three problems: prompt sensitivity ("Are you measuring something intrinsic to the model or is it an artifact of your prompt?"), construct validity, contamination (GPT-4 "perfect results on a coding benchmark before September 5, 2021 and zero afterwards").
+- **Greedy vs sampling / degeneration in code**:
+ - Hot or Cold? AdapT (2023) — https://arxiv.org/abs/2309.02772 — "code tokens can be divided into two categories: challenging tokens that are difficult to predict and confident tokens that can be easily inferred"; adaptive temperature per token beats fixed-temperature decoding (numbers not in abstract).
+ - Code Copycat Conundrum (2025) — https://arxiv.org/html/2504.12608v1 — "10,399 code snippets containing repetition, among which 9,346 cases (89.9%) exceeded the predefined maximum token limit"; 20 repetition patterns at character/statement/block level; "Top-k Sampling with k=10 achieved rep metric improvements of 72.3%" vs greedy; their DeRep gives "208.3% over greedy search" Pass@1 improvement on their repetition set.
+ - Rethinking Repetition Problems of LLMs in Code Generation (2025) — https://arxiv.org/abs/2505.10402 — structural repetition "possesses a fixed structure, which can be inherently reflected in grammar"; grammar-based RPG decays likelihood of repetition-inducing tokens.
+- Transfer: report mean ± SE over ≥10 seeds (≥30 for small sets) and question-level paired differences; clustered SE if several prompts share a scene/object; watch for greedy-decoding repetition loops that hit the token cap (boilerplate-heavy dialects) and treat "hit max tokens" as a distinct failure class; a contamination probe = re-authored twin prompts (GSM1k style).
+
+---
+
+## B. VLM-as-judge and image-generation evaluation protocols
+
+### B1. GenEval (Ghosh et al. NeurIPS 2023)
+- URL: https://arxiv.org/html/2310.11513
+- Six tasks: single object, two object, counting, colors, position, attribute binding; "553 prompts" from templates over "80 MS COCO class names", "11 basic color terms", counts "2, 3, or 4", positions "above, below, to the left of, or to the right of".
+- Pipeline: "Mask2Former trained on MS COCO" (Swin-S); "For all tasks except for counting, we take objects with a confidence above the default threshold of 0.3. For counting, we find that a higher threshold of 0.9 gives the highest human agreement"; colours by "CLIP ViT-L/14" zero-shot on the cropped, background-masked object.
+- Human study: "6,000 annotations on a total of 1,200 images", "5 annotations for each image"; "GenEval obtains 83% agreement with human annotators, where pairwise interannotator agreement is 88%"; on unanimous cases "GenEval obtains 91% overall agreement ... while CLIPScore obtains 87%".
+- Failure modes: "Holes in the object which are incorporated into the segmentation mask can mislead downstream color classification"; "Images with overlapping objects of the same type are difficult for object detectors"; "Simpler artistic renderings are out-of-distribution for the detector".
+- GenEval 2 (Dec 2025) — https://arxiv.org/html/2512.16853v1 — benchmark drift: "GenEval is now saturated, with Gemini 2.5 Flash Image achieving a score of 96.7%"; detector breaks as "the T2I output distribution shifted farther from COCO"; deviation from human scores grew to "17.7%"; GenEval 2: 800 prompts, Soft-TIFA scorer ("generates one question per atom of each prompt ... assigns soft scores"), "AUROC of 94.5%" vs VQAScore 92.4% / TIFA 91.6%; "85.1% of the data points were labeled unanimously by the 3 annotators".
+- Transfer: detector-based checks on renders (object present, count, colour, relative position) are cheap and human-aligned at release but drift; thresholds per sub-task must be tuned against a human set and re-validated over time.
+
+### B2. T2I-CompBench / T2I-CompBench++ (Huang et al. 2023 / TPAMI 2025)
+- URLs: https://arxiv.org/html/2307.06350 ; v3 (++) https://arxiv.org/html/2307.06350v3 ; https://github.com/Karine-Huang/T2I-CompBench
+- ++: 8,000 prompts, four categories (attribute binding: color/shape/texture; object relationships: 2D/3D-spatial, non-spatial; generative numeracy; complex).
+- Metrics: "Disentangled BLIP-VQA" for attributes; UniDet-based spatial rule: first object is left of the second "if x₁<x₂, |x₁-x₂|>|y₁-y₂|" and IoU < 0.1; CLIPScore for non-spatial; 3-in-1 = average of CLIPScore, BLIP-VQA, UniDet for complex; MLLM: MiniGPT4-CoT, GPT-4V, ShareGPT4V.
+- Human correlation (Kendall τ / Spearman ρ): CLIP — color 0.1938/0.2773, shape 0.0555/0.0821, texture 0.2890/0.4008, 2D-spatial 0.2741/0.3548; BLIP-VQA — color 0.6297/0.7958, shape 0.2707/0.3795, texture 0.5177/0.6995; UniDet 2D-spatial 0.4756/0.5136; MiniGPT4-CoT — color 0.3156/0.4151, shape 0.1300/0.1805, texture 0.3453/0.4664, 2D-spatial 0.1096/0.1239; GPT-4V — color 0.5242/0.6465, shape 0.2668/0.3402, texture 0.3944/0.4987, 2D-spatial 0.3456/0.4038, 3D-spatial 0.2560/0.3202, numeracy 0.3777/0.4651, non-spatial 0.4756/0.5337, complex 0.5070/0.5942.
+- Pitfall: "Share-CoT tends to give less diverse ratings, regardless of the prompt provided. GPT-4V can comprehend the evaluation prompts, but it is less able to convert into exact grades."
+- Transfer: specialised per-category checkers (VQA for attributes, detector for spatial) beat both CLIP and a general MLLM; MLLMs are weakest at converting judgement into calibrated numeric grades — prefer binary/checklist questions.
+
+### B3. LLM-as-a-judge / MT-Bench (Zheng et al. NeurIPS 2023)
+- URL: https://arxiv.org/html/2306.05685
+- Position bias (Table 2, fraction consistent when answer order swapped): GPT-4 "65.0%" default / "66.2%" rename; Claude-v1 "23.8%" / "56.2%"; GPT-3.5 "46.2%" / "51.2%"; "Only GPT-4 outputs consistent results in more than 60% of cases." Mitigation: swap positions and only count a win if consistent, otherwise tie.
+- Verbosity bias ("repetitive list" attack): fooled Claude-v1 and GPT-3.5 in "91.3%" of cases, GPT-4 only "8.7%".
+- Self-enhancement: "GPT-4 favors itself with a 10% higher win rate; Claude-v1 favors itself with a 25% higher win rate ... GPT-3.5 does not favor itself" — "our study cannot determine whether the models exhibit a self-enhancement bias".
+- Limited reasoning: math-grading failures "14/20" default, "6/20" CoT, "3/20" reference-guided; recommendation "a reference-guided method, in which we first generate LLM judge's answer independently, and then display it as a reference answer in the judge prompt".
+- Agreement: "The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%)."
+- Transfer: always evaluate both orderings; give the judge a reference (GT render) rather than asking it to reason from scratch; report agreement with and without ties.
+
+### B4. MLLM-as-a-Judge (Chen et al. ICML 2024)
+- URL: https://arxiv.org/html/2402.04788
+- "4,414 image-instruction pairs" from "10 diverse domains"; three settings: Scoring (1–5), Pair Comparison (with tie), Batch Ranking.
+- GPT-4V agreement: scoring Pearson 0.490 (70% human agreement); pair comparison 0.636 accuracy (79.3% agreement); batch ranking Levenshtein 0.361 (62.1%). Gemini: 0.304 / 0.509 / 0.432.
+- Biases: egocentric ("GPT-4V exhibits a slight degree of Egocentricity"); position ("LLaVA replicates this sequence in 88.2% of responses"); length ("GPT-4V and Gemini showing average gains of 0.6 and 0.75 points, respectively" when answers were expanded); hallucination worse in batch ranking than pair comparison.
+- Transfer: pairwise > absolute scoring > batch ranking for VLM judges; expect ~0.6 scoring-correlation ceilings; verbosity of model output (long code comments) can inflate judge scores if the judge sees the code.
+
+### B5. Judging the Judges (Thakur et al. 2024)
+- URL: https://arxiv.org/html/2406.12624
+- TriviaQA, "a random sample of 400 questions"; humans: "Scott's π of 96.2±1.07", "average percentage agreement was 98.52%±0.42%". GPT-4 Turbo and Llama-3 70B ≈ 88 Scott's π; lexical "contains" baseline ≈ 64 π yet rank-correlation "ρ of 0.99"; "Most judge models have rank correlations above 0.7".
+- Pitfalls: "high percent agreement can still give scores that differ 10-20 points from the human-assigned scores"; leniency ("evaluate responses as 'correct' when their evaluation criteria are not completely aligned", P+ "significantly higher than 0.5"); prompt sensitivity ("other models lose alignment with increased instructions").
+- Transfer: report Cohen's κ / Scott's π, not raw agreement; a cheap deterministic check can rank models as well as a judge even when its per-item agreement is poor.
+
+### B6. GPT-4V(ision) as a Generalist Evaluator (Zhang et al. 2023)
+- URL: https://arxiv.org/html/2311.01361
+- Correlations with humans: captioning Pearson 0.401/Spearman 0.499 (hard-negative 0.687/0.612); text-to-image relevance 0.653/0.577, visual clarity 0.318/0.194, object accuracy 0.587/0.584; image editing Spearman 0.4450 (semantic) and 0.4090 (realism) vs "CLIPScore gets Spearman's correlation of 0.0437 for Semantic Consistency"; multi-image-to-text 0.826/0.794.
+- Position bias: consistency "92%" image-to-text vs "51.5%" text-to-image (Table 7). Pairwise agreement with humans on text-to-image "41.43%" (random 33%).
+- Pitfall: "GPT-4V attributed higher scores for visual clarity compared to those determined by human evaluators"; weaker on generated images than on captions.
+- Transfer: expect VLM judges to be far better at "does the render match the text" than at "is the render high quality"; position consistency on images can be near chance — swap.
+
+### B7. VIEScore (Ku et al. 2023) and ImagenHub (Ku et al. 2023)
+- URLs: https://arxiv.org/html/2312.14867 ; https://arxiv.org/abs/2310.01596
+- VIEScore: Semantic Consistency (SC) and Perceptual Quality (PQ) sub-scores 0–10; overall O = [min(α1..αi)·min(β1..βi)]^(1/2) using minimums "to emphasize the importance of meeting all criteria without exception".
+- Spearman with humans over 7 tasks: human-human 0.45; GPT-4o 0.40; GPT-4V 0.33; LLaVA 0.09. "VIEScore (with open-source MLLM) is significantly weaker"; MLLMs "fail to detect minor changes made in image editing, such as small patch edits".
+- ImagenHub: 7 tasks, SC and PQ human ratings; "inter-worker agreement of Krippendorff's alpha on 76% models with a value higher than 0.4"; "None of the existing automatic metrics has a Spearman's correlation higher than 0.2 except subject-driven image generation".
+- Transfer: geometric-mean-of-minimums aggregation penalises any single failed criterion; report human-human correlation as the ceiling; small local defects are invisible to MLLMs.
+
+### B8. GenAI-Bench / VQAScore (Lin et al. 2024) and GenAI-Arena (Jiang et al. 2024)
+- URLs: https://arxiv.org/html/2406.13743 ; https://arxiv.org/html/2406.04485
+- VQAScore = "the probability of a 'Yes' answer to a simple question like 'Does this figure show {text}?'"; GenAI-Bench "1,600 prompts sourced from professional designers"; 3 raters, 1–5 Likert, Krippendorff α 0.72 (images).
+- Pairwise accuracy on GenAI-Bench (image): VQAScore 64.1, CLIPScore 50.8, PickScore 57.1, HPSv2 49.6, ImageReward 56.6; Pearson/Kendall: VQAScore 49.9/39.8, CLIPScore 16.4/11.8, PickScore 35.4/25.0, ImageReward 35.0/24.0, HPSv2 13.9/9.6.
+- GenAI-Arena: anonymous side-by-side, four options, Bradley–Terry; "over 9000 votes" (6300 T2I, 1154 edit, 2024 video); MLLM judges vs human votes: "the best model GPT-4o only achieves an average accuracy of 49.19" (45.59 / 53.54 / 48.46); Gemini-1.5-Pro 48.94; open-source 0–37.81. "93.07%" of sampled votes judged reasonable.
+- Transfer: a yes/no VQA probability is a stronger alignment metric than CLIP or preference models; MLLM pairwise preference on generated visuals is near chance for aesthetics — restrict the judge to verifiable questions.
+
+### B9. DreamBench++ (Peng et al. 2024)
+- URL: https://arxiv.org/html/2406.16855
+- "150 images and 1,350 prompts"; GPT-4o rates concept preservation and prompt following on "integers ranging from 0 (very poor) to 4 (excellent)" with task-specific guidelines and a two-step "Internal Thinking" prompt; "7 human annotators", "each instance is rated by at least two humans".
+- Alignment: "DreamBench++ achieves 79.64% and 93.18% agreement with human's evaluation in concept preservation and prompt following ... +32.59% and +37.23% higher than traditional DINO and CLIP metrics" (Table 4 uses mean Pearson).
+- Transfer: a rubric-anchored 0–4 integer scale with explicit "what to look at" (shape, colour, texture) is more human-aligned than embedding similarity for identity preservation.
+
+### B10. Preference/quality scorers: HPS v2, PickScore, ImageReward
+- HPS v2 (Wu et al. 2023) — https://arxiv.org/html/2306.09341 — "HPD v2 comprises 798,090 human preference choices on 433,760 pairs of images"; 57 contractors; test groups annotated by 10 annotators; prompts rewritten by ChatGPT to remove style words; accuracy on HPD v2 test: CLIP ViT-H/14 65.1, Aesthetic 76.8, ImageReward 74.0, HPS v1 77.6, PickScore 79.8, HPS v2 83.3; single-human vs averaged-human 85.0.
+- PickScore (Kirstain et al. 2023) — https://arxiv.org/html/2305.01569 — "over 500,000 examples and 35,000 distinct prompts"; preference accuracy: Random 56.8, Human Expert 68.0, Aesthetics 56.8, CLIP-H 60.8, ImageReward 61.1, HPS 66.7, PickScore 70.5; model-ranking correlation with real users PickScore 0.790 vs CLIP-H 0.313, ImageReward 0.492, HPS 0.670; "FID (-0.900) ... surprisingly, exhibits a strong negative correlation".
+- ImageReward (Xu et al. 2023) — https://arxiv.org/html/2304.05977 — "8,878 prompts, resulting in 136,892 compared pairs"; dimensions alignment/fidelity/harmlessness, 7-level ratings; annotator-vs-ensemble agreement 53.9%±5.8%, researcher-vs-ensemble 73.4%±6.2%; preference accuracy ImageReward 65.14%, BLIP 57.76%, Aesthetic 57.35%, CLIP 54.82%.
+- Transfer: these are trained on photoreal/artistic T2I distributions; on renders of generated 3D they are out-of-distribution (see T3Bench/MATE-3D numbers); FID can anti-correlate with humans.
+
+### B11. CLIPScore, DINO, LPIPS, SSIM
+- CLIPScore (Hessel et al. 2021) — https://ar5iv.labs.arxiv.org/html/2104.08718 — "CLIP-S(c,v)=w∗max(cos(c,v),0)" with "w=2.5"; RefCLIP-S = harmonic mean of CLIP-S and max reference cosine; Kendall τ Flickr8K-Expert: CLIP-S 51.2, RefCLIP-S 53.0, CIDEr 43.9, SPICE 44.9, BLEU-4 30.8; Flickr8K-CF τb: 34.4 / 36.4 / 24.6 / 24.4 / 16.9. Stated weakness on captions needing "richer contextual knowledge". Insensitivity to composition is documented by others: T2I-CompBench shape τ 0.0555 (B2), GenAI-Bench pairwise 50.8 ≈ chance (B8).
+- DINO / CLIP-I / CLIP-T (DreamBooth, Ruiz et al. 2022) — https://ar5iv.labs.arxiv.org/html/2208.12242 — DINO = "average pairwise cosine similarity between the ViT-S/16 DINO embeddings of generated and real images"; CLIP-I same with CLIP; CLIP-T = "average cosine similarity between prompt and image CLIP embeddings"; DINO preferred because its self-supervised training distinguishes instances, whereas CLIP is trained on text and ignores details absent from captions.
+- LPIPS (Zhang et al. 2018) — https://ar5iv.labs.arxiv.org/html/1801.03924 — unit-normalise deep features per channel, weight channels by w^l, L2, spatial average, sum over layers; BAPPS 2AFC agreement (all distortions): human 73.9%, PSNR 63.2%, SSIM 63.1%, FSIM 63.8%, LPIPS lin 69.2–70.0%, tune 69.6–69.8%, scratch 70.0–70.2%. Caveat: SSIM "was not designed for situations where geometric distortion is a large factor".
+- SSIM (Wang et al. 2004) — https://en.wikipedia.org/wiki/Structural_similarity_index_measure ; CW-SSIM paper https://live.ece.utexas.edu/publications/2009/sampat_tip_nov09.pdf — spatial-domain SSIM "is highly sensitive to translation, scaling, and rotation of images"; alternatives (CW-SSIM) were built to be "insensitive to 'nonstructured' geometric image distortions ... caused by nuisance factors, such as, relative movement of the image acquisition device".
+- Transfer: pixel/structure metrics (SSIM/LPIPS) only make sense with identical camera, lighting and framing; DINO similarity is the better identity/shape proxy than CLIP-I; CLIPScore is near chance on compositional/spatial errors.
+
+### B12. MMMU / MMMU-Pro (Yue et al. 2024)
+- URLs: https://arxiv.org/html/2311.16502 ; https://arxiv.org/html/2409.02813
+- MMMU: "11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks", "30 subjects and 183 subfields", "30 highly heterogeneous image types", collected by "over 50 university students"; dedup by "lexical overlap and source URL similarity"; "Approximately 10% of the problems, classified as very easy" removed; expert accuracy "88.6%"; GPT-4V "55.7%"; error analysis of 150 GPT-4V errors: "35%" perceptual, "29%" lack of knowledge, "26%" reasoning.
+- MMMU-Pro: "We begin by filtering out questions that can be answered by text-only LLMs" (four text models, 10 trials, drop if ≥3 of 4 correct in the majority of trials); options "from four to ten"; "1730 questions" plus a vision-only screenshot version ("3,460 questions"); drops "ranging from 16.8% to 26.9% across models" (GPT-4o 69.1% → 49.7%).
+- Transfer: run a text-only ablation (no render / no image) to detect shortcut solvability of 3D judging prompts; expand distractors.
+
+### B13. 3D-spatial VLM benchmarks (brief)
+- 3DSRBench — https://arxiv.org/abs/2412.07825 (ICCV); VSI-Bench — https://arxiv.org/abs/2412.14171 ("5,000 QA pairs that span eight 3D spatial cognition tasks" from ScanNet/ScanNet++/ARKitScenes); SpatialRGPT-Bench — https://arxiv.org/pdf/2406.01584 (nuScenes/KITTI/SUNRGBD/ARKitScenes/Hypersim); Open3D-VQA — https://arxiv.org/abs/2503.11094 ("73k QA pairs spanning 7 general spatial reasoning tasks"). VisionArena notes "current VLMs struggle with spatial reasoning and planning tasks" ( https://arxiv.org/abs/2412.08687 ).
+- Transfer: VLM judges are weakest exactly on relative-position/metric questions — verify positions from the scene graph, not from the judge.
+
+### B14. VisionArena (Chou et al. 2024)
+- URL: https://arxiv.org/html/2412.08687
+- "230K real-world conversations", "73K unique users, 45 VLMs"; VisionArena-Bench: 500 prompts, GPT-4o judge vs a GPT-4-Turbo anchor; "Spearman Correlation: 97.3%", Kendall 89.7% with live arena; "Length is by far the most influential stylistic factor"; user-vs-expert agreement "0.72 (excluding ties) and 0.56 (including ties)".
+- Transfer: anchor-based pairwise judging reproduces arena rankings; length bias must be controlled (strip comments/prose from code before judging).
+
+### B15. Text-to-3D evaluation protocols
+- **T3Bench** (He et al. 2023) — https://arxiv.org/html/2310.02977 — 300 prompts (100 each: single object, single object with surroundings, multiple objects), GPT-4 generated, filtered for proper nouns. Quality: render "from 161 locations" (level-2 icosahedron), five focal lengths, ImageReward per view, "regional convolution" for t=3, take "the highest score from all viewpoints". Alignment: 12-view BLIP captions → GPT-4 merges into one 3D caption → GPT-4 scores 1–5 vs prompt. Human correlation: quality Spearman 0.784 / Kendall 0.636; alignment Spearman 0.780 / Kendall 0.701; single-view CLIP Spearman 0.638; "1,260 scores" on 30% of prompts.
+- **GPTEval3D** (Wu et al. CVPR 2024) — https://arxiv.org/html/2401.04092 — criteria: text–asset alignment, 3D plausibility, texture details, geometry details, texture–geometry coherency; pairwise: "a large image containing renderings of the 3D asset from four or nine viewpoints" plus surface-normal renders, both orderings; Elo: "σ=arg min_σ ∑_{i≠j} A_ij log(1+10^{(σ_j−σ_i)/400})", init 1000; ~110 GPT-4 prompts on a complexity×creativity grid. Kendall τ vs human Elo: alignment 0.821, plausibility 0.641, coherency 0.564, texture 0.821, geometry 0.795, avg 0.710; CLIP-E avg 0.628, Aesthetic-S 0.671, PickScore 0.562. Human expert κ > 0.53; general users ≈ 0.3. Limitations: "GPT-4V hallucinations", position bias, adversarial attacks, "quadratically growing" comparisons.
+- **3DGen-Bench / 3DGen-Arena** (2025) — https://arxiv.org/html/2503.21745 — "1,020 prompts, including 510 texts and 510 images"; 19 models; "13,680 battle pairs" → "68,400 unique votes" (expert), "56,100" score entries, "8,045 anonymous votes"; five dimensions (geometry plausibility, geometry details, texture quality, geometry–texture coherence, prompt–asset alignment); shown as "360° panoramic videos" in "normal maps, textureless geometry, and fully textured renderings"; 3DGen-Score (CLIP-ViT-H/14, multi-view) pairwise alignment 0.725 (T2-3D) / 0.767 (I2-3D), Kendall τ 0.711 / 0.856; 3DGen-Eval (LLaVA) 0.672 / 0.731; both beat GPT-4V on several dimensions.
+- **Eval3D** (Duggal et al. CVPR 2025) — https://arxiv.org/html/2504.18509 — consistency-among-tools metrics: geometric consistency (analytic normals vs normals predicted by a depth model, "pixel-wise angular difference" inliers at 23°); semantic consistency (per-vertex DINO variance across views); structural consistency (Stable-Zero123 novel view vs actual render, DreamSim); text–3D alignment (TIFA-style QA with LLaVA-NeXT-7B, "globally aligned if and only if, for some viewpoint, the answers of all its neighboring views are aligned"); aesthetics (GPT-4o Elo or ImageReward). 160 prompts (80 single / 80 multi), 10 annotators. Human agreement: geometric 83.0% (GPT-4V 46.9%, ImageReward 44.3%); structural 69.2% (68.9% / 63.7%); aesthetics 87.4% (85.6%, T3Bench 76.5%); text-3D 88.7% (72.8%, T3Bench 56.7%); inter-annotator 83.1%–97.2%.
+- **MATE-3D / HyperScore** (ICCV 2025) — https://arxiv.org/html/2412.11170v1 — 160 prompts × 8 methods = "1,280 textured meshes"; "11-level impairment scale (ranging from 0 to 10)" per ITU-T P.910; "1,280×4×21=107,520 annotations" (21 subjects); SRCC overall: CLIPScore 0.510, ImageReward 0.623, HyperScore 0.792.
+- **T23D-CompBench / Rank2Score** (2025) — https://arxiv.org/html/2509.23841v1 — "3,600 textured meshes", "twelve fine-grained quality dimensions", "129,600 reliable individual annotations"; SRCC Rank2Score 0.908 vs HyperScore 0.809, Q-Align 0.709, HPS v2 0.688.
+- **3D Arena** (2025) — https://arxiv.org/abs/2506.18787 — "123,243 votes from 8,096 users across 19 state-of-the-art models"; ELO; iso3d 100 prompts; "99.75% user authenticity"; presentation bias: "Gaussian splat outputs achieving a 16.6 ELO advantage over meshes" and "textured models receiving a 144.1 ELO advantage over untextured models"; recommends "multi-criteria assessment, task-oriented evaluation, and format-aware comparison".
+- **3D-DefectBench** (2026) — https://arxiv.org/html/2607.10826 — factorial study of VLM defect-detection pipelines: camera (6-view oblique / 14-view equatorial / 14-view multi-ring), visual input (RGB, RGB+depth/normals, geometry-only, normal-only), prompt schema (compact binary / definition-guided / rubric-guided binary / rubric-guided checklist), 12 judge models; nine binary defects (5 geometry, 4 texture); "VLM model explains the most variance"; "RGB is worth far more than any added geometric channel" (model-choice gap ≈0.14 macro MCC vs ≈0.04 for visual input); "The 14-view equatorial and 14-view multi-ring turntables do not improve on the compact six-view oblique turntable (best-vs-worst camera gap ∼0.02 macro MCC)"; best VLM 0.298 expert-geometry MCC vs held-out expert 0.519; recommended default "six-view oblique RGB turntable with a rubric-guided checklist prompt".
+- **Cyc3D** (2026) — https://arxiv.org/abs/2608.28080 — "View-Cycle Structural Consistency, a closed-loop render-regenerate-align protocol"; usability axes "geometric structure, reference-image fidelity, mesh discretization and efficiency, and UV parameterization quality"; "even the strongest methods achieve cycle-stability scores below 48". Human-agreement numbers: not found in abstract.
+- **Cap3D** (Luo et al. 2023) — https://arxiv.org/abs/2306.07279 — multi-view BLIP-2 captions, CLIP filtering, GPT-4 consolidation; "660k 3D-text pairs" on Objaverse; judged against "41k human annotations"; "Cap3D surpasses human-authored descriptions in terms of quality, cost, and speed".
+- Transfer: (i) multi-view grids + normal maps + both orderings + Elo is the standard VLM-judge protocol for 3D, with Kendall τ ≈ 0.7 vs humans; (ii) untextured/format presentation shifts human votes by >100 Elo, so fix the presentation format; (iii) consistency-based metrics (Eval3D) beat MLLM judges on geometry; (iv) a 6-view oblique RGB turntable is as good as 14 views for defect detection; (v) Cap3D-style multi-view captioning is a defensible way to author prompts from existing assets.
+
+---
+
+## C. 3D shape metric methodology
+
+### C1. Chamfer distance / F-score / IoU — Tatarchenko et al. CVPR 2019
+- URL: https://ar5iv.labs.arxiv.org/html/1905.03678
+- CD(G,R) = (1/|R|)∑_r min_g ||r−g||₂ + (1/|G|)∑_g min_r ||g−r||₂ (they use L2, non-squared, mean over both directions).
+- F-score = "the harmonic mean between precision and recall"; precision = "the percentage of reconstructed points that lie within a certain distance to the ground truth"; recall = "the percentage of points on the ground truth that lie within a certain distance to the reconstruction"; sampled "10K points from the surface"; IoU at 128³.
+- Why not CD/IoU: CD "significantly perturbed by the geometric layout of outliers"; IoU is "dominated by the interior parts of objects" and "Low to mid-range scores indicate a significant discrepancy between two shapes". Recommendation: F-score "at distance thresholds of 1% and below" of the bounding-volume side.
+- Finding: retrieval and clustering baselines "outperform recent state-of-the-art methods" — networks do recognition, not reconstruction.
+- Transfer: report F@τ (τ = 1% of the normalised bbox side) as the headline shape metric, CD as secondary; always include a retrieval baseline (nearest training asset) to show the benchmark is not solvable by recall.
+
+### C2. Occupancy Networks protocol (Mescheder et al. CVPR 2019)
+- URL: https://ar5iv.labs.arxiv.org/html/1812.03828
+- Volumetric IoU: "randomly sampling 100k points from the bounding volume and determining if the points lie inside our outside the ground truth / predicted mesh"; Chamfer-L1: "the mean of an accuracy and and a completeness metric. The accuracy metric is defined as the mean distance of points on the output mesh to their nearest neighbors on the ground truth mesh. The completeness metric is defined similarly, but in opposite direction"; "randomly sampling 100k points from both meshes and using a KD-tree"; units: "we use 1/10 times the maximal edge length of the current object's bounding box as unit 1" (this is where the ×10 convention comes from); normal consistency: "the mean absolute dot product of the normals in one mesh and the normals at the corresponding nearest neighbors in the other mesh"; watertight GT via Stutz et al. code.
+- Transfer: 100k surface samples, CD-L1 in units of bbox/10, normal consistency as a cheap surface-quality signal; requires watertight meshes for IoU (non-watertight OpenSCAD/Blender output must be handled explicitly).
+
+### C3. Point-set distribution metrics — Achlioptas et al. 2018, PointFlow (Yang et al. 2019)
+- URLs: https://ar5iv.labs.arxiv.org/html/1707.02392 ; https://ar5iv.labs.arxiv.org/html/1906.12320
+- CD (squared form) d_CH = ∑_{x∈S1} min_y ||x−y||₂² + ∑_{y∈S2} min_x ||x−y||₂²; EMD d_EMD = min_φ:S1→S2 ∑ ||x−φ(x)||₂ (bijection; equal cardinality).
+- JSD: voxel-occupancy marginal distribution divergence; COV: "the fraction of the point clouds in B that were matched to point clouds in A"; MMD: "the average of distances in the matching"; 1-NNA: "Leave-one-out accuracy of the 1-NN classifier", 50% = indistinguishable.
+- Pitfalls: "CD fails to distinguish the inferiority of the r-GAN samples" when density clusters (one summand vanishes); JSD "Only considers the marginal point distributions but not the distribution of individual shapes"; MMD "Very insensitive to low-quality point clouds"; COV "Does not evaluate the quality of generated point clouds". PointFlow uses 2048 points, data "normalized to have zero-mean per axis and unit-variance globally".
+- Transfer: for unconditional/creative 3D-code generation use 1-NNA with CD and EMD; for conditional tasks use per-item CD/F-score; be explicit whether CD is squared-L2 (Achlioptas/PointFlow) or L1/L2-mean (OccNet/Tatarchenko) — values are not comparable across conventions.
+
+### C4. Density-aware Chamfer Distance (Wu et al. NeurIPS 2021)
+- URLs: https://arxiv.org/abs/2111.12702 ; https://arxiv.org/html/2111.12702
+- CD pitfalls: "CD is usually insensitive to mismatched local density", "unbounded value range induces a heavy influence from the outliers"; toy example: "CD hardly changes with the imbalance ratio" at low noise, and "under a high noise level, higher imbalance even results in a lower CD". EMD "dominated by global distribution while overlooks the fidelity of detailed structures" and slow.
+- DCD: d_DCD(S1,S2) = ½[ (1/|S1|)∑_{x∈S1}(1 − (1/n_ŷ)·e^{−α||x−ŷ||₂}) + (1/|S2|)∑_{y∈S2}(1 − (1/n_x̂)·e^{−α||y−x̂||₂}) ], ŷ = NN of x in S2, n_ŷ = query frequency of ŷ; bounded [0,1]; "we fix α=1000 for evaluation"; training α∈[40,100].
+- Transfer: a bounded, density-aware variant is safer than raw CD when generated meshes have wildly different tessellation density (typical of procedural code output).
+
+### C5. Alignment and normalisation before CD — Monnier et al. 2022 (UNICORN)
+- URL: https://ar5iv.labs.arxiv.org/html/2204.10310 (Appendix C.1)
+- "Meshes are centered and normalized so that they exactly fit inside the cube of unit length [-0.5,0.5]³; this is important to obtain results that are comparable. Second, we sample 100k points on the mesh surfaces."; gradient-based ICP "minimizes by gradient descent the Chamfer-L2 distance between the point clouds by jointly optimizing 3 translation parameters, 6 rotation parameters and 3 scaling parameters" (Adam, lr 0.01, 100 iterations; classical ICP "diverge[s] when optimizing scale"); "an ICP pre-processing is crucial for an unbiased 3D reconstruction evaluation" — identical shapes under a different canonical transform otherwise score "dramatically poor".
+- Transfer: for code-generated shapes whose origin/axis convention is unspecified (OpenSCAD vs Blender Z-up etc.), normalise to the unit cube and run scale-aware ICP (or evaluate at a small set of canonical rotations) before CD/F-score; report both aligned and unaligned numbers.
+
+### C6. CAD-specific: DeepCAD (Wu et al. ICCV 2021)
+- URLs: https://ar5iv.labs.arxiv.org/html/2105.09492 ; https://github.com/ChrisWu1997/DeepCAD
+- CD: "we evaluate CD by uniformly sampling 2000 points on the surfaces of reference shape and recovered shape"; median CD reported (Table 2: 0.787 ×10³ for DeepCAD) because of outliers; Invalid ratio = "the percentage of the output CAD models that fail to be converted to point clouds" (i.e. the CAD kernel cannot build a valid solid); Table 2 invalid ratios: Ours+Aug 2.72, Ours 3.30, Alt-ArcMid 3.26, Alt-Trans 3.30, Alt-Rel 3.51, Alt-Regr 4.32 (%). Generation: COV 78.13, MMD 1.45 (×10²), JSD 3.76 (×10²), "randomly sample a reference set of 1000 shapes and generate 3000 shapes", repeated three times; README: `python evaluate_ae_cd.py --src ... --parallel` "for chamfer distance and invalid ratio".
+- Handling empty/degenerate output: DeepCAD excludes invalid models from CD and reports the invalid ratio separately (no penalty CD is assigned) — the two numbers must be read together.
+- Transfer: report (a) invalid/empty-mesh ratio, (b) median CD over valid outputs, (c) mean CD or F-score over all with a defined penalty for invalid outputs; never silently drop failures.
+
+---
+
+## Cross-cutting recommendations distilled (each traceable to the items above)
+1. Estimator: unbiased pass@k with n>k samples (A1); n=10 at T=0.2/top-p 0.95 for pass@1 (A4); greedy only with batch size 1 and a repetition/token-cap failure class (A9, A16).
+2. Noise: mean ± SE via CLT, clustered SE for prompts sharing a scene, paired differences between models, ≥10 seeds (A16); ~1,000 independent items to detect 3-point gaps (A16).
+3. Task validation: 3-annotator 0–3 severity for underspecified prompt and over-specific checker (A7); adversarial test augmentation of GT (A2); text-only ablation of the judge (B12); retrieval baseline (C1).
+4. Automatic geometry: F@1% + CD-L1 (×10 bbox units) + normal consistency on 100k samples after unit-cube normalisation and scale-aware ICP (C1, C2, C5); DCD if tessellation density varies (C4); invalid ratio + median CD separately (C6).
+5. VLM judge: multi-view grid (6 oblique views suffice for defects; 4–9 views + normals for pairwise), both orderings, reference render supplied, per-task checklist with binary questions, absolute 10-point or 0–4 rubric sub-scores, report κ / Scott's π and rank correlation with a human arena (A13, B3, B4, B5, B9, B15).
+6. Image metrics: CLIPScore is near chance on composition (B2, B8); SSIM/LPIPS need identical camera/lighting (B11); DINO for identity (B11); preference models are OOD on renders (B10, B15).
diff --git a/eval/llm/docs/survey_synthesis.md b/eval/llm/docs/survey_synthesis.md
new file mode 100644
index 00000000..7edaf5a2
--- /dev/null
+++ b/eval/llm/docs/survey_synthesis.md
@@ -0,0 +1,80 @@
+# Evaluating LLMs/VLMs on 3D-code generation: a survey synthesis (as of 2026-09-08)
+
+Scope: public benchmarks, datasets and metrics where the model output is a *program* that produces 3D content (bpy, CadQuery/OpenSCAD/FreeCAD/FeatureScript, Infinigen/procedural, GLSL, three.js, scene-layout and animation code), plus general code / VLM-judge / 3D-metric methodology that transfers. Per-benchmark records (≈130 entries, every number copied from a primary source) are in `00_records_all.md` next to this file; this document is the synthesis. Numbers below are quoted as reported; "n/f" = not found in an accessible primary source.
+
+## 1. Comparison table
+
+| Benchmark (year) | Input → language | Size / construction | Execution env | Automatic metrics | Judge / human | Availability |
+|---|---|---|---|---|---|---|
+| [BlenderGym](https://arxiv.org/abs/2504.01786) (CVPR'25) | start `.blend`+script + goal render → edited bpy | 245 instances, 5 task types, hand-crafted from BlenderAlchemy/Infinigen/BlenderKit; human baseline on ~20% | Blender (version n/f), ≥3 views, 8-min limit | Photometric loss, N-CLIP = 1−CLIP, Chamfer (blend-shape) | VLM verifier for best-of-k; human–VLM agreement 0.66 vs inter-human 0.79 | [HF data](https://huggingface.co/datasets/richard-guyunqi/BG_bench_data), no license field |
+| [BlenderLLM / CADBench](https://arxiv.org/abs/2412.14203) (2024) | text → bpy | 500 sim + 200 "wild" test; BlendNet 50k Self-Instruct, 2k human-verified | Blender, 4 renders/script | E_syntax (render without error) | GPT-4o on 12–15 binary checkpoints/sample (8 sub-dims); κ 0.791 vs human | Apache-2.0, [HF](https://huggingface.co/FreedomIntelligence/BlenderLLM) |
+| [3D-PreMise](https://arxiv.org/abs/2401.06437) (2024) | fully-specified text → bpy | 57 objects | Blender | pass/fail by point-cloud match within δ; CD | — | code n/f |
+| [3DCodeBench](https://arxiv.org/abs/2606.01057) (2026) | text (3 caption styles) or 4 views → Blender 5.0 bpy | 212 Infinigen categories (eval) / 12.7k seed instances; GT = migrated Infinigen factories, median 387 lines | Blender 5.0 Cycles, sandbox, 240 s, 4 canonical views | Exec = runs ∧ ≥1 mesh; SigLIP-2 & DINOv3 view-paired; CD; Uni3D; "conditional" vs "penalized" | 3DCodeArena BT-Elo (~3,100 votes, bootstrap CI); SigLIP-2 r=0.964, DINOv3 ρ=0.972 vs Elo | CC BY 4.0, [HF](https://huggingface.co/datasets/YipengGao/3DCode), [code](https://github.com/gaoypeng/3dcodebench) |
+| [BlenderRAG](https://arxiv.org/abs/2605.00632) (2026) | text → bpy | 500 expert-validated triplets (Claude-drafted); eval on 30 OOD prompts | Blender | compile %, normalized CLIP | — | CC BY 4.0, [repo](https://github.com/MaxRondelli/BlenderRAG) |
+| [CodeGen-3D](https://scholarworks.sjsu.edu/faculty_rsca/6835/) (IEEE Access 2026) | text → Blender script | 100 Objaverse/Cap3D prompts | Blender | error rate, multi-view CLIP | GPT-4o pairwise | code n/f |
+| [VIGA / BlenderBench](https://arxiv.org/abs/2601.11109) (2026) | target image → iterative Blender edits | 27–30 episodes, 3 tracks | Blender, ≤10 rounds | PL, N-CLIP, success rate | GPT-4o 0–5 | MIT, [HF](https://huggingface.co/datasets/DietCoke4671/blenderbench) |
+| [SceneCraft](https://arxiv.org/abs/2403.01248) (ICML'24) | text → bpy scene w/ asset retrieval | 40 queries w/ GT constraints + Sintel | Blender | CLIP, constraint score, layout-matrix sim, FVD | human win-rate | code n/f |
+| [SceneEval-500](https://arxiv.org/abs/2503.14756) (2025/26) | text → any layout+meshes | 500 descriptions (100 human, 400 o4-mini+validated), 3 tiers | mesh-level checks | CNT, OOR/OAR (ray-cast), COL, SUP, NAV, OOB | GPT-4o for attributes; Qwen2.5-VL-7B tested | [GitHub](https://github.com/3dlg-hcvc/SceneEval) |
+| [Holodeck](https://arxiv.org/abs/2312.09067) / [LayoutGPT](https://arxiv.org/abs/2305.15393) | text → layout program | 120 paired scenes / 3D-FRONT splits | AI2-THOR / renders | OOB rate, KL, FID, CLIP | human pairwise (Holodeck) | Apache-2.0 / CC BY 4.0 |
+| [SimWorlds / 4DBuildBench](https://arxiv.org/abs/2607.01766) (2026) | text → Blender 5.1 dynamic scene code | 50 hand-authored prompts, D1–D3 | Blender 5.1 (bake) | MPR/SPR engine-state predicates (anti-cheat for faked keyframes) | GPT-5.5 itemized binary judge | CC BY 4.0 |
+| [Text2CAD](https://arxiv.org/abs/2409.17106) (NeurIPS'24) | text (L0–L3) → DeepCAD sequence | ~170k models × 4 prompts; LLaVA-NeXT + Mixtral captions; v1.1 filters basic shapes to ~5% | OCC kernel | F1 (Hungarian), CD ×10³ median/mean, IR | GPT-4V pairwise, 5 designers | CC BY-NC-SA 4.0 |
+| [CADPrompt / CadCodeVerify](https://arxiv.org/abs/2410.05340) (ICLR'25) | text → CadQuery | 200 DeepCAD objects; 2 prompt writers + selector + verifier; expert GT | CadQuery, ≤2 repair iters | PCD (1k pts, ICP), Hausdorff, IoGT, compile % | — | [repo](https://github.com/Kamel773/CAD_Code_Generation), no license file |
+| [Text2CAD-Bench](https://arxiv.org/abs/2605.18430) (2026) | text (geometric / sequence style) → CadQuery | 600 human-curated × 2 styles; L1 200/L2 200/L3 100/L4 100; 3 annotators | CadQuery 2.4, 60 s, isolated | CD (30k pts, ×10³), IR, IoU 256³ | GLM-4.6V, 8 views, 0–10 × 5 dims (L4); humans re-scored subset | CC BY 4.0; code "after accept" |
+| [BenchCAD](https://arxiv.org/abs/2605.10865) (2026) | 4 ortho views → CadQuery; QA; edit | 17,900 programs, 106 industrial families (49% ISO/DIN tables), 3 tiers, 10 OOD families | sandbox, 30 s, volume ≤1e-6 mm³ = degenerate | IoU 256³ (+rot-inv), CD, exec %, essential-op recall, edit acc = clip((IoU_gen−IoU_orig)/(1−IoU_orig)) | blind baselines as floors | data CC BY 4.0, [code](https://github.com/BenchCAD/BenchCAD-main) MIT |
+| [CADBench (DeCoDELab)](https://arxiv.org/abs/2605.10873) (2026) | 5 modalities (views, PBR, mesh, noisy mesh) → CadQuery | 18k from DeepCAD/Fusion360/ABC/MCB/Objaverse | CadQuery | Procrustes-aligned CD, voxel IoU, Surface-IoU (τ=1% diag), VSR | — | [HF](https://huggingface.co/datasets/DeCoDELab/CADBench) (license n/f) |
+| [CADTestBench](https://arxiv.org/abs/2605.07807) (2026) | text → CadQuery | 200 CADPrompt items → 5,937 property tests (≈15/sample), 1,275 mutants | CadQuery B-Rep predicates | Pass-Rate, Requirement Score, IR | tests vs human F1 0.938 (VLM judge 0.549, CD AUC 0.663) | CC BY 4.0, [HF](https://huggingface.co/datasets/dimitrismallis/CADTestBench) |
+| [MUSE](https://arxiv.org/abs/2605.28579) (2026) | design spec → CadQuery assemblies | 106 instances, expert seeds + LLM expansion + human review | code → OCCT validity cascade | executes, watertight/manifold/no-overlap | Gemini 3.1 Pro rubric; r=0.713 (sub-criteria) | CC BY 4.0 / MIT |
+| [P3D-Bench](https://arxiv.org/abs/2606.11152) (2026) | text / image / annotated image → JSON, OpenSCAD, CadQuery, three.js | 400 + 400 + 203 from Text2CAD v1.1 & Fusion360; GPT-5.5-written specs, DINOv2 dedup, 3 tiers | native toolchains; 4-step alignment | Valid; CD, [email protected]/0.01, NC, IoU; NoOE/InvN/NM topology; PartMatchF1 | Gemini 3.1 Pro QA-S/QA-P + J-Sem/Geo/Aes (1–10) | paper only, code "future" |
+| [CAD-Recode](https://arxiv.org/abs/2412.14042) / [cadrille](https://arxiv.org/abs/2505.22914) | PC / views / text → CadQuery | DeepCAD test 8,046, Fusion360 1,725, CC3D 2,973 | CadQuery | median CD (8,192 pts ×10³), IoU, IR; **best-of-10 vs GT** | — | CC BY-NC 4.0 data / Apache-2.0 |
+| [RealCADBench](https://arxiv.org/html/2609.03773) (2026) | text / drawing / photo / render → FreeCAD 1.1.1 Python | 12,632 industrial tasks (1,770 reported), 5 practitioners verify | FreeCAD | exec (non-empty export), Solid/Surface IoU 96³ after signed-PCA | Kimi K2.6 rubric vs input evidence | code "to be released" |
+| [SGP-Bench](https://arxiv.org/abs/2408.08313) (ICLR'25) | program (SVG / CAD DSL) → MCQ | 4,340 SVG + 2,400 CAD Qs, GPT-4o-generated, human-checked; SE(2)-perturbed twins | none (understanding) | accuracy | — | MIT code, [HF](https://huggingface.co/sgp-bench) |
+| [ShaderMatch / ShaderEval-2](https://huggingface.co/spaces/Vipitis/Shadermatch) (LLM4Code@ICSE'25) | comment+header → GLSL body | 467 (paper) / 394 (HF v0.3) functions from Shadertoy API, tree-sitter, SPDX per shader | wgpu-shadertoy, 512×288, 10 fixed timestamps | 8 ordered labels: incomplete, c0/c1/c2-clone, code_error, c4-clone (exact pixels), single_color, variation | — | inputs public, generations gated |
+| [WorldCoder-Bench](https://arxiv.org/abs/2606.01869) (2026) | spec (+glb) → three.js | 2,026 tasks (Core 205, Robust 615 perturbed); 5 PhD seeds → LLM expansion → runtime-validated contracts | Playwright headless Chromium, `window.__3D_STATE__` | V-Cov = hidden assertions passed | — (DOM/screenshot τb=−0.02 vs state) | CC BY 4.0 (anon repo) |
+| [ArtifactsBench](https://arxiv.org/abs/2507.04952) (2025) | text → HTML/JS (incl. three.js tasks) | 1,825; MinHash+semantic dedup; retrospective 30/40/30 tiers | Playwright 1024×768, 3 staged screenshots | — | Gemini-2.5-Pro 10-item checklist; 90.95% pairwise human agreement, 94.4% Footrule vs WebDev Arena | CC BY-NC 4.0 |
+| [Design2Code](https://arxiv.org/abs/2403.03163) (NAACL'25) | screenshot → HTML | 484 C4 pages, manual filter; HARD 80 | browser | CLIP (text masked), Block-Match, Text (Dice), Position, CIEDE2000 | 5-annotator pairwise; metrics predict preference at 79.9% | CC BY 4.0 |
+| [VLMaterial](https://arxiv.org/abs/2501.18623) (ICLR'25) | image → Blender 3.3 material code | 1,640 valid → ~550k augmented; test 44/64/64 | Blender | style loss, SWD, CLIP, correctness (valid fraction) | user study | [repo](https://github.com/mit-gfx/VLMaterial) |
+
+Not found as public benchmarks (searched): a multi-model **OpenSCAD** benchmark other than P3D-Bench; any text→GLSL benchmark with rendered metrics; "ThreeJS-Bench"; WGSL/HLSL; p5.js; Unreal Blueprint; "GenEval-3D"; 3D-Fixup is diffusion image editing, not code; BlenderMCP is a tool with no eval paper; a Blender keyframe-animation code benchmark (closest: 4DBuildBench, Code4D with 10 prompts, SceneActBench dynamic track).
+
+## 2. Metric taxonomy, pitfalls, and what tracks humans
+
+**(a) Validity / execution rate.** Definitions differ by an order of magnitude in strictness: "no exception" (Text-to-CadQuery, BlenderLLM E_syntax), "≥1 mesh object" ([3DCodeBench](https://arxiv.org/html/2606.01057)), "non-empty export" ([RealCADBench](https://arxiv.org/html/2609.03773)), "volume > 1e-6 mm³" ([BenchCAD](https://arxiv.org/html/2605.10865v1)), "compiles ∧ watertight" ([CADEvolve](https://arxiv.org/html/2602.16317v1)), "valid solid, no non-manifold" ([CADBench](https://arxiv.org/html/2605.10873v1)), "executes ∧ B-Rep ∧ STEP round-trip" ([CADEngBench](https://arxiv.org/abs/2608.09296)). Pitfalls reported: (i) *empty output passes* unless a mesh/volume gate exists — 3DCodeBench's formula explicitly requires `|Mesh| ≥ 1`; (ii) *compile–correctness divergence*: Mage found compile-pass anti-correlated with mechanism F1 (≈0.12 at 43% runtime pass, [arXiv 2605.07342](https://arxiv.org/abs/2605.07342)); Text2CAD-Bench found domain models with IR 2–11% but CD 219–267 and IoU≈0 ([arXiv 2605.18430](https://arxiv.org/html/2605.18430v1)); WorldCoder-Bench found screenshot/DOM scores uncorrelated with hidden-state correctness (τb = −0.02, [arXiv 2606.01869](https://arxiv.org/abs/2606.01869)); (iii) *survivor bias*: computing CD/IoU only on executed outputs rewards models that fail often — RealCADBench's Kimi K3 has 34% exec but the highest IoU among survivors; Text2CAD-Bench flags it; [IterCAD-Bench](https://arxiv.org/abs/2606.13368) introduces AUC-TR that counts non-executing samples; 3DCodeBench reports both "conditional" and "penalized (failure = 0)" aggregates. (iv) *API drift as a confound*: Blender 4.x→5.0 deprecations were the dominant failure in 3DCodeBench (two models excluded at <10% exec); ManiBench names "syntactic hallucination" of non-existent APIs ([arXiv 2603.13251](https://arxiv.org/html/2603.13251)). P3D-Bench's error taxonomy (syntax / undefined reference / parameter / geometry) and its finding that OpenSCAD is the most stable format while CadQuery has the highest syntax-failure rate are directly reusable.
+
+**(b) pass@k.** Use the unbiased estimator pass@k = E[1 − C(n−c,k)/C(n,k)] with n > k samples ([Chen et al. 2021](https://arxiv.org/abs/2107.03374)); LiveCodeBench uses n=10 at T=0.2/top-p 0.95 ([arXiv 2403.07974](https://arxiv.org/html/2403.07974)). In 3D code, pass requires a geometric oracle: 3D-PreMise's GPT-4 Pass@1 17.5% vs Pass@5 19.3% ([arXiv 2401.06437](https://arxiv.org/html/2401.06437v1)) shows how flat pass@k is when failures are systematic. Beware *oracle best-of-N*: CAD-Recode and cadrille select the min-CD candidate of 10 *against ground truth*, so their CD/IoU are not pass@1 numbers ([arXiv 2412.14042](https://arxiv.org/html/2412.14042)). Beware mislabelled metrics: Text-to-CadQuery's "top-1 exact match 69.3%" is a Gemini render-judge "same object?" rate, not string match ([arXiv 2505.06507](https://arxiv.org/html/2505.06507)).
+
+**(c) Chamfer / EMD / F-score / IoU.** Conventions are mutually incompatible: 1k points + ICP (CADPrompt), 2k (DeepCAD/GenCAD), 8,192 (CAD-Recode), 10k squared (Text-to-CadQuery, CADSmith), 30k unit-bbox (Text2CAD-Bench), 100k (CADFS, OccNet, MeshCoder); squared-L2 (Achlioptas/PointFlow) vs L1-mean in bbox/10 units (OccNet, [arXiv 1812.03828](https://ar5iv.labs.arxiv.org/html/1812.03828)); median vs mean (Text2CAD: median 0.37 vs mean 26.41 ×10³; CADEvolve and CADmium choose median "to reduce invalidity/outlier bias"). Alignment: [UNICORN](https://ar5iv.labs.arxiv.org/html/2204.10310) calls unit-cube normalisation + scale-aware gradient ICP "crucial for an unbiased evaluation"; CADBench uses Procrustes, RealCADBench/CAD-Coder signed-PCA over rotation combos, P3D-Bench a 4-step normalise/translate/rotate/bounded-scale search; Blender is Z-up while three.js is Y-up, so unaligned CD is meaningless across dialects. Known failure modes: CD is outlier-dominated ([Tatarchenko et al. 2019](https://ar5iv.labs.arxiv.org/html/1905.03678), who recommend F-score at τ ≤ 1% of bbox side) and density-blind ([DCD, NeurIPS'21](https://arxiv.org/abs/2111.12702)); CD-based RL rewards get hacked on thin parts via sparse sampling ([CAD-Coder-CoT](https://arxiv.org/html/2505.19713)); wrong-topology objects can beat correct ones on CD/IoU ([EvoCAD](https://arxiv.org/html/2510.11631v1)); a house can score well on Chamfer and fail every structural test ([DreamHouse](https://arxiv.org/html/2603.24866v1)); CD/IoU "fail to account for the space of valid design variations" a prompt allows — property tests agree with humans at F1 0.938 vs CD AUC 0.663 ([CADTestBench](https://arxiv.org/html/2605.07807v1)). IoU needs watertight solids (resolution 32³ MeshCoder → 96³ RealCADBench → 256³ BenchCAD/Text2CAD-Bench; exact CSG in P3D-Bench); three.js triangle soups are non-manifold and were excluded from topology/part scoring in P3D-Bench. Topology-aware additions now common: Euler characteristic (CADmium, EvoCAD), open/inverted/non-manifold edge ratios (P3D), Betti numbers (CADGenBench), OCCT checks (MUSE), SegE/DangEL/SIR/FluxEE (CAD-MLLM).
+
+**(d) Render-based image metrics.** Pixel loss/SSIM/LPIPS are valid only with identical camera, lighting and framing (BlenderGym's edit setting); SSIM is "highly sensitive to translation, scaling, and rotation" and LPIPS beats it only modestly on 2AFC (69–70% vs 63.1%, [Zhang et al. 2018](https://ar5iv.labs.arxiv.org/html/1801.03924)). CLIPScore saturates and is compositionally blind: Kendall τ 0.06 on shape in [T2I-CompBench](https://arxiv.org/html/2307.06350v3), pairwise accuracy 50.8% ≈ chance on GenAI-Bench ([arXiv 2406.13743](https://arxiv.org/html/2406.13743)), gameable by text copied into the image (ROT13 test in [TikZero](https://arxiv.org/html/2503.11509)), and CodeGen-3D reports judge preferences diverging from CLIP. What does track humans on 3D renders: view-paired SigLIP-2 (Pearson 0.964) and DINOv3 (Spearman 0.972) against 3DCodeBench's Elo; multi-view 3DGen-Score alignment 0.725 vs CLIP 0.661 ([3DGen-Bench](https://arxiv.org/html/2503.21745v1)); DINO is the better identity/shape proxy than CLIP-I ([DreamBooth](https://ar5iv.labs.arxiv.org/html/2208.12242)). Preference scorers (HPSv2, PickScore, ImageReward) are trained on photoreal T2I and correlate only 0.51–0.62 SRCC on textured meshes ([MATE-3D](https://arxiv.org/html/2412.11170v1)); FID anti-correlates with users (−0.90, [PickScore](https://arxiv.org/html/2305.01569)).
+
+**(e) VLM/LLM judges.** Position bias is severe: GPT-4 only 65% consistent under answer swap ([MT-Bench](https://arxiv.org/html/2306.05685)); GPT-4V's text-to-image position consistency 51.5% ([Zhang et al. 2023](https://arxiv.org/html/2311.01361)); BlenderGym's Qwen verifier "consistently favor the second edit candidate". Absolute scoring correlates poorly (GPT-4V Pearson 0.49; pairwise 0.636, [MLLM-as-a-Judge](https://arxiv.org/html/2402.04788)); MLLMs "are less able to convert into exact grades" (T2I-CompBench). What works: binary checklists derived from the prompt — BlenderLLM's 12–15 checkpoints reach κ 0.791 vs human, ArtifactsBench's 10-item checklist 90.95% pairwise agreement (only 71.34% with a Qwen2.5-VL-72B judge); multi-view grids + normal maps + both orderings + Elo (GPTEval3D, avg Kendall τ 0.71, [arXiv 2401.04092](https://arxiv.org/html/2401.04092)); rubric VLM on already-validated geometry (MUSE r=0.713). Limits: geometry judgments are weak (Eval3D's consistency metrics 83% vs GPT-4V 46.9% on geometry, [arXiv 2504.18509](https://arxiv.org/html/2504.18509)); VLMs miss small local edits ([VIEScore](https://arxiv.org/html/2312.14867)); VLM judges score 0.549 F1 vs 0.938 for property tests (CADTestBench); presentation shifts human votes (+144 Elo textured vs untextured, [3D Arena](https://arxiv.org/abs/2506.18787)); CADFusion warns of same-family bias with a GPT-4o judge, Text2CAD-Bench found none with GLM-4.6V. [3D-DefectBench](https://arxiv.org/html/2607.10826) shows the judge model explains most variance, RGB beats added depth/normal channels, and a 6-view oblique turntable matches 14 views. Report Cohen's κ / Scott's π rather than raw agreement ([Judging the Judges](https://arxiv.org/html/2406.12624)).
+
+**(f) Attribute / constraint / edit metrics.** Deterministic checks are the most human-aligned family: BlendNet checkpoints; SceneEval count/attribute/relation tests (all methods < 30% attribute satisfaction); SimWorlds MPR/SPR predicates with anti-cheat for faked keyframes; WorldCoder hidden state contracts calibrated by mutation testing; BenchCAD essential-op recall (loft/sweep/helix near-zero for pretrained models) and edit accuracy normalised by headroom, revealing ~64% of "successful" edits silently corrupt unrelated features. Sequence-level F1/edit distance (Text2CAD Hungarian F1, DeepCAD command/param accuracy) only applies when the target language is a fixed DSL.
+
+**(g) Decoding degeneration and noise.** Greedy decoding produces repetition loops that hit the token cap (89.9% of repetitive outputs, [arXiv 2504.12608](https://arxiv.org/html/2504.12608v1)); ShaderMatch's `incomplete_generation` bucket reaches 37% for small models; MBPP found greedy best for 1 sample but sampling better for 10 ([arXiv 2108.07732](https://ar5iv.labs.arxiv.org/html/2108.07732)). Pass@1 SD is 5–15 points across seeds on small sets, temperature swings up to 15% ([Sober Look](https://arxiv.org/html/2504.07086)); "deterministic" API settings vary up to 15% ([arXiv 2408.04667](https://arxiv.org/abs/2408.04667)); detecting a 3-point gap needs ≈969 independent items at 80% power ([Miller 2024](https://arxiv.org/html/2411.00640)).
+
+## 3. Dataset-construction best practices distilled
+
+1. **Prompt authoring.** Human-authored, multi-annotator prompts (Text2CAD-Bench: 3 engineers check executability, description–geometry correspondence, tier, cross-style consistency; CADPrompt: 2 writers + selector + verifier; BenchCAD experts with ISO tables; MUSE expert seeds → LLM expansion → human review) versus caption-derived prompts (Text2CAD: LLaVA-NeXT multi-view captions + Mixtral, later re-annotated with Qwen2-VL/Qwen2.5-72B; [Cap3D](https://arxiv.org/abs/2306.07279); 3DCodeBench captions validated by Gemini 3.1 Pro and humans). Provide two styles (geometric vs procedural) — Text2CAD-Bench shows geometric prompts win at L1–L2 and procedural at L3, and frontier IR on text→CadQuery spans 11%–93% across papers purely from prompt format. Prompts derived from the GT representation leak (P3D-Bench: QA-P > QA-S because specs were written from JSON).
+2. **Tiered difficulty from a model-independent criterion**: operation count/type (Text2CAD-Bench L1–L4, CADSmith 1–3/3–8/5–15 ops), parameter complexity (BenchCAD easy/medium/hard, >45-point cliff), furniture count (SceneEval), external ratings (LiveCodeBench). Retrospective solve-rate tiers (ArtifactsBench 30/40/30, BigCodeBench-Hard) are acceptable if documented.
+3. **Must-have checklists per item**, authored with the prompt: 12–15 binary checkpoints (BlendNet), ~15 property tests with mutation score > 90% (CADTestBench), 10-dimension checklist calibrated to κ ≥ 0.8 among annotators (ArtifactsBench), keep-in/keep-out volumes (CADGenBench). Budget a SWE-bench-Verified-style pass (3 annotators, 0–3 severity for underspecified prompt / over-specific checker; 68.3% of SWE-bench was discarded, [source](https://www.swebench.com/verified.html)) and expect ~10% of reference solutions to be defective ([EvalPlus](https://arxiv.org/html/2305.01210)).
+4. **Reference renders**: fixed camera/lighting, 4 canonical azimuths at 45/135/225/315° (3DCodeBench), 6 ortho + 2 iso (CADEvolve), 10 views (Text2CAD); a 6-view oblique RGB turntable suffices for defect detection (3D-DefectBench); ship textured and untextured GLBs because presentation shifts votes (3D Arena).
+5. **Seed / perturbation variants**: 60 seeds per category (3DCodeBench), 3 parameter-perturbed twins per core task (WorldCoder-Robust), SE(2)-perturbed programs (SGP-Bench), 15 CMA-ES parameter draws per generator (CADEvolve). These double as contamination probes (GSM1k-style twins).
+6. **Dedup and balance**: DINOv2 near-duplicate removal (P3D), MinHash + semantic + screenshot perceptual hash (ArtifactsBench), ROUGE-L on prompts (T3Bench), class dedup (3DCodeBench); cap trivial classes (Text2CAD v1.1 and CADmium reduce cuboids from ~25% to 5%).
+7. **Contamination and held-out families**: the DeepCAD 8,046-model test split is shared by Text2CAD, CADmium, UniCAD and IterCAD and sits in the training data of most fine-tuned CAD models; use held-out families (BenchCAD: 10 OOD families; Fusion360 is outside ABC; MCB), private GT with leaderboard-only scoring (CADGenBench), gated generations (Shadereval-runs), date-stamped items with post-cutoff reporting (LiveCodeBench), blind-input baselines as floors (BenchCAD), and a text-only/no-image ablation of the judge prompt ([MMMU-Pro](https://arxiv.org/html/2409.02813)). Always include a retrieval baseline (nearest training asset) — Tatarchenko showed retrieval matches SOTA reconstruction.
+8. **Licensing**: Fusion 360 Gallery is non-commercial and non-redistributable; CAD-Recode, Text2CAD, CAD-Editor, CADFusion, ArtifactsBench are CC BY-NC(-SA); Shadertoy shaders default to CC BY-NC-SA 3.0 (tag SPDX per item); Onshape-derived data follows Onshape ToS; Infinigen is BSD-3.
+
+## 4. Recommended protocol for a new multi-dialect benchmark (bpy, CadQuery, OpenSCAD, GLSL, three.js; text→code and image→code; 8B open reference)
+
+**Tasks and items.** Three tracks: text→code, image→code (4 fixed views, 45/135/225/315°), and edit (start program + goal, graded FAIL_TO_PASS/PASS_TO_PASS style: new checks must pass, existing checks must still pass). Per dialect ≥ 200 items (≥ 1,000 total if 3-point model differences must be resolved); tiers by operation count; each item ships prompt (two styles), reference program, reference GLB, 4–6 reference renders, 10–15 binary must-have checks, and ≥ 3 seed/parameter twins. Pin toolchains: Blender 5.x headless Cycles, 240 s; CadQuery 2.6 + trimesh, 60 s; OpenSCAD CLI (nightly/Manifold backend) → STL; GLSL via wgpu-shadertoy at 512×288 at ≥ 10 non-periodic timestamps; three.js in Playwright headless Chromium with a `window.__3D_STATE__`-style export plus GLTF export for mesh metrics. Record the toolchain version in every result row (API-version drift is the largest single confound).
+
+**Metrics to report (all four layers, never one).**
+1. *Validity*: exec = runs within budget ∧ ≥ 1 mesh ∧ volume > ε; plus separate flags for watertight, timeout, and token-cap-hit (repetition), and an error taxonomy (syntax / undefined API / parameter / geometry). For GLSL add ShaderMatch's `single_color` and `incomplete` labels.
+2. *Geometry*: normalise to unit cube, scale-aware ICP (or search 24 canonical rotations), 100k surface samples; headline **F-score@1% bbox**, plus CD-L1 (bbox/10 units, median and mean), voxel IoU at 128³ only for watertight outputs, normal consistency; Euler/open-edge/non-manifold ratios for CAD dialects; skip topology for three.js. Report *penalized* (failure = worst) as primary and *conditional* as secondary.
+3. *Render*: view-paired DINOv3 and SigLIP-2 cosine (the two metrics with r ≥ 0.96 to human Elo in 3DCodeBench); LPIPS only in the edit track (same camera); no CLIPScore headline.
+4. *Semantics*: must-have checklist pass rate judged by a frontier VLM given the reference render, both orderings, binary questions only; report κ against a 200-item human-labelled subset and rank correlation with a pairwise human arena (BT-Elo, ≥ 1,000 bootstrap resamples). Composite = geometric mean of layer scores (VIEScore's min-based aggregation is the alternative), never a single scalar.
+
+**Decoding and sampling.** Reference 8B baseline (e.g. Qwen2.5-Coder-7B/Qwen3-8B; comparable public points: Qwen2.5-Coder-7B 0.353 avg / 31.4% syntax errors on CADBench-Sim before self-improvement, Llama-3.1-8B 46.5 on SGP-Bench SVG, deepseek-coder-6.7b ShaderMatch error rate 0.308): n = 10 samples at T = 0.2 / top-p 0.95, seeds fixed, batch size 1; report unbiased pass@1 and pass@5 on the validity+checklist oracle and mean metric over samples; also run greedy as a diagnostic with the token-cap/repetition rate reported (expect degeneration on boilerplate-heavy OpenSCAD/GLSL). API models: provider defaults, n = 5, thinking budget stated. Multi-turn (≤ 2 retries with truncated traceback) and agent-harness settings are separate rows — 3DCodeBench shows retries lift exec 0.70 → 0.97 without changing conditional quality, so single-shot and multi-turn must never be mixed.
+
+**Noise reporting.** Mean ± CLT standard error per metric, clustered by object family; question-level paired differences with CIs for model-vs-model claims; ≥ 3 full seeds (≥ 30 bootstrap draws for pass@k); per-tier and per-dialect breakdowns; state exact temperature/top-p/max-tokens; flag any model excluded for < 10% exec rather than dropping it silently.
+
+**Contamination controls.** Held-out object families never released with code; seed-twin evaluation (score gap between canonical and twin items flags memorisation); release-date stamping; a blind baseline (no prompt / black image) as floor; a retrieval baseline as a ceiling check; text-only judge ablation.
diff --git a/eval/llm/download.py b/eval/llm/download.py
new file mode 100644
index 00000000..81ecdf0c
--- /dev/null
+++ b/eval/llm/download.py
@@ -0,0 +1,49 @@
+"""Download the evaluation assets from the Hub into $CV3D_EVAL_DATA/hub (needs a token with access to
+`ilabai/*`; `hf auth login` or HF_TOKEN).
+
+ --core 212 3DCodeBench tasks (YipengGao/3DCode/3DCodeBench), the five held-out test parquets and
+ mixes.json (ilabai/3dcodeverse-llamafactory), the 3dcodeverse dataset card + _reports/ +
+ factories metadata (~30 MB)
+ --views the official 4 GT reference views per bench task (3DCodeBench_ModelLogs/inputs) (~930 MB)
+ --logs frontier-model generation logs of the official benchmark (text_to_3D / image_to_3D parquets),
+ re-executable here for reference numbers (~16 MB)
+ --gt-tar 3dcodebench/factories_geo/samples-000.tar with the canonical object.glb per factory (2.1 GB)
+"""
+from __future__ import annotations
+
+import argparse
+
+from huggingface_hub import snapshot_download
+
+from . import config
+
+
+def dl(repo: str, patterns: list[str], sub: str, workers: int = 16) -> None:
+ p = snapshot_download(repo, repo_type="dataset", allow_patterns=patterns, local_dir=str(config.HUB_DIR / sub), max_workers=workers)
+ print(f"[download] {repo} {patterns[:2]}... -> {p}", flush=True)
+
+
+def main() -> None:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--core", action="store_true")
+ ap.add_argument("--views", action="store_true")
+ ap.add_argument("--logs", action="store_true")
+ ap.add_argument("--gt-tar", action="store_true")
+ a = ap.parse_args()
+ if not any(vars(a).values()):
+ a.core = True
+ if a.core:
+ dl("YipengGao/3DCode", ["3DCodeBench/*", "3DCodeBench_ModelLogs/README.md", "README.md"], "3DCode")
+ dl("ilabai/3dcodeverse-llamafactory", ["test/*", "*.json", "README.md"], "3dcodeverse-llamafactory")
+ dl("ilabai/3dcodeverse", ["_reports/*", "dataset_info*.json", "README.md", "3dcodebench/factories_geo/metadata.parquet",
+ "3dcodebench/factories_tex/metadata.parquet", "data/build_report_text.json"], "3dcodeverse")
+ if a.views:
+ dl("YipengGao/3DCode", ["3DCodeBench_ModelLogs/inputs/*"], "3DCode")
+ if a.logs:
+ dl("YipengGao/3DCode", ["3DCodeBench_ModelLogs/data/text_to_3D.parquet", "3DCodeBench_ModelLogs/data/image_to_3D.parquet"], "3DCode")
+ if a.gt_tar:
+ dl("ilabai/3dcodeverse", ["3dcodebench/factories_geo/samples-000.tar"], "3dcodeverse", workers=4)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/eval/llm/execute.py b/eval/llm/execute.py
new file mode 100644
index 00000000..5f6c87ef
--- /dev/null
+++ b/eval/llm/execute.py
@@ -0,0 +1,66 @@
+"""Execute every generated program in a gen dir with the suite's executor → <gen_dir>/exec_results.jsonl."""
+from __future__ import annotations
+
+import json
+from collections import Counter
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+
+from . import executors
+from .executors import CODE_EXT
+
+# Blender and Chromium are heavy; CadQuery/OpenSCAD/glslang are light.
+DEFAULT_WORKERS = {"blender": 8, "cadquery": 8, "openscad": 6, "glsl": 6, "threejs": 4}
+
+
+def _one(dialect: str, task_dir: Path, timeout: int | None) -> dict:
+ code_path = task_dir / f"code.{CODE_EXT[dialect]}"
+ code = code_path.read_text() if code_path.exists() else ""
+ if not code.strip():
+ rep = {"status": "FAIL", "error": "empty answer (no code extracted)", "latency_s": 0.0, "mesh": None}
+ else:
+ rep = executors.run(dialect, code, str(task_dir / "exec"), timeout)
+ rep = {k: v for k, v in rep.items() if k not in ("traceback", "stdout_tail")} | {"id": task_dir.name}
+ return rep
+
+
+def execute_dir(gen_dir: Path, dialect: str, workers: int | None = None, timeout: int | None = None,
+ ids: list[str] | None = None, resume: bool = True) -> list[dict]:
+ gen_dir = Path(gen_dir)
+ out_path = gen_dir / "exec_results.jsonl"
+ prev: dict[str, dict] = {}
+ if resume and out_path.exists():
+ for l in out_path.open():
+ r = json.loads(l)
+ prev[r["id"]] = r
+ task_dirs = sorted(p for p in gen_dir.iterdir() if p.is_dir() and (ids is None or p.name in ids))
+ todo = [p for p in task_dirs if p.name not in prev]
+ workers = workers or DEFAULT_WORKERS[dialect]
+ print(f"[exec] {gen_dir.name} ({dialect}): {len(todo)} to run, {len(prev)} cached, {workers} workers", flush=True)
+ res = list(prev.values())
+ with ThreadPoolExecutor(workers) as ex:
+ futs = {ex.submit(_one, dialect, p, timeout): p for p in todo}
+ for i, fu in enumerate(as_completed(futs), 1):
+ res.append(fu.result())
+ if i % 25 == 0:
+ print(f"[exec] {gen_dir.name} {i}/{len(todo)}", flush=True)
+ res.sort(key=lambda r: r["id"])
+ with out_path.open("w") as f:
+ for r in res:
+ f.write(json.dumps(r) + "\n")
+ c = Counter(r["status"] for r in res)
+ print(f"[exec] {gen_dir.name}: {dict(c)} | OK {c.get('OK', 0)}/{len(res)} = {100 * c.get('OK', 0) / max(1, len(res)):.1f}%", flush=True)
+ return res
+
+
+if __name__ == "__main__":
+ import argparse
+
+ ap = argparse.ArgumentParser()
+ ap.add_argument("gen_dir", type=Path)
+ ap.add_argument("--dialect", required=True, choices=sorted(CODE_EXT))
+ ap.add_argument("--workers", type=int)
+ ap.add_argument("--timeout", type=int)
+ ap.add_argument("--no-resume", action="store_true")
+ a = ap.parse_args()
+ execute_dir(a.gen_dir, a.dialect, a.workers, a.timeout, resume=not a.no_resume)
diff --git a/eval/llm/executors/__init__.py b/eval/llm/executors/__init__.py
new file mode 100644
index 00000000..2f95c955
--- /dev/null
+++ b/eval/llm/executors/__init__.py
@@ -0,0 +1,36 @@
+"""Executors: run one generated program of a dialect and report what it produced.
+
+Every executor has the signature ``run(code: str, workdir: str, timeout: int | None = None) -> dict`` and
+returns a JSON-serialisable report with a **uniform status vocabulary**:
+
+ OK ran, produced non-empty output (mesh / compiled shader / non-blank canvas)
+ EMPTY ran without error but produced nothing usable (no mesh, blank canvas)
+ FAIL the program raised / failed to compile / JS error
+ CRASH the host (Blender, node, browser) died without a report
+ TIMEOUT exceeded the per-dialect wall-clock limit
+
+plus ``error`` (str | None), ``latency_s`` (float) and, where applicable, ``mesh`` (path to GLB/STL or None),
+``render`` (for glsl: OK/STATIC/FAIL) and ``shot`` (screenshot path).
+
+Per-dialect wall-clock limits (seconds) — identical for every suite of that dialect, unlike the old code
+which used 300 for 3DCodeBench and 180 for the Blender held-out set.
+"""
+from __future__ import annotations
+
+from . import blender, cadquery, glsl, openscad, threejs
+
+TIMEOUTS = {"blender": 300, "cadquery": 90, "openscad": 120, "glsl": 60, "threejs": 60}
+CODE_EXT = {"blender": "py", "cadquery": "py", "openscad": "scad", "glsl": "frag", "threejs": "html"}
+RUNNERS = {
+ "blender": blender.run,
+ "cadquery": cadquery.run,
+ "openscad": openscad.run,
+ "glsl": glsl.run,
+ "threejs": threejs.run,
+}
+
+STATUSES = ("OK", "EMPTY", "FAIL", "CRASH", "TIMEOUT", "MISSING")
+
+
+def run(dialect: str, code: str, workdir: str, timeout: int | None = None) -> dict:
+ return RUNNERS[dialect](code, workdir, timeout or TIMEOUTS[dialect])
diff --git a/eval/llm/executors/blender.py b/eval/llm/executors/blender.py
new file mode 100644
index 00000000..c2c1e6ea
--- /dev/null
+++ b/eval/llm/executors/blender.py
@@ -0,0 +1,48 @@
+"""Blender-Python executor: headless Blender 5.x runs the script, exports a GLB, writes a JSON report.
+
+The in-Blender part is ``blender_runner.py`` (verbatim copy of finetune/eval/blender_runner.py): it deletes
+every default object and purges orphan data first (so an empty script cannot pass on the default cube),
+runs the script with ``runpy``, counts depsgraph-evaluated MESH objects, and exports ``export_yup=True``.
+"""
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+import time
+from pathlib import Path
+
+from .. import config
+
+RUNNER = Path(__file__).with_name("blender_runner.py")
+
+
+def run(code: str, workdir: str, timeout: int = 300) -> dict:
+ blender = config.find_blender()
+ if blender is None:
+ return {"status": "CRASH", "error": "blender not found (set CV3D_BLENDER)", "latency_s": 0.0, "mesh": None}
+ wd = Path(workdir).resolve()
+ wd.mkdir(parents=True, exist_ok=True)
+ script, glb, rep_path = wd / "code.py", wd / "out.glb", wd / "exec.json"
+ script.write_text(code if code.endswith("\n") else code + "\n")
+ for stale in (glb, rep_path):
+ if stale.exists():
+ stale.unlink()
+ cmd = [str(blender), "-b", "--factory-startup", "-noaudio", "--python", str(RUNNER), "--",
+ "--script", str(script), "--out", str(glb), "--report", str(rep_path)]
+ t0 = time.time()
+ env = config.tool_env(HOME="/tmp", XDG_CONFIG_HOME="/tmp/.bcfg", SEED=os.environ.get("SEED", "0"))
+ try:
+ p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=timeout,
+ text=True, errors="replace", env=env, cwd=str(wd))
+ if rep_path.exists():
+ rep = json.loads(rep_path.read_text())
+ else:
+ rep = {"status": "CRASH", "error": p.stdout[-1500:]}
+ except subprocess.TimeoutExpired:
+ rep = {"status": "TIMEOUT", "error": f"timeout {timeout}s"}
+ rep["latency_s"] = round(time.time() - t0, 1)
+ rep["mesh"] = str(glb) if rep.get("status") == "OK" and glb.exists() else None
+ if rep.get("status") == "OK" and rep["mesh"] is None:
+ rep["status"] = "EMPTY"
+ return rep
diff --git a/finetune/3dcodeverse_eval/blender_runner.py b/eval/llm/executors/blender_runner.py
similarity index 100%
rename from finetune/3dcodeverse_eval/blender_runner.py
rename to eval/llm/executors/blender_runner.py
diff --git a/eval/llm/executors/cadquery.py b/eval/llm/executors/cadquery.py
new file mode 100644
index 00000000..facdeb35
--- /dev/null
+++ b/eval/llm/executors/cadquery.py
@@ -0,0 +1,83 @@
+"""CadQuery executor: runs the script in a subprocess interpreter that has cadquery, exports STL.
+
+Result discovery (ported from finetune/eval/dialect_runners.py): export calls are intercepted (scripts export
+STEP/STL to arbitrary paths), ``show_object``/``debug`` are injected, and the result object is the last
+captured export → a global named ``result/final/model/part/solid/shape/body/assembly/assy/obj/res`` → any
+Workplane/Shape/Assembly global. Assemblies are compounded; empty shapes are rejected.
+"""
+from __future__ import annotations
+
+import json
+import os
+import re
+import subprocess
+import time
+from pathlib import Path
+
+from .. import config
+
+CQ_RUNNER = r'''
+import sys, json, runpy, traceback, os, importlib
+os.environ.setdefault("OMP_NUM_THREADS", "1")
+script, out_stl = sys.argv[1], sys.argv[2]
+rep = {"status": "FAIL", "error": None}
+try:
+ import cadquery as cq
+ sys.argv = [script, out_stl]
+ _orig_export = cq.exporters.export
+ captured = []
+ def _capture(o, *a, **k): captured.append(o)
+ for modname in ("cadquery.occ_impl.exporters", "cadquery.exporters"):
+ try:
+ _m = importlib.import_module(modname); _m.export = _capture
+ except Exception: pass
+ try: cq.exporters.export = _capture
+ except Exception: pass
+ g = runpy.run_path(script, run_name="__main__", init_globals={"show_object": _capture, "debug": _capture})
+ obj = None
+ for c in reversed(captured):
+ if isinstance(c, (cq.Workplane, cq.Shape, cq.Assembly)): obj = c; break
+ if obj is None:
+ for name in ("result", "final", "model", "part", "solid", "shape", "body", "assembly", "assy", "obj", "res"):
+ if isinstance(g.get(name), (cq.Workplane, cq.Shape, cq.Assembly)): obj = g[name]; break
+ if obj is None:
+ cands = [v for v in g.values() if isinstance(v, (cq.Workplane, cq.Shape, cq.Assembly))]
+ obj = cands[-1] if cands else None
+ if obj is None: raise RuntimeError("no CadQuery result object found (expected variable `result`)")
+ if isinstance(obj, cq.Assembly): obj = obj.toCompound()
+ if isinstance(obj, cq.Workplane):
+ vals = obj.vals()
+ if not vals: raise RuntimeError("empty workplane")
+ obj = vals[0] if len(vals) == 1 else cq.Compound.makeCompound([v for v in vals if isinstance(v, cq.Shape)])
+ if not isinstance(obj, cq.Shape): raise RuntimeError(f"unsupported object {type(obj)}")
+ if obj.Volume() <= 1e-9 and not obj.Faces(): raise RuntimeError("empty shape")
+ _orig_export(obj, out_stl, tolerance=0.01, angularTolerance=0.1)
+ rep["status"] = "OK"
+except BaseException as e:
+ rep["error"] = (type(e).__name__ + ": " + str(e))[:500]; rep["traceback"] = traceback.format_exc()[-1500:]
+print("CQ_REPORT " + json.dumps(rep))
+'''
+
+
+def run(code: str, workdir: str, timeout: int = 90) -> dict:
+ wd = Path(workdir).resolve()
+ wd.mkdir(parents=True, exist_ok=True)
+ script, stl, runner = wd / "code.py", wd / "out.stl", wd / "_runner.py"
+ script.write_text(code)
+ runner.write_text(CQ_RUNNER)
+ if stl.exists():
+ stl.unlink()
+ t0 = time.time()
+ try:
+ p = subprocess.run([config.cadquery_python(), str(runner), str(script), str(stl)], cwd=str(wd),
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=timeout, text=True, errors="replace",
+ env={**os.environ, "OMP_NUM_THREADS": "1"})
+ m = re.search(r"CQ_REPORT (\{.*\})", p.stdout)
+ rep = json.loads(m.group(1)) if m else {"status": "CRASH", "error": p.stdout[-800:]}
+ except subprocess.TimeoutExpired:
+ rep = {"status": "TIMEOUT", "error": f"timeout {timeout}s"}
+ rep["latency_s"] = round(time.time() - t0, 1)
+ rep["mesh"] = str(stl) if rep["status"] == "OK" and stl.exists() and stl.stat().st_size > 100 else None
+ if rep["status"] == "OK" and rep["mesh"] is None:
+ rep["status"] = "EMPTY"
+ return rep
diff --git a/eval/llm/executors/glsl.py b/eval/llm/executors/glsl.py
new file mode 100644
index 00000000..a08b2b29
--- /dev/null
+++ b/eval/llm/executors/glsl.py
@@ -0,0 +1,158 @@
+"""GLSL (Shadertoy-style) executor: compile with glslang, then optionally render in headless Chromium.
+
+Compile: the model's ``mainImage`` body is wrapped in a Shadertoy prelude (GLSL ES 3.00 uniforms incl.
+``iFrameRate`` — the old eval prelude lacked it and failed shaders that used it) and checked with
+``glslang -S frag``.
+
+Render (``render=True``, default): a WebGL2 page draws two frames (iTime = 0 and 1.7 s) and measures
+distinct quantised colours + pixel std. ``render`` ∈ {OK, STATIC, FAIL}; a shader that compiles but paints
+a constant colour is STATIC. The suite-level ``status`` stays "compile OK" for comparability with the
+historical numbers; ``render`` is reported alongside as the stricter signal.
+"""
+from __future__ import annotations
+
+import json
+import re
+import subprocess
+import time
+from pathlib import Path
+
+from .. import config
+
+PRELUDE = """#version 300 es
+precision highp float; precision highp int;
+uniform vec3 iResolution; uniform float iTime; uniform float iTimeDelta; uniform int iFrame; uniform float iFrameRate;
+uniform vec4 iMouse; uniform vec4 iDate; uniform float iSampleRate;
+uniform sampler2D iChannel0; uniform sampler2D iChannel1; uniform sampler2D iChannel2; uniform sampler2D iChannel3;
+uniform vec3 iChannelResolution[4]; uniform float iChannelTime[4];
+out vec4 _outColor;
+"""
+MAIN = "\nvoid main(){ vec4 c = vec4(0.0); mainImage(c, gl_FragCoord.xy); _outColor = c; }\n"
+
+_UNIFORM_RE = re.compile(r"\s*uniform\s+.*\b(iResolution|iTime|iTimeDelta|iFrame|iFrameRate|iMouse|iDate|iSampleRate|iChannel[0-3]|iChannelResolution|iChannelTime)\b")
+
+
+def normalise(code: str) -> str:
+ """Drop #version / precision / Shadertoy-uniform / `out vec4` lines that the prelude already provides."""
+ out = []
+ for ln in code.splitlines():
+ if re.match(r"\s*#\s*version\b", ln) or re.match(r"\s*precision\s+(low|medium|high)p\s+", ln):
+ continue
+ if _UNIFORM_RE.match(ln) or re.match(r"\s*out\s+vec4\s+\w+\s*;", ln):
+ continue
+ out.append(ln)
+ return "\n".join(out)
+
+
+HTML = """<!doctype html><html><body style="margin:0"><canvas id=c width=320 height=240></canvas><script>
+const FS_BODY = %s;
+const vs = `#version 300 es\nin vec2 p; void main(){ gl_Position = vec4(p,0.,1.); }`;
+const fsHead = %s;
+const fsTail = `\nvoid main(){ mainImage(_outColor, gl_FragCoord.xy); }`;
+window.__result = {ok:false, err:null};
+(function(){
+ const c = document.getElementById('c');
+ const gl = c.getContext('webgl2', {preserveDrawingBuffer:true});
+ if (!gl) { window.__result.err = 'no webgl2'; return; }
+ function sh(t, src){ const s = gl.createShader(t); gl.shaderSource(s, src); gl.compileShader(s);
+ if (!gl.getShaderParameter(s, gl.COMPILE_STATUS)) throw new Error(gl.getShaderInfoLog(s).split('\\n')[0]); return s; }
+ try {
+ const p = gl.createProgram();
+ gl.attachShader(p, sh(gl.VERTEX_SHADER, vs));
+ gl.attachShader(p, sh(gl.FRAGMENT_SHADER, fsHead + FS_BODY + fsTail));
+ gl.linkProgram(p);
+ if (!gl.getProgramParameter(p, gl.LINK_STATUS)) throw new Error(gl.getProgramInfoLog(p).split('\\n')[0]);
+ gl.useProgram(p);
+ const buf = gl.createBuffer(); gl.bindBuffer(gl.ARRAY_BUFFER, buf);
+ gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1,-1, 3,-1, -1,3]), gl.STATIC_DRAW);
+ const loc = gl.getAttribLocation(p, 'p'); gl.enableVertexAttribArray(loc); gl.vertexAttribPointer(loc, 2, gl.FLOAT, false, 0, 0);
+ const tex = gl.createTexture(); gl.bindTexture(gl.TEXTURE_2D, tex);
+ gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, new Uint8Array([128,128,128,255]));
+ for (let i=0;i<4;i++){ const u = gl.getUniformLocation(p, 'iChannel'+i); if (u) gl.uniform1i(u, 0); }
+ window.__draw = function(t){
+ gl.uniform3f(gl.getUniformLocation(p,'iResolution'), c.width, c.height, 1);
+ gl.uniform1f(gl.getUniformLocation(p,'iTime'), t);
+ gl.uniform1f(gl.getUniformLocation(p,'iTimeDelta'), 1/60);
+ gl.uniform1i(gl.getUniformLocation(p,'iFrame'), Math.floor(t*60));
+ gl.uniform1f(gl.getUniformLocation(p,'iFrameRate'), 60);
+ gl.viewport(0,0,c.width,c.height); gl.drawArrays(gl.TRIANGLES, 0, 3);
+ const px = new Uint8Array(c.width*c.height*4);
+ gl.readPixels(0,0,c.width,c.height, gl.RGBA, gl.UNSIGNED_BYTE, px);
+ const seen = new Set(); let sum=0, sum2=0, n=0;
+ for (let i=0;i<px.length;i+=4*7){ const r=px[i]>>4, g=px[i+1]>>4, b=px[i+2]>>4;
+ seen.add((r<<8)|(g<<4)|b); const v=(px[i]+px[i+1]+px[i+2])/3; sum+=v; sum2+=v*v; n++; }
+ const mean=sum/n; return {distinct: seen.size, std: Math.sqrt(Math.max(0, sum2/n - mean*mean)), mean};
+ };
+ window.__result.ok = true;
+ } catch(e) { window.__result.err = String(e.message || e); }
+})();
+</script></body></html>"""
+
+
+def compile_check(src: str, workdir: Path, timeout: int) -> dict:
+ exe = config.find_glslang()
+ if exe is None:
+ return {"status": "CRASH", "error": "glslang not found (set CV3D_GLSLANG)"}
+ fp = workdir / "shader.frag"
+ fp.write_text(PRELUDE + src + MAIN)
+ try:
+ p = subprocess.run([str(exe), "-S", "frag", str(fp)], stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+ timeout=timeout, text=True, errors="replace")
+ return {"status": "OK" if p.returncode == 0 else "FAIL", "error": None if p.returncode == 0 else p.stdout[-600:]}
+ except subprocess.TimeoutExpired:
+ return {"status": "TIMEOUT", "error": "glslang timeout"}
+
+
+def render_check(src: str, workdir: Path, timeout: int) -> dict:
+ """WebGL2 render in headless Chromium → {render: OK|STATIC|FAIL, distinct, std, changed, shot, render_error}."""
+ rep = {"render": "FAIL", "render_error": None, "distinct": 0, "std": 0.0, "changed": False, "shot": None}
+ page_path = workdir / "index.html"
+ page_path.write_text(HTML % (json.dumps(src), json.dumps(PRELUDE.replace("_outColor", "_outColor"))))
+ try:
+ from playwright.sync_api import sync_playwright
+ with sync_playwright() as pw:
+ b = pw.chromium.launch(args=["--use-gl=angle", "--use-angle=swiftshader", "--enable-unsafe-swiftshader", "--no-sandbox"])
+ pg = b.new_page()
+ errs: list[str] = []
+ pg.on("pageerror", lambda e: errs.append(str(e)[:200]))
+ pg.goto("file://" + str(page_path), wait_until="load", timeout=timeout * 1000)
+ res = pg.evaluate("window.__result")
+ if not res or not res.get("ok"):
+ rep["render_error"] = (res or {}).get("err") or (errs[0] if errs else "no result")
+ b.close()
+ return rep
+ a = pg.evaluate("window.__draw(0.0)")
+ c = pg.evaluate("window.__draw(1.7)")
+ try:
+ shot = workdir / "frame.png"
+ pg.screenshot(path=str(shot))
+ rep["shot"] = str(shot)
+ except Exception: # noqa: BLE001
+ pass
+ b.close()
+ rep["distinct"] = max(a["distinct"], c["distinct"])
+ rep["std"] = round(max(a["std"], c["std"]), 2)
+ rep["changed"] = abs(a["mean"] - c["mean"]) > 0.5 or a["distinct"] != c["distinct"]
+ rep["render"] = "OK" if (rep["distinct"] > 3 and rep["std"] > 2.0) else "STATIC"
+ if rep["render"] == "STATIC":
+ rep["render_error"] = f"uniform image (distinct={rep['distinct']}, std={rep['std']})"
+ except Exception as e: # noqa: BLE001
+ rep["render_error"] = str(e)[:200]
+ return rep
+
+
+def run(code: str, workdir: str, timeout: int = 60, render: bool = True) -> dict:
+ wd = Path(workdir).resolve()
+ wd.mkdir(parents=True, exist_ok=True)
+ (wd / "code.frag").write_text(code)
+ t0 = time.time()
+ src = normalise(code)
+ if "mainImage" not in src:
+ rep = {"status": "FAIL", "error": "no mainImage()"}
+ else:
+ rep = compile_check(src, wd, timeout)
+ if rep["status"] == "OK" and render:
+ rep.update(render_check(src, wd, timeout))
+ rep["latency_s"] = round(time.time() - t0, 2)
+ rep["mesh"] = None
+ return rep
diff --git a/eval/llm/executors/openscad.py b/eval/llm/executors/openscad.py
new file mode 100644
index 00000000..3d799e6e
--- /dev/null
+++ b/eval/llm/executors/openscad.py
@@ -0,0 +1,33 @@
+"""OpenSCAD executor: ``openscad -o out.stl --export-format binstl model.scad`` (needs a 2025+ build)."""
+from __future__ import annotations
+
+import subprocess
+import time
+from pathlib import Path
+
+from .. import config
+
+
+def run(code: str, workdir: str, timeout: int = 120) -> dict:
+ exe = config.find_openscad()
+ if exe is None:
+ return {"status": "CRASH", "error": "openscad not found (set CV3D_OPENSCAD)", "latency_s": 0.0, "mesh": None}
+ wd = Path(workdir).resolve()
+ wd.mkdir(parents=True, exist_ok=True)
+ scad, stl = wd / "code.scad", wd / "out.stl"
+ scad.write_text(code)
+ if stl.exists():
+ stl.unlink()
+ t0 = time.time()
+ try:
+ p = subprocess.run([str(exe), "-o", str(stl), "--export-format", "binstl", str(scad)], cwd=str(wd),
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=timeout, text=True, errors="replace",
+ env=config.tool_env(HOME=str(wd), QT_QPA_PLATFORM="offscreen"))
+ ok = p.returncode == 0 and stl.exists() and stl.stat().st_size > 100
+ rep = {"status": "OK" if ok else ("EMPTY" if p.returncode == 0 else "FAIL"),
+ "error": None if ok else p.stdout[-800:]}
+ except subprocess.TimeoutExpired:
+ rep = {"status": "TIMEOUT", "error": f"timeout {timeout}s"}
+ rep["latency_s"] = round(time.time() - t0, 1)
+ rep["mesh"] = str(stl) if rep["status"] == "OK" else None
+ return rep
diff --git a/eval/llm/executors/threejs.py b/eval/llm/executors/threejs.py
new file mode 100644
index 00000000..bda3106e
--- /dev/null
+++ b/eval/llm/executors/threejs.py
@@ -0,0 +1,91 @@
+"""three.js executor: load a single-file HTML page in headless Chromium (Playwright), screenshot, decide.
+
+status OK = page loaded, no uncaught JS error / failed module import / failed request, a <canvas> exists and
+ the composited screenshot is not (nearly) uniform.
+ EMPTY = no canvas or blank canvas. FAIL = JS/import error. TIMEOUT = page load exceeded the limit.
+
+Pages usually import three.js from a CDN via an importmap, so the browser needs network access. Set
+``CV3D_THREE_LOCAL=/path/to/three/build`` (a directory holding ``three.module.js`` and ``jsm/``) to rewrite
+``https://cdn.jsdelivr.net/npm/three@X/build/three.module.js`` / ``…/examples/jsm/`` (and the unpkg /
+esm.sh equivalents) to that local copy and run offline.
+"""
+from __future__ import annotations
+
+import os
+import re
+import time
+from pathlib import Path
+
+_CDN_RE = re.compile(r"https://(?:cdn\.jsdelivr\.net/npm|unpkg\.com|esm\.sh)/three@[^/\"']+/(build/three\.module\.js|examples/jsm/)")
+
+
+def localise(html: str) -> str:
+ local = os.environ.get("CV3D_THREE_LOCAL")
+ if not local:
+ return html
+ root = Path(local).resolve()
+
+ def sub(m: re.Match) -> str:
+ tail = m.group(1)
+ return f"file://{root}/three.module.js" if tail.startswith("build/") else f"file://{root}/jsm/"
+
+ return _CDN_RE.sub(sub, html)
+
+
+def run(code: str, workdir: str, timeout: int = 60, settle_ms: int = 4000) -> dict:
+ wd = Path(workdir).resolve()
+ wd.mkdir(parents=True, exist_ok=True)
+ page_path = wd / "index.html"
+ page_path.write_text(localise(code))
+ rep: dict = {"status": "FAIL", "error": None, "console_errors": [], "canvas_nonblank": None, "latency_s": 0.0,
+ "shot": None, "mesh": None}
+ t0 = time.time()
+ try:
+ from playwright.sync_api import TimeoutError as PWTimeout
+ from playwright.sync_api import sync_playwright
+ with sync_playwright() as p:
+ browser = p.chromium.launch(headless=True, args=["--use-gl=angle", "--use-angle=swiftshader", "--enable-unsafe-swiftshader",
+ "--ignore-gpu-blocklist", "--no-sandbox", "--disable-dev-shm-usage",
+ "--allow-file-access-from-files"])
+ page = browser.new_page(viewport={"width": 640, "height": 480})
+ errs: list[str] = []
+ page.on("pageerror", lambda e: errs.append("pageerror: " + str(e)[:300]))
+ page.on("console", lambda m: errs.append("console.error: " + m.text[:300]) if m.type == "error" else None)
+ page.on("requestfailed", lambda r: errs.append("requestfailed: " + r.url[:200]))
+ try:
+ page.goto("file://" + str(page_path), wait_until="load", timeout=timeout * 1000)
+ except PWTimeout:
+ rep.update(status="TIMEOUT", error=f"page load > {timeout}s")
+ browser.close()
+ rep["latency_s"] = round(time.time() - t0, 1)
+ return rep
+ page.wait_for_timeout(settle_ms)
+ n_canvas = page.evaluate("document.querySelectorAll('canvas').length")
+ shot = wd / "shot.png"
+ page.screenshot(path=str(shot))
+ rep["shot"] = str(shot)
+ browser.close()
+ rep["console_errors"] = errs[:10]
+ fatal = [e for e in errs if e.startswith("pageerror") or "Failed to resolve module" in e or "Uncaught" in e
+ or e.startswith("requestfailed")]
+ from PIL import Image
+ import numpy as np
+ im = np.array(Image.open(shot).convert("RGB"))
+ q = im[::6, ::6].reshape(-1, 3) // 16
+ distinct = len(set(map(tuple, q)))
+ rep["shot_distinct"], rep["shot_std"] = distinct, float(im.std())
+ nonblank = distinct > 6 and float(im.std()) > 4.0
+ rep["canvas_nonblank"] = bool(n_canvas) and nonblank
+ if fatal:
+ rep.update(status="FAIL", error=fatal[0])
+ elif not n_canvas:
+ rep.update(status="EMPTY", error="no <canvas>")
+ elif not nonblank:
+ rep.update(status="EMPTY", error=f"blank canvas (distinct={distinct}, std={im.std():.1f})")
+ else:
+ rep["status"] = "OK"
+ except Exception as e: # noqa: BLE001
+ rep["error"] = (type(e).__name__ + ": " + str(e))[:400]
+ rep["status"] = "CRASH"
+ rep["latency_s"] = round(time.time() - t0, 1)
+ return rep
diff --git a/finetune/3dcodeverse_eval/extract.py b/eval/llm/extract.py
similarity index 96%
rename from finetune/3dcodeverse_eval/extract.py
rename to eval/llm/extract.py
index 55eaa783..514a2444 100644
--- a/finetune/3dcodeverse_eval/extract.py
+++ b/eval/llm/extract.py
@@ -1,5 +1,8 @@
"""Dialect-aware code extraction from free-form model output.
+Copied from finetune/eval/extract.py (2026-09-08) with one fix: the shell-prompt penalty no longer
+fires on a leading `# comment` line. Behaviour is otherwise identical, so numbers stay comparable.
+
Why this exists: instruct/reasoning models do not answer with bare code. A single response may contain
a <think> block (often with *draft* code), prose, several fenced blocks (setup shell commands, a JSON
snippet, the real program, then a "here's a variant" block), or no fence at all. Taking "the longest
@@ -61,7 +64,7 @@ DIALECTS = {
# blocks that are never the answer
NEGATIVE = [
(re.compile(r"^\s*(pip|conda|apt|sudo|blender|python)\s+(install|-m|-b)\b", re.M), 4),
- (re.compile(r"^\s*[$#>]\s+\w"), 3), # shell prompt
+ (re.compile(r"^\s*[$>]\s+\w"), 3), # shell prompt ("#" excluded: it is a comment in every dialect we score)
(re.compile(r"^\s*\{[\s\S]*\}\s*$"), 2), # bare JSON
(re.compile(r"^\s*(Traceback|Error:|error:)", re.M), 3), # pasted error log
]
diff --git a/eval/llm/generate.py b/eval/llm/generate.py
new file mode 100644
index 00000000..44c828c7
--- /dev/null
+++ b/eval/llm/generate.py
@@ -0,0 +1,205 @@
+"""Generation: turn prompt rows into <gen_dir>/<id>/{code.<ext>, raw.txt} + gens.jsonl, one backend at a time.
+
+Backends
+ vllm in-process vLLM engine (local open models; text and image prompts; one load for many suites)
+ openai any OpenAI-compatible chat endpoint: a vLLM server (`vllm serve …`), OpenRouter, GPT, or a proxy —
+ set --base-url / OPENAI_BASE_URL and OPENAI_API_KEY (dummy key ok for local servers)
+ anthropic Claude models through the Anthropic SDK (ANTHROPIC_API_KEY)
+ reference writes the suite's reference program as the "answer" — the executor/metric self-test
+
+Every generated answer goes through `extract.extract(text, dialect)` (dialect-aware fenced-block selection,
+think-block stripping) and the extraction metadata is kept per row, so format failures are separable from
+capability failures in the report.
+
+gens.jsonl row: {id, n_new_tokens, finished, truncated, has_code_block, latency_s, extract{…}, error?}
+"""
+from __future__ import annotations
+
+import base64
+import json
+import mimetypes
+import os
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+
+from . import config
+from .executors import CODE_EXT
+from .extract import extract, summarize
+from .suites import resolve
+
+
+def _data_url(path: Path) -> str:
+ """Images as JPEG data-URLs (thumbnail 768 px, quality 88) — identical to finetune/eval/generate_vllm_img.py."""
+ import io
+ from PIL import Image
+
+ im = Image.open(path).convert("RGB")
+ im.thumbnail((768, 768))
+ buf = io.BytesIO()
+ im.save(buf, format="JPEG", quality=88)
+ return "data:image/jpeg;base64," + base64.b64encode(buf.getvalue()).decode()
+
+
+def to_openai_messages(row: dict) -> list[dict]:
+ """Chat messages in OpenAI content-parts form; `<image>` placeholders become image parts placed first."""
+ msgs = []
+ images = [resolve(p) for p in row.get("images", [])]
+ for m in row["messages"]:
+ if m["role"] == "user" and images:
+ text = m["content"].replace("<image>", "").strip()
+ parts = [{"type": "image_url", "image_url": {"url": _data_url(p)}} for p in images]
+ parts.append({"type": "text", "text": text})
+ msgs.append({"role": "user", "content": parts})
+ else:
+ msgs.append({"role": m["role"], "content": m["content"]})
+ return msgs
+
+
+def write_answer(gen_dir: Path, row: dict, text: str, meta: dict) -> dict:
+ d = gen_dir / row["id"]
+ d.mkdir(parents=True, exist_ok=True)
+ code, xmeta = extract(text or "", row["dialect"])
+ (d / f"code.{CODE_EXT[row['dialect']]}").write_text(code + "\n")
+ (d / "raw.txt").write_text(text or "")
+ rec = {"id": row["id"], **meta, "has_code_block": "```" in (text or ""), "extract": xmeta}
+ return rec
+
+
+def finish(gen_dir: Path, recs: list[dict], info: dict) -> None:
+ recs.sort(key=lambda r: r["id"])
+ with (gen_dir / "gens.jsonl").open("w") as f:
+ for r in recs:
+ f.write(json.dumps(r) + "\n")
+ stats = summarize([r["extract"] for r in recs])
+ n = max(1, len(recs))
+ stats.update(info, n_rows=len(recs), truncated=sum(bool(r.get("truncated")) for r in recs),
+ errors=sum(bool(r.get("error")) for r in recs),
+ mean_new_tokens=round(sum(r.get("n_new_tokens") or 0 for r in recs) / n, 1))
+ (gen_dir / "gen_stats.json").write_text(json.dumps(stats, indent=2))
+ print(f"[gen] {gen_dir.name}: {len(recs)} rows | truncated {stats['truncated']} | no_fence {stats['no_fence']} "
+ f"| multi_block {stats['multi_block']} | mean_new_tokens {stats['mean_new_tokens']}", flush=True)
+
+
+# ----------------------------------------------------------------------------------------------------------- vLLM
+class VllmEngine:
+ def __init__(self, model: str, tp: int = 1, max_model_len: int = 16384, gpu_mem: float = 0.90,
+ max_images: int = 4, max_num_seqs: int | None = None, dtype: str = "bfloat16"):
+ # WSL2: vLLM ≥0.28 refuses to start without pinned memory ("UVA is not available") unless this is set
+ os.environ.setdefault("VLLM_WSL2_ENABLE_PIN_MEMORY", "1")
+ # no CUDA toolkit on this box: FlashInfer's JIT sampler needs nvcc; the torch sampler does not
+ os.environ.setdefault("VLLM_USE_FLASHINFER_SAMPLER", "0")
+ from vllm import LLM
+
+ self.model = model
+ kw = {"max_num_seqs": max_num_seqs} if max_num_seqs else {}
+ self.llm = LLM(model=model, dtype=dtype, tensor_parallel_size=tp, max_model_len=max_model_len,
+ gpu_memory_utilization=gpu_mem, enable_prefix_caching=True, trust_remote_code=True,
+ limit_mm_per_prompt={"image": max_images}, **kw)
+
+ def generate(self, rows: list[dict], gen_dir: Path, temperature: float, max_new_tokens: int, seed: int = 0,
+ no_think: bool = True, top_p: float = 0.95) -> None:
+ from vllm import SamplingParams
+
+ gen_dir.mkdir(parents=True, exist_ok=True)
+ sp = SamplingParams(temperature=temperature, top_p=top_p if temperature > 0 else 1.0, max_tokens=max_new_tokens, seed=seed)
+ kw = {"chat_template_kwargs": {"enable_thinking": False}} if no_think else {}
+ msgs = [to_openai_messages(r) for r in rows]
+ t0 = time.time()
+ outs = self.llm.chat(msgs, sp, use_tqdm=True, **kw)
+ el = time.time() - t0
+ recs, ntok = [], 0
+ for r, o in zip(rows, outs):
+ out = o.outputs[0]
+ ntok += len(out.token_ids)
+ recs.append(write_answer(gen_dir, r, out.text, {"n_new_tokens": len(out.token_ids),
+ "finished": out.finish_reason == "stop",
+ "truncated": out.finish_reason == "length"}))
+ finish(gen_dir, recs, {"backend": "vllm", "model": self.model, "temperature": temperature, "seed": seed,
+ "max_new_tokens": max_new_tokens, "no_think": no_think, "tok_per_s": round(ntok / max(el, 1e-6), 1),
+ "wall_s": round(el, 1)})
+
+
+# ----------------------------------------------------------------------------------------------------- API backends
+def _openai_call(client, model: str, row: dict, temperature: float, max_new_tokens: int, no_think: bool,
+ extra_body: dict | None) -> dict:
+ t0 = time.time()
+ body = dict(extra_body or {})
+ if no_think:
+ body.setdefault("chat_template_kwargs", {"enable_thinking": False})
+ resp = client.chat.completions.create(model=model, messages=to_openai_messages(row), temperature=temperature,
+ max_tokens=max_new_tokens, extra_body=body or None)
+ ch = resp.choices[0]
+ text = ch.message.content or ""
+ usage = resp.usage
+ return {"text": text, "n_new_tokens": getattr(usage, "completion_tokens", None) if usage else None,
+ "finished": ch.finish_reason == "stop", "truncated": ch.finish_reason == "length",
+ "latency_s": round(time.time() - t0, 1)}
+
+
+def _anthropic_call(client, model: str, row: dict, temperature: float, max_new_tokens: int) -> dict:
+ t0 = time.time()
+ system = "\n".join(m["content"] for m in row["messages"] if m["role"] == "system")
+ images = [resolve(p) for p in row.get("images", [])]
+ content = []
+ for p in images:
+ mt = mimetypes.guess_type(str(p))[0] or "image/png"
+ content.append({"type": "image", "source": {"type": "base64", "media_type": mt,
+ "data": base64.b64encode(p.read_bytes()).decode()}})
+ user = "\n".join(m["content"] for m in row["messages"] if m["role"] == "user").replace("<image>", "").strip()
+ content.append({"type": "text", "text": user})
+ resp = client.messages.create(model=model, system=system or None, max_tokens=max_new_tokens,
+ temperature=temperature, messages=[{"role": "user", "content": content}])
+ text = "".join(b.text for b in resp.content if getattr(b, "type", "") == "text")
+ return {"text": text, "n_new_tokens": resp.usage.output_tokens, "finished": resp.stop_reason == "end_turn",
+ "truncated": resp.stop_reason == "max_tokens", "latency_s": round(time.time() - t0, 1)}
+
+
+def api_generate(backend: str, model: str, rows: list[dict], gen_dir: Path, temperature: float, max_new_tokens: int,
+ workers: int = 8, base_url: str | None = None, no_think: bool = True, extra_body: dict | None = None,
+ resume: bool = True) -> None:
+ gen_dir.mkdir(parents=True, exist_ok=True)
+ done: dict[str, dict] = {}
+ if resume and (gen_dir / "gens.jsonl").exists():
+ for l in (gen_dir / "gens.jsonl").open():
+ r = json.loads(l)
+ if not r.get("error"):
+ done[r["id"]] = r
+ todo = [r for r in rows if r["id"] not in done]
+ if backend == "openai":
+ from openai import OpenAI
+ client = OpenAI(base_url=base_url or os.environ.get("OPENAI_BASE_URL"),
+ api_key=os.environ.get("OPENAI_API_KEY", "sk-local"), timeout=600, max_retries=2)
+ call = lambda r: _openai_call(client, model, r, temperature, max_new_tokens, no_think, extra_body) # noqa: E731
+ elif backend == "anthropic":
+ import anthropic
+ client = anthropic.Anthropic(timeout=600, max_retries=2)
+ call = lambda r: _anthropic_call(client, model, r, temperature, max_new_tokens) # noqa: E731
+ else:
+ raise ValueError(backend)
+ recs = list(done.values())
+ t0 = time.time()
+ with ThreadPoolExecutor(workers) as ex:
+ futs = {ex.submit(call, r): r for r in todo}
+ for i, fu in enumerate(as_completed(futs), 1):
+ r = futs[fu]
+ try:
+ res = fu.result()
+ recs.append(write_answer(gen_dir, r, res.pop("text"), res))
+ except Exception as e: # noqa: BLE001
+ recs.append(write_answer(gen_dir, r, "", {"error": f"{type(e).__name__}: {str(e)[:300]}"}))
+ if i % 20 == 0:
+ print(f"[gen] {gen_dir.name} {i}/{len(todo)}", flush=True)
+ finish(gen_dir, recs, {"backend": backend, "model": model, "temperature": temperature, "max_new_tokens": max_new_tokens,
+ "no_think": no_think, "wall_s": round(time.time() - t0, 1), "resumed": len(done)})
+
+
+def reference_generate(rows: list[dict], gen_dir: Path) -> None:
+ gen_dir.mkdir(parents=True, exist_ok=True)
+ recs = []
+ for r in rows:
+ code = r["reference"].get("code") or ""
+ fence = {"blender": "python", "cadquery": "python", "openscad": "openscad", "glsl": "glsl", "threejs": "html"}[r["dialect"]]
+ text = code if code.lstrip().startswith("```") else f"```{fence}\n{code}\n```"
+ recs.append(write_answer(gen_dir, r, text, {"n_new_tokens": 0, "finished": True, "truncated": False}))
+ finish(gen_dir, recs, {"backend": "reference", "model": "reference", "temperature": 0.0, "max_new_tokens": 0})
diff --git a/eval/llm/image_sim.py b/eval/llm/image_sim.py
new file mode 100644
index 00000000..5ee175a9
--- /dev/null
+++ b/eval/llm/image_sim.py
@@ -0,0 +1,154 @@
+"""Render-based similarity between generated and reference views (the official 3DCodeBench image metric).
+
+For each task: 4 rendered views of the generated mesh vs the 4 reference views (same turntable azimuths),
+embedded with SigLIP-2 (google/siglip2-so400m-patch16-naflex, image features) or DINOv3
+(facebook/dinov3-vitl16-pretrain-lvd1689m, pooler output) — the two encoders whose scores track the
+3DCodeBench human arena (Pearson 0.964 / Spearman 0.972). Two aggregates over the 4×4 cosine matrix:
+
+ view_paired mean of the diagonal (same azimuth) — penalises a 90/180° yaw mismatch
+ best_assignment Hungarian 1-to-1 matching — immune to view permutations
+
+and two averages: conditional (tasks with all 4 renders) and penalized (missing renders = 0, over all tasks).
+A text→image score (prompt vs rendered views, SigLIP-2 only) is available for text suites.
+
+Weights are loaded from the HF cache (local_files_only fallback); needs torch + transformers + a GPU.
+"""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import numpy as np
+
+ENCODERS = {
+ "siglip2": {"model_id": "google/siglip2-so400m-patch16-naflex", "kind": "siglip"},
+ "siglip2-base": {"model_id": "google/siglip2-base-patch16-224", "kind": "siglip"},
+ "dinov3": {"model_id": "facebook/dinov3-vitl16-pretrain-lvd1689m", "kind": "pooler"},
+ "dinov2": {"model_id": "facebook/dinov2-base", "kind": "pooler"},
+}
+VIEWS = ["Image_005.png", "Image_015.png", "Image_025.png", "Image_035.png"]
+
+
+class Encoder:
+ def __init__(self, name: str = "siglip2", device: str = "cuda"):
+ import torch
+ from transformers import AutoImageProcessor, AutoModel, AutoProcessor
+
+ spec = ENCODERS[name]
+ self.name, self.kind, self.device = name, spec["kind"], device
+
+ def load(cls, mid):
+ try:
+ return cls.from_pretrained(mid)
+ except OSError:
+ return cls.from_pretrained(mid, local_files_only=True)
+
+ self.processor = load(AutoProcessor if self.kind == "siglip" else AutoImageProcessor, spec["model_id"])
+ self.model = load(AutoModel, spec["model_id"]).to(device).eval()
+ self.torch = torch
+
+ def embed_images(self, paths: list[Path], batch_size: int = 16) -> np.ndarray:
+ from PIL import Image
+
+ out = []
+ for i in range(0, len(paths), batch_size):
+ imgs = [_flatten(Image.open(p)) for p in paths[i:i + batch_size]]
+ inputs = self.processor(images=imgs, return_tensors="pt").to(self.device)
+ with self.torch.no_grad():
+ if self.kind == "siglip":
+ f = self.model.get_image_features(**inputs)
+ f = f if isinstance(f, self.torch.Tensor) else f.pooler_output
+ else:
+ f = self.model(**inputs).pooler_output
+ f = f / f.norm(dim=-1, keepdim=True)
+ out.append(f.float().cpu().numpy())
+ return np.concatenate(out, 0) if out else np.zeros((0, 1))
+
+ def embed_texts(self, texts: list[str]) -> np.ndarray:
+ assert self.kind == "siglip", "text embeddings need SigLIP"
+ inputs = self.processor(text=texts, return_tensors="pt", padding="max_length", truncation=True, max_length=64).to(self.device)
+ with self.torch.no_grad():
+ f = self.model.get_text_features(**inputs)
+ f = f if isinstance(f, self.torch.Tensor) else f.pooler_output
+ f = f / f.norm(dim=-1, keepdim=True)
+ return f.float().cpu().numpy()
+
+
+def _flatten(im):
+ """RGBA renders on a transparent film → composite on the dark world colour so encoders see what a viewer sees."""
+ from PIL import Image
+
+ if im.mode == "RGBA":
+ bg = Image.new("RGBA", im.size, (3, 3, 5, 255))
+ im = Image.alpha_composite(bg, im)
+ return im.convert("RGB")
+
+
+def pair_scores(gen: list[np.ndarray | None], ref: list[np.ndarray | None]) -> dict:
+ from scipy.optimize import linear_sum_assignment
+
+ m = [[float(gen[i] @ ref[j]) if gen[i] is not None and ref[j] is not None else None for j in range(4)] for i in range(4)]
+ diag = [m[v][v] for v in range(4) if m[v][v] is not None]
+ vi = [i for i in range(4) if gen[i] is not None]
+ vj = [j for j in range(4) if ref[j] is not None]
+ assigned = None
+ if vi and vj:
+ sub = np.array([[m[i][j] for j in vj] for i in vi])
+ r, c = linear_sum_assignment(-sub)
+ assigned = float(sub[r, c].mean())
+ return {"n_paired": len(diag), "view_paired": float(np.mean(diag)) if diag else None, "best_assignment": assigned, "cos_matrix": m}
+
+
+def score_gen_dir(gen_dir: Path, rows: dict[str, dict], resolve, encoder_name: str = "siglip2", device: str = "cuda",
+ text_sim: bool = False) -> dict:
+ """rows: {id: prompt row}; writes <gen_dir>/image_sim_<enc>.jsonl and returns the aggregate."""
+ enc = Encoder(encoder_name, device)
+ paths, index = [], []
+ per = {}
+ for tid, row in rows.items():
+ refs = [resolve(p) for p in row["reference"].get("renders", [])]
+ ref_by = {p.name: p for p in refs}
+ gen_dir_views = gen_dir / tid / "exec" / "renders"
+ per[tid] = {"gen": [None] * 4, "ref": [None] * 4}
+ for k, v in enumerate(VIEWS):
+ g = gen_dir_views / v
+ if g.exists():
+ index.append((tid, "gen", k)); paths.append(g)
+ if v in ref_by and ref_by[v].exists():
+ index.append((tid, "ref", k)); paths.append(ref_by[v])
+ if not paths:
+ return {"encoder": encoder_name, "n": len(rows), "error": "no renders"}
+ embs = enc.embed_images(paths)
+ for (tid, kind, k), e in zip(index, embs):
+ per[tid][kind][k] = e
+ text_emb = None
+ if text_sim and enc.kind == "siglip":
+ prompts = [next((m["content"] for m in rows[t]["messages"] if m["role"] == "user"), "") for t in rows]
+ text_emb = dict(zip(rows, enc.embed_texts([p.replace("<image>", "").strip()[:800] for p in prompts])))
+ recs = []
+ for tid in rows:
+ s = pair_scores(per[tid]["gen"], per[tid]["ref"])
+ rec = {"id": tid, "n_gen_views": sum(g is not None for g in per[tid]["gen"]),
+ "n_ref_views": sum(r is not None for r in per[tid]["ref"]), **{k: s[k] for k in ("n_paired", "view_paired", "best_assignment")}}
+ if text_emb is not None:
+ gv = [g for g in per[tid]["gen"] if g is not None]
+ rec["text_image"] = float(np.mean([g @ text_emb[tid] for g in gv])) if gv else None
+ recs.append(rec)
+ with (gen_dir / f"image_sim_{encoder_name}.jsonl").open("w") as f:
+ for r in recs:
+ f.write(json.dumps(r) + "\n")
+ n = len(recs)
+ full = [r for r in recs if r["n_paired"] == 4]
+
+ def agg(field):
+ cond = float(np.mean([r[field] for r in full])) if full else None
+ pen = float(sum((r[field] or 0.0) for r in recs) / n) if n else None
+ return {"conditional": cond, "penalized": pen}
+
+ summ = {"encoder": encoder_name, "model_id": ENCODERS[encoder_name]["model_id"], "n": n, "n_full": len(full),
+ "view_paired": agg("view_paired"), "best_assignment": agg("best_assignment")}
+ if text_emb is not None:
+ with_t = [r for r in recs if r.get("text_image") is not None]
+ summ["text_image"] = {"conditional": float(np.mean([r["text_image"] for r in with_t])) if with_t else None,
+ "penalized": float(sum((r.get("text_image") or 0.0) for r in recs) / n) if n else None}
+ return summ
diff --git a/eval/llm/judge.py b/eval/llm/judge.py
new file mode 100644
index 00000000..cdd52ba4
--- /dev/null
+++ b/eval/llm/judge.py
@@ -0,0 +1,284 @@
+"""VLM judge for generated objects — the quality layer for prompts that have no ground truth.
+
+Two modes, both judged from the 4 canonical renders (render_views.py) of the generated mesh:
+
+ absolute one object → JSON scores 1–5 on identity / structure / detail (+ overall) against the prompt
+ (and the reference views, if the suite has them). Cheap, but absolute LLM grades are noisy and
+ drift between judge models — use them for triage, not for close comparisons.
+ pairwise two runs on the same prompt → winner a / b / tie / both_bad, judged in BOTH orders; a winner
+ counts only when the two orders agree (position bias is the dominant failure of VLM judges —
+ MT-Bench 65 % consistency, GPT-4V 51 %). The pairwise rubric is the official 3DCodeBench arena
+ prompt verbatim (metrics/llm_judge/prompts/image_judge.txt).
+
+Backends: any OpenAI-compatible chat endpoint with image input (a local vLLM server running Qwen3.5-9B /
+Qwen3-VL, OpenRouter, GPT) via --base-url, or Claude via --backend anthropic. Results go to
+<gen_dir>/judge_<tag>.jsonl (absolute) or <out>/pairwise_<a>_vs_<b>.jsonl.
+
+ python -m llm.judge absolute --run qwen3_8b --suite 3dcodebench_text --judge-model gpt-5.4 --base-url …
+ python -m llm.judge pairwise --run-a qwen3_8b --run-b official_gpt-5.5 --suite 3dcodebench_official_text …
+"""
+from __future__ import annotations
+
+import argparse
+import base64
+import json
+import os
+import random
+import re
+import time
+from collections import Counter
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+
+from . import config
+from .render_views import VIEWS
+from .suites import load_prompts, resolve
+
+PAIRWISE_SYSTEM = (config.PKG_DIR / "data" / "official_prompts" / "image_judge.txt").read_text() \
+ if (config.PKG_DIR / "data" / "official_prompts" / "image_judge.txt").exists() else ""
+
+ABSOLUTE_SYSTEM = """You are an impartial expert judge of 3D models. An AI system was given a text prompt (and, for
+image-to-3D tasks, reference images) and asked to produce a 3D object as code. You see four renders of the
+resulting object from canonical viewpoints (azimuths 45/135/225/315 degrees). Judge ONLY from the renders.
+
+Score each criterion on an integer scale 1-5 (1 = fails completely, 3 = partially right, 5 = fully convincing):
+ identity is it recognizably the requested object at all?
+ structure are the major parts present, in the right place, in plausible proportion, physically connected
+ (no floating or interpenetrating pieces)?
+ detail shape fidelity and surface plausibility of the parts that exist; ornament and repeating features
+ the prompt asks for are real geometry, not flat stand-ins.
+ reference (image-to-3D only, else null) faithfulness to the reference images: silhouette, proportions,
+ distinctive features. Colour / material is NOT a criterion; renders may be untextured grey.
+overall your holistic 1-5 grade of "is this a good answer to the prompt", not an average.
+
+Do NOT reward polygon count or visual flair when the object is wrong; a clean simple correct shape beats a busy
+unrecognizable mess. Output STRICTLY one JSON object on one line, no markdown fences, no extra text:
+{"identity": int, "structure": int, "detail": int, "reference": int|null, "overall": int, "reasoning": "<one to three sentences>"}"""
+
+
+def _img_part(path: Path) -> dict:
+ b64 = base64.b64encode(path.read_bytes()).decode()
+ mt = "image/png" if path.suffix.lower() == ".png" else "image/jpeg"
+ return {"type": "image_url", "image_url": {"url": f"data:{mt};base64,{b64}"}}
+
+
+def _renders(gen_dir: Path, tid: str) -> list[Path]:
+ d = gen_dir / tid / "exec" / "renders"
+ return [d / v for v in VIEWS if (d / v).exists()]
+
+
+def _parse_json(text: str) -> dict | None:
+ cleaned = re.sub(r"^```(?:json)?\s*|\s*```$", "", text.strip(), flags=re.S)
+ m = re.search(r"\{.*\}", cleaned, re.S)
+ if not m:
+ return None
+ try:
+ return json.loads(m.group(0))
+ except json.JSONDecodeError:
+ return None
+
+
+class JudgeClient:
+ def __init__(self, backend: str, model: str, base_url: str | None = None, temperature: float = 0.0, max_tokens: int = 600):
+ self.backend, self.model, self.temperature, self.max_tokens = backend, model, temperature, max_tokens
+ if backend == "openai":
+ from openai import OpenAI
+ self.client = OpenAI(base_url=base_url or os.environ.get("OPENAI_BASE_URL"),
+ api_key=os.environ.get("OPENAI_API_KEY", "sk-local"), timeout=300, max_retries=2)
+ elif backend == "anthropic":
+ import anthropic
+ self.client = anthropic.Anthropic(timeout=300, max_retries=2)
+ else:
+ raise ValueError(backend)
+
+ def ask(self, system: str, parts: list[dict]) -> str:
+ if self.backend == "openai":
+ r = self.client.chat.completions.create(model=self.model, temperature=self.temperature, max_tokens=self.max_tokens,
+ messages=[{"role": "system", "content": system}, {"role": "user", "content": parts}])
+ return r.choices[0].message.content or ""
+ content = []
+ for p in parts:
+ if p["type"] == "text":
+ content.append({"type": "text", "text": p["text"]})
+ else:
+ url = p["image_url"]["url"]
+ mt, b64 = url.split(";base64,")
+ content.append({"type": "image", "source": {"type": "base64", "media_type": mt[5:], "data": b64}})
+ r = self.client.messages.create(model=self.model, system=system, max_tokens=self.max_tokens, temperature=self.temperature,
+ messages=[{"role": "user", "content": content}])
+ return "".join(b.text for b in r.content if getattr(b, "type", "") == "text")
+
+
+def _prompt_text(row: dict) -> str:
+ return next((m["content"] for m in row["messages"] if m["role"] == "user"), "").replace("<image>", "").strip()
+
+
+def _ref_parts(row: dict, max_refs: int = 4) -> list[dict]:
+ refs = [resolve(p) for p in row.get("images", [])][:max_refs]
+ if not refs:
+ return []
+ return [{"type": "text", "text": f"Reference image(s) the system was asked to reconstruct ({len(refs)}):"}] + [_img_part(p) for p in refs]
+
+
+# ------------------------------------------------------------------------------------------------------ absolute
+def judge_absolute_one(jc: JudgeClient, row: dict, gen_dir: Path) -> dict:
+ tid = row["id"]
+ rends = _renders(gen_dir, tid)
+ if len(rends) < 4:
+ return {"id": tid, "status": "NO_RENDERS", "n_renders": len(rends)}
+ parts = [{"type": "text", "text": "PROMPT given to the system:\n" + _prompt_text(row)}] + _ref_parts(row) + \
+ [{"type": "text", "text": "Four renders of the system's object (azimuths 45, 135, 225, 315 degrees):"}] + \
+ [_img_part(p) for p in rends] + [{"type": "text", "text": "Return the JSON verdict now."}]
+ t0 = time.time()
+ try:
+ raw = jc.ask(ABSOLUTE_SYSTEM, parts)
+ except Exception as e: # noqa: BLE001
+ return {"id": tid, "status": "ERROR", "error": f"{type(e).__name__}: {str(e)[:200]}"}
+ v = _parse_json(raw)
+ if not v or "overall" not in v:
+ return {"id": tid, "status": "PARSE_FAIL", "raw": raw[:500]}
+ return {"id": tid, "status": "OK", **{k: v.get(k) for k in ("identity", "structure", "detail", "reference", "overall")},
+ "reasoning": str(v.get("reasoning", ""))[:400], "latency_s": round(time.time() - t0, 1)}
+
+
+def judge_absolute(run: str, suite: str, jc: JudgeClient, tag: str, workers: int = 4, limit: int | None = None,
+ out_root: Path = config.OUT_DIR, resume: bool = True) -> dict:
+ gen_dir = out_root / run / suite
+ rows = load_prompts(suite, limit)
+ out_path = gen_dir / f"judge_{tag}.jsonl"
+ done = {}
+ if resume and out_path.exists():
+ done = {json.loads(l)["id"]: json.loads(l) for l in out_path.open() if l.strip()}
+ done = {k: v for k, v in done.items() if v.get("status") == "OK"}
+ todo = [r for r in rows if r["id"] not in done]
+ recs = list(done.values())
+ with ThreadPoolExecutor(workers) as ex:
+ for i, fu in enumerate(as_completed({ex.submit(judge_absolute_one, jc, r, gen_dir): r for r in todo}), 1):
+ recs.append(fu.result())
+ if i % 20 == 0:
+ print(f"[judge] {run}/{suite} {i}/{len(todo)}", flush=True)
+ recs.sort(key=lambda r: r["id"])
+ with out_path.open("w") as f:
+ for r in recs:
+ f.write(json.dumps(r, ensure_ascii=False) + "\n")
+ ok = [r for r in recs if r["status"] == "OK"]
+ n = len(rows)
+
+ def mean(k, cond):
+ vals = [r[k] for r in ok if isinstance(r.get(k), (int, float))]
+ if not vals:
+ return None
+ return sum(vals) / len(vals) if cond else sum(vals) / n # penalized: missing render / failure = 0
+
+ summ = {"judge": f"{jc.backend}:{jc.model}", "tag": tag, "n": n, "n_judged": len(ok),
+ "statuses": dict(Counter(r["status"] for r in recs)),
+ **{f"{k}_cond": mean(k, True) for k in ("identity", "structure", "detail", "reference", "overall")},
+ **{f"{k}_pen": mean(k, False) for k in ("identity", "structure", "detail", "overall")}}
+ (gen_dir / f"judge_{tag}_summary.json").write_text(json.dumps(summ, indent=2))
+ print(f"[judge] {run}/{suite}: judged {len(ok)}/{n} | overall cond {summ['overall_cond']} pen {summ['overall_pen']} "
+ f"| identity {summ['identity_cond']} structure {summ['structure_cond']} detail {summ['detail_cond']}", flush=True)
+ return summ
+
+
+# ------------------------------------------------------------------------------------------------------ pairwise
+def _pair_parts(row: dict, ra: list[Path], rb: list[Path]) -> list[dict]:
+ parts = [{"type": "text", "text": "PROMPT given to both systems:\n" + _prompt_text(row)}] + _ref_parts(row)
+ parts.append({"type": "text", "text": "Four renders of System A's 3D object:"})
+ parts += [_img_part(p) for p in ra]
+ parts.append({"type": "text", "text": "Four renders of System B's 3D object:"})
+ parts += [_img_part(p) for p in rb]
+ parts.append({"type": "text", "text": "Return the JSON verdict now."})
+ return parts
+
+
+def judge_pair_one(jc: JudgeClient, row: dict, dir_a: Path, dir_b: Path) -> dict:
+ tid = row["id"]
+ ra, rb = _renders(dir_a, tid), _renders(dir_b, tid)
+ rec = {"id": tid, "a_rendered": len(ra) == 4, "b_rendered": len(rb) == 4}
+ if len(ra) < 4 and len(rb) < 4:
+ return {**rec, "winner": "both_bad", "status": "NO_RENDERS"}
+ if len(ra) < 4:
+ return {**rec, "winner": "b", "status": "A_NO_RENDER"}
+ if len(rb) < 4:
+ return {**rec, "winner": "a", "status": "B_NO_RENDER"}
+ verdicts = []
+ for swap in (False, True):
+ x, y = (rb, ra) if swap else (ra, rb)
+ try:
+ raw = jc.ask(PAIRWISE_SYSTEM, _pair_parts(row, x, y))
+ except Exception as e: # noqa: BLE001
+ return {**rec, "winner": None, "status": "ERROR", "error": f"{type(e).__name__}: {str(e)[:200]}"}
+ v = _parse_json(raw) or {}
+ w = v.get("winner")
+ if w in ("a", "b") and swap:
+ w = "b" if w == "a" else "a"
+ verdicts.append({"swap": swap, "winner": w, "reasoning": str(v.get("reasoning", ""))[:300]})
+ w0, w1 = verdicts[0]["winner"], verdicts[1]["winner"]
+ if w0 is None or w1 is None:
+ final, status = None, "PARSE_FAIL"
+ elif w0 == w1:
+ final, status = w0, "OK"
+ else:
+ final, status = "tie", "DISAGREE" # order-dependent verdict → not a winner
+ return {**rec, "winner": final, "status": status, "verdicts": verdicts}
+
+
+def judge_pairwise(run_a: str, run_b: str, suite: str, jc: JudgeClient, tag: str, workers: int = 4, limit: int | None = None,
+ out_root: Path = config.OUT_DIR) -> dict:
+ dir_a, dir_b = out_root / run_a / suite, out_root / run_b / suite
+ rows = load_prompts(suite, limit)
+ recs = []
+ with ThreadPoolExecutor(workers) as ex:
+ for i, fu in enumerate(as_completed({ex.submit(judge_pair_one, jc, r, dir_a, dir_b): r for r in rows}), 1):
+ recs.append(fu.result())
+ if i % 20 == 0:
+ print(f"[judge] pairwise {i}/{len(rows)}", flush=True)
+ recs.sort(key=lambda r: r["id"])
+ out_dir = out_root / "_pairwise"
+ out_dir.mkdir(exist_ok=True)
+ out_path = out_dir / f"{suite}__{run_a}__vs__{run_b}__{tag}.jsonl"
+ with out_path.open("w") as f:
+ for r in recs:
+ f.write(json.dumps(r, ensure_ascii=False) + "\n")
+ c = Counter(r["winner"] for r in recs)
+ decided = c.get("a", 0) + c.get("b", 0)
+ summ = {"suite": suite, "run_a": run_a, "run_b": run_b, "judge": f"{jc.backend}:{jc.model}", "n": len(recs),
+ "a_wins": c.get("a", 0), "b_wins": c.get("b", 0), "ties": c.get("tie", 0), "both_bad": c.get("both_bad", 0),
+ "order_disagreements": sum(r.get("status") == "DISAGREE" for r in recs),
+ "win_rate_a": (c.get("a", 0) + 0.5 * c.get("tie", 0)) / len(recs) if recs else None,
+ "win_rate_a_decided": c.get("a", 0) / decided if decided else None}
+ out_path.with_suffix(".summary.json").write_text(json.dumps(summ, indent=2))
+ print(f"[judge] {suite} {run_a} vs {run_b}: A {summ['a_wins']} / B {summ['b_wins']} / tie {summ['ties']} / both_bad {summ['both_bad']} "
+ f"(order disagreements {summ['order_disagreements']}) → win-rate A {summ['win_rate_a']:.3f}", flush=True)
+ return summ
+
+
+def main() -> None:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("mode", choices=["absolute", "pairwise"])
+ ap.add_argument("--run", help="absolute: run name")
+ ap.add_argument("--run-a")
+ ap.add_argument("--run-b")
+ ap.add_argument("--suite", required=True)
+ ap.add_argument("--backend", choices=["openai", "anthropic"], default="openai")
+ ap.add_argument("--judge-model", required=True)
+ ap.add_argument("--base-url", default=None)
+ ap.add_argument("--tag", default=None, help="label for the output files (default: judge model)")
+ ap.add_argument("--workers", type=int, default=4)
+ ap.add_argument("--limit", type=int, default=None)
+ ap.add_argument("--out", type=Path, default=config.OUT_DIR)
+ a = ap.parse_args()
+ jc = JudgeClient(a.backend, a.judge_model, a.base_url)
+ tag = a.tag or re.sub(r"[^A-Za-z0-9._-]+", "_", a.judge_model)
+ if a.mode == "absolute":
+ if not a.run:
+ raise SystemExit("--run is required for absolute mode")
+ judge_absolute(a.run, a.suite, jc, tag, a.workers, a.limit, a.out)
+ else:
+ if not (a.run_a and a.run_b):
+ raise SystemExit("--run-a and --run-b are required for pairwise mode")
+ judge_pairwise(a.run_a, a.run_b, a.suite, jc, tag, a.workers, a.limit, a.out)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/eval/llm/metrics.py b/eval/llm/metrics.py
new file mode 100644
index 00000000..af2350b2
--- /dev/null
+++ b/eval/llm/metrics.py
@@ -0,0 +1,187 @@
+"""Metric primitives. Definitions are documented in docs/metrics.md; keep the two in sync.
+
+Geometry (generated mesh vs reference mesh, both normalised to a unit bounding box):
+ chamfer symmetric Chamfer-L2 distance = mean_a min_b ||a-b|| + mean_b min_a ||a-b|| (sum of the two
+ directional means — the convention used throughout finetune/docs/REPORT.md; halve it for the
+ "average" convention used by some CAD papers)
+ f@tau F-score at threshold tau: precision = frac of generated points within tau of the reference,
+ recall = frac of reference points within tau of the generated mesh, F = 2PR/(P+R)
+ iou_vox volumetric IoU on a 64^3 voxel grid of the normalised bounding box (solid voxelisation; needs
+ reasonably closed meshes — reported as None when either voxelisation is empty)
+ rotation search: the generated point cloud is rotated by k*90 deg (k = 0..3) about the up axis and the
+ rotation with the smallest Chamfer is kept (objects may face any of the four cardinal directions)
+
+Aggregates:
+ exec_rate OK / n
+ f@tau_mean_scored mean over executed tasks; f@tau_mean_all fail counted as 0 (the headline number)
+ pass@k unbiased estimator of Chen et al. 2021 given n samples with c successes per task
+"""
+from __future__ import annotations
+
+import math
+from pathlib import Path
+
+import numpy as np
+
+UP_AXIS = {"glb": 1, "stl": 2} # glTF is Y-up; CadQuery/OpenSCAD STL are Z-up
+
+
+def load_points(path: str | Path, n: int = 10000, seed: int = 0) -> np.ndarray | None:
+ import trimesh
+
+ m = trimesh.load(str(path), force="mesh")
+ if isinstance(m, trimesh.Scene):
+ geoms = list(m.geometry.values())
+ if not geoms:
+ return None
+ m = trimesh.util.concatenate(geoms)
+ if m.is_empty or len(m.faces) == 0:
+ return None
+ m = normalise(m)
+ if m is None:
+ return None
+ pts, _ = trimesh.sample.sample_surface(m, n, seed=seed)
+ return np.asarray(pts)
+
+
+def normalise(m):
+ """Centre at the bbox centre, scale so the largest bbox extent is 1 (in place; returns None if degenerate)."""
+ v = m.vertices
+ c = (v.max(0) + v.min(0)) / 2
+ s = (v.max(0) - v.min(0)).max()
+ if not np.isfinite(s) or s <= 0:
+ return None
+ m.apply_translation(-c)
+ m.apply_scale(1.0 / s)
+ return m
+
+
+def rot_about(p: np.ndarray, axis: int, k: int) -> np.ndarray:
+ th = np.pi / 2 * k
+ c, s = np.cos(th), np.sin(th)
+ if axis == 1: # Y-up
+ R = np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])
+ elif axis == 2: # Z-up
+ R = np.array([[c, -s, 0], [s, c, 0], [0, 0, 1]])
+ else:
+ R = np.array([[1, 0, 0], [0, c, -s], [0, s, c]])
+ return p @ R.T
+
+
+def chamfer_f(a: np.ndarray, b: np.ndarray, taus=(0.05, 0.1)) -> tuple[float, dict[str, float]]:
+ from scipy.spatial import cKDTree
+
+ ta, tb = cKDTree(a), cKDTree(b)
+ da, _ = tb.query(a)
+ db, _ = ta.query(b)
+ cd = float(da.mean() + db.mean())
+ fs = {}
+ for t in taus:
+ p = float((da < t).mean())
+ r = float((db < t).mean())
+ fs[f"f@{t}"] = 2 * p * r / (p + r) if p + r > 0 else 0.0
+ return cd, fs
+
+
+def voxel_iou(gen_path: str | Path, ref_path: str | Path, pitch_div: int = 64) -> float | None:
+ """Solid-voxel IoU in the unit cube, or None when a mesh cannot be voxelised."""
+ import trimesh
+
+ def solid(path):
+ m = trimesh.load(str(path), force="mesh")
+ if isinstance(m, trimesh.Scene):
+ m = trimesh.util.concatenate(list(m.geometry.values()))
+ if m.is_empty or len(m.faces) == 0 or normalise(m) is None:
+ return None
+ vox = m.voxelized(pitch=1.0 / pitch_div).fill()
+ return vox
+
+ try:
+ a, b = solid(gen_path), solid(ref_path)
+ if a is None or b is None:
+ return None
+ # align both grids on the unit cube centred at the origin
+ grid = np.zeros((pitch_div + 2,) * 3, dtype=bool)
+
+ def paint(vox):
+ g = grid.copy()
+ idx = np.round((vox.points + 0.5) * pitch_div).astype(int)
+ idx = idx[(idx >= 0).all(1) & (idx < pitch_div + 2).all(1)]
+ g[idx[:, 0], idx[:, 1], idx[:, 2]] = True
+ return g
+
+ ga, gb = paint(a), paint(b)
+ inter, union = np.logical_and(ga, gb).sum(), np.logical_or(ga, gb).sum()
+ return float(inter / union) if union else None
+ except Exception: # noqa: BLE001
+ return None
+
+
+def geometry_metrics(gen_path: str | Path, ref_path: str | Path, n_points: int = 10000, up_axis: int | None = None,
+ with_iou: bool = True) -> dict:
+ """Chamfer / [email protected] / [email protected] (best of 4 yaw rotations) + voxel IoU between two mesh files."""
+ if up_axis is None:
+ up_axis = UP_AXIS.get(Path(ref_path).suffix.lower().lstrip("."), 1)
+ pg, pr = load_points(gen_path, n_points), load_points(ref_path, n_points)
+ if pg is None or pr is None:
+ return {"metric_error": "empty mesh"}
+ best = None
+ for k in range(4):
+ cd, fs = chamfer_f(rot_about(pg, up_axis, k), pr)
+ if best is None or cd < best[0]:
+ best = (cd, fs, k)
+ out = {"chamfer": best[0], **best[1], "rot_k": best[2]}
+ if with_iou:
+ out["iou_vox"] = voxel_iou(gen_path, ref_path)
+ return out
+
+
+def chamfer_official(gen_path: str | Path, ref_path: str | Path, n_points: int = 8192, seed: int = 0) -> dict:
+ """The official 3DCodeBench Chamfer (metrics/shape_chamfer.py): centroid-centred, unit-SPHERE normalised,
+ n_points surface samples, CD = mean min ||.||^2 + mean min ||.||^2 (squared), min over 4 yaw rotations
+ about Z of the generated cloud. Reported separately from `chamfer` (unit-bbox, unsquared) so the numbers
+ are comparable to the paper's frontier-model table."""
+ import trimesh
+ from scipy.spatial import cKDTree
+
+ rng = np.random.default_rng(seed)
+
+ def pts(path):
+ m = trimesh.load(str(path), force="mesh")
+ if isinstance(m, trimesh.Scene):
+ m = trimesh.util.concatenate(list(m.geometry.values()))
+ if m.is_empty or len(m.faces) == 0:
+ return None
+ p, _ = trimesh.sample.sample_surface(m, n_points, seed=int(rng.integers(0, 2**31 - 1)))
+ p = np.asarray(p, dtype=np.float64)
+ p = p - p.mean(0, keepdims=True)
+ r = np.linalg.norm(p, axis=1).max()
+ return p / r if r > 1e-9 else p
+
+ def cd2(a, b):
+ da, _ = cKDTree(b).query(a)
+ db, _ = cKDTree(a).query(b)
+ return float((da ** 2).mean() + (db ** 2).mean())
+
+ ref, gen = pts(ref_path), pts(gen_path)
+ if ref is None or gen is None:
+ return {"cd_official": None, "cd_official_yawmin": None}
+ per_yaw = [cd2(ref, rot_about(gen, 2, k)) for k in range(4)]
+ return {"cd_official": per_yaw[0], "cd_official_yawmin": float(min(per_yaw))}
+
+
+def pass_at_k(n: int, c: int, k: int) -> float:
+ """Unbiased pass@k (Chen et al. 2021): 1 - C(n-c, k) / C(n, k)."""
+ if n - c < k:
+ return 1.0
+ return 1.0 - math.prod((n - c - i) / (n - i) for i in range(k))
+
+
+def mean(xs) -> float | None:
+ xs = [x for x in xs if x is not None and not (isinstance(x, float) and math.isnan(x))]
+ return float(np.mean(xs)) if xs else None
+
+
+def median(xs) -> float | None:
+ xs = [x for x in xs if x is not None]
+ return float(np.median(xs)) if xs else None
diff --git a/eval/llm/render_views.py b/eval/llm/render_views.py
new file mode 100644
index 00000000..c4ded924
--- /dev/null
+++ b/eval/llm/render_views.py
@@ -0,0 +1,170 @@
+"""Render a GLB from the 4 canonical 3DCodeBench views (Image_005/015/025/035 = azimuth 45/135/225/315 deg).
+
+Camera / light / material setup is the official one (3dcodebench core/render.py): the object is centred at
+the origin, cam(θ) = (1.8·extent·cos θ, 1.8·extent·sin θ, 0.6·extent), 50 mm lens, three area lights, a
+neutral grey material on every mesh, dark world, transparent film, Cycles (GPU if available) at 512 px.
+The official renderer re-executes the generated *script*; we render the exported GLB instead (identical
+geometry; the script's own materials survive the GLB round trip, and like the official rig a neutral grey is
+applied only to meshes without one — `keep_materials=False` renders everything as grey clay).
+
+ render_glb_views(glb, out_dir) -> {status, n_views, latency_s, error}
+ render_dir(gen_dir) -> renders <gen_dir>/<id>/exec/renders/Image_0xx.png for every OK task
+"""
+from __future__ import annotations
+
+import json
+import subprocess
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+
+from . import config
+
+VIEWS = ["Image_005.png", "Image_015.png", "Image_025.png", "Image_035.png"]
+
+BLENDER_SCRIPT = r'''
+import sys, os, json, math, time, traceback
+import bpy
+from mathutils import Vector
+argv = sys.argv[sys.argv.index("--") + 1:]
+glb, out_dir, samples, res, engine = argv[0], argv[1], int(argv[2]), int(argv[3]), argv[4]
+keep_materials = len(argv) < 6 or argv[5] != "strip"
+REF_VIEWS = [(5, "Image_005.png"), (15, "Image_015.png"), (25, "Image_025.png"), (35, "Image_035.png")]
+rec = {"status": None, "n_views": 0, "latency_s": None, "error": None, "extent": None}
+t0 = time.time()
+try:
+ bpy.ops.object.select_all(action="SELECT"); bpy.ops.object.delete()
+ for col in (bpy.data.meshes, bpy.data.curves, bpy.data.cameras, bpy.data.lights, bpy.data.materials, bpy.data.node_groups):
+ for item in list(col): col.remove(item)
+ bpy.ops.import_scene.gltf(filepath=glb)
+ for o in list(bpy.context.scene.objects):
+ if o.type in ("CAMERA", "LIGHT"): bpy.data.objects.remove(o, do_unlink=True)
+ meshes = [o for o in bpy.context.scene.objects if o.type == "MESH"]
+ if not meshes: raise RuntimeError("no mesh in GLB")
+ bpy.context.view_layer.update()
+ corners = [o.matrix_world @ Vector(c) for o in meshes for c in o.bound_box]
+ xs, ys, zs = zip(*[(v.x, v.y, v.z) for v in corners])
+ cx, cy, cz = (min(xs)+max(xs))/2, (min(ys)+max(ys))/2, (min(zs)+max(zs))/2
+ extent = max(max(xs)-min(xs), max(ys)-min(ys), max(zs)-min(zs)) or 1.0
+ roots = [o for o in bpy.context.scene.objects if o.parent is None]
+ for o in roots: o.location = o.location - Vector((cx, cy, cz))
+ center = Vector((0, 0, 0)); cam_r, cam_h = extent * 1.8, extent * 0.6
+ w = bpy.context.scene.world or bpy.data.worlds.new("World"); bpy.context.scene.world = w; w.use_nodes = True
+ for n in list(w.node_tree.nodes): w.node_tree.nodes.remove(n)
+ bg = w.node_tree.nodes.new("ShaderNodeBackground"); out = w.node_tree.nodes.new("ShaderNodeOutputWorld")
+ bg.inputs[0].default_value = (0.012, 0.013, 0.021, 1); bg.inputs[1].default_value = 1.0
+ w.node_tree.links.new(bg.outputs[0], out.inputs[0])
+ def add_light(name, loc, energy, sz):
+ bpy.ops.object.light_add(type="AREA", location=loc); lt = bpy.context.object
+ lt.name, lt.data.energy, lt.data.size = name, energy, extent * sz
+ lt.rotation_euler = (center - Vector(loc)).to_track_quat("-Z", "Y").to_euler()
+ add_light("Key", (cam_r*0.9, -cam_r*0.7, cam_h*1.7), 1200, 0.9)
+ add_light("Fill", (-cam_r*0.6, -cam_r*0.4, cam_h), 400, 1.2)
+ add_light("Rim", (0, cam_r*0.9, cam_h*1.3), 600, 0.7)
+ mat = bpy.data.materials.new("eval_default"); mat.use_nodes = True
+ bsdf = mat.node_tree.nodes.get("Principled BSDF")
+ if bsdf:
+ bsdf.inputs["Base Color"].default_value = (0.72, 0.72, 0.75, 1); bsdf.inputs["Roughness"].default_value = 0.65
+ for o in meshes: # official rig: default material only where the script assigned none
+ if keep_materials and o.data.materials and any(m is not None for m in o.data.materials): continue
+ o.data.materials.clear(); o.data.materials.append(mat)
+ bpy.ops.object.camera_add(location=(cam_r, 0, cam_h)); cam = bpy.context.object
+ bpy.context.scene.camera = cam; cam.data.lens = 50; cam.data.clip_end = extent * 25
+ sc = bpy.context.scene
+ sc.render.resolution_x = sc.render.resolution_y = res
+ sc.render.image_settings.file_format = "PNG"; sc.render.image_settings.color_mode = "RGBA"
+ sc.render.film_transparent = True; sc.render.engine = engine
+ if engine == "CYCLES":
+ sc.cycles.device = "CPU"
+ try:
+ pr = bpy.context.preferences.addons["cycles"].preferences
+ for dev_type in ("OPTIX", "CUDA"):
+ try: pr.compute_device_type = dev_type
+ except TypeError: continue
+ if hasattr(pr, "refresh_devices"): pr.refresh_devices()
+ elif hasattr(pr, "get_devices"): pr.get_devices()
+ gpus = [d for d in pr.devices if d.type == dev_type]
+ if gpus:
+ for d in pr.devices: d.use = (d.type == dev_type)
+ sc.cycles.device = "GPU"; break
+ except Exception: sc.cycles.device = "CPU"
+ sc.cycles.samples = samples; sc.cycles.use_denoising = True
+ else:
+ for attr in ("taa_render_samples", "samples"):
+ try: setattr(sc.eevee, attr, samples); break
+ except AttributeError: pass
+ for frame_idx, fname in REF_VIEWS:
+ a = 2 * math.pi * frame_idx / 40
+ cam.location = (cam_r * math.cos(a), cam_r * math.sin(a), cam_h)
+ cam.rotation_euler = (center - cam.location).to_track_quat("-Z", "Y").to_euler()
+ sc.render.filepath = os.path.join(out_dir, fname); bpy.ops.render.render(write_still=True)
+ rec["n_views"] += 1
+ rec["extent"] = float(extent); rec["status"] = "OK" if rec["n_views"] == 4 else "ERR_RENDER"
+except Exception as e:
+ rec["status"] = "ERR_RENDER"; rec["error"] = f"{type(e).__name__}: {e}\n{traceback.format_exc()[-800:]}"
+rec["latency_s"] = round(time.time() - t0, 2)
+open(os.path.join(out_dir, "render_log.json"), "w").write(json.dumps(rec, indent=1))
+'''
+
+
+def render_glb_views(glb: str | Path, out_dir: str | Path, samples: int = 64, resolution: int = 512, engine: str = "CYCLES",
+ timeout: int = 240, overwrite: bool = False, keep_materials: bool = True) -> dict:
+ out_dir = Path(out_dir)
+ out_dir.mkdir(parents=True, exist_ok=True)
+ log = out_dir / "render_log.json"
+ if log.exists() and not overwrite and all((out_dir / v).exists() for v in VIEWS):
+ return json.loads(log.read_text()) | {"cached": True}
+ blender = config.find_blender()
+ if blender is None:
+ return {"status": "ERR_RENDER", "error": "blender not found", "n_views": 0, "latency_s": 0.0}
+ script = out_dir / "_render_views.py"
+ script.write_text(BLENDER_SCRIPT)
+ cmd = [str(blender), "-b", "--factory-startup", "-noaudio", "--python", str(script), "--",
+ str(Path(glb).resolve()), str(out_dir), str(samples), str(resolution), engine, "keep" if keep_materials else "strip"]
+ t0 = time.time()
+ try:
+ subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, env=config.tool_env(HOME="/tmp", XDG_CONFIG_HOME="/tmp/.bcfg"))
+ except subprocess.TimeoutExpired:
+ rec = {"status": "ERR_TIMEOUT", "error": f"render > {timeout}s", "n_views": 0, "latency_s": round(time.time() - t0, 1)}
+ log.write_text(json.dumps(rec))
+ return rec
+ if log.exists():
+ return json.loads(log.read_text())
+ rec = {"status": "ERR_NOLOG", "error": "blender exited without a log", "n_views": 0, "latency_s": round(time.time() - t0, 1)}
+ log.write_text(json.dumps(rec))
+ return rec
+
+
+def render_dir(gen_dir: Path, workers: int = 2, samples: int = 64, resolution: int = 512, engine: str = "CYCLES") -> dict[str, dict]:
+ """Render every task whose exec_results status is OK; returns {id: render_log}."""
+ gen_dir = Path(gen_dir)
+ ex = {json.loads(l)["id"]: json.loads(l) for l in (gen_dir / "exec_results.jsonl").open() if l.strip()}
+ jobs = [(tid, e["mesh"]) for tid, e in ex.items() if e.get("status") == "OK" and e.get("mesh") and str(e["mesh"]).endswith(".glb")]
+ print(f"[render] {gen_dir.name}: {len(jobs)} GLBs, {workers} workers, {engine} {samples}spp {resolution}px", flush=True)
+ out: dict[str, dict] = {}
+ with ThreadPoolExecutor(workers) as pool:
+ futs = {pool.submit(render_glb_views, mesh, gen_dir / tid / "exec" / "renders", samples, resolution, engine): tid for tid, mesh in jobs}
+ for i, fu in enumerate(as_completed(futs), 1):
+ out[futs[fu]] = fu.result()
+ if i % 25 == 0:
+ print(f"[render] {gen_dir.name} {i}/{len(jobs)}", flush=True)
+ ok = sum(r.get("status") == "OK" for r in out.values())
+ print(f"[render] {gen_dir.name}: {ok}/{len(jobs)} rendered", flush=True)
+ return out
+
+
+if __name__ == "__main__":
+ import argparse
+
+ ap = argparse.ArgumentParser()
+ ap.add_argument("target", help="a .glb file or a gen dir")
+ ap.add_argument("--out", default=None)
+ ap.add_argument("--workers", type=int, default=2)
+ ap.add_argument("--engine", default="CYCLES")
+ ap.add_argument("--samples", type=int, default=64)
+ a = ap.parse_args()
+ t = Path(a.target)
+ if t.is_file():
+ print(json.dumps(render_glb_views(t, a.out or t.parent / "renders", samples=a.samples, engine=a.engine), indent=1))
+ else:
+ render_dir(t, a.workers, a.samples, engine=a.engine)
diff --git a/eval/llm/report.py b/eval/llm/report.py
new file mode 100644
index 00000000..4891cd25
--- /dev/null
+++ b/eval/llm/report.py
@@ -0,0 +1,113 @@
+"""Collect summary.json files under an output root into one table (markdown + json).
+
+Layout expected: <out>/<run>/<suite>[/sN]/summary.json (run = model tag, sN = sample index for pass@k runs).
+pass@k / best-of-N are computed per (run, suite) from the metrics.jsonl of every sample directory.
+"""
+from __future__ import annotations
+
+import json
+from collections import defaultdict
+from pathlib import Path
+
+from . import metrics
+
+COLS = ["run", "suite", "n", "exec", "exec_rate", "f05_all", "f10_ok", "cd_ok", "iou_ok", "siglip2", "dinov3", "floating", "watertight", "judge",
+ "render_rate", "trunc", "tok", "no_fence", "multi_blk", "T"]
+
+
+def fmt(v, kind="f"):
+ if v is None:
+ return "–"
+ if kind == "pct":
+ return f"{100 * v:.1f}%"
+ if kind == "f":
+ return f"{v:.3f}"
+ return str(v)
+
+
+def collect(root: Path) -> list[dict]:
+ rows = []
+ for p in sorted(root.rglob("summary.json")):
+ s = json.loads(p.read_text())
+ rel = p.relative_to(root).parts # run / suite / [sN] / summary.json
+ s["run"], s["_dir"] = rel[0], str(p.parent)
+ s["sample"] = rel[2] if len(rel) == 4 else "s0"
+ for j in sorted(p.parent.glob("judge_*_summary.json")): # latest judge tag wins
+ js = json.loads(j.read_text())
+ s["judge_overall_cond"], s["judge_overall_pen"], s["judge"] = js.get("overall_cond"), js.get("overall_pen"), js.get("judge")
+ rows.append(s)
+ return rows
+
+
+def pass_at_k(rows: list[dict]) -> list[dict]:
+ """Group sample dirs per (run, suite) → [email protected], best-of-N [email protected](all)."""
+ groups: dict[tuple[str, str], list[dict]] = defaultdict(list)
+ for r in rows:
+ groups[(r["run"], r["suite"])].append(r)
+ out = []
+ for (run, suite), rs in groups.items():
+ if len(rs) < 2:
+ continue
+ per_task: dict[str, list[dict]] = defaultdict(list)
+ for r in rs:
+ for l in (Path(r["_dir"]) / "metrics.jsonl").open():
+ m = json.loads(l)
+ per_task[m["id"]].append(m)
+ n = len(rs)
+ ks = [k for k in (1, 2, 4, 8, n) if k <= n]
+ rec = {"run": run, "suite": suite, "n_samples": n, "n_tasks": len(per_task)}
+ for k in sorted(set(ks)):
+ rec[f"pass@{k}"] = metrics.mean([metrics.pass_at_k(len(v), sum(m["status"] == "OK" for m in v), k) for v in per_task.values()])
+ if any("[email protected]" in m for v in per_task.values() for m in v):
+ rec["best_of_n_f05_all"] = metrics.mean([max([m.get("[email protected]", 0.0) or 0.0 for m in v]) for v in per_task.values()])
+ out.append(rec)
+ return out
+
+
+def markdown(rows: list[dict]) -> str:
+ lines = ["| " + " | ".join(COLS) + " |", "|" + "---|" * len(COLS)]
+ for s in sorted(rows, key=lambda r: (r["run"], r["suite"], r["sample"])):
+ g, x = s.get("gen", {}), s.get("extract", {})
+ im, st = s.get("image_sim", {}), s.get("structural", {})
+ sig = (im.get("siglip2") or {}).get("view_paired", {}).get("conditional")
+ din = (im.get("dinov3") or {}).get("view_paired", {}).get("conditional")
+ vals = [s["run"], s["suite"] + ("" if s["sample"] == "s0" else f"/{s['sample']}"), s["n"], s["exec_ok"],
+ fmt(s["exec_rate"], "pct"), fmt(s.get("f05_mean_all")), fmt(s.get("f10_mean_scored")),
+ fmt(s.get("chamfer_mean_scored")), fmt(s.get("iou_mean_scored")), fmt(sig), fmt(din),
+ fmt(st.get("floating_part_rate_cond")), fmt(st.get("is_watertight_cond"), "pct"), fmt(s.get("judge_overall_cond")),
+ fmt(s.get("render_rate"), "pct"),
+ s.get("truncated"), fmt(s.get("mean_new_tokens"), "i") if s.get("mean_new_tokens") is None else int(s["mean_new_tokens"]),
+ x.get("no_fence"), x.get("multi_block"), g.get("temperature")]
+ lines.append("| " + " | ".join(str(v) for v in vals) + " |")
+ return "\n".join(lines)
+
+
+def build(root: Path, out_md: Path | None = None) -> str:
+ rows = collect(root)
+ pk = pass_at_k(rows)
+ md = ["# 3dcodeverse_eval results", f"root: `{root}`", "",
+ "exec = OK / n; f05_all = mean [email protected] with failures as 0 (headline geometry number); f10_ok / cd_ok / iou_ok = mean over "
+ "executed tasks; render_rate = GLSL compiled *and* painted a non-uniform image; trunc = hit max_new_tokens; "
+ "siglip2 / dinov3 = official view-paired similarity, conditional; floating = official floating-part rate (cond.); watertight = "
+ "share of executed meshes that are watertight; judge = VLM absolute overall 1-5 (cond.); no_fence / multi_blk = extraction diagnostics.", "", markdown(rows)]
+ if pk:
+ md += ["", "## pass@k over sampled runs", "", "| run | suite | samples | tasks | " + " | ".join(k for k in pk[0] if k.startswith("pass@")) + " | best-of-N [email protected] |",
+ "|---|---|---|---|" + "---|" * (len([k for k in pk[0] if k.startswith("pass@")]) + 1)]
+ for r in pk:
+ pks = " | ".join(fmt(r[k]) for k in r if k.startswith("pass@"))
+ md.append(f"| {r['run']} | {r['suite']} | {r['n_samples']} | {r['n_tasks']} | {pks} | {fmt(r.get('best_of_n_f05_all'))} |")
+ text = "\n".join(md) + "\n"
+ (out_md or root / "report.md").write_text(text)
+ (root / "report.json").write_text(json.dumps({"summaries": rows, "pass_at_k": pk}, indent=2))
+ return text
+
+
+if __name__ == "__main__":
+ import argparse
+ from . import config
+
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--root", type=Path, default=config.OUT_DIR)
+ ap.add_argument("--out", type=Path, default=None)
+ a = ap.parse_args()
+ print(build(a.root, a.out))
diff --git a/eval/llm/rescore_logs.py b/eval/llm/rescore_logs.py
new file mode 100644
index 00000000..6fc9a781
--- /dev/null
+++ b/eval/llm/rescore_logs.py
@@ -0,0 +1,64 @@
+"""Re-score the official 3DCodeBench frontier-model logs with this package's executor and metrics.
+
+`YipengGao/3DCode/3DCodeBench_ModelLogs/data/{text_to_3D,image_to_3D}.parquet` hold every generated script of
+the paper's 12 models (one row per trial, `code` column). Writing them into the gen-dir layout lets
+`run_eval --stages exec render score` produce exec / Chamfer / SigLIP-2 / DINOv3 numbers for the frontier
+models on *this* machine — the calibration that tells whether a local 8B number is comparable to the paper.
+
+ python -m llm.rescore_logs --setting text_to_3D --models gpt-5.5 claude-opus-4-7
+ python -m llm.run_eval --backend reference --run official_gpt-5.5 --suites 3dcodebench_official_text --stages exec render score
+
+Runs are named `official_<model>`; the suite is `3dcodebench_official_text` (or `_img4`).
+"""
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from . import config
+from .generate import finish, write_answer
+from .suites import load_prompts
+
+SUITE = {"text_to_3D": "3dcodebench_official_text", "image_to_3D": "3dcodebench_official_img4"}
+
+
+def main() -> None:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--setting", choices=list(SUITE), default="text_to_3D")
+ ap.add_argument("--models", nargs="*", default=None, help="model ids as in the parquet (default: all)")
+ ap.add_argument("--out", type=Path, default=config.OUT_DIR)
+ ap.add_argument("--list", action="store_true")
+ a = ap.parse_args()
+ import pandas as pd
+
+ p = config.HUB_DIR / "3DCode" / "3DCodeBench_ModelLogs" / "data" / f"{a.setting}.parquet"
+ if not p.exists():
+ raise SystemExit(f"{p} missing — run download.py --logs")
+ df = pd.read_parquet(p)
+ counts = df.groupby("model").size()
+ if a.list:
+ print(df.assign(ok=df.status.eq("OK")).groupby("model").agg(n=("ok", "size"), ok=("ok", "mean")).sort_values("ok"))
+ return
+ suite = SUITE[a.setting]
+ rows = {r["id"]: r for r in load_prompts(suite)}
+ for model in (a.models or sorted(counts.index)):
+ sub = df[df.model == model].drop_duplicates("instance")
+ gen_dir = a.out / f"official_{model}" / suite
+ recs = []
+ for _, r in sub.iterrows():
+ tid = r["instance"]
+ if tid not in rows:
+ continue
+ code = r["code"] if isinstance(r["code"], str) else ""
+ text = code if code.lstrip().startswith("```") or not code.strip() else f"```python\n{code}\n```"
+ recs.append(write_answer(gen_dir, rows[tid], text, {"n_new_tokens": int(r["output_tokens"]) if r.get("output_tokens") == r.get("output_tokens") else None,
+ "finished": r["status"] == "OK", "truncated": False,
+ "official_status": r["status"], "cost_usd": r.get("cost_usd")}))
+ finish(gen_dir, recs, {"backend": "official_logs", "model": model, "temperature": float(sub.temperature.iloc[0]) if "temperature" in sub else None,
+ "max_new_tokens": None, "setting": a.setting})
+ print(f"[rescore] {model}: {len(recs)} instances -> {gen_dir}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/eval/llm/run_eval.py b/eval/llm/run_eval.py
new file mode 100644
index 00000000..42e61e17
--- /dev/null
+++ b/eval/llm/run_eval.py
@@ -0,0 +1,135 @@
+"""One command: generate → execute → score → report, for one model over one or more suites.
+
+ python -m llm.run_eval --backend vllm --model /path/Qwen3-8B --run qwen3_8b --suites headline heldout
+ python -m llm.run_eval --backend openai --base-url http://localhost:8000/v1 --model Qwen3-8B --run q8 --suites all
+ python -m llm.run_eval --backend reference --run reference --suites all # executor self-test
+ python -m llm.run_eval ... --samples 4 --temperature 0.7 # pass@k
+
+Outputs land in $CV3D_EVAL_OUT/<run>/<suite>[/sN]/ and a report.md is rebuilt over the whole output root.
+Stages can be skipped/resumed: --stages gen exec score (default: all); generation of an existing gens.jsonl is
+reused unless --force.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import time
+from pathlib import Path
+
+from . import config, generate, report
+from .execute import execute_dir
+from .score import score_dir
+from .suites import expand, get_suite, load_prompts
+
+
+def main() -> None:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--backend", choices=["vllm", "openai", "anthropic", "reference"], required=True)
+ ap.add_argument("--model", default=None, help="model path / HF id / API model name")
+ ap.add_argument("--run", required=True, help="output folder name (model tag)")
+ ap.add_argument("--suites", nargs="+", default=["headline"], help="suite names or tags: all headline text vlm heldout rubric long-output")
+ ap.add_argument("--out", type=Path, default=config.OUT_DIR)
+ ap.add_argument("--limit", type=int, default=None, help="first N prompts of every suite (smoke tests)")
+ ap.add_argument("--temperature", type=float, default=None, help="override the suite default")
+ ap.add_argument("--max-new-tokens", type=int, default=None, help="override the suite default")
+ ap.add_argument("--samples", type=int, default=1, help="N sampled generations per prompt (pass@k); seeds 0..N-1")
+ ap.add_argument("--think", action="store_true", help="leave the model's thinking mode on (default: enable_thinking=False)")
+ ap.add_argument("--stages", nargs="+", default=["gen", "exec", "render", "score"], choices=["gen", "exec", "render", "score"],
+ help="render = 4 canonical views of every OK mesh (suites that score image_sim, or any suite when --judge-model is set)")
+ ap.add_argument("--render-workers", type=int, default=2)
+ ap.add_argument("--render-engine", default="CYCLES", choices=["CYCLES", "BLENDER_EEVEE"])
+ ap.add_argument("--image-encoders", nargs="*", default=["siglip2", "dinov3"])
+ ap.add_argument("--force", action="store_true", help="regenerate even if gens.jsonl exists")
+ ap.add_argument("--ref", choices=["executed", "canonical"], default="executed")
+ # vllm
+ ap.add_argument("--tp", type=int, default=1)
+ ap.add_argument("--max-model-len", type=int, default=16384)
+ ap.add_argument("--gpu-mem", type=float, default=0.90)
+ ap.add_argument("--max-num-seqs", type=int, default=None)
+ ap.add_argument("--max-images", type=int, default=4)
+ # api
+ ap.add_argument("--base-url", default=None)
+ ap.add_argument("--workers", type=int, default=8, help="API concurrency")
+ ap.add_argument("--exec-workers", type=int, default=None)
+ ap.add_argument("--score-workers", type=int, default=4, help="geometry-metric processes (each loads full meshes; keep small on <64 GB RAM)")
+ # optional VLM judge (absolute 1-5 grades from the 4 renders; see judge.py)
+ ap.add_argument("--judge-model", default=None, help="run judge.py absolute mode after scoring, e.g. gpt-5.4 or a local VLM")
+ ap.add_argument("--judge-backend", choices=["openai", "anthropic"], default="openai")
+ ap.add_argument("--judge-base-url", default=None)
+ a = ap.parse_args()
+
+ suites = expand(a.suites)
+ run_dir = a.out / a.run
+ run_dir.mkdir(parents=True, exist_ok=True)
+ (run_dir / "run_args.json").write_text(json.dumps(vars(a), default=str, indent=2))
+ print(f"[run] {a.run}: suites {suites} -> {run_dir}", flush=True)
+
+ plan = [] # (suite, sample_idx, gen_dir)
+ for s in suites:
+ for k in range(a.samples):
+ gen_dir = run_dir / s / (f"s{k}" if a.samples > 1 else "")
+ plan.append((s, k, gen_dir))
+
+ t0 = time.time()
+ if "gen" in a.stages:
+ engine = None
+ for s, k, gen_dir in plan:
+ spec = get_suite(s)
+ rows = load_prompts(s, a.limit)
+ if (gen_dir / "gens.jsonl").exists() and not a.force:
+ print(f"[gen] {s} s{k}: gens.jsonl exists, skipping (use --force)", flush=True)
+ continue
+ temp = a.temperature if a.temperature is not None else spec.temperature
+ if a.samples > 1 and temp == 0.0:
+ temp = 0.7
+ mnt = a.max_new_tokens or spec.max_new_tokens
+ if a.backend == "reference":
+ generate.reference_generate(rows, gen_dir)
+ elif a.backend == "vllm":
+ if engine is None:
+ engine = generate.VllmEngine(a.model, tp=a.tp, max_model_len=a.max_model_len, gpu_mem=a.gpu_mem,
+ max_images=a.max_images if any(r.get("images") for r in rows) else 0,
+ max_num_seqs=a.max_num_seqs)
+ engine.generate(rows, gen_dir, temp, mnt, seed=k, no_think=not a.think)
+ else:
+ generate.api_generate(a.backend, a.model, rows, gen_dir, temp, mnt, workers=a.workers, base_url=a.base_url,
+ no_think=not a.think)
+ if engine is not None:
+ del engine
+ try:
+ import gc
+ import torch
+ gc.collect()
+ torch.cuda.empty_cache()
+ except Exception: # noqa: BLE001
+ pass
+ print(f"[run] generation done in {(time.time() - t0) / 60:.1f} min", flush=True)
+
+ if "exec" in a.stages:
+ for s, k, gen_dir in plan:
+ if not gen_dir.exists():
+ continue
+ execute_dir(gen_dir, get_suite(s).dialect, workers=a.exec_workers)
+ if "render" in a.stages:
+ from .render_views import render_dir
+ for s, k, gen_dir in plan:
+ if gen_dir.exists() and ("image_sim" in get_suite(s).metrics or a.judge_model) and (gen_dir / "exec_results.jsonl").exists():
+ render_dir(gen_dir, workers=a.render_workers, engine=a.render_engine)
+ if "score" in a.stages:
+ for s, k, gen_dir in plan:
+ if not gen_dir.exists():
+ continue
+ score_dir(gen_dir, s, ref=a.ref, workers=a.score_workers, image_encoders=tuple(a.image_encoders))
+ if a.judge_model:
+ from .judge import JudgeClient, judge_absolute
+ jc = JudgeClient(a.judge_backend, a.judge_model, a.judge_base_url)
+ tag = "".join(c if c.isalnum() or c in "._-" else "_" for c in a.judge_model)
+ for s, k, gen_dir in plan:
+ if gen_dir.exists():
+ judge_absolute(a.run, s if a.samples == 1 else f"{s}/s{k}", jc, tag, workers=a.workers, limit=a.limit, out_root=a.out)
+ report.build(a.out)
+ print(f"[run] done in {(time.time() - t0) / 60:.1f} min; report at {a.out / 'report.md'}", flush=True)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/eval/llm/score.py b/eval/llm/score.py
new file mode 100644
index 00000000..527775c2
--- /dev/null
+++ b/eval/llm/score.py
@@ -0,0 +1,214 @@
+"""Score one gen dir of one suite → <gen_dir>/metrics.jsonl + summary.json (one schema for every suite).
+
+summary.json
+ suite, dialect, model, n, statuses{OK,EMPTY,FAIL,CRASH,TIMEOUT,MISSING}, exec_ok, exec_rate,
+ n_ref_ok (tasks whose reference produced a mesh — the denominator of the *_all geometry means),
+ n_scored, chamfer_mean_scored, chamfer_median_scored, f05_mean_scored, f10_mean_scored, iou_mean_scored,
+ f05_mean_all, f10_mean_all, iou_mean_all (fail = 0, over n_ref_ok)
+ render_ok, render_rate (glsl: compiled AND painted a non-uniform image)
+ truncated, mean_new_tokens, extract{no_fence, multi_block, chosen_not_last, from_think_fallback, ...}
+ gen{backend, model, temperature, seed, max_new_tokens, no_think}
+"""
+from __future__ import annotations
+
+import json
+from collections import Counter
+
+import numpy as np
+from concurrent.futures import ProcessPoolExecutor
+from pathlib import Path
+
+from . import metrics
+from .suites import get_suite, load_prompts, resolve
+
+
+def _geom(args):
+ gen_mesh, ref_mesh, n_points = args
+ try:
+ return metrics.geometry_metrics(gen_mesh, ref_mesh, n_points=n_points)
+ except Exception as e: # noqa: BLE001
+ return {"metric_error": f"{type(e).__name__}: {str(e)[:200]}"}
+
+
+def _geom_official(args):
+ gen_mesh, ref_mesh = args
+ try:
+ return metrics.chamfer_official(gen_mesh, ref_mesh)
+ except Exception as e: # noqa: BLE001
+ return {"metric_error_official": f"{type(e).__name__}: {str(e)[:200]}"}
+
+
+STRUCT_FIELDS = ("n_components", "floating_part_rate", "floating_area_fraction", "largest_component_volume_fraction",
+ "nonmanifold_edge_rate", "boundary_edge_rate", "is_watertight", "ground_contact", "com_over_support",
+ "com_support_margin")
+STRUCT_PENALTY = {"floating_part_rate": 1.0, "floating_area_fraction": 1.0, "largest_component_volume_fraction": 0.0,
+ "nonmanifold_edge_rate": 1.0, "boundary_edge_rate": 1.0, "is_watertight": 0.0, "ground_contact": 0.0,
+ "com_over_support": 0.0}
+
+
+def _struct(args):
+ """Official structural-integrity fields (structural.py) for one generated mesh (+ component ratio vs the reference)."""
+ gen_mesh, ref_mesh = args
+ from . import structural
+
+ up = 1 if str(gen_mesh).endswith(".glb") else 2
+ try:
+ rec, st = structural.score_glb(gen_mesh, 0.01, up, 1e-3, 100_000)
+ if rec is None:
+ return {"struct_status": st}
+ out = {"struct_status": st, **{k: rec.get(k) for k in STRUCT_FIELDS}}
+ if ref_mesh:
+ rrec, rst = structural.score_glb(ref_mesh, 0.01, up, 1e-3, 100_000)
+ if rrec:
+ out["ref_n_components"] = rrec["n_components"]
+ out["abs_log2_component_ratio"] = abs(float(np.log2(max(rec["n_components"] / max(rrec["n_components"], 1), 1e-9))))
+ return out
+ except Exception as e: # noqa: BLE001
+ return {"struct_status": f"SCORE_FAIL: {type(e).__name__}: {str(e)[:120]}"}
+
+
+def score_dir(gen_dir: Path, suite: str, ref: str = "executed", workers: int = 4, with_geometry: bool = True,
+ image_encoders: tuple[str, ...] = ("siglip2", "dinov3"), device: str = "cuda") -> dict:
+ gen_dir = Path(gen_dir)
+ spec = get_suite(suite)
+ rows = {r["id"]: r for r in load_prompts(suite)}
+ ex = {}
+ if (gen_dir / "exec_results.jsonl").exists():
+ ex = {json.loads(l)["id"]: json.loads(l) for l in (gen_dir / "exec_results.jsonl").open() if l.strip()}
+ gens = {}
+ if (gen_dir / "gens.jsonl").exists():
+ gens = {json.loads(l)["id"]: json.loads(l) for l in (gen_dir / "gens.jsonl").open() if l.strip()}
+ gen_stats = json.loads((gen_dir / "gen_stats.json").read_text()) if (gen_dir / "gen_stats.json").exists() else {}
+ n_prompts_total = len(rows)
+ if gens: # a --limit run: score only what was generated, but say so in the summary
+ rows = {tid: r for tid, r in rows.items() if tid in gens}
+
+ out_rows, geom_jobs, struct_jobs = [], [], []
+ for tid, row in rows.items():
+ e = ex.get(tid, {})
+ r = {"id": tid, "status": e.get("status", "MISSING"), "error": (e.get("error") or None) and str(e.get("error"))[:200],
+ "latency_s": e.get("latency_s"), "n_new_tokens": gens.get(tid, {}).get("n_new_tokens"),
+ "truncated": gens.get(tid, {}).get("truncated")}
+ if "render" in spec.metrics:
+ r["render"] = e.get("render")
+ ref_rel = row["reference"].get("mesh")
+ if ref == "canonical" and ref_rel:
+ cand = resolve(ref_rel).with_name("ref_canonical.glb")
+ ref_rel = str(cand.relative_to(resolve("."))) if cand.exists() else ref_rel
+ ref_mesh = resolve(ref_rel) if ref_rel else None
+ r["has_ref"] = bool(ref_mesh and ref_mesh.exists())
+ if with_geometry and "geometry" in spec.metrics and r["status"] == "OK" and e.get("mesh") and r["has_ref"]:
+ geom_jobs.append((len(out_rows), (e["mesh"], str(ref_mesh), spec.n_points)))
+ if with_geometry and spec.dialect in ("blender", "cadquery", "openscad") and r["status"] == "OK" and e.get("mesh"):
+ struct_jobs.append((len(out_rows), (e["mesh"], str(ref_mesh) if r["has_ref"] else None)))
+ out_rows.append(r)
+
+ if geom_jobs:
+ with ProcessPoolExecutor(workers) as pool:
+ for (i, _), res in zip(geom_jobs, pool.map(_geom, [j for _, j in geom_jobs])):
+ out_rows[i].update(res)
+ if "chamfer_official" in spec.metrics:
+ for (i, _), res in zip(geom_jobs, pool.map(_geom_official, [(j[0], j[1]) for _, j in geom_jobs])):
+ out_rows[i].update(res)
+
+ if struct_jobs:
+ with ProcessPoolExecutor(workers) as pool:
+ for (i, _), res in zip(struct_jobs, pool.map(_struct, [j for _, j in struct_jobs])):
+ out_rows[i].update(res)
+
+ image_summ = {}
+ if "image_sim" in spec.metrics and any(r["reference"].get("renders") for r in rows.values()):
+ rendered = {p.parent.parent.parent.name for p in gen_dir.glob("*/exec/renders/Image_035.png")}
+ for tid in rows:
+ out_rows[[r["id"] for r in out_rows].index(tid)]["rendered"] = tid in rendered
+ if rendered:
+ from . import image_sim
+ for enc in image_encoders:
+ try:
+ image_summ[enc] = image_sim.score_gen_dir(gen_dir, rows, resolve, enc, device, text_sim=(enc.startswith("siglip") and not any(r.get("images") for r in rows.values())))
+ per = {json.loads(l)["id"]: json.loads(l) for l in (gen_dir / f"image_sim_{enc}.jsonl").open()}
+ for r in out_rows:
+ p = per.get(r["id"], {})
+ r[f"{enc}_paired"], r[f"{enc}_assigned"] = p.get("view_paired"), p.get("best_assignment")
+ if "text_image" in p:
+ r[f"{enc}_text"] = p["text_image"]
+ except Exception as e: # noqa: BLE001
+ image_summ[enc] = {"error": f"{type(e).__name__}: {str(e)[:200]}"}
+ print(f"[score] image_sim {enc} failed: {image_summ[enc]['error']}", flush=True)
+
+ with (gen_dir / "metrics.jsonl").open("w") as f:
+ for r in out_rows:
+ f.write(json.dumps(r) + "\n")
+
+ n = len(out_rows)
+ st = Counter(r["status"] for r in out_rows)
+ ok = st.get("OK", 0)
+ summ = {"suite": suite, "dialect": spec.dialect, "model": gen_stats.get("model"), "n": n, "n_prompts_total": n_prompts_total,
+ "statuses": dict(st),
+ "exec_ok": ok, "exec_rate": ok / n if n else None,
+ "truncated": sum(bool(r.get("truncated")) for r in out_rows),
+ "mean_new_tokens": metrics.mean([r.get("n_new_tokens") for r in out_rows]),
+ "extract": {k: gen_stats.get(k) for k in ("no_fence", "multi_block", "chosen_not_last", "from_think_fallback",
+ "syntax_valid", "no_dialect_marker", "with_think")},
+ "gen": {k: gen_stats.get(k) for k in ("backend", "model", "temperature", "seed", "max_new_tokens", "no_think",
+ "tok_per_s", "wall_s")}}
+ if "geometry" in spec.metrics:
+ with_ref = [r for r in out_rows if r["has_ref"]]
+ scored = [r for r in with_ref if "chamfer" in r]
+ n_ref = len(with_ref)
+ summ.update(n_ref_ok=n_ref, n_scored=len(scored),
+ chamfer_mean_scored=metrics.mean([r["chamfer"] for r in scored]),
+ chamfer_median_scored=metrics.median([r["chamfer"] for r in scored]),
+ f05_mean_scored=metrics.mean([r["[email protected]"] for r in scored]),
+ f10_mean_scored=metrics.mean([r["[email protected]"] for r in scored]),
+ iou_mean_scored=metrics.mean([r.get("iou_vox") for r in scored]),
+ f05_mean_all=(sum(r["[email protected]"] for r in scored) / n_ref) if n_ref else None,
+ f10_mean_all=(sum(r["[email protected]"] for r in scored) / n_ref) if n_ref else None,
+ iou_mean_all=(sum(r.get("iou_vox") or 0 for r in scored) / n_ref) if n_ref else None)
+ if struct_jobs:
+ sc = [r for r in out_rows if r.get("struct_status") == "OK"]
+ summ["structural"] = {"n_scored": len(sc)}
+ for k, pen in STRUCT_PENALTY.items():
+ vals = [float(r[k]) for r in sc if r.get(k) is not None]
+ summ["structural"][f"{k}_cond"] = metrics.mean(vals)
+ summ["structural"][f"{k}_pen"] = (sum(vals) + pen * (n - len(vals))) / n if n else None
+ summ["structural"]["n_components_median_cond"] = metrics.median([r.get("n_components") for r in sc])
+ summ["structural"]["abs_log2_component_ratio_cond"] = metrics.mean([r.get("abs_log2_component_ratio") for r in sc])
+ if "chamfer_official" in spec.metrics:
+ sc = [r for r in out_rows if r.get("cd_official_yawmin") is not None]
+ pen = 1.5 * max((r["cd_official_yawmin"] for r in sc), default=0.0) # official penalty: 1.5 x worst observed
+ summ.update(cd_official_cond=metrics.mean([r["cd_official"] for r in sc]),
+ cd_official_yawmin_cond=metrics.mean([r["cd_official_yawmin"] for r in sc]),
+ cd_official_yawmin_pen=(sum(r.get("cd_official_yawmin") if r.get("cd_official_yawmin") is not None else pen for r in out_rows) / n) if n else None)
+ if image_summ:
+ summ["image_sim"] = image_summ
+ if "render" in spec.metrics:
+ rok = sum(r.get("render") == "OK" for r in out_rows)
+ summ.update(render_ok=rok, render_rate=rok / n if n else None,
+ render_static=sum(r.get("render") == "STATIC" for r in out_rows))
+ (gen_dir / "summary.json").write_text(json.dumps(summ, indent=2))
+ line = f"[score] {suite} {summ.get('model')}: exec {ok}/{n} = {100 * (summ['exec_rate'] or 0):.1f}%"
+ if "geometry" in spec.metrics:
+ line += f" | [email protected] all {summ['f05_mean_all'] or 0:.3f} (ok {summ['f05_mean_scored'] or 0:.3f}) | CD {summ['chamfer_mean_scored'] or 0:.3f} | IoU {summ['iou_mean_scored'] or 0:.3f}"
+ if "render" in spec.metrics:
+ line += f" | render OK {summ['render_ok']}/{n}"
+ if struct_jobs and summ.get("structural", {}).get("floating_part_rate_cond") is not None:
+ st_ = summ["structural"]
+ line += f" | floating {st_['floating_part_rate_cond']:.3f} | watertight {st_['is_watertight_cond']:.2f} | stable {st_['com_over_support_cond']:.2f}"
+ for enc, s in image_summ.items():
+ if "view_paired" in s:
+ line += f" | {enc} paired {s['view_paired']['penalized'] or 0:.3f} (cond {s['view_paired']['conditional'] or 0:.3f})"
+ print(line, flush=True)
+ return summ
+
+
+if __name__ == "__main__":
+ import argparse
+
+ ap = argparse.ArgumentParser()
+ ap.add_argument("gen_dir", type=Path)
+ ap.add_argument("--suite", required=True)
+ ap.add_argument("--ref", choices=["executed", "canonical"], default="executed")
+ ap.add_argument("--workers", type=int, default=8)
+ a = ap.parse_args()
+ score_dir(a.gen_dir, a.suite, a.ref, a.workers)
diff --git a/eval/llm/setup.sh b/eval/llm/setup.sh
new file mode 100644
index 00000000..2ccf89e2
--- /dev/null
+++ b/eval/llm/setup.sh
@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+# One-shot environment setup for 3dcodeverse_eval on a Linux box (no sudo needed).
+# bash llm/setup.sh # everything below
+# CV3D_TOOLS=/opt/cv3d-tools bash setup.sh # tools elsewhere
+# Installs: conda env `cv3d-eval` (vLLM, executors, metrics), Blender 5.0.1, OpenSCAD nightly AppImage
+# (+ libglvnd unpacked locally), glslang 16.5, Playwright Chromium, scipy/shapely inside Blender's python.
+# Re-runnable: every step skips what already exists. Afterwards: `python -m llm.config`.
+set -euo pipefail
+
+TOOLS="${CV3D_TOOLS:-$HOME/3dcodeverse_data/tools}"
+ENV_NAME="${CV3D_ENV:-cv3d-eval}"
+XLIBS="${CV3D_XLIBS:-$HOME/.local/xlibs/usr/lib/x86_64-linux-gnu}"
+mkdir -p "$TOOLS" "$HOME/.local/bin"
+
+log() { printf '\n\033[1;34m[setup]\033[0m %s\n' "$*"; }
+
+# ---------------------------------------------------------------------------------------------- conda env
+if ! conda env list | grep -qE "^${ENV_NAME}\s"; then
+ log "creating conda env $ENV_NAME"
+ conda create -y -n "$ENV_NAME" python=3.12
+fi
+PY="$(conda run -n "$ENV_NAME" python -c 'import sys; print(sys.executable)')"
+log "python: $PY"
+"$PY" -m ensurepip --upgrade >/dev/null 2>&1 || true
+"$PY" -m pip install -q --upgrade pip
+"$PY" -m pip install -q vllm trimesh scipy pyarrow pandas datasets huggingface_hub openai anthropic cadquery playwright \
+ pyyaml tqdm pillow "transformers>=4.50" accelerate pytest
+"$PY" -m playwright install chromium
+
+# ---------------------------------------------------------------------------------------------- Blender 5.0.1
+if [ ! -x "$TOOLS/blender-5.0.1-linux-x64/blender" ]; then
+ log "downloading Blender 5.0.1"
+ curl -L -o "$TOOLS/blender.tar.xz" https://download.blender.org/release/Blender5.0/blender-5.0.1-linux-x64.tar.xz
+ tar -xJf "$TOOLS/blender.tar.xz" -C "$TOOLS" && rm "$TOOLS/blender.tar.xz"
+fi
+BPY="$(ls -d "$TOOLS"/blender-5.0.1-linux-x64/5.0/python/bin/python3* | head -1)"
+"$BPY" -m ensurepip >/dev/null 2>&1 || true
+"$BPY" -m pip install -q "scipy>=1.14" shapely # 15/212 reference scripts import scipy; one needs shapely
+# libSM / libICE / libglvnd are often missing on headless boxes; unpack the Ubuntu debs locally (no sudo)
+mkdir -p "$HOME/.local/xlibs"
+if [ ! -e "$XLIBS/libOpenGL.so.0" ] || [ ! -e "$XLIBS/libSM.so.6" ]; then
+ log "unpacking libSM/libICE/libglvnd into ~/.local/xlibs"
+ tmp="$(mktemp -d)"; pushd "$tmp" >/dev/null
+ apt-get download libsm6 libice6 libglvnd0 libopengl0 libglx0 2>/dev/null || true
+ for d in *.deb; do [ -f "$d" ] && dpkg-deb -x "$d" "$HOME/.local/xlibs"; done
+ popd >/dev/null; rm -rf "$tmp"
+fi
+cat > "$HOME/.local/bin/blender-5.0" <<EOF
+#!/bin/bash
+export LD_LIBRARY_PATH="$XLIBS:\${LD_LIBRARY_PATH:-}"
+exec "$TOOLS/blender-5.0.1-linux-x64/blender" "\$@"
+EOF
+chmod +x "$HOME/.local/bin/blender-5.0"
+"$HOME/.local/bin/blender-5.0" --version | head -1
+
+# ---------------------------------------------------------------------------------------------- OpenSCAD (nightly, --export-format binstl)
+if [ ! -x "$TOOLS/openscad/AppRun" ]; then
+ log "downloading OpenSCAD nightly AppImage"
+ latest="$(curl -s https://files.openscad.org/snapshots/ | grep -oE 'OpenSCAD-[0-9.]+-x86_64\.AppImage' | sort -u | tail -1)"
+ curl -L -o "$TOOLS/openscad.AppImage" "https://files.openscad.org/snapshots/$latest"
+ chmod +x "$TOOLS/openscad.AppImage"
+ (cd "$TOOLS" && ./openscad.AppImage --appimage-extract >/dev/null && rm -rf openscad && mv squashfs-root openscad)
+fi
+LD_LIBRARY_PATH="$XLIBS:${LD_LIBRARY_PATH:-}" QT_QPA_PLATFORM=offscreen "$TOOLS/openscad/AppRun" --version 2>&1 | head -1
+
+# ---------------------------------------------------------------------------------------------- glslang
+if [ ! -x "$TOOLS/glslang/bin/glslang" ]; then
+ log "downloading glslang 16.5.0"
+ curl -L -o "$TOOLS/glslang.tgz" https://github.com/KhronosGroup/glslang/releases/download/16.5.0/glslang-16.5.0-linux-x86_64-release.tar.gz
+ mkdir -p "$TOOLS/glslang" && tar -xzf "$TOOLS/glslang.tgz" -C "$TOOLS/glslang" && rm "$TOOLS/glslang.tgz"
+fi
+"$TOOLS/glslang/bin/glslang" --version | head -1
+
+log "done. Next: hf auth login (token with ilabai/* access), then"
+echo " $PY -m llm.config && $PY -m llm.download --core --views --logs && $PY -m llm.build_prompts && SEED=0 $PY -m llm.build_refs"
diff --git a/eval/llm/structural.py b/eval/llm/structural.py
new file mode 100644
index 00000000..eb36d795
--- /dev/null
+++ b/eval/llm/structural.py
@@ -0,0 +1,1061 @@
+# Vendored verbatim from the official 3DCodeBench repo (metrics/structural_integrity.py, github.com/gaoypeng/3dcodebench, 2026-09)
+# so the numbers match the paper. Used through score_glb(); the CLI below still works standalone.
+#!/usr/bin/env python3
+"""Structural-integrity metric — is the mesh one coherent object, or a pile of parts?
+
+Motivation
+----------
+Perceptual scores (SigLIP-2 / DINOv3 image similarity, Uni3D, even Chamfer on
+sampled surface points) are computed on *appearance* or on *surface samples*.
+None of them can tell the difference between
+
+ (a) a chair whose four legs are welded to the seat, and
+ (b) a chair whose four legs float 3 cm below the seat,
+
+because both render nearly identically from a turntable and both produce almost
+the same point cloud. This scorer measures the properties those metrics are
+blind to: connectivity, fragmentation, manifoldness, and static plausibility.
+
+Inputs (same layout as shape_chamfer.py)
+----------------------------------------
+ reference: <reference-dir>/<inst>/glb/<inst>.glb
+ generated: <results-dir>/<inst>/glb/<inst>.glb
+
+Run with only ``--reference-dir`` (no ``--results-dir``) to score the reference
+meshes themselves and establish the GT baseline. This is not optional bookkeeping:
+the Infinigen procedural factories emit meshes that are frequently non-watertight
+and almost never single-component (a fern legitimately has one component per
+leaflet), so several of these numbers are only interpretable as a *ratio to the
+reference*, never as absolute quality.
+
+Pipeline (per mesh)
+-------------------
+ 1. ``trimesh.load(force="mesh")`` — concatenates all scene primitives into one
+ Trimesh, exactly like shape_chamfer.py.
+ 2. Normalize to a unit bounding sphere: subtract the area-weighted surface
+ centroid, divide by max ||v||. Same normalization target as
+ shape_chamfer.py (which uses the mean of its surface samples; the
+ area-weighted centroid is the deterministic version of that estimator).
+ Doing this *before* welding also makes the weld tolerance scale-free.
+ 3. Weld vertices by position with ``merge_vertices(merge_tex=True,
+ merge_norm=True)``. This is load-bearing: glTF splits vertices at UV seams
+ and normal discontinuities, so an unwelded Infinigen tree reports 2.26M
+ "components" for 3.95M faces (i.e. raw triangle soup). Ignoring UV/normal
+ splits is the only way to get a topologically meaningful component count.
+ 4. Connected components on the **vertex** graph (faces that share a vertex are
+ in the same component), not on ``mesh.face_adjacency``. trimesh's
+ face_adjacency silently drops any edge incident to 3+ faces, so on a
+ non-manifold mesh it *over*-reports components — welding a mesh can make its
+ face_adjacency component count go **up**, which is nonsense for our purpose.
+ 5. Everything else is vectorized per-component via ``np.bincount`` on the face
+ labels; no per-component submesh is ever materialized (some GT meshes have
+ 350k+ components).
+
+Metrics
+-------
+n_components connected-component count (vertex graph)
+component_ratio n_components / n_components_ref
+log2_component_ratio signed log2 of the above (0 = matches ref)
+abs_log2_component_ratio |log2 ratio| — the aggregation-friendly form
+largest_component_volume_fraction |vol(largest)| / sum_c |vol(c)|. KEY
+ fragmentation measure. Each component's
+ volume is the divergence-theorem integral
+ taken about *that component's own* area
+ centroid, so an open planar sheet integrates
+ to ~0 instead of to an origin-dependent
+ garbage value.
+largest_component_area_fraction same but surface area. Always well defined,
+ including for meshes that are pure sheets.
+largest_component_face_fraction same but face count (tessellation-dependent,
+ diagnostic only).
+floating_part_rate fraction of components not within eps of any
+ other component.
+floating_area_fraction surface area of those components / total.
+floating_volume_fraction volume of those components / total.
+nonmanifold_edge_rate edges incident to >=3 faces / unique edges.
+boundary_edge_rate edges incident to exactly 1 face / unique
+ edges (open boundary, NOT non-manifold).
+is_watertight, euler_number trimesh topology summary.
+ground_contact is there a real base? True iff the footprint
+ (points within eps of the min-up plane) has
+ >=3 points and a 2D convex-hull area of at
+ least --ground-min-area-frac of the
+ horizontal bbox area. This is a *degeneracy*
+ filter, not a flatness test: a sphere passes
+ (its eps-band cap has radius ~sqrt(2 R eps),
+ ~5% of the bbox area at eps_frac=1%), a cone
+ balanced on its apex fails (radius grows
+ only linearly in eps, ~2e-4 of bbox area),
+ and an object suspended above its own lowest
+ stray vertex fails. Expect it to be True for
+ almost everything; read
+ ground_contact_area_frac instead if you want
+ a graded signal.
+ground_contact_area_frac the raw continuous version of the above.
+com_over_support standard static-stability test: is the
+ horizontal projection of the centre of mass
+ inside the convex hull of the footprint?
+com_support_margin signed distance from the COM to the support
+ polygon boundary / horizontal bbox diagonal.
+ Positive = inside = stable. Continuous, so
+ it degrades gracefully; prefer it to the bool.
+
+Choice of epsilon
+-----------------
+eps = --eps-frac * (bounding-box diagonal of the normalized mesh), default
+--eps-frac = 0.01, i.e. **1% of the object's bbox diagonal**. Rationale: this is
+roughly the width of a visible seam at typical render resolution (a 512px render
+of an object filling the frame puts 1% of the diagonal at ~5px), and it is an
+order of magnitude above the float32 quantization of a glTF vertex buffer
+(~1e-7 relative) so it cannot be tripped by export noise. It is also small
+enough that a genuinely detached chair leg — the failure mode the reviewer asked
+about — sits well outside it. Sweep it with --eps-frac if you want a sensitivity
+curve; 0.005 and 0.02 are the natural bracketing values.
+
+Contact test
+------------
+Contact between components is decided on an axis-aligned grid of cell size eps:
+two components are "in contact" iff they have surface points in the same or in
+one of the 26 adjacent cells. Any two points at distance <= eps necessarily fall
+in the same or an adjacent cell, so **no true contact at <= eps is ever missed**;
+the test can additionally accept pairs up to sqrt(3)*eps ~ 1.73 eps apart. The
+bias is therefore toward calling things connected, i.e. toward *under*-reporting
+floating parts, which is the conservative direction for a claim about model
+failure. The alternative (an exact all-pairs KD-tree) is O(C^2) tree builds and
+is not tractable at 350k components.
+
+The point set used for contact / footprint / COM is the welded vertices plus
+enough area-proportional surface samples to reach ~1 point per eps^2 of surface
+(capped). Vertices alone are not sufficient: an LLM-generated mesh may be a
+12-triangle box whose vertices are metres apart, and a sphere resting on the
+middle of that box's top face would be scored as floating.
+
+Up axis
+-------
+The spec talks about the "z-min plane". These GLBs are Blender exports and the
+glTF exporter converts Z-up to **Y-up**; trimesh does not convert back
+(verified: TableDining_seed0 has extent [2.32, 0.70, 1.17], the 0.70 being the
+table height on Y). ``--up-axis`` therefore defaults to ``y``. Everything below
+says "up" where the spec says "z".
+
+Aggregation (matches shape_chamfer.py / image_similarity.py)
+------------------------------------------------------------
+ conditional — averaged over instances whose mesh loaded and had faces.
+ "Given that a mesh came out at all, how sound was it?"
+ penalized — instances with no/empty/broken mesh are folded in at the
+ worst possible value for each field (floating_part_rate -> 1,
+ largest_component_volume_fraction -> 0, every boolean -> False,
+ com_support_margin -> the worst observed in the run).
+ "Overall structural soundness, including hard failures."
+Booleans aggregate as rates; scalars report mean / median / std / p10 / p90.
+
+Output
+------
+ per-instance JSONL -> --out (default <out-dir>/structural_integrity.jsonl)
+ aggregate JSON -> same stem + .json
+
+Usage
+-----
+ # GT baseline (no model involved)
+ python metrics/structural_integrity.py \
+ --reference-dir data/data --workers 6 \
+ --out notes/e5_structural/reference_baseline.jsonl
+
+ # A model run, with ratios against the reference
+ python metrics/structural_integrity.py \
+ --results-dir results/text_to_3D/gemini-3-flash-preview \
+ --reference-dir data/data --workers 8
+
+ # Sanity checks on hand-built meshes
+ python metrics/structural_integrity.py --selftest
+"""
+
+import argparse
+import json
+import math
+import os
+import sys
+import time
+import traceback
+from concurrent.futures import ProcessPoolExecutor, as_completed
+from pathlib import Path
+
+import numpy as np
+import trimesh
+from scipy.sparse import coo_matrix
+from scipy.sparse.csgraph import connected_components as _csgraph_cc
+from scipy.spatial import ConvexHull
+
+EVAL_ROOT = Path(__file__).resolve().parent.parent
+DEFAULT_DATA_ROOT = EVAL_ROOT / "data"
+
+UP_AXIS_INDEX = {"x": 0, "y": 1, "z": 2}
+
+# Fields that are booleans in the per-instance record and aggregate as rates.
+BOOL_FIELDS = ["is_watertight", "is_single_component", "ground_contact",
+ "com_over_support", "winding_consistent"]
+
+# Scalar fields -> value used when the instance produced no usable mesh.
+# None means "use the worst value observed in this run".
+PENALTY = {
+ "n_components": None,
+ "largest_component_volume_fraction": 0.0,
+ "largest_component_area_fraction": 0.0,
+ "largest_component_face_fraction": 0.0,
+ "floating_part_rate": 1.0,
+ "floating_area_fraction": 1.0,
+ "floating_volume_fraction": 1.0,
+ "nonmanifold_edge_rate": 1.0,
+ "boundary_edge_rate": 1.0,
+ "ground_contact_area_frac": 0.0,
+ "com_support_margin": None,
+ "abs_log2_component_ratio": None,
+}
+SCALAR_FIELDS = list(PENALTY.keys())
+
+# Direction of "worse" for fields whose penalty is run-relative.
+WORSE_IS_MAX = {"n_components": True, "com_support_margin": False,
+ "abs_log2_component_ratio": True}
+
+
+# --------------------------------------------------------------------------- #
+# args
+# --------------------------------------------------------------------------- #
+def parse_args(argv=None):
+ p = argparse.ArgumentParser(
+ description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--results-dir", type=Path, default=None,
+ help="Model results tree: <results-dir>/<inst>/glb/<inst>.glb. "
+ "Omit to score the reference meshes themselves.")
+ p.add_argument("--reference-dir", type=Path, default=DEFAULT_DATA_ROOT,
+ help="Reference tree: <reference-dir>/<inst>/glb/<inst>.glb")
+ p.add_argument("--workers", type=int, default=8)
+ p.add_argument("--out", type=Path, default=None,
+ help="Per-instance JSONL path. The aggregate summary is "
+ "written next to it with a .json suffix.")
+ p.add_argument("--eps-frac", type=float, default=0.01,
+ help="Contact/footprint tolerance as a fraction of the "
+ "normalized bbox diagonal (default 0.01 = 1%%).")
+ p.add_argument("--ground-min-area-frac", type=float, default=1e-3,
+ help="Footprint convex-hull area, as a fraction of the "
+ "horizontal bbox area, below which the base counts as "
+ "degenerate (default 1e-3).")
+ p.add_argument("--up-axis", choices=["x", "y", "z"], default="y",
+ help="Vertical axis. Blender->glTF exports are Y-up "
+ "(default y).")
+ p.add_argument("--max-contact-samples", type=int, default=100_000,
+ help="Cap on extra surface samples added to the contact "
+ "point set (on top of every welded vertex).")
+ p.add_argument("--instances", nargs="*", default=None)
+ p.add_argument("--limit", type=int, default=None,
+ help="Cap number of instances (debug).")
+ p.add_argument("--selftest", action="store_true",
+ help="Run the hand-built sanity meshes and exit.")
+ return p.parse_args(argv)
+
+
+# --------------------------------------------------------------------------- #
+# geometry helpers
+# --------------------------------------------------------------------------- #
+def _hull_2d(points_2d):
+ """2D convex hull; returns (hull, area) or (None, 0.0) if degenerate."""
+ if len(points_2d) < 3:
+ return None, 0.0
+ try:
+ h = ConvexHull(np.ascontiguousarray(points_2d, dtype=np.float64))
+ except Exception:
+ return None, 0.0
+ # For a 2D hull scipy reports `volume` = enclosed area, `area` = perimeter.
+ return h, float(h.volume)
+
+
+def normalize_unit_sphere(mesh):
+ """Center on the area-weighted surface centroid, scale to max ||v|| = 1.
+
+ Mesh analogue of shape_chamfer.py's point-cloud normalization. Returns the
+ scale factor that was applied (useful only for logging).
+ """
+ af = mesh.area_faces
+ tot = float(af.sum())
+ if tot > 0:
+ center = (mesh.triangles_center * af[:, None]).sum(0) / tot
+ else:
+ center = np.asarray(mesh.vertices).mean(0)
+ mesh.apply_translation(-center)
+ r = float(np.linalg.norm(np.asarray(mesh.vertices), axis=1).max())
+ if r > 1e-12:
+ mesh.apply_scale(1.0 / r)
+ return r
+
+
+def vertex_graph_components(mesh):
+ """Connected components where two faces are connected iff they share a vertex.
+
+ Returns (face_label int32[F], n_components).
+
+ Deliberately NOT mesh.face_adjacency: trimesh builds face_adjacency only
+ from edges incident to exactly two faces, so a T-junction or any 3-faces-
+ to-an-edge configuration severs the graph. On Bird_seed0 that inflates the
+ count from 8329 real components to whatever the manifold patches happen to
+ be. The vertex graph is the definition that matches "is this one object".
+ """
+ faces = np.asarray(mesh.faces)
+ n_v = len(mesh.vertices)
+ if len(faces) == 0:
+ return np.zeros(0, dtype=np.int32), 0
+ e = np.concatenate([faces[:, [0, 1]], faces[:, [1, 2]], faces[:, [2, 0]]])
+ data = np.ones(len(e), dtype=np.int8)
+ g = coo_matrix((data, (e[:, 0], e[:, 1])), shape=(n_v, n_v))
+ _, vlab = _csgraph_cc(g, directed=False, return_labels=True)
+ face_label = vlab[faces[:, 0]]
+ # Drop labels belonging to isolated (unreferenced) vertices and compact.
+ uniq, face_label = np.unique(face_label, return_inverse=True)
+ return face_label.astype(np.int32), int(len(uniq))
+
+
+def per_component_area_volume(mesh, face_label, n_comp):
+ """Vectorized per-component surface area and (origin-free) volume.
+
+ Volume uses the divergence theorem with each component's own area centroid
+ as the origin: V_c = sum_{f in c} (a-o).((b-o) x (c-o)) / 6.
+ For a closed component this equals the enclosed volume (origin-independent).
+ For an open planar sheet every tetrahedron is degenerate and the sum is ~0,
+ which is exactly the behaviour we want -- otherwise a leaf far from the
+ world origin would be credited with a huge phantom volume and would
+ dominate largest_component_volume_fraction.
+ """
+ af = np.asarray(mesh.area_faces, dtype=np.float64)
+ area = np.bincount(face_label, weights=af, minlength=n_comp)
+
+ # Area centroid per component.
+ tc = np.asarray(mesh.triangles_center, dtype=np.float64)
+ cen = np.stack([np.bincount(face_label, weights=tc[:, k] * af,
+ minlength=n_comp) for k in range(3)], axis=1)
+ with np.errstate(invalid="ignore", divide="ignore"):
+ cen = np.where(area[:, None] > 0, cen / np.maximum(area, 1e-300)[:, None], 0.0)
+
+ tri = np.asarray(mesh.triangles, dtype=np.float64)
+ o = cen[face_label]
+ a = tri[:, 0] - o
+ b = tri[:, 1] - o
+ c = tri[:, 2] - o
+ vf = np.einsum("ij,ij->i", a, np.cross(b, c)) / 6.0
+ vol = np.abs(np.bincount(face_label, weights=vf, minlength=n_comp))
+ return area, vol
+
+
+def contact_point_set(mesh, face_label, eps, max_samples, rng):
+ """Welded vertices + area-proportional surface samples, each with a label.
+
+ Vertices alone under-sample coarse geometry (a 12-triangle box), which would
+ make a sphere sitting on that box read as floating. We top up to roughly one
+ point per eps^2 of surface area, capped.
+ """
+ verts = np.asarray(mesh.vertices, dtype=np.float64)
+ faces = np.asarray(mesh.faces)
+ v_lab = np.zeros(len(verts), dtype=np.int32)
+ # Label every referenced vertex by (any) incident face's component.
+ v_lab[faces.reshape(-1)] = np.repeat(face_label, 3)
+
+ pts = [verts]
+ labs = [v_lab]
+
+ total_area = float(mesh.area)
+ want = int(total_area / max(eps * eps, 1e-12)) - len(verts)
+ n_extra = int(np.clip(want, 0, max_samples))
+ if n_extra > 0:
+ try:
+ s_pts, s_fidx = trimesh.sample.sample_surface(
+ mesh, n_extra, seed=int(rng.integers(0, 2 ** 31 - 1)))
+ pts.append(np.asarray(s_pts, dtype=np.float64))
+ labs.append(face_label[np.asarray(s_fidx)])
+ except Exception:
+ pass
+ return np.concatenate(pts), np.concatenate(labs).astype(np.int64)
+
+
+def _cell_keys(points, eps):
+ """Pack floor(p/eps) into a single int64 key per point."""
+ ijk = np.floor(points / eps).astype(np.int64)
+ ijk -= ijk.min(axis=0) # shift to non-negative
+ m = ijk.max(axis=0) + 3 # +3 leaves room for +/-1 probes
+ base_y = int(m[2])
+ base_x = int(m[1]) * base_y
+ return (ijk[:, 0] + 1) * base_x + (ijk[:, 1] + 1) * base_y + (ijk[:, 2] + 1), \
+ (base_x, base_y)
+
+
+def components_in_contact(points, labels, n_comp, eps):
+ """Boolean[n_comp]: does this component have another component within eps?
+
+ Grid method: two points within eps always land in the same or a 26-adjacent
+ cell of an eps-sized grid, so this never misses a true contact; it can
+ over-accept out to sqrt(3)*eps. See module docstring.
+ """
+ touching = np.zeros(n_comp, dtype=bool)
+ if n_comp <= 1 or len(points) == 0:
+ return touching
+
+ keys, (bx, by) = _cell_keys(points, eps)
+
+ # Unique (cell, component) pairs, then per-cell summary.
+ pair = np.stack([keys, labels], axis=1)
+ pair = np.unique(pair, axis=0) # sorted by cell then component
+ p_cell, p_lab = pair[:, 0], pair[:, 1]
+
+ cells, first_idx, counts = np.unique(p_cell, return_index=True,
+ return_counts=True)
+ cell_first_lab = p_lab[first_idx] # only meaningful when count==1
+
+ offsets = [dx * bx + dy * by + dz
+ for dx in (-1, 0, 1) for dy in (-1, 0, 1) for dz in (-1, 0, 1)]
+
+ hit = np.zeros(len(p_cell), dtype=bool)
+ for off in offsets:
+ probe = p_cell + off
+ idx = np.searchsorted(cells, probe)
+ np.clip(idx, 0, len(cells) - 1, out=idx)
+ ok = cells[idx] == probe
+ # Neighbour cell holds a *different* component if it holds >1 component,
+ # or holds exactly one that isn't ours.
+ diff = (counts[idx] > 1) | (cell_first_lab[idx] != p_lab)
+ hit |= ok & diff
+
+ np.logical_or.at(touching, p_lab, hit)
+ return touching
+
+
+def support_metrics(points, mesh, up, eps, ground_min_area_frac, is_watertight):
+ """Ground contact + static stability.
+
+ Footprint = points within eps of the minimum-up plane, projected to the two
+ horizontal axes. Support polygon = convex hull of that footprint.
+ """
+ horiz = [i for i in range(3) if i != up]
+ lo = points.min(axis=0)
+ hi = points.max(axis=0)
+ ext = hi - lo
+ bbox_h_area = float(ext[horiz[0]] * ext[horiz[1]])
+ bbox_h_diag = float(math.hypot(ext[horiz[0]], ext[horiz[1]]))
+
+ mask = points[:, up] <= (lo[up] + eps)
+ foot = points[mask][:, horiz]
+ n_foot = int(mask.sum())
+
+ hull, hull_area = _hull_2d(foot)
+ area_frac = hull_area / bbox_h_area if bbox_h_area > 1e-12 else 0.0
+ ground_contact = bool(n_foot >= 3 and area_frac >= ground_min_area_frac)
+
+ # Centre of mass. mesh.center_mass is the volume integral and is meaningless
+ # for an open mesh, so fall back to the area-weighted surface centroid.
+ com_mode = "volume"
+ com = None
+ if is_watertight:
+ try:
+ cm = np.asarray(mesh.center_mass, dtype=np.float64)
+ if np.all(np.isfinite(cm)) and abs(mesh.volume) > 1e-12:
+ com = cm
+ except Exception:
+ com = None
+ if com is None:
+ com_mode = "area"
+ af = np.asarray(mesh.area_faces, dtype=np.float64)
+ tot = float(af.sum())
+ if tot > 0:
+ com = (np.asarray(mesh.triangles_center) * af[:, None]).sum(0) / tot
+ else:
+ com = np.asarray(mesh.vertices, dtype=np.float64).mean(0)
+
+ com_h = np.asarray(com, dtype=np.float64)[horiz]
+ if hull is None:
+ com_over = False
+ margin = None
+ else:
+ # ConvexHull.equations: outward normals, inside <=> all n.x + d <= 0.
+ vals = hull.equations[:, :2] @ com_h + hull.equations[:, 2]
+ signed = -float(vals.max()) # >0 inside
+ com_over = bool(signed >= 0.0)
+ margin = signed / bbox_h_diag if bbox_h_diag > 1e-12 else 0.0
+
+ return {
+ "ground_contact": ground_contact,
+ "ground_contact_area_frac": float(area_frac),
+ "n_footprint_points": n_foot,
+ "footprint_point_frac": float(n_foot / max(len(points), 1)),
+ "com_over_support": com_over,
+ "com_support_margin": margin,
+ "com_mode": com_mode,
+ "bbox_extent": [float(x) for x in ext],
+ }
+
+
+# --------------------------------------------------------------------------- #
+# per-mesh scoring
+# --------------------------------------------------------------------------- #
+def score_mesh_object(mesh, eps_frac, up, ground_min_area_frac,
+ max_contact_samples, seed=0):
+ """Score an already-loaded Trimesh. Normalizes and welds in place."""
+ rng = np.random.default_rng(seed)
+
+ if mesh is None or len(getattr(mesh, "faces", [])) == 0:
+ return None
+
+ normalize_unit_sphere(mesh)
+ # Weld by position, ignoring UV / normal splits introduced by glTF.
+ try:
+ mesh.merge_vertices(merge_tex=True, merge_norm=True)
+ except Exception:
+ pass
+ if len(mesh.faces) == 0:
+ return None
+
+ face_label, n_comp = vertex_graph_components(mesh)
+ if n_comp == 0:
+ return None
+
+ ext = mesh.extents
+ bbox_diag = float(np.linalg.norm(ext))
+ eps = eps_frac * bbox_diag
+ if not np.isfinite(eps) or eps <= 0:
+ eps = eps_frac * 2.0
+
+ area_c, vol_c = per_component_area_volume(mesh, face_label, n_comp)
+ face_c = np.bincount(face_label, minlength=n_comp).astype(np.float64)
+ tot_area, tot_vol, tot_face = area_c.sum(), vol_c.sum(), face_c.sum()
+
+ def frac(arr, tot):
+ return float(arr.max() / tot) if tot > 1e-15 else None
+
+ pts, plab = contact_point_set(mesh, face_label, eps, max_contact_samples, rng)
+ touching = components_in_contact(pts, plab, n_comp, eps)
+ if n_comp <= 1:
+ floating = np.zeros(n_comp, dtype=bool)
+ else:
+ floating = ~touching
+ n_float = int(floating.sum())
+
+ # Edge manifoldness.
+ inv = np.asarray(mesh.edges_unique_inverse)
+ n_unique_edges = int(len(mesh.edges_unique))
+ if n_unique_edges > 0:
+ per_edge = np.bincount(inv, minlength=n_unique_edges)
+ nonmanifold = int((per_edge >= 3).sum())
+ boundary = int((per_edge == 1).sum())
+ else:
+ nonmanifold = boundary = 0
+
+ try:
+ watertight = bool(mesh.is_watertight)
+ except Exception:
+ watertight = False
+ try:
+ winding = bool(mesh.is_winding_consistent)
+ except Exception:
+ winding = False
+ try:
+ euler = int(mesh.euler_number)
+ except Exception:
+ euler = None
+
+ rec = {
+ "n_vertices": int(len(mesh.vertices)),
+ "n_faces": int(len(mesh.faces)),
+ "n_edges_unique": n_unique_edges,
+ "n_components": int(n_comp),
+ "is_single_component": bool(n_comp == 1),
+ "largest_component_volume_fraction": frac(vol_c, tot_vol),
+ "largest_component_area_fraction": frac(area_c, tot_area),
+ "largest_component_face_fraction": frac(face_c, tot_face),
+ "n_floating_components": n_float,
+ "floating_part_rate": float(n_float / n_comp),
+ "floating_area_fraction": (float(area_c[floating].sum() / tot_area)
+ if tot_area > 1e-15 else 0.0),
+ "floating_volume_fraction": (float(vol_c[floating].sum() / tot_vol)
+ if tot_vol > 1e-15 else 0.0),
+ "n_nonmanifold_edges": nonmanifold,
+ "nonmanifold_edge_rate": float(nonmanifold / max(n_unique_edges, 1)),
+ "n_boundary_edges": boundary,
+ "boundary_edge_rate": float(boundary / max(n_unique_edges, 1)),
+ "is_watertight": watertight,
+ "winding_consistent": winding,
+ "euler_number": euler,
+ "surface_area": float(tot_area),
+ "abs_volume_sum": float(tot_vol),
+ "bbox_diag": bbox_diag,
+ "eps": float(eps),
+ "n_contact_points": int(len(pts)),
+ }
+ rec.update(support_metrics(pts, mesh, up, eps, ground_min_area_frac,
+ watertight))
+ return rec
+
+
+def score_glb(glb_path, eps_frac, up, ground_min_area_frac,
+ max_contact_samples, seed=0):
+ """Load, score. Returns (record_or_None, status)."""
+ try:
+ mesh = trimesh.load(glb_path, force="mesh")
+ except Exception:
+ return None, "LOAD_FAIL"
+ if mesh is None or not hasattr(mesh, "faces") or len(mesh.faces) == 0:
+ return None, "EMPTY_MESH"
+ try:
+ rec = score_mesh_object(mesh, eps_frac, up, ground_min_area_frac,
+ max_contact_samples, seed=seed)
+ except Exception:
+ return None, "SCORE_FAIL:" + traceback.format_exc(limit=2).strip().splitlines()[-1][:160]
+ if rec is None:
+ return None, "EMPTY_MESH"
+ return rec, "OK"
+
+
+def _worker(job):
+ (inst, ref_glb, gen_glb, eps_frac, up, gmaf, mcs) = job
+ t0 = time.time()
+ out = {"instance": inst}
+ ref_rec = gen_rec = None
+
+ if ref_glb is not None:
+ ref_glb = Path(ref_glb)
+ if ref_glb.exists():
+ ref_rec, st = score_glb(str(ref_glb), eps_frac, up, gmaf, mcs)
+ out["ref_status"] = st
+ else:
+ out["ref_status"] = "NO_REF_GLB"
+ if gen_glb is not None:
+ gen_glb = Path(gen_glb)
+ if gen_glb.exists():
+ gen_rec, st = score_glb(str(gen_glb), eps_frac, up, gmaf, mcs)
+ out["gen_status"] = st
+ else:
+ out["gen_status"] = "NO_GEN_GLB"
+
+ if gen_glb is None:
+ # reference-only mode: the reference IS the subject
+ out["status"] = out.get("ref_status", "NO_REF_GLB")
+ if ref_rec:
+ out.update(ref_rec)
+ else:
+ out["status"] = out.get("gen_status", "NO_GEN_GLB")
+ if gen_rec:
+ out.update(gen_rec)
+ if ref_rec:
+ out["ref_n_components"] = ref_rec["n_components"]
+ out["ref_is_watertight"] = ref_rec["is_watertight"]
+ out["ref_largest_component_volume_fraction"] = \
+ ref_rec["largest_component_volume_fraction"]
+ out["ref_floating_part_rate"] = ref_rec["floating_part_rate"]
+ if gen_rec:
+ r = gen_rec["n_components"] / max(ref_rec["n_components"], 1)
+ out["component_ratio"] = float(r)
+ out["log2_component_ratio"] = float(np.log2(max(r, 1e-9)))
+ out["abs_log2_component_ratio"] = abs(out["log2_component_ratio"])
+
+ out["seconds"] = round(time.time() - t0, 2)
+ return out
+
+
+# --------------------------------------------------------------------------- #
+# aggregation
+# --------------------------------------------------------------------------- #
+def _stats(vals):
+ v = np.asarray([x for x in vals if x is not None and np.isfinite(x)],
+ dtype=np.float64)
+ if len(v) == 0:
+ return None
+ return {"n": int(len(v)), "mean": float(v.mean()), "median": float(np.median(v)),
+ "std": float(v.std()), "p10": float(np.percentile(v, 10)),
+ "p90": float(np.percentile(v, 90)), "min": float(v.min()),
+ "max": float(v.max())}
+
+
+def aggregate(rows):
+ """conditional (OK rows only) vs penalized (failures at worst value)."""
+ ok = [r for r in rows if r.get("status") == "OK"]
+ n, n_ok = len(rows), len(ok)
+
+ penalties = {}
+ for f in SCALAR_FIELDS:
+ p = PENALTY[f]
+ if p is None:
+ vals = [r.get(f) for r in ok if r.get(f) is not None]
+ vals = [v for v in vals if np.isfinite(v)]
+ if vals:
+ p = max(vals) if WORSE_IS_MAX.get(f, True) else min(vals)
+ else:
+ p = None
+ penalties[f] = p
+
+ out = {"n_instances": n, "n_ok": n_ok,
+ "ok_rate": (n_ok / n) if n else None,
+ "penalties": penalties, "scalars": {}, "booleans": {}}
+
+ for f in SCALAR_FIELDS:
+ cond = _stats([r.get(f) for r in ok])
+ p = penalties[f]
+ if p is None:
+ pen = None
+ else:
+ pen = _stats([(r.get(f) if r.get("status") == "OK"
+ and r.get(f) is not None else p) for r in rows])
+ out["scalars"][f] = {"conditional": cond, "penalized": pen}
+
+ for f in BOOL_FIELDS:
+ vals = [r.get(f) for r in ok if r.get(f) is not None]
+ cond = (float(np.mean(vals)) if vals else None)
+ pen_vals = [bool(r.get(f)) if r.get("status") == "OK" else False
+ for r in rows]
+ out["booleans"][f] = {
+ "conditional_rate": cond,
+ "penalized_rate": float(np.mean(pen_vals)) if pen_vals else None,
+ "n_true_conditional": int(sum(1 for v in vals if v)),
+ "n_conditional": len(vals),
+ }
+
+ from collections import Counter
+ out["status_counts"] = dict(Counter(r.get("status") for r in rows))
+ return out
+
+
+# --------------------------------------------------------------------------- #
+# selftest
+# --------------------------------------------------------------------------- #
+def _sphere(radius=1.0, center=(0, 0, 0), subdiv=3):
+ m = trimesh.creation.icosphere(subdivisions=subdiv, radius=radius)
+ m.apply_translation(np.asarray(center, dtype=np.float64))
+ return m
+
+
+def selftest(eps_frac=0.01, up=1, gmaf=1e-3, mcs=100_000, verbose=True):
+ """Hand-built meshes with known answers. Returns (n_pass, n_fail, lines)."""
+ lines, npass, nfail = [], 0, 0
+
+ def check(name, got, want, tol=None):
+ nonlocal npass, nfail
+ if tol is None:
+ ok = (got == want)
+ else:
+ ok = (got is not None) and abs(got - want) <= tol
+ npass, nfail = npass + int(ok), nfail + int(not ok)
+ g = f"{got:.4f}" if isinstance(got, float) else str(got)
+ w = f"{want:.4f}" if isinstance(want, float) else str(want)
+ lines.append(f" [{'PASS' if ok else 'FAIL'}] {name}: got={g} want={w}"
+ + (f" (tol {tol})" if tol else ""))
+ return ok
+
+ def score(m, tag):
+ lines.append(f"\n--- {tag} ---")
+ r = score_mesh_object(m.copy(), eps_frac, up, gmaf, mcs)
+ lines.append(f" n_comp={r['n_components']} "
+ f"lcvf={r['largest_component_volume_fraction']} "
+ f"lcaf={r['largest_component_area_fraction']} "
+ f"float_rate={r['floating_part_rate']} "
+ f"wt={r['is_watertight']} euler={r['euler_number']} "
+ f"nm={r['nonmanifold_edge_rate']:.4f} "
+ f"ground={r['ground_contact']} "
+ f"gcaf={r['ground_contact_area_frac']:.4f} "
+ f"com_ok={r['com_over_support']} "
+ f"margin={r['com_support_margin']}")
+ return r
+
+ # 1. Single sphere: one component, watertight, no floaters, euler 2.
+ r = score(_sphere(1.0), "single sphere")
+ check("n_components", r["n_components"], 1)
+ check("is_single_component", r["is_single_component"], True)
+ check("is_watertight", r["is_watertight"], True)
+ check("euler_number", r["euler_number"], 2)
+ check("largest_component_volume_fraction",
+ r["largest_component_volume_fraction"], 1.0, 1e-9)
+ check("largest_component_area_fraction",
+ r["largest_component_area_fraction"], 1.0, 1e-9)
+ check("floating_part_rate", r["floating_part_rate"], 0.0, 1e-9)
+ check("nonmanifold_edge_rate", r["nonmanifold_edge_rate"], 0.0, 1e-9)
+ # A sphere is tangent to its ground plane at a point, but within the eps
+ # band the contact patch is a cap of radius ~sqrt(2*R*eps), i.e. ~5% of the
+ # horizontal bbox area at eps_frac=1%. So it passes -- correctly, a ball
+ # does rest on the floor. ground_contact is a *degeneracy* filter, not a
+ # flatness test; see the cone case below for what it actually rejects.
+ check("ground_contact (sphere rests on floor)", r["ground_contact"], True)
+
+ # 1b. Cone balanced on its apex: the genuinely degenerate base. Footprint
+ # radius grows only linearly in eps, so the area fraction stays ~2e-4.
+ cone = trimesh.creation.cone(radius=1.0, height=2.0, sections=64)
+ cone.apply_transform(trimesh.transformations.rotation_matrix(
+ np.pi / 2, [1, 0, 0])) # apex down along -y
+ r = score(cone, "cone balanced on its apex")
+ check("ground_contact (apex only)", r["ground_contact"], False)
+
+ # 1c. Same cone, flipped to sit on its base.
+ cone2 = trimesh.creation.cone(radius=1.0, height=2.0, sections=64)
+ cone2.apply_transform(trimesh.transformations.rotation_matrix(
+ -np.pi / 2, [1, 0, 0])) # base down
+ r = score(cone2, "cone sitting on its base")
+ check("ground_contact (flat base)", r["ground_contact"], True)
+ check("com_over_support (cone on base)", r["com_over_support"], True)
+
+ # 2. Two equal disjoint spheres, far apart: 2 components, both floating,
+ # largest volume fraction 0.5.
+ a, b = _sphere(1.0, (-3, 0, 0)), _sphere(1.0, (3, 0, 0))
+ r = score(trimesh.util.concatenate([a, b]), "two disjoint equal spheres")
+ check("n_components", r["n_components"], 2)
+ check("is_single_component", r["is_single_component"], False)
+ check("largest_component_volume_fraction",
+ r["largest_component_volume_fraction"], 0.5, 1e-6)
+ check("largest_component_area_fraction",
+ r["largest_component_area_fraction"], 0.5, 1e-6)
+ check("floating_part_rate", r["floating_part_rate"], 1.0, 1e-9)
+ check("floating_area_fraction", r["floating_area_fraction"], 1.0, 1e-9)
+ check("is_watertight (2 closed spheres)", r["is_watertight"], True)
+ check("euler_number (2 spheres)", r["euler_number"], 4)
+
+ # 3. Unequal disjoint spheres r=1 and r=0.5: volume fraction 8/9.
+ a, b = _sphere(1.0, (-3, 0, 0)), _sphere(0.5, (3, 0, 0))
+ r = score(trimesh.util.concatenate([a, b]), "disjoint spheres r=1, r=0.5")
+ check("n_components", r["n_components"], 2)
+ check("largest_component_volume_fraction",
+ r["largest_component_volume_fraction"], 8.0 / 9.0, 2e-3)
+ check("largest_component_area_fraction",
+ r["largest_component_area_fraction"], 4.0 / 5.0, 2e-3)
+
+ # 4. Sphere floating above a box: the reviewer's failure mode.
+ # Box 4x1x4 sitting at y in [0,1]; sphere r=1 centred at y=2.6, so its
+ # bottom is 0.6 above the box -> ~12% of the bbox diagonal, far above eps.
+ box = trimesh.creation.box(extents=(4, 1, 4))
+ box.apply_translation([0, 0.5, 0])
+ sph = _sphere(1.0, (0, 2.6, 0))
+ r = score(trimesh.util.concatenate([box, sph]), "sphere FLOATING above box")
+ check("n_components", r["n_components"], 2)
+ check("floating_part_rate", r["floating_part_rate"], 1.0, 1e-9)
+ check("ground_contact (box base)", r["ground_contact"], True)
+ check("com_over_support", r["com_over_support"], True)
+
+ # 5. Same box + sphere, but the sphere is embedded in the box top.
+ # Still 2 topological components, but 0 floating parts.
+ sph = _sphere(1.0, (0, 1.9, 0)) # bottom at y=0.9, inside the box
+ r = score(trimesh.util.concatenate([box, sph]), "sphere RESTING on box")
+ check("n_components", r["n_components"], 2)
+ check("floating_part_rate", r["floating_part_rate"], 0.0, 1e-9)
+ check("ground_contact", r["ground_contact"], True)
+ check("com_over_support", r["com_over_support"], True)
+
+ # 6. Static instability: a tall thin column whose mass hangs off the side of
+ # a small base. COM must fall outside the support polygon.
+ base = trimesh.creation.box(extents=(0.4, 0.2, 0.4))
+ base.apply_translation([0, 0.1, 0])
+ slab = trimesh.creation.box(extents=(3.0, 0.2, 0.4))
+ slab.apply_translation([1.6, 0.3, 0]) # cantilevered far to +x
+ r = score(trimesh.util.concatenate([base, slab]), "cantilever (unstable)")
+ check("ground_contact", r["ground_contact"], True)
+ check("com_over_support (should be False)", r["com_over_support"], False)
+ check("com_support_margin < 0", bool(r["com_support_margin"] < 0), True)
+
+ # 7. Open sheet: a single square, no volume. Volume fraction must fall back
+ # gracefully and area fraction must still be 1.0.
+ plane = trimesh.Trimesh(
+ vertices=np.array([[0, 0, 0], [1, 0, 0], [1, 0, 1], [0, 0, 1]],
+ dtype=np.float64),
+ faces=np.array([[0, 1, 2], [0, 2, 3]]))
+ r = score(plane, "single open sheet (flat quad)")
+ check("n_components", r["n_components"], 1)
+ check("is_watertight", r["is_watertight"], False)
+ check("largest_component_area_fraction",
+ r["largest_component_area_fraction"], 1.0, 1e-9)
+ check("boundary_edge_rate", r["boundary_edge_rate"], 4.0 / 5.0, 1e-9)
+ check("ground_contact (flat sheet is its own base)", r["ground_contact"], True)
+
+ # 8. Non-manifold: three squares sharing one edge (a "T" / fin).
+ v = np.array([[0, 0, 0], [1, 0, 0], # shared edge 0-1
+ [0, 1, 0], [1, 1, 0],
+ [0, -1, 0], [1, -1, 0],
+ [0, 0, 1], [1, 0, 1]], dtype=np.float64)
+ f = np.array([[0, 1, 3], [0, 3, 2],
+ [0, 1, 5], [0, 5, 4],
+ [0, 1, 7], [0, 7, 6]])
+ r = score(trimesh.Trimesh(vertices=v, faces=f), "three fins sharing an edge")
+ check("n_components (fins are one object)", r["n_components"], 1)
+ check("n_nonmanifold_edges >= 1", bool(r["n_nonmanifold_edges"] >= 1), True)
+ check("is_watertight", r["is_watertight"], False)
+
+ # 9. Fragmentation sweep: 1 big sphere + N tiny far-apart spheres. The volume
+ # fraction must stay near 1 while n_components explodes -- this is the
+ # property that makes largest_component_volume_fraction more honest than
+ # a raw component count.
+ parts = [_sphere(1.0, (0, 0, 0))]
+ for i in range(20):
+ parts.append(_sphere(0.05, (3 + 0.3 * i, 0, 0), subdiv=1))
+ r = score(trimesh.util.concatenate(parts), "1 big + 20 tiny spheres")
+ check("n_components", r["n_components"], 21)
+ check("largest_component_volume_fraction > 0.99",
+ bool(r["largest_component_volume_fraction"] > 0.99), True)
+
+ # 10. eps sensitivity: two spheres separated by a hair (0.2% of bbox diag)
+ # are "in contact" at eps=1%, and "floating" at eps=0.1%.
+ gap = 0.01
+ a = _sphere(1.0, (-1 - gap / 2, 0, 0))
+ b = _sphere(1.0, (1 + gap / 2, 0, 0))
+ near = trimesh.util.concatenate([a, b])
+ r1 = score_mesh_object(near.copy(), 0.01, up, gmaf, mcs)
+ r2 = score_mesh_object(near.copy(), 0.0005, up, gmaf, mcs)
+ lines.append(f"\n--- eps sensitivity (gap={gap}) ---")
+ lines.append(f" eps_frac=0.01 -> floating_part_rate={r1['floating_part_rate']}")
+ lines.append(f" eps_frac=0.0005 -> floating_part_rate={r2['floating_part_rate']}")
+ check("near-touching spheres joined at eps=1%",
+ r1["floating_part_rate"], 0.0, 1e-9)
+ check("near-touching spheres split at eps=0.05%",
+ r2["floating_part_rate"], 1.0, 1e-9)
+
+ lines.append(f"\n{npass} passed, {nfail} failed")
+ if verbose:
+ print("\n".join(lines))
+ return npass, nfail, lines
+
+
+# --------------------------------------------------------------------------- #
+# main
+# --------------------------------------------------------------------------- #
+def list_instances(root, override=None):
+ insts = sorted(d.name for d in root.iterdir()
+ if d.is_dir() and not d.name.startswith("_"))
+ if override:
+ keep = set(override)
+ insts = [i for i in insts if i in keep]
+ return insts
+
+
+def main(argv=None):
+ args = parse_args(argv)
+
+ if args.selftest:
+ npass, nfail, _ = selftest(eps_frac=args.eps_frac,
+ up=UP_AXIS_INDEX[args.up_axis],
+ gmaf=args.ground_min_area_frac,
+ mcs=args.max_contact_samples)
+ return 0 if nfail == 0 else 1
+
+ ref_root = args.reference_dir
+ res_root = args.results_dir
+ reference_only = res_root is None
+
+ if reference_only:
+ if not ref_root or not ref_root.exists():
+ raise SystemExit(f"No such reference dir: {ref_root}")
+ insts = list_instances(ref_root, args.instances)
+ subject = f"REFERENCE {ref_root}"
+ out_default = ref_root / "_metrics" / "structural_integrity.jsonl"
+ else:
+ if not res_root.exists():
+ raise SystemExit(f"No such results dir: {res_root}")
+ insts = list_instances(res_root, args.instances)
+ subject = f"{res_root}"
+ out_default = res_root / "_metrics" / "structural_integrity.jsonl"
+
+ if args.limit:
+ insts = insts[:args.limit]
+ if not insts:
+ raise SystemExit("No instances.")
+
+ out_path = args.out or out_default
+ out_path.parent.mkdir(parents=True, exist_ok=True)
+ sum_path = out_path.with_suffix(".json")
+
+ up = UP_AXIS_INDEX[args.up_axis]
+ jobs = []
+ for inst in insts:
+ ref_glb = (ref_root / inst / "glb" / f"{inst}.glb") if ref_root else None
+ gen_glb = None if reference_only else (res_root / inst / "glb" / f"{inst}.glb")
+ jobs.append((inst,
+ str(ref_glb) if ref_glb else None,
+ str(gen_glb) if gen_glb else None,
+ args.eps_frac, up, args.ground_min_area_frac,
+ args.max_contact_samples))
+
+ # Big meshes first: they dominate wall-clock, and starting them early keeps
+ # the tail short. Peak RSS is bounded by --workers, so keep it modest.
+ def _sz(j):
+ p = j[2] or j[1]
+ try:
+ return -os.path.getsize(p)
+ except Exception:
+ return 0
+ jobs.sort(key=_sz)
+
+ print(f"structural_integrity: {subject}")
+ print(f" instances={len(jobs)} workers={args.workers} "
+ f"eps_frac={args.eps_frac} up_axis={args.up_axis} "
+ f"ground_min_area_frac={args.ground_min_area_frac}")
+
+ rows, t0 = [], time.time()
+ if args.workers <= 1:
+ for i, j in enumerate(jobs):
+ rows.append(_worker(j))
+ print(f" [{i+1}/{len(jobs)}] {rows[-1]['instance']} "
+ f"{rows[-1]['status']} {rows[-1]['seconds']}s", flush=True)
+ else:
+ with ProcessPoolExecutor(max_workers=args.workers) as ex:
+ futs = {ex.submit(_worker, j): j[0] for j in jobs}
+ for i, fut in enumerate(as_completed(futs)):
+ try:
+ rows.append(fut.result())
+ except Exception as e:
+ rows.append({"instance": futs[fut],
+ "status": f"WORKER_FAIL:{e}"})
+ if (i + 1) % 10 == 0 or i == len(jobs) - 1:
+ ok = sum(1 for r in rows if r.get("status") == "OK")
+ print(f" [{i+1}/{len(jobs)}] ok={ok} "
+ f"elapsed={time.time()-t0:.1f}s", flush=True)
+
+ rows.sort(key=lambda r: r["instance"])
+ with out_path.open("w") as fh:
+ for r in rows:
+ fh.write(json.dumps(r) + "\n")
+
+ summary = aggregate(rows)
+ summary.update({
+ "subject": subject,
+ "mode": "reference_only" if reference_only else "results",
+ "results_dir": str(res_root) if res_root else None,
+ "reference_dir": str(ref_root) if ref_root else None,
+ "eps_frac": args.eps_frac,
+ "up_axis": args.up_axis,
+ "ground_min_area_frac": args.ground_min_area_frac,
+ "normalization": "unit_sphere(area_weighted_centroid)",
+ "weld": "merge_vertices(merge_tex=True, merge_norm=True) post-normalize",
+ "component_definition": "vertex-graph connected components",
+ "contact_test": "eps-grid, 26-neighbourhood (detects <=eps, may accept <=sqrt(3)eps)",
+ "jsonl": str(out_path),
+ "elapsed_s": round(time.time() - t0, 1),
+ })
+ sum_path.write_text(json.dumps(summary, indent=2))
+
+ # stdout
+ def f2(x, d=3):
+ return " -- " if x is None else f"{x:.{d}f}"
+ s = summary["scalars"]
+ b = summary["booleans"]
+ print()
+ print(f"=== structural integrity — {subject} ===")
+ print(f" instances={summary['n_instances']} ok={summary['n_ok']} "
+ f"({summary['ok_rate']:.1%})" if summary["ok_rate"] is not None else "")
+ print(f" status: {summary['status_counts']}")
+ for f in ["n_components", "largest_component_volume_fraction",
+ "largest_component_area_fraction", "floating_part_rate",
+ "floating_area_fraction", "nonmanifold_edge_rate",
+ "boundary_edge_rate", "ground_contact_area_frac",
+ "com_support_margin"]:
+ c = s[f]["conditional"]
+ p = s[f]["penalized"]
+ if c is None:
+ continue
+ print(f" {f:38s} cond mean={f2(c['mean'],4):>10} med={f2(c['median'],4):>10}"
+ f" pen mean={f2(p['mean'],4) if p else ' -- ':>10}")
+ for f in BOOL_FIELDS:
+ print(f" {f:38s} cond rate={f2(b[f]['conditional_rate']):>8}"
+ f" pen rate={f2(b[f]['penalized_rate']):>8}")
+ print(f" Wrote {out_path}")
+ print(f" Wrote {sum_path}")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/eval/llm/suites.py b/eval/llm/suites.py
new file mode 100644
index 00000000..4f299322
--- /dev/null
+++ b/eval/llm/suites.py
@@ -0,0 +1,124 @@
+"""Suite registry + prompt loading.
+
+A *suite* is one prompt file under data/prompts/<name>.jsonl plus how to run and score it. Everything a
+runner needs is derived from the rows themselves (dialect, images, reference), so adding a suite = adding a
+JSONL; the table below only pins per-suite decoding defaults and which metrics apply.
+
+Decoding defaults follow finetune/docs/REPORT.md §13.4: greedy for Blender/CadQuery, and for the
+long-output dialects (OpenSCAD / GLSL / three.js) sampling at T=0.7 is the *primary* protocol because
+greedy decoding degenerates into runaway loops there; run both when you can afford it.
+"""
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from . import config
+
+
+@dataclass(frozen=True)
+class SuiteSpec:
+ name: str
+ dialect: str
+ kind: str # "gt" (reference mesh), "exec" (execution only), "rubric" (no GT, judge optional)
+ temperature: float = 0.0
+ max_new_tokens: int = 8192
+ n_points: int = 10000
+ metrics: tuple[str, ...] = ("exec",)
+ description: str = ""
+ tags: tuple[str, ...] = field(default_factory=tuple)
+
+
+SUITES: dict[str, SuiteSpec] = {
+ "3dcodebench_text": SuiteSpec("3dcodebench_text", "blender", "gt", 0.0, 8192, 10000, ("exec", "geometry"),
+ "212 Infinigen-family objects, structured instruction → bpy; GT mesh", ("headline", "text")),
+ "3dcodebench_desc": SuiteSpec("3dcodebench_desc", "blender", "gt", 0.0, 8192, 10000, ("exec", "geometry"),
+ "same 212 objects, short natural description", ("text",)),
+ "3dcodebench_img1": SuiteSpec("3dcodebench_img1", "blender", "gt", 0.0, 8192, 10000, ("exec", "geometry"),
+ "image→code: 1 GT view", ("vlm",)),
+ "3dcodebench_img4": SuiteSpec("3dcodebench_img4", "blender", "gt", 0.0, 8192, 10000, ("exec", "geometry"),
+ "image→code: 4 GT views", ("vlm",)),
+ "3dcodebench_img4text": SuiteSpec("3dcodebench_img4text", "blender", "gt", 0.0, 8192, 10000, ("exec", "geometry"),
+ "image+text→code: 4 GT views + description", ("vlm",)),
+ # the official 3DCodeBench protocol (paper arXiv:2606.01057): raw-python contract, T=0.7, 16k budget,
+ # exec + SigLIP-2 / DINOv3 view similarity + squared unit-sphere Chamfer, conditional and penalized
+ "3dcodebench_official_text": SuiteSpec("3dcodebench_official_text", "blender", "gt", 0.7, 16384, 8192,
+ ("exec", "geometry", "chamfer_official", "image_sim"),
+ "official text_to_3D: description prompt, official system prompt, official metrics", ("official", "text")),
+ "3dcodebench_official_img4": SuiteSpec("3dcodebench_official_img4", "blender", "gt", 0.7, 16384, 8192,
+ ("exec", "geometry", "chamfer_official", "image_sim"),
+ "official image_to_3D: 4 GT views, official system prompt, official metrics", ("official", "vlm")),
+ "heldout_blender": SuiteSpec("heldout_blender", "blender", "gt", 0.0, 8192, 5000, ("exec", "geometry"),
+ "103 held-out bioinspired3d/non-test-factory prompts; GT = executed reference", ("heldout",)),
+ "heldout_cadquery": SuiteSpec("heldout_cadquery", "cadquery", "gt", 0.0, 8192, 5000, ("exec", "geometry"),
+ "200 held-out DeepCAD/Articraft prompts; GT = executed reference", ("heldout",)),
+ "heldout_openscad": SuiteSpec("heldout_openscad", "openscad", "gt", 0.7, 8192, 5000, ("exec", "geometry"),
+ "50 held-out Thingiverse prompts; GT = executed reference", ("heldout", "long-output")),
+ "heldout_glsl": SuiteSpec("heldout_glsl", "glsl", "exec", 0.7, 8192, 0, ("exec", "render"),
+ "200 held-out Shadertoy prompts; compile + WebGL render check", ("heldout", "long-output")),
+ "heldout_threejs": SuiteSpec("heldout_threejs", "threejs", "exec", 0.7, 12288, 0, ("exec",),
+ "40 held-out single-file three.js pages; headless-Chromium check", ("heldout", "long-output")),
+}
+
+
+def _battery_specs() -> dict[str, SuiteSpec]:
+ out = {}
+ for p in sorted(config.PROMPTS_DIR.glob("harness_*.jsonl")):
+ name = p.stem
+ with p.open() as f:
+ first = json.loads(f.readline())
+ dialect = first["dialect"]
+ out[name] = SuiteSpec(name, dialect, "rubric", 0.7 if dialect in ("glsl", "threejs") else 0.0, 8192, 0,
+ ("exec", "render") if dialect == "glsl" else ("exec",),
+ f"harness battery {name}: execution + optional VLM judge, no GT", ("rubric",))
+ return out
+
+
+def all_suites() -> dict[str, SuiteSpec]:
+ return {**SUITES, **_battery_specs()}
+
+
+def get_suite(name: str) -> SuiteSpec:
+ s = all_suites().get(name)
+ if s is None:
+ raise KeyError(f"unknown suite {name!r}; known: {sorted(all_suites())}")
+ return s
+
+
+def prompt_path(name: str) -> Path:
+ return config.PROMPTS_DIR / f"{name}.jsonl"
+
+
+def load_prompts(name: str, limit: int | None = None) -> list[dict]:
+ p = prompt_path(name)
+ if not p.exists():
+ raise FileNotFoundError(f"{p} — run `python -m llm.build_prompts` first")
+ rows = [json.loads(l) for l in p.open() if l.strip()]
+ return rows[:limit] if limit else rows
+
+
+def resolve(rel: str | None) -> Path | None:
+ return (config.DATA_DIR / rel) if rel else None
+
+
+def expand(names: list[str]) -> list[str]:
+ """Expand suite names / tags: 'all', 'text', 'vlm', 'heldout', 'rubric', 'headline', 'long-output'."""
+ known = all_suites()
+ out: list[str] = []
+ for n in names:
+ if n == "all":
+ out += [k for k, s in known.items() if "rubric" not in s.tags]
+ elif n in known:
+ out.append(n)
+ else:
+ hit = [k for k, s in known.items() if n in s.tags]
+ if not hit:
+ raise KeyError(f"unknown suite or tag {n!r}")
+ out += hit
+ seen, uniq = set(), []
+ for n in out:
+ if n not in seen:
+ seen.add(n)
+ uniq.append(n)
+ return uniq
diff --git a/eval/llm/tests/test_smoke.py b/eval/llm/tests/test_smoke.py
new file mode 100644
index 00000000..cb112e6a
--- /dev/null
+++ b/eval/llm/tests/test_smoke.py
@@ -0,0 +1,84 @@
+"""Offline smoke tests: extraction, metrics arithmetic, suite registry, and (if tools exist) one executor each.
+
+ ~/miniconda3/envs/cv3d-eval/bin/python -m pytest llm/tests -q
+"""
+from __future__ import annotations
+
+import importlib
+import math
+import tempfile
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+PKG = "llm"
+extract = importlib.import_module(f"{PKG}.extract")
+metrics = importlib.import_module(f"{PKG}.metrics")
+suites = importlib.import_module(f"{PKG}.suites")
+config = importlib.import_module(f"{PKG}.config")
+executors = importlib.import_module(f"{PKG}.executors")
+
+
+def test_extract_comment_first_line_not_penalised():
+ text = "```python\n# build a cube\nimport bpy\nbpy.ops.mesh.primitive_cube_add()\n```"
+ code, meta = extract.extract(text, "blender")
+ assert code.startswith("# build a cube") and meta["valid"] and meta["score"] >= 10
+
+
+def test_extract_prefers_last_dialect_block():
+ text = "```bash\npip install cadquery\n```\n```python\nimport cadquery as cq\nresult = cq.Workplane('XY').box(1,2,3)\n```"
+ code, meta = extract.extract(text, "cadquery")
+ assert "Workplane" in code and meta["chosen"] == 1
+
+
+def test_pass_at_k():
+ assert metrics.pass_at_k(4, 0, 1) == 0.0
+ assert metrics.pass_at_k(4, 4, 1) == 1.0
+ assert math.isclose(metrics.pass_at_k(4, 1, 1), 0.25)
+ assert math.isclose(metrics.pass_at_k(4, 1, 4), 1.0)
+
+
+def test_chamfer_identity():
+ p = np.random.default_rng(0).random((500, 3))
+ cd, fs = metrics.chamfer_f(p, p)
+ assert cd == 0.0 and fs["[email protected]"] == 1.0
+
+
+def test_suite_registry():
+ s = suites.get_suite("3dcodebench_text")
+ assert s.dialect == "blender" and "geometry" in s.metrics
+ assert "heldout_glsl" in suites.expand(["long-output"])
+
+
[email protected](config.find_blender() is None, reason="no blender")
+def test_blender_executor_empty_vs_ok():
+ with tempfile.TemporaryDirectory() as d:
+ rep = executors.run("blender", "import bpy\nbpy.ops.mesh.primitive_cube_add()\n", d + "/ok")
+ assert rep["status"] == "OK" and rep["mesh"]
+ rep = executors.run("blender", "import bpy\n", d + "/empty")
+ assert rep["status"] == "EMPTY"
+ rep = executors.run("blender", "import bpy\nbpy.ops.scene.clear()\n", d + "/fail")
+ assert rep["status"] == "FAIL"
+
+
+def test_cadquery_executor():
+ pytest.importorskip("cadquery")
+ with tempfile.TemporaryDirectory() as d:
+ rep = executors.run("cadquery", "import cadquery as cq\nresult = cq.Workplane('XY').box(1,1,1)\n", d)
+ assert rep["status"] == "OK" and Path(rep["mesh"]).exists()
+
+
[email protected](config.find_openscad() is None, reason="no openscad")
+def test_openscad_executor():
+ with tempfile.TemporaryDirectory() as d:
+ assert executors.run("openscad", "cube(10);", d)["status"] == "OK"
+
+
[email protected](config.find_glslang() is None, reason="no glslang")
+def test_glsl_executor_compile_only():
+ with tempfile.TemporaryDirectory() as d:
+ rep = executors.glsl.run("void mainImage(out vec4 c, in vec2 f){ c = vec4(f/iResolution.xy, 0.5+0.5*sin(iTime), 1.); }", d, render=False)
+ assert rep["status"] == "OK"
+ rep = executors.glsl.run("void mainImage(out vec4 c, in vec2 f){ c = vec4(1.0) }", d + "/bad", render=False)
+ assert rep["status"] == "FAIL"
diff --git a/eval/pyproject.toml b/eval/pyproject.toml
new file mode 100644
index 00000000..32b00681
--- /dev/null
+++ b/eval/pyproject.toml
@@ -0,0 +1,23 @@
+# Tooling config for the evaluation code. Nothing here is installed: `bench` and `llm` are run
+# from this directory (`cd eval && python bench/<script>.py …`, `cd eval && python -m pytest`).
+[tool.ruff]
+line-length = 100
+target-version = "py313"
+extend-exclude = ["bench/out", "llm"] # llm/ keeps its own style; bench/out is run data
+
+[tool.ruff.lint]
+select = ["E", "W", "F", "I", "B", "UP", "SIM"]
+ignore = ["E501"]
+
+[tool.ruff.lint.isort]
+known-first-party = ["bench", "codeverse", "tests"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+timeout = 600
+addopts = ["-m", "not live", "-n", "auto", "--dist", "worksteal"]
+markers = [
+ "live: needs network + API keys (deselect with -m 'not live')",
+ "blender: needs a Blender binary",
+ "node: needs node + runtime_js deps",
+]
diff --git a/harness/tests/compare_bench/__init__.py b/eval/tests/__init__.py
similarity index 100%
rename from harness/tests/compare_bench/__init__.py
rename to eval/tests/__init__.py
diff --git a/harness/tests/compare_bench/conftest.py b/eval/tests/conftest.py
similarity index 86%
rename from harness/tests/compare_bench/conftest.py
rename to eval/tests/conftest.py
index 8ebbd269..6af79fca 100644
--- a/harness/tests/compare_bench/conftest.py
+++ b/eval/tests/conftest.py
@@ -8,9 +8,25 @@ from typing import Any
from PIL import Image
-REPO = Path(__file__).resolve().parents[2]
-if str(REPO) not in sys.path:
- sys.path.insert(0, str(REPO))
+REPO = Path(__file__).resolve().parents[1] # eval/ — the `bench` package lives here
+HARNESS = REPO.parent / "harness" # the tree under evaluation: ITS codeverse, not an install
+for _p in (HARNESS, REPO):
+ if str(_p) not in sys.path:
+ sys.path.insert(0, str(_p))
+
+
+def _harness_test_module(rel: str, name: str) -> Any:
+ """A helper module of the harness's own test suite, loaded by path: `tests` here is
+ eval/tests, so `tests.flywheel_cli.conftest` cannot be imported by name."""
+ import importlib.util
+
+ spec = importlib.util.spec_from_file_location(name, HARNESS / "tests" / rel)
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+make_fake_run = _harness_test_module("flywheel_cli/conftest.py", "_harness_flywheel_conftest").make_fake_run
from bench._fixed_eval import EvalOutcome # noqa: E402
from bench._oneshot import MODEL_FILE, OneShotResult # noqa: E402
diff --git a/harness/tests/compare_bench/test_ab_plan.py b/eval/tests/test_ab_plan.py
similarity index 99%
rename from harness/tests/compare_bench/test_ab_plan.py
rename to eval/tests/test_ab_plan.py
index 210ad26b..79b423d2 100644
--- a/harness/tests/compare_bench/test_ab_plan.py
+++ b/eval/tests/test_ab_plan.py
@@ -8,7 +8,7 @@ from pathlib import Path
import pytest
-REPO = Path(__file__).resolve().parents[2]
+REPO = Path(__file__).resolve().parents[1]
if str(REPO) not in sys.path:
sys.path.insert(0, str(REPO))
diff --git a/eval/tests/test_ab_plan_switches.py b/eval/tests/test_ab_plan_switches.py
new file mode 100644
index 00000000..3a118614
--- /dev/null
+++ b/eval/tests/test_ab_plan_switches.py
@@ -0,0 +1,37 @@
+"""`bench/ab_plan.py` refuses an A/B whose only switch nothing reads (the switch registry is
+`codeverse/tracks/plan_features.py`, tested with the harness)."""
+
+from __future__ import annotations
+
+import pytest
+
+
+def test_ab_plan_refuses_an_ab_whose_only_switch_is_dead(capsys):
+ """`--variant-env CV3D_PLAN_FEATURES=all` produced a full battery and the verdict
+ 'keep, mean delta +0.344' for two byte-identical arms. It must not start."""
+ import bench.ab_plan as A
+
+ with pytest.raises(SystemExit):
+ A.main(["--prompts", "p.yaml", "--out", "o", "--variant-env", "CV3D_PLAN_FEATURES=all"])
+ err = capsys.readouterr().err
+ assert "nothing reads" in err and "CV3D_PLAN_FEATURES" in err
+
+
+def test_ab_plan_still_accepts_a_live_switch(monkeypatch, capsys):
+ """The guard must not block a real A/B: it fires only when EVERY key is dead.
+
+ Offline — --no-preflight and --allow-siblings keep the provider health check and the
+ docs/COST.md §23 admission check out of it, and run_ab itself is stubbed."""
+ import bench.ab_plan as A
+
+ seen: list[dict[str, str]] = []
+ class _V:
+ decision, reason, caution = "keep", "stubbed", ""
+
+ monkeypatch.setattr(A, "run_ab", lambda battery, out, opts, **kw: (seen.append(opts.variant_env), _V())[1])
+ for argv in (["--variant-env", "CV3D_PLAN_BRIEF=off"],
+ ["--variant-env", "CV3D_PLAN_FEATURES=all", "--variant-env", "CV3D_PLAN_BRIEF=off"]):
+ A.main(["--prompts", "p.yaml", "--out", "o", "--no-preflight", "--allow-siblings", *argv])
+
+ assert len(seen) == 2, capsys.readouterr()
+ assert "CV3D_PLAN_BRIEF" in seen[0]
diff --git a/harness/tests/compare_bench/test_articulated_compare.py b/eval/tests/test_articulated_compare.py
similarity index 99%
rename from harness/tests/compare_bench/test_articulated_compare.py
rename to eval/tests/test_articulated_compare.py
index 49a66734..40f2da63 100644
--- a/harness/tests/compare_bench/test_articulated_compare.py
+++ b/eval/tests/test_articulated_compare.py
@@ -7,7 +7,7 @@ from pathlib import Path
import pytest
-REPO = Path(__file__).resolve().parents[2]
+REPO = Path(__file__).resolve().parents[1]
if str(REPO) not in sys.path:
sys.path.insert(0, str(REPO))
diff --git a/harness/tests/flywheel_cli/test_bench.py b/eval/tests/test_bench.py
similarity index 99%
rename from harness/tests/flywheel_cli/test_bench.py
rename to eval/tests/test_bench.py
index ff53c88e..ac49de9e 100644
--- a/harness/tests/flywheel_cli/test_bench.py
+++ b/eval/tests/test_bench.py
@@ -11,9 +11,9 @@ from bench.report import build_report, load_results
from bench.run_bench import Battery, BenchOptions, run_battery
from codeverse.contracts.run import RunRecord, RunStatus
from codeverse.workspace import Workspace
-from tests.flywheel_cli.conftest import make_fake_run
+from tests.conftest import make_fake_run
-REPO = Path(__file__).resolve().parents[2]
+REPO = Path(__file__).resolve().parents[1]
BATTERIES = sorted((REPO / "bench" / "prompts").glob("*.yaml"))
diff --git a/harness/tests/compare_bench/test_bench_refs.py b/eval/tests/test_bench_refs.py
similarity index 96%
rename from harness/tests/compare_bench/test_bench_refs.py
rename to eval/tests/test_bench_refs.py
index 58ad5ec7..d49d6f76 100644
--- a/harness/tests/compare_bench/test_bench_refs.py
+++ b/eval/tests/test_bench_refs.py
@@ -7,7 +7,7 @@ from pathlib import Path
import yaml
-sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "bench"))
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "bench"))
from run_bench import Battery, BenchPrompt, discover_references # noqa: E402
from codeverse.contracts.common import Track # noqa: E402
diff --git a/harness/tests/compare_bench/test_bench_shared.py b/eval/tests/test_bench_shared.py
similarity index 99%
rename from harness/tests/compare_bench/test_bench_shared.py
rename to eval/tests/test_bench_shared.py
index cb474874..0e7a8ecf 100644
--- a/harness/tests/compare_bench/test_bench_shared.py
+++ b/eval/tests/test_bench_shared.py
@@ -10,7 +10,7 @@ import pytest
import bench.compare_backends as cb
import bench.run_bench as rb
from bench.run_bench import Battery, BenchOptions, build_spec, select_prompts, spec_for
-from tests.compare_bench.conftest import BATTERY
+from tests.conftest import BATTERY
def _battery() -> Battery:
diff --git a/harness/tests/compare_bench/test_bestofk_report.py b/eval/tests/test_bestofk_report.py
similarity index 100%
rename from harness/tests/compare_bench/test_bestofk_report.py
rename to eval/tests/test_bestofk_report.py
diff --git a/harness/tests/compare_bench/test_compare_matrix.py b/eval/tests/test_compare_matrix.py
similarity index 99%
rename from harness/tests/compare_bench/test_compare_matrix.py
rename to eval/tests/test_compare_matrix.py
index babdd67c..e481c2a8 100644
--- a/harness/tests/compare_bench/test_compare_matrix.py
+++ b/eval/tests/test_compare_matrix.py
@@ -8,7 +8,7 @@ from pathlib import Path
import pytest
-REPO = Path(__file__).resolve().parents[2]
+REPO = Path(__file__).resolve().parents[1]
if str(REPO) not in sys.path:
sys.path.insert(0, str(REPO))
@@ -30,7 +30,7 @@ from bench.compare_backends import ( # noqa: E402
spec_for,
)
from bench.run_bench import Battery # noqa: E402
-from tests.compare_bench.conftest import ( # noqa: E402
+from tests.conftest import ( # noqa: E402
BAD,
BATTERY,
GOOD,
diff --git a/eval/tests/test_complexity_report.py b/eval/tests/test_complexity_report.py
new file mode 100644
index 00000000..f5b91d5a
--- /dev/null
+++ b/eval/tests/test_complexity_report.py
@@ -0,0 +1,121 @@
+"""`bench/complexity_report.py`: does score rise or fall with complexity, and what does a
+complexity point cost? (The block a record carries is tested with the harness.)"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from bench import complexity_report as CR
+from codeverse.contracts.artifacts import Measurement
+
+REPO = Path(__file__).resolve().parents[1]
+
+
+def _measurement(index: float, parts: int = 8) -> Measurement:
+ return Measurement(
+ bbox_min=(0, 0, 0), bbox_max=(1, 1, 1), extents=(1, 1, 1), center=(0.5, 0.5, 0.5),
+ tri_count=1000, n_meshes=parts, n_islands=parts,
+ extra={"complexity": {"version": 1, "index": index, "band": "moderate", "part_count": parts,
+ "tri_count": 1000, "materials": 3, "silhouette": 4.0,
+ "feature_density": 500.0, "symmetry_groups": 1, "hollowness": 0.2,
+ "assembly_depth": 1, "components": {}, "extra": {}}},
+ )
+
+
+
+# ------------------------------------------------------------------ the report
+def _write_run(root: Path, slug: str, *, index: float, overall: float, detail: float,
+ cost: float = 1.0, minutes: float = 10.0) -> Path:
+ run = root / "runs" / slug
+ (run / "artifacts").mkdir(parents=True)
+ rec = {
+ "spec": {"track": "static_object", "language": "blender", "tags": ["bench", "hard"]},
+ "plan": {"parts": [{"name": "a"}, {"name": "b"}]},
+ "status": "passed", "best_round": 0,
+ "total_usage": {"cost_usd": cost},
+ "telemetry": {"cost": {"wall_clock_s": minutes * 60}},
+ "rounds": [{
+ "index": 0, "gates": [],
+ "measurement": _measurement(index).model_dump(mode="json"),
+ "judgment": {"overall": overall, "passed": overall >= 0.72,
+ "scores": {c: detail for c in CR.CRITERIA}, "issues": []},
+ }],
+ }
+ (run / "record.json").write_text(json.dumps(rec))
+ return run
+
+
+def test_collect_and_report_over_a_battery(tmp_path: Path) -> None:
+ battery = tmp_path / "batt"
+ _write_run(battery, "low", index=0.35, overall=0.85, detail=0.85, cost=0.5, minutes=5)
+ _write_run(battery, "mid", index=0.55, overall=0.70, detail=0.70, cost=1.5, minutes=20)
+ _write_run(battery, "high", index=0.75, overall=0.55, detail=0.55, cost=3.0, minutes=40)
+ rows, skipped = CR.collect([battery])
+ assert skipped == 0 and len(rows) == 3
+ assert {r["slug"] for r in rows} == {"low", "mid", "high"}
+ assert rows[0]["plan_parts"] == 2 and rows[0]["n_materials"] == 3
+ # the whole point of the study: a monotone fall shows up as a strong negative r
+ xs, ys = CR._pairs(rows, "index", "overall")
+ assert CR.pearson(xs, ys) < -0.98
+ assert CR.spearman(xs, ys) == pytest.approx(-1.0)
+ money = CR.dollars_per_point(rows)
+ assert money["runs"] == 3 and money["total_usd"] == pytest.approx(5.0)
+ assert money["usd_per_point"] == pytest.approx(5.0 / (100 * (0.35 + 0.55 + 0.75)), abs=5e-5)
+ report = CR.render_report(rows, skipped)
+ assert "score vs complexity" in report and "cost of complexity" in report
+ assert "| batt/high |" in report
+
+
+def test_battery_expectations_flag_out_of_band_runs(tmp_path: Path) -> None:
+ battery = tmp_path / "batt"
+ _write_run(battery, "cx_a03_stool", index=0.30, overall=0.9, detail=0.9)
+ _write_run(battery, "cx_d25_engine", index=0.40, overall=0.9, detail=0.9)
+ rows, _ = CR.collect([battery])
+ yaml_path = tmp_path / "b.yaml"
+ yaml_path.write_text(
+ "name: b\ntrack: static_object\nlanguage: blender\nprompts:\n"
+ " - id: cx_a03_stool\n tier: easy\n prompt: p\n expected_complexity: [0.25, 0.42]\n"
+ " - id: cx_d25_engine\n tier: hard\n prompt: p\n expected_complexity: [0.68, 0.88]\n"
+ " - id: cx_missing\n tier: hard\n prompt: p\n expected_complexity: [0.5, 0.6]\n"
+ )
+ table = CR.battery_expectations(yaml_path, rows)
+ assert "| cx_a03_stool | 0.25–0.42 | 0.300 | yes |" in table
+ assert "LOW" in table # the engine collapsed to a stool
+ assert "not run" in table # a prompt with no run is still listed
+
+
+def test_real_battery_bands_are_sane() -> None:
+ """complexity_v3 is the ladder: every band inside [0,1], ordered, and the
+ prompts must actually climb."""
+ import yaml
+
+ data = yaml.safe_load((REPO / "bench" / "prompts" / "complexity_v3.yaml").read_text())
+ bands = [(p["id"], p["expected_complexity"]) for p in data["prompts"]]
+ assert len(bands) == 12
+ for pid, (lo, hi) in bands:
+ assert 0.0 < lo < hi <= 1.0, pid
+ lows = [lo for _, (lo, _hi) in bands]
+ assert lows == sorted(lows) and lows[-1] > lows[0] + 0.3
+
+
+def test_row_recomputes_the_vector_when_the_record_has_none(tmp_path: Path) -> None:
+ """The historic corpus predates the vector: the report must fall back to the
+ GLB rather than skipping the run."""
+ import trimesh
+
+ run = tmp_path / "runs" / "old"
+ (run / "artifacts").mkdir(parents=True)
+ trimesh.Scene({"Box": trimesh.creation.box(extents=(0.3, 0.3, 0.3))}).export(
+ str(run / "artifacts" / "object.glb"))
+ (run / "record.json").write_text(json.dumps({
+ "spec": {"track": "static_object", "language": "blender", "tags": []},
+ "plan": {"parts": []}, "status": "passed", "best_round": 0,
+ "total_usage": {"cost_usd": 1.0}, "rounds": [{
+ "index": 0, "gates": [],
+ "judgment": {"overall": 0.5, "passed": False,
+ "scores": dict.fromkeys(CR.CRITERIA, 0.5), "issues": []}}]}))
+ row = CR.row_for(run, "old_battery")
+ assert row is not None and row["part_count"] == 1 and row["index"] > 0
diff --git a/harness/tests/compare_bench/test_degraded.py b/eval/tests/test_degraded.py
similarity index 97%
rename from harness/tests/compare_bench/test_degraded.py
rename to eval/tests/test_degraded.py
index 2c9fa7f9..aa6c0432 100644
--- a/harness/tests/compare_bench/test_degraded.py
+++ b/eval/tests/test_degraded.py
@@ -12,7 +12,7 @@ from pathlib import Path
import pytest
-REPO = Path(__file__).resolve().parents[2]
+REPO = Path(__file__).resolve().parents[1]
if str(REPO) not in sys.path:
sys.path.insert(0, str(REPO))
@@ -27,7 +27,7 @@ from bench.compare_backends import ( # noqa: E402
from bench.run_bench import Battery # noqa: E402
from codeverse.contracts.common import Usage # noqa: E402
from codeverse.contracts.run import RoundRecord, RunRecord, RunStatus # noqa: E402
-from tests.compare_bench.conftest import BATTERY, GOOD, FakeEvaluator # noqa: E402
+from tests.conftest import BATTERY, GOOD, FakeEvaluator # noqa: E402
def _track(status: RunStatus, stop_reason: str, rounds: int, cost: float, aborted: int = 0):
diff --git a/harness/tests/compare_bench/test_fixed_eval_language.py b/eval/tests/test_fixed_eval_language.py
similarity index 100%
rename from harness/tests/compare_bench/test_fixed_eval_language.py
rename to eval/tests/test_fixed_eval_language.py
diff --git a/harness/tests/compare_bench/test_infra_failures.py b/eval/tests/test_infra_failures.py
similarity index 98%
rename from harness/tests/compare_bench/test_infra_failures.py
rename to eval/tests/test_infra_failures.py
index c2b4fe29..63b7a9ab 100644
--- a/harness/tests/compare_bench/test_infra_failures.py
+++ b/eval/tests/test_infra_failures.py
@@ -110,7 +110,7 @@ def test_both_failure_paths_classify_the_same_way(tmp_path):
from bench.compare_backends import CompareDeps, CompareOptions, parse_arm, run_cell
from bench.run_bench import Battery
from codeverse.models.base import ModelError
- from tests.compare_bench.conftest import BATTERY, FakeEvaluator
+ from tests.conftest import BATTERY, FakeEvaluator
def verdicts(outage: Exception, out: Path) -> dict[str, tuple[str, float | None]]:
class DeadModel: # the real ApiOneShot around a model that raises the outage
@@ -204,7 +204,7 @@ def test_a_repair_lost_to_an_outage_drops_the_cell_instead_of_scoring_the_pre_re
from bench.compare_backends import CompareDeps, CompareOptions, parse_arm, run_cell
from bench.run_bench import Battery
from codeverse.contracts.common import Usage
- from tests.compare_bench.conftest import BATTERY, GOOD, FakeEvaluator
+ from tests.conftest import BATTERY, GOOD, FakeEvaluator
broken = GOOD.format(score=0.5).replace("\n", "\n# BOOM\n", 1)
@@ -251,7 +251,7 @@ def test_a_harness_planning_failure_is_a_zero_not_a_dropped_cell(tmp_path):
from bench.compare_backends import CompareDeps, CompareOptions, parse_arm, run_cell
from bench.run_bench import Battery
from codeverse.tracks.planner import PlanningError
- from tests.compare_bench.conftest import BATTERY, FakeEvaluator
+ from tests.conftest import BATTERY, FakeEvaluator
def no_plan(spec, ws, resume):
raise PlanningError("plan did not validate after re-ask: joint X references unknown link(s)")
@@ -288,7 +288,7 @@ def test_a_classifier_crash_still_records_the_cell(tmp_path):
"""End to end: a failure whose classification blows up is still a row on disk."""
from bench import compare_backends as cb
from bench.run_bench import Battery
- from tests.compare_bench.conftest import BATTERY, FakeEvaluator
+ from tests.conftest import BATTERY, FakeEvaluator
def boom(spec, ws, resume):
raise ClassifierTrap("planner died")
diff --git a/harness/tests/compare_bench/test_live_compare.py b/eval/tests/test_live_compare.py
similarity index 97%
rename from harness/tests/compare_bench/test_live_compare.py
rename to eval/tests/test_live_compare.py
index 3f8fb932..76edb9f6 100644
--- a/harness/tests/compare_bench/test_live_compare.py
+++ b/eval/tests/test_live_compare.py
@@ -7,7 +7,7 @@ from pathlib import Path
import pytest
-REPO = Path(__file__).resolve().parents[2]
+REPO = Path(__file__).resolve().parents[1]
if str(REPO) not in sys.path:
sys.path.insert(0, str(REPO))
diff --git a/harness/tests/compare_bench/test_oneshot.py b/eval/tests/test_oneshot.py
similarity index 99%
rename from harness/tests/compare_bench/test_oneshot.py
rename to eval/tests/test_oneshot.py
index 236b36a7..11decb9d 100644
--- a/harness/tests/compare_bench/test_oneshot.py
+++ b/eval/tests/test_oneshot.py
@@ -7,7 +7,7 @@ from pathlib import Path
import pytest
-REPO = Path(__file__).resolve().parents[2]
+REPO = Path(__file__).resolve().parents[1]
if str(REPO) not in sys.path:
sys.path.insert(0, str(REPO))
diff --git a/harness/tests/compare_bench/test_paired_compare.py b/eval/tests/test_paired_compare.py
similarity index 99%
rename from harness/tests/compare_bench/test_paired_compare.py
rename to eval/tests/test_paired_compare.py
index c58dff78..e43f4d45 100644
--- a/harness/tests/compare_bench/test_paired_compare.py
+++ b/eval/tests/test_paired_compare.py
@@ -6,7 +6,7 @@ import json
import sys
from pathlib import Path
-REPO = Path(__file__).resolve().parents[2]
+REPO = Path(__file__).resolve().parents[1]
if str(REPO) not in sys.path:
sys.path.insert(0, str(REPO))
diff --git a/harness/tests/compare_bench/test_pin_plan.py b/eval/tests/test_pin_plan.py
similarity index 94%
rename from harness/tests/compare_bench/test_pin_plan.py
rename to eval/tests/test_pin_plan.py
index f1ef5bf3..52b1f59b 100644
--- a/harness/tests/compare_bench/test_pin_plan.py
+++ b/eval/tests/test_pin_plan.py
@@ -90,7 +90,7 @@ def test_pin_pair_seeds_every_arm_from_one_plan(tmp_path, monkeypatch):
from bench import ab_plan
from bench.run_bench import Battery
- battery = Battery.load(ab_plan.REPO / "bench" / "prompts" / "compare_v1.yaml")
+ battery = Battery.load(Path(ab_plan.__file__).resolve().parent / "prompts" / "compare_v1.yaml")
item = battery.prompts[0]
opts = ab_plan.AbOptions(variant_env={"CV3D_SKILLS": "1"}, pin_plan=True)
calls: list[Path] = []
@@ -127,7 +127,7 @@ def test_pin_pair_reuses_the_plan_when_the_pair_is_retried(tmp_path, monkeypatch
from bench import ab_plan
from bench.run_bench import Battery
- battery = Battery.load(ab_plan.REPO / "bench" / "prompts" / "compare_v1.yaml")
+ battery = Battery.load(Path(ab_plan.__file__).resolve().parent / "prompts" / "compare_v1.yaml")
item = battery.prompts[0]
opts = ab_plan.AbOptions(variant_env={"CV3D_SKILLS": "1"}, pin_plan=True)
n = 0
@@ -160,7 +160,7 @@ def test_a_plan_side_variant_env_is_refused_by_the_cli():
from bench import ab_plan
with pytest.raises(SystemExit):
- ab_plan.main(["--prompts", str(ab_plan.REPO / "bench" / "prompts" / "compare_v1.yaml"),
+ ab_plan.main(["--prompts", str(Path(ab_plan.__file__).resolve().parent / "prompts" / "compare_v1.yaml"),
"--out", "/tmp/never", "--pin-plan", "--no-preflight",
"--variant-env", "CV3D_PLAN_BRIEF=1"])
@@ -201,9 +201,9 @@ def test_the_seeded_plan_and_the_running_plan_share_one_stage_key():
keys, the helper called on the spec, the attribute path -- must match exactly.
"""
- harness = Path(__file__).resolve().parents[2]
- seeded = _stage_call_inputs(harness / "bench" / "pin_plan.py", "plan_once")
- running = _stage_call_inputs(harness / "codeverse" / "tracks" / "lifecycle.py", "run")
+ here = Path(__file__).resolve().parents[1] # eval/
+ seeded = _stage_call_inputs(here / "bench" / "pin_plan.py", "plan_once")
+ running = _stage_call_inputs(here.parent / "harness" / "codeverse" / "tracks" / "lifecycle.py", "run")
def norm(dump: str) -> str:
return dump.replace("Name(id='self',", "Name(id='track',")
diff --git a/harness/tests/compare_bench/test_plan_stage.py b/eval/tests/test_plan_stage.py
similarity index 98%
rename from harness/tests/compare_bench/test_plan_stage.py
rename to eval/tests/test_plan_stage.py
index af0a9a19..9f11c66e 100644
--- a/harness/tests/compare_bench/test_plan_stage.py
+++ b/eval/tests/test_plan_stage.py
@@ -13,7 +13,7 @@ import pytest
from bench.plan_stage_report import failure_class, fisher_exact, outcome, report
-DATA = Path(__file__).resolve().parents[2] / "bench" / "data" / "plan_stage"
+DATA = Path(__file__).resolve().parents[1] / "bench" / "data" / "plan_stage"
def test_fisher_exact_reproduces_the_number_d52_cites() -> None:
diff --git a/harness/tests/judges/test_replay.py b/eval/tests/test_replay.py
similarity index 100%
rename from harness/tests/judges/test_replay.py
rename to eval/tests/test_replay.py
diff --git a/harness/tests/compare_bench/test_scene_regate.py b/eval/tests/test_scene_regate.py
similarity index 100%
rename from harness/tests/compare_bench/test_scene_regate.py
rename to eval/tests/test_scene_regate.py
diff --git a/harness/tests/compare_bench/test_scene_stats.py b/eval/tests/test_scene_stats.py
similarity index 100%
rename from harness/tests/compare_bench/test_scene_stats.py
rename to eval/tests/test_scene_stats.py
diff --git a/harness/tests/compare_bench/test_session_stats.py b/eval/tests/test_session_stats.py
similarity index 100%
rename from harness/tests/compare_bench/test_session_stats.py
rename to eval/tests/test_session_stats.py
diff --git a/eval/tests/test_skill_targets_readout.py b/eval/tests/test_skill_targets_readout.py
new file mode 100644
index 00000000..d5b7914f
--- /dev/null
+++ b/eval/tests/test_skill_targets_readout.py
@@ -0,0 +1,144 @@
+"""`bench/skill_targets.py`: the per-skill deterministic readout over a recorded battery or A/B dir.
+(The claims table itself is tested with the harness: harness/tests/skills/test_targets.py.)"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from codeverse.addons.skill_targets import TARGETS
+
+
+# --------------------------------------------------------------------------- the readout
+def _record(language: str, gates: list[dict], *, build_ok: bool = True) -> dict:
+ return {"spec": {"language": language}, "final_score": 0.5, "status": "plateau",
+ "rounds": [{"index": 0, "gates": gates, "build": {"ok": build_ok}}]}
+
+
+def _battery(tmp_path: Path, cells: dict[str, dict]) -> Path:
+ root = tmp_path / "battery"
+ for name, rec in cells.items():
+ d = root / "runs" / name
+ d.mkdir(parents=True)
+ (d / "record.json").write_text(json.dumps(rec))
+ return root
+
+
+def _finding(gate: str, message: str, severity: str = "warn") -> dict:
+ return {"gate": gate, "severity": severity, "message": message}
+
+
+CONNECTIVITY = [_finding("connectivity", "'Lid' and 'Body' interpenetrate by ≈7.4 mm (12% of surface samples inside)"),
+ _finding("connectivity", "'Leg_0' and 'Seat' interpenetrate by ≈2.1 mm (3% of surface samples inside)"),
+ _finding("connectivity", "all 7 parts are connected (9 contacts, gap ≤ 1 mm)", "info")]
+CONTRACT = [_finding("contract", "part 'Seat' bbox deviates from the plan (worst 2.1× tolerance)"),
+ _finding("contract", "'Leg' (each instance) bbox deviates from the plan (worst 1.4× tolerance)")]
+
+
+def test_the_readout_counts_the_kinds_the_row_names(tmp_path):
+ from bench.skill_targets import battery_rows, load_runs
+
+ root = _battery(tmp_path, {
+ "chair": _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY},
+ {"gate": "contract", "findings": CONTRACT}]),
+ "lamp": _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY[2:]},
+ {"gate": "contract", "findings": []}]),
+ })
+ rows = {r["skill"]: r for r in battery_rows(load_runs(root), list(TARGETS), which="last", cache=None)}
+ assert rows["cv3d-part-contact"]["n"] == 2
+ assert rows["cv3d-part-contact"]["mean"] == 1.0 # 2 pairs + 0, INFO ignored
+ assert rows["cv3d-bbox-contract"]["mean"] == 0.5 # part_bbox only
+ assert rows["cv3d-repeats-and-mirrors"]["mean"] == 0.5 # instance_bbox only
+
+
+def test_a_row_only_counts_the_languages_it_is_defined_over(tmp_path):
+ from bench.skill_targets import battery_rows, load_runs
+
+ root = _battery(tmp_path, {
+ "robot": _record("urdf_blender", [{"gate": "joint_sweep", "findings": [
+ _finding("joint_sweep", "links 'a' and 'b' overlap by 6.0 mm at pose j=1.0", "error"),
+ _finding("joint_sweep", "links 'a' and 'c' overlap by 3.0 mm at rest", "warn")]}]),
+ "chair": _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY}]),
+ })
+ rows = {r["skill"]: r for r in battery_rows(load_runs(root), list(TARGETS), which="last", cache=None)}
+ assert rows["cv3d-urdf-joints"]["n"] == 1 # the blender run is not eligible
+ assert rows["cv3d-urdf-joints"]["mean"] == 1.0 # ERRORs only; the rest-pose WARN is not one
+ assert rows["cv3d-part-contact"]["n"] == 2 # both languages carry a part graph
+
+
+def test_an_ungraded_run_is_not_in_the_population(tmp_path):
+ """A concurrency probe or an e2e smoke run has no gate report; counting it would give
+ a GLB-derived row a different n from the gate-derived rows beside it."""
+ from bench.skill_targets import battery_rows, load_runs
+
+ root = _battery(tmp_path, {
+ "chair": _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY}]),
+ "probe": {"spec": {"language": "blender"}, "rounds": [{"index": 0, "gates": []}]},
+ })
+ rows = {r["skill"]: r for r in battery_rows(load_runs(root), list(TARGETS), which="last", cache=None)}
+ assert rows["cv3d-part-contact"]["n"] == 1
+
+
+def test_the_ab_mode_pairs_by_prompt(tmp_path):
+ from bench.skill_targets import ab_arms, ab_rows
+
+ root = tmp_path / "ab"
+ for arm, findings in (("control", CONNECTIVITY), ("variant", CONNECTIVITY[1:])):
+ d = root / "arms" / arm / "cells" / "chair" / "run"
+ d.mkdir(parents=True)
+ (d / "record.json").write_text(json.dumps(
+ _record("blender", [{"gate": "connectivity", "findings": findings}])))
+ arms = ab_arms(root)
+ assert arms is not None
+ rows = {r["skill"]: r for r in ab_rows(arms, list(TARGETS), which="last", cache=None)}
+ r = rows["cv3d-part-contact"]
+ assert (r["n"], r["control_mean"], r["variant_mean"], r["mean_delta"]) == (1, 2.0, 1.0, -1.0)
+ assert (r["better"], r["worse"], r["tied"]) == (1, 0, 0)
+
+
+def test_a_battery_dir_is_not_mistaken_for_an_ab_dir(tmp_path):
+ from bench.skill_targets import ab_arms
+
+ root = _battery(tmp_path, {"chair": _record("blender", [])})
+ assert ab_arms(root) is None
+
+
+def test_the_cli_runs_over_a_synthetic_battery(tmp_path, capsys):
+ from bench.skill_targets import main
+
+ root = _battery(tmp_path, {"chair": _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY}])})
+ assert main([str(root), "--json", "--no-cache"]) == 0
+ payload = json.loads(capsys.readouterr().out)
+ assert payload["mode"] == "battery"
+ row = next(r for r in payload["rows"] if r["skill"] == "cv3d-part-contact")
+ assert row["n"] == 1 and row["mean"] == 2.0
+
+
+def test_the_confidence_block_says_how_many_pairs_an_effect_needs():
+ from bench.skill_targets import confidence
+
+ c = confidence([-1.0, 1.0, -1.0, 1.0], control_mean=4.0)
+ assert c["sd"] == pytest.approx(1.1547, rel=1e-3)
+ assert c["ci95"] == pytest.approx(2 * c["se"])
+ assert c["resolvable_effect"] == 1.0 # 25% of a control mean of 4
+ assert c["n_to_resolve"] == 5 # (2*1.1547/1.0)^2
+ assert confidence([1.0], control_mean=4.0)["sd"] is None
+
+
+def test_a_retried_cell_pairs_on_the_attempt_that_reached_a_gate(tmp_path):
+ """A provider outage leaves `...flash` and `...flash.attempt1` side by side in one cell;
+ pairing the ungraded one would drop a prompt that actually ran."""
+ from bench.skill_targets import ab_arms, ab_rows
+
+ root = tmp_path / "ab"
+ for arm in ("control", "variant"):
+ cell = root / "arms" / arm / "cells" / "chair"
+ for leaf, rec in (("run_a", {"spec": {"language": "blender"}, "rounds": []}),
+ ("run_a.attempt1",
+ _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY}]))):
+ (cell / leaf).mkdir(parents=True)
+ (cell / leaf / "record.json").write_text(json.dumps(rec))
+ rows = {r["skill"]: r for r in ab_rows(ab_arms(root), list(TARGETS), which="last", cache=None)}
+ assert rows["cv3d-part-contact"]["n"] == 1
diff --git a/harness/tests/bench_prompts/test_v2_batteries.py b/eval/tests/test_v2_batteries.py
similarity index 99%
rename from harness/tests/bench_prompts/test_v2_batteries.py
rename to eval/tests/test_v2_batteries.py
index b90ac37e..49cbfd2a 100644
--- a/harness/tests/bench_prompts/test_v2_batteries.py
+++ b/eval/tests/test_v2_batteries.py
@@ -23,7 +23,7 @@ from pathlib import Path
import pytest
-REPO = Path(__file__).resolve().parents[2]
+REPO = Path(__file__).resolve().parents[1]
if str(REPO) not in sys.path:
sys.path.insert(0, str(REPO))
diff --git a/harness/tests/compare_bench/test_worktree_import.py b/eval/tests/test_worktree_import.py
similarity index 97%
rename from harness/tests/compare_bench/test_worktree_import.py
rename to eval/tests/test_worktree_import.py
index 01d33592..dd7c1c36 100644
--- a/harness/tests/compare_bench/test_worktree_import.py
+++ b/eval/tests/test_worktree_import.py
@@ -21,8 +21,8 @@ from pathlib import Path
import pytest
-HARNESS = Path(__file__).resolve().parents[2]
-BENCH = HARNESS / "bench"
+HARNESS = Path(__file__).resolve().parents[2] / "harness"
+BENCH = Path(__file__).resolve().parents[1] / "bench"
CODEVERSE_IMPORT = re.compile(r"^\s*(?:from|import)\s+codeverse\b", re.M)
#: every bench module that bootstraps sys.path because it is also run as a script
SCRIPTS = sorted(p for p in BENCH.glob("*.py")
diff --git a/finetune/3dcodeverse_eval/README.md b/finetune/3dcodeverse_eval/README.md
deleted file mode 100644
index 1a04fe94..00000000
--- a/finetune/3dcodeverse_eval/README.md
+++ /dev/null
@@ -1,52 +0,0 @@
-# 3DCodeVerse evaluation
-
-Scores a model by **executing what it generates**, not by comparing text. A 3D program either runs and
-produces the geometry or it does not, so every number here is a pass rate from a real interpreter.
-
-```bash
-source env.sh # optional on this box; all vars already default to these values
-GPUS=0 TP=1 ./run_eval.sh /path/to/model my_run
-GPUS=0 TP=1 ./run_eval.sh /path/to/model my_run --temp 0.7 --seed 1 # sampled decoding
-GPUS=0 TP=1 ./run_eval.sh /path/to/model my_run --suites blender threejs
-```
-
-Results: `$EVAL_OUT/<name>_<suite>/summary.json`, aggregated into `$EVAL_OUT/<name>_report.json`.
-
-## Suites
-
-| suite | what runs the code | pass = |
-|---|---|---|
-| `bench` | Blender 5.0.1 (+scipy) | script executes **and** geometry matches ground truth |
-| `blender` | Blender 5.0.1 | script executes, scene is non-empty |
-| `cadquery` | CadQuery | solid builds and exports |
-| `openscad` | OpenSCAD CLI | compiles to a non-empty mesh |
-| `glsl` | `glslangValidator` | shader compiles |
-| `threejs` | headless Chromium, WebGL2 | page renders without a JS error and draws something |
-
-## Layout
-
-`run_eval.sh` is the only entry point. It generates once with **one vLLM engine for all suites** (loading a
-large model per suite costs more than the generation itself), then executes and scores each suite.
-
-- generation — `generate_multi.py` (all suites, one engine), `generate_vllm.py` (single suite),
- `generate_vllm_img.py` (image → code), `generate.py` (HF fallback, no vLLM)
-- extraction — `extract.py`, dialect-aware: what counts as the answer differs per dialect
-- execution — `dialect_runners.py`, `blender_runner.py`, `threejs_runner.py`, `glsl_render.py`, `render_glb.py`
-- scoring — `run_bench.py` + `metrics.py` (bench), `dialect_eval.py`, `blender_dialect_eval.py`,
- `threejs_eval.py`, aggregation in `results_all.py`
-- `free_gpus.sh` — lists cards with enough free memory (the box is shared)
-
-## Two guards that are there for a reason
-
-**It refuses to score an empty generation directory.** An engine that never started leaves no generations,
-and every scorer reports that as 0%. A real 0% and a failed launch then look identical — which is how a
-rescore of two healthy models once came back as 0/200 on every suite.
-
-**It re-checks the GPU right before launching.** On a shared box a card that was free when it was picked can
-be full by the time vLLM starts; the harness switches to whichever card is free now, or waits.
-
-## Reading the numbers
-
-Report long-output dialects under **both** greedy and sampled decoding. A fine-tuned model that scores 6% on
-OpenSCAD greedy and 44% sampled has not forgotten the dialect — it is in a repetition loop and never emits
-the closing token. Preference pairs on termination fix it, and afterwards greedy beats sampling again.
diff --git a/finetune/3dcodeverse_eval/blender_dialect_eval.py b/finetune/3dcodeverse_eval/blender_dialect_eval.py
deleted file mode 100644
index e242a475..00000000
--- a/finetune/3dcodeverse_eval/blender_dialect_eval.py
+++ /dev/null
@@ -1,44 +0,0 @@
-"""Held-out Blender test (bioinspired3d-style): execute generated & reference scripts in Blender, compare meshes."""
-import argparse, json, os, re, subprocess, sys, numpy as np
-from concurrent.futures import ThreadPoolExecutor
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from metrics import load_points, chamfer_f, rot_y
-BLENDER = os.environ.get("BLENDER_BIN", "/wekafs/ict/hx_624/tools/blender-5.0.1-linux-x64/blender"); RUNNER = os.path.join(os.path.dirname(os.path.abspath(__file__)), "blender_runner.py")
-ap = argparse.ArgumentParser(); ap.add_argument("--test", required=True); ap.add_argument("--gen_dir", required=True); ap.add_argument("--workers", type=int, default=32); a = ap.parse_args()
-tests = [json.loads(l) for l in open(a.test)]; ref_cache = os.path.join(os.path.dirname(a.test), "ref_exec_blender")
-def code_of(msg):
- m = re.findall(r"```[a-zA-Z]*\n(.*?)```", msg, flags=re.S); return (max(m, key=len) if m else msg).strip()
-def run_blender(code, d):
- d = os.path.abspath(d); os.makedirs(d, exist_ok=True); sp = os.path.join(d, "code.py"); glb = os.path.join(d, "out.glb"); rp = os.path.join(d, "exec.json")
- if not os.path.exists(sp): open(sp, "w").write(code)
- try:
- subprocess.run([BLENDER, "-b", "--factory-startup", "-noaudio", "--python", RUNNER, "--", "--script", sp, "--out", glb, "--report", rp], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=180, env={**os.environ, "HOME": "/tmp"})
- r = json.load(open(rp)) if os.path.exists(rp) else {"status": "CRASH"}
- except subprocess.TimeoutExpired: r = {"status": "TIMEOUT"}
- r["mesh"] = glb if r.get("status") == "OK" else None; return r
-def ref_job(t):
- tid = t["id"].replace("/", "__"); d = os.path.join(ref_cache, tid)
- if os.path.exists(os.path.join(d, "rep.json")): return tid, json.load(open(os.path.join(d, "rep.json")))
- rep = run_blender(code_of(t["messages"][-1]["content"]), d); json.dump(rep, open(os.path.join(d, "rep.json"), "w")); return tid, rep
-def gen_job(t):
- tid = t["id"].replace("/", "__"); d = os.path.join(a.gen_dir, tid)
- if not os.path.exists(os.path.join(d, "code.py")): return tid, {"status": "MISSING", "mesh": None}
- return tid, run_blender(None, d)
-with ThreadPoolExecutor(a.workers) as ex: refs = dict(ex.map(ref_job, tests)); gens = dict(ex.map(gen_job, tests))
-rows = []
-for t in tests:
- tid = t["id"].replace("/", "__"); r = {"id": tid, "ref": refs[tid].get("status"), "gen": gens[tid].get("status"), "gen_error": (gens[tid].get("error") or "")[:200]}
- if r["ref"] == "OK" and r["gen"] == "OK":
- try:
- pr = load_points(refs[tid]["mesh"], 5000); pg = load_points(gens[tid]["mesh"], 5000)
- if pr is not None and pg is not None:
- best = min((chamfer_f(rot_y(pg, k), pr) for k in range(4)), key=lambda x: x[0]); r["chamfer"] = best[0]; r.update(best[1])
- except Exception as e: r["metric_error"] = str(e)[:100]
- rows.append(r)
-n = len(rows); sc = [r for r in rows if "[email protected]" in r]
-summary = {"dialect": "blender_heldout", "n": n, "ref_exec_ok": sum(r["ref"] == "OK" for r in rows), "gen_exec_ok": sum(r["gen"] == "OK" for r in rows), "gen_exec_rate": round(sum(r["gen"] == "OK" for r in rows) / n, 4),
- "n_scored": len(sc), "[email protected]_mean_scored": round(float(np.mean([r["[email protected]"] for r in sc])), 4) if sc else None, "[email protected]_mean_all": round(float(sum(r.get("[email protected]", 0) for r in rows) / n), 4), "chamfer_mean_scored": round(float(np.mean([r["chamfer"] for r in sc])), 4) if sc else None}
-os.makedirs(a.gen_dir, exist_ok=True)
-with open(os.path.join(a.gen_dir, "dialect_metrics.jsonl"), "w") as f:
- for r in rows: f.write(json.dumps(r) + "\n")
-json.dump(summary, open(os.path.join(a.gen_dir, "summary.json"), "w"), indent=2); print(json.dumps(summary))
diff --git a/finetune/3dcodeverse_eval/dialect_eval.py b/finetune/3dcodeverse_eval/dialect_eval.py
deleted file mode 100644
index 92f87a02..00000000
--- a/finetune/3dcodeverse_eval/dialect_eval.py
+++ /dev/null
@@ -1,43 +0,0 @@
-"""Evaluate a model on a non-Blender dialect test set: generate (vLLM, done separately) -> execute generated & reference -> geometry metrics vs reference mesh.
-usage: dialect_eval.py --dialect cadquery --test data/multidialect/cadquery/test.jsonl --gen_dir <dir with <id>/code.py> [--workers 32]"""
-import argparse, json, os, re, sys, numpy as np
-from concurrent.futures import ThreadPoolExecutor
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from dialect_runners import RUNNERS
-from metrics import load_points, chamfer_f, rot_y
-ap = argparse.ArgumentParser(); ap.add_argument("--dialect", required=True); ap.add_argument("--test", required=True); ap.add_argument("--gen_dir", required=True); ap.add_argument("--workers", type=int, default=32); ap.add_argument("--ref_cache", default=None)
-a = ap.parse_args()
-run = RUNNERS[a.dialect]
-tests = [json.loads(l) for l in open(a.test)]
-ref_cache = a.ref_cache or os.path.join(os.path.dirname(a.test), f"ref_exec_{a.dialect}")
-def code_of(msg):
- m = re.findall(r"```[a-zA-Z]*\n(.*?)```", msg, flags=re.S); return (max(m, key=len) if m else msg).strip()
-def ref_job(t):
- tid = t["id"].replace("/", "__"); d = os.path.join(ref_cache, tid)
- if os.path.exists(os.path.join(d, "rep.json")): return tid, json.load(open(os.path.join(d, "rep.json")))
- rep = run(code_of(t["messages"][-1]["content"]), d); json.dump(rep, open(os.path.join(d, "rep.json"), "w")); return tid, rep
-def gen_job(t):
- tid = t["id"].replace("/", "__"); d = os.path.join(a.gen_dir, tid); cp = os.path.join(d, "code.py")
- if not os.path.exists(cp): return tid, {"status": "MISSING", "error": "no generation", "mesh": None}
- rep = run(open(cp).read(), os.path.join(d, "exec")); return tid, rep
-with ThreadPoolExecutor(a.workers) as ex:
- refs = dict(ex.map(ref_job, tests)); gens = dict(ex.map(gen_job, tests))
-rows = []; np.random.seed(0)
-for t in tests:
- tid = t["id"].replace("/", "__"); r = {"id": tid, "ref": refs[tid]["status"], "gen": gens[tid]["status"], "gen_error": (gens[tid].get("error") or "")[:200]}
- if a.dialect != "glsl" and refs[tid]["status"] == "OK" and gens[tid]["status"] == "OK":
- try:
- pr = load_points(refs[tid]["mesh"], 5000); pg = load_points(gens[tid]["mesh"], 5000)
- if pr is not None and pg is not None:
- best = min((chamfer_f(rot_y(pg, k), pr) for k in range(4)), key=lambda x: x[0]); r["chamfer"] = best[0]; r.update(best[1])
- except Exception as e: r["metric_error"] = str(e)[:100]
- rows.append(r)
-n = len(rows); ref_ok = sum(r["ref"] == "OK" for r in rows); gen_ok = sum(r["gen"] == "OK" for r in rows); sc = [r for r in rows if "[email protected]" in r]
-summary = {"dialect": a.dialect, "n": n, "ref_exec_ok": ref_ok, "gen_exec_ok": gen_ok, "gen_exec_rate": round(gen_ok / n, 4), "n_scored": len(sc),
- "[email protected]_mean_scored": round(float(np.mean([r["[email protected]"] for r in sc])), 4) if sc else None, "[email protected]_mean_scored": round(float(np.mean([r["[email protected]"] for r in sc])), 4) if sc else None,
- "[email protected]_mean_all": round(float(sum(r.get("[email protected]", 0) for r in rows) / n), 4), "chamfer_mean_scored": round(float(np.mean([r["chamfer"] for r in sc])), 4) if sc else None}
-os.makedirs(a.gen_dir, exist_ok=True)
-with open(os.path.join(a.gen_dir, "dialect_metrics.jsonl"), "w") as f:
- for r in rows: f.write(json.dumps(r) + "\n")
-json.dump(summary, open(os.path.join(a.gen_dir, "summary.json"), "w"), indent=2); print(json.dumps(summary))
-from collections import Counter; print("gen failure modes:", Counter(re.sub(r"'[^']*'", "'<x>'", (r["gen_error"] or r["gen"]).split("\n")[0][:70]) for r in rows if r["gen"] != "OK").most_common(8))
diff --git a/finetune/3dcodeverse_eval/dialect_runners.py b/finetune/3dcodeverse_eval/dialect_runners.py
deleted file mode 100644
index cd988489..00000000
--- a/finetune/3dcodeverse_eval/dialect_runners.py
+++ /dev/null
@@ -1,101 +0,0 @@
-"""Executors for non-Blender dialects. Each run_* returns dict(status, error, mesh_path or None, latency_s)."""
-import os, subprocess, sys, time, json, tempfile, re
-OPENSCAD = os.environ.get("OPENSCAD_BIN", "/wekafs/ict/hx_624/tools/openscad/squashfs-root/AppRun")
-GLSLANG = os.environ.get("GLSLANG_BIN", "/wekafs/ict/hx_624/anaconda3/envs/llmft/bin/glslangValidator")
-PY = os.environ.get("EVAL_PYTHON", "/wekafs/ict/hx_624/anaconda3/envs/llmft/bin/python")
-CQ_RUNNER = r'''
-import sys, json, runpy, traceback, os
-os.environ.setdefault("OMP_NUM_THREADS","1")
-script, out_stl = sys.argv[1], sys.argv[2]
-rep = {"status":"FAIL","error":None}
-try:
- import cadquery as cq
- sys.argv = [script, out_stl]
- _orig_export = cq.exporters.export
- captured = []
- def _capture(o, *a, **k): captured.append(o)
- # neutralise export calls (the scripts export STEP/STL to arbitrary paths) but remember the exported object
- for modname in ("cadquery.occ_impl.exporters", "cadquery.exporters"):
- try:
- import importlib; _m = importlib.import_module(modname); _m.export = _capture
- except Exception: pass
- try: cq.exporters.export = _capture
- except Exception: pass
- g = runpy.run_path(script, run_name="__main__", init_globals={"show_object": _capture, "debug": _capture})
- obj = None
- for c in reversed(captured):
- if isinstance(c, (cq.Workplane, cq.Shape, cq.Assembly)): obj = c; break
- if obj is None:
- for name in ("result", "final", "model", "part", "solid", "shape", "body", "assembly", "assy", "obj", "res"):
- if isinstance(g.get(name), (cq.Workplane, cq.Shape, cq.Assembly)): obj = g[name]; break
- if obj is None:
- cands = [v for v in g.values() if isinstance(v, (cq.Workplane, cq.Shape, cq.Assembly))]
- obj = cands[-1] if cands else None
- if obj is None: raise RuntimeError("no CadQuery result object found (expected variable `result`)")
- if isinstance(obj, cq.Assembly): obj = obj.toCompound()
- if isinstance(obj, cq.Workplane):
- vals = obj.vals()
- if not vals: raise RuntimeError("empty workplane")
- obj = vals[0] if len(vals)==1 else cq.Compound.makeCompound([v for v in vals if isinstance(v, cq.Shape)])
- if not isinstance(obj, cq.Shape): raise RuntimeError(f"unsupported object {type(obj)}")
- if obj.Volume() <= 1e-9 and not obj.Faces(): raise RuntimeError("empty shape")
- _orig_export(obj, out_stl, tolerance=0.01, angularTolerance=0.1)
- rep["status"]="OK"
-except BaseException as e:
- rep["error"]=(type(e).__name__+": "+str(e))[:500]; rep["traceback"]=traceback.format_exc()[-1500:]
-print("CQ_REPORT "+json.dumps(rep))
-'''
-def run_cadquery(code, workdir, timeout=90):
- workdir = os.path.abspath(workdir); os.makedirs(workdir, exist_ok=True); sp = os.path.join(workdir, "script.py"); stl = os.path.join(workdir, "out.stl"); rp = os.path.join(workdir, "runner.py")
- open(sp, "w").write(code); open(rp, "w").write(CQ_RUNNER)
- t0 = time.time()
- try:
- p = subprocess.run([PY, rp, sp, stl], cwd=workdir, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=timeout, text=True, errors="replace")
- m = re.search(r"CQ_REPORT (\{.*\})", p.stdout)
- rep = json.loads(m.group(1)) if m else {"status": "CRASH", "error": p.stdout[-800:]}
- except subprocess.TimeoutExpired:
- rep = {"status": "TIMEOUT", "error": f"timeout {timeout}s"}
- rep["latency_s"] = round(time.time() - t0, 1); rep["mesh"] = stl if rep["status"] == "OK" and os.path.exists(stl) else None
- if rep["status"] == "OK" and rep["mesh"] is None: rep["status"] = "EMPTY"
- return rep
-def run_openscad(code, workdir, timeout=120):
- workdir = os.path.abspath(workdir); os.makedirs(workdir, exist_ok=True); sc = os.path.join(workdir, "model.scad"); stl = os.path.join(workdir, "out.stl")
- open(sc, "w").write(code); t0 = time.time()
- try:
- p = subprocess.run([OPENSCAD, "-o", stl, "--export-format", "binstl", sc], cwd=workdir, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=timeout, text=True, errors="replace",
- env={**os.environ, "HOME": workdir, "QT_QPA_PLATFORM": "offscreen"})
- ok = p.returncode == 0 and os.path.exists(stl) and os.path.getsize(stl) > 100
- rep = {"status": "OK" if ok else ("EMPTY" if p.returncode == 0 else "FAIL"), "error": None if ok else p.stdout[-800:]}
- except subprocess.TimeoutExpired:
- rep = {"status": "TIMEOUT", "error": f"timeout {timeout}s"}
- rep["latency_s"] = round(time.time() - t0, 1); rep["mesh"] = stl if rep["status"] == "OK" else None
- return rep
-GLSL_PRELUDE = """#version 300 es
-precision highp float; precision highp int;
-uniform vec3 iResolution; uniform float iTime; uniform float iTimeDelta; uniform int iFrame; uniform vec4 iMouse; uniform vec4 iDate; uniform float iSampleRate;
-uniform sampler2D iChannel0; uniform sampler2D iChannel1; uniform sampler2D iChannel2; uniform sampler2D iChannel3;
-uniform vec3 iChannelResolution[4]; uniform float iChannelTime[4];
-out vec4 _outColor;
-"""
-GLSL_MAIN = "\nvoid main(){ vec4 c = vec4(0.0); mainImage(c, gl_FragCoord.xy); _outColor = c; }\n"
-def run_glsl(code, workdir, timeout=30):
- workdir = os.path.abspath(workdir); os.makedirs(workdir, exist_ok=True); fp = os.path.join(workdir, "shader.frag")
- # normalise: drop #version / precision lines and re-declarations of Shadertoy uniforms / the out variable (the prelude provides them)
- import re as _re
- lines = []
- for ln in code.splitlines():
- if _re.match(r"\s*#\s*version\b", ln) or _re.match(r"\s*precision\s+(low|medium|high)p\s+", ln): continue
- if _re.match(r"\s*uniform\s+.*\b(iResolution|iTime|iTimeDelta|iFrame|iMouse|iDate|iSampleRate|iChannel[0-3]|iChannelResolution|iChannelTime)\b", ln): continue
- if _re.match(r"\s*out\s+vec4\s+\w+\s*;", ln): continue
- lines.append(ln)
- src = "\n".join(lines)
- if "mainImage" not in src: return {"status": "FAIL", "error": "no mainImage()", "latency_s": 0, "mesh": None}
- open(fp, "w").write(GLSL_PRELUDE + src + GLSL_MAIN); t0 = time.time()
- try:
- p = subprocess.run([GLSLANG, "-S", "frag", fp], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=timeout, text=True, errors="replace")
- rep = {"status": "OK" if p.returncode == 0 else "FAIL", "error": None if p.returncode == 0 else p.stdout[-600:]}
- except subprocess.TimeoutExpired:
- rep = {"status": "TIMEOUT", "error": "timeout"}
- rep["latency_s"] = round(time.time() - t0, 2); rep["mesh"] = None
- return rep
-RUNNERS = {"cadquery": run_cadquery, "openscad": run_openscad, "glsl": run_glsl}
diff --git a/finetune/3dcodeverse_eval/env.sh b/finetune/3dcodeverse_eval/env.sh
deleted file mode 100644
index 6f1aad09..00000000
--- a/finetune/3dcodeverse_eval/env.sh
+++ /dev/null
@@ -1,12 +0,0 @@
-# Tool locations for 3DCodeVerse evaluation. `source env.sh` before run_eval.sh, or export your own.
-# Every variable already defaults to the value below, so on this box sourcing it is optional.
-export PROJECT_ROOT=${PROJECT_ROOT:-/wekafs/ict/hx_624/llm-ft} # holds data/multidialect, data/sft_v1
-export EVAL_OUT=${EVAL_OUT:-$PROJECT_ROOT/eval/out} # generations + summaries land here
-export BENCH_DATA=${BENCH_DATA:-/wekafs/ict/hx_624/data/3dcodebench/data}
-export BLENDER_BIN=${BLENDER_BIN:-/wekafs/ict/hx_624/tools/blender-5.0.1-linux-x64/blender}
-export OPENSCAD_BIN=${OPENSCAD_BIN:-/wekafs/ict/hx_624/tools/openscad/squashfs-root/AppRun}
-export GLSLANG_BIN=${GLSLANG_BIN:-/wekafs/ict/hx_624/anaconda3/envs/llmft/bin/glslangValidator}
-export EVAL_PYTHON=${EVAL_PYTHON:-/wekafs/ict/hx_624/anaconda3/envs/llmft/bin/python}
-export PLAYWRIGHT_BROWSERS_PATH=${PLAYWRIGHT_BROWSERS_PATH:-/wekafs/ict/hx_624/cache/ms-playwright}
-export VLLM_ENV=${VLLM_ENV:-vllm} # conda env with vLLM (generation)
-export EXEC_ENV=${EXEC_ENV:-llmft} # conda env with the runners (execution + scoring)
diff --git a/finetune/3dcodeverse_eval/free_gpus.sh b/finetune/3dcodeverse_eval/free_gpus.sh
deleted file mode 100755
index b61c9376..00000000
--- a/finetune/3dcodeverse_eval/free_gpus.sh
+++ /dev/null
@@ -1,16 +0,0 @@
-#!/bin/bash
-# Print the GPUs with at least <need_free_mib> free — other users share this box, so "completely empty" is the
-# wrong test; what matters is whether OUR job fits.
-# scripts/free_gpus.sh [need_free_mib] [gpu_list] default 60000 (a 9B LoRA at cutoff 8192), all GPUs
-# The list used to be hardcoded to 0-3 from when this box exposed four cards, which silently hid 4-7 from every
-# script that auto-selects a GPU.
-NEED=${1:-60000}
-LIST=${2:-$(nvidia-smi --query-gpu=index --format=csv,noheader | tr '\n' ' ')}
-OUT=""
-for i in $LIST; do
- read -r TOT USED <<< "$(nvidia-smi --query-gpu=memory.total,memory.used --format=csv,noheader,nounits -i $i | tr ',' ' ')"
- [ -z "${TOT:-}" ] && continue
- FREE=$((TOT - USED))
- [ "$FREE" -ge "$NEED" ] && OUT="$OUT,$i"
-done
-echo "${OUT#,}"
diff --git a/finetune/3dcodeverse_eval/generate.py b/finetune/3dcodeverse_eval/generate.py
deleted file mode 100644
index cabaf670..00000000
--- a/finetune/3dcodeverse_eval/generate.py
+++ /dev/null
@@ -1,81 +0,0 @@
-"""Batched generation for the 3DCodeBench prompts with a HF causal LM. Writes <out>/<task>/code.py + gens.jsonl"""
-import argparse, json, os, re, time, torch
-from transformers import AutoTokenizer, AutoModelForCausalLM
-
-def extract_code(text):
- m = re.findall(r"```[a-zA-Z0-9_+.-]*[ \t]*\n(.*?)```", text, flags=re.S) # any language fence (python/glsl/openscad/...)
- if m:
- return max(m, key=len).strip("\n")
- m = re.search(r"```[a-zA-Z0-9_+.-]*[ \t]*\n(.*)$", text, flags=re.S) # unterminated block
- if m: return m.group(1).strip("\n")
- return text.strip()
-
-def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--model", required=True)
- ap.add_argument("--prompts", required=True)
- ap.add_argument("--out", required=True)
- ap.add_argument("--batch_size", type=int, default=16)
- ap.add_argument("--max_new_tokens", type=int, default=6144)
- ap.add_argument("--temperature", type=float, default=0.0)
- ap.add_argument("--top_p", type=float, default=0.95)
- ap.add_argument("--shard", type=int, default=0); ap.add_argument("--num_shards", type=int, default=1)
- ap.add_argument("--attn", default="auto")
- ap.add_argument("--limit", type=int, default=None)
- ap.add_argument("--no_think", action="store_true", help="pass enable_thinking=False to the chat template (Qwen3/3.5)")
- a = ap.parse_args()
- os.makedirs(a.out, exist_ok=True)
- attn = a.attn
- if attn == "auto":
- try: import flash_attn; attn = "flash_attention_2" # noqa
- except Exception: attn = "sdpa"
- tok = AutoTokenizer.from_pretrained(a.model, padding_side="left")
- if tok.pad_token is None: tok.pad_token = tok.eos_token
- try:
- model = AutoModelForCausalLM.from_pretrained(a.model, dtype=torch.bfloat16, attn_implementation=attn, device_map="cuda")
- except Exception as e: # multimodal wrappers (e.g. Qwen3.5 = Qwen3_5ForConditionalGeneration)
- print("[gen] AutoModelForCausalLM failed (%s); falling back to AutoModelForImageTextToText" % str(e)[:200], flush=True)
- from transformers import AutoModelForImageTextToText
- model = AutoModelForImageTextToText.from_pretrained(a.model, dtype=torch.bfloat16, attn_implementation=attn, device_map="cuda")
- model.eval()
- rows = [json.loads(l) for l in open(a.prompts)]
- rows = rows[a.shard::a.num_shards]
- if a.limit: rows = rows[:a.limit]
- # skip already-done
- done = set()
- outf = f"{a.out}/gens_shard{a.shard}.jsonl"
- if os.path.exists(outf):
- done = {json.loads(l)["task"] for l in open(outf)}
- rows = [r for r in rows if r["task"] not in done]
- print(f"[gen] {len(rows)} prompts to do (shard {a.shard}/{a.num_shards}) attn={attn}", flush=True)
- # sort by prompt length for efficient batching
- tmpl_kwargs = {"enable_thinking": False} if a.no_think else {}
- texts = [tok.apply_chat_template(r["messages"], tokenize=False, add_generation_prompt=True, **tmpl_kwargs) for r in rows]
- if texts: print("[gen] prompt tail example:", repr(texts[0][-200:]), flush=True)
- order = sorted(range(len(rows)), key=lambda i: len(texts[i]))
- gen_kwargs = dict(max_new_tokens=a.max_new_tokens, pad_token_id=tok.pad_token_id)
- if a.temperature > 0: gen_kwargs.update(do_sample=True, temperature=a.temperature, top_p=a.top_p)
- else: gen_kwargs.update(do_sample=False)
- t0 = time.time(); ntok = 0
- with open(outf, "a") as f:
- for b in range(0, len(order), a.batch_size):
- idx = order[b:b + a.batch_size]
- enc = tok([texts[i] for i in idx], return_tensors="pt", padding=True).to("cuda")
- with torch.no_grad():
- out = model.generate(**enc, **gen_kwargs)
- gen = out[:, enc["input_ids"].shape[1]:]
- for j, i in enumerate(idx):
- ids = gen[j]
- n = int((ids != tok.pad_token_id).sum())
- text = tok.decode(ids, skip_special_tokens=True)
- code = extract_code(text)
- finished = bool((ids == tok.eos_token_id).any()) or any((ids == t).any() for t in tok.all_special_ids if t != tok.pad_token_id) if n < a.max_new_tokens else False
- task = rows[i]["task"]; d = f"{a.out}/{task}"; os.makedirs(d, exist_ok=True)
- open(f"{d}/code.py", "w").write(code + "\n"); open(f"{d}/raw.txt", "w").write(text)
- f.write(json.dumps({"task": task, "n_new_tokens": n, "finished": finished, "has_code_block": "```" in text}) + "\n"); f.flush()
- ntok += n
- el = time.time() - t0
- print(f"[gen] {min(b + a.batch_size, len(order))}/{len(order)} done | {ntok} tok | {ntok/el:.0f} tok/s | {el/60:.1f} min", flush=True)
-
-if __name__ == "__main__":
- main()
diff --git a/finetune/3dcodeverse_eval/generate_multi.py b/finetune/3dcodeverse_eval/generate_multi.py
deleted file mode 100644
index db6f08f8..00000000
--- a/finetune/3dcodeverse_eval/generate_multi.py
+++ /dev/null
@@ -1,114 +0,0 @@
-"""Run several evaluation suites through ONE vLLM engine (loading a 27B twice costs more than generating).
-
- python eval/generate_multi.py --model M --tp 2 --spec '[{"name":"bench","prompts":"...","out":"...","dialect":"blender"}, ...]'
- [--no_think] [--temperature 0.7] [--seed 1] [--max_new_tokens 12288] [--max_model_len 16384] [--limit N]
-
-Writes, per job: <out>/<task>/{code.py,raw.txt}, <out>/gens_shard0.jsonl, <out>/extract_stats.json — identical to
-generate_vllm.py, so every downstream executor/metric script works unchanged.
-"""
-import argparse
-import json
-import os
-import sys
-import time
-
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from extract import extract, summarize
-
-
-
-def auto_gpu_mem(requested, tp=1, headroom_gib=4.0, model_dir=None):
- """`gpu_memory_utilization` is a fraction of the card's TOTAL memory, but this box is shared — asking for
- 0.88 of a card that another user already half fills makes vLLM refuse to start, and asking for exactly what
- is free makes its KV-cache profiling OOM instead. Derive the fraction from what is actually free, and scale
- the headroom with the model: a 51 GB checkpoint needs far more slack than a 18 GB one."""
- try:
- import subprocess, glob, os
- if model_dir and os.path.isdir(model_dir):
- gib = sum(os.path.getsize(f) for f in glob.glob(os.path.join(model_dir, "*.safetensors"))) / 1024**3
- if gib > 30:
- headroom_gib = max(headroom_gib, 10.0)
- out = subprocess.run(["nvidia-smi", "--query-gpu=memory.total,memory.used", "--format=csv,noheader,nounits"],
- capture_output=True, text=True, timeout=30).stdout.strip().splitlines()
- vis = os.environ.get("CUDA_VISIBLE_DEVICES", "")
- idx = [int(x) for x in vis.split(",") if x.strip().isdigit()] or list(range(len(out)))
- worst = 1.0
- for i in idx[:max(1, tp)]:
- tot, used = (int(v) for v in out[i].split(","))
- free_gib = (tot - used) / 1024.0
- worst = min(worst, max(0.10, (free_gib - headroom_gib) / (tot / 1024.0)))
- if worst < requested:
- print(f"[gen] gpu_memory_utilization {requested} -> {worst:.2f} (shared card, {headroom_gib:.0f} GiB headroom)", flush=True)
- return round(worst, 2)
- except Exception as e:
- print(f"[gen] auto_gpu_mem failed ({e}); keeping {requested}", flush=True)
- return requested
-
-
-def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--model", required=True)
- ap.add_argument("--spec", required=True, help="JSON list (or @file) of {name,prompts,out,dialect}")
- ap.add_argument("--max_new_tokens", type=int, default=8192)
- ap.add_argument("--max_model_len", type=int, default=12288)
- ap.add_argument("--temperature", type=float, default=0.0)
- ap.add_argument("--seed", type=int, default=0)
- ap.add_argument("--no_think", action="store_true")
- ap.add_argument("--tp", type=int, default=1)
- ap.add_argument("--gpu_mem", type=float, default=0.88)
- # Qwen3.5/3.8 carry linear-attention layers and vLLM needs one Mamba cache block per concurrent sequence;
- # the default 1024 exceeds the blocks a 27B leaves free and the engine refuses to start
- ap.add_argument("--max_num_seqs", type=int, default=None)
- ap.add_argument("--limit", type=int, default=None)
- a = ap.parse_args()
- spec = json.load(open(a.spec[1:])) if a.spec.startswith("@") else json.loads(a.spec)
-
- from vllm import LLM, SamplingParams
- t_load = time.time()
- llm = LLM(model=a.model, dtype="bfloat16", tensor_parallel_size=a.tp, max_model_len=a.max_model_len,
- gpu_memory_utilization=auto_gpu_mem(a.gpu_mem, getattr(a, 'tp', 1), model_dir=a.model), enable_prefix_caching=True,
- limit_mm_per_prompt={"image": 0, "video": 0}, trust_remote_code=True,
- **({"max_num_seqs": a.max_num_seqs} if a.max_num_seqs else {}))
- print(f"[gen-multi] engine up in {time.time()-t_load:.0f}s | jobs: {[j['name'] for j in spec]}", flush=True)
- sp = SamplingParams(temperature=a.temperature, top_p=0.95 if a.temperature > 0 else 1.0,
- max_tokens=a.max_new_tokens, seed=a.seed)
- kw = {"chat_template_kwargs": {"enable_thinking": False}} if a.no_think else {}
-
- for job in spec:
- rows = [json.loads(l) for l in open(job["prompts"])]
- if a.limit:
- rows = rows[: a.limit]
- os.makedirs(job["out"], exist_ok=True)
- t0 = time.time()
- outs = llm.chat([r["messages"] for r in rows], sp, use_tqdm=True, **kw)
- el = time.time() - t0
- ntok, metas = 0, []
- with open(f"{job['out']}/gens_shard0.jsonl", "w") as f:
- for r, o in zip(rows, outs):
- text = o.outputs[0].text
- n = len(o.outputs[0].token_ids)
- ntok += n
- d = f"{job['out']}/{r['task']}"
- os.makedirs(d, exist_ok=True)
- code, meta = extract(text, job.get("dialect", "auto"))
- metas.append(meta)
- open(f"{d}/code.py", "w").write(code + "\n")
- open(f"{d}/raw.txt", "w").write(text)
- f.write(json.dumps({"task": r["task"], "n_new_tokens": n,
- "finished": o.outputs[0].finish_reason == "stop",
- "has_code_block": "```" in text, "extract": meta}) + "\n")
- stats = summarize(metas)
- stats.update({"model": a.model, "dialect": job.get("dialect"), "temperature": a.temperature,
- "no_think": a.no_think, "max_new_tokens": a.max_new_tokens,
- "tok_per_s": round(ntok / max(el, 1e-6), 1), "mean_new_tokens": round(ntok / max(1, len(rows)), 1),
- "truncated": sum(1 for r, o in zip(rows, outs) if o.outputs[0].finish_reason != "stop")})
- json.dump(stats, open(f"{job['out']}/extract_stats.json", "w"), indent=2)
- print(f"[gen-multi] {job['name']}: {len(rows)} prompts | {ntok} tok | {ntok/el:.0f} tok/s | {el/60:.1f} min", flush=True)
- print("[extract] " + json.dumps({k: stats[k] for k in ("with_think", "no_fence", "from_think_fallback",
- "multi_block", "chosen_not_last", "syntax_valid",
- "no_dialect_marker", "truncated")}), flush=True)
- print("[gen-multi] ALL GENERATION DONE", flush=True)
-
-
-if __name__ == "__main__":
- main()
diff --git a/finetune/3dcodeverse_eval/generate_vllm.py b/finetune/3dcodeverse_eval/generate_vllm.py
deleted file mode 100644
index d0593ba9..00000000
--- a/finetune/3dcodeverse_eval/generate_vllm.py
+++ /dev/null
@@ -1,82 +0,0 @@
-"""vLLM version of generate.py: same outputs (<out>/<task>/code.py + raw.txt, <out>/gens_shard0.jsonl)."""
-import argparse, json, os, re, time, sys
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from generate import extract_code as extract_code_legacy
-from extract import extract, summarize
-
-
-def auto_gpu_mem(requested, tp=1, headroom_gib=4.0, model_dir=None):
- """`gpu_memory_utilization` is a fraction of the card's TOTAL memory, but this box is shared — asking for
- 0.88 of a card that another user already half fills makes vLLM refuse to start, and asking for exactly what
- is free makes its KV-cache profiling OOM instead. Derive the fraction from what is actually free, and scale
- the headroom with the model: a 51 GB checkpoint needs far more slack than a 18 GB one."""
- try:
- import subprocess, glob, os
- if model_dir and os.path.isdir(model_dir):
- gib = sum(os.path.getsize(f) for f in glob.glob(os.path.join(model_dir, "*.safetensors"))) / 1024**3
- if gib > 30:
- headroom_gib = max(headroom_gib, 10.0)
- out = subprocess.run(["nvidia-smi", "--query-gpu=memory.total,memory.used", "--format=csv,noheader,nounits"],
- capture_output=True, text=True, timeout=30).stdout.strip().splitlines()
- vis = os.environ.get("CUDA_VISIBLE_DEVICES", "")
- idx = [int(x) for x in vis.split(",") if x.strip().isdigit()] or list(range(len(out)))
- worst = 1.0
- for i in idx[:max(1, tp)]:
- tot, used = (int(v) for v in out[i].split(","))
- free_gib = (tot - used) / 1024.0
- worst = min(worst, max(0.10, (free_gib - headroom_gib) / (tot / 1024.0)))
- if worst < requested:
- print(f"[gen] gpu_memory_utilization {requested} -> {worst:.2f} (shared card, {headroom_gib:.0f} GiB headroom)", flush=True)
- return round(worst, 2)
- except Exception as e:
- print(f"[gen] auto_gpu_mem failed ({e}); keeping {requested}", flush=True)
- return requested
-
-
-def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--model", required=True); ap.add_argument("--prompts", required=True); ap.add_argument("--out", required=True)
- ap.add_argument("--max_new_tokens", type=int, default=6144); ap.add_argument("--temperature", type=float, default=0.0)
- ap.add_argument("--max_model_len", type=int, default=12288); ap.add_argument("--no_think", action="store_true")
- ap.add_argument("--dialect", default="auto", help="blender|cadquery|openscad|glsl|threejs|auto — steers code extraction from free-form answers")
- ap.add_argument("--legacy_extract", action="store_true", help="use the old longest-fenced-block rule")
- ap.add_argument("--tp", type=int, default=1); ap.add_argument("--gpu_mem", type=float, default=0.88)
- # Qwen3.5/3.8 carry linear-attention layers and vLLM needs one Mamba cache block per concurrent sequence;
- # the default 1024 exceeds the blocks a 27B leaves free and the engine refuses to start
- ap.add_argument("--max_num_seqs", type=int, default=None); ap.add_argument("--limit", type=int, default=None); ap.add_argument("--seed", type=int, default=0)
- a = ap.parse_args()
- from vllm import LLM, SamplingParams
- os.makedirs(a.out, exist_ok=True)
- rows = [json.loads(l) for l in open(a.prompts)]
- if a.limit: rows = rows[:a.limit]
- llm = LLM(model=a.model, dtype="bfloat16", tensor_parallel_size=a.tp, max_model_len=a.max_model_len, gpu_memory_utilization=auto_gpu_mem(a.gpu_mem, getattr(a, 'tp', 1), model_dir=a.model),
- enable_prefix_caching=True, limit_mm_per_prompt={"image": 0, "video": 0}, trust_remote_code=True,
- **({"max_num_seqs": a.max_num_seqs} if a.max_num_seqs else {}))
- sp = SamplingParams(temperature=a.temperature, top_p=0.95 if a.temperature > 0 else 1.0, max_tokens=a.max_new_tokens, seed=a.seed)
- kw = {"chat_template_kwargs": {"enable_thinking": False}} if a.no_think else {}
- t0 = time.time()
- outs = llm.chat([r["messages"] for r in rows], sp, use_tqdm=True, **kw)
- el = time.time() - t0; ntok = 0
- metas = []
- with open(f"{a.out}/gens_shard0.jsonl", "w") as f:
- for r, o in zip(rows, outs):
- text = o.outputs[0].text; n = len(o.outputs[0].token_ids); ntok += n
- d = f"{a.out}/{r['task']}"; os.makedirs(d, exist_ok=True)
- if a.legacy_extract:
- code, meta = extract_code_legacy(text), {}
- else:
- code, meta = extract(text, a.dialect)
- metas.append(meta)
- open(f"{d}/code.py", "w").write(code + "\n"); open(f"{d}/raw.txt", "w").write(text)
- f.write(json.dumps({"task": r["task"], "n_new_tokens": n, "finished": o.outputs[0].finish_reason == "stop",
- "has_code_block": "```" in text, "extract": meta}) + "\n")
- stats = summarize(metas) if not a.legacy_extract else {"legacy": True}
- stats.update({"model": a.model, "dialect": a.dialect, "temperature": a.temperature, "no_think": a.no_think,
- "max_new_tokens": a.max_new_tokens, "tok_per_s": round(ntok / el, 1), "mean_new_tokens": round(ntok / max(1, len(rows)), 1)})
- json.dump(stats, open(f"{a.out}/extract_stats.json", "w"), indent=2)
- print(f"[gen-vllm] {len(rows)} prompts | {ntok} tok | {ntok/el:.0f} tok/s | {el/60:.1f} min", flush=True)
- if not a.legacy_extract:
- print("[extract] " + json.dumps({k: stats[k] for k in ("with_think", "no_fence", "from_think_fallback", "multi_block", "chosen_not_last", "syntax_valid", "no_dialect_marker")}), flush=True)
-
-if __name__ == "__main__":
- main()
diff --git a/finetune/3dcodeverse_eval/generate_vllm_img.py b/finetune/3dcodeverse_eval/generate_vllm_img.py
deleted file mode 100644
index 123c682b..00000000
--- a/finetune/3dcodeverse_eval/generate_vllm_img.py
+++ /dev/null
@@ -1,98 +0,0 @@
-"""vLLM generation with image inputs (multimodal prompts), same outputs as generate_vllm.py.
-
-Prompt file rows: {"task", "images": [paths], "messages": [{role, content}]} where the user content contains one
-`<image>` per image. The images are attached as multi_modal_data so the model actually sees them.
-"""
-import argparse, json, os, sys, time
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from extract import extract, summarize
-
-
-def auto_gpu_mem(requested, tp=1, headroom_gib=4.0, model_dir=None):
- """`gpu_memory_utilization` is a fraction of the card's TOTAL memory, but this box is shared — asking for
- 0.88 of a card that another user already half fills makes vLLM refuse to start, and asking for exactly what
- is free makes its KV-cache profiling OOM instead. Derive the fraction from what is actually free, and scale
- the headroom with the model: a 51 GB checkpoint needs far more slack than a 18 GB one."""
- try:
- import subprocess, glob, os
- if model_dir and os.path.isdir(model_dir):
- gib = sum(os.path.getsize(f) for f in glob.glob(os.path.join(model_dir, "*.safetensors"))) / 1024**3
- if gib > 30:
- headroom_gib = max(headroom_gib, 10.0)
- out = subprocess.run(["nvidia-smi", "--query-gpu=memory.total,memory.used", "--format=csv,noheader,nounits"],
- capture_output=True, text=True, timeout=30).stdout.strip().splitlines()
- vis = os.environ.get("CUDA_VISIBLE_DEVICES", "")
- idx = [int(x) for x in vis.split(",") if x.strip().isdigit()] or list(range(len(out)))
- worst = 1.0
- for i in idx[:max(1, tp)]:
- tot, used = (int(v) for v in out[i].split(","))
- free_gib = (tot - used) / 1024.0
- worst = min(worst, max(0.10, (free_gib - headroom_gib) / (tot / 1024.0)))
- if worst < requested:
- print(f"[gen] gpu_memory_utilization {requested} -> {worst:.2f} (shared card, {headroom_gib:.0f} GiB headroom)", flush=True)
- return round(worst, 2)
- except Exception as e:
- print(f"[gen] auto_gpu_mem failed ({e}); keeping {requested}", flush=True)
- return requested
-
-
-def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--model", required=True); ap.add_argument("--prompts", required=True); ap.add_argument("--out", required=True)
- ap.add_argument("--max_new_tokens", type=int, default=32768); ap.add_argument("--max_model_len", type=int, default=12288)
- ap.add_argument("--temperature", type=float, default=0.0); ap.add_argument("--seed", type=int, default=0)
- ap.add_argument("--tp", type=int, default=1); ap.add_argument("--gpu_mem", type=float, default=0.88)
- # Qwen3.5/3.8 carry linear-attention layers and vLLM needs one Mamba cache block per concurrent sequence;
- # the default 1024 exceeds the blocks a 27B leaves free and the engine refuses to start
- ap.add_argument("--max_num_seqs", type=int, default=None)
- ap.add_argument("--presence_penalty", type=float, default=0.0)
- ap.add_argument("--repetition_penalty", type=float, default=1.0)
- ap.add_argument("--dialect", default="blender"); ap.add_argument("--limit", type=int, default=None)
- a = ap.parse_args()
- from vllm import LLM, SamplingParams
- from PIL import Image
- rows = [json.loads(l) for l in open(a.prompts) if l.strip()]
- if a.limit: rows = rows[:a.limit]
- os.makedirs(a.out, exist_ok=True)
- llm = LLM(model=a.model, dtype="bfloat16", tensor_parallel_size=a.tp, max_model_len=a.max_model_len,
- gpu_memory_utilization=auto_gpu_mem(a.gpu_mem, getattr(a, 'tp', 1), model_dir=a.model), limit_mm_per_prompt={"image": 4, "video": 0}, trust_remote_code=True,
- **({"max_num_seqs": a.max_num_seqs} if a.max_num_seqs else {}))
- sp = SamplingParams(temperature=a.temperature, top_p=0.95 if a.temperature > 0 else 1.0,
- presence_penalty=a.presence_penalty, repetition_penalty=a.repetition_penalty,
- max_tokens=a.max_new_tokens, seed=a.seed)
- import base64, io
- def data_url(path):
- im = Image.open(path).convert("RGB")
- im.thumbnail((768, 768))
- b = io.BytesIO(); im.save(b, "JPEG", quality=88)
- return "data:image/jpeg;base64," + base64.b64encode(b.getvalue()).decode()
- convs = []
- for r in rows:
- msgs = []
- for m in r["messages"]:
- if m["role"] == "user" and r.get("images"):
- parts = [{"type": "image_url", "image_url": {"url": data_url(p)}} for p in r["images"]]
- parts.append({"type": "text", "text": m["content"].replace("<image>", "").strip()})
- msgs.append({"role": "user", "content": parts})
- else:
- msgs.append(m)
- convs.append(msgs)
- t0 = time.time()
- outs = llm.chat(convs, sp, use_tqdm=True, chat_template_kwargs={"enable_thinking": False})
- el = time.time() - t0; ntok = 0; metas = []
- with open(f"{a.out}/gens_shard0.jsonl", "w") as f:
- for r, o in zip(rows, outs):
- text = o.outputs[0].text; n = len(o.outputs[0].token_ids); ntok += n
- d = f"{a.out}/{r['task']}"; os.makedirs(d, exist_ok=True)
- code, meta = extract(text, a.dialect); metas.append(meta)
- open(f"{d}/code.py", "w").write(code + "\n"); open(f"{d}/raw.txt", "w").write(text)
- f.write(json.dumps({"task": r["task"], "n_new_tokens": n,
- "finished": o.outputs[0].finish_reason == "stop",
- "has_code_block": "```" in text, "extract": meta}) + "\n")
- st = summarize(metas); st.update({"model": a.model, "images": True, "tok_per_s": round(ntok/max(el,1e-6), 1),
- "mean_new_tokens": round(ntok/max(1, len(rows)), 1)})
- json.dump(st, open(f"{a.out}/extract_stats.json", "w"), indent=2)
- print(f"[gen-vllm-img] {len(rows)} prompts | {ntok} tok | {ntok/el:.0f} tok/s | {el/60:.1f} min", flush=True)
-
-if __name__ == "__main__":
- main()
diff --git a/finetune/3dcodeverse_eval/glsl_render.py b/finetune/3dcodeverse_eval/glsl_render.py
deleted file mode 100644
index 5061be39..00000000
--- a/finetune/3dcodeverse_eval/glsl_render.py
+++ /dev/null
@@ -1,119 +0,0 @@
-"""Render a Shadertoy fragment shader in a real WebGL2 context and check that it draws something.
-
-Compiling is necessary but not sufficient: a shader can compile and still paint a constant colour. This runs the
-shader in headless Chromium (the same browser the three.js runner uses), samples two frames (iTime = 0 and 1.7 s)
-and reports, per frame, how many distinct quantised colours the canvas holds and its pixel standard deviation.
-
- render_glsl(code, workdir) -> {status, error, distinct, std, changed, shot}
- status OK renders a non-uniform image
- STATIC renders, but the image is (near) uniform — usually a black screen
- FAIL shader/link error inside the browser, or the page threw
- changed True if frame(1.7s) differs from frame(0) — i.e. the shader animates
-
-usage as a script: python eval/glsl_render.py file1.glsl [file2.glsl ...]
-"""
-import json
-import os
-import sys
-
-HTML = """<!doctype html><html><body style="margin:0">
-<canvas id=c width=320 height=240></canvas>
-<script>
-const FS_BODY = %s;
-const vs = `#version 300 es
-in vec2 p; void main(){ gl_Position = vec4(p,0.,1.); }`;
-const fsHead = `#version 300 es
-precision highp float; precision highp int;
-uniform vec3 iResolution; uniform float iTime; uniform float iTimeDelta; uniform int iFrame;
-uniform vec4 iMouse; uniform vec4 iDate; uniform float iSampleRate; uniform float iFrameRate;
-uniform float iChannelTime[4]; uniform vec3 iChannelResolution[4];
-uniform sampler2D iChannel0; uniform sampler2D iChannel1; uniform sampler2D iChannel2; uniform sampler2D iChannel3;
-out vec4 _fragColor;
-`;
-const fsTail = `
-void main(){ mainImage(_fragColor, gl_FragCoord.xy); }`;
-window.__result = {ok:false, err:null};
-(function(){
- const c = document.getElementById('c');
- const gl = c.getContext('webgl2', {preserveDrawingBuffer:true});
- if (!gl) { window.__result.err = 'no webgl2'; return; }
- function sh(t, src){ const s = gl.createShader(t); gl.shaderSource(s, src); gl.compileShader(s);
- if (!gl.getShaderParameter(s, gl.COMPILE_STATUS)) throw new Error(gl.getShaderInfoLog(s).split('\\n')[0]); return s; }
- try {
- const body = FS_BODY.replace(/^\\s*#version[^\\n]*\\n/, '');
- const p = gl.createProgram();
- gl.attachShader(p, sh(gl.VERTEX_SHADER, vs));
- gl.attachShader(p, sh(gl.FRAGMENT_SHADER, fsHead + body + fsTail));
- gl.linkProgram(p);
- if (!gl.getProgramParameter(p, gl.LINK_STATUS)) throw new Error(gl.getProgramInfoLog(p).split('\\n')[0]);
- gl.useProgram(p);
- const buf = gl.createBuffer(); gl.bindBuffer(gl.ARRAY_BUFFER, buf);
- gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1,-1, 3,-1, -1,3]), gl.STATIC_DRAW);
- const loc = gl.getAttribLocation(p, 'p'); gl.enableVertexAttribArray(loc); gl.vertexAttribPointer(loc, 2, gl.FLOAT, false, 0, 0);
- const tex = gl.createTexture(); gl.bindTexture(gl.TEXTURE_2D, tex);
- gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, new Uint8Array([128,128,128,255]));
- for (let i=0;i<4;i++){ const u = gl.getUniformLocation(p, 'iChannel'+i); if (u) gl.uniform1i(u, 0); }
- window.__draw = function(t){
- gl.uniform3f(gl.getUniformLocation(p,'iResolution'), c.width, c.height, 1);
- gl.uniform1f(gl.getUniformLocation(p,'iTime'), t);
- gl.uniform1f(gl.getUniformLocation(p,'iTimeDelta'), 1/60);
- gl.uniform1i(gl.getUniformLocation(p,'iFrame'), Math.floor(t*60));
- gl.uniform1f(gl.getUniformLocation(p,'iFrameRate'), 60);
- gl.viewport(0,0,c.width,c.height); gl.drawArrays(gl.TRIANGLES, 0, 3);
- const px = new Uint8Array(c.width*c.height*4);
- gl.readPixels(0,0,c.width,c.height, gl.RGBA, gl.UNSIGNED_BYTE, px);
- const seen = new Set(); let sum=0, sum2=0, n=0;
- for (let i=0;i<px.length;i+=4*7){ const r=px[i]>>4, g=px[i+1]>>4, b=px[i+2]>>4;
- seen.add((r<<8)|(g<<4)|b); const v=(px[i]+px[i+1]+px[i+2])/3; sum+=v; sum2+=v*v; n++; }
- const mean=sum/n; return {distinct: seen.size, std: Math.sqrt(Math.max(0, sum2/n - mean*mean)), mean};
- };
- window.__result.ok = true;
- } catch(e) { window.__result.err = String(e.message || e); }
-})();
-</script></body></html>"""
-
-
-def render_glsl(code, workdir, timeout=40):
- os.makedirs(workdir, exist_ok=True)
- html = HTML % json.dumps(code)
- p = os.path.join(workdir, "index.html")
- open(p, "w").write(html)
- rep = {"status": "FAIL", "error": None, "distinct": 0, "std": 0.0, "changed": False, "shot": None}
- try:
- from playwright.sync_api import sync_playwright
- with sync_playwright() as pw:
- b = pw.chromium.launch(args=["--use-gl=swiftshader", "--enable-unsafe-swiftshader", "--no-sandbox"])
- pg = b.new_page()
- errs = []
- pg.on("pageerror", lambda e: errs.append(str(e)[:200]))
- pg.goto("file://" + os.path.abspath(p), wait_until="load", timeout=timeout * 1000)
- res = pg.evaluate("window.__result")
- if not res or not res.get("ok"):
- rep["error"] = (res or {}).get("err") or (errs[0] if errs else "no result")
- b.close()
- return rep
- a = pg.evaluate("window.__draw(0.0)")
- c = pg.evaluate("window.__draw(1.7)")
- try: # a screenshot failure must not discard a good render
- shot = os.path.join(workdir, "frame.png")
- pg.screenshot(path=shot)
- rep["shot"] = shot
- except Exception:
- rep["shot"] = None
- b.close()
- rep["distinct"] = max(a["distinct"], c["distinct"])
- rep["std"] = round(max(a["std"], c["std"]), 2)
- rep["changed"] = abs(a["mean"] - c["mean"]) > 0.5 or a["distinct"] != c["distinct"]
- rep["status"] = "OK" if (rep["distinct"] > 3 and rep["std"] > 2.0) else "STATIC"
- if rep["status"] == "STATIC":
- rep["error"] = f"uniform image (distinct={rep['distinct']}, std={rep['std']})"
- except Exception as e:
- rep["error"] = str(e)[:200]
- return rep
-
-
-if __name__ == "__main__":
- import tempfile
- for f in sys.argv[1:]:
- r = render_glsl(open(f).read(), tempfile.mkdtemp())
- print(f"{r['status']:6s} distinct={r['distinct']:3d} std={r['std']:6.2f} animated={r['changed']} {os.path.basename(f)} {r['error'] or ''}")
diff --git a/finetune/3dcodeverse_eval/metrics.py b/finetune/3dcodeverse_eval/metrics.py
deleted file mode 100644
index c3c9160e..00000000
--- a/finetune/3dcodeverse_eval/metrics.py
+++ /dev/null
@@ -1,77 +0,0 @@
-"""Geometry metrics between generated GLB and the 3DCodeBench ground-truth GLB.
-Normalisation: center at bbox center, scale so max bbox extent = 1. Metrics: symmetric Chamfer-L2 distance (mean NN distance),
-F-score @0.05 / @0.1, best over 4 yaw rotations (objects may face any direction). Writes <gen_dir>/metrics.jsonl + summary.json"""
-import argparse, json, os, numpy as np
-import trimesh
-from scipy.spatial import cKDTree
-
-BENCH = os.environ.get("BENCH_DATA", "/wekafs/ict/hx_624/data/3dcodebench/data")
-
-def load_points(path, n):
- m = trimesh.load(path, force="mesh")
- if isinstance(m, trimesh.Scene): m = trimesh.util.concatenate(list(m.geometry.values()))
- if m.is_empty or len(m.faces) == 0: return None
- v = m.vertices
- c = (v.max(0) + v.min(0)) / 2; s = (v.max(0) - v.min(0)).max()
- if s <= 0: return None
- m.apply_translation(-c); m.apply_scale(1.0 / s)
- pts, _ = trimesh.sample.sample_surface(m, n)
- return np.asarray(pts)
-
-def chamfer_f(a, b, taus=(0.05, 0.1)):
- ta, tb = cKDTree(a), cKDTree(b)
- da, _ = tb.query(a); db, _ = ta.query(b)
- cd = float(da.mean() + db.mean())
- fs = {}
- for t in taus:
- p = float((da < t).mean()); r = float((db < t).mean())
- fs[f"f@{t}"] = 2 * p * r / (p + r) if p + r > 0 else 0.0
- return cd, fs
-
-def rot_y(p, k):
- th = np.pi / 2 * k; c, s = np.cos(th), np.sin(th)
- R = np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])
- return p @ R.T
-
-def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--gen_dir", required=True)
- ap.add_argument("--n_points", type=int, default=10000)
- a = ap.parse_args()
- np.random.seed(0)
- ex = {json.loads(l)["task"]: json.loads(l) for l in open(os.path.join(a.gen_dir, "exec_results.jsonl"))}
- tasks = sorted(os.listdir(BENCH))
- rows = []
- for t in tasks:
- r = {"task": t, "exec": ex.get(t, {}).get("status", "MISSING")}
- gen_glb = os.path.join(a.gen_dir, t, "out.glb"); gt_glb = os.path.join(BENCH, t, "glb", f"{t}.glb")
- if r["exec"] == "OK" and os.path.exists(gen_glb) and os.path.exists(gt_glb):
- try:
- pg = load_points(gen_glb, a.n_points); pt = load_points(gt_glb, a.n_points)
- if pg is not None and pt is not None:
- best = None
- for k in range(4):
- cd, fs = chamfer_f(rot_y(pg, k), pt)
- if best is None or cd < best[0]: best = (cd, fs)
- r.update(chamfer=best[0], **best[1])
- except Exception as e:
- r["metric_error"] = str(e)[:200]
- rows.append(r)
- with open(os.path.join(a.gen_dir, "metrics.jsonl"), "w") as f:
- for r in rows: f.write(json.dumps(r) + "\n")
- n = len(rows); ok = [r for r in rows if r["exec"] == "OK"]; sc = [r for r in rows if "chamfer" in r]
- # Failed / empty generations get the worst score for the aggregate (chamfer: mean of worst-10 for penalty; f: 0)
- summary = {
- "n_tasks": n, "exec_ok": len(ok), "exec_ok_rate": round(len(ok) / n, 4), "n_scored": len(sc),
- "chamfer_mean_scored": round(float(np.mean([r["chamfer"] for r in sc])), 4) if sc else None,
- "chamfer_median_scored": round(float(np.median([r["chamfer"] for r in sc])), 4) if sc else None,
- "[email protected]_mean_scored": round(float(np.mean([r["[email protected]"] for r in sc])), 4) if sc else None,
- "[email protected]_mean_scored": round(float(np.mean([r["[email protected]"] for r in sc])), 4) if sc else None,
- "[email protected]_mean_all(fail=0)": round(float(sum(r.get("[email protected]", 0) for r in rows) / n), 4),
- "[email protected]_mean_all(fail=0)": round(float(sum(r.get("[email protected]", 0) for r in rows) / n), 4),
- }
- json.dump(summary, open(os.path.join(a.gen_dir, "summary.json"), "w"), indent=2)
- print(json.dumps(summary, indent=2))
-
-if __name__ == "__main__":
- main()
diff --git a/finetune/3dcodeverse_eval/render_glb.py b/finetune/3dcodeverse_eval/render_glb.py
deleted file mode 100644
index 21fc6e21..00000000
--- a/finetune/3dcodeverse_eval/render_glb.py
+++ /dev/null
@@ -1,29 +0,0 @@
-"""Blender headless: render a GLB to PNG with an auto-framed 3/4 camera (workbench, solid shading).
-usage: blender -b --factory-startup -noaudio --python eval/render_glb.py -- --glb in.glb --out out.png [--size 320]"""
-import bpy, sys, math, argparse
-from mathutils import Vector
-argv = sys.argv[sys.argv.index("--")+1:] if "--" in sys.argv else []
-ap = argparse.ArgumentParser(); ap.add_argument("--glb"); ap.add_argument("--out"); ap.add_argument("--size", type=int, default=320); a = ap.parse_args(argv)
-bpy.ops.wm.read_factory_settings(use_empty=True)
-bpy.ops.import_scene.gltf(filepath=a.glb)
-meshes = [o for o in bpy.context.scene.objects if o.type == "MESH"]
-if not meshes: raise SystemExit("no mesh")
-# bounds
-pts = [o.matrix_world @ Vector(c) for o in meshes for c in o.bound_box]
-mn = Vector((min(p.x for p in pts), min(p.y for p in pts), min(p.z for p in pts))); mx = Vector((max(p.x for p in pts), max(p.y for p in pts), max(p.z for p in pts)))
-center = (mn + mx) / 2; radius = max((mx - mn).length / 2, 1e-3)
-cam_data = bpy.data.cameras.new("cam"); cam = bpy.data.objects.new("cam", cam_data); bpy.context.scene.collection.objects.link(cam)
-direction = Vector((1.0, -1.2, 0.8)).normalized(); cam.location = center + direction * radius * 2.6
-cam.rotation_euler = (center - cam.location).to_track_quat("-Z", "Y").to_euler(); cam_data.lens = 40; cam_data.clip_end = radius * 100
-bpy.context.scene.camera = cam
-sc = bpy.context.scene; sc.render.engine = "CYCLES"; sc.cycles.device = "CPU"; sc.cycles.samples = 32; sc.cycles.use_denoising = False
-sc.render.resolution_x = a.size; sc.render.resolution_y = a.size; sc.render.film_transparent = False
-w = bpy.data.worlds.new("w"); w.use_nodes = True; bg = w.node_tree.nodes["Background"]; bg.inputs[0].default_value = (0.92, 0.92, 0.92, 1); bg.inputs[1].default_value = 1.0; sc.world = w
-sun_d = bpy.data.lights.new("sun", "SUN"); sun_d.energy = 3.0; sun = bpy.data.objects.new("sun", sun_d); sc.collection.objects.link(sun)
-sun.rotation_euler = (math.radians(45), math.radians(20), math.radians(30))
-import random; random.seed(0)
-for o in meshes:
- mat = bpy.data.materials.new(f"m_{o.name}"); mat.use_nodes = True; bsdf = mat.node_tree.nodes.get("Principled BSDF")
- if bsdf: bsdf.inputs["Base Color"].default_value = (random.uniform(0.25, 0.85), random.uniform(0.25, 0.85), random.uniform(0.25, 0.85), 1); bsdf.inputs["Roughness"].default_value = 0.6
- o.data.materials.clear(); o.data.materials.append(mat)
-sc.render.filepath = a.out; bpy.ops.render.render(write_still=True)
diff --git a/finetune/3dcodeverse_eval/results_all.py b/finetune/3dcodeverse_eval/results_all.py
deleted file mode 100644
index 64a9273e..00000000
--- a/finetune/3dcodeverse_eval/results_all.py
+++ /dev/null
@@ -1,59 +0,0 @@
-"""Aggregate every suite of one eval_all run (or compare several runs) into a table.
-
- python eval/results_all.py <name> [<name2> ...] -> markdown table + eval/out/<name>_report.json
-"""
-import json
-import os
-import sys
-
-SUITES = [("bench", "3DCodeBench (212)"), ("blender", "Blender held-out (103)"), ("cadquery", "CadQuery (200)"),
- ("openscad", "OpenSCAD (50)"), ("glsl", "GLSL (200)"), ("threejs", "three.js (40)")]
-ROOT = os.environ.get("EVAL_OUT", "/wekafs/ict/hx_624/llm-ft/eval/out")
-
-
-def suite_row(name, suite):
- d = os.path.join(ROOT, f"{name}_{suite}")
- s_path, x_path = os.path.join(d, "summary.json"), os.path.join(d, "extract_stats.json")
- if not os.path.exists(s_path):
- return None
- s = json.load(open(s_path))
- x = json.load(open(x_path)) if os.path.exists(x_path) else {}
- if suite == "bench":
- rate, f05, f01 = s.get("exec_ok_rate"), s.get("[email protected]_mean_all(fail=0)"), s.get("[email protected]_mean_scored")
- n_ok, n = s.get("exec_ok"), s.get("n_tasks")
- elif suite == "threejs":
- rate, f05, f01 = s.get("exec_rate"), None, None
- n_ok, n = s.get("exec_ok"), s.get("n")
- else:
- rate, f05, f01 = s.get("gen_exec_rate"), s.get("[email protected]_mean_all"), s.get("[email protected]_mean_scored")
- n_ok, n = s.get("gen_exec_ok"), s.get("n")
- return {"suite": suite, "n": n, "exec_ok": n_ok, "exec_rate": rate, "[email protected]_all": f05, "[email protected]_scored": f01,
- "mean_new_tokens": x.get("mean_new_tokens"), "with_think": x.get("with_think"), "no_fence": x.get("no_fence"),
- "multi_block": x.get("multi_block"), "chosen_not_last": x.get("chosen_not_last"),
- "syntax_valid": x.get("syntax_valid"), "no_dialect_marker": x.get("no_dialect_marker"),
- "from_think_fallback": x.get("from_think_fallback")}
-
-
-def main(names):
- print("| model | suite | exec | rate | [email protected](all) | [email protected](ok) | out tok | think | no-fence | multi-blk | not-last | no-marker |")
- print("|---|---|---|---|---|---|---|---|---|---|---|---|")
- for name in names:
- rows = []
- for suite, _label in SUITES:
- r = suite_row(name, suite)
- if not r:
- continue
- rows.append(r)
- pct = f"{100*r['exec_rate']:.1f}%" if r["exec_rate"] is not None else "–"
- f05 = f"{r['[email protected]_all']:.3f}" if r["[email protected]_all"] is not None else "–"
- f01 = f"{r['[email protected]_scored']:.3f}" if r["[email protected]_scored"] is not None else "–"
- print(f"| {name} | {suite} | {r['exec_ok']}/{r['n']} | {pct} | {f05} | {f01} | "
- f"{r['mean_new_tokens'] or '–'} | {r['with_think'] if r['with_think'] is not None else '–'} | "
- f"{r['no_fence'] if r['no_fence'] is not None else '–'} | {r['multi_block'] if r['multi_block'] is not None else '–'} | "
- f"{r['chosen_not_last'] if r['chosen_not_last'] is not None else '–'} | {r['no_dialect_marker'] if r['no_dialect_marker'] is not None else '–'} |")
- if rows:
- json.dump({"name": name, "suites": rows}, open(os.path.join(ROOT, f"{name}_report.json"), "w"), indent=2)
-
-
-if __name__ == "__main__":
- main(sys.argv[1:] or ["md_xl"])
diff --git a/finetune/3dcodeverse_eval/run_bench.py b/finetune/3dcodeverse_eval/run_bench.py
deleted file mode 100644
index e1409d7a..00000000
--- a/finetune/3dcodeverse_eval/run_bench.py
+++ /dev/null
@@ -1,49 +0,0 @@
-"""Execute every <gen_dir>/<task>/code.py in headless Blender (parallel), export GLB, collect reports -> <gen_dir>/exec_results.jsonl"""
-import os
-import argparse, json, os, subprocess, sys, time
-from concurrent.futures import ThreadPoolExecutor, as_completed
-BLENDER = os.environ.get("BLENDER_BIN", "/wekafs/ict/hx_624/tools/blender-5.0.1-linux-x64/blender")
-RUNNER = os.path.join(os.path.dirname(os.path.abspath(__file__)), "blender_runner.py")
-
-def run_one(task_dir, timeout):
- task = os.path.basename(task_dir)
- script = os.path.join(task_dir, "code.py"); glb = os.path.join(task_dir, "out.glb"); rep = os.path.join(task_dir, "exec.json")
- if os.path.exists(glb): os.remove(glb)
- cmd = [BLENDER, "-b", "--factory-startup", "-noaudio", "--python", RUNNER, "--", "--script", script, "--out", glb, "--report", rep]
- t0 = time.time()
- try:
- p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=timeout, text=True, errors="replace",
- env={**os.environ, "HOME": "/tmp", "XDG_CONFIG_HOME": "/tmp/.bcfg"})
- if os.path.exists(rep):
- r = json.load(open(rep))
- else:
- r = {"status": "CRASH", "error": p.stdout[-1500:], "latency_s": round(time.time() - t0, 1)}
- except subprocess.TimeoutExpired:
- r = {"status": "TIMEOUT", "error": f"timeout {timeout}s", "latency_s": timeout}
- r["task"] = task
- return r
-
-def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--gen_dir", required=True)
- ap.add_argument("--workers", type=int, default=16)
- ap.add_argument("--timeout", type=int, default=300)
- a = ap.parse_args()
- a.gen_dir = os.path.abspath(a.gen_dir) # the Blender runner chdir()s into the script dir -> all paths must be absolute
- tasks = sorted(d for d in os.listdir(a.gen_dir) if os.path.isfile(os.path.join(a.gen_dir, d, "code.py")))
- print(f"[exec] {len(tasks)} scripts, {a.workers} workers", flush=True)
- res = []
- with ThreadPoolExecutor(a.workers) as ex:
- futs = {ex.submit(run_one, os.path.join(a.gen_dir, t), a.timeout): t for t in tasks}
- for i, fu in enumerate(as_completed(futs)):
- r = fu.result(); res.append(r)
- if (i + 1) % 20 == 0: print(f"[exec] {i+1}/{len(tasks)}", flush=True)
- res.sort(key=lambda r: r["task"])
- with open(os.path.join(a.gen_dir, "exec_results.jsonl"), "w") as f:
- for r in res: f.write(json.dumps(r) + "\n")
- from collections import Counter
- c = Counter(r["status"] for r in res)
- print("[exec] status counts:", dict(c), f"| OK rate = {c.get('OK',0)}/{len(res)} = {100*c.get('OK',0)/max(len(res),1):.1f}%")
-
-if __name__ == "__main__":
- main()
diff --git a/finetune/3dcodeverse_eval/run_eval.sh b/finetune/3dcodeverse_eval/run_eval.sh
deleted file mode 100755
index 20d315c1..00000000
--- a/finetune/3dcodeverse_eval/run_eval.sh
+++ /dev/null
@@ -1,112 +0,0 @@
-#!/bin/bash
-# 3DCodeVerse evaluation: one command scores ANY model by EXECUTING what it generates -- 3DCodeBench
-# (Blender exec + geometry against ground truth) and five held-out dialect sets (CadQuery / OpenSCAD /
-# GLSL / Blender / three.js), with dialect-aware code extraction.
-#
-# GPUS=0 TP=1 ./run_eval.sh <model_dir> <name> [--think] [--temp 0.7] [--seed 1] [--max_new 32768] [--suites ...]
-#
-# --think keep the model's reasoning on (default: enable_thinking=false)
-# --temp/--seed sampling (default greedy). Report long-output dialects under BOTH -- greedy alone
-# understates them badly when a model runs away.
-# --suites subset of: bench cadquery openscad glsl blender threejs (default: all)
-#
-# Paths are env vars so this folder runs on another box: PROJECT_ROOT (holds data/), EVAL_OUT,
-# BLENDER_BIN, OPENSCAD_BIN, GLSLANG_BIN, BENCH_DATA, VLLM_ENV, EXEC_ENV. See env.sh and README.md.
-# results: $EVAL_OUT/<name>_<suite>/summary.json + $EVAL_OUT/<name>_report.json (one row per suite)
-set -uo pipefail
-HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-PROJECT_ROOT="${PROJECT_ROOT:-/wekafs/ict/hx_624/llm-ft}"
-export EVAL_OUT="${EVAL_OUT:-$PROJECT_ROOT/eval/out}"
-VLLM_ENV="${VLLM_ENV:-vllm}"; EXEC_ENV="${EXEC_ENV:-llmft}"
-CONDA_SH="${CONDA_SH:-/wekafs/ict/hx_624/anaconda3/etc/profile.d/conda.sh}"
-mkdir -p "$EVAL_OUT"
-MODEL=$(readlink -f "$1"); NAME=$2; shift 2
-GPU=${GPUS:-0}; TP=${TP:-1}; THINK=""; TEMP=0.0; SEED=0; MAXNEW=32768 # the models take 262k context and our prompts are ~300 tokens; a low cap silently turns "unfinished" into "wrong"
-SUITES="bench cadquery openscad glsl blender threejs"
-while [ $# -gt 0 ]; do
- case "$1" in
- --think) THINK="1"; shift;;
- --temp) TEMP=$2; shift 2;;
- --seed) SEED=$2; shift 2;;
- --max_new) MAXNEW=$2; shift 2;;
- --suites) shift; SUITES=""; while [ $# -gt 0 ] && [[ "$1" != --* ]]; do SUITES="$SUITES $1"; shift; done;;
- *) echo "unknown arg $1"; exit 2;;
- esac
-done
-cd "$PROJECT_ROOT"
-source "$CONDA_SH"
-export VLLM_CACHE_ROOT=/wekafs/ict/hx_624/cache/vllm HF_HOME=/wekafs/ict/hx_624/cache/huggingface VLLM_LOGGING_LEVEL=WARNING
-export PLAYWRIGHT_BROWSERS_PATH=/wekafs/ict/hx_624/cache/ms-playwright CUDA_VISIBLE_DEVICES=$GPU
-NOTHINK="--no_think"; [ -n "$THINK" ] && NOTHINK=""
-MAXLEN=$((MAXNEW + 4096))
-echo "[eval_all] $(date) model=$MODEL name=$NAME gpus=$GPU tp=$TP think=${THINK:-0} temp=$TEMP suites=$SUITES"
-
-# ---------- generation: ONE engine for all suites (loading a 27B per suite costs more than generating) ----------
-conda activate "$VLLM_ENV"
-SPEC=$(mktemp /tmp/hx_624_evalspec_XXXX.json)
-python - "$NAME" "$SUITES" "$SPEC" <<'PY'
-import json, sys, os
-name, suites, spec_path = sys.argv[1], sys.argv[2].split(), sys.argv[3]
-jobs = []
-for s in suites:
- out = f"{os.environ['EVAL_OUT']}/{name}_{s}"; os.makedirs(out, exist_ok=True)
- if s == "bench":
- prompts, dia = "data/sft_v1/bench_prompts.jsonl", "blender"
- elif s == "threejs":
- prompts, dia = os.environ["EVAL_OUT"] + "/_3js_prompts.jsonl", "threejs"
- else:
- prompts, dia = f"{out}/prompts.jsonl", s
- with open(prompts, "w") as f:
- for l in open(f"data/multidialect/{s}/test.jsonl"):
- r = json.loads(l); f.write(json.dumps({"task": r["id"].replace("/", "__"), "messages": r["messages"][:2]}) + "\n")
- jobs.append({"name": s, "prompts": prompts, "out": out, "dialect": dia})
-json.dump(jobs, open(spec_path, "w"))
-print("[eval_all] suites:", [j["name"] for j in jobs])
-PY
-# On a shared box a card that was free when it was picked can be full by the time vLLM starts, and the engine
-# dies with no generations at all. Re-check right before launching, and fall back to whichever card is free now.
-for ATTEMPT in 1 2 3; do
- FREE_NOW=$("$HERE"/free_gpus.sh 40000)
- case ",$FREE_NOW," in
- *",$GPU,"*) ;;
- *) NEW=$(echo "$FREE_NOW" | cut -d, -f1)
- if [ -n "$NEW" ]; then
- echo "[eval_all] GPU $GPU filled up since it was picked; switching to $NEW"
- GPU=$NEW; export CUDA_VISIBLE_DEVICES=$GPU
- else
- echo "[eval_all] no card has 40 GB free; waiting 120s (attempt $ATTEMPT/3)"; sleep 120; continue
- fi;;
- esac
- python "$HERE"/generate_multi.py --model $MODEL --spec @$SPEC --tp $TP $NOTHINK \
- --max_new_tokens $MAXNEW --max_model_len $MAXLEN --temperature $TEMP --seed $SEED 2>&1 \
- | grep -E "gen-multi|extract\]|Error|Traceback" | tail -40
- # one generation directory per task; anything less means the engine never ran
- if [ "$(find $EVAL_OUT/${NAME}_$(echo $SUITES | awk '{print $1}') -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l)" -ge 5 ]; then break; fi
- echo "[eval_all] generation produced nothing on attempt $ATTEMPT; retrying"
- sleep 60
-done
-rm -f $SPEC
-
-# ---------- execution + metrics (llmft env) ----------
-conda activate "$EXEC_ENV"
-for S in $SUITES; do
- OUT=$EVAL_OUT/${NAME}_${S}
- # An engine that never started leaves an empty directory, and every scorer below reports that as 0%. A real
- # 0% and a failed launch then look identical -- which is how a rescore of two healthy models came back as
- # 0/200 on every suite. Refuse to score instead.
- NGEN=$(find $OUT -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l)
- if [ "${NGEN:-0}" -lt 5 ]; then
- echo "[eval_all] $S PRODUCED NOTHING ($NGEN generations) — not scoring. See the generation log above."
- continue
- fi
- echo "[eval_all] exec $S"
- case $S in
- bench) python "$HERE"/run_bench.py --gen_dir $OUT --workers 32 --timeout 300 2>&1 | grep -E "status counts" | tail -1
- python "$HERE"/metrics.py --gen_dir $OUT 2>&1 | tail -1;;
- blender) python "$HERE"/blender_dialect_eval.py --test data/multidialect/blender/test.jsonl --gen_dir $OUT --workers 32 2>&1 | tail -1;;
- threejs) python "$HERE"/threejs_eval.py --gen_dir $OUT 2>&1 | tail -2;;
- *) python "$HERE"/dialect_eval.py --dialect $S --test data/multidialect/$S/test.jsonl --gen_dir $OUT --workers 32 2>&1 | tail -1;;
- esac
-done
-python "$HERE"/results_all.py $NAME
-echo "[eval_all] DONE $NAME"
diff --git a/finetune/3dcodeverse_eval/threejs_eval.py b/finetune/3dcodeverse_eval/threejs_eval.py
deleted file mode 100644
index ab969ad3..00000000
--- a/finetune/3dcodeverse_eval/threejs_eval.py
+++ /dev/null
@@ -1,17 +0,0 @@
-"""Run every <gen_dir>/<task>/code.py (HTML) through the headless-browser runner (parallel) and summarise."""
-import argparse, json, os, sys
-from concurrent.futures import ThreadPoolExecutor
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))); from threejs_runner import run_threejs
-ap = argparse.ArgumentParser(); ap.add_argument("--gen_dir", required=True); ap.add_argument("--workers", type=int, default=8); a = ap.parse_args()
-tasks = sorted(d for d in os.listdir(a.gen_dir) if os.path.isfile(os.path.join(a.gen_dir, d, "code.py")))
-def job(t):
- html = open(os.path.join(a.gen_dir, t, "code.py")).read()
- r = run_threejs(html, os.path.join(a.gen_dir, t, "exec")); r["task"] = t; r.pop("shot", None); return r
-with ThreadPoolExecutor(a.workers) as ex: res = list(ex.map(job, tasks))
-from collections import Counter
-c = Counter(r["status"] for r in res)
-with open(os.path.join(a.gen_dir, "threejs_results.jsonl"), "w") as f:
- for r in res: f.write(json.dumps(r) + "\n")
-summ = {"dialect": "threejs", "n": len(res), "exec_ok": c.get("OK", 0), "exec_rate": round(c.get("OK", 0) / max(1, len(res)), 4), "counts": dict(c)}
-json.dump(summ, open(os.path.join(a.gen_dir, "summary.json"), "w"), indent=2); print(json.dumps(summ))
-print("failure modes:", Counter((r["error"] or r["status"])[:80] for r in res if r["status"] != "OK").most_common(6))
diff --git a/finetune/3dcodeverse_eval/threejs_runner.py b/finetune/3dcodeverse_eval/threejs_runner.py
deleted file mode 100644
index c44f6a36..00000000
--- a/finetune/3dcodeverse_eval/threejs_runner.py
+++ /dev/null
@@ -1,56 +0,0 @@
-"""Headless-Chromium executor for single-file three.js HTML (Playwright). Returns dict(status, error, console_errors, canvas_nonblank, latency_s, shot).
-status OK = page loaded, no uncaught JS error / failed module import, a <canvas> exists and its rendered pixels are not (nearly) uniform."""
-import os, sys, json, time, base64, io
-def run_threejs(html, workdir, timeout=40, settle_ms=4000):
- os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/wekafs/ict/hx_624/cache/ms-playwright")
- workdir = os.path.abspath(workdir); os.makedirs(workdir, exist_ok=True)
- hp = os.path.join(workdir, "index.html"); open(hp, "w").write(html)
- rep = {"status": "FAIL", "error": None, "console_errors": [], "canvas_nonblank": None, "latency_s": 0, "shot": None}
- t0 = time.time()
- try:
- from playwright.sync_api import sync_playwright
- with sync_playwright() as p:
- browser = p.chromium.launch(headless=True, args=["--use-gl=angle", "--use-angle=swiftshader", "--enable-unsafe-swiftshader", "--ignore-gpu-blocklist", "--no-sandbox", "--disable-dev-shm-usage"])
- page = browser.new_page(viewport={"width": 640, "height": 480})
- errs = []
- page.on("pageerror", lambda e: errs.append("pageerror: " + str(e)[:300]))
- page.on("console", lambda m: errs.append("console.error: " + m.text[:300]) if m.type == "error" else None)
- page.on("requestfailed", lambda r: errs.append("requestfailed: " + r.url[:200]))
- page.goto("file://" + hp, wait_until="load", timeout=timeout * 1000)
- page.wait_for_timeout(settle_ms)
- # canvas check: count distinct pixels over a downsampled grid
- info = page.evaluate("""() => {
- const cs = document.querySelectorAll('canvas'); if (!cs.length) return {n:0};
- const c = cs[0]; const w=c.width, h=c.height; if (!w||!h) return {n:cs.length, w, h, distinct:0};
- let gl = c.getContext('webgl2') || c.getContext('webgl');
- let pix = null;
- if (gl) { pix = new Uint8Array(w*h*4); try { gl.readPixels(0,0,w,h,gl.RGBA,gl.UNSIGNED_BYTE,pix); } catch(e) { pix=null; } }
- if (!pix) { try { const t=document.createElement('canvas'); t.width=w; t.height=h; const ctx=t.getContext('2d'); ctx.drawImage(c,0,0); pix=ctx.getImageData(0,0,w,h).data; } catch(e) { return {n:cs.length, w, h, distinct:-1, err:String(e)}; } }
- const seen = new Set(); const step = Math.max(1, Math.floor((w*h)/4000));
- for (let i=0;i<w*h;i+=step){ const o=i*4; seen.add((pix[o]>>3)+','+(pix[o+1]>>3)+','+(pix[o+2]>>3)); if (seen.size>64) break; }
- return {n:cs.length, w, h, distinct:seen.size};
- }""")
- shot = os.path.join(workdir, "shot.png"); page.screenshot(path=shot); rep["shot"] = shot
- browser.close()
- rep["console_errors"] = errs[:10]
- fatal = [e for e in errs if e.startswith("pageerror") or "Failed to resolve module" in e or "Uncaught" in e or e.startswith("requestfailed")]
- # blank check from the composited screenshot (the WebGL drawing buffer is cleared after compositing, so in-page readPixels is unreliable)
- try:
- from PIL import Image; import numpy as np
- im = np.array(Image.open(shot).convert("RGB")); q = (im[::6, ::6].reshape(-1, 3) // 16)
- distinct = len(set(map(tuple, q))); rep["shot_distinct"] = distinct; rep["shot_std"] = float(im.std())
- nonblank = distinct > 6 and float(im.std()) > 4.0
- except Exception as e:
- nonblank = info.get("distinct", 0) > 3
- rep["canvas_nonblank"] = bool(info.get("n", 0)) and nonblank
- if fatal: rep["status"] = "FAIL"; rep["error"] = fatal[0]
- elif not info.get("n"): rep["status"] = "EMPTY"; rep["error"] = "no <canvas>"
- elif not rep["canvas_nonblank"]: rep["status"] = "EMPTY"; rep["error"] = f"blank canvas (shot distinct={rep.get('shot_distinct')})"
- else: rep["status"] = "OK"
- except Exception as e:
- rep["error"] = (type(e).__name__ + ": " + str(e))[:400]
- if "Timeout" in rep["error"]: rep["status"] = "TIMEOUT"
- rep["latency_s"] = round(time.time() - t0, 1)
- return rep
-if __name__ == "__main__":
- print(json.dumps(run_threejs(open(sys.argv[1]).read(), sys.argv[2]), indent=1))
diff --git a/harness/CLAUDE.md b/harness/CLAUDE.md
index 5e0e4c6d..5e0f1eb3 100644
--- a/harness/CLAUDE.md
+++ b/harness/CLAUDE.md
@@ -17,8 +17,8 @@ data-flywheel record per run.
Read `docs/ARCHITECTURE.md` (design, package map, authoring contracts, what a run
really does, known limits), `docs/INTERFACES.md` (binding cross-package signatures as
built), `docs/RUNBOOK.md` (how to run / resume / extend), `docs/DECISIONS.md`
-(ADR-style laws + deviations), `docs/EVAL.md` (evaluation protocol + judge
-calibration) and `docs/COMPLEXITY.md` (the objective complexity vector + the
+(ADR-style laws + deviations), `eval/docs/EVAL.md` (evaluation protocol + judge
+calibration) and `eval/docs/COMPLEXITY.md` (the objective complexity vector + the
score-vs-complexity corpus study) before changing anything.
## Laws (do not break)
@@ -40,7 +40,7 @@ score-vs-complexity corpus study) before changing anything.
## Supported versions (what other people can run this on)
python **3.13** (one fixed version — owner's decision 2026-08-26; no floor, no matrix) ·
node **20.6+** · Blender 4.2+ · Linux x86_64. **There is no CI** (removed 2026-08-26 by the
-owner): run `ruff check codeverse bench tests` and the offline suite locally BEFORE every
+owner): run `ruff check codeverse tests` and the offline suite locally BEFORE every
push, with `set -o pipefail` so a `| tail` cannot swallow a red exit. The version lives in
`pyproject` `requires-python` + ruff `target-version` + `setup.sh`, and
`spatial/node.py:NODE_MIN` + `runtime_js/package.json` `engines` for node, pinned together by
@@ -92,17 +92,17 @@ cd /home/yipeng/3dcodeverse/harness
3dcode resume <slug> · 3dcode status <slug> · 3dcode render <slug> [--mode wire] · 3dcode judge <slug> [--model ... --n 3]
3dcode texture pass <slug> [--no-judge] · 3dcode texture scene-pack <slug> · 3dcode texture show <slug>
3dcode tools list · 3dcode tools measure --workspace runs/<slug> · 3dcode mcp --workspace runs/<slug>
-3dcode cost <slug> · 3dcode cost --runs-dir bench/out/<battery> · 3dcode cost cache <slug>
+3dcode cost <slug> · 3dcode cost --runs-dir eval/bench/out/<battery> · 3dcode cost cache <slug>
3dcode cost prices [--stale] · 3dcode cost profiles · 3dcode cost estimate gemini:gemini-3.1-pro-preview --in 12000
3dcode flywheel export runs/ dataset/ [--pack --drop-duplicates --captions-dir d/] · 3dcode flywheel pairs runs/ pairs.jsonl
3dcode flywheel caption <slug> [--out dir] # the gallery is `3dcode gallery build --embed`
3dcode gallery serve [ROOTS...] [--port 8765] [--reload] · 3dcode gallery build --out gallery.html [--embed]
-3dcode bench run bench/prompts/static_objects_v1.yaml --generator ... --judge gemini:gemini-3.1-pro-preview
+3dcode bench run eval/bench/prompts/static_objects_v1.yaml --generator ... --judge gemini:gemini-3.1-pro-preview
python -m codeverse.addons.calibration runs/<slug>... --model gemini:gemini-3.1-pro-preview --n 3 --out out/
-python bench/complexity_report.py bench/out --recursive # score-vs-complexity + $/complexity point (docs/COMPLEXITY.md)
-python bench/compare_backends.py --prompts bench/prompts/compare_v1.yaml --arms harness:gemini-cli:gemini-3.6-flash,oneshot:claude-code --judge gemini:gemini-3.1-pro-preview --out bench/out/compare_v1
-python -m pytest tests -q -m "not live" # 2 938 tests, ~65 s (real Blender + headless Chrome + CadQuery)
-python -m pytest tests -q -m "not live and not blender and not node" # pure python: 2 390 tests, ~40 s
+python eval/bench/complexity_report.py eval/bench/out --recursive # score-vs-complexity + $/complexity point (eval/docs/COMPLEXITY.md)
+python eval/bench/compare_backends.py --prompts eval/bench/prompts/compare_v1.yaml --arms harness:gemini-cli:gemini-3.6-flash,oneshot:claude-code --judge gemini:gemini-3.1-pro-preview --out eval/bench/out/compare_v1
+python -m pytest tests -q -m "not live" # 2 714 tests, ~65 s (real Blender + headless Chrome + CadQuery)
+python -m pytest tests -q -m "not live and not blender and not node" # pure python: 2 166 tests, ~40 s (the 224 bench tests: `cd eval && python -m pytest`)
# (the counts drift every commit — `--collect-only` is the answer, not a number in this file)
# both run PARALLEL by default (pytest-xdist, -n auto --dist worksteal, in pyproject addopts).
# A nested pytest inside a test MUST pass -n0 or it forks another full set of workers.
diff --git a/harness/README.md b/harness/README.md
index 789e4b78..1ba47552 100644
--- a/harness/README.md
+++ b/harness/README.md
@@ -25,7 +25,7 @@ renders, judges, refines, textures and records every run as data-flywheel materi
quality tiers + dedupe (+ parquet, tar locators), preference/repair/trajectory
pairs, captions, sqlite index, HTML gallery.
* **Looking at results:** `3dcode gallery serve` — a local page over `runs/` +
- `bench/out/*/runs` with filters, a per-filter summary strip and a detail page per run;
+ `eval/bench/out/*/runs` with filters, a per-filter summary strip and a detail page per run;
it serves the run directories too, so every link (sheet, renders, `src/`, `object.glb`
in an orbit viewer, `record.json`) actually opens. `3dcode gallery build --embed` writes
the same page as one shareable file.
@@ -63,7 +63,7 @@ codeverse/
gallery (the runs browser), dataset (export, tiers, preference/repair pairs,
captions, sqlite index), costreport (`3dcode cost`), calibration, skill_targets
cli/ the typer CLI · doctor.py the environment checks behind `3dcode doctor`
-bench/ the evaluation harness around the harness: run_bench, compare_backends
+eval/bench/ the evaluation harness around the harness: run_bench, compare_backends
(A/B matrix), ab_plan, infra-failure classification
runtime_js/ node side: [email protected] + headless-Chrome render/probe hosts
```
@@ -97,4 +97,4 @@ multi-pass OpenGL program (gemini-cli) 0.70 → 0.86 pass, $0.74; neon-rain shad
Docs: `docs/INSTALL.md` (install / prerequisites / doctor troubleshooting) ·
`docs/ARCHITECTURE.md` (design + what a run does) · `docs/INTERFACES.md`
(signatures) · `docs/RUNBOOK.md` (operate / extend) · `docs/DECISIONS.md` (ADRs) ·
-`docs/EVAL.md` (evaluation protocol + judge calibration) · `CLAUDE.md` (working rules).
+`eval/docs/EVAL.md` (evaluation protocol + judge calibration) · `CLAUDE.md` (working rules).
diff --git a/harness/codeverse/cli/_common.py b/harness/codeverse/cli/_common.py
index c3082772..fb746247 100644
--- a/harness/codeverse/cli/_common.py
+++ b/harness/codeverse/cli/_common.py
@@ -160,13 +160,17 @@ def lazy(module: str, attr: str | None = None) -> Any:
raise CliError(f"{module}.{attr} is missing ({e}); the sub-package may be incomplete.", code=2) from e
+#: the evaluation code lives NEXT TO the harness (``<repo>/eval``), not inside it
+EVAL_ROOT = REPO_ROOT.parent / "eval"
+
+
def import_bench() -> Any:
- """``bench`` lives at the repo root (not inside the package); make it importable."""
+ """``bench`` is ``<repo>/eval/bench`` (not part of the package); make it importable."""
try:
return importlib.import_module("bench")
except ImportError:
- if str(REPO_ROOT) not in sys.path:
- sys.path.insert(0, str(REPO_ROOT))
+ if str(EVAL_ROOT) not in sys.path:
+ sys.path.insert(0, str(EVAL_ROOT))
return lazy("bench")
diff --git a/harness/codeverse/cli/main.py b/harness/codeverse/cli/main.py
index c571e68e..54a8823e 100644
--- a/harness/codeverse/cli/main.py
+++ b/harness/codeverse/cli/main.py
@@ -94,8 +94,8 @@ bench_app = typer.Typer(no_args_is_help=True)
@bench_app.command("run")
def run_cmd(
- battery: Annotated[Path, typer.Argument(help="bench/prompts/<name>.yaml")],
- out: Annotated[Path | None, typer.Option("--out", help="default bench/out/<battery name>")] = None,
+ battery: Annotated[Path, typer.Argument(help="eval/bench/prompts/<name>.yaml")],
+ out: Annotated[Path | None, typer.Option("--out", help="default eval/bench/out/<battery name>")] = None,
generator: Annotated[str | None, typer.Option("--generator")] = None,
planner: Annotated[str | None, typer.Option("--planner")] = None,
judge: Annotated[str | None, typer.Option("--judge", help="fixed judge model for the whole battery")] = None,
@@ -121,7 +121,7 @@ def run_cmd(
opts = run_bench.BenchOptions(generator=generator, planner=planner, judge=judge, rounds=rounds, max_minutes=max_minutes,
limit=limit, ids=ids or [], tiers=tiers or [],
redo_status=[x for x in redo_status.split(",") if x], **par)
- out_dir = out or (C.REPO_ROOT / "bench" / "out" / battery.stem)
+ out_dir = out or (C.EVAL_ROOT / "bench" / "out" / battery.stem)
console.print(f"battery={battery} out={out_dir} generator={generator or 'default'} judge={judge or 'default'}")
def _on(res) -> None:
diff --git a/harness/docs/ARCHITECTURE.md b/harness/docs/ARCHITECTURE.md
index e30f19eb..07cabeb7 100644
--- a/harness/docs/ARCHITECTURE.md
+++ b/harness/docs/ARCHITECTURE.md
@@ -6,8 +6,8 @@ runtime (`runtime_js/`) for everything Three.js / headless Chrome, and moderngl
for the graphics track. Reconciled against the code and the live runs on
2026-08-23 (waves 2–3 + fix batch 1); design history and deviations are in
`docs/DECISIONS.md`, how-to in `docs/RUNBOOK.md`, evaluation protocol in
-`docs/EVAL.md`, complexity measurement + the score-vs-complexity study in
-`docs/COMPLEXITY.md`, binding signatures in `docs/INTERFACES.md`.
+`eval/docs/EVAL.md`, complexity measurement + the score-vs-complexity study in
+`eval/docs/COMPLEXITY.md`, binding signatures in `docs/INTERFACES.md`.
## 0. What it is
@@ -129,7 +129,7 @@ codeverse/
gl_render.py (GlHost), frame_stats.py (gl_frames),
sheet.py (montage_2x2, crop_region), measure.py, connectivity.py,
contract.py (authoring-frame hints), sections.py, silhouette.py, probes.py,
- complexity.py (objective complexity vector -> Measurement.extra, docs/COMPLEXITY.md),
+ complexity.py (objective complexity vector -> Measurement.extra, eval/docs/COMPLEXITY.md),
joints*.py + joints_collide.py (deterministic penetration), registry.py,
tools.py (every @tool registration since 2026-08-28, the joint_sweep body included;
spatial siblings are plain imports — lazy() guards only codeverse.languages /
@@ -189,7 +189,7 @@ codeverse/
(CV3D_PLAN_BRIEF), plan budgets and the worked examples),
plan_features.py (CV3D_PLAN_FEATURES: one switch per plan-loop change, so each
can be A/B'd alone, + pin_plan_blockers() deciding when two arms may share
- one plan — docs/EVAL.md §8.1),
+ one plan — eval/docs/EVAL.md §8.1),
depth.py,
skills_hook.py (the round's view of codeverse/skills: attach before generating,
probe reads after — a no-op unless CV3D_SKILLS is on)
@@ -226,9 +226,9 @@ codeverse/
flywheel_cmd.py, texture_cmd.py, cost_cmd.py (`3dcode cost`), layout_cmd.py, doctor.py
(`--skills` checks the library + its discovery wiring),
skills_cmd.py (`3dcode skills list|show|validate|report` — the read-rate report)
-bench/ run_bench.py, report.py (renders through codeverse/addons/gallery), compare_backends.py
+eval/bench/ run_bench.py, report.py (renders through codeverse/addons/gallery), compare_backends.py
(preflights every model it needs; --wait-for-provider / --no-preflight),
- _infra.py (outage vs model failure: infra_failed / budget_exhausted, docs/EVAL.md §7),
+ _infra.py (outage vs model failure: infra_failed / budget_exhausted, eval/docs/EVAL.md §7),
pin_plan.py (seed one plan into both arms so the paired delta stops carrying
the planner's spread — permitted only by plan_features.pin_plan_blockers),
_compare_report.py (arm table incl. the `dropped` / `over budget` loss columns;
@@ -564,7 +564,7 @@ side-car mode); the gallery is `3dcode gallery build --embed`
(the flywheel package has no renderer of its own).
`codeverse/addons/gallery/` is the **local** answer to the same question: `3dcode gallery
-serve` indexes `runs/` + every `bench/out/*/runs`, serves the page **and the run
+serve` indexes `runs/` + every `eval/bench/out/*/runs`, serves the page **and the run
directories** on 127.0.0.1 (so every link opens: sheet, renders, `src/`, `object.glb`
in an orbit viewer built on the vendored three.js, `record.json`), and re-reads a
run's record per request so a battery that is still writing shows up live. A run
diff --git a/harness/docs/COST.md b/harness/docs/COST.md
index 2cbcb904..81ed3bab 100644
--- a/harness/docs/COST.md
+++ b/harness/docs/COST.md
@@ -1,23 +1,23 @@
# Cost audit — where the money actually goes
-Measured on **61 recorded runs** (`runs/e2e_*` ×5, `bench/out/{static_v1_flash,
+Measured on **61 recorded runs** (`runs/e2e_*` ×5, `eval/bench/out/{static_v1_flash,
articulated_v1_flash,graphics_v1_flash,scenes_v1_flash}` ×46, the 10 cells of
-`bench/out/compare_v1_live2`) on 2026-08-23. Every number below comes from the
+`eval/bench/out/compare_v1_live2`) on 2026-08-23. Every number below comes from the
recorded telemetry, not from an estimate: `record.json`, `events.jsonl` and the
per-call `usage` rows inside `trajectories/*/transcript.jsonl`.
Reproduce:
```
-3dcode cost show runs bench/out/static_v1_flash … # console
-3dcode cost show runs bench/out/* --md docs/cost_report.md # full tables
+3dcode cost show runs eval/bench/out/static_v1_flash … # console
+3dcode cost show runs eval/bench/out/* --md docs/cost_report.md # full tables
3dcode cost show runs --recheck # re-price with today's table
3dcode cost prices [--unverified] # the price table + provenance
```
The audit is `codeverse/addons/costreport/audit.py`; it reconstructs a per-call ledger from
old runs (`codeverse/cost/reconstruct.py`), so it works on every run recorded so
-far — no re-instrumentation needed. `bench/out/compare_v1_full` (a partial,
+far — no re-instrumentation needed. `eval/bench/out/compare_v1_full` (a partial,
superseded compare battery) is excluded.
> **Re-run 2026-08-23 (wave 3).** Every figure in Part I below was recomputed
@@ -354,7 +354,7 @@ Savings are estimated **on this data set** (61 runs, $86.30) unless stated.
| 3 | ~~**Cap agent turns at ~25 and compact old tool results.** 39.3% of the agent bill is turn ≥20; 19 of the 33 sessions that ran ≥50 turns were cut off by their own budget.~~ **Tested, rejected: +$0.02 and −0.21 score** (A/B, n=3 per arm, §17) — the cap is off by default. | est. $8–15, **measured $0** | high (A/B) | `RoundPolicy.agent_max_turns=0`; still settable per caller |
| 4 | **Fold the off-record spend into the budget** (cut rounds, post-hoc texture passes). | $0 saved, **$1.22 of blindness removed** | high | §6; the ledger (`codeverse.cost.record_call`) makes it automatic |
| 5 | **Scene track: 73% of scene spend is assets+zones, 0/5 passed.** Trim the per-zone context (each zone session re-sends the whole scene contract) and judge assets before zones start. | ~$1/run of $2.83 | medium | `tracks/scene*.py` |
-| 6 | **Drop `oneshot:claude-code` from default compare arms** ($1.04/artifact at 0.673 — the worst score per dollar measured). | bench-only | high | `bench/compare_backends.py` arms |
+| 6 | **Drop `oneshot:claude-code` from default compare arms** ($1.04/artifact at 0.673 — the worst score per dollar measured). | bench-only | high | `eval/bench/compare_backends.py` arms |
| 7 | **Gate the texture pass on the materials criterion** (< 0.7) — 5 passes ran, 2 shipped, $1.10 spent. | ~$0.5 | medium | `texturing/run.py` |
| 8 | **Keep the pro judge; do not "save" money there** (§8). | avoids a **$45** increase | high | – |
| 9 | **Price-table corrections** (§7) — accuracy, not savings: the reported bill was 3.1% low overall and 3.7x low on gemini-cli runs. | – | done | `models/pricing.py` |
@@ -466,7 +466,7 @@ generator sessions only and reads both label forms — `baseline_c<k>` (live) an
**Who opens a ledger.** `3dcode make` / `3dcode resume` (`cli.main._run_track`),
`3dcode texture pass` (with `create=False`), **and the bench drivers** —
-`bench/run_bench.py` opens one per prompt and `bench/compare_backends.py` one per
+`eval/bench/run_bench.py` opens one per prompt and `eval/bench/compare_backends.py` one per
cell (plus a nested one for a harness arm's own run, and one for the pairwise
arena). The batteries produce most of the runs in this repo, so until wave 3
most priced rows were going to the per-process fallback log. `run_ledger` nests by holding
@@ -591,7 +591,7 @@ the default for every profile.**
### Payload v3 (D47, 2026-08-31): 5 montages for the 14-view rig
-The rig A/B (docs/EVAL.md judge-experiments log) re-priced the verdict: the adopted
+The rig A/B (eval/docs/EVAL.md judge-experiments log) re-priced the verdict: the adopted
payload C (14 views + clay, 5 montages + 2 crops) bills **$0.155/verdict at pro n=3
(40.9k input tokens)** against the old 8-view payload's $0.146–0.150/30.0–34.0k — a
+3–6 % price for the only payload change that has survived multiplicity (+0.038
@@ -834,8 +834,8 @@ finding is that **rotation was never the constraint**: the pool was running at 0
its request quota and 1.6 % (median) of its token quota.
Everything below is measured on this box (24 cores, 22 keys, `gemini-3.7-flash`), from
-the 6 014 live priced calls in `bench/out/**` and three probes; the reusable one is
-`bench/concurrency_probe.py`.
+the 6 014 live priced calls in `eval/bench/out/**` and three probes; the reusable one is
+`eval/bench/concurrency_probe.py`.
## 18. The baseline: 3 % of the quota, and 17.7 h asleep
@@ -901,7 +901,7 @@ shows it is not theoretical.
## 20. The measured knee — and why one number is not enough
-`bench/concurrency_probe.py` runs a fixed workload at several in-flight levels with a
+`eval/bench/concurrency_probe.py` runs a fixed workload at several in-flight levels with a
fresh `KeyPool` each time, so the `ok / 429 / 5xx` counters are exact deltas.
**Generator-shaped (12 k-token prompt, 128 calls per level):**
@@ -1010,7 +1010,7 @@ time. A single sustained outage is the case the gate was designed for and is no
covered by this measurement — that is why the mechanism is kept rather than deleted.
So `Settings.rate.storm_gate` defaults to **False**. The mechanism, its counters and
the probe flag are kept so the experiment is reproducible
-(`CV3D_RATE__STORM_GATE=1`, or `bench/concurrency_probe.py --storm-gate`) — the same
+(`CV3D_RATE__STORM_GATE=1`, or `eval/bench/concurrency_probe.py --storm-gate`) — the same
treatment §13 gave cache-friendly prompt ordering.
**Caveat on the knee under a storm.** §20's knee optimises *throughput* — total calls
@@ -1054,7 +1054,7 @@ storm consumed **5.59 h**; with the deadline it consumes **15.1 min**:
says so, in both the storm branch and ordinary backoff. A slow call that is *making
progress* is never cut — the deadline bounds retrying, not the call. Past it the cell
is `infra_failed`, which is excluded from every rate and re-runnable with
-`--redo-status infra_failed` (`docs/EVAL.md` §7).
+`--redo-status infra_failed` (`eval/docs/EVAL.md` §7).
**The wall-clock ceiling is only checked when money is spent.** `BudgetGuard.check()`
is called from `spend()`/`charge()`, so a run whose calls never *complete* is never
@@ -1130,13 +1130,13 @@ Three consequences:
read by nothing until 2026-08-24 — three launches that "set" it ran at 64).
`codeverse.models.health.pool_budget()` reads every sibling's cap from `/proc/<pid>/environ`
and reports used / headroom; `3dcode doctor`'s `pool sharing` row prints it, and
- `bench/ab_plan.py` refuses to start only when its own need does not FIT the headroom.
+ `eval/bench/ab_plan.py` refuses to start only when its own need does not FIT the headroom.
The first version of this rule counted *processes* and made every agent wait for an
empty machine — which stalled an entire A/B wave behind two batteries that were
themselves parked. A budget, not a head-count.
4. **The accounting only works if every process is visible to it.** `pool_budget()`
matches on argv, and until 2026-08-24 it matched only the *script path* spelling
- (`python bench/ab_plan.py`). A live eight-prompt A/B launched as
+ (`python eval/bench/ab_plan.py`). A live eight-prompt A/B launched as
`python -m bench.ab_plan` — driver plus two 16-in-flight cell children — was therefore
invisible: the budget read `16/64, headroom 48` on a machine already at 48, and the
next agent to check it would have launched 32 more and taken the machine to 80. Both
@@ -1181,7 +1181,7 @@ costs one 2-minute re-probe, starting wrongly costs hours. This one is a judgem
call, not a measured optimum, and it is written here so it can be revisited with data.
What did work, in production, on those two lost cells: they were recorded
-`infra_failed` with `score=None` rather than a hard 0.0 (§7 of `docs/EVAL.md`), and the
+`infra_failed` with `score=None` rather than a hard 0.0 (§7 of `eval/docs/EVAL.md`), and the
retry deadline (§22) stopped each at ~15 min instead of the 56-87 min the same cells
burned earlier the same morning.
@@ -1302,7 +1302,7 @@ defaulted so old rows load); `3dcode cost` adds a per-key table and a
## 28. Where the time goes — the 2026-08-26 audit
51 storm-day runs against 52 baseline runs, every stage and every model call, scripts in
-`bench/time_audit/` (read-only over `bench/out`). The numbers that decide what to build next:
+`eval/bench/time_audit/` (read-only over `eval/bench/out`). The numbers that decide what to build next:
* A blender object run is **1 811 s** median on a healthy provider and **4 726 s** under the storm;
build + gates + render together are 3–19 % of a baseline run and 1–4 % of a storm run. The
@@ -1379,7 +1379,7 @@ come from.
**verdict** — so every gate that ran and answered FAIL reached the model as a broken call.
**Selector** (every number in this section, unless another one is named): the files
-`bench/out/*/**/run/trajectories/*/stdout.json` **as they stood at 2026-09-02 20:20 UTC**
+`eval/bench/out/*/**/run/trajectories/*/stdout.json` **as they stood at 2026-09-02 20:20 UTC**
— the corpus is live and grows with every battery, so a later re-run reads larger
numbers (2026-09-03 00:30: 390 files, 6 154 calls, 60 of 302 rounds) — 386 of them, of
which **224 are non-empty** — one per gemini-cli session that reported stats; the other 162 are
@@ -1398,7 +1398,7 @@ MCP tool calls, 1 720 (28 %) reported as errors.**
| `check_connectivity` | 923 | 14 % | — |
| `isolate` | 35 | 49 % | — (these ARE mostly real usage errors: an unknown part name) |
-(The gate column is a second selector: all 239 `run/record.json` under `bench/out/`,
+(The gate column is a second selector: all 239 `run/record.json` under `eval/bench/out/`,
411 rounds. An earlier draft of this section quoted 217 sessions / 1 404 sweeps / 63 %
— that was the same corpus minus the `.attempt1` retry sessions and did not reproduce;
these numbers do, with the glob above.)
@@ -1480,9 +1480,9 @@ with them. The 62 % vs 19 % gap between the sweep TOOL's verdict and the round'
GATE is a separate question — the tool flags any overlap past `tol_m` and any floating
link, while `sweep_findings` downgrades small rest overlaps and hinge gaps to WARN.
-**Measured after the fix** (`bench/out/wave2_lean`, the same battery and config as the
+**Measured after the fix** (`eval/bench/out/wave2_lean`, the same battery and config as the
`aa_articulated` A/A above, 2026-09-03). Both columns are printed by
-`python bench/session_stats.py bench/out/aa_articulated bench/out/wave2_lean`, which reads
+`python eval/bench/session_stats.py eval/bench/out/aa_articulated eval/bench/out/wave2_lean`, which reads
each session's own stats block and each round's recorded `usage`:
| per generator request | before (`aa_articulated`) | after (`wave2_lean`) |
@@ -1497,7 +1497,7 @@ each session's own stats block and each round's recorded `usage`:
**Correction (2026-09-03), and what it does NOT explain.** The after column first read
1.4 % (48 / 3 480), 13 736 uncached tokens and median \$0.967 over "108 sessions / 5 082
requests". It was computed by hand, by a method that is not in the tree, and
-`bench/session_stats.py` does not reproduce it: the ratios are 1.87x on MCP calls and on
+`eval/bench/session_stats.py` does not reproduce it: the ratios are 1.87x on MCP calls and on
requests but only 1.06x on sessions, and no mechanism tested accounts for that shape —
following the `run/telemetry/trajectories` symlink doubles the file count exactly (2.0x on
both batteries), counting every tool instead of the MCP ones gives 2 756 not 3 480, and no
diff --git a/harness/docs/INSTALL.md b/harness/docs/INSTALL.md
index f5f1bae3..c5bffc93 100644
--- a/harness/docs/INSTALL.md
+++ b/harness/docs/INSTALL.md
@@ -576,7 +576,7 @@ things: `renders/` and `judge_images/` (render + montage cache), `textures/` and
regenerate), `gpu_probe.json`, `browser_*.lock` (the shared
headless-Chrome endpoint file — do not delete it while renders are running).
-**Your data is not in either cache.** `runs/`, `bench/out/` and any exported
+**Your data is not in either cache.** `runs/`, `eval/bench/out/` and any exported
dataset are the flywheel output — delete them deliberately, never as part of a
cleanup. On this machine `harness/bench/out/` is read-only reference material
(the early `e2e_*` reference runs were archived off-repo on 2026-08-29).
diff --git a/harness/docs/INTERFACES.md b/harness/docs/INTERFACES.md
index 50de1aea..fdd5cbe0 100644
--- a/harness/docs/INTERFACES.md
+++ b/harness/docs/INTERFACES.md
@@ -280,7 +280,7 @@ from codeverse.spatial.complexity import (ComplexityVector, COMPLEXITY_WEIGHTS,
complexity_of_glb, complexity_of_parts, band_of)
complexity_of_glb(glb) -> ComplexityVector # part_count, assembly_depth, tri_count, materials, silhouette,
# feature_density, symmetry_groups, hollowness, + index 0-1 (documented weights) and band
- # (trivial|simple|moderate|complex|intricate). Deterministic, no VLM/render. docs/COMPLEXITY.md
+ # (trivial|simple|moderate|complex|intricate). Deterministic, no VLM/render. eval/docs/COMPLEXITY.md
from codeverse.spatial.connectivity import check_connectivity # (glb, *, gap_m=…, …, language="", planned_edges=()) — planned_edges: (child, parent | (copies…)); Δ language selects the
from codeverse.spatial.contract import check_contract # frame of fix hints; both gates emit hints in the AUTHORING frame
# (labelled "blender frame: Z-up, -Y front" etc.), GLB vectors in data
@@ -553,7 +553,7 @@ from codeverse.record._git import read_tree_at, diff_between, changed_files_betw
from codeverse.record.record import unique_files, SUBRUN_DIRS, BATTERY_MARKERS
# unique_files(root, name) -> [Path]: every file called `name` under root ONCE per file on disk (follows
# the run/telemetry/trajectories symlink and collapses it; skips SUBRUN_DIRS = {_cand, _assets}) — the
- # one walker behind bench/session_stats.py, bench/coupling_stats.py and cost.reconstruct.find_runs
+ # one walker behind eval/bench/session_stats.py, eval/bench/coupling_stats.py and cost.reconstruct.find_runs
# diff_between(ws, before, after, *, max_bytes=None) -> (text, untruncated size, was_truncated)
# changed_files_between(ws, before, after) -> [path]; commit_exists(ws, commit) -> bool
# both under GIT_SAFE_DIFF_FLAGS (--no-ext-diff --no-textconv) on top of workspace.GIT_SAFE_FLAGS
@@ -569,7 +569,7 @@ from codeverse.addons.dataset.captions import caption_sample # Δ (ws, record, m
from codeverse.addons.gallery import build_index, default_roots, build_static, serve, GalleryApp # THE local gallery
# build_index(roots) -> GalleryIndex (sections of RunEntry; never raises per run)
# build_static(roots, out_html, *, embed=False) -> (path, n, index)
- # render_static(index, *, embed=…, extra_html="") — bench/report.py's page
+ # render_static(index, *, embed=…, extra_html="") — eval/bench/report.py's page
# GalleryApp(roots, reload=False).route(path, query) -> Response (pure, testable)
# serve(roots, *, host=None, host_explicit=False, port=8765, reload=False)
from codeverse.addons.gallery.urls import safe_join # (root, rel) -> Path inside root, else PathError
diff --git a/harness/docs/RUNBOOK.md b/harness/docs/RUNBOOK.md
index 58312651..b2c8df51 100644
--- a/harness/docs/RUNBOOK.md
+++ b/harness/docs/RUNBOOK.md
@@ -116,7 +116,7 @@ events.
```bash
3dcode gallery serve # ./runs + every ./bench/out/*/runs → http://127.0.0.1:8765/
-3dcode gallery serve bench/out/static_v2_flash/runs --port 9000 --reload --no-open
+3dcode gallery serve eval/bench/out/static_v2_flash/runs --port 9000 --reload --no-open
3dcode gallery build --out gallery.html [--embed] # one self-contained file (--embed inlines the sheets)
```
`serve` indexes the run roots (records only — ~85 runs in ~0.15 s) and serves the run
@@ -225,14 +225,14 @@ runs out of memory Chrome reaps the render tab and the driver reports
`Attempted to use detached Frame '<id>'`: the scene built, every gate ran, and the judge
was skipped for want of pixels — three of six cells of `scenes_v1` on 2026-09-05, $6.75 of
generation already paid for. The scene drivers now retry once on a browser of their own
-(`spatial/render_scene.run_scene_script`), and `bench/scene_stats.py` reports such rounds
+(`spatial/render_scene.run_scene_script`), and `eval/bench/scene_stats.py` reports such rounds
in a `lost to the box` column instead of counting them as gate failures. A battery whose
report shows that column non-zero has to be re-run for those cells before it is read as a
statement about the generator.
**A battery launched with `3dcode` from a worktree runs the MAIN checkout's code.** `3dcode`
is a console script, so `sys.path[0]` is the venv's `bin`, never the cwd, and `import
-codeverse` finds the editable install. `--out bench/out/<name>` IS relative to the cwd,
+codeverse` finds the editable install. `--out eval/bench/out/<name>` IS relative to the cwd,
so the OUTPUT lands in the worktree while the CODE that produced it is the main tree's —
an arm that looks like it is testing your branch and is testing `main` (measured
2026-09-05: the first `scenes_v1` battery, launched from `local/worktrees/integrate`, ran
@@ -267,22 +267,22 @@ call the `texture_pass` / `texture_preview` tools mid-session.
## 7. Benchmarks
```bash
-3dcode bench run bench/prompts/static_objects_v1.yaml --generator single-shot:gemini:gemini-3.7-flash \
- --judge gemini:gemini-3.1-pro-preview --rounds 2 --parallel 4 [--tier easy] [--id furn_easy_stool] [--limit 6] [--out bench/out/x]
-3dcode bench report bench/out/static_objects_v1 # report.md + self-contained report.html (gallery)
-python bench/compare_backends.py --prompts bench/prompts/compare_v1.yaml \
- --arms harness:gemini-cli:gemini-3.7-flash,oneshot:claude-code --judge gemini:gemini-3.1-pro-preview --out bench/out/compare_v1
+3dcode bench run eval/bench/prompts/static_objects_v1.yaml --generator single-shot:gemini:gemini-3.7-flash \
+ --judge gemini:gemini-3.1-pro-preview --rounds 2 --parallel 4 [--tier easy] [--id furn_easy_stool] [--limit 6] [--out eval/bench/out/x]
+3dcode bench report eval/bench/out/static_objects_v1 # report.md + self-contained report.html (gallery)
+python eval/bench/compare_backends.py --prompts eval/bench/prompts/compare_v1.yaml \
+ --arms harness:gemini-cli:gemini-3.7-flash,oneshot:claude-code --judge gemini:gemini-3.1-pro-preview --out eval/bench/out/compare_v1
```
Results stream to `results.jsonl` (resumable). Batteries: `static_objects_v1` (24),
`articulated_v1` (12), `scenes_v1` (12), `compare_v1` (8, harness-vs-one-shot).
-Protocol and judge calibration: `docs/EVAL.md`.
+Protocol and judge calibration: `eval/docs/EVAL.md`.
`compare_backends` **preflights every model it needs** (one ~20 s probe each) and
refuses to start when one is not serving — a dead provider does not fail fast on its
own, it lets each cell burn its full retry budget first. `--wait-for-provider 60`
parks until it recovers instead; `--no-preflight` skips the check. Cells that a
provider outage kills anyway are recorded `infra_failed`, excluded from every rate,
-and re-run with `--redo-status infra_failed` (see `docs/EVAL.md` §7).
+and re-run with `--redo-status infra_failed` (see `eval/docs/EVAL.md` §7).
### 7.v Templates are read at render time — a new required variable breaks live workers like a moved name
@@ -358,7 +358,7 @@ model, produced the zero — check `cell.json`'s `error`).
plan model in `contracts/plan.py`, `.j2` prompts, branch in `tracks/__init__.py`
(`get_track` forwards `**options` to constructors). `tracks/graphics.py` is the
template for a track with its own planner and no GLB.
-* **New bench battery**: `bench/prompts/<name>.yaml` with `name, track, language,
+* **New bench battery**: `eval/bench/prompts/<name>.yaml` with `name, track, language,
prompts[{id, tier, category, prompt, must_have, dimensions_m}]`.
diff --git a/harness/docs/RUN_LAYOUT.md b/harness/docs/RUN_LAYOUT.md
index 30b2d716..2e39632f 100644
--- a/harness/docs/RUN_LAYOUT.md
+++ b/harness/docs/RUN_LAYOUT.md
@@ -1,6 +1,6 @@
# Run directory layout
-One run = one directory (`runs/<slug>/`, `bench/out/<battery>/runs/<slug>/`).
+One run = one directory (`runs/<slug>/`, `eval/bench/out/<battery>/runs/<slug>/`).
It answers three different questions, and since 2026-08-23 it keeps them in
three separate buckets:
diff --git a/harness/docs/SKILLS.md b/harness/docs/SKILLS.md
index 84c8cb82..3d4a215b 100644
--- a/harness/docs/SKILLS.md
+++ b/harness/docs/SKILLS.md
@@ -147,7 +147,7 @@ record refers to a routing decision by id, so recycling it would silently relabe
`measured` for provenance and have **no measured effect**; five are. Every bundle also
declares **one deterministic quantity it claims to move**, in its own frontmatter
(`target_metric`, `target_direction`, `target_baseline`) and in `codeverse/addons/skill_targets.py`.
-`python bench/skill_targets.py bench/out` prints them all — per battery or paired across an
+`python eval/bench/skill_targets.py eval/bench/out` prints them all — per battery or paired across an
A/B's two arms — and **`docs/SKILLS_LEDGER.md` is the row-by-row maintenance surface**.
`cadquery-forms` and `threejs-forms` stay `inherited-unverified` and routed off: their
@@ -180,7 +180,7 @@ per track), every gate finding kind the corpus produces, and 2,000 seeded random
including junk tracks and languages.
`finding_kind()` classifies **wider** than it routes — 113 distinct WARN/ERROR message
-shapes mined from `bench/out`, with a 43-row golden fixture
+shapes mined from `eval/bench/out`, with a 43-row golden fixture
(`tests/skills/data/gate_findings.json`) asserting no `unknown`. Naming a defect costs
nothing; routing one is a claim that a skill helps. INFO findings are census, not defects,
and classify to `None` — routing off "all 7 parts are connected" would attach the
@@ -199,8 +199,8 @@ goes into `record.prompt_hashes` as `skill:<name>`, so "what text decided this r
answerable. Per session, `telemetry/skills.jsonl` gets one line.
```
-3dcode skills report bench/out/<battery> # deep-read rate per skill per backend
-3dcode skills report bench/out/<battery> --json
+3dcode skills report eval/bench/out/<battery> # deep-read rate per skill per backend
+3dcode skills report eval/bench/out/<battery> --json
3dcode doctor --skills # library + discovery wiring
```
@@ -300,20 +300,20 @@ Two contradiction checks are worth separating, because they answer different que
### The rig
-`bench/ab_plan.py`, arms differing in exactly one thing — `CV3D_SKILLS=on` on the variant —
+`eval/bench/ab_plan.py`, arms differing in exactly one thing — `CV3D_SKILLS=on` on the variant —
on `static_objects_v2`, `--rounds 1`, paired, generator `gemini-cli:gemini-3.6-flash`,
fixed judge `gemini:gemini-3.1-pro-preview` at `n_samples 2`, `--max-in-flight 8` against
32–48 of pool headroom, `--wait-for-provider`. The eight prompt ids are **the same eight
-`bench/out/plan_loop/C0` used for its A/A**, so the noise floor below was measured on this
+`eval/bench/out/plan_loop/C0` used for its A/A**, so the noise floor below was measured on this
exact battery, on this rig, on this day.
The switch is generation-side: the planner call happens in the `plan` stage before any
skill is attached, and `attach_for_round` only touches generation and repair rounds. That
-matters for §8.1 of `docs/EVAL.md` and is the reason the next run of this can pin the plan.
+matters for §8.1 of `eval/docs/EVAL.md` and is the reason the next run of this can pin the plan.
### The noise floor, measured first
-`bench/out/plan_loop/C0`, 2026-08-25, **two identical arms** on these eight prompts:
+`eval/bench/out/plan_loop/C0`, 2026-08-25, **two identical arms** on these eight prompts:
| | |
|---|---|
@@ -331,7 +331,7 @@ prompts regressed past 0.03 with byte-identical arms. So at n = 8 on this rig th
not a test of the change; nothing can pass it. That is a fact about the instrument, and it
is the honest first line of any verdict it produces.
-`docs/EVAL.md` §8.1 opened C0's worst pair and found the cause: the two identical arms
+`eval/docs/EVAL.md` §8.1 opened C0's worst pair and found the cause: the two identical arms
planned 1 part and 10 parts for the same pitcher pump. The dominant variance term is the
**planner**, not the judge and not the generator.
@@ -339,14 +339,14 @@ planned 1 part and 10 parts for the same pitcher pump. The dominant variance ter
Recorded here because it is the more useful half of this section. The first run of this
A/B was **void**: `ab_plan.spawn_cell` starts each child as a file path, so `sys.path[0]`
-is `bench/`, and `ab_plan` imported `codeverse._compat` *above* its own `sys.path`
+is `eval/bench/`, and `ab_plan` imported `codeverse._compat` *above* its own `sys.path`
bootstrap — which let the editable install resolve `codeverse` to the **main tree**. Both
arms ran a harness with no `codeverse/skills` package at all. No error, no warning; the
variant workspace simply had no `.agents/skills` directory and no `skills.attached` event,
and the run would have reported "no effect" with a straight face.
The plan-loop wave hit the same thing hours earlier
-(`bench/out/plan_loop/C0/invalid_attempt1_maintree_import`) and worked around it with
+(`eval/bench/out/plan_loop/C0/invalid_attempt1_maintree_import`) and worked around it with
`PYTHONPATH` in a launch script. It is fixed in the code now — bootstrap first, a guard
that refuses to start against a foreign `codeverse`, and
`tests/compare_bench/test_worktree_import.py` — because a workaround protects whoever
@@ -354,7 +354,7 @@ remembers it, not the run.
### What this A/B actually measured
-`bench/out/ab_skills`, **still running when this was written** — re-read `summary.md` for the
+`eval/bench/out/ab_skills`, **still running when this was written** — re-read `summary.md` for the
current state, and re-run with `--redo-status infra_failed` before quoting it as final.
Weather was hostile: the Gemini pool sat at 0–4 of 6 keys answering for most of the window,
so the rig parked on its preflight for two hours and then lost whole pairs to 503 storms
@@ -419,7 +419,7 @@ delivery mechanism for `api-agent` is a pointer nobody follows, and that is fixa
The design planned around this: make the **gate counts** the primary readout, since they
are deterministic and are what a contact skill actually targets, and put the judged score
-second under a sign test. `bench/ab_gate_rates.py` computes them, paired per prompt.
+second under a sign test. `eval/bench/ab_gate_rates.py` computes them, paired per prompt.
Run it over the SAME A/A — two identical arms — and it says:
@@ -441,7 +441,7 @@ So at n = 8 with a free plan, neither readout can separate this switch from noth
### What to try next, in order
-0. **Pin the plan** (`docs/EVAL.md` §8.1). The skills switch is generation-side — the
+0. **Pin the plan** (`eval/docs/EVAL.md` §8.1). The skills switch is generation-side — the
planner runs in the `plan` stage before anything is attached — so plan-once-write-both is
valid here, and it removes the dominant variance term from *both* readouts instead of
averaging it down. It also costs one planner call *less* per pair. This is the single
@@ -515,7 +515,7 @@ chose to go deeper" (§1).
4. Add its rows to `ROUTES` in `codeverse/skills/registry.py`, in the same commit.
5. Add a `Target` row to `codeverse/addons/skill_targets.py` and the matching `target_*` keys to
the frontmatter — the ONE deterministic quantity the bundle claims to move, its
- direction, and its baseline from `python bench/skill_targets.py bench/out`. If no
+ direction, and its baseline from `python eval/bench/skill_targets.py eval/bench/out`. If no
deterministic instrument can see the claim, say so with `measurable=False` and a
`caveat`: that is a finding about the bundle, not a gap to paper over with a judged
criterion. Then add its row to `docs/SKILLS_LEDGER.md`.
@@ -567,10 +567,10 @@ so; the rest are live.
pipeline step, so it reaches no `record.json`. `cv3d-threejs-shader-traps` is therefore
**not instrumented**, which is a stronger statement than "always clean". Merge it into
`rounds[].gates` before spending anything on that bundle.
-* **`bench/_fixed_eval.FixedEvaluator` is blender-only.** It pins `get_runtime(BLENDER)` and
+* **`eval/bench/_fixed_eval.FixedEvaluator` is blender-only.** It pins `get_runtime(BLENDER)` and
runs only lint and connectivity — never contract, joint_sweep, scene_frames or gl_frames.
The *judged* column of any non-blender A/B is meaningless. The primary readout is safe:
- `bench/skill_targets.py` reads each arm's own harness record.
+ `eval/bench/skill_targets.py` reads each arm's own harness record.
* **The atime probe is blind in any git workspace, and its control proves it.** The control
bundle came back "opened" in 27 of 33 sessions; the only 6 sessions where it stayed clean
were the 6 scene sessions, where nothing was delivered. So the shipped report is honest
@@ -598,7 +598,7 @@ so; the rest are live.
scene stages deliver skills, this is the obvious comparison.
* **Corpus percentages are checked for provenance, not recomputed.** A body's "47 graded
blender runs" is a *slice*; without each claim declaring its query, a tight recomputation
- compares two different populations. Enforced today: a claim may not exceed what `bench/out`
+ compares two different populations. Enforced today: a claim may not exceed what `eval/bench/out`
holds, `measured` needs n ≥ 20, every rate names its battery / n / date.
* **`cadquery` and `threejs` bundles are routed off** until each language reaches 20 graded
runs. As of this wave: cadquery **4**, threejs **3**.
diff --git a/harness/docs/SKILLS_LEDGER.md b/harness/docs/SKILLS_LEDGER.md
index 5917e64c..b8dded22 100644
--- a/harness/docs/SKILLS_LEDGER.md
+++ b/harness/docs/SKILLS_LEDGER.md
@@ -15,23 +15,23 @@ Three files hold it up:
| file | what it is |
|---|---|
| `codeverse/addons/skill_targets.py` | the table — one `Target` row per bundle |
-| `bench/skill_targets.py` | the readout — one command, battery **or** A/B |
+| `eval/bench/skill_targets.py` | the readout — one command, battery **or** A/B |
| `tests/skills/test_targets.py` | the pins — the rows exist, the gate kinds are still live, the frontmatter agrees, the readout counts what the row says |
```
-python bench/skill_targets.py bench/out # every baseline, whole corpus
-python bench/skill_targets.py bench/out --round first # where a bundle acts, before repair
-python bench/skill_targets.py bench/out/ab_skills # control vs variant, paired
-python bench/skill_targets.py bench/out --skill cv3d-part-contact --per-run --json
+python eval/bench/skill_targets.py eval/bench/out # every baseline, whole corpus
+python eval/bench/skill_targets.py eval/bench/out --round first # where a bundle acts, before repair
+python eval/bench/skill_targets.py eval/bench/out/ab_skills # control vs variant, paired
+python eval/bench/skill_targets.py eval/bench/out --skill cv3d-part-contact --per-run --json
```
---
## 0a. In flight — the first whole-library pinned A/B (2026-08-26)
-`bench/out/fancy_v1/{bl_a,bl_b,cq,tj}`: 16 hard, design-realistic object prompts (blender 10,
+`eval/bench/out/fancy_v1/{bl_a,bl_b,cq,tj}`: 16 hard, design-realistic object prompts (blender 10,
cadquery 3, threejs 3), `CV3D_SKILLS=1` (the whole library) against OFF, plans pinned, the same
-fixed judge, on all 22 keys. Read it out with `python bench/skill_targets.py bench/out/fancy_v1/<driver>`
+fixed judge, on all 22 keys. Read it out with `python eval/bench/skill_targets.py eval/bench/out/fancy_v1/<driver>`
per driver, or the merged paired table. What it can and cannot say, stated before the data
lands: n ≤ 16 pairs against a judged-score floor of 0.202 resolves a ~0.2 mean move, not a
per-bundle effect (§5); its value is the first paired data on the library *as shipped* and the
@@ -62,7 +62,7 @@ interpenetration (15–18 mm WARNs) plus flat materials. With the lamp re-aggre
library helps, and the two large negatives are real defects in the ON runs, not judge noise.
Skills stay OFF. Pairs still in flight (carousel_horse, lever_espresso, smock_windmill,
gate_valve, turbocharger, marimba, jacobs_chuck redo) land in `results.jsonl` per driver;
-merge with `python bench/skill_targets.py` or the paired readout (scratch `fancy_readout.py`
+merge with `python eval/bench/skill_targets.py` or the paired readout (scratch `fancy_readout.py`
— rows are only paired when both arms are `scored`).
## 0. What this wave decided (2026-08-25, curate)
@@ -94,12 +94,12 @@ say `measured` because its defect descriptions were derived from our corpus, and
## 1. Why these are gate numbers and not scores — and why that was not enough
-An 8-prompt A/A of `bench/ab_plan.py` put the paired sd of the judged score at **0.202** and
-printed *~408 paired prompts to resolve +0.02* (`docs/EVAL.md` §8). Every row below is read
+An 8-prompt A/A of `eval/bench/ab_plan.py` put the paired sd of the judged score at **0.202** and
+printed *~408 paired prompts to resolve +0.02* (`eval/docs/EVAL.md` §8). Every row below is read
from a gate report, a `BuildResult`, the exported GLB or the sampled frames — never the judge.
The premise this wave was built on was that **pinning the plan** would collapse the noise,
-because `docs/EVAL.md` §8.1 showed the variance is the planner's. Pinning was built, wired
+because `eval/docs/EVAL.md` §8.1 showed the variance is the planner's. Pinning was built, wired
and verified in production (one planner call per pair, both arms cache-HIT on the same
`inputs_hash`). **The premise was half wrong, and that is the wave's most useful finding.**
@@ -132,8 +132,8 @@ away most of the signal. 24 pairs is affordable; 449 is not.
## 2. The ledger
-Baselines are recomputed 2026-08-25 from every run under `bench/out` that a gate reported on
-(`python bench/skill_targets.py bench/out`). Both rounds are shown where they differ, because
+Baselines are recomputed 2026-08-25 from every run under `eval/bench/out` that a gate reported on
+(`python eval/bench/skill_targets.py eval/bench/out`). Both rounds are shown where they differ, because
**several bundles act on the first round and the repair loop hides them by the last.**
Graded runs per language (a run is a sample **iff** it has a round carrying a gate report):
@@ -255,9 +255,9 @@ it lands these four rows stay `mixed`, not `measured`.
have reported "no effect" for a switch it never tested. **Fixed** (`entry_of(spec)` reads
`contracts.common.ENTRY_FILE`). This is why no graphics or three.js bundle has ever had a
readable A/B.
-* **`bench/_fixed_eval.FixedEvaluator` is blender-only.** It pins `get_runtime(BLENDER)` and
+* **`eval/bench/_fixed_eval.FixedEvaluator` is blender-only.** It pins `get_runtime(BLENDER)` and
runs only lint and connectivity — never contract, joint_sweep, scene_frames or gl_frames.
- Not fixed; it does not sink the primary readout, because `bench/skill_targets.py` reads
+ Not fixed; it does not sink the primary readout, because `eval/bench/skill_targets.py` reads
each arm's own harness record. Know that the *judged* column of any non-blender A/B is
meaningless.
* **`telemetry.record_exact_read` is wired to `read_file` only** (`agents/api_agent.py`), so
diff --git a/harness/tests/flywheel_cli/test_complexity_record.py b/harness/tests/flywheel_cli/test_complexity_record.py
index 7e9a09da..506b338b 100644
--- a/harness/tests/flywheel_cli/test_complexity_record.py
+++ b/harness/tests/flywheel_cli/test_complexity_record.py
@@ -1,20 +1,16 @@
-"""The complexity block on a record, and the score-vs-complexity report.
+"""The complexity block on a record.
``record.extra["complexity"]`` is what makes a verdict readable: it says how much
-artifact the score was earned on. ``bench/complexity_report.py`` is the study
-that turns a corpus of those into "does score rise or fall with complexity, and
-what does a complexity point cost?" (docs/COMPLEXITY.md).
+artifact the score was earned on. (The score-vs-complexity report over a corpus of
+these is `eval/bench/complexity_report.py`, tested in `eval/tests`.)
"""
from __future__ import annotations
-import json
from datetime import UTC, datetime
-from pathlib import Path
import pytest
-from bench import complexity_report as CR
from codeverse.contracts.artifacts import Judgment, Measurement
from codeverse.contracts.common import Backends, Language, Track, Usage
from codeverse.contracts.plan import BBox, PartPlan, StaticPlan
@@ -22,8 +18,6 @@ from codeverse.contracts.run import RoundRecord, RunRecord, RunStatus
from codeverse.contracts.spec import Spec
from codeverse.record.record import complexity_block, fill_derived, round_summary
-REPO = Path(__file__).resolve().parents[2]
-
def _measurement(index: float, parts: int = 8) -> Measurement:
return Measurement(
@@ -80,98 +74,3 @@ def test_no_measurement_means_no_block() -> None:
rec = fill_derived(_record([_round(0, 0.6, None)], best=0))
assert "complexity" not in rec.extra
assert round_summary(rec.rounds[0])["complexity"] is None
-
-
-# ------------------------------------------------------------------ the report
-def _write_run(root: Path, slug: str, *, index: float, overall: float, detail: float,
- cost: float = 1.0, minutes: float = 10.0) -> Path:
- run = root / "runs" / slug
- (run / "artifacts").mkdir(parents=True)
- rec = {
- "spec": {"track": "static_object", "language": "blender", "tags": ["bench", "hard"]},
- "plan": {"parts": [{"name": "a"}, {"name": "b"}]},
- "status": "passed", "best_round": 0,
- "total_usage": {"cost_usd": cost},
- "telemetry": {"cost": {"wall_clock_s": minutes * 60}},
- "rounds": [{
- "index": 0, "gates": [],
- "measurement": _measurement(index).model_dump(mode="json"),
- "judgment": {"overall": overall, "passed": overall >= 0.72,
- "scores": {c: detail for c in CR.CRITERIA}, "issues": []},
- }],
- }
- (run / "record.json").write_text(json.dumps(rec))
- return run
-
-
-def test_collect_and_report_over_a_battery(tmp_path: Path) -> None:
- battery = tmp_path / "batt"
- _write_run(battery, "low", index=0.35, overall=0.85, detail=0.85, cost=0.5, minutes=5)
- _write_run(battery, "mid", index=0.55, overall=0.70, detail=0.70, cost=1.5, minutes=20)
- _write_run(battery, "high", index=0.75, overall=0.55, detail=0.55, cost=3.0, minutes=40)
- rows, skipped = CR.collect([battery])
- assert skipped == 0 and len(rows) == 3
- assert {r["slug"] for r in rows} == {"low", "mid", "high"}
- assert rows[0]["plan_parts"] == 2 and rows[0]["n_materials"] == 3
- # the whole point of the study: a monotone fall shows up as a strong negative r
- xs, ys = CR._pairs(rows, "index", "overall")
- assert CR.pearson(xs, ys) < -0.98
- assert CR.spearman(xs, ys) == pytest.approx(-1.0)
- money = CR.dollars_per_point(rows)
- assert money["runs"] == 3 and money["total_usd"] == pytest.approx(5.0)
- assert money["usd_per_point"] == pytest.approx(5.0 / (100 * (0.35 + 0.55 + 0.75)), abs=5e-5)
- report = CR.render_report(rows, skipped)
- assert "score vs complexity" in report and "cost of complexity" in report
- assert "| batt/high |" in report
-
-
-def test_battery_expectations_flag_out_of_band_runs(tmp_path: Path) -> None:
- battery = tmp_path / "batt"
- _write_run(battery, "cx_a03_stool", index=0.30, overall=0.9, detail=0.9)
- _write_run(battery, "cx_d25_engine", index=0.40, overall=0.9, detail=0.9)
- rows, _ = CR.collect([battery])
- yaml_path = tmp_path / "b.yaml"
- yaml_path.write_text(
- "name: b\ntrack: static_object\nlanguage: blender\nprompts:\n"
- " - id: cx_a03_stool\n tier: easy\n prompt: p\n expected_complexity: [0.25, 0.42]\n"
- " - id: cx_d25_engine\n tier: hard\n prompt: p\n expected_complexity: [0.68, 0.88]\n"
- " - id: cx_missing\n tier: hard\n prompt: p\n expected_complexity: [0.5, 0.6]\n"
- )
- table = CR.battery_expectations(yaml_path, rows)
- assert "| cx_a03_stool | 0.25–0.42 | 0.300 | yes |" in table
- assert "LOW" in table # the engine collapsed to a stool
- assert "not run" in table # a prompt with no run is still listed
-
-
-def test_real_battery_bands_are_sane() -> None:
- """complexity_v3 is the ladder: every band inside [0,1], ordered, and the
- prompts must actually climb."""
- import yaml
-
- data = yaml.safe_load((REPO / "bench" / "prompts" / "complexity_v3.yaml").read_text())
- bands = [(p["id"], p["expected_complexity"]) for p in data["prompts"]]
- assert len(bands) == 12
- for pid, (lo, hi) in bands:
- assert 0.0 < lo < hi <= 1.0, pid
- lows = [lo for _, (lo, _hi) in bands]
- assert lows == sorted(lows) and lows[-1] > lows[0] + 0.3
-
-
-def test_row_recomputes_the_vector_when_the_record_has_none(tmp_path: Path) -> None:
- """The historic corpus predates the vector: the report must fall back to the
- GLB rather than skipping the run."""
- import trimesh
-
- run = tmp_path / "runs" / "old"
- (run / "artifacts").mkdir(parents=True)
- trimesh.Scene({"Box": trimesh.creation.box(extents=(0.3, 0.3, 0.3))}).export(
- str(run / "artifacts" / "object.glb"))
- (run / "record.json").write_text(json.dumps({
- "spec": {"track": "static_object", "language": "blender", "tags": []},
- "plan": {"parts": []}, "status": "passed", "best_round": 0,
- "total_usage": {"cost_usd": 1.0}, "rounds": [{
- "index": 0, "gates": [],
- "judgment": {"overall": 0.5, "passed": False,
- "scores": dict.fromkeys(CR.CRITERIA, 0.5), "issues": []}}]}))
- row = CR.row_for(run, "old_battery")
- assert row is not None and row["part_count"] == 1 and row["index"] > 0
diff --git a/harness/tests/orchestrator_tracks/test_plan_features.py b/harness/tests/orchestrator_tracks/test_plan_features.py
index e061b5e5..3da5e671 100644
--- a/harness/tests/orchestrator_tracks/test_plan_features.py
+++ b/harness/tests/orchestrator_tracks/test_plan_features.py
@@ -18,17 +18,12 @@ HARNESS = Path(__file__).resolve().parents[2]
def _sources() -> tuple[tuple[str, str], ...]:
"""(relpath, text) for every harness source, read ONCE per session.
- ``bench/out`` is excluded on purpose: it is gitignored battery output (5 781 of the
- 5 978 .py files this used to walk, all LLM-authored ``model.py``). Reading it cost
- ~0.65 s per parametrized case × 13, and it was also WRONG — a generated model.py
- that happens to contain a switch name would satisfy _readers() or fail the
- dead-switch guard, from a file that is not part of the harness at all.
+ Only ``codeverse`` counts: a switch is live when the HARNESS reads it. The evaluation
+ scripts (``eval/bench``) set switches for an A/B; they are not what makes one live.
"""
out = []
- for d in ("codeverse", "bench"):
- for p in (HARNESS / d).rglob("*.py"):
- if p.name == "plan_features.py" or "out" in p.relative_to(HARNESS).parts:
- continue
+ for p in (HARNESS / "codeverse").rglob("*.py"):
+ if p.name != "plan_features.py":
out.append((str(p.relative_to(HARNESS)), p.read_text(errors="replace")))
return tuple(out)
@@ -61,37 +56,6 @@ def test_the_plan_features_switch_is_declared_dead():
assert F.dead_env_keys({F.PLAN_FEATURES_ENV: "all", "CV3D_PLAN_BRIEF": "off"}) == [F.PLAN_FEATURES_ENV]
-def test_ab_plan_refuses_an_ab_whose_only_switch_is_dead(capsys):
- """`--variant-env CV3D_PLAN_FEATURES=all` produced a full battery and the verdict
- 'keep, mean delta +0.344' for two byte-identical arms. It must not start."""
- import bench.ab_plan as A
-
- with pytest.raises(SystemExit):
- A.main(["--prompts", "p.yaml", "--out", "o", "--variant-env", "CV3D_PLAN_FEATURES=all"])
- err = capsys.readouterr().err
- assert "nothing reads" in err and "CV3D_PLAN_FEATURES" in err
-
-
-def test_ab_plan_still_accepts_a_live_switch(monkeypatch, capsys):
- """The guard must not block a real A/B: it fires only when EVERY key is dead.
-
- Offline — --no-preflight and --allow-siblings keep the provider health check and the
- docs/COST.md §23 admission check out of it, and run_ab itself is stubbed."""
- import bench.ab_plan as A
-
- seen: list[dict[str, str]] = []
- class _V:
- decision, reason, caution = "keep", "stubbed", ""
-
- monkeypatch.setattr(A, "run_ab", lambda battery, out, opts, **kw: (seen.append(opts.variant_env), _V())[1])
- for argv in (["--variant-env", "CV3D_PLAN_BRIEF=off"],
- ["--variant-env", "CV3D_PLAN_FEATURES=all", "--variant-env", "CV3D_PLAN_BRIEF=off"]):
- A.main(["--prompts", "p.yaml", "--out", "o", "--no-preflight", "--allow-siblings", *argv])
-
- assert len(seen) == 2, capsys.readouterr()
- assert "CV3D_PLAN_BRIEF" in seen[0]
-
-
# --------------------------------------------------------------------- pin-plan safety
def test_a_generation_side_switch_may_share_one_plan():
"""`contacts` renders a table into the BUILDER's prompt from an unchanged plan, so
diff --git a/harness/tests/skills/test_freshness.py b/harness/tests/skills/test_freshness.py
index 40983c56..8e671902 100644
--- a/harness/tests/skills/test_freshness.py
+++ b/harness/tests/skills/test_freshness.py
@@ -87,7 +87,7 @@ def _live_vocabulary() -> set[str]:
"""
words: set[str] = set()
for root, exts in ((HARNESS / "codeverse", {".py", ".md", ".j2", ".yaml", ".yml", ".toml"}),
- (HARNESS / "bench", {".py", ".yaml", ".yml"}),
+ (HARNESS.parent / "eval" / "bench", {".py", ".yaml", ".yml"}), # battery + prompt ids a bundle cites
(HARNESS / "runtime_js", {".js", ".mjs", ".glsl", ".json"})):
if not root.is_dir():
continue
@@ -98,7 +98,7 @@ def _live_vocabulary() -> set[str]:
or p.parent.parent.name.startswith("cv3d-")):
continue
words |= set(re.findall(r"[A-Za-z_][A-Za-z0-9_]*", p.read_text(errors="ignore")))
- out = HARNESS / "bench" / "out"
+ out = HARNESS.parent / "eval" / "bench" / "out"
if out.is_dir():
words |= {d.name for d in out.iterdir() if d.is_dir()}
return words
diff --git a/harness/tests/skills/test_targets.py b/harness/tests/skills/test_targets.py
index 794d0c1f..48cb4515 100644
--- a/harness/tests/skills/test_targets.py
+++ b/harness/tests/skills/test_targets.py
@@ -2,7 +2,6 @@
from __future__ import annotations
-import json
import re
from pathlib import Path
@@ -108,139 +107,6 @@ def test_every_actionable_gl_frames_message_classifies():
)
-# --------------------------------------------------------------------------- the readout
-def _record(language: str, gates: list[dict], *, build_ok: bool = True) -> dict:
- return {"spec": {"language": language}, "final_score": 0.5, "status": "plateau",
- "rounds": [{"index": 0, "gates": gates, "build": {"ok": build_ok}}]}
-
-
-def _battery(tmp_path: Path, cells: dict[str, dict]) -> Path:
- root = tmp_path / "battery"
- for name, rec in cells.items():
- d = root / "runs" / name
- d.mkdir(parents=True)
- (d / "record.json").write_text(json.dumps(rec))
- return root
-
-
-def _finding(gate: str, message: str, severity: str = "warn") -> dict:
- return {"gate": gate, "severity": severity, "message": message}
-
-
-CONNECTIVITY = [_finding("connectivity", "'Lid' and 'Body' interpenetrate by ≈7.4 mm (12% of surface samples inside)"),
- _finding("connectivity", "'Leg_0' and 'Seat' interpenetrate by ≈2.1 mm (3% of surface samples inside)"),
- _finding("connectivity", "all 7 parts are connected (9 contacts, gap ≤ 1 mm)", "info")]
-CONTRACT = [_finding("contract", "part 'Seat' bbox deviates from the plan (worst 2.1× tolerance)"),
- _finding("contract", "'Leg' (each instance) bbox deviates from the plan (worst 1.4× tolerance)")]
-
-
-def test_the_readout_counts_the_kinds_the_row_names(tmp_path):
- from bench.skill_targets import battery_rows, load_runs
-
- root = _battery(tmp_path, {
- "chair": _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY},
- {"gate": "contract", "findings": CONTRACT}]),
- "lamp": _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY[2:]},
- {"gate": "contract", "findings": []}]),
- })
- rows = {r["skill"]: r for r in battery_rows(load_runs(root), list(TARGETS), which="last", cache=None)}
- assert rows["cv3d-part-contact"]["n"] == 2
- assert rows["cv3d-part-contact"]["mean"] == 1.0 # 2 pairs + 0, INFO ignored
- assert rows["cv3d-bbox-contract"]["mean"] == 0.5 # part_bbox only
- assert rows["cv3d-repeats-and-mirrors"]["mean"] == 0.5 # instance_bbox only
-
-
-def test_a_row_only_counts_the_languages_it_is_defined_over(tmp_path):
- from bench.skill_targets import battery_rows, load_runs
-
- root = _battery(tmp_path, {
- "robot": _record("urdf_blender", [{"gate": "joint_sweep", "findings": [
- _finding("joint_sweep", "links 'a' and 'b' overlap by 6.0 mm at pose j=1.0", "error"),
- _finding("joint_sweep", "links 'a' and 'c' overlap by 3.0 mm at rest", "warn")]}]),
- "chair": _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY}]),
- })
- rows = {r["skill"]: r for r in battery_rows(load_runs(root), list(TARGETS), which="last", cache=None)}
- assert rows["cv3d-urdf-joints"]["n"] == 1 # the blender run is not eligible
- assert rows["cv3d-urdf-joints"]["mean"] == 1.0 # ERRORs only; the rest-pose WARN is not one
- assert rows["cv3d-part-contact"]["n"] == 2 # both languages carry a part graph
-
-
-def test_an_ungraded_run_is_not_in_the_population(tmp_path):
- """A concurrency probe or an e2e smoke run has no gate report; counting it would give
- a GLB-derived row a different n from the gate-derived rows beside it."""
- from bench.skill_targets import battery_rows, load_runs
-
- root = _battery(tmp_path, {
- "chair": _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY}]),
- "probe": {"spec": {"language": "blender"}, "rounds": [{"index": 0, "gates": []}]},
- })
- rows = {r["skill"]: r for r in battery_rows(load_runs(root), list(TARGETS), which="last", cache=None)}
- assert rows["cv3d-part-contact"]["n"] == 1
-
-
-def test_the_ab_mode_pairs_by_prompt(tmp_path):
- from bench.skill_targets import ab_arms, ab_rows
-
- root = tmp_path / "ab"
- for arm, findings in (("control", CONNECTIVITY), ("variant", CONNECTIVITY[1:])):
- d = root / "arms" / arm / "cells" / "chair" / "run"
- d.mkdir(parents=True)
- (d / "record.json").write_text(json.dumps(
- _record("blender", [{"gate": "connectivity", "findings": findings}])))
- arms = ab_arms(root)
- assert arms is not None
- rows = {r["skill"]: r for r in ab_rows(arms, list(TARGETS), which="last", cache=None)}
- r = rows["cv3d-part-contact"]
- assert (r["n"], r["control_mean"], r["variant_mean"], r["mean_delta"]) == (1, 2.0, 1.0, -1.0)
- assert (r["better"], r["worse"], r["tied"]) == (1, 0, 0)
-
-
-def test_a_battery_dir_is_not_mistaken_for_an_ab_dir(tmp_path):
- from bench.skill_targets import ab_arms
-
- root = _battery(tmp_path, {"chair": _record("blender", [])})
- assert ab_arms(root) is None
-
-
-def test_the_cli_runs_over_a_synthetic_battery(tmp_path, capsys):
- from bench.skill_targets import main
-
- root = _battery(tmp_path, {"chair": _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY}])})
- assert main([str(root), "--json", "--no-cache"]) == 0
- payload = json.loads(capsys.readouterr().out)
- assert payload["mode"] == "battery"
- row = next(r for r in payload["rows"] if r["skill"] == "cv3d-part-contact")
- assert row["n"] == 1 and row["mean"] == 2.0
-
-
-def test_the_confidence_block_says_how_many_pairs_an_effect_needs():
- from bench.skill_targets import confidence
-
- c = confidence([-1.0, 1.0, -1.0, 1.0], control_mean=4.0)
- assert c["sd"] == pytest.approx(1.1547, rel=1e-3)
- assert c["ci95"] == pytest.approx(2 * c["se"])
- assert c["resolvable_effect"] == 1.0 # 25% of a control mean of 4
- assert c["n_to_resolve"] == 5 # (2*1.1547/1.0)^2
- assert confidence([1.0], control_mean=4.0)["sd"] is None
-
-
-def test_a_retried_cell_pairs_on_the_attempt_that_reached_a_gate(tmp_path):
- """A provider outage leaves `...flash` and `...flash.attempt1` side by side in one cell;
- pairing the ungraded one would drop a prompt that actually ran."""
- from bench.skill_targets import ab_arms, ab_rows
-
- root = tmp_path / "ab"
- for arm in ("control", "variant"):
- cell = root / "arms" / arm / "cells" / "chair"
- for leaf, rec in (("run_a", {"spec": {"language": "blender"}, "rounds": []}),
- ("run_a.attempt1",
- _record("blender", [{"gate": "connectivity", "findings": CONNECTIVITY}]))):
- (cell / leaf).mkdir(parents=True)
- (cell / leaf / "record.json").write_text(json.dumps(rec))
- rows = {r["skill"]: r for r in ab_rows(ab_arms(root), list(TARGETS), which="last", cache=None)}
- assert rows["cv3d-part-contact"]["n"] == 1
-
-
def test_3dcode_skills_list_shows_every_bundles_claim():
"""The ledger has to be visible from the CLI, or it is a document nobody opens."""
from typer.testing import CliRunner