Skip to content

Official Codebase of 3DCodeVerse

Python

Repo docs for the plugin layout; eval/llm keeps its ten battery prompt files and its own JSONL reader

gaoypeng1ab2ebfd450f
diff --git a/CLAUDE.md b/CLAUDE.md
index b7a88138..4503c5a1 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -1,14 +1,15 @@
# 3dcodeverse (repo root)
-The harness lives in **`harness/`** — read `harness/CLAUDE.md` +
-`harness/docs/ARCHITECTURE.md` before touching it. Install with
-`pip install -e harness`. Run tests with
-`cd harness && python -m pytest tests -q -m "not live"`.
+**`harness/`** is the `3dcodeverse` plugin for Claude Code, Codex and Antigravity: skills, a judge
+subagent and the `3dcode` command-line instruments (build, measure, render, gates, judge pack, score).
+Read `harness/CLAUDE.md` + `harness/docs/ARCHITECTURE.md` before touching it. The repo root is its
+marketplace (`.claude-plugin/marketplace.json`). Install: `bash harness/setup.sh`. Tests live outside
+the repo (`~/3dcodeverse_tests`); never add a `tests/` folder back under `harness/`.
-**`eval/`** evaluates things and is never imported by the harness: `eval/bench` evaluates the HARNESS
-(batteries — `cd eval && python -m bench.run_bench <battery.yaml>` —
-A/B rigs, reports; its tests: `cd eval && python -m pytest`), `eval/llm` evaluates a bare LLM/VLM
-(was `finetune/3dcodeverse_eval`). Read `eval/README.md` first.
+**`flywheel/`** (`c3d_flywheel`, CLI `c3d-flywheel`) reads runs recorded by the old CLI harness (v0.0.x):
+show/status, cost, gallery, dataset export/pairs/refine, pick. It depends on the plugin, never the reverse.
+
+**`eval/llm`** evaluates a bare LLM/VLM on 3D coding (no harness in the loop); read `eval/README.md`.
Other top-level folders are separate components: `toolkits/` owns the general data CLI and source curation;
`finetune/` owns training and its data preparation. The canonical LLaMA-Factory data scripts live in
diff --git a/README.md b/README.md
index 179020dc..9f0a7e1c 100644
--- a/README.md
+++ b/README.md
@@ -78,32 +78,31 @@ on 805 fixed tasks:
## Quick Start
-The harness needs Python 3.13, Node 20.6+, Git, Gemini API keys and the Antigravity CLI (`agy`, the
-default generator). Blender 5.2 LTS is optional (the `blender`, `urdf_blender` and `scene_blender`
-languages). Full guide: [harness/docs/INSTALL.md](harness/docs/INSTALL.md).
-The default judge is `gemini:gemini-4-argon` with thinking `medium`.
+The harness ships as **`3dcodeverse`, a plugin for Claude Code, Codex and Antigravity**: the host agent
+writes the code with its own model, and the plugin's skills and `3dcode` instruments build, measure,
+render, gate and score it (a fresh-context subagent judges). It needs Python 3.13, Node 20.6+ and Git;
+Blender 5.2 LTS for the `blender`, `urdf_blender` and `scene_blender` languages.
+Full guide: [harness/README.md](harness/README.md) and [harness/docs/INSTALL.md](harness/docs/INSTALL.md).
```bash
-cd harness
-bash setup.sh # pip install -e . + runtime_js + headless Chrome + the agy / claude-code / codex CLIs at their latest, then `3dcodeverse doctor`
-export GEMINI_API_KEYS="key1,key2" # agy runs in Gemini-API mode on this key pool
-3dcodeverse doctor # every row OK or WARN; a FAIL row says what to fix
-3dcodeverse make "a classic park bench" --track static_object --language threejs
-3dcodeverse gallery serve # browse every run at http://127.0.0.1:8765/
+git clone https://github.com/gaoypeng/3dcodeverse && cd 3dcodeverse
+bash harness/setup.sh # the `3dcode` instruments + runtime_js + headless Chrome, then doctor
+claude plugin marketplace add . && claude plugin install 3dcodeverse@3dcodeverse # Claude Code
+codex plugin marketplace add . && codex plugin add 3dcodeverse@3dcodeverse # Codex
+agy plugin install harness # Antigravity
```
-The optional generators claude-code and codex use your local subscriptions: `--generator claude-code:<model>`
-or `--generator codex:<model>`. Tracks, languages and costs per track: [harness/README.md](harness/README.md) and
-[harness/docs/RUNBOOK.md](harness/docs/RUNBOOK.md); evaluation: [eval/README.md](eval/README.md).
+Then ask your agent for a 3D thing, e.g. "Make a mid-century wooden dining chair in Blender bpy."
## Repository Structure
| Directory | Description | Documentation |
| :-- | :-- | :-- |
-| [`harness/`](harness/) | Agent-based 3D content generation, execution, rendering, and verification. | [Harness guide](harness/README.md) |
+| [`harness/`](harness/) | The `3dcodeverse` agent plugin: skills, judge subagent, and the `3dcode` build / measure / render / gate / score instruments. | [Plugin guide](harness/README.md) |
+| [`flywheel/`](flywheel/) | Reader for runs recorded by the v0.0.x CLI harness: gallery, cost, dataset export, pick. | [Flywheel guide](flywheel/README.md) |
| [`toolkits/`](toolkits/) | General data CLI and per-source corpus curation. | [Data toolkit guide](toolkits/README.md) |
| [`finetune/`](finetune/) | LLaMA-Factory training-data preparation, supervised fine-tuning, and preference optimization for open models. | [Training guide](finetune/README.md) |
-| [`eval/`](eval/) | Benchmarks and evaluation pipelines for the harness and standalone models. | [Evaluation guide](eval/README.md) |
+| [`eval/`](eval/) | Evaluation of standalone LLMs / VLMs on 3D coding. | [Evaluation guide](eval/README.md) |
## Citation
diff --git a/eval/README.md b/eval/README.md
index 62772b68..b02954af 100644
--- a/eval/README.md
+++ b/eval/README.md
@@ -1,38 +1,16 @@
-# eval/ — two evaluations, kept apart from what they evaluate
+# eval/ — evaluating a bare LLM / VLM on 3D coding
-| folder | what it evaluates | how |
-|---|---|---|
-| [`bench/`](bench/) | **the harness** (`../harness`): does plan → build → gates → judge → refine beat asking a model once? | prompt batteries run through the harness and through one-shot / bare-agent arms, judged by ONE fixed evaluator; paired A/B rigs for harness switches; offline re-judging and reports |
-| [`llm/`](llm/) | **a bare LLM / VLM** on 3D coding — no harness in the loop | text → code and image → code over 3DCodeBench, the held-out dialect sets and ten harness batteries; every answer is executed; geometry and image metrics against references |
-
-`docs/` holds the evaluation write-ups: `EVAL.md` (protocol, judge calibration, every recorded
-comparison), `COMPLEXITY.md` (the complexity vector and the score-vs-complexity study),
-`PAPER_WRITING.md`. `tests/` tests the `bench` scripts.
-
-The dependency is one-way: `bench` and `llm` import `codeverse3d` (the harness package); nothing under
-`harness/` imports anything here, and the harness test suite passes without this folder. The two
-evaluations share one set of battery files — `bench/prompts/*.yaml` — and `llm` reads the ten it can
-ask one-shot from there (`llm/config.py: HARNESS_BATTERIES`).
-
-Harness evaluation defaults to judge `gemini:gemini-4-argon` with thinking `medium`.
-Keep the judge model and sampling settings fixed across comparison arms; historical reports
-retain the judge they were measured with.
+[`llm/`](llm/) asks a model one-shot — no plugin, no tools in the loop — over 3DCodeBench, the held-out
+dialect sets and ten prompt batteries (`llm/batteries/*.yaml`); every answer is executed and scored with
+geometry and image metrics against references. `docs/` holds the earlier evaluation write-ups
+(`EVAL.md`, `COMPLEXITY.md`, `PAPER_WRITING.md`); the harness benchmark they describe (`eval/bench`) ran
+on the v0.0.x CLI harness and lives in those releases.
```bash
cd eval
-pip install -e ../harness # once; both evaluations import codeverse3d
-python -m bench.run_bench bench/prompts/static_objects_v1.yaml --generator agy:gemini-4-argon
-python -m bench.report bench/out/static_objects_v1 # rebuild report.md + report.html
-python bench/compare_backends.py --prompts bench/prompts/compare_v1.yaml --arms harness:agy:gemini-4-argon,oneshot:gemini:gemini-3.8-flash --out bench/out/compare_v1
-python -m llm.config # doctor for the LLM evaluation: tools and assets it can / cannot find
-python -m pytest # the bench tests (offline)
-ruff check .
+python -m llm.config # what the evaluation can / cannot find
+python -m pytest llm/tests # offline tests
```
-Run data is never in git: `bench/out/` (battery output) and `llm/data/prompts/*.jsonl` (built by
-`python -m llm.build_prompts` from the gated Hub assets) are ignored. Every bench report reads its
-journals through `bench/_jsonl.py` (a truncated last line costs one row; the last row per cell wins)
-and states its uncertainty through `bench/stats.py` (the 95 % t-interval, the exact sign test). Every `bench` script bootstraps
-its own import path — `../harness` for THIS tree's `codeverse3d`, `.` for the `bench` package — so an
-editable install of a different checkout cannot silently stand in for the tree under test
-(`tests/test_worktree_import.py`).
+Run data is never in git: `llm/data/prompts/*.jsonl` is built by `python -m llm.build_prompts` from the
+gated Hub assets.
diff --git a/eval/llm/_jsonl.py b/eval/llm/_jsonl.py
index 6ec0aeba..22a6750e 100644
--- a/eval/llm/_jsonl.py
+++ b/eval/llm/_jsonl.py
@@ -10,13 +10,25 @@ byte) is skipped and that row simply runs again; a missing file has no rows."""
from __future__ import annotations
import json
-from functools import partial
from pathlib import Path
-from codeverse3d.proc import read_jsonl_lenient, write_text_atomic
+from codeverse3d.proc import write_text_atomic
-#: every JSON object line of a file, in order
-read_rows = partial(read_jsonl_lenient, dicts_only=True)
+
+def read_rows(path: Path | str) -> list[dict]:
+ """Every JSON object line of ``path``, in order; a line that does not parse is skipped."""
+ p = Path(path)
+ if not p.is_file():
+ return []
+ rows = []
+ for line in p.read_text(encoding="utf-8", errors="replace").splitlines():
+ try:
+ row = json.loads(line)
+ except ValueError:
+ continue
+ if isinstance(row, dict):
+ rows.append(row)
+ return rows
def write_rows(path: Path, rows: list[dict], *, ensure_ascii: bool = True) -> None:
diff --git a/eval/llm/batteries/articulated_v2.yaml b/eval/llm/batteries/articulated_v2.yaml
new file mode 100644
index 00000000..87ac89e3
--- /dev/null
+++ b/eval/llm/batteries/articulated_v2.yaml
@@ -0,0 +1,227 @@
+# articulated_v2 — multi-DoF battery (v1: easy 75% pass; hard discriminated at 50%).
+#
+# Every prompt has >= 3 joints and at least one COUPLED constraint: a clearance or
+# rest condition that only holds when several joints are posed together (folded stack
+# thickness, sashes passing in separate tracks, a lid resting on a prop stick).
+# must_have items name the MOTION with a quantity ("drawer slides 0.25 m out toward
+# the front without hitting the frame") so the judge can verify each DoF in the
+# per-joint limit poses. The harness renders rest + ONE joint at a time (lower/
+# upper limit) and IGNORES <mimic>, so coupled conditions are stated against the
+# rest pose or against the collision sweep, which samples random combined poses
+# and reports interpenetration as gate errors.
+# Difficulty axes: joint count (3-8 DoF), coupled clearances, mixed joint types
+# (revolute + prismatic + continuous in one model).
+# Expected discrimination: medium 30-60% pass (3-4 loosely coupled joints), hard
+# 0-30% pass (4+ joints with clearance coupling — v1 showed baseline collapse to
+# 0.25 on such prompts, with large refine deltas).
+name: articulated_v2
+track: articulated_object
+language: urdf_blender
+description: >
+ 14 articulated prompts (URDF + bpy link meshes), all >= 3 DoF with coupled
+ constraints. 4 medium, 10 hard. must_have names each joint's motion with a
+ quantity and at least one cross-joint clearance verifiable in the rest pose
+ or by the collision sweep over random combined poses.
+prompts:
+ # ------------------------------------------------------------------ medium
+ - id: art_med_architect_lamp
+ tier: medium
+ category: lighting
+ prompt: >
+ an adjustable architect lamp with four revolute joints: base swivel, shoulder,
+ elbow and head tilt, with parallel link bars beside the lower arm
+ must_have:
+ - "weighted round base"
+ - "arm assembly swivels at least 180 degrees about the vertical base axis"
+ - "lower arm about 0.35 m long, shoulder revolute raises it from 20 to 80 degrees"
+ - "upper arm about 0.35 m long, elbow revolute with at least 120 degrees range"
+ - "head tilt revolute at the wrist, shade opens downward"
+ - "twin parallel link bars alongside the lower arm"
+ - "shoulder at its lower limit folds the lower arm down toward the base without touching it"
+ - id: art_med_tool_chest
+ tier: medium
+ category: tools
+ prompt: >
+ a machinist tool chest with a rear-hinged lid over a top tray and three
+ drawers of decreasing height on slides
+ must_have:
+ - "chest carcass about 0.60 m wide and 0.40 m tall"
+ - "lid hinged along the rear edge, opens 90 degrees to expose a top tray"
+ - "3 drawers of decreasing height from bottom to top"
+ - "each drawer slides 0.25 m out toward the front without hitting its neighbours"
+ - "drawer pulls on every drawer front"
+ - "with the lid closed, each fully opened drawer clears the lid overhang"
+ - id: art_med_camera_tripod
+ tier: medium
+ category: props
+ prompt: >
+ a camera tripod: three legs that splay at the crown, a sliding lower section
+ per leg, and a pan-tilt head with a mounting plate
+ must_have:
+ - "3 legs, each splaying by revolute from vertical to 25 degrees at the crown"
+ - "each leg's lower section slides out 0.40 m along the leg axis"
+ - "pan joint at the head rotates continuously about the vertical axis"
+ - "tilt joint pitches the mounting plate from -90 to +30 degrees"
+ - "rest pose: legs at equal splay and full extension, mounting plate level about 1.4 m up"
+ - "rubber feet at all 3 leg tips, resting on the ground plane in the rest pose"
+ - id: art_med_dutch_door
+ tier: medium
+ category: architecture
+ prompt: >
+ a dutch stable door in its frame: independent upper and lower leaves on the
+ same jamb, a slide bolt that joins them, and a latch handle on the lower leaf
+ must_have:
+ - "door frame with both leaves fitted, total opening about 2.0 m tall"
+ - "upper leaf swings open 100 degrees on hinges at the jamb"
+ - "lower leaf swings open 100 degrees on hinges at the same jamb"
+ - "slide bolt on the upper leaf travels 0.08 m down into the lower leaf edge"
+ - "bolt at its extended limit spans the seam into a notch in the lower leaf edge (leaves closed at rest)"
+ - "shelf ledge on top of the lower leaf clears the swinging upper leaf in every sweep pose"
+ - "latch handle on the lower leaf"
+ # -------------------------------------------------------------------- hard
+ - id: art_hard_rolltop_desk
+ tier: hard
+ category: furniture
+ prompt: >
+ a roll-top desk: a tambour of hinged slats that retracts along curved side
+ tracks to expose pigeonholes and the desktop, plus two pedestal drawers
+ per side
+ must_have:
+ - "desk carcass with 2 pedestals and a knee-hole"
+ - "tambour approximated as at least 8 slats riding in curved side tracks"
+ - "tambour travels from fully closed at rest to fully open at its joint limit, moving toward the rear"
+ - "in the rest pose the closed tambour's bottom rail meets the desktop with under 0.01 m gap"
+ - "at the tambour's open limit at least 4 pigeonhole compartments are exposed"
+ - "2 drawers per pedestal, each slides 0.30 m out toward the front"
+ - "open drawers do not intersect the knee-hole space or each other"
+ - "drawer pulls and a writing surface about 0.75 m above the floor"
+ - id: art_hard_folding_workbench
+ tier: hard
+ category: tools
+ prompt: >
+ a folding workbench: a top on two leg frames that fold inward, each held
+ open by a diagonal locking brace with an elbow pivot
+ must_have:
+ - "top about 1.2 m by 0.6 m, work height 0.85 m when deployed"
+ - "2 leg frames, each folding inward by revolute at the top rail"
+ - "2 locking braces, each with an elbow revolute that snaps straight when deployed"
+ - "each brace elbow folds through at least 90 degrees"
+ - "each leg frame at its folded limit lies within 0.15 m of the top's underside, with no collision in any sweep pose"
+ - "rest pose: deployed, all 4 feet on the ground and braces straight"
+ - id: art_hard_parallel_clamp
+ tier: hard
+ category: tools
+ prompt: >
+ a parallel-jaw bench clamp: a fixed jaw and a moving jaw on two guide rods,
+ driven by a lead screw with a crank handle and a swivel foot pad
+ must_have:
+ - "fixed jaw body and moving jaw about 0.12 m wide"
+ - "2 guide rods passing through the moving jaw"
+ - "lead screw between the rods, crank handle rotates continuously"
+ - "moving jaw slides 0.12 m along the guide rods toward the fixed jaw"
+ - "jaw faces remain parallel over the whole travel"
+ - "swivel pad on the screw tip tilts up to 5 degrees against the fixed jaw"
+ - "at the moving jaw's closed limit the jaw faces meet with no interpenetration"
+ - id: art_hard_sash_window
+ tier: hard
+ category: architecture
+ prompt: >
+ a double-hung sash window: two counterweighted sashes sliding in separate
+ tracks of one frame, with a rotating lock lever on the meeting rail
+ must_have:
+ - "outer frame about 0.9 m by 1.5 m with sill and casing"
+ - "2 sashes, each divided into at least 4 panes by muntins"
+ - "lower sash slides up 0.40 m in the inner track"
+ - "upper sash slides down 0.35 m in the outer track"
+ - "sashes pass each other with visible clearance, never touching"
+ - "lock lever on the meeting rail rotates 90 degrees"
+ - "lower sash at its upper limit overlaps the upper sash, both visible in front view"
+ - id: art_hard_grand_piano
+ tier: hard
+ category: musical_instruments
+ prompt: >
+ a baby grand piano with a hinged main lid held by a prop stick, a rotating
+ fallboard over the keyboard, three pedals on a lyre, and a bench with a
+ hinged storage lid
+ must_have:
+ - "curved-rim piano body on 3 legs"
+ - "keyboard with white and black key rows exposed when the fallboard is open"
+ - "fallboard rotates 90 degrees from covering the keys to lying flat behind them"
+ - "main lid hinged along the straight side, held open about 30 degrees at its lower limit, hinging up to about 45 degrees"
+ - "prop stick pivots up from the rim; at its raised limit the stick tip meets the lid's underside within 0.01 m"
+ - "prop stick at rest lies flat inside the rim"
+ - "3 pedals on a lyre post under the body"
+ - "bench with a storage lid hinged at the rear, opening 60 degrees"
+ - id: art_hard_umbrella
+ tier: hard
+ category: props
+ prompt: >
+ an umbrella with a simplified rib mechanism: a runner sliding on the shaft
+ drives at least six ribs that pivot at the crown, with canopy panels
+ stretched between the ribs
+ must_have:
+ - "central shaft about 0.9 m with a curved handle"
+ - "runner slides 0.30 m up the shaft"
+ - "at least 6 ribs, each revolute at the crown"
+ - "ribs swing from folded along the shaft to 80 degrees open"
+ - "stretcher links connect the runner to each rib mid-point in the open rest pose"
+ - "rest pose: open, with canopy panels stretched between adjacent ribs"
+ - "each rib at its folded limit lies within 0.05 m of the shaft without crossing it"
+ - id: art_hard_casement_window
+ tier: hard
+ category: architecture
+ prompt: >
+ a casement window driven by a sill crank: the sash swings outward on side
+ hinges while a folding operator arm with a sliding shoe pushes its bottom rail
+ must_have:
+ - "frame about 0.6 m by 1.2 m with a gridded sash"
+ - "sash swings outward 90 degrees on hinges at one side edge"
+ - "crank handle at the sill rotates continuously"
+ - "operator arm with an elbow revolute links the sill to the sash bottom rail"
+ - "operator arm shoe slides 0.30 m along a track in the sash bottom rail"
+ - "rest pose: window closed, operator arm folded flat over the sill without touching the sash"
+ - "latch lever on the free edge rotates to lock the sash"
+ - id: art_hard_step_ladder
+ tier: hard
+ category: tools
+ prompt: >
+ an A-frame step ladder: a five-step front frame and a rear frame hinged at
+ the top cap, held open by two spreader arms with center elbows
+ must_have:
+ - "front frame with 5 flat steps"
+ - "rear frame hinged to the front frame at the top cap"
+ - "frames open from parallel to about 30 degrees apart"
+ - "2 spreader arms, each with a center elbow revolute"
+ - "rest pose: open, spreaders locked straight between the frames"
+ - "rear frame at its closed limit lies parallel to the front frame within 0.10 m; spreaders fold without hitting the steps in any sweep pose"
+ - "top cap platform and 4 feet touching the ground in the open rest pose"
+ - id: art_hard_drawleaf_table
+ tier: hard
+ category: furniture
+ prompt: >
+ a draw-leaf dining table: two end leaves stowed under the fixed center top
+ slide out horizontally on tapered runners, plus an apron drawer on one long
+ side
+ must_have:
+ - "fixed center top about 1.2 m by 0.9 m on 4 legs, surface about 0.75 m high"
+ - "2 end leaves about 0.45 m deep stowed under the center top at rest"
+ - "each leaf slides 0.45 m outward along the table's long axis"
+ - "each leaf rides on 2 tapered runners that emerge from under the top as it extends"
+ - "extended leaf clears the center top edge with a gap under 0.02 m"
+ - "apron drawer on one long side slides 0.25 m out"
+ - "leaves and drawer extend without touching the legs or each other in any sweep pose"
+ - "rest pose: both leaves fully stowed inside the center top's footprint"
+ - id: art_hard_scissor_mirror
+ tier: hard
+ category: household
+ prompt: >
+ a wall-mounted scissor extension mirror: a pantograph of three X-links
+ between a wall plate and a mirror head that also swivels and tilts
+ must_have:
+ - "wall plate anchoring the mechanism"
+ - "pantograph of at least 3 X-links with revolute pivots at centers and ends"
+ - "rest pose: mirror head retracted 0.15 m from the wall plate; each pantograph joint at its limit moves the head farther from the wall"
+ - "all pantograph bars lie in a horizontal plane, scissoring parallel to the floor"
+ - "mirror head swivels at least 90 degrees about its vertical post"
+ - "mirror tilts at least 30 degrees about its horizontal axis"
+ - "X-links nest without bar intersections in the retracted rest pose"
diff --git a/eval/llm/batteries/compare_v4.yaml b/eval/llm/batteries/compare_v4.yaml
new file mode 100644
index 00000000..2f6988d8
--- /dev/null
+++ b/eval/llm/batteries/compare_v4.yaml
@@ -0,0 +1,713 @@
+# compare_v4 — harness-vs-oneshot arms on the UNION of the two static batteries (40 prompts).
+#
+# Why: the compare_v2/v3 batteries hold 8/12 prompts, and docs/report.html measured that an
+# 8-prompt paired battery resolves a real difference of ~0.25, not the 0.02 the profiles are
+# tuned to. This battery is every static_objects_v2 prompt (4 medium controls + 16 hard, all
+# with dimensions_m) plus every static_objects_v1 prompt whose text is not already in v2
+# (8 easy + 4 medium + 8 hard), ids kept VERBATIM from their source battery so recorded
+# cells of those batteries stay comparable. Prompt-text duplicates dropped from v1:
+# furn_med_dining_chair, veh_med_pickup, tool_med_hand_drill, appl_med_toaster (== ctrl_*).
+#
+# must_have is the FIXED judge's acceptance checklist for every arm (no plan-derived items)
+# so harness and one-shot are judged alike. Tiers: 8 easy / 8 medium / 24 hard.
+name: compare_v4
+track: static_object
+language: blender
+description: >
+ 40 static-object prompts (blender / bpy) — the union of static_objects_v2 and
+ static_objects_v1 by prompt text — used to compare the harness loop against raw
+ one-shot generation under ONE fixed judge with enough prompts for a paired CI
+ narrower than the effects claimed. 8 easy / 8 medium / 24 hard.
+prompts:
+- id: ctrl_med_dining_chair
+ tier: medium
+ category: furniture
+ tags:
+ - control
+ - axis:decomposition
+ prompt: a mid-century wooden dining chair with a slatted backrest and tapered legs
+ must_have:
+ - four tapered legs, each thinner at the floor than at the seat
+ - seat slab about 0.42 m wide
+ - seat surface about 0.45 m above the floor
+ - backrest with at least 3 vertical slats
+ - back legs continue up into the backrest posts
+ - backrest top rail between 0.80 m and 0.95 m above the floor
+ - all four feet touch the ground plane
+ - no leg or slat intersects the seat slab
+ dimensions_m:
+ width: 0.45
+ depth: 0.5
+ height: 0.9
+- id: ctrl_med_pickup
+ tier: medium
+ category: vehicles
+ tags:
+ - control
+ - axis:decomposition
+ prompt: a pickup truck with a two-door cab, open cargo bed, wheel arches and side mirrors
+ must_have:
+ - cab with windshield and 2 side windows
+ - open rear bed with 3 walls and a tailgate
+ - 4 wheels with hubs, all touching the ground
+ - wheel diameter about 0.75 m
+ - front grille and 2 headlights
+ - 2 side mirrors on stalks
+ - bed floor about 0.80 m above the ground
+ - overall length about 5.2 m
+ dimensions_m:
+ length: 5.2
+ width: 1.9
+ height: 1.8
+- id: ctrl_med_toaster
+ tier: medium
+ category: appliances
+ tags:
+ - control
+ - axis:curved_organic
+ prompt: a two-slot retro toaster with rounded chrome body, a lever on the side, a browning dial and
+ a crumb tray
+ must_have:
+ - 2 bread slots on the top face
+ - each slot about 0.13 m long and under 0.04 m wide
+ - lever on one end face
+ - round dial knob on the front face
+ - rounded body with no sharp vertical edges
+ - crumb tray seam line near the base
+ - 4 small feet
+ - body about 0.28 m wide and 0.19 m tall
+ dimensions_m:
+ width: 0.28
+ depth: 0.17
+ height: 0.19
+- id: ctrl_med_hand_drill
+ tier: medium
+ category: tools
+ tags:
+ - control
+ - axis:decomposition
+ prompt: a cordless power drill with a pistol grip, trigger, battery pack in the base of the grip and
+ a keyless chuck holding a drill bit
+ must_have:
+ - pistol-grip body with the grip angled back from vertical
+ - trigger recessed at the top of the grip
+ - battery block under the grip, wider than the grip
+ - chuck at the front, about 0.04 m diameter
+ - drill bit protruding about 0.08 m from the chuck
+ - side vents on the motor housing
+ - overall length about 0.22 m from chuck tip to body rear
+ - overall height about 0.24 m from battery base to body top
+ dimensions_m:
+ length: 0.22
+ height: 0.24
+- id: mech_hard_coffee_grinder
+ tier: hard
+ category: mechanisms
+ tags:
+ - axis:decomposition
+ prompt: 'a hand-crank coffee grinder: square wooden box body with a small front drawer, cast-iron dome
+ top with an open hopper funnel, a conical burr visible inside the funnel throat, and a top crank arm
+ with a wooden knob
+
+ '
+ must_have:
+ - square wooden box body about 0.14 m wide and 0.12 m tall
+ - front drawer about 0.05 m tall with a round knob, in the lower half of the box
+ - drawer front sits flush in its opening with a visible seam
+ - metal dome top mounted on the box
+ - open hopper funnel about 0.08 m across at the rim
+ - conical burr visible inside the funnel throat
+ - central spindle about 0.01 m thick rising from the burr through the dome
+ - crank arm about 0.11 m long attached to the spindle top
+ - wooden knob on a short vertical pin at the crank's free end
+ - adjustment nut on the spindle above the crank hub
+ dimensions_m:
+ width: 0.14
+ depth: 0.14
+ height: 0.3
+- id: mech_hard_machinist_vise
+ tier: hard
+ category: tools
+ tags:
+ - axis:decomposition
+ - axis:thin_features
+ prompt: 'a machinist bench vise with an acme lead screw showing its thread helix, twin cylindrical guide
+ rails, a swivel base with two lock handles, serrated jaw plates and a sliding tommy-bar handle
+
+ '
+ must_have:
+ - fixed jaw and sliding jaw with a 0.05 m open gap between them
+ - flat jaw plates about 0.10 m wide with serration grooves
+ - lead screw between the jaws with at least 6 visible thread turns
+ - 2 cylindrical guide rails flanking the screw
+ - sliding jaw rides on the rails and the screw
+ - tommy bar about 0.16 m long through the screw head, with ball ends
+ - swivel base disc under the vise body
+ - 2 lock handles on the swivel base rim
+ - mounting flange with 3 bolt holes
+ - overall body length about 0.30 m
+ dimensions_m:
+ length: 0.3
+ width: 0.14
+ height: 0.18
+- id: mech_hard_drill_press
+ tier: hard
+ category: tools
+ tags:
+ - axis:decomposition
+ prompt: 'a benchtop drill press: cast base, vertical column, adjustable work table with a crank, head
+ casting with a belt housing, chuck with a bit over the table, three-spoke quill feed lever and a depth-stop
+ rod
+
+ '
+ must_have:
+ - flat base about 0.30 m by 0.20 m with 2 mounting slots
+ - vertical column about 0.05 m diameter rising from the base
+ - work table clamped to the column between base and head
+ - table raise crank handle below the table
+ - head casting at the top of the column
+ - belt housing on top of the head with a lid seam
+ - cylindrical motor behind the column, connected to the head
+ - chuck about 0.04 m diameter under the head with a drill bit pointing down
+ - bit tip aligned over the table center
+ - quill feed lever with 3 spokes ending in ball knobs
+ - depth-stop rod with 2 nuts beside the quill
+ - overall height about 0.85 m
+ dimensions_m:
+ height: 0.85
+ depth: 0.45
+- id: mech_hard_pitcher_pump
+ tier: hard
+ category: garden
+ tags:
+ - axis:decomposition
+ - axis:curved_organic
+ prompt: 'a cast-iron pitcher water pump on a wooden platform: fluted cylindrical body, curved downward
+ spout, long lever handle raised mid-stroke, connecting link to the piston rod, domed cap and a bolted
+ base flange
+
+ '
+ must_have:
+ - cylindrical pump body about 0.09 m diameter
+ - at least 6 vertical flutes on the body
+ - base flange with 4 bolt heads
+ - square wooden platform under the flange
+ - curved spout leaving the body two-thirds up, opening facing down
+ - domed cap on top of the body
+ - lever handle about 0.40 m long, raised at an angle
+ - handle pivots in a clevis bracket at the top rear of the body
+ - connecting link from the handle to a piston rod entering the cap
+ - overall height about 0.65 m
+ dimensions_m:
+ height: 0.65
+- id: thin_hard_garden_arch
+ tier: hard
+ category: garden
+ tags:
+ - axis:thin_features
+ prompt: 'a wrought-iron garden arch: two lattice side frames, a semicircular arch head filled with S-scrolls,
+ spear-point finials and a tie bar at the spring line
+
+ '
+ must_have:
+ - 2 side frames, each of 2 uprights about 0.01 m thick
+ - diamond lattice bars between the uprights of each side frame
+ - semicircular arch head spanning about 1.2 m between the side frames
+ - at least 6 S-scrolls filling the arch head
+ - at least 4 scrolls in each side frame
+ - 3 spear-point finials on top of the arch
+ - horizontal tie bar connecting both sides at the spring line
+ - small ground plates under all 4 upright feet
+ - overall height about 2.3 m
+ - 'all bars slender: no member thicker than 0.02 m'
+ dimensions_m:
+ width: 1.4
+ depth: 0.5
+ height: 2.3
+- id: thin_hard_violin
+ tier: hard
+ category: musical_instruments
+ tags:
+ - axis:thin_features
+ - axis:curved_organic
+ prompt: 'a violin: hourglass body with C-bout waists, arched top with two f-holes, a bridge, four strings
+ from tailpiece over the bridge along the fingerboard to the pegbox, a scroll head, four tuning pegs
+ and a chin rest
+
+ '
+ must_have:
+ - hourglass body with 2 concave C-bout waist cutaways
+ - body length about 0.36 m
+ - top plate visibly arched, not flat
+ - 2 f-holes through the top, one each side of the bridge
+ - bridge about 0.03 m tall standing upright between the f-holes
+ - 4 strings running in parallel from tailpiece to pegbox
+ - strings pass over the bridge with a visible break angle
+ - black fingerboard extending over the body without touching it
+ - spiral scroll at the head
+ - 4 tuning pegs, 2 per side of the pegbox
+ - tailpiece anchored toward the end button
+ - chin rest beside the tailpiece
+ - total length about 0.60 m
+ dimensions_m:
+ length: 0.6
+ width: 0.21
+ height: 0.09
+- id: thin_hard_penny_farthing
+ tier: hard
+ category: vehicles
+ tags:
+ - axis:thin_features
+ - axis:curved_organic
+ prompt: 'a penny-farthing bicycle: huge spoked front wheel with pedal cranks on its hub, small spoked
+ rear wheel, curved backbone tube, front fork, handlebar and a sprung leather saddle
+
+ '
+ must_have:
+ - front wheel about 1.3 m diameter with at least 24 thin spokes
+ - rear wheel about 0.40 m diameter with at least 12 spokes
+ - tyres no thicker than 0.03 m
+ - backbone tube arcing from the fork crown down to the rear wheel fork
+ - front fork straddling the big wheel to its hub
+ - handlebar with 2 grips above the front hub
+ - 2 pedal cranks on the front hub with pedals, opposed at 180 degrees
+ - saddle on the backbone behind the handlebar, on a visible spring
+ - small mounting step peg on the backbone above the rear wheel
+ - both wheels touch the ground plane
+ dimensions_m:
+ length: 1.7
+ height: 1.5
+- id: thin_hard_birdcage
+ tier: hard
+ category: household
+ tags:
+ - axis:thin_features
+ prompt: 'a round wire birdcage with an onion-dome top, a hanging ring, a wire door with a latch, a trapeze
+ perch hanging inside, a straight perch, two feed cups and a base tray with a rolled rim
+
+ '
+ must_have:
+ - round cage about 0.35 m diameter
+ - at least 20 vertical wires around the circumference
+ - wires about 0.003 m thick
+ - 3 horizontal rings tying the wires
+ - onion-dome top converging to a finial
+ - hanging ring at the very top
+ - rectangular door frame set into the wires with a small latch
+ - trapeze perch hanging inside from the dome on 2 wires
+ - straight perch spanning the cage between opposite walls
+ - 2 feed cups mounted at a ring
+ - solid base tray with a rolled rim
+ - overall height about 0.55 m
+ dimensions_m:
+ height: 0.55
+- id: org_hard_bonsai_rock
+ tier: hard
+ category: plants
+ tags:
+ - axis:curved_organic
+ prompt: 'a root-over-rock bonsai: a shallow oval pot with soil, a rounded rock gripped by exposed roots,
+ a tapering trunk with two pronounced curves, and five distinct flattened foliage pads
+
+ '
+ must_have:
+ - shallow oval pot about 0.30 m long with a rim and small feet
+ - soil surface inside the pot
+ - rounded rock about 0.12 m tall sitting on the soil
+ - at least 4 roots gripping over the rock faces and entering the soil
+ - trunk rising from the rock top with at least 2 pronounced curves
+ - trunk tapers from about 0.04 m at the base to under 0.015 m at the apex
+ - 5 distinct flattened foliage pads with gaps between them
+ - lowest branch extends beyond the pot rim
+ - no foliage pad touches the soil or the rock
+ - overall height about 0.50 m
+ dimensions_m:
+ height: 0.5
+ length: 0.45
+- id: org_hard_seated_cat
+ tier: hard
+ category: animals
+ tags:
+ - axis:curved_organic
+ prompt: 'a domestic cat sitting upright: haunches on the ground, straight vertical front legs, tail
+ curled around beside the front paws, short muzzle and triangular ears
+
+ '
+ must_have:
+ - upright seated pose with haunches on the ground and chest raised
+ - 2 straight front legs, nearly vertical
+ - front paws on the ground between the haunches
+ - hind legs folded with thighs bulging wider than the chest
+ - tail about 0.30 m long curling around to lie beside the front paws
+ - head with a short muzzle, 2 eyes and a nose
+ - 2 triangular ears about 0.04 m tall
+ - back forms a smooth continuous curve from neck to tail base
+ - silhouette narrows from haunches up to the shoulders
+ - overall height about 0.35 m
+ dimensions_m:
+ height: 0.35
+ depth: 0.25
+- id: org_hard_octopus_rock
+ tier: hard
+ category: animals
+ tags:
+ - axis:curved_organic
+ - axis:thin_features
+ prompt: 'an octopus draped over a low rock: bulbous mantle, raised eyes, eight tapering arms curling
+ in different directions, with sucker discs visible along the underside of at least three arms
+
+ '
+ must_have:
+ - bulbous mantle about 0.20 m long
+ - 2 raised eye bumps on the head
+ - 8 arms radiating from the head
+ - each arm tapers from about 0.05 m at the base to a point
+ - arm tips curl, at least 4 curling in different directions
+ - sucker discs along the underside of at least 3 arms
+ - low rock under the body, arms draping over its sides
+ - at least 2 arms touch the ground plane
+ - overall arm spread about 0.8 m
+ dimensions_m:
+ length: 0.8
+ height: 0.3
+- id: arch_hard_bay_window
+ tier: hard
+ category: buildings
+ tags:
+ - axis:arch_detail
+ - axis:decomposition
+ prompt: 'a Victorian bay window facade section: a three-sided bay projecting from a clapboard wall,
+ a sash window with divided panes in each face, panel aprons, corner pilasters, scroll brackets and
+ a dentil cornice under a faceted roof cap
+
+ '
+ must_have:
+ - three-sided bay projecting from a flat wall section
+ - center face wider than the 2 angled side faces
+ - side faces angled about 45 degrees from the wall
+ - sash window in each of the 3 faces
+ - upper sash of each window divided into at least 4 panes by muntins
+ - continuous sill band under all 3 windows
+ - recessed panel apron under each window
+ - pilasters at the 4 vertical bay corners
+ - cornice with at least 10 dentil blocks across the bay
+ - at least 4 scroll brackets under the bay roof
+ - three-faceted roof cap over the bay
+ - horizontal siding boards on the flat wall section
+ dimensions_m:
+ width: 3.4
+ height: 3.2
+ depth: 1.0
+- id: arch_hard_spiral_stair
+ tier: hard
+ category: buildings
+ tags:
+ - axis:arch_detail
+ - axis:thin_features
+ prompt: 'a cast-iron spiral staircase: central column, wedge treads winding a full turn, a baluster
+ at each tread''s outer edge, a continuous helical handrail and a railed top landing
+
+ '
+ must_have:
+ - central column about 0.10 m diameter
+ - at least 12 wedge-shaped treads attached to the column
+ - treads climb evenly through at least 360 degrees total rotation
+ - 'open risers: visible gaps between consecutive treads'
+ - 1 baluster at the outer edge of each tread
+ - continuous helical handrail on top of the balusters
+ - top landing platform with its own railing
+ - base plate at the bottom of the column
+ - each tread about 0.70 m from column to outer edge
+ - overall height about 2.7 m
+ dimensions_m:
+ height: 2.7
+ width: 1.6
+- id: arch_hard_teahouse
+ tier: hard
+ category: buildings
+ tags:
+ - axis:arch_detail
+ prompt: 'a Japanese teahouse: raised wooden floor on short posts, a veranda strip along the front, sliding
+ shoji panels with visible grids, corner posts, and a hipped roof with deep eaves showing exposed rafter
+ tails
+
+ '
+ must_have:
+ - floor platform raised about 0.5 m on short posts
+ - wooden veranda strip along the front edge
+ - at least 3 sliding shoji panels in the front wall
+ - each shoji panel gridded into at least 12 rectangles
+ - square corner posts at the 4 corners
+ - hipped roof with eaves overhanging about 0.6 m
+ - at least 8 exposed rafter tails visible under each eave side
+ - ridge cap along the roof top
+ - solid back and side walls contrasting with the shoji front
+ - step stone below the veranda
+ dimensions_m:
+ width: 3.5
+ depth: 3.0
+ height: 3.0
+- id: comp_hard_drafting_table
+ tier: hard
+ category: furniture
+ tags:
+ - axis:multi_material
+ - axis:decomposition
+ prompt: 'a drafting table setup: tilted drawing board on a trestle frame with quadrant tilt brackets,
+ a parallel ruler riding on side rails, a pen tray, a sheet of paper held by two corner clips, a clamp-on
+ two-arm lamp aimed at the board, and a stool with a footring
+
+ '
+ must_have:
+ - drawing board about 1.0 m by 0.7 m tilted about 20 degrees
+ - parallel ruler bar spanning the board width
+ - ruler rides on a rail or wire at each side edge
+ - pen tray along the bottom edge of the board
+ - trestle legs with a cross stretcher
+ - quadrant tilt bracket with a knob at each of the board's 2 sides
+ - sheet of paper on the board held by clips at 2 corners
+ - lamp clamped to a top corner of the board
+ - lamp has 2 arm segments and a shade aimed down at the board
+ - stool with a round seat about 0.65 m high
+ - stool footring between its legs
+ - board top edge about 1.1 m above the floor
+ dimensions_m:
+ width: 1.1
+ depth: 0.9
+ height: 1.2
+- id: comp_hard_telescope_tripod
+ tier: hard
+ category: props
+ tags:
+ - axis:multi_material
+ - axis:decomposition
+ prompt: 'a brass refractor telescope on a wooden tripod: main tube with an objective ring, a thinner
+ draw tube with an eyepiece cup, a clamping collar and cradle on a pan-tilt head, three splayed legs
+ joined by a spreader tray, metal feet
+
+ '
+ must_have:
+ - main tube about 0.7 m long and 0.08 m diameter
+ - objective ring at the front of the tube
+ - draw tube about 0.05 m diameter extending from the rear, thinner than the main tube
+ - eyepiece cup at the draw tube end
+ - clamping collar around the main tube's balance point
+ - cradle connecting the collar to a pan-tilt head
+ - tension knob on the head
+ - 3 wooden legs splayed outward, each about 1.2 m long
+ - triangular spreader tray connecting the legs at about 0.7 m height
+ - metal feet at the leg tips
+ - tube tilted upward between 20 and 40 degrees
+ - eyepiece height about 1.5 m above the ground
+ dimensions_m:
+ height: 1.7
+- id: furn_easy_stool
+ tier: easy
+ category: furniture
+ prompt: a simple three-legged round wooden stool, 45 cm tall
+ must_have:
+ - round seat
+ - three splayed legs of equal length
+ - legs touch the ground
+ dimensions_m:
+ height: 0.45
+- id: furn_hard_rolltop_desk
+ tier: hard
+ category: furniture
+ prompt: a writing desk with two pedestals of three drawers each, a central knee-hole, a leather inlay
+ top and brass drawer pulls
+ must_have:
+ - two drawer pedestals
+ - three drawers per pedestal with visible fronts
+ - knee-hole gap
+ - brass pull handles
+ - desk top overhangs
+- id: veh_easy_toy_car
+ tier: easy
+ category: vehicles
+ prompt: a chunky toy car with a rounded body and four thick wheels
+ must_have:
+ - one-piece rounded body
+ - four wheels touching the ground
+ - wheels protrude sideways
+- id: veh_hard_tractor
+ tier: hard
+ category: vehicles
+ prompt: a farm tractor with large rear tyres with treads, small front wheels, an exposed engine block,
+ exhaust stack, driver seat, steering wheel and a roll bar
+ must_have:
+ - rear wheels much larger than front wheels
+ - tyre tread detail
+ - exhaust stack
+ - seat and steering wheel
+ - roll bar
+ - engine hood
+- id: tool_easy_hammer
+ tier: easy
+ category: tools
+ prompt: a claw hammer with a wooden handle and a steel head
+ must_have:
+ - cylindrical handle
+ - steel head with a flat face
+ - forked claw on the back
+- id: tool_hard_bench_vise
+ tier: hard
+ category: tools
+ prompt: a cast-iron bench vise with fixed and sliding jaws, a lead screw with a tommy bar handle, guide
+ rails, a small anvil face and mounting base with bolt holes
+ must_have:
+ - two jaws with a visible gap
+ - sliding jaw on guide rails
+ - lead screw with sliding T handle
+ - anvil flat
+ - base with bolt holes
+- id: appl_easy_kettle
+ tier: easy
+ category: appliances
+ prompt: an electric kettle with a spout, a lid and a side handle on a round base
+ must_have:
+ - pot body
+ - pouring spout
+ - handle opposite the spout
+ - lid on top
+ - round base
+- id: appl_hard_espresso
+ tier: hard
+ category: appliances
+ prompt: an espresso machine with a group head and portafilter, a steam wand, a drip tray with grate,
+ a water tank at the back, and a top-mounted cup warmer with a pressure gauge on the front panel
+ must_have:
+ - portafilter with handle under a group head
+ - steam wand on the side
+ - drip tray with grate
+ - water tank
+ - pressure gauge
+ - cup warmer rail
+- id: plant_easy_cactus
+ tier: easy
+ category: plants
+ prompt: a potted barrel cactus in a terracotta pot with soil
+ must_have:
+ - ribbed round cactus body
+ - terracotta pot with rim
+ - soil surface
+- id: plant_med_palm
+ tier: medium
+ category: plants
+ prompt: a small potted palm with a textured trunk and eight arching fronds made of many leaflets
+ must_have:
+ - textured trunk
+ - at least eight fronds
+ - fronds arch downward
+ - leaflets along each frond
+ - pot
+- id: plant_hard_bonsai
+ tier: hard
+ category: plants
+ prompt: a bonsai tree with a twisted thick trunk, three layered foliage pads, exposed roots over a rock
+ and a shallow oval ceramic tray with moss
+ must_have:
+ - twisted trunk with taper
+ - at least three distinct foliage pads
+ - exposed roots gripping a rock
+ - shallow oval tray
+ - moss surface
+- id: bld_easy_shed
+ tier: easy
+ category: buildings
+ prompt: a small garden shed with a pitched roof, one door and one window
+ must_have:
+ - box walls
+ - pitched roof overhang
+ - door on front
+ - window
+- id: bld_med_lighthouse
+ tier: medium
+ category: buildings
+ prompt: a tapered lighthouse with red and white bands, a gallery with railing, a glass lantern room
+ and a domed cap, on a rocky base
+ must_have:
+ - tapered tower
+ - alternating bands
+ - gallery walkway with railing
+ - glass lantern room
+ - dome
+ - rock base
+- id: bld_hard_windmill
+ tier: hard
+ category: buildings
+ prompt: a Dutch windmill with an octagonal tapered body, thatched cap, four lattice sails, a balcony
+ stage with railing, a door, and two shuttered windows
+ must_have:
+ - octagonal tapered body
+ - cap
+ - four lattice sails in a cross
+ - balcony with railing
+ - door
+ - shuttered windows
+- id: toy_easy_blocks
+ tier: easy
+ category: toys
+ prompt: a stack of five alphabet blocks of decreasing size in a slightly twisted tower
+ must_have:
+ - five cubes
+ - decreasing size upward
+ - each slightly rotated
+ - all stacked touching
+- id: toy_med_rocking_horse
+ tier: medium
+ category: toys
+ prompt: a wooden rocking horse with curved rockers, a saddle, handles and a mane
+ must_have:
+ - horse body with head and four legs
+ - two curved rockers
+ - saddle
+ - handle grips
+ - mane
+- id: toy_hard_train
+ tier: hard
+ category: toys
+ prompt: a toy steam locomotive with a boiler, smokestack, cab with windows, cowcatcher, six driving
+ wheels with connecting rods, and a coal tender with four wheels
+ must_have:
+ - cylindrical boiler
+ - smokestack
+ - cab
+ - cowcatcher
+ - six large wheels with rods
+ - separate tender with coal load and four wheels
+- id: mus_easy_drum
+ tier: easy
+ category: musical_instruments
+ prompt: a snare drum with tension rods and two drumsticks resting on top
+ must_have:
+ - cylindrical shell
+ - top head
+ - at least eight tension rods around the rim
+ - two sticks
+- id: mus_med_acoustic_guitar
+ tier: medium
+ category: musical_instruments
+ prompt: an acoustic guitar with a figure-eight body, sound hole, bridge, fretted neck, headstock with
+ six tuning pegs and six strings
+ must_have:
+ - figure-eight body
+ - sound hole
+ - bridge
+ - neck with frets
+ - headstock
+ - six tuning pegs
+ - six strings
+- id: mus_hard_drum_kit
+ tier: hard
+ category: musical_instruments
+ prompt: a five-piece drum kit with a bass drum with pedal, snare on a stand, two rack toms, a floor
+ tom, a hi-hat and two cymbals on stands, and a throne
+ must_have:
+ - bass drum with pedal
+ - snare on stand
+ - two mounted toms
+ - floor tom with legs
+ - hi-hat with two cymbals
+ - two crash/ride cymbals on stands
+ - drum throne
diff --git a/eval/llm/batteries/complexity_v3.yaml b/eval/llm/batteries/complexity_v3.yaml
new file mode 100644
index 00000000..4e9317e1
--- /dev/null
+++ b/eval/llm/batteries/complexity_v3.yaml
@@ -0,0 +1,249 @@
+# complexity_v3 — the COMPLEXITY LADDER battery.
+#
+# Purpose: make the framework's complexity CEILING measurable over time. The other
+# batteries ask "can it build this?"; this one asks "how much artifact can it build
+# before the score falls over?". The 12 prompts climb one axis on purpose — the
+# number of nameable parts, from a 3-part stool to a 25-part beam engine with a
+# visible mechanism — while the secondary axes (thin features, repetition, hollow
+# sections, mouldings, mechanism) are switched on in a documented order.
+#
+# Every prompt declares `expected_complexity: [lo, hi]`, the band the BUILT artifact's
+# objective complexity index (codeverse3d/spatial/complexity.py, docs/COMPLEXITY.md)
+# should land in. A run that scores well but lands BELOW its band did the easy thing
+# instead of the asked thing — the report calls that out as LOW, and it is the failure
+# mode this battery exists to catch. `expected_complexity` is not read by run_bench
+# (extra keys are ignored); `bench/complexity_report.py --battery` reads it.
+#
+# python -m bench.run_bench bench/prompts/complexity_v3.yaml --out bench/out/complexity_v3 \
+# --generator agy:gemini-3.6-flash --judge gemini:gemini-3.1-pro-preview
+# python bench/complexity_report.py bench/out/complexity_v3 --battery bench/prompts/complexity_v3.yaml
+#
+# Ladder (measured index of the nearest corpus run is in brackets):
+# a-tier 3-6 parts solid primitives + one curve 0.25-0.52 [furn_easy_stool 0.40]
+# b-tier 8-12 parts drawers, panes, first mechanism 0.45-0.66 [ctrl_med_dining_chair 0.54]
+# c-tier 14-18 parts repetition, spokes, mouldings 0.55-0.75 [comp_hard_telescope 0.65]
+# d-tier 20-25 parts thin repetition + working mechanism 0.62-0.85 [arch_hard_spiral_stair 0.84]
+name: complexity_v3
+track: static_object
+language: blender
+description: >
+ 12 static-object prompts on one deliberate complexity ladder (3 → 25 nameable parts,
+ with thin features, repeated elements, hollow sections, mouldings and a visible
+ mechanism switched on in order). Each prompt states the objective complexity band
+ its built artifact is expected to reach, so the ceiling can be tracked over time
+ instead of argued about.
+prompts:
+ # ------------------------------------------------------------------ a-tier (3-6 parts)
+ - id: cx_a03_milking_stool
+ tier: easy
+ category: furniture
+ tags: ["axis:parts:3", "ladder:a"]
+ expected_complexity: [0.25, 0.42]
+ prompt: a three-legged wooden milking stool with a round dished seat and splayed round legs
+ must_have:
+ - "round seat about 0.30 m across"
+ - "seat top dished (concave) by about 0.008 m, not flat"
+ - "exactly three round legs"
+ - "legs splay outward so the floor circle is wider than the seat"
+ - "seat surface about 0.45 m above the floor"
+ - "leg diameter about 0.04 m at the seat"
+ - "all three feet touch the ground plane"
+ - "chamfered seat edge, chamfer about 0.006 m"
+ dimensions_m: {width: 0.36, depth: 0.36, height: 0.45}
+ - id: cx_a05_wall_shelf
+ tier: easy
+ category: furniture
+ tags: ["axis:parts:5", "ladder:a"]
+ expected_complexity: [0.30, 0.46]
+ prompt: an oak wall shelf on two cast-iron L brackets, with a moulded front edge and a small back lip
+ must_have:
+ - "shelf board about 0.80 m wide and 0.22 m deep"
+ - "board thickness about 0.025 m"
+ - "moulded (profiled, not square) front edge"
+ - "back lip about 0.03 m tall along the rear edge"
+ - "exactly two L brackets under the board"
+ - "each bracket has a vertical arm, a horizontal arm and a diagonal brace"
+ - "four screw heads visible on the bracket vertical arms"
+ - "brackets set in about 0.10 m from each end"
+ dimensions_m: {width: 0.80, depth: 0.24, height: 0.20}
+ - id: cx_a06_watering_can
+ tier: easy
+ category: tools
+ tags: ["axis:parts:6", "axis:hollow", "axis:curved", "ladder:a"]
+ expected_complexity: [0.38, 0.54]
+ prompt: a galvanised steel watering can with a long tapered spout, a rose head, an arched carry handle and a rear grip handle
+ must_have:
+ - "tapered body about 0.26 m tall with a visibly open top"
+ - "visible wall thickness of about 0.003 m at the open rim"
+ - "long spout about 0.30 m long, rising from the body base to above the rim"
+ - "perforated rose head about 0.07 m across on the spout end"
+ - "arched carry handle over the top"
+ - "separate rear grip handle"
+ - "rolled bead about 0.006 m thick around the body rim"
+ - "body diameter about 0.22 m at the base"
+ dimensions_m: {width: 0.46, depth: 0.22, height: 0.34}
+ # ------------------------------------------------------------------ b-tier (8-12 parts)
+ - id: cx_b08_writing_desk
+ tier: medium
+ category: furniture
+ tags: ["axis:parts:8", "axis:moulding", "ladder:b"]
+ expected_complexity: [0.44, 0.60]
+ prompt: a walnut writing desk with one central drawer, turned tapered legs, a moulded top edge and a leather inlay panel
+ must_have:
+ - "desk top about 1.20 m by 0.60 m"
+ - "top surface about 0.75 m above the floor"
+ - "moulded (profiled) top edge on all four sides, profile depth about 0.012 m"
+ - "leather inlay panel recessed about 0.003 m into the top"
+ - "one drawer in the apron with a visible drawer front"
+ - "drawer pull about 0.09 m wide, modelled as separate geometry"
+ - "four turned legs, tapering from about 0.06 m to 0.035 m"
+ - "apron rails between the legs on three sides"
+ dimensions_m: {width: 1.20, depth: 0.60, height: 0.75}
+ - id: cx_b10_gas_lantern
+ tier: medium
+ category: lighting
+ tags: ["axis:parts:10", "axis:thin_features", "axis:hollow", "ladder:b"]
+ expected_complexity: [0.48, 0.64]
+ prompt: a Victorian cast-iron gas lantern head on a fluted post - four glass panes in a tapered glazing frame, a vented crown cap and a finial
+ must_have:
+ - "fluted post about 2.2 m tall"
+ - "tapered four-sided glazing frame with visible corner bars under 0.02 m thick"
+ - "four separate glass panes about 0.004 m thick set inside the frame"
+ - "vented crown cap above the glazing"
+ - "finial about 0.12 m tall on top of the cap"
+ - "flared collar where the head meets the post"
+ - "square base plate about 0.35 m across"
+ - "ladder rest bar projecting from the post below the head"
+ dimensions_m: {width: 0.42, depth: 0.42, height: 2.60}
+ - id: cx_b12_sewing_machine
+ tier: medium
+ category: machines
+ tags: ["axis:parts:12", "axis:mechanism", "ladder:b"]
+ expected_complexity: [0.52, 0.68]
+ prompt: a cast-iron hand-crank sewing machine on a wooden base - curved arm, needle bar, presser foot, spool pin, hand wheel and drive belt
+ must_have:
+ - "wooden base about 0.40 m by 0.20 m"
+ - "curved cast-iron arm reaching about 0.22 m forward over the bed"
+ - "needle bar about 0.05 m long hanging from the arm head with a needle below it"
+ - "presser foot about 0.02 m long on the bed under the needle"
+ - "hand wheel about 0.09 m across on the right of the pillar"
+ - "crank handle on the hand wheel, offset from its centre"
+ - "spool pin about 0.05 m tall standing on top of the arm"
+ - "bobbin winder on the pillar"
+ - "flat bed plate with a visible needle plate"
+ - "tension discs on the arm face"
+ dimensions_m: {width: 0.40, depth: 0.20, height: 0.32}
+ # ------------------------------------------------------------------ c-tier (14-18 parts)
+ - id: cx_c14_ship_binnacle
+ tier: hard
+ category: instruments
+ tags: ["axis:parts:14", "axis:repetition", "axis:curved", "ladder:c"]
+ expected_complexity: [0.55, 0.72]
+ prompt: a brass ship's binnacle with a domed hood, a compass bowl on gimbals, two spherical correctors on side arms and a fluted pedestal on a stepped base
+ must_have:
+ - "fluted pedestal about 1.1 m tall"
+ - "domed hood about 0.30 m across with a viewing window opening"
+ - "compass bowl about 0.20 m across inside the hood"
+ - "two gimbal rings under 0.01 m thick around the bowl"
+ - "two spherical correctors, one on each side arm"
+ - "side arms clamped to the pedestal at the same height"
+ - "stepped circular base about 0.45 m across"
+ - "at least eight flutes around the pedestal"
+ - "hinged lamp housing on the hood"
+ - "spherical corrector diameter about 0.16 m"
+ dimensions_m: {width: 0.90, depth: 0.45, height: 1.30}
+ - id: cx_c16_apothecary_cabinet
+ tier: hard
+ category: furniture
+ tags: ["axis:parts:16", "axis:repetition", "axis:moulding", "ladder:c"]
+ expected_complexity: [0.56, 0.74]
+ prompt: an apothecary cabinet - a twelve-drawer grid in a moulded carcass on a plinth, each drawer with a turned knob and a brass label frame
+ must_have:
+ - "carcass about 0.90 m wide and 1.10 m tall"
+ - "twelve drawer fronts in a 4 by 3 grid"
+ - "each drawer front is inset, with a visible reveal about 0.003 m around it"
+ - "a turned knob about 0.03 m across on every drawer front"
+ - "a rectangular label frame on every drawer front"
+ - "moulded cornice about 0.05 m tall along the top"
+ - "plinth base about 0.10 m tall"
+ - "vertical divider between drawer columns"
+ - "carcass depth about 0.35 m"
+ - "drawer front height about 0.20 m"
+ dimensions_m: {width: 0.90, depth: 0.35, height: 1.10}
+ - id: cx_c18_orrery
+ tier: hard
+ category: instruments
+ tags: ["axis:parts:18", "axis:mechanism", "axis:thin_features", "ladder:c"]
+ expected_complexity: [0.60, 0.78]
+ prompt: a brass tabletop orrery - a central sun on a spindle, three planet arms of different lengths each carrying a sphere, a moon on a sub-arm, an exposed gear train and a turned wooden base
+ must_have:
+ - "turned wooden base about 0.30 m across"
+ - "central spindle carrying a sun sphere about 0.06 m across"
+ - "exactly three planet arms of different lengths"
+ - "a planet sphere at the end of each arm, all different sizes"
+ - "a moon on a short sub-arm off the outermost planet"
+ - "at least three toothed gears, the largest about 0.10 m across, visible under the arms"
+ - "gear teeth modelled as geometry, not painted"
+ - "arms under 0.008 m thick"
+ - "a crank handle about 0.05 m long on the side of the gear housing"
+ - "engraved zodiac ring about 0.015 m wide around the base rim"
+ dimensions_m: {width: 0.40, depth: 0.40, height: 0.28}
+ # ------------------------------------------------------------------ d-tier (20-25 parts)
+ - id: cx_d20_floor_loom
+ tier: hard
+ category: machines
+ tags: ["axis:parts:20", "axis:thin_features", "axis:repetition", "ladder:d"]
+ expected_complexity: [0.62, 0.82]
+ prompt: a wooden four-shaft floor loom - side frames, warp and cloth beams, four heddle shafts hung on cords, a swinging beater with a reed, two treadles and a bench
+ must_have:
+ - "two side frames about 1.1 m tall joined by cross rails"
+ - "warp beam at the back and cloth beam at the front, both round and about 0.08 m across"
+ - "exactly four heddle shafts hanging in the middle of the frame"
+ - "at least six vertical heddle wires per shaft, under 0.004 m thick"
+ - "swinging beater about 0.7 m long pivoted at the top, carrying a reed"
+ - "at least twelve reed dents modelled in the beater"
+ - "two treadles about 0.6 m long at floor level, connected upward by cords"
+ - "ratchet wheel and pawl on the cloth beam"
+ - "separate bench in front of the loom"
+ - "overall depth about 1.2 m"
+ dimensions_m: {width: 1.10, depth: 1.60, height: 1.20}
+ - id: cx_d22_letterpress
+ tier: hard
+ category: machines
+ tags: ["axis:parts:22", "axis:mechanism", "axis:repetition", "ladder:d"]
+ expected_complexity: [0.64, 0.84]
+ prompt: a cast-iron platen letterpress - a round ink disc, two ink rollers on a saddle, a hinged platen, a chase bed, a treadle, a flywheel with a crank and a scrolled cast frame on legs
+ must_have:
+ - "scrolled cast-iron side frames about 1.4 m tall"
+ - "round ink disc about 0.35 m across at the top, tilted forward"
+ - "two ink rollers about 0.05 m across on a roller saddle between disc and bed"
+ - "hinged platen about 0.25 m by 0.20 m that faces the chase bed"
+ - "chase bed with a rectangular chase about 0.20 m by 0.15 m inside it"
+ - "flywheel about 0.55 m across on the right side"
+ - "crank arm connecting the flywheel to the platen linkage"
+ - "treadle bar about 0.5 m long at floor level linked upward by a rod"
+ - "at least six spokes in the flywheel"
+ - "gripper fingers on the platen face"
+ - "throw-off lever on the frame"
+ dimensions_m: {width: 0.90, depth: 1.20, height: 1.60}
+ - id: cx_d25_beam_engine
+ tier: hard
+ category: machines
+ tags: ["axis:parts:25", "axis:mechanism", "axis:repetition", "axis:thin_features", "ladder:d"]
+ expected_complexity: [0.68, 0.88]
+ prompt: a model Watt beam engine on a cast base - a rocking overhead beam on an A-frame column, piston cylinder with rod and crosshead, connecting rod to a crank, a spoked flywheel, a centrifugal governor with two balls, and exposed valve linkage
+ must_have:
+ - "cast base plate about 0.60 m by 0.30 m"
+ - "A-frame column carrying the beam pivot about 0.55 m above the base"
+ - "rocking beam about 0.45 m long with an eye at each end"
+ - "vertical piston cylinder about 0.09 m across with a visible rod emerging from its top"
+ - "crosshead about 0.05 m wide joining the piston rod to the beam end"
+ - "connecting rod from the other beam end down to a crank"
+ - "spoked flywheel about 0.28 m across with at least six spokes"
+ - "crankshaft supported in two pedestal bearings"
+ - "centrifugal governor about 0.15 m tall with two arms and two balls about 0.03 m across"
+ - "governor driven by a belt or gear off the crankshaft"
+ - "valve rod running alongside the cylinder"
+ - "parallel motion linkage between the beam end and the piston rod"
+ - "flywheel rim under 0.03 m thick"
+ dimensions_m: {width: 0.60, depth: 0.30, height: 0.75}
diff --git a/eval/llm/batteries/fancy_v1_blender.yaml b/eval/llm/batteries/fancy_v1_blender.yaml
new file mode 100644
index 00000000..d20e853a
--- /dev/null
+++ b/eval/llm/batteries/fancy_v1_blender.yaml
@@ -0,0 +1,286 @@
+# fancy_v1 — blender slice (10 prompts). Complex, picture-worthy static objects, every
+# prompt written from how the real object is actually built: load-bearing parts, how they
+# attach, proportions in metres, where materials change, and the 2-4 details that make it
+# read as THAT object. Objects chosen for a distinctive silhouette, real internal structure,
+# repeats-and-mirrors (spokes, slats, teeth, fins) and at least one hollow. Blender slice
+# carries the organic and bevelled forms. All hard tier; ids fancy_bl_<snake>.
+# Sibling files: fancy_v1_cadquery.yaml, fancy_v1_threejs.yaml.
+name: fancy_v1_blender
+track: static_object
+language: blender
+description: >
+ fancy_v1 blender slice: 10 hard static objects with real construction described the way
+ a product designer would (Windsor chair, harp, ship's wheel, carousel horse, orrery ...),
+ each with a countable falsifiable must_have list and realistic overall dimensions.
+prompts:
+ # Steam-bent bow socketed into a saddled seat, turned spindles and splayed legs: organic
+ # carving plus a large count of thin repeated parts that must all land in the seat.
+ - id: fancy_bl_windsor_chair
+ tier: hard
+ category: furniture
+ tags: [fancy_v1, hard]
+ prompt: >
+ a sack-back Windsor armchair in the traditional construction: a thick elm seat plank,
+ about 0.55 m wide and 0.05 m thick, carved into a saddle with a raised pommel at the front; a steam-bent ash bow of round section about 0.02 m
+ thick rising in a continuous arch whose two ends socket down into the seat behind the
+ sitter; a flat steam-bent arm rail wrapping around at elbow height whose ends terminate
+ in small hand rests, supported at the front by a turned arm post on each side; seven
+ tapered spindles passing through holes in the arm rail up into the bow, plus four
+ shorter spindles under each arm; four turned legs with baluster and ring profiles
+ raked outward and splayed to the sides, socketed into the underside of the seat, tied
+ by an H-stretcher of three turned rungs. Milk-paint black-green, worn through to pale wood on the arm tops.
+ must_have:
+ - "seat plank about 0.55 m wide with a visibly scooped saddle, not a flat slab"
+ - "a continuous round-section bow arching over the back and entering the seat at exactly two points"
+ - "a flat arm rail at about 0.68 m above the floor wrapping around both sides of the sitter"
+ - "at least 7 long spindles pass through the arm rail and continue up into the bow"
+ - "at least 4 short spindles under each arm between arm rail and seat"
+ - "four turned legs raked outward so the feet sit wider than the seat"
+ - "all four feet touch the ground plane"
+ - "paint worn through to pale wood on the arm tops as a second material"
+ dimensions_m: {width: 0.62, depth: 0.58, height: 1.02}
+ # A concert harp: a real triangular load path (pillar, neck, soundboard-in-body) and 47
+ # graduated strings in a plane, with coloured C and F strings and a hollow resonator.
+ - id: fancy_bl_concert_harp
+ tier: hard
+ category: instruments
+ tags: [fancy_v1, hard]
+ prompt: >
+ a concert pedal harp built as the real triangle: a fluted gilded column (the pillar)
+ standing about 1.75 m tall at the front, a hollow tapered spruce sound body at the
+ back, its rounded staved back closed and its flat soundboard facing the player,
+ widening from about 0.10 m at the top to 0.50 m at the base; a curved maple neck
+ arching from the top of the column down to the top of the body with the brass action
+ plate running along its side carrying a row of tuning pins; 47 strings, all in one
+ plane, strung from pins on the neck down to a straight rib on the soundboard centre,
+ graduating from about 0.08 m long at the treble to 1.50 m long at the bass; every
+ seventh string red (the Cs) and one in four blue (the Fs), the others natural gut colour; a base plate with seven pedals in a row and
+ two rear feet; gold-leaf ornament on the column capital and the crown.
+ must_have:
+ - "a fluted front pillar about 1.75 m tall, standing vertically at the front"
+ - "a hollow sound body at the back that widens toward the base and shows a rounded staved back"
+ - "a curved neck joining the pillar top to the body top, forming a closed triangle"
+ - "at least 40 strings, all lying in one plane, each attached at both ends"
+ - "strings graduate in length from short at the treble end to long at the bass end"
+ - "some strings coloured red and some blue among natural-coloured ones"
+ - "seven pedals visible on the base plate"
+ - "gold-coloured ornament on the column capital distinct from the wood body"
+ dimensions_m: {width: 0.55, depth: 1.0, height: 1.85}
+ # Ship's wheel on a binnacle: turned spokes through a laminated rim, a hub with a brass
+ # boss, the wheel mounted on a pedestal that is itself a hollow box with a compass bowl.
+ - id: fancy_bl_ships_wheel
+ tier: hard
+ category: vehicles
+ tags: [fancy_v1, hard]
+ prompt: >
+ a ship's steering wheel on its binnacle: a mahogany rim about 1.20 m across built from
+ a laminated ring of round section about 0.05 m thick, through which pass eight turned
+ spokes; each spoke runs from the central hub, through the rim, and out beyond it as a
+ turned handle with a bulbous knob, so the handles sit about 0.14 m proud of the rim;
+ the hub is a thick disc with a domed brass boss at its centre; the wheel is mounted
+ vertically on a horizontal brass axle that runs back into a hollow teak binnacle
+ pedestal about 1.10 m tall, square in plan with chamfered corners, brass bands top
+ and bottom; on top of the binnacle a brass compass bowl under a glazed hemispherical
+ hood with a hinged lamp housing on the side; a brass grab rail around the pedestal top.
+ must_have:
+ - "a circular rim about 1.20 m in diameter with a round cross-section"
+ - "exactly 8 spokes spaced evenly, each passing through the rim and continuing outward as a handle"
+ - "each handle ends in a bulbous turned knob outside the rim"
+ - "a thick hub disc at the centre with a domed brass boss"
+ - "the wheel stands vertically in front of a square-plan pedestal about 1.10 m tall"
+ - "a hemispherical glazed compass hood on top of the pedestal"
+ - "brass bands and boss distinct in colour from the dark wood rim and spokes"
+ - "the pedestal base sits flat on the ground plane"
+ dimensions_m: {width: 1.35, depth: 0.75, height: 1.75}
+ # Carousel horse: organic sculpted body in a gallop pose with a brass pole through it,
+ # carved tack, glass jewels, a decorated saddle; the pole is the hard rigid reference.
+ - id: fancy_bl_carousel_horse
+ tier: hard
+ category: toys
+ tags: [fancy_v1, hard]
+ prompt: >
+ a carousel jumper horse in the full-gallop carving pose: the body about 1.50 m from
+ nose to tail root, front legs tucked up under the chest, hind legs stretched straight
+ back, the head turned slightly to the outside (the romance side) with an open mouth
+ and flared nostrils, a carved flowing mane falling to one side and a long swept tail;
+ a polished brass pole about 0.045 m thick passing vertically through the body just
+ behind the withers and exiting under the belly, continuing down to a round base plate;
+ carved tack in relief: a saddle with a raised cantle and a saddle blanket with a
+ scalloped trailing edge, a breast strap, a bridle with a browband; the romance side
+ dressed with at least six faceted glass jewels set in gilded rosettes along the saddle
+ blanket and breast strap; painted a dapple white body with a gold-and-red saddle, a
+ black hoof on each leg and a horseshoe on each hoof.
+ must_have:
+ - "a horse body about 1.50 m long with both front legs tucked up and both hind legs stretched back"
+ - "no hoof touches the ground; the horse is held up only by the pole"
+ - "a vertical brass pole passes through the body behind the withers and exits under the belly"
+ - "the pole continues down to a round base plate on the ground"
+ - "a saddle with a raised cantle and a blanket with a scalloped rear edge"
+ - "at least 6 jewel-like faceted bosses along the saddle blanket or breast strap"
+ - "a flowing mane and a long tail carved as separate masses from the body"
+ - "at least 3 distinct paint colours: pale body, red or gold tack, black hooves"
+ dimensions_m: {width: 0.55, depth: 1.6, height: 1.85}
+ # Brass orrery: nested concentric arms at different heights, gear stack under the base,
+ # spheres on stalks with a moon sub-arm, a scalloped skirt, hollow base with visible gears.
+ - id: fancy_bl_orrery
+ tier: hard
+ category: instruments
+ tags: [fancy_v1, hard]
+ prompt: >
+ a Georgian-style brass orrery: a round mahogany base about 0.45 m in diameter with a
+ turned edge, carrying a brass calendar ring engraved with the zodiac around its rim on
+ twelve small turned pillars; under the ring, in the hollow between base and ring, a
+ visible stack of brass spur gears of decreasing size driving a central vertical shaft;
+ at the top of the shaft a gilt sun ball about 0.03 m across on a short stalk; six planet
+ arms of flat brass strip radiating at six different heights on nested concentric
+ sleeves around the shaft, each arm ending in a polished sphere on a small vertical
+ stalk, lengths increasing from 0.06 m for Mercury to 0.22 m for Saturn; the Earth arm
+ carries a small sub-arm with a tiny moon and a tilted brass ring around the Earth ball;
+ Saturn wears a thin flat ring; a crank handle on the side of the base drives the gears.
+ must_have:
+ - "a round wooden base about 0.45 m in diameter"
+ - "a brass ring around the rim supported on at least 12 small pillars"
+ - "at least 4 visible spur gears of different sizes in the hollow under the ring"
+ - "exactly 6 radial arms at six different heights, each ending in a sphere on a stalk"
+ - "arm lengths increase outward so the outermost sphere is farthest from the shaft"
+ - "one arm carries a small sub-arm with a tiny moon beside its planet"
+ - "the outermost sphere has a thin flat ring around it"
+ - "brass parts distinct in colour from the dark wood base"
+ dimensions_m: {width: 0.5, depth: 0.5, height: 0.42}
+ # A pipe organ facade: a carved case with a hollow impost, graduated speaking pipes in
+ # mitred flats and towers, a keyboard console; a hard count-and-graduation object.
+ - id: fancy_bl_chamber_organ
+ tier: hard
+ category: instruments
+ tags: [fancy_v1, hard]
+ prompt: >
+ a chamber pipe organ in an oak case about 2.20 m wide and 3.20 m tall: a lower case
+ with two panelled doors, a projecting keydesk with two manuals of 56 keys each (the
+ upper set back from the lower), a music stand and eight stop knobs on each side jamb; above a moulded impost, the facade of speaking pipes arranged as a central
+ round tower of seven pipes in a semicircle, a flat of nine pipes on each side, and an
+ outer tower of five pipes on each side, the towers standing proud of the flats; all
+ facade pipes are polished tin cylinders with conical feet standing in a toe-board,
+ a mouth with a cut-up and upper lip cut in each one about a fifth of the way up, and
+ pipe lengths graduated symmetrically from tallest at the centre of each tower to
+ shortest at the flat ends; carved pierced pipe shades filling the arch above each
+ tower and flat; a crown cornice with a carved centre finial.
+ must_have:
+ - "an oak case about 2.20 m wide and 3.20 m tall standing on the ground plane"
+ - "a central tower of pipes standing proud of two flanking flats"
+ - "an outer tower of pipes at each end, making 5 pipe groups in a left-right mirror"
+ - "at least 30 facade pipes in total, all polished tin colour"
+ - "pipe heights within each group graduate symmetrically about the group centre"
+ - "each pipe has a conical foot and a visible mouth notch in its lower fifth"
+ - "two keyboards visible on a projecting keydesk, the upper set back from the lower"
+ - "at least 8 stop knobs in a column on each side of the keyboards"
+ dimensions_m: {width: 2.2, depth: 1.1, height: 3.2}
+ # Victorian cast-iron street lamp: a fluted tapered column, a hollow four-sided lantern
+ # with glazed panes and a finial, ladder bar, acanthus base; architectural detail object.
+ - id: fancy_bl_gas_street_lamp
+ tier: hard
+ category: architecture
+ tags: [fancy_v1, hard]
+ prompt: >
+ a Victorian cast-iron gas street lamp about 4.2 m tall: a square stepped plinth at
+ the base about 0.45 m across, an octagonal pedestal with a cast acanthus-leaf collar
+ at its top, a fluted tapered column of 16 flutes narrowing from 0.20 m to 0.12 m over
+ its height with two moulded astragal rings, a short horizontal ladder bar with a
+ ball at each end just below the lantern for the lamplighter; the lantern is a hollow
+ four-sided tapered box, wider at the top, with a glazed pane in each face held by
+ thin glazing bars, a sloped four-sided pyramidal roof with a vented cap and a pointed
+ finial, and a copper gas-mantle burner visible inside the glass; the iron painted
+ gloss black with the acanthus collar and finial picked out in gold leaf; the lantern
+ glass slightly frosted.
+ must_have:
+ - "a stepped square plinth at the base sitting on the ground plane"
+ - "a tapered column with at least 12 visible flutes"
+ - "a cast leaf collar at the top of the pedestal, gold-coloured"
+ - "a horizontal ladder bar with a ball at each end just below the lantern"
+ - "a four-sided lantern that is wider at the top than at the bottom"
+ - "a glazed pane in each of the four lantern faces"
+ - "a pyramidal roof on the lantern with a finial at its apex"
+ - "a burner element visible inside the lantern"
+ dimensions_m: {width: 0.55, depth: 0.55, height: 4.2}
+ # Brass microscope: a hollow drawtube, triple nosepiece with objectives, rack-and-pinion
+ # focus, a mirror on a gimbal under a stage with clips, a horseshoe foot; optics object.
+ - id: fancy_bl_brass_microscope
+ tier: hard
+ category: optics
+ tags: [fancy_v1, hard]
+ prompt: >
+ a late-Victorian brass compound microscope: a heavy horseshoe-shaped foot, two
+ trunnion pillars carrying a pivoted limb so the whole body tilts back, a square stage
+ about 0.09 m across with two spring stage clips and a central aperture, beneath it a
+ concave mirror about 0.04 m across on a gimbal fork that swings under the stage;
+ above the stage a curved limb carrying the coarse focus rack, with a pair of large
+ knurled coarse-focus knobs on a shaft either side and a smaller fine-focus knob on top
+ of the limb; the body tube about 0.03 m in diameter and 0.16 m long, with an inner
+ drawtube telescoped out of its top ending in an eyepiece cap; at the bottom of the
+ body tube a revolving nosepiece carrying three objectives of different lengths in a
+ rotating triple; lacquered polished brass throughout with black japanned foot and
+ the limb, silvered mirror, and a black eyepiece cap.
+ must_have:
+ - "a horseshoe-shaped foot sitting flat on the ground plane"
+ - "the body is pivoted between two pillars rising from the foot"
+ - "a square stage with two stage clips on its top surface"
+ - "a round mirror on a fork below the stage"
+ - "a vertical body tube with a narrower drawtube extending from its top"
+ - "exactly 3 objectives of different lengths hanging from a rotating nosepiece"
+ - "two large matching knurled focus knobs on a common shaft either side of the limb"
+ - "brass body distinct in colour from the black foot and the black eyepiece cap"
+ dimensions_m: {width: 0.15, depth: 0.2, height: 0.36}
+ # Espresso lever machine: a chrome dome boiler, a spring-lever group with a long handle,
+ # portafilter locked in, a drip tray grille, gauge, eagle finial; appliance in curves.
+ - id: fancy_bl_lever_espresso
+ tier: hard
+ category: appliances
+ tags: [fancy_v1, hard]
+ prompt: >
+ a classic Italian lever espresso machine of the domestic kind: a polished chrome
+ boiler cylinder about 0.16 m in diameter and 0.30 m tall standing on a cast alloy
+ base with four rubber feet; a domed lid on the boiler with a brass eagle finial on
+ top and a round pressure gauge with a white dial set into the front of the boiler; a
+ brass group head cast into the front of the boiler near its base, with a long
+ cylindrical spring-lever housing rising above it and a lever arm about 0.30 m long
+ with a black bakelite knob angled up and back; a portafilter with a black handle
+ locked into the group with two spouts pointing down at a cup; a steam wand of thin
+ pipe with a black tap knob on the side of the boiler; a pull-out drip tray with a
+ slotted chrome grille under the group; a sight glass tube on the boiler side showing
+ water level.
+ must_have:
+ - "a vertical cylindrical boiler about 0.16 m in diameter with a domed lid"
+ - "a finial on the top of the dome"
+ - "a round gauge with a pale dial set into the boiler front"
+ - "a group head on the boiler front near its base with a vertical spring housing above it"
+ - "a lever arm at least 0.25 m long with a black knob angled up and back from the group"
+ - "a portafilter locked into the group with two spouts pointing down"
+ - "a drip tray with a slotted grille under the group"
+ - "chrome boiler, brass group head and black knobs as three distinct materials"
+ dimensions_m: {width: 0.24, depth: 0.32, height: 0.55}
+ # Orbital shaker-incubator: a radiused housing with a hinged clear lid on gas struts, a
+ # perforated platform, six clamped flasks with liquid; lab equipment with transparency.
+ - id: fancy_bl_lab_orbital_shaker
+ tier: hard
+ category: lab_equipment
+ tags: [fancy_v1, hard]
+ prompt: >
+ a benchtop orbital shaker-incubator as used in a microbiology lab: a squat white
+ powder-coated steel housing about 0.60 m wide, 0.55 m deep and 0.45 m tall with
+ radiused vertical corners and a recessed front control panel carrying a small dark
+ display, a speed knob, a temperature knob and two rocker switches; a clear hinged
+ acrylic lid on the top, propped open at about 60 degrees on two gas struts, revealing
+ the platform inside: a flat aluminium tray on the orbital drive with a grid of holes
+ and two rows of spring-wire flask clamps, each clamp a ring of curved spring fingers,
+ holding six 250 ml conical Erlenmeyer flasks with orange screw caps and pale broth
+ inside; a stainless handle on the lid front edge; four levelling feet; a louvred
+ vent strip on each side of the housing; a coiled black mains lead at the back.
+ must_have:
+ - "a white boxy housing about 0.60 m wide with rounded vertical corners"
+ - "a transparent lid hinged at the back, propped open on two struts"
+ - "a flat platform tray inside with a visible grid of holes"
+ - "exactly 6 conical flasks standing on the platform in two rows"
+ - "each flask is held by a ring of spring fingers around its base"
+ - "each flask has an orange cap and a visible liquid level inside"
+ - "a front control panel with two knobs and a small dark display"
+ - "the housing stands on four feet on the ground plane"
+ dimensions_m: {width: 0.62, depth: 0.58, height: 0.9}
diff --git a/eval/llm/batteries/fancy_v1_cadquery.yaml b/eval/llm/batteries/fancy_v1_cadquery.yaml
new file mode 100644
index 00000000..5a3e10f5
--- /dev/null
+++ b/eval/llm/batteries/fancy_v1_cadquery.yaml
@@ -0,0 +1,97 @@
+# fancy_v1 — cadquery slice (3 prompts). Machined, turned and cast objects that suit a
+# B-rep kernel: fillets, bores, bolt circles, gear teeth, knurls and threads. Every prompt
+# names the real construction of the object; must_have items are single falsifiable
+# clauses with counts. All hard tier; ids fancy_cq_<snake>.
+# Sibling files: fancy_v1_blender.yaml, fancy_v1_threejs.yaml.
+name: fancy_v1_cadquery
+track: static_object
+language: cadquery
+description: >
+ fancy_v1 cadquery slice: 3 hard machined/cast objects (Jacobs chuck on an arbor, a
+ cast-iron handwheel gate valve, a turbocharger housing) with bores, bolt circles, knurls
+ and fillets that a CAD kernel can express exactly.
+prompts:
+ # Jacobs chuck: three jaws on a scroll in a tapered body, a knurled sleeve, a key with a
+ # pinion, the arbor behind — turned parts with a knurl and radial repeats at 120 degrees.
+ - id: fancy_cq_jacobs_chuck
+ tier: hard
+ category: tools
+ tags: [fancy_v1, hard]
+ prompt: >
+ a keyed Jacobs drill chuck on a Morse-taper arbor, as a machined assembly: a steel
+ chuck body about 0.045 m in diameter and 0.075 m long, turned with a shallow taper
+ toward the nose; over its middle a rotating knurled sleeve about 0.052 m in diameter
+ with straight knurling of at least 60 ribs and a chamfered front edge; at the nose,
+ three hardened jaws set at 120 degrees in angled slots, each jaw a rectangular bar
+ with a ground gripping face, protruding about 0.012 m and open around a 0.008 m gap
+ holding a twist drill bit about 0.09 m long with two helical flutes; on the rear
+ of the sleeve a ring of bevel gear teeth, at least 24 teeth, into which a chuck key
+ with a matching bevel pinion and a T-bar handle is engaged in one of three key holes
+ drilled into the body at 120 degrees; behind the body a tapered arbor about 0.10 m
+ long with a flat tang at its end; fillets at every shoulder.
+ must_have:
+ - "a turned chuck body with a knurled sleeve of at least 60 straight ribs around it"
+ - "exactly 3 jaws at 120 degree spacing at the nose, angled inward toward the axis"
+ - "a drill bit with two helical flutes gripped between the three jaws, coaxial with the body"
+ - "a ring of at least 24 bevel gear teeth on the rear face of the sleeve"
+ - "a chuck key with a bevel pinion engaged into a hole in the body, its T-bar handle outside"
+ - "a tapered arbor behind the body ending in a flat tang"
+ - "body, sleeve and arbor are all coaxial"
+ - "visible fillets at the shoulders between arbor and body"
+ dimensions_m: {width: 0.11, depth: 0.09, height: 0.27}
+ # Cast-iron gate valve: flanged body with bolt circles, a bonnet with its own bolt ring,
+ # a yoke, a rising stem with thread, a spoked handwheel; castings plus machined faces.
+ - id: fancy_cq_gate_valve
+ tier: hard
+ category: architecture
+ tags: [fancy_v1, hard]
+ prompt: >
+ a DN100 cast-iron rising-stem gate valve as installed in a pipe run: a bulbous cast
+ body with a horizontal bore, a raised-face flange at each end about 0.22 m in
+ diameter and 0.024 m thick with eight bolt holes on a 0.18 m pitch circle, each hole
+ fitted with a hex bolt and nut;
+ a bonnet flange on top about 0.16 m across with six bolts on its own circle, the
+ bonnet rising as a domed casting into a yoke of two curved arms that meet at a
+ threaded stem nut boss; a rising stem about 0.025 m in diameter passing up through
+ the yoke, its upper part visibly threaded with at least 10 turns showing above the
+ handwheel; a cast handwheel about 0.30 m in diameter with five curved spokes, a
+ dished rim of round section and a hub with a square drive and a retaining nut; a gland with two studs where the stem enters the bonnet; castings dark blue-grey,
+ flange faces bright, handwheel red.
+ must_have:
+ - "a horizontal through-bore body with a flange at each end"
+ - "exactly 8 bolt holes on each end flange, each fitted with a bolt and nut"
+ - "a bonnet flange on top with exactly 6 bolts on its own circle"
+ - "a two-armed yoke rising from the bonnet meeting at a central boss"
+ - "a vertical stem passing through the boss with at least 10 thread turns visible above the handwheel"
+ - "a handwheel about 0.30 m in diameter with exactly 5 spokes"
+ - "the handwheel red, the flange faces bright metal, the body a dark cast colour"
+ - "the body, bonnet, yoke, stem and handwheel share one vertical axis"
+ dimensions_m: {width: 0.35, depth: 0.32, height: 0.62}
+ # Turbocharger: two cast volutes, a machined centre bearing housing, a finned compressor
+ # wheel and turbine wheel visible in their inlets, V-band and bolt flanges, an actuator.
+ - id: fancy_cq_turbocharger
+ tier: hard
+ category: vehicles
+ tags: [fancy_v1, hard]
+ prompt: >
+ an automotive turbocharger as a machined and cast assembly: a snail-shaped aluminium
+ compressor volute about 0.16 m across whose scroll grows in section around the axis
+ to a tangential outlet spigot, and
+ an axial inlet bore about 0.055 m across in which the compressor wheel is visible: a hub with 12 thin curved blades; a
+ turbine housing on the other side, a cast-iron volute about 0.14 m across with a
+ square inlet flange carrying four bolt holes, and a round outlet with a V-band
+ flange; between the volutes a cylindrical centre bearing housing about 0.07 m in
+ diameter with a boss for the oil feed banjo bolt on top and a drain flange with two
+ studs underneath, joined to each housing by a ring of six clamp bolts; a wastegate
+ actuator can on a bracket on the compressor housing, its rod running to a lever on the
+ turbine housing; fillets at every casting junction.
+ must_have:
+ - "two snail-shaped volutes on a common axis with a cylindrical bearing housing between them"
+ - "the compressor volute has a tangential outlet spigot whose scroll grows in section around the axis"
+ - "a compressor wheel with at least 10 curved blades visible inside the axial inlet bore"
+ - "a square inlet flange on the turbine housing with exactly 4 bolt holes"
+ - "a V-band flange on the turbine outlet"
+ - "a ring of at least 6 bolts where each volute meets the centre housing"
+ - "a round actuator can on a bracket with a rod running to a lever on the turbine housing"
+ - "aluminium-coloured compressor housing distinct from the dark iron turbine housing"
+ dimensions_m: {width: 0.28, depth: 0.22, height: 0.2}
diff --git a/eval/llm/batteries/fancy_v1_threejs.yaml b/eval/llm/batteries/fancy_v1_threejs.yaml
new file mode 100644
index 00000000..227f3fdb
--- /dev/null
+++ b/eval/llm/batteries/fancy_v1_threejs.yaml
@@ -0,0 +1,99 @@
+# fancy_v1 — threejs slice (3 prompts). Objects that read through colour separation and
+# instancing of repeated parts (keys, blades, slats, tines): a good fit for procedural
+# three.js meshes with per-part materials. Every prompt names the real construction;
+# must_have items are single falsifiable clauses with counts. All hard tier; ids
+# fancy_tj_<snake>. Sibling files: fancy_v1_blender.yaml, fancy_v1_cadquery.yaml.
+name: fancy_v1_threejs
+track: static_object
+language: threejs
+description: >
+ fancy_v1 threejs slice: 3 hard static objects built from many repeated coloured parts
+ (a Wurlitzer-style jukebox, a Dutch windmill with lattice sails, a xylophone-style
+ marimba) that reward instancing and clean material separation.
+prompts:
+ # Jukebox: arched cabinet with bubble tubes, a chrome grille, a visible record carousel
+ # behind glass, a row of selector buttons — emissive colour bands plus repeated parts.
+ - id: fancy_tj_bubbler_jukebox
+ tier: hard
+ category: appliances
+ tags: [fancy_v1, hard]
+ prompt: >
+ a 1940s bubbler-style jukebox about 1.50 m tall: a wooden cabinet with a rounded arch
+ top, its front built as concentric arches of translucent coloured plastic panels
+ (yellow, red and green from the outside in) lit from behind, with two clear bubble
+ tubes following the outer arch; inside the arch a curved glass window showing the
+ record mechanism: a carousel of 24 vertical 78 rpm records on edge in a slotted
+ rack, with a chrome tonearm and a turntable in front; below the window a sloped
+ title strip holder with two columns of 12 title cards each behind a glass strip,
+ flanked by a row of 24 ivory selector buttons in two banks; the lower front a
+ chrome pierced speaker grille of at least eight vertical bars over a red cloth
+ backing, with two chrome coin-slot plates; four short cabinet feet; the cabinet
+ sides in dark walnut with a chrome trim strip up each front corner.
+ must_have:
+ - "a cabinet about 1.50 m tall with a rounded arched top"
+ - "at least 3 concentric coloured arch bands, distinct hues, on the front"
+ - "two bubble tubes following the outer arch on each side"
+ - "a curved window inside the arch through which at least 20 records are visible standing on edge"
+ - "two columns of title cards behind a glass strip below the window"
+ - "at least 20 selector buttons in two banks"
+ - "a speaker grille of at least 8 vertical chrome bars over a red backing on the lower front"
+ - "the coloured arch bands emissive so they read as lit"
+ dimensions_m: {width: 0.85, depth: 0.65, height: 1.52}
+ # Dutch smock windmill: an octagonal thatched body on a brick base, a rotating cap, four
+ # lattice sails with dozens of instanced slats, a tail pole and a stage gallery.
+ - id: fancy_tj_smock_windmill
+ tier: hard
+ category: architecture
+ tags: [fancy_v1, hard]
+ prompt: >
+ a Dutch smock windmill about 18 m tall: a tapered octagonal tower, the lower 5 m in
+ red brick with a door and two arched windows, the upper part thatched over a timber
+ frame with eight corner posts; a stage
+ gallery ringing the tower at the brick-to-thatch line, its deck on radial joists
+ with a railing of at least 40 balusters; a boat-shaped rotating cap on top, its
+ plank cladding laid horizontally, carrying the windshaft at a slight upward tilt; four
+ common sails fixed at 90 degrees on a cast-iron cross, each sail a long whip
+ about 12 m from the shaft with a lattice of at least 18 sail bars across it, the
+ lattice on one side of the whip only with a leading board on the other, the sails
+ set in the X position; a tail pole angling down from the back of the cap to a wheel
+ on the stage; sail lattices white,
+ the cap tarred black, the thatch grey-brown.
+ must_have:
+ - "an octagonal tapered tower that is brick below and thatch above"
+ - "a gallery deck ringing the tower at the material change with at least 40 balusters"
+ - "a rotating cap on top with horizontal plank cladding"
+ - "exactly 4 sails on a central cross set at 90 degree spacing"
+ - "each sail about 12 m from the shaft with at least 18 lattice bars"
+ - "a tail pole from the back of the cap running down to the gallery"
+ - "white sails, black cap and brown thatch as three distinct colours"
+ - "the tower base sits on the ground plane and no sail tip reaches the ground"
+ dimensions_m: {width: 26.0, depth: 26.0, height: 18.5}
+ # Concert marimba: 49 graduated rosewood bars in two rows on a frame, a resonator tube
+ # under each bar graduated in length, a wheeled frame; instancing with per-bar scaling.
+ - id: fancy_tj_concert_marimba
+ tier: hard
+ category: instruments
+ tags: [fancy_v1, hard]
+ prompt: >
+ a concert marimba with a 4-octave range: 49 rosewood bars laid out like a piano
+ keyboard on a wooden frame about 2.10 m long and 0.92 m high, the naturals in a
+ front row of 29 bars and the accidentals in a raised rear row of 20 bars offset
+ behind the gaps, the accidental row set back and about 0.03 m higher; bars
+ graduated in both length and width from about 0.45 m long and 0.075 m wide at the
+ bass end to 0.15 m long and 0.04 m wide at the treble, each bar resting on a cord through
+ nodal holes strung between posts on the rails; beneath every bar a vertical golden-coloured
+ aluminium resonator tube whose length matches the bar's pitch, longest under the
+ bass bars and stepping down toward the treble, so the resonator array forms a curved
+ wedge; the bass end of the frame taller than the treble end;
+ four legs on rubber-tyred casters; two mallets with yarn heads resting across the
+ bass bars.
+ must_have:
+ - "a front row of at least 25 bars and a raised rear row of at least 18 bars"
+ - "the rear row is offset behind the gaps of the front row and sits higher"
+ - "bars graduate in length from long at one end to short at the other"
+ - "a vertical resonator tube under every bar"
+ - "resonator tube lengths step down from the bass end to the treble end"
+ - "bars are a rosewood colour and resonators a golden metal colour"
+ - "four legs on casters, all touching the ground plane"
+ - "overall length about 2.10 m"
+ dimensions_m: {width: 2.15, depth: 0.85, height: 0.95}
diff --git a/eval/llm/batteries/graphics_v2.yaml b/eval/llm/batteries/graphics_v2.yaml
new file mode 100644
index 00000000..4d8bcceb
--- /dev/null
+++ b/eval/llm/batteries/graphics_v2.yaml
@@ -0,0 +1,170 @@
+# graphics_v2 — shader / OpenGL battery.
+#
+# Difficulty axes:
+# raymarching depth full raymarched scenes with material response (SSS-ish glow,
+# fractal surface detail, correct-looking gear meshing) — v1's
+# only raymarch prompt scored 0.30, the strongest discriminator
+# feedback state multi-pass programs whose look emerges from a feedback buffer
+# (reaction-diffusion, ink advection) — cannot be faked in one pass
+# stylisation control hitting a named art style with countable features (ukiyo-e wave,
+# CRT boot) instead of generic noise beauty
+# instancing scale opengl_python prompts with explicit instance counts and a
+# structured motion the VLM can verify across frames
+# Per-prompt `language: opengl_python` marks the raw-OpenGL prompts (battery default
+# is glsl_shader); the bench loader honours the per-prompt override (BenchPrompt.language).
+# Expected discrimination: medium 40-70% pass; hard 0-30% pass — feedback-buffer and
+# instancing prompts also stress the build/runtime, where v1's opengl prompt errored.
+name: graphics_v2
+track: graphics
+language: glsl_shader
+description: >
+ 10 graphics prompts: 3 medium stylised shaders, 7 hard (raymarched scenes,
+ feedback-buffer simulations, large-scale instancing). must_have items are
+ frame-verifiable features with counts and motion statements.
+prompts:
+ # ------------------------------------------------------------- raymarched (hard)
+ - id: glsl_hard_mushroom_forest
+ tier: hard
+ category: raymarch
+ prompt: >
+ a raymarched night mushroom forest: giant glowing mushrooms whose caps
+ transmit light like wax, thin stems, drifting spore motes, soft shadows and
+ a slow camera glide between the stems
+ must_have:
+ - "at least 5 mushrooms with domed caps on visible stems"
+ - "caps glow from inside with a soft falloff toward the rim, brighter where thin"
+ - "ground plane receiving soft shadows from the stems"
+ - "drifting bright spore motes between the mushrooms"
+ - "dark blue or violet night ambience with the caps as main light sources"
+ - "camera glides forward between the stems over the loop"
+ - id: glsl_hard_infinite_corridor
+ tier: hard
+ category: raymarch
+ prompt: >
+ a raymarched infinite corridor: repeating arched segments receding to a
+ vanishing point, walls carved with self-similar fractal relief that gains
+ detail as the camera approaches, cool fog in the distance, endless forward
+ motion
+ must_have:
+ - "corridor of repeating arched segments receding to a vanishing point"
+ - "at least 6 segment repetitions visible at once"
+ - "wall surfaces show self-similar relief with at least 3 visible scales of detail"
+ - "distance fog fades the far segments"
+ - "forward motion continues across all sampled frames without the camera clipping the walls"
+ - "light sources along the corridor alternate or repeat with the segments"
+ - id: glsl_hard_clockwork_gears
+ tier: hard
+ category: raymarch
+ prompt: >
+ a raymarched brass clockwork: at least three meshing gears with visible
+ teeth turning at consistent ratios, a slowly swinging pendulum disc behind
+ them, metallic highlights and a dark workshop background
+ must_have:
+ - "at least 3 gears with clearly separated teeth"
+ - "adjacent gears rotate in opposite directions"
+ - "a small gear visibly turns faster than a large one it meshes with"
+ - "teeth of meshing gears interleave without overlapping"
+ - "pendulum disc swings periodically behind the gears"
+ - "metallic specular highlights on the gear faces"
+ # ---------------------------------------------------- feedback simulations (hard)
+ - id: ogl_hard_reaction_diffusion
+ tier: hard
+ category: simulation
+ language: opengl_python
+ prompt: >
+ a reaction-diffusion coral garden grown in a feedback buffer: patterns seeded
+ at several points spread outward into labyrinthine coral growth, shaded with
+ height-like lighting and an ocean palette, growth visible over the run
+ must_have:
+ - "labyrinthine or spotted pattern characteristic of reaction-diffusion"
+ - "pattern grows outward from at least 3 seed points over time"
+ - "later frames show strictly more covered area than early frames"
+ - "pattern edges shaded with a relief or rim-light effect"
+ - "ocean palette: teal to orange or pink gradient mapped to the pattern"
+ - "the pattern boundary advances in every successive sampled frame"
+ - id: ogl_hard_ink_swirls
+ tier: hard
+ category: simulation
+ language: opengl_python
+ prompt: >
+ fluid ink in water via a feedback advection buffer: dark ink blobs injected
+ periodically curl into filament swirls that stretch and fold without
+ dissipating instantly, on a paper-white background
+ must_have:
+ - "ink blobs appear at intervals during the run"
+ - "blobs stretch into curling filaments over subsequent frames"
+ - "filaments fold back on themselves forming spiral structures"
+ - "earlier ink stays visible as faded wisps while new ink is injected"
+ - "paper-white background with dark ink, no full-frame darkening over time"
+ # ------------------------------------------------------------- stylised (medium)
+ - id: glsl_med_ukiyoe_wave
+ tier: medium
+ category: stylised
+ prompt: >
+ a looping ukiyo-e style great wave: a large curling crest with claw-like foam
+ fingers, flat indigo color bands with paper-cream foam, smaller waves in
+ front, and a seamless loop where the crest rises, curls and collapses
+ must_have:
+ - "one dominant wave with a forward-curling crest"
+ - "foam edge broken into claw-like fingers"
+ - "flat color bands in indigo and cream, no photoreal gradients"
+ - "at least 2 smaller waves in the foreground"
+ - "across the sampled frames the crest rises, curls forward and collapses"
+ - id: glsl_med_crt_boot
+ tier: medium
+ category: stylised
+ prompt: >
+ a retro CRT boot sequence: black screen flickers on with a horizontal flash,
+ green blocky glyph rows type in line by line, a blinking cursor, scanlines,
+ barrel distortion at the screen corners and phosphor glow
+ must_have:
+ - "earliest sampled frame shows the turn-on state: mostly dark or a bright horizontal flash band"
+ - "rows of blocky glyphs appear line by line down the screen"
+ - "blinking rectangular cursor at the end of the last row"
+ - "visible scanlines across the whole frame"
+ - "image bulges with barrel distortion, corners darkened"
+ - "green phosphor glow bleeding around bright glyphs"
+ - id: glsl_med_kaleido_mandala
+ tier: medium
+ category: stylised
+ prompt: >
+ a kaleidoscopic mandala: 12-fold mirror symmetry, at least three concentric
+ pattern rings rotating at different speeds in alternating directions, petal
+ shapes morphing smoothly, jewel palette on a dark background
+ must_have:
+ - "12-fold mirror symmetry about the center"
+ - "at least 3 concentric pattern rings"
+ - "adjacent rings rotate in opposite directions at different speeds"
+ - "petal or lobe shapes morph smoothly over time"
+ - "saturated jewel palette on a dark background, center stays centered"
+ # ------------------------------------------------------ instancing scale (hard)
+ - id: ogl_hard_boids_murmuration
+ tier: hard
+ category: instancing
+ language: opengl_python
+ prompt: >
+ a starling murmuration of 30000 instanced boids over a dusk landscape
+ silhouette: the flock stretches, folds and splits into lobes that merge
+ again, dense cores darker than the edges, slow camera pan
+ must_have:
+ - "flock of thousands of small instances, far more than any countable number"
+ - "flock forms one connected shape that stretches and folds over time"
+ - "flock splits into at least 2 lobes and merges again during the run"
+ - "denser flock regions read darker or thicker than sparse edges"
+ - "dusk gradient sky with a dark landscape silhouette at the bottom"
+ - "camera pans slowly while the flock stays in frame"
+ - id: ogl_hard_voxel_city
+ tier: hard
+ category: instancing
+ language: opengl_python
+ prompt: >
+ a fly-through of an instanced voxel city with a full day-night cycle:
+ cube-built towers of varying heights on a street grid, sun light and long
+ shadows by day, lit windows by night, the camera weaving between towers
+ must_have:
+ - "city of cube-built towers on a regular street grid"
+ - "at least 50 towers of clearly varying heights"
+ - "continuous camera flight between the towers without entering any tower"
+ - "day phase: directional sun with visible long shadows"
+ - "night phase: dark sky with emissive lit windows on the towers"
+ - "smooth day-to-night transition within the run"
diff --git a/eval/llm/batteries/static_objects_v1.yaml b/eval/llm/batteries/static_objects_v1.yaml
new file mode 100644
index 00000000..cb9705a7
--- /dev/null
+++ b/eval/llm/batteries/static_objects_v1.yaml
@@ -0,0 +1,137 @@
+name: static_objects_v1
+track: static_object
+language: blender
+description: >
+ 24 static-object prompts, 8 categories × 3 tiers (easy / medium / hard).
+ Every prompt carries must_have items the planner turns into acceptance checks.
+ Same battery runs under any generator; keep the judge model fixed across arms.
+prompts:
+ # ---------------------------------------------------------------- furniture
+ - id: furn_easy_stool
+ tier: easy
+ category: furniture
+ prompt: a simple three-legged round wooden stool, 45 cm tall
+ must_have: [round seat, three splayed legs of equal length, legs touch the ground]
+ dimensions_m: {height: 0.45}
+ - id: furn_med_dining_chair
+ tier: medium
+ category: furniture
+ prompt: a mid-century wooden dining chair with a slatted backrest and tapered legs
+ must_have: [four tapered legs, seat slab, backrest with at least three vertical slats, back legs continue into back posts]
+ - id: furn_hard_rolltop_desk
+ tier: hard
+ category: furniture
+ prompt: a writing desk with two pedestals of three drawers each, a central knee-hole, a leather inlay top and brass drawer pulls
+ must_have: [two drawer pedestals, three drawers per pedestal with visible fronts, knee-hole gap, brass pull handles, desk top overhangs]
+ # ---------------------------------------------------------------- vehicles
+ - id: veh_easy_toy_car
+ tier: easy
+ category: vehicles
+ prompt: a chunky toy car with a rounded body and four thick wheels
+ must_have: [one-piece rounded body, four wheels touching the ground, wheels protrude sideways]
+ - id: veh_med_pickup
+ tier: medium
+ category: vehicles
+ prompt: a pickup truck with a two-door cab, open cargo bed, wheel arches and side mirrors
+ must_have: [cab with windows, open rear bed, four wheels with hubs, front grille and headlights, side mirrors]
+ - id: veh_hard_tractor
+ tier: hard
+ category: vehicles
+ prompt: a farm tractor with large rear tyres with treads, small front wheels, an exposed engine block, exhaust stack, driver seat, steering wheel and a roll bar
+ must_have: [rear wheels much larger than front wheels, tyre tread detail, exhaust stack, seat and steering wheel, roll bar, engine hood]
+ # ---------------------------------------------------------------- tools
+ - id: tool_easy_hammer
+ tier: easy
+ category: tools
+ prompt: a claw hammer with a wooden handle and a steel head
+ must_have: [cylindrical handle, steel head with a flat face, forked claw on the back]
+ - id: tool_med_hand_drill
+ tier: medium
+ category: tools
+ prompt: a cordless power drill with a pistol grip, trigger, battery pack in the base of the grip and a keyless chuck holding a drill bit
+ must_have: [pistol grip body, trigger, battery block under grip, chuck with a bit, side vents]
+ - id: tool_hard_bench_vise
+ tier: hard
+ category: tools
+ prompt: a cast-iron bench vise with fixed and sliding jaws, a lead screw with a tommy bar handle, guide rails, a small anvil face and mounting base with bolt holes
+ must_have: [two jaws with a visible gap, sliding jaw on guide rails, lead screw with sliding T handle, anvil flat, base with bolt holes]
+ # ---------------------------------------------------------------- appliances
+ - id: appl_easy_kettle
+ tier: easy
+ category: appliances
+ prompt: an electric kettle with a spout, a lid and a side handle on a round base
+ must_have: [pot body, pouring spout, handle opposite the spout, lid on top, round base]
+ - id: appl_med_toaster
+ tier: medium
+ category: appliances
+ prompt: a two-slot retro toaster with rounded chrome body, a lever on the side, a browning dial and a crumb tray
+ must_have: [two bread slots on top, side lever, dial knob, rounded body, small feet]
+ - id: appl_hard_espresso
+ tier: hard
+ category: appliances
+ prompt: an espresso machine with a group head and portafilter, a steam wand, a drip tray with grate, a water tank at the back, and a top-mounted cup warmer with a pressure gauge on the front panel
+ must_have: [portafilter with handle under a group head, steam wand on the side, drip tray with grate, water tank, pressure gauge, cup warmer rail]
+ # ---------------------------------------------------------------- plants
+ - id: plant_easy_cactus
+ tier: easy
+ category: plants
+ prompt: a potted barrel cactus in a terracotta pot with soil
+ must_have: [ribbed round cactus body, terracotta pot with rim, soil surface]
+ - id: plant_med_palm
+ tier: medium
+ category: plants
+ prompt: a small potted palm with a textured trunk and eight arching fronds made of many leaflets
+ must_have: [textured trunk, at least eight fronds, fronds arch downward, leaflets along each frond, pot]
+ - id: plant_hard_bonsai
+ tier: hard
+ category: plants
+ prompt: a bonsai tree with a twisted thick trunk, three layered foliage pads, exposed roots over a rock and a shallow oval ceramic tray with moss
+ must_have: [twisted trunk with taper, at least three distinct foliage pads, exposed roots gripping a rock, shallow oval tray, moss surface]
+ # ---------------------------------------------------------------- buildings
+ - id: bld_easy_shed
+ tier: easy
+ category: buildings
+ prompt: a small garden shed with a pitched roof, one door and one window
+ must_have: [box walls, pitched roof overhang, door on front, window]
+ - id: bld_med_lighthouse
+ tier: medium
+ category: buildings
+ prompt: a tapered lighthouse with red and white bands, a gallery with railing, a glass lantern room and a domed cap, on a rocky base
+ must_have: [tapered tower, alternating bands, gallery walkway with railing, glass lantern room, dome, rock base]
+ - id: bld_hard_windmill
+ tier: hard
+ category: buildings
+ prompt: a Dutch windmill with an octagonal tapered body, thatched cap, four lattice sails, a balcony stage with railing, a door, and two shuttered windows
+ must_have: [octagonal tapered body, cap, four lattice sails in a cross, balcony with railing, door, shuttered windows]
+ # ---------------------------------------------------------------- toys
+ - id: toy_easy_blocks
+ tier: easy
+ category: toys
+ prompt: a stack of five alphabet blocks of decreasing size in a slightly twisted tower
+ must_have: [five cubes, decreasing size upward, each slightly rotated, all stacked touching]
+ - id: toy_med_rocking_horse
+ tier: medium
+ category: toys
+ prompt: a wooden rocking horse with curved rockers, a saddle, handles and a mane
+ must_have: [horse body with head and four legs, two curved rockers, saddle, handle grips, mane]
+ - id: toy_hard_train
+ tier: hard
+ category: toys
+ prompt: a toy steam locomotive with a boiler, smokestack, cab with windows, cowcatcher, six driving wheels with connecting rods, and a coal tender with four wheels
+ must_have: [cylindrical boiler, smokestack, cab, cowcatcher, six large wheels with rods, separate tender with coal load and four wheels]
+ # ---------------------------------------------------------------- musical instruments
+ - id: mus_easy_drum
+ tier: easy
+ category: musical_instruments
+ prompt: a snare drum with tension rods and two drumsticks resting on top
+ must_have: [cylindrical shell, top head, at least eight tension rods around the rim, two sticks]
+ - id: mus_med_acoustic_guitar
+ tier: medium
+ category: musical_instruments
+ prompt: an acoustic guitar with a figure-eight body, sound hole, bridge, fretted neck, headstock with six tuning pegs and six strings
+ must_have: [figure-eight body, sound hole, bridge, neck with frets, headstock, six tuning pegs, six strings]
+ - id: mus_hard_drum_kit
+ tier: hard
+ category: musical_instruments
+ prompt: a five-piece drum kit with a bass drum with pedal, snare on a stand, two rack toms, a floor tom, a hi-hat and two cymbals on stands, and a throne
+ must_have: [bass drum with pedal, snare on stand, two mounted toms, floor tom with legs, hi-hat with two cymbals, two crash/ride cymbals on stands, drum throne]
diff --git a/eval/llm/batteries/static_objects_v2.yaml b/eval/llm/batteries/static_objects_v2.yaml
new file mode 100644
index 00000000..bf9f339d
--- /dev/null
+++ b/eval/llm/batteries/static_objects_v2.yaml
@@ -0,0 +1,425 @@
+# static_objects_v2 — hard-tier battery (v1 easy saturates: 100% pass, 0.85-0.96 with 3.7-flash).
+#
+# Difficulty axes (named per prompt via tags):
+# axis:decomposition 10+ distinct nameable parts that must fit together (v1 hard failure
+# modes were floating parts / interpenetration / stray islands — these
+# prompts maximise part-to-part contact surface count)
+# axis:thin_features wires, spokes, scrolls, strings at 3-30 mm thickness
+# axis:curved_organic compound-curved / organic surfaces (no box-stack shortcut)
+# axis:arch_detail repeated countable architectural elements (dentils, rafters, panes)
+# axis:multi_material compositions of 2+ objects in distinct materials that must relate
+# Controls: 4 medium prompts whose PROMPT TEXT is copied verbatim from v1 medium
+# (furniture / vehicles / appliances / tools — 2/4 passed in static_v1_flash, the
+# discriminating band of that run) so
+# generation difficulty is comparable across versions; their must_have lists are extended
+# to the v2 checklist standard (8+ items, half with dims), so judge scores are v2-internal.
+# Expected discrimination: controls ~30-60% pass; hard tier 0-30% pass with score spread
+# 0.4-0.7 driven by assembly_fit and geometry_detail, not intent_fidelity.
+name: static_objects_v2
+track: static_object
+language: blender
+description: >
+ 20 static-object prompts: 4 medium controls (prompt text shared with v1 medium for
+ cross-version comparison) and 16 hard prompts across five named difficulty axes
+ (decomposition depth, thin features, curved/organic surfaces, architectural detail,
+ multi-material composition). Every prompt carries 8-16 nameable parts in must_have,
+ at least half with measurable dimensions.
+prompts:
+ # ------------------------------------------------------------- controls (v1 medium text)
+ - id: ctrl_med_dining_chair
+ tier: medium
+ category: furniture
+ tags: [control, "axis:decomposition"]
+ prompt: a mid-century wooden dining chair with a slatted backrest and tapered legs
+ must_have:
+ - "four tapered legs, each thinner at the floor than at the seat"
+ - "seat slab about 0.42 m wide"
+ - "seat surface about 0.45 m above the floor"
+ - "backrest with at least 3 vertical slats"
+ - "back legs continue up into the backrest posts"
+ - "backrest top rail between 0.80 m and 0.95 m above the floor"
+ - "all four feet touch the ground plane"
+ - "no leg or slat intersects the seat slab"
+ dimensions_m: {width: 0.45, depth: 0.50, height: 0.90}
+ - id: ctrl_med_pickup
+ tier: medium
+ category: vehicles
+ tags: [control, "axis:decomposition"]
+ prompt: a pickup truck with a two-door cab, open cargo bed, wheel arches and side mirrors
+ must_have:
+ - "cab with windshield and 2 side windows"
+ - "open rear bed with 3 walls and a tailgate"
+ - "4 wheels with hubs, all touching the ground"
+ - "wheel diameter about 0.75 m"
+ - "front grille and 2 headlights"
+ - "2 side mirrors on stalks"
+ - "bed floor about 0.80 m above the ground"
+ - "overall length about 5.2 m"
+ dimensions_m: {length: 5.2, width: 1.9, height: 1.8}
+ - id: ctrl_med_toaster
+ tier: medium
+ category: appliances
+ tags: [control, "axis:curved_organic"]
+ prompt: a two-slot retro toaster with rounded chrome body, a lever on the side, a browning dial and a crumb tray
+ must_have:
+ - "2 bread slots on the top face"
+ - "each slot about 0.13 m long and under 0.04 m wide"
+ - "lever on one end face"
+ - "round dial knob on the front face"
+ - "rounded body with no sharp vertical edges"
+ - "crumb tray seam line near the base"
+ - "4 small feet"
+ - "body about 0.28 m wide and 0.19 m tall"
+ dimensions_m: {width: 0.28, depth: 0.17, height: 0.19}
+ - id: ctrl_med_hand_drill
+ tier: medium
+ category: tools
+ tags: [control, "axis:decomposition"]
+ prompt: a cordless power drill with a pistol grip, trigger, battery pack in the base of the grip and a keyless chuck holding a drill bit
+ must_have:
+ - "pistol-grip body with the grip angled back from vertical"
+ - "trigger recessed at the top of the grip"
+ - "battery block under the grip, wider than the grip"
+ - "chuck at the front, about 0.04 m diameter"
+ - "drill bit protruding about 0.08 m from the chuck"
+ - "side vents on the motor housing"
+ - "overall length about 0.22 m from chuck tip to body rear"
+ - "overall height about 0.24 m from battery base to body top"
+ dimensions_m: {length: 0.22, height: 0.24}
+ # ------------------------------------------------------- mechanical assemblies (hard)
+ - id: mech_hard_coffee_grinder
+ tier: hard
+ category: mechanisms
+ tags: ["axis:decomposition"]
+ prompt: >
+ a hand-crank coffee grinder: square wooden box body with a small front drawer,
+ cast-iron dome top with an open hopper funnel, a conical burr visible inside
+ the funnel throat, and a top crank arm with a wooden knob
+ must_have:
+ - "square wooden box body about 0.14 m wide and 0.12 m tall"
+ - "front drawer about 0.05 m tall with a round knob, in the lower half of the box"
+ - "drawer front sits flush in its opening with a visible seam"
+ - "metal dome top mounted on the box"
+ - "open hopper funnel about 0.08 m across at the rim"
+ - "conical burr visible inside the funnel throat"
+ - "central spindle about 0.01 m thick rising from the burr through the dome"
+ - "crank arm about 0.11 m long attached to the spindle top"
+ - "wooden knob on a short vertical pin at the crank's free end"
+ - "adjustment nut on the spindle above the crank hub"
+ dimensions_m: {width: 0.14, depth: 0.14, height: 0.30}
+ - id: mech_hard_machinist_vise
+ tier: hard
+ category: tools
+ tags: ["axis:decomposition", "axis:thin_features"]
+ prompt: >
+ a machinist bench vise with an acme lead screw showing its thread helix, twin
+ cylindrical guide rails, a swivel base with two lock handles, serrated jaw
+ plates and a sliding tommy-bar handle
+ must_have:
+ - "fixed jaw and sliding jaw with a 0.05 m open gap between them"
+ - "flat jaw plates about 0.10 m wide with serration grooves"
+ - "lead screw between the jaws with at least 6 visible thread turns"
+ - "2 cylindrical guide rails flanking the screw"
+ - "sliding jaw rides on the rails and the screw"
+ - "tommy bar about 0.16 m long through the screw head, with ball ends"
+ - "swivel base disc under the vise body"
+ - "2 lock handles on the swivel base rim"
+ - "mounting flange with 3 bolt holes"
+ - "overall body length about 0.30 m"
+ dimensions_m: {length: 0.30, width: 0.14, height: 0.18}
+ - id: mech_hard_drill_press
+ tier: hard
+ category: tools
+ tags: ["axis:decomposition"]
+ prompt: >
+ a benchtop drill press: cast base, vertical column, adjustable work table with
+ a crank, head casting with a belt housing, chuck with a bit over the table,
+ three-spoke quill feed lever and a depth-stop rod
+ must_have:
+ - "flat base about 0.30 m by 0.20 m with 2 mounting slots"
+ - "vertical column about 0.05 m diameter rising from the base"
+ - "work table clamped to the column between base and head"
+ - "table raise crank handle below the table"
+ - "head casting at the top of the column"
+ - "belt housing on top of the head with a lid seam"
+ - "cylindrical motor behind the column, connected to the head"
+ - "chuck about 0.04 m diameter under the head with a drill bit pointing down"
+ - "bit tip aligned over the table center"
+ - "quill feed lever with 3 spokes ending in ball knobs"
+ - "depth-stop rod with 2 nuts beside the quill"
+ - "overall height about 0.85 m"
+ dimensions_m: {height: 0.85, depth: 0.45}
+ - id: mech_hard_pitcher_pump
+ tier: hard
+ category: garden
+ tags: ["axis:decomposition", "axis:curved_organic"]
+ prompt: >
+ a cast-iron pitcher water pump on a wooden platform: fluted cylindrical body,
+ curved downward spout, long lever handle raised mid-stroke, connecting link to
+ the piston rod, domed cap and a bolted base flange
+ must_have:
+ - "cylindrical pump body about 0.09 m diameter"
+ - "at least 6 vertical flutes on the body"
+ - "base flange with 4 bolt heads"
+ - "square wooden platform under the flange"
+ - "curved spout leaving the body two-thirds up, opening facing down"
+ - "domed cap on top of the body"
+ - "lever handle about 0.40 m long, raised at an angle"
+ - "handle pivots in a clevis bracket at the top rear of the body"
+ - "connecting link from the handle to a piston rod entering the cap"
+ - "overall height about 0.65 m"
+ dimensions_m: {height: 0.65}
+ # ------------------------------------------------------- thin / curved structures (hard)
+ - id: thin_hard_garden_arch
+ tier: hard
+ category: garden
+ tags: ["axis:thin_features"]
+ prompt: >
+ a wrought-iron garden arch: two lattice side frames, a semicircular arch head
+ filled with S-scrolls, spear-point finials and a tie bar at the spring line
+ must_have:
+ - "2 side frames, each of 2 uprights about 0.01 m thick"
+ - "diamond lattice bars between the uprights of each side frame"
+ - "semicircular arch head spanning about 1.2 m between the side frames"
+ - "at least 6 S-scrolls filling the arch head"
+ - "at least 4 scrolls in each side frame"
+ - "3 spear-point finials on top of the arch"
+ - "horizontal tie bar connecting both sides at the spring line"
+ - "small ground plates under all 4 upright feet"
+ - "overall height about 2.3 m"
+ - "all bars slender: no member thicker than 0.02 m"
+ dimensions_m: {width: 1.4, depth: 0.5, height: 2.3}
+ - id: thin_hard_violin
+ tier: hard
+ category: musical_instruments
+ tags: ["axis:thin_features", "axis:curved_organic"]
+ prompt: >
+ a violin: hourglass body with C-bout waists, arched top with two f-holes, a
+ bridge, four strings from tailpiece over the bridge along the fingerboard to
+ the pegbox, a scroll head, four tuning pegs and a chin rest
+ must_have:
+ - "hourglass body with 2 concave C-bout waist cutaways"
+ - "body length about 0.36 m"
+ - "top plate visibly arched, not flat"
+ - "2 f-holes through the top, one each side of the bridge"
+ - "bridge about 0.03 m tall standing upright between the f-holes"
+ - "4 strings running in parallel from tailpiece to pegbox"
+ - "strings pass over the bridge with a visible break angle"
+ - "black fingerboard extending over the body without touching it"
+ - "spiral scroll at the head"
+ - "4 tuning pegs, 2 per side of the pegbox"
+ - "tailpiece anchored toward the end button"
+ - "chin rest beside the tailpiece"
+ - "total length about 0.60 m"
+ dimensions_m: {length: 0.60, width: 0.21, height: 0.09}
+ - id: thin_hard_penny_farthing
+ tier: hard
+ category: vehicles
+ tags: ["axis:thin_features", "axis:curved_organic"]
+ prompt: >
+ a penny-farthing bicycle: huge spoked front wheel with pedal cranks on its hub,
+ small spoked rear wheel, curved backbone tube, front fork, handlebar and a
+ sprung leather saddle
+ must_have:
+ - "front wheel about 1.3 m diameter with at least 24 thin spokes"
+ - "rear wheel about 0.40 m diameter with at least 12 spokes"
+ - "tyres no thicker than 0.03 m"
+ - "backbone tube arcing from the fork crown down to the rear wheel fork"
+ - "front fork straddling the big wheel to its hub"
+ - "handlebar with 2 grips above the front hub"
+ - "2 pedal cranks on the front hub with pedals, opposed at 180 degrees"
+ - "saddle on the backbone behind the handlebar, on a visible spring"
+ - "small mounting step peg on the backbone above the rear wheel"
+ - "both wheels touch the ground plane"
+ dimensions_m: {length: 1.7, height: 1.5}
+ - id: thin_hard_birdcage
+ tier: hard
+ category: household
+ tags: ["axis:thin_features"]
+ prompt: >
+ a round wire birdcage with an onion-dome top, a hanging ring, a wire door with
+ a latch, a trapeze perch hanging inside, a straight perch, two feed cups and a
+ base tray with a rolled rim
+ must_have:
+ - "round cage about 0.35 m diameter"
+ - "at least 20 vertical wires around the circumference"
+ - "wires about 0.003 m thick"
+ - "3 horizontal rings tying the wires"
+ - "onion-dome top converging to a finial"
+ - "hanging ring at the very top"
+ - "rectangular door frame set into the wires with a small latch"
+ - "trapeze perch hanging inside from the dome on 2 wires"
+ - "straight perch spanning the cage between opposite walls"
+ - "2 feed cups mounted at a ring"
+ - "solid base tray with a rolled rim"
+ - "overall height about 0.55 m"
+ dimensions_m: {height: 0.55}
+ # ------------------------------------------------------------------ organic (hard)
+ - id: org_hard_bonsai_rock
+ tier: hard
+ category: plants
+ tags: ["axis:curved_organic"]
+ prompt: >
+ a root-over-rock bonsai: a shallow oval pot with soil, a rounded rock gripped
+ by exposed roots, a tapering trunk with two pronounced curves, and five
+ distinct flattened foliage pads
+ must_have:
+ - "shallow oval pot about 0.30 m long with a rim and small feet"
+ - "soil surface inside the pot"
+ - "rounded rock about 0.12 m tall sitting on the soil"
+ - "at least 4 roots gripping over the rock faces and entering the soil"
+ - "trunk rising from the rock top with at least 2 pronounced curves"
+ - "trunk tapers from about 0.04 m at the base to under 0.015 m at the apex"
+ - "5 distinct flattened foliage pads with gaps between them"
+ - "lowest branch extends beyond the pot rim"
+ - "no foliage pad touches the soil or the rock"
+ - "overall height about 0.50 m"
+ dimensions_m: {height: 0.50, length: 0.45}
+ - id: org_hard_seated_cat
+ tier: hard
+ category: animals
+ tags: ["axis:curved_organic"]
+ prompt: >
+ a domestic cat sitting upright: haunches on the ground, straight vertical
+ front legs, tail curled around beside the front paws, short muzzle and
+ triangular ears
+ must_have:
+ - "upright seated pose with haunches on the ground and chest raised"
+ - "2 straight front legs, nearly vertical"
+ - "front paws on the ground between the haunches"
+ - "hind legs folded with thighs bulging wider than the chest"
+ - "tail about 0.30 m long curling around to lie beside the front paws"
+ - "head with a short muzzle, 2 eyes and a nose"
+ - "2 triangular ears about 0.04 m tall"
+ - "back forms a smooth continuous curve from neck to tail base"
+ - "silhouette narrows from haunches up to the shoulders"
+ - "overall height about 0.35 m"
+ dimensions_m: {height: 0.35, depth: 0.25}
+ - id: org_hard_octopus_rock
+ tier: hard
+ category: animals
+ tags: ["axis:curved_organic", "axis:thin_features"]
+ prompt: >
+ an octopus draped over a low rock: bulbous mantle, raised eyes, eight
+ tapering arms curling in different directions, with sucker discs visible
+ along the underside of at least three arms
+ must_have:
+ - "bulbous mantle about 0.20 m long"
+ - "2 raised eye bumps on the head"
+ - "8 arms radiating from the head"
+ - "each arm tapers from about 0.05 m at the base to a point"
+ - "arm tips curl, at least 4 curling in different directions"
+ - "sucker discs along the underside of at least 3 arms"
+ - "low rock under the body, arms draping over its sides"
+ - "at least 2 arms touch the ground plane"
+ - "overall arm spread about 0.8 m"
+ dimensions_m: {length: 0.8, height: 0.3}
+ # ------------------------------------------------------- architectural detail (hard)
+ - id: arch_hard_bay_window
+ tier: hard
+ category: buildings
+ tags: ["axis:arch_detail", "axis:decomposition"]
+ prompt: >
+ a Victorian bay window facade section: a three-sided bay projecting from a
+ clapboard wall, a sash window with divided panes in each face, panel aprons,
+ corner pilasters, scroll brackets and a dentil cornice under a faceted roof cap
+ must_have:
+ - "three-sided bay projecting from a flat wall section"
+ - "center face wider than the 2 angled side faces"
+ - "side faces angled about 45 degrees from the wall"
+ - "sash window in each of the 3 faces"
+ - "upper sash of each window divided into at least 4 panes by muntins"
+ - "continuous sill band under all 3 windows"
+ - "recessed panel apron under each window"
+ - "pilasters at the 4 vertical bay corners"
+ - "cornice with at least 10 dentil blocks across the bay"
+ - "at least 4 scroll brackets under the bay roof"
+ - "three-faceted roof cap over the bay"
+ - "horizontal siding boards on the flat wall section"
+ dimensions_m: {width: 3.4, height: 3.2, depth: 1.0}
+ - id: arch_hard_spiral_stair
+ tier: hard
+ category: buildings
+ tags: ["axis:arch_detail", "axis:thin_features"]
+ prompt: >
+ a cast-iron spiral staircase: central column, wedge treads winding a full
+ turn, a baluster at each tread's outer edge, a continuous helical handrail
+ and a railed top landing
+ must_have:
+ - "central column about 0.10 m diameter"
+ - "at least 12 wedge-shaped treads attached to the column"
+ - "treads climb evenly through at least 360 degrees total rotation"
+ - "open risers: visible gaps between consecutive treads"
+ - "1 baluster at the outer edge of each tread"
+ - "continuous helical handrail on top of the balusters"
+ - "top landing platform with its own railing"
+ - "base plate at the bottom of the column"
+ - "each tread about 0.70 m from column to outer edge"
+ - "overall height about 2.7 m"
+ dimensions_m: {height: 2.7, width: 1.6}
+ - id: arch_hard_teahouse
+ tier: hard
+ category: buildings
+ tags: ["axis:arch_detail"]
+ prompt: >
+ a Japanese teahouse: raised wooden floor on short posts, a veranda strip
+ along the front, sliding shoji panels with visible grids, corner posts, and
+ a hipped roof with deep eaves showing exposed rafter tails
+ must_have:
+ - "floor platform raised about 0.5 m on short posts"
+ - "wooden veranda strip along the front edge"
+ - "at least 3 sliding shoji panels in the front wall"
+ - "each shoji panel gridded into at least 12 rectangles"
+ - "square corner posts at the 4 corners"
+ - "hipped roof with eaves overhanging about 0.6 m"
+ - "at least 8 exposed rafter tails visible under each eave side"
+ - "ridge cap along the roof top"
+ - "solid back and side walls contrasting with the shoji front"
+ - "step stone below the veranda"
+ dimensions_m: {width: 3.5, depth: 3.0, height: 3.0}
+ # ------------------------------------------------------- multi-material compositions (hard)
+ - id: comp_hard_drafting_table
+ tier: hard
+ category: furniture
+ tags: ["axis:multi_material", "axis:decomposition"]
+ prompt: >
+ a drafting table setup: tilted drawing board on a trestle frame with quadrant
+ tilt brackets, a parallel ruler riding on side rails, a pen tray, a sheet of
+ paper held by two corner clips, a clamp-on two-arm lamp aimed at the board,
+ and a stool with a footring
+ must_have:
+ - "drawing board about 1.0 m by 0.7 m tilted about 20 degrees"
+ - "parallel ruler bar spanning the board width"
+ - "ruler rides on a rail or wire at each side edge"
+ - "pen tray along the bottom edge of the board"
+ - "trestle legs with a cross stretcher"
+ - "quadrant tilt bracket with a knob at each of the board's 2 sides"
+ - "sheet of paper on the board held by clips at 2 corners"
+ - "lamp clamped to a top corner of the board"
+ - "lamp has 2 arm segments and a shade aimed down at the board"
+ - "stool with a round seat about 0.65 m high"
+ - "stool footring between its legs"
+ - "board top edge about 1.1 m above the floor"
+ dimensions_m: {width: 1.1, depth: 0.9, height: 1.2}
+ - id: comp_hard_telescope_tripod
+ tier: hard
+ category: props
+ tags: ["axis:multi_material", "axis:decomposition"]
+ prompt: >
+ a brass refractor telescope on a wooden tripod: main tube with an objective
+ ring, a thinner draw tube with an eyepiece cup, a clamping collar and cradle
+ on a pan-tilt head, three splayed legs joined by a spreader tray, metal feet
+ must_have:
+ - "main tube about 0.7 m long and 0.08 m diameter"
+ - "objective ring at the front of the tube"
+ - "draw tube about 0.05 m diameter extending from the rear, thinner than the main tube"
+ - "eyepiece cup at the draw tube end"
+ - "clamping collar around the main tube's balance point"
+ - "cradle connecting the collar to a pan-tilt head"
+ - "tension knob on the head"
+ - "3 wooden legs splayed outward, each about 1.2 m long"
+ - "triangular spreader tray connecting the legs at about 0.7 m height"
+ - "metal feet at the leg tips"
+ - "tube tilted upward between 20 and 40 degrees"
+ - "eyepiece height about 1.5 m above the ground"
+ dimensions_m: {height: 1.7}
diff --git a/eval/llm/batteries/teaser_v1_static.yaml b/eval/llm/batteries/teaser_v1_static.yaml
new file mode 100644
index 00000000..c2acb9db
--- /dev/null
+++ b/eval/llm/batteries/teaser_v1_static.yaml
@@ -0,0 +1,306 @@
+# Teaser showcase battery, split one file per track because a `Battery` binds exactly
+# one track + language (bench/run_bench.py). Cross-language prompts inside a track use
+# the per-prompt `language:` override. These four files ARE the source; edit them here.
+# Regenerate: python /tmp/claude-1000/-home-yipeng/256e9f01-5a86-4d6d-9b4b-3d662f71921d/scratchpad/teaser/make_batteries.py
+# Heroes (run with codex:gpt-5.6-sol, everything else agy:gemini-3.6-flash): tsr_obj_penny_farthing, tsr_cad_radial_engine
+name: teaser_v1_static
+track: static_object
+language: blender
+description: 'teaser_v1 — static_object slice (7 prompts). Showcase battery: picked for the picture,
+ every prompt carries its own `why` for the selection evidence.'
+prompts:
+- id: tsr_obj_penny_farthing
+ tier: hard
+ category: vehicles
+ tags:
+ - hero
+ - texture
+ - axis:thin_features
+ - material:iron+brass+leather
+ prompt: 'a Victorian penny-farthing bicycle: a very tall spoked front wheel, a small rear wheel, a curved
+ backbone frame, a sprung leather saddle above the front hub, pedals on the front hub, curved handlebars
+ and polished brass fittings
+
+ '
+ must_have:
+ - front wheel about 1.30 m in diameter with at least 32 straight spokes running hub to rim
+ - rear wheel about 0.40 m in diameter with at least 16 spokes
+ - curved backbone tube sweeping from the front hub head down to the rear fork
+ - leather saddle about 0.26 m long on a sprung post above the front wheel
+ - two cranks with pedals on the front hub, 180 degrees apart
+ - handlebars about 0.55 m wide with two grips
+ - solid tyre band on both rims
+ - brass or nickel fittings at the hubs, the head and the saddle post, distinct in colour from the black
+ frame
+ - both wheels touch the ground plane
+ - no spoke, fork or backbone tube intersects a wheel rim
+ dimensions_m:
+ width: 0.6
+ depth: 1.65
+ height: 1.55
+- id: tsr_obj_lighthouse
+ tier: medium
+ category: buildings
+ tags:
+ - texture
+ - axis:arch_detail
+ - material:stone+glass+copper
+ prompt: 'a coastal lighthouse on a rock plinth: a tapered stone tower with painted bands, a corbelled
+ gallery with a railing, a glazed lantern room with a fresnel lens and a domed copper roof with a weathervane,
+ and a keeper''s door at the base
+
+ '
+ must_have:
+ - tapered tower about 14 m tall, wider at the base than under the gallery
+ - at least 3 painted horizontal bands around the tower
+ - corbelled gallery ring projecting at least 0.5 m beyond the tower under the lantern room
+ - gallery railing with at least 16 balusters
+ - lantern room with at least 8 glazed panes between vertical astragals
+ - fresnel lens drum visible inside the lantern room
+ - domed roof with a finial or weathervane on top
+ - keeper's door with a frame and at least 3 small windows up the tower
+ - rock plinth the tower stands on, tower base fully seated on it
+ - stone tower, glass lantern panes and a copper dome as three distinct materials
+ dimensions_m:
+ width: 6.0
+ depth: 6.0
+ height: 18.0
+- id: tsr_obj_gramophone
+ tier: hard
+ category: props
+ tags:
+ - texture
+ - axis:curved_organic
+ - material:brass+oak+bakelite
+ prompt: 'an antique wind-up gramophone: a big flared brass horn on an elbow, an oak case with a felted
+ turntable and a shellac record, a curved tonearm with a sound box, a crank handle on the side and
+ turned feet
+
+ '
+ must_have:
+ - flared horn whose bell is about 0.55 m across, opening from a throat under 0.06 m along a smooth curved
+ flare
+ - elbow joining the horn throat to the sound box
+ - oak case about 0.40 x 0.40 x 0.22 m with a moulded lip
+ - turntable platter about 0.30 m in diameter on the case top
+ - record disc lying flat on the platter with a centre label
+ - curved tonearm from the case to the sound box, with the needle over the record
+ - crank handle on the side of the case
+ - four turned feet, all touching the ground plane
+ - polished brass horn against dark oak and a black record — three clearly different materials
+ - the horn does not intersect the case or the platter
+ dimensions_m:
+ width: 0.6
+ depth: 0.55
+ height: 0.75
+- id: tsr_cad_radial_engine
+ tier: hard
+ category: mechanisms
+ tags:
+ - hero
+ - texture
+ - axis:decomposition
+ - material:aluminium+steel+brass
+ language: cadquery
+ prompt: 'a five-cylinder radial aero engine: a round crankcase with a bolted front cover, five finned
+ cylinder barrels in a star, domed heads with rocker covers, a pushrod tube per cylinder, an exhaust
+ stub on each head and a splined propeller hub
+
+ '
+ must_have:
+ - five cylinder barrels evenly spaced 72 degrees apart around the crankcase axis
+ - at least 8 cooling fins on every barrel, all the same pitch
+ - domed head with a rocker cover on each cylinder
+ - one pushrod tube per cylinder running from the crankcase up to its head
+ - circular crankcase about 0.36 m in diameter
+ - front cover with at least 8 bolt heads on a circular pattern
+ - propeller hub on the crankcase axis with at least 6 bolt holes
+ - an exhaust stub on each head, all pointing the same way around the ring
+ - pale aluminium barrels, a darker steel crankcase and brass pushrod ends as distinct part colours
+ - no barrel, pushrod or exhaust stub intersects a neighbour
+ dimensions_m:
+ width: 0.9
+ depth: 0.42
+ height: 0.9
+- id: tsr_cad_portafilter
+ tier: medium
+ category: tools
+ tags:
+ - texture
+ - material:steel+walnut
+ - product_shot
+ language: cadquery
+ prompt: 'a double-spout espresso portafilter: a polished stainless basket holder with three locking
+ lugs, a knurled collar, two tapered spouts and a turned walnut handle on a chrome ferrule
+
+ '
+ must_have:
+ - cylindrical basket body about 0.075 m outside diameter and 0.035 m deep
+ - filter basket inside the body with a visible rim below the body lip
+ - three locking lugs evenly spaced around the body
+ - knurled or ribbed collar between the body and the ferrule
+ - two tapered spouts under the body, splayed at least 20 degrees apart
+ - turned walnut handle about 0.12 m long, thicker in the middle, on the body axis
+ - chrome ferrule joining handle to body
+ - steel body and chrome ferrule against a dark wood handle, three distinct part colours
+ - handle, ferrule and body are coaxial and touch without gaps
+ dimensions_m:
+ width: 0.09
+ depth: 0.24
+ height: 0.09
+- id: tsr_tjs_arcade_cabinet
+ tier: medium
+ category: props
+ tags:
+ - emissive
+ - material:laminate+glass+plastic
+ language: threejs
+ prompt: 'an upright 1980s arcade cabinet: a sloped control panel with a ball-top joystick and six buttons,
+ a recessed CRT screen behind a bezel, a glowing backlit marquee, a speaker grille, side art panels,
+ a coin door and T-molding trim
+
+ '
+ must_have:
+ - cabinet about 1.75 m tall, 0.70 m wide, 0.85 m deep, standing on the ground plane
+ - sloped control panel with one ball-top joystick and at least 6 round buttons in two rows
+ - screen recessed behind a bezel, the screen surface emissive so it reads as switched on
+ - backlit marquee across the top, emissive and brighter than the cabinet body
+ - speaker grille with at least 12 holes above the screen
+ - coin door with two coin slots and a coin return cup
+ - T-molding trim strip running along both side edges
+ - side art panel in a saturated colour distinct from the black cabinet body
+ - no button or joystick sinks below the control panel surface
+ dimensions_m:
+ width: 0.7
+ depth: 0.85
+ height: 1.75
+- id: tsr_tjs_deco_chandelier
+ tier: hard
+ category: lighting
+ tags:
+ - instancing
+ - material:brass+glass
+ - radial_symmetry
+ language: threejs
+ prompt: 'an art-deco brass chandelier: a central stem on a ceiling canopy and chain, two tiers of curved
+ arms carrying frosted candle shades, stepped faceted glass drops hanging in rings from each tier and
+ a fluted glass bowl at the crown
+
+ '
+ must_have:
+ - central brass stem with a ceiling canopy and a chain of at least 5 links above it
+ - upper tier of 6 curved arms and a lower tier of 8 curved arms, each tier evenly spaced
+ - a frosted candle shade at the end of every arm
+ - at least 40 faceted glass drops hanging in even rings below the two tiers
+ - fluted glass bowl at the crown under the stem
+ - overall diameter about 0.90 m and overall height about 1.10 m
+ - warm brass arms and stem against clear or frosted glass drops and shades
+ - every drop hangs free below its tier and no drop intersects an arm
+ dimensions_m:
+ width: 0.9
+ depth: 0.9
+ height: 1.1
+
+# ===========================================================================
+# SHARPENED RE-RUNS — these are the prompts the teaser page actually ships.
+# The original id each one replaces is kept above it, so the pair stays legible;
+# see docs/TEASER.md for the picture each attempt produced.
+# NOTE: BenchPrompt (bench/run_bench.py) carries only id/prompt/tier/category/
+# must_have/dimensions_m/tags/language. The re-runs also used --style, --must-not
+# and --candidates on the CLI; those are recorded as comments here rather than as
+# keys, because an unknown key is dropped silently.
+# ===========================================================================
+# SHIPPED. Supersedes tsr_cad_radial_engine (same judged 0.600, broken picture: one barrel
+# buried through the ground plane, exhaust stubs reading as detached sticks, no stand, all grey).
+
+# CLI flags used, not expressible in this schema:
+# --style "clean machined-metal product shot: pale satin aluminium fins, dark grey cast steel crankcase, warm polished brass pushrod tubes and black rocker covers, crisp fillets and chamfers on every machined edge"
+# --must-not "a cylinder barrel buried in or passing through the ground plane"
+# --must-not "exhaust stubs or pushrods detached from the heads, floating in space, or pointing in inconsistent directions"
+# --must-not "an all-grey untextured look with no colour separation between crankcase, barrels and pushrods"
+# --candidates 2
+# --profile quality
+- id: tsr_cad_radial_engine_v2
+ tier: hard
+ category: mechanisms
+ tags:
+ - teaser
+ - rerun
+ - hero
+ language: cadquery
+ prompt: 'a five-cylinder radial aero engine displayed on a low steel stand, in the correct museum orientation:
+ ONE cylinder points straight UP at top dead centre and the other four are at 72 degree steps, so there
+ is NO cylinder pointing straight down and the engine sits clear of the ground on its stand. A dark grey
+ machined steel crankcase carries a bolted circular front cover and a splined propeller hub on the axis;
+ each of the five barrels is a pale satin aluminium tube wrapped in evenly pitched square cooling fins,
+ capped by a domed head with a black rocker cover; a slim brass pushrod tube lies flush along the front
+ of each barrel from the crankcase to its rocker cover; a short polished steel exhaust stub leaves each
+ head, all five swept the same way around the ring.
+
+ '
+ must_have:
+ - one cylinder barrel pointing straight up at top dead centre, the other four at 72 degree steps around
+ the crankcase axis
+ - no cylinder barrel pointing straight down; the lowest part of the engine is the steel stand, and the stand
+ sits flat on the ground plane
+ - a low steel stand under the crankcase, welded to it, carrying the whole engine clear of the ground
+ - at least 8 cooling fins on every barrel, all the same pitch and the same outer diameter
+ - a domed head with a black rocker cover on each cylinder, sitting flush on the top of its barrel
+ - one brass pushrod tube per cylinder, lying flush along the barrel from the crankcase to its rocker cover,
+ touching both
+ - circular crankcase about 0.36 m in diameter with a front cover carrying at least 8 bolt heads on a circular
+ pattern
+ - propeller hub on the crankcase axis with at least 6 bolt holes
+ - one exhaust stub per head, each rooted flush in its head, all five angled the same way around the ring
+ - pale aluminium barrels, a darker steel crankcase, brass pushrod tubes and black rocker covers as four
+ clearly distinct part colours
+ - no barrel, pushrod or exhaust stub intersects or floats away from a neighbour
+ dimensions_m:
+ width: 0.9
+ depth: 0.42
+ height: 0.98
+# SHIPPED at a LOWER judged score than the run it replaces (0.600 vs 0.841): v1 was a clean brass
+# armature with plain white shades and ZERO hanging crystal, i.e. none of the sparkle the prompt
+# was picked for.
+
+# CLI flags used, not expressible in this schema:
+# --style "expensive lighting-catalogue product shot: warm polished brass against clear cut crystal and frosted glass, every facet catching a highlight, strong specular sparkle, no flat matte surfaces"
+# --must-not "a chandelier with no hanging crystal drops, or with the drops too small or too few to see"
+# --must-not "plain white cylinder shades with no finial"
+# --must-not "drops merged into the arms or hidden inside the tiers"
+# --candidates 2
+# --profile quality
+- id: tsr_tjs_deco_chandelier_v2
+ tier: hard
+ category: lighting
+ tags:
+ - teaser
+ - rerun
+ language: threejs
+ prompt: 'an art-deco brass chandelier dripping with cut crystal: a stepped brass ceiling canopy and a chain
+ drop to a fluted central stem; two tiers of curved brass arms, six above and eight below, each ending
+ in a frosted glass candle sleeve with a small tapered flame-tip finial; and hanging from the rim of BOTH
+ tiers, in even concentric rings, more than forty stepped faceted crystal drops - long slender prisms with
+ cut facets, each hanging free on its own short brass link so the whole lower half of the fixture reads
+ as a curtain of glass; a fluted clear glass bowl at the crown under the stem.
+
+ '
+ must_have:
+ - central fluted brass stem with a stepped ceiling canopy and a chain of at least 5 visible links above
+ it
+ - an upper tier of 6 curved brass arms and a lower tier of 8 curved brass arms, each tier evenly spaced
+ - a frosted glass candle sleeve with a small tapered flame-tip finial at the end of every arm
+ - MORE THAN 40 faceted crystal drops, clearly visible as separate slender prisms, hanging in even concentric
+ rings from the rims of BOTH tiers
+ - each crystal drop is at least 0.06 m long, hangs vertically on its own short brass link, and hangs free
+ below its tier
+ - the rings of drops are the dominant visual feature of the lower half of the chandelier
+ - fluted clear glass bowl at the crown under the stem
+ - overall diameter about 0.90 m and overall height about 1.10 m
+ - warm brass arms and stem clearly distinct in colour from the clear crystal drops and the frosted white
+ shades
+ - no drop intersects an arm, a shade or another drop
+ dimensions_m:
+ width: 0.9
+ depth: 0.9
+ height: 1.1
diff --git a/eval/llm/build_prompts.py b/eval/llm/build_prompts.py
index 4133c48e..31c52e74 100644
--- a/eval/llm/build_prompts.py
+++ b/eval/llm/build_prompts.py
@@ -19,7 +19,7 @@ Suites built here (sizes are what the assets give):
3dcodebench_img4text 212 Blender 4 views + description
heldout_blender 103, heldout_cadquery 200, heldout_openscad 50, heldout_glsl 200, heldout_threejs 40
(3dcodeverse/3dcodeverse-2m finetune/test/*.parquet, never trained on)
- harness_<battery> rubric-judged batteries copied from harness/bench/prompts (no GT; scored by
+ harness_<battery> rubric-judged batteries in eval/llm/batteries (no GT; scored by
execution + optional VLM judge), one suite per YAML
System prompts are the exact strings used in finetune/docs/REPORT.md so numbers stay comparable; the
@@ -205,7 +205,7 @@ def build_batteries() -> None:
"reference": {"code": None, "mesh": None, "renders": []},
"meta": {"tier": p.get("tier"), "category": p.get("category"), "must_have": p.get("must_have", []),
"dimensions_m": p.get("dimensions_m"), "tags": p.get("tags", []), "track": bat.get("track"),
- "source": f"harness/bench/prompts/{y.name}"}})
+ "source": f"eval/llm/batteries/{y.name}"}})
if rows:
write_jsonl(rows, f"harness_{bat['name']}")
diff --git a/eval/llm/config.py b/eval/llm/config.py
index 137b48e4..71b2163e 100644
--- a/eval/llm/config.py
+++ b/eval/llm/config.py
@@ -25,9 +25,8 @@ from pathlib import Path
PKG_DIR = Path(__file__).resolve().parent
PROMPTS_DIR = PKG_DIR / "data" / "prompts"
-# the harness batteries are ONE set of files, owned by the harness evaluation next door (eval/bench/prompts);
-# these ten are the ones a bare LLM can be asked one-shot (static + graphics, no plan, no tools)
-BATTERIES_DIR = PKG_DIR.parent / "bench" / "prompts"
+# the ten harness prompt batteries a bare LLM can be asked one-shot (static + graphics, no plan, no tools)
+BATTERIES_DIR = PKG_DIR / "batteries"
HARNESS_BATTERIES = ("articulated_v2", "compare_v4", "complexity_v3", "fancy_v1_blender", "fancy_v1_cadquery",
"fancy_v1_threejs", "graphics_v2", "static_objects_v1", "static_objects_v2", "teaser_v1_static")
diff --git a/eval/pyproject.toml b/eval/pyproject.toml
index 4d3fddb0..07a18f67 100644
--- a/eval/pyproject.toml
+++ b/eval/pyproject.toml
@@ -1,19 +1,18 @@
-# Tooling config for the evaluation code. Nothing here is installed: `bench` and `llm` are run
-# from this directory (`cd eval && python bench/<script>.py …`, `cd eval && python -m pytest`).
+# Tooling config for the evaluation code. Nothing here is installed: `llm` runs from this directory.
[tool.ruff]
line-length = 100
target-version = "py313"
-extend-exclude = ["bench/out", "llm"] # llm/ keeps its own style; bench/out is run data
+extend-exclude = ["llm"] # llm/ keeps its own style
[tool.ruff.lint]
select = ["E", "W", "F", "I", "B", "UP", "SIM"]
ignore = ["E501"]
[tool.ruff.lint.isort]
-known-first-party = ["bench", "codeverse3d", "tests"]
+known-first-party = ["llm", "codeverse3d"]
[tool.pytest.ini_options]
-testpaths = ["tests"]
+testpaths = ["llm/tests"]
timeout = 600
addopts = ["-m", "not live", "-n", "auto", "--dist", "worksteal"]
markers = [