Official Codebase of 3DCodeVerse
Python
Repo docs for the plugin layout; eval/llm keeps its ten battery prompt files and its own JSONL reader
gaoypeng1ab2ebfd450f
diff --git a/CLAUDE.md b/CLAUDE.mdindex b7a88138..4503c5a1 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -1,14 +1,15 @@# 3dcodeverse (repo root)-The harness lives in **`harness/`** — read `harness/CLAUDE.md` +-`harness/docs/ARCHITECTURE.md` before touching it. Install with-`pip install -e harness`. Run tests with-`cd harness && python -m pytest tests -q -m "not live"`.+**`harness/`** is the `3dcodeverse` plugin for Claude Code, Codex and Antigravity: skills, a judge+subagent and the `3dcode` command-line instruments (build, measure, render, gates, judge pack, score).+Read `harness/CLAUDE.md` + `harness/docs/ARCHITECTURE.md` before touching it. The repo root is its+marketplace (`.claude-plugin/marketplace.json`). Install: `bash harness/setup.sh`. Tests live outside+the repo (`~/3dcodeverse_tests`); never add a `tests/` folder back under `harness/`.-**`eval/`** evaluates things and is never imported by the harness: `eval/bench` evaluates the HARNESS-(batteries — `cd eval && python -m bench.run_bench <battery.yaml>` —-A/B rigs, reports; its tests: `cd eval && python -m pytest`), `eval/llm` evaluates a bare LLM/VLM-(was `finetune/3dcodeverse_eval`). Read `eval/README.md` first.+**`flywheel/`** (`c3d_flywheel`, CLI `c3d-flywheel`) reads runs recorded by the old CLI harness (v0.0.x):+show/status, cost, gallery, dataset export/pairs/refine, pick. It depends on the plugin, never the reverse.++**`eval/llm`** evaluates a bare LLM/VLM on 3D coding (no harness in the loop); read `eval/README.md`.Other top-level folders are separate components: `toolkits/` owns the general data CLI and source curation;`finetune/` owns training and its data preparation. The canonical LLaMA-Factory data scripts live indiff --git a/README.md b/README.mdindex 179020dc..9f0a7e1c 100644--- a/README.md+++ b/README.md@@ -78,32 +78,31 @@ on 805 fixed tasks:## Quick Start-The harness needs Python 3.13, Node 20.6+, Git, Gemini API keys and the Antigravity CLI (`agy`, the-default generator). Blender 5.2 LTS is optional (the `blender`, `urdf_blender` and `scene_blender`-languages). Full guide: [harness/docs/INSTALL.md](harness/docs/INSTALL.md).-The default judge is `gemini:gemini-4-argon` with thinking `medium`.+The harness ships as **`3dcodeverse`, a plugin for Claude Code, Codex and Antigravity**: the host agent+writes the code with its own model, and the plugin's skills and `3dcode` instruments build, measure,+render, gate and score it (a fresh-context subagent judges). It needs Python 3.13, Node 20.6+ and Git;+Blender 5.2 LTS for the `blender`, `urdf_blender` and `scene_blender` languages.+Full guide: [harness/README.md](harness/README.md) and [harness/docs/INSTALL.md](harness/docs/INSTALL.md).```bash-cd harness-bash setup.sh # pip install -e . + runtime_js + headless Chrome + the agy / claude-code / codex CLIs at their latest, then `3dcodeverse doctor`-export GEMINI_API_KEYS="key1,key2" # agy runs in Gemini-API mode on this key pool-3dcodeverse doctor # every row OK or WARN; a FAIL row says what to fix-3dcodeverse make "a classic park bench" --track static_object --language threejs-3dcodeverse gallery serve # browse every run at http://127.0.0.1:8765/+git clone https://github.com/gaoypeng/3dcodeverse && cd 3dcodeverse+bash harness/setup.sh # the `3dcode` instruments + runtime_js + headless Chrome, then doctor+claude plugin marketplace add . && claude plugin install 3dcodeverse@3dcodeverse # Claude Code+codex plugin marketplace add . && codex plugin add 3dcodeverse@3dcodeverse # Codex+agy plugin install harness # Antigravity```-The optional generators claude-code and codex use your local subscriptions: `--generator claude-code:<model>`-or `--generator codex:<model>`. Tracks, languages and costs per track: [harness/README.md](harness/README.md) and-[harness/docs/RUNBOOK.md](harness/docs/RUNBOOK.md); evaluation: [eval/README.md](eval/README.md).+Then ask your agent for a 3D thing, e.g. "Make a mid-century wooden dining chair in Blender bpy."## Repository Structure| Directory | Description | Documentation || :-- | :-- | :-- |-| [`harness/`](harness/) | Agent-based 3D content generation, execution, rendering, and verification. | [Harness guide](harness/README.md) |+| [`harness/`](harness/) | The `3dcodeverse` agent plugin: skills, judge subagent, and the `3dcode` build / measure / render / gate / score instruments. | [Plugin guide](harness/README.md) |+| [`flywheel/`](flywheel/) | Reader for runs recorded by the v0.0.x CLI harness: gallery, cost, dataset export, pick. | [Flywheel guide](flywheel/README.md) || [`toolkits/`](toolkits/) | General data CLI and per-source corpus curation. | [Data toolkit guide](toolkits/README.md) || [`finetune/`](finetune/) | LLaMA-Factory training-data preparation, supervised fine-tuning, and preference optimization for open models. | [Training guide](finetune/README.md) |-| [`eval/`](eval/) | Benchmarks and evaluation pipelines for the harness and standalone models. | [Evaluation guide](eval/README.md) |+| [`eval/`](eval/) | Evaluation of standalone LLMs / VLMs on 3D coding. | [Evaluation guide](eval/README.md) |## Citationdiff --git a/eval/README.md b/eval/README.mdindex 62772b68..b02954af 100644--- a/eval/README.md+++ b/eval/README.md@@ -1,38 +1,16 @@-# eval/ — two evaluations, kept apart from what they evaluate+# eval/ — evaluating a bare LLM / VLM on 3D coding-| folder | what it evaluates | how |-|---|---|---|-| [`bench/`](bench/) | **the harness** (`../harness`): does plan → build → gates → judge → refine beat asking a model once? | prompt batteries run through the harness and through one-shot / bare-agent arms, judged by ONE fixed evaluator; paired A/B rigs for harness switches; offline re-judging and reports |-| [`llm/`](llm/) | **a bare LLM / VLM** on 3D coding — no harness in the loop | text → code and image → code over 3DCodeBench, the held-out dialect sets and ten harness batteries; every answer is executed; geometry and image metrics against references |--`docs/` holds the evaluation write-ups: `EVAL.md` (protocol, judge calibration, every recorded-comparison), `COMPLEXITY.md` (the complexity vector and the score-vs-complexity study),-`PAPER_WRITING.md`. `tests/` tests the `bench` scripts.--The dependency is one-way: `bench` and `llm` import `codeverse3d` (the harness package); nothing under-`harness/` imports anything here, and the harness test suite passes without this folder. The two-evaluations share one set of battery files — `bench/prompts/*.yaml` — and `llm` reads the ten it can-ask one-shot from there (`llm/config.py: HARNESS_BATTERIES`).--Harness evaluation defaults to judge `gemini:gemini-4-argon` with thinking `medium`.-Keep the judge model and sampling settings fixed across comparison arms; historical reports-retain the judge they were measured with.+[`llm/`](llm/) asks a model one-shot — no plugin, no tools in the loop — over 3DCodeBench, the held-out+dialect sets and ten prompt batteries (`llm/batteries/*.yaml`); every answer is executed and scored with+geometry and image metrics against references. `docs/` holds the earlier evaluation write-ups+(`EVAL.md`, `COMPLEXITY.md`, `PAPER_WRITING.md`); the harness benchmark they describe (`eval/bench`) ran+on the v0.0.x CLI harness and lives in those releases.```bashcd eval-pip install -e ../harness # once; both evaluations import codeverse3d-python -m bench.run_bench bench/prompts/static_objects_v1.yaml --generator agy:gemini-4-argon-python -m bench.report bench/out/static_objects_v1 # rebuild report.md + report.html-python bench/compare_backends.py --prompts bench/prompts/compare_v1.yaml --arms harness:agy:gemini-4-argon,oneshot:gemini:gemini-3.8-flash --out bench/out/compare_v1-python -m llm.config # doctor for the LLM evaluation: tools and assets it can / cannot find-python -m pytest # the bench tests (offline)-ruff check .+python -m llm.config # what the evaluation can / cannot find+python -m pytest llm/tests # offline tests```-Run data is never in git: `bench/out/` (battery output) and `llm/data/prompts/*.jsonl` (built by-`python -m llm.build_prompts` from the gated Hub assets) are ignored. Every bench report reads its-journals through `bench/_jsonl.py` (a truncated last line costs one row; the last row per cell wins)-and states its uncertainty through `bench/stats.py` (the 95 % t-interval, the exact sign test). Every `bench` script bootstraps-its own import path — `../harness` for THIS tree's `codeverse3d`, `.` for the `bench` package — so an-editable install of a different checkout cannot silently stand in for the tree under test-(`tests/test_worktree_import.py`).+Run data is never in git: `llm/data/prompts/*.jsonl` is built by `python -m llm.build_prompts` from the+gated Hub assets.diff --git a/eval/llm/_jsonl.py b/eval/llm/_jsonl.pyindex 6ec0aeba..22a6750e 100644--- a/eval/llm/_jsonl.py+++ b/eval/llm/_jsonl.py@@ -10,13 +10,25 @@ byte) is skipped and that row simply runs again; a missing file has no rows."""from __future__ import annotationsimport json-from functools import partialfrom pathlib import Path-from codeverse3d.proc import read_jsonl_lenient, write_text_atomic+from codeverse3d.proc import write_text_atomic-#: every JSON object line of a file, in order-read_rows = partial(read_jsonl_lenient, dicts_only=True)++def read_rows(path: Path | str) -> list[dict]:+ """Every JSON object line of ``path``, in order; a line that does not parse is skipped."""+ p = Path(path)+ if not p.is_file():+ return []+ rows = []+ for line in p.read_text(encoding="utf-8", errors="replace").splitlines():+ try:+ row = json.loads(line)+ except ValueError:+ continue+ if isinstance(row, dict):+ rows.append(row)+ return rowsdef write_rows(path: Path, rows: list[dict], *, ensure_ascii: bool = True) -> None:diff --git a/eval/llm/batteries/articulated_v2.yaml b/eval/llm/batteries/articulated_v2.yamlnew file mode 100644index 00000000..87ac89e3--- /dev/null+++ b/eval/llm/batteries/articulated_v2.yaml@@ -0,0 +1,227 @@+# articulated_v2 — multi-DoF battery (v1: easy 75% pass; hard discriminated at 50%).+#+# Every prompt has >= 3 joints and at least one COUPLED constraint: a clearance or+# rest condition that only holds when several joints are posed together (folded stack+# thickness, sashes passing in separate tracks, a lid resting on a prop stick).+# must_have items name the MOTION with a quantity ("drawer slides 0.25 m out toward+# the front without hitting the frame") so the judge can verify each DoF in the+# per-joint limit poses. The harness renders rest + ONE joint at a time (lower/+# upper limit) and IGNORES <mimic>, so coupled conditions are stated against the+# rest pose or against the collision sweep, which samples random combined poses+# and reports interpenetration as gate errors.+# Difficulty axes: joint count (3-8 DoF), coupled clearances, mixed joint types+# (revolute + prismatic + continuous in one model).+# Expected discrimination: medium 30-60% pass (3-4 loosely coupled joints), hard+# 0-30% pass (4+ joints with clearance coupling — v1 showed baseline collapse to+# 0.25 on such prompts, with large refine deltas).+name: articulated_v2+track: articulated_object+language: urdf_blender+description: >+ 14 articulated prompts (URDF + bpy link meshes), all >= 3 DoF with coupled+ constraints. 4 medium, 10 hard. must_have names each joint's motion with a+ quantity and at least one cross-joint clearance verifiable in the rest pose+ or by the collision sweep over random combined poses.+prompts:+ # ------------------------------------------------------------------ medium+ - id: art_med_architect_lamp+ tier: medium+ category: lighting+ prompt: >+ an adjustable architect lamp with four revolute joints: base swivel, shoulder,+ elbow and head tilt, with parallel link bars beside the lower arm+ must_have:+ - "weighted round base"+ - "arm assembly swivels at least 180 degrees about the vertical base axis"+ - "lower arm about 0.35 m long, shoulder revolute raises it from 20 to 80 degrees"+ - "upper arm about 0.35 m long, elbow revolute with at least 120 degrees range"+ - "head tilt revolute at the wrist, shade opens downward"+ - "twin parallel link bars alongside the lower arm"+ - "shoulder at its lower limit folds the lower arm down toward the base without touching it"+ - id: art_med_tool_chest+ tier: medium+ category: tools+ prompt: >+ a machinist tool chest with a rear-hinged lid over a top tray and three+ drawers of decreasing height on slides+ must_have:+ - "chest carcass about 0.60 m wide and 0.40 m tall"+ - "lid hinged along the rear edge, opens 90 degrees to expose a top tray"+ - "3 drawers of decreasing height from bottom to top"+ - "each drawer slides 0.25 m out toward the front without hitting its neighbours"+ - "drawer pulls on every drawer front"+ - "with the lid closed, each fully opened drawer clears the lid overhang"+ - id: art_med_camera_tripod+ tier: medium+ category: props+ prompt: >+ a camera tripod: three legs that splay at the crown, a sliding lower section+ per leg, and a pan-tilt head with a mounting plate+ must_have:+ - "3 legs, each splaying by revolute from vertical to 25 degrees at the crown"+ - "each leg's lower section slides out 0.40 m along the leg axis"+ - "pan joint at the head rotates continuously about the vertical axis"+ - "tilt joint pitches the mounting plate from -90 to +30 degrees"+ - "rest pose: legs at equal splay and full extension, mounting plate level about 1.4 m up"+ - "rubber feet at all 3 leg tips, resting on the ground plane in the rest pose"+ - id: art_med_dutch_door+ tier: medium+ category: architecture+ prompt: >+ a dutch stable door in its frame: independent upper and lower leaves on the+ same jamb, a slide bolt that joins them, and a latch handle on the lower leaf+ must_have:+ - "door frame with both leaves fitted, total opening about 2.0 m tall"+ - "upper leaf swings open 100 degrees on hinges at the jamb"+ - "lower leaf swings open 100 degrees on hinges at the same jamb"+ - "slide bolt on the upper leaf travels 0.08 m down into the lower leaf edge"+ - "bolt at its extended limit spans the seam into a notch in the lower leaf edge (leaves closed at rest)"+ - "shelf ledge on top of the lower leaf clears the swinging upper leaf in every sweep pose"+ - "latch handle on the lower leaf"+ # -------------------------------------------------------------------- hard+ - id: art_hard_rolltop_desk+ tier: hard+ category: furniture+ prompt: >+ a roll-top desk: a tambour of hinged slats that retracts along curved side+ tracks to expose pigeonholes and the desktop, plus two pedestal drawers+ per side+ must_have:+ - "desk carcass with 2 pedestals and a knee-hole"+ - "tambour approximated as at least 8 slats riding in curved side tracks"+ - "tambour travels from fully closed at rest to fully open at its joint limit, moving toward the rear"+ - "in the rest pose the closed tambour's bottom rail meets the desktop with under 0.01 m gap"+ - "at the tambour's open limit at least 4 pigeonhole compartments are exposed"+ - "2 drawers per pedestal, each slides 0.30 m out toward the front"+ - "open drawers do not intersect the knee-hole space or each other"+ - "drawer pulls and a writing surface about 0.75 m above the floor"+ - id: art_hard_folding_workbench+ tier: hard+ category: tools+ prompt: >+ a folding workbench: a top on two leg frames that fold inward, each held+ open by a diagonal locking brace with an elbow pivot+ must_have:+ - "top about 1.2 m by 0.6 m, work height 0.85 m when deployed"+ - "2 leg frames, each folding inward by revolute at the top rail"+ - "2 locking braces, each with an elbow revolute that snaps straight when deployed"+ - "each brace elbow folds through at least 90 degrees"+ - "each leg frame at its folded limit lies within 0.15 m of the top's underside, with no collision in any sweep pose"+ - "rest pose: deployed, all 4 feet on the ground and braces straight"+ - id: art_hard_parallel_clamp+ tier: hard+ category: tools+ prompt: >+ a parallel-jaw bench clamp: a fixed jaw and a moving jaw on two guide rods,+ driven by a lead screw with a crank handle and a swivel foot pad+ must_have:+ - "fixed jaw body and moving jaw about 0.12 m wide"+ - "2 guide rods passing through the moving jaw"+ - "lead screw between the rods, crank handle rotates continuously"+ - "moving jaw slides 0.12 m along the guide rods toward the fixed jaw"+ - "jaw faces remain parallel over the whole travel"+ - "swivel pad on the screw tip tilts up to 5 degrees against the fixed jaw"+ - "at the moving jaw's closed limit the jaw faces meet with no interpenetration"+ - id: art_hard_sash_window+ tier: hard+ category: architecture+ prompt: >+ a double-hung sash window: two counterweighted sashes sliding in separate+ tracks of one frame, with a rotating lock lever on the meeting rail+ must_have:+ - "outer frame about 0.9 m by 1.5 m with sill and casing"+ - "2 sashes, each divided into at least 4 panes by muntins"+ - "lower sash slides up 0.40 m in the inner track"+ - "upper sash slides down 0.35 m in the outer track"+ - "sashes pass each other with visible clearance, never touching"+ - "lock lever on the meeting rail rotates 90 degrees"+ - "lower sash at its upper limit overlaps the upper sash, both visible in front view"+ - id: art_hard_grand_piano+ tier: hard+ category: musical_instruments+ prompt: >+ a baby grand piano with a hinged main lid held by a prop stick, a rotating+ fallboard over the keyboard, three pedals on a lyre, and a bench with a+ hinged storage lid+ must_have:+ - "curved-rim piano body on 3 legs"+ - "keyboard with white and black key rows exposed when the fallboard is open"+ - "fallboard rotates 90 degrees from covering the keys to lying flat behind them"+ - "main lid hinged along the straight side, held open about 30 degrees at its lower limit, hinging up to about 45 degrees"+ - "prop stick pivots up from the rim; at its raised limit the stick tip meets the lid's underside within 0.01 m"+ - "prop stick at rest lies flat inside the rim"+ - "3 pedals on a lyre post under the body"+ - "bench with a storage lid hinged at the rear, opening 60 degrees"+ - id: art_hard_umbrella+ tier: hard+ category: props+ prompt: >+ an umbrella with a simplified rib mechanism: a runner sliding on the shaft+ drives at least six ribs that pivot at the crown, with canopy panels+ stretched between the ribs+ must_have:+ - "central shaft about 0.9 m with a curved handle"+ - "runner slides 0.30 m up the shaft"+ - "at least 6 ribs, each revolute at the crown"+ - "ribs swing from folded along the shaft to 80 degrees open"+ - "stretcher links connect the runner to each rib mid-point in the open rest pose"+ - "rest pose: open, with canopy panels stretched between adjacent ribs"+ - "each rib at its folded limit lies within 0.05 m of the shaft without crossing it"+ - id: art_hard_casement_window+ tier: hard+ category: architecture+ prompt: >+ a casement window driven by a sill crank: the sash swings outward on side+ hinges while a folding operator arm with a sliding shoe pushes its bottom rail+ must_have:+ - "frame about 0.6 m by 1.2 m with a gridded sash"+ - "sash swings outward 90 degrees on hinges at one side edge"+ - "crank handle at the sill rotates continuously"+ - "operator arm with an elbow revolute links the sill to the sash bottom rail"+ - "operator arm shoe slides 0.30 m along a track in the sash bottom rail"+ - "rest pose: window closed, operator arm folded flat over the sill without touching the sash"+ - "latch lever on the free edge rotates to lock the sash"+ - id: art_hard_step_ladder+ tier: hard+ category: tools+ prompt: >+ an A-frame step ladder: a five-step front frame and a rear frame hinged at+ the top cap, held open by two spreader arms with center elbows+ must_have:+ - "front frame with 5 flat steps"+ - "rear frame hinged to the front frame at the top cap"+ - "frames open from parallel to about 30 degrees apart"+ - "2 spreader arms, each with a center elbow revolute"+ - "rest pose: open, spreaders locked straight between the frames"+ - "rear frame at its closed limit lies parallel to the front frame within 0.10 m; spreaders fold without hitting the steps in any sweep pose"+ - "top cap platform and 4 feet touching the ground in the open rest pose"+ - id: art_hard_drawleaf_table+ tier: hard+ category: furniture+ prompt: >+ a draw-leaf dining table: two end leaves stowed under the fixed center top+ slide out horizontally on tapered runners, plus an apron drawer on one long+ side+ must_have:+ - "fixed center top about 1.2 m by 0.9 m on 4 legs, surface about 0.75 m high"+ - "2 end leaves about 0.45 m deep stowed under the center top at rest"+ - "each leaf slides 0.45 m outward along the table's long axis"+ - "each leaf rides on 2 tapered runners that emerge from under the top as it extends"+ - "extended leaf clears the center top edge with a gap under 0.02 m"+ - "apron drawer on one long side slides 0.25 m out"+ - "leaves and drawer extend without touching the legs or each other in any sweep pose"+ - "rest pose: both leaves fully stowed inside the center top's footprint"+ - id: art_hard_scissor_mirror+ tier: hard+ category: household+ prompt: >+ a wall-mounted scissor extension mirror: a pantograph of three X-links+ between a wall plate and a mirror head that also swivels and tilts+ must_have:+ - "wall plate anchoring the mechanism"+ - "pantograph of at least 3 X-links with revolute pivots at centers and ends"+ - "rest pose: mirror head retracted 0.15 m from the wall plate; each pantograph joint at its limit moves the head farther from the wall"+ - "all pantograph bars lie in a horizontal plane, scissoring parallel to the floor"+ - "mirror head swivels at least 90 degrees about its vertical post"+ - "mirror tilts at least 30 degrees about its horizontal axis"+ - "X-links nest without bar intersections in the retracted rest pose"diff --git a/eval/llm/batteries/compare_v4.yaml b/eval/llm/batteries/compare_v4.yamlnew file mode 100644index 00000000..2f6988d8--- /dev/null+++ b/eval/llm/batteries/compare_v4.yaml@@ -0,0 +1,713 @@+# compare_v4 — harness-vs-oneshot arms on the UNION of the two static batteries (40 prompts).+#+# Why: the compare_v2/v3 batteries hold 8/12 prompts, and docs/report.html measured that an+# 8-prompt paired battery resolves a real difference of ~0.25, not the 0.02 the profiles are+# tuned to. This battery is every static_objects_v2 prompt (4 medium controls + 16 hard, all+# with dimensions_m) plus every static_objects_v1 prompt whose text is not already in v2+# (8 easy + 4 medium + 8 hard), ids kept VERBATIM from their source battery so recorded+# cells of those batteries stay comparable. Prompt-text duplicates dropped from v1:+# furn_med_dining_chair, veh_med_pickup, tool_med_hand_drill, appl_med_toaster (== ctrl_*).+#+# must_have is the FIXED judge's acceptance checklist for every arm (no plan-derived items)+# so harness and one-shot are judged alike. Tiers: 8 easy / 8 medium / 24 hard.+name: compare_v4+track: static_object+language: blender+description: >+ 40 static-object prompts (blender / bpy) — the union of static_objects_v2 and+ static_objects_v1 by prompt text — used to compare the harness loop against raw+ one-shot generation under ONE fixed judge with enough prompts for a paired CI+ narrower than the effects claimed. 8 easy / 8 medium / 24 hard.+prompts:+- id: ctrl_med_dining_chair+ tier: medium+ category: furniture+ tags:+ - control+ - axis:decomposition+ prompt: a mid-century wooden dining chair with a slatted backrest and tapered legs+ must_have:+ - four tapered legs, each thinner at the floor than at the seat+ - seat slab about 0.42 m wide+ - seat surface about 0.45 m above the floor+ - backrest with at least 3 vertical slats+ - back legs continue up into the backrest posts+ - backrest top rail between 0.80 m and 0.95 m above the floor+ - all four feet touch the ground plane+ - no leg or slat intersects the seat slab+ dimensions_m:+ width: 0.45+ depth: 0.5+ height: 0.9+- id: ctrl_med_pickup+ tier: medium+ category: vehicles+ tags:+ - control+ - axis:decomposition+ prompt: a pickup truck with a two-door cab, open cargo bed, wheel arches and side mirrors+ must_have:+ - cab with windshield and 2 side windows+ - open rear bed with 3 walls and a tailgate+ - 4 wheels with hubs, all touching the ground+ - wheel diameter about 0.75 m+ - front grille and 2 headlights+ - 2 side mirrors on stalks+ - bed floor about 0.80 m above the ground+ - overall length about 5.2 m+ dimensions_m:+ length: 5.2+ width: 1.9+ height: 1.8+- id: ctrl_med_toaster+ tier: medium+ category: appliances+ tags:+ - control+ - axis:curved_organic+ prompt: a two-slot retro toaster with rounded chrome body, a lever on the side, a browning dial and+ a crumb tray+ must_have:+ - 2 bread slots on the top face+ - each slot about 0.13 m long and under 0.04 m wide+ - lever on one end face+ - round dial knob on the front face+ - rounded body with no sharp vertical edges+ - crumb tray seam line near the base+ - 4 small feet+ - body about 0.28 m wide and 0.19 m tall+ dimensions_m:+ width: 0.28+ depth: 0.17+ height: 0.19+- id: ctrl_med_hand_drill+ tier: medium+ category: tools+ tags:+ - control+ - axis:decomposition+ prompt: a cordless power drill with a pistol grip, trigger, battery pack in the base of the grip and+ a keyless chuck holding a drill bit+ must_have:+ - pistol-grip body with the grip angled back from vertical+ - trigger recessed at the top of the grip+ - battery block under the grip, wider than the grip+ - chuck at the front, about 0.04 m diameter+ - drill bit protruding about 0.08 m from the chuck+ - side vents on the motor housing+ - overall length about 0.22 m from chuck tip to body rear+ - overall height about 0.24 m from battery base to body top+ dimensions_m:+ length: 0.22+ height: 0.24+- id: mech_hard_coffee_grinder+ tier: hard+ category: mechanisms+ tags:+ - axis:decomposition+ prompt: 'a hand-crank coffee grinder: square wooden box body with a small front drawer, cast-iron dome+ top with an open hopper funnel, a conical burr visible inside the funnel throat, and a top crank arm+ with a wooden knob++ '+ must_have:+ - square wooden box body about 0.14 m wide and 0.12 m tall+ - front drawer about 0.05 m tall with a round knob, in the lower half of the box+ - drawer front sits flush in its opening with a visible seam+ - metal dome top mounted on the box+ - open hopper funnel about 0.08 m across at the rim+ - conical burr visible inside the funnel throat+ - central spindle about 0.01 m thick rising from the burr through the dome+ - crank arm about 0.11 m long attached to the spindle top+ - wooden knob on a short vertical pin at the crank's free end+ - adjustment nut on the spindle above the crank hub+ dimensions_m:+ width: 0.14+ depth: 0.14+ height: 0.3+- id: mech_hard_machinist_vise+ tier: hard+ category: tools+ tags:+ - axis:decomposition+ - axis:thin_features+ prompt: 'a machinist bench vise with an acme lead screw showing its thread helix, twin cylindrical guide+ rails, a swivel base with two lock handles, serrated jaw plates and a sliding tommy-bar handle++ '+ must_have:+ - fixed jaw and sliding jaw with a 0.05 m open gap between them+ - flat jaw plates about 0.10 m wide with serration grooves+ - lead screw between the jaws with at least 6 visible thread turns+ - 2 cylindrical guide rails flanking the screw+ - sliding jaw rides on the rails and the screw+ - tommy bar about 0.16 m long through the screw head, with ball ends+ - swivel base disc under the vise body+ - 2 lock handles on the swivel base rim+ - mounting flange with 3 bolt holes+ - overall body length about 0.30 m+ dimensions_m:+ length: 0.3+ width: 0.14+ height: 0.18+- id: mech_hard_drill_press+ tier: hard+ category: tools+ tags:+ - axis:decomposition+ prompt: 'a benchtop drill press: cast base, vertical column, adjustable work table with a crank, head+ casting with a belt housing, chuck with a bit over the table, three-spoke quill feed lever and a depth-stop+ rod++ '+ must_have:+ - flat base about 0.30 m by 0.20 m with 2 mounting slots+ - vertical column about 0.05 m diameter rising from the base+ - work table clamped to the column between base and head+ - table raise crank handle below the table+ - head casting at the top of the column+ - belt housing on top of the head with a lid seam+ - cylindrical motor behind the column, connected to the head+ - chuck about 0.04 m diameter under the head with a drill bit pointing down+ - bit tip aligned over the table center+ - quill feed lever with 3 spokes ending in ball knobs+ - depth-stop rod with 2 nuts beside the quill+ - overall height about 0.85 m+ dimensions_m:+ height: 0.85+ depth: 0.45+- id: mech_hard_pitcher_pump+ tier: hard+ category: garden+ tags:+ - axis:decomposition+ - axis:curved_organic+ prompt: 'a cast-iron pitcher water pump on a wooden platform: fluted cylindrical body, curved downward+ spout, long lever handle raised mid-stroke, connecting link to the piston rod, domed cap and a bolted+ base flange++ '+ must_have:+ - cylindrical pump body about 0.09 m diameter+ - at least 6 vertical flutes on the body+ - base flange with 4 bolt heads+ - square wooden platform under the flange+ - curved spout leaving the body two-thirds up, opening facing down+ - domed cap on top of the body+ - lever handle about 0.40 m long, raised at an angle+ - handle pivots in a clevis bracket at the top rear of the body+ - connecting link from the handle to a piston rod entering the cap+ - overall height about 0.65 m+ dimensions_m:+ height: 0.65+- id: thin_hard_garden_arch+ tier: hard+ category: garden+ tags:+ - axis:thin_features+ prompt: 'a wrought-iron garden arch: two lattice side frames, a semicircular arch head filled with S-scrolls,+ spear-point finials and a tie bar at the spring line++ '+ must_have:+ - 2 side frames, each of 2 uprights about 0.01 m thick+ - diamond lattice bars between the uprights of each side frame+ - semicircular arch head spanning about 1.2 m between the side frames+ - at least 6 S-scrolls filling the arch head+ - at least 4 scrolls in each side frame+ - 3 spear-point finials on top of the arch+ - horizontal tie bar connecting both sides at the spring line+ - small ground plates under all 4 upright feet+ - overall height about 2.3 m+ - 'all bars slender: no member thicker than 0.02 m'+ dimensions_m:+ width: 1.4+ depth: 0.5+ height: 2.3+- id: thin_hard_violin+ tier: hard+ category: musical_instruments+ tags:+ - axis:thin_features+ - axis:curved_organic+ prompt: 'a violin: hourglass body with C-bout waists, arched top with two f-holes, a bridge, four strings+ from tailpiece over the bridge along the fingerboard to the pegbox, a scroll head, four tuning pegs+ and a chin rest++ '+ must_have:+ - hourglass body with 2 concave C-bout waist cutaways+ - body length about 0.36 m+ - top plate visibly arched, not flat+ - 2 f-holes through the top, one each side of the bridge+ - bridge about 0.03 m tall standing upright between the f-holes+ - 4 strings running in parallel from tailpiece to pegbox+ - strings pass over the bridge with a visible break angle+ - black fingerboard extending over the body without touching it+ - spiral scroll at the head+ - 4 tuning pegs, 2 per side of the pegbox+ - tailpiece anchored toward the end button+ - chin rest beside the tailpiece+ - total length about 0.60 m+ dimensions_m:+ length: 0.6+ width: 0.21+ height: 0.09+- id: thin_hard_penny_farthing+ tier: hard+ category: vehicles+ tags:+ - axis:thin_features+ - axis:curved_organic+ prompt: 'a penny-farthing bicycle: huge spoked front wheel with pedal cranks on its hub, small spoked+ rear wheel, curved backbone tube, front fork, handlebar and a sprung leather saddle++ '+ must_have:+ - front wheel about 1.3 m diameter with at least 24 thin spokes+ - rear wheel about 0.40 m diameter with at least 12 spokes+ - tyres no thicker than 0.03 m+ - backbone tube arcing from the fork crown down to the rear wheel fork+ - front fork straddling the big wheel to its hub+ - handlebar with 2 grips above the front hub+ - 2 pedal cranks on the front hub with pedals, opposed at 180 degrees+ - saddle on the backbone behind the handlebar, on a visible spring+ - small mounting step peg on the backbone above the rear wheel+ - both wheels touch the ground plane+ dimensions_m:+ length: 1.7+ height: 1.5+- id: thin_hard_birdcage+ tier: hard+ category: household+ tags:+ - axis:thin_features+ prompt: 'a round wire birdcage with an onion-dome top, a hanging ring, a wire door with a latch, a trapeze+ perch hanging inside, a straight perch, two feed cups and a base tray with a rolled rim++ '+ must_have:+ - round cage about 0.35 m diameter+ - at least 20 vertical wires around the circumference+ - wires about 0.003 m thick+ - 3 horizontal rings tying the wires+ - onion-dome top converging to a finial+ - hanging ring at the very top+ - rectangular door frame set into the wires with a small latch+ - trapeze perch hanging inside from the dome on 2 wires+ - straight perch spanning the cage between opposite walls+ - 2 feed cups mounted at a ring+ - solid base tray with a rolled rim+ - overall height about 0.55 m+ dimensions_m:+ height: 0.55+- id: org_hard_bonsai_rock+ tier: hard+ category: plants+ tags:+ - axis:curved_organic+ prompt: 'a root-over-rock bonsai: a shallow oval pot with soil, a rounded rock gripped by exposed roots,+ a tapering trunk with two pronounced curves, and five distinct flattened foliage pads++ '+ must_have:+ - shallow oval pot about 0.30 m long with a rim and small feet+ - soil surface inside the pot+ - rounded rock about 0.12 m tall sitting on the soil+ - at least 4 roots gripping over the rock faces and entering the soil+ - trunk rising from the rock top with at least 2 pronounced curves+ - trunk tapers from about 0.04 m at the base to under 0.015 m at the apex+ - 5 distinct flattened foliage pads with gaps between them+ - lowest branch extends beyond the pot rim+ - no foliage pad touches the soil or the rock+ - overall height about 0.50 m+ dimensions_m:+ height: 0.5+ length: 0.45+- id: org_hard_seated_cat+ tier: hard+ category: animals+ tags:+ - axis:curved_organic+ prompt: 'a domestic cat sitting upright: haunches on the ground, straight vertical front legs, tail+ curled around beside the front paws, short muzzle and triangular ears++ '+ must_have:+ - upright seated pose with haunches on the ground and chest raised+ - 2 straight front legs, nearly vertical+ - front paws on the ground between the haunches+ - hind legs folded with thighs bulging wider than the chest+ - tail about 0.30 m long curling around to lie beside the front paws+ - head with a short muzzle, 2 eyes and a nose+ - 2 triangular ears about 0.04 m tall+ - back forms a smooth continuous curve from neck to tail base+ - silhouette narrows from haunches up to the shoulders+ - overall height about 0.35 m+ dimensions_m:+ height: 0.35+ depth: 0.25+- id: org_hard_octopus_rock+ tier: hard+ category: animals+ tags:+ - axis:curved_organic+ - axis:thin_features+ prompt: 'an octopus draped over a low rock: bulbous mantle, raised eyes, eight tapering arms curling+ in different directions, with sucker discs visible along the underside of at least three arms++ '+ must_have:+ - bulbous mantle about 0.20 m long+ - 2 raised eye bumps on the head+ - 8 arms radiating from the head+ - each arm tapers from about 0.05 m at the base to a point+ - arm tips curl, at least 4 curling in different directions+ - sucker discs along the underside of at least 3 arms+ - low rock under the body, arms draping over its sides+ - at least 2 arms touch the ground plane+ - overall arm spread about 0.8 m+ dimensions_m:+ length: 0.8+ height: 0.3+- id: arch_hard_bay_window+ tier: hard+ category: buildings+ tags:+ - axis:arch_detail+ - axis:decomposition+ prompt: 'a Victorian bay window facade section: a three-sided bay projecting from a clapboard wall,+ a sash window with divided panes in each face, panel aprons, corner pilasters, scroll brackets and+ a dentil cornice under a faceted roof cap++ '+ must_have:+ - three-sided bay projecting from a flat wall section+ - center face wider than the 2 angled side faces+ - side faces angled about 45 degrees from the wall+ - sash window in each of the 3 faces+ - upper sash of each window divided into at least 4 panes by muntins+ - continuous sill band under all 3 windows+ - recessed panel apron under each window+ - pilasters at the 4 vertical bay corners+ - cornice with at least 10 dentil blocks across the bay+ - at least 4 scroll brackets under the bay roof+ - three-faceted roof cap over the bay+ - horizontal siding boards on the flat wall section+ dimensions_m:+ width: 3.4+ height: 3.2+ depth: 1.0+- id: arch_hard_spiral_stair+ tier: hard+ category: buildings+ tags:+ - axis:arch_detail+ - axis:thin_features+ prompt: 'a cast-iron spiral staircase: central column, wedge treads winding a full turn, a baluster+ at each tread''s outer edge, a continuous helical handrail and a railed top landing++ '+ must_have:+ - central column about 0.10 m diameter+ - at least 12 wedge-shaped treads attached to the column+ - treads climb evenly through at least 360 degrees total rotation+ - 'open risers: visible gaps between consecutive treads'+ - 1 baluster at the outer edge of each tread+ - continuous helical handrail on top of the balusters+ - top landing platform with its own railing+ - base plate at the bottom of the column+ - each tread about 0.70 m from column to outer edge+ - overall height about 2.7 m+ dimensions_m:+ height: 2.7+ width: 1.6+- id: arch_hard_teahouse+ tier: hard+ category: buildings+ tags:+ - axis:arch_detail+ prompt: 'a Japanese teahouse: raised wooden floor on short posts, a veranda strip along the front, sliding+ shoji panels with visible grids, corner posts, and a hipped roof with deep eaves showing exposed rafter+ tails++ '+ must_have:+ - floor platform raised about 0.5 m on short posts+ - wooden veranda strip along the front edge+ - at least 3 sliding shoji panels in the front wall+ - each shoji panel gridded into at least 12 rectangles+ - square corner posts at the 4 corners+ - hipped roof with eaves overhanging about 0.6 m+ - at least 8 exposed rafter tails visible under each eave side+ - ridge cap along the roof top+ - solid back and side walls contrasting with the shoji front+ - step stone below the veranda+ dimensions_m:+ width: 3.5+ depth: 3.0+ height: 3.0+- id: comp_hard_drafting_table+ tier: hard+ category: furniture+ tags:+ - axis:multi_material+ - axis:decomposition+ prompt: 'a drafting table setup: tilted drawing board on a trestle frame with quadrant tilt brackets,+ a parallel ruler riding on side rails, a pen tray, a sheet of paper held by two corner clips, a clamp-on+ two-arm lamp aimed at the board, and a stool with a footring++ '+ must_have:+ - drawing board about 1.0 m by 0.7 m tilted about 20 degrees+ - parallel ruler bar spanning the board width+ - ruler rides on a rail or wire at each side edge+ - pen tray along the bottom edge of the board+ - trestle legs with a cross stretcher+ - quadrant tilt bracket with a knob at each of the board's 2 sides+ - sheet of paper on the board held by clips at 2 corners+ - lamp clamped to a top corner of the board+ - lamp has 2 arm segments and a shade aimed down at the board+ - stool with a round seat about 0.65 m high+ - stool footring between its legs+ - board top edge about 1.1 m above the floor+ dimensions_m:+ width: 1.1+ depth: 0.9+ height: 1.2+- id: comp_hard_telescope_tripod+ tier: hard+ category: props+ tags:+ - axis:multi_material+ - axis:decomposition+ prompt: 'a brass refractor telescope on a wooden tripod: main tube with an objective ring, a thinner+ draw tube with an eyepiece cup, a clamping collar and cradle on a pan-tilt head, three splayed legs+ joined by a spreader tray, metal feet++ '+ must_have:+ - main tube about 0.7 m long and 0.08 m diameter+ - objective ring at the front of the tube+ - draw tube about 0.05 m diameter extending from the rear, thinner than the main tube+ - eyepiece cup at the draw tube end+ - clamping collar around the main tube's balance point+ - cradle connecting the collar to a pan-tilt head+ - tension knob on the head+ - 3 wooden legs splayed outward, each about 1.2 m long+ - triangular spreader tray connecting the legs at about 0.7 m height+ - metal feet at the leg tips+ - tube tilted upward between 20 and 40 degrees+ - eyepiece height about 1.5 m above the ground+ dimensions_m:+ height: 1.7+- id: furn_easy_stool+ tier: easy+ category: furniture+ prompt: a simple three-legged round wooden stool, 45 cm tall+ must_have:+ - round seat+ - three splayed legs of equal length+ - legs touch the ground+ dimensions_m:+ height: 0.45+- id: furn_hard_rolltop_desk+ tier: hard+ category: furniture+ prompt: a writing desk with two pedestals of three drawers each, a central knee-hole, a leather inlay+ top and brass drawer pulls+ must_have:+ - two drawer pedestals+ - three drawers per pedestal with visible fronts+ - knee-hole gap+ - brass pull handles+ - desk top overhangs+- id: veh_easy_toy_car+ tier: easy+ category: vehicles+ prompt: a chunky toy car with a rounded body and four thick wheels+ must_have:+ - one-piece rounded body+ - four wheels touching the ground+ - wheels protrude sideways+- id: veh_hard_tractor+ tier: hard+ category: vehicles+ prompt: a farm tractor with large rear tyres with treads, small front wheels, an exposed engine block,+ exhaust stack, driver seat, steering wheel and a roll bar+ must_have:+ - rear wheels much larger than front wheels+ - tyre tread detail+ - exhaust stack+ - seat and steering wheel+ - roll bar+ - engine hood+- id: tool_easy_hammer+ tier: easy+ category: tools+ prompt: a claw hammer with a wooden handle and a steel head+ must_have:+ - cylindrical handle+ - steel head with a flat face+ - forked claw on the back+- id: tool_hard_bench_vise+ tier: hard+ category: tools+ prompt: a cast-iron bench vise with fixed and sliding jaws, a lead screw with a tommy bar handle, guide+ rails, a small anvil face and mounting base with bolt holes+ must_have:+ - two jaws with a visible gap+ - sliding jaw on guide rails+ - lead screw with sliding T handle+ - anvil flat+ - base with bolt holes+- id: appl_easy_kettle+ tier: easy+ category: appliances+ prompt: an electric kettle with a spout, a lid and a side handle on a round base+ must_have:+ - pot body+ - pouring spout+ - handle opposite the spout+ - lid on top+ - round base+- id: appl_hard_espresso+ tier: hard+ category: appliances+ prompt: an espresso machine with a group head and portafilter, a steam wand, a drip tray with grate,+ a water tank at the back, and a top-mounted cup warmer with a pressure gauge on the front panel+ must_have:+ - portafilter with handle under a group head+ - steam wand on the side+ - drip tray with grate+ - water tank+ - pressure gauge+ - cup warmer rail+- id: plant_easy_cactus+ tier: easy+ category: plants+ prompt: a potted barrel cactus in a terracotta pot with soil+ must_have:+ - ribbed round cactus body+ - terracotta pot with rim+ - soil surface+- id: plant_med_palm+ tier: medium+ category: plants+ prompt: a small potted palm with a textured trunk and eight arching fronds made of many leaflets+ must_have:+ - textured trunk+ - at least eight fronds+ - fronds arch downward+ - leaflets along each frond+ - pot+- id: plant_hard_bonsai+ tier: hard+ category: plants+ prompt: a bonsai tree with a twisted thick trunk, three layered foliage pads, exposed roots over a rock+ and a shallow oval ceramic tray with moss+ must_have:+ - twisted trunk with taper+ - at least three distinct foliage pads+ - exposed roots gripping a rock+ - shallow oval tray+ - moss surface+- id: bld_easy_shed+ tier: easy+ category: buildings+ prompt: a small garden shed with a pitched roof, one door and one window+ must_have:+ - box walls+ - pitched roof overhang+ - door on front+ - window+- id: bld_med_lighthouse+ tier: medium+ category: buildings+ prompt: a tapered lighthouse with red and white bands, a gallery with railing, a glass lantern room+ and a domed cap, on a rocky base+ must_have:+ - tapered tower+ - alternating bands+ - gallery walkway with railing+ - glass lantern room+ - dome+ - rock base+- id: bld_hard_windmill+ tier: hard+ category: buildings+ prompt: a Dutch windmill with an octagonal tapered body, thatched cap, four lattice sails, a balcony+ stage with railing, a door, and two shuttered windows+ must_have:+ - octagonal tapered body+ - cap+ - four lattice sails in a cross+ - balcony with railing+ - door+ - shuttered windows+- id: toy_easy_blocks+ tier: easy+ category: toys+ prompt: a stack of five alphabet blocks of decreasing size in a slightly twisted tower+ must_have:+ - five cubes+ - decreasing size upward+ - each slightly rotated+ - all stacked touching+- id: toy_med_rocking_horse+ tier: medium+ category: toys+ prompt: a wooden rocking horse with curved rockers, a saddle, handles and a mane+ must_have:+ - horse body with head and four legs+ - two curved rockers+ - saddle+ - handle grips+ - mane+- id: toy_hard_train+ tier: hard+ category: toys+ prompt: a toy steam locomotive with a boiler, smokestack, cab with windows, cowcatcher, six driving+ wheels with connecting rods, and a coal tender with four wheels+ must_have:+ - cylindrical boiler+ - smokestack+ - cab+ - cowcatcher+ - six large wheels with rods+ - separate tender with coal load and four wheels+- id: mus_easy_drum+ tier: easy+ category: musical_instruments+ prompt: a snare drum with tension rods and two drumsticks resting on top+ must_have:+ - cylindrical shell+ - top head+ - at least eight tension rods around the rim+ - two sticks+- id: mus_med_acoustic_guitar+ tier: medium+ category: musical_instruments+ prompt: an acoustic guitar with a figure-eight body, sound hole, bridge, fretted neck, headstock with+ six tuning pegs and six strings+ must_have:+ - figure-eight body+ - sound hole+ - bridge+ - neck with frets+ - headstock+ - six tuning pegs+ - six strings+- id: mus_hard_drum_kit+ tier: hard+ category: musical_instruments+ prompt: a five-piece drum kit with a bass drum with pedal, snare on a stand, two rack toms, a floor+ tom, a hi-hat and two cymbals on stands, and a throne+ must_have:+ - bass drum with pedal+ - snare on stand+ - two mounted toms+ - floor tom with legs+ - hi-hat with two cymbals+ - two crash/ride cymbals on stands+ - drum thronediff --git a/eval/llm/batteries/complexity_v3.yaml b/eval/llm/batteries/complexity_v3.yamlnew file mode 100644index 00000000..4e9317e1--- /dev/null+++ b/eval/llm/batteries/complexity_v3.yaml@@ -0,0 +1,249 @@+# complexity_v3 — the COMPLEXITY LADDER battery.+#+# Purpose: make the framework's complexity CEILING measurable over time. The other+# batteries ask "can it build this?"; this one asks "how much artifact can it build+# before the score falls over?". The 12 prompts climb one axis on purpose — the+# number of nameable parts, from a 3-part stool to a 25-part beam engine with a+# visible mechanism — while the secondary axes (thin features, repetition, hollow+# sections, mouldings, mechanism) are switched on in a documented order.+#+# Every prompt declares `expected_complexity: [lo, hi]`, the band the BUILT artifact's+# objective complexity index (codeverse3d/spatial/complexity.py, docs/COMPLEXITY.md)+# should land in. A run that scores well but lands BELOW its band did the easy thing+# instead of the asked thing — the report calls that out as LOW, and it is the failure+# mode this battery exists to catch. `expected_complexity` is not read by run_bench+# (extra keys are ignored); `bench/complexity_report.py --battery` reads it.+#+# python -m bench.run_bench bench/prompts/complexity_v3.yaml --out bench/out/complexity_v3 \+# --generator agy:gemini-3.6-flash --judge gemini:gemini-3.1-pro-preview+# python bench/complexity_report.py bench/out/complexity_v3 --battery bench/prompts/complexity_v3.yaml+#+# Ladder (measured index of the nearest corpus run is in brackets):+# a-tier 3-6 parts solid primitives + one curve 0.25-0.52 [furn_easy_stool 0.40]+# b-tier 8-12 parts drawers, panes, first mechanism 0.45-0.66 [ctrl_med_dining_chair 0.54]+# c-tier 14-18 parts repetition, spokes, mouldings 0.55-0.75 [comp_hard_telescope 0.65]+# d-tier 20-25 parts thin repetition + working mechanism 0.62-0.85 [arch_hard_spiral_stair 0.84]+name: complexity_v3+track: static_object+language: blender+description: >+ 12 static-object prompts on one deliberate complexity ladder (3 → 25 nameable parts,+ with thin features, repeated elements, hollow sections, mouldings and a visible+ mechanism switched on in order). Each prompt states the objective complexity band+ its built artifact is expected to reach, so the ceiling can be tracked over time+ instead of argued about.+prompts:+ # ------------------------------------------------------------------ a-tier (3-6 parts)+ - id: cx_a03_milking_stool+ tier: easy+ category: furniture+ tags: ["axis:parts:3", "ladder:a"]+ expected_complexity: [0.25, 0.42]+ prompt: a three-legged wooden milking stool with a round dished seat and splayed round legs+ must_have:+ - "round seat about 0.30 m across"+ - "seat top dished (concave) by about 0.008 m, not flat"+ - "exactly three round legs"+ - "legs splay outward so the floor circle is wider than the seat"+ - "seat surface about 0.45 m above the floor"+ - "leg diameter about 0.04 m at the seat"+ - "all three feet touch the ground plane"+ - "chamfered seat edge, chamfer about 0.006 m"+ dimensions_m: {width: 0.36, depth: 0.36, height: 0.45}+ - id: cx_a05_wall_shelf+ tier: easy+ category: furniture+ tags: ["axis:parts:5", "ladder:a"]+ expected_complexity: [0.30, 0.46]+ prompt: an oak wall shelf on two cast-iron L brackets, with a moulded front edge and a small back lip+ must_have:+ - "shelf board about 0.80 m wide and 0.22 m deep"+ - "board thickness about 0.025 m"+ - "moulded (profiled, not square) front edge"+ - "back lip about 0.03 m tall along the rear edge"+ - "exactly two L brackets under the board"+ - "each bracket has a vertical arm, a horizontal arm and a diagonal brace"+ - "four screw heads visible on the bracket vertical arms"+ - "brackets set in about 0.10 m from each end"+ dimensions_m: {width: 0.80, depth: 0.24, height: 0.20}+ - id: cx_a06_watering_can+ tier: easy+ category: tools+ tags: ["axis:parts:6", "axis:hollow", "axis:curved", "ladder:a"]+ expected_complexity: [0.38, 0.54]+ prompt: a galvanised steel watering can with a long tapered spout, a rose head, an arched carry handle and a rear grip handle+ must_have:+ - "tapered body about 0.26 m tall with a visibly open top"+ - "visible wall thickness of about 0.003 m at the open rim"+ - "long spout about 0.30 m long, rising from the body base to above the rim"+ - "perforated rose head about 0.07 m across on the spout end"+ - "arched carry handle over the top"+ - "separate rear grip handle"+ - "rolled bead about 0.006 m thick around the body rim"+ - "body diameter about 0.22 m at the base"+ dimensions_m: {width: 0.46, depth: 0.22, height: 0.34}+ # ------------------------------------------------------------------ b-tier (8-12 parts)+ - id: cx_b08_writing_desk+ tier: medium+ category: furniture+ tags: ["axis:parts:8", "axis:moulding", "ladder:b"]+ expected_complexity: [0.44, 0.60]+ prompt: a walnut writing desk with one central drawer, turned tapered legs, a moulded top edge and a leather inlay panel+ must_have:+ - "desk top about 1.20 m by 0.60 m"+ - "top surface about 0.75 m above the floor"+ - "moulded (profiled) top edge on all four sides, profile depth about 0.012 m"+ - "leather inlay panel recessed about 0.003 m into the top"+ - "one drawer in the apron with a visible drawer front"+ - "drawer pull about 0.09 m wide, modelled as separate geometry"+ - "four turned legs, tapering from about 0.06 m to 0.035 m"+ - "apron rails between the legs on three sides"+ dimensions_m: {width: 1.20, depth: 0.60, height: 0.75}+ - id: cx_b10_gas_lantern+ tier: medium+ category: lighting+ tags: ["axis:parts:10", "axis:thin_features", "axis:hollow", "ladder:b"]+ expected_complexity: [0.48, 0.64]+ prompt: a Victorian cast-iron gas lantern head on a fluted post - four glass panes in a tapered glazing frame, a vented crown cap and a finial+ must_have:+ - "fluted post about 2.2 m tall"+ - "tapered four-sided glazing frame with visible corner bars under 0.02 m thick"+ - "four separate glass panes about 0.004 m thick set inside the frame"+ - "vented crown cap above the glazing"+ - "finial about 0.12 m tall on top of the cap"+ - "flared collar where the head meets the post"+ - "square base plate about 0.35 m across"+ - "ladder rest bar projecting from the post below the head"+ dimensions_m: {width: 0.42, depth: 0.42, height: 2.60}+ - id: cx_b12_sewing_machine+ tier: medium+ category: machines+ tags: ["axis:parts:12", "axis:mechanism", "ladder:b"]+ expected_complexity: [0.52, 0.68]+ prompt: a cast-iron hand-crank sewing machine on a wooden base - curved arm, needle bar, presser foot, spool pin, hand wheel and drive belt+ must_have:+ - "wooden base about 0.40 m by 0.20 m"+ - "curved cast-iron arm reaching about 0.22 m forward over the bed"+ - "needle bar about 0.05 m long hanging from the arm head with a needle below it"+ - "presser foot about 0.02 m long on the bed under the needle"+ - "hand wheel about 0.09 m across on the right of the pillar"+ - "crank handle on the hand wheel, offset from its centre"+ - "spool pin about 0.05 m tall standing on top of the arm"+ - "bobbin winder on the pillar"+ - "flat bed plate with a visible needle plate"+ - "tension discs on the arm face"+ dimensions_m: {width: 0.40, depth: 0.20, height: 0.32}+ # ------------------------------------------------------------------ c-tier (14-18 parts)+ - id: cx_c14_ship_binnacle+ tier: hard+ category: instruments+ tags: ["axis:parts:14", "axis:repetition", "axis:curved", "ladder:c"]+ expected_complexity: [0.55, 0.72]+ prompt: a brass ship's binnacle with a domed hood, a compass bowl on gimbals, two spherical correctors on side arms and a fluted pedestal on a stepped base+ must_have:+ - "fluted pedestal about 1.1 m tall"+ - "domed hood about 0.30 m across with a viewing window opening"+ - "compass bowl about 0.20 m across inside the hood"+ - "two gimbal rings under 0.01 m thick around the bowl"+ - "two spherical correctors, one on each side arm"+ - "side arms clamped to the pedestal at the same height"+ - "stepped circular base about 0.45 m across"+ - "at least eight flutes around the pedestal"+ - "hinged lamp housing on the hood"+ - "spherical corrector diameter about 0.16 m"+ dimensions_m: {width: 0.90, depth: 0.45, height: 1.30}+ - id: cx_c16_apothecary_cabinet+ tier: hard+ category: furniture+ tags: ["axis:parts:16", "axis:repetition", "axis:moulding", "ladder:c"]+ expected_complexity: [0.56, 0.74]+ prompt: an apothecary cabinet - a twelve-drawer grid in a moulded carcass on a plinth, each drawer with a turned knob and a brass label frame+ must_have:+ - "carcass about 0.90 m wide and 1.10 m tall"+ - "twelve drawer fronts in a 4 by 3 grid"+ - "each drawer front is inset, with a visible reveal about 0.003 m around it"+ - "a turned knob about 0.03 m across on every drawer front"+ - "a rectangular label frame on every drawer front"+ - "moulded cornice about 0.05 m tall along the top"+ - "plinth base about 0.10 m tall"+ - "vertical divider between drawer columns"+ - "carcass depth about 0.35 m"+ - "drawer front height about 0.20 m"+ dimensions_m: {width: 0.90, depth: 0.35, height: 1.10}+ - id: cx_c18_orrery+ tier: hard+ category: instruments+ tags: ["axis:parts:18", "axis:mechanism", "axis:thin_features", "ladder:c"]+ expected_complexity: [0.60, 0.78]+ prompt: a brass tabletop orrery - a central sun on a spindle, three planet arms of different lengths each carrying a sphere, a moon on a sub-arm, an exposed gear train and a turned wooden base+ must_have:+ - "turned wooden base about 0.30 m across"+ - "central spindle carrying a sun sphere about 0.06 m across"+ - "exactly three planet arms of different lengths"+ - "a planet sphere at the end of each arm, all different sizes"+ - "a moon on a short sub-arm off the outermost planet"+ - "at least three toothed gears, the largest about 0.10 m across, visible under the arms"+ - "gear teeth modelled as geometry, not painted"+ - "arms under 0.008 m thick"+ - "a crank handle about 0.05 m long on the side of the gear housing"+ - "engraved zodiac ring about 0.015 m wide around the base rim"+ dimensions_m: {width: 0.40, depth: 0.40, height: 0.28}+ # ------------------------------------------------------------------ d-tier (20-25 parts)+ - id: cx_d20_floor_loom+ tier: hard+ category: machines+ tags: ["axis:parts:20", "axis:thin_features", "axis:repetition", "ladder:d"]+ expected_complexity: [0.62, 0.82]+ prompt: a wooden four-shaft floor loom - side frames, warp and cloth beams, four heddle shafts hung on cords, a swinging beater with a reed, two treadles and a bench+ must_have:+ - "two side frames about 1.1 m tall joined by cross rails"+ - "warp beam at the back and cloth beam at the front, both round and about 0.08 m across"+ - "exactly four heddle shafts hanging in the middle of the frame"+ - "at least six vertical heddle wires per shaft, under 0.004 m thick"+ - "swinging beater about 0.7 m long pivoted at the top, carrying a reed"+ - "at least twelve reed dents modelled in the beater"+ - "two treadles about 0.6 m long at floor level, connected upward by cords"+ - "ratchet wheel and pawl on the cloth beam"+ - "separate bench in front of the loom"+ - "overall depth about 1.2 m"+ dimensions_m: {width: 1.10, depth: 1.60, height: 1.20}+ - id: cx_d22_letterpress+ tier: hard+ category: machines+ tags: ["axis:parts:22", "axis:mechanism", "axis:repetition", "ladder:d"]+ expected_complexity: [0.64, 0.84]+ prompt: a cast-iron platen letterpress - a round ink disc, two ink rollers on a saddle, a hinged platen, a chase bed, a treadle, a flywheel with a crank and a scrolled cast frame on legs+ must_have:+ - "scrolled cast-iron side frames about 1.4 m tall"+ - "round ink disc about 0.35 m across at the top, tilted forward"+ - "two ink rollers about 0.05 m across on a roller saddle between disc and bed"+ - "hinged platen about 0.25 m by 0.20 m that faces the chase bed"+ - "chase bed with a rectangular chase about 0.20 m by 0.15 m inside it"+ - "flywheel about 0.55 m across on the right side"+ - "crank arm connecting the flywheel to the platen linkage"+ - "treadle bar about 0.5 m long at floor level linked upward by a rod"+ - "at least six spokes in the flywheel"+ - "gripper fingers on the platen face"+ - "throw-off lever on the frame"+ dimensions_m: {width: 0.90, depth: 1.20, height: 1.60}+ - id: cx_d25_beam_engine+ tier: hard+ category: machines+ tags: ["axis:parts:25", "axis:mechanism", "axis:repetition", "axis:thin_features", "ladder:d"]+ expected_complexity: [0.68, 0.88]+ prompt: a model Watt beam engine on a cast base - a rocking overhead beam on an A-frame column, piston cylinder with rod and crosshead, connecting rod to a crank, a spoked flywheel, a centrifugal governor with two balls, and exposed valve linkage+ must_have:+ - "cast base plate about 0.60 m by 0.30 m"+ - "A-frame column carrying the beam pivot about 0.55 m above the base"+ - "rocking beam about 0.45 m long with an eye at each end"+ - "vertical piston cylinder about 0.09 m across with a visible rod emerging from its top"+ - "crosshead about 0.05 m wide joining the piston rod to the beam end"+ - "connecting rod from the other beam end down to a crank"+ - "spoked flywheel about 0.28 m across with at least six spokes"+ - "crankshaft supported in two pedestal bearings"+ - "centrifugal governor about 0.15 m tall with two arms and two balls about 0.03 m across"+ - "governor driven by a belt or gear off the crankshaft"+ - "valve rod running alongside the cylinder"+ - "parallel motion linkage between the beam end and the piston rod"+ - "flywheel rim under 0.03 m thick"+ dimensions_m: {width: 0.60, depth: 0.30, height: 0.75}diff --git a/eval/llm/batteries/fancy_v1_blender.yaml b/eval/llm/batteries/fancy_v1_blender.yamlnew file mode 100644index 00000000..d20e853a--- /dev/null+++ b/eval/llm/batteries/fancy_v1_blender.yaml@@ -0,0 +1,286 @@+# fancy_v1 — blender slice (10 prompts). Complex, picture-worthy static objects, every+# prompt written from how the real object is actually built: load-bearing parts, how they+# attach, proportions in metres, where materials change, and the 2-4 details that make it+# read as THAT object. Objects chosen for a distinctive silhouette, real internal structure,+# repeats-and-mirrors (spokes, slats, teeth, fins) and at least one hollow. Blender slice+# carries the organic and bevelled forms. All hard tier; ids fancy_bl_<snake>.+# Sibling files: fancy_v1_cadquery.yaml, fancy_v1_threejs.yaml.+name: fancy_v1_blender+track: static_object+language: blender+description: >+ fancy_v1 blender slice: 10 hard static objects with real construction described the way+ a product designer would (Windsor chair, harp, ship's wheel, carousel horse, orrery ...),+ each with a countable falsifiable must_have list and realistic overall dimensions.+prompts:+ # Steam-bent bow socketed into a saddled seat, turned spindles and splayed legs: organic+ # carving plus a large count of thin repeated parts that must all land in the seat.+ - id: fancy_bl_windsor_chair+ tier: hard+ category: furniture+ tags: [fancy_v1, hard]+ prompt: >+ a sack-back Windsor armchair in the traditional construction: a thick elm seat plank,+ about 0.55 m wide and 0.05 m thick, carved into a saddle with a raised pommel at the front; a steam-bent ash bow of round section about 0.02 m+ thick rising in a continuous arch whose two ends socket down into the seat behind the+ sitter; a flat steam-bent arm rail wrapping around at elbow height whose ends terminate+ in small hand rests, supported at the front by a turned arm post on each side; seven+ tapered spindles passing through holes in the arm rail up into the bow, plus four+ shorter spindles under each arm; four turned legs with baluster and ring profiles+ raked outward and splayed to the sides, socketed into the underside of the seat, tied+ by an H-stretcher of three turned rungs. Milk-paint black-green, worn through to pale wood on the arm tops.+ must_have:+ - "seat plank about 0.55 m wide with a visibly scooped saddle, not a flat slab"+ - "a continuous round-section bow arching over the back and entering the seat at exactly two points"+ - "a flat arm rail at about 0.68 m above the floor wrapping around both sides of the sitter"+ - "at least 7 long spindles pass through the arm rail and continue up into the bow"+ - "at least 4 short spindles under each arm between arm rail and seat"+ - "four turned legs raked outward so the feet sit wider than the seat"+ - "all four feet touch the ground plane"+ - "paint worn through to pale wood on the arm tops as a second material"+ dimensions_m: {width: 0.62, depth: 0.58, height: 1.02}+ # A concert harp: a real triangular load path (pillar, neck, soundboard-in-body) and 47+ # graduated strings in a plane, with coloured C and F strings and a hollow resonator.+ - id: fancy_bl_concert_harp+ tier: hard+ category: instruments+ tags: [fancy_v1, hard]+ prompt: >+ a concert pedal harp built as the real triangle: a fluted gilded column (the pillar)+ standing about 1.75 m tall at the front, a hollow tapered spruce sound body at the+ back, its rounded staved back closed and its flat soundboard facing the player,+ widening from about 0.10 m at the top to 0.50 m at the base; a curved maple neck+ arching from the top of the column down to the top of the body with the brass action+ plate running along its side carrying a row of tuning pins; 47 strings, all in one+ plane, strung from pins on the neck down to a straight rib on the soundboard centre,+ graduating from about 0.08 m long at the treble to 1.50 m long at the bass; every+ seventh string red (the Cs) and one in four blue (the Fs), the others natural gut colour; a base plate with seven pedals in a row and+ two rear feet; gold-leaf ornament on the column capital and the crown.+ must_have:+ - "a fluted front pillar about 1.75 m tall, standing vertically at the front"+ - "a hollow sound body at the back that widens toward the base and shows a rounded staved back"+ - "a curved neck joining the pillar top to the body top, forming a closed triangle"+ - "at least 40 strings, all lying in one plane, each attached at both ends"+ - "strings graduate in length from short at the treble end to long at the bass end"+ - "some strings coloured red and some blue among natural-coloured ones"+ - "seven pedals visible on the base plate"+ - "gold-coloured ornament on the column capital distinct from the wood body"+ dimensions_m: {width: 0.55, depth: 1.0, height: 1.85}+ # Ship's wheel on a binnacle: turned spokes through a laminated rim, a hub with a brass+ # boss, the wheel mounted on a pedestal that is itself a hollow box with a compass bowl.+ - id: fancy_bl_ships_wheel+ tier: hard+ category: vehicles+ tags: [fancy_v1, hard]+ prompt: >+ a ship's steering wheel on its binnacle: a mahogany rim about 1.20 m across built from+ a laminated ring of round section about 0.05 m thick, through which pass eight turned+ spokes; each spoke runs from the central hub, through the rim, and out beyond it as a+ turned handle with a bulbous knob, so the handles sit about 0.14 m proud of the rim;+ the hub is a thick disc with a domed brass boss at its centre; the wheel is mounted+ vertically on a horizontal brass axle that runs back into a hollow teak binnacle+ pedestal about 1.10 m tall, square in plan with chamfered corners, brass bands top+ and bottom; on top of the binnacle a brass compass bowl under a glazed hemispherical+ hood with a hinged lamp housing on the side; a brass grab rail around the pedestal top.+ must_have:+ - "a circular rim about 1.20 m in diameter with a round cross-section"+ - "exactly 8 spokes spaced evenly, each passing through the rim and continuing outward as a handle"+ - "each handle ends in a bulbous turned knob outside the rim"+ - "a thick hub disc at the centre with a domed brass boss"+ - "the wheel stands vertically in front of a square-plan pedestal about 1.10 m tall"+ - "a hemispherical glazed compass hood on top of the pedestal"+ - "brass bands and boss distinct in colour from the dark wood rim and spokes"+ - "the pedestal base sits flat on the ground plane"+ dimensions_m: {width: 1.35, depth: 0.75, height: 1.75}+ # Carousel horse: organic sculpted body in a gallop pose with a brass pole through it,+ # carved tack, glass jewels, a decorated saddle; the pole is the hard rigid reference.+ - id: fancy_bl_carousel_horse+ tier: hard+ category: toys+ tags: [fancy_v1, hard]+ prompt: >+ a carousel jumper horse in the full-gallop carving pose: the body about 1.50 m from+ nose to tail root, front legs tucked up under the chest, hind legs stretched straight+ back, the head turned slightly to the outside (the romance side) with an open mouth+ and flared nostrils, a carved flowing mane falling to one side and a long swept tail;+ a polished brass pole about 0.045 m thick passing vertically through the body just+ behind the withers and exiting under the belly, continuing down to a round base plate;+ carved tack in relief: a saddle with a raised cantle and a saddle blanket with a+ scalloped trailing edge, a breast strap, a bridle with a browband; the romance side+ dressed with at least six faceted glass jewels set in gilded rosettes along the saddle+ blanket and breast strap; painted a dapple white body with a gold-and-red saddle, a+ black hoof on each leg and a horseshoe on each hoof.+ must_have:+ - "a horse body about 1.50 m long with both front legs tucked up and both hind legs stretched back"+ - "no hoof touches the ground; the horse is held up only by the pole"+ - "a vertical brass pole passes through the body behind the withers and exits under the belly"+ - "the pole continues down to a round base plate on the ground"+ - "a saddle with a raised cantle and a blanket with a scalloped rear edge"+ - "at least 6 jewel-like faceted bosses along the saddle blanket or breast strap"+ - "a flowing mane and a long tail carved as separate masses from the body"+ - "at least 3 distinct paint colours: pale body, red or gold tack, black hooves"+ dimensions_m: {width: 0.55, depth: 1.6, height: 1.85}+ # Brass orrery: nested concentric arms at different heights, gear stack under the base,+ # spheres on stalks with a moon sub-arm, a scalloped skirt, hollow base with visible gears.+ - id: fancy_bl_orrery+ tier: hard+ category: instruments+ tags: [fancy_v1, hard]+ prompt: >+ a Georgian-style brass orrery: a round mahogany base about 0.45 m in diameter with a+ turned edge, carrying a brass calendar ring engraved with the zodiac around its rim on+ twelve small turned pillars; under the ring, in the hollow between base and ring, a+ visible stack of brass spur gears of decreasing size driving a central vertical shaft;+ at the top of the shaft a gilt sun ball about 0.03 m across on a short stalk; six planet+ arms of flat brass strip radiating at six different heights on nested concentric+ sleeves around the shaft, each arm ending in a polished sphere on a small vertical+ stalk, lengths increasing from 0.06 m for Mercury to 0.22 m for Saturn; the Earth arm+ carries a small sub-arm with a tiny moon and a tilted brass ring around the Earth ball;+ Saturn wears a thin flat ring; a crank handle on the side of the base drives the gears.+ must_have:+ - "a round wooden base about 0.45 m in diameter"+ - "a brass ring around the rim supported on at least 12 small pillars"+ - "at least 4 visible spur gears of different sizes in the hollow under the ring"+ - "exactly 6 radial arms at six different heights, each ending in a sphere on a stalk"+ - "arm lengths increase outward so the outermost sphere is farthest from the shaft"+ - "one arm carries a small sub-arm with a tiny moon beside its planet"+ - "the outermost sphere has a thin flat ring around it"+ - "brass parts distinct in colour from the dark wood base"+ dimensions_m: {width: 0.5, depth: 0.5, height: 0.42}+ # A pipe organ facade: a carved case with a hollow impost, graduated speaking pipes in+ # mitred flats and towers, a keyboard console; a hard count-and-graduation object.+ - id: fancy_bl_chamber_organ+ tier: hard+ category: instruments+ tags: [fancy_v1, hard]+ prompt: >+ a chamber pipe organ in an oak case about 2.20 m wide and 3.20 m tall: a lower case+ with two panelled doors, a projecting keydesk with two manuals of 56 keys each (the+ upper set back from the lower), a music stand and eight stop knobs on each side jamb; above a moulded impost, the facade of speaking pipes arranged as a central+ round tower of seven pipes in a semicircle, a flat of nine pipes on each side, and an+ outer tower of five pipes on each side, the towers standing proud of the flats; all+ facade pipes are polished tin cylinders with conical feet standing in a toe-board,+ a mouth with a cut-up and upper lip cut in each one about a fifth of the way up, and+ pipe lengths graduated symmetrically from tallest at the centre of each tower to+ shortest at the flat ends; carved pierced pipe shades filling the arch above each+ tower and flat; a crown cornice with a carved centre finial.+ must_have:+ - "an oak case about 2.20 m wide and 3.20 m tall standing on the ground plane"+ - "a central tower of pipes standing proud of two flanking flats"+ - "an outer tower of pipes at each end, making 5 pipe groups in a left-right mirror"+ - "at least 30 facade pipes in total, all polished tin colour"+ - "pipe heights within each group graduate symmetrically about the group centre"+ - "each pipe has a conical foot and a visible mouth notch in its lower fifth"+ - "two keyboards visible on a projecting keydesk, the upper set back from the lower"+ - "at least 8 stop knobs in a column on each side of the keyboards"+ dimensions_m: {width: 2.2, depth: 1.1, height: 3.2}+ # Victorian cast-iron street lamp: a fluted tapered column, a hollow four-sided lantern+ # with glazed panes and a finial, ladder bar, acanthus base; architectural detail object.+ - id: fancy_bl_gas_street_lamp+ tier: hard+ category: architecture+ tags: [fancy_v1, hard]+ prompt: >+ a Victorian cast-iron gas street lamp about 4.2 m tall: a square stepped plinth at+ the base about 0.45 m across, an octagonal pedestal with a cast acanthus-leaf collar+ at its top, a fluted tapered column of 16 flutes narrowing from 0.20 m to 0.12 m over+ its height with two moulded astragal rings, a short horizontal ladder bar with a+ ball at each end just below the lantern for the lamplighter; the lantern is a hollow+ four-sided tapered box, wider at the top, with a glazed pane in each face held by+ thin glazing bars, a sloped four-sided pyramidal roof with a vented cap and a pointed+ finial, and a copper gas-mantle burner visible inside the glass; the iron painted+ gloss black with the acanthus collar and finial picked out in gold leaf; the lantern+ glass slightly frosted.+ must_have:+ - "a stepped square plinth at the base sitting on the ground plane"+ - "a tapered column with at least 12 visible flutes"+ - "a cast leaf collar at the top of the pedestal, gold-coloured"+ - "a horizontal ladder bar with a ball at each end just below the lantern"+ - "a four-sided lantern that is wider at the top than at the bottom"+ - "a glazed pane in each of the four lantern faces"+ - "a pyramidal roof on the lantern with a finial at its apex"+ - "a burner element visible inside the lantern"+ dimensions_m: {width: 0.55, depth: 0.55, height: 4.2}+ # Brass microscope: a hollow drawtube, triple nosepiece with objectives, rack-and-pinion+ # focus, a mirror on a gimbal under a stage with clips, a horseshoe foot; optics object.+ - id: fancy_bl_brass_microscope+ tier: hard+ category: optics+ tags: [fancy_v1, hard]+ prompt: >+ a late-Victorian brass compound microscope: a heavy horseshoe-shaped foot, two+ trunnion pillars carrying a pivoted limb so the whole body tilts back, a square stage+ about 0.09 m across with two spring stage clips and a central aperture, beneath it a+ concave mirror about 0.04 m across on a gimbal fork that swings under the stage;+ above the stage a curved limb carrying the coarse focus rack, with a pair of large+ knurled coarse-focus knobs on a shaft either side and a smaller fine-focus knob on top+ of the limb; the body tube about 0.03 m in diameter and 0.16 m long, with an inner+ drawtube telescoped out of its top ending in an eyepiece cap; at the bottom of the+ body tube a revolving nosepiece carrying three objectives of different lengths in a+ rotating triple; lacquered polished brass throughout with black japanned foot and+ the limb, silvered mirror, and a black eyepiece cap.+ must_have:+ - "a horseshoe-shaped foot sitting flat on the ground plane"+ - "the body is pivoted between two pillars rising from the foot"+ - "a square stage with two stage clips on its top surface"+ - "a round mirror on a fork below the stage"+ - "a vertical body tube with a narrower drawtube extending from its top"+ - "exactly 3 objectives of different lengths hanging from a rotating nosepiece"+ - "two large matching knurled focus knobs on a common shaft either side of the limb"+ - "brass body distinct in colour from the black foot and the black eyepiece cap"+ dimensions_m: {width: 0.15, depth: 0.2, height: 0.36}+ # Espresso lever machine: a chrome dome boiler, a spring-lever group with a long handle,+ # portafilter locked in, a drip tray grille, gauge, eagle finial; appliance in curves.+ - id: fancy_bl_lever_espresso+ tier: hard+ category: appliances+ tags: [fancy_v1, hard]+ prompt: >+ a classic Italian lever espresso machine of the domestic kind: a polished chrome+ boiler cylinder about 0.16 m in diameter and 0.30 m tall standing on a cast alloy+ base with four rubber feet; a domed lid on the boiler with a brass eagle finial on+ top and a round pressure gauge with a white dial set into the front of the boiler; a+ brass group head cast into the front of the boiler near its base, with a long+ cylindrical spring-lever housing rising above it and a lever arm about 0.30 m long+ with a black bakelite knob angled up and back; a portafilter with a black handle+ locked into the group with two spouts pointing down at a cup; a steam wand of thin+ pipe with a black tap knob on the side of the boiler; a pull-out drip tray with a+ slotted chrome grille under the group; a sight glass tube on the boiler side showing+ water level.+ must_have:+ - "a vertical cylindrical boiler about 0.16 m in diameter with a domed lid"+ - "a finial on the top of the dome"+ - "a round gauge with a pale dial set into the boiler front"+ - "a group head on the boiler front near its base with a vertical spring housing above it"+ - "a lever arm at least 0.25 m long with a black knob angled up and back from the group"+ - "a portafilter locked into the group with two spouts pointing down"+ - "a drip tray with a slotted grille under the group"+ - "chrome boiler, brass group head and black knobs as three distinct materials"+ dimensions_m: {width: 0.24, depth: 0.32, height: 0.55}+ # Orbital shaker-incubator: a radiused housing with a hinged clear lid on gas struts, a+ # perforated platform, six clamped flasks with liquid; lab equipment with transparency.+ - id: fancy_bl_lab_orbital_shaker+ tier: hard+ category: lab_equipment+ tags: [fancy_v1, hard]+ prompt: >+ a benchtop orbital shaker-incubator as used in a microbiology lab: a squat white+ powder-coated steel housing about 0.60 m wide, 0.55 m deep and 0.45 m tall with+ radiused vertical corners and a recessed front control panel carrying a small dark+ display, a speed knob, a temperature knob and two rocker switches; a clear hinged+ acrylic lid on the top, propped open at about 60 degrees on two gas struts, revealing+ the platform inside: a flat aluminium tray on the orbital drive with a grid of holes+ and two rows of spring-wire flask clamps, each clamp a ring of curved spring fingers,+ holding six 250 ml conical Erlenmeyer flasks with orange screw caps and pale broth+ inside; a stainless handle on the lid front edge; four levelling feet; a louvred+ vent strip on each side of the housing; a coiled black mains lead at the back.+ must_have:+ - "a white boxy housing about 0.60 m wide with rounded vertical corners"+ - "a transparent lid hinged at the back, propped open on two struts"+ - "a flat platform tray inside with a visible grid of holes"+ - "exactly 6 conical flasks standing on the platform in two rows"+ - "each flask is held by a ring of spring fingers around its base"+ - "each flask has an orange cap and a visible liquid level inside"+ - "a front control panel with two knobs and a small dark display"+ - "the housing stands on four feet on the ground plane"+ dimensions_m: {width: 0.62, depth: 0.58, height: 0.9}diff --git a/eval/llm/batteries/fancy_v1_cadquery.yaml b/eval/llm/batteries/fancy_v1_cadquery.yamlnew file mode 100644index 00000000..5a3e10f5--- /dev/null+++ b/eval/llm/batteries/fancy_v1_cadquery.yaml@@ -0,0 +1,97 @@+# fancy_v1 — cadquery slice (3 prompts). Machined, turned and cast objects that suit a+# B-rep kernel: fillets, bores, bolt circles, gear teeth, knurls and threads. Every prompt+# names the real construction of the object; must_have items are single falsifiable+# clauses with counts. All hard tier; ids fancy_cq_<snake>.+# Sibling files: fancy_v1_blender.yaml, fancy_v1_threejs.yaml.+name: fancy_v1_cadquery+track: static_object+language: cadquery+description: >+ fancy_v1 cadquery slice: 3 hard machined/cast objects (Jacobs chuck on an arbor, a+ cast-iron handwheel gate valve, a turbocharger housing) with bores, bolt circles, knurls+ and fillets that a CAD kernel can express exactly.+prompts:+ # Jacobs chuck: three jaws on a scroll in a tapered body, a knurled sleeve, a key with a+ # pinion, the arbor behind — turned parts with a knurl and radial repeats at 120 degrees.+ - id: fancy_cq_jacobs_chuck+ tier: hard+ category: tools+ tags: [fancy_v1, hard]+ prompt: >+ a keyed Jacobs drill chuck on a Morse-taper arbor, as a machined assembly: a steel+ chuck body about 0.045 m in diameter and 0.075 m long, turned with a shallow taper+ toward the nose; over its middle a rotating knurled sleeve about 0.052 m in diameter+ with straight knurling of at least 60 ribs and a chamfered front edge; at the nose,+ three hardened jaws set at 120 degrees in angled slots, each jaw a rectangular bar+ with a ground gripping face, protruding about 0.012 m and open around a 0.008 m gap+ holding a twist drill bit about 0.09 m long with two helical flutes; on the rear+ of the sleeve a ring of bevel gear teeth, at least 24 teeth, into which a chuck key+ with a matching bevel pinion and a T-bar handle is engaged in one of three key holes+ drilled into the body at 120 degrees; behind the body a tapered arbor about 0.10 m+ long with a flat tang at its end; fillets at every shoulder.+ must_have:+ - "a turned chuck body with a knurled sleeve of at least 60 straight ribs around it"+ - "exactly 3 jaws at 120 degree spacing at the nose, angled inward toward the axis"+ - "a drill bit with two helical flutes gripped between the three jaws, coaxial with the body"+ - "a ring of at least 24 bevel gear teeth on the rear face of the sleeve"+ - "a chuck key with a bevel pinion engaged into a hole in the body, its T-bar handle outside"+ - "a tapered arbor behind the body ending in a flat tang"+ - "body, sleeve and arbor are all coaxial"+ - "visible fillets at the shoulders between arbor and body"+ dimensions_m: {width: 0.11, depth: 0.09, height: 0.27}+ # Cast-iron gate valve: flanged body with bolt circles, a bonnet with its own bolt ring,+ # a yoke, a rising stem with thread, a spoked handwheel; castings plus machined faces.+ - id: fancy_cq_gate_valve+ tier: hard+ category: architecture+ tags: [fancy_v1, hard]+ prompt: >+ a DN100 cast-iron rising-stem gate valve as installed in a pipe run: a bulbous cast+ body with a horizontal bore, a raised-face flange at each end about 0.22 m in+ diameter and 0.024 m thick with eight bolt holes on a 0.18 m pitch circle, each hole+ fitted with a hex bolt and nut;+ a bonnet flange on top about 0.16 m across with six bolts on its own circle, the+ bonnet rising as a domed casting into a yoke of two curved arms that meet at a+ threaded stem nut boss; a rising stem about 0.025 m in diameter passing up through+ the yoke, its upper part visibly threaded with at least 10 turns showing above the+ handwheel; a cast handwheel about 0.30 m in diameter with five curved spokes, a+ dished rim of round section and a hub with a square drive and a retaining nut; a gland with two studs where the stem enters the bonnet; castings dark blue-grey,+ flange faces bright, handwheel red.+ must_have:+ - "a horizontal through-bore body with a flange at each end"+ - "exactly 8 bolt holes on each end flange, each fitted with a bolt and nut"+ - "a bonnet flange on top with exactly 6 bolts on its own circle"+ - "a two-armed yoke rising from the bonnet meeting at a central boss"+ - "a vertical stem passing through the boss with at least 10 thread turns visible above the handwheel"+ - "a handwheel about 0.30 m in diameter with exactly 5 spokes"+ - "the handwheel red, the flange faces bright metal, the body a dark cast colour"+ - "the body, bonnet, yoke, stem and handwheel share one vertical axis"+ dimensions_m: {width: 0.35, depth: 0.32, height: 0.62}+ # Turbocharger: two cast volutes, a machined centre bearing housing, a finned compressor+ # wheel and turbine wheel visible in their inlets, V-band and bolt flanges, an actuator.+ - id: fancy_cq_turbocharger+ tier: hard+ category: vehicles+ tags: [fancy_v1, hard]+ prompt: >+ an automotive turbocharger as a machined and cast assembly: a snail-shaped aluminium+ compressor volute about 0.16 m across whose scroll grows in section around the axis+ to a tangential outlet spigot, and+ an axial inlet bore about 0.055 m across in which the compressor wheel is visible: a hub with 12 thin curved blades; a+ turbine housing on the other side, a cast-iron volute about 0.14 m across with a+ square inlet flange carrying four bolt holes, and a round outlet with a V-band+ flange; between the volutes a cylindrical centre bearing housing about 0.07 m in+ diameter with a boss for the oil feed banjo bolt on top and a drain flange with two+ studs underneath, joined to each housing by a ring of six clamp bolts; a wastegate+ actuator can on a bracket on the compressor housing, its rod running to a lever on the+ turbine housing; fillets at every casting junction.+ must_have:+ - "two snail-shaped volutes on a common axis with a cylindrical bearing housing between them"+ - "the compressor volute has a tangential outlet spigot whose scroll grows in section around the axis"+ - "a compressor wheel with at least 10 curved blades visible inside the axial inlet bore"+ - "a square inlet flange on the turbine housing with exactly 4 bolt holes"+ - "a V-band flange on the turbine outlet"+ - "a ring of at least 6 bolts where each volute meets the centre housing"+ - "a round actuator can on a bracket with a rod running to a lever on the turbine housing"+ - "aluminium-coloured compressor housing distinct from the dark iron turbine housing"+ dimensions_m: {width: 0.28, depth: 0.22, height: 0.2}diff --git a/eval/llm/batteries/fancy_v1_threejs.yaml b/eval/llm/batteries/fancy_v1_threejs.yamlnew file mode 100644index 00000000..227f3fdb--- /dev/null+++ b/eval/llm/batteries/fancy_v1_threejs.yaml@@ -0,0 +1,99 @@+# fancy_v1 — threejs slice (3 prompts). Objects that read through colour separation and+# instancing of repeated parts (keys, blades, slats, tines): a good fit for procedural+# three.js meshes with per-part materials. Every prompt names the real construction;+# must_have items are single falsifiable clauses with counts. All hard tier; ids+# fancy_tj_<snake>. Sibling files: fancy_v1_blender.yaml, fancy_v1_cadquery.yaml.+name: fancy_v1_threejs+track: static_object+language: threejs+description: >+ fancy_v1 threejs slice: 3 hard static objects built from many repeated coloured parts+ (a Wurlitzer-style jukebox, a Dutch windmill with lattice sails, a xylophone-style+ marimba) that reward instancing and clean material separation.+prompts:+ # Jukebox: arched cabinet with bubble tubes, a chrome grille, a visible record carousel+ # behind glass, a row of selector buttons — emissive colour bands plus repeated parts.+ - id: fancy_tj_bubbler_jukebox+ tier: hard+ category: appliances+ tags: [fancy_v1, hard]+ prompt: >+ a 1940s bubbler-style jukebox about 1.50 m tall: a wooden cabinet with a rounded arch+ top, its front built as concentric arches of translucent coloured plastic panels+ (yellow, red and green from the outside in) lit from behind, with two clear bubble+ tubes following the outer arch; inside the arch a curved glass window showing the+ record mechanism: a carousel of 24 vertical 78 rpm records on edge in a slotted+ rack, with a chrome tonearm and a turntable in front; below the window a sloped+ title strip holder with two columns of 12 title cards each behind a glass strip,+ flanked by a row of 24 ivory selector buttons in two banks; the lower front a+ chrome pierced speaker grille of at least eight vertical bars over a red cloth+ backing, with two chrome coin-slot plates; four short cabinet feet; the cabinet+ sides in dark walnut with a chrome trim strip up each front corner.+ must_have:+ - "a cabinet about 1.50 m tall with a rounded arched top"+ - "at least 3 concentric coloured arch bands, distinct hues, on the front"+ - "two bubble tubes following the outer arch on each side"+ - "a curved window inside the arch through which at least 20 records are visible standing on edge"+ - "two columns of title cards behind a glass strip below the window"+ - "at least 20 selector buttons in two banks"+ - "a speaker grille of at least 8 vertical chrome bars over a red backing on the lower front"+ - "the coloured arch bands emissive so they read as lit"+ dimensions_m: {width: 0.85, depth: 0.65, height: 1.52}+ # Dutch smock windmill: an octagonal thatched body on a brick base, a rotating cap, four+ # lattice sails with dozens of instanced slats, a tail pole and a stage gallery.+ - id: fancy_tj_smock_windmill+ tier: hard+ category: architecture+ tags: [fancy_v1, hard]+ prompt: >+ a Dutch smock windmill about 18 m tall: a tapered octagonal tower, the lower 5 m in+ red brick with a door and two arched windows, the upper part thatched over a timber+ frame with eight corner posts; a stage+ gallery ringing the tower at the brick-to-thatch line, its deck on radial joists+ with a railing of at least 40 balusters; a boat-shaped rotating cap on top, its+ plank cladding laid horizontally, carrying the windshaft at a slight upward tilt; four+ common sails fixed at 90 degrees on a cast-iron cross, each sail a long whip+ about 12 m from the shaft with a lattice of at least 18 sail bars across it, the+ lattice on one side of the whip only with a leading board on the other, the sails+ set in the X position; a tail pole angling down from the back of the cap to a wheel+ on the stage; sail lattices white,+ the cap tarred black, the thatch grey-brown.+ must_have:+ - "an octagonal tapered tower that is brick below and thatch above"+ - "a gallery deck ringing the tower at the material change with at least 40 balusters"+ - "a rotating cap on top with horizontal plank cladding"+ - "exactly 4 sails on a central cross set at 90 degree spacing"+ - "each sail about 12 m from the shaft with at least 18 lattice bars"+ - "a tail pole from the back of the cap running down to the gallery"+ - "white sails, black cap and brown thatch as three distinct colours"+ - "the tower base sits on the ground plane and no sail tip reaches the ground"+ dimensions_m: {width: 26.0, depth: 26.0, height: 18.5}+ # Concert marimba: 49 graduated rosewood bars in two rows on a frame, a resonator tube+ # under each bar graduated in length, a wheeled frame; instancing with per-bar scaling.+ - id: fancy_tj_concert_marimba+ tier: hard+ category: instruments+ tags: [fancy_v1, hard]+ prompt: >+ a concert marimba with a 4-octave range: 49 rosewood bars laid out like a piano+ keyboard on a wooden frame about 2.10 m long and 0.92 m high, the naturals in a+ front row of 29 bars and the accidentals in a raised rear row of 20 bars offset+ behind the gaps, the accidental row set back and about 0.03 m higher; bars+ graduated in both length and width from about 0.45 m long and 0.075 m wide at the+ bass end to 0.15 m long and 0.04 m wide at the treble, each bar resting on a cord through+ nodal holes strung between posts on the rails; beneath every bar a vertical golden-coloured+ aluminium resonator tube whose length matches the bar's pitch, longest under the+ bass bars and stepping down toward the treble, so the resonator array forms a curved+ wedge; the bass end of the frame taller than the treble end;+ four legs on rubber-tyred casters; two mallets with yarn heads resting across the+ bass bars.+ must_have:+ - "a front row of at least 25 bars and a raised rear row of at least 18 bars"+ - "the rear row is offset behind the gaps of the front row and sits higher"+ - "bars graduate in length from long at one end to short at the other"+ - "a vertical resonator tube under every bar"+ - "resonator tube lengths step down from the bass end to the treble end"+ - "bars are a rosewood colour and resonators a golden metal colour"+ - "four legs on casters, all touching the ground plane"+ - "overall length about 2.10 m"+ dimensions_m: {width: 2.15, depth: 0.85, height: 0.95}diff --git a/eval/llm/batteries/graphics_v2.yaml b/eval/llm/batteries/graphics_v2.yamlnew file mode 100644index 00000000..4d8bcceb--- /dev/null+++ b/eval/llm/batteries/graphics_v2.yaml@@ -0,0 +1,170 @@+# graphics_v2 — shader / OpenGL battery.+#+# Difficulty axes:+# raymarching depth full raymarched scenes with material response (SSS-ish glow,+# fractal surface detail, correct-looking gear meshing) — v1's+# only raymarch prompt scored 0.30, the strongest discriminator+# feedback state multi-pass programs whose look emerges from a feedback buffer+# (reaction-diffusion, ink advection) — cannot be faked in one pass+# stylisation control hitting a named art style with countable features (ukiyo-e wave,+# CRT boot) instead of generic noise beauty+# instancing scale opengl_python prompts with explicit instance counts and a+# structured motion the VLM can verify across frames+# Per-prompt `language: opengl_python` marks the raw-OpenGL prompts (battery default+# is glsl_shader); the bench loader honours the per-prompt override (BenchPrompt.language).+# Expected discrimination: medium 40-70% pass; hard 0-30% pass — feedback-buffer and+# instancing prompts also stress the build/runtime, where v1's opengl prompt errored.+name: graphics_v2+track: graphics+language: glsl_shader+description: >+ 10 graphics prompts: 3 medium stylised shaders, 7 hard (raymarched scenes,+ feedback-buffer simulations, large-scale instancing). must_have items are+ frame-verifiable features with counts and motion statements.+prompts:+ # ------------------------------------------------------------- raymarched (hard)+ - id: glsl_hard_mushroom_forest+ tier: hard+ category: raymarch+ prompt: >+ a raymarched night mushroom forest: giant glowing mushrooms whose caps+ transmit light like wax, thin stems, drifting spore motes, soft shadows and+ a slow camera glide between the stems+ must_have:+ - "at least 5 mushrooms with domed caps on visible stems"+ - "caps glow from inside with a soft falloff toward the rim, brighter where thin"+ - "ground plane receiving soft shadows from the stems"+ - "drifting bright spore motes between the mushrooms"+ - "dark blue or violet night ambience with the caps as main light sources"+ - "camera glides forward between the stems over the loop"+ - id: glsl_hard_infinite_corridor+ tier: hard+ category: raymarch+ prompt: >+ a raymarched infinite corridor: repeating arched segments receding to a+ vanishing point, walls carved with self-similar fractal relief that gains+ detail as the camera approaches, cool fog in the distance, endless forward+ motion+ must_have:+ - "corridor of repeating arched segments receding to a vanishing point"+ - "at least 6 segment repetitions visible at once"+ - "wall surfaces show self-similar relief with at least 3 visible scales of detail"+ - "distance fog fades the far segments"+ - "forward motion continues across all sampled frames without the camera clipping the walls"+ - "light sources along the corridor alternate or repeat with the segments"+ - id: glsl_hard_clockwork_gears+ tier: hard+ category: raymarch+ prompt: >+ a raymarched brass clockwork: at least three meshing gears with visible+ teeth turning at consistent ratios, a slowly swinging pendulum disc behind+ them, metallic highlights and a dark workshop background+ must_have:+ - "at least 3 gears with clearly separated teeth"+ - "adjacent gears rotate in opposite directions"+ - "a small gear visibly turns faster than a large one it meshes with"+ - "teeth of meshing gears interleave without overlapping"+ - "pendulum disc swings periodically behind the gears"+ - "metallic specular highlights on the gear faces"+ # ---------------------------------------------------- feedback simulations (hard)+ - id: ogl_hard_reaction_diffusion+ tier: hard+ category: simulation+ language: opengl_python+ prompt: >+ a reaction-diffusion coral garden grown in a feedback buffer: patterns seeded+ at several points spread outward into labyrinthine coral growth, shaded with+ height-like lighting and an ocean palette, growth visible over the run+ must_have:+ - "labyrinthine or spotted pattern characteristic of reaction-diffusion"+ - "pattern grows outward from at least 3 seed points over time"+ - "later frames show strictly more covered area than early frames"+ - "pattern edges shaded with a relief or rim-light effect"+ - "ocean palette: teal to orange or pink gradient mapped to the pattern"+ - "the pattern boundary advances in every successive sampled frame"+ - id: ogl_hard_ink_swirls+ tier: hard+ category: simulation+ language: opengl_python+ prompt: >+ fluid ink in water via a feedback advection buffer: dark ink blobs injected+ periodically curl into filament swirls that stretch and fold without+ dissipating instantly, on a paper-white background+ must_have:+ - "ink blobs appear at intervals during the run"+ - "blobs stretch into curling filaments over subsequent frames"+ - "filaments fold back on themselves forming spiral structures"+ - "earlier ink stays visible as faded wisps while new ink is injected"+ - "paper-white background with dark ink, no full-frame darkening over time"+ # ------------------------------------------------------------- stylised (medium)+ - id: glsl_med_ukiyoe_wave+ tier: medium+ category: stylised+ prompt: >+ a looping ukiyo-e style great wave: a large curling crest with claw-like foam+ fingers, flat indigo color bands with paper-cream foam, smaller waves in+ front, and a seamless loop where the crest rises, curls and collapses+ must_have:+ - "one dominant wave with a forward-curling crest"+ - "foam edge broken into claw-like fingers"+ - "flat color bands in indigo and cream, no photoreal gradients"+ - "at least 2 smaller waves in the foreground"+ - "across the sampled frames the crest rises, curls forward and collapses"+ - id: glsl_med_crt_boot+ tier: medium+ category: stylised+ prompt: >+ a retro CRT boot sequence: black screen flickers on with a horizontal flash,+ green blocky glyph rows type in line by line, a blinking cursor, scanlines,+ barrel distortion at the screen corners and phosphor glow+ must_have:+ - "earliest sampled frame shows the turn-on state: mostly dark or a bright horizontal flash band"+ - "rows of blocky glyphs appear line by line down the screen"+ - "blinking rectangular cursor at the end of the last row"+ - "visible scanlines across the whole frame"+ - "image bulges with barrel distortion, corners darkened"+ - "green phosphor glow bleeding around bright glyphs"+ - id: glsl_med_kaleido_mandala+ tier: medium+ category: stylised+ prompt: >+ a kaleidoscopic mandala: 12-fold mirror symmetry, at least three concentric+ pattern rings rotating at different speeds in alternating directions, petal+ shapes morphing smoothly, jewel palette on a dark background+ must_have:+ - "12-fold mirror symmetry about the center"+ - "at least 3 concentric pattern rings"+ - "adjacent rings rotate in opposite directions at different speeds"+ - "petal or lobe shapes morph smoothly over time"+ - "saturated jewel palette on a dark background, center stays centered"+ # ------------------------------------------------------ instancing scale (hard)+ - id: ogl_hard_boids_murmuration+ tier: hard+ category: instancing+ language: opengl_python+ prompt: >+ a starling murmuration of 30000 instanced boids over a dusk landscape+ silhouette: the flock stretches, folds and splits into lobes that merge+ again, dense cores darker than the edges, slow camera pan+ must_have:+ - "flock of thousands of small instances, far more than any countable number"+ - "flock forms one connected shape that stretches and folds over time"+ - "flock splits into at least 2 lobes and merges again during the run"+ - "denser flock regions read darker or thicker than sparse edges"+ - "dusk gradient sky with a dark landscape silhouette at the bottom"+ - "camera pans slowly while the flock stays in frame"+ - id: ogl_hard_voxel_city+ tier: hard+ category: instancing+ language: opengl_python+ prompt: >+ a fly-through of an instanced voxel city with a full day-night cycle:+ cube-built towers of varying heights on a street grid, sun light and long+ shadows by day, lit windows by night, the camera weaving between towers+ must_have:+ - "city of cube-built towers on a regular street grid"+ - "at least 50 towers of clearly varying heights"+ - "continuous camera flight between the towers without entering any tower"+ - "day phase: directional sun with visible long shadows"+ - "night phase: dark sky with emissive lit windows on the towers"+ - "smooth day-to-night transition within the run"diff --git a/eval/llm/batteries/static_objects_v1.yaml b/eval/llm/batteries/static_objects_v1.yamlnew file mode 100644index 00000000..cb9705a7--- /dev/null+++ b/eval/llm/batteries/static_objects_v1.yaml@@ -0,0 +1,137 @@+name: static_objects_v1+track: static_object+language: blender+description: >+ 24 static-object prompts, 8 categories × 3 tiers (easy / medium / hard).+ Every prompt carries must_have items the planner turns into acceptance checks.+ Same battery runs under any generator; keep the judge model fixed across arms.+prompts:+ # ---------------------------------------------------------------- furniture+ - id: furn_easy_stool+ tier: easy+ category: furniture+ prompt: a simple three-legged round wooden stool, 45 cm tall+ must_have: [round seat, three splayed legs of equal length, legs touch the ground]+ dimensions_m: {height: 0.45}+ - id: furn_med_dining_chair+ tier: medium+ category: furniture+ prompt: a mid-century wooden dining chair with a slatted backrest and tapered legs+ must_have: [four tapered legs, seat slab, backrest with at least three vertical slats, back legs continue into back posts]+ - id: furn_hard_rolltop_desk+ tier: hard+ category: furniture+ prompt: a writing desk with two pedestals of three drawers each, a central knee-hole, a leather inlay top and brass drawer pulls+ must_have: [two drawer pedestals, three drawers per pedestal with visible fronts, knee-hole gap, brass pull handles, desk top overhangs]+ # ---------------------------------------------------------------- vehicles+ - id: veh_easy_toy_car+ tier: easy+ category: vehicles+ prompt: a chunky toy car with a rounded body and four thick wheels+ must_have: [one-piece rounded body, four wheels touching the ground, wheels protrude sideways]+ - id: veh_med_pickup+ tier: medium+ category: vehicles+ prompt: a pickup truck with a two-door cab, open cargo bed, wheel arches and side mirrors+ must_have: [cab with windows, open rear bed, four wheels with hubs, front grille and headlights, side mirrors]+ - id: veh_hard_tractor+ tier: hard+ category: vehicles+ prompt: a farm tractor with large rear tyres with treads, small front wheels, an exposed engine block, exhaust stack, driver seat, steering wheel and a roll bar+ must_have: [rear wheels much larger than front wheels, tyre tread detail, exhaust stack, seat and steering wheel, roll bar, engine hood]+ # ---------------------------------------------------------------- tools+ - id: tool_easy_hammer+ tier: easy+ category: tools+ prompt: a claw hammer with a wooden handle and a steel head+ must_have: [cylindrical handle, steel head with a flat face, forked claw on the back]+ - id: tool_med_hand_drill+ tier: medium+ category: tools+ prompt: a cordless power drill with a pistol grip, trigger, battery pack in the base of the grip and a keyless chuck holding a drill bit+ must_have: [pistol grip body, trigger, battery block under grip, chuck with a bit, side vents]+ - id: tool_hard_bench_vise+ tier: hard+ category: tools+ prompt: a cast-iron bench vise with fixed and sliding jaws, a lead screw with a tommy bar handle, guide rails, a small anvil face and mounting base with bolt holes+ must_have: [two jaws with a visible gap, sliding jaw on guide rails, lead screw with sliding T handle, anvil flat, base with bolt holes]+ # ---------------------------------------------------------------- appliances+ - id: appl_easy_kettle+ tier: easy+ category: appliances+ prompt: an electric kettle with a spout, a lid and a side handle on a round base+ must_have: [pot body, pouring spout, handle opposite the spout, lid on top, round base]+ - id: appl_med_toaster+ tier: medium+ category: appliances+ prompt: a two-slot retro toaster with rounded chrome body, a lever on the side, a browning dial and a crumb tray+ must_have: [two bread slots on top, side lever, dial knob, rounded body, small feet]+ - id: appl_hard_espresso+ tier: hard+ category: appliances+ prompt: an espresso machine with a group head and portafilter, a steam wand, a drip tray with grate, a water tank at the back, and a top-mounted cup warmer with a pressure gauge on the front panel+ must_have: [portafilter with handle under a group head, steam wand on the side, drip tray with grate, water tank, pressure gauge, cup warmer rail]+ # ---------------------------------------------------------------- plants+ - id: plant_easy_cactus+ tier: easy+ category: plants+ prompt: a potted barrel cactus in a terracotta pot with soil+ must_have: [ribbed round cactus body, terracotta pot with rim, soil surface]+ - id: plant_med_palm+ tier: medium+ category: plants+ prompt: a small potted palm with a textured trunk and eight arching fronds made of many leaflets+ must_have: [textured trunk, at least eight fronds, fronds arch downward, leaflets along each frond, pot]+ - id: plant_hard_bonsai+ tier: hard+ category: plants+ prompt: a bonsai tree with a twisted thick trunk, three layered foliage pads, exposed roots over a rock and a shallow oval ceramic tray with moss+ must_have: [twisted trunk with taper, at least three distinct foliage pads, exposed roots gripping a rock, shallow oval tray, moss surface]+ # ---------------------------------------------------------------- buildings+ - id: bld_easy_shed+ tier: easy+ category: buildings+ prompt: a small garden shed with a pitched roof, one door and one window+ must_have: [box walls, pitched roof overhang, door on front, window]+ - id: bld_med_lighthouse+ tier: medium+ category: buildings+ prompt: a tapered lighthouse with red and white bands, a gallery with railing, a glass lantern room and a domed cap, on a rocky base+ must_have: [tapered tower, alternating bands, gallery walkway with railing, glass lantern room, dome, rock base]+ - id: bld_hard_windmill+ tier: hard+ category: buildings+ prompt: a Dutch windmill with an octagonal tapered body, thatched cap, four lattice sails, a balcony stage with railing, a door, and two shuttered windows+ must_have: [octagonal tapered body, cap, four lattice sails in a cross, balcony with railing, door, shuttered windows]+ # ---------------------------------------------------------------- toys+ - id: toy_easy_blocks+ tier: easy+ category: toys+ prompt: a stack of five alphabet blocks of decreasing size in a slightly twisted tower+ must_have: [five cubes, decreasing size upward, each slightly rotated, all stacked touching]+ - id: toy_med_rocking_horse+ tier: medium+ category: toys+ prompt: a wooden rocking horse with curved rockers, a saddle, handles and a mane+ must_have: [horse body with head and four legs, two curved rockers, saddle, handle grips, mane]+ - id: toy_hard_train+ tier: hard+ category: toys+ prompt: a toy steam locomotive with a boiler, smokestack, cab with windows, cowcatcher, six driving wheels with connecting rods, and a coal tender with four wheels+ must_have: [cylindrical boiler, smokestack, cab, cowcatcher, six large wheels with rods, separate tender with coal load and four wheels]+ # ---------------------------------------------------------------- musical instruments+ - id: mus_easy_drum+ tier: easy+ category: musical_instruments+ prompt: a snare drum with tension rods and two drumsticks resting on top+ must_have: [cylindrical shell, top head, at least eight tension rods around the rim, two sticks]+ - id: mus_med_acoustic_guitar+ tier: medium+ category: musical_instruments+ prompt: an acoustic guitar with a figure-eight body, sound hole, bridge, fretted neck, headstock with six tuning pegs and six strings+ must_have: [figure-eight body, sound hole, bridge, neck with frets, headstock, six tuning pegs, six strings]+ - id: mus_hard_drum_kit+ tier: hard+ category: musical_instruments+ prompt: a five-piece drum kit with a bass drum with pedal, snare on a stand, two rack toms, a floor tom, a hi-hat and two cymbals on stands, and a throne+ must_have: [bass drum with pedal, snare on stand, two mounted toms, floor tom with legs, hi-hat with two cymbals, two crash/ride cymbals on stands, drum throne]diff --git a/eval/llm/batteries/static_objects_v2.yaml b/eval/llm/batteries/static_objects_v2.yamlnew file mode 100644index 00000000..bf9f339d--- /dev/null+++ b/eval/llm/batteries/static_objects_v2.yaml@@ -0,0 +1,425 @@+# static_objects_v2 — hard-tier battery (v1 easy saturates: 100% pass, 0.85-0.96 with 3.7-flash).+#+# Difficulty axes (named per prompt via tags):+# axis:decomposition 10+ distinct nameable parts that must fit together (v1 hard failure+# modes were floating parts / interpenetration / stray islands — these+# prompts maximise part-to-part contact surface count)+# axis:thin_features wires, spokes, scrolls, strings at 3-30 mm thickness+# axis:curved_organic compound-curved / organic surfaces (no box-stack shortcut)+# axis:arch_detail repeated countable architectural elements (dentils, rafters, panes)+# axis:multi_material compositions of 2+ objects in distinct materials that must relate+# Controls: 4 medium prompts whose PROMPT TEXT is copied verbatim from v1 medium+# (furniture / vehicles / appliances / tools — 2/4 passed in static_v1_flash, the+# discriminating band of that run) so+# generation difficulty is comparable across versions; their must_have lists are extended+# to the v2 checklist standard (8+ items, half with dims), so judge scores are v2-internal.+# Expected discrimination: controls ~30-60% pass; hard tier 0-30% pass with score spread+# 0.4-0.7 driven by assembly_fit and geometry_detail, not intent_fidelity.+name: static_objects_v2+track: static_object+language: blender+description: >+ 20 static-object prompts: 4 medium controls (prompt text shared with v1 medium for+ cross-version comparison) and 16 hard prompts across five named difficulty axes+ (decomposition depth, thin features, curved/organic surfaces, architectural detail,+ multi-material composition). Every prompt carries 8-16 nameable parts in must_have,+ at least half with measurable dimensions.+prompts:+ # ------------------------------------------------------------- controls (v1 medium text)+ - id: ctrl_med_dining_chair+ tier: medium+ category: furniture+ tags: [control, "axis:decomposition"]+ prompt: a mid-century wooden dining chair with a slatted backrest and tapered legs+ must_have:+ - "four tapered legs, each thinner at the floor than at the seat"+ - "seat slab about 0.42 m wide"+ - "seat surface about 0.45 m above the floor"+ - "backrest with at least 3 vertical slats"+ - "back legs continue up into the backrest posts"+ - "backrest top rail between 0.80 m and 0.95 m above the floor"+ - "all four feet touch the ground plane"+ - "no leg or slat intersects the seat slab"+ dimensions_m: {width: 0.45, depth: 0.50, height: 0.90}+ - id: ctrl_med_pickup+ tier: medium+ category: vehicles+ tags: [control, "axis:decomposition"]+ prompt: a pickup truck with a two-door cab, open cargo bed, wheel arches and side mirrors+ must_have:+ - "cab with windshield and 2 side windows"+ - "open rear bed with 3 walls and a tailgate"+ - "4 wheels with hubs, all touching the ground"+ - "wheel diameter about 0.75 m"+ - "front grille and 2 headlights"+ - "2 side mirrors on stalks"+ - "bed floor about 0.80 m above the ground"+ - "overall length about 5.2 m"+ dimensions_m: {length: 5.2, width: 1.9, height: 1.8}+ - id: ctrl_med_toaster+ tier: medium+ category: appliances+ tags: [control, "axis:curved_organic"]+ prompt: a two-slot retro toaster with rounded chrome body, a lever on the side, a browning dial and a crumb tray+ must_have:+ - "2 bread slots on the top face"+ - "each slot about 0.13 m long and under 0.04 m wide"+ - "lever on one end face"+ - "round dial knob on the front face"+ - "rounded body with no sharp vertical edges"+ - "crumb tray seam line near the base"+ - "4 small feet"+ - "body about 0.28 m wide and 0.19 m tall"+ dimensions_m: {width: 0.28, depth: 0.17, height: 0.19}+ - id: ctrl_med_hand_drill+ tier: medium+ category: tools+ tags: [control, "axis:decomposition"]+ prompt: a cordless power drill with a pistol grip, trigger, battery pack in the base of the grip and a keyless chuck holding a drill bit+ must_have:+ - "pistol-grip body with the grip angled back from vertical"+ - "trigger recessed at the top of the grip"+ - "battery block under the grip, wider than the grip"+ - "chuck at the front, about 0.04 m diameter"+ - "drill bit protruding about 0.08 m from the chuck"+ - "side vents on the motor housing"+ - "overall length about 0.22 m from chuck tip to body rear"+ - "overall height about 0.24 m from battery base to body top"+ dimensions_m: {length: 0.22, height: 0.24}+ # ------------------------------------------------------- mechanical assemblies (hard)+ - id: mech_hard_coffee_grinder+ tier: hard+ category: mechanisms+ tags: ["axis:decomposition"]+ prompt: >+ a hand-crank coffee grinder: square wooden box body with a small front drawer,+ cast-iron dome top with an open hopper funnel, a conical burr visible inside+ the funnel throat, and a top crank arm with a wooden knob+ must_have:+ - "square wooden box body about 0.14 m wide and 0.12 m tall"+ - "front drawer about 0.05 m tall with a round knob, in the lower half of the box"+ - "drawer front sits flush in its opening with a visible seam"+ - "metal dome top mounted on the box"+ - "open hopper funnel about 0.08 m across at the rim"+ - "conical burr visible inside the funnel throat"+ - "central spindle about 0.01 m thick rising from the burr through the dome"+ - "crank arm about 0.11 m long attached to the spindle top"+ - "wooden knob on a short vertical pin at the crank's free end"+ - "adjustment nut on the spindle above the crank hub"+ dimensions_m: {width: 0.14, depth: 0.14, height: 0.30}+ - id: mech_hard_machinist_vise+ tier: hard+ category: tools+ tags: ["axis:decomposition", "axis:thin_features"]+ prompt: >+ a machinist bench vise with an acme lead screw showing its thread helix, twin+ cylindrical guide rails, a swivel base with two lock handles, serrated jaw+ plates and a sliding tommy-bar handle+ must_have:+ - "fixed jaw and sliding jaw with a 0.05 m open gap between them"+ - "flat jaw plates about 0.10 m wide with serration grooves"+ - "lead screw between the jaws with at least 6 visible thread turns"+ - "2 cylindrical guide rails flanking the screw"+ - "sliding jaw rides on the rails and the screw"+ - "tommy bar about 0.16 m long through the screw head, with ball ends"+ - "swivel base disc under the vise body"+ - "2 lock handles on the swivel base rim"+ - "mounting flange with 3 bolt holes"+ - "overall body length about 0.30 m"+ dimensions_m: {length: 0.30, width: 0.14, height: 0.18}+ - id: mech_hard_drill_press+ tier: hard+ category: tools+ tags: ["axis:decomposition"]+ prompt: >+ a benchtop drill press: cast base, vertical column, adjustable work table with+ a crank, head casting with a belt housing, chuck with a bit over the table,+ three-spoke quill feed lever and a depth-stop rod+ must_have:+ - "flat base about 0.30 m by 0.20 m with 2 mounting slots"+ - "vertical column about 0.05 m diameter rising from the base"+ - "work table clamped to the column between base and head"+ - "table raise crank handle below the table"+ - "head casting at the top of the column"+ - "belt housing on top of the head with a lid seam"+ - "cylindrical motor behind the column, connected to the head"+ - "chuck about 0.04 m diameter under the head with a drill bit pointing down"+ - "bit tip aligned over the table center"+ - "quill feed lever with 3 spokes ending in ball knobs"+ - "depth-stop rod with 2 nuts beside the quill"+ - "overall height about 0.85 m"+ dimensions_m: {height: 0.85, depth: 0.45}+ - id: mech_hard_pitcher_pump+ tier: hard+ category: garden+ tags: ["axis:decomposition", "axis:curved_organic"]+ prompt: >+ a cast-iron pitcher water pump on a wooden platform: fluted cylindrical body,+ curved downward spout, long lever handle raised mid-stroke, connecting link to+ the piston rod, domed cap and a bolted base flange+ must_have:+ - "cylindrical pump body about 0.09 m diameter"+ - "at least 6 vertical flutes on the body"+ - "base flange with 4 bolt heads"+ - "square wooden platform under the flange"+ - "curved spout leaving the body two-thirds up, opening facing down"+ - "domed cap on top of the body"+ - "lever handle about 0.40 m long, raised at an angle"+ - "handle pivots in a clevis bracket at the top rear of the body"+ - "connecting link from the handle to a piston rod entering the cap"+ - "overall height about 0.65 m"+ dimensions_m: {height: 0.65}+ # ------------------------------------------------------- thin / curved structures (hard)+ - id: thin_hard_garden_arch+ tier: hard+ category: garden+ tags: ["axis:thin_features"]+ prompt: >+ a wrought-iron garden arch: two lattice side frames, a semicircular arch head+ filled with S-scrolls, spear-point finials and a tie bar at the spring line+ must_have:+ - "2 side frames, each of 2 uprights about 0.01 m thick"+ - "diamond lattice bars between the uprights of each side frame"+ - "semicircular arch head spanning about 1.2 m between the side frames"+ - "at least 6 S-scrolls filling the arch head"+ - "at least 4 scrolls in each side frame"+ - "3 spear-point finials on top of the arch"+ - "horizontal tie bar connecting both sides at the spring line"+ - "small ground plates under all 4 upright feet"+ - "overall height about 2.3 m"+ - "all bars slender: no member thicker than 0.02 m"+ dimensions_m: {width: 1.4, depth: 0.5, height: 2.3}+ - id: thin_hard_violin+ tier: hard+ category: musical_instruments+ tags: ["axis:thin_features", "axis:curved_organic"]+ prompt: >+ a violin: hourglass body with C-bout waists, arched top with two f-holes, a+ bridge, four strings from tailpiece over the bridge along the fingerboard to+ the pegbox, a scroll head, four tuning pegs and a chin rest+ must_have:+ - "hourglass body with 2 concave C-bout waist cutaways"+ - "body length about 0.36 m"+ - "top plate visibly arched, not flat"+ - "2 f-holes through the top, one each side of the bridge"+ - "bridge about 0.03 m tall standing upright between the f-holes"+ - "4 strings running in parallel from tailpiece to pegbox"+ - "strings pass over the bridge with a visible break angle"+ - "black fingerboard extending over the body without touching it"+ - "spiral scroll at the head"+ - "4 tuning pegs, 2 per side of the pegbox"+ - "tailpiece anchored toward the end button"+ - "chin rest beside the tailpiece"+ - "total length about 0.60 m"+ dimensions_m: {length: 0.60, width: 0.21, height: 0.09}+ - id: thin_hard_penny_farthing+ tier: hard+ category: vehicles+ tags: ["axis:thin_features", "axis:curved_organic"]+ prompt: >+ a penny-farthing bicycle: huge spoked front wheel with pedal cranks on its hub,+ small spoked rear wheel, curved backbone tube, front fork, handlebar and a+ sprung leather saddle+ must_have:+ - "front wheel about 1.3 m diameter with at least 24 thin spokes"+ - "rear wheel about 0.40 m diameter with at least 12 spokes"+ - "tyres no thicker than 0.03 m"+ - "backbone tube arcing from the fork crown down to the rear wheel fork"+ - "front fork straddling the big wheel to its hub"+ - "handlebar with 2 grips above the front hub"+ - "2 pedal cranks on the front hub with pedals, opposed at 180 degrees"+ - "saddle on the backbone behind the handlebar, on a visible spring"+ - "small mounting step peg on the backbone above the rear wheel"+ - "both wheels touch the ground plane"+ dimensions_m: {length: 1.7, height: 1.5}+ - id: thin_hard_birdcage+ tier: hard+ category: household+ tags: ["axis:thin_features"]+ prompt: >+ a round wire birdcage with an onion-dome top, a hanging ring, a wire door with+ a latch, a trapeze perch hanging inside, a straight perch, two feed cups and a+ base tray with a rolled rim+ must_have:+ - "round cage about 0.35 m diameter"+ - "at least 20 vertical wires around the circumference"+ - "wires about 0.003 m thick"+ - "3 horizontal rings tying the wires"+ - "onion-dome top converging to a finial"+ - "hanging ring at the very top"+ - "rectangular door frame set into the wires with a small latch"+ - "trapeze perch hanging inside from the dome on 2 wires"+ - "straight perch spanning the cage between opposite walls"+ - "2 feed cups mounted at a ring"+ - "solid base tray with a rolled rim"+ - "overall height about 0.55 m"+ dimensions_m: {height: 0.55}+ # ------------------------------------------------------------------ organic (hard)+ - id: org_hard_bonsai_rock+ tier: hard+ category: plants+ tags: ["axis:curved_organic"]+ prompt: >+ a root-over-rock bonsai: a shallow oval pot with soil, a rounded rock gripped+ by exposed roots, a tapering trunk with two pronounced curves, and five+ distinct flattened foliage pads+ must_have:+ - "shallow oval pot about 0.30 m long with a rim and small feet"+ - "soil surface inside the pot"+ - "rounded rock about 0.12 m tall sitting on the soil"+ - "at least 4 roots gripping over the rock faces and entering the soil"+ - "trunk rising from the rock top with at least 2 pronounced curves"+ - "trunk tapers from about 0.04 m at the base to under 0.015 m at the apex"+ - "5 distinct flattened foliage pads with gaps between them"+ - "lowest branch extends beyond the pot rim"+ - "no foliage pad touches the soil or the rock"+ - "overall height about 0.50 m"+ dimensions_m: {height: 0.50, length: 0.45}+ - id: org_hard_seated_cat+ tier: hard+ category: animals+ tags: ["axis:curved_organic"]+ prompt: >+ a domestic cat sitting upright: haunches on the ground, straight vertical+ front legs, tail curled around beside the front paws, short muzzle and+ triangular ears+ must_have:+ - "upright seated pose with haunches on the ground and chest raised"+ - "2 straight front legs, nearly vertical"+ - "front paws on the ground between the haunches"+ - "hind legs folded with thighs bulging wider than the chest"+ - "tail about 0.30 m long curling around to lie beside the front paws"+ - "head with a short muzzle, 2 eyes and a nose"+ - "2 triangular ears about 0.04 m tall"+ - "back forms a smooth continuous curve from neck to tail base"+ - "silhouette narrows from haunches up to the shoulders"+ - "overall height about 0.35 m"+ dimensions_m: {height: 0.35, depth: 0.25}+ - id: org_hard_octopus_rock+ tier: hard+ category: animals+ tags: ["axis:curved_organic", "axis:thin_features"]+ prompt: >+ an octopus draped over a low rock: bulbous mantle, raised eyes, eight+ tapering arms curling in different directions, with sucker discs visible+ along the underside of at least three arms+ must_have:+ - "bulbous mantle about 0.20 m long"+ - "2 raised eye bumps on the head"+ - "8 arms radiating from the head"+ - "each arm tapers from about 0.05 m at the base to a point"+ - "arm tips curl, at least 4 curling in different directions"+ - "sucker discs along the underside of at least 3 arms"+ - "low rock under the body, arms draping over its sides"+ - "at least 2 arms touch the ground plane"+ - "overall arm spread about 0.8 m"+ dimensions_m: {length: 0.8, height: 0.3}+ # ------------------------------------------------------- architectural detail (hard)+ - id: arch_hard_bay_window+ tier: hard+ category: buildings+ tags: ["axis:arch_detail", "axis:decomposition"]+ prompt: >+ a Victorian bay window facade section: a three-sided bay projecting from a+ clapboard wall, a sash window with divided panes in each face, panel aprons,+ corner pilasters, scroll brackets and a dentil cornice under a faceted roof cap+ must_have:+ - "three-sided bay projecting from a flat wall section"+ - "center face wider than the 2 angled side faces"+ - "side faces angled about 45 degrees from the wall"+ - "sash window in each of the 3 faces"+ - "upper sash of each window divided into at least 4 panes by muntins"+ - "continuous sill band under all 3 windows"+ - "recessed panel apron under each window"+ - "pilasters at the 4 vertical bay corners"+ - "cornice with at least 10 dentil blocks across the bay"+ - "at least 4 scroll brackets under the bay roof"+ - "three-faceted roof cap over the bay"+ - "horizontal siding boards on the flat wall section"+ dimensions_m: {width: 3.4, height: 3.2, depth: 1.0}+ - id: arch_hard_spiral_stair+ tier: hard+ category: buildings+ tags: ["axis:arch_detail", "axis:thin_features"]+ prompt: >+ a cast-iron spiral staircase: central column, wedge treads winding a full+ turn, a baluster at each tread's outer edge, a continuous helical handrail+ and a railed top landing+ must_have:+ - "central column about 0.10 m diameter"+ - "at least 12 wedge-shaped treads attached to the column"+ - "treads climb evenly through at least 360 degrees total rotation"+ - "open risers: visible gaps between consecutive treads"+ - "1 baluster at the outer edge of each tread"+ - "continuous helical handrail on top of the balusters"+ - "top landing platform with its own railing"+ - "base plate at the bottom of the column"+ - "each tread about 0.70 m from column to outer edge"+ - "overall height about 2.7 m"+ dimensions_m: {height: 2.7, width: 1.6}+ - id: arch_hard_teahouse+ tier: hard+ category: buildings+ tags: ["axis:arch_detail"]+ prompt: >+ a Japanese teahouse: raised wooden floor on short posts, a veranda strip+ along the front, sliding shoji panels with visible grids, corner posts, and+ a hipped roof with deep eaves showing exposed rafter tails+ must_have:+ - "floor platform raised about 0.5 m on short posts"+ - "wooden veranda strip along the front edge"+ - "at least 3 sliding shoji panels in the front wall"+ - "each shoji panel gridded into at least 12 rectangles"+ - "square corner posts at the 4 corners"+ - "hipped roof with eaves overhanging about 0.6 m"+ - "at least 8 exposed rafter tails visible under each eave side"+ - "ridge cap along the roof top"+ - "solid back and side walls contrasting with the shoji front"+ - "step stone below the veranda"+ dimensions_m: {width: 3.5, depth: 3.0, height: 3.0}+ # ------------------------------------------------------- multi-material compositions (hard)+ - id: comp_hard_drafting_table+ tier: hard+ category: furniture+ tags: ["axis:multi_material", "axis:decomposition"]+ prompt: >+ a drafting table setup: tilted drawing board on a trestle frame with quadrant+ tilt brackets, a parallel ruler riding on side rails, a pen tray, a sheet of+ paper held by two corner clips, a clamp-on two-arm lamp aimed at the board,+ and a stool with a footring+ must_have:+ - "drawing board about 1.0 m by 0.7 m tilted about 20 degrees"+ - "parallel ruler bar spanning the board width"+ - "ruler rides on a rail or wire at each side edge"+ - "pen tray along the bottom edge of the board"+ - "trestle legs with a cross stretcher"+ - "quadrant tilt bracket with a knob at each of the board's 2 sides"+ - "sheet of paper on the board held by clips at 2 corners"+ - "lamp clamped to a top corner of the board"+ - "lamp has 2 arm segments and a shade aimed down at the board"+ - "stool with a round seat about 0.65 m high"+ - "stool footring between its legs"+ - "board top edge about 1.1 m above the floor"+ dimensions_m: {width: 1.1, depth: 0.9, height: 1.2}+ - id: comp_hard_telescope_tripod+ tier: hard+ category: props+ tags: ["axis:multi_material", "axis:decomposition"]+ prompt: >+ a brass refractor telescope on a wooden tripod: main tube with an objective+ ring, a thinner draw tube with an eyepiece cup, a clamping collar and cradle+ on a pan-tilt head, three splayed legs joined by a spreader tray, metal feet+ must_have:+ - "main tube about 0.7 m long and 0.08 m diameter"+ - "objective ring at the front of the tube"+ - "draw tube about 0.05 m diameter extending from the rear, thinner than the main tube"+ - "eyepiece cup at the draw tube end"+ - "clamping collar around the main tube's balance point"+ - "cradle connecting the collar to a pan-tilt head"+ - "tension knob on the head"+ - "3 wooden legs splayed outward, each about 1.2 m long"+ - "triangular spreader tray connecting the legs at about 0.7 m height"+ - "metal feet at the leg tips"+ - "tube tilted upward between 20 and 40 degrees"+ - "eyepiece height about 1.5 m above the ground"+ dimensions_m: {height: 1.7}diff --git a/eval/llm/batteries/teaser_v1_static.yaml b/eval/llm/batteries/teaser_v1_static.yamlnew file mode 100644index 00000000..c2acb9db--- /dev/null+++ b/eval/llm/batteries/teaser_v1_static.yaml@@ -0,0 +1,306 @@+# Teaser showcase battery, split one file per track because a `Battery` binds exactly+# one track + language (bench/run_bench.py). Cross-language prompts inside a track use+# the per-prompt `language:` override. These four files ARE the source; edit them here.+# Regenerate: python /tmp/claude-1000/-home-yipeng/256e9f01-5a86-4d6d-9b4b-3d662f71921d/scratchpad/teaser/make_batteries.py+# Heroes (run with codex:gpt-5.6-sol, everything else agy:gemini-3.6-flash): tsr_obj_penny_farthing, tsr_cad_radial_engine+name: teaser_v1_static+track: static_object+language: blender+description: 'teaser_v1 — static_object slice (7 prompts). Showcase battery: picked for the picture,+ every prompt carries its own `why` for the selection evidence.'+prompts:+- id: tsr_obj_penny_farthing+ tier: hard+ category: vehicles+ tags:+ - hero+ - texture+ - axis:thin_features+ - material:iron+brass+leather+ prompt: 'a Victorian penny-farthing bicycle: a very tall spoked front wheel, a small rear wheel, a curved+ backbone frame, a sprung leather saddle above the front hub, pedals on the front hub, curved handlebars+ and polished brass fittings++ '+ must_have:+ - front wheel about 1.30 m in diameter with at least 32 straight spokes running hub to rim+ - rear wheel about 0.40 m in diameter with at least 16 spokes+ - curved backbone tube sweeping from the front hub head down to the rear fork+ - leather saddle about 0.26 m long on a sprung post above the front wheel+ - two cranks with pedals on the front hub, 180 degrees apart+ - handlebars about 0.55 m wide with two grips+ - solid tyre band on both rims+ - brass or nickel fittings at the hubs, the head and the saddle post, distinct in colour from the black+ frame+ - both wheels touch the ground plane+ - no spoke, fork or backbone tube intersects a wheel rim+ dimensions_m:+ width: 0.6+ depth: 1.65+ height: 1.55+- id: tsr_obj_lighthouse+ tier: medium+ category: buildings+ tags:+ - texture+ - axis:arch_detail+ - material:stone+glass+copper+ prompt: 'a coastal lighthouse on a rock plinth: a tapered stone tower with painted bands, a corbelled+ gallery with a railing, a glazed lantern room with a fresnel lens and a domed copper roof with a weathervane,+ and a keeper''s door at the base++ '+ must_have:+ - tapered tower about 14 m tall, wider at the base than under the gallery+ - at least 3 painted horizontal bands around the tower+ - corbelled gallery ring projecting at least 0.5 m beyond the tower under the lantern room+ - gallery railing with at least 16 balusters+ - lantern room with at least 8 glazed panes between vertical astragals+ - fresnel lens drum visible inside the lantern room+ - domed roof with a finial or weathervane on top+ - keeper's door with a frame and at least 3 small windows up the tower+ - rock plinth the tower stands on, tower base fully seated on it+ - stone tower, glass lantern panes and a copper dome as three distinct materials+ dimensions_m:+ width: 6.0+ depth: 6.0+ height: 18.0+- id: tsr_obj_gramophone+ tier: hard+ category: props+ tags:+ - texture+ - axis:curved_organic+ - material:brass+oak+bakelite+ prompt: 'an antique wind-up gramophone: a big flared brass horn on an elbow, an oak case with a felted+ turntable and a shellac record, a curved tonearm with a sound box, a crank handle on the side and+ turned feet++ '+ must_have:+ - flared horn whose bell is about 0.55 m across, opening from a throat under 0.06 m along a smooth curved+ flare+ - elbow joining the horn throat to the sound box+ - oak case about 0.40 x 0.40 x 0.22 m with a moulded lip+ - turntable platter about 0.30 m in diameter on the case top+ - record disc lying flat on the platter with a centre label+ - curved tonearm from the case to the sound box, with the needle over the record+ - crank handle on the side of the case+ - four turned feet, all touching the ground plane+ - polished brass horn against dark oak and a black record — three clearly different materials+ - the horn does not intersect the case or the platter+ dimensions_m:+ width: 0.6+ depth: 0.55+ height: 0.75+- id: tsr_cad_radial_engine+ tier: hard+ category: mechanisms+ tags:+ - hero+ - texture+ - axis:decomposition+ - material:aluminium+steel+brass+ language: cadquery+ prompt: 'a five-cylinder radial aero engine: a round crankcase with a bolted front cover, five finned+ cylinder barrels in a star, domed heads with rocker covers, a pushrod tube per cylinder, an exhaust+ stub on each head and a splined propeller hub++ '+ must_have:+ - five cylinder barrels evenly spaced 72 degrees apart around the crankcase axis+ - at least 8 cooling fins on every barrel, all the same pitch+ - domed head with a rocker cover on each cylinder+ - one pushrod tube per cylinder running from the crankcase up to its head+ - circular crankcase about 0.36 m in diameter+ - front cover with at least 8 bolt heads on a circular pattern+ - propeller hub on the crankcase axis with at least 6 bolt holes+ - an exhaust stub on each head, all pointing the same way around the ring+ - pale aluminium barrels, a darker steel crankcase and brass pushrod ends as distinct part colours+ - no barrel, pushrod or exhaust stub intersects a neighbour+ dimensions_m:+ width: 0.9+ depth: 0.42+ height: 0.9+- id: tsr_cad_portafilter+ tier: medium+ category: tools+ tags:+ - texture+ - material:steel+walnut+ - product_shot+ language: cadquery+ prompt: 'a double-spout espresso portafilter: a polished stainless basket holder with three locking+ lugs, a knurled collar, two tapered spouts and a turned walnut handle on a chrome ferrule++ '+ must_have:+ - cylindrical basket body about 0.075 m outside diameter and 0.035 m deep+ - filter basket inside the body with a visible rim below the body lip+ - three locking lugs evenly spaced around the body+ - knurled or ribbed collar between the body and the ferrule+ - two tapered spouts under the body, splayed at least 20 degrees apart+ - turned walnut handle about 0.12 m long, thicker in the middle, on the body axis+ - chrome ferrule joining handle to body+ - steel body and chrome ferrule against a dark wood handle, three distinct part colours+ - handle, ferrule and body are coaxial and touch without gaps+ dimensions_m:+ width: 0.09+ depth: 0.24+ height: 0.09+- id: tsr_tjs_arcade_cabinet+ tier: medium+ category: props+ tags:+ - emissive+ - material:laminate+glass+plastic+ language: threejs+ prompt: 'an upright 1980s arcade cabinet: a sloped control panel with a ball-top joystick and six buttons,+ a recessed CRT screen behind a bezel, a glowing backlit marquee, a speaker grille, side art panels,+ a coin door and T-molding trim++ '+ must_have:+ - cabinet about 1.75 m tall, 0.70 m wide, 0.85 m deep, standing on the ground plane+ - sloped control panel with one ball-top joystick and at least 6 round buttons in two rows+ - screen recessed behind a bezel, the screen surface emissive so it reads as switched on+ - backlit marquee across the top, emissive and brighter than the cabinet body+ - speaker grille with at least 12 holes above the screen+ - coin door with two coin slots and a coin return cup+ - T-molding trim strip running along both side edges+ - side art panel in a saturated colour distinct from the black cabinet body+ - no button or joystick sinks below the control panel surface+ dimensions_m:+ width: 0.7+ depth: 0.85+ height: 1.75+- id: tsr_tjs_deco_chandelier+ tier: hard+ category: lighting+ tags:+ - instancing+ - material:brass+glass+ - radial_symmetry+ language: threejs+ prompt: 'an art-deco brass chandelier: a central stem on a ceiling canopy and chain, two tiers of curved+ arms carrying frosted candle shades, stepped faceted glass drops hanging in rings from each tier and+ a fluted glass bowl at the crown++ '+ must_have:+ - central brass stem with a ceiling canopy and a chain of at least 5 links above it+ - upper tier of 6 curved arms and a lower tier of 8 curved arms, each tier evenly spaced+ - a frosted candle shade at the end of every arm+ - at least 40 faceted glass drops hanging in even rings below the two tiers+ - fluted glass bowl at the crown under the stem+ - overall diameter about 0.90 m and overall height about 1.10 m+ - warm brass arms and stem against clear or frosted glass drops and shades+ - every drop hangs free below its tier and no drop intersects an arm+ dimensions_m:+ width: 0.9+ depth: 0.9+ height: 1.1++# ===========================================================================+# SHARPENED RE-RUNS — these are the prompts the teaser page actually ships.+# The original id each one replaces is kept above it, so the pair stays legible;+# see docs/TEASER.md for the picture each attempt produced.+# NOTE: BenchPrompt (bench/run_bench.py) carries only id/prompt/tier/category/+# must_have/dimensions_m/tags/language. The re-runs also used --style, --must-not+# and --candidates on the CLI; those are recorded as comments here rather than as+# keys, because an unknown key is dropped silently.+# ===========================================================================+# SHIPPED. Supersedes tsr_cad_radial_engine (same judged 0.600, broken picture: one barrel+# buried through the ground plane, exhaust stubs reading as detached sticks, no stand, all grey).++# CLI flags used, not expressible in this schema:+# --style "clean machined-metal product shot: pale satin aluminium fins, dark grey cast steel crankcase, warm polished brass pushrod tubes and black rocker covers, crisp fillets and chamfers on every machined edge"+# --must-not "a cylinder barrel buried in or passing through the ground plane"+# --must-not "exhaust stubs or pushrods detached from the heads, floating in space, or pointing in inconsistent directions"+# --must-not "an all-grey untextured look with no colour separation between crankcase, barrels and pushrods"+# --candidates 2+# --profile quality+- id: tsr_cad_radial_engine_v2+ tier: hard+ category: mechanisms+ tags:+ - teaser+ - rerun+ - hero+ language: cadquery+ prompt: 'a five-cylinder radial aero engine displayed on a low steel stand, in the correct museum orientation:+ ONE cylinder points straight UP at top dead centre and the other four are at 72 degree steps, so there+ is NO cylinder pointing straight down and the engine sits clear of the ground on its stand. A dark grey+ machined steel crankcase carries a bolted circular front cover and a splined propeller hub on the axis;+ each of the five barrels is a pale satin aluminium tube wrapped in evenly pitched square cooling fins,+ capped by a domed head with a black rocker cover; a slim brass pushrod tube lies flush along the front+ of each barrel from the crankcase to its rocker cover; a short polished steel exhaust stub leaves each+ head, all five swept the same way around the ring.++ '+ must_have:+ - one cylinder barrel pointing straight up at top dead centre, the other four at 72 degree steps around+ the crankcase axis+ - no cylinder barrel pointing straight down; the lowest part of the engine is the steel stand, and the stand+ sits flat on the ground plane+ - a low steel stand under the crankcase, welded to it, carrying the whole engine clear of the ground+ - at least 8 cooling fins on every barrel, all the same pitch and the same outer diameter+ - a domed head with a black rocker cover on each cylinder, sitting flush on the top of its barrel+ - one brass pushrod tube per cylinder, lying flush along the barrel from the crankcase to its rocker cover,+ touching both+ - circular crankcase about 0.36 m in diameter with a front cover carrying at least 8 bolt heads on a circular+ pattern+ - propeller hub on the crankcase axis with at least 6 bolt holes+ - one exhaust stub per head, each rooted flush in its head, all five angled the same way around the ring+ - pale aluminium barrels, a darker steel crankcase, brass pushrod tubes and black rocker covers as four+ clearly distinct part colours+ - no barrel, pushrod or exhaust stub intersects or floats away from a neighbour+ dimensions_m:+ width: 0.9+ depth: 0.42+ height: 0.98+# SHIPPED at a LOWER judged score than the run it replaces (0.600 vs 0.841): v1 was a clean brass+# armature with plain white shades and ZERO hanging crystal, i.e. none of the sparkle the prompt+# was picked for.++# CLI flags used, not expressible in this schema:+# --style "expensive lighting-catalogue product shot: warm polished brass against clear cut crystal and frosted glass, every facet catching a highlight, strong specular sparkle, no flat matte surfaces"+# --must-not "a chandelier with no hanging crystal drops, or with the drops too small or too few to see"+# --must-not "plain white cylinder shades with no finial"+# --must-not "drops merged into the arms or hidden inside the tiers"+# --candidates 2+# --profile quality+- id: tsr_tjs_deco_chandelier_v2+ tier: hard+ category: lighting+ tags:+ - teaser+ - rerun+ language: threejs+ prompt: 'an art-deco brass chandelier dripping with cut crystal: a stepped brass ceiling canopy and a chain+ drop to a fluted central stem; two tiers of curved brass arms, six above and eight below, each ending+ in a frosted glass candle sleeve with a small tapered flame-tip finial; and hanging from the rim of BOTH+ tiers, in even concentric rings, more than forty stepped faceted crystal drops - long slender prisms with+ cut facets, each hanging free on its own short brass link so the whole lower half of the fixture reads+ as a curtain of glass; a fluted clear glass bowl at the crown under the stem.++ '+ must_have:+ - central fluted brass stem with a stepped ceiling canopy and a chain of at least 5 visible links above+ it+ - an upper tier of 6 curved brass arms and a lower tier of 8 curved brass arms, each tier evenly spaced+ - a frosted glass candle sleeve with a small tapered flame-tip finial at the end of every arm+ - MORE THAN 40 faceted crystal drops, clearly visible as separate slender prisms, hanging in even concentric+ rings from the rims of BOTH tiers+ - each crystal drop is at least 0.06 m long, hangs vertically on its own short brass link, and hangs free+ below its tier+ - the rings of drops are the dominant visual feature of the lower half of the chandelier+ - fluted clear glass bowl at the crown under the stem+ - overall diameter about 0.90 m and overall height about 1.10 m+ - warm brass arms and stem clearly distinct in colour from the clear crystal drops and the frosted white+ shades+ - no drop intersects an arm, a shade or another drop+ dimensions_m:+ width: 0.9+ depth: 0.9+ height: 1.1diff --git a/eval/llm/build_prompts.py b/eval/llm/build_prompts.pyindex 4133c48e..31c52e74 100644--- a/eval/llm/build_prompts.py+++ b/eval/llm/build_prompts.py@@ -19,7 +19,7 @@ Suites built here (sizes are what the assets give):3dcodebench_img4text 212 Blender 4 views + descriptionheldout_blender 103, heldout_cadquery 200, heldout_openscad 50, heldout_glsl 200, heldout_threejs 40(3dcodeverse/3dcodeverse-2m finetune/test/*.parquet, never trained on)- harness_<battery> rubric-judged batteries copied from harness/bench/prompts (no GT; scored by+ harness_<battery> rubric-judged batteries in eval/llm/batteries (no GT; scored byexecution + optional VLM judge), one suite per YAMLSystem prompts are the exact strings used in finetune/docs/REPORT.md so numbers stay comparable; the@@ -205,7 +205,7 @@ def build_batteries() -> None:"reference": {"code": None, "mesh": None, "renders": []},"meta": {"tier": p.get("tier"), "category": p.get("category"), "must_have": p.get("must_have", []),"dimensions_m": p.get("dimensions_m"), "tags": p.get("tags", []), "track": bat.get("track"),- "source": f"harness/bench/prompts/{y.name}"}})+ "source": f"eval/llm/batteries/{y.name}"}})if rows:write_jsonl(rows, f"harness_{bat['name']}")diff --git a/eval/llm/config.py b/eval/llm/config.pyindex 137b48e4..71b2163e 100644--- a/eval/llm/config.py+++ b/eval/llm/config.py@@ -25,9 +25,8 @@ from pathlib import PathPKG_DIR = Path(__file__).resolve().parentPROMPTS_DIR = PKG_DIR / "data" / "prompts"-# the harness batteries are ONE set of files, owned by the harness evaluation next door (eval/bench/prompts);-# these ten are the ones a bare LLM can be asked one-shot (static + graphics, no plan, no tools)-BATTERIES_DIR = PKG_DIR.parent / "bench" / "prompts"+# the ten harness prompt batteries a bare LLM can be asked one-shot (static + graphics, no plan, no tools)+BATTERIES_DIR = PKG_DIR / "batteries"HARNESS_BATTERIES = ("articulated_v2", "compare_v4", "complexity_v3", "fancy_v1_blender", "fancy_v1_cadquery","fancy_v1_threejs", "graphics_v2", "static_objects_v1", "static_objects_v2", "teaser_v1_static")diff --git a/eval/pyproject.toml b/eval/pyproject.tomlindex 4d3fddb0..07a18f67 100644--- a/eval/pyproject.toml+++ b/eval/pyproject.toml@@ -1,19 +1,18 @@-# Tooling config for the evaluation code. Nothing here is installed: `bench` and `llm` are run-# from this directory (`cd eval && python bench/<script>.py …`, `cd eval && python -m pytest`).+# Tooling config for the evaluation code. Nothing here is installed: `llm` runs from this directory.[tool.ruff]line-length = 100target-version = "py313"-extend-exclude = ["bench/out", "llm"] # llm/ keeps its own style; bench/out is run data+extend-exclude = ["llm"] # llm/ keeps its own style[tool.ruff.lint]select = ["E", "W", "F", "I", "B", "UP", "SIM"]ignore = ["E501"][tool.ruff.lint.isort]-known-first-party = ["bench", "codeverse3d", "tests"]+known-first-party = ["llm", "codeverse3d"][tool.pytest.ini_options]-testpaths = ["tests"]+testpaths = ["llm/tests"]timeout = 600addopts = ["-m", "not live", "-n", "auto", "--dist", "worksteal"]markers = [