3D engine that AI agents drive
Add light-pick-hard suite task and measure branches on it (part of #67)
Nguyen Xuan An2bde7eea6c3c
diff --git a/bench/agent-modeling/README.md b/bench/agent-modeling/README.mdindex a22882b0..ca3cd491 100644--- a/bench/agent-modeling/README.md+++ b/bench/agent-modeling/README.md@@ -266,7 +266,7 @@ Result: agents ignored branches in all 12 runs. Sonnet wrote the right shade in## Best of three, picked from a shadow (2026-11-01, issue #67)-`render-pick` starts from a gallery: a floor, a back wall, a painting on it, a pillar 1.5 m in front of the painting and one shadow-casting spotlight at [-1.8, 2.4, 1.5], aimed at the painting. The prompt offers two more places, [0, 2.4, 1.5] and [1.8, 2.4, 1.5], and asks for the one from which the whole painting is lit with no pillar shadow on it. It gives no shadow geometry and does not mention branches. The check `litPainting` reads the exported recipe and traces rays itself: exactly one spot light, at an offered place, aimed at the painting's centre (0.05 rad), a cone that reaches every corner (at most 0.9 rad), shadows on, and none of 15 points on the painting's front face is hit by a ray-to-box test against the pillar. Computed over the 15 points, the pillar hides 6 from the first place, 6 from the second and none from the third, so only the third place passes. The pillar, painting, wall and floor must stay put. `tests/agent-modeling-suite.test.js` grades the reference and each shortcut (a shadowed place, shadows off, a light off the offered places, two lights, a moved pillar). The wall and floor are not graded beyond part count.+`render-pick` starts from a gallery: a floor, a back wall, a painting on it, a pillar 1.5 m in front of the painting and one shadow-casting spotlight at [-1.8, 2.4, 1.5], aimed at the painting. The prompt offers two more places, [0, 2.4, 1.5] and [1.8, 2.4, 1.5], and asks for the one from which the whole painting is lit with no pillar shadow on it. It gives no shadow geometry and does not mention branches. The check `litPainting` reads the exported recipe and traces rays itself: exactly one spot light, at an offered place, aimed at the painting's centre (0.05 rad), a cone that reaches every corner (at most 0.9 rad), shadows on, and none of 861 points (a 41 by 21 grid, 15 points when the task was first measured) on the painting's front face is hit by a ray-to-box test against the pillar. Computed over the 15 points, the pillar hides 6 from the first place, 6 from the second and none from the third, so only the third place passes. The pillar, painting, wall and floor must stay put. `tests/agent-modeling-suite.test.js` grades the reference and each shortcut (a shadowed place, shadows off, a light off the offered places, two lights, a moved pillar). The wall and floor are not graded beyond part count.| Model | Branches | Pass | Calls (runs) | Errors | Cost | `branch` calls | `scene_look` calls || --- | --- | --- | --- | --- | --- | --- | --- |@@ -279,6 +279,19 @@ Result (measured): all 10 runs passed and none used branches. Sonnet picked theThe other half of the issue, `npm run smoke:agent-branches` on SwiftShader: passed (fork, shared snapshot, review, labels, keep, hostile names), including the label check.+## Best of five lights, none of them clear (2026-11-03, issue #67)++`light-pick-hard` starts from a gallery with a 2 m by 1.2 m painting, six thin posts at different depths and heights, and a spotlight at the first of five offered places (x = -2.4, -1.2, 0, 1.2, 2.4 at height 2.6, depth 1.5). Every place puts some post shadow on the painting. The prompt asks for the place that leaves the least of it in shadow, and does not mention branches. The shares in shadow, over a 41 by 21 grid of the painting, are 24, 24, 37, 10 and 27 per cent, so the fourth place wins by 14 points and no place is clear. The posts were found by a random search for a layout whose best place is neither the start, the middle nor an end and wins by a wide margin. The check `litPainting` with `least: true` needs one spotlight at an offered place, aimed at the painting's centre, shadows on, a cone that reaches the painting, and a shadow share no larger than any other offered place; the posts and painting must stay put. `tests/agent-modeling-suite.test.js` grades the reference, each of the four other places and the shortcuts (shadows off, a light off the offered places, two lights, a moved post).++| Model | Branches | Pass | Calls (runs) | Errors | Cost | `branch` calls | `scene_look` calls |+| --- | --- | --- | --- | --- | --- | --- | --- |+| Haiku | on | 3/3 | 5, 9, 9 | 0 | $0.25 | 0, 5, 6 | 1, 0, 0 (`scene_review`: 0, 1, 2) |+| Haiku | off | 3/3 | 5, 5, 4 | 0 | $0.20 | 0 | 1, 1, 1 |+| Sonnet | on | 3/3 | 3, 3, 4 | 1 | $1.04 | 0 | 0 |+| Sonnet | off | 3/3 | 3, 3, 4 | 1 | $1.04 | 0 | 0 |++Calls count tool calls except `ToolSearch`. Result (measured, 3 runs per cell): branches did not save calls. Haiku used branches in 2 of 3 runs with branches on, 5 and 6 calls each, and ended with more calls (mean 7.7) than with branches off (mean 4.7), where it rendered one view with `scene_look` and picked. Both settings passed 3/3. Sonnet again used no branches and no render: 3 calls (create, apply, export) in two runs and 4 in the third (one `Bash` call) with branches on or off, and all 6 runs picked the best place. Sonnet's runs spent about 3,700 to 4,500 output tokens against Haiku's 10,000 to 15,000, so it worked the answer out before its first apply; the transcripts do not say how. This task is harder to trace than `render-pick` (the shares are 10 to 37 per cent from six posts), yet Sonnet still solved it with no look, so a trace this long is not out of its reach. The issue's calls condition is not met: no model made fewer calls with branches, and Haiku made more. I did not change the task after the runs. What this does not show: a judgement a model cannot compute (colour, exposure, composition) where the best of several variants shows only in a render, which needs a check that renders instead of tracing rays.+## A walk around two holes and a rock (2026-11-01, issue #64)`terrain-path-walk` builds 40 m of hills with two round openings (radius 3 m at the origin, 2.5 m at [9, 9]) and a 3 m by 2 m by 3 m `rock` at [-8, -8], then walks a `walker` node from [-15, -15] to [15, 15] in 10 s. The check on the exported recipe samples 41 times: the position stays within 1.5 m of the ground height, at least 0.3 m outside both openings and at least 0.3 m from the rock's footprint, and starts and ends within 0.75 m of the two points. The `through` check confirms both openings are cut clean through. The robot GLB cannot stand on a recipe terrain (asset scenes hold no recipe geometry), so the character is a box node, like `terrain-hole-walk`. The prompt does not mention the path query: agents find it in the `scene_inspect` schema (`path`) and in `scene_capabilities` `navigation`. `pathQueries` in each record counts `scene_inspect` calls that passed `path`.diff --git a/bench/agent-modeling/check.mjs b/bench/agent-modeling/check.mjsindex efcc34ff..0878d0a8 100644--- a/bench/agent-modeling/check.mjs+++ b/bench/agent-modeling/check.mjs@@ -101,8 +101,8 @@ function checkShell(check, measures) {return { pass: problems.length === 0, detail: problems.join('; ') || `${check.node} between ${round(nearest)} and ${round(farthest)} m of the centre` };}-const PAINTING_SAMPLES_ACROSS = 5;-const PAINTING_SAMPLES_UP = 3;+const PAINTING_SAMPLES_ACROSS = 41;+const PAINTING_SAMPLES_UP = 21;const AIM_TOLERANCE = 0.05;const POSITION_TOLERANCE = 0.05;@@ -131,11 +131,32 @@ const angleBetween = (first, second) => {};// The one spotlight must sit at an offered place, aim at the painting with a cone that reaches all of it and cast-// shadows, and no ray from it to the front face of the painting may meet the blocker.+// shadows. With `blocker` (one node) no ray from it to the front face of the painting may meet that node. With+// `blockers` (several nodes) and `least: true`, the share of painting points in shadow at the chosen place must be the+// smallest over all offered places, which only a render or a long trace tells.+function shadowedPoints(position, painting, blockerBounds) {+ const frontZ = painting.max[2];+ let shadowed = 0;+ let total = 0;+ for (let across = 0; across < PAINTING_SAMPLES_ACROSS; across++) {+ for (let up = 0; up < PAINTING_SAMPLES_UP; up++) {+ const point = [+ painting.min[0] + (painting.max[0] - painting.min[0]) * across / (PAINTING_SAMPLES_ACROSS - 1),+ painting.min[1] + (painting.max[1] - painting.min[1]) * up / (PAINTING_SAMPLES_UP - 1),+ frontZ,+ ];+ total++;+ if (blockerBounds.some(blocker => segmentHitsBox(position, point, blocker.min, blocker.max))) shadowed++;+ }+ }+ return { shadowed, total };+}+function checkLitPainting(check, measures) {const painting = measures.nodeBounds(check.painting);- const blocker = measures.nodeBounds(check.blocker);- if (!painting || !blocker) return { pass: false, detail: `no drawn node ${check.painting} or ${check.blocker}` };+ const blockerIds = check.blockers ?? [check.blocker];+ const blockerBounds = blockerIds.map(id => measures.nodeBounds(id));+ if (!painting || blockerBounds.some(bounds => !bounds)) return { pass: false, detail: `no drawn node ${check.painting} or ${blockerIds.join(', ')}` };const lights = measures.spotLights;if (lights.length !== 1) return { pass: false, detail: `${lights.length} spot lights, need exactly 1` };const [light] = lights;@@ -151,19 +172,14 @@ function checkLitPainting(check, measures) {if (aimError > AIM_TOLERANCE) problems.push(`aimed ${round(aimError)} rad off the centre`);if (light.angle < reach || light.angle > check.maxAngle) problems.push(`cone ${round(light.angle)} rad, needs ${round(reach)} to ${check.maxAngle}`);if (light.castShadow !== true) problems.push('shadows are off');- let shadowed = 0;- for (let across = 0; across < PAINTING_SAMPLES_ACROSS; across++) {- for (let up = 0; up < PAINTING_SAMPLES_UP; up++) {- const point = [- painting.min[0] + (painting.max[0] - painting.min[0]) * across / (PAINTING_SAMPLES_ACROSS - 1),- painting.min[1] + (painting.max[1] - painting.min[1]) * up / (PAINTING_SAMPLES_UP - 1),- frontZ,- ];- if (segmentHitsBox(light.position, point, blocker.min, blocker.max)) shadowed++;- }+ const { shadowed, total } = shadowedPoints(light.position, painting, blockerBounds);+ if (check.least) {+ const shares = check.positions.map(place => shadowedPoints(place, painting, blockerBounds).shadowed);+ if (shadowed > Math.min(...shares)) problems.push(`place ${placeIndex + 1} shades ${shadowed} of ${total} points, place ${shares.indexOf(Math.min(...shares)) + 1} shades ${Math.min(...shares)}`);+ } else if (shadowed > 0) {+ problems.push(`${shadowed} of ${total} painting points are in shadow`);}- if (shadowed > 0) problems.push(`${shadowed} of ${PAINTING_SAMPLES_ACROSS * PAINTING_SAMPLES_UP} painting points are in shadow`);- return { pass: problems.length === 0, detail: problems.join('; ') || `place ${placeIndex + 1}, no shadow on the painting` };+ return { pass: problems.length === 0, detail: problems.join('; ') || `place ${placeIndex + 1}, ${shadowed} of ${total} points in shadow` };}function checkOne(check, measures) {diff --git a/bench/agent-modeling/colonnade.mjs b/bench/agent-modeling/colonnade.mjsnew file mode 100644index 00000000..4eee3d39--- /dev/null+++ b/bench/agent-modeling/colonnade.mjs@@ -0,0 +1,74 @@+// A gallery with six thin posts and a painting, and five places for the one spotlight. The posts stand at different+// depths and heights, so each place throws a different pattern of thin shadows over the painting. The prompt asks for the+// place that leaves the least of the painting in shadow. Shadow shares at the five places are 24, 24, 37, 10 and 27 per+// cent, so no place is clear and the best cannot be read off the layout: a render of each place (or a fork per place) tells+// it, and a trace by hand means testing 6 posts against 5 places over the whole painting. The check traces the rays itself,+// so it grades the pick and not the method.++export const COLONNADE_LIGHT_POSITIONS = [[-2.4, 2.6, 1.5], [-1.2, 2.6, 1.5], [0, 2.6, 1.5], [1.2, 2.6, 1.5], [2.4, 2.6, 1.5]];+export const COLONNADE_BEST_PLACE = 3;+const PAINTING_CENTRE = [0, 1.5, -1.98];+const SPOT_ANGLE = 0.5;++// Each post: id, x, depth z (the post is 0.08 m thick in z), bottom, top, width in x.+const POSTS = [+ ['post-1', 1.745, 0.611, 0, 2.632, 0.088],+ ['post-2', -1.345, -0.333, 0.737, 3.082, 0.058],+ ['post-3', -0.125, 0.845, 0, 2.736, 0.095],+ ['post-4', -1.048, -0.926, 1.293, 3.17, 0.07],+ ['post-5', -0.869, -0.014, 1.077, 2.877, 0.066],+ ['post-6', -0.168, -0.856, 1.351, 2.314, 0.084],+];++function aimedAt(position, target) {+ const offset = target.map((value, axis) => value - position[axis]);+ const length = Math.hypot(...offset);+ return offset.map(value => Number((value / length).toFixed(4)));+}++function colonnadeRecipe(lightPosition) {+ const parts = [+ { id: 'floor', geometry: 'block', scale: [6, 0.05, 6], translation: [0, -0.025, 0] },+ { id: 'wall', geometry: 'block', scale: [6, 3.4, 0.1], translation: [0, 1.7, -2.05] },+ { id: 'painting', geometry: 'block', scale: [2, 1.2, 0.04], translation: PAINTING_CENTRE },+ ...POSTS.map(([id, x, z, bottom, top, width]) => ({+ id, geometry: 'block', scale: [width, top - bottom, 0.08], translation: [x, (top + bottom) / 2, z],+ })),+ ];+ return {+ format: '67js.scene', schemaVersion: 1,+ geometries: [{ id: 'block', type: 'box', bevel: 0 }],+ materials: [{ id: 'plaster', color: [0.8, 0.78, 0.72] }],+ nodes: parts.map(part => ({ material: 'plaster', ...part })),+ environment: {+ spotLights: [{+ position: lightPosition, direction: aimedAt(lightPosition, PAINTING_CENTRE),+ angle: SPOT_ANGLE, intensity: 60, castShadow: true,+ }],+ },+ };+}++/** The scene as the task starts: the spotlight at the first place. */+export const COLONNADE_START = colonnadeRecipe(COLONNADE_LIGHT_POSITIONS[0]);++/** The right pick: the fourth place, which leaves the least of the painting in shadow. */+export const COLONNADE_SOLUTION = colonnadeRecipe(COLONNADE_LIGHT_POSITIONS[COLONNADE_BEST_PLACE]);++export const COLONNADE_PROMPT = `The gallery scene below has a floor, a back wall, a 2 m by 1.2 m painting on the wall and six thin posts standing at+different depths and heights in the room. One spotlight with shadows on lights the painting; it is at [-2.4, 2.6, 1.5] and+aimed at the painting's centre. Four other places could hold it: [-1.2, 2.6, 1.5], [0, 2.6, 1.5], [1.2, 2.6, 1.5] and+[2.4, 2.6, 1.5]. Every place puts some post shadow on the painting. Keep the spotlight at the one place that leaves the+least of the painting in shadow. Leave it aimed at the painting's centre, with shadows on and its cone wide enough to+reach the whole painting. Leave the floor, wall, painting and posts as they are, and keep this one light.`;++export const COLONNADE_CHECKS = [+ { kind: 'lint' },+ { kind: 'parts', min: 9, max: 9 },+ { kind: 'centreAt', node: 'painting', position: PAINTING_CENTRE },+ ...POSTS.map(([id, x, z, bottom, top]) => ({ kind: 'centreAt', node: id, position: [x, (top + bottom) / 2, z] })),+ {+ kind: 'litPainting', painting: 'painting', blockers: POSTS.map(([id]) => id), least: true,+ positions: COLONNADE_LIGHT_POSITIONS, maxAngle: 0.9,+ },+];diff --git a/bench/agent-modeling/references.mjs b/bench/agent-modeling/references.mjsindex ece5d7f5..581a47d4 100644--- a/bench/agent-modeling/references.mjs+++ b/bench/agent-modeling/references.mjs@@ -9,6 +9,7 @@ import { walkerRecipe } from './walker.mjs';import { KETTLE_RESIZED, KETTLE_RESIZED_PLAIN } from './kettle.mjs';import { LAMP_SOLUTION } from './lamp.mjs';import { GALLERY_SOLUTION } from './gallery.mjs';+import { COLONNADE_SOLUTION } from './colonnade.mjs';import { DESK_FINAL, DESK_FINAL_PLAIN } from './desk.mjs';import { CABINET_FINAL, CABINET_FINAL_PLAIN } from './cabinet.mjs';@@ -168,6 +169,7 @@ function robotArmSolver(root, middle, end, target) {export const REFERENCES = {'best-of-three': LAMP_SOLUTION,'render-pick': GALLERY_SOLUTION,+ 'light-pick-hard': COLONNADE_SOLUTION,'kettle-resize': KETTLE_RESIZED,'kettle-resize-plain': KETTLE_RESIZED_PLAIN,'desk-resize': DESK_FINAL,diff --git a/bench/agent-modeling/results/2026-11-03-05-41-haiku-hard-haiku-on.json b/bench/agent-modeling/results/2026-11-03-05-41-haiku-hard-haiku-on.jsonnew file mode 100644index 00000000..cfd03fe5--- /dev/null+++ b/bench/agent-modeling/results/2026-11-03-05-41-haiku-hard-haiku-on.json@@ -0,0 +1,194 @@+{+ "run": "2026-11-03-05-41-haiku-hard-haiku-on",+ "model": "claude-haiku-5-5",+ "date": "2026-11-03T05:45:57.277Z",+ "skill": false,+ "branches": "on",+ "relations": "on",+ "runsPerTask": 3,+ "passed": 3,+ "tasks": 3,+ "toolCalls": 23,+ "costUsd": 0.25,+ "byModel": {+ "claude-haiku-5-5": {+ "runs": 3,+ "passed": 3,+ "successRate": 1,+ "toolCalls": 23,+ "toolErrors": 0,+ "inputTokens": 1842818,+ "outputTokens": 36593,+ "costUsd": 0.25,+ "tasks": {+ "light-pick-hard": {+ "runs": 3,+ "passed": 3,+ "successRate": 1,+ "toolCalls": 23,+ "toolErrors": 0,+ "inputTokens": 1842818,+ "outputTokens": 36593,+ "costUsd": 0.25+ }+ }+ }+ },+ "records": [+ {+ "task": "light-pick-hard",+ "attempt": 1,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-haiku-5-5",+ "toolCalls": 5,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_capabilities": 1,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_look": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-41-haiku-hard-haiku-on/look/1/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-41-haiku-hard-haiku-on/look/1/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-41-haiku-hard-haiku-on/look/1/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 7,+ "costUsd": 0.05559846,+ "inputTokens": 446255,+ "outputTokens": 14889,+ "wallSeconds": 74,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "light-pick-hard",+ "attempt": 2,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-haiku-5-5",+ "toolCalls": 9,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_inspect": 1,+ "scene_apply": 5,+ "scene_review": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 5,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-41-haiku-hard-haiku-on/look/2/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-41-haiku-hard-haiku-on/look/2/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-41-haiku-hard-haiku-on/look/2/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 11,+ "costUsd": 0.15583933000000003,+ "inputTokens": 710783,+ "outputTokens": 11866,+ "wallSeconds": 85,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "light-pick-hard",+ "attempt": 3,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-haiku-5-5",+ "toolCalls": 9,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_apply": 5,+ "scene_review": 2,+ "scene_export_recipe": 1+ },+ "branchCalls": 6,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-41-haiku-hard-haiku-on/look/3/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-41-haiku-hard-haiku-on/look/3/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-41-haiku-hard-haiku-on/look/3/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 11,+ "costUsd": 0.041729640000000005,+ "inputTokens": 685780,+ "outputTokens": 9838,+ "wallSeconds": 76,+ "exitCode": 0,+ "stopped": "success"+ }+ ]+}diff --git a/bench/agent-modeling/results/2026-11-03-05-45-haiku-hard-haiku-off.json b/bench/agent-modeling/results/2026-11-03-05-45-haiku-hard-haiku-off.jsonnew file mode 100644index 00000000..0d820e5d--- /dev/null+++ b/bench/agent-modeling/results/2026-11-03-05-45-haiku-hard-haiku-off.json@@ -0,0 +1,194 @@+{+ "run": "2026-11-03-05-45-haiku-hard-haiku-off",+ "model": "claude-haiku-5-5",+ "date": "2026-11-03T05:50:07.196Z",+ "skill": false,+ "branches": "off",+ "relations": "on",+ "runsPerTask": 3,+ "passed": 3,+ "tasks": 3,+ "toolCalls": 14,+ "costUsd": 0.2,+ "byModel": {+ "claude-haiku-5-5": {+ "runs": 3,+ "passed": 3,+ "successRate": 1,+ "toolCalls": 14,+ "toolErrors": 0,+ "inputTokens": 1471163,+ "outputTokens": 37643,+ "costUsd": 0.2,+ "tasks": {+ "light-pick-hard": {+ "runs": 3,+ "passed": 3,+ "successRate": 1,+ "toolCalls": 14,+ "toolErrors": 0,+ "inputTokens": 1471163,+ "outputTokens": 37643,+ "costUsd": 0.2+ }+ }+ }+ },+ "records": [+ {+ "task": "light-pick-hard",+ "attempt": 1,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-haiku-5-5",+ "toolCalls": 5,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 2,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_look": 1,+ "scene_export_recipe": 1,+ "scene_dispose": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-45-haiku-hard-haiku-off/look/1/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-45-haiku-hard-haiku-off/look/1/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-45-haiku-hard-haiku-off/look/1/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 8,+ "costUsd": 0.02655633,+ "inputTokens": 584802,+ "outputTokens": 11472,+ "wallSeconds": 80,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "light-pick-hard",+ "attempt": 2,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-haiku-5-5",+ "toolCalls": 5,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_inspect": 1,+ "scene_apply": 1,+ "scene_look": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-45-haiku-hard-haiku-off/look/2/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-45-haiku-hard-haiku-off/look/2/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-45-haiku-hard-haiku-off/look/2/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 7,+ "costUsd": 0.025110910000000004,+ "inputTokens": 408118,+ "outputTokens": 11864,+ "wallSeconds": 69,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "light-pick-hard",+ "attempt": 3,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-haiku-5-5",+ "toolCalls": 4,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_look": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-45-haiku-hard-haiku-off/look/3/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-45-haiku-hard-haiku-off/look/3/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-45-haiku-hard-haiku-off/look/3/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 6,+ "costUsd": 0.14809182999999998,+ "inputTokens": 478243,+ "outputTokens": 14307,+ "wallSeconds": 90,+ "exitCode": 0,+ "stopped": "success"+ }+ ]+}diff --git a/bench/agent-modeling/results/2026-11-03-05-50-sonnet-hard-sonnet-on.json b/bench/agent-modeling/results/2026-11-03-05-50-sonnet-hard-sonnet-on.jsonnew file mode 100644index 00000000..4932acf0--- /dev/null+++ b/bench/agent-modeling/results/2026-11-03-05-50-sonnet-hard-sonnet-on.json@@ -0,0 +1,190 @@+{+ "run": "2026-11-03-05-50-sonnet-hard-sonnet-on",+ "model": "claude-sonnet-5-5",+ "date": "2026-11-03T05:52:34.904Z",+ "skill": false,+ "branches": "on",+ "relations": "on",+ "runsPerTask": 3,+ "passed": 3,+ "tasks": 3,+ "toolCalls": 10,+ "costUsd": 1.04,+ "byModel": {+ "claude-sonnet-5-5": {+ "runs": 3,+ "passed": 3,+ "successRate": 1,+ "toolCalls": 10,+ "toolErrors": 1,+ "inputTokens": 1176456,+ "outputTokens": 12071,+ "costUsd": 1.04,+ "tasks": {+ "light-pick-hard": {+ "runs": 3,+ "passed": 3,+ "successRate": 1,+ "toolCalls": 10,+ "toolErrors": 1,+ "inputTokens": 1176456,+ "outputTokens": 12071,+ "costUsd": 1.04+ }+ }+ }+ },+ "records": [+ {+ "task": "light-pick-hard",+ "attempt": 1,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 3,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-50-sonnet-hard-sonnet-on/look/1/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-50-sonnet-hard-sonnet-on/look/1/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-50-sonnet-hard-sonnet-on/look/1/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 5,+ "costUsd": 0.34067140000000007,+ "inputTokens": 365667,+ "outputTokens": 3825,+ "wallSeconds": 39,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "light-pick-hard",+ "attempt": 2,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 3,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-50-sonnet-hard-sonnet-on/look/2/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-50-sonnet-hard-sonnet-on/look/2/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-50-sonnet-hard-sonnet-on/look/2/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 5,+ "costUsd": 0.33698690000000003,+ "inputTokens": 363648,+ "outputTokens": 3685,+ "wallSeconds": 38,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "light-pick-hard",+ "attempt": 3,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 4,+ "toolErrors": 1,+ "toolsByName": {+ "ToolSearch": 1,+ "Bash": 1,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-50-sonnet-hard-sonnet-on/look/3/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-50-sonnet-hard-sonnet-on/look/3/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-50-sonnet-hard-sonnet-on/look/3/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 6,+ "costUsd": 0.35745009999999994,+ "inputTokens": 447141,+ "outputTokens": 4561,+ "wallSeconds": 56,+ "exitCode": 0,+ "stopped": "success"+ }+ ]+}diff --git a/bench/agent-modeling/results/2026-11-03-05-52-sonnet-hard-sonnet-off.json b/bench/agent-modeling/results/2026-11-03-05-52-sonnet-hard-sonnet-off.jsonnew file mode 100644index 00000000..111f7ce4--- /dev/null+++ b/bench/agent-modeling/results/2026-11-03-05-52-sonnet-hard-sonnet-off.json@@ -0,0 +1,190 @@+{+ "run": "2026-11-03-05-52-sonnet-hard-sonnet-off",+ "model": "claude-sonnet-5-5",+ "date": "2026-11-03T05:54:41.647Z",+ "skill": false,+ "branches": "off",+ "relations": "on",+ "runsPerTask": 3,+ "passed": 3,+ "tasks": 3,+ "toolCalls": 10,+ "costUsd": 1.04,+ "byModel": {+ "claude-sonnet-5-5": {+ "runs": 3,+ "passed": 3,+ "successRate": 1,+ "toolCalls": 10,+ "toolErrors": 1,+ "inputTokens": 1179322,+ "outputTokens": 12210,+ "costUsd": 1.04,+ "tasks": {+ "light-pick-hard": {+ "runs": 3,+ "passed": 3,+ "successRate": 1,+ "toolCalls": 10,+ "toolErrors": 1,+ "inputTokens": 1179322,+ "outputTokens": 12210,+ "costUsd": 1.04+ }+ }+ }+ },+ "records": [+ {+ "task": "light-pick-hard",+ "attempt": 1,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 3,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-52-sonnet-hard-sonnet-off/look/1/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-52-sonnet-hard-sonnet-off/look/1/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-52-sonnet-hard-sonnet-off/look/1/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 5,+ "costUsd": 0.3389434,+ "inputTokens": 364784,+ "outputTokens": 3748,+ "wallSeconds": 39,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "light-pick-hard",+ "attempt": 2,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 3,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-52-sonnet-hard-sonnet-off/look/2/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-52-sonnet-hard-sonnet-off/look/2/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-52-sonnet-hard-sonnet-off/look/2/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 5,+ "costUsd": 0.3421309,+ "inputTokens": 365842,+ "outputTokens": 3938,+ "wallSeconds": 37,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "light-pick-hard",+ "attempt": 3,+ "pass": true,+ "passed": 10,+ "total": 10,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 4,+ "toolErrors": 1,+ "toolsByName": {+ "ToolSearch": 1,+ "Bash": 1,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-05-52-sonnet-hard-sonnet-off/look/3/light-pick-hard-front.png",+ "artifacts/agent-modeling/2026-11-03-05-52-sonnet-hard-sonnet-off/look/3/light-pick-hard-side.png",+ "artifacts/agent-modeling/2026-11-03-05-52-sonnet-hard-sonnet-off/look/3/light-pick-hard-three-quarter.png"+ ]+ },+ "turns": 6,+ "costUsd": 0.3583705,+ "inputTokens": 448696,+ "outputTokens": 4524,+ "wallSeconds": 45,+ "exitCode": 0,+ "stopped": "success"+ }+ ]+}diff --git a/bench/agent-modeling/tasks.mjs b/bench/agent-modeling/tasks.mjsindex 99054010..cece71ca 100644--- a/bench/agent-modeling/tasks.mjs+++ b/bench/agent-modeling/tasks.mjs@@ -7,6 +7,7 @@ import { DESK_START, DESK_START_PLAIN, DESK_CHECKS } from './desk.mjs';import { CABINET_START, CABINET_START_PLAIN, CABINET_CHECKS } from './cabinet.mjs';import { LAMP_START, LAMP_PROMPT, LAMP_CHECKS } from './lamp.mjs';import { GALLERY_START, GALLERY_PROMPT, GALLERY_CHECKS } from './gallery.mjs';+import { COLONNADE_START, COLONNADE_PROMPT, COLONNADE_CHECKS } from './colonnade.mjs';import { TERRAIN_TASKS } from './terrain.mjs';export const PROMPT_PREAMBLE = `You are modeling with the 67js MCP tools. Build the object described below as a procedural recipe scene:@@ -565,6 +566,7 @@ exported recipe is what gets graded.`,},{ id: 'best-of-three', startRecipe: LAMP_START, prompt: LAMP_PROMPT, checks: LAMP_CHECKS },{ id: 'render-pick', startRecipe: GALLERY_START, prompt: GALLERY_PROMPT, checks: GALLERY_CHECKS },+ { id: 'light-pick-hard', startRecipe: COLONNADE_START, prompt: COLONNADE_PROMPT, checks: COLONNADE_CHECKS },...KETTLE_TASKS,...DESK_TASKS,...CABINET_TASKS,diff --git a/changes/issue-67.md b/changes/issue-67.mdnew file mode 100644index 00000000..00814dc7--- /dev/null+++ b/changes/issue-67.md@@ -0,0 +1 @@+- Add the suite task `light-pick-hard` for branches (part of issue #67): a gallery with six thin posts and five places for one spotlight, where the place that leaves the least of the painting in shadow (10 per cent against 24 to 37) can only be told by rendering each place or tracing 6 posts by hand. `litPainting` gained `blockers` and `least`, and samples a 41 by 21 grid of the painting. The reference and the wrong places and shortcuts are graded in `tests/agent-modeling-suite.test.js`. Calls with branches on and off are in `bench/agent-modeling/README.md`.diff --git a/tests/agent-modeling-suite.test.js b/tests/agent-modeling-suite.test.jsindex 2ffe9917..442e128a 100644--- a/tests/agent-modeling-suite.test.js+++ b/tests/agent-modeling-suite.test.js@@ -4,6 +4,7 @@ import { gradeRecipe } from '../bench/agent-modeling/check.mjs';import { TASKS, taskById, taskPrompt } from '../bench/agent-modeling/tasks.mjs';import { REFERENCES } from '../bench/agent-modeling/references.mjs';import { GALLERY_LIGHT_POSITIONS } from '../bench/agent-modeling/gallery.mjs';+import { COLONNADE_BEST_PLACE, COLONNADE_LIGHT_POSITIONS } from '../bench/agent-modeling/colonnade.mjs';const failedKinds = grade => grade.results.filter(result => !result.pass).map(result => result.label ?? result.kind);@@ -220,3 +221,37 @@ test('the render pick passes the unshadowed light and fails the two shadowed plamovedPillar.nodes.find(node => node.id === 'pillar').translation = [3, 1.3, -0.5];assert.ok(failedKinds(await gradeRecipe(task, movedPillar)).includes('centreAt'));});++test('the hard light pick passes the least shadowed place and fails every other place and shortcut', async () => {+ const task = taskById('light-pick-hard');+ assert.deepEqual(failedKinds(await gradeRecipe(task, structuredClone(REFERENCES['light-pick-hard']))), []);++ for (const [index, place] of COLONNADE_LIGHT_POSITIONS.entries()) {+ if (index === COLONNADE_BEST_PLACE) continue;+ const wrongPlace = structuredClone(REFERENCES['light-pick-hard']);+ const [light] = wrongPlace.environment.spotLights;+ light.position = place;+ const toCentre = [0 - place[0], 1.5 - place[1], -1.98 - place[2]];+ const length = Math.hypot(...toCentre);+ light.direction = toCentre.map(value => value / length);+ const grade = await gradeRecipe(task, wrongPlace);+ assert.deepEqual(failedKinds(grade), ['litPainting'], `place ${index + 1}`);+ assert.match(grade.results.at(-1).detail, /shades \d+ of 861 points/);+ }++ const shadowsOff = structuredClone(REFERENCES['light-pick-hard']);+ shadowsOff.environment.spotLights[0].castShadow = false;+ assert.deepEqual(failedKinds(await gradeRecipe(task, shadowsOff)), ['litPainting']);++ const offeredPlaceMissed = structuredClone(REFERENCES['light-pick-hard']);+ offeredPlaceMissed.environment.spotLights[0].position = [1.7, 2.6, 1.5];+ assert.deepEqual(failedKinds(await gradeRecipe(task, offeredPlaceMissed)), ['litPainting']);++ const twoLights = structuredClone(REFERENCES['light-pick-hard']);+ twoLights.environment.spotLights.push(structuredClone(twoLights.environment.spotLights[0]));+ assert.deepEqual(failedKinds(await gradeRecipe(task, twoLights)), ['litPainting']);++ const movedPost = structuredClone(REFERENCES['light-pick-hard']);+ movedPost.nodes.find(node => node.id === 'post-3').translation[0] = 3;+ assert.ok(failedKinds(await gradeRecipe(task, movedPost)).includes('centreAt'));+});