3D engine that AI agents drive
Agent bench: 28 graded tasks, k runs per model, pass@1 and pass^k summary (issue #647)
Nguyen Xuan An2a55afe43e5e
diff --git a/bench/agent-modeling/README.md b/bench/agent-modeling/README.mdindex 4d240a05..ef6b18fd 100644--- a/bench/agent-modeling/README.md+++ b/bench/agent-modeling/README.md@@ -51,6 +51,11 @@ own mesh report:| workshop | Long: a whole scene, a floor slab with a table, a chair and a mug, 12 parts that rest on each other || robot-shot | Long: the robot with both hands placed by IK, then a side-on camera, checked from the exported recipe |+| camera-closeup, camera-orbit | Frame the statue in a close-up; a 6 second camera orbit of at least 180 degrees (`scenes.mjs`) |+| light-key-fill, light-spot-shadow | A warm sun plus a cool point fill; one shadow-casting spotlight aimed at the statue |+| look-metal, look-fog | A polished gold material; a thin cool fog with a readable exposure |+| sunset-timelapse, film-reveal | Sun direction and colour tracks over 4 s; a 6 second dolly-in reveal with an exposure fade |+`references.mjs` holds one hand-built solution per task. `tests/agent-modeling-suite.test.js` checks that everyreference passes and that broken variants fail. That keeps a wrong checker from costing an agent run.@@ -72,6 +77,38 @@ the session, goes to `results/<run>.json`. Grade one recipe by hand withCompare runs only when they use the same model and suite version. A single run per task is noisy, so read differencesof one or two tasks as noise.+### The agent bench++`bench.mjs` is the whole bench in one command. It runs the 28 tasks of `suite.mjs` (8 modeling, 3 revision, 3 scene,+3 look, 4 light, 3 camera, 2 animation, 2 film), k times each, on each model, through headless `claude -p` with the 67js+MCP config. It calls `run.mjs` once per model, so every run is a fresh session in its own empty temp folder with only the+67js tools, and the grader reads the exported recipe, never the transcript. Graders are the checks above plus the state+checks in `scene-checks.mjs` (camera aim and orbit, lights, materials, fog, sun and camera tracks) and the rendered-look+check, and every task has a reference solution that the tests grade. Runs use the Claude Code subscription (there is no API+key), so the summary reports tokens, calls and seconds and never dollars.++The baseline, on the current main. It is long (28 tasks, 3 runs, 3 models run one after another), so start it in the+background:++```sh+node bench/agent-modeling/bench.mjs --claude <path to claude> --models haiku,sonnet,opus --runs 3 --label baseline --backend cpu+```++It prints a table with pass@1 (average success over all runs), pass^k (share of tasks that passed on all k runs),+`any of k`, calls, input and output tokens and seconds per run, then a line per model by kind. The file+`results/bench-<label>.json` holds that summary and every run's record. `--smoke` runs the two smoke tasks once to prove the+pipeline, `--tasks a,b` picks tasks, and `--resume` (same label, models and `--runs`) continues a stopped bench.++To compare a branch with the baseline, run the same command with another label on the branch, then:++```sh+node bench/agent-modeling/compare-bench.mjs bench/agent-modeling/results/bench-baseline.json bench/agent-modeling/results/bench-<branch label>.json+```++It prints the totals side by side per model and the tasks whose pass count changed, and exits with 1 when a model's pass@1+drops by more than `--tolerance` (0.05). Use the same models, `--runs` and backend on both sides. A task that moves by one+run out of three is noise; read the totals and the tasks that moved by two runs or more.+### Several runs and several models`--runs 3` repeats every task three times, each in a fresh session. Raw files are named `<task>-<attempt>`, each recorddiff --git a/bench/agent-modeling/bench-summary.mjs b/bench/agent-modeling/bench-summary.mjsnew file mode 100644index 00000000..183c2322--- /dev/null+++ b/bench/agent-modeling/bench-summary.mjs@@ -0,0 +1,62 @@+// The bench's summary numbers. Runs go through Claude Code on a subscription, so cost is never reported in dollars:+// the figures are success, tool calls, tokens and wall time.+// pass@1 average success rate over every run (the chance one try works)+// pass^k share of tasks that passed on all k runs (the chance an agent can be relied on)+// pass@k share of tasks that passed on at least one of the k runs+import { kindOf } from './suite.mjs';++const mean = (values, digits = 1) => (values.length ? Number((values.reduce((total, value) => total + value, 0) / values.length).toFixed(digits)) : null);++function groupBy(records, keyOf) {+ const groups = new Map();+ for (const record of records) {+ const key = keyOf(record);+ groups.set(key, [...(groups.get(key) ?? []), record]);+ }+ return groups;+}++/** Numbers for a group of records that belong to tasks run `runs` times each. */+export function benchTotals(records, runs) {+ const perTask = [...groupBy(records, record => record.task).values()];+ const everyRun = perTask.filter(own => own.length >= runs && own.every(record => record.pass));+ const anyRun = perTask.filter(own => own.some(record => record.pass));+ const share = count => (perTask.length ? Number((count / perTask.length).toFixed(3)) : null);+ return {+ tasks: perTask.length,+ runs: records.length,+ passAt1: records.length ? Number((records.filter(record => record.pass).length / records.length).toFixed(3)) : null,+ passPowK: share(everyRun.length),+ passAtK: share(anyRun.length),+ callsPerRun: mean(records.map(record => record.toolCalls)),+ errorsPerRun: mean(records.map(record => record.toolErrors)),+ inputTokensPerRun: mean(records.map(record => record.inputTokens), 0),+ outputTokensPerRun: mean(records.map(record => record.outputTokens), 0),+ secondsPerRun: mean(records.map(record => record.wallSeconds)),+ };+}++/** Totals for the model, by kind of task and by task. `byModel` maps a requested model alias to its records. */+export function benchSummary(byModel, runs) {+ return Object.fromEntries(Object.entries(byModel).map(([model, records]) => [model, {+ ...benchTotals(records, runs),+ kinds: Object.fromEntries([...groupBy(records, record => kindOf[record.task] ?? 'other')].map(([kind, own]) => [kind, benchTotals(own, runs)])),+ taskResults: Object.fromEntries([...groupBy(records, record => record.task)].map(([task, own]) => [task, { passed: own.filter(record => record.pass).length, runs: own.length }])),+ }]));+}++const percent = value => (value === null ? '?' : `${Math.round(value * 100)}%`);++export function formatSummaryTable(summary, runs) {+ const header = ['model', 'tasks', 'runs', 'pass@1', `pass^${runs}`, `any of ${runs}`, 'calls/run', 'tokens in/run', 'tokens out/run', 'seconds/run'];+ const rows = Object.entries(summary).map(([model, totals]) => [+ model, totals.tasks, totals.runs, percent(totals.passAt1), percent(totals.passPowK), percent(totals.passAtK),+ totals.callsPerRun, totals.inputTokensPerRun, totals.outputTokensPerRun, totals.secondsPerRun,+ ].map(String));+ const widths = header.map((title, column) => Math.max(title.length, ...rows.map(row => row[column].length)));+ const line = cells => cells.map((cell, column) => cell.padEnd(widths[column])).join(' ');+ const kindLines = Object.entries(summary).flatMap(([model, totals]) => [+ `${model} by kind: ${Object.entries(totals.kinds).map(([kind, own]) => `${kind} ${percent(own.passAt1)}`).join(', ')}`,+ ]);+ return [line(header), ...rows.map(line), '', ...kindLines].join('\n');+}diff --git a/bench/agent-modeling/bench.mjs b/bench/agent-modeling/bench.mjsnew file mode 100644index 00000000..7f0b7605--- /dev/null+++ b/bench/agent-modeling/bench.mjs@@ -0,0 +1,72 @@+// The agent bench in one command: every task in suite.mjs, k runs each, on each model, through headless+// `claude -p` with the 67js MCP config (run.mjs does the per-run work: clean temp folder, MCP server only, grader).+// Runs use the Claude Code subscription, so the summary has tokens, calls and time and never dollars.+//+// node bench/agent-modeling/bench.mjs --claude <claude executable> --models haiku,sonnet,opus --runs 3 --label baseline+// [--tasks crate,look-fog] [--smoke] [--backend gpu|cpu] [--budget 3] [--timeout 1200] [--resume]+//+// --smoke runs the two smoke tasks once. --resume continues the same label (same models, same --runs): attempts already+// recorded are skipped. The summary goes to bench/agent-modeling/results/bench-<label>.json and holds the per-run records+// of every model, so `compare-bench.mjs` can diff it against another bench file.+import { spawn } from 'node:child_process';+import { mkdir, readFile, writeFile } from 'node:fs/promises';+import { join, resolve } from 'node:path';+import { parseArgs } from 'node:util';+import { benchSummary, formatSummaryTable } from './bench-summary.mjs';+import { BENCH_TASKS, SMOKE_TASKS } from './suite.mjs';++const resultsDirectory = join(resolve(import.meta.dirname), 'results');++const { values: options } = parseArgs({+ options: {+ claude: { type: 'string', default: process.env.CLAUDE_EXECUTABLE ?? 'claude' },+ models: { type: 'string', default: 'haiku,sonnet,opus' },+ runs: { type: 'string', default: '3' },+ label: { type: 'string', default: 'bench' },+ tasks: { type: 'string' },+ smoke: { type: 'boolean', default: false },+ backend: { type: 'string', default: 'gpu' },+ budget: { type: 'string', default: '3' },+ timeout: { type: 'string', default: '1200' },+ resume: { type: 'boolean', default: false },+ },+});++const runs = options.smoke ? 1 : Number(options.runs);+if (!Number.isInteger(runs) || runs < 1) throw new RangeError('--runs must be a whole number from 1');+const taskIds = options.tasks ? options.tasks.split(',') : options.smoke ? SMOKE_TASKS : BENCH_TASKS.map(task => task.id);+const models = options.models.split(',');+const benchName = options.smoke ? `smoke-${options.label}` : options.label;++function runModel(model) {+ const runName = `bench-${benchName}-${model}`;+ const argumentsList = [+ join(import.meta.dirname, 'run.mjs'),+ '--claude', options.claude, '--model', model, '--tasks', taskIds.join(','), '--runs', String(runs),+ '--backend', options.backend, '--budget', options.budget, '--timeout', options.timeout,+ '--label', benchName, '--run-name', runName,+ ...(options.resume ? ['--resume', runName] : []),+ ];+ return new Promise((resolvePromise, reject) => {+ const child = spawn(process.execPath, argumentsList, { stdio: 'inherit', windowsHide: true });+ child.on('error', reject);+ child.on('close', async code => {+ if (code !== 0) return reject(new Error(`run.mjs for ${model} exited with ${code}`));+ const summary = JSON.parse(await readFile(join(resultsDirectory, `${runName}.json`), 'utf8'));+ // The session's cost estimate is not what a subscription run costs, so the bench file leaves it out.+ resolvePromise(summary.records.map(({ costUsd, ...record }) => record));+ });+ });+}++await mkdir(resultsDirectory, { recursive: true });+const recordsByModel = {};+for (const model of models) {+ console.log(`\n== ${model}: ${taskIds.length} tasks x ${runs} runs ==`);+ recordsByModel[model] = await runModel(model);+}++const summary = benchSummary(recordsByModel, runs);+const benchFile = join(resultsDirectory, `bench-${benchName}.json`);+await writeFile(benchFile, `${JSON.stringify({ bench: benchName, date: new Date().toISOString(), runsPerTask: runs, tasks: taskIds, backend: options.backend, summary, recordsByModel }, null, 2)}\n`);+console.log(`\n${formatSummaryTable(summary, runs)}\n\nBench file: bench/agent-modeling/results/bench-${benchName}.json`);diff --git a/bench/agent-modeling/check.mjs b/bench/agent-modeling/check.mjsindex b8e0e5cd..e6ce36b9 100644--- a/bench/agent-modeling/check.mjs+++ b/bench/agent-modeling/check.mjs@@ -3,6 +3,7 @@ import { readFile } from 'node:fs/promises';import { fileURLToPath } from 'node:url';import { measureAssetRecipe, measureRecipe } from './measure.mjs';import { taskById } from './tasks.mjs';+import { checkSceneState, isSceneCheck } from './scene-checks.mjs';import { REQUIRED_HUMANOID_BONES } from '../../src/retarget.js';const AXES = { x: 0, y: 1, z: 2 };@@ -414,6 +415,7 @@ export function checkOne(check, measures) {return { pass: everyPart && measures.uv.stretch <= check.maxStretch, detail: everyPart ? `stretch ${round(measures.uv.stretch)}` : `${measures.uv.parts}/${measures.uv.of} nodes have UVs` };}default:+ if (isSceneCheck(check.kind)) return checkSceneState(check, measures.recipe);throw new TypeError(`Unknown check kind ${check.kind}.`);}}@@ -444,6 +446,7 @@ export async function gradeRecipe(task, recipe) {// Keep the reason: a browser or page error must show in the check's detail, not read as a dim picture.measures.renderedLook = await measureRenderedLook(recipe).catch(error => ({ drawError: error.message }));}+ measures.recipe = recipe;const results = task.checks.map(check => ({ kind: check.kind, ...(check.label ? { label: check.label } : {}), ...checkOne(check, measures) }));const passed = results.filter(result => result.pass).length;return { pass: passed === results.length, passed, total: results.length, results };diff --git a/bench/agent-modeling/compare-bench.mjs b/bench/agent-modeling/compare-bench.mjsnew file mode 100644index 00000000..74e4a8e3--- /dev/null+++ b/bench/agent-modeling/compare-bench.mjs@@ -0,0 +1,42 @@+// Compare a candidate bench file against a baseline bench file, per model that appears in both:+//+// node bench/agent-modeling/compare-bench.mjs <baseline bench file> <candidate bench file> [--tolerance 0.05]+//+// Prints pass@1, pass^k, calls and tokens per run side by side, then the tasks whose pass count changed. Exits with 1+// when a model's pass@1 drops by more than the tolerance (default 0.05). With k runs per task, a swing of one run on+// one task is noise: judge a branch by the totals and by tasks that moved by two runs or more.+import { readFile } from 'node:fs/promises';+import { fileURLToPath } from 'node:url';+import { parseArgs } from 'node:util';++const percent = value => (value === null || value === undefined ? '?' : `${Math.round(value * 100)}%`);+const signed = value => (value > 0 ? `+${value}` : String(value));++/** Differences per model: totals side by side and the tasks whose passes changed. */+export function compareBench(baseline, candidate, tolerance = 0.05) {+ const lines = [];+ let regressed = false;+ for (const model of Object.keys(baseline.summary).filter(name => name in candidate.summary)) {+ const before = baseline.summary[model];+ const after = candidate.summary[model];+ const drop = Number(((before.passAt1 ?? 0) - (after.passAt1 ?? 0)).toFixed(3));+ if (drop > tolerance) regressed = true;+ lines.push(`${model}: pass@1 ${percent(before.passAt1)} -> ${percent(after.passAt1)}${drop > tolerance ? ' REGRESSION' : ''}, pass^k ${percent(before.passPowK)} -> ${percent(after.passPowK)}, calls/run ${before.callsPerRun} -> ${after.callsPerRun}, tokens in/run ${before.inputTokensPerRun} -> ${after.inputTokensPerRun}, tokens out/run ${before.outputTokensPerRun} -> ${after.outputTokensPerRun}`);+ const tasks = new Set([...Object.keys(before.taskResults), ...Object.keys(after.taskResults)]);+ for (const task of tasks) {+ const passedBefore = before.taskResults[task]?.passed ?? 0;+ const passedAfter = after.taskResults[task]?.passed ?? 0;+ if (passedBefore !== passedAfter) lines.push(` ${task}: ${passedBefore}/${before.taskResults[task]?.runs ?? 0} -> ${passedAfter}/${after.taskResults[task]?.runs ?? 0} (${signed(passedAfter - passedBefore)})`);+ }+ }+ return { regressed, text: lines.join('\n') || 'No model appears in both files.' };+}++if (process.argv[1] === fileURLToPath(import.meta.url)) {+ const { values, positionals } = parseArgs({ allowPositionals: true, options: { tolerance: { type: 'string', default: '0.05' } } });+ const [baselinePath, candidatePath] = positionals;+ const [baseline, candidate] = await Promise.all([baselinePath, candidatePath].map(async path => JSON.parse(await readFile(path, 'utf8'))));+ const result = compareBench(baseline, candidate, Number(values.tolerance));+ console.log(result.text);+ process.exitCode = result.regressed ? 1 : 0;+}diff --git a/bench/agent-modeling/references.mjs b/bench/agent-modeling/references.mjsindex 71c7ae79..d5db72c2 100644--- a/bench/agent-modeling/references.mjs+++ b/bench/agent-modeling/references.mjs@@ -13,6 +13,7 @@ import { DUSK_SOLUTION } from './dusk.mjs';import { COLONNADE_SOLUTION } from './colonnade.mjs';import { DESK_FINAL, DESK_FINAL_PLAIN } from './desk.mjs';import { CABINET_FINAL, CABINET_FINAL_PLAIN } from './cabinet.mjs';+import { SCENE_REFERENCES } from './scenes.mjs';import { EASING_PLAIN_SOLUTION, EASING_SOLUTION } from './easing.mjs';const material = [{ id: 'paint', color: [0.6, 0.55, 0.5] }];@@ -169,6 +170,7 @@ function robotArmSolver(root, middle, end, target) {}export const REFERENCES = {+ ...SCENE_REFERENCES,'best-of-three': LAMP_SOLUTION,'render-pick': GALLERY_SOLUTION,'light-pick-hard': COLONNADE_SOLUTION,diff --git a/bench/agent-modeling/results/bench-smoke-pipeline-haiku.json b/bench/agent-modeling/results/bench-smoke-pipeline-haiku.jsonnew file mode 100644index 00000000..e4da9df7--- /dev/null+++ b/bench/agent-modeling/results/bench-smoke-pipeline-haiku.json@@ -0,0 +1,151 @@+{+ "run": "bench-smoke-pipeline-haiku",+ "model": "claude-haiku-5-5",+ "date": "2026-11-08T20:37:12.587Z",+ "skill": false,+ "branches": "on",+ "relations": "on",+ "runsPerTask": 1,+ "passed": 2,+ "tasks": 2,+ "toolCalls": 8,+ "costUsd": 0.14,+ "byModel": {+ "claude-haiku-5-5": {+ "runs": 2,+ "passed": 2,+ "successRate": 1,+ "toolCalls": 8,+ "toolErrors": 0,+ "inputTokens": 958962,+ "outputTokens": 4795,+ "costUsd": 0.14,+ "tasks": {+ "bolt": {+ "runs": 1,+ "passed": 1,+ "successRate": 1,+ "toolCalls": 5,+ "toolErrors": 0,+ "inputTokens": 582713,+ "outputTokens": 3047,+ "costUsd": 0.12+ },+ "camera-closeup": {+ "runs": 1,+ "passed": 1,+ "successRate": 1,+ "toolCalls": 3,+ "toolErrors": 0,+ "inputTokens": 376249,+ "outputTokens": 1748,+ "costUsd": 0.02+ }+ }+ }+ },+ "records": [+ {+ "task": "bolt",+ "attempt": 1,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "model": "claude-haiku-5-5",+ "toolCalls": 5,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "mesh_inspect": 1,+ "scene_look": 1,+ "scene_export_recipe": 1,+ "scene_dispose": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "cylinder"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-front.png",+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-side.png",+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-three-quarter.png"+ ]+ },+ "turns": 7,+ "costUsd": 0.11833810000000002,+ "inputTokens": 582713,+ "outputTokens": 3047,+ "wallSeconds": 34,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "camera-closeup",+ "attempt": 1,+ "pass": true,+ "passed": 3,+ "total": 3,+ "failed": [],+ "model": "claude-haiku-5-5",+ "toolCalls": 3,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-front.png",+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-side.png",+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-three-quarter.png"+ ]+ },+ "turns": 5,+ "costUsd": 0.017836499999999998,+ "inputTokens": 376249,+ "outputTokens": 1748,+ "wallSeconds": 40,+ "exitCode": 0,+ "stopped": "success"+ }+ ]+}diff --git a/bench/agent-modeling/results/bench-smoke-pipeline.json b/bench/agent-modeling/results/bench-smoke-pipeline.jsonnew file mode 100644index 00000000..9ad28dea--- /dev/null+++ b/bench/agent-modeling/results/bench-smoke-pipeline.json@@ -0,0 +1,164 @@+{+ "bench": "smoke-pipeline",+ "date": "2026-11-08T20:37:48.098Z",+ "runsPerTask": 1,+ "tasks": [+ "bolt",+ "camera-closeup"+ ],+ "backend": "cpu",+ "summary": {+ "haiku": {+ "tasks": 2,+ "runs": 2,+ "passAt1": 1,+ "passPowK": 1,+ "passAtK": 1,+ "callsPerRun": 4,+ "errorsPerRun": 0,+ "inputTokensPerRun": 479481,+ "outputTokensPerRun": 2398,+ "secondsPerRun": 37,+ "kinds": {+ "modeling": {+ "tasks": 1,+ "runs": 1,+ "passAt1": 1,+ "passPowK": 1,+ "passAtK": 1,+ "callsPerRun": 5,+ "errorsPerRun": 0,+ "inputTokensPerRun": 582713,+ "outputTokensPerRun": 3047,+ "secondsPerRun": 34+ },+ "camera": {+ "tasks": 1,+ "runs": 1,+ "passAt1": 1,+ "passPowK": 1,+ "passAtK": 1,+ "callsPerRun": 3,+ "errorsPerRun": 0,+ "inputTokensPerRun": 376249,+ "outputTokensPerRun": 1748,+ "secondsPerRun": 40+ }+ },+ "taskResults": {+ "bolt": {+ "passed": 1,+ "runs": 1+ },+ "camera-closeup": {+ "passed": 1,+ "runs": 1+ }+ }+ }+ },+ "recordsByModel": {+ "haiku": [+ {+ "task": "bolt",+ "attempt": 1,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "model": "claude-haiku-5-5",+ "toolCalls": 5,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "mesh_inspect": 1,+ "scene_look": 1,+ "scene_export_recipe": 1,+ "scene_dispose": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "cylinder"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-front.png",+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-side.png",+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-three-quarter.png"+ ]+ },+ "turns": 7,+ "inputTokens": 582713,+ "outputTokens": 3047,+ "wallSeconds": 34,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "camera-closeup",+ "attempt": 1,+ "pass": true,+ "passed": 3,+ "total": 3,+ "failed": [],+ "model": "claude-haiku-5-5",+ "toolCalls": 3,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_apply": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-front.png",+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-side.png",+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-three-quarter.png"+ ]+ },+ "turns": 5,+ "inputTokens": 376249,+ "outputTokens": 1748,+ "wallSeconds": 40,+ "exitCode": 0,+ "stopped": "success"+ }+ ]+ }+}diff --git a/bench/agent-modeling/run.mjs b/bench/agent-modeling/run.mjsindex 3b4b9c1d..095b39f3 100644--- a/bench/agent-modeling/run.mjs+++ b/bench/agent-modeling/run.mjs@@ -2,7 +2,7 @@// server, then grades the recipe the agent exported.//// node bench/agent-modeling/run.mjs --claude <claude executable> [--model sonnet] [--tasks crate,bolt]-// [--runs 3] [--budget 3] [--timeout 1200] [--label baseline] [--skill] [--branches off] [--relations off] [--backend gpu|cpu] [--resume <run name>]+// [--runs 3] [--run-name <name>] [--budget 3] [--timeout 1200] [--label baseline] [--skill] [--branches off] [--relations off] [--backend gpu|cpu] [--resume <run name>]//// --runs repeats every task that many times, each in a fresh session; the summary reports success rate, tool calls,// tool errors, tokens and cost for the model and for each task.@@ -44,11 +44,12 @@ const { values: options } = parseArgs({branches: { type: 'string', default: 'on' },relations: { type: 'string', default: 'on' },resume: { type: 'string' },+ 'run-name': { type: 'string' },backend: { type: 'string', default: 'gpu' },},});-const runName = options.resume ?? `${new Date().toISOString().slice(0, 16).replace(/[:T]/g, '-')}-${options.model}-${options.label}`;+const runName = options.resume ?? options['run-name'] ?? `${new Date().toISOString().slice(0, 16).replace(/[:T]/g, '-')}-${options.model}-${options.label}`;const rawDirectory = join(repository, 'artifacts', 'agent-modeling', runName);const resultsDirectory = join(repository, 'bench', 'agent-modeling', 'results');diff --git a/bench/agent-modeling/scene-checks.mjs b/bench/agent-modeling/scene-checks.mjsnew file mode 100644index 00000000..b2d39745--- /dev/null+++ b/bench/agent-modeling/scene-checks.mjs@@ -0,0 +1,170 @@+// Checks on the state of an exported scene recipe: camera, lights, materials, sky and timeline tracks. They read the+// recipe's own fields, so they grade what the agent left behind and never how it got there. Keyframe checks look at the+// keyed values themselves, not at what an interpolation does between them.++const round = value => Number(value.toFixed(2));+const distance = (first, second) => Math.hypot(...first.map((value, axis) => value - second[axis]));+const inRange = (value, [lowest, highest]) => value >= lowest && value <= highest;++function tracksOf(recipe, kind, property) {+ return (recipe.motion?.tracks ?? []).filter(track => track.kind === kind && track.property === property);+}++function keyedValues(track) {+ return track.keyframes.map(keyframe => keyframe.value);+}++function lastTime(track) {+ return track.keyframes.reduce((latest, keyframe) => Math.max(latest, keyframe.time), 0);+}++function cameraAims(check, recipe) {+ const { camera } = recipe;+ const node = recipe.nodes.find(candidate => candidate.id === check.node);+ if (!camera || !node) return { pass: false, detail: 'no camera or no such node in the recipe' };+ const centre = node.translation ?? [0, 0, 0];+ const toCentre = centre.map((value, axis) => value - camera.eye[axis]);+ const toTarget = camera.target.map((value, axis) => value - camera.eye[axis]);+ const along = toCentre.reduce((total, value, axis) => total + value * toTarget[axis], 0) / Math.hypot(...toTarget);+ const miss = Math.sqrt(Math.max(0, Math.hypot(...toCentre) ** 2 - along ** 2));+ const eyeDistance = Math.hypot(...toCentre);+ const fov = camera.fov ?? 45;+ const problems = [];+ if (along <= 0 || miss > check.maxMiss) problems.push(`sight line misses the node by ${round(miss)} m`);+ if (!inRange(eyeDistance, check.distance)) problems.push(`eye ${round(eyeDistance)} m away`);+ if (!inRange(fov, check.fov)) problems.push(`fov ${fov}`);+ return { pass: problems.length === 0, detail: problems.join('; ') || `miss ${round(miss)} m, ${round(eyeDistance)} m away, fov ${fov}` };+}++function cameraOrbit(check, recipe) {+ const [eyeTrack] = tracksOf(recipe, 'camera', 'eye');+ const [targetTrack] = tracksOf(recipe, 'camera', 'target');+ if (!eyeTrack || !targetTrack) return { pass: false, detail: 'the camera eye and target are not both keyed' };+ const problems = [];+ const duration = lastTime(eyeTrack);+ if (!inRange(duration, check.duration)) problems.push(`eye track lasts ${round(duration)} s`);+ const eyes = keyedValues(eyeTrack);+ const radii = eyes.map(eye => Math.hypot(eye[0] - check.around[0], eye[2] - check.around[2]));+ if (!radii.every(radius => inRange(radius, check.radius))) problems.push(`radius ${round(Math.min(...radii))} to ${round(Math.max(...radii))} m`);+ if (!eyes.every(eye => inRange(eye[1], check.height))) problems.push('eye leaves the height range');+ let sweep = 0;+ for (let index = 1; index < eyes.length; index++) {+ const before = Math.atan2(eyes[index - 1][2] - check.around[2], eyes[index - 1][0] - check.around[0]);+ const after = Math.atan2(eyes[index][2] - check.around[2], eyes[index][0] - check.around[0]);+ let step = after - before;+ while (step > Math.PI) step -= 2 * Math.PI;+ while (step < -Math.PI) step += 2 * Math.PI;+ sweep += step;+ }+ const degrees = Math.abs(sweep) * 180 / Math.PI;+ if (degrees < check.sweepDegrees) problems.push(`sweeps ${Math.round(degrees)} degrees`);+ if (!keyedValues(targetTrack).every(target => distance(target, check.around) <= 0.4)) problems.push('camera target leaves the statue');+ return { pass: problems.length === 0, detail: problems.join('; ') || `sweeps ${Math.round(degrees)} degrees at ${round(Math.min(...radii))} to ${round(Math.max(...radii))} m` };+}++function cameraDolly(check, recipe) {+ const [eyeTrack] = tracksOf(recipe, 'camera', 'eye');+ const [targetTrack] = tracksOf(recipe, 'camera', 'target');+ if (!eyeTrack || !targetTrack) return { pass: false, detail: 'the camera eye and target are not both keyed' };+ const problems = [];+ const duration = lastTime(eyeTrack);+ if (!inRange(duration, check.duration)) problems.push(`eye track lasts ${round(duration)} s`);+ const eyes = keyedValues(eyeTrack);+ const start = distance(eyes[0], check.toward);+ const end = distance(eyes.at(-1), check.toward);+ if (!inRange(start, check.startDistance)) problems.push(`starts ${round(start)} m away`);+ if (!inRange(end, check.endDistance)) problems.push(`ends ${round(end)} m away`);+ if (!keyedValues(targetTrack).every(target => distance(target, check.toward) <= 0.4)) problems.push('camera target leaves the statue');+ return { pass: problems.length === 0, detail: problems.join('; ') || `dolly from ${round(start)} m to ${round(end)} m in ${round(duration)} s` };+}++function lights(check, recipe) {+ const list = recipe.environment?.[check.field] ?? [];+ const problems = [];+ if (list.length < check.min || list.length > check.max) problems.push(`${list.length} ${check.field}`);+ for (const light of list) {+ if (check.castShadow && light.castShadow !== true) problems.push('a light has shadows off');+ if (check.cool && !(light.color && light.color[2] > light.color[0])) problems.push('a light is not cool (blue above red)');+ if (check.minX !== undefined && light.position[0] < check.minX) problems.push(`a light sits at x ${light.position[0]}`);+ if (check.minHeight !== undefined && light.position[1] < check.minHeight) problems.push(`a light is only ${light.position[1]} m high`);+ if (check.aimAt) {+ const toTarget = check.aimAt.position.map((value, axis) => value - light.position[axis]);+ const length = Math.hypot(...toTarget);+ const directionLength = Math.hypot(...light.direction);+ const cosine = toTarget.reduce((total, value, axis) => total + value * light.direction[axis], 0) / (length * directionLength);+ const miss = length * Math.sqrt(Math.max(0, 1 - cosine ** 2));+ if (cosine <= 0 || miss > check.aimAt.maxMiss) problems.push(`a light misses its target by ${round(miss)} m`);+ }+ }+ return { pass: problems.length === 0, detail: problems.join('; ') || `${list.length} ${check.field} as asked` };+}++function sunLight(check, recipe) {+ const { keyColor = [1, 1, 1], keyIntensity = 0 } = recipe.environment ?? {};+ const problems = [];+ if (keyIntensity < check.minIntensity) problems.push(`sun intensity ${keyIntensity}`);+ if (check.warm && !(keyColor[0] > keyColor[2] * 1.2)) problems.push(`sun colour ${keyColor.map(round)} is not warm`);+ return { pass: problems.length === 0, detail: problems.join('; ') || `sun ${keyColor.map(round)} at ${keyIntensity}` };+}++function materialOf(check, recipe) {+ const node = recipe.nodes.find(candidate => candidate.id === check.node);+ const material = recipe.materials.find(candidate => candidate.id === node?.material);+ if (!material) return { pass: false, detail: `node ${check.node} has no material` };+ const { color = [1, 1, 1], metallic = 0, roughness = 1 } = material;+ const problems = [];+ if (check.metallic && !inRange(metallic, check.metallic)) problems.push(`metallic ${metallic}`);+ if (check.roughness && !inRange(roughness, check.roughness)) problems.push(`roughness ${roughness}`);+ if (check.warm && !(color[0] > color[2])) problems.push('colour is not warm');+ return { pass: problems.length === 0, detail: problems.join('; ') || `metallic ${metallic}, roughness ${roughness}` };+}++function fieldValue(check, source, name) {+ const value = source?.[check.field];+ if (value === undefined || value === null) return { pass: false, detail: `${name}.${check.field} is not set` };+ if (check.range) return { pass: inRange(value, check.range), detail: `${check.field} ${value}` };+ const channelsOk = value.every(channel => channel >= 0.5 && channel <= 0.9);+ return { pass: Boolean(channelsOk && value[2] >= value[0]), detail: `${check.field} ${value.map(round)}` };+}++function track(check, recipe) {+ const [found] = tracksOf(recipe, check.trackKind, check.property);+ if (!found) return { pass: false, detail: `no ${check.trackKind} ${check.property} track` };+ const duration = lastTime(found);+ const values = keyedValues(found);+ const span = typeof values[0] === 'number'+ ? Math.max(...values) - Math.min(...values)+ : Math.max(...values[0].map((_, axis) => Math.max(...values.map(value => value[axis])) - Math.min(...values.map(value => value[axis]))));+ const problems = [];+ if (!inRange(duration, check.duration)) problems.push(`lasts ${round(duration)} s`);+ if (span < check.minSpan) problems.push(`values move only ${round(span)}`);+ return { pass: problems.length === 0, detail: problems.join('; ') || `${round(duration)} s, values move ${round(span)}` };+}++function trackRedder(check, recipe) {+ const [found] = tracksOf(recipe, check.trackKind, check.property);+ if (!found) return { pass: false, detail: `no ${check.trackKind} ${check.property} track` };+ const [first] = keyedValues(found);+ const last = keyedValues(found).at(-1);+ const ratio = colour => colour[0] / Math.max(colour[2], 0.001);+ return { pass: ratio(last) > ratio(first) * 1.3, detail: `red over blue ${round(ratio(first))} to ${round(ratio(last))}` };+}++function trackLower(check, recipe) {+ const [found] = tracksOf(recipe, check.trackKind, check.property);+ if (!found) return { pass: false, detail: `no ${check.trackKind} ${check.property} track` };+ const slope = vector => Math.abs(vector[check.component]) / Math.hypot(...vector);+ const [first] = keyedValues(found);+ const last = keyedValues(found).at(-1);+ return { pass: slope(last) < slope(first) * 0.6, detail: `steepness ${round(slope(first))} to ${round(slope(last))}` };+}++const SCENE_CHECKS = { cameraAims, cameraOrbit, cameraDolly, lights, sunLight, materialOf, track, trackRedder, trackLower,+ environmentValue: (check, recipe) => fieldValue(check, recipe.environment, 'environment'),+ effectsValue: (check, recipe) => fieldValue(check, recipe.effects, 'effects') };++export const isSceneCheck = kind => kind in SCENE_CHECKS;++export function checkSceneState(check, recipe) {+ return SCENE_CHECKS[check.kind](check, recipe);+}diff --git a/bench/agent-modeling/scenes.mjs b/bench/agent-modeling/scenes.mjsnew file mode 100644index 00000000..575f1f0f--- /dev/null+++ b/bench/agent-modeling/scenes.mjs@@ -0,0 +1,204 @@+// Scene, look, light, camera and film tasks for the agent bench. Each starts from a small recipe given in the prompt and+// asks for a change to the camera, the lights, a material, the sky or a timeline. The grader reads the exported recipe+// (scene-checks.mjs), so it grades the state the agent left and never the calls it made.++const SCENE_NODES = [+ { id: 'floor', geometry: 'block', material: 'stone', scale: [8, 0.1, 8], translation: [0, -0.05, 0] },+ { id: 'plinth', geometry: 'block', material: 'stone', scale: [0.8, 0.6, 0.8], translation: [0, 0.3, 0] },+ { id: 'statue', geometry: 'block', material: 'bronze', scale: [0.4, 1, 0.3], translation: [0, 1.1, 0] },+];+const STATUE_CENTRE = [0, 1.1, 0];++function stageRecipe(extra = {}) {+ return {+ format: '67js.scene', schemaVersion: 1,+ geometries: [{ id: 'block', type: 'box', bevel: 0.01 }],+ materials: [{ id: 'stone', color: [0.7, 0.68, 0.64] }, { id: 'bronze', color: [0.45, 0.3, 0.2] }],+ nodes: structuredClone(SCENE_NODES),+ camera: { target: [0, 1, 0], eye: [6, 4, 8], up: [0, 1, 0], fov: 50, near: 0.1, far: 60 },+ ...extra,+ };+}++const STAGE = 'The stage scene below has a floor, a plinth and a statue (node "statue", centred at [0, 1.1, 0], about 1 m tall).';+const FINISH = 'When it is finished, call scene_export_recipe once: the exported recipe is what gets graded.';+const PLAIN_PREAMBLE = 'You are working with the 67js MCP tools. Units are metres and Y is up.';++const CLOSEUP_CHECKS = [+ { kind: 'lint' },+ { kind: 'parts', min: 3, max: 3 },+ { kind: 'cameraAims', node: 'statue', maxMiss: 0.3, distance: [1.5, 3.5], fov: [20, 50] },+];++const ORBIT_CHECKS = [+ { kind: 'lint' },+ { kind: 'cameraOrbit', around: STATUE_CENTRE, duration: [5.5, 6.5], radius: [2.5, 6], height: [0.8, 3], sweepDegrees: 180 },+];++const KEY_FILL_CHECKS = [+ { kind: 'lint' },+ { kind: 'parts', min: 3, max: 3 },+ { kind: 'sunLight', warm: true, minIntensity: 1, label: 'warm key sun' },+ { kind: 'lights', field: 'pointLights', min: 1, max: 2, cool: true, minX: 0.5, label: 'cool point fill on the right' },+];++const SPOT_CHECKS = [+ { kind: 'lint' },+ { kind: 'lights', field: 'spotLights', min: 1, max: 1, castShadow: true, aimAt: { position: STATUE_CENTRE, maxMiss: 0.3 }, minHeight: 2.5, label: 'one shadow spot above, aimed at the statue' },+ { kind: 'centreAt', node: 'statue', position: STATUE_CENTRE },+];++const METAL_CHECKS = [+ { kind: 'lint' },+ { kind: 'materialOf', node: 'statue', metallic: [0.85, 1], roughness: [0.15, 0.5], warm: true, label: 'polished gold on the statue' },+ { kind: 'materialOf', node: 'plinth', metallic: [0, 0.2], label: 'plinth stays stone' },+ { kind: 'centreAt', node: 'statue', position: STATUE_CENTRE },+];++const FOG_CHECKS = [+ { kind: 'lint' },+ { kind: 'environmentValue', field: 'fogDensity', range: [0.03, 0.3], label: 'thin fog' },+ { kind: 'environmentValue', field: 'fogColor', cool: true, label: 'cool grey fog' },+ { kind: 'effectsValue', field: 'exposure', range: [0.8, 1.6], label: 'exposure stays readable' },+ { kind: 'parts', min: 3, max: 3 },+];++const SUNSET_CHECKS = [+ { kind: 'lint' },+ { kind: 'track', trackKind: 'sun', property: 'direction', duration: [3.5, 4.5], minSpan: 3.5, label: 'sun direction moves over 4 s' },+ { kind: 'trackRedder', trackKind: 'sun', property: 'color', label: 'sun colour gets warmer' },+ { kind: 'trackLower', trackKind: 'sun', property: 'direction', component: 1, label: 'sun sinks' },+];++const FILM_CHECKS = [+ { kind: 'lint' },+ { kind: 'parts', min: 3, max: 3 },+ { kind: 'cameraDolly', toward: STATUE_CENTRE, duration: [5.5, 6.5], startDistance: [6, 14], endDistance: [1.5, 3.5] },+ { kind: 'track', trackKind: 'effects', property: 'exposure', duration: [5.5, 6.5], minSpan: 0.2, label: 'exposure fades up' },+ { kind: 'track', trackKind: 'camera', property: 'target', duration: [5.5, 6.5], minSpan: 0, label: 'camera target keyed' },+];++export const SCENE_TASKS = [+ {+ id: 'camera-closeup',+ startRecipe: stageRecipe(),+ ownPreamble: true,+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Set the camera for a close-up of the statue: the camera looks at the statue (within 0.3 m of its centre),+the eye is 1.5 to 3.5 m from the statue and the field of view is between 20 and 50 degrees. Do not move or change the objects. ${FINISH}`,+ checks: CLOSEUP_CHECKS,+ },+ {+ id: 'camera-orbit',+ startRecipe: stageRecipe(),+ ownPreamble: true,+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Add a 6 second camera move: the camera circles the statue by at least half a turn (180 degrees seen from above),+keeping 2.5 to 6 m from the statue's vertical axis and between 0.8 and 3 m above the ground, and always looking at the statue's centre. Key the camera eye and target. ${FINISH}`,+ checks: ORBIT_CHECKS,+ },+ {+ id: 'light-key-fill',+ startRecipe: stageRecipe(),+ ownPreamble: true,+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Light it as a portrait: a warm key light (the sun: intensity at least 1, red clearly above blue in its colour) plus one cool point light as a fill (blue above red) at least 0.5 m to the right of the statue (x of 0.5 or more). Do not change the objects. ${FINISH}`,+ checks: KEY_FILL_CHECKS,+ },+ {+ id: 'light-spot-shadow',+ startRecipe: stageRecipe(),+ ownPreamble: true,+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Add exactly one spotlight with shadows on, at least 2.5 m above the ground, aimed at the statue's centre (within 0.3 m).+Do not change the objects. ${FINISH}`,+ checks: SPOT_CHECKS,+ },+ {+ id: 'look-metal',+ startRecipe: stageRecipe(),+ ownPreamble: true,+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Make the statue polished gold: metallic at least 0.85, roughness between 0.15 and 0.5, and a warm colour (red above blue).+The plinth and floor stay as they are. ${FINISH}`,+ checks: METAL_CHECKS,+ },+ {+ id: 'look-fog',+ startRecipe: stageRecipe(),+ ownPreamble: true,+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Give it a foggy morning look: fog density between 0.03 and 0.3 with a cool grey fog colour (blue at least as high as red,+all channels between 0.5 and 0.9), and keep the exposure between 0.8 and 1.6 so it stays readable. Do not change the objects. ${FINISH}`,+ checks: FOG_CHECKS,+ },+ {+ id: 'sunset-timelapse',+ startRecipe: stageRecipe({ environment: { lightDirection: [-1, -4, -1], keyColor: [3, 2.8, 2.5], keyIntensity: 3 } }),+ ownPreamble: true,+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Add a 4 second sunset: key the sun direction so it sinks (its downward component shrinks toward the horizon) and key the sun+colour so it turns warmer (red up, blue down) between time 0 and time 4. ${FINISH}`,+ checks: SUNSET_CHECKS,+ },+ {+ id: 'film-reveal',+ startRecipe: stageRecipe(),+ ownPreamble: true,+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Make a 6 second reveal shot: the camera starts 6 to 14 m from the statue and dollies in to 1.5 to 3.5 m from it, looking at the+statue's centre the whole way (key the camera eye and target), while the exposure fades up by at least 0.2 over the same 6 seconds (key effects exposure). ${FINISH}`,+ checks: FILM_CHECKS,+ },+];++function trackOf(kind, property, keyframes, extra = {}) {+ return { kind, property, interpolation: 'LINEAR', keyframes: keyframes.map(([time, value]) => ({ time, value })), ...extra };+}++function orbitKeyframes(radius, height, steps, seconds) {+ return Array.from({ length: steps + 1 }, (_, step) => {+ const angle = Math.PI * 1.2 * (step / steps);+ return [seconds * step / steps, [Number((radius * Math.cos(angle)).toFixed(3)), height, Number((radius * Math.sin(angle)).toFixed(3))]];+ });+}++export const SCENE_REFERENCES = {+ 'camera-closeup': stageRecipe({ camera: { target: [0, 1.1, 0], eye: [0, 1.4, 2.5], up: [0, 1, 0], fov: 35, near: 0.1, far: 60 } }),+ 'camera-orbit': stageRecipe({+ motion: {+ duration: 6,+ tracks: [+ trackOf('camera', 'eye', orbitKeyframes(4, 2, 12, 6)),+ trackOf('camera', 'target', [[0, STATUE_CENTRE], [6, STATUE_CENTRE]]),+ ],+ },+ }),+ 'light-key-fill': stageRecipe({+ environment: {+ lightDirection: [0.6, -0.5, -0.6], keyColor: [1, 0.75, 0.5], keyIntensity: 2.5,+ pointLights: [{ position: [2, 1.5, 2], color: [0.5, 0.65, 1], intensity: 6 }],+ },+ }),+ 'light-spot-shadow': stageRecipe({+ environment: { spotLights: [{ position: [1, 4, 1.5], direction: [-0.2857, -0.8286, -0.4286], angle: 0.5, intensity: 40, castShadow: true }] },+ }),+ 'look-metal': (() => {+ const recipe = stageRecipe();+ recipe.materials = recipe.materials.map(material => (material.id === 'bronze' ? { id: 'bronze', color: [1, 0.78, 0.34], metallic: 1, roughness: 0.3 } : material));+ return recipe;+ })(),+ 'look-fog': stageRecipe({ environment: { fogDensity: 0.1, fogColor: [0.72, 0.75, 0.8] }, effects: { exposure: 1.1 } }),+ 'sunset-timelapse': stageRecipe({+ environment: { lightDirection: [-1, -4, -1], keyColor: [3, 2.8, 2.5], keyIntensity: 3 },+ motion: {+ duration: 4,+ tracks: [+ trackOf('sun', 'direction', [[0, [-1, -4, -1]], [4, [-4, -0.3, -1]]]),+ trackOf('sun', 'color', [[0, [3, 2.8, 2.5]], [4, [3.4, 1.4, 0.6]]]),+ ],+ },+ }),+ 'film-reveal': stageRecipe({+ motion: {+ duration: 6,+ tracks: [+ trackOf('camera', 'eye', [[0, [0, 3, 10]], [6, [0, 1.5, 2.5]]]),+ trackOf('camera', 'target', [[0, STATUE_CENTRE], [6, STATUE_CENTRE]]),+ trackOf('effects', 'exposure', [[0, 0.6], [6, 1.2]]),+ ],+ },+ }),+};diff --git a/bench/agent-modeling/suite.mjs b/bench/agent-modeling/suite.mjsnew file mode 100644index 00000000..08ebca1c--- /dev/null+++ b/bench/agent-modeling/suite.mjs@@ -0,0 +1,38 @@+// The agent bench: the tasks that make up one bench run, each tagged with the kind of work it tests. Ids come from+// tasks.mjs (every one has a reference solution and an automatic grader). SMOKE_TASKS is the two-task subset used to+// prove the pipeline end to end without paying for a full run.++export const BENCH_TASKS = [+ { id: 'crate', kind: 'modeling' },+ { id: 'bolt', kind: 'modeling' },+ { id: 'table', kind: 'modeling' },+ { id: 'mug', kind: 'modeling' },+ { id: 'gear', kind: 'modeling' },+ { id: 'pillar-court', kind: 'modeling' },+ { id: 'chair', kind: 'modeling' },+ { id: 'crate-uv', kind: 'modeling' },+ { id: 'crate-revision', kind: 'revision' },+ { id: 'bracket-revision', kind: 'revision' },+ { id: 'kettle-resize', kind: 'revision' },+ { id: 'workshop', kind: 'scene' },+ { id: 'desk-resize', kind: 'scene' },+ { id: 'terrain-hole-walk', kind: 'scene' },+ { id: 'look-pick', kind: 'look' },+ { id: 'look-metal', kind: 'look' },+ { id: 'look-fog', kind: 'look' },+ { id: 'render-pick', kind: 'light' },+ { id: 'light-pick-hard', kind: 'light' },+ { id: 'light-key-fill', kind: 'light' },+ { id: 'light-spot-shadow', kind: 'light' },+ { id: 'camera-closeup', kind: 'camera' },+ { id: 'camera-orbit', kind: 'camera' },+ { id: 'robot-shot', kind: 'camera' },+ { id: 'overshoot-lift', kind: 'animation' },+ { id: 'robot-walk-wave', kind: 'animation' },+ { id: 'sunset-timelapse', kind: 'film' },+ { id: 'film-reveal', kind: 'film' },+];++export const SMOKE_TASKS = ['bolt', 'camera-closeup'];++export const kindOf = Object.fromEntries(BENCH_TASKS.map(task => [task.id, task.kind]));diff --git a/bench/agent-modeling/tasks.mjs b/bench/agent-modeling/tasks.mjsindex af3036d6..5f121abe 100644--- a/bench/agent-modeling/tasks.mjs+++ b/bench/agent-modeling/tasks.mjs@@ -11,6 +11,7 @@ import { DUSK_START, DUSK_PROMPT, DUSK_CHECKS } from './dusk.mjs';import { COLONNADE_START, COLONNADE_PROMPT, COLONNADE_CHECKS } from './colonnade.mjs';import { TERRAIN_TASKS } from './terrain.mjs';import { EASING_TASKS } from './easing.mjs';+import { SCENE_TASKS } from './scenes.mjs';export const PROMPT_PREAMBLE = `You are modeling with the 67js MCP tools. Build the object described below as a procedural recipe scene:start it with scene_create, refine it with mesh_edit, mesh_stack and scene_apply, and look at it with scene_look when@@ -588,6 +589,7 @@ scene. When it is finished, call scene_export_recipe once: the exported recipe i...LONG_TASKS,...TERRAIN_TASKS,...EASING_TASKS,+ ...SCENE_TASKS,];/** The task text an agent receives, with the starting recipe appended when the task changes an existing scene. */diff --git a/changes/647.md b/changes/647.mdnew file mode 100644index 00000000..370c2312--- /dev/null+++ b/changes/647.md@@ -0,0 +1 @@+- bench: `bench/agent-modeling/bench.mjs` runs a 28-task agent bench (modeling, revision, scene, look, light, camera, animation and film tasks, each with an automatic grader) k times per model through headless `claude -p`, and prints pass@1, pass^k, calls, tokens and time per model; `compare-bench.mjs` diffs a branch against a baseline file (#647)diff --git a/package.json b/package.jsonindex a7a6955c..40808847 100644--- a/package.json+++ b/package.json@@ -181,6 +181,7 @@"smoke:agent-review": "node scripts/smoke-agent-review.mjs","smoke:agent-branches": "node scripts/smoke-agent-branches.mjs","parity": "node scripts/parity-report.mjs",+ "bench:agent": "node bench/agent-modeling/bench.mjs","reference": "node bench/reference/run.mjs","reference:line-art": "node bench/reference/line-art.mjs","size": "node scripts/measure-size.mjs",diff --git a/tests/agent-bench.test.js b/tests/agent-bench.test.jsnew file mode 100644index 00000000..ae75e1e8--- /dev/null+++ b/tests/agent-bench.test.js@@ -0,0 +1,72 @@+import assert from 'node:assert/strict';+import test from 'node:test';+import { benchSummary, benchTotals } from '../bench/agent-modeling/bench-summary.mjs';+import { compareBench } from '../bench/agent-modeling/compare-bench.mjs';+import { gradeRecipe } from '../bench/agent-modeling/check.mjs';+import { REFERENCES } from '../bench/agent-modeling/references.mjs';+import { BENCH_TASKS, SMOKE_TASKS } from '../bench/agent-modeling/suite.mjs';+import { taskById } from '../bench/agent-modeling/tasks.mjs';++const failedKinds = grade => grade.results.filter(result => !result.pass).map(result => result.label ?? result.kind);++test('the bench has 20 to 30 tasks, each known to the suite, covering every kind of work', () => {+ assert.ok(BENCH_TASKS.length >= 20 && BENCH_TASKS.length <= 30, `${BENCH_TASKS.length} tasks`);+ assert.equal(new Set(BENCH_TASKS.map(task => task.id)).size, BENCH_TASKS.length);+ for (const { id } of BENCH_TASKS) assert.ok(taskById(id).checks.length > 0, id);+ for (const id of SMOKE_TASKS) assert.ok(BENCH_TASKS.some(task => task.id === id), id);+ const kinds = new Set(BENCH_TASKS.map(task => task.kind));+ for (const kind of ['modeling', 'scene', 'look', 'light', 'camera', 'animation', 'film']) assert.ok(kinds.has(kind), kind);+});++test('the camera, light, look and film tasks reject plausible wrong answers', async () => {+ const wrong = (id, change) => {+ const recipe = structuredClone(REFERENCES[id]);+ change(recipe);+ return gradeRecipe(taskById(id), recipe);+ };+ assert.deepEqual(failedKinds(await wrong('camera-closeup', recipe => { recipe.camera.eye = [6, 4, 8]; })), ['cameraAims']);+ assert.deepEqual(failedKinds(await wrong('camera-closeup', recipe => { recipe.camera.target = [3, 1, 0]; })), ['cameraAims']);+ assert.deepEqual(failedKinds(await wrong('camera-orbit', recipe => { recipe.motion.tracks[0].keyframes.length = 4; })), ['cameraOrbit']);+ assert.deepEqual(failedKinds(await wrong('camera-orbit', recipe => { delete recipe.motion; })), ['cameraOrbit']);+ assert.deepEqual(failedKinds(await wrong('light-spot-shadow', recipe => { recipe.environment.spotLights[0].castShadow = false; })), ['one shadow spot above, aimed at the statue']);+ assert.deepEqual(failedKinds(await wrong('light-spot-shadow', recipe => { recipe.environment.spotLights[0].direction = [1, 0, 0]; })), ['one shadow spot above, aimed at the statue']);+ assert.deepEqual(failedKinds(await wrong('light-key-fill', recipe => { recipe.environment.pointLights[0].color = [1, 0.5, 0.3]; })), ['cool point fill on the right']);+ assert.deepEqual(failedKinds(await wrong('look-metal', recipe => { recipe.materials[1].metallic = 0; })), ['polished gold on the statue']);+ assert.deepEqual(failedKinds(await wrong('look-fog', recipe => { recipe.environment.fogDensity = 0.001; })), ['thin fog']);+ assert.deepEqual(failedKinds(await wrong('sunset-timelapse', recipe => { recipe.motion.tracks[1].keyframes[1].value = [1, 1, 1]; })), ['sun colour gets warmer']);+ assert.deepEqual(failedKinds(await wrong('film-reveal', recipe => { recipe.motion.tracks[0].keyframes[1].value = [0, 3, 9]; })), ['cameraDolly']);+ assert.deepEqual(failedKinds(await wrong('film-reveal', recipe => { recipe.motion.tracks.pop(); })), ['exposure fades up']);+});++const record = (task, pass, extra = {}) => ({ task, pass, toolCalls: 10, toolErrors: 1, inputTokens: 1000, outputTokens: 100, wallSeconds: 30, ...extra });++test('pass@1 is the share of runs that pass, pass^k needs every run of a task to pass', () => {+ const records = [+ record('crate', true), record('crate', true), record('crate', true),+ record('bolt', true), record('bolt', false), record('bolt', true),+ record('mug', false), record('mug', false), record('mug', false),+ ];+ const totals = benchTotals(records, 3);+ assert.equal(totals.passAt1, 0.556);+ assert.equal(totals.passPowK, 0.333);+ assert.equal(totals.passAtK, 0.667);+ assert.equal(totals.callsPerRun, 10);+ assert.equal(totals.secondsPerRun, 30);+});++test('a task with fewer runs than k does not count toward pass^k', () => {+ assert.equal(benchTotals([record('crate', true), record('crate', true)], 3).passPowK, 0);+});++test('the summary groups by model and kind, and the comparison flags a pass@1 drop and the tasks that moved', () => {+ const baselineRecords = { haiku: [record('crate', true), record('crate', true), record('look-fog', true), record('look-fog', false)] };+ const candidateRecords = { haiku: [record('crate', true), record('crate', false), record('look-fog', false), record('look-fog', false)] };+ const baseline = { summary: benchSummary(baselineRecords, 2) };+ const candidate = { summary: benchSummary(candidateRecords, 2) };+ assert.equal(baseline.summary.haiku.kinds.look.passAt1, 0.5);+ const result = compareBench(baseline, candidate);+ assert.equal(result.regressed, true);+ assert.match(result.text, /REGRESSION/);+ assert.match(result.text, /crate: 2\/2 -> 1\/2 \(-1\)/);+ assert.equal(compareBench(baseline, baseline).regressed, false);+});