Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Agent bench: 28 graded tasks, k runs per model, pass@1 and pass^k summary (issue #647)

Nguyen Xuan An2a55afe43e5e
diff --git a/bench/agent-modeling/README.md b/bench/agent-modeling/README.md
index 4d240a05..ef6b18fd 100644
--- a/bench/agent-modeling/README.md
+++ b/bench/agent-modeling/README.md
@@ -51,6 +51,11 @@ own mesh report:
| workshop | Long: a whole scene, a floor slab with a table, a chair and a mug, 12 parts that rest on each other |
| robot-shot | Long: the robot with both hands placed by IK, then a side-on camera, checked from the exported recipe |
+| camera-closeup, camera-orbit | Frame the statue in a close-up; a 6 second camera orbit of at least 180 degrees (`scenes.mjs`) |
+| light-key-fill, light-spot-shadow | A warm sun plus a cool point fill; one shadow-casting spotlight aimed at the statue |
+| look-metal, look-fog | A polished gold material; a thin cool fog with a readable exposure |
+| sunset-timelapse, film-reveal | Sun direction and colour tracks over 4 s; a 6 second dolly-in reveal with an exposure fade |
+
`references.mjs` holds one hand-built solution per task. `tests/agent-modeling-suite.test.js` checks that every
reference passes and that broken variants fail. That keeps a wrong checker from costing an agent run.
@@ -72,6 +77,38 @@ the session, goes to `results/<run>.json`. Grade one recipe by hand with
Compare runs only when they use the same model and suite version. A single run per task is noisy, so read differences
of one or two tasks as noise.
+### The agent bench
+
+`bench.mjs` is the whole bench in one command. It runs the 28 tasks of `suite.mjs` (8 modeling, 3 revision, 3 scene,
+3 look, 4 light, 3 camera, 2 animation, 2 film), k times each, on each model, through headless `claude -p` with the 67js
+MCP config. It calls `run.mjs` once per model, so every run is a fresh session in its own empty temp folder with only the
+67js tools, and the grader reads the exported recipe, never the transcript. Graders are the checks above plus the state
+checks in `scene-checks.mjs` (camera aim and orbit, lights, materials, fog, sun and camera tracks) and the rendered-look
+check, and every task has a reference solution that the tests grade. Runs use the Claude Code subscription (there is no API
+key), so the summary reports tokens, calls and seconds and never dollars.
+
+The baseline, on the current main. It is long (28 tasks, 3 runs, 3 models run one after another), so start it in the
+background:
+
+```sh
+node bench/agent-modeling/bench.mjs --claude <path to claude> --models haiku,sonnet,opus --runs 3 --label baseline --backend cpu
+```
+
+It prints a table with pass@1 (average success over all runs), pass^k (share of tasks that passed on all k runs),
+`any of k`, calls, input and output tokens and seconds per run, then a line per model by kind. The file
+`results/bench-<label>.json` holds that summary and every run's record. `--smoke` runs the two smoke tasks once to prove the
+pipeline, `--tasks a,b` picks tasks, and `--resume` (same label, models and `--runs`) continues a stopped bench.
+
+To compare a branch with the baseline, run the same command with another label on the branch, then:
+
+```sh
+node bench/agent-modeling/compare-bench.mjs bench/agent-modeling/results/bench-baseline.json bench/agent-modeling/results/bench-<branch label>.json
+```
+
+It prints the totals side by side per model and the tasks whose pass count changed, and exits with 1 when a model's pass@1
+drops by more than `--tolerance` (0.05). Use the same models, `--runs` and backend on both sides. A task that moves by one
+run out of three is noise; read the totals and the tasks that moved by two runs or more.
+
### Several runs and several models
`--runs 3` repeats every task three times, each in a fresh session. Raw files are named `<task>-<attempt>`, each record
diff --git a/bench/agent-modeling/bench-summary.mjs b/bench/agent-modeling/bench-summary.mjs
new file mode 100644
index 00000000..183c2322
--- /dev/null
+++ b/bench/agent-modeling/bench-summary.mjs
@@ -0,0 +1,62 @@
+// The bench's summary numbers. Runs go through Claude Code on a subscription, so cost is never reported in dollars:
+// the figures are success, tool calls, tokens and wall time.
+// pass@1 average success rate over every run (the chance one try works)
+// pass^k share of tasks that passed on all k runs (the chance an agent can be relied on)
+// pass@k share of tasks that passed on at least one of the k runs
+import { kindOf } from './suite.mjs';
+
+const mean = (values, digits = 1) => (values.length ? Number((values.reduce((total, value) => total + value, 0) / values.length).toFixed(digits)) : null);
+
+function groupBy(records, keyOf) {
+ const groups = new Map();
+ for (const record of records) {
+ const key = keyOf(record);
+ groups.set(key, [...(groups.get(key) ?? []), record]);
+ }
+ return groups;
+}
+
+/** Numbers for a group of records that belong to tasks run `runs` times each. */
+export function benchTotals(records, runs) {
+ const perTask = [...groupBy(records, record => record.task).values()];
+ const everyRun = perTask.filter(own => own.length >= runs && own.every(record => record.pass));
+ const anyRun = perTask.filter(own => own.some(record => record.pass));
+ const share = count => (perTask.length ? Number((count / perTask.length).toFixed(3)) : null);
+ return {
+ tasks: perTask.length,
+ runs: records.length,
+ passAt1: records.length ? Number((records.filter(record => record.pass).length / records.length).toFixed(3)) : null,
+ passPowK: share(everyRun.length),
+ passAtK: share(anyRun.length),
+ callsPerRun: mean(records.map(record => record.toolCalls)),
+ errorsPerRun: mean(records.map(record => record.toolErrors)),
+ inputTokensPerRun: mean(records.map(record => record.inputTokens), 0),
+ outputTokensPerRun: mean(records.map(record => record.outputTokens), 0),
+ secondsPerRun: mean(records.map(record => record.wallSeconds)),
+ };
+}
+
+/** Totals for the model, by kind of task and by task. `byModel` maps a requested model alias to its records. */
+export function benchSummary(byModel, runs) {
+ return Object.fromEntries(Object.entries(byModel).map(([model, records]) => [model, {
+ ...benchTotals(records, runs),
+ kinds: Object.fromEntries([...groupBy(records, record => kindOf[record.task] ?? 'other')].map(([kind, own]) => [kind, benchTotals(own, runs)])),
+ taskResults: Object.fromEntries([...groupBy(records, record => record.task)].map(([task, own]) => [task, { passed: own.filter(record => record.pass).length, runs: own.length }])),
+ }]));
+}
+
+const percent = value => (value === null ? '?' : `${Math.round(value * 100)}%`);
+
+export function formatSummaryTable(summary, runs) {
+ const header = ['model', 'tasks', 'runs', 'pass@1', `pass^${runs}`, `any of ${runs}`, 'calls/run', 'tokens in/run', 'tokens out/run', 'seconds/run'];
+ const rows = Object.entries(summary).map(([model, totals]) => [
+ model, totals.tasks, totals.runs, percent(totals.passAt1), percent(totals.passPowK), percent(totals.passAtK),
+ totals.callsPerRun, totals.inputTokensPerRun, totals.outputTokensPerRun, totals.secondsPerRun,
+ ].map(String));
+ const widths = header.map((title, column) => Math.max(title.length, ...rows.map(row => row[column].length)));
+ const line = cells => cells.map((cell, column) => cell.padEnd(widths[column])).join(' ');
+ const kindLines = Object.entries(summary).flatMap(([model, totals]) => [
+ `${model} by kind: ${Object.entries(totals.kinds).map(([kind, own]) => `${kind} ${percent(own.passAt1)}`).join(', ')}`,
+ ]);
+ return [line(header), ...rows.map(line), '', ...kindLines].join('\n');
+}
diff --git a/bench/agent-modeling/bench.mjs b/bench/agent-modeling/bench.mjs
new file mode 100644
index 00000000..7f0b7605
--- /dev/null
+++ b/bench/agent-modeling/bench.mjs
@@ -0,0 +1,72 @@
+// The agent bench in one command: every task in suite.mjs, k runs each, on each model, through headless
+// `claude -p` with the 67js MCP config (run.mjs does the per-run work: clean temp folder, MCP server only, grader).
+// Runs use the Claude Code subscription, so the summary has tokens, calls and time and never dollars.
+//
+// node bench/agent-modeling/bench.mjs --claude <claude executable> --models haiku,sonnet,opus --runs 3 --label baseline
+// [--tasks crate,look-fog] [--smoke] [--backend gpu|cpu] [--budget 3] [--timeout 1200] [--resume]
+//
+// --smoke runs the two smoke tasks once. --resume continues the same label (same models, same --runs): attempts already
+// recorded are skipped. The summary goes to bench/agent-modeling/results/bench-<label>.json and holds the per-run records
+// of every model, so `compare-bench.mjs` can diff it against another bench file.
+import { spawn } from 'node:child_process';
+import { mkdir, readFile, writeFile } from 'node:fs/promises';
+import { join, resolve } from 'node:path';
+import { parseArgs } from 'node:util';
+import { benchSummary, formatSummaryTable } from './bench-summary.mjs';
+import { BENCH_TASKS, SMOKE_TASKS } from './suite.mjs';
+
+const resultsDirectory = join(resolve(import.meta.dirname), 'results');
+
+const { values: options } = parseArgs({
+ options: {
+ claude: { type: 'string', default: process.env.CLAUDE_EXECUTABLE ?? 'claude' },
+ models: { type: 'string', default: 'haiku,sonnet,opus' },
+ runs: { type: 'string', default: '3' },
+ label: { type: 'string', default: 'bench' },
+ tasks: { type: 'string' },
+ smoke: { type: 'boolean', default: false },
+ backend: { type: 'string', default: 'gpu' },
+ budget: { type: 'string', default: '3' },
+ timeout: { type: 'string', default: '1200' },
+ resume: { type: 'boolean', default: false },
+ },
+});
+
+const runs = options.smoke ? 1 : Number(options.runs);
+if (!Number.isInteger(runs) || runs < 1) throw new RangeError('--runs must be a whole number from 1');
+const taskIds = options.tasks ? options.tasks.split(',') : options.smoke ? SMOKE_TASKS : BENCH_TASKS.map(task => task.id);
+const models = options.models.split(',');
+const benchName = options.smoke ? `smoke-${options.label}` : options.label;
+
+function runModel(model) {
+ const runName = `bench-${benchName}-${model}`;
+ const argumentsList = [
+ join(import.meta.dirname, 'run.mjs'),
+ '--claude', options.claude, '--model', model, '--tasks', taskIds.join(','), '--runs', String(runs),
+ '--backend', options.backend, '--budget', options.budget, '--timeout', options.timeout,
+ '--label', benchName, '--run-name', runName,
+ ...(options.resume ? ['--resume', runName] : []),
+ ];
+ return new Promise((resolvePromise, reject) => {
+ const child = spawn(process.execPath, argumentsList, { stdio: 'inherit', windowsHide: true });
+ child.on('error', reject);
+ child.on('close', async code => {
+ if (code !== 0) return reject(new Error(`run.mjs for ${model} exited with ${code}`));
+ const summary = JSON.parse(await readFile(join(resultsDirectory, `${runName}.json`), 'utf8'));
+ // The session's cost estimate is not what a subscription run costs, so the bench file leaves it out.
+ resolvePromise(summary.records.map(({ costUsd, ...record }) => record));
+ });
+ });
+}
+
+await mkdir(resultsDirectory, { recursive: true });
+const recordsByModel = {};
+for (const model of models) {
+ console.log(`\n== ${model}: ${taskIds.length} tasks x ${runs} runs ==`);
+ recordsByModel[model] = await runModel(model);
+}
+
+const summary = benchSummary(recordsByModel, runs);
+const benchFile = join(resultsDirectory, `bench-${benchName}.json`);
+await writeFile(benchFile, `${JSON.stringify({ bench: benchName, date: new Date().toISOString(), runsPerTask: runs, tasks: taskIds, backend: options.backend, summary, recordsByModel }, null, 2)}\n`);
+console.log(`\n${formatSummaryTable(summary, runs)}\n\nBench file: bench/agent-modeling/results/bench-${benchName}.json`);
diff --git a/bench/agent-modeling/check.mjs b/bench/agent-modeling/check.mjs
index b8e0e5cd..e6ce36b9 100644
--- a/bench/agent-modeling/check.mjs
+++ b/bench/agent-modeling/check.mjs
@@ -3,6 +3,7 @@ import { readFile } from 'node:fs/promises';
import { fileURLToPath } from 'node:url';
import { measureAssetRecipe, measureRecipe } from './measure.mjs';
import { taskById } from './tasks.mjs';
+import { checkSceneState, isSceneCheck } from './scene-checks.mjs';
import { REQUIRED_HUMANOID_BONES } from '../../src/retarget.js';
const AXES = { x: 0, y: 1, z: 2 };
@@ -414,6 +415,7 @@ export function checkOne(check, measures) {
return { pass: everyPart && measures.uv.stretch <= check.maxStretch, detail: everyPart ? `stretch ${round(measures.uv.stretch)}` : `${measures.uv.parts}/${measures.uv.of} nodes have UVs` };
}
default:
+ if (isSceneCheck(check.kind)) return checkSceneState(check, measures.recipe);
throw new TypeError(`Unknown check kind ${check.kind}.`);
}
}
@@ -444,6 +446,7 @@ export async function gradeRecipe(task, recipe) {
// Keep the reason: a browser or page error must show in the check's detail, not read as a dim picture.
measures.renderedLook = await measureRenderedLook(recipe).catch(error => ({ drawError: error.message }));
}
+ measures.recipe = recipe;
const results = task.checks.map(check => ({ kind: check.kind, ...(check.label ? { label: check.label } : {}), ...checkOne(check, measures) }));
const passed = results.filter(result => result.pass).length;
return { pass: passed === results.length, passed, total: results.length, results };
diff --git a/bench/agent-modeling/compare-bench.mjs b/bench/agent-modeling/compare-bench.mjs
new file mode 100644
index 00000000..74e4a8e3
--- /dev/null
+++ b/bench/agent-modeling/compare-bench.mjs
@@ -0,0 +1,42 @@
+// Compare a candidate bench file against a baseline bench file, per model that appears in both:
+//
+// node bench/agent-modeling/compare-bench.mjs <baseline bench file> <candidate bench file> [--tolerance 0.05]
+//
+// Prints pass@1, pass^k, calls and tokens per run side by side, then the tasks whose pass count changed. Exits with 1
+// when a model's pass@1 drops by more than the tolerance (default 0.05). With k runs per task, a swing of one run on
+// one task is noise: judge a branch by the totals and by tasks that moved by two runs or more.
+import { readFile } from 'node:fs/promises';
+import { fileURLToPath } from 'node:url';
+import { parseArgs } from 'node:util';
+
+const percent = value => (value === null || value === undefined ? '?' : `${Math.round(value * 100)}%`);
+const signed = value => (value > 0 ? `+${value}` : String(value));
+
+/** Differences per model: totals side by side and the tasks whose passes changed. */
+export function compareBench(baseline, candidate, tolerance = 0.05) {
+ const lines = [];
+ let regressed = false;
+ for (const model of Object.keys(baseline.summary).filter(name => name in candidate.summary)) {
+ const before = baseline.summary[model];
+ const after = candidate.summary[model];
+ const drop = Number(((before.passAt1 ?? 0) - (after.passAt1 ?? 0)).toFixed(3));
+ if (drop > tolerance) regressed = true;
+ lines.push(`${model}: pass@1 ${percent(before.passAt1)} -> ${percent(after.passAt1)}${drop > tolerance ? ' REGRESSION' : ''}, pass^k ${percent(before.passPowK)} -> ${percent(after.passPowK)}, calls/run ${before.callsPerRun} -> ${after.callsPerRun}, tokens in/run ${before.inputTokensPerRun} -> ${after.inputTokensPerRun}, tokens out/run ${before.outputTokensPerRun} -> ${after.outputTokensPerRun}`);
+ const tasks = new Set([...Object.keys(before.taskResults), ...Object.keys(after.taskResults)]);
+ for (const task of tasks) {
+ const passedBefore = before.taskResults[task]?.passed ?? 0;
+ const passedAfter = after.taskResults[task]?.passed ?? 0;
+ if (passedBefore !== passedAfter) lines.push(` ${task}: ${passedBefore}/${before.taskResults[task]?.runs ?? 0} -> ${passedAfter}/${after.taskResults[task]?.runs ?? 0} (${signed(passedAfter - passedBefore)})`);
+ }
+ }
+ return { regressed, text: lines.join('\n') || 'No model appears in both files.' };
+}
+
+if (process.argv[1] === fileURLToPath(import.meta.url)) {
+ const { values, positionals } = parseArgs({ allowPositionals: true, options: { tolerance: { type: 'string', default: '0.05' } } });
+ const [baselinePath, candidatePath] = positionals;
+ const [baseline, candidate] = await Promise.all([baselinePath, candidatePath].map(async path => JSON.parse(await readFile(path, 'utf8'))));
+ const result = compareBench(baseline, candidate, Number(values.tolerance));
+ console.log(result.text);
+ process.exitCode = result.regressed ? 1 : 0;
+}
diff --git a/bench/agent-modeling/references.mjs b/bench/agent-modeling/references.mjs
index 71c7ae79..d5db72c2 100644
--- a/bench/agent-modeling/references.mjs
+++ b/bench/agent-modeling/references.mjs
@@ -13,6 +13,7 @@ import { DUSK_SOLUTION } from './dusk.mjs';
import { COLONNADE_SOLUTION } from './colonnade.mjs';
import { DESK_FINAL, DESK_FINAL_PLAIN } from './desk.mjs';
import { CABINET_FINAL, CABINET_FINAL_PLAIN } from './cabinet.mjs';
+import { SCENE_REFERENCES } from './scenes.mjs';
import { EASING_PLAIN_SOLUTION, EASING_SOLUTION } from './easing.mjs';
const material = [{ id: 'paint', color: [0.6, 0.55, 0.5] }];
@@ -169,6 +170,7 @@ function robotArmSolver(root, middle, end, target) {
}
export const REFERENCES = {
+ ...SCENE_REFERENCES,
'best-of-three': LAMP_SOLUTION,
'render-pick': GALLERY_SOLUTION,
'light-pick-hard': COLONNADE_SOLUTION,
diff --git a/bench/agent-modeling/results/bench-smoke-pipeline-haiku.json b/bench/agent-modeling/results/bench-smoke-pipeline-haiku.json
new file mode 100644
index 00000000..e4da9df7
--- /dev/null
+++ b/bench/agent-modeling/results/bench-smoke-pipeline-haiku.json
@@ -0,0 +1,151 @@
+{
+ "run": "bench-smoke-pipeline-haiku",
+ "model": "claude-haiku-5-5",
+ "date": "2026-11-08T20:37:12.587Z",
+ "skill": false,
+ "branches": "on",
+ "relations": "on",
+ "runsPerTask": 1,
+ "passed": 2,
+ "tasks": 2,
+ "toolCalls": 8,
+ "costUsd": 0.14,
+ "byModel": {
+ "claude-haiku-5-5": {
+ "runs": 2,
+ "passed": 2,
+ "successRate": 1,
+ "toolCalls": 8,
+ "toolErrors": 0,
+ "inputTokens": 958962,
+ "outputTokens": 4795,
+ "costUsd": 0.14,
+ "tasks": {
+ "bolt": {
+ "runs": 1,
+ "passed": 1,
+ "successRate": 1,
+ "toolCalls": 5,
+ "toolErrors": 0,
+ "inputTokens": 582713,
+ "outputTokens": 3047,
+ "costUsd": 0.12
+ },
+ "camera-closeup": {
+ "runs": 1,
+ "passed": 1,
+ "successRate": 1,
+ "toolCalls": 3,
+ "toolErrors": 0,
+ "inputTokens": 376249,
+ "outputTokens": 1748,
+ "costUsd": 0.02
+ }
+ }
+ }
+ },
+ "records": [
+ {
+ "task": "bolt",
+ "attempt": 1,
+ "pass": true,
+ "passed": 7,
+ "total": 7,
+ "failed": [],
+ "model": "claude-haiku-5-5",
+ "toolCalls": 5,
+ "toolErrors": 0,
+ "toolsByName": {
+ "ToolSearch": 1,
+ "scene_create": 1,
+ "mesh_inspect": 1,
+ "scene_look": 1,
+ "scene_export_recipe": 1,
+ "scene_dispose": 1
+ },
+ "branchCalls": 0,
+ "relationCalls": 0,
+ "pathQueries": 0,
+ "features": {
+ "geometryTypes": [
+ "cylinder"
+ ],
+ "meshOperations": [],
+ "motionIk": [],
+ "motionTracks": 0
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ },
+ "images": [
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-front.png",
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-side.png",
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-three-quarter.png"
+ ]
+ },
+ "turns": 7,
+ "costUsd": 0.11833810000000002,
+ "inputTokens": 582713,
+ "outputTokens": 3047,
+ "wallSeconds": 34,
+ "exitCode": 0,
+ "stopped": "success"
+ },
+ {
+ "task": "camera-closeup",
+ "attempt": 1,
+ "pass": true,
+ "passed": 3,
+ "total": 3,
+ "failed": [],
+ "model": "claude-haiku-5-5",
+ "toolCalls": 3,
+ "toolErrors": 0,
+ "toolsByName": {
+ "ToolSearch": 1,
+ "scene_create": 1,
+ "scene_apply": 1,
+ "scene_export_recipe": 1
+ },
+ "branchCalls": 0,
+ "relationCalls": 0,
+ "pathQueries": 0,
+ "features": {
+ "geometryTypes": [
+ "box"
+ ],
+ "meshOperations": [],
+ "motionIk": [],
+ "motionTracks": 0
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ },
+ "images": [
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-front.png",
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-side.png",
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-three-quarter.png"
+ ]
+ },
+ "turns": 5,
+ "costUsd": 0.017836499999999998,
+ "inputTokens": 376249,
+ "outputTokens": 1748,
+ "wallSeconds": 40,
+ "exitCode": 0,
+ "stopped": "success"
+ }
+ ]
+}
diff --git a/bench/agent-modeling/results/bench-smoke-pipeline.json b/bench/agent-modeling/results/bench-smoke-pipeline.json
new file mode 100644
index 00000000..9ad28dea
--- /dev/null
+++ b/bench/agent-modeling/results/bench-smoke-pipeline.json
@@ -0,0 +1,164 @@
+{
+ "bench": "smoke-pipeline",
+ "date": "2026-11-08T20:37:48.098Z",
+ "runsPerTask": 1,
+ "tasks": [
+ "bolt",
+ "camera-closeup"
+ ],
+ "backend": "cpu",
+ "summary": {
+ "haiku": {
+ "tasks": 2,
+ "runs": 2,
+ "passAt1": 1,
+ "passPowK": 1,
+ "passAtK": 1,
+ "callsPerRun": 4,
+ "errorsPerRun": 0,
+ "inputTokensPerRun": 479481,
+ "outputTokensPerRun": 2398,
+ "secondsPerRun": 37,
+ "kinds": {
+ "modeling": {
+ "tasks": 1,
+ "runs": 1,
+ "passAt1": 1,
+ "passPowK": 1,
+ "passAtK": 1,
+ "callsPerRun": 5,
+ "errorsPerRun": 0,
+ "inputTokensPerRun": 582713,
+ "outputTokensPerRun": 3047,
+ "secondsPerRun": 34
+ },
+ "camera": {
+ "tasks": 1,
+ "runs": 1,
+ "passAt1": 1,
+ "passPowK": 1,
+ "passAtK": 1,
+ "callsPerRun": 3,
+ "errorsPerRun": 0,
+ "inputTokensPerRun": 376249,
+ "outputTokensPerRun": 1748,
+ "secondsPerRun": 40
+ }
+ },
+ "taskResults": {
+ "bolt": {
+ "passed": 1,
+ "runs": 1
+ },
+ "camera-closeup": {
+ "passed": 1,
+ "runs": 1
+ }
+ }
+ }
+ },
+ "recordsByModel": {
+ "haiku": [
+ {
+ "task": "bolt",
+ "attempt": 1,
+ "pass": true,
+ "passed": 7,
+ "total": 7,
+ "failed": [],
+ "model": "claude-haiku-5-5",
+ "toolCalls": 5,
+ "toolErrors": 0,
+ "toolsByName": {
+ "ToolSearch": 1,
+ "scene_create": 1,
+ "mesh_inspect": 1,
+ "scene_look": 1,
+ "scene_export_recipe": 1,
+ "scene_dispose": 1
+ },
+ "branchCalls": 0,
+ "relationCalls": 0,
+ "pathQueries": 0,
+ "features": {
+ "geometryTypes": [
+ "cylinder"
+ ],
+ "meshOperations": [],
+ "motionIk": [],
+ "motionTracks": 0
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ },
+ "images": [
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-front.png",
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-side.png",
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/bolt-three-quarter.png"
+ ]
+ },
+ "turns": 7,
+ "inputTokens": 582713,
+ "outputTokens": 3047,
+ "wallSeconds": 34,
+ "exitCode": 0,
+ "stopped": "success"
+ },
+ {
+ "task": "camera-closeup",
+ "attempt": 1,
+ "pass": true,
+ "passed": 3,
+ "total": 3,
+ "failed": [],
+ "model": "claude-haiku-5-5",
+ "toolCalls": 3,
+ "toolErrors": 0,
+ "toolsByName": {
+ "ToolSearch": 1,
+ "scene_create": 1,
+ "scene_apply": 1,
+ "scene_export_recipe": 1
+ },
+ "branchCalls": 0,
+ "relationCalls": 0,
+ "pathQueries": 0,
+ "features": {
+ "geometryTypes": [
+ "box"
+ ],
+ "meshOperations": [],
+ "motionIk": [],
+ "motionTracks": 0
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ },
+ "images": [
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-front.png",
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-side.png",
+ "artifacts/agent-modeling/bench-smoke-pipeline-haiku/look/1/camera-closeup-three-quarter.png"
+ ]
+ },
+ "turns": 5,
+ "inputTokens": 376249,
+ "outputTokens": 1748,
+ "wallSeconds": 40,
+ "exitCode": 0,
+ "stopped": "success"
+ }
+ ]
+ }
+}
diff --git a/bench/agent-modeling/run.mjs b/bench/agent-modeling/run.mjs
index 3b4b9c1d..095b39f3 100644
--- a/bench/agent-modeling/run.mjs
+++ b/bench/agent-modeling/run.mjs
@@ -2,7 +2,7 @@
// server, then grades the recipe the agent exported.
//
// node bench/agent-modeling/run.mjs --claude <claude executable> [--model sonnet] [--tasks crate,bolt]
-// [--runs 3] [--budget 3] [--timeout 1200] [--label baseline] [--skill] [--branches off] [--relations off] [--backend gpu|cpu] [--resume <run name>]
+// [--runs 3] [--run-name <name>] [--budget 3] [--timeout 1200] [--label baseline] [--skill] [--branches off] [--relations off] [--backend gpu|cpu] [--resume <run name>]
//
// --runs repeats every task that many times, each in a fresh session; the summary reports success rate, tool calls,
// tool errors, tokens and cost for the model and for each task.
@@ -44,11 +44,12 @@ const { values: options } = parseArgs({
branches: { type: 'string', default: 'on' },
relations: { type: 'string', default: 'on' },
resume: { type: 'string' },
+ 'run-name': { type: 'string' },
backend: { type: 'string', default: 'gpu' },
},
});
-const runName = options.resume ?? `${new Date().toISOString().slice(0, 16).replace(/[:T]/g, '-')}-${options.model}-${options.label}`;
+const runName = options.resume ?? options['run-name'] ?? `${new Date().toISOString().slice(0, 16).replace(/[:T]/g, '-')}-${options.model}-${options.label}`;
const rawDirectory = join(repository, 'artifacts', 'agent-modeling', runName);
const resultsDirectory = join(repository, 'bench', 'agent-modeling', 'results');
diff --git a/bench/agent-modeling/scene-checks.mjs b/bench/agent-modeling/scene-checks.mjs
new file mode 100644
index 00000000..b2d39745
--- /dev/null
+++ b/bench/agent-modeling/scene-checks.mjs
@@ -0,0 +1,170 @@
+// Checks on the state of an exported scene recipe: camera, lights, materials, sky and timeline tracks. They read the
+// recipe's own fields, so they grade what the agent left behind and never how it got there. Keyframe checks look at the
+// keyed values themselves, not at what an interpolation does between them.
+
+const round = value => Number(value.toFixed(2));
+const distance = (first, second) => Math.hypot(...first.map((value, axis) => value - second[axis]));
+const inRange = (value, [lowest, highest]) => value >= lowest && value <= highest;
+
+function tracksOf(recipe, kind, property) {
+ return (recipe.motion?.tracks ?? []).filter(track => track.kind === kind && track.property === property);
+}
+
+function keyedValues(track) {
+ return track.keyframes.map(keyframe => keyframe.value);
+}
+
+function lastTime(track) {
+ return track.keyframes.reduce((latest, keyframe) => Math.max(latest, keyframe.time), 0);
+}
+
+function cameraAims(check, recipe) {
+ const { camera } = recipe;
+ const node = recipe.nodes.find(candidate => candidate.id === check.node);
+ if (!camera || !node) return { pass: false, detail: 'no camera or no such node in the recipe' };
+ const centre = node.translation ?? [0, 0, 0];
+ const toCentre = centre.map((value, axis) => value - camera.eye[axis]);
+ const toTarget = camera.target.map((value, axis) => value - camera.eye[axis]);
+ const along = toCentre.reduce((total, value, axis) => total + value * toTarget[axis], 0) / Math.hypot(...toTarget);
+ const miss = Math.sqrt(Math.max(0, Math.hypot(...toCentre) ** 2 - along ** 2));
+ const eyeDistance = Math.hypot(...toCentre);
+ const fov = camera.fov ?? 45;
+ const problems = [];
+ if (along <= 0 || miss > check.maxMiss) problems.push(`sight line misses the node by ${round(miss)} m`);
+ if (!inRange(eyeDistance, check.distance)) problems.push(`eye ${round(eyeDistance)} m away`);
+ if (!inRange(fov, check.fov)) problems.push(`fov ${fov}`);
+ return { pass: problems.length === 0, detail: problems.join('; ') || `miss ${round(miss)} m, ${round(eyeDistance)} m away, fov ${fov}` };
+}
+
+function cameraOrbit(check, recipe) {
+ const [eyeTrack] = tracksOf(recipe, 'camera', 'eye');
+ const [targetTrack] = tracksOf(recipe, 'camera', 'target');
+ if (!eyeTrack || !targetTrack) return { pass: false, detail: 'the camera eye and target are not both keyed' };
+ const problems = [];
+ const duration = lastTime(eyeTrack);
+ if (!inRange(duration, check.duration)) problems.push(`eye track lasts ${round(duration)} s`);
+ const eyes = keyedValues(eyeTrack);
+ const radii = eyes.map(eye => Math.hypot(eye[0] - check.around[0], eye[2] - check.around[2]));
+ if (!radii.every(radius => inRange(radius, check.radius))) problems.push(`radius ${round(Math.min(...radii))} to ${round(Math.max(...radii))} m`);
+ if (!eyes.every(eye => inRange(eye[1], check.height))) problems.push('eye leaves the height range');
+ let sweep = 0;
+ for (let index = 1; index < eyes.length; index++) {
+ const before = Math.atan2(eyes[index - 1][2] - check.around[2], eyes[index - 1][0] - check.around[0]);
+ const after = Math.atan2(eyes[index][2] - check.around[2], eyes[index][0] - check.around[0]);
+ let step = after - before;
+ while (step > Math.PI) step -= 2 * Math.PI;
+ while (step < -Math.PI) step += 2 * Math.PI;
+ sweep += step;
+ }
+ const degrees = Math.abs(sweep) * 180 / Math.PI;
+ if (degrees < check.sweepDegrees) problems.push(`sweeps ${Math.round(degrees)} degrees`);
+ if (!keyedValues(targetTrack).every(target => distance(target, check.around) <= 0.4)) problems.push('camera target leaves the statue');
+ return { pass: problems.length === 0, detail: problems.join('; ') || `sweeps ${Math.round(degrees)} degrees at ${round(Math.min(...radii))} to ${round(Math.max(...radii))} m` };
+}
+
+function cameraDolly(check, recipe) {
+ const [eyeTrack] = tracksOf(recipe, 'camera', 'eye');
+ const [targetTrack] = tracksOf(recipe, 'camera', 'target');
+ if (!eyeTrack || !targetTrack) return { pass: false, detail: 'the camera eye and target are not both keyed' };
+ const problems = [];
+ const duration = lastTime(eyeTrack);
+ if (!inRange(duration, check.duration)) problems.push(`eye track lasts ${round(duration)} s`);
+ const eyes = keyedValues(eyeTrack);
+ const start = distance(eyes[0], check.toward);
+ const end = distance(eyes.at(-1), check.toward);
+ if (!inRange(start, check.startDistance)) problems.push(`starts ${round(start)} m away`);
+ if (!inRange(end, check.endDistance)) problems.push(`ends ${round(end)} m away`);
+ if (!keyedValues(targetTrack).every(target => distance(target, check.toward) <= 0.4)) problems.push('camera target leaves the statue');
+ return { pass: problems.length === 0, detail: problems.join('; ') || `dolly from ${round(start)} m to ${round(end)} m in ${round(duration)} s` };
+}
+
+function lights(check, recipe) {
+ const list = recipe.environment?.[check.field] ?? [];
+ const problems = [];
+ if (list.length < check.min || list.length > check.max) problems.push(`${list.length} ${check.field}`);
+ for (const light of list) {
+ if (check.castShadow && light.castShadow !== true) problems.push('a light has shadows off');
+ if (check.cool && !(light.color && light.color[2] > light.color[0])) problems.push('a light is not cool (blue above red)');
+ if (check.minX !== undefined && light.position[0] < check.minX) problems.push(`a light sits at x ${light.position[0]}`);
+ if (check.minHeight !== undefined && light.position[1] < check.minHeight) problems.push(`a light is only ${light.position[1]} m high`);
+ if (check.aimAt) {
+ const toTarget = check.aimAt.position.map((value, axis) => value - light.position[axis]);
+ const length = Math.hypot(...toTarget);
+ const directionLength = Math.hypot(...light.direction);
+ const cosine = toTarget.reduce((total, value, axis) => total + value * light.direction[axis], 0) / (length * directionLength);
+ const miss = length * Math.sqrt(Math.max(0, 1 - cosine ** 2));
+ if (cosine <= 0 || miss > check.aimAt.maxMiss) problems.push(`a light misses its target by ${round(miss)} m`);
+ }
+ }
+ return { pass: problems.length === 0, detail: problems.join('; ') || `${list.length} ${check.field} as asked` };
+}
+
+function sunLight(check, recipe) {
+ const { keyColor = [1, 1, 1], keyIntensity = 0 } = recipe.environment ?? {};
+ const problems = [];
+ if (keyIntensity < check.minIntensity) problems.push(`sun intensity ${keyIntensity}`);
+ if (check.warm && !(keyColor[0] > keyColor[2] * 1.2)) problems.push(`sun colour ${keyColor.map(round)} is not warm`);
+ return { pass: problems.length === 0, detail: problems.join('; ') || `sun ${keyColor.map(round)} at ${keyIntensity}` };
+}
+
+function materialOf(check, recipe) {
+ const node = recipe.nodes.find(candidate => candidate.id === check.node);
+ const material = recipe.materials.find(candidate => candidate.id === node?.material);
+ if (!material) return { pass: false, detail: `node ${check.node} has no material` };
+ const { color = [1, 1, 1], metallic = 0, roughness = 1 } = material;
+ const problems = [];
+ if (check.metallic && !inRange(metallic, check.metallic)) problems.push(`metallic ${metallic}`);
+ if (check.roughness && !inRange(roughness, check.roughness)) problems.push(`roughness ${roughness}`);
+ if (check.warm && !(color[0] > color[2])) problems.push('colour is not warm');
+ return { pass: problems.length === 0, detail: problems.join('; ') || `metallic ${metallic}, roughness ${roughness}` };
+}
+
+function fieldValue(check, source, name) {
+ const value = source?.[check.field];
+ if (value === undefined || value === null) return { pass: false, detail: `${name}.${check.field} is not set` };
+ if (check.range) return { pass: inRange(value, check.range), detail: `${check.field} ${value}` };
+ const channelsOk = value.every(channel => channel >= 0.5 && channel <= 0.9);
+ return { pass: Boolean(channelsOk && value[2] >= value[0]), detail: `${check.field} ${value.map(round)}` };
+}
+
+function track(check, recipe) {
+ const [found] = tracksOf(recipe, check.trackKind, check.property);
+ if (!found) return { pass: false, detail: `no ${check.trackKind} ${check.property} track` };
+ const duration = lastTime(found);
+ const values = keyedValues(found);
+ const span = typeof values[0] === 'number'
+ ? Math.max(...values) - Math.min(...values)
+ : Math.max(...values[0].map((_, axis) => Math.max(...values.map(value => value[axis])) - Math.min(...values.map(value => value[axis]))));
+ const problems = [];
+ if (!inRange(duration, check.duration)) problems.push(`lasts ${round(duration)} s`);
+ if (span < check.minSpan) problems.push(`values move only ${round(span)}`);
+ return { pass: problems.length === 0, detail: problems.join('; ') || `${round(duration)} s, values move ${round(span)}` };
+}
+
+function trackRedder(check, recipe) {
+ const [found] = tracksOf(recipe, check.trackKind, check.property);
+ if (!found) return { pass: false, detail: `no ${check.trackKind} ${check.property} track` };
+ const [first] = keyedValues(found);
+ const last = keyedValues(found).at(-1);
+ const ratio = colour => colour[0] / Math.max(colour[2], 0.001);
+ return { pass: ratio(last) > ratio(first) * 1.3, detail: `red over blue ${round(ratio(first))} to ${round(ratio(last))}` };
+}
+
+function trackLower(check, recipe) {
+ const [found] = tracksOf(recipe, check.trackKind, check.property);
+ if (!found) return { pass: false, detail: `no ${check.trackKind} ${check.property} track` };
+ const slope = vector => Math.abs(vector[check.component]) / Math.hypot(...vector);
+ const [first] = keyedValues(found);
+ const last = keyedValues(found).at(-1);
+ return { pass: slope(last) < slope(first) * 0.6, detail: `steepness ${round(slope(first))} to ${round(slope(last))}` };
+}
+
+const SCENE_CHECKS = { cameraAims, cameraOrbit, cameraDolly, lights, sunLight, materialOf, track, trackRedder, trackLower,
+ environmentValue: (check, recipe) => fieldValue(check, recipe.environment, 'environment'),
+ effectsValue: (check, recipe) => fieldValue(check, recipe.effects, 'effects') };
+
+export const isSceneCheck = kind => kind in SCENE_CHECKS;
+
+export function checkSceneState(check, recipe) {
+ return SCENE_CHECKS[check.kind](check, recipe);
+}
diff --git a/bench/agent-modeling/scenes.mjs b/bench/agent-modeling/scenes.mjs
new file mode 100644
index 00000000..575f1f0f
--- /dev/null
+++ b/bench/agent-modeling/scenes.mjs
@@ -0,0 +1,204 @@
+// Scene, look, light, camera and film tasks for the agent bench. Each starts from a small recipe given in the prompt and
+// asks for a change to the camera, the lights, a material, the sky or a timeline. The grader reads the exported recipe
+// (scene-checks.mjs), so it grades the state the agent left and never the calls it made.
+
+const SCENE_NODES = [
+ { id: 'floor', geometry: 'block', material: 'stone', scale: [8, 0.1, 8], translation: [0, -0.05, 0] },
+ { id: 'plinth', geometry: 'block', material: 'stone', scale: [0.8, 0.6, 0.8], translation: [0, 0.3, 0] },
+ { id: 'statue', geometry: 'block', material: 'bronze', scale: [0.4, 1, 0.3], translation: [0, 1.1, 0] },
+];
+const STATUE_CENTRE = [0, 1.1, 0];
+
+function stageRecipe(extra = {}) {
+ return {
+ format: '67js.scene', schemaVersion: 1,
+ geometries: [{ id: 'block', type: 'box', bevel: 0.01 }],
+ materials: [{ id: 'stone', color: [0.7, 0.68, 0.64] }, { id: 'bronze', color: [0.45, 0.3, 0.2] }],
+ nodes: structuredClone(SCENE_NODES),
+ camera: { target: [0, 1, 0], eye: [6, 4, 8], up: [0, 1, 0], fov: 50, near: 0.1, far: 60 },
+ ...extra,
+ };
+}
+
+const STAGE = 'The stage scene below has a floor, a plinth and a statue (node "statue", centred at [0, 1.1, 0], about 1 m tall).';
+const FINISH = 'When it is finished, call scene_export_recipe once: the exported recipe is what gets graded.';
+const PLAIN_PREAMBLE = 'You are working with the 67js MCP tools. Units are metres and Y is up.';
+
+const CLOSEUP_CHECKS = [
+ { kind: 'lint' },
+ { kind: 'parts', min: 3, max: 3 },
+ { kind: 'cameraAims', node: 'statue', maxMiss: 0.3, distance: [1.5, 3.5], fov: [20, 50] },
+];
+
+const ORBIT_CHECKS = [
+ { kind: 'lint' },
+ { kind: 'cameraOrbit', around: STATUE_CENTRE, duration: [5.5, 6.5], radius: [2.5, 6], height: [0.8, 3], sweepDegrees: 180 },
+];
+
+const KEY_FILL_CHECKS = [
+ { kind: 'lint' },
+ { kind: 'parts', min: 3, max: 3 },
+ { kind: 'sunLight', warm: true, minIntensity: 1, label: 'warm key sun' },
+ { kind: 'lights', field: 'pointLights', min: 1, max: 2, cool: true, minX: 0.5, label: 'cool point fill on the right' },
+];
+
+const SPOT_CHECKS = [
+ { kind: 'lint' },
+ { kind: 'lights', field: 'spotLights', min: 1, max: 1, castShadow: true, aimAt: { position: STATUE_CENTRE, maxMiss: 0.3 }, minHeight: 2.5, label: 'one shadow spot above, aimed at the statue' },
+ { kind: 'centreAt', node: 'statue', position: STATUE_CENTRE },
+];
+
+const METAL_CHECKS = [
+ { kind: 'lint' },
+ { kind: 'materialOf', node: 'statue', metallic: [0.85, 1], roughness: [0.15, 0.5], warm: true, label: 'polished gold on the statue' },
+ { kind: 'materialOf', node: 'plinth', metallic: [0, 0.2], label: 'plinth stays stone' },
+ { kind: 'centreAt', node: 'statue', position: STATUE_CENTRE },
+];
+
+const FOG_CHECKS = [
+ { kind: 'lint' },
+ { kind: 'environmentValue', field: 'fogDensity', range: [0.03, 0.3], label: 'thin fog' },
+ { kind: 'environmentValue', field: 'fogColor', cool: true, label: 'cool grey fog' },
+ { kind: 'effectsValue', field: 'exposure', range: [0.8, 1.6], label: 'exposure stays readable' },
+ { kind: 'parts', min: 3, max: 3 },
+];
+
+const SUNSET_CHECKS = [
+ { kind: 'lint' },
+ { kind: 'track', trackKind: 'sun', property: 'direction', duration: [3.5, 4.5], minSpan: 3.5, label: 'sun direction moves over 4 s' },
+ { kind: 'trackRedder', trackKind: 'sun', property: 'color', label: 'sun colour gets warmer' },
+ { kind: 'trackLower', trackKind: 'sun', property: 'direction', component: 1, label: 'sun sinks' },
+];
+
+const FILM_CHECKS = [
+ { kind: 'lint' },
+ { kind: 'parts', min: 3, max: 3 },
+ { kind: 'cameraDolly', toward: STATUE_CENTRE, duration: [5.5, 6.5], startDistance: [6, 14], endDistance: [1.5, 3.5] },
+ { kind: 'track', trackKind: 'effects', property: 'exposure', duration: [5.5, 6.5], minSpan: 0.2, label: 'exposure fades up' },
+ { kind: 'track', trackKind: 'camera', property: 'target', duration: [5.5, 6.5], minSpan: 0, label: 'camera target keyed' },
+];
+
+export const SCENE_TASKS = [
+ {
+ id: 'camera-closeup',
+ startRecipe: stageRecipe(),
+ ownPreamble: true,
+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Set the camera for a close-up of the statue: the camera looks at the statue (within 0.3 m of its centre),
+the eye is 1.5 to 3.5 m from the statue and the field of view is between 20 and 50 degrees. Do not move or change the objects. ${FINISH}`,
+ checks: CLOSEUP_CHECKS,
+ },
+ {
+ id: 'camera-orbit',
+ startRecipe: stageRecipe(),
+ ownPreamble: true,
+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Add a 6 second camera move: the camera circles the statue by at least half a turn (180 degrees seen from above),
+keeping 2.5 to 6 m from the statue's vertical axis and between 0.8 and 3 m above the ground, and always looking at the statue's centre. Key the camera eye and target. ${FINISH}`,
+ checks: ORBIT_CHECKS,
+ },
+ {
+ id: 'light-key-fill',
+ startRecipe: stageRecipe(),
+ ownPreamble: true,
+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Light it as a portrait: a warm key light (the sun: intensity at least 1, red clearly above blue in its colour) plus one cool point light as a fill (blue above red) at least 0.5 m to the right of the statue (x of 0.5 or more). Do not change the objects. ${FINISH}`,
+ checks: KEY_FILL_CHECKS,
+ },
+ {
+ id: 'light-spot-shadow',
+ startRecipe: stageRecipe(),
+ ownPreamble: true,
+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Add exactly one spotlight with shadows on, at least 2.5 m above the ground, aimed at the statue's centre (within 0.3 m).
+Do not change the objects. ${FINISH}`,
+ checks: SPOT_CHECKS,
+ },
+ {
+ id: 'look-metal',
+ startRecipe: stageRecipe(),
+ ownPreamble: true,
+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Make the statue polished gold: metallic at least 0.85, roughness between 0.15 and 0.5, and a warm colour (red above blue).
+The plinth and floor stay as they are. ${FINISH}`,
+ checks: METAL_CHECKS,
+ },
+ {
+ id: 'look-fog',
+ startRecipe: stageRecipe(),
+ ownPreamble: true,
+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Give it a foggy morning look: fog density between 0.03 and 0.3 with a cool grey fog colour (blue at least as high as red,
+all channels between 0.5 and 0.9), and keep the exposure between 0.8 and 1.6 so it stays readable. Do not change the objects. ${FINISH}`,
+ checks: FOG_CHECKS,
+ },
+ {
+ id: 'sunset-timelapse',
+ startRecipe: stageRecipe({ environment: { lightDirection: [-1, -4, -1], keyColor: [3, 2.8, 2.5], keyIntensity: 3 } }),
+ ownPreamble: true,
+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Add a 4 second sunset: key the sun direction so it sinks (its downward component shrinks toward the horizon) and key the sun
+colour so it turns warmer (red up, blue down) between time 0 and time 4. ${FINISH}`,
+ checks: SUNSET_CHECKS,
+ },
+ {
+ id: 'film-reveal',
+ startRecipe: stageRecipe(),
+ ownPreamble: true,
+ prompt: `${PLAIN_PREAMBLE} ${STAGE} Make a 6 second reveal shot: the camera starts 6 to 14 m from the statue and dollies in to 1.5 to 3.5 m from it, looking at the
+statue's centre the whole way (key the camera eye and target), while the exposure fades up by at least 0.2 over the same 6 seconds (key effects exposure). ${FINISH}`,
+ checks: FILM_CHECKS,
+ },
+];
+
+function trackOf(kind, property, keyframes, extra = {}) {
+ return { kind, property, interpolation: 'LINEAR', keyframes: keyframes.map(([time, value]) => ({ time, value })), ...extra };
+}
+
+function orbitKeyframes(radius, height, steps, seconds) {
+ return Array.from({ length: steps + 1 }, (_, step) => {
+ const angle = Math.PI * 1.2 * (step / steps);
+ return [seconds * step / steps, [Number((radius * Math.cos(angle)).toFixed(3)), height, Number((radius * Math.sin(angle)).toFixed(3))]];
+ });
+}
+
+export const SCENE_REFERENCES = {
+ 'camera-closeup': stageRecipe({ camera: { target: [0, 1.1, 0], eye: [0, 1.4, 2.5], up: [0, 1, 0], fov: 35, near: 0.1, far: 60 } }),
+ 'camera-orbit': stageRecipe({
+ motion: {
+ duration: 6,
+ tracks: [
+ trackOf('camera', 'eye', orbitKeyframes(4, 2, 12, 6)),
+ trackOf('camera', 'target', [[0, STATUE_CENTRE], [6, STATUE_CENTRE]]),
+ ],
+ },
+ }),
+ 'light-key-fill': stageRecipe({
+ environment: {
+ lightDirection: [0.6, -0.5, -0.6], keyColor: [1, 0.75, 0.5], keyIntensity: 2.5,
+ pointLights: [{ position: [2, 1.5, 2], color: [0.5, 0.65, 1], intensity: 6 }],
+ },
+ }),
+ 'light-spot-shadow': stageRecipe({
+ environment: { spotLights: [{ position: [1, 4, 1.5], direction: [-0.2857, -0.8286, -0.4286], angle: 0.5, intensity: 40, castShadow: true }] },
+ }),
+ 'look-metal': (() => {
+ const recipe = stageRecipe();
+ recipe.materials = recipe.materials.map(material => (material.id === 'bronze' ? { id: 'bronze', color: [1, 0.78, 0.34], metallic: 1, roughness: 0.3 } : material));
+ return recipe;
+ })(),
+ 'look-fog': stageRecipe({ environment: { fogDensity: 0.1, fogColor: [0.72, 0.75, 0.8] }, effects: { exposure: 1.1 } }),
+ 'sunset-timelapse': stageRecipe({
+ environment: { lightDirection: [-1, -4, -1], keyColor: [3, 2.8, 2.5], keyIntensity: 3 },
+ motion: {
+ duration: 4,
+ tracks: [
+ trackOf('sun', 'direction', [[0, [-1, -4, -1]], [4, [-4, -0.3, -1]]]),
+ trackOf('sun', 'color', [[0, [3, 2.8, 2.5]], [4, [3.4, 1.4, 0.6]]]),
+ ],
+ },
+ }),
+ 'film-reveal': stageRecipe({
+ motion: {
+ duration: 6,
+ tracks: [
+ trackOf('camera', 'eye', [[0, [0, 3, 10]], [6, [0, 1.5, 2.5]]]),
+ trackOf('camera', 'target', [[0, STATUE_CENTRE], [6, STATUE_CENTRE]]),
+ trackOf('effects', 'exposure', [[0, 0.6], [6, 1.2]]),
+ ],
+ },
+ }),
+};
diff --git a/bench/agent-modeling/suite.mjs b/bench/agent-modeling/suite.mjs
new file mode 100644
index 00000000..08ebca1c
--- /dev/null
+++ b/bench/agent-modeling/suite.mjs
@@ -0,0 +1,38 @@
+// The agent bench: the tasks that make up one bench run, each tagged with the kind of work it tests. Ids come from
+// tasks.mjs (every one has a reference solution and an automatic grader). SMOKE_TASKS is the two-task subset used to
+// prove the pipeline end to end without paying for a full run.
+
+export const BENCH_TASKS = [
+ { id: 'crate', kind: 'modeling' },
+ { id: 'bolt', kind: 'modeling' },
+ { id: 'table', kind: 'modeling' },
+ { id: 'mug', kind: 'modeling' },
+ { id: 'gear', kind: 'modeling' },
+ { id: 'pillar-court', kind: 'modeling' },
+ { id: 'chair', kind: 'modeling' },
+ { id: 'crate-uv', kind: 'modeling' },
+ { id: 'crate-revision', kind: 'revision' },
+ { id: 'bracket-revision', kind: 'revision' },
+ { id: 'kettle-resize', kind: 'revision' },
+ { id: 'workshop', kind: 'scene' },
+ { id: 'desk-resize', kind: 'scene' },
+ { id: 'terrain-hole-walk', kind: 'scene' },
+ { id: 'look-pick', kind: 'look' },
+ { id: 'look-metal', kind: 'look' },
+ { id: 'look-fog', kind: 'look' },
+ { id: 'render-pick', kind: 'light' },
+ { id: 'light-pick-hard', kind: 'light' },
+ { id: 'light-key-fill', kind: 'light' },
+ { id: 'light-spot-shadow', kind: 'light' },
+ { id: 'camera-closeup', kind: 'camera' },
+ { id: 'camera-orbit', kind: 'camera' },
+ { id: 'robot-shot', kind: 'camera' },
+ { id: 'overshoot-lift', kind: 'animation' },
+ { id: 'robot-walk-wave', kind: 'animation' },
+ { id: 'sunset-timelapse', kind: 'film' },
+ { id: 'film-reveal', kind: 'film' },
+];
+
+export const SMOKE_TASKS = ['bolt', 'camera-closeup'];
+
+export const kindOf = Object.fromEntries(BENCH_TASKS.map(task => [task.id, task.kind]));
diff --git a/bench/agent-modeling/tasks.mjs b/bench/agent-modeling/tasks.mjs
index af3036d6..5f121abe 100644
--- a/bench/agent-modeling/tasks.mjs
+++ b/bench/agent-modeling/tasks.mjs
@@ -11,6 +11,7 @@ import { DUSK_START, DUSK_PROMPT, DUSK_CHECKS } from './dusk.mjs';
import { COLONNADE_START, COLONNADE_PROMPT, COLONNADE_CHECKS } from './colonnade.mjs';
import { TERRAIN_TASKS } from './terrain.mjs';
import { EASING_TASKS } from './easing.mjs';
+import { SCENE_TASKS } from './scenes.mjs';
export const PROMPT_PREAMBLE = `You are modeling with the 67js MCP tools. Build the object described below as a procedural recipe scene:
start it with scene_create, refine it with mesh_edit, mesh_stack and scene_apply, and look at it with scene_look when
@@ -588,6 +589,7 @@ scene. When it is finished, call scene_export_recipe once: the exported recipe i
...LONG_TASKS,
...TERRAIN_TASKS,
...EASING_TASKS,
+ ...SCENE_TASKS,
];
/** The task text an agent receives, with the starting recipe appended when the task changes an existing scene. */
diff --git a/changes/647.md b/changes/647.md
new file mode 100644
index 00000000..370c2312
--- /dev/null
+++ b/changes/647.md
@@ -0,0 +1 @@
+- bench: `bench/agent-modeling/bench.mjs` runs a 28-task agent bench (modeling, revision, scene, look, light, camera, animation and film tasks, each with an automatic grader) k times per model through headless `claude -p`, and prints pass@1, pass^k, calls, tokens and time per model; `compare-bench.mjs` diffs a branch against a baseline file (#647)
diff --git a/package.json b/package.json
index a7a6955c..40808847 100644
--- a/package.json
+++ b/package.json
@@ -181,6 +181,7 @@
"smoke:agent-review": "node scripts/smoke-agent-review.mjs",
"smoke:agent-branches": "node scripts/smoke-agent-branches.mjs",
"parity": "node scripts/parity-report.mjs",
+ "bench:agent": "node bench/agent-modeling/bench.mjs",
"reference": "node bench/reference/run.mjs",
"reference:line-art": "node bench/reference/line-art.mjs",
"size": "node scripts/measure-size.mjs",
diff --git a/tests/agent-bench.test.js b/tests/agent-bench.test.js
new file mode 100644
index 00000000..ae75e1e8
--- /dev/null
+++ b/tests/agent-bench.test.js
@@ -0,0 +1,72 @@
+import assert from 'node:assert/strict';
+import test from 'node:test';
+import { benchSummary, benchTotals } from '../bench/agent-modeling/bench-summary.mjs';
+import { compareBench } from '../bench/agent-modeling/compare-bench.mjs';
+import { gradeRecipe } from '../bench/agent-modeling/check.mjs';
+import { REFERENCES } from '../bench/agent-modeling/references.mjs';
+import { BENCH_TASKS, SMOKE_TASKS } from '../bench/agent-modeling/suite.mjs';
+import { taskById } from '../bench/agent-modeling/tasks.mjs';
+
+const failedKinds = grade => grade.results.filter(result => !result.pass).map(result => result.label ?? result.kind);
+
+test('the bench has 20 to 30 tasks, each known to the suite, covering every kind of work', () => {
+ assert.ok(BENCH_TASKS.length >= 20 && BENCH_TASKS.length <= 30, `${BENCH_TASKS.length} tasks`);
+ assert.equal(new Set(BENCH_TASKS.map(task => task.id)).size, BENCH_TASKS.length);
+ for (const { id } of BENCH_TASKS) assert.ok(taskById(id).checks.length > 0, id);
+ for (const id of SMOKE_TASKS) assert.ok(BENCH_TASKS.some(task => task.id === id), id);
+ const kinds = new Set(BENCH_TASKS.map(task => task.kind));
+ for (const kind of ['modeling', 'scene', 'look', 'light', 'camera', 'animation', 'film']) assert.ok(kinds.has(kind), kind);
+});
+
+test('the camera, light, look and film tasks reject plausible wrong answers', async () => {
+ const wrong = (id, change) => {
+ const recipe = structuredClone(REFERENCES[id]);
+ change(recipe);
+ return gradeRecipe(taskById(id), recipe);
+ };
+ assert.deepEqual(failedKinds(await wrong('camera-closeup', recipe => { recipe.camera.eye = [6, 4, 8]; })), ['cameraAims']);
+ assert.deepEqual(failedKinds(await wrong('camera-closeup', recipe => { recipe.camera.target = [3, 1, 0]; })), ['cameraAims']);
+ assert.deepEqual(failedKinds(await wrong('camera-orbit', recipe => { recipe.motion.tracks[0].keyframes.length = 4; })), ['cameraOrbit']);
+ assert.deepEqual(failedKinds(await wrong('camera-orbit', recipe => { delete recipe.motion; })), ['cameraOrbit']);
+ assert.deepEqual(failedKinds(await wrong('light-spot-shadow', recipe => { recipe.environment.spotLights[0].castShadow = false; })), ['one shadow spot above, aimed at the statue']);
+ assert.deepEqual(failedKinds(await wrong('light-spot-shadow', recipe => { recipe.environment.spotLights[0].direction = [1, 0, 0]; })), ['one shadow spot above, aimed at the statue']);
+ assert.deepEqual(failedKinds(await wrong('light-key-fill', recipe => { recipe.environment.pointLights[0].color = [1, 0.5, 0.3]; })), ['cool point fill on the right']);
+ assert.deepEqual(failedKinds(await wrong('look-metal', recipe => { recipe.materials[1].metallic = 0; })), ['polished gold on the statue']);
+ assert.deepEqual(failedKinds(await wrong('look-fog', recipe => { recipe.environment.fogDensity = 0.001; })), ['thin fog']);
+ assert.deepEqual(failedKinds(await wrong('sunset-timelapse', recipe => { recipe.motion.tracks[1].keyframes[1].value = [1, 1, 1]; })), ['sun colour gets warmer']);
+ assert.deepEqual(failedKinds(await wrong('film-reveal', recipe => { recipe.motion.tracks[0].keyframes[1].value = [0, 3, 9]; })), ['cameraDolly']);
+ assert.deepEqual(failedKinds(await wrong('film-reveal', recipe => { recipe.motion.tracks.pop(); })), ['exposure fades up']);
+});
+
+const record = (task, pass, extra = {}) => ({ task, pass, toolCalls: 10, toolErrors: 1, inputTokens: 1000, outputTokens: 100, wallSeconds: 30, ...extra });
+
+test('pass@1 is the share of runs that pass, pass^k needs every run of a task to pass', () => {
+ const records = [
+ record('crate', true), record('crate', true), record('crate', true),
+ record('bolt', true), record('bolt', false), record('bolt', true),
+ record('mug', false), record('mug', false), record('mug', false),
+ ];
+ const totals = benchTotals(records, 3);
+ assert.equal(totals.passAt1, 0.556);
+ assert.equal(totals.passPowK, 0.333);
+ assert.equal(totals.passAtK, 0.667);
+ assert.equal(totals.callsPerRun, 10);
+ assert.equal(totals.secondsPerRun, 30);
+});
+
+test('a task with fewer runs than k does not count toward pass^k', () => {
+ assert.equal(benchTotals([record('crate', true), record('crate', true)], 3).passPowK, 0);
+});
+
+test('the summary groups by model and kind, and the comparison flags a pass@1 drop and the tasks that moved', () => {
+ const baselineRecords = { haiku: [record('crate', true), record('crate', true), record('look-fog', true), record('look-fog', false)] };
+ const candidateRecords = { haiku: [record('crate', true), record('crate', false), record('look-fog', false), record('look-fog', false)] };
+ const baseline = { summary: benchSummary(baselineRecords, 2) };
+ const candidate = { summary: benchSummary(candidateRecords, 2) };
+ assert.equal(baseline.summary.haiku.kinds.look.passAt1, 0.5);
+ const result = compareBench(baseline, candidate);
+ assert.equal(result.regressed, true);
+ assert.match(result.text, /REGRESSION/);
+ assert.match(result.text, /crate: 2\/2 -> 1\/2 \(-1\)/);
+ assert.equal(compareBench(baseline, baseline).regressed, false);
+});