3D engine that AI agents drive
Judge 67js and three.js renders blind and in pairs (#987)
Nguyen Xuan Andc962a669f6f
diff --git a/bench/agent-modeling/README.md b/bench/agent-modeling/README.mdindex 720005a4..c8c9d70a 100644--- a/bench/agent-modeling/README.md+++ b/bench/agent-modeling/README.md@@ -474,6 +474,21 @@ The three.js half of issue #31 and the 67js side of the same table. The Blender- A first attempt of the 67js run failed on every task because the checkout had no `build/mcp` folder (`npm run build:lib` creates it); those results were discarded. The first three.js run stopped after `bolt` when its process disappeared, so `window`, `table`, `fence` and `bracket` were run again as a second run and joined with `compare-threejs.mjs`.- Limits: 1 run per cell, one model, tasks the 67js suite was built around (they favour 67js's checks, not three.js). The `window` three.js failure label came from an earlier wording of the same check.+### Blind pairwise look judge (2026-11-13, issue #987)++The silhouette look score does not say which render a person or a vision model prefers. `pairwise-judge.mjs` shows the render of system A and the render of system B for one task as "first" and "second", in a seeded random order (`default-judge.mjs`'s `firstSideOf`; the repeats of one task swap sides), and asks which is the better finished render. It records the winner per task and attempt, then wins per system and an exact two-sided binomial p against 0.5. A control shows system A's own render on both sides: any preference for "first" or "second" is position bias, and a pairwise result no larger than the control's is not a preference for a system. Calls go through headless `claude -p` on the subscription, one per task, attempt and pairing.++```sh+node bench/agent-modeling/pairwise-judge.mjs --name <result name> --model sonnet --tasks crate,bolt,window,table,fence,bracket --attempts 1 \+ --a "67js MCP=artifacts/agent-modeling/<67js run>/look/{attempt}/{task}-three-quarter.png" \+ --b "three.js by code=artifacts/agent-modeling/<three.js run>/{task}-{attempt}.png"+node bench/agent-modeling/compare-threejs.mjs <67js run json> <three.js run json> <output json> --pairwise bench/agent-modeling/results/<result name>.json+```++Each side is `label=image pattern` with `{task}` and `{attempt}` filled in. `--control-only` skips the pairing and runs only the control on side A; `--seed` changes the order. The judge writes `results/<result name>.json` (summary and one record per call, with the note the judge gave), and `compare-threejs.mjs --pairwise` prints wins per system, the p and the control next to the usual tables. Run records also carry `fixRounds`, next to `toolCalls` and `toolErrors`: the number of write steps after the first render (`scene_apply` or `scene_create_asset` after the first `scene_look` or render for 67js; `Write` or `Edit` after the first `node` run for three.js). A run that renders once has 0. Records from before this change have none and print `-`.++Run on 2026-11-13 (measured): the raw renders of the 2026-11-07 head-to-head run are no longer on disk (only the scene numbers are in the result file), so the pairing was not re-run. The control ran on the 67js three-quarter renders that exist (`crate`, `bolt`, `table`, `chair`, `gear`, three attempts each, Sonnet, 15 calls, $1.76): the judge picked "first" 7 times and "second" 8 times, p = 1.0, so no position bias at this size. `results/2026-11-13-pairwise-control-67js.json`. The next three.js run saves `scene.png` in `artifacts/agent-modeling/<run>/`, which is what the `--b` pattern above reads.+### Blender MCP half (2026-11-07, issue #31)The official Blender MCP server (Blender Lab, v1.0.3, add-on and server) driving Blender 5.1.2, same model, same six tasks, 1 run each, in `results/2026-11-07-blender-mcp.json`. The server runs model-written Python with no guards, so `blender.mjs` never runs it on the benchmark machine: Blender (headless, `--command blender_mcp`) and the HTTP MCP server run inside a throwaway Docker container (`blender-container/`, x86_64 under Rosetta on an Apple silicon laptop, one scratch folder mounted, a fresh container per task). `claude -p` reaches the server through an SSH port forward and gets only the MCP tools: no Bash and no file tools on its own machine. The prompt asks the agent to leave `geometry.json` (world-space triangles, Y up) and `scene.png` in the container's `/work`; the script copies them back and grades them exactly like the three.js run.diff --git a/bench/agent-modeling/compare-threejs.mjs b/bench/agent-modeling/compare-threejs.mjsindex 6a701b31..34115674 100644--- a/bench/agent-modeling/compare-threejs.mjs+++ b/bench/agent-modeling/compare-threejs.mjs@@ -1,9 +1,13 @@// Joins one 67js run (run.mjs) and one three.js run (threejs.mjs) into the comparison file of issue #31 and prints// the markdown tables.-// node bench/agent-modeling/compare-threejs.mjs <67js run json> <three.js run json> <output json>+// node bench/agent-modeling/compare-threejs.mjs <67js run json> <three.js run json> <output json> [--pairwise <pairwise-judge result json>]+// With --pairwise it also prints wins per system, the exact binomial p and the position-bias control of pairwise-judge.mjs.import { readFile, writeFile } from 'node:fs/promises';+import { parseArgs } from 'node:util';+import { pairwiseTable } from './pairwise-judge.mjs';-const [mcpPath, codePath, outputPath] = process.argv.slice(2);+const { values: options, positionals } = parseArgs({ allowPositionals: true, options: { pairwise: { type: 'string' } } });+const [mcpPath, codePath, outputPath] = positionals;const mcpRun = JSON.parse(await readFile(mcpPath, 'utf8'));// The three.js run may be split over several files (comma separated) when a run was stopped and the rest redone.const codeRuns = await Promise.all(codePath.split(',').map(async path => JSON.parse(await readFile(path, 'utf8'))));@@ -20,15 +24,18 @@ const lookOf = record => record.look?.silhouette?.mean ?? null;const tokensOf = record => record.inputTokens + record.outputTokens;const thousands = value => `${Math.round(value / 1000)}k`;+// Records from before fix rounds were counted have none; a mean needs every record to carry one.+const fixRoundsOf = own => (own.every(record => typeof record.fixRounds === 'number') ? Number(mean(own.map(record => record.fixRounds)).toFixed(1)) : null);+const taskIds = [...new Set(records.map(record => record.task))];-const lines = ['| Task | System | Pass | Tool calls | Tool errors | Tokens in / out | Look | Failures |', '| --- | --- | --- | --- | --- | --- | --- | --- |'];+const lines = ['| Task | System | Pass | Tool calls | Tool errors | Fix rounds | Tokens in / out | Look | Failures |', '| --- | --- | --- | --- | --- | --- | --- | --- | --- |'];for (const task of taskIds) {for (const { name } of SYSTEMS) {const record = records.find(candidate => candidate.task === task && candidate.system === name);- lines.push(`| ${task} | ${name} | ${record.pass ? 'yes' : 'no'} (${record.passed}/${record.total}) | ${record.toolCalls} | ${record.toolErrors} | ${thousands(record.inputTokens)} / ${thousands(record.outputTokens)} | ${lookOf(record)?.toFixed(3) ?? '-'} | ${record.failed.join('; ') || '-'} |`);+ lines.push(`| ${task} | ${name} | ${record.pass ? 'yes' : 'no'} (${record.passed}/${record.total}) | ${record.toolCalls} | ${record.toolErrors} | ${record.fixRounds ?? '-'} | ${thousands(record.inputTokens)} / ${thousands(record.outputTokens)} | ${lookOf(record)?.toFixed(3) ?? '-'} | ${record.failed.join('; ') || '-'} |`);}}-const totals = ['| System | Pass | Mean tool calls | Mean tokens | Mean look | Mean wall seconds |', '| --- | --- | --- | --- | --- | --- |'];+const totals = ['| System | Pass | Mean tool calls | Mean fix rounds | Mean tokens | Mean look | Mean wall seconds |', '| --- | --- | --- | --- | --- | --- | --- |'];const summary = {};for (const { name } of SYSTEMS) {const own = records.filter(record => record.system === name);@@ -37,14 +44,16 @@ for (const { name } of SYSTEMS) {runs: own.length,passed: own.filter(record => record.pass).length,meanToolCalls: Number(mean(own.map(record => record.toolCalls)).toFixed(1)),+ meanFixRounds: fixRoundsOf(own),meanTokens: Math.round(mean(own.map(tokensOf))),meanLook: Number(mean(looks).toFixed(3)),meanWallSeconds: Math.round(mean(own.map(record => record.wallSeconds))),};const row = summary[name];- totals.push(`| ${name} | ${row.passed}/${row.runs} | ${row.meanToolCalls} | ${thousands(row.meanTokens)} | ${row.meanLook} | ${row.meanWallSeconds} |`);+ totals.push(`| ${name} | ${row.passed}/${row.runs} | ${row.meanToolCalls} | ${row.meanFixRounds ?? '-'} | ${thousands(row.meanTokens)} | ${row.meanLook} | ${row.meanWallSeconds} |`);}+const pairwise = options.pairwise ? JSON.parse(await readFile(options.pairwise, 'utf8')) : null;const output = {run: '2026-11-07-threejs-vs-67js',issue: 31,@@ -54,7 +63,8 @@ const output = {threeVersion: codeRun.threeVersion,runsPerTaskPerSystem: 1,summary,+ ...(pairwise ? { pairwise } : {}),records,};await writeFile(outputPath, `${JSON.stringify(output, null, 2)}\n`);-console.log(`${totals.join('\n')}\n\n${lines.join('\n')}`);+console.log(`${totals.join('\n')}\n\n${lines.join('\n')}${pairwise ? `\n\n${pairwiseTable(pairwise)}` : ''}`);diff --git a/bench/agent-modeling/default-judge.mjs b/bench/agent-modeling/default-judge.mjsindex df5ddb9a..13e3dbe5 100644--- a/bench/agent-modeling/default-judge.mjs+++ b/bench/agent-modeling/default-judge.mjs@@ -103,10 +103,22 @@ async function renderVariant({ label, noFloor, directory }) {console.log(`${label}: ${COLD_TASKS.length * VIEW_ORDER.length} images`);}-function imageBlock(bytes) {+export function imageBlock(bytes) {return { type: 'image', source: { type: 'base64', media_type: 'image/png', data: bytes.toString('base64') } };}+/** One headless `claude -p` call on the subscription with no tools. `content` is a list of text and image blocks. */+export function callClaudeJudge({ claude, model, content }) {+ const message = JSON.stringify({ type: 'user', message: { role: 'user', content } });+ const result = spawnSync(claude, ['-p', '--model', model, '--input-format', 'stream-json', '--output-format', 'stream-json', '--verbose',+ '--tools', '', '--system-prompt', JUDGE_ROLE, '--strict-mcp-config', '--no-session-persistence', '--max-budget-usd', '0.2'],+ { input: `${message}\n`, encoding: 'utf8', timeout: 300000, maxBuffer: 64 * 1024 * 1024 });+ const final = result.stdout.split('\n').filter(Boolean).map(line => JSON.parse(line)).find(event => event.type === 'result');+ const text = String(final?.result ?? '');+ const match = /WINNER:\s*(first|second)/i.exec(text);+ return { answer: match ? match[1].toLowerCase() : null, costUsd: final?.total_cost_usd ?? 0, note: text.split('\n').find(line => line.trim()) };+}+function askJudge({ claude, model, firstDirectory, secondDirectory, taskId }) {return readImages({ firstDirectory, secondDirectory, taskId }).then(images => {const content = [{ type: 'text', text: JUDGE_QUESTION }];@@ -114,14 +126,7 @@ function askJudge({ claude, model, firstDirectory, secondDirectory, taskId }) {content.push({ type: 'text', text: `View "${view}", first render:` }, imageBlock(images[view].first));content.push({ type: 'text', text: `View "${view}", second render:` }, imageBlock(images[view].second));}- const message = JSON.stringify({ type: 'user', message: { role: 'user', content } });- const result = spawnSync(claude, ['-p', '--model', model, '--input-format', 'stream-json', '--output-format', 'stream-json', '--verbose',- '--tools', '', '--system-prompt', JUDGE_ROLE, '--strict-mcp-config', '--no-session-persistence', '--max-budget-usd', '0.2'],- { input: `${message}\n`, encoding: 'utf8', timeout: 300000, maxBuffer: 64 * 1024 * 1024 });- const final = result.stdout.split('\n').filter(Boolean).map(line => JSON.parse(line)).find(event => event.type === 'result');- const text = String(final?.result ?? '');- const match = /WINNER:\s*(first|second)/i.exec(text);- return { answer: match ? match[1].toLowerCase() : null, costUsd: final?.total_cost_usd ?? 0, note: text.split('\n').find(line => line.trim()) };+ return callClaudeJudge({ claude, model, content });});}diff --git a/bench/agent-modeling/fix-rounds.mjs b/bench/agent-modeling/fix-rounds.mjsnew file mode 100644index 00000000..a7d60cf2--- /dev/null+++ b/bench/agent-modeling/fix-rounds.mjs@@ -0,0 +1,41 @@+// Fix rounds: how many times an agent changed the model after it first looked at a render.+// A run that builds the object and renders once has 0. Each write or edit step after the first render is one round,+// so a run that looks, edits, looks and edits again has 2. It is recorded next to `toolCalls` and `toolErrors`.++const MCP_RENDER_TOOLS = ['scene_look', 'scene_render_submit'];+const MCP_WRITE_TOOLS = ['scene_apply', 'scene_create_asset'];+const CODE_WRITE_TOOLS = ['Write', 'Edit'];+const CODE_RENDER_COMMAND = /build\.mjs|\bnode\b/;++/** The tool name of a 67js MCP call, with or without the `mcp__<server>__` prefix. */+function mcpToolName(step) {+ return step.name.replace(/^mcp__.*?__/, '');+}++export function isMcpRender(step) {+ return MCP_RENDER_TOOLS.includes(mcpToolName(step));+}++export function isMcpWrite(step) {+ return MCP_WRITE_TOOLS.includes(mcpToolName(step));+}++/** The three.js agent renders by running its build script with Bash. */+export function isCodeRender(step) {+ return step.name === 'Bash' && CODE_RENDER_COMMAND.test(String(step.input?.command ?? ''));+}++export function isCodeWrite(step) {+ return CODE_WRITE_TOOLS.includes(step.name);+}++/**+ * @param {{ name: string, input?: object }[]} steps tool calls in the order the agent made them+ * @param {{ isRender: (step) => boolean, isWrite: (step) => boolean }} kinds+ * @returns {number} write steps after the first render; 0 when the agent never rendered+ */+export function countFixRounds(steps, { isRender, isWrite }) {+ const firstRender = steps.findIndex(isRender);+ if (firstRender === -1) return 0;+ return steps.slice(firstRender + 1).filter(isWrite).length;+}diff --git a/bench/agent-modeling/pairwise-judge.mjs b/bench/agent-modeling/pairwise-judge.mjsnew file mode 100644index 00000000..b7241e81--- /dev/null+++ b/bench/agent-modeling/pairwise-judge.mjs@@ -0,0 +1,168 @@+// Blind pairwise look judge for two systems that built the same task (issue #987). A vision model sees the render of+// system A and the render of system B as "first" and "second", in a seeded random order, and says which looks like the+// better finished render of the object. It never learns which system made which. Wins per system and an exact binomial+// p against 0.5 tell whether the judge prefers one. A control pairs one system's render with itself: any preference for+// "first" or "second" is position bias, and a result near the control's size is not a preference for a system.+//+// node bench/agent-modeling/pairwise-judge.mjs --claude <path> --model <name> --name <result name> --tasks crate,bolt+// --a "67js MCP=artifacts/agent-modeling/<run>/look/{attempt}/{task}-three-quarter.png"+// --b "three.js by code=artifacts/agent-modeling/<run>/{task}-{attempt}.png" [--attempts 1] [--seed 53] [--control-only]+// Each side is `label=image pattern`; `{task}` and `{attempt}` are filled in. One judge call per task, attempt and+// pairing (the pair, then the control). The result is written to bench/agent-modeling/results/<name>.json, and+// compare-threejs.mjs --pairwise prints it. Calls go through headless `claude -p` on the subscription, never the API.+import { mkdir, readFile, writeFile } from 'node:fs/promises';+import { dirname, join, resolve } from 'node:path';+import { parseArgs } from 'node:util';+import { binomialP, callClaudeJudge, firstSideOf, imageBlock } from './default-judge.mjs';++const repository = resolve(import.meta.dirname, '../..');+const DEFAULT_SEED = 53;++export const PAIRWISE_QUESTION = `You see two renders of the same 3D object, drawn by two different systems. The first render comes first, the second render second.+Which one looks like the better finished render of that object: a believable shape, proportions, surface and lighting, grounded on a surface, with nothing missing or broken?+Judge only the pictures. Give one short sentence naming what decided it, then as the very last line exactly: WINNER: first or WINNER: second`;++/** Splits `label=pattern` at the first equals sign. */+export function parseSide(argument) {+ const separator = String(argument ?? '').indexOf('=');+ if (separator < 1 || separator === argument.length - 1) throw new RangeError('Give each side as "label=image pattern", for example "67js=dir/{task}-{attempt}.png".');+ return { label: argument.slice(0, separator), pattern: argument.slice(separator + 1) };+}++function imagePathOf(side, task, attempt) {+ return resolve(side.pattern.replaceAll('{task}', task).replaceAll('{attempt}', String(attempt)));+}++/**+ * One entry per task and attempt, with the seeded order of the two renders. A control pairing shows side A's own+ * image on both sides, so its two inputs are identical.+ */+export function planPairs({ tasks, attempts = 1, seed = DEFAULT_SEED, sides, control = false }) {+ const pairs = [];+ for (const [taskIndex, task] of tasks.entries()) {+ for (let attempt = 1; attempt <= attempts; attempt++) {+ const firstSide = firstSideOf(seed, taskIndex, attempt - 1);+ const secondSide = firstSide === 'a' ? 'b' : 'a';+ const own = { a: sides.a, b: control ? sides.a : sides.b };+ pairs.push({+ task,+ attempt,+ firstSide,+ firstLabel: own[firstSide].label,+ secondLabel: own[secondSide].label,+ firstImage: imagePathOf(own[firstSide], task, attempt),+ secondImage: imagePathOf(own[secondSide], task, attempt),+ });+ }+ }+ return pairs;+}++/** Asks `ask(pair)` once per pair and maps the answer back to the system that won. `ask` returns { answer, costUsd, note }. */+export async function judgePairs({ pairs, ask }) {+ const records = [];+ for (const pair of pairs) {+ const verdict = await ask(pair);+ const answer = verdict.answer ?? null;+ const winnerSide = answer === null ? null : answer === 'first' ? pair.firstSide : (pair.firstSide === 'a' ? 'b' : 'a');+ records.push({+ task: pair.task,+ attempt: pair.attempt,+ firstSide: pair.firstSide,+ answer,+ winnerSide,+ winner: answer === null ? null : answer === 'first' ? pair.firstLabel : pair.secondLabel,+ costUsd: verdict.costUsd ?? 0,+ note: verdict.note,+ });+ }+ return records;+}++/** Wins per system and the exact p for the pairing; for the control, how often "first" was picked and the p of that. */+export function summarizePairing({ records, labels, control }) {+ const decided = records.filter(record => record.answer !== null);+ const costUsd = Number(records.reduce((sum, record) => sum + record.costUsd, 0).toFixed(4));+ if (control) {+ const pickedFirst = decided.filter(record => record.answer === 'first').length;+ return { trials: decided.length, undecided: records.length - decided.length, pickedFirst, pickedSecond: decided.length - pickedFirst, pBias: binomialP(pickedFirst, decided.length), costUsd };+ }+ const winsB = decided.filter(record => record.winnerSide === 'b').length;+ return { trials: decided.length, undecided: records.length - decided.length, wins: { [labels.a]: decided.length - winsB, [labels.b]: winsB }, pValue: binomialP(winsB, decided.length), costUsd };+}++async function readImageBlock(path) {+ return imageBlock(await readFile(path));+}++/** The judge call for real runs: both images and the question in one message to headless Claude. */+export function claudeAsk({ claude, model }) {+ return async pair => {+ const content = [+ { type: 'text', text: PAIRWISE_QUESTION },+ { type: 'text', text: 'First render:' },+ await readImageBlock(pair.firstImage),+ { type: 'text', text: 'Second render:' },+ await readImageBlock(pair.secondImage),+ ];+ return callClaudeJudge({ claude, model, content });+ };+}++/** Runs the pairing (unless controlOnly) and the control, and returns the result file contents. */+export async function runPairwise({ ask, tasks, attempts, seed, sides, controlOnly = false, model = null }) {+ const labels = { a: sides.a.label, b: sides.b.label };+ const result = { mode: 'pairwise', model, seed, tasks, attempts, labels, pairing: null, control: null };+ if (!controlOnly) {+ const records = await judgePairs({ pairs: planPairs({ tasks, attempts, seed, sides }), ask });+ result.pairing = { summary: summarizePairing({ records, labels, control: false }), records };+ }+ const controlRecords = await judgePairs({ pairs: planPairs({ tasks, attempts, seed, sides, control: true }), ask });+ result.control = { label: labels.a, summary: summarizePairing({ records: controlRecords, labels, control: true }), records: controlRecords };+ return result;+}++/** Markdown lines for a result file, as compare-threejs.mjs prints them. */+export function pairwiseTable(result) {+ const lines = [];+ if (result.pairing) {+ const { summary } = result.pairing;+ const [labelA, labelB] = [result.labels.a, result.labels.b];+ lines.push('| Pairwise judge | Trials | Wins | Exact p |', '| --- | --- | --- | --- |');+ lines.push(`| ${labelA} vs ${labelB} | ${summary.trials} | ${labelA} ${summary.wins[labelA]}, ${labelB} ${summary.wins[labelB]} | ${summary.pValue.toFixed(4)} |`);+ }+ const control = result.control.summary;+ lines.push('', '| Control (same render on both sides) | Trials | Picked first | Picked second | Exact p of position bias |', '| --- | --- | --- | --- | --- |');+ lines.push(`| ${result.control.label} | ${control.trials} | ${control.pickedFirst} | ${control.pickedSecond} | ${control.pBias.toFixed(4)} |`);+ return lines.join('\n');+}++if (process.argv[1] && resolve(process.argv[1]) === import.meta.filename) {+ const { values } = parseArgs({ options: {+ claude: { type: 'string', default: process.env.CLAUDE_EXECUTABLE ?? 'claude' },+ model: { type: 'string', default: 'sonnet' },+ name: { type: 'string' },+ tasks: { type: 'string' },+ a: { type: 'string' },+ b: { type: 'string' },+ attempts: { type: 'string', default: '1' },+ seed: { type: 'string', default: String(DEFAULT_SEED) },+ 'control-only': { type: 'boolean', default: false },+ } });+ if (!values.name || !values.tasks || !values.a) throw new Error('Give --name, --tasks and --a (and --b unless --control-only).');+ const sides = { a: parseSide(values.a), b: parseSide(values.b ?? values.a) };+ const result = await runPairwise({+ ask: claudeAsk({ claude: values.claude, model: values.model }),+ tasks: values.tasks.split(','),+ attempts: Number(values.attempts),+ seed: Number(values.seed),+ sides,+ controlOnly: values['control-only'],+ model: values.model,+ });+ const outputPath = join(repository, 'bench', 'agent-modeling', 'results', `${values.name}.json`);+ await mkdir(dirname(outputPath), { recursive: true });+ await writeFile(outputPath, `${JSON.stringify(result, null, 2)}\n`);+ console.log(pairwiseTable(result));+ console.log(`\nWritten to bench/agent-modeling/results/${values.name}.json`);+}diff --git a/bench/agent-modeling/results/2026-11-13-pairwise-control-67js.json b/bench/agent-modeling/results/2026-11-13-pairwise-control-67js.jsonnew file mode 100644index 00000000..faa14985--- /dev/null+++ b/bench/agent-modeling/results/2026-11-13-pairwise-control-67js.json@@ -0,0 +1,181 @@+{+ "mode": "pairwise",+ "model": "sonnet",+ "seed": 53,+ "tasks": [+ "crate",+ "bolt",+ "table",+ "chair",+ "gear"+ ],+ "attempts": 3,+ "labels": {+ "a": "67js MCP",+ "b": "67js MCP"+ },+ "pairing": null,+ "control": {+ "label": "67js MCP",+ "summary": {+ "trials": 15,+ "undecided": 0,+ "pickedFirst": 7,+ "pickedSecond": 8,+ "pBias": 1,+ "costUsd": 1.7564+ },+ "records": [+ {+ "task": "crate",+ "attempt": 1,+ "firstSide": "a",+ "answer": "first",+ "winnerSide": "a",+ "winner": "67js MCP",+ "costUsd": 0.133934,+ "note": "The two renders look identical in shape, proportions, surface, lighting and grounding, so nothing separates them and the pick is arbitrary."+ },+ {+ "task": "crate",+ "attempt": 2,+ "firstSide": "b",+ "answer": "first",+ "winnerSide": "b",+ "winner": "67js MCP",+ "costUsd": 0.133982,+ "note": "The two renders look identical in shape, proportions, surface detail, lighting and shadow, so nothing separates them and I'm picking arbitrarily."+ },+ {+ "task": "crate",+ "attempt": 3,+ "firstSide": "a",+ "answer": "second",+ "winnerSide": "b",+ "winner": "67js MCP",+ "costUsd": 0.133892,+ "note": "The two renders look identical in shape, lighting, shadow, and surface, so nothing separates them; I'm picking the second by default."+ },+ {+ "task": "bolt",+ "attempt": 1,+ "firstSide": "b",+ "answer": "second",+ "winnerSide": "a",+ "winner": "67js MCP",+ "costUsd": 0.13393,+ "note": "The two renders look identical in shape, proportions, surface, lighting and shadow, so nothing separates them and I'm picking the second by default."+ },+ {+ "task": "bolt",+ "attempt": 2,+ "firstSide": "a",+ "answer": "second",+ "winnerSide": "b",+ "winner": "67js MCP",+ "costUsd": 0.13391999999999998,+ "note": "The two renders look identical in shape, lighting, shadow and grounding, so nothing visible separates them; I'm picking the second by default."+ },+ {+ "task": "bolt",+ "attempt": 3,+ "firstSide": "a",+ "answer": "second",+ "winnerSide": "b",+ "winner": "67js MCP",+ "costUsd": 0.0075659,+ "note": "The two renders look identical in shape, proportions, shading and shadow, so nothing separates them; I'm picking the second arbitrarily."+ },+ {+ "task": "table",+ "attempt": 1,+ "firstSide": "b",+ "answer": "second",+ "winnerSide": "a",+ "winner": "67js MCP",+ "costUsd": 0.133938,+ "note": "The two renders look identical in shape, proportions, lighting, shadow and grounding, so nothing separates them, and I'm picking the second by default."+ },+ {+ "task": "table",+ "attempt": 2,+ "firstSide": "a",+ "answer": "second",+ "winnerSide": "b",+ "winner": "67js MCP",+ "costUsd": 0.0075739,+ "note": "The two renders look identical in shape, proportions, surface, lighting and shadow, so nothing separates them; I'm picking the second by default."+ },+ {+ "task": "table",+ "attempt": 3,+ "firstSide": "b",+ "answer": "first",+ "winnerSide": "b",+ "winner": "67js MCP",+ "costUsd": 0.133984,+ "note": "The two renders look identical in shape, proportions, lighting, shadow and grounding, with no visible difference to separate them, so I'm picking the first by default."+ },+ {+ "task": "chair",+ "attempt": 1,+ "firstSide": "a",+ "answer": "first",+ "winnerSide": "a",+ "winner": "67js MCP",+ "costUsd": 0.133856,+ "note": "The two renders look identical in shape, proportions, surface, lighting and grounding, with no visible difference to separate them."+ },+ {+ "task": "chair",+ "attempt": 2,+ "firstSide": "b",+ "answer": "first",+ "winnerSide": "b",+ "winner": "67js MCP",+ "costUsd": 0.134016,+ "note": "The two renders look identical in shape, proportions, surface, lighting, shadow and grounding, so nothing separates them and I'm picking one arbitrarily."+ },+ {+ "task": "chair",+ "attempt": 3,+ "firstSide": "b",+ "answer": "first",+ "winnerSide": "b",+ "winner": "67js MCP",+ "costUsd": 0.13398200000000002,+ "note": "The two renders look identical in shape, proportions, surface, lighting and shadow, with no visible difference to decide on, so I'm picking arbitrarily."+ },+ {+ "task": "gear",+ "attempt": 1,+ "firstSide": "a",+ "answer": "second",+ "winnerSide": "b",+ "winner": "67js MCP",+ "costUsd": 0.13392600000000002,+ "note": "The two renders look identical in shape, proportions, surface, lighting and grounding, so nothing separates them; I'm picking the second by default."+ },+ {+ "task": "gear",+ "attempt": 2,+ "firstSide": "b",+ "answer": "second",+ "winnerSide": "a",+ "winner": "67js MCP",+ "costUsd": 0.13396999999999998,+ "note": "The two renders look identical in shape, proportions, surface, lighting and grounding, so nothing separates them and I'm picking the second arbitrarily."+ },+ {+ "task": "gear",+ "attempt": 3,+ "firstSide": "b",+ "answer": "first",+ "winnerSide": "b",+ "winner": "67js MCP",+ "costUsd": 0.133934,+ "note": "The two renders look identical in shape, proportions, lighting, and shadow, so nothing separates them; I'm picking arbitrarily."+ }+ ]+ }+}diff --git a/bench/agent-modeling/run.mjs b/bench/agent-modeling/run.mjsindex 657d8c3c..b761f15e 100644--- a/bench/agent-modeling/run.mjs+++ b/bench/agent-modeling/run.mjs@@ -24,6 +24,7 @@ import { parseArgs } from 'node:util';import { createRunGuard } from './run-guard.mjs';import { removeLocalPaths, totalsByModel, totalsOf } from './aggregate.mjs';import { gradeRecipe, recipeFeatures } from './check.mjs';+import { countFixRounds, isMcpRender, isMcpWrite } from './fix-rounds.mjs';import { scoreLookSafely } from './look.mjs';import { classifyZeroCalls } from './zero-calls.mjs';import { PROMPT_PREAMBLE, TASKS, taskById, taskPrompt } from './tasks.mjs';@@ -109,7 +110,8 @@ function textOf(content) {/** Tool calls, errors, the model, usage and the last exported recipe from a stream-json transcript. */function readTranscript(output) {- const summary = { model: null, toolCalls: 0, toolErrors: 0, toolsByName: {}, branchCalls: 0, relationCalls: 0, pathQueries: 0, recipe: null, result: null };+ const summary = { model: null, toolCalls: 0, toolErrors: 0, toolsByName: {}, branchCalls: 0, relationCalls: 0, pathQueries: 0, fixRounds: 0, recipe: null, result: null };+ const steps = [];const exportCalls = new Set();for (const line of output.split('\n')) {if (!line.trim().startsWith('{')) continue;@@ -126,6 +128,7 @@ function readTranscript(output) {const name = block.name.replace(`mcp__${MCP_SERVER_NAME}__`, '');// ToolSearch only loads the MCP tool schemas; it is harness overhead, not modeling work.if (block.name !== 'ToolSearch') summary.toolCalls++;+ steps.push({ name, input: block.input });summary.toolsByName[name] = (summary.toolsByName[name] ?? 0) + 1;if (block.input?.branch !== undefined || block.input?.shots?.some(shot => shot.branch !== undefined)) summary.branchCalls++;// The recipe a task hands over may carry relations itself; only calls that send a non-empty list count.@@ -149,6 +152,7 @@ function readTranscript(output) {}if (event.type === 'result') summary.result = event;}+ summary.fixRounds = countFixRounds(steps, { isRender: isMcpRender, isWrite: isMcpWrite });return summary;}@@ -194,6 +198,7 @@ async function runTask(task, attempt, systemPromptAddition) {model: transcript.model,toolCalls: transcript.toolCalls,toolErrors: transcript.toolErrors,+ fixRounds: transcript.fixRounds,toolsByName: transcript.toolsByName,branchCalls: transcript.branchCalls,relationCalls: transcript.relationCalls,diff --git a/bench/agent-modeling/threejs.mjs b/bench/agent-modeling/threejs.mjsindex ff0a376c..ef532fb1 100644--- a/bench/agent-modeling/threejs.mjs+++ b/bench/agent-modeling/threejs.mjs@@ -20,6 +20,7 @@ import { parseArgs } from 'node:util';import { createRunGuard } from './run-guard.mjs';import { removeLocalPaths, totalsByModel, totalsOf } from './aggregate.mjs';import { checkOne } from './check.mjs';+import { countFixRounds, isCodeRender, isCodeWrite } from './fix-rounds.mjs';import { camerasForTask, worldGeometriesOf } from './look.mjs';import { countTips, lineHits, sectionLoops, signedVolume } from './measure.mjs';import { checkPng } from './png-check.mjs';@@ -136,7 +137,8 @@ export async function scoreSilhouetteOfGeometries(taskId, rawGeometries) {}export function readTranscript(output) {- const summary = { model: null, toolCalls: 0, toolErrors: 0, toolsByName: {}, result: null };+ const summary = { model: null, toolCalls: 0, toolErrors: 0, toolsByName: {}, fixRounds: 0, result: null };+ const steps = [];for (const line of output.split('\n')) {if (!line.trim().startsWith('{')) continue;let event;@@ -151,6 +153,7 @@ export function readTranscript(output) {if (block.type !== 'tool_use' || block.name === 'ToolSearch') continue;summary.toolCalls++;summary.toolsByName[block.name] = (summary.toolsByName[block.name] ?? 0) + 1;+ steps.push({ name: block.name, input: block.input });}}if (event.type === 'user') {@@ -160,6 +163,7 @@ export function readTranscript(output) {}if (event.type === 'result') summary.result = event;}+ summary.fixRounds = countFixRounds(steps, { isRender: isCodeRender, isWrite: isCodeWrite });return summary;}@@ -267,6 +271,7 @@ if (process.argv[1] === fileURLToPath(import.meta.url)) {model: transcript.model,toolCalls: transcript.toolCalls,toolErrors: transcript.toolErrors,+ fixRounds: transcript.fixRounds,toolsByName: transcript.toolsByName,look: silhouette ? { silhouette } : null,turns: transcript.result?.num_turns ?? null,diff --git a/changes/987.md b/changes/987.mdnew file mode 100644index 00000000..e2263ead--- /dev/null+++ b/changes/987.md@@ -0,0 +1 @@+- Added a blind pairwise look judge for 67js and three.js renders of one task (seeded first and second order, a same-render control for position bias, wins per system with an exact binomial p), printed by `compare-threejs.mjs --pairwise`, and a `fixRounds` count in run records (#987).diff --git a/tests/agent-modeling-suite.test.js b/tests/agent-modeling-suite.test.jsindex 6de781a3..2e6807d1 100644--- a/tests/agent-modeling-suite.test.js+++ b/tests/agent-modeling-suite.test.js@@ -7,6 +7,10 @@ import { GALLERY_LIGHT_POSITIONS } from '../bench/agent-modeling/gallery.mjs';import { COLONNADE_BEST_PLACE, COLONNADE_LIGHT_POSITIONS } from '../bench/agent-modeling/colonnade.mjs';import { DUSK_GRADES } from '../bench/agent-modeling/dusk.mjs';import { EASING_PLAIN_SOLUTION } from '../bench/agent-modeling/easing.mjs';+import { binomialP } from '../bench/agent-modeling/default-judge.mjs';+import { countFixRounds, isMcpRender, isMcpWrite } from '../bench/agent-modeling/fix-rounds.mjs';+import { judgePairs, pairwiseTable, planPairs, runPairwise, summarizePairing } from '../bench/agent-modeling/pairwise-judge.mjs';+import { readTranscript as readThreejsTranscript } from '../bench/agent-modeling/threejs.mjs';test('the look pick passes the golden grade and fails the dim, the pale and a moved scene', async () => {const task = taskById('look-pick');@@ -298,3 +302,97 @@ test('the plain overshoot task forbids easings in its prompt and grades the handassert.match(task.prompt, /do not use the easing field/);assert.deepEqual(failedKinds(await gradeRecipe(task, structuredClone(REFERENCES['overshoot-lift-plain']))), []);});++const PAIRWISE_SIDES = {+ a: { label: '67js MCP', pattern: 'renders/mcp/{attempt}/{task}.png' },+ b: { label: 'three.js by code', pattern: 'renders/code/{task}-{attempt}.png' },+};++test('the pairwise order is a function of the seed, and repeats of one task swap sides', () => {+ const tasks = ['crate', 'bolt', 'table'];+ const plan = seed => planPairs({ tasks, attempts: 3, seed, sides: PAIRWISE_SIDES }).map(pair => `${pair.task}${pair.attempt}${pair.firstSide}`);+ assert.deepEqual(plan(53), plan(53));+ assert.notDeepEqual(plan(53), plan(54));+ const pairs = planPairs({ tasks, attempts: 3, seed: 53, sides: PAIRWISE_SIDES });+ for (const pair of pairs.filter(candidate => candidate.attempt === 2)) {+ const first = pairs.find(candidate => candidate.task === pair.task && candidate.attempt === 1);+ assert.notEqual(pair.firstSide, first.firstSide, 'attempt 2 shows the other side first');+ }+ const crate = pairs[0];+ assert.equal(crate.firstLabel, PAIRWISE_SIDES[crate.firstSide].label);+ assert.match(crate.firstImage.replaceAll('\\', '/'), crate.firstSide === 'a' ? /renders\/mcp\/1\/crate\.png$/ : /renders\/code\/crate-1\.png$/);+});++test('the pairwise win count follows a fixed set of answers and its exact p', async () => {+ const tasks = ['crate', 'bolt', 'table', 'fence', 'window', 'bracket'];+ const pairs = planPairs({ tasks, attempts: 1, seed: 53, sides: PAIRWISE_SIDES });+ // The stub always names side B's image, wherever it sits, except for the last task, which it gives to side A.+ const ask = async pair => {+ const wantedSide = pair.task === 'bracket' ? 'a' : 'b';+ return { answer: pair.firstSide === wantedSide ? 'first' : 'second', costUsd: 0.01 };+ };+ const records = await judgePairs({ pairs, ask });+ const summary = summarizePairing({ records, labels: { a: '67js MCP', b: 'three.js by code' }, control: false });+ assert.deepEqual(summary.wins, { '67js MCP': 1, 'three.js by code': 5 });+ assert.equal(summary.trials, 6);+ assert.ok(Math.abs(summary.pValue - 0.21875) < 1e-9, 'two-sided p of 5 of 6 is 14/64');+ assert.equal(summary.costUsd, 0.06);+ const undecided = await judgePairs({ pairs: pairs.slice(0, 2), ask: async () => ({ answer: null }) });+ assert.equal(summarizePairing({ records: undecided, labels: { a: 'a', b: 'b' }, control: false }).trials, 0);+});++test('the binomial p matches known values', () => {+ assert.equal(binomialP(0, 0), 1);+ assert.equal(binomialP(9, 18), 1);+ assert.ok(Math.abs(binomialP(18, 18) - 2 * 0.5 ** 18) < 1e-12);+ assert.ok(Math.abs(binomialP(0, 10) - 2 / 1024) < 1e-12);+ assert.ok(Math.abs(binomialP(13, 18) - 0.096) < 0.001, '13 of 18 is not significant');+});++test('the control pairs one render with itself and reports position bias', async () => {+ const pairs = planPairs({ tasks: ['crate', 'bolt', 'table', 'chair'], attempts: 2, seed: 53, sides: PAIRWISE_SIDES, control: true });+ assert.equal(pairs.length, 8);+ for (const pair of pairs) {+ assert.equal(pair.firstImage, pair.secondImage, 'identical input on both sides');+ assert.equal(pair.firstLabel, '67js MCP');+ assert.equal(pair.secondLabel, '67js MCP');+ }+ const records = await judgePairs({ pairs, ask: async () => ({ answer: 'first' }) });+ const summary = summarizePairing({ records, labels: { a: '67js MCP', b: 'three.js by code' }, control: true });+ assert.equal(summary.pickedFirst, 8);+ assert.equal(summary.pickedSecond, 0);+ assert.ok(summary.pBias < 0.01, 'always picking first is position bias');+});++test('runPairwise calls the judge once per pair and once per control pair, with no model', async () => {+ let calls = 0;+ const result = await runPairwise({ ask: async () => { calls++; return { answer: calls % 2 ? 'first' : 'second' }; }, tasks: ['crate', 'bolt'], attempts: 2, seed: 53, sides: PAIRWISE_SIDES });+ assert.equal(calls, 8);+ assert.equal(result.pairing.records.length, 4);+ assert.equal(result.control.records.length, 4);+ assert.match(pairwiseTable(result), /Exact p/);+ calls = 0;+ const controlOnly = await runPairwise({ ask: async () => { calls++; return { answer: 'first' }; }, tasks: ['crate'], attempts: 3, seed: 53, sides: PAIRWISE_SIDES, controlOnly: true });+ assert.equal(calls, 3);+ assert.equal(controlOnly.pairing, null);+});++test('fix rounds count the writes after the first render, in both systems', () => {+ const mcpSteps = ['scene_create', 'scene_apply', 'scene_look', 'scene_apply', 'scene_look', 'scene_apply', 'scene_export_recipe']+ .map(name => ({ name: `mcp__67js__${name}` }));+ assert.equal(countFixRounds(mcpSteps, { isRender: isMcpRender, isWrite: isMcpWrite }), 2);+ assert.equal(countFixRounds(mcpSteps.slice(0, 2), { isRender: isMcpRender, isWrite: isMcpWrite }), 0, 'never rendered');++ const toolUse = (name, input) => JSON.stringify({ type: 'assistant', message: { content: [{ type: 'tool_use', id: name, name, input }] } });+ const transcript = [+ toolUse('Write', { file_path: 'build.mjs' }),+ toolUse('Bash', { command: 'node build.mjs' }),+ toolUse('Edit', { file_path: 'build.mjs' }),+ toolUse('Bash', { command: 'node build.mjs' }),+ toolUse('Write', { file_path: 'build.mjs' }),+ toolUse('Bash', { command: 'node build.mjs' }),+ ].join('\n');+ const summary = readThreejsTranscript(transcript);+ assert.equal(summary.toolCalls, 6);+ assert.equal(summary.fixRounds, 2);+});