3D engine that AI agents drive
Compare 67js MCP with three.js by code on six suite tasks (part of #31)
Nguyen Xuan An91ab410faafb
diff --git a/bench/agent-modeling/README.md b/bench/agent-modeling/README.mdindex ee760092..a7cf83a7 100644--- a/bench/agent-modeling/README.md+++ b/bench/agent-modeling/README.md@@ -394,3 +394,37 @@ Not run:### Antigravity CLI (2026-11-07)`llms-only.mjs --agent agy --agy <agy executable> --model <model>` drives the Antigravity CLI, which offers Gemini 4 Argon, Gemini 3.8 Flash and GPT-OSS 120B. A first run of 2 attempts per model passed 0 of 6, but not on the docs: headless mode denies every shell command it cannot prompt for, so the agents read `llms.txt` and the package, wrote `render.mjs` (Gemini 4 Argon), and were refused `node render.mjs`. Those results are not kept. A valid run needs a session-only allow rule for commands, passed with `--settings`, which An has to approve; the runner records any web tool call as `usedWeb` and fails it, since the CLI cannot switch its web tools off.++## Agent friendliness: 67js MCP against three.js by code (2026-11-07, issue #31)++The three.js half of issue #31 and the 67js side of the same table. The Blender MCP half needs a VM and is not run. Model `sonnet`, headless `claude -p` on the subscription, 6 suite tasks (`crate`, `bolt`, `window`, `table`, `fence`, `bracket`), 1 run each per system, so 12 records in `results/2026-11-07-threejs-vs-67js.json`. Tokens and tool calls are reported; there is no API dollar figure.++- 67js MCP: `run.mjs`, the task prompt behind the usual MCP preamble.+- three.js: `threejs.mjs`, the same task text behind a different preamble. The agent gets an empty folder with `three` installed from npm, Bash, Read, Write, Edit, Glob and Grep, no web and no MCP. It writes `build.mjs`, which builds the object, writes `geometry.json` (world-space triangles per mesh) and renders `scene.png` with headless Chrome on SwiftShader.+- Same grading: the task's own checks run on `geometry.json` (corners welded before the closed-surface, hole and loop counts; the grader passes the suite's reference geometry of all six tasks, see `tests/agent-threejs-grade.test.js`), and the look score is the suite's silhouette overlap from the three fixed cameras. A three.js run passes when every check passes and `scene.png` is a real picture.++| System | Pass | Mean tool calls | Mean tokens (in+out) | Mean look | Mean wall seconds |+| --- | --- | --- | --- | --- | --- |+| 67js MCP | 6/6 | 3.2 | 375k | 0.932 | 24 |+| three.js by code | 5/6 | 8.8 | 269k | 0.926 | 359 |++| Task | System | Pass | Tool calls (errors) | Look | Failure |+| --- | --- | --- | --- | --- | --- |+| crate | 67js MCP | yes | 4 (0) | 1.000 | |+| crate | three.js | yes | 9 (1) | 1.000 | |+| bolt | 67js MCP | yes | 2 (0) | 1.000 | |+| bolt | three.js | yes | 7 (3) | 0.964 | |+| window | 67js MCP | yes | 4 (0) | 1.000 | |+| window | three.js | no, 4/5 | 11 (0) | 1.000 | closed-surface check: the frame has open or doubled edges |+| table | 67js MCP | yes | 2 (0) | 0.592 | |+| table | three.js | yes | 8 (1) | 0.592 | |+| fence | 67js MCP | yes | 2 (0) | 1.000 | |+| fence | three.js | yes | 11 (1) | 1.000 | |+| bracket | 67js MCP | yes | 5 (0) | 1.000 | |+| bracket | three.js | yes | 7 (0) | 1.000 | |++- Measured: 67js needed about a third of the tool calls and finished in about 24 s against about 6 minutes, most of the three.js time going to getting headless Chrome to write a PNG. Both systems built the same shapes: the look means are within noise, and `table` scores 0.592 for both, which points at the task's reference rather than at either system.+- Tokens are not lower for 67js: inferred: the MCP tool schemas and server instructions are re-read on every call, so its input tokens are higher even with fewer calls. The three.js `table` run reports only 44k input tokens although it ran 871 s, so treat that one count as unreliable.+- 67js failures: none in the checks. One harness fault: the `bracket` look images failed because the look page could not fetch the boolean wasm (`both async and sync fetching of the wasm failed`), which threw away the whole look record, silhouette included. The silhouette was scored again afterwards from the exported recipe, and the record carries `imagesError`. `scoreLook` should score the silhouette before it renders images.+- A first attempt of the 67js run failed on every task because the checkout had no `build/mcp` folder (`npm run build:lib` creates it); those results were discarded. The first three.js run stopped after `bolt` when its process disappeared, so `window`, `table`, `fence` and `bracket` were run again as a second run and joined with `compare-threejs.mjs`.+- Limits: 1 run per cell, one model, tasks the 67js suite was built around (they favour 67js's checks, not three.js). The `window` three.js failure label came from an earlier wording of the same check.diff --git a/bench/agent-modeling/check.mjs b/bench/agent-modeling/check.mjsindex c59736a7..b8e0e5cd 100644--- a/bench/agent-modeling/check.mjs+++ b/bench/agent-modeling/check.mjs@@ -201,7 +201,7 @@ function checkLitPainting(check, measures) {return { pass: problems.length === 0, detail: problems.join('; ') || `place ${placeIndex + 1}, ${shadowed} of ${total} points in shadow` };}-function checkOne(check, measures) {+export function checkOne(check, measures) {switch (check.kind) {case 'lint':return { pass: measures.lintProblems === 0 && measures.closedParts === measures.parts, detail: measures.lintProblems ? measures.problemKinds.join(', ') : `${measures.closedParts}/${measures.parts} parts closed` };diff --git a/bench/agent-modeling/compare-threejs.mjs b/bench/agent-modeling/compare-threejs.mjsnew file mode 100644index 00000000..6a701b31--- /dev/null+++ b/bench/agent-modeling/compare-threejs.mjs@@ -0,0 +1,60 @@+// Joins one 67js run (run.mjs) and one three.js run (threejs.mjs) into the comparison file of issue #31 and prints+// the markdown tables.+// node bench/agent-modeling/compare-threejs.mjs <67js run json> <three.js run json> <output json>+import { readFile, writeFile } from 'node:fs/promises';++const [mcpPath, codePath, outputPath] = process.argv.slice(2);+const mcpRun = JSON.parse(await readFile(mcpPath, 'utf8'));+// The three.js run may be split over several files (comma separated) when a run was stopped and the rest redone.+const codeRuns = await Promise.all(codePath.split(',').map(async path => JSON.parse(await readFile(path, 'utf8'))));+const codeRun = { ...codeRuns[0], records: codeRuns.flatMap(run => run.records) };++const SYSTEMS = [+ { name: '67js MCP', run: mcpRun },+ { name: 'three.js by code', run: codeRun },+];+const records = SYSTEMS.flatMap(({ name, run }) => run.records.map(record => ({ system: name, ...record })));++const mean = values => values.reduce((sum, value) => sum + value, 0) / values.length;+const lookOf = record => record.look?.silhouette?.mean ?? null;+const tokensOf = record => record.inputTokens + record.outputTokens;+const thousands = value => `${Math.round(value / 1000)}k`;++const taskIds = [...new Set(records.map(record => record.task))];+const lines = ['| Task | System | Pass | Tool calls | Tool errors | Tokens in / out | Look | Failures |', '| --- | --- | --- | --- | --- | --- | --- | --- |'];+for (const task of taskIds) {+ for (const { name } of SYSTEMS) {+ const record = records.find(candidate => candidate.task === task && candidate.system === name);+ lines.push(`| ${task} | ${name} | ${record.pass ? 'yes' : 'no'} (${record.passed}/${record.total}) | ${record.toolCalls} | ${record.toolErrors} | ${thousands(record.inputTokens)} / ${thousands(record.outputTokens)} | ${lookOf(record)?.toFixed(3) ?? '-'} | ${record.failed.join('; ') || '-'} |`);+ }+}+const totals = ['| System | Pass | Mean tool calls | Mean tokens | Mean look | Mean wall seconds |', '| --- | --- | --- | --- | --- | --- |'];+const summary = {};+for (const { name } of SYSTEMS) {+ const own = records.filter(record => record.system === name);+ const looks = own.map(lookOf).filter(value => value !== null);+ summary[name] = {+ runs: own.length,+ passed: own.filter(record => record.pass).length,+ meanToolCalls: Number(mean(own.map(record => record.toolCalls)).toFixed(1)),+ meanTokens: Math.round(mean(own.map(tokensOf))),+ meanLook: Number(mean(looks).toFixed(3)),+ meanWallSeconds: Math.round(mean(own.map(record => record.wallSeconds))),+ };+ const row = summary[name];+ totals.push(`| ${name} | ${row.passed}/${row.runs} | ${row.meanToolCalls} | ${thousands(row.meanTokens)} | ${row.meanLook} | ${row.meanWallSeconds} |`);+}++const output = {+ run: '2026-11-07-threejs-vs-67js',+ issue: 31,+ model: mcpRun.model,+ date: new Date().toISOString(),+ scope: 'three.js half and 67js MCP side; the official Blender MCP half needs a VM and is not run',+ threeVersion: codeRun.threeVersion,+ runsPerTaskPerSystem: 1,+ summary,+ records,+};+await writeFile(outputPath, `${JSON.stringify(output, null, 2)}\n`);+console.log(`${totals.join('\n')}\n\n${lines.join('\n')}`);diff --git a/bench/agent-modeling/measure.mjs b/bench/agent-modeling/measure.mjsindex 7f503841..c06b872a 100644--- a/bench/agent-modeling/measure.mjs+++ b/bench/agent-modeling/measure.mjs@@ -59,7 +59,7 @@ const triangleArea = (positions, first, second, third) => {return Math.hypot(ay * bz - az * by, az * bx - ax * bz, ax * by - ay * bx) / 2;};-function signedVolume({ positions, indices }) {+export function signedVolume({ positions, indices }) {let volume = 0;for (let triangle = 0; triangle < indices.length; triangle += 3) {const [first, second, third] = [indices[triangle], indices[triangle + 1], indices[triangle + 2]];@@ -136,7 +136,7 @@ export function sectionLoops(geometries, axis, at) {* Count separate outer tips around the thinnest axis (gear teeth, star points). Vertices near the outer radius are* joined when a triangle edge runs between them along the rim, so a wide flat tooth tip still counts once.*/-function countTips(geometries, bounds) {+export function countTips(geometries, bounds) {const extents = [0, 1, 2].map(axis => bounds.max[axis] - bounds.min[axis]);const thinAxis = extents.indexOf(Math.min(...extents));const [firstAxis, secondAxis] = [0, 1, 2].filter(axis => axis !== thinAxis);@@ -199,7 +199,7 @@ function surfaceHeightAt({ positions, indices }, pointX, pointZ) {return null;}-function lineHits(geometries, axis, point) {+export function lineHits(geometries, axis, point) {const axisIndex = { x: 0, y: 1, z: 2 }[axis];const [firstAxis, secondAxis] = [0, 1, 2].filter(candidate => candidate !== axisIndex);let hits = 0;diff --git a/bench/agent-modeling/results/2026-11-07-threejs-vs-67js.json b/bench/agent-modeling/results/2026-11-07-threejs-vs-67js.jsonnew file mode 100644index 00000000..a73f4400--- /dev/null+++ b/bench/agent-modeling/results/2026-11-07-threejs-vs-67js.json@@ -0,0 +1,580 @@+{+ "run": "2026-11-07-threejs-vs-67js",+ "issue": 31,+ "model": "claude-sonnet-5-5",+ "date": "2026-11-07T16:28:52.048Z",+ "scope": "three.js half and 67js MCP side; the official Blender MCP half needs a VM and is not run",+ "threeVersion": "0.187.0",+ "runsPerTaskPerSystem": 1,+ "summary": {+ "67js MCP": {+ "runs": 6,+ "passed": 6,+ "meanToolCalls": 3.2,+ "meanTokens": 374760,+ "meanLook": 0.932,+ "meanWallSeconds": 24+ },+ "three.js by code": {+ "runs": 6,+ "passed": 5,+ "meanToolCalls": 8.8,+ "meanTokens": 268655,+ "meanLook": 0.926,+ "meanWallSeconds": 359+ }+ },+ "records": [+ {+ "system": "67js MCP",+ "task": "crate",+ "attempt": 1,+ "pass": true,+ "passed": 8,+ "total": 8,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 4,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_look": 1,+ "mesh_edit": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "mesh",+ "box"+ ],+ "meshOperations": [+ "inset_individual"+ ],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "crate-front.png",+ "crate-side.png",+ "crate-three-quarter.png"+ ]+ },+ "turns": 6,+ "inputTokens": 445726,+ "outputTokens": 1751,+ "wallSeconds": 27,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "system": "67js MCP",+ "task": "bolt",+ "attempt": 1,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 2,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "cylinder"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "bolt-front.png",+ "bolt-side.png",+ "bolt-three-quarter.png"+ ]+ },+ "turns": 4,+ "inputTokens": 232980,+ "outputTokens": 1112,+ "wallSeconds": 21,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "system": "67js MCP",+ "task": "window",+ "attempt": 1,+ "pass": true,+ "passed": 5,+ "total": 5,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 4,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 2,+ "scene_create": 1,+ "scene_inspect": 1,+ "mesh_inspect": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "extrude"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "window-front.png",+ "window-side.png",+ "window-three-quarter.png"+ ]+ },+ "turns": 7,+ "inputTokens": 546049,+ "outputTokens": 1464,+ "wallSeconds": 26,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "system": "67js MCP",+ "task": "table",+ "attempt": 1,+ "pass": true,+ "passed": 6,+ "total": 6,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 2,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 0.1684,+ "three-quarter": 0.6073+ },+ "mean": 0.5919+ },+ "images": [+ "table-front.png",+ "table-side.png",+ "table-three-quarter.png"+ ]+ },+ "turns": 4,+ "inputTokens": 232283,+ "outputTokens": 1109,+ "wallSeconds": 18,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "system": "67js MCP",+ "task": "fence",+ "attempt": 1,+ "pass": true,+ "passed": 6,+ "total": 6,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 2,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "fence-front.png",+ "fence-side.png",+ "fence-three-quarter.png"+ ]+ },+ "turns": 4,+ "inputTokens": 234011,+ "outputTokens": 1253,+ "wallSeconds": 20,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "system": "67js MCP",+ "task": "bracket",+ "attempt": 1,+ "pass": true,+ "passed": 6,+ "total": 6,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 5,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "mesh_inspect": 1,+ "mesh_edit": 1,+ "scene_apply": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "mesh"+ ],+ "meshOperations": [+ "boolean"+ ],+ "motionIk": [],+ "motionTracks": 0+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "imagesError": "the look harness page could not fetch the boolean wasm, so no preview images; the silhouette is scored from the exported recipe afterwards"+ },+ "turns": 7,+ "inputTokens": 548871,+ "outputTokens": 1949,+ "wallSeconds": 30,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "system": "three.js by code",+ "task": "crate",+ "attempt": 1,+ "pass": true,+ "passed": 8,+ "total": 8,+ "pngPass": true,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 9,+ "toolErrors": 1,+ "toolsByName": {+ "Bash": 4,+ "TaskStop": 2,+ "Edit": 2,+ "Read": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ }+ },+ "turns": 10,+ "inputTokens": 317368,+ "outputTokens": 6905,+ "wallSeconds": 392,+ "exitCode": 0,+ "stopped": "success",+ "filesWritten": [+ ".chrome-profile",+ "build.mjs",+ "geometry.json",+ "package.json",+ "scene.png"+ ]+ },+ {+ "system": "three.js by code",+ "task": "bolt",+ "attempt": 1,+ "pass": true,+ "passed": 7,+ "total": 7,+ "pngPass": true,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 7,+ "toolErrors": 3,+ "toolsByName": {+ "Bash": 6,+ "Read": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 0.9621,+ "side": 0.9621,+ "three-quarter": 0.9666+ },+ "mean": 0.9636+ }+ },+ "turns": 8,+ "inputTokens": 308935,+ "outputTokens": 5416,+ "wallSeconds": 182,+ "exitCode": 0,+ "stopped": "success",+ "filesWritten": [+ "build.mjs",+ "chrome-profile",+ "geometry.json",+ "package.json",+ "render.html",+ "scene.png"+ ]+ },+ {+ "system": "three.js by code",+ "task": "window",+ "attempt": 1,+ "pass": false,+ "passed": 4,+ "total": 5,+ "pngPass": true,+ "failed": [+ "lint: open edges"+ ],+ "model": "claude-sonnet-5-5",+ "toolCalls": 11,+ "toolErrors": 0,+ "toolsByName": {+ "Bash": 5,+ "TaskStop": 2,+ "Edit": 3,+ "Read": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ }+ },+ "turns": 12,+ "inputTokens": 330002,+ "outputTokens": 4616,+ "wallSeconds": 302,+ "exitCode": 0,+ "stopped": "success",+ "filesWritten": [+ "build.mjs",+ "geometry.json",+ "package.json",+ "render.html",+ "scene.png"+ ]+ },+ {+ "system": "three.js by code",+ "task": "table",+ "attempt": 1,+ "pass": true,+ "passed": 6,+ "total": 6,+ "pngPass": true,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 8,+ "toolErrors": 1,+ "toolsByName": {+ "Bash": 7,+ "Read": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 0.1684,+ "three-quarter": 0.6073+ },+ "mean": 0.5919+ }+ },+ "turns": 1,+ "inputTokens": 44307,+ "outputTokens": 84,+ "wallSeconds": 871,+ "exitCode": 0,+ "stopped": "success",+ "filesWritten": [+ "build.mjs",+ "geometry.json",+ "package.json",+ "scene.png"+ ]+ },+ {+ "system": "three.js by code",+ "task": "fence",+ "attempt": 1,+ "pass": true,+ "passed": 6,+ "total": 6,+ "pngPass": true,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 11,+ "toolErrors": 1,+ "toolsByName": {+ "Bash": 5,+ "TaskStop": 2,+ "PowerShell": 1,+ "Edit": 2,+ "Read": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ }+ },+ "turns": 13,+ "inputTokens": 349860,+ "outputTokens": 3328,+ "wallSeconds": 292,+ "exitCode": 0,+ "stopped": "success",+ "filesWritten": [+ "build.mjs",+ "geometry.json",+ "package.json",+ "scene.png"+ ]+ },+ {+ "system": "three.js by code",+ "task": "bracket",+ "attempt": 1,+ "pass": true,+ "passed": 6,+ "total": 6,+ "pngPass": true,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 7,+ "toolErrors": 0,+ "toolsByName": {+ "Bash": 4,+ "Write": 1,+ "Read": 2+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ }+ },+ "turns": 8,+ "inputTokens": 234719,+ "outputTokens": 6392,+ "wallSeconds": 114,+ "exitCode": 0,+ "stopped": "success",+ "filesWritten": [+ "build.mjs",+ "geometry.json",+ "package.json",+ "render.html",+ "scene.png"+ ]+ }+ ]+}diff --git a/bench/agent-modeling/threejs.mjs b/bench/agent-modeling/threejs.mjsnew file mode 100644index 00000000..83fed2c3--- /dev/null+++ b/bench/agent-modeling/threejs.mjs@@ -0,0 +1,292 @@+// The three.js side of the agent friendliness comparison (issue #31): the same task prompts and model as run.mjs, but the+// agent has no MCP. It gets an empty folder with `three` installed from npm and writes code.+//+// node bench/agent-modeling/threejs.mjs --claude <claude executable> [--model sonnet] [--tasks crate,bolt] [--runs 1]+// [--budget 3] [--timeout 1200] [--label threejs]+//+// The agent writes build.mjs, which builds the object with three.js, writes geometry.json (the world-space triangles of+// every mesh) and renders scene.png with headless Chrome. The same task checks and the same silhouette look score as+// the 67js runs grade geometry.json, so the two systems are measured on the same shape. A run passes when every check+// passes and scene.png is a real picture.+//+// Raw transcripts go to artifacts/agent-modeling/<run>/ (ignored by Git); the summary goes to+// bench/agent-modeling/results/<run>.json.+import { execFileSync, spawn } from 'node:child_process';+import { mkdir, mkdtemp, readFile, readdir, rm, writeFile, cp } from 'node:fs/promises';+import { tmpdir } from 'node:os';+import { join, resolve } from 'node:path';+import { fileURLToPath } from 'node:url';+import { parseArgs } from 'node:util';+import { removeLocalPaths, totalsByModel, totalsOf } from './aggregate.mjs';+import { checkOne } from './check.mjs';+import { camerasForTask, worldGeometriesOf } from './look.mjs';+import { countTips, lineHits, sectionLoops, signedVolume } from './measure.mjs';+import { checkPng } from './png-check.mjs';+import { REFERENCES } from './references.mjs';+import { intersectionOverUnion, rasterizeSilhouette } from './silhouette.mjs';+import { taskById } from './tasks.mjs';++const repository = resolve(import.meta.dirname, '../..');+const BUILT_IN_TOOLS_DENIED = ['WebFetch', 'WebSearch', 'Agent', 'Task', 'NotebookEdit'];+const WELD_SCALE = 1e5;++export const THREEJS_PREAMBLE = `You are modeling with three.js code. The folder you are in has three installed from npm (node_modules/three). Build the+object described below as a closed, watertight solid, in metres with Y up. Rest the object on the ground (lowest point at+Y = 0), centred on X = 0 and Z = 0.+Write a Node.js script named build.mjs that does three things and run it:+1. Builds the object with three.js.+2. Writes geometry.json: a JSON array with one entry { "positions": [...], "indices": [...] } per mesh, with every vertex in+ world space (apply each mesh's world matrix) and indices as a flat triangle list.+3. Renders the scene to scene.png, 512 by 512 pixels, with headless Google Chrome (installed; use --use-angle=swiftshader+ --enable-unsafe-swiftshader --enable-webgl --ignore-gpu-blocklist so it needs no GPU). There is no internet access.+The files build.mjs, geometry.json and scene.png are what gets graded. When they exist, reply with one line saying so.`;++/** Measures an array of { positions, indices } meshes like measure.mjs does for a recipe, welding shared corners first. */+export function measureGeometries(rawGeometries) {+ const geometries = rawGeometries.map(({ positions, indices }) => ({+ positions: Float32Array.from(positions),+ indices: Uint32Array.from(indices ?? Array.from({ length: positions.length / 3 }, (_, index) => index)),+ }));+ const bounds = { min: [Infinity, Infinity, Infinity], max: [-Infinity, -Infinity, -Infinity] };+ let signed = 0;+ let triangles = 0;+ let vertexCount = 0;+ let edgeCount = 0;+ let faceCount = 0;+ let components = 0;+ let closedComponents = 0;+ for (const geometry of geometries) {+ const weldedIndex = new Map();+ const welded = [];+ for (let vertex = 0; vertex < geometry.positions.length / 3; vertex++) {+ const point = [0, 1, 2].map(axis => geometry.positions[vertex * 3 + axis]);+ for (let axis = 0; axis < 3; axis++) {+ bounds.min[axis] = Math.min(bounds.min[axis], point[axis]);+ bounds.max[axis] = Math.max(bounds.max[axis], point[axis]);+ }+ const key = point.map(value => Math.round(value * WELD_SCALE)).join(',');+ if (!weldedIndex.has(key)) weldedIndex.set(key, weldedIndex.size);+ welded.push(weldedIndex.get(key));+ }+ signed += signedVolume(geometry);+ triangles += geometry.indices.length / 3;+ const edgeUses = new Map();+ const parent = Array.from({ length: weldedIndex.size }, (_, index) => index);+ const find = vertex => (parent[vertex] === vertex ? vertex : (parent[vertex] = find(parent[vertex])));+ let faces = 0;+ for (let triangle = 0; triangle < geometry.indices.length; triangle += 3) {+ const corners = [0, 1, 2].map(corner => welded[geometry.indices[triangle + corner]]);+ if (new Set(corners).size < 3) continue;+ faces++;+ for (let corner = 0; corner < 3; corner++) {+ const [from, to] = [corners[corner], corners[(corner + 1) % 3]];+ const key = from < to ? `${from}-${to}` : `${to}-${from}`;+ edgeUses.set(key, (edgeUses.get(key) ?? 0) + 1);+ parent[find(from)] = find(to);+ }+ }+ const roots = new Set(Array.from({ length: weldedIndex.size }, (_, vertex) => find(vertex)));+ components += roots.size;+ const closed = [...edgeUses.values()].every(uses => uses === 2);+ if (closed) closedComponents += roots.size;+ vertexCount += weldedIndex.size;+ edgeCount += edgeUses.size;+ faceCount += faces;+ }+ const allClosed = closedComponents === components;+ const volume = Math.abs(signed);+ return {+ parts: components,+ closedParts: closedComponents,+ lintProblems: allClosed ? 0 : components - closedComponents,+ problemKinds: allClosed ? [] : ['open or non-manifold edges'],+ holes: allClosed ? (2 * components - (vertexCount - edgeCount + faceCount)) / 2 : null,+ volume,+ triangles,+ bounds,+ size: [0, 1, 2].map(axis => bounds.max[axis] - bounds.min[axis]),+ tips: countTips(geometries, bounds),+ section: (axis, at) => sectionLoops(geometries, axis, at),+ lineHits: (axis, point) => lineHits(geometries, axis, point),+ };+}++/** Same shape as gradeRecipe: { pass, passed, total, results }. Tasks with checks this module cannot measure fail loudly. */+export function gradeGeometries(task, rawGeometries) {+ const unsupported = task.checks.filter(check => !['lint', 'parts', 'size', 'extent', 'ground', 'holes', 'volume', 'section', 'through', 'tips', 'triangles'].includes(check.kind));+ if (unsupported.length) throw new RangeError(`threejs.mjs cannot grade ${unsupported.map(check => check.kind).join(', ')}`);+ const measures = measureGeometries(rawGeometries);+ const results = task.checks.map(check => ({ kind: check.kind, ...(check.label ? { label: check.label } : {}), ...checkOne(check, measures) }));+ const passed = results.filter(result => result.pass).length;+ return { pass: passed === results.length, passed, total: results.length, results };+}++/** The look score of the suite (silhouette overlap from the three fixed cameras) for loose geometries. */+export async function scoreSilhouetteOfGeometries(taskId, rawGeometries) {+ const views = await camerasForTask(taskId);+ const reference = await worldGeometriesOf(REFERENCES[taskId]);+ const candidate = rawGeometries.map(({ positions, indices }) => ({ positions: Float32Array.from(positions), indices: Uint32Array.from(indices) }));+ const overlap = {};+ for (const { name, camera } of views) {+ overlap[name] = Number(intersectionOverUnion(rasterizeSilhouette(candidate, camera), rasterizeSilhouette(reference, camera)).toFixed(4));+ }+ const values = Object.values(overlap);+ return { views: overlap, mean: Number((values.reduce((sum, value) => sum + value, 0) / values.length).toFixed(4)) };+}++function readTranscript(output) {+ const summary = { model: null, toolCalls: 0, toolErrors: 0, toolsByName: {}, result: null };+ for (const line of output.split('\n')) {+ if (!line.trim().startsWith('{')) continue;+ let event;+ try {+ event = JSON.parse(line);+ } catch {+ continue;+ }+ if (event.type === 'system' && event.subtype === 'init') summary.model = event.model;+ if (event.type === 'assistant') {+ for (const block of event.message?.content ?? []) {+ if (block.type !== 'tool_use' || block.name === 'ToolSearch') continue;+ summary.toolCalls++;+ summary.toolsByName[block.name] = (summary.toolsByName[block.name] ?? 0) + 1;+ }+ }+ if (event.type === 'user') {+ for (const block of event.message?.content ?? []) {+ if (block.type === 'tool_result' && block.is_error) summary.toolErrors++;+ }+ }+ if (event.type === 'result') summary.result = event;+ }+ return summary;+}++async function removeFolder(folder) {+ try {+ await rm(folder, { recursive: true, force: true, maxRetries: 10, retryDelay: 500 });+ } catch (error) {+ console.warn(`Left a temporary folder behind: ${error.code}`);+ }+}++if (process.argv[1] === fileURLToPath(import.meta.url)) {+ const { values: options } = parseArgs({+ options: {+ claude: { type: 'string', default: process.env.CLAUDE_EXECUTABLE ?? 'claude' },+ model: { type: 'string', default: 'sonnet' },+ tasks: { type: 'string', default: 'crate,bolt,window,table,fence,bracket' },+ runs: { type: 'string', default: '1' },+ budget: { type: 'string', default: '3' },+ timeout: { type: 'string', default: '1200' },+ label: { type: 'string', default: 'threejs' },+ },+ });+ const npmExecutable = process.platform === 'win32' ? 'npm.cmd' : 'npm';+ const runName = `${new Date().toISOString().slice(0, 16).replace(/[:T]/g, '-')}-${options.model}-${options.label}`;+ const rawDirectory = join(repository, 'artifacts', 'agent-modeling', runName);+ const resultsDirectory = join(repository, 'bench', 'agent-modeling', 'results');+ await mkdir(rawDirectory, { recursive: true });+ await mkdir(resultsDirectory, { recursive: true });++ // One install of three from npm, copied into each attempt's fresh folder.+ const template = await mkdtemp(join(tmpdir(), '67js-threejs-template-'));+ await writeFile(join(template, 'package.json'), JSON.stringify({ name: 'threejs-task', private: true, type: 'module' }));+ execFileSync(npmExecutable, ['install', 'three', '--ignore-scripts', '--no-audit', '--no-fund'], { cwd: template, stdio: 'ignore', shell: process.platform === 'win32' });+ const threeVersion = JSON.parse(await readFile(join(template, 'node_modules', 'three', 'package.json'), 'utf8')).version;++ const runAgent = (prompt, workingDirectory) => new Promise(resolvePromise => {+ const started = performance.now();+ const child = spawn(options.claude, [+ '-p', prompt, '--model', options.model, '--output-format', 'stream-json', '--verbose', '--strict-mcp-config',+ '--allowedTools', 'Bash', 'Read', 'Write', 'Edit', 'Glob', 'Grep',+ '--disallowedTools', ...BUILT_IN_TOOLS_DENIED,+ '--permission-mode', 'dontAsk', '--setting-sources', 'project', '--no-session-persistence', '--max-budget-usd', options.budget,+ ], { cwd: workingDirectory, windowsHide: true, stdio: ['ignore', 'pipe', 'pipe'] });+ let output = '';+ let errors = '';+ child.stdout.on('data', chunk => { output += chunk; });+ child.stderr.on('data', chunk => { errors += chunk; });+ const timer = setTimeout(() => child.kill(), Number(options.timeout) * 1000);+ child.on('close', code => {+ clearTimeout(timer);+ resolvePromise({ output, errors, code, wallMs: Math.round(performance.now() - started) });+ });+ });++ const records = [];+ const summarize = () => ({+ run: runName,+ system: 'three.js by code',+ threeVersion,+ model: records.find(record => record.model)?.model ?? options.model,+ date: new Date().toISOString(),+ runsPerTask: Number(options.runs),+ passed: records.filter(record => record.pass).length,+ tasks: records.length,+ toolCalls: records.reduce((sum, record) => sum + record.toolCalls, 0),+ byModel: totalsByModel(records),+ records,+ });+ try {+ for (let attempt = 1; attempt <= Number(options.runs); attempt++) {+ for (const task of options.tasks.split(',').map(taskById)) {+ const workingDirectory = await mkdtemp(join(tmpdir(), '67js-threejs-task-'));+ try {+ await cp(template, workingDirectory, { recursive: true });+ const run = await runAgent(`${THREEJS_PREAMBLE}\n\n${task.prompt.replace(/\n/g, ' ')}`, workingDirectory);+ const transcript = readTranscript(run.output);+ await writeFile(join(rawDirectory, `${task.id}-${attempt}.jsonl`), run.output);+ let geometries = null;+ let grade = { pass: false, passed: 0, total: task.checks.length, results: [{ kind: 'geometry', pass: false, detail: 'no geometry.json' }] };+ try {+ geometries = JSON.parse(await readFile(join(workingDirectory, 'geometry.json'), 'utf8'));+ grade = gradeGeometries(task, geometries);+ } catch (error) {+ grade.results[0].detail = `geometry.json unreadable: ${error.message.split('\n')[0]}`;+ }+ const png = await checkPng(join(workingDirectory, 'scene.png'));+ if (geometries) {+ await cp(join(workingDirectory, 'scene.png'), join(rawDirectory, `${task.id}-${attempt}.png`)).catch(() => {});+ await cp(join(workingDirectory, 'build.mjs'), join(rawDirectory, `${task.id}-${attempt}.build.mjs`)).catch(() => {});+ }+ let silhouette = null;+ if (geometries) silhouette = await scoreSilhouetteOfGeometries(task.id, geometries).catch(error => ({ error: error.message.split('\n')[0] }));+ const usage = transcript.result?.usage ?? {};+ const failed = grade.results.filter(result => !result.pass).map(result => `${result.label ?? result.kind}: ${result.detail}`);+ if (!png.pass) failed.push(`scene.png: ${png.detail}`);+ const record = {+ task: task.id,+ attempt,+ pass: grade.pass && png.pass,+ passed: grade.passed,+ total: grade.total,+ pngPass: png.pass,+ failed: failed.map(removeLocalPaths),+ model: transcript.model,+ toolCalls: transcript.toolCalls,+ toolErrors: transcript.toolErrors,+ toolsByName: transcript.toolsByName,+ look: silhouette ? { silhouette } : null,+ turns: transcript.result?.num_turns ?? null,+ costUsd: null,+ inputTokens: (usage.input_tokens ?? 0) + (usage.cache_read_input_tokens ?? 0) + (usage.cache_creation_input_tokens ?? 0),+ outputTokens: usage.output_tokens ?? 0,+ wallSeconds: Math.round(run.wallMs / 1000),+ exitCode: run.code,+ stopped: transcript.result?.subtype ?? 'no result',+ filesWritten: (await readdir(workingDirectory)).filter(name => !['node_modules', 'package-lock.json'].includes(name)),+ };+ records.push(record);+ console.log(`${record.task.padEnd(10)} #${attempt} ${record.pass ? 'PASS' : 'FAIL'} ${record.passed}/${record.total} png ${png.pass} calls ${record.toolCalls} (errors ${record.toolErrors}) look ${silhouette?.mean ?? '-'} ${record.wallSeconds}s${failed.length ? ` ${failed.join(' | ')}` : ''}`);+ await writeFile(join(resultsDirectory, `${runName}.json`), `${JSON.stringify(summarize(), null, 2)}\n`);+ } finally {+ await removeFolder(workingDirectory);+ }+ }+ }+ } finally {+ await removeFolder(template);+ }+ console.log(`${records.filter(record => record.pass).length}/${records.length} passed, ${totalsOf(records).toolCalls} tool calls. Summary: bench/agent-modeling/results/${runName}.json`);+}diff --git a/changes/31.md b/changes/31.mdnew file mode 100644index 00000000..5e9be688--- /dev/null+++ b/changes/31.md@@ -0,0 +1 @@+- Compare agent friendliness of 67js MCP against three.js written as code on six suite tasks with the same model and the same checks and look score (issue #31, three.js half; the Blender MCP half still needs a VM): `bench/agent-modeling/threejs.mjs` runs the three.js side and `results/2026-11-07-threejs-vs-67js.json` holds both.diff --git a/tests/agent-threejs-grade.test.js b/tests/agent-threejs-grade.test.jsnew file mode 100644index 00000000..4de9dd8f--- /dev/null+++ b/tests/agent-threejs-grade.test.js@@ -0,0 +1,25 @@+import assert from 'node:assert/strict';+import test from 'node:test';+import { worldGeometriesOf } from '../bench/agent-modeling/look.mjs';+import { REFERENCES } from '../bench/agent-modeling/references.mjs';+import { gradeGeometries, scoreSilhouetteOfGeometries } from '../bench/agent-modeling/threejs.mjs';+import { taskById } from '../bench/agent-modeling/tasks.mjs';++const asJson = geometries => geometries.map(({ positions, indices }) => ({ positions: Array.from(positions), indices: Array.from(indices) }));++test('the three.js grader passes the reference geometry of each sampled task and scores its look as 1', async () => {+ for (const id of ['crate', 'bolt', 'window', 'table', 'fence', 'bracket']) {+ const geometries = asJson(await worldGeometriesOf(REFERENCES[id]));+ const grade = gradeGeometries(taskById(id), geometries);+ assert.equal(grade.pass, true, `${id}: ${JSON.stringify(grade.results.filter(result => !result.pass))}`);+ assert.equal((await scoreSilhouetteOfGeometries(id, geometries)).mean, 1);+ }+});++test('the three.js grader fails an open surface of the wrong size', () => {+ const crate = taskById('crate');+ const quad = { positions: [0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0], indices: [0, 1, 2, 0, 2, 3] };+ const grade = gradeGeometries(crate, [quad]);+ assert.equal(grade.pass, false);+ assert.equal(grade.results.find(result => result.kind === 'lint').pass, false);+});