Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Compare 67js MCP with three.js by code on six suite tasks (part of #31)

Nguyen Xuan An91ab410faafb
diff --git a/bench/agent-modeling/README.md b/bench/agent-modeling/README.md
index ee760092..a7cf83a7 100644
--- a/bench/agent-modeling/README.md
+++ b/bench/agent-modeling/README.md
@@ -394,3 +394,37 @@ Not run:
### Antigravity CLI (2026-11-07)
`llms-only.mjs --agent agy --agy <agy executable> --model <model>` drives the Antigravity CLI, which offers Gemini 4 Argon, Gemini 3.8 Flash and GPT-OSS 120B. A first run of 2 attempts per model passed 0 of 6, but not on the docs: headless mode denies every shell command it cannot prompt for, so the agents read `llms.txt` and the package, wrote `render.mjs` (Gemini 4 Argon), and were refused `node render.mjs`. Those results are not kept. A valid run needs a session-only allow rule for commands, passed with `--settings`, which An has to approve; the runner records any web tool call as `usedWeb` and fails it, since the CLI cannot switch its web tools off.
+
+## Agent friendliness: 67js MCP against three.js by code (2026-11-07, issue #31)
+
+The three.js half of issue #31 and the 67js side of the same table. The Blender MCP half needs a VM and is not run. Model `sonnet`, headless `claude -p` on the subscription, 6 suite tasks (`crate`, `bolt`, `window`, `table`, `fence`, `bracket`), 1 run each per system, so 12 records in `results/2026-11-07-threejs-vs-67js.json`. Tokens and tool calls are reported; there is no API dollar figure.
+
+- 67js MCP: `run.mjs`, the task prompt behind the usual MCP preamble.
+- three.js: `threejs.mjs`, the same task text behind a different preamble. The agent gets an empty folder with `three` installed from npm, Bash, Read, Write, Edit, Glob and Grep, no web and no MCP. It writes `build.mjs`, which builds the object, writes `geometry.json` (world-space triangles per mesh) and renders `scene.png` with headless Chrome on SwiftShader.
+- Same grading: the task's own checks run on `geometry.json` (corners welded before the closed-surface, hole and loop counts; the grader passes the suite's reference geometry of all six tasks, see `tests/agent-threejs-grade.test.js`), and the look score is the suite's silhouette overlap from the three fixed cameras. A three.js run passes when every check passes and `scene.png` is a real picture.
+
+| System | Pass | Mean tool calls | Mean tokens (in+out) | Mean look | Mean wall seconds |
+| --- | --- | --- | --- | --- | --- |
+| 67js MCP | 6/6 | 3.2 | 375k | 0.932 | 24 |
+| three.js by code | 5/6 | 8.8 | 269k | 0.926 | 359 |
+
+| Task | System | Pass | Tool calls (errors) | Look | Failure |
+| --- | --- | --- | --- | --- | --- |
+| crate | 67js MCP | yes | 4 (0) | 1.000 | |
+| crate | three.js | yes | 9 (1) | 1.000 | |
+| bolt | 67js MCP | yes | 2 (0) | 1.000 | |
+| bolt | three.js | yes | 7 (3) | 0.964 | |
+| window | 67js MCP | yes | 4 (0) | 1.000 | |
+| window | three.js | no, 4/5 | 11 (0) | 1.000 | closed-surface check: the frame has open or doubled edges |
+| table | 67js MCP | yes | 2 (0) | 0.592 | |
+| table | three.js | yes | 8 (1) | 0.592 | |
+| fence | 67js MCP | yes | 2 (0) | 1.000 | |
+| fence | three.js | yes | 11 (1) | 1.000 | |
+| bracket | 67js MCP | yes | 5 (0) | 1.000 | |
+| bracket | three.js | yes | 7 (0) | 1.000 | |
+
+- Measured: 67js needed about a third of the tool calls and finished in about 24 s against about 6 minutes, most of the three.js time going to getting headless Chrome to write a PNG. Both systems built the same shapes: the look means are within noise, and `table` scores 0.592 for both, which points at the task's reference rather than at either system.
+- Tokens are not lower for 67js: inferred: the MCP tool schemas and server instructions are re-read on every call, so its input tokens are higher even with fewer calls. The three.js `table` run reports only 44k input tokens although it ran 871 s, so treat that one count as unreliable.
+- 67js failures: none in the checks. One harness fault: the `bracket` look images failed because the look page could not fetch the boolean wasm (`both async and sync fetching of the wasm failed`), which threw away the whole look record, silhouette included. The silhouette was scored again afterwards from the exported recipe, and the record carries `imagesError`. `scoreLook` should score the silhouette before it renders images.
+- A first attempt of the 67js run failed on every task because the checkout had no `build/mcp` folder (`npm run build:lib` creates it); those results were discarded. The first three.js run stopped after `bolt` when its process disappeared, so `window`, `table`, `fence` and `bracket` were run again as a second run and joined with `compare-threejs.mjs`.
+- Limits: 1 run per cell, one model, tasks the 67js suite was built around (they favour 67js's checks, not three.js). The `window` three.js failure label came from an earlier wording of the same check.
diff --git a/bench/agent-modeling/check.mjs b/bench/agent-modeling/check.mjs
index c59736a7..b8e0e5cd 100644
--- a/bench/agent-modeling/check.mjs
+++ b/bench/agent-modeling/check.mjs
@@ -201,7 +201,7 @@ function checkLitPainting(check, measures) {
return { pass: problems.length === 0, detail: problems.join('; ') || `place ${placeIndex + 1}, ${shadowed} of ${total} points in shadow` };
}
-function checkOne(check, measures) {
+export function checkOne(check, measures) {
switch (check.kind) {
case 'lint':
return { pass: measures.lintProblems === 0 && measures.closedParts === measures.parts, detail: measures.lintProblems ? measures.problemKinds.join(', ') : `${measures.closedParts}/${measures.parts} parts closed` };
diff --git a/bench/agent-modeling/compare-threejs.mjs b/bench/agent-modeling/compare-threejs.mjs
new file mode 100644
index 00000000..6a701b31
--- /dev/null
+++ b/bench/agent-modeling/compare-threejs.mjs
@@ -0,0 +1,60 @@
+// Joins one 67js run (run.mjs) and one three.js run (threejs.mjs) into the comparison file of issue #31 and prints
+// the markdown tables.
+// node bench/agent-modeling/compare-threejs.mjs <67js run json> <three.js run json> <output json>
+import { readFile, writeFile } from 'node:fs/promises';
+
+const [mcpPath, codePath, outputPath] = process.argv.slice(2);
+const mcpRun = JSON.parse(await readFile(mcpPath, 'utf8'));
+// The three.js run may be split over several files (comma separated) when a run was stopped and the rest redone.
+const codeRuns = await Promise.all(codePath.split(',').map(async path => JSON.parse(await readFile(path, 'utf8'))));
+const codeRun = { ...codeRuns[0], records: codeRuns.flatMap(run => run.records) };
+
+const SYSTEMS = [
+ { name: '67js MCP', run: mcpRun },
+ { name: 'three.js by code', run: codeRun },
+];
+const records = SYSTEMS.flatMap(({ name, run }) => run.records.map(record => ({ system: name, ...record })));
+
+const mean = values => values.reduce((sum, value) => sum + value, 0) / values.length;
+const lookOf = record => record.look?.silhouette?.mean ?? null;
+const tokensOf = record => record.inputTokens + record.outputTokens;
+const thousands = value => `${Math.round(value / 1000)}k`;
+
+const taskIds = [...new Set(records.map(record => record.task))];
+const lines = ['| Task | System | Pass | Tool calls | Tool errors | Tokens in / out | Look | Failures |', '| --- | --- | --- | --- | --- | --- | --- | --- |'];
+for (const task of taskIds) {
+ for (const { name } of SYSTEMS) {
+ const record = records.find(candidate => candidate.task === task && candidate.system === name);
+ lines.push(`| ${task} | ${name} | ${record.pass ? 'yes' : 'no'} (${record.passed}/${record.total}) | ${record.toolCalls} | ${record.toolErrors} | ${thousands(record.inputTokens)} / ${thousands(record.outputTokens)} | ${lookOf(record)?.toFixed(3) ?? '-'} | ${record.failed.join('; ') || '-'} |`);
+ }
+}
+const totals = ['| System | Pass | Mean tool calls | Mean tokens | Mean look | Mean wall seconds |', '| --- | --- | --- | --- | --- | --- |'];
+const summary = {};
+for (const { name } of SYSTEMS) {
+ const own = records.filter(record => record.system === name);
+ const looks = own.map(lookOf).filter(value => value !== null);
+ summary[name] = {
+ runs: own.length,
+ passed: own.filter(record => record.pass).length,
+ meanToolCalls: Number(mean(own.map(record => record.toolCalls)).toFixed(1)),
+ meanTokens: Math.round(mean(own.map(tokensOf))),
+ meanLook: Number(mean(looks).toFixed(3)),
+ meanWallSeconds: Math.round(mean(own.map(record => record.wallSeconds))),
+ };
+ const row = summary[name];
+ totals.push(`| ${name} | ${row.passed}/${row.runs} | ${row.meanToolCalls} | ${thousands(row.meanTokens)} | ${row.meanLook} | ${row.meanWallSeconds} |`);
+}
+
+const output = {
+ run: '2026-11-07-threejs-vs-67js',
+ issue: 31,
+ model: mcpRun.model,
+ date: new Date().toISOString(),
+ scope: 'three.js half and 67js MCP side; the official Blender MCP half needs a VM and is not run',
+ threeVersion: codeRun.threeVersion,
+ runsPerTaskPerSystem: 1,
+ summary,
+ records,
+};
+await writeFile(outputPath, `${JSON.stringify(output, null, 2)}\n`);
+console.log(`${totals.join('\n')}\n\n${lines.join('\n')}`);
diff --git a/bench/agent-modeling/measure.mjs b/bench/agent-modeling/measure.mjs
index 7f503841..c06b872a 100644
--- a/bench/agent-modeling/measure.mjs
+++ b/bench/agent-modeling/measure.mjs
@@ -59,7 +59,7 @@ const triangleArea = (positions, first, second, third) => {
return Math.hypot(ay * bz - az * by, az * bx - ax * bz, ax * by - ay * bx) / 2;
};
-function signedVolume({ positions, indices }) {
+export function signedVolume({ positions, indices }) {
let volume = 0;
for (let triangle = 0; triangle < indices.length; triangle += 3) {
const [first, second, third] = [indices[triangle], indices[triangle + 1], indices[triangle + 2]];
@@ -136,7 +136,7 @@ export function sectionLoops(geometries, axis, at) {
* Count separate outer tips around the thinnest axis (gear teeth, star points). Vertices near the outer radius are
* joined when a triangle edge runs between them along the rim, so a wide flat tooth tip still counts once.
*/
-function countTips(geometries, bounds) {
+export function countTips(geometries, bounds) {
const extents = [0, 1, 2].map(axis => bounds.max[axis] - bounds.min[axis]);
const thinAxis = extents.indexOf(Math.min(...extents));
const [firstAxis, secondAxis] = [0, 1, 2].filter(axis => axis !== thinAxis);
@@ -199,7 +199,7 @@ function surfaceHeightAt({ positions, indices }, pointX, pointZ) {
return null;
}
-function lineHits(geometries, axis, point) {
+export function lineHits(geometries, axis, point) {
const axisIndex = { x: 0, y: 1, z: 2 }[axis];
const [firstAxis, secondAxis] = [0, 1, 2].filter(candidate => candidate !== axisIndex);
let hits = 0;
diff --git a/bench/agent-modeling/results/2026-11-07-threejs-vs-67js.json b/bench/agent-modeling/results/2026-11-07-threejs-vs-67js.json
new file mode 100644
index 00000000..a73f4400
--- /dev/null
+++ b/bench/agent-modeling/results/2026-11-07-threejs-vs-67js.json
@@ -0,0 +1,580 @@
+{
+ "run": "2026-11-07-threejs-vs-67js",
+ "issue": 31,
+ "model": "claude-sonnet-5-5",
+ "date": "2026-11-07T16:28:52.048Z",
+ "scope": "three.js half and 67js MCP side; the official Blender MCP half needs a VM and is not run",
+ "threeVersion": "0.187.0",
+ "runsPerTaskPerSystem": 1,
+ "summary": {
+ "67js MCP": {
+ "runs": 6,
+ "passed": 6,
+ "meanToolCalls": 3.2,
+ "meanTokens": 374760,
+ "meanLook": 0.932,
+ "meanWallSeconds": 24
+ },
+ "three.js by code": {
+ "runs": 6,
+ "passed": 5,
+ "meanToolCalls": 8.8,
+ "meanTokens": 268655,
+ "meanLook": 0.926,
+ "meanWallSeconds": 359
+ }
+ },
+ "records": [
+ {
+ "system": "67js MCP",
+ "task": "crate",
+ "attempt": 1,
+ "pass": true,
+ "passed": 8,
+ "total": 8,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 4,
+ "toolErrors": 0,
+ "toolsByName": {
+ "ToolSearch": 1,
+ "scene_create": 1,
+ "scene_look": 1,
+ "mesh_edit": 1,
+ "scene_export_recipe": 1
+ },
+ "branchCalls": 0,
+ "relationCalls": 0,
+ "pathQueries": 0,
+ "features": {
+ "geometryTypes": [
+ "mesh",
+ "box"
+ ],
+ "meshOperations": [
+ "inset_individual"
+ ],
+ "motionIk": [],
+ "motionTracks": 0
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ },
+ "images": [
+ "crate-front.png",
+ "crate-side.png",
+ "crate-three-quarter.png"
+ ]
+ },
+ "turns": 6,
+ "inputTokens": 445726,
+ "outputTokens": 1751,
+ "wallSeconds": 27,
+ "exitCode": 0,
+ "stopped": "success"
+ },
+ {
+ "system": "67js MCP",
+ "task": "bolt",
+ "attempt": 1,
+ "pass": true,
+ "passed": 7,
+ "total": 7,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 2,
+ "toolErrors": 0,
+ "toolsByName": {
+ "ToolSearch": 1,
+ "scene_create": 1,
+ "scene_export_recipe": 1
+ },
+ "branchCalls": 0,
+ "relationCalls": 0,
+ "pathQueries": 0,
+ "features": {
+ "geometryTypes": [
+ "cylinder"
+ ],
+ "meshOperations": [],
+ "motionIk": [],
+ "motionTracks": 0
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ },
+ "images": [
+ "bolt-front.png",
+ "bolt-side.png",
+ "bolt-three-quarter.png"
+ ]
+ },
+ "turns": 4,
+ "inputTokens": 232980,
+ "outputTokens": 1112,
+ "wallSeconds": 21,
+ "exitCode": 0,
+ "stopped": "success"
+ },
+ {
+ "system": "67js MCP",
+ "task": "window",
+ "attempt": 1,
+ "pass": true,
+ "passed": 5,
+ "total": 5,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 4,
+ "toolErrors": 0,
+ "toolsByName": {
+ "ToolSearch": 2,
+ "scene_create": 1,
+ "scene_inspect": 1,
+ "mesh_inspect": 1,
+ "scene_export_recipe": 1
+ },
+ "branchCalls": 0,
+ "relationCalls": 0,
+ "pathQueries": 0,
+ "features": {
+ "geometryTypes": [
+ "extrude"
+ ],
+ "meshOperations": [],
+ "motionIk": [],
+ "motionTracks": 0
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ },
+ "images": [
+ "window-front.png",
+ "window-side.png",
+ "window-three-quarter.png"
+ ]
+ },
+ "turns": 7,
+ "inputTokens": 546049,
+ "outputTokens": 1464,
+ "wallSeconds": 26,
+ "exitCode": 0,
+ "stopped": "success"
+ },
+ {
+ "system": "67js MCP",
+ "task": "table",
+ "attempt": 1,
+ "pass": true,
+ "passed": 6,
+ "total": 6,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 2,
+ "toolErrors": 0,
+ "toolsByName": {
+ "ToolSearch": 1,
+ "scene_create": 1,
+ "scene_export_recipe": 1
+ },
+ "branchCalls": 0,
+ "relationCalls": 0,
+ "pathQueries": 0,
+ "features": {
+ "geometryTypes": [
+ "box"
+ ],
+ "meshOperations": [],
+ "motionIk": [],
+ "motionTracks": 0
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 0.1684,
+ "three-quarter": 0.6073
+ },
+ "mean": 0.5919
+ },
+ "images": [
+ "table-front.png",
+ "table-side.png",
+ "table-three-quarter.png"
+ ]
+ },
+ "turns": 4,
+ "inputTokens": 232283,
+ "outputTokens": 1109,
+ "wallSeconds": 18,
+ "exitCode": 0,
+ "stopped": "success"
+ },
+ {
+ "system": "67js MCP",
+ "task": "fence",
+ "attempt": 1,
+ "pass": true,
+ "passed": 6,
+ "total": 6,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 2,
+ "toolErrors": 0,
+ "toolsByName": {
+ "ToolSearch": 1,
+ "scene_create": 1,
+ "scene_export_recipe": 1
+ },
+ "branchCalls": 0,
+ "relationCalls": 0,
+ "pathQueries": 0,
+ "features": {
+ "geometryTypes": [
+ "box"
+ ],
+ "meshOperations": [],
+ "motionIk": [],
+ "motionTracks": 0
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ },
+ "images": [
+ "fence-front.png",
+ "fence-side.png",
+ "fence-three-quarter.png"
+ ]
+ },
+ "turns": 4,
+ "inputTokens": 234011,
+ "outputTokens": 1253,
+ "wallSeconds": 20,
+ "exitCode": 0,
+ "stopped": "success"
+ },
+ {
+ "system": "67js MCP",
+ "task": "bracket",
+ "attempt": 1,
+ "pass": true,
+ "passed": 6,
+ "total": 6,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 5,
+ "toolErrors": 0,
+ "toolsByName": {
+ "ToolSearch": 1,
+ "scene_create": 1,
+ "mesh_inspect": 1,
+ "mesh_edit": 1,
+ "scene_apply": 1,
+ "scene_export_recipe": 1
+ },
+ "branchCalls": 0,
+ "relationCalls": 0,
+ "pathQueries": 0,
+ "features": {
+ "geometryTypes": [
+ "mesh"
+ ],
+ "meshOperations": [
+ "boolean"
+ ],
+ "motionIk": [],
+ "motionTracks": 0
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ },
+ "imagesError": "the look harness page could not fetch the boolean wasm, so no preview images; the silhouette is scored from the exported recipe afterwards"
+ },
+ "turns": 7,
+ "inputTokens": 548871,
+ "outputTokens": 1949,
+ "wallSeconds": 30,
+ "exitCode": 0,
+ "stopped": "success"
+ },
+ {
+ "system": "three.js by code",
+ "task": "crate",
+ "attempt": 1,
+ "pass": true,
+ "passed": 8,
+ "total": 8,
+ "pngPass": true,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 9,
+ "toolErrors": 1,
+ "toolsByName": {
+ "Bash": 4,
+ "TaskStop": 2,
+ "Edit": 2,
+ "Read": 1
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ }
+ },
+ "turns": 10,
+ "inputTokens": 317368,
+ "outputTokens": 6905,
+ "wallSeconds": 392,
+ "exitCode": 0,
+ "stopped": "success",
+ "filesWritten": [
+ ".chrome-profile",
+ "build.mjs",
+ "geometry.json",
+ "package.json",
+ "scene.png"
+ ]
+ },
+ {
+ "system": "three.js by code",
+ "task": "bolt",
+ "attempt": 1,
+ "pass": true,
+ "passed": 7,
+ "total": 7,
+ "pngPass": true,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 7,
+ "toolErrors": 3,
+ "toolsByName": {
+ "Bash": 6,
+ "Read": 1
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 0.9621,
+ "side": 0.9621,
+ "three-quarter": 0.9666
+ },
+ "mean": 0.9636
+ }
+ },
+ "turns": 8,
+ "inputTokens": 308935,
+ "outputTokens": 5416,
+ "wallSeconds": 182,
+ "exitCode": 0,
+ "stopped": "success",
+ "filesWritten": [
+ "build.mjs",
+ "chrome-profile",
+ "geometry.json",
+ "package.json",
+ "render.html",
+ "scene.png"
+ ]
+ },
+ {
+ "system": "three.js by code",
+ "task": "window",
+ "attempt": 1,
+ "pass": false,
+ "passed": 4,
+ "total": 5,
+ "pngPass": true,
+ "failed": [
+ "lint: open edges"
+ ],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 11,
+ "toolErrors": 0,
+ "toolsByName": {
+ "Bash": 5,
+ "TaskStop": 2,
+ "Edit": 3,
+ "Read": 1
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ }
+ },
+ "turns": 12,
+ "inputTokens": 330002,
+ "outputTokens": 4616,
+ "wallSeconds": 302,
+ "exitCode": 0,
+ "stopped": "success",
+ "filesWritten": [
+ "build.mjs",
+ "geometry.json",
+ "package.json",
+ "render.html",
+ "scene.png"
+ ]
+ },
+ {
+ "system": "three.js by code",
+ "task": "table",
+ "attempt": 1,
+ "pass": true,
+ "passed": 6,
+ "total": 6,
+ "pngPass": true,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 8,
+ "toolErrors": 1,
+ "toolsByName": {
+ "Bash": 7,
+ "Read": 1
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 0.1684,
+ "three-quarter": 0.6073
+ },
+ "mean": 0.5919
+ }
+ },
+ "turns": 1,
+ "inputTokens": 44307,
+ "outputTokens": 84,
+ "wallSeconds": 871,
+ "exitCode": 0,
+ "stopped": "success",
+ "filesWritten": [
+ "build.mjs",
+ "geometry.json",
+ "package.json",
+ "scene.png"
+ ]
+ },
+ {
+ "system": "three.js by code",
+ "task": "fence",
+ "attempt": 1,
+ "pass": true,
+ "passed": 6,
+ "total": 6,
+ "pngPass": true,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 11,
+ "toolErrors": 1,
+ "toolsByName": {
+ "Bash": 5,
+ "TaskStop": 2,
+ "PowerShell": 1,
+ "Edit": 2,
+ "Read": 1
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ }
+ },
+ "turns": 13,
+ "inputTokens": 349860,
+ "outputTokens": 3328,
+ "wallSeconds": 292,
+ "exitCode": 0,
+ "stopped": "success",
+ "filesWritten": [
+ "build.mjs",
+ "geometry.json",
+ "package.json",
+ "scene.png"
+ ]
+ },
+ {
+ "system": "three.js by code",
+ "task": "bracket",
+ "attempt": 1,
+ "pass": true,
+ "passed": 6,
+ "total": 6,
+ "pngPass": true,
+ "failed": [],
+ "model": "claude-sonnet-5-5",
+ "toolCalls": 7,
+ "toolErrors": 0,
+ "toolsByName": {
+ "Bash": 4,
+ "Write": 1,
+ "Read": 2
+ },
+ "look": {
+ "silhouette": {
+ "views": {
+ "front": 1,
+ "side": 1,
+ "three-quarter": 1
+ },
+ "mean": 1
+ }
+ },
+ "turns": 8,
+ "inputTokens": 234719,
+ "outputTokens": 6392,
+ "wallSeconds": 114,
+ "exitCode": 0,
+ "stopped": "success",
+ "filesWritten": [
+ "build.mjs",
+ "geometry.json",
+ "package.json",
+ "render.html",
+ "scene.png"
+ ]
+ }
+ ]
+}
diff --git a/bench/agent-modeling/threejs.mjs b/bench/agent-modeling/threejs.mjs
new file mode 100644
index 00000000..83fed2c3
--- /dev/null
+++ b/bench/agent-modeling/threejs.mjs
@@ -0,0 +1,292 @@
+// The three.js side of the agent friendliness comparison (issue #31): the same task prompts and model as run.mjs, but the
+// agent has no MCP. It gets an empty folder with `three` installed from npm and writes code.
+//
+// node bench/agent-modeling/threejs.mjs --claude <claude executable> [--model sonnet] [--tasks crate,bolt] [--runs 1]
+// [--budget 3] [--timeout 1200] [--label threejs]
+//
+// The agent writes build.mjs, which builds the object with three.js, writes geometry.json (the world-space triangles of
+// every mesh) and renders scene.png with headless Chrome. The same task checks and the same silhouette look score as
+// the 67js runs grade geometry.json, so the two systems are measured on the same shape. A run passes when every check
+// passes and scene.png is a real picture.
+//
+// Raw transcripts go to artifacts/agent-modeling/<run>/ (ignored by Git); the summary goes to
+// bench/agent-modeling/results/<run>.json.
+import { execFileSync, spawn } from 'node:child_process';
+import { mkdir, mkdtemp, readFile, readdir, rm, writeFile, cp } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { parseArgs } from 'node:util';
+import { removeLocalPaths, totalsByModel, totalsOf } from './aggregate.mjs';
+import { checkOne } from './check.mjs';
+import { camerasForTask, worldGeometriesOf } from './look.mjs';
+import { countTips, lineHits, sectionLoops, signedVolume } from './measure.mjs';
+import { checkPng } from './png-check.mjs';
+import { REFERENCES } from './references.mjs';
+import { intersectionOverUnion, rasterizeSilhouette } from './silhouette.mjs';
+import { taskById } from './tasks.mjs';
+
+const repository = resolve(import.meta.dirname, '../..');
+const BUILT_IN_TOOLS_DENIED = ['WebFetch', 'WebSearch', 'Agent', 'Task', 'NotebookEdit'];
+const WELD_SCALE = 1e5;
+
+export const THREEJS_PREAMBLE = `You are modeling with three.js code. The folder you are in has three installed from npm (node_modules/three). Build the
+object described below as a closed, watertight solid, in metres with Y up. Rest the object on the ground (lowest point at
+Y = 0), centred on X = 0 and Z = 0.
+Write a Node.js script named build.mjs that does three things and run it:
+1. Builds the object with three.js.
+2. Writes geometry.json: a JSON array with one entry { "positions": [...], "indices": [...] } per mesh, with every vertex in
+ world space (apply each mesh's world matrix) and indices as a flat triangle list.
+3. Renders the scene to scene.png, 512 by 512 pixels, with headless Google Chrome (installed; use --use-angle=swiftshader
+ --enable-unsafe-swiftshader --enable-webgl --ignore-gpu-blocklist so it needs no GPU). There is no internet access.
+The files build.mjs, geometry.json and scene.png are what gets graded. When they exist, reply with one line saying so.`;
+
+/** Measures an array of { positions, indices } meshes like measure.mjs does for a recipe, welding shared corners first. */
+export function measureGeometries(rawGeometries) {
+ const geometries = rawGeometries.map(({ positions, indices }) => ({
+ positions: Float32Array.from(positions),
+ indices: Uint32Array.from(indices ?? Array.from({ length: positions.length / 3 }, (_, index) => index)),
+ }));
+ const bounds = { min: [Infinity, Infinity, Infinity], max: [-Infinity, -Infinity, -Infinity] };
+ let signed = 0;
+ let triangles = 0;
+ let vertexCount = 0;
+ let edgeCount = 0;
+ let faceCount = 0;
+ let components = 0;
+ let closedComponents = 0;
+ for (const geometry of geometries) {
+ const weldedIndex = new Map();
+ const welded = [];
+ for (let vertex = 0; vertex < geometry.positions.length / 3; vertex++) {
+ const point = [0, 1, 2].map(axis => geometry.positions[vertex * 3 + axis]);
+ for (let axis = 0; axis < 3; axis++) {
+ bounds.min[axis] = Math.min(bounds.min[axis], point[axis]);
+ bounds.max[axis] = Math.max(bounds.max[axis], point[axis]);
+ }
+ const key = point.map(value => Math.round(value * WELD_SCALE)).join(',');
+ if (!weldedIndex.has(key)) weldedIndex.set(key, weldedIndex.size);
+ welded.push(weldedIndex.get(key));
+ }
+ signed += signedVolume(geometry);
+ triangles += geometry.indices.length / 3;
+ const edgeUses = new Map();
+ const parent = Array.from({ length: weldedIndex.size }, (_, index) => index);
+ const find = vertex => (parent[vertex] === vertex ? vertex : (parent[vertex] = find(parent[vertex])));
+ let faces = 0;
+ for (let triangle = 0; triangle < geometry.indices.length; triangle += 3) {
+ const corners = [0, 1, 2].map(corner => welded[geometry.indices[triangle + corner]]);
+ if (new Set(corners).size < 3) continue;
+ faces++;
+ for (let corner = 0; corner < 3; corner++) {
+ const [from, to] = [corners[corner], corners[(corner + 1) % 3]];
+ const key = from < to ? `${from}-${to}` : `${to}-${from}`;
+ edgeUses.set(key, (edgeUses.get(key) ?? 0) + 1);
+ parent[find(from)] = find(to);
+ }
+ }
+ const roots = new Set(Array.from({ length: weldedIndex.size }, (_, vertex) => find(vertex)));
+ components += roots.size;
+ const closed = [...edgeUses.values()].every(uses => uses === 2);
+ if (closed) closedComponents += roots.size;
+ vertexCount += weldedIndex.size;
+ edgeCount += edgeUses.size;
+ faceCount += faces;
+ }
+ const allClosed = closedComponents === components;
+ const volume = Math.abs(signed);
+ return {
+ parts: components,
+ closedParts: closedComponents,
+ lintProblems: allClosed ? 0 : components - closedComponents,
+ problemKinds: allClosed ? [] : ['open or non-manifold edges'],
+ holes: allClosed ? (2 * components - (vertexCount - edgeCount + faceCount)) / 2 : null,
+ volume,
+ triangles,
+ bounds,
+ size: [0, 1, 2].map(axis => bounds.max[axis] - bounds.min[axis]),
+ tips: countTips(geometries, bounds),
+ section: (axis, at) => sectionLoops(geometries, axis, at),
+ lineHits: (axis, point) => lineHits(geometries, axis, point),
+ };
+}
+
+/** Same shape as gradeRecipe: { pass, passed, total, results }. Tasks with checks this module cannot measure fail loudly. */
+export function gradeGeometries(task, rawGeometries) {
+ const unsupported = task.checks.filter(check => !['lint', 'parts', 'size', 'extent', 'ground', 'holes', 'volume', 'section', 'through', 'tips', 'triangles'].includes(check.kind));
+ if (unsupported.length) throw new RangeError(`threejs.mjs cannot grade ${unsupported.map(check => check.kind).join(', ')}`);
+ const measures = measureGeometries(rawGeometries);
+ const results = task.checks.map(check => ({ kind: check.kind, ...(check.label ? { label: check.label } : {}), ...checkOne(check, measures) }));
+ const passed = results.filter(result => result.pass).length;
+ return { pass: passed === results.length, passed, total: results.length, results };
+}
+
+/** The look score of the suite (silhouette overlap from the three fixed cameras) for loose geometries. */
+export async function scoreSilhouetteOfGeometries(taskId, rawGeometries) {
+ const views = await camerasForTask(taskId);
+ const reference = await worldGeometriesOf(REFERENCES[taskId]);
+ const candidate = rawGeometries.map(({ positions, indices }) => ({ positions: Float32Array.from(positions), indices: Uint32Array.from(indices) }));
+ const overlap = {};
+ for (const { name, camera } of views) {
+ overlap[name] = Number(intersectionOverUnion(rasterizeSilhouette(candidate, camera), rasterizeSilhouette(reference, camera)).toFixed(4));
+ }
+ const values = Object.values(overlap);
+ return { views: overlap, mean: Number((values.reduce((sum, value) => sum + value, 0) / values.length).toFixed(4)) };
+}
+
+function readTranscript(output) {
+ const summary = { model: null, toolCalls: 0, toolErrors: 0, toolsByName: {}, result: null };
+ for (const line of output.split('\n')) {
+ if (!line.trim().startsWith('{')) continue;
+ let event;
+ try {
+ event = JSON.parse(line);
+ } catch {
+ continue;
+ }
+ if (event.type === 'system' && event.subtype === 'init') summary.model = event.model;
+ if (event.type === 'assistant') {
+ for (const block of event.message?.content ?? []) {
+ if (block.type !== 'tool_use' || block.name === 'ToolSearch') continue;
+ summary.toolCalls++;
+ summary.toolsByName[block.name] = (summary.toolsByName[block.name] ?? 0) + 1;
+ }
+ }
+ if (event.type === 'user') {
+ for (const block of event.message?.content ?? []) {
+ if (block.type === 'tool_result' && block.is_error) summary.toolErrors++;
+ }
+ }
+ if (event.type === 'result') summary.result = event;
+ }
+ return summary;
+}
+
+async function removeFolder(folder) {
+ try {
+ await rm(folder, { recursive: true, force: true, maxRetries: 10, retryDelay: 500 });
+ } catch (error) {
+ console.warn(`Left a temporary folder behind: ${error.code}`);
+ }
+}
+
+if (process.argv[1] === fileURLToPath(import.meta.url)) {
+ const { values: options } = parseArgs({
+ options: {
+ claude: { type: 'string', default: process.env.CLAUDE_EXECUTABLE ?? 'claude' },
+ model: { type: 'string', default: 'sonnet' },
+ tasks: { type: 'string', default: 'crate,bolt,window,table,fence,bracket' },
+ runs: { type: 'string', default: '1' },
+ budget: { type: 'string', default: '3' },
+ timeout: { type: 'string', default: '1200' },
+ label: { type: 'string', default: 'threejs' },
+ },
+ });
+ const npmExecutable = process.platform === 'win32' ? 'npm.cmd' : 'npm';
+ const runName = `${new Date().toISOString().slice(0, 16).replace(/[:T]/g, '-')}-${options.model}-${options.label}`;
+ const rawDirectory = join(repository, 'artifacts', 'agent-modeling', runName);
+ const resultsDirectory = join(repository, 'bench', 'agent-modeling', 'results');
+ await mkdir(rawDirectory, { recursive: true });
+ await mkdir(resultsDirectory, { recursive: true });
+
+ // One install of three from npm, copied into each attempt's fresh folder.
+ const template = await mkdtemp(join(tmpdir(), '67js-threejs-template-'));
+ await writeFile(join(template, 'package.json'), JSON.stringify({ name: 'threejs-task', private: true, type: 'module' }));
+ execFileSync(npmExecutable, ['install', 'three', '--ignore-scripts', '--no-audit', '--no-fund'], { cwd: template, stdio: 'ignore', shell: process.platform === 'win32' });
+ const threeVersion = JSON.parse(await readFile(join(template, 'node_modules', 'three', 'package.json'), 'utf8')).version;
+
+ const runAgent = (prompt, workingDirectory) => new Promise(resolvePromise => {
+ const started = performance.now();
+ const child = spawn(options.claude, [
+ '-p', prompt, '--model', options.model, '--output-format', 'stream-json', '--verbose', '--strict-mcp-config',
+ '--allowedTools', 'Bash', 'Read', 'Write', 'Edit', 'Glob', 'Grep',
+ '--disallowedTools', ...BUILT_IN_TOOLS_DENIED,
+ '--permission-mode', 'dontAsk', '--setting-sources', 'project', '--no-session-persistence', '--max-budget-usd', options.budget,
+ ], { cwd: workingDirectory, windowsHide: true, stdio: ['ignore', 'pipe', 'pipe'] });
+ let output = '';
+ let errors = '';
+ child.stdout.on('data', chunk => { output += chunk; });
+ child.stderr.on('data', chunk => { errors += chunk; });
+ const timer = setTimeout(() => child.kill(), Number(options.timeout) * 1000);
+ child.on('close', code => {
+ clearTimeout(timer);
+ resolvePromise({ output, errors, code, wallMs: Math.round(performance.now() - started) });
+ });
+ });
+
+ const records = [];
+ const summarize = () => ({
+ run: runName,
+ system: 'three.js by code',
+ threeVersion,
+ model: records.find(record => record.model)?.model ?? options.model,
+ date: new Date().toISOString(),
+ runsPerTask: Number(options.runs),
+ passed: records.filter(record => record.pass).length,
+ tasks: records.length,
+ toolCalls: records.reduce((sum, record) => sum + record.toolCalls, 0),
+ byModel: totalsByModel(records),
+ records,
+ });
+ try {
+ for (let attempt = 1; attempt <= Number(options.runs); attempt++) {
+ for (const task of options.tasks.split(',').map(taskById)) {
+ const workingDirectory = await mkdtemp(join(tmpdir(), '67js-threejs-task-'));
+ try {
+ await cp(template, workingDirectory, { recursive: true });
+ const run = await runAgent(`${THREEJS_PREAMBLE}\n\n${task.prompt.replace(/\n/g, ' ')}`, workingDirectory);
+ const transcript = readTranscript(run.output);
+ await writeFile(join(rawDirectory, `${task.id}-${attempt}.jsonl`), run.output);
+ let geometries = null;
+ let grade = { pass: false, passed: 0, total: task.checks.length, results: [{ kind: 'geometry', pass: false, detail: 'no geometry.json' }] };
+ try {
+ geometries = JSON.parse(await readFile(join(workingDirectory, 'geometry.json'), 'utf8'));
+ grade = gradeGeometries(task, geometries);
+ } catch (error) {
+ grade.results[0].detail = `geometry.json unreadable: ${error.message.split('\n')[0]}`;
+ }
+ const png = await checkPng(join(workingDirectory, 'scene.png'));
+ if (geometries) {
+ await cp(join(workingDirectory, 'scene.png'), join(rawDirectory, `${task.id}-${attempt}.png`)).catch(() => {});
+ await cp(join(workingDirectory, 'build.mjs'), join(rawDirectory, `${task.id}-${attempt}.build.mjs`)).catch(() => {});
+ }
+ let silhouette = null;
+ if (geometries) silhouette = await scoreSilhouetteOfGeometries(task.id, geometries).catch(error => ({ error: error.message.split('\n')[0] }));
+ const usage = transcript.result?.usage ?? {};
+ const failed = grade.results.filter(result => !result.pass).map(result => `${result.label ?? result.kind}: ${result.detail}`);
+ if (!png.pass) failed.push(`scene.png: ${png.detail}`);
+ const record = {
+ task: task.id,
+ attempt,
+ pass: grade.pass && png.pass,
+ passed: grade.passed,
+ total: grade.total,
+ pngPass: png.pass,
+ failed: failed.map(removeLocalPaths),
+ model: transcript.model,
+ toolCalls: transcript.toolCalls,
+ toolErrors: transcript.toolErrors,
+ toolsByName: transcript.toolsByName,
+ look: silhouette ? { silhouette } : null,
+ turns: transcript.result?.num_turns ?? null,
+ costUsd: null,
+ inputTokens: (usage.input_tokens ?? 0) + (usage.cache_read_input_tokens ?? 0) + (usage.cache_creation_input_tokens ?? 0),
+ outputTokens: usage.output_tokens ?? 0,
+ wallSeconds: Math.round(run.wallMs / 1000),
+ exitCode: run.code,
+ stopped: transcript.result?.subtype ?? 'no result',
+ filesWritten: (await readdir(workingDirectory)).filter(name => !['node_modules', 'package-lock.json'].includes(name)),
+ };
+ records.push(record);
+ console.log(`${record.task.padEnd(10)} #${attempt} ${record.pass ? 'PASS' : 'FAIL'} ${record.passed}/${record.total} png ${png.pass} calls ${record.toolCalls} (errors ${record.toolErrors}) look ${silhouette?.mean ?? '-'} ${record.wallSeconds}s${failed.length ? ` ${failed.join(' | ')}` : ''}`);
+ await writeFile(join(resultsDirectory, `${runName}.json`), `${JSON.stringify(summarize(), null, 2)}\n`);
+ } finally {
+ await removeFolder(workingDirectory);
+ }
+ }
+ }
+ } finally {
+ await removeFolder(template);
+ }
+ console.log(`${records.filter(record => record.pass).length}/${records.length} passed, ${totalsOf(records).toolCalls} tool calls. Summary: bench/agent-modeling/results/${runName}.json`);
+}
diff --git a/changes/31.md b/changes/31.md
new file mode 100644
index 00000000..5e9be688
--- /dev/null
+++ b/changes/31.md
@@ -0,0 +1 @@
+- Compare agent friendliness of 67js MCP against three.js written as code on six suite tasks with the same model and the same checks and look score (issue #31, three.js half; the Blender MCP half still needs a VM): `bench/agent-modeling/threejs.mjs` runs the three.js side and `results/2026-11-07-threejs-vs-67js.json` holds both.
diff --git a/tests/agent-threejs-grade.test.js b/tests/agent-threejs-grade.test.js
new file mode 100644
index 00000000..4de9dd8f
--- /dev/null
+++ b/tests/agent-threejs-grade.test.js
@@ -0,0 +1,25 @@
+import assert from 'node:assert/strict';
+import test from 'node:test';
+import { worldGeometriesOf } from '../bench/agent-modeling/look.mjs';
+import { REFERENCES } from '../bench/agent-modeling/references.mjs';
+import { gradeGeometries, scoreSilhouetteOfGeometries } from '../bench/agent-modeling/threejs.mjs';
+import { taskById } from '../bench/agent-modeling/tasks.mjs';
+
+const asJson = geometries => geometries.map(({ positions, indices }) => ({ positions: Array.from(positions), indices: Array.from(indices) }));
+
+test('the three.js grader passes the reference geometry of each sampled task and scores its look as 1', async () => {
+ for (const id of ['crate', 'bolt', 'window', 'table', 'fence', 'bracket']) {
+ const geometries = asJson(await worldGeometriesOf(REFERENCES[id]));
+ const grade = gradeGeometries(taskById(id), geometries);
+ assert.equal(grade.pass, true, `${id}: ${JSON.stringify(grade.results.filter(result => !result.pass))}`);
+ assert.equal((await scoreSilhouetteOfGeometries(id, geometries)).mean, 1);
+ }
+});
+
+test('the three.js grader fails an open surface of the wrong size', () => {
+ const crate = taskById('crate');
+ const quad = { positions: [0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0], indices: [0, 1, 2, 0, 2, 3] };
+ const grade = gradeGeometries(crate, [quad]);
+ assert.equal(grade.pass, false);
+ assert.equal(grade.results.find(result => result.kind === 'lint').pass, false);
+});