3D engine that AI agents drive
Test the docs and MCP server with Claude Code models and without MCP (part of #69)
Nguyen Xuan An22f1f334ce09
diff --git a/bench/agent-modeling/README.md b/bench/agent-modeling/README.mdindex 5df71fb6..9409f635 100644--- a/bench/agent-modeling/README.md+++ b/bench/agent-modeling/README.md@@ -351,3 +351,42 @@ Result (measured, 3 runs per cell): branches did not save calls where the task wTool calls exclude `ToolSearch`. Each run was `scene_create` with the motion inside it and one `scene_export_recipe`, with no errors and no retry.Result (measured, 3 runs per task): the call count is the same with and without the easing, because Sonnet never used one. In the free arm all 3 runs wrote 6 plain keyframes on a `LINEAR` track with the peak at 1.35 m and passed on the first try. The 2-keyframe `easeOutBack` version in the reference was not found or not chosen. So the issue's condition is met as a comparison, and the answer is no difference in calls, 2 against 2. It does not show that the easing saves calls when an agent is told to use it, and the plain arm needed no retry, unlike the hand search that placed the reference peak. Not measured: Haiku, a prompt that names easings, or a harder curve.++## Other clients and models (2026-11-04, issue #69)++`results/2026-11-04-clients-models.json`. Claude Code runs used the 5-task sample `crate,bolt,window,table,mug`, 2 runs each, on the CPU backend, with `--model haiku` and `--model sonnet`. Pass rate and means are per run.++| Client and model | Task | Pass | Mean tool calls | Mean turns |+| --- | --- | --- | --- | --- |+| MCP Inspector CLI 2.13.0 | `tools/list`, `scene_create` | pass, 22 tools | | |+| MCP Python SDK 2.3.0 | `initialize`, `tools/list`, `scene_create`, `scene_look` | pass, 20480 byte PNG | | |+| Claude Code, haiku | all five MCP tasks | 10 of 10 | 7.0 | 9.0 |+| Claude Code, sonnet | all five MCP tasks | 9 of 10 | 3.2 | 5.3 |+| Claude Code, haiku | `llms.txt` and the package, no MCP | 2 of 2 | 14.0 | 15.0 |+| Claude Code, sonnet | `llms.txt` and the package, no MCP | 2 of 2 | 10.5 | 11.5 |++By task (pass of 2, mean tool calls):++| Task | haiku | sonnet |+| --- | --- | --- |+| crate | 2, 6.5 | 1, 2.0 |+| bolt | 2, 11.0 | 2, 2.0 |+| window | 2, 6.0 | 2, 3.5 |+| table | 2, 5.5 | 2, 2.0 |+| mug | 2, 6.0 | 2, 6.5 |++- Haiku passed everything but needed about twice the tool calls. Sonnet used fewer calls and failed one `crate` run: it wrote a single `inset_region` over four faces in 2 calls, never inspected the mesh, and got one loop per panel instead of two. That is a model shortcut, not a schema defect, and one failure in ten is within the noise of this sample.+- The no-MCP task is `llms-only.mjs`: a fresh folder with the package installed from a packed archive and a copy of `llms.txt`, no web access, no MCP. The agent writes a script that renders a red sphere on a ground plane to `scene.png`; `png-check.mjs` requires a real 512 px PNG with many colours. Both models pass.+- Friction found: in both sonnet runs the first `cd node_modules/67js` failed, because the guide's script assumes the `67js` alias install and the task installed `@codepawlhq/67js` plainly. Fixed in `docs/guide.md` and `public/llms.txt`, which now state both install names. These two sonnet runs predate that fix.+- The Inspector CLI starts a new server per command, so it cannot show a preview image; that is the client, not the server.+- Every page that `llms.txt` and the README link to is checked against the package and the site by `tests/other-clients.test.js`.++Not run:++| Client | Reason |+| --- | --- |+| Non-Anthropic models | None is free here: a hosted model needs an API key and spend, and a local LLM is deliberately not installed |+| Cursor | Interactive desktop client with its own account; no headless mode to drive |+| Gemini CLI | Needs a Google sign-in or API key, which this setup does not have |+| VS Code | Interactive client behind a Copilot subscription that is not set up here |+| Claude Desktop | A GUI app that cannot be driven without computer use |diff --git a/bench/agent-modeling/llms-only.mjs b/bench/agent-modeling/llms-only.mjsnew file mode 100644index 00000000..c25d2d1e--- /dev/null+++ b/bench/agent-modeling/llms-only.mjs@@ -0,0 +1,173 @@+// A coding task with no MCP: the agent has `llms.txt` and the installed npm package, and must write a script that+// renders a scene to a PNG. This measures the path a coding agent takes when it reads the docs instead of calling tools.+// node bench/agent-modeling/llms-only.mjs --claude <claude executable> [--model sonnet] [--runs 2] [--budget 1.5] [--timeout 900] [--label llms-only]+// Each attempt gets a fresh folder with the package installed from a packed archive, a copy of public/llms.txt and+// nothing else. The check opens scene.png and requires a real image with content (png-check.mjs).+import { spawn, execFileSync } from 'node:child_process';+import { copyFile, mkdir, mkdtemp, readdir, rm, writeFile } from 'node:fs/promises';+import { tmpdir } from 'node:os';+import { join, resolve } from 'node:path';+import { parseArgs } from 'node:util';+import { removeLocalPaths, totalsByModel } from './aggregate.mjs';+import { checkPng } from './png-check.mjs';++const repository = resolve(import.meta.dirname, '../..');+const TASK_ID = 'llms-only-png';+const BUILT_IN_TOOLS_DENIED = ['WebFetch', 'WebSearch', 'Agent', 'Task', 'NotebookEdit'];+const PROMPT = `Write a Node.js script named render.mjs in the current folder that renders a 3D scene to a PNG file named scene.png, 512 by 512 pixels, and run it.+The scene: a red sphere resting on a grey ground plane, seen from a three-quarter view above.+The 67js package is installed in node_modules. This folder also holds llms.txt, the map of its documentation. You have no internet access and no MCP tools, so use only llms.txt and the files in node_modules.+Google Chrome is installed. When scene.png exists, reply with one line saying so.`;++const { values: options } = parseArgs({+ options: {+ claude: { type: 'string', default: process.env.CLAUDE_EXECUTABLE ?? 'claude' },+ model: { type: 'string', default: 'sonnet' },+ runs: { type: 'string', default: '2' },+ budget: { type: 'string', default: '1.5' },+ timeout: { type: 'string', default: '900' },+ label: { type: 'string', default: 'llms-only' },+ },+});++const runName = `${new Date().toISOString().slice(0, 16).replace(/[:T]/g, '-')}-${options.model}-${options.label}`;+const rawDirectory = join(repository, 'artifacts', 'agent-modeling', runName);+const resultsDirectory = join(repository, 'bench', 'agent-modeling', 'results');+const npmExecutable = process.platform === 'win32' ? 'npm.cmd' : 'npm';++function packArchive(destination) {+ const output = execFileSync(npmExecutable, ['pack', '--ignore-scripts', '--json', '--pack-destination', destination], { cwd: repository, encoding: 'utf8', shell: process.platform === 'win32' });+ const packed = JSON.parse(output);+ return join(destination, (Array.isArray(packed) ? packed[0] : Object.values(packed)[0]).filename);+}++function runClaude(workingDirectory) {+ const argumentsList = [+ '-p', PROMPT,+ '--model', options.model,+ '--output-format', 'stream-json', '--verbose',+ '--strict-mcp-config',+ '--allowedTools', 'Bash', 'Read', 'Write', 'Edit', 'Glob', 'Grep',+ '--disallowedTools', ...BUILT_IN_TOOLS_DENIED,+ '--permission-mode', 'dontAsk',+ '--setting-sources', 'project',+ '--no-session-persistence',+ '--max-budget-usd', options.budget,+ ];+ return new Promise(resolvePromise => {+ const started = performance.now();+ const child = spawn(options.claude, argumentsList, { cwd: workingDirectory, windowsHide: true, stdio: ['ignore', 'pipe', 'pipe'] });+ let output = '';+ let errors = '';+ child.stdout.on('data', chunk => {+ output += chunk;+ });+ child.stderr.on('data', chunk => {+ errors += chunk;+ });+ const timer = setTimeout(() => child.kill(), Number(options.timeout) * 1000);+ child.on('close', code => {+ clearTimeout(timer);+ resolvePromise({ output, errors, code, wallMs: Math.round(performance.now() - started) });+ });+ });+}++function readTranscript(output) {+ const summary = { model: null, toolCalls: 0, toolErrors: 0, toolsByName: {}, result: null };+ for (const line of output.split('\n')) {+ if (!line.trim().startsWith('{')) continue;+ let event;+ try {+ event = JSON.parse(line);+ } catch {+ continue;+ }+ if (event.type === 'system' && event.subtype === 'init') summary.model = event.model;+ if (event.type === 'assistant') {+ for (const block of event.message?.content ?? []) {+ if (block.type !== 'tool_use') continue;+ summary.toolCalls++;+ summary.toolsByName[block.name] = (summary.toolsByName[block.name] ?? 0) + 1;+ }+ }+ if (event.type === 'user') {+ for (const block of event.message?.content ?? []) {+ if (block.type === 'tool_result' && block.is_error) summary.toolErrors++;+ }+ }+ if (event.type === 'result') summary.result = event;+ }+ return summary;+}++async function removeFolder(folder) {+ try {+ await rm(folder, { recursive: true, force: true, maxRetries: 10, retryDelay: 500 });+ } catch (error) {+ console.warn(`Left a temporary folder behind: ${error.code}`);+ }+}++async function runAttempt(archivePath, attempt) {+ const workingDirectory = await mkdtemp(join(tmpdir(), '67js-llms-only-'));+ try {+ await writeFile(join(workingDirectory, 'package.json'), JSON.stringify({ name: 'llms-only-task', private: true, type: 'module' }));+ execFileSync(npmExecutable, ['install', archivePath, '--ignore-scripts', '--no-audit', '--no-fund'], { cwd: workingDirectory, stdio: 'ignore', shell: process.platform === 'win32' });+ await copyFile(join(repository, 'public', 'llms.txt'), join(workingDirectory, 'llms.txt'));+ const run = await runClaude(workingDirectory);+ const transcript = readTranscript(run.output);+ const grade = await checkPng(join(workingDirectory, 'scene.png'));+ await writeFile(join(rawDirectory, `${TASK_ID}-${attempt}.jsonl`), run.output);+ const filesWritten = (await readdir(workingDirectory)).filter(name => !['node_modules', 'package-lock.json'].includes(name));+ const usage = transcript.result?.usage ?? {};+ return {+ task: TASK_ID,+ attempt,+ pass: grade.pass,+ detail: removeLocalPaths(grade.detail),+ filesWritten,+ model: transcript.model,+ toolCalls: transcript.toolCalls,+ toolErrors: transcript.toolErrors,+ toolsByName: transcript.toolsByName,+ turns: transcript.result?.num_turns ?? null,+ costUsd: transcript.result?.total_cost_usd ?? null,+ inputTokens: (usage.input_tokens ?? 0) + (usage.cache_read_input_tokens ?? 0) + (usage.cache_creation_input_tokens ?? 0),+ outputTokens: usage.output_tokens ?? 0,+ wallSeconds: Math.round(run.wallMs / 1000),+ stopped: transcript.result?.subtype ?? 'no result',+ };+ } finally {+ await removeFolder(workingDirectory);+ }+}++await mkdir(rawDirectory, { recursive: true });+await mkdir(resultsDirectory, { recursive: true });+const archiveFolder = await mkdtemp(join(tmpdir(), '67js-llms-only-pack-'));+const records = [];+try {+ const archivePath = packArchive(archiveFolder);+ for (let attempt = 1; attempt <= Number(options.runs); attempt++) {+ const record = await runAttempt(archivePath, attempt);+ records.push(record);+ console.log(`${TASK_ID} #${attempt} ${record.pass ? 'PASS' : 'FAIL'} ${record.detail} calls ${record.toolCalls} (errors ${record.toolErrors}) turns ${record.turns} $${record.costUsd ?? '?'} ${record.wallSeconds}s`);+ }+} finally {+ await removeFolder(archiveFolder);+}+const summary = {+ run: runName,+ model: records.find(record => record.model)?.model ?? options.model,+ date: new Date().toISOString(),+ runsPerTask: Number(options.runs),+ passed: records.filter(record => record.pass).length,+ tasks: records.length,+ toolCalls: records.reduce((total, record) => total + record.toolCalls, 0),+ costUsd: Number(records.reduce((total, record) => total + (record.costUsd ?? 0), 0).toFixed(2)),+ byModel: totalsByModel(records),+ records,+};+await writeFile(join(resultsDirectory, `${runName}.json`), `${JSON.stringify(summary, null, 2)}\n`);+console.log(`${summary.passed}/${summary.tasks} passed, ${summary.toolCalls} tool calls, $${summary.costUsd}. Summary: bench/agent-modeling/results/${runName}.json`);diff --git a/bench/agent-modeling/png-check.mjs b/bench/agent-modeling/png-check.mjsnew file mode 100644index 00000000..c3267c80--- /dev/null+++ b/bench/agent-modeling/png-check.mjs@@ -0,0 +1,76 @@+// Checks that a file is a real PNG with content: a valid signature, at least the given size, and enough different pixels.+// Used by llms-only.mjs. Reads 8-bit RGB and RGBA, non-interlaced PNGs, which is what canvas.toDataURL produces.+import { readFile } from 'node:fs/promises';+import { inflateSync } from 'node:zlib';++const SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);++function paeth(left, above, upperLeft) {+ const estimate = left + above - upperLeft;+ const distanceLeft = Math.abs(estimate - left);+ const distanceAbove = Math.abs(estimate - above);+ const distanceUpperLeft = Math.abs(estimate - upperLeft);+ if (distanceLeft <= distanceAbove && distanceLeft <= distanceUpperLeft) return left;+ return distanceAbove <= distanceUpperLeft ? above : upperLeft;+}++export function decodePng(buffer) {+ if (buffer.length < 33 || !buffer.subarray(0, 8).equals(SIGNATURE)) throw new Error('not a PNG: bad signature');+ let header = null;+ const dataChunks = [];+ for (let offset = 8; offset + 8 <= buffer.length;) {+ const length = buffer.readUInt32BE(offset);+ const type = buffer.toString('latin1', offset + 4, offset + 8);+ const body = buffer.subarray(offset + 8, offset + 8 + length);+ if (type === 'IHDR') header = { width: body.readUInt32BE(0), height: body.readUInt32BE(4), bitDepth: body[8], colorType: body[9], interlace: body[12] };+ if (type === 'IDAT') dataChunks.push(body);+ offset += 12 + length;+ }+ if (!header) throw new Error('not a PNG: no IHDR');+ const channels = { 2: 3, 6: 4 }[header.colorType];+ if (!channels || header.bitDepth !== 8 || header.interlace !== 0) throw new Error(`unsupported PNG layout: colour type ${header.colorType}, ${header.bitDepth} bits`);+ const raw = inflateSync(Buffer.concat(dataChunks));+ const rowBytes = header.width * channels;+ const pixels = Buffer.alloc(rowBytes * header.height);+ for (let row = 0; row < header.height; row++) {+ const filter = raw[row * (rowBytes + 1)];+ for (let column = 0; column < rowBytes; column++) {+ const value = raw[row * (rowBytes + 1) + 1 + column];+ const left = column >= channels ? pixels[row * rowBytes + column - channels] : 0;+ const above = row > 0 ? pixels[(row - 1) * rowBytes + column] : 0;+ const upperLeft = row > 0 && column >= channels ? pixels[(row - 1) * rowBytes + column - channels] : 0;+ const predictor = [0, left, above, (left + above) >> 1, paeth(left, above, upperLeft)][filter];+ pixels[row * rowBytes + column] = (value + predictor) & 255;+ }+ }+ return { ...header, channels, pixels };+}++/** Returns { pass, detail } for a PNG at `path`. */+export async function checkPng(path, { minimumSide = 256, minimumBytes = 2000, minimumColours = 16, minimumDifferingShare = 0.02 } = {}) {+ let buffer;+ try {+ buffer = await readFile(path);+ } catch {+ return { pass: false, detail: 'the file does not exist' };+ }+ try {+ const image = decodePng(buffer);+ if (image.width < minimumSide || image.height < minimumSide) return { pass: false, detail: `too small: ${image.width}x${image.height}` };+ if (buffer.length < minimumBytes) return { pass: false, detail: `only ${buffer.length} bytes` };+ const colours = new Set();+ const corner = image.pixels.subarray(0, image.channels).toString('hex');+ let differing = 0;+ for (let index = 0; index < image.pixels.length; index += image.channels) {+ const colour = image.pixels.subarray(index, index + image.channels).toString('hex');+ colours.add(colour);+ if (colour !== corner) differing++;+ }+ const share = differing / (image.width * image.height);+ if (colours.size < minimumColours) return { pass: false, detail: `only ${colours.size} distinct colours` };+ if (share < minimumDifferingShare) return { pass: false, detail: `${(share * 100).toFixed(1)}% of pixels differ from the corner` };+ return { pass: true, detail: `${image.width}x${image.height}, ${colours.size} colours, ${(share * 100).toFixed(0)}% differ from the corner` };+ } catch (error) {+ return { pass: false, detail: error.message };+ }+}diff --git a/bench/agent-modeling/results/2026-11-04-clients-models.json b/bench/agent-modeling/results/2026-11-04-clients-models.jsonnew file mode 100644index 00000000..3e492a5c--- /dev/null+++ b/bench/agent-modeling/results/2026-11-04-clients-models.json@@ -0,0 +1,488 @@+{+ "date": "2026-11-04",+ "suite": "agent-modeling 5-task sample (crate, bolt, window, table, mug), 2 runs each, CPU backend",+ "protocol": [+ {+ "client": "MCP Inspector CLI",+ "version": "2.13.0",+ "result": "tools/list shows 22 tools and scene_create works; scene_look in a second command fails because the CLI starts a new server for each command, so no scene exists (client behaviour, not a defect)"+ },+ {+ "client": "MCP Python SDK",+ "version": "2.3.0",+ "result": "initialize, tools/list (22 tools), scene_create and scene_look all pass; the preview is a 20480 byte PNG"+ }+ ],+ "claudeCode": [+ {+ "client": "claude-code-mcp",+ "modelAlias": "haiku",+ "model": "claude-haiku-5-5",+ "runs": 10,+ "passed": 10,+ "meanToolCalls": 7,+ "meanTurns": 9,+ "costUsd": 1.51,+ "tasks": [+ {+ "task": "crate",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 6.5,+ "meanTurns": 8.5+ },+ {+ "task": "bolt",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 11,+ "meanTurns": 13+ },+ {+ "task": "window",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 6,+ "meanTurns": 8+ },+ {+ "task": "table",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 5.5,+ "meanTurns": 7.5+ },+ {+ "task": "mug",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 6,+ "meanTurns": 8+ }+ ],+ "records": [+ {+ "task": "crate",+ "attempt": 1,+ "pass": true,+ "passed": 8,+ "total": 8,+ "failed": [],+ "toolCalls": 7,+ "toolErrors": 1,+ "turns": 9,+ "costUsd": 0.15819959999999997,+ "wallSeconds": 47+ },+ {+ "task": "bolt",+ "attempt": 1,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "toolCalls": 8,+ "toolErrors": 2,+ "turns": 10,+ "costUsd": 0.14608125,+ "wallSeconds": 30+ },+ {+ "task": "window",+ "attempt": 1,+ "pass": true,+ "passed": 5,+ "total": 5,+ "failed": [],+ "toolCalls": 6,+ "toolErrors": 0,+ "turns": 8,+ "costUsd": 0.14025845,+ "wallSeconds": 34+ },+ {+ "task": "table",+ "attempt": 1,+ "pass": true,+ "passed": 6,+ "total": 6,+ "failed": [],+ "toolCalls": 5,+ "toolErrors": 0,+ "turns": 7,+ "costUsd": 0.11627955000000002,+ "wallSeconds": 24+ },+ {+ "task": "mug",+ "attempt": 1,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "toolCalls": 6,+ "toolErrors": 0,+ "turns": 8,+ "costUsd": 0.17300064999999998,+ "wallSeconds": 78+ },+ {+ "task": "crate",+ "attempt": 2,+ "pass": true,+ "passed": 8,+ "total": 8,+ "failed": [],+ "toolCalls": 6,+ "toolErrors": 0,+ "turns": 8,+ "costUsd": 0.14595139999999998,+ "wallSeconds": 37+ },+ {+ "task": "bolt",+ "attempt": 2,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "toolCalls": 14,+ "toolErrors": 1,+ "turns": 16,+ "costUsd": 0.19055564999999997,+ "wallSeconds": 60+ },+ {+ "task": "window",+ "attempt": 2,+ "pass": true,+ "passed": 5,+ "total": 5,+ "failed": [],+ "toolCalls": 6,+ "toolErrors": 0,+ "turns": 8,+ "costUsd": 0.14460429999999996,+ "wallSeconds": 49+ },+ {+ "task": "table",+ "attempt": 2,+ "pass": true,+ "passed": 6,+ "total": 6,+ "failed": [],+ "toolCalls": 6,+ "toolErrors": 0,+ "turns": 8,+ "costUsd": 0.13450725000000002,+ "wallSeconds": 26+ },+ {+ "task": "mug",+ "attempt": 2,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "toolCalls": 6,+ "toolErrors": 0,+ "turns": 8,+ "costUsd": 0.16062745,+ "wallSeconds": 67+ }+ ]+ },+ {+ "client": "claude-code-mcp",+ "modelAlias": "sonnet",+ "model": "claude-sonnet-5-5",+ "runs": 10,+ "passed": 9,+ "meanToolCalls": 3.2,+ "meanTurns": 5.3,+ "costUsd": 4.39,+ "tasks": [+ {+ "task": "crate",+ "runs": 2,+ "passed": 1,+ "meanToolCalls": 2,+ "meanTurns": 4+ },+ {+ "task": "bolt",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 2,+ "meanTurns": 4+ },+ {+ "task": "window",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 3.5,+ "meanTurns": 6+ },+ {+ "task": "table",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 2,+ "meanTurns": 4+ },+ {+ "task": "mug",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 6.5,+ "meanTurns": 8.5+ }+ ],+ "records": [+ {+ "task": "crate",+ "attempt": 1,+ "pass": false,+ "passed": 5,+ "total": 8,+ "failed": [+ "volume: 0.2235 m3",+ "recessed panel on +X: 1 loops at x = 0.29",+ "recessed panel on -Z: 1 loops at z = -0.29"+ ],+ "toolCalls": 2,+ "toolErrors": 0,+ "turns": 4,+ "costUsd": 0.41402540000000004,+ "wallSeconds": 23+ },+ {+ "task": "bolt",+ "attempt": 1,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "toolCalls": 2,+ "toolErrors": 0,+ "turns": 4,+ "costUsd": 0.41078580000000003,+ "wallSeconds": 21+ },+ {+ "task": "window",+ "attempt": 1,+ "pass": true,+ "passed": 5,+ "total": 5,+ "failed": [],+ "toolCalls": 4,+ "toolErrors": 0,+ "turns": 6,+ "costUsd": 0.458918,+ "wallSeconds": 21+ },+ {+ "task": "table",+ "attempt": 1,+ "pass": true,+ "passed": 6,+ "total": 6,+ "failed": [],+ "toolCalls": 2,+ "toolErrors": 0,+ "turns": 4,+ "costUsd": 0.38984240000000003,+ "wallSeconds": 17+ },+ {+ "task": "mug",+ "attempt": 1,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "toolCalls": 6,+ "toolErrors": 1,+ "turns": 8,+ "costUsd": 0.521204,+ "wallSeconds": 41+ },+ {+ "task": "crate",+ "attempt": 2,+ "pass": true,+ "passed": 8,+ "total": 8,+ "failed": [],+ "toolCalls": 2,+ "toolErrors": 0,+ "turns": 4,+ "costUsd": 0.41737079999999993,+ "wallSeconds": 20+ },+ {+ "task": "bolt",+ "attempt": 2,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "toolCalls": 2,+ "toolErrors": 0,+ "turns": 4,+ "costUsd": 0.40829260000000006,+ "wallSeconds": 22+ },+ {+ "task": "window",+ "attempt": 2,+ "pass": true,+ "passed": 5,+ "total": 5,+ "failed": [],+ "toolCalls": 3,+ "toolErrors": 0,+ "turns": 6,+ "costUsd": 0.4602924,+ "wallSeconds": 24+ },+ {+ "task": "table",+ "attempt": 2,+ "pass": true,+ "passed": 6,+ "total": 6,+ "failed": [],+ "toolCalls": 2,+ "toolErrors": 0,+ "turns": 4,+ "costUsd": 0.38937520000000003,+ "wallSeconds": 18+ },+ {+ "task": "mug",+ "attempt": 2,+ "pass": true,+ "passed": 7,+ "total": 7,+ "failed": [],+ "toolCalls": 7,+ "toolErrors": 0,+ "turns": 9,+ "costUsd": 0.5163774000000001,+ "wallSeconds": 34+ }+ ]+ },+ {+ "client": "claude-code-llms-only",+ "modelAlias": "haiku",+ "model": "claude-haiku-5-5",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 14,+ "meanTurns": 15,+ "costUsd": 0.04,+ "tasks": [+ {+ "task": "llms-only-png",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 14,+ "meanTurns": 15+ }+ ],+ "records": [+ {+ "task": "llms-only-png",+ "attempt": 1,+ "pass": true,+ "detail": "512x512, 18069 colours, 85% differ from the corner",+ "toolCalls": 15,+ "toolErrors": 1,+ "turns": 16,+ "costUsd": 0.024062040000000003,+ "wallSeconds": 65+ },+ {+ "task": "llms-only-png",+ "attempt": 2,+ "pass": true,+ "detail": "512x512, 7916 colours, 84% differ from the corner",+ "toolCalls": 13,+ "toolErrors": 0,+ "turns": 14,+ "costUsd": 0.01483305,+ "wallSeconds": 47+ }+ ]+ },+ {+ "client": "claude-code-llms-only",+ "modelAlias": "sonnet",+ "model": "claude-sonnet-5-5",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 10.5,+ "meanTurns": 11.5,+ "costUsd": 0.46,+ "tasks": [+ {+ "task": "llms-only-png",+ "runs": 2,+ "passed": 2,+ "meanToolCalls": 10.5,+ "meanTurns": 11.5+ }+ ],+ "records": [+ {+ "task": "llms-only-png",+ "attempt": 1,+ "pass": true,+ "detail": "512x512, 12232 colours, 77% differ from the corner",+ "toolCalls": 10,+ "toolErrors": 2,+ "turns": 11,+ "costUsd": 0.28118660000000006,+ "wallSeconds": 31+ },+ {+ "task": "llms-only-png",+ "attempt": 2,+ "pass": true,+ "detail": "512x512, 13179 colours, 78% differ from the corner",+ "toolCalls": 11,+ "toolErrors": 2,+ "turns": 12,+ "costUsd": 0.18302639999999998,+ "wallSeconds": 31+ }+ ]+ }+ ],+ "notRun": [+ {+ "client": "Non-Anthropic models",+ "reason": "No free non-Anthropic model is available here: a hosted one needs an API key and spend, and a local LLM is not installed on this machine on purpose."+ },+ {+ "client": "Cursor",+ "reason": "Needs an interactive desktop client and its own account or subscription; no headless mode to drive."+ },+ {+ "client": "Gemini CLI",+ "reason": "Needs a Google sign-in or API key, which this setup does not have."+ },+ {+ "client": "VS Code",+ "reason": "An interactive client with Copilot behind a subscription that is not set up here; no headless mode."+ },+ {+ "client": "Claude Desktop",+ "reason": "A GUI app that cannot be driven without computer use, which is only used on request."+ }+ ]+}diff --git a/changes/69.md b/changes/69.mdnew file mode 100644index 00000000..397ce376--- /dev/null+++ b/changes/69.md@@ -0,0 +1 @@+- Record how other clients and Claude models use the docs and the MCP server (issue #69). A 5-task sample ran on haiku and sonnet through Claude Code, a no-MCP task gives an agent only `llms.txt` and the installed package (`bench/agent-modeling/llms-only.mjs`), and the results are in `bench/agent-modeling/results/2026-11-04-clients-models.json`. The guide and `llms.txt` now say that the render script assumes the `67js` alias install and what to change after a plain `npm install @codepawlhq/67js`; both sonnet runs first tried `node_modules/67js` and failed.diff --git a/docs/guide.md b/docs/guide.mdindex 51ca5f92..721eede5 100644--- a/docs/guide.md+++ b/docs/guide.md@@ -74,7 +74,8 @@ The bundled ES module includes its dependencies. Serve the HTML over HTTP. ReleaA coding agent that has no MCP client can still produce an image from the installed package.The renderer needs a browser, so the script serves one page over HTTP and reads the canvas back through `playwright-core`, which the package already depends on.-It needs an installed Google Chrome. Use `executablePath` instead of `channel` for another Chromium. Run it from the folder that holds `node_modules`:+It needs an installed Google Chrome. Use `executablePath` instead of `channel` for another Chromium. Run it from the folder that holds `node_modules`.+The import map assumes the alias install (`node_modules/67js`). After a plain `npm install @codepawlhq/67js`, use `node_modules/@codepawlhq/67js` in the map instead.```jsimport { createServer } from 'node:http';diff --git a/public/llms.txt b/public/llms.txtindex 65a59b2b..d2113856 100644--- a/public/llms.txt+++ b/public/llms.txt@@ -28,6 +28,7 @@ Keep renderer-facing geometry immutable. Call `renderer.updateParts(session.partThis addon does not expose an MCP server or a render job queue.The npm package includes a prebuilt stdio MCP server. After alias installation, run `npx 67js-mcp --backend gpu --memory 192` with Node.js 20.19+ and installed Chrome.+Install with `npm install 67js@npm:@codepawlhq/67js` so imports stay `67js`. A plain `npm install @codepawlhq/67js` puts the package in `node_modules/@codepawlhq/67js` instead.For MCP clients, invoke Node directly with the absolute path to `node_modules/67js/bin/67js-mcp.mjs`. Read `/mcp.md` for configuration.The browser starts only when creating a scene. No source checkout, Vite build or browser download is required.diff --git a/tests/agent-llms-only-check.test.js b/tests/agent-llms-only-check.test.jsnew file mode 100644index 00000000..34f496c8--- /dev/null+++ b/tests/agent-llms-only-check.test.js@@ -0,0 +1,76 @@+import assert from 'node:assert/strict';+import { mkdtemp, rm, writeFile } from 'node:fs/promises';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import test from 'node:test';+import { deflateSync } from 'node:zlib';+import { checkPng } from '../bench/agent-modeling/png-check.mjs';++const crcTable = Array.from({ length: 256 }, (_, index) => {+ let value = index;+ for (let bit = 0; bit < 8; bit++) value = value & 1 ? 0xedb88320 ^ (value >>> 1) : value >>> 1;+ return value >>> 0;+});++function crc32(bytes) {+ let crc = 0xffffffff;+ for (const byte of bytes) crc = crcTable[(crc ^ byte) & 255] ^ (crc >>> 8);+ return (crc ^ 0xffffffff) >>> 0;+}++function chunk(type, body) {+ const head = Buffer.alloc(8);+ head.writeUInt32BE(body.length, 0);+ head.write(type, 4, 'latin1');+ const tail = Buffer.alloc(4);+ tail.writeUInt32BE(crc32(Buffer.concat([head.subarray(4), body])), 0);+ return Buffer.concat([head, body, tail]);+}++/** An RGB PNG of `size` squared pixels whose colour comes from `pixelAt(x, y)`, every row stored with filter 0. */+function makePng(size, pixelAt) {+ const header = Buffer.alloc(13);+ header.writeUInt32BE(size, 0);+ header.writeUInt32BE(size, 4);+ header[8] = 8;+ header[9] = 2;+ const rows = [];+ for (let y = 0; y < size; y++) {+ const row = Buffer.alloc(1 + size * 3);+ for (let x = 0; x < size; x++) row.set(pixelAt(x, y), 1 + x * 3);+ rows.push(row);+ }+ const signature = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);+ return Buffer.concat([signature, chunk('IHDR', header), chunk('IDAT', deflateSync(Buffer.concat(rows))), chunk('IEND', Buffer.alloc(0))]);+}++async function checkBytes(bytes) {+ const folder = await mkdtemp(join(tmpdir(), '67js-png-check-'));+ try {+ const path = join(folder, 'scene.png');+ await writeFile(path, bytes);+ return await checkPng(path, { minimumBytes: 0 });+ } finally {+ await rm(folder, { recursive: true, force: true });+ }+}++test('a picture with a shaded disc on a plain background passes', async () => {+ const disc = makePng(300, (x, y) => {+ const distance = Math.hypot(x - 150, y - 150);+ return distance < 90 ? [255 - Math.floor(distance), 40, 30 + Math.floor(distance)] : [30, 30, 30];+ });+ assert.equal((await checkBytes(disc)).pass, true);+});++test('a flat picture, a tiny picture and a file that is not a PNG fail', async () => {+ assert.equal((await checkBytes(makePng(300, () => [30, 30, 30]))).pass, false);+ assert.equal((await checkBytes(makePng(64, (x, y) => [x * 3, y * 3, 0]))).pass, false);+ assert.equal((await checkBytes(Buffer.from('not an image at all, just text'))).pass, false);+});++test('a missing file fails with a reason', async () => {+ const result = await checkPng(join(tmpdir(), '67js-no-such-scene.png'));+ assert.equal(result.pass, false);+ assert.match(result.detail, /does not exist/);+});