3D engine that AI agents drive
Measure tool cost against the base branch instead of a shared budget file
Nguyen Xuan Anc2d5ce37797f
diff --git a/.github/workflows/test.yml b/.github/workflows/test.ymlindex 8811ea8a..57a1809d 100644--- a/.github/workflows/test.yml+++ b/.github/workflows/test.yml@@ -33,8 +33,9 @@ jobs:- uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0with:path: node_modules/.cache/size-base- key: size-base-${{ steps.base.outputs.sha }}-${{ hashFiles('scripts/measure-size.mjs', 'scripts/size-*.mjs', 'package-lock.json') }}+ key: size-base-${{ steps.base.outputs.sha }}-${{ hashFiles('scripts/measure-size.mjs', 'scripts/size-*.mjs', 'scripts/*tool-cost*.mjs', 'package-lock.json') }}- run: npm run size -- --base "$SIZE_BASE_REF"+ - run: npm run tool-cost -- --base "$SIZE_BASE_REF"- run: npm test- run: npm run test:isolated- run: npm run prepare:releasediff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 9ccc683d..f5ed9676 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -35,7 +35,9 @@ Run `npm run smoke` or the `smoke:*` script for the area that you changed, and lA change to skinning, glTF loading, poses or IK also needs `npm run smoke:character`, which takes a real character through the MCP server. Run `npm run prepare:release` first.Screenshots and other output go to `artifacts/`, which Git ignores.-Every pull request runs `npm audit`, `npm run size`, `npm test`, `npm run test:isolated` (each test file in its own process), `prepare:release`, `smoke:package` and `smoke:mcp-package` in the `Test` workflow. `smoke:mcp-package` draws through the MCP server on SwiftShader in Chrome. `npm run size` compares every bundle with `origin/main`. If a bundle grows past its allowance (64 bytes gzip for core and the scenes), add `changes/<issue>.size.md` with the line `- entry:core: <why it is worth the bytes>` and say why in the pull request. Never edit `bench/size/budgets.json` or `baseline.json` in a pull request. `bench/size/README.md` has the details.+Every pull request runs `npm audit`, `npm run size`, `npm run tool-cost`, `npm test`, `npm run test:isolated` (each test file in its own process), `prepare:release`, `smoke:package` and `smoke:mcp-package` in the `Test` workflow. `smoke:mcp-package` draws through the MCP server on SwiftShader in Chrome. `npm run size` compares every bundle with `origin/main`. If a bundle grows past its allowance (64 bytes gzip for core and the scenes), add `changes/<issue>.size.md` with the line `- entry:core: <why it is worth the bytes>` and say why in the pull request. Never edit `bench/size/budgets.json` or `baseline.json` in a pull request. `bench/size/README.md` has the details.++`npm run tool-cost` does the same for what the MCP tools cost an agent. If a tool schema grows by more than 64 bytes, a typical result by more than 128 bytes, or the schema total by more than 128 bytes, first move rare detail into `scene_capabilities` or a shared schema. If the growth is worth it, add a line to `changes/<issue>.size.md`, such as `- tool:scene_create: <why it is worth the bytes>`, `- tool-total: <why>` or `- result:scene_capabilities: <why>`. Never edit `bench/tool-cost-budgets.json` in a pull request. `bench/tool-cost/README.md` has the details.Branch protection on `main` requires the `test` check to pass before a merge. Merge a pull request only when `GitGuardian Security Checks` also passes.@@ -75,7 +77,7 @@ Keep site assets out of the npm package: the `files` list in `package.json` deciMaintainers make releases. `package.json` holds the version, and the publish workflow runs when it changes on `main`.1. Update the version in `package.json` and in `src/version.js`, then run `npm run changes:fold`. It writes the files in `changes/` into `CHANGELOG.md` as that version's section and deletes them. Read `npm run changes` first to see what it will fold. It also deletes the `changes/*.size.md` notes.-2. Run `npm run size:release`. It resets the size budgets and the baseline to the release sizes. Read the diff of `bench/size/budgets.json`.+2. Run `npm run size:release` and `npm run tool-cost:release`. They reset the size budgets, the baseline and the tool cost ceilings to the release numbers. Read the diffs of `bench/size/budgets.json` and `bench/tool-cost-budgets.json`.3. Run `npm test`, `npm run build` and `npm run smoke:package`.4. Commit and push to `main`.5. Wait for the `Publish 67js` workflow. It tests the package and publishes the new version through the npm trusted publisher.diff --git a/bench/tool-cost-budgets.json b/bench/tool-cost-budgets.jsonindex 7325ffa9..553b44d0 100644--- a/bench/tool-cost-budgets.json+++ b/bench/tool-cost-budgets.json@@ -1,5 +1,4 @@{- "measured": "2026-10-31, from scripts/measure-tool-cost.mjs; raised by 800 total and 150 for scene_apply for branch and labels fields (no new tool); raised by 650 total and 80 for scene_apply for clip layers, bone masks and retargeting through MCP; raised by 20 for scene_create and 20 for the scene_capabilities result after merging the expression language with the material graph; was 373388 schema bytes before shared IDs and recipe references; trimmed after the render-passes merge: material graph, depth of field, motion blur and passes details moved to scene_capabilities, budgets set to the measured value plus 0.5% (total 188267, scene_capabilities result 5167); raised by 95 for scene_apply and 690 for the scene_capabilities result for node relations; curve and text geometry entries shortened to a pointer at scene_capabilities curveText, budgets set to the measured value plus 0.5% (total 189468, scene_capabilities result 6399); then six modifier ops (weld, triangulate, edge_split, smooth, simple_deform, cast) added as one compact schema entry with details in scene_capabilities meshModifiers, entry and mesh_stack description shortened: total 190382 (budget unchanged at 190420), mesh_edit 34216 (budget 34220), scene_capabilities result 7042 (budget 7050); mesh_stack 11682 (budget 11690); after merging imported mesh topology with the modifier ops: raised the total to 190480 (measured 190471), mesh_edit to 34300 (measured 34293) and the scene_capabilities result to 7520 (measured 7512) for imported mesh edits; then cloth (recipe cloth, set-cloth, grid geometry) added with the total unchanged at 190480 (measured 189159): physics descriptions shortened and set-physics now takes the same record reference as set-motion, scene_create raised to 62150 and scene_apply lowered to 44130, scene_capabilities result raised to 8190 (measured 8176) for the cloth details; then particles and hair (recipe `particles`, set-particles) added with the total unchanged at 190480 (measured 189642, scene_create raised to 62350 and scene_apply to 44370 for the recipe particles reference and set-particles): the section is one record reference described in scene_capabilities, so the scene_capabilities result rose to 9570 (measured 9523) for the particle and hair fields; after merging the film timeline (recipe timeline, set-timeline) with particles: total 190719 (budget 190730), scene_create 62464, scene_apply 44620, scene_capabilities result 9902; then the path tracer renderer setting (drop z.int bounds, integers still checked by the job validator): total raised to 190830 (measured 190818); after merging the render limits and light bake with the path tracer: measured total 190013, scene_create raised to 62680 (measured 62667), scene_apply to 44860 (measured 44852) and the scene_capabilities result to 10620 (measured 10613) for the bake section and job size limits; then material layers anisotropy and iridescence (issue #51): the 16 layer fields take one short line each in the schema and their ranges and notes moved to scene_capabilities scene.materialLayers, so the scene_capabilities result rose to 12230 (measured 12168); then the worker bake limit added workerMaxSeconds to the bake limits (result 13361, budget raised to 13370); then the sculpt and shrinkwrap mesh edits (issue #47) with the total unchanged at 190830 (measured 190612): the face rule and tag prefix descriptions shortened (the face rule text repeats in every select field), so mesh_edit rose to 33980 (measured 33960) and mesh_stack fell to 11030 (measured 11014), and the scene_capabilities result rose to 13350 (measured 13338) for the meshSculpt notes; then recipe light links and shadow cascades (issue #52): environment fields added to the shared environment schema with the total unchanged at 190830 (measured 190507) by shortening its descriptions, and the scene_capabilities result raised to 13870 (measured 13865) for lightLinking and shadowCascades; then the nine remaining modifiers (issue #42: screw, skin, wireframe, lattice, curve, hook, wave, weighted_normal, smooth_by_angle) added their notes to meshModifiers and a free modifier record replaced the modifier schema entry, so the schemas shrank and the scene_capabilities result rose to 15900 (measured 15814); path query of scene_inspect (issue #64): one record field of 205 bytes in the scene_inspect schema (total 190732, within its budget) and the navigation notes in scene_capabilities, so the result rose to 14500 (measured 14494); reflection probes (issue #55): the bake.probes line, and the scene_capabilities result raised to 16600 (measured 16541) after merging with main; volumeFog shadowAmbient (issue #165): scene_create raised to 62945 (measured 62940) and scene_apply to 45040 (measured 45037); keyframe easing (issue #286): one easing string field in each of the motion track schemas, scene_create raised to 63450 (measured 63444) and the total to 191000 (measured 190994); glTF import line gltfImport (issue #280): the scene_capabilities result raised to 16701 (measured 16681); depth bias line depthBias (issue #310): the scene_capabilities result raised to 17000 (measured 16979); view transforms and looks (issue #282): one viewTransform line in scene_capabilities, so the result is raised to 17020 (measured 16996); both depthBias and viewTransform lines together: raised to 17320 (measured 17292); outlines and toon light (issue #285), after merging depthBias and view transforms: environment.lines is one free record and material toon one short line, total 192150 (measured 192127), scene_create 63800 (measured 63778), scene_apply 45480 (measured 45463), scene_create_asset 19500 (measured 19480), and the scene_capabilities result 17670 (measured 17647) for the lineArt entry","schemaTotalBytes": 192150,"schemaToolBytes": {"scene_create": 63800,diff --git a/bench/tool-cost/README.md b/bench/tool-cost/README.mdnew file mode 100644index 00000000..19ffb928--- /dev/null+++ b/bench/tool-cost/README.md@@ -0,0 +1,44 @@+# Tool cost++Every MCP tool schema is read on every turn, so their size is a cost to each agent. `npm run tool-cost` measures three kinds of bytes and compares them with the same numbers on the base branch. The `Test` workflow runs it on every pull request. A pull request never edits a tool cost number, so two pull requests do not conflict over one.++## What it measures++- `tool:<name>`: the description and input schema of one tool.+- `tool-total`: all tool schemas together.+- `result:<name>`: a typical result of one tool, from `scripts/measure-tool-cost.mjs`.++`node scripts/measure-tool-cost.mjs` prints the full report with estimated tokens.++## How the check decides++The check measures the base branch with its own copy of the script, then compares each cost. It reuses the machinery of the size check: the same base ref, the same cache by base commit and the same notes. On your machine the base is `origin/main`, from the commit where your branch left it.++A cost fails when it grows by more than its allowance:++- A tool schema: 64 bytes.+- A typical result: 128 bytes.+- The schema total: 128 bytes.++The allowance is about one short schema field. A change that adds a field to one or two tools passes. A new cost that the base does not have is reported and never fails. A cost that shrinks never fails.++## When a check fails++1. Move rare detail into `scene_capabilities`, or reference a shared schema. This is the best fix, because the schemas are read on every turn and `scene_capabilities` is read once.+2. If the growth is worth its cost, add `changes/<issue>.size.md` to your pull request, with one line for each cost that you accept:++```md+- tool:scene_create: the new environment field costs 90 bytes+- tool-total: the same field+- result:scene_capabilities: its notes are 140 bytes+```++3. State the same reason in the pull request.++The check reads only the notes that your branch adds, so a note from another pull request never accepts your growth. Your note is a new file, so it conflicts with nobody. A size note can hold bundle lines and tool cost lines together.++## Release ceilings++`bench/tool-cost-budgets.json` holds the absolute ceilings. They are reported on a pull request and enforced with `npm run tool-cost -- --absolute`. A release refreshes them with `npm run tool-cost:release`, which sets each ceiling 2% over the measured value. Read the diff, because a reset hides growth since the last reset.++The file once held a `measured` note that every pull request extended. It is gone, and `git log -p bench/tool-cost-budgets.json` keeps the history.diff --git a/changes/345.md b/changes/345.mdnew file mode 100644index 00000000..6059efaa--- /dev/null+++ b/changes/345.md@@ -0,0 +1 @@+- Measure tool schema and result bytes against the base branch with `npm run tool-cost` (issue #345). A pull request that grows a tool by more than 64 bytes, a result by more than 128 or the total by more than 128 accepts it in its own `changes/<issue>.size.md`, so `bench/tool-cost-budgets.json` no longer conflicts. The `measured` history string is gone and the absolute budgets are release ceilings, refreshed by `npm run tool-cost:release`.diff --git a/changes/README.md b/changes/README.mdindex 2fb910ee..d8f930f4 100644--- a/changes/README.md+++ b/changes/README.md@@ -10,4 +10,4 @@ Do not edit `CHANGELOG.md` in a pull request. Two pull requests then never touch`npm run changes` prints the pending notes. At release time `npm run changes:fold` writes them into `CHANGELOG.md` as the section for the version in `package.json` and deletes the files.-A size note, `changes/<issue>.size.md`, is not a changelog bullet. It accepts a bundle size increase, and `bench/size/README.md` says how. `changes:fold` leaves it out of the changelog and deletes it.+A size note, `changes/<issue>.size.md`, is not a changelog bullet. It accepts a bundle size increase, and `bench/size/README.md` says how. `changes:fold` leaves it out of the changelog and deletes it. The same note accepts a tool cost increase, with lines such as `- tool:scene_create: <reason>`, and `bench/tool-cost/README.md` says how.diff --git a/docs/mcp.md b/docs/mcp.mdindex a1b82ff3..6e910a21 100644--- a/docs/mcp.md+++ b/docs/mcp.md@@ -149,7 +149,7 @@ Every error an agent can reach names what failed and what to do next. `tests/too### Tool cost and diagnostics-Every tool schema is read on every turn, so `node scripts/measure-tool-cost.mjs` reports the bytes and estimated tokens of each schema and of each typical result, and `tests/tool-cost.test.js` fails when one passes `bench/tool-cost-budgets.json`. The schemas total 173,216 bytes (about 49,500 tokens). To save space, a mesh geometry's `edits` and `stack` in a recipe, and `motion` in `scene_apply` and asset recipes, are advertised as plain objects: the full field list is in `mesh_edit`, `mesh_stack` and `scene_create`, and the backend still validates every field and names the valid choices when one is wrong. A small mesh change is far cheaper as `mesh_edit` (187 bytes in the measured case) than as a whole recipe rewrite (1,254 bytes with the dispose).+Every tool schema is read on every turn, so `node scripts/measure-tool-cost.mjs` reports the bytes and estimated tokens of each schema and of each typical result, and `npm run tool-cost` fails a pull request when one grows past its allowance against the base branch (`bench/tool-cost/README.md`). The schemas total 173,216 bytes (about 49,500 tokens). To save space, a mesh geometry's `edits` and `stack` in a recipe, and `motion` in `scene_apply` and asset recipes, are advertised as plain objects: the full field list is in `mesh_edit`, `mesh_stack` and `scene_create`, and the backend still validates every field and names the valid choices when one is wrong. A small mesh change is far cheaper as `mesh_edit` (187 bytes in the measured case) than as a whole recipe rewrite (1,254 bytes with the dispose).`mesh_inspect` lint adds warnings for `intersecting-parts`, `self-intersection`, `thin-wall`, `floating-part` and `asymmetric`. They never set `ok` to false, run within face budgets, and list any check they skipped in `skippedChecks`. `scene_inspect` returns a `diagnostics` list only when there is something to say: ignored material fields, blended transmission, unused materials or geometries, and for imported assets the decoder's not-drawn features, rigs with no joints or no skin, and assets with no clips. Each item has `kind`, `severity`, `target`, `message` and `fix`.diff --git a/package.json b/package.jsonindex 76ac02c5..d050d236 100644--- a/package.json+++ b/package.json@@ -149,6 +149,8 @@"reference": "node bench/reference/run.mjs","size": "node scripts/measure-size.mjs","size:release": "node scripts/measure-size.mjs --set-budgets",+ "tool-cost": "node scripts/check-tool-cost.mjs",+ "tool-cost:release": "node scripts/check-tool-cost.mjs --set-budgets","benchmark:runtime": "node bench/runtime/run.mjs","benchmark:crowd": "node scripts/benchmark-crowd.mjs","check:tool-schemas": "node scripts/check-tool-schemas.mjs",diff --git a/scripts/check-tool-cost.mjs b/scripts/check-tool-cost.mjsnew file mode 100644index 00000000..552f2bcf--- /dev/null+++ b/scripts/check-tool-cost.mjs@@ -0,0 +1,73 @@+// Checks what the MCP tools cost an agent: npm run tool-cost+// A pull request fails when a tool schema, the schema total or a typical result grows past its allowance against the base branch+// (--base, default origin/main) unless its own changes/*.size.md accepts it. --absolute also enforces the release ceilings in+// bench/tool-cost-budgets.json. --set-budgets refreshes the ceilings at release.+import { readFile, writeFile } from 'node:fs/promises';+import { fileURLToPath } from 'node:url';+import { measureToolCost } from './measure-tool-cost.mjs';+import { DEFAULT_BASE_REF, baseRefExists, measureBase, readAcceptedBundles } from './size-base.mjs';+import { budgetsFrom, ceilingProblems, compareCostsToBase, costsFrom, formatCostRows } from './tool-cost-growth.mjs';++const BUDGETS_PATH = 'bench/tool-cost-budgets.json';+// The measuring script imports the source beside itself, so the base branch runs its own copy of it.+const TOOL_COST_METHOD = {+ script: fileURLToPath(new URL('./measure-tool-cost.mjs', import.meta.url)),+ runInBase: true,+ files: ['measure-tool-cost.mjs', 'check-tool-cost.mjs', 'tool-cost-growth.mjs', 'size-base.mjs'],+};++const argumentsGiven = process.argv.slice(2);+const flags = new Set(argumentsGiven);++function valueAfter(flag) {+ const index = argumentsGiven.indexOf(flag);+ return index === -1 ? undefined : argumentsGiven[index + 1];+}++const requestedBase = valueAfter('--base');+const baseRef = requestedBase ?? DEFAULT_BASE_REF;+if (requestedBase && !baseRefExists(baseRef)) {+ console.error(`The base ref ${baseRef} does not exist. Fetch it first, for example: git fetch origin main`);+ process.exit(1);+}+const refreshing = flags.has('--set-budgets');+const checkGrowth = baseRefExists(baseRef) && !refreshing;+const baseMeasurement = checkGrowth ? measureBase(baseRef, TOOL_COST_METHOD) : null;+const measured = costsFrom(await measureToolCost());+const budgets = JSON.parse(await readFile(BUDGETS_PATH, 'utf8'));+const problems = [];++if (refreshing) {+ await writeFile(BUDGETS_PATH, `${JSON.stringify(budgetsFrom(measured, budgets), null, 2)}\n`);+ console.log(`Wrote the release ceilings to ${BUDGETS_PATH}.`);+ process.exit(0);+}++if (checkGrowth) {+ const accepted = readAcceptedBundles(baseRef);+ const growth = compareCostsToBase(measured, costsFrom(await baseMeasurement), accepted);+ console.log(`Tool cost against ${baseRef}, in bytes:`);+ console.log(formatCostRows(growth.rows));+ for (const [name, reason] of accepted) {+ if (name in measured) console.log(`Accepted: ${name}, ${reason}`);+ }+ problems.push(...growth.problems);+} else {+ console.log(`No ${baseRef} to compare with, so the growth check is skipped. Fetch it, or pass --base <ref>.`);+}++// The absolute budgets are the release ceilings. They are reported on a pull request and enforced with --absolute.+const overCeiling = ceilingProblems(measured, budgets);+if (overCeiling.length) {+ if (flags.has('--absolute')) {+ problems.push(...overCeiling);+ } else {+ console.log(`\n${overCeiling.length} cost(s) are past their release ceiling. A release refreshes them with npm run tool-cost:release. List them with npm run tool-cost -- --absolute.`);+ }+}+if (problems.length) {+ console.error(`\n${problems.length} tool cost problem(s):`);+ for (const problem of problems) console.error(`- ${problem}`);+ process.exit(1);+}+console.log('\nNo tool cost grew past its allowance.');diff --git a/scripts/size-base.mjs b/scripts/size-base.mjsindex 00b03b85..cf2e143d 100644--- a/scripts/size-base.mjs+++ b/scripts/size-base.mjs@@ -3,7 +3,7 @@ import { execFile, spawnSync } from 'node:child_process';import { createHash } from 'node:crypto';import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from 'node:fs';import { tmpdir } from 'node:os';-import { join, resolve } from 'node:path';+import { basename, join, resolve } from 'node:path';import { promisify } from 'node:util';import { fileURLToPath } from 'node:url';import { parseSizeNote } from './size-growth.mjs';@@ -11,8 +11,9 @@ import { parseSizeNote } from './size-growth.mjs';export const DEFAULT_BASE_REF = 'origin/main';const SIZE_NOTE_PATTERN = /^changes\/[^/]+\.size\.md$/;const SCRIPTS_FOLDER = fileURLToPath(new URL('.', import.meta.url));-const MEASURE_SCRIPT = join(SCRIPTS_FOLDER, 'measure-size.mjs');-const METHOD_FILES = ['measure-size.mjs', 'size-base.mjs', 'size-budget.mjs', 'size-growth.mjs'];+// A measuring method names the script that prints JSON for --json and the files that decide its numbers. A script that imports+// the source relative to itself runs from the base checkout (runInBase), so it measures the base source.+export const SIZE_METHOD = { script: join(SCRIPTS_FOLDER, 'measure-size.mjs'), runInBase: false, files: ['measure-size.mjs', 'size-base.mjs', 'size-budget.mjs', 'size-growth.mjs'] };// CI caches this folder, keyed by the base commit, so a run on an unchanged base skips the second measurement.export const BASE_CACHE_FOLDER = 'node_modules/.cache/size-base';const MAX_OUTPUT_BYTES = 256 * 1024 * 1024;@@ -50,15 +51,16 @@ export function readAcceptedBundles(baseRef, cwd = process.cwd()) {}// The cache key holds everything that decides the base numbers: the base commit, the measuring method and the lock file.-function cacheFileFor(baseRef) {+function cacheFileFor(baseRef, method) {const hash = createHash('sha256');hash.update(mergeBaseOf(baseRef));- for (const file of METHOD_FILES) hash.update(readFileSync(join(SCRIPTS_FOLDER, file)));+ hash.update(basename(method.script));+ for (const file of method.files) hash.update(readFileSync(join(SCRIPTS_FOLDER, file)));if (existsSync('package-lock.json')) hash.update(readFileSync('package-lock.json'));return join(BASE_CACHE_FOLDER, `${hash.digest('hex').slice(0, 24)}.json`);}-async function measureCheckout(baseRef) {+async function measureCheckout(baseRef, method) {const folder = mkdtempSync(join(tmpdir(), 'size-base-'));const checkout = join(folder, 'base');try {@@ -70,7 +72,7 @@ async function measureCheckout(baseRef) {const extracted = spawnSync('git', ['checkout-index', '--all', '--force', `--prefix=${checkout}/`], { encoding: 'utf8', windowsHide: true, env: environment });if (extracted.status !== 0) throw new Error(`Could not unpack ${baseRef}: ${extracted.stderr}`);if (existsSync('node_modules')) symlinkSync(resolve('node_modules'), join(checkout, 'node_modules'), 'junction');- const { stdout } = await execFileAsync(process.execPath, [MEASURE_SCRIPT, '--json'], { cwd: checkout, maxBuffer: MAX_OUTPUT_BYTES, windowsHide: true });+ const { stdout } = await execFileAsync(process.execPath, [method.runInBase ? join(checkout, 'scripts', basename(method.script)) : method.script, '--json'], { cwd: checkout, maxBuffer: MAX_OUTPUT_BYTES, windowsHide: true });return JSON.parse(stdout);} finally {rmSync(join(checkout, 'node_modules'), { force: true });@@ -80,10 +82,10 @@ async function measureCheckout(baseRef) {// Measures the base branch with this checkout's script, so both sides use the same method. It runs beside the// measurement of the pull request, and a cached result for the same base commit skips it.-export async function measureBase(baseRef) {- const cacheFile = cacheFileFor(baseRef);+export async function measureBase(baseRef, method = SIZE_METHOD) {+ const cacheFile = cacheFileFor(baseRef, method);if (existsSync(cacheFile)) return JSON.parse(readFileSync(cacheFile, 'utf8'));- const measured = await measureCheckout(baseRef);+ const measured = await measureCheckout(baseRef, method);mkdirSync(BASE_CACHE_FOLDER, { recursive: true });writeFileSync(cacheFile, JSON.stringify(measured));return measured;diff --git a/scripts/tool-cost-growth.mjs b/scripts/tool-cost-growth.mjsnew file mode 100644index 00000000..9e55558a--- /dev/null+++ b/scripts/tool-cost-growth.mjs@@ -0,0 +1,92 @@+// The growth gate for what MCP tools cost an agent. A pull request is measured against the base branch, so no pull request+// edits a number that another one edits. It reuses the size notes: acceptInstruction comes from size-growth.mjs.+import { acceptInstruction } from './size-growth.mjs';++export const TOOL_ALLOWANCE_BYTES = 64;+export const RESULT_ALLOWANCE_BYTES = 128;+export const TOTAL_ALLOWANCE_BYTES = 128;+export const TOTAL_NAME = 'tool-total';+const BUDGET_MARGIN = 1.02;+const BUDGET_ROUNDING_BYTES = 10;+const MOVE_ADVICE = 'Move rare detail into scene_capabilities or reference a shared schema.';++/** Flattens a report from measureToolCost() into named byte counts: tool:<name>, tool-total and result:<name>. */+export function costsFrom(report) {+ const costs = { [TOTAL_NAME]: report.schemaTotal.bytes };+ for (const row of report.schemas) costs[`tool:${row.name}`] = row.bytes;+ for (const row of report.results) costs[`result:${row.name}`] = row.bytes;+ return costs;+}++export function costAllowance(name) {+ if (name === TOTAL_NAME) return TOTAL_ALLOWANCE_BYTES;+ return name.startsWith('result:') ? RESULT_ALLOWANCE_BYTES : TOOL_ALLOWANCE_BYTES;+}++/** Compares each cost with the same cost on the base branch. A cost the base does not have is reported, not failed. */+export function compareCostsToBase(measured, baseMeasured, accepted = new Map()) {+ const rows = [];+ const problems = [];+ for (const [name, bytes] of Object.entries(measured)) {+ const base = baseMeasured[name];+ const growth = base === undefined ? null : bytes - base;+ const allowance = base === undefined ? null : costAllowance(name);+ const isAccepted = accepted.has(name);+ rows.push({ name, bytes, base: base ?? null, growth, allowance, accepted: isAccepted });+ if (growth !== null && growth > allowance && !isAccepted) {+ problems.push(`${name} grew by ${growth} bytes against the base branch (${base} to ${bytes}), over its allowance of ${allowance}. ${MOVE_ADVICE} ${acceptInstruction(name)}`);+ }+ }+ return { rows, problems };+}++function noteFor(row) {+ if (row.growth === null) return 'new, reported only';+ if (row.growth <= row.allowance) return '';+ return row.accepted ? 'accepted' : 'OVER';+}++/** Prints the costs that moved. An unchanged cost adds nothing, so the table stays short. */+export function formatCostRows(rows) {+ const moved = rows.filter(row => row.growth !== 0);+ if (!moved.length) return 'No tool schema or result changed against the base branch.';+ const header = `${'cost'.padEnd(34)}${'bytes'.padStart(9)}${'base'.padStart(9)}${'growth'.padStart(9)}${'allowed'.padStart(9)} note`;+ const lines = moved.map(row => [+ row.name.padEnd(34), String(row.bytes).padStart(9), String(row.base ?? '-').padStart(9),+ String(row.growth ?? '-').padStart(9), String(row.allowance ?? '-').padStart(9), ` ${noteFor(row)}`,+ ].join(''));+ return [header, ...lines].join('\n');+}++function ceilingFor(bytes) {+ return Math.ceil((bytes * BUDGET_MARGIN) / BUDGET_ROUNDING_BYTES) * BUDGET_ROUNDING_BYTES;+}++/** Release ceilings: 2% over what is measured now. Only the large tools get their own line, the rest share schemaOtherToolBytes. */+export function budgetsFrom(measured, previous) {+ const budgets = { ...previous, schemaToolBytes: {}, resultBytes: {} };+ budgets.schemaTotalBytes = ceilingFor(measured[TOTAL_NAME]);+ for (const [name, bytes] of Object.entries(measured)) {+ if (name.startsWith('tool:') && bytes > previous.schemaOtherToolBytes) budgets.schemaToolBytes[name.slice('tool:'.length)] = ceilingFor(bytes);+ if (name.startsWith('result:')) budgets.resultBytes[name.slice('result:'.length)] = ceilingFor(bytes);+ }+ return budgets;+}++/** The release ceilings in bench/tool-cost-budgets.json that the measured costs are past, as problems. */+export function ceilingProblems(measured, budgets) {+ const problems = [];+ const check = (name, bytes, ceiling) => {+ if (bytes > ceiling) problems.push(`${name} is ${bytes} bytes, over its release ceiling of ${ceiling} by ${bytes - ceiling}. ${MOVE_ADVICE}`);+ };+ check(TOTAL_NAME, measured[TOTAL_NAME], budgets.schemaTotalBytes);+ for (const [name, bytes] of Object.entries(measured)) {+ if (name.startsWith('tool:')) check(name, bytes, budgets.schemaToolBytes[name.slice('tool:'.length)] ?? budgets.schemaOtherToolBytes);+ if (name.startsWith('result:')) {+ const ceiling = budgets.resultBytes[name.slice('result:'.length)];+ if (ceiling === undefined) problems.push(`${name} has no release ceiling. A release adds one with npm run tool-cost:release.`);+ else check(name, bytes, ceiling);+ }+ }+ return problems;+}diff --git a/tests/tool-cost-growth.test.js b/tests/tool-cost-growth.test.jsnew file mode 100644index 00000000..cf04fa19--- /dev/null+++ b/tests/tool-cost-growth.test.js@@ -0,0 +1,107 @@+import assert from 'node:assert/strict';+import { spawnSync } from 'node:child_process';+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import test from 'node:test';+import { readAcceptedBundles } from '../scripts/size-base.mjs';+import { budgetsFrom, ceilingProblems, compareCostsToBase, costAllowance, costsFrom } from '../scripts/tool-cost-growth.mjs';++const report = {+ schemaTotal: { bytes: 20000 },+ schemas: [{ name: 'scene_create', bytes: 12000 }, { name: 'scene_apply', bytes: 8000 }],+ results: [{ name: 'scene_capabilities', bytes: 300 }],+};++test('costs are named tool:<name>, tool-total and result:<name>', () => {+ assert.deepEqual(costsFrom(report), { 'tool-total': 20000, 'tool:scene_create': 12000, 'tool:scene_apply': 8000, 'result:scene_capabilities': 300 });+});++test('a tool gets 64 bytes, a result and the total 128', () => {+ assert.equal(costAllowance('tool:scene_create'), 64);+ assert.equal(costAllowance('result:scene_capabilities'), 128);+ assert.equal(costAllowance('tool-total'), 128);+});++test('a tool that grows past the allowance fails with the move advice and the accept instruction', () => {+ const { problems } = compareCostsToBase({ 'tool:scene_create': 700 }, { 'tool:scene_create': 600 });+ assert.equal(problems.length, 1);+ assert.match(problems[0], /tool:scene_create grew by 100 bytes against the base branch \(600 to 700\), over its allowance of 64/);+ assert.match(problems[0], /Move rare detail into scene_capabilities or reference a shared schema/);+ assert.match(problems[0], /Reduce it, or accept it: add changes\/<issue>\.size\.md/);+ assert.match(problems[0], /- tool:scene_create: <why it is worth the bytes>/);+});++test('growth inside the allowance, a shrink and a new cost pass', () => {+ const measured = { 'tool:scene_create': 664, 'tool:scene_apply': 300, 'tool:fresh': 90 };+ const base = { 'tool:scene_create': 600, 'tool:scene_apply': 400 };+ const { rows, problems } = compareCostsToBase(measured, base);+ assert.deepEqual(problems, []);+ assert.deepEqual(rows.map(row => row.growth), [64, -100, null]);+});++test('a note accepts only the costs it names', () => {+ const measured = { 'tool-total': 1500, 'tool:scene_create': 700, 'result:scene_capabilities': 600 };+ const base = { 'tool-total': 1000, 'tool:scene_create': 600, 'result:scene_capabilities': 300 };+ const accepted = new Map([['tool:scene_create', 'a new field'], ['tool-total', 'the same field']]);+ const { problems } = compareCostsToBase(measured, base, accepted);+ assert.equal(problems.length, 1);+ assert.match(problems[0], /result:scene_capabilities grew by 300/);+});++test('release ceilings are listed as problems, and a release resets them', () => {+ const budgets = { schemaTotalBytes: 19000, schemaToolBytes: { scene_create: 12500 }, schemaOtherToolBytes: 4000, resultBytes: { scene_capabilities: 320 } };+ const measured = costsFrom(report);+ const problems = ceilingProblems(measured, budgets);+ assert.equal(problems.length, 2);+ assert.match(problems[0], /tool-total is 20000 bytes, over its release ceiling of 19000 by 1000/);+ assert.match(problems[1], /tool:scene_apply is 8000 bytes, over its release ceiling of 4000 by 4000/);+ const reset = budgetsFrom(measured, budgets);+ assert.deepEqual(ceilingProblems(measured, reset), []);+ assert.equal(reset.schemaOtherToolBytes, 4000);+});++function runGit(folder, ...args) {+ const identity = ['-c', 'user.name=Test', '-c', '[email protected]', '-c', 'commit.gpgsign=false', '-c', 'core.autocrlf=false'];+ const result = spawnSync('git', [...identity, ...args], { cwd: folder, encoding: 'utf8' });+ assert.equal(result.status, 0, `${args.join(' ')}: ${result.stderr}`);+ return result.stdout;+}++function commitFile(folder, path, text, message) {+ mkdirSync(join(folder, path, '..'), { recursive: true });+ writeFileSync(join(folder, path), text);+ runGit(folder, 'add', '.');+ runGit(folder, 'commit', '-q', '-m', message);+}++test('two pull requests that grow different tools merge in sequence and neither touches the shared budget file', () => {+ const folder = mkdtempSync(join(tmpdir(), 'tool-cost-sim-'));+ try {+ runGit(folder, 'init', '-q', '-b', 'main');+ commitFile(folder, 'bench/tool-cost-budgets.json', '{\n "schemaTotalBytes": 1000\n}\n', 'budgets');+ commitFile(folder, 'src/create.js', 'export const create = 1;\n', 'create');+ commitFile(folder, 'src/apply.js', 'export const apply = 1;\n', 'apply');+ runGit(folder, 'update-ref', 'refs/remotes/origin/main', 'HEAD');++ runGit(folder, 'checkout', '-q', '-b', 'grow-create');+ commitFile(folder, 'src/create.js', 'export const create = 2;\n', 'grow create');+ commitFile(folder, 'changes/301.size.md', '- tool:scene_create: a new field\n- tool-total: the same field\n', 'accept create');+ runGit(folder, 'checkout', '-q', 'main');+ runGit(folder, 'checkout', '-q', '-b', 'grow-apply');+ commitFile(folder, 'src/apply.js', 'export const apply = 2;\n', 'grow apply');+ commitFile(folder, 'changes/302.size.md', '- tool:scene_apply: a new field\n- result:scene_capabilities: its notes\n', 'accept apply');++ assert.deepEqual([...readAcceptedBundles('origin/main', folder).keys()], ['tool:scene_apply', 'result:scene_capabilities']);+ runGit(folder, 'checkout', '-q', 'grow-create');+ assert.deepEqual([...readAcceptedBundles('origin/main', folder).keys()], ['tool:scene_create', 'tool-total']);++ runGit(folder, 'checkout', '-q', 'main');+ runGit(folder, 'merge', '-q', '--no-ff', '-m', 'merge create', 'grow-create');+ runGit(folder, 'merge', '-q', '--no-ff', '-m', 'merge apply', 'grow-apply');+ const touched = runGit(folder, 'diff', '--name-only', 'origin/main', 'HEAD');+ assert.equal(touched.includes('bench/tool-cost-budgets.json'), false);+ } finally {+ rmSync(folder, { recursive: true, force: true });+ }+});diff --git a/tests/tool-cost.test.js b/tests/tool-cost.test.jsindex cbebe7e3..dd8c054b 100644--- a/tests/tool-cost.test.js+++ b/tests/tool-cost.test.js@@ -2,28 +2,24 @@ import test from 'node:test';import assert from 'node:assert/strict';import { readFileSync } from 'node:fs';import { measureToolCost } from '../scripts/measure-tool-cost.mjs';+import { ceilingProblems, costsFrom } from '../scripts/tool-cost-growth.mjs';const budgets = JSON.parse(readFileSync(new URL('../bench/tool-cost-budgets.json', import.meta.url), 'utf8'));const report = await measureToolCost();+const costs = costsFrom(report);-test('the tool schemas an agent reads on every turn stay under their total budget', () => {- assert.ok(report.schemaTotal.bytes <= budgets.schemaTotalBytes,- `tool schemas are ${report.schemaTotal.bytes} bytes, over the ${budgets.schemaTotalBytes} budget. Move rare detail into scene_capabilities or reference a shared schema; raise bench/tool-cost-budgets.json only for a feature that needs it.`);-});--test('each large tool schema and every small one stays under its own budget', () => {- for (const row of report.schemas) {- const budget = budgets.schemaToolBytes[row.name] ?? budgets.schemaOtherToolBytes;- assert.ok(row.bytes <= budget, `${row.name} is ${row.bytes} bytes, over its ${budget} budget.`);+// Growth against the base branch is checked by npm run tool-cost, and tests/tool-cost-growth.test.js tests its rules.+// The ceilings are release numbers: a pull request past one is reported by npm run tool-cost and never fails here.+test('every typical result has a release ceiling, and the budget file holds no history string', () => {+ for (const name of Object.keys(costs).filter(key => key.startsWith('result:'))) {+ const ceiling = budgets.resultBytes[name.slice('result:'.length)];+ assert.ok(ceiling !== undefined, `${name} has no release ceiling; run npm run tool-cost:release.`);}+ assert.equal(budgets.measured, undefined, 'bench/tool-cost-budgets.json holds no history string. Git history keeps it.');});-test('typical results stay under their budgets', () => {- for (const row of report.results) {- const budget = budgets.resultBytes[row.name];- assert.ok(budget !== undefined, `${row.name} has no result budget; add one to bench/tool-cost-budgets.json.`);- assert.ok(row.bytes <= budget, `${row.name} result is ${row.bytes} bytes, over its ${budget} budget.`);- }+test('the release ceilings come back as a list of problems', () => {+ assert.ok(Array.isArray(ceilingProblems(costs, budgets)));});test('a small mesh change as a mesh_edit call is smaller than rewriting the recipe', () => {