Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Measure tool cost against the base branch instead of a shared budget file

Nguyen Xuan Anc2d5ce37797f
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 8811ea8a..57a1809d 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -33,8 +33,9 @@ jobs:
- uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: node_modules/.cache/size-base
- key: size-base-${{ steps.base.outputs.sha }}-${{ hashFiles('scripts/measure-size.mjs', 'scripts/size-*.mjs', 'package-lock.json') }}
+ key: size-base-${{ steps.base.outputs.sha }}-${{ hashFiles('scripts/measure-size.mjs', 'scripts/size-*.mjs', 'scripts/*tool-cost*.mjs', 'package-lock.json') }}
- run: npm run size -- --base "$SIZE_BASE_REF"
+ - run: npm run tool-cost -- --base "$SIZE_BASE_REF"
- run: npm test
- run: npm run test:isolated
- run: npm run prepare:release
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 9ccc683d..f5ed9676 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -35,7 +35,9 @@ Run `npm run smoke` or the `smoke:*` script for the area that you changed, and l
A change to skinning, glTF loading, poses or IK also needs `npm run smoke:character`, which takes a real character through the MCP server. Run `npm run prepare:release` first.
Screenshots and other output go to `artifacts/`, which Git ignores.
-Every pull request runs `npm audit`, `npm run size`, `npm test`, `npm run test:isolated` (each test file in its own process), `prepare:release`, `smoke:package` and `smoke:mcp-package` in the `Test` workflow. `smoke:mcp-package` draws through the MCP server on SwiftShader in Chrome. `npm run size` compares every bundle with `origin/main`. If a bundle grows past its allowance (64 bytes gzip for core and the scenes), add `changes/<issue>.size.md` with the line `- entry:core: <why it is worth the bytes>` and say why in the pull request. Never edit `bench/size/budgets.json` or `baseline.json` in a pull request. `bench/size/README.md` has the details.
+Every pull request runs `npm audit`, `npm run size`, `npm run tool-cost`, `npm test`, `npm run test:isolated` (each test file in its own process), `prepare:release`, `smoke:package` and `smoke:mcp-package` in the `Test` workflow. `smoke:mcp-package` draws through the MCP server on SwiftShader in Chrome. `npm run size` compares every bundle with `origin/main`. If a bundle grows past its allowance (64 bytes gzip for core and the scenes), add `changes/<issue>.size.md` with the line `- entry:core: <why it is worth the bytes>` and say why in the pull request. Never edit `bench/size/budgets.json` or `baseline.json` in a pull request. `bench/size/README.md` has the details.
+
+`npm run tool-cost` does the same for what the MCP tools cost an agent. If a tool schema grows by more than 64 bytes, a typical result by more than 128 bytes, or the schema total by more than 128 bytes, first move rare detail into `scene_capabilities` or a shared schema. If the growth is worth it, add a line to `changes/<issue>.size.md`, such as `- tool:scene_create: <why it is worth the bytes>`, `- tool-total: <why>` or `- result:scene_capabilities: <why>`. Never edit `bench/tool-cost-budgets.json` in a pull request. `bench/tool-cost/README.md` has the details.
Branch protection on `main` requires the `test` check to pass before a merge. Merge a pull request only when `GitGuardian Security Checks` also passes.
@@ -75,7 +77,7 @@ Keep site assets out of the npm package: the `files` list in `package.json` deci
Maintainers make releases. `package.json` holds the version, and the publish workflow runs when it changes on `main`.
1. Update the version in `package.json` and in `src/version.js`, then run `npm run changes:fold`. It writes the files in `changes/` into `CHANGELOG.md` as that version's section and deletes them. Read `npm run changes` first to see what it will fold. It also deletes the `changes/*.size.md` notes.
-2. Run `npm run size:release`. It resets the size budgets and the baseline to the release sizes. Read the diff of `bench/size/budgets.json`.
+2. Run `npm run size:release` and `npm run tool-cost:release`. They reset the size budgets, the baseline and the tool cost ceilings to the release numbers. Read the diffs of `bench/size/budgets.json` and `bench/tool-cost-budgets.json`.
3. Run `npm test`, `npm run build` and `npm run smoke:package`.
4. Commit and push to `main`.
5. Wait for the `Publish 67js` workflow. It tests the package and publishes the new version through the npm trusted publisher.
diff --git a/bench/tool-cost-budgets.json b/bench/tool-cost-budgets.json
index 7325ffa9..553b44d0 100644
--- a/bench/tool-cost-budgets.json
+++ b/bench/tool-cost-budgets.json
@@ -1,5 +1,4 @@
{
- "measured": "2026-10-31, from scripts/measure-tool-cost.mjs; raised by 800 total and 150 for scene_apply for branch and labels fields (no new tool); raised by 650 total and 80 for scene_apply for clip layers, bone masks and retargeting through MCP; raised by 20 for scene_create and 20 for the scene_capabilities result after merging the expression language with the material graph; was 373388 schema bytes before shared IDs and recipe references; trimmed after the render-passes merge: material graph, depth of field, motion blur and passes details moved to scene_capabilities, budgets set to the measured value plus 0.5% (total 188267, scene_capabilities result 5167); raised by 95 for scene_apply and 690 for the scene_capabilities result for node relations; curve and text geometry entries shortened to a pointer at scene_capabilities curveText, budgets set to the measured value plus 0.5% (total 189468, scene_capabilities result 6399); then six modifier ops (weld, triangulate, edge_split, smooth, simple_deform, cast) added as one compact schema entry with details in scene_capabilities meshModifiers, entry and mesh_stack description shortened: total 190382 (budget unchanged at 190420), mesh_edit 34216 (budget 34220), scene_capabilities result 7042 (budget 7050); mesh_stack 11682 (budget 11690); after merging imported mesh topology with the modifier ops: raised the total to 190480 (measured 190471), mesh_edit to 34300 (measured 34293) and the scene_capabilities result to 7520 (measured 7512) for imported mesh edits; then cloth (recipe cloth, set-cloth, grid geometry) added with the total unchanged at 190480 (measured 189159): physics descriptions shortened and set-physics now takes the same record reference as set-motion, scene_create raised to 62150 and scene_apply lowered to 44130, scene_capabilities result raised to 8190 (measured 8176) for the cloth details; then particles and hair (recipe `particles`, set-particles) added with the total unchanged at 190480 (measured 189642, scene_create raised to 62350 and scene_apply to 44370 for the recipe particles reference and set-particles): the section is one record reference described in scene_capabilities, so the scene_capabilities result rose to 9570 (measured 9523) for the particle and hair fields; after merging the film timeline (recipe timeline, set-timeline) with particles: total 190719 (budget 190730), scene_create 62464, scene_apply 44620, scene_capabilities result 9902; then the path tracer renderer setting (drop z.int bounds, integers still checked by the job validator): total raised to 190830 (measured 190818); after merging the render limits and light bake with the path tracer: measured total 190013, scene_create raised to 62680 (measured 62667), scene_apply to 44860 (measured 44852) and the scene_capabilities result to 10620 (measured 10613) for the bake section and job size limits; then material layers anisotropy and iridescence (issue #51): the 16 layer fields take one short line each in the schema and their ranges and notes moved to scene_capabilities scene.materialLayers, so the scene_capabilities result rose to 12230 (measured 12168); then the worker bake limit added workerMaxSeconds to the bake limits (result 13361, budget raised to 13370); then the sculpt and shrinkwrap mesh edits (issue #47) with the total unchanged at 190830 (measured 190612): the face rule and tag prefix descriptions shortened (the face rule text repeats in every select field), so mesh_edit rose to 33980 (measured 33960) and mesh_stack fell to 11030 (measured 11014), and the scene_capabilities result rose to 13350 (measured 13338) for the meshSculpt notes; then recipe light links and shadow cascades (issue #52): environment fields added to the shared environment schema with the total unchanged at 190830 (measured 190507) by shortening its descriptions, and the scene_capabilities result raised to 13870 (measured 13865) for lightLinking and shadowCascades; then the nine remaining modifiers (issue #42: screw, skin, wireframe, lattice, curve, hook, wave, weighted_normal, smooth_by_angle) added their notes to meshModifiers and a free modifier record replaced the modifier schema entry, so the schemas shrank and the scene_capabilities result rose to 15900 (measured 15814); path query of scene_inspect (issue #64): one record field of 205 bytes in the scene_inspect schema (total 190732, within its budget) and the navigation notes in scene_capabilities, so the result rose to 14500 (measured 14494); reflection probes (issue #55): the bake.probes line, and the scene_capabilities result raised to 16600 (measured 16541) after merging with main; volumeFog shadowAmbient (issue #165): scene_create raised to 62945 (measured 62940) and scene_apply to 45040 (measured 45037); keyframe easing (issue #286): one easing string field in each of the motion track schemas, scene_create raised to 63450 (measured 63444) and the total to 191000 (measured 190994); glTF import line gltfImport (issue #280): the scene_capabilities result raised to 16701 (measured 16681); depth bias line depthBias (issue #310): the scene_capabilities result raised to 17000 (measured 16979); view transforms and looks (issue #282): one viewTransform line in scene_capabilities, so the result is raised to 17020 (measured 16996); both depthBias and viewTransform lines together: raised to 17320 (measured 17292); outlines and toon light (issue #285), after merging depthBias and view transforms: environment.lines is one free record and material toon one short line, total 192150 (measured 192127), scene_create 63800 (measured 63778), scene_apply 45480 (measured 45463), scene_create_asset 19500 (measured 19480), and the scene_capabilities result 17670 (measured 17647) for the lineArt entry",
"schemaTotalBytes": 192150,
"schemaToolBytes": {
"scene_create": 63800,
diff --git a/bench/tool-cost/README.md b/bench/tool-cost/README.md
new file mode 100644
index 00000000..19ffb928
--- /dev/null
+++ b/bench/tool-cost/README.md
@@ -0,0 +1,44 @@
+# Tool cost
+
+Every MCP tool schema is read on every turn, so their size is a cost to each agent. `npm run tool-cost` measures three kinds of bytes and compares them with the same numbers on the base branch. The `Test` workflow runs it on every pull request. A pull request never edits a tool cost number, so two pull requests do not conflict over one.
+
+## What it measures
+
+- `tool:<name>`: the description and input schema of one tool.
+- `tool-total`: all tool schemas together.
+- `result:<name>`: a typical result of one tool, from `scripts/measure-tool-cost.mjs`.
+
+`node scripts/measure-tool-cost.mjs` prints the full report with estimated tokens.
+
+## How the check decides
+
+The check measures the base branch with its own copy of the script, then compares each cost. It reuses the machinery of the size check: the same base ref, the same cache by base commit and the same notes. On your machine the base is `origin/main`, from the commit where your branch left it.
+
+A cost fails when it grows by more than its allowance:
+
+- A tool schema: 64 bytes.
+- A typical result: 128 bytes.
+- The schema total: 128 bytes.
+
+The allowance is about one short schema field. A change that adds a field to one or two tools passes. A new cost that the base does not have is reported and never fails. A cost that shrinks never fails.
+
+## When a check fails
+
+1. Move rare detail into `scene_capabilities`, or reference a shared schema. This is the best fix, because the schemas are read on every turn and `scene_capabilities` is read once.
+2. If the growth is worth its cost, add `changes/<issue>.size.md` to your pull request, with one line for each cost that you accept:
+
+```md
+- tool:scene_create: the new environment field costs 90 bytes
+- tool-total: the same field
+- result:scene_capabilities: its notes are 140 bytes
+```
+
+3. State the same reason in the pull request.
+
+The check reads only the notes that your branch adds, so a note from another pull request never accepts your growth. Your note is a new file, so it conflicts with nobody. A size note can hold bundle lines and tool cost lines together.
+
+## Release ceilings
+
+`bench/tool-cost-budgets.json` holds the absolute ceilings. They are reported on a pull request and enforced with `npm run tool-cost -- --absolute`. A release refreshes them with `npm run tool-cost:release`, which sets each ceiling 2% over the measured value. Read the diff, because a reset hides growth since the last reset.
+
+The file once held a `measured` note that every pull request extended. It is gone, and `git log -p bench/tool-cost-budgets.json` keeps the history.
diff --git a/changes/345.md b/changes/345.md
new file mode 100644
index 00000000..6059efaa
--- /dev/null
+++ b/changes/345.md
@@ -0,0 +1 @@
+- Measure tool schema and result bytes against the base branch with `npm run tool-cost` (issue #345). A pull request that grows a tool by more than 64 bytes, a result by more than 128 or the total by more than 128 accepts it in its own `changes/<issue>.size.md`, so `bench/tool-cost-budgets.json` no longer conflicts. The `measured` history string is gone and the absolute budgets are release ceilings, refreshed by `npm run tool-cost:release`.
diff --git a/changes/README.md b/changes/README.md
index 2fb910ee..d8f930f4 100644
--- a/changes/README.md
+++ b/changes/README.md
@@ -10,4 +10,4 @@ Do not edit `CHANGELOG.md` in a pull request. Two pull requests then never touch
`npm run changes` prints the pending notes. At release time `npm run changes:fold` writes them into `CHANGELOG.md` as the section for the version in `package.json` and deletes the files.
-A size note, `changes/<issue>.size.md`, is not a changelog bullet. It accepts a bundle size increase, and `bench/size/README.md` says how. `changes:fold` leaves it out of the changelog and deletes it.
+A size note, `changes/<issue>.size.md`, is not a changelog bullet. It accepts a bundle size increase, and `bench/size/README.md` says how. `changes:fold` leaves it out of the changelog and deletes it. The same note accepts a tool cost increase, with lines such as `- tool:scene_create: <reason>`, and `bench/tool-cost/README.md` says how.
diff --git a/docs/mcp.md b/docs/mcp.md
index a1b82ff3..6e910a21 100644
--- a/docs/mcp.md
+++ b/docs/mcp.md
@@ -149,7 +149,7 @@ Every error an agent can reach names what failed and what to do next. `tests/too
### Tool cost and diagnostics
-Every tool schema is read on every turn, so `node scripts/measure-tool-cost.mjs` reports the bytes and estimated tokens of each schema and of each typical result, and `tests/tool-cost.test.js` fails when one passes `bench/tool-cost-budgets.json`. The schemas total 173,216 bytes (about 49,500 tokens). To save space, a mesh geometry's `edits` and `stack` in a recipe, and `motion` in `scene_apply` and asset recipes, are advertised as plain objects: the full field list is in `mesh_edit`, `mesh_stack` and `scene_create`, and the backend still validates every field and names the valid choices when one is wrong. A small mesh change is far cheaper as `mesh_edit` (187 bytes in the measured case) than as a whole recipe rewrite (1,254 bytes with the dispose).
+Every tool schema is read on every turn, so `node scripts/measure-tool-cost.mjs` reports the bytes and estimated tokens of each schema and of each typical result, and `npm run tool-cost` fails a pull request when one grows past its allowance against the base branch (`bench/tool-cost/README.md`). The schemas total 173,216 bytes (about 49,500 tokens). To save space, a mesh geometry's `edits` and `stack` in a recipe, and `motion` in `scene_apply` and asset recipes, are advertised as plain objects: the full field list is in `mesh_edit`, `mesh_stack` and `scene_create`, and the backend still validates every field and names the valid choices when one is wrong. A small mesh change is far cheaper as `mesh_edit` (187 bytes in the measured case) than as a whole recipe rewrite (1,254 bytes with the dispose).
`mesh_inspect` lint adds warnings for `intersecting-parts`, `self-intersection`, `thin-wall`, `floating-part` and `asymmetric`. They never set `ok` to false, run within face budgets, and list any check they skipped in `skippedChecks`. `scene_inspect` returns a `diagnostics` list only when there is something to say: ignored material fields, blended transmission, unused materials or geometries, and for imported assets the decoder's not-drawn features, rigs with no joints or no skin, and assets with no clips. Each item has `kind`, `severity`, `target`, `message` and `fix`.
diff --git a/package.json b/package.json
index 76ac02c5..d050d236 100644
--- a/package.json
+++ b/package.json
@@ -149,6 +149,8 @@
"reference": "node bench/reference/run.mjs",
"size": "node scripts/measure-size.mjs",
"size:release": "node scripts/measure-size.mjs --set-budgets",
+ "tool-cost": "node scripts/check-tool-cost.mjs",
+ "tool-cost:release": "node scripts/check-tool-cost.mjs --set-budgets",
"benchmark:runtime": "node bench/runtime/run.mjs",
"benchmark:crowd": "node scripts/benchmark-crowd.mjs",
"check:tool-schemas": "node scripts/check-tool-schemas.mjs",
diff --git a/scripts/check-tool-cost.mjs b/scripts/check-tool-cost.mjs
new file mode 100644
index 00000000..552f2bcf
--- /dev/null
+++ b/scripts/check-tool-cost.mjs
@@ -0,0 +1,73 @@
+// Checks what the MCP tools cost an agent: npm run tool-cost
+// A pull request fails when a tool schema, the schema total or a typical result grows past its allowance against the base branch
+// (--base, default origin/main) unless its own changes/*.size.md accepts it. --absolute also enforces the release ceilings in
+// bench/tool-cost-budgets.json. --set-budgets refreshes the ceilings at release.
+import { readFile, writeFile } from 'node:fs/promises';
+import { fileURLToPath } from 'node:url';
+import { measureToolCost } from './measure-tool-cost.mjs';
+import { DEFAULT_BASE_REF, baseRefExists, measureBase, readAcceptedBundles } from './size-base.mjs';
+import { budgetsFrom, ceilingProblems, compareCostsToBase, costsFrom, formatCostRows } from './tool-cost-growth.mjs';
+
+const BUDGETS_PATH = 'bench/tool-cost-budgets.json';
+// The measuring script imports the source beside itself, so the base branch runs its own copy of it.
+const TOOL_COST_METHOD = {
+ script: fileURLToPath(new URL('./measure-tool-cost.mjs', import.meta.url)),
+ runInBase: true,
+ files: ['measure-tool-cost.mjs', 'check-tool-cost.mjs', 'tool-cost-growth.mjs', 'size-base.mjs'],
+};
+
+const argumentsGiven = process.argv.slice(2);
+const flags = new Set(argumentsGiven);
+
+function valueAfter(flag) {
+ const index = argumentsGiven.indexOf(flag);
+ return index === -1 ? undefined : argumentsGiven[index + 1];
+}
+
+const requestedBase = valueAfter('--base');
+const baseRef = requestedBase ?? DEFAULT_BASE_REF;
+if (requestedBase && !baseRefExists(baseRef)) {
+ console.error(`The base ref ${baseRef} does not exist. Fetch it first, for example: git fetch origin main`);
+ process.exit(1);
+}
+const refreshing = flags.has('--set-budgets');
+const checkGrowth = baseRefExists(baseRef) && !refreshing;
+const baseMeasurement = checkGrowth ? measureBase(baseRef, TOOL_COST_METHOD) : null;
+const measured = costsFrom(await measureToolCost());
+const budgets = JSON.parse(await readFile(BUDGETS_PATH, 'utf8'));
+const problems = [];
+
+if (refreshing) {
+ await writeFile(BUDGETS_PATH, `${JSON.stringify(budgetsFrom(measured, budgets), null, 2)}\n`);
+ console.log(`Wrote the release ceilings to ${BUDGETS_PATH}.`);
+ process.exit(0);
+}
+
+if (checkGrowth) {
+ const accepted = readAcceptedBundles(baseRef);
+ const growth = compareCostsToBase(measured, costsFrom(await baseMeasurement), accepted);
+ console.log(`Tool cost against ${baseRef}, in bytes:`);
+ console.log(formatCostRows(growth.rows));
+ for (const [name, reason] of accepted) {
+ if (name in measured) console.log(`Accepted: ${name}, ${reason}`);
+ }
+ problems.push(...growth.problems);
+} else {
+ console.log(`No ${baseRef} to compare with, so the growth check is skipped. Fetch it, or pass --base <ref>.`);
+}
+
+// The absolute budgets are the release ceilings. They are reported on a pull request and enforced with --absolute.
+const overCeiling = ceilingProblems(measured, budgets);
+if (overCeiling.length) {
+ if (flags.has('--absolute')) {
+ problems.push(...overCeiling);
+ } else {
+ console.log(`\n${overCeiling.length} cost(s) are past their release ceiling. A release refreshes them with npm run tool-cost:release. List them with npm run tool-cost -- --absolute.`);
+ }
+}
+if (problems.length) {
+ console.error(`\n${problems.length} tool cost problem(s):`);
+ for (const problem of problems) console.error(`- ${problem}`);
+ process.exit(1);
+}
+console.log('\nNo tool cost grew past its allowance.');
diff --git a/scripts/size-base.mjs b/scripts/size-base.mjs
index 00b03b85..cf2e143d 100644
--- a/scripts/size-base.mjs
+++ b/scripts/size-base.mjs
@@ -3,7 +3,7 @@ import { execFile, spawnSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
-import { join, resolve } from 'node:path';
+import { basename, join, resolve } from 'node:path';
import { promisify } from 'node:util';
import { fileURLToPath } from 'node:url';
import { parseSizeNote } from './size-growth.mjs';
@@ -11,8 +11,9 @@ import { parseSizeNote } from './size-growth.mjs';
export const DEFAULT_BASE_REF = 'origin/main';
const SIZE_NOTE_PATTERN = /^changes\/[^/]+\.size\.md$/;
const SCRIPTS_FOLDER = fileURLToPath(new URL('.', import.meta.url));
-const MEASURE_SCRIPT = join(SCRIPTS_FOLDER, 'measure-size.mjs');
-const METHOD_FILES = ['measure-size.mjs', 'size-base.mjs', 'size-budget.mjs', 'size-growth.mjs'];
+// A measuring method names the script that prints JSON for --json and the files that decide its numbers. A script that imports
+// the source relative to itself runs from the base checkout (runInBase), so it measures the base source.
+export const SIZE_METHOD = { script: join(SCRIPTS_FOLDER, 'measure-size.mjs'), runInBase: false, files: ['measure-size.mjs', 'size-base.mjs', 'size-budget.mjs', 'size-growth.mjs'] };
// CI caches this folder, keyed by the base commit, so a run on an unchanged base skips the second measurement.
export const BASE_CACHE_FOLDER = 'node_modules/.cache/size-base';
const MAX_OUTPUT_BYTES = 256 * 1024 * 1024;
@@ -50,15 +51,16 @@ export function readAcceptedBundles(baseRef, cwd = process.cwd()) {
}
// The cache key holds everything that decides the base numbers: the base commit, the measuring method and the lock file.
-function cacheFileFor(baseRef) {
+function cacheFileFor(baseRef, method) {
const hash = createHash('sha256');
hash.update(mergeBaseOf(baseRef));
- for (const file of METHOD_FILES) hash.update(readFileSync(join(SCRIPTS_FOLDER, file)));
+ hash.update(basename(method.script));
+ for (const file of method.files) hash.update(readFileSync(join(SCRIPTS_FOLDER, file)));
if (existsSync('package-lock.json')) hash.update(readFileSync('package-lock.json'));
return join(BASE_CACHE_FOLDER, `${hash.digest('hex').slice(0, 24)}.json`);
}
-async function measureCheckout(baseRef) {
+async function measureCheckout(baseRef, method) {
const folder = mkdtempSync(join(tmpdir(), 'size-base-'));
const checkout = join(folder, 'base');
try {
@@ -70,7 +72,7 @@ async function measureCheckout(baseRef) {
const extracted = spawnSync('git', ['checkout-index', '--all', '--force', `--prefix=${checkout}/`], { encoding: 'utf8', windowsHide: true, env: environment });
if (extracted.status !== 0) throw new Error(`Could not unpack ${baseRef}: ${extracted.stderr}`);
if (existsSync('node_modules')) symlinkSync(resolve('node_modules'), join(checkout, 'node_modules'), 'junction');
- const { stdout } = await execFileAsync(process.execPath, [MEASURE_SCRIPT, '--json'], { cwd: checkout, maxBuffer: MAX_OUTPUT_BYTES, windowsHide: true });
+ const { stdout } = await execFileAsync(process.execPath, [method.runInBase ? join(checkout, 'scripts', basename(method.script)) : method.script, '--json'], { cwd: checkout, maxBuffer: MAX_OUTPUT_BYTES, windowsHide: true });
return JSON.parse(stdout);
} finally {
rmSync(join(checkout, 'node_modules'), { force: true });
@@ -80,10 +82,10 @@ async function measureCheckout(baseRef) {
// Measures the base branch with this checkout's script, so both sides use the same method. It runs beside the
// measurement of the pull request, and a cached result for the same base commit skips it.
-export async function measureBase(baseRef) {
- const cacheFile = cacheFileFor(baseRef);
+export async function measureBase(baseRef, method = SIZE_METHOD) {
+ const cacheFile = cacheFileFor(baseRef, method);
if (existsSync(cacheFile)) return JSON.parse(readFileSync(cacheFile, 'utf8'));
- const measured = await measureCheckout(baseRef);
+ const measured = await measureCheckout(baseRef, method);
mkdirSync(BASE_CACHE_FOLDER, { recursive: true });
writeFileSync(cacheFile, JSON.stringify(measured));
return measured;
diff --git a/scripts/tool-cost-growth.mjs b/scripts/tool-cost-growth.mjs
new file mode 100644
index 00000000..9e55558a
--- /dev/null
+++ b/scripts/tool-cost-growth.mjs
@@ -0,0 +1,92 @@
+// The growth gate for what MCP tools cost an agent. A pull request is measured against the base branch, so no pull request
+// edits a number that another one edits. It reuses the size notes: acceptInstruction comes from size-growth.mjs.
+import { acceptInstruction } from './size-growth.mjs';
+
+export const TOOL_ALLOWANCE_BYTES = 64;
+export const RESULT_ALLOWANCE_BYTES = 128;
+export const TOTAL_ALLOWANCE_BYTES = 128;
+export const TOTAL_NAME = 'tool-total';
+const BUDGET_MARGIN = 1.02;
+const BUDGET_ROUNDING_BYTES = 10;
+const MOVE_ADVICE = 'Move rare detail into scene_capabilities or reference a shared schema.';
+
+/** Flattens a report from measureToolCost() into named byte counts: tool:<name>, tool-total and result:<name>. */
+export function costsFrom(report) {
+ const costs = { [TOTAL_NAME]: report.schemaTotal.bytes };
+ for (const row of report.schemas) costs[`tool:${row.name}`] = row.bytes;
+ for (const row of report.results) costs[`result:${row.name}`] = row.bytes;
+ return costs;
+}
+
+export function costAllowance(name) {
+ if (name === TOTAL_NAME) return TOTAL_ALLOWANCE_BYTES;
+ return name.startsWith('result:') ? RESULT_ALLOWANCE_BYTES : TOOL_ALLOWANCE_BYTES;
+}
+
+/** Compares each cost with the same cost on the base branch. A cost the base does not have is reported, not failed. */
+export function compareCostsToBase(measured, baseMeasured, accepted = new Map()) {
+ const rows = [];
+ const problems = [];
+ for (const [name, bytes] of Object.entries(measured)) {
+ const base = baseMeasured[name];
+ const growth = base === undefined ? null : bytes - base;
+ const allowance = base === undefined ? null : costAllowance(name);
+ const isAccepted = accepted.has(name);
+ rows.push({ name, bytes, base: base ?? null, growth, allowance, accepted: isAccepted });
+ if (growth !== null && growth > allowance && !isAccepted) {
+ problems.push(`${name} grew by ${growth} bytes against the base branch (${base} to ${bytes}), over its allowance of ${allowance}. ${MOVE_ADVICE} ${acceptInstruction(name)}`);
+ }
+ }
+ return { rows, problems };
+}
+
+function noteFor(row) {
+ if (row.growth === null) return 'new, reported only';
+ if (row.growth <= row.allowance) return '';
+ return row.accepted ? 'accepted' : 'OVER';
+}
+
+/** Prints the costs that moved. An unchanged cost adds nothing, so the table stays short. */
+export function formatCostRows(rows) {
+ const moved = rows.filter(row => row.growth !== 0);
+ if (!moved.length) return 'No tool schema or result changed against the base branch.';
+ const header = `${'cost'.padEnd(34)}${'bytes'.padStart(9)}${'base'.padStart(9)}${'growth'.padStart(9)}${'allowed'.padStart(9)} note`;
+ const lines = moved.map(row => [
+ row.name.padEnd(34), String(row.bytes).padStart(9), String(row.base ?? '-').padStart(9),
+ String(row.growth ?? '-').padStart(9), String(row.allowance ?? '-').padStart(9), ` ${noteFor(row)}`,
+ ].join(''));
+ return [header, ...lines].join('\n');
+}
+
+function ceilingFor(bytes) {
+ return Math.ceil((bytes * BUDGET_MARGIN) / BUDGET_ROUNDING_BYTES) * BUDGET_ROUNDING_BYTES;
+}
+
+/** Release ceilings: 2% over what is measured now. Only the large tools get their own line, the rest share schemaOtherToolBytes. */
+export function budgetsFrom(measured, previous) {
+ const budgets = { ...previous, schemaToolBytes: {}, resultBytes: {} };
+ budgets.schemaTotalBytes = ceilingFor(measured[TOTAL_NAME]);
+ for (const [name, bytes] of Object.entries(measured)) {
+ if (name.startsWith('tool:') && bytes > previous.schemaOtherToolBytes) budgets.schemaToolBytes[name.slice('tool:'.length)] = ceilingFor(bytes);
+ if (name.startsWith('result:')) budgets.resultBytes[name.slice('result:'.length)] = ceilingFor(bytes);
+ }
+ return budgets;
+}
+
+/** The release ceilings in bench/tool-cost-budgets.json that the measured costs are past, as problems. */
+export function ceilingProblems(measured, budgets) {
+ const problems = [];
+ const check = (name, bytes, ceiling) => {
+ if (bytes > ceiling) problems.push(`${name} is ${bytes} bytes, over its release ceiling of ${ceiling} by ${bytes - ceiling}. ${MOVE_ADVICE}`);
+ };
+ check(TOTAL_NAME, measured[TOTAL_NAME], budgets.schemaTotalBytes);
+ for (const [name, bytes] of Object.entries(measured)) {
+ if (name.startsWith('tool:')) check(name, bytes, budgets.schemaToolBytes[name.slice('tool:'.length)] ?? budgets.schemaOtherToolBytes);
+ if (name.startsWith('result:')) {
+ const ceiling = budgets.resultBytes[name.slice('result:'.length)];
+ if (ceiling === undefined) problems.push(`${name} has no release ceiling. A release adds one with npm run tool-cost:release.`);
+ else check(name, bytes, ceiling);
+ }
+ }
+ return problems;
+}
diff --git a/tests/tool-cost-growth.test.js b/tests/tool-cost-growth.test.js
new file mode 100644
index 00000000..cf04fa19
--- /dev/null
+++ b/tests/tool-cost-growth.test.js
@@ -0,0 +1,107 @@
+import assert from 'node:assert/strict';
+import { spawnSync } from 'node:child_process';
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import test from 'node:test';
+import { readAcceptedBundles } from '../scripts/size-base.mjs';
+import { budgetsFrom, ceilingProblems, compareCostsToBase, costAllowance, costsFrom } from '../scripts/tool-cost-growth.mjs';
+
+const report = {
+ schemaTotal: { bytes: 20000 },
+ schemas: [{ name: 'scene_create', bytes: 12000 }, { name: 'scene_apply', bytes: 8000 }],
+ results: [{ name: 'scene_capabilities', bytes: 300 }],
+};
+
+test('costs are named tool:<name>, tool-total and result:<name>', () => {
+ assert.deepEqual(costsFrom(report), { 'tool-total': 20000, 'tool:scene_create': 12000, 'tool:scene_apply': 8000, 'result:scene_capabilities': 300 });
+});
+
+test('a tool gets 64 bytes, a result and the total 128', () => {
+ assert.equal(costAllowance('tool:scene_create'), 64);
+ assert.equal(costAllowance('result:scene_capabilities'), 128);
+ assert.equal(costAllowance('tool-total'), 128);
+});
+
+test('a tool that grows past the allowance fails with the move advice and the accept instruction', () => {
+ const { problems } = compareCostsToBase({ 'tool:scene_create': 700 }, { 'tool:scene_create': 600 });
+ assert.equal(problems.length, 1);
+ assert.match(problems[0], /tool:scene_create grew by 100 bytes against the base branch \(600 to 700\), over its allowance of 64/);
+ assert.match(problems[0], /Move rare detail into scene_capabilities or reference a shared schema/);
+ assert.match(problems[0], /Reduce it, or accept it: add changes\/<issue>\.size\.md/);
+ assert.match(problems[0], /- tool:scene_create: <why it is worth the bytes>/);
+});
+
+test('growth inside the allowance, a shrink and a new cost pass', () => {
+ const measured = { 'tool:scene_create': 664, 'tool:scene_apply': 300, 'tool:fresh': 90 };
+ const base = { 'tool:scene_create': 600, 'tool:scene_apply': 400 };
+ const { rows, problems } = compareCostsToBase(measured, base);
+ assert.deepEqual(problems, []);
+ assert.deepEqual(rows.map(row => row.growth), [64, -100, null]);
+});
+
+test('a note accepts only the costs it names', () => {
+ const measured = { 'tool-total': 1500, 'tool:scene_create': 700, 'result:scene_capabilities': 600 };
+ const base = { 'tool-total': 1000, 'tool:scene_create': 600, 'result:scene_capabilities': 300 };
+ const accepted = new Map([['tool:scene_create', 'a new field'], ['tool-total', 'the same field']]);
+ const { problems } = compareCostsToBase(measured, base, accepted);
+ assert.equal(problems.length, 1);
+ assert.match(problems[0], /result:scene_capabilities grew by 300/);
+});
+
+test('release ceilings are listed as problems, and a release resets them', () => {
+ const budgets = { schemaTotalBytes: 19000, schemaToolBytes: { scene_create: 12500 }, schemaOtherToolBytes: 4000, resultBytes: { scene_capabilities: 320 } };
+ const measured = costsFrom(report);
+ const problems = ceilingProblems(measured, budgets);
+ assert.equal(problems.length, 2);
+ assert.match(problems[0], /tool-total is 20000 bytes, over its release ceiling of 19000 by 1000/);
+ assert.match(problems[1], /tool:scene_apply is 8000 bytes, over its release ceiling of 4000 by 4000/);
+ const reset = budgetsFrom(measured, budgets);
+ assert.deepEqual(ceilingProblems(measured, reset), []);
+ assert.equal(reset.schemaOtherToolBytes, 4000);
+});
+
+function runGit(folder, ...args) {
+ const identity = ['-c', 'user.name=Test', '-c', '[email protected]', '-c', 'commit.gpgsign=false', '-c', 'core.autocrlf=false'];
+ const result = spawnSync('git', [...identity, ...args], { cwd: folder, encoding: 'utf8' });
+ assert.equal(result.status, 0, `${args.join(' ')}: ${result.stderr}`);
+ return result.stdout;
+}
+
+function commitFile(folder, path, text, message) {
+ mkdirSync(join(folder, path, '..'), { recursive: true });
+ writeFileSync(join(folder, path), text);
+ runGit(folder, 'add', '.');
+ runGit(folder, 'commit', '-q', '-m', message);
+}
+
+test('two pull requests that grow different tools merge in sequence and neither touches the shared budget file', () => {
+ const folder = mkdtempSync(join(tmpdir(), 'tool-cost-sim-'));
+ try {
+ runGit(folder, 'init', '-q', '-b', 'main');
+ commitFile(folder, 'bench/tool-cost-budgets.json', '{\n "schemaTotalBytes": 1000\n}\n', 'budgets');
+ commitFile(folder, 'src/create.js', 'export const create = 1;\n', 'create');
+ commitFile(folder, 'src/apply.js', 'export const apply = 1;\n', 'apply');
+ runGit(folder, 'update-ref', 'refs/remotes/origin/main', 'HEAD');
+
+ runGit(folder, 'checkout', '-q', '-b', 'grow-create');
+ commitFile(folder, 'src/create.js', 'export const create = 2;\n', 'grow create');
+ commitFile(folder, 'changes/301.size.md', '- tool:scene_create: a new field\n- tool-total: the same field\n', 'accept create');
+ runGit(folder, 'checkout', '-q', 'main');
+ runGit(folder, 'checkout', '-q', '-b', 'grow-apply');
+ commitFile(folder, 'src/apply.js', 'export const apply = 2;\n', 'grow apply');
+ commitFile(folder, 'changes/302.size.md', '- tool:scene_apply: a new field\n- result:scene_capabilities: its notes\n', 'accept apply');
+
+ assert.deepEqual([...readAcceptedBundles('origin/main', folder).keys()], ['tool:scene_apply', 'result:scene_capabilities']);
+ runGit(folder, 'checkout', '-q', 'grow-create');
+ assert.deepEqual([...readAcceptedBundles('origin/main', folder).keys()], ['tool:scene_create', 'tool-total']);
+
+ runGit(folder, 'checkout', '-q', 'main');
+ runGit(folder, 'merge', '-q', '--no-ff', '-m', 'merge create', 'grow-create');
+ runGit(folder, 'merge', '-q', '--no-ff', '-m', 'merge apply', 'grow-apply');
+ const touched = runGit(folder, 'diff', '--name-only', 'origin/main', 'HEAD');
+ assert.equal(touched.includes('bench/tool-cost-budgets.json'), false);
+ } finally {
+ rmSync(folder, { recursive: true, force: true });
+ }
+});
diff --git a/tests/tool-cost.test.js b/tests/tool-cost.test.js
index cbebe7e3..dd8c054b 100644
--- a/tests/tool-cost.test.js
+++ b/tests/tool-cost.test.js
@@ -2,28 +2,24 @@ import test from 'node:test';
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import { measureToolCost } from '../scripts/measure-tool-cost.mjs';
+import { ceilingProblems, costsFrom } from '../scripts/tool-cost-growth.mjs';
const budgets = JSON.parse(readFileSync(new URL('../bench/tool-cost-budgets.json', import.meta.url), 'utf8'));
const report = await measureToolCost();
+const costs = costsFrom(report);
-test('the tool schemas an agent reads on every turn stay under their total budget', () => {
- assert.ok(report.schemaTotal.bytes <= budgets.schemaTotalBytes,
- `tool schemas are ${report.schemaTotal.bytes} bytes, over the ${budgets.schemaTotalBytes} budget. Move rare detail into scene_capabilities or reference a shared schema; raise bench/tool-cost-budgets.json only for a feature that needs it.`);
-});
-
-test('each large tool schema and every small one stays under its own budget', () => {
- for (const row of report.schemas) {
- const budget = budgets.schemaToolBytes[row.name] ?? budgets.schemaOtherToolBytes;
- assert.ok(row.bytes <= budget, `${row.name} is ${row.bytes} bytes, over its ${budget} budget.`);
+// Growth against the base branch is checked by npm run tool-cost, and tests/tool-cost-growth.test.js tests its rules.
+// The ceilings are release numbers: a pull request past one is reported by npm run tool-cost and never fails here.
+test('every typical result has a release ceiling, and the budget file holds no history string', () => {
+ for (const name of Object.keys(costs).filter(key => key.startsWith('result:'))) {
+ const ceiling = budgets.resultBytes[name.slice('result:'.length)];
+ assert.ok(ceiling !== undefined, `${name} has no release ceiling; run npm run tool-cost:release.`);
}
+ assert.equal(budgets.measured, undefined, 'bench/tool-cost-budgets.json holds no history string. Git history keeps it.');
});
-test('typical results stay under their budgets', () => {
- for (const row of report.results) {
- const budget = budgets.resultBytes[row.name];
- assert.ok(budget !== undefined, `${row.name} has no result budget; add one to bench/tool-cost-budgets.json.`);
- assert.ok(row.bytes <= budget, `${row.name} result is ${row.bytes} bytes, over its ${budget} budget.`);
- }
+test('the release ceilings come back as a list of problems', () => {
+ assert.ok(Array.isArray(ceilingProblems(costs, budgets)));
});
test('a small mesh change as a mesh_edit call is smaller than rewriting the recipe', () => {