Skip to content

armgit / kiln

Public Imported from GitHub
0

Build and revise procedural 3D assets with your coding agent. Local MCP server, CLI and TypeScript engine with rendered review, editable source and GLB export.

fix(coverage): give the ratchet a scope, and correct why it needed one (#93)

Matthew Kissingeree6d5d6effc2
diff --git a/AGENTS.md b/AGENTS.md
index 40461b18..f4f39ba8 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -88,7 +88,8 @@ do not lower them without an explicit measured rationale. Live model tests are o
`bun run test:live` and may spend money.
Tests and CI pin `KILN_RENDER=cpu`. The coverage ratchet must not vary by whether the runner has a
-GPU.
+GPU. It is measured over `src/` alone -- the shipped engine -- so nothing you change under
+`scripts/` can move it; those tests still run, and their correctness is their own assertions' job.
Use strict test-driven development for behavior changes: add a focused failing test, observe the
expected failure, implement the smallest fix, and rerun both the focused and full gates. Preserve
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5b67f1fd..0986736e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,35 @@
Changes to `@kiln/engine`. Source and installable packages are distributed through
GitHub. The package is not published on the npm registry.
+## The coverage ratchet has a scope now — 2026-09-12
+
+- The previous entry blamed 162 lines of lost coverage slack on helper scripts being
+ measured and never executed. The number was right; the mechanism was wrong. Bun
+ instruments only files it loads, and `coveragePathIgnorePatterns` already dropped
+ `**/*.test.mjs`, so those tools were never in the report — there is no `SF:` record
+ for `dispatch-asset`, `upload-posters`, `geometry-experiments` or `harness`.
+- Exactly **four** files under `scripts/` were instrumented, because tests import them:
+ `evaluation/observe-shipping.mjs`, `build-runtime.mjs`, `evaluation/conditions.ts` and
+ `authorship.ts`. 229 lines against the engine's 45,726 — 0.5% of the denominator, and
+ worth 0.10 points. Two of the four are dense single-expression files, so reflowing
+ them inflated their line counts while their covered lines stayed put. That is where
+ the 162 lines went.
+- So the finding is narrower and worse than "helper scripts dilute the ratchet": the
+ engine's coverage contract had a **repo-only formatting input**. A number this
+ repository enforces as policy could be moved by reformatting code that does not ship.
+- `bunfig.toml` now ignores `scripts/**` for coverage. Those tests still run — they are
+ the gates — but their sources leave the denominator, so the ratchet is a statement
+ about the shipped engine and nothing else. Re-measured on the narrowed scope: **95.39%
+ functions, 92.59% lines**.
+- Narrowing hands back 44 lines of slack no new test earned, so `lines` goes 92 → **92.1**
+ to hold the 224-line margin 13.3 chose rather than pocket it. `functions` stays at 94,
+ where the margin is unchanged in practice. Tightening further is a separate decision.
+- And the record carries its own scope now. `measuredBaseline` requires `measuredOver`,
+ the gate refuses a baseline without it, and it prints it: `Recorded baseline:
+ functions 95.39%, lines 92.59% (over src/, [email protected], 2026-09-12)`. A percentage
+ without its scope is as ambiguous as a threshold without a baseline — which is what
+ made the wrong explanation above plausible enough to write down.
+
## The repository's own gates are linted now — 2026-09-12
- Adding a file to `scripts/` and running `biome check` on its path printed "No files
@@ -19,13 +48,10 @@ GitHub. The package is not published on the npm registry.
- Left, with the cost measured rather than guessed. The rest of `scripts/` is one-off
tools, several authored as dense single-expression lines. Taking the whole directory
is 26 lint findings, a 1,530-line reflow (3,078 lines to 4,608, pure formatting), and
- **162 lines of coverage slack** — `test:coverage` measures `scripts`, those tools are
- never executed, so the reflow lands in the uncovered denominator and lines fall
- 92.49% to 92.14% against a threshold of 92. Measured by taking the full pass and
- running the gate.
-- Which says something about the ratchet worth recording: line coverage over manual
- helper scripts is noise inside a number this repository treats as a contract.
- Narrowing what `test:coverage` measures is the real fix, and a separate decision.
+ **162 lines of coverage slack** — lines fall 92.49% to 92.14% against a threshold of
+ 92. Measured by taking the full pass and running the gate.
+- **Corrected below**: the mechanism first given for that third cost was wrong, and it
+ pointed at the wrong fix.
- `render-service/` stays out for an unrelated reason: its sources are CRLF by
`.gitattributes`, and the formatter would rewrite the line endings.
- One incidental finding: Biome prints at most 20 diagnostics by default, so a count
diff --git a/bunfig.toml b/bunfig.toml
index 9e579b75..5658eb27 100644
--- a/bunfig.toml
+++ b/bunfig.toml
@@ -3,6 +3,15 @@ coverageSkipTestFiles = true
coverageReporter = ["text", "lcov"]
coverageDir = "coverage"
coveragePathIgnorePatterns = [
+ # The ratchet is a contract about the shipped engine, so it is measured over
+ # `src/` alone. `scripts/` is repo-only -- it is not in `files` or `exports`,
+ # and its correctness is asserted by its own tests, which still run here. Four
+ # of its files were being instrumented because a test imports them, 229 lines
+ # against the engine's 45,726, and that was enough for a FORMATTING change in a
+ # repo-only script to move the engine's coverage contract: reflowing two dense
+ # single-expression files took lines from 92.49% to 92.14%. A number treated as
+ # a contract should not have an input like that.
+ "scripts/**",
"**/*.test.ts",
"**/*.test.mjs",
"**/__tests__/**",
diff --git a/coverage-thresholds.json b/coverage-thresholds.json
index 73b162f5..9430470d 100644
--- a/coverage-thresholds.json
+++ b/coverage-thresholds.json
@@ -1,12 +1,13 @@
{
"measuredBaseline": {
- "functions": 95.27,
- "lines": 92.49,
+ "functions": 95.39,
+ "lines": 92.59,
+ "measuredOver": "src/",
"measuredUnder": "[email protected]",
"measuredOn": "2026-09-12"
},
"thresholds": {
"functions": 94,
- "lines": 92
+ "lines": 92.1
}
}
diff --git a/docs/plans/repo-size-and-r2-migration-2026-09-10.md b/docs/plans/repo-size-and-r2-migration-2026-09-10.md
index 01a0b073..c9b8db04 100644
--- a/docs/plans/repo-size-and-r2-migration-2026-09-10.md
+++ b/docs/plans/repo-size-and-r2-migration-2026-09-10.md
@@ -1600,7 +1600,18 @@ never by more than 41 levels, at a shadow edge, with a mean delta of 0.01--0.09.
There is nothing to choose between them by eye, which is what the 1:1 crop of the
pavilion's shadow edge confirms. `pcf` stays the shipped name.
-**Turning shadows on is renderer work, not a preset flag.** Four things stand in the
+**Decided 2026-09-12: the gallery does not cast shadows.** The owner's call, and the
+measurements support it. The site's `<Canvas>` already grounds an asset with drei
+`ContactShadows`, which runs its own depth pass and never consults
+`shadowMap.type`, so the surface a visitor actually looks at has the cue that
+matters. A directional shadow would add 2-6% of pixels at a maximum delta of
+27-54/255, from a shipped 3/4 view whose camera sits six degrees from the key light
+-- close to the worst angle for showing one -- in exchange for the four pieces of
+silently-failing renderer work below. **This row is closed, not deferred.** The
+analysis stays because it is the reason, and because anything that revisits the
+question starts from these four rather than rediscovering them.
+
+**Turning shadows on is renderer work, not a preset flag.** Four things stood in the
way, and the first three were each found by the probe producing no shadow at all:
1. **A receiver.** The service renders an asset against a flat background with no
@@ -1668,6 +1679,7 @@ the installed tree.
| 14.1 | Re-ground 7.12 / SEP-2640 against an authoritative source | **Done 2026-09-12.** Deferred still, for a stronger reason. See below |
| 14.2 | Establish what actually blocks the `ai` 7 family, and gate it | **Done 2026-09-12.** `@strands-agents/[email protected]` is latest, declares peer `@ai-sdk/provider: ^3.0.0`, and its `VercelModel` is typed on `LanguageModelV3` in 21 places. `@openrouter/[email protected]` requires `ai: ^7.0.0`; `[email protected]` depends on `@ai-sdk/[email protected]`. The family cannot be taken until Strands ships a release accepting the v4 spec -- no budget changes that. `scripts/peer-ranges.test.mjs` now fails on a peer range the version beside it does not meet, verified by patching the installed `@openrouter/ai-sdk-provider` manifest to peer `ai: ^7.0.0` and watching it report `installed 6.0.282` |
| 14.3 | Assert the prompt-cache breakpoint on the wire, offline | **Done 2026-09-12.** Both native transports captured in `src/agent/providers.test.ts`, both differential. See below |
+| 14.5 | Give the coverage ratchet a scope, so repo-only code cannot move the engine's contract | **Done 2026-09-12.** Measured over `src/` alone; baseline re-measured at 95.39% functions / 92.59% lines; `lines` raised 92 to 92.1 so the narrowing does not quietly hand back slack. See below |
| 14.4 | Bring the repository's own gates under the lint gate, which they were never under | **Done 2026-09-12** for the gates and their tests: 416 files checked where 405 were, and the tree reports nothing. The one-off tools beside them are measured and deliberately left, below |
### 14.1 -- the decisive fact is not the SEP's status
@@ -1771,17 +1783,27 @@ was one 1,300-character line before this pass. Taking the whole directory costs:
lines where the fix is unreviewable and the file is no more readable after.
- **A 1,530-line reflow.** `scripts/` goes from 3,078 lines to 4,608, a 50% growth
that is entirely formatting.
-- **162 lines of coverage slack.** `test:coverage` measures `src scripts`, the
- one-off tools are never executed by a test, so the reflow lands almost entirely
- in the uncovered denominator: lines fall 92.49% to 92.14% against a threshold of
- 92, leaving 62 lines of room where there were 224. Measured by taking the full
+- **162 lines of coverage slack.** Lines fall 92.49% to 92.14% against a threshold
+ of 92, leaving 62 lines of room where there were 224. Measured by taking the full
pass and running the gate, not estimated.
-That third cost is the interesting one, because it says something the ratchet has
-been quietly absorbing all along: line coverage over manual helper scripts is
-noise in a number the repository treats as a contract. Narrowing what
-`test:coverage` measures is the real fix and it is a separate decision, so it is
-recorded here rather than folded in.
+**Correction, 2026-09-12, to the sentence this row first carried.** It said the
+one-off tools are never executed so the reflow lands in the uncovered denominator.
+That number is right and that mechanism is wrong, and the wrong mechanism pointed
+at the wrong fix. Bun instruments only files it loads, and
+`coveragePathIgnorePatterns` already dropped `**/*.test.mjs`, so the one-off tools
+were **never in the report at all**: no `SF:` record for `dispatch-asset`,
+`promote-asset`, `upload-posters`, `curate-texture-library`, `geometry-experiments`
+or `harness`. Exactly four files under `scripts/` were instrumented, because tests
+import them -- `evaluation/observe-shipping.mjs`, `build-runtime.mjs`,
+`evaluation/conditions.ts` and `authorship.ts` -- totalling 229 lines against the
+engine's 45,726, which is 0.5% of the denominator. Two of those four are dense
+single-expression files, and reflowing them inflated their line counts while their
+covered lines stayed where they were. That is where the 162 lines went.
+
+So the finding is not "helper scripts dilute the ratchet in bulk"; `scripts/` was
+only ever dragging the total down by 0.10 points. It is narrower and worse: **the
+engine's coverage contract had a repo-only formatting input at all.** Fixed in 14.5.
`render-service/` stays out for an unrelated reason: its `src/**` is CRLF by
`.gitattributes` and Biome's formatter would rewrite the line endings.
@@ -1790,3 +1812,37 @@ One incidental finding worth keeping. Biome caps output at 20 diagnostics by
default, so `bun run lint` reported 20 findings in `scripts/` when there were 26.
`--error-on-warnings` still fails the gate, so nothing escapes -- but a count read
off that output is a floor, not a total.
+
+### 14.5 -- the ratchet now says what it measures
+
+The owner's call on 14.4's open question was to take the proper fix rather than the
+cheap one. Measuring first changed what the proper fix was: see the correction in
+14.4. The problem was never volume, it was that a number the repository enforces as
+policy could be moved by reformatting code that does not ship.
+
+`bunfig.toml` ignores `scripts/**` for coverage. Those tests still run -- they are
+the gates -- but their sources leave the denominator, so the ratchet is a statement
+about the shipped engine and nothing else. Re-measured on the narrowed scope:
+**95.39% functions (3168/3321), 92.59% lines (42336/45726)**, both a little higher
+than the blended numbers because the four instrumented `scripts/` files sat at
+72.93%.
+
+That narrowing hands back 44 lines of slack no new test earned, so `lines` goes from
+92 to **92.1**, holding the 224-line margin 13.3 chose instead of pocketing it --
+222 lines now. `functions` stays at 94, where the margin is unchanged in practice
+(1.39 points against 1.27, 46 functions against 42). Tightening further is a
+separate decision with its own friction and is not taken here.
+
+**The record now carries its own scope.** `measuredBaseline` requires a
+`measuredOver`, the gate refuses a baseline without one, and it prints it:
+`Recorded baseline: functions 95.39%, lines 92.59% (over src/, [email protected],
+2026-09-12)`. A percentage without its scope is as ambiguous as a threshold without
+a baseline, and that ambiguity is exactly what made 14.4's first explanation
+plausible enough to write down. Verified by a case in
+`scripts/check-coverage.test.mjs` that omits the field, and by
+`reliability.test.mjs` pinning both the raised threshold and the scope --
+`bunfig.toml` losing the `scripts/**` line puts the formatting input straight back.
+
+`AGENTS.md` states the scope where an agent reads it, beside the existing rule that
+the ratchet must not vary by whether the runner has a GPU. Same class of rule: the
+contract must not depend on things that are not the engine.
diff --git a/scripts/check-coverage.mjs b/scripts/check-coverage.mjs
index 8f176263..6e4cd56b 100644
--- a/scripts/check-coverage.mjs
+++ b/scripts/check-coverage.mjs
@@ -55,6 +55,14 @@ if (
if (typeof baseline?.functions !== 'number' || typeof baseline?.lines !== 'number') {
throw new Error('coverage thresholds must record the measuredBaseline they were ratcheted from');
}
+// And what it was measured OVER. A percentage without its scope is as ambiguous
+// as a threshold without a baseline, and that ambiguity is not theoretical: this
+// gate measured `src` plus four incidentally-imported files under `scripts/` for
+// months, which was enough for reformatting a repo-only script to move the
+// engine's contract -- and enough to make the cause hard to find afterwards.
+if (typeof baseline?.measuredOver !== 'string' || baseline.measuredOver.length === 0) {
+ throw new Error('coverage measuredBaseline must record what it was measuredOver');
+}
for (const metric of ['functions', 'lines']) {
if (thresholds[metric] > baseline[metric]) {
throw new Error(
@@ -80,7 +88,9 @@ const summary = `${report('functions', 'functions')}, ${report('lines', 'lines')
// Printed on the way past, not only on failure: the recorded baseline is the number
// the next ratchet decision is made against, and a log that shows it beside the
// current measurement makes a stale record visible without a second run.
-const measuredWhere = [baseline.measuredUnder, baseline.measuredOn].filter(Boolean).join(', ');
+const measuredWhere = [`over ${baseline.measuredOver}`, baseline.measuredUnder, baseline.measuredOn]
+ .filter(Boolean)
+ .join(', ');
const provenance = [
`Recorded baseline: functions ${baseline.functions.toFixed(2)}%, lines ${baseline.lines.toFixed(2)}%`,
measuredWhere ? ` (${measuredWhere})` : '',
diff --git a/scripts/check-coverage.test.mjs b/scripts/check-coverage.test.mjs
index 8cbc25af..b4966067 100644
--- a/scripts/check-coverage.test.mjs
+++ b/scripts/check-coverage.test.mjs
@@ -10,7 +10,10 @@ import { fileURLToPath } from 'node:url';
const checker = fileURLToPath(new URL('./check-coverage.mjs', import.meta.url));
/** Two functions and two lines, half of each covered: a flat 50% in both metrics. */
-async function fixture(thresholds, measuredBaseline = { functions: 100, lines: 100 }) {
+async function fixture(
+ thresholds,
+ measuredBaseline = { functions: 100, lines: 100, measuredOver: 'src/' },
+) {
const directory = await mkdtemp(join(tmpdir(), 'kiln-coverage-'));
await writeFile(
join(directory, 'lcov.info'),
@@ -54,7 +57,7 @@ test('aggregate coverage checker accepts coverage at the ratchet', async () => {
expect(result.status, result.stderr || result.stdout).toBe(0);
expect(result.stdout.trim().split('\n')).toEqual([
'Coverage gate passed: functions 50.00% (minimum 50.00%, 0 functions of slack), lines 50.00% (minimum 50.00%, 0 lines of slack)',
- 'Recorded baseline: functions 100.00%, lines 100.00%',
+ 'Recorded baseline: functions 100.00%, lines 100.00% (over src/)',
]);
} finally {
await rm(directory, { recursive: true, force: true });
@@ -94,7 +97,10 @@ test('a shortfall is reported in whole functions and lines, not only in percent'
// above anything ever measured fails in whatever change runs next, and reads there
// as that change's regression.
test('a threshold above the recorded measured baseline is rejected as self-contradictory', async () => {
- const directory = await fixture({ functions: 50, lines: 50 }, { functions: 49, lines: 100 });
+ const directory = await fixture(
+ { functions: 50, lines: 50 },
+ { functions: 49, lines: 100, measuredOver: 'src/' },
+ );
try {
const result = run(directory);
@@ -123,3 +129,20 @@ test('a thresholds file that records no measured baseline is rejected', async ()
await rm(directory, { recursive: true, force: true });
}
});
+
+// A percentage without its scope is as ambiguous as a threshold without a
+// baseline. This gate measured `src` plus four incidentally-imported files under
+// `scripts/` for months: enough for reformatting a repo-only script to move the
+// engine's contract by 0.35 points, and enough to make the cause hard to find
+// once it had. The scope is part of the record now, so it cannot go unstated.
+test('a measured baseline that does not say what it measured is rejected', async () => {
+ const directory = await fixture({ functions: 50, lines: 50 }, { functions: 100, lines: 100 });
+ try {
+ const result = run(directory);
+
+ expect(result.status).not.toBe(0);
+ expect(result.stderr).toContain('must record what it was measuredOver');
+ } finally {
+ await rm(directory, { recursive: true, force: true });
+ }
+});
diff --git a/scripts/reliability.test.mjs b/scripts/reliability.test.mjs
index 3020982a..6ca0fd19 100644
--- a/scripts/reliability.test.mjs
+++ b/scripts/reliability.test.mjs
@@ -92,6 +92,12 @@ describe('repository reliability contracts', () => {
expect(bunfig).toContain('coverageReporter = ["text", "lcov"]');
expect(bunfig).not.toContain('coverageThreshold =');
expect(bunfig).toContain('coveragePathIgnorePatterns = [');
+ // The ratchet is a contract about the shipped engine, measured over `src/`
+ // alone. Without this line four files under `scripts/` are instrumented
+ // because tests import them, and reflowing two dense ones moved the engine's
+ // coverage by 0.35 points -- a formatting change in repo-only code moving a
+ // number the repository treats as policy.
+ expect(bunfig).toContain('"scripts/**",');
// The enforced ratchets are policy: they must not move without someone
// noticing, so they stay literal here. The measured baseline is an
// observation, and it legitimately moves whenever code lands — so it is
@@ -100,7 +106,11 @@ describe('repository reliability contracts', () => {
// drift, and the copy in the test is the one that turns a stale sentence
// into a red build with no idea which of the three is right.
expect(Object.keys(thresholds).sort()).toEqual(['measuredBaseline', 'thresholds']);
- expect(thresholds.thresholds).toEqual({ functions: 94, lines: 92 });
+ expect(thresholds.thresholds).toEqual({ functions: 94, lines: 92.1 });
+ // Narrowing the scope handed back 44 lines of slack that no new test earned.
+ // `lines` moves 92 -> 92.1 to hold the margin 13.3 chose rather than pocket
+ // it; tightening beyond that is a separate decision with its own friction.
+ expect(thresholds.measuredBaseline.measuredOver).toBe('src/');
const { functions, lines } = thresholds.measuredBaseline;
expect(functions).toBeGreaterThanOrEqual(thresholds.thresholds.functions);
expect(lines).toBeGreaterThanOrEqual(thresholds.thresholds.lines);