Skip to content

armgit / kiln

Public Imported from GitHub
0

Build and revise procedural 3D assets with your coding agent. Local MCP server, CLI and TypeScript engine with rendered review, editable source and GLB export.

ci: the gates stop reporting and start blocking (#84)

Matthew Kissinger1797542835c4
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c127657a..2bee96f0 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -71,26 +71,34 @@ jobs:
windows-checks:
name: typecheck · lint · test (Windows)
runs-on: windows-2022
- # Not blocking yet, deliberately. A retained Windows coverage run aborted on entry
- # to `examples.test.ts` with a native GLib fault --
+ # Blocking as of 2026-09-12, on accumulated evidence rather than on one clean run.
+ #
+ # It landed reporting-only because a retained Windows coverage run had aborted on
+ # entry to `examples.test.ts` with a native GLib fault --
# `g_system_thread_free: The handle is invalid during CloseHandle` -- inside
- # libvips/sharp under Bun's threading. It is INTERMITTENT: two of three reproduction
- # attempts passed, and a later full gate ran 1,709 tests clean. A required check that
- # fails at random is worse than no check, so this reports rather than blocks.
+ # libvips/sharp under Bun's threading, and that fault was INTERMITTENT: two of three
+ # reproduction attempts passed. A required check that fails at random is worse than
+ # no check at all. The promotion criterion written here was consecutive green
+ # builds, which a single run cannot supply.
#
- # It was worth running because the fault was last seen on Bun 1.3.14 and sharp
- # 0.35.3, and both have since moved -- Bun 1.4 in particular fixed several test
- # isolation leaks. The first run answered it: 1,831 pass / 0 fail in 124s, the
- # same counts as Linux, with no GLib fault. So the September 5 failure does not
- # reproduce on the current toolchain. That is ONE green run against a fault that
- # already passed two of three attempts once, which is why this still reports
- # rather than blocks -- promote it once it has been green across consecutive
- # builds, which is evidence one run cannot supply.
+ # The whole history of this job is nine runs: seven green, two red, and not one
+ # recurrence of the fault. Four of the greens are consecutive and two of those are
+ # on main. Both reds were new code from the session that added the job -- a
+ # `require('sharp/package.json')` that sharp's `exports` map blocks, and a
+ # `new URL('..', import.meta.url).pathname` that yields `/D:/a/kiln/src/` and makes
+ # every fs call ENOENT. Neither is reachable from a POSIX host. That is the argument
+ # for promoting it: the failures were true, and a check that only reports is a check
+ # whose findings arrive after the merge.
#
# `test` rather than `test:coverage`: the original crash was under coverage, but the
- # ratchet in bunfig.toml is measured on Linux, so a coverage run here would entangle
- # a platform question with a threshold question and fail for the wrong reason.
- continue-on-error: true
+ # ratchet in coverage-thresholds.json is measured on Linux, so a coverage run here
+ # would entangle a platform question with a threshold question and fail for the
+ # wrong reason.
+ #
+ # Blocking also means a hang is a blocked merge rather than a slow report, so the
+ # job gets a bound. The suite runs in about two minutes here; thirty minutes is room
+ # for a cold runner, not for a deadlock.
+ timeout-minutes: 30
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a9056a18..b6767540 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,36 @@
Changes to `@kiln/engine`. Source and installable packages are distributed through
GitHub. The package is not published on the npm registry.
+## The gates stop reporting and start blocking — 2026-09-12
+
+- **The Windows job blocks.** It landed reporting-only on purpose, because the fault it
+ was added to rule out — a native GLib `g_system_thread_free` / invalid `CloseHandle`
+ inside libvips/sharp under Bun's threading — was intermittent, and a required check
+ that fails at random is worse than no check. Its whole history is now nine runs: seven
+ green, two red, and not one recurrence. Both reds were new code from the session that
+ added the job — a `require('sharp/package.json')` that sharp's `exports` map blocks,
+ and a `new URL('..', import.meta.url).pathname` that yields `/D:/a/kiln/src/` — and
+ neither is reachable from a POSIX host. The failures were true, which is the argument
+ for blocking rather than against it.
+- Blocking means a hang is a blocked merge rather than a slow report, so the job now has
+ a 30-minute bound. The suite runs there in about two minutes.
+- A test asserts no CI job carries `continue-on-error` at job level, so this cannot be
+ demoted by one line slipping back in. The single step-level exception is the native
+ dependency inventory, which exists to attach libvips and GLib versions to a run if the
+ fault ever returns and must never be why Windows reports red. Indentation is the
+ distinction and the test reads it as such.
+- **The coverage ratchet moves to 94% functions / 92% lines**, from 92/91. Measured
+ 95.27% and 92.49% on this tree, which leaves 42 functions and 224 lines of room.
+- That "42 functions" is the other half of the change: the gate now reports its margin
+ in the unit a person can act on. A percentage says whether the gate passed; turning it
+ into work needs LCOV totals that are not in front of whoever tripped it. Slack rounds
+ down and a shortfall rounds up, so neither reads as more comfortable than it is.
+- The gate also refuses a threshold set above the baseline it records. That is the one
+ mistake the coverage comparison cannot catch by itself: a threshold above anything
+ ever measured fails in whatever change happens to run next, and reads there as that
+ change's regression. A thresholds file with no recorded baseline is refused for the
+ same reason — a ratchet is raised to a number somebody measured.
+
## Dependencies current in range, and the toolchain gate gets its own test — 2026-09-12
- In-range refresh, deliberately kept apart from the deferred majors: `ai` 6.0.222 →
diff --git a/coverage-thresholds.json b/coverage-thresholds.json
index 5ec1eb08..73b162f5 100644
--- a/coverage-thresholds.json
+++ b/coverage-thresholds.json
@@ -1,12 +1,12 @@
{
"measuredBaseline": {
- "functions": 95.25,
- "lines": 92.48,
+ "functions": 95.27,
+ "lines": 92.49,
"measuredUnder": "[email protected]",
- "measuredOn": "2026-09-11"
+ "measuredOn": "2026-09-12"
},
"thresholds": {
- "functions": 92,
- "lines": 91
+ "functions": 94,
+ "lines": 92
}
}
diff --git a/docs/plans/repo-size-and-r2-migration-2026-09-10.md b/docs/plans/repo-size-and-r2-migration-2026-09-10.md
index a8e19e59..83539294 100644
--- a/docs/plans/repo-size-and-r2-migration-2026-09-10.md
+++ b/docs/plans/repo-size-and-r2-migration-2026-09-10.md
@@ -1572,8 +1572,8 @@ than for more care.
| --- | --- | --- |
| 13.1 | `site/` to three 0.186.0, and `check:toolchain` gains a cross-manifest guard: `three` must be byte-identical in every manifest that pins it, and `@types/three` must track its minor | **Done 2026-09-12.** Verified by injecting three separate defects -- the exact site drift that shipped, a `render-service` drift, and a `@types/three` left a minor behind -- and confirming each is caught and names the offending manifest |
| 13.2 | Safe in-range dependency refresh, kept strictly separate from the deferred majors: `ai` 6.0.222 to 6.0.282, `openai` 6.46.0 to 6.49.0, `@ai-sdk/provider` 3.0.14 to 3.0.16, `@aws-sdk/client-bedrock-runtime` 3.1083.0 to 3.1131.0, and in `site/` react/react-dom 19.3.0 and vite 8.3.0 | **Done 2026-09-12.** Every other pin in the tree was already at latest. Two things came out of it. The `@types/three` rule generalized to every `@types/*` whose runtime the same manifest pins, which is what brought `@types/react` and `@types/react-dom` along with react 19.3 rather than leaving them a release line behind. And `check-toolchain.mjs` gained **its own test**: it takes `--root`, and each case stages a real copy of the repository's files, mutates one value, and asserts the gate names it. Three extensions had each been verified by hand once; nothing stopped a later edit from leaving a rule that could no longer fail |
-| 13.3 | Raise the coverage ratchet from 92/91 to 94/92. Measured 95.27% functions / 92.49% lines, so this keeps roughly 1.3 and 0.5 points for normal churn | Approved, not started |
-| 13.4 | Promote the Windows job from `continue-on-error` to blocking | Approved, not started. The suite itself has never failed there; both red runs were new code of this session's own -- a `sharp/package.json` require and a `URL.pathname` that yields `/D:/...` -- which is the job doing its job |
+| 13.3 | Raise the coverage ratchet from 92/91 to 94/92. Measured 95.27% functions / 92.49% lines, so this keeps roughly 1.3 and 0.5 points for normal churn | **Done 2026-09-12.** In absolute terms that is 42 functions and 224 lines of room, which is what the gate now prints -- a percentage says whether it passed, and turning that into work needs LCOV totals the person who tripped it does not have. The gate also refuses a threshold set above the baseline it records, and refuses a thresholds file that records no baseline: a threshold above anything ever measured fails in whatever change runs next and reads there as that change's regression |
+| 13.4 | Promote the Windows job from `continue-on-error` to blocking | **Done 2026-09-12 in the repository; one GitHub setting outstanding.** Nine runs with the job: seven green, two red, no recurrence of the GLib fault, and four consecutive greens including two on main. Both reds were this session's own new code and both were real Windows-only defects, which is the argument for blocking. Given a 30-minute bound, since a blocking job that hangs is a blocked merge. A test now rejects `continue-on-error` at job level so this cannot be demoted by one line. **Outstanding:** main's branch protection lists only `typecheck . lint . test`, so a red Windows run fails the workflow and still permits the merge until the Windows context is added there |
| 13.5 | Clear the one substantive lint finding: an unused import in `src/__tests__/optimize.test.ts`. Everything else in the 14/11 baseline is `useTemplate` and `useOptionalChain` style | Approved, not started |
| 13.6 | Decide shadows by comparison, not assumption. r186 deleted the `PCFSoftShadowMap` implementation; `render-service` now names `PCFShadowMap`, and `VSMShadowMap` is the soft filter that survives. Render gallery presets under both and choose by eye, because this is the only user-visible regression in the r186 upgrade and it lands on marketing surface | Approved, not started |
diff --git a/scripts/check-coverage.mjs b/scripts/check-coverage.mjs
index 98e91a0b..8f176263 100644
--- a/scripts/check-coverage.mjs
+++ b/scripts/check-coverage.mjs
@@ -15,12 +15,29 @@ function percentage(hit, found, label) {
return (hit / found) * 100;
}
+/**
+ * How far the ratchet is from biting, counted in the unit a person can act on.
+ *
+ * A percentage says whether the gate passed; it does not say how much room is left,
+ * and the arithmetic to find out needs numbers that only appear in the LCOV file. A
+ * stranger who trips this gate should be told "seven functions short", not handed two
+ * percentages and a subtraction. Slack rounds down and a shortfall rounds up, so
+ * neither is ever reported as more comfortable than it is.
+ */
+function margin(hit, found, threshold, unit) {
+ const exact = hit - (threshold / 100) * found;
+ const count = exact >= 0 ? Math.floor(exact) : Math.ceil(-exact);
+ const noun = count === 1 ? unit.replace(/s$/u, '') : unit;
+ return `${count} ${noun} ${exact >= 0 ? 'of slack' : 'short'}`;
+}
+
const [lcov, configurationText] = await Promise.all([
readFile(lcovPath, 'utf8'),
readFile(thresholdsPath, 'utf8'),
]);
const configuration = JSON.parse(configurationText);
const thresholds = configuration.thresholds;
+const baseline = configuration.measuredBaseline;
if (
typeof thresholds?.functions !== 'number' ||
@@ -30,19 +47,50 @@ if (
) {
throw new Error('coverage thresholds must contain non-negative functions and lines percentages');
}
+// A threshold is only ever raised to a number somebody measured. Requiring the
+// baseline, and requiring it to sit at or above the threshold, is what keeps the
+// ratchet from being set on a guess -- the one mistake this file cannot otherwise
+// catch, because a threshold above anything ever observed reads exactly like a
+// regression in whatever change happens to run next.
+if (typeof baseline?.functions !== 'number' || typeof baseline?.lines !== 'number') {
+ throw new Error('coverage thresholds must record the measuredBaseline they were ratcheted from');
+}
+for (const metric of ['functions', 'lines']) {
+ if (thresholds[metric] > baseline[metric]) {
+ throw new Error(
+ `${metric} threshold ${thresholds[metric]}% exceeds the recorded measured baseline ${baseline[metric]}%`,
+ );
+ }
+}
+const counts = {
+ functions: { hit: totalField(lcov, 'FNH'), found: totalField(lcov, 'FNF') },
+ lines: { hit: totalField(lcov, 'LH'), found: totalField(lcov, 'LF') },
+};
const actual = {
- functions: percentage(totalField(lcov, 'FNH'), totalField(lcov, 'FNF'), 'functions'),
- lines: percentage(totalField(lcov, 'LH'), totalField(lcov, 'LF'), 'lines'),
+ functions: percentage(counts.functions.hit, counts.functions.found, 'functions'),
+ lines: percentage(counts.lines.hit, counts.lines.found, 'lines'),
};
const failures = Object.entries(thresholds).filter(
([metric, threshold]) => actual[metric] < threshold,
);
-const summary = `functions ${actual.functions.toFixed(2)}% (minimum ${thresholds.functions.toFixed(2)}%), lines ${actual.lines.toFixed(2)}% (minimum ${thresholds.lines.toFixed(2)}%)`;
+const report = (metric, unit) =>
+ `${metric} ${actual[metric].toFixed(2)}% (minimum ${thresholds[metric].toFixed(2)}%, ${margin(counts[metric].hit, counts[metric].found, thresholds[metric], unit)})`;
+const summary = `${report('functions', 'functions')}, ${report('lines', 'lines')}`;
+// Printed on the way past, not only on failure: the recorded baseline is the number
+// the next ratchet decision is made against, and a log that shows it beside the
+// current measurement makes a stale record visible without a second run.
+const measuredWhere = [baseline.measuredUnder, baseline.measuredOn].filter(Boolean).join(', ');
+const provenance = [
+ `Recorded baseline: functions ${baseline.functions.toFixed(2)}%, lines ${baseline.lines.toFixed(2)}%`,
+ measuredWhere ? ` (${measuredWhere})` : '',
+].join('');
if (failures.length > 0) {
console.error(`Coverage gate failed: ${summary}`);
+ console.error(provenance);
process.exit(1);
}
console.log(`Coverage gate passed: ${summary}`);
+console.log(provenance);
diff --git a/scripts/check-coverage.test.mjs b/scripts/check-coverage.test.mjs
index 34462139..7f5181bf 100644
--- a/scripts/check-coverage.test.mjs
+++ b/scripts/check-coverage.test.mjs
@@ -9,7 +9,8 @@ import { fileURLToPath } from 'node:url';
// fileURLToPath is the portable conversion.
const checker = fileURLToPath(new URL('./check-coverage.mjs', import.meta.url));
-async function fixture(thresholds) {
+/** Two functions and two lines, half of each covered: a flat 50% in both metrics. */
+async function fixture(thresholds, measuredBaseline = { functions: 100, lines: 100 }) {
const directory = await mkdtemp(join(tmpdir(), 'kiln-coverage-'));
await writeFile(
join(directory, 'lcov.info'),
@@ -30,23 +31,28 @@ async function fixture(thresholds) {
'',
].join('\n'),
);
- await writeFile(join(directory, 'thresholds.json'), JSON.stringify({ thresholds }));
+ await writeFile(join(directory, 'thresholds.json'), JSON.stringify({ measuredBaseline, thresholds }));
return directory;
}
+function run(directory) {
+ return spawnSync(
+ process.execPath,
+ [checker, join(directory, 'lcov.info'), join(directory, 'thresholds.json')],
+ { encoding: 'utf8' },
+ );
+}
+
test('aggregate coverage checker accepts coverage at the ratchet', async () => {
const directory = await fixture({ functions: 50, lines: 50 });
try {
- const result = spawnSync(
- process.execPath,
- [checker, join(directory, 'lcov.info'), join(directory, 'thresholds.json')],
- { encoding: 'utf8' },
- );
+ const result = run(directory);
expect(result.status, result.stderr || result.stdout).toBe(0);
- expect(result.stdout.trim()).toBe(
- 'Coverage gate passed: functions 50.00% (minimum 50.00%), lines 50.00% (minimum 50.00%)',
- );
+ expect(result.stdout.trim().split('\n')).toEqual([
+ 'Coverage gate passed: functions 50.00% (minimum 50.00%, 0 functions of slack), lines 50.00% (minimum 50.00%, 0 lines of slack)',
+ 'Recorded baseline: functions 100.00%, lines 100.00%',
+ ]);
} finally {
await rm(directory, { recursive: true, force: true });
}
@@ -55,16 +61,61 @@ test('aggregate coverage checker accepts coverage at the ratchet', async () => {
test('aggregate coverage checker rejects coverage below either ratchet', async () => {
const directory = await fixture({ functions: 50.01, lines: 50.01 });
try {
- const result = spawnSync(
- process.execPath,
- [checker, join(directory, 'lcov.info'), join(directory, 'thresholds.json')],
- { encoding: 'utf8' },
+ const result = run(directory);
+
+ expect(result.status).toBe(1);
+ expect(result.stderr.trim().split('\n')[0]).toBe(
+ 'Coverage gate failed: functions 50.00% (minimum 50.01%, 1 function short), lines 50.00% (minimum 50.01%, 1 line short)',
);
+ } finally {
+ await rm(directory, { recursive: true, force: true });
+ }
+});
+
+// The number a person can act on. A shortfall in percent needs the LCOV totals to
+// turn into work, and those are not in front of whoever tripped the gate.
+test('a shortfall is reported in whole functions and lines, not only in percent', async () => {
+ const directory = await fixture({ functions: 90, lines: 90 });
+ try {
+ const result = run(directory);
expect(result.status).toBe(1);
- expect(result.stderr.trim()).toBe(
- 'Coverage gate failed: functions 50.00% (minimum 50.01%), lines 50.00% (minimum 50.01%)',
+ expect(result.stderr).toContain('1 function short');
+ expect(result.stderr).toContain('1 line short');
+ } finally {
+ await rm(directory, { recursive: true, force: true });
+ }
+});
+
+// The one mistake the coverage comparison cannot catch by itself: a threshold set
+// above anything ever measured fails in whatever change runs next, and reads there
+// as that change's regression.
+test('a threshold above the recorded measured baseline is rejected as self-contradictory', async () => {
+ const directory = await fixture({ functions: 50, lines: 50 }, { functions: 49, lines: 100 });
+ try {
+ const result = run(directory);
+
+ expect(result.status).not.toBe(0);
+ expect(result.stderr).toContain(
+ 'functions threshold 50% exceeds the recorded measured baseline 49%',
+ );
+ } finally {
+ await rm(directory, { recursive: true, force: true });
+ }
+});
+
+test('a thresholds file that records no measured baseline is rejected', async () => {
+ const directory = await mkdtemp(join(tmpdir(), 'kiln-coverage-'));
+ try {
+ await writeFile(join(directory, 'lcov.info'), 'FNF:1\nFNH:1\nLF:1\nLH:1\n');
+ await writeFile(
+ join(directory, 'thresholds.json'),
+ JSON.stringify({ thresholds: { functions: 90, lines: 90 } }),
);
+ const result = run(directory);
+
+ expect(result.status).not.toBe(0);
+ expect(result.stderr).toContain('must record the measuredBaseline they were ratcheted from');
} finally {
await rm(directory, { recursive: true, force: true });
}
diff --git a/scripts/reliability.test.mjs b/scripts/reliability.test.mjs
index 3d34d632..8c752bed 100644
--- a/scripts/reliability.test.mjs
+++ b/scripts/reliability.test.mjs
@@ -64,7 +64,7 @@ describe('repository reliability contracts', () => {
// drift, and the copy in the test is the one that turns a stale sentence
// into a red build with no idea which of the three is right.
expect(Object.keys(thresholds).sort()).toEqual(['measuredBaseline', 'thresholds']);
- expect(thresholds.thresholds).toEqual({ functions: 92, lines: 91 });
+ expect(thresholds.thresholds).toEqual({ functions: 94, lines: 92 });
const { functions, lines } = thresholds.measuredBaseline;
expect(functions).toBeGreaterThanOrEqual(thresholds.thresholds.functions);
expect(lines).toBeGreaterThanOrEqual(thresholds.thresholds.lines);
@@ -75,6 +75,28 @@ describe('repository reliability contracts', () => {
expect(workflow).toContain('path: coverage/lcov.info');
});
+ // The Windows gate spent a day reporting rather than blocking, on purpose, while an
+ // intermittent native fault was ruled out. Demoting it again is a decision, not a
+ // detail, so it cannot happen by dropping one line back into the file unnoticed.
+ //
+ // This asserts the repository's half. The other half is a GitHub setting: the job's
+ // check context has to be listed in main's branch protection, or a red Windows run
+ // fails the workflow and still permits the merge.
+ test('every CI job blocks; none of them merely reports', async () => {
+ const workflow = await readText('.github/workflows/ci.yml');
+
+ expect(workflow).toContain('typecheck \u00b7 lint \u00b7 test (Windows)');
+ // Indentation is the distinction, and it is the whole point. At job level -- four
+ // spaces, a sibling of `runs-on` -- `continue-on-error` demotes an entire gate to a
+ // report. At step level it excuses one command. There must be none of the first.
+ expect(workflow.match(/^ {4}continue-on-error:/gmu)).toBeNull();
+ // The one at step level is the native dependency inventory: it exists so that IF
+ // the GLib fault returns, the libvips and GLib versions arrive attached to that run.
+ // It must never be why Windows reports red, so it is failure-tolerant by design.
+ expect(workflow.match(/^ {8}continue-on-error:/gmu)).toHaveLength(1);
+ expect(workflow).toContain('Report the native dependency inventory');
+ });
+
test('standalone agent context stays concise and names the safety-critical paths', async () => {
const agents = await readText('AGENTS.md');