Skip to content

Autonomous development for real products. Governed agents build from specs you approve, follow the rules you set, and nothing merges until another agent proves it. Your decisions, judgement and taste are the bar that work must clear. Declare your vision and let agents build it.

feat(gate): retain bounded flake evidence [A1,C7,D1,H16]

Corey Olson121784823f2c
diff --git a/docs/api.md b/docs/api.md
index 92ecace8..4870925a 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -105,6 +105,8 @@ For example: `flow: review wait average 61 min (1 measured; > 60 min); bottlenec
| `decisions` | `version:number`, `decisions:array` |
| `ledger` | `version:number`, `items:array`, `stats:object` |
| `log` | `version:number`, `events:array` |
+| `flakes` | `version:number`, `flakes:array`, `unclassified:array`, `regressions:array`, `stable:array` |
+| `flakes rerun` | `version:number`, `flakes:object`, `file:string` |
| `rigor` | `version:number`, `causes:array`, `uncaused:number` |
| `rigor draft` | `version:number`, `key:string`, `reads:string`, `evidence:object`, `proof:object`, `drafts:array` |
| `add` | `version:number`, `item:object` |
@@ -253,6 +255,10 @@ The HTTP move is `{verb: "fact", item: 12, agent: "web-1", args: {kind: "measure
Rigor is how errors become doctrine or specs. It starts by noticing a failure cause that repeats. A cause key comes only from structured fields: `test:<file>::<title>`, `refusal:<CODE>`, `row:<id>` or `doctrine:<id>`. A reject (`verify <id> reject ... --cause <key>`) and a `reopen <id> --note "..." --cause <key>` record the key they name as `cause` in their event detail. A reopen after a red landing also gives one `test:` key per failing test in its `landingFailure.tests`, including reopens recorded before cause keys existed. Notes are never parsed, so an older reject with no cause counts as uncaused. A malformed key is refused with `BAD_CAUSE` before the move. Merge conflicts never count: a landing test whose name starts with `conflicts with`, and `refusal:MERGE_CONFLICT`.
+`pullboard flakes` lists recorded test failures: confirmed flake sightings, unclassified failures, possible regressions and stable red targets. Each observation retains the pinned source and complete raw/profile digests. Raw artifacts stay in the local common Git directory under `pullboard/flake-evidence/`; another device reading the board does not automatically receive those files. A paired pass never changes the original red verdict.
+
+The coordinator can run `pullboard flakes rerun on|off`, then commit `pullboard.json` to apply that policy. Paired diagnostics default off; recording remains on. On allows one exact-target pair for a supported structured Node target at unchanged source, with a pinned-main comparison. A green main is a possible regression. Unknown selectors, changed source and unsupported remote comparisons remain unclassified. A second confirmed sighting drafts ordinary strong repair work; it approves no spec or doctrine row. Off preserves the released landing diagnosis and its first legacy defect. Enabled landing consumes the saved pair, including after interruption, without another diagnostic measurement.
+
`pullboard rigor` is read-only. It lists each cause that repeated: on 2 items, or on 1 item at 2 different commits. Each line shows the key, how it reads (`work` for `doctrine:` and `refusal:`, `product` for `row:`, `both` for `test:`), its failure count and its items. Its JSON result is `{version, causes: [{key, reads, occurrences, neverAgain, items: [{id, commits}]}], uncaused}`; `uncaused` counts rejects and reopens with no cause key.
The person can mark a cause never-again from the main checkout's terminal: `pullboard fact <id> root-cause "<why>" --cause <key> --never-again --as person`. The fact is recorded as the person's, carries `cause` and `neverAgain: true`, and `rigor` lists its cause at once. An agent or a fact without `--as person` is refused with `RIGOR_PERSON_ONLY`. Over the relay, a fact move with person mode, a never-again or a cause key is accepted only from an authenticated person sender; an agent, a coordinator seat or the machine credential is refused with `RELAY_PERSON_ONLY` whatever channel the move names, and nothing is recorded. A move that carries a cause key needs move engine 30; moves without one replay as before.
diff --git a/docs/formats.md b/docs/formats.md
index eb320cdc..d4406aed 100644
--- a/docs/formats.md
+++ b/docs/formats.md
@@ -152,6 +152,10 @@ Engine 55 (`ROW_DRAFT_ENGINE`) adds `draftSpecRow`: an agent in a lane worktree
Engine 56 (`BOARD_SENTINEL_ENGINE`) adds board-owned watch proposals, exact person approve/decline decisions, ordered due-run claims and completed run receipts. Approval binds the entire execution definition; changing it creates a pending proposal. Replay records history without executing commands. Every non-cancelled red run records `sentinel:<name>` for Rigor; feed notices only mark failure and recovery transitions. Older engines refuse these operations and this new cause form before replay. Published engine-55 readers reject these moves by their newer engine number before interpreting an operation.
+Engine 59 (`NATIVE_FLAKE_ENGINE`) adds `recordFlakeObservation` and typed diagnostic relationships in landing-batch receipts. A registered reporting agent supplies a bounded observation: stable id, relative test file/name (or null when unknown), pinned head/tree/main, gate kind, classification, observation time, machine-load sample, diagnostic duration, command digest, and original/paired/main receipt references. A receipt contains its result, byte count, SHA-256 and optional profile digest; full raw output and profiles stay under the common Git directory's `pullboard/flake-evidence/<id>/`, outside board events. Those local artifacts are not automatically copied to another device by a board snapshot. A duplicate observation id returns the original receipt; different evidence under that id is refused. The original red remains authoritative.
+
+The board's committed `flakes.rerun` setting defaults off. Off retains released execution, including landing's whole-file diagnosis and first legacy `landing_flake`; the new ledger still records the original failure. On permits one exact-target diagnostic for a supported structured Node test at unchanged source, never green-by-retry. Built-in `node --test` is supported; a `bin/run-tests.js` script is recognized only when its committed bytes on candidate and pinned main match the bundled Native runner. Its filename alone does not establish selector support, and a probe that executes other targets remains unclassified. A pinned green main marks a possible regression, not a flake. Unknown selectors, source changes or unsupported remote execution evidence remain unclassified; no local substitute is run for a remote admission. A confirmed second sighting drafts an ordinary strong repair item, not an approved spec or doctrine row. A landed repair reads fixed until another sighting. Enabled landing consumes its captured pair without another legacy diagnosis or first-sighting legacy defect; saved typed relationships prevent remeasurement on adoption. Diagnostic-recording failures are reported while the original red and completed evidence remain retained. Older engines refuse the new operation and typed landing relationships before changing rows or the replay cursor; legacy `landing_flake` meaning and the schema/event-log versions are unchanged.
+
Engine 60 (`SUBMITTED_CONFLICT_ENGINE`) lets a conflict reopen return a submitted head, not only an accepted one, and lets any joined seat record it. Before it offers a review, `pullboard next --verify` merges every submitted head onto the trunk in memory, as landing does; a head that conflicts goes back to its builder, open and unreviewed, with Git's conflicting files and the trunk commit, and its builder is told which files and that the one step back is to claim it, merge main and submit again. The reopen sets no accept aside and records no verdict; its next review is a whole one. A hand-written `--conflict` on a submission, which names no trunk commit, is still refused. Engines 57 to 59 are allocated to other changes; older engines refuse this move before replay.
The CLI declares its engine in every relay request's `X-Pullboard-Engine` header.
@@ -382,6 +386,8 @@ The `event` table is a SQLite schema object governed by `SCHEMA_VERSION`; its ap
| `answer` | answerer | `shout`, `to`, `decision`, `request`, `answers`, `outcome`, `channel`, engine-20 `implies`, `supersedes` |
| `row_decision` | person | `record`, `channel` |
| `row_draft` | builder | engine-55 `row`, `written` |
+| `flake-observation` | reporting agent | engine-59 `engine`, `observation` |
+| `flake-repair` | reporting agent | engine-59 `engine`, `key`, `item` |
| `row_apply` | coordinator | `events` |
<!-- events:end -->
diff --git a/src/board-stats.js b/src/board-stats.js
new file mode 100644
index 00000000..6ea15df0
--- /dev/null
+++ b/src/board-stats.js
@@ -0,0 +1,18 @@
+/** Item status and verdict totals for board summaries. */
+import { listItems } from './board.js';
+
+/**
+ * Counts for the status line and the ledger's summary: items by status, verdicts by decision.
+ *
+ * @param {any} board
+ * @returns {{ items: Record<string, number>, accepted: number, rejected: number }}
+ */
+export function stats(board) {
+ const items = { open: 0, claimed: 0, submitted: 0, verified: 0, withdrawn: 0 };
+ for (const item of listItems(board, { all: true })) items[item.item_status] += 1;
+ const verdicts = board.db
+ .prepare('SELECT verdict_decision AS decision, COUNT(*) AS total FROM verdict GROUP BY verdict_decision')
+ .all();
+ const count = (decision) => verdicts.find((row) => row.decision === decision)?.total ?? 0;
+ return { items, accepted: count('ACCEPT'), rejected: count('REJECT') };
+}
diff --git a/src/board.js b/src/board.js
index 927af01f..fa15d4b8 100644
--- a/src/board.js
+++ b/src/board.js
@@ -97,6 +97,11 @@ import {
} from './landing-exceptions.js';
import { parseSpec } from './spec.js';
import { invariantEvidence } from './submit-invariants.js';
+import { landingBatches } from './landing-batch-history.js';
+import {
+ NATIVE_FLAKE_ENGINE,
+ carriesFlakeDiagnostics,
+} from './flake-engine.js';
import { preserveInvariantEvidence } from './submit-invariants.js';
import { validateLandingBatch } from './submit-invariants.js';
import { validateLandingRevision } from './submit-invariants.js';
@@ -105,6 +110,7 @@ import { SUBMISSION_PROOF_ENGINE } from './submit-invariants.js';
export { landingFlakes, landingWaiverCovers, landingWaiverKind };
export { acceptCheck } from './check-accept.js';
export { proposeSentinel, decideSentinel, beginSentinelRun, finishSentinelRun } from './sentinels.js';
+export { flakeLedger, recordFlakeObservation } from './flakes.js';
/** The seat readers live in seats.js; the board re-exports them so `store.*` callers keep one import. */
export { agentAt, agentCreator, listAgents, nextSeatId, seatEnrolled, seatLineage, withVerifierStanding } from './seats.js';
@@ -2655,35 +2661,16 @@ export function recentShouts(board, limit = 40) {
return board.db.prepare('SELECT * FROM shout ORDER BY shout_id DESC LIMIT ?').all(limit);
}
-/**
- * Counts for the status line and the ledger's summary: items by status, verdicts by decision.
- *
- * @param {any} board
- * @returns {{ items: Record<string, number>, accepted: number, rejected: number }}
- */
-export function stats(board) {
- const items = { open: 0, claimed: 0, submitted: 0, verified: 0, withdrawn: 0 };
- for (const item of listItems(board, { all: true })) items[item.item_status] += 1;
- const verdicts = board.db
- .prepare('SELECT verdict_decision AS decision, COUNT(*) AS total FROM verdict GROUP BY verdict_decision')
- .all();
- const count = (decision) => verdicts.find((row) => row.decision === decision)?.total ?? 0;
- return { items, accepted: count('ACCEPT'), rejected: count('REJECT') };
-}
-
-/** Read the latest append-only snapshot of each landing batch [B3,R3,R2]. */
-export function landingBatches(board) {
- const batches = new Map();
- for (const row of board.db.prepare("SELECT event_detail FROM event WHERE event_kind = 'landing_batch' ORDER BY event_id").all()) {
- const { batch } = JSON.parse(row.event_detail);
- batches.set(batch.id, batch);
- }
- return [...batches.values()];
-}
+export { stats } from './board-stats.js';
+
+export { landingBatches };
/** Append one coordinator-owned batch revision without changing the board schema [B3,R3]. */
export function recordLandingBatch(board, { agentId, batch, revision = 0 }) {
coordinatorOnly(agentId, 'records landing batches');
+ if (carriesFlakeDiagnostics(batch) && (board.executionEngineVersion ?? ENGINE_VERSION) < NATIVE_FLAKE_ENGINE) {
+ throw new Refused('ENGINE_VERSION', `typed landing diagnostics require engine ${NATIVE_FLAKE_ENGINE}; upgrade Pullboard`);
+ }
const evidence = (board.executionEngineVersion ?? ENGINE_VERSION) >= SUBMISSION_PROOF_ENGINE ? invariantEvidence(batch?.invariantEvidence) : undefined;
validateLandingBatch(batch, revision);
if ((board.executionEngineVersion ?? ENGINE_VERSION) >= VERDICT_REVIEW_ENGINE
diff --git a/src/cli.js b/src/cli.js
index fd7449a1..01dc47be 100644
--- a/src/cli.js
+++ b/src/cli.js
@@ -42,6 +42,7 @@ import {
} from './next-steps.js';
import { reviewerStartup } from './next-steps.js';
import * as store from './board.js';
+import * as nativeFlakes from './native-flake-cli.js';
import {
CONFIG_FILE,
COORDINATOR,
@@ -2315,6 +2316,7 @@ function refuseUncommittedSetup(mainRoot, config) {
*/
function readCommands(io, { first, second, rest, values }) {
return {
+ flakes: () => nativeFlakes.command(context(io), { first, second, rest }, withBoard),
doctrine: async () => {
if (first !== 'check' || second || rest.length) throw new Refused('USAGE', 'pullboard doctrine check');
const ctx = context(io);
@@ -2754,7 +2756,8 @@ async function submitHere(ctx, id) {
submitInvariants: acceptedMain.config.submitInvariants,
// The accepted project runner interprets its current invariant files; candidate config cannot replace it.
...(acceptedMain.config.submitInvariants.length ? { affectedTests: acceptedMain.config.affectedTests } : {}) };
- const gate = await runSubmitGate(root, submitConfig, { base: claimHead, trunk: acceptedMain.commit, check: frozenCheck(held), agent: me.id, item: String(id), onWait: queuedItemLease(ctx, held, me.id, 'build'), onBrowserFallback: browserAdmissionReporter(ctx.io, ctx.io.relayJson) });
+ const gate = await runSubmitGate(root, submitConfig, { base: claimHead, trunk: acceptedMain.commit, check: frozenCheck(held), agent: me.id, item: String(id), onWait: queuedItemLease(ctx, held, me.id, 'build'), onBrowserFallback: browserAdmissionReporter(ctx.io, ctx.io.relayJson),
+ onFailure: nativeFlakes.observer(ctx, { kind: 'submit', main: acceptedMain.commit, config: submitConfig }, { ordered, withBoard, resourceAgent }) });
// Only the person's live waivers can pass a red proof, failure by failure; the item's own check is never waived [B3].
const waived = !gate.isGreen && gate.check?.green ? withBoard(ctx, (board) => waivedFailures(board, accountedFailures(gate, root))) : null;
if (!gate.isGreen && !waived) {
@@ -3012,7 +3015,7 @@ async function verifyHere(ctx, id, { second, values }) {
const browser = !browserFreeChecksEnabled() || await checkNeedsBrowser(root, frozenCheck(item), { commit: checkedCommit, onFallback: browserAdmissionReporter(ctx.io, ctx.io.relayJson) });
// A private checked composition needs this machine's live local gate markers [C7,V4].
const shortEligible = Boolean(await explicitCheckFiles(root, frozenCheck(item), { commit: checkedCommit }));
- check = await withGateSlot(root, (lease) => checkAtCommit(root, checkedItem, { waitMs: lease.waitMs, gateLease: lease }), { browser, remote: false, agent: me.id, item: String(id), kind: 'review', itemCheck: true, shortEligible, onWait: queuedItemLease(ctx, item, me.id, 'review') });
+ check = await withGateSlot(root, nativeFlakes.verify(ctx, checkedItem, frozenCheck(item), { board, main: trunk.commit, commit: checkedCommit }, { ordered, withBoard, resourceAgent }), { browser, remote: false, agent: me.id, item: String(id), kind: 'review', itemCheck: true, shortEligible, onWait: queuedItemLease(ctx, item, me.id, 'review') });
} else {
check = checkAtCommit(root, checkedItem);
}
@@ -3583,7 +3586,7 @@ function workCommands(io, args) {
await runStaticChecks(ctx.info.root, (item.item_claim_head ? itemPolicy(ctx.info.root, item) : mainPolicy(ctx.info.root)).config, { agent: caller, item: String(item.item_id), onWait: queuedItemLease(ctx, item, caller, 'build'), onBrowserFallback: browserAdmissionReporter(io, values.json) });
const browser = !browserFreeChecksEnabled() || await checkNeedsBrowser(ctx.info.root, check, { onFallback: browserAdmissionReporter(io, values.json) });
const shortEligible = Boolean(await explicitCheckFiles(ctx.info.root, check));
- const run = await withGateSlot(ctx.info.root, (lease) => runProfiledShell(ctx.info.root, check, { waitMs: lease.waitMs, artifactPrefix: `pullboard-check-${item.item_id}`, persistLog: true, gateLease: lease }), { browser, agent: caller, item: String(item.item_id), kind: 'check', itemCheck: true, shortEligible, onWait: queuedItemLease(ctx, item, caller, 'build') });
+ const run = await withGateSlot(ctx.info.root, nativeFlakes.check(ctx, item, check, { ordered, withBoard, resourceAgent }), { browser, agent: caller, item: String(item.item_id), kind: 'check', itemCheck: true, shortEligible, onWait: queuedItemLease(ctx, item, caller, 'build') });
refuseSandboxBlocked(run);
io.result?.({ id: item.item_id, green: run.isGreen, seconds: run.seconds, check, by, report: run.isGreen ? '' : digestOf(run.output) });
const timings = timingDigest(run.profile);
@@ -4512,7 +4515,8 @@ async function runCommand(argv, io) {
throw error;
}
const waivers = values.landing ? withBoard(ctx, (board) => liveWaiverProjection(board)) : [];
- const gate = await runGate(ctx.info.root, config, { landing: values.landing, agent: resourceAgent(ctx), onWait: gateWaitReporter(io), stages: Boolean(values.landing), waivers });
+ const gate = await runGate(ctx.info.root, config, { landing: values.landing, agent: resourceAgent(ctx), onWait: gateWaitReporter(io), stages: Boolean(values.landing), waivers,
+ onFailure: nativeFlakes.observer(ctx, { kind: values.landing ? 'land' : 'gate', config }, { ordered, withBoard, resourceAgent }) });
// A landing gate honours the person's live waivers exactly as land does; a red run never stamps, so it stays waived, not green [B3,R3].
const waived = values.landing && !gate.isGreen ? withBoard(ctx, (board) => waivedFailures(board, accountedFailures(gate, ctx.info.root))) : null;
if (waived) {
diff --git a/src/config.js b/src/config.js
index fb74f5ba..7503e8d7 100644
--- a/src/config.js
+++ b/src/config.js
@@ -164,6 +164,9 @@ export function configProblems(config) {
if (typeof config.spec !== 'string' || !config.spec) problems.push('"spec" names a file');
if (typeof config.practice !== 'string' || !config.practice) problems.push('"practice" names the doctrine file');
if (typeof config.gate !== 'string') problems.push('"gate" is a shell command, like "npm test"');
+ if (config.flakes !== undefined && (!config.flakes || typeof config.flakes !== 'object' || Array.isArray(config.flakes)
+ || config.flakes.rerun !== undefined && typeof config.flakes.rerun !== 'boolean'
+ || Object.keys(config.flakes).some(key => key !== 'rerun'))) problems.push('"flakes" has only the optional boolean "rerun", off by default');
if (!['inside', 'beside'].includes(config.worktrees)) problems.push('"worktrees" is "inside" or "beside"');
if (config.submitGate !== undefined && config.submitGate !== 'landing') {
problems.push('"submitGate" is "landing" when the full suite should be deferred until landing');
diff --git a/src/engine.js b/src/engine.js
index d5afe4e9..2f44adab 100644
--- a/src/engine.js
+++ b/src/engine.js
@@ -48,6 +48,10 @@ import {
SUBMISSION_PROOF_ENGINE,
submissionProof,
} from './submit-invariants.js';
+import {
+ NATIVE_FLAKE_ENGINE,
+ carriesFlakeDiagnostics,
+} from './flake-engine.js';
const ITEM_ID_FIRST_ARGUMENT = new Set(['acceptCheck', 'editItem', 'escalate', 'recordAttempt', 'claim', 'release', 'submit', 'reserveReview', 'verify', 'merged', 'withdraw', 'refreeze', 'reopen', 'appendFact']);
@@ -60,6 +64,7 @@ export const ENGINE_OPERATIONS = Object.freeze([
'recordRowDecisions', 'applyRowDecisions', 'draftSpecRow', 'appendFact',
'recordLandingBatch', 'recordLandingWaiver', 'recordLandingFlake', 'finishLandingBatch', 'acceptCheck',
'recordHealthFindings', 'proposeSentinel', 'decideSentinel', 'beginSentinelRun', 'finishSentinelRun',
+ 'recordFlakeObservation',
]);
/** Engine version at which each ordered operation first has stable replay semantics. */
@@ -68,6 +73,8 @@ const OPERATION_ENGINE = Object.freeze({ draftSpecRow: ROW_DRAFT_ENGINE, orderMi
/** Dependency edits, causes, sealed reviews, submission proofs, trunk conflicts, health observations and milestone targets retain distinct recorded engine meanings. */
function minimumOperationEngine(operation, args, board) {
+ if (operation === 'recordFlakeObservation') return NATIVE_FLAKE_ENGINE;
+ if (operation === 'recordLandingBatch' && carriesFlakeDiagnostics(args[0]?.batch)) return NATIVE_FLAKE_ENGINE;
if (['proposeSentinel', 'decideSentinel', 'beginSentinelRun', 'finishSentinelRun'].includes(operation) || typeof args[1]?.cause === 'string' && args[1].cause.startsWith('sentinel:')) return BOARD_SENTINEL_ENGINE;
if (operation === 'recordHealthFindings') return HEALTH_SENTINEL_ENGINE;
const options = args[1];
diff --git a/src/flake-command.js b/src/flake-command.js
new file mode 100644
index 00000000..17a89281
--- /dev/null
+++ b/src/flake-command.js
@@ -0,0 +1,35 @@
+/** Read durable observations or opt this board into one diagnostic pair, never into retry acceptance [C7,H16]. */
+import {
+ readFileSync,
+ writeFileSync,
+} from 'node:fs';
+import { join } from 'node:path';
+import { CONFIG_FILE } from './config.js';
+import { flakeLedger } from './flakes.js';
+import { Refused } from './refused.js';
+
+/** List actual flake sightings separately from unknown failures and candidate regressions. */
+export function flakesCommand(ctx, { first, second, rest }, withBoard) {
+ if (first === 'rerun' && ['on', 'off'].includes(second) && !rest.length) {
+ if (!ctx.info.isMain) throw new Refused('NOT_MAIN', 'only the coordinator sets this board’s diagnostic policy; ask it to run pullboard flakes rerun on or off');
+ const file = join(ctx.info.root, CONFIG_FILE);
+ const config = JSON.parse(readFileSync(file, 'utf8'));
+ config.flakes = { rerun: second === 'on' };
+ writeFileSync(file, JSON.stringify(config, null, 2) + '\n');
+ ctx.io.result?.({ flakes: config.flakes, file: CONFIG_FILE });
+ ctx.io.say(`paired flake diagnostics ${second}; commit ${CONFIG_FILE} to apply this board policy. The original red remains red.`);
+ return 0;
+ }
+ if (first || second || rest.length) throw new Refused('USAGE', 'pullboard flakes lists observations; pullboard flakes rerun on|off sets the coordinator’s opt-in policy');
+ return withBoard(ctx, board => {
+ const rows = flakeLedger(board);
+ const flakes = rows.filter(row => row.count > 0);
+ const unclassified = rows.filter(row => row.count === 0 && row.classification === 'unclassified');
+ const regressions = rows.filter(row => row.count === 0 && row.classification === 'possible-regression');
+ const stable = rows.filter(row => row.count === 0 && row.classification === 'stable-red');
+ ctx.io.result?.({ flakes, unclassified, regressions, stable });
+ for (const row of rows) ctx.io.say(`${row.status}: ${row.file ?? '(unclassified gate)'}${row.name ? ' :: ' + row.name : ''}; count ${row.count}; first ${row.firstSeen}; last ${row.lastSeen}${row.item ? '; repair #' + row.item : ''}`);
+ if (!rows.length) ctx.io.say('no recorded test failures; paired diagnostics are off unless this board opts in with pullboard flakes rerun on');
+ return 0;
+ });
+}
diff --git a/src/flake-engine.js b/src/flake-engine.js
new file mode 100644
index 00000000..7aa91ca6
--- /dev/null
+++ b/src/flake-engine.js
@@ -0,0 +1,8 @@
+/** Coordinator allocation17419: typed Native flake observations start here [H16]. */
+export const NATIVE_FLAKE_ENGINE = 59;
+
+/** Recognize new landing receipt relationships without interpreting malformed batch input. */
+export function carriesFlakeDiagnostics(batch) {
+ return Boolean(batch && (Object.hasOwn(batch, 'flakeDiagnostics')
+ || Array.isArray(batch.gateResults) && batch.gateResults.some(result => result && Object.hasOwn(result, 'flakeDiagnostics'))));
+}
diff --git a/src/flakes.js b/src/flakes.js
new file mode 100644
index 00000000..cc4e51bc
--- /dev/null
+++ b/src/flakes.js
@@ -0,0 +1,122 @@
+/** Typed failure observations preserve evidence separately from person-authorized waivers [C7,D1,H16]. */
+import { createHash } from 'node:crypto';
+import {
+ addItem,
+ atomic,
+ getItem,
+ listAgents,
+ logEvent,
+} from './board.js';
+import { NATIVE_FLAKE_ENGINE } from './flake-engine.js';
+import { ENGINE_VERSION } from './machine.js';
+import { Refused } from './refused.js';
+
+const SHA = /^(?:[a-f0-9]{40}|[a-f0-9]{64})$/u;
+const DIGEST = /^[a-f0-9]{64}$/u;
+const CLASSIFICATIONS = ['flake', 'possible-regression', 'stable-red', 'unclassified'];
+
+/** Require portable artifact names, never a machine's private checkout or home path. */
+function receipt(value) {
+ if (!value || typeof value.green !== 'boolean' || !DIGEST.test(value.sha256 ?? '')
+ || !Number.isSafeInteger(value.bytes) || value.bytes < 0 || typeof value.artifact !== 'string'
+ || !/^flake-evidence\/[a-zA-Z0-9-]+\/[a-zA-Z0-9.-]+$/u.test(value.artifact)
+ || value.profileSha256 !== null && !DIGEST.test(value.profileSha256 ?? '')) {
+ throw new Refused('FLAKE_EVIDENCE', 'retain the complete original and diagnostic receipts with their digests before recording a flake');
+ }
+ return { green: value.green, sha256: value.sha256, bytes: value.bytes, artifact: value.artifact, profileSha256: value.profileSha256 };
+}
+
+/** Normalize only bounded, structured evidence; a successful diagnostic cannot replace its original red. */
+export function flakeObservation(value) {
+ const text = (entry, limit) => typeof entry === 'string' && entry.length > 0 && entry.length <= limit && !/[\r\n\0]/u.test(entry);
+ if (!value || !text(value.id, 80) || !/^[a-zA-Z0-9-]+$/u.test(value.id) || !SHA.test(value.head ?? '')
+ || !SHA.test(value.tree ?? '') || value.main !== null && !SHA.test(value.main ?? '')
+ || !['check', 'submit', 'verify', 'land', 'gate'].includes(value.kind) || !CLASSIFICATIONS.includes(value.classification)
+ || !DIGEST.test(value.commandSha256 ?? '') || !text(value.reason, 2048) || !Number.isFinite(Date.parse(value.at))
+ || !Number.isFinite(value.diagnosticMs) || value.diagnosticMs < 0) {
+ throw new Refused('FLAKE_EVIDENCE', 'a failure observation needs its pinned source, gate kind, time and complete receipts');
+ }
+ const file = value.file ?? null, name = value.name ?? null;
+ if (file !== null && (!text(file, 4096) || file.startsWith('/') || file.includes('\\') || file.split('/').some(part => !part || part === '.' || part === '..'))
+ || name !== null && !text(name, 8192) || (file === null) !== (name === null)
+ || value.classification !== 'unclassified' && file === null) {
+ throw new Refused('FLAKE_EVIDENCE', 'identify the test with its committed relative file and structured name, or leave it unclassified');
+ }
+ const original = receipt(value.original);
+ const paired = value.paired === null ? null : receipt(value.paired);
+ const mainReceipt = value.mainReceipt === null ? null : receipt(value.mainReceipt);
+ if (original.green || value.classification === 'flake' && (!paired?.green || value.main !== value.head && (!mainReceipt || mainReceipt.green))
+ || value.classification === 'possible-regression' && !mainReceipt?.green
+ || value.classification === 'stable-red' && (!paired || paired.green)) {
+ throw new Refused('FLAKE_EVIDENCE', 'a flake needs a red original and one green pair; a pinned green main is a possible regression instead');
+ }
+ const load = value.load;
+ if (!load || !Array.isArray(load.average) || load.average.length !== 3 || load.average.some(number => !Number.isFinite(number) || number < 0)
+ || !Number.isSafeInteger(load.cpus) || load.cpus < 1 || !Number.isSafeInteger(load.freeMB) || load.freeMB < 0) {
+ throw new Refused('FLAKE_EVIDENCE', 'record observed machine load with this failure');
+ }
+ return { id: value.id, file, name, head: value.head, tree: value.tree, main: value.main, kind: value.kind,
+ classification: value.classification, commandSha256: value.commandSha256, reason: value.reason, at: new Date(value.at).toISOString(),
+ diagnosticMs: value.diagnosticMs, load: { average: [...load.average], cpus: load.cpus, freeMB: load.freeMB }, original, paired, mainReceipt };
+}
+
+/** A stable identity follows the actual test, rather than its line number, machine or latest source head. */
+function observationKey(value) {
+ return createHash('sha256').update(JSON.stringify(value.file === null ? ['unknown', value.commandSha256] : [value.file, value.name])).digest('hex');
+}
+
+/** Read only allocated typed events, preserving every original observation and repair relationship. */
+export function flakeLedger(board) {
+ const rows = new Map();
+ for (const event of board.db.prepare("SELECT * FROM event WHERE event_kind IN ('flake-observation','flake-repair') ORDER BY event_id").all()) {
+ const detail = JSON.parse(event.event_detail);
+ if (detail.engine !== NATIVE_FLAKE_ENGINE) continue;
+ if (event.event_kind === 'flake-repair') {
+ const row = rows.get(detail.key);
+ if (row) row.item = detail.item;
+ continue;
+ }
+ const value = detail.observation;
+ const key = observationKey(value);
+ let row = rows.get(key);
+ if (!row) { row = { key, file: value.file, name: value.name, count: 0, firstSeen: event.event_at, lastSeen: event.event_at, item: null, status: 'unclassified', observations: [] }; rows.set(key, row); }
+ row.observations.push(value);
+ row.lastSeen = event.event_at;
+ row.lastEventId = event.event_id;
+ row.classification = value.classification;
+ if (value.classification === 'flake') row.count += 1;
+ row.status = row.count ? 'active' : value.classification;
+ }
+ for (const row of rows.values()) if (row.item !== null) {
+ const item = getItem(board, row.item);
+ const merged = board.db.prepare("SELECT MAX(event_id) AS id FROM event WHERE event_kind='merged' AND item_id=?").get(row.item);
+ if (item.item_merged_commit && merged.id > row.lastEventId) row.status = 'fixed';
+ }
+ return [...rows.values()];
+}
+
+/** Atomically append one observation, drafting ordinary repair work only on a second confirmed sighting. */
+export function recordFlakeObservation(board, { agentId, observation, lane }) {
+ if ((board.executionEngineVersion ?? ENGINE_VERSION) < NATIVE_FLAKE_ENGINE) throw new Refused('ENGINE_VERSION', `flake observations require engine ${NATIVE_FLAKE_ENGINE}; upgrade Pullboard`);
+ if (!listAgents(board).some(agent => agent.agent_id === agentId)) throw new Refused('NOT_JOINED', 'join this board before recording a flake observation');
+ const value = flakeObservation(observation);
+ return atomic(board, () => {
+ const existing = flakeLedger(board).flatMap(row => row.observations).find(row => row.id === value.id);
+ if (existing) {
+ if (JSON.stringify(existing) !== JSON.stringify(value)) throw new Refused('FLAKE_OBSERVATION_CHANGED', 'an observation id already binds different evidence; preserve the original and use a new id');
+ return { observation: existing, duplicate: true };
+ }
+ const key = observationKey(value);
+ logEvent(board, agentId, 'flake-observation', null, { engine: NATIVE_FLAKE_ENGINE, observation: value });
+ const row = flakeLedger(board).find(entry => entry.key === key);
+ if (row.count >= 2 && row.item === null) {
+ const item = addItem(board, { by: agentId, lane, route: 'strong', title: `Flaky test: ${value.file} :: ${value.name}`.slice(0, 240),
+ criterion: `Identify and correct why ${value.file} :: ${value.name} fails intermittently; preserve the original red and paired evidence, prove the correction without retries, and pass the gate.`,
+ brief: 'Rigor repair draft from repeated typed flake observations. The original red remains the gate result. No spec or doctrine decision was approved.\n\n'
+ + row.observations.map(entry => JSON.stringify({ id: entry.id, head: entry.head, at: entry.at, kind: entry.kind, original: entry.original, paired: entry.paired, main: entry.mainReceipt })).join('\n') });
+ logEvent(board, agentId, 'flake-repair', item, { engine: NATIVE_FLAKE_ENGINE, key, item });
+ return { observation: value, item };
+ }
+ return { observation: value, item: row.item };
+ });
+}
diff --git a/src/gate.js b/src/gate.js
index e44d7670..5d42f629 100644
--- a/src/gate.js
+++ b/src/gate.js
@@ -543,12 +543,16 @@ export async function withGateSlot(root, action, { landing = false, itemCheck =
* @param {{ trustStamp?: boolean, onWait?: (state: object) => void, landing?: boolean, agent?: string, item?: string|number }} [options]
* @returns {Promise<{ isGreen: boolean, isCached: boolean, output: string, seconds: number, log: string, profile?: object, profilePath?: string }>}
*/
-export async function runGate(root, config, { trustStamp = true, onWait, landing = false, agent, item = '', stages = false, waivers = [] } = {}) {
+export async function runGate(root, config, { trustStamp = true, onWait, landing = false, agent, item = '', stages = false, waivers = [], onFailure } = {}) {
if (!config.gate.trim()) {
throw missingGate(root);
}
if (trustStamp && isStampedGreen(root)) return { isGreen: true, isCached: true, output: '', seconds: 0, log: '' };
- return await withGateSlot(root, (lease) => fullGateRun(root, config, trustStamp, { waitMs: lease.waitMs, gateLease: lease, landing, stages, waivers }), {
+ return await withGateSlot(root, async (lease) => {
+ const run = fullGateRun(root, config, trustStamp, { waitMs: lease.waitMs, gateLease: lease, landing, stages, waivers });
+ if (!run.isGreen && onFailure) await onFailure(run, { root, command: config.gate, lease });
+ return run;
+ }, {
landing, browser: true, agent, item, kind: landing ? 'landing' : 'gate', onWait,
});
}
@@ -592,7 +596,7 @@ export async function runStaticChecks(root, config, { agent, item = '', onWait,
* @param {{base:string|null, trunk?:string, changed?:string[], check:string, agent?:string, item?:string|number, kind?:string, onWait?:(state:object)=>void, onBrowserFallback?:(decision:{file:string,reason:string})=>void}} options
* @returns {Promise<any>}
*/
-export async function runSubmitGate(root, config, { base, trunk, changed, check, agent, item = '', kind = 'submit', onWait, onBrowserFallback }) {
+export async function runSubmitGate(root, config, { base, trunk, changed, check, agent, item = '', kind = 'submit', onWait, onBrowserFallback, onFailure }) {
if (!config.gate.trim()) throw missingGate(root);
await runStaticChecks(root, config, { agent, item, onWait, onBrowserFallback });
const selection = config.affectedTests
@@ -616,7 +620,7 @@ export async function runSubmitGate(root, config, { base, trunk, changed, check,
? (proofFiles.length > 0 && (!allowBrowserFreeChecks || await testsNeedBrowser(root, proofFiles, { onFallback: onBrowserFallback })))
: (selection.full || !allowBrowserFreeChecks || await testsNeedBrowser(root, proofFiles, { onFallback: onBrowserFallback }));
const browser = checkUsesBrowser || proofUsesBrowser;
- return await withGateSlot(root, (lease) => {
+ return await withGateSlot(root, async (lease) => {
const startedAt = Date.now();
const log = gitPath(root, 'pullboard-submit.log');
const checkLog = `${log}.check.log`;
@@ -646,6 +650,7 @@ export async function runSubmitGate(root, config, { base, trunk, changed, check,
};
if (!criterion.isGreen) {
writeFileSync(log, criterion.output);
+ if (onFailure) await onFailure({ ...criterion, log: checkLog }, { root, command: check, lease });
return finish({ isGreen: false, sandboxBlocked: criterion.sandboxBlocked, isCached: false, output: criterion.output, seconds: 0, log: '' });
}
// The proof already owns this lease: its phase profile has no additional queue wait.
@@ -661,6 +666,8 @@ export async function runSubmitGate(root, config, { base, trunk, changed, check,
}
const phase = deferFullFallback ? `submit invariants: ${invariants.length ? invariants.join(', ') : 'none'}; full suite deferred to the landing gate` : selection.full ? 'full gate' : 'affected tests';
writeFileSync(log, `item check:\n${criterion.output}\n${phase}:\n${proof.output}`);
+ if (!proof.isGreen && onFailure) await onFailure(proof, { root,
+ command: !deferFullFallback && selection.full ? config.gate : `${config.affectedTests.trimEnd()} ${proofFiles.map(path => commandWord('./' + path)).join(' ')}`, lease });
return finish(proof);
}, { browser, agent, item, kind, itemCheck: true, onWait });
}
diff --git a/src/help.js b/src/help.js
index 8dea26be..8e4f0e09 100644
--- a/src/help.js
+++ b/src/help.js
@@ -187,6 +187,9 @@ Receipts
when that queue is empty, the action says none waiting or waits for open work
pullboard ledger markdown: what was built, by whom, verified by whom
pullboard log [id] every move, in order
+ pullboard flakes recorded failures, their first and last sightings, counts and repair status
+ pullboard flakes rerun on|off coordinator: opt into one diagnostic pair per failed test; off by default
+ commit pullboard.json to apply; a passing diagnostic never turns the original red green
pullboard rigor failure causes that repeated: on 2 items, or 1 item at 2 commits;
each with its items and whether it reads as work, product or both
pullboard rigor draft <key> "<rule>" --check "<command>" --red <commit> --green <commit> [--section <letter>]
@@ -257,6 +260,7 @@ const HELP_EXAMPLES = {
claim: 'pullboard claim 12', submit: 'pullboard submit 12', 'next --verify': 'pullboard next --verify',
fact: 'pullboard fact 12 measurement "The check passes in 8 seconds"',
why: 'pullboard why 12',
+ flakes: 'pullboard flakes', 'flakes rerun': 'pullboard flakes rerun on',
verify: 'pullboard verify 12 accept --note "removed the fix; the test failed"',
watch: 'pullboard watch --to web', answer: 'pullboard answer 12 "done"', pass: 'pullboard pass 12 "please decide"',
status: 'pullboard status', view: 'pullboard view', digest: 'pullboard digest --out briefing.md', mcp: 'pullboard mcp', stats: 'pullboard stats --since 2026-10-08', log: 'pullboard log 12', ledger: 'pullboard ledger',
diff --git a/src/json.js b/src/json.js
index c525ab03..3a26128c 100644
--- a/src/json.js
+++ b/src/json.js
@@ -52,6 +52,8 @@ export const JSON_SHAPES = {
decisions: shape({ decisions: 'array' }),
ledger: shape({ items: 'array', stats: 'object' }),
log: shape({ events: 'array' }),
+ flakes: shape({ flakes: 'array', unclassified: 'array', regressions: 'array', stable: 'array' }),
+ 'flakes rerun': shape({ flakes: 'object', file: 'string' }),
rigor: shape({ causes: 'array', uncaused: 'number' }),
'rigor draft': shape({ key: 'string', reads: 'string', evidence: 'object', proof: 'object', drafts: 'array' }),
add: shape({ item: 'object' }),
diff --git a/src/land.js b/src/land.js
index e5d71fb6..c9ae4551 100644
--- a/src/land.js
+++ b/src/land.js
@@ -37,6 +37,10 @@ import {
import { laneOf } from './lanes.js';
import { Refused } from './refused.js';
import { landingFileTimes } from './submit-invariants.js';
+import {
+ flakeSource,
+ observeNativeFailure,
+} from './native-flakes.js';
import {
staleVerdictRefusal,
verdictProjector,
@@ -248,7 +252,15 @@ async function gateGroup(state, label) {
mkdirSync(directory, { recursive: true });
const logPath = join(directory, `${label}-${state.batch.revision}.log`);
await save(state, { state: 'gating', tip: headCommit(state.batch.root), logPath });
- const gate = await runGate(state.batch.root, state.config, { trustStamp: false, landing: true, agent: 'coordinator', onWait: state.onWait, stages: true, waivers: liveWaiverProjection(state.board) });
+ const pin = flakeSource(state.batch.root, { commit, main: state.batch.base });
+ const gate = await runGate(state.batch.root, state.config, { trustStamp: false, landing: true, agent: 'coordinator', onWait: state.onWait, stages: true, waivers: liveWaiverProjection(state.board),
+ onFailure: (run, { root, command, lease }) => observeNativeFailure({ root, config: state.config, pin, run, command, lease, kind: 'land',
+ record: observation => {
+ const owner = observation.file ? laneOf(state.config, observation.file) : 'coordinator';
+ const lane = owner === 'coordinator' ? store.getItem(state.board, state.batch.items[0].id).item_lane : owner;
+ return state.write('recordFlakeObservation', [{ agentId: 'coordinator', observation, lane }]);
+ } }),
+ });
copyFileSync(gate.log, logPath);
if (gate.profilePath) copyFileSync(gate.profilePath, `${logPath}.profile.json`);
if (headCommit(state.batch.root) !== commit || headTree(state.batch.root) !== tree || !isClean(state.batch.root)) {
@@ -262,14 +274,25 @@ async function gateGroup(state, label) {
submissions: state.batch.items.map(({ id, commit }) => ({ id, commit })),
failures: [...new Set(landingFailures(gate.output, state.batch.root, { excludeCancelled: true }).map(failure => failure.file).filter(Boolean))],
fileTimes: landingFileTimes(state.batch.root, gate.profile) };
- await save(state, { gateResults: [...state.gates.values()].map(({ tree: testedTree, isGreen, failures, accounted, complete, requiresGreen, logPath: log }) =>
- ({ tree: testedTree, isGreen, failures, accounted: accounted ?? null, complete: complete ?? null, requiresGreen: Boolean(requiresGreen), logPath: log })),
+ await save(state, { gateResults: [...state.gates.values()].map(({ tree: testedTree, isGreen, failures, accounted, complete, requiresGreen, logPath: log, flakeDiagnostics }) =>
+ ({ tree: testedTree, isGreen, failures, accounted: accounted ?? null, complete: complete ?? null, requiresGreen: Boolean(requiresGreen), logPath: log,
+ ...(flakeDiagnostics ? { flakeDiagnostics } : {}) })),
invariantEvidence: { ...evidence, gates: [...evidence.gates, tested] } });
return result;
}
/** Run each failing file once alone, with the same queued machine capacity as other checks. */
-async function diagnose(state, failures) {
+async function diagnose(state, failures, gate) {
+ // The enabled observer already made the only allowed exact-target pair under the original lease.
+ // A saved gate retains that association, so adopting it cannot repeat a confidence measurement.
+ if (state.config.flakes?.rerun === true) {
+ const observations = gate?.flakeDiagnostics?.observations ?? [];
+ if (gate?.flakeDiagnostics?.error || !observations.length || observations.some(row => !row.recorded || row.classification === 'unclassified')) {
+ await save(state, { state: 'blocked', failures, flakeDiagnostics: gate?.flakeDiagnostics });
+ throw new Refused('LAND_DIAGNOSTIC', `the original red has no complete classified diagnostic; no additional run or landing is allowed; inspect ${gate.logPath} and pullboard flakes`);
+ }
+ return [...new Set(observations.filter(row => row.classification === 'flake' && row.file).map(row => row.file))];
+ }
if (!failures.length || failures.some((entry) => !entry.file)) throw new Refused('LAND_DIAGNOSTIC', `the gate did not name a failing test file; inspect ${state.batch.logPath} and make its reporter name source files`);
const files = [...new Set(failures.map((entry) => entry.file))];
const flakes = [];
@@ -441,7 +464,7 @@ async function bisect(state, items, failures, label = 'split') {
const gate = await gateGroup(state, `${label}-${index}`);
if (gate.isGreen || waivedProof(state, gate)) green.push(...build.items);
else {
- const flakyFiles = await diagnose(state, gate.failures);
+ const flakyFiles = await diagnose(state, gate.failures, gate);
if (flakyFiles.length) {
await fileFlakes(state, gate, flakyFiles);
state.flaky = true;
@@ -467,6 +490,11 @@ async function returnCulprits(state) {
/** File a flake once against its named test and leave the batch stopped. */
async function fileFlakes(state, gate, files) {
+ if (state.config.flakes?.rerun === true) {
+ await save(state, { state: 'blocked', failures: gate.failures, flakeDiagnostics: gate.flakeDiagnostics });
+ state.say(`flake observed: ${files.join(', ')}; the original gate stays red and nothing lands; inspect pullboard flakes`);
+ return;
+ }
const flakes = [];
for (const file of files) {
const owner = laneOf(state.config, file);
@@ -619,7 +647,7 @@ export async function land({ root, board, config, write, say, owner, maximum = I
let gate = await gateGroup(state, 'batch');
let proof = gate.isGreen ? null : waivedProof(state, gate);
if (!gate.isGreen && !proof) {
- const flakyFiles = await diagnose(state, gate.failures);
+ const flakyFiles = await diagnose(state, gate.failures, gate);
if (flakyFiles.length) { await fileFlakes(state, gate, flakyFiles); state.flaky = true; }
else {
const green = await bisect(state, build.items, gate.failures);
diff --git a/src/landing-batch-history.js b/src/landing-batch-history.js
new file mode 100644
index 00000000..7339a4e6
--- /dev/null
+++ b/src/landing-batch-history.js
@@ -0,0 +1,10 @@
+/** Read-only projection of append-only landing receipts, shared by board operations and clients. */
+/** Read the latest append-only snapshot of each landing batch [B3,R3,R2]. */
+export function landingBatches(board) {
+ const batches = new Map();
+ for (const row of board.db.prepare("SELECT event_detail FROM event WHERE event_kind = 'landing_batch' ORDER BY event_id").all()) {
+ const { batch } = JSON.parse(row.event_detail);
+ batches.set(batch.id, batch);
+ }
+ return [...batches.values()];
+}
diff --git a/src/machine.js b/src/machine.js
index 96cde24f..9eec5164 100644
--- a/src/machine.js
+++ b/src/machine.js
@@ -10,6 +10,7 @@
*/
import { RIGOR_NOTICE_ENGINE } from './rigor-source.js';
+import { NATIVE_FLAKE_ENGINE } from './flake-engine.js';
/** First writer for withdraw-and-remove milestone behavior; shipment remains engine 45. */
export const WITHDRAW_MILESTONE_ENGINE = 51;
@@ -32,6 +33,7 @@ export const SUBMITTED_CONFLICT_ENGINE = 60;
export const ENGINE_VERSION = Math.max(
SUBMITTED_CONFLICT_ENGINE,
+ NATIVE_FLAKE_ENGINE,
ROW_DRAFT_ENGINE,
AUTO_ESCALATE_ENGINE,
RIGOR_NOTICE_ENGINE,
diff --git a/src/native-flake-cli.js b/src/native-flake-cli.js
new file mode 100644
index 00000000..cff1366e
--- /dev/null
+++ b/src/native-flake-cli.js
@@ -0,0 +1,60 @@
+/** CLI diagnostic adapters keep policy capture, execution and evidence inside the existing gate admission. */
+import { readFileSync } from 'node:fs';
+import { COORDINATOR } from './config.js';
+import { laneOf } from './lanes.js';
+import {
+ checkAtCommit,
+ mainPolicy,
+ policyAt,
+} from './trusted-policy.js';
+import { runProfiledShell } from './gate.js';
+import {
+ diagnosticUnavailable,
+ flakeSource,
+ observeNativeFailure,
+} from './native-flakes.js';
+export { flakesCommand as command } from './flake-command.js';
+
+/** Capture owner policy and source before admission, recording through the CLI's ordinary ordered board moves. */
+export function observer(ctx, { kind, board, main, commit, privateCheck = false, config = ctx.config } = {}, { ordered, withBoard, resourceAgent }) {
+ let acceptedConfig, pin, agentId;
+ try {
+ const owner = main ? policyAt(ctx.info.root, main) : mainPolicy(ctx.info.root);
+ acceptedConfig = { ...config, flakes: owner.config.flakes };
+ pin = flakeSource(ctx.info.root, { main: main ?? owner.commit, commit, privateCheck });
+ agentId = resourceAgent(ctx);
+ } catch (error) {
+ // Diagnostics cannot block an existing check when its coordinator is detached or its observer cannot initialize.
+ return async run => { if (!run.isGreen) diagnosticUnavailable({ run }, error); };
+ }
+ return async (run, { root = ctx.info.root, command, lease }) => {
+ await observeNativeFailure({ root, config: acceptedConfig, pin, run, command, lease, kind,
+ record: observation => {
+ const ownerLane = observation.file ? laneOf(acceptedConfig, observation.file) : COORDINATOR;
+ const lane = ownerLane === COORDINATOR ? Object.keys(acceptedConfig.lanes)[0] : ownerLane;
+ const write = target => ordered(ctx, target, 'recordFlakeObservation', [{ agentId, observation, lane }]);
+ return board ? write(board) : withBoard(ctx, write);
+ } });
+ };
+}
+
+/** Prepare an ordinary check callback before it queues; the callback observes red under that same lease. */
+export function check(ctx, item, command, services) {
+ const observe = observer(ctx, { kind: 'check' }, services);
+ return async lease => {
+ const run = runProfiledShell(ctx.info.root, command, { waitMs: lease.waitMs, artifactPrefix: `pullboard-check-${item.item_id}`, persistLog: true, gateLease: lease });
+ if (!run.isGreen) await observe(run, { command, lease });
+ return run;
+ };
+}
+
+/** Prepare the private frozen-check callback without losing its retained raw output, profile or execution root. */
+export function verify(ctx, item, command, options, services) {
+ const observe = observer(ctx, { ...options, kind: 'verify', privateCheck: true }, services);
+ return async lease => {
+ const result = checkAtCommit(ctx.info.root, item, { waitMs: lease.waitMs, gateLease: lease });
+ if (result.state === 'red') await observe({ isGreen: false, output: result.outputPath ? readFileSync(result.outputPath, 'utf8') : result.output,
+ log: result.outputPath, profilePath: result.profilePath, executionRoot: result.executionRoot, executionSourceUnchanged: result.executionSourceUnchanged }, { root: ctx.info.root, command, lease });
+ return result;
+ };
+}
diff --git a/src/native-flakes.js b/src/native-flakes.js
new file mode 100644
index 00000000..6d2a2930
--- /dev/null
+++ b/src/native-flakes.js
@@ -0,0 +1,255 @@
+/** One bounded diagnostic pair under the existing admission; the original gate remains authoritative [C7,H16]. */
+import {
+ createHash,
+ randomUUID,
+} from 'node:crypto';
+import {
+ availableParallelism,
+ freemem,
+ loadavg,
+} from 'node:os';
+import {
+ closeSync,
+ constants,
+ copyFileSync,
+ mkdirSync,
+ openSync,
+ readFileSync,
+ readSync,
+ readdirSync,
+ statSync,
+ writeFileSync,
+} from 'node:fs';
+import {
+ isAbsolute,
+ join,
+ relative,
+} from 'node:path';
+import {
+ headCommit,
+ headTree,
+ invalidateGitFacts,
+ isClean,
+ tryGit,
+ untracked,
+} from './git.js';
+import {
+ checkAtCommit,
+ mainPolicy,
+} from './trusted-policy.js';
+import { fileURLToPath } from 'node:url';
+import { runProfiledShell } from './gate.js';
+
+/** Hash immutable bytes without retaining a potentially credential-bearing command in board data. */
+function digest(value) { return createHash('sha256').update(value).digest('hex'); }
+
+/** Hash large raw evidence in bounded chunks rather than loading another complete log into memory. */
+function fileDigest(file) {
+ const hash = createHash('sha256');
+ const descriptor = openSync(file, 'r');
+ const buffer = Buffer.alloc(65536);
+ try {
+ let length;
+ while ((length = readSync(descriptor, buffer, 0, buffer.length, null))) hash.update(buffer.subarray(0, length));
+ } finally { closeSync(descriptor); }
+ return hash.digest('hex');
+}
+
+/** Quote one literal selector argument, preserving its bytes without shell interpolation. */
+function word(value) { return "'" + value.replaceAll("'", "'\\''") + "'"; }
+
+/** Bind the source before the original run; uncertainty is recorded rather than silently retried. */
+export function flakeSource(root, { commit, main, privateCheck = false } = {}) {
+ invalidateGitFacts();
+ const head = commit ?? headCommit(root);
+ const tree = head && tryGit(root, ['rev-parse', '--verify', head + '^{tree}']);
+ let trunk = main ?? null;
+ if (!trunk) { try { trunk = mainPolicy(root).commit; } catch { /* Missing main prevents classification. */ } }
+ return { head, tree: tree?.status === 0 ? tree.stdout.trim() : null, main: trunk, privateCheck,
+ clean: privateCheck || isClean(root) && untracked(root).length === 0,
+ load: { average: loadavg(), cpus: availableParallelism(), freeMB: Math.floor(freemem() / 1048576) } };
+}
+
+/** Read structured Node TAP blocks only; nested selectors and cancelled parents stay unsupported. */
+function tapTests(output, root) {
+ const rows = [];
+ let active;
+ for (const raw of output.split('\n')) {
+ const match = /^(\s*)(not ok|ok)\s+\d+\s+-\s+(.*)$/u.exec(raw);
+ if (match) {
+ active = { name: match[3], passed: match[2] === 'ok', indent: match[1].length, file: null, skip: /\s+#\s*(?:SKIP|TODO)\b/iu.test(match[3]), type: null, failureType: null };
+ rows.push(active); continue;
+ }
+ if (!active) continue;
+ const field = /^(\s*)(location|type|failureType):\s*(.*?)\s*$/u.exec(raw);
+ if (!field || field[1].length !== active.indent + 2) continue;
+ const value = field[3].replace(/^(['"])(.*)\1$/u, '$2');
+ if (field[2] === 'location') {
+ const path = value.replace(/:\d+(?::\d+)?$/u, '');
+ const file = isAbsolute(path) ? relative(root, path) : path.replace(/^\.\//u, '');
+ if (file && !file.startsWith('../') && !file.includes('\\')) active.file = file;
+ } else active[field[2]] = value;
+ }
+ return rows;
+}
+
+/** Use only a known structured Node runner; arbitrary shell, Jest and Vitest commands are never replaced. */
+function nodePrefix(command, config, root, pin) {
+ const executable = '(?:node|/[^\\s\'";|&`$]+/node)';
+ const own = new RegExp('^(' + executable + ')\\s+((?:\\./)?bin/run-tests\\.js)(?:\\s|$)', 'u');
+ const builtin = new RegExp('^(' + executable + ')\\s+--test(?:\\s|$)', 'u');
+ const match = own.exec(command);
+ if (match) return knownRunner(root, pin) ? match[1] + ' ' + match[2] : null;
+ const native = builtin.exec(command);
+ if (native) {
+ const rest = command.slice(native[0].length);
+ if (/[;&|`$()<>\r\n]/u.test(rest) || /(?:^|\s)--(?!test-reporter=tap(?:\s|$))\S+/u.test(rest)) return null;
+ return native[1] + ' --test --test-reporter=tap';
+ }
+ if (new RegExp('^' + executable + '\\s', 'u').test(command)) return null;
+ const configured = typeof config.affectedTests === 'string' ? own.exec(config.affectedTests) : null;
+ return configured && knownRunner(root, pin) && config.affectedTests.trim() === configured[1] + ' ' + configured[2]
+ ? config.affectedTests.trim() : null;
+}
+
+/** Recognize the bundled runner by committed bytes, never by a conventional filename in an end-user repo. */
+function knownRunner(root, pin) {
+ try {
+ const bundled = tryGit(root, ['hash-object', '--no-filters', fileURLToPath(new URL('../bin/run-tests.js', import.meta.url))]);
+ if (bundled.status !== 0) return false;
+ return [...new Set([pin.head, pin.main].filter(Boolean))].every(commit => {
+ if (!committedTest(root, commit, 'bin/run-tests.js')) return false;
+ const source = tryGit(root, ['rev-parse', '--verify', commit + ':bin/run-tests.js']);
+ return source.status === 0 && source.stdout === bundled.stdout;
+ });
+ } catch { return false; }
+}
+
+/** Confirm the file is a committed regular test at the pinned source, without reading a live substitute. */
+function committedTest(root, commit, file) {
+ if (!file || !/\.(?:[cm]?js)$/u.test(file) || file.startsWith('/') || file.includes('\\') || file.split('/').some(part => !part || part === '..' || part === '.')) return false;
+ const result = tryGit(root, ['--literal-pathspecs', 'ls-tree', '-z', commit, '--', file]);
+ return result.status === 0 && /^(?:100644|100755) blob [a-f0-9]+\t/u.test(result.stdout) && result.stdout.split('\0').filter(Boolean).length === 1;
+}
+
+/** Copy the whole output/profile to unique durable evidence before any diagnostic executes. */
+function saveReceipt(directory, id, label, run) {
+ const file = join(directory, label + '.log');
+ const source = run.log || run.logPath;
+ if (source) copyFileSync(source, file, constants.COPYFILE_EXCL);
+ else writeFileSync(file, run.output ?? '', { flag: 'wx', mode: 0o600 });
+ let profileSha256 = null;
+ if (run.profilePath) {
+ copyFileSync(run.profilePath, join(directory, label + '.profile.json'), constants.COPYFILE_EXCL);
+ profileSha256 = fileDigest(run.profilePath);
+ }
+ return { green: run.isGreen, sha256: fileDigest(file), bytes: statSync(file).size, artifact: 'flake-evidence/' + id + '/' + label + '.log', profileSha256 };
+}
+
+/** A probe proves one selected target only if its real completion is present, not merely a zero exit. */
+function targetResult(run, root, name) {
+ if (run.executionSourceUnchanged === false) return null;
+ const matches = tapTests(run.output ?? '', run.executionRoot ?? root).filter(row => row.type !== 'suite' && !row.skip);
+ if (matches.length !== 1 || matches[0].indent !== 0 || matches[0].name !== name || ['subtestsFailed', 'cancelledByParent', 'testAborted'].includes(matches[0].failureType)) return null;
+ return matches[0].passed === run.isGreen ? run.isGreen : null;
+}
+
+/** Run an exact-target private source comparison, preserving the trusted check's full receipt. */
+function privateProbe(root, commit, command, lease, policyCommit) {
+ const item = { item_id: 0, item_commit: commit, item_claim_head: policyCommit, item_frozen: JSON.stringify({ check: command }) };
+ const check = checkAtCommit(root, item, { gateLease: lease });
+ const output = check.outputPath ? readFileSync(check.outputPath, 'utf8') : '';
+ return { isGreen: check.state === 'pass', unverified: check.state === 'unverified', output, log: check.outputPath,
+ executionRoot: check.executionRoot, executionSourceUnchanged: check.executionSourceUnchanged, profilePath: check.profilePath, profile: check.profile };
+}
+
+/** Report only a bounded code when diagnostic work fails; never replace the original gate's red result. */
+export async function observeNativeFailure(options) {
+ const started = Date.now();
+ try { await observeFailure(options); }
+ catch (error) { diagnosticUnavailable(options, error, started); }
+}
+
+/** Record a bounded diagnostic failure without replacing the original run or exposing arbitrary exception text. */
+export function diagnosticUnavailable(options, error, started = Date.now()) {
+ const code = diagnosticCode(error);
+ options.run.flakeDiagnostics = { ...(options.run.flakeDiagnostics ?? { observations: [] }), wallMs: Date.now() - started, error: code };
+ const message = `failure diagnostics could not finish (${code}); the original result stays red; inspect the retained gate log and pullboard flakes\n`;
+ try {
+ if (typeof options.onError === 'function') options.onError({ code, message });
+ else process.stderr.write(message);
+ } catch { /* An unavailable diagnostic channel cannot change the original verdict. */ }
+}
+
+/** Retain an operational code without echoing arbitrary command text or credentials from an exception. */
+function diagnosticCode(error) {
+ return /^[A-Z][A-Z0-9_]{0,79}$/u.test(error?.code ?? '') ? error.code : 'FLAKE_DIAGNOSTIC_FAILED';
+}
+
+/** Observe one red run, optionally run one pair, then persist typed results without changing its verdict. */
+async function observeFailure({ root, config, pin, run, command, lease, kind, record }) {
+ if (run.isGreen || !pin.head || !pin.tree) return;
+ const started = Date.now();
+ const observedAt = new Date().toISOString();
+ const load = { average: loadavg(), cpus: availableParallelism(), freeMB: Math.floor(freemem() / 1048576) };
+ const prefix = nodePrefix(command, config, root, pin);
+ const parsed = tapTests(run.output ?? '', run.executionRoot ?? root).filter(row => !row.passed && row.type !== 'suite' && !['subtestsFailed', 'cancelledByParent', 'testAborted'].includes(row.failureType));
+ const failures = parsed.length ? parsed : [{ file: null, name: null, indent: null }];
+ const common = tryGit(root, ['rev-parse', '--path-format=absolute', '--git-common-dir']);
+ if (common.status !== 0) throw new Error('Cannot retain flake evidence without the board Git directory.');
+ const evidence = join(common.stdout.trim(), 'pullboard', 'flake-evidence');
+ mkdirSync(evidence, { recursive: true, mode: 0o700 });
+ const observations = [];
+ for (const failure of failures) {
+ const targetStarted = Date.now();
+ const id = randomUUID();
+ const directory = join(evidence, id); mkdirSync(directory, { mode: 0o700 });
+ const original = saveReceipt(directory, id, 'original', run);
+ let paired = null, mainReceipt = null, classification = 'unclassified';
+ let reason = config.flakes?.rerun === true ? 'No supported structured Node test selector is available.' : 'Paired reruns are off for this board; the original red is retained.';
+ const named = failure.file && failure.name && committedTest(root, pin.head, failure.file);
+ invalidateGitFacts();
+ const changed = run.executionSourceUnchanged === false || !pin.privateCheck && (headCommit(root) !== pin.head || headTree(root) !== pin.tree || !isClean(root) || untracked(root).length > 0);
+ if (!pin.clean || changed) reason = 'The source is dirty or changed; no unchanged-head diagnostic is claimed.';
+ else if (lease?.runner) reason = 'This remote admission has no supported pinned-main comparison; no local substitute is run.';
+ else if (config.flakes?.rerun === true && prefix && named && failure.indent === 0) {
+ const pattern = '^' + failure.name.replace(/[.*+?^${}()|[\]\\]/gu, '\\$&') + '$';
+ const selected = prefix + ' --test-name-pattern ' + word(pattern) + ' ' + word(failure.file);
+ const pairRun = pin.privateCheck ? privateProbe(root, pin.head, selected, lease, pin.main ?? pin.head)
+ : runProfiledShell(root, selected, { gateLease: lease });
+ paired = saveReceipt(directory, id, 'paired', pairRun);
+ const pairResult = pairRun.unverified ? null : targetResult(pairRun, root, failure.name);
+ let mainResult = pin.main === pin.head ? false : null;
+ if (pin.main && pin.main !== pin.head && committedTest(root, pin.main, failure.file)) {
+ const mainRun = privateProbe(root, pin.main, selected, lease, pin.main);
+ mainReceipt = saveReceipt(directory, id, 'main', mainRun);
+ mainResult = mainRun.unverified ? null : targetResult(mainRun, root, failure.name);
+ }
+ if (mainResult === true) { classification = 'possible-regression'; reason = 'The exact target passes on pinned main; this candidate failure is not a flake observation.'; }
+ else if (pairResult === true && mainResult === false) { classification = 'flake'; reason = 'The red target passed its one paired rerun at the same pinned head; original red retained.'; }
+ else if (pairResult === false) { classification = 'stable-red'; reason = 'The target remains red in its one paired rerun; original red retained.'; }
+ else reason = 'A probe did not establish the exact target at its pinned source; it remains unclassified.';
+ invalidateGitFacts();
+ if (!pin.privateCheck && (headCommit(root) !== pin.head || headTree(root) !== pin.tree || !isClean(root) || untracked(root).length > 0)) {
+ classification = 'unclassified'; reason = 'The source changed during diagnostics; no unchanged-head classification is claimed.';
+ }
+ }
+ const observation = { id, file: named ? failure.file : null, name: named ? failure.name : null, head: pin.head, tree: pin.tree, main: pin.main,
+ kind, classification, reason, commandSha256: digest(command), at: observedAt, diagnosticMs: Date.now() - targetStarted,
+ load, original, paired, mainReceipt };
+ writeFileSync(join(directory, 'observation.json'), JSON.stringify(observation, null, 2) + '\n', { flag: 'wx', mode: 0o600 });
+ const manifest = Object.fromEntries(readdirSync(directory).map(file => [file, fileDigest(join(directory, file))]));
+ writeFileSync(join(directory, 'manifest.json'), JSON.stringify(manifest, null, 2) + '\n', { flag: 'wx', mode: 0o600 });
+ const summary = { id, classification, file: observation.file, name: observation.name, paired: paired !== null, recorded: false };
+ observations.push(summary);
+ run.flakeDiagnostics = { wallMs: Date.now() - started, observations };
+ try { await record(observation); summary.recorded = true; }
+ catch (error) {
+ // The completed evidence manifest remains immutable; the recording failure gets its own addendum.
+ try { writeFileSync(join(directory, 'recording-error.addendum.json'), JSON.stringify({ code: diagnosticCode(error), originalRetained: true }) + '\n', { flag: 'wx', mode: 0o600 }); }
+ catch { /* A failing disk cannot justify replacing the original verdict. */ }
+ throw error;
+ }
+ }
+ run.flakeDiagnostics = { wallMs: Date.now() - started, observations };
+}
diff --git a/src/relay-sender.js b/src/relay-sender.js
index 1991f723..ea5ed7cc 100644
--- a/src/relay-sender.js
+++ b/src/relay-sender.js
@@ -10,7 +10,7 @@ export function relayMoveActor(move) {
if (move.operation === 'addItem') return args[0]?.by ?? null;
if (move.operation === 'shout') return args[0]?.from ?? null;
if (['proposeSentinel', 'decideSentinel', 'beginSentinelRun', 'finishSentinelRun'].includes(move.operation)) return args[0]?.agentId ?? null;
- if (move.operation === 'recordHealthFindings') return args[0]?.agentId ?? null;
+ if (['recordHealthFindings', 'recordFlakeObservation'].includes(move.operation)) return args[0]?.agentId ?? null;
if (['holdLane', 'releaseLane'].includes(move.operation) && args[1]?.asPerson) return 'person';
if (move.operation === 'release') return args[1] ?? null;
if (['reserveNextReview', 'addMilestone', 'recordRowDecisions', 'applyRowDecisions', 'draftSpecRow', 'recordLandingBatch', 'recordLandingWaiver', 'recordLandingFlake', 'finishLandingBatch'].includes(move.operation)) return args[0]?.agentId ?? null;
diff --git a/src/rigor.js b/src/rigor.js
index d8e623c0..624aaf0f 100644
--- a/src/rigor.js
+++ b/src/rigor.js
@@ -7,6 +7,7 @@ import { COORDINATOR } from './config.js';
import { Refused } from './refused.js';
import { HEALTH_SENTINEL_ENGINE } from './health-receipts.js';
import { BOARD_SENTINEL_ENGINE } from './sentinel-engine.js';
+import { NATIVE_FLAKE_ENGINE } from './flake-engine.js';
/** The kinds of cause key and their shapes, in the order help and refusals name them. */
export const CAUSE_FORMS = Object.freeze(['test:<file>::<title>', 'refusal:<CODE>', 'row:<id>', 'doctrine:<id>', 'sentinel:<name>']);
@@ -97,8 +98,10 @@ export function causeReading(key) {
* @returns {string[]}
*/
export function eventCauses(event) {
- if (!['reject', 'reopen', 'health-failure', 'sentinel-failure'].includes(event.event_kind)) return [];
+ if (!['reject', 'reopen', 'health-failure', 'sentinel-failure', 'flake-observation'].includes(event.event_kind)) return [];
const detail = JSON.parse(event.event_detail);
+ if (event.event_kind === 'flake-observation') return detail.engine === NATIVE_FLAKE_ENGINE && detail.observation?.classification === 'flake'
+ ? [testCause(detail.observation)] : [];
if (event.event_kind === 'health-failure' && (event.event_by !== 'sentinel' || detail.engine !== HEALTH_SENTINEL_ENGINE)) return [];
if (event.event_kind === 'sentinel-failure' && (event.event_by !== 'sentinel' || detail.engine !== BOARD_SENTINEL_ENGINE)) return [];
const keys = [];
@@ -139,7 +142,7 @@ export function rigorReport(events, { person }) {
let uncaused = 0;
/** The running record for one cause key. */
const entry = (key) => {
- if (!byKey.has(key)) byKey.set(key, { key, reads: causeReading(key), items: new Map(), occurrences: 0, healthOccurrences: 0, neverAgain: false });
+ if (!byKey.has(key)) byKey.set(key, { key, reads: causeReading(key), items: new Map(), occurrences: 0, healthOccurrences: 0, flakeOccurrences: 0, neverAgain: false });
return byKey.get(key);
};
for (const event of events) {
@@ -147,6 +150,8 @@ export function rigorReport(events, { person }) {
&& (event.event_by !== 'sentinel' || JSON.parse(event.event_detail)?.engine !== HEALTH_SENTINEL_ENGINE)) continue;
if (event.event_kind === 'sentinel-failure'
&& (event.event_by !== 'sentinel' || JSON.parse(event.event_detail)?.engine !== BOARD_SENTINEL_ENGINE)) continue;
+ if (event.event_kind === 'flake-observation' && (JSON.parse(event.event_detail)?.engine !== NATIVE_FLAKE_ENGINE
+ || JSON.parse(event.event_detail)?.observation?.classification !== 'flake')) continue;
const never = neverAgainCause(event, person);
if (never && !isMergeConflict(never)) {
const found = entry(never);
@@ -154,7 +159,7 @@ export function rigorReport(events, { person }) {
if (event.item_id !== null && !found.items.has(event.item_id)) found.items.set(event.item_id, new Set());
continue;
}
- if (!['reject', 'reopen', 'health-failure', 'sentinel-failure'].includes(event.event_kind)) continue;
+ if (!['reject', 'reopen', 'health-failure', 'sentinel-failure', 'flake-observation'].includes(event.event_kind)) continue;
const keys = eventCauses(event);
if (!keys.length) {
uncaused += 1;
@@ -164,6 +169,7 @@ export function rigorReport(events, { person }) {
for (const key of keys.filter((candidate) => !isMergeConflict(candidate))) {
const found = entry(key);
found.occurrences += 1;
+ if (event.event_kind === 'flake-observation') found.flakeOccurrences += 1;
if (['health-failure', 'sentinel-failure'].includes(event.event_kind)) found.healthOccurrences += 1;
if (event.item_id === null) continue;
const commits = found.items.get(event.item_id) ?? new Set();
@@ -172,7 +178,7 @@ export function rigorReport(events, { person }) {
}
}
const causes = [...byKey.values()]
- .filter(found => found.neverAgain || found.healthOccurrences >= 2 || found.items.size >= 2 || [...found.items.values()].some(commits => commits.size >= 2))
+ .filter(found => found.neverAgain || found.healthOccurrences >= 2 || found.flakeOccurrences >= 2 || found.items.size >= 2 || [...found.items.values()].some(commits => commits.size >= 2))
.map((found) => ({ key: found.key, reads: found.reads, occurrences: found.occurrences, neverAgain: found.neverAgain,
items: [...found.items].map(([id, commits]) => ({ id, commits: [...commits] })).sort((a, b) => a.id - b.id) }))
.sort((a, b) => Number(b.neverAgain) - Number(a.neverAgain) || b.items.length - a.items.length || b.occurrences - a.occurrences || a.key.localeCompare(b.key));
diff --git a/src/trusted-policy.js b/src/trusted-policy.js
index f6adadb8..4f751867 100644
--- a/src/trusted-policy.js
+++ b/src/trusted-policy.js
@@ -211,12 +211,14 @@ export function checkAtCommit(root, item, { waitMs = 0, gateSlotHeld = false, ga
let installLogPath = '';
let checkLogPath = '';
let timingProfile = { files: [], tests: [] };
+ let checkedSourceReady = false;
/** Format one completed check phase with the private captures and durable artifact locations. */
const finish = (state, stage) => {
const runner = timingRunner(command);
const events = checkLogPath && existsSync(checkLogPath) ? timingFromLog(checkLogPath, runner) : timingFromOutput(checkOutput, runner);
timingProfile = { ...events, wallMs: Math.max(0, Date.now() - checkStartedAt) };
- return checkResult(state, stage, installOutput, checkOutput, { root, item, commandRoot: copy, installLogPath, checkLogPath, timingProfile, waitMs });
+ return checkResult(state, stage, installOutput, checkOutput, { root, item, commandRoot: copy, installLogPath, checkLogPath, timingProfile, waitMs,
+ executionSourceUnchanged: checkedSourceReady && privateSourceUnchanged(copy, item.item_commit, env) });
};
try {
const common = policyGit(root, ['rev-parse', '--path-format=absolute', '--git-common-dir']).trim();
@@ -242,6 +244,7 @@ export function checkAtCommit(root, item, { waitMs = 0, gateSlotHeld = false, ga
}
const checkout = clone.status === 0 && spawnSync('git', ['checkout', '--quiet', '--detach', item.item_commit], { cwd: copy, env, stdio: 'ignore' });
if (!checkout || checkout.status !== 0) return finish('unverified', 'clone or checkout could not start');
+ checkedSourceReady = true;
const deadline = Date.now() + timeout;
if (install) {
installLogPath = join(scratch, 'install.log');
@@ -404,8 +407,16 @@ function streamSanitizedLog(path, fallback, label, writeArtifact, onLine = () =>
return { tail, sawOutput };
}
+/** Capture the private clone before cleanup, so a diagnostic cannot call edited source an unchanged-head measurement. */
+function privateSourceUnchanged(root, commit, env) {
+ const state = spawnSync('git', ['--no-replace-objects', 'status', '--porcelain=v2', '--branch', '--untracked-files=normal'], { cwd: root, env, encoding: 'utf8' });
+ if (state.status !== 0) return false;
+ const lines = state.stdout.split('\n').filter(Boolean);
+ return lines.includes('# branch.oid ' + commit) && lines.every(line => line.startsWith('# '));
+}
+
/** Persist a sanitized full diagnostic outside the temporary checkout so a refusal can name it. */
-function checkResult(state, stage, installCapture, checkCapture, { root = '', item = {}, commandRoot = root, installLogPath = '', checkLogPath = '', timingProfile = null, waitMs = 0 } = {}) {
+function checkResult(state, stage, installCapture, checkCapture, { root = '', item = {}, commandRoot = root, installLogPath = '', checkLogPath = '', timingProfile = null, waitMs = 0, executionSourceUnchanged } = {}) {
const boundedInstall = secretScanned(installCapture, 'check install capture');
const boundedCheck = secretScanned(checkCapture, 'frozen check capture');
const install = digestOf(boundedInstall) || '(no output)';
@@ -480,5 +491,7 @@ function checkResult(state, stage, installCapture, checkCapture, { root = '', it
const failure = state === 'red' ? 'failed' : /timed out/u.test(stage) ? 'timed out'
: /could not start/u.test(stage) ? "couldn't start" : state === 'unverified' ? 'could not be verified' : 'passed';
const output = `install output:\n${boundedInstall || '(no output)'}\ncheck output:\n${boundedCheck || '(not run or no output)'}`;
- return { sandboxBlocked, state, green: state === 'pass', checked: true, stage, failure, report, output, outputTail, outputPath, outputError, profilePath, profile: timingProfile ? { ...timingProfile, profilePath } : null };
+ return { sandboxBlocked, state, green: state === 'pass', checked: true, stage, failure, report, output, outputTail, outputPath, outputError, profilePath,
+ executionRoot: commandRoot, ...(executionSourceUnchanged === undefined ? {} : { executionSourceUnchanged }),
+ profile: timingProfile ? { ...timingProfile, profilePath } : null };
}
diff --git a/test/api.test.js b/test/api.test.js
index 8bc372aa..d3f14884 100644
--- a/test/api.test.js
+++ b/test/api.test.js
@@ -1006,6 +1006,15 @@ test('[A1] every catalog command and subcommand has a real CLI exercise', () =>
assert.deepEqual(landing.items, []);
json(source, source.repo, 'roadmap');
json(source, source.repo, 'queue');
+ const failures = json(source, source.repo, 'flakes');
+ assert.deepEqual([failures.flakes, failures.unclassified, failures.regressions, failures.stable], [[], [], [], []]);
+ const failurePolicy = join(source.repo, 'pullboard.json');
+ const originalPolicy = readFileSync(failurePolicy, 'utf8');
+ assert.equal(json(source, source.repo, 'flakes', ['rerun', 'on'], 'rerun').flakes.rerun, true);
+ assert.equal(JSON.parse(readFileSync(failurePolicy, 'utf8')).flakes.rerun, true, 'the coordinator edits the private project policy');
+ assert.equal(json(source, source.repo, 'flakes', ['rerun', 'off'], 'rerun').flakes.rerun, false);
+ assert.equal(JSON.parse(readFileSync(failurePolicy, 'utf8')).flakes.rerun, false);
+ writeFileSync(failurePolicy, originalPolicy, 'utf8'); // These uncommitted policy edits do not alter the later catalog fixture.
json(source, source.repo, 'rigor');
const watch = json(source, source.repo, 'sentinel', ['propose', 'catalog-watch', '--check', 'true', '--every', 'hourly', '--owner', 'coordinator'], 'propose');
assert.equal(watch.sentinel.status, 'pending');
diff --git a/test/flakes-ledger.test.js b/test/flakes-ledger.test.js
new file mode 100644
index 00000000..396cbc65
--- /dev/null
+++ b/test/flakes-ledger.test.js
@@ -0,0 +1,101 @@
+/** Synthetic public-store/replay fixtures: these receipt inputs do not claim a gate was executed [H16]. */
+import assert from 'node:assert/strict';
+import { createHash } from 'node:crypto';
+import { test } from 'node:test';
+import * as store from '../src/board.js';
+import { applyEngineMove, appliedSequence, prepareEngineMove } from '../src/engine.js';
+import { exportBoard, importBoard } from '../src/exchange.js';
+import { submitFixture } from './board-submit-fixture.js';
+
+/** Seed only a private in-memory board, using ordinary registration operations and a controlled clock. */
+function ledgerFixture(t) {
+ let time = Date.parse('2026-10-20T00:00:00Z');
+ const board = store.openBoard(':memory:', { now: () => new Date(time) });
+ t.after(() => store.closeBoard(board));
+ store.register(board, { lane: 'coordinator', path: '/synthetic/main' });
+ const builder = store.register(board, { lane: 'app', path: '/synthetic/build', route: 'strong' });
+ const reviewer = store.register(board, { lane: 'app', path: '/synthetic/review', route: 'strong' });
+ return { board, builder, reviewer, advance: () => { time += 1000; }, at: () => new Date(time).toISOString() };
+}
+
+/** Make bounded synthetic receipt metadata solely for validation, ledger and replay contracts. */
+function observation(box, id) {
+ const receipt = green => {
+ const bytes = Buffer.from(green ? 'synthetic green receipt' : 'synthetic red receipt');
+ return { green, sha256: createHash('sha256').update(bytes).digest('hex'), bytes: bytes.length,
+ artifact: 'flake-evidence/' + id + '/' + (green ? 'paired' : 'original') + '.log', profileSha256: null };
+ };
+ return { id, file: 'app/test.test.js', name: 'synthetic ledger target', head: 'a'.repeat(40), tree: 'b'.repeat(40), main: 'a'.repeat(40),
+ kind: 'check', classification: 'flake', reason: 'Synthetic unit fixture only; no gate run is claimed.', commandSha256: 'c'.repeat(64),
+ at: box.at(), diagnosticMs: 1, load: { average: [0, 0, 0], cpus: 1, freeMB: 100 }, original: receipt(false), paired: receipt(true), mainReceipt: null };
+}
+
+/** Exercise the public observation operation with the same lane and registered actor as an actual adapter. */
+function record(box, value) { return store.recordFlakeObservation(box.board, { agentId: box.builder, observation: value, lane: 'app' }); }
+
+test('one observation is immutable and idempotent; a second drafts ordinary repair work [D1,H16]', t => {
+ const box = ledgerFixture(t), first = observation(box, 'synthetic-one');
+ record(box, first);
+ const count = store.events(box.board).length;
+ assert.equal(record(box, first).duplicate, true);
+ assert.equal(store.events(box.board).length, count);
+ assert.equal(store.flakeLedger(box.board)[0].count, 1);
+ assert.equal(store.flakeLedger(box.board)[0].item, null);
+ assert.throws(() => record(box, { ...first, reason: 'different evidence under an existing identity' }), error => error.code === 'FLAKE_OBSERVATION_CHANGED');
+ box.advance(); record(box, observation(box, 'synthetic-two'));
+ const row = store.flakeLedger(box.board)[0], repair = store.getItem(box.board, row.item);
+ assert.equal(row.count, 2); assert.equal(row.observations.length, 2);
+ assert.equal(repair.item_status, 'open'); assert.equal(repair.item_route, 'strong');
+ assert.match(repair.item_brief, /synthetic-one[^]*synthetic-two/u);
+ assert.ok(!store.events(box.board).some(event => /row_decision|doctrine/u.test(event.event_kind)), 'drafting never approves governing rows');
+});
+
+test('a landed repair is fixed until the same test is seen again [D1,H16]', t => {
+ const box = ledgerFixture(t);
+ record(box, observation(box, 'synthetic-one')); box.advance(); record(box, observation(box, 'synthetic-two'));
+ const row = store.flakeLedger(box.board)[0], id = row.item;
+ store.claim(box.board, id, { agentId: box.builder, lane: 'app', leaseMs: 3600000,
+ freeze: () => ({ text: 'synthetic repair criterion', digest: 'd'.repeat(64) }) });
+ submitFixture(box.board, id, { agentId: box.builder, commit: 'e'.repeat(40), tree: 'f'.repeat(40) });
+ store.verify(box.board, id, { agentId: box.reviewer, decision: 'ACCEPT', head: 'e'.repeat(40), digest: 'd'.repeat(64), policy: 'any',
+ note: 'Synthetic history fixture: no engineering acceptance or gate execution is claimed.' });
+ box.advance(); store.merged(box.board, id, { agentId: 'coordinator', commit: '1'.repeat(40), note: 'Synthetic history only; no real repository changed.' });
+ assert.equal(store.flakeLedger(box.board)[0].status, 'fixed');
+ box.advance(); record(box, observation(box, 'synthetic-three'));
+ const again = store.flakeLedger(box.board)[0];
+ assert.equal(again.status, 'active'); assert.equal(again.count, 3);
+ assert.equal(again.firstSeen, row.firstSeen); assert.ok(again.lastSeen > row.lastSeen);
+});
+
+test('flake observations replay exactly and duplicate engine receipts do not add sightings [H16]', t => {
+ const box = ledgerFixture(t), replica = store.openBoard(':memory:');
+ t.after(() => store.closeBoard(replica));
+ importBoard(replica, exportBoard(box.board));
+ for (let sequence = 1; sequence <= 2; sequence++) {
+ const move = prepareEngineMove(box.board, 'recordFlakeObservation', [{ agentId: box.builder, observation: observation(box, 'synthetic-' + sequence), lane: 'app' }],
+ { id: 'synthetic-move-' + sequence, actor: box.builder });
+ const order = { sequence, at: box.at() };
+ const original = applyEngineMove(box.board, move, order), replayed = applyEngineMove(replica, move, order);
+ assert.deepEqual(replayed, original);
+ assert.deepEqual(applyEngineMove(replica, move, order), original, 'the duplicate returns its immutable engine receipt');
+ box.advance();
+ }
+ assert.deepEqual(store.flakeLedger(replica), store.flakeLedger(box.board));
+ assert.equal(store.flakeLedger(replica)[0].count, 2); assert.equal(appliedSequence(replica), 2);
+ const source = exportBoard(box.board), target = exportBoard(replica);
+ for (const [table, rows] of Object.entries(source.tables)) if (table !== 'board_meta') assert.deepEqual(target.tables[table], rows, table);
+});
+
+test('old engines, unjoined actors and impossible evidence cannot add observations [H16]', t => {
+ const box = ledgerFixture(t), value = observation(box, 'synthetic-one');
+ assert.throws(() => store.recordFlakeObservation(box.board, { agentId: 'not-joined', observation: value, lane: 'app' }), error => error.code === 'NOT_JOINED');
+ assert.throws(() => record(box, { ...value, original: { ...value.original, green: true } }), error => error.code === 'FLAKE_EVIDENCE');
+ const move = prepareEngineMove(box.board, 'recordFlakeObservation', [{ agentId: box.builder, observation: value, lane: 'app' }], { id: 'synthetic-old-engine', actor: box.builder });
+ assert.throws(() => applyEngineMove(box.board, { ...move, engine: 58 }, { sequence: 1, at: box.at() }), error => error.code === 'RELAY_MOVE');
+ box.board.executionEngineVersion = 58;
+ try {
+ assert.throws(() => record(box, value), error => error.code === 'ENGINE_VERSION');
+ assert.throws(() => store.recordLandingBatch(box.board, { agentId: 'coordinator', batch: { flakeDiagnostics: {} } }), error => error.code === 'ENGINE_VERSION');
+ } finally { delete box.board.executionEngineVersion; }
+ assert.deepEqual(store.flakeLedger(box.board), []);
+});
diff --git a/test/flakes.test.js b/test/flakes.test.js
new file mode 100644
index 00000000..6224522a
--- /dev/null
+++ b/test/flakes.test.js
@@ -0,0 +1,482 @@
+/** @pullboard-test-input source-tree */
+/** Native flake observations preserve the first red verdict and carry evidence into repair work [C7,D1,H16]. */
+import assert from 'node:assert/strict';
+import { createHash } from 'node:crypto';
+import { appendFileSync, cpSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, symlinkSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join, resolve } from 'node:path';
+import { test } from 'node:test';
+import { runFixtureChild, runFixtureGit } from './fixture-child.js';
+import { flakeSource, observeNativeFailure } from '../src/native-flakes.js';
+import * as store from '../src/board.js';
+import { land } from '../src/land.js';
+
+const BIN = resolve(import.meta.dirname, '../bin/pullboard.js');
+const TITLE = 'a controlled intermittent assertion [C7]';
+
+/** Initialize a real private board and Git history; all gate commands use only this synthetic project. */
+function flakyProject(t, { rerun = true, reporter = 'tap', alwaysPass = false } = {}) {
+ const scratch = realpathSync(mkdtempSync(join(tmpdir(), 'pullboard-native-flake-')));
+ t.after(() => rmSync(scratch, { recursive: true, force: true }));
+ const root = join(scratch, 'repo'); mkdirSync(root);
+ const env = { ...process.env, GIT_CONFIG_GLOBAL: '/dev/null', GIT_CONFIG_NOSYSTEM: '1',
+ GIT_AUTHOR_NAME: 'Fixture Agent', GIT_AUTHOR_EMAIL: '[email protected]',
+ GIT_COMMITTER_NAME: 'Fixture Agent', GIT_COMMITTER_EMAIL: '[email protected]',
+ PULLBOARD_HOME: join(scratch, 'home'), PULLBOARD_MACHINE_HOME: join(scratch, 'machine'), PULLBOARD_VIEW_BACKGROUND: '0' };
+ delete env.PULLBOARD_RELAY_TOKEN; delete env.NODE_TEST_CONTEXT;
+ /** Run Git only within this fixture with a private deterministic identity. */
+ const git = (...args) => runFixtureGit(args, { cwd: root, env });
+ /** Exercise the real CLI, retaining only its structured receipt for assertions. */
+ const cliAt = (cwd, ...args) => {
+ const result = runFixtureChild(process.execPath, [BIN, ...args, '--json'], { cwd, env, encoding: 'utf8' });
+ return { code: result.status, document: result.stdout?.trim() ? JSON.parse(result.stdout) : null, failure: result.failure };
+ };
+ /** Keep the coordinator's normal command path separate from its actual joined worktrees. */
+ const cli = (...args) => cliAt(root, ...args);
+ git('init', '-q', '-b', 'main');
+ assert.equal(cli('init').code, 0);
+ mkdirSync(join(root, 'node_modules/.bin'), { recursive: true });
+ symlinkSync(BIN, join(root, 'node_modules/.bin/pullboard'));
+ appendFileSync(join(root, '.git/info/exclude'), '\nnode_modules/\n');
+ const file = join(root, 'pullboard.json');
+ const config = JSON.parse(readFileSync(file, 'utf8'));
+ config.gate = `node --test --test-reporter=${reporter} test/alternating.test.mjs`;
+ config.flakes = rerun === null ? undefined : { rerun };
+ // These synthetic checks intentionally match the already-configured gate; a private warning policy permits the controlled failure to be armed after submission.
+ config.checks = { ...config.checks, provesNothing: 'warn' };
+ config.lanes = { build: { owns: ['src/', 'test/'], specs: [] } };
+ writeFileSync(file, JSON.stringify(config, null, 2) + '\n');
+ mkdirSync(join(root, 'test'));
+ const counter = join(scratch, 'observed-target-runs');
+ if (alwaysPass === 'controlled') writeFileSync(counter + '.mode', 'pass');
+ writeTarget(root, counter, alwaysPass);
+ git('add', '-A'); git('commit', '-q', '-m', 'chore: prepare private flake fixture');
+ return { root, scratch, env, git, cli, cliAt, counter, lane: 'build' };
+}
+
+/** Keep the exact same test identity on main and a candidate, with independently controlled behavior. */
+function writeTarget(root, counter, alwaysPass = false, mutate = false) {
+ const assertion = "assert.equal(readFileSync(marker, 'utf8').trim().split('\\n').length % 2, 0);";
+ writeFileSync(join(root, 'test/alternating.test.mjs'), `import assert from 'node:assert/strict';
+import { appendFileSync, readFileSync } from 'node:fs';
+import { test } from 'node:test';
+/** Record real executions; the fixture controls whether the assertion is passing or intermittent. */
+test(${JSON.stringify(TITLE)}, () => {
+ const marker = ${JSON.stringify(counter)};
+ appendFileSync(marker, 'run\\n');
+ ${mutate ? (mutate === 'controlled' ? "if (readFileSync(marker + '.mode', 'utf8') !== 'pass') " : '') + 'appendFileSync(new URL(import.meta.url), ' + JSON.stringify('\n// fixture source changed\n') + ');' : ''}
+ ${alwaysPass === 'controlled' ? "if (readFileSync(marker + '.mode', 'utf8') !== 'pass') " + assertion : alwaysPass ? 'assert.ok(true);' : assertion}
+});\n`);
+}
+
+/** Count the target's actual executions independently of any agent's reported verdict. */
+function targetRuns(box) { return readFileSync(box.counter, 'utf8').trim().split('\n').length; }
+
+/** Verify the whole immutable evidence, including profiles, before the private fixture is removed. */
+function assertReceipts(box, observation) {
+ for (const value of [observation.original, observation.paired, observation.mainReceipt].filter(Boolean)) {
+ const file = join(box.root, '.git/pullboard', value.artifact);
+ const bytes = readFileSync(file);
+ assert.equal(bytes.length, value.bytes);
+ assert.equal(createHash('sha256').update(bytes).digest('hex'), value.sha256);
+ if (value.profileSha256 !== null) assert.equal(createHash('sha256').update(readFileSync(file.replace(/\.log$/u, '.profile.json'))).digest('hex'), value.profileSha256);
+ }
+}
+
+/** Join through the actual CLI, using a separate session-bound worktree for a candidate. */
+function privateBuilder(box) {
+ const made = box.cli('worktree', box.lane);
+ assert.equal(made.code, 0);
+ const root = made.document.path;
+ /** Run a candidate command without taking the coordinator's seat. */
+ const cli = (...args) => box.cliAt(root, ...args);
+ /** Commit only this private candidate's source. */
+ const git = (...args) => runFixtureGit(args, { cwd: root, env: box.env });
+ return { root, cli, git };
+}
+
+/** Set an exact repo-gate check, which the normal baseline shortcut records without a background command. */
+function checkedItem(box) {
+ const check = JSON.parse(readFileSync(join(box.root, 'pullboard.json'), 'utf8')).gate;
+ const added = box.cli('add', box.lane, 'Controlled Native check', '--criterion', 'The controlled target passes without accepting retries.',
+ '--route', 'light', '--brief', 'Files: test/alternating.test.mjs\nTest: run its configured Node TAP check. Preserve the original failed verdict while retaining exact diagnostic receipts.', '--check', check, '--wait');
+ assert.equal(added.code, 0, JSON.stringify(added.document));
+ return added.document.item.item_id;
+}
+
+test('a red gate records one paired pass and drafts repair work on a second sighting [C7,D1,H16]', t => {
+ const box = flakyProject(t);
+ const head = box.git('rev-parse', 'HEAD').trim();
+ const first = box.cli('gate');
+ assert.equal(first.code, 1, 'a paired pass never turns the original red gate green');
+ assert.equal(first.document.green, false);
+ if (targetRuns(box) !== 2) {
+ const diagnostic = box.cli('flakes').document;
+ const observation = diagnostic?.unclassified?.[0]?.observations?.[0];
+ console.log(JSON.stringify({ flakeDiagnostic: diagnostic, originalBody: observation
+ ? readFileSync(join(box.root, '.git/pullboard', observation.original.artifact), 'utf8') : first.document.report }));
+ }
+ assert.equal(targetRuns(box), 2, 'the configured target executes once and gets exactly one paired rerun');
+ const listed = box.cli('flakes');
+ assert.equal(listed.code, 0);
+ assert.equal(listed.document.flakes.length, 1);
+ const row = listed.document.flakes[0];
+ assert.equal(row.file, 'test/alternating.test.mjs');
+ assert.equal(row.name, TITLE);
+ assert.equal(row.count, 1);
+ assert.equal(row.status, 'active');
+ assert.equal(row.observations[0].head, head);
+ assert.equal(row.observations[0].original.green, false);
+ assert.equal(row.observations[0].paired.green, true);
+ assertReceipts(box, row.observations[0]);
+ assert.equal(row.item, null, 'one sighting records evidence without inventing a repair item');
+ const second = box.cli('gate');
+ assert.equal(second.code, 1);
+ assert.equal(second.document.green, false);
+ assert.equal(targetRuns(box), 4, 'a second red adds one new pair, no confidence loop');
+ const repeated = box.cli('flakes').document.flakes[0];
+ assert.equal(repeated.count, 2);
+ assert.ok(repeated.firstSeen <= repeated.lastSeen);
+ assert.ok(Number.isSafeInteger(repeated.item), 'Rigor drafts one ordinary fix item after the second confirmed sighting');
+ const item = box.cli('show', String(repeated.item)).document;
+ assert.equal(item.item_status, 'open');
+ assert.match(item.item_title, /flaky test/iu);
+ assert.match(item.item_brief, /original.*paired/isu, 'the repair work carries both observations as evidence');
+ for (const observation of repeated.observations) assertReceipts(box, observation);
+ const causes = box.cli('rigor').document.causes;
+ assert.equal(causes.find(row => row.key === 'test:test/alternating.test.mjs::' + TITLE)?.occurrences, 2);
+ assert.equal(box.git('status', '--porcelain').trim(), '', 'Rigor changes no source, spec or doctrine file');
+ assert.equal(box.git('rev-parse', 'HEAD').trim(), head, 'diagnostics and drafting never move the source head');
+});
+
+test('the ledger records a red by default without running a second command [C7,H16]', t => {
+ const box = flakyProject(t, { rerun: null });
+ assert.equal(box.cli('gate').code, 1);
+ assert.equal(targetRuns(box), 1);
+ const report = box.cli('flakes').document;
+ assert.equal(report.flakes.length, 0);
+ assert.equal(report.unclassified.length, 1);
+ const observation = report.unclassified[0].observations[0];
+ assert.equal(observation.paired, null);
+ assert.match(observation.reason, /off/iu);
+ assertReceipts(box, observation);
+});
+
+test('an unsupported Node reporter stays unclassified rather than guessing a selector [C7,H16]', t => {
+ const box = flakyProject(t, { reporter: 'spec' });
+ assert.equal(box.cli('gate').code, 1);
+ assert.equal(targetRuns(box), 1);
+ const report = box.cli('flakes').document;
+ assert.equal(report.flakes.length, 0);
+ assert.equal(report.unclassified[0].observations[0].paired, null);
+ assertReceipts(box, report.unclassified[0].observations[0]);
+});
+
+test('the pinned bundled runner selects one target and skips unrelated bodies [C7,H16]', t => {
+ const box = flakyProject(t);
+ const unrelated = join(box.scratch, 'bundled-unrelated-runs');
+ mkdirSync(join(box.root, 'bin'));
+ cpSync(resolve(import.meta.dirname, '../bin/run-tests.js'), join(box.root, 'bin/run-tests.js'));
+ cpSync(resolve(import.meta.dirname, '../src'), join(box.root, 'src'), { recursive: true });
+ writeFileSync(join(box.root, 'package.json'), '{"type":"module"}\n');
+ writeFileSync(join(box.root, 'test/unrelated.test.mjs'), `import { test } from 'node:test';
+import { appendFileSync } from 'node:fs';
+test('unrelated bundled execution [C7]', () => appendFileSync(${JSON.stringify(unrelated)}, 'run\\n'));
+`);
+ const file = join(box.root, 'pullboard.json');
+ const config = JSON.parse(readFileSync(file, 'utf8'));
+ config.gate = 'node bin/run-tests.js test/alternating.test.mjs test/unrelated.test.mjs';
+ config.affectedTests = 'node bin/run-tests.js';
+ writeFileSync(file, JSON.stringify(config, null, 2) + '\n');
+ box.git('add', '-A'); box.git('commit', '-q', '-m', 'chore: private bundled runner');
+ assert.equal(box.cli('gate').code, 1);
+ assert.equal(targetRuns(box), 2, 'the known pinned runner executes only its one exact-target pair');
+ assert.equal(readFileSync(unrelated, 'utf8'), 'run\n', 'the unrelated body executes only in the original gate');
+ const report = box.cli('flakes').document;
+ assert.equal(report.flakes.length, 1);
+ assert.equal(report.flakes[0].observations[0].paired.green, true);
+ assertReceipts(box, report.flakes[0].observations[0]);
+});
+
+test('an unknown script named run-tests cannot rerun unrelated tests or confirm a flake [C7,H16]', t => {
+ const box = flakyProject(t);
+ const unrelated = join(box.scratch, 'observed-unrelated-runs');
+ mkdirSync(join(box.root, 'bin'));
+ writeFileSync(join(box.root, 'bin/run-tests.js'), `import { spawnSync } from 'node:child_process';
+// This user's runner deliberately does not implement Node's test-name selector.
+const run = spawnSync(process.execPath, ['--test', '--test-reporter=tap', 'test/alternating.test.mjs', 'test/unrelated.test.mjs'], { env: process.env, stdio: 'inherit' });
+process.exitCode = run.status;
+`);
+ writeFileSync(join(box.root, 'test/unrelated.test.mjs'), `import { test } from 'node:test';
+import { appendFileSync } from 'node:fs';
+test('unrelated execution [C7]', () => appendFileSync(${JSON.stringify(unrelated)}, 'run\\n'));
+`);
+ const file = join(box.root, 'pullboard.json');
+ const config = JSON.parse(readFileSync(file, 'utf8'));
+ config.gate = 'node bin/run-tests.js';
+ config.affectedTests = 'node bin/run-tests.js';
+ writeFileSync(file, JSON.stringify(config, null, 2) + '\n');
+ box.git('add', '-A'); box.git('commit', '-q', '-m', 'chore: private unknown runner');
+ assert.equal(box.cli('gate').code, 1, 'the original assertion remains red');
+ assert.equal(targetRuns(box), 1, 'a conventional script filename does not authorize a paired rerun');
+ assert.equal(readFileSync(unrelated, 'utf8'), 'run\n', 'the unrelated test runs only in the original gate');
+ const report = box.cli('flakes').document;
+ assert.equal(report.flakes.length, 0);
+ assert.equal(report.unclassified.length, 1);
+ assert.equal(report.unclassified[0].observations[0].paired, null);
+ assertReceipts(box, report.unclassified[0].observations[0]);
+});
+
+test('a green pinned main excludes a candidate that passes its paired rerun [C7,H16]', t => {
+ const box = flakyProject(t, { alwaysPass: true });
+ const main = box.git('rev-parse', 'HEAD').trim();
+ const builder = privateBuilder(box);
+ writeTarget(builder.root, box.counter);
+ builder.git('add', '-A'); builder.git('commit', '-q', '-m', 'chore: prepare private candidate');
+ const head = builder.git('rev-parse', 'HEAD').trim();
+ const gate = builder.cli('gate');
+ assert.equal(gate.code, 1);
+ assert.equal(gate.document.green, false);
+ assert.equal(targetRuns(box), 3, 'one original, one candidate pair and one independently pinned main comparison');
+ const report = box.cli('flakes').document;
+ assert.equal(report.flakes.length, 0);
+ assert.equal(report.regressions.length, 1);
+ assert.equal(report.regressions[0].item, null);
+ const observation = report.regressions[0].observations[0];
+ assert.equal(observation.head, head);
+ assert.equal(observation.main, main);
+ assert.equal(observation.paired.green, true);
+ assert.equal(observation.mainReceipt.green, true);
+ assertReceipts(box, observation);
+});
+
+test('a source change during the original run prevents a paired measurement [C7,H16]', t => {
+ const box = flakyProject(t);
+ writeTarget(box.root, box.counter, false, true);
+ box.git('add', '-A'); box.git('commit', '-q', '-m', 'chore: prepare controlled source mutation');
+ assert.equal(box.cli('gate').code, 1);
+ assert.equal(targetRuns(box), 1);
+ const report = box.cli('flakes').document;
+ assert.equal(report.flakes.length, 0);
+ assert.equal(report.unclassified[0].observations[0].paired, null);
+ assert.match(report.unclassified[0].observations[0].reason, /changed/iu);
+ assertReceipts(box, report.unclassified[0].observations[0]);
+});
+
+test('the actual check command retains its red and records one pair [C7,V2,H16]', t => {
+ const box = flakyProject(t);
+ const id = checkedItem(box);
+ const result = box.cli('check', String(id), '--yes');
+ assert.equal(result.code, 1);
+ assert.equal(result.document.green, false);
+ assert.equal(targetRuns(box), 2);
+ const observation = box.cli('flakes').document.flakes[0].observations[0];
+ assert.equal(observation.kind, 'check');
+ assert.equal(observation.original.green, false);
+ assert.equal(observation.paired.green, true);
+ assertReceipts(box, observation);
+});
+
+test('a red submit records diagnostics without submitting the item [C7,V4,H16]', t => {
+ const box = flakyProject(t);
+ const id = checkedItem(box);
+ const builder = privateBuilder(box);
+ const claimed = builder.cli('claim', String(id));
+ assert.equal(claimed.code, 0, JSON.stringify(claimed.document));
+ builder.git('commit', '--allow-empty', '-q', '-m', 'chore: controlled submission candidate');
+ const result = builder.cli('submit', String(id));
+ assert.equal(result.code, 1);
+ assert.equal(box.cli('show', String(id)).document.item_status, 'claimed');
+ assert.equal(targetRuns(box), 3, 'one original and pair on candidate, plus one pinned main comparison');
+ const observation = box.cli('flakes').document.flakes[0].observations[0];
+ assert.equal(observation.kind, 'submit');
+ assert.equal(observation.original.green, false);
+ assert.equal(observation.paired.green, true);
+ assertReceipts(box, observation);
+});
+
+test('an independent private verification check stays red after its one passing pair [C7,V4,H16]', t => {
+ const box = flakyProject(t, { alwaysPass: 'controlled' });
+ const id = checkedItem(box);
+ const builder = privateBuilder(box);
+ const claimed = builder.cli('claim', String(id));
+ assert.equal(claimed.code, 0, JSON.stringify(claimed.document));
+ builder.git('commit', '--allow-empty', '-q', '-m', 'chore: controlled review candidate');
+ const submitted = builder.cli('submit', String(id));
+ assert.equal(submitted.code, 0, 'the fixture submits with a real green proof before its external failure is enabled');
+ const head = box.cli('show', String(id)).document.item_commit;
+ writeFileSync(box.counter, ''); writeFileSync(box.counter + '.mode', 'paired');
+ const reviewer = privateBuilder(box);
+ reviewer.git('checkout', '-q', '--detach', head);
+ const result = reviewer.cli('verify', String(id), 'accept');
+ assert.equal(result.code, 1);
+ assert.equal(result.document.error.code, 'CHECK_RED');
+ const item = box.cli('show', String(id)).document;
+ assert.equal(item.item_status, 'submitted', 'a green diagnostic cannot authorize a verdict');
+ assert.equal(item.item_verdict, null);
+ assert.equal(targetRuns(box), 3, 'one private original, one private candidate pair and one pinned main control');
+ const observation = box.cli('flakes').document.flakes[0].observations[0];
+ assert.equal(observation.kind, 'verify');
+ assert.equal(observation.head, head);
+ assert.equal(observation.original.green, false);
+ assert.equal(observation.paired.green, true);
+ assertReceipts(box, observation);
+});
+
+test('a frozen private check that changes its source records no unchanged-source pair [C7,V4,H16]', t => {
+ const box = flakyProject(t, { alwaysPass: 'controlled' });
+ const id = checkedItem(box);
+ const builder = privateBuilder(box);
+ assert.equal(builder.cli('claim', String(id)).code, 0);
+ writeTarget(builder.root, box.counter, 'controlled', 'controlled');
+ builder.git('add', '-A'); builder.git('commit', '-q', '-m', 'chore: controlled private-check mutation');
+ const submitted = builder.cli('submit', String(id));
+ assert.equal(submitted.code, 0, JSON.stringify(submitted.document));
+ const head = box.cli('show', String(id)).document.item_commit;
+ writeFileSync(box.counter, ''); writeFileSync(box.counter + '.mode', 'paired');
+ const reviewer = privateBuilder(box);
+ reviewer.git('checkout', '-q', '--detach', head);
+ const result = reviewer.cli('verify', String(id), 'accept');
+ assert.equal(result.document.error?.code, 'CHECK_RED');
+ assert.equal(targetRuns(box), 1, 'changing the private checked source prevents a diagnostic pair');
+ const report = box.cli('flakes').document;
+ assert.equal(report.flakes.length, 0);
+ const observation = report.unclassified[0].observations[0];
+ assert.equal(observation.paired, null);
+ assert.match(observation.reason, /source.*changed/iu);
+ assertReceipts(box, observation);
+});
+
+/** Prepare one genuinely submitted and accepted candidate with a private local bare destination. */
+function readyLanding(t, rerun) {
+ const box = flakyProject(t, { alwaysPass: 'controlled', rerun });
+ const remote = join(box.scratch, 'private-origin.git');
+ box.git('init', '-q', '--bare', remote);
+ box.git('remote', 'add', 'origin', remote);
+ box.git('push', '-q', '-u', 'origin', 'main');
+ const base = box.git('rev-parse', 'HEAD').trim();
+ const id = checkedItem(box);
+ const builder = privateBuilder(box);
+ const claimed = builder.cli('claim', String(id));
+ assert.equal(claimed.code, 0, JSON.stringify(claimed.document));
+ builder.git('commit', '--allow-empty', '-q', '-m', 'chore: prepare controlled landing candidate');
+ const submitted = builder.cli('submit', String(id));
+ assert.equal(submitted.code, 0, JSON.stringify(submitted.document));
+ const head = box.cli('show', String(id)).document.item_commit;
+ const reviewer = privateBuilder(box);
+ reviewer.git('checkout', '-q', '--detach', head);
+ const accepted = reviewer.cli('verify', String(id), 'accept', '--note', 'Private fixture: its exact controlled check passed before the external failure was armed.');
+ assert.equal(accepted.code, 0, JSON.stringify(accepted.document));
+ writeFileSync(box.counter, '');
+ writeFileSync(box.counter + '.mode', 'paired');
+ return { ...box, remote, base, id };
+}
+
+test('enabled landing retains its original red without a second diagnostic or first legacy defect [C7,H16]', t => {
+ const box = readyLanding(t, true);
+ const result = box.cli('land');
+ assert.equal(result.code, 0, JSON.stringify(result.document));
+ assert.equal(result.document.batch.state, 'blocked');
+ assert.deepEqual(result.document.landed, []);
+ assert.deepEqual(result.document.flakes, [], 'typed observations do not file a first-sighting legacy landing_flake');
+ assert.equal(result.document.batch.gateResults[0].isGreen, false, 'the diagnostic pass never replaces the original red');
+ assert.equal(targetRuns(box), 3, 'one original, one exact-target pair, one independently pinned main control; no legacy whole-file remeasurement');
+ const report = box.cli('flakes').document;
+ assert.equal(report.flakes.length, 1);
+ assert.equal(report.flakes[0].count, 1);
+ assert.equal(report.flakes[0].item, null, 'first sighting does not invent repair work');
+ assert.equal(report.flakes[0].observations[0].kind, 'land');
+ assertReceipts(box, report.flakes[0].observations[0]);
+ assert.equal(box.git('rev-parse', 'HEAD').trim(), box.base);
+ assert.equal(box.git('--git-dir=' + box.remote, 'rev-parse', 'main').trim(), box.base, 'the private destination received no landing');
+ const refused = box.cli('land', '--adopt');
+ assert.equal(refused.document.error?.code, 'LAND_ADOPT', 'a terminal blocked batch is not an interrupted landing');
+ assert.equal(targetRuns(box), 3, 'a refused adoption adds no measurement');
+ assert.equal(box.cli('flakes').document.flakes[0].count, 1);
+});
+
+/** Interrupt the dependency at its final blocked write after real diagnostics were durably recorded, then use normal CLI adoption. */
+test('an interrupted landing consumes its saved diagnostic without inventing another measurement [C7,H16]', async t => {
+ const box = readyLanding(t, true);
+ // The direct public API call cannot inherit cliAt's private identity environment; set only this synthetic repo's identity.
+ box.git('config', '--local', 'user.name', 'Fixture Agent');
+ box.git('config', '--local', 'user.email', '[email protected]');
+ const board = store.openBoard(join(box.root, '.git/pullboard/board.sqlite'));
+ try {
+ const config = JSON.parse(readFileSync(join(box.root, 'pullboard.json'), 'utf8'));
+ // A direct API call must not hand Node's outer test-worker protocol to its own configured Node test command.
+ const context = process.env.NODE_TEST_CONTEXT;
+ delete process.env.NODE_TEST_CONTEXT;
+ let completed;
+ try {
+ await assert.rejects(land({ root: box.root, board, config, owner: 'private-interrupted-owner', say: () => {},
+ write: async (operation, args) => {
+ if (operation === 'recordLandingBatch' && args[0].batch.state === 'blocked') throw new Error('controlled interruption after retained diagnostic');
+ return store[operation](board, ...args);
+ } }).then(result => { completed = result; return result; }), /controlled interruption after retained diagnostic/u);
+ } catch (error) {
+ const saved = store.landingBatches(board).at(-1);
+ t.diagnostic(JSON.stringify({ unexpectedCompletion: completed?.batch?.state ?? null, targetRuns: targetRuns(box), savedState: saved?.state }));
+ for (const gate of saved?.gateResults ?? []) if (gate.logPath) t.diagnostic(readFileSync(gate.logPath, 'utf8'));
+ throw error;
+ } finally {
+ if (context === undefined) delete process.env.NODE_TEST_CONTEXT;
+ else process.env.NODE_TEST_CONTEXT = context;
+ }
+ const saved = store.landingBatches(board).at(-1);
+ assert.equal(saved.state, 'gating');
+ assert.equal(saved.gateResults[0].isGreen, false);
+ assert.equal(saved.gateResults[0].flakeDiagnostics.observations[0].recorded, true);
+ } finally { store.closeBoard(board); }
+ assert.equal(targetRuns(box), 3, 'the real interrupted landing completed only its original, pair and pinned-main comparison');
+ const adopted = box.cli('land', '--adopt');
+ assert.equal(adopted.code, 0, JSON.stringify(adopted.document));
+ assert.equal(adopted.document.batch.state, 'blocked');
+ assert.equal(targetRuns(box), 3, 'normal adoption consumes the retained original and pair, never a confidence rerun');
+ assert.equal(box.cli('flakes').document.flakes[0].count, 1, 'adoption does not invent a second sighting');
+ assert.equal(box.git('--git-dir=' + box.remote, 'rev-parse', 'main').trim(), box.base);
+});
+
+test('landing with reruns off preserves the released whole-file diagnosis and first legacy defect [C7,H16]', t => {
+ const box = readyLanding(t, false);
+ const result = box.cli('land');
+ assert.equal(result.code, 0, JSON.stringify(result.document));
+ assert.equal(result.document.batch.state, 'flaky');
+ assert.deepEqual(result.document.landed, []);
+ assert.equal(result.document.flakes.length, 1, 'the released first-sighting legacy defect remains');
+ assert.equal(targetRuns(box), 2, 'the released original and whole-file diagnosis remain, with no new paired rerun');
+ const report = box.cli('flakes').document;
+ assert.equal(report.flakes.length, 0, 'a whole-file diagnosis is not relabeled as exact-target confirmation');
+ assert.equal(report.unclassified.length, 1, 'the new ledger still retains the original red');
+ assert.equal(report.unclassified[0].observations[0].paired, null);
+ assertReceipts(box, report.unclassified[0].observations[0]);
+ assert.equal(box.git('rev-parse', 'HEAD').trim(), box.base);
+ assert.equal(box.git('--git-dir=' + box.remote, 'rev-parse', 'main').trim(), box.base);
+});
+
+test('a ledger recording failure cannot replace the original red or discard its evidence [C7,H16]', async t => {
+ const box = flakyProject(t, { rerun: false });
+ const config = JSON.parse(readFileSync(join(box.root, 'pullboard.json'), 'utf8'));
+ const pin = flakeSource(box.root);
+ const result = runFixtureChild(process.execPath, ['--test', '--test-reporter=tap', 'test/alternating.test.mjs'], { cwd: box.root, env: box.env, encoding: 'utf8' });
+ assert.equal(result.status, 1, 'the original assertion really failed before the forced recording error');
+ const run = { isGreen: false, output: result.stdout + result.stderr };
+ const notices = [];
+ await observeNativeFailure({ root: box.root, config, pin, run, command: config.gate, kind: 'check',
+ record: () => { throw Object.assign(new Error('private injected ledger failure'), { code: 'SYNTHETIC_LEDGER_UNAVAILABLE' }); },
+ onError: notice => notices.push(notice),
+ });
+ assert.equal(run.isGreen, false);
+ assert.equal(targetRuns(box), 1, 'a recording failure adds no execution');
+ assert.equal(run.flakeDiagnostics.error, 'SYNTHETIC_LEDGER_UNAVAILABLE');
+ assert.equal(run.flakeDiagnostics.observations[0].recorded, false, 'the result never claims the missing ledger entry was recorded');
+ assert.equal(notices.length, 1);
+ assert.match(notices[0].message, /original result stays red/u);
+ const directory = join(box.root, '.git/pullboard/flake-evidence', run.flakeDiagnostics.observations[0].id);
+ const retained = JSON.parse(readFileSync(join(directory, 'observation.json'), 'utf8'));
+ assertReceipts(box, retained);
+ const manifest = JSON.parse(readFileSync(join(directory, 'manifest.json'), 'utf8'));
+ for (const [file, digest] of Object.entries(manifest)) assert.equal(createHash('sha256').update(readFileSync(join(directory, file))).digest('hex'), digest);
+ assert.equal(Object.hasOwn(manifest, 'recording-error.addendum.json'), false, 'the original manifest remains immutable');
+ assert.equal(JSON.parse(readFileSync(join(directory, 'recording-error.addendum.json'), 'utf8')).code, 'SYNTHETIC_LEDGER_UNAVAILABLE');
+});
diff --git a/test/verdict-check.test.js b/test/verdict-check.test.js
index 4f378bd7..c50f9605 100644
--- a/test/verdict-check.test.js
+++ b/test/verdict-check.test.js
@@ -1,7 +1,7 @@
/** The frozen check outcome travels with accepts without changing the board schema [V19,V8]. */
import assert from 'node:assert/strict';
-import { runFixtureExecFile as execFileSync, runFixtureChild, runFixtureGit } from './fixture-child.js';
-import { chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { runFixtureExecFile as execFileSync, runFixtureChild, runFixtureGit, safeFixtureDiagnostic } from './fixture-child.js';
+import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { test } from 'node:test';
@@ -30,6 +30,7 @@ function sandbox() {
PULLBOARD_HOME: join(dir, 'pullboard-home'),
PULLBOARD_MACHINE_HOME: join(dir, 'machine-home'),
});
+ delete env.NODE_TEST_CONTEXT; // A fixture CLI/compiler child is a normal process, not its parent's Node test worker.
mkdirSync(env.HOME);
/** Run isolated Git commands for this fixture. */
const git = (cwd, ...args) => runFixtureGit(args, { cwd, env, encoding: 'utf8', stdio: 'pipe' });
@@ -63,7 +64,7 @@ function project(box) {
/** Run a fixture command and fail with its captured diagnostics if it refuses. */
function succeeds(box, cwd, ...args) {
const result = box.run(cwd, ...args);
- assert.equal(result.code, 0, result.failure ?? `${result.out}\n${result.err}`);
+ assert.equal(result.code, 0, result.failure ? `${result.failure}\nstdout:\n${safeFixtureDiagnostic(result.out, box.env)}` : `${result.out}\n${result.err}`);
return result.out;
}
@@ -108,6 +109,18 @@ test('accept records none or green in its event and JSON, and says when unchecke
box.git(repo, 'switch', 'main');
const checked = submitted(box, repo, web, 'Checked accept', 'node --check web/item-3.js');
+ succeeds(box, repo, 'check', String(checked.id), '--yes'); // Diagnostics must not block an existing frozen check while main is detached for review.
+ const checkedSource = join(repo, 'web', `item-${checked.id}.js`);
+ const originalSource = readFileSync(checkedSource, 'utf8');
+ writeFileSync(checkedSource, 'const = ;\n'); // Keep the negative compiler case unambiguous; Node22 can silently accept invalid detected-ESM .js (#65202).
+ const reference = runFixtureChild(process.execPath, ['--check', checkedSource], { cwd: repo, env: box.env, encoding: 'utf8' });
+ assert.equal(reference.status, 1, safeFixtureDiagnostic(JSON.stringify({ phase: 'direct compiler reference', status: reference.status,
+ stdout: reference.stdout, stderr: reference.stderr, source: readFileSync(checkedSource, 'utf8'), inheritedTestContext: Boolean(box.env.NODE_TEST_CONTEXT) }), box.env));
+ const redCheck = box.run(repo, 'check', String(checked.id), '--yes', '--json');
+ assert.equal(redCheck.code, 1, safeFixtureDiagnostic(JSON.stringify({ phase: 'real check after direct compiler red', redCheck, source: readFileSync(checkedSource, 'utf8') }), box.env));
+ assert.equal(JSON.parse(redCheck.out).green, false);
+ assert.match(redCheck.err, /failure diagnostics could not finish \(NO_POLICY\).*original result stays red/u);
+ writeFileSync(checkedSource, originalSource);
const checkedJson = JSON.parse(succeeds(box, repo, 'verify', String(checked.id), 'accept', '--as', 'coordinator', '--note', 'the frozen check passed', '--json'));
assert.equal(checkedJson.check, 'green');
const checkedEvent = JSON.parse(succeeds(box, repo, 'log', String(checked.id), '--json')).events.find((event) => event.event_kind === 'accept');