Skip to content

Autonomous development for real products. Governed agents build from specs you approve, follow the rules you set, and nothing merges until another agent proves it. Your decisions, judgement and taste are the bar that work must clear. Declare your vision and let agents build it.

fix(submit): allow exact known main failures [V4,B3,P4]

Corey Olsonbd93bcfe12f4
diff --git a/docs/lifecycle.md b/docs/lifecycle.md
index 112a4ac0..ad20a8d9 100644
--- a/docs/lifecycle.md
+++ b/docs/lifecycle.md
@@ -164,14 +164,14 @@ Each move checks its guards in this order and refuses with the first one that do
| GUARD_NOT_FOUND | the frozen item check passes at the submitted commit, or its clean reviewed composition for a stale acceptance | the named guard is not in the submitted commit; ask the coordinator to fix it |
| BAR_NOT_READY | an opted-in new bar has a named red baseline, measurement notes, reviewer edges and a revision-bound signature from another parent session, bar ready is on and this bar was frozen or edited after activation | the author completes the brief, then another parent runs pullboard bar sign <id> |
| CRITERION_PROVES_NOTHING | the item's check does not already pass on main, or the person accepted that exact check, only when accepted main sets checks.provesNothing to "hold" | set a check that fails before the work, or the person runs pullboard check accept <id> |
-| GATE_RED | the gate, which submit runs itself every time, is green at HEAD | fix what the digest names, commit, submit again |
-| RUNNER_GUI_SESSION_REQUIRED | the gate, which submit runs itself every time, is green at HEAD | nothing to do: the run reruns locally; log in to a GUI session on the runner before sending GUI-marked files there |
-| LAND_TIMEOUT_CUSTODY | the gate, which submit runs itself every time, is green at HEAD | inspect the saved timed gate cleanup state and stop only its proven owned processes before starting another landing |
-| LAND_GATE_OUTPUT | the gate, which submit runs itself every time, is green at HEAD | inspect the retained timed gate output directory and free disk space, then start another landing |
-| SLOT_DB_LOCKED | the gate, which submit runs itself every time, is green at HEAD | the run never started, so it is not a red; rerun the same command once the named process releases the slot database |
-| TIMING_SLOT_REQUIRED | the gate, which submit runs itself every time, is green at HEAD | start the timing check outside its inherited ordinary lease and acquire the machine gate with timing true |
-| RUNNER_BROWSER_SLOT_REQUIRED | the gate, which submit runs itself every time, is green at HEAD | nothing to do: the run reruns on a local gate slot; if it repeats, set gate.browserByLaunch the same on this machine and that runner |
-| PIPEFAIL_UNAVAILABLE | the gate, which submit runs itself every time, is green at HEAD | rewrite the gate without a pipe |
+| GATE_RED | the item check is green; submission proof is green or every failure matches exact main or a live person waiver | fix what the digest names, commit, submit again |
+| RUNNER_GUI_SESSION_REQUIRED | the item check is green; submission proof is green or every failure matches exact main or a live person waiver | nothing to do: the run reruns locally; log in to a GUI session on the runner before sending GUI-marked files there |
+| LAND_TIMEOUT_CUSTODY | the item check is green; submission proof is green or every failure matches exact main or a live person waiver | inspect the saved timed gate cleanup state and stop only its proven owned processes before starting another landing |
+| LAND_GATE_OUTPUT | the item check is green; submission proof is green or every failure matches exact main or a live person waiver | inspect the retained timed gate output directory and free disk space, then start another landing |
+| SLOT_DB_LOCKED | the item check is green; submission proof is green or every failure matches exact main or a live person waiver | the run never started, so it is not a red; rerun the same command once the named process releases the slot database |
+| TIMING_SLOT_REQUIRED | the item check is green; submission proof is green or every failure matches exact main or a live person waiver | start the timing check outside its inherited ordinary lease and acquire the machine gate with timing true |
+| RUNNER_BROWSER_SLOT_REQUIRED | the item check is green; submission proof is green or every failure matches exact main or a live person waiver | nothing to do: the run reruns on a local gate slot; if it repeats, set gate.browserByLaunch the same on this machine and that runner |
+| PIPEFAIL_UNAVAILABLE | the item check is green; submission proof is green or every failure matches exact main or a live person waiver | rewrite the gate without a pipe |
| ITEM_LEASE_LOST | the caller still owns the same live item claim or review and can renew it against the frozen criterion, while submit or an accept waits for a machine slot, and once immediately before admission | claim or reserve the item again before retrying the command |
| A5_GRAMMAR_VERSION | the caller still owns the same live item claim or review and can renew it against the frozen criterion, while submit or an accept waits for a machine slot, and once immediately before admission | upgrade Pullboard or restore a spec written for grammar 1, then retry |
| SPEC_BEHIND_MAIN | the caller still owns the same live item claim or review and can renew it against the frozen criterion, while submit or an accept waits for a machine slot, and once immediately before admission | merge main into this worktree, then renew the claim and retry |
diff --git a/src/board.js b/src/board.js
index 38e71024..57302eb6 100644
--- a/src/board.js
+++ b/src/board.js
@@ -1156,7 +1156,7 @@ export function submit(board, id, { agentId, commit, tree, files = [], policyCom
gateGreen: null,
treeStillDuringGate: null,
submissionProofValid: () => {
- if ((board.executionEngineVersion ?? ENGINE_VERSION) >= SUBMISSION_PROOF_ENGINE) recordedProof = submissionProof(proof);
+ if ((board.executionEngineVersion ?? ENGINE_VERSION) >= SUBMISSION_PROOF_ENGINE) recordedProof = submissionProof(proof, board.executionEngineVersion ?? ENGINE_VERSION);
return null;
},
childrenDone: () => {
diff --git a/src/commands/submit.js b/src/commands/submit.js
index 112325d9..f837863f 100644
--- a/src/commands/submit.js
+++ b/src/commands/submit.js
@@ -86,6 +86,8 @@ import {
withBoard,
} from '../cli.js';
import { recordedRecovery } from '../recorded-recovery.js';
+import { failureSignatures } from '../main-failure-matches.js';
+import { matchingMainFailures } from '../main-failure-matches.js';
/**
* Submit: a clean tree, nothing untracked, the gate green at HEAD, then the board (V4).
@@ -196,15 +198,27 @@ export async function submitHere(ctx, id, infra) {
...(acceptedMain.config.submitInvariants.length ? { affectedTests: acceptedMain.config.affectedTests } : {}) };
const gate = await runSubmitGate(root, submitConfig, { base: claimHead, trunk: acceptedMain.commit, check: frozenCheck(held), agent: me.id, item: String(id), onWait: queuedItemLease(ctx, held, me.id, 'build'), onBrowserFallback: browserAdmissionReporter(ctx.io, ctx.io.relayJson),
onFailure: nativeFlakes.observer(ctx, { kind: 'submit', main: acceptedMain.commit, config: submitConfig }, { ordered, withBoard, resourceAgent }) });
- // Only the person's live waivers can pass a red proof, failure by failure; the item's own check is never waived [B3].
+ // Only fully accounted exact-main failures or live person waivers pass a red proof; the item check stays mandatory [B3].
const waived = !gate.isGreen && gate.check?.green ? withBoard(ctx, (board) => waivedFailures(board, accountedFailures(gate, root))) : null;
- if (!gate.isGreen && !waived) {
- const message = `the submission proof is red at ${commit.slice(0, 12)}; fix it, commit, submit again. ${submitGateReport(gate)}`;
+ let comparison;
+ let knownMainFailures;
+ const failures = !gate.isGreen && gate.check?.green && !waived ? failureSignatures(gate, root) : null;
+ if (failures) {
+ try {
+ comparison = await compareMainFailures(root, gate.profile, gateWaitReporter(ctx.io), gate.output, { commit: acceptedMain.commit, detailed: true });
+ knownMainFailures = matchingMainFailures(failures, comparison, acceptedMain.commit);
+ } catch { /* Unavailable exact-main proof never authorizes a red submission. */ }
+ }
+ if (!gate.isGreen && !waived && !knownMainFailures) {
+ const accountingNote = gate.check?.green && !failures ? '\nknown-main acceptance unavailable: the proof has no complete named failure diagnostics' : '';
+ const message = `the submission proof is red at ${commit.slice(0, 12)}; fix it, commit, submit again. ${submitGateReport(gate)}${accountingNote}`;
// No submission can follow this verdict. Read-only main comparisons must not retain the seat's write lease.
CHECKOUT_LEASES.get(ctx.io)?.release();
CHECKOUT_LEASES.delete(ctx.io);
- let comparison = { main: null, files: [], unavailable: 'main comparison could not be completed' };
- try { comparison = await compareMainFailures(root, gate.profile, gateWaitReporter(ctx.io), gate.output); } catch { /* Keep the original gate refusal if main evidence is unavailable. */ }
+ comparison ??= { main: null, files: [], unavailable: 'main comparison could not be completed' };
+ if (!failures) {
+ try { comparison = await compareMainFailures(root, gate.profile, gateWaitReporter(ctx.io), gate.output); } catch { /* Keep the original gate refusal if main evidence is unavailable. */ }
+ }
const mainNote = withBoard(ctx, board => mainRedMessage(root, board, gate.profile, comparison));
throw new Refused('GATE_RED', `${message}${mainNote ? `\n${mainNote}` : ''}`, 'fix the reported test or gate, commit the change, and run pullboard submit <id> again');
}
@@ -221,18 +235,22 @@ export async function submitHere(ctx, id, infra) {
refuseUnverifiedStack(root, board, store.getItem(board, id), commit, currentTrunk);
// The waived run is written to the item's thread before the submission, so no waived proof reads as green.
if (waived) await ordered(ctx, board, 'appendFact', [id, { agentId: me.id, kind: 'measurement', text: `submission proof at ${commit.slice(0, 12)} was red and is recorded as waived, not green; every failure is covered by a live person waiver:\n${waivedLines.join('\n')}` }]);
+ if (knownMainFailures) await ordered(ctx, board, 'appendFact', [id, { agentId: me.id, kind: 'measurement', text: `submission proof at ${commit.slice(0, 12)} remains red; known main failures at ${acceptedMain.commit}: ${knownMainFailures.failures.map(row => `${row.file} (${row.name})`).join('; ')}. The item check passed; no new or unaccounted failure is accepted.` }]);
return await ordered(ctx, board, 'submit', [id, { agentId: me.id, commit, tree: headTree(root) ?? '', files: filesSince(root, id, claimHead, commit), policyCommit: acceptedMain.commit, ...(infraRecord ? { infra: infraRecord } : {}),
- submissionProof: { version: 1, files: gate.files, full: gate.full, deferred: gate.deferred, reason: gate.reason, ...(gate.deferredFiles ? { deferredFiles: gate.deferredFiles } : {}) },
+ submissionProof: { version: 1, files: gate.files, full: gate.full, deferred: gate.deferred, reason: gate.reason, ...(gate.deferredFiles ? { deferredFiles: gate.deferredFiles } : {}), ...(knownMainFailures ? { knownMainFailures } : {}) },
...(gate.deferred ? { note: gate.deferredFiles ? deferredSelectionNote(gate.deferredFiles) : 'full suite deferred to the landing gate' } : {}) }]);
});
// A large submit still goes through; the note only suggests a chain of smaller items (#1162).
const sizeNote = oversizeNote(changedLines(root, currentTrunk.commit, commit), acceptedMain.config.itemSizeBudget);
if (sizeNote) ctx.io.say(`note: ${sizeNote}`);
- const report = waived
+ const report = knownMainFailures
+ ? `submission proof red with ${knownMainFailures.failures.length} known main failure(s) at ${knownMainFailures.main}; item check green; no new failures accepted. ${submitGateReport(gate)}`
+ : waived
? `submission proof waived, not green: ${waived.length} failure(s), each covered by a live person waiver; the whole output is in ${gate.log}\n${waivedLines.join('\n')}`
: submitGateReport(gate);
return reportSubmission(ctx, id, commit, { green: gate.isGreen, report, files: gate.files, full: gate.full, deferred: gate.deferred, deferFullFallback: gate.deferFullFallback, ...(gate.deferredFiles ? { deferredFiles: gate.deferredFiles } : {}), reason: gate.reason, check: gate.check,
log: gate.log, profile: gate.profile, profilePath: gate.profilePath, proofLog: gate.proofLog, proofProfilePath: gate.proofProfilePath,
+ ...(knownMainFailures ? { knownMainFailures } : {}),
...(waived ? { waived: waived.map(({ file, name, waiver }) => ({ file, name, waiver: waiver.id, reason: waiver.reason, until: waiver.until })) } : {}) }, doctrine);
}
diff --git a/src/known-main-failures-engine.js b/src/known-main-failures-engine.js
new file mode 100644
index 00000000..b0834620
--- /dev/null
+++ b/src/known-main-failures-engine.js
@@ -0,0 +1,4 @@
+/** Known-main submission receipts preserve exact failure identity and historical replay [V4,B3]. */
+import { unassignedEngine } from './engine-next.js';
+
+export const KNOWN_MAIN_FAILURES_ENGINE = unassignedEngine();
diff --git a/src/machine.js b/src/machine.js
index d1a79f70..b87fca78 100644
--- a/src/machine.js
+++ b/src/machine.js
@@ -132,7 +132,7 @@ export const GUARDS = [
{ id: 'barReady', refuse: 'BAR_NOT_READY', rule: 'an opted-in new bar has a named red baseline, measurement notes, reviewer edges and a revision-bound signature from another parent session', next: 'the author completes the brief, then another parent runs pullboard bar sign <id>', source: 'board', when: 'bar ready is on and this bar was frozen or edited after activation' },
{ id: 'checkProvesSomething', refuse: 'CRITERION_PROVES_NOTHING', rule: "the item's check does not already pass on main, or the person accepted that exact check", next: 'set a check that fails before the work, or the person runs pullboard check accept <id>', source: 'cli', when: 'only when accepted main sets checks.provesNothing to "hold"' },
- { id: 'gateGreen', refuse: 'GATE_RED', alsoRefuses: [{ code: 'RUNNER_GUI_SESSION_REQUIRED', next: 'nothing to do: the run reruns locally; log in to a GUI session on the runner before sending GUI-marked files there' }, { code: 'LAND_TIMEOUT_CUSTODY', next: 'inspect the saved timed gate cleanup state and stop only its proven owned processes before starting another landing' }, { code: 'LAND_GATE_OUTPUT', next: 'inspect the retained timed gate output directory and free disk space, then start another landing' }, { code: 'SLOT_DB_LOCKED', next: 'the run never started, so it is not a red; rerun the same command once the named process releases the slot database' }, { code: 'TIMING_SLOT_REQUIRED', next: 'start the timing check outside its inherited ordinary lease and acquire the machine gate with timing true' }, { code: 'RUNNER_BROWSER_SLOT_REQUIRED', next: 'nothing to do: the run reruns on a local gate slot; if it repeats, set gate.browserByLaunch the same on this machine and that runner' }, { code: 'PIPEFAIL_UNAVAILABLE', next: 'rewrite the gate without a pipe' }], rule: 'the gate, which submit runs itself every time, is green at HEAD', next: 'fix what the digest names, commit, submit again', source: 'cli' },
+ { id: 'gateGreen', refuse: 'GATE_RED', alsoRefuses: [{ code: 'RUNNER_GUI_SESSION_REQUIRED', next: 'nothing to do: the run reruns locally; log in to a GUI session on the runner before sending GUI-marked files there' }, { code: 'LAND_TIMEOUT_CUSTODY', next: 'inspect the saved timed gate cleanup state and stop only its proven owned processes before starting another landing' }, { code: 'LAND_GATE_OUTPUT', next: 'inspect the retained timed gate output directory and free disk space, then start another landing' }, { code: 'SLOT_DB_LOCKED', next: 'the run never started, so it is not a red; rerun the same command once the named process releases the slot database' }, { code: 'TIMING_SLOT_REQUIRED', next: 'start the timing check outside its inherited ordinary lease and acquire the machine gate with timing true' }, { code: 'RUNNER_BROWSER_SLOT_REQUIRED', next: 'nothing to do: the run reruns on a local gate slot; if it repeats, set gate.browserByLaunch the same on this machine and that runner' }, { code: 'PIPEFAIL_UNAVAILABLE', next: 'rewrite the gate without a pipe' }], rule: 'the item check is green; submission proof is green or every failure matches exact main or a live person waiver', next: 'fix what the digest names, commit, submit again', source: 'cli' },
{ id: 'queuedItemLease', refuse: 'ITEM_LEASE_LOST', alsoRefuses: [{ code: 'A5_GRAMMAR_VERSION', next: 'upgrade Pullboard or restore a spec written for grammar 1, then retry' }, { code: 'SPEC_BEHIND_MAIN', next: 'merge main into this worktree, then renew the claim and retry' }, { code: 'NOT_A_REPO', next: 'restore the worktree Git metadata, then retry the command' }, { code: 'NO_CONFIG', next: 'restore pullboard.json in this worktree, then retry the command' }], rule: 'the caller still owns the same live item claim or review and can renew it against the frozen criterion', next: 'claim or reserve the item again before retrying the command', source: 'cli', when: 'while submit or an accept waits for a machine slot, and once immediately before admission' },
{ id: 'treeStillDuringGate', refuse: 'MOVED_DURING_GATE', rule: 'when the gate ends, HEAD and every tracked file are as they were when it started', next: 'leave the worktree alone until the gate finishes, then submit again', source: 'cli' },
{ id: 'childrenDone', refuse: 'CHILDREN_OPEN', rule: 'every child item is verified or withdrawn', next: 'finish the child items, or the coordinator withdraws them', source: 'board' },
diff --git a/src/main-failure-matches.js b/src/main-failure-matches.js
new file mode 100644
index 00000000..289197ad
--- /dev/null
+++ b/src/main-failure-matches.js
@@ -0,0 +1,54 @@
+/** Match completely accounted test failures without granting a file-wide main-red exemption [V4,B3]. */
+import { createHash } from 'node:crypto';
+import { realpathSync } from 'node:fs';
+import { accountedFailures } from './land.js';
+
+/** Keep deterministic failure diagnostics, excluding only timing, source location and stack presentation. */
+function diagnostics(output, root) {
+ const rows = [];
+ let active;
+ let omitted = false;
+ for (const line of output.split('\n')) {
+ const start = /^(\s*)not ok\s+\d+\s+-\s+(.*)$/u.exec(line);
+ if (start) {
+ active = { name: start[2].replace(/\s+#.*$/u, ''), indent: start[1].length + 2, lines: [] };
+ rows.push(active); omitted = false; continue;
+ }
+ if (/^\s*ok\s+\d+/u.test(line)) { active = null; continue; }
+ if (!active || !line.startsWith(' '.repeat(active.indent))) continue;
+ const value = line.slice(active.indent);
+ if (/^(?:duration_ms|location|stack):/u.test(value)) { omitted = true; continue; }
+ if (/^[\w-]+:/u.test(value)) omitted = false;
+ if (!omitted && value !== '---' && value !== '...') active.lines.push(value.replaceAll(root, '<root>'));
+ }
+ return rows.map(row => ({ ...row,
+ signature: row.lines.some(line => line.startsWith('error:')) && row.lines.some(line => line.startsWith('failureType:'))
+ ? createHash('sha256').update(row.lines.join('\n')).digest('hex') : null }));
+}
+
+/** Null means some process failure or diagnostic cannot be accounted for; it never authorizes acceptance. */
+export function failureSignatures(run, root) {
+ let executionRoot = run.executionRoot ?? root;
+ if (!run.executionRunner) {
+ try { executionRoot = realpathSync(executionRoot); }
+ catch { /* An unavailable path cannot invent a failure location. */ }
+ }
+ const failures = accountedFailures({ ...run, executionRoot }, executionRoot);
+ if (!failures) return null;
+ const stages = run.stages.flatMap(stage => stage.pipeline ?? [stage]);
+ const details = stages.filter(stage => stage.status !== 0).flatMap(stage => diagnostics(stage.output ?? '', executionRoot));
+ if (details.length !== failures.length) return null;
+ return failures.map((failure, index) => ({ ...failure, signature: details[index].name === failure.name ? details[index].signature : null }))
+ .every(row => row.file && row.signature) ? failures.map((failure, index) => ({ ...failure, signature: details[index].signature })) : null;
+}
+
+/** Every actual candidate failure must match the exact main test and its failure details. */
+export function matchingMainFailures(failures, comparison, main) {
+ if (!failures?.length || comparison.main !== main || comparison.unavailable) return null;
+ for (const failure of failures) {
+ const file = comparison.files.find(row => row.file === failure.file);
+ const matches = file?.failures?.filter(row => row.file === failure.file && row.name === failure.name);
+ if (file?.result !== 'red' || matches?.length !== 1 || matches[0].signature !== failure.signature) return null;
+ }
+ return { main, failures };
+}
diff --git a/src/main-red.js b/src/main-red.js
index 5d3f7ebd..7cfc6ce5 100644
--- a/src/main-red.js
+++ b/src/main-red.js
@@ -19,6 +19,7 @@ import { tmpdir } from 'node:os';
import { stripVTControlCharacters } from 'node:util';
import { fileURLToPath } from 'node:url';
import {
+ runProfiledShell,
runProfiledShellAsync,
withGateSlot,
} from './gate.js';
@@ -34,6 +35,7 @@ import { parseSpec } from './spec.js';
import { listItems } from './board.js';
import { takeResource } from './resources.js';
import { Refused } from './refused.js';
+import { failureSignatures } from './main-failure-matches.js';
const CACHE_RELATIVE = 'pullboard/main-red.json';
@@ -102,7 +104,11 @@ function readCache(file) {
if (!/^[0-9a-f]{40,64}$/u.test(commit) || !files || typeof files !== 'object' || Array.isArray(files)) continue;
entries[commit] = {};
for (const [path, outcome] of Object.entries(files)) {
- if (typeof path === 'string' && path && ['red', 'green'].includes(outcome?.result)) entries[commit][path] = { result: outcome.result };
+ if (typeof path === 'string' && path && ['red', 'green'].includes(outcome?.result)) {
+ const failures = Array.isArray(outcome.failures) && outcome.failures.every(row => row?.file === path
+ && typeof row.name === 'string' && row.name && /^[a-f0-9]{64}$/u.test(row.signature)) ? outcome.failures : undefined;
+ entries[commit][path] = { result: outcome.result, ...(failures ? { failures } : {}) };
+ }
}
}
return { version: 1, entries };
@@ -162,10 +168,11 @@ async function withMainCheckout(root, commit, run) {
* @param {object} profile
* @param {(state:object)=>void} [onWait]
* @param {string} [output] - The failed phase's unchanged reporter output.
+ * @param {{commit?: string, detailed?: boolean}} [options] - Capture the submit base and require exact test identities.
* @returns {Promise<{ main: string|null, files: Array<{file:string,result:'red'|'green'}>, unavailable?: string }>}
*/
-export async function compareMainFailures(root, profile, onWait, output = '') {
- const main = mainCheckout(root);
+export async function compareMainFailures(root, profile, onWait, output = '', { commit, detailed = false } = {}) {
+ const main = commit ? { commit } : mainCheckout(root);
if (!main?.commit) return { main: null, files: [], unavailable: 'the primary checkout has no committed main snapshot' };
const policy = policyAt(root, main.commit);
if (typeof policy.config.affectedTests !== 'string' || !policy.config.affectedTests.trim()) {
@@ -181,29 +188,34 @@ export async function compareMainFailures(root, profile, onWait, output = '') {
});
try {
let cache = readCache(file);
- const missing = files.filter(path => !['red', 'green'].includes(cache.entries[main.commit]?.[path]?.result));
+ const missing = files.filter(path => !['red', 'green'].includes(cache.entries[main.commit]?.[path]?.result)
+ || detailed && !Array.isArray(cache.entries[main.commit][path].failures));
let failed = null;
for (const path of missing) {
try {
- await withGateSlot(root, async lease => {
- const measured = await withMainCheckout(root, main.commit, async checkout => {
+ const measured = await withMainCheckout(root, main.commit, async checkout => {
+ // Match #1219: admission belongs to the clean exact-main checkout, including its runner source.
+ return withGateSlot(checkout, async lease => {
const command = `${policy.config.affectedTests.trimEnd()} ${shellWord('./' + path)}`;
- const run = await runProfiledShellAsync(checkout, command, { gateLease: lease, waitMs: lease.waitMs });
- return { result: run.isGreen ? 'green' : 'red' };
- });
- if (measured.unavailable) throw new Error(measured.unavailable);
- // The cache lease excludes other comparisons; persist each result before yielding again.
- cache = readCache(file);
- cache.entries[main.commit] ??= {};
- cache.entries[main.commit][path] = measured;
- writeCache(file, cache);
- }, { diagnostic: true, browser: true, agent: process.env.PULLBOARD_AGENT, kind: 'main-red', onWait });
+ const run = detailed ? runProfiledShell(checkout, command, { gateLease: lease, waitMs: lease.waitMs, stages: true })
+ : await runProfiledShellAsync(checkout, command, { gateLease: lease, waitMs: lease.waitMs });
+ const failures = detailed ? run.isGreen ? [] : failureSignatures(run, checkout) : null;
+ if (detailed && !run.isGreen && !failures) throw new Error('focused main output has no complete named failure account');
+ return { result: run.isGreen ? 'green' : 'red', ...(failures ? { failures } : {}) };
+ }, { diagnostic: true, browser: true, agent: process.env.PULLBOARD_AGENT, kind: 'main-red', onWait });
+ });
+ if (measured.unavailable) throw new Error(measured.unavailable);
+ cache = readCache(file);
+ cache.entries[main.commit] ??= {};
+ cache.entries[main.commit][path] = measured;
+ writeCache(file, cache);
}
catch (error) { failed ??= error; }
}
if (failed) return { main: main.commit, files: [], unavailable: failed.message ?? 'main comparison could not be completed' };
cache = readCache(file);
- return { main: main.commit, files: files.map(path => ({ file: path, result: cache.entries[main.commit][path].result })) };
+ return { main: main.commit, files: files.map(path => ({ file: path, ...(detailed
+ ? cache.entries[main.commit][path] : { result: cache.entries[main.commit][path].result }) })) };
} finally { cacheLease.release(); }
}
@@ -219,6 +231,9 @@ export function mainRedAttribution(root, board, comparison) {
/** Add only evidence the exact-main focused reruns established to an existing GATE_RED refusal. */
export function mainRedMessage(root, board, profile, comparison) {
+ if (comparison.unavailable === 'focused main output has no complete named failure account') {
+ return 'main not checked: focused main output has no complete named failure account';
+ }
if (comparison.unavailable === 'no focused test runner configured') {
return 'main not checked: no focused test runner configured (affectedTests)';
}
diff --git a/src/submit-invariants.js b/src/submit-invariants.js
index 3f6a004e..4128b8bc 100644
--- a/src/submit-invariants.js
+++ b/src/submit-invariants.js
@@ -2,6 +2,7 @@
import { Refused } from './refused.js';
import { isAbsolute } from 'node:path';
import { relative } from 'node:path';
+import { KNOWN_MAIN_FAILURES_ENGINE } from './known-main-failures-engine.js';
export const SUBMISSION_PROOF_ENGINE = 38;
const BUDGET_MS = 60_000;
@@ -15,15 +16,22 @@ function projectFile(value) {
}
/** Normalize the receipt required by modern submits; legacy replay skips this guard. */
-export function submissionProof(value) {
+export function submissionProof(value, engine = KNOWN_MAIN_FAILURES_ENGINE) {
if (!value || value.version !== 1 || typeof value.full !== 'boolean'
|| typeof value.deferred !== 'boolean' || typeof value.reason !== 'string'
|| value.reason.length > 8192 || !Array.isArray(value.files) || value.files.some(file => !projectFile(file))
|| value.deferredFiles !== undefined && (!value.deferred || !Array.isArray(value.deferredFiles) || value.deferredFiles.some(file => !projectFile(file)))) {
throw new Refused('SUBMISSION_PROOF', 'a submission proof needs its version, selected project files, full-fallback reason and deferred flag; rerun submit with the current CLI');
}
+ const known = value.knownMainFailures;
+ if (known !== undefined && (engine < KNOWN_MAIN_FAILURES_ENGINE || !known || !COMMIT.test(known.main ?? '')
+ || !Array.isArray(known.failures) || !known.failures.length || known.failures.some(row => !projectFile(row?.file)
+ || typeof row.name !== 'string' || !row.name || row.name.length > 8192 || !/^[a-f0-9]{64}$/u.test(row.signature ?? '')))) {
+ throw new Refused('SUBMISSION_PROOF', 'known main failures need their exact main commit, project test files, names and failure signatures on a supporting engine');
+ }
return { version: 1, files: [...new Set(value.files)].sort(), full: value.full,
- deferred: value.deferred, reason: value.reason, ...(value.deferredFiles ? { deferredFiles: [...new Set(value.deferredFiles)].sort() } : {}) };
+ deferred: value.deferred, reason: value.reason, ...(value.deferredFiles ? { deferredFiles: [...new Set(value.deferredFiles)].sort() } : {}),
+ ...(known ? { knownMainFailures: { main: known.main, failures: known.failures.map(({ file, name, signature }) => ({ file, name, signature })) } } : {}) };
}
/** Accept only measured finite nonnegative durations; unavailable is always null. */
diff --git a/test/main-red.test.js b/test/main-red.test.js
index 3dc53d8b..4567d9b7 100644
--- a/test/main-red.test.js
+++ b/test/main-red.test.js
@@ -1,4 +1,4 @@
-/** A red submit compares only its failed files with exact main and guides an inherited rework [V1]. */
+/** A red item check remains mandatory and compares its failed files with main for recovery guidance [V1,V4]. */
import assert from 'node:assert/strict';
import '../src/main-red.js'; // The private driver below invokes this production entry point.
import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, statSync, writeFileSync } from 'node:fs';
@@ -36,6 +36,11 @@ function commit(root, env, subject = 'chore: fixture state') {
return runFixtureGit(['rev-parse', 'HEAD'], { cwd: root, env }).trim();
}
+/** Count real runner calls separately from the bar-baseline calls made while defining items. */
+function runCount(box) {
+ return existsSync(box.env.MAIN_RED_RUNS) ? readFileSync(box.env.MAIN_RED_RUNS, 'utf8').trim().split('\n').length : 0;
+}
+
/** Quote one executable path for the fixture's real CLI shim. */
function shellWord(value) { return "'" + String(value).replaceAll("'", "'\\''") + "'"; }
@@ -91,7 +96,8 @@ function addItems(box, rows) {
return rows.map(({ title, specId, route = 'mid' }) => {
const file = { G1: 'test/inherited.test.js', G2: 'test/unrelated.test.js', G3: 'test/earlier.test.js', G4: 'test/middle.test.js' }[specId];
const result = cli(box.root, box.env, 'add', 'core', title, '--specs', specId, '--route', route,
- '--criterion', title, '--check', 'true', '--brief', `Files: ${file}\nTest: ${title}`);
+ // These recovery cases exercise an unsatisfied item check; inherited proof failures have separate acceptance coverage.
+ '--criterion', title, '--check', 'node test/runner.js', '--brief', `Files: ${file}\nTest: ${title}`);
assert.equal(result.code, 0, result.err || result.out);
return Number(/^#(\d+)$/mu.exec(result.out)?.[1]);
});
@@ -277,13 +283,14 @@ test('an inherited red reads native spec reporter failure locations [V1]', t =>
assert.equal(submitter.run('claim', String(submitItem)).code, 0);
put(submitter.root, 'test/inherited.test.js', `${FAILING_TEST}// candidate-only comment\n`);
commit(submitter.root, box.env, 'test(core): select spec-reported red [G2]');
+ const before = runCount(box);
const unexplained = submitter.run('submit', String(submitItem));
assert.match(unexplained.err, /\[INFRA_NOTE_REQUIRED\]/u, 'test edits need an explanation before any inherited-red comparison');
const red = submitter.run('submit', String(submitItem), '--infra', 'append a candidate-only test marker to exercise native spec failure reporting');
inheritedMessage(red, { row: 'G1', item: rework });
const refusedItem = JSON.parse(submitter.run('show', String(submitItem), '--json').out);
assert.equal(refusedItem.thread.some(entry => entry.kind === 'submit'), false, 'a red gate records no successful submit or infra note');
- assert.equal(readFileSync(box.env.MAIN_RED_RUNS, 'utf8').trim().split('\n').length, 2,
+ assert.equal(runCount(box), before + 2,
'the unchanged spec reporter command runs once on the candidate and once on exact main');
});
@@ -305,12 +312,13 @@ test('a red with unreadable failure files says why main was not checked [V1]', t
assert.equal(submitter.run('claim', String(submitItem)).code, 0);
put(submitter.root, 'test/inherited.test.js', `${FAILING_TEST}// candidate-only comment\n`);
commit(submitter.root, box.env, 'test(core): select unreadable runner red [G2]');
+ const before = runCount(box);
const red = submitter.run('submit', String(submitItem), '--infra', 'append a candidate-only test marker to exercise unreadable runner failure reporting');
assert.equal(red.code, 1, red.out || red.err);
assert.match(red.err, /\[GATE_RED\]/u);
assert.match(red.err, /main not checked: could not read which tests failed/u);
assert.doesNotMatch(red.err, /also red on main|inherited main red/u);
- assert.equal(readFileSync(box.env.MAIN_RED_RUNS, 'utf8').trim().split('\n').length, 1,
+ assert.equal(runCount(box), before + 1,
'a location outside the spec reporter failure footer is not guessed to be a failed file');
});
@@ -358,6 +366,7 @@ test('an inherited red names the rework that owns it [V1]', async (t) => {
assert.equal(submitter.run('claim', String(submitItem)).code, 0);
put(submitter.root, 'test/inherited.test.js', `${FAILING_TEST}// candidate-only comment\n`);
commit(submitter.root, box.env, 'test(core): select inherited red [G1]');
+ const before = runCount(box);
let red = submitter.run('submit', String(submitItem), '--infra', 'append a candidate-only test marker to exercise exact-main comparison caching');
inheritedMessage(red, { row: 'G1', item: rework });
const main = runFixtureGit(['rev-parse', 'main'], { cwd: box.root, env: box.env }).trim();
@@ -368,7 +377,7 @@ test('an inherited red names the rework that owns it [V1]', async (t) => {
assert.equal((readFileSync(cachePath).length > 0), true);
assert.equal(statSync(cachePath).mode & 0o077, 0, 'the result cache is private to this machine account');
const firstCount = readFileSync(box.env.MAIN_RED_RUNS, 'utf8').trim().split('\n').length;
- assert.equal(firstCount, 2, 'one selected candidate run and one exact-main run occurred');
+ assert.equal(firstCount, before + 2, 'one selected candidate run and one exact-main run occurred');
put(submitter.root, 'test/inherited.test.js', `${FAILING_TEST}// second candidate-only comment\n`);
commit(submitter.root, box.env, 'test(core): retry inherited red [G1]');
@@ -481,10 +490,11 @@ test('an inherited red names the rework that owns it [V1]', async (t) => {
assert.equal(submitter.run('claim', String(submitItem)).code, 0);
put(submitter.root, 'test/inherited.test.js', `${FAILING_TEST}// candidate-only comment\n`);
commit(submitter.root, box.env, 'test(core): select red without focused runner [G2]');
+ const before = runCount(box);
const red = submitter.run('submit', String(submitItem), '--infra', 'append a candidate-only test marker to exercise missing focused-runner guidance');
assert.equal(red.code, 1, red.out || red.err);
assert.match(red.err, /\[GATE_RED\]/u);
assert.match(red.err, /main not checked: no focused test runner configured \(affectedTests\)/u);
- assert.equal(readFileSync(box.env.MAIN_RED_RUNS, 'utf8').trim().split('\n').length, 1, 'the main comparison does not run a second full gate');
+ assert.equal(runCount(box), before + 1, 'the main comparison does not run a second full gate');
});
});
diff --git a/test/submit-compare.test.js b/test/submit-compare.test.js
index 9c8589ea..e7a6a699 100644
--- a/test/submit-compare.test.js
+++ b/test/submit-compare.test.js
@@ -207,9 +207,17 @@ test('a red submit reruns its failed files one slot at a time within the pool [V
}
assert.equal(maximum, 1, 'one submit leaves the other machine slot available');
assert.equal(active, 0);
- assert.deepEqual(JSON.parse(readFileSync(box.cache, 'utf8')).entries[box.main], {
+ const cached = JSON.parse(readFileSync(box.cache, 'utf8')).entries[box.main];
+ assert.deepEqual(Object.fromEntries(Object.entries(cached).map(([file, row]) => [file, { result: row.result }])), {
'test/a.test.js': { result: 'green' }, 'test/b.test.js': { result: 'red' }, 'test/c.test.js': { result: 'green' },
});
+ assert.deepEqual(cached['test/a.test.js'].failures, []);
+ assert.deepEqual(cached['test/c.test.js'].failures, []);
+ assert.equal(cached['test/b.test.js'].failures.length, 1);
+ const [known] = cached['test/b.test.js'].failures;
+ assert.equal(known.file, 'test/b.test.js');
+ assert.equal(known.name, 'b native comparison [G1]');
+ assert.match(known.signature, /^[a-f0-9]{64}$/u);
assert.equal(rows(box.machineQueue, 'SELECT token FROM holder').length, 0, 'all machine leases are released');
const before = events(box).length;
await assertRed(box, launch(box, 'submit', String(box.item)));
diff --git a/test/submit-main-red.test.js b/test/submit-main-red.test.js
index 1a6f2c98..619daec9 100644
--- a/test/submit-main-red.test.js
+++ b/test/submit-main-red.test.js
@@ -5,6 +5,7 @@ import { chmodSync } from 'node:fs';
import { mkdirSync } from 'node:fs';
import { mkdtempSync } from 'node:fs';
import { realpathSync } from 'node:fs';
+import { readFileSync } from 'node:fs';
import { rmSync } from 'node:fs';
import { writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
@@ -16,6 +17,10 @@ import { DatabaseSync } from 'node:sqlite';
import { AGENT_SHELL_MARKERS } from '../src/person.js';
import { runFixtureChild } from './fixture-child.js';
import { runFixtureGit } from './fixture-child.js';
+import { submissionProof } from '../src/submit-invariants.js';
+import { KNOWN_MAIN_FAILURES_ENGINE } from '../src/known-main-failures-engine.js';
+import { matchingMainFailures } from '../src/main-failure-matches.js';
+import { failureSignatures } from '../src/main-failure-matches.js';
const BIN = join(import.meta.dirname, '../bin/pullboard.js');
@@ -106,8 +111,13 @@ function submittedProof(box) {
test('submit accepts a test that fails the same way on main and refuses one red only with the item [V4,B3]', t => {
const known = fixture(t);
+ const cache = join(known.root, '.git/pullboard/main-red.json');
+ writeFileSync(cache, JSON.stringify({ version: 1, entries: { [known.main]: { 'test/inherited.test.js': { result: 'red' } } } }), { mode: 0o600 });
const result = cli(known, known.builder, 'submit', String(known.item));
- assert.equal(result.status, 0, result.stdout + result.stderr);
+ const proofLog = runFixtureGit(['rev-parse', '--git-path', 'pullboard-submit.log.proof.log'], { cwd: known.builder, env: known.env }).trim();
+ const observed = result.status ? { cache: JSON.parse(readFileSync(cache, 'utf8')),
+ parsed: failureSignatures({ isGreen: false, stages: [{ status: 1, output: readFileSync(proofLog, 'utf8') }] }, known.builder) } : null;
+ assert.equal(result.status, 0, JSON.stringify({ response: result.stdout + result.stderr, observed }));
const report = JSON.parse(result.stdout);
assert.equal(report.gate.green, false, 'an inherited failure stays red in its receipt');
assert.equal(report.gate.knownMainFailures.main, known.main);
@@ -117,10 +127,34 @@ test('submit accepts a test that fails the same way on main and refuses one red
const changed = fixture(t, { candidate: source('shared failure [G1]') + source('new item failure [G1]').split('\n').slice(2).join('\n') });
const refused = cli(changed, changed.builder, 'submit', String(changed.item));
assert.equal(refused.status, 1, refused.stdout + refused.stderr);
- assert.match(refused.stderr, /GATE_RED/u);
+ assert.equal(JSON.parse(refused.stdout).error.code, 'GATE_RED');
assert.equal(submittedProof(changed), null, 'a new failure in an already-red file still prevents submit');
});
+test('known main receipts preserve exact identities and refuse unsupported historical envelopes [V4,B3,H16]', () => {
+ const failure = { file: 'test/one.test.js', name: 'one failure [V4]', signature: 'a'.repeat(64) };
+ const main = 'b'.repeat(40);
+ const proof = { version: 1, files: [failure.file], full: false, deferred: false, reason: '',
+ knownMainFailures: { main, failures: [failure] } };
+ assert.deepEqual(submissionProof(proof, KNOWN_MAIN_FAILURES_ENGINE).knownMainFailures, proof.knownMainFailures);
+ assert.throws(() => submissionProof(proof, KNOWN_MAIN_FAILURES_ENGINE - 1), { code: 'SUBMISSION_PROOF' });
+ const historical = { version: 1, files: [], full: true, deferred: false, reason: 'historical full gate' };
+ assert.deepEqual(submissionProof(historical, KNOWN_MAIN_FAILURES_ENGINE - 1), historical);
+ for (const bad of [{ main, failures: [] }, { main: 'unknown', failures: [failure] },
+ { main, failures: [{ ...failure, file: '../outside.test.js' }] }, { main, failures: [{ ...failure, signature: null }] }]) {
+ assert.throws(() => submissionProof({ ...proof, knownMainFailures: bad }), { code: 'SUBMISSION_PROOF' });
+ }
+ const compared = { main, files: [{ file: failure.file, result: 'red', failures: [failure] }] };
+ assert.deepEqual(matchingMainFailures([failure], compared, main), proof.knownMainFailures);
+ assert.equal(matchingMainFailures([failure], compared, 'c'.repeat(40)), null);
+ assert.equal(matchingMainFailures([failure], { ...compared, unavailable: 'timeout' }, main), null);
+ assert.equal(matchingMainFailures([failure], { main, files: [{ file: failure.file, result: 'red' }] }, main), null);
+ assert.equal(matchingMainFailures([{ ...failure, name: 'different failure' }], compared, main), null);
+ assert.equal(matchingMainFailures([{ ...failure, file: 'test/other.test.js' }], compared, main), null);
+ assert.equal(matchingMainFailures([{ ...failure, signature: 'd'.repeat(64) }], compared, main), null);
+ assert.equal(matchingMainFailures([failure], { main, files: [{ file: failure.file, result: 'red', failures: [failure, failure] }] }, main), null);
+});
+
test('submit refuses changed failure details, an unaccounted stage, and a red frozen check [V4,B3]', t => {
for (const options of [
{ candidate: source('shared failure [G1]', 3) },
@@ -130,7 +164,7 @@ test('submit refuses changed failure details, an unaccounted stage, and a red fr
const box = fixture(t, options);
const result = cli(box, box.builder, 'submit', String(box.item));
assert.equal(result.status, 1, result.stdout + result.stderr);
- assert.match(result.stderr, /GATE_RED/u);
+ assert.equal(JSON.parse(result.stdout).error.code, 'GATE_RED');
assert.equal(submittedProof(box), null);
}
});