Skip to content

Autonomous development for real products. Governed agents build from specs you approve, follow the rules you set, and nothing merges until another agent proves it. Your decisions, judgement and taste are the bar that work must clear. Declare your vision and let agents build it.

feat(sentinel): run due watches on trunk outside resume [N26,A6]

Corey Olsonb8efd267b3d8
diff --git a/docs/api.md b/docs/api.md
index b09d3fdb..2b1c5018 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -607,7 +607,11 @@ A project may declare a `staticChecks` command in `pullboard.json` (empty by def
`pullboard sentinel propose <name> --check "<command>" --every 10m|hourly|daily --owner <seat or lane>` proposes a board row. Optional `--directory <relative path>`, `--timeout <milliseconds>` and `--next "<step>"` are part of the exact proposal. The directory defaults to this project, and must stay inside it; timeout defaults to 30000 ms and is a hang ceiling. The person approves or declines through the same person-channel guard as spec rows: `pullboard sentinel approve <name>` or `pullboard sentinel decline <name> --reason "<why>"`. An agent cannot approve. Any new command, directory, schedule, owner, timeout or next step needs fresh approval. No row in SPEC.md is edited.
-An approved watch runs when due on `resume` and through the background view’s ordinary resume worker. The existing board and machine `health.sentinel: false` switches suppress it. Ordered run leases prevent duplicate concurrent observers. External checks run outside the database write lock, in the approved directory, with bounded final output and an owned-child timeout; recorded replay never executes a command. On POSIX hosts shutdown drains the owned process group; Windows uses direct-child termination. A failed check notifies its owner once with its name, final output and next step; staying red posts no duplicate, and recovery posts one cleared notice. Every red run remains a `sentinel:<name>` cause receipt for Rigor, without inventing a code change. Cancellation records its outcome without creating a failure or recovery notice. `pullboard sentinels --json` lists current rows, their owners, approval, latest result and current proposal’s run history. This is engine 56; earlier engines refuse the new moves before replay.
+`resume` and the background view never execute approved watch commands. When watches are due, resume prints at most one line pointing to `pullboard sentinel run`. That explicit command prepares a clean checkout of the committed local trunk, runs due approved watches there in their approved relative directories, records their results on the original board, and removes the prepared checkout after the commands finish. It never chooses an agent's working tree as the execution directory or rewrites an approved command. Relative project inputs come from committed trunk; explicitly approved absolute paths retain their literal meaning.
+
+Schedule `pullboard sentinel run` independently of work sessions. For example, an hourly server cron entry is `0 * * * * cd /path/to/project && pullboard sentinel run`. A scheduled CI job can run the same command after its ordinary checkout and board-access setup; its schedule calls this command, not `resume`. The checkout used by the timer must already have normal access to the board. `pullboard sentinel run --help` describes both scheduling options. These examples do not install a timer or change approval.
+
+The existing board and machine `health.sentinel: false` switches suppress execution. Ordered run leases prevent duplicate concurrent scheduled runs. External checks run outside the database write lock, with bounded final output and an owned-child timeout; recorded replay never executes a command. On POSIX hosts shutdown drains the owned process group; Windows uses direct-child termination. A failed check notifies its owner once with its name, final output and next step; staying red posts no duplicate, and recovery posts one cleared notice. Every red run remains a `sentinel:<name>` cause receipt for Rigor, without inventing a code change. Cancellation records its outcome without creating a failure or recovery notice. `pullboard sentinels --json` lists current rows, their owners, approval, latest result and current proposal’s run history. This is engine 56; earlier engines refuse the new moves before replay.
## Approval provenance
diff --git a/src/commands/resume.js b/src/commands/resume.js
index e74da4f9..761b8223 100644
--- a/src/commands/resume.js
+++ b/src/commands/resume.js
@@ -152,7 +152,7 @@ export async function resumeHere(io, view) {
catch (error) { if (!(error instanceof Refused) || error.code !== 'HEALTH_SCAN_STALE') throw error; }
}
});
- await resumeSentinels({ ctx, withBoard, whoAmI, ordered });
+ resumeSentinels({ ctx, withBoard });
const { root, isMain } = ctx.info;
const mainRoot = primaryCheckoutRoot(ctx.info);
const worktreeConfig = mainRoot ? (isMain ? ctx.config : loadConfig(mainRoot)) : null;
diff --git a/src/help.js b/src/help.js
index 629142c7..b9b5a668 100644
--- a/src/help.js
+++ b/src/help.js
@@ -95,6 +95,7 @@ Work
pullboard sentinel propose <name> --check "<command>" --every 10m|hourly|daily --owner <seat or lane> [--directory <path>] [--timeout <milliseconds>] [--next "<step>"]
propose an exact board watch; only the person may approve it
pullboard sentinel approve <name> person: approve this command, directory and schedule
+ pullboard sentinel run run due approved watches in a prepared trunk checkout
pullboard sentinel decline <name> --reason "<why>" person: prevent future runs
pullboard sentinels approvals, owners and the last result; does not run checks
pullboard loop declare <name> --every 10m --owner <seat or lane> coordinator: declare a recurring job's cadence
@@ -362,6 +363,11 @@ const HELP_FLAG_EXPLANATIONS = {
/** Lines a command's own help adds after its example. */
const HELP_NOTES = {
+ 'sentinel run': [
+ 'Schedule this command with cron on a server or a scheduled CI job that already has ordinary board access.',
+ 'For example: 0 * * * * cd /path/to/project && pullboard sentinel run',
+ 'Each run prepares committed local trunk; resume and the background view never execute watch commands.',
+ ],
watch: [
'Claude Code: run pullboard watch --to <your id> under the Monitor tool, and each event line wakes the seat.',
'Any other harness: run it as a long-running command and treat each stdout line as an event.',
diff --git a/src/sentinel-command.js b/src/sentinel-command.js
index e376e9f0..6a972f2c 100644
--- a/src/sentinel-command.js
+++ b/src/sentinel-command.js
@@ -2,8 +2,8 @@
import { PERSON } from './board.js';
import { requirePersonChannel } from './person.js';
import { Refused } from './refused.js';
-import { sentinels } from './sentinels.js';
-import { runDueSentinels } from './sentinel-runner.js';
+import { dueSentinels, sentinels } from './sentinels.js';
+import { runDueSentinels, withSentinelTrunk } from './sentinel-runner.js';
import { healthSentinelEnabled } from './settings.js';
/** The watch command's existing string options, shared with the CLI parser. */
@@ -16,21 +16,38 @@ export const SENTINEL_OPTIONS = {
sentinel: { type: 'boolean' },
};
-/** Run due approved watches from ordinary resume, respecting the health switches. */
-export async function resumeSentinels({ ctx, withBoard, whoAmI, ordered }) {
+/** Resume reports due watches in one line and never starts their commands. */
+export function resumeSentinels({ ctx, withBoard }) {
+ if (!healthSentinelEnabled(ctx.config)) return;
+ return withBoard(ctx, board => {
+ if (dueSentinels(board).length) ctx.io.say('Sentinels are due; schedule pullboard sentinel run to run them in a prepared trunk checkout.');
+ });
+}
+
+/** Run due approved watches on pinned trunk, recording through the original caller's board. */
+async function runSentinels({ ctx, withBoard, whoAmI, ordered }) {
return withBoard(ctx, async board => {
const me = whoAmI(ctx, board);
- if (!healthSentinelEnabled(ctx.config)) return;
- await runDueSentinels({ root: ctx.info.root, board, agentId: me.id,
- ordered: (operation, args) => ordered(ctx, board, operation, args) });
+ if (!healthSentinelEnabled(ctx.config) || !dueSentinels(board).length) {
+ ctx.io.result?.({ sentinels: [] });
+ ctx.io.say('No enabled sentinel is due.');
+ return 0;
+ }
+ const { results, trunk } = await withSentinelTrunk(ctx.info.root, root => runDueSentinels({ root, board, agentId: me.id,
+ ordered: (operation, args) => ordered(ctx, board, operation, args) }));
+ ctx.io.result?.({ sentinels: results, trunk });
+ ctx.io.say(`Sentinels: ${results.length} run from trunk ${trunk.branch}@${trunk.commit.slice(0, 12)}.`);
+ for (const row of results) ctx.io.say(`${row.name}: ${row.lastRun.green ? 'green' : 'red'}; owner ${row.owner}`);
+ return 0;
});
}
/** Keep command parsing separate from the CLI's process and server setup. */
export async function sentinelCommand({ ctx, words, values, withBoard, whoAmI, ordered }) {
const [verb, name, ...extra] = words;
+ if (verb === 'run' && !name && !extra.length) return runSentinels({ ctx, withBoard, whoAmI, ordered });
if (extra.length || !['propose', 'approve', 'decline'].includes(verb) || !name) {
- throw new Refused('USAGE', 'use pullboard sentinel propose <name> --check "<command>" --every 10m --owner <seat or lane>, sentinel approve <name>, sentinel decline <name> --reason "<why>", or pullboard sentinels');
+ throw new Refused('USAGE', 'use pullboard sentinel run, sentinel propose <name> --check "<command>" --every 10m --owner <seat or lane>, sentinel approve <name>, sentinel decline <name> --reason "<why>", or pullboard sentinels');
}
if (verb !== 'propose') requirePersonChannel(ctx.io.personChannel);
return withBoard(ctx, async board => {
diff --git a/src/sentinel-runner.js b/src/sentinel-runner.js
index cb2035ea..e7d2d1ba 100644
--- a/src/sentinel-runner.js
+++ b/src/sentinel-runner.js
@@ -3,13 +3,34 @@ import { spawn } from 'node:child_process';
import { spawnSync } from 'node:child_process';
import { randomUUID } from 'node:crypto';
import { realpathSync } from 'node:fs';
+import { mkdtempSync, rmSync } from 'node:fs';
import { relative } from 'node:path';
import { resolve } from 'node:path';
import { sep } from 'node:path';
+import { join } from 'node:path';
+import { tmpdir } from 'node:os';
import { performance } from 'node:perf_hooks';
import { Refused } from './refused.js';
import { parseProcessCensus } from './process-census.js';
import { sentinels } from './sentinels.js';
+import { gitChildEnv, mainCheckout } from './git.js';
+
+/** Prepare immutable local trunk in a private clone, leaving every live checkout untouched. */
+export async function withSentinelTrunk(root, action) {
+ const trunk = mainCheckout(root);
+ if (!trunk?.commit) throw new Refused('SENTINEL_TRUNK', 'commit the primary branch before running sentinels: pullboard sentinel run');
+ const directory = mkdtempSync(join(tmpdir(), 'pullboard-sentinel-trunk-'));
+ const checkout = join(directory, 'checkout');
+ const env = gitChildEnv(root);
+ try {
+ const clone = spawnSync('git', ['clone', '--quiet', '--shared', '--no-checkout', '--', root, checkout], { env, encoding: 'utf8' });
+ const selected = clone.status === 0 && spawnSync('git', ['checkout', '--quiet', '--detach', trunk.commit], { cwd: checkout, env, encoding: 'utf8' });
+ if (!selected || selected.status !== 0) throw new Refused('SENTINEL_TRUNK', 'the trunk checkout could not be prepared; inspect the primary branch and retry pullboard sentinel run');
+ return { results: await action(checkout), trunk };
+ } finally {
+ rmSync(directory, { recursive: true, force: true });
+ }
+}
/** Keep only bounded final output; a noisy check cannot grow a board receipt without limit. */
function outputTail(previous, chunk) {
@@ -69,7 +90,7 @@ export function executeSentinel(root, row, { signal, environment = process.env }
const path = relative(base, directory);
if (path === '..' || path.startsWith('..' + sep)) throw new Refused('BAD_SENTINEL', 'the approved sentinel directory must remain inside its project');
} catch (error) { reject(error); return; }
- if (signal?.aborted) { reject(new Refused('SENTINEL_ABORTED', 'the background view stopped before this check began; retry on resume')); return; }
+ if (signal?.aborted) { reject(new Refused('SENTINEL_ABORTED', 'the sentinel run stopped before this check began; retry pullboard sentinel run')); return; }
const start = performance.now();
const group = process.platform !== 'win32';
// The shell waits on a private start pipe until its detached group identity is captured.
@@ -106,7 +127,7 @@ export function executeSentinel(root, row, { signal, environment = process.env }
const timer = setTimeout(() => { timedOut = true; stop(); }, row.timeoutMs);
timer.unref();
signal?.addEventListener('abort', stop, { once: true });
- /** A background resume child also drains its approved command before honoring view shutdown. */
+ /** A scheduled run drains its approved command before honoring shutdown. */
const interrupt = () => { interrupted = true; stop(); };
process.once('SIGTERM', interrupt); process.once('SIGINT', interrupt);
/** Remove only this run's handlers, leaving the embedding process's other handlers untouched. */
diff --git a/src/sentinels.js b/src/sentinels.js
index 80e14f57..44179dc6 100644
--- a/src/sentinels.js
+++ b/src/sentinels.js
@@ -156,15 +156,26 @@ export function decideSentinel(board, { agentId, channel, name, digest, decision
});
}
-/** Atomically claim a due approved check, so simultaneous resume/view workers cannot run it twice. */
+/** Apply the same cadence and live-run rules to a notice and an atomic run claim. */
+function isDue(row, now) {
+ return row.status === 'approved' && !(row.activeRun && Date.parse(row.activeRun.until) > now)
+ && !(row.lastRun && !row.lastRun.aborted && Date.parse(row.lastRun.at) + INTERVALS[row.every] > now);
+}
+
+/** List due watches without starting a command or consuming a run lease. */
+export function dueSentinels(board) {
+ const now = board.clock.now().getTime();
+ return sentinels(board).filter(row => isDue(row, now));
+}
+
+/** Atomically claim a due approved check, so simultaneous scheduled workers cannot run it twice. */
export function beginSentinelRun(board, { agentId, name, digest, run }) {
return atomic(board, () => {
requireSentinelActor(board, agentId);
const row = currentSentinel(board, name, digest);
const now = board.clock.now().getTime();
if (typeof run !== 'string' || !/^[a-zA-Z0-9-]{1,80}$/u.test(run)) throw new Refused('BAD_SENTINEL', 'start a sentinel with a unique run id');
- if (row.status !== 'approved' || row.activeRun && Date.parse(row.activeRun.until) > now
- || row.lastRun && !row.lastRun.aborted && Date.parse(row.lastRun.at) + INTERVALS[row.every] > now) return null;
+ if (!isDue(row, now)) return null;
const until = new Date(now + row.timeoutMs + 10_000).toISOString(); // budget: command hang ceiling plus exact-child shutdown grace.
logEvent(board, agentId, 'sentinel-start', null, { name, digest, run, until, engine: BOARD_SENTINEL_ENGINE });
return { ...row, run, until };
diff --git a/test/sentinels.test.js b/test/sentinels.test.js
index 3b830bef..b7567523 100644
--- a/test/sentinels.test.js
+++ b/test/sentinels.test.js
@@ -212,10 +212,13 @@ test('a sentinel draft waits in Needs-you with its evidence [A6,D1,D3]', async t
assert.deepEqual(after, ['staging-site'], 'the person’s approval closes the call');
const watched = sentinels(box.board).find(row => row.name === 'rigor-row-a6');
assert.equal(watched.status, 'approved');
- assert.equal(watched.lastRun?.green, true, 'approving it starts it: the next resume runs the watch, now green');
+ assert.equal(watched.lastRun, null, 'approval and resume leave execution to the scheduled command');
+ const explicit = await box.cli(['sentinel', 'run']);
+ assert.equal(explicit.code, 0, explicit.error || JSON.stringify(explicit.document));
+ assert.equal(sentinels(box.board).find(row => row.name === 'rigor-row-a6').lastRun?.green, true);
});
-test('an approved sentinel runs by itself and reports to its owner once [A6,D1,D3]', async t => {
+test('an approved sentinel execution reports to its owner once [A6,D1,D3]', async t => {
const box = fixture(t);
mkdirSync(join(box.root, 'watch'));
const check = checker(box, "import {existsSync,appendFileSync} from 'node:fs'; appendFileSync('runs','run\\n'); console.log('check receipt'); process.exit(existsSync('green')?0:1);");
@@ -227,7 +230,7 @@ test('an approved sentinel runs by itself and reports to its owner once [A6,D1,D
const approved = await box.cli(['sentinel', 'approve', 'staging'], 'view');
assert.equal(approved.code, 0, approved.error);
assert.equal(approved.document.sentinel.status, 'approved');
- assert.equal((await box.cli(['resume'])).code, 0, 'the real automatic resume caller executes the due approved watch');
+ await box.run();
let notices = store.allShouts(box.board).filter(row => row.shout_from === 'sentinel');
assert.equal(notices.length, 1); assert.equal(notices[0].shout_to, box.agent);
assert.match(notices[0].shout_text, /staging[\s\S]*check receipt[\s\S]*Next: repair the staging check/u);
@@ -465,7 +468,7 @@ test('a successful watch drains a descendant that ignores TERM before publishing
assert.ok(!remaining || remaining.state.startsWith('Z'), 'no receipt precedes the stubborn descendant shutdown');
});
-test('the background observer runs an approved board watch through its real resume child [A6,D3]', async t => {
+test('the background observer leaves approved board watches for explicit runs [A6,D3]', async t => {
const box = fixture(t);
const check = checker(box, "import {appendFileSync} from 'node:fs'; appendFileSync('background-runs','run\\n'); console.log('background receipt'); process.exit(1);");
const row = store.proposeSentinel(box.board, { agentId: box.agent, row: { name: 'background', check, every: '10m', owner: box.agent } });
@@ -497,13 +500,13 @@ test('the background observer runs an approved board watch through its real resu
writeFileSync(settingsFile, settings);
await tick();
assert.deepEqual(errors, []);
- assert.equal(readFileSync(runs, 'utf8'), 'run\n', 'the default background callback launches the real resume child');
+ assert.equal(existsSync(runs), false, 'the real background resume child never executes an approved watch');
const notices = store.allShouts(box.board).filter(shout => shout.shout_text.startsWith('Sentinel failed: background'));
- assert.equal(notices.length, 1);
- assert.equal(notices[0].shout_to, box.agent);
- assert.match(notices[0].shout_text, /background receipt/u);
+ assert.equal(notices.length, 0, 'resume cannot invent a result or owner alert');
+ assert.equal(sentinels(box.board)[0].lastRun, null);
+ assert.equal(store.events(box.board).some(row => row.event_kind === 'sentinel-start'), false);
await tick();
- assert.equal(readFileSync(runs, 'utf8'), 'run\n', 'the next callback honors the recorded due time');
+ assert.equal(existsSync(runs), false, 'a second health callback still leaves the watch to its scheduled command');
await observer.close();
assert.equal(cancelled, true);
});