Autonomous development for real products. Governed agents build from specs you approve, follow the rules you set, and nothing merges until another agent proves it. Your decisions, judgement and taste are the bar that work must clear. Declare your vision and let agents build it.
JavaScript agentic-codingai-agentsai-governanceautonomous-agentsclaude-codeclicode-reviewcodexcoding-agentsdeveloper-toolsgit-hooksllmlocal-firstmulti-agentsoftware-factoryspec-driven-developmentverificationvibe-coding
fix(relay): separate monitor samples and phone observations [H5,H17]
Corey Olsoncfad8d3d643d
diff --git a/docs/architecture.md b/docs/architecture.mdindex 2dab370b..9f9da78d 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -128,6 +128,18 @@ The relay never receives a board key in a form it can use. A phone gets it wrapp**One order for every move.** The relay gives each record the next sequence number atomically and refuses repeats and gaps (`insert` in `relay/journal.js`). Every client applies records in that order with the CLI's own rules.+## Machine-local relay canary++A relay canary is an opt-in monitor on an ordinary client machine. It checks the authenticated board move path by signing in as its configured ordinary seat, sending a nonce-bearing shout through the normal ordered move, pulling the fresh event and unsealing the exact returned move. A successful `/health` response alone is not a round trip. The canary does not alter the relay API or add a server-side table.++`pullboard relay canary on [--timeout <milliseconds>]` stores `relay.canary` in this machine's settings: an enabled flag, an absolute checkout path, the registered non-coordinator `agentId`, and an optional deadline (5 seconds by default; 100–25,000 ms when set). The credentials stay in the existing local link and key stores. The setting binds the run to that checkout and seat. To monitor the dedicated board from Studio and MacBook, link that board normally on each non-relay host and configure one ordinary seat per machine after the feature lands. `pullboard relay canary off` disables an existing setting (or leaves an absent setting unset); `status` reads local observations without a network request; `run` executes one probe; `watch` runs an immediate probe and then checks once per minute until stopped with Ctrl-C. The watch is separate from the ordinary ten-minute health scan.++Each machine keeps its observations and minute schedule in a private SQLite file at `<git common dir>/pullboard/relay-health.sqlite`. A fresh result is healthy or degraded; after two minutes without a recent sample, status is unknown. Two consecutive failed probes create an alert. A latency alert requires a complete ten-minute window and at least ten successful samples, with p95 above two seconds. The local Needs-you entry reports the actual queued move count from that machine's outbox. It is local telemetry: it is not included in sealed board presentations or sent to other devices as health state.++On an alert or recovery, the configured seat sends notices through ordinary native ordered shouts to the coordinator and person. If the relay or native notification route is unavailable, the notice remains pending locally and is retried; a later delivery is labeled delayed. A local alert or successful native write is not a phone acknowledgement. Closed-phone push delivery is a separate notification path.++For an operational drill, interrupt the authenticated sign-in, move-upload or fresh-event-read route used by the client, then restore that route and run a complete probe. Do not substitute an HTTP health endpoint for those client routes. The expected recovery is a later successful sealed round trip; if its notice could not travel during the outage, it remains delayed. Configuration alone does not mean either production host has been linked or has completed a successful probe.+## RedeploysA new relay process on the same `/data` volume keeps every account, link, credential hash, paired phone, wrapped key, approval and journal; `test/relay-deploy.test.js` restarts the relay and proves that no one re-pairs. A restart ends what lives only in memory, so a half-finished sign-in or pairing starts again. The service runs as a single replica, since one writer owns these SQLite files (`relay/README.md`).diff --git a/docs/formats.md b/docs/formats.mdindex 931e0fe9..c8801baa 100644--- a/docs/formats.md+++ b/docs/formats.md@@ -725,3 +725,16 @@ The activity-delivery engine, numbered at landing, adds recipient-owned `recordAThe machine resource database adds `waiter.runner_only` and `holder.runner_execution`, both integer booleans defaulting to zero. A runner-only gate ticket cannot consume local or short capacity while a runner is registered. The trusted SSH worker sets `runner_execution` on its own actual private gate holder; inherited admission reads that holder rather than accepting an environment-only claim of remote execution. These are local, additive scheduling metadata, not board move or relay formats.The CPU-sentinel engine, numbered at landing, names an agent-worktree process after more than twenty CPU minutes, once per host, PID and start time. Current `health-scan` receipts carry `pendingCpu`, the findings whose coordinator notice was held by the existing ten-notice cap. Later scans drain that durable queue without inventing another health-failure episode. An older engine refuses while this queue is pending, preserving its history; older receipts retain their original fields. Observation never signals a process.++## Machine-local relay canary++The canary setting is part of the machine's `settings.json`, not project `pullboard.json` or the native board. Its optional `relay.canary` object is `{enabled, root, agentId, deadlineMs}`. `root` is an absolute checkout path, `agentId` is an ordinary registered non-coordinator seat, and `deadlineMs` defaults to 5000 and may be 100 through 25000. `pullboard relay canary off` sets an existing object to `enabled: false` (or leaves an absent setting unset); it creates no board event or relay credential.++The client-local file `<git common dir>/pullboard/relay-health.sqlite` is separate from `board.sqlite` and from the relay's journal. It contains:++| Table | Rows | Meaning |+| --- | --- | --- |+| `relay_health` | One row with `id = 1` and a JSON `value` | Version-one local history: monitoring start; bounded timestamped probe samples (`ok`, failed step, safe code, elapsed milliseconds and relay version); failure count; degraded/alert timestamps; p95; and durable pending alert or recovery notices. |+| `canary_schedule` | One row with `id = 1` and numeric `next_due` | The next eligible minute, claimed atomically so multiple local views of the same repository do not start duplicate probes. |++The containing directory is created with mode 0700 and the SQLite file is created with mode 0600. A status read does not create it or contact the relay. Pending notices are removed only after the ordinary native notification path completes; a failed or interrupted notification leaves the notice available for retry. These records are machine-local display and scheduling state. They add no board table, event, engine field, relay journal record or sealed presentation field.diff --git a/relay/browser-client.js b/relay/browser-client.jsindex 8e729385..abae0474 100644--- a/relay/browser-client.js+++ b/relay/browser-client.js@@ -7,7 +7,7 @@ import { ENGINE_VERSION } from './engine.js';import { Refused } from './refused.js';import { SNAPSHOT_BODY_BYTES } from './relay-limits.js';import { enqueueStream, followStream } from './browser-stream.js';-import { emptyBoardStep, homeLineDismissed, homeScreenLine, noticeLines, rememberOwnedRequestId, showRelayScreen, storedOwnedRequestState } from './browser-notice.js';+import { recordBrowserReadFailure, clearBrowserReadFailure, emptyBoardStep, homeLineDismissed, homeScreenLine, noticeLines, rememberOwnedRequestId, showRelayScreen, storedOwnedRequestState } from './browser-notice.js';import { registerApp, requestPairingPersistence } from './install.js';import { browserJSON, withBrowserDeadline } from './browser-deadline.js';import { phoneApprovalStatus, placePhoneApprovals, forgetPhoneApprovals } from './browser-approval-card.js';@@ -84,11 +84,11 @@ async function checkAuthorization(path, status, onDenied, error) {if (error?.code === 'DEVICE_REVOKED' && onDenied) { await onDenied(error); return; }if (![401, 403].includes(status) || !onDenied) return;if (['/api/v1/boards', '/api/v1/devices/session'].includes(new URL(path, location.origin).pathname)) onDenied();- else await documentAt('/api/v1/devices/session', onDenied);+ else await rawDocumentAt('/api/v1/devices/session', onDenied);}/** Fetch only same-origin API documents, retaining stable refusal guidance and no provider details. */-async function documentAt(path, onDenied, options = {}) {+async function rawDocumentAt(path, onDenied, options = {}) {const headers = new Headers(options.headers ?? {});headers.set('x-pullboard-engine', String(ENGINE_VERSION));const { response, document } = await browserJSON(path, { ...options, headers, credentials: 'same-origin', cache: 'no-store', redirect: 'error' });@@ -121,6 +121,7 @@ export async function createTransport({ onUpdate = () => {}, onOpenBoard = () =>let warnings = [];let failure = '';let failureBoard = null;+ const degradedBoards = new Map();let accessLost = false;let accessError = null;const access = new AbortController();@@ -132,6 +133,18 @@ export async function createTransport({ onUpdate = () => {}, onOpenBoard = () =>if (pair && (!BOARD.test(pair.board ?? '') || typeof pair.key !== 'string')) pair = null;try { if (pair) decodeBoardKey(pair.key); } catch { pair = null; }+ /** Observe failed board reads on this phone; user-intent refusals are not transport health. */+ async function documentAt(path, onDenied, options = {}) {+ try { return await rawDocumentAt(path, onDenied, options); }+ catch (error) {+ const id = /^\/api\/v[12]\/boards\/([^/]+)\//u.exec(new URL(path, location.origin).pathname)?.[1];+ if (id && (!options.method || options.method === 'GET') && !['NO_ITEM', 'NO_SHOUT', 'BAD_REQUEST'].includes(error?.code)) {+ recordBrowserReadFailure(degradedBoards, id); notice();+ }+ throw error;+ }+ }+/** Preserve shared collections across updates: the cockpit shallow-copies state and revocation must clear those arrays too. */function installState(entry, value) {if (!entry.state) { entry.state = value; return; }@@ -258,13 +271,16 @@ export async function createTransport({ onUpdate = () => {}, onOpenBoard = () =>if (!element) return;element.replaceChildren();const ownedRequestIds = new Map();+ const queuedRequests = new Map();for (const [id, entry] of paired) {const key = OWNED + id;let state = { owned: new Set() };try { state = storedOwnedRequestState(localStorage, key, entry.state?.personRequests); } catch { /* Missing device storage can only hide this device's own notice. */ }ownedRequestIds.set(id, state.owned);+ try { queuedRequests.set(id, pendingRequests(entry).length); }+ catch { queuedRequests.set(id, null); }}- const lines = noticeLines({ available, paired, ownedRequestIds, waitingBoards, sizeProblems, approvedBoards, pendingLinks, warnings, exclusions,+ const lines = noticeLines({ available, paired, ownedRequestIds, degradedBoards, queuedRequests, waitingBoards, sizeProblems, approvedBoards, pendingLinks, warnings, exclusions,failure: failure || pendingLinksFailure || exclusionsFailure });for (const text of lines) {const paragraph = document.createElement('p');@@ -355,6 +371,15 @@ export async function createTransport({ onUpdate = () => {}, onOpenBoard = () =>/** Authenticate the latest native snapshot before installing its presentation in this device. */async function snapshot(entry) {+ try { return await readSnapshot(entry); }+ catch (error) {+ if (!access.signal.aborted) { recordBrowserReadFailure(degradedBoards, entry.id); notice(); }+ throw error;+ }+ }++ /** Decode the actual checkpoint before a later complete prefix can clear degraded status. */+ async function readSnapshot(entry) {let path = '/api/v' + (entry.snapshotApiVersion === 2 ? '2' : '1') + '/boards/' + entry.id + '/state';const settings = {credentials: 'same-origin', cache: 'no-store', redirect: 'error',@@ -474,6 +499,15 @@ export async function createTransport({ onUpdate = () => {}, onOpenBoard = () =>/** Read any durable prefix before attaching the live stream, including compaction repair. */async function catchUp(entry) {+ try { return await readCompletePrefix(entry); }+ catch (error) {+ if (!access.signal.aborted) { recordBrowserReadFailure(degradedBoards, entry.id); notice(); }+ throw error;+ }+ }++ /** Validate every received sealed event before this board can recover its local health. */+ async function readCompletePrefix(entry) {let document;try { document = await boardEvents(entry); }catch (error) {@@ -484,6 +518,8 @@ export async function createTransport({ onUpdate = () => {}, onOpenBoard = () =>warning(document.warning, entry.id);for (const value of document.warnings ?? []) warning(value, entry.id);for (const row of document.events) await receive(entry, row, true);+ clearBrowserReadFailure(degradedBoards, entry.id);+ if (failureBoard === entry.id) { failure = ''; failureBoard = null; }}/** Handle a relay stream message in sequence, matching EventSource behavior with an engine header. */@@ -508,14 +544,14 @@ export async function createTransport({ onUpdate = () => {}, onOpenBoard = () =>if (['AUTH_REQUIRED', 'NO_REPO_ACCESS', 'TOKEN_BOARD'].includes(error?.code)) await documentAt('/api/v1/devices/session', denied);failure = error ? '[' + error.code + '] ' + error.message : 'The relay stream ended. Refresh this board.';failureBoard = entry.id;+ recordBrowserReadFailure(degradedBoards, entry.id);notice();}return;}if (document.version !== 1) throw new Error('Upgrade this browser client to read the relay API.');await receive(entry, document.event);- failure = '';- failureBoard = null;+ // One event is not a successful full read, and cannot clear another board's failure.notice();onUpdate();}@@ -554,6 +590,7 @@ export async function createTransport({ onUpdate = () => {}, onOpenBoard = () =>if (['NO_BOARD', 'NO_SNAPSHOT'].includes(error?.code)) { waitForFirstSnapshot(entry); return; }failure = error.message;failureBoard = entry.id;+ recordBrowserReadFailure(degradedBoards, entry.id);notice();if (fatal) stream.close();},@@ -646,6 +683,7 @@ export async function createTransport({ onUpdate = () => {}, onOpenBoard = () =>rememberOwnedRequestId(localStorage, OWNED + entry.id, entry.outbox.id);}} catch { throw new Refused('REQUEST_STORAGE', 'This browser cannot save the sealed request. Enable device storage before sending it.'); }+ notice();}/** Read each independently saved request so another tab cannot overwrite or clear its intent. */diff --git a/relay/browser-notice.js b/relay/browser-notice.jsindex c0d4f014..cc594a44 100644--- a/relay/browser-notice.js+++ b/relay/browser-notice.js@@ -32,9 +32,27 @@ export function storedOwnedRequestState(storage, key, requests) {return { ...state, owned: new Set(owned) };}+/** Preserve this browser's first failed read per board; another board cannot clear it. */+export function recordBrowserReadFailure(degradedBoards, id, now = Date.now()) {+ if (!degradedBoards.has(id)) degradedBoards.set(id, new Date(now).toISOString());+}++/** Clear only the board whose authenticated snapshot and complete prefix were read successfully. */+export function clearBrowserReadFailure(degradedBoards, id) {+ degradedBoards.delete(id);+}+/** Build truthful device notices from authorized boards and their decrypted pending requests. */-export function noticeLines({ available, paired, ownedRequestIds = new Map(), waitingBoards = new Map(), sizeProblems = new Map(), approvedBoards = new Set(), pendingLinks = [], warnings = [], failure = '', exclusions = [] }) {+export function noticeLines({ available, paired, ownedRequestIds = new Map(), waitingBoards = new Map(), sizeProblems = new Map(), approvedBoards = new Set(), pendingLinks = [], warnings = [], failure = '', exclusions = [], degradedBoards = new Map(), queuedRequests = new Map() }) {const lines = [];+ for (const [id, entry] of paired) {+ const since = degradedBoards.get(id);+ const queued = queuedRequests.get(id);+ if (!since && !(queued > 0) && queued !== null) continue;+ const name = entry.repository ?? id;+ const count = queued === null ? 'queued request count unavailable' : `${queued ?? 0} sealed request${queued === 1 ? '' : 's'} queued locally`;+ lines.push(`${name}: ${since ? `relay degraded since ${since.slice(11, 16)}; ` : ''}${count}.${queued > 0 ? ' This device has not received an acknowledgement for these queued requests.' : ''}`);+ }const waiting = [...paired.entries()].reduce((count, [id, entry]) => {const owned = ownedRequestIds.get(id) ?? new Set();return count + reconcileOwnedRequests(entry.state?.personRequests, owned).waiting.length;diff --git a/src/relay-canary-health.js b/src/relay-canary-health.jsindex 03d43edf..23abcf7d 100644--- a/src/relay-canary-health.js+++ b/src/relay-canary-health.js@@ -33,10 +33,13 @@ function historyAt(file) {/** Format only observations this client made, with a real locally queued move count. */function projection(history, queued, now) {- const last = history?.samples?.at(-1);+ const monitor = history?.samples?.at(-1);+ const client = history?.client?.sample;+ const last = history?.alertSince ? monitor : (!monitor || client && Date.parse(client.at) > Date.parse(monitor.at) ? client : monitor);+ const degradedSince = history?.alertSince ? history.degradedSince : last && last === client ? history?.client?.degradedSince : history?.degradedSince;const stale = !last || now - Date.parse(last.at) > 120000;- const status = !last || stale ? 'unknown' : history.degradedSince ? 'degraded' : 'healthy';- const since = history?.degradedSince ?? null;+ const status = !last || stale ? 'unknown' : degradedSince ? 'degraded' : 'healthy';+ const since = degradedSince ?? null;const time = since ? since.slice(11, 16) : '';const text = status === 'degraded'? `relay degraded since ${time}; your moves are queued locally (${queued})`@@ -78,6 +81,7 @@ function sampleOf(result, now) {function advance(previous, sample, now) {const history = previous ?? { version: 1, monitoringSince: sample.at, samples: [], consecutiveFailures: 0,degradedSince: null, alertSince: null, p95Ms: null };+ history.monitoringSince ??= sample.at;history.samples = history.samples.filter(entry => Date.parse(entry.at) >= now - WINDOW_MS && Date.parse(entry.at) <= now);history.samples.push(sample);history.samples = history.samples.slice(-128);@@ -120,16 +124,14 @@ export function observeRelayHealthFile(file, result, { now = Date.now(), alertsdb.exec('PRAGMA busy_timeout=1000; CREATE TABLE IF NOT EXISTS relay_health(id INTEGER PRIMARY KEY CHECK(id=1), value TEXT NOT NULL); BEGIN IMMEDIATE');const saved = db.prepare('SELECT value FROM relay_health WHERE id=1').get();const previous = saved ? JSON.parse(saved.value) : null;- const priorAlert = previous?.alertSince ?? null;- const priorFailures = previous?.consecutiveFailures ?? 0;- const priorNotices = structuredClone(previous?.pendingNotices ?? []);- const updated = advance(previous, sample, now);- // Ordinary client transport failures update status; only the configured canary raises monitor alerts.- if (!alerts) {- updated.transition = null; updated.finding = null;- updated.history.alertSince = priorAlert;- updated.history.consecutiveFailures = priorFailures;- updated.history.pendingNotices = priorNotices;+ let updated;+ if (alerts) updated = advance(previous, sample, now);+ else {+ // User traffic reports its own recent health without becoming a minute monitor sample.+ const history = previous ?? { version: 1, monitoringSince: null, samples: [], consecutiveFailures: 0,+ degradedSince: null, alertSince: null, p95Ms: null };+ history.client = { sample, degradedSince: sample.ok ? null : history.client?.degradedSince ?? sample.at };+ updated = { history, transition: null, finding: null };}db.prepare('INSERT INTO relay_health(id,value) VALUES (1,?) ON CONFLICT(id) DO UPDATE SET value=excluded.value').run(JSON.stringify(updated.history));diff --git a/src/relay-canary-watch.js b/src/relay-canary-watch.jsindex 9314094f..c537a0a8 100644--- a/src/relay-canary-watch.js+++ b/src/relay-canary-watch.js@@ -3,6 +3,7 @@ import { probeRelayCanary } from './relay-canary-probe.js';import {claimRelayCanaryTick,observeRelayHealth,+ readRelayHealth,pendingRelayCanaryNotices,markCanaryNoticeRecorded,} from './relay-canary-health.js';@@ -23,7 +24,7 @@ export async function runRelayCanary(value, { io = {}, now = Date.now, notify }await notify(notice, observation);markCanaryNoticeRecorded(config.root, notice.id);}- return { observation, ...update };+ return { observation, ...update, health: readRelayHealth(config.root, { now: now() }) };}/** Start one due local tick; shutdown aborts and awaits the active probe instead of abandoning a write. */diff --git a/test/relay-browser-stream.test.js b/test/relay-browser-stream.test.jsindex f24922d9..e7516b16 100644--- a/test/relay-browser-stream.test.js+++ b/test/relay-browser-stream.test.js@@ -2,7 +2,7 @@import assert from 'node:assert/strict';import { test } from 'node:test';import { enqueueStream, followStream } from '../relay/browser-stream.js';-import { noticeLines, reconcileOwnedRequests, rememberOwnedRequestId, storedOwnedRequestState } from '../relay/browser-notice.js';+import { recordBrowserReadFailure, clearBrowserReadFailure, noticeLines, reconcileOwnedRequests, rememberOwnedRequestId, storedOwnedRequestState } from '../relay/browser-notice.js';/** Build a finite standards-shaped stream response for a deterministic transport probe. */function eventResponse(id) {@@ -210,3 +210,25 @@ test('checkpoint timeout notices survive activity and recovery on another pairedpaired.get(a).checkpointTimedOut = false;assert.deepEqual(lines(), [], 'every covering checkpoint clears only its own timeout');});+++test('phone degradation and durable queued notices remain local to each board [H5,H17]', () => {+ const degradedBoards = new Map();+ recordBrowserReadFailure(degradedBoards, 'a', Date.parse('2026-01-01T10:00:00Z'));+ recordBrowserReadFailure(degradedBoards, 'a', Date.parse('2026-01-01T10:01:00Z'));+ recordBrowserReadFailure(degradedBoards, 'b', Date.parse('2026-01-01T10:02:00Z'));+ clearBrowserReadFailure(degradedBoards, 'b');+ assert.equal(degradedBoards.get('a'), '2026-01-01T10:00:00.000Z', 'repeated failure keeps the first timestamp and other-board recovery does not clear it');+ const paired = new Map([['a', { repository: 'repo/a', state: { personRequests: [] } }], ['b', { repository: 'repo/b' }]]);+ const queuedRequests = new Map([['a', 2], ['b', 0]]);+ const lines = noticeLines({ available: [{ id: 'a', repository: 'repo/a' }, { id: 'b', repository: 'repo/b' }], paired, degradedBoards, queuedRequests });+ assert.equal(lines.length, 1);+ assert.match(lines[0], /repo\/a: relay degraded since 10:00; 2 sealed requests queued locally/u);+ assert.match(lines[0], /has not received an acknowledgement/u, 'queued ciphertext is not an acknowledged waiting request');+ clearBrowserReadFailure(degradedBoards, 'a');+ assert.match(noticeLines({ available: [{ id: 'a' }], paired: new Map([['a', paired.get('a')]]), degradedBoards, queuedRequests })[0], /2 sealed requests queued locally/u, 'a successful read does not discard unsent ciphertext');+ queuedRequests.set('a', 0);+ assert.deepEqual(noticeLines({ available: [{ id: 'a' }], paired: new Map([['a', paired.get('a')]]), degradedBoards, queuedRequests }), []);+ queuedRequests.set('a', null);+ assert.match(noticeLines({ available: [{ id: 'a' }], paired: new Map([['a', paired.get('a')]]), degradedBoards, queuedRequests })[0], /count unavailable/u, 'unreadable device storage cannot claim zero pending requests');+});diff --git a/test/relay-canary-watch.test.js b/test/relay-canary-watch.test.jsindex 0b0ca1f1..ddc788d0 100644--- a/test/relay-canary-watch.test.js+++ b/test/relay-canary-watch.test.js@@ -175,8 +175,9 @@ test('a pending alert notice retries after delivery failure and is removed onlyassert.equal(attemptIds.length, 1, 'the first failed probe did not create a notification');now += 60_000;- await withFixtureEnvironment(box.env, () => runRelayCanary(config, { now: () => now,+ const delivered = await withFixtureEnvironment(box.env, () => runRelayCanary(config, { now: () => now,notify: async notice => { attemptIds.push(notice.id); } }));+ assert.equal(delivered.health.pendingNotifications, 0, 'returned status reflects completed notice recording');assert.deepEqual(attemptIds, [pending[0].id, pending[0].id], 'retry uses the same durable notice identity');assert.deepEqual(pendingRelayCanaryNotices(box.root), [], 'successful ordinary notice recording clears the pending delivery');