Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Wakeshift drainer: lease-token settle, per-row failure backoff, unmatched emits end the watch, members pause reporting back, enable resets a stale URL
garrytanfee991ae6d8e
diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.mdindex 064884b585..ad2ee2b9b5 100644--- a/docs/guides/error-codes.md+++ b/docs/guides/error-codes.md@@ -5194,7 +5194,7 @@ More: [docs/guides/wakeshift.md#troubleshooting](../../docs/guides/wakeshift.md#| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable ||---|---|---|---|---|---|---|-| gbrain wakeshift enable refused: this brain has live access members, and this release reports back only on brains that serve their owner alone. | Every member of the brain owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. Shared brains get a terms step in a later release; until then reporting back stays off there. | Leave reporting back off on this brain; agents can use wait_job or gbrain jobs wait <id>. | user | `gbrain doctor --json` | 1 | no |+| Reporting back refused (gbrain wakeshift enable, or watch_job after a member was added): this brain has live access members, and this release reports back only on brains that serve their owner alone. | Every member of the brain owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. Shared brains get a terms step in a later release; until then reporting back stays off there. | Leave reporting back off on this brain; agents can use wait_job or gbrain jobs wait <id>. | user | `gbrain doctor --json` | 1 | no |More: [docs/guides/wakeshift.md#who-it-is-for](../../docs/guides/wakeshift.md#who-it-is-for)diff --git a/docs/guides/wakeshift.md b/docs/guides/wakeshift.mdindex 611605961b..3b642ff9e7 100644--- a/docs/guides/wakeshift.md+++ b/docs/guides/wakeshift.md@@ -26,7 +26,7 @@ A Wakeshift key is an Ed25519 private key: `wst_` followed by 50 base58 characte## Who it is for-This release reports back only on brains that serve their owner alone. `gbrain wakeshift enable` refuses a brain with live access members (`gbrain access member list`), because every member of the owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. Shared brains get a terms step in a later release.+This release reports back only on brains that serve their owner alone. `gbrain wakeshift enable` refuses a brain with live access members (`gbrain access member list`), because every member of the owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. A member added after enable pauses reporting back: watch_job refuses with `wakeshift_shared_brain`, no wakes are sent, and `gbrain doctor` warns. Shared brains get a terms step in a later release.A watch belongs to the credential that made it: the OAuth client or token, or the local stdio pipe. `list_watches` and `unwatch` touch only the caller's own watches. `watch_job` runs the same owner check as `get_job`, so an agent-scoped token can watch only the jobs it submitted. A job it doesn't own reads exactly like a missing one, and nothing is sent. If the credential is revoked or its client is deleted before the job finishes, the watch ends without a wake.@@ -46,7 +46,7 @@ Delivery is a pull, not a hook. `gbrain serve` (stdio and HTTP) and the job workOn PGLite there is no background worker. A job submitted from a stdio session runs when `gbrain jobs work` drains the queue, after `gbrain serve` stops, and that run delivers its wakes before it exits. A wake it can't deliver arrives from the next `gbrain serve`.-Each watch lasts 7 days and delivers once. A failed delivery is retried at most 5 times, after 30 s, 2 min, 8 min and 30 min (a rate limit waits its `Retry-After`). Any other refusal is final. A watch that dies is ended at Wakeshift, so the conversation is told. Jobs are never blocked or slowed by Wakeshift: it is down only means the wake comes late or not at all.+Each watch lasts 7 days and delivers once. A failed delivery is retried at most 5 times, after 30 s, 2 min, 8 min and 30 min (a rate limit waits its `Retry-After`). Any other refusal is final. A watch that dies is ended at Wakeshift, so the conversation is told. A wake Wakeshift matched to no open conversation (the user ended it, or it expired there) ends the watch with `last_error: unmatched` rather than counting as delivered. Jobs are never blocked or slowed by Wakeshift: it is down only means the wake comes late or not at all.Repo-less brains work the same way. Watches are operational database rows, not brain content.diff --git a/src/commands/doctor/checks/wakeshift.ts b/src/commands/doctor/checks/wakeshift.tsindex 9bb1c57bce..afef509203 100644--- a/src/commands/doctor/checks/wakeshift.ts+++ b/src/commands/doctor/checks/wakeshift.ts@@ -2,13 +2,15 @@* wakeshift (plan PR 1 item 9): reporting back through Wakeshift. Off is* `ok` (disabled by choice). It warns when the key file is readable by other* users, when reporting back is enabled but no usable key is found (the watch- * tools stay hidden), and when watches died (5 failed attempts or a final+ * tools stay hidden), when the brain gained live access members after enable+ * (reporting back pauses), and when watches died (5 failed attempts or a final* Wakeshift refusal) or hit `id_reused` (a bug signal) in the last 7 days.* Shows the key's kid fingerprint only, never the key. Guide: docs/guides/wakeshift.md.*/import { loadConfigFileOnly } from '../../../core/config.ts';import { parseWakeshiftKey } from '../../../core/wakeshift/client.ts';import { readWakeshiftKey, wakeshiftFileConfig, wakeshiftKeyFile } from '../../../core/wakeshift/config.ts';+import { liveAccessMembers } from '../../../core/wakeshift/drain.ts';import type { Check } from '../../doctor.ts';import { connectedEngine, type DoctorContext, type DoctorEntry } from '../context.ts';@@ -38,13 +40,16 @@ async function runWakeshiftCheck(ctx: DoctorContext): Promise<Check[]> {counts[r.state] = (counts[r.state] ?? 0) + Number(r.n);if (r.id_reused) counts.id_reused! += Number(r.n);}- const details = { enabled, key_source: read.source, kid, watches_7d: counts, docs: DOCS };- const message = counts.id_reused! > 0+ const members = await liveAccessMembers(connectedEngine(ctx));+ const details = { enabled, key_source: read.source, kid, watches_7d: counts, live_members: members, docs: DOCS };+ const message = members > 0+ ? `Reporting back is paused: this brain now has ${members} live access member(s), and this release reports back only on a brain that serves its owner alone. watch_job refuses and no wakes are sent. Fix: gbrain wakeshift disable, or remove the members.`+ : counts.id_reused! > 0? `${counts.id_reused} watch(es) were refused with id_reused in the last 7 days: a gbrain bug, so please report it with this output.`: counts.dead! > 0? `${counts.dead} wake(s) died in the last 7 days (5 failed attempts or a final Wakeshift refusal); the agents were told the watch ended. Check the key and the app with gbrain wakeshift test.`: `Reporting back through Wakeshift is on (key kid ${kid}); ${counts.open} open watch(es), ${counts.delivered} delivered in the last 7 days.`;- checks.push({ name: 'wakeshift', status: counts.id_reused! > 0 || counts.dead! > 0 ? 'warn' : 'ok', message, details });+ checks.push({ name: 'wakeshift', status: members > 0 || counts.id_reused! > 0 || counts.dead! > 0 ? 'warn' : 'ok', message, details });}return checks;}diff --git a/src/commands/wakeshift.ts b/src/commands/wakeshift.tsindex f3cc3e96f9..6b18f3c60f 100644--- a/src/commands/wakeshift.ts+++ b/src/commands/wakeshift.ts@@ -16,7 +16,7 @@ import { requireConsent } from '../core/consent.ts';import { WakeshiftError, WakeshiftKeyError, normalizeWakeshiftUrl, parseWakeshiftKey, wakeshiftClient, type WakeshiftKey } from '../core/wakeshift/client.ts';import { readWakeshiftKey, wakeshiftFileConfig, wakeshiftKeyFile, wakeshiftPublished, wakeshiftSecret } from '../core/wakeshift/config.ts';import { wakeshiftManifest } from '../core/wakeshift/events.ts';-import { endAtWakeshift } from '../core/wakeshift/drain.ts';+import { endAtWakeshift, liveAccessMembers } from '../core/wakeshift/drain.ts';import { setCliExitVerdict } from '../core/cli-force-exit.ts';const HELP = `gbrain wakeshift <command> — report back to agent conversations when jobs finish (off by default)@@ -59,11 +59,6 @@ function loadKey(): { key: WakeshiftKey; source: 'env' | 'file' } {}}-async function liveMembers(engine: BrainEngine): Promise<number> {- const [row] = await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM access_members WHERE disabled_at IS NULL AND tombstoned_at IS NULL');- return Number(row?.n ?? 0);-}-async function watchCounts(engine: BrainEngine): Promise<Record<string, number>> {try {const rows = await engine.executeRaw<{ state: string; last_error: string | null; n: number }>(@@ -122,7 +117,7 @@ async function enable(engine: BrainEngine, args: string[]): Promise<void> {console.error(`[wakeshift] saved the key to ${wakeshiftKeyFile()} (0600).`);}const { key, source } = loadKey();- const members = await liveMembers(engine);+ const members = await liveAccessMembers(engine);if (members > 0) {throw opError('wakeshift_shared_brain', `This brain has ${members} live access member(s); this release reports back only on a brain that serves its owner alone. Nothing was turned on.`,'Leave reporting back off on this brain; agents wait for a job with wait_job or gbrain jobs wait and the job id.');@@ -147,7 +142,11 @@ async function enable(engine: BrainEngine, args: string[]): Promise<void> {'Check the key belongs to your Wakeshift app (gbrain wakeshift test) and retry.', { reason: e instanceof WakeshiftError ? e.code : 'error' });}wakeshiftSecret();- writeFilePlane(w => ({ ...w, enabled: true, ...(url !== 'https://app.wakeshift.ai' ? { url } : {}) }));+ writeFilePlane(w => {+ const next: { enabled?: boolean; url?: string } = { ...w, enabled: true, url };+ if (url === 'https://app.wakeshift.ai') delete next.url;+ return next;+ });say(args.includes('--json'), { enabled: true, url, app: { id: app?.id, name: app?.name }, key: { source, kid: key.kid } }, [`Wakeshift reporting back is on (app ${app?.name ?? '?'}, key kid ${key.kid}, from ${source}).`,'Agents now see watch_job, list_watches and unwatch; gbrain serve and the job worker deliver wakes.',diff --git a/src/core/error-registry.ts b/src/core/error-registry.tsindex 4b7b2a85aa..999f15865e 100644--- a/src/core/error-registry.ts+++ b/src/core/error-registry.ts@@ -848,7 +848,7 @@ export const CODES = {wakeshift_needs_conversation: { class: 'caller', summary: "watch_job wakes an MCP conversation, and a local CLI run has none to wake.", docs: 'docs/guides/wakeshift.md#delivery-and-latency', suggestion: 'Run gbrain jobs wait <id> to block until the job finishes.' },wakeshift_not_enabled: { class: 'caller', summary: "Reporting back through Wakeshift is off on this brain (the owner has not run gbrain wakeshift enable, or the key is missing).", why: "Wakeshift is off by default; only the brain owner turns it on, on the brain host, because it sends wake metadata to a third-party service.", docs: 'docs/guides/wakeshift.md#turn-it-on', suggestion: 'Use wait_job (or gbrain jobs wait <id>) instead, or ask the brain owner to run gbrain wakeshift enable.' },wakeshift_refused: { class: 'caller', summary: "Wakeshift refused the request; the reason carries its error code. Nothing was watched or sent.", why: "Wakeshift checks each request (the signed key, the event and its arguments, the conversation's session key). A refusal other than a rate limit does not go away on retry.", docs: 'docs/guides/wakeshift.md#troubleshooting', suggestion: 'Call wait_job to wait without Wakeshift; the brain owner can run gbrain wakeshift test to check the key and the app.' },- wakeshift_shared_brain: { class: 'host_only', actor: 'user', summary: "gbrain wakeshift enable refused: this brain has live access members, and this release reports back only on brains that serve their owner alone.", why: "Every member of the brain owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. Shared brains get a terms step in a later release; until then reporting back stays off there.", docs: 'docs/guides/wakeshift.md#who-it-is-for', suggestion: 'Leave reporting back off on this brain; agents can use wait_job or gbrain jobs wait <id>.' },+ wakeshift_shared_brain: { class: 'host_only', actor: 'user', summary: "Reporting back refused (gbrain wakeshift enable, or watch_job after a member was added): this brain has live access members, and this release reports back only on brains that serve their owner alone.", why: "Every member of the brain owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. Shared brains get a terms step in a later release; until then reporting back stays off there.", docs: 'docs/guides/wakeshift.md#who-it-is-for', suggestion: 'Leave reporting back off on this brain; agents can use wait_job or gbrain jobs wait <id>.' },wakeshift_unavailable: { class: 'unavailable', retryable: true, summary: "Wakeshift could not be reached or answered with a server error or rate limit; nothing was watched.", why: "gbrain makes one request with a 10 s timeout and never retries inside a tool call, so a slow or down Wakeshift never holds the call.", docs: 'docs/guides/wakeshift.md#troubleshooting', suggestion: 'Retry in a minute, or call wait_job to wait without Wakeshift.' },withdrawal_capacity: { class: 'caller', summary: "Withdrawal capacity." },withdrawal_mirror_failed: { class: 'unavailable', summary: "A page's Git or embedding work is withheld because its withdrawal mirror failed before rewriting the page's file; the file may still hold the withdrawn text.", why: "A withdrawal's Git and embedding effects touch only the pages whose file its mirror recorded rewriting (GBRA-130 A-12); Git never commits a page the mirror passed without rewriting its file. Retrying the request's effects reruns the mirror for this page; once the mirror commits, the withheld work is re-armed automatically.", suggestion: 'Read the parked effects with gbrain doctor --only parked_effects, fix the cause the mirror recorded, then preview gbrain sources writer retry-effects <source> --request-id <id> --dry-run and run it without --dry-run.' },diff --git a/src/core/ops/wakeshift.ts b/src/core/ops/wakeshift.tsindex b515c23242..3439abdafa 100644--- a/src/core/ops/wakeshift.ts+++ b/src/core/ops/wakeshift.ts@@ -16,7 +16,7 @@ import { memberIdentityFor } from '../access/caller.ts';import { WakeshiftError, type WakeshiftCall } from '../wakeshift/client.ts';import { wakeshiftBrainKey, wakeshiftRuntime, wakeshiftSecret, wakeshiftUserId } from '../wakeshift/config.ts';import { JOB_FINISHED } from '../wakeshift/events.ts';-import { drainWakes, endAtWakeshift } from '../wakeshift/drain.ts';+import { drainWakes, endAtWakeshift, liveAccessMembers } from '../wakeshift/drain.ts';const WATCH_TTL = '7d';const TERMINAL = new Set(['completed', 'failed', 'dead', 'cancelled']);@@ -106,6 +106,11 @@ const watch_job: Operation = {}const runtime = wakeshiftRuntime();if (!runtime) throw notEnabled();+ const members = await liveAccessMembers(ctx.engine);+ if (members > 0) {+ throw opError('wakeshift_shared_brain', `This brain now has ${members} live access member(s), so reporting back is paused; nothing was watched.`,+ `Call wait_job ${id} to wait for the job.`, { fix: waitFix(id) });+ }const owner = agentOwnerFence(ctx, 'watch_job');if (owner !== null) await assertJobOwned(ctx, id, owner);const [job] = await ctx.engine.executeRaw<{ status: string }>('SELECT status FROM minion_jobs WHERE id = $1', [id]);diff --git a/src/core/wakeshift/drain.ts b/src/core/wakeshift/drain.tsindex b2356c43ac..94fa266af2 100644--- a/src/core/wakeshift/drain.ts+++ b/src/core/wakeshift/drain.ts@@ -12,7 +12,8 @@* never emit one row; 5 attempts backing off 30 s, 2 m, 8 m, 30 m (a 429 waits* its Retry-After within the same cap); any other 4xx is final. A dead,* revoked or expired watch is ended at Wakeshift once, so the conversation is- * told. Off or keyless: the tick returns before touching the database.+ * told. Off or keyless: the tick returns before touching the database. A brain+ * with live access members sends nothing (watch_job refuses there too).*/import type { BrainEngine } from '../engine.ts';import { memberIdentityFor } from '../access/caller.ts';@@ -45,7 +46,7 @@ export interface WakeRow {finished_us: string | null;}-export interface DrainReport { delivered: number; retried: number; dead: number; ended: number; skipped?: 'off' }+export interface DrainReport { delivered: number; retried: number; dead: number; ended: number; skipped?: 'off' | 'shared' }const CANDIDATES_SQL = (byId: boolean) => `SELECT w.id, w.event, w.target, w.principal_kind, w.principal_id, w.member_id, w.wake_user, w.ws_subscription_id, w.attempts,@@ -61,20 +62,36 @@ SELECT w.id, w.event, w.target, w.principal_kind, w.principal_id, w.member_id, wORDER BY w.next_attempt_at, w.idLIMIT ${WAKE_DRAIN_BATCH}`;-async function claim(engine: BrainEngine, id: number): Promise<boolean> {- const rows = await engine.executeRaw<{ id: number }>(+/** Lease one open row; the lease's timestamp in epoch microseconds (text, so no driver rounds it) is the token settle() must match. */+async function claim(engine: BrainEngine, id: number): Promise<string | null> {+ const rows = await engine.executeRaw<{ lease: string }>(`UPDATE wake_subscriptions SET lease_until = now() + interval '${LEASE_S} seconds', updated_at = now()- WHERE id = $1 AND state = 'open' AND (lease_until IS NULL OR lease_until < now()) RETURNING id`, [id]);- return rows.length > 0;+ WHERE id = $1 AND state = 'open' AND (lease_until IS NULL OR lease_until < now()) RETURNING (floor(extract(epoch FROM lease_until) * 1000000))::bigint::text AS lease`, [id]);+ return rows[0]?.lease ?? null;}-async function settle(engine: BrainEngine, id: number, state: 'open' | 'delivered' | 'dead' | 'ended', fields: { marker?: string; attempts?: number; delayS?: number; error?: string | null }): Promise<void> {- await engine.executeRaw(+/** Settle a leased row; false when it was ended (unwatch, disable, expiry) or re-leased meanwhile, which then wins. */+async function settle(engine: BrainEngine, id: number, lease: string, state: 'open' | 'delivered' | 'dead' | 'ended', fields: { marker?: string; attempts?: number; delayS?: number; error?: string | null }): Promise<boolean> {+ const rows = await engine.executeRaw(`UPDATE wake_subscriptions SET state = $2, lease_until = NULL, updated_at = now(),delivered_marker = COALESCE($3, delivered_marker), attempts = COALESCE($4, attempts),next_attempt_at = CASE WHEN $5::int IS NULL THEN next_attempt_at ELSE now() + make_interval(secs => $5::int) END,last_error = $6- WHERE id = $1`, [id, state, fields.marker ?? null, fields.attempts ?? null, fields.delayS ?? null, fields.error ?? null]);+ WHERE id = $1 AND state = 'open' AND (floor(extract(epoch FROM lease_until) * 1000000))::bigint::text = $7 RETURNING id`,+ [id, state, fields.marker ?? null, fields.attempts ?? null, fields.delayS ?? null, fields.error ?? null, lease]);+ return rows.length > 0;+}++/** Live access members: reporting back runs only on a brain that serves its owner alone (docs/guides/wakeshift.md#who-it-is-for). */+export async function liveAccessMembers(engine: BrainEngine): Promise<number> {+ const [row] = await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM access_members WHERE disabled_at IS NULL AND tombstoned_at IS NULL');+ return Number(row?.n ?? 0);+}++function retryDelayS(attempts: number, err?: WakeshiftError): number {+ const cap = WAKE_RETRY_SCHEDULE_S[WAKE_RETRY_SCHEDULE_S.length - 1];+ if (err?.status === 429 && err.retryAfterS !== undefined) return Math.min(cap, Math.max(1, Math.ceil(err.retryAfterS)));+ return WAKE_RETRY_SCHEDULE_S[attempts - 1] ?? cap;}/** The subscribing credential still exists and, when it was bound to a member, that member is still live. */@@ -101,43 +118,44 @@ function jobOf(row: WakeRow): JobSnapshot | null {}async function deliver(engine: BrainEngine, api: WakeshiftApi, row: WakeRow, report: DrainReport): Promise<void> {- if (!(await claim(engine, row.id))) return;+ const lease = await claim(engine, row.id);+ if (!lease) return;+ const attempts = row.attempts + 1;try {if (!(await principalLive(engine, row))) {await endAtWakeshift(api, row);- await settle(engine, row.id, 'ended', { error: 'principal_revoked' });- report.ended++;+ if (await settle(engine, row.id, lease, 'ended', { error: 'principal_revoked' })) report.ended++;return;}const job = jobOf(row);const marker = jobMarker(job);const { payload, text } = jobWake(Number(row.target), job);+ let sent;try {- await api.emit({+ sent = await api.emit({name: row.event, arguments: { job: Number(row.target) }, payload, text,id: wakeEmitId(row.event, row.target, marker, row.id),to: [{ user: row.wake_user, subscription: row.ws_subscription_id }],});- await settle(engine, row.id, 'delivered', { marker, error: null });- report.delivered++;} catch (e) {const err = e instanceof WakeshiftError ? e : new WakeshiftError(0, 'network', String(e));- const attempts = row.attempts + 1;const retryable = err.status === 0 || err.status === 429 || err.status >= 500;if (!retryable || attempts >= WAKE_MAX_ATTEMPTS) {await endAtWakeshift(api, row);- await settle(engine, row.id, 'dead', { attempts, error: err.code });- report.dead++;+ if (await settle(engine, row.id, lease, 'dead', { attempts, error: err.code })) report.dead++;return;}- const cap = WAKE_RETRY_SCHEDULE_S[WAKE_RETRY_SCHEDULE_S.length - 1];- const base = WAKE_RETRY_SCHEDULE_S[attempts - 1] ?? cap;- const delayS = err.status === 429 && err.retryAfterS !== undefined ? Math.min(cap, Math.max(1, Math.ceil(err.retryAfterS))) : base;- await settle(engine, row.id, 'open', { attempts, delayS, error: err.code });- report.retried++;+ if (await settle(engine, row.id, lease, 'open', { attempts, delayS: retryDelayS(attempts, err), error: err.code })) report.retried++;+ return;+ }+ if (sent.matched > 0 || sent.duplicate) {+ if (await settle(engine, row.id, lease, 'delivered', { marker, error: null })) report.delivered++;+ } else if (await settle(engine, row.id, lease, 'ended', { marker, error: 'unmatched' })) {+ report.ended++;}} catch (e) {- await settle(engine, row.id, 'open', { error: 'drain_error' }).catch(() => {});+ const dead = attempts >= WAKE_MAX_ATTEMPTS;+ await settle(engine, row.id, lease, dead ? 'dead' : 'open', { attempts, ...(dead ? {} : { delayS: retryDelayS(attempts) }), error: 'drain_error' }).catch(() => false);throw e;}}@@ -150,12 +168,17 @@ export async function drainWakes(engine: BrainEngine, opts: { ids?: number[]; apconst report: DrainReport = { delivered: 0, retried: 0, dead: 0, ended: 0 };const api = opts.api ?? wakeshiftRuntime()?.api;if (!api) return { ...report, skipped: 'off' };+ if (await liveAccessMembers(engine) > 0) return { ...report, skipped: 'shared' };const expired = await engine.executeRaw<{ id: number; wake_user: string; ws_subscription_id: string }>(`UPDATE wake_subscriptions SET state = 'ended', last_error = 'expired', updated_at = now()WHERE state = 'open' AND expires_at < now() RETURNING id, wake_user, ws_subscription_id`);report.ended += expired.length;const rows = await engine.executeRaw<WakeRow>(CANDIDATES_SQL(opts.ids !== undefined), opts.ids !== undefined ? [opts.ids] : []);- for (const row of rows) await deliver(engine, api, row, report);+ let failure: unknown = null;+ for (const row of rows) {+ try { await deliver(engine, api, row, report); } catch (e) { failure ??= e; }+ }+ if (failure !== null) throw failure;return report;}@@ -171,13 +194,20 @@ export async function drainWakesUntilIdle(engine: BrainEngine): Promise<DrainRepreturn total;}-/** The 5 s drainer timer for a long-lived process (serve, worker). Never throws, never keeps the process alive. */+/** The 5 s drainer timer for a long-lived process (serve, worker). Never throws, never keeps the process alive; logs each distinct failure once to stderr. */export function startWakeDrain(engine: BrainEngine, opts: { intervalMs?: number; onError?: (e: unknown) => void } = {}): { stop(): void } {+ const logged = new Set<string>();+ const onError = opts.onError ?? ((e: unknown) => {+ const message = e instanceof Error ? e.message : String(e);+ if (logged.has(message) || logged.size >= 20) return;+ logged.add(message);+ console.error(`[wakeshift] wake drain failed: ${message}`);+ });let running = false;const timer = setInterval(() => {if (running) return;running = true;- drainWakes(engine).catch(e => opts.onError?.(e)).finally(() => { running = false; });+ drainWakes(engine).catch(onError).finally(() => { running = false; });}, opts.intervalMs ?? WAKE_DRAIN_TICK_MS);timer.unref?.();return { stop: () => clearInterval(timer) };diff --git a/test/helpers/fake-wakeshift.ts b/test/helpers/fake-wakeshift.tsindex efe03dbf6f..811cf4d353 100644--- a/test/helpers/fake-wakeshift.ts+++ b/test/helpers/fake-wakeshift.ts@@ -15,7 +15,7 @@ type Answer = { status: number; body: unknown; headers?: Record<string, string>export interface FakeWakeshift {url: string;calls: FakeCall[];- respond: Map<string, (body: any) => Answer>;+ respond: Map<string, (body: any) => Answer | Promise<Answer>>;stop(): void;}@@ -26,7 +26,7 @@ export function startFakeWakeshift(rawKey: string): FakeWakeshift {const kid = createHash('sha256').update(spki.subarray(spki.length - 32)).digest().subarray(0, 16).toString('base64url');const seenJti = new Set<string>();const calls: FakeCall[] = [];- const respond = new Map<string, (body: any) => Answer>();+ const respond = new Map<string, (body: any) => Answer | Promise<Answer>>();let subSeq = 0;const subs = new Map<string, string>();const server = Bun.serve({@@ -52,7 +52,7 @@ export function startFakeWakeshift(rawKey: string): FakeWakeshift {if (problem) return Response.json({ error: { code: 'unauthenticated', message: problem } }, { status: 401 });const override = respond.get(path);if (override) {- const a = override(body);+ const a = await override(body);return Response.json(a.body, { status: a.status, headers: a.headers ?? {} });}switch (path) {diff --git a/test/wakeshift.serial.test.ts b/test/wakeshift.serial.test.tsindex 59652773b4..5802eb596d 100644--- a/test/wakeshift.serial.test.ts+++ b/test/wakeshift.serial.test.ts@@ -231,6 +231,74 @@ describe('failure schedule', () => {expect(emits()).toHaveLength(0);});+ test('an unwatch while a drainer holds the lease stays ended when the emit then fails', async () => {+ const id = await seedJob('active');+ await dispatchToolCall(engine, 'watch_job', { id }, stdio);+ await finish(id);+ fake.respond.set('/events/emit', async () => {+ await dispatchToolCall(engine, 'unwatch', { job: id }, stdio);+ return { status: 500, body: { error: { code: 'internal', message: 'boom' } } };+ });+ expect(await drainWakes(engine)).toMatchObject({ retried: 0 });+ const [row] = await engine.executeRaw<{ state: string; attempts: number }>('SELECT state, attempts FROM wake_subscriptions');+ expect(row).toMatchObject({ state: 'ended', attempts: 0 });+ });++ test('a row whose processing throws backs off and does not block the rows behind it', async () => {+ const a = await seedJob('active');+ const b = await seedJob('active');+ await dispatchToolCall(engine, 'watch_job', { id: a }, stdio);+ await dispatchToolCall(engine, 'watch_job', { id: b }, stdio);+ await finish(a); await finish(b);+ await engine.executeRaw("UPDATE wake_subscriptions SET principal_kind = 'oauth_client', principal_id = 'poison-client' WHERE target = $1", [String(a)]);+ const poisoned = new Proxy(engine, {+ get(t, k, r) {+ if (k !== 'executeRaw') return Reflect.get(t, k, r);+ return (sql: string, params?: unknown[]) => sql.includes('FROM oauth_clients') && params?.[0] === 'poison-client'+ ? Promise.reject(new Error('lookup failed')) : t.executeRaw(sql, params);+ },+ });+ await expect(drainWakes(poisoned)).rejects.toThrow('lookup failed');+ const rows = await engine.executeRaw<{ target: string; state: string; attempts: number; delay: number; lease: boolean }>(+ 'SELECT target, state, attempts, round(extract(epoch FROM next_attempt_at - now()))::int AS delay, lease_until IS NOT NULL AS lease FROM wake_subscriptions ORDER BY id');+ expect(rows[0]).toMatchObject({ target: String(a), state: 'open', attempts: 1, delay: 30, lease: false });+ expect(rows[1]).toMatchObject({ target: String(b), state: 'delivered' });+ expect(emits()).toHaveLength(1);+ });++ test('an emit nobody matched ends the watch instead of calling it delivered; a duplicate counts as delivered', async () => {+ const a = await seedJob('active');+ const b = await seedJob('active');+ await dispatchToolCall(engine, 'watch_job', { id: a }, stdio);+ await dispatchToolCall(engine, 'watch_job', { id: b }, stdio);+ await finish(a); await finish(b);+ fake.respond.set('/events/emit', e => (e.arguments.job === a+ ? { status: 200, body: { matched: 0, duplicate: false, unmatched: e.to, ended: [] } }+ : { status: 200, body: { matched: 0, duplicate: true, unmatched: [], ended: [] } }));+ expect(await drainWakes(engine)).toMatchObject({ delivered: 1, ended: 1 });+ const rows = await engine.executeRaw<{ target: string; state: string; last_error: string | null }>('SELECT target, state, last_error FROM wake_subscriptions ORDER BY id');+ expect(rows).toEqual([{ target: String(a), state: 'ended', last_error: 'unmatched' }, { target: String(b), state: 'delivered', last_error: null }]);+ });++ test('a member added after enable pauses reporting back: watch_job refuses and the drainer sends nothing', async () => {+ const id = await seedJob('active');+ await dispatchToolCall(engine, 'watch_job', { id }, stdio);+ await finish(id);+ await engine.executeRaw("INSERT INTO access_members (id) VALUES ('alice-example') ON CONFLICT DO NOTHING");+ try {+ const other = await seedJob('active');+ const r = await dispatchToolCall(engine, 'watch_job', { id: other }, stdio);+ expect(r.isError).toBe(true);+ const env = JSON.parse(r.content[0]!.text);+ expect(env.code ?? env.error).toBe('wakeshift_shared_brain');+ expect(await drainWakes(engine)).toMatchObject({ skipped: 'shared' });+ expect(emits()).toHaveLength(0);+ expect(fake.calls.filter(c => c.path === '/event-subscriptions')).toHaveLength(1);+ } finally {+ await engine.executeRaw("DELETE FROM access_members WHERE id = 'alice-example'");+ }+ });+test('Wakeshift down during watch_job: a clear error with the wait_job fix and no row', async () => {const id = await seedJob('active');fake.respond.set('/event-subscriptions', () => ({ status: 503, body: { error: { code: 'unavailable', message: 'down' } } }));@@ -320,6 +388,25 @@ describe('gbrain wakeshift', () => {}});+ test('enable with the default URL drops a custom URL saved earlier', async () => {+ writeConfig({ enabled: true, url: 'http://127.0.0.1:9' });+ const realFetch = globalThis.fetch;+ const hosts: string[] = [];+ globalThis.fetch = (async (input: string | URL | Request) => {+ const u = new URL(typeof input === 'string' || input instanceof URL ? input : input.url);+ hosts.push(u.origin);+ return Response.json(u.pathname.endsWith('/app') ? { id: 'app_test', name: 'Test app' } : { events: [] });+ }) as typeof fetch;+ try {+ const r = await cli(['enable', '--yes', '--url', 'https://app.wakeshift.ai']);+ expect(r.error).toBeUndefined();+ } finally {+ globalThis.fetch = realFetch;+ }+ expect([...new Set(hosts)]).toEqual(['https://app.wakeshift.ai']);+ expect(JSON.parse(readFileSync(join(home, '.gbrain', 'config.json'), 'utf8')).wakeshift).toEqual({ enabled: true });+ });+test('enable without --yes off a terminal refuses through consent, sending nothing', async () => {writeConfig(null);const r = await cli(['enable', '--url', fake.url]);