Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Wakeshift drainer: lease-token settle, per-row failure backoff, unmatched emits end the watch, members pause reporting back, enable resets a stale URL

garrytanfee991ae6d8e
diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.md
index 064884b585..ad2ee2b9b5 100644
--- a/docs/guides/error-codes.md
+++ b/docs/guides/error-codes.md
@@ -5194,7 +5194,7 @@ More: [docs/guides/wakeshift.md#troubleshooting](../../docs/guides/wakeshift.md#
| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
|---|---|---|---|---|---|---|
-| gbrain wakeshift enable refused: this brain has live access members, and this release reports back only on brains that serve their owner alone. | Every member of the brain owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. Shared brains get a terms step in a later release; until then reporting back stays off there. | Leave reporting back off on this brain; agents can use wait_job or gbrain jobs wait <id>. | user | `gbrain doctor --json` | 1 | no |
+| Reporting back refused (gbrain wakeshift enable, or watch_job after a member was added): this brain has live access members, and this release reports back only on brains that serve their owner alone. | Every member of the brain owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. Shared brains get a terms step in a later release; until then reporting back stays off there. | Leave reporting back off on this brain; agents can use wait_job or gbrain jobs wait <id>. | user | `gbrain doctor --json` | 1 | no |
More: [docs/guides/wakeshift.md#who-it-is-for](../../docs/guides/wakeshift.md#who-it-is-for)
diff --git a/docs/guides/wakeshift.md b/docs/guides/wakeshift.md
index 611605961b..3b642ff9e7 100644
--- a/docs/guides/wakeshift.md
+++ b/docs/guides/wakeshift.md
@@ -26,7 +26,7 @@ A Wakeshift key is an Ed25519 private key: `wst_` followed by 50 base58 characte
## Who it is for
-This release reports back only on brains that serve their owner alone. `gbrain wakeshift enable` refuses a brain with live access members (`gbrain access member list`), because every member of the owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. Shared brains get a terms step in a later release.
+This release reports back only on brains that serve their owner alone. `gbrain wakeshift enable` refuses a brain with live access members (`gbrain access member list`), because every member of the owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. A member added after enable pauses reporting back: watch_job refuses with `wakeshift_shared_brain`, no wakes are sent, and `gbrain doctor` warns. Shared brains get a terms step in a later release.
A watch belongs to the credential that made it: the OAuth client or token, or the local stdio pipe. `list_watches` and `unwatch` touch only the caller's own watches. `watch_job` runs the same owner check as `get_job`, so an agent-scoped token can watch only the jobs it submitted. A job it doesn't own reads exactly like a missing one, and nothing is sent. If the credential is revoked or its client is deleted before the job finishes, the watch ends without a wake.
@@ -46,7 +46,7 @@ Delivery is a pull, not a hook. `gbrain serve` (stdio and HTTP) and the job work
On PGLite there is no background worker. A job submitted from a stdio session runs when `gbrain jobs work` drains the queue, after `gbrain serve` stops, and that run delivers its wakes before it exits. A wake it can't deliver arrives from the next `gbrain serve`.
-Each watch lasts 7 days and delivers once. A failed delivery is retried at most 5 times, after 30 s, 2 min, 8 min and 30 min (a rate limit waits its `Retry-After`). Any other refusal is final. A watch that dies is ended at Wakeshift, so the conversation is told. Jobs are never blocked or slowed by Wakeshift: it is down only means the wake comes late or not at all.
+Each watch lasts 7 days and delivers once. A failed delivery is retried at most 5 times, after 30 s, 2 min, 8 min and 30 min (a rate limit waits its `Retry-After`). Any other refusal is final. A watch that dies is ended at Wakeshift, so the conversation is told. A wake Wakeshift matched to no open conversation (the user ended it, or it expired there) ends the watch with `last_error: unmatched` rather than counting as delivered. Jobs are never blocked or slowed by Wakeshift: it is down only means the wake comes late or not at all.
Repo-less brains work the same way. Watches are operational database rows, not brain content.
diff --git a/src/commands/doctor/checks/wakeshift.ts b/src/commands/doctor/checks/wakeshift.ts
index 9bb1c57bce..afef509203 100644
--- a/src/commands/doctor/checks/wakeshift.ts
+++ b/src/commands/doctor/checks/wakeshift.ts
@@ -2,13 +2,15 @@
* wakeshift (plan PR 1 item 9): reporting back through Wakeshift. Off is
* `ok` (disabled by choice). It warns when the key file is readable by other
* users, when reporting back is enabled but no usable key is found (the watch
- * tools stay hidden), and when watches died (5 failed attempts or a final
+ * tools stay hidden), when the brain gained live access members after enable
+ * (reporting back pauses), and when watches died (5 failed attempts or a final
* Wakeshift refusal) or hit `id_reused` (a bug signal) in the last 7 days.
* Shows the key's kid fingerprint only, never the key. Guide: docs/guides/wakeshift.md.
*/
import { loadConfigFileOnly } from '../../../core/config.ts';
import { parseWakeshiftKey } from '../../../core/wakeshift/client.ts';
import { readWakeshiftKey, wakeshiftFileConfig, wakeshiftKeyFile } from '../../../core/wakeshift/config.ts';
+import { liveAccessMembers } from '../../../core/wakeshift/drain.ts';
import type { Check } from '../../doctor.ts';
import { connectedEngine, type DoctorContext, type DoctorEntry } from '../context.ts';
@@ -38,13 +40,16 @@ async function runWakeshiftCheck(ctx: DoctorContext): Promise<Check[]> {
counts[r.state] = (counts[r.state] ?? 0) + Number(r.n);
if (r.id_reused) counts.id_reused! += Number(r.n);
}
- const details = { enabled, key_source: read.source, kid, watches_7d: counts, docs: DOCS };
- const message = counts.id_reused! > 0
+ const members = await liveAccessMembers(connectedEngine(ctx));
+ const details = { enabled, key_source: read.source, kid, watches_7d: counts, live_members: members, docs: DOCS };
+ const message = members > 0
+ ? `Reporting back is paused: this brain now has ${members} live access member(s), and this release reports back only on a brain that serves its owner alone. watch_job refuses and no wakes are sent. Fix: gbrain wakeshift disable, or remove the members.`
+ : counts.id_reused! > 0
? `${counts.id_reused} watch(es) were refused with id_reused in the last 7 days: a gbrain bug, so please report it with this output.`
: counts.dead! > 0
? `${counts.dead} wake(s) died in the last 7 days (5 failed attempts or a final Wakeshift refusal); the agents were told the watch ended. Check the key and the app with gbrain wakeshift test.`
: `Reporting back through Wakeshift is on (key kid ${kid}); ${counts.open} open watch(es), ${counts.delivered} delivered in the last 7 days.`;
- checks.push({ name: 'wakeshift', status: counts.id_reused! > 0 || counts.dead! > 0 ? 'warn' : 'ok', message, details });
+ checks.push({ name: 'wakeshift', status: members > 0 || counts.id_reused! > 0 || counts.dead! > 0 ? 'warn' : 'ok', message, details });
}
return checks;
}
diff --git a/src/commands/wakeshift.ts b/src/commands/wakeshift.ts
index f3cc3e96f9..6b18f3c60f 100644
--- a/src/commands/wakeshift.ts
+++ b/src/commands/wakeshift.ts
@@ -16,7 +16,7 @@ import { requireConsent } from '../core/consent.ts';
import { WakeshiftError, WakeshiftKeyError, normalizeWakeshiftUrl, parseWakeshiftKey, wakeshiftClient, type WakeshiftKey } from '../core/wakeshift/client.ts';
import { readWakeshiftKey, wakeshiftFileConfig, wakeshiftKeyFile, wakeshiftPublished, wakeshiftSecret } from '../core/wakeshift/config.ts';
import { wakeshiftManifest } from '../core/wakeshift/events.ts';
-import { endAtWakeshift } from '../core/wakeshift/drain.ts';
+import { endAtWakeshift, liveAccessMembers } from '../core/wakeshift/drain.ts';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
const HELP = `gbrain wakeshift <command> — report back to agent conversations when jobs finish (off by default)
@@ -59,11 +59,6 @@ function loadKey(): { key: WakeshiftKey; source: 'env' | 'file' } {
}
}
-async function liveMembers(engine: BrainEngine): Promise<number> {
- const [row] = await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM access_members WHERE disabled_at IS NULL AND tombstoned_at IS NULL');
- return Number(row?.n ?? 0);
-}
-
async function watchCounts(engine: BrainEngine): Promise<Record<string, number>> {
try {
const rows = await engine.executeRaw<{ state: string; last_error: string | null; n: number }>(
@@ -122,7 +117,7 @@ async function enable(engine: BrainEngine, args: string[]): Promise<void> {
console.error(`[wakeshift] saved the key to ${wakeshiftKeyFile()} (0600).`);
}
const { key, source } = loadKey();
- const members = await liveMembers(engine);
+ const members = await liveAccessMembers(engine);
if (members > 0) {
throw opError('wakeshift_shared_brain', `This brain has ${members} live access member(s); this release reports back only on a brain that serves its owner alone. Nothing was turned on.`,
'Leave reporting back off on this brain; agents wait for a job with wait_job or gbrain jobs wait and the job id.');
@@ -147,7 +142,11 @@ async function enable(engine: BrainEngine, args: string[]): Promise<void> {
'Check the key belongs to your Wakeshift app (gbrain wakeshift test) and retry.', { reason: e instanceof WakeshiftError ? e.code : 'error' });
}
wakeshiftSecret();
- writeFilePlane(w => ({ ...w, enabled: true, ...(url !== 'https://app.wakeshift.ai' ? { url } : {}) }));
+ writeFilePlane(w => {
+ const next: { enabled?: boolean; url?: string } = { ...w, enabled: true, url };
+ if (url === 'https://app.wakeshift.ai') delete next.url;
+ return next;
+ });
say(args.includes('--json'), { enabled: true, url, app: { id: app?.id, name: app?.name }, key: { source, kid: key.kid } }, [
`Wakeshift reporting back is on (app ${app?.name ?? '?'}, key kid ${key.kid}, from ${source}).`,
'Agents now see watch_job, list_watches and unwatch; gbrain serve and the job worker deliver wakes.',
diff --git a/src/core/error-registry.ts b/src/core/error-registry.ts
index 4b7b2a85aa..999f15865e 100644
--- a/src/core/error-registry.ts
+++ b/src/core/error-registry.ts
@@ -848,7 +848,7 @@ export const CODES = {
wakeshift_needs_conversation: { class: 'caller', summary: "watch_job wakes an MCP conversation, and a local CLI run has none to wake.", docs: 'docs/guides/wakeshift.md#delivery-and-latency', suggestion: 'Run gbrain jobs wait <id> to block until the job finishes.' },
wakeshift_not_enabled: { class: 'caller', summary: "Reporting back through Wakeshift is off on this brain (the owner has not run gbrain wakeshift enable, or the key is missing).", why: "Wakeshift is off by default; only the brain owner turns it on, on the brain host, because it sends wake metadata to a third-party service.", docs: 'docs/guides/wakeshift.md#turn-it-on', suggestion: 'Use wait_job (or gbrain jobs wait <id>) instead, or ask the brain owner to run gbrain wakeshift enable.' },
wakeshift_refused: { class: 'caller', summary: "Wakeshift refused the request; the reason carries its error code. Nothing was watched or sent.", why: "Wakeshift checks each request (the signed key, the event and its arguments, the conversation's session key). A refusal other than a rate limit does not go away on retry.", docs: 'docs/guides/wakeshift.md#troubleshooting', suggestion: 'Call wait_job to wait without Wakeshift; the brain owner can run gbrain wakeshift test to check the key and the app.' },
- wakeshift_shared_brain: { class: 'host_only', actor: 'user', summary: "gbrain wakeshift enable refused: this brain has live access members, and this release reports back only on brains that serve their owner alone.", why: "Every member of the brain owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. Shared brains get a terms step in a later release; until then reporting back stays off there.", docs: 'docs/guides/wakeshift.md#who-it-is-for', suggestion: 'Leave reporting back off on this brain; agents can use wait_job or gbrain jobs wait <id>.' },
+ wakeshift_shared_brain: { class: 'host_only', actor: 'user', summary: "Reporting back refused (gbrain wakeshift enable, or watch_job after a member was added): this brain has live access members, and this release reports back only on brains that serve their owner alone.", why: "Every member of the brain owner's Wakeshift organization can read the wakes it sends, and Wakeshift's terms make the operator pass its Connector Terms to each member. Shared brains get a terms step in a later release; until then reporting back stays off there.", docs: 'docs/guides/wakeshift.md#who-it-is-for', suggestion: 'Leave reporting back off on this brain; agents can use wait_job or gbrain jobs wait <id>.' },
wakeshift_unavailable: { class: 'unavailable', retryable: true, summary: "Wakeshift could not be reached or answered with a server error or rate limit; nothing was watched.", why: "gbrain makes one request with a 10 s timeout and never retries inside a tool call, so a slow or down Wakeshift never holds the call.", docs: 'docs/guides/wakeshift.md#troubleshooting', suggestion: 'Retry in a minute, or call wait_job to wait without Wakeshift.' },
withdrawal_capacity: { class: 'caller', summary: "Withdrawal capacity." },
withdrawal_mirror_failed: { class: 'unavailable', summary: "A page's Git or embedding work is withheld because its withdrawal mirror failed before rewriting the page's file; the file may still hold the withdrawn text.", why: "A withdrawal's Git and embedding effects touch only the pages whose file its mirror recorded rewriting (GBRA-130 A-12); Git never commits a page the mirror passed without rewriting its file. Retrying the request's effects reruns the mirror for this page; once the mirror commits, the withheld work is re-armed automatically.", suggestion: 'Read the parked effects with gbrain doctor --only parked_effects, fix the cause the mirror recorded, then preview gbrain sources writer retry-effects <source> --request-id <id> --dry-run and run it without --dry-run.' },
diff --git a/src/core/ops/wakeshift.ts b/src/core/ops/wakeshift.ts
index b515c23242..3439abdafa 100644
--- a/src/core/ops/wakeshift.ts
+++ b/src/core/ops/wakeshift.ts
@@ -16,7 +16,7 @@ import { memberIdentityFor } from '../access/caller.ts';
import { WakeshiftError, type WakeshiftCall } from '../wakeshift/client.ts';
import { wakeshiftBrainKey, wakeshiftRuntime, wakeshiftSecret, wakeshiftUserId } from '../wakeshift/config.ts';
import { JOB_FINISHED } from '../wakeshift/events.ts';
-import { drainWakes, endAtWakeshift } from '../wakeshift/drain.ts';
+import { drainWakes, endAtWakeshift, liveAccessMembers } from '../wakeshift/drain.ts';
const WATCH_TTL = '7d';
const TERMINAL = new Set(['completed', 'failed', 'dead', 'cancelled']);
@@ -106,6 +106,11 @@ const watch_job: Operation = {
}
const runtime = wakeshiftRuntime();
if (!runtime) throw notEnabled();
+ const members = await liveAccessMembers(ctx.engine);
+ if (members > 0) {
+ throw opError('wakeshift_shared_brain', `This brain now has ${members} live access member(s), so reporting back is paused; nothing was watched.`,
+ `Call wait_job ${id} to wait for the job.`, { fix: waitFix(id) });
+ }
const owner = agentOwnerFence(ctx, 'watch_job');
if (owner !== null) await assertJobOwned(ctx, id, owner);
const [job] = await ctx.engine.executeRaw<{ status: string }>('SELECT status FROM minion_jobs WHERE id = $1', [id]);
diff --git a/src/core/wakeshift/drain.ts b/src/core/wakeshift/drain.ts
index b2356c43ac..94fa266af2 100644
--- a/src/core/wakeshift/drain.ts
+++ b/src/core/wakeshift/drain.ts
@@ -12,7 +12,8 @@
* never emit one row; 5 attempts backing off 30 s, 2 m, 8 m, 30 m (a 429 waits
* its Retry-After within the same cap); any other 4xx is final. A dead,
* revoked or expired watch is ended at Wakeshift once, so the conversation is
- * told. Off or keyless: the tick returns before touching the database.
+ * told. Off or keyless: the tick returns before touching the database. A brain
+ * with live access members sends nothing (watch_job refuses there too).
*/
import type { BrainEngine } from '../engine.ts';
import { memberIdentityFor } from '../access/caller.ts';
@@ -45,7 +46,7 @@ export interface WakeRow {
finished_us: string | null;
}
-export interface DrainReport { delivered: number; retried: number; dead: number; ended: number; skipped?: 'off' }
+export interface DrainReport { delivered: number; retried: number; dead: number; ended: number; skipped?: 'off' | 'shared' }
const CANDIDATES_SQL = (byId: boolean) => `
SELECT w.id, w.event, w.target, w.principal_kind, w.principal_id, w.member_id, w.wake_user, w.ws_subscription_id, w.attempts,
@@ -61,20 +62,36 @@ SELECT w.id, w.event, w.target, w.principal_kind, w.principal_id, w.member_id, w
ORDER BY w.next_attempt_at, w.id
LIMIT ${WAKE_DRAIN_BATCH}`;
-async function claim(engine: BrainEngine, id: number): Promise<boolean> {
- const rows = await engine.executeRaw<{ id: number }>(
+/** Lease one open row; the lease's timestamp in epoch microseconds (text, so no driver rounds it) is the token settle() must match. */
+async function claim(engine: BrainEngine, id: number): Promise<string | null> {
+ const rows = await engine.executeRaw<{ lease: string }>(
`UPDATE wake_subscriptions SET lease_until = now() + interval '${LEASE_S} seconds', updated_at = now()
- WHERE id = $1 AND state = 'open' AND (lease_until IS NULL OR lease_until < now()) RETURNING id`, [id]);
- return rows.length > 0;
+ WHERE id = $1 AND state = 'open' AND (lease_until IS NULL OR lease_until < now()) RETURNING (floor(extract(epoch FROM lease_until) * 1000000))::bigint::text AS lease`, [id]);
+ return rows[0]?.lease ?? null;
}
-async function settle(engine: BrainEngine, id: number, state: 'open' | 'delivered' | 'dead' | 'ended', fields: { marker?: string; attempts?: number; delayS?: number; error?: string | null }): Promise<void> {
- await engine.executeRaw(
+/** Settle a leased row; false when it was ended (unwatch, disable, expiry) or re-leased meanwhile, which then wins. */
+async function settle(engine: BrainEngine, id: number, lease: string, state: 'open' | 'delivered' | 'dead' | 'ended', fields: { marker?: string; attempts?: number; delayS?: number; error?: string | null }): Promise<boolean> {
+ const rows = await engine.executeRaw(
`UPDATE wake_subscriptions SET state = $2, lease_until = NULL, updated_at = now(),
delivered_marker = COALESCE($3, delivered_marker), attempts = COALESCE($4, attempts),
next_attempt_at = CASE WHEN $5::int IS NULL THEN next_attempt_at ELSE now() + make_interval(secs => $5::int) END,
last_error = $6
- WHERE id = $1`, [id, state, fields.marker ?? null, fields.attempts ?? null, fields.delayS ?? null, fields.error ?? null]);
+ WHERE id = $1 AND state = 'open' AND (floor(extract(epoch FROM lease_until) * 1000000))::bigint::text = $7 RETURNING id`,
+ [id, state, fields.marker ?? null, fields.attempts ?? null, fields.delayS ?? null, fields.error ?? null, lease]);
+ return rows.length > 0;
+}
+
+/** Live access members: reporting back runs only on a brain that serves its owner alone (docs/guides/wakeshift.md#who-it-is-for). */
+export async function liveAccessMembers(engine: BrainEngine): Promise<number> {
+ const [row] = await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM access_members WHERE disabled_at IS NULL AND tombstoned_at IS NULL');
+ return Number(row?.n ?? 0);
+}
+
+function retryDelayS(attempts: number, err?: WakeshiftError): number {
+ const cap = WAKE_RETRY_SCHEDULE_S[WAKE_RETRY_SCHEDULE_S.length - 1];
+ if (err?.status === 429 && err.retryAfterS !== undefined) return Math.min(cap, Math.max(1, Math.ceil(err.retryAfterS)));
+ return WAKE_RETRY_SCHEDULE_S[attempts - 1] ?? cap;
}
/** The subscribing credential still exists and, when it was bound to a member, that member is still live. */
@@ -101,43 +118,44 @@ function jobOf(row: WakeRow): JobSnapshot | null {
}
async function deliver(engine: BrainEngine, api: WakeshiftApi, row: WakeRow, report: DrainReport): Promise<void> {
- if (!(await claim(engine, row.id))) return;
+ const lease = await claim(engine, row.id);
+ if (!lease) return;
+ const attempts = row.attempts + 1;
try {
if (!(await principalLive(engine, row))) {
await endAtWakeshift(api, row);
- await settle(engine, row.id, 'ended', { error: 'principal_revoked' });
- report.ended++;
+ if (await settle(engine, row.id, lease, 'ended', { error: 'principal_revoked' })) report.ended++;
return;
}
const job = jobOf(row);
const marker = jobMarker(job);
const { payload, text } = jobWake(Number(row.target), job);
+ let sent;
try {
- await api.emit({
+ sent = await api.emit({
name: row.event, arguments: { job: Number(row.target) }, payload, text,
id: wakeEmitId(row.event, row.target, marker, row.id),
to: [{ user: row.wake_user, subscription: row.ws_subscription_id }],
});
- await settle(engine, row.id, 'delivered', { marker, error: null });
- report.delivered++;
} catch (e) {
const err = e instanceof WakeshiftError ? e : new WakeshiftError(0, 'network', String(e));
- const attempts = row.attempts + 1;
const retryable = err.status === 0 || err.status === 429 || err.status >= 500;
if (!retryable || attempts >= WAKE_MAX_ATTEMPTS) {
await endAtWakeshift(api, row);
- await settle(engine, row.id, 'dead', { attempts, error: err.code });
- report.dead++;
+ if (await settle(engine, row.id, lease, 'dead', { attempts, error: err.code })) report.dead++;
return;
}
- const cap = WAKE_RETRY_SCHEDULE_S[WAKE_RETRY_SCHEDULE_S.length - 1];
- const base = WAKE_RETRY_SCHEDULE_S[attempts - 1] ?? cap;
- const delayS = err.status === 429 && err.retryAfterS !== undefined ? Math.min(cap, Math.max(1, Math.ceil(err.retryAfterS))) : base;
- await settle(engine, row.id, 'open', { attempts, delayS, error: err.code });
- report.retried++;
+ if (await settle(engine, row.id, lease, 'open', { attempts, delayS: retryDelayS(attempts, err), error: err.code })) report.retried++;
+ return;
+ }
+ if (sent.matched > 0 || sent.duplicate) {
+ if (await settle(engine, row.id, lease, 'delivered', { marker, error: null })) report.delivered++;
+ } else if (await settle(engine, row.id, lease, 'ended', { marker, error: 'unmatched' })) {
+ report.ended++;
}
} catch (e) {
- await settle(engine, row.id, 'open', { error: 'drain_error' }).catch(() => {});
+ const dead = attempts >= WAKE_MAX_ATTEMPTS;
+ await settle(engine, row.id, lease, dead ? 'dead' : 'open', { attempts, ...(dead ? {} : { delayS: retryDelayS(attempts) }), error: 'drain_error' }).catch(() => false);
throw e;
}
}
@@ -150,12 +168,17 @@ export async function drainWakes(engine: BrainEngine, opts: { ids?: number[]; ap
const report: DrainReport = { delivered: 0, retried: 0, dead: 0, ended: 0 };
const api = opts.api ?? wakeshiftRuntime()?.api;
if (!api) return { ...report, skipped: 'off' };
+ if (await liveAccessMembers(engine) > 0) return { ...report, skipped: 'shared' };
const expired = await engine.executeRaw<{ id: number; wake_user: string; ws_subscription_id: string }>(
`UPDATE wake_subscriptions SET state = 'ended', last_error = 'expired', updated_at = now()
WHERE state = 'open' AND expires_at < now() RETURNING id, wake_user, ws_subscription_id`);
report.ended += expired.length;
const rows = await engine.executeRaw<WakeRow>(CANDIDATES_SQL(opts.ids !== undefined), opts.ids !== undefined ? [opts.ids] : []);
- for (const row of rows) await deliver(engine, api, row, report);
+ let failure: unknown = null;
+ for (const row of rows) {
+ try { await deliver(engine, api, row, report); } catch (e) { failure ??= e; }
+ }
+ if (failure !== null) throw failure;
return report;
}
@@ -171,13 +194,20 @@ export async function drainWakesUntilIdle(engine: BrainEngine): Promise<DrainRep
return total;
}
-/** The 5 s drainer timer for a long-lived process (serve, worker). Never throws, never keeps the process alive. */
+/** The 5 s drainer timer for a long-lived process (serve, worker). Never throws, never keeps the process alive; logs each distinct failure once to stderr. */
export function startWakeDrain(engine: BrainEngine, opts: { intervalMs?: number; onError?: (e: unknown) => void } = {}): { stop(): void } {
+ const logged = new Set<string>();
+ const onError = opts.onError ?? ((e: unknown) => {
+ const message = e instanceof Error ? e.message : String(e);
+ if (logged.has(message) || logged.size >= 20) return;
+ logged.add(message);
+ console.error(`[wakeshift] wake drain failed: ${message}`);
+ });
let running = false;
const timer = setInterval(() => {
if (running) return;
running = true;
- drainWakes(engine).catch(e => opts.onError?.(e)).finally(() => { running = false; });
+ drainWakes(engine).catch(onError).finally(() => { running = false; });
}, opts.intervalMs ?? WAKE_DRAIN_TICK_MS);
timer.unref?.();
return { stop: () => clearInterval(timer) };
diff --git a/test/helpers/fake-wakeshift.ts b/test/helpers/fake-wakeshift.ts
index efe03dbf6f..811cf4d353 100644
--- a/test/helpers/fake-wakeshift.ts
+++ b/test/helpers/fake-wakeshift.ts
@@ -15,7 +15,7 @@ type Answer = { status: number; body: unknown; headers?: Record<string, string>
export interface FakeWakeshift {
url: string;
calls: FakeCall[];
- respond: Map<string, (body: any) => Answer>;
+ respond: Map<string, (body: any) => Answer | Promise<Answer>>;
stop(): void;
}
@@ -26,7 +26,7 @@ export function startFakeWakeshift(rawKey: string): FakeWakeshift {
const kid = createHash('sha256').update(spki.subarray(spki.length - 32)).digest().subarray(0, 16).toString('base64url');
const seenJti = new Set<string>();
const calls: FakeCall[] = [];
- const respond = new Map<string, (body: any) => Answer>();
+ const respond = new Map<string, (body: any) => Answer | Promise<Answer>>();
let subSeq = 0;
const subs = new Map<string, string>();
const server = Bun.serve({
@@ -52,7 +52,7 @@ export function startFakeWakeshift(rawKey: string): FakeWakeshift {
if (problem) return Response.json({ error: { code: 'unauthenticated', message: problem } }, { status: 401 });
const override = respond.get(path);
if (override) {
- const a = override(body);
+ const a = await override(body);
return Response.json(a.body, { status: a.status, headers: a.headers ?? {} });
}
switch (path) {
diff --git a/test/wakeshift.serial.test.ts b/test/wakeshift.serial.test.ts
index 59652773b4..5802eb596d 100644
--- a/test/wakeshift.serial.test.ts
+++ b/test/wakeshift.serial.test.ts
@@ -231,6 +231,74 @@ describe('failure schedule', () => {
expect(emits()).toHaveLength(0);
});
+ test('an unwatch while a drainer holds the lease stays ended when the emit then fails', async () => {
+ const id = await seedJob('active');
+ await dispatchToolCall(engine, 'watch_job', { id }, stdio);
+ await finish(id);
+ fake.respond.set('/events/emit', async () => {
+ await dispatchToolCall(engine, 'unwatch', { job: id }, stdio);
+ return { status: 500, body: { error: { code: 'internal', message: 'boom' } } };
+ });
+ expect(await drainWakes(engine)).toMatchObject({ retried: 0 });
+ const [row] = await engine.executeRaw<{ state: string; attempts: number }>('SELECT state, attempts FROM wake_subscriptions');
+ expect(row).toMatchObject({ state: 'ended', attempts: 0 });
+ });
+
+ test('a row whose processing throws backs off and does not block the rows behind it', async () => {
+ const a = await seedJob('active');
+ const b = await seedJob('active');
+ await dispatchToolCall(engine, 'watch_job', { id: a }, stdio);
+ await dispatchToolCall(engine, 'watch_job', { id: b }, stdio);
+ await finish(a); await finish(b);
+ await engine.executeRaw("UPDATE wake_subscriptions SET principal_kind = 'oauth_client', principal_id = 'poison-client' WHERE target = $1", [String(a)]);
+ const poisoned = new Proxy(engine, {
+ get(t, k, r) {
+ if (k !== 'executeRaw') return Reflect.get(t, k, r);
+ return (sql: string, params?: unknown[]) => sql.includes('FROM oauth_clients') && params?.[0] === 'poison-client'
+ ? Promise.reject(new Error('lookup failed')) : t.executeRaw(sql, params);
+ },
+ });
+ await expect(drainWakes(poisoned)).rejects.toThrow('lookup failed');
+ const rows = await engine.executeRaw<{ target: string; state: string; attempts: number; delay: number; lease: boolean }>(
+ 'SELECT target, state, attempts, round(extract(epoch FROM next_attempt_at - now()))::int AS delay, lease_until IS NOT NULL AS lease FROM wake_subscriptions ORDER BY id');
+ expect(rows[0]).toMatchObject({ target: String(a), state: 'open', attempts: 1, delay: 30, lease: false });
+ expect(rows[1]).toMatchObject({ target: String(b), state: 'delivered' });
+ expect(emits()).toHaveLength(1);
+ });
+
+ test('an emit nobody matched ends the watch instead of calling it delivered; a duplicate counts as delivered', async () => {
+ const a = await seedJob('active');
+ const b = await seedJob('active');
+ await dispatchToolCall(engine, 'watch_job', { id: a }, stdio);
+ await dispatchToolCall(engine, 'watch_job', { id: b }, stdio);
+ await finish(a); await finish(b);
+ fake.respond.set('/events/emit', e => (e.arguments.job === a
+ ? { status: 200, body: { matched: 0, duplicate: false, unmatched: e.to, ended: [] } }
+ : { status: 200, body: { matched: 0, duplicate: true, unmatched: [], ended: [] } }));
+ expect(await drainWakes(engine)).toMatchObject({ delivered: 1, ended: 1 });
+ const rows = await engine.executeRaw<{ target: string; state: string; last_error: string | null }>('SELECT target, state, last_error FROM wake_subscriptions ORDER BY id');
+ expect(rows).toEqual([{ target: String(a), state: 'ended', last_error: 'unmatched' }, { target: String(b), state: 'delivered', last_error: null }]);
+ });
+
+ test('a member added after enable pauses reporting back: watch_job refuses and the drainer sends nothing', async () => {
+ const id = await seedJob('active');
+ await dispatchToolCall(engine, 'watch_job', { id }, stdio);
+ await finish(id);
+ await engine.executeRaw("INSERT INTO access_members (id) VALUES ('alice-example') ON CONFLICT DO NOTHING");
+ try {
+ const other = await seedJob('active');
+ const r = await dispatchToolCall(engine, 'watch_job', { id: other }, stdio);
+ expect(r.isError).toBe(true);
+ const env = JSON.parse(r.content[0]!.text);
+ expect(env.code ?? env.error).toBe('wakeshift_shared_brain');
+ expect(await drainWakes(engine)).toMatchObject({ skipped: 'shared' });
+ expect(emits()).toHaveLength(0);
+ expect(fake.calls.filter(c => c.path === '/event-subscriptions')).toHaveLength(1);
+ } finally {
+ await engine.executeRaw("DELETE FROM access_members WHERE id = 'alice-example'");
+ }
+ });
+
test('Wakeshift down during watch_job: a clear error with the wait_job fix and no row', async () => {
const id = await seedJob('active');
fake.respond.set('/event-subscriptions', () => ({ status: 503, body: { error: { code: 'unavailable', message: 'down' } } }));
@@ -320,6 +388,25 @@ describe('gbrain wakeshift', () => {
}
});
+ test('enable with the default URL drops a custom URL saved earlier', async () => {
+ writeConfig({ enabled: true, url: 'http://127.0.0.1:9' });
+ const realFetch = globalThis.fetch;
+ const hosts: string[] = [];
+ globalThis.fetch = (async (input: string | URL | Request) => {
+ const u = new URL(typeof input === 'string' || input instanceof URL ? input : input.url);
+ hosts.push(u.origin);
+ return Response.json(u.pathname.endsWith('/app') ? { id: 'app_test', name: 'Test app' } : { events: [] });
+ }) as typeof fetch;
+ try {
+ const r = await cli(['enable', '--yes', '--url', 'https://app.wakeshift.ai']);
+ expect(r.error).toBeUndefined();
+ } finally {
+ globalThis.fetch = realFetch;
+ }
+ expect([...new Set(hosts)]).toEqual(['https://app.wakeshift.ai']);
+ expect(JSON.parse(readFileSync(join(home, '.gbrain', 'config.json'), 'utf8')).wakeshift).toEqual({ enabled: true });
+ });
+
test('enable without --yes off a terminal refuses through consent, sending nothing', async () => {
writeConfig(null);
const r = await cli(['enable', '--url', fake.url]);