Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Record cron runs reported while serve or the job worker holds a PGLite brain; revoking a credential always ends its watches (GST-2679) (#9369)

Garry Tan5b2fbd1ba437
diff --git a/changelog.d/garrytan-wakeshift-emit-spool.md b/changelog.d/garrytan-wakeshift-emit-spool.md
new file mode 100644
index 0000000000..5a55bca0ea
--- /dev/null
+++ b/changelog.d/garrytan-wakeshift-emit-spool.md
@@ -0,0 +1,8 @@
+**On a PGLite brain held by `gbrain serve` or the job worker, `gbrain wakeshift emit cron.finished` now records the run instead of dropping it, and revoking a credential always ends its watches.**
+
+### Fixed
+
+- A cron run that finished while `gbrain serve` or `gbrain jobs work` held the PGLite brain was lost. Emit now saves it as a small private file (0600) under the brain's own gbrain home, and the process holding the brain records it within seconds with its original finish time, so a watcher still wakes. Two processes reading the folder at once never record a run twice. A file that is malformed, oversized or has an impossible time goes to `rejected/` with one stderr line and never holds back the runs after it. At most 1,000 runs wait per brain. Postgres brains and brains with reporting back off never use the folder.
+- The next direct `emit` also records any leftover runs, and it records its own run even when the leftovers can't be read.
+- `gbrain auth revoke --id <id>` with a legacy token id written in uppercase now ends that token's watches and asks Wakeshift to forget its user.
+- When removing an access member failed to ask Wakeshift to forget that member's user, the wake drainer now asks again. A member who is still active keeps their Wakeshift user.
diff --git a/docs/architecture/key-files/wakeshift.md b/docs/architecture/key-files/wakeshift.md
index 4d108390b8..8bb3366fb1 100644
--- a/docs/architecture/key-files/wakeshift.md
+++ b/docs/architecture/key-files/wakeshift.md
@@ -3,5 +3,5 @@
[Subsystem index](../KEY_FILES.md). Read only the entries relevant to your change.
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
-- `src/core/ops/wakeshift.ts` + `src/core/wakeshift/{client,config,events,drain,terms,forget}.ts` + `src/core/wakeshift-env-key.ts` + `src/commands/wakeshift.ts` + `src/commands/doctor/checks/wakeshift.ts` — report back to agent conversations through Wakeshift (docs/guides/wakeshift.md), off by default. `client.ts` is the in-repo HTTP client (no SDK): key parsing (`wst_` + 50 base58 with a CRC32; legacy formats refused), a fresh EdDSA `app+jwt` assertion per request, one `fetch` with a 10 s timeout and no retry, loopback-only http; endpoints subscribe, unsubscribe, end, emit, manifest merge and `/users/forget`. `config.ts` holds the host-only state: `wakeshift.enabled`/`url`/`terms_url`/`privacy_url`/`manifest_hash` in the file plane, the key (WAKESHIFT_API_KEY, which `wakeshift-env-key.ts` moves out of the environment on every config load so children never inherit it, the first value seen winning; else a 0600 `~/.gbrain/wakeshift.key`), the 0600 HMAC secret and the opaque Wakeshift user (HMAC over brain and principal key: `member:<id>` when bound, else `<kind>:<id>`); `wakeshiftRuntime().file` carries the file plane to the ops and the drainer, so core never reads config twice; `wakeshiftPublished` is publish gate `mcp.publish_wakeshift`, which `mcp/publish-gates.ts` resolves from the file plane and key presence only (never the DB plane; not in `PUBLISH_GATE_KEYS`). Three events (`events.ts`): `gbrain.job.finished` (`watch_job`, the get_job owner fence from `ops/jobs.ts`), `gbrain.write.settled` (`watch_write_request`, write scope, the receipt read path `receiptAccess`/`receiptVisible`/`receiptNotFound` exported from `ops/persistence.ts`, so a foreign id reads exactly like get_write_request's not_found; `target_ref` holds the `persistence_requests.id`), `gbrain.cron.finished` (`watch_cron`, admin scope, names in `wake_crons`; `gbrain wakeshift emit cron.finished` only inserts a `wake_cron_runs` row and exits 0). Every watch goes through `openWatch` (subscribe `once`... (line truncated to 2000 chars)
+- `src/core/ops/wakeshift.ts` + `src/core/wakeshift/{client,config,events,drain,terms,forget,spool}.ts` + `src/core/wakeshift-env-key.ts` + `src/commands/wakeshift.ts` + `src/commands/doctor/checks/wakeshift.ts` — report back to agent conversations through Wakeshift (docs/guides/wakeshift.md), off by default. `client.ts` is the in-repo HTTP client (no SDK): key parsing (`wst_` + 50 base58 with a CRC32; legacy formats refused), a fresh EdDSA `app+jwt` assertion per request, one `fetch` with a 10 s timeout and no retry, loopback-only http; endpoints subscribe, unsubscribe, end, emit, manifest merge and `/users/forget`. `config.ts` holds the host-only state: `wakeshift.enabled`/`url`/`terms_url`/`privacy_url`/`manifest_hash` in the file plane, the key (WAKESHIFT_API_KEY, which `wakeshift-env-key.ts` moves out of the environment on every config load so children never inherit it, the first value seen winning; else a 0600 `~/.gbrain/wakeshift.key`), the 0600 HMAC secret and the opaque Wakeshift user (HMAC over brain and principal key: `member:<id>` when bound, else `<kind>:<id>`); `wakeshiftRuntime().file` carries the file plane to the ops and the drainer, so core never reads config twice; `wakeshiftPublished` is publish gate `mcp.publish_wakeshift`, which `mcp/publish-gates.ts` resolves from the file plane and key presence only (never the DB plane; not in `PUBLISH_GATE_KEYS`). Three events (`events.ts`): `gbrain.job.finished` (`watch_job`, the get_job owner fence from `ops/jobs.ts`), `gbrain.write.settled` (`watch_write_request`, write scope, the receipt read path `receiptAccess`/`receiptVisible`/`receiptNotFound` exported from `ops/persistence.ts`, so a foreign id reads exactly like get_write_request's not_found; `target_ref` holds the `persistence_requests.id`), `gbrain.cron.finished` (`watch_cron`, admin scope, names in `wake_crons`; `gbrain wakeshift emit cron.finished` only inserts a `wake_cron_runs` row and exits 0; on a PGLite brain held by serve or the worker it writes the run to `spool.ts`'s per-brain spool, which the drainer ingests). Every watch goes through `openWatch` (subscribe `once`... (line truncated to 2000 chars)
- `src/commands/wakeshift-watch.ts` + `src/commands/setup-openclaw-wakeshift.ts` + `src/core/report-back-notice.ts` + `src/commands/doctor/checks/wakeshift-openclaw.ts` — the agent-facing edges of reporting back. `wakeshift-watch.ts` is `gbrain wakeshift watch job|write|cron` (the only `wakeshift` subcommand a thin client routes, through `callRemoteTool`; locally it calls the op with `remote: false`, whose refusal names `gbrain jobs wait`); Wakeshift's addition is always the last stdout line and a `--session-key` is passed through, never stored. `setup-openclaw-wakeshift.ts` is `gbrain setup openclaw --wakeshift`: `requireConsent` (persistent_install, egress), then OpenClaw's own installer for the one pinned `WAKESHIFT_OPENCLAW_VERSION`, an injectable runner, Wakeshift's allowNetworkHooks question printed on a non-loopback gateway and never answered; `wakeshift-openclaw.ts` warns when the installed plugin differs from the pin. `report-back-notice.ts` is the muteable `wakeshift_report_back` coaching notice after a remote `submit_job`/`submit_agent` (the notice ledger's once-per-session dedupe); `mcp/instructions.ts` adds one sentence only while `watch_job` is callable. Tests: `test/wakeshift-watch.test.ts`, `test/setup-openclaw-wakeshift.test.ts`, `test/wakeshift-nudge.test.ts`.
diff --git a/docs/guides/wakeshift.md b/docs/guides/wakeshift.md
index f4386a77d8..e28b985709 100644
--- a/docs/guides/wakeshift.md
+++ b/docs/guides/wakeshift.md
@@ -48,7 +48,7 @@ status=ok
gbrain-run wakeshift emit cron.finished --name nightly-digest --status "$status"
```
-`emit` only records the run (name, status, time) in the brain; the drainer wakes the watchers on its next tick. It never calls Wakeshift itself, never waits, and exits 0 when reporting back is off (with one line on stderr) or when the name isn't registered (it records the run and says so; `gbrain doctor` lists unregistered names it saw). It also exits 0 when it can't open the brain, and then the run is not recorded: on a PGLite brain that happens whenever `gbrain serve` or a job worker holds the brain, so on PGLite a watch on a cron only wakes for runs that finish while neither is running. A Postgres brain has no such lock. `gbrain wakeshift cron list` shows the names with their last run, and `cron remove <name>` removes one and ends its open watches. Runs older than 30 days are dropped. No report path or slug goes into the wake: the agent reads the output where the cron writes it.
+`emit` only records the run (name, status, time) in the brain; the drainer wakes the watchers on its next tick. It never calls Wakeshift itself, never waits, and exits 0 when reporting back is off (with one line on stderr) or when the name isn't registered (it records the run and says so; `gbrain doctor` lists unregistered names it saw). It also exits 0 when it can't open the brain. On a PGLite brain that happens whenever `gbrain serve` or a job worker holds the brain, so `emit` writes the run to a small file under `~/.gbrain/wakeshift-spool/` instead, and the holder records it on its next 5-second tick with the time the cron actually finished; the next `emit` that opens the brain records any left over. At most 1,000 runs wait there per brain, and a file that isn't a valid run is moved to `rejected/` in that folder and named on stderr. Nothing is spooled while reporting back is off. A Postgres brain has no such lock: when it can't be reached the run is not recorded. `gbrain wakeshift cron list` shows the names with their last run, and `cron remove <name>` removes one and ends its open watches. Runs older than 30 days are dropped. No report path or slug goes into the wake: the agent reads the output where the cron writes it.
## The key
diff --git a/scripts/core-world-reads-baseline.tsv b/scripts/core-world-reads-baseline.tsv
index 4e8acf0964..72e58ff27a 100644
--- a/scripts/core-world-reads-baseline.tsv
+++ b/scripts/core-world-reads-baseline.tsv
@@ -1193,6 +1193,7 @@ src/core/verbs/usage-log.ts fs_import 1 W7: filesystem boundary moves to the loc
src/core/wakeshift/config.ts fs_import 1 stays: the Wakeshift key and HMAC secret are host files (0600) by design, never engine settings (docs/guides/wakeshift.md#the-key)
src/core/wakeshift/config.ts load_config 1 stays: wakeshift.enabled is read from the host file plane only, so no DB-plane or remote writer can turn reporting back on
src/core/wakeshift/config.ts process_env 2 stays: an explicit env argument is compared with process.env to decide whether the key moves into env-key.ts
+src/core/wakeshift/spool.ts fs_import 1 stays: the cron-run spool is a 0600 host file under the brain's own gbrain home that emit writes while the PGLite brain is held, read by the drainer (docs/guides/wakeshift.md)
src/core/wakeshift-env-key.ts process_env 2 stays: WAKESHIFT_API_KEY is taken out of the process environment whenever config loads, so child processes do not inherit the signing key
src/core/workspace-push.ts fs_import 1 W7: filesystem boundary moves to the local app
src/core/workspace-push.ts process_env 2 W6: settings plane out of the core
diff --git a/scripts/serial-files.tsv b/scripts/serial-files.tsv
index 4cacaa7a16..d05de457fa 100644
--- a/scripts/serial-files.tsv
+++ b/scripts/serial-files.tsv
@@ -473,4 +473,5 @@ test/synopsis-sweep-8744.serial.test.ts R1 R1: mutates process.env (GBRAIN_HOME,
test/release-channel-w1.serial.test.ts R1 R1: mutates process.env (GBRAIN_HOME, GBRAIN_RELEASE_CHANNEL) and stubs globalThis.fetch
test/hermes-snapshot-7067.serial.test.ts R2 R2: mock.module(node:fs) wraps copyFileSync to inject a writer and WAL checkpoint mid-snapshot
test/wakeshift-pr2.serial.test.ts global-state global-state: R3: constructs a second PGLiteEngine inside a test (the git-backed brain for write.settled); binds a fake Wakeshift HTTP listener
+test/wakeshift-emit-spool.serial.test.ts global-state global-state: resets the module-level Wakeshift env-key cache (resetWakeshiftEnvKeyForTests) and starts the process-wide persistence consumer; binds a fake Wakeshift HTTP listener
test/push-secret-scan-gaps.serial.test.ts global-state global-state: sets GBRAIN_HOME, PATH (a git wrapper for the index race) and GBRAIN_GIT_ALLOW_FILE_TRANSPORT through withEnv for real commits and pushes to local bare remotes
diff --git a/src/cli/main.ts b/src/cli/main.ts
index 011871e58c..e1ccf891d0 100755
--- a/src/cli/main.ts
+++ b/src/cli/main.ts
@@ -2523,7 +2523,7 @@ async function connectCliOnlyEngine(command: string, args: string[]): Promise<Br
if (command === 'recall') oneShotRead = !args.includes('--watch');
engine = await connectEngine({ probeOnly: findCliCommand(command)?.startup === 'observational' || (command === 'jobs' && args[0] === 'supervisor' && args[1] === 'status'), oneShotRead });
} catch (serveConnectError) {
- if (command === 'wakeshift' && args[0] === 'emit') return (await import('../commands/wakeshift.ts')).reportEmitConnectFailure(args.slice(1), serveConnectError), null;
+ if (command === 'wakeshift' && args[0] === 'emit') return await (await import('../commands/wakeshift.ts')).reportEmitConnectFailure(args.slice(1), serveConnectError, dbMarkerBrainId());
if (command === 'jobs' && args[0] === 'child-readiness') {
const { writeChildReadinessFailure } = await import('../core/minions/child-readiness.ts');
process.exit(writeChildReadinessFailure(serveConnectError));
diff --git a/src/commands/wakeshift.ts b/src/commands/wakeshift.ts
index dc86e50460..6267ee6ba8 100644
--- a/src/commands/wakeshift.ts
+++ b/src/commands/wakeshift.ts
@@ -14,7 +14,9 @@
*/
import { readFileSync, renameSync, writeFileSync } from 'node:fs';
import type { BrainEngine } from '../core/engine.ts';
-import { loadConfigFileOnly, saveConfig, type GBrainConfig } from '../core/config.ts';
+import { loadConfig, loadConfigFileOnly, saveConfig, type GBrainConfig } from '../core/config.ts';
+import { managedFilesystemDatastorePath } from '../core/persistence/filesystem-guard.ts';
+import { SPOOL_MAX_FILES, ingestCronSpool, recordCronRun, spoolCronRun } from '../core/wakeshift/spool.ts';
import { opError } from '../core/ops/contract.ts';
import { requireConsent } from '../core/consent.ts';
import { WakeshiftError, WakeshiftKeyError, normalizeWakeshiftUrl, parseWakeshiftKey, wakeshiftClient, type WakeshiftKey } from '../core/wakeshift/client.ts';
@@ -284,14 +286,42 @@ function emitArgs(args: string[]): { name: string; status: 'ok' | 'failed' } | n
return null;
}
+/** The datastore path of the PGLite brain this CLI resolved (`brainId` names a mount), or null for Postgres. */
+async function pgliteDatastore(brainId: string | undefined): Promise<string | null> {
+ if (brainId && brainId !== 'host') {
+ const mount = (await import('../core/brain-registry.ts')).loadMounts().find(m => m.id === brainId);
+ return mount?.engine === 'pglite' ? mount.database_path ?? null : null;
+ }
+ const config = loadConfig();
+ return config?.engine === 'pglite' ? config.database_path ?? null : null;
+}
+
/**
- * The brain could not be opened for `emit` (a live serve holding the PGLite lock, a database outage): the run is not
- * recorded, but the cron script still keeps its own exit status, as the emit contract promises.
+ * The brain could not be opened for `emit`. On a PGLite brain held by `gbrain serve` or `gbrain jobs work`, the run
+ * goes to the brain's spool (spool.ts) for that process's wake drainer to record; otherwise (Postgres, reporting back
+ * off, a full spool) it is not recorded. Either way one stderr line, and the cron script keeps its own exit status.
*/
-export function reportEmitConnectFailure(args: string[], error: unknown): void {
+export async function reportEmitConnectFailure(args: string[], error: unknown, brainId?: string): Promise<null> {
const parsed = emitArgs(args);
- if (!parsed) return;
- console.error(`[wakeshift] could not record the ${parsed.name} run: the brain could not be opened (${error instanceof Error ? error.message.split('\n')[0] : String(error)}).`);
+ if (!parsed) return null;
+ const reason = error instanceof Error ? error.message.split('\n')[0] : String(error);
+ try {
+ const datastore = await pgliteDatastore(brainId);
+ if (datastore && wakeshiftPublished()) {
+ const spooled = spoolCronRun(datastore, parsed);
+ if (spooled !== 'full') {
+ console.error(`[wakeshift] the brain could not be opened (${reason}), so the ${parsed.name} run was spooled: the wake drainer of gbrain serve or gbrain jobs work records it within seconds.`);
+ return null;
+ }
+ console.error(`[wakeshift] could not record the ${parsed.name} run: ${SPOOL_MAX_FILES} runs already wait in the spool, which only gbrain serve or gbrain jobs work drains.`);
+ return null;
+ }
+ } catch (e) {
+ console.error(`[wakeshift] could not spool the ${parsed.name} run: ${e instanceof Error ? e.message : String(e)}`);
+ return null;
+ }
+ console.error(`[wakeshift] could not record the ${parsed.name} run: the brain could not be opened (${reason}).`);
+ return null;
}
async function emit(engine: BrainEngine, args: string[]): Promise<void> {
@@ -303,10 +333,9 @@ async function emit(engine: BrainEngine, args: string[]): Promise<void> {
return;
}
try {
- await engine.executeRaw('INSERT INTO wake_cron_runs (name, status) VALUES ($1, $2)', [name, status]);
- await engine.executeRaw("DELETE FROM wake_cron_runs WHERE name = $1 AND finished_at < now() - interval '30 days'", [name]);
- const registered = await engine.executeRaw('SELECT 1 FROM wake_crons WHERE name = $1', [name]);
- if (registered.length === 0) console.error(`[wakeshift] cron ${name} is not registered, so no agent can watch it: run gbrain wakeshift cron add ${name}.`);
+ const datastore = managedFilesystemDatastorePath(engine);
+ if (datastore) await ingestCronSpool(engine, datastore).catch(e => console.error(`[wakeshift] could not take the spooled cron runs: ${e instanceof Error ? e.message : String(e)}`));
+ await recordCronRun(engine, { name, status });
} catch (e) {
console.error(`[wakeshift] could not record the ${name} run: ${e instanceof Error ? e.message : String(e)}`);
}
diff --git a/src/core/cli-flag-registry.generated.ts b/src/core/cli-flag-registry.generated.ts
index f8a259a4f2..383d3fa8d0 100644
--- a/src/core/cli-flag-registry.generated.ts
+++ b/src/core/cli-flag-registry.generated.ts
@@ -143,7 +143,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
'trust': ['--accept-all', '--accept-pending', '--all', '--as', '--background', '--batch-size', '--brain', '--break-lock', '--db-url', '--dry-run', '--expect', '--explain', '--follow', '--from', '--help', '--http', '--json', '--kind', '--label-lowerings', '--legacy-lock', '--no-accept-pending', '--only', '--path', '--probe', '--progress-interval', '--progress-json', '--quiet', '--reason', '--reason-family', '--relower', '--remove', '--repair-unknown-markers', '--resume', '--room', '--since', '--slug', '--source', '--source-id', '--timeout', '--timings', '--undo', '--uri-prefix', '--version', '--wait', '--yes'],
'upgrade': ['--accept-reviewed-inventory', '--acknowledge-no-backup', '--admin-intent', '--all', '--all-applied', '--apply', '--apply-clean-hunks', '--backend', '--backup-confirmed', '--brain', '--catch-up', '--changed', '--channel', '--check', '--check-downgrade', '--check-only', '--client', '--concurrency', '--confirm-quiesced', '--confirm-relocated-root', '--content-root', '--cost-estimate', '--dry-run', '--embedding-dimensions', '--embedding-model', '--exit-code', '--expand', '--expect', '--expected-epoch', '--expected-state', '--explain', '--export-db-only', '--export-source', '--ff-only', '--force', '--force-all', '--force-orchestrator', '--force-retry', '--force-schema', '--format', '--held', '--help', '--host-dir', '--id', '--images', '--inject-bootstrap', '--inline', '--install', '--instructions-only', '--interval', '--is-ancestor', '--json', '--limit', '--list', '--manifest', '--markdown', '--max-rss', '--max-usd', '--may-act-for', '--member', '--migrate-only', '--migration', '--mode', '--multimodal', '--name', '--name-only', '--no-autopilot-install', '--no-bun-floor-check', '--no-embed', '--no-expand', '--no-inject', '--no-restart', '--no-worker', '--non-interactive', '--now', '--once', '--only', '--oversized', '--path', '--pause', '--phase', '--prefix', '--quiet', '--readers', '--reads-unscoped', '--reason', '--refs', '--remediation-plan', '--repo', '--require-db', '--restart-only', '--resume', '--retry-global', '--scope', '--search-mode', '--segmented', '--self-transfer', '--skills-dir', '--skip-verify', '--slug', '--slugs-file', '--source', '--stale', '--statement-timeout', '--status', '--stop-mcp', '--swap-only', '--synopsis', '--tags', '--target', '--token', '--uninstall', '--user', '--vectors', '--verify', '--version', '--workers', '--yes'],
'waiting': ['--all', '--as-of', '--brain', '--dry-run', '--fail-fast', '--help', '--json', '--note', '--source', '--stale-ok', '--status', '--tools-json', '--top', '--type'],
- 'wakeshift': ['--apply', '--brain', '--conversation-id', '--expect', '--harness', '--help', '--json', '--key-stdin', '--max-cost', '--max-usd', '--name', '--non-interactive', '--privacy-url', '--probe', '--session-key', '--source', '--status', '--terms-url', '--trust', '--url', '--yes'],
+ 'wakeshift': ['--apply', '--brain', '--conversation-id', '--db-url', '--expect', '--harness', '--help', '--json', '--key-stdin', '--max-cost', '--max-usd', '--name', '--non-interactive', '--path', '--privacy-url', '--probe', '--session-key', '--source', '--status', '--terms-url', '--trust', '--url', '--yes'],
'watch': ['--brain', '--help', '--json', '--max-pages', '--min-confidence', '--source', '--window-turns'],
'whoknows': ['--access', '--add', '--all', '--allowed-operations', '--as', '--asof', '--body', '--bound-slug-prefixes', '--brain', '--card', '--client', '--cursor', '--date', '--dated-since', '--days', '--entities', '--expected-revision', '--explain', '--federated-read', '--follow', '--from-index', '--full', '--fuzzy', '--group-by', '--help', '--http', '--id', '--include', '--include-deleted', '--json', '--kind', '--limit', '--max-usd', '--mcp-only', '--model', '--multimodal', '--no-autopilot-install', '--of', '--only', '--page', '--params', '--patch', '--path', '--phase', '--prefer', '--privacy-url', '--purge', '--recent', '--refresh-operations', '--request-id', '--row', '--scopes', '--session', '--since', '--slugs', '--source', '--sources', '--stale', '--surface', '--terms-url', '--to', '--updated-after', '--url', '--yes'],
};
diff --git a/src/core/wakeshift/drain.ts b/src/core/wakeshift/drain.ts
index b6ff076c45..5e9e7d63d4 100644
--- a/src/core/wakeshift/drain.ts
+++ b/src/core/wakeshift/drain.ts
@@ -31,6 +31,8 @@ import { WakeshiftError, type WakeshiftApi } from './client.ts';
import { wakeshiftRuntime, type WakeshiftRuntime } from './config.ts';
import { CRON_FINISHED, JOB_FINISHED, WRITE_SETTLED, cronMarker, cronWake, jobMarker, jobWake, wakeEmitId, writeMarker, writeWake, type JobSnapshot, type WriteSnapshot } from './events.ts';
import { operatorTerms, termsAccepted, type OperatorTerms } from './terms.ts';
+import { ingestCronSpool } from './spool.ts';
+import { managedFilesystemDatastorePath } from '../persistence/filesystem-guard.ts';
export const WAKE_RETRY_SCHEDULE_S = [30, 120, 480, 1800] as const;
export const WAKE_MAX_ATTEMPTS = 5;
@@ -184,10 +186,16 @@ function wakeFor(row: WakeRow): { marker: string; args: Record<string, unknown>;
return { marker: jobMarker(job), args: { job: Number(row.target) }, ...jobWake(Number(row.target), job) };
}
-/** Queue a Wakeshift forget for a revoked credential's own user (a member's user is forgotten when the member is removed, forget.ts). */
+/**
+ * Queue a Wakeshift forget for a revoked credential's own user, or for a removed member's user: removal queues it
+ * (forget.ts), and this catches a removal whose queueing failed. A revoked credential of a member who is still live
+ * queues nothing, since the member's user outlives that credential.
+ */
async function queueForget(engine: BrainEngine, row: WakeRow): Promise<void> {
- if (row.member_id || row.principal_kind === 'local_stdio') return;
- await engine.executeRaw(`INSERT INTO wake_forgets (wake_user) VALUES ($1) ON CONFLICT (wake_user) WHERE state = 'pending' DO NOTHING`, [row.wake_user]);
+ if (!row.member_id && row.principal_kind === 'local_stdio') return;
+ await engine.executeRaw(`INSERT INTO wake_forgets (wake_user) SELECT $1
+ WHERE $2::text IS NULL OR NOT EXISTS (SELECT 1 FROM access_members WHERE id = $2 AND tombstoned_at IS NULL)
+ ON CONFLICT (wake_user) WHERE state = 'pending' DO NOTHING`, [row.wake_user, row.member_id]);
}
async function deliver(engine: BrainEngine, api: WakeshiftApi, row: WakeRow, report: DrainReport, terms: { shared: boolean; current: OperatorTerms | null }): Promise<void> {
@@ -268,7 +276,8 @@ async function drainForgets(engine: BrainEngine, api: WakeshiftApi, report: Drai
}
/**
- * One reconciliation pass: ends expired watches, sends queued forgets, then delivers up to 20 due watches (only `ids`
+ * One reconciliation pass: ends expired watches, records cron runs spooled while the PGLite brain was held
+ * (spool.ts), sends queued forgets, then delivers up to 20 due watches (only `ids`
* when given). Returns `skipped: 'off'` without a query when reporting back is off or keyless. `runtime` is the
* caller's (the watch tools) or a test seam.
*/
@@ -282,9 +291,11 @@ export async function drainWakes(engine: BrainEngine, opts: { ids?: number[]; ru
`UPDATE wake_subscriptions SET state = 'ended', last_error = 'expired', updated_at = now()
WHERE state = 'open' AND expires_at < now() RETURNING id, wake_user, ws_subscription_id`);
report.ended += expired.length;
+ let failure: unknown = null;
+ const datastore = opts.ids === undefined ? managedFilesystemDatastorePath(engine) : undefined;
+ if (datastore) await ingestCronSpool(engine, datastore).catch(e => { failure = e; });
if (opts.ids === undefined) await drainForgets(engine, api, report);
const rows = await engine.executeRaw<WakeRow>(CANDIDATES_SQL(opts.ids !== undefined), opts.ids !== undefined ? [opts.ids] : []);
- let failure: unknown = null;
for (const row of rows) {
try { await deliver(engine, api, row, report, terms); } catch (e) { failure ??= e; }
}
diff --git a/src/core/wakeshift/forget.ts b/src/core/wakeshift/forget.ts
index 95fc175e06..f6a98ea662 100644
--- a/src/core/wakeshift/forget.ts
+++ b/src/core/wakeshift/forget.ts
@@ -27,12 +27,13 @@ export async function forgetWakePrincipal(sql: SqlQuery, who: ForgetTarget): Pro
await sql`DELETE FROM wake_terms_acceptances WHERE principal_key = ${`member:${who.member}`}`;
return { ended: ended.length, queued: queued.length };
}
+ const id = who.kind === 'legacy_token' ? who.id.toLowerCase() : who.id;
const queued = await sql`INSERT INTO wake_forgets (wake_user) SELECT DISTINCT wake_user FROM wake_subscriptions
- WHERE principal_kind = ${who.kind} AND principal_id = ${who.id} AND member_id IS NULL
+ WHERE principal_kind = ${who.kind} AND principal_id = ${id} AND member_id IS NULL
ON CONFLICT (wake_user) WHERE state = 'pending' DO NOTHING RETURNING id`;
const ended = await sql`UPDATE wake_subscriptions SET state = 'ended', last_error = 'principal_revoked', lease_until = NULL, updated_at = now()
- WHERE principal_kind = ${who.kind} AND principal_id = ${who.id} AND state = 'open' RETURNING id`;
- await sql`DELETE FROM wake_terms_acceptances WHERE principal_key = ${`${who.kind}:${who.id}`}`;
+ WHERE principal_kind = ${who.kind} AND principal_id = ${id} AND state = 'open' RETURNING id`;
+ await sql`DELETE FROM wake_terms_acceptances WHERE principal_key = ${`${who.kind}:${id}`}`;
return { ended: ended.length, queued: queued.length };
} catch (e) {
console.error(`[wakeshift] could not end the removed principal's watches or queue its forget: ${e instanceof Error ? e.message : String(e)}`);
diff --git a/src/core/wakeshift/spool.ts b/src/core/wakeshift/spool.ts
new file mode 100644
index 0000000000..ed9c35748a
--- /dev/null
+++ b/src/core/wakeshift/spool.ts
@@ -0,0 +1,112 @@
+/**
+ * The cron-run spool (GST-2679). On a PGLite brain, `gbrain wakeshift emit cron.finished` at the end of a cron script
+ * cannot open the brain while `gbrain serve` or `gbrain jobs work` holds its single-writer lock. Instead of dropping
+ * the run, emit writes it as one small 0600 file under the brain's own gbrain home
+ * (~/.gbrain/wakeshift-spool/<hash of the datastore path>/), and the 5 s wake drainer that serve and the worker both
+ * run ingests it: it claims each file by an atomic rename, so two drainers never take the same file, inserts the run
+ * with its original finished time and deletes the file. The insert skips a run already recorded with the same name,
+ * status and time, so a claim left behind by a crash between the insert and the delete is replayed harmlessly.
+ *
+ * Bounded: at most SPOOL_MAX_FILES runs wait per brain (emit says so and drops the run past that), and a file over
+ * SPOOL_MAX_BYTES or one that does not parse (a finished time before 2000 or over a day ahead included) is moved to rejected/ with one stderr line, never retried. Nothing is
+ * spooled while reporting back is off, and Postgres brains never spool: their emit connects or reports the outage.
+ */
+import { createHash, randomBytes } from 'node:crypto';
+import { mkdirSync, readdirSync, readFileSync, realpathSync, renameSync, rmSync, statSync, utimesSync, writeFileSync } from 'node:fs';
+import { join, resolve } from 'node:path';
+import type { BrainEngine } from '../engine.ts';
+import { gbrainPath } from '../config.ts';
+import { cronName } from './events.ts';
+
+export const SPOOL_MAX_FILES = 1000;
+export const SPOOL_MAX_BYTES = 1024;
+export const SPOOL_INGEST_BATCH = 200;
+const STALE_CLAIM_MS = 10 * 60_000;
+const EARLIEST_RUN_MS = Date.UTC(2000, 0, 1);
+const LATEST_RUN_SKEW_MS = 24 * 3600_000;
+const PENDING = /^\d{13}-[0-9a-f]{16}\.json$/;
+const CLAIMED = /^\d{13}-[0-9a-f]{16}\.json\.claimed-[0-9a-f]{16}$/;
+
+export interface CronRun { name: string; status: 'ok' | 'failed'; finished_at: string }
+
+/**
+ * The spool directory of the PGLite brain at `databasePath`, under this gbrain home, keyed by the datastore's
+ * canonical path as the PGLite lock is, so two spellings of one datastore (a symlink) share one spool. A path that
+ * cannot be resolved keys by its absolute spelling.
+ */
+export function cronSpoolDir(databasePath: string): string {
+ let path = resolve(databasePath);
+ try { path = realpathSync(path); } catch { /* not resolvable: the absolute spelling */ }
+ return gbrainPath('wakeshift-spool', createHash('sha256').update(path).digest('hex').slice(0, 16));
+}
+
+/** Record one run: the direct emit (no time, so now()) or a spooled one (its own time, skipped when already recorded). */
+export async function recordCronRun(engine: BrainEngine, run: { name: string; status: string; finished_at?: string }): Promise<void> {
+ await engine.executeRaw(
+ `INSERT INTO wake_cron_runs (name, status, finished_at) SELECT $1, $2, COALESCE($3::timestamptz, now())
+ WHERE $3::timestamptz IS NULL OR NOT EXISTS (SELECT 1 FROM wake_cron_runs WHERE name = $1 AND status = $2 AND finished_at = $3::timestamptz)`,
+ [run.name, run.status, run.finished_at ?? null]);
+ await engine.executeRaw("DELETE FROM wake_cron_runs WHERE name = $1 AND finished_at < now() - interval '30 days'", [run.name]);
+ const registered = await engine.executeRaw('SELECT 1 FROM wake_crons WHERE name = $1', [run.name]);
+ if (registered.length === 0) console.error(`[wakeshift] cron ${run.name} is not registered, so no agent can watch it: run gbrain wakeshift cron add ${run.name}.`);
+}
+
+/** Write one run to the spool; 'full' when SPOOL_MAX_FILES runs already wait. */
+export function spoolCronRun(databasePath: string, run: { name: string; status: 'ok' | 'failed' }, now = new Date()): { file: string } | 'full' {
+ const dir = cronSpoolDir(databasePath);
+ mkdirSync(dir, { recursive: true, mode: 0o700 });
+ if (readdirSync(dir).filter(f => PENDING.test(f) || CLAIMED.test(f)).length >= SPOOL_MAX_FILES) return 'full';
+ const base = `${String(now.getTime()).padStart(13, '0')}-${randomBytes(8).toString('hex')}.json`;
+ const tmp = join(dir, `.${base}.tmp`);
+ writeFileSync(tmp, JSON.stringify({ name: run.name, status: run.status, finished_at: now.toISOString() } satisfies CronRun), { mode: 0o600, flag: 'wx' });
+ renameSync(tmp, join(dir, base));
+ return { file: join(dir, base) };
+}
+
+function parseRun(path: string): CronRun | string {
+ const size = statSync(path).size;
+ if (size > SPOOL_MAX_BYTES) return `it is ${size} bytes (at most ${SPOOL_MAX_BYTES})`;
+ let raw: unknown;
+ try { raw = JSON.parse(readFileSync(path, 'utf8')); } catch { return 'it is not JSON'; }
+ const r = raw as Partial<CronRun> | null;
+ const time = typeof r?.finished_at === 'string' ? Date.parse(r.finished_at) : NaN;
+ if (!r || typeof r.name !== 'string' || cronName(r.name) !== r.name || (r.status !== 'ok' && r.status !== 'failed') || !(time >= EARLIEST_RUN_MS && time <= Date.now() + LATEST_RUN_SKEW_MS)) {
+ return 'it is not a cron run (name, status ok|failed, finished_at)';
+ }
+ return { name: r.name, status: r.status, finished_at: new Date(time).toISOString() };
+}
+
+/**
+ * Ingest up to SPOOL_INGEST_BATCH spooled runs of the brain at `databasePath`, oldest first, plus claims older than
+ * 10 minutes (a drainer that died between claim and delete). A failed insert puts the file back for the next pass.
+ */
+export async function ingestCronSpool(engine: BrainEngine, databasePath: string): Promise<{ recorded: number; rejected: number }> {
+ const dir = cronSpoolDir(databasePath);
+ const out = { recorded: 0, rejected: 0 };
+ let names: string[];
+ try { names = readdirSync(dir); } catch { return out; }
+ const stale = (f: string) => { try { return Date.now() - statSync(join(dir, f)).mtimeMs > STALE_CLAIM_MS; } catch { return false; } };
+ const due = names.filter(f => PENDING.test(f) || (CLAIMED.test(f) && stale(f))).sort().slice(0, SPOOL_INGEST_BATCH);
+ for (const f of due) {
+ const pending = f.replace(/\.claimed-[0-9a-f]{16}$/, '');
+ const claim = join(dir, `${pending}.claimed-${randomBytes(8).toString('hex')}`);
+ try { renameSync(join(dir, f), claim); utimesSync(claim, new Date(), new Date()); } catch { continue; }
+ const run = parseRun(claim);
+ if (typeof run === 'string') {
+ mkdirSync(join(dir, 'rejected'), { recursive: true, mode: 0o700 });
+ renameSync(claim, join(dir, 'rejected', pending));
+ console.error(`[wakeshift] moved spooled cron run ${pending} to ${join(dir, 'rejected')}: ${run}.`);
+ out.rejected++;
+ continue;
+ }
+ try {
+ await recordCronRun(engine, run);
+ } catch (e) {
+ renameSync(claim, join(dir, pending));
+ throw e;
+ }
+ rmSync(claim, { force: true });
+ out.recorded++;
+ }
+ return out;
+}
diff --git a/test/wakeshift-emit-spool.serial.test.ts b/test/wakeshift-emit-spool.serial.test.ts
new file mode 100644
index 0000000000..6470659f6f
--- /dev/null
+++ b/test/wakeshift-emit-spool.serial.test.ts
@@ -0,0 +1,219 @@
+/**
+ * GST-2679: `gbrain wakeshift emit cron.finished` on a PGLite brain held by another gbrain process (serve, the job
+ * worker) spools the run under the brain's own gbrain home, and the holder's wake drainer records it with its original
+ * finished time. Repo-less PGLite brain on disk, fake Wakeshift (test/helpers/fake-wakeshift.ts).
+ *
+ * Serial: every test runs under withEnv with GBRAIN_HOME at a temporary home holding config.json and the 0600 key file.
+ */
+import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
+import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync, symlinkSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { dispatchToolCall } from '../src/mcp/dispatch.ts';
+import { encodeWakeshiftKey } from '../src/core/wakeshift/client.ts';
+import { drainWakes } from '../src/core/wakeshift/drain.ts';
+import { resetWakeshiftEnvKeyForTests } from '../src/core/wakeshift/config.ts';
+import { wakeshiftManifestHash } from '../src/core/wakeshift/events.ts';
+import { SPOOL_MAX_BYTES, SPOOL_MAX_FILES, cronSpoolDir, ingestCronSpool, spoolCronRun } from '../src/core/wakeshift/spool.ts';
+import { registerLocalWriter } from '../src/core/persistence/identity.ts';
+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+import { reportEmitConnectFailure, runWakeshift } from '../src/commands/wakeshift.ts';
+import { startFakeWakeshift, type FakeWakeshift } from './helpers/fake-wakeshift.ts';
+import { withEnv } from './helpers/with-env.ts';
+
+const REPO = fileURLToPath(new URL('..', import.meta.url)).replace(/\/$/, '');
+const KEY = encodeWakeshiftKey(Buffer.alloc(32, 22));
+let engine: PGLiteEngine;
+let fake: FakeWakeshift;
+let home: string;
+let datastore: string;
+
+const inHome = <T>(fn: () => Promise<T> | T) => withEnv({ GBRAIN_HOME: home, WAKESHIFT_API_KEY: undefined }, async () => {
+ resetWakeshiftEnvKeyForTests();
+ return fn();
+});
+
+function writeConfig(config: Record<string, unknown>): void {
+ writeFileSync(join(home, '.gbrain', 'config.json'), JSON.stringify(config), { mode: 0o600 });
+}
+const pgliteOn = () => writeConfig({ engine: 'pglite', database_path: datastore, wakeshift: { enabled: true, url: fake.url, manifest_hash: wakeshiftManifestHash() } });
+const spoolDir = () => inHome(() => cronSpoolDir(datastore));
+const pending = async () => { const dir = await spoolDir(); return existsSync(dir) ? readdirSync(dir).filter(f => f.endsWith('.json') || f.includes('.claimed-')) : []; };
+const runs = () => engine.executeRaw<{ name: string; status: string; finished_at: Date }>('SELECT name, status, finished_at FROM wake_cron_runs ORDER BY id');
+const emits = () => fake.calls.filter(c => c.path === '/events/emit');
+const log = { info() {}, warn() {}, error() {} };
+const stdio = { remote: true, transport: 'stdio' as const, sourceId: 'default', config: { engine: 'pglite' }, logger: log, mcpCaller: { client: 'claude-code' } };
+
+async function captured<T>(fn: () => Promise<T>): Promise<{ value: T; out: string; err: string }> {
+ const [log0, err0] = [console.log, console.error];
+ let out = '', err = '';
+ console.log = (...a: unknown[]) => { out += a.map(String).join(' ') + '\n'; };
+ console.error = (...a: unknown[]) => { err += a.map(String).join(' ') + '\n'; };
+ try { return { value: await inHome(fn), out, err }; } finally { console.log = log0; console.error = err0; }
+}
+
+beforeAll(async () => {
+ home = mkdtempSync(join(tmpdir(), 'gbrain-wakeshift-spool-'));
+ mkdirSync(join(home, '.gbrain'), { recursive: true });
+ writeFileSync(join(home, '.gbrain', 'wakeshift.key'), KEY + '\n', { mode: 0o600 });
+ datastore = join(home, '.gbrain', 'brain.pglite');
+ fake = startFakeWakeshift(KEY);
+ engine = new PGLiteEngine();
+ await inHome(async () => {
+ await engine.connect({ engine: 'pglite', database_path: datastore });
+ await engine.initSchema();
+ await registerLocalWriter(engine, 'stdio', { sourceIds: ['*'], operations: null, scopes: ['read', 'write'], slugPrefixes: null });
+ });
+}, 120_000);
+
+afterAll(async () => {
+ fake.stop();
+ await disposePersistenceConsumer(engine);
+ await engine.disconnect();
+ rmSync(home, { recursive: true, force: true });
+});
+
+beforeEach(async () => {
+ pgliteOn();
+ fake.calls.length = 0;
+ fake.respond.clear();
+ rmSync(join(home, '.gbrain', 'wakeshift-spool'), { recursive: true, force: true });
+ for (const table of ['wake_subscriptions', 'wake_crons', 'wake_cron_runs']) await engine.executeRaw(`DELETE FROM ${table}`);
+});
+
+describe('emit while another gbrain process holds the PGLite brain', () => {
+ test('the real CLI exits 0 and spools the run; the holder\'s drainer records it with its own time and wakes the watcher', async () => {
+ await captured(() => runWakeshift(engine, ['cron', 'add', 'nightly-example']));
+ const watch = await inHome(() => dispatchToolCall(engine, 'watch_cron', { name: 'nightly-example' }, stdio as never));
+ expect(watch.isError).toBeFalsy();
+ const before = Date.now();
+ const env: Record<string, string | undefined> = { ...process.env, GBRAIN_HOME: home, WAKESHIFT_API_KEY: undefined, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined };
+ const proc = Bun.spawn(['bun', 'run', `${REPO}/src/cli.ts`, 'wakeshift', 'emit', 'cron.finished', '--name', 'nightly-example', '--status', 'failed'], { cwd: REPO, env, stdout: 'pipe', stderr: 'pipe' });
+ const [stderr, exitCode] = await Promise.all([new Response(proc.stderr).text(), proc.exited]);
+ const after = Date.now();
+ expect(exitCode).toBe(0);
+ expect(stderr).toContain('so the nightly-example run was spooled');
+ const files = await pending();
+ expect(files).toHaveLength(1);
+ const file = join(await spoolDir(), files[0]!);
+ expect(statSync(file).mode & 0o777).toBe(0o600);
+ expect(file.startsWith(join(home, '.gbrain', 'wakeshift-spool'))).toBe(true);
+ const spooledAt = Date.parse(JSON.parse(readFileSync(file, 'utf8')).finished_at);
+ expect(spooledAt).toBeGreaterThanOrEqual(before);
+ expect(spooledAt).toBeLessThanOrEqual(after);
+ expect(await runs()).toHaveLength(0);
+ await Bun.sleep(20);
+ expect(await inHome(() => drainWakes(engine))).toMatchObject({ delivered: 1 });
+ const recorded = await runs();
+ expect(recorded.map(r => [r.name, r.status])).toEqual([['nightly-example', 'failed']]);
+ expect(new Date(recorded[0]!.finished_at).getTime()).toBe(spooledAt);
+ expect(await pending()).toHaveLength(0);
+ expect(emits()).toHaveLength(1);
+ expect(emits()[0]!.body.payload).toMatchObject({ name: 'nightly-example', status: 'failed' });
+ expect((await inHome(() => drainWakes(engine))).delivered).toBe(0);
+ expect(await runs()).toHaveLength(1);
+ }, 120_000);
+});
+
+describe('the spool', () => {
+ test('two drainers ingesting at once record each spooled run exactly once, and a replayed claim is not recorded again', async () => {
+ await inHome(() => { for (let i = 0; i < 5; i++) spoolCronRun(datastore, { name: 'nightly-example', status: 'ok' }, new Date(Date.now() - (5 - i) * 1000)); });
+ await engine.executeRaw("INSERT INTO wake_crons (name) VALUES ('nightly-example')");
+ const [a, b] = await inHome(() => Promise.all([ingestCronSpool(engine, datastore), ingestCronSpool(engine, datastore)]));
+ expect(a.recorded + b.recorded).toBe(5);
+ expect(await runs()).toHaveLength(5);
+ expect(await pending()).toHaveLength(0);
+
+ const dir = await spoolDir();
+ const [run] = await runs();
+ const stale = join(dir, `${String(new Date(run!.finished_at).getTime()).padStart(13, '0')}-00112233aabbccdd.json.claimed-0011223344556677`);
+ writeFileSync(stale, JSON.stringify({ name: run!.name, status: run!.status, finished_at: new Date(run!.finished_at).toISOString() }), { mode: 0o600 });
+ const old = new Date(Date.now() - 11 * 60_000);
+ (await import('node:fs')).utimesSync(stale, old, old);
+ expect(await inHome(() => ingestCronSpool(engine, datastore))).toEqual({ recorded: 1, rejected: 0 });
+ expect(await runs()).toHaveLength(5);
+ expect(await pending()).toHaveLength(0);
+ });
+
+ test('a malformed or oversized file is moved to rejected/ and reported once, never retried', async () => {
+ const dir = await spoolDir();
+ mkdirSync(dir, { recursive: true });
+ writeFileSync(join(dir, '0000000000001-aaaaaaaaaaaaaaaa.json'), '{not json', { mode: 0o600 });
+ writeFileSync(join(dir, '0000000000002-bbbbbbbbbbbbbbbb.json'), JSON.stringify({ name: 'x', status: 'ok', finished_at: new Date().toISOString(), pad: 'p'.repeat(SPOOL_MAX_BYTES) }), { mode: 0o600 });
+ writeFileSync(join(dir, '0000000000003-cccccccccccccccc.json'), JSON.stringify({ name: 'Bad Name', status: 'ok', finished_at: new Date().toISOString() }), { mode: 0o600 });
+ const first = await captured(() => ingestCronSpool(engine, datastore));
+ expect(first.value).toEqual({ recorded: 0, rejected: 3 });
+ expect(first.err).toContain('not JSON');
+ expect(first.err).toContain(`at most ${SPOOL_MAX_BYTES}`);
+ expect(first.err).toContain('not a cron run');
+ expect(readdirSync(join(dir, 'rejected')).sort()).toHaveLength(3);
+ expect(await pending()).toHaveLength(0);
+ const again = await captured(() => ingestCronSpool(engine, datastore));
+ expect(again.value).toEqual({ recorded: 0, rejected: 0 });
+ expect(again.err).toBe('');
+ expect(await runs()).toHaveLength(0);
+ });
+
+ test('a spooled time the database cannot store is moved to rejected/ and does not hold back the runs after it', async () => {
+ const dir = await spoolDir();
+ mkdirSync(dir, { recursive: true });
+ writeFileSync(join(dir, '0000000000001-aaaaaaaaaaaaaaaa.json'), JSON.stringify({ name: 'nightly-example', status: 'ok', finished_at: '0000-01-01T00:00:00.000Z' }), { mode: 0o600 });
+ await inHome(() => spoolCronRun(datastore, { name: 'nightly-example', status: 'failed' }));
+ const r = await captured(() => ingestCronSpool(engine, datastore));
+ expect(r.value).toEqual({ recorded: 1, rejected: 1 });
+ expect(r.err).toContain('not a cron run');
+ expect(readdirSync(join(dir, 'rejected'))).toEqual(['0000000000001-aaaaaaaaaaaaaaaa.json']);
+ expect((await runs()).map(x => x.status)).toEqual(['failed']);
+ expect(await pending()).toHaveLength(0);
+ });
+
+ test('a direct emit records its own run even when the spool cannot be ingested', async () => {
+ const dir = await spoolDir();
+ mkdirSync(dir, { recursive: true });
+ writeFileSync(join(dir, 'rejected'), '', { mode: 0o600 });
+ writeFileSync(join(dir, '0000000000001-aaaaaaaaaaaaaaaa.json'), '{not json', { mode: 0o600 });
+ const r = await captured(() => runWakeshift(engine, ['emit', 'cron.finished', '--name', 'nightly-example', '--status', 'ok']));
+ expect(r.err).toContain('could not take the spooled cron runs');
+ expect((await runs()).map(x => [x.name, x.status])).toEqual([['nightly-example', 'ok']]);
+ });
+
+ test('a run spooled through a symlink to the datastore is ingested by the drainer that opened the real path', async () => {
+ const link = join(home, 'brain-link.pglite');
+ rmSync(link, { force: true });
+ symlinkSync(datastore, link);
+ await engine.executeRaw("INSERT INTO wake_crons (name) VALUES ('nightly-example')");
+ await inHome(() => spoolCronRun(link, { name: 'nightly-example', status: 'ok' }));
+ await inHome(() => drainWakes(engine));
+ expect((await runs()).map(x => [x.name, x.status])).toEqual([['nightly-example', 'ok']]);
+ expect(await pending()).toHaveLength(0);
+ });
+
+ test('a spooled run of an unregistered cron is recorded with the same hint as the direct path, and the next direct emit takes leftovers', async () => {
+ await inHome(() => spoolCronRun(datastore, { name: 'never-added', status: 'ok' }));
+ const r = await captured(() => runWakeshift(engine, ['emit', 'cron.finished', '--name', 'never-added', '--status', 'failed']));
+ expect(r.err).toContain('cron never-added is not registered');
+ expect((await runs()).map(x => x.status)).toEqual(['ok', 'failed']);
+ expect(await pending()).toHaveLength(0);
+ });
+
+ test('a full spool keeps the cron script\'s status and says why the run was dropped', async () => {
+ const dir = await spoolDir();
+ mkdirSync(dir, { recursive: true });
+ for (let i = 0; i < SPOOL_MAX_FILES; i++) writeFileSync(join(dir, `${String(i).padStart(13, '0')}-${i.toString(16).padStart(16, '0')}.json`), '{}');
+ const r = await captured(() => reportEmitConnectFailure(['cron.finished', '--name', 'nightly-example', '--status', 'ok'], new Error('busy')));
+ expect(r.err).toContain(`${SPOOL_MAX_FILES} runs already wait in the spool`);
+ expect(await pending()).toHaveLength(SPOOL_MAX_FILES);
+ });
+
+ test('nothing is spooled when reporting back is off, or on a Postgres brain', async () => {
+ writeConfig({ engine: 'pglite', database_path: datastore, wakeshift: { enabled: false } });
+ const off = await captured(() => reportEmitConnectFailure(['cron.finished', '--name', 'nightly-example', '--status', 'ok'], new Error('busy')));
+ expect(off.err).toBe('[wakeshift] could not record the nightly-example run: the brain could not be opened (busy).\n');
+ writeConfig({ engine: 'postgres', database_url: 'postgresql://[email protected]:1/none', wakeshift: { enabled: true, url: fake.url, manifest_hash: wakeshiftManifestHash() } });
+ const pg = await captured(() => reportEmitConnectFailure(['cron.finished', '--name', 'nightly-example', '--status', 'ok'], new Error('refused')));
+ expect(pg.err).toBe('[wakeshift] could not record the nightly-example run: the brain could not be opened (refused).\n');
+ expect(existsSync(join(home, '.gbrain', 'wakeshift-spool'))).toBe(false);
+ });
+});
diff --git a/test/wakeshift-pr2.serial.test.ts b/test/wakeshift-pr2.serial.test.ts
index 2de48a23c4..36dab49d6c 100644
--- a/test/wakeshift-pr2.serial.test.ts
+++ b/test/wakeshift-pr2.serial.test.ts
@@ -421,6 +421,38 @@ describe('forget on remove', () => {
expect(await drain()).toMatchObject({ forgotten: 1 });
});
+ test('token revoke by an uppercase id still ends the token\'s watches and queues its forget', async () => {
+ const minted = await mintLegacyToken(engine, { name: 'wake-token-upper', takesHolders: ['world'], scopes: ['read', 'write', 'admin'] });
+ const ctx = { remote: true, transport: 'http' as const, sourceId: 'default', config: { engine: 'pglite' }, logger: log,
+ auth: { token: 't', clientId: minted.id, principal: { kind: 'legacy_token' as const, id: minted.id }, scopes: ['read', 'write', 'admin'] } };
+ expect(body(await call('watch_job', { id: await seedJob() }, ctx))).toMatchObject({ watching: true });
+ writeConfig({ enabled: false, url: fake.url });
+ expect(await revokeLegacyTokenById(sqlQueryForEngine(engine), minted.id.toUpperCase())).toBe(true);
+ expect(await engine.executeRaw('SELECT state, last_error FROM wake_subscriptions')).toEqual([{ state: 'ended', last_error: 'principal_revoked' }]);
+ expect(await engine.executeRaw('SELECT state FROM wake_forgets')).toEqual([{ state: 'pending' }]);
+ });
+
+ test('a removed member whose forget was never queued is forgotten by the drainer; a live member\'s revoked credential is not', async () => {
+ await addMember('alice-example');
+ await addMember('bob-example');
+ await addClient('client-m1', 'alice-example');
+ await addClient('client-m2', 'bob-example');
+ writeConfig({ enabled: true, url: fake.url, terms_url: TERMS, privacy_url: PRIVACY });
+ for (const c of ['client-m1', 'client-m2']) {
+ const step = body(await call('watch_job', { id: await seedJob() }, client(c)));
+ await call('accept_wakeshift_terms', { terms_version: step.terms_version }, client(c));
+ expect(body(await call('watch_job', { id: await seedJob() }, client(c)))).toMatchObject({ watching: true });
+ }
+ await engine.executeRaw("UPDATE access_members SET tombstoned_at = now(), disabled_at = now() WHERE id = 'alice-example'");
+ await engine.executeRaw("UPDATE oauth_clients SET deleted_at = now() WHERE client_id = 'client-m2'");
+ await engine.executeRaw("UPDATE minion_jobs SET status = 'completed', finished_at = now()");
+ expect(await drain()).toMatchObject({ delivered: 0, ended: 2 });
+ const [alice] = await engine.executeRaw<{ wake_user: string }>("SELECT wake_user FROM wake_subscriptions WHERE member_id = 'alice-example'");
+ expect(await engine.executeRaw('SELECT wake_user, state FROM wake_forgets')).toEqual([{ wake_user: alice!.wake_user, state: 'pending' }]);
+ expect(await drain()).toMatchObject({ forgotten: 1 });
+ expect(paths('/users/forget').map(c => c.body)).toEqual([{ user: alice!.wake_user }]);
+ });
+
test('a refused forget is retried on the schedule and then dies', async () => {
await engine.executeRaw("INSERT INTO wake_forgets (wake_user) VALUES ('0123456789abcdef0123456789abcdef')");
fake.respond.set('/users/forget', () => ({ status: 503, body: { error: { code: 'unavailable', message: 'down' } } }));