Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
A replaced CLI registration's stopped remove no longer blocks deactivate, automatic purges are left to the next run, and writer status lists stopped requests
garrytan426e0a69ed45
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2bc00857c4..59e8e8989d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,14 +12,14 @@ identifiers and attribution are available in the pre-removal Git revision## [Unreleased]-**Retrying an interrupted `gbrain sources remove` or `purge` can no longer delete a re-added source's pages.** A managed remove or purge deletes a large source's pages in batches, and a run that stops partway can be finished by rerunning it with the same `--request-id`. Until now nothing tied that request id to the source it was started against: if the source was removed and added again under the same ID in between, the rerun deleted the new source's pages. The first batch now records which incarnation of the source the request was admitted against, in the same transaction as its first deletion. A rerun against the same incarnation still resumes and finishes; a rerun after the source was replaced refuses with `source_retire_superseded` and changes nothing. Restoring, archiving or otherwise changing the source after a run stopped abandons that run the same way, so a purge stopped before a restore can't hold up `deactivate` or graduation for good, and both ask the user before a stopped run is finished, since finishing deletes the rest of the source's pages.+**Retrying an interrupted `gbrain sources remove` or `purge` can no longer delete a re-added source's pages.** A managed remove or purge deletes a large source's pages in batches, and a run that stops partway can be finished by rerunning it with the same `--request-id`. Until now nothing tied that request id to the source it was started against: if the source was removed and added again under the same ID in between, the rerun deleted the new source's pages. The first batch now records which incarnation of the source the request was admitted against, in the same transaction as its first deletion. A rerun against the same incarnation still resumes and finishes; a rerun after the source was replaced refuses with `source_retire_superseded` and changes nothing. Restoring, archiving or otherwise changing the source after a run stopped abandons that run the same way, so a purge stopped before a restore can't hold up `deactivate` or graduation for good, and both ask the user before a stopped run is finished, since finishing deletes the rest of the source's pages. A stopped automatic expired-archive purge is left to the next automatic run instead of being handed to a person, and `gbrain sources writer status --source <id> --json` now lists a source's stopped and superseded remove and purge requests.### Itemized changes-- `src/core/persistence/topology-receipts.ts`: `admitTopologyRetire` records a retire admission (a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation, holding one lifetime ID and a full receipt reservation after the same quota check `assertTopologyChangeCapacity` made); `recordTopologyChange` settles a request's own admission `committed` and trues up its receipt bytes; `supersedeTopologyRetires` settles other requests' admissions `failed` when a remove or purge deletes their incarnation, or when any other lifecycle operation (archive, restore, rebind, claim) commits a change to it.+- `src/core/persistence/topology-receipts.ts`: `admitTopologyRetire` records a retire admission (a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation, holding one lifetime ID and a full receipt reservation after the same quota check `assertTopologyChangeCapacity` made); `recordTopologyChange` settles a request's own admission `committed` and trues up its receipt bytes; `supersedeTopologyRetires` settles other requests' admissions `failed` when a remove or purge deletes their incarnation, or when any other lifecycle operation (archive, restore, rebind, claim) commits a change to it; it skips only the caller's own (principal, request id) pair, so a retry under a replaced CLI registration settles the old registration's admission too.- `src/core/persistence/source-lifecycle.ts`: a request id whose admission names another incarnation than the source registered now refuses `source_retire_superseded` before any lock or deletion (a dry run too) and settles the admission `failed`, so a later retry refuses the same way; the batches and the final transaction resume past their own admission (#8984).-- `src/core/error-registry.ts`: new caller code `source_retire_superseded`, with a read-only writer-status fix.-- `src/core/persistence/deactivation.ts`, `graduation-drain.ts`: an unfinished retire is listed as a blocker that names its request id and asks the user whether to finish it (rerun the same command with its request id) or keep what is left (another lifecycle change abandons it); graduation marks it `needsUser`.+- `src/core/error-registry.ts`: new caller code `source_retire_superseded`, with a read-only writer-status fix; `sources writer status --json` gains `topology_requests` (recovering admissions and superseded requests).+- `src/core/persistence/deactivation.ts`, `graduation-drain.ts`: an unfinished retire is listed as a blocker that names its request id and asks the user whether to finish it (rerun the same command with its request id) or keep what is left (another lifecycle change abandons it); graduation marks it `needsUser`. A stopped automatic expired-archive purge (its admission records `expired_only`) instead says the next automatic run finishes it, with `needsUser: false`.- Tests: `test/source-retire-incarnation.test.ts` (PGLite) and `test/e2e/source-retire-incarnation-postgres.test.ts`: a retry after the final transaction failed, and a retry after committed batches already deleted part of the source, both refuse once the source was replaced and keep the new source's pages (both resolved `committed` on master); a same-incarnation retry resumes and settles; a restore after an interrupted purge abandons it, so its retry refuses and no blocker stays. `test/source-retire-batched.test.ts` and its Postgres twin now expect the interrupted run's admission row.## [0.60.629.0] - 2026-11-09diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.mdindex 78fd7f03a4..bca5c5c8c0 100644--- a/docs/architecture/key-files/core-persistence-continued.md+++ b/docs/architecture/key-files/core-persistence-continued.md@@ -41,4 +41,4 @@ The #6188 fence check and fence repair live in [fence repair](core-fence-repair.- `src/core/postgres-engine/reserved-transactions.ts` with `isPersistenceDeclaration` (`src/core/persistence/protocol.ts`) — BEGIN pipelining. `isPersistenceDeclaration` (GBRA-156) accepts exactly the `protocol.ts` declarations (transaction-local `set_config` calls and the brain row FOR SHARE). Admission, the FIFO claim, the effect claims and source guards, the receipt true-up, the single publication (`coordinator.ts`) and the consumer's lent lane transactions (`consumer.ts` `onLane`, through `reservedTransactions`) pass it as `TransactionOptions.pipelineFirst`, so behind a transaction-mode pooler the declaration goes out with BEGIN (one round trip less per transaction) while every later statement still waits for BEGIN's answer; if BEGIN fails, only the declaration has run, in autocommit, where it changes nothing. A reserved transaction gates every statement its body can reach, savepoint and nested-transaction bodies and `file()` included. A sync lane group's publication passes none: its begun mark must follow BEGIN's answer (`sync-lanes.ts` `awaitLaneBegin`). Tests: `test/persistence-pipeline-first.test.ts` (the predicate and the reviewed call sites), `test/e2e/pgbouncer-transaction-pipeline-6930.test.ts`.- `src/core/links-fence.ts` + `src/core/links-fence-projection.ts` — the `## Links` fence (G5): the markdown form of authored (manual) edges, `<!--- gbrain:links:begin -->` / `end` around `| to | type | source | valid_from | valid_until | context |`, one row per (to, type, source), source defaulting to `manual`; a reconciliation-managed provenance (`MANAGED_LINK_SOURCES`, exported here and re-exported by `ops/links.ts`), a bad date or a duplicate key is a skipped row with a warning. `parseLinksFence` (state `absent` / `parsed` / `malformed`), the deterministic `renderLinksFence` (rows sorted by key), `upsertLinksFenceRow` (omitted context and dates keep the stored ones; identical row = unchanged body), `removeLinksFenceRows` (leaves an empty fence), `replaceOrInsertLinksFence` (a `## Links` section at the end of compiled_truth, above an unclosed code block) and `maskLinksFence`, which `extractBodyLinks` applies so the fence cells are never also extracted as markdown, wikilink or bare-slug edges. A rewrite of a fence that parses with warnings throws `LinksFenceRewriteRefusal`. Cells use `escapeFenceCell` plus the same one-level marker encoding for `gbrain:links:*`. `projectLinksFence(tx, page, rows)` reconciles only what the page's fence owns: edges with origin_page_id = the page and origin_field `links_fence` (deleted, context-updated, inserted by key) and `manual` transitions with origin_page_id = the page, then refreshes the touched relationships; a row whose target is not a live page in the source yields no edge. `linksFenceProjectionRows` returns null for an absent or malformed fence, so such a body changes no edge (an absent fence retires nothing, as for facts; unlike facts no held generation is recorded, so after a hand deletion of the whole section the database keeps edges git no longer has until `remove_link` or the G3 indexer), and drops rows typed with a link type the active pack does not declare. Callers: `prepareCanonicalProjections` (every projected page publication, quarantined and segmented-unread bodies excepted), `prepareAutomaticLinks` (put_page auto-links and the managed links maintenance, so `extract --stale` restores a lost fence edge) and `persistence/manual-links.ts`. Database-only manual edges (origin_field NULL) are never touched by any of them. Pinned by `test/links-fence.test.ts` and `test/managed-links-fence.test.ts`.-- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt other than this request's own admission, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`, `lockTopologyPrincipal`, and on the first batch `admitTopologyRetire` from `topology-receipts.ts`, which checks the receipt quotas at the largest receipt and records the admission) and, when one would refuse or replay, stops without deleting so the final transaction reports it, or throws that refusal before deleting; a batch that fails `write_pending` is retried at half the size (floor 1), so heavy pages still fit the bound; a refusal after some batches committed says how many pages that run already deleted; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. #8984: the first batch's transaction records the admission, a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation it deletes from (`isRetireAdmission`); `recordTopologyChange` settles it `committed`, and a remove or purge that deletes that incarnation under another request, or any other lifecycle operation that commits a change to it (archive, restore, rebind, claim), settles it `failed` (`supersedeTopologyRetires`). A retry of the request_id against another incarnation (the source was removed and re-added) refuses `source_retire_superseded`, settles the admission `failed` and deletes nothing; deactivation and graduation list an unsettled admission as a blocker that names its request_id and asks the user whether to finish it or keep what is left. Tests: `test/source-retire-batched.test.ts` (PGLite), `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages), `test/source-retire-incarnation.test.ts` and `test/e2e/source-retire-incarnation-postgres.test.ts`.+- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt other than this request's own admission, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`, `lockTopologyPrincipal`, and on the first batch `admitTopologyRetire` from `topology-receipts.ts`, which checks the receipt quotas at the largest receipt and records the admission) and, when one would refuse or replay, stops without deleting so the final transaction reports it, or throws that refusal before deleting; a batch that fails `write_pending` is retried at half the size (floor 1), so heavy pages still fit the bound; a refusal after some batches committed says how many pages that run already deleted; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. #8984: the first batch's transaction records the admission, a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation it deletes from (`isRetireAdmission`); `recordTopologyChange` settles it `committed`, and a remove or purge that deletes that incarnation under another request, or any other lifecycle operation that commits a change to it (archive, restore, rebind, claim), settles it `failed` (`supersedeTopologyRetires`, which skips only the caller's own principal and request_id pair). A retry of the request_id against another incarnation (the source was removed and re-added) refuses `source_retire_superseded`, settles the admission `failed` and deletes nothing; deactivation and graduation list an unsettled admission as a blocker that names its request_id and asks the user whether to finish it or keep what is left; an automatic expired-archive purge's admission (`expired_only` in its outcome) is left to the next automatic run with `needsUser: false`, and `sources writer status --json` lists the stopped and superseded requests (`topology_requests`). Tests: `test/source-retire-batched.test.ts` (PGLite), `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages), `test/source-retire-incarnation.test.ts` and `test/e2e/source-retire-incarnation-postgres.test.ts`.diff --git a/src/core/error-registry.ts b/src/core/error-registry.tsindex 3894f22a40..61e9002302 100644--- a/src/core/error-registry.ts+++ b/src/core/error-registry.ts@@ -751,7 +751,7 @@ export const CODES = {source_not_ready: { class: 'caller', summary: "A committed source and approved schema are required." },source_profile_no_backfill: { class: 'unavailable', summary: "This source profile disables automatic embedding backfill." },source_referenced: { class: 'caller', summary: "OAuth clients still reference this source." },- source_retire_superseded: { class: 'caller', summary: "A retried source remove or purge was superseded: its source has since been removed, replaced or changed by another lifecycle operation; nothing was changed.", why: "A remove or purge records the source incarnation it was admitted against before it deletes its first page, so a retry of its request id can never delete the pages of a source re-added under the same ID, and a later lifecycle change to the source (such as a restore) abandons the request. A request refused this way stays refused.", suggestion: "Do not retry this request id. Read the source's current state; removing the current source is a new decision for the user, made with a new request id.", fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', '{source_id}', '--json'], consent: [], actor: 'agent', why: "Read-only: shows the source's current incarnation, binding and pending work.", requires_exclusive: false } },+ source_retire_superseded: { class: 'caller', summary: "A retried source remove or purge was superseded: its source has since been removed, replaced or changed by another lifecycle operation; nothing was changed.", why: "A remove or purge records the source incarnation it was admitted against before it deletes its first page, so a retry of its request id can never delete the pages of a source re-added under the same ID, and a later lifecycle change to the source (such as a restore) abandons the request. A request refused this way stays refused.", suggestion: "Do not retry this request id. Read the source's current state; removing the current source is a new decision for the user, made with a new request id.", fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', '{source_id}', '--json'], consent: [], actor: 'agent', why: "Read-only: shows the source's current incarnation, binding, pending work and its stopped or superseded remove and purge requests (topology_requests).", requires_exclusive: false } },source_root_conflict: { class: 'caller', summary: "Multiple active sources alias this canonical skill root; resolve ownership before publishing." },source_writeback_required: { class: 'host_only', summary: "Canonical preparation requires a source-content correction; this profile never writes repository files." },sources_not_readable: { class: 'host_only', retryable: false, summary: "An unscoped search, query, recall or list_pages came back empty and was refused: every source this connection can read (the default source and federated sources) holds no page (for recall and query, no world fact either), while other sources on the brain hold pages it cannot read (not federated, and no source grant).", why: "Answering would return an empty result that reads like a clean miss. The refusal names the host command that federates a source or grants this connection the sources it should read; it never names a hidden source and never widens access. A scope that holds pages but matched nothing, a granted token, an OAuth client, a bound connection and a read that names a source are never refused.", docs: 'docs/guides/multi-source-brains.md#an-unscoped-connection-whose-sources-hold-no-pages', suggestion: 'Ask the user which source these connections should read; the brain host runs `gbrain sources federate <source>`, or grants the token its sources with `gbrain auth rescope --token <name> --sources <ids>`.' },diff --git a/src/core/persistence/administration.ts b/src/core/persistence/administration.tsindex 5c219fad52..d917ce0a46 100644--- a/src/core/persistence/administration.ts+++ b/src/core/persistence/administration.ts@@ -317,6 +317,10 @@ export async function runPersistenceAdministration(engine: BrainEngine, operatio// #6317 (B3): every process holding a consumer on this host, from the heartbeat rows (kind, pid, nonce, mode, age, wedge report).host: { host_id: existingHostIdFor(engine), ...await (await import('./consumer-diagnostics.ts')).hostConsumersReport(engine, existingHostIdFor(engine)) },worktree_refreshes: await (await import('./worktree-refresh.ts')).activeWorktreeRefreshes(engine, params.source_id as string | undefined),+ // #8984: stopped remove/purge requests (retire admissions) and the ones a later lifecycle change superseded.+ topology_requests: await engine.executeRaw(`SELECT request_id::text AS request_id,operation,source_id,source_incarnation::text AS source_incarnation,state,outcome->>'error' AS error,created_at::text AS created_at+ FROM persistence_topology_changes WHERE ($1::text IS NULL OR source_id=$1) AND operation IN ('remove','purge') AND recovery IS NULL+ AND (state='recovering' OR state='failed' AND outcome->>'error'='source_retire_superseded') ORDER BY created_at DESC LIMIT 20`, [params.source_id === undefined ? null : source(params.source_id)]),// Q1: each canonical checkout on this host against its last fetched upstream: commits it holds that the upstream lacks, and its porcelain.checkouts: await writerStatusCheckouts(bindings as Array<{ source_id: string; local_path: string | null }>),// #6317 (B4): per managed source, whether sync data moves (shares the reader of sources status, doctor and writer movement).diff --git a/src/core/persistence/deactivation.ts b/src/core/persistence/deactivation.tsindex 48afd5edcd..83ab2bc41b 100644--- a/src/core/persistence/deactivation.ts+++ b/src/core/persistence/deactivation.ts@@ -114,11 +114,13 @@ export async function deactivationBlockers(engine: BrainEngine): Promise<Deactiv: r.state === 'running' && !r.recovering ? 'wait for the owner to finish publishing it, then rerun deactivate': `gbrain sync --source ${r.source_id} --no-pull` });}- const topology = await engine.executeRaw<{ id: string; operation: string; source_id: string; request_id: string; admission: boolean }>(- "SELECT id::text AS id, operation, source_id, request_id::text AS request_id, (state='recovering' AND recovery IS NULL) AS admission FROM persistence_topology_changes WHERE state='recovering' OR recovery IS NOT NULL ORDER BY created_at LIMIT 20");+ const topology = await engine.executeRaw<{ id: string; operation: string; source_id: string; request_id: string; admission: boolean; automatic: boolean }>(+ "SELECT id::text AS id, operation, source_id, request_id::text AS request_id, (state='recovering' AND recovery IS NULL) AS admission, COALESCE(outcome->>'expired_only','false')='true' AS automatic FROM persistence_topology_changes WHERE state='recovering' OR recovery IS NOT NULL ORDER BY created_at LIMIT 20");for (const t of topology) blockers.push({ kind: 'topology_recovery', id: t.id, source_id: t.source_id,- detail: t.admission ? `${t.operation} of ${t.source_id} (request ${t.request_id}) stopped before it finished` : `${t.operation} for ${t.source_id} is recovering`,- exit: t.admission ? `ask the user whether to finish it, because finishing deletes the rest of ${t.source_id}'s pages: to finish, rerun the same gbrain sources ${t.operation} command with all its original flags and --request-id ${t.request_id} (an automatic expired-archive purge finishes on its next run); to keep what is left, any other lifecycle change to the source (gbrain sources restore of an archived source, or archive then restore) abandons the request; then rerun deactivate`+ detail: !t.admission ? `${t.operation} for ${t.source_id} is recovering`+ : `${t.automatic ? 'automatic expired-archive purge' : t.operation} of ${t.source_id} (request ${t.request_id}) stopped before it finished`,+ exit: t.admission && t.automatic ? `the next automatic expired-archive purge finishes it (gbrain sources writer status --source ${t.source_id} --json lists the request), then rerun deactivate`+ : t.admission ? `ask the user whether to finish it, because finishing deletes the rest of ${t.source_id}'s pages: to finish, rerun the same gbrain sources ${t.operation} command with all its original flags and --request-id ${t.request_id} (an automatic expired-archive purge finishes on its next run); to keep what is left, any other lifecycle change to the source (gbrain sources restore of an archived source, or archive then restore) abandons the request; then rerun deactivate`: `gbrain sources writer status ${t.source_id} --json, then let the owner finish recovery` });// GBRA-130 A-15: only work that can still change blocks, as activation checks; terminal failures are failedEffectHistory's.const effects = await engine.executeRaw<EffectRow>(`${EFFECT_ROWS} WHERE e.state IN ('queued','running') OR e.recovery IS NOT NULL ORDER BY e.id LIMIT 50`);diff --git a/src/core/persistence/graduation-drain.ts b/src/core/persistence/graduation-drain.tsindex c3328ae42e..9a77a2d6f0 100644--- a/src/core/persistence/graduation-drain.ts+++ b/src/core/persistence/graduation-drain.ts@@ -58,13 +58,15 @@ export async function graduationBlockers(source: BrainEngine, hostId: string): Pblockers.push({ kind: 'request', id: r.request_id, detail, needsUser: false });}}- const topology = await source.executeRaw<{ id: string; operation: string; source_id: string; request_id: string; admission: boolean }>(- "SELECT id::text AS id, operation, source_id, request_id::text AS request_id, (state = 'recovering' AND recovery IS NULL) AS admission FROM persistence_topology_changes WHERE state = 'recovering' OR recovery IS NOT NULL ORDER BY created_at");+ const topology = await source.executeRaw<{ id: string; operation: string; source_id: string; request_id: string; admission: boolean; automatic: boolean }>(+ "SELECT id::text AS id, operation, source_id, request_id::text AS request_id, (state = 'recovering' AND recovery IS NULL) AS admission, COALESCE(outcome->>'expired_only', 'false') = 'true' AS automatic FROM persistence_topology_changes WHERE state = 'recovering' OR recovery IS NOT NULL ORDER BY created_at");for (const t of topology) {- blockers.push({ kind: 'topology_recovery', id: t.id, detail: t.admission+ blockers.push({ kind: 'topology_recovery', id: t.id, detail: t.admission && t.automatic+ ? `automatic expired-archive purge of ${t.source_id} (request ${t.request_id}) stopped before it finished; the next automatic expired-archive purge finishes it`+ : t.admission? `${t.operation} of ${t.source_id} (request ${t.request_id}) stopped before it finished; ask the user whether to finish it (rerun the same command with all its original flags and --request-id ${t.request_id}, which deletes the rest of its pages) or keep what is left (any other lifecycle change to the source, such as a restore, abandons the request)`: `${t.operation} for ${t.source_id} is recovering`,- argv: ['gbrain', 'sources', 'writer', 'status', t.source_id, '--json'], needsUser: t.admission });+ argv: ['gbrain', 'sources', 'writer', 'status', t.source_id, '--json'], needsUser: t.admission && !t.automatic });}const effects = await source.executeRaw<{ id: string; kind: string; state: string; error_code: string | null; source_id: string | null; request_id: string }>(`SELECT e.id::text AS id, e.kind, e.state, e.error_code, COALESCE(e.source_id, r.source_id) AS source_id, r.request_id::text AS request_iddiff --git a/src/core/persistence/source-lifecycle.ts b/src/core/persistence/source-lifecycle.tsindex 2ce92ee813..4a0f9120a9 100644--- a/src/core/persistence/source-lifecycle.ts+++ b/src/core/persistence/source-lifecycle.ts@@ -222,7 +222,7 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceLawait assertTopologyRequestsSettled(tx,sources,bindings.map(binding=>binding.worktree_id));await lockTopologyPrincipal(tx,principal);if(!admitted) await admitTopologyRetire(tx,{principal,requestId,intent,operation:input.operation,sourceId:input.sourceId,- incarnation:source.incarnation,worktrees:bindings.map(binding=>binding.worktree_id)});+ incarnation:source.incarnation,worktrees:bindings.map(binding=>binding.worktree_id),expiredOnly:input.expiredOnly});const gone=await withCoordinatedWrite(tx,sources,async()=>{deleting=true;const rows=await tx.executeRaw('DELETE FROM pages WHERE id IN (SELECT id FROM pages WHERE source_id=$1 LIMIT $2) RETURNING id',[input.sourceId,limit]);@@ -339,9 +339,9 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceLpagesDeleted+=Number(impact.count);await tx.executeRaw('DELETE FROM sources WHERE id=$1 AND incarnation=$2::uuid',[input.sourceId,incarnation]);await tx.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1 AND source_incarnation=$2::uuid',[input.sourceId,incarnation]);- await supersedeTopologyRetires(tx,input.sourceId,incarnation,requestId);+ await supersedeTopologyRetires(tx,input.sourceId,incarnation,principal,requestId);}- if(source&&!['remove','purge'].includes(input.operation)) await supersedeTopologyRetires(tx,input.sourceId,source.incarnation,requestId);+ if(source&&!['remove','purge'].includes(input.operation)) await supersedeTopologyRetires(tx,input.sourceId,source.incarnation,principal,requestId);const contentReceipts=input.operation==='rebind'?await rebindContentReceipts(tx,worktrees,hostIdFor(tx),relocated?undefined:[input.sourceId]):[];await advanceTopology(tx,worktrees);for(const id of worktrees){diff --git a/src/core/persistence/topology-receipts.ts b/src/core/persistence/topology-receipts.tsindex 70d21a0932..ef62810e0f 100644--- a/src/core/persistence/topology-receipts.ts+++ b/src/core/persistence/topology-receipts.ts@@ -85,7 +85,7 @@ export const RETIRE_SUPERSEDED='source_retire_superseded';export function isSupersededRetire(row:TopologyChange):boolean{return row.state==='failed'&&row.outcome?.error===RETIRE_SUPERSEDED;}-export async function admitTopologyRetire(tx:BrainEngine,input:{principal:string;requestId:string;intent:unknown;operation:string;sourceId:string;incarnation:string;worktrees:string[]}):Promise<void>{+export async function admitTopologyRetire(tx:BrainEngine,input:{principal:string;requestId:string;intent:unknown;operation:string;sourceId:string;incarnation:string;worktrees:string[];expiredOnly?:boolean}):Promise<void>{const limits=await readJournalLimits(tx);const reservation=receiptReservation({});const counters=await lockCounters(tx,['brain',principalKey({kind:'local_cli',id:input.principal})]);@@ -94,7 +94,7 @@ export async function admitTopologyRetire(tx:BrainEngine,input:{principal:string(principal_id,request_id,digest,operation,source_id,source_incarnation,worktree_ids,state,terminal_bytes,outcome)VALUES($1::uuid,$2::uuid,$3,$4,$5,$6::uuid,$7::uuid[],'recovering',$8,$9::text::jsonb)`,[input.principal,input.requestId,digest(input.intent),input.operation,input.sourceId,input.incarnation,input.worktrees,reservation,- JSON.stringify({operation:input.operation,source_id:input.sourceId,source_incarnation:input.incarnation})]);+ JSON.stringify({operation:input.operation,source_id:input.sourceId,source_incarnation:input.incarnation,...(input.expiredOnly?{expired_only:true}:{})})]);for(const counter of counters) await tx.executeRaw('UPDATE persistence_counters SET lifetime_ids=lifetime_ids+1,terminal_bytes=terminal_bytes+$2 WHERE key=$1',[counter.key,reservation]);}/** Settles a retire admission and trues its reserved receipt bytes up to the outcome it keeps; null when already settled. */@@ -107,10 +107,10 @@ export async function settleTopologyRetire(tx:BrainEngine,row:TopologyChange,stafor(const counter of counters) await tx.executeRaw('UPDATE persistence_counters SET terminal_bytes=terminal_bytes+$2 WHERE key=$1',[counter.key,terminalBytes-Number(row.terminal_bytes)]);return settled;}-/** A remove or purge that deleted incarnation X fails every other request still admitted against X: none can resume. */-export async function supersedeTopologyRetires(tx:BrainEngine,sourceId:string,incarnation:string,requestId:string):Promise<void>{+/** A lifecycle change that deleted or changed incarnation X fails every other request still admitted against X: none can resume. Requests are keyed by (principal, request_id), so the same request_id under another CLI registration is another request. */+export async function supersedeTopologyRetires(tx:BrainEngine,sourceId:string,incarnation:string,principal:string,requestId:string):Promise<void>{const rows=await tx.executeRaw<TopologyChange>(`SELECT * FROM persistence_topology_changes WHERE source_id=$1 AND source_incarnation=$2::uuid- AND state='recovering' AND recovery IS NULL AND operation IN ('remove','purge') AND request_id<>$3::uuid ORDER BY id`,[sourceId,incarnation,requestId]);+ AND state='recovering' AND recovery IS NULL AND operation IN ('remove','purge') AND NOT (principal_id=$3::uuid AND request_id=$4::uuid) ORDER BY id`,[sourceId,incarnation,principal,requestId]);for(const row of rows) await settleTopologyRetire(tx,row,'failed',{...row.outcome,error:RETIRE_SUPERSEDED});}/** Caller retains owner/source guards and completes final state in this same transaction. */diff --git a/test/source-retire-incarnation.test.ts b/test/source-retire-incarnation.test.tsindex ebb609eab1..64790779a6 100644--- a/test/source-retire-incarnation.test.ts+++ b/test/source-retire-incarnation.test.ts@@ -12,13 +12,13 @@ import { tmpdir } from 'node:os';import { PGLiteEngine } from '../src/core/pglite-engine.ts';import { resetPgliteState } from './helpers/reset-pglite.ts';import { withEnv } from './helpers/with-env.ts';-import { registerLocalWriter } from '../src/core/persistence/identity.ts';+import { localHostId, registerLocalWriter } from '../src/core/persistence/identity.ts';import { claimWorktree } from '../src/core/persistence/ownership.ts';import { runManagedSourceLifecycle, SOURCE_RETIRE_BATCH_PAGES } from '../src/core/persistence/source-lifecycle.ts';import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';import { deactivationBlockers } from '../src/core/persistence/deactivation.ts';import { graduationBlockers } from '../src/core/persistence/graduation-drain.ts';-import { localHostId } from '../src/core/persistence/identity.ts';+import { runPersistenceAdministration } from '../src/core/persistence/administration.ts';let engine: PGLiteEngine;beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); });@@ -160,3 +160,53 @@ test('a retry against the same incarnation resumes, finishes and settles its recexpect((await deactivationBlockers(engine)).filter(b => b.kind === 'topology_recovery')).toEqual([]);expect(await incarnation()).toBeUndefined();}), 60_000);++test('a retry under a replaced CLI registration finishes and settles the old registration\'s stopped admission', () => fixture(SOURCE_RETIRE_BATCH_PAGES + 3, async () => {+ const requestId = randomUUID();+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });+ await expect(failFinalTransaction(remove)).rejects.toThrow('injected final retire failure');+ const [old] = await engine.executeRaw<{ principal_id: string }>('SELECT principal_id::text AS principal_id FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId]);+ const replaced = await registerLocalWriter(engine, 'cli', undefined, true);+ expect(replaced.id).not.toBe(old.principal_id);+ expect(await remove()).toMatchObject({ state: 'committed', operation: 'remove' });+ expect(await engine.executeRaw('SELECT principal_id::text AS principal_id, state FROM persistence_topology_changes WHERE request_id=$1::uuid ORDER BY created_at', [requestId]))+ .toEqual([{ principal_id: old.principal_id, state: 'failed' }, { principal_id: replaced.id, state: 'committed' }]);+ expect((await deactivationBlockers(engine)).filter(b => b.kind === 'topology_recovery')).toEqual([]);+ expect((await graduationBlockers(engine, localHostId())).filter(b => b.kind === 'topology_recovery')).toEqual([]);+}), 60_000);++test('a stopped automatic expired-archive purge is left to the next automatic run, not handed to a person', () => fixture(SOURCE_RETIRE_BATCH_PAGES * 2, async () => {+ expect(await runManagedSourceLifecycle(engine, { operation: 'archive', sourceId: source })).toMatchObject({ state: 'committed' });+ await engine.executeRaw("UPDATE sources SET archive_expires_at=now()-interval '1 hour' WHERE id=$1", [source]);+ const first = await incarnation();+ const requestId = randomUUID();+ const purge = () => runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true, expiredOnly: true, expectedIncarnation: first, requestId });+ await expect(failAfterFirstBatch(purge)).rejects.toThrow('injected retire batch failure');+ expect(await change(requestId)).toMatchObject({ state: 'recovering', recovery: null });+ const deactivate = (await deactivationBlockers(engine)).find(b => b.kind === 'topology_recovery');+ expect(deactivate?.detail).toContain(`automatic expired-archive purge of ${source} (request ${requestId})`);+ expect(deactivate?.exit).toMatch(/^the next automatic expired-archive purge finishes it/);+ expect(deactivate?.exit).not.toContain('ask the user');+ const graduation = (await graduationBlockers(engine, localHostId())).find(b => b.kind === 'topology_recovery');+ expect(graduation).toMatchObject({ needsUser: false, argv: ['gbrain', 'sources', 'writer', 'status', source, '--json'] });+ expect(graduation?.detail).toContain('the next automatic expired-archive purge finishes it');+ expect(await runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true, expiredOnly: true, expectedIncarnation: first })).toMatchObject({ state: 'committed' });+ expect(await change(requestId)).toMatchObject({ state: 'failed' });+ expect((await deactivationBlockers(engine)).filter(b => b.kind === 'topology_recovery')).toEqual([]);+}), 60_000);++test('the superseded refusal points at a writer status that lists the stopped and superseded requests', () => fixture(SOURCE_RETIRE_BATCH_PAGES + 3, async root => {+ const requestId = randomUUID();+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });+ await expect(failFinalTransaction(remove)).rejects.toThrow('injected final retire failure');+ const status = async () => (await runPersistenceAdministration(engine, 'writer_status', { source_id: source }) as { topology_requests: unknown[] }).topology_requests;+ expect(await status()).toEqual([expect.objectContaining({ request_id: requestId, operation: 'remove', state: 'recovering' })]);+ await unmanaged(async () => {+ await engine.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1', [source]);+ await engine.executeRaw('DELETE FROM sources WHERE id=$1', [source]);+ });+ await readd(root, 2);+ const refusal = await remove().then(() => null, (e: { code?: string; fix?: { argv?: string[] } }) => e);+ expect(refusal).toMatchObject({ code: 'source_retire_superseded', fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', source, '--json'] } });+ expect(await status()).toEqual([expect.objectContaining({ request_id: requestId, operation: 'remove', state: 'failed', error: 'source_retire_superseded' })]);+}), 60_000);