Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
A restore or other change to a source abandons its stopped remove or purge, and blockers ask the user before finishing one
garrytanbdae8277da5d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1d56e1a333..2bc00857c4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,15 +12,15 @@ identifiers and attribution are available in the pre-removal Git revision## [Unreleased]-**Retrying an interrupted `gbrain sources remove` or `purge` can no longer delete a re-added source's pages.** A managed remove or purge deletes a large source's pages in batches, and a run that stops partway can be finished by rerunning it with the same `--request-id`. Until now nothing tied that request id to the source it was started against: if the source was removed and added again under the same ID in between, the rerun deleted the new source's pages. The first batch now records which incarnation of the source the request was admitted against, in the same transaction as its first deletion. A rerun against the same incarnation still resumes and finishes; a rerun after the source was replaced refuses with `source_retire_superseded` and changes nothing.+**Retrying an interrupted `gbrain sources remove` or `purge` can no longer delete a re-added source's pages.** A managed remove or purge deletes a large source's pages in batches, and a run that stops partway can be finished by rerunning it with the same `--request-id`. Until now nothing tied that request id to the source it was started against: if the source was removed and added again under the same ID in between, the rerun deleted the new source's pages. The first batch now records which incarnation of the source the request was admitted against, in the same transaction as its first deletion. A rerun against the same incarnation still resumes and finishes; a rerun after the source was replaced refuses with `source_retire_superseded` and changes nothing. Restoring, archiving or otherwise changing the source after a run stopped abandons that run the same way, so a purge stopped before a restore can't hold up `deactivate` or graduation for good, and both ask the user before a stopped run is finished, since finishing deletes the rest of the source's pages.### Itemized changes-- `src/core/persistence/topology-receipts.ts`: `admitTopologyRetire` records a retire admission (a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation, holding one lifetime ID and a full receipt reservation after the same quota check `assertTopologyChangeCapacity` made); `recordTopologyChange` settles a request's own admission `committed` and trues up its receipt bytes; `supersedeTopologyRetires` settles other requests' admissions `failed` when a remove or purge deletes their incarnation.+- `src/core/persistence/topology-receipts.ts`: `admitTopologyRetire` records a retire admission (a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation, holding one lifetime ID and a full receipt reservation after the same quota check `assertTopologyChangeCapacity` made); `recordTopologyChange` settles a request's own admission `committed` and trues up its receipt bytes; `supersedeTopologyRetires` settles other requests' admissions `failed` when a remove or purge deletes their incarnation, or when any other lifecycle operation (archive, restore, rebind, claim) commits a change to it.- `src/core/persistence/source-lifecycle.ts`: a request id whose admission names another incarnation than the source registered now refuses `source_retire_superseded` before any lock or deletion (a dry run too) and settles the admission `failed`, so a later retry refuses the same way; the batches and the final transaction resume past their own admission (#8984).- `src/core/error-registry.ts`: new caller code `source_retire_superseded`, with a read-only writer-status fix.-- `src/core/persistence/deactivation.ts`, `graduation-drain.ts`: an unfinished retire is listed as a blocker that names its request id and how to finish it.-- Tests: `test/source-retire-incarnation.test.ts` (PGLite) and `test/e2e/source-retire-incarnation-postgres.test.ts`: a retry after the final transaction failed, and a retry after committed batches already deleted part of the source, both refuse once the source was replaced and keep the new source's pages (both resolved `committed` on master); a same-incarnation retry resumes and settles. `test/source-retire-batched.test.ts` and its Postgres twin now expect the interrupted run's admission row.+- `src/core/persistence/deactivation.ts`, `graduation-drain.ts`: an unfinished retire is listed as a blocker that names its request id and asks the user whether to finish it (rerun the same command with its request id) or keep what is left (another lifecycle change abandons it); graduation marks it `needsUser`.+- Tests: `test/source-retire-incarnation.test.ts` (PGLite) and `test/e2e/source-retire-incarnation-postgres.test.ts`: a retry after the final transaction failed, and a retry after committed batches already deleted part of the source, both refuse once the source was replaced and keep the new source's pages (both resolved `committed` on master); a same-incarnation retry resumes and settles; a restore after an interrupted purge abandons it, so its retry refuses and no blocker stays. `test/source-retire-batched.test.ts` and its Postgres twin now expect the interrupted run's admission row.## [0.60.629.0] - 2026-11-09diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.mdindex 38d9479c58..78fd7f03a4 100644--- a/docs/architecture/key-files/core-persistence-continued.md+++ b/docs/architecture/key-files/core-persistence-continued.md@@ -41,4 +41,4 @@ The #6188 fence check and fence repair live in [fence repair](core-fence-repair.- `src/core/postgres-engine/reserved-transactions.ts` with `isPersistenceDeclaration` (`src/core/persistence/protocol.ts`) — BEGIN pipelining. `isPersistenceDeclaration` (GBRA-156) accepts exactly the `protocol.ts` declarations (transaction-local `set_config` calls and the brain row FOR SHARE). Admission, the FIFO claim, the effect claims and source guards, the receipt true-up, the single publication (`coordinator.ts`) and the consumer's lent lane transactions (`consumer.ts` `onLane`, through `reservedTransactions`) pass it as `TransactionOptions.pipelineFirst`, so behind a transaction-mode pooler the declaration goes out with BEGIN (one round trip less per transaction) while every later statement still waits for BEGIN's answer; if BEGIN fails, only the declaration has run, in autocommit, where it changes nothing. A reserved transaction gates every statement its body can reach, savepoint and nested-transaction bodies and `file()` included. A sync lane group's publication passes none: its begun mark must follow BEGIN's answer (`sync-lanes.ts` `awaitLaneBegin`). Tests: `test/persistence-pipeline-first.test.ts` (the predicate and the reviewed call sites), `test/e2e/pgbouncer-transaction-pipeline-6930.test.ts`.- `src/core/links-fence.ts` + `src/core/links-fence-projection.ts` — the `## Links` fence (G5): the markdown form of authored (manual) edges, `<!--- gbrain:links:begin -->` / `end` around `| to | type | source | valid_from | valid_until | context |`, one row per (to, type, source), source defaulting to `manual`; a reconciliation-managed provenance (`MANAGED_LINK_SOURCES`, exported here and re-exported by `ops/links.ts`), a bad date or a duplicate key is a skipped row with a warning. `parseLinksFence` (state `absent` / `parsed` / `malformed`), the deterministic `renderLinksFence` (rows sorted by key), `upsertLinksFenceRow` (omitted context and dates keep the stored ones; identical row = unchanged body), `removeLinksFenceRows` (leaves an empty fence), `replaceOrInsertLinksFence` (a `## Links` section at the end of compiled_truth, above an unclosed code block) and `maskLinksFence`, which `extractBodyLinks` applies so the fence cells are never also extracted as markdown, wikilink or bare-slug edges. A rewrite of a fence that parses with warnings throws `LinksFenceRewriteRefusal`. Cells use `escapeFenceCell` plus the same one-level marker encoding for `gbrain:links:*`. `projectLinksFence(tx, page, rows)` reconciles only what the page's fence owns: edges with origin_page_id = the page and origin_field `links_fence` (deleted, context-updated, inserted by key) and `manual` transitions with origin_page_id = the page, then refreshes the touched relationships; a row whose target is not a live page in the source yields no edge. `linksFenceProjectionRows` returns null for an absent or malformed fence, so such a body changes no edge (an absent fence retires nothing, as for facts; unlike facts no held generation is recorded, so after a hand deletion of the whole section the database keeps edges git no longer has until `remove_link` or the G3 indexer), and drops rows typed with a link type the active pack does not declare. Callers: `prepareCanonicalProjections` (every projected page publication, quarantined and segmented-unread bodies excepted), `prepareAutomaticLinks` (put_page auto-links and the managed links maintenance, so `extract --stale` restores a lost fence edge) and `persistence/manual-links.ts`. Database-only manual edges (origin_field NULL) are never touched by any of them. Pinned by `test/links-fence.test.ts` and `test/managed-links-fence.test.ts`.-- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt other than this request's own admission, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`, `lockTopologyPrincipal`, and on the first batch `admitTopologyRetire` from `topology-receipts.ts`, which checks the receipt quotas at the largest receipt and records the admission) and, when one would refuse or replay, stops without deleting so the final transaction reports it, or throws that refusal before deleting; a batch that fails `write_pending` is retried at half the size (floor 1), so heavy pages still fit the bound; a refusal after some batches committed says how many pages that run already deleted; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. #8984: the first batch's transaction records the admission, a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation it deletes from (`isRetireAdmission`); `recordTopologyChange` settles it `committed`, and a remove or purge that deletes that incarnation under another request settles it `failed` (`supersedeTopologyRetires`). A retry of the request_id against another incarnation (the source was removed and re-added) refuses `source_retire_superseded`, settles the admission `failed` and deletes nothing; deactivation and graduation list an unsettled admission as a blocker that names its request_id. Tests: `test/source-retire-batched.test.ts` (PGLite), `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages), `test/source-retire-incarnation.test.ts` and `test/e2e/source-retire-incarnation-postgres.test.ts`.+- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt other than this request's own admission, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`, `lockTopologyPrincipal`, and on the first batch `admitTopologyRetire` from `topology-receipts.ts`, which checks the receipt quotas at the largest receipt and records the admission) and, when one would refuse or replay, stops without deleting so the final transaction reports it, or throws that refusal before deleting; a batch that fails `write_pending` is retried at half the size (floor 1), so heavy pages still fit the bound; a refusal after some batches committed says how many pages that run already deleted; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. #8984: the first batch's transaction records the admission, a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation it deletes from (`isRetireAdmission`); `recordTopologyChange` settles it `committed`, and a remove or purge that deletes that incarnation under another request, or any other lifecycle operation that commits a change to it (archive, restore, rebind, claim), settles it `failed` (`supersedeTopologyRetires`). A retry of the request_id against another incarnation (the source was removed and re-added) refuses `source_retire_superseded`, settles the admission `failed` and deletes nothing; deactivation and graduation list an unsettled admission as a blocker that names its request_id and asks the user whether to finish it or keep what is left. Tests: `test/source-retire-batched.test.ts` (PGLite), `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages), `test/source-retire-incarnation.test.ts` and `test/e2e/source-retire-incarnation-postgres.test.ts`.diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.mdindex b908e1eb35..1603e04ba3 100644--- a/docs/guides/error-codes.md+++ b/docs/guides/error-codes.md@@ -4624,7 +4624,7 @@ More: [docs/guides/write-refusals.md#slug_page_mismatch](../../docs/guides/write| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable ||---|---|---|---|---|---|---|-| A retried source remove or purge was admitted against an earlier incarnation of its source, which has since been removed or replaced; nothing was changed. | A remove or purge records the source incarnation it was admitted against before it deletes its first page, so a retry of its request id can never delete the pages of a source re-added under the same ID. A request refused this way stays refused. | Do not retry this request id. Read the source's current state; removing the current source is a new decision for the user, made with a new request id. Run: gbrain sources writer status --source '{source_id}' --json | agent | `repeat the read that failed` | 1 | no |+| A retried source remove or purge was superseded: its source has since been removed, replaced or changed by another lifecycle operation; nothing was changed. | A remove or purge records the source incarnation it was admitted against before it deletes its first page, so a retry of its request id can never delete the pages of a source re-added under the same ID, and a later lifecycle change to the source (such as a restore) abandons the request. A request refused this way stays refused. | Do not retry this request id. Read the source's current state; removing the current source is a new decision for the user, made with a new request id. Run: gbrain sources writer status --source '{source_id}' --json | agent | `repeat the read that failed` | 1 | no |### source_root_conflictdiff --git a/src/core/error-registry.ts b/src/core/error-registry.tsindex c27c30a216..3894f22a40 100644--- a/src/core/error-registry.ts+++ b/src/core/error-registry.ts@@ -751,7 +751,7 @@ export const CODES = {source_not_ready: { class: 'caller', summary: "A committed source and approved schema are required." },source_profile_no_backfill: { class: 'unavailable', summary: "This source profile disables automatic embedding backfill." },source_referenced: { class: 'caller', summary: "OAuth clients still reference this source." },- source_retire_superseded: { class: 'caller', summary: "A retried source remove or purge was admitted against an earlier incarnation of its source, which has since been removed or replaced; nothing was changed.", why: "A remove or purge records the source incarnation it was admitted against before it deletes its first page, so a retry of its request id can never delete the pages of a source re-added under the same ID. A request refused this way stays refused.", suggestion: "Do not retry this request id. Read the source's current state; removing the current source is a new decision for the user, made with a new request id.", fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', '{source_id}', '--json'], consent: [], actor: 'agent', why: "Read-only: shows the source's current incarnation, binding and pending work.", requires_exclusive: false } },+ source_retire_superseded: { class: 'caller', summary: "A retried source remove or purge was superseded: its source has since been removed, replaced or changed by another lifecycle operation; nothing was changed.", why: "A remove or purge records the source incarnation it was admitted against before it deletes its first page, so a retry of its request id can never delete the pages of a source re-added under the same ID, and a later lifecycle change to the source (such as a restore) abandons the request. A request refused this way stays refused.", suggestion: "Do not retry this request id. Read the source's current state; removing the current source is a new decision for the user, made with a new request id.", fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', '{source_id}', '--json'], consent: [], actor: 'agent', why: "Read-only: shows the source's current incarnation, binding and pending work.", requires_exclusive: false } },source_root_conflict: { class: 'caller', summary: "Multiple active sources alias this canonical skill root; resolve ownership before publishing." },source_writeback_required: { class: 'host_only', summary: "Canonical preparation requires a source-content correction; this profile never writes repository files." },sources_not_readable: { class: 'host_only', retryable: false, summary: "An unscoped search, query, recall or list_pages came back empty and was refused: every source this connection can read (the default source and federated sources) holds no page (for recall and query, no world fact either), while other sources on the brain hold pages it cannot read (not federated, and no source grant).", why: "Answering would return an empty result that reads like a clean miss. The refusal names the host command that federates a source or grants this connection the sources it should read; it never names a hidden source and never widens access. A scope that holds pages but matched nothing, a granted token, an OAuth client, a bound connection and a read that names a source are never refused.", docs: 'docs/guides/multi-source-brains.md#an-unscoped-connection-whose-sources-hold-no-pages', suggestion: 'Ask the user which source these connections should read; the brain host runs `gbrain sources federate <source>`, or grants the token its sources with `gbrain auth rescope --token <name> --sources <ids>`.' },diff --git a/src/core/persistence/deactivation.ts b/src/core/persistence/deactivation.tsindex 66023a96f4..48afd5edcd 100644--- a/src/core/persistence/deactivation.ts+++ b/src/core/persistence/deactivation.ts@@ -118,7 +118,7 @@ export async function deactivationBlockers(engine: BrainEngine): Promise<Deactiv"SELECT id::text AS id, operation, source_id, request_id::text AS request_id, (state='recovering' AND recovery IS NULL) AS admission FROM persistence_topology_changes WHERE state='recovering' OR recovery IS NOT NULL ORDER BY created_at LIMIT 20");for (const t of topology) blockers.push({ kind: 'topology_recovery', id: t.id, source_id: t.source_id,detail: t.admission ? `${t.operation} of ${t.source_id} (request ${t.request_id}) stopped before it finished` : `${t.operation} for ${t.source_id} is recovering`,- exit: t.admission ? `rerun gbrain sources ${t.operation} ${t.source_id} --confirm-destructive --request-id ${t.request_id} (the user already confirmed it; it resumes, or refuses with source_retire_superseded if the source was replaced), then rerun deactivate`+ exit: t.admission ? `ask the user whether to finish it, because finishing deletes the rest of ${t.source_id}'s pages: to finish, rerun the same gbrain sources ${t.operation} command with all its original flags and --request-id ${t.request_id} (an automatic expired-archive purge finishes on its next run); to keep what is left, any other lifecycle change to the source (gbrain sources restore of an archived source, or archive then restore) abandons the request; then rerun deactivate`: `gbrain sources writer status ${t.source_id} --json, then let the owner finish recovery` });// GBRA-130 A-15: only work that can still change blocks, as activation checks; terminal failures are failedEffectHistory's.const effects = await engine.executeRaw<EffectRow>(`${EFFECT_ROWS} WHERE e.state IN ('queued','running') OR e.recovery IS NOT NULL ORDER BY e.id LIMIT 50`);diff --git a/src/core/persistence/graduation-drain.ts b/src/core/persistence/graduation-drain.tsindex 47a6d3714e..c3328ae42e 100644--- a/src/core/persistence/graduation-drain.ts+++ b/src/core/persistence/graduation-drain.ts@@ -62,9 +62,9 @@ export async function graduationBlockers(source: BrainEngine, hostId: string): P"SELECT id::text AS id, operation, source_id, request_id::text AS request_id, (state = 'recovering' AND recovery IS NULL) AS admission FROM persistence_topology_changes WHERE state = 'recovering' OR recovery IS NOT NULL ORDER BY created_at");for (const t of topology) {blockers.push({ kind: 'topology_recovery', id: t.id, detail: t.admission- ? `${t.operation} of ${t.source_id} (request ${t.request_id}) stopped before it finished; rerun it with --request-id ${t.request_id} to finish it`+ ? `${t.operation} of ${t.source_id} (request ${t.request_id}) stopped before it finished; ask the user whether to finish it (rerun the same command with all its original flags and --request-id ${t.request_id}, which deletes the rest of its pages) or keep what is left (any other lifecycle change to the source, such as a restore, abandons the request)`: `${t.operation} for ${t.source_id} is recovering`,- argv: ['gbrain', 'sources', 'writer', 'status', t.source_id, '--json'], needsUser: false });+ argv: ['gbrain', 'sources', 'writer', 'status', t.source_id, '--json'], needsUser: t.admission });}const effects = await source.executeRaw<{ id: string; kind: string; state: string; error_code: string | null; source_id: string | null; request_id: string }>(`SELECT e.id::text AS id, e.kind, e.state, e.error_code, COALESCE(e.source_id, r.source_id) AS source_id, r.request_id::text AS request_iddiff --git a/src/core/persistence/source-lifecycle.ts b/src/core/persistence/source-lifecycle.tsindex 77061b917a..2ce92ee813 100644--- a/src/core/persistence/source-lifecycle.ts+++ b/src/core/persistence/source-lifecycle.ts@@ -54,10 +54,10 @@ function ownerStatusFix(sourceId:string):Action{{argv:['gbrain','sources','writer','status','--source',sourceId,'--json']});}-/** #8984: this request_id was admitted against an incarnation of the source that has since been removed or replaced. */+/** #8984: the source this request_id was admitted against has since been removed, replaced or changed by another lifecycle operation. */function retireSuperseded(row:TopologyChange):OperationError{- return opError(RETIRE_SUPERSEDED,`This ${row.operation} request was admitted against an earlier incarnation of source '${row.source_id}', which has since been removed or replaced.`,- `Nothing was changed: request ${row.request_id} can only act on incarnation ${row.source_incarnation} of '${row.source_id}', so retrying it can never delete the pages of the source registered under that ID now. Do not retry it. Read the source's current state with the command in fix; removing the current source is a new decision for the user, made with a new request id.`,+ return opError(RETIRE_SUPERSEDED,`This ${row.operation} request was superseded: source '${row.source_id}' has since been removed, replaced or changed by another lifecycle operation.`,+ `Nothing was changed: request ${row.request_id} can only act on incarnation ${row.source_incarnation} of '${row.source_id}' as it stood when the request was admitted, so retrying it can never delete pages the source holds now. Do not retry it. Read the source's current state with the command in fix; removing the current source is a new decision for the user, made with a new request id.`,{fix:ownerStatusFix(row.source_id)});}/** A prior receipt of this request_id, replayed; a retire admission (#8984) returns null so the run resumes. */@@ -341,6 +341,7 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceLawait tx.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1 AND source_incarnation=$2::uuid',[input.sourceId,incarnation]);await supersedeTopologyRetires(tx,input.sourceId,incarnation,requestId);}+ if(source&&!['remove','purge'].includes(input.operation)) await supersedeTopologyRetires(tx,input.sourceId,source.incarnation,requestId);const contentReceipts=input.operation==='rebind'?await rebindContentReceipts(tx,worktrees,hostIdFor(tx),relocated?undefined:[input.sourceId]):[];await advanceTopology(tx,worktrees);for(const id of worktrees){diff --git a/test/source-retire-incarnation.test.ts b/test/source-retire-incarnation.test.tsindex da1029aa70..ebb609eab1 100644--- a/test/source-retire-incarnation.test.ts+++ b/test/source-retire-incarnation.test.ts@@ -17,6 +17,8 @@ import { claimWorktree } from '../src/core/persistence/ownership.ts';import { runManagedSourceLifecycle, SOURCE_RETIRE_BATCH_PAGES } from '../src/core/persistence/source-lifecycle.ts';import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';import { deactivationBlockers } from '../src/core/persistence/deactivation.ts';+import { graduationBlockers } from '../src/core/persistence/graduation-drain.ts';+import { localHostId } from '../src/core/persistence/identity.ts';let engine: PGLiteEngine;beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); });@@ -61,6 +63,20 @@ async function failFinalTransaction<T>(run: () => Promise<T>): Promise<T> {await engine.executeRaw('DROP FUNCTION IF EXISTS retire_final_fn()');}}+/** Arms a failure of the second page batch, so exactly one batch of SOURCE_RETIRE_BATCH_PAGES pages commits. */+async function failAfterFirstBatch<T>(run: () => Promise<T>): Promise<T> {+ await engine.executeRaw('CREATE TABLE retire_fail(armed boolean NOT NULL, seen integer NOT NULL)');+ await engine.executeRaw(`CREATE FUNCTION retire_fail_fn() RETURNS trigger LANGUAGE plpgsql AS $$ DECLARE n integer; BEGIN+ UPDATE retire_fail SET seen=seen+1 WHERE armed RETURNING seen INTO n;+ IF n > ${SOURCE_RETIRE_BATCH_PAGES} THEN RAISE EXCEPTION 'injected retire batch failure'; END IF; RETURN OLD; END $$`);+ await engine.executeRaw('CREATE TRIGGER retire_fail BEFORE DELETE ON pages FOR EACH ROW EXECUTE FUNCTION retire_fail_fn()');+ try {+ await engine.executeRaw('INSERT INTO retire_fail VALUES(true,0)');+ return await run();+ } finally {+ for (const sql of ['DROP TRIGGER IF EXISTS retire_fail ON pages', 'DROP FUNCTION IF EXISTS retire_fail_fn()', 'DROP TABLE IF EXISTS retire_fail']) await engine.executeRaw(sql);+ }+}/** Raw page writes stand in for an import; the managed guard admits them only while the brain is briefly unmanaged. */async function unmanaged(run: () => Promise<unknown>) {await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');@@ -88,35 +104,40 @@ test('a retry after the source was removed and re-added refuses and keeps the ne}), 60_000);test('a batched retry after partial deletion refuses once the source was replaced out of band, and settles its record', () => fixture(SOURCE_RETIRE_BATCH_PAGES * 2 + 17, async root => {- await engine.executeRaw('CREATE TABLE retire_fail(armed boolean NOT NULL, seen integer NOT NULL)');- await engine.executeRaw(`CREATE FUNCTION retire_fail_fn() RETURNS trigger LANGUAGE plpgsql AS $$ DECLARE n integer; BEGIN- UPDATE retire_fail SET seen=seen+1 WHERE armed RETURNING seen INTO n;- IF n > ${SOURCE_RETIRE_BATCH_PAGES} THEN RAISE EXCEPTION 'injected retire batch failure'; END IF; RETURN OLD; END $$`);- await engine.executeRaw('CREATE TRIGGER retire_fail BEFORE DELETE ON pages FOR EACH ROW EXECUTE FUNCTION retire_fail_fn()');- try {- await engine.executeRaw('INSERT INTO retire_fail VALUES(true,0)');- const requestId = randomUUID();- const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });- const first = await incarnation();- await expect(remove()).rejects.toThrow('injected retire batch failure');- await engine.executeRaw('DELETE FROM retire_fail');- expect(await pages()).toBe(SOURCE_RETIRE_BATCH_PAGES + 17);- expect(await change(requestId)).toMatchObject({ state: 'recovering', source_incarnation: first });- await unmanaged(async () => {- await engine.executeRaw('DELETE FROM pages WHERE source_id=$1', [source]);- await engine.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1', [source]);- await engine.executeRaw('DELETE FROM sources WHERE id=$1', [source]);- });- await readd(root, 3);- await expect(remove()).rejects.toMatchObject({ code: 'source_retire_superseded' });- expect(await change(requestId)).toMatchObject({ state: 'failed', source_incarnation: first });- await expect(remove()).rejects.toMatchObject({ code: 'source_retire_superseded' });- await expect(runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId, dryRun: true }))- .rejects.toMatchObject({ code: 'source_retire_superseded' });- expect(await pages()).toBe(3);- } finally {- for (const sql of ['DROP TRIGGER IF EXISTS retire_fail ON pages', 'DROP FUNCTION IF EXISTS retire_fail_fn()', 'DROP TABLE IF EXISTS retire_fail']) await engine.executeRaw(sql);- }+ const requestId = randomUUID();+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });+ const first = await incarnation();+ await expect(failAfterFirstBatch(remove)).rejects.toThrow('injected retire batch failure');+ expect(await pages()).toBe(SOURCE_RETIRE_BATCH_PAGES + 17);+ expect(await change(requestId)).toMatchObject({ state: 'recovering', source_incarnation: first });+ await unmanaged(async () => {+ await engine.executeRaw('DELETE FROM pages WHERE source_id=$1', [source]);+ await engine.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1', [source]);+ await engine.executeRaw('DELETE FROM sources WHERE id=$1', [source]);+ });+ await readd(root, 3);+ await expect(remove()).rejects.toMatchObject({ code: 'source_retire_superseded' });+ expect(await change(requestId)).toMatchObject({ state: 'failed', source_incarnation: first });+ await expect(remove()).rejects.toMatchObject({ code: 'source_retire_superseded' });+ await expect(runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId, dryRun: true }))+ .rejects.toMatchObject({ code: 'source_retire_superseded' });+ expect(await pages()).toBe(3);+}), 60_000);++test('a restore after an interrupted purge abandons the purge: its retry refuses and nothing blocks deactivate or graduation', () => fixture(SOURCE_RETIRE_BATCH_PAGES * 2, async () => {+ expect(await runManagedSourceLifecycle(engine, { operation: 'archive', sourceId: source })).toMatchObject({ state: 'committed' });+ const requestId = randomUUID();+ const purge = () => runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true, requestId });+ const first = await incarnation();+ await expect(failAfterFirstBatch(purge)).rejects.toThrow('injected retire batch failure');+ expect(await pages()).toBe(SOURCE_RETIRE_BATCH_PAGES);+ expect(await runManagedSourceLifecycle(engine, { operation: 'restore', sourceId: source })).toMatchObject({ state: 'committed' });+ expect(await change(requestId)).toMatchObject({ state: 'failed', source_incarnation: first });+ for (let i = 0; i < 2; i++) await expect(purge()).rejects.toMatchObject({ code: 'source_retire_superseded' });+ expect(await pages()).toBe(SOURCE_RETIRE_BATCH_PAGES);+ expect(await incarnation()).toBe(first);+ expect((await deactivationBlockers(engine)).filter(b => b.kind === 'topology_recovery')).toEqual([]);+ expect((await graduationBlockers(engine, localHostId())).filter(b => b.kind === 'topology_recovery')).toEqual([]);}), 60_000);test('a retry against the same incarnation resumes, finishes and settles its record committed', () => fixture(SOURCE_RETIRE_BATCH_PAGES + 3, async () => {@@ -128,7 +149,10 @@ test('a retry against the same incarnation resumes, finishes and settles its recexpect(await pages()).toBe(0);expect(await change(requestId)).toMatchObject({ state: 'recovering', source_incarnation: first });expect((await deactivationBlockers(engine)).find(b => b.kind === 'topology_recovery')).toMatchObject({- detail: `remove of ${source} (request ${requestId}) stopped before it finished`, exit: expect.stringContaining(`--request-id ${requestId}`) });+ detail: `remove of ${source} (request ${requestId}) stopped before it finished`,+ exit: expect.stringMatching(new RegExp(`^ask the user .*--request-id ${requestId}`)) });+ expect((await graduationBlockers(engine, localHostId())).find(b => b.kind === 'topology_recovery')).toMatchObject({+ argv: ['gbrain', 'sources', 'writer', 'status', source, '--json'], needsUser: true });const receipt = await remove();expect(receipt).toMatchObject({ state: 'committed', operation: 'remove', source_incarnation: first });expect(await remove()).toEqual(receipt);