Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: a retried source remove never deletes a re-added source's pages
garrytan7db21e689e2e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 24dbecea03..1d56e1a333 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,18 @@ credits are retained; no result has been reassigned to another provider. Originaidentifiers and attribution are available in the pre-removal Git revision`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).+## [Unreleased]++**Retrying an interrupted `gbrain sources remove` or `purge` can no longer delete a re-added source's pages.** A managed remove or purge deletes a large source's pages in batches, and a run that stops partway can be finished by rerunning it with the same `--request-id`. Until now nothing tied that request id to the source it was started against: if the source was removed and added again under the same ID in between, the rerun deleted the new source's pages. The first batch now records which incarnation of the source the request was admitted against, in the same transaction as its first deletion. A rerun against the same incarnation still resumes and finishes; a rerun after the source was replaced refuses with `source_retire_superseded` and changes nothing.++### Itemized changes++- `src/core/persistence/topology-receipts.ts`: `admitTopologyRetire` records a retire admission (a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation, holding one lifetime ID and a full receipt reservation after the same quota check `assertTopologyChangeCapacity` made); `recordTopologyChange` settles a request's own admission `committed` and trues up its receipt bytes; `supersedeTopologyRetires` settles other requests' admissions `failed` when a remove or purge deletes their incarnation.+- `src/core/persistence/source-lifecycle.ts`: a request id whose admission names another incarnation than the source registered now refuses `source_retire_superseded` before any lock or deletion (a dry run too) and settles the admission `failed`, so a later retry refuses the same way; the batches and the final transaction resume past their own admission (#8984).+- `src/core/error-registry.ts`: new caller code `source_retire_superseded`, with a read-only writer-status fix.+- `src/core/persistence/deactivation.ts`, `graduation-drain.ts`: an unfinished retire is listed as a blocker that names its request id and how to finish it.+- Tests: `test/source-retire-incarnation.test.ts` (PGLite) and `test/e2e/source-retire-incarnation-postgres.test.ts`: a retry after the final transaction failed, and a retry after committed batches already deleted part of the source, both refuse once the source was replaced and keep the new source's pages (both resolved `committed` on master); a same-incarnation retry resumes and settles. `test/source-retire-batched.test.ts` and its Postgres twin now expect the interrupted run's admission row.+## [0.60.629.0] - 2026-11-09**`gbrain repair file-drift` now agrees with sync on a held origin-twin page.** A legacy page left as an origin twin with no recorded file (no import record) could be held `file_drift` by sync while `repair file-drift --slug <slug>` reported `no_drift`. The hold's own suggested fix couldn't clear it, and scripts that use the preview as a drift check passed falsely. The preview now reads such a twin against its slug-path file, the same way sync imports it, and lists it with an apply command.diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.mdindex d4dc45df41..38d9479c58 100644--- a/docs/architecture/key-files/core-persistence-continued.md+++ b/docs/architecture/key-files/core-persistence-continued.md@@ -41,4 +41,4 @@ The #6188 fence check and fence repair live in [fence repair](core-fence-repair.- `src/core/postgres-engine/reserved-transactions.ts` with `isPersistenceDeclaration` (`src/core/persistence/protocol.ts`) — BEGIN pipelining. `isPersistenceDeclaration` (GBRA-156) accepts exactly the `protocol.ts` declarations (transaction-local `set_config` calls and the brain row FOR SHARE). Admission, the FIFO claim, the effect claims and source guards, the receipt true-up, the single publication (`coordinator.ts`) and the consumer's lent lane transactions (`consumer.ts` `onLane`, through `reservedTransactions`) pass it as `TransactionOptions.pipelineFirst`, so behind a transaction-mode pooler the declaration goes out with BEGIN (one round trip less per transaction) while every later statement still waits for BEGIN's answer; if BEGIN fails, only the declaration has run, in autocommit, where it changes nothing. A reserved transaction gates every statement its body can reach, savepoint and nested-transaction bodies and `file()` included. A sync lane group's publication passes none: its begun mark must follow BEGIN's answer (`sync-lanes.ts` `awaitLaneBegin`). Tests: `test/persistence-pipeline-first.test.ts` (the predicate and the reviewed call sites), `test/e2e/pgbouncer-transaction-pipeline-6930.test.ts`.- `src/core/links-fence.ts` + `src/core/links-fence-projection.ts` — the `## Links` fence (G5): the markdown form of authored (manual) edges, `<!--- gbrain:links:begin -->` / `end` around `| to | type | source | valid_from | valid_until | context |`, one row per (to, type, source), source defaulting to `manual`; a reconciliation-managed provenance (`MANAGED_LINK_SOURCES`, exported here and re-exported by `ops/links.ts`), a bad date or a duplicate key is a skipped row with a warning. `parseLinksFence` (state `absent` / `parsed` / `malformed`), the deterministic `renderLinksFence` (rows sorted by key), `upsertLinksFenceRow` (omitted context and dates keep the stored ones; identical row = unchanged body), `removeLinksFenceRows` (leaves an empty fence), `replaceOrInsertLinksFence` (a `## Links` section at the end of compiled_truth, above an unclosed code block) and `maskLinksFence`, which `extractBodyLinks` applies so the fence cells are never also extracted as markdown, wikilink or bare-slug edges. A rewrite of a fence that parses with warnings throws `LinksFenceRewriteRefusal`. Cells use `escapeFenceCell` plus the same one-level marker encoding for `gbrain:links:*`. `projectLinksFence(tx, page, rows)` reconciles only what the page's fence owns: edges with origin_page_id = the page and origin_field `links_fence` (deleted, context-updated, inserted by key) and `manual` transitions with origin_page_id = the page, then refreshes the touched relationships; a row whose target is not a live page in the source yields no edge. `linksFenceProjectionRows` returns null for an absent or malformed fence, so such a body changes no edge (an absent fence retires nothing, as for facts; unlike facts no held generation is recorded, so after a hand deletion of the whole section the database keeps edges git no longer has until `remove_link` or the G3 indexer), and drops rows typed with a link type the active pack does not declare. Callers: `prepareCanonicalProjections` (every projected page publication, quarantined and segmented-unread bodies excepted), `prepareAutomaticLinks` (put_page auto-links and the managed links maintenance, so `extract --stale` restores a lost fence edge) and `persistence/manual-links.ts`. Database-only manual edges (origin_field NULL) are never touched by any of them. Pinned by `test/links-fence.test.ts` and `test/managed-links-fence.test.ts`.-- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`, `lockTopologyPrincipal`, and `assertTopologyChangeCapacity` from `topology-receipts.ts`, the receipt quotas at the largest receipt) and, when one would refuse or replay, stops without deleting so the final transaction reports it, or throws that refusal before deleting; a batch that fails `write_pending` is retried at half the size (floor 1), so heavy pages still fit the bound; a refusal after some batches committed says how many pages that run already deleted; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. Tests: `test/source-retire-batched.test.ts` (PGLite) and `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages).+- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt other than this request's own admission, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`, `lockTopologyPrincipal`, and on the first batch `admitTopologyRetire` from `topology-receipts.ts`, which checks the receipt quotas at the largest receipt and records the admission) and, when one would refuse or replay, stops without deleting so the final transaction reports it, or throws that refusal before deleting; a batch that fails `write_pending` is retried at half the size (floor 1), so heavy pages still fit the bound; a refusal after some batches committed says how many pages that run already deleted; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. #8984: the first batch's transaction records the admission, a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation it deletes from (`isRetireAdmission`); `recordTopologyChange` settles it `committed`, and a remove or purge that deletes that incarnation under another request settles it `failed` (`supersedeTopologyRetires`). A retry of the request_id against another incarnation (the source was removed and re-added) refuses `source_retire_superseded`, settles the admission `failed` and deletes nothing; deactivation and graduation list an unsettled admission as a blocker that names its request_id. Tests: `test/source-retire-batched.test.ts` (PGLite), `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages), `test/source-retire-incarnation.test.ts` and `test/e2e/source-retire-incarnation-postgres.test.ts`.diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.mdindex 917a8556ba..b908e1eb35 100644--- a/docs/guides/error-codes.md+++ b/docs/guides/error-codes.md@@ -4618,6 +4618,14 @@ More: [docs/guides/write-refusals.md#slug_page_mismatch](../../docs/guides/write|---|---|---|---|---|---|---|| OAuth clients still reference this source. | The request itself was wrong or no longer matches the brain; nothing was changed. | Correct the request using the message above, then retry. | agent | `repeat the read that failed` | 1 | no |+### source_retire_superseded++<a id="source_retire_superseded"></a>++| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |+|---|---|---|---|---|---|---|+| A retried source remove or purge was admitted against an earlier incarnation of its source, which has since been removed or replaced; nothing was changed. | A remove or purge records the source incarnation it was admitted against before it deletes its first page, so a retry of its request id can never delete the pages of a source re-added under the same ID. A request refused this way stays refused. | Do not retry this request id. Read the source's current state; removing the current source is a new decision for the user, made with a new request id. Run: gbrain sources writer status --source '{source_id}' --json | agent | `repeat the read that failed` | 1 | no |+### source_root_conflict<a id="source_root_conflict"></a>diff --git a/src/core/error-registry.ts b/src/core/error-registry.tsindex af6d21fed8..c27c30a216 100644--- a/src/core/error-registry.ts+++ b/src/core/error-registry.ts@@ -751,6 +751,7 @@ export const CODES = {source_not_ready: { class: 'caller', summary: "A committed source and approved schema are required." },source_profile_no_backfill: { class: 'unavailable', summary: "This source profile disables automatic embedding backfill." },source_referenced: { class: 'caller', summary: "OAuth clients still reference this source." },+ source_retire_superseded: { class: 'caller', summary: "A retried source remove or purge was admitted against an earlier incarnation of its source, which has since been removed or replaced; nothing was changed.", why: "A remove or purge records the source incarnation it was admitted against before it deletes its first page, so a retry of its request id can never delete the pages of a source re-added under the same ID. A request refused this way stays refused.", suggestion: "Do not retry this request id. Read the source's current state; removing the current source is a new decision for the user, made with a new request id.", fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', '{source_id}', '--json'], consent: [], actor: 'agent', why: "Read-only: shows the source's current incarnation, binding and pending work.", requires_exclusive: false } },source_root_conflict: { class: 'caller', summary: "Multiple active sources alias this canonical skill root; resolve ownership before publishing." },source_writeback_required: { class: 'host_only', summary: "Canonical preparation requires a source-content correction; this profile never writes repository files." },sources_not_readable: { class: 'host_only', retryable: false, summary: "An unscoped search, query, recall or list_pages came back empty and was refused: every source this connection can read (the default source and federated sources) holds no page (for recall and query, no world fact either), while other sources on the brain hold pages it cannot read (not federated, and no source grant).", why: "Answering would return an empty result that reads like a clean miss. The refusal names the host command that federates a source or grants this connection the sources it should read; it never names a hidden source and never widens access. A scope that holds pages but matched nothing, a granted token, an OAuth client, a bound connection and a read that names a source are never refused.", docs: 'docs/guides/multi-source-brains.md#an-unscoped-connection-whose-sources-hold-no-pages', suggestion: 'Ask the user which source these connections should read; the brain host runs `gbrain sources federate <source>`, or grants the token its sources with `gbrain auth rescope --token <name> --sources <ids>`.' },diff --git a/src/core/persistence/deactivation.ts b/src/core/persistence/deactivation.tsindex 049d094fd9..66023a96f4 100644--- a/src/core/persistence/deactivation.ts+++ b/src/core/persistence/deactivation.ts@@ -114,10 +114,12 @@ export async function deactivationBlockers(engine: BrainEngine): Promise<Deactiv: r.state === 'running' && !r.recovering ? 'wait for the owner to finish publishing it, then rerun deactivate': `gbrain sync --source ${r.source_id} --no-pull` });}- const topology = await engine.executeRaw<{ id: string; operation: string; source_id: string }>(- "SELECT id::text AS id, operation, source_id FROM persistence_topology_changes WHERE state='recovering' OR recovery IS NOT NULL ORDER BY created_at LIMIT 20");- for (const t of topology) blockers.push({ kind: 'topology_recovery', id: t.id, source_id: t.source_id, detail: `${t.operation} for ${t.source_id} is recovering`,- exit: `gbrain sources writer status ${t.source_id} --json, then let the owner finish recovery` });+ const topology = await engine.executeRaw<{ id: string; operation: string; source_id: string; request_id: string; admission: boolean }>(+ "SELECT id::text AS id, operation, source_id, request_id::text AS request_id, (state='recovering' AND recovery IS NULL) AS admission FROM persistence_topology_changes WHERE state='recovering' OR recovery IS NOT NULL ORDER BY created_at LIMIT 20");+ for (const t of topology) blockers.push({ kind: 'topology_recovery', id: t.id, source_id: t.source_id,+ detail: t.admission ? `${t.operation} of ${t.source_id} (request ${t.request_id}) stopped before it finished` : `${t.operation} for ${t.source_id} is recovering`,+ exit: t.admission ? `rerun gbrain sources ${t.operation} ${t.source_id} --confirm-destructive --request-id ${t.request_id} (the user already confirmed it; it resumes, or refuses with source_retire_superseded if the source was replaced), then rerun deactivate`+ : `gbrain sources writer status ${t.source_id} --json, then let the owner finish recovery` });// GBRA-130 A-15: only work that can still change blocks, as activation checks; terminal failures are failedEffectHistory's.const effects = await engine.executeRaw<EffectRow>(`${EFFECT_ROWS} WHERE e.state IN ('queued','running') OR e.recovery IS NOT NULL ORDER BY e.id LIMIT 50`);for (const e of effects) {diff --git a/src/core/persistence/graduation-drain.ts b/src/core/persistence/graduation-drain.tsindex 222f637d59..47a6d3714e 100644--- a/src/core/persistence/graduation-drain.ts+++ b/src/core/persistence/graduation-drain.ts@@ -58,10 +58,12 @@ export async function graduationBlockers(source: BrainEngine, hostId: string): Pblockers.push({ kind: 'request', id: r.request_id, detail, needsUser: false });}}- const topology = await source.executeRaw<{ id: string; operation: string; source_id: string }>(- "SELECT id::text AS id, operation, source_id FROM persistence_topology_changes WHERE state = 'recovering' OR recovery IS NOT NULL ORDER BY created_at");+ const topology = await source.executeRaw<{ id: string; operation: string; source_id: string; request_id: string; admission: boolean }>(+ "SELECT id::text AS id, operation, source_id, request_id::text AS request_id, (state = 'recovering' AND recovery IS NULL) AS admission FROM persistence_topology_changes WHERE state = 'recovering' OR recovery IS NOT NULL ORDER BY created_at");for (const t of topology) {- blockers.push({ kind: 'topology_recovery', id: t.id, detail: `${t.operation} for ${t.source_id} is recovering`,+ blockers.push({ kind: 'topology_recovery', id: t.id, detail: t.admission+ ? `${t.operation} of ${t.source_id} (request ${t.request_id}) stopped before it finished; rerun it with --request-id ${t.request_id} to finish it`+ : `${t.operation} for ${t.source_id} is recovering`,argv: ['gbrain', 'sources', 'writer', 'status', t.source_id, '--json'], needsUser: false });}const effects = await source.executeRaw<{ id: string; kind: string; state: string; error_code: string | null; source_id: string | null; request_id: string }>(diff --git a/src/core/persistence/source-lifecycle.ts b/src/core/persistence/source-lifecycle.tsindex f1060c3793..77061b917a 100644--- a/src/core/persistence/source-lifecycle.ts+++ b/src/core/persistence/source-lifecycle.ts@@ -17,7 +17,7 @@ import { containsPath, getWorktreeBinding, humanManifestProgress, type WorktreeBimport { manifestDifferences, storedManifestScope, type WorktreeManifestScope } from './worktree-manifest.ts';import { hostIdFor } from './identity.ts';import { advanceTopology, assertTopologyRequestsSettled, inspectTopologyPhysicalRoot, lockTopologyPrincipal, lockTopologyRows, settleTopologyRequests, topologyCanonicalStamp, topologyPrincipal, withTopologyLocks } from './topology-locks.ts';-import { assertTopologyChangeCapacity, priorTopologyChange, recordTopologyChange, topologyReceipt } from './topology-receipts.ts';+import { admitTopologyRetire, isRetireAdmission, isSupersededRetire, priorTopologyChange, RETIRE_SUPERSEDED, recordTopologyChange, settleTopologyRetire, supersedeTopologyRetires, topologyReceipt, type TopologyChange } from './topology-receipts.ts';import { isWriteRequestId } from './types.ts';import { withCoordinatedWrite } from './context.ts';import { principalAttribution } from './attribution.ts';@@ -54,6 +54,19 @@ function ownerStatusFix(sourceId:string):Action{{argv:['gbrain','sources','writer','status','--source',sourceId,'--json']});}+/** #8984: this request_id was admitted against an incarnation of the source that has since been removed or replaced. */+function retireSuperseded(row:TopologyChange):OperationError{+ return opError(RETIRE_SUPERSEDED,`This ${row.operation} request was admitted against an earlier incarnation of source '${row.source_id}', which has since been removed or replaced.`,+ `Nothing was changed: request ${row.request_id} can only act on incarnation ${row.source_incarnation} of '${row.source_id}', so retrying it can never delete the pages of the source registered under that ID now. Do not retry it. Read the source's current state with the command in fix; removing the current source is a new decision for the user, made with a new request id.`,+ {fix:ownerStatusFix(row.source_id)});+}+/** A prior receipt of this request_id, replayed; a retire admission (#8984) returns null so the run resumes. */+function replayTopologyChange(row:TopologyChange|null):Record<string,unknown>|null{+ if(!row||isRetireAdmission(row))return null;+ if(isSupersededRetire(row))throw retireSuperseded(row);+ return topologyReceipt(row);+}+function localRoot(path:string,create=false):{source:string;worktree:string}{if(!isAbsolute(path)||path.includes('\0')) throw opError('invalid_params','Source path must be an absolute directory on this host.','Pass the source directory as an absolute path on the brain host, then rerun the command; nothing was changed.');@@ -133,7 +146,8 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceLconst principal=await topologyPrincipal(engine);const intent={...input,requestId:undefined,dryRun:undefined,automaticClaim:undefined};const prior=await priorTopologyChange(engine,principal,requestId,intent);- if(prior) return topologyReceipt(prior);+ const replayed=replayTopologyChange(prior);+ if(replayed) return replayed;if(input.requireGitContent&&input.path&&(!isInsideGitRepo(input.path)||!hasTrackedContent(input.path)))throw opError('not_a_git_repo','The source path must contain committed Git content. Use --force to register an ordinary directory.',`Commit the content in that directory with git first, or, if '${input.sourceId}' is meant to be a plain directory, rerun the add with --force (sync then imports its files without Git; a file removed from it keeps its page); nothing was registered.`);@@ -146,6 +160,10 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceL`Pass the canonical directory for '${input.sourceId}' (the one the command in fix reports, or its verified successor), then rerun the ${input.operation}.`,{fix:ownerStatusFix(input.sourceId)});const [before]=await engine.executeRaw<SourceState>('SELECT id,incarnation,archived,local_path,config,name,last_commit FROM sources WHERE id=$1',[input.sourceId]);+ if(prior&&prior.source_incarnation!==before?.incarnation){+ if(!input.dryRun) await topologyTransaction(engine,async tx=>{await lockTopologyPrincipal(tx,principal);await settleTopologyRetire(tx,prior,'failed',{...prior.outcome,error:RETIRE_SUPERSEDED});});+ throw retireSuperseded(prior);+ }if(input.operation!=='add'&&!before) throw opError('not_found','Source not found.',`No source '${input.sourceId}' is registered on this brain. Check the ID against the command in fix, which lists archived sources too.`,{fix:sourcesListFix()});if(input.expectedIncarnation&&input.expectedIncarnation!==before?.incarnation) throw opError('source_changed','The source was replaced.',@@ -197,12 +215,14 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceLconst sources=await lockTopologyRows(tx,input.sourceId,bindings);const [source]=await tx.executeRaw<{incarnation:string;archived:boolean;due:boolean}>(`SELECT incarnation,archived,${input.expiredOnly?EXPIRED_ARCHIVE_SQL:'true'} AS due FROM sources WHERE id=$1`,[input.sourceId]);+ const admitted=await priorTopologyChange(tx,principal,requestId,intent);if(source?.incarnation!==before!.incarnation||!source.due||input.operation==='purge'&&!input.expiredOnly&&!source.archived- ||await priorTopologyChange(tx,principal,requestId,intent)+ ||admitted&&!(isRetireAdmission(admitted)&&admitted.source_incarnation===source.incarnation)||(await tx.executeRaw('SELECT client_id FROM oauth_clients WHERE source_id=$1 LIMIT 1',[input.sourceId])).length) return 0;await assertTopologyRequestsSettled(tx,sources,bindings.map(binding=>binding.worktree_id));await lockTopologyPrincipal(tx,principal);- await assertTopologyChangeCapacity(tx,{principal,operation:input.operation,sourceId:input.sourceId});+ if(!admitted) await admitTopologyRetire(tx,{principal,requestId,intent,operation:input.operation,sourceId:input.sourceId,+ incarnation:source.incarnation,worktrees:bindings.map(binding=>binding.worktree_id)});const gone=await withCoordinatedWrite(tx,sources,async()=>{deleting=true;const rows=await tx.executeRaw('DELETE FROM pages WHERE id IN (SELECT id FROM pages WHERE source_id=$1 LIMIT $2) RETURNING id',[input.sourceId,limit]);@@ -226,8 +246,8 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceLif(input.operation==='claim'&&!input.automaticClaim) await assertWriterAdminUnlocked(tx);const [source]=await tx.executeRaw<SourceState>('SELECT id,incarnation,archived,local_path,config,name,last_commit FROM sources WHERE id=$1',[input.sourceId]);if(source)source.config=parseSourceConfig(source.config);- const repeated=await priorTopologyChange(tx,principal,requestId,intent);- if(repeated){await lockTopologyPrincipal(tx,principal);return topologyReceipt(repeated);}+ const repeated=replayTopologyChange(await priorTopologyChange(tx,principal,requestId,intent));+ if(repeated){await lockTopologyPrincipal(tx,principal);return repeated;}if((source?.incarnation??null)!==(before?.incarnation??null)) throw opError('source_changed','The source changed during lifecycle preparation.',`Source '${input.sourceId}' was replaced while this ${input.operation} was being prepared; the transaction rolled back and nothing changed. Check its current state with the command in fix before running the ${input.operation} again.`,{fix:ownerStatusFix(input.sourceId)});@@ -319,6 +339,7 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceLpagesDeleted+=Number(impact.count);await tx.executeRaw('DELETE FROM sources WHERE id=$1 AND incarnation=$2::uuid',[input.sourceId,incarnation]);await tx.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1 AND source_incarnation=$2::uuid',[input.sourceId,incarnation]);+ await supersedeTopologyRetires(tx,input.sourceId,incarnation,requestId);}const contentReceipts=input.operation==='rebind'?await rebindContentReceipts(tx,worktrees,hostIdFor(tx),relocated?undefined:[input.sourceId]):[];await advanceTopology(tx,worktrees);diff --git a/src/core/persistence/topology-receipts.ts b/src/core/persistence/topology-receipts.tsindex 30f2ce415c..70d21a0932 100644--- a/src/core/persistence/topology-receipts.ts+++ b/src/core/persistence/topology-receipts.ts@@ -31,7 +31,11 @@ export async function recordTopologyChange(tx:BrainEngine, input:{principal:striconst limits=await readJournalLimits(tx);const key=principalKey({kind:'local_cli',id:input.principal});const counters=await lockCounters(tx,['brain',key,...input.worktrees.map(id=>`worktree:${id}`)]);- await priorTopologyChange(tx,input.principal,input.requestId,input.intent);+ const admitted=await priorTopologyChange(tx,input.principal,input.requestId,input.intent);+ if(admitted&&!recovery&&isRetireAdmission(admitted)){+ const settled=await settleTopologyRetire(tx,admitted,'committed',outcome);+ if(settled)return settled;+ }for(const counter of counters){const brain=counter.key==='brain';if(!counter.key.startsWith('worktree:')){@@ -68,11 +72,46 @@ async function assertQuota(tx:BrainEngine,counter:Counter,limits:JournalLimits,tthrow quotaExhausted(`${brain?'brain':'principal'} ${ids?'permanent request IDs':'reserved receipt bytes'} (lifecycle ${input.operation} of source ${input.sourceId})`,journalLimitKey(setting),await oneYearCapacity(tx,counter.key,ids?'LifetimeIds':'TerminalBytes',used,limits[setting]),used,limits[setting],ids?'lifetime_ids':'terminal_bytes');}-/** #7702/#8905: the lifetime-ID and receipt-byte quotas recordTopologyChange enforces, at the largest receipt it can record, so a retire refuses before it deletes anything. */-export async function assertTopologyChangeCapacity(tx:BrainEngine,input:{principal:string;operation:string;sourceId:string}):Promise<void>{+/**+ * #8984: a remove or purge that deletes pages in batches (#7702/#8905) first records, in the transaction of its first+ * batch, which incarnation of the source its request_id was admitted against: a recovering row with no recovery+ * record, holding one lifetime ID and a full receipt reservation (so it refuses on a spent quota before deleting+ * anything). recordTopologyChange settles it committed; a retry against another incarnation settles it failed.+ */+export function isRetireAdmission(row:TopologyChange):boolean{+ return row.state==='recovering'&&row.recovery===null&&['remove','purge'].includes(row.operation);+}+export const RETIRE_SUPERSEDED='source_retire_superseded';+export function isSupersededRetire(row:TopologyChange):boolean{+ return row.state==='failed'&&row.outcome?.error===RETIRE_SUPERSEDED;+}+export async function admitTopologyRetire(tx:BrainEngine,input:{principal:string;requestId:string;intent:unknown;operation:string;sourceId:string;incarnation:string;worktrees:string[]}):Promise<void>{const limits=await readJournalLimits(tx);- for(const counter of await lockCounters(tx,['brain',principalKey({kind:'local_cli',id:input.principal})]))- await assertQuota(tx,counter,limits,receiptReservation({}),input);+ const reservation=receiptReservation({});+ const counters=await lockCounters(tx,['brain',principalKey({kind:'local_cli',id:input.principal})]);+ for(const counter of counters) await assertQuota(tx,counter,limits,reservation,input);+ await tx.executeRaw(`INSERT INTO persistence_topology_changes+ (principal_id,request_id,digest,operation,source_id,source_incarnation,worktree_ids,state,terminal_bytes,outcome)+ VALUES($1::uuid,$2::uuid,$3,$4,$5,$6::uuid,$7::uuid[],'recovering',$8,$9::text::jsonb)`,+ [input.principal,input.requestId,digest(input.intent),input.operation,input.sourceId,input.incarnation,input.worktrees,reservation,+ JSON.stringify({operation:input.operation,source_id:input.sourceId,source_incarnation:input.incarnation})]);+ for(const counter of counters) await tx.executeRaw('UPDATE persistence_counters SET lifetime_ids=lifetime_ids+1,terminal_bytes=terminal_bytes+$2 WHERE key=$1',[counter.key,reservation]);+}+/** Settles a retire admission and trues its reserved receipt bytes up to the outcome it keeps; null when already settled. */+export async function settleTopologyRetire(tx:BrainEngine,row:TopologyChange,state:'committed'|'failed',outcome:Record<string,unknown>):Promise<TopologyChange|null>{+ const counters=await lockCounters(tx,['brain',principalKey({kind:'local_cli',id:row.principal_id})]);+ const terminalBytes=Math.min(receiptReservation({}),retainedReceiptBytes({outcome},{stage:'terminal'}));+ const [settled]=await tx.executeRaw<TopologyChange>(`UPDATE persistence_topology_changes SET state=$2,outcome=$3::text::jsonb,terminal_bytes=$4,updated_at=now()+ WHERE id=$1::uuid AND state='recovering' AND recovery IS NULL RETURNING *`,[row.id,state,JSON.stringify(outcome),terminalBytes]);+ if(!settled)return null;+ for(const counter of counters) await tx.executeRaw('UPDATE persistence_counters SET terminal_bytes=terminal_bytes+$2 WHERE key=$1',[counter.key,terminalBytes-Number(row.terminal_bytes)]);+ return settled;+}+/** A remove or purge that deleted incarnation X fails every other request still admitted against X: none can resume. */+export async function supersedeTopologyRetires(tx:BrainEngine,sourceId:string,incarnation:string,requestId:string):Promise<void>{+ const rows=await tx.executeRaw<TopologyChange>(`SELECT * FROM persistence_topology_changes WHERE source_id=$1 AND source_incarnation=$2::uuid+ AND state='recovering' AND recovery IS NULL AND operation IN ('remove','purge') AND request_id<>$3::uuid ORDER BY id`,[sourceId,incarnation,requestId]);+ for(const row of rows) await settleTopologyRetire(tx,row,'failed',{...row.outcome,error:RETIRE_SUPERSEDED});}/** Caller retains owner/source guards and completes final state in this same transaction. */export async function releaseTopologyReservation(tx:BrainEngine,row:TopologyChange):Promise<boolean>{diff --git a/test/e2e/source-retire-batched-postgres.test.ts b/test/e2e/source-retire-batched-postgres.test.tsindex 830ddeec83..ef6121e89f 100644--- a/test/e2e/source-retire-batched-postgres.test.ts+++ b/test/e2e/source-retire-batched-postgres.test.ts@@ -97,7 +97,7 @@ describe('managed remove/purge of a large source (postgres)', () => {const left = (await counts(engine, source)).pages;expect(left).toBe(PAGES - STALL_AFTER);expect(await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).toHaveLength(1);- expect(await engine.executeRaw('SELECT id FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId])).toHaveLength(0);+ expect(await engine.executeRaw('SELECT state,recovery FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId])).toEqual([{ state: 'recovering', recovery: null }]);await engine.executeRaw('DELETE FROM retire_stall');const receipt = await remove();expect(receipt).toMatchObject({ state: 'committed', operation: 'remove', pages_deleted: left });diff --git a/test/e2e/source-retire-incarnation-postgres.test.ts b/test/e2e/source-retire-incarnation-postgres.test.tsnew file mode 100644index 0000000000..f8af946a93--- /dev/null+++ b/test/e2e/source-retire-incarnation-postgres.test.ts@@ -0,0 +1,114 @@+/**+ * #8984 on real Postgres: a retry of an interrupted managed remove/purge after its source was removed and re-added+ * refuses with source_retire_superseded and keeps the new source's pages, including after batches already deleted+ * part of the old source.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { randomUUID } from 'node:crypto';+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';+import { join } from 'node:path';+import { tmpdir } from 'node:os';+import type { BrainEngine } from '../../src/core/engine.ts';+import { registerLocalWriter } from '../../src/core/persistence/identity.ts';+import { claimWorktree } from '../../src/core/persistence/ownership.ts';+import { runManagedSourceLifecycle, SOURCE_RETIRE_BATCH_PAGES } from '../../src/core/persistence/source-lifecycle.ts';+import { disposePersistenceConsumer } from '../../src/core/persistence/service.ts';+import { persistencePostgresTemplate } from '../helpers/persistence-postgres.ts';+import { requirePostgresTestDatabase } from '../helpers/test-backends.ts';+import { withEnv } from '../helpers/with-env.ts';++const source = 'retire-again';+const PAGES = SOURCE_RETIRE_BATCH_PAGES * 2 + 17;+const databaseUrl = requirePostgresTestDatabase();+let template: Awaited<ReturnType<typeof persistencePostgresTemplate>>;+beforeAll(async () => {+ template = await persistencePostgresTemplate(databaseUrl, 2, async engine => {+ for (const id of [source, 'keep-other']) await engine.executeRaw('INSERT INTO sources(id,name) VALUES($1,$1)', [id]);+ await seed(engine, source, PAGES);+ await seed(engine, 'keep-other', 20);+ });+}, 600_000);+afterAll(async () => { await template?.dispose(); });++function seed(engine: BrainEngine, id: string, n: number, from = 1) {+ return engine.executeRaw(`INSERT INTO pages(source_id,slug,type,title,compiled_truth)+ SELECT $1,'notes/p-'||g,'note','Page '||g,'Body '||g FROM generate_series($2::int,$3::int) g`, [id, from, from + n - 1]);+}+const pages = async (engine: BrainEngine, id = source) => (await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM pages WHERE source_id=$1', [id]))[0].n;+const incarnation = async (engine: BrainEngine) => (await engine.executeRaw<{ incarnation: string }>('SELECT incarnation FROM sources WHERE id=$1', [source]))[0]?.incarnation;+const change = async (engine: BrainEngine, requestId: string) => (await engine.executeRaw<{ state: string; source_incarnation: string }>(+ 'SELECT state,source_incarnation FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId]))[0];+/** Raw page writes stand in for an import; the managed guard admits them only while the brain is briefly unmanaged. */+async function unmanaged(engine: BrainEngine, run: () => Promise<unknown>) {+ await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');+ try { await run(); } finally { await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1'); }+}++describe('retrying a managed remove after its source was re-added (postgres)', () => {+ async function fixture(run: (engine: BrainEngine, root: string) => Promise<void>) {+ const { engine, close } = await template.clone();+ const home = mkdtempSync(join(tmpdir(), 'gbrain-retire-again-'));+ try {+ await withEnv({ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined }, async () => {+ await registerLocalWriter(engine, 'cli');+ for (const id of [source, 'keep-other']) {+ const root = join(home, id); mkdirSync(root);+ writeFileSync(join(root, 'example.md'), '---\ntitle: Example\ntype: note\n---\n\nCanonical\n');+ await engine.executeRaw('UPDATE sources SET local_path=$2 WHERE id=$1', [id, root]);+ await claimWorktree(engine, id, root);+ }+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');+ await run(engine, join(home, source));+ expect(await pages(engine, 'keep-other')).toBe(20);+ });+ } finally { await disposePersistenceConsumer(engine); await close(); rmSync(home, { recursive: true, force: true }); }+ }+ async function readd(engine: BrainEngine, root: string, n: number) {+ expect(await runManagedSourceLifecycle(engine, { operation: 'add', sourceId: source, path: root })).toMatchObject({ state: 'committed' });+ await unmanaged(engine, () => seed(engine, source, n, 9000));+ }++ test('a retry after the final transaction failed and the source was removed and re-added refuses', () => fixture(async (engine, root) => {+ await engine.executeRaw(`CREATE FUNCTION retire_final_fn() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN+ RAISE EXCEPTION 'injected final retire failure'; END $$`);+ await engine.executeRaw('CREATE TRIGGER retire_final BEFORE DELETE ON sources FOR EACH ROW EXECUTE FUNCTION retire_final_fn()');+ const requestId = randomUUID();+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });+ const first = await incarnation(engine);+ try { await expect(remove()).rejects.toThrow('injected final retire failure'); }+ finally { await engine.executeRaw('DROP TRIGGER retire_final ON sources'); await engine.executeRaw('DROP FUNCTION retire_final_fn()'); }+ expect(await pages(engine)).toBe(0);+ expect(await change(engine, requestId)).toMatchObject({ state: 'recovering', source_incarnation: first });+ expect(await runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true })).toMatchObject({ state: 'committed' });+ expect(await change(engine, requestId)).toMatchObject({ state: 'failed', source_incarnation: first });+ await readd(engine, root, 1);+ for (let i = 0; i < 2; i++) await expect(remove()).rejects.toMatchObject({ code: 'source_retire_superseded' });+ expect(await pages(engine)).toBe(1);+ }), 300_000);++ test('a batched retry after partial deletion refuses once the source was replaced, and settles its record', () => fixture(async (engine, root) => {+ await engine.executeRaw('CREATE TABLE retire_fail(armed boolean NOT NULL, seen integer NOT NULL)');+ await engine.executeRaw(`CREATE FUNCTION retire_fail_fn() RETURNS trigger LANGUAGE plpgsql AS $$ DECLARE n integer; BEGIN+ UPDATE retire_fail SET seen=seen+1 WHERE armed RETURNING seen INTO n;+ IF n > ${SOURCE_RETIRE_BATCH_PAGES} THEN RAISE EXCEPTION 'injected retire batch failure'; END IF; RETURN OLD; END $$`);+ await engine.executeRaw('CREATE TRIGGER retire_fail BEFORE DELETE ON pages FOR EACH ROW EXECUTE FUNCTION retire_fail_fn()');+ await engine.executeRaw('INSERT INTO retire_fail VALUES(true,0)');+ const requestId = randomUUID();+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });+ const first = await incarnation(engine);+ await expect(remove()).rejects.toThrow('injected retire batch failure');+ await engine.executeRaw('DELETE FROM retire_fail');+ expect(await pages(engine)).toBe(PAGES - SOURCE_RETIRE_BATCH_PAGES);+ expect(await change(engine, requestId)).toMatchObject({ state: 'recovering', source_incarnation: first });+ await unmanaged(engine, async () => {+ await engine.executeRaw('DELETE FROM pages WHERE source_id=$1', [source]);+ await engine.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1', [source]);+ await engine.executeRaw('DELETE FROM sources WHERE id=$1', [source]);+ });+ await readd(engine, root, 3);+ await expect(remove()).rejects.toMatchObject({ code: 'source_retire_superseded' });+ expect(await change(engine, requestId)).toMatchObject({ state: 'failed', source_incarnation: first });+ await expect(remove()).rejects.toMatchObject({ code: 'source_retire_superseded' });+ expect(await pages(engine)).toBe(3);+ }), 300_000);+});diff --git a/test/source-retire-batched.test.ts b/test/source-retire-batched.test.tsindex 845fbd98ac..c329f5a54c 100644--- a/test/source-retire-batched.test.ts+++ b/test/source-retire-batched.test.ts@@ -135,7 +135,7 @@ test('an interrupted batch keeps the committed batches and the same request_id rawait expect(remove()).rejects.toThrow('injected retire batch failure');expect(await pages(source)).toBe(PAGES - SOURCE_RETIRE_BATCH_PAGES * 2);expect(await registered()).toBe(1);- expect(await engine.executeRaw('SELECT id FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId])).toHaveLength(0);+ expect(await engine.executeRaw('SELECT state,recovery FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId])).toEqual([{ state: 'recovering', recovery: null }]);await engine.executeRaw('DELETE FROM retire_fail');const receipt = await remove();expect(receipt).toMatchObject({ state: 'committed', operation: 'remove', pages_deleted: PAGES - SOURCE_RETIRE_BATCH_PAGES * 2 });diff --git a/test/source-retire-incarnation.test.ts b/test/source-retire-incarnation.test.tsnew file mode 100644index 0000000000..da1029aa70--- /dev/null+++ b/test/source-retire-incarnation.test.ts@@ -0,0 +1,138 @@+/**+ * #8984 on PGLite: a managed remove or purge records which incarnation of its source it was admitted against, durably+ * and atomically with its first deleted batch. A retry of that request_id after the source was removed and re-added+ * refuses with source_retire_superseded and never deletes the new source's pages; a retry against the same+ * incarnation still resumes and finishes.+ */+import { afterAll, beforeAll, beforeEach, expect, test } from 'bun:test';+import { randomUUID } from 'node:crypto';+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';+import { join } from 'node:path';+import { tmpdir } from 'node:os';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { resetPgliteState } from './helpers/reset-pglite.ts';+import { withEnv } from './helpers/with-env.ts';+import { registerLocalWriter } from '../src/core/persistence/identity.ts';+import { claimWorktree } from '../src/core/persistence/ownership.ts';+import { runManagedSourceLifecycle, SOURCE_RETIRE_BATCH_PAGES } from '../src/core/persistence/source-lifecycle.ts';+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';+import { deactivationBlockers } from '../src/core/persistence/deactivation.ts';++let engine: PGLiteEngine;+beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); });+afterAll(async () => { await disposePersistenceConsumer(engine); await engine.disconnect(); });+beforeEach(async () => { await resetPgliteState(engine); });++const source = 'retire-again';+const pages = async (id = source) => (await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM pages WHERE source_id=$1', [id]))[0].n;+const incarnation = async () => (await engine.executeRaw<{ incarnation: string }>('SELECT incarnation FROM sources WHERE id=$1', [source]))[0]?.incarnation;+const change = async (requestId: string) => (await engine.executeRaw<{ state: string; source_incarnation: string; recovery: unknown }>(+ 'SELECT state,source_incarnation,recovery FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId]))[0];+const seed = (id: string, n: number, from = 1) => engine.executeRaw(`INSERT INTO pages(source_id,slug,type,title,compiled_truth)+ SELECT $1,'notes/p-'||g,'note','Page '||g,'Body '||g FROM generate_series($2::int,$3::int) g`, [id, from, from + n - 1]);++async function fixture(count: number, run: (root: string) => Promise<void>) {+ const home = mkdtempSync(join(tmpdir(), 'gbrain-retire-again-'));+ try {+ await withEnv({ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined }, async () => {+ await registerLocalWriter(engine, 'cli');+ const roots: Record<string, string> = {};+ for (const id of [source, 'keep-other']) {+ const root = roots[id] = join(home, id); mkdirSync(root);+ writeFileSync(join(root, 'example.md'), '---\ntitle: Example\ntype: note\n---\n\nCanonical\n');+ await engine.executeRaw('INSERT INTO sources(id,name,local_path) VALUES($1,$1,$2)', [id, root]);+ await claimWorktree(engine, id, root);+ await seed(id, id === source ? count : 20);+ }+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');+ await run(roots[source]);+ expect(await pages('keep-other')).toBe(20);+ });+ } finally { rmSync(home, { recursive: true, force: true }); }+}++/** Arms a one-shot failure of the final topology transaction (its DELETE FROM sources), after every page batch committed. */+async function failFinalTransaction<T>(run: () => Promise<T>): Promise<T> {+ await engine.executeRaw(`CREATE FUNCTION retire_final_fn() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN+ RAISE EXCEPTION 'injected final retire failure'; END $$`);+ await engine.executeRaw('CREATE TRIGGER retire_final BEFORE DELETE ON sources FOR EACH ROW EXECUTE FUNCTION retire_final_fn()');+ try { return await run(); } finally {+ await engine.executeRaw('DROP TRIGGER IF EXISTS retire_final ON sources');+ await engine.executeRaw('DROP FUNCTION IF EXISTS retire_final_fn()');+ }+}+/** Raw page writes stand in for an import; the managed guard admits them only while the brain is briefly unmanaged. */+async function unmanaged(run: () => Promise<unknown>) {+ await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');+ try { await run(); } finally { await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1'); }+}+async function readd(root: string, count: number) {+ expect(await runManagedSourceLifecycle(engine, { operation: 'add', sourceId: source, path: root })).toMatchObject({ state: 'committed' });+ await unmanaged(() => seed(source, count, 9000));+}++test('a retry after the source was removed and re-added refuses and keeps the new source\'s pages', () => fixture(1, async root => {+ const requestId = randomUUID();+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });+ const first = await incarnation();+ await expect(failFinalTransaction(remove)).rejects.toThrow('injected final retire failure');+ expect(await pages()).toBe(0);+ expect(await change(requestId)).toMatchObject({ state: 'recovering', source_incarnation: first, recovery: null });+ expect(await runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true })).toMatchObject({ state: 'committed' });+ expect(await change(requestId)).toMatchObject({ state: 'failed', source_incarnation: first });+ await readd(root, 1);+ expect(await incarnation()).not.toBe(first);+ for (let i = 0; i < 2; i++) await expect(remove()).rejects.toMatchObject({ code: 'source_retire_superseded' });+ expect(await pages()).toBe(1);+ expect(await incarnation()).toBeDefined();+}), 60_000);++test('a batched retry after partial deletion refuses once the source was replaced out of band, and settles its record', () => fixture(SOURCE_RETIRE_BATCH_PAGES * 2 + 17, async root => {+ await engine.executeRaw('CREATE TABLE retire_fail(armed boolean NOT NULL, seen integer NOT NULL)');+ await engine.executeRaw(`CREATE FUNCTION retire_fail_fn() RETURNS trigger LANGUAGE plpgsql AS $$ DECLARE n integer; BEGIN+ UPDATE retire_fail SET seen=seen+1 WHERE armed RETURNING seen INTO n;+ IF n > ${SOURCE_RETIRE_BATCH_PAGES} THEN RAISE EXCEPTION 'injected retire batch failure'; END IF; RETURN OLD; END $$`);+ await engine.executeRaw('CREATE TRIGGER retire_fail BEFORE DELETE ON pages FOR EACH ROW EXECUTE FUNCTION retire_fail_fn()');+ try {+ await engine.executeRaw('INSERT INTO retire_fail VALUES(true,0)');+ const requestId = randomUUID();+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });+ const first = await incarnation();+ await expect(remove()).rejects.toThrow('injected retire batch failure');+ await engine.executeRaw('DELETE FROM retire_fail');+ expect(await pages()).toBe(SOURCE_RETIRE_BATCH_PAGES + 17);+ expect(await change(requestId)).toMatchObject({ state: 'recovering', source_incarnation: first });+ await unmanaged(async () => {+ await engine.executeRaw('DELETE FROM pages WHERE source_id=$1', [source]);+ await engine.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1', [source]);+ await engine.executeRaw('DELETE FROM sources WHERE id=$1', [source]);+ });+ await readd(root, 3);+ await expect(remove()).rejects.toMatchObject({ code: 'source_retire_superseded' });+ expect(await change(requestId)).toMatchObject({ state: 'failed', source_incarnation: first });+ await expect(remove()).rejects.toMatchObject({ code: 'source_retire_superseded' });+ await expect(runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId, dryRun: true }))+ .rejects.toMatchObject({ code: 'source_retire_superseded' });+ expect(await pages()).toBe(3);+ } finally {+ for (const sql of ['DROP TRIGGER IF EXISTS retire_fail ON pages', 'DROP FUNCTION IF EXISTS retire_fail_fn()', 'DROP TABLE IF EXISTS retire_fail']) await engine.executeRaw(sql);+ }+}), 60_000);++test('a retry against the same incarnation resumes, finishes and settles its record committed', () => fixture(SOURCE_RETIRE_BATCH_PAGES + 3, async () => {+ const requestId = randomUUID();+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });+ const first = await incarnation();+ const err = await failFinalTransaction(remove).then(() => null, (e: Error & { suggestion?: string }) => e);+ expect(err?.message).toContain('injected final retire failure');+ expect(await pages()).toBe(0);+ expect(await change(requestId)).toMatchObject({ state: 'recovering', source_incarnation: first });+ expect((await deactivationBlockers(engine)).find(b => b.kind === 'topology_recovery')).toMatchObject({+ detail: `remove of ${source} (request ${requestId}) stopped before it finished`, exit: expect.stringContaining(`--request-id ${requestId}`) });+ const receipt = await remove();+ expect(receipt).toMatchObject({ state: 'committed', operation: 'remove', source_incarnation: first });+ expect(await remove()).toEqual(receipt);+ expect(await change(requestId)).toMatchObject({ state: 'committed', recovery: null });+ expect((await deactivationBlockers(engine)).filter(b => b.kind === 'topology_recovery')).toEqual([]);+ expect(await incarnation()).toBeUndefined();+}), 60_000);