Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Batch managed source remove/purge page deletes under the topology bound
garrytan8c5ee9e2d654
diff --git a/docs/architecture/canonical-writers.tsv b/docs/architecture/canonical-writers.tsvindex d546f89093..88c627d803 100644--- a/docs/architecture/canonical-writers.tsv+++ b/docs/architecture/canonical-writers.tsv@@ -161,7 +161,7 @@ src/core/persistence/sync-image.ts 2 coordinator The managed sync image branch psrc/core/persistence/sync-run.ts 1 coordinator The managed up-to-date heartbeat stamps only sources.last_sync_at inside withCoordinatedWrite, scoped to the checked source id and incarnation; no checkpoint, commit or content-freshness column changes.src/core/persistence/sync-pass-run.ts 2 coordinator #6712 PR 4b: the pass's up-to-date and no-checkpoint stamps write only sources.last_sync_at inside withCoordinatedWrite, after assertActive and assertDrainLease, scoped to the source id and incarnation; no checkpoint, commit or content-freshness column changes.src/commands/enrich.ts 0 coordinator A managed brain preflights maintenance authority before any model spend and publishes each enriched page through publishMaintenancePage; unmanaged enrichment writes through the put_page operation.-src/core/persistence/source-lifecycle.ts 7 coordinator Native affected roots precede topology, source, principal and receipt guards; every shared-root generation and old queued intent changes together.+src/core/persistence/source-lifecycle.ts 8 coordinator Native affected roots precede topology, source, principal and receipt guards; every shared-root generation and old queued intent changes together. A remove or purge first deletes the source's pages in bounded batches under the same native locks, each batch behind the final transaction's preconditions (#7702/#8905).src/core/persistence/topology-recovery.ts 2 coordinator Verified directory-swap recovery checks source incarnation, current logical manifest and physical identity before topology commitment; permanent receipt IDs survive deletion.src/core/persistence/connector-sync.ts 4 coordinator Connector freshness and source-scoped deletion commit through original/current authority, source and revision guards, canonical publication and checkpoint CAS. A skipped unchanged checkpoint save stamps sources.last_sync_at/newest_content_at (never page content) inside withCoordinatedWrite, in a transaction that first asserts the connector sync lease, scoped to the source incarnation (#5686/#5470). Its importFromContent calls carry prepare and run inside the coordinator publication.src/core/persistence/prepared-maintenance.ts 5 coordinator Named local maintenance uses native owner publication; contributing semantic facts and take lifecycle are revalidated and committed together with canonical files and receipts; a chronicle event page and its depth timeline projection commit in one publication; v0.32.2 fact adoption re-locks the pinned legacy facts and their fence positions and sets their row positions ahead of the canonical projection in the same publication.; consolidation closes a fenced fact from the fence cell it publishes with the take, an unfenced one in the database (G4)diff --git a/docs/architecture/coordinated-writes.tsv b/docs/architecture/coordinated-writes.tsvindex 79be8aeaa5..acafa38ae9 100644--- a/docs/architecture/coordinated-writes.tsv+++ b/docs/architecture/coordinated-writes.tsv@@ -21,7 +21,7 @@ src/core/persistence/connector-sync.ts stampFreshness 1 operational Stamps last_src/core/persistence/coordinator.ts publishMutation 2 door The page-mutation door itself: the prepared publication and its after-effects commit here after the file is in place.src/core/persistence/group-publish.ts publishGroup 1 door Group publication of door requests: the same prepared publications as publishMutation, committed as one group.src/core/persistence/links-maintenance.ts derive 1 projection Re-derives markdown, frontmatter and mention edges from each page's current text, bound to the revision it read; the indexer reproduces them.-src/core/persistence/source-lifecycle.ts runManagedSourceLifecycle 1 operational Source topology changes (remove, purge, archive) on the sources and binding rows; source administration, not memory content.+src/core/persistence/source-lifecycle.ts runManagedSourceLifecycle 2 operational Source topology changes (remove, purge, archive) on the sources and binding rows, and the bounded page-deletion batches a remove or purge runs first (#7702/#8905); source administration, not memory content.src/core/persistence/sync-pass-run.ts stampIdlePass 1 operational An idle sync pass stamps last_sync_at and its pass row; sync bookkeeping, not memory content.src/core/persistence/sync-pass-run.ts stampUnchangedPass 1 operational An unchanged sync pass stamps last_sync_at and its converged watermark; sync bookkeeping, not memory content.src/core/persistence/sync-run.ts runManagedSync 1 operational A sync that found nothing stamps the freshness heartbeat and releases holds for paths that left the source; sync bookkeeping, not memory content.diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.mdindex d774948873..bc157b81cd 100644--- a/docs/architecture/key-files/core-persistence-continued.md+++ b/docs/architecture/key-files/core-persistence-continued.md@@ -41,3 +41,4 @@ The #6188 fence check and fence repair live in [fence repair](core-fence-repair.- `src/core/postgres-engine/reserved-transactions.ts` with `isPersistenceDeclaration` (`src/core/persistence/protocol.ts`) — BEGIN pipelining. `isPersistenceDeclaration` (GBRA-156) accepts exactly the `protocol.ts` declarations (transaction-local `set_config` calls and the brain row FOR SHARE). Admission, the FIFO claim, the effect claims and source guards, the receipt true-up, the single publication (`coordinator.ts`) and the consumer's lent lane transactions (`consumer.ts` `onLane`, through `reservedTransactions`) pass it as `TransactionOptions.pipelineFirst`, so behind a transaction-mode pooler the declaration goes out with BEGIN (one round trip less per transaction) while every later statement still waits for BEGIN's answer; if BEGIN fails, only the declaration has run, in autocommit, where it changes nothing. A reserved transaction gates every statement its body can reach, savepoint and nested-transaction bodies and `file()` included. A sync lane group's publication passes none: its begun mark must follow BEGIN's answer (`sync-lanes.ts` `awaitLaneBegin`). Tests: `test/persistence-pipeline-first.test.ts` (the predicate and the reviewed call sites), `test/e2e/pgbouncer-transaction-pipeline-6930.test.ts`.- `src/core/links-fence.ts` + `src/core/links-fence-projection.ts` — the `## Links` fence (G5): the markdown form of authored (manual) edges, `<!--- gbrain:links:begin -->` / `end` around `| to | type | source | valid_from | valid_until | context |`, one row per (to, type, source), source defaulting to `manual`; a reconciliation-managed provenance (`MANAGED_LINK_SOURCES`, exported here and re-exported by `ops/links.ts`), a bad date or a duplicate key is a skipped row with a warning. `parseLinksFence` (state `absent` / `parsed` / `malformed`), the deterministic `renderLinksFence` (rows sorted by key), `upsertLinksFenceRow` (omitted context and dates keep the stored ones; identical row = unchanged body), `removeLinksFenceRows` (leaves an empty fence), `replaceOrInsertLinksFence` (a `## Links` section at the end of compiled_truth, above an unclosed code block) and `maskLinksFence`, which `extractBodyLinks` applies so the fence cells are never also extracted as markdown, wikilink or bare-slug edges. A rewrite of a fence that parses with warnings throws `LinksFenceRewriteRefusal`. Cells use `escapeFenceCell` plus the same one-level marker encoding for `gbrain:links:*`. `projectLinksFence(tx, page, rows)` reconciles only what the page's fence owns: edges with origin_page_id = the page and origin_field `links_fence` (deleted, context-updated, inserted by key) and `manual` transitions with origin_page_id = the page, then refreshes the touched relationships; a row whose target is not a live page in the source yields no edge. `linksFenceProjectionRows` returns null for an absent or malformed fence, so such a body changes no edge (an absent fence retires nothing, as for facts; unlike facts no held generation is recorded, so after a hand deletion of the whole section the database keeps edges git no longer has until `remove_link` or the G3 indexer), and drops rows typed with a link type the active pack does not declare. Callers: `prepareCanonicalProjections` (every projected page publication, quarantined and segmented-unread bodies excepted), `prepareAutomaticLinks` (put_page auto-links and the managed links maintenance, so `extract --stale` restores a lost fence edge) and `persistence/manual-links.ts`. Database-only manual edges (origin_field NULL) are never touched by any of them. Pinned by `test/links-fence.test.ts` and `test/managed-links-fence.test.ts`.+- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`) and, when one would refuse or replay, stops without deleting so the final transaction reports it; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. Tests: `test/source-retire-batched.test.ts` (PGLite) and `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages).diff --git a/skills/data-loss-gate/SKILL.md b/skills/data-loss-gate/SKILL.mdindex 59805d6047..148b9e4fdb 100644--- a/skills/data-loss-gate/SKILL.md+++ b/skills/data-loss-gate/SKILL.md@@ -93,7 +93,9 @@ Brain / database level (gbrain-specific):`--yes` only after the user agrees.- **Source removal** — `gbrain sources remove <id>` deletes the source ANDevery page in it. The `--confirm-destructive` flag does not substitute for- the card.+ the card. A managed removal deletes pages in committed batches: one that+ stops early (`write_pending`) has already deleted some, so say so and rerun it+ to finish.- **Mount removal** — `gbrain mounts remove <id>` only removes the localregistration (the mounted brain's database survives; re-add to recover). Gateit anyway when the flow ALSO plans to delete the mount's underlying databasediff --git a/skills/skills.lock.json b/skills/skills.lock.jsonindex 76abef9092..0a50cead1c 100644--- a/skills/skills.lock.json+++ b/skills/skills.lock.json@@ -71,7 +71,7 @@"daily-task-manager/SKILL.md": "e93264a83b5fd0a19b2f06a322b3a4ad92d418f6a968c1cd51f4bea79cbca4d9","daily-task-manager/routing-eval.jsonl": "e98b2592e65c26952242436a960d8b2061e375b269004a18c230cbcf1d692ff9","daily-task-prep/SKILL.md": "5ea839012b48761ed8d7a9bab056a2566587d6f7fe044e3907d77b20f8eec5ac",- "data-loss-gate/SKILL.md": "c1736e0985121e60ffeba993b68b5c722c9285220057b5ef72059f111aabcb18",+ "data-loss-gate/SKILL.md": "656d7e108a88fa3ac7d7d41dfb79f02b7dc592d286a7cd6a7fa63503550e49ba","data-loss-gate/routing-eval.jsonl": "825358a5600641061a02b25097cd84c1c28bb0359a7608a10b9f25f661bd88f9","data-research/SKILL.md": "39f9648a21a7074d6ad7db789e93e6d7085449a6cdaffacf8b1660eb68636b9d","db-repair/SKILL.md": "87a34fe18812f3b6ec37381ae69c205e30747300098ebe3fe41f8460d0ba8346",diff --git a/src/core/persistence/source-lifecycle.ts b/src/core/persistence/source-lifecycle.tsindex e8ccb10596..a3b295d6af 100644--- a/src/core/persistence/source-lifecycle.ts+++ b/src/core/persistence/source-lifecycle.ts@@ -16,7 +16,7 @@ import { isInsideGitRepo, hasTrackedContent } from '../git-remote.ts';import { containsPath, getWorktreeBinding, humanManifestProgress, type WorktreeBinding, type WorktreeManifest, worktreeManifest } from './ownership.ts';import { manifestDifferences, storedManifestScope, type WorktreeManifestScope } from './worktree-manifest.ts';import { hostIdFor } from './identity.ts';-import { advanceTopology, inspectTopologyPhysicalRoot, lockTopologyPrincipal, lockTopologyRows, settleTopologyRequests, topologyCanonicalStamp, topologyPrincipal, withTopologyLocks } from './topology-locks.ts';+import { advanceTopology, assertTopologyRequestsSettled, inspectTopologyPhysicalRoot, lockTopologyPrincipal, lockTopologyRows, settleTopologyRequests, topologyCanonicalStamp, topologyPrincipal, withTopologyLocks } from './topology-locks.ts';import { priorTopologyChange, recordTopologyChange, topologyReceipt } from './topology-receipts.ts';import { isWriteRequestId } from './types.ts';import { withCoordinatedWrite } from './context.ts';@@ -42,6 +42,8 @@ export interface SourceLifecycleInput {expectedAdminState?:string;automaticClaim?:boolean;}+/** #7702/#8905: pages one retire batch deletes; each batch (with its cascaded chunks, links and versions) stays far under the 5 s topology statement bound. */+export const SOURCE_RETIRE_BATCH_PAGES=500;interface SourceState {id:string;incarnation:string;archived:boolean;local_path:string|null;config:Record<string,unknown>;name:string;last_commit:string|null;}function sourcesListFix():Action{@@ -181,6 +183,26 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceLif(input.operation==='rebind'&&root&¤tRoot&¤tRoot!==root.worktree&&manifests.has(currentRoot)&&manifests.has(root.worktree)&&storedManifestScope(manifests.get(currentRoot))!==storedManifestScope(manifests.get(root.worktree)))manifests.set(root.worktree,worktreeManifest(root.worktree,{progress:humanManifestProgress(),scope:storedManifestScope(manifests.get(currentRoot))}));+ // #7702/#8905: a remove or purge first deletes the source's pages in bounded batches, each its own transaction behind+ // the final transaction's preconditions; any precondition that would refuse or replay stops batching and the final+ // transaction reports it. Committed batches are durable, so a retry with the same request_id resumes.+ let retiredPages=0;+ if(['remove','purge'].includes(input.operation)&&input.confirmDestructive&&!missingCheckout&&!admission)+ for(let deleted=SOURCE_RETIRE_BATCH_PAGES;deleted===SOURCE_RETIRE_BATCH_PAGES;) deleted=await topologyTransaction(engine,async tx=>{+ await assertWriterAdminState(tx,input.expectedAdminState);+ const sources=await lockTopologyRows(tx,input.sourceId,bindings);+ const [source]=await tx.executeRaw<{incarnation:string;archived:boolean;due:boolean}>(+ `SELECT incarnation,archived,${input.expiredOnly?EXPIRED_ARCHIVE_SQL:'true'} AS due FROM sources WHERE id=$1`,[input.sourceId]);+ if(source?.incarnation!==before!.incarnation||!source.due||input.operation==='purge'&&!input.expiredOnly&&!source.archived+ ||await priorTopologyChange(tx,principal,requestId,intent)+ ||(await tx.executeRaw('SELECT client_id FROM oauth_clients WHERE source_id=$1 LIMIT 1',[input.sourceId])).length) return 0;+ await assertTopologyRequestsSettled(tx,sources,bindings.map(binding=>binding.worktree_id));+ const gone=await withCoordinatedWrite(tx,sources,()=>tx.executeRaw(+ 'DELETE FROM pages WHERE id IN (SELECT id FROM pages WHERE source_id=$1 LIMIT $2) RETURNING id',[input.sourceId,SOURCE_RETIRE_BATCH_PAGES]),+ principalAttribution({kind:'local_cli',id:principal}));+ retiredPages+=gone.length;+ return gone.length;+ });return topologyTransaction(engine,async tx=>{await assertWriterAdminState(tx,input.expectedAdminState);await tx.executeRaw("SELECT set_config('synchronous_commit','on',true)");@@ -248,7 +270,7 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceLawait tx.executeRaw("SELECT set_config('gbrain.topology_change','on',true)");const result=await withCoordinatedWrite(tx,sources,async()=>{let incarnation=source?.incarnation??randomUUID();- let pagesDeleted=0;+ let pagesDeleted=retiredPages;if(input.operation==='add'||input.operation==='claim'){if(root&&input.createDirectory&&!existsSync(root.source)){mkdirSync(root.source,{recursive:true});flushTopologyDirectory(dirname(root.source));}if(source) await tx.executeRaw("UPDATE sources SET local_path=$2,name=COALESCE($3,name),config=$4::text::jsonb WHERE id=$1",@@ -278,7 +300,7 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceL`Nothing was removed. OAuth client ${refs[0].client_id} (and possibly others) is bound to '${input.sourceId}'; list them with the command in fix and ask the user before revoking any, since revoking disconnects that client.`,{fix:readFix('Lists OAuth clients with the source each is bound to.',{argv:['gbrain','auth','clients','--json']})});const [impact]=await tx.executeRaw<{count:string}>('SELECT count(*)::text AS count FROM pages WHERE source_id=$1',[input.sourceId]);- pagesDeleted=Number(impact.count);+ pagesDeleted+=Number(impact.count);await tx.executeRaw('DELETE FROM sources WHERE id=$1 AND incarnation=$2::uuid',[input.sourceId,incarnation]);await tx.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1 AND source_incarnation=$2::uuid',[input.sourceId,incarnation]);}diff --git a/src/core/persistence/topology-locks.ts b/src/core/persistence/topology-locks.tsindex f8c61799dc..c00fc6de70 100644--- a/src/core/persistence/topology-locks.ts+++ b/src/core/persistence/topology-locks.ts@@ -147,8 +147,8 @@ export async function topologyCanonicalStamp(tx:BrainEngine,worktreeId:string):Preturn row.stamp;}-/** No provider or filesystem wait occurs while these database guards are held. */-export async function settleTopologyRequests(tx: BrainEngine, sources: string[], worktrees: string[], principal: string): Promise<number> {+/** Read-only half of settleTopologyRequests: refuses while a publication, recovery or blocking effect is unfinished. */+export async function assertTopologyRequestsSettled(tx: BrainEngine, sources: string[], worktrees: string[]): Promise<WriteRequest[]> {const requests = await tx.executeRaw<WriteRequest>(`SELECT * FROM persistence_requestsWHERE (source_id=ANY($1::text[]) OR worktree_id=ANY($2::uuid[]))AND (state IN ('queued','deferred','running','recovering') OR recovery IS NOT NULL) ORDER BY sequence`, [sources,worktrees]);@@ -167,6 +167,12 @@ export async function settleTopologyRequests(tx: BrainEngine, sources: string[],`Effect ${blocked.id} on source ${source} (a queued withdrawal mirror, a running effect or one holding a recovery record) has not finished, so the lifecycle change rolled back and nothing changed. The owner's effect worker settles it; check it with the command in fix, which lists it with its page and request. Then run the same lifecycle command again with the same request_id.`,{fix:ownerStatusFix(source)});}+ return requests;+}++/** No provider or filesystem wait occurs while these database guards are held. */+export async function settleTopologyRequests(tx: BrainEngine, sources: string[], worktrees: string[], principal: string): Promise<number> {+ const requests = await assertTopologyRequestsSettled(tx, sources, worktrees);const pending = requests.filter(row => row.state === 'queued' || row.state === 'deferred');// Revocation and publication use the same principal rows. Administrative// invalidation may terminate revoked work but still respects guard ordering.diff --git a/src/core/persistence/topology-transaction.ts b/src/core/persistence/topology-transaction.tsindex 0cf16c27a6..057957c947 100644--- a/src/core/persistence/topology-transaction.ts+++ b/src/core/persistence/topology-transaction.ts@@ -29,7 +29,7 @@ export async function topologyTransaction<T>(engine: BrainEngine, run: (tx: Brai});} catch (error) {if (['55P03','57014'].includes((error as { code?: string }).code ?? '')) {- throw new OperationError('write_pending', 'The source transaction exceeded its bounded database wait.', 'Retry the same lifecycle request_id after the current writer finishes.');+ throw new OperationError('write_pending', 'The source transaction exceeded its bounded database wait.', 'Retry the same lifecycle request_id after the current writer finishes; work it already committed, such as the pages a source removal deleted, is kept and the retry resumes from there.');}throw error;} finally { release(); }diff --git a/test/e2e/source-retire-batched-postgres.test.ts b/test/e2e/source-retire-batched-postgres.test.tsnew file mode 100644index 0000000000..830ddeec83--- /dev/null+++ b/test/e2e/source-retire-batched-postgres.test.ts@@ -0,0 +1,108 @@+/**+ * #7702 / #8905: a managed remove or purge of a large source deletes its pages in bounded batches, each in its own+ * short topology transaction, so no statement meets the 5 s topology bound; a retry with the same request_id+ * resumes after an interrupted batch and finishes.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { randomUUID } from 'node:crypto';+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';+import { join } from 'node:path';+import { tmpdir } from 'node:os';+import type { BrainEngine } from '../../src/core/engine.ts';+import { registerLocalWriter } from '../../src/core/persistence/identity.ts';+import { claimWorktree } from '../../src/core/persistence/ownership.ts';+import { runManagedSourceLifecycle } from '../../src/core/persistence/source-lifecycle.ts';+import { disposePersistenceConsumer } from '../../src/core/persistence/service.ts';+import { persistencePostgresTemplate } from '../helpers/persistence-postgres.ts';+import { requirePostgresTestDatabase } from '../helpers/test-backends.ts';+import { withEnv } from '../helpers/with-env.ts';++const PAGES = 60_000;+const STALL_AFTER = PAGES / 2;+const databaseUrl = requirePostgresTestDatabase();+let template: Awaited<ReturnType<typeof persistencePostgresTemplate>>;+beforeAll(async () => {+ template = await persistencePostgresTemplate(databaseUrl, 4, async engine => {+ for (const id of ['retire-large', 'keep-other']) await engine.executeRaw('INSERT INTO sources(id,name) VALUES($1,$1)', [id]);+ await seed(engine, 'retire-large', PAGES);+ await seed(engine, 'keep-other', 50);+ });+}, 600_000);+afterAll(async () => { await template?.dispose(); });++async function seed(engine: BrainEngine, source: string, pages: number) {+ for (let from = 1; from <= pages; from += 5000) await engine.executeRaw(`INSERT INTO pages(source_id,slug,type,title,compiled_truth)+ SELECT $1,'notes/p-'||g,'note','Page '||g,'Body of page '||g FROM generate_series($2::int,LEAST($2::int+4999,$3::int)) g`, [source, from, pages]);+ await engine.executeRaw(`INSERT INTO content_chunks(page_id,chunk_index,chunk_text)+ SELECT p.id,c,'chunk '||c||' of '||p.slug FROM pages p CROSS JOIN generate_series(0,2) c WHERE p.source_id=$1`, [source]);+ await engine.executeRaw(`INSERT INTO links(from_page_id,to_page_id) SELECT p.id,q.id FROM pages p+ JOIN pages q ON q.source_id=p.source_id AND q.slug='notes/p-'||(split_part(p.slug,'-',2)::int % $2::int + 1) WHERE p.source_id=$1`, [source, pages]);+}+async function counts(engine: BrainEngine, source: string) {+ const [row] = await engine.executeRaw<{ pages: number; chunks: number; links: number }>(`SELECT+ (SELECT count(*)::int FROM pages WHERE source_id=$1) AS pages,+ (SELECT count(*)::int FROM content_chunks c JOIN pages p ON p.id=c.page_id WHERE p.source_id=$1) AS chunks,+ (SELECT count(*)::int FROM links l JOIN pages p ON p.id=l.from_page_id WHERE p.source_id=$1) AS links`, [source]);+ return row;+}++describe('managed remove/purge of a large source (postgres)', () => {+ async function fixture(run: (engine: BrainEngine, source: string) => Promise<void>) {+ const { engine, close } = await template.clone();+ const home = mkdtempSync(join(tmpdir(), 'gbrain-retire-batched-'));+ try {+ await withEnv({ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined }, async () => {+ await registerLocalWriter(engine, 'cli');+ for (const id of ['retire-large', 'keep-other']) {+ const root = join(home, id); mkdirSync(root);+ writeFileSync(join(root, 'example.md'), '---\ntitle: Example\ntype: note\n---\n\nCanonical\n');+ await engine.executeRaw('UPDATE sources SET local_path=$2 WHERE id=$1', [id, root]);+ await claimWorktree(engine, id, root);+ }+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');+ await run(engine, 'retire-large');+ expect(await counts(engine, 'keep-other')).toEqual({ pages: 50, chunks: 150, links: 50 });+ });+ } finally { await disposePersistenceConsumer(engine); await close(); rmSync(home, { recursive: true, force: true }); }+ }++ test('remove deletes every page, chunk and link of a source too large for one bounded statement', () => fixture(async (engine, source) => {+ const started = performance.now();+ const receipt = await runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true })+ .finally(() => console.log(`[retire-batched] remove of ${PAGES} pages: ${Math.round(performance.now() - started)} ms`));+ expect(receipt).toMatchObject({ state: 'committed', operation: 'remove', pages_deleted: PAGES });+ expect(await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).toHaveLength(0);+ expect(await engine.executeRaw('SELECT source_id FROM persistence_source_bindings WHERE source_id=$1', [source])).toHaveLength(0);+ expect(await counts(engine, source)).toEqual({ pages: 0, chunks: 0, links: 0 });+ }), 300_000);++ test('purge of an archived large source completes', () => fixture(async (engine, source) => {+ await runManagedSourceLifecycle(engine, { operation: 'archive', sourceId: source });+ const receipt = await runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true });+ expect(receipt).toMatchObject({ state: 'committed', operation: 'purge', pages_deleted: PAGES });+ expect(await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).toHaveLength(0);+ expect(await counts(engine, source)).toEqual({ pages: 0, chunks: 0, links: 0 });+ }), 300_000);++ test('a batch that times out keeps earlier batches and the same request_id resumes to completion', () => fixture(async (engine, source) => {+ await engine.executeRaw('CREATE TABLE retire_stall(armed boolean NOT NULL, seen integer NOT NULL)');+ await engine.executeRaw(`CREATE FUNCTION retire_stall_fn() RETURNS trigger LANGUAGE plpgsql AS $$ DECLARE n integer; BEGIN+ UPDATE retire_stall SET seen=seen+1 WHERE armed RETURNING seen INTO n;+ IF n = ${STALL_AFTER + 1} THEN PERFORM pg_sleep(6); END IF; RETURN OLD; END $$`);+ await engine.executeRaw('CREATE TRIGGER retire_stall BEFORE DELETE ON pages FOR EACH ROW EXECUTE FUNCTION retire_stall_fn()');+ await engine.executeRaw('INSERT INTO retire_stall VALUES(true,0)');+ const requestId = randomUUID();+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });+ await expect(remove()).rejects.toMatchObject({ code: 'write_pending' });+ const left = (await counts(engine, source)).pages;+ expect(left).toBe(PAGES - STALL_AFTER);+ expect(await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).toHaveLength(1);+ expect(await engine.executeRaw('SELECT id FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId])).toHaveLength(0);+ await engine.executeRaw('DELETE FROM retire_stall');+ const receipt = await remove();+ expect(receipt).toMatchObject({ state: 'committed', operation: 'remove', pages_deleted: left });+ expect(await remove()).toEqual(receipt);+ expect(await counts(engine, source)).toEqual({ pages: 0, chunks: 0, links: 0 });+ expect(await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).toHaveLength(0);+ }), 300_000);+});diff --git a/test/helpers/persistence-postgres.ts b/test/helpers/persistence-postgres.tsindex b1da8095a1..b945e4407d 100644--- a/test/helpers/persistence-postgres.ts+++ b/test/helpers/persistence-postgres.ts@@ -20,8 +20,9 @@ export async function isolatedPersistencePostgres(databaseUrl:string, connection* One migrated template per test file: each clone (CREATE DATABASE ... TEMPLATE) is its own fresh brain,* so permanent receipt IDs never cross tests, without re-running every migration per test.*/-export async function persistencePostgresTemplate(databaseUrl:string, poolSize=4):Promise<{clone:()=>Promise<{engine:PostgresEngine;close:()=>Promise<void>}>;dispose:()=>Promise<void>}> {+export async function persistencePostgresTemplate(databaseUrl:string, poolSize=4, prepare?:(engine:PostgresEngine)=>Promise<void>):Promise<{clone:()=>Promise<{engine:PostgresEngine;close:()=>Promise<void>}>;dispose:()=>Promise<void>}> {const template=await isolatedPersistencePostgres(databaseUrl);+ try { await prepare?.(template.engine); } catch (error) { await template.close(); throw error; }await template.engine.disconnect();const name=new URL(template.databaseUrl).pathname.slice(1);const admin=postgres(databaseUrl,{max:1,prepare:false});diff --git a/test/source-retire-batched.test.ts b/test/source-retire-batched.test.tsnew file mode 100644index 0000000000..78e9f50251--- /dev/null+++ b/test/source-retire-batched.test.ts@@ -0,0 +1,108 @@+/**+ * #7702 / #8905 on PGLite: a managed remove or purge deletes a source's pages in bounded batches behind the same+ * preconditions as its final transaction, never deletes for a dry run or a refused operation, and a retry with the+ * same request_id resumes after an interrupted batch.+ */+import { afterAll, beforeAll, beforeEach, expect, test } from 'bun:test';+import { randomUUID } from 'node:crypto';+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';+import { join } from 'node:path';+import { tmpdir } from 'node:os';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { resetPgliteState } from './helpers/reset-pglite.ts';+import { withEnv } from './helpers/with-env.ts';+import { registerLocalWriter } from '../src/core/persistence/identity.ts';+import { claimWorktree } from '../src/core/persistence/ownership.ts';+import { runManagedSourceLifecycle, SOURCE_RETIRE_BATCH_PAGES } from '../src/core/persistence/source-lifecycle.ts';+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';++let engine: PGLiteEngine;++beforeAll(async () => {+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+});++afterAll(async () => {+ await disposePersistenceConsumer(engine);+ await engine.disconnect();+});++beforeEach(async () => {+ await resetPgliteState(engine);+});++const PAGES = SOURCE_RETIRE_BATCH_PAGES * 2 + 17;+const source = 'retire-batched';++async function fixture(run: () => Promise<void>) {+ const home = mkdtempSync(join(tmpdir(), 'gbrain-retire-batched-'));+ try {+ await withEnv({ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined }, async () => {+ await registerLocalWriter(engine, 'cli');+ for (const id of [source, 'keep-other']) {+ const root = join(home, id); mkdirSync(root);+ writeFileSync(join(root, 'example.md'), '---\ntitle: Example\ntype: note\n---\n\nCanonical\n');+ await engine.executeRaw('INSERT INTO sources(id,name,local_path) VALUES($1,$1,$2)', [id, root]);+ await claimWorktree(engine, id, root);+ await engine.executeRaw(`INSERT INTO pages(source_id,slug,type,title,compiled_truth)+ SELECT $1,'notes/p-'||g,'note','Page '||g,'Body '||g FROM generate_series(1,$2::int) g`, [id, id === source ? PAGES : 20]);+ await engine.executeRaw(`INSERT INTO content_chunks(page_id,chunk_index,chunk_text)+ SELECT id,0,'chunk of '||slug FROM pages WHERE source_id=$1`, [id]);+ }+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');+ await run();+ expect(await pages('keep-other')).toBe(20);+ });+ } finally { rmSync(home, { recursive: true, force: true }); }+}+const pages = async (id: string) => (await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM pages WHERE source_id=$1', [id]))[0].n;+const registered = async () => (await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).length;++test('remove and purge of a source larger than one batch delete every page and count them all', () => fixture(async () => {+ await runManagedSourceLifecycle(engine, { operation: 'archive', sourceId: source });+ expect(await runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true }))+ .toMatchObject({ state: 'committed', operation: 'purge', pages_deleted: PAGES });+ expect(await registered()).toBe(0);+ expect(await pages(source)).toBe(0);+ expect((await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM content_chunks c LEFT JOIN pages p ON p.id=c.page_id WHERE p.id IS NULL`))[0].n).toBe(0);+}), 60_000);++test('a dry run or a refused remove/purge deletes no page', () => fixture(async () => {+ await runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, dryRun: true });+ await expect(runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source })).rejects.toMatchObject({ code: 'invalid_params' });+ await expect(runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true })).rejects.toMatchObject({ code: 'invalid_params' });+ await runManagedSourceLifecycle(engine, { operation: 'archive', sourceId: source });+ expect(await runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true, expiredOnly: true }))+ .toMatchObject({ state: 'committed', noop: true });+ await expect(runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, expectedIncarnation: randomUUID() }))+ .rejects.toMatchObject({ code: 'source_changed' });+ expect(await pages(source)).toBe(PAGES);+ expect(await registered()).toBe(1);+}), 60_000);++test('an interrupted batch keeps the committed batches and the same request_id resumes to completion', () => fixture(async () => {+ await engine.executeRaw('CREATE TABLE retire_fail(armed boolean NOT NULL, seen integer NOT NULL)');+ await engine.executeRaw(`CREATE FUNCTION retire_fail_fn() RETURNS trigger LANGUAGE plpgsql AS $$ DECLARE n integer; BEGIN+ UPDATE retire_fail SET seen=seen+1 WHERE armed RETURNING seen INTO n;+ IF n > ${SOURCE_RETIRE_BATCH_PAGES * 2} THEN RAISE EXCEPTION 'injected retire batch failure'; END IF; RETURN OLD; END $$`);+ await engine.executeRaw('CREATE TRIGGER retire_fail BEFORE DELETE ON pages FOR EACH ROW EXECUTE FUNCTION retire_fail_fn()');+ try {+ await engine.executeRaw('INSERT INTO retire_fail VALUES(true,0)');+ const requestId = randomUUID();+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });+ await expect(remove()).rejects.toThrow('injected retire batch failure');+ expect(await pages(source)).toBe(PAGES - SOURCE_RETIRE_BATCH_PAGES * 2);+ expect(await registered()).toBe(1);+ expect(await engine.executeRaw('SELECT id FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId])).toHaveLength(0);+ await engine.executeRaw('DELETE FROM retire_fail');+ const receipt = await remove();+ expect(receipt).toMatchObject({ state: 'committed', operation: 'remove', pages_deleted: PAGES - SOURCE_RETIRE_BATCH_PAGES * 2 });+ expect(await remove()).toEqual(receipt);+ expect(await pages(source)).toBe(0);+ expect(await registered()).toBe(0);+ } finally {+ for (const sql of ['DROP TRIGGER IF EXISTS retire_fail ON pages', 'DROP FUNCTION IF EXISTS retire_fail_fn()', 'DROP TABLE IF EXISTS retire_fail']) await engine.executeRaw(sql);+ }+}), 60_000);