Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Batch managed source remove/purge page deletes under the topology bound

garrytan8c5ee9e2d654
diff --git a/docs/architecture/canonical-writers.tsv b/docs/architecture/canonical-writers.tsv
index d546f89093..88c627d803 100644
--- a/docs/architecture/canonical-writers.tsv
+++ b/docs/architecture/canonical-writers.tsv
@@ -161,7 +161,7 @@ src/core/persistence/sync-image.ts 2 coordinator The managed sync image branch p
src/core/persistence/sync-run.ts 1 coordinator The managed up-to-date heartbeat stamps only sources.last_sync_at inside withCoordinatedWrite, scoped to the checked source id and incarnation; no checkpoint, commit or content-freshness column changes.
src/core/persistence/sync-pass-run.ts 2 coordinator #6712 PR 4b: the pass's up-to-date and no-checkpoint stamps write only sources.last_sync_at inside withCoordinatedWrite, after assertActive and assertDrainLease, scoped to the source id and incarnation; no checkpoint, commit or content-freshness column changes.
src/commands/enrich.ts 0 coordinator A managed brain preflights maintenance authority before any model spend and publishes each enriched page through publishMaintenancePage; unmanaged enrichment writes through the put_page operation.
-src/core/persistence/source-lifecycle.ts 7 coordinator Native affected roots precede topology, source, principal and receipt guards; every shared-root generation and old queued intent changes together.
+src/core/persistence/source-lifecycle.ts 8 coordinator Native affected roots precede topology, source, principal and receipt guards; every shared-root generation and old queued intent changes together. A remove or purge first deletes the source's pages in bounded batches under the same native locks, each batch behind the final transaction's preconditions (#7702/#8905).
src/core/persistence/topology-recovery.ts 2 coordinator Verified directory-swap recovery checks source incarnation, current logical manifest and physical identity before topology commitment; permanent receipt IDs survive deletion.
src/core/persistence/connector-sync.ts 4 coordinator Connector freshness and source-scoped deletion commit through original/current authority, source and revision guards, canonical publication and checkpoint CAS. A skipped unchanged checkpoint save stamps sources.last_sync_at/newest_content_at (never page content) inside withCoordinatedWrite, in a transaction that first asserts the connector sync lease, scoped to the source incarnation (#5686/#5470). Its importFromContent calls carry prepare and run inside the coordinator publication.
src/core/persistence/prepared-maintenance.ts 5 coordinator Named local maintenance uses native owner publication; contributing semantic facts and take lifecycle are revalidated and committed together with canonical files and receipts; a chronicle event page and its depth timeline projection commit in one publication; v0.32.2 fact adoption re-locks the pinned legacy facts and their fence positions and sets their row positions ahead of the canonical projection in the same publication.; consolidation closes a fenced fact from the fence cell it publishes with the take, an unfenced one in the database (G4)
diff --git a/docs/architecture/coordinated-writes.tsv b/docs/architecture/coordinated-writes.tsv
index 79be8aeaa5..acafa38ae9 100644
--- a/docs/architecture/coordinated-writes.tsv
+++ b/docs/architecture/coordinated-writes.tsv
@@ -21,7 +21,7 @@ src/core/persistence/connector-sync.ts stampFreshness 1 operational Stamps last_
src/core/persistence/coordinator.ts publishMutation 2 door The page-mutation door itself: the prepared publication and its after-effects commit here after the file is in place.
src/core/persistence/group-publish.ts publishGroup 1 door Group publication of door requests: the same prepared publications as publishMutation, committed as one group.
src/core/persistence/links-maintenance.ts derive 1 projection Re-derives markdown, frontmatter and mention edges from each page's current text, bound to the revision it read; the indexer reproduces them.
-src/core/persistence/source-lifecycle.ts runManagedSourceLifecycle 1 operational Source topology changes (remove, purge, archive) on the sources and binding rows; source administration, not memory content.
+src/core/persistence/source-lifecycle.ts runManagedSourceLifecycle 2 operational Source topology changes (remove, purge, archive) on the sources and binding rows, and the bounded page-deletion batches a remove or purge runs first (#7702/#8905); source administration, not memory content.
src/core/persistence/sync-pass-run.ts stampIdlePass 1 operational An idle sync pass stamps last_sync_at and its pass row; sync bookkeeping, not memory content.
src/core/persistence/sync-pass-run.ts stampUnchangedPass 1 operational An unchanged sync pass stamps last_sync_at and its converged watermark; sync bookkeeping, not memory content.
src/core/persistence/sync-run.ts runManagedSync 1 operational A sync that found nothing stamps the freshness heartbeat and releases holds for paths that left the source; sync bookkeeping, not memory content.
diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.md
index d774948873..bc157b81cd 100644
--- a/docs/architecture/key-files/core-persistence-continued.md
+++ b/docs/architecture/key-files/core-persistence-continued.md
@@ -41,3 +41,4 @@ The #6188 fence check and fence repair live in [fence repair](core-fence-repair.
- `src/core/postgres-engine/reserved-transactions.ts` with `isPersistenceDeclaration` (`src/core/persistence/protocol.ts`) — BEGIN pipelining. `isPersistenceDeclaration` (GBRA-156) accepts exactly the `protocol.ts` declarations (transaction-local `set_config` calls and the brain row FOR SHARE). Admission, the FIFO claim, the effect claims and source guards, the receipt true-up, the single publication (`coordinator.ts`) and the consumer's lent lane transactions (`consumer.ts` `onLane`, through `reservedTransactions`) pass it as `TransactionOptions.pipelineFirst`, so behind a transaction-mode pooler the declaration goes out with BEGIN (one round trip less per transaction) while every later statement still waits for BEGIN's answer; if BEGIN fails, only the declaration has run, in autocommit, where it changes nothing. A reserved transaction gates every statement its body can reach, savepoint and nested-transaction bodies and `file()` included. A sync lane group's publication passes none: its begun mark must follow BEGIN's answer (`sync-lanes.ts` `awaitLaneBegin`). Tests: `test/persistence-pipeline-first.test.ts` (the predicate and the reviewed call sites), `test/e2e/pgbouncer-transaction-pipeline-6930.test.ts`.
- `src/core/links-fence.ts` + `src/core/links-fence-projection.ts` — the `## Links` fence (G5): the markdown form of authored (manual) edges, `<!--- gbrain:links:begin -->` / `end` around `| to | type | source | valid_from | valid_until | context |`, one row per (to, type, source), source defaulting to `manual`; a reconciliation-managed provenance (`MANAGED_LINK_SOURCES`, exported here and re-exported by `ops/links.ts`), a bad date or a duplicate key is a skipped row with a warning. `parseLinksFence` (state `absent` / `parsed` / `malformed`), the deterministic `renderLinksFence` (rows sorted by key), `upsertLinksFenceRow` (omitted context and dates keep the stored ones; identical row = unchanged body), `removeLinksFenceRows` (leaves an empty fence), `replaceOrInsertLinksFence` (a `## Links` section at the end of compiled_truth, above an unclosed code block) and `maskLinksFence`, which `extractBodyLinks` applies so the fence cells are never also extracted as markdown, wikilink or bare-slug edges. A rewrite of a fence that parses with warnings throws `LinksFenceRewriteRefusal`. Cells use `escapeFenceCell` plus the same one-level marker encoding for `gbrain:links:*`. `projectLinksFence(tx, page, rows)` reconciles only what the page's fence owns: edges with origin_page_id = the page and origin_field `links_fence` (deleted, context-updated, inserted by key) and `manual` transitions with origin_page_id = the page, then refreshes the touched relationships; a row whose target is not a live page in the source yields no edge. `linksFenceProjectionRows` returns null for an absent or malformed fence, so such a body changes no edge (an absent fence retires nothing, as for facts; unlike facts no held generation is recorded, so after a hand deletion of the whole section the database keeps edges git no longer has until `remove_link` or the G3 indexer), and drops rows typed with a link type the active pack does not declare. Callers: `prepareCanonicalProjections` (every projected page publication, quarantined and segmented-unread bodies excepted), `prepareAutomaticLinks` (put_page auto-links and the managed links maintenance, so `extract --stale` restores a lost fence edge) and `persistence/manual-links.ts`. Database-only manual edges (origin_field NULL) are never touched by any of them. Pinned by `test/links-fence.test.ts` and `test/managed-links-fence.test.ts`.
+- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`) and, when one would refuse or replay, stops without deleting so the final transaction reports it; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. Tests: `test/source-retire-batched.test.ts` (PGLite) and `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages).
diff --git a/skills/data-loss-gate/SKILL.md b/skills/data-loss-gate/SKILL.md
index 59805d6047..148b9e4fdb 100644
--- a/skills/data-loss-gate/SKILL.md
+++ b/skills/data-loss-gate/SKILL.md
@@ -93,7 +93,9 @@ Brain / database level (gbrain-specific):
`--yes` only after the user agrees.
- **Source removal** — `gbrain sources remove <id>` deletes the source AND
every page in it. The `--confirm-destructive` flag does not substitute for
- the card.
+ the card. A managed removal deletes pages in committed batches: one that
+ stops early (`write_pending`) has already deleted some, so say so and rerun it
+ to finish.
- **Mount removal** — `gbrain mounts remove <id>` only removes the local
registration (the mounted brain's database survives; re-add to recover). Gate
it anyway when the flow ALSO plans to delete the mount's underlying database
diff --git a/skills/skills.lock.json b/skills/skills.lock.json
index 76abef9092..0a50cead1c 100644
--- a/skills/skills.lock.json
+++ b/skills/skills.lock.json
@@ -71,7 +71,7 @@
"daily-task-manager/SKILL.md": "e93264a83b5fd0a19b2f06a322b3a4ad92d418f6a968c1cd51f4bea79cbca4d9",
"daily-task-manager/routing-eval.jsonl": "e98b2592e65c26952242436a960d8b2061e375b269004a18c230cbcf1d692ff9",
"daily-task-prep/SKILL.md": "5ea839012b48761ed8d7a9bab056a2566587d6f7fe044e3907d77b20f8eec5ac",
- "data-loss-gate/SKILL.md": "c1736e0985121e60ffeba993b68b5c722c9285220057b5ef72059f111aabcb18",
+ "data-loss-gate/SKILL.md": "656d7e108a88fa3ac7d7d41dfb79f02b7dc592d286a7cd6a7fa63503550e49ba",
"data-loss-gate/routing-eval.jsonl": "825358a5600641061a02b25097cd84c1c28bb0359a7608a10b9f25f661bd88f9",
"data-research/SKILL.md": "39f9648a21a7074d6ad7db789e93e6d7085449a6cdaffacf8b1660eb68636b9d",
"db-repair/SKILL.md": "87a34fe18812f3b6ec37381ae69c205e30747300098ebe3fe41f8460d0ba8346",
diff --git a/src/core/persistence/source-lifecycle.ts b/src/core/persistence/source-lifecycle.ts
index e8ccb10596..a3b295d6af 100644
--- a/src/core/persistence/source-lifecycle.ts
+++ b/src/core/persistence/source-lifecycle.ts
@@ -16,7 +16,7 @@ import { isInsideGitRepo, hasTrackedContent } from '../git-remote.ts';
import { containsPath, getWorktreeBinding, humanManifestProgress, type WorktreeBinding, type WorktreeManifest, worktreeManifest } from './ownership.ts';
import { manifestDifferences, storedManifestScope, type WorktreeManifestScope } from './worktree-manifest.ts';
import { hostIdFor } from './identity.ts';
-import { advanceTopology, inspectTopologyPhysicalRoot, lockTopologyPrincipal, lockTopologyRows, settleTopologyRequests, topologyCanonicalStamp, topologyPrincipal, withTopologyLocks } from './topology-locks.ts';
+import { advanceTopology, assertTopologyRequestsSettled, inspectTopologyPhysicalRoot, lockTopologyPrincipal, lockTopologyRows, settleTopologyRequests, topologyCanonicalStamp, topologyPrincipal, withTopologyLocks } from './topology-locks.ts';
import { priorTopologyChange, recordTopologyChange, topologyReceipt } from './topology-receipts.ts';
import { isWriteRequestId } from './types.ts';
import { withCoordinatedWrite } from './context.ts';
@@ -42,6 +42,8 @@ export interface SourceLifecycleInput {
expectedAdminState?:string;
automaticClaim?:boolean;
}
+/** #7702/#8905: pages one retire batch deletes; each batch (with its cascaded chunks, links and versions) stays far under the 5 s topology statement bound. */
+export const SOURCE_RETIRE_BATCH_PAGES=500;
interface SourceState {id:string;incarnation:string;archived:boolean;local_path:string|null;config:Record<string,unknown>;name:string;last_commit:string|null;}
function sourcesListFix():Action{
@@ -181,6 +183,26 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceL
if(input.operation==='rebind'&&root&&currentRoot&&currentRoot!==root.worktree&&manifests.has(currentRoot)&&manifests.has(root.worktree)
&&storedManifestScope(manifests.get(currentRoot))!==storedManifestScope(manifests.get(root.worktree)))
manifests.set(root.worktree,worktreeManifest(root.worktree,{progress:humanManifestProgress(),scope:storedManifestScope(manifests.get(currentRoot))}));
+ // #7702/#8905: a remove or purge first deletes the source's pages in bounded batches, each its own transaction behind
+ // the final transaction's preconditions; any precondition that would refuse or replay stops batching and the final
+ // transaction reports it. Committed batches are durable, so a retry with the same request_id resumes.
+ let retiredPages=0;
+ if(['remove','purge'].includes(input.operation)&&input.confirmDestructive&&!missingCheckout&&!admission)
+ for(let deleted=SOURCE_RETIRE_BATCH_PAGES;deleted===SOURCE_RETIRE_BATCH_PAGES;) deleted=await topologyTransaction(engine,async tx=>{
+ await assertWriterAdminState(tx,input.expectedAdminState);
+ const sources=await lockTopologyRows(tx,input.sourceId,bindings);
+ const [source]=await tx.executeRaw<{incarnation:string;archived:boolean;due:boolean}>(
+ `SELECT incarnation,archived,${input.expiredOnly?EXPIRED_ARCHIVE_SQL:'true'} AS due FROM sources WHERE id=$1`,[input.sourceId]);
+ if(source?.incarnation!==before!.incarnation||!source.due||input.operation==='purge'&&!input.expiredOnly&&!source.archived
+ ||await priorTopologyChange(tx,principal,requestId,intent)
+ ||(await tx.executeRaw('SELECT client_id FROM oauth_clients WHERE source_id=$1 LIMIT 1',[input.sourceId])).length) return 0;
+ await assertTopologyRequestsSettled(tx,sources,bindings.map(binding=>binding.worktree_id));
+ const gone=await withCoordinatedWrite(tx,sources,()=>tx.executeRaw(
+ 'DELETE FROM pages WHERE id IN (SELECT id FROM pages WHERE source_id=$1 LIMIT $2) RETURNING id',[input.sourceId,SOURCE_RETIRE_BATCH_PAGES]),
+ principalAttribution({kind:'local_cli',id:principal}));
+ retiredPages+=gone.length;
+ return gone.length;
+ });
return topologyTransaction(engine,async tx=>{
await assertWriterAdminState(tx,input.expectedAdminState);
await tx.executeRaw("SELECT set_config('synchronous_commit','on',true)");
@@ -248,7 +270,7 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceL
await tx.executeRaw("SELECT set_config('gbrain.topology_change','on',true)");
const result=await withCoordinatedWrite(tx,sources,async()=>{
let incarnation=source?.incarnation??randomUUID();
- let pagesDeleted=0;
+ let pagesDeleted=retiredPages;
if(input.operation==='add'||input.operation==='claim'){
if(root&&input.createDirectory&&!existsSync(root.source)){mkdirSync(root.source,{recursive:true});flushTopologyDirectory(dirname(root.source));}
if(source) await tx.executeRaw("UPDATE sources SET local_path=$2,name=COALESCE($3,name),config=$4::text::jsonb WHERE id=$1",
@@ -278,7 +300,7 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceL
`Nothing was removed. OAuth client ${refs[0].client_id} (and possibly others) is bound to '${input.sourceId}'; list them with the command in fix and ask the user before revoking any, since revoking disconnects that client.`,
{fix:readFix('Lists OAuth clients with the source each is bound to.',{argv:['gbrain','auth','clients','--json']})});
const [impact]=await tx.executeRaw<{count:string}>('SELECT count(*)::text AS count FROM pages WHERE source_id=$1',[input.sourceId]);
- pagesDeleted=Number(impact.count);
+ pagesDeleted+=Number(impact.count);
await tx.executeRaw('DELETE FROM sources WHERE id=$1 AND incarnation=$2::uuid',[input.sourceId,incarnation]);
await tx.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1 AND source_incarnation=$2::uuid',[input.sourceId,incarnation]);
}
diff --git a/src/core/persistence/topology-locks.ts b/src/core/persistence/topology-locks.ts
index f8c61799dc..c00fc6de70 100644
--- a/src/core/persistence/topology-locks.ts
+++ b/src/core/persistence/topology-locks.ts
@@ -147,8 +147,8 @@ export async function topologyCanonicalStamp(tx:BrainEngine,worktreeId:string):P
return row.stamp;
}
-/** No provider or filesystem wait occurs while these database guards are held. */
-export async function settleTopologyRequests(tx: BrainEngine, sources: string[], worktrees: string[], principal: string): Promise<number> {
+/** Read-only half of settleTopologyRequests: refuses while a publication, recovery or blocking effect is unfinished. */
+export async function assertTopologyRequestsSettled(tx: BrainEngine, sources: string[], worktrees: string[]): Promise<WriteRequest[]> {
const requests = await tx.executeRaw<WriteRequest>(`SELECT * FROM persistence_requests
WHERE (source_id=ANY($1::text[]) OR worktree_id=ANY($2::uuid[]))
AND (state IN ('queued','deferred','running','recovering') OR recovery IS NOT NULL) ORDER BY sequence`, [sources,worktrees]);
@@ -167,6 +167,12 @@ export async function settleTopologyRequests(tx: BrainEngine, sources: string[],
`Effect ${blocked.id} on source ${source} (a queued withdrawal mirror, a running effect or one holding a recovery record) has not finished, so the lifecycle change rolled back and nothing changed. The owner's effect worker settles it; check it with the command in fix, which lists it with its page and request. Then run the same lifecycle command again with the same request_id.`,
{fix:ownerStatusFix(source)});
}
+ return requests;
+}
+
+/** No provider or filesystem wait occurs while these database guards are held. */
+export async function settleTopologyRequests(tx: BrainEngine, sources: string[], worktrees: string[], principal: string): Promise<number> {
+ const requests = await assertTopologyRequestsSettled(tx, sources, worktrees);
const pending = requests.filter(row => row.state === 'queued' || row.state === 'deferred');
// Revocation and publication use the same principal rows. Administrative
// invalidation may terminate revoked work but still respects guard ordering.
diff --git a/src/core/persistence/topology-transaction.ts b/src/core/persistence/topology-transaction.ts
index 0cf16c27a6..057957c947 100644
--- a/src/core/persistence/topology-transaction.ts
+++ b/src/core/persistence/topology-transaction.ts
@@ -29,7 +29,7 @@ export async function topologyTransaction<T>(engine: BrainEngine, run: (tx: Brai
});
} catch (error) {
if (['55P03','57014'].includes((error as { code?: string }).code ?? '')) {
- throw new OperationError('write_pending', 'The source transaction exceeded its bounded database wait.', 'Retry the same lifecycle request_id after the current writer finishes.');
+ throw new OperationError('write_pending', 'The source transaction exceeded its bounded database wait.', 'Retry the same lifecycle request_id after the current writer finishes; work it already committed, such as the pages a source removal deleted, is kept and the retry resumes from there.');
}
throw error;
} finally { release(); }
diff --git a/test/e2e/source-retire-batched-postgres.test.ts b/test/e2e/source-retire-batched-postgres.test.ts
new file mode 100644
index 0000000000..830ddeec83
--- /dev/null
+++ b/test/e2e/source-retire-batched-postgres.test.ts
@@ -0,0 +1,108 @@
+/**
+ * #7702 / #8905: a managed remove or purge of a large source deletes its pages in bounded batches, each in its own
+ * short topology transaction, so no statement meets the 5 s topology bound; a retry with the same request_id
+ * resumes after an interrupted batch and finishes.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { randomUUID } from 'node:crypto';
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { tmpdir } from 'node:os';
+import type { BrainEngine } from '../../src/core/engine.ts';
+import { registerLocalWriter } from '../../src/core/persistence/identity.ts';
+import { claimWorktree } from '../../src/core/persistence/ownership.ts';
+import { runManagedSourceLifecycle } from '../../src/core/persistence/source-lifecycle.ts';
+import { disposePersistenceConsumer } from '../../src/core/persistence/service.ts';
+import { persistencePostgresTemplate } from '../helpers/persistence-postgres.ts';
+import { requirePostgresTestDatabase } from '../helpers/test-backends.ts';
+import { withEnv } from '../helpers/with-env.ts';
+
+const PAGES = 60_000;
+const STALL_AFTER = PAGES / 2;
+const databaseUrl = requirePostgresTestDatabase();
+let template: Awaited<ReturnType<typeof persistencePostgresTemplate>>;
+beforeAll(async () => {
+ template = await persistencePostgresTemplate(databaseUrl, 4, async engine => {
+ for (const id of ['retire-large', 'keep-other']) await engine.executeRaw('INSERT INTO sources(id,name) VALUES($1,$1)', [id]);
+ await seed(engine, 'retire-large', PAGES);
+ await seed(engine, 'keep-other', 50);
+ });
+}, 600_000);
+afterAll(async () => { await template?.dispose(); });
+
+async function seed(engine: BrainEngine, source: string, pages: number) {
+ for (let from = 1; from <= pages; from += 5000) await engine.executeRaw(`INSERT INTO pages(source_id,slug,type,title,compiled_truth)
+ SELECT $1,'notes/p-'||g,'note','Page '||g,'Body of page '||g FROM generate_series($2::int,LEAST($2::int+4999,$3::int)) g`, [source, from, pages]);
+ await engine.executeRaw(`INSERT INTO content_chunks(page_id,chunk_index,chunk_text)
+ SELECT p.id,c,'chunk '||c||' of '||p.slug FROM pages p CROSS JOIN generate_series(0,2) c WHERE p.source_id=$1`, [source]);
+ await engine.executeRaw(`INSERT INTO links(from_page_id,to_page_id) SELECT p.id,q.id FROM pages p
+ JOIN pages q ON q.source_id=p.source_id AND q.slug='notes/p-'||(split_part(p.slug,'-',2)::int % $2::int + 1) WHERE p.source_id=$1`, [source, pages]);
+}
+async function counts(engine: BrainEngine, source: string) {
+ const [row] = await engine.executeRaw<{ pages: number; chunks: number; links: number }>(`SELECT
+ (SELECT count(*)::int FROM pages WHERE source_id=$1) AS pages,
+ (SELECT count(*)::int FROM content_chunks c JOIN pages p ON p.id=c.page_id WHERE p.source_id=$1) AS chunks,
+ (SELECT count(*)::int FROM links l JOIN pages p ON p.id=l.from_page_id WHERE p.source_id=$1) AS links`, [source]);
+ return row;
+}
+
+describe('managed remove/purge of a large source (postgres)', () => {
+ async function fixture(run: (engine: BrainEngine, source: string) => Promise<void>) {
+ const { engine, close } = await template.clone();
+ const home = mkdtempSync(join(tmpdir(), 'gbrain-retire-batched-'));
+ try {
+ await withEnv({ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined }, async () => {
+ await registerLocalWriter(engine, 'cli');
+ for (const id of ['retire-large', 'keep-other']) {
+ const root = join(home, id); mkdirSync(root);
+ writeFileSync(join(root, 'example.md'), '---\ntitle: Example\ntype: note\n---\n\nCanonical\n');
+ await engine.executeRaw('UPDATE sources SET local_path=$2 WHERE id=$1', [id, root]);
+ await claimWorktree(engine, id, root);
+ }
+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');
+ await run(engine, 'retire-large');
+ expect(await counts(engine, 'keep-other')).toEqual({ pages: 50, chunks: 150, links: 50 });
+ });
+ } finally { await disposePersistenceConsumer(engine); await close(); rmSync(home, { recursive: true, force: true }); }
+ }
+
+ test('remove deletes every page, chunk and link of a source too large for one bounded statement', () => fixture(async (engine, source) => {
+ const started = performance.now();
+ const receipt = await runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true })
+ .finally(() => console.log(`[retire-batched] remove of ${PAGES} pages: ${Math.round(performance.now() - started)} ms`));
+ expect(receipt).toMatchObject({ state: 'committed', operation: 'remove', pages_deleted: PAGES });
+ expect(await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).toHaveLength(0);
+ expect(await engine.executeRaw('SELECT source_id FROM persistence_source_bindings WHERE source_id=$1', [source])).toHaveLength(0);
+ expect(await counts(engine, source)).toEqual({ pages: 0, chunks: 0, links: 0 });
+ }), 300_000);
+
+ test('purge of an archived large source completes', () => fixture(async (engine, source) => {
+ await runManagedSourceLifecycle(engine, { operation: 'archive', sourceId: source });
+ const receipt = await runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true });
+ expect(receipt).toMatchObject({ state: 'committed', operation: 'purge', pages_deleted: PAGES });
+ expect(await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).toHaveLength(0);
+ expect(await counts(engine, source)).toEqual({ pages: 0, chunks: 0, links: 0 });
+ }), 300_000);
+
+ test('a batch that times out keeps earlier batches and the same request_id resumes to completion', () => fixture(async (engine, source) => {
+ await engine.executeRaw('CREATE TABLE retire_stall(armed boolean NOT NULL, seen integer NOT NULL)');
+ await engine.executeRaw(`CREATE FUNCTION retire_stall_fn() RETURNS trigger LANGUAGE plpgsql AS $$ DECLARE n integer; BEGIN
+ UPDATE retire_stall SET seen=seen+1 WHERE armed RETURNING seen INTO n;
+ IF n = ${STALL_AFTER + 1} THEN PERFORM pg_sleep(6); END IF; RETURN OLD; END $$`);
+ await engine.executeRaw('CREATE TRIGGER retire_stall BEFORE DELETE ON pages FOR EACH ROW EXECUTE FUNCTION retire_stall_fn()');
+ await engine.executeRaw('INSERT INTO retire_stall VALUES(true,0)');
+ const requestId = randomUUID();
+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });
+ await expect(remove()).rejects.toMatchObject({ code: 'write_pending' });
+ const left = (await counts(engine, source)).pages;
+ expect(left).toBe(PAGES - STALL_AFTER);
+ expect(await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).toHaveLength(1);
+ expect(await engine.executeRaw('SELECT id FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId])).toHaveLength(0);
+ await engine.executeRaw('DELETE FROM retire_stall');
+ const receipt = await remove();
+ expect(receipt).toMatchObject({ state: 'committed', operation: 'remove', pages_deleted: left });
+ expect(await remove()).toEqual(receipt);
+ expect(await counts(engine, source)).toEqual({ pages: 0, chunks: 0, links: 0 });
+ expect(await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).toHaveLength(0);
+ }), 300_000);
+});
diff --git a/test/helpers/persistence-postgres.ts b/test/helpers/persistence-postgres.ts
index b1da8095a1..b945e4407d 100644
--- a/test/helpers/persistence-postgres.ts
+++ b/test/helpers/persistence-postgres.ts
@@ -20,8 +20,9 @@ export async function isolatedPersistencePostgres(databaseUrl:string, connection
* One migrated template per test file: each clone (CREATE DATABASE ... TEMPLATE) is its own fresh brain,
* so permanent receipt IDs never cross tests, without re-running every migration per test.
*/
-export async function persistencePostgresTemplate(databaseUrl:string, poolSize=4):Promise<{clone:()=>Promise<{engine:PostgresEngine;close:()=>Promise<void>}>;dispose:()=>Promise<void>}> {
+export async function persistencePostgresTemplate(databaseUrl:string, poolSize=4, prepare?:(engine:PostgresEngine)=>Promise<void>):Promise<{clone:()=>Promise<{engine:PostgresEngine;close:()=>Promise<void>}>;dispose:()=>Promise<void>}> {
const template=await isolatedPersistencePostgres(databaseUrl);
+ try { await prepare?.(template.engine); } catch (error) { await template.close(); throw error; }
await template.engine.disconnect();
const name=new URL(template.databaseUrl).pathname.slice(1);
const admin=postgres(databaseUrl,{max:1,prepare:false});
diff --git a/test/source-retire-batched.test.ts b/test/source-retire-batched.test.ts
new file mode 100644
index 0000000000..78e9f50251
--- /dev/null
+++ b/test/source-retire-batched.test.ts
@@ -0,0 +1,108 @@
+/**
+ * #7702 / #8905 on PGLite: a managed remove or purge deletes a source's pages in bounded batches behind the same
+ * preconditions as its final transaction, never deletes for a dry run or a refused operation, and a retry with the
+ * same request_id resumes after an interrupted batch.
+ */
+import { afterAll, beforeAll, beforeEach, expect, test } from 'bun:test';
+import { randomUUID } from 'node:crypto';
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { tmpdir } from 'node:os';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { withEnv } from './helpers/with-env.ts';
+import { registerLocalWriter } from '../src/core/persistence/identity.ts';
+import { claimWorktree } from '../src/core/persistence/ownership.ts';
+import { runManagedSourceLifecycle, SOURCE_RETIRE_BATCH_PAGES } from '../src/core/persistence/source-lifecycle.ts';
+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await disposePersistenceConsumer(engine);
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+});
+
+const PAGES = SOURCE_RETIRE_BATCH_PAGES * 2 + 17;
+const source = 'retire-batched';
+
+async function fixture(run: () => Promise<void>) {
+ const home = mkdtempSync(join(tmpdir(), 'gbrain-retire-batched-'));
+ try {
+ await withEnv({ GBRAIN_HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined }, async () => {
+ await registerLocalWriter(engine, 'cli');
+ for (const id of [source, 'keep-other']) {
+ const root = join(home, id); mkdirSync(root);
+ writeFileSync(join(root, 'example.md'), '---\ntitle: Example\ntype: note\n---\n\nCanonical\n');
+ await engine.executeRaw('INSERT INTO sources(id,name,local_path) VALUES($1,$1,$2)', [id, root]);
+ await claimWorktree(engine, id, root);
+ await engine.executeRaw(`INSERT INTO pages(source_id,slug,type,title,compiled_truth)
+ SELECT $1,'notes/p-'||g,'note','Page '||g,'Body '||g FROM generate_series(1,$2::int) g`, [id, id === source ? PAGES : 20]);
+ await engine.executeRaw(`INSERT INTO content_chunks(page_id,chunk_index,chunk_text)
+ SELECT id,0,'chunk of '||slug FROM pages WHERE source_id=$1`, [id]);
+ }
+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');
+ await run();
+ expect(await pages('keep-other')).toBe(20);
+ });
+ } finally { rmSync(home, { recursive: true, force: true }); }
+}
+const pages = async (id: string) => (await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM pages WHERE source_id=$1', [id]))[0].n;
+const registered = async () => (await engine.executeRaw('SELECT id FROM sources WHERE id=$1', [source])).length;
+
+test('remove and purge of a source larger than one batch delete every page and count them all', () => fixture(async () => {
+ await runManagedSourceLifecycle(engine, { operation: 'archive', sourceId: source });
+ expect(await runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true }))
+ .toMatchObject({ state: 'committed', operation: 'purge', pages_deleted: PAGES });
+ expect(await registered()).toBe(0);
+ expect(await pages(source)).toBe(0);
+ expect((await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM content_chunks c LEFT JOIN pages p ON p.id=c.page_id WHERE p.id IS NULL`))[0].n).toBe(0);
+}), 60_000);
+
+test('a dry run or a refused remove/purge deletes no page', () => fixture(async () => {
+ await runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, dryRun: true });
+ await expect(runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source })).rejects.toMatchObject({ code: 'invalid_params' });
+ await expect(runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true })).rejects.toMatchObject({ code: 'invalid_params' });
+ await runManagedSourceLifecycle(engine, { operation: 'archive', sourceId: source });
+ expect(await runManagedSourceLifecycle(engine, { operation: 'purge', sourceId: source, confirmDestructive: true, expiredOnly: true }))
+ .toMatchObject({ state: 'committed', noop: true });
+ await expect(runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, expectedIncarnation: randomUUID() }))
+ .rejects.toMatchObject({ code: 'source_changed' });
+ expect(await pages(source)).toBe(PAGES);
+ expect(await registered()).toBe(1);
+}), 60_000);
+
+test('an interrupted batch keeps the committed batches and the same request_id resumes to completion', () => fixture(async () => {
+ await engine.executeRaw('CREATE TABLE retire_fail(armed boolean NOT NULL, seen integer NOT NULL)');
+ await engine.executeRaw(`CREATE FUNCTION retire_fail_fn() RETURNS trigger LANGUAGE plpgsql AS $$ DECLARE n integer; BEGIN
+ UPDATE retire_fail SET seen=seen+1 WHERE armed RETURNING seen INTO n;
+ IF n > ${SOURCE_RETIRE_BATCH_PAGES * 2} THEN RAISE EXCEPTION 'injected retire batch failure'; END IF; RETURN OLD; END $$`);
+ await engine.executeRaw('CREATE TRIGGER retire_fail BEFORE DELETE ON pages FOR EACH ROW EXECUTE FUNCTION retire_fail_fn()');
+ try {
+ await engine.executeRaw('INSERT INTO retire_fail VALUES(true,0)');
+ const requestId = randomUUID();
+ const remove = () => runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true, requestId });
+ await expect(remove()).rejects.toThrow('injected retire batch failure');
+ expect(await pages(source)).toBe(PAGES - SOURCE_RETIRE_BATCH_PAGES * 2);
+ expect(await registered()).toBe(1);
+ expect(await engine.executeRaw('SELECT id FROM persistence_topology_changes WHERE request_id=$1::uuid', [requestId])).toHaveLength(0);
+ await engine.executeRaw('DELETE FROM retire_fail');
+ const receipt = await remove();
+ expect(receipt).toMatchObject({ state: 'committed', operation: 'remove', pages_deleted: PAGES - SOURCE_RETIRE_BATCH_PAGES * 2 });
+ expect(await remove()).toEqual(receipt);
+ expect(await pages(source)).toBe(0);
+ expect(await registered()).toBe(0);
+ } finally {
+ for (const sql of ['DROP TRIGGER IF EXISTS retire_fail ON pages', 'DROP FUNCTION IF EXISTS retire_fail_fn()', 'DROP TABLE IF EXISTS retire_fail']) await engine.executeRaw(sql);
+ }
+}), 60_000);