Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(sources): retire batches refuse before deleting on a spent receipt quota or a revoked CLI writer, and halve on the statement bound

garrytancaf2f3f5fd24
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8cadb056cf..b56f35741e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,20 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**`gbrain upgrade` no longer consults the ClawHub registry; a large Google address book, and a large source's removal, can now finish.** Advisory: an unofficial package on the ClawHub registry shares the name gbrain. `gbrain upgrade`, and autopilot's unattended self-upgrade, treated any install it could not otherwise classify (for example a release binary run under its published file name, `gbrain-linux-x64`) as a ClawHub install whenever the `clawhub` CLI was on the host, and ran `clawhub update gbrain`, which installs whatever the registry resolves for that name from any publisher. Upgrade now never runs it: a compiled binary is recognised whatever its file name, and an install it cannot classify prints the manual steps. When Google's People sync token expires, the contacts re-list now checkpoints after every page and resumes there, so a large address book no longer restarts from page one in every sync job and Calendar and Gmail sync again. A managed `gbrain sources remove` or `purge` of a large source now deletes its pages in bounded batches under the 5-second statement bound instead of one cascading statement that could never finish, and a rerun resumes. The secret scanner also catches keys right after terminal color codes, `\xNN` and `\UXXXXXXXX` escapes and quoted-printable `=3D`, and redacts a bearer token containing an escaped slash in full.
+
+### Itemized changes
+
+- `src/commands/upgrade.ts`: `detectInstallMethod` drops the `clawhub` method and its `clawhub --version` probe; a process whose `argv[1]` is Bun's compiled entrypoint (`/$bunfs/`, `B:\~BUN\`) is `binary`; the `clawhub` upgrade branch and hint are gone. `src/core/self-upgrade.ts`: `canSelfUpdate` no longer accepts `clawhub`. `src/commands/check-update.ts`: no `clawhub update` hint (#8464).
+- `src/core/google/people-calendar-sweep.ts`, `google-clients.ts`: a full contacts re-list goes page by page (`listConnectionsPage`), saves `contacts_relist` (next page token and field mask) after each page lands through the same state store as Gmail's checkpoints (no git, so repo-less brains work), resumes there, restarts once from page 1 when Google rejects a saved page token (400/410), and lists at most 10 pages per run so Calendar and Gmail still run (#8906).
+- `src/core/persistence/source-lifecycle.ts`: managed `remove`/`purge` deletes pages 500 at a time, each batch its own bounded transaction that first rechecks the final transaction's preconditions (including the CLI writer registration and the receipt quotas, `assertTopologyChangeCapacity`) and halves its size when it hits the statement bound; a refusal after committed batches says how many pages that run deleted. `topology-transaction.ts`: the `write_pending` hint says a retry resumes (#7702, #8905).
+- `src/core/secret-scan.ts`: the escape-boundary twins also fire after `ESC[…m` (raw or written `\u001b[`, `\x1b[`, `\033[`), `\xNN`, `\UXXXXXXXX` and `=3D`; the `bearer` value accepts `\/` (#8266).
+- Tests: `test/upgrade.serial.test.ts`, `test/self-upgrade.test.ts` (a renamed binary or an unclassified install with a working `clawhub` CLI never runs it); `test/contacts-relist-checkpoint-8906.slow.test.ts`, `test/contacts-relist-checkpoint-managed-8906.test.ts`; `test/source-retire-batched.test.ts`, `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages); `test/secret-scan-encoded-boundary-8266.test.ts`.
+
+Reported and first fixed by @javieraldape (#8910, #8764) and @Masashi-Ono0611 (#7704, #8907, #8908).
+
## [0.60.620.0] - 2026-11-09
**A diverged managed checkout now publishes its local commits on refresh by default when Git durability is on, after a secret scan of those commits.** Commits a script made with plain `git commit` in a managed checkout used to strand there whenever the remote moved: every `gbrain sources refresh` refused `refresh_diverged` and asked the user, unless `sources.refresh_publish_diverged` was set to `true`. With the setting unset, a worktree whose Git durability is on (`gbrain sources writer git-durability <source> --enable`, or the legacy hook) now publishes them itself: the local commits merged with the upstream in memory, never a rebase or force-push, then the fast-forward. The publish child scans, under its publish lock and for the exact commits it pushes, what they add: commit messages, added lines (including a value a later commit removed) and merge-commit conflict resolutions. A value that looks like a secret, added text too large to scan, or a commit that changes `.gbrain-scan-allow` or a `.gitattributes` refuses the new `publish_secret_found` naming the file, line, rule and commit, never the value, with nothing pushed; commits the scan cannot read in full refuse `publish_scan_unreadable` instead. The rewritten-upstream refusal and the `--expect-remote` bound are unchanged. `false` turns it off; `true` still publishes with durability off.
diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.md
index bc157b81cd..d4dc45df41 100644
--- a/docs/architecture/key-files/core-persistence-continued.md
+++ b/docs/architecture/key-files/core-persistence-continued.md
@@ -41,4 +41,4 @@ The #6188 fence check and fence repair live in [fence repair](core-fence-repair.
- `src/core/postgres-engine/reserved-transactions.ts` with `isPersistenceDeclaration` (`src/core/persistence/protocol.ts`) — BEGIN pipelining. `isPersistenceDeclaration` (GBRA-156) accepts exactly the `protocol.ts` declarations (transaction-local `set_config` calls and the brain row FOR SHARE). Admission, the FIFO claim, the effect claims and source guards, the receipt true-up, the single publication (`coordinator.ts`) and the consumer's lent lane transactions (`consumer.ts` `onLane`, through `reservedTransactions`) pass it as `TransactionOptions.pipelineFirst`, so behind a transaction-mode pooler the declaration goes out with BEGIN (one round trip less per transaction) while every later statement still waits for BEGIN's answer; if BEGIN fails, only the declaration has run, in autocommit, where it changes nothing. A reserved transaction gates every statement its body can reach, savepoint and nested-transaction bodies and `file()` included. A sync lane group's publication passes none: its begun mark must follow BEGIN's answer (`sync-lanes.ts` `awaitLaneBegin`). Tests: `test/persistence-pipeline-first.test.ts` (the predicate and the reviewed call sites), `test/e2e/pgbouncer-transaction-pipeline-6930.test.ts`.
- `src/core/links-fence.ts` + `src/core/links-fence-projection.ts` — the `## Links` fence (G5): the markdown form of authored (manual) edges, `<!--- gbrain:links:begin -->` / `end` around `| to | type | source | valid_from | valid_until | context |`, one row per (to, type, source), source defaulting to `manual`; a reconciliation-managed provenance (`MANAGED_LINK_SOURCES`, exported here and re-exported by `ops/links.ts`), a bad date or a duplicate key is a skipped row with a warning. `parseLinksFence` (state `absent` / `parsed` / `malformed`), the deterministic `renderLinksFence` (rows sorted by key), `upsertLinksFenceRow` (omitted context and dates keep the stored ones; identical row = unchanged body), `removeLinksFenceRows` (leaves an empty fence), `replaceOrInsertLinksFence` (a `## Links` section at the end of compiled_truth, above an unclosed code block) and `maskLinksFence`, which `extractBodyLinks` applies so the fence cells are never also extracted as markdown, wikilink or bare-slug edges. A rewrite of a fence that parses with warnings throws `LinksFenceRewriteRefusal`. Cells use `escapeFenceCell` plus the same one-level marker encoding for `gbrain:links:*`. `projectLinksFence(tx, page, rows)` reconciles only what the page's fence owns: edges with origin_page_id = the page and origin_field `links_fence` (deleted, context-updated, inserted by key) and `manual` transitions with origin_page_id = the page, then refreshes the touched relationships; a row whose target is not a live page in the source yields no edge. `linksFenceProjectionRows` returns null for an absent or malformed fence, so such a body changes no edge (an absent fence retires nothing, as for facts; unlike facts no held generation is recorded, so after a hand deletion of the whole section the database keeps edges git no longer has until `remove_link` or the G3 indexer), and drops rows typed with a link type the active pack does not declare. Callers: `prepareCanonicalProjections` (every projected page publication, quarantined and segmented-unread bodies excepted), `prepareAutomaticLinks` (put_page auto-links and the managed links maintenance, so `extract --stale` restores a lost fence edge) and `persistence/manual-links.ts`. Database-only manual edges (origin_field NULL) are never touched by any of them. Pinned by `test/links-fence.test.ts` and `test/managed-links-fence.test.ts`.
-- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`) and, when one would refuse or replay, stops without deleting so the final transaction reports it; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. Tests: `test/source-retire-batched.test.ts` (PGLite) and `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages).
+- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`, `lockTopologyPrincipal`, and `assertTopologyChangeCapacity` from `topology-receipts.ts`, the receipt quotas at the largest receipt) and, when one would refuse or replay, stops without deleting so the final transaction reports it, or throws that refusal before deleting; a batch that fails `write_pending` is retried at half the size (floor 1), so heavy pages still fit the bound; a refusal after some batches committed says how many pages that run already deleted; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. Tests: `test/source-retire-batched.test.ts` (PGLite) and `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages).
diff --git a/docs/architecture/key-files/core-utilities-1.md b/docs/architecture/key-files/core-utilities-1.md
index 7a321d7383..7a43f4ab1c 100644
--- a/docs/architecture/key-files/core-utilities-1.md
+++ b/docs/architecture/key-files/core-utilities-1.md
@@ -13,7 +13,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/brain-score-recommendations.ts` — pure data layer consumed by both `gbrain doctor --remediation-plan` / `--remediate` and `gbrain features`. Extraction-watermark staleness produces source-scoped `extract.stale` against database pages, including DB-only pages without a checkout; it does not imply a filesystem sync or block embedding on one. `RecommendationContext.staleExtractionBlocked` (probed by `staleExtractionBlocked()` in `remediation/context.ts`, shared with autopilot) withholds `extract.stale` and classifies `links_extraction_lag` as blocked while the remediated source, or any source with stale pages in an all-source sweep, cannot load its active schema pack, so no dispatch fails every cycle. `computeRecommendations(checks, opts)` returns `Remediation[]` with stable `id`, content-hash `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on` (references stable ids, not check names — so plan order is reproducible). `classifyChecks(report)` triages every doctor check three-state into `remediable | human_only | blocked` (`human_only` covers RLS warnings and other human-judgment gates; `blocked` covers dependency chains where a parent check failed). `maxReachableScore(checks)` computes the ceiling for empty/under-configured brains (no entity pages → graph_coverage caps at 70; no embedding key → embedding_coverage caps at 60). Cost estimates pull from `anthropic-pricing.ts` (synthesize/patterns/consolidate) and `embedding-pricing.ts` (embed jobs). Pinned by `test/brain-score-recommendations.test.ts` (~27 cases incl. determinism, content-hash idempotency, DB-backed checkpoint provenance, three-state triage).
-- `src/core/bun-floor.ts` — the Bun floor of an upgrade target (#5855): `readBunLinkTarget` (`git fetch`, the fetched upstream SHA, `package.json` at that SHA; the swap fast-forwards to that SHA), `readPackageTarget` (`package.json` at the ref a global install's `github:` spec names, default branch when unpinned; ClawHub reads the canonical default branch), `parseReleasePackage` (only a `>=X.Y.Z` `engines.bun` is a floor; any other shape is unreadable), `readHostBun` (the lower of the `bun` on PATH and the Bun at `process.execPath`) and `evaluateBunFloor` (the refusal text with path, and a path-free copy for the self-upgrade audit). `BUN_FLOOR_EXIT_CODE` (78) marks a `gbrain upgrade` refused for the floor, which the autopilot channel treats as a hold. Tests: `bun-floor.test.ts`, `autopilot-bun-floor.test.ts`, `upgrade-bun-link-arc.serial.test.ts`.
+- `src/core/bun-floor.ts` — the Bun floor of an upgrade target (#5855): `readBunLinkTarget` (`git fetch`, the fetched upstream SHA, `package.json` at that SHA; the swap fast-forwards to that SHA), `readPackageTarget` (`package.json` at the ref a global install's `github:` spec names, default branch when unpinned; a null spec reads the canonical default branch), `parseReleasePackage` (only a `>=X.Y.Z` `engines.bun` is a floor; any other shape is unreadable), `readHostBun` (the lower of the `bun` on PATH and the Bun at `process.execPath`) and `evaluateBunFloor` (the refusal text with path, and a path-free copy for the self-upgrade audit). `BUN_FLOOR_EXIT_CODE` (78) marks a `gbrain upgrade` refused for the floor, which the autopilot channel treats as a hold. Tests: `bun-floor.test.ts`, `autopilot-bun-floor.test.ts`, `upgrade-bun-link-arc.serial.test.ts`.
- `src/core/orphan-policy.ts` — the shared orphan-reporting policy (`gbrain orphans`, doctor's orphan ratio, `get_health`): convention constants plus per-brain `orphans.exclude_prefixes` / `orphans.exclude_slugs` overrides. Two renderers over the same constants: `shouldExcludeFromOrphanReporting(slug, overrides, meta)` (TypeScript) and `orphanExclusionSql(alias, overrides)` (a never-NULL SQL predicate; every list value is a bound parameter, only the identifier-checked alias is spliced; regexes are spelled for Postgres's engine: ASCII digits, `.` stops at line terminators). Quarantine stays outside the SQL predicate. Parity pinned on a generated slug/type corpus by `test/orphan-policy-sql-parity.test.ts` and its Postgres arm in `test/e2e/engine-sql-health-parity.test.ts`.
- `src/core/planner-stats.ts` + `src/core/planner-stats-schema.ts` — F4b row-delta ANALYZE for PGLite (no autovacuum). Migration f4_planner_stats (`sqlFor.pglite` only; a Postgres no-op) adds `planner_stats_deltas` / `planner_stats_state` and AFTER STATEMENT transition-table triggers on pages, links, facts, takes, content_chunks and timeline_entries, so modification counts commit or roll back with the data and survive restarts. `maybeRefreshPlannerStats(engine, reason)` ANALYZEs tables whose pending count exceeds max(500, 10% of reltuples) (or that have rows but no pg_stats) and publishes a watermark read before the ANALYZE, so a concurrent modification stays pending; checks are throttled to one per 2 s per engine. Call sites: PGLite planner-sensitive reads (`beforePlannerRead`, deferring a table whose last ANALYZE exceeded `planner.first_read_budget_ms`), import and managed sync every `import.analyze_every_pages`, stale link drains and the `extract --source db` links walk of at least `LINK_DRAIN_ANALYZE_MIN_STALE` (500) pages through `plannerStatsForLinkDrain` (first page, then the same cadence; `link_transitions`/`link_relationships`, no deltas, with `links`), the cycle after extract_facts, the consumer idle tick, and `refreshProjectionStatistics` (full ANALYZE publishes every watermark). `planner.auto_analyze=false` / `GBRAIN_PLANNER_AUTO_ANALYZE=0` disables them all. Doctor `planner_stats_stale` (Postgres: `pg_stat_user_tables` plus absent `missingSearchStatistics`) and repair kind `planner-stats` keep working. `readExpressionIndexesWithoutStats` (both engines; lhf wave 2 item 6) lists expression indexes on the hot tables with entries (index `reltuples > 0`) but no `pg_stats` row under their own name, skipping indexes whose statistics the role cannot see; every `maybeRefreshPlannerStats` reason except `first_read` analyzes their tables (a search never pays the probe), doctor names each with its own reason, and the repair plans their tables (on Postgres the operator-run recovery). Guide: `docs/guides/planner-stats.md`. Tests: `test/planner-stats*.test.ts` (`planner-stats-link-drain.test.ts` EXPLAINs the drain's snapshot read), `test/e2e/planner-stats-postgres.test.ts`.
diff --git a/src/core/persistence/source-lifecycle.ts b/src/core/persistence/source-lifecycle.ts
index a3b295d6af..2d8da653a2 100644
--- a/src/core/persistence/source-lifecycle.ts
+++ b/src/core/persistence/source-lifecycle.ts
@@ -17,7 +17,7 @@ import { containsPath, getWorktreeBinding, humanManifestProgress, type WorktreeB
import { manifestDifferences, storedManifestScope, type WorktreeManifestScope } from './worktree-manifest.ts';
import { hostIdFor } from './identity.ts';
import { advanceTopology, assertTopologyRequestsSettled, inspectTopologyPhysicalRoot, lockTopologyPrincipal, lockTopologyRows, settleTopologyRequests, topologyCanonicalStamp, topologyPrincipal, withTopologyLocks } from './topology-locks.ts';
-import { priorTopologyChange, recordTopologyChange, topologyReceipt } from './topology-receipts.ts';
+import { assertTopologyChangeCapacity, priorTopologyChange, recordTopologyChange, topologyReceipt } from './topology-receipts.ts';
import { isWriteRequestId } from './types.ts';
import { withCoordinatedWrite } from './context.ts';
import { principalAttribution } from './attribution.ts';
@@ -185,10 +185,12 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceL
manifests.set(root.worktree,worktreeManifest(root.worktree,{progress:humanManifestProgress(),scope:storedManifestScope(manifests.get(currentRoot))}));
// #7702/#8905: a remove or purge first deletes the source's pages in bounded batches, each its own transaction behind
// the final transaction's preconditions; any precondition that would refuse or replay stops batching and the final
- // transaction reports it. Committed batches are durable, so a retry with the same request_id resumes.
+ // transaction reports it. Committed batches are durable, so a retry with the same request_id resumes. A batch that
+ // hits the statement bound is retried at half the size, so heavy pages still fit.
let retiredPages=0;
+ try{
if(['remove','purge'].includes(input.operation)&&input.confirmDestructive&&!missingCheckout&&!admission)
- for(let deleted=SOURCE_RETIRE_BATCH_PAGES;deleted===SOURCE_RETIRE_BATCH_PAGES;) deleted=await topologyTransaction(engine,async tx=>{
+ for(let limit=SOURCE_RETIRE_BATCH_PAGES,deleted=limit;deleted===limit;) try{ deleted=await topologyTransaction(engine,async tx=>{
await assertWriterAdminState(tx,input.expectedAdminState);
const sources=await lockTopologyRows(tx,input.sourceId,bindings);
const [source]=await tx.executeRaw<{incarnation:string;archived:boolean;due:boolean}>(
@@ -197,13 +199,18 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceL
||await priorTopologyChange(tx,principal,requestId,intent)
||(await tx.executeRaw('SELECT client_id FROM oauth_clients WHERE source_id=$1 LIMIT 1',[input.sourceId])).length) return 0;
await assertTopologyRequestsSettled(tx,sources,bindings.map(binding=>binding.worktree_id));
+ await lockTopologyPrincipal(tx,principal);
+ await assertTopologyChangeCapacity(tx,{principal,operation:input.operation,sourceId:input.sourceId});
const gone=await withCoordinatedWrite(tx,sources,()=>tx.executeRaw(
- 'DELETE FROM pages WHERE id IN (SELECT id FROM pages WHERE source_id=$1 LIMIT $2) RETURNING id',[input.sourceId,SOURCE_RETIRE_BATCH_PAGES]),
+ 'DELETE FROM pages WHERE id IN (SELECT id FROM pages WHERE source_id=$1 LIMIT $2) RETURNING id',[input.sourceId,limit]),
principalAttribution({kind:'local_cli',id:principal}));
retiredPages+=gone.length;
return gone.length;
- });
- return topologyTransaction(engine,async tx=>{
+ }); }catch(error){
+ if(!(error instanceof OperationError&&error.code==='write_pending'&&limit>1)) throw error;
+ deleted=limit=Math.ceil(limit/2);
+ }
+ return await topologyTransaction(engine,async tx=>{
await assertWriterAdminState(tx,input.expectedAdminState);
await tx.executeRaw("SELECT set_config('synchronous_commit','on',true)");
const sources=await lockTopologyRows(tx,input.sourceId,bindings);
@@ -297,7 +304,7 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceL
}else{
const refs=await tx.executeRaw<{client_id:string}>('SELECT client_id FROM oauth_clients WHERE source_id=$1 LIMIT 1',[input.sourceId]);
if(refs.length) throw opError('source_referenced','OAuth clients still reference this source. Revoke and remove those registrations first.',
- `Nothing was removed. OAuth client ${refs[0].client_id} (and possibly others) is bound to '${input.sourceId}'; list them with the command in fix and ask the user before revoking any, since revoking disconnects that client.`,
+ `${retiredPages?`Only the ${retiredPages} page(s) this run already deleted were removed.`:'Nothing was removed.'} OAuth client ${refs[0].client_id} (and possibly others) is bound to '${input.sourceId}'; list them with the command in fix and ask the user before revoking any, since revoking disconnects that client.`,
{fix:readFix('Lists OAuth clients with the source each is bound to.',{argv:['gbrain','auth','clients','--json']})});
const [impact]=await tx.executeRaw<{count:string}>('SELECT count(*)::text AS count FROM pages WHERE source_id=$1',[input.sourceId]);
pagesDeleted+=Number(impact.count);
@@ -323,6 +330,11 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceL
const row=await recordTopologyChange(tx,{principal,requestId,intent,operation:input.operation,sourceId:input.sourceId,incarnation:source?.incarnation??String(result.source_incarnation),worktrees},result);
return topologyReceipt(row);
});
+ }catch(error){
+ if(retiredPages&&error instanceof OperationError&&!error.suggestion?.startsWith('Only the'))
+ error.suggestion=`This run had already deleted ${retiredPages} page(s) of source '${input.sourceId}' in committed batches before it stopped; the source and its remaining pages are still registered. ${error.suggestion??''}`.trim();
+ throw error;
+ }
},root?.worktree,5000,{relocation:input.operation==='rebind'});
}
diff --git a/src/core/persistence/topology-receipts.ts b/src/core/persistence/topology-receipts.ts
index d4bbee1e68..30f2ce415c 100644
--- a/src/core/persistence/topology-receipts.ts
+++ b/src/core/persistence/topology-receipts.ts
@@ -35,19 +35,11 @@ export async function recordTopologyChange(tx:BrainEngine, input:{principal:stri
for(const counter of counters){
const brain=counter.key==='brain';
if(!counter.key.startsWith('worktree:')){
+ await assertQuota(tx,counter,limits,terminalBytes,input);
if(recovery && (Number(counter.outstanding_count)+1>(brain?limits.brainOutstanding:limits.principalOutstanding)
||Number(counter.intent_bytes)+intentBytes>(brain?limits.brainIntentBytes:limits.principalIntentBytes)))
throw opError('queue_capacity','Pending lifecycle work exceeds available brain/principal request or intent capacity.',
`Too much lifecycle work is still pending, so the ${input.operation} of source ${input.sourceId} was not accepted. Let the pending work settle, then submit it again; raising ${journalLimitKey(brain?'brainOutstanding':'principalOutstanding')} is the user's call.`, { fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', input.sourceId, '--json'], consent: [], actor: 'agent', why: 'Shows the pending lifecycle and recovery work that holds this capacity.', requires_exclusive: false } });
- if(Number(counter.lifetime_ids)+1>(brain?limits.brainLifetimeIds:limits.principalLifetimeIds)
- ||Number(counter.terminal_bytes)+terminalBytes>(brain?limits.brainTerminalBytes:limits.principalTerminalBytes))
- {
- const ids=Number(counter.lifetime_ids)+1>(brain?limits.brainLifetimeIds:limits.principalLifetimeIds);
- const setting=`${brain?'brain':'principal'}${ids?'LifetimeIds':'TerminalBytes'}` as const;
- const used=Number(ids?counter.lifetime_ids:counter.terminal_bytes);
- throw quotaExhausted(`${brain?'brain':'principal'} ${ids?'permanent request IDs':'reserved receipt bytes'} (lifecycle ${input.operation} of source ${input.sourceId})`,
- journalLimitKey(setting),await oneYearCapacity(tx,counter.key,ids?'LifetimeIds':'TerminalBytes',used,limits[setting]),used,limits[setting],ids?'lifetime_ids':'terminal_bytes');
- }
}
if(recoveryBytes && counter.key!==key && Number(counter.recovery_bytes)+recoveryBytes>(brain?limits.brainRecoveryBytes:limits.worktreeRecoveryBytes))
throw opError('queue_capacity','The lifecycle recovery exceeds available brain/worktree capacity.',
@@ -65,6 +57,23 @@ export async function recordTopologyChange(tx:BrainEngine, input:{principal:stri
}
return row;
}
+type JournalLimits=Awaited<ReturnType<typeof readJournalLimits>>;
+type Counter=Awaited<ReturnType<typeof lockCounters>>[number];
+async function assertQuota(tx:BrainEngine,counter:Counter,limits:JournalLimits,terminalBytes:number,input:{operation:string;sourceId:string}):Promise<void>{
+ const brain=counter.key==='brain';
+ const ids=Number(counter.lifetime_ids)+1>(brain?limits.brainLifetimeIds:limits.principalLifetimeIds);
+ if(!ids&&Number(counter.terminal_bytes)+terminalBytes<=(brain?limits.brainTerminalBytes:limits.principalTerminalBytes)) return;
+ const setting=`${brain?'brain':'principal'}${ids?'LifetimeIds':'TerminalBytes'}` as const;
+ const used=Number(ids?counter.lifetime_ids:counter.terminal_bytes);
+ throw quotaExhausted(`${brain?'brain':'principal'} ${ids?'permanent request IDs':'reserved receipt bytes'} (lifecycle ${input.operation} of source ${input.sourceId})`,
+ journalLimitKey(setting),await oneYearCapacity(tx,counter.key,ids?'LifetimeIds':'TerminalBytes',used,limits[setting]),used,limits[setting],ids?'lifetime_ids':'terminal_bytes');
+}
+/** #7702/#8905: the lifetime-ID and receipt-byte quotas recordTopologyChange enforces, at the largest receipt it can record, so a retire refuses before it deletes anything. */
+export async function assertTopologyChangeCapacity(tx:BrainEngine,input:{principal:string;operation:string;sourceId:string}):Promise<void>{
+ const limits=await readJournalLimits(tx);
+ for(const counter of await lockCounters(tx,['brain',principalKey({kind:'local_cli',id:input.principal})]))
+ await assertQuota(tx,counter,limits,receiptReservation({}),input);
+}
/** Caller retains owner/source guards and completes final state in this same transaction. */
export async function releaseTopologyReservation(tx:BrainEngine,row:TopologyChange):Promise<boolean>{
// Revocation or removal prevents publication, never cleanup of its prior
diff --git a/test/source-retire-batched.test.ts b/test/source-retire-batched.test.ts
index 78e9f50251..de9fa6daae 100644
--- a/test/source-retire-batched.test.ts
+++ b/test/source-retire-batched.test.ts
@@ -15,6 +15,7 @@ import { registerLocalWriter } from '../src/core/persistence/identity.ts';
import { claimWorktree } from '../src/core/persistence/ownership.ts';
import { runManagedSourceLifecycle, SOURCE_RETIRE_BATCH_PAGES } from '../src/core/persistence/source-lifecycle.ts';
import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+import { journalLimitKey } from '../src/core/persistence/limits.ts';
let engine: PGLiteEngine;
@@ -106,3 +107,35 @@ test('an interrupted batch keeps the committed batches and the same request_id r
for (const sql of ['DROP TRIGGER IF EXISTS retire_fail ON pages', 'DROP FUNCTION IF EXISTS retire_fail_fn()', 'DROP TABLE IF EXISTS retire_fail']) await engine.executeRaw(sql);
}
}), 60_000);
+
+test('a remove refuses before deleting when the receipt quota is spent, and a CLI writer revoked mid-run stops it at the next batch', () => fixture(async () => {
+ const [brain] = await engine.executeRaw<{ n: string }>("SELECT lifetime_ids::text AS n FROM persistence_counters WHERE key='brain'");
+ await engine.executeRaw('INSERT INTO config(key,value) VALUES($1,$2) ON CONFLICT(key) DO UPDATE SET value=EXCLUDED.value', [journalLimitKey('brainLifetimeIds'), brain?.n ?? '0']);
+ await expect(runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true })).rejects.toMatchObject({ code: 'write_quota_exhausted' });
+ expect(await pages(source)).toBe(PAGES);
+ await engine.executeRaw('DELETE FROM config WHERE key=$1', [journalLimitKey('brainLifetimeIds')]);
+ const revoke = "CREATE FUNCTION retire_revoke_fn() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN UPDATE persistence_local_writers SET revoked_at=now() WHERE lane='cli'; RETURN NULL; END $$";
+ await engine.executeRaw(revoke);
+ await engine.executeRaw('CREATE TRIGGER retire_revoke AFTER DELETE ON pages FOR EACH STATEMENT EXECUTE FUNCTION retire_revoke_fn()');
+ try {
+ await expect(runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true }))
+ .rejects.toMatchObject({ code: 'permission_denied', suggestion: expect.stringContaining(`already deleted ${SOURCE_RETIRE_BATCH_PAGES} page(s)`) });
+ expect(await pages(source)).toBe(PAGES - SOURCE_RETIRE_BATCH_PAGES);
+ expect(await registered()).toBe(1);
+ } finally {
+ for (const sql of ['DROP TRIGGER IF EXISTS retire_revoke ON pages', 'DROP FUNCTION IF EXISTS retire_revoke_fn()']) await engine.executeRaw(sql);
+ }
+}), 60_000);
+
+test('a batch that hits the statement bound is retried at half the size until it fits', () => fixture(async () => {
+ await engine.executeRaw(`CREATE FUNCTION retire_bound_fn() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN
+ IF (SELECT count(*) FROM gone) > 100 THEN RAISE EXCEPTION 'canceling statement due to statement timeout' USING ERRCODE = 'query_canceled'; END IF; RETURN NULL; END $$`);
+ await engine.executeRaw('CREATE TRIGGER retire_bound AFTER DELETE ON pages REFERENCING OLD TABLE AS gone FOR EACH STATEMENT EXECUTE FUNCTION retire_bound_fn()');
+ try {
+ expect(await runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: source, confirmDestructive: true }))
+ .toMatchObject({ state: 'committed', operation: 'remove', pages_deleted: PAGES });
+ expect(await pages(source)).toBe(0);
+ } finally {
+ for (const sql of ['DROP TRIGGER IF EXISTS retire_bound ON pages', 'DROP FUNCTION IF EXISTS retire_bound_fn()']) await engine.executeRaw(sql);
+ }
+}), 60_000);