Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Removing a source, a connector delete and a late eval capture no longer leave a deleted page's slug behind

garrytan99dcd70aea04
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4be6f55b73..f9fef5222c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,6 +25,23 @@ identifiers and attribution are available in the pre-removal Git revision
- `src/core/deletion-inventory.ts`: both tables are `swept` (`page_names`); the fact receipt still probes them, since a fact purge sweeps neither.
- Tests: `test/purge-name-stores-9068.test.ts` (PGLite, and Postgres through `test/postgres-unit-arms.txt`).
+### Removing a source, a connector delete and a late eval capture no longer leave a deleted page's slug behind
+
+**Three more ways a deleted page's slug outlived the page are closed, and the federation warning stops counting private pages.** Before, only a page purge scrubbed the stores that name a page without a foreign key to it.
+
+| What you did | Before | Now |
+|---|---|---|
+| A query or search with eval capture on, whose capture runs after the response, racing a page purge | the capture could write the purged slug back into `eval_candidates.retrieved_slugs` | the insert keeps only slugs whose page still exists, checked under the same row lock the feedback writers take |
+| A connector, GitHub or Google sync deletes a page (`deletePages`) | retrieval logs, ratings, weights, the volunteer log, the gazetteer and captured eval slugs kept naming it | scrubbed in the delete's transaction |
+| `gbrain sources remove`, `sources purge`, the expired-archive sweep, or a managed `remove`/`purge` | the source's retrieval logs, ratings, volunteer rows and captured eval slugs stayed | deleted in the statement that deletes the source, and swept again right after for a write that raced it |
+| `gbrain sources federate` or `sources restore` on a source with a remote-written `visibility: private` page and no readers | the warning named that page, though no-grant connections still cannot read it | the warning applies the read paths' privacy rules, unless `search.remote_private_pages` or `GBRAIN_REMOTE_PRIVATE_PAGES` relaxes them |
+
+- `src/core/feedback/store.ts`: `insertEvalCandidate` is the one eval capture insert both engines run.
+- `src/core/engine-sql/pages.ts`: `deletePages` runs `scrubPurgedSlugs` and `scrubPurgedNameStores` after the delete, in one transaction. `deletePage` is unchanged: the page purge, its main caller, scrubs and records the stores in its receipt.
+- `src/core/source-delete.ts`: `removedSourceScrubCtes` runs inside `deleteSourceRow` and the managed lifecycle's source delete, then once more ungated. `retrieval_weights` and `mention_gazetteer_entries` already cascaded from `sources`.
+- `src/core/access/federation-exposure.ts`: the exposure read adds `privacyPartSql` unless `remotePrivacyRelaxed`.
+- Tests: `test/purge-scrub-followups-9068.test.ts` and `test/federation-exposure-private-9068.test.ts` (PGLite, and Postgres through `test/postgres-unit-arms.txt`). `test/eval-candidates.test.ts` now seeds the pages its round-trip captures, since a capture keeps only live slugs.
+
## [0.60.650.0] - 2026-11-10
**Private-labels windows publish together and finish past declined pages.** `gbrain repair private-labels --apply --expect <hash>` on a brain with managed persistence now admits a window's pages together and publishes them in groups, instead of one page and one wait at a time. On a 1,000-page window with Git commits and a second queue consumer (`scripts/bench/private-labels-apply.ts`), it went from 181.4 to 1365.3 pages a minute with no added database latency, and from 19.9 to 37.1 pages a minute with 20 ms added to each database round trip. Git commits for the window dropped from 990 to 22. The preview no longer claims a re-embed for every page: only pages stored under per-chunk synopsis, or with a chunk that has no vector yet, count.
diff --git a/docs/architecture/key-files/access.md b/docs/architecture/key-files/access.md
index 063a4f8909..7feda72682 100644
--- a/docs/architecture/key-files/access.md
+++ b/docs/architecture/key-files/access.md
@@ -4,5 +4,5 @@
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
- `src/core/access/` (GBRA-218) — per-member read access inside one source. `principals.ts` normalizes frontmatter `readable_by` (bare id = `member:<id>`, `group:<g>`, ASCII case fold, space trim; anything malformed is `['!nobody']`); it is the row twin of the SQL `gbrain_access_readers()` (`test/access-principals.test.ts` holds them equal). A BEFORE trigger keeps `pages.page_readers` equal to that function (direct writes to the column are recomputed), and predicates test the array, never the function, which costs ~0.2 ms per row. A page that names readers with no derived value yet (written before the migration) reads as nobody until `gbrain repair access` (`repair/access.ts`). `members.ts` (members, groups, tombstoned ids never reused, bindings), `token-binding.ts` (the verifiers' join; a credential bound to a disabled, tombstoned or missing member is refused with `access_member_disabled`) and `caller.ts` (`remoteCallerAccess`: the identity dispatch puts on the caller-access record; stdio binds through `mcp.stdio_member`; publication re-resolves from the stored principal). In `search/private-visibility.ts` every page fragment is `accessAllowedSql` AND the privacy part: plain fragments use the empty identity (over-hide, never leak), `*ForCaller` fragments the caller's, and `GBRAIN_REMOTE_PRIVATE_PAGES`, `search.remote_private_pages` and `granted_sources` relax only the privacy part (`privacyForCaller`). A derivative of a page that names readers is hidden from every remote caller (`originAllowedSql`, which calls the SQL function `gbrain_access_origin_restricted()` only for atom, question, concept, event or `derived_from` rows, so each read plans one call instead of four subplans). Both parts lead with column tests (`page_readers IS NULL`, `type`) so row estimates follow column statistics; an expression test such as `->'readable_by' IS NULL` estimates almost no rows and sent restricted vector searches into nested-loop sequential scans. The index walk's page probe (`INDEX_WALK_PAGE_COLUMNS`) projects `page_readers`. Config `access.restrictions_present` (set by a trigger, sticky) keeps every pre-A1 path on a brain with no restriction. `check:remote-privacy-reads` keeps the two relaxations read only in `private-visibility.ts`. Tests: `test/member-read-access.test.ts` (remote matrix, both engines), `test/access-*.test.ts`. Folder rules (A2) keep whole folders of a source to named members. `folder-rules.ts` is the grammar (`.gbrain/access.yml`: `rules:` of `id`, `prefix`, `readable_by`), `classifyRulebook` (each change is narrowing or widening; the target installs only the narrowing part) and `matchingRuleIds`. `policy.ts` owns the one applied rulebook for both stores: `access_rules` (source-qualified ids), the append-only `access_policy_revisions` log, `access_pending_coverage` (prefixes that read as nobody while a narrowing restamps, or while a committed file is invalid) and `sources.config.access_store`, persisted at the first rule and changed only by `policy-migrate.ts` (journaled in `access_policy_migrations`). `proposeRevision` installs narrowing at once under a per-source row lock and holds widening for `applyApproved` (`plan_hash`, `preview_changed`); `restampSource` rewrites `pages.folder_rule_ids` in id windows, then clears pending coverage under the lock; re-proposing the same rulebook finishes an interrupted restamp. A trigger stamps `folder_rule_ids` from the SQL function `gbrain_access_folder_rules()` on insert and on slug or source change, never from git on the serve path. `policy-git.ts` is the git store (files committed through the commit door, applied by `syncAccessPolicyAt` from sync preflight and before and after every managed pass (`performManagedSync`: sync, drain, `sources refresh`, git conversion), read with `git show` at the synced commit; a database with no revision yet replays each committed `access.yml` since the `access.approved` baseline, so a rebuild holds a removal a plain commit made); `policy-transfer.ts` carries the applied rulebook through `gbrain export` and installs it before any page in `gbrain import`; `policy-notice.ts` is the `access_policy_pending` degraded notice. In `private-visibility.ts`, `folderRulesAllowedSql` sits inside `accessAllowedSql` (`folder_rule_ids IS NULL OR folder_rule_ids <@ <satisfied ids>` plus coverage conjuncts), so no hatch, grant or `--as` relaxes it, and the unrestricted short form tests `folder_rule_ids IS NULL` next to `page_readers IS NULL` (v283 triggers set `access.restrictions_present` on the first rule or pending prefix); `caller.ts`'s `folderAccessFor` fails closed to whole-source coverage. CLI: `src/commands/access-rules.ts`; doctor: `src/commands/doctor/checks/folder-access.ts`. Tests: `test/access-folder-rules.test.ts` (database store, both engines), `test/access-folder-rules-git.test.ts`, `test/access-folder-transfer.test.ts`, `test/access-folder-doctor.test.ts`. Shared agents (B, GBRA-231): `audience.ts` parses the audience a harness asserts (`X-Gbrain-On-Behalf-Of` or request-level `_meta.on_behalf_of`, 1-50 principals, `guest` = the empty identity; the same key inside tool arguments is refused) and appends each assertion to `access-audience-YYYY-Www.jsonl`. Dispatch resolves it on every remote call through `caller.ts` (`resolveAudienceIdentities`: every entry must be in the credential's live `may_act_for`, then name an enabled member or existing group); the caller-access record then carries `audienceIdentities`, and `accessAllowedForAllSql` needs one readers conjunct per identity, folder rules the intersection of each entry's satisfied set (`folderAccessForAll`). Privacy relaxes only for a one-member audience. Writes carry the audience as `WriteAuthority.actingFor` (re-resolved at publication) and agent jobs as `grant.actingFor`. Every remote response carries `_meta.access` (`appliedAccessSummary`), which `whoami` returns as `access.applied`; doctor `access_may_act_for_wildcard` (`checks/shared-agent-access.ts`) warns on `*`. Tests: `test/access-on-behalf-of.test.ts` (both engines), `test/e2e/serve-http-on-behalf-of.test.ts`. Attested actor (C1, GST-359): `actor-model.ts` (dependency-free: `x-qm-actor`, `acting_user`, `EXTERNAL_ID_RE` = 1-256 visible ASCII, `attestRightFrom`: the right counts only for a member-less credential whose token is legacy or `client_credentials`, read from `oauth_tokens.grant_type`) and `actor.ts`. `resolveAttestedActor` runs in dispatch before any work: an invalid id refuses `actor_not_attested` (`invalid_actor_id`) before any lookup, then the right, the op list (`attest_ops`, default `get_write_request,memory_recall,memory_write`), `acting_user` = header (stripped), and `actorCoverage` (the one coverage test: live right plus a live link in `access_member_links` to an enabled member, compared exactly) or `actor_unlinked`. `ACTOR_DEFERRED_OPS` (`capture`, `extract_facts`, take writes, `list_write_conflicts`) are never attestable until each is keyed to the member: refused by `parseAttestOps`, `bindCredential`, every call and the stored re-check, which also re-checks `attest_ops` for a queued write's operation and each job tool call. Under the header serve-http lists only attested tools and refuses `resources/read`. `attestedCallerAccess` is a one-member audience with privacy never relaxed (no granted sources, no hatch), never B's `may_act_for` path. Writes store `WriteAuthority.actingFor = {system, external_id, member}` and jobs `grant.actor`; `storedActorIdentity` re-resolves both through `actorCoverage` and refuses when the member changed. `linkMember` / `unlinkMember` / `listMemberLinks` back `gbrain access member link|unlink|links`; `--attest-actor` / `--attest-ops` on `auth rescope` (`members.ts`) only for a member-less legacy token or confidential client-credentials client. Migration v286. Tests: `test/access-qm-actor.test.ts` (both engines), `test/e2e/serve-http-qm-actor.test.ts`. Member facts (C2, GST-359): `member-facts.ts`. A remote `remember` from an attested actor or a lone-member credential (`callerMember`, never an audience of several) writes to `memory/members/<member>` (`memberFactPage`; `persistence/memory-prepare.ts` creates the page with `readable_by: [member:<id>]`), `visibility: private`, `infer_entity` off, the entity kept as `entity_slug`, unless the call passes `share: true` (GST-2243), which takes the ordinary world route to the subject's page; a member's explicit `visibility: "world"` is refused in `remember`'s validation (`member_share_required`), so a batch with one such item saves nothing. A member id whose page slug `slugifyPath` would change (`a--b`, `zed-`) is refused (`invalid_params`). The page must name that member alone (`memberPageIsOwn`, checked at submission and again in preparation, `memberPageNotOwned`), and remote writes under `memory/members/` go only to the caller's own page (`assertMemberPageWriter`, called from `ops/context.ts` `enforceClientSlugFence`, so every remote page write passes it; trusted local callers are unaffected). Reads widen a world-only fact list only by `memberFactAdmitSql` (a private fact on the caller's own live member page whose readers and folder rules admit the caller; `factVisibilitySql` in `engine-sql/facts.ts` and `search/fact-relevance.ts`), and `privateProvenanceFilterUnderPolicy` judges a member-page fact's source page as the caller (every other fact keeps the empty identity, so a fact from a page that names readers stays hidden from every remote caller). Writers never cross pages: `facts/single-prepare.ts` (`SingleFactIntent.page_slug`) and `trust/supersede-handlers.ts` dedup, supersede and contest only within the member page, and no other write matches a fact there (`onMemberPageSql`); consolidation skips facts whose page names readers (`readersSourcedFactSql`). Per-person keys: `callerIdentityKey` joins the hot-memory cache key (`facts/meta-hook.ts`, `::xid=`), `callerSessionKey` the `delta` cursor (`ops/facts.ts`), `sameAttestedActor` gates receipts (`persistence/authority.ts` `ownRequestAccessible`) and replays (`persistence/journal.ts`) on the stored actor, else the stored audience, and `get_agent_job` and the owner-fenced `get_job`, `wait_job`, `get_job_progress`, `list_jobs` and `cancel_job` match the job's `grant.actor` (system, external id, member) and `grant.actingFor` (the same distinct audience entries; `jobActorSql` in `minions/types.ts`), so a job or receipt with an actor or audience reads as missing to the bare client and to other actors and audiences, and one with neither to every actor and audience. Tests: `test/access-member-facts.test.ts` (both engines), `test/access-member-sweep.test.ts` (every attestable op as Bob, the bare client and an unbound credential; `test/helpers/member-sweep.ts`), `test/e2e/serve-http-member-sweep.test.ts` (two members over a real `serve --http` on a keyless repo-less brain).
-- `src/core/access/federation-exposure.ts` (#8982) — `remotePagesFederationExposes(engine, source)`: the live pages of a source that name no readers and no folder rule and whose live revision (`pages.revision_principal_*`) or any version (`page_versions.write_principal_*`) a token or OAuth client wrote (revoked or deleted included; no record keeps a credential's member at write time, so an unbound, rebound or deleted one counts and each example names its member now; a tag change moves the revision attribution without a version, hiding the earlier author): count, the first five slugs, and shell-quoted `who-can-read` / `access rule add` commands. `remotePagesFederationExposesOrWarn` runs it, after the change commits, only for a change that added the source to the no-grant floor (federate reads the prior state under the source row lock), and turns a failed read into a stderr warning, so the caller's federation never changes. Read by `sources federate`, the unmanaged `sources restore` (stderr; `--json` as `pages_exposed`) and the managed `source_lifecycle` restore result. Pinned by `test/sources-federate-member-pages.test.ts`.
+- `src/core/access/federation-exposure.ts` (#8982) — `remotePagesFederationExposes(engine, source)`: the live pages of a source that name no readers and no folder rule, that the read paths' privacy rules (`privacyPartSql`) do not hide from a no-grant connection (#9068; skipped when `remotePrivacyRelaxed` says an operator opted out), and whose live revision (`pages.revision_principal_*`) or any version (`page_versions.write_principal_*`) a token or OAuth client wrote (revoked or deleted included; no record keeps a credential's member at write time, so an unbound, rebound or deleted one counts and each example names its member now; a tag change moves the revision attribution without a version, hiding the earlier author): count, the first five slugs, and shell-quoted `who-can-read` / `access rule add` commands. `remotePagesFederationExposesOrWarn` runs it, after the change commits, only for a change that added the source to the no-grant floor (federate reads the prior state under the source row lock), and turns a failed read into a stderr warning, so the caller's federation never changes. Read by `sources federate`, the unmanaged `sources restore` (stderr; `--json` as `pages_exposed`) and the managed `source_lifecycle` restore result. Pinned by `test/sources-federate-member-pages.test.ts` and `test/federation-exposure-private-9068.test.ts`.
- `src/core/ops/qm-memory.ts` (GST-359 D) — QM's memory provider pair. `memory_recall` reads the attested person's own last-24-hour facts on their member page (`listFactsSince` with `sourceMarkdownSlug`), then `rankFactsByQuestion` over the facts they may read, renders one plain line per fact (`plainFactText` strips fence and heading markers) and cuts at a line within `max_chars` (`boundFactLines`). A call that names `source_id` or `source_ids` (full surface only) reads as that scope instead: `recallSources` resolves them through `requestSourceScope` (a source outside the credential's grant is `permission_denied`), `SCOPE_READ_ACCESS` empties identity, actor and audience, so no own facts and no member-page fact are read, restricted pages stay hidden and privacy is never relaxed; the query is ranked on its first 2,000 characters (`MEMORY_RECALL_QUERY_CHARS`). `memory_write` validates 1-20 facts, derives each request id from uuidv5(member, idempotency_key) (`memoryWriteRequestId`) and saves through `remember`'s handler with `ctx.attestedOp = 'memory_write'`, which the stored write authority keeps (`WriteAuthority.attestedOp`) so publication and receipt reads re-check that operation's attest right. `qmMemoryToolResult` (dispatch) moves the result into `structuredContent` and leaves the model text in `content`; `unlinkedRecallResult` is the empty success dispatch's `actorRefusal` returns for an unlinked person. Contract fixture: `scripts/build-qm-memory-contract.ts` writes `docs/contracts/qm-memory-provider/v1.json` (frozen, byte-identical) and `v2.json` (the scope read and write; `regen:all` artifacts). Scope writes (GST-1113): `memory_write {source_id}` saves on `memory/scope` in a scope source listed in the credential's `attest_sources` (migration v293, on `access_tokens` and `oauth_clients`); `access/attest-sources.ts` holds `perCallWriteSource` (the one exemption from the one-write-source checks in `page-mutations.ts`, `memory-mutations.ts` and `persistence/authority.ts`, keyed to that slug and re-checked at publication through `actorCoverage(..., source)`), the sticky mark `sources.config.qm_scope` (set by `applyAttestSources` under row locks; `isSourceFederated` ignores federation on it), `assertScopeSourceGrant` (in `insertClientGrant`, `persistGrant`, `writeTokenGrant` and `insertUnifiedToken`) and `withoutScopeSources` (a person recall never reads a scope source). `member-facts.ts` `assertScopePageWriter` keeps other remote writers off `memory/scope`. Tests: `test/qm-scope-writes.test.ts` (both engines), `test/qm-memory-tools.test.ts` (both engines), `test/qm-memory-contract.test.ts` (drift), `test/e2e/serve-http-qm-memory.test.ts` (keyless `serve --http`).
diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.md
index c6c36ba2d6..ffa1a2b09f 100644
--- a/docs/architecture/key-files/core-persistence-continued.md
+++ b/docs/architecture/key-files/core-persistence-continued.md
@@ -41,4 +41,4 @@ The #6188 fence check and fence repair live in [fence repair](core-fence-repair.
- `src/core/postgres-engine/reserved-transactions.ts` with `isPersistenceDeclaration` (`src/core/persistence/protocol.ts`) — BEGIN pipelining. `isPersistenceDeclaration` (GBRA-156) accepts exactly the `protocol.ts` declarations (transaction-local `set_config` calls and the brain row FOR SHARE). Admission, the FIFO claim, the effect claims and source guards, the receipt true-up, the single publication (`coordinator.ts`) and the consumer's lent lane transactions (`consumer.ts` `onLane`, through `reservedTransactions`) pass it as `TransactionOptions.pipelineFirst`, so behind a transaction-mode pooler the declaration goes out with BEGIN (one round trip less per transaction) while every later statement still waits for BEGIN's answer; if BEGIN fails, only the declaration has run, in autocommit, where it changes nothing. A reserved transaction gates every statement its body can reach, savepoint and nested-transaction bodies and `file()` included. A sync lane group's publication passes none: its begun mark must follow BEGIN's answer (`sync-lanes.ts` `awaitLaneBegin`). Tests: `test/persistence-pipeline-first.test.ts` (the predicate and the reviewed call sites), `test/e2e/pgbouncer-transaction-pipeline-6930.test.ts`.
- `src/core/links-fence.ts` + `src/core/links-fence-projection.ts` — the `## Links` fence (G5): the markdown form of authored (manual) edges, `<!--- gbrain:links:begin -->` / `end` around `| to | type | source | valid_from | valid_until | context |`, one row per (to, type, source), source defaulting to `manual`; a reconciliation-managed provenance (`MANAGED_LINK_SOURCES`, exported here and re-exported by `ops/links.ts`), a bad date or a duplicate key is a skipped row with a warning. `parseLinksFence` (state `absent` / `parsed` / `malformed`), the deterministic `renderLinksFence` (rows sorted by key), `upsertLinksFenceRow` (omitted context and dates keep the stored ones; identical row = unchanged body), `removeLinksFenceRows` (leaves an empty fence), `replaceOrInsertLinksFence` (a `## Links` section at the end of compiled_truth, above an unclosed code block) and `maskLinksFence`, which `extractBodyLinks` applies so the fence cells are never also extracted as markdown, wikilink or bare-slug edges. A rewrite of a fence that parses with warnings throws `LinksFenceRewriteRefusal`. Cells use `escapeFenceCell` plus the same one-level marker encoding for `gbrain:links:*`. `projectLinksFence(tx, page, rows)` reconciles only what the page's fence owns: edges with origin_page_id = the page and origin_field `links_fence` (deleted, context-updated, inserted by key) and `manual` transitions with origin_page_id = the page, then refreshes the touched relationships; a row whose target is not a live page in the source yields no edge. `linksFenceProjectionRows` returns null for an absent or malformed fence, so such a body changes no edge (an absent fence retires nothing, as for facts; unlike facts no held generation is recorded, so after a hand deletion of the whole section the database keeps edges git no longer has until `remove_link` or the G3 indexer), and drops rows typed with a link type the active pack does not declare. Callers: `prepareCanonicalProjections` (every projected page publication, quarantined and segmented-unread bodies excepted), `prepareAutomaticLinks` (put_page auto-links and the managed links maintenance, so `extract --stale` restores a lost fence edge) and `persistence/manual-links.ts`. Database-only manual edges (origin_field NULL) are never touched by any of them. Pinned by `test/links-fence.test.ts` and `test/managed-links-fence.test.ts`.
-- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt other than this request's own admission, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`, `lockTopologyPrincipal`, and on the first batch `admitTopologyRetire` from `topology-receipts.ts`, which checks the receipt quotas at the largest receipt and records the admission) and, when one would refuse or replay, stops without deleting so the final transaction reports it, or throws that refusal before deleting; a batch that fails `write_pending` is retried at half the size (floor 1), so heavy pages still fit the bound; a refusal after some batches committed says how many pages that run already deleted; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. #8984: the first batch's transaction records the admission, a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation it deletes from (`isRetireAdmission`); `recordTopologyChange` settles it `committed`, and a remove or purge that deletes that incarnation under another request, or any other lifecycle operation that commits a change to it (archive, restore, rebind, claim), settles it `failed` (`supersedeTopologyRetires`, which skips only the caller's own principal and request_id pair). A retry of the request_id against another incarnation (the source was removed and re-added) refuses `source_retire_superseded`, settles the admission `failed` and deletes nothing, also when a since-replaced CLI registration (another principal) admitted that request_id for a remove or purge, since idempotency is per principal; an admission bound to the current incarnation still resumes under the new registration; deactivation and graduation list an unsettled admission as a blocker that names its request_id and asks the user whether to finish it or keep what is left; an automatic expired-archive purge's admission (`expired_only` in its outcome) is left to the next automatic run with `needsUser: false`, and `sources writer status --json` lists the stopped and superseded requests (`topology_requests`). Tests: `test/source-retire-batched.test.ts` (PGLite), `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages), `test/source-retire-incarnation.test.ts` and `test/e2e/source-retire-incarnation-postgres.test.ts`.
+- `src/core/persistence/source-lifecycle.ts` retire batches with `topology-transaction.ts` (#7702/#8905) — every `topologyTransaction` runs under `statement_timeout=5s` and `lock_timeout=1s` and maps SQLSTATE 57014/55P03 to `write_pending`, so a managed `remove` or `purge` first deletes the source's pages in batches of `SOURCE_RETIRE_BATCH_PAGES` (500; about 110 ms per batch with chunks and links on Postgres) under the native locks `withTopologyLocks` holds, each batch its own `topologyTransaction`. A batch re-checks the final transaction's preconditions (admin state, `lockTopologyRows`, the incarnation read before the locks, archived or `EXPIRED_ARCHIVE_SQL` due, no prior receipt other than this request's own admission, no `oauth_clients` reference, `assertTopologyRequestsSettled` from `topology-locks.ts`, `lockTopologyPrincipal`, and on the first batch `admitTopologyRetire` from `topology-receipts.ts`, which checks the receipt quotas at the largest receipt and records the admission) and, when one would refuse or replay, stops without deleting so the final transaction reports it, or throws that refusal before deleting; a batch that fails `write_pending` is retried at half the size (floor 1), so heavy pages still fit the bound; a refusal after some batches committed says how many pages that run already deleted; a dry run, a missing checkout, an unconfirmed operation and an admission-hooked call never batch. The final transaction's `sources` delete runs `removedSourceScrubCtes` (#9068: the source's retrieval and volunteer rows, and its slugs in eval candidates), so a run that stops after some batches leaves those rows until a retry commits. Each batch deletes inside `withCoordinatedWrite` with local-CLI attribution (the `pages` guard needs the source in `gbrain.write_sources`; only the final `sources` delete sets `gbrain.topology_change`). Committed batches are durable: a retry (same or new request_id) resumes, and the receipt's `pages_deleted` counts the pages deleted by the run that commits it. #8984: the first batch's transaction records the admission, a `persistence_topology_changes` row in state `recovering` with no recovery record, bound to the source incarnation it deletes from (`isRetireAdmission`); `recordTopologyChange` settles it `committed`, and a remove or purge that deletes that incarnation under another request, or any other lifecycle operation that commits a change to it (archive, restore, rebind, claim), settles it `failed` (`supersedeTopologyRetires`, which skips only the caller's own principal and request_id pair). A retry of the request_id against another incarnation (the source was removed and re-added) refuses `source_retire_superseded`, settles the admission `failed` and deletes nothing, also when a since-replaced CLI registration (another principal) admitted that request_id for a remove or purge, since idempotency is per principal; an admission bound to the current incarnation still resumes under the new registration; deactivation and graduation list an unsettled admission as a blocker that names its request_id and asks the user whether to finish it or keep what is left; an automatic expired-archive purge's admission (`expired_only` in its outcome) is left to the next automatic run with `needsUser: false`, and `sources writer status --json` lists the stopped and superseded requests (`topology_requests`). Tests: `test/source-retire-batched.test.ts` (PGLite), `test/e2e/source-retire-batched-postgres.test.ts` (60,000 pages), `test/source-retire-incarnation.test.ts` and `test/e2e/source-retire-incarnation-postgres.test.ts`.
diff --git a/docs/architecture/key-files/core-services-3.md b/docs/architecture/key-files/core-services-3.md
index cb0d4b61c2..b314a056a4 100644
--- a/docs/architecture/key-files/core-services-3.md
+++ b/docs/architecture/key-files/core-services-3.md
@@ -27,7 +27,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/verbs/usage-log.ts` — observability sidecar: one JSONL line per verb call at `~/.gbrain/integrations/memory-verbs/usage.jsonl` (gbrainPath — GBRAIN_HOME honored; `brainId()` = the resolved gbrain home). LOCAL ONLY, never uploaded, stats-only (lock-free 10MB rotation may drop lines; O_APPEND line-atomic, best-effort on Windows). `logVerbUsage` is fire-and-forget (never awaited, never throws); written from the DISPATCH layer so param-validation failures count. `readVerbUsage`/`earliestVerbUsageTs` feed `gbrain protocol stats` (incl. measured TTHW vs the init-stamped `protocol_installed_at`) and the doctor `memory_verbs_usage` check.
-- `src/core/feedback/` — use-attributed retrieval feedback (guide: `docs/guides/retrieval-feedback.md`). `settings.ts` resolves `feedback.*` with a 30s cache (`effectiveInfluence` is 0 when off). `record.ts`: `recordAnswer(ctx, { op, pages, links })` mints `ans_<ULID>`, returns it at once and enqueues the event (pages with the retrieved `content_hash`, relational path edges) on a bounded write-behind queue (cap 1,000, drop-oldest counted, drained by the background-work registry as `retrieval-feedback`); the citation signal (rating 4, half alpha) rides the same queue. Only callers that may change shared ranking record: `canTeachSource` allows the owner's CLI (`remote === false`) and auth-less stdio, and remote callers with write scope on that source, no slug fence, no operation allowlist excluding `rate_answer`, not delegated. `recordThinkAnswer` marks cited pages (synthesis `ok`, bare slug unambiguous across gathered sources). `feedbackMetaFields` adds `answer_id`/`feedback` only to rateable answers (callers that cannot rate see unchanged output). `rate.ts` `rateAnswer` implements `rate_answer`: whole-answer ratings (think/synthesize cited pages, else gathered; search-shaped returned pages; plus path edges into rated pages) or targeted `pages: [{ ref, rating }]` (pages only); skips `stale_revision`/`unverifiable_revision`/`not_authorized_source`; receipts carry weight before/after and the multiplier; refusals are registered codes. `store.ts` owns all SQL (TEXT[] binds, booleans bound as text): `insertRetrievalEvents`, `applyRatings` (one statement: idempotent feedback insert per element, then an upsert whose `DO UPDATE` recomputes from the locked row, so concurrent ratings never lose an update), `readWeights`, `pruneRetrievalFeedback` (purge phase), `resetRetrievalWeights`, `feedbackStatus`, `scrubPurgedSlugs` (#8821: both hard-delete paths, the page purge and the expired-tombstone purge, remove a purged slug's source-scoped rows as a page key or either end of an edge key, and drop it from `eval_candidates.retrieved_slugs`, in the purge transaction; the page-purge receipt checks each store). Pinned by `test/feedback-store.test.ts` and `test/e2e/feedback-parity.test.ts`.
+- `src/core/feedback/` — use-attributed retrieval feedback (guide: `docs/guides/retrieval-feedback.md`). `settings.ts` resolves `feedback.*` with a 30s cache (`effectiveInfluence` is 0 when off). `record.ts`: `recordAnswer(ctx, { op, pages, links })` mints `ans_<ULID>`, returns it at once and enqueues the event (pages with the retrieved `content_hash`, relational path edges) on a bounded write-behind queue (cap 1,000, drop-oldest counted, drained by the background-work registry as `retrieval-feedback`); the citation signal (rating 4, half alpha) rides the same queue. Only callers that may change shared ranking record: `canTeachSource` allows the owner's CLI (`remote === false`) and auth-less stdio, and remote callers with write scope on that source, no slug fence, no operation allowlist excluding `rate_answer`, not delegated. `recordThinkAnswer` marks cited pages (synthesis `ok`, bare slug unambiguous across gathered sources). `feedbackMetaFields` adds `answer_id`/`feedback` only to rateable answers (callers that cannot rate see unchanged output). `rate.ts` `rateAnswer` implements `rate_answer`: whole-answer ratings (think/synthesize cited pages, else gathered; search-shaped returned pages; plus path edges into rated pages) or targeted `pages: [{ ref, rating }]` (pages only); skips `stale_revision`/`unverifiable_revision`/`not_authorized_source`; receipts carry weight before/after and the multiplier; refusals are registered codes. `store.ts` owns all SQL (TEXT[] binds, booleans bound as text): `insertRetrievalEvents`, `applyRatings` (one statement: idempotent feedback insert per element, then an upsert whose `DO UPDATE` recomputes from the locked row, so concurrent ratings never lose an update), `readWeights`, `pruneRetrievalFeedback` (purge phase), `resetRetrievalWeights`, `feedbackStatus`, `scrubPurgedSlugs` (#8821: every hard delete of a page row, the page purge, the expired-tombstone purge and the engines' `deletePages` (connector, GitHub and Google deletes; #9068), removes a deleted slug's source-scoped rows as a page key or either end of an edge key, and drops it from `eval_candidates.retrieved_slugs` (`scrubCandidateSlugs`), in the delete's transaction; the page-purge receipt checks each store), `insertEvalCandidate` (#9068: the eval capture insert, keeping only slugs whose page row still exists, probed under the same key-share lock as the feedback writers). Pinned by `test/feedback-store.test.ts` and `test/e2e/feedback-parity.test.ts`.
- `src/core/ops/feedback.ts` — the `rate_answer` op (scope `write`, CLI `gbrain rate <answer_id> <rating>`), delegating to `feedback/rate.ts`. Answers carry `answer_id` from `ops/search.ts` (`search`, `query` via `buildRetrievalResponseMeta`), `ops/takes.ts` (`think`), `verbs.ts` (`synthesize`) and `ops/facts.ts` (`recall` page arm on the hybrid path).
diff --git a/docs/architecture/key-files/core-utilities-1.md b/docs/architecture/key-files/core-utilities-1.md
index 7a43f4ab1c..0e3ad04ff9 100644
--- a/docs/architecture/key-files/core-utilities-1.md
+++ b/docs/architecture/key-files/core-utilities-1.md
@@ -53,7 +53,7 @@ The `src/core/cycle.ts` entry is on its own page: [Core Cycle primitive](core-cy
- `src/core/eval-capture-scrub.ts` — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
-- `src/core/eval-capture.ts` — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers (catches MCP + CLI + subagent tool-bridge from one site). Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. Capture is off by default — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Contributors set `export GBRAIN_CONTRIBUTOR_MODE=1`. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
+- `src/core/eval-capture.ts` — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers (catches MCP + CLI + subagent tool-bridge from one site). Fire-and-forget; the insert (`insertEvalCandidate` in `feedback/store.ts`, both engines) keeps only slugs a page of the captured sources still holds, probed under the feedback writers' key-share lock, so a capture deferred past its response cannot write a purged slug back (#9068). Failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. Capture is off by default — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Contributors set `export GBRAIN_CONTRIBUTOR_MODE=1`. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
- `src/core/filing-audit.ts` + `skills/_brain-filing-rules.json` — Check 6 of `check-resolvable`. Parses `writes_pages:` / `writes_to:` frontmatter on skills and audits their filing claims against the filing-rules JSON (error severity). Internal `parseFrontmatter` is a thin wrapper over the shared `src/core/skill-frontmatter.ts` parser so both filing-audit and skill-brain-first read the same shape (`tools?`, `triggers?`, `brain_first?: 'exempt'`, typed `brain_first_typo`) from one source of truth.
diff --git a/docs/architecture/key-files/files-and-sync-1.md b/docs/architecture/key-files/files-and-sync-1.md
index 5f22b705a0..d08280740f 100644
--- a/docs/architecture/key-files/files-and-sync-1.md
+++ b/docs/architecture/key-files/files-and-sync-1.md
@@ -51,7 +51,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/source-resolver.ts` — 6-tier source resolution. `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside `resolveSourceId()` (unchanged). `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'sole_non_default', 'brain_default', 'seed_default']` (7 entries; order matches priority). Tier `sole_non_default` slots between `local_path` and `brain_default`: when NO `sources.default` config is set AND exactly one registered source has `local_path` AND isn't `'default'` AND the `'default'` source holds no active pages (the emptiness guard — the tier's charter is rescuing brains whose `'default'` is empty; when `'default'` carries an established corpus, auto-routing would hijack every bare write into the side-source, so the resolver falls through to `seed_default` and prints a one-line stderr notice naming both sides so the suppressed flip is diagnosable, same `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1` suppression knob as the routing nudge; a failed pages probe on an exotic/legacy schema keeps the auto-route), auto-route to it; archived sources excluded (try/catch for pre-v34 brains); private `pickSoleNonDefaultSource(engine)` shared by both resolver entry points so they cannot drift. Exported `formatSoleNonDefaultNudge(sourceId): string | null` builds the user-facing stderr nudge (null when `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1`). `resolveCliSyncSource` in `src/commands/sync/run.ts` calls `resolveSourceWithTier` unconditionally so the tier fires; `src/commands/import.ts:96-128` mirrors with the tier-gated nudge. Consumed by `gbrain sources current`, `import --source-id`, `extract --source-id`, and the `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (`withEnv()` per test-isolation lint), `test/source-resolver-sole-non-default.test.ts` (20 cases incl. both-ways emptiness-guard + probe-failure fallback), `test/sync-sole-non-default-routing.test.ts` (3 PGLite cases driving real `runSync`), `test/pages-source-scoping-4329.test.ts` (real-engine both-ways guard coverage). Also exports `noGrantFederatedScope(engine, hasSourceGrant, sourceId)`, the widening decision transports share: returns `localFederatedSourceIds(..., 'seed_default')` ONLY when `hasSourceGrant === false` (the legacy-bearer no-grant floor), `undefined` for a granted token, for an OAuth client (flag `undefined` — a falsy gate here would widen every OAuth client), and when the resolver throws (best-effort; the scalar scope stands rather than failing the request). Called by `src/commands/serve-http.ts`; pinned by `test/no-grant-federated-scope.test.ts` (6 cases). `explicitReadBinding(engine, sourceId, tier)` (#5081) is the explicit-tier twin used by the stdio transport for `GBRAIN_SOURCE` (`env`) and `.gbrain-source` (`dotfile`) bindings: `{ sourceId, via, sourceIds, optedOut }`, where `sourceIds` is the same set `localFederatedSourceIds` computes (shared `federatedSetFor`, so an isolated anchor admits only itself) and `optedOut` lists `federated === false` sources for the denial hint; `undefined` for every other tier. Pinned by `test/explicit-read-admission-5081.test.ts`. Unscoped default-write guard family (the `seed_default` tier is the only one that actually lands a write in `'default'`): `assessDefaultWriteGuard(engine)` → `DefaultWriteAssessment { shouldGuard, defaultPages, nonDefaultPages, nonDefaultSources, failed? }` — an unindexed full-`pages` aggregate; `shouldGuard` when ≥1 non-default source holds pages AND non-default pages outnumber default's; a query failure returns the fail-open no-guard verdict stamped `failed: true`, which cachers/latchers must not treat as settled. `assessDefaultWriteGuardOnce(engine)` is the per-engine WeakMap memo (arms on the first SUCCESSFUL assessment whatever its verdict, forgets a failed attempt so the next caller retries, coalesces concurrent callers; `__resetDefaultWriteGuardMemo` is the test seam). `assessUnscopedDefaultWrite(engine, tier, mutating)` → `{ warning, assessed }` (`assessed: false` only when the aggregate failed); `formatDefaultWriteWarning(a, sourceFlag?)` / `formatDefaultWriteRefusal(cmd, a)` render the two surfaces; `defaultWriteAllowedByEnv()` is the `GBRAIN_ALLOW_DEFAULT_WRITE=1` escape hatch. Consumers: `sync` refuses (dry-run warns), `import` warns via the memo, MCP stdio's `createDefaultWriteAdvisory` prints once. Also exports `isResolverUserError(e)` — the ONE predicate for this module's user-facing throws (`Source "…" not found.` / `not found or is archived.`, `Invalid --source value`, `Invalid GBRAIN_SOURCE value`), consumed by dream.ts, agent.ts and code-scope.ts so a CLI command turns an unknown/archived source into a clean stderr line + exit 1 while anything else keeps propagating. A request snapshot serves tiers 4, 5.5 (its rows carry the 'default' emptiness probe, `default_has_pages`), `assertSourceExists` and the federated set from one sources read, sent with the config table (`resolveSourceWithTier` reads tier 5's key beside the tier-4 listing) (GBRA-156).
-- `src/core/source-delete.ts` — `deleteSourceRow`: every unmanaged source delete (`sources remove`, `sources purge`, the `sources_remove` op, expired-archive purge) deletes the `sources` row and that incarnation's `persistence_source_bindings` row in one statement; bindings have no FK to `sources`. Bindings of other incarnations are left to `gbrain repair orphan-bindings` (`src/core/persistence/orphan-bindings.ts`, doctor `orphan_persistence_bindings`). The unmanaged claim check (`sync-authority.ts`) and doctor `unbound_source` count only a binding of the source's current incarnation. `EXPIRED_ARCHIVE_SQL` is the one purge-eligibility predicate (unmanaged sweep, its per-row DELETE and the managed `expiredOnly` purge): both archive timestamps must be later than the epoch and the expiry past; `UNVERIFIED_ARCHIVE_SQL` lists archived rows with a missing or epoch timestamp, which the sweep reports blocked with restore-then-archive guidance instead of purging.
+- `src/core/source-delete.ts` — `deleteSourceRow`: every unmanaged source delete (`sources remove`, `sources purge`, the `sources_remove` op, expired-archive purge) deletes the `sources` row and that incarnation's `persistence_source_bindings` row in one statement; bindings have no FK to `sources`. #9068: that statement also runs `removedSourceScrubCtes` (the managed `remove`/`purge` in `source-lifecycle.ts` runs them in its final transaction too), which delete the source's `retrieval_event_pages`, `retrieval_event_links`, `retrieval_feedback` and `context_volunteer_events` rows and strip from `eval_candidates.retrieved_slugs` each slug no other source the candidate covers has a page for (`retrieval_weights` and `mention_gazetteer_entries` cascade from `sources`); a second statement runs them again ungated, for a feedback write or capture that held a page row while the delete waited. Pinned by `test/purge-scrub-followups-9068.test.ts`. Bindings of other incarnations are left to `gbrain repair orphan-bindings` (`src/core/persistence/orphan-bindings.ts`, doctor `orphan_persistence_bindings`). The unmanaged claim check (`sync-authority.ts`) and doctor `unbound_source` count only a binding of the source's current incarnation. `EXPIRED_ARCHIVE_SQL` is the one purge-eligibility predicate (unmanaged sweep, its per-row DELETE and the managed `expiredOnly` purge): both archive timestamps must be later than the epoch and the expiry past; `UNVERIFIED_ARCHIVE_SQL` lists archived rows with a missing or epoch timestamp, which the sweep reports blocked with restore-then-archive guidance instead of purging.
- `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`. `assertNoOverlappingPath(engine, id, path)` is the ONE overlapping-path guard shared by every surface that binds a `local_path` (`addSource`, `gbrain sources set-path`): a path equal to, nested inside, or enclosing another source's `local_path` throws `SourceOpError('overlapping_path')` — overlapping trees make sync/write-through attribute files to the wrong source, so no add or repair path may skip it. The guard compares by spelling AND by realpath (both the candidate and each sibling `local_path` are `realpathSync`'d when they exist, fail-open to the spelling otherwise), so a symlink resolving into or over another source's tree cannot slip past it; the stored pointer and the error message keep the operator's spelling. #6946: `isOwnedClone`/`defaultCloneDir` live in `src/core/url-clone.ts`, re-exported here.
- `src/core/sync-concurrency.ts` — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the sites can't drift. `DEFAULT_PARALLEL_SOURCES = 4` is a SEPARATE constant for the per-source fan-out under `gbrain sync --all` — kept distinct from `DEFAULT_PARALLEL_WORKERS` because total live Postgres connections per wave ≈ `DEFAULT_PARALLEL_SOURCES × DEFAULT_PARALLEL_WORKERS × 2 (per-file pool)` = 32 at both defaults (each per-file worker opens its own `PostgresEngine` with `poolSize = min(2, resolvePoolSize(2))`); `sync.ts` warns when `parallel × workers × 2 > 16`. `resolveWorkersWithClamp(engine, override, commandName, fileCount)` wraps `autoConcurrency` with a per-command stderr clamp warning on PGLite (per-(command, requested) dedup via module-scoped warned-once set with `_resetWorkersClampWarningsForTest()` seam) and is the canonical surface for every bulk-command `--workers N` flag (extract-conversation-facts, extract, edges-backfill, reindex-multimodal, reindex, reindex-code); embed.ts deliberately bypasses it and keeps `GBRAIN_EMBED_CONCURRENCY || 20`. `resolveMaxConnections()` (reads `GBRAIN_MAX_CONNECTIONS`, undefined when unset) + `clampWorkersForConnectionBudget(workers, perWorkerPool, maxConnections, parentPool)` back the opt-in single-sync connection-footprint clamp so a big sync stays under a low pooler cap (`parent_pool + workers×perWorkerPool ≤ budget`); `gbrain doctor`'s `pool_budget` check (`computePoolBudgetCheck` / `checkPoolBudget` in `src/commands/doctor/checks/consolidation-cycle.ts`, re-exported by `src/commands/doctor.ts`) warns when the budget leaves no room for a worker, pointing at `GBRAIN_POOL_SIZE=2`. Pinned by `test/pglite-workers-clamp.test.ts`.
diff --git a/docs/architecture/key-files/memory-purge.md b/docs/architecture/key-files/memory-purge.md
index 8148c544df..72e6eefdf0 100644
--- a/docs/architecture/key-files/memory-purge.md
+++ b/docs/architecture/key-files/memory-purge.md
@@ -26,7 +26,7 @@ User-facing contract: [expire versus purge](../../guides/memory-boundaries.md#pu
- `src/core/persistence/purge-deleted.ts` — `purgeDeletedPagesCoordinated` backs the cycle `purge` phase, the purge job, `purge_deleted_pages` and `gbrain pages purge-deleted`. Unmanaged brains keep the engine bulk delete, after the belief-ledger expiry of recorded sources' tombstones: each is its own transaction and item (ItemRun site `purge.recorded_tombstones`), a run-scoped failure or a systemic run stops the purge, and any other failure lists the tombstone in `blocked` with its classified code, counts it in `failed` (no `error`, so the phase warns) and keeps it out of the bulk delete (`purgeDeletedPages` `excludeIds`); a content failure counts toward the same park as the managed purge, a transient one is retried next run, and parked tombstones are skipped as `purge_parked` unless `retryParked` (the unmanaged preview excludes them too). A managed brain purges each tombstone older than the window (skipping archived sources, which the source lifecycle purges) through `delete_page` with `purge: true` and the internal `retention: true` bound to its revision (retention is not an owner purge; see [belief ledger](belief-ledger.md)). Writer state is read once per run (`writerOnboardingPreflight`): a source whose writer is `recovery_required` submits nothing and its tombstones are listed in `blocked` as `writer_not_active` (counted in `inactive`). Revision races are deferred to the next run; a parked tombstone (`purge-parks.ts`) is skipped as `purge_parked` (counted in `parked`) unless `retryParked`, and any other failure is listed in `blocked` and sets `error`. `selectPurgeCandidates` is the one eligibility selector (preview, consent, `purge_deleted_pages` op, run); #6884: the run submits only while the writer has outstanding-request room (`readOutstandingRoom` in `journal.ts`, at `persistence.capacity.background_pause_pct` of the cap for a background run); the rest, and the rest after a `queue_capacity` refusal, is `blocked` as `write_capacity_deferred` and counted in `capacity_deferred`, never a failure or a park. `purgePhaseStatus`/`purgePhaseNote` shape the cycle purge phase (fail on `error`; warn on `writer_not_active`, parked, capacity-deferred or an unmanaged `failed` tombstone). The engine bulk delete (`engine-sql/pages.ts` `purgeDeletedPages`) also deletes, in its transaction, the purged pages' stagings (by source and slug) and the segment texts only their manifests referenced. Tests: `test/managed-writers-w3.test.ts`, `test/purge-deleted-parks.test.ts`, `test/purge-deleted-capacity-6884.test.ts`, `test/pathological/a49-unmanaged-purge-tombstones.test.ts`.
- `src/core/persistence/derived-retirement-hold.ts` — GBRA-161: the purge hold for every derived retirement. `DERIVED_RETIRED_BY` lists the `retired_by` stamps derived writers put on the pages they retire (extract-atoms, managed-reextract, stale-atoms, life-chronicle); `derivedRetirementPurgeHold(alias)` keeps a stamped page out of every purge route (engine bulk purge and its preview, the coordinated selection and delete) until `DERIVED_RETIREMENT_HOLD_DAYS` (30) after `deleted_at` (the database clock, never the `retired_at` text); `retireDerivedPages` versions, soft-deletes and stamps a batch in one attributed transaction (unmanaged extract_atoms). `assertRetirementStampUnchanged` is the page preparer's refusal (`reserved_frontmatter_key`) of a remote write that adds or changes one of `RESERVED_RETIREMENT_KEYS`. The hold length follows `generations.restore_days` (`RESTORE_DAYS_SQL`, read inside the purge statement; 30 by default) and `retireDerivedPagesInTransaction` is the in-transaction form generation adapters retire through. An explicit `gbrain delete --purge` is not held.
- `src/core/generations/ledger-scrub.ts` — GBRA-161: purge reaches the generation ledger. A page purge (`page-purge.ts`) nulls the purged item's preimages in `derived_generation_items` and every held proposal in `derived_generations` naming it; a fact purge (`facts/purge.ts`) nulls every preimage, proposal and held context carrying the claim. Both are `swept` (`generation_ledger`) in the deletion inventory, so the verification probes them; `gbrain repair generations` then refuses a scrubbed item with `no_preimage`.
-- `src/core/persistence/purge-name-stores.ts` — #9068: `scrubPurgedNameStores(engine, sourceId, slugs)` deletes the source's `context_volunteer_events` rows (whose rationale quotes the matched name) and `mention_gazetteer_entries` rows (`name_norm -> target_slug`) for purged slugs. Every hard delete of a page calls it inside its transaction (`purgePageInTransaction`, `purgeDeletedPages`, `expireRecordedTombstones`, the restore purge replay in `belief/purge-mirror.ts`); a soft delete keeps both. Both tables are `swept` (`page_names`) in the deletion inventory; the page receipt checks them by slug, and the fact receipt probes them (rationale, name_norm) and reports a match as `probed_reported`, because a fact purge sweeps neither.
+- `src/core/persistence/purge-name-stores.ts` — #9068: `scrubPurgedNameStores(engine, sourceId, slugs)` deletes the source's `context_volunteer_events` rows (whose rationale quotes the matched name) and `mention_gazetteer_entries` rows (`name_norm -> target_slug`) for purged slugs. Every hard delete of a page calls it inside its transaction (`purgePageInTransaction`, `purgeDeletedPages`, `expireRecordedTombstones`, the restore purge replay in `belief/purge-mirror.ts`, and the engines' batch `deletePages`); removing a source deletes the source's volunteer rows (`removedSourceScrubCtes` in `source-delete.ts`) and its gazetteer rows by cascade; a soft delete keeps both. Both tables are `swept` (`page_names`) in the deletion inventory; the page receipt checks them by slug, and the fact receipt probes them (rationale, name_norm) and reports a match as `probed_reported`, because a fact purge sweeps neither.
- `src/core/persistence/atom-retirement-hold.ts` — #7013: the one SQL rule for atoms `gbrain repair atoms-restore` restores. `atomRetirementsSql(scope)` is set-based (CTEs over `pages` joined on source and origin key, linear in the atoms in scope) and returns every non-imported atom soft-deleted with `retired_by: managed-reextract` with `origin_live` (a live page origin in its source, or a transcript), `replaced` (a live atom of the same origin carries another stripped source hash), `latest` (its hash is the hash of the origin's latest `managed-reextract` retirement) and `restorable` (all three right); `atomOriginScopeSql` narrows it to one atom's source and origin through the frontmatter GIN index for the apply recheck; `atomRetirementPurgeHold`, which keeps such an atom out of every batch purge selection (engine `purgeDeletedPages` preview and delete, the belief-ledger expiry, `selectPurgeCandidates`) for `ATOM_RESTORE_HOLD_DAYS` (30) after `deleted_at`, next to the #6729 `identityDowngradePurgeHold`. An explicit `delete --purge` is not held. Static strings only, so `engine-sql/pages.ts` imports it.
- `src/core/repair/atoms-restore.ts` — `gbrain repair atoms-restore` (#7013), explicit-only and preview-bound: lists atoms an empty managed re-extraction retired (`retired_by: managed-reextract`) that `atomRetirementsSql` (`persistence/atom-retirement-hold.ts`) marks `restorable`, each with origin, stripped source hash and `retired_at`, and residuals `restorable`, `origins`, `origin_gone`, `replaced`, `older_generation`; `--apply --expect <hash>` restores the approved set, rechecking each atom (id, incarnation, revision, origin, hash, `retired_at`, rule match) and reporting `changed_since_preview` otherwise. Managed: one `managed_maintenance_restore_reextract_atom` request per atom (`deleted_page: true`, bound to the deleted revision, request id in the outcome detail); `prepareAtomRestore` (dispatched from `prepared-maintenance.ts`) rechecks the rule under the atom and origin page locks and publishes a revision-bound `revert_version` to `version_id`, the newest live version of the atom at its source hash without `retired_by` (the one the coordinated delete archived; file written back, `allowMissingFile`); none is `no_version`, lowered to the `derive:atoms` tier of its origin with taint edges. Unmanaged: `createVersion`, `restorePage` and the two keys dropped in one `maintenanceTransaction`. Extraction state is not touched. Test: `test/pathological/h5-atoms-restore.test.ts`.
- `src/core/repair/atom-fence-rows.ts` — `gbrain repair atom-fence-rows`, explicit-only and preview-bound: `findAtomFenceRows` lists live page-bound `visibility: world` atoms whose title, body or `source_quote` contains (case and whitespace folded, claims of at least `MIN_CLAIM_CHARS`) the claim of a facts row of their live source page whose visibility is not world or a takes row whose holder is not world, absent from the page's `publicBody`; keyset pages of 500 atoms whose source page names a facts or takes fence, each source page read once, segmented source pages counted `origin_unread`. The preview lists atom, source page and row kinds and numbers (never row text) under a saved hash; `--apply --expect <hash>` rechecks each atom and sets `visibility: private` through a revision-bound put_page (`narrowed`), else `changed_since_preview`. Doctor `atom_fence_rows` shares the scan. Test: `test/repair-atom-fence-rows.test.ts` (unmanaged and managed, PGLite and Postgres).
diff --git a/src/core/access/federation-exposure.ts b/src/core/access/federation-exposure.ts
index 8d9c62c3f2..cd9ea9c823 100644
--- a/src/core/access/federation-exposure.ts
+++ b/src/core/access/federation-exposure.ts
@@ -5,7 +5,9 @@
* becomes readable by every connection with no source grant. This read names those pages so the operator sees them;
* it writes nothing and never refuses the change.
*
- * A page counts when it names no readers and no folder rule and the writer of its live revision or of any earlier
+ * A page counts when it names no readers and no folder rule, the read paths' privacy rules (privacyPartSql: visibility,
+ * aliases, pinned questions, lineage) do not hide it from a no-grant connection (#9068; the search.remote_private_pages
+ * and GBRAIN_REMOTE_PRIVATE_PAGES opt-outs relax them here too), and the writer of its live revision or of any earlier
* version (the write attribution columns) is a remote credential: a token or OAuth client, revoked or deleted since. No
* record keeps which member a credential was bound to when it wrote, so a credential unbound or rebound since then
* counts too, and each example names the member it is bound to now (null when none or deleted). A write that moves
@@ -13,7 +15,8 @@
* repo-less brain answers the same way.
*/
import { shellQuote } from '../agent-output.ts';
-import type { SqlEngine } from '../persistence/model.ts';
+import type { BrainEngine } from '../engine.ts';
+import { privacyPartSql, remotePrivacyRelaxed } from '../search/private-visibility.ts';
export interface PagesExposed {
count: number;
@@ -23,10 +26,11 @@ export interface PagesExposed {
}
/** The pages of `sourceId` a remote credential wrote that name no readers (the first five by slug), or null when there are none. */
-export async function remotePagesFederationExposes(engine: SqlEngine, sourceId: string): Promise<PagesExposed | null> {
+export async function remotePagesFederationExposes(engine: BrainEngine, sourceId: string): Promise<PagesExposed | null> {
+ const hidden = await remotePrivacyRelaxed(engine) ? '' : ` AND ${privacyPartSql('p')}`;
const rows = await engine.executeRaw<{ slug: string; member: string | null; total: number }>(
- `WITH open AS (SELECT id, slug, revision_principal_kind, revision_principal_id FROM pages
- WHERE source_id = $1 AND deleted_at IS NULL AND page_readers IS NULL AND folder_rule_ids IS NULL),
+ `WITH open AS (SELECT p.id, p.slug, p.revision_principal_kind, p.revision_principal_id FROM pages p
+ WHERE p.source_id = $1 AND p.deleted_at IS NULL AND p.page_readers IS NULL AND p.folder_rule_ids IS NULL${hidden}),
authored AS (
SELECT slug, revision_principal_kind AS kind, revision_principal_id AS pid FROM open
UNION ALL
@@ -54,7 +58,7 @@ export async function remotePagesFederationExposes(engine: SqlEngine, sourceId:
* remotePagesFederationExposes when `federating` (the change added the source to the no-grant floor), else null. A
* failed read is reported on stderr and reads as null, so the check never changes whether or how the federation happens.
*/
-export async function remotePagesFederationExposesOrWarn(engine: SqlEngine, sourceId: string, federating: boolean): Promise<PagesExposed | null> {
+export async function remotePagesFederationExposesOrWarn(engine: BrainEngine, sourceId: string, federating: boolean): Promise<PagesExposed | null> {
if (!federating) return null;
return remotePagesFederationExposes(engine, sourceId).catch((e: unknown) => {
console.error(`Warning: could not check ${shellQuote([sourceId])} for remote-written pages without readers: ${e instanceof Error ? e.message : String(e)}`);
diff --git a/src/core/engine-sql/pages.ts b/src/core/engine-sql/pages.ts
index 29b2343ccb..c52ae58851 100644
--- a/src/core/engine-sql/pages.ts
+++ b/src/core/engine-sql/pages.ts
@@ -207,9 +207,12 @@ export async function deletePage(exec: SqlExecutor, slug: string, opts?: { sourc
/**
* v0.41.19.0 — batch delete primitive. See BrainEngine.deletePages JSDoc.
- * Single SQL round-trip per call; caller is responsible for chunking input
- * to <= DELETE_BATCH_SIZE. RETURNING slug projects the actually-deleted set
- * so the caller can filter pagesAffected.
+ * Caller is responsible for chunking input to <= DELETE_BATCH_SIZE.
+ * RETURNING slug projects the actually-deleted set so the caller can filter
+ * pagesAffected. #9068: the rows that name a deleted page outside any foreign
+ * key (retrieval logs and ratings, the volunteer log, the gazetteer, captured
+ * eval slugs) go in the same transaction, after the page row: a feedback write
+ * holding the row commits first and is scrubbed, and a later one finds no page.
*/
export async function deletePages(exec: SqlExecutor, slugs: string[], opts: { sourceId: string }): Promise<string[]> {
if (slugs.length === 0) return [];
@@ -218,12 +221,17 @@ export async function deletePages(exec: SqlExecutor, slugs: string[], opts: { so
`deletePages: input size ${slugs.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`,
);
}
- const { rows } = await exec.run<{ slug: string }>(sqlFragment`
- DELETE FROM pages
- WHERE slug = ANY(${slugs}::text[]) AND source_id = ${opts.sourceId}
- RETURNING slug
- `);
- return rows.map(r => r.slug);
+ return exec.transaction(async (tx) => {
+ const { rows } = await tx.run<{ slug: string }>(sqlFragment`
+ DELETE FROM pages
+ WHERE slug = ANY(${slugs}::text[]) AND source_id = ${opts.sourceId}
+ RETURNING slug
+ `);
+ const deleted = rows.map(r => r.slug);
+ await scrubPurgedSlugs(tx, opts.sourceId, deleted);
+ await scrubPurgedNameStores(tx, opts.sourceId, deleted);
+ return deleted;
+ });
}
/**
diff --git a/src/core/engine.ts b/src/core/engine.ts
index 5f51dbf0b9..be0c636b99 100644
--- a/src/core/engine.ts
+++ b/src/core/engine.ts
@@ -940,6 +940,9 @@ export interface BrainEngine {
* the sync loop falls back to per-slug `deletePage` to log unrecoverable
* failures to `failedFiles`). `gbrain sync` calls this on EVERY run that
* sees a deleted file — it is NOT admin-only.
+ *
+ * Unlike `deletePages` it scrubs nothing else: the page purge, its main
+ * caller, scrubs those stores itself and records them in its receipt.
*/
deletePage(slug: string, opts?: { sourceId?: string }): Promise<void>;
/**
@@ -960,7 +963,13 @@ export interface BrainEngine {
* phases don't waste lookups on phantom slugs (paths that were in the
* deletion list but had no DB row).
*
- * ATOMICITY: one statement, one transaction. The whole batch commits or
+ * #9068: in the same transaction, after the delete, it scrubs the rows
+ * that name a deleted page outside any foreign key: retrieval_event_pages,
+ * retrieval_event_links, retrieval_feedback, retrieval_weights,
+ * context_volunteer_events, mention_gazetteer_entries and the slug in
+ * eval_candidates.retrieved_slugs (scrubPurgedSlugs, scrubPurgedNameStores).
+ *
+ * ATOMICITY: one transaction. The whole batch commits or
* the whole batch rolls back. Coarser than the per-row `deletePage`
* cadence — a mid-loop abort or transient connection failure can roll
* back up to `DELETE_BATCH_SIZE - 1` successful deletes from the
diff --git a/src/core/feedback/store.ts b/src/core/feedback/store.ts
index ecf244762b..d7efb947ff 100644
--- a/src/core/feedback/store.ts
+++ b/src/core/feedback/store.ts
@@ -5,6 +5,7 @@
* run the same SQL through `engine.executeRaw` with positional binds (no JSONB).
*/
import type { BrainEngine } from '../engine.ts';
+import type { EvalCandidateInput } from '../types.ts';
export type FeedbackOp = 'query' | 'search' | 'think' | 'synthesize' | 'recall';
export type FeedbackSignal = 'explicit' | 'cited';
@@ -329,6 +330,32 @@ export async function scrubPurgedSlugs(
};
}
+/**
+ * #9068: insert one captured eval candidate with only the slugs a page of its sources (any source when it names none)
+ * still holds, in retrieval order. Each probe takes the feedback writers' key-share lock on the page row, so a capture
+ * deferred past its query's response either commits before a purge takes the row, and the purge's scrub strips it, or
+ * waits for the purge and drops the slug. Both engines run this statement.
+ */
+export async function insertEvalCandidate(engine: Pick<BrainEngine, 'executeRaw'>, input: EvalCandidateInput): Promise<number> {
+ const rows = await engine.executeRaw<{ id: number | string }>(
+ `INSERT INTO eval_candidates (
+ tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids,
+ expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied,
+ latency_ms, remote, job_id, subagent_id, embedding_column
+ ) VALUES ($1, $2,
+ ARRAY(SELECT u.s FROM unnest($3::text[]) WITH ORDINALITY AS u(s, n)
+ WHERE EXISTS (SELECT 1 FROM pages pg WHERE pg.slug = u.s
+ AND (cardinality($5::text[]) = 0 OR pg.source_id = ANY($5::text[])) FOR KEY SHARE)
+ ORDER BY u.n),
+ $4::int[], $5::text[], $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)
+ RETURNING id`,
+ [input.tool_name, input.query, input.retrieved_slugs, input.retrieved_chunk_ids, input.source_ids, input.expand_enabled, input.detail,
+ input.detail_resolved, input.vector_enabled, input.expansion_applied, input.latency_ms, input.remote, input.job_id, input.subagent_id,
+ input.embedding_column ?? null],
+ );
+ return Number(rows[0]!.id);
+}
+
export async function resetRetrievalWeights(
engine: BrainEngine,
opts: { sourceId?: string; slug?: string },
diff --git a/src/core/persistence/source-lifecycle.ts b/src/core/persistence/source-lifecycle.ts
index 6c960d5e97..7132e5816b 100644
--- a/src/core/persistence/source-lifecycle.ts
+++ b/src/core/persistence/source-lifecycle.ts
@@ -8,7 +8,7 @@ import { readFix } from '../ops/op-fix.ts';
import type { Action } from '../agent-output.ts';
import { catalogueError } from '../error-catalogue.ts';
import { isValidSourceId } from '../source-id.ts';
-import { EXPIRED_ARCHIVE_SQL } from '../source-delete.ts';
+import { EXPIRED_ARCHIVE_SQL, removedSourceScrubCtes } from '../source-delete.ts';
import { parseSourceConfig } from '../sources-load.ts';
import { redactSourceConfig } from '../source-config-redact.ts';
import { discoverGitRoot } from '../sync-git.ts';
@@ -342,7 +342,8 @@ export async function runManagedSourceLifecycle(engine:BrainEngine,input:SourceL
{fix:readFix('Lists OAuth clients with the source each is bound to.',{argv:['gbrain','auth','clients','--json']})});
const [impact]=await tx.executeRaw<{count:string}>('SELECT count(*)::text AS count FROM pages WHERE source_id=$1',[input.sourceId]);
pagesDeleted+=Number(impact.count);
- await tx.executeRaw('DELETE FROM sources WHERE id=$1 AND incarnation=$2::uuid',[input.sourceId,incarnation]);
+ await tx.executeRaw(`WITH gone AS (DELETE FROM sources WHERE id=$1 AND incarnation=$2::uuid RETURNING id),${removedSourceScrubCtes('EXISTS (SELECT 1 FROM gone)')} SELECT 1`,[input.sourceId,incarnation]);
+ await tx.executeRaw(`WITH ${removedSourceScrubCtes('true')} SELECT 1`,[input.sourceId]);
await tx.executeRaw('DELETE FROM persistence_source_bindings WHERE source_id=$1 AND source_incarnation=$2::uuid',[input.sourceId,incarnation]);
await supersedeTopologyRetires(tx,input.sourceId,incarnation,principal,requestId);
}
diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts
index fe709980a3..46699a1d57 100644
--- a/src/core/pglite-engine.ts
+++ b/src/core/pglite-engine.ts
@@ -173,6 +173,7 @@ import * as keywordPagesImpl from './engine-sql/keyword-pages.ts';
import { applyForwardReferenceBootstrap, pgliteBootstrapTarget } from './engine-sql/bootstrap.ts';
import { wellFormedJson } from './text-safe.ts';
import { ENGINE_ROOT } from './engine-root.ts';
+import { insertEvalCandidate } from './feedback/store.ts';
/**
* #4284 — opt-in out-of-band watchdog for a PGLite disconnect with a live
@@ -2996,32 +2997,7 @@ export class PGLiteEngine implements BrainEngine {
// Eval capture (v0.25.0). See BrainEngine interface docs.
async logEvalCandidate(input: EvalCandidateInput): Promise<number> {
- const { rows } = await this.db.query<{ id: number }>(
- `INSERT INTO eval_candidates (
- tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids,
- expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied,
- latency_ms, remote, job_id, subagent_id, embedding_column
- ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)
- RETURNING id`,
- [
- input.tool_name,
- input.query,
- input.retrieved_slugs,
- input.retrieved_chunk_ids,
- input.source_ids,
- input.expand_enabled,
- input.detail,
- input.detail_resolved,
- input.vector_enabled,
- input.expansion_applied,
- input.latency_ms,
- input.remote,
- input.job_id,
- input.subagent_id,
- input.embedding_column ?? null,
- ]
- );
- return rows[0]!.id;
+ return insertEvalCandidate(this, input);
}
async listEvalCandidates(filter?: { since?: Date; limit?: number; tool?: 'query' | 'search' }): Promise<EvalCandidate[]> {
diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts
index 5a6ee81546..9a1ecd891e 100644
--- a/src/core/postgres-engine.ts
+++ b/src/core/postgres-engine.ts
@@ -182,6 +182,7 @@ import { scopedRead, unscopedExecutor } from './engine-sql/brands.ts';
import { writerStamp } from './persistence/writer-versions.ts';
import { wellFormedJson } from './text-safe.ts';
import { ENGINE_ROOT } from './engine-root.ts';
+import { insertEvalCandidate } from './feedback/store.ts';
function escapeSqlStringLiteral(value: string): string {
return value.replace(/'/g, "''");
@@ -3082,20 +3083,7 @@ export class PostgresEngine implements BrainEngine {
// Eval capture (v0.25.0). See BrainEngine interface docs.
async logEvalCandidate(input: EvalCandidateInput): Promise<number> {
- const sql = this.sql;
- const rows = await sql`
- INSERT INTO eval_candidates (
- tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids,
- expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied,
- latency_ms, remote, job_id, subagent_id, embedding_column
- ) VALUES (
- ${input.tool_name}, ${input.query}, ${input.retrieved_slugs}, ${input.retrieved_chunk_ids}, ${input.source_ids},
- ${input.expand_enabled}, ${input.detail}, ${input.detail_resolved}, ${input.vector_enabled}, ${input.expansion_applied},
- ${input.latency_ms}, ${input.remote}, ${input.job_id}, ${input.subagent_id}, ${input.embedding_column ?? null}
- )
- RETURNING id
- `;
- return rows[0]!.id as number;
+ return insertEvalCandidate(this, input);
}
async listEvalCandidates(filter?: { since?: Date; limit?: number; tool?: 'query' | 'search' }): Promise<EvalCandidate[]> {
diff --git a/src/core/source-delete.ts b/src/core/source-delete.ts
index 288a7d00ab..20e398411b 100644
--- a/src/core/source-delete.ts
+++ b/src/core/source-delete.ts
@@ -17,7 +17,9 @@ export const unverifiedArchiveReason = (id: string) =>
* binding (#5732). `persistence_source_bindings` has no FK to `sources`, so a
* delete that leaves the binding would make a same-id replacement read as
* claimed. A binding of another incarnation is never touched here; `gbrain
- * repair orphan-bindings` removes those. Returns whether a row was deleted.
+ * repair orphan-bindings` removes those. The same statement runs
+ * removedSourceScrubCtes, and a second statement repeats it once the row is
+ * gone. Returns whether a row was deleted.
*/
export async function deleteSourceRow(
engine: Pick<BrainEngine, 'executeRaw'>,
@@ -30,9 +32,30 @@ export async function deleteSourceRow(
const rows = await engine.executeRaw<{ id: string }>(
`WITH gone AS (DELETE FROM sources WHERE id = $1 ${expired} RETURNING id, incarnation),
unbound AS (DELETE FROM persistence_source_bindings b USING gone
- WHERE b.source_id = gone.id AND b.source_incarnation = gone.incarnation)
+ WHERE b.source_id = gone.id AND b.source_incarnation = gone.incarnation),
+ ${removedSourceScrubCtes('EXISTS (SELECT 1 FROM gone)')}
SELECT id FROM gone`,
[id],
);
- return rows.length > 0;
+ if (!rows.length) return false;
+ await engine.executeRaw(`WITH ${removedSourceScrubCtes('true')} SELECT 1`, [id]);
+ return true;
+}
+
+/**
+ * #9068: CTEs that remove what names a removed source's pages outside a foreign key, for source `$1`, each only
+ * when `gate` holds: the source's retrieval_event_pages, retrieval_event_links, retrieval_feedback and
+ * context_volunteer_events rows, and each eval candidate's captured slugs that no other source the candidate covers
+ * still has a page for (retrieval_weights and mention_gazetteer_entries cascade from sources). Every removal runs
+ * them in the statement that deletes the sources row (gated on that delete), so the source and those rows go
+ * together, then again, ungated, in a later statement: that statement reads past a feedback write or eval capture
+ * that held one of the source's page rows while the delete waited for it (a later one finds no page).
+ */
+export function removedSourceScrubCtes(gate: string): string {
+ const deletes = ['retrieval_event_pages', 'retrieval_event_links', 'retrieval_feedback', 'context_volunteer_events']
+ .map(table => `scrub_${table} AS (DELETE FROM ${table} WHERE source_id = $1 AND ${gate})`);
+ const elsewhere = (slug: string) => `EXISTS (SELECT 1 FROM pages pg WHERE pg.slug = ${slug} AND pg.source_id = ANY(c.source_ids) AND pg.source_id <> $1)`;
+ return [...deletes, `scrub_eval_candidates AS (UPDATE eval_candidates c
+ SET retrieved_slugs = ARRAY(SELECT u.s FROM unnest(c.retrieved_slugs) WITH ORDINALITY AS u(s, n) WHERE ${elsewhere('u.s')} ORDER BY u.n)
+ WHERE $1 = ANY(c.source_ids) AND EXISTS (SELECT 1 FROM unnest(c.retrieved_slugs) AS r(s) WHERE NOT ${elsewhere('r.s')}) AND ${gate})`].join(',\n ');
}
diff --git a/test/eval-candidates.test.ts b/test/eval-candidates.test.ts
index b5065dfca3..7909f24ee1 100644
--- a/test/eval-candidates.test.ts
+++ b/test/eval-candidates.test.ts
@@ -11,6 +11,7 @@
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import type { EvalCandidateInput, EvalCaptureFailureReason } from '../src/core/types.ts';
+import { importFromContent } from '../src/core/import-file.ts';
let engine: PGLiteEngine;
@@ -18,6 +19,9 @@ beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
+ for (const slug of ['people/alice-example', 'companies/acme-example']) {
+ await importFromContent(engine, slug, `---\ntitle: ${slug}\ntype: note\n---\n# ${slug}\n`, { noEmbed: true, sourceId: 'default' });
+ }
});
afterAll(async () => {
diff --git a/test/federation-exposure-private-9068.test.ts b/test/federation-exposure-private-9068.test.ts
new file mode 100644
index 0000000000..374e861ae0
--- /dev/null
+++ b/test/federation-exposure-private-9068.test.ts
@@ -0,0 +1,109 @@
+/**
+ * #9068 follow-up: the federation exposure warning counts only pages a no-grant connection would read.
+ *
+ * 1. Protects: a remote credential writes a `visibility: private` page with no readable_by into an isolated source.
+ * Federating the source does not make that page readable to a connection with no source grant (the read paths'
+ * privacy predicate hides it), so `sources federate` no longer names it; the open page it wrote still is. With
+ * search.remote_private_pages=visible the read paths show private pages, and the warning names it again.
+ * 2. Fails when: the warning ignores the privacy predicate (it named the private page on master) or ignores the
+ * operator opt-out (it would stay silent about a page the floor then reads).
+ * 3. Existing coverage: sources-federate-member-pages (PGLite, open pages only).
+ * 4. Real seams: PGLite, and Postgres when DATABASE_URL is set (test/postgres-unit-arms.txt); tokens from the token
+ * service verified by the OAuth provider, put_page and get_page through MCP dispatch, the real sources command;
+ * repo-less brain.
+ */
+import { afterAll, beforeAll, describe, expect, spyOn, test } from 'bun:test';
+import { randomUUID } from 'node:crypto';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts';
+import { sqlQueryForEngine } from '../src/core/sql-query.ts';
+import { mintLegacyToken } from '../src/core/token-mint.ts';
+import { dispatchToolCall } from '../src/mcp/dispatch.ts';
+import { runSources } from '../src/commands/sources.ts';
+import { noGrantFederatedScope } from '../src/core/source-resolver.ts';
+import { __resetPrivateVisibilityCacheForTests, REMOTE_PRIVATE_PAGES_KEY } from '../src/core/search/private-visibility.ts';
+import type { AuthInfo } from '../src/core/ops/contract.ts';
+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const SRC = 'source-a';
+const engines: BrainEngine[] = [];
+let closePostgres: (() => Promise<void>) | undefined;
+const tokens = new Map<BrainEngine, Record<string, string>>();
+const note = (title: string, body: string, extra: string[] = []) => ['---', 'type: note', 'title: ' + title, ...extra, '---', '# ' + title, '', body].join('\n');
+
+async function call(engine: BrainEngine, who: string, op: string, params: Record<string, unknown>) {
+ const provider = new GBrainOAuthProvider({ sql: sqlQueryForEngine(engine), transaction: fn => engine.transaction(tx => fn(sqlQueryForEngine(tx))) });
+ const auth = await provider.verifyAccessToken(tokens.get(engine)![who]!) as unknown as AuthInfo;
+ const sourceId = auth.sourceId ?? 'default';
+ const localFederatedSourceIds = await noGrantFederatedScope(engine, auth.hasSourceGrant, sourceId);
+ const out = await dispatchToolCall(engine, op, { ...params }, { remote: true, transport: 'http', auth, sourceId, takesHoldersAllowList: ['world'],
+ ...(localFederatedSourceIds ? { localFederatedSourceIds } : {}) });
+ return { isError: out.isError === true, text: out.content.map(c => c.text).join('\n') };
+}
+
+async function sources(engine: BrainEngine, args: string[]): Promise<{ stdout: string; stderr: string }> {
+ const out: string[] = [], err: string[] = [];
+ const log = spyOn(console, 'log').mockImplementation((...a: unknown[]) => { out.push(a.join(' ')); });
+ const warn = spyOn(console, 'error').mockImplementation((...a: unknown[]) => { err.push(a.join(' ')); });
+ try { await runSources(engine, args); } finally { log.mockRestore(); warn.mockRestore(); }
+ return { stdout: out.join('\n'), stderr: err.join('\n') };
+}
+
+beforeAll(async () => {
+ const backends = testBackends();
+ if (backends.includes('pglite')) { const lite = new PGLiteEngine(); await lite.connect({ engine: 'pglite' } as never); await lite.initSchema(); engines.push(lite); }
+ if (backends.includes('postgres')) {
+ const pg = await isolatedPersistencePostgres(process.env.DATABASE_URL!);
+ engines.push(pg.engine); closePostgres = pg.close;
+ }
+ for (const engine of engines) {
+ await engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1)', [SRC]);
+ await engine.executeRaw("INSERT INTO access_members (id) VALUES ('alice-example')");
+ const minted: Record<string, string> = {};
+ for (const [name, member, grant] of [['alice', 'alice-example', [SRC, 'default']], ['public', null, undefined]] as const) {
+ const t = await mintLegacyToken(engine, { name, scopes: ['read', 'write'], takesHolders: ['world'], ...(grant ? { sourceGrant: [...grant] } : {}) });
+ minted[name] = t.token;
+ if (member) await engine.executeRaw('UPDATE access_tokens SET member_id = $2 WHERE id = $1::uuid', [t.id, member]);
+ }
+ tokens.set(engine, minted);
+ }
+}, 120_000);
+afterAll(async () => { await engines.find(e => e.kind === 'pglite')?.disconnect(); await closePostgres?.(); }, 60_000);
+
+describe('#9068 the federation exposure warning applies the read paths\' privacy predicate', () => {
+ test('a private page with no readers is not named; the open page is; the opt-out names both', async () => {
+ for (const engine of engines) {
+ __resetPrivateVisibilityCacheForTests();
+ const canary = randomUUID().slice(0, 8);
+ const hidden = 'notes/alice-private-' + canary, open = 'notes/alice-open-' + canary;
+ expect((await call(engine, 'alice', 'put_page', { slug: hidden, content: note('Alice private', 'Private ' + canary + '.', ['visibility: private']), request_id: randomUUID() })).isError).toBe(false);
+ expect((await call(engine, 'alice', 'put_page', { slug: open, content: note('Alice open', 'Open ' + canary + '.'), request_id: randomUUID() })).isError).toBe(false);
+ const rows = await engine.executeRaw<{ slug: string; page_readers: unknown; visibility: string | null }>(
+ "SELECT slug, page_readers, frontmatter->>'visibility' AS visibility FROM pages WHERE source_id = $1 ORDER BY slug", [SRC]);
+ expect(rows).toEqual([{ slug: open, page_readers: null, visibility: null }, { slug: hidden, page_readers: null, visibility: 'private' }]);
+
+ const out = await sources(engine, ['federate', SRC, '--json']);
+ expect({ engine: engine.kind, exposed: JSON.parse(out.stdout).pages_exposed }).toEqual({ engine: engine.kind, exposed: {
+ count: 1, examples: [{ slug: open, member: 'alice-example' }],
+ check: 'gbrain access who-can-read ' + open + ' --source ' + SRC,
+ restrict: 'gbrain access rule add --prefix notes/ --readers alice-example --source ' + SRC } });
+ expect((await call(engine, 'public', 'get_page', { slug: open, include_content: true })).text).toContain('Open ' + canary + '.');
+ expect((await call(engine, 'public', 'get_page', { slug: hidden, include_content: true })).text).not.toContain('Private ' + canary + '.');
+
+ await sources(engine, ['unfederate', SRC]);
+ await engine.setConfig(REMOTE_PRIVATE_PAGES_KEY, 'visible');
+ __resetPrivateVisibilityCacheForTests();
+ try {
+ const relaxed = await sources(engine, ['federate', SRC, '--json']);
+ expect(JSON.parse(relaxed.stdout).pages_exposed).toMatchObject({ count: 2, examples: [{ slug: open }, { slug: hidden }] });
+ expect((await call(engine, 'public', 'get_page', { slug: hidden, include_content: true })).text).toContain('Private ' + canary + '.');
+ } finally {
+ await engine.executeRaw('DELETE FROM config WHERE key = $1', [REMOTE_PRIVATE_PAGES_KEY]);
+ __resetPrivateVisibilityCacheForTests();
+ await sources(engine, ['unfederate', SRC]);
+ }
+ }
+ }, 240_000);
+});
diff --git a/test/federation-exposure-warn.test.ts b/test/federation-exposure-warn.test.ts
index 2d721616f1..c035d0825d 100644
--- a/test/federation-exposure-warn.test.ts
+++ b/test/federation-exposure-warn.test.ts
@@ -8,11 +8,11 @@
*/
import { describe, expect, spyOn, test } from 'bun:test';
import { pagesExposedLines, remotePagesFederationExposesOrWarn, type PagesExposed } from '../src/core/access/federation-exposure.ts';
-import type { SqlEngine } from '../src/core/persistence/model.ts';
+import type { BrainEngine } from '../src/core/engine.ts';
-function engineWith(executeRaw: () => Promise<never>): { engine: SqlEngine; calls: () => number } {
+function engineWith(executeRaw: () => Promise<never>): { engine: BrainEngine; calls: () => number } {
let n = 0;
- return { engine: { executeRaw: async () => { n++; return executeRaw(); } } as unknown as SqlEngine, calls: () => n };
+ return { engine: { executeRaw: async () => { n++; return executeRaw(); } } as unknown as BrainEngine, calls: () => n };
}
describe('#8982 federation exposure read never changes the federation', () => {
diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txt
index a0e6e515ba..fc4c7c1a6f 100644
--- a/test/postgres-unit-arms.txt
+++ b/test/postgres-unit-arms.txt
@@ -105,6 +105,7 @@ test/facts-entity-moved-7602.test.ts
test/facts-insert-guards-7105.test.ts
test/facts-publication-large-fence-6898.test.ts
test/facts-subject-infer-7405.test.ts
+test/federation-exposure-private-9068.test.ts
test/fence-absent-append-keeps-rows.test.ts
test/fence-fact-append.test.ts
test/fence-first-generation-8819.test.ts
@@ -470,6 +471,7 @@ test/purge-global-marker.test.ts
test/purge-missing-cache-6911.test.ts
test/purge-name-stores-9068.test.ts
test/purge-read-surfaces.test.ts
+test/purge-scrub-followups-9068.test.ts
test/purge-sync-tombstone-hold-7130.test.ts
test/qm-memory-tools.test.ts
test/qm-scope-writes.test.ts
diff --git a/test/purge-scrub-followups-9068.test.ts b/test/purge-scrub-followups-9068.test.ts
new file mode 100644
index 0000000000..9fb3d3d2b7
--- /dev/null
+++ b/test/purge-scrub-followups-9068.test.ts
@@ -0,0 +1,230 @@
+/**
+ * #9068 follow-up: every delete that removes a page row also removes the rows that name it outside a foreign key, and
+ * an eval capture that lands after a purge does not write the purged slug back.
+ *
+ * 1. Protects: (a) a deferred eval capture (the query response went out before the insert ran) that lands after the
+ * page purge stores only slugs whose page still exists, and on Postgres one that runs while the purge is still open
+ * waits on the page row and drops the slug; (b) the connector delete (deleteConnectorPage -> deletePages) scrubs
+ * retrieval logs, ratings, weights, the volunteer log, the gazetteer and the eval candidates' captured slugs for
+ * that source in the delete's transaction;
+ * (c) removing a source (unmanaged removeSource -> deleteSourceRow, and the managed lifecycle remove) drops every
+ * such row of the source, and strips its slugs from eval candidates no other covered source still answers.
+ * 2. Fails when: logEvalCandidate inserts slugs without probing the page row under a key-share lock, a hard delete or a
+ * source removal leaves any of those rows behind, or a scrub reaches another source's rows or a slug another
+ * source the candidate covers still holds.
+ * 3. Existing coverage: purge-retrieval-feedback-8821 and purge-name-stores-9068 pin delete_page purge and the
+ * tombstone purge only; nothing pinned the capture, the connector delete, or source removal.
+ * 4. Real seams: PGLite, and Postgres when DATABASE_URL is set (test/postgres-unit-arms.txt); the feedback store's
+ * writers, captureEvalCandidate under withResponseDeferral, the delete_page op, deleteConnectorPage, removeSource,
+ * runManagedSourceLifecycle; repo-less brain, no provider calls.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { randomUUID } from 'node:crypto';
+import { mkdirSync, mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import type { BrainEngine } from '../src/core/engine.ts';
+import type { EvalCandidateInput } from '../src/core/types.ts';
+import type { OperationContext } from '../src/core/ops/contract.ts';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { operations } from '../src/core/operations.ts';
+import { importFromContent } from '../src/core/import-file.ts';
+import { applyRatings, insertRetrievalEvents, scrubPurgedSlugs } from '../src/core/feedback/store.ts';
+import { insertVolunteerEvents } from '../src/core/context/volunteer-events.ts';
+import { captureEvalCandidate } from '../src/core/eval-capture.ts';
+import { withResponseDeferral } from '../src/core/background-work.ts';
+import { deleteConnectorPage } from '../src/core/connectors/api-source.ts';
+import { removeSource } from '../src/core/sources-ops.ts';
+import { registerLocalWriter } from '../src/core/persistence/identity.ts';
+import { claimWorktree } from '../src/core/persistence/ownership.ts';
+import { runManagedSourceLifecycle } from '../src/core/persistence/source-lifecycle.ts';
+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
+import { withEnv } from './helpers/with-env.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const home = mkdtempSync(join(tmpdir(), 'gbrain-purge-followups-'));
+const engines: BrainEngine[] = [];
+let closePostgres: (() => Promise<void>) | undefined;
+const OTHER = 'companies/acme-example';
+
+const ctx = (engine: BrainEngine): OperationContext => ({ engine, sourceId: 'default', remote: false, dryRun: false,
+ config: { engine: engine.kind, embedding_disabled: true, storage: { backend: 'local', bucket: 'b', localPath: join(home, 'storage') } },
+ logger: { info() {}, warn() {}, error() {} } } as OperationContext);
+const op = (name: string) => operations.find(o => o.name === name)!;
+
+const candidate = (query: string, slugs: string[], sources: string[]): EvalCandidateInput => ({ tool_name: 'query', query, retrieved_slugs: slugs,
+ retrieved_chunk_ids: [], source_ids: sources, expand_enabled: null, detail: null, detail_resolved: null, vector_enabled: false,
+ expansion_applied: false, latency_ms: 1, remote: false, job_id: null, subagent_id: null });
+const capturedSlugs = async (engine: BrainEngine, query: string) =>
+ (await engine.executeRaw<{ retrieved_slugs: string[] }>('SELECT retrieved_slugs FROM eval_candidates WHERE query = $1', [query])).map(r => r.retrieved_slugs);
+
+async function page(engine: BrainEngine, sourceId: string, slug: string) {
+ await engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT DO NOTHING', [sourceId]);
+ await importFromContent(engine, slug, `---\ntitle: ${slug}\ntype: note\n---\n# ${slug}\n\nBody text ${randomUUID()}.\n`, { noEmbed: true, sourceId });
+}
+
+/** The page and OTHER in `sourceId`, and every store that names the page: a retrieval event (page and edge), page and edge ratings, a volunteer pointer, a gazetteer entry. */
+async function traced(engine: BrainEngine, sourceId: string, slug: string) {
+ await page(engine, sourceId, slug);
+ await page(engine, sourceId, OTHER);
+ const id = randomUUID(), edge = `${slug}|mentions|${OTHER}`;
+ await insertRetrievalEvents(engine, [{ id, client_id: 'local', op: 'query', links: [{ source_id: sourceId, edge_key: edge, to_slug: OTHER }],
+ pages: [{ source_id: sourceId, slug, content_hash: null, rank: 1, cited: true }] }]);
+ await applyRatings(engine, { eventId: id, clientId: 'local', signal: 'explicit', alpha: 0.2, targets: [
+ { kind: 'page', source_id: sourceId, key: slug, rating: 5, content_hash: null }, { kind: 'link', source_id: sourceId, key: edge, rating: 5, content_hash: null }] });
+ await insertVolunteerEvents(engine, [{ source_id: sourceId, slug, confidence: 0.9, match_arm: 'alias', rationale: `alias match "${slug}"`, channel: 'op' }]);
+ await engine.executeRaw('INSERT INTO mention_gazetteer_entries (source_id, name_norm, target_slug) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING', [sourceId, slug, slug]);
+}
+
+/** Rows per store that still name `slug` in `sourceId` (any row of the source when null). */
+async function residue(engine: BrainEngine, sourceId: string, slug: string | null) {
+ const n = async (sql: string) => (await engine.executeRaw(sql, [sourceId, slug])).length;
+ const named = (col: string) => `($2::text IS NULL OR strpos(${col}, $2::text) > 0)`;
+ return {
+ retrieval_event_pages: await n(`SELECT 1 FROM retrieval_event_pages WHERE source_id = $1 AND ${named('slug')}`),
+ retrieval_event_links: await n(`SELECT 1 FROM retrieval_event_links WHERE source_id = $1 AND ${named("edge_key || '|' || to_slug")}`),
+ retrieval_feedback: await n(`SELECT 1 FROM retrieval_feedback WHERE source_id = $1 AND ${named('element_key')}`),
+ retrieval_weights: await n(`SELECT 1 FROM retrieval_weights WHERE source_id = $1 AND ${named('element_key')}`),
+ context_volunteer_events: await n(`SELECT 1 FROM context_volunteer_events WHERE source_id = $1 AND ${named('slug')}`),
+ mention_gazetteer_entries: await n(`SELECT 1 FROM mention_gazetteer_entries WHERE source_id = $1 AND ${named('target_slug')}`),
+ };
+}
+const NONE = { retrieval_event_pages: 0, retrieval_event_links: 0, retrieval_feedback: 0, retrieval_weights: 0, context_volunteer_events: 0, mention_gazetteer_entries: 0 };
+
+beforeAll(async () => withEnv({ GBRAIN_HOME: home }, async () => {
+ const backends = testBackends();
+ if (backends.includes('pglite')) { const lite = new PGLiteEngine(); await lite.connect({}); await lite.initSchema(); engines.push(lite); }
+ if (backends.includes('postgres')) {
+ const pg = await isolatedPersistencePostgres(process.env.DATABASE_URL!);
+ engines.push(pg.engine); closePostgres = pg.close;
+ }
+ for (const engine of engines) await registerLocalWriter(engine, 'cli');
+}), 120_000);
+afterAll(async () => {
+ for (const engine of engines) await disposePersistenceConsumer(engine);
+ await engines.find(e => e.kind === 'pglite')?.disconnect(); await closePostgres?.(); rmSync(home, { recursive: true, force: true });
+}, 60_000);
+
+describe('#9068 an eval capture does not write a purged slug back', () => {
+ test('a deferred capture that lands after delete_page purge:true keeps only live slugs', () => withEnv({ GBRAIN_HOME: home }, async () => {
+ for (const engine of engines) {
+ const slug = 'people/alice-capture';
+ await page(engine, 'default', slug);
+ await page(engine, 'default', OTHER);
+ const query = `deferred capture ${randomUUID()}`;
+ let captured!: Promise<void>;
+ const deferred = await withResponseDeferral(async () => {
+ captured = captureEvalCandidate(engine, { tool_name: 'query', query, latency_ms: 1, remote: true, expand_enabled: null, detail: null, job_id: null, subagent_id: null,
+ results: [{ slug, source_id: 'default', chunk_id: 1 }, { slug: OTHER, source_id: 'default', chunk_id: 2 }] as never,
+ meta: { detail_resolved: null, vector_enabled: false, expansion_applied: false } as never }, { scrub_pii: false });
+ });
+ const snapshot = await engine.readPageSnapshot(slug, { sourceId: 'default' });
+ const result = await op('delete_page').handler(ctx(engine), { slug, purge: true, request_id: randomUUID(), expected_revision: snapshot!.revision }) as Record<string, any>;
+ expect(result.status).toBe('purged');
+ expect(deferred.release()).toBe(1);
+ await captured;
+ expect({ engine: engine.kind, slugs: await capturedSlugs(engine, query) }).toEqual({ engine: engine.kind, slugs: [[OTHER]] });
+ }
+ }), 120_000);
+
+ test('a capture keeps, in order, each slug a page of its sources still holds', async () => {
+ for (const engine of engines) {
+ await page(engine, 'default', 'notes/kept-a');
+ await page(engine, 'source-b', 'notes/kept-b');
+ const query = `kept ${randomUUID()}`;
+ await engine.logEvalCandidate(candidate(query, ['notes/kept-b', 'notes/never-existed', 'notes/kept-a'], ['default', 'source-b']));
+ expect({ engine: engine.kind, slugs: await capturedSlugs(engine, query) }).toEqual({ engine: engine.kind, slugs: [['notes/kept-b', 'notes/kept-a']] });
+ }
+ }, 120_000);
+
+ const postgresTest = testBackends().includes('postgres') ? test : test.skip;
+ postgresTest('on Postgres a capture during an open purge waits for it and drops the slug', async () => {
+ const engine = engines.find(e => e.kind === 'postgres')!;
+ const slug = 'people/alice-race';
+ await page(engine, 'default', slug);
+ await page(engine, 'default', OTHER);
+ const query = `race ${randomUUID()}`;
+ let late: Promise<number> | null = null;
+ await engine.transaction(async tx => {
+ await tx.executeRaw(`DELETE FROM pages WHERE source_id = 'default' AND slug = $1`, [slug]);
+ await scrubPurgedSlugs(tx, 'default', [slug]);
+ late = engine.logEvalCandidate(candidate(query, [slug, OTHER], ['default']));
+ await new Promise(r => setTimeout(r, 500));
+ });
+ await late;
+ expect(await capturedSlugs(engine, query)).toEqual([[OTHER]]);
+ }, 120_000);
+});
+
+describe('#9068 the batch hard delete scrubs what names the page', () => {
+ test('the connector delete (deletePages) scrubs the source\'s rows and leaves other sources and pages', () => withEnv({ GBRAIN_HOME: home }, async () => {
+ for (const engine of engines) {
+ const slug = 'notes/connector-item';
+ for (const sourceId of ['default', 'source-b']) await traced(engine, sourceId, slug);
+ const solo = `connector solo ${randomUUID()}`, shared = `connector shared ${randomUUID()}`;
+ await engine.logEvalCandidate(candidate(solo, [slug, OTHER], ['default']));
+ await engine.logEvalCandidate(candidate(shared, [slug, OTHER], ['default', 'source-b']));
+ const summary = { added: 0, modified: 0, deleted: 0, chunksCreated: 0, embedded: 0, pagesAffected: [], factsRetired: 0 };
+ await deleteConnectorPage({ engine, sourceId: 'default', managed: null, summary, label: 'Meetings', tag: 'meetings', record: 'database', dir: null },
+ `${slug}.md`, 'delete', 'deleted upstream');
+ expect(summary.deleted).toBe(1);
+ expect({ engine: engine.kind, residue: await residue(engine, 'default', slug) }).toEqual({ engine: engine.kind, residue: NONE });
+ expect(await residue(engine, 'source-b', slug)).toEqual({ retrieval_event_pages: 1, retrieval_event_links: 1, retrieval_feedback: 2, retrieval_weights: 2,
+ context_volunteer_events: 1, mention_gazetteer_entries: 1 });
+ expect(await capturedSlugs(engine, solo)).toEqual([[OTHER]]);
+ expect(await capturedSlugs(engine, shared)).toEqual([[slug, OTHER]]);
+ expect(await engine.executeRaw(`SELECT 1 FROM pages WHERE source_id = 'default' AND slug = $1`, [OTHER])).toHaveLength(1);
+ }
+ }), 120_000);
+
+});
+
+describe('#9068 removing a source scrubs every row that names its pages', () => {
+ test('unmanaged removeSource drops the source\'s rows and strips its slugs from eval candidates', () => withEnv({ GBRAIN_HOME: home }, async () => {
+ for (const engine of engines) {
+ const gone = 'source-gone', slug = 'notes/gone-page';
+ await traced(engine, gone, slug);
+ await traced(engine, 'default', 'notes/default-page');
+ await engine.executeRaw(`INSERT INTO context_volunteer_events (source_id, slug, confidence, match_arm, rationale, channel) VALUES ($1, 'notes/orphan', 0.5, 'alias', 'alias match "orphan"', 'op')`, [gone]);
+ const solo = `source solo ${randomUUID()}`, shared = `source shared ${randomUUID()}`;
+ await engine.logEvalCandidate(candidate(solo, [slug, OTHER], [gone]));
+ await engine.logEvalCandidate(candidate(shared, [slug, OTHER], [gone, 'default']));
+ const before = await residue(engine, 'default', null);
+ const result = await removeSource(engine, { id: gone, yes: true });
+ expect(result.pages_deleted).toBe(2);
+ expect({ engine: engine.kind, residue: await residue(engine, gone, null) }).toEqual({ engine: engine.kind, residue: NONE });
+ expect(await residue(engine, 'default', null)).toEqual(before);
+ expect(await capturedSlugs(engine, solo)).toEqual([[]]);
+ expect(await capturedSlugs(engine, shared)).toEqual([[OTHER]]);
+ }
+ }), 120_000);
+
+ test('the managed lifecycle remove does the same', async () => {
+ for (const engine of engines) {
+ const lifecycleHome = mkdtempSync(join(tmpdir(), 'gbrain-purge-followups-managed-'));
+ try {
+ await withEnv({ GBRAIN_HOME: lifecycleHome, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined }, async () => {
+ const managed = 'source-managed', slug = 'notes/managed-page', root = join(lifecycleHome, 'canonical');
+ mkdirSync(root);
+ await registerLocalWriter(engine, 'cli');
+ await engine.executeRaw('INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2)', [managed, root]);
+ await traced(engine, managed, slug);
+ const query = `managed ${randomUUID()}`;
+ await engine.logEvalCandidate(candidate(query, [slug, OTHER], [managed]));
+ await claimWorktree(engine, managed, root);
+ await engine.executeRaw('UPDATE persistence_brain SET enabled = true WHERE singleton = 1');
+ try {
+ const result = await runManagedSourceLifecycle(engine, { operation: 'remove', sourceId: managed, confirmDestructive: true, requestId: randomUUID() });
+ expect(result).toMatchObject({ state: 'committed' });
+ } finally {
+ await engine.executeRaw('UPDATE persistence_brain SET enabled = false WHERE singleton = 1');
+ await disposePersistenceConsumer(engine);
+ }
+ expect({ engine: engine.kind, residue: await residue(engine, managed, null) }).toEqual({ engine: engine.kind, residue: NONE });
+ expect(await capturedSlugs(engine, query)).toEqual([[]]);
+ });
+ } finally { rmSync(lifecycleHome, { recursive: true, force: true }); }
+ }
+ }, 180_000);
+});