Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Let local reads on an upgraded brain use the concept-input function

garrytan039c1b83fe7d
diff --git a/docs/architecture/key-files/core-search-1.md b/docs/architecture/key-files/core-search-1.md
index 6f25ca7ca7..1eb3f8e74b 100644
--- a/docs/architecture/key-files/core-search-1.md
+++ b/docs/architecture/key-files/core-search-1.md
@@ -65,7 +65,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/search/modes-report.ts` — `buildModesReport(engine)` → `SearchModesReport` (`schema_version: 2`), the read-only dashboard behind `gbrain search modes` and the `search_modes` MCP op: `active_mode` + validity, the per-knob `resolved` attribution (`attributeKnob` over `KNOB_DESCRIPTIONS`, including the five `reranker_*` knobs), the three frozen `bundles`, `config_keys`, and `reranker_readiness` — `{model, enabled, ready, required_key, key_present, self_hosted, fix}` from `rerankerReadinessForEngine` + `describeRerankerFix` (a thrown readiness check still yields a `ready: false` verdict whose `fix` says to run `gbrain doctor`; the block never vanishes silently). `redactReadinessForRemote(report)` is what the `search_modes` op returns when `ctx.remote !== false`: `required_key`, `key_present` and `fix` are dropped and `self_hosted` is forced false — which env vars exist on the host is fingerprinting data and the fix names them; `ready` stays because it is observable anyway (reranked results carry `rerank_score`). `src/commands/search.ts:formatModesText` renders the runtime `Reranker:` line (`off (resolved) — …` / `<model> (enabled) — <KEY> present` / `<model> (enabled but NOT running) — <fix>`) and a per-bundle `reranker=… topNIn=… autocut=…` line. Pinned by `test/modes-report-reranker.test.ts`. The CLI dashboard leads with command-specific expansion: `query` follows the mode's expansion default (off in all modes) unless the call passes `expand`, while `search` is off. It discloses provider prerequisites/text flow and distinguishes retained bundle configuration from effective command behavior.
-- `src/core/search/private-visibility.ts` — shared page-visibility resolver and SQL predicates. Explicit local trust disables page filtering; operator env/config opt-outs keep their semantics; failed config reads enforce privacy. `privatePagesFilterFragment` authorizes concrete page rows and `isPrivatePage` is its row-side twin: explicit `private` hides a page, absent visibility is world for ordinary pages and private for extracted atoms (`type = 'atom'`) and synthesized concepts (`type = 'concept'` with `synthesized_by`), because their origin may be a private page or transcript. The page predicate also follows derived origins: an atom whose `source_slug` page is explicitly private, and a synthesized concept with a private input atom (or an input whose origin is private, including an origin a rename moved after a later rename onto the old slug dropped its alias: `slug_rename_history`, GST-2687) over `concept-provenance` edges, are private whatever their own field says, so a later private flip of an origin reaches derived pages before any repair. The concept-input test is the SQL function `gbrain_concept_inputs_private(page_id)` (migration v311, never inlined, so a read plans one call per page instead of the subplans); inside a remote request on a brain whose schema predates v311 (`RemotePolicy.schemaVersion`) the same test runs inline, so reads keep working before the migration applies. Chronicle events follow `event.depth` (fail closed). Declared lineage matches `derived_from` slugs with `= ANY(ARRAY(...))`, not `IN (SELECT ... jsonb_array_elements_text(...))`: same rows, but the planner priced the IN form as a 100-row semi-join per page and pushed every read carrying the rule past `jit_above_cost` (`test/e2e/private-visibility-plan-postgres.test.ts`, which measures with `links` and `slug_rename_history` both unanalyzed and analyzed; test `test/concept-rename-history-gst2687.test.ts`, both engines). `privateSnapshotFilterFragment` is the same rule on a row's own frontmatter for `page_versions`. `effectiveVisibility(origin)` (transcript or missing origin private, page origin per the read rule) and `strictestVisibility` are used by both `extract_atoms` paths, concept synthesis and `gbrain repair visibility`. Remote publication rechecks (`persistence/page-visibility.ts`) and the alias-inject recheck in `hybrid.ts` use the same rule. Tests: `test/private-visibility.test.ts`, `test/derived-page-visibility.test.ts` (PGLite and Postgres); link origins and timeline projections use independent ID-based predicates. `privateProvenanceFilterFragment` hides a fact whose provenance page (`source_markdown_slug`) is private; the fact-list reads in `engine-sql/facts.ts` (by entity, since, keyset, session, supersessions, trajectory) apply it behind `FactListOpts.excludePrivate` / `TrajectoryOpts.excludePrivate`, which every remote fact surface resolves with `resolveExcludePrivatePages` (`test/remote-fact-list-private-provenance.test.ts`, both engines). Remote fact writes apply the same predicate behind an `excludePrivate` option: `forget` (`guardRemoteForget`, the row lock in `persistence/memory-mutations.ts`, `recordFactWithdrawal`, `similarActiveAfterForget`) and `remember` (`decideSingleFact`'s exact and cosine arms, `decideReplacement`), so a fact remote reads hide is `not_found` to a remote write and never matched as its duplicate (`test/remote-fact-write-private-provenance.test.ts`, both engines). Legacy slug-set helpers only enumerate names; they never authorize another row's data. `ops/context.ts:readPolicyOpts` combines canonical source precedence, resolved privacy and holder permissions. Both engines filter content/history/graph/identity/analytics before limits or aggregation; current and historical visibility are checked independently while page privacy is enforced. Semantic response caching remains disabled, so visibility-key folding alone never authorizes reuse.
+- `src/core/search/private-visibility.ts` — shared page-visibility resolver and SQL predicates. Explicit local trust disables page filtering; operator env/config opt-outs keep their semantics; failed config reads enforce privacy. `privatePagesFilterFragment` authorizes concrete page rows and `isPrivatePage` is its row-side twin: explicit `private` hides a page, absent visibility is world for ordinary pages and private for extracted atoms (`type = 'atom'`) and synthesized concepts (`type = 'concept'` with `synthesized_by`), because their origin may be a private page or transcript. The page predicate also follows derived origins: an atom whose `source_slug` page is explicitly private, and a synthesized concept with a private input atom (or an input whose origin is private, including an origin a rename moved after a later rename onto the old slug dropped its alias: `slug_rename_history`, GST-2687) over `concept-provenance` edges, are private whatever their own field says, so a later private flip of an origin reaches derived pages before any repair. The concept-input test is the SQL function `gbrain_concept_inputs_private(page_id)` (migration v311, never inlined, so a read plans one call per page instead of the subplans); a read uses it when its schema version is v311 or later: the one a remote, scope or attested record carries (`RemotePolicy.schemaVersion`), or for a local read the one its scoped engine recorded at connect or migration (`schemaVersionSeen`, `migrate-pending.ts`); otherwise the same test runs inline, so reads keep working before the migration applies. Chronicle events follow `event.depth` (fail closed). Declared lineage matches `derived_from` slugs with `= ANY(ARRAY(...))`, not `IN (SELECT ... jsonb_array_elements_text(...))`: same rows, but the planner priced the IN form as a 100-row semi-join per page and pushed every read carrying the rule past `jit_above_cost` (`test/e2e/private-visibility-plan-postgres.test.ts`, which measures with `links` and `slug_rename_history` both unanalyzed and analyzed; test `test/concept-rename-history-gst2687.test.ts`, both engines). `privateSnapshotFilterFragment` is the same rule on a row's own frontmatter for `page_versions`. `effectiveVisibility(origin)` (transcript or missing origin private, page origin per the read rule) and `strictestVisibility` are used by both `extract_atoms` paths, concept synthesis and `gbrain repair visibility`. Remote publication rechecks (`persistence/page-visibility.ts`) and the alias-inject recheck in `hybrid.ts` use the same rule. Tests: `test/private-visibility.test.ts`, `test/derived-page-visibility.test.ts` (PGLite and Postgres); link origins and timeline projections use independent ID-based predicates. `privateProvenanceFilterFragment` hides a fact whose provenance page (`source_markdown_slug`) is private; the fact-list reads in `engine-sql/facts.ts` (by entity, since, keyset, session, supersessions, trajectory) apply it behind `FactListOpts.excludePrivate` / `TrajectoryOpts.excludePrivate`, which every remote fact surface resolves with `resolveExcludePrivatePages` (`test/remote-fact-list-private-provenance.test.ts`, both engines). Remote fact writes apply the same predicate behind an `excludePrivate` option: `forget` (`guardRemoteForget`, the row lock in `persistence/memory-mutations.ts`, `recordFactWithdrawal`, `similarActiveAfterForget`) and `remember` (`decideSingleFact`'s exact and cosine arms, `decideReplacement`), so a fact remote reads hide is `not_found` to a remote write and never matched as its duplicate (`test/remote-fact-write-private-provenance.test.ts`, both engines). Legacy slug-set helpers only enumerate names; they never authorize another row's data. `ops/context.ts:readPolicyOpts` combines canonical source precedence, resolved privacy and holder permissions. Both engines filter content/history/graph/identity/analytics before limits or aggregation; current and historical visibility are checked independently while page privacy is enforced. Semantic response caching remains disabled, so visibility-key folding alone never authorizes reuse.
- `src/core/search/projection-readiness.ts` — one JIT-off statement probes visible unsealed pages: ready, projection_pending, by_design (only protected/archived unsealed) or unknown; LIMIT 1 arms use the v161 pending index. Cached per scope by DB generation (page clock, pages filenode, archive flags, no open tx); `GBRAIN_PROJECTION_READINESS_CACHE=0` disables.
diff --git a/scripts/module-size-limits.tsv b/scripts/module-size-limits.tsv
index 077728fcc7..520546ed02 100644
--- a/scripts/module-size-limits.tsv
+++ b/scripts/module-size-limits.tsv
@@ -18,7 +18,7 @@ src/core/cycle/synthesize.ts 3244 ratchet +3 #7805: each captured session synthe
src/core/cycle/extract-atoms.ts 1635 ratchet crossed the 1500 unlisted cap in the atoms/extraction fix wave: +21 embed-aware cost gate (#4878), +65 atoms-array anchor scan (#4913); peel candidate: the parse block (parseAtomsOutcome / parseArrayAtOffset / atomsFromParsedArray) into extract-atoms-parse.ts; +35: #4853 transcript-origin atoms link to their conversation page; +130: #4916 transcript failure-count/tombstone (extract_atoms_transcript_state, TODOS P2(b)); wave composition: #4916 transcript-state skip + #4948 zero-atom prompt + #4943 meetings source-id summed; +3: wave review — a later `[]` is an honest zero-yield, a non-atom-shaped array is a counted failure; +13: wave review — explicit budget keeps the cap over an unpriced embed route ($0-priced); +14: wave review — transcript-state write errors surfaced, missing table warns once; merged reliability/shared-skills source integration; +37 fix wave 3: stable undated atom dates + stale-atom retirement (C-14); +3 #5525: effectiveVisibility stamped on non-managed atoms; fix wave 3 x fix wave 2 (v0.60.5.0) composition: +3 at the merge; +6 fix wave 3 lane A (#5601): accepted-pending atom batches count as progress (helper in persistence/accepted-pending.ts); +4 fix wave 4: #5705 transcript-as-data system prompt lines and its message helper, #5627 response schema import (the atom-type list moved to extract-atoms-schema.ts); +30 #5809/#5832: drain hard/soft stop signals (ExtractAtomsOpts.signal/stopSignal, pre-commit abort checks, no strike or post-stop writes after a hard stop) and the cancellable backlog count; +1 #5812: transcript items are paste-stripped (corpusTextForExtraction import) before the input cut; +1 #5820: extract_atoms discovery passes selfCaptureSessionIds like synthesize; +10 #5856/#5854: drain attempt state (one BudgetTracker + one publish wait across batches) and the opt-in connector email/meeting exclusion (predicate in connector-atoms.ts); +1 Foundations 2 Lane C: maintenanceTransaction import (attributed stale-atom retirement); +5 fix wave 9 lane P (#5041): opt-in atom write-through hook and retired-atom slugs returned for file removal (logic in derived-write-through.ts); the parse block (parseAtomsOutcome / parseArrayAtOffset / atomsFromParsedArray) is peeled into extract-atoms-parse.ts (-193, fix wave 10 lane B); +35: #4853 transcript-origin atoms link to their conversation page; +130: #4916 transcript failure-count/tombstone (extract_atoms_transcript_state, TODOS P2(b)); wave composition: #4916 transcript-state skip + #4948 zero-atom prompt + #4943 meetings source-id summed; +3: wave review — a later `[]` is an honest zero-yield, a non-atom-shaped array is a counted failure; +13: wave review — explicit budget keeps the cap over an unpriced embed route ($0-priced); +14: wave review — transcript-state write errors surfaced, missing table warns once; merged reliability/shared-skills source integration; +37 fix wave 3: stable undated atom dates + stale-atom retirement (C-14); +3 #5525: effectiveVisibility stamped on non-managed atoms; fix wave 3 x fix wave 2 (v0.60.5.0) composition: +3 at the merge; +6 fix wave 3 lane A (#5601): accepted-pending atom batches count as progress (helper in persistence/accepted-pending.ts); +4 fix wave 4: #5705 transcript-as-data system prompt lines and its message helper, #5627 response schema import (the atom-type list moved to extract-atoms-schema.ts); +30 #5809/#5832: drain hard/soft stop signals (ExtractAtomsOpts.signal/stopSignal, pre-commit abort checks, no strike or post-stop writes after a hard stop) and the cancellable backlog count; +1 #5812: transcript items are paste-stripped (corpusTextForExtraction import) before the input cut; +1 #5820: extract_atoms discovery passes selfCaptureSessionIds like synthesize; +10 #5856/#5854: drain attempt state (one BudgetTracker + one publish wait across batches) and the opt-in connector email/meeting exclusion (predicate in connector-atoms.ts); +1 Foundations 2 Lane C: maintenanceTransaction import (attributed stale-atom retirement); +5 fix wave 9 lane P (#5041): opt-in atom write-through hook and retired-atom slugs returned for file removal (logic in derived-write-through.ts); +17 #5031: same-page case-insensitive atom adoption fallback (one page-atom scan per source page per run); +12 P2 atomsPrompt helper: extraction.date_grounding source observation date + rule in the atoms prompt; +4 GBRA-75: octet_length-bounded body-length predicate (minCompiledTruthChars) for discovery and the backlog count; GBRA-130 A-42 (#6804): dot-led atom stems and the legacy-slug adoption guard; re-measured at the Train A merge of #8948: 1633; re-measured at the Train C merge onto Train B (#9246 onto #9223): 1633 -> 1635, both sides' growth
src/core/engine.ts 2890 ratchet +5 #7191: findPages/countPagesBy signatures and type imports (SQL in engine-sql/page-lookup.ts); grown v0.46.26.0 megawave: relational/visibility/usage contract surface (#4352 #4218); grown wave-g (#4524 #4527): orphans-mode contract + removeLink deleted count; wave-k: +4 fail-closed TrajectoryOpts.remote contract doc (#3855), +2 FactListOpts.unconsolidatedOnly (#4070), +14 getAllConfig bulk-read contract (#3980); grown v0.46.26.0 megawave: relational/visibility/usage contract surface (#4352 #4218); grown wave-g (#4524 #4527): orphans-mode contract + removeLink deleted count; wave-k: +4 fail-closed TrajectoryOpts.remote contract doc (#3855), +2 FactListOpts.unconsolidatedOnly (#4070), +14 getAllConfig bulk-read contract (#3980); +1: getBacklinkCounts page-id contract doc (#4380); +9: FactListOpts.grep SQL pre-limit contract (#3851); +13: dream-verdict TTL contract (#4069); +33: softDeletePages batch soft-delete contract (#4587); +8: #4592 scoped getStats/getHealth contract doc; +5 read-time TTL validity contract docs (active vs history read semantics); +12: #4280 findOrphanPages type/quarantined metadata contract; +14: traversePathsDetailed row-cap contract (ship-review); +12: resolveSlugWithAliasDetailed owning-source contract (ship-review) | merged v0.48.1.0: master #4788 + wave traversePathsDetailed/resolveSlugWithAliasDetailed; master: | merged v0.48.1.0 true-up; corrective release: read-policy signatures and Chronicle scope; +7: #4882 listFactsSince entitySlug/sessionId composition doc; #5105: canonical page state shared module and transaction/snapshot delegates; #5105: guarded derived projection preparation and read seals; #5105: resident-stop registration and direct transaction contract; #5105: +1 raw preservation contract for legacy engine migration; merged company ingestion/schema routing changes; safety wave: fact embedding model and text identity contract; +5 write-path wave: findDuplicatePage excludeSlug + id-first ranking contract; +2 fw3 A14: addTag tagSource provenance contract; evidence delivery: getChunkWindows interface member + type import (+3); #5761: +4 markPagesAttendanceBlocked member + countStalePagesForExtraction attendance option; +4 #5888: FactListOpts.fingerprint + FactRow.fact_fingerprint contract (hot-memory duplicate collapse); +1 GBRA-49: LinkReadScope temporal opts on the link read contract; +4 #5984: upsertChunks sealChunkerVersion and createVersion preimage contract docs (round-trip diet); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2683 measured); +10 P7 relationalChainHop member + ChainHopOpts/ChainHopEdge imports (SQL in search/read-enrichment.ts); +14 contributor audit wave P0: listFactsKeyset member (delta facts keyset read contract) + FactRow.created_at_iso; +1 #5375: get_versions limit/include_body type import (GetVersionsOpts, PageVersionRows); 3 P2 speaker attribution: FactAttribution type + FactRow/NewFact.attributed_to fields + findCandidateDuplicates attributedTo option contract; +4 #5984 lane apply diet: setPageAliases inline option and upsertChunks pageId contract docs; +10 #6278: executeRaw timeoutMs contract (transaction-local statement_timeout for bounded preparation reads); +2 GBRA-69: replaceDerivedLinksBatch member; master v0.60.136.0 merge into #5575 memory trust (both sides kept); +6 GBRA-67: readPageSnapshotsBatch interface member (BrainEngine signatures cannot move); +1 GBRA-75 wave 10: optional findDuplicatePages member (batched findDuplicatePage, contract in engine-sql/pages.ts); +4 FactListOpts/TrajectoryOpts.excludePrivate fact-provenance visibility contract (interface fields cannot move); +18 GBRA-76: TransactionOptions (abort signal) interface and the transaction/transactionDirect signatures (abort logic in postgres-engine/transaction-abort.ts); +2 W14 P3.11 (#4381): LinkBatchInput.resolution_type on top of W13; +1 C2: findCandidateDuplicates arm:'keyword' contract doc (facts.candidate_fusion); re-summed at the master 6ccf30c3e merge (2708 integration + 2707 master - 2707 base); +5 Cat 40 Hard fix wave F2 (#6271): countKeywordPages/searchKeywordPages BrainEngine signatures (SQL in engine-sql/keyword-pages.ts); +6 W15 P1.2 (#6454): countStaleFrontmatterKeyPages/listStaleFrontmatterKeyPages/stripStaleFrontmatterKeys contract (SQL in engine-sql/pages.ts); +2 large pages Wave A sub-a: getChunkPage signature (get_chunks paging; SQL in engine-sql/chunks.ts); +5 #6696 B3: readEmbeddingStateCounts/listEmbeddingStateRows signatures + type import (SQL in engine-sql/chunks.ts); +2 Cat 40 next wave part 1: non-ranking link types on the relationalFanout/relationalChainHop/getBacklinkCounts/getAdjacencyBoosts contract docs (policy in link-ranking-policy.ts; re-summed at the 0.60.242.0 merge: master 2794 + 2); +26 Wave B PR 1: segmented page storage members (allocateBodyOccurrenceIds..sweepSegments) + page-body type import (SQL in engine-sql/page-body.ts); merge v0.60.248.0: Wave B PR 1 and master growth compose at 2822; +1 LHF wave 2 item 1: executeRaw opts gain jitOff/prepare (the JIT-off typed-search type check; logic in ops/search-type-filter.ts); +9 PR2 fix lane X: BrainEngine.keywordBoundedRead, one unscoped read under the keyword statement bound for the segmented-region read (interface member and its doc cannot move): 2823 + 9; +9 GBRA-177: TakesListOpts.oldestFirst/sinceOnOrBefore/ungradedFirst for the grade_takes window (interface fields cannot move; SQL in engine-sql/takes.ts): 2832 + 9; +6 GBRA-156: TransactionOptions.pipelineFirst (the declaration goes out with BEGIN behind a transaction-mode pooler; predicate in persistence/protocol.ts); interface fields cannot move; +1 GBRA-161 PR 2: deprecation notes on insertFacts({ deleteForPageFirst }), deleteFactsForPage and expireFact (interface docs cannot move); +7 GBRA-156: TransactionOptions.pipelineEarly (search read bodies go out behind BEGIN; predicate in postgres-engine/search-settings.ts); interface fields cannot move; +2 GBRA-156: TransactionOptions.readOnly (search read transactions open BEGIN READ ONLY; 2858 measured); grown v0.46.26.0 megawave: relational/visibility/usage contract surface (#4352 #4218); re-summed at the Wave B PR 4 merge with v0.60.452.0 (2866 measured); re-summed at the memory train #8481 merge with v0.60.576.0 (2874 measured); +1 GST-731: TrajectoryPoint.valid_until (interface fields cannot move); +9 #9068: deletePage/deletePages scrub contract docs (interface docs cannot move); +2 GBRA-467: optional holdSchemaMigrationLock member (interface members cannot move; logic in schema-migration-lock.ts); re-measured at the Train A merge of #8948: 2889; +1 #9295: wantedProducers option on the stale-page list signature (interface fields cannot move; SQL in engine-sql/pages.ts)
src/core/import-file.ts 2091 ratchet +2 #7880: importFromContent's holdNoncanonicalPrivate option (interface field; the hold is screenImportContent's); +2 #7848: importFromFile holds a non-canonical private request (predicate and hold in persistence/visibility-request.ts); +2 W1 belief ledger: the legacy import is an unmanaged projection point (logic in belief/projection-point.ts); +2: pre-publication withdrawal validation; grown v0.46.26.0 megawave: OCR budget + image import + embed-retry rewire (#3973 #3374); grown wave-g (#4530): per-model chunk-token cap wiring; +24 #4548: row-level visibility-aware fence merge on remote write-back (replaces the #2044 whole-block swap + #4553/#4555 warn-only paths); +7 #4546: merge extended to the timeline-embedded fence; +5 #4107: maybeOcr gates on the OCR model's provider; +3 #3893; +7: #4702 disabled_patterns pass-through to the sanity gate; composition true-up (fix-wave multi-train); +8 #4903: importCodeFile writes source_path so the full-sync reconcile can see code pages; -11: wave review — local fence append replaced by facts-fence.replaceOrInsertFactsFence (#4756 rule); security: durable queued authority and confined filesystem dispatch; master integration: grown v0.46.26.0 megawave: OCR budget + image import + embed-retry rewire (#3973 #3374); grown wave-g (#4530): per-model chunk-token cap wiring; +24 #4548: row-level visibility-aware fence merge on remote write-back (replaces the #2044 whole-block swap + #4553/#4555 warn-only paths); +7 #4546: merge extended to the timeline-embedded fence; +5 #4107: maybeOcr gates on the OCR model's provider; +3 #3893; +7: #4702 disabled_patterns pass-through to the sanity gate; composition true-up (fix-wave multi-train); +8 #4903: importCodeFile writes source_path so the full-sync reconcile can see code pages; -11: wave review — local fence append replaced by facts-fence.replaceOrInsertFactsFence (#4756 rule); harness support: boundary wiring to shared grants, isolated setup, withdrawal and delegated guards; merged v0.49: withdrawal preservation after data-only frontmatter parsing; #5105: small legacy writer/refusal and shared revision CAS delegates; #5105: +8 prepared contextual retrieval policy and retry boundary wiring; upstream integration: preserve independently reviewed master additions with branch facade extraction; #5252 wave: shared provider-free Markdown and code preparation; merged company ingestion/schema routing changes; master retrieval/projection integration with company ingestion; shared content: prepare managed image imports without bypassing canonical publication; +29 write-path wave: identity move/collision wiring (logic in import-identity.ts), deferred embedding on provider failure, image index completeness, file-time date anchor; +22 fw3 A13: markdown re-import reuses unchanged chunk vectors under the seal/wrapper/signature/protected-fence gate; +2 fw3 A12: importFromFile firstCommitAt option; +3 fw3 A14: managed imports keep add-only tags; +4 #5675: record the file origin move inference binds to (logic in import-identity.ts); fw3 integration: A12/A13/A14 (+27) and #5675 (+4) compose; +17 wave-2 W4: safe-chunk re-seal of unchanged legacy markdown/code pages (#5050 #5247) and embedding-input provenance on inline imports (#5553); -1 fix wave 2 (v0.60.5.0) integration: W4 delta re-summed over the merged master line count; fix wave 3 x fix wave 2 (v0.60.5.0) composition: +16 at the merge; +11 #4616: markdown and code imports keep the usable vectors of a partially refused batch and leave refused chunks for embed --stale; lowered to 2230: #5621 contextual-mode resolution peeled to core/import-contextual-mode.ts (shared with gbrain repair contextual-mode); +2 wave 8 lane D1 #5158: key-order fallback equality call (comparison lives in page-state/import-guard.ts); +1 F1c: direct imports run in maintenanceTransaction (write attribution; helper in persistence/attribution.ts); lowered to 2178: #5988 content screen and content-sanity config peeled to core/import-screen.ts; +1 #6007: the prepared apply returns its page id and seal (PreparedImportApplied); +1 #5333: trusted imports drop a stale oversized embed_skip/content_flag the gate no longer sets, and an embed_skip transition defeats both unchanged fast paths; wave 10 collector: lane E (#5333) and the other lanes compose at 2183; grown #6188 (+17): fence options interface, result fields and the Tier 1 step call (logic in fence-repair/import-step.ts); +2 #5984 single-write fast path: the coordinated existingSnapshot option (interface field) composes with the lane apply diet at 2202; +1 GBRA-69: a coordinated apply defers its chunker seal to the caller (sealImportedPage in page-state/projections.ts); lowered to 2023: #5001 the edge-mapping loop moved to code-chunks.ts mapCodeEdges (shared with the prepared apply); W14 P3.13 (#5001) lowered 2049→2023 by moving the edge loop to mapCodeEdges; +2 W13 PR1 on top (2025 measured); +2 #6529: the unchanged-image skip path records source_path (refreshSourcePath, as #4588 does for Markdown); +4 large pages A6/T33: code files keep CODE_FILE_MAX_BYTES, a page over PAGE_INLINE_MAX_BYTES commits with NULL vectors for the metered stale sweep; +1 #6722: an embed_skip page is chunked and never embedded (noEmbed folds the marker); merge v0.60.240.0: large pages and master growth compose at 2032; +24 Wave B lane 1d: the segmented storage step (stage before publication, flip in the apply, seal and post-commit projection; logic in persistence/segmented-apply.ts and page-body/ingest.ts) and the streamed-import dispatch over the single-string ceiling (logic in page-body/streamed-import.ts); merge v0.60.248.0: Wave B PR 1 and master growth compose at 2056; +4 Wave B review: a rename re-preparation reuses the staged segmented body (reuseSegments option) and the remote fence merge reads an unread body as fence-free (fenceBody in page-body/page-shape.ts); +2 Wave B gem A: a coordinated demotion versions the segmented page before demoting it (createVersion order); lanes compose at 2062; +2 Wave B lane 2q: ImportResult.searchable (interface field; the value comes from SegmentedImport.afterCommit);+2: pre-publication withdrawal validation; +9 #7172: kept chunk rows move to their new index and a bounded managed publication defers its chunk replacement (logic in page-state/chunk-moves.ts and page-state/projections.ts continueDeferredProjection); merge v0.60.332.0: #7172 and master growth compose at 2075; +7 GBRA-283 B3a (#8523): the three file import entry points re-enter an embedded engine's scope (needsEngineScope in core/engine-scope.ts), importFromContent runs its screened body in that scope as a closure, and the deferred embed resumes the import's scope; re-summed at the memory train #8481 merge with v0.60.576.0 (2088 measured); +3 GST-2112: importFromContent's expectedBaseRevision option and base check (a webhook capture binds its write to its readers check)
-src/core/migrate.ts 616 ratchet schema-migration runner only; migrations live in src/core/schema-migrations/ (W3); +3 #5216: resumable page revision backfill runs after every migration pass while pages.knowledge_revision is nullable; re-summed at the Train A merge onto master v0.60.708.0: master 597 + #8930 hardenFunctionSearchPaths wiring 19 = 616
+src/core/migrate.ts 618 ratchet schema-migration runner only; migrations live in src/core/schema-migrations/ (W3); +3 #5216: resumable page revision backfill runs after every migration pass while pages.knowledge_revision is nullable; re-summed at the Train A merge onto master v0.60.708.0: master 597 + #8930 hardenFunctionSearchPaths wiring 19 = 616; +2 GST-2687: each version read and write records it on the engine (noteSchemaVersion) = 618
src/core/operations.ts 249 ratchet +1 #7168: graphRankOperations import (ops/graph-rank.ts; spread shares the orphans line); peel target: containment sprint C4-C7; grown v0.46.26.0 megawave op wiring; +chat-connectors wiring (master merge); +5 v0.47 gmail-loops: loopsOperations spread + OP_AREAS; #5105: receipt operation registry wiring; shared-skills integration on company-ingestion base; +2 security wave: withOutputRedaction applied at op registration; +1 #5616: pageEditOperations import (spread shares the pages line); +1 Foundations 1 F1b: attributionOperations import (spread shares the admin line); +1 agent operator wave A6: ops/notices.ts import (mute_notice); +2 agent operator closeout A1: registerOpRoutes import + call (routing logic in fix-routing.ts); +2 retrieval feedback: feedbackOperations import and spread (ops/feedback.ts); +1 #6007: pageBatchOperations import (spread shares the pages line); +1 P5: wanted_pages area row; lowered GBRA-75 W11 cold start: OP_AREAS + finalize step moved to operation-load.ts (shared with the CLI per-op loader); +1 C4 pinned questions: questionsOperations import and spread (ops/questions.ts; its areas live in operation-load.ts OP_AREAS); +1 Wave B append: pageAppendOperations import (spread shares the pages line); +1 Wave B upload sessions: pageUploadOperations import (spread shares the pages line); +1 #7191: search-exact import; +1 GBRA-156: whois import (ops/whois.ts; spread shares the chunks line); +1 GST-359 D: qmMemoryOperations import (ops/qm-memory.ts; spread shares the memory line); re-summed at the train R merge with v0.60.574.0: 246 + 1; re-summed at the memory train #8481 merge with v0.60.576.0 (248 measured); +1 Wakeshift PR 1: wakeshiftOperations import (ops/wakeshift.ts; its spread shares the jobs line)
src/core/pglite-engine.ts 3103 ratchet +9 #7191: findPages/countPagesBy delegates and imports (SQL in engine-sql/page-lookup.ts); +4 GBRA-283 B1 (#8455): initSchema seeds an engine opened from settings with settings.embedding and refuses a mismatch, and its open skips the snapshot fast path (logic in embedded-engine.ts; engine methods cannot move); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); 4 Foundations 2 Lane F: autocommit write statements take the WAL checkpoint guard (classifier writesWal in pglite-engine/checkpoint-guard.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); 4 Foundations 2 Lane F: autocommit write statements take the WAL checkpoint guard (classifier writesWal in pglite-engine/checkpoint-guard.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); +1 #5984: upsertChunks guards pass the per-transaction config memo (transactionMemo in page-state/transactions.ts); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2983 measured); 4 Foundations 2 Lane F: autocommit write statements take the WAL checkpoint guard (classifier writesWal in pglite-engine/checkpoint-guard.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); +1 #5984: upsertChunks guards pass the per-transaction config memo (transactionMemo in page-state/transactions.ts); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2983 measured); +4 P7 relationalChainHop delegate; +8 contributor audit wave P0: listFactsKeyset delegate (SQL in engine-sql/facts.ts); +1 #5375: get_versions limit/include_body type import (GetVersionsOpts, PageVersionRows); +2 #5984 lane apply diet: putPage/setPageAliases write inline in the caller's page transaction (engine methods cannot move); +4 GBRA-69: replaceDerivedLinksBatch delegate (logic in derived-links.ts); +5 GBRA-67: readPageSnapshotsBatch delegate (batch logic in page-snapshot-batch.ts); +1 GBRA-75 wave 10: findDuplicatePages delegate (SQL in engine-sql/pages.ts); +2 GBRA-76: transaction/transactionDirect take TransactionOptions and refuse an already-aborted signal before BEGIN; +2 #4772 (W14 PR3): entityTypes HealthDeps delegate + import (policy in schema-pack/entity-types.ts, SQL in engine-sql/health.ts); +2 W14 P3.12 (#4772): entityTypes HealthDeps delegate + import on top of W13; +11 Cat 40 Hard fix wave F2 (#6271) on top: countKeywordPages/searchKeywordPages delegations (SQL in engine-sql/keyword-pages.ts) and the rankQuery score parameter (3023 measured); +3 W15 P1.2 (#6454): three one-line stale-frontmatter-key delegates (SQL in engine-sql/pages.ts); +6 large pages Wave A sub-a: getChunkPage delegate (SQL in engine-sql/chunks.ts); +11 #6696 B3: readEmbeddingStateCounts/listEmbeddingStateRows delegations + type import; +22 Wave B lane 1a: segmented page storage delegations (allocateBodyOccurrenceIds..sweepSegments, SQL in engine-sql/page-body.ts) and their imports; +7 LHF wave 2 item 5: request config snapshot delegate lines in getConfig/getAllConfig/setConfig/unsetConfig (logic in config-snapshot.ts); +3 PR2 fix lane B: logIngest stamps writer_version/writer_host_id/writer_segment_reader and getIngestLog names its columns (engine methods cannot move); integrated at the Wave B PR 2 master merge (v0.60.264.0): master 3072 + 3; +5 PR2 fix lane X: keywordBoundedRead runs the read on the engine itself (engine methods cannot move): 3075 + 5; +2 GBRA-130 B-6 (#6860): putDreamVerdict passes its payload through wellFormedJson (engine methods cannot move); +1 GBRA-130 A-1: held option on the stale-page list/count signatures (SQL in engine-sql/pages.ts); +1 GBRA-130 integration: the A-1 (links_extract_failure) and B-6 (wellFormedJson putDreamVerdict) engine hunks composed at the lane merge; +2 #6948: putPage runs assertOriginUnclaimed in its page transaction and restorePage runs assertRevivalUnclaimed (logic in persistence/origin-guard.ts; engine methods cannot move): 3080 + 2; GBRA-130 composed with #6948 at the master 0e0cd27b merge: re-summed to the merged line count (3085); +2 #7159: the frontmatter where clause in searchKeyword and its import (SQL in search/frontmatter-where.ts; engine methods cannot move); +2 GBRA-156 keyword OR rarity: the import and the searchKeyword OR retry through rankOrFallbackByRarity with an orFallback rank parameter like Postgres (logic in search/keyword-or-rarity.ts; engine methods cannot move); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); re-summed at the Wave B PR 4 merge with v0.60.452.0 (3097 measured); +2 GBRA-283 Wave A (#8460): the ENGINE_ROOT import and field (engine identity for per-engine caches; class fields cannot move)
src/core/postgres-engine.ts 3182 ratchet +9 #7191: findPages/countPagesBy scoped-read delegates and imports (SQL in engine-sql/page-lookup.ts); +6 GBRA-283 B1 (#8455): an engine opened from settings routes its direct URL and statement-description cache from its settings, and initSchema seeds settings.embedding and refuses a mismatch (logic in embedded-engine.ts; engine methods cannot move); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); ceiling rationale: docs/architecture/key-files/engines-2.md entry for src/core/postgres-engine.ts; evidence delivery: getChunkWindows scoped-read delegate + type import (+5; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +10 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3242 #5824: searchVector SQL moved to search/vector-statement.ts, +explainVectorSearch/runVectorAttempt (-156); +6 #5730: onpoisoned wiring on the instance/module/connection-manager pools + poisonedDiscards in getPoolDiagnostics (counter in pool-gauge.ts); +5 #5801: onCheckout observer seam (fires after reserve resolution / transaction-callback entry; listeners in pool-gauge.ts); lowered to 3119 #5889: searchTitles SQL moved to engine-sql/titles.ts (-133); lowered to 2914 F4a: getHealth SQL moved to engine-sql/health.ts (-216); +2 #5984: transactions track held page guards like PGLite (withHeldPageKeys), so repeated guards in one transaction cost no round trips; +1 #5984: traceSqlOptions import for the env-gated GBRAIN_SQL_TRACE bench hook (src/core/sql-trace.ts); v0.60.48.0 merge: #5984 growth re-summed over v0.60.47.0 (2917 measured); +2 engine graduation (GBRA-50 G2a): initSchema sizing parameter (the source embedding layout sizes a fresh target); +1 #5984: upsertChunks guards pass the per-transaction config memo (transactionMemo in page-state/transactions.ts); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2920 measured); +4 P7 relationalChainHop delegate; +8 contributor audit wave P0: listFactsKeyset delegate (SQL in engine-sql/facts.ts); +1 #5375: get_versions limit/include_body type import (GetVersionsOpts, PageVersionRows); 6 #6085: initSchema pins advisory lock 42, bootstrap, schema replay and unlock to one reserved backend when the DDL pool has more than one slot (gauge only, no long-hold permit), released in finally; wave 10 collector: lane E (#6085), lane G (#5375) and master compose at 2939; +2 #6110: search transactions pass jitOff to withScopedReadTransaction (SET LOCAL jit = off in postgres-engine/search-settings.ts); +2 #5984 lane apply diet: putPage/setPageAliases write inline in the caller's page transaction (engine methods cannot move); +1 #5984 single-write fast path composes with the lane apply diet at 2944; +2 #5984 shared parameter types: the instance pool passes shared_types and initSchema clears them after migrations (helpers in db.ts); +2 #6276: the remote alias-resolving readPageSnapshot runs with JIT off (withScopedReadTransaction jitOff); +5 #6278: executeRaw timeoutMs delegate (statement in postgres-engine/bounded-statement.ts); +23 #6278 PR 1.5: cancelSettleMs() and the settle timer in runUnsafe that discards a reserved connection whose cancel a transaction-mode pooler never completed; +3 #6317 lane 2: getPoolDiagnostics().pool from the vendored driver (postgres-engine/pool-stats.ts) and the prepare mark; +4 GBRA-69: replaceDerivedLinksBatch delegate (logic in derived-links.ts); +1 GBRA-69 extract batching: replaceDerivedLinksBatch delegate (batch logic in derived-links.ts); +9 GBRA-75: getBacklinkCounts and getHealth statements run JIT-off through the private jitOffRead transaction helper (engine methods cannot move); +3 GBRA-75 wave 7: searchKeyword and searchVector start the once-per-engine background search statistics guard (logic in search/projection-statistics.ts); +7 GBRA-67: readPageSnapshotsBatch scoped-read delegate (batch logic in page-snapshot-batch.ts); +1 GBRA-75 wave 10: findDuplicatePages scoped-read delegate (SQL in engine-sql/pages.ts); +7 #6383: poolIncidents hooks on the instance/module/connection-manager pools (counter and lines in pool-gauge.ts, budget in db.ts) and stuckDiscards/buildFailures in getPoolDiagnostics; +3 GBRA-76: transaction/transactionDirect take TransactionOptions; transactionOn attaches the begin handle to withTransactionAbort (logic in postgres-engine/transaction-abort.ts); +2 W14 P3.12 (#4772): entityTypes HealthDeps delegate + import on top of W13; +1 #5227: schema.sql replay under withSchemaLockTimeout (wave 14 PR1); +1 driver desync tripwire: desyncDiscards in getPoolDiagnostics (counter and line in pool-gauge.ts); +11 Cat 40 Hard fix wave F2 (#6271) on top: countKeywordPages/searchKeywordPages scoped-read delegations (SQL in engine-sql/keyword-pages.ts) and the rankQuery score parameter (3027 measured); +3 W15 P1.2 (#6454): three one-line stale-frontmatter-key delegates, reads scoped (SQL in engine-sql/pages.ts); +12 wave 16 G5.1 (#6495): keywordStatementTimeout memo field, its connect/disconnect resets and the keywordStatementTimeoutSetting accessor, plus one awaited setting per keyword/chunk/CJK transaction (resolver in postgres-engine/keyword-statement-timeout.ts); wave 16 G5.1 gate fix: keywordStatementTimeoutSetting resolves the knob once per connection on the first keyword statement from the saved connect config (connect issues no statement for it; 3042 measured); +7 large pages Wave A sub-a: getChunkPage scoped-read delegate (SQL in engine-sql/chunks.ts); +11 #6696 B3: readEmbeddingStateCounts/listEmbeddingStateRows delegations + type import; +22 Wave B lane 1a: segmented page storage delegations (allocateBodyOccurrenceIds..sweepSegments, SQL in engine-sql/page-body.ts) and their imports; +3 wave 18b lane 2 (#6495): keywordTimeouts negative-cache field and the searchKeyword runner wrap (logic in postgres-engine/keyword-timeout-cache.ts); integrated at wave 18b: master 3082 + 3; +3 PR2 fix lane B: logIngest stamps writer_version/writer_host_id/writer_segment_reader and getIngestLog names its columns (engine methods cannot move); integrated at the Wave B PR 2 master merge (v0.60.264.0): master 3085 + 3; +3 PR2 fix lane E (E1): getChunks/getChunkPage/getChunkWindows run the manifest privilege probe in their scoped transaction (chunkManifestReads in search/safe-chunks.ts); integrated at the lane E merge onto b75a69a6d: 3088 + 3; +6 PR2 fix lane X: searchVector, explainVectorSearch, the hasMore witness and runVectorAttempt pick the chunk-manifest statement form inside each bounded attempt transaction, so its statement_timeout bounds the probe (form memo vectorStatementForms in search/vector-statement.ts; engine methods cannot move): 3091 + 6; +5 PR2 fix lane X: keywordBoundedRead runs the read in withScopedReadTransaction under the keyword statement timeout setting (engine methods cannot move): 3097 + 5; +2 GBRA-130 B-6 (#6860): putDreamVerdict passes its payload through wellFormedJson (engine methods cannot move); +1 GBRA-130 A-1: held option on the stale-page list/count signatures (SQL in engine-sql/pages.ts); +1 GBRA-130 integration: the A-1 (links_extract_failure) and B-6 (wellFormedJson putDreamVerdict) engine hunks composed at the lane merge; +2 #6948: putPage runs assertOriginUnclaimed in its page transaction and restorePage runs assertRevivalUnclaimed (logic in persistence/origin-guard.ts; engine methods cannot move): 3102 + 2; GBRA-130 composed with #6948 at the master 0e0cd27b merge: re-summed to the merged line count (3107); +3 #7159: the frontmatter where clause in the two keyword typesClause sites and its import (SQL in search/frontmatter-where.ts; engine methods cannot move); GBRA-156 retrieval wave (#7201) composed with #7194 at the merge: re-summed (3114 measured); +2 warm CLI socket (#7218): connectedDatabaseUrl getter (the URL the engine connected with, bound into the socket identity), composed over the GBRA-156 retrieval wave (3116 measured); +1 GBRA-133 PR 2b: transactionOn hands the outermost transaction its commit checks (page-state/transactions.ts withCommitChecks), composed with the warm CLI socket at the merge (3117 measured); +18 GBRA-156: the private readTransaction helper behind withScopedReadTransaction and jitOffRead answers a top-level read before its COMMIT settles (engine methods cannot move; 3135 measured); +1 GBRA-156: transactionOn and runUnsafe hold a per-search connection slot (import of search/connection-budget.ts; 3136 measured); +2 GBRA-156: transactionOn passes TransactionOptions.pipelineFirst to the pool begin as { first } (driver gate in vendor/postgres index.js begin); an engine method cannot move; composed with the warm CLI socket and GBRA-133 PR 2b at the master v0.60.344.0 merge (3119 measured); GBRA-151 #7183/#7187/#7188 re-summed at the v0.60.402.0 merge (3135 measured); +2 #7183 (GBRA-151) at the v0.60.412.0 merge: read-write-transactions import and transactionOn wraps its connection in readWriteTransactions (logic in postgres-engine/read-write-transactions.ts; engine methods cannot move): 3135 + 2; +7 GBRA-156: transactionOn opens readOnly transactions as BEGIN READ ONLY and a top-level read inherits its request signal, runUnsafe refuses an aborted request (logic in request-signal.ts; engine methods cannot move): 3137 + 7; +3 GBRA-156 keyword OR rarity: the import and the searchKeyword OR retry through rankOrFallbackByRarity (logic in search/keyword-or-rarity.ts; engine methods cannot move); +2 wave-b PR 4 merged with #7218 (both sides added imports); re-summed at the Wave B PR 4 merge with v0.60.452.0 (3155 measured); +2 GBRA-283 Wave A (#8460): the ENGINE_ROOT import and field (engine identity for per-engine caches; class fields cannot move); +7 GBRA-283 B1 review: the instance pool passes settings connection fields explicitly, initSchema reads the embedding identity again under the schema lock, and reconnect ends the superseded manager (engine methods cannot move); re-summed at the train R merge with v0.60.574.0: 3170 + 4; +4 #6318: withScopedReadTransaction runs a bounded preparation's read under its deadline (read-deadline.ts; reserved-connection envelope runDeadlineRead in postgres-engine/bounded-statement.ts) and adds statement_timeout to the transaction settings (engine methods cannot move): re-summed at the 0.60.583.0 merge: 3174 + 4; +2 TLS default: the pg-tls import and the instance pool builds its options through withPgTls and turns a no-TLS refusal into postgres_tls_required (logic in pg-tls.ts; engine methods cannot move): 3178 + 2; +2 #7776 (GBRA-404 salvage): the schema-client-behind import and its assertSchemaNotAhead call in initSchema, re-summed on master 78419f76 (3179 + 2); +1 #9295: wantedProducers option on the listStalePagesForExtraction delegate signature (engine methods cannot move)
diff --git a/src/core/migrate-pending.ts b/src/core/migrate-pending.ts
index bd5b31fee2..584ec71bdd 100644
--- a/src/core/migrate-pending.ts
+++ b/src/core/migrate-pending.ts
@@ -1,6 +1,20 @@
import type { BrainEngine } from './engine.ts';
+import { engineKey } from './engine-key.ts';
import { LATEST_SCHEMA_VERSION } from './schema-migrations/latest.generated.ts';
+const SCHEMA_VERSION_SEEN = Symbol('gbrain.schemaVersionSeen');
+type Recorded = { [SCHEMA_VERSION_SEEN]?: number };
+
+/** GST-2687: records config `version` on the engine as it just read or wrote it (the connect gate below, migrations); a transaction or view records on its engine. */
+export function noteSchemaVersion(engine: object, version: number): void {
+ (engineKey(engine) as Recorded)[SCHEMA_VERSION_SEEN] = Number.isInteger(version) ? version : undefined;
+}
+
+/** The schema version `engine` (or the engine a transaction or view belongs to) last recorded, read with no query; undefined when none was. */
+export function schemaVersionSeen(engine: object): number | undefined {
+ return (engineKey(engine) as Recorded)[SCHEMA_VERSION_SEEN];
+}
+
/**
* Cheap probe: does this engine have schema migrations pending?
*
@@ -23,6 +37,7 @@ export async function hasPendingMigrations(engine: BrainEngine): Promise<boolean
try {
const currentStr = await engine.getConfig('version');
const current = parseInt(currentStr || '1', 10);
+ noteSchemaVersion(engine, current);
return current < LATEST_SCHEMA_VERSION;
} catch {
return true;
diff --git a/src/core/migrate.ts b/src/core/migrate.ts
index 7464c9d362..2c27b62e80 100644
--- a/src/core/migrate.ts
+++ b/src/core/migrate.ts
@@ -5,7 +5,7 @@ import { resolveSchemaLockTimeoutMs, schemaLockBlockedError } from './postgres-e
import { SCHEMA_MIGRATION_LOCK_APPLICATION_NAME } from './postgres-engine/init-schema-lock.ts';
import { assertSchemaMigrationLockHeld, withSchemaMigrationLock } from './schema-migration-lock.ts';
import { MIGRATIONS } from './schema-migrations/registry.generated.ts';
-import { hasPendingMigrations } from './migrate-pending.ts';
+import { hasPendingMigrations, noteSchemaVersion } from './migrate-pending.ts';
import { assertOwnedTableShapes } from './owned-table-shapes.ts';
import { runMigrationStatement, setQuietMigrationNotices } from './schema-migrations/helpers.ts';
import { MIGRATION_PRELUDES, runMigrationPrelude } from './migration-preludes.ts';
@@ -380,6 +380,7 @@ export async function runMigrations(engine: BrainEngine): Promise<{ applied: num
async function runMigrationsLocked(engine: BrainEngine): Promise<{ applied: number; current: number }> {
const currentStr = await engine.getConfig('version');
const current = parseInt(currentStr || '1', 10);
+ noteSchemaVersion(engine, current);
// Sort by version ascending so array insertion order doesn't affect
// correctness. Migrations MUST run in version order; if v16 accidentally
@@ -510,6 +511,7 @@ async function runMigrationsLocked(engine: BrainEngine): Promise<{ applied: numb
// Update version after both SQL and handler succeed. Inside the same
// catch so a stamp-write failure is also NAMED in quiet mode.
await engine.setConfig('version', String(m.version));
+ noteSchemaVersion(engine, m.version);
} catch (err) {
// Quiet fresh-install replay: name the failing migration — without the
// per-step lines, the error would otherwise be anonymous.
diff --git a/src/core/search/private-visibility.ts b/src/core/search/private-visibility.ts
index eab1542596..18f3e6a3a4 100644
--- a/src/core/search/private-visibility.ts
+++ b/src/core/search/private-visibility.ts
@@ -22,7 +22,8 @@
import { AsyncLocalStorage } from 'node:async_hooks';
import type { BrainEngine } from '../engine.ts';
-import { embeddedKey, embeddedKeyInScope, unscopedWithOpenEmbeddedEngines } from '../engine-scope-store.ts';
+import { embeddedKey, embeddedKeyInScope, engineScopeStore, unscopedWithOpenEmbeddedEngines } from '../engine-scope-store.ts';
+import { schemaVersionSeen } from '../migrate-pending.ts';
import { requestConfigRows } from '../config-snapshot.ts';
import { isValidSourceId } from '../source-id.ts';
import { asciiLower, asciiLowerSql, edgeTrimSql } from '../edge-whitespace.ts';
@@ -701,15 +702,18 @@ export function privateSnapshotFilterFragment(alias: string): string {
* move whose alias a later rename onto the old slug dropped (#8920). A concept's
* inputs are judged the same way, moved origins included (GST-2687), by the SQL
* function gbrain_concept_inputs_private (migration v311), one planned call per
- * page instead of the subplans, under a caller record that carries a schema version at or above v311 (a remote
- * request, a scope or attested read). Over a shadowed `pages` (privacyPartSql), for a local caller, and on a brain
- * whose schema predates v311 or whose version could not be read, the same test runs inline, as gbrain_access_origin_restricted is
+ * page instead of the subplans, when the read's schema version is at or above v311: the one its remote, scope or
+ * attested caller record carries, or, with no remote policy, the one the engine in scope recorded at connect or
+ * migration (schemaVersionSeen). Over a shadowed `pages` (privacyPartSql), outside an engine scope, and on a brain
+ * whose schema predates v311 or whose version was not read, the same test runs inline, as gbrain_access_origin_restricted is
* skipped inside a request on a brain that never set access.restrictions_present.
*/
function derivedOriginPrivateSql(p: string, shadowedPages: boolean): string {
const privateOrigin = (atom: string, alias: string) => `EXISTS (SELECT 1 FROM pages ${alias} WHERE ${alias}.source_id = ${atom}.source_id
AND ${alias}.slug = ANY(${atomOriginSlugsSql(atom)}) AND ${alias}.frontmatter->>'visibility' = 'private')`;
- const schema = currentCallerAccess().remotePolicy?.schemaVersion;
+ const { remotePolicy } = currentCallerAccess();
+ const scoped = engineScopeStore.getStore();
+ const schema = remotePolicy ? remotePolicy.schemaVersion : scoped ? schemaVersionSeen(scoped) : undefined;
const conceptInputsPrivate = !shadowedPages && schema !== undefined && schema >= v311.version ? `gbrain_concept_inputs_private((${p}.id)::integer)`
: `EXISTS (SELECT 1 FROM links derived_input_link
JOIN pages derived_input ON derived_input.id = derived_input_link.to_page_id
@@ -1141,7 +1145,7 @@ export async function remotePrivacyRelaxed(engine: BrainEngine): Promise<boolean
export interface ServesPolicy { several: boolean; from: 'config' | 'members'; since: number | null; enforce: boolean }
/** `servesUnknown`: the config read failed, so whom the brain serves is not known (never read as one person, GST-363). */
-/** `schemaVersion`: config `version` as the request read it; below v311, or absent, the concept-input test runs inline (derivedOriginPrivateSql). */
+/** `schemaVersion`: config `version` as the request read it; below v311, or absent, the concept-input test runs inline (derivedOriginPrivateSql). A local read has no policy and uses its engine's recorded version instead. */
export interface RemotePolicy { privacyRelaxed: boolean; restricted: boolean; ownerMember?: string | null; ownerReadsRestricted?: boolean; membersReadLineage?: boolean; serves?: ServesPolicy; servesUnknown?: boolean; schemaVersion?: number }
/** GST-363: an explicit access.serves wins over the trigger-kept member count; enforcement is on unless set to false. */
diff --git a/test/concept-rename-history-gst2687.test.ts b/test/concept-rename-history-gst2687.test.ts
index e18d6f6494..0bacf713fb 100644
--- a/test/concept-rename-history-gst2687.test.ts
+++ b/test/concept-rename-history-gst2687.test.ts
@@ -32,6 +32,8 @@ import { bindCredential } from '../src/core/access/members.ts';
import { renamePageOntoSlug } from '../src/commands/sync/rename-collision.ts';
import { derivedVisibilityCheck } from '../src/commands/doctor/checks/derived-visibility.ts';
import { runMigrations } from '../src/core/migrate.ts';
+import { hasPendingMigrations } from '../src/core/migrate-pending.ts';
+import { withEngineScope } from '../src/core/engine-scope.ts';
import { v311 } from '../src/core/schema-migrations/v311-concept-inputs-private.ts';
import type { AuthInfo } from '../src/core/ops/contract.ts';
import { readsUnscopedOnPurpose } from './helpers/reads-unscoped.ts';
@@ -190,11 +192,13 @@ for (const kind of testBackends()) {
await engine.setConfig('version', String(v311.version - 1));
try {
__resetPrivateVisibilityCacheForTests();
- const visible = await engine.transaction(async tx => {
+ expect(await hasPendingMigrations(engine)).toBe(true); // the connect gate's read records the version, as for a CLI call
+ const visible = await withEngineScope(engine, () => engine.transaction(async tx => {
+ expect(privatePagesFilterFragment('p')).not.toContain('gbrain_concept_inputs_private(');
const rows = await tx.executeRaw<{ slug: string }>(`SELECT p.slug FROM pages p WHERE p.source_id = $1 AND ${privatePagesFilterFragment('p')}`, [SRC]);
expect(await tx.executeRaw<{ ok: number }>('SELECT 1 AS ok')).toEqual([{ ok: 1 }]);
return rows.map(row => row.slug);
- });
+ }));
expect({ concept: visible.includes(CONCEPT), atom: visible.includes(ATOM) }).toEqual({ concept: false, atom: false });
} finally {
await runMigrations(engine);
@@ -211,5 +215,16 @@ for (const kind of testBackends()) {
`SELECT p.slug FROM pages p WHERE p.source_id = $1 AND ${privatePagesFilterFragment('p')}`, [SRC])).map(row => row.slug));
expect(visible.includes(CONCEPT)).toBe(false);
}, 60_000);
+
+ test('(k) on a v311 brain a local read with no caller record takes the function form in its engine scope and hides the concept', async () => {
+ __resetPrivateVisibilityCacheForTests();
+ expect(await hasPendingMigrations(engine)).toBe(false);
+ const usesFunction = (scope: object) => withEngineScope(scope, () => privatePagesFilterFragment('p')).includes('gbrain_concept_inputs_private(');
+ expect({ engine: usesFunction(engine), unrecorded: usesFunction({}) }).toEqual({ engine: true, unrecorded: false });
+ expect(privatePagesFilterFragment('p')).not.toContain('gbrain_concept_inputs_private(');
+ const visible = await withEngineScope(engine, () => engine.transaction(async tx => (await tx.executeRaw<{ slug: string }>(
+ `SELECT p.slug FROM pages p WHERE p.source_id = $1 AND ${privatePagesFilterFragment('p')}`, [SRC])).map(row => row.slug)));
+ expect({ concept: visible.includes(CONCEPT), atom: visible.includes(ATOM) }).toEqual({ concept: false, atom: false });
+ }, 60_000);
});
}
diff --git a/test/e2e/private-visibility-plan-postgres.test.ts b/test/e2e/private-visibility-plan-postgres.test.ts
index 95e9250899..46d3460e71 100644
--- a/test/e2e/private-visibility-plan-postgres.test.ts
+++ b/test/e2e/private-visibility-plan-postgres.test.ts
@@ -25,6 +25,7 @@ import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
import { getEngine, hasDatabase, setupDB, teardownDB } from './helpers.ts';
import { privatePagesFilterFragment, withCallerAccess } from '../../src/core/search/private-visibility.ts';
import { v311 } from '../../src/core/schema-migrations/v311-concept-inputs-private.ts';
+import { withEngineScope } from '../../src/core/engine-scope.ts';
const PAGES = 3_000;
/** The slug_rename_history rows the analyzed case writes. */
@@ -50,14 +51,22 @@ function remoteFragment(): string {
return fragment;
}
-/** The estimated cost per page of the IN spelling (before) and the emitted rule (after), and the jit budget per page. */
-async function costPerPage(): Promise<{ before: number; after: number; local: number; budget: number }> {
+/**
+ * The estimated cost per page of the IN spelling (before), the remote rule (after), a local read's rule in its v311
+ * engine's scope (local), the inline concept-input test (inline), and the jit budget per page. Inline is what a read
+ * plans before v311 applies or with no recorded version: the mid-upgrade state, logged and not held to the budget.
+ */
+async function costPerPage(): Promise<{ before: number; after: number; local: number; inline: number; budget: number }> {
const [{ jit_above_cost }] = await getEngine().executeRaw<{ jit_above_cost: string }>(`SELECT current_setting('jit_above_cost') AS jit_above_cost`);
const fragment = remoteFragment();
+ const localFragment = withEngineScope(getEngine(), () => privatePagesFilterFragment('p'));
+ expect(localFragment).toContain('gbrain_concept_inputs_private(');
+ const inlineFragment = privatePagesFilterFragment('p');
+ expect(inlineFragment).not.toContain('gbrain_concept_inputs_private(');
const scan = (predicate: string) => planCost(`SELECT count(*) FROM pages p WHERE ${predicate}`);
const base = await scan('true');
const perPage = async (predicate: string) => ((await scan(predicate)) - base) / PAGES;
- return { before: await perPage(inSubqueryForm(fragment)), after: await perPage(fragment), local: await perPage(privatePagesFilterFragment('p')), budget: Number(jit_above_cost) / 500 };
+ return { before: await perPage(inSubqueryForm(fragment)), after: await perPage(fragment), local: await perPage(localFragment), inline: await perPage(inlineFragment), budget: Number(jit_above_cost) / 500 };
}
/** Both spellings hide the same pages, and each rule of the fixture fires. */
@@ -114,10 +123,11 @@ async function expectSameHiddenPages(): Promise<void> {
});
test('the rule costs the planner little enough per page that a 500-page read stays under jit_above_cost', async () => {
- const { before, after, local, budget } = await costPerPage();
- console.info(JSON.stringify({ metric: 'private-visibility-cost-per-page', before, after, local, budget }));
+ const { before, after, local, inline, budget } = await costPerPage();
+ console.info(JSON.stringify({ metric: 'private-visibility-cost-per-page', before, after, local, inline, budget }));
if (before <= budget) throw new Error(`Precondition failed: the IN spelling already costs ${before} per page (budget ${budget}); this check cannot discriminate.`);
expect(after).toBeLessThan(budget);
+ expect(local).toBeLessThan(budget);
});
test('it stays under jit_above_cost with provenance links and rename history analyzed', async () => {
@@ -137,8 +147,8 @@ async function expectSameHiddenPages(): Promise<void> {
const [{ links }] = await engine.executeRaw<{ links: number }>(`SELECT count(*)::int AS links FROM links WHERE link_source = 'concept-provenance'`);
expect(links).toBe(Math.ceil((PAGES - 4) / 23) * 2);
await expectSameHiddenPages();
- const { before, after, local, budget } = await costPerPage();
- console.info(JSON.stringify({ metric: 'private-visibility-cost-per-page-analyzed', before, after, local, budget }));
+ const { before, after, local, inline, budget } = await costPerPage();
+ console.info(JSON.stringify({ metric: 'private-visibility-cost-per-page-analyzed', before, after, local, inline, budget }));
if (before <= budget) throw new Error(`Precondition failed: the IN spelling already costs ${before} per page (budget ${budget}); this check cannot discriminate.`);
expect(after).toBeLessThan(budget);
});