Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
52.3 KB
Raw Blame

Key files: Engines (2/2)

Subsystem index. Read only the entries relevant to your change. Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.

  • patches/[email protected] — auditable delta for the driver ownership and cancellation repair already applied to the shipped vendor/postgres ESM/CommonJS/workerd source (not an install hook; bash vendor/update-postgres.sh --check verifies it), consumed by PostgresEngine.runUnsafe through #postgres. Abortable reservations settle atomically; shared transaction queues isolate cancellable SQL from tagged/savepoint siblings while preserving unfenced pipelining. A pooled connection (pool max above 1) or a released reservation whose last ReadyForQuery status is not idle is terminated instead of reused, and the onpoisoned(status) option reports it (pinned by test/e2e/postgres-poisoned-connection.test.ts). See PostgreSQL cancellation ownership for settlement, deadline, transport and clean-install invariants. Pinned by test/e2e/persistence-chaos.test.ts. The patch also shares each pool's described parameter types across its connections (shared_types; db.resolveSharedTypes() reads the GBRAIN_PG_TYPE_CACHE=0 kill switch at the four pool construction sites, db.clearSharedTypes() runs after initSchema applies migrations), so a first run on another connection, and every run under prepare: false, skips the describe round trip; pinned by test/e2e/postgres-shared-types.test.ts. #6355: closed() settles a startup's internal query (the array-types fetch) and clears its saved error before reconnecting for initial, so a backend terminated during startup neither hands the caller's first statement a stale 57P01 on the fresh connection nor leaves an unhandled rejection (fetchArrayTypes also swallows its own failure); it clears the pending write buffer and its immediate handle, so a close that catches a buffered statement does not leave the next StartupMessage appended and unscheduled (a pool slot stuck at CONNECT_TIMEOUT); pinned by test/postgres-driver-startup-death.test.ts against a scripted wire-protocol server. A query cancelled after it was written stays current until its own ReadyForQuery, so its answer is never read for the next query, and rejects 57014; no server cancel is sent for it while statements are pipelined behind it, because a CancelRequest stops whatever the backend is running (pinned by test/e2e/postgres-driver-cancel-pipeline.test.ts, direct and PgBouncer). A statement answer that arrives with no current query retires the connection with CONNECTION_DESYNC and calls ondesync({ message, connection, statement }); PoolIncidentCounter logs pg_connection_desync (pinned by test/e2e/postgres-driver-desync-tripwire.test.ts). #6930: array_types hands every pool for one database target the array types the process's first connection fetched, so later connections skip the pg_type round trip; transaction_pipeline pipelines on a connection its owner holds (sql.begin, sql.reserve) while every statement in flight leaves it inside a transaction and never behind COMMIT/ROLLBACK/END/ABORT/PREPARE TRANSACTION, so a transaction-mode pooler URL (max_pipeline 0, db.transactionPipelineOptions, GBRAIN_PG_TXN_PIPELINE=0 off) keeps #6543's boundary; pinned by test/pathological/pg-round-trips-6930.test.ts and test/e2e/pgbouncer-transaction-pipeline-6930.test.ts. GST-2260: the first RevalidateCachedQuery error (0A000: a prepared SELECT */RETURNING * whose table changed) bumps a process-wide plan generation, so every connection drops its prepared statements before its next query: one failure per process per migration, not per pooled connection (test/stale-plan-heal.test.ts).

  • src/core/pg-type-cache.ts — the store shared_types uses: one SharedParameterTypes Map per database target in this process (sharedParameterTypes(url)), keyed by SHA-256 digests of the driver's (types, SQL) key, and saved under <gbrain home>/cache/pg-types/ (0600) for the next process. db.checkPoolAndLoadSharedTypes reads its scope (server_version_num, config.version, and the database as the server names it, so a proxy and a direct URL share a file) in each new pool's connection check and loads it; a different scope is ignored and replaced, a migration's clearSharedTypes deletes the file. GBRAIN_PG_TYPE_CACHE_PERSIST=0 keeps it in-process. Pinned by test/pg-type-cache.test.ts and the new-process case in test/e2e/postgres-shared-types.test.ts. #6930: sharedArrayTypes(url) is the per-target, in-process-only array_types store (extension OIDs belong to one catalog); GBRAIN_PG_TYPE_CACHE=0 turns it off and clearSharedTypes empties it.

  • src/core/engine-sql/ — one SQL implementation per migrated storage domain (the domains are the migrated rows of scripts/engine-sql-baseline.tsv), run by both engines through one executor contract:

    PGLiteEngine / PostgresEngine   one-line delegations; RLS scoping per method as on master
          | get engineSql()         fresh adapter per access, never stored (transaction clones)
          v
    engine-sql/<domain>.ts          Postgres master text; sqlFragment composition; capabilities
          |  ScopedRead / LegacyUnscopedRead brands (brands.ts), row kinds (normalize.ts)
          v
    engine-sql/executor.ts          SqlExecutor: query/run, executeRaw, unsafe, transaction; {rows, affectedRows}
          |
          +--> dialect-pglite.ts    db.query on the engine's cached, checkpoint-guarded handle
          +--> dialect-postgres.ts  runUnsafe(conn, sql, params, {signal?, prepare: true, simple: false}); no gauge
    

    Outside engine.transaction(), engineSql hands engine-sql guardedHandle (src/core/pglite-engine/checkpoint-guard.ts): WAL-writing statements take runStatement, executor transactions runOutermost, like executeRaw and transaction(). Before that, engine-sql autocommit writes such as markPagesExtractedBatch never probed WAL; past the automatic trigger (about 528 MB with max_wal_size 1 GB) the crossing UPDATE ran the checkpoint inline and spun forever (GBRA-69). Pinned by test/pglite-checkpoint-guard.test.ts and test/fixtures/pglite-engine-sql-wal-worker.ts.

    Inputs engine-sql never resolves arrive already resolved: source scope (sourceIds > sourceId > 'default'), the registry-active embedding column (bare name, quoted with quoteIdentifier), page-state guards as callbacks (lockPageKeys / readPageSnapshot) and the engine's transaction() where master used it. A read master ran inside withScopedReadTransaction keeps that literal call in the engine method (the RLS AST inventory counts direct sites) and brands the tx handle's executor: scopedRead(this.engineSqlOn(tx)) on Postgres, scopedRead(this.engineSql) on PGLite.

    fragment.ts numbers placeholders at render time, so text is byte-identical to the tagged templates it replaced (test/fixtures/goldens/sql-text/). Capabilities (maxBindParamsPerStatement, transactionAdvisoryLocks, probesEmbeddingCast) carry the real engine differences; docs/designs/refactor-wave-1/w1-inventory.md classifies every method. Page-state orchestration stays in the engines: putPage's transaction, page-key lock and revision check, and the updateSlug / setPageAliases transactions, call pages.ts on the transaction clone's executor; getPage / readPageSnapshot keep their SQL in page-state/snapshot.ts. Reads with an early return that must not open a scoped transaction (listPrefixSampledPages, listCorpusSample) take a ScopedReadRunner. Nothing here imports an engine façade (check:layering); SQL may splice only constant text (check:engine-sql-dynamic); brands are unforgeable (check:engine-sql-brands); engine SQL only shrinks (check:engine-sql-ratchet). Tests: docs/TESTING.md "Engine-sql".

  • src/core/engine-sql/facts.ts listFactsKeyset + src/core/engine-sql/pages.ts keyset predicates — delta's two keyset reads. Both bind the cursor timestamp as ${value}::text::timestamptz: a bare ::timestamptz param is typed by the postgres.js driver, which serializes a string through a JS Date and truncates it to milliseconds (on Postgres that re-selected the whole last millisecond every wake and never drained a >limit cluster inside one millisecond). listFactsKeyset reads ascending (created_at, id) strictly after the keyset (id: null = strictly after the timestamp) and projects created_at_iso at column precision; recall's newest-first listFactsSince is unchanged. test/engine-sql-facts-keyset.test.ts, test/delta-cursor-integrity.test.ts (both engines).

  • src/core/engine-sql/keyword-pages.ts + src/core/search/keyword-statement.ts — the strict keyword count (BrainEngine.countKeywordPages) and the keyword-mode page query (BrainEngine.searchKeywordPages) for both engines. buildKeywordMatch is the ONE WHERE builder both statements read: searchKeyword's strict match (websearch FTS on content_chunks.search_vector, or the CJK LIKE/ILIKE terms), never its OR retry, with the same page filters, hard excludes and buildVisibilityClause (private pages, safe chunks, so no private fence text). A query with slug handles (search/keyword-handles.ts) adds a second arm under the same filters, the chunks of the pages whose slug basename equals a handle (idx_pages_slug_basename), scored 1 + rank so they lead; the count is a UNION of both arms and buildKeywordHandlePagesStatement reads the handle arm alone for searchKeyword. The match returns its WHERE params first and the score-only rank params after them (whereParamCount), so the count binds only what its WHERE reads (a CJK count bound the term-frequency params and failed on both engines; test/sql-bind-placeholders.test.ts). The count is count(*) over SELECT DISTINCT page_id … LIMIT 10001; a non-timeout count failure logs its SQLSTATE once per process (keyword-paging.ts); the page query takes each page's best chunk before paging (no global chunk cap), orders by (score DESC, page_id ASC) with float8 scores so the keyset cursor round-trips, and pages those (page_id, chunk_id, score) rows before it reads the page and chunk rows it returns (#7082; test/search/keyword-pages-limit-first.test.ts, Postgres arm in test/e2e/keyword-pages-limit-first-postgres.test.ts). Postgres runs both inside withScopedReadTransaction with a statement timeout (the count's set from the caller's deadline); PGLite has none.

  • src/core/search/exact-scan.ts + src/core/ops/search-exact.ts + src/cli/search-exact-format.ts — #7191: search_exact (CLI gbrain search --exact, routed in cli/main.ts before ranked-search flags parse): an unranked literal scan of content_chunks.chunk_text (words in order, any whitespace between, case-insensitive unless case_sensitive; the ARE comes from phrasePattern, with a LIKE prefilter on the longest word) in keyset batches of pages by id (buildExactBatchStatement), under ranked search's visibility (live page, unarchived source, quarantine, pinned questions, chunkVisibleFilter, source scope, private pages and the trust floor for untrusted callers, safeChunksFilter for remote) and the hard-excluded prefixes. complete is true only when the scope was scanned to the end with no limit or deadline stop and no unreadable page (unchunked, projection_pending, remote unsealed, scan_timeout); a Postgres statement timeout halves the batch, down to one page. resume_after continues a bounded scan. Registered filtered (min_trust, stampPageTrust), output redaction retrieval. Guide: exact search. Tests: test/search-exact.test.ts, test/e2e/search-exact-postgres.test.ts.

  • src/core/engine-sql/titles.ts + src/core/search/title-statement.ts — the page-grain title arm (BrainEngine.searchTitles) for both engines: buildSearchTitlesStatement in search/ (SQL composed outside engine-sql like cjk-keyword-sql.ts; also used by the plan-proof E2E test/e2e/title-arm-postgres.test.ts) and searchTitles(scoped, query, opts, dialect) in engine-sql, where dialect carries the engine differences kept from master (Postgres: 8s per-attempt statement_timeout, enable_seqscan off for the relaxed local re-run; PGLite: the timeline stale probe). Both orderings sort an exact-title key first (lower(btrim(title, ' "''' || chr(160))) = normalizeAlias(query), bound separately from the websearch text so the OR fallback never rebinds it; #5889). The websearch text is read through gbrain_fts_input($1) (#6370), the same CJK/ASCII boundary split the index applies, so a glued title query matches. Safe-chunks callers and the title-only OR re-run match ts_filter(search_vector, '{a}') @@ q (weight A is the title alone) and rank on the same slice: served by the expression index idx_pages_title_search (GBRA-156, v267), which holds title lexemes only, so a term in timeline text never makes its page a candidate; title-only for negated and phrase terms, following the stored search_vector language. Postgres runs each attempt with the keyword statement's work_mem (statementSettings). Pages with a NULL search_vector are absent from the remote arm. #7471: a 2-4 word name query ORs in its given-name variants (titleNameVariants: "Matt Example" also matches the title "Matthew Example"), each its own bound websearch query; the OR fallback of a query of at most RARITY_MAX_TERMS words runs once per word in one scoped read and rankByRarity puts titles holding the rarest word first (ts_rank_cd knows no term rarity, so a surname's one title used to tie thousands of first-name titles and lose on page id). #7491: OR-retry titles holding every identifier the query names (holdsIdentifiers, search/keyword-identifier-probe.ts) lead untagged; the rest stay keyword_relaxed. #7082: a local caller's strict attempt matches the whole vector but ranks at most titleWholeVectorCandidates() (5,000) matching pages, the lowest page ids (the candidates CTE, ordered by id); a cut attempt reports keyword_candidates_incomplete (reason candidate_budget) and puts the title-only statement's rows first. GST-2899: a local caller's OR re-run first runs the whole-vector statement with completeOnly (the same cap, an unordered count, rows only when every matching page fits, else one row marked candidates_cut), so a reworded question still reaches pages whose timeline names its subject; when the set does not fit (a date, a common word) it runs the title-only statement as before. Plan proof: test/e2e/title-any-word-retry-postgres.test.ts.

  • src/core/engine-sql/page-lookup.ts + src/core/page-find.ts + src/core/page-counts.ts + src/core/ops/find-pages.ts + src/core/ops/stats-by.ts + src/cli/format-page-lookup.ts — find-by-name and grouped counts (gbrain find / find_pages, gbrain stats --by / get_stats by, since, collapse_aliases). parseFindPattern turns a glob (*/?, whole slug or source path), a slug prefix (contains /) or a name (title or last slug segment substring) into one escapeLikePattern-escaped ILIKE pattern; names are served by idx_pages_trgm + idx_pages_slug_trgm, while globs and prefixes also match source_path (no trigram index). Rows sort by slug then source in the C collation, carry count(*) OVER () as total, and the op reports complete (limit default 50, cap 500 local, 100 remote). Find hides the ranked-search hard excludes (resolveHardExcludes + buildHardExcludeClause) unless the pattern's literal lead is inside one, and names them in hidden_prefixes. countPagesBy returns per (source, type) counts plus raw and distinct-slug totals (per type group through an unnest mapping and GROUPING SETS), so cross-source twins are never silently double counted; since is the query effective-date bound (resolveDateBoundary). Every caller sees live pages in unarchived sources; a remote caller's readPolicyOpts policy adds pageReadFilter(..., live). collapseTypes lets the active pack decide (declared stays, alias joins canonical), else normalizes (last segment, lower case, singular) and joins a -<name> suffix to an existing group; each group lists member_types with its rule (exact, pack_alias, normalized, suffix). Without the grouping params get_stats returns its unchanged counters. Tests: test/page-find-pattern-7191.test.ts, test/page-lookup-7191.test.ts (Postgres arm: test/e2e/page-lookup-7191.test.ts).

  • src/core/engine-sql/health.ts — BrainEngine.getHealth once for both engines (migrated health). Three statements, no per-page rows returned: the counters (pages, chunks, links, embedding coverage, missing embeddings, dead links, entity link/timeline coverage), the linkable scope grouped by page type with islanded and with-timeline counts (the orphan policy rendered in SQL by orphanExclusionSql; the brain_score timeline grading applies per type), and most_connected as live, in-scope link endpoints grouped per entity page (ties break by slug in code-point order; a link whose far endpoint is soft-deleted does not count, #6940). The linkable groups are a separate statement because folding them into the counters crosses Postgres's JIT cost thresholds at about 20k pages. Per-page and per-link checks that misplan without planner statistics (PGLite before its first ANALYZE) are primary-key lookups the planner cannot reorder (scalar subqueries, a LATERAL fenced with OFFSET 0), so the statements stay linear with or without statistics. Scope rules: sourceIds > sourceId > brain-wide, duplicate ids count once, a link counts only when both endpoints are in scope, endpoint liveness in both directions, quarantined pages out of the entity denominators and the linkable scope. Entity pages (#4772) are the types the active schema pack(s) declare as entities, resolved per source by HealthDeps.entityTypes (src/core/schema-pack/entity-types.ts) and bound as text[] parameters (entityTypePredicateSql: one list, or a CASE p.source_id when sources' packs disagree); a pack that does not load binds an empty list and the result carries entity_types_status: 'pack_unavailable', never the old hardcoded list. Reads take LegacyUnscopedRead (pool, no scope transaction). Engine deps are four functions (embeddingColumn, countStalePagesForExtraction, getConfig, entityTypes). Pinned field by field against the pre-F4a implementation (test/helpers/legacy-get-health.ts, whose most_connected was corrected for #6940 as an intended change) by test/engine-sql-health-equality.test.ts and test/e2e/engine-sql-health-parity.test.ts (direct Postgres and PgBouncer); the pack-driven entity predicate by test/health-entity-types-pack.test.ts and test/e2e/health-entity-types-pack.test.ts; most_connected liveness by test/health-most-connected-liveness-6940.test.ts (+ Postgres twin).

  • src/core/engine-sql/graph-rank.ts — #7168: rankEntitiesStatement(opts), the one rank_entities aggregate for both engines (a statement builder: the op runs it through executeRaw with its statement deadline, like extractionLagStatement). Edges are distinct (from, to, link_type) triples between live, in-scope endpoints (both endpoints scoped by sourceIds), self-links, non-ranking types (rankingLinkSql) and, unless includeMentions, mention rows excluded. Retired pages (retiredPageSql) fold into their superseded_by/merged_into target in the same source through a recursive chain over retired pages only, capped at MAX_RETIREMENT_HOPS; edges dedupe after folding and folded self-links drop. With remote every endpoint, chain hop, fold target and non-null link origin must be in scope, live, unquarantined, in an unarchived source and (with excludePrivate) not private, so a hidden target never folds. Ordered by the direction's count, then slug and source id (COLLATE "C"), LIMIT after folding. Tests: test/rank-entities.test.ts.

  • src/core/engine-sql/exact-identity.ts — GBRA-156: exactIdentityStatement(opts), the whois candidate read (a statement builder the op runs through executeRaw): a UNION of equality probes each served by an existing index, lower(title) (idx_pages_source_title_lower; the raw query is also lowered by the database), the slug basename (idx_pages_slug_basename, accent-folded by the slug grammar), page_aliases.alias_norm, and one GIN containment arm per address and shape (email: x, email: [x], emails: x, emails: [x]; one OR of them plans a scan). Every probe and row stays in sourceIds; rows are live, in an unarchived source, of a type the EntityTypeFilter admits, and with remote unquarantined and (with excludePrivate) not private. The rows are candidates; the op verifies each. Accent-insensitive titles are found only through the lowered title (same accents, or none in the title) or the slug basename; a case-insensitive frontmatter email only through its as-written or lowercase value or a declared alias. Closing those gaps needs expression indexes (a migration). Tests: the index-plan case in test/whois.test.ts and test/e2e/whois-postgres.test.ts.

  • src/core/engine-sql/chunks.ts upsertChunksOnce — a chunk replacement inside the caller's transaction: the batched chunkWriteInvalidations guard, then one prune (moveDisplacedChunksAndPrune: a row whose vector still matches its text (movableChunkSql) and whose exact text and source reappear at another index is parked at a negative index and moved there; rows outside the new set are deleted), then one multi-row INSERT per jsonRowBatches batch (at most ROW_BATCH_MAX_ROWS rows and ROW_BATCH_MAX_BYTES encoded bytes; the first is pipelined with the seal). A failure in any batch rolls back the prune and every earlier batch. Tests: engine-sql-chunks-batched-write.test.ts, the multi-batch case in helpers/engine-sql-rollback-cases.ts. Its sealChunks is BrainEngine.sealChunks, run inside the engine's page transaction: lock and read the live page, refuse a missing, deleted or segmented page, an archived source and a stale expectedRevision, then delete the chunks, insert the new ones with the seal chunker version, set the search vector and text_projection_revision, and drop the page's queued projection job. Both engines delegate to it; the engine parity suite seals its fixture chunks through it.

  • src/core/pglite-resetwal.ts — pg_resetwal for PGLite NodeFS data dirs, in TypeScript (a port of the electric-sql/pglite pg_resetwal contribution, PR #994, Apache-2.0). Validates the PG17 pg_control layout fail-closed (WalResetUnsupportedError on any unsupported shape — PG_VERSION ≠ 17, control ≠ 8192 bytes, control version ≠ 1700, bad seg/block size), removes stale postmaster.pid + old WAL segments + archive_status/summaries entries, writes a replacement shutdown-checkpoint WAL segment + CRC32C'd pg_control. Both file writes are atomic + durable (tmp cleared then opened 'wx' so a pre-planted symlink at the predictable tmp name can never redirect the write, + fsync(tmp) + rename + fsync(parent dir)); write order is segment-first/control-last so a mid-write kill leaves a state that still fails startup and the next attempt re-runs (idempotent — a torn pair can never claim success). WAL segment size is capped at 64MB (pglite ships 16MB; the Postgres-general 1GB bound would let a corrupt-but-plausible control field drive a 1GB allocation on the repair path). Exports the shared PG17 layout literals (PG_CONTROL_FILE_SIZE, isWalSegmentName) consumed by pglite-repair.ts. LAYOUT COUPLING: any pglite bump past PG17 must revisit this file together with the ./vector export blocker (TODOS.md "pglite upgrade blocker" entry). Pinned by test/pglite-resetwal.test.ts.

  • src/core/pooler-notice.ts — the transaction-pooler banner (db.connect() with prepared statements off), printed once per database target per gbrain home: a marker under <gbrain home>/notices/pooler-prepare-off/ named by a digest of the credential-free target (typeCacheTarget) and the banner text, so a reworded banner or another database prints again; an unwritable home prints every time. gbrain engine status (prepare=false) and doctor's pgbouncer_prepare report the setting on demand. Pinned by test/pooler-banner-once.test.ts.

  • src/core/pool-budget.ts + src/core/persistence/pool-capacity.ts — physical-pool shared long-hold ceilings retain one ordinary/control connection across engine clones; pool size one pauses canonical consumption with writer_pool_capacity. withReservedConnection(fn, { selfContained: true }) (migration DDL, buildIndexOnline, backfill batches: fn touches only its reserved connection, like a transaction) may take the only connection of a one-connection ordinary pool, never the only direct session; pinned by test/pool-budget.test.ts and test/e2e/pool-size-one-migrations.test.ts. Publication concurrency is capped at two per actual Postgres pool and one per PGLite owner. persistence/takes-prepare.ts journals semantic takes with frozen dates/resolver identity and original/current holder ceilings; canonical row updates and receipt commit together. Publication retains the actual affected holders privately in authority JSON, so narrowed holder grants also hide committed/replayed receipts after input compaction.

  • src/core/pool-gauge.ts — CheckoutGauge: approximate in-flight counters at the engine's raw/direct/reserved/tx seams, surfaced via duck-typed PostgresEngine.getPoolDiagnostics() (no BrainEngine change, no PGLite stub). HONESTY CONTRACT in the module doc: tagged-template traffic is untracked; consumers must label counts as a subset and never derive waiter/available figures. Fail-open (clamped release, try/finally around sync-throwing runUnsafe). Consumed by db-probe.ts. Pinned by test/pool-gauge.test.ts. PoisonedDiscardCounter counts the driver's non-idle connection discards and writes one fixed-shape pg_connection_poisoned warn line each; the engine wires it into every pool it creates and reports it as poisonedDiscards. onCheckout(listener) (exposed as PostgresEngine.onCheckout) fires after a connection is actually obtained: signalled raw reserve resolution, transaction-callback entry and withReservedConnection reserve resolution; it never fires for a call still waiting (pinned by test/e2e/postgres-checkout-observer.test.ts). formatBuildFailureWarning appends param=$n (or param=? for an array element) to an UNDEFINED_VALUE line from the driver's onbuilderror(code, statement, parameter) (#6543).

  • src/core/postgres-engine.ts — Postgres + pgvector implementation (Supabase / self-hosted). The package-private #postgres import maps directly to the shipped cancellation-capable vendor/postgres source (the same source is embedded by compiled builds); vendor/README.md and the hash-verifying updater own provenance and regeneration. postgres-engine/cancellation.ts shares actual reserved-owner capability detection, typed postgres_cancellation_unavailable errors, and abort-bounded reservation with late-owner release; unsignalled diagnostic queries remain usable. Signalled SQL must retain cancellation/discard safety, including through transaction-mode poolers. The migrated storage domains' methods are one-line delegations into src/core/engine-sql/<domain>.ts over the private engineSql getter (a fresh Postgres adapter over this.sql on every access; RLS-scoped reads brand the withScopedReadTransaction handle with scopedRead), Postgres-only code stays under src/core/postgres-engine/, and the façade keeps its full public surface. addLinksBatch/addTimelineEntriesBatch/addTakesBatch pass the batch as one JSONB document — INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(...) JOIN pages ... bound through executeRawJsonb({ rows }) — which encodes arbitrary free text safely (an unnest(${arr}::text[]) array-literal path would crash Postgres with "malformed array literal" on free text such as calendar/Zoom context) and sidesteps the 65535-parameter cap; takes declares native recordset column types (page_id int, weight real, active boolean, …) so no per-element casts; all three are batchRetry-wrapped. disconnect() runs drainBackgroundWorkBeforeDisconnect() before pool teardown (engine parity with PGLiteEngine — mode 'disconnect', so residual telemetry buffers drop symmetrically on both engines; guarded so a never-connected/already-torn-down engine skips the drain). searchKeyword/searchVector scope statement_timeout via sql.begin + a transaction-local set_config so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection; the keyword-side bound (searchKeyword, countKeywordPages, searchKeywordPages, searchTitles, searchKeywordChunks, _searchKeywordCJK) is search.keyword_statement_timeout_ms (#6495; default 8000, range 1000-60000), resolved once per connection by keywordStatementTimeoutSetting() on the first keyword statement (postgres-engine/keyword-statement-timeout.ts: the EngineConfig.keyword_statement_timeout_ms value the engine connected with, else the DB plane's row read by one parameterless pool statement so no pg-types cache entry is described and connect() issues nothing for it, else the default; never a request parameter), with countKeywordPages keeping a caller's narrower timeoutMs. Every search transaction (keyword, keyword chunks, CJK, title, the vector attempts and the vector hasMore witness) also passes jitOff and runs with jit = off (postgres-engine/search-settings.ts): the correlated visibility subplans push these short reads past the JIT cost thresholds, and LLVM compilation, not execution, dominated their latency on large brains. withScopedReadTransaction applies every setting a search read needs (RLS app.scopes, jit, the caller's settings: statement_timeout, the relaxed arms' enable_seqscan, the vector scan settings from vectorScanSettings, a thunk read after BEGIN) in one withSearchSettings statement, SELECT set_config('<name>', $n, true), …, so a top-level read is BEGIN, settings, query, COMMIT; statement_timeout stays a literal name with a bound value so the vendored driver's #6543 budget reads it. Inside a caller's transaction the search runs in a savepoint, reads the caller's values in one statement and restores every one afterwards (a failed read rolls them back with the savepoint). A top-level read transaction opts in to begin({ early: isPipelinableSearchRead }) (GBRA-156): its settings statement and its reads (single SELECT or WITH statements naming no write, lock, sequence or session setting) go out behind BEGIN before BEGIN's answer on any pool that pipelines inside a transaction, one round trip instead of two. Pinned by test/search-settings-one-statement.test.ts and test/e2e/search-query-contract-postgres.test.ts. executeRaw(sql, params, { jitOff: true }) (postgres-engine/jit-off-statement.ts) gives single statements the same setting only when it pays: the server's jit is read once per pool, a JIT-off server runs the plain statement, and a JIT-on server runs BEGIN; SET LOCAL jit = off, the statement and COMMIT pipelined in the runBoundedStatement envelope (postgres-engine/bounded-statement.ts, which keeps timeoutMs/deadlineAt and the abort signal); inside a caller's transaction it reads, overrides and restores the caller's value. Pinned by test/e2e/typed-search-jit-postgres.test.ts. getEmbeddingsByChunkIds (in src/core/engine-sql/chunks.ts, both engines) uses tryParseEmbedding so one corrupt row skips+warns instead of killing the query. searchKeyword/searchKeywordChunks/searchVector apply source-aware ranking by inlining the source-factor CASE and NOT (col LIKE …) hard-exclude from src/core/search/sql-ranking.ts; searchVector is a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in outer SELECT) carrying p.source_id inner→outer. _savedConfig retains the connect config; reconnect() tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures, and by batchRetry on a retryable connection error). Concurrent callers share one in-flight _reconnectPromise (they await the single reconnect rather than racing a half-rebuilt pool); ownership re-samples through the atomic db.connect() token on the connect leg. reconnect(ctx?) accepts the triggering error and records a pool-recovery audit event (reap_detected/reconnect_other/reconnect_succeeded/reconnect_failed) for the pool_reap_health doctor check. executeRaw is a single-statement passthrough — no per-call retry (unsound for non-idempotent statements; recovery is supervisor-driven). connect() applies resolveSessionTimeouts() from db.ts as connection-time startup parameters (statement_timeout, idle_in_transaction_session_timeout) so orphan pgbouncer backends can't hold locks for hours. The chunk SQL below lives once in src/core/engine-sql/chunks.ts; the engine resolves the active column and source scope, owns the retry/transaction wrapper and the RLS scope transaction, and passes them in. countStaleChunks()+listStaleChunks() server-side-filter on embedding IS NULL for embed --stale (eliminates ~76 MB/call client-side pull); upsertChunks() routes text-embedding writes through the column registry (both engines in parity): a caller-resolved opts.embeddingColumn descriptor wins; otherwise the DB-plane search_embedding_column + embedding_columns config rows resolve via resolveWriteColumnFromConfigRows to the SAME active column + cast the read side searches (legacy embedding::vector on pre-registry brains; so a registry-routed brain never writes with a dimension mismatch against the legacy column). The INSERT column list, the $N::vector|::halfvec(N) cast, and every embedding reference in the ON CONFLICT CASE branches use the resolved column; it resets both the active column AND embedded_at to NULL when chunk_text changes without a new embedding. Pinned by test/e2e/upsert-chunks-registry-column.test.ts (PGLite always; Postgres DATABASE_URL-gated). initSchema() calls applyForwardReferenceBootstrap(conn) BEFORE replaying SCHEMA_SQL — a one-line delegation to the shared src/core/engine-sql/bootstrap.ts (Postgres dialect hooks: current_schema() probes plus the Postgres-blob-only dream_verdicts.expires_at probe); the entire probe path runs on the DDL connection threaded from initSchema (so concurrent bootstraps cannot race on a Supabase pooler). initSchema runs the bootstrap, the schema replay and the migrations under the schema migration lock (GBRA-467, holdSchemaMigrationLock, key 42), the same lock runMigrations() takes when gbrain apply-migrations calls it directly (withSchemaMigrationLock), so no two runners apply a migration at once; the runner reads the schema version once the lock is held and checks it is still held before each migration, and PGLite (one process per datastore) serializes in-process (src/core/schema-migration-lock.ts, test/schema-migration-lock-pglite.test.ts). The lock is pg_try_advisory_xact_lock(42) inside a READ COMMITTED transaction held open on a backend reserved from the DDL pool, or from the statement pool when the DDL pool has one connection (a short-lived command's idle direct connection would time out and drop it); it holds through a transaction-mode pooler and is released when its holder dies. Only when every pool has one connection does it take the session lock. A timed-out waiter is told not to terminate a holder tagged gbrain-schema-migration-lock. The bootstrap and replay run on a second reserved backend (checkout gauge only, no long-hold permit, so pending transaction: false migrations can still reserve); a one-slot pool runs them unreserved. Pinned by test/e2e/postgres-initschema-advisory-lock.test.ts test/e2e/schema-migration-lock-postgres.test.ts (direct and PgBouncer passes) and test/e2e/schema-migration-lock-direct-pool-postgres.test.ts. disconnect() is idempotent — _connectionStyle tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than clobbering the singleton; and a module-style engine only calls db.disconnect() when it owns the singleton (_ownsModuleSingleton, set from the db.connect() creation token), so a borrower probe engine's teardown leaves the cycle owner's connection intact. Pinned by test/e2e/postgres-engine-disconnect-idempotency.test.ts + test/postgres-engine-singleton-lifecycle.test.ts (concurrent-connect ownership, db.disconnect() detaching the singleton before awaiting end(), and the instance-reconnect _reconnecting guard). getBrainScore empty-brain parity with PGLite — 100/100 with breakdown 35/25/15/15/10 when pageCount === 0 (both engines must agree to keep doctor-report-remote.serial.test.ts deterministic). Implements deletePages(slugs, {sourceId}): Promise<string[]> via DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug (single round-trip; caller chunks); resolveSlugsByPaths does SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2; FK cascades through content_chunks/links/tags/raw_data/timeline_entries/page_versions, files.page_id+links.origin_page_id go SET NULL; throws when input exceeds DELETE_BATCH_SIZE (from src/core/engine-constants.ts); both short-circuit on empty input. Implements the embedding-signature stale-detection quartet (sumStaleChunkChars, setPageEmbeddingSignature, invalidateStaleSignatureEmbeddings, widened countStaleChunks, all accept optional signature extending "stale" to model/dims-swap drift via the pages.embedding_signature JOIN, NULL grandfathered; the embedding IS NULL server-side filter is preserved as the no-signature fast path); Pinned by test/e2e/engine-parity.test.ts. Retry classifiers, ontology/recency helpers, and disconnect/pool-recovery audit writers bind statically; only the two ai/gateway.ts fallback lookups stay lazy and line-marked, in parity with PGLite. insertFact + insertFacts do not hardcode tx.unsafe(\'${embedLit}'::vector`)for the embedding column.resolveFactsEmbeddingCast()(private) probespg_attributeonce per engine instance (cached in_factsEmbeddingCastSuffix) and returns '::halfvec'when migration v40 created the column as halfvec, else'::vector'; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). withReservedConnectionroutes long-hold reserved work (CREATE INDEX CONCURRENTLY,transaction: falsemigration DDL, backfill write batches) to the DIRECT session lane when dual-pool is active, so multi-minute holds do not pin the worker's shared read pool — never rerouted inside an open transaction (same guard shape asexecuteRawDirect), semaphore-capped below direct_pool_sizewith deliberately NO minimum-1 floor (atdirect_pool_size=1a floor would let one long reserve consume the only direct session and starve the claim/renewLock heartbeats — the same starvation on the direct lane), overflowing to the shared read pool when the direct lane has no spare capacity or is unavailable; the permit is released on both fn throw and reserve failure.getPoolDiagnostics()(duck-typed, no BrainEngine change) surfaces theCheckoutGaugein-flight counters fromsrc/core/pool-gauge.tsat the raw/direct/reserved/tx seams — a tracked SUBSET (tagged-template traffic is untracked) thatdb-probe.tslabels honestly in health-probe failure lines. Pinned bytest/postgres-engine-reserved-routing.test.ts. getRawDatajoinspagesand addsp.deleted_at IS NULLunlessopts.includeDeleted(thealivefragment next toprivacyin all six branches) so raw session metadata follows the page soft-delete; PGLite mirrors it with onewhere.push; pinned by the standalone getRawData describe in test/e2e/engine-parity.test.ts(PGLite arm always runs; Postgres arm when DATABASE_URL is set).includeDeletedis NOT on the sharedPageReadScope(only getRawData honors it) — it is declared per-method asgetRawData(slug, source?, opts?: PageReadScope & { includeDeleted?: boolean }) in engine.ts and both engines; resolved only by trusted callers (export, engine migration; the ingest probe passes it but only ever runs against a live page), never from MCP parameters (get_raw_databuilds its scope fromreadPolicyOpts(ctx)). #6278 PR 1.5: runUnsafearms a settle timer after a signalled statement'scancel(); a statement still pending cancelSettleMs() later (GBRAIN_CANCEL_SETTLE_MS, default 2000) has its reserved connection discarded so the awaiting caller settles with CONNECTION_DESTROYEDinstead of parking on a round-trip a transaction-mode pooler never completed (backend inClientRead). For excludePrivate callers searchKeywordruns its full-text match in anOFFSET 0 subquery (search/keyword-chunk-match.ts): inside the join the private-visibility subplans inflate every candidate plan's cost, so a cheaper bitmap scan fell within the planner's 1% fuzz of a serial seq scan that detoasts every chunk's search_vector(about 1.1 s for a common term at 250k chunks); local callers keep the unfenced, parallel-capable join. Rows, order and scores are pinned on both engines bytest/search/keyword-arm-ranking.test.ts(goldentest/fixtures/keyword-arm-ranking.json, which also pins one keyword statement per search and two when the strict query matches nothing and the OR fallback runs) and its Postgres twin test/e2e/keyword-arm-ranking-postgres.test.ts`.

  • src/core/search/connection-budget.ts — per-op pooled-connection budget for search and query. SEARCH_CONNECTION_LIMIT (4) is the most connections one op holds at once. withSearchConnectionBudget opens the scope; the ops' metered wrapper in src/core/ops/search.ts and dispatchToolCall (src/mcp/dispatch.ts, so pre-op reads and the response-meta hook count too) both enter it, and a nested call shares the enclosing budget. holdSearchConnection is called at the engine's connection seams, PostgresEngine#transactionOn (a top-level transaction, held until its COMMIT settles even when a read transaction answers early), executeRaw's JIT-off and statement-timeout envelopes (which reserve their own connection) and runUnsafe (one statement). A read whose signal aborts while it waits for a slot leaves the queue and rejects AbortError. Work inside a held slot runs outside the scope, so statements on a transaction's own connection never wait on a second slot. A read started under withUrgentSearchConnections (hybrid search's lexical, relational and vector arms, hybrid/arms.ts) takes the next free slot ahead of waiting ordinary reads (the keyword count and entity resolution an op starts early for its response; GBRA-156, test/search-connection-budget-urgent.test.ts). Outside a scope, and on PGLite, nothing waits. These reads use the ordinary read pool (the transaction pooler behind a split route), never the direct/session lane. Pinned on Postgres at the wire by test/e2e/search-connection-budget-postgres.test.ts. Size ceiling: scripts/module-size-limits.tsv (per-change raise history: git log -p on that file). The ceiling covers every BrainEngine method whose SQL has not moved to src/core/engine-sql/ (rows in scripts/engine-sql-baseline.tsv), written as parity twins of the PGLite methods (search arms, graph traversal caps, source-scoped stats and health, page lifecycle and soft delete, chunk writes, transaction routing). Migrated domains are one-line delegations over the private engineSql getter (a fresh adapter over this.sql on every access); reads that were RLS-scoped on master keep their own withScopedReadTransaction call and hand engine-sql scopedRead(this.engineSqlOn(tx)), an adapter over the scoped handle; runUnsafe forwards prepare/simple only when the engine-sql adapter sets them.

  • src/core/schema-client-behind.ts — #7776: assertSchemaNotAhead(query) runs on the schema-setup connection before both engines' schema replay: a config.version above LATEST_SCHEMA_VERSION refuses client_behind (fix gbrain upgrade, host_admin) before the bootstrap and replay, so an older binary never reinstalls its superseded function bodies over a newer schema. No config table or no version row (a fresh install) is not behind. apply-migrations refuses the same way before any orchestrator and before a --force-schema/--force-all schema pass, whose drift repairs would rewrite newer objects (schemaAheadRefusal, connectForForcedSchema in apply-migrations-schema.ts). Pinned by test/init-schema-client-behind.test.ts and its Postgres e2e twin.

  • src/core/snapshot-schema-inputs.ts — the inputs of the PGLite snapshot schema hash (computeSnapshotSchemaHash in pglite-engine.ts hashes each file's raw bytes). Computed, not hand-kept: the static import closure (import, export ... from; type-only imports skipped) of SNAPSHOT_SCHEMA_ROOTS (pglite-schema.ts, schema-migrations/registry.generated.ts, migrate.ts, engine-sql/bootstrap.ts) plus SNAPSHOT_SCHEMA_LEAVES; every literal dynamic import() inside the closure must be classified as a leaf or in SNAPSHOT_DYNAMIC_IMPORTS_NOT_HASHED. Pure (the caller supplies file access). test/snapshot-inputs-closure.test.ts names any missing file and checks every pglite-snapshot-* CI cache key covers the same inputs.

  • src/core/sql-trace.ts — env-gated wire-level SQL trace for benchmarks. traceSqlOptions(options, pool) returns the postgres.js options untouched unless GBRAIN_SQL_TRACE=<file.jsonl> is set; then it supplies a tracing socket and sets application_name to gbrain:<GBRAIN_SQL_TRACE_LABEL or command>:<pid>:<pool>. All four pool construction sites use it (db.ts module singleton, PostgresEngine instance pool, ConnectionManager read and direct pools), so executeRaw, tx.unsafe, the engine-sql adapter and tagged templates are all seen. Each Sync, simple Query or describe Flush and its reply is one JSONL record (statement text without parameter values, start, duration, process, pool, client connection, backend pid, SQLSTATE on error); a connect record covers TCP plus the startup handshake. TLS connections are recorded once as tls_untraced. Records are buffered and flushed every second and at exit. Consumed by scripts/bench/managed-sync-catchup.ts (record); pinned by test/sql-trace.test.ts and test/e2e/sql-trace-postgres.test.ts.

  • src/core/supabase-admin.ts — Supabase Management-API client (listProjects, discoverPoolerUrl, extractProjectRef — refs are lowercase ALPHANUMERIC, [a-z0-9]+). Consumed by the init --prefer-postgres ladder's rung 2 (DISCOVERY only — project creation stays dashboard guidance; discovered URLs are candidates, connect-probed before persisting) and by the classifier's Supabase hints. The access token is never persisted, logged, or written to receipts/config.

  • src/core/worker-pool.ts — (must-abort tags: BUDGET_EXHAUSTED and SPEND_GROUP_REFUSED, a job's spend-group refusal) Canonical sliding-pool + bounded-semaphore primitive (used by the src/commands/embed.ts sliding-pool sites and src/commands/eval-cross-modal.ts's runWithLimit semaphore). Two exports: runSlidingPool<T>({items, workers, onItem, signal?, onError?, failureLabel?, onProgress?}) + runWithLimit<TIn, TOut>({items, limit, fn, signal?}). Atomicity invariant: const idx = nextIdx++ is one synchronous JS statement (no await between read and write — guaranteed by the single-threaded event loop), documented in the module header AND enforced by scripts/check-worker-pool-atomicity.sh (wired into bun run verify), which rejects importing worker_threads in any consuming file and inserting await between the nextIdx read and write. MUST_ABORT_ERROR_TAGS set is seeded with BUDGET_EXHAUSTED from src/core/budget/budget-tracker.ts; tagged errors (matched via err.tag === 'BUDGET_EXHAUSTED' to avoid cross-module import) bypass onError and hard-abort the pool via AbortController.abort() to in-flight onItem, then the pool JOINS every worker (each finishes its current item) before rethrowing — the budget cap is a structural ceiling under concurrency and no worker is left running detached past the caller's catch. failures[] shape is {idx, label, error} records (NOT full items; callers supply failureLabel(item) => string) for bounded memory under huge brains. Pinned by test/worker-pool.test.ts + test/scripts/check-worker-pool-atomicity.test.ts. Drives every --workers N bulk command.

  • src/core/search-vector-backfill.ts — keyset-batched rebuilds of stored keyword vectors, shared by reindex-search-vector (every row) and the fts_cjk_boundary migration (an optional where limits it to rows with a CJK boundary). backfillChunkVectors recomputes content_chunks.search_vector with chunkSearchVectorSql; backfillPageVectors (#6374) reads each batch through an UPDATE-to-self of id (RETURNING id, slug, source_id, timeline, knowledge_revision plus whether a fact_withdrawals/fact_purges row covers the page or its whole source), takes the seal's effective timeline (the raw column, or for a withdrawn/purged page readPageSnapshot's overlaid timeline, so a database withdrawal never returns to the vector), sanitizes it with sanitizeRemoteBody as the seal does, and writes search_vector through pageSearchVectorSql in one UPDATE … FROM unnest(…) guarded by the revision read (a page edited in between is skipped; its seal reindexes it). Each statement scans at most BACKFILL_RANGE (50 000) ids past the cursor and rebuilds at most BACKFILL_BATCH_SIZE (5000) of them, never one transaction; the cursor advances to the range end even when nothing matched (so a filtered run on a brain with no CJK text never scans the whole table in one statement), and the run stops at the max(id) the first batch read; the cursor persists after every batch under backfill.<checkpoint>.last_id and the caller clears it. Pinned by test/reindex-search-vector.serial.test.ts, test/reindex-search-vector-pages.serial.test.ts, test/fts-cjk-boundary.test.ts, test/search-vector-backfill-withdrawal.test.ts, test/search-vector-backfill-range.test.ts. FTS write expressions (src/core/fts-language.ts, #6370): FTS_INPUT_FUNCTION_SQL (gbrain_fts_input(text), IMMUTABLE STRICT PARALLEL SAFE, search_path = pg_catalog: two regexp_replace passes built from CJK_SLUG_CHARS that insert a space at every CJK→ASCII-alnum and ASCII-alnum→CJK boundary; a byte-for-byte no-op without CJK, idempotent; spliced into src/schema.sql by build:schema), FTS_CJK_BOUNDARY_REGEX (the rows the fts_cjk_boundary migration rebuilds), pageSearchVectorSql (the sealed page vector used by the seal, the import seal and search-vector-backfill.ts), chunkSearchVectorSql, and pageSearchVectorTriggerFnSql/chunkSearchVectorTriggerFnSql (the two trigger functions under a language, byte-identical to src/schema.sql's for english so the blob replay after a migration leaves the catalog unchanged; used by reindex-search-vector and the migration). Every indexing to_tsvector reads through gbrain_fts_input; applyFtsLanguagePolicy's to_tsvector('english', prefix rewrite still matches. Pinned by test/fts-cjk-boundary.test.ts, test/fts-cjk-boundary-search.test.ts, test/e2e/fts-cjk-boundary-postgres.test.ts.

  • src/core/schema-migrations/v232-fts-cjk-boundary.ts — #6370, handler-only and idempotent: creates gbrain_fts_input, recreates both search_vector trigger functions under getFtsLanguage() with the shared builders, then rebuilds only pages (title/timeline) and chunks (chunk_text/doc_comment/symbol_name_qualified) matching FTS_CJK_BOUNDARY_REGEX through search-vector-backfill.ts (checkpoints fts_cjk_pages/fts_cjk_content_chunks, cleared at the end); every other row's vector is byte-identical under the helper and is not touched. Pinned by test/fts-cjk-boundary.test.ts and test/e2e/fts-cjk-boundary-postgres.test.ts.

  • src/core/schema-migrations/v245-page-search-vector-bounded-timeline.ts — Large pages, handler-only and idempotent: recreates the pages search_vector trigger function under getFtsLanguage() with the bounded timeline (PAGE_TIMELINE_FTS_MAX_CHARS) and entries (PAGE_TIMELINE_ENTRIES_FTS_MAX_CHARS) cuts, then rebuilds only pages whose timeline is longer than half the timeline cap through search-vector-backfill.ts (checkpoint page_vector_bounded_timeline, cleared at the end); shorter timelines index byte-identically and are not touched. Pinned by test/large-pages/page-timeline-tsvector-bound.test.ts and test/e2e/page-timeline-tsvector-bound-postgres.test.ts.

Continued in Engines (2/2, continued).