Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Name executeRaw's JIT-off and statement-timeout reads among the search connection budget's seams

garrytan28a6f9abb653
diff --git a/docs/architecture/key-files/engines-2.md b/docs/architecture/key-files/engines-2.md
index 470de90102..a3b88272da 100644
--- a/docs/architecture/key-files/engines-2.md
+++ b/docs/architecture/key-files/engines-2.md
@@ -48,7 +48,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/pool-gauge.ts` — `CheckoutGauge`: approximate in-flight counters at the engine's raw/direct/reserved/tx seams, surfaced via duck-typed `PostgresEngine.getPoolDiagnostics()` (no BrainEngine change, no PGLite stub). HONESTY CONTRACT in the module doc: tagged-template traffic is untracked; consumers must label counts as a subset and never derive waiter/available figures. Fail-open (clamped release, try/finally around sync-throwing runUnsafe). Consumed by db-probe.ts. Pinned by `test/pool-gauge.test.ts`. `PoisonedDiscardCounter` counts the driver's non-idle connection discards and writes one fixed-shape `pg_connection_poisoned` warn line each; the engine wires it into every pool it creates and reports it as `poisonedDiscards`. `onCheckout(listener)` (exposed as `PostgresEngine.onCheckout`) fires after a connection is actually obtained: signalled raw reserve resolution, transaction-callback entry and `withReservedConnection` reserve resolution; it never fires for a call still waiting (pinned by `test/e2e/postgres-checkout-observer.test.ts`). `formatBuildFailureWarning` appends `param=$n` (or `param=?` for an array element) to an `UNDEFINED_VALUE` line from the driver's `onbuilderror(code, statement, parameter)` (#6543).
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). The package-private `#postgres` import maps directly to the shipped cancellation-capable `vendor/postgres` source (the same source is embedded by compiled builds); `vendor/README.md` and the hash-verifying updater own provenance and regeneration. `postgres-engine/cancellation.ts` shares actual reserved-owner capability detection, typed `postgres_cancellation_unavailable` errors, and abort-bounded reservation with late-owner release; unsignalled diagnostic queries remain usable. Signalled SQL must retain cancellation/discard safety, including through transaction-mode poolers. The migrated storage domains' methods are one-line delegations into `src/core/engine-sql/<domain>.ts` over the private `engineSql` getter (a fresh Postgres adapter over `this.sql` on every access; RLS-scoped reads brand the `withScopedReadTransaction` handle with `scopedRead`), Postgres-only code stays under `src/core/postgres-engine/`, and the façade keeps its full public surface. `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the batch as one JSONB document — `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(...) JOIN pages ...` bound through `executeRawJsonb({ rows })` — which encodes arbitrary free text safely (an `unnest(${arr}::text[])` array-literal path would crash Postgres with "malformed array literal" on free text such as calendar/Zoom context) and sidesteps the 65535-parameter cap; takes declares native recordset column types (`page_id int, weight real, active boolean, …`) so no per-element casts; all three are `batchRetry`-wrapped. `disconnect()` runs `drainBackgroundWorkBeforeDisconnect()` before pool teardown (engine parity with PGLiteEngine — mode `'disconnect'`, so residual telemetry buffers drop symmetrically on both engines; guarded so a never-connected/already-torn-down engine skips the drain). `searchKeyword`/`searchVector` scope `statement_timeout` via `sql.begin` + a transaction-local `set_config` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection; the keyword-side bound (`searchKeyword`, `countKeywordPages`, `searchKeywordPages`, `searchTitles`, `searchKeywordChunks`, `_searchKeywordCJK`) is `search.keyword_statement_timeout_ms` (#6495; default 8000, range 1000-60000), resolved once per connection by `keywordStatementTimeoutSetting()` on the first keyword statement (`postgres-engine/keyword-statement-timeout.ts`: the `EngineConfig.keyword_statement_timeout_ms` value the engine connected with, else the DB plane's row read by one parameterless pool statement so no pg-types cache entry is described and `connect()` issues nothing for it, else the default; never a request parameter), with `countKeywordPages` keeping a caller's narrower `timeoutMs`. Every search transaction (keyword, keyword chunks, CJK, title, the vector attempts and the vector `hasMore` witness) also passes `jitOff` and runs with `jit = off` (`postgres-engine/search-settings.ts`): the correlated visibility subplans push these short reads past the JIT cost thresholds, and LLVM compilation, not execution, dominated their latency on large brains. `withScopedReadTransaction` applies every setting a search read needs (RLS `app.scopes`, `jit`, the caller's `settings`: `statement_timeout`, the relaxed arms' `enable_seqscan`, the vector scan settings from `vectorScanSettings`, a thunk read after BEGIN) in one `withSearchSettings` statement, `SELECT set_config('<name>', $n, true), …`, so a top-level read is BEGIN, settings, query, COMMIT; `statement_timeout` stays a literal name with a bound value so the vendored driver's #6543 budget reads it. Inside a caller's transaction the search runs in a savepoint, reads the caller's values in one statement and restores every one afterwards (a failed read rolls them back with the savepoint). A top-level read transaction opts in to `begin({ early: isPipelinableSearchRead })` (GBRA-156): its settings statement and its reads (single SELECT or WITH statements naming no write, lock, sequence or session setting) go out behind BEGIN before BEGIN's answer on any pool that pipelines inside a transaction, one round trip instead of two. Pinned by `test/search-settings-one-statement.test.ts` and `test/e2e/search-query-contract-postgres.test.ts`. `executeRaw(sql, params, { jitOff: true })` (`postgres-engine/jit-off-statement.ts`) gives single statements the same setting only when it pays: the server's `jit` is read once per pool, a JIT-off server runs the plain statement, and a JIT-on server runs `BEGIN; SET LOCAL jit = off`, the statement and `COMMIT` pipelined in the `runBoundedStatement` envelope (`postgres-engine/bounded-statement.ts`, which keeps `timeoutMs`/`deadlineAt` and the abort signal); inside a caller's transaction it reads, overrides and restores the caller's value. Pinned by `test/e2e/typed-search-jit-postgres.test.ts`. `getEmbeddingsByChunkIds` (in `src/core/engine-sql/chunks.ts`, both engines) uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude from `src/core/search/sql-ranking.ts`; `searchVector` is a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in outer SELECT) carrying `p.source_id` inner→outer. `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures, and by `batchRetry` on a retryable connection error). Concurrent callers share one in-flight `_reconnectPromise` (they await the single reconnect rather than racing a half-rebuilt pool); ownership re-samples through the atomic `db.connect()` token on the connect leg. `reconnect(ctx?)` accepts the triggering error and records a pool-recovery audit event (`reap_detected`/`reconnect_other`/`reconnect_succeeded`/`reconnect_failed`) for the `pool_reap_health` doctor check. `executeRaw` is a single-statement passthrough — no per-call retry (unsound for non-idempotent statements; recovery is supervisor-driven). `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. The chunk SQL below lives once in `src/core/engine-sql/chunks.ts`; the engine resolves the active column and source scope, owns the retry/transaction wrapper and the RLS scope transaction, and passes them in. `countStaleChunks()`+`listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale` (eliminates ~76 MB/call client-side pull); `upsertChunks()` routes text-embedding writes through the column registry (both engines in parity): a caller-resolved `opts.embeddingColumn` descriptor wins; otherwise the DB-plane `search_embedding_column` + `embedding_columns` config rows resolve via `resolveWriteColumnFromConfigRows` to the SAME active column + cast the read side searches (legacy `embedding`::vector on pre-registry brains; so a registry-routed brain never writes with a dimension mismatch against the legacy column). The INSERT column list, the `$N::vector|::halfvec(N)` cast, and every `embedding` reference in the ON CONFLICT CASE branches use the resolved column; it resets both the active column AND `embedded_at` to NULL when chunk_text changes without a new embedding. Pinned by `test/e2e/upsert-chunks-registry-column.test.ts` (PGLite always; Postgres DATABASE_URL-gated). `initSchema()` calls `applyForwardReferenceBootstrap(conn)` BEFORE replaying SCHEMA_SQL — a one-line delegation to the shared `src/core/engine-sql/bootstrap.ts` (Postgres dialect hooks: `current_schema()` probes plus the Postgres-blob-only `dream_verdicts.expires_at` probe); the entire probe path runs on the DDL connection threaded from `initSchema` (so concurrent bootstraps cannot race on a Supabase pooler). `initSchema` runs the bootstrap, the schema replay and the migrations under the schema migration lock (GBRA-467, `holdSchemaMigrationLock`, key 42), the same lock `runMigrations()` takes when `gbrain apply-migrations` calls it directly (`withSchemaMigrationLock`), so no two runners apply a migration at once; the runner reads the schema version once the lock is held and checks it is still held before each migration, and PGLite (one process per datastore) serializes in-process (`src/core/schema-migration-lock.ts`, `test/schema-migration-lock-pglite.test.ts`). The lock is `pg_try_advisory_xact_lock(42)` inside a READ COMMITTED transaction held open on a backend reserved from the DDL pool, or from the statement pool when the DDL pool has one connection (a short-lived command's idle direct connection would time out and drop it); it holds through a transaction-mode pooler and is released when its holder dies. Only when every pool has one connection does it take the session lock. A timed-out waiter is told not to terminate a holder tagged `gbrain-schema-migration-lock`. The bootstrap and replay run on a second reserved backend (checkout gauge only, no long-hold permit, so pending `transaction: false` migrations can still reserve); a one-slot pool runs them unreserved. Pinned by `test/e2e/postgres-initschema-advisory-lock.test.ts` `test/e2e/schema-migration-lock-postgres.test.ts` (direct and PgBouncer passes) and `test/e2e/schema-migration-lock-direct-pool-postgres.test.ts`. `disconnect()` is idempotent — `_connectionStyle` tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than clobbering the singleton; and a module-style engine only calls `db.disconnect()` when it owns the singleton (`_ownsModuleSingleton`, set from the `db.connect()` creation token), so a borrower probe engine's teardown leaves the cycle owner's connection intact. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` + `test/postgres-engine-singleton-lifecycle.test.ts` (concurrent-connect ownership, `db.disconnect()` detaching the singleton before awaiting `end()`, and the instance-reconnect `_reconnecting` guard). `getBrainScore` empty-brain parity with PGLite — 100/100 with breakdown 35/25/15/15/10 when `pageCount === 0` (both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic). Implements `deletePages(slugs, {sourceId}): Promise<string[]>` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single round-trip; caller chunks); `resolveSlugsByPaths` does `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2`; FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`, `files.page_id`+`links.origin_page_id` go SET NULL; throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both short-circuit on empty input. Implements the embedding-signature stale-detection quartet (`sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, widened `countStaleChunks`, all accept optional `signature` extending "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN, NULL grandfathered; the `embedding IS NULL` server-side filter is preserved as the no-signature fast path); Pinned by `test/e2e/engine-parity.test.ts`. Retry classifiers, ontology/recency helpers, and disconnect/pool-recovery audit writers bind statically; only the two `ai/gateway.ts` fallback lookups stay lazy and line-marked, in parity with PGLite. `insertFact` + `insertFacts` do not hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. `resolveFactsEmbeddingCast()` (private) probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). `withReservedConnection` routes long-hold reserved work (CREATE INDEX CONCURRENTLY, `transaction: false` migration DDL, backfill write batches) to the DIRECT session lane when dual-pool is active, so multi-minute holds do not pin the worker's shared read pool — never rerouted inside an open transaction (same guard shape as `executeRawDirect`), semaphore-capped below `direct_pool_size` with deliberately NO minimum-1 floor (at `direct_pool_size=1` a floor would let one long reserve consume the only direct session and starve the claim/renewLock heartbeats — the same starvation on the direct lane), overflowing to the shared read pool when the direct lane has no spare capacity or is unavailable; the permit is released on both fn throw and reserve failure. `getPoolDiagnostics()` (duck-typed, no BrainEngine change) surfaces the `CheckoutGauge` in-flight counters from `src/core/pool-gauge.ts` at the raw/direct/reserved/tx seams — a tracked SUBSET (tagged-template traffic is untracked) that `db-probe.ts` labels honestly in health-probe failure lines. Pinned by `test/postgres-engine-reserved-routing.test.ts`. `getRawData` joins `pages` and adds `p.deleted_at IS NULL` unless `opts.includeDeleted` (the `alive` fragment next to `privacy` in all six branches) so raw session metadata follows the page soft-delete; PGLite mirrors it with one `where.push`; pinned by the standalone getRawData describe in `test/e2e/engine-parity.test.ts` (PGLite arm always runs; Postgres arm when DATABASE_URL is set). `includeDeleted` is NOT on the shared `PageReadScope` (only getRawData honors it) — it is declared per-method as `getRawData(slug, source?, opts?: PageReadScope & { includeDeleted?: boolean })` in engine.ts and both engines; resolved only by trusted callers (export, engine migration; the ingest probe passes it but only ever runs against a live page), never from MCP parameters (`get_raw_data` builds its scope from `readPolicyOpts(ctx)`). #6278 PR 1.5: `runUnsafe` arms a settle timer after a signalled statement's `cancel()`; a statement still pending `cancelSettleMs()` later (`GBRAIN_CANCEL_SETTLE_MS`, default 2000) has its reserved connection discarded so the awaiting caller settles with `CONNECTION_DESTROYED` instead of parking on a round-trip a transaction-mode pooler never completed (backend in `ClientRead`). For excludePrivate callers `searchKeyword` runs its full-text match in an `OFFSET 0` subquery (`search/keyword-chunk-match.ts`): inside the join the private-visibility subplans inflate every candidate plan's cost, so a cheaper bitmap scan fell within the planner's 1% fuzz of a serial seq scan that detoasts every chunk's `search_vector` (about 1.1 s for a common term at 250k chunks); local callers keep the unfenced, parallel-capable join. Rows, order and scores are pinned on both engines by `test/search/keyword-arm-ranking.test.ts` (golden `test/fixtures/keyword-arm-ranking.json`, which also pins one keyword statement per search and two when the strict query matches nothing and the OR fallback runs) and its Postgres twin `test/e2e/keyword-arm-ranking-postgres.test.ts`.
-- `src/core/search/connection-budget.ts` — per-op pooled-connection budget for `search` and `query`. `SEARCH_CONNECTION_LIMIT` (4) is the most connections one op holds at once. `withSearchConnectionBudget` opens the scope; the ops' `metered` wrapper in `src/core/ops/search.ts` and `dispatchToolCall` (`src/mcp/dispatch.ts`, so pre-op reads and the response-meta hook count too) both enter it, and a nested call shares the enclosing budget. `holdSearchConnection` is called at the engine's two connection seams, `PostgresEngine#transactionOn` (a top-level transaction, held until its COMMIT settles even when a read transaction answers early) and `runUnsafe` (one statement). Work inside a held slot runs outside the scope, so statements on a transaction's own connection never wait on a second slot. A read started under `withUrgentSearchConnections` (hybrid search's lexical, relational and vector arms, `hybrid/arms.ts`) takes the next free slot ahead of waiting ordinary reads (the keyword count and entity resolution an op starts early for its response; GBRA-156, `test/search-connection-budget-urgent.test.ts`). Outside a scope, and on PGLite, nothing waits. These reads use the ordinary read pool (the transaction pooler behind a split route), never the direct/session lane. Pinned on Postgres at the wire by `test/e2e/search-connection-budget-postgres.test.ts`.
+- `src/core/search/connection-budget.ts` — per-op pooled-connection budget for `search` and `query`. `SEARCH_CONNECTION_LIMIT` (4) is the most connections one op holds at once. `withSearchConnectionBudget` opens the scope; the ops' `metered` wrapper in `src/core/ops/search.ts` and `dispatchToolCall` (`src/mcp/dispatch.ts`, so pre-op reads and the response-meta hook count too) both enter it, and a nested call shares the enclosing budget. `holdSearchConnection` is called at the engine's connection seams, `PostgresEngine#transactionOn` (a top-level transaction, held until its COMMIT settles even when a read transaction answers early), `executeRaw`'s JIT-off and statement-timeout envelopes (which reserve their own connection) and `runUnsafe` (one statement). A read whose `signal` aborts while it waits for a slot leaves the queue and rejects `AbortError`. Work inside a held slot runs outside the scope, so statements on a transaction's own connection never wait on a second slot. A read started under `withUrgentSearchConnections` (hybrid search's lexical, relational and vector arms, `hybrid/arms.ts`) takes the next free slot ahead of waiting ordinary reads (the keyword count and entity resolution an op starts early for its response; GBRA-156, `test/search-connection-budget-urgent.test.ts`). Outside a scope, and on PGLite, nothing waits. These reads use the ordinary read pool (the transaction pooler behind a split route), never the direct/session lane. Pinned on Postgres at the wire by `test/e2e/search-connection-budget-postgres.test.ts`.
Size ceiling: `scripts/module-size-limits.tsv` (per-change raise history: `git log -p` on that file). The ceiling covers every BrainEngine method whose SQL has not moved to `src/core/engine-sql/` (rows in `scripts/engine-sql-baseline.tsv`), written as parity twins of the PGLite methods (search arms, graph traversal caps, source-scoped stats and health, page lifecycle and soft delete, chunk writes, transaction routing). Migrated domains are one-line delegations over the private `engineSql` getter (a fresh adapter over `this.sql` on every access); reads that were RLS-scoped on master keep their own `withScopedReadTransaction` call and hand engine-sql `scopedRead(this.engineSqlOn(tx))`, an adapter over the scoped handle; `runUnsafe` forwards `prepare`/`simple` only when the engine-sql adapter sets them.
- `src/core/schema-client-behind.ts` — #7776: `assertSchemaNotAhead(query)` runs on the schema-setup connection before both engines' schema replay: a `config.version` above `LATEST_SCHEMA_VERSION` refuses `client_behind` (fix `gbrain upgrade`, `host_admin`) before the bootstrap and replay, so an older binary never reinstalls its superseded function bodies over a newer schema. No config table or no version row (a fresh install) is not behind. `apply-migrations` refuses the same way before any orchestrator and before a `--force-schema`/`--force-all` schema pass, whose drift repairs would rewrite newer objects (`schemaAheadRefusal`, `connectForForcedSchema` in `apply-migrations-schema.ts`). Pinned by `test/init-schema-client-behind.test.ts` and its Postgres e2e twin.