Garry's Opinionated OpenClaw/Hermes Agent Brain
BrainEngine map
This map lists every member of the BrainEngine interface in gbrain (TypeScript), grouped by family. For each member it gives the signature, what it does, the SQL it runs, where PGLiteEngine and PostgresEngine differ, and the tests that cover it.
- Reference:
garrytan/gbrainat commit8bbfd0b5e8(master, 2026-11-16). Everypath:linecitation is relative to the gbrain repo root at that commit. - Interface:
src/core/engine.ts:830-2890. Engines:src/core/pglite-engine.ts(3,088 lines),src/core/postgres-engine.ts(3,182 lines), plussrc/core/pglite-engine/*andsrc/core/postgres-engine/*. Shared SQL:src/core/engine-sql/*.ts(about 15,000 lines with the engines). - Method: each family was read by a separate research pass. Counts are per declaration. Line numbers were read at the commit above. Test coverage was found by grep, not by running the suite. Items a pass could not verify say so in place.
Confirmed build direction: the first build target is a Rust storage engine that implements BrainEngine as a third engine inside gbrain, next to PGLite and Postgres (option B, confirmed by Garry). The binding is still open; a napi-rs native module is the leading option. The last two sections of this map serve that target:
- What a Rust engine must do to pass engine-parity has the per-method parity notes for
test/e2e/engine-parity.test.ts. - Adding a third engine and binding options covers engine selection and napi-rs vs sidecar vs WASM vs
bun:ffifor Bun.
Contents
| Section | Interface lines | Members |
|---|---|---|
| Lifecycle, Pages and Search | 830-1238 | 41 (incl. the kind property) |
| Chunks, Extraction staleness and Links/Graph | 1240-1841 | 45 |
| Tags, Timeline, Chronicle, Ontology, Raw data, Files, Takes, Dream and Contradictions | 1842-2195 | 43 |
| Facts, Segmented storage, Versions, Stats, Sync, Config and Migration support | 2196-2890 | 70 |
| What a Rust engine must do to pass engine-parity | test file | 60 covered, 137 not covered |
| Adding a third engine and binding options | engine selection | n/a |
The four family sections count 199 declarations. The parity section counts 197 callable members; the difference is the kind property and how overloads were counted.
Headline findings
- The SQL is mostly shared. Migrated storage domains keep their SQL once in
src/core/engine-sql/and both engines delegate to it. Engine differences are in the wrappers: Postgres adds statement timeouts, JIT-off reads, reserved connections, read-only transactions, real cancellation and an exact-scan vector fallback. PGLite adds planner-stat refreshes and a single-process lock. - Some engine differences are visible to callers.
searchKeywordChunksignorestype,types,whereandexclude_slugson PGLite. Thestaleflag on keyword results is computed on PGLite and a constantfalseon Postgres. A Rust engine has to pick one behavior per method; the Postgres behavior is the safer reference. - Several interface doc comments disagree with the code (
addLinksBatch,getEffectiveDates,createVersion,setPageAliases,updatePageContextualRetrievalState,insertFacts). A Rust engine should follow the code and fix the doc first. test/e2e/engine-parity.test.tscovers only 60 of 197 members and needsDATABASE_URL. 46 of its calls are raw Postgres SQL throughexecuteRaw. A Rust engine cannot pass it unchanged; see the parity section.
Lifecycle, Pages and Search
Scope: BrainEngine interface members in src/core/engine.ts:830-1238 (interface opens at :830, the kind discriminator is at :832, and the range ends before the Search-adjacent searchKeywordChunks, which is not on the interface). Engines: src/core/pglite-engine.ts (PGLite, 3088 lines) and src/core/postgres-engine.ts (Postgres, 3182 lines). Domain SQL lives in src/core/engine-sql/*.ts and is shared by both engines.
Member count: 41 interface members in scope (1 property, 40 methods). Optional members: holdSchemaMigrationLock? (engine.ts:848), findDuplicatePage? (:922), findDuplicatePages? (:926). Every other member is required. Note initSchema() is declared with no arguments (:846), but Postgres takes an optional { embedding? } (postgres-engine.ts:576).
Plumbing used by these methods
| File | Role | Key lines |
|---|---|---|
src/core/engine-factory.ts |
createEngine(config) calls constructEngine, registers the chat usage sink for the engine, and wraps engine.disconnect so the sink is deregistered before the original disconnect. Every production engine passes through here. |
:10-31 (wrap at :24-28) |
src/core/engine-sql/executor.ts |
SqlExecutor contract: dialect, capabilities, query(sql, params), run(fragment), executeRaw (rows only), unsafe (raw driver, no gauge), transaction(fn). Results are { rows, affectedRows } (EO18). Errors pass through unchanged. DialectCapabilities: maxBindParamsPerStatement (PGLite 30,000, Postgres Infinity), transactionAdvisoryLocks (Postgres true), probesEmbeddingCast (Postgres true). |
interface :110-123; capabilities :87-108; jsonbParam :133 |
src/core/engine-sql/dialect-pglite.ts |
Wraps one PGLite handle. pgliteQuery pre-checks the abort signal, then races the statement against abort. The statement itself is not cancelled (WASM has no cancel). PGLITE_CAPABILITIES at :25-29. transaction calls db.transaction and returns a new executor over the tx handle. |
:42-56 query; :58-76 executor; :71-74 transaction |
src/core/engine-sql/dialect-postgres.ts |
Wraps one postgres.js handle (pool, reserved conn or tx). query/run go through runUnsafe with prepare: true, simple: false (named prepared statements; PgBouncer prepare:false still honored). executeRaw bumps the raw checkout gauge. unsafe is a bare conn.unsafe. transaction uses readWriteTransactions(conn).begin. POSTGRES_CAPABILITIES at :129-133. jsonbParam binds via conn.json (:137). |
:144-170; query :145-146; executeRaw :155-162; unsafe :163-165; transaction :166-168 |
src/core/engine-sql/fragment.ts |
sqlFragment tagged template builds {strings, values}. Nested fragments splice, trustedSql(text) splices constant identifier or ORDER BY text, anything else becomes a positional parameter. renderFragment numbers $1..$n only at render time. joinFragments for lists. No identifier escaping: identifiers come from constant allowlists. |
sqlFragment :46; trustedSql :75; joinFragments :80; renderFragment :94 |
src/core/engine-sql/brands.ts |
Read brands. ScopedRead (:174) is an executor branded as running inside withScopedReadTransaction on Postgres (RLS app.scopes bound when enabled); scopedRead() (:184) is an identity cast used only on the handle that helper yields (PGLite brands its own executor). LegacyUnscopedRead (:177) marks reads that ran unscoped on master (EO4 inventory), created by unscopedExecutor(exec, reason) (:193). Brand keys are checked by scripts/check-engine-sql-brands.ts. |
:174-196 |
src/core/engine-sql/normalize.ts |
compileRowNormalizer(kinds) converts only declared columns. Kinds: jsonb (parse text), bigint (Number), date (Date), vector (parseEmbedding), text[]. Used e.g. by the domain-bank row mapper (last_retrieved_at: 'date') in engine-sql/pages.ts. |
ColumnKind :212; compile :227-236 |
Engine-side wiring for all methods below: each engine exposes an engineSql getter that builds a fresh executor over the current handle on every access (never cached), so transaction() clones (created via Object.create(this)) run domain SQL inside the transaction. Postgres: postgres-engine.ts:317-325. PGLite: pglite-engine.ts:794 engineSql getter.
Lifecycle
| Member (engine.ts) | Signature | What it does | PGLite (pglite-engine.ts) |
Postgres (postgres-engine.ts) |
|---|---|---|---|---|
kind (:832) |
readonly kind: 'postgres' | 'pglite' |
Discriminator for migrations and consumers. | kind = 'pglite' as const (:739) |
kind = 'postgres' as const (:226) |
connect (:835) |
connect(config: EngineConfig): Promise<void> |
Opens the datastore. | connect :799 → _connectWithRootRegistration (registers managed filesystem engine) → _connectInternal (:887). Takes a kernel file lock (acquireLock(dataDir), pglite-lock.ts) for file-backed DBs, then PGlite.create({dataDir}), configures codecs. In-memory: optional GBRAIN_PGLITE_SNAPSHOT restore (skips schema replay). Wasm-abort on a persistent dir triggers attemptWalRepairAndRetry (in-place WAL repair, with consent-gated refusal). Stores _savedConfig for reconnect. |
connect :433. Two modes: (a) instance pool when config.poolSize is set: own postgres(url) with max from resolvePoolSize, idle_timeout 20, session timeouts (statement_timeout, idle_in_transaction_session_timeout), prepare per URL, ConnectionManager wrapper; (b) module singleton via db.connect (db.ts), shared with other engines. types: { bigint: postgres.BigInt } on instance pools. |
disconnect (:836) |
disconnect(): Promise<void> |
Tears the datastore down. Idempotent (memoized call). | disconnect :1050. Sets requested flag, awaits pending connect, then _closeInternal (:1088): runs the _beforeDisconnect barrier, sets _db = null, stopAndDrain on DB work, drain background work, checkpoint (failure warns and retains ownership), db.close(), release kernel lock. Close watchdog arms SIGTERM/SIGKILL timers. A failed close poisons the engine (_closePoison) and retains the lock. |
disconnect :516 → disconnectInternal (:526): runs _beforeDisconnect stops, logs disconnect audit, drains background work, closes ConnectionManager (direct pool), db.endPoolBounded(this._sql) (bounded wait, #1972). Module-singleton teardown only by the engine that created it (_ownsModuleSingleton). |
reconnect (:845) |
reconnect(ctx?: { error?: unknown }): Promise<void> |
Recovers a dropped connection from the config captured at connect(). Callers must use it instead of disconnect + bare connect (#2034). |
reconnect :1139: no-op if never connected or in-memory (database_path unset, to preserve state). File-backed: disconnect() then connect(savedConfig). |
reconnect :2859 (far from the other lifecycle methods). Instance pool: build-then-swap (new pool proven live before old is ended); _reconnecting guard; ctx.error classified by isConnectionEndedError to audit reap_detected vs reconnect_other (logPoolRecovery). Module singleton: db.connect(savedConfig) then refresh ConnectionManager read pool. |
initSchema (:846) |
initSchema(): Promise<void> (Postgres: opts?: {embedding?}) |
Creates or replays the schema and applies migrations. | initSchema :1147. Skipped entirely if a snapshot was loaded. Dims/model from embedded settings, gateway, or defaults; stored embedding identity wins. Runs under withSchemaMigrationLock (in-process serialization, since no holdSchemaMigrationLock): assertSchemaNotAhead (#7776), applyForwardReferenceBootstrap, db.exec(getPGLiteSchema(dims, model)), runMigrations(this), then registerManagedFilesystemEngine. No verifySchema or zombie-index sweep. |
initSchema :576. Pool = connectionManager.ddl() when dual-pool, else this.sql. Dims/model as PGLite. Runs under withSchemaMigrationLock (:619): assertSchemaNotAhead (:620), applyForwardReferenceBootstrap(conn), retrySchemaReplayOnDeadlock(withSchemaLockTimeout(conn, conn.unsafe(schemaSql))) (:642; session lock_timeout on a reserved backend, #5227), runMigrations, verifySchema(this) (self-heals missing columns), dropZombieIndexes(this) (best-effort, invalid HNSW indexes). Clears shared types if anything changed. |
holdSchemaMigrationLock? (:848) |
holdSchemaMigrationLock?<T>(fn: (assertHeld) => Promise<T>): Promise<T> |
GBRA-467 cross-process lock (key 42). Optional: when absent, schema-migration-lock.ts:22-30 serializes in-process only. |
Not implemented. The schema-migration-lock.ts wrapper serializes within the process (PGLite datastore lock admits one process). |
holdSchemaMigrationLock :687. Uses the DDL pool if max>1, else the statement pool if max>1, else a session lock (holdSchemaMigrationSessionLock). Reserved-connection form (holdSchemaMigrationTransactionLock) polls pg_try_advisory_xact_lock(42) (init-schema-lock.ts:189-203). Session form uses pg_advisory_lock(42) (init-schema-lock.ts:130-137) and pg_advisory_unlock (:237). |
transaction (:849) |
transaction<T>(fn: (engine) => Promise<T>, opts?: TransactionOptions): Promise<T> |
Runs fn on a clone engine bound to a transaction; nested calls become savepoints. |
transaction :1209. Checks only opts.signal (abort before start). Uses db.transaction, runs under _dbWork.admit + PgliteCheckpointGuard.runOutermost (checkpoint blocked during tx), composable tx handle, withHeldPageKeys. Ignores readOnly and pipelineEarly. A nested call on a tx clone runs inside the same tx. |
transaction :716 → transactionOn (:726). readOnly opens BEGIN READ ONLY (not wrapped by readWriteTransactions). checkoutGauge.acquire('tx') (top-level only). Options pipelineEarly, pipelineFirst, signal (abort via withTransactionAbort), commit checks (withCommitChecks), held page keys, holdSearchConnection. Clone sets sql to the tx handle; nested call uses conn.begin(run) (savepoint). |
transactionDirect (:851) |
transactionDirect<T>(fn, opts?) |
Short control transaction on the direct route. Honors nesting. | transactionDirect :1225: return this.transaction(fn, opts) (no direct route exists). |
transactionDirect :720: when dual-pool and top-level, uses connectionManager.ddl() (direct pool), else this.sql; then transactionOn. |
registerBeforeDisconnect (:853) |
registerBeforeDisconnect(stop: () => Promise<void>): () => void |
Registers a mandatory resident-consumer stop barrier that runs before datastore or pool teardown. Returns an unregister function. | registerBeforeDisconnect :753. Adds to _beforeDisconnect Set; barrier runs at the start of _closeInternal, before _db = null. |
registerBeforeDisconnect :230. Same Set. Barrier runs first in disconnectInternal (:524). |
withReservedConnection (:859) |
withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>, opts?: { route?: 'ordinary'; selfContained?: boolean }): Promise<T> |
Runs fn on one dedicated backend. Release automatic. route: 'ordinary' skips the direct pool. selfContained lets a one-connection pool lend the connection. |
withReservedConnection :1196 (opts not accepted). No pool: passes the single db through. conn.executeRaw only; no transaction member. |
withReservedConnection :763. Picks the direct DDL pool when dual-pool and not route:'ordinary', guarded by tryAcquirePoolLongHold (permit; throws PoolCapacityError if none). Gauge reserved acquired before pool.reserve(). conn.executeRaw uses reserved.unsafe (AbortSignal ignored). conn.transaction = transactionOn(reservedTransactions(reserved)). Release in finally (double-release tolerant). |
Pages CRUD
Shared SQL for most methods is in src/core/engine-sql/pages.ts (both engines call pagesImpl.* with this.engineSql). Differences are in how the read/write is scoped and wrapped (noted per row).
| Member (engine.ts) | Signature (compact) | What it does and SQL | Engine differences |
|---|---|---|---|
getPage (:869) |
getPage(slug, opts?: GetPageOpts): Promise<Page | null> |
Returns readPageSnapshot(...).page, or null. Soft-deleted rows return null unless includeDeleted. |
PGLite :1230 calls readPageSnapshot directly. Postgres :850 same contract. |
readPageSnapshot (:870) |
readPageSnapshot(slug, opts?: PageSnapshotOptions): Promise<PageSnapshot | null> |
Canonical page snapshot (src/core/page-state/snapshot.ts:66-120): one page row by slug. With resolveAlias, falls through slug_aliases to the canonical page. Ordering (p.slug = $1) DESC, (p.source_id = $n) DESC, p.source_id ASC LIMIT 1. Filters: deleted_at unless includeDeleted; requireLiveSource (sources not archived); requireUnambiguous (count(*) OVER ()). Joins sources.incarnation, tags (jsonb_agg), fact-withdrawal and fact-purge aggregates (fact_purges). |
PGLite :1234 executeRaw directly, no scope. Postgres :854 runs inside withScopedReadTransaction(sourceIds, sourceId) with prepare: true; it uses JIT off and an alwaysTransaction wrap only when resolveAlias && excludePrivate (#6276). |
readPageSnapshotsBatch (:875) |
readPageSnapshotsBatch(refs: {slug, sourceId}[], opts?: {maxBytes?, includeDeleted?, absentIsNull?}): Promise<PageSnapshotBatch> |
Exact-ref batch (src/core/page-snapshot-batch.ts:42-60). VALUES-style wanted list with cumulative bytes (SUM ... ROWS UNBOUNDED PRECEDING AND 1 PRECEDING) and returns the longest prefix whose bodies fit maxBytes. Same snapshot joins as readPageSnapshot. |
PGLite :1238 executeRaw. Postgres :861 binds the batch's distinct sourceIds for RLS through withScopedReadTransaction. |
lockPageKeys (:877) |
lockPageKeys(keys: readonly PageKey[]): Promise<void> |
Holds exact page identities (including absent rows) through commit. Throws unless called on a transaction engine. Takes only keys not already held (sorted acquisition via lockUnheldPageKeys, page-state/guards.ts:114-119). |
Both engines identical in shape: PGLite :1242 (lockUnheldPageKeys), Postgres :867 (lockUnheldPageKeys). |
putPage (:891) |
putPage(slug, page: PageInput, opts?: PageWriteOptions): Promise<Page> |
Validates slug, runs in transaction(write) (or inline when already in a page tx and opts.inline). In order: lockPageKeys; revision precondition check when expectedRevision/force set (readPageSnapshot includeDeleted); assertOriginUnclaimed(revive:true) (persistence/origin-guard.ts:112); then pages.ts:103-204 INSERT ... ON CONFLICT (source_id, slug) DO UPDATE with deleted_at = NULL (revival), COALESCE preservation for effective_date, import_filename, source_path, source_kind, source_uri, ingested_via, ingested_at, contextual retrieval columns when provided, chunker_version via bodyWriteChunkVersion, RETURNING ... . Blank-body guard: if the incoming body is blank and not allowEmptyOverwrite, reads the prior row and refuses to blank a non-empty or segmented page (pages.ts:116-133). |
PGLite :1262 passes a reread callback to pagesImpl.putPage for the PGLite edge case where RETURNING yields zero rows. Postgres :887 passes none, and maps origin unique violations through classifyOriginUniqueViolation. |
findDuplicatePage? (:922) |
findDuplicatePage?(sourceId, opts: {hash, frontmatterId?, excludeSlug?}): Promise<{slug,id} | null> |
Import dedup pre-check (#1309). pages.ts:57-80: WHERE source_id = $ AND deleted_at IS NULL AND (content_hash = $hash OR (frontmatter->>'id' = $fmId AND $fmId IS NOT NULL)), excluding excludeSlug. Orders frontmatter-id match first, then id. LIMIT 1. |
PGLite :1252 uses scopedRead(this.engineSql) with no scope transaction. Postgres :877 runs in withScopedReadTransaction(undefined, sourceId). |
findDuplicatePages? (:926) |
findDuplicatePages?(sourceId, inputs[]): Promise<Array<{slug,id}|null>> |
Batched version (GBRA-75 w10). pages.ts:82-101: jsonb_to_recordset of inputs joined to pages, DISTINCT ON (k.n), same predicate and ordering. Returns one entry per input, in order. |
PGLite :1258; Postgres :883. Same SQL. |
deletePage (:952) |
deletePage(slug, opts?: {sourceId?}): Promise<void> |
Hard delete. pages.ts:206-208: DELETE FROM pages WHERE slug = $ AND source_id = $ (source defaults to 'default'). Cascades via FKs only. No scrub of name/feedback stores. Used by purge, sync per-slug fallback, and test teardown. |
PGLite :1276 and Postgres :901 both call pagesImpl.deletePage(this.engineSql, ...). |
deletePages (:989) |
deletePages(slugs: string[], opts: {sourceId}): Promise<string[]> |
Batch hard delete, one transaction. pages.ts:221-247: DELETE ... WHERE slug = ANY($::text[]) AND source_id = $ RETURNING slug, then scrubs by slug: scrubPurgedSlugs, scrubPurgedNameStores, scrubPurgedSlugStores, scrubPurgedSlugHolds. Throws above DELETE_BATCH_SIZE (500, engine-constants.ts:25). Empty input returns []. |
Same SQL on both engines via this.engineSql (PGLite :1286, Postgres :911). |
resolveSlugsByPaths (:1007) |
resolveSlugsByPaths(paths: string[], opts: {sourceId}): Promise<Map<string,string>> |
Path to slug map. pages.ts:249-274: SELECT slug, source_path, deleted_at IS NULL AS live ... WHERE source_path = ANY($) AND source_id = $. Live rows win over tombstones; several live rows resolve via pickCanonicalOrigin; ambiguous paths are omitted. Throws above 500 items. |
Both use unscopedExecutor(...) (LegacyUnscopedRead, "unscoped on master"): PGLite :1294, Postgres :919. |
softDeletePage (:1017) |
softDeletePage(slug, opts?: {sourceId?}): Promise<{slug} | null> |
pages.ts:276-295: UPDATE pages SET deleted_at = now() WHERE slug = $ AND deleted_at IS NULL [AND source_id = $] RETURNING slug. Null if nothing flipped. No cascade. Note: sourceId has no 'default' fallback; when omitted, the predicate matches the slug in every source. |
PGLite :1301, Postgres :926 (both pagesImpl.softDeletePage(this.engineSql, ...)). |
softDeletePages (:1050) |
softDeletePages(slugs, opts: {sourceId}): Promise<string[]> |
Batch soft delete (#4587). pages.ts:297-310: UPDATE ... SET deleted_at = now() WHERE slug = ANY(...) AND source_id = $ AND deleted_at IS NULL RETURNING slug. Empty returns []; over 500 throws. Re-runs never refresh deleted_at (72h clock). |
PGLite :1314, Postgres :937. Same SQL. |
restorePage (:1056) |
restorePage(slug, opts?: {sourceId?}): Promise<boolean> |
pages.ts:312-327: one CTE statement. UPDATE pages SET deleted_at = NULL, updated_at = now() WHERE slug = $ AND deleted_at IS NOT NULL [AND source_id], plus DELETE FROM op_checkpoints for the owed-purge record (GST-2766). Runs inside transaction after assertRevivalUnclaimed (persistence/origin-guard.ts:156). |
PGLite :1318, Postgres :941 (same wrapping). |
purgeDeletedPages (:1071) |
purgeDeletedPages(olderThanHours, opts?: {dryRun?, excludeIds?}): Promise<{slugs, count, pages?}> |
Hard purge after the soft-delete window (pages.ts:329-403). Hours floored, clamped at 0. Dry run: same WHERE predicate as a SELECT, returns pages. Real run, one transaction: collects shas from page_body_manifest; DELETE FROM pages WHERE deleted_at IS NOT NULL AND deleted_at < now() - interval excluding rows under holds (identity-downgrade checkpoint, atomRetirementPurgeHold, derivedRetirementPurgeHold, derivedOriginPurgeHold) and excludeIds; then expireGonePageFacts, deletes page_body_stagings rows, deletes orphaned page_segment_texts, and scrubs per source (purged slugs, name stores, slug stores, slug holds). |
PGLite :1322 and Postgres :945 both use this.engineSql. Same SQL; PG runs it on the pool or a tx clone. |
listPages (:1079) |
listPages(filters?: PageFilters): Promise<Page[]> |
pages.ts:484-561. Filters: type, tag (JOIN tags), updated_after or keyset (updated_at, slug), slugPrefix (LIKE with escaped metachars), excludeSlugPrefixes (starts_with), slugAfter, sourceIds/sourceId, includeDeleted (default hides), excludePrivate (privatePagesFilterForCaller), effective_after/effective_before, frontmatter where, sort via PAGE_SORT_SQL allowlist, listColumnsOnly. LIMIT default 100. Projects updated_at_iso with microseconds for keyset resume. |
PGLite :1377 wraps in plannerRead (runs beforePlannerRead stats settle). Postgres :1009 wraps in withScopedReadTransaction(filters.sourceIds, filters.sourceId). |
findPages (:1081) |
findPages(query: FindPagesQuery): Promise<FoundPages> |
Live pages whose slug, source_path or title matches a find pattern (engine-sql/page-lookup.ts:43). |
PGLite :1380 via plannerRead; Postgres :1012 via withScopedReadTransaction. |
countPagesBy (:1083) |
countPagesBy(query: PageCountQuery, groupTypes?: TypeGrouper): Promise<PageCounts> |
Live page counts per (source, type). With groupTypes, distinct-slug totals per type group (page-lookup.ts:72). |
PGLite :1383 via plannerRead; Postgres :1015 via withScopedReadTransaction. |
resolveSlugs (:1097) |
resolveSlugs(partial: string, opts?: PageReadScope): Promise<string[]> |
Fuzzy slug resolver. pages.ts:752-~800: exact slug match first (deleted_at IS NULL + scope + privacy). Otherwise up to 5 rows: WHERE deleted_at IS NULL AND (title % $partial OR slug ILIKE '%partial%'), ORDER BY similarity(title, $partial) DESC LIMIT 5 (pg_trgm). Scope: sourceIds array wins over sourceId; neither = unscoped (back-compat). |
Both use unscopedExecutor(this.engineSql, ...): PGLite :1415, Postgres :1056 (direct, no scope tx). Needs pg_trgm on both (PGLite ships its own extension set). |
getAllSlugs (:1108) |
getAllSlugs(opts?: {sourceId?}): Promise<Set<string>> |
pages.ts:563-572: SELECT slug FROM pages or ... WHERE source_id = $. No deleted_at filter, so soft-deleted slugs are included. |
PGLite :1387 scopedRead(this.engineSql) direct; Postgres :1019 via withScopedReadTransaction(undefined, opts.sourceId). |
listAllPageRefs (:1127) |
listAllPageRefs(): Promise<Array<{slug, source_id, updated_at: Date}>> |
pages.ts:574-589: SELECT slug, source_id, updated_at FROM pages WHERE deleted_at IS NULL ORDER BY source_id, slug. Cross-source, unscoped. |
Both unscopedExecutor(this.engineSql, ...): PGLite :1391, Postgres :1023. |
listAllSources (:1145) |
listAllSources(opts?: {includeArchived?, localPathOnly?}): Promise<SourceRow[]> |
engine-sql/sources.ts:14-41: SELECT id, name, local_path, last_sync_at, config FROM sources WHERE (includeArchived OR archived IS NOT TRUE) AND (!localPathOnly OR (local_path IS NOT NULL AND config->>'role' IS DISTINCT FROM 'storage_manifest')) ORDER BY (id='default') DESC, id. Config parsed if text. |
Both unscopedExecutor(...): PGLite :1396, Postgres :1028 (sources table is not RLS-scoped in master). |
updateSourceConfig (:1165) |
updateSourceConfig(sourceId, patch: Record<string,unknown>): Promise<boolean> |
Atomic JSONB merge (engine-sql/sources.ts:43-81): `UPDATE sources SET config = |
|
listPrefixSampledPages (:1184) |
listPrefixSampledPages(opts: DomainBankSampleOpts): Promise<DomainBankRow[]> |
pages.ts:619-~700. CTE prefix_pages: prefix = substring(slug from '^[^/]+/[^/]+'), connection_count = COUNT(page_links.to_page_id), last_retrieved_at = GREATEST(p.last_retrieved_at, page_retrievals.last_retrieved_at). Filters: deleted_at IS NULL, body_storage = 'inline', excludeSlugs, source scope. ROW_NUMBER() ... PARTITION BY prefix ordered by stale score (if staleBias, threshold default 90 days), then connection_count DESC, then slug. One page per requested prefix. Representative chunk = lowest chunk_index with embedding IS NOT NULL. |
PGLite :1407 scopedRead(this.engineSql) direct. Postgres :1046 wraps reads in withScopedReadTransaction(opts.sourceIds, opts.sourceId). |
listCorpusSample (:1195) |
listCorpusSample(opts: CorpusSampleOpts): Promise<DomainBankRow[]> |
pages.ts:704-750: optional SELECT setseed(clamped float8) on the same executor when opts.seed is a number, then ORDER BY RANDOM() LIMIT n over the same filters (inline bodies, not deleted, exclusions, source scope). Same representative-chunk subquery. |
PGLite :1411 direct. Postgres :1052 passes alwaysTransaction: typeof opts.seed === 'number', so the seed and the select share one transaction and connection. |
Search
Keyword search shares one builder family in src/core/search/* (keyword-chunk-match.ts, keyword-statement.ts, keyword-or-rarity.ts, title-statement.ts, vector-statement.ts, cjk-keyword-sql.ts). Engine code wires them to the executor.
| Member (engine.ts) | Signature (compact) | What it does | PGLite (pglite-engine.ts) |
Postgres (postgres-engine.ts) |
|---|---|---|---|---|
searchKeyword (:1198) |
searchKeyword(query, opts?: SearchOpts): Promise<SearchResult[]> |
Page-grain keyword search, best chunk per page. Shape: content_chunks.search_vector @@ websearch_to_tsquery(<ftsLang>, $1), ts_rank(...) * keyword phrase factor * sourceFactorCase (source boosts, detail gate), ROW_NUMBER() OVER (PARTITION BY page_id ...) per page, innerLimit = min(limit*3, cap*3), visibility clause (soft-delete, archived source), hard-exclude prefixes, cc.modality = 'text'. OR fallback (orFallback) reruns with OR-of-terms through the same path. CJK queries go to _searchKeywordCJK. |
searchKeyword :1433. Plain db.query with beforePlannerRead settle (planner-stats.ts:389). No statement timeout, no JIT setting, no work_mem, no candidate LIMIT (KEYWORD_RANK_CANDIDATES not applied). Emits stale per row via timeline_entries MAX(created_at) comparison. |
searchKeyword :1066 (builder :1078-1262). Uses withScopedReadTransaction with alwaysTransaction: true, jitOff: true and settings: statement_timeout from keywordStatementTimeoutSetting() (default 8000 ms, range 1000-60000; keyword-statement-timeout.ts), enable_seqscan: 'off' on relaxed or candidate-join attempts, KEYWORD_STATEMENT_SETTINGS (work_mem 32MB, keyword-chunk-match.ts:40). Candidate rows capped at KEYWORD_RANK_CANDIDATES = 50,000 (keyword-chunk-match.ts:30). Timeouts routed through keywordTimeouts negative cache (#6495; :222, :1231). Emits false AS stale. |
countKeywordPages (:1200) |
countKeywordPages(query, opts?: SearchOpts & {timeoutMs?}): Promise<number> |
Cat 40 Hard F2 strict count only (never OR retry): buildKeywordCountStatement (search/keyword-statement.ts), capped at KEYWORD_COUNT_CAP + 1 = 10,001 (keyword-statement.ts:35). Via engine-sql/keyword-pages.ts:24-33. |
countKeywordPages :1571 via keywordPagesImpl.countKeywordPages(read => read(scopedRead(this.engineSql)), ...) with empty dialect (no timeout). |
countKeywordPages :1266. Scoped read with alwaysTransaction, jitOff, settings; statement_timeout = timeoutMs override when given, else keyword timeout; plus KEYWORD_STATEMENT_SETTINGS. |
searchKeywordPages (:1202) |
searchKeywordPages(query, opts: SearchOpts | undefined, page: KeywordPageWindow): Promise<SearchResult[]> |
One window of matching pages (best chunk each) ordered (score DESC, page_id ASC), no global chunk cap. Statement from buildKeywordPagesStatement (keyword-statement.ts). engine-sql/keyword-pages.ts:36-40. Also searchHandlePages (keyword-pages.ts:43-47) for slug-handle hits (GBRA-108). |
searchKeywordPages :1574 (no timeout, no settings). |
searchKeywordPages :1269. Same scoped transaction as countKeywordPages, with the keyword timeout and KEYWORD_STATEMENT_SETTINGS. |
searchTitles (:1223) |
searchTitles(query, opts?: SearchOpts): Promise<SearchResult[]> |
Page-grain title arm (fix/title-retrieval-arm D1). search/title-statement.ts: pages whose title matches websearch_to_tsquery, ORDER BY exact_title DESC, score DESC, p.id ASC (exact-title key #5889), LIMIT/OFFSET, then LEFT JOIN LATERAL one representative chunk (compiled_truth preferred, then lowest chunk_index). Same page-grain filters and visibility as keyword. AND then OR fallback (engine-sql/titles.ts:67). No query-length gate. |
searchTitles :1578 via titlesImpl.searchTitles with relaxedPrefersIndex: false, staleProbe: true (so stale is computed) and no timeout. |
searchTitles :1273 via titlesImpl.searchTitles with statementTimeout from keyword timeout, statementSettings: KEYWORD_STATEMENT_SETTINGS, relaxedPrefersIndex: true, staleProbe: false, and chunk-manifest reads. |
searchVector (:1224) |
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]> |
ANN search. search/vector-statement.ts:310 buildVectorSearchStatement: two-stage CTE. The candidate stage keeps a pure distance ORDER BY so HNSW is usable; outer stages re-rank by source factor. Freshness placement (#5824), scope binding and index-walk or scope-scan variants are in the same builder. Iterative HNSW scan used when supportsHnswIterativeScan(extversion) says pgvector is 0.8+ (vector-index.ts:141-144). Pool attempts with remainingMs budget. |
searchVector :1726. Same statement builder with dialect: 'pglite' (adds timeline stale). this.vectorIterativeScan probe cached on the engine. Each attempt in its own db.transaction with withVectorSettings. No exact-scan fallback (comment at :1727). Calls beforePlannerRead settle. |
searchVector :1450. vectorScope(opts) (scope binding), then searchIndexWalk / searchVectorPool with attempts via runVectorAttempt (scoped read, jitOff, vectorScanSettings incl. iterative scan and per-attempt statement_timeout). Falls back to an exact scan (stmt.exactSql) when the pool is short. explainVectorSearch (EXPLAIN only) shares the same path. |
getEmbeddingsByChunkIds (:1238) |
getEmbeddingsByChunkIds(ids: number[], column = 'embedding'): Promise<Map<number, Float32Array>> |
Rehydrates embeddings for rescoring. engine-sql/chunks.ts:1085-1119: column must match COLUMN_NAME_REGEX (else EmbeddingColumnNotRegisteredError) and is identifier-quoted. SELECT cc.id, vector_send(cc.<col>::vector) AS embedding FROM content_chunks cc JOIN pages p ... WHERE cc.id = ANY($::int[]) AND cc.<col> IS NOT NULL AND chunkVisibleFilter. Binary vector_send decoded with decodeVectorSend. Empty input returns an empty Map. |
getEmbeddingsByChunkIds :1762 via chunksImpl with unscopedExecutor (LegacyUnscopedRead). |
getEmbeddingsByChunkIds :1541, same brand and SQL. No scope transaction (unscoped on master), though the visibility filter stays in SQL. |
Engine-only search members (not on the interface, same range): searchKeywordChunks (Postgres :1291; PGLite :1638) is the chunk-grain anchor primitive with no page dedup. Postgres also has _searchKeywordCJK (postgres-engine.ts:1435) and explainVectorSearch.
CJK keyword fallback (engine-sql/cjk-search.ts): both engines route CJK queries (hasCJK) away from websearch_to_tsquery to an ILIKE term-AND scan (search/cjk-keyword-sql.ts). CJK_CANDIDATE_CAP = 2000 (cjk-search.ts:55). PGLite picks a capped vs uncapped variant from the planner chunk estimate versus PGLITE_CJK_CAPPED_CHUNKS = 50,000 (cjk-search.ts:56, :171). Postgres runs it under the keyword statement timeout and scoped read (postgres-engine.ts _searchKeywordCJK).
Engine-only members adjacent to this range (for completeness)
Not on the interface, but on one or both engines in the same area: connectForRestore, connectWithHeldLock, closeRetainingLock, holdsDatastoreLock, walRepairReceipt (PGLite); onCheckout, getPoolDiagnostics, keywordBoundedRead, refreshPageBody, updatePageContextualRetrievalState, migrateFactsToCanonical, explainVectorSearch, searchKeywordChunks (Postgres and PGLite as noted). putPage also gets its own write-side guard assertOriginUnclaimed from persistence/origin-guard.ts.
Tests
Counts are reference counts from rg over test/ (not distinct tests). Gating: test/e2e/engine-parity.test.ts header says it is gated by DATABASE_URL and skips without Postgres, but the PGLite half always runs.
| Member | Main coverage |
|---|---|
connect, disconnect, initSchema |
engine-parity.test.ts (17 each); pglite-engine.test.ts (connect 2, initSchema 3, disconnect 1); pglite-engine-disconnect.serial.test.ts; postgres-engine.test.ts; postgres-engine-config-reconnect.test.ts; engine-surface-coverage.test.ts (method sweep, see below). |
reconnect |
postgres-engine-config-reconnect.test.ts (27); supervisor-health-reconnect.test.ts (26); core/retry-reconnect.test.ts (25); connection-resilience.test.ts (16); pglite-reconnect.serial.test.ts; e2e/postgres-reconnect-singleton.test.ts (17); postgres-engine-getter-selfheal.test.ts. |
holdSchemaMigrationLock |
Only test/helpers/postgres-engine-sql-cases.ts (1 ref). No dedicated test of the cross-process lock was found in the search; withSchemaMigrationLock callers are not individually checked. |
registerBeforeDisconnect |
pglite-engine-disconnect.serial.test.ts; persistence-reconnect.serial.test.ts; engine-control-routing.test.ts; search-settings-one-statement.test.ts; engine-keyed-caches.test.ts. |
transaction, transactionDirect |
engine-sql-transaction.test.ts; engine-sql-transaction-parity.test.ts; helpers/engine-sql-rollback-cases.ts; postgres-engine-reserved-routing.test.ts; direct-pool-profile.test.ts; autopilot-transaction-fence.test.ts; engine-control-routing.test.ts. |
withReservedConnection |
postgres-engine-reserved-routing.test.ts (24); worker-lock-renewal.test.ts (25); sync-drain-connection.test.ts (14); gauge goldens test/fixtures/goldens/postgres-engine-gauge/acquires-by-case.json. |
getPage, readPageSnapshot |
engine-parity.test.ts (getPage 32); pglite-engine.test.ts (getPage 13); helpers/postgres-engine-sql-cases.ts; many (readPageSnapshot in ~507 files). |
readPageSnapshotsBatch |
helpers/page-snapshot-batch-cases.ts; helpers/import-reads-cases.ts; entity-names-parity.test.ts; postgres-rls-scope-inventory.test.ts. |
lockPageKeys |
belief-ledger-purge.test.ts; belief-ledger-purge-review.test.ts; link-cross-source.test.ts; persistence-memory-mutations.test.ts; helpers/postgres-engine-sql-cases.ts. |
putPage |
engine-parity.test.ts (74); pglite-engine.test.ts (93). Heaviest-used write in the suite. |
findDuplicatePage, findDuplicatePages |
import-dedup-frontmatter-id.test.ts; import-reads.test.ts; e2e/import-reads.test.ts; import-identity-move.test.ts; helpers/import-reads-cases.ts. |
deletePage |
pglite-engine.test.ts (4); engine-parity.test.ts (1); pages-soft-delete.test.ts; ~36 files. |
deletePages, resolveSlugsByPaths |
engine-parity.test.ts (3 each); sync-delete-batch.test.ts (18). |
softDeletePage, softDeletePages, restorePage |
pages-soft-delete.test.ts (22); sync-soft-delete.serial.test.ts; engine-parity.test.ts (softDeletePage 10, softDeletePages 5, restorePage 9); sync-rename-reconcile.serial.test.ts. |
purgeDeletedPages |
pages-soft-delete.test.ts; e2e/purge-deleted-dryrun-postgres.test.ts; purge-expires-page-facts.test.ts; pathological/h5-atoms-restore.test.ts; pathological/g0-derived-retirement-hold.test.ts. |
listPages |
pglite-engine.test.ts (14); engine-parity.test.ts (2); ~58 files. |
findPages, countPagesBy |
helpers/postgres-engine-sql-cases.ts; engine-surface-coverage.test.ts; list-pages-empty-type-notice-gbra449.test.ts (countPagesBy). |
resolveSlugs |
pglite-engine.test.ts (4); helpers/postgres-engine-sql-cases.ts; validate-params.test.ts; operations-fuzzy-source-scope.test.ts; call-federated-scope.test.ts; e2e/engine-content-privacy.test.ts. |
getAllSlugs |
pglite-engine.test.ts (3); engine-parity via helpers; link-extraction.test.ts; e2e/integrity-batch.test.ts; source-id-tx-regression.test.ts. |
listAllPageRefs |
engine-parity.test.ts (3); ~9 files. |
listAllSources |
list-all-sources.test.ts (24); e2e/list-all-sources-postgres.test.ts (20); e2e/autopilot-fanout-postgres.test.ts; cycle-last-full-cycle-at.test.ts; storage-manifest.test.ts. |
updateSourceConfig |
cycle-stamp-write-failure.test.ts (11); helpers/engine-sql-rollback-cases.ts; e2e/list-all-sources-postgres.test.ts; connector-checkpoint-identity.test.ts. |
listPrefixSampledPages, listCorpusSample |
page-retrievals.test.ts; brainstorm-timeout.test.ts; brainstorm/cost-guardrails.test.ts; postgres-rls-scope-inventory.test.ts. |
searchKeyword |
engine-parity.test.ts (44); pglite-engine.test.ts (23); keyword-timeout-negative-cache-6495.test.ts (44); keyword-statement-timeout-knob-6495.test.ts (26); e2e/search-quality.test.ts; e2e/search-exclude.test.ts; cjk-keyword-fallback.test.ts (20); e2e/engine-parity-cjk.test.ts. |
countKeywordPages, searchKeywordPages |
search-keyword-paging.test.ts (15); e2e/search-keyword-paging-parity.test.ts; search/keyword-pages-limit-first.test.ts; search/slug-handles.serial.test.ts (handles); search/keyword-arm-access.test.ts. |
searchTitles |
engine-parity.test.ts (6); search/title-retrieval-arm.test.ts (21); e2e/title-arm-postgres.test.ts; keyword-timeout-negative-cache-6495.test.ts. |
searchVector |
engine-parity.test.ts (10); pglite-engine.test.ts (6); search/searchvector-escalation.test.ts (12); e2e/vector-candidate-safety-postgres.test.ts (15); e2e/vector-plan-real-column-postgres.test.ts (11). |
getEmbeddingsByChunkIds |
engine-sql-chunks.test.ts; cosine-rescore-chunkless-headstart.test.ts; postgres-engine.test.ts; e2e/search-quality.test.ts; e2e/embedding-column-pglite.test.ts; unified-multimodal.serial.test.ts. |
Cross-cutting: test/engine-surface-coverage.test.ts checks that every interface method exists on PGLiteEngine.prototype and that every prototype member is classified (Plan D9). test/e2e/engine-parity.test.ts runs identical corpora through both engines. test/engine-sql-*.test.ts pin the shared SQL, transaction and normalizer behavior per executor. Golden SQL text lives in test/fixtures/goldens/sql-text/ and RLS scope inventory in test/fixtures/goldens/rls-scope-inventory.json.
Parity risks noted
- Keyword timeouts, JIT and work_mem: Postgres bounds keyword, title, count and CJK reads with
statement_timeout(default 8000 ms, configurable 1-60 s), JIT off andwork_mem 32MB. PGLite has none of these, so a slow query can run unbounded there. A Rust engine needs an explicit per-query deadline or it will not match Postgres behavior. - Keyword candidate bound: Postgres caps keyword candidates at 50,000 chunks (
KEYWORD_RANK_CANDIDATES). PGLite does not, so PGLite can rank more candidates and return different top-N under load. staleflag: PGLite computes per rowstalefromtimeline_entries. PostgressearchKeywordreturnsfalse AS stale(postgres-engine.ts:1217), andsearchTitleson Postgres disables the stale probe (staleProbe: false). Consumers see differentstalevalues per engine.- Vector search: Postgres has an exact-scan fallback and iterative HNSW scan (pgvector 0.8+), with an index-walk path and per-attempt budgets. PGLite has no exact fallback and no pool escalation beyond its own attempts (pglite-engine.ts:1733 comment). Result sets can differ when the ANN pool is short.
- Planner stats: PGLite runs
beforePlannerRead(ANALYZE settle) around reads (planner-stats.ts:389-420). No planner-stats call was found in the Postgres engine file. Plan shapes differ, which can change ties inORDER BY score. - Transactions: Postgres honors
readOnly(BEGIN READ ONLY),pipelineEarly/pipelineFirst, signal abort and commit checks. PGLite honors onlysignal. A read-only transaction on PGLite can write. withReservedConnection: PGLite ignoresrouteandselfContained, has notransactionmember, and does not enforce pool permits. Postgres enforcesPoolCapacityErrorand direct-pool routing.reconnect: PGLite in-memory is a deliberate no-op (state preserved). File-backed PGLite does a full disconnect and reconnect. Postgres rebuilds pools with build-then-swap. Callers that rely on "reconnect restores the same state" behave differently in-memory vs file-backed PGLite.- Schema lock: Postgres uses cross-process advisory key 42 (session or transaction form). PGLite serializes in-process only (
schema-migration-lock.ts:22-30). PGLiteinitSchemaalso skipsverifySchemaand the zombie-index sweep that Postgres runs after migrations. - Soft delete asymmetry:
softDeletePagehas no'default'source fallback. IfsourceIdis omitted, it can flip the same slug in every source (pages.ts:276-295), whiledeletePagedefaults to'default'.deletePages,softDeletePagesandresolveSlugsByPathsrequiresourceId. getAllSlugsincludes soft-deleted pages (nodeleted_atpredicate, pages.ts:563-572). Its JSDoc says "every page in the brain", so callers doing existence checks can see tombstones.- Unscoped reads on Postgres:
resolveSlugs,resolveSlugsByPaths,listAllPageRefs,listAllSources,getEmbeddingsByChunkIdsrun on the pool without an RLS scope transaction (LegacyUnscopedRead brand). The Rust rewrite has no RLS, so these become plain reads, but any access control must be enforced in the app layer. - Blank-body guard:
putPagerefuses to overwrite a non-empty or segmented body with a blank one unlessallowEmptyOverwrite(pages.ts:116-133). Both engines share this logic, but PGLite'srereadcallback differs from Postgres'sRETURNINGassumption, and the Rust port must keep the same guard. - Batch limits:
deletePages,softDeletePages,resolveSlugsByPathsthrow aboveDELETE_BATCH_SIZE= 500 (engine-constants.ts:25). Callers must chunk; a Rust port must keep the cap or the contract changes. - Bind-parameter ceiling: PGLite batches under 30,000 bind params (
PGLITE_CAPABILITIES, dialect-pglite.ts:25-29). Postgres is unbounded. Batched writes must stay under the PGLite ceiling if both engines stay in use during migration. - Statement cancellation: PGLite cannot cancel a running statement (WASM). The abort race in
pgliteQueryonly settles the caller, the statement finishes in the background (dialect-pglite.ts:42-56). Postgres sends a real cancel. Timeout behavior differs under contention. - Lock timeouts: Postgres schema replay runs under
lock_timeout(schema-lock-timeout.ts) andretrySchemaReplayOnDeadlock. PGLite has no equivalent. - Vector column binding:
getEmbeddingsByChunkIdsvalidates the column name by regex and quotes it. The Rust port must keep the same allowlist, since column names are interpolated. listPrefixSampledPagesrepresentative chunk usesembedding IS NOT NULL(hardcoded column) even thoughgetEmbeddingsByChunkIdstakes a column. Keep both behaviors aligned.- Sampling determinism:
listCorpusSamplewithseeddepends onsetseedfollowed byORDER BY RANDOM()on the same connection (Postgres forces a transaction). PGLite is single-connection, so it matches, but the Rust engine must reproduce the same seeded order or accept a different sample.
Chunks, Extraction staleness and Links/Graph
Scope: BrainEngine interface declarations in src/core/engine.ts lines 1240-1841 (the // Chunks, extraction-freshness and // Links blocks, ending before // Tags at line 1842). That range declares 45 methods: 15 chunk/embedding, 7 extraction staleness, 6 link writes, 17 link/graph reads. None of them is marked optional (?) in the interface.
Engine implementations compared: src/core/pglite-engine.ts (PGLiteEngine, 1812-2078) and src/core/postgres-engine.ts (PostgresEngine, 1626-1899). SQL lives in src/core/engine-sql/chunks.ts, pages.ts, links.ts, and in src/core/search/read-enrichment.ts (the graph and boost readers both engines share, via executeRaw), plus src/core/derived-links.ts.
Engine difference, in short: the SQL text is shared. The two engines differ in the read transaction wrapper and the planner/JIT handling around it:
- PGLite calls
scopedRead(this.engineSql)directly, with no transaction. Several reads are wrapped inplannerRead(...)(pglite-engine.ts:1992, 2015, 2029, 2077), which can runANALYZE-style planner refreshes (src/core/planner-stats.ts:389-410). Postgres never callsplannerRead. - PostgresEngine wraps scoped reads in
withScopedReadTransaction(sourceIds, sourceId, tx => ...)(postgres-engine.ts:368), which sets the source/RLS scope in a transaction. It also threadschunkManifestReadsinto chunk reads (postgres-engine.ts:1642, 1650, 1658, 1856).chunkManifestReadsreturnstrueon every non-Postgres engine (src/core/search/safe-chunks.ts:107-113), so PGLite omitting the argument is equivalent by construction. getBacklinkCountson Postgres runs injitOffRead(postgres-engine.ts:1863-1867,prepare: true). PGLite runs it throughexecuteRaw(pglite-engine.ts:2043).- The
unscopedExecutor(..., 'EO4 inventory')wrapper marks reads that are deliberately unscoped on the master path. PGLite and Postgres both use it forfindByTitleFuzzy,traverseGraph,traversePathsDetailed,findOrphanPages,sumStaleChunkChars,getPageTimestamps, and thelinks/pagesreads that take no scope.
A. Chunks (engine.ts 1240-1423)
Shared helpers used below: chunkReadWhere (engine-sql/chunks.ts:396-408) is the visibility predicate for chunk reads (page slug, source scope p.source_id = ANY($::text[]) or = $, chunkVisibleFilter, optional private/safe filters). staleChunkWhere (chunks.ts:509-534) is the stale predicate: p.deleted_at IS NULL, the active embedding column IS NULL (or signature drift), embed_skip excluded, and budgetLimited excluded. chunkGenerationFilter (search/safe-chunks.ts:77) gates segmented bodies through page_body_manifest.
| # | Method (engine.ts line) | Signature (compact) | What it does | SQL / delegation | PGLite vs Postgres | Tests |
|---|---|---|---|---|---|---|
| 1 | upsertChunks (1269) |
(slug, chunks: ChunkInput[], opts?: {sourceId, embeddingColumn, expectedRevision, sealChunkerVersion, pageId, deferSeal} & BatchOpts): void |
Replaces a page's chunk set. Inserts or updates chunks by (page_id, chunk_index) and moves or prunes displaced chunks. Stamps embedded_text_hash, embedded_at, embedding_pending_since. Optional seal of pages.chunker_version. |
upsertChunksOnce (chunks.ts:70): SELECT id FROM pages ... FOR UPDATE (or the seal UPDATE pages SET chunker_version); optional expectedRevision check; moveDisplacedChunksAndPrune (chunks.ts:360, unnest(int[],int[]) for moves, md5(chunk_text) for match); one batched INSERT INTO content_chunks ... SELECT ... FROM jsonb_to_recordset($json::text::jsonb) ... ON CONFLICT (page_id, chunk_index) DO UPDATE (chunks.ts:260-285), rows batched via jsonRowBatches. Column routed by resolveWriteColumnFromConfigRows and cast with vectorCastSuffix (search/embedding-column.ts:279: ::vector or ::halfvec(N)). embedding_image::vector for image chunks. md5(c.chunk_text) for embedded_text_hash (#4246). |
PGLite upsertChunks (pglite-engine.ts:1812) runs _upsertChunksOnce via batchRetry + transaction (single connection, _chunkWritesInTransaction shortcut). Postgres (postgres-engine.ts:1626) same logic with lockPageKeys, readPageSnapshot. Same SQL. |
upsertChunks: about 96 files reference it. Parity: e2e/engine-parity.test.ts:915-943 (round trip, sealed vs unsealed). PGLite: pglite-engine.test.ts:519-560. Also e2e/upsert-chunks-registry-column.test.ts, embedding-migration.test.ts. |
| 2 | getChunks (1279) |
(slug, opts?: PageReadScope & {includeEmbedding, includeUnsealed}): Chunk[] |
Every chunk of a page in document order. Vectors omitted unless includeEmbedding (returns the active column AS embedding). Returns embedding_is_null as a boolean. Default scope is 'default' source when neither sourceId nor sourceIds is set. |
chunks.ts:410-462: SELECT ... FROM content_chunks cc JOIN pages p ... WHERE chunkReadWhere ORDER BY chunkDocumentOrder. (cc.<col> IS NULL) AS embedding_is_null. Optional ordinal window function row_number() OVER (PARTITION BY page_id ...) for segmented pages. |
PGLite pglite-engine.ts:1828 (scopedRead, no tx). Postgres postgres-engine.ts:1645 (withScopedReadTransaction + chunkManifestReads). |
getChunks: about 108 files. Parity: e2e/engine-parity.test.ts:951-982 (#2555 sourceIds[] parity), :937-948 (includeUnsealed). PGLite: pglite-engine.test.ts:229, 519-537. |
| 3 | getChunkPage (1281) |
(slug, opts: PageReadScope & {fromIndex, limit}): {chunks, total_chunks, max_chunk_index} |
Paged get_chunks: chunks with chunk_index in [fromIndex, fromIndex+limit), plus total count and max index. Segmented pages use document position instead. |
chunks.ts:474-495: one aggregate sum(count(*)), max(chunk_index), bool_or(body_generation IS NOT NULL), then getChunks with an indexRange. |
PGLite pglite-engine.ts:1835. Postgres postgres-engine.ts:1653. |
get-chunks-paging.test.ts, engine-surface-coverage.test.ts, e2e/segmented-switch-off-rls.test.ts. No parity test. |
| 4 | getChunkWindows (1283) |
(requests: ChunkWindowRequest[], opts: ChunkWindowOpts): ChunkWindowPage[] |
Batched evidence read keyed by page_id. Re-authorized. Returns page metadata plus chunks in each requested index window, capped at maxRows. |
chunks.ts:1147-1190: WITH req AS (SELECT * FROM unnest(int[], int[], int[], int[]) AS r(page_id, lo, hi, prio)), auth CTE (source, archived, deleted, projection, quarantine, private, safe filters), chunk_rows with LIMIT maxRows+1, UNION ALL. Batch pattern: four-array unnest. |
PGLite pglite-engine.ts:1826: scopedRead(this.engineSql) (no tx). Postgres postgres-engine.ts:1640: withScopedReadTransaction + chunkManifestReads. |
getChunkWindows: about 19 files. Parity: e2e/evidence-delivery-parity.test.ts, evidence-delivery.test.ts, evidence-long-page.test.ts, evidence-tail-trim.test.ts, e2e/segmented-rls-scope.test.ts, postgres-rls-scope-inventory.test.ts. |
| 5 | countStaleChunks (1299) |
(opts?: {sourceId, signature, includeNullSignature}): number |
Count of chunks needing embedding: active column IS NULL. With signature, also embedded chunks whose page signature differs (NULL signature grandfathered unless includeNullSignature). |
chunks.ts:536-547: SELECT count(*)::int FROM content_chunks cc JOIN pages p ... WHERE staleChunkWhere. No vector op; IS NULL / <> on text signature only. |
PGLite pglite-engine.ts:1850: scopedRead, no tx. Postgres postgres-engine.ts:1670: withScopedReadTransaction(undefined, sourceId). |
countStaleChunks: about 47 files. Parity: e2e/engine-parity.test.ts does not cover it. PGLite: embedding-state.test.ts, embedding-truth-predicates.serial.test.ts, embed-oversize-heal-drain.serial.test.ts, embed-partial-failure-3037.serial.test.ts, embed-stale-progress-5968.serial.test.ts. |
| 6 | sumStaleChunkChars (1316) |
(opts?: same as countStaleChunks): number |
Sum of LENGTH(chunk_text) over stale chunks (cost preview). |
chunks.ts:549-564: SELECT COALESCE(SUM(chars),0)::bigint FROM (...). With signature, UNION ALL of the unembedded arm and the drifted-page arm, so each arm uses its own index (#8811). |
Both: unscopedExecutor(..., 'chunks: unscoped on master (EO4 inventory)'), no source scope transaction (pglite-engine.ts:1855, postgres-engine.ts:1675). |
sumStaleChunkChars: about 17 files. sum-stale-chunk-chars.test.ts, sum-stale-chunk-chars-8811.test.ts, embedding-signature-stale.test.ts, e2e/migrate-embeddings-postgres.test.ts, helpers/postgres-engine-sql-cases.ts. |
| 7 | setPageEmbeddingSignature (1322) |
(slug, opts: {sourceId?, signature}): void |
Stamps pages.embedding_signature after (re)embedding a page. |
chunks.ts:566-571: UPDATE pages SET embedding_signature = $ WHERE slug = $ AND source_id = $. No deleted_at filter. |
Same SQL. PGLite pglite-engine.ts:1860, Postgres postgres-engine.ts:1680. |
setPageEmbeddingSignature: about 24 files. embedding-signature-stale.test.ts, embedding-restamp-w3.test.ts, migrate-embeddings-hardening.serial.test.ts, embed-stale-signature-pagination.serial.test.ts. |
| 8 | invalidateStaleSignatureEmbeddings (1339) |
(opts: {signature, sourceId?, includeNullSignature?}): number |
NULLs embedding, embedded_at and embedding_pending_since for chunks whose page signature differs from the current one. Returns chunk count. |
chunks.ts:582-609: inTransaction(...), lockEmbeddingSources(tx, sourceId), then UPDATE content_chunks cc SET <col> = NULL ... FROM pages p WHERE ... AND NOT currentSpaceChunkFragment(column, model, dims) AND sigClause RETURNING cc.page_id. Uses splitEmbeddingSignature for model/dims. |
Both: this.transaction(fn) plus resolveActiveEmbeddingColumnFromEngine (pglite-engine.ts:1864, postgres-engine.ts:1684). Same SQL. |
invalidateStaleSignatureEmbeddings: about 17 files. e2e/migrate-embeddings-postgres.test.ts, e2e/embedding-tombstones-postgres.test.ts, embedding-migration.test.ts, e2e/upsert-chunks-registry-column.test.ts, helpers/engine-sql-rollback-cases.ts, helpers/postgres-engine-sql-cases.ts. |
| 9 | invalidateContentDriftEmbeddings (1351) |
(opts?: {sourceId?}): number |
NULLs vectors whose stored embedded_text_hash no longer equals md5(chunk_text) (#4246). Loops in 5000-row batches, one transaction each, no saved cursor (#7117). |
chunks.ts:685-698 loops invalidateContentDriftBatch (chunks.ts:638-676): WITH b AS (SELECT id FROM content_chunks WHERE id > $after ORDER BY id LIMIT n), w AS (UPDATE content_chunks cc SET <col> = NULL, embedded_at = NULL, embedded_text_hash = NULL ... WHERE cc.embedded_text_hash <> md5(cc.chunk_text) AND NOT embed_skip AND NOT budgetLimited RETURNING 1). The cursor and limit are inlined, not bound. |
Both: this.transaction(fn) (pglite-engine.ts:1869, postgres-engine.ts:1689). Same SQL. |
invalidateContentDriftEmbeddings: about 17 files. e2e/content-drift-sweep-postgres.test.ts, e2e/embedding-tombstones-postgres.test.ts, e2e/upsert-chunks-registry-column.test.ts, helpers/engine-sql-rollback-cases.ts, helpers/postgres-engine-sql-cases.ts. |
| 10 | listStaleChunks (1368) |
`(opts?: {batchSize=2000, afterPageId, afterChunkIndex, sourceId, orderBy:'page_id' | 'updated_desc', afterUpdatedAt}): StaleChunkRow[]` | Keyset-paged stale chunks (NULL active column, not embed_skip, not budget-limited, live page). Cursor is (page_id, chunk_index) or (updated_at DESC, page_id, chunk_index). |
chunks.ts:700-813: four SELECT ... FROM content_chunks cc JOIN pages p WHERE cc.<col> IS NULL ... ORDER BY p.updated_at DESC NULLS LAST, p.id, cc.chunk_index LIMIT n variants, or ORDER BY cc.page_id, cc.chunk_index with (page_id, chunk_index) > ($, $). No vector op. |
PGLite pglite-engine.ts:1874: scopedRead, no tx. Postgres postgres-engine.ts:1694: withScopedReadTransaction(undefined, sourceId). |
| 11 | countChunklessPagesWithContent (1396) |
(opts?: {sourceId?}): number |
Pages with non-empty compiled_truth / timeline (or a segmented body) and zero chunks, excluding quarantined pages and sealed text projections (#8473). |
chunks.ts:826-847: SELECT count(*)::int FROM pages p WHERE deleted_at IS NULL AND (compiled_truth <> '' OR timeline <> '' OR segmented body) AND quarantine filter AND (text_projection_revision IS DISTINCT FROM knowledge_revision OR embed_skip) AND NOT EXISTS (SELECT 1 FROM content_chunks cc WHERE cc.page_id = p.id). |
PGLite pglite-engine.ts:1886 (scopedRead). Postgres postgres-engine.ts:1706 (withScopedReadTransaction). |
countChunklessPagesWithContent: about 16 files. Parity: e2e/engine-parity.test.ts (chunkless detection parity, around line 1177). PGLite: embedding-state.test.ts, embed-stale-chunkless-pages.serial.test.ts, embedding-state-projection-empty-8473.test.ts. |
| 12 | listChunklessPagesWithContent (1408) |
(opts?: {batchSize=50, afterPageId, sourceId}): ChunklessPageRow[] |
Keyset page (by id) of chunkless pages with their full compiled_truth and timeline. Small default batch on purpose. |
chunks.ts:848-870: SELECT p.id, p.slug, p.source_id, p.compiled_truth, p.timeline FROM pages p WHERE <chunklessPagesWhere> AND p.id > $ ORDER BY p.id LIMIT n. |
PGLite pglite-engine.ts:1890. Postgres postgres-engine.ts:1710. Same SQL. |
listChunklessPagesWithContent: about 13 files. Parity: e2e/engine-parity.test.ts, helpers/postgres-engine-sql-cases.ts. PGLite: embed-stale-chunkless-pages.serial.test.ts, embed.serial.test.ts. |
| 13 | readEmbeddingStateCounts (1414) |
`(opts: {mode: 'ops' | 'full', sourceIds?, purgeWindowHours}): EmbeddingStateCounts` | Counts per embedding state (embedded, pending, keyword_only, quarantined, empty, deleted_page_chunks, chunkless_healable). ops mode uses a partial index on the active column IS NULL, and falls back to a planner estimate for embedded. |
chunks.ts:922-1046: pg_index/pg_class lookup for a covering partial index; aggregate with count(*) FILTER (...); EXPLAIN (FORMAT JSON) for the estimated embedded count. embed_skip marker bytes counted. |
PGLite pglite-engine.ts:1894. Postgres postgres-engine.ts:1714 (withScopedReadTransaction(sourceIds)). The pg_index probe and the EXPLAIN estimate behave differently on PGLite (plannerRows, chunks.ts:1005). |
| 14 | listEmbeddingStateRows (1416) |
(opts: {sourceIds?, slugs?, states[], limit, purgeWindowHours}): EmbeddingStateRow[] |
One row per (page, state), ordered (state, source_id, slug), at most limit. |
chunks.ts:1020-1044: SELECT ... FROM (...) r ORDER BY r.state, r.source_id, r.slug LIMIT n over the same state CASE expressions as the counts. |
PGLite pglite-engine.ts:1899. Postgres postgres-engine.ts:1719. |
listEmbeddingStateRows: about 12 files. e2e/embedding-state-postgres.test.ts, embedding-state.test.ts, helpers/postgres-engine-sql-cases.ts. |
| 15 | deleteChunks (1422) |
(slug, opts?: {sourceId?}): void |
Deletes every chunk of a page and clears pages.text_projection_revision (the seal). |
chunks.ts:1048-1066: WITH page AS (SELECT id FROM pages WHERE slug=$ AND source_id=$ FOR UPDATE), gone AS (DELETE FROM content_chunks WHERE page_id=(SELECT id FROM page) RETURNING page_id) UPDATE pages SET text_projection_revision = NULL WHERE id=... AND .... Page row locked before chunks (lock order, #8502). |
PGLite pglite-engine.ts:1904, no transaction wrapper. Postgres postgres-engine.ts:1724. Same SQL. |
deleteChunks: about 16 files. Parity: e2e/delete-chunks-lock-order-8502.test.ts. PGLite: pglite-engine.test.ts:565-572, source-id-tx-regression.test.ts, import-file.test.ts. |
Notes for section A:
- Embedding-column routing is resolved per call by
resolveActiveEmbeddingColumnFromEngine(pglite-engine.ts:1831, 1851;postgres-engine.ts:1648, 1671). The write path resolves from config rows inside the upsert statement's own executor (chunks.ts:139-170). - pgvector: in this range, the only pgvector operations are the cast on insert (
c.embedding::vectoror::halfvec(N),chunks.ts:260) andembedding_image::vector. Stale and drift predicates useIS NULLandmd5(), not vector distance operators. Vector reads (vector_send, cosine) are outside this range (engine-sql/chunks.ts:1085-1131,getEmbeddingsByChunkIdsandcosineByChunkIdsStatement, declared at engine.ts:1238 and before). - Batch pattern:
upsertChunksusesjsonb_to_recordset(one JSON document per batch,jsonRowBatches), notunnest.moveDisplacedChunksAndPrune(chunks.ts:389) andgetChunkWindows(chunks.ts:1157) useunnestover parallel int arrays. - Recursive CTE: none in this section.
B. Extraction staleness (engine.ts 1424-1482)
The predicate is stalePagesWhere in engine-sql/pages.ts (line 827). A page is stale when deleted_at IS NULL, not quarantined, and links_extracted_at IS NULL OR links_extracted_at < versionTs OR updated_at > links_extracted_at OR a wanted-producer origin is stale. Options: attendance ('exclude' | 'blocked', by links_attendance_blocked_revision vs knowledge_revision), held ('exclude' | 'only', linkExtractionHeld), and a guest-source exclusion. This is the shared watermark behind gbrain extract --stale and the links_extraction_lag doctor check.
| # | Method (engine.ts line) | Signature (compact) | What it does | SQL / delegation | PGLite vs Postgres | Tests |
|---|---|---|---|---|---|---|
| 16 | countStalePagesForExtraction (1438) |
(opts?: {sourceId, versionTs, attendance, held, wantedProducers}): number |
Counts pages needing (re)extraction. | pages.ts:871-875: SELECT count(*)::int FROM pages WHERE stalePagesWhere. extractionLagStatement (pages.ts:855) is the combined total/stale/blocked/held query used by doctor. |
PGLite pglite-engine.ts:1908, scopedRead. Postgres postgres-engine.ts:1728, withScopedReadTransaction(undefined, sourceId). |
countStalePagesForExtraction: about 35 files. Parity: e2e/engine-parity.test.ts:1103-1176 (stale-page extraction methods). PGLite: extract-stamp-source-3957.test.ts, guest-derived-registry.test.ts, helpers/postgres-engine-sql-cases.ts. |
| 17 | listStalePagesForExtraction (1446) |
(opts: {batchSize, afterPageId?, sourceId?, versionTs?, held?: 'exclude', wantedProducers?}): StalePageRow[] |
Keyset page (by id) of stale pages with body and frontmatter, so the caller extracts without getPage per row. |
pages.ts:877-900: SELECT id, slug, ..., compiled_truth, timeline, frontmatter, updated_at, to_char(updated_at AT TIME ZONE 'UTC', ...) AS updated_at_iso FROM pages WHERE stalePagesWhere AND id > $ ORDER BY id LIMIT n. |
PGLite pglite-engine.ts:1912, scopedRead. Postgres postgres-engine.ts:1732, withScopedReadTransaction(undefined, sourceId). Same SQL. |
listStalePagesForExtraction: about 17 files. Parity: e2e/engine-parity.test.ts:1103-1176. PGLite: extract-stale.test.ts, attendance-retrieval.test.ts, extract-endpoint-revision.serial.test.ts, maintenance-idle-gbra438.test.ts. |
| 18 | markPagesExtractedBatch (1474) |
(refs: {slug, source_id, extractedAt?}[], defaultExtractedAt: string): number |
Stamps links_extracted_at per page. Clears attendance-blocked marker and extract-failure record. Returns rows actually stamped (#3957 shortfall signal). |
pages.ts:902-929: UPDATE pages p SET links_extracted_at = v.ts::timestamptz, links_attendance_blocked_revision = NULL, ... FROM unnest($slugs::text[], $srcs::text[], $tss::text[]) AS v(slug, source_id, ts) WHERE p.slug = v.slug AND p.source_id = v.source_id. Three-array unnest. |
Same SQL. PGLite pglite-engine.ts:1923 and Postgres postgres-engine.ts:1743, both with this.engineSql, no scope transaction. |
markPagesExtractedBatch: about 15 files. Parity: e2e/engine-parity.test.ts:1129-1150. PGLite: extract-stamp-source-3957.test.ts, pglite-checkpoint-guard.test.ts, fixtures/pglite-engine-sql-wal-worker.ts, persistence-managed-lifecycle.test.ts. |
| 19 | markPagesAttendanceBlocked (1476) |
(refs: {slug, source_id, revision}[]): number |
Marks pages stale only because an attendee did not resolve, at their current knowledge revision (#5761). | pages.ts:931-952: UPDATE pages p SET links_attendance_blocked_revision = v.revision::uuid, links_attendance_blocked_at = now() FROM unnest(slug[], source_id[], revision[]) AS v(...) WHERE ... AND p.knowledge_revision = v.revision::uuid AND p.deleted_at IS NULL. Three-array unnest. |
Same SQL. PGLite pglite-engine.ts:1927, Postgres postgres-engine.ts:1747. |
markPagesAttendanceBlocked: about 10 files. helpers/postgres-engine-sql-cases.ts, helpers/engine-sql-rollback-cases.ts, engine-surface-coverage.test.ts. No behavior parity test in e2e/engine-parity.test.ts. |
| 20 | countStaleFrontmatterKeyPages (1478) |
(opts: {sourceIds: string[]}): {source_id, n}[] |
Per source, live pages whose frontmatter still carries a reserved key (type, title, tags, slug) (#6454). |
pages.ts:954-961: `SELECT source_id, count(*)::bigint FROM pages WHERE source_id = ANY($) AND deleted_at IS NULL AND frontmatter ? |
ARRAY[...] GROUP BY source_id. JSONB ? |
` operator. |
| 21 | listStaleFrontmatterKeyPages (1480) |
(opts: {sourceId, afterId, limit}): {id, slug, keys[]}[] |
Keyset page of those pages with the reserved keys each carries. | pages.ts:963-972: SELECT id, slug, ARRAY(SELECT k FROM unnest(ARRAY[...]::text[]) AS k WHERE frontmatter ? k ORDER BY k) AS keys ... WHERE source_id = $ AND ... AND id > $ ORDER BY id LIMIT n. |
PGLite pglite-engine.ts:1931. Postgres postgres-engine.ts:1751, scope transaction on sourceId. |
listStaleFrontmatterKeyPages: about 10 files, mostly golden and inventory tests plus repair-frontmatter-keys-sweep-6454.test.ts. |
| 22 | stripStaleFrontmatterKeys (1482) |
(opts: {sourceId, ids: number[]}): number |
Strips reserved keys from the named live pages of one source (coordinated maintenance write). | pages.ts:974-981: `UPDATE pages SET frontmatter = frontmatter - ARRAY[...]::text[] WHERE source_id = $ AND id = ANY($ids::int[]) AND deleted_at IS NULL AND frontmatter ? |
ARRAY[...]`. The revision trigger advances each changed row. | PGLite pglite-engine.ts:1932 and Postgres postgres-engine.ts:1752, both with this.engineSql. Same SQL. |
Notes for section B:
pages.tsshares one stale predicate between count and list (stalePagesWhere), so the count and the batch cursor cannot drift apart.- Extraction staleness has no recursive CTE and no pgvector use. Batch pattern:
markPagesExtractedBatchandmarkPagesAttendanceBlockeduse three-arrayunnest(text[]).stripStaleFrontmatterKeysusesid = ANY(int[]). markPagesExtractedBatchis also the stamp that themarkPagesAttendanceBlockedrevision check must agree with. A concurrent edit makesupdated_at > links_extracted_at, so the page stays stale (v0.42.7 D4).
C. Links: writes (engine.ts 1484-1573)
| # | Method (engine.ts line) | Signature (compact) | What it does | SQL / delegation | PGLite vs Postgres | Tests |
|---|---|---|---|---|---|---|
| 23 | addLink (1496) |
(from, to, context?, linkType?, linkSource?='markdown', originSlug?, originField?, opts?: {fromSourceId, toSourceId, originSourceId}): void |
Single edge upsert. Endpoints are resolved source-qualified in one statement. Throws PageMissingError if either endpoint is missing. Temporal link types refresh relationship state. |
links.ts:102-150: WITH endpoint_state AS (SELECT (SELECT id FROM pages WHERE slug=$ AND source_id=$ FOR KEY SHARE) AS from_id, ...), upserted AS (INSERT INTO links (...) SELECT ... FROM endpoint_state WHERE from_id IS NOT NULL AND to_id IS NOT NULL ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO UPDATE SET context = EXCLUDED.context, origin_field = EXCLUDED.origin_field RETURNING 1) SELECT .... Calls refreshRelationships for temporal types. |
PGLite pglite-engine.ts:1935 and Postgres postgres-engine.ts:1755 both call linksImpl.addLink(this.engineSql, ...). Same SQL. |
addLink: about 105 files. Parity: e2e/engine-parity.test.ts:797-830 (soft-delete visibility), :805, 831. PGLite: pglite-engine.test.ts, source-id-tx-regression.test.ts, derived-visibility-repair.test.ts. |
| 24 | addLinksBatch (1521) |
(links: LinkBatchInput[], opts?: BatchOpts): number |
Bulk edge insert. Returns rows inserted (conflicts excluded). Temporal types run refreshRelationships in a transaction. |
links.ts:216-240: executeRawJsonb(target, INSERT INTO links (...) SELECT f.id, t.id, v.* FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(...) JOIN pages f ON f.slug = v.from_slug AND f.source_id = v.from_source_id JOIN pages t ... LEFT JOIN pages o ... ON CONFLICT (...) DO NOTHING RETURNING ...). Wrapped in batchRetry. |
PGLite pglite-engine.ts:1948 and Postgres postgres-engine.ts:1768, both batchRetry. Same SQL. |
addLinksBatch: about 77 files. Parity: e2e/engine-parity.test.ts (addLinksBatch-based fixtures). PGLite: link-extraction-dir-whitelist-2576.test.ts, p2-acceptance.test.ts, extract-fs-source-id.test.ts, doctor-stale-mentions.test.ts. |
| 25 | replaceDerivedLinks (1522) |
(origin: DerivedLinkOrigin, links: LinkBatchInput[], opts?: DerivedLinkReplacementOptions): {created, removed} |
Atomically replaces all derived edges for one origin page: delete then insert, with page-key locks and an extraction-generation check (DerivedLinkSettingsChangedError). |
derived-links.ts:170-189: engine.transaction(tx => pipelined(primeRelationSemantics, lockPageKeys(...)); readPageSnapshot; publishDerivedLinks(...)). beliefs.commit(engine). |
Shared code path: both engines call replaceDerivedLinks(this, origin, links, opts) (pglite-engine.ts:1953, postgres-engine.ts:1773). No engine-specific SQL. |
replaceDerivedLinks: about 22 files. postgres-rls-scope-inventory.test.ts, link-cross-source.test.ts, attendance-retrieval.test.ts, extract-endpoint-revision.serial.test.ts, helpers/postgres-engine-sql-cases.ts. No parity test. |
| 26 | replaceDerivedLinksBatch (1524) |
(items: DerivedLinkBatchItem[]): {created, removed}[] |
Same as above for many origins in one transaction. Any failure rolls back all of them. | derived-links.ts:202-: shared prime, page guards (one sorted lock), extraction-generation check, origin reads, then per-origin publishDerivedLinks. |
Shared path (pglite-engine.ts:1957, postgres-engine.ts:1777). |
replaceDerivedLinksBatch: about 12 files. extract-endpoint-revision.serial.test.ts, helpers/extract-db-batch-fixture.ts, helpers/postgres-engine-sql-cases.ts, engine-surface-coverage.test.ts. |
| 27 | removeLink (1538) |
(from, to, linkType?, linkSource?, opts?: {fromSourceId, toSourceId}): number |
Deletes edges between a pair. Optional type and provenance filters. Returns rows deleted (#4527). | links.ts:306-: four branches (type and source, type only, source only, neither). Each is DELETE FROM links WHERE from_page_id = (SELECT id FROM pages WHERE slug=$ AND source_id=$) AND to_page_id = (...) [AND link_type = $] [AND link_source IS NOT DISTINCT FROM $] RETURNING from_page_id, to_page_id, link_type, then refreshRelationships. |
PGLite pglite-engine.ts:1977, Postgres postgres-engine.ts:1797. Same SQL. |
removeLink: about 17 files. PGLite: pglite-engine.test.ts, source-id-tx-regression.test.ts. Also managed-links-fence.test.ts, canonical-writer-inventory.test.ts, helpers/link-temporal-scenario.ts. |
| 28 | removeLinksByPagesAndSource (1564) |
(pages: {slug, source_id}[], opts: {linkSource, keepTypedNerPairs?}): number |
Bulk delete of derived links for a set of from pages, scoped to one link_source. Keeps typed_ner rows whose pair is in the keep list (#3674). |
links.ts:258-: WITH scope AS (SELECT f.id AS from_id FROM jsonb_to_recordset(($2::jsonb)->'pages') ... JOIN pages f ...), keep AS (SELECT ... FROM jsonb_to_recordset(...'keep') ...), DELETE FROM links l USING scope s WHERE l.from_page_id = s.from_id AND l.link_source = $1 AND NOT (COALESCE(l.link_kind,'')='typed_ner' AND EXISTS (SELECT 1 FROM keep k ...)) RETURNING l.from_page_id, l.to_page_id, l.link_type, then refreshRelationships. Bound via executeRawJsonb. |
PGLite pglite-engine.ts:1964, Postgres postgres-engine.ts:1784. Comments say "identical SQL shape" (parity). |
removeLinksByPagesAndSource: about 11 files. Parity: e2e/engine-parity.test.ts. PGLite: extract-by-mention-rebuild.test.ts, helpers/postgres-engine-sql-cases.ts, engine-surface-coverage.test.ts. |
Notes for section C:
- The engine.ts JSDoc for
addLinksBatch(engine.ts:1507-1511) says "single multi-row INSERT...SELECT FROM (VALUES) JOIN pages". The implementation isjsonb_to_recordset(links.ts:216-227, #1861). This doc is stale. - Batch pattern:
addLinksBatchandremoveLinksByPagesAndSourceboth bind their rows as one JSONB document viajsonb_to_recordset, notunnest. That avoids the 65535 parameter cap and the text[] literal crash (#1861). - No recursive CTE in the writes.
ON CONFLICTtargets the composite(from_page_id, to_page_id, link_type, link_source, origin_page_id)key.
D. Links: reads, graph, boosts (engine.ts 1585-1841)
Shared readers in src/core/search/read-enrichment.ts take a ReadQuery (the engine's executeRaw). The SQL in them is the same on both engines. The engines differ only in the transaction and planner wrapper around them, as described at the top.
| # | Method (engine.ts line) | Signature (compact) | What it does | SQL / delegation | PGLite vs Postgres | Tests |
|---|---|---|---|---|---|---|
| 29 | getLinks (1590) |
(slug, opts?: LinkReadScope): Link[] |
Outbound edges of a page. sourceId scopes the from page. sourceIds (#2200) constrains from, to and origin. Optional excludePrivate and temporal filters. |
links.ts:366-: SELECT f.slug AS from_slug ... FROM links l JOIN pages f ... JOIN pages t ... WHERE f.slug = $ AND <linkPageSql> .... Scope by sourceIds = ANY($::text[]) on all three endpoints. Optional TEMPORAL_LINK_SELECT_SQL. |
PGLite pglite-engine.ts:1987: scopedRead(this.engineSql), no tx. Postgres postgres-engine.ts:1807: withScopedReadTransaction(opts?.sourceIds, opts?.sourceId). |
getLinks: about 60 files. Parity: e2e/engine-parity.test.ts:1487-1528 (sourceIds[] three-endpoint, scalar cross-source, unscoped identity). PGLite: persistence-graph-publication.test.ts, extract-atoms-provenance-links.test.ts, sync-soft-delete.serial.test.ts, helpers/engine-sql-rollback-cases.ts. |
| 30 | getBacklinks (1596) |
(slug, opts?: LinkReadScope): Link[] |
Inbound edges of a page. Same scope rules as getLinks, applied to the to side. |
links.ts:437-: mirror of getLinks with t.slug = $. |
PGLite pglite-engine.ts:1991: plannerRead(this, this._pageTransaction, () => scopedRead(...)). Postgres postgres-engine.ts:1811: withScopedReadTransaction. The plannerRead wrapper is PGLite-only. |
getBacklinks: about 36 files. Parity: e2e/engine-parity.test.ts:1529 (sourceIds[] both endpoints), e2e/links-read-order.test.ts, e2e/federated-link-reads-postgres.test.ts, e2e/graph-quality.test.ts. |
| 31 | listLinkSources (1604) |
(opts?: {sourceId?, sourceIds?}): {link_source, count}[] |
Provenance counts, for gbrain link-sources (v114, #1941). An edge counts only when its from, to and live origin are in scope. |
links.ts:501-525: SELECT l.link_source, COUNT(*)::int FROM links l JOIN pages f ... JOIN pages t ... [WHERE scope] GROUP BY l.link_source ORDER BY count DESC, l.link_source ASC NULLS LAST. |
PGLite pglite-engine.ts:1995: scopedRead. Postgres postgres-engine.ts:1815: withScopedReadTransaction. Same SQL. |
listLinkSources: about 11 files. Parity: e2e/engine-parity.test.ts:983-1001 (ordered counts). Also helpers/postgres-engine-sql-cases.ts, engine-surface-coverage.test.ts. |
| 32 | findByTitleFuzzy (1629) |
`(name, dirPrefix?, minSimilarity=0.55, sourceId?): {slug, similarity} | null` | Best page whose title is similar to name (pg_trgm). Zero embedding or LLM cost. |
pages.ts:983-1031: WHERE title % $name AND similarity(title, $name) >= $min AND slug LIKE $prefix [AND source_id = $] AND deleted_at IS NULL ORDER BY sim DESC, slug ASC LIMIT 1. The % operator uses the trigram GIN index idx_pages_trgm when minSimilarity >= 0.3. |
Both: unscopedExecutor(this.engineSql, 'pages: unscoped on master (EO4 inventory)') (pglite-engine.ts:2007, postgres-engine.ts:1827). Same SQL. |
| 33 | traverseGraph (1643) |
(slug, depth=5, opts?: TraverseGraphOpts): GraphNode[] |
Breadth-first node walk from a seed. Each page appears once, at its shortest depth, with its outbound {to_slug, link_type} list. Scope (sourceId, sourceIds), privacy, live-origin and temporal filters apply. |
links.ts:659-709: WITH RECURSIVE bfs(depth, frontier, visited) AS (...), walk(id, depth, pos) AS (...). Built by bfsWalkCtes (links.ts:609-650): frontier/visited are int[] arrays, CROSS JOIN LATERAL unnest(bfs.frontier), anti-join NOT EXISTS (SELECT 1 FROM unnest(bfs.visited) v(id) ...), layer cap LIMIT least(frontierCap, budget), and total budget TRAVERSE_WALK_ROW_CAP + 1 - cardinality(visited) (20000). Final SELECT ... WHERE w.pos <= TRAVERSE_PATH_ROW_CAP, with jsonb_agg(DISTINCT ...) for outbound links. |
Both: unscopedExecutor(...) on pglite-engine.ts:2015 (wrapped in plannerRead) and postgres-engine.ts:1835 (not wrapped). Same SQL. |
traverseGraph: about 19 files. Parity: e2e/engine-parity.test.ts:2068-2118 (depth 1..3, cycle guard, sourceId and sourceIds). PGLite: pglite-engine.test.ts, traverse-walk-cap.test.ts, soft-delete-link-visibility.test.ts, e2e/source-isolation-pglite.test.ts. |
| 34 | traversePaths (1658) |
`(slug, opts?: PageReadScope & {depth=5, linkType?, direction='out' | 'in' | 'both'}): GraphPath[]` | Edge-based BFS. Returns each qualifying edge once, at one past the depth of its visited endpoint. | pglite-engine.ts:2018 and postgres-engine.ts:1838 both delegate to traversePathsDetailed(...).paths. No separate SQL. |
| 35 | traversePathsDetailed (1672) |
(slug, opts?: LinkReadScope & {depth, linkType, direction}): {paths: GraphPath[], truncated: boolean} |
Same walk as traversePaths, plus a truncated flag. The walk is bounded by TRAVERSE_WALK_ROW_CAP (20000). The final result by TRAVERSE_PATH_ROW_CAP (5000), counting raw rows before the in-memory dedup. |
links.ts:711: WITH RECURSIVE walk AS (bfs CTE from bfsWalkCtes with maxLayer = depth - 1), nodes AS (... WHERE w.pos <= CAP). The edges SELECT depends on direction: out joins links.from_page_id, in joins links.to_page_id, both UNIONs the two in a derived table e (links l joined on the frontier side and its depth). LIMIT CAP + 1, dedup in JS by `from_id |
to_id | link_type |
| 36 | relationalFanout (1697) |
(seeds: string[], opts?: RelationalFanoutOpts): RelationalFanoutRow[] |
Typed-edge fan-out for the relational recall arm (v0.43). From a seed array, returns reached nodes ranked hop ASC, edge_count DESC, source_id, slug, with via_link_types, the shortest path and edges, and a canonical_chunk_id. Within-source only. mentions excluded unless includeMentions. Non-ranking link types excluded. |
search/read-enrichment.ts:19-115: WITH RECURSIVE walk AS (SELECT seed rows ... ARRAY[p.id] AS visited, ARRAY[p.slug] AS path, ... UNION ALL SELECT p2 ... FROM walk w JOIN links l ... WHERE w.depth < $2 AND NOT (p2.id = ANY(w.visited)) AND p2.source_id = w.seed_source AND ... AND rankingLinkSql('l') AND mentionsFilter ...). The outer GROUP BY n.source_id, n.slug, n.id, MIN(depth), COUNT(DISTINCT last_link_type), array_agg ... ORDER BY depth, array_length(path), path. Depth is clamped to 1..3 (default 2), limit to 1..200 (default 50), direction defaults to both. Canonical chunk via a correlated LIMIT 1 subquery. |
PGLite pglite-engine.ts:2032: readRelationalFanout(this.executeRaw.bind(this), seeds, opts) (default chunkManifest = true). Postgres postgres-engine.ts:1852: passes await chunkManifestReads(this). On PGLite that returns true, so both are equivalent. |
relationalFanout: about 15 files. Parity: e2e/engine-parity.test.ts:1337-1400 (relationalFanout parity), relational-recall.test.ts, e2e/read-enrichment-privacy.test.ts, attendance-retrieval.test.ts, e2e/segmented-switch-off-rls.test.ts. |
| 37 | relationalChainHop (1710) |
(frontierPageIds: number[], opts: ChainHopOpts): ChainHopEdge[] |
One oriented expansion step of a multi-hop relational chain. Returns logical edges from frontier nodes, oriented by type signature (canonical, stored, flipped, uncertain). Caps per-frontier neighbors (neighborCap <= 1000). Computes from_degree with saturation at 300. |
search/read-enrichment.ts:117-241: WITH f AS (SELECT DISTINCT unnest(ARRAY[...]::int[]) AS id), cand AS (outBranch UNION inBranch), classified, semantic, stepped, logical (GROUP BY fid, next_id), ranked (ROW_NUMBER, COUNT OVER per frontier), deg_rows/deg (LATERAL, LIMIT 300). Frontier ids are inlined as an ARRAY[...]::int[] literal, not bound. Each branch uses OFFSET 0 as an optimization fence. This is not a recursive CTE: the multi-hop loop is in the caller (runRelationalChain). |
PGLite pglite-engine.ts:2039: readChainHop(this.executeRaw.bind(this), frontierPageIds, opts). Postgres postgres-engine.ts:1859: same, plus chunkManifestReads. Equivalent on PGLite. |
relationalChainHop: about 10 files. Parity: e2e/engine-parity.test.ts:1409-1486 (multi-hop chain identical across engines). relational-recall.test.ts, helpers/amends-ranking-scenarios.ts, relational-chain-hub.slow.test.ts. |
| 38 | getBacklinkCounts (1717) |
(pageIds: number[], opts?: PageReadScope): Map<number, number> |
Distinct linking pages per target page id, for the hybrid-search backlink boost. Mentions and non-ranking types excluded. Ids with zero links map to 0. Keyed by page id (#4380). | search/read-enrichment.ts:243-263: SELECT p.id AS page_id, COUNT(DISTINCT l.from_page_id)::int AS cnt FROM pages p LEFT JOIN links l ON l.to_page_id = p.id AND l.from_page_id <> p.id AND link_source IS DISTINCT FROM 'mentions' AND rankingLinkSql('l') [AND EXISTS contributor AND origin] WHERE p.id = ANY($1::int[]) AND <target filter> GROUP BY p.id. Single query. |
PGLite pglite-engine.ts:2043: readBacklinkCounts(this.executeRaw.bind(this), ...). Postgres postgres-engine.ts:1863: wraps the same call in jitOffRead with tx.unsafe(query, params, { prepare: true }) (JIT off, because compile time dominated at 50k pages, per the inline comment). |
getBacklinkCounts: about 19 files. Parity: e2e/engine-parity.test.ts (around the boost tests). e2e/jit-off-reads-postgres.test.ts, search/post-fusion-concurrent-reads.test.ts, search/metadata-boost-gate.test.ts, e2e/read-enrichment-privacy.test.ts, helpers/postgres-engine-sql-cases.ts. |
| 39 | getAdjacencyBoosts (1744) |
(pageIds: number[], opts?: PageReadScope): Map<number, {hits, cross_source_hits}> |
For a page-id set, returns in-set hits (distinct from-pages) and cross_source_hits (distinct other source ids), restricted to the induced subgraph. Returns only pages with hits >= 1. |
search/read-enrichment.ts:265-288: SELECT l.to_page_id, COUNT(DISTINCT l.from_page_id)::int AS hits, COUNT(DISTINCT CASE WHEN p.source_id <> t.source_id THEN p.source_id END)::int AS cross_source_hits FROM links l JOIN pages p ... JOIN pages t ... WHERE l.from_page_id = ANY(ARRAY[...]::int[]) AND l.to_page_id = ANY(ARRAY[...]::int[]) AND ... GROUP BY l.to_page_id HAVING COUNT(DISTINCT l.from_page_id) >= 1. Ids are inlined as an ARRAY[...]::int[] literal, not a bound parameter, to keep the planner off a generic plan (search/adjacency-generic-plan.test.ts). |
PGLite pglite-engine.ts:2047 and Postgres postgres-engine.ts:1869, both via readAdjacencyBoosts(this.executeRaw.bind(this), ...). Same SQL, no transaction wrapper on either. |
getAdjacencyBoosts: about 12 files. search/adjacency-generic-plan.test.ts, e2e/graph-signals-engine.test.ts, helpers/amends-ranking-scenarios.ts, helpers/postgres-engine-sql-cases.ts, search/post-fusion-concurrent-reads.test.ts. No parity test. |
| 40 | getContentFlagsByPageIds (1753) |
(pageIds: number[], opts?: PageReadScope): Map<number, {reason, detail}> |
frontmatter.content_flag per page (#1699). Post-fusion warning channel for search results. |
search/read-enrichment.ts:290-299: SELECT p.id, p.frontmatter->'content_flag'->>'reason', ...->>'detail' FROM pages p WHERE p.id = ANY($1::int[]) AND p.frontmatter ? 'content_flag' AND <filter>. |
PGLite pglite-engine.ts:2051, Postgres postgres-engine.ts:1873. Same readContentFlags. |
getContentFlagsByPageIds: about 10 files. e2e/read-enrichment-privacy.test.ts, helpers/postgres-engine-sql-cases.ts, quarantine-cli.test.ts, hybrid-arm-rethrow.test.ts. No parity test. |
| 41 | getUnverifiedExtractionPageIds (1769) |
(pageIds: number[], opts?: PageReadScope): Map<number, {unverified, status}> |
For every page with frontmatter.status, the raw status and whether it is an unverified auto-extracted stub (#160, #4220). |
search/read-enrichment.ts:301-309: SELECT p.id, p.frontmatter->>'status', (<unverifiedExtractionFragment('p')>) AS unverified FROM pages p WHERE p.id = ANY($1::int[]) AND p.frontmatter->>'status' IS NOT NULL AND <filter>. The predicate is unverifiedExtractionFragment (src/core/extraction-review.ts). |
PGLite pglite-engine.ts:2055, Postgres postgres-engine.ts:1877. Same SQL. |
getUnverifiedExtractionPageIds: about 11 files. e2e/extraction-review-postgres.test.ts, extraction-review.test.ts, helpers/postgres-engine-sql-cases.ts, hybrid-arm-rethrow.test.ts. No parity test. |
| 42 | getPageTimestamps (1781, @deprecated v0.29.1) |
(slugs: string[]): Map<string, Date> |
updated_at (or created_at) per slug. Kept for PR #618 callers. Not source-qualified. |
pages.ts:1033: SELECT slug, COALESCE(updated_at, created_at) ... WHERE slug = ANY($). |
Both: unscopedExecutor(this.engineSql, 'pages: unscoped on master (EO4 inventory)') (pglite-engine.ts:2059, postgres-engine.ts:1881). Same SQL. |
getPageTimestamps: about 10 files. title-fuzzy-index.test.ts, helpers/postgres-engine-sql-cases.ts, engine-surface-coverage.test.ts. No parity test. |
| 43 | getEffectiveDates (1791) |
(refs: {slug, source_id}[], opts?: PageReadScope): Map<'source::slug', Date> |
COALESCE(effective_date, updated_at, created_at) per (slug, source_id). Drives the recency boost (v0.29.1). |
search/read-enrichment.ts:311-320: SELECT p.slug, p.source_id, COALESCE(p.effective_date, p.updated_at, p.created_at) AS ts FROM pages p JOIN unnest($1::text[], $2::text[]) AS u(slug, source_id) ON p.slug = u.slug AND p.source_id = u.source_id WHERE <filter>. Two-array unnest join. Key format is ${source_id}::${slug}. |
PGLite pglite-engine.ts:2063, Postgres postgres-engine.ts:1885. Same SQL. |
getEffectiveDates: about 11 files. e2e/read-enrichment-privacy.test.ts, hybrid-recency-config.test.ts, search/metadata-boost-gate.test.ts, search/post-fusion-concurrent-reads.test.ts, search.test.ts. No parity test. |
| 44 | getSalienceScores (1801) |
(refs: {slug, source_id}[], opts?: PageReadPolicy): Map<'source::slug', number> |
Salience per (slug, source_id) = COALESCE(emotional_weight,0)*5 + ln(1 + count(distinct active takes)). With a takesHoldersAllowList, the emotional term is 0 and only takes from allowed holders count (search/read-enrichment.ts:322-345). |
search/read-enrichment.ts:322-345: SELECT p.slug, p.source_id, (COALESCE(p.emotional_weight,0)*5 + ln(1 + COUNT(DISTINCT t.id))) AS score FROM pages p JOIN unnest($1::text[], $2::text[]) AS u(...) ON ... LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE [AND t.holder = ANY($::text[])] WHERE <filter> GROUP BY p.id. |
PGLite pglite-engine.ts:2067, Postgres postgres-engine.ts:1889. Same SQL. The salience.ts file (getRecentSalience, listEnrichCandidates, findAnomalies) is a separate, non-interface path. |
getSalienceScores: about 11 files. Parity: e2e/engine-parity-salience.test.ts (dedicated parity file). Also e2e/read-enrichment-privacy.test.ts, search/metadata-boost-gate.test.ts, search.test.ts. |
| 45 | findOrphanPages (1830) |
`(opts?: PageReadScope & {sourceId?, sourceIds?, mode?: 'islanded' | 'inbound', scopeInbound?}): {slug, title, domain, type?, quarantined?, source_id?}[]` | Pages with no live inbound link (inbound mode) or with neither inbound nor live outbound (islanded, the default, #4524). Candidate set scoped by source; inbound counts from any source unless scopeInbound. |
links.ts:528-598: SELECT p.slug, COALESCE(p.title, p.slug), p.frontmatter->>'domain', p.type, (NOT <QUARANTINE>) AS quarantined, p.source_id FROM pages p WHERE p.deleted_at IS NULL [AND p.source_id = $] AND NOT EXISTS (SELECT 1 FROM links l JOIN pages src ... WHERE l.to_page_id = p.id AND src.deleted_at IS NULL [sourceScope] [liveOriginFilter]) [islanded: AND NOT EXISTS (outbound to live target)] ORDER BY p.slug. |
PGLite pglite-engine.ts:2071: wrapped in plannerRead(...). Postgres postgres-engine.ts:1893: not wrapped. Both pass unscopedExecutor(...). Same SQL. |
Notes for section D:
- Recursive CTEs:
traverseGraph(links.ts:659),traversePathsDetailed(links.ts:711) andrelationalFanout(read-enrichment.ts:19). The first two use the array-frontier BFS built bybfsWalkCtes(links.ts:609). The third walks paths withpathandvisitedarrays and clamps depth to 3.relationalChainHop,getBacklinkCounts,getAdjacencyBoostsand the rest are not recursive. - pgvector: none in this section.
getAdjacencyBoosts,getBacklinkCountsandrelationalChainHoprun onlinks/pagesjoins only. Canonical chunk ids come fromcontent_chunksthrough a correlatedLIMIT 1subquery. - Batch
unnestpatterns:getEffectiveDatesandgetSalienceScoresjoinunnest(text[], text[])on(slug, source_id).traverseGraph/traversePathsuseunnest(bfs.frontier)andunnest(bfs.visited)in array CTEs.relationalChainHopusesunneston an inlined id array. Other bulk reads useANY($n::int[]). - Read-policy seam: every read here authorizes endpoints, hops and origins before returning (
pageReadFilter,originFilterinread-enrichment.ts:11-16). Scoping bysourceId/sourceIdsis applied by the same helper inlinks.ts(sourceScope,links.ts:652).
Test coverage summary
- Parity (both engines,
DATABASE_URL-gated, runs PGLite half always):test/e2e/engine-parity.test.ts. Covers:addLink,getChunks,listLinkSources,findOrphanPages, the stale-extraction group (countStalePagesForExtraction,listStalePagesForExtraction,markPagesExtractedBatch),countChunklessPagesWithContent/listChunklessPagesWithContent,relationalFanout,relationalChainHop,getLinks,getBacklinks,traverseGraph,traversePaths,traversePathsDetailed,removeLinksByPagesAndSource,listStalePagesForExtraction. test/e2e/engine-parity-salience.test.tscoversgetSalienceScores.test/pglite-engine.test.tscovers PGLite-only behavior:upsertChunksround trip and mixed-case slugs (519-560),deleteChunks(565-572),countStaleChunks(597-703),removeLink,getChunks.- No parity or PGLite test directly names these methods:
getChunkPage(onlyget-chunks-paging.test.ts),getChunkWindows(evidence tests, Postgres RLS scope),sumStaleChunkChars,setPageEmbeddingSignature,invalidateStaleSignatureEmbeddings,invalidateContentDriftEmbeddings,readEmbeddingStateCounts,listEmbeddingStateRows,markPagesAttendanceBlocked,countStaleFrontmatterKeyPages,listStaleFrontmatterKeyPages,stripStaleFrontmatterKeys,replaceDerivedLinks,replaceDerivedLinksBatch,findByTitleFuzzy,getAdjacencyBoosts,getContentFlagsByPageIds,getUnverifiedExtractionPageIds,getPageTimestamps,getEffectiveDates,getBacklinkCounts(the last is covered bye2e/jit-off-reads-postgres.test.ts). Several of these are only exercised throughtest/helpers/postgres-engine-sql-cases.tsandtest/engine-surface-coverage.test.ts(inventory-style tests).
Parity risks noted
- Planner wrapper asymmetry. PGLite wraps
getBacklinks,traverseGraph,traversePathsDetailed,traversePaths' inner call andfindOrphanPagesinplannerRead(pglite-engine.ts:1992, 2015, 2029, 2077), which can run planner-stat refreshes. Postgres does not. A Rust engine has no equivalent, so timing and plan behavior differ and nothing in the tests pins it. - Transaction and RLS scope asymmetry. Postgres wraps scoped reads in
withScopedReadTransaction(postgres-engine.ts:368), which sets the source/RLS scope. PGLite runs the same SQL on the bare connection. The source-scope predicates are in SQL in both, so behavior matches, but a Rust engine must enforce the same scope in its own read path. - JIT off on Postgres only.
getBacklinkCountsruns injitOffReadon Postgres (postgres-engine.ts:1863). PGLite has no JIT, so this is a performance-only divergence, but the inline comment says plan compilation dominated at 50k pages. - Generic-plan sensitivity.
getAdjacencyBoostsandrelationalChainHopinline integer id arrays and useOFFSET 0fences, to keep the planner from choosing a generic plan (search/read-enrichment.ts:117-130, 265-288). A Rust engine has no plan cache, so these are not needed, but the output ordering they rely on must hold. - Traversal caps.
TRAVERSE_WALK_ROW_CAP= 20000 andTRAVERSE_PATH_ROW_CAP= 5000 (engine-constants.ts:37, 48) bound the walk, andtraversePathsDetailedreportstruncated. The cap counts raw rows before dedup, so a Rust version must apply the cap at the same point or thetruncatedflag changes. - Traversal ordering.
traversePathsorders bydepth, from_slug, to_slug, from_source_id, to_source_id.traverseGraphorders bydepth, slug, source_id.relationalFanoutorders byhop ASC, edge_count DESC, source_id, slug. Ties and collation (ORDER BY slug) must match on both engines. - Stale
addLinksBatchJSDoc.engine.ts:1507-1511says "INSERT...SELECT FROM (VALUES)". The implementation isjsonb_to_recordset(links.ts:216-227, #1861). The doc is wrong and should not be copied. getEffectiveDatesJSDoc contradicts itself.engine.ts:1788-1790says "Returns NULL for refs with no row; map omits them." The implementation omits them (read-enrichment.ts:311-320). Callers must treat a missing key as absent.- Stacked JSDoc on
findOrphanPages.engine.ts:1802-1829has two JSDoc blocks in sequence; themodesemantics (#4524) sit above a separate block. TheexcludePrivateoption is accepted by both implementations but missing from the interface type (engine.ts:1830-1834). - Unscoped master reads.
sumStaleChunkChars,findByTitleFuzzy,traverseGraph,traversePathsDetailed,findOrphanPages,getPageTimestampsuseunscopedExecutor(..., 'EO4 inventory')on both engines. Their scope comes from the caller's options, not the engine. Any Rust read path that enforces scope centrally would change these results. - Segmented-body chunk visibility.
getChunks,getChunkPage,getChunkWindows,countStaleChunks,listStaleChunksandrelationalFanoutall gate onchunkGenerationFilter/chunkGenerationProbe(search/safe-chunks.ts:77-89). It readspage_body_manifest(the segmented-body manifest), and the flagchunkManifestonly changes behavior on Postgres (safe-chunks.ts:107-113). Rust must replicate the manifest semantics exactly, or segmented pages will read differently. - Content-drift sweep has no saved cursor.
invalidateContentDriftEmbeddingsloops in 5000-row transactions with no saved cursor (chunks.ts:612, 685-698). A crash mid-loop re-examines rows, which is safe by design (the predicate is idempotent) but the batch boundary is observable in the returned count. - Upsert is retried as a whole.
upsertChunksandaddLinksBatchare wrapped inbatchRetry(pglite-engine.ts:1814-1815, 1950;postgres-engine.ts:1628-1629, 1770). The engine doc says "callers MUST NOT wrap externally." A Rust engine that retries differently changes the commit-ambiguous replay story (engine.ts:1248-1253, 1512-1520). - Embedding column routing. Every chunk read and stale count resolves the active column by a registry lookup (
resolveActiveEmbeddingColumnFromEngine,search/embedding-column.ts:694), and the write path resolves it again from config rows inside the upsert. A fallback to the legacyembeddingcolumn on a broken registry (fallbackToLegacy: true) is a silent behavior change for diagnostics and reads. - Halfvec and dimension casts.
upsertChunkswrites::halfvec(N)or::vectordepending on the resolved column (search/embedding-column.ts:279-283). A Rust store must keep the same dimension and type rules, or writes to a registry-routed brain fail with dimension mismatch errors.
Tags, Timeline, Chronicle, Ontology, Raw data, Files, Takes, Dream and Contradictions
Scope: the 43 BrainEngine methods declared in src/core/engine.ts:1842-2195 (interface). No method in this range is optional (no name?() declarations). Paths below are relative to the gbrain repo root the gbrain repo root. Line numbers are file:line in that repo.
How delegation works in this range:
- Interface:
src/core/engine.ts:1842-2194. - Engine methods are implemented twice, once in
src/core/pglite-engine.tsand once insrc/core/postgres-engine.ts. Most methods are thin wrappers that callsrc/core/engine-sql/*.ts, which hold one shared SQL text for both engines (engine-sql/takes.ts:1-9,timeline.ts:1-11,tags.ts:1-7,files.ts:1-7). - Two families are not in engine-sql and are inlined per engine: ontology (
mergeOntologyFact,getOntology,discoverOntologyDimensions,findOntologyConflicts),putRawData/getRawData, and the dream verdict cache. Their SQL text is written twice, once withdb.query($n)in PGLite and once with postgres.js tagged templates in Postgres. - Reads in engine-sql take a
LegacyUnscopedReadbrand (engine-sql/brands.ts:33), meaning master ran them on the pool with no RLS scope transaction. Both engines obtain it viaunscopedExecutor(...)(brands.ts:49). - Dialect executors:
engine-sql/dialect-pglite.ts:58-76andengine-sql/dialect-postgres.ts:68-94. PGLite binds JSONB params withJSON.stringify(dialect-pglite.ts:33). Postgres binds them withconn.json(dialect-postgres.ts:61).
Shared helpers referenced below:
mutatePageTaginsrc/core/page-state/tags.ts:5-22.- Ontology helpers in
src/core/chronicle/ontology.ts:valueHash(:12),normalizeDimension(:32),isNovelDimension(:43),isBackdatedObservation(:54),ONTOLOGY_SUPERSEDE_GUARD(:65). deletedPageProjectionHiddenSqlinsrc/core/eligibility/sql.ts:55.privatePagesFilterForCaller/privateFactFilterForCallerinsrc/core/search/private-visibility.ts:611and:992.buildTakeRows/buildTimelineRowsinsrc/core/batch-rows.ts:177and:161.finalizeScorecard/deriveResolutionTupleinsrc/core/takes-resolution.ts:90and:22.finalizeLastSeeninsrc/core/chronicle/last-seen.ts:11.DREAM_VERDICT_TTL_SECONDS = 30 * 86400atsrc/core/engine.ts:528.
Engine line map (PGLite / Postgres): every method is declared in both engine files, so there are no PGLite-only or Postgres-only methods in this range.
Tags
| Method (iface line) | Signature | What it does | SQL / impl |
|---|---|---|---|
addTag (1851) |
addTag(slug, tag, opts?: {sourceId?, tagSource?: 'frontmatter'}): Promise<void> |
Adds a tag to a page. Idempotent. An explicit add claims the row. | PGLite pglite-engine.ts:2081, Postgres postgres-engine.ts:1903. Both call mutatePageTag in one transaction. Inside it: lockPageKeys, then SELECT id FROM pages WHERE source_id=$1 AND slug=$2. Missing page throws addTag failed: page ... not found. Then INSERT INTO tags(page_id,tag,tag_source) VALUES(...,'added') ON CONFLICT (page_id,tag) DO UPDATE SET tag_source='added'. With tagSource:'frontmatter' it inserts 'frontmatter' and updates only WHERE tags.tag_source IS NULL (page-state/tags.ts:13-17). |
removeTag (1852) |
removeTag(slug, tag, opts?: {sourceId?}): Promise<void> |
Removes one tag. | Same mutatePageTag path with add=false (page-state/tags.ts:18-20): DELETE FROM tags WHERE page_id=(SELECT id FROM pages WHERE source_id=$1 AND slug=$2) AND tag=$3. No existence check, so a missing page or tag is a silent no-op. |
getTags (1859) |
getTags(slug, opts?: {sourceId?, sourceIds?, excludePrivate?, liveOnly?}): Promise<string[]> |
Returns sorted distinct tag names for a slug. sourceIds[] wins over sourceId. Unscoped reads default to source 'default'. |
engine-sql/tags.ts:12-34. SELECT DISTINCT tag FROM tags WHERE page_id IN (SELECT id FROM pages WHERE slug=$ AND <scope> <privacy> <live>) ORDER BY tag. Scope is source_id = ANY($::text[]) or source_id = $. IN (not =) is used so a federated slug in several sources does not throw. excludePrivate adds privatePagesFilterForCaller('pages'). liveOnly adds deleted_at IS NULL. Read is LegacyUnscopedRead (tags.ts:6). |
Parity: both engines call tagsImpl.getTags with the same SQL. addTag/removeTag are the same shared function. The dialect difference is only the lock call. PGLITE_CAPABILITIES.transactionAdvisoryLocks=false (dialect-pglite.ts:27) and Postgres sets it to true (dialect-postgres.ts:55). I did not trace what lockPageKeys does on each dialect.
Timeline
| Method (iface line) | Signature | What it does | SQL / impl |
|---|---|---|---|
addTimelineEntry (1880) |
addTimelineEntry(slug, entry: TimelineInput, opts?: {skipExistenceCheck?, sourceId?}): Promise<boolean> |
Inserts one timeline row. Returns true if inserted, false if deduped. Throws PageMissingError when the page is missing and skipExistenceCheck is not set. |
PGLite pglite-engine.ts:2094, Postgres postgres-engine.ts:1916, both call engine-sql/timeline.ts:28-67. One CTE: page_state selects pages by (slug, source_id) FOR KEY SHARE, then inserted does INSERT INTO timeline_entries(page_id,date,source,summary,detail) ... ON CONFLICT (page_id,date,md5(summary),source) DO NOTHING RETURNING 1. Free-text fields go through sanitizeForJsonb. Dedup index is idx_timeline_dedup (schema.sql:670). |
addTimelineEntriesBatch (1897) |
addTimelineEntriesBatch(entries: TimelineBatchInput[], opts?: BatchOpts): Promise<number> |
Bulk insert. Returns count of rows actually inserted. Empty input returns 0 with no query. | Both engines: batchRetry(...) wrapper (pglite-engine.ts:2102, postgres-engine.ts:1924). Impl timeline.ts:70-90: executeRawJsonb over jsonb_to_recordset(($1::jsonb)->'rows') AS v(slug,date,source,summary,detail,source_id), JOIN pages p ON p.slug=v.slug AND p.source_id=v.source_id AND p.deleted_at IS NULL, ON CONFLICT (page_id,date,md5(summary),source) DO NOTHING RETURNING 1. Rows built by buildTimelineRows (batch-rows.ts:161). |
getTimeline (1898) |
getTimeline(slug, opts?: TimelineOpts): Promise<TimelineEntry[]> |
Entries for a slug, newest first. Supports after, before, limit (default 100), excludePrivate, eligibility, sourceIds/sourceId. |
timeline.ts:93-126. FROM timeline_entries te JOIN pages p ON p.id=te.page_id when scoped. When unscoped it uses sources s CROSS JOIN LATERAL (SELECT * FROM pages WHERE source_id=s.id AND slug=$ OFFSET 0) p JOIN timeline_entries te (:114-118). The OFFSET 0 fences the planner from driving off pages (comment cites Cat7-1). Ordering: te.date DESC, te.id DESC LIMIT n. Read is LegacyUnscopedRead. |
Parity: both engines share engine-sql/timeline.ts. The addTimelineEntry CTE with FOR KEY SHARE and the getTimeline LATERAL are Postgres planner or locking features. PGLite runs them on its single-connection WASM Postgres, so the same SQL text is used and the locking is not exercised the same way.
Life Chronicle (timeline reads)
All five filter pages.deleted_at IS NULL on the depth page and hide soft-deleted event projections. The shared shape is in timeline.ts:129-143 (chronicleSourceCond). It applies a source scope: sourceIds first, then sourceId, applied to p or to the event alias ep (ep=true). excludePrivate adds privatePagesFilterForCaller('p') and privateTimelineEventFilterForCaller('te').
| Method (iface line) | Signature | What it does | SQL / impl |
|---|---|---|---|
getTimelineForDate (1904) |
getTimelineForDate(date: string, opts?: ChronicleTimelineOpts): Promise<ChronicleTimelineRow[]> |
Rows for one day, or the ISO week when opts.week. Limit default 200. |
timeline.ts:146-166. Week bounds are date_trunc('week', d)::date through +6 days, else d::date. Joins timeline_entries te JOIN pages p ... AND p.deleted_at IS NULL LEFT JOIN pages ep ON ep.id=te.event_page_id. Filter (te.event_page_id IS NULL OR ep.deleted_at IS NULL). Order COALESCE(ep.effective_date, te.date::timestamptz) ASC, te.id ASC. Returns kind from ep.frontmatter->'event'->>'kind'. |
getSince (1906) |
getSince(date: string, opts?: ChronicleTimelineOpts): Promise<ChronicleTimelineRow[]> |
Rows on or after date. Optional kind filter. Limit 200. |
timeline.ts:169-188. Same join and order as above. Adds te.date >= $date::date and an optional ep.frontmatter->'event'->>'kind' = $kind. |
getOnThisDay (1908) |
getOnThisDay(opts?: PageReadScope & {date?, limit?}): Promise<ChronicleTimelineRow[]> |
Same month and day as date (default today) in prior years only. Limit 50. |
timeline.ts:191-211. EXTRACT(MONTH/DAY FROM te.date) = EXTRACT(... FROM target), te.date < target. target is $date::date or current_date (DB session clock). Order te.date DESC, te.id ASC. |
getLastSeen (1910) |
getLastSeen(entitySlug, opts?: PageReadScope & {asof?}): Promise<LastSeenResult> |
Most recent date an entity is seen: its own page's timeline, or an event whose who array names it exactly or by wikilink ([[slug]], `[[slug |
label]]`). Substring matches are excluded. |
upsertEventProjection (1917) |
upsertEventProjection(opts: {depthSlug, eventSlug, date, summary, detail?, sourceId?}): Promise<{projected: boolean}> |
Upserts the date-index row for an event page, keyed (event_page_id, date). Returns projected=false when either slug is missing. |
timeline.ts:256-269. `INSERT INTO timeline_entries(page_id,date,source,summary,detail,event_page_id) SELECT dp.id, $date::date, 'life-chronicle:event:' |
Parity for this family: PGLite (pglite-engine.ts:2111-2131) and Postgres (postgres-engine.ts:1933-1952) both call timelineImpl with the same unscoped executor. Both getOnThisDay and getLastSeen depend on the database session date (current_date), not on the process clock. getLastSeen's days_ago uses the JS clock. Parity risk is listed at the end.
Ontology (rides facts)
Ontology is inlined per engine, not in engine-sql. The shared logic is in chronicle/ontology.ts. The facts table is defined elsewhere in this research set. Here the relevant columns are source_id, entity_slug, dimension, value, value_hash, dim_status, confidence, source_markdown_slug, valid_from, valid_until, expired_at, superseded_by, consolidated_into, visibility, trust_tier.
| Method (iface line) | Signature | What it does | SQL / impl |
|---|---|---|---|
mergeOntologyFact (1928) |
mergeOntologyFact(obs: OntologyObservationInput): Promise<OntologyMergeResult> |
Records one observation (entity has dimension=value). Returns action of noop, corroborated, superseded_prior or inserted. Defaults: confidence 0.7, visibility private, status quarantined for novel dimensions, else active. |
PGLite pglite-engine.ts:2131-2198 (this.db.query). Postgres postgres-engine.ts:1953-2023 (tagged templates). Steps: (1) normalize dimension, valueHash (ontology.ts:12-32); (2) find current open row: expired_at IS NULL AND valid_until IS NULL AND (dim_status IS NULL OR dim_status='active'), ORDER BY valid_from DESC NULLS LAST, confidence DESC, id DESC LIMIT 1; (3a) same value and not backdated: check for an existing row from the same provenance (source_markdown_slug) at or after current valid_from; if found, noop; else insert a row with expired_at=now() and consolidated_into=current.id, ON CONFLICT (source_id,entity_slug,dimension,value_hash,source_markdown_slug,valid_from) WHERE dimension IS NOT NULL DO NOTHING RETURNING id, giving corroborated; (3b) otherwise insert a new row, then if current is open and status is active and the new valid_from is not earlier, close the prior row with UPDATE facts SET valid_until=COALESCE($validFrom, now()), superseded_by=$new WHERE id=$cur AND valid_until IS NULL + ONTOLOGY_SUPERSEDE_GUARD, giving superseded_prior, else inserted. A backdated conflicting value is inserted without touching the prior row. |
getOntology (1930) |
getOntology(entitySlug, opts?: OntologyReadOpts): Promise<OntologyValue[]> |
Current resolved value per dimension at asof (default now). Honors minConfidence (default 0), includeQuarantined, sourceId/sourceIds, visibility[], excludePrivate. |
PGLite pglite-engine.ts:2200-2227, Postgres postgres-engine.ts:2025-2050. SELECT DISTINCT ON (dimension) dimension, value, confidence, source_markdown_slug AS source, valid_from, valid_until AS valid_to, COALESCE(dim_status,'active') AS status, id AS fact_id FROM facts WHERE entity_slug=$ AND dimension IS NOT NULL AND expired_at IS NULL <scope> <deletedPageHidden> [<privateFactFilter>] <visibility> AND valid_from <= asof AND valid_until > asof AND confidence >= $min AND ($includeQ OR dim_status IS NULL OR dim_status='active') ORDER BY dimension, valid_from DESC NULLS LAST, confidence DESC, id DESC. Page-visibility gate runs before DISTINCT ON (comment at postgres-engine.ts:2033). |
discoverOntologyDimensions (1932) |
discoverOntologyDimensions(opts?: {sourceId?, sourceIds?}): Promise<OntologyDimensionStat[]> |
Lists every dimension in the brain with entity and observation counts. | PGLite pglite-engine.ts:2229-2244, Postgres postgres-engine.ts:2052-2063. SELECT dimension, count(DISTINCT entity_slug)::int AS entities, count(*)::int AS observations FROM facts WHERE dimension IS NOT NULL AND expired_at IS NULL <scope> GROUP BY dimension ORDER BY entities DESC, dimension. No privacy filter in this query. |
findOntologyConflicts (1934) |
findOntologyConflicts(opts?: PageReadScope & {minConfidence?, visibility?}): Promise<OntologyConflict[]> |
Dimensions with at least 2 distinct current values from at least 2 distinct provenances. | PGLite pglite-engine.ts:2246-2270, Postgres postgres-engine.ts:2065-2091. CTE cur selects open rows (expired_at IS NULL AND valid_until IS NULL AND dim_status active, confidence >= $min, scope, page gates, visibility). Then GROUP BY entity_slug, dimension HAVING count(DISTINCT value) >= 2 AND count(DISTINCT source) >= 2, returning json_agg(json_build_object('value',..,'source',..,'confidence',..,'fact_id',..)). |
Parity: the logic is identical. Driver differences only. PGLite builds params as an array with $n placeholders and a dynamic scope. Postgres uses sql\...`fragments andsql.unsafe()for the privacy fragment. PGLite parsesjson_aggtext withJSON.parse when needed (pglite-engine.ts:2272`).
Raw data
| Method (iface line) | Signature | What it does | SQL / impl |
|---|---|---|---|
putRawData (1943) |
putRawData(slug, source, data: object, opts?: {sourceId?}): Promise<void> |
Upserts one raw_data sidecar row per (page, source). Throws if the page is missing. |
PGLite pglite-engine.ts:2277-2320, Postgres postgres-engine.ts:2093-2128. With sourceId: INSERT INTO raw_data(page_id,source,data) SELECT id,$2,$3::jsonb FROM pages WHERE slug=$1 AND source_id=$4 ON CONFLICT (page_id,source) DO UPDATE SET data=EXCLUDED.data, fetched_at=now() RETURNING id. Without sourceId: the same without the source filter (matches every same-slug page; comment at postgres-engine.ts:2101, pglite-engine.ts:2284). Zero rows throws putRawData failed: page ... not found. The two engines differ only in JSON binding: PGLite JSON.stringify with $3::jsonb, Postgres sql.json. |
getRawData (1949) |
getRawData(slug, source?, opts?: PageReadScope & {includeDeleted?}): Promise<RawData[]> |
Returns {source, data, fetched_at} rows for a slug. Hides rows of soft-deleted pages unless includeDeleted. |
PGLite pglite-engine.ts:2322-2352 builds a dynamic WHERE. Postgres postgres-engine.ts:2130-2168 has explicit branches. Both produce FROM raw_data rd JOIN pages p ON p.id=rd.page_id WHERE p.slug=$1 [AND rd.source=$] [AND p.deleted_at IS NULL] [AND p.source_id = ANY/=] [AND privatePagesFilter]. sourceIds[] wins over sourceId. Read is unscoped on master, not LegacyUnscopedRead in the engine (this method has no engine-sql home). |
Files
Files are binary asset metadata only. Image bytes never enter the DB. Shared SQL is in engine-sql/files.ts, used by both engines. Schema: files (schema.sql:1136-1149), UNIQUE(storage_path).
| Method (iface line) | Signature | What it does | SQL / impl |
|---|---|---|---|
upsertFile (1954) |
upsertFile(spec: FileSpec): Promise<{id, created}> |
Upserts a file row keyed on storage_path. created is true on insert. |
files.ts:50-68. INSERT INTO files(source_id,page_slug,page_id,filename,storage_path,mime_type,size_bytes,content_hash,metadata) VALUES(...) ON CONFLICT (storage_path) DO UPDATE SET page_slug, page_id, filename, mime_type, size_bytes, content_hash, metadata = EXCLUDED.* RETURNING id, (xmax = 0) AS created. source_id defaults to 'default', metadata to {} via jsonbParam. The DO UPDATE list does not touch source_id. |
getFile (1955) |
getFile(sourceId, storagePath): Promise<FileRow | null> |
Looks up one file by source and path. | files.ts:70-78. SELECT ... FROM files WHERE source_id=$1 AND storage_path=$2 LIMIT 1. Read is LegacyUnscopedRead. |
listFilesForPage (1956) |
listFilesForPage(pageId: number): Promise<FileRow[]> |
Files attached to a page. | files.ts:80-88. SELECT ... FROM files WHERE page_id=$1 ORDER BY created_at ASC. |
Both engines: pglite-engine.ts:2354-2364 and postgres-engine.ts:2170-2180 delegate to filesImpl. Engine-level difference: none in SQL. The write path uses this.engineSql on both.
Takes (typed, weighted, attributed claims)
Takes live in table takes, keyed (page_id, row_num), with active, superseded_by, resolved_* columns and optional embedding (pgvector) plus embedding_model, embedded_text_hash, embedded_at. Takes carry no source_id; source scope goes through the page join. Engine-level methods are one-liners that delegate to engine-sql/takes.ts for both engines (PGLite pglite-engine.ts:2571-2706, Postgres postgres-engine.ts:2430-2562). The addTakesBatch / updateTakeEmbeddings wrappers pass batchRetry to the impl.
| Method (iface line) | Signature | What it does | SQL / impl (engine-sql/takes.ts) |
|---|---|---|---|
addTakesBatch (1978) |
addTakesBatch(rows: TakeBatchInput[], opts?: BatchOpts): Promise<number> |
Bulk insert or update takes. Returns count of rows inserted, updated, or already current. Weight is clamped to [0,1] and rounded, with a TAKES_WEIGHT_CLAMPED stderr line. |
:47-102. buildTakeRows (batch-rows.ts:177) clamps weight and NUL-strips claim and source. Then executeRawJsonb: INSERT INTO takes(...) SELECT ... FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(...) ON CONFLICT (page_id,row_num) DO UPDATE SET embedding/embedded_at/embedding_model/embedded_text_hash = NULL when claim changes, claim, kind, holder, weight, since_date, until_date, source, superseded_by, active, updated_at=now() WHERE (...) IS DISTINCT FROM (...). A row that is already current is not written. An intra-batch duplicate (page_id,row_num) fails the statement (:62-63). |
updateTakeEmbeddings (1981) |
updateTakeEmbeddings(rows: TakeEmbeddingInput[], opts?: BatchOpts): Promise<number> |
Stores vectors for active takes. Vectors whose claim changed in flight are discarded. Returns count updated. | :458-510. Validates take_id (positive int, no duplicates), claim and model strings, finite floats. UPDATE takes AS t SET embedding=v.embedding::vector, embedded_at=now(), embedding_model=v.model, embedded_text_hash=md5(v.claim), updated_at=now() FROM jsonb_to_recordset(...) AS v(take_id bigint, embedding text, claim text, model text) WHERE t.id=v.take_id AND t.active AND t.claim=v.claim RETURNING t.id. |
listTakes (1984) |
listTakes(opts?: TakesListOpts): Promise<Take[]> |
Filtered list of takes with page_slug. Default active=true, limit 100 (max 500), offset. |
:294-341. FROM takes t JOIN pages p. Filters: page_id, page_slug, holder, kind, active (NULL means any), resolved (true/false/undefined), takesHoldersAllowList (holder = ANY), sinceOnOrBefore (text compare with partial-date padding), excludePrivate, sourceIds/sourceId (p.source_id), eligibility (projectionEligibleSql). Sort: ungradedFirst (EXISTS in take_grade_cache), then sortBy in since_date / weight / created_at (default), with oldestFirst, tie-break t.id DESC. |
searchTakes (1991) |
searchTakes(query, opts?): Promise<TakeHit[]> |
Keyword (trigram) search over active take claims. Limit 30 (max 100). | :343-371. word_similarity(query, t.claim)::real AS score FROM takes t JOIN pages p WHERE t.active AND $query <% t.claim [privacy] [holder allow-list] [source scope] [eligibility] ORDER BY score DESC, t.weight DESC LIMIT n. Uses the pg_trgm <% operator. Rows go through takeHitRowToHit (BigInt coercion). |
searchTakesVector (1997) |
searchTakesVector(embedding: Float32Array, opts?): Promise<TakeHit[]> |
Cosine search over stored take vectors. Limit 30 (max 100). | :373-406. Vector is rendered as [v1,v2,...]. SELECT ..., (1 - (t.embedding <=> $vec::vector))::real AS score FROM takes t JOIN pages p WHERE t.active AND t.embedding IS NOT NULL [privacy] [holder] [source] [eligibility] ORDER BY t.embedding <=> $vec::vector LIMIT n. Rows with no embedding simply do not match, so the result is [] when none are embedded. No dimension check in SQL. |
getTakeEmbeddings (2003) |
getTakeEmbeddings(ids: number[]): Promise<Map<number, Float32Array>> |
Vectors by take id. Mirrors getEmbeddingsByChunkIds. |
:408-419. SELECT id, embedding FROM takes WHERE id = ANY($ids::bigint[]) AND embedding IS NOT NULL. Parsed with tryParseEmbedding. Returns an empty Map for empty input. |
countStaleTakes (2006) |
countStaleTakes(opts?: StaleTakeOpts): Promise<number> |
Count of stale takes (missing vector, claim drift, or other model or width when the caller names a target). | :427-444. Shared predicate staleTakeWhere (:427-436): t.active AND p.deleted_at IS NULL AND EXISTS(sources s WHERE s.id=p.source_id AND NOT s.archived) AND (sourceId filter) AND (t.embedding IS NULL OR t.embedded_text_hash IS DISTINCT FROM md5(t.claim) OR (model given AND embedding_model IS DISTINCT FROM model) OR (dims given AND vector_dims(embedding) <> dims)). Read is LegacyUnscopedRead. |
listStaleTakes (2009) |
listStaleTakes(opts?: StaleTakeOpts): Promise<StaleTakeRow[]> |
Same predicate, returns take_id, page_slug, row_num, claim. No vector in payload. |
:446-456. Same predicate, ORDER BY t.id LIMIT 100000. The hard cap is silent. |
updateTake (2016) |
updateTake(pageId, rowNum, fields: {weight?, since_date?, source?}): Promise<void> |
Mutates weight, since_date or source. Cannot change claim, kind or holder. | :512-538. Weight is normalized with normalizeWeightForStorage (clamp, stderr note). UPDATE takes SET weight=COALESCE($w::real, weight), since_date=COALESCE(...), source=COALESCE(...), updated_at=now() WHERE page_id=$ AND row_num=$ RETURNING 1. Zero rows throws TAKE_ROW_NOT_FOUND. |
supersedeTake (2030) |
supersedeTake(pageId, oldRow, newRow): Promise<{oldRow, newRow}> |
Transactionally marks the old row inactive and appends a new row at MAX(row_num)+1. Refuses resolved takes. |
:540-569, inside exec.transaction. (1) SELECT resolved_at FROM takes WHERE page_id AND row_num=oldRow: missing throws TAKE_ROW_NOT_FOUND, resolved throws TAKE_RESOLVED_IMMUTABLE. (2) SELECT COALESCE(MAX(row_num),0)+1 AS next FROM takes WHERE page_id=$. (3) INSERT INTO takes(...) VALUES(...), weight clamped to [0,1] (default 0.5). (4) UPDATE takes SET active=false, superseded_by=newRow, updated_at=now() WHERE page_id AND row_num=oldRow. No row lock is taken for the MAX step. |
resolveTake (2047) |
resolveTake(pageId, rowNum, resolution: TakeResolution): Promise<void> |
Sets the resolved state on a take. Immutable once set. | :571-592. Missing row throws TAKE_ROW_NOT_FOUND. Already resolved throws TAKE_ALREADY_RESOLVED. deriveResolutionTuple (takes-resolution.ts:22) maps quality (correct, incorrect, partial, unresolvable) to outcome (true, false, null). quality wins if both are set and disagree, and throws. UPDATE takes SET resolved_at=now(), resolved_quality, resolved_outcome, resolved_value, resolved_unit, resolved_source, resolved_by, updated_at=now() WHERE page_id AND row_num. |
getScorecard (2062) |
getScorecard(opts: TakesScorecardOpts, allowList: string[] | undefined): Promise<TakesScorecard> |
Aggregate calibration: totals, accuracy, Brier, partial rate, unresolvable rate. | :599-638. One aggregate SELECT: COUNT(*) FILTER (WHERE kind='bet') as total_bets, FILTER (resolved_quality IN ('correct','incorrect','partial')) as resolved, FILTER counts for correct, incorrect, partial, unresolvable, and AVG(POWER(weight - (correct?1:0),2)) over correct and incorrect as brier. Filters: holder, domainPrefix (EXISTS page slug LIKE prefix%), since/until (on since_date), allow-list (holder = ANY, inside the GROUP BY scope, so hidden holders contribute zero), source scope, excludePrivate. finalizeScorecard (takes-resolution.ts:90): accuracy = correct/(correct+incorrect), brier null if no binary bet, partial_rate = partial/resolved, unresolvable_rate = unresolvable/(resolved+unresolvable). |
getCalibrationCurve (2070) |
getCalibrationCurve(opts: CalibrationCurveOpts, allowList): Promise<CalibrationBucket[]> |
Bins resolved correct and incorrect bets by stated weight. Returns observed hit rate and predicted mean per bucket. Excludes partial. | :650-694. bucketSize clamped to (0,1], default 0.1. maxIdx = floor(1/bucketSize)-1. CTE binned: bucket_idx = LEAST(FLOOR(weight::numeric / bucketSize::numeric)::int, maxIdx), hit = (resolved_quality='correct')::int, over resolved_quality IN ('correct','incorrect') with holder, allow-list, source and privacy filters. Output bucket_lo, bucket_hi (bucket_idx*size as numeric then ::float), n = COUNT(*), observed = AVG(hit), predicted = AVG(weight). Only buckets that contain rows are returned. NUMERIC arithmetic is used on purpose so bucket boundaries like 0.7/0.1 do not drift (comment :661-665). |
addSynthesisEvidence (2073) |
addSynthesisEvidence(rows: SynthesisEvidenceInput[]): Promise<number> |
Records which takes a synthesis page cited. Idempotent. Returns number inserted. | :696-712. INSERT INTO synthesis_evidence(synthesis_page_id,take_page_id,take_row_num,citation_index) SELECT v.* FROM unnest($int[], $int[], $int[], $int[]) AS v(...) ON CONFLICT (synthesis_page_id,take_page_id,take_row_num) DO NOTHING RETURNING 1. Returns result.length. |
Engine-level differences: none in SQL. Both engines call the same takesImpl functions with the same executor shape. Typed-signature difference only: PGLite's searchTakes and searchTakesVector opts type (pglite-engine.ts:2648-2658) omits sourceId and sourceIds, while Postgres's includes them (postgres-engine.ts:2507-2513). Runtime passes the full opts object through, so behavior matches.
Dream-cycle verdict cache
Table dream_verdicts, keyed (file_path, content_hash). Index dream_verdicts_expires_idx (schema.sql:1505). expires_at defaults to now()+30 days at the table level. Writes set it explicitly.
| Method (iface line) | Signature | What it does | SQL / impl |
|---|---|---|---|
getDreamVerdict (2078) |
getDreamVerdict(filePath, contentHash): Promise<DreamVerdict | null> |
Cache lookup. Expired rows are misses. NULL-expiry rows count as hits. | PGLite pglite-engine.ts:2367-2400, Postgres postgres-engine.ts:2183-2216. SELECT worth_processing, reasons, judged_at, score, content_type, segments, entities, model, triage_version FROM dream_verdicts WHERE file_path=$ AND content_hash=$ AND (expires_at IS NULL OR expires_at > now()). Maps null arrays to [], judged_at to ISO string. Note the NULL-expiry branch, documented as the #4657 bootstrap window (postgres-engine.ts:2200; PGLite pglite-engine.ts:2383). |
putDreamVerdict (2079) |
putDreamVerdict(filePath, contentHash, verdict: DreamVerdictInput): Promise<void> |
Upserts a verdict and resets the TTL. | PGLite pglite-engine.ts:2402-2430, Postgres postgres-engine.ts:2218-2243. wellFormedJson(verdict) first. INSERT INTO dream_verdicts(file_path,content_hash,worth_processing,reasons,score,content_type,segments,entities,model,triage_version,expires_at) VALUES(...,now()+make_interval(secs => TTL)) ON CONFLICT (file_path,content_hash) DO UPDATE SET <all fields>=EXCLUDED.*, judged_at=now(), expires_at=EXCLUDED.expires_at. TTL = DREAM_VERDICT_TTL_SECONDS (engine.ts:528). JSON columns: PGLite $4::jsonb with JSON.stringify, Postgres sql.json. |
sweepDreamVerdicts (2084) |
sweepDreamVerdicts(): Promise<number> |
Deletes expired verdicts. Housekeeping only. | PGLite pglite-engine.ts:2432 returns affectedRows. Postgres postgres-engine.ts:2245 returns count. Both run DELETE FROM dream_verdicts WHERE expires_at <= now(). NULL-expiry rows are not swept. |
Contradiction probe (batched takes, cache, run history)
Tables: eval_contradictions_runs (schema.sql:1631), eval_contradictions_cache (schema.sql:1614, expiry index at :1626), and take_grade_cache (referenced by listTakes ordering, schema.sql:1727). All SQL lives in engine-sql/takes.ts for both engines.
| Method (iface line) | Signature | What it does | SQL / impl |
|---|---|---|---|
listActiveTakesForPages (2102) |
listActiveTakesForPages(pageIds: number[], opts?: {takesHoldersAllowList?}): Promise<Map<number, Take[]>> |
Active takes for many pages in one round-trip. Every requested page id is a key, with [] if it has none. |
takes.ts:109-135. SELECT t.*, p.slug AS page_slug FROM takes t JOIN pages p ON p.id=t.page_id WHERE t.page_id = ANY($::int[]) AND t.active = true AND (allow-list IS NULL OR holder = ANY(...)) ORDER BY t.page_id, t.row_num. Read is LegacyUnscopedRead. No deleted_at or privacy filter. |
writeContradictionsRun (2115) |
writeContradictionsRun(row: {run_id, judge_model, prompt_version, queries_evaluated, queries_with_contradiction, total_contradictions_flagged, wilson_ci_lower, wilson_ci_upper, judge_errors_total, cost_usd_total, duration_ms, source_tier_breakdown, report_json}): Promise<boolean> |
Persists one probe run row. Idempotent on run_id. Returns true if inserted. |
takes.ts:141-174. INSERT INTO eval_contradictions_runs(...) VALUES(...,${jsonbParam(source_tier_breakdown)},${jsonbParam(report_json)}) ON CONFLICT (run_id) DO NOTHING, returning affectedRows > 0. |
loadContradictionsTrend (2137) |
loadContradictionsTrend(days: number): Promise<Array<{run_id, ran_at, ... report_json}>> |
Runs from the last N days, newest first. | takes.ts:180-221. Cutoff is computed in JS: Date.now() - max(0,days)*86400000. SELECT ... FROM eval_contradictions_runs WHERE ran_at >= $cutoff ORDER BY ran_at DESC. No LIMIT. Numeric columns go through Number(). ran_at becomes an ISO string. Read is LegacyUnscopedRead. |
getContradictionCacheEntry (2163) |
getContradictionCacheEntry(key: {chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy}): Promise<Record<string, unknown> | null> |
Judge-verdict cache lookup. Key is the full five-part key, so prompt or truncation changes miss. | takes.ts:227-247. SELECT verdict FROM eval_contradictions_cache WHERE <5 key cols> AND expires_at > now() LIMIT 1. Read is LegacyUnscopedRead. |
putContradictionCacheEntry (2180) |
putContradictionCacheEntry(opts: {...5 key cols, verdict, ttl_seconds?}): Promise<void> |
Upserts a verdict and slides the TTL. | takes.ts:253-279. TTL = max(60, ttl_seconds ?? 30d). expiresAt = new Date(Date.now() + ttl*1000) is computed in JS. INSERT INTO eval_contradictions_cache(...,verdict,expires_at) VALUES(...,${jsonbParam(verdict)},$expiresAt) ON CONFLICT (5 key cols) DO UPDATE SET verdict, expires_at, created_at=now(). |
sweepContradictionCache (2194) |
sweepContradictionCache(): Promise<number> |
Deletes expired judge-cache rows. Called periodically from cache.ts. |
takes.ts:282-287. DELETE FROM eval_contradictions_cache WHERE expires_at <= now(), returns affectedRows. |
Engine-level: all seven delegate to takesImpl in both engines (PGLite pglite-engine.ts:2575-2642, Postgres postgres-engine.ts:2434-2500). Writes use this.engineSql (SqlExecutor), and the loadContradictionsTrend, listActiveTakesForPages, and cache reads use unscopedExecutor.
Tests covering this range
Counts are the number of files under test/ that name the method (grep \bname\b). "Parity" means the test runs on both engines through describeBoth in test/e2e/engine-parity.test.ts:33. That helper is gated by SKIP_PG, which is set when no DATABASE_URL is present (:32). test/pglite-engine.test.ts has no takes, ontology, chronicle, file, dream or contradiction tests. Its only coverage here is tags, timeline and raw-data basics.
| Method | Files naming it | Parity / direct coverage |
|---|---|---|
addTag, removeTag |
66 / 16 | engine-parity.test.ts:189-201 (getTags excludePrivate, liveOnly, both engines), :1434-1435 (sourceId scoping); pglite-engine.test.ts:766-787 (add, remove, dup idempotent) |
getTags |
42 | engine-parity.test.ts:189, :1468-1470 (sourceIds[] grant); pglite-engine.test.ts:772-786 |
addTimelineEntry |
66 | engine-parity.test.ts:287, :1444-1446 (sourceId); pglite-engine.test.ts:800-814; test/e2e/chronicle-event-projection-parity.test.ts (describeBoth) |
addTimelineEntriesBatch |
34 | pglite-engine.test.ts:823+ (batch); test/links-timeline-jsonb-poison.test.ts; engine-parity.test.ts count 1 |
getTimeline |
46 | engine-parity.test.ts:1536-1549 (sourceIds, D5A window), :1557-1584 (Cat7-1 unscoped plan, both engines); pglite-engine.test.ts:800-820 |
getTimelineForDate, getSince, getOnThisDay |
15 / 11 / 11 | test/chronicle-ops-scope.test.ts, test/chronicle-extract.test.ts, test/e2e/chronicle-event-projection-parity.test.ts. No direct parity test in engine-parity.test.ts (0 hits). |
getLastSeen |
12 | test/e2e/chronicle-last-seen-postgres.test.ts (Postgres only, name-gated). No PGLite parity hit. |
upsertEventProjection |
16 | test/e2e/chronicle-event-projection-parity.test.ts (describeBoth), test/chronicle-invite-projection.test.ts |
mergeOntologyFact |
27 | test/e2e/ontology-merge-parity.test.ts (describeBoth), test/chronicle-ontology.test.ts, test/ontology-backdated-same-value.test.ts, test/trust-guarded-supersession.test.ts, test/pathological/b-10-ontology-rekey-collision.test.ts, test/chronicle-ontology-ops.test.ts |
getOntology |
13 | test/e2e/ontology-merge-parity.test.ts, test/chronicle-ontology-private-visibility.test.ts |
discoverOntologyDimensions |
10 | test/chronicle-ontology.test.ts, test/managed-ontology-propose.test.ts (no parity hit) |
findOntologyConflicts |
13 | test/chronicle-ontology.test.ts, test/e2e/ontology-merge-parity.test.ts |
putRawData |
26 | engine-parity.test.ts:2714-2731 (soft-delete filter, PGLite always and Postgres when DATABASE_URL is set); pglite-engine.test.ts (count 2) |
getRawData |
20 | engine-parity.test.ts:2714-2731; pglite-engine.test.ts (count 2) |
upsertFile |
12 | test/engine-upsertFile.test.ts (PGLite only) |
getFile |
13 | test/engine-upsertFile.test.ts |
listFilesForPage |
11 | test/engine-upsertFile.test.ts |
addTakesBatch |
77 | test/e2e/takes-postgres.test.ts (Postgres, DATABASE_URL-gated), test/e2e/takes-weight-rounding-postgres.test.ts, test/takes-weight-rounding.test.ts, test/e2e/jsonb-batch-poison-postgres.test.ts, test/e2e/takes-scorecard-parity.test.ts (uses PGLite and Postgres) |
updateTakeEmbeddings |
12 | test/takes-vector-lifecycle-5885.serial.test.ts (PGLite), test/e2e/takes-horizon-postgres.test.ts |
listTakes |
24 | test/list-takes-tie-order.test.ts, test/takes-source-scope.test.ts, test/takes-mcp-allowlist.test.ts, test/e2e/serve-http-takes-holders.test.ts |
searchTakes |
19 | test/e2e/takes-postgres.test.ts (pg_trgm), test/takes-source-scope.test.ts, test/eligibility-reads.test.ts |
searchTakesVector |
15 | test/e2e/takes-postgres.test.ts (pgvector), test/takes-vector-lifecycle-5885.serial.test.ts |
getTakeEmbeddings |
10 | test/takes-vector-lifecycle-5885.serial.test.ts |
countStaleTakes, listStaleTakes |
12 / 13 | test/takes-vector-lifecycle-5885.serial.test.ts |
updateTake |
15 | test/takes-weight-rounding.test.ts, test/takes-grades-cli.test.ts |
supersedeTake |
14 | test/e2e/takes-postgres.test.ts (supersede path on real PG), test/extract-takes.test.ts |
resolveTake |
24 | test/e2e/takes-postgres.test.ts (immutability), test/grade-takes-window.test.ts, test/eval-takes-quality-boundaries.test.ts |
getScorecard |
16 | test/e2e/takes-scorecard-parity.test.ts (both engines), test/calibration-profile.test.ts |
getCalibrationCurve |
14 | test/e2e/takes-scorecard-parity.test.ts (both engines, bucket boundaries), test/e2e/calibration-profile-write.test.ts |
addSynthesisEvidence |
11 | test/e2e/takes-postgres.test.ts (FK cascade on take delete), test/think-save-source.test.ts |
getDreamVerdict |
16 | test/e2e/engine-parity.test.ts:389-413 (triage-v1 round-trip, identical shape on both engines), :432 (legacy row); test/e2e/dream-triage-backoff-postgres.test.ts; test/pathological/b6-triage-verdict-surrogate.test.ts |
putDreamVerdict |
23 | test/e2e/engine-parity.test.ts:408-413 (both engines, score update); test/synthesize-triage-backoff.test.ts; test/e2e/dream-synthesize-pglite.test.ts |
sweepDreamVerdicts |
7 | Few direct tests; test/synthesize-triage-backoff.test.ts covers the TTL and expiry behavior |
listActiveTakesForPages |
11 | test/eval-contradictions-engine.test.ts (PGLite), test/e2e/eval-contradictions-postgres.test.ts (Postgres) |
writeContradictionsRun, loadContradictionsTrend |
16 / 12 | test/e2e/eval-contradictions-postgres.test.ts, test/eval-contradictions-engine.test.ts, test/eval-takes-quality-runner.serial.test.ts |
getContradictionCacheEntry, putContradictionCacheEntry, sweepContradictionCache |
11 each | test/eval-contradictions/no-valid-until-write.test.ts, test/eval-contradictions-engine.test.ts, test/e2e/eval-contradictions-postgres.test.ts |
Summary of coverage: 43 methods in range. Eight have parity tests that run on both engines in engine-parity.test.ts: addTag, getTags, addTimelineEntry, getTimeline, putRawData, getRawData (soft-delete), getDreamVerdict, putDreamVerdict. removeTag and addTimelineEntriesBatch appear once in that file, but not in a dedicated parity assertion I could confirm. The other methods have no hit in engine-parity.test.ts; their parity coverage, where it exists, sits in the separate *-parity.test.ts files listed above. Chronicle ontology and takes have parity tests in separate files (ontology-merge-parity.test.ts, chronicle-event-projection-parity.test.ts, takes-scorecard-parity.test.ts). Several methods have Postgres-only tests (takes-postgres.test.ts, eval-contradictions-postgres.test.ts, chronicle-last-seen-postgres.test.ts), gated by DATABASE_URL.
Parity risks noted
- Clock sources differ. Reads and sweeps compare against the DB clock (
now(),current_date), e.g.getDreamVerdict,sweepDreamVerdicts,getContradictionCacheEntry,sweepContradictionCache,getOnThisDay,getLastSeen,getOntologydefaultasof. Some writes and computations use the JS clock:putContradictionCacheEntryexpiry (takes.ts:262-263),loadContradictionsTrendcutoff (takes.ts:195), andfinalizeLastSeen'sdays_ago(last-seen.ts:11-21). Clock skew between app and DB changes results. A Rust rewrite has to choose one clock and keep both sides on it. - Transaction-local trust GUC.
ONTOLOGY_SUPERSEDE_GUARD(chronicle/ontology.ts:65) readscurrent_setting('gbrain.write_trust_tier', true)in the same transaction. Ontology supersession depends on that session setting, not on a parameter. An embedded store needs an equivalent per-transaction writer tier. - Postgres-specific SQL features with no direct equivalent:
DISTINCT ON,json_agg/json_build_object,jsonb_to_recordset(batch writes),jsonb_array_elements_text,unnest(int[]),ON CONFLICT ... WHERE <partial index>(timeline_event_dedup, facts dedup),RETURNING (xmax = 0) AS created(upsertFile created flag),md5()as a dedup key (timelinemd5(summary)),FOR KEY SHARE(addTimelineEntry),LATERAL ... OFFSET 0planner fence (getTimeline),make_interval,date_trunc('week'),EXTRACT(MONTH/DAY),pg_trgm<%andword_similarity,pgvector<=>andvector_dims. - Calibration bucketing relies on NUMERIC arithmetic (
takes.ts:661-665). Float math would put boundary weights in a different bucket. The Postgres driver also needs explicit::floatcasts on bound scalars (takes.ts:644-648), so scalars are text-bound there. Rust must do exact decimal bucketing. filesidentity mismatch. The DDL isUNIQUE(storage_path)(schema.sql:1149), whilegetFileand the file comment atengine-sql/files.ts:41describe identity as(source_id, storage_path). TheON CONFLICT (storage_path) DO UPDATEnever updatessource_id. A second source writing the same storage_path keeps the first source's row, andgetFile(secondSource, path)returns null.supersedeTakeallocatesrow_numasMAX(row_num)+1without a row lock (takes.ts:554). Two concurrent supersedes on one page can compute the same next row number. A Rust rewrite should allocate with a lock or sequence.addTakesBatchON CONFLICT is DO UPDATE (takes.ts:62-63). An intra-batch duplicate(page_id,row_num)fails the whole statement, and the claim-change reset of embeddings (takes.ts:81-84) must be reproduced exactly.updateTakeEmbeddingssilently drops a vector when the claim changed in flight (WHERE t.claim = v.claim,takes.ts:502). It returns the updated count only, so the caller cannot tell a dropped vector from a successful one.listStaleTakeshas a hardLIMIT 100000(takes.ts:453). Excess rows are dropped with no signal.loadContradictionsTrendhas no LIMIT (takes.ts:196-205). Result size is unbounded.searchTakesVectorhas no dimension check in SQL (takes.ts:385-401). Mixed-width vectors fail in pgvector. The Rust version must decide what to do with mixed widths.getLastSeenmatches wikilinks withLIKE ... ESCAPE '\\'onwhostrings (timeline.ts:243-245). The escape rule has to be reproduced exactly, or wikilink matches will differ.- Reads on master are
LegacyUnscopedRead(engine-sql/brands.ts:33): no RLS scope transaction. Any Rust rewrite that adds scope enforcement will change behavior in this range. The brands doc calls widening a TODO, not a drive-by (brands.ts:17-21). - Dialect cancellation differs. PGLite cannot cancel a running statement;
pgliteQueryraces a late abort (dialect-pglite.ts:42-56). Postgres uses real cancellation. Batch-write abort behavior is not identical. - Lock capability differs.
transactionAdvisoryLocksisfalseon PGLite (dialect-pglite.ts:27) andtrueon Postgres (dialect-postgres.ts:55).mutatePageTagcallslockPageKeyson both. I did not trace the lock mechanism itself. - Type-only drift: PGLite's
searchTakes/searchTakesVectoropts type (pglite-engine.ts:2648-2658) omitssourceId/sourceIds. Runtime behavior is unchanged. resolveTakeandupdateTakehave no transaction and do a SELECT then an UPDATE (takes.ts:571-576). Concurrent resolves can race the immutability check. The engine's error code is the only guard.putRawDatawithoutsourceIdmatches every same-slug page (postgres-engine.ts:2101,pglite-engine.ts:2284). Multi-source brains can get a 21000 error. The sourceId path avoids this.
Facts, Segmented storage, Versions, Stats, Sync, Config and Migration support
Scope: BrainEngine interface members declared in src/core/engine.ts lines 2196-2890 (70 methods, confirmed by listing every member in that range). Interface line numbers below are engine.ts:N. Implementations: PGLiteEngine = src/core/pglite-engine.ts, PostgresEngine = src/core/postgres-engine.ts, shared SQL = src/core/engine-sql/*.ts. "Shared" means one SQL implementation runs on both engines through SqlExecutor. "Inline" means each engine writes its own SQL.
Groups in this part: Facts (14), Segmented page storage (11), Derivation watermarks (3), Versions (3), Stats and health (2), Ingest log (2), Sync (9), Config (5), Migration and raw SQL (5), Code edges (6), Eval capture (5), Salience and anomalies (5). Total 70.
Engine plumbing used by this range
| Item | PGLite | Postgres | Notes |
|---|---|---|---|
| Shared executor | engineSql getter: pgliteExecutor(db or guardedHandle(db)), pglite-engine.ts:794-795 |
engineSql / engineSqlOn(conn), postgres-engine.ts:317-322 |
Executor built per access so transaction clones run on their own handle. |
| Dialect capabilities | PGLITE_CAPABILITIES = maxBindParamsPerStatement 30000, transactionAdvisoryLocks false, probesEmbeddingCast false, engine-sql/dialect-pglite.ts:25-29 |
maxBindParamsPerStatement Infinity, transactionAdvisoryLocks true, probesEmbeddingCast true, engine-sql/dialect-postgres.ts:54-56 |
These three flags drive most PGLite/PG branches in the shared SQL below. |
| Unscoped read brand | unscopedExecutor(engineSql, label) wraps reads; LegacyUnscopedRead type, engine-sql/brands.ts |
same | Every read in this range is on master with no source scope (EO4 inventory). |
| Transactions | transaction() = db.transaction with composable handle, savepoint on nesting, checkpoint guard runOutermost, pglite-engine.ts:1209-1223. Single connection, no advisory locks. |
transactionOn(), postgres-engine.ts:726-760: BEGIN READ WRITE unless readOnly (postgres-engine/read-write-transactions.ts), checkoutGauge 'tx', signal abort via transaction-abort.ts, commit checks, savepoints on nested clones. |
Interface declaration at engine.ts:849-851 (outside this range; owned by the lifecycle part). |
| Statement timeout | None. PGLite runs read on itself and ignores timeoutMs |
Per-statement statement_timeout (keyword-statement-timeout.ts, default 8000 ms, range 1000-60000) |
See keywordBoundedRead. |
| Pool gauge / reserved connections | None (single in-process connection) | checkoutGauge acquire/release for raw, direct, tx checkouts (executeRaw 2998-3008, executeRawDirect 3053-3058, transactionOn), real driver queue numbers in postgres-engine/pool-stats.ts:1-11; reserveWithCancellation (postgres-engine/cancellation.ts:17) reserves a dedicated connection for each cancellable statement |
Pool diagnostics and reserved-connection behavior exist only on Postgres. |
| WAL checkpoint guard | PgliteCheckpointGuard, pglite-engine/checkpoint-guard.ts:1-25; writesWal(sql) regex decides autocommit writes |
n/a | Only in PGLite. |
| Config snapshot | requestConfigValue / requestConfigRows, config-snapshot.ts:161-172, used by getConfig / getAllConfig on both engines |
same | Request-scoped snapshot first, then the engine's own read. |
Facts / hot memory (14 methods)
Shared SQL: src/core/engine-sql/facts.ts. Both engines delegate; PGLite passes undefined for the embedding-cast resolver, Postgres passes () => this.resolveFactsEmbeddingCast() (postgres-engine.ts:2271-2304, which probes pg_attribute for facts.embedding and returns ::halfvec or ::vector, cached on the root engine).
Test-count convention: "eng-parity" = test/e2e/engine-parity.test.ts (Postgres arm needs DATABASE_URL, PGLite arm always runs); "pglite" = test/pglite-engine.test.ts; "files" = number of test/ files that mention the method name anywhere (grep, so an upper bound on real coverage).
| Method (iface line) | Impl PGLite / PG | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
insertFact(input: NewFact, ctx: {source_id, supersedeId?}): Promise<{id, status}> (engine.ts:2215) |
pglite-engine.ts:2447 / postgres-engine.ts:2306; shared facts.ts:88-172 |
Two paths. (a) No supersedeId: one INSERT INTO facts (...) VALUES (...) RETURNING id, status 'inserted'. (b) With supersedeId: exec.transaction, then INSERT, then UPDATE facts SET expired_at = now(), superseded_by = <new id>, valid_until = CASE ... END WHERE id = supersedeId AND expired_at IS NULL, status 'superseded'. Per-entity lock: pg_advisory_xact_lock(hashtextextended(source_id || ':' || entity_slug, 0)) when transactionAdvisoryLocks (facts.ts:120-123, 149-152). Vector is an inlined literal '[...]'::vector or ::halfvec (facts.ts:105, 133); embedded_text_hash = md5(fact) when an embedding and model are supplied. The method never returns 'duplicate': dedup is the caller's job (engine.ts:2200-2214 says it can, the code cannot). |
PGLite: no advisory lock, plain insert not wrapped in a transaction (facts.ts:168-170), cast always ::vector (probesEmbeddingCast false). PG: lock, cast probe. eng-parity 8; pglite 0; facts-engine.test.ts, facts-separation-pglite.test.ts; 171 files mention it. |
insertFacts(rows, ctx: {source_id}, opts?: {deleteForPageFirst?}): {inserted, ids, warnings, deleted} (engine.ts:2258) |
pglite-engine.ts:2458 / postgres-engine.ts:2317; shared facts.ts:260-370 |
One exec.transaction. (1) If deleteForPageFirst: DELETE FROM facts WHERE source_id AND source_markdown_slug = slug [AND NOT (COALESCE(source,'') LIKE ANY(prefix%))] [AND NOT (row_num IS NULL AND expired_at IS NOT NULL)] (facts.ts:290-313), same tx as the inserts. (2) set_config(FACT_WITHDRAWAL_STAMP_DEFERRED_SETTING, 'on', true) (facts.ts:204), then multi-row INSERT ... SELECT FROM jsonb_to_recordset(<json text>) ... ON CONFLICT (source_id, source_markdown_slug, row_num) WHERE row_num IS NOT NULL DO NOTHING RETURNING id, source_markdown_slug, row_num, in chunks of FACT_INSERT_CHUNK = 500 (facts.ts:185, 221-248); the setting is reset afterwards (facts.ts:256). (3) Second pass: superseded by #N refs resolved by one SELECT over facts joined to a JSON recordset, then one UPDATE facts SET superseded_by ... FROM jsonb_to_recordset (facts.ts:329-366); warnings from facts/supersede-resolve.ts. Vector cast: r.embedding::vector or ::halfvec per castSuffix. |
PGLite: cast fixed ::vector; no lock needed. Returned ids are filtered (facts.ts:318), so they are NOT index-aligned with rows when ON CONFLICT skips a row (doc at engine.ts:2236 says input order). eng-parity 0; pglite 0; 50 files (generations-deprecated-fact-writers.test.ts, extract-facts-reconcile-drain.test.ts, fence-chunk-overlay.test.ts, conversation-facts-managed.test.ts). |
deleteFactsForPage(slug, source_id, opts?): {deleted} (engine.ts:2305, @deprecated GBRA-161) |
pglite-engine.ts:2466 / postgres-engine.ts:2325; shared facts.ts:372-403 |
DELETE FROM facts WHERE source_id = $ AND source_markdown_slug = $ with the same optional excludeSourcePrefixes and preserveExpiredLegacy filters as above. Self-committing on the root executor (note at facts.ts:285-288). Hard delete. |
Identical on both engines. eng-parity 0; pglite 0; 18 files (generations-deprecated-fact-writers.test.ts, facts-thin-client-provenance.test.ts, canonical-writer-inventory.test.ts, extract-facts-phase.test.ts). |
expireFact(id, opts?: {supersededBy?, at?}): boolean (engine.ts:2316, @deprecated) |
pglite-engine.ts:2454 / postgres-engine.ts:2313; shared facts.ts:174-182 |
UPDATE facts SET expired_at = $at, superseded_by = COALESCE($supersededBy, superseded_by) WHERE id = $ AND expired_at IS NULL; returns affectedRows > 0. Never deletes. |
Identical. eng-parity 0; pglite 0; 24 files (facts-engine.test.ts, recall-extensions.test.ts, write-attribution-legacy.test.ts). |
listFactsByEntity(source_id, entitySlug, opts?): FactRow[] (engine.ts:2319) |
pglite-engine.ts:2474 / postgres-engine.ts:2333; shared facts.ts:405-437 |
SELECT * FROM facts WHERE source_id AND entity_slug [AND source_session]. Default activeOnly: expired_at IS NULL AND (valid_until IS NULL OR valid_until > now()). Optional: consolidated_at IS NULL, kind = ANY, visibility clause, privacy clause, audit-row exclusion (source != ALL(AUDIT_ROW_SOURCES)), fact ILIKE grep with escaped pattern, eligibility clause. ORDER BY valid_from DESC, id DESC LIMIT OFFSET (default limit 50). |
Identical. eng-parity 3; pglite 0; facts-engine.test.ts, facts-visibility.test.ts, entity-names-parity.test.ts, facts-separation-pglite.test.ts. |
listFactsSince(source_id, since: Date, opts?): FactRow[] (engine.ts:2333) |
pglite-engine.ts:2482 / postgres-engine.ts:2341; shared facts.ts:439-464 |
WHERE source_id AND created_at >= since (or COALESCE(valid_from, created_at) with eventTime), plus recallNarrowing (facts.ts:79-86: entitySlug, sourceMarkdownSlug, sessionId, excludeAuditRows, grep). Active filter as above. ORDER BY created_at DESC, id DESC LIMIT OFFSET. fingerprint adds gbrain_fact_fingerprint(fact). |
Identical. eng-parity 1; pglite 0; hot-memory-invalidation.test.ts, facts-recall-audit-rows.test.ts, facts-engine.test.ts, ambient-recall.test.ts. |
listFactsKeyset(source_id, after: {createdAt, id|null} | null, opts?): FactRow[] (engine.ts:2345) |
pglite-engine.ts:2490 / postgres-engine.ts:2349; shared facts.ts:477-503 |
Keyset read, OLDEST first: AND (created_at > ts::text::timestamptz OR (created_at = ts AND id > $id)), or created_at > ts when id is null. Adds to_char(created_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS created_at_iso so microseconds survive. ORDER BY created_at ASC, id ASC LIMIT. |
Identical. eng-parity 0; pglite 0; delta-cursor-integrity.test.ts, engine-surface-coverage.test.ts, helpers/postgres-engine-sql-cases.ts. |
listFactsBySession(source_id, sessionId, opts?): FactRow[] (engine.ts:2352) |
pglite-engine.ts:2498 / postgres-engine.ts:2357; shared facts.ts:505-533 |
WHERE source_id AND source_session = $, active/kind/visibility/privacy/audit/grep/eligibility filters, ORDER BY created_at DESC, id DESC LIMIT OFFSET. |
Identical. eng-parity 1; pglite 0; facts-separation-pglite.test.ts, facts-engine.test.ts, e2e/facts-separation-postgres.test.ts. |
listSupersessions(source_id, opts?): FactRow[] (engine.ts:2370) |
pglite-engine.ts:2506 / postgres-engine.ts:2365; shared facts.ts:535-559 |
WHERE source_id AND superseded_by IS NOT NULL (not expired_at, so ontology and fence supersessions appear), optional COALESCE(expired_at, valid_until) >= since, visibility, privacy, eligibility. ORDER BY COALESCE(expired_at, valid_until) DESC, id DESC LIMIT (no OFFSET). |
Identical. eng-parity 2; pglite 0; chronicle-ontology.test.ts, recall-extensions.test.ts, facts-engine.test.ts, e2e/facts-fence-reconcile-postgres.test.ts. |
countUnconsolidatedFacts(source_id): number (engine.ts:2382) |
pglite-engine.ts:2513 / postgres-engine.ts:2372; shared facts.ts:561-578 |
SELECT COUNT(*)::int FROM facts WHERE source_id AND entity_slug IS NOT NULL AND consolidated_at IS NULL AND expired_at IS NULL AND (valid_until IS NULL OR valid_until > now()) AND source != ALL(AUDIT_ROW_SOURCES). |
Identical. eng-parity 1; pglite 0; recall-extensions.test.ts, facts-recall-audit-rows.test.ts, facts/ttl-validity.test.ts. |
findCandidateDuplicates(source_id, entitySlug, factText, opts?): FactRow[] (engine.ts:2393) |
pglite-engine.ts:2517 / postgres-engine.ts:2376; shared facts.ts:580-666 |
k clamped to 1..20, default 5. Three arms. (a) arm: 'keyword': findKeywordCandidates (facts.ts:638-666) builds an OR-of-lexemes tsquery from plainto_tsquery(lang, text), filters by to_tsvector(...) @@ q, orders by ts_rank_cd DESC, created_at DESC, id DESC. (b) opts.embedding given (needs embeddingModel, else []): WHERE embedding IS NOT NULL AND embedding_model = $ AND embedded_text_hash = md5(fact) AND vector_dims(embedding) = n ORDER BY embedding <=> '<literal>'::vector LIMIT k. (c) Otherwise ORDER BY created_at DESC, id DESC LIMIT k. All arms: active + validity filter, audit-row exclusion, attributed_to IS NULL OR = speaker, and onlyPage or excludeMemberPages (onOwnedMemoryPageSql). |
Identical SQL; PGLite always uses the ::vector literal. eng-parity 4; pglite 0; facts-engine.test.ts, facts/candidate-fusion.test.ts, facts/speaker-attribution.test.ts, facts/ttl-validity.test.ts. |
consolidateFact(id, takeId): void (engine.ts:2404) |
pglite-engine.ts:2530 / postgres-engine.ts:2385; shared facts.ts:668-670 |
UPDATE facts SET consolidated_at = now(), consolidated_into = $takeId WHERE id = $id. Never deletes. |
Identical. eng-parity 0; pglite 0; facts-engine.test.ts, write-attribution-legacy.test.ts, engine-surface-coverage.test.ts. |
findTrajectory(opts: TrajectoryOpts): TrajectoryPoint[] (engine.ts:2422) |
pglite-engine.ts:2526 / postgres-engine.ts:2389; shared facts.ts:672-740 |
SELECT id, valid_from, valid_until, claim_*, event_type, fact, source_session, source_markdown_slug, embedding FROM facts WHERE source (scalar sourceId, default 'default', or source_id = ANY(sourceIds)), entity_slug = $, expired_at IS NULL. Remote callers (remote !== false, fail-closed) get visibility = 'world' only. Optional metric/kind/since/until. Embedding returned only when embedding_model = (SELECT value FROM config WHERE key='embedding_model') and embedded_text_hash = md5(fact) (inline subquery, facts.ts:709). ORDER BY valid_from DESC, id DESC LIMIT (default 100, cap 500), then reversed to chronological. No validity filter by design (engine.ts:2415-2420). |
Identical. eng-parity 3; pglite 0; engine-find-trajectory.test.ts, trajectory-timeout-timers.test.ts, think-trajectory-injection.test.ts, migrate.test.ts. |
getFactsHealth(source_id): FactsHealth (engine.ts:2425) |
pglite-engine.ts:2534 / postgres-engine.ts:2393; shared facts.ts:742-781 |
Two statements. (1) One row of COUNT(*) FILTER (...): total_active (active and not lapsed), total_today (created_at > now() - 24h), total_week (7 days), total_expired (expired_at IS NOT NULL or lapsed), total_consolidated. (2) Top 5 entities by active count, ORDER BY count DESC, entity_slug ASC. |
Identical. eng-parity 1; pglite 0; facts-doctor-shape.test.ts, facts-engine.test.ts, facts/ttl-validity.test.ts. |
Segmented page storage (11 methods)
Shared SQL: src/core/engine-sql/page-body.ts. Limits: src/core/page-body/limits.ts (STAGE_BATCH_BYTES = 4 MiB for both engines, line 65-68; SEGMENT_TEXT_GC_GRACE_MS = 1 h, line 132; BODY_STAGING_IDLE_MS = 1 h, line 134; BODY_STAGING_HARD_MS = 24 h, line 135). Batches yield the event loop with setTimeout(0) (page-body.ts:50). Both engines call the shared functions with this.engineSql (PGLite pglite-engine.ts:2539-2562, PG postgres-engine.ts:2398-2421); only readSegmentTexts, readManifest and readSegmentDerivations use the unscoped read brand.
Test note for the whole family: test/large-pages/segment-gc.test.ts runs the GC helper cases on PGLite, test/e2e/segment-gc-parity.test.ts imports PostgresEngine (Postgres side). test/e2e/segmented-publish-tiling-postgres.test.ts and test/large-pages/segmented-publish-tiling.serial.test.ts cover the flip. No engine-parity or pglite-engine.test.ts hits for any of these 11 methods (eng-parity 0, pglite 0 for each).
| Method (iface line) | Impl | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
allocateBodyOccurrenceIds(count): number[] (engine.ts:2429) |
shared page-body.ts:58-62 |
SELECT nextval('page_body_occurrence_id_seq') AS id FROM generate_series(1, $count). count <= 0 returns [] without SQL. Ids are never reused. |
Identical. files 12 (helpers/segment-gc-cases.ts, helpers/segmented-page.ts, helpers/page-body-fixtures.ts). |
stageSegmentTexts(sourceId, texts, opts?: {stagingId?}): {inserted} (engine.ts:2431) |
shared page-body.ts:69-100 |
Batches by bytes (STAGE_BATCH_BYTES) and by floor(maxBindParams / 5) rows (page-body.ts:72, 92). Per batch: if stagingId, INSERT INTO page_body_staging_texts (staging_id, source_id, sha256) SELECT ... FROM unnest(...) ON CONFLICT DO NOTHING (records the text under the staging so a page purge keeps it). Then INSERT INTO page_segment_texts (source_id, sha256, byte_len, line_count, body) VALUES ... ON CONFLICT (source_id, sha256) DO UPDATE SET created_at = now() RETURNING (xmax = 0) AS inserted. inserted counts fresh rows only. |
Row cap differs: PGLite maxBindParams 30000 gives 6000 rows per statement; PG Infinity means no row cap (only bytes). files 15. |
readSegmentTexts(sourceId, sha256s: Uint8Array[]): Map<hexSha, bytes> (engine.ts:2433) |
shared page-body.ts:102-108 (unscoped read) |
SELECT encode(t.sha256,'hex') AS sha, t.body FROM page_segment_texts t WHERE t.source_id = $ AND t.sha256 = ANY(SELECT decode(h,'hex') FROM unnest($text[]) AS h). Missing shas are absent from the map. Empty input returns an empty Map. |
Identical. files 26. |
stageManifestRows(rows: ManifestRowInput[]): void (engine.ts:2435) |
shared page-body.ts:110-132 |
Rows serialized by jsonRowBatches (batch-rows.ts) and inserted with INSERT INTO page_body_manifest (page_id, occurrence_id, gen_from, seq, region, source_id, sha256, byte_start, byte_len, line_start, line_count, cut_kind, entry_state, protected_regions, facts_slots, static_proj_bytes, static_proj_lines, outline, date_min, date_max, hash_state, staging_id) SELECT ... FROM jsonb_to_recordset(...). Hex sha and hash_state decoded to bytea. No transaction around the batches. |
Identical. files 13. |
readManifest(pageId, opts?: {generation?, stagingId?, region?, lastN?}): ManifestRow[] (engine.ts:2437) |
shared page-body.ts:152-170 (unscoped read) |
Metadata only, never the body. Generation g = COALESCE($generation, pages.body_generation). Live rows: staging_id IS NULL AND gen_from <= g AND (gen_to IS NULL OR gen_to > g). Staged rows: staging_id = $stagingId. Optional region. ORDER BY seq, or with lastN: ORDER BY seq DESC LIMIT n then reversed. date_min/date_max rendered with to_char(..., 'YYYY-MM-DD'). |
Identical. files 32 (helpers/rechunk-segmented-cases.ts, helpers/segment-gc-cases.ts, helpers/conversation-facts-recovery-cases.ts). |
stageSegmentChunks(pageId, generation, chunks, opts?: {embeddingColumn?}): number[] (engine.ts:2439) |
shared page-body.ts:186-257 |
One exec.transaction. (1) PG only (transactionAdvisoryLocks): SELECT 1 FROM pages WHERE id = $ FOR KEY SHARE, then pg_advisory_xact_lock(hashtextextended('gbrain:page-body-chunks:' || pageId, 0)) (page-body.ts:193-197). (2) SELECT body_generation, body_storage, slug FROM pages; refuses with revision_changed when generation < live, the page is inline, or an occurrence has no published manifest row at the live generation (page-body.ts:199-217). (3) chunk_index = COALESCE(MAX(chunk_index), -1) + 1 for the page, never reused (page-body.ts:219-221). (4) Model and write column come from config keys search_embedding_column, embedding_columns, embedding_model (page-body.ts:173-184). (5) INSERT INTO content_chunks (...) SELECT ... FROM jsonb_to_recordset(...) in byte-bounded batches, embedding::vector (or the configured vector cast), embedding_image::vector, body_generation = $generation, embedded_at or embedding_pending_since set. Returned indexes are in input order. |
PGLite skips step (1), so there is no page row lock or advisory lock; the generation checks still run. files 14 (helpers/segment-gc-cases.ts, helpers/rechunk-segmented-cases.ts, helpers/page-body-fixtures.ts). |
createSegmentedPageShell(slug, meta, {sourceId}): {id} (engine.ts:2441) |
shared page-body.ts:270-282 |
INSERT INTO pages (source_id, slug, type, page_kind, title, compiled_truth NULL, timeline NULL, frontmatter, updated_at, effective_date..., chunker_version, source_*, ingested_*, body_storage 'segmented', body_generation 0, body_bytes 0) VALUES (...) RETURNING id. Slug through validateSlug. Origin claim is checked by the caller (page-body.ts:272 comment). |
Identical. files 11. |
flipBodyGeneration(input: BodyFlipInput): {retiredChunks, retireDeferred} (engine.ts:2443) |
shared page-body.ts:290-354 |
Asserts newGeneration === expectedGeneration + 1 (page-body.ts:291). One exec.transaction: (1) SELECT body_generation, slug, source_id FROM pages WHERE id FOR UPDATE. Deleted page throws page_identity_changed; generation mismatch throws revision_changed (page-body.ts:292-308). (2) SELECT state FROM page_body_stagings WHERE id FOR UPDATE; must be open, sealed or prepared, else body_staging_expired (page-body.ts:309-310). (3) Chunks to retire counted; if more than retireMaxRows the delete is deferred (page-body.ts:314-321). (4) UPDATE page_body_manifest SET gen_to = newGen for closeOccurrences; DELETE FROM content_chunks for the retire predicate; UPDATE page_body_manifest SET staging_id = NULL (publishes staged rows). (5) UPDATE pages SET body_storage='segmented', compiled_truth=NULL, timeline=NULL, body_generation=new, content_hash, body_bytes, truth_bytes, body_lines, segment_count, segmenter_version, truth_lead, timeline_lead, body_flags, type, page_kind, title, frontmatter, updated_at=now(), deleted_at=NULL, COALESCE(...) for effective_date and source fields. (6) GST-1209 check: liveTilingSql/untiledSql (page-body/schema.ts) must tile the body, else body_staging_expired reason generation_incomplete (page-body.ts:345-351). Comment at page-body.ts:287-288 notes this fires trg_pages_search_vector. |
Same statements; PGLite relies on being single-writer (no FOR UPDATE contention). files 13 (helpers/segmented-page.ts, helpers/segment-gc-cases.ts, helpers/segmented-publish-tiling-cases.ts, helpers/engine-sql-rollback-cases.ts). |
openBodyStaging(input: BodyStagingInput): void (engine.ts:2445) |
shared page-body.ts:375-383 |
INSERT INTO page_body_stagings (id, kind, principal_kind, principal_id, client_request_id, source_id, source_incarnation, slug, page_id, mode, base_revision, base_generation, declared_bytes, declared_sha256 (hex to bytea), reserved_bytes, idle_expires_at, hard_expires_at) VALUES (...). A staging is a GC root while open, sealed or prepared. |
Identical. files 18 (refresh-barrier-purge-gst1602.test.ts, helpers/segment-gc-cases.ts, helpers/postgres-engine-sql-cases.ts). |
setBodyStagingState(id, state, opts?: {commitRequestId?}): void (engine.ts:2447) |
shared page-body.ts:391-401 |
UPDATE page_body_stagings SET state, updated_at = now(), idle_expires_at = CASE WHEN live THEN LEAST(hard_expires_at, now() + idle) ELSE idle_expires_at END, commit_request_id = COALESCE($new, commit_request_id) WHERE id AND state IN live RETURNING id. If no row: aborted/expired return quietly; the same state is a no-op; any other case throws body_staging_expired. |
Identical. files 16. |
sweepSegments(opts?: {sourceId?, pageId?, graceMs?, expireStagings?, now?}): {retiredChunksDeleted, manifestRowsDeleted, textsDeleted, stagingsExpired} (engine.ts:2449) |
shared page-body.ts:425-487 |
Six statements in order. (1) If expireStagings: CTE UPDATE page_body_stagings ... SET state='expired' for live stagings past hard or idle deadline (idle applies to non-upload kinds, or uploads still open) and not held by a non-terminal persistence_requests.intent->body_ref; RELEASE_STAGED_CTE releases reserved bytes (persistence/limits.ts). Then delete parts of non-live stagings and terminal stagings older than 24 h. (2) Delete chunks with body_generation > page live generation not owned by a live staging. (3) Delete chunks of occurrences owned only by dead stagings at or below the live generation (page-body.ts:458-462). (4) Delete staged manifest rows of dead stagings (RETURNING sha when pageId set). (5) Delete closed manifest rows not referenced by the live generation or any page_versions row (page-body.ts:467-471). (6) Delete page_segment_texts older than the grace window, unreferenced by any manifest row, not recorded by a live staging (page_body_staging_texts), and not needed by stagingNeedsTextSql; FOR UPDATE SKIP LOCKED. With pageId, only shas released by steps (4)-(5) are candidates. |
Identical SQL; PGLite opts.now binds the same. e2e/segment-gc-parity.test.ts uses PostgresEngine; large-pages/segment-gc.test.ts uses PGLiteEngine. files 19. |
Per-segment derivation watermarks (3 methods)
Shared SQL: page-body.ts:489-551. Kinds (page-body/schema.ts:194): links, mentions, timeline, fences, conversation_facts, facts_backstop, synopsis. States (schema.ts:195): done, partial, failed, skipped. Types in page-body/derivations.ts.
| Method (iface line) | Impl | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
readSegmentDerivations(pageId, opts?: {kind?, occurrenceIds?}): SegmentDerivationRow[] (engine.ts:2452) |
shared page-body.ts:492-513 (unscoped read) |
SELECT ... FROM page_segment_derivations d WHERE d.page_id = $ [AND d.pipeline = kind] [AND d.occurrence_id = ANY(ids)] ORDER BY d.pipeline, d.occurrence_id. created_at/updated_at rendered as UTC text. Empty occurrenceIds returns []. |
Identical. files 15 (helpers/segment-derivation-cases.ts, engine-sql-segment-derivations.test.ts, e2e/segment-derivations-postgres.test.ts is the Postgres-named runner). |
upsertSegmentDerivations(rows): void (engine.ts:2454) |
shared page-body.ts:521-542 |
Batches by bytes; INSERT INTO page_segment_derivations (page_id, pipeline, occurrence_id, pipeline_version, segment_sha256, context_hash, state, output, receipt) SELECT ... FROM jsonb_to_recordset(...) ON CONFLICT (page_id, pipeline, occurrence_id) DO UPDATE SET pipeline_version, segment_sha256, context_hash, state, output, receipt, updated_at = now() (created_at kept). If stored < input, throws DerivationWriteRefused after storing the others. Batches are NOT one transaction. |
Identical. files 10. |
retireSegmentDerivations(pageId, liveOccurrenceIds, opts?: {kind?}): number (engine.ts:2456) |
shared page-body.ts:545-551 |
DELETE FROM page_segment_derivations WHERE page_id = $ [AND pipeline = kind] AND NOT (occurrence_id = ANY(live)); returns affectedRows. |
Identical. files 8. |
Versions (3 methods)
| Method (iface line) | Impl PGLite / PG | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
createVersion(slug, opts?: {sourceId?, preimage?} & VersionHistory): PageVersion (engine.ts:2465) |
pglite-engine.ts:2709 / postgres-engine.ts:2565 -> page-state/versions.ts:25-50 (createPageVersion) |
With preimage: insert directly. Without it: engine.transaction -> tx.lockPageKeys([{sourceId, slug}]) -> tx.readPageSnapshot(slug, {sourceId, includeDeleted: true}); missing page throws createVersion failed. Insert: INSERT INTO page_versions (page_id, compiled_truth, frontmatter, knowledge_revision, timeline, title, type, tags, is_deleted, source_path, change_reason) VALUES (...) RETURNING * (versions.ts:30-35). historyGap runs UPDATE pages SET version_history_gap_at = now(). Segmented pages insert a reference row (body_storage 'segmented', body_generation, body_bytes, content_hash, no body) and prune by pruneSegmentedVersions (versions.ts:56-70). Large inline pages prune older rows with DELETE ... OFFSET and markPrunedHistory (versions.ts:34-38). Default source: sourceId ?? 'default' (pglite-engine.ts:2710, postgres-engine.ts:2566). |
Identical. files 70; eng-parity 0; pglite-engine.test.ts 3; 70 files (e.g. generation-identity-visibility-9386.test.ts, belief-ledger-w5-time-model.test.ts). See parity risk 3. |
getVersions<B>(slug, opts?: GetVersionsOpts<B>): PageVersionRows<B> (engine.ts:2471) |
pglite-engine.ts:2713 / postgres-engine.ts:2569 -> shared pages.ts:1079-1105 |
SELECT <version columns>, <segmented columns>, replaced_because, <valid-time columns> FROM page_versions pv JOIN pages p ON p.id = pv.page_id WHERE p.slug = $ [AND p.source_id = scope] [privacy filter when excludePrivate] AND rederive-hidden predicate ORDER BY pv.snapshot_at DESC, pv.id DESC [LIMIT]. includeBody: false selects metadata columns only. Without a source scope it returns same-slug versions across all sources (matches engine.ts:2467-2470). Segmented rows get numeric body_generation and body_bytes. |
Identical. files 23; eng-parity 0; pglite-engine.test.ts 3; 23 files (e.g. version-history-privacy.test.ts, persistence-version-deletion.test.ts). |
revertToVersion(slug, versionId, opts?: {sourceId?}): void (engine.ts:2477) |
pglite-engine.ts:2717 / postgres-engine.ts:2573 -> shared pages.ts:1108-1155 |
Refuses first (segmentedRevertRefusal) if the page or the version is segmented, before any write. Otherwise UPDATE pages SET compiled_truth = pv.compiled_truth, frontmatter = pv.frontmatter, chunker_version = bodyWriteChunkVersion(...), updated_at = now() FROM page_versions pv WHERE pages.slug = $ [AND pages.source_id = $] AND pv.id = $ AND pv.page_id = pages.id. Does not snapshot the current state first, does not touch title/type/timeline/tags, does not re-chunk or re-embed. |
Identical. files 15; eng-parity 0; pglite-engine.test.ts 2; 15 files (e.g. write-attribution-legacy.test.ts, chunker-version-insert-default.test.ts). |
Stats and health (2 methods)
| Method (iface line) | Impl PGLite / PG | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
getStats(opts?: {sourceId?, sourceIds?}): BrainStats (engine.ts:2488) |
pglite-engine.ts:2726-2778 / postgres-engine.ts:2582-2641. Inline SQL in both engines (not in engine-sql). |
Scope array = sourceIds ?? [sourceId] ?? null, bound as $1::text[]. One scalar query: page_count (live pages), chunk_count, embedded_count (cc.<active embedding column> IS NOT NULL, column from activeEmbeddingColId({fallbackToLegacy: true})), link_count (both endpoints in scope), tag_count (count(DISTINCT tag)), timeline_entry_count. Second query: pages_by_type (live pages, GROUP BY type). Scope logic matches engine.ts:2481-2487. |
PG builds the column as sql.unsafe(colId) and runs via this.sql (no read transaction, no JIT wrapper). PGLite runs through this.db.query. Same statements otherwise. eng-parity 5; pglite-engine.test.ts 2; 46 files (e.g. features-no-sync.test.ts, degraded-engine-recovered-surface-9224.test.ts). |
getHealth(opts?: {sourceId?, sourceIds?}): BrainHealth (engine.ts:2489) |
pglite-engine.ts:2780-2787 / postgres-engine.ts:2643-2652 -> shared engine-sql/health.ts:96-273 |
Three statements, plus two engine callbacks. (1) Counters CTE: entity pages (entityTypePredicateSql, live, not quarantined), chunk totals with embed_skip eligibility, dead_links (link to missing page), link_count, entity_page_count, link_coverage, timeline_coverage. (2) linkable_pages AS MATERIALIZED: grouped by type, islanded and with-timeline counts, using orphanExclusionSql. (3) most_connected: top 5 entity pages by in+out links (ORDER BY link_count DESC, slug COLLATE "C", id). Plus countStalePagesForExtraction and effectiveLinkExtractorWatermark. Score computed in TS: embed 35, link density 25, timeline 15, orphans 15, dead links 10 (health.ts:238-249). Endpoint checks are PK lookups fenced with OFFSET 0 (planner note health.ts:20-27). |
PG wraps each statement in jitOffRead (postgres-engine.ts:2644-2645). PGLite wraps the whole call in plannerRead (pglite-engine.ts:2781). Same shared SQL. eng-parity 4; pglite-engine.test.ts 10; 54 files (e.g. features-no-sync.test.ts, autopilot-tick-health-7236.test.ts, helpers/legacy-get-health.ts). |
Ingest log (2 methods)
| Method (iface line) | Impl PGLite / PG | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
logIngest(entry: IngestLogInput): void (engine.ts:2492) |
pglite-engine.ts:2790-2801 / postgres-engine.ts:2655-2666. Inline SQL. |
INSERT INTO ingest_log (source_id, source_type, source_ref, pages_updated, summary, writer_version, writer_host_id, writer_segment_reader) VALUES (..., $pages_updated::jsonb, ..., $host::uuid, true). source_id defaults to 'default'. Writer stamp from writerStamp({mint: false, engine}) (persistence/writer-versions.ts:36). |
PGLite passes JSON text; PG uses sql.json(...). eng-parity 0; pglite-engine.test.ts 2; 19 files (e.g. doctor-facts-absorb-history-6761.test.ts, import-cancellation.serial.test.ts). |
getIngestLog(opts?: {limit?, sourceIds?}): IngestLogEntry[] (engine.ts:2497) |
pglite-engine.ts:2803-2820 / postgres-engine.ts:2668-2684. Inline SQL. |
SELECT id, source_id, source_type, source_ref, pages_updated, summary, created_at FROM ingest_log [WHERE source_id = ANY($::text[])] ORDER BY created_at DESC LIMIT $ (default 50). source_id ?? 'default' fallback applied in TS. |
PG uses a template fragment for the scope; PGLite picks one of two SQL strings. eng-parity 0; pglite-engine.test.ts 2; 11 files (e.g. e2e/connectors-sync-pglite.test.ts, e2e/sync.test.ts). |
Sync: slug, links, aliases, page body (9 methods)
| Method (iface line) | Impl PGLite / PG | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
updateSlug(oldSlug, newSlug, opts?: {sourceId?}): number (engine.ts:2511) |
pglite-engine.ts:2823-2828 / postgres-engine.ts:2687-2692 -> shared pages.ts:1157-1177 |
validateSlug(newSlug), then this.transaction(...). Inside: readPageSnapshot(old, {sourceId, includeDeleted: true}) and createPageVersion snapshot of the old state. UPDATE pages SET slug = $new, updated_at = now() WHERE slug = $old AND source_id = $ RETURNING id. If anything moved: recordRenameAlias (writes slug_aliases, page-state/rename-alias.ts:20) and moveSlugBindings (rename-alias.ts:86). Returns the number of rows moved; zero means no row, no throw. |
Same. eng-parity 0; pglite-engine.test.ts 2; 31 files (e.g. sync-rename-collision.test.ts). |
rewriteLinks(oldSlug, newSlug): void (engine.ts:2512) |
pglite-engine.ts:2830-2832 / postgres-engine.ts:2694-2699 |
No-op on both. Links use integer page_id FKs, so a rename needs no link rewrite. Textual [[wikilinks]] in bodies are NOT rewritten. |
Same. eng-parity 0; pglite 0; 8 files. Listed in the UNCALLED allowlist of test/engine-surface-coverage.test.ts as a documented stub. |
resolveSlugWithAlias(slug, sourceOrSources, opts?: {excludePrivate?}): string (engine.ts:2535) |
pglite-engine.ts:2834-2840 / postgres-engine.ts:2701-2707 |
Thin wrapper: (await resolveSlugWithAliasDetailed(...))?.canonical_slug ?? slug. |
Same. eng-parity 2; pglite 0; phantom-redirect-merge.test.ts, managed-facts-writers.test.ts, type-unification-full-flow.test.ts. |
resolveSlugWithAliasDetailed(slug, sourceOrSources, opts?): {canonical_slug, source_id} | null (engine.ts:2548) |
pglite-engine.ts:2842-2848 / postgres-engine.ts:2709-2715 -> shared pages.ts:1194-1223 (unscoped read) |
SELECT canonical_slug, source_id FROM slug_aliases WHERE alias_slug = $ AND source_id = ANY($sources::text[]) [privacy: EXISTS live page the caller can read] ORDER BY array_position($sources, source_id), id. First row wins; more than one row emits a once-per-process multi_match warning. Pre-v105 brain (missing table slug_aliases): returns null (isMissingSchemaObject). |
Same. eng-parity 9; pglite 0; e2e/engine-content-privacy.test.ts, schema-error-write-paths.test.ts, fixtures/goldens/sql-text/pages.json. |
resolveAliases(aliasNorms: string[], opts?: PageReadScope): Map<alias_norm, Array<{slug, source_id}>> (engine.ts:2567) |
pglite-engine.ts:2850-2855 / postgres-engine.ts:2717-2722 -> search/read-enrichment.ts:343 readAliases |
Same function for both, given executeRaw.bind(this). CTE a AS MATERIALIZED (SELECT alias_norm, slug, source_id, rank FROM page_aliases WHERE alias_norm = ANY($1) AND <scope>), rank 2 for origin subject, 1 for declared, 0 for frontmatter. Joined to live readable pages with CROSS JOIN LATERAL (... OFFSET 0). A derived alias (rank > 0) is dropped when another readable page has the exact same lowercase title. Output sorted by alias, source, rank. Throws on pre-v110 brains (caller wraps). |
Same. eng-parity 0; pglite 0; 22 files (read-aliases-title-scope.test.ts, search/qualified-alias-visibility.test.ts, search/pre-migration-failopen.test.ts). |
setPageAliases(slug, sourceId, aliasNorms: string[], opts?: {inline?}): void (engine.ts:2580) |
pglite-engine.ts:2857-2860 / postgres-engine.ts:2724-2727 -> shared pages.ts:1179-1192 |
inline and already in a page transaction: run directly; else this.transaction(...). Inside: tx.lockPageKeys([{sourceId, slug}]); DELETE FROM page_aliases WHERE source_id = $ AND slug = $ AND origin = 'frontmatter'; if any aliases: INSERT INTO page_aliases (source_id, alias_norm, slug) SELECT ... FROM unnest($::text[]) ON CONFLICT DO NOTHING. Only frontmatter-origin rows are replaced (derived declared and subject rows are left alone). |
Same. eng-parity 0; pglite 0; 37 files (migrate-engine-raw-include-deleted.test.ts, recall-entity-keying-4828.test.ts, canonical-writer-inventory.test.ts). See parity risk 8. |
refreshPageBody(slug, sourceId, compiledTruth, timeline, contentHash): void (engine.ts:2604) |
pglite-engine.ts:1329-1337 / postgres-engine.ts:952-960 -> shared pages.ts:405-428 |
UPDATE pages SET compiled_truth = $1, timeline = $2, content_hash = $3, chunker_version = bodyWriteChunkVersion('$1','$2'), updated_at = now() WHERE source_id = $ AND slug = $ AND deleted_at IS NULL. Leaves frontmatter, chunks, links, embeddings, tags and takes alone (engine.ts:2593-2599: used by the extract_facts phantom redirect). |
Same. eng-parity 0; pglite 0; 21 files (phantom-redirect-engine-parity.test.ts, fence-chunk-overlay.test.ts, facts-write-path-failures.test.ts, canonical-writer-inventory.test.ts). |
updatePageContextualRetrievalState(slug, sourceId, mode, corpusGeneration: string | null): void (engine.ts:2629) |
pglite-engine.ts:1339-1346 / postgres-engine.ts:962-969 -> shared pages.ts:431-... (transactionMemo passed in) |
Two shapes. (a) mode !== 'none': UPDATE pages SET contextual_retrieval_mode = $mode, corpus_generation = $gen, updated_at = now() WHERE source_id AND slug AND deleted_at IS NULL. (b) mode === 'none': reads config (search_embedding_column, embedding_columns, embedding_model) through the memo, then one statement with previous AS MATERIALIZED (... FOR UPDATE) and changed CTE that sets the mode and corpus_generation, and for a page whose previous mode was title or per_chunk_synopsis (and not embed_skip) also NULLs embedding_signature, then UPDATE content_chunks sets the active embedding column to NULL, embedded_at, embedded_text_hash, embedding_input_hash to NULL and embedding_pending_since = now(). |
Same SQL. Doc says this method only stamps the two columns plus updated_at (engine.ts:2612-2628); branch (b) also clears chunk vectors. See parity risk 9. eng-parity 0; pglite 0; 21 files (contextual-retrieval-service-pure.test.ts, persistence-embedding-effects.test.ts, page-projection-vector-preservation.test.ts). |
migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId): {migrated} (engine.ts:2650) |
pglite-engine.ts:1348-1375 / postgres-engine.ts:971-1007 |
(1) dropPhantomMigrationDuplicates(tx, sourceId, canonical, phantom) (facts/ontology-rekey.ts:44-47) first removes phantom observations the canonical page already holds (idx_facts_ontology_stint_dedup collision, B-10). (2) UPDATE facts SET entity_slug = $canonical, source_markdown_slug = $canonical, row_num = row_num + COALESCE((SELECT MAX(f2.row_num) FROM facts f2 WHERE f2.source_id AND f2.source_markdown_slug = $canonical AND f2.row_num IS NOT NULL), 0) WHERE source_id AND source_markdown_slug = $phantom AND expired_at IS NULL RETURNING id. NULL row_num stays NULL. Expired rows stay on the phantom slug. Re-run is a no-op. |
Same SQL. PG issues it through the tagged template on this.sql, PGLite via this.db.query. eng-parity 0; pglite 0; 15 files (phantom-redirect-engine-parity.test.ts, phantom-redirect.test.ts, e2e/phantom-redirect.test.ts). |
Config (5 methods)
Config table is config (key, value). Both engines call requestConfigValue / requestConfigRows first (config-snapshot.ts). PG wraps the SQL in connRetry (postgres-engine.ts:2753-2765: withRetry with the bulk-retry options and reconnect), so a dropped pooler connection is retried and the pool rebuilt. PGLite has no retry wrapper.
| Method (iface line) | Impl PGLite / PG | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
getConfig(key): string | null (engine.ts:2657) |
pglite-engine.ts:2863-2868 / postgres-engine.ts:2767-2777 |
Request snapshot first; else SELECT value FROM config WHERE key = $. Returns null when the key is unset. |
PG only: connRetry. eng-parity 0; pglite 4; 489 files (ubiquitous). |
setConfig(key, value): void (engine.ts:2658) |
pglite-engine.ts:2870-2877 / postgres-engine.ts:2779-2787 |
INSERT INTO config (key, value) VALUES ($1, $2) ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value; then noteRequestConfigWrite. Autocommit, no explicit transaction. |
PG only: connRetry (idempotent upsert). pglite 2; 1387 files. |
unsetConfig(key): number (engine.ts:2664) |
pglite-engine.ts:2879-2886 / postgres-engine.ts:2789-2796 |
DELETE FROM config WHERE key = $. Returns affected rows (0 or 1). Calls noteRequestConfigWrite(key, null). |
Row count from affectedRows (PGLite) vs count (PG). eng-parity 0; pglite 0; 228 files. |
listConfigKeys(prefix): string[] (engine.ts:2670) |
pglite-engine.ts:2888-2896 / postgres-engine.ts:2797-2808 |
The prefix is LIKE-escaped (\, %, _) in TS, then SELECT key FROM config WHERE key LIKE $pattern ESCAPE '\' ORDER BY key. Literal prefix match, no glob. |
PG wraps in connRetry. eng-parity 0; pglite 0; 18 files (config-db-merge.test.ts, search-introspection-ops.test.ts, postgres-engine-config-reconnect.test.ts). |
getAllConfig(): Record<string, string> (engine.ts:2684) |
pglite-engine.ts:2898-2905 / postgres-engine.ts:2810-2821 |
Request snapshot if one applies (requestConfigRows); else SELECT key, value FROM config (whole table, one round trip). Used by loadConfigWithEngine (about 44 keys per connect, engine.ts:2672-2683). |
PG wraps in connRetry. eng-parity 0; pglite 0; 20 files (config-request-snapshot.test.ts, read-waves-gbra133.test.ts, requester-scope.test.ts). |
Migration support and raw SQL (5 methods)
| Method (iface line) | Impl PGLite / PG | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
runMigration(version: number, sql: string): void (engine.ts:2687) |
pglite-engine.ts:2908-2910 / postgres-engine.ts:2824-2827 |
Runs the migration text unmodified: PGLite this.db.exec(sql) (multi-statement), PG this.sql.unsafe(sqlStr) with no parameters. The version argument is ignored by both (_version). The runner that chooses versions is src/core/migrate.ts (not read in this part). |
Different APIs: PGLite exec vs PG unsafe; multi-statement behavior must match on both (not verified here). eng-parity 0; pglite 0; 36 files (migrate.test.ts, v273-validate-timeouts.test.ts, function-search-path-pglite.test.ts, oauth-client-registered-via-migration.test.ts). |
getChunksWithEmbeddings(slug, opts?: {sourceId?, includeUnsealed?}): Chunk[] (engine.ts:2692) |
pglite-engine.ts:2912-2914 / postgres-engine.ts:2829-2831 -> shared chunks.ts:1193-1209 (unscoped read) |
SELECT cc.* FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE <chunkVisibleFilter('p','cc') or chunkGenerationFilter when includeUnsealed> AND p.slug = $ [AND p.source_id = $] ORDER BY cc.chunk_index. Raw preservation tools may include unverified chunks (engine.ts:2691). |
Identical. eng-parity 0; pglite-engine.test.ts 2; 14 files (e.g. incremental-chunking.test.ts). |
executeRaw<T>(sql, params?, opts?: {signal?, timeoutMs?, deadlineAt?, jitOff?, prepare?}): T[] (engine.ts:2714) |
pglite-engine.ts:2916-2942 / postgres-engine.ts:2990-3018 (+ runUnsafe 2934-2992) |
PGLite: pre-check signal (AbortError), then db.query(sql, params). Autocommit WAL-writing statements (per writesWal) go through PgliteCheckpointGuard.runStatement (pglite-engine/checkpoint-guard.ts). An abort rejects the caller but the statement keeps running in WASM. timeoutMs, deadlineAt, jitOff, prepare are ignored. Postgres: throwIfRequestAborted(), checkoutGauge 'raw'. If jitOff or timeoutMs (outside a transaction): holdSearchConnection, then JIT-off envelope (jit-off-statement.ts) or runBoundedStatement (BEGIN; SET LOCAL statement_timeout; <sql>; COMMIT as one pipelined round trip, bounded-statement.ts, 57014 on timeout). Else retryReadOnlyAutocommit(this.sql, runUnsafe). runUnsafe reserves a connection when a signal is present, sets prepare (default true, #5984), and on abort sends CancelRequest, then discards the connection after cancelSettleMs. No automatic retry of mutations on connection death (comment postgres-engine.ts:3011-3017). |
PGLite has no server-side cancel, no timeout, no JIT control. PG retries only a read-only-transaction refusal (SQLSTATE 25006) by reopening BEGIN READ WRITE (read-write-transactions.ts:65-76). eng-parity 46; pglite 1; 3285 files (ubiquitous). |
executeRawDirect<T>(sql, params?, opts?: {signal?}): T[] (engine.ts:2732) |
pglite-engine.ts:2955-2961 (delegates to executeRaw) / postgres-engine.ts:3034-3059 |
PG: signal pre-check before pool wait; if inside an open transaction (_sql is the reserved tx connection), stays on it; else if dual pool is active, uses connectionManager.ddl() (session-mode direct pool, port 5432 per engine.ts:2720-2729); else this.sql. checkoutGauge 'direct', then retryReadOnlyAutocommit. Single-statement UPDATEs only (Minion lock hot path). |
PGLite is identical to executeRaw (no pooler). eng-parity 0; pglite 0; 30 files (persistence-diagnostics.test.ts, pool-gauge.test.ts, engine-control-routing.test.ts, sync-run-ordering.serial.test.ts). |
keywordBoundedRead<T>(read: (db: Pick<BrainEngine,'executeRaw'>) => Promise<T>): T (engine.ts:2745) |
pglite-engine.ts:2945-2947 / postgres-engine.ts:1507-1510 |
PG: resolve keyword_statement_timeout_ms once per connection (keyword-statement-timeout.ts, default 8000, range 1000-60000), then withScopedReadTransaction(undefined, undefined, tx => read(engineSqlOn(tx)), {alwaysTransaction: true, settings: {statement_timeout}}) so every statement in read is bounded and transaction-local. Used for the segmented-region read with its chunk-manifest probe (engine.ts:2740-2744). PGLite: read(this), no timeout. |
PG-only bound (57014 on overrun); PGLite unbounded. eng-parity 0; pglite 0; 9 files (read-transaction-early-result.test.ts, postgres-rls-scope-inventory.test.ts, engine-surface-coverage.test.ts). |
Code edges (6 methods)
Shared SQL: src/core/engine-sql/code-edges.ts. Tables: code_edges_chunk (resolved, to_chunk_id set), code_edges_symbol (unresolved, to_symbol_qualified set; resolved_chunk_id in edge_metadata for resolver-stamped rows). Batch sizes: floor(maxBindParams / binds-per-row) (code-edges.ts:14-16): PGLite 30000 binds, PG unbounded.
| Method (iface line) | Impl PGLite / PG | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
addCodeEdges(edges): number (engine.ts:2756) |
pglite-engine.ts:2980-2982 / postgres-engine.ts:3071-3073 -> shared code-edges.ts:31-60 |
Resolved edges: INSERT INTO code_edges_chunk (from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id) VALUES ... ON CONFLICT (from_chunk_id, to_chunk_id, edge_type) DO NOTHING. Unresolved: INSERT INTO code_edges_symbol (...) ON CONFLICT (from_chunk_id, to_symbol_qualified, edge_type) DO NOTHING. edge_metadata as per-row ::text::jsonb. Returns affected rows. |
Same SQL; PGLite batches by bind count. eng-parity 1; pglite 0; 20 files (code-edges.test.ts, e2e/code-edges-jsonb-postgres.test.ts). |
deleteCodeEdgesForChunks(chunkIds: number[]): void (engine.ts:2765) |
pglite-engine.ts:2984-2986 / postgres-engine.ts:3075-3077 -> shared code-edges.ts:62-68 |
DELETE FROM code_edges_chunk WHERE from_chunk_id = ANY(...) OR to_chunk_id = ANY(...), then DELETE FROM code_edges_symbol WHERE from_chunk_id = ANY(...). Both directions for the resolved table, outbound for the symbol table. Empty input returns early. |
Same. eng-parity 0; pglite 0; 10 files (code-edges.test.ts). |
getCallersOf(qualifiedName, opts?: {sourceId?, allSources?, limit?, chunkId?}): CodeEdgeResult[] (engine.ts:2773) |
pglite-engine.ts:2988-2994 / postgres-engine.ts:3079-3085 -> shared code-edges.ts:75-102 (unscoped read) |
UNION ALL of code_edges_chunk (to_symbol_qualified = $name or to_chunk_id = $chunkId) and code_edges_symbol (to_symbol_qualified = $name or (edge_metadata->>'resolved_chunk_id')::int = $chunkId), each filtered by currentCodeEdgeFilter(...) (code-intel/read-scope.ts), optional source_id = $ unless allSources. Shared LIMIT default 100, cap 500. |
Same. eng-parity 0; pglite 0; 24 files (import-code-edges-source-id.test.ts, code-intel/n13-edge-resolution.test.ts, e2e/cli-source-scoping-pglite.test.ts). |
getCalleesOf(qualifiedName, opts?: {sourceId?, allSources?, limit?, bareFallback?, chunkId?}): CodeEdgeResult[] (engine.ts:2783) |
pglite-engine.ts:2995-3000 / postgres-engine.ts:3086-3091 -> shared code-edges.ts:104-134 |
Edges whose from_symbol_qualified matches (or whose from-chunk is chunkId), same source and current-filter rules. bareFallback (#4670): if zero rows and the name has no ., #, :, re-key on from_chunk_id IN (SELECT id FROM content_chunks WHERE symbol_name = $). |
Same. eng-parity 8; pglite 0; 18 files (import-code-edges-source-id.test.ts, code-intel/recursive-walk-resolved-ids-5439.test.ts, code-edges.test.ts). |
getEdgesByChunk(chunkId, opts?: {direction?, edgeType?, limit?}): CodeEdgeResult[] (engine.ts:2793) |
pglite-engine.ts:3002-3007 -> PGLite-only pglite-engine/code-edges.ts:17-54 / postgres-engine.ts:3093-3098 -> shared code-edges.ts:141-172 |
direction default both; limit default 50 cap 200. Two statements: code_edges_chunk (to_chunk_id = $1 for in, from_chunk_id = $1 for out, either for both) and, unless in, code_edges_symbol WHERE from_chunk_id = $1. Each table has its own LIMIT. edgeType bound as $3/edge_type = $. Rows mapped by shared rowToCodeEdge (code-edges.ts:182-195). |
Two implementations kept in sync by hand (see parity risk 6). eng-parity 0; pglite 0; 12 files (code-edges.test.ts, e2e/code-edges-read-parity.test.ts, code-intel/n13-edge-resolution.test.ts). |
searchKeywordChunks(query, opts?: SearchOpts): SearchResult[] (engine.ts:2804) |
pglite-engine.ts:1638-1724 / postgres-engine.ts:1291-1415 (inline SQL both) |
Chunk-grain FTS: ts_rank(cc.search_vector, websearch_to_tsquery(lang, $1)) * sourceFactorCase AS score, WHERE cc.search_vector @@ websearch_to_tsquery(...), hard-exclude clause, visibility clause, detail: 'low' restricts to compiled_truth, ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT OFFSET. CJK queries go to _searchKeywordCJK (dedup: false) on both. No page dedup (that is searchKeyword). |
Not equivalent. PG additionally honors opts.type, opts.types, opts.where (frontmatter filter), opts.exclude_slugs. PGLite ignores all four. PG returns constant stale = false; PGLite computes stale from timeline_entries. eng-parity 7; pglite 1; 27 files. See parity risk 1. |
Eval capture (5 methods)
Table: eval_candidates and eval_capture_failures. Writers and readers are small raw statements. Insert helper: src/core/feedback/store.ts:340 (insertEvalCandidate, shared by both engines).
| Method (iface line) | Impl PGLite / PG | What it does and SQL | PGLite vs PG; tests |
|---|---|---|---|
logEvalCandidate(input: EvalCandidateInput): number (engine.ts:2812) |
pglite-engine.ts:3010-3012 / postgres-engine.ts:3101-3103 -> feedback/store.ts:340- |
INSERT INTO eval_candidates (tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids, expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied, latency_ms, remote, job_id, subagent_id, embedding_column) VALUES (...) RETURNING id. retrieved_slugs is filtered to slugs that exist (FOR KEY SHARE) or facts/<id> rows that exist, in input order. |
Same. eng-parity 0; pglite 0; 17 files (eval-candidates.test.ts, eval-capture-drain.test.ts, e2e/eval-replay-column.test.ts). |
listEvalCandidates(filter?: {since?, limit?, tool?}): EvalCandidate[] (engine.ts:2814) |
pglite-engine.ts:3014-3036 / postgres-engine.ts:3105-3130 |
SELECT * FROM eval_candidates WHERE created_at >= $since [AND tool_name = $tool] ORDER BY created_at DESC, id DESC LIMIT $. limit default 1000, cap 100000; since default epoch. |
Same SQL; PG via tagged template. eng-parity 0; pglite 0; 11 files (eval-candidates.test.ts, e2e/eval-capture.test.ts, mcp-eval-capture.test.ts). |
deleteEvalCandidatesBefore(date: Date): number (engine.ts:2816) |
pglite-engine.ts:3038-3044 / postgres-engine.ts:3132-3138 |
DELETE FROM eval_candidates WHERE created_at < $date RETURNING id; returns rows length. |
Same. eng-parity 0; pglite 0; 7 files. |
logEvalCaptureFailure(reason): void (engine.ts:2818) |
pglite-engine.ts:3046-3051 / postgres-engine.ts:3140-3143 |
INSERT INTO eval_capture_failures (reason) VALUES ($). Best effort (callers swallow errors, engine.ts:2811). |
Same. eng-parity 0; pglite 0; 9 files (eval-candidates.test.ts, eval-capture-drain.test.ts). |
listEvalCaptureFailures(filter?: {since?}): EvalCaptureFailure[] (engine.ts:2820) |
pglite-engine.ts:3053-3060 / postgres-engine.ts:3145-3154 |
SELECT * FROM eval_capture_failures WHERE ts >= $since ORDER BY ts DESC. |
Same. eng-parity 0; pglite 0; 8 files (eval-candidates.test.ts, mcp-eval-capture.test.ts). |
Salience, anomalies and enrich candidates (5 methods)
Shared SQL: src/core/engine-sql/salience.ts. Statement text is Postgres master text (engine.ts comment at salience.ts:2-6); PGLite runs the same statements. Every read is unscoped (LegacyUnscopedRead). Both engines call salienceImpl.* with unscopedExecutor(this.engineSql, ...) (PGLite pglite-engine.ts:3069-3087, PG postgres-engine.ts:3163-3181), except setEmotionalWeightBatch, which passes this.engineSql (a write).
| Method (iface line) | What it does and SQL | Tests |
|---|---|---|
batchLoadEmotionalInputs(slugs?: string[]): EmotionalWeightInputRow[] (engine.ts:2842) |
salience.ts:27-75. Two CTEs avoid a tag x take cross product: page_tags (array_agg(DISTINCT tag)) and page_takes (json_agg of active takes: holder, weight, kind, active). pages LEFT JOIN both on page_id. WHERE p.slug = ANY($slugs) when given. Each row carries source_id. |
eng-parity 0; pglite 0; 13 files (recompute-emotional-weight.test.ts, e2e/salience-pglite.test.ts, backfill-base.test.ts). |
setEmotionalWeightBatch(rows: EmotionalWeightWriteRow[]): number (engine.ts:2853) |
salience.ts:77-103. UPDATE pages SET emotional_weight = u.weight, salience_touched_at = now() FROM unnest($slugs::text[], $sourceIds::text[], $weights::real[]) AS u(slug, source_id, weight) WHERE pages.slug = u.slug AND pages.source_id = u.source_id AND pages.emotional_weight IS DISTINCT FROM u.weight RETURNING 1. Composite key on (slug, source_id). Returns changed rows only. |
eng-parity 0 (salience parity file is separate: e2e/engine-parity-salience.test.ts); pglite 0; 16 files (recompute-emotional-weight.test.ts, e2e/multi-source-emotional-weight-pglite.test.ts). |
getRecentSalience(opts: SalienceOpts): SalienceResult[] (engine.ts:2863) |
salience.ts:105-182. Boundary computed in JS and bound as timestamptz (salience.ts:110). Window: GREATEST(updated_at, COALESCE(salience_touched_at, updated_at)) >= boundary (or updated_at when holders are restricted). Score: emotional_weight * 5 + ln(1 + COUNT(DISTINCT active take id)) + recency component (buildRecencyComponentSql, flat by default, per-prefix decay when recency_bias: 'on'). Excludes briefings/% unless the prefix asks for it; optional slugPrefix LIKE with escape; source scope; pageReadFilter (privacy). GROUP BY p.id ORDER BY score DESC LIMIT (default 20, cap 100). |
eng-parity 0; pglite 0; 13 files (e2e/salience-pglite.test.ts, e2e/engine-parity-salience.test.ts, e2e/salience-anomalies-source-isolation-pglite.test.ts). |
listEnrichCandidates(opts: EnrichCandidatesOpts): EnrichCandidate[] (engine.ts:2877) |
salience.ts:184-250. Empty types returns [] without SQL. WHERE deleted_at IS NULL AND type = ANY($types) AND (char_length(compiled_truth) + char_length(COALESCE(timeline,''))) < threshold plus source scope, optional re-enrich guard (frontmatter->>'enriched_at' older than cutoff), and dream_generated IS DISTINCT FROM 'true'. inbound_count is a correlated count over links with link_source IS DISTINCT FROM 'mentions' and rankingLinkSql. ORDER BY ENRICH_ORDER_SQL[order] (default inbound-links), LIMIT 1..5000. Lightweight projection, no bodies. |
eng-parity 2; pglite 0; 12 files (e2e/enrich-pglite.test.ts, e2e/read-enrichment-privacy.test.ts, amends-ranking-scenarios.ts). |
findAnomalies(opts: AnomaliesOpts): AnomalyResult[] (engine.ts:2889) |
salience.ts:252- (two aggregate statements visible at lines 274-320: tag cohort, then type cohort). Baseline window [since - lookback, since), today [since, since + 1 day). Both use generate_series day densification with zero fill (so sparse cohorts are not under-counted). Counts are COUNT(DISTINCT p.id) per cohort per day from pages.updated_at. The final sigma test runs in TS: computeAnomaliesFromBuckets (cycle/anomaly.ts). Default sigma 3.0, lookback 30 days. |
eng-parity 0; pglite 0; 14 files (e2e/anomalies-pglite.test.ts, e2e/salience-anomalies-source-isolation-pglite.test.ts, e2e/engine-parity-salience.test.ts). |
Parity risks noted
searchKeywordChunksfilter drift: Postgres honorstype,types,where(frontmatter filter) andexclude_slugs; PGLite (pglite-engine.ts:1638-1724) ignores all four. Thestaleflag also differs (PG constantfalseatpostgres-engine.ts:1398; PGLite computed fromtimeline_entriesatpglite-engine.ts:1713). Any caller that narrows anchors by type gets different results per engine. Rust needs one filter contract tested on both.- Fact insert concurrency: the per-entity advisory lock (
facts.ts:120-123,149-152) exists only on Postgres. PGLite inserts do not serialize the dedup window; a Rust embedded engine that is single-writer is fine, but the dedup contract must not rely on the lock. createVersiondefault source: whensourceIdis omitted,createPageVersionuses'default'(pglite-engine.ts:2710,postgres-engine.ts:2566). The interface doc (engine.ts:2459-2463) says the bare-slug lookup picks whichever row Postgres returns first. The code does not do that. Rust should follow the code, or the doc should be fixed first.insertFactsreturn alignment:idsdrops ON CONFLICT-skipped rows (facts.ts:318), so callers cannot map index to id, contrary to the doc atengine.ts:2236. Keep the contract explicit (returnnullper input, or document the filter).- Embedding cast: Postgres probes
facts.embeddingfor::halfvecvs::vector(postgres-engine.ts:2271-2304); PGLite always casts::vector. A Rust store with one fixed vector type must not depend on the probe; the dimension and type must be identical to what both engines read back. getEdgesByChunkhas two implementations (pglite-engine/code-edges.tsandengine-sql/code-edges.ts:141-172). Theengine-sqlheader (code-edges.ts:6-10) still says PGLite runs "one UNION with a shared LIMIT", but the PGLite file now applies a per-table LIMIT. Comment and code have drifted; the Rust version needs one behavior.executeRawsemantics: PGLite ignorestimeoutMs,deadlineAt,jitOff,prepareand cannot cancel a running statement. Postgres cancels withCancelRequestand bounds withstatement_timeout. Callers that rely on a timeout (search, Minion locks) see no bound on PGLite. The interface doc (engine.ts:2696-2713) describes Postgres behavior.setPageAliasesreplaces onlyorigin = 'frontmatter'rows (pages.ts:1188). The interface doc (engine.ts:2572-2579) says it deletes "the page's existing page_aliases rows". Declared and subject aliases survive a frontmatter rewrite on purpose; the doc is out of date.updatePageContextualRetrievalStatemode'none'also clears chunk vectors andembedding_signaturewhen the old mode wastitleorper_chunk_synopsis(pages.ts:431-...). The interface doc (engine.ts:2612-2628) says it stamps only the two columns plusupdated_at. A Rust port that follows the doc would keep stale vectors.- Segmented flip and staging:
flipBodyGenerationrelies onFOR UPDATEon the page row and staging row (page-body.ts:294,309), andstageSegmentChunkson advisory lock plusFOR KEY SHARE(page-body.ts:193-197, Postgres only). PGLite has neither. A Rust embedded store needs its own serialization (a per-page mutex or single writer) and must reproduce the generation checks (revision_changed,body_staging_expired,generation_incomplete). upsertSegmentDerivationsis not atomic across batches: earlier batches commit beforeDerivationWriteRefusedis thrown (page-body.ts:521-542).sweepSegmentsruns six statements without one transaction (page-body.ts:425-487); its correctness depends on the ordering and grace windows (1 h texts, 1 h idle and 24 h hard staging deadlines,page-body/limits.ts:132-135).- Chunk indexes:
stageSegmentChunksderiveschunk_indexfromMAX(chunk_index) + 1per page and never reuses an index (page-body.ts:219-221). Deleted chunks leave gaps, so a Rust store must not compact indexes. runMigrationruns raw SQL text: PGLitedb.execvs PGsql.unsafe(pglite-engine.ts:2908,postgres-engine.ts:2824). Multi-statement and transaction-control behavior of migration files must be checked on the Rust side. Theversionargument is ignored.getStatsandgetHealthrun their own statements on each engine (getStatsis inline in both;getHealthshareshealth.tsbut is wrapped differently: PGjitOffReadper statement, PGLiteplannerRead). Results must match across engines; the planner-robustness notes athealth.ts:20-27mean a Rust rewrite cannot rely on query plans for correctness.resolveAliasesthrows on pre-v110 brains (engine.ts:2563-2566), whileresolveSlugWithAliasDetailedreturns null on pre-v105 brains (pages.ts:1194-1223). Callers depend on these two different fail modes.- Config: Postgres retries connection-class failures on every config accessor (
postgres-engine.ts:2753-2765); PGLite has no equivalent. The request-scoped config snapshot (config-snapshot.ts) is used by both, so reads can be served from a snapshot taken before a write in the same request. - Facts health and recall reads use
now()for validity (valid_until > now()) in SQL on both engines, so results depend on the wall clock at read time (TTL honesty,facts.ts:419-427). A Rust engine has to evaluate the same predicate at read time, not at write time.
What a Rust engine must do to pass engine-parity
Scope: test/e2e/engine-parity.test.ts (2762 lines, commit 8bbfd0b5e8), its helpers, and docs/TESTING.md. The file compares the Postgres engine against the in-process PGLite engine. No engine list is parametrized. A Rust engine has to be added by hand to every describe block. Five things block a pass as the file stands:
- The file is gated by
DATABASE_URL. Without it, 16 of its 17 describe blocks skip, including the PGLite half. Only thegetRawDatablock runs (1 test). - About 46
executeRawcalls and 17installFixtureChunkscalls rely on Postgres SQL and on the page-projection protocol. A Rust engine must either run that SQL or the test must be rewritten to typed methods. - Test 1592 reads
EXPLAIN (FORMAT JSON)plans and asserts index names. This is Postgres-only. - The file never compares scores (no
toBeCloseTo). Ordering and exact nullability are what matter. undefinedvsnullmatters in several places (see section 4).
Counts: 17 describe blocks (16 describeBoth + 1 ungated describe). 74 static test( calls. The for (const q of QUERIES) loop at line 137 turns one test into 3 runs. With DATABASE_URL set, 76 tests run. Without it, 1 runs.
1. How the parity test is structured
Gating. SKIP_PG = !hasDatabase() and describeBoth = SKIP_PG ? describe.skip : describe (test/e2e/engine-parity.test.ts:32-33). hasDatabase() is !!process.env.DATABASE_URL (test/e2e/helpers.ts, hasDatabase). The header comment at lines 4-14 says "Always runs the PGLite half". That is wrong: the whole block is inside describeBoth, so PGLite is skipped too. The only always-on block is the getRawData describe at lines 2714-2762. Its arms array (arms, 2716-2723) pushes pglite always and postgres only when !SKIP_PG. That array is the closest existing pattern for a multi-engine list.
Engine construction. Postgres arm: setupDB() or setupLegacyEmbeddingDB() from test/e2e/helpers.ts. Both go through db.ts and PostgresEngine. setupDB TRUNCATEs every table in ALL_TABLES and re-seeds config (test/e2e/helpers.ts, setupDB). setupLegacyEmbeddingDB also configures the gateway to LEGACY_EMBEDDING_CONFIG (test/helpers/legacy-embedding-config.ts: openai:text-embedding-3-large, 1536 dims) and alters the vector column widths. Isolated variant: isolatedPersistencePostgres() in test/helpers/persistence-postgres.ts creates a fresh database per describe (used at 2386-2404 for open_loops only).
PGLite arm: new PGLiteEngine(); await e.connect({}); await e.initSchema(); (for example 126-128, 1368-1370, 1457-1459). Each describe creates both and seeds them with the same function, usually seedX(eng) run twice.
Teardown: afterAll calls pgliteEngine.disconnect() and teardownDB() (30s timeout). beforeAll budgets are 90s, or 180s for the dense-hub block (2201).
Fixtures.
- Main corpus:
SEED_PAGES(49-89) with 6 pages, each withembeddingDimand a one-hot embedding frombasisEmbedding(idx, 1536)(35-39,Float32Arrayof length 1536). QUERIES(112-116):'fat code thin harness','fat code thin harness part 3','fat code production'.seedEngine(91-110):putPage+installFixtureChunksfor each page, onecompiled_truthchunk withtoken_count = body.split(/\s+/).length.- Other seeds:
seedRelational(1338-1359),seedFederated(1428-1447),seedTraversal(2023-2039),seedDenseHub(test/helpers/dense-hub.ts, 200 inbound + 200 outbound spokes, ring offsets, 5200 edges),seedAliasOwners(2241-2256),seedKeyset(1624-1639), CJK corpus (1952-1969),seedin the timeline plan block (1561-1568). - Helper
installFixtureChunks(test/helpers/page-projection.ts) callsreadProjectionSnapshot, theninstallPageProjection(..., {seal: true}). These are insrc/core/page-state/projections.ts. They useengine.transaction,engine.lockPageKeys,engine.readPageSnapshot,engine.executeRaw("UPDATE pages SET text_projection_revision=knowledge_revision, search_vector=..."), andpage_projection_jobs. So a Rust engine must implement the projection revision model for the fixture helper to work at all.
Comparison style.
- Most tests run the same call on both engines and
expect(pgX).toEqual(pgliteX), or comparenew Set(...), or compare a derived shape string. - Exact expectations are then pinned against absolute values, so both engines cannot be wrong in the same way.
- Ordering:
toEqualon arrays is order-sensitive. Many tests.sort()first, which means they do not check order (see table). - Scores: the only score asserted is
DreamVerdict.scoreexact0.31(line 417). Search scores are never compared. - Timestamps: checked with
instanceof Date,Number.isFinite(getTime()),toMatch(/\.\d{6}Z$/), or exact ISO strings (for example'2026-09-01T23:59:59.000Z'at 2477).
Plugging in a third engine. Concrete steps, based on the current code:
- Extend the engine kind union:
readonly kind: 'postgres' | 'pglite'(src/core/engine.ts:832),EngineConfig.engine?: 'postgres' | 'pglite'(src/core/types.ts:2343), and theswitchinsrc/core/engine-construct.ts:9-22(thedefaultthrows). - Add an engine list to the test. Two existing helpers show the pattern.
test/helpers/two-engine.tshastype EngineKind = 'pglite' | 'postgres'andopenTestBrain(kind, settings)throughcreateEngine({engine: kind}).test/helpers/test-backends.tshasTestBackend = 'pglite' | 'postgres',testBackends(), and envGBRAIN_TEST_BACKEND(valuespostgres,postgres-direct,pgbouncer; anything else throws). The parity file reads neither. Its matrix runs (scripts/e2e-backend-matrix.txt:33,scripts/e2e-matrix.ts:36) only rerun the same file. - Replace
pgEngine/pgliteEnginepairs witharms(as in 2714-2729), or add a third variable to each of the 16beforeAllblocks. - Gate the Rust arm on its own env var, and make the PGLite arm run without
DATABASE_URL. Otherwise the Rust comparison can only run where Postgres exists..github/workflows/e2e.yml:423runs the file astier1 engine-parity. - Replace or skip the EXPLAIN-based block (1557-1605) and the
jsonb_typeofprobes (1744-1752) for Rust. They inspect Postgres catalogs. - Replace the
executeRawseeding with typed calls, or provide a Postgres-SQL subset executor (see section 4, SQL surface).
2. Per-method table: methods covered by engine-parity
Covered means a direct .method( call in test/e2e/engine-parity.test.ts. "Indirect" methods are covered only by helper code (listed in section 3). Line numbers are for the parity file. "Rust must" lists what a passing engine must return or do.
| Method (engine.ts) | Parity tests (describe / it, line) | What is asserted | Rust engine must |
|---|---|---|---|
connect (835) |
every describe; e.g. 126-127, 1368-1369, 2394 | connect({}) must work for PGLite |
accept {} and open an ephemeral brain |
disconnect (836) |
afterAll blocks, e.g. 133, 1375, 2400 | none (cleanup) | disconnect without error, release files |
initSchema (846) |
every describe; 128, 1370, 2395 | none (must succeed) | create schema idempotently; a second call is a no-op |
getPage (869) |
"provenance columns" 688-725 (703-724); "provenance COALESCE" 727-773; "putPage restores soft-deleted" 775-795 (785, 793); "restorePage arc" 2327-2353 (2338-2378); "unscoped getPage tiebreak" 1817-1833 (1819, 1828); "putPage empty-overwrite" 1852-1870; "revival" 909-949 (930-934); "getRawData soft-delete" 2731-2761 | returns null on miss (toBeNull); source_path, source_kind, source_uri, ingested_via exact; ingested_at and deleted_at are Date instances; unscoped read prefers source_id='default', then smallest source_id |
return null (not undefined) on miss; expose the 4 provenance columns plus source_path; ingested_at and deleted_at as Date; ORDER BY (source_id = 'default') DESC, source_id ASC for unscoped reads; includeDeleted option |
putPage (891) |
used in ~68 sites across the file | returns a Page with slug; blank overwrite of non-empty page rejects with /refusing to overwrite non-empty page/ (1860) unless allowEmptyOverwrite: true (1864-1868); provenance COALESCE-preserve on re-write without provenance (749-773, title becomes V2, source_kind stays) |
same blank-overwrite guard and message; COALESCE on provenance columns; restore soft-deleted row under the same slug (revival, 909-949); frontmatter visibility, quarantine, embed_skip, message_id, thread_id, subject carried through (see email citation 440-545) |
deletePages (989) |
"v0.41.19.0 deletePages parity" 849-875 (862-863) | returns only slugs that existed (ghosts excluded); pages are then null via getPage (869-874); result is sorted before compare | hard delete; return confirmed-deleted slugs; ignore unknown slugs |
deletePage (952) |
2759 (raw data test) | none beyond completion | delete one page for {sourceId} |
resolveSlugsByPaths (1007) |
"resolveSlugsByPaths parity" 1008-1031 | Map size 2 for 3 input paths; missing path is undefined (1029-1030) |
return a Map<path, slug>; omit missing keys |
softDeletePage (1017) |
"#3754 ... getLinks/getBacklinks/traversePaths" 797-821 (811); "#3754 traverseGraph" 823-847 (836); "restorePage arc" 2327-2353 (2332-2335); "two-source restore" 2355-2381 (2366-2367); raw data 2739 | returns {slug} first time, null second time (2335); row hidden from default reads; links are not cascaded (they stay in the table but are hidden from reads); deleted_at set |
soft delete: set deleted_at, return {slug} or null; hide from reads and from link/traversal reads |
softDeletePages (1050) |
"#4587 softDeletePages parity" 877-907 (889-893, 905, 922) | returns only rows that transitioned (ghost and already-deleted excluded; the pre-deleted row's deleted_at is not re-stamped, comment 883-884); [] on empty input; rows stay recoverable (includeDeleted: true) |
deleted_at IS NULL predicate; return slugs transitioned; empty input returns [] |
restorePage (1056) |
"restorePage arc" 2327-2353 (2346, 2351); "two-source" 2355-2381 (2368, 2376-2377); raw data 2757 | true exactly once, then false; scoped restore does not touch another source's row |
returns boolean; only restores rows with deleted_at IS NOT NULL in the given source |
listPages (1079) |
"ambient recall keyset" 1668-1768 via drainKeyset (1643-1666, call at 1652) |
keyset cursor {updatedAfterKeyset:{updatedAt, slug}} with sort: 'updated_asc', limit: 3, slugPrefix: 'ks/'; strict greater-than on the (updated_at, slug) tuple (1700-1708); 10-page tie cluster must drain in full with no dupes (1689-1698); batch rows have updated_at as a Date (1662 calls .toISOString()) |
implement the tuple keyset (updated_at > ts OR (updated_at = ts AND slug > s)); return updated_at as Date; sort by updated_at ASC, slug ASC in that mode |
listAllPageRefs (1127) |
"#4304 listAllPageRefs" 313-327 | every row updated_at instanceof Date and finite; source_id::slug list identical in order across engines (324-326) |
return Date for updated_at; deterministic order |
searchKeyword (1198) |
"top-5 slugs" 137-152 (139-150, 3 queries); "exclude_slugs and type" 154-169 (156-168); "orFallback relaxed" 205-233 (214-232); "orFallback parity" 673-681 (676-679); "hard-exclude" 547-591 (578-590); "detail=high" 593-621 (607-620); "searchTitles parity" 630-668 (652-654); "email citation" 440-545 (466, 502); "CJK" 1948-2014 (1988-1997, 2009-2012) | top-1 slug strict equal (146); top-5 set equal (150); exclude_slugs removes slug; type filter returns only that type (167); strict AND query with a missing term returns [] (216, 676); keyword_relaxed === true on every row when orFallback fires (224-245); test/ slugs hidden by default and shown with include_slug_prefixes: ['test/'] (580, 589); CJK order (1990): full ordered slug list equal across engines; sourceIds: ['nonexistent'] returns [] (2011-2012); detail:'high' must keep openclaw/chat/ pages (613-614) and must change ordering on at least one engine (620) |
strict AND by default; OR fallback opt-in with keyword_relaxed on every row; hard-exclude test/ by default; include_slug_prefixes, exclude_slugs, type, sourceIds; CJK branch (ILIKE or equivalent) that gives the same ordering; detail option that changes ranking; frontmatter email fields on results |
searchTitles (1223) |
"searchTitles orFallback" 235-249 (240-248); "exact-title hit" 630-668 (657-667) | title OR-fallback rows all keyword_relaxed === true (244-245); exact title hit found with title tokens absent from body (659); row has chunk_source: 'compiled_truth' and identical chunk_text (665-667) |
title arm returns SearchResult shape with chunk_source and chunk_text from the representative chunk |
searchVector (1224) |
"searchVector: top result" 251-257 (256); "escalation" 347-387 (381-386); "email citation" 440-545 (470-471, 538-539) | top-1 slug equal (256); with 120 near-duplicate chunks on one page and 8 sparse pages, limit: 6 returns exactly 6 distinct pages; top-5 slugs ordered equal (385); result carries message_id/thread_id/source_subject |
take Float32Array of length 1536; per-page DISTINCT collapse; bounded escalation so a dense page cannot starve the result; return score and projected metadata |
upsertChunks (1269) |
"#4587 revival" 909-949 (915-918, 936-938); "chunkless-page" 1177-1220 (1186-1188) | after upsertChunks the default getChunks returns [] (942) because the write is unsealed; getChunks(slug, {includeUnsealed:true}) returns the one chunk (943-945) |
keep unsealed chunks invisible to sealed reads; expose includeUnsealed |
getChunks (1279) |
"#2555 getChunks sourceIds[]" 951-981 (970-979, 976-977); "revival" 942-945; "CJK chunks" 2683 (ids used) | federated sourceIds beats scalar; out-of-grant array returns [] (973-974), never falls through to default; unset opts keep 'default' floor (976-977); embedding is null unless requested (979) |
embedding: null by default; chunk.id numeric (used as from_chunk_id at 2684) |
countChunklessPagesWithContent (1396) |
"chunkless-page" 1177-1220 (1203-1204) | count 2 for the source: cp/stub and cp/skipped (embed_skip with body, #7174); excludes chunked, empty, and quarantined pages |
count pages with non-empty content and zero chunk rows, excluding quarantined and empty |
listChunklessPagesWithContent (1408) |
same block (1206-1217) | ordered ['cp/stub','cp/skipped'] equal (1208-1209); compiled_truth text round-trips (1210-1211); unscoped call with batchSize: 10000 includes cp/stub (1216-1219) |
same filter, same order (insertion or slug order must match Postgres; test pins equality only) |
countStalePagesForExtraction (1438) |
"stale-page extraction" 1103-1170 (1117-1118, 1144-1145, 1154-1158, 1168-1169) | counts: NULL watermark 3; after stamp 2; without versionTs 1; with versionTs: '2026-05-31T00:00:00Z' 2; edited-since arm (updated_at > links_extracted_at) 2 |
three arms: NULL watermark, versionTs, updated-since |
listStalePagesForExtraction (1446) |
same block (1121-1127, 1137-1140) | sorted slugs ['sp/1','sp/2','sp/3']; compiled_truth truthy (1126); updated_at instanceof Date (1127); the row exposes updated_at_iso as a string (1140) |
return both updated_at (Date) and updated_at_iso (string) |
markPagesExtractedBatch (1474) |
same block (1139-1142, 1150) | stamps move the stale count (1144-1145); per-ref extractedAt honoured; defaultExtractedAt used when ref has none (1150) |
writes links_extracted_at per ref with the given timestamp |
addLink (1496) |
many: 805, 831, 919, 939, 989-991, 1053, 1240-1242, 1354-1358, 1410-1411, 1436-1443, 2032-2036 | link visible via getLinks/getBacklinks; cross-source links with fromSourceId/toSourceId (1053); origin options (1442-1443); re-adding after soft delete gives exactly one link (947); missing endpoint: parity test does not check the error |
upsert on identity key (from, to, link_type, link_source, source ids); accept origin and originSourceId fields |
addLinksBatch (1521) |
"removeLinksByPagesAndSource" 1877-1946 (1897) | 5 rows inserted including link_kind: 'typed_ner' (1902-1905) |
batch insert with link_kind, link_source, from_source_id, to_source_id |
removeLinksByPagesAndSource (1564) |
1877-1946 (1926-1944) | removed count 3 (2 plain mentions + 1 stale typed_ner); survivors ordered: `rlps/to-x | markdown |
getLinks (1590) |
federated 1449-1553 (1487-1502, 1504-1515, 1517-1527); soft-delete 797-821 (808); 2032-2036 traverse | exact link shape with from_source_id, from_slug, to_source_id, to_slug, origin_source_id, origin_slug, link_type (1475-1485); out-of-grant origin edge withheld (1498-1501); origin_slug is null-able; far endpoint fed/outside excluded under grant (1492) |
scoped and unscoped reads with identity fields; origin-grant filter (F1); hide soft-deleted endpoints |
getBacklinks (1596) |
soft-delete 797-821 (807, 813-815); federated 1529-1534 (1530-1533); unscoped 1523-1526 | empty after the source is soft-deleted (813-815); from_source_id:from_slug equals ['beta:fed/target'] (1533) |
same as getLinks, in the inbound direction |
listLinkSources (1604) |
"v114 listLinkSources" 983-1006 (996-997, 1003-1005) | rows filtered to two sources equal across engines (1001); citation-graph (count 2) comes before manual (count 1) (1005) |
group by link_source (NULL allowed), ORDER BY count DESC |
traverseGraph (1643) |
"traverseGraph / traversePaths (D7)" 2041-2181 (2070-2116); "#3754 traverseGraph" 823-847 (833, 840-845) | node set and depth identical for depths 1-3 (2070-2072); exact depths at depth 3: a=0, b=1, c=2, x=2, d=3 (2079-2084); cycle guard: depth 25 terminates with 5 nodes and max depth 3 (2089-2097); cross-source node hidden with sourceId (2105) and shown with sourceIds (2112); empty for out-of-scope seed (2115-2116); frontierCap: 10 (843) |
recursive walk with visited set; per-node links array; scope filters on seed, step and links aggregate; frontierCap option |
traversePaths (1658) |
2041-2181 (2122-2177); soft-delete 797-821 (809, 817-819) | ordered edge list from>to:type@depth must match exactly, with ORDER BY depth, from_slug, to_slug (comment 2065); concrete pins at 2128-2141; linkType filter returns one edge (2155); context survives (2157-2158); 'both' emits each edge once (2137-2142) |
BFS with visited set; ORDER BY depth, from, to; the same edge set for in/out/both |
traversePathsDetailed (1672) |
"traversePaths row cap" 2190-2234 (2212-2213, 2227-2228) | truncated: true at depth 3 both-direction from the hub (2214-2215); paths.length <= 5000 (TRAVERSE_PATH_ROW_CAP, src/core/engine-constants.ts:37); all 400 depth-1 edges survive (2219); truncated: false when under cap (2229-2232) |
cap at 5000 with a +1 probe row; shallowest-first before cut |
relationalFanout (1697) |
"relationalFanout" 1361-1407 (1389-1406) | ordered shape equal: `source_id:slug:hop:edge_count:via_link_types:path:canonical_chunk_id(set |
null)(1385);invested_inin-direction returns exactly['people/ep-inv-a','people/ep-inv-b'](1392);mentions` edges excluded (1399) |
findOrphanPages (1830) |
"findOrphanPages source scoping" 1039-1099 (1063); "#4280 projection" 1275-1334 (1296-1307) | mode: 'islanded' (default) vs 'inbound'; scalar and federated scopes; returns {slug, type, quarantined} (1297); quarantined shell has quarantined: true (1306); cross-source inbound counts (A2 comment 1037) |
return type and quarantined on each row; support both modes; scope by source |
addTag (1851) |
"getTags excludePrivate/liveOnly" 189-203 (193-194); "getTags sourceIds[]" 1468-1473 (1434-1435) | tag rows exist for a page; tag on soft-deleted page still stored | tag table with (source_id, slug, tag) identity |
getTags (1859) |
189-203 (198-201); 1468-1473 (1469-1470) | excludePrivate: true removes tags from a visibility: 'private' page (199); liveOnly: true returns [] for a soft-deleted page (201); under federated sourceIds only the beta tag (1472) |
options excludePrivate, liveOnly, sourceIds |
addTimelineEntry (1880) |
"#3754 ..." 797-821 (none direct); "federated" 1428-1447 (1444, 1446); "unscoped timeline plan" 1557-1568 (1565-1566) | two entries added via addTimelineEntry to the same page with different dates |
store date, source, summary, detail |
getTimeline (1898) |
federated 1536-1543 (1537-1538); D5A date window 1545-1552 (1548-1549); unscoped plan 1584-1590 (1586-1589) | federated: sorted summaries ['fed event','late event'] (1540); unscoped: ['beta event','default event'] (1589), i.e. newest first (2026-03-03 before 2026-02-02); after / before windows identical (1546-1551) |
ORDER BY date DESC; cross-source union for unscoped reads; date-window filters |
putRawData (1943) |
2731-2761 (2736) | raw row stored | stores data as JSON per (page, source) |
getRawData (1949) |
2714-2762 (2737-2758) | hidden ([]) for every WHERE shape after soft delete (2742-2749); includeDeleted: true returns it (2753-2755); restore brings it back (2758) |
filter deleted_at IS NULL in all shapes unless includeDeleted |
getDreamVerdict (2078) |
"#4152 dream verdict" 389-438 (412-413, 432) | score exact 0.31 after upsert (417); segments and entities are real arrays, not JSON strings (419-421); model, triage_version = 1 (422-423); legacy row (boolean era) has score, triage_version null, segments and entities [] (433-436) |
store segments/entities as JSON arrays; score as f64; return null for missing score fields (not undefined) |
putDreamVerdict (2079) |
389-438 (408, 410) | upsert on (file_path, content_hash); second put overwrites score (410-417) |
upsert on PK |
insertFact (2215) |
"facts TTL" 2523-2653 (2557-2592) | valid_until in the past hides a fact from active reads; embedding_model round-trips (2583); embedded_text_hash equals md5(fact) (2584); ontology supersession via valid_until + superseded_by (2588-2599) |
compute embedded_text_hash as MD5 hex of fact; accept valid_until and embedding |
listFactsByEntity (2319) |
2523-2653 (2580, 2604, 2611) | activeOnly: true default excludes lapsed and superseded rows (2628); activeOnly: false returns full history (2640-2643); compared as sorted list |
activeOnly option; history includes expired rows |
listFactsSince (2333) |
2605 | entitySlug filter; sorted texts equal (2630) |
filter by created_at >= since |
listFactsBySession (2352) |
2606 | returns the two durable/future facts (2631) | session filter; active-only |
listSupersessions (2370) |
2612 | 'ttl superseded old' present (2645): valid_until-closed row still listed |
include rows with superseded_by set |
countUnconsolidatedFacts (2382) |
2618 | 4 (2651) |
count active rows not consolidated |
findCandidateDuplicates (2393) |
2607-2610 | recency arm (2632) returns active rows; embedding arm ['ttl embed live'] (2634, 2638); keyword arm k:5, arm:'keyword' ranks by shared terms, most first (2636) |
three arms (recency, embedding with embeddingModel, keyword); exclude lapsed rows |
findTrajectory (2422) |
"findTrajectory newest" 171-187 (183-186) | with limit: 3 over 6 points (values 0..5 by date), returns [3,4,5] in ascending date order (185) |
take newest N then return ascending by date |
getFactsHealth (2425) |
2602-2649 (2647-2649) | total_active: 4, total_expired: 3, top_entities sorted ENTITY:3, EMB_ENTITY:1 |
counts with validity rules (lapsed and closed = expired) |
getStats (2488) |
"#4280 ..." 1275-1334 (1328-1333) | raw counts: pages_by_type {person: 4, atom: 1} (1330), page_count 5 (1332); quarantined shell still counted as person |
count all rows, no policy filter |
getHealth (2489) |
"#4280 ..." (1313-1323) | entity_page_count: 3, linkable_page_count: 3, orphan_pages: 3, page_count: 5 (1323); quarantined shell and atom excluded from denominators |
apply quarantine and type policy in health math |
resolveSlugWithAlias (2535) |
"resolveSlugWithAliasDetailed" 2288-2303 (2297-2298) | string equals the detailed canonical (2297-2298) | return canonical slug string |
resolveSlugWithAliasDetailed (2548) |
2258-2286 (2278-2285, 2293-2296) | {canonical_slug, source_id}; winner follows scope order (['par-b','par-a'] gives par-b, 2296); scope with no match returns null (2284-2285); federated ['par-a','par-b'] returns par-b for par/old-b (2283) |
ordered scope resolution with first-match-wins |
executeRaw (2714) |
46 call sites, e.g. 173-175, 195, 270, 427, 953, 1014-1017, 1107, 1151, 1163, 1180, 1232-1239, 1279, 1429, 1562-1567, 1629, 1681, 1744, 1788, 1910, 2024, 2243-2250, 2359, 2414, 2490, 2553, 2596 | seeds and checks using raw SQL: INSERT ... SELECT generate_series, now() - interval '2 hours', ON CONFLICT DO NOTHING/UPDATE, $1::timestamptz, '{}'::jsonb, UPDATE pages SET deleted_at = now(), DELETE FROM facts, ANALYZE (1567), SELECT jsonb_typeof(...) (1744-1749) |
run Postgres-dialect SQL, or provide a typed replacement (see section 4) |
addCodeEdges (2756) |
"getCalleesOf bare-name" 2659-2708 (2684) | edge stored with from_chunk_id = chunk id, to_symbol_qualified, edge_type: 'calls' |
store code edges keyed by chunk id |
getCalleesOf (2783) |
2659-2708 (2699-2705) | exact name returns 0 rows without the option (2699); bareFallback: true returns ValidateRequest for bare SubmitAsync (2700-2701); qualified and delimited names unchanged (2702); wrong qualifier or Submit_sync returns 0 (2703-2704); scope to unknown source returns 0 (2705) |
exact match first; bare-name fallback on symbol_name only when option set and input has no delimiter |
searchKeywordChunks (2804) |
468-469, 536-537, 2001-2008 (2002-2006) | chunk-grain ordered slug#chunk_index equal (2004); top slug notes/tokyo-meeting (2005) |
chunk-level keyword rows with CJK fallback |
listEnrichCandidates (2877) |
"v0.41.39 listEnrichCandidates" 1222-1266 (1246, 1257-1264) | filtered rows slug:inbound_count:body_len equal and ordered (1258); long page excluded by thinThreshold: 400 (1262); order alice(2) > bob(1) > widget(0) by inbound (1263-1264); inbound_count for alice is 2 (1265) |
thin filter on body length; source-aware inbound count (NULL link_source counts); ORDER BY inbound DESC |
Indirect coverage (no direct .method( in the parity file, but reached through helpers used in it):
readPageSnapshot,lockPageKeys,transaction: viainstallFixtureChunks(test/helpers/page-projection.ts, 17 call sites) andreadProjectionSnapshot/sealPageTextProjection(src/core/page-state/projections.ts:39,:159).relationalChainHop: viarunRelationalChainin the chain test (1409-1421; helpersrc/core/search/relational-chain.ts).getBacklinks,getLinks,getTimeline,listFactsByEntity,resolveAliases,searchKeyword: viabuildEntityCard(259-311;src/core/verbs/entity-card.ts).getConfig,getLinks,getBacklinks,transaction: vialinkEntityIdentity/listEntityIdentities(329-345;src/core/entity-identity.ts).executeRaw: viaupsertSessionContextState/getSessionContextState(1711-1767;src/core/context/session-state.ts) andupsertOpenLoop/closeOpenLoop/listOpenLoops(2383-2521;src/core/loops/loops-store.ts).engine.kindandexecuteRaw("EXPLAIN (FORMAT JSON) ..."): viatimelinePageScans(test/helpers/timeline-plan.ts, used at 1595).
3. Engine methods with NO parity coverage
Total BrainEngine members in src/core/engine.ts:830-2890: 197. Covered directly: 60 (section 2). Not covered by the parity file: 137. The list below is grouped; test files are a sample from rg -l '\.<name>\b' test and are not exhaustive. Files marked "helpers only" are referenced only from test/helpers/*.ts case tables (for example postgres-engine-sql-cases.ts, engine-sql-rollback-cases.ts).
- Lifecycle / transaction:
reconnect(pglite-reconnect.serial.test.ts,supervisor-health-reconnect.test.ts),holdSchemaMigrationLock(no.holdSchemaMigrationLock(caller found in test/),transaction(trust-channel-writes.test.ts,client-grants.test.ts),transactionDirect(autopilot-transaction-fence.test.ts,engine-control-routing.test.ts),registerBeforeDisconnect(persistence-reconnect.serial.test.ts),withReservedConnection(migrate.test.ts,pool-gauge.test.ts),runMigration(v273-validate-timeouts.test.ts). - Pages:
readPageSnapshot(fact-withdrawal-normalized.test.ts, indirect here),readPageSnapshotsBatch(postgres-rls-scope-inventory.test.ts),lockPageKeys(belief-ledger-purge.test.ts; indirect here),findDuplicatePage(import-dedup-frontmatter-id.test.ts),findDuplicatePages(helpers only),purgeDeletedPages(pages-soft-delete.test.ts,purge-deleted-dryrun-postgres.test.ts),findPages(helpers only),countPagesBy(list-pages-empty-type-notice-gbra449.test.ts),resolveSlugs(pglite-engine.test.ts),getAllSlugs(integrity-batch.test.ts),listAllSources/updateSourceConfig(list-all-sources-postgres.test.ts),listPrefixSampledPages/listCorpusSample(engine-surface-coverage.test.ts,page-retrievals.test.ts),updateSlug(origin-guard-6948.test.ts),rewriteLinks(engine-surface-coverage.test.ts),resolveAliases(alias-hop.test.ts; indirect here),setPageAliases(by-mention.test.ts),refreshPageBody(fence-chunk-overlay.test.ts),updatePageContextualRetrievalState(import-revision-concurrency.test.ts),migrateFactsToCanonical(phantom-redirect.test.ts),getPageTimestamps(engine-surface-coverage.test.ts),getEffectiveDates(read-enrichment-privacy.test.ts). - Search / retrieval:
countKeywordPages(search-keyword-paging-parity.test.ts),searchKeywordPages(keyword-arm-access.test.ts),getEmbeddingsByChunkIds(search-quality.test.ts),getChunkPage(get-chunks-paging.test.ts),getChunkWindows(evidence-delivery-parity.test.ts),keywordBoundedRead(search-read-keyword-bound.test.ts),getChunksWithEmbeddings(pglite-engine.test.ts). - Embedding state:
countStaleChunks(embedding-state.test.ts),sumStaleChunkChars(embedding-signature-stale.test.ts),setPageEmbeddingSignature(embedding-restamp-w3.test.ts),invalidateStaleSignatureEmbeddings(embedding-signature-stale.test.ts),invalidateContentDriftEmbeddings(content-drift-sweep-postgres.test.ts),listStaleChunks(embed-stale-pagination.test.ts),readEmbeddingStateCounts(helpers only),listEmbeddingStateRows(embedding-state.test.ts),deleteChunks(delete-chunks-lock-order-8502.test.ts),markPagesAttendanceBlocked(helpers only),countStaleFrontmatterKeyPages(repair-frontmatter-keys-sweep-6454.test.ts),listStaleFrontmatterKeyPages/stripStaleFrontmatterKeys(helpers only). - Links / graph:
replaceDerivedLinks(link-cross-source.test.ts),replaceDerivedLinksBatch(extract-endpoint-revision.serial.test.ts),removeLink(pglite-engine.test.ts),findByTitleFuzzy(title-fuzzy-index.test.ts),getBacklinkCounts(read-enrichment-privacy.test.ts,jit-off-reads-postgres.test.ts),getAdjacencyBoosts(graph-signals-engine.test.ts),getContentFlagsByPageIds(quarantine-cli.test.ts),getUnverifiedExtractionPageIds(extraction-review.test.ts),getSalienceScores(engine-parity-salience.test.ts),relationalChainHop(relational-chain.test.ts; indirect here). - Tags / timeline / chronicle:
removeTag(pglite-engine.test.ts),addTimelineEntriesBatch(timeline-date-batch.test.ts),getTimelineForDate(chronicle-event-projection-parity.test.ts),getSince(chronicle-timeline-reads.test.ts),getOnThisDay(chronicle-delight.test.ts),getLastSeen(chronicle-last-seen-postgres.test.ts),upsertEventProjection(chronicle-event-projection-parity.test.ts),mergeOntologyFact,getOntology,discoverOntologyDimensions,findOntologyConflicts(ontology-merge-parity.test.ts,chronicle-ontology.test.ts). - Files:
upsertFile,listFilesForPage(engine-upsertFile.test.ts),getFile(import-image-file.test.ts). - Takes:
addTakesBatch(takes-engine.test.ts),updateTakeEmbeddings,countStaleTakes,listStaleTakes(takes-vector-lifecycle-5885.serial.test.ts),listTakes(takes-source-scope.test.ts),searchTakes,searchTakesVector(takes-postgres.test.ts),getTakeEmbeddings(takes-engine.test.ts),updateTake,supersedeTake,addSynthesisEvidence(takes-engine.test.ts),resolveTake(take-resolution-survives-publish.test.ts),getScorecard,getCalibrationCurve(takes-scorecard-parity.test.ts). - Dream / contradictions:
sweepDreamVerdicts(dream-verdict-cache-ttl.test.ts),listActiveTakesForPages,writeContradictionsRun,loadContradictionsTrend,getContradictionCacheEntry,putContradictionCacheEntry,sweepContradictionCache(eval-contradictions-engine.test.ts,eval-contradictions-postgres.test.ts). - Facts:
insertFacts(fence-chunk-overlay.test.ts,extract-facts-reconcile-drain.test.ts),deleteFactsForPage(facts-fence-reconcile-postgres.test.ts),expireFact,consolidateFact(facts-engine.test.ts),listFactsKeyset(engine-sql-facts-keyset.test.ts,turn-context.test.ts). - Segments / body generations:
allocateBodyOccurrenceIds,stageSegmentTexts,readSegmentTexts,stageSegmentChunks,createSegmentedPageShell,flipBodyGeneration,openBodyStaging,setBodyStagingState(engine-sql-page-body.test.ts),stageManifestRows(segmented-storage-visibility.test.ts),readManifest(edit-segmented.test.ts),sweepSegments(lp-h-purge-orphan-segment-text.test.ts),readSegmentDerivations(segmented-conversation-facts.test.ts),upsertSegmentDerivations,retireSegmentDerivations(helpers only:segment-derivation-cases.ts). - Versions / ingest:
createVersion(generation-identity-visibility-9386.test.ts),getVersions(persistence-version-deletion.test.ts),revertToVersion(segmented-versions.test.ts),logIngest(import-cancellation.serial.test.ts),getIngestLog(sync.test.ts). - Config:
getConfig(conversation-facts-managed.test.ts; indirect here),setConfig(trust-release-strict-home-gst2676.test.ts),unsetConfig(doctor-fence-integrity.test.ts),listConfigKeys(config-db-merge.test.ts),getAllConfig(config-request-snapshot.test.ts). - Code intelligence:
deleteCodeEdgesForChunks(code-edges.test.ts),getCallersOf(cli-source-scoping-pglite.test.ts),getEdgesByChunk(code-edges.test.ts),executeRawDirect(persistence-diagnostics.test.ts). - Eval capture:
logEvalCandidate,listEvalCandidates,deleteEvalCandidatesBefore,logEvalCaptureFailure,listEvalCaptureFailures(eval-candidates.test.ts,eval-capture.test.ts). - Salience / anomalies:
batchLoadEmotionalInputs(salience-pglite.test.ts),setEmotionalWeightBatch,getRecentSalience,findAnomalies(salience-anomalies-source-isolation-pglite.test.ts;engine-parity-salience.test.tscovers them cross-engine but is DATABASE_URL-gated, like this file).
Note for the Rust plan: many uncovered methods (for example getRecentSalience, findAnomalies, getContradictions*, takes) are covered only by PGLite unit tests, so parity for them is not checked anywhere. The Rust engine will need its own parity cases if they matter.
4. Cross-cutting requirements a Rust engine must meet
Error classes and messages.
- The parity file asserts one error message:
refusing to overwrite non-empty page(/refusing to overwrite non-empty page/, line 1860). The Rust engine must throw an error whose message contains that text, fromputPage. PageMissingError(src/core/engine-errors.ts:18-35, messagesaddLink failed: <endpoint> page "<slug>" (source=<id>) not foundandaddTimelineEntry failed: page ...) is not asserted in this file. Other tests check it, so the Rust engine still needs it.- Rejections are expected for blank overwrites only. Other operations return
null,[],falseor0for misses.
Null vs undefined vs missing.
toBeUndefined()for absent email fields:result?.message_id,thread_id,source_subjectmust beundefined(505-507, 541-543). Returningnullfails the test. Omit these keys when not applicable.thread_idis set when frontmatter has it, even for non-email pages (506).toBeNull()for absent dream fields:scoreandtriage_versionmust benull, whilesegmentsandentitiesmust be[](433-436).getPagemiss:toBeNull()(785, 899, 1862 etc.).getChunksembedding:toBeNull()when not requested (979).deleted_at:nullor Date; the test usesdeleted_at ?? null(934) so eithernullorundefinedfor live rows would pass there, butgetPage(... includeDeleted)requires a Date (2343).resolveSlugsByPathsmiss:Map.get()returnsundefined(1029-1030), notnull.resolveSlugWithAliasDetailedno match:toBeNull()(2284-2285).restorePage/softDeletePage: booleans and{slug}ornull(2335).
Timestamps.
Dateobjects (must beinstanceof Date):listAllPageRefs.updated_at(319-320),listStalePagesForExtraction.updated_at(1127),getPage.ingested_at(723-724),getPage.deleted_at(2343),listPagesrowsupdated_at(1662 calls.toISOString()on them).- ISO strings:
EntityCard.last_touched.last_timeline_date(298, 304) andopen_threads[].datemust bestring | null(299-301).listStalePagesForExtraction.updated_at_isois a string (1140).session_context_state.last_wake_atmust match/\.\d{6}Z$/(1735) and equal the input instant2026-08-05T12:00:00.000Zafternew Date()(1734), so microsecond precision is required there. Loops:due_atequals'2026-09-01T23:59:59.000Z'(2477), i.e. millisecond precision;opened_atis a string (2455). Parity compares strings across engines exactly (1736), so both engines must format identically. - Timeline
dateis a string (src/core/types.ts:1742).created_atis a Date (types.ts:1746). - Keyset cursor compares
updated_atas a timestamp plus slug. The keyset fixtures use whole-millisecond values (1616-1618), whilesession_context_state.last_wake_atis pinned at microsecond precision (1730-1735).
IDs.
- Numeric.
chunk.idis passed back asfrom_chunk_id(2683-2684), so it must round-trip as a number. loopId: 2 ** 40(2511) must work without error and return zero rows (beyond int4). Use 64-bit ids foropen_loopsand any table with a serial id in this file.canonical_chunk_idis compared only for presence (1380-1385), because its absolute value depends on sequence state.page_idis normalized to 0 in the chain test (1416), so exact values are not compared.slug_aliases,pages.id,facts.idare used as opaque numbers (newRow.id, 2592-2599).
JSON shapes.
dream_verdicts.segmentsandentitiesmust be JSON arrays, not JSON strings (419-421). The Postgres path had a double-encode regression (#2339). A Rust engine must store native JSON arrays and return them as arrays.session_context_state.standing_entitiesandsurfaced_slugsmust be JSON arrays in storage:jsonb_typeof = 'array'(1744-1752). A Rust engine cannot run that SQL probe; it needs an equivalent typed check.loops.evidencemust come back as an array withmessage_idandquote(2448-2452).pages.frontmatterkeys drive search result metadata (message_id,thread_id,subjectgated by an email identity, 440-545).subjectalone must not setsource_subject(499-508).- Tag/link/timeline keys,
contexttext, andfacttext must round-trip byte-for-byte (for example'a verbatim line with "quotes" and unicode', line 400).
Source scoping.
- Scalar
sourceId, federatedsourceIds[], and unset (default floor'default') must behave as in 951-981 and 2104-2116. - Out-of-grant
sourceIdsreturns empty and never falls back to'default'(973-974). - Unscoped link reads must expose exact
from_source_id/to_source_idfor every row (1521-1526). - Origin filter (F1): an edge whose
origin_source_idis outside the grant is hidden undersourceIds(1442-1443, 1498-1501). - Unscoped
getPageandgetTimelinepick the default source first (1819-1830, 1589).
Soft delete and projection visibility.
- Soft delete sets
deleted_atand does not cascade (889-907). Links and tags stay in storage but are hidden from reads (797-821, 823-847). - Re-adding a soft-deleted slug revives the same row (909-949):
deleted_atclears, content updates, chunk and link sets are replaced, not duplicated (942-948). - Unsealed chunks are hidden from default
getChunksuntil sealed (942-945).
Vector search.
- Input:
Float32Arrayof length 1536 (basisEmbedding, line 35). Stored vectors must accept 1536 dims (and legacy 1536 viasetupLegacyEmbeddingDB). - Test vectors are one-hot basis vectors, so many scores are exactly
0or1and ties are common. The test asserts only top-1 (256), aSetof 6 pages (386), and the ordered top-5 on the escalation fixture, whose scores are distinct:0.99 - i*0.0005(367) and0.6 - p*0.001(375). A tie-break rule is therefore not pinned by the parity file. - Float tolerance: none asserted. Dream score is compared exactly (
0.31, 417), so store scores as f64. - Escalation (347-387): a dense page with 120 chunks must not starve the result. Expect 6 distinct pages with
limit: 6. The pre-fix engine returned 1 page. The inner candidate pool is 100 chunks (comment 349-351).
Keyword ranking.
- Strict AND by default. OR fallback only when
orFallback: true, and then every row carrieskeyword_relaxed: true(205-233, 673-681). - Top-1 slug must equal across engines for the three QUERIES (139-146). Top-5 must be the same set (150). Ordering beyond top-1 is not required, except where the test compares ordered arrays (CJK 1990, 1997; typed 168).
detail: 'high'must change ordering on at least one engine (593-621), and must still include chat pages (612-614).- CJK queries (
東京 会議,미팅 김대리,予算) must return the matching page as top-1 with identical ordered slugs on both engines (1987-2006). The corpus includes an ASCII decoy page (1955), but the assertions check only CJK top-1 and ordered equality, not the decoy's absence. - Title arm: exact title tokens must hit a page whose body lacks them (630-668).
Traversal and caps.
traversePathsorder isdepth, from_slug, to_slug(2065).traverseGraphnode order is not compared (sorted in tests).- Cycle guard: A to B to A must terminate at depth 25 with 5 nodes and max depth 3 (2089-2097). Each edge appears once in
'both'(2137-2142). - Row cap:
TRAVERSE_PATH_ROW_CAP = 5000(src/core/engine-constants.ts:37), one probe row beyond it, andtruncatedflag (2212-2232). Shallowest edges survive the cut (2218-2219).
Time-based fixtures.
- Facts TTL uses
Date.now()-relativevalid_until(2551-2568). The Rust engine must compare against its own clock consistently withnow()used in raw SQL seeds (for example 1164, 1629, 2596). Clock skew between client and DB broke a test once (1131-1135), so pick one clock source. - Entity card counts
new Date().toISOString().slice(0,10)as a timeline date (289).
Chronicle, relational, and other helpers. buildEntityCard (src/core/verbs/entity-card.ts) expects exact active_fact_count 105 for remote and 106 for local (297, 309-310). This needs listFactsByEntity (world-only for remote), getTimeline, getLinks, getBacklinks, resolveAliases, and searchKeyword.
Parity risks noted
- Harness is Postgres-bound.
test/e2e/helpers.ts(setupDB,setupLegacyEmbeddingDB,teardownDB) usessrc/core/db.tsandPostgresEnginedirectly, and TRUNCATEs tables. A Rust engine needs its own setup helper; there is no engine-level switch to reuse. - Gating hides PGLite. Everything except the
getRawDatablock is insidedescribeBoth. WithoutDATABASE_URL, the file runs 1 test. The header comment (lines 12-14) anddocs/TESTING.md:509("each E2E file keeps a PGLite arm in the unit lane") both say otherwise. A Rust arm will be hidden the same way unless the gating changes. - Raw SQL surface. 46
executeRawsites plusinstallFixtureChunksrely on Postgres SQL (now(),interval,generate_series,ON CONFLICT,::jsonb,::timestamptz,ANALYZE) and on the projection tables (page_projection_jobs,text_projection_revision,knowledge_revision,search_vector). An embedded Rust engine that does not run this SQL cannot pass without test rewrites. - Postgres-only introspection.
test/helpers/timeline-plan.tsrunsEXPLAIN (FORMAT JSON)and assertsIndex Name/Index Cond(1592-1604). Replace it with a Rust query-plan assertion or drop it for Rust. - Catalog probe.
jsonb_typeofchecks (1744-1752) inspect the stored type in Postgres. Rust needs an equivalent typed check or to rely on the read-back shape. - Null vs undefined.
toBeUndefined(505-507, 541-543) vstoBeNull(433-435) on different fields. A generic "null for missing" serializer fails one set of tests. - Weak assertions hide drift. No score comparison;
detail=highonly asserts "differs on at least one engine" (620);listPages/listAllPageRefscompare parity but not absolute order;getTimelinefederated results are sorted. A Rust engine can pass these tests while ordering differs. Add stronger Rust-only checks where ordering matters. - Fixture protocol coupling.
installFixtureChunks(17 uses) depends on the projection revision protocol insrc/core/page-state/projections.ts. Any Rust storage model must implement that protocol, or the fixture must be rewritten to call a public seal API. - Unsealed chunk visibility.
upsertChunkswrites are invisible to defaultgetChunksuntil sealed (942-945). A Rust engine that exposes chunks immediately fails 942. - Uncovered surface. 137 of 197 interface members have no parity test (section 3). Many are covered only by PGLite-only tests (salience, takes, contradictions, ontology, segments).
- Clock coupling. Facts TTL and stale-extraction tests mix
Date.now()(client) andnow()(server) clocks (1131-1135, 2551-2568, 2596). A Rust engine with one clock may disagree with the test's expectations at boundaries. - Engine-kind plumbing is closed.
BrainEngine.kind,EngineConfig.engine, andconstructEngineaccept onlypostgresandpglite(src/core/engine.ts:832,src/core/types.ts:2343,src/core/engine-construct.ts:9-22). A Rust kind requires changes in all three. - Matrix runs do not test a third engine.
scripts/e2e-backend-matrix.txt:33and.github/workflows/e2e.yml:423only rerun this file against direct Postgres and PgBouncer.GBRAIN_TEST_BACKENDis not read by this file, so the matrix never exercises a new backend unless it is wired intotestBackends()and this file.
Adding a third engine and binding options
Base path for all repo citations: the gbrain repo root (commit 8bbfd0b5e8, read-only). Paths below are relative to it.
Engine selection today
How an engine is chosen. GBrainConfig.engine is the two-value union 'postgres' | 'pglite' (src/core/config.ts:37). It is loaded from ~/.gbrain/config.json (config.ts:196; the directory honors GBRAIN_HOME, config.ts:1977). loadConfig infers the engine: a DATABASE_URL env var forces postgres; otherwise the file's engine wins, and a file with only database_path is pglite (config.ts:738-746). toEngineConfig copies engine, database_url, database_path into the runtime EngineConfig (config.ts:1968-1975).
The discriminator. EngineConfig.engine?: 'postgres' | 'pglite' is at src/core/types.ts:2343. BrainEngine.kind: 'postgres' | 'pglite' is at src/core/engine.ts:832. DegradedEngineOptions.kind is at src/core/degraded-engine.ts:63. EngineSettings.engine (embedding host) is at src/core/embedded-engine.ts:14. EngineStatusReport.effective_engine and config_file_engine are at src/commands/engine-status.ts:108.
Construction chain.
| File | Role | Lines |
|---|---|---|
src/core/engine-construct.ts |
constructEngine switch: case 'pglite' dynamic-imports pglite-engine.ts, case 'postgres' imports postgres-engine.ts; default throws Unknown engine type ... Supported engines: postgres, pglite |
12-23 |
src/core/engine-factory.ts |
createEngine = constructEngine + chat-usage sink registration + wraps disconnect |
whole file (31 lines) |
src/core/engine-connect.ts |
connectEngineForServe: loadConfig, createEngine(toEngineConfig(config)), then connectWithRetry; PGLite one-shot reads set pglite_shared_buffers, Postgres one-shot sets pool_max: 2 |
76-110 (one-shot constants 34-42) |
src/core/engine-open.ts |
Experimental openEngine(settings) for embedding hosts. checkSettings refuses any engine other than the two (:94); SETTINGS_FIELDS names database: ['url','directUrl','path']; constructEngine at :166; PGLite only accepts database.path, Postgres only database.url (:92-106) |
94-106 |
src/core/embedded-engine.ts |
Settings types and markEmbeddedEngine registry; engine-kind-agnostic except the engine union |
11-20 |
src/core/degraded-engine.ts |
Lazy-reconnect proxy for gbrain serve when Postgres is down at startup. Scope is Postgres only (header comment, lines 17-24); kind is an explicit getter (:234-235) that defaults to postgres while dead |
17-24, 63, 116-240 |
src/core/engine-key.ts, src/core/engine-root.ts |
engineKey and ENGINE_ROOT symbol; kind-agnostic (no change needed) |
whole files |
Config and CLI surfaces.
gbrain init --pglitesetsisPGLite(src/commands/init.ts:56). Bareinitwith an existing Postgres config is preserved unless--pgliteis passed (init.ts:170-181). PGLite init createscreateEngine({engine:'pglite'})atinit.ts:1144. Init hints and error text namegbrain init --pglite --no-embedding(src/core/engine-connect.ts:59-64,init.ts:598-609).--pgliteappears 54 times in 18 files, includingsrc/commands/init.ts,setup.ts,bootstrap.ts,reinit-pglite.ts,doctor/checks/db-connection.ts,doctor/checks/pglite-worker.ts,src/mcp/status-mode.ts,src/cli/top-help.ts,src/core/error-registry.ts,src/core/cli-flag-registry.generated.ts(generated; regenerate, not hand-edit). Exact generator not verified.gbrain engine status [--json] [--probe] [--brain <id>](src/commands/engine-status.ts:1-80) is engine-free. It dispatches beforeconnectEngine. PGLite lock state is reported viainspectLockHolder(pglite_lock). It has no generic lock or probe path for a non-SQL engine.gbrain reinit-pgliterefuses unlesscfg.engine === 'pglite'(src/commands/reinit-pglite.ts:42).gbrain migrate --to postgres|pgliteand graduation:src/commands/migrate-engine.ts,migrate-graduation.ts.
Size of the branch surface. Counts are regex-based, so treat them as approximate.
| Pattern | Count | Where |
|---|---|---|
'pglite' string literal (all of src/) |
~353 occurrences in ~186 files | broad |
'postgres' string literal (all of src/) |
~339 occurrences in ~171 files | broad |
| Strict `kind ===/!== 'pglite' | 'postgres'` | ~268 sites |
sqlFor per-engine SQL |
42 uses in 34 files | schema-migrations/types.ts:29 (sqlFor?: { postgres?: string; pglite?: string }); picked at src/core/migrate.ts:548 (m.sqlFor?.[engine.kind] ?? m.sql) |
| Transaction branch on kind | 1 | src/core/migrate.ts:221 (`useTransaction |
| Doctor checks mentioning PGLite | 37 of 207 check files | src/commands/doctor/checks/ (e.g. pglite-worker.ts) |
Main categories of branch:
- Migrations.
sqlFordialect picks,pglite-only repair and lock migrations. A non-SQL engine can use the engine-agnosticsqlfallback only if its migrations are dialect-free, which current raw-SQL migrations are not. - Persistence and locking. The PGLite single-writer lock (
src/core/pglite-lock.ts,PgliteBusyErrorat lines 19-25) and the persistence owner socket (src/core/persistence/local-client.ts, "Engine-free CLI routing to the process that already owns a PGLite brain"). The owner socket is the existing single-writer routing for CLI, MCP and cron. - Doctor and repair.
commands/doctor/checks/*,pglite-repair.ts,pglite-resetwal.ts,pglite-leftovers-check.ts,db-repair.ts. - Ops and searches. Postgres-only features (pg_trgm, statement timeouts, pool) are gated on
kind === 'postgres'. Engine-sql domain modules (src/core/engine-sql/) have two dialect adapters,dialect-pglite.tsanddialect-postgres.ts.
Docs. docs/ENGINES.md says "Two engines ship today" (line 7). The "Adding a new engine" guide is at docs/ENGINES.md:1566-1596. It says a non-SQL engine "implements every method itself" (line 1594 onward). Its step 4 says to store {"engine":"myengine"} in config. That is not accurate today: GBrainConfig.engine and EngineConfig.engine are closed unions (config.ts:37, types.ts:2343), and constructEngine throws on unknown values (engine-construct.ts:22). Adding a third kind needs type changes too. The capability matrix at ENGINES.md:1598-1611 covers only the two engines.
What adding a third engine kind would touch
Minimum set (non-SQL Rust store, kind 'rust' as a placeholder name):
| # | Change | File(s) |
|---|---|---|
| 1 | Widen BrainEngine.kind union |
src/core/engine.ts:832 |
| 2 | Widen EngineConfig.engine and GBrainConfig.engine unions; decide config key for a storage path (e.g. database_path reuse vs a new key) |
src/core/types.ts:2343, src/core/config.ts:37, config.ts:738-751 (inference), config.ts:1968-1975 (toEngineConfig) |
| 3 | New case in constructEngine (dynamic import of the Rust binding module) |
src/core/engine-construct.ts:12-23 |
| 4 | Engine class implementing all of BrainEngine (the interface is roughly 100+ methods; see engine.ts:830-2890). Non-SQL engines implement each method directly; SQL-dialect adapters are optional |
new src/core/<name>-engine.ts |
| 5 | Widen EngineSettings.engine, checkSettings engine guard, and the database shape rules |
src/core/embedded-engine.ts:14, src/core/engine-open.ts:94-106 |
| 6 | Widen DegradedEngineOptions.kind if the degraded path should ever cover it; today it is Postgres-only |
src/core/degraded-engine.ts:17-24, 63 |
| 7 | Widen EngineStatusReport engine unions; add lock or probe fields for the new engine |
src/commands/engine-status.ts:108-110 (and the mount?.engine mapping at :268) |
| 8 | init flag (e.g. --rust) and isPGLite-style branches; init.ts:56, init.ts:170-181, init.ts:1144 |
src/commands/init.ts, src/core/cli-flag-registry.generated.ts |
| 9 | Migrations: every sqlFor map has only `postgres |
pglite keys (schema-migrations/types.ts:29); add a third key or require an engine-agnostic sql/handler. Check each of the ~29 strict kind branches in schema-migrations/` |
| 10 | Persistence: owner socket, graduation (persistence/engine-graduation.ts, graduation-custody.ts) and pglite-lock-based single-writer logic are PGLite-specific. A third engine needs its own answer to "who is the writer" |
src/core/persistence/*, src/core/pglite-lock.ts, src/core/persistence/native-lock.ts |
| 11 | Doctor checks: commands/doctor/checks/* (22 files under commands/doctor/ branch on kind; 37 check files mention PGLite) |
src/commands/doctor/checks/ |
| 12 | Tests: test/e2e/engine-parity.test.ts, test/pglite-engine.test.ts, test/engine-factory.test.ts, test/engine-open*.test.ts; add the third engine to the parity matrix |
test/ |
| 13 | Docs: docs/ENGINES.md (sections "Adding a new engine" and the capability matrix), docs/ENGINES.md:1-40 intro |
docs/ENGINES.md |
Unchanged (kind-agnostic): src/cli.ts, src/mcp/server.ts, chunkers, embedding.ts, hybrid search over SearchResult[], and most of src/core/engine-sql/<domain>.ts (per ENGINES.md:1594-1596).
Native module precedent: native/locks/
Contents. native/locks/locks.c is a C Node-API v3 addon (NAPI_MODULE_INIT at line 301, napi_define_properties at line 333) exposing openLock, tryLock, close, plus Windows IPC, plus export publication (export-publication.h). POSIX uses flock; Windows uses LockFileEx.
native/locks/README.md: design and CI notes. Says all eight addons are checked in so source installs work withbun install --frozen-lockfile --ignore-scripts.native/locks/manifest.json:version 1,napi 3,zig 0.14.1,input_sha256, and per-artifactsha256andbytesfor eight targets (linux-x64-glibc,linux-arm64-glibc,linux-x64-musl,linux-arm64-musl,darwin-x64,darwin-arm64,win32-x64,win32-arm64).native/locks/package.json: private package boundary, not a package. Its README note says this prevents plugin hosts from treating the whole checkout as one native package (#6026).native/locks/prebuilds/*.node: exactly the eight binaries, checked into git.native/locks/vendor/node-v22.15.0/:node_api.h,node_api_types.h,js_native_api.h,js_native_api_types.h,LICENSE, vendored from Node v22.15.0.native/locks/abi-check.c: compile-time layout checks (_Static_assertfor Darwinstat, open flags, lock constants). Compiled only on macOS against the real SDK innative-locks.yml.native/locks/darwin-abi.h,windows-napi.h(resolves used Node-API symbols from the running executable, so it does not load a separatenode.exe),windows-ipc.h,export-publication.h.
Build. scripts/native/build.ts compiles with pinned Zig 0.14.1 via zig cc -target <triple> -shared -fPIC -O2 -s -DNAPI_VERSION=3. Targets are glibc 2.17 baseline, musl, x86_64-macos.13.0, and x86_64-windows-gnu. It sets SOURCE_DATE_EPOCH and -Wl,--build-id=none for reproducible bytes (build.ts:39-50). scripts/native/toolchain.json pins archive URLs and SHA-256 (zig for x86_64/aarch64 linux, macOS, Windows). scripts/native/verify.ts recomputes input_sha256 and compares artifact bytes against a rebuild; --write-manifest regenerates the manifest. The README says the rebuild must be byte-identical across CI runners.
Load from Bun. src/core/persistence/native-lock.ts:
loadBindingSync()(lines 94-140) picks a literalrequire('../../../native/locks/prebuilds/<target>.node')byprocess.platform/process.arch/detect-libc (familySync,GLIBC,MUSL). Literal requires are deliberate: "Bun embeds each .node asset in its compiled executables. Dynamic prebuild discovery does not embed them." (comment at lines 105-107).- It validates
binding.target === targetand that required functions exist, then caches. Errors becomeNativeLockUnavailableErrorwith codewriter_lock_unavailable(lines 27-33). - Lazy: loaded only by callers that need a lock.
nativeLockCapability()andnativeExportPublisher()expose capability checks. - No
process.dlopen, nocreateRequirefor.node, no loader module beyond this one file.src/core/cli-socket/peer-cred.ts:20,:38andsrc/core/pid1-reaper.ts:91-94usebun:ffidlopenagainst system libs (libSystem.B.dylib, libc), not gbrain-owned addons.
CI.
.github/workflows/native-locks.yml(reusable):nativejob matrix overlinux-x64-glibc,linux-arm64-glibc,darwin-arm64,darwin-x64,win32-x64,win32-arm64on Bun 1.4.4. Each cell runsscripts/native/build.ts --target <t>,verify.ts --rebuilt, the native lock tests,compiled-smoke.ts, and the data-safety suites. A separatemusljob (line 336) runslinux-x64-muslandlinux-arm64-muslin musl userspace. Scope is narrowed on PRs (scope=smoke|primary|full)..github/workflows/test.yml,heavy-tests.yml,persistence-validation.ymlalso reference native paths..github/workflows/release.yml:121-131builds release binaries for only two targets:bun-darwin-arm64(macos-26) andbun-linux-x64. Line 170 runsbun build --compile --no-compile-autoload-bunfig ... --target=<t> --outfile bin/<artifact> src/cli.ts src/core/skills/resolver/. Line 175 runsbun scripts/native/verify.ts && bun scripts/native/compiled-smoke.ts --binary bin/<artifact>, which asserts the executable embeds the matching addon.- Tests:
test/native-lock.test.ts,test/scripts/native-lock-prebuilds.test.ts(tamper and matrix-coverage checks),test/native-locks-package-boundary.test.ts.
Compiled binaries. bun build --compile embeds .node files reached by a literal require. Computed paths are not embedded. The compiled binary does not use the repo's import ... with { type: 'file' } pattern for addons; that pattern is used for PGLite WASM and tree-sitter WASM (src/core/pglite-embedded-assets.ts:1-40, src/core/pglite-embedded-asset-paths.ts). Verified in source only; the packaging assertion is in release.yml:175.
Binding options
Current Bun facts used below (sources at the end):
- Bun supports Node-API and
require()/process.dlopen()of.nodefiles. Compatibility is "best effort" for the exact addon, Bun version, OS and arch. Bun 1.4 aligned N-API status and error behavior with Node 26 and addednapi_create_async_workand threadsafe functions with teardown fixes. - napi-rs treats Bun as best effort; its Bun test job is non-blocking upstream, so each addon must be tested on Bun before support is claimed.
bun:ffiis still experimental; Bun's docs say not to rely on it in production.dlopen,JSCallbackare engine-native in current Bun.threadsafe: truecallbacks marshal to the JS thread with a per-call cost.node:wasiin Bun is partial:wasiImportandstart()work,getImportObject()andsock_acceptare missing, and WASI preview1 has nommap.
Option 1: napi-rs native addon (in-process)
| Aspect | Assessment |
|---|---|
| Bun fit | Good for the .node precedent already in the repo. Bun 1.4.4 is the repo floor (package.json:261-262, >=1.4.4). Bun 1.4 covers async work and threadsafe functions. napi-rs says Bun is best effort, so the repo must own a Bun test lane. |
| Async and threads | napi_create_async_work and threadsafe functions are supported. Use them for blocking Rust I/O so the JS thread is not blocked. Teardown rules still apply: join native threads before the environment is torn down. |
| Buffers and embeddings | Input Buffer/TypedArray can be read in place with napi_get_buffer_info (already used in export-publication.h:509). Copy-in is safe. Zero-copy out to JS (external ArrayBuffer with a finalizer) is a napi feature whose behavior under Bun must be verified before depending on it; not verified here. |
| Prebuild matrix | Same 8 targets as native/locks. napi-rs cross-build: musl via cargo-zigbuild (adds -crt-static off); Windows MSVC via cargo-xwin (GNU Windows not supported in that mode). The repo's zig-cc pipeline does not build Rust. Use napi build -x or native CI runners per target, and keep the byte-compare or rebuild-verify step if the repo wants the verify.ts guarantee. Windows arm64 and musl are the least-tested on Bun per the napi-rs note. |
bun build --compile |
Works if the .node is reached by a literal require, as native-lock.ts:108-122 does. Bun may extract the addon to a temp file at runtime; TMPDIR must be absolute (Bun issue 45054). Verify with the existing release.yml:175 style assertion. |
| Crash isolation | None. A segfault, abort or uncaught Rust panic kills the gbrain process (CLI, serve, cron). Every exported function needs catch_unwind. |
| Multi-process | Each gbrain process would open the store in-process, so the store needs a single-writer story. The repo already has native/locks flock and the owner-socket routing used for PGLite; both can be reused. Two addons would load in one process (lock addon plus engine addon). |
| Per-call overhead | Lowest of the four: a napi call and marshalling, no syscalls or serialization. |
| Startup | One dlopen per process. No extra process. |
| Effort | Medium: Rust crate, napi bindings for the BrainEngine surface (or a narrow C-ABI-like surface wrapped in TS), prebuild matrix, Bun CI lane, compile smoke. |
| Risk | Medium-high. Crash blast radius; Bun napi gaps for newer APIs; prebuild matrix cost (8 binaries, reproducibility vs cargo output). |
| Parity tests | Run test/e2e/engine-parity.test.ts with the third engine factory, same as PGLite and Postgres; add a native-lock-prebuilds-style test (binary target check, manifest hashes) and a compiled-binary smoke (scripts/native/compiled-smoke.ts pattern). |
Option 2: Rust sidecar process (stdio or unix socket, JSON-RPC or binary framing)
| Aspect | Assessment |
|---|---|
| Bun fit | Pure TS client; no Node-API. Bun child_process / Bun.spawn plus a unix socket. Matches the repo's existing owner-socket and cli-socket design (src/core/cli-socket/server.ts header: "The serve side of the warm CLI socket"; src/core/persistence/local-client.ts). |
| Crash isolation | Full. A Rust crash kills the sidecar, not gbrain; the TS side can restart and reconnect (the repo already has reconnect machinery for Postgres in degraded-engine.ts). |
| Multi-process | Natural single writer: the sidecar owns the store and serializes CLI, MCP and cron requests. This is the main architectural advantage given the repo's current PGLite single-writer lock and owner-socket routing. Requires the sidecar's own lifecycle (start, lock, shutdown, stale socket cleanup, peer-uid checks as in cli-socket/peer-cred.ts). |
| Per-call overhead | Highest. A syscall or socket round trip plus framing per call. JSON is expensive for embeddings (f32 vectors); use length-prefixed binary frames for vectors and bulk page bodies. Not measured. |
| Startup | Per-CLI-invocation cost: spawn the sidecar and open the store, unless the sidecar stays resident (the owner socket pattern). The repo already tunes one-shot reads (engine-connect.ts:34-42), so this matters. Measure before committing. |
bun build --compile |
The sidecar binary must be embedded as a file asset (import ... with { type: 'file' }, the pglite-embedded-assets.ts pattern), extracted or read from /$bunfs, and run with the executable bit preserved. The exec-bit and extraction behavior in compiled binaries is not verified here. |
| Prebuild matrix | Standard Rust cross-compile per target (no Node-API ABI matrix); the output is one executable per platform rather than a .node. Windows and musl still need runners. |
| Effort | Medium-large: protocol design, supervisor, lifecycle, error mapping into opError shapes, reconnect, embed-and-extract for compiled binaries, plus the same CI matrix. |
| Risk | Medium. Latency and marshalling; two processes to keep in sync on version; a second lock domain; sidecar orphaning on crash. |
| Parity tests | Same engine-parity suite via the TS client. Add process-kill and restart tests (the persistence-chaos suites are the model), stale-socket cleanup, and a concurrency test for CLI + MCP + cron against one sidecar. |
Option 3: WASM (wasm32-wasi / wasm-bindgen)
| Aspect | Assessment |
|---|---|
| Bun fit | Mirrors PGLite's model: WASM loaded in-process, assets embedded via with { type: 'file' } (pglite-embedded-assets.ts). No native packaging per platform: one artifact runs everywhere. |
| Filesystem | WASI preview1 preopens work in Bun but are partial and not a security sandbox (Node and Bun docs both say so). No mmap: a storage engine must use linear memory and ordinary file I/O, which constrains page-cache and index design. |
| Locks | WASI has no flock. The single-writer lock in native/locks is a native addon. A WASM engine would need a host import (napi or ffi) for locking or accept single-process use. This is a real gap for the CLI + MCP + cron model. |
| Threads | WASM shared memory and atomics work with workers; WASI itself has no thread creation. Bun supports the shared-memory path, but this is more complex than a native build. |
| Performance | Unknown for this workload; usually slower than native for vector and index work, and no native SIMD guarantees. Not measured. |
| Crash isolation | In-process trap, so a Rust panic is a WASM trap that can still kill the gbrain process unless caught at the boundary. |
| Effort | Large for a storage engine: WASI file semantics, locking, memory limits (wasm32 is 4 GB addressable), and performance work. |
| Risk | High for a storage engine; low risk for pure compute such as a chunker or ranker. |
| Parity tests | Same engine-parity suite; plus WASI file-semantics tests (fsync, rename, lock failure) and a memory-growth test. |
Option 4: bun:ffi with a C ABI
| Aspect | Assessment |
|---|---|
| Bun fit | Bun-only, experimental per Bun's docs. gbrain already uses bun:ffi for small syscalls (src/core/cli-socket/peer-cred.ts:20-40, src/core/pid1-reaper.ts:89-94), so the team has the pattern. |
| Surface | Rust cdylib with a C header (cbindgen) and dlopen(path, { symbols }). Scalar and pointer calls are straightforward; a storage API with owned buffers and handles needs manual free functions. |
| Callbacks | JSCallback is not thread-safe by default; threadsafe: true adds a per-call marshalling cost. Avoid callbacks from Rust threads in the design. |
| Async | No Node-API async-work equivalent; async must be built with Rust threads and polling or a threadsafe callback. |
bun build --compile |
Same embedding question as napi: the .dlopen path must be a file that exists at runtime; extraction needed. Not verified for ffi specifically. |
| Crash isolation | None (same as napi). |
| Effort | Medium for scalars, large for a full BrainEngine. |
| Risk | High: experimental API; Bun may change it; no Node-API stability. |
| Parity tests | Same as napi; add ABI and free-function leak tests. |
Comparison summary
| napi-rs in-process | Sidecar process | WASM | bun:ffi | |
|---|---|---|---|---|
| Bun maturity | Stable N-API, best effort for napi-rs | Not Bun-specific (plain IPC) | WASI partial in Bun | Experimental |
| Crash isolation | None | Full | None (trap) | None |
| Single writer across CLI+MCP+cron | Needs lock addon and owner socket | Natural | Needs host lock (no flock in WASI) | Same as napi |
| Per-call cost | Lowest | Highest | Medium | Low |
| Startup | Lowest | Spawn + open per CLI call unless resident | Compile WASM per process (cache?) | Lowest |
| Prebuild matrix | 8 (or 6+musl) .node per target |
One executable per target | One .wasm |
8 shared libs |
bun build --compile |
Literal require embeds; may extract |
File asset + extract + exec bit | with {type:'file'} (PGLite pattern) |
Like napi |
| Effort | M | M-L | L | M (scalar) to L |
| Risk | Medium-high | Medium | High for storage | High |
Recommendation (open question for the repo owner)
Open question: does the first Rust engine need crash isolation and a single writer across CLI, MCP and cron, or is in-process speed the priority?
- If speed and simplicity matter most: napi-rs in-process. It reuses the proven
.nodeload and compile-embed path innative-lock.ts, has the lowest per-call cost, and fits the Bun-only floor. Cost: a Rust panic or segfault takes the whole gbrain process, so every export must catch panics, and the store needs a single-writer answer (reusenative/locksflock and the owner socket). - If crash isolation and one writer matter most: sidecar. It maps onto the existing owner-socket design, but it adds per-call marshalling and startup cost that must be measured against the one-shot read paths in
engine-connect.ts. - WASM is a poor fit for a storage engine given no mmap and no flock in WASI. bun:ffi is experimental and better kept for small syscalls.
My suggestion, not decided: start with napi-rs in-process, keep the engine surface behind constructEngine, and keep a sidecar as the fallback if crash isolation becomes a requirement. Both need the same engine-parity suite, and both need a Bun CI lane because napi-rs does not block on Bun.
Sources
- Bun Node-API docs: https://bun.com/docs/runtime/node-api (summary read via search; full page not fetched)
- Bun 1.4 release: https://bun.com/blog/bun-v1.4 ; Bun 1.4.1: https://bun.com/blog/bun-v1.4.1
- Bun FFI docs: https://bun.com/docs/runtime/ffi ; https://bun.com/docs/api/ffi.md
- Bun executables / embedding
.node: https://bun.com/docs/compiler/executables ; compile extraction issue: https://github.com/oven-sh/bun/issues/45054 - Bun WASI reference: https://bun.com/reference/node/wasi
- Bun napi async work and threadsafe function teardown: https://github.com/oven-sh/bun/pull/37075 (via search summary)
- napi-rs Bun support: https://napi.rs/docs/more/support-compatibility
- napi-rs cross-build: https://napi.rs/docs/cross-build ; CLI: https://napi.rs/docs/cli/build
Caveat: Bun and napi-rs facts above come from web search summaries of those pages, not from fetching and reading each page in full. The Bun ffi and WASI details were partly summarized from secondary result text. Items marked "not verified" were not tested in this repo or on Bun.
Parity risks noted
- The kind discriminator is a closed union in at least five places (
engine.ts:832,types.ts:2343,config.ts:37,embedded-engine.ts:14,degraded-engine.ts:63, plusengine-status.ts:108). Missing one will type-check or silently fall through. - About 29 kind branches sit in
schema-migrations/, andsqlForonly knowspostgres|pglite. A non-SQL engine must not silently get the Postgres or PGLite SQL. migrate.ts:221treatspgliteas the transactional-migration special case; a third engine's transaction semantics must match.- PGLite single-writer lock and owner-socket routing are the only multi-process coordination; a third engine needs an equivalent or CLI+MCP+cron can double-write.
degraded-engine.tsis Postgres-only; a third engine has no degraded-serve story unless added.engine-open.tscheckSettingsrefuses unknown engines, andopenEngineis the embedding surface; a third engine must be added there or embedders cannot open it.- The
--pgliteinit flag andinit.ts:170-181"preserve existing postgres engine" logic need a parallel rule for the new engine, orinitcan flip an existing brain. docs/ENGINES.md:1566-1596"Adding a new engine" step 4 implies a free-form engine string that the types reject; the doc is out of date with the code.- Compiled binaries: addon and sidecar embedding is verified in source (
native-lock.ts:105-107,release.yml:175) but not for a napi-rs addon or a file-asset sidecar on this repo's targets. - Release binaries build only
linux-x64anddarwin-arm64(release.yml:121-131), while the lock addons cover 8 targets; a Rust engine would need an explicit target policy. - napi-rs Bun support is best effort upstream; without a repo-owned Bun CI lane on each target, a Bun regression can ship unseen.